Real-time emotion recognition for sales

Research output: Chapter in Book/Report/Conference proceedingConference article in proceedingsScientificpeer-review

7 Citations (Scopus)
217 Downloads (Pure)


Positive emotion is a pre-condition to any sales contract. Likewise, the ability to perceive the emotions of a customer impacts sales performance.To support emotional perception in buyer-seller interactions, we propose an audio-visual emotion recognition system that can recognize eight emotions: neutral, calm, sad, happy, angry, fearful, surprised, and disgusted. We reduced noise in audio samples and we applied transfer learning for image feature extraction based on a pre-trained deep neural network VGG16. For emotion recognition, we successfully obtained an audio emotion-recognition accuracy of 62.51% and 68% and video emotion-recognition accuracy of 97.13% and 97.77% on the Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS) and Surrey Audio-Visual Expressed Emotion (SAVEE) datasets respectively. For the combination of the two models, our proposed merging mechanism without re-training achieved an accuracy of close to 100% on both datasets. Finally, we demonstrated our system for a customer satisfaction use case in a real customer-to-salesperson interaction using audio and video models, achieving an average accuracy of 78%.

Original languageEnglish
Title of host publicationProceedings of 16th International Conference on Mobility, Sensing and Networking (MSN 2020)
Number of pages8
ISBN (Electronic)978-1-7281-9916-0
Publication statusPublished - Apr 2021
MoE publication typeA4 Conference publication
EventInternational Conference on Mobility, Sensing and Networking - Tokyo, Japan
Duration: 17 Dec 202019 Dec 2020
Conference number: 16


ConferenceInternational Conference on Mobility, Sensing and Networking
Abbreviated titleMSN


  • Customer satisfaction
  • Deep learning
  • Emotion recognition
  • Internet of Things
  • Transfer learning


Dive into the research topics of 'Real-time emotion recognition for sales'. Together they form a unique fingerprint.

Cite this