10.57647/j.mjee.2025.1902.29

Hybrid Attention-based Deep Learning Network For Emotion Recognition by ECG Signal

PDF
  • Mahtab Vaezi1,
  • Mehdi Nasri*2,3,
  • Farhad Azimifar1,
  • Mahdi Mosleh4,
  1. Department of Biomedical Engineering, Isfahan (Khorasgan) Branch, Islamic Azad University Isfahan, Iran
  2. Department of Biomedical Engineering, Isfahan (Khorasgan) Branch, Islamic Azad University, Isfahan, Iran
  3. Efficiency and Smartization of Energy Systems Research Center, Khomeinishahr Branch, Islamic Azad University, Isfahan, Iran
  4. Department of Computer Engineering, Isfahan (Khorasgan) Branch, Islamic Azad University Isfahan, Iran

Received: 2025-03-12

Revised: 0025-02-24

Accepted: 2025-04-26

Published in Issue 2025-06-01

Copyright (c) 2025 Mahtab Vaezi, Mehdi Nasri, Farhad Azimifar, Mahdi Mosleh (Author)

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

How to Cite

Vaezi, M., Nasri, M., Azimifar, F., & Mosleh, M. (2025). Hybrid Attention-based Deep Learning Network For Emotion Recognition by ECG Signal. Majlesi Journal of Electrical Engineering, 19(2 (June 2025). https://doi.org/10.57647/j.mjee.2025.1902.29
Crossref
Scopus
Google Scholar
Europe PMC

PDF views: 106

Abstract

Emotions play an important role in our daily activities, decision-making, and artificial intelligence needs to identify emotions to interact constructively with its audience. In this paper, an intelligent method for two-dimensional emotion recognition is proposed. The ECG signal available in the DREAMER database has been used to recognize emotions because of the high correlation of this signal with emotions and easy recording. First step for valence and arousal recognition, the ECG signal is entered into the deep learning network, which is a combination of CNN and LSTM. CNN performs feature extraction and LSTM performs data classification. The attention mechanism aims to optimize the weights and improve the performance of the network, overseeing the proposed deep learning network. Using the proposed method, valence and emanation were identified with 95% and 94% accuracy, respectively. The proposed hybrid network is very suitable for high-dimensional data, and the use of the attention mechanism helps to improve the performance of the network by preventing overfit and getting stuck in local optimal.

Keywords

  • Emotion,
  • CNN,
  • LSTM,
  • ECG,
  • Attention Mechanism
PDF

References

  1. T. Baltrušaitis, C. Ahuja, and L.-P. Morency, "Multimodal machine learning: A survey and taxonomy," IEEE transactions on pattern analysis machine intelligence, vol. 41, no. 2, pp. 423-443, 2018. https://doi.org/10.1109/TPAMI.2018.2798607
  2. N. Ahmed, Z. Al Aghbari, and S. Girija, "A systematic survey on multimodal emotion recognition using learning algorithms," Intelligent Systems with Applications, vol. 17, p. 200171, 2023. https://doi.org/10.1016/j.iswa.2022.200171
  3. S.-W. Byun, J.-H. Kim, and S.-P. Lee, "Multi-modal emotion recognition using speech features and text-embedding," Applied Sciences, vol. 11, no. 17, p. 7967, 2021. https://doi.org/10.3390/app11177967
  4. Z. Lian, Y. Li, J.-H. Tao, J. Huang, and M.-Y. Niu, "Expression analysis based on face regions in real-world conditions," International Journal of Automation Computing, vol. 17, pp. 96-107, 2020. https://doi.org/10.1007/s11633-019-1176-9
  5. J. A. Russell, "A circumplex model of affect," Journal of personality social psychology, vol. 39, no. 6, p. 1161, 1980. https://doi.org/10.1037/h0077714
  6. A. Geetha, T. Mala, D. Priyanka, and E. Uma, "Multimodal Emotion Recognition with deep learning: advancements, challenges, and future directions," Information Fusion, vol. 105, p. 102218, 2024. https://doi.org/10.1016/j.inffus.2023.102218
  7. C. E. Izard, "The many meanings/aspects of emotion: Definitions, functions, activation, and regulation," Emotion Review, vol. 2, no. 4, pp. 363-370, 2010. https://doi.org/10.1177/1754073910374661
  8. C. Kauschke, D. Bahn, M. Vesker, and G. Schwarzer, "The role of emotional valence for the processing of facial and verbal stimuli—positivity or negativity bias?," Frontiers in psychology, vol. 10, p. 444703, 2019. https://doi.org/10.3389/fpsyg.2019.01654
  9. S. Kim and S.-P. Lee, "A BiLSTM–Transformer and 2D CNN Architecture for Emotion Recognition from Speech," Electronics, vol. 12, no. 19, p. 4034, 2023. https://doi.org/10.3390/electronics12194034
  10. D. Avola et al., "Spatio-Temporal Image-Based Encoded Atlases for EEG Emotion Recognition," International Journal of Neural Systems, vol. 34, no. 5, p. 2450024, 2024. http://dx.doi.org/10.1142/S0129065724500242
  11. Y. Chen, H. Zhang, J. Long, and Y. Xie, "Temporal shift residual network for EEG-based emotion recognition: a 3D feature image sequence approach," Multimedia Tools Applications, vol. 83, no. 15, pp. 45739-45759, 2024. https://doi.org/10.1007/s11042-023-17142-7
  12. N. Kumari and R. Bhatia, "Saliency map and deep learning based efficient facial emotion recognition technique for facial images," Multimedia Tools Applications, vol. 83, no. 12, pp. 36841-36864, 2024. https://doi.org/10.1007/s11042-023-16220-0
  13. M. K. Chowdary, T. N. Nguyen, and D. J. Hemanth, "Deep learning-based facial emotion recognition for human–computer interaction applications," Neural Computing Applications, vol. 35, no. 32, pp. 23311-23328, 2023. https://doi.org/10.1007/s00521-021-06012-8
  14. K. Korovai, D. Zhelezniakov, O. Yakovchuk, O. Radyvonenko, N. Sakhnenko, and I. Deriuga, "Handwriting Enhancement: Recognition-Based and Recognition-Independent Approaches for On-device Online Handwritten Text Alignment," IEEE Access, 2024. https://doi.org/10.1109/ACCESS.2024.3412433
  15. S. Zhang, Y. Yang, C. Chen, X. Zhang, Q. Leng, and X. Zhao, "Deep learning-based multimodal emotion recognition from audio, visual, and text modalities: A systematic review of recent advancements and future prospects," Expert Systems with Applications, vol. 237, p. 121692, 2024. https://doi.org/10.1016/j.eswa.2023.121692
  16. P. Chakraborty, S. Ahmed, M. A. Yousuf, A. Azad, S. A. Alyami, and M. A. Moni, "A human-robot interaction system calculating visual focus of human’s attention level," IEEE Access, vol. 9, pp. 93409-93421, 2021. https://doi.org/10.1109/ACCESS.2021.3091642
  17. S. Mishra, N. Bhatnagar, P. P, and S. T. R, "Speech emotion recognition and classification using hybrid deep CNN and BiLSTM model," Multimedia Tools Applications, vol. 83, no. 13, pp. 37603-37620, 2024. https://doi.org/10.1007/s11042-023-16849-x
  18. M. Aslan, M. Baykara, and T. B. Alakuş, "Analysis of brain areas in emotion recognition from EEG signals with deep learning methods," Multimedia Tools Applications, vol. 83, no. 11, pp. 32423-32452, 2024. https://doi.org/10.1007/s11042-023-16696-w
  19. M. Vaezi, M. Nasri, F. Azimifar, and M. Mosleh, "BMPA-DSL: Binary Marine Predators Algorithm to Identify Driver’s Different Levels of Stress," presented at the 2023 14th International Conference on Information and Knowledge Technology (IKT), 2023. https://doi.org/10.1109/IKT62039.2023.10433031
  20. A. Dessai and H. Virani, "Multimodal and Multidomain Feature Fusion for Emotion Classification Based on Electrocardiogram and Galvanic Skin Response Signals," Sci, vol. 6, no. 1, p. 10, 2024. https://doi.org/10.3390/sci6010010
  21. J. Cheng et al., "Emotion recognition from multi-channel EEG via deep forest," IEEE Journal of Biomedical Health Informatics, vol. 25, no. 2, pp. 453-464, 2020. https://doi.org/10.1109/JBHI.2020.2995767
  22. M. Vaezi, M. Nasri, F. Azimifar, and M. Mosleh, "Driver Assistance System for Stress Recognition by Handcrafted Feature Extraction and Convolutional Neural Network," in 2024 20th CSI International Symposium on Artificial Intelligence and Signal Processing (AISP), 2024, pp. 1-5: IEEE. https://doi.org/10.1109/AISP61396.2024.10475225
  23. T. Fan et al., "A new deep convolutional neural network incorporating attentional mechanisms for ECG emotion recognition," Computers in Biology Medicine, vol. 159, p. 106938, 2023. https://doi.org/10.1016/j.compbiomed.2023.106938
  24. P. Chakraborty, M. A. Yousuf, and S. Rahman, "Predicting level of visual focus of human’s attention using machine learning approaches," in Proceedings of International Conference on Trends in Computational and Cognitive Engineering: Proceedings of TCCE 2020, 2021, pp. 683-694: Springer. https://doi.org/10.1007/978-981-33-4673-4_56
  25. M. J. Al Dujaili, A. Ebrahimi-Moghadam, and A. Fatlawi, "Speech emotion recognition based on SVM and KNN classifications fusion," International Journal of Electrical Computer Engineering, vol. 11, no. 2, p. 1259, 2021. https://doi.org/0.11591/ijece.v11i2.pp1259-1264
  26. X. Liu, X. Cheng, and K. Lee, "GA-SVM-based facial emotion recognition using facial geometric features," IEEE Sensors Journal, vol. 21, no. 10, pp. 11532-11542, 2020. https://doi.org/10.1109/JSEN.2020.3028075
  27. M. A. Hasnul, N. A. Ab. Aziz, and A. Abd. Aziz, "Augmenting ECG Data with Multiple Filters for a Better Emotion Recognition System," Arabian Journal for Science Engineering, pp. 1-22, 2023. https://doi.org/10.1007/s13369-022-07585-9
  28. Q. Gao, C.-h. Wang, Z. Wang, X.-l. Song, E.-z. Dong, and Y. Song, "EEG based emotion recognition using fusion feature extraction method," vol. 79, no. 37, pp. 27057-27074, 2020. https://doi.org/10.1007/s11042-020-09354-y
  29. X. Wang, J. Zhang, C. He, H. Wu, and L. Cheng, "A novel emotion recognition method based on the feature fusion of single-lead EEG and ECG signals," IEEE Internet of Things Journal, 2023. https://doi.org/10.1109/JIOT.2023.3320269
  30. S. Wang, J. Qu, Y. Zhang, and Y. Zhang, "Multimodal emotion recognition from EEG signals and facial expressions," IEEE Access, vol. 11, pp. 33061-33068, 2023. https://doi.org/10.1109/ACCESS.2023.3263670
  31. M. Vaezi and M. Nasri, "AS3-SAE: Automatic Sleep Stages Scoring using Stacked Autoencoders," Frontiers in Biomedical Technologies‏, pp. 400-416, 2023. https://doi.org/10.18502/fbt.v10i4.13722
  32. S. Katsigiannis and N. Ramzan, "DREAMER: A database for emotion recognition through EEG and ECG signals from wireless low-cost off-the-shelf devices," IEEE journal of biomedical health informatics, vol. 22, no. 1, pp. 98-107, 2017. https://doi.org/10.1109/JBHI.2017.2688239
  33. S. N. M. S. Ismail, N. A. A. Aziz, and S. Z. Ibrahim, "A comparison of emotion recognition system using electrocardiogram (ECG) and photoplethysmogram (PPG)," Journal of King Saud University-Computer Information Sciences, vol. 34, no. 6, pp. 3539-3558, 2022. https://doi.org/10.1016/j.jksuci.2022.04.012
  34. M. U. Hossain, M. A. Rahman, M. M. Islam, A. Akhter, M. A. Uddin, and B. K. Paul, "Automatic driver distraction detection using deep convolutional neural networks," Intelligent Systems with Applications, vol. 14, p. 200075, 2022. https://doi.org/10.1016/j.iswa.2022.200075
  35. R. Yamashita, M. Nishio, R. K. G. Do, and K. Togashi, "Convolutional neural networks: an overview and application in radiology," Insights into imaging, vol. 9, pp. 611-629, 2018. https://doi.org/10.1007/s13244-018-0639-9
  36. Q. Zhang, T. Gao, X. Liu, and Y. Zheng, "Public environment emotion prediction model using LSTM network," Sustainability, vol. 12, no. 4, p. 1665, 2020. https://doi.org/10.3390/su12041665
  37. Z. Liu, W. Zhou, and H. Li, "AB-LSTM: Attention-based bidirectional LSTM model for scene text detection," ACM Transactions on Multimedia Computing, Communications, Applications, vol. 15, no. 4, pp. 1-23, 2019. https://doi.org/10.1145/3356728
  38. S. M. Alarcao and M. J. Fonseca, "Emotions recognition using EEG signals: A survey," IEEE Transactions on Affective Computing, vol. 10, no. 3, pp. 374-393, 2017. https://doi.org/10.1109/TAFFC.2017.2714671
  39. Y.-L. Hsu, J.-S. Wang, W.-C. Chiang, and C.-H. Hung, "Automatic ECG-based emotion recognition in music listening," IEEE Transactions on Affective Computing, vol. 11, no. 1, pp. 85-99, 2017. https://doi.org/10.1109/TAFFC.2017.2781732
  40. S. Nita, S. Bitam, M. Heidet, and A. Mellouk, "A new data augmentation convolutional neural network for human emotion recognition based on ECG signals," Biomedical Signal Processing Control, vol. 75, p. 103580, 2022. https://doi.org/10.1016/j.bspc.2022.103580
  41. C. M. T. Khan, N. A. Ab Aziz, J. E. Raja, S. W. B. Nawawi, and P. Rani, "Evaluation of machine learning algorithms for emotions recognition using electrocardiogram," Emerging Science Journal, vol. 7, no. 1, pp. 147-161, 2022. http://dx.doi.org/10.28991/ESJ-2023-07-01-011