A journal of IEEE and CAA , publishes high-quality papers in English on original theoretical/experimental research and development in all areas of automation
Advanced Search
Volume 9 Issue 9
Sep.  2022

IEEE/CAA Journal of Automatica Sinica

  • JCR Impact Factor: 11.8, Top 4% (SCI Q1)
    CiteScore: 17.6, Top 3% (Q1)
    Google Scholar h5-index: 77， TOP 5
Turn off MathJax
Article Contents
Article Navigation >  IEEE/CAA Journal of Automatica Sinica  > 2022  >  9(9): 1612-1626
Y. X. Wang, S. Qiu, D. Li, C. D. Du, B.-L. Lu, and H. G. He, “Multi-modal domain adaptation variational autoencoder for EEG-based emotion recognition,” IEEE/CAA J. Autom. Sinica, vol. 9, no. 9, pp. 1612–1626, Sept. 2022. doi: 10.1109/JAS.2022.105515
Citation: Y. X. Wang, S. Qiu, D. Li, C. D. Du, B.-L. Lu, and H. G. He, “Multi-modal domain adaptation variational autoencoder for EEG-based emotion recognition,” IEEE/CAA J. Autom. Sinica, vol. 9, no. 9, pp. 1612–1626, Sept. 2022. doi: 10.1109/JAS.2022.105515
shu

Multi-Modal Domain Adaptation Variational Autoencoder for EEG-Based Emotion Recognition

doi: 10.1109/JAS.2022.105515
  • 1.

    Research Center for Brain-inspired Intelligence, National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Science, Beijing 100190, and the University of Chinese Academy of Sciences, Beijing 100049

  • 2.

    Beijing Institute of Control and Electronic Technology, Beijing 100038, China

  • 3.

    School of Mathematics and Information Sciences, Yantai University, Yantai 264003, China

  • 4.

    Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Science, Beijing, China

  • 5.

    Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai 200240, China

Funds:  This work was supported in part by National Natural Science Foundation of China (61976209, 62020106015, U21A20388); in part by the CAS International Collaboration Key Project (173211KYSB20190024); and in part by the Strategic Priority Research Program of CAS (XDB32040000)
More Information
    Abstract
  • Traditional electroencephalograph (EEG)-based emotion recognition requires a large number of calibration samples to build a model for a specific subject, which restricts the application of the affective brain computer interface (BCI) in practice. We attempt to use the multi-modal data from the past session to realize emotion recognition in the case of a small amount of calibration samples. To solve this problem, we propose a multi-modal domain adaptive variational autoencoder (MMDA-VAE) method, which learns shared cross-domain latent representations of the multi-modal data. Our method builds a multi-modal variational autoencoder (MVAE) to project the data of multiple modalities into a common space. Through adversarial learning and cycle-consistency regularization, our method can reduce the distribution difference of each domain on the shared latent representation layer and realize the transfer of knowledge. Extensive experiments are conducted on two public datasets, SEED and SEED-IV, and the results show the superiority of our proposed method. Our work can effectively improve the performance of emotion recognition with a small amount of labelled multi-modal data.

     

    • FullText(HTML)
  • loading
    • References(53)
  • [1]
    S. Spence, “Descartes’ error: Emotion, reason and the human brain,” BMJ, vol. 310, no. 6988, p. 1213, 1995.
    [2]
    B. Shneiderman, C. Plaisant, M. Cohen, S. Jacobs, N. Elmqvist, and N. Diakopoulos, Designing the User Interface: Strategies for Effective Human-Computer Interaction. Pearson, 2016.
    [3]
    P. Ekman, “Expression and the nature of emotion,” Approaches to emotion, vol. 3, pp. 19–344, 1984.
    [4]
    Z. Zeng, M. Pantic, G. I. Roisman, and T. S. Huang, “A survey of affect recognition methods: Audio, visual, and spontaneous expressions,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 31, no. 1, pp. 39–58, 2009. doi: 10.1109/TPAMI.2008.52
    [5]
    M. Coulson, “Attributing emotion to static body postures: Recognition accuracy, confusions, and viewpoint dependence,” Journal of Nonverbal Behavior, vol. 28, no. 2, pp. 117–139, 2004. doi: 10.1023/B:JONB.0000023655.25550.be
    [6]
    H. Gunes, B. Schuller, M. Pantic, and R. Cowie, “Emotion representation, analysis and synthesis in continuous space: A survey,” in Proc. IEEE Face and Gesture, pp. 827–834, 2011.
    [7]
    R. A. Calvo and S. D’Mello, “Affect detection: An interdisciplinary review of models, methods, and their applications,” IEEE Trans. Affective Computing, vol. 1, no. 1, pp. 18–37, 2010. doi: 10.1109/T-AFFC.2010.1
    [8]
    F. Lotte, M. Congedo, A. Lécuyer, F. Lamarche, and B. Arnaldi, “A review of classification algorithms for EEG-based brain–computer interfaces,” Journal of Neural Engineering, vol. 4, no. 2, p. R1, 2007.
    [9]
    J. Zhang, Z. Yin, P. Cheng, and S. Nichele, “Emotion recognition using multi-modal data and machine learning techniques: A tutorial and review,” Information Fusion, 2020.
    [10]
    M. Soleymani, S. Asghari-Esfeden, Y. Fu, and M. Pantic, “Analysis of EEG signals and facial expressions for continuous emotion detection,” IEEE Trans. Affective Computing, vol. 7, no. 1, pp. 17–28, 2015.
    [11]
    Y. Lu, W.-L. Zheng, B. Li, and B.-L. Lu, “Combining eye movements and EEG to enhance emotion recognition,” in Proc. 24th Int. Joint Conf. Artificial Intelligence, 2015.
    [12]
    H. Ranganathan, S. Chakraborty, and S. Panchanathan, “Multimodal emotion recognition using deep learning architectures,” in Proc. IEEE Winter Conf. Applications of Computer Vision, pp. 1–9, 2016.
    [13]
    C. Mühl, B. Allison, A. Nijholt, and G. Chanel, “A survey of affective brain computer interfaces: Principles, state-of-the-art, and challenges,” Brain-Computer Interfaces, vol. 1, no. 2, pp. 66–84, 2014. doi: 10.1080/2326263X.2014.912881
    [14]
    D. H. Lee and A. K. Anderson, “Reading what the mind thinks from how the eye sees,” Psychological Science, vol. 28, no. 4, pp. 494–503, 2017. doi: 10.1177/0956797616687364
    [15]
    M. Soleymani, M. Pantic, and T. Pun, “Multi-modal emotion recognition in response to videos,” IEEE Trans. Affective Computing, vol. 3, no. 2, pp. 211–223, 2011.
    [16]
    K. Weiss, T. M. Khoshgoftaar, and D. Wang, “A survey of transfer learning,” Journal of Big Data, vol. 3, no. 1, p. 9, 2016.
    [17]
    T. Baltrušaitis, C. Ahuja, and L.-P. Morency, “Multimodal machine learning: A survey and taxonomy,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 41, no. 2, pp. 423–443, 2018.
    [18]
    E. Principi, D. Rossetti, S. Squartini, and F. Piazza, “Unsupervised electric motor fault detection by using deep autoencoders,” IEEE/CAA J. Autom. Sinica, vol. 6, no. 2, pp. 441–451, 2019. doi: 10.1109/JAS.2019.1911393
    [19]
    L. Pang, S. Zhu, and C.-W. Ngo, “Deep multi-modal learning for affective analysis and retrieval,” IEEE Trans. Multimedia, vol. 17, no. 11, pp. 2008–2020, 2015. doi: 10.1109/TMM.2015.2482228
    [20]
    W. Liu, W.-L. Zheng, and B.-L. Lu, “Emotion recognition using multi-modal deep learning,” in Proc. Int. Conf. Neural Information Processing, pp. 521–529, Springer, 2016.
    [21]
    P. Tzirakis, G. Trigeorgis, M. A. Nicolaou, B. W. Schuller, and S. Zafeiriou, “End-to-end multi-modal emotion recognition using deep neural networks,” IEEE Journal of Selected Topics in Signal Processing, vol. 11, no. 8, pp. 1301–1309, 2017. doi: 10.1109/JSTSP.2017.2764438
    [22]
    X. Kang, F. Ren, and Y. Wu, “Exploring latent semantic information for textual emotion recognition in blog articles,” IEEE/CAA J. Autom. Sinica, vol. 5, no. 1, pp. 204–216, 2017.
    [23]
    J. Chen, X. Luo, and M. Zhou, “Hierarchical particle swarm optimization-incorporated latent factor analysis for large-scale incomplete matrices,” IEEE Trans. Big Data, 2021. DOI: 10.1109/TBDATA.2021.3090905
    [24]
    W.-L. Zheng and B.-L. Lu, “Personalizing EEG-based affective models with transfer learning,” in Proc. 25th Int. Joint Conf. Artificial Intelligence, pp. 2732–2738, AAAI Press, 2016.
    [25]
    P. Sinno Jialin, I. W. Tsang, J. T. Kwok, and Y. Qiang, “Domain adaptation via transfer component analysis,” IEEE Trans. Neural Networks, vol. 22, no. 2, pp. 199–210, 2011. doi: 10.1109/TNN.2010.2091281
    [26]
    B. Schölkopf, A. Smola, and K.-R. Müller, “Nonlinear component analysis as a kernel eigenvalue problem,” Neural Computation, vol. 10, no. 5, pp. 1299–1319, 1998. doi: 10.1162/089976698300017467
    [27]
    R. Collobert, F. Sinz, J. Weston, L. Bottou, and T. Joachims, “Large scale transductive SVMs,” Journal of Machine Learning Research, vol. 7, no. 1, pp. 1687–1712, 2006.
    [28]
    E. Sangineto, G. Zen, E. Ricci, and N. Sebe, “We are not all equal: Personalizing models for facial expression analysis with transductive parameter transfer,” in Proc. ACM Int. Conf. Multimedia, pp. 357–366, 2014.
    [29]
    X. Chai, Q. Wang, Y. Zhao, Y. Li, D. Liu, X. Liu, and O. Bai, “A fast, efficient domain adaptation technique for cross-domain electroencephalography (EEG)-based emotion recognition,” Sensors, vol. 17, no. 5, p. 1014, 2017.
    [30]
    Y. Li, W. Zheng, Y. Zong, Z. Cui, T. Zhang, and X. Zhou, “A bihemisphere domain adversarial neural network model for EEG emotion recognition,” IEEE Trans. Affective Computing, vol. 12, no. 2, pp. 494–504, 2021. doi: 10.1109/TAFFC.2018.2885474
    [31]
    J. Li, S. Qiu, C. Du, Y. Wang, and H. He, “Domain adaptation for EEG emotion recognition based on latent representation similarity,” IEEE Trans. Cognitive and Developmental Systems, vol. 12, no. 2, pp. 344–353, 2020. doi: 10.1109/TCDS.2019.2949306
    [32]
    D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” arXiv preprint arXiv: 1312.6114, 2013.
    [33]
    T. Shen, T. Lei, R. Barzilay, and T. Jaakkola, “Style transfer from nonparallel text by cross-alignment,” in Advances in Neural Information Processing Systems, pp. 6830–6841, 2017.
    [34]
    M.-Y. Liu, T. Breuel, and J. Kautz, “Unsupervised image-to-image translation networks,” in Advances in Neural Information Processing Systems, pp. 700–708, 2017.
    [35]
    M. Shang, X. Luo, Z. Liu, J. Chen, Y. Yuan, and M. Zhou, “Randomized latent factor model for high-dimensional and sparse matrices from industrial applications,” IEEE/CAA J. Autom. Sinica, vol. 6, no. 1, pp. 131–141, 2018.
    [36]
    X. Luo, Z. Liu, S. Li, M. Shang, and Z. Wang, “A fast non-negative latent factor model based on generalized momentum method,” IEEE Trans. Systems,Man,and Cybernetics: Systems, vol. 51, no. 1, pp. 610–620, 2021. doi: 10.1109/TSMC.2018.2875452
    [37]
    D. Wu, Y. He, X. Luo, and M. Zhou, “A latent factor analysis-based approach to online sparse streaming feature selection,” IEEE Trans. Systems,Man,and Cybernetics: Systems, 2021. DOI: 10.1109/TSMC.2021.3096065
    [38]
    M. Wu and N. Goodman, “Multi-modal generative models for scalable weakly-supervised learning,” in Advances in Neural Information Processing Systems, pp. 5575–5585, 2018.
    [39]
    G. E. Hinton, “Training products of experts by minimizing contrastive divergence,” Neural Computation, vol. 14, no. 8, pp. 1771–1800, 2002. doi: 10.1162/089976602760128018
    [40]
    W.-L. Zheng, W. Liu, Y. Lu, B.-L. Lu, and A. Cichocki, “Emotionmeter: A multi-modal framework for recognizing human emotions,” IEEE Trans. Cybernetics, vol. 49, no. 3, pp. 1110–1122, 2018.
    [41]
    W.-L. Zheng, J.-Y. Zhu, and B.-L. Lu, “Identifying stable patterns over time for emotion recognition from EEG,” IEEE Trans. Affective Computing, vol. 10, no. 3, pp. 417–429, 2019. doi: 10.1109/TAFFC.2017.2712143
    [42]
    M. S. AL-Quraishi, I. Elamvazuthi, T. B. Tang, A.-Q. Muhammad, S. Parasuraman, and A. Borboni, “Multi-modal fusion approach based on EEG and EMG signals for lower limb movement recognition,” IEEE Sensors Journal, vol. 21, no. 24, pp. 27640–37650, 2021. doi: 10.1109/JSEN.2021.3119074
    [43]
    E. Tzeng, J. Hoffman, T. Darrell, and K. Saenko, “Simultaneous deep transfer across domains and tasks,” in Proc. IEEE Int. Conf. Computer Vision, pp. 4068–4076, 2015.
    [44]
    M. Long, Y. Cao, J. Wang, and M. Jordan, “Learning transferable features with deep adaptation networks,” in Proc. Int. Conf. Machine Learning, pp. 97–105, 2015.
    [45]
    Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and V. Lempitsky, “Domain-adversarial training of neural networks,” The Journal of Machine Learning Research, vol. 17, no. 1, pp. 2096–2030, 2016.
    [46]
    M. Long, H. Zhu, J. Wang, and M. I. Jordan, “Deep transfer learning with joint adaptation networks,” in Proc. 34th Int. Conf. Machine Learning-Volume 70, pp. 2208–2217, JMLR. org, 2017.
    [47]
    P. Haeusser, T. Frerix, A. Mordvintsev, and D. Cremers, “Associative domain adaptation,” in Proc. IEEE Int. Conf. Computer Vision, pp. 2765–2773, 2017.
    [48]
    M. Long, Z. Cao, J. Wang, and M. I. Jordan, “Conditional adversarial domain adaptation,” in Advances in Neural Information Processing Systems, pp. 1640–1650, 2018.
    [49]
    M.-Y. Liu and O. Tuzel, “Coupled generative adversarial networks,” in Advances in Neural Information Processing Systems, pp. 469–477, 2016.
    [50]
    C. Yu, J. Wang, Y. Chen, and M. Huang, “Transfer learning with dynamic adversarial adaptation network,” in Proc. IEEE Int. Conf. Data Mining, pp. 778–786, 2019.
    [51]
    R. Horlings, D. Datcu, and L. J. Rothkrantz, “Emotion recognition using brain activity,” in Proc. 9th Int. Conf. Computer Systems and Technologies and Workshop for Ph.D. Students in Computing, 2008.
    [52]
    S. Katsigiannis and N. Ramzan, “Dreamer: A database for emotion recognition through EEG and ECG signals from wireless low-cost off-the-shelf devices,” IEEE Journal of Biomedical and Health Informatics, vol. 22, no. 1, pp. 98–107, 2017.
    [53]
    M. Liu, D. Fan, X. Zhang, and X. Gong, “Human emotion recognition based on galvanic skin response signal feature selection and SVM,” in Proc. Int. Conf. Smart City and Systems Engineering, pp. 157–160, 2016.
    • Related Articles
    • Supplements(0)

Catalog

    通讯作者: 陈斌, bchen63@163.com
    • 1. 

      沈阳化工大学材料科学与工程学院 沈阳 110142

    1. 本站搜索
    2. 百度学术搜索
    3. 万方数据库搜索
    4. CNKI搜索

    Figures(6)  / Tables(15)

    Get Citation
    PDF
    XML

    Article Metrics

    Article views (684) PDF downloads(147) Cited by()

    Highlights

    • Using multi-modal data to realize emotion recognition under small samples condition
    • MMDA-VAE learns cross-domain latent representations and solves missing modality problem
    • Adversarial loss and cycle-consistency loss reduce cross-domain distribution difference
    • Performance on two public EEG/eye movement datasets proved the method's superiority

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return