Multimodal transformer and directed relational graph learning for emotion recognition in conversations with potential applications in college student mental health research
Abstract
Multimodal emotion recognition in conversations aims to identify the emotion expressed in each utterance by jointly analyzing textual, acoustic, visual, and contextual information. However, existing methods still face challenges in coordinating intra-modal dependency modeling, cross-modal interaction, and graph-based conversational reasoning. To address these challenges, this paper proposes MTG-ERC, a multimodal conversational emotion recognition framework that integrates transformer-based attention mechanisms with directed relational graph learning. First, textual, acoustic, and visual features are projected into a unified representation space. Intra-modal self-attention and cross-modal attention are then employed to capture modality-specific contextual dependencies and complementary interactions among modalities. Subsequently, a directed multi-relational graph module combines relational graph convolution and Graph Transformer operations to model temporal context, speaker-related interactions, and multimodal dependencies among utterances. Finally, global attention-based representations and local graph-based contextual representations are integrated for utterance-level emotion classification. Experiments on the IEMOCAP and MELD datasets show that MTG-ERC achieves competitive performance compared with the evaluated conversational emotion recognition baselines. Ablation results further indicate that intra-modal attention, cross-modal interaction, and directed relational…
Read the paper
Related studies
- Chaos and quantum thermalization
- The Dalton quantum chemistry program system
- Cord Blood Leptin and Adiponectin as Predictors of Adiposity in Children at 3 Years of Age: A Prospective Cohort Study
- Quantitative metabolomics by H-NMR and LC-MS/MS confirms altered metabolic pathways in diabetes
- Stable adiabatic quantum batteries
- A novel self-lipid antigen targets human T cells against CD1c(+) leukemias
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.