Enhancing Automated Interview Performance Evaluation Using Multimodal Emotion Detection and Retrieval-Augmented Generation
Abstract
Conventional interview assessment is based on human perception, tending towards subjectivity, variability, and inefficiencies, particularly when evaluating large groups of candidates. To overcome these drawbacks, we introduce an AI-based automated interview assessment platform that combines multimodal emotion recognition with Retrieval-Augmented Generation (RAG) to deliver objective, data-driven analysis. Our method evaluates verbal (text-based answers), non-verbal (facial reactions, gaze, and body language), and paralinguistic (speech tone, pitch, and clarity) signals to provide a complete assessment of interview performance. By utilizing GPT-3.5 with LLaMA Index, the model reads and places key findings from interview transcripts into context to provide comprehensive and explainable feedback. Moreover, OpenFace and Wav2Vec are utilized for face expression and speech emotion recognition that are combined utilizing a hybrid multimodal approach in order to do a strong assessment. The assessment is done via a Total Score metric, examining candidates on understanding content (30%), confidence, and fluency (20%), emotional control (20%), non-verbal communication (15%), and paralinguistic features (15%). Experimental validation on the MIT Interview Dataset shows that the system proposed here enhances assessment accuracy, fairness, and consistency, outperforming human evaluators in delivering standardized reviews. Such discoveries show the promise of multimodal AI-driven…
Read the paper
Related studies
- Knowledge Management Perspective of Generative Artificial Intelligence
- Navigating Generative Artificial Intelligence Promises and Perils for Knowledge and Creative Work
- Characterizing the evolving research on enterprise content management
- Large language models and multimodal foundation models for precision oncology
- From Generative AI to Extended Reality: Multidisciplinary Perspectives on the Challenges, Opportunities, and Future of Educational Computing
- Generative Artificial Intelligence: A New Engine for Advancing Environmental Science and Engineering
Go deeper in Evidence Hub
Evidence Hub adds a study data sheet (design, sample size, evidence level, key numbers), an AI summary and a chat grounded in the paper's text.