An Integrated Multimodal Approach to Emotion and Empathy

Empathy is a fundamental component of human social life, supporting connection, cooperation, and emotional understanding. Despite its central role in human functioning, empathy remains complex and not fully understood, particularly in dynamic, real-world contexts where emotions unfold across multiple channels simultaneously. Much of the existing literature relies on simplified tasks and single-modality measures, limiting our understanding of how empathy operates as an embodied and interactive process. The present dissertation addresses this gap by adopting a multimodal approach to examine how neural, behavioral, and physiological signals jointly contribute to emotion perception and empathic responding. The three studies in this dissertation were designed to capture the dynamic and multimodal nature of emotional experience and empathy while progressively increasing ecological validity. Study 1 focused on subjective emotional experience and tested whether affective states could be decoded from portable EEG recordings using classical machine learning approaches. Study 2 extended this framework by incorporating facial expression dynamics alongside EEG activity to examine how individuals perceive and resonate with others’ emotions. Study 3 further expanded this work by introducing a dyadic, real-life interaction paradigm and adding autonomic physiological measures, allowing for the examination of empathy as it unfolds between interaction partners in real time. Across these studies, results demonstrate that emotion and empathy cannot be fully captured by any single modality. Instead, neural activity, expressive behavior, and physiological responses provide complementary and partially overlapping information about emotional states. The integration of these modalities improves the prediction of emotional experience and highlights individual differences in how people rely on different channels during emotion perception. By demonstrating the value of combining neural, behavioral, and physiological measures, this dissertation contributes to a more comprehensive understanding of empathy and offers insights with potential applications in domains such as psychotherapy, education, and human-centered technology, where accurate emotional understanding is essential.