Efecan Yılmaz, Neural and Ocular Correlates of Conceptual Grounding in Verbal Interaction: A Multimodal Hyperscanning Approach

Ph.D. Candidate: Efecan Yılmaz
Program: Cognitive Science
Date: 05.09.2024 / 15:00
Place: A-108

Abstract: Effective communication depends on interlocutors agreeing on how to resolve ambiguities and constructing conceptual representations into shared meanings. To achieve this, continuous adaptation by all participants to establish and to maintain alignments is required, which is known as grounding in communication. Previous work on grounding has largely explored linguistic and ocular phenomena, whereas neural correlates have not been explicitly studied. In this thesis, hyperscanning and verbal interaction analysis approaches are combined to investigate the process of establishing a common ground – a social aspect of communication – using a multi-modal approach that includes EEG, eye tracking, and fNIRS devices.

Eye-tracking results indicated that with establishment of common ground, shared gaze becomes less central with explicit verbal communication taking precedence. Gaze entropy reduces as interlocutors move from high workload visual feature extraction to utilizing stable common meanings. EEG results showed that increased inter-brain synchrony in the frontal and parietal electrode sites in delta and theta bands, which aligns with sustained attention and memory processes that correlate with shared linguistic contexts in communication. The synchrony observed over different frequency bands highlights the collaborative nature of communication and the role of parietal electrode sites in facilitating effective communication. fNIRS hyperscanning results were inconclusive due to device malfunctions. However, single device hemodynamic response analysis revealed significant effects of repeated trials and experiment conditions on PFC oxygenation, supporting the role of this region in interaction and communication, particularly in relation to working memory and communicative intent. The findings reveal that dyadic communication involves multiplex synchrony, relying on both verbal and non-verbal cues as they established a common ground, with both neural and ocular underpinnings.