At the second International Symposium on Multimodal Communication (MMSYM 2024) in Frankfurt, Bielefeld Professor Dr Petra Wagner, project leader of A02, will give a keynote speech. The lecture on “The multimodal expression of (non-)understanding in dyadic explanations - some lessons learned” will focus on the research of the A02 project.
During an explanation, explainers rely on the explainee's feedback regarding their understanding and cognitive load or attention. Based on a variety of verbal and non-verbal feedback, explainers can dynamically adapt their strategy by, for example, changing the pace, repeating or omitting parts of the explanation or shifting the focus of the explanation.
'In my talk, I will report insights from the TRR318 „Constructing Explainability“ subproject A02 on „Monitoring the understanding of explanations“, in which we gather and investigate multimodal signals of (non-)understanding in explanations, see how they evolve in course of ongoing explanations, and how they are interpreted and reacted to,’ explains Professor Wagner.
Among other things, the recording and rich multimodal annotation of a corpus of 87 dyadic board game explanations will be discussed. Furthermore, a recall task will be used to show more information about the annotation of different levels of (non-)comprehension. Another thematic focus will be the dynamics of floor management across different phases of the explanations. Professor Wagner also explains: ‘I will present some insights on how explainers adapt their multimodal behavior to different explainees and show how verbal and non-verbal information are combined in a model for classifying understanding and non-understanding respectively. Throughout, I will also address the various challenges we were faced with.’
Petra Wagner is Professor of Phonetics and Head of the Phonetics Research Group at Bielefeld University. Her research focuses on phonetics, prosody and multimodal prosody as well as speech synthesis, conversational speech acts and human-machine interaction. The symposium will take place from 25 to 27 September 2024.
