November 6-7, 2023 at Paderborn University

Unpacking understanding in interaction: Video studies of technologies in use

In this talk, I’ll show how qualitative video analysis can respond to some of the methodological challenges in studying ‘understanding’ and ‘explanation’ in interaction. I’ll briefly present the approach drawing on ethnomethodology and conversation analysis, which has been extensively applied to study human-machine interaction and contributed to the development of interdisciplinary fields such as HCI and CSCW.

Then, I will draw on past and current empirical studies undertaken with colleagues to discuss current topics and foundational concepts and suggest ways to re-specify understanding and explanation as continuous, collaborative accomplishments. I will unpack how, for example, recipient design in the shaping of embodied action, through its publicly available features, sheds light on participants’ understanding of each other’s emerging conduct and interactional competencies. I will also discuss what different data collection methods allow; and conclude with remaining challenges and future directions in  the light of current developments of technologies in specialised settings.

Cognitive skills: the building blocks of human intelligence

Humans have the amazing capacity to perform new tasks with little or no instruction. To explain this remarkable ability, I propose that people, when faced with a new task, compose the necessary knowledge for that task using cognitive skills as building blocks. In our cognitive modeling research, we have shown how a small set of skills can instantiated into a variety of task models, and provide explanations for phenomena such as attentional blink and task switching costs without having to rely on assumptions about limitations of the brain. 

If cognitive skills are the building blocks of cognition, is important to be able to identify them, and study how they are learned. To identify cognitive skills, we use a hybrid approach, in which we use bottom-up machine learning methods to use individual differences in student performance to construct a knowledge graph, in which each node represents a combination of skills, and a possible knowledge state of the student.


As a pilot, we constructed a knowledge graph for an arithmetic course in the mid-level vocational education (MBO) in the Netherlands. The basis for this graph was an math entry test, which, according to the publisher, addressed several specific topics, such as length measurements, weight, clock time, etc. However, when we constructed a knowledge graph from data from 2480 students, we found that students do not differ on mastery of those topics, but rather on more general underlying skills, such as general arithmetic skills, reading skills and multi-step reasoning. 

In order to assess whether the analysis of learning materials can improve learning, I will report on a pilot study where we give advice to students on the basis of their knowledge state.