Mutual Minds

How people and intelligent systems learn to adapt to each other

Cars, cockpits, production lines and AI conversation partners now predict and adapt to the people who use them. People adapt in return. Mutual Minds studies this two-way adaptation: when it stays stable and useful, when it breaks down, and how to design systems that respect human limits.

Mutual Minds is a research initiative of the Human-IST Institute at the University of Fribourg. It brings together two research platforms, MindCraft (the body: perception, gaze and movement) and Harbour (conversation), and the Specialised Master in Digital Neuroscience, which trains students on both.

Five research questions

We study one loop: an operator and an operating environment, linked by an adaptive interface, over timescales from milliseconds to years.

  1. Regulating perception and action. How do people control what they see and do when a system acts with them in a closed loop?

  2. Signals of adaptation. Which measurable signals show that a person is adapting, becoming unstable or losing a sense of control?

  3. Internal models. How do people's expectations of a system change as the system becomes more predictive and autonomous?

  4. Timescales. How do breakdowns and learning unfold from seconds to months?

  5. Design. How can adaptive environments stay within human cognitive limits?
Diagram: an operator and an operating environment linked by an adaptive interface. Action flows from operator to environment, perception flows back. A timescale bar runs from milliseconds to years. Numbered badges place the five research questions on the loop.

The platforms: MindCraft and Harbour

Diagram of the two platforms. MindCraft: an operator and a simulator, recording eye tracking, EEG, skin conductance, heart activity, head motion and motor telemetry; projects U+, MAP-Drive, HABIT and SkyMind. Harbour: a speaker and an AI partner, recording one audio track per speaker, prosody, voice quality, speech acts and turn-taking over weeks of practice; projects PRISM and three studies. A shared pipeline below: time-aligned recording, behavioural tokens, perception-action cycles, comparable models.

MindCraft

Embodied interaction and sensorimotor regulation

Driving and flight simulators with time-aligned recording of eye movements, EEG, skin conductance, heart activity, head motion and control inputs. We measure how people control and adapt their actions, and how workload, uncertainty and skill change under pressure.

Domains: mobility, aviation, manufacturing, extended reality.

Harbour

Communication, coordination and social regulation

A conversation-training platform where people practise structured dialogue with adaptive AI partners. Each speaker is recorded on a separate audio track, so we can analyse turn-taking, prosody and coordination phrase by phrase, and follow how a person's way of communicating develops over weeks.

Domains: education, teamwork, accessibility.

  • Projects

    U+: behavioural tokens. Our shared method. U+ turns continuous signals (gaze, movement, brain and body signals) into discrete behavioural tokens and perception-action cycles, so that driving, flying, assembly work and conversation can be compared in the same terms. The long-term aim is active-inference models of the operator.

    MAP-Drive: driving. How do drivers learn to anticipate as a route becomes familiar? 66 participants drove a 14-segment urban circuit while we recorded six signals at 500 Hz (415 aligned recordings). First results: the road segment can be decoded from behaviour with 71.7% balanced accuracy (four classes, chance 25%).

    HABIT: Industry 5.0. Human-centred AI for behaviour learning on a real production line at ICAM Strasbourg-Europe, fusing eye tracking, motion sensors, video and neuroscience annotations. A PhD project with the University of Strasbourg, ICAM and the iCube laboratory, funded by the ENACT AI Cluster, co-supervised by Dr Samy Rima and Asst. Prof. Rabih Amhaz.

    SkyMind: aviation. Applies U+ in a flight simulator to study how pilots regulate their actions, manage workload and keep situational awareness.

  • Projects

    PRISM: conversation analysis. A pipeline that describes every utterance on three levels: its form (question, statement), its purpose (informing, requesting, agreeing, hedging) and its sound (pitch, rhythm, voice quality, speaking rate). Aligning what is said with how it is said makes the mechanics of conversation measurable.

    Study 1: Reading the signals. Which changes in the voice go with different conversational strategies, for example hedging an opinion or warming up to a topic? We look for reliable, real-time indicators of how speakers adjust to their partner.

    Study 2: Mapping conversational growth. We follow the same people over weeks of practice and measure the range and flexibility of how they communicate, not only what they say.

    Study 3: Conversations that adapt. We use the indicators from Study 1 and the profiles from Study 2 to adjust the difficulty of a practice conversation as it happens, and test whether this speeds up learning.

Teaching: Digital Neuroscience

Students of the Specialised Master in Digital Neuroscience work directly on MindCraft and Harbour: they run experiments, analyse multimodal data and design adaptive systems. Several master's theses come from Mutual Minds projects.

Selected Outputs

Publications

  • Bussard, A., Roosch, L., Schmid, M. C., & Rima, S. (2025). MAP-DRIVE: Optimizing driver monitoring systems through perception-action sequences and neurophysiological insights. NAT'25 Proceedings, 50–53. BTU Cottbus-Senftenberg. doi:10.26127/BTUOpen-7215
  • Gurbanov, K., Almakdessi, N., Kacimi, F., Bobenrieth, C., Chabrol, G., Rima, S., & Amhaz, R. (2026). HABIT: An AI-driven multimodal framework for human-centric productivity in Industry 5.0. FAIM 2026, Lecture Notes in Mechanical Engineering, Springer (in press).

In preparation

  • MAP-Drive: a multimodal dataset of urban driving behaviour.
  • Scanpath reorganisation during route familiarisation in urban driving.
  • Pressure-state tokenisation of motor signatures in simulated driving.

Datasets

  • MAP-Drive: multimodal driving dataset (EEG, eye tracking, skin conductance, heart activity, head motion, motor telemetry).
  • Harbour: longitudinal corpus of practice conversations (collection ongoing).

Team

Dr. Samy Rima

Initiative lead, Principal Investigator
Lecturer and Senior Researcher, Human-IST
Coordinator of the Digital Neuroscience master. Leads MindCraft and Harbour.

Prof. Denis Lalanne

Co-PI, Full Professor, Director of Human-IST
Human-computer interaction, multimodal interfaces, visual analytics, human-AI teaming.

Prof. Michael C. Schmid

Co-PI, Full Professor, University of Fribourg
Visual neuroscience, neural dynamics, attention and perception.

Ass. Prof. Rabih Amhaz

Co-PI, University of Strasbourg and ICAM
Signal processing, computational modelling, applied mathematics.

Sofia Panagiotakou

MSc student, Digital Neuroscience

Vithusan Ramalingam

MSc student, Digital Neuroscience

Lukas Maurer

MSc student, Medicine

Alumni (MindCraft): Alessia Bussard, Lorenzo Roosch, Zeynep Aydin, Celine Rohrer, Emile Alyev.

Work with us

We welcome research partners, companies that want to host a thesis, and students looking for a project. Write to samy.rima@unifr.ch or book a 30-minute call.

Institutionnal Partners

University of Fribourg · Human-IST Institute · University of Strasbourg · ICAM · iCube Laboratory · ENACT AI Cluster