Temporal Lag Effects in Multimodal Learning Analytics: Physiological–Behavioral Characterization

Abstract

Multimodal learning analytics typically aligns physiological and behavioral streams synchronously, ignoring the temporal lag between physiological response and overt behavior. Across six students collecting 1-second stress and 5-second heart-rate streams with face-attention inference, we find that lag variability — not mean lag — is what tracks learning outcomes, and the relationship is stronger in active labs than passive lectures.

Problem & Motivation

Multimodal learning analytics typically integrates visual attention and physiological signals to infer learner state, but uses synchronous alignment or simple feature stacking. The temporal-lag characteristics between physiological response and overt behavior across modalities are under-investigated — yet this cross-modal time-delay structure likely carries important information about the learning process.

Method

We recruited 6 students for a three-week multimodal data-collection study in a Python programming course. Using the Garmin vívoactive 5 watch with the Garmin Health Companion SDK we acquired 1-second-resolution stress index (HRV-derived) and 5-second heart-rate data, while a camera captured facial imagery for attention inference. Temporal alignment plus cross-correlation analysis quantified physiological–behavioral lag features at the instructional-event level.

Findings

• Mean temporal lag shows only weak association with academic performance (stress r = 0.20, heart rate r = 0.09).
• Lag variability is moderately positively correlated with performance (stress r = 0.55, heart rate r = 0.52) — a stronger discriminator than mean lag.
• Active lab sessions show stronger physiological–behavioral coupling than passive lecture sessions.
• Learners able to modulate their engagement pattern by context show better performance, consistent with self-regulated learning theory.

Implications

Instructors can use real-time coupling indicators to rebalance activity types: encourage tighter physiological–behavioral coupling during labs while tolerating looser coupling during lectures. Lag variability offers a new, interpretable metric for multimodal learning analytics, enabling more fine-grained instructional adaptation and learning support.

Citation

@inproceedings{chang2026temporal_lag,
  author    = {Yan-Yu Chang and Yu-Zhen Chai and Chia-Kai Chang},
  title     = {Temporal Lag Effects in Multimodal Learning Analytics: Physiological--Behavioral Characterization},
  booktitle = {Proc. IEEE Int. Conf. on Advanced Learning Technologies (ICALT)},
  year      = {2026},
  month     = jul,
}