Psychophysiological Approaches for Assessing User States and Performance in Interactive Entertainment

Abstract

Psychophysiological approaches offer a powerful means to investigate the cognitive and emotional states that shape user experience, capturing both transient reactions and sustained dynamics in digital game environments. However, knowledge remains limited regarding how psychophysiological methods can be systematically applied to study user experience in games, spanning issues of methodology, ecological validity, analytical approaches, and integration with other measurement tools. This research investigated how multimodal psychophysiological signals can be used to evaluate user experience in terms of cognitive and emotional states, and to predict performance across diverse game contexts. In addition, the research examined how psychophysiological measures can be triangulated with self-reports and behavioural data to provide convergent and complementary insights. This investigation was realised through a multimodal psychophysiological approach, integrating ocular, cardiac, and neural measures with behavioural data, self-reports, and interviews to capture user experience in a comprehensive manner. I operationalised this approach across four empirical studies, each targeting different facets of user experience in interactive games. The first study (N=15) applied self-reports and behavioural observation to assess dynamic changes in emotional states and in-game actions under increasing difficulty, showing that higher difficulty elicited heightened arousal and negative emotions while also revealing substantial individual variability in coping strategies. To probe the cognitive mechanisms underlying these behavioural differences, the second study (N=35) employed eye-tracking to identify systematic differences in visual attention strategies between high- and low-performing players, establishing visual strategies as quantifiable indicators of cognitive processes underlying performance variation. Moving from explanation to prediction, the third study (N=35) integrated ocular and cardiac signals to prospectively predict performance outcomes across sessions. The models achieved reliable accuracy, demonstrating that anticipatory information about performance trajectories is embedded in the early dynamics of interaction. Extending beyond challenge-based play, the final study (N=13) employed EEG to examine narrative interaction mechanics, showing that mechanics requiring complex cognition and affective appraisal elicited stronger neural activity and underscoring the role of interactive structure, beyond narrative content, in shaping engagement and affective experience. In addition, I triangulated behavioural observations, physiological signals, self-reports, and interviews, integrating implicit physiological reactions with explicit self-appraisals and overt actions to offer a more holistic perspective on the multidimensional nature of experience.

This research demonstrates that multimodal psychophysiological signals provide robust means to evaluate cognitive and emotional states and to anticipate performance outcomes in digital games, thereby advancing approaches for user state assessment in interactive systems. It provides an integrated empirical account of attention, engagement, affective responses, and performance in interactive game contexts. It also provides practical implications for the use of multimodal assessment in anticipatory user analytics, supportive feedback systems, and narrative design. Moreover, this work shows that effective multimodal assessment depends both on careful signal selection and on systematic triangulation across complementary measurement methods. In doing so, it advances the systematic integration and ecologically grounded application of psychophysiological approaches in interactive systems. Show more