Keywords

1 Introduction

Cyber-Physical Systems (CPS) [1] refer to environments that are sensitive and responsive to people; they integrate a variety of devices operating in concert to support human activities in an unobtrusive way, using intelligence that is hidden in the network connecting them. Designing a CPS implies planning for user interactions that are contextual and open-ended, triggered by the unrestricted activity of the users within the environment; therefore, CPS design is influenced by user-centric methods where the user is placed at the centre of the design activity and asked to give feedback through evaluations and tests to improve the design, or even co-create the design with a design team [2]. Pavlovic [2] observes an enlargement of current user experience (UX) design practices towards the design of CPSes and proposes Ambient UX as a novel suitable approach for the emerging applications. Ambient UX approach, thus, targets the design of physical environments in which activities are enhanced by digitized services.

Within the research field, UX has been studied by various authors that have been proposing approaches towards reasoning on UX beyond mere usability. Norman [3] talks about “emotional design” in terms of visceral, behavioural, and reflective experiences, while Hassenzahl [4] underlines that products have hedonic attributes, besides the pragmatic ones (e.g., an ability to evoke feelings of pleasure). Desmet [5] provides a conceptual model for emotional responses that results from the perception of consumer products. In this context, MacDonald and Atwood  [6] suggest that the exploration of UX evaluation methods, implying both pragmatic and hedonic dimensions, is a valuable research direction for real-world interaction design projects, as it is recognized that usability is not enough.

Pragmatic and hedonic dimensions of UX can be observed as correspondent to “what” and “why”, where the main aim is to understand the “why” behind the “what” [7]. This is to say that investigating experiences is a complex quest and requires merging of diverse inquiry methods as well as diverse professional fields that conduct investigations. Such professional fields that tackle research on UX span from Human-Computer Interaction to Design of Services, Interfaces and Spaces, as well as Marketing and Communication. All of these fields investigate diverse dimensions of experience; bringing together diverse methods of inquiry could help to comprehend UX in a more holistic manner and facing its complexity accordingly. In UX Design an in-depth investigation of complexity of the human experience, with respect to the interaction with spaces and contexts, is a key issue, suitable to provide insights and suggestions in all phases of the design process, from the conception of innovative solutions to the progressive improvement of existing systems and services [8]. Evidence-based investigation of users’ experience provides inspiration in the design of new solutions [9, 10]; it supports the reduction of the mismatch between the value proposition as it is conceived by designers with respect to users [8] provides information about the users’ efforts in their interaction with the adaptive ambient and their satisfaction [11]; it provides awareness about the diverse human attitudes, and supports interoperability, consistency and coherence of connected systems [12].

This paper aims to respond to the need of facing UX investigation in a holistic manner, by proposing a methodology for investigating the experience of users of CPSes based on quadrangulation of methods borrowed from diverse fields of practices that shown to be efficient for comprehending hedonic dimensions of experiences, i.e. users’ affective responses. The major contribution of the current research lies in the methodological integration, firstly adopting a theoretical stance and secondly providing a discussion of the approach as applied at the PHEEL (PHysiology, Emotion, and Experience, Lab) research laboratory (which authors are part of), based on case-studies developed in collaborations with industries.

The present work proposes quadrangulation of diverse levels of analysis for investigating systematically different facets of UX. The here presented approach is based on the assumption that the crossing of information, provided by monitoring of physiological parameters with the record of the behaviours of users and their subjective perception of the interactive process with environments and contexts, can provide meaningful insights about the “how” of the experience and subsequently hints on the “why”. The paper presents the methods and an overview of some meaningful application. Each presented method is compared in terms of methodological specificities including time accuracy, instrumentation invasiveness, the possibility to assess top-down or bottom-up user responses, and the role of the researcher. From such an appraisal, a comprehensive framework is drafted highlighting applicability best practices, critical success factors and limitations associated with each method.

The discussion is centred on the importance of the experimental model in terms of scientific reliability, validity and transparency associated with the four layers of user experience analysis as provided by the experimental studies.

2 Framework of Reference

The present work encompasses an initial overview of the methods employed to investigate systematically different facets of UX. The overview summarises the scientific background of the experimental activities performed by the PHEEL Lab, a research laboratory created in collaboration by the Departments of Design, Management Engineering and Bioengineering within Politecnico di Milano, and focused on multi-purposes analysis of user experience. Investigation approaches are conceived as belonging to four layers of analysis, namely (i) physiological, (ii) behavioural, (iii) self-reported, and (iv) expert evaluation. After an illustration of the four layers’ perspective and tools, a methodological comparison is presented to discuss quadrangulation modalities.

2.1 Physiological Level of Analysis

Physiological analyses are intended to gauge user reactions based on responses related to either the individual’s central or peripheral nervous system activity. In its broadest sense, this level of investigation encompasses both a focus on the cognitive as well as the affective user responses [13]. The theoretical foundation of this level is rooted in the established mind-body relationship, which posits that for every person “every change in the physiological state is accompanied by an appropriate change in the mental-emotional state, conscious or unconscious, and conversely, every change in the mental-emotional state, conscious or unconscious, is accompanied by an appropriate change in the physiological state” [14]. According to this principle, which stems from early research on physiological responses specificity [15], by means of the measurement of the physiological state of a person, information about her psychological state can be assessed.

This line of thought is not novel. Indeed, evidence both in communication and behavioural research is found in previous literature. For instance, Kroeber-Riel [16] investigated individual’s psychobiological activations to test the efficacy of advertising or Weinberg and Gottwald [17] conceptualised the assessment of perceived arousal during impulse buying processes through electrodermal activations. However, thanks to the technological and scientific developments, the last decade has seen a rising interest in the adoption of tools for assessing users’ reactions, beyond the clinical setting. The fields of consumer neuroscience and cognitive psychology provide exemplary evidence [18, 19]. The former specifically aims at understanding instinctive conscious and unconscious human behaviours in response to marketing stimuli to ultimately provide information about the underlying consumer’s cognitive and affective mechanisms and improve behavioural predictions [20, 21].

The added value of the physiological layer of analysis lays in the possibility of gauging user’s unconscious information about their preferences, overcoming fundamental limitations related to traditional research methods, such as surveys or interviews. Indeed, such methods of investigation based on self-reporting may be impaired by biases related to self-deception, social desirability or acquiescence [20, 22]. By doing so, these analysis techniques are expected to support the early product or service design providing accurate and quantifiable insights into the user experience itself [23].

Concretely, the physiological level of analysis includes the analysis of physical responses such as (i) cortical and sub-cortical activations, as a direct quantitative measure of the central nervous system activity; (ii) cardiac and respiratory responses, as an expression of sympathetic and parasympathetic bodily activations; (iii) electrodermal activity, as an manifestation of sympathetic activations; (iv) muscular responses, related the muscle action potential produced by motor units; (v) peripheral temperature, as changes in skin temperature of the human body; (vi) pupil dilation, as the variation in the size of the eye pupil; and (vii) other biometric methods related to the measurement of nervous system-related activity (e.g. salivary or lacrimal glands responses).

In non-clinical settings, such analysis is employed for the investigation of a vast amount of cognitive and affective processes as, for instance, user experienced engagement during the use of a mobile app [24] or decision-making dynamics [25]. Overall, previous literature has shown that through the analysis of physiological activations it is possible to assess individual’s exogenous and endogenous attention [26], affect [27], memory-related processes [28], and approach-withdrawal tendencies [29], arousal and stress [30].

2.2 Behavioural Level of Analysis

Behavioural responses encompass information related to instinctive nonverbal responses of the user. Contrastingly to physiological responses that gauge internal bodily reactions related to the nervous system activity, the behavioural level aims at assessing user involuntary expressions and behavioural exteriorisation. The study of nonverbal behaviours has been the centre of interest of various disciplines ranging from social psychology to anthropology [31], and from psychopathology [32] to sociology and marketing [33].

A specific fil rouge that bonds such distant fields is found in the analysis of the person’s affective states. Indeed, significant evidence shows how expressions of affect can be conveyed through nonverbal cues [34] or that deception may be linked to specific nonverbal correlates [35]. Following such a line of thought, behavioural analyses result to be an alternative way to overcome biases related to self-reports in-field interviews. The observation of nonverbal behaviours enables descriptions of user behaviours both in social and individual settings, thus providing further sources of information about the individual’s intentions.

As conceived by the present research, the behavioural level of analysis is focused on user responses such as (i) facial expressions, as the combinations of movements of facial landmarks such as eyes, eyebrows, lips, mouth, nose, and cheeks; (ii) gaze behaviour, as the analysis of the typologies of eye contact, ocular movements, gaze direction, and blink rates; (iii) paralinguistic cues, including the investigation of vocal pitch, loudness or amplitude, pitch pattern variations or pauses; (iv) proxemics, namely the postural patterns and structuring of space adopted from the user; (v) kinesics, as the body movements including head and limbs moves; and (vi) interaction behaviours, as the actions performed by the user during an interaction including touch and haptic behaviours.

Previous literature has broadly investigated nonverbal behaviours and various codings have been proposed [36]. Facial expression analysis may be performed through different methods (i.e. experts coding based on the Facial Action Coding System, by means of surface electromyography or through real-time software analyses). Despite the assessment methodology, facial expression analysis is commonly adopted to investigate user reactions in terms of arousal and valence, generally related to universal emotions [37].

Eye movement and gaze direction are classified according to voluntarily or involuntarily moves such as eye direction, focus change, or objects following [38]. Gaze analysis is commonly employed to track visual saliency during space exploration or during an interaction with a digital interface [39]. In addition, blink rates, as the measure of the frequency of eye blinks has often related to stress-related indexes [40].

Verbal communication features are an additional parameter considered. Specifically, paralinguistic cues as vocal pitch, intensity, frequency, glottal characteristics, and speech rate are among the common parameters analysed [36, 40]. These extracted features have been related to affective states such as frustration, annoyance, tiredness, or amusement classifiable based on valence and arousal [41] or potential stress levels [42].

Proxemics and kinesics represent a broad cluster of non-verbal cues where the physical behaviour is exercised to express or convey information. Such behaviour embraces body poses, posture, hand movements as well as body motion. Common metrics include distance, body orientation, touch, limb static positions or movements to provide inference on user engagement, disengagement or distress [36].

Lastly, interaction behaviours embrace the relational aspects occurring between the user, the object of investigation and the environment. Interaction behaviours may occur by means of devoted input peripherals (e.g. keyboard, mouse, touchscreen in case of a digital interface), or by means of specific event recording. Interactive patterns provide information about the user’s understanding of an artefact as well as affective responses related to the interaction.

2.3 Self-reported Level of Analysis

Self-reported analyses are a family of research approaches based on personal statements stemming either from a conversation between the user and the researcher or as written user’s opinions. This level of analysis represents a widely adopted methodology in social sciences to investigate user’s understanding, opinions, attitudes or feelings [43]. Self-reports are commonly supported by a constructivist stance, positing that individual’s understanding is not objective. Therefore, these research approaches are intended to investigate user’s rationalisation of phenomena [44, 45].

Differently from the physiological and behavioural levels of analysis, the self-reported process grants an opportunity to explore the meanings that individuals give the object of interaction. Indeed, this layer aims at studying the content of thoughts and to support the respondent to articulate perceptions and understandings otherwise not observable. Such process proves specifically valuable in case of longitudinal research, where self-reports are collected over a period of time [43].

In its essence, the self-reported layer encompasses: (i) interviews, intended as a conversation between the user and an interviewer on a specific topic; (ii) surveys and questionnaires, as a collection of written questions; (iii) think aloud, referred to the analysis of the user’s verbalisations during a task; and (iv) focus groups, as inquiries taking place in a group setting. Interviews consist of a process where the interviewer works directly with the respondent and seeks to understand the central themes of the user experience. Such a practice may be carried out in either a structured, semi-structured, or unstructured way [46].

Surveys and questionnaires encompass a set of structured queries provided in written form. Such methods of investigation include both open and closed questions which participants are required to answer in an individual manner. Personal answers do commonly guarantee the responder’s anonymity especially in case of sensitive and personal topics of research. Overall, surveys and questionnaires result suitable for probability sampling and represent a tool associated with easier study replicability [45].

Think aloud investigations represent a process where the individual is asked to verbalise her thought while being involved in a task [47]. It consists of a source of information on the user’s cognitive processes, which are gathered during a specific interaction, object of the study. However, unlike other methods for verbal data gathering, the participant is neither interrupted, nor suggestive prompts are provided. Instead, the subject is encouraged to express her understanding and interpretation of the task at hand.

Lastly, focus groups consist of group interviews with a heterogeneous set of respondents. Participants are asked for their perceptions, opinions, or attitudes towards the object of investigation in a way that they are open to discuss with peers under the guidance of a moderator [48]. The added value related to focus groups stems from the group interaction to novel idea generation, as respondents build on peers’ comments. Specifically, the technique proves valuable to extract insights about complex issues and topics.

2.4 Expert Evaluation

The three levels of analysis described in the previous paragraphs refer to methods and techniques that can provide information on the experience of people interacting with CPSes, in experiments where users are directly involved and monitored while performing free activities or goal-oriented actions.

The fourth level of the quadrangulation scheme is the one we refer to as expert evaluation, and it provides the framework for the creation of suitable experiments and data processing, aimed at generating design hints and opportunities for the CPS. Expert here is a design professional that manages with his/her sensibility, developed over time and through experience working on projects, to model a CPS with respect to the possible user paths enabled by the environment and technologies, and conceptual description of the activities enabled/constrained by the characteristics of the system.

The final aim of our investigation is the production of knowledge that supports the design and re-design of CPSes with respect to the fulfilment of user needs and expectations, with an emphasis on the freedom of action and resilience. The description of a CPS in terms of tasks and paths of users [8], flows of activities and processes, and the envisioning of structural dimensions of the physical and digital architectures [2] is a preliminary activity in the investigation of experience, and the benchmark for the interpretation of data provided by experiments. For this reason, expert evaluation is crucial for tailoring an evaluation process related to experiences. Such professional figure is knowledgeable about the whole design process flow as well as the aspects of a product-service system that can be redesigned for supporting desirable experiences. This level of analysis provides insights within an overall context of usage, potential expectations from the usage, as well as the adequacy of the design concept for undertaking foreseen activities. The insights provided by the experiments with users can confirm and enrich the interpretation and expectations of the experts, or they can offer different and unforeseen perspectives on possible experiences related to the interaction with a CPS emerging from the quadrangulation process.

Roots of this method that we employ within the work of our research group can be found among UX research methods referred to as heuristic evaluation. Author Nielsen [49, 50] defines heuristic evaluation as a “usability engineering method for finding the usability problems in a user interface design so that they can be attended to as part of an iterative design process”. In this context, such evaluation implies having a number of usability evaluators within the research team that can analyse the interface according to recognized usability principles, which are referred to as heuristics.

Usability principles, i.e. heuristics, according to Nielsen [50] are: (1) visibility of system status, (2) match between system and the real world, (3) user control and freedom, (4) consistency and standards, (5) error prevention, (6) recognition rather than recall, (7) flexibility and efficiency of use, (8) aesthetic and minimalist design, (9) helping users recognize, diagnose, and recover from errors, (10) help and documentation. It is to note, however, that evaluators can develop further category-specific heuristics according to peculiarities of a design product that is being analysed. One of such examples is performing a competitive analysis among existing products within a specific category, and further extracting principles that would support the definition of usability problematics [51].

Heuristic evaluation principles refer to usability aspects of an interface, while in our research group we expand the expert evaluation on a broader level considering experiences beyond mere usability problematics. This is due to the nature of projects that the group deals with, where projects of interaction design go beyond graphical user interfaces and shape complex cyber-physical service systems. Role of a design professional as an evaluator is to identify all the problematic and/or potentially dominant aspects of design that can influence user experience. Such evaluation brings to a further definition of a study protocol for investigating UX according to very specific design aspects and features, even those not among the recognized heuristics.

In this context, expert evaluation refers to conducting an analysis of a design product from the side of design professionals, in order to identify a further study protocol. This activity ensures having a tailored UX research methodology according to the very specific design in question, which comprises of some of the previously described physiological, behavioural, and self-reported methods. After defining a protocol and conducting studies, a design professional performs another activity by translating the research insights into potential design recommendations.

The scope of identifying a necessity for an activity such as expert evaluation, within the process of investigation of user’s experience, is to underline the importance of including a design professional within the research methodology. Such a professional is capable of detecting problematics and features with design potential in order to further improve/re-design a product-service system by bridging the research and design process.

The four described layers of analysis (physiological, behavioural, self-reported, expert evaluation) are parts of a quadrangulation strategy for investigating users’ experiences. Table 1 in Sect. 3.1 on quadrangulation strategies summarises the structure of each layer and the related information provided.

3 Methodological Considerations

Each presented perspective and measurement tool demonstrate specificities in terms of skills and knowledge, methodological procedure and context of use requirements. However, the present discussion intentionally leaves aside punctual considerations related to each tool for the sake of framework generalization. Accordingly, we develop a broad discussion that encompasses applicability features of each layer of analysis. Our argumentation begins with the comparison of the four layers in terms of temporal accuracy, instrumentation invasiveness (i.e. the necessity to potentially insert disturbing elements in the analysed experience), possibility to assess instinctive (i.e. top-down or bottom-up) user responses, and the role of the researcher. Next, we draft possible quadrangulation strategies combining each layer’s features.

3.1 Methodological Perspective Appraisal

Each layer provides a different lens of analysis (Table 1); each one is characterised by a specific level of impact on users (as an instance, eye-tracking and the monitoring of physiological parameters require the user to wear instruments that in some circumstances can affect the experience; video recording and direct observation of contexts are subjected to different constraints of privacy depending on the location of the experiments), and some require calibration and settings that should be carefully handled to interfere with the normal development of events. In terms of temporal accuracy, more precisely the possibility to assess real-time information, the physiological layer grants continuous data gathering. Temporal information may range from milliseconds (in case of cortical responses gathered through EEG) to minutes (in relation to peripheral temperature analyses). Such information may be collected prior, during and after the experience, in the event of long-term physiological recordings. Despite the temporal accuracy, the physiological recording is commonly bounded to the physical application of measuring instrumentation to the body of the subject, which may restrain movements either completely (in the case of sub-cortical inferences) or partially (in the event of electrodermal activity measurements). On this line of thought, physiological measurement tools imply a high degree of invasiveness. Furthermore, the physiological layer of analysis allows the possibility to measure instinctive user responses, related to physiological involuntary responses. Such a possibility is provided in the face of a fairly actively involved researcher that must monitor physiological responses during each experimental session. In general terms, we argue that the physiological layer is characterised by a robust temporal accuracy, a high degree of invasiveness, a significant possibility of gathering user’s instinctive responses and a fairly active role of the researcher.

By the same token, the behavioural level of analysis shows good temporal accuracy. Gaze behaviour or facial expressions are commonly accounted in the range of seconds, as well as proxemics and kinesics that are frequently assessed in time windows of a few seconds. Information is commonly gathered during the experience through less invasive technology which, for the large majority, is not in contact with the user’s body (such as ambient cameras, infrared cameras, or audio-recording devices). Like the physiological layer, behavioural analyses grant the possibility to assess user’s instinctive and involuntary responses in the face of a fairly active researcher that should often actively code user responses during the investigation. Based on such reasons, we posit that the behavioural level shows an average level of temporal accuracy, low invasiveness, a significant possibility of collecting instinctive responses and a fairly active role of the researcher.

During experiments, the accuracy of timing for physiological and behavioural analysis is crucial for the aim of producing information about the correlation between the events, the activities, the stimuli enacting actions, and the reactions that take place in the context.

Conversely, the self-reported layer of analysis shows distinct features. Despite specific methodologies that allow collecting self-reported responses of the user during the experience (e.g. think aloud), self-reported measures commonly are assessed either prior or after the experience. Self-reported analyses investigate the experience rationalisation of the user, providing a time-lag between the user action and the data collection. As concerns the experience invasiveness, self-reports do not constitute any concern, since they do not insert any element of potential disturb. Furthermore, this layer of analysis does not provide, by its nature, insights on the instinctive user responses rather it shed light on the user’s cognitive understanding. Self-reported investigations do require, however, a committed role of the researcher who has often to actively conduct interviews or moderate focus groups discussions.

Table 1. Confrontation of four layers of analysis for investigating user experience.
Full size table

Expert evaluation as an activity is not defined by a certain timeframe, but within the overall study methodology is meant to save time by identifying specific design features that the study should focus on. This level of analysis engages in two phases of the study, in the very initial phase when the study protocol is being defined, and in the final phase when it comes to interpreting research results. This level of analysis differs from the other three as it does not involve direct feedback from the users, rather it relies on the expertise of the design professional. Therefore, the design professional in this case takes on a user’s perspective, performing diverse tasks with a design product, while simultaneously being knowledgeable about the design process and certain heuristics that comply with good design. Information provided from this activity is a definition of design features to be tested, through one or more methods deriving from the other three levels of analysis previously described.

3.2 Quadrangulation Strategy

At the PHEEL Lab, each experiment follows a typical process including a sequence of activities that we summarize in four main steps: (i) framing; (ii) preparation; (iii) test; (iv) data post-processing.

Our research is usually commissioned by an institution or company that manages a service, product or system, and that intends to analyse the user experience in order to seize the opportunities for improvement or, more simply, to understand the real attitudes of people towards their design. We also collaborate with companies that are developing innovative and experimental solutions, and therefore we support the development process by carrying out tests on prototypes.

The framing phase includes a preliminary inspection of the artefacts that will provide the context of analysis, and a collaboration with the stakeholder to identify goals and priorities of the investigations. Second phase, the preparation, includes the expert evaluation of the artefacts performed by designers and giving as output a conceptual description of the context including topology, structure, design characteristics; the expert evaluation also provides a description of the main tasks of users of the CPS in the timeline, and of the expected procedures of interactions with it. Based on these descriptions, we prepare an experimental protocol including tasks and activities we will focus on during the tests with users. Finally, we define a strategy for recruiting people to be involved in the experiments as representatives of the final users of a CPS. The third phase is dedicated to the tests with users, that we perform keeping under control potential external interferences on the normal development of activities. The last phase is dedicated to the data post-processing with an approach that is schematically reported in Fig. 1.

Fig. 1.
figure 1

Quadrangulation of layers of analysis for investigating user experience according to the activities within an experiment flow.

Full size image

3.3 Case Studies of Investigating User Experience

The methodology has been applied in experiments performed by the PHEEL Lab in a variety of application fields, from entertainment to retail, from gaming to driving experience. Most projects are funded by industries requiring investigation to support the design of very innovative solutions and the upgrading of the existing ones; due to the industrial sponsorship, most experiments are subjected to non-disclosure agreements, but we can provide here an overview of some experiments we performed in some fields we consider as meaningful for the purposes of the present work. We recall here: (i) experiments in large retail environments, (ii) analysis of the driving experience, (iii) investigation of playing.

In regard to retail as a domain of human experience, we consider physical stores as interesting for the experimental use of digital solutions employed to support information and decision processes. Retail spaces are complex environments where the senses are overstimulated, the attention of users is disputed by intentional activities and external inductions to action, and where task-positive and task-negative perception processes alternate frantically. Large stores provide several different concurring layers of information, related to the spatial organization and including the references to conventional or innovative semantic category-management of goods. In stores, users perform tasks under the compulsion of personal tasks and in free exploration of the space, sometimes also guided by information searched on personal devices. The application of the assessment methodology based on our four different analysis provides insights on cognitive and emotional experiences with respect to the spatial organization, the single objects and products, the social interactions and the evaluation of the journey. With this aim, we commonly track the user’s cortical activations, cardiac, respiratory, and ocular activity as well as user’s verbalisations. We report in Fig. 2 an archetypal setting.

Fig. 2.
figure 2

Illustrative research setting in the shopping context

Full size image

We are presently performing an investigation of the driving experience in real-world environments and in simulation rooms. The entire automotive sector is currently undergoing a phase of major changes due to the transition from combustion engines to electric ones, to the increasingly massive role played by sensors and digital technologies, and to the development of technologies for autonomous vehicle driving. The evolution of engines and control systems requires an extensive investigation of the new modalities of engagement of drivers, that involve all senses and the development of automatism. The application of our approach for the analysis of the experience of drivers supports the understanding of the impacts related to the sensorial stimuli provided by the new cars (including sound and haptic effects), related to the physical characteristics of the vehicles, as well as with respect to personal attitudes and expertise of pilots. In such a setting, we specifically monitor users’ cardiac and respiratory activity to infer states of arousal, as an expression of the physiological layer of analysis. Furthermore, we track users’ gaze activity in conjunction with driving interaction patterns, as indexes of behavioural expressions. Self-reporting takes place prior and after the experience to compare expectations with actual users’ rationalisations. Expert evaluation represents a fundamental layer in several process phases. First, these include the framing and preparation phases to outline user interaction possibilities in relation to context topology and expected users’ mental mappings. Second, expert evaluations steer the conception of the experimental protocol in the definition of tasks and procedures requested to the user. Lastly, during the data post-processing, expert evaluation has the role of bridging research outcomes with the future (re)design process. A typical research setting is shown in Fig. 3.

Fig. 3.
figure 3

Illustrative research setting in the driving context

Full size image

In collaboration with one of the Italian state’s concessionary agencies, we conducted research on gambling with digital games by applying our methodology of investigation in the laboratory and in game arcades. In the controlled experimental setting we explored player’s physiological and behavioural reactions to slot machines gambling outcomes and in-game features. The physiological layer of analysis encompassed most of the responses listed (i.e. cortical, cardiac, respiratory, dermal, and pupillary responses) due to the seated posture of the subjects. The behavioural layer embraced the gaze, proxemics, and interactive behaviours, whereas the self-reported layer included surveys prior to and after the experience. Expert evaluation played a central role in the understanding of the interaction dynamics with the gaming platform in the problem setting stage as well as in the generation of conclusive insights. The experiment allowed to produce metrics and models of the different ways of engagement in relation to the intrinsic characteristics of the games, highlighting the potential to leverage on such physiological responses to prevent possible unhealthy gambling behaviours. In particular, users’ behaviours underscored the possible exploration of in-game pop-up message usage to nudge online responsible gambling behaviours [52]. Secondly, we explored social dynamics in the ecological setting, namely arcades and slot halls. In such settings, we relied on unobtrusive behavioural expressions gauged from facial expressions, proxemics, paralinguistic cues, and kinesics gathered from ambient cameras. Our findings showed that the presence of other players might influence the individual gambling conduct, constituting an element of prevention in the onset of negative valence behavioural responses [53].

4 Conclusion

The development of physical/digital environments is based on the integration of traditional and innovative technologies aimed at improving lifestyles, wellbeing and providing sustainable cities. The introduction of digital technologies in these environments has impacts on human activities and induces new ways of actions/interactions that can only be partially predicted in the design phases. The development of suitable approaches for the analysis of the user experience plays a fundamental role in the development of CPSes, providing the means for an awareness of the impacts of the design choices. The development of neuroscience and the availability of instruments for monitoring physiological responses enable new approaches for such analysis, offering the means to understand the impacts of the design choices with respect to the variety of human characteristics and attitudes. By developing more comprehensive approaches to the investigation of user experiences in CPSes, the authors intend to provide a contribution to the development of design methods capable to support awareness and responsibility within the design processes.