Keywords

16.1 The Rise of Artificial Intelligence: Technologies for the Interaction Between Human and Machine

Digitally connected systems, techniques and methods of artificial intelligence (AI) and machine learning (ML) are changing our world. Humans develop such technologies in order to satisfy intentions, thereby anticipating our needs and thus making life easier. Computers trade our shares, we have cars that park themselves, and flying is almost completely automated. Virtually every area has benefited from the tremendous progress in digitization and AI, from military to the medical field and manufacturing. These progresses not only gradually transform the way we are interacting with technological products and services, but also differently influences our sensorimotor and cognitive capacities and skills. Osiurak et al. [1] summarizes these gradual technological developments over time into three levels that describe human-technology interaction with physical (affordance design), sophisticated (automation and interface design) and symbiotic (embodied and cognitive design) technologies. In the future, we will experience a massive influence on people’s everyday lives and working environment: the interaction with digital products and connected machines, technologies and services is becoming more self-evident and a core competence of the future requiring new modes of interaction and cognitive abilities. Future trends, like voice, gesture or thought operated technologies indicate an increasing symbiosis of human and technology, referring to a new phase of interaction called symbiotic intelligence [1]. The authors claim that the sophisticated technology of the future will ultimately become more and more unconscious to humans in order to maybe become one with them—the goal is the intuitive handling of technology in order to minimize the interaction effort with technical products.

Historically, the development and use of tools is strongly related to human evolution and intelligence. With the rise of AI and digitally connected products, we have access to an enormous variety of data and information. This enables us to develop interfaces that support us in how we think, what we know, how we decide and act. This transformation can be summarized under the concept of cognitive enhancement or cognitive augmentation [2]. It describes a very broad spectrum of techniques and approaches, such as performance-enhancing drugs, medical implants and prostheses and human-computer interfaces, which lead to improved abilities and may probably transcend our existing cognitive boundaries. Nevertheless, the increasing integration of technology in our everyday life and working environments entails new challenges and potential for conflicts. Often, humans with their individual preferences, skills and needs find themselves overlooked in the development of future technology. The resulting solutions, while technologically advanced, may nevertheless offer limited gains in terms of the productivity, creativity, and health of the users in question.

16.2 Embodied and Situated Minds: How We Use, Act and Think with Technology

If smart and adaptive technologies that support or even expand our cognitive abilities are the future, then it is essential to consider an optimal design so that such technologies are geared to the user’s needs and contribute to a human-centred, efficient and accepted technology.

The human-centred development of technologies and interfaces for the interaction between human and machines is becoming more and more important. In order to achieve increased productivity with a concurrent contribution to the subjective well-being of employees, digital equipment needs to be seamlessly and intuitively integrated in everyday working life [3]. Intelligent systems should support the user rather than hamper the interaction due to its inherent complexity. Instead of creating frustration, the system should motivate the user by providing a positive user experience during the interaction [4]. Positive user experiences in daily human-technology interactions are extremely important for both the individual person and the organization: from the human factors point of view, positive user experiences contribute to the subjective perception of competence, have a positive effect on mental health and consequently lead to motivated action, increased productivity and job satisfaction, which are important factors for the enterprise [5,6,7].

Research from the cognitive neurosciences on embodied intelligence shows that intelligence cannot be assigned purely to brain functions without regarding the human in its situated surroundings. Thus, intelligent behaviour and decision-making develop primarily from the interaction between brain, body and environment [8]. We use our entire environment, including integrated technologies, as an extended mind or memory storage [9], for example to facilitate knowledge retrieval and to reduce the cognitive demands of a task. Familiar technical aids include our fingers for counting, GPS devices for navigation, or the use of the internet for knowledge retrieval. We use such technical aids to either simplify or surrender tasks completely. Hence, we are permanently engaged in what can be called cognitive offloading [10].

16.3 Connecting Brain and Machine: Brain-Computer Interfaces

From a technological perspective, computers and machines are increasingly capable of learning, communicating and making decisions. Thus, the interaction between humans and technology gains additional dynamics. Speech or gesture and mimic recognition are increasingly replacing former input devices such as a mouse and keyboard. There is steady progress in the development of measurement techniques, sensor technologies and miniaturization of techniques for recording neurophysiological activity coupled with advanced signal processing, statistics and machine learning. Based on these developments, we gathered a tremendous understanding of cognitive functions and emotional processes underlying human behaviour, decision-making and social interactions over the last decades. Thus, brain and physiological signals allow us to derive many facets of subtle mental user states, like attention, affect, movement intention, cognitive workload and many more. One key invention for researching brain-based interactions between humans and machines is called Brain-Computer Interfaces (BCIs) [11, 12]. The BCI is currently the most direct form of an interface for the interaction between a user and a technical system.

16.4 Measurement Technologies and Applications of Brain-Computer Interfaces

The backbone of BCIs is technology for the real-time recording of neurophysiological activity that can be divided into invasive and non-invasive measurement techniques. Invasive recording techniques require brain surgery to implant electrodes directly into the brain. These recordings are further subdivided into brain-surface electrodes, like e.g. electrocorticography (ECoG) and brain-penetrating microelectrodes (for a comprehensive overview, see Thakor [13]). Non-invasive recordings for BCIs are subdivided into (a) portable measurement techniques, like electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) and (b) stationary systems like magnetoencephalography (MEG) and functional magnetic resonance imaging (fMRI). EEG and MEG allow measuring the electromagnetic activity of multiple cortical neurons directly by recording voltage fluctuations via electrodes on the scalp (EEG) or by using very sensitive superconducting sensors (so-called SQUIDs, superconducting quantum interference device in MEG) [14, 15]. Both fMRI and fNIRS measure neuronal activity indirectly. These techniques record metabolic processes related to neuronal activity by capturing hemodynamic changes in the blood flow. Hence, they enable the precise localization of the activation and deactivation of certain brain regions [16, 17].

Since its beginnings in the 1970s [18], BCI research focussed primarily on clinical and medical applications. The main purpose was to provide users that have physical or perceptual limitations with a communication tool or to allow them to control a technical device, for example for locked-in or stroke patients. In such applications, certain mental states that the user voluntarily generated are decoded while circumventing any muscular activity [19,20,21,22,23,24]. BCIs enable the development of assistive and restorative technologies to control wheelchairs [25], orthoses [26,27,28], prostheses [29], service robots [30] and web-applications [31]. Besides active control for users with motor impairments, BCIs are also applicable for neurofeedback training, for example to treat patients with psychiatric disorders such as depression, schizophrenia or attentional deficits [32,33,34,35,36]. Advances in these fields have led to a boost in mobile technologies and sophisticated machine-learning algorithms that can be further exploited for monitoring healthy users and laying the basis for non-medical applications of BCIs [11, 12, 37, 38].

16.5 Neuroadaptive Technology and Its Potential for Future Human-Computer Interaction Applications

In our everyday life, technical systems are becoming more and more prominent and serve the purpose of supporting us in our daily routines. Interactive machines and adaptive systems obtain information from the user’s interaction behaviour [39] through integrated sensors (e.g. smartphones) or environmental sensors (e.g. cameras) [40,41,42]. Such intelligent systems are summarized under the term context-aware systems [43, 44]. Context-aware systems are able to adapt the interaction based on the current context information (including information about the purpose of use, objective to be achieved, and tasks), thus making machines sensitive to physical environments, locations and situations. Examples range from very simple adaptations such as screen brightness to the current time of day and ambient lighting; lane-keeping and distance assistants in (autonomous) vehicles; and cooperating industrial robots and service robots for domestic use for the elderly.

However, the user with her individual preferences, skills and abilities receives less attention. In order to provide an optimal interaction between user and adaptive and autonomous technologies, it is important to take not only environmental and contextual conditions, but also the current user state (with her preferences and intentions) into account appropriately. Thus, machines need an understanding of the user, information about the user that goes beyond the bare necessities for controlling the machine. Therefore, it is a major prerequisite that technology reacts sensitively and promptly to its users, to create a collaborative and assistive interaction. Over the last years, the use of machine-learning algorithms for computational user modelling has increased substantially [45,46,47]. The basic idea is that computational user models represent more fine-graded aspects of the user, such as skills, preferences and cognitive abilities, as well as contextual changes such as selective attention, working memory load and the current emotional state and mood. It allows system adaptation to complex situations without inflexible dependence on predetermined programs [48, 49] and provides the basis for a symbiotic interaction between user and machine to collaborate and cooperate in making collective decisions.

For a long time, clinical and medical applications have been the primary goals of BCI research. In classical approaches, active control-based BCIs consider decoding brain activity to map it to commands that can drive an application that is running on a computer or a device. Examples for assistive or restorative BCIs are the P300 speller [50] that uses an event-related potential correlating with attentional resources, menu selection and exoskeleton control using Steady-State-Visual-Evoked-Potential (SSVEP) paradigms that are natural responses to visual stimuli at specific frequencies [51, 52], or binary selection through motor imagery paradigms that are produced by voluntarily modulating certain oscillatory sensorimotor rhythms [20, 26,27,28].

The introduction of passive BCIs [53] as a new concept coupled with new mobile and deployable sensor technologies for EEG and fNIRS [54,55,56] and advanced signal processing and machine-learning algorithms [37, 38, 57,58,59,60] for artefact correction and classification of cognitive and emotional states makes BCIs ready for non-clinical usage [11, 12, 61,62,63]. In classical BCIs, the machine-learning algorithms that are used focus mainly on the number of bits transmitted per minute and the successful classification rates of extracted brain patterns. In the passive BCI concept, the bit transmission rate is not the primary interest, but rather to focus on augmenting human-computer interaction. Hence, users do not need to carry out any mental actions actively to produce brain patterns that are translated to computer action. To the contrary, the passive BCI concept is envisioned as a continuous brain-monitoring process that is used to stratify the user according to his/her cognitive or emotional state. This provides the basis for extended computational user models in a human-machine control loop. Furthermore, this loop requires a precise knowledge of the psychological processes and corresponding neurophysiological correlates on which the adaptive computer system depends. Open-loop EEG or fNIRS-based passive BCIs to monitor psychological processes such as user engagement, user intention, selective attention, emotional engagement and workload in students, drivers, pilots or air traffic controllers have already been introduced [64,65,66,67,68,69,70,71,72,73]. Affective reactions such as valence and arousal are another source of user information that can serve as possible input to adaptive computer systems [61,62,63, 74, 75]. In a closed-loop human-computer interaction paradigm, this information enriches a user model to enable not only a concrete command, but also an adequate system adaptation to the user’s preferences, skills and abilities. With the help of sophisticated signal processing and machine-learning techniques, neurophysiological signals can be interpreted in the sense of a continuous representation of the user’s condition and provide information about psychological processes such as cognition, emotion and motivation. In a control loop, the estimated user model serves as an input variable in order to optimally support the goal of user interaction and certain user needs by intelligent system adaptation.

The developments in brain-based interactions enable the design of a neuroadaptive system loop [61,62,63, 76,77,78]. These loops are currently being discussed as a strategy to make adaptive and autonomous technologies more user-oriented and augment human-computer interaction. Possible future neuroadaptive applications, among others, are for example:

  • Intelligent vehicles that dynamically adapt the level of automation of the driving task to the current intention, attentional or workload level of the driver [67, 79,80,81,82].

  • Interactive e-learning programs that adapt speed and difficulty to the cognitive abilities of the user [73, 83].

  • Neurofeedback-based interfaces to promote subjective well-being by training concentration and relaxation [84].

  • Personalized internet applications that capture affective user reaction to adapt their content, presentation and interaction mechanisms to individual needs and preferences [61,62,63].

  • Collaborative robots that react sensitively to user intentions, emotions and attentional levels [85,86,87,88].

There are two main benefits of implicit user interaction via a neuroadaptive control loop for personalized system applications. (1) Complementarity: There is no interference with other activities in the interaction cycle. The neuroadaptive control loop expands the communication between human and machine and thereby contributes to the self-learning process of the system towards individual user abilities, skills and preferences. (2) Assistance for automation: The neuroadaptive system can help to increase the situation-awareness of the user towards automated system behaviour. A transparent system behaviour develops by considering implicit user reactions during longer periods of interaction. Consequently, the user is expected to experience feelings of control over and trust in the automated system. Future research will reveal the extent to which these expectations are correct.

The increasing research interest in the still very young field of neuroadaptive technologies can also be observed at human factor engineeringFootnote 1 and affective computingFootnote 2 conferences and newly emerging, popular conferences such as neuroergonomicsFootnote 3 and neuroadaptive technology.Footnote 4 Innovation-friendly companies, e.g. from the automobile industry, already use brain and physiological methods for their consumer research. Furthermore, technology companies from Silicon Valley, like Facebook Footnote 5 or Elon Musk—founded NeuraLink Footnote 6—invest in research on invasive closed-loop neuroadaptive technologies.

Although the use and advantages of neuroadaptive interaction are indisputable, there are still some major gaps and challenges to overcome before they can be applied outside of lab conditions. (1) Understanding brain functions out of the lab: There is still a significant lack of basic knowledge on human brain functions in complex real-world situations where individuals perform activities in natural environments and social contexts. In such environments, signal analysis and machine-learning interpretation is still very challenging. Robust algorithms to deal with real-world artefacts that strongly exceed the signals of interest are still needed. (2) Integration of context information: Neurophysiological signals cannot be interpreted in isolation, but must be analysed and classified in a given context of application. While context is controlled and known in the lab, real-life applications require the combination of context information to adapt technical systems to user states and social situations under real-world conditions.

Consequently, considerable research effort is needed to realize closed-loop solutions based on brain signals robust enough to deal with the high complexity of real-life applications [11, 12, 89].

The primary goal of a future neuroadaptive technology research agenda is to consistently align the increasing intelligence and autonomy of technical assistive systems with the emotional needs and cognitive abilities of the human—a human-centred neuroadaptive technology research roadmap. By their individual adaptability, neuroadaptive technologies contribute significantly to a human-centred, efficient and acceptable technology. The applied research in this field and the possible transfer into real-life applications requires a strong transdisciplinary research agenda. Due to its far-reaching social implications, research and developments does not only have to face technological, but also social challenges, like including questions about cognitive liberty, mental privacy, mental integrity and psychological integrity. In addition to computer science and neuroscience, the integration of further disciplines is needed, such as positive psychology that aims to foster human flourishing and well-being by researching positive emotions and its influences during human-technology interaction as well as ethics and social sciences. If neuroadaptive technologies are applied in non-medical areas, they must be consistently oriented to the needs and ethical values of the users and society.