Keywords

1 Introduction

The reports on new and emerging risks (ESENER-3) by European Agency for Safety and Health at Work (EU-OSHA) (2019) as part of the third European survey of enterprises and recent Eurostat data (2023) show that occupational health and safety risks are being analyzed with a greater focus on psychosocial risks related to work and new technologies. On the one hand, critical points are highlighted in relation to psychosocial and ergonomic factors mainly related to human-machine interaction, and on the other hand, the role that mechanisation and digitalisation have to play in mitigating these risks is explored (Niciejewska and Idzikowski 2022). According to the analysis of statistical data from the main European databases, such as EU-OSHA (2023), Eurostat (2023) and WHO (World Health Organization) (2023), the construction, transport and storage, manufacturing, agriculture, silviculture and fishing sectors together account for about two thirds (63.1%) of all fatal occupational accidents and more than two fifths (44.1%) of all non-fatal occupational accidents in 2020.

In this year, of all fatal accidents at work in the EU, the manufacturing sector (15.2%) had the highest share. The highest share of non-fatal occupational accidents in the EU occurred on industrial sites for the manufacturing sector, totaling 78.1%, with loss of control of machines, tools or transport and handling equipment (20.8% of the total), movement of the body under or with physical stress (18.4%) and slipping or falling (17.6%) being the most common causes. Loss of control of machines, tools or transport and handling equipment was also the most common cause of fatal accidents at work, accounting for 23.2% of the total number of work-related fatalities in the EU in 2020. For the same year, the most common contact modes for non-fatal accidents in the EU included: physical or mental stress (23% of all non-fatal accidents); impact with a stationary object (in other words, the victim was in motion - 21.0%); contact with a sharp/sharp or rough/rough agent (14.7%); and being struck by a moving object/collision (11.5%) (Eurostat 2023).

A detailed review of the relevant context through a critical analysis of the state of the art and national and international standards has shown that operators are often unable to develop or maintain adequate levels of awareness, due to risk factors (see Fig. 1) that assess the ergonomic, psychological and organizational factors and interaction between the “operator” and “the environment”. In particular, it’s found from statistical data and literature research, that the greatest risks are related to ergonomic (Dekker et al. 2021), transversal and physical aspects such as the high complexity of systems, lack of experience in the use of systems and inadequate training and workstation design with respect to the physical-dimensional compatibility of the operator (ESENER 2019; Nawi et al. 2022; Razali et al. 2022). Against this, current “augmented” safety devices have been researched and systematized to support the operator in machine use and maintenance activities and to assess the operator’s psychophysical state.

Fig. 1.
figure 1

Histogram re-processed by comparing and synchronising statistical data on occupational safety in industrial establishments (ESENER 2019)

Full size image

2 Human Error, Mental Workload and Human Reliability

The concept of the working environment encompasses, in a wide sense, the man-machine-environment paradigm in constant interaction between the physical and psychological spheres. It is therefore important to consider all the factors that contribute to the occurrence of an error or accident. A correct design of work activities and workstations must consider the cognitive load and anthropometry of the operator. With the aim to address these issues, it is important to estimate workers’ awareness of their own safety (Körner et al. 2019) and relate it to their state of mental stress and fatigue in order to improve interaction with machines. It is essential to analyse the actions between the operator and the machines, emphasising the distinction between correct/incorrect behaviour and unintentional error (such as malfunctions, faults, interface incompatibility, etc.), assessing environmental factors (microclimate, noise, lighting, etc.) and the working conditions related to the design of the workstation (La Fata et al. 2023).

One of the major causes of accidents is the physical and mental fatigue of the operator.

Fatigue is defined as a decrease in mental and/or physical performance caused by cognitive overload and physical exertion. Worker fatigue has been introduced as one of the main factors that increase the error rate of workers and lead to unsafe work actions, negatively affecting their alertness, reaction time and mental acuity. Therefore, the quantification of fatigue is fundamental in relation to occupational health and safety. In addition to being a physiological response of the human body that can prevent overload, fatigue is a symptom associated with several diseases and health conditions (Kirwan 2017). Fatigue impairs cognitive and/or motor performance, reducing work efficiency, productivity and product quality, and increasing the risk of injury and death (Körner et al. 2019). Since excessive or insufficient mental workload may be associated with reduced efficiency and safety of human-machine interactions, cognitive and physical stress must be analysed and evaluated in order to design new integrated and adaptive systems that can assist the operator (Derosière et al. 2013).

It is necessary to assess the impact of human factors on risk through Human Reliability Analysis (HRA), the functions of ‘identifying which errors can occur (human error identification), what is the probability of these errors (error quantification) and, finally, identifying ways in which the probability and consequences of errors can be reduced (error reduction) (Kirwan 2017). Ayaz (2012) states that mental workload reflects “how hard the brain is working to meet the demands of the task”. Therefore, the last three decades have witnessed a revolution in understanding the brain processes that regulate human performance and attention of workers (Dehais et al. 2020). Moreover, HRA applications are still scarce in the manufacturing sector, where human errors are often overlooked and, therefore, there is a need for the creation of safety tools, based on neuroimaging technologies, that are sophisticated and portable and allow for a non-invasive examination of the ‘brain at work’ in real time, monitoring the operator and providing him with all the information he needs to complete his tasks while minimising risk factors. Cognitive evaluation emphasises the role of operators as they are involved in the acquisition and processing of information through a holistic assessment of how design influences the acquisition of information and, conversely, how the operator’s mental models (process understanding) influence the acquisition and processing of information. This point of view is significant in the context of contemporary digital implants, where humans generally act as decision-makers and perform monitoring, diagnosing, and prognostication functions. Based on this, it is possible to develop countermeasures, such as integrated intelligent systems, that “protect” the worker from problems related to the psychophysical sphere and ensure the safety and quality of work in manufacturing environments.

3 Design, Usability and Human Factors for HMI Evaluation

The interconnection between operator and industrial machine requires adequate training of the worker in the use of the machine and knowledge of the characteristics of human control and human modelling for machine design (Lu et al. 2022).

It is necessary to approach the study of Human Factors from a neuroscientific perspective, looking at the new paradigms of Industry 5.0 and focusing on the operator and his capabilities (Lombardi et al. 2023). Human-machine interaction, therefore, is considered in the totality of its aspects, including those related to human reliability, fatigue and physical/cognitive stress (Perrey et al. 2010) and systems usability.

There is a shift from a task-centric view of design to a human-centric view, in which the entire system is designed to ensure well-being and usability (Buono et al. 2021); tools such as sensors, body scanners, and devices for detecting biometric and cognitive parameters allow for the immediate detection of specific user characteristics. Several researchers have attempted to address consumer usage behaviour regarding smart wearable devices by extracting potential technical and psychological factors using user-oriented theories and models (Park 2020). This approach allows modelling the user experience through an ergonomic and cognitive study, the analysis of interaction systems and the evaluation of usability and accessibility requirements (Buono et al. 2021), thus helping to optimise the relationship between the user, device and its environment.

A total of 35 HRA methods are identified in literature, divided into first- and second-generation methods (Bell and Holroyd 2009), which allow risk assessors to predict and quantify the probability of human error (Kim 2001). First-generation methods focus on human actions (errors) based on skills and rules, but do not consider context, human organisational factors and cognitive aspects. These methods include the technique for human error-rate prediction (THERP), which involves performing a task analysis to identify human involvement, define the sequence of events that must be performed to ensure safety, and then quantify these sequences using a human error probability database (Derosière et al. 2013). THERP also includes the calculation of dependency, which means that the success or failure of the current action is related to the previous action or task. Second-generation HRA methods, on the other hand, introduce cognitive models to characterise human behaviour in the workplace, searching for the root causes of human errors in the application of mental processes based on perception, thinking, memory and action decision strategy (La Fata et al. 2023). These methods include personal, contextual and cognitive factors (e.g. Performance Shaping Factors - PSFs) that can influence workers’ performance. In particular, the most significant PSFs on which primary actions can be taken to improve worker reliability during task performance can be identified, which are: “Training”, “Procedures”, “Interface”, “Time pressure”, “Complexity”, “Workload, stress and Stressors”, “Environment”, “Physiological parameters”, “Work process”. The fundamental aim of these approaches/analyses is to make the system safer by decreasing various hazards and reducing the possibility of human error. For example, guidelines for the design of HMI (Human Machine Interface) in a control room should be geared towards ensuring the ease of interaction of a user-operator with the computer interface by analysing various aspects such as the readability of text on an interface (e.g. font colour, size), level of detail (too much or too little), and alarm status that differs from normal system operating conditions. The main source of human error is a mismatch between human capabilities and system requirements. It is essential that system design considers user characteristics and needs in relation to anthropometric variability, usage activities, and different levels of skill, experience and knowledge under different conditions of use and for each user category. It is necessary to analyse human-machine interactions by systemising human and neuroergonomics factors such as physical-dimensional, performance (physical workload or body posture), behavioural and ability (e.g. personality, self-efficacy, personal control and motivation); cognitive (e.g. cognitive fatigue, cognitive load, decline in attention); physiological (e.g. heart rate, muscle oxygenation, body temperature, drowsiness, respiratory rate, sweating) and environmental (e.g. ultrasound, noise, microclimate, vibration, lighting) (see Fig. 2). Through the neuroergonomic approach and analysis, cognitive constructs of interest (e.g. motivation or mental load) can be assessed and analysed in highly controlled artificial environments in the laboratory. Through user-centred design, one aims to improve safety and well-being in workplaces or everyday environments with the help of neurophysiological measures that enable an understanding of the mental mechanisms of workers subjected to specific work demands (Parasuraman and Wilson 2008). Therefore, cognitive neuroergonomics investigates cognitive states and their impact on information processing in the workplace based on neurophysiological data. The continuous assessment and monitoring of mental states and/or cognitive processing can contribute to improving safety and well-being at work (Parasuraman and Wilson 2008). Measures of mental workload can be classified according to performance, in relation to the subjective self-assessment process or in response to psychophysiology or neurophysiology (Dehais et al. 2020).

Fig. 2.
figure 2

Characterisation and synchronisation of HMI and neuroergonomic factors.

Full size image

4 Neuroimaging Technologies Comparison for Cognitive and Perceptual Processes’ Detection

Research into the behaviour of workers, the monitoring of neuroergonomics parameters during collaborative work, and the monitoring of attention and fatigue contributed to a better understanding of the events occurring and indicate the specifics of workers’ behaviour (Savković et al. 2022). The use of neuroimaging technologies for the analysis and assessment of cognitive stress makes early and objective detection possible in the event of declining levels of attention (Derosière et al. 2013) and concentration. EEG systems, for instance, provide the possibility of a continuous and objective measurement of workers’ attention, assisting the monitoring of their performance.

Advances in sensor technology have made it possible to objectively measure various aspects of human cognition (Shahab et al. 2021); measures of mental workload can be classified as performance-based, or related to the subjective self-assessment process, or associated with psychophysiology or neurophysiology. Physiological measures such as eye-tracking, electroencephalography (EEG), heart rate variability (HRV) and galvanic skin response (GSR) have proven to be useful markers in providing critical information on human cognition.

Among the technologies for detecting neuroergonomic processes is Neuroimaging, or brain imaging, which uses various techniques to directly or indirectly map the structure, function or pharmacology of the nervous system. Functional Neuroimaging is used to diagnose metabolic diseases and lesions on a fine scale and is widely used in psychological, neurological and cognitive research and in the construction of brain-computer interfaces. The main approaches for assessing cognitive load include direct measurements and indirect physiological ones (Lu et al. 2022). It is important to focus on indirect measurements, which is those that estimate the mental stress or safety awareness of workers based on their performance or physiological data obtained through sensors or specialised devices. Performance is generally assessed by response time or error in completing a task, but the worker’s psycho-physical state must also be taken into account through the detection and analysis of physiological parameters such as brain response feedback (UNI EN ISO 10075-3:2005), which could be heart rate (electrocardiogram - ECG), skin conductance (electrodermal activity - EDA), muscle oxygenation (electromyography - EMG). Considering that the prefrontal cortex (PFC) is functionally connected with several regions of the brain, this region mediates the complex interactions between motor function and emotion (Doi et al. 2013; Dehais et al. 2020) and performs a control function during routine cognitive operations, such as action selection, retrieval/updating in working memory and monitoring (Kirwan 2017). Michael Posner (1980) pioneered a network approach to resource operation in the early days of neuroimaging. His influential analysis describes how specific networks were dedicated to particular functions of attention regulation, e.g. alertness, orientation, focus. There is evidence that a higher probability of failure is associated with PFC deactivation, for operational performance in which failure can compromise the safety of oneself and others, a higher probability of failure can also provoke strong emotional responses that are associated with stress and cognitive interference, which can function as distractors from the task at hand (Dehais et al. 2020).

Neuroimaging methods fall into two categories: those that reflect metabolic brain processes associated with neural activity, such as functional magnetic resonance imaging (fMRI) and transcranial Doppler sonography (TCD), and those that directly measure neural activity, such as electroencephalography (EEG) and event-related potentials (ERPs). The merits and disadvantages of these techniques can be considered in terms of three criteria: (a) spatial resolution in localising neural activity within the brain, (b) temporal resolution in identifying the timing of neural processing, and (c) ease of use in HF/E. Neuroimaging findings have supported the distinction between perceptual/cognitive, verbal/spatial and focal/environmental visual processing (Parasuraman and Wilson 2008).

Among the most popular neuroimaging technologies, the ones mainly used for the assessment of cognitive load during work tasks by means of PFC analysis are: electroencephalography (EEG), which is a technique that detects electrical activities generated by the brain. The EEG signal is an effective signal for representing changes in the autonomic nervous system. The level of mental stress is frequently reflected by an increase or reduction in brain activity in the frequency band. The study conducted by Al-Shargie et al. (2016), used arithmetic tasks as stimuli to induce different levels of mental stress, which could then be classified according to EEG signals; functional near-infrared spectroscopy (fNIRS), a non-invasive functional neuroimaging technology widely used to detect physiological factors related to brain activity. It has a higher spatial resolution than EEG and better temporal resolution than functional magnetic resonance imaging (fMRI) (Russo et al. 2023). NIRS measures light intensity after passing through a tissue (Song et al. 2020; Perrey et al. 2010; Varandas et al. 2022). It is believed that NIRS measurement imposes considerably less physical and psychological burden than current neuroimaging techniques (Doi et al. 2013). Aghajani et al. (2017) demonstrate that a hybrid system (EEG + fNIRS) allows higher classification accuracy for mental workload than using EEG or fNIRS alone. A recent study achieved over 90% accuracy in distinguishing between stress and non-stress conditions during a mental arithmetic task using combined EEG and fNIRS (Al-Shargie et al. 2016). Since mental workload and psychological stress share many physiological markers, it can be difficult to distinguish between them. For example, both mental workload and acute stress are known to affect heart rate and heart rate variability (Parent et al. 2019), justifying the need for an integrated system of multiple sensors in order to ensure physiological feedback. Both technologies have limitations (e.g. external disturbance factors, motion artefacts, etc.), which is why the combination of EEG-fNIRS revealed better results, indicating that additional data sources may be useful for the detection of cognitive fatigue. Furthermore, the combination of different sensors such as NIRS and EMG are particularly useful for emotional studies, providing a method for examining the theoretical process of fatigue. In this case, a decrease in haemoglobin can lead to a lack of muscle oxygenation, which can lead to fatigue. The combination of measurements is not contaminated by technical interference as both methods are based on different working principles (Balconi and Molteni 2016).

5 Devices and Integrated Systems for “Augmented Safety” in Workplaces

In manufacturing industries, which require workers to be mentally alert and to carry out repetitive tasks using specific muscle groups, cognitive and physical load levels are assessed using technologies such as electroencephalography (EEG) for cognitive tasks such as calculation or continuous performance that cause activation of the frontal region of the brain (Eyam et al. 2021), the electrocardiogram (ECG) (Parent et al. 2019), eye tracking also used to monitor driver fatigue (Balconi and Molteni 2016) and muscle electromyography (EMG) (Savković et al. 2022) for localised muscle fatigue to monitor, for example, arm-shoulder muscle load in car assembly workers (Ferguson et al. 2013). To estimate mental stress in the area of industrial environments, it is therefore necessary to study the overall function of brain activity in the workplace. Supervisors must ensure that workers are immediately aware of potential dangers to avoid accidents. Using wearable devices, workers can inform their supervisors about their location, fatigue levels, health status and surroundings (Alberto et al. 2018). This digital connectivity and data transparency allows supervisors to remotely observe workers, check safety compliance, assess potential hazards and send early warnings or requests for help. The following are examples of smart wearable devices (see Fig. 3), such as helmets, headbands, gloves, glasses, textiles, etc. that use advanced sensing technologies, such as eye-tracking, haptic feedback or neuroimaging technologies, have shown great promise in various application fields, including manufacturing. These devices are designed to provide real-time feedback and improve the user experience, making them ideal for applications where precision and accuracy are crucial.

Figure 1 shows an overview containing a collection of devices currently on the market, software and designs that are used today for monitoring, tracking and providing ‘augmented’ information to assist the operator in the use and maintenance of machines.

Among the devices examined, we can distinguish several categories: (1) headbands; (2) helmets; (3) gloves; (4) goggles; (5) textiles; (6) exoskeletons; (7) smartwatches; (8) and other devices and/or software. Many of the devices examined are integrated systems combining different technologies. The state of the art reveals the increasing use in industry of tools and systems to improve the safety and health of operators, including: (a) technologies for detecting physiological parameters (heart rate) and biofeedback; (b) cognitive parameter detection technologies (EEG and fNIRS); (c) eye tracking; (d) remote control of machines; (e) real-time sensing and data transfer; (f) virtual reality; (g) gyroscope and accelerometer systems; (h) tactile feedback and force translators; (i) biomechanical overload assistance.

There are wearable tools and connected work platforms on the market, designed towards monitoring safety at work (safe lifting of heavy loads, lifting assistance, ergonomics, hazard identification, sleep monitoring, fatigue and stress management due to extreme temperatures); increasing worker productivity (asset monitoring, augmented and virtual reality, gesture and movement control, cognitive parameter detection, work stress management); monitoring health (work-related musculoskeletal disorders, movement disorders, respiratory disorders, cardiovascular health, etc.) (Patel et al. 2022). In particular, the devices examined are designed to monitor the operator, the machine and the environment and to provide ‘augmented’ information. Protecting and improving the safety, health and productivity of workers is crucial for companies. In this regard, intelligent systems (deployable sensors and analytics) play a key role in facilitating continuous monitoring, management and forecasting of workplace risks and organisational resources. In this paper, we examined recent trends in commercial wearable technologies and connected worker solutions applied to different work environments to promote ergonomics, situational awareness, injury risk management, efficient workflow and healthy behavioural and cognitive habits. While most devices monitor human performance (e.g., biomechanical functions, physical activity and/or physiological signals), new intelligent systems are being introduced to actively monitor and manage mental health (e.g., stress, emotions, moods) using brainwave sensing, biofeedback and human-in-the-loop models.

Most of the devices examined concern technologies related to the detection of cognitive parameters as, due to the growing interest in and request for these, and thanks to the rapid advances in the micro- and nanoelectronics industry, biomedical device manufacturers have been able to drastically reduce the size of EEG and fNIRS devices (Di Flumeri et al. 2019), allowing them to be used on a daily basis within the reach of all types of users. Despite this, such existing systems are often not aesthetically appealing or able to arouse positive emotions and thus influence and improve the user experience (Radüntz and Meffert 2019). Some solutions turn out to be cumbersome or painful for the operator when used over a long period of time (e.g. used during an eight-hour shift), with the risk of altering user behaviour. There are also solutions that wet and soil the hair and scalp (Di Flumeri et al. 2019) as they are made of the abrasive paste and electrolyte gel, which, although minimally invasive and not harmful, are sticky products. Another important factor is the electrode-skin impedance, which must be controlled and adjusted to obtain acceptable low values; and also the awareness of being observed and detected, and thus effectively prevented from performing the activities (Hanzal et al. 2023).

For example, if we consider devices that use neuroimaging technologies such as EEG or NIRS (e.g. devices such as Emotiv Insight, Mendi or even Muse) they are discrete and (almost) invisible to others so as not to hinder the normal behaviour of the worker. This is achieved with band-based systems or the use of electrode pads, but these only cover a small area of the skull and therefore do not allow a clear separation of cortical sources or of the origins of them and are extremely sensitive to artefacts created by movements. The NirSport device, which uses Near Infrared Functional Spectroscopy technology, is lightweight and easily transportable, so that it can be used for measurements in real locations and not just in laboratory setups. However, the parameters measured by measuring the oxygen concentration in the blood vessels of the cerebral cortex being examined have an increasing and decreasing phase that results in the detection of inaccurate cerebral activation, as tissue oxygenation artefacts. Therefore, as well as devices recording the physiological parameters of the operator, there is a need to integrate different parameters, making it necessary to use adjacent systems for a complete overview of the state of health and to improve safety at work (e.g. the Honeywell BioHarness device).Or some devices are cumbersome, to the point of preventing the operator from moving freely and easil around the workstation or switching between workstations, tasks and activities, without removing the device. For example, the Haptx Gloves G1 remote equipment control gloves, which provide realistic tactile feedback. They feature a lightweight, wireless Airpack, worn like a backpack, that generates compressed air and precisely controls its flow; with this device, it is impossible to work on other tasks without removing the glove, and the device worn on the shoulders is bulky, heavy and needs to be powered if worn for an entire work shift (8 h), as the device’s autonomy is approximately 3 h.

This can increase the possibility of cognitive overload and mental strain and often negatively affect the level of interaction and collaboration with other operators. In addition, there is a need to overcome the difficulties currently encountered in managing communication and systematising heterogeneous data collected from different devices (often with incompatible SW) in relation to different parameters (physiological, cognitive, environmental).

Fig. 3.
figure 3

Overview of intelligent ‘augmented’ security devices.

Full size image

6 Conclusion

Integrated, adaptive and intelligent systems can help reduce the damaging effects of human error during the execution of complicated tasks, relying on a sensitive and accurate collection of metrics to identify the various levels and combinations of mental effort and psychological stress in real time (Parent et al. 2019).

For the human factors discipline, the study of mental workload serves two main functions: (a) to quantify the transitions between operators and a set of task demands or technological systems or operational protocols, and (b) to predict the probability of performance failure during operational scenarios, which may be safety-critical (Dehais et al. 2020). A challenge the field must deal with is to delineate a consistent relationship between the measurement of mental workload and performance quality, based on complex interactions between the person and the task. Both mental workload and psychological stress are common in work environments and the two concepts can have a mutual impact on each other. In this sense, maintaining a certain level of stress may also be desirable during particular situations, such as during training, where stress may help to consolidate information in the memory. It can be argued, then, that different situations lead to different optimal levels of workload and stress may result from different situations and identifying these ‘weak points’ could maximize desirable effects such as performance and learning.

Neuroergonomic approaches based on measurements of the human brain’s hemodynamic or electromagnetic activity, combined with feedback from other technologies, can provide a sensitive and reliable assessment of human mental workload in complex work environments, in order to accurately assess mental workload, which could help mitigate errors and enable early intervention by predicting the decline in performance that may result from overwork or under-stimulation. In particular, the introduction and use of digital technologies such as wearable devices and artificial intelligence represent an opportunity to support production processes and safety in the workplace, as long as the main criticalities concerning, for example, the wearability of the devices (such as the inability to complete certain functions or the incorrect positioning of the system with respect to the underlying body area or anatomical point, producing discomfort) are remedied efficiency (the inability to act correctly in synergy with the user’s movements and the wearable’s expectations); and discomfort (such as friction, restriction to movement, excessive temperature, etc.)., or possible pressure problems resulting from prolonged use of wearables).Based on the analysis conducted in this paper, a comprehensive approach is needed to address the complex issues of occupational safety and health, particularly in the manufacturing sector. The combination of different technologies, including neuroimaging, can provide valuable insights into the cognitive and physiological aspects of work and help identify potential sources of work overload and related psychosocial risks. It is therefore essential to develop an integrated system capable of analysing and evaluating multiple data sources to provide a complete picture of the operator, the machine and the working environment and ensure the well-being of workers.