Keywords

1 Introduction

Expert medical systems have been around for decades. One of the first examples is MYCIN, developed by Shortlife in 1976 at Stanford University, representing the first research effort able to solve complex real-world problems and provide clinical assistance [1]. The recent advances in Artificial Intelligence have brought about a new generation of expert systems, empowered with cognitive capabilities such as machine learning, reasoning and decision-making. They include off-the-shelf applications such as IBM’s Watson, HP’s Autonomy and Palantir but also their precursors, a set of in-house-built intelligent decision support systems using scorecards and dashboards to improve clinical outcomes.

Principles of Soft Systems Thinking, ETHICS and AIM have been used for the purpose of the analysis of human–computer interaction in this chapter which include analysing the health-care industry from a systemic point of view and focusing on the people system rather than the IS technology they use to do their jobs ([2], pp. 18–19; [3], pp. 3–5). To avoid confusion between the notion of system in IT (sum of technology and applications) and the one in Systems Thinking (people and technology), intelligent systems such as Watson will be referred to as intelligent agents. A clinical decision support system (CDSS) incorporates established clinical knowledge which is constantly updated with patient information in order to improve the patient care standards and includes a knowledge base software to integrate patient information with the knowledge base and a user interface for the clinician to interact with the system. The intelligent agent is a large-scale CDSS which is intended to deal with expert knowledge only and is able to process both structured and unstructured data ([4], p. 5; [5], p. 504).

Research

The initial aim of the chapter was to outline the considerations that various decision-makers have been taken into account and others that have been ignored before allocating resources for the implementation of intelligent medical agents. Given the limited academic works specifically targeting intelligent agents and the lack of feedback available from medical staff who have used them in real-world practice to comment on their effectiveness, the scope of the chapter was amended to a balanced account of the consequences of adopting intelligent agents in health-care organisations affecting the medical staff’s day-to-day job. The pre-adoption considerations advised in this chapter are based on the previous deployments of CDSS as outlined in academic publications, the initial feedback of users who participated in testing the intelligent agents at work and case studies with advertising character sponsored by intelligent agents’ software vendors. Caution needs to be employed in using results from these case studies by constantly comparing the outcomes they present with academic conclusions based on the adoption of CDSS.

The motivation of the authors was the publicity and investment that intelligent agents like Watson have benefitted from since the beginning of their trial deployments in health-care organisations. The feedback that has surfaced so far seems more concerned with the potential technical capabilities of these agents rather than the impression they have made on clinical personnel [6]. This chapter does not present itself as a comprehensive review of all the consequences associated with the adoption of intelligent agents. This chapter researches the stakeholders with decisional powers in the adoption process and their involvement in a technology-assisted medical process. It also aims to outline some examples that should be considered before the final deployment of intelligent agents in order to ensure a smooth integration of the technology in the daily jobs of both doctors and administrative staff in health care.

Thus, this chapter is structured as follows. The first section sets up the context used to briefly review intelligent medical agents from a socio-technical point of view. The second section provides a brief account of how expert systems have evolved into intelligent agents. The third section introduces a systemic view of the health-care industry and starts analysing the people system around it by considering the administrative decision-makers. The stakeholders’ views are discussed in Sect. 4. In Sect. 5, the ethical aspects are reflected upon. Finally, conclusive remarks are presented in Sect. 6.

2 Considering Intelligent Systems

An expert system is defined as “a piece of software which uses databases of expert knowledge to offer advice or make decisions in areas such as medical diagnosis” (Oxford Dictionary). There are two key aspects connected to the technology behind expert systems: they are built to simulate the judgement and cognitive processes of the human brain and they are processing expert knowledge and experience from a particular field ([7], pp. 4–12). Both these aspects point to the human element’s decisive role in creating the expert system: the data that is fed into the system, regardless of whether it is development data or the knowledge that medical staff has gathered over the years.

The technology behind expert medical systems was first pioneered in the early 1960s using programs that performed statistical analysis. The Dendral (1961) project and the software it produced represents the first use of Artificial Intelligence in biomedical research and was developed by Joshua Lederberg (geneticist) and chemistry professor Carl Djerassi. The 1970s brought a wave of IT systems performing diagnoses and making therapy recommendations which included PIP, CASNET, INTERNIST, CADUCEUS and PUFF [1].

Liebowitz ([8], pp. 32-1, 32-2) predicted that the stand-alone systems named above would evolve into fully integrated information systems that would also connect to hospital database systems and medical devices (e.g. EMRs, ECG and EEG, CT and MRI). The predicted evolution started to materialise after 2010 through advanced systems employing technologies such as machine learning, natural language processing and speech recognition brought about by the advances in Artificial Intelligence. Examples of systems already on the market and adopted by medical bodies include IBM Watson, HP Autonomy and Palantir. The recent support that expert medical systems have received both from technology companies and medical bodies is partly justified by the increased use of technology in health care and diverse stakeholder demands.

Health care has recently been under scrutiny after a series of failures to achieve its targets ([9], pp. 1–4), both at a scalable level (deadlines, budget, patient waiting time) and at a less quantifiable one—“quality of care”.

In their attempt to become value-based organisations, medical bodies are struggling to maximise their services’ value, achieving best outcomes at lowest costs, and at the same time working towards patient-centred systems organised to meet their patients’ needs [10]. Patients start being seen as customers and technology as a catalyst to improve their satisfaction with the service they receive. In an environment with a decreasing number of experts and increased demands and pressure, expert medical systems are often seen as strong arguments in favour of a technology-assisted health care.

The promoted advances in cognitive computing and artificial intelligence together with the pressure put on human capabilities have paved the way for what has been advertised by software providers as more cost and time effective and accurate technical solutions. The predicted benefits of intelligent agents such as Watson (e.g. immense memory space, processing of unstructured data) recommended them as viable candidates [4]. However, their technical development has tended to bypass or avoid an analysis concerned with what their role in medical care should be and whether medical staff actually needs the assistance of intelligent agents. The possible roles discussed in this chapter are replacement of the human doctor, guarantor of diagnostic accuracy or human dependent repository of data [11].

3 Impacted Parties

Health-care delivery is a highly complex environment and consists of numerous loosely connected and independent systems and subsystems which make it difficult to assess its overall business value on one side and quantifiable clinical outcomes on the other ([12], pp. 1–7).

Health care and Hospitals as Socio-Technical Systems

The health-care industry and hospitals can both be seen and analysed as socio-technical systems, a heterogeneous ensemble of people, technology and legacy practices that are expected to work together for the benefit of the patient without neglecting the notion of work satisfaction for the medical personnel. Employing systemic thinking techniques can facilitate a differentiation between the system as a whole and the sum of its parts ([2], pp. 18–19).

When looking at the health-care system as a whole, the ensemble is made of inputs (patient queries), internal processes (medical interventions) and outputs (patient treatments and work satisfaction). An intelligent agent can be designed to store data inputs in one place, process them and ensure a smooth data flow through the internal processes regardless of whether they are tests performed by care staff or administrative reports for health-care management.

A comprehensive intelligent agent might face a number of challenges before ensuring a smooth transition of data across various departments and stakeholders. First of all, different stakeholders have completely different priorities. Administrative staff and management team’s jobs focus on budgets, feasibility strategy and resource allocation. The IT department’s main responsibility is a viable IT strategy, security risks, upgrades and maintenance. Doctors and nurses need to think about time management, patients’ care and effectiveness of treatments and also job satisfaction and work–family balance. They might have developed their own work routine and repository of data without necessarily following the policies put in place by administrative staff and the IT department. At the same time, they might need other stakeholders’ permission to perform certain tests or prescribe certain treatments and definitely depend on their resource allocation. The interaction between medical personnel and their professional areas brings with it more complexity. While more tests and procedures contribute towards increased diagnostic accuracy for doctors, they increase an already strained budget and put more pressure on the administrative staff ([13], pp. 85–86; [14]). In an industry where technical skills are not distributed evenly across generations, there will always be a two-way training. Introducing an intelligent agent might put additional strain on the younger, technology-savvy generation who might feel responsible for teaching the older one how to use it [15]. However, the older generation includes the real-world experts, clinicians with many years of real-world practice who will need to teach the younger generation to develop their intuition and know when to question the intelligent agent’s judgement. Another issue of this industry is the technology gap between the IT departments and medical personnel. Dr. Atul Gawande, a Harvard University surgeon, summarised this issue by explaining that “part of the bafflement occurs because the folks who know how to make such systems (i.e. intelligent agents) don’t understand how the clinical encounter actually operates” [16].

To make the challenges even more complex, the health-care system is part of a constantly changing environment made up of government, regulatory bodies, technology companies, medical insurance companies and, more importantly, prospective patients. Although a clear definition and delimitation is envisaged, the system can only be analysed as part of the environment, being characterised by connectivity and a high degree of influence for external factors ([2], p. 20). In state-funded health care, the government is the main investor and the patients do not have the advantages of a competitive market.

Administrative Decision-Makers

The health-care business sector provides a hint of how the idea of profit can influence the role of technology in health care. The money flow and assumed financial motivation of using intelligent agents in health care is beyond the focus of this chapter. What remains within focus is the interaction between intelligent medical agents, management and IT support staff in health care, whose concerns include resource allocation, reporting and performance both for people and technology.

The health-care managements’ motivation to support the use of intelligent agents is mentioned by Shortliffe (1979) cited by Liebowitz ([8], pp. 32-1, 32-2), who argues that an expert system should only be used if it improves the standard of quality of care at a justifiable cost in time or money or if it maintains the same level of quality by saving time or money.

Some of the technical information that appeals to political decision-makers and managers includes the following: IBM Watson is considered the first system to understand questions posed in natural language and research the entire body of medical knowledge and patient records to create a diagnosis plan in 3 s ([17], pp. 1050–1054). HP IDOL is described as recognising concepts, patterns and ideas in unstructured natural language descriptions delivering a significant impact on the productivity and efficiency of health-care professionals at the point of care. It is intended to contribute towards informing strategic decision-making, as an early warning system, or as a system to benchmark drug deployment, yielding rapid results [18]. Apart from the ability to search and interpret vast amounts of data which is virtually impossible for a human doctor, medical intelligent agents have been described as having a better diagnosis precision when it comes to known cases. IBM claims that Watson’s successful diagnosis rate for lung cancer is 90 % as opposed to only 50 % for human doctors [19]. Another element that supports the use of expert systems is connectivity and integration. While previous systems such as MYCIN were operating in isolation, contemporary expert systems are being developed with the intention to be able to interact with medical equipment such as EMRs and HIS, contributing towards improvements in the quality of care and more efficient resource management ([17], pp. 1053–1059). However, there is a danger that in many cases systems might be looked upon as a silver bullet. By reducing the exploration of possible leads and replacing them with a more certain path in assigning a diagnostic, it is seen to have the potential to address the medical personnel shortage and also support financial savings [20]. This is in line with what many decision-makers strapped for staff want to hear. However, the majority of these benefits resulted from various simulations and testing activities conducted by potential software vendors in collaboration with medical institutions that have a rich expertise of technical solutions such as the Memorial Sloan Kettering Cancer Center. One of the arguments supporting this statement is the vast gap between the theoretical benefits predicted and the real-world outcomes for digitised records systems in the UK ([21], pp. 92–107). The interoperability advertised by developers must also be regarded with caution. First of all, multiple data formatting might lead to brand loyalty issues and eventually the question of market monopolisation by certain vendors. Secondly, not all medical institutions start from the same level of technology adoption. While some might have successfully implemented electronic record systems, others are still operating with paper-based ones. They support mutual learning and knowledge sharing and coordination and might not be simply replaceable by their electronic counterparts ([15], pp. 79–83; [21], pp. 105–107).

The evolution of expert information systems in general has been shaped by the advances in technology. When Watson developers first decided that health care could benefit from its capabilities, they looked at the masses of unstructured data resulting from care processes but not necessarily at how people working with that data make sense of it [22]. Doctors did not say “we need help in trying to memorise millions of medical journals” but rather factors such as misdiagnosis rate, shortage of staff or failure to achieve waiting-time targets signalled opportunities for improvement [4, 23]. This means that there may be significant mismatch between problem solution and expected outcomes.

The medical and IT professionals share a vital responsibility: the data that the intelligent agent will learn and the format they will use to redistribute the information between people and technology. Specialists with wide practical expertise tend to develop their own “language”, jargon and internal collaborative code of practice (20]; [24; [25], pp. 2–3; [26]). The health-care industry is characterised by a certain level of discipline, a specific way of managing conflicting statements and an arbitrary level of detail. Barley et al. refer to the abstract models of work used in analysing a system as representations of provisional theories which might or might not capture the essence of people’s activity [15]. The interpretation required to load and unload data into an intelligent agent will cause further contextual difficulties. Overcoming those contextual difficulties can ultimately dictate the efficiency of the agent ([21], p. 105 [5]).

Around 75 % of medical students and junior doctors in the UK own a smartphone and occasionally use 1–5 medical applications ([27], p. 121), but the percentage is significantly lower with more senior care providers. Before being able to use an intelligent system, medical professionals will need training and technical support which will make an IT support team an absolute necessity with a guaranteed budget share [22]. This is an investment in organisational change and requires significant resources to be successful.

The use of intelligent agents is yet to be widely spread in practice, and their real benefits and limitations are still to be identified. However, some medical institutions have pioneered their use and claimed expected benefits from informative results. A preliminary announcement from the partnership between IBM’s Watson Group and Cleveland Clinic, Ohio, claimed that researchers at the clinic will use IBM’s Watson Genomics Analytics to enhance the use of personalised medicine based on the patient’s DNA. While doctors don’t have the time or the tools to explore specific treatment alternatives for individual patients based on their unique genetic configuration, Watson is said to be able to solve this problem [28]. On the same note, Watson has already ingested all 23 million medical papers in the National Library of Medicine (MEDLINE) and can access that data in milliseconds [29].

When Watson’s developers first envisaged to target the health-care industry, they regarded the patient’s case as a problem scenario. The need they identified was patients and caregivers are overwhelmed with “hoards” of unstructured, ever-changing data. The initial aim of the technology they created was to provide resources needed to rationalise important medical decisions [30]. Testing showed promising results in areas such as drug prescription, drug-to-drug interaction and drug-associated complications ([4], pp. 5–10). When moving to less predictable areas, it is vital to ensure that the intelligent agent is able to cope with clinicians not following all its instructions and support them along the path dictated by their practical expertise rather than predetermined, ideal scenarios.

As senior decision-makers, the investors (private/government) and health-care management personnel carry responsibility in assigning one of the following roles to an intelligent medical agent: replacement of human doctor, guarantor of diagnostic accuracy, human dependent repository of data or support for human doctor’s decision-making practices.

4 Consequences of Intelligent Agents’ Implementation in Health-care Organisations

The Demand

Improved living standards and advances in technology have made people more demanding over the years [20]. They want to be healthier and they want to live longer. If they get sick, they want to know the cause after being investigated for as little time as possible and to be given a quick, efficient treatment. In their view, there is no room for errors ([31], pp. 583–585).

The percentage of medical errors is situated between 3 % and 5 %, while 40 % of ambulatory malpractice claims are made for assigning an erroneous diagnosis. A study from John Hopkins University reveals that 40,500 patients die in intensive care in the USA as a result of diagnostic errors ([32], pp. 1–3).

The demographic increase and life expectancy growth have led to an increase in the number of patients whom medical bodies need to provide care to. According to a report by HSCIC [33], in 2013–2014 NHS personnel dealt with 42,400 NHS hospital admissions per day. The figure is 870 (2.1 %) more per day on average than in 2012–2013, while the greatest number of admissions by age band was for patients aged 65–69 (1.3 million, equivalent of 5.5 %). Although people tend to live longer, the healthy life expectancy has not increased at the same rate which leads to an increased need for care for an increased number of patients ([9], pp. 1–4).

These factors have led to challenges in dividing patients into categories and assessing their needs based on the affiliation to a single category. An example of the type of patients includes the elderly who represent the major consumers of health-related services including primary care (GPs), secondary care (hospitals), community (social nurses) and social care (care homes) [34]. They prefer doctors with whom they build long-lasting relationships based on patience, empathy and trust and at the same time demand relatively long and frequent consults and attention from the medical staff [35]. Other types are the younger and middle-aged patients who have very limited time and patience for health check-ups and prefer quick results to human relations and empathy. Being surrounded by smart, mobile devices many of them with built-in medical functions (i.e. applications that check blood pressure, intelligent fitness activity trackers), they tend to trust human doctors better if their view is confirmed by an app or a medical website. They also might be willing to get a second opinion, even if that is only a Google search of their symptoms and therefore might be more prone to support the utilisation of an intelligent agent such as Watson in health care ([10], pp. 516–517). However, their awareness of technology might make them more concerned regarding IS security and more demanding when it comes to their medical data handling.

Financial costs are an essential aspect to be discussed in relation to the patients’ view and acceptance of intelligent agents as part of the care process. A UK-based review conducted in 2011 has revealed that in practice, patients were billed more after the introduction of clinical decision support systems because the computer recommended additional tests and also because it was easier for doctors to order them on an online-based system ([21], pp. 92–107).

The above examples show how various factors can influence patients’ preferences towards intelligent medical agents. When looking at health care in general from a systemic point of view, it is vital to understand that patients are an essential stakeholder with increased decision-making responsibilities [14]. The systems thinking theory strongly argues that a contextual and holistic review of individual circumstances can be much more efficient than identifying broad categories of stakeholders and depersonalising the systems analysis ([2], p. 28; [3], pp. 3–5). Elements of disruption such as trust in technology and data security need to be taken into account in the use of intelligent agents interacting with patients. While people may be aware of intelligent medical agents through mass communication, they may have more questions when the doctor mentions, adds data or retrieves information from the agent. Additionally, doctors who have seen intelligent agents at work fear that Watson’s ability to identify many possible diagnoses will encourage patients to ask for even more tests and procedures, setting off a cost-inflating “diagnostic cascade” [23]. This might distract the clinicians from the contextually relevant and truly needed health-care solutions that apply to particular circumstances.

Supply

When it comes to the supply side, the users of an expert system can be divided into many categories. They can be doctors or medical support staff, and then the doctors could be researchers (professors), specialists, GPs or junior doctors; the support staff can include nurses, carers, receptionists or health advisors. They each have specific jobs, but more importantly, from a systems analysis point of view, they are individuals. They have specific work requirements, personal aspirations and ways to achieve excellence. They have different competences and abilities and may not always be able to identify contextual exceptions where the agent cannot be relied upon. The implementation of expert systems should not be dictated by how much medical literature a system can compile or its successful diagnosis rate.

As mentioned before, the key in analysing those ways is seeing patients as individuals and taking into account their particular characteristics, not classifying them as a homogeneous group. Data quality can only come under scrutiny at some point because of what is recognisable as “little data”, which is personal and immediate and a context-specific alternative to Big data ([36], pp. 355–356). Trust and recognition for the experts authoring the data fed into the intelligent agent are major factors of influence for the data users. Most of the times, in medical care, information is fit for use when the doctor or nurse trusts it or knows how competent the colleague who provided it is ([15], pp. 80–86). On the other hand, doctors considered competent by their colleagues might use their practical experience and tacit knowledge in many situations and the resulting “data” stays invisible and will not be published anywhere. A major risk for the future sustainability of intelligent medical agents is that experienced doctors will not be motivated to repeatedly question the knowledge of the agent while junior doctors might get into the habit of relying on the expert agent, following a robot-like set of instructions without being incentivised to expand their individual knowledge by experimenting in practice and eventually hinder them from developing intuition ([20], [3], p. 3; [21], pp. 100–107]). The downside might be that they lose motivation to learn and remember endless variations of the same case because they know that they can access this kind of information and even more in less than 3 s ([37], p. 986; [38], pp. 188–195; [23]), resulting in the appearance of functional stupidity [39]. Conversely, more experienced doctors might find the large number of alerts and recommendations repeatedly displayed by intelligent agents disturbing and distracting, so in practice counterproductive, and start turning them off without necessarily paying attention to every single one. To find a balance in the number of reminders that an intelligent agent should flag, developers need to consider more the doctors’ preferences and less the protection from lawsuits that vendors might face in the future ([21], pp. 106–107; [5]; [40], pp. 503–505).

This refers back to the actual need for an expert system; 75 % of diagnostic errors in the USA are reported to be related to cognitive factors which would translate in physician’s judgement limitations ([33], pp. 1–3). At the same time, there is little known about the opposite (e.g. when cognitive factors help identify exceptions).

To link back to the patient and their view of the situation, doctors cannot be seen as IT support workers (the patient queries them; they query a database and come up with the highest probability response) and IT experts cannot be seen as replacements for doctors. There is a large category of IT-skilled patients who google their symptoms, but in the end, they all see human doctors for an accurate diagnosis and treatment. Historically, the medical profession has been highly respected because of its human interaction and trust in the healing abilities of its people [16]. Patients are not prepared, at least at the moment, to compete with the intelligent agent for the clinician’s attention ([21], p. 80).

As with other computerised systems in different industries, there might be champions and there might be saboteurs. The difference in health care is highly hierarchical, based on long years of experience culture. Even if junior doctors and patients might be impressed by the technical specifications of an expert system, if a senior consultant with hundreds of hours of experience thinks the system is not viable, there are serious concerns to be considered [35].

At the same time, the number of experienced medical staff is decreasing. The figure for global health workforce shortage was 7.2 million in 2013, with a prediction to dramatically increase to 12.5 million by 2035 [38, 41]. The UK has temporarily found a solution to its shortage of medical experts by recruiting medical personnel, especially nurses from abroad ([42], pp. 558–561).

The advances in medicine and medical technology, discovery of new diseases and mutations of known ones and the enormous market of treatments represent too much information for a single doctor to learn and put into practice at the same time. Additionally, the medical knowledge generated by research and practice doubles every 7 years. The human body contains a number of variables that is simply too large for a human to monitor ([17], pp. 1051–1059). Young people who are currently studying medicine have been born in an era when a smart, mobile device is almost an extension of the human body. It is difficult to believe that they will be refractory towards technology enablement at their workplace, even if that is a hospital.

At the same time, the doctors’ views are conflicting. According to Herbert Chase, a professor of clinical medicine at Columbia University and member of IBM’s Watson Healthcare Advisory Board, “it’s not humanly possible to practice the best possible medicine. We need machines”. Given the creation rate for medical literature, a physician would need to read around 600 h per month in order to stay current ([43], pp. 21–27). Other doctors do not consider this a priority. Physician Mark Graber who heads the Society to Improve Diagnosis in Medicine thinks that “doctors have enough knowledge”. On the other hand, some suggest that intelligent agents such as Watson and Autonomy could overcome difficulties linked to the soft side of the human doctors and provide unbiased second opinions [16]. But then it can also be argued that human experts are capable of making contextually relevant decisions because of the same bias [44]. Other positive predicted outcomes would include encouraging patient questions, decrease duplicate data and the solving the issue of illegible hand writing by linking the agent to an electronic record system and prescription system ([4], pp. 5–7). There are many more questions that can arise upon the actual implementation of the technology in health-care organisations. Leaving aside the natural resistance to change, doctors will want and need to understand how the technology works. This will count towards a number of training sessions and also practice ones. Additionally, the doctors will put pressure on making the processes as transparent as possible, as without seeing the internal reasoning of the intelligent agent, they will not be able to understand and validate the final diagnostic. Apart from that, technology has proven breakable over the years. Having a patient on the operating table in desperate need of support and a technical fault with the intelligent agent will leave little time tolerance for escalation and troubleshooting processes. This points to technical support once again, but as opposed to other technical industries, with potentially deadly consequences. IT support people might have serious difficulties [20].

Patients expect timely and personalised care putting pressure on physicians to see immediate results in consultation, diagnosis, treatment and recovery. Their expectations play an important part in the role that expert medical systems can be assigned over the next few years. The need for human interaction and reassurance rules out the role of replacement of human doctor. For them, care is more important than protocols and predefined care strategies. They want care personnel to be attentive to their individual needs which can be easily dismissed by a depersonalised intelligent agent ([15], pp. 80–90). The agent’s role of human-dependent repository of data would not take advantage of the intelligent capabilities of the expert system, leaving the most likely role to be the guarantor of diagnostic accuracy ([25], p. 2).

As empowered as the technology might be, the ideology behind it is to simulate the cognitive capacities of the human brain. However, at least at the moment, a machine cannot be enriched with imagination, creativity or feelings. No human doctor is able to read through 23 million medical papers before providing a diagnosis, but at the same time, no computer can simulate human intuition or empathy. Modern computers have displayed outstanding results in terms of data processing, but medicine and patient care is about people, and ultimately, people should be the main decision-makers.

5 Ethical Aspects

Using an expert medical system as guarantor of diagnostic accuracy carries a sum of ethical concerns and liability issues. Bringing an intelligent agent to the medical act does not relieve medical staff from accountability and liability. As long as the people providing care are responsible for it in the patients’ eyes, they should also be the ones who decide how much they can trust and rely on the intelligent agents ([45], pp. 3–6).

One of the general issues with intelligent agents is “depersonalisation”. They have an inherent tendency to “empty out” the context of local interactions, specifically validity and authority, resulting in a lack of relevance. Intelligent agents capture professional expertise by formalisation—deploying impersonal knowledge, classificatory systems and procedures to shape, monitor, standardise and render calculable the work they support [34].

“Depersonalisation” leads to two follow-up concerns. The need for less medical experts might contribute towards a resistance to change showed by medical personnel who might feel excluded from the medical act. At the same time, if the intelligent agents will prove beneficial only when used by experts, it will lead to the exact opposite: more qualified experts are needed instead of mediocre personnel. Secondly, one of the major issues which has triggered long debates is the issue of liability in case of malpractice ([45], pp. 5–10). Preliminary studies suggest that intelligent agents will fall in the same category as robotic surgery or cyber-medicine when it comes to regulations. Only one death has been registered in the USA after using robotic surgery where the lawsuit was settled outside the court, so at the moment, there is limited expertise ([17], pp. 1053–1055).

Even before malpractice, there might be a series of conflicts between doctors and the intelligent agents. Pointing back to the roles that intelligent agents might be assigned in this context, there will have to be clear policies in place to clarify who or what has supreme authority. If the doctor can disagree with the intelligent agent and the treatment is unsuccessful, there is a follow-up question of who takes responsibility for the action. So far, doctors who have used Watson have superficially dismissed the discussion of responsibility and best course of action. Eric Topol, a genomics professor at the Scripps Research Institute, argued that since doctors ultimately make a final diagnosis, there’s no need for regulation like the one used for traditional devices used to treat patients [46].

Transparency and justifiable reasoning for the intelligent agent’s processes and outputs are the key (a necessity for a doctor to be able to make a relevant judgment). However, giving doctors and support staff access to the internal structure of the intelligent agent makes the technology more vulnerable to manipulation or potentially unlawful data collection. Going a bit further and linking the agent to a pharmaceutical database through the treatments it might suggest deepens the concerns regarding data processing, data collection and sharing and, eventually, market competition issues as the agent might develop a preference for a certain medicine or producer.

Expert medical systems have been prototyped for over 50 years, but their cognitive capabilities have not appealed to physicians in practice. Recent developments in medicine generated enormous quantities of information that physicians would need to learn and update constantly. However, the time dedicated to learning would prevent them from being able to maintain the standard quality of care while dealing with their regular number of patients. At the same time, adding a piece of technology as intermediary might further increase that time and introduce additional issues with training and utilisation, therefore making the intelligent agent a suboptimal solution to the problem. Additionally, while computers are better than humans at storing, remembering and processing data, it is vital to outline that human understanding and machine understanding are significantly different. For an intelligent agent, the data it learns is a succession of symbols and its lack of consciousness prevents it from grasping how the manipulation of the data could impact a patient’s life [47].

6 Conclusion

While various stakeholders in the health-care industry have very different opinions regarding the feasibility of using Watson for medical diagnosis, an essential feature distinguishes itself—that is individuality. Managers, patients and doctors can be seen as part of a system but cannot be simply divided into two basic categories: tech savvy and tech reluctant. Their particular characteristics, needs and expectations not only dictate their acceptance of intelligent agents in their daily lives (medical ones in this case) but also how they are being applied (e.g. their role in the decision-making process). Various characteristics will lead to various scenarios in real life which need to be considered before widely deploying intelligent agents.

The medical world might not be prepared to cope with an autonomous intelligent agent but, at the same time, might highly benefit from a combination of guarantor of diagnostic accuracy and human-dependent repository of data roles [16]. The intelligent agent may help with accuracy but not contextual relevance. If the artificial system proposes a decision how will the doctor sustain the ability to ensure it is the correct one? Doctors might save time by querying a machine but will waste valuable time in learning how to use it and to judge it. Perhaps the biggest difficulty will be to overcome potential functional stupidity and to sustain professional competency and capability.

Ideally, to balance the two, intelligent agents would tackle complicated and standardised problems while human doctors would focus on complex matters which require contextual deviation in decision-making as opposed to standardised assumptions. However, this would make the intelligent factor redundant and ignore stringent health-care issues such as shortage of personnel and human errors. Intelligent agents are here to stay, but expecting them to combine machine processing capabilities with emotional intelligence is an unrealistic short-term expectation. Getting them to work closely with humans, learn from experts that are dealing with people and their individual needs on a daily basis could take us one step closer to autonomous intelligent medical agents. The immediate priority is to enrich the agents with comprehensive learning capabilities, to make them able to cope with lessons that clinicians have learnt from practice not from books and, ultimately, to ensure that the agent will get this knowledge along the way by following the clinicians’ path rather than distracting them by dictating a completely new one.