1 Introduction

We aim to address the core of the ethical debates surrounding the use of decision-making systems based on machine-learning (hereafter: ML systems) for high-stakes decisions.Footnote 1 There is an abundant, and increasingly growing literature focused on the ethical problems of using ML systems for decision-making, e.g. in medicine and healthcare, criminal justice, job application assessment, insurance and loan qualification, etc. (Wexler 2017; Rudin 2019; Varshney and Alemzadeh 2017; Flores et al. 2016; Wang et al. 2019; Yeung 2019; Russell and Norvig 2016). The literature is mostly focused on the requests for explanations of the functioning of ML systems or of particular decisions (Miller 2017; Gilpin et al. 2018; Guidotti et al. 2018; Mittelstadt et al. 2019; Paez 2019; Samek et al. 2019; Rudin 2019). The pressure from both the academic literature and the various governmental institutions and NGOs to provide such explanations resulted in the project of Explainable AI (UNI Global Union 2018; Floridi et al. 2018; Hoffman et al. 2018; Gilpin et al. 2018; Ribera and Lapedriza 2019). It is strongly advocated that a right to an explanation is in itself an important right for the subjects of the decisions (Wachter et al. 2016, 2018; Samek et al. 2019; Goodman and Flaxman 2017; Zednik 2019; Guidotti et al. 2018; Zerilli et al. 2019). Designing various technical tools that provide insight into the functioning of ML is seen as a solution that should satisfy the right to an explanation.

We argue, however, that explanatory efforts only provide a limited value for achieving ethical AI. In cases when an algorithmic decision has already led to detrimental effects, e.g., a mistaken diagnosis, an unfair decision concerning parole release, or even more radically, a large-scale destruction caused by autonomous systems in warfare (Sparrow 2007; Asaro 2012; Apps 2021), it is important to be able to assign responsibility and legal liability, just as in human decision-making. The explanatory efforts concerning ML, however, do not automatically point to the locus of responsibility for the detrimental decisions. Moreover, it seems that there are deeply ingrained reasons why it may be extremely difficult to assign responsibility in the context of ML decision-making. The problems seem to stem from the characteristic way in which ML systems work, and it is not clear whether it is possible to overcome the ethical problems while keeping the advantages that ML provides. This paper aims to explain in what ways the inherent characteristics of ML create obstacles for assigning moral responsibility and legal liability.

The paper is structured as follows. In Sect. 2 we present the (mostly) inherent characteristics of ML systems. In Sect. 2.1 we briefly expose the main advantages of using ML systems for decision-making, which make them appealing for use in a variety of fields. In Sect. 2.2 we indicate the epistemic consequences of the inherent characteristics of ML. In Sect. 2.3 we address the attempts to remedy these consequences through the xAI project, and indicate why these attempts are not sufficient for achieving ethical AI.

In Sect. 3 we analyze the negative aspects of using ML for decision-making, which involve potentially harmful or rights-infringing decisions, and the problem of determining the locus of responsibility. In 3.1 we discuss the relationship between the characteristics of ML and the problems with control and responsibility. In 3.2 we present step-by-step how these issues arise in the ML decision-making process. We show that the problems with assigning responsibility arise from the inherent characteristics of ML presented in Sect. 2.

In Sect. 4 we consider potential ethical (Sect. 4.1) and legal (Sect. 4.2) approaches to solving this problem, and discuss the difficulties of adopting each of the proposed solutions. We conclude that the inherent characteristics of the way in which ML functions give rise to both the advantages of using ML and the difficulties in assigning moral and legal responsibility. This opens the question of the costs and benefits of using the systems which, although highly efficient, are inherently evasive to human control and difficult to regulate.

2 Inherent characteristics of ML

There are several characteristics integral to the way in which ML functions that we find as having significant ethical implications. We will now briefly describe these characteristics, and then proceed to elaborate on their epistemic and ethical consequences.

The first characteristic—semi-autonomy—refers to the self-learning of the ML algorithm. In the training phase, a parameter-updating algorithm learns the model from the patterns in the training data. The result of the training process is a mathematical model, which is a function that maps input features to output features and provides predictions for new inputs based on these patterns (Boge and Grünke 2019; Grossmann et al. 2021). For example, in ANNs, learning consists of refining or adjusting the weights of parameters based on the patterns in the data, where the weights indicate the relevance of a parameter for a particular output—a classification or a prediction. In other words, the higher the weight assigned to the parameter, the more important the parameter is for the given output (Hart 2019). For example, if a high weight is assigned to the parameter of age in the criminal risk assessment, this means that age is an important indicator of whether the defendant will be classified as a low-risk or high-risk category. Since the classifications and predictions such as these are determined by the semi-autonomous process of learning, the engineers do not have direct oversight or control over what the machine learns from the data and what classifications or predictions it will give as output.Footnote 2

The second important characteristic of ML systems is complexity. Namely, several aspects of ML systems involve unprecedented quantities, such as data and operations involved in the decision-making processes. First, the massive datasets that ML systems learn from, often called ‘big data’, meaning “rapidly collected, complex data in such unprecedented quantities that terabytes (1012 bytes), petabytes (1015 bytes) or even zettabytes (1021 bytes) of storage may be required”, and the data themselves also often have extremely large dimensionality (Wyber et al. 2015). The size of datasets is one aspect of the overall complexity. The other aspects concern the complexity of the ML models and involve the great number of paths the information could travel between input and output through hidden layers. For example, as the number of nodes in a layer, as well as the number of layers, grow in an ANN, this leads to having a number of, say, 108 of paths along which information can travel between layers (Boge and Grünke 2019).

Another important characteristic of a large number of ML systems is that they may behave non-deterministically. In this context, non-determinism means that, upon each execution, the model may produce different outputs based on the same input. The training algorithms can lead to different behaviors of the model, for example, if the same training data is used in a different order. When tested against the test dataset, the models (that were trained on the same training dataset) show slightly different performances. In contrast, deterministic modeling generates consistent output for the same input in each execution. This kind of modeling is used when the relationship between the variables is determined and there is no randomness or uncertainty. Non-deterministic modeling, on the other hand, is used when the relationship between the variables involved is unknown or uncertain, and it is more suitable for finding approximate solutions relying on the likelihood estimation of the probability of events (Mehta 2022; Ombach 2014).

2.1 Pros: why use ML

These inherent properties of ML bring about several advantages over traditional, human decision-making, such as accuracy, but also efficiency and reliability. Machine learning predictions prove to be more accurate than human predictions in many areas, in the sense that they produce fewer errors in classification or prediction tasks (Goh et al. 2020; Lee 2020). ML systems can also process tremendous amounts of data in a short period of time and provide predictions and recommendations for a large number of cases. This high efficiency is crucial in fields where decisions need to be made in a limited time (e.g. healthcare). Since ML systems can also process much larger amounts of data than humans, their capacities allow taking more parameters into account than human decision-makers. This may enable the ML system to capture more regularities and correlations in the data than humans would, but also the correlations that are not obvious to humans as being relevant.Footnote 3

In virtue of these advantages, namely, efficiency in processing a tremendous amount of data and providing highly accurate predictions, ML was found to be suitable for application in many critical areas. In healthcare, for example, ML is immensely helpful in diagnostics, therapy recommendations, and automation of tasks that would take too much time and effort from doctors and patients. ML is able to perform a thorough analysis and organize large datasets with many data points such as patient files, hospital records, etc. For example, using ML in diagnostics (e.g. the InnerEye project designed to differentiate healthy cells and tumors on 3D radiological images) results in an increase in diagnostic accuracy as well as reaching results much faster, which allows starting the treatments earlier. In therapy recommendation, for example, IBM’s Watson Oncology system uses the patient history to suggest multiple potential treatment options, thereby taking into account the personal specifics of each patient, such as drug interactions (Tkachenko 2021). Similarly, in the justice delivery system, the application of ML, due to its efficiency, has the capacity to reduce the pendency of cases and the number of unresolved cases in courts, which in turn affects the efficiency of the judiciary system, and ultimately has an impact on people’s access to justice (Pant 2021). In the banking and finance sectors, ML has proved highly efficient in fraud and money laundering detection. Fraudulent behavior is detected by ML algorithms that in real-time can examine an enormous number of data points, transaction parameters, and consumer behavior patterns (Sidelov 2021). The ability for self-learning and the capacity to handle enormous amounts of data thus make ML helpful for performing tasks that are too demanding for humans.

2.2 Epistemic consequences

The characteristics of ML that bring about its advantages also have specific epistemic consequences. ML models are often described as opaque or black boxes since the functioning of the model and the paths leading to a particular decision are not fully epistemically accessible and comprehensible to humans.Footnote 4 Opacity is particularly relevant in situations when it is necessary to determine whether there was an error in the decision-making process, and what was its precise cause. Here we briefly explain the different types of epistemic inaccessibility to gain precision and clarity concerning how ethical problems arise.

Semi-autonomy of the ML models results in the lack of epistemic accessibility. This form of opacity has not been addressed sufficiently in the literature, however, it was recognized by a number of authors as separate from the opacity that stems from the complexity of the models (Humphreys 2004; Schembera 2017; Boge and Grünke 2019; Srećković et al. 2022). We emphasize it as a distinct form of opacity because of its relevance for the ethical consequences we discuss in Sects. 3 and 4. The lack of epistemic accessibility concerns the lack of an explicit representation of the learned information. This may be described in different terms, e.g. as a “lack of an explicit algorithm linking the initial inputs with the final outputs” (Humphreys 2004, p. 149). As Matthias (2004) puts it, “[c]onnectionist systems lack an explicit representation, and the contained information can only be deduced from their behavior. (…) [W]e cannot:—have a look at the information that is stored inside the network, and, even more importantly;—see what information is not represented inside it” (p. 178–9.). This means, in short, that even the engineers do not have epistemic access to the exact processes of decision-making that lead from the input to the output.

The second reason for opacity is the lack of comprehensibility. One way in which ML is not comprehensible concerns the quantities involved. This makes the paths of information in ML models too numerous for any human to be able to ‘walk through’ in a normal life span (Boge and Grünke 2019; Lipton 2016). The massive leap in the quantities of elements involved in the ML decision-making processes raises them far beyond human cognitive capacities. More importantly, there is no explanation in human-understandable terms for why the ML model made a particular choice. The problem concerns mapping the information that is represented in the neural network into human-understandable information. Burrell (2016) calls this a mismatch between the mathematical procedures in ML and human styles of semantic interpretation, in the sense that what the NN represents is not suitable for human comprehension.Footnote 5

The third epistemic consequence of the way in which ML functions is the lack of predictability. The former two consequences concern the fact that the processes between the input and the output are either inaccessible or incomprehensible. The lack of predictability, however, concerns the dependence of output upon the input. Since ML modeling is often non-deterministic in the sense that the same input can lead to a different output, this makes the decision-making based on such modeling inconsistent. For example, based on the same parameters relevant to a decision on a parole hearing, an ML system may lead to different predictions, leading subsequently to different decisions for the subject. In other words, the characteristic way of ML modeling leads to epistemic uncertainty and unpredictability regarding its decisions.Footnote 6

The fourth, separate and somewhat artificial type of epistemic inaccessibility is the block of access via proprietary software: the corporations and other legal entities are protecting the code or a part of the code of the software they own, and as a result, other parties cannot be adequately informed about the exact process behind the decision-making. Since it is, in principle, always possible to remove this kind of block of access by removing restrictions on the code, it is not inherent and thus does not have the theoretical consequences that the other epistemic dimensions have. However, since it could still affect the ability to find the source of the potential errors or biases involved in the decision-making (discussed in the following sections), these practical consequences make them also worth considering.

To sum up, the lack of accessibility, the lack of comprehensibility, the lack of predictability, and the block of access are different epistemic obstacles posed by the often unavoidable characteristics of ML decision-making. They are generally present in ML decision-making but may become especially problematic in cases when ML decisions cause harm to the subjects of the decisions.

2.3 Explainable AI

Making decisions about people’s lives based on opaque decision-making raises ethical questions of whether this decision-making is trustworthy, justified, and ethical (Wexler 2017; Rudin 2019; Varshney and Alemzadeh 2017; Flores et al. 2016; Wang et al. 2019; Yeung 2019; Russell and Norvig 2016). The standard response to this issue is requesting explanations of decisions. ‘The Explainable AI (xAI) Project’ is the result of these tendencies, and its aim is often stated as increasing the level of insight into ML models by various technical tools.

However, the xAI project is by no means an easy solution and faces its own challenges. First, there are problems of determining what exactly are the explanations that are being sought in the sense that the very notion of explanation in this context is underspecified.Footnote 7 The term ‘explanation’ in the xAI literature and in the external requests for explainability mainly appears in the cluster of interconnected but also insufficiently specified notions such as comprehensibility, interpretability, transparency, understanding (Lipton 2016), and there is no consensus on what constitutes either of these notions in the context of machine learning, nor is it clear how to measure them (Molnar 2019). The majority of the technical literature seems to rely on an idiosyncratic conception of explanation, exemplified by any tool that offers any kind of insight into the model’s working. Many of the explanation methods offered in this field are thus criticized for various reasons, for being incomprehensible, inconclusive, potentially misleading, or not proper explanations (Mittelstadt et al. 2019; Rudin 2019).

There might be additional limitations of the xAI project that result from the characteristics of ML, which functions semi-autonomously, non-deterministically, and involves tremendous complexity on several levels. Some aspects of ML decision-making are inaccessible for representation by any xAI tools, since they “never take durable, observable forms” and exist in memory only temporarily (Ananny and Crawford 2018). Furthermore, the xAI methods cannot in principle faithfully represent the original model’s calculations, since they perform different calculations than the original.Footnote 8 If they performed the same calculations, they would, in fact, be equal to the original model and would ipso facto be too complex to comprehend (Rudin 2019). The xAI methods, therefore, do not reflect exactly how the decisions were reached. Moreover, because the original model is opaque, we are not in a position to determine how much the xAI representation diverges from the original decision-making. Some authors find this problematic for reaching an adequate explanation, and raise the question of the extent to which the xAI itself can be trustworthy, and whether it can grant trustworthiness to the original model (Rudin 2019). It might be responded that every explanation rests on simplification and idealization and as such is not completely faithful to its explanandum. The truth is, what constitutes a good explanation is still a matter of debate in epistemology and philosophy of science, and, as we noted at the beginning of this section, is even more unclear in a novel field such as ML.

These are the problems that reflect the current state of the art in xAI and may be remedied with further research. Some authors, however, object to the pursuit of complete transparency of ML decision-making, as it supposedly results from a double standard, since human decision-making is non-transparent as well. Human brain is also considered a black box, and there is no access to all the steps that human decision-makers implement in reaching decisions (Zerilli et al. 2019). A human judge, for example, may make a decision about a parole hearing that is racially biased but also rationalize the decision, a posteriori, by appealing to more acceptable reasons. It may seem that the request for complete transparency of ML decision-making poses unrealistically high standards that are not able to be satisfied in the context of human decision-making. However, there is an important difference between these two contexts. Notwithstanding the non-transparent processes of decision-making, in the human context, the locus of responsibility is clear—the person who has made the decision is responsible if it is established that the decision is unfair and/or has led to harmful consequences. In the context of ML, however, the specificities of its way of functioning pose serious obstacles to determining the locus of responsibility for errors and harm. The xAI methods may indeed reveal some aspects of the functioning of a particular model. However, the attainable level of transparency provided by the xAI may suffice for purposes such as model improvement or providing the end-users with some sort of explanation of the decisions, but it does not suffice for resolving the issue of responsibility. We discuss this issue extensively in Sects. 3 and 4, and show in what ways the unavoidable characteristics of ML pose obstacles to assigning responsibility for errors and harm of ML decision-making.

3 Cons: harm and responsibility

Damaging consequences that may arise from the use of ML decision-making can be divided into separate categories: tangible harm and human rights violations. Although tangible harm and violations of human rights are separate issues, they often coincide. As Yeung (2019) notes, a human rights violation does not have to include tangible harm, and vice versa: (a) when Facebook removed the famous photograph of a naked 9-year-old girl fleeing bombs during the Vietnam War, it could be understood as a violation of the right to freedom of expression and information, even though there was no tangible harm; (b) a self-driving car injuring a wild animal inflicts obvious tangible harm, but it does not violate a human right (Yeung 2019). Nevertheless, here we consider these consequences jointly since our points apply to both. Since opaque ML models are used for making decisions in many fields that fundamentally affect human lives, such as medical diagnosis and treatment, credit scores in loan or job applications, allowing probation, etc., decisions made in these fields are able to both violate human rights and cause tangible harm to its subjects. For example, the decisions based on a flawed automated process may wrongfully deny someone of their freedom (in criminal justice applications), cause prolonged illness, or even directly harm the subjects of the decisions (e.g. in medical applications). Harmful decisions may be caused either by individual errors or by systematic biases embedded in the ML decision-making processes.

Harmful or rights-infringing decisions which result from faulty decision-making raise the issue of assigning responsibility. Responsibility can be moral or legal. The terms related to legal responsibility, such as accountability and liability are often used interchangeably (Cornock 2011; Yeung 2019). Since these terms are often confused or used inconsistently in the literature, we will use them according to the following definitions. By ‘responsibility’ we always mean ‘moral responsibility’, that is, we refer to the abstract ethical concept which, when attributed to an agent, implies that the agent who is responsible for some action or outcome deserves moral praise or moral blame (Zimmerman 1997). Responsibility is sometimes considered closely related to the concepts of ‘duty’ and ‘obligation’, which the agent is expected to uphold (Talbert 2022). In the philosophical literature, moral agency is considered a precondition of moral responsibility, and moral agency, in turn, requires properties such as consciousness, intentionality, and free will (Johnson 2006; Sparrow 2007; Asaro 2014 and Hanson 2009).Footnote 9 Whenever we are talking about legal responsibility, we will use the term ‘liability’. By liability, we refer to the related concept, also attributable to moral agents, which implies that there are effective institutional mechanisms that are expected to be deployed in cases of damaging outcomes, including the mechanisms to appropriately sanction responsible agents.

Although moral and legal responsibilities are most often linked, they are nevertheless independent. We can have cases of moral responsibility in which there is no legal responsibility, as well as cases of legal responsibility without clearly defined moral responsibility. When it comes to regulating an area, such as the use of ML, it would be ideal to determine moral responsibility and build a legal framework based on it. In that case, the agent responsible would also be liable and thus subjectable to the established institutional mechanisms of liability. There are many cases, however, where moral responsibility and legal liability do not coincide. For example, an agent may be found responsible for some harmful outcome, but if there are no institutional mechanisms of liability, the agent would bear only the consequences which remain in the abstract sphere of moral judgment without practical consequences or sanctions.Footnote 10 Or conversely, there may be cases where the moral issue is too vague, and it may only be legally regulated by circumventing the issue of moral responsibility and defining liability independently.

Decision-making based on ML systems involves distinct difficulties in assigning both responsibility and liability for damaging decisions. These difficulties arise from the inherent characteristics of ML models themselves and are further aggravated by the complex and dynamic socio-technological context of the development and use of these models. The fact that this type of decision-making involves errors and potential harm is not particularly problematic in itself, since any decision-making system is flawed and prone to harmful errors. However, when it comes to ML decision-making systems, a specific problem arises with attributing responsibility for errors and damage.

In the following sections, we show that the responsibility issues stem from properties that are unavoidable and are an integral part of how ML functions.

3.1 Control and responsibility

In Sect. 2 we discussed how ML decision-making is inherently characterized by semi-autonomy, non-deterministic modeling, and complexity that transcends human cognitive capacities. Here we show how these characteristics reflect on the issues of control and responsibility. First, an ML model autonomously adjusts the decision-making parameters without direct human interference. This means that determining the most crucial factors of decisions is delegated to the model, and humans have no direct control over the processes through which the decision-making proceeds. The lack of control is worsened by the quantitative characteristics of ML decision-making. The volume of information, and the complexity involved in ML decision-making lead to a transcendence of human cognitive capacities (Humphreys 2009; Srećković et al. 2022). The quantitative characteristics thus make it extremely difficult for humans to trace or comprehend the processes of decision-making, which complicates efforts to control the processes.Footnote 11

An additional aspect of ML decision-making that aggravates the problem of assigning responsibility concerns the data that is being fed to the ML model. The data has been gathered over long periods, and only reflects the previous practice of various sectors. The model learns the parameter weights from the correlations among the given data and produces predictions and decisions based on what it has learned. Since algorithms learn from existing patterns, they tend to learn human biases as well. If the data are biased (as it often is), or possesses inaccuracies, this may lead to biased decisions. Quite often, biased models can already be in use (long) before the bias is revealed and fixed. For example, the algorithm widely used by U.S. hospitals and insurers for healthcare allocation was found to prioritize white people over black people when individuals of both groups had equal health conditions. The algorithm based the patients’ risk scores on previous medical costs, and ended up identifying as a priority the social groups with higher access to, and ability to pay for medical care (Price 2019). Similarly, the criminological software used for predicting the risk of a future criminal offense in the U.S. judicial system was found to be biased against African American defendants, basing its predictions on the previous practices of allowing and denying parole to defendants with similar background profiles (Angwin et al. 2016). Finally, Amazon’s hiring algorithms showed clearly sexist preferences toward male candidates. Since it was programmed to replicate existing hiring practices and the patterns of profiles of those previously employed, “the system taught itself that male candidates were preferable” (Lauret 2019). Even though sometimes biased data is all that is available, we cannot just blame the data and call it a day. This is why engineers look for ways to fix the problem, and the xAI methods can help recognize biases and enable them to take measures to correct them. This is not a straightforward task, and some biases can still slip through despite the measures taken (Zhao et al. 2022). It may be difficult to determine who is responsible for biased decisions while the system, which was developed in good faith and was not known to be biased, was used.Footnote 12 Hidden biases in the data can thus additionally blur the locus of responsibility along with the other features we discuss.

In addition to the inherent characteristics of ML, the lack of control is aggravated by external factors. Because of the various advantages provided by these systems in many different fields, there is a growing trend in using ML systems for decision-making. In combination with the number of sectors using ML systems and the number of people subject to automated decisions, along with the fact that these numbers will most probably tend to grow as well (Butler 2016), this would allow for worldwide automated decision-making on a massive scale in a variety of contexts. Even at the current rate of usage, it is not clear whether it is possible for humans to inspect the correctness of the decision-making processes occurring in various fields.Footnote 13 With the future expansion of the use of ML systems, the possibility of supervision should be diminished even further.

3.2 Problems at every step of assigning responsibility

A number of steps need to be taken to achieve the adequate assignment of responsibility. Here we describe in more detail how the characteristics of ML complicate this process at each step. The first step of assigning responsibility is to determine whether the ML decision-making process was correct and fair. Due to the characteristics presented in Sect. 2.2, namely, lack of accessibility, predictability and comprehensibility, as well as the external factors such as the massive and growing use of ML, it is not clear how humans could carry out sufficient monitoring of the correctness and fairness. This leads to a general loss of human control in the sense that it becomes exceptionally difficult for humans to conduct meaningful scrutiny and surveillance over the ML processes necessary for assessing the correctness and fairness of decision-making. Thus a significant (and a growing) number of cases of automated decision-making must remain unexamined, and there are bound to be errors that will never be recognized.

For the remaining cases where inspection did expose errors in the process, the second step—determining the source of error—is obstructed by the epistemic characteristics of ML processes. Due to the lack of accessibility and lack of comprehensibility, it is often difficult to determine whether an error is a consequence of the model itself or of the data on which it was learned. Although there are techniques for determining this indirectly (by making changes and deducing from the behavior of the model), in practice it is not always a realistically manageable task. Additionally, the factor of blocked access due to proprietary software means that third parties (e.g. independent committees or other parties) cannot properly access the causes of errors or biases in the decision-making process. This could practically mean that assignment of responsibility is in the hands of the ML system’s private owners, who may not have incentives to investigate the causes of errors or biases.

In the even smaller subset of cases where it was possible to pinpoint the source of error, another difficulty occurs in determining whether the error could have been predicted, as predictability is relevant for assigning responsibility (Yeung 2019).Footnote 14 However, the epistemic consequences we have discussed in Sect. 2.2 hinder predictability in multiple ways. Due to the autonomous, non-deterministic, and complex nature of ML processes, or due to the errors coming from the data gathered over long periods, it seems extremely difficult for anyone involved in developing or operating the ML model to predict the errors in the decision-making process. Given that the very technique of machine learning is such that the engineer has no direct and complete control over what decisions the machine will make, it is questionable whether human agents should be held morally responsible for the unpredictable decisions of the machine (see Sect. 4.1 for more details).

Alternatively, even if in some cases it is established that errors and harmful consequences could have realistically been predicted and avoided and that humans are responsible, a complex question arises as to which of all the agents involved in the process should be held responsible. ML models are hardly ever developed and used by a single individual. Rather, the models are commonly developed by a team or even teams of experts, who often compartmentalize the specific development tasks. This problem is named the ‘many hands’ problem in the literature (Yeung 2019). It may be encountered in fields where the traditional conditions for responsibility, such as intent, knowledge, freedom of action, etc. are distributed over many different individuals, and none of the individuals separately meet all the conditions (Nissenbaum 1996). Since multiple individuals and mostly teams of individuals are involved in making, approving, and using the ML models, the assignment of responsibility becomes a much less straightforward task. There are many candidates: the company that designed the model, the specific people involved in the design or just the people in the management positions in the process, the institution that uses the decision model, or some higher governing body that authorized the use of the model, the government of the state that allowed such automated decision-making, the ML model itself (if machines are afforded the status of moral agents), or a combination of the above candidates.

An additional difficulty in determining the locus of responsibility arises from the socio-technical context of the application of ML models. It is what we call the ‘two-way nescience’, which arises from the fact that a number of experts who develop ML models often know very little about the fields in which the models will be used (for example, in judiciary or banking systems); and conversely, numerous experts in the fields in which the model is being used know little or nothing about the functioning of the ML (for example, a judge who uses ML system for a parole hearing). This way of usage of ML systems means that the persons who develop them and the persons who apply them are quite often located at different epistemic positions, due to often disparate expertise. Consequently, the errors that may occur in the design or usage of models could result from the difference in the epistemic positions of agents involved (e.g. a developer not properly understanding the details of the judiciary system, or a judge not properly understanding the dependence of the model’s score on the data). It is not clear in this context how to correctly assess who should have predicted the detrimental outcomes of the ML model application. In such complex contexts, it is very difficult to achieve an objective assessment of the responsibility of each individual agent because the body assessing liability would have to have expertise in multiple areas, and have insight into all the relevant factors from all areas in which the different agents operate. Making a credible assessment of responsibility may require a dialogue of experts from different sectors involved in a given ML application, which does not appear to be easily achievable in practice.

Finally, for the smallest subset of cases in which all of these obstacles are evaded or overcome, and it is clearly established who are the persons responsible for the harmful consequences, yet another difficulty is posed by the absence of elaborate legal and professional mechanisms that should ensure that the responsible person really bears the consequences (Mittelstadt 2019).

4 Potential responses to the responsibility problems

At the most general level, there are two main approaches for resolving the issue of responsibility.Footnote 15 The first approach would be to resolve the issue of moral responsibility and then to build institutional and legal norms based on that solution. We call this approach ‘responsibility first’. Within it, we consider the possible options of attributing moral responsibility for ML decisions, particularly with regard to the fact that both human and autonomous machine actors are involved. In considering this approach, we are primarily interested in potential ways to provide practical guidelines for legally regulating the liability for harmful ML decisions based on moral solutions.Footnote 16

However, since ethical debates are complex and require long-term deliberations, the second, more expeditious approach also seems reasonable to consider. We call this approach ‘liability first’. It involves putting the issue of moral responsibility aside and constructing a legal framework that could regulate this area and protect end-users as well as other involved parties without addressing the locus of moral responsibility.

4.1 ‘Responsibility first’

The first, obvious option is to attribute responsibility to human actors (Hall 2001; Goertzel 2002; Johnson 2006; Sullins 2006; Bryson 2010). The rationale behind this option is that the ultimate responsibility for the actions of machines, no matter how intelligent and autonomous they may seem, is always human because it is humans who “determine their [the machines’] goals and behavior, either directly or indirectly through specifying their intelligence, or even more indirectly by specifying how they acquire their own intelligence” (Bryson 2010, p. 65). This is a similar line of thinking to Lady Lovelace’s Objection which notes that machines can only do whatever we know how to order them to perform, regardless of the level of their sophistication (Turing 1999; Gunkel 2020). However, as suggested in the previous section, this path is fraught with difficulties. The main difficulty in attributing responsibility to humans is their lack of control over the processes through which ML models reach decisions. The causes for the loss of control—machine semi-autonomy, non-deterministic modeling, and the complexity that transcends human cognitive capacities—open a specific question of the extent to which human beings are morally responsible for the decisions made by machines. Commonly, responsibility is attributed to human beings on the basis of their autonomy and control as decision-makers (Fischer and Ravizza 1998; McKenna 2008; Lunney and Oliphant 2013; Yeung 2019). If the control over the decision-making is transferred to a machine, it may be argued that the responsibility also belongs to the machine and that it would be unjust to hold humans responsible for the machines’ decisions (Matthias 2004). Thus, it may seem unjust also to hold humans responsible for the absurdly large number of (semi-autonomously reached and unpredictable) decisions that people have no control over, and over which we cannot realistically expect to be properly monitored or inspected. Attributing responsibility to humans would mean holding human agents responsible for consequences that they could not have foreseen or prevented. Additionally, even if we are prone to accept that humans are responsible, the already mentioned problems of many hands and two-way nescience would pose significant practical obstacles regarding which particular humans we should declare responsible.

Another option would be to hold the machines responsible for the decisions they make. The development of autonomous and social machines undermines the pure instrumentalist view of technology, as it seems that autonomous technologies have transcended the role of being mere human tools, and have instead come to occupy the role of human agents themselves (Gunkel 2020). This is most evident in the examples of decision-making systems or self-driving vehicles. The option to assign responsibility to machines may, on the other hand, seem implausible because it is generally accepted that attributing responsibility requires moral agency. Attributing moral agency to an entity, in turn, requires the possession of mental qualities such as consciousness, intention, etc. (Johnson 2006; Sparrow 2007; Asaro 2014; Hanson 2009), which does not seem plausible to assign even to the most sophisticated machines, at least at the current stage of development.

Notwithstanding the theoretical debates on the moral status of autonomous machines, there are also important practical problems with this option. Even if we accepted attributing responsibility to machines, it is not clear what practical consequences this would have. What exactly does it mean in practice to consider a machine morally responsible? The question is not only whether machines could be seen as persons and as having genuine moral status; the more important question is whether it would make sense, from a moral (and consequently legal) perspective, to treat machines as persons in the same way that we currently treat human agents and organizations (Gunkel 2020). Thus, one problem with this option is that it has very unclear practical consequences. Another problem is that it is potentially unfair, in that it may lead to underestimating, or even ignoring the further role and responsibility of human agents, and this may be abused for evading the responsibility of humans involved (Johnson and Miller 2008; Gunkel 2020). If the practical consequences are not adequately specified, this direction might boil down to practically holding nobody responsible, which we address in the following passages.

The third option would be to adopt a model of hybrid moral responsibility, or ‘extended agency theory’ that would imply that responsibility is shared between humans and machines (Johnson 2006; Hanson 2009; Verbeek 2011). This option relies on understanding responsibility as distributed across a network of agents involved in the decision-making, including both humans and machines, as well as organizations. It is based on the intuition that all agents involved need to be assigned responsibility for the results of the decision-making. However, this solution also has unclear implications. It still needs to be decided how exactly to distribute the responsibility, namely, what aspects of it belong to machines, and what aspects should be assigned to humans (Gunkel 2020). In addition, it seems extremely difficult to determine who of all the individual agents involved, be it human or artificial, is responsible for which aspect of the harmful decision. Similarly to the option of assigning responsibility solely to machines, this option suffers from having unclear practical consequences, as well as being potentially unjust. Namely, if it is not specified exactly which part or aspect of the responsibility belongs to humans, and which to machines, then the persons who work with the machines, the so-called ‘humans in the loop,’ might serve as the default culprits whenever the machine reaches a harmful decision. Alternatively, if the shared responsibility was to be interpreted in a contrary way—leaning more to the side of the machines—the presence of a machine in the decision-making could serve as a default excuse for human error and negligence (Siponen 2004; Johnson and Miller 2008; Mowshowitz 2008). Both tendencies of interpreting shared responsibility carry obvious moral dangers. This suggests that it needs to be precisely elaborated on which way the participation of machines in decision-making affects the attribution of responsibility to human agents.

Finally, it could be argued that in some cases nobody is responsible for the decisions of the machines. Humans are not responsible because they cannot control machine decisions, provided that the decision-making models are constructed with the best intention. In addition, machines are not responsible because they do not meet the epistemic conditions for being considered moral agents. If the error or bias stems from the data collected over many years, it may seem that there is nobody to whom this type of error or bias should be attributed. Such a scenario makes it implausible to assign responsibility to anyone in particular, even to the machine. The damage caused as a result of machine decisions would then be observed analogously to the consequences of a force majeure, such as the damage caused by, say, atypical weather conditions. Due to the high stakes involved in many areas of ML decision-making, and the gravity of the damage that can be caused, this option seems unacceptable, as it implies practically abandoning the assignment of responsibility. It may be asked whether it is unfair to use autonomous systems if no one can take moral responsibility for harmful decisions. Some authors have taken the failure to assign moral responsibility as an argument in favor of completely abandoning the use of autonomous systems (Sparrow 2007; Asaro 2012).

Determining how to assign moral responsibility for harm done by ML decision-making thus seems extremely difficult. This is why a more practicable approach may be taken in the absence of an ethical resolution, or until the issue of moral responsibility is resolved. It might be practically more useful to try to define liability, even if moral responsibility is not sorted out. We will now present possible directions in resolving the issue of liability, which bypass the problems of finding morally responsible agents.

4.2 ‘Liability first’

As things currently stand, there are no established professionally or legally endorsed liability mechanisms for the regulation of the AI sector. The development of AI technology is proceeding faster than the development of legal and regulatory mechanisms to control and supervise it (Mittelstadt 2019). We thus propose potential solutions which might be introduced from regulatory practices of other sectors.

One possible direction would be to enable end-users to participate in the decision-making process by presenting them with the potential risks and benefits of using ML and letting them consent to such a decision-making system being applied to their case. This solution would be similar to the informed consent practice used in other fields. For example, in medicine, informed consent means that patients are presented with the risks of a procedure and that they can consent to take those risks. By that means, informed consent is a type of involving the user in a decision about their case. The user thus takes on part of the responsibility for the risks the decision-making carries.

While this option could provide the basis for expedient regulation, it should be borne in mind that because of the inherent characteristics of ML, the informed consent practice may face certain challenges that do not arise in other fields. As already mentioned, the epistemic characteristics of ML often complicate the detection of errors and the attribution of responsibility for those errors. This involves determining whether the harm was a consequence of recklessness or abuse, rather than of an accidental error. In other fields, the cases of such unethical actions are not covered by informed consent, and the liability cannot be disclaimed. Since ML often precludes differentiating the two classes of cases, adopting the informed consent practice would mean considering all cases of harm to be unproblematic by default. In this way, the practice of informed consent bears the risk of being used as an alibi for human negligence or abuse.

Another direction in regulating liability in ML would be to proclaim that the liability is borne by the institution which benefits by either selling the ML software or by using it. The regulation would specify the types and amounts of compensation afforded to the affected party by the institution in cases of harm caused by ML decision-making. Such regulation already exists in other fields, in the form of the court ordering an institution to pay compensation or damages to the person for the harm suffered, known in many countries as the no-fault compensation system (Henry et al. 2015). This, again, does not require assigning moral responsibility for resolving liability. This solution has great practical advantages because it does not require finding an individual culprit or the exact source of the error that led to the detrimental outcome. This makes it especially convenient for the context of ML where it is incredibly difficult to determine this. No-fault liability would require that the end-user shows that the ML error was a causative factor in the resultant detrimental outcome, irrespective of who specifically is to blame. Thus, no-fault liability demands “proof of causation rather than proof of fault” (Gaine 2003). This direction of assigning liability to companies or institutions might seem more fair to the general public than the informed consent practice because liability is born by someone who profits from the use of ML, instead of someone who is negatively affected by it.

It should be noted that this solution comes with its own difficulties, albeit they do not arise from the inherent characteristics of ML, but rather from more general societal factors. Namely, AI technology promises to be tremendously profitable for those who own it. If the profit heavily outweighs the compensations, the penalties may not have noticeable practical significance, and the agents labeled liable might have no financial incentive to uphold any ethical principles when designing or using ML systems.

Legal solutions, however, desirable and urgent, without an ethical basis, do not seem to be the ideal solution and leave room for concern about the direction in which the AI sector will develop. The decisions about the directions of development and the kinds of AI technology are made mostly by private companies, not with the public interest in mind, but rather guided by market logic. In combination with the already mentioned increasing massivity of the use of ML in diverse areas of life, and the impact this decision-making has on human lives, the AI sector bears enormous power to shape the future of humanity. The unresolved issue of moral responsibility thus remains significant and should not be dismissed even if the legal regulation is well-established.

5 Conclusion

Using ML models for decision-making has important advantages over human decision-makers. The semi-autonomy of the model in processing large amounts of data, and the capacity to process the quantities which by far transcend human cognitive abilities help make ML models highly efficient and accurate in their predictions. On the downside, when faulty ML processes cause harm to the subjects of decisions, assigning responsibility for the harm faces serious difficulties, due to the unavoidable and inherent properties of ML systems that make them immune to human control and surveillance.

It seems that the further development of the ML field is directed toward ubiquitous automated decision-making on a massive scale in a variety of contexts, and that the problems with control and responsibility are more likely to be amplified in the future. Given the accelerated development of the field, resolving the regulation problems seems urgent to avoid possible abuse, and protect all actors involved in the decision-making.

To sum up, we uncover a conflict within the field of machine learning. What was considered its main advantage (handing over enormous quantities of information-processing tasks to a ‘machine’, which can perform them incredibly faster than humans) has turned out to be precisely the aspect that produces ethical problems for which it is not clear how they could be overcome while keeping the advantages. It is an open question whether the advantages of the use of ML are worth the potentially irresolvable issues of moral responsibility in cases of harm caused by the decisions, and whether it is advisable to abandon the use of ML models for high-stakes decisions (Sparrow 2007), and go back either to human decision-making, or at least to simpler, transparent decision-making models (the latter was suggested by Rudin (2019) based on a different, but complementary rationale). Further research is needed to weigh the costs and benefits of such a ‘regress’.