Introduction

I want to thank Drs. Farisco, Evers, and Salles for their compelling cautionary statement regarding the ethical evaluation of artificial intelligence. The themes in their work merit additional exploration with both critical and expansive eyes and I will follow their outline regarding essential criteria for artificial intelligence as well as both theoretical and practical issues in intelligence and ethics. Their work invites comparison to findings in both cognitive psychology and neuroscience (and popular culture)—the questions raised about deliberation and moral agency draw immediate parallels to our own processing as the model with which we have the most direct experience. Ultimately, there are ontological questions as to whether we ought to feel challenged by the possibility of artificial moral agents and whether we ought to welcome a potentially different model of ethical reasoning, considering the cognitive architecture for our own moral reasoning can be (charitably) described as haphazard.

Essential Features of AI

In their introduction, Farisco, Evers, and Salles introduce the controversy in defining artificial intelligence, suggesting the minimal criteria of “perception of the environment through sensors; reasoning/decision-making on data perceived; and actuation through executors.” Self-learning machines epitomize these criteria as the apparent endpoint of a continuum of less “intelligent” machines. Artificial intelligence takes many forms from programmed algorithms to autonomous vehicles and robots.

There is immediate appeal to the criteria and obvious parallels to our own cognitive experience of the world around us. Throughout our history, we have fashioned ourselves to be reasonable and mostly rational beings, perceiving the world through our own phenomenological filters, weighing and evaluating what we perceive, drawing inferences and other logical relationships, and acting based on those perceptions. It only stands to reason that we would use a similar understanding in conceptualizing an artificial cognitive agent. There is a prima facie truth in this model, but is it sufficient?

What is Reasoning?

There are legitimate questions about what constitutes reasoning (Goel 2007; Stanovich & West 2000)—the input and weighing of evidence is certainly part of it, but we find that there are limitations in “reasoning” human and non-human animals. If we confine “reasoning” to purely rational processes (e.g., absent emotional connection, instincts, etc.) in a Kantian or Cartesian sense, then none of us really qualifies. Our cognitive architecture weaves emotional and preconscious processing throughout our reasoning process (Bargh 1997; Clore & Ketelaar, 1997; Damasio 1995; Evans 2010; Fauconnier & Turner 2002; Franklin et al. 2014; Haidt 2001; Homan 2003; Isen & Diamond 1989; Logan 1989; Prinz 2015; Smith 1997; Turner 2000; Uleman & Bargh 1989; Wyer 1997). These intuitive and emotional responses are not volitional and are much deeper structures in our brains. As such, contemporary arguments about human “reasoning” must necessarily account for a much more complex cognitive model.

Farisco, Evers, and Salles are not exploring human reasoners, and as such, it isn’t clear that the architecture of human reasoning and agency would be applicable here. However, in the spirit of their caution, I raise these issues because they also explain some potential concerns about artificial agents, whether programmed or naturally learning. They note that there are fundamental elements of human cognition that are absent in current AI and may prove impossible to translate into artificial experience (e.g., counterfactual reasoning and emotional experience). They rightly note that these challenges may serve as barriers in particular ethical contexts (childcare, healthcare), but there are additional concerns as well. If artificial agents have deficits in emotional processing, for instance, it is entirely possible that their moral calculus may yield the ethical problems identified early in this discussion by Wallach & Allen (2009)—both top–down and bottom-up moral methodologies are insufficient. We cannot easily optimize for a pure utilitarian calculus any more than we can create a Napoleonic code sufficient for deontological absolute prohibitions or contextualization of prima facie principles. As such, “reasoning” in this case will either be incomplete (from our perspective) or yield morally repugnant or incoherent recommendations. I will be returning to these concerns as this article progresses.

Potential Biases in Reasoning

Empirical problems emerge in artificial ethical reasoning regardless of whether we use programmed ethical rules or allow machines to learn on their own. If we trust artificial agents to generate moral rules by their own understanding of interactions between other moral agents, we risk the development of artificial agents representing our worst inclinations and opinions (Angulo 2018). Without constraints on what they learn and how, “self-learning” artificial agents quickly can be manipulated into parroting horrific human sentiments. While it may be possible to program constraints (e.g., watching for the use of particular keywords during interactions), the fluid nature of natural language, the ease with which evil intentions can co-opt benign phenomena (Anti-Defamation League n.d), and a wealth of other concerns make these constraints reactive rather than proactive. Programmers would have to update their agents constantly and without real-time awareness given this fluidity. Simply put, it would seem that self-learning artificial agents would be vulnerable to manipulation and contextually naïve.

On the other hand, if we do not adopt a machine-learning model, we run the risk of a host of induced biases based on the nature of our cultural and personal preferences (Lloyd 2018; Sweeney 2013). Algorithmic bias reflects the programmer who is culturally situated and influenced, increasing the risk that data analysis will yield skewed results. There is already empirical verification of this, whether we consider cases like Amazon’s hiring practices (Dastin 2018) or risk assessments of recidivism reflecting racial biases (Angwin et al. 2016). In essence, just as human reasoning reflects myriad backstage emotional, intuitional, and preconscious elements, there are significant risks associated with artificial agents.

Contextually Appropriate Reasoning and Decision-Making

The crux of the concerns raised here is that “reasoning/decision-making based on the data perceived” is likely going to be an insufficient descriptor of artificial intelligence. Rather, it would be preferable to modify this criterion to reflect more contextually appropriate reasoning and decision-making. At a practical level, everyday experience demonstrates the need for contextually appropriate responses—it is entirely possible for someone to engage in a reasoning process (decision-making based on data presented) but yield results that are highly inappropriate. Having spent several years working in behavioral health, there are plenty of people capable of processing information in ways that do not gel with what we consider “normal” reasoningFootnote 1. Whether we are describing patients with psychiatric conditions or the inferences of young children, the conclusions reached by a reasoning process might not be contextually-appropriate and hence an inappropriate understanding of artificial intelligence. To be fair, contextualization does emerge as a concern in their article but not as clearly or early as might be desired.

The contextualization problem combined with the machine-learning issues suggest that the criterion include some threshold before responding/acting, whether we are discussing actual artificial agent actions (such as those of autonomous vehicles or robots) or those of information processing (such as those of AI agents which simply make recommendations in professional or commercial contexts). Artificial intelligence takes many forms—Farisco, Evers, and Salles rightly note that the issue at hand is not necessarily one of humanity-destroying proportions, but there are clearly deficits when an artificial agent suggests that because one has watched A Bug’s Life (insects working together) that they would enjoy The Human Centipede (body horror). Analyzing my viewing habits is certainly working with data input, but the culmination of that particular inductive reasoning process is clearly insufficiently informed and yields an uncogent conclusion.

Intelligence

Farisco, Evers, and Salles note that there is no real agreement in the definition of intelligence. Paralleling this, attempting to define intelligence is outside both the purview of this paper and the skillset of this author. However, there are some elements of human cognition (and moral cognition) that are better understood and serve as a useful starting point for discussions of artificial intelligence.

Cognitive psychology has made significant inroads in modeling our thought processes. The past several decades of research have suggested that we have processes that are both linear and intuitive, yielding language of “hot and cold” and “System I/II” processing (Gilovich et al. 2002; Kahneman 2011). These processes reflect both our labor-intensive systemic processing as well as our quick intuitive judgments—we employ both routinely when evaluating novel situations. When we have to decide quickly (e.g., when we have to make snap decisions), we employ System I. When we have the luxury of deciding between options (e.g., when we have the ability to spend some time reflecting on our options), we routinely employ System II. Both systems have strengths and weaknesses—our intuitive judgments allow us to make rapid decisions, but they can also be sources of error (e.g., we might be unduly influenced by a particular memory regardless of how appropriate it is for the current situation). Our more labor-intensive decision-making processes don’t necessarily fall into the same traps as our cognitive heuristics but don’t lend themselves to situations requiring faster responses. Cognitive heuristics are not the only elements of our cognition; we also employ emotional valence (whether memories have positive or negative emotional states attached to them), we are vulnerable to framing (how information is presented) and priming effects (how unrelated information presented first shapes our response to subsequent information), and backstage cognition elements like preconscious associations, cultural context, and socialization. All of these elements influence how we make decisions, most are not volitional, and the results can be difficult to predict. As Wallach and Allen note, “humans are hybrid decision-makers with unique approaches to moral choices, honed over time and altered by their own distinctive experiences” (Wallach and Allen 2009, p. 178). Neither our day-to-day nor our ethical judgments are linear thought processes.

There would seem to be advantages to dual-processing models, allowing for both quick and more methodological decision-making processes. Our particular cognitive architecture, however, developed gradually through socialization rather than through programming (Carter 2014; Dunbar & Schultz 2007; Eisenberger 2013; Frith, 2007; Garrod & Pickering 2004; Hari et al.2015; Hurlemann et al. 2010; Saxe 2006). This sensitized us to emotional cues and an ability to engage in moral imagination and perspective taking—tasks that Farisco, Evers, and Salles note currently elude artificial agents. This raises questions as to the extent we would want similar processes in our artificial agents—would we want them to think like us? Should we model the “non-rational” elements of moral decision-making? Our socialization also yielded a cognitive architecture vulnerable to social shaming and isolation, a predisposition to favor perceived ingroups, and a significant aversion to conflicts within the group. This cognitive architecture becomes vulnerable to peer pressures, groupthink, and a host of other group-specific deficits in decision-making (Asch 1951, 1956; Camerer et al. 2004; Baumeister & Leary 1995; Falk & Bassett 2017; Janis 1982). If we elect not to program the socialized and non-rational elements of cognition, we would need to accept a rational model that is foreign to us—an “Other” whose moral experience is artificial and alien to us and whose rational conclusions may differ radically from our own.

Ethics

Moral agency (human or otherwise) defies easy explanation, and an attempt at fully accounting for it is well outside the purview of the present work. However, if we are concerned about the capacity for agency in artificial agents, it is worth highlighting a few facets of our reasoning that may serve as barriers.

At a superficial level, programming agency would seem to be a straightforward task—optimize the utility of actions performed (utilitarianism) while accepting constraints on actions or outcomes that violate these concerns (deontology). Attempting to realize this in an artificial form is much more difficult (Wallach and Allen 2009)—what constitutes “utility” for which population and what are the limits of reasonably affected agents (e.g., who matters and how far into the future are we predicting outcomes)? What limits should need to be placed—an absolute prohibition against killing/harming other moral agents or are these acceptable if they are limited? How do we decide those limits? These are basic questions that raise complicated issues which may defy coding and are but a sample of many other factors we would normally expect ethical agents to weighFootnote 2.

The practical issues that Farisco, Evers, and Salles note are compelling, especially the emotional barriers faced by artificial moral agents. As alluded to above, emotional processing is critical to human moral agency and it isn’t uncommon for a lack of empathy and emotion to draw comparisons to the agent in question being more of a robot instead of a human being. Popular rhetoric aside, there are problems that arise when humans engage in ethical deliberation without emotional valence, compassion, and other agent relative concerns. A purely rational utilitarian calculus quickly yields horrific abuses so long as the majority benefits (e.g., involuntary experimentation for the betterment of mankind, rights abuses justified by raising the overall quality of life or security of the population in question, and so on). Depending on the outcomes we seek to optimize (i.e., what the “good” to be obtained actually is), we can easily see how a purely rational application of utility maximization yields outcomes most agents would find undesirable. If we adhere to a purely rational deontological framework, we can quickly yield results that are implicitly damaging to our relationships (e.g., always telling the unvarnished truth without worrying about compassion or contextual appropriateness). Simply put, purely rational ethical agents could easily yield outcomes that we find undesirable despite the coherence and logic of the reasoning. Efforts to program emotion and emotion-based reasoning raise a host of other concerns including the dilemma about deception noted by Farisco, Evers, and Salles as well as the concerns raised here and elsewhere about biases in the coding.

As the authors note, an apparently unique human attribute that informs this contextualization of ethical methodology is our ability to use inductive and abductive reasoning styles that would not be readily apparent to an artificial agent. This raises a unique dilemma for artificial agents, however. If we recognize that there is a potential deficit in their reasoning, this can be addressed (potentially) by making them think more like human agents. But is this really desirable? Do we want them to think like we do? A significant part of our neurological development can be linked to the process of socialization and “in group” maintenance (Carter 2014; Dunbar & Schultz 2007; Eisenberger 2013; Frith 2007; Garrod & Pickering 2004; Hurlemann et al. 2010; Saxe 2006) with resultant problems. Our brains evaluate scenarios differently in groups rather than in isolation (Hari et al. 2015; Schilbach et al. 2013), they easily fall victim to groupthink and other fallacies, they are not function specific (e.g., regions of emotional processing are also used in social processing), and so on. There is a danger that making them more like us will expose artificial agents to similar deficits and distractions.

At a more concrete level, Farisco, Evers, and Salles note issues that may arise in the context of health- and childcare. These seem to be largely valid, but questions remain. First, while there may be roles for artificial intelligence to play in both of these fields, it isn’t clear why we would consider offsetting direct care and human contact in the first place. While some data indicate a shortage of caregivers in the future (Sharkey & Sharkey 2012), it isn’t clear that artificial caregivers would be an appropriate solution. Pew surveys find Americans have a mix of worry and willingness about receiving care from artificial agents (Smith & Anderson 2017) while data from Japan and other countries suggest a significantly greater willingness to receive care while simultaneously indicating that they would not feel pressure to form a bond as they would with human care (Broadbent et al. 2011; Jiji 2018). Interpersonal connection is critical, and hallmarks of both fields include empathy and compassion, both of which defy current programming. If we were able to simulate them, it is not clear that patients or children would be able to form bonds with them (Sharkey & Sharkey 2011)Footnote 3. If the artificial caregiver is not perceived as a moral agent capable of forming genuine bonds, it stands to reason that there would be difficulties with trusting them in the course of careFootnote 4. It is likely that artificial agents may be more trusted in environments allowing for information analysis and synthesis rather than the direct care of vulnerable populations and other circumstances as artificial intelligence is already employed in this capacity elsewhere (Whitby 2008).

Farisco, Evers, and Salles present additional theoretical ethical concerns, and two of these deserve additional attention. First, they briefly raise the possibility of machines developing their own capacity for moral/ethical reasoning but suggest it is unlikely. Second, they raise concerns about human agents experiencing an ontological crisis concerning our perceived unique ability to engage in moral reasoning. Both of these suggest an existential crisis emerging from artificial agents becoming ‘too much like us’—i.e., bridging the ontological divide between humanity and technology.

First, it isn’t clear why the development of unique moral/ethical capacities is unlikely. If we accept this premise and still seek to create ethical artificial agents, we will encounter the difficulties noted above about coding artificial agents. If we reject this premise, it is not difficult to imagine circumstances that would allow artificial agents to develop their own moral or ethical principles and methodologies (albeit not in a form familiar to us; they would not necessarily generate anthropocentric moral or ethical principles). This suggestion is entirely compatible with observable phenomena of artificial agents—novel language development, machine learning, etc. all suggest that machines would be able to create their own optimization schemas (Lewis et al. 2017). These can be expressed in terms of both rules to follow as well as moral principles (any initial preprogrammed goal that allows for adaptation by the artificial agent or what the artificial agent itself chooses to be optimized). This does, however, yield the possibility of the nightmare scenarios envisioned in popular culture where humanity may simply be another variable to consider rather than an end in itself. Discussions of any particular schemas or frameworks here is beyond the scope of the argument, but there does seem to be some consistent mechanism by which artificial agents could solve their own optimization problems.

The second issue is also compelling—the ontological crisis of the non-unique problem of moral reasoning. As with the originating moral principles problem, it isn’t clear that this ontological crisis is guaranteed or would be severe. There are already other parallels in non-human animal agency—we see behaviors similar to our own in other species and do not seem to begrudge them tool usage, complex language, emotional lives, basic economies, and so on. Many of the behaviors that seemed to be “unique” to us are clearly shared. It isn’t clear that moral reasoning should be much different. Popular culture has been addressing this phenomenon for decades, creating both sympathetic and terrifying characters (from Data on Star Trek to the morally complex and ideologically divided Cylons of Battlestar Galactica and “hosts” of Westworld). Some of these portrayals explicitly attempt to blur the line between biological and synthetic, creating characters with the same apparent will and volition as human agents and synthetic biology nearly identical to our own. These apparently functionalist portrayals suggest a willingness to consider the issue while recognizing that their resultant moral systems may be at odds with our own. Admittedly, this is not a strong argument—willingness to engage with science fiction does not necessarily translate into a willingness to engage with a concrete challenge. However, science fiction has introduced many elements of technology taken for granted now—this at least raises the possibility that a parallel process may occur when it comes to a synthetic moral agent.

Conclusion

Ultimately, the caution suggested by Farisco, Evers, and Salles is reasonable—it is quite easy to reach both dystopian and idyllic conclusions about the future of artificial intelligence absent serious exploration of both the conceptual and practical issues they raise. Some of the concerns they note seem to be warranted, others require additional justification. The degree to which we are integrating artificial intelligence into our daily lives suggests that some of their ontological concerns warrant some skepticism. If we intentionally avoid some of the known pitfalls in our cognitive architecture, we cannot help but create moral agents that are dissimilar from us. Knowing that they will be different from us from the start could offset some of the existential uncertainty—if anything, the experience of an alien “other” in moral agency might serve to overcome some of the out-group biases we experience currently towards our fellow human agents.