1 Introduction

Face recognition technology (FRT) is a type of artificial intelligence (AI) that has become increasingly utilized in modern liberal democracies (Bjerring & Busch, 2021; Crawford et al., 2016; Helbing et al., 2017; OECD, 2019; West & Allen, 2018). This paper is on the moral hazards of a type of facial analysis software, which the author calls facial profiling technology (FPT), that is designed to predict personality traits. At the time of this writing, oversight organizations (mostly in the US and Europe) have only considered the ethical use of FRT (The United States Department of Justice, 2020; Galbally et al., 2019; FRA, 2018; FRA Focus, 2019; ICO, 2019; Lewis & Crumpler, 2021). FPT programs, which are currently in use by the United States (U.S.) government, have not been analyzed by any policy or oversight organization. Moreover, this is the first paper to philosophically examine the anti-liberal implications of its use and to develop regulatory safeguards for its use in liberal democracies.

This paper proceeds as follows: First, a brief overview of FRT and FPT will emphasize how these related technologies reinvigorate the same concerns associated with the pseudoscience of physiognomy. Then, ‘A Liberal Argument Against Facial Discrimination’ (LAAFD) is presented and defended. In sections IV and V, the author makes the case that government use of FPT risks perpetuating wrongful discrimination that is incompatible with liberal values by proposing a possible world scenario in which FPT is used by government. The author concludes by developing a set of prima facie FPT policies for liberal democracies to adopt for protecting the central tenets of classical liberalism if FPT is to be used by government.

2 Faception: a case study

The use of general FRT has received an abundance of attention in legal, philosophical, and public-media spheres in recent years (Brey, 2004; de Laat, 2018; Hale, 2005; Selinger & Hartzog, 2020a; 2020b; Ringrose & Ramiee, 2020). The author of this paper has previously published the first philosophical analysis of the use of racially and gender biased FRT by governments in liberal democracies and has argued against its use until such biases are ameliorated (Gentzel, 2021). However, at the time of this writing, only FRT has been addressed in the academic literature. While general FRT uses AI algorithms to detect facial patterns, which can then match a face to photographs of other faces within a large database (Garvie et al., 2016), some facial analysis programs contain more advanced algorithms designed to find subtle facial changes that are correlated with specific emotions and lying (Bittle, 2020; Rhue, 2018; Rhue, 2019). FPT (facial profiling technology) is a type of facial analysis that is alleged to be able to predict specific personality or character traits of an individual by using facial analysis artificial intelligence that focuses on facial structure.

At the time of this writing, Israeli technology company, Faception, is the most prominent FPT program. According to the Washington Post, Faception claims that its program can, ‘…take one look at a person’s face and realize character traits that are undetectable to the human eye. Faception said it’s already signed a contract with a homeland security agency to help identify terrorists. The company said its technology also can be used to identify everything from great poker players to extroverts, pedophiles, geniuses and white collar-criminals.’ (McFarland, 2016). The company website describes its own technology as follows: ‘Faception can analyze faces from video streams (recorded and live), cameras, or online/offline databases, encode the faces in proprietary image descriptors and match an individual with various personality traits and types with a high level of accuracy. We develop proprietary classifiers, each describing a certain personality type or trait such as an Extrovert, a person with High IQ, Professional Poker Player or a Terrorist. Ultimately, we can score facial images on a set of classifiers and provide our clients with a better understanding of their customers, the people in front of them or in front of their cameras’ (Faception, 2022). Faception software has been trained on thousands of images of examples, known as datasets. According to the chief technology officer, the software was able to classify 9 of the 11 terrorists who committed the 2016 Paris attacks. Additionally, Faception was able to identify 25 out of 27 poker players in a database of images (Adee, 2016).

Faception’s website features a detailed advertisement and instruction manual explaining the indicated use of Faception for law enforcement agencies (https://www.faception.com/hls-and-public-safety). According to Faception, its AI can be wielded by all levels of law enforcement (federal, state, local, border security, etc.) to ‘detect and apprehend potential offenders or criminals before they have the opportunity to do harm’. Faception explicitly distinguishes its technology from traditional face recognition technology that matches the faces of known criminals from a database. The company also states that this software is superior to other traditional forms of criminal profiling, such as behavior detection, fingerprinting, and mass surveillance. As an alternative, Faception states that the main use of its FPT is for police to scan large groups of people, detect certain people that Faception identifies as matching a certain classifier (as being aggressive; a potential robber; a potential terrorist; a potential murderer; a potential pedophile), and then police can detain those individuals whose faces match the relevant classifier for interrogation and investigation. Faception provides case studies to illustrate the intended use of its facial profiling technology. The following is an example: “Murder Case: someone was murdered in a place/event. The Police has the images of all attendees during the crime event. Now the Police will search for suspects that Faception shows high level of potential to be murders. The Police intelligence will get a list of images of suspects with high score [means potential to be suspected as Murders].” (https://www.faception.com/hls-and-public-safety).

While Faception claims its FPT can be used in conjunction with databases of known criminal offenders, its main function is to pick out, at random, faces of individuals that match a particular classifier so that police can detain and investigate before that individual commits a crime. Faception argues that its facial profiling is better than traditional FRT, behavior detection, finger printing, and surveillance, and suggests that law enforcement replace traditional crime prevention and detection strategies with the use of its proprietary AI technology.

One immediate critique of the use of FPT programs is that it rekindles concerns about the use of the old tradition of physiognomy. Physiognomy, simply defined, is ‘…the art of reading personality traits from faces.’ (Hassin & Trope, 2000). Writing for the American Psychological Association Handbook on Nonverbal Communication, Daniel Re and Nicolas Rule provide a thorough history of physiognomy, from ancient to modern times. In ancient times, Aristotle had suggested that individuals with structures similar to particular animals would share in that animal’s temperament (Re … Rule, 2016). In contemporary psychological literature, Re and Rule note that there exists an extensive discussion about whether and how facial features can correlate with personality traits. The authors provide a literature review that identifies various possible mechanisms by which facial features and personality could be connected; including biological causes (i.e.: testosterone effects both facial morphology and personality traits like dominance) and environmental causes (i.e.: external environmental influences can lead to early puberty and influence hormones, which could affect both facial features and personality). Despite such connections between facial features and personality, the authors are careful to emphasize that these correlations are ‘minor to moderate’, and overgeneralization ought to be avoided (Re … Rule, 2016).

Notwithstanding the recent psychological research on the connection between personality and the face, the history of physiognomy as it was once practiced has a sinister tradition. Writing for the New York Times Opinion section in 2019, Sahil Chinoy recounts the sordid history behind the 19th century attempts to link particular facial features to aspects of their owner’s character. Francis Galton, a prominent advocate for eugenics in the 19th century, made unsuccessful attempts to find similarities among men convicted of crimes. In a similar vein, 19th century Italian physiognomist Cesare Lombroso argued in favor of using facial and body measurements to separate so-called ‘intellectually inferior’ children from their ‘normal’ counterparts. Chinoy concludes that modern AI algorithms that purport to predict personality from facial features are both inaccurate and run the risk of perpetuating and disguising harmful human prejudices (Chinoy, 2019). Writing for Medium.com, Averyl Dietering recounts the racist intentions behind many of the traditional physiognomists from European backgrounds (Dietering, 2019). Black Ethiopian features, for example, were associated with those individuals being fornicators, undisciplined, and malicious. This inaccurate and prejudicial use of physiognomy should provide a cautionary tale regarding modern FPT programs that purport to be able to accurately perform those functions that the old pseudoscience failed at achieving.

3 A liberal argument against facial discrimination (LAAFD)

The previous section established that FPT is currently being used by the United States government. In response to this fact, the author of this manuscript develops and defends the following argument:

3.1 Liberal argument against facial discrimination (LAAFD)

  1. 1)

    Equal treatment before the law is an essential tenet of classical liberalism.

  2. 2)

    Unjustified discrimination practiced by the government is incompatible with equal treatment before the law.

  3. 3)

    Unjustified discrimination practiced by the government is incompatible with classical liberalism. (From 1 to 2)

  4. 4)

    Government use of FPT could lead to government practicing unjustified discrimination.

  5. 5)

    Therefore, government use of FPT could be incompatible with treating all individuals equally before the law. (From 2, 3 and 4)

  6. 6)

    Therefore, government use of FPT could be incompatible with classical liberalism. (From 3, 4 and 5)

The LAAFD is formally valid, and the bulk of this paper establishes the truth of each premise, with particular focus on premise four.

The first three premises state that unjust discrimination is incompatible with both classical liberalism and equality before the law. Plenty of important work has been done on the relationship between equality and classical liberalism (Vincent, 2009; Mill, 1859; Gaus, 2014; Hayek, 1960; Locke, 1689; Arneson, 1999). Contemporary philosopher Thommie Shelby sums up this important relationship thusly: ‘It is a central if not defining tenet of liberalism that all persons are to be regarded as free and equal in a just society’ (Shelby, 2004). Currently, most modern liberal democracies codify this idea of equal treatment under the law (14th Amendment of the US Constitution; United Nations, 1948; CFR, 2000;). As such, an extended defense of these premises is beyond the scope of this paper. Therefore, it will be assumed for the sake of argument that classical liberalism entails equal treatment before the law.

Much like the relationship between equality and the classical liberal tradition, the notion that unjust discrimination is incompatible with political equality needs no elaborate defense here, and therefore can also be assumed for the sake of argument. That said, some important remarks are worth mentioning in service to the main argument of this manuscript. To discriminate between two groups or two individuals is to treat one group or one individual differently from another group or individual. According to Andrew Altman, there are more than six different theories of wrongful discrimination (Altman, 2020). The two theories relevant to this paper have been articulated by Joel Feinberg and Deborah Hellman.

According to Feinberg (who echoes Aristotle), two individuals who are equal in all relevant respects deserve (morally) to be treated equally. To treat one group or individual differently from another group or individual when no morally relevant difference exists would be arbitrary, and therefore, unjust. Conversely, when the basis upon which discrimination is practiced is due to a morally relevant difference between two groups or individuals, then that discrimination can be said to be non-arbitrary, and therefore, justified (Feinberg, 1973). James Rachels provided an apt example: An employer who discriminates against blacks and Jews when race and religion are irrelevant discriminates arbitrarily because the properties of being black or Jewish are not causally relevant to the ability to perform a job. Conversely, an employer who discriminates against the blind, when visual acuity is an essential component for job performance, has a morally relevant reason to discriminate against the blind, because blindness is a disqualifying causal property. Therefore, such discrimination is not arbitrary, and is therefore justified (Rachels, 2004).

Deborah Hellman’s theory of unjust discrimination assumes a liberal context and builds upon the arbitrary/non-arbitrary distinction by adding the demeaning condition (though she maintains that not all forms of arbitrary discrimination are wrongful), which has the benefit of dealing with more difficult cases. Accordingly, this is why ‘Separating students by last name feels quite different than separating students by race…though each can be done for good or bad reasons and each may be related or unrelated to some legitimate purpose’ (Hellman, 2008, p. 7).

Hellman’s more sophisticated theory of discrimination posits that discrimination is wrong when it fails to recognize the equal moral worth of all persons by demeaning the personhood of individuals or groups, especially those who were historically disadvantaged within the cultural context. Discrimination that demeans contains two components: it expresses a disregard for the equal moral worth of persons while also causing actions that are effective at subordinating or oppressing. For A to wrongfully discriminate against B, A must be in a social position relative to B whereby A’s social status is higher than B’s and A’s actions effectively demean B by subordinating or degrading B. Historical context can play a role in determining whether A demeans B, as can relative social power dynamics between A and B. To illustrate the latter, an employee who spits at her boss simply expresses disrespect and does not demean her boss, whereas a pedestrian who spits at a homeless man demeans him. To illustrate the former, a policy enacted and enforced by a health insurance company that picks out battered women as a group and charges them higher premiums based on actuarial calculations that imply that they cost the company more money in healthcare expenses demeans that group of women because women have been historically disadvantaged in the broader economy and the insurance companies occupy a relative social status higher than the women who are either denied coverage or charged higher premiums (Hellman, 2008).

More concretely, Hellman outlines the conditions surrounding a policy or regulation that ought to be considered to determine if that policy or regulation demeans a particular group. Hellman points out that one should carefully consider the ‘…the actor, the content of the regulation, and the context’ (Hellman2008, p. 64). For example, if the actor represents a powerful authority (either political or economic), then that increases the likelihood that the policy demeans. In terms of the content of the policy, it could range from mere recommendations to outright denials of service, or in the case of the criminal law, criminal punishment. And finally, the context in which the policy is being implemented can increase the likelihood of demeaning discrimination. An American federal agency that picks out Native Americans or Black Americans as distinct groups to be treated differently from other Americans presents a higher risk of demeaning those groups because of the longstanding historical injustice those groups experienced at the hands of the American federal government. Hellman admits that while many cases will turn out to be obvious instances of policies that demean groups and therefore wrongfully discriminate, there is also likely to be honest disagreement on particularly difficult cases. One thing to note is that the question of whether policy X demeans group Y should be viewed as an objective judgment so that how particular individuals feel about a policy does not on its own influence whether a policy demeans. While the recognition of this point is important, the details are not crucial to the application of Hellman’s theory to the case of FPT used by government. Later sections of this paper show that while FPT used by the government may not necessarily result in unjust discrimination, its widespread use could seriously risk violating both the traditional arbitrary/non-arbitrary distinction and Hellman’s demeaning conditions for when discrimination is unjust.

To be sure, liberal democracies have at times throughout history discriminated against groups in ways that have demeaned them and in ways that violated the liberal value of equality before the law. One historical example is the Chinese Exclusion Act of 1882. Signed by United States president Chester A. Arthur, it placed a 10-year ban on Chinese manual laborers from entering the United States (National Archives). Pre-existing racial animus against U.S. residents of Asian descent became increasingly commonplace, and anti-Asian sentiments were commercialized and expressed openly in popular culture.Footnote 1 In the years following the passage of the Chinese Exclusion Act two infamous massacres perpetrated by white townspeople against Chinese immigrants–the Rock Springs massacre of 1885 and the Hells Canyon massacre of 1887– revealed the tragic results of racial tensions intensified by government fiat. Both massacres combined resulted in the murder and dismemberment of no less than 60 innocent Chinese residents.Footnote 2 This horrific example illustrates how government approved discrimination against a group emboldens portions of the citizenry to act in atrocious ways with impunity. Although many other similar historical examples exist, all current and future laws and practices in liberal democracies that involve the use of new technology like Faception should explicitly recognize the equal moral worth of groups and individuals and protect the equal treatment of all individuals before the law.Footnote 3

To sum up: discrimination is unjust when it is arbitrary or in the absence of a morally relevant reason. Additionally, discrimination that fails to treat individuals or groups as moral equals by demeaning those individuals or groups, especially those who have been historically disadvantaged, is likewise unjust. Unjust government discrimination can lead to horrific societal outcomes, as demonstrated by the Chinese Exclusion Act. Therefore, the first, second, and third premises of the LAAFD are true.

3.2 The fourth premise: government use of FPT could lead to government practicing unjustified discrimination

The next two sections defend the fourth premise, which states that the government’s use of FPT (namely, Faception) risks leading to government practicing unjustified discrimination. In this section, it will be argued that government discrimination based on facial features could lead to the same widespread injustice that government discrimination based on race would cause. The key move in the argument is to show how government use of Faception could give rise to a context in which government categorizes and segregates specific groups of people from the rest of the population, resulting in treating them differently in the absence of a morally relevant reason. Moreover, this segregation could also fulfill Hellman’s condition of discrimination that demeans a particular group, some of whom may already face historical disadvantage. A thought experiment utilizing two possible worlds will be presented to illustrate how discrimination and segregation based on facial-profiling technology could be as demeaning and unjust as discrimination and segregation based on race. Both types of discrimination could lead to tragic consequences, the extreme of which could parity the Chinese Exclusion Act and its consequences. While the unjust discrimination described in this section could take place in the future, it is not destined to happen with Faception. The purpose of this paper, and the overall argument, is to identify the risks that come with the use of FPT in western liberal societies, so that these risks can be mitigated through policies and best practices, which the author develops in sections IV and V of this paper.

Recall from section II that Faception advertised its facial profiling software to government law enforcement agencies to identify individuals whose faces match a specific classifier so that government agents could focus on those individuals as potential threats. The case studies that Faception proposes to illustrate how the technology is intended to be utilized by law enforcement is clear: Faception is used to analyze the faces of all individuals present at the scene of a murder (from, say, security camera footage). It should be used to identify a list of suspects based on the facial features of individuals that Faception matches as potential murderers. Police would then be able to focus on those individuals whom the program indicates have the facial features of a murderer. Faception has a classifier called ‘murderer’, which separates faces based on whether a particular face matches that classifier or not. To illustrate how government-initiated discrimination based on facial features could be pernicious and unjust in the same way that government-initiated discrimination based on race would be pernicious and unjust, consider two possible-world scenarios.

3.2.1 World 1 (face-based discrimination)

For the purposes of this discussion, call a person whose face matches Faception’s classifier for murderer, a ‘murder-face’. Corresponding classifiers would identify those with ‘pedophile-face’, ‘terrorist-face’, and so on. Now imagine a possible world, call it World 1, in which the government utilizes this face-classifier software as a threat management and crime-prevention tool. In such a world, categories of people would begin to form ‘in the eyes’ of government and law enforcement. Those identified as having murder-face based on Faception’s classifiers would be picked out and targeted for surveillance before such individuals have committed any crime. Such categorization would be done in the name of public safety and the collective good. The same would hold for individuals whose facial features match the FPT’s program classifiers for ‘pedophile’, ‘thief’, ‘drug-dealer’, and ‘terrorist’. Those with pedophile-face, so picked out and labeled by an FPT program, would be surveilled, and treated differently, by law enforcement before any crime were committed by such individuals. Perhaps such facial features are identified by an FPT at a young age, and every young person bearing a murder-face, pedophile-face, etc. would be carefully watched and scrutinized beginning in childhood and continuing for life. They could be ‘marked’ for life. For society to reduce crime before it happens (Faception calls it ‘anonymous threat detection’), individuals with faces that correspond to the criminal facial classifiers could also be added to government registries or ‘lists’ before they commit a crime, solely on the basis of their facial features. These lists could either be overtly advertised or covertly maintained. Society at large could come to view those with murder-face and other face-classified features as pariahs to be avoided, stigmatized, and excluded from polite society. In the most extreme version of this scenario, and as was seen in the example of the Chinese Exclusion Act of 1882, extreme animus and violence against individuals with murder-face and pedophile-face could become a normal and accepted form of protecting societal interests in safety.

3.2.2 World 2 (race-based discrimination)

To demonstrate why such discrimination practiced by a liberal democratic government would be unjustified, it would be illuminating to substitute the term ‘face’ in the description of World 1 with the term ‘race’, and it becomes obvious how a government program of this type of discrimination would be outrageous and incompatible with the liberal value of equality before the law. Imagine a possible world (World 2) in which a liberal government identifies people by racial traits. Instead of separating people according to face classifiers, certain groups of individuals are identified as being black, white, Asian, or Hispanic and so-on. Those individuals who possess characteristics of certain racial groups (like the face classifiers in World 1) that have been deemed statistically more likely to commit criminal acts (murder, theft, terrorism, etc.) are picked out and targeted for surveillance before such individuals have committed any crime. Such categorization would be made in the name of public safety and the collective good. Suppose in this possible world, people classified as black are the ones picked out and surveilled as being more likely to commit murder. Perhaps black people are identified at a young age, and everyone bearing traits of being black would be carefully watched and scrutinized from childhood. Such individuals would be ‘marked’ for life. For society to reduce crime before it happens (i.e.: to practice anonymous threat detection), black individuals could be added to government registries or lists before they commit a crime, solely on the basis of their race (being black). Society at large could come to view those classified as being black as pariahs to be avoided, stigmatized, and excluded from polite society. In the most extreme version of this scenario, and as was seen in the example of the Chinese Exclusion Act of 1882, extreme animus and violence against black individuals could become a normal and accepted form of protecting societal interests in safety.

World 1 and World 2 both involve government discrimination based on an immutable trait (either belonging to a racial group or having a certain face-type) the possession of which has been statistically linked with some dangerous behavior (being a criminal). Recall from the first part of Section III that classical liberalism requires that government treat individuals equally before the law, and that discrimination between groups or individuals is unjust when it is arbitrary to individuals or groups. This condition is present in both World 1 and 2. In both World 1 and 2, statistical discrimination is arbitrary and unjust because the class membership in each case is only correlated with the dangerous behavior. The property of being a member of a particular class in both cases (of having murder-face and of being black) does not necessarily contain the property of being a criminal (or being a murderer, a pedophile, a thief, or a terrorist), even though there may be a statistical correlation between class-membership and dangerous behavior, because class-membership is not the relevant causal factor that explains that dangerous behavior. This is why it is unjustified to treat individuals differently based on harmful stereotypes about group membership. Being black and being a criminal are not causally linked, just as being Irish is not causally linked with driving while drunk and being East Asian is not causally linked with having strong mathematical abilities. Likewise, there are plenty of non-black individuals who engage criminal behavior, many non-Irish motorists who drive while drunk, and many non-East Asians who possess strong mathematical abilities. The causal factors behind certain behaviors are complex and multifactorial and go far beyond mere attributions to group membership. The same analysis applies to World 1 where statistical discrimination links face-types with criminality. Even though Faception claims that the possession of a certain set of facial features that matches a corresponding classifier is statistically linked with a corresponding dangerous behavior, merely possessing a face that matches a classifier is not the casual factor for murder, theft, pedophilia, and terrorism. One’s face does not cause one to murder, just as one’s blackness, or Irishness, does not cause one to commit crimes or to drive drunk, even though statistical correlations may (or may not) exist.

Joel Feinberg aptly explains this crucial technique of identifying a morally relevant factor that justifies discrimination as follows: ‘The correlation between statistical class membership and a specified type of behavior…does not connect that behavior to any causally relevant factor operating in each member of the class. That a given person is a member of the statistically dangerous class is a ground for suspecting that he might have a property that is causally connected with danger, but the class membership itself is not that property’ (Feinberg, 1984, pg. 201). According to this analysis, even if it were the case in World 2 that being a member of a certain race is statistically correlated with criminality, discriminating against an individual solely based on that class membership would be arbitrary (and hence, unjust) because being black is not a causally relevant factor for believing that that individual is a criminal. Similarly, in World 1, being a member of a certain class of face-types (having a murder-face, for instance) might be statistically correlated with criminality, but having a certain face-type is not a property that causes one to murder. In that case, for the government to discriminate against individuals would also be arbitrary and therefore unjust.

Thus far in the analysis, it has been shown that FPT used by government in World 1 to categorize certain groups of people based on facial structure as being criminally inclined is unjust and incompatible with liberal values because it is arbitrary in the same way that categorizing certain groups based on racial grounds is arbitrary, unjust, and incompatible with liberal values in World 2. The next section will consider an immediate objection to this argument. Part of the reply to this objection will invoke Deborah Hellman’s theory of wrongful discrimination as demeaning, which will provide reason to believe that facial discrimination in World 1 could also be demeaning to groups with particular facial structures, and therefore, would be unjustly discriminatory, in the same way that racial discrimination in World 2 is demeaning, and therefore, unjustly discriminatory.

3.3 The genetic link objection and reply

There is an immediate objection to the argument in the preceding section. Call this the Genetic Link Objection, and it goes as follows: There is a difference between statistical correlations and the subsequent discrimination which occurs in World 2 than that which appears in World (1) One can concede that discrimination based on the statistical correlations between race and criminality are arbitrary in the sense that class membership is not the causal property that explains the dangerous behavior. However, discrimination in World 1, which is based on a statistical correlation between face-type and criminality, is not arbitrary, and therefore justified, because this statistical discrimination points to the causally relevant factor that connects face-type with dangerous behavior: genetics. Indeed, (so the argument goes) Faception claims on its own website that the facial classifiers are based on the DNA (genes) of individuals, because ‘our face is a reflection of our DNA’ (Faception, 2022). As such, Faception’s classifiers are based on the genetic profiles of the people whose faces match each classifier. An individual singled out as having a murder-face in World 1 is not singled out due to a mere correlation between class membership and their corresponding behaviors like in World (2) Instead, murder-face is a class membership that picks out a certain genotype, and that genotype is the causally relevant factor that explains murderous behavior. Consequently, individuals discriminated against by the government in World 1 based on facial traits are singled out because of the causally relevant factor of how their genotype gives rise to their criminality. Therefore, race-based discrimination is arbitrary and unjust in World 2 while face-based discrimination is not arbitrary, and therefore, justified, in World 1. Premise four of the Liberal Argument Against Facial Discrimination is false, and the argument is unsound.

There are two main counterarguments to the Genetic Link Objection. First: Faception’s current evidence used to support the claim about one’s face reflecting one’s DNA must be called into question. The second counterargument contains three main steps: Even if one’s facial structures are influenced by one’s genes, there are significant uncertainties about the extent to which a specific genotype causes socially complex behaviors like murder, pedophilia, theft, and terrorism, and how those behaviors relate to specific facial structures. This uncertainty would still render such discrimination arbitrary. Moreover, even if face-structure were a causally relevant factor for increased risk for criminality, such discrimination could turn out to be demeaning, thus fulfilling Hellman’s theory of wrongful discrimination.

The first counterargument to the Genetic Link Objection draws attention to Faception’s inadequate explanation of the technology. Focusing on Faception’s advertising information is important because it is the only company at the time of this manuscript that markets facial profiling technology and is under contract with the Unites States federal government. Faception’s website features a section called ‘Theory Behind the Technology” (https://www.faception.com/about-us). The main assumption is that personality is determined by genes and that facial features reflect our genes. Faception argues that their technology can identify individuals with faces of murderers, terrorists, pedophiles, and thieves because the software is identifying genes that determine these behaviors. But one problem is that Faception’s information is not well documented. For example, Faception mentions, (but does not cite) only one study to support the claim that genes give rise to our personality (Archontaki et al., 2013). The author of this manuscript searched the details mentioned on the website and was able to find the relevant study. In that study, psychologists at the University of Edinburg studied 837 twin pairs to determine the extent to which genetic influences determine psychological well-being in adults. While the tools used to measure well-being included personality traits like autonomy and positive relations with others, the study itself did not address the important traits of Faception’s classifiers, like murder, theft, pedophilia, and terrorism. Although the study seems to suggest that a strong genetic component may be involved in determining a person’s sense of psychological well-being (the authors refer to the ancient Greek term, Eudaimonia), the extent to which specific genes can predict and distinguish between complex forms of criminality like murder and pedophilia was not established by this study, and therefore, has not been established by Faception. Given the high social, economic, and civil rights stakes described in World 1 that depend on the accuracy of this software much more evidence for the causal claim connecting genes and personality needs to be provided by Faception to justify its use by law enforcement in liberal democracies.

Moreover, Faception offers weak evidence to support the claim that a person’s face is a reliable predictor of genotypes that predict criminal behavior. Especially worrisome is Faception’s reliance on mouse model research and ancient physiognomy. Direct quotes from the website include: “Working on mice, researchers have identified thousands of small regions of DNA that influence the way facial features develop. The researchers said that although the work was carried out on animals, the human face was likely to develop in the same way”, and, “… in Chinese history, there have been people that have studied the ‘mapping of the face’ for thousands of years.”(https://www.faception.com/about-us).The first problem is that this technology is based on research from mice, the conclusions of which do not necessarily apply to human beings. Given the high social stakes associated with the use of this technology, human studies are needed. Even more problematic than the reliance on animal models is the implicit legitimatization of ancient Chinese physiognomy. Section II of this manuscript provided the sinister, racially charged, and pseudoscientific foundations of the application and theory of physiognomy. It should be clear from that section that while ancient Chinese physiognomy may or may not have been falsified, ancient pre-scientific speculations about mapping faces should not be utilized as a reason to think that a person’s face reflects a person’s DNA, because many of these traditions were steeped in prejudice and bigotry.

Even if Faception were able to provide more robust evidence, that would not be dispositive evidence to vindicate its use by a liberal democracy. Even if a person’s face does reflect her genes, it does not follow that any resulting government discrimination is automatically justified. It could still be the case that the relevant genes picked out by Faception’s classifiers are not the causally relevant factors connecting an individual’s class membership to the dangerous behavior. The causal factors underlying complex dangerous behaviors like murder and pedophilia could include a plethora of influential causal factors in addition to or independent from a person’s genes as expressed as facial features. Returning to the research cited in section II by Daniel Re and Nicolas Rule on correlations between face and personality, there is considerable contemporary debate about correlations between types of faces and personality types. According to Re and Rule, there are four possible causal relationships between face and personality.

First, perhaps the face and personality share a common biological cause. To back this possibility, the authors consider the literature that connects the presence of testosterone with a dominant personality and a corresponding face-type. Testosterone guides prenatal brain development (Chowenbreed et al., 1989), enhances aggression (Archer, 1991; Mattsson et al., 1980; Mazur & Booth, 1998; Olweus et al., 1980, 1988), and affects facial development (Penton-Voak & Chen, 2004; Roney et al., 2006). Second, it is possible that an unstable environment during childhood could elevate risk of early onset puberty. Environmental causes include violence in the surrounding neighborhood, divorce, or parental conflict (Belsky et al., 1991; Wierson et al., 1993). As a result, Re and Rule suggest that puberty related testosterone levels being elevated at a younger could lead to a certain type of facial changes occurring at a younger age. The environmental conflict experienced at a young age could simultaneously and independently lead to a more aggressive personality. Third, Re and Rule consider the possibility that facial appearance could shape how a person develops her own personality. Maybe people who happen to be born with more dominant faces are more likely to be treated by others with more submissive or fear-laden reactions. As a result, people with dominant faces ‘learn’, through the reactions from others, to be more dominant and aggressive. It should be noted that there does not seem to be any direct evidence to back this possibility. Finally, it is possible that a person’s personality could shape facial structure. The idea here is that an individual with a dominant or aggressive personality to begin with would be more likely to make angry facial expressions and maintain those facial expressions for longer periods compared to others not disposed to aggression. An example of this phenomenon includes the facial expression of eyebrow-furrowing and its connection with perceived dominance (Keating, 1985; Keating et al., 1981; Zebrowitz & Lee, 1999). Perhaps dominant and aggressive people engage in prolonged furrowing of the eye-brows, which gives rise to a face that exudes dominance and aggression to others.

These four possible relationships between facial structure and personality paint an ambiguous picture. The extent to which a person’s face reflects their DNA, as Faception claims, is substantially underdetermined at the time of this manuscript. Re and Rule suspect that all four possible relationships between face and personality could be relevant, with no single relationship being causally exclusive. It is also possible that one or more of these relationships could be irrelevant. The upshot on the literature from Re and Rule is that any correlations between facial structure and personality are ‘minor to moderate’, and overgeneralization ought to be avoided (Re & Rule, 2016). The authors provide a strong warning against overgeneralizations, which could lead to unfair and ‘deleterious’ social implications. For example, research suggests that people with faces perceived as attractive receive more lenient prison sentences than do people with faces perceived as less attractive who commit the same crimes (Stewart, 1980). Here would be a crucial example of arbitrary and unjust discrimination committed by the government. In this case, it would be unjust because a person’s face is not a causally relevant property related to the crime being committed.

Suppose the science behind Faception turns out to be robust and that face structure is a causally relevant factor that influences criminality. Even so, the use of Faception could still threaten classical liberal principles. The discrimination described in World 1 could fulfill Deborah Hellman’s theory of wrongful discrimination, possibly resulting in society-wide discrimination that demeans the moral worth of those with certain facial features and would therefore be incompatible with liberal values. Recall from section III the conditions Hellman proposes under which A wrongfully discriminates against B. A wrongfully discriminates against B when A demeans B by failing to recognize the equal moral worth of B and when A occupies a position of power over B. This asymmetry between A and B often takes the form of social, political, or economic differences between A and B, where A is ‘higher’ than B, and / or it can also take the form of a historical context in which B represents a group that has been culturally, politically, economically, or otherwise oppressed, underrepresented, or denied equal consideration. Hellman’s criteria further include the consideration of a policy’s actor, content, and context as key determinants on the likelihood that a policy demeans a particular group. The discrimination in World 1 based on facial structure could fulfill all these criteria of demeaning discrimination. In World 1, Faception’s facial classifiers would pick out certain groups as being criminally inclined based on facial structure. Labeling a group as criminally inclined automatically invokes beliefs of moral inferiority. More specifically, the history of physiognomy recounted in section II established the historical pseudoscientific attempt to associate facial structures with supposed moral and racial inferiority of certain groups of people. Faception would rekindle this degrading paradigm while offering a false illusion of scientific legitimacy. This consideration is significant in determining how FPT use in World 1 would demean (by subordinating and degrading) groups with certain facial structures as being morally defective and racially inferior, as it would be a regressive return to preemptively labeling certain people as ‘less than’ and inferior, and thus worthy of unequal consideration in political and economic life. Moreover, there is a vast political asymmetry between the actor and those affected: the federal government using Faception versus those groups being picked out based on facial structure. The government has a monopoly on force and has the power to take the freedom, property, and lives of individuals, as well as the power to sway public opinion, as was seen in the example of the Chinese Exclusion Act. To make matters worse, government policies under the auspices of physiognomy were historically used to degrade and deny opportunities to certain ethnic groups. The content of the policy would not be a mere recommendation in World 1. It would instantiate government surveillance of certain groups before crimes are committed, along with the potential deprivations of political and economic freedom without due process. Therefore, the discrimination described in World 1 could fulfill all of Hellman’s criteria for when discrimination is demeaning, and therefore, would be wrong and incompatible with liberal values.

To sum up: There are multiple reasons to reject the Genetic Link Objection that claims that a person’s face and genes causally relate to violent or criminal personalities. The first counterargument pointed out that Faception’s evidence to support such a claim is surprisingly weak and relies on pseudoscientific appeals to physiognomy. The second counterargument contained two main steps. First, it was shown that even if facial structure and personality were causally linked, the extent to which such a link is determined by genes, environment, or an interaction of both, is scientifically underdetermined. Therefore, discrimination based on Faception classifiers would still be arbitrary and unjust in World 1. Second, it was shown that even if a solid scientific link between genes, facial structures, and criminality could be demonstrated, discrimination based on Faception’s classifiers may fulfill Deborah Hellman’s theory of wrongful discrimination as demeaning and could therefore be unjust and incompatible with liberal values in World 1.

As a result, the Genetic Link Objection to the defense of premise four fails, and premise four of the LAAFD remains true: Government use of FPT could lead to government practicing unjustified discrimination. Premises one through four of the LAAFD have been shown to be true. Therefore, the LAAFD is valid and sound. The conclusion is therefore true: government use of FPT could be incompatible with classical liberalism.

4 Policy recommendations for FPT use in liberal democracies

The LAAFD and its defense do not demonstrate that the use of FPT is immoral per se. Instead, this analysis demonstrates that there are serious tensions between the essential values of classical liberal democracy and the use of FPT by the government as described in World 1. If a liberal democracy is to uphold the essential value of treating everyone equally before the law, then that government’s use of FPT seriously risks perpetuating unjust discrimination against members of its society.

This paper is the first philosophical analysis on the moral problems with FPT used by liberal democratic governments. As such, this section is devoted to outlining the most important policy proposals for liberal governments to consider should FPT programs like Faception be used by law enforcement. What follows is neither a complete nor detailed list of policies. Such a task is for another paper. These are merely the first steps to guard against the most egregious threats to liberalism that FPT poses to citizens. The following policy recommendations should therefore be considered necessary but not sufficient as prima facie steps toward protecting liberal democratic values if FPT is to be utilized by liberal democracies. Additional work is needed on this topic to further develop the policies that should govern FPT.

5 Transparency

The first policy recommendation regulating government FPT use in liberal democracies is transparency. A central tenet of classical liberal theory, originating in Locke’s Second Treatise on Government, is the idea that government gets its authority from the consent of the people. Accordingly, all citizens possess pre-political natural rights, and to protect those natural rights, citizens form a representative government to act as a third-party neutral arbiter whose function is to protect the natural rights of citizens (Locke, 1689). This means that the people should have a right to know when and how FPT is being used by law enforcement. More specially, it also entails that the people should have access to datasets, the specific facial classifiers, and the research that supports the accuracy and validity of the technology. Because the government is ultimately accountable to the people in classical liberal theory, FPT use by the government must remain out in the open and without secret lists or registries. Some requirements on transparency could include two main pathways:

  1. 1)

    Informing the public that the government uses FPT.

The first step in transparency is for the people to be aware that FPT technology exists and that it is being used by the government that they elected to represent their interests and political values. In a liberal democracy, the government serves the people, so the burden is placed on the government to publicly disclose to the people the methods of mass surveillance that are being used for their own safety and security. This implies that public knowledge and awareness should not come from freedom of information requests or investigative reporting; instead, the government should preemptively announce and educate the public on FPT and its uses.

  1. 2)

    The public has access to details about the software being used (i.e.: datasets, the specific facial classifiers, and the foundational research upon which the software is based).

The second step for transparency is for the public to have access to the important details regarding the technology itself. Transparency without information is an empty promise. As with many uses of AI, FPT programs make use of datasets from which to ‘learn’ how to make associations and categories. As has been pointed out by researchers in the field of AI, the content of the dataset is vitally important for discerning how accurate the AI will perform in its specified function and provides a glimpse into potential problems. For example, it has been shown that incomplete datasets in FRT programs used by law enforcement have yielded inaccurate performance when applied to female and racially diverse faces (Buolamwini & Gebru, 2018; Garvie, 2019; Gentzel, 2021). This racial bias in FRT used by law enforcement in the U.S. has led to at least one innocent citizen being falsely arrested and detained (Allyn, 2020; Fussell, 2020). It is important for the datasets on which FPT programs are being trained to be available to discern if similar biases are being imbedded into the systems. Additionally, the public should know what face-classifiers are being utilized by the government. This information informs the citizens about what face-types are being picked out and scrutinized. Finally, technology is only as valid as the research and theories upon which it is based. The public should therefore be given access to all of the background research and theories of the FPT program being used.

5.1 Consent

Another policy recommendation for FPT use by the government in liberal democracy is for consent to be obtained before and while the technology is used. Consent can also be withdrawn after it is given, should the effects of FPT be deemed to be problematic or unacceptably harmful. Within this context, consent should be understood in terms of both the collective level (democratic votes) and individual level (when FPT is used in specific locations where individuals could be affected). Two bulwarks of classical liberalism are the concepts of government authority originating from the consent of the people and that there is a presumption in favor of liberty when it comes to individual freedom from government intrusions. From these two liberal assumptions follows the need for government to acknowledge that FPT used by law enforcement has the potential to infringe on the liberty of individuals and groups before crimes have been committed, and therefore the government should also acknowledge that the vulnerable individuals and communities need to have a voice with respect to the use of this technology. The idea of obtaining consent in liberal democracies before an authority exercises force or power over individuals is not without historical precedence. In the context of medical ethics, the idea of informed consent is held to be sacrosanct, especially in western democracies in the wake of historical atrocities wherein governments and scientists denied the bodily autonomy of innocent people in medical experiments or the practice of medicine. The atrocities of the Nazi experiments in World War II and the Tuskegee Syphilis experiment (Centers for Disease Control and Prevention) in the U.S., and many other examples have given rise to formal declarations of the ethical guidelines that govern liberal democratic societies for protecting the autonomy of individuals from unjust authority in the medical context. These declarations include the Nuremberg Code (United States Holocaust Museum), the Belmont Report (U.S. Department of Health & Human Services), and the Declaration of Helsinki (World Medical Association). These documents codify the importance of obtaining consent before and during medical experiments. Similarly, the use of novel biometric technology in a liberal democracy by the government that has the potential to violate liberal values in the ways articulated in this paper should likewise be utilized only through the enduring consent of the people.

5.2 Oversight and regulation

Since the topic of this paper is the government use of FPT, another important policy consideration is oversight and accountability. This includes regular and publicly available reports on the specific uses of FPT and their impact on individuals and groups in society. The enforcement of oversight and accountability of the use of FPT can come from the legislative branch of government, the judicial branch of government and ultimately the citizens themselves.

In the U.S., the legislative branch (Congress) serves the people through democratic representation as duly elected officials. As such, oversight on the use of FPT by the government should, in part, come from laws that are passed to regulate its use. Some of these laws could involve the use of the judicial branch (courts) to limit the government’s use of FPT. Some states and municipalities have, through local and state legislative acts, passed laws regulating government use of general FRT. For example, Arizona passed SB 1583 in 2021, which requires government agencies at the state and local levels who want to install FRT or other forms of surveillance to first seek authorization from the city council in which it is to be used (The State of Arizona 2021). Some states have passed laws that require law enforcement agencies that wish to use FRT to first obtain a warrant or court order before using FRT in a criminal investigation. Massachusetts has passed such a law, which utilizes both the legislative and judicial branches of government to oversee the use of FRT (The Commonwealth of Massachusetts 2020). Another type of oversight could restrict or make explicit the conditions under which FPT is permitted to be used by law enforcement. Utah has passed a law that states police are permitted to use FRT only when investigating felonies and violent crimes (Lewis & Crumpler, 2021). The state of New York has passed a law, Assembly Bill A954, that restricts the use of FRT in schools (The New York State Senate 2022). Given the concerns that were raised in World 1 and World 2 scenarios in Section IV of this paper, restricting or even banning the use of FPT on children would be a prudent oversight decision to consider. All of these forms of legislation for FRT can be proposed and passed to oversee the use of FPT, with enforcement delegated to the courts.

Outside of the lawmaking and judicial aspects of oversight, re-establishing The U.S. Office of Technology Assessment (OTA) should be a priority in overseeing the ethical use of FPT on citizens. Darrell M. West of the Center for Technology Innovation advocates for this thesis with respect to AI oversight in a recent report for The Brookings Institute (West, 2021). The OTA was an agency in the U.S. legislature created in the 1970’s to utilize impartial scientific studies and methods to provide congress with guidance on the harms and benefits of the most influential emerging technologies. It was defunded and disbanded in 1995 as part of a de-regulation campaign (Congressional Research Service, 2020). Restoring this important, impartial, and scientifically informed oversight committee to monitor and report on FPT used by law enforcement is vital to maintain the values of liberal democracy in the United States.

The preceding list of policy recommendations regarding FPT used by liberal governments is incomplete in scope and content. This list should be construed as only a prima facie attempt at policy proposals that could begin to protect the civil liberties of citizens in liberal democracies that choose to adopt FPT for law enforcement and threat assessment. More research and policy recommendations are urgently needed in this area to ensure the maintenance of the liberal value of equality before the law, and to prevent the social evils of pernicious discrimination described in World 1 in section III of this paper.

6 Future research

This paper focused on law enforcement use of FPT that analyzes immutable facial structure to predict personality traits and found that its use poses serious risks to liberal values. As mentioned in section II, many varieties of facial analysis programs exist and are being developed for use in other areas. For example, facial analysis is currently being used in medicine to predict the risk of and to diagnose diseases. The AI programs can detect subtle facial features associated with genetic disorders (i.e.: Down Syndrome and Turner Syndrome) with accuracy greater than 90% in some instances (Qiang et al., 2022). Facial analysis is also being developed to predict health problems in aging, patient behavior, pain levels, depression, and medication compliance (AMA J & Ethics, 2019). An important question for liberal democracies is whether government provided healthcare that utilizes facial analysis for disease prediction and diagnosis would pose similar risks (like the ones discussed in this paper) to equality before the law and ultimately lead to unjust government discrimination. While the LAAFD did not conclude that all forms of facial analysis are wrongful or inherently inconsistent with liberalism, it would be important for liberal democracies to examine the risks and normative boundaries of its use for government provided healthcare. Good or bad outcomes could occur. One could imagine scenarios in which government-provided healthcare systems that heavily utilize facial analysis to predict diseases in specific people could label those people as future-diseased, and then either deny healthcare (stigmatization) or marshal additional healthcare resources to those people (beneficence). Two related questions need to be addressed in future research: (a) Is government-provided healthcare enhanced with facial analysis consistent with equality before the law (since it would entail treating people very differently) and (b) Is treating people differently based on different medical risks and needs a morally relevant reason that justifies discrimination, and does it demean the equal moral worth of all individuals? The answers to these questions are beyond the scope of this paper, but the author fully acknowledges their importance.Footnote 4

Another important area for future research is the use of facial analysis programs to predict criminality not based on immutable facial structure like Faception, but instead by detecting subtle cues in facial expressions in individuals. There are two main issues that require further research regarding the use of this technology in liberal democracies. The first issue is racial bias. The author of this paper has previously published on racially biased FRT, including this type of facial expression software (Gentzel, 2021). This software detects angry-looking faces in large crowds, which can be used by law enforcement to identify threats from potential criminals. Researcher Lauren Rhue discovered in 2018 that two different facial analysis programs trained for this task were significantly more likely to interpret black faces as angry or contemptuous (and therefore threatening) than their white counterparts (Rhue, 2018, 2019). The unjust disparate impact the use of this technology would have on groups already experiencing bias and discrimination in many western democracies would render its use incompatible with liberal democratic values, at least until all bias has been eliminated. The second issue is supposing that bias can be eliminated, an important question remains: does profiling potential criminals based on facial expression (mutable traits) instead of facial structure (immutable traits) present with different moral concerns within liberal democracy than the FPT technology discussed in this manuscript? Could the former be less risky to liberal values than the latter? Renee Jorgensen Bolinger has argued from an epistemological bent that in social situations, epistemic mistakes based on immutable traits (like race) come with higher social costs by perpetuating additional harms to members of the stereotyped group than mistakes made from mutable considerations (like clothing), and therefore epistemic justification for a particular belief in social situations must take this fact into account (Bolinger, 2020). This seems correct, and the LAAFD presented in this paper does not conclude that the use of this technology would be inherently wrongfully discriminatory or violate equality before the law. Since facial expressions of individuals are mutable across time, a careful philosophical analysis could find that the use of expression detecting technology would be less problematic than FPT and more akin to (comparatively) less controversial profiling techniques that rely on behavioral patterns of suspects. In the U.S., United State Supreme Court rulings like Terry v. Ohio establish the legal contours within which law enforcement may utilize behavioral patterns of suspects to warrant searches and seizures, and future research is needed to situate the use of expression detecting FRT within the existing legal and philosophical context (Terry v. Ohio). On the other hand, the reliability of the expression detecting AI could play a role in determining its legitimate role in liberal democracy. The reliability concerns associated with the use of the polygraph (lie detector) in the legal system has rendered its use in court trials as admissible only under well-defined conditions and in only some (and not all) jurisdictions (The U.S. Department of Justice Archives). Expression detecting technology could, upon closer examination, never reach the reliability threshold to be trustworthy for the modern liberal context.Footnote 5

7 Conclusion

This paper presented the first philosophical analysis of the ethical problems associated with FPT currently being used by the U.S. government. The author developed and defended the LAAFD, which concluded that government use of FPT in liberal democracies poses serious risks to the classical liberal value of equal treatment before the law. A key move in the argument was to show, by utilizing two hypothetical scenarios described in Worlds 1 and 2, that discriminating against groups of people with certain face-types could be unjust and incompatible with classical liberalism, mutatis mutandis, for the same reasons why discriminating against groups based on race would be unjust and incompatible with classical liberalism. While this hypothetical futuristic scenario is not preordained to occur, this thought experiment and its corresponding analysis should be taken as a cautionary tale of how government use of FPT could lead to discrimination that is arbitrary (Feinberg’s criteria) and possibly demeaning (Hellman’s criteria).

Given these grave moral concerns associated with FPT used by liberal governments, the author of this paper developed a prima facie outline of policies for liberal governments to adopt to mitigate unjust discrimination and the associated social harms. At a minimum, FPT use by liberal governments should be tempered with detailed regulations that emphasize transparency, consent, oversight, and regulation. This novel analysis delivers a warning that new technologies that bring promise can also bring peril, and that their use in liberal democracies should always be balanced against a continued commitment to the liberal values intended to protect the principles most sacred to liberal democracy.