Surveying Judges about artificial intelligence: profession, judicial adjudication, and legal principles

Abstract

Artificial Intelligence (AI) is set to bring changes to legal systems. These technologies may have positive practical implications when it comes to access, efficiency, and accuracy in Justice. However, there are still many uncertainties and challenges associated with the implementation of AI in the legal space. In this research, we surveyed Judges on critical challenges related to the Judging Profession in the AI paradigm; Automated Adjudication; and Legal Principles. Our results suggest that (i) Judges are hesitant about changes in their profession. They signal the need for adequate training that fosters legal literacy in AI, but are less open to changes in legal writing or their social and institutional role; (ii) Judges believe higher levels of automation only lead to fair outcomes if used in earlier phases of adjudication; (iii) Judges believe and are concerned about AI leading to Techno-Legal Positivism; and (iv) Judges consider that Legal AI technologies may have a positive impact in some legal principles, as long as everyone has equal access to those technologies and cybersecurity and judge on the loop safeguards are in place; and (v) Judges are strongly concerned about the de-humanization of Justice. They consider that assessing evidence, analyzing arguments, and deciding on a legal case should be inherently human. By surveying these practitioners, we aim to foster a responsible, inclusive, and transparent innovation in Justice.

1 Introduction

Artificial Intelligence (AI) can no longer be ignored in the legal field. Traditionally, this field has been slow in adopting new technologies (Simshaw 2018; Baker 2018), but AI is finally set to bring changes to legal systems. In the current data-intensive paradigm, Legal AI amounts to a multitude of methods, tools, and technologies sustained by Machine Learning algorithms that are used to solve problems in the legal space (Bansal et al. 2019; Rosili et al. 2021; Dyevre 2020).

Legal AI operates as a supportive, replacement, or disruptive technology (Sourdin 2018). At the most basic level, supportive Legal AI assists in informing, supporting, and advising legal professionals in tasks, such as research and document review (Sourdin 2018; Baker 2018). At a higher level, replacement Legal AI may take over simple disputes, thus relegating Judges to an appellate or review function to ensure meaningful human oversight of the technology (Sourdin 2018; Santoni de Sio and Van den Hoven 2018).

At the highest and most speculative level, Legal AI may eventually disrupt the legal system, thus paving the way to novel forms of Justice (Sourdin 2018).

These technologies may have positive practical implications when it comes to access, efficiency, and accuracy in Justice (Davis 2019). Legal AI is already able to help individuals, litigants, and Judges with simplifying, organizing, and managing legal information (Simshaw 2018; Reiling 2020). These technologies automate processes and parse large amounts of data, thus saving practitioners a substantial amount of time (Rothmann and Rossouw 2020; Park et al. 2022; Edwards and Miller 2019), and mitigating court backlog problems (Bielen et al. 2018; Fjelstul et al. 2023), and human error (Surden 2014; Barysė and Sarel 2023).

Despite the positive outlook, there are still many uncertainties and challenges associated with the implementation of Legal AI in Justice (Sourdin 2018; Poppe 2019; Land and Aronson 2020; Završnik 2020; Alarie et al. 2018; Markovic 2019). The increased use of these technologies entails a disaggregation of legal work (Agrawal et al. 2019), raising concerns about changes in legal professions and displacement of legal jobs (Brooks et al. 2020; Markovic 2019). Moreover, it is also unclear how to balance values of efficiency, trust, and fairness in AI-powered judicial adjudication (Xu and Wang 2021; Araujo et al. 2020; Barysė and Sarel 2023; Lockey et al. 2021; Aarts 2020). These technologies may also impact core Legal Principles (Zalnieriute 2021), such as Equality or Judicial Secrecy, but for now, it remains unclear whether such impact will be positive or negative.

Along the same lines, Legal AI technologies are unlikely to solve structural problems in Justice. For Legal AI to realize its potential, it needs to be supported by adequate policies. For example, Legal AI has the potential to promote efficiency in Justice, yet policies are needed to prevent understaffed Courts from relying on Al as a force multiplier to allow more work to be done with limited resources (Campbell 2020).

To foster a responsible innovation approach to Legal AI, entailing a transparent and interactive process in which developers, stakeholders, and societal actors become mutually responsive for the ethical acceptability of the technology (Von Schomberg 2011), it is critical to engage and learn the perspectives of legal practitioners about the challenges and uncertainties concerning Legal AI.

Recent empirical studies have made significant contributions for this literature (Yalcin et al. 2023; Helberger et al. 2020; Barysė and Sarel 2023; Xu and Wang 2021). Some studies surveyed individuals on their beliefs, acceptance and intentions to support Legal AI technologies (Barysė 2022a, b), while others explored whether algorithmic judicial decision-making can be trusted (Yalcin et al. 2023) and perceived as fair (Araujo et al. 2020; Helberger et al. 2020; Barysė and Sarel 2023). However, these studies provide very little information on the views of Judges, the quintessential decision-makers in the current judicial paradigm, and have also failed to investigate important challenges related to Legal AI.

This exploratory study aims to survey Judges currently affiliated with Portuguese Courts on the impact of Legal AI in the Profession, Judicial Adjudication, and Legal Principles. We believe that Portugal is a good case study for this research, provided that it has a well-established civil law system and Judges have some exposure to court technologies, such as e-services or videoconferencing. Moreover, it is an interesting environment to explore the perceptions of Judges about technology as a driver of efficiency, given that the Portuguese Judicial System has been heavily scrutinized for the extensive length of judicial proceedings.¹

Our comparative logistical advantage in this country allowed us to deploy the survey on the three levels of the Judicial System, from lower courts to the Supreme Court of Justice, thus accounting for a wide diversity of Judges and courts in the country. By surveying such a diverse mix of Judges, this study is expected to foster a responsible innovation approach in Legal AI that is inclusive of a broad range of perspectives (Stilgoe et al. 2020; Owen et al. 2013).

The article is organized as follows: first, we provide some reflections on the impact of Legal AI on the legal profession, judicial adjudication, and legal principles, thus framing the research questions addressed in this study. Subsequently, we describe the research methodology. Later, we present and discuss the results and main findings. Finally, we draw conclusions and present directions for further research related to Legal AI.

2 Legal profession

Legal AI technologies are expected to impact legal professions. While there are certainly many ways to characterize the legal profession, in this research we frame it through the elements of Legal Education; Legal Writing; and Social Role.

Future generations of practitioners will likely experience the impact of Legal AI starting in Law School. The legal curriculum needs to be adjusted to prepare these practitioners for the AI paradigm (Ma and Hou 2021; Vučić 2023; Janoski-Haehlen and Starnes 2020; Carrel 2018). Such endeavor represents an opportunity for Law Schools to embrace change and prepare practitioners to become more consistent, objective, and precise in their professional practices (Reid 2018). Yet, it is still unclear how to expand the legal training to include AI while also fostering core competencies in legal decision-making and problem-solving (Carrel 2018).

Law Schools may introduce materials designed to foster legal literacy on AI to prepare practitioners for the legal challenges associated with AI. These materials would follow the traditional structure of legal courses, focusing on the theoretical aspects of the legal challenges associated with AI technologies. Law Schools may also introduce materials designed to foster technical literacy on AI to prepare practitioners to use these tools in their research and professional endeavors. Such courses would have an interdisciplinary and practical nature, focusing, for instance, on applying Machine Learning and Natural Language Processing techniques to extract information from legal text data (Savelka et al. 2020).

Reflecting on Legal Education in the AI paradigm also requires some considerations about the training of Judges. In Civil Law Systems, such as Portugal, Judges are recruited by Judicial Schools through a competitive entrance exam (Spáč 2018), and receive extensive formal training in theoretical and procedural subjects (Muniz-Arguelles and Fraticelli-Torres 1985). Accordingly, it is also necessary to further investigate the integration of AI in the training of these practitioners.

Another core element of the legal profession is legal writing. A staple in the legal field, legal writing traditionally relies on coherent arguments about the law, featuring logical deduction, inference, and normative claims (van Boom et al. 2018). It tends to be elegant, convoluted and rich in jargon (Pollman 2001; Osbeck 2011). Increased automation of judicial adjudication may require some adjustments to this particular style of writing, to make it more amenable to data extraction and automation.

A particular challenge for automated extraction and analysis of legal data concerns the nuances in the narratives underlying legal decisions. In Portugal, a controversial legal decision that cited the Bible and a 1884 Penal Code to frame a case of domestic violence,² prompted a public debate about Judges including their subjective views in legal decisions. One study analyzed the controversial decision and reported that the Judge formulated a reasoning apparently based on a generic content of common experience only to give voice to a highly subjective judgment (Pinto 2021).

To facilitate the automation of the judicial adjudication, legal writing may need to evolve to become more objective and structured, in other words, closer to programming languages. However, given the reluctance of the legal community to change their practices (Reid 2018), the innovation onus with respect to automation and extraction of legal data may fall entirely on the technology community.

Due to increased digitalization and automation in Courts, the social and institutional role of Judges may also undergo some changes. Assuming that Legal AI will free Judges from repetitive and time-consuming work, these practitioners may have more time to engage in ancillary activities, such as legal research, teaching, management of Courts, or even community projects.

In Civil Law systems Judges are regarded as bureaucrats well-versed in the technicalities, idiosyncrasies, and complexities of the Law, yet slightly isolated from the larger legal and social community (Eltis 2014). There are historical reasons for such isolation, which concern the need for impartiality and authority over other legal professionals and fellow citizens, but there are also practical reasons related to the day-to-day court workload.

The phenomenon of Judicial Isolation has been described in the literature as part of a wide-ranging and deep acculturation process (Zimmerman 2000; Eltis 2014). We speculate that this phenomenon may be even more prevalent in countries, such as Portugal, where Judges are recruited shortly after graduation and have little professional experience outside of court. Whereas impartiality concerns should remain relevant in the novel paradigm in Justice, Legal AI represents an opportunity to mitigate  such isolation, by allowing these practitioners to have the time for greater immersion in the community.

In the spirit of responsible innovation, we believe that it is relevant to gain knowledge about the perspectives of practitioners about the impact of Legal AI technologies on these core elements of their profession. In this research we survey Judges about their views on these matters. Although our sample is limited to Judges currently working in Portuguese courts, this information, should also be useful for Legal AI innovation in foreign jurisdictions.

3 Adjudication

Adjudication is a cornerstone of procedural fairness and trust in courts (Burke and Leben 2007; Burke 2020). Legal AI technologies allow the automation of some processes that underlie judicial adjudication and have the potential to save legal professionals a substantial amount of time, thus mitigating burnout (Rothmann and Rossouw 2020; Park et al. 2022; Edwards and Miller 2019) and court backlog problems (Bielen et al. 2018; Fjelstul et al. 2023). However, these technologies also raise important concerns about fairness.

Justice is a human endeavor. The introduction of Legal AI technologies in courts may undermine sentiments of Justice if individuals consider machine adjudication as less fair than human adjudication (Chen et al. 2022). Empirical research seems to substantiate these concerns. A recent study reported that court users have higher trust in human Judges, especially when legal cases involve emotional complexities (Yalcin et al. 2023).

Along the same lines, another recent study investigated fairness expectations in different phases of automated adjudication (Barysė and Sarel 2023). The authors Dovilė Barysė and Roee Sarelrelied relied on a taxonomy that divides adjudication into four different stages of decision-making (i) Information acquisition; (ii) Information analysis; (iii) Information selection; and (iv) Information implementation) and surveyed individuals on the levels of automation that would most likely ensure the fairest legal outcome in each adjudication phase (Barysė and Sarel 2023).

In this research, we build heavily on the study authored by Barysė and Sarel to investigate the perceptions of Judges regarding the levels of automation in different phases of adjudication leading to the fairest outcomes. It is quite convenient to replicate their study with a sample of Judges, given that Barysė and Sarel characterized the adjudication phases from the standpoint of the Judge: (i) information acquisition is the process of acquiring information (legislation, legal doctrine, jurisprudence) to resolve a legal case; (ii) information analysis concerns the critical analysis of the information and arguments relevant for a legal case; (iii) information selection concerns the process of selecting the best decision for a case, and (iv) information implementation concerns the preparation of the legal decision.

Moreover, the mentioned authors have also suggested that their investigation is more relevant in inquisitorial legal systems, provided that Judges have a more active role in the investigation, when compared to Judges in adversarial legal systems (Barysė and Sarel 2023). By replicating the study mentioned above (Barysė and Sarel 2023) on a sample of Judges currently working in Portuguese Judicial courts, we aim to further understand what would be considered a fair combination of human and algorithmic adjudication in an inquisitorial system.

4 Legal principles

Legal principles are basic standards of Justice and Morality (Dworkin 1971; Moore 1996; Dworkin 2013) that guide legal practitioners in the interpretation and application of the law. The relevance of these principles is well-expressed in the words of Ronald Dworkin, who considers that the law is better seen, not as a set of individuated rules emanating from a historical source, but holistically, as a constellation of principles owing their status as law as much to their moral correctness as to their historical, institutional basis (Moore 1996).

These principles gain prominence in spaces of Equitable Justice, where there is more room for discretion beyond the letter of the law, to ensure that the outcome is morally and ethically fair. The implementation of Legal AI in Justice, raises questions about legal principles.

Whereas any legal system features trade-offs between Equitable and Codified Justice (Re and Solow-Niederman 2019), provided that the strengths of Legal AI are hallmarks of Codified Justice (efficiency, uniformity, and predictability (Re and Solow-Niederman 2019), it is speculated whether these technologies will reinforce Codified Justice over Equitable Justice (Re and Solow-Niederman 2019). In extremis, Legal AI may prompt a resurgence of codified Legal Positivism. In a Techno-Positivism system, legally relevant variables are established in advance and standardized automated procedures are routinely applied to a set of facts, without much consideration for particular facts and circumstances (Re and Solow-Niederman 2019). In such system, there would be limited room for Legal Principles.

Even if Positivism is not revived, Legal AI technologies may still impact core Legal Principles, such as Judicial Secrecy or Access to Justice. So far, however, this topic remains unclear and largely unexplored in the literature. In this study, we investigate the views of Judges on a potential revival of Positivism and also on the impact of Legal AI on core Legal Principles.

Provided that participants in this study are Portuguese Judges, we believe it is important to characterize each principle in light of Portuguese Law. We surveyed Judges on six Legal Principles. Most of these Principles are stated in the Portuguese Constitution. This is the case of the Principle of Equality, which states that All citizens have the same social dignity and are equal before the law and No one may be privileged, benefited, harmed, deprived of any right or exempt from any duty due to ancestry, sex, race, language, territory of origin, religion, political or ideological convictions, education, economic situation, social condition or sexual orientation (Article 13); Principle of Access to Justice, which states that Everyone is guaranteed access to the law and to the courts to defend their legally protected rights and interests, and Justice cannot be denied due to insufficient economic means (Article 20/1); Principle of Judicial Secrecy, which states that The law defines and ensures adequate protection of the secrecy of justice (Article 20/3); and Principle of Legal Certainty which is a corollary of the Rule of Law (Article 2).

Others are stated in civil procedural law, as it is the case of the Principle of Simple and Clear Language, which states that The court must, in all its acts, and in particular in citations, notifications and other communications addressed directly to the parties and other natural and legal persons, preferably use simple and clear language (Article 9-A); and the Principle of Freedom in the Assessment of Evidence, which states that The Judge freely assesses the evidence according to his prudent conviction about each fact; free assessment does not cover facts for which the law requires special formality to prove, nor those that can only be proven by documents or that are fully proven, either by documents, or by agreement or confession of the parties, (Article 607/5), also stated in criminal procedural law (Article 127: Unless otherwise provided by law, evidence is assessed according to the rules of experience and the free conviction of the competent authority).

Although defined in light of Portuguese law, these principles relate to critical standards in any democratic legal system. Accordingly, our findings may be relevant to other jurisdictions. In particular, Civil Law Jurisdictions, which rely more heavily on codified Legal Principles, when compared to Common Law Systems.

5 Research questions

Based on the previous background about the impact of Legal AI technologies on the Judging Profession, Judicial Adjudication, and Legal Principles, the research questions formulated in this empirical study are:

• 1. What are the perspectives of Judges on whether legal education should be adjusted to the new AI paradigm?

• 2. What are the perspectives of Judges on whether legal writing should become more structured and objective to facilitate automation?

• 3. What are the perspectives of Judges on the social role of the Judge in the Legal AI paradigm?

• 4. What are the perspectives of Judges on which levels of automation in different adjudication phases lead to the fairest outcomes?

• 5. What are the perspectives of Judges on the risk that Legal AI may lead to a Techno-Legal Positivism?

• 6. What are the perspectives of Judges on the impact of Legal AI in core Legal Principles?

6 Methodology

In this research study we relied on a survey featuring both quantitative (numerically rated) and qualitative (open-ended) questions (Ponto 2015; Creswell and Creswell 2017) to gain insight into the views of Judges with respect to Legal AI. The study was submitted to the Tufts University Institutional Review Board (IRB) and received an exempt determination. The methodology in this study consisted of three phases: (i) Development of Survey; (ii) Data Collection; and (iii) Data Analysis.

6.1 Development and overview of survey

Upon reviewing the literature and identifying the need for empirical research on Legal AI, a collaboration was established with the Portuguese Supreme Court of Justice to further explore this research. Subsequently, a survey was developed in Qualtrics,³ an online software tool for creating and implementing surveys. The content validity of the survey was reviewed by a panel of (n = 9) Judges in three separate sessions. The survey was developed in both English and Portuguese languages, for the purpose of the Tufts University IRB process, but it was deployed in Portuguese. Both versions can be found in the Supplementary Materials.

The final version of the survey features seven groups of questions (Table 1). The group related to the impact of AI on the Judging Profession includes questions about Legal Education, Legal Writing, and Social Role of Judge. On Legal Education, we surveyed participants about potential changes to the legal curricula, to prepare Judges for the novel technology paradigm. The options included in the survey encompassed Legal literacy on AI (Legal training for the challenges related to AI (AI Law)); Technical literacy on AI, (Training in computation and AI algorithms (AI algorithms); Training in Probability and Statistics (Prob & Stats)); and Status quo (Current legal education and training is adequate for the AI technology paradigm). For participants who did not select the Status Quo option, the survey features a follow-up question about the moment that such changes legal education should take place, featuring a combination of Law School, Judge Traineeship, and Professional/Continuing Education of Judges.

On Legal Writing, we surveyed participants about potential changes to the traditional legal writing style, to facilitate automation and use of AI in Courts. The options included in the survey encompassed Major Changes to Legal Writing (Writing in legal decisions and rulings should become more objective and structured to facilitate the AI processes); No Changes & Summaries (No changes to the writing style but complete, structured, and objective summaries on each case are made available); and No Changes (AI algorithms should only be used when they have the ability to interpret the complexity of legal writing).

On Social Role of Judge, we assumed that digitalization, automation, and AI will contribute to a greater de-bureaucratization of Justice and will free Judges from repetitive and time-consuming tasks. We surveyed participants about potential changes to the traditional social role of Judges in the AI paradigm. The options included in the survey correspond to different possible roles that Judges may take on or become more involved with, such as Research and Teaching (Academia); Management of Court; Management of Judicial Adjudication; Engaging in Community Projects, and one option was also included to keep the Status Quo provided that no changes, should be made to the current institutional and social role. In this question, the participants could select two options.

The group of questions related to Judicial Adjudication aims to investigate the automation levels in different phases of adjudication that Judges consider as leading to the fairest legal outcomes. This group includes four questions corresponding to the different phases of adjudication [(i) Information Acquisition; (ii) Information Analysis; (iii) Information Selection; and (iv) Information Implementation). In each phase, five statements are presented to the participants, reflecting different levels of automation ((i) No Automation; (ii) Intermediate Automation; (iii) High Automation; and (v) Full Automation]. For example, in Information Implementation, which requires the Judge to write the legal decision, the following statements are presented: (i) Only the Judge can prepare the judicial document with the final decision and its reasons; (ii) The software program generates the text with the argument and final decision and the Judge can accept, edit, or reject the text; (iii) The software program prepares the text with the argument and final decision. The Judge has a certain time limit to reject the text. In more complex cases, the Judge himself prepares the text of the sentence; (iv) The software program prepares a text with the argument and final decision. The Judge has access to this text, but can only edit the text in exceptional cases; (v) The software program prepares a text with the arguments and final decision, automatically generates the sentence, and sends it to the parties, with the Judge only reviewing this text if one of the parties appeals A full description of the automation levels can be found in the Supplementary Materials.

On Impact of AI in Legal Paradigm & Legal Principles, we surveyed participants on their perceptions about the potential impact of Legal AI technologies in prompting a novel Techno-Legal Positivism Paradigm, and we also invited participants to share their views and comments about the impact of Legal AI on core Legal Principles.

Table 1 Overview of survey

6.2 Data collection and analysis

The survey was deployed via email among Judges currently working in Portuguese Judicial Courts. Three different institutions deployed the survey: The Portuguese Supreme Court of Justice (Supremo Tribunal de Justiça) and the Intermediate Appeal Courts (Tribunais da Relação) deployed the survey among Judges affiliated with those courts; and the Superior Council of the Judiciary deployed the survey among Judges working on the Lower Courts (Tribunais de Primeira Instância). The survey was accessible through a Qualtrics anonymous link that was made available online from March 27th 2023 to April 25th 2023. The data was stored in Qualtrics.

For the data analysis, we first proceeded with pre-processing the data. The original csv file exported from Qualtrics featured 390 responses, however, after removing the responses in which participants did not provide consent (N = 48) and incomplete responses (N = 63), i.e., responses featuring more than 3 missing questions, our final dataset consisted of 279 responses.

Upon pre-processing the data, we computed a reliability score using Flanagan’s formula on the responses related to the technology adoption and adjudication, which were captured in numerical variables, and report a coefficient score of 0.736, which indicates internal consistency of data. Subsequently, we proceeded with the statistical analyses in Python. Finally, we used a qualitative research approach to analyze the textual data provided in the open-ended questions, about the impact of Legal AI technologies in Legal Principles.

7 Results

7.1 Characterization of the sample population of Judges

The participants in this study are (N = 279) Judges currently working in Portuguese Judicial Courts. We consider the sample population diverse and representative of the population of Judges in Portugal, with respect to Gender (prevalence of female Judges), Stage of Career (prevalence of mid and late-career Judges), Type of Court (prevalence of Judges working in the Lower Courts) and District (prevalence of Judges based in Lisbon) (Table2).

Table 2 Sample of population of Judges

To gain insight into the personal innovativeness of the participants in our study and investigate whether the sample is biased towards technology, we computed a technology acceptance score based on the Innovation Diffusion Theory (Rogers 2010):

$$\begin{aligned} a = \sum _{f=i}^{F}\frac{F}{\textrm{max}(F)}, \end{aligned}$$

(1)

where a is the acceptance score, f $\in$ F is a set of technology acceptance factors defined in this study as $F = [adoption, trust, compatibility, advantage]$ and max(F) is the maximum score possible associated with each factor.

Adoption relates to the technology adopter categories. These categories are defined as innovators (individuals who seek and embrace innovation), early adopters (individuals who are open to innovation but not so risky as innovators), early majority (individuals who adopt innovation just prior to the average member of the social system), late majority (individuals who are slower to adopt and are skeptical about innovation), and laggards (individuals who are suspicious of new services, technologies, or processes and are the last to adopt an innovation) (Rogers 2010). In our study we assigned scores ranging from 1 to 6 to the survey options associated with each of these categories (1 = laggards and 6 = innovators). Accordingly, the $max(adoption) = 6$. Trust relates to the trust in technologies (Rogers 2010) used in court, assessed on a 5-point Likert scale where $max(trust) = 5$. Compatibility relates to the compatibility of technologies with the values and mission of courts, assessed in a 5-point Likert scale where $max(compatibility) = 5$. Advantage relates to the relative advantage of technologies in improving efficiency in courts, decreasing errors, and improving the experience of court users—each advantage item was assessed on a 5-point Likert scale where $max(advantage) = 15$.

Upon computing the final a score for each participant, these scores were feature-scaled (each score was divided by the max score) to range between [0,1]. As illustrated in Fig. 1, the distribution of technology acceptance resembles a normal distribution, which is consistent with the models described in the literature (Kaminski 2011). This indicates that our sample of Judges is not biased, i.e., it is neither conservative nor innovative towards technology.

Fig. 1

Technology adoption score

7.2 Profession

7.2.1 Legal education

What are the perspectives of Judges on whether legal education should be adjusted to the new AI paradigm? In this study, we report that Judges consider that legal education should be adjusted to foster legal literacy about AI, to prepare Judges for legal cases related to AI, and that such materials be offered throughout all the phases of legal training (Law School, Judge Traineeship, and Professional/Continuing Education of Judges) (Figs. 2 and 3).

Fig. 2

Legal education in the AI paradigm: training of Judges

Fig. 3

Legal education in the AI paradigm: training phase (LS stands for Law school; JT stands for Judge Traineeship; and CE stands for Continuous Education)

7.2.2 Legal writing

What are the perspectives of Judges on whether legal writing should become more structured and objective to facilitate automation? We report that Judges consider that legal writing should not undergo any changes for the purpose of facilitating automation. The participants in this study believe that AI algorithms should only be used in courts when they have the ability to interpret the complexities and nuances of legal writing. If some innovation were to take place with respect to legal writing, our results indicate that Judges have a preference for making available objective and structured summaries about each legal case, so that it would be easier for algorithms to read the data (Fig. 4).

Fig. 4

Legal writing in the AI paradigm

7.3 Social role of Judge

What are the perspectives of Judges on the social role of the Judge in the Legal AI paradigm? We report that the Judges who participated in this study consider that their social and institutional role should remain the same in the novel AI paradigm and that any changes should be related to increased engagement in the management of judicial adjudication (Fig. 5).

Fig. 5

Social role of Judge in the legal AI paradigm

7.4 Adjudication

What are the perspectives of Judges on which levels of automation in different adjudication phases lead to the fairest outcomes? We report that Judges believe automation may lead to fair outcomes if used in the earlier phases of adjudication. Judges are open to higher automation in the Information Acquisition phase, intermediate automation in the phases of Analysis and Selection of Information, and low automation in the Implementation of Information phase. We also report that Judges are more open to automation for selecting information when compared to analyzing information (Fig. 6).

Fig. 6

Levels of automation in judicial adjudication leading to the fairest outcomes

7.5 Legal principles

What are the perspectives of Judges on the risk that Legal AI may lead to a Techno-Legal Positivism? We report that the vast majority of Judges who participated in the study believe (85%) and worry (63%) about Legal AI leading to a revival of (Techno-Legal) Positivism (Table 3).

Table 3 AI leading to techno-legal positivism

What are the perspectives of Judges on the impact of Legal AI in core Legal Principles? We asked participants to provide comments about the impact of Legal AI technologies on particular Legal Principles. We report a total of 109 comments, which were divided into positive, negative, and neutral⁴. In our sample, we observe a prevalence of positive comments associated with the Principle of Equality and negative comments associated with the Principle of Freedom in the Assessment of Evidence (Table 4). A more detailed reflection on these comments is provided in the Discussion section. All comments written by participants are available in the Supplementary Materials.

Table 4 Impact of legal AI technologies in legal principles

8 Discussion

This study provides valuable insights into the views of Judges on challenges and uncertainties related to Legal AI. These insights allow us to further reflect on the novel technology paradigm in Justice. We frame this discussion around three themes: (i) Judge in the AI paradigm, (ii) Automated Adjudication, and (iii) Legal Principles in Techno-Legal Positivism. Each theme opens avenues for further empirical and theoretical research.

8.1 The Judge in the AI paradigm

The first challenge we addressed in this study concerns the impact of Legal AI technologies in the Legal Profession. We aimed to gain insight into the perspectives of Judges on the judging profession in the Legal AI paradigm. For this purpose, we surveyed the participants in this study on potential changes in core elements of their profession (Legal Education, Legal Writing, Social Role). In general, our results suggest that Judges are hesitant about changes in their profession.

Regarding Legal Education, our results indicate that Judges acknowledge that the legal curricula should be adjusted to prepare practitioners for the novel technological paradigm. Participants in this study signal a need for improving legal literacy about AI. On the contrary, they showed less interest in improving technical literacy about AI. This may be explained by a sense of urgency in providing solutions to the legal challenges related to AI, but also by the perspective that there is little added value in learning the mathematical and computational foundations of AI. We speculate that, when confronted with the prospect of learning these materials, Judges fail to grasp the importance of these foundational concepts in their day-to-day work.

Our results substantiate the need for incorporating materials that foster legal literacy in AI in the Law School curricula. These empirical insights are aligned with recommendations that have been formulated and published in the legal scholarship, Johnson and Shen (2020); Reid (2018). A study that examined the course catalogs of United States-based Law Schools has recommended introducing a Law & AI course in every Law School (Johnson and Shen 2020). They reported that most of Law & AI courses are overview courses, i.e., broad courses that incorporate different aspects of AI, but they consider that specialized courses should also be developed, as these courses allow students to gain a deeper understanding of the impact of AI in a particular area of Law (Johnson and Shen 2020).

Further research is needed to identify the best way to incorporate these materials in the legal curricula in a way that facilitates the critical thinking and exploration of the moral and ethical questions associated with AI, Data, and Algorithms (Reid 2018; Ryan 2021).

The results of this study also indicate that Judges are quite conservative when it comes to Legal Writing. This is not surprising, as the traditional legal writing style is considered a staple of the judging profession. Most participants in this study do not wish to engage in a more objective and structured writing style to facilitate the use of AI algorithms, although they are open to having structured summaries of legal cases made available.

In Europe, an effort has been made in the digitalization and identifiability of legal cases through the European Case Law Identifier (ECLI), a uniform identifier for EU legal cases. A summary is provided along with each case, but those summaries fail to show information about the critical elements of the case. There are many ways to compose a summary that would provide a thorough representation of the case, thus facilitating not only the use of AI algorithms but also the comprehension of the case by citizens who are not trained in the Law.

In the spirit of the FAIR Guiding Principles for Scientific Data Management and Stewardship, which states that data should be described with rich metadata featuring a plurality of accurate and relevant attributes, we consider that summaries of legal cases should also include information about the parties involved in the dispute (without compromising anonymization), a review of the arguments used to support the decision, and a detailed account of the final decision.

Participants in our study were also conservative about the Social Role of the Judge in the AI paradigm. Most Judges consider that their social and institutional role should not change, and any changes should be for Judges to be more involved in management activities, namely, managing Judicial Adjudication and also managing the Court. In contrast, participants in this study reject involvement in social and community projects. Reluctance to participate in such activities may be rooted in the incompatibilities with the function of the Judge. We speculate that participants in this study may consider that community involvement undermines their impartiality and independence.

In general, the results of this study suggest that Judges are not open to striking changes in their profession. These practitioners would most likely prefer that legal educators, policymakers, and technology developers make incremental rather than disruptive changes. Such an incremental approach may allow legal professionals to better adjust to the new Legal AI paradigm.

8.2 Automated adjudication

The second challenge we addressed in this study concerns the impact of Legal AI in Judicial Adjudication. We investigated the perceptions of Judges concerning the levels of automation in different adjudication phases leading to the fairest outcomes. For this purpose, we used a simplified model of judicial adjudication featuring four phases ((i) Information Acquisition, (ii) Information Analysis, (iii) Information Selection, and (iv) Information Implementation).

As mentioned earlier, this investigation is particularly relevant in inquisitorial systems, such as Portugal. In these systems, the Judge is actively involved in investigating the facts of the case to ascertain the truth. Therefore, it is relevant to understand the views of Judges on what would be a fair combination of human and algorithmic adjudication.

The results suggest that Judges believe higher levels of automation only lead to fair outcomes if used in earlier phases of adjudication (21% of participants have a preference for high and full automation in the Information Acquisition phase, whereas only around 3% of participants believe such levels should be used in the remaining adjudication phases).

The participants in this study are open to intermediate levels of automation in the Information Acquisition, low/intermediate automation in the Information Analysis phase, intermediate automation in the Information Selection phase, and low automation in the Information Implementation phase.

Following Barysė and Sarel (Barysė and Sarel 2023), we also consider that the preferences of Judges for automation of adjudication may be driven by the relative advantage of Legal AI in acquiring and selecting information rather than its advantage in analyzing it. While further research is needed to determine the particular reasons and motivations of Judges regarding their automation preferences, taking into account the comments made by participants in this study, which will be further explored below, we speculate that Judges believe that analyzing arguments and making a final decision on a legal case should be inherently human.

This is aligned with the perspective that AI should be a tool to be used by practitioners but should not replace them in critical aspects of their profession, especially in domains such as Justice or Medicine. Along those lines, a study that surveyed medical doctors reported a perspective in which these practitioners regard AI as a helpful tool that will allow doctors to have the time to focus on Medicine (Martinho et al. 2021). In the case of Justice, it seems that Judges would welcome Legal AI technologies to take over some of the more bureaucratic aspects of judicial adjudication, but not others that relate more to the core of Justice, such as analyzing arguments and preparing legal decisions.

Our findings have important implications for legal and technology communities. In recent years, an increased amount of attention has been paid to normative aspects of Legal AI. However, often those discussions focus on speculative technology, without much consideration for the state-of-the-art or input from stakeholders and developers. This study is yet another reminder about the need to mitigate speculation and ground the normative debates about AI. Rather than focusing on some artificial super system set to take over difficult human moral and legal decisions (Robot Judge), both legal and technology communities should focus on the complexities of developing and integrating AI technologies in the phases of acquisition and selection of information.

8.3 Legal principles in techno-legal positivism

The third challenge we addressed in this study concerns Legal Principles. We investigated the views of Judges on whether there is a risk of a revival of Positivism in the Legal AI paradigm. Our results suggest that most participants in our study believe (and worry) that AI will lead to Techno-Legal Positivism. In such a system, there would be little room for Legal Principles, as morally-charged flexible elements in the Law.

However, we also found that Judges have some positive views when asked about the potential impact of Legal AI on particular Legal Principles. We posit that such views relate mainly to the potential of Legal AI to promote the underlying values associated with Legal Principles, and not so much to the preservation of their formal structure in the legal system.⁵

The positive outlook on the impact of AI in Legal Principles is anchored in values of innovation and uniformity. Judges understand the value of technology in the legal space and consider that Legal AI may promote or facilitate the underlying values associated with some Legal Principles. One Judge wrote that these technologies would lead to a better preservation of secrecy of Justice and easier identification of their perpetrators, and another Judge wrote that the progressive use of new technologies will enhance the use of progressively simpler and clearer language.

Judges also consider that the uniformity associated with legal technologies will lead to increased equality. They believe that AI will bring gains to equality, because decisions will tend to be predictable, with no undesirable contradictions occurring for the same de facto situation, but it is also cautioned that the impact is positive if all citizens have access to these technologies.

Indeed, the Justice Gap is a point of contention. Whereas some Judges consider that Legal AI will mitigate this gap (The impact of new AI technologies will be positive on the Principle of Equality, to blur a Justice for the ‘poor’ and a Justice for the ‘rich’, since the latter is essentially based on a labyrinth and formal ‘imbroglio’ of the process so that it never comes to an end), others are concerned about accessibility to Legal AI as a critical step to access the Legal System (The impact will necessarily be positive, based on the assumption that ordinary citizens also have generalized access to these new technologies, as Justice users will be granted greater autonomy in accessing information from the courts, contributing, if associated with other actions, to greater legal literacy and a better understanding of Justice by people in general).

To ensure that Legal AI technologies have a positive impact in core Legal Principles, Judges consider that safeguards must be placed, namely, with respect to Cybersecurity. There are concerns about who has access to and manages the systems. One Judge wrote that the impact on secrecy can be positive, as long as the system’s security is safeguarded, without the possibility of third-party intervention, and the access can be monitored and another Judge wrote that the use of new Artificial Intelligence technologies could undermine the Principle of Secrecy of Justice, since it ignores who manages the access platforms.

Another safeguard that is often mentioned by the participants is the Judge on the loop. Participants consider that Legal AI is a useful tool, but Judges need to keep control of judicial decisions (Artificial intelligence tools should only be used as a source of information for the Judge to prepare the decision, without interfering in the slightest in the decision-making process; It should be a tool to help the Judge, but not a tool that replaces the Judge; New technologies must serve as an aid to human decision-making, never as a substitute for it, and limits must be imposed on their intervention so as not to relegate the role of the Judge).

The need for a Judge on the Loop is emphasized in the context of the Principle of Freedom in Assessment of Evidence. Judges consider that this is the core of their profession and that it needs to be secured by a human (It is in the field of assessing evidence (the core of the art of being a Judge) that I see greater difficulty and dangers in the application of AI technologies; AI is totally incompatible with the evaluation of evidence; In this specific context and considering that the principle in question is particularly relevant at the sentencing stage, I believe that new technologies should not intervene once the facts have been established).

Along the same lines, participants in this study showed a strong concern about the De-humanization of Justice, as they fear that Legal AI technologies will weaken the human factor in Justice (Justice is a human concept, made for human beings and by human beings, and any decision-making process is both rational and emotional; The new technologies will de-characterize and dehumanize the exercise of the judicial function, as we know it, in terms of efficiency (speed and costs) enhancing the alteration of the truth-finding process (lawyers will present the case in a way that guarantees a greater probability of success), more judicial errors (Judge tends to follow the solutions proposed by the system), crystallization of jurisprudence (Judge tends to repeat the majority solution) and less fair results (the repetition and linkage to the majority solution will prevent the weighing of the particularities of the case, that escape the great statistical trends).

Our exploratory study lays the ground for further and more detailed research about the impact of Legal AI on Legal Principles. The main message for developers and policy-makers is that Justice is a human endeavor. Technological innovation should, therefore, allow core elements of adjudication to remain inherently human.

9 Limitations

There are some important limitations in this study. The first limitation concerns the methodology (survey). To investigate the views of Judges about Legal AI, we used a survey, which is a research tool that extracts the opinions of participants separately, thus failing to uncover latent connections and nuanced perspectives about AI. Moreover, in the testing sessions with different cohorts of Judges, it was clear that, to foster engagement in this research, the survey needed to be concise. Despite our efforts in designing such a concise survey, we reported many incomplete surveys (N=111), which were removed from the analysis. We believe there are several reasons for these incomplete surveys, namely, poor exposure to empirical research in the legal field, which makes these practitioners less prone to engage in these types of studies; little time availability to complete the survey given their intense workload; and poor engagement or interest in AI.

The second limitation concerns the conceptualization of critical elements in this research, namely, Legal AI and Judicial Adjudication. In the survey, rather than providing a formal definition or a list of AI technologies that may be used in the legal space, we opted to frame Legal AI as a transformative technology that may prompt changes in the legal profession; automation technology that automates different tasks of judicial adjudication; and disruptive technology that may disrupt the current legal paradigm. While our conceptualization offers some advantages, by allowing participants to reflect on AI without detailing particular technologies within the constraints of existing legal frameworks, it may also introduce some vagueness. We also resorted to a simplified model of the judicial function. We used an over-simplistic four-stage model of Judicial Adjudication ((i) information acquisition, (ii)information analysis (iii) information selection and (iii) information implementation), which does not account for the complexities of Judicial Systems.

The third limitation concerns the participants in this study. We focused only on Judges, thus leaving out other legal practitioners, such as lawyers or judicial clerks, who also need to be involved in developing and implementing Legal AI technologies. Moreover, we did not assess the AI literacy of the Judges who participated in the study. We computed a Technology Acceptance Score (Sect. 7.1), to evaluate if the the sample of Judges was balanced with respect to acceptance of technology, and we also presented AI as a technology featuring particular characteristics ( transformative, automation, disruptive), so that participants could grasp the concepts without too much background knowledge of AI. However, we consider that having insights about the AI literacy of Judges would improve this study, as it would allow us to expand our statistical analyses and draw additional conclusions.

10 Conclusion

AI is set to bring changes to the Legal domain. In this study, we surveyed Judges about critical uncertainties and challenges related to Legal AI. We focused on the challenges related to the impact of these technologies in the Judging Profession, Judicial Adjudication, and Legal Principles.

Our findings suggest that Judges are hesitant to change the core elements of their profession. They signal the need for training that fosters legal literacy in AI, but are less open to changes in legal writing or their social and institutional role. We also report that the Judges who participated in this study are more open to higher automation levels in the early phases of judicial adjudication. They believe assessing evidence, analyzing arguments, and deciding on a legal case should be inherently human. Finally, our findings suggest that Judges are concerned about AI leading to Techno-Legal Positivism. They consider that Legal AI technologies may have a positive impact on some Legal Principles, as long as everyone has equal access to those technologies and Cybersecurity safeguards are in place. They also emphasize the need for a Judge in the Loop: Justice is a human endeavor.

This study opens avenues for future research in Legal Education, Automated Adjudication, and Legal Theory. Legal education studies are needed to guide Law Schools in developing materials and courses about AI. Moreover, additional studies should extend our research about the automation of judicial adjudication. It is relevant to frame automated adjudication, particularly the phases related to acquiring and selecting information, within particular legal systems, thus accounting for legal, logistical, and behavioral complexities. Finally, future studies in Legal Theory should explore the normative foundations, potential, and perils of Techno-Legal Positivism.