Introduction

Healthcare systems are coping with the challenge of countering the exponential growth of healthcare costs worldwide, which has far outpaced the growth rate of Gross Domestic Product (GDP) in individual states. These constrained finances, combined with an aging population and an increase in chronic diseases, could jeopardize their sustainability [1, 2]. The value of Artificial Intelligence (AI) in healthcare and its ability to address healthcare delivery issues has been a subject of discussion within the scientific community for several years [3,4,5]. Expectations for the use of AI in healthcare are high, as these technologies have demonstrated their potential as valuable support in various medical specialties, including mental health [6], radiology [7], oncology [8] and ophthalmology [9], as well as in improving waitlist reduction [10], treatment adherence [11] and therapy personalization [12]. Key stakeholders are pushing for a concrete integration of AI into clinical practice, transcending the theoretical or experimental boundaries discussed so far [13, 14]. However, although the rapidly growing list of AI-based clinical algorithms approved by the Food and Drug Administration (FDA), their real-life usage remains limited [14]. The challenges encountered on the path to integrating AI into clinical practice go well beyond the initial development and evaluation phase. As a result, it is crucial for the scientific community to have a clear understanding of medical technologies based on AI and how they are regulated, to assess their access in the toolkit available to healthcare professionals. The aim of this manuscript is to provide an overview of the primary uses of AI in the healthcare system and to discuss its future while shedding light on the major issues related to implications and gray areas within international regulatory processes. Figure 1 illustrates the main applications of AI in various aspects of health care.

Fig. 1
figure 1

Main applications of AI in several aspects of health care. References: Medical imaging and diagnostics [15]; Clinical research [16]; Rehabilitation [17]; Virtual Patient Care [18]; Patient engagement [19]; Administrative applications [20]; Adherence to therapy [21]; Clinical decision making [22].

Full size image

Definition

AI is the technology able to perform human cognitive functions through the creation of algorithms, such as learning and problem solving [23]. The recent Executive Order 15 U.S.C. 9401 [3] from the White House provided a very concise definition of AI: “a machine-based system that can, for a given set of human-defined objectives, make predictions, recommendations or decisions influencing real or virtual environments.” AI is not a ubiquitous universal technology, but is composed of different subfields that, individually or in combination, add intelligence to applications. The different subgroups are illustrated in Table I.

Table I Definitions of AI Subgroups
Full size table

Clinical Research and Drug Discovery

Effective preclinical preparation is essential for quality clinical studies, involving the identification of promising active ingredients and their targets, as well as the delineation of the experimental strategy for regulatory approval. Errors during this stage can condemn clinical studies to failure. Machine learning can assist researchers in reducing the inefficiencies in the preclinical process. The processes where AI can be used in clinical research are summarized in Fig. 2. The main ones are further detailed below:

Fig. 2
figure 2

Processes of using AI in clinical research.

Full size image

Identification of the Pharmacological Target and Mechanism of Action

AI is able to accelerate the process and enhance the identification of active ingredients’ targets and generating new molecules by synthesizing large amounts of current research, elucidating medicine mechanisms of action, and predictively shaping protein structures and potential target interactions [36]. For instance, in the case a drug under investigation acts differently in vivo compared with what expected, AI is able to generate and analyse large amounts of data to improve the understanding of the medicine’s mechanism. Greater knowledge of the mechanism of action and interaction with the target can help increase the likelihood of testing drugs in populaces who take advantages from them [37].

Management of Clinical Trial Participants

Managing participants in clinical experimentation involves selecting target patient populations, recruiting patients, and retaining participants. Participant dropouts and non-adherence to protocol standards may bring studies to overcome allowed timing or costs or abort to generate usable data. In fact, it was assessed that among 33.6% and 52.4% of phase 1–3 clinical studies fail to proceed to the next experimental phase, reducing the probability of a drug tested in phase I gaining approval by 13.8%. ML approaches may ease the more effective and equitable recognition, enrolling, and retention of participants through rapid analysis of extensive databases of prior research [38]. Moreover, machine learning can identify patterns which can be employed to choose phenotypes of patient with a higher likelihood of benefiting from the tested drug. Unstructured data is condemnatory for phenotyping and selecting cohorts more representative, leading that including further patient data is a critical step toward selecting strong and cross-Sect. [39, 40]. Two general overtures are available to increase retention of participants and the adherence of protocol utilizing machine learning-assisted procedures. The first is to employ ML to identify and intercede with people who highly risk to be not compliant with the study [41]. The second overtures involve using machine learning to reduce the study burden on participants, for instance, during data collection, and thus improving their journey. In fact, patients generate critical content for clinical studies even outside the clinical trial context, such as reporting side effects, which ML can process and incorporate into the clinical study [42].

Recruitment

When a specific cohort is identified, there has been promising evidence for natural language processing identifying patients who match the desideratum phenotype, which or else would require a substantial human resource investment [19].

Identification of Endpoints

AI can further be used on data processing for outcome selection. The current approach is performed manually by a committee of clinicians and involves a high-intensity sorting and classification of events, which aligns perfectly with the capabilities of artificial intelligence, resulting in reduced time and costs [43].

Data Analysis

Data gathered in clinical studies, registries or clinical practices are important sources for generating hypotheses and risk assessment where machine learning is particularly suited.

Ethical Implications

The application of AI in healthcare raises a series of ethical and legal issues. In the past, healthcare decisions were made by healthcare professionals themselves. The fact that intelligent technologies autonomously make decisions or assist in various processes raises questions of responsibility, transparency, consent, and confidentiality. The most significant challenge to address is certainly transparency, as most AI algorithms are difficult to understand or interpret. The second aspect is the moral obligation, i.e., the obligation to accept responsibility for one’s actions, which is hard to define in the case of AI usage. Moral obligation could be attributed to both the developer and the operator who decides to use artificial intelligence. Based on the discussion, in order to mitigate risks and maximize the potential of AI in healthcare, the World Health Organization (WHO) has provided principles as a basis for control and governance, summarized in Fig. 3 [44].

Fig. 3
figure 3

Fundamental ethical principles for the correct management of AI in healthcare.

Full size image

Main Challenges

The main challenges for the correct use of AI in healthcare are linked to critical aspects arising from current ML approaches, which heavily rely on training data. Even in unsupervised learning, theoretically independent from direct human monitoring, a connection to supervision processes emerges. This raises significant concerns regarding data quality, security, and privacy, reinforcing the need for in-depth analysis [46]. Regarding public health research and practice, big data poses three key issues: the risk of unintentional disclosure of personally identifiable information (e.g., through the use of online tools), the potential increase in data dimensionality making it challenging to determine ‘deductive disclosure’ of personally identifiable information, and the challenge of identifying and maintaining ethical research standards in the face of emerging technologies that can alter generally accepted privacy norms (e.g., GPS, drones, social media, etc.) [47]. Data quality emerges as a crucial element to ensure the validity of ML models, as the reliability of results is closely connected to the quality of data used in the training process. Another area of concern is the risk associated with potentially biased or incorrect statistical models. It becomes necessary to conduct audits on the ‘black box,’ i.e., understand the internal functioning of models. Furthermore, there is a need to improve system interoperability, aiming to ensure harmonious collaboration and compatibility among them. This approach is crucial to mitigate risks and promote greater transparency in the use of ML/DL models in healthcare contexts [45].

Regulatory Framework in the United States

The regulatory process for AI-based medical devices in the United States is primarily managed by the Food and Drug Administration (FDA), the government agency responsible for regulating medical devices, medicines, and food in the United States. In April 2019, the U.S. Food and Drug Administration (FDA) proposed a regulatory framework for AI in medicine [46]. Here is how the regulatory process for AI-based medical devices in the United States works:

  • Device Classification: The first step involves the classification of AI-based medical devices. The FDA classifies medical devices into three main classes: Class I (low risk), Class II (moderate risk), and Class III (high risk). The class determines the level of required regulation.

  • Premarket Authorization Request: For Class II and Class III devices, the manufacturer must submit a Premarket Approval (PMA) or a 510(k) premarket authorization request to the FDA, depending on the level of risk. The request must include clinical data, evidence of device safety, and effectiveness [47].

  • FDA Evaluation: The FDA carefully evaluates the authorization request, examining the data presented by the manufacturer. This evaluation includes the analysis of clinical data and information on device safety and effectiveness. The FDA may request additional information or conduct inspections at the manufacturer’s facility.

  • Approval or Rejection: After completing the evaluation, the FDA can decide to approve the AI-based medical device, reject the approval, or request additional data or modifications. If approval is granted, the device can be marketed in the United States.

  • Post-Market Monitoring: After marketing, the FDA continues to monitor the safety and effectiveness of AI-based medical devices through adverse event reports, inspections, and periodic reviews.

It is noteworthy that the FDA is constantly working to develop specific guidelines and regulations for AI-based medical devices to address the unique challenges posed by these technologies. The agency has developed the “Artificial Intelligence/Machine Learning (AI/ML)—Based Software as a Medical Device (SaMD) Action Plan”, a new action plan which outlines activities for regulating AI-based software as Medical Devices (SaMD) [48]. The action plan emphasizes the need for specific regulatory attention given the rapid and constant evolution of these tools. The document illustrates the agency’s effort to reimagine a premarket review approach based on the total product lifecycle (TPLC). The FDA regulates products to ensure they are safe for consumers. In the healthcare sector, machine learning tools can pose safety concerns, as neglecting potential negative impacts can increase the risk and danger for already marginalized and discriminated groups in healthcare. Considerations of health disparities should not be out of scope or an optional dimension to consider when developing machine learning tools for medicine. To that end, considerations of health disparities can be integrated into the FDA’s AI/ML regulation both in the pre-market and post-market phases. In addition to conventional approaches, the FDA has outlined a specific pathway for SaMD technologies in its Digital Health Innovation Action Plan, offering a streamlined path to precertification, certifying the developer themselves rather than the product [49]. The TPLC facilitates continuous monitoring of the SaMD product throughout its development and post-market performance, as periodic disclosure of performance data to the FDA has been mandated. As a result, the agency aims to strike a balance between reducing the overall regulatory burden while ensuring efficacy and safety.

Regulatory Framework in the European Union

In European Union (EU) AI-based medical devices are not approved by a centralized agency. The regulatory process is as follows [50, 51]:

  • Low-risk devices (Class I): For devices considered low-risk (Class I), the manufacturer is responsible for ensuring that the product complies with regulations. Pre-market approval by a central regulatory authority is not required.

  • High-risk devices (Classes IIa, IIb, and III): Devices considered high-risk are overseen by private Notified Bodies, which are accredited and authorized independent organizations responsible for evaluating and approving such devices. These bodies verify that the devices meet European regulatory requirements before they can be placed on the market.

This European system relies on third-party assessment by Notified Bodies to ensure compliance with regulations before devices can be placed on the market. Therefore, EU delegates a significant portion of approval responsibility to accredited private entities, while the FDA in the United States plays a central role in the approval of medical devices. The differences between the European and American regulatory processes are illustrated in Table II.

Table II Differences Between the American and European Regulatory Processes
Full size table

The European regulatory system for AI-based medical devices has some critical issues and challenges, including:

  • Variation in Notified Body procedures: since Notified Bodies are private and operate in different European countries, procedures and standards may vary between countries. This can lead to a lack of uniformity in the assessment and approval of medical devices in Europe.

  • Need for adequate training and resources: Notified Bodies must have the necessary resources and expertise to conduct accurate assessments and approve medical devices. The quality of assessments may vary depending on the involved body.

  • Risks to patient safety: proper evaluation of medical devices is crucial to ensure patient safety. Any errors or deficiencies in the assessment procedures could pose risks to patients.

  • Delayed regulation: the European approval process involving Notified Bodies may be slower than that in the United States, potentially resulting in delays in European patients’ access to new medical technologies.

  • Coordination and standardization: variation in processes and standards among different European countries can create complexity and challenges in marketing medical devices throughout the European Union.

  • Post-market monitoring: after marketing, adequate monitoring of medical devices is essential to ensure long-term safety and effectiveness.

Overall, these challenges are related to the balance between the need to ensure stringent regulation for the safety and effectiveness of medical devices and the need to provide timely access to new medical technologies for patients. EU is addressing these challenges by improving coordination among Notified Bodies and establishing clearer guidelines for the evaluation and approval of AI-based medical devices [52].

Discussion and Conclusions

As illustrated in Table II, the regulatory processes for AI-based medical devices in the USA and the EU exhibit both similarities and differences, reflecting distinct approaches to ensure the safety and effectiveness of healthcare technologies. In the USA, the FDA governs the regulatory landscape for AI-based medical devices. The process involves classifying devices based on risk, with subsequent premarket authorization requests for Class II and III devices. The FDA evaluates clinical data and safety information, either approving or rejecting the device. Post-market monitoring is crucial, ensuring continued safety and effectiveness. The FDA is actively adapting to the challenges posed by AI, emphasizing a lifecycle approach, continuous monitoring, and efforts to streamline regulatory pathways. In contrast, the EU has a decentralized approach. Low-risk devices undergo self-assessment by manufacturers, while high-risk devices rely on private Notified Bodies for assessment and approval. Challenges arise from potential variation in procedures and standards among different countries, necessitating coordinated efforts. Potential delays in approval and risks to patient safety are concerns, prompting the need for robust post-market monitoring and improved coordination among Notified Bodies. The key distinction lies in the central role of the FDA in the USA, acting as the primary regulatory authority, while the EU delegates approval responsibilities to accredited private entities. This difference may contribute to variations in regulatory speed and potential challenges in uniformity across Europe. Both regulatory systems cope with the balance between stringent safety measures and the timely introduction of innovative medical technologies. The FDA’s emphasis on continuous monitoring aligns with the EU’s acknowledgment of post-market surveillance’s importance. Addressing these challenges requires ongoing efforts in both regions. The FDA is developing specific guidelines for AI-based devices to ensure rapid adaptation to evolving technologies. In the EU, there is a need for improved coordination among notified bodies and clearer guidelines. Furthermore, overall the integration of artificial intelligence into clinical practice presents many challenges. Acceptance and trust among healthcare professionals are pivotal for the successful integration of AI in healthcare. In addition, ensuring interoperability is a critical aspect. AI systems must seamlessly integrate with existing electronic health record systems, ensuring a smooth flow of data and facilitating easy access for professionals to the insights generated by AI. In the ever-evolving landscape of data privacy and security, AI algorithms must adhere to stringent regulations and robust information security practices. It is also becoming increasingly imperative to establish reliable evaluation criteria, to ensure that AI solutions meet clinical effectiveness and safety standards before being deployed for patient care. Furthermore, addressing bias in both data and decisions is a crucial ethical consideration for AI systems. Moreover, the implementation of AI in healthcare environments demands substantial investments in infrastructure and training, posing challenges for facilities with limited resources. The ethical complexities surrounding AI in medicine extend to the evolving landscape of legal responsibility. Clarity in determining legal responsibility for AI errors or incorrect decisions is still under development, and defining regulations and standards in this domain is crucial. To achieve all these aims is crucial to balance collaboration between regulatory bodies, healthcare professionals, and technology developers to create a robust and adaptive regulatory framework. Addressing these challenges requires a combination of technical expertise, resources, international collaboration, and a smart and balanced regulatory approach [53].