Agentic AI and Multi-Agent Collaboration in Healthcare: A Comprehensive Survey of Architectures, Clinical Safety, and Future Directions

Abstract

In the healthcare and clinical domain, artificial intelligence (AI) is evolving from earlier models that primarily predicted outcomes or generated content toward agentic AI systems that demonstrate the capability to make decisions and complete tasks autonomously. Previous AI research contributed significantly to disease identification, deep learning applications, large language models (LLMs), and generative AI, yet these systems function primarily as assistive tools, generating text or predictions without directly interacting with clinical infrastructures. Recent research trends are therefore increasingly oriented toward agentic AI systems that extend beyond traditional predictive and generative model performance. This manuscript provides a detailed review of the current state of agentic AI, starting with the evolution of AI and the concept of a medical agent. A medical agent refers to an intelligent AI system designed to assist in clinical or administrative tasks by analyzing data, supporting decision making, and interacting with healthcare environments. Its underlying agentic AI architecture integrates planning, memory, reasoning, and environmental interaction to enable autonomous tool use, multi-agent collaboration, and continuous perception–decision–action loops across diverse healthcare applications and clinical workflows. The review further examines safety mechanisms, including human-in-the-loop oversight, self-verification strategies, and regulatory alignment frameworks, designed to ensure reliability, accountability, compliance, and safe deployment in regulated healthcare environments. Our findings indicate that a large number of AI agents have been introduced in various manuscripts for healthcare applications; however, fully autonomous systems remain challenging to achieve, as AI still faces limitations related to reliability, interpretability, data dependency, and integration within complex clinical workflows. In response, this survey shifts the focus from task-specific model performance to system-level autonomy and workflow orchestration, providing a structured foundation for understanding the design, deployment, governance, and limitations of agentic AI systems in modern healthcare ecosystems.

I. Introduction

Today's clinical facilities increasingly depend on AI to enhance diagnostic accuracy, optimize workflows, and improve patient outcomes. The evolution of AI in healthcare can be broadly categorized into three major waves: predictive AI, which focuses on pattern recognition and outcome prediction using statistical models and deep learning — such as convolutional neural networks in medical imaging (X-ray, CT, MRI) and recurrent and long short-term memory networks in physiological time-series signals (ECG, EEG) — followed by generative AI, which focuses on content generation, representation learning, and knowledge synthesis through large language models, generative adversarial networks, and transformer-based generative architectures.

The third and emerging wave is agentic AI, which represents a fundamental shift from predictive and generative paradigms toward autonomous, goal-driven systems capable of acting within complex clinical environments. Agentic systems are designed to perceive context, reason over time, plan multi-step actions, and execute decisions across interconnected healthcare workflows by integrating large language models with planning, memory, tool use, and environmental interaction, enabling direct engagement with hospital information systems, appointment scheduling platforms, billing infrastructures, clinical alerting pipelines, and pre-screening processes. Despite the impressive performance of modern generative models on medical examinations and clinical reasoning benchmarks, a critical limitation known as the "agency gap" remains: current LLMs largely function as passive responders that generate recommendations or textual outputs without the capability to autonomously coordinate tasks, adapt to dynamic clinical states, or execute multi-step actions within real-world healthcare systems.

This paper goes beyond a conventional literature survey by offering a well-organized, application-oriented framework: the definition of the medical agent and its core components, levels of autonomy from Level 0 to Level 5, single-agent and multi-agent architectures, their clinical, administrative, and research applications, safety mechanisms, technical and practical challenges, and actionable recommendations for future research. The review followed a PRISMA-style semi-systematic methodology covering 100 research papers indexed primarily in IEEE, Springer, Elsevier, PubMed, arXiv, MDPI, and ACM, documenting their distribution across databases, publication years, study types, clinical risk levels, and deployment status.

II. The Medical Agent and Levels of Autonomy

A medical agent is defined as an intelligent AI system designed to assist in clinical or administrative tasks by analyzing data, supporting decision making, and interacting with healthcare environments. Unlike predictive models that learn correlations between clinical variables and target outcomes, or generative models that synthesize content, the medical agent operates within a continuous perception–decision–action loop: it perceives the clinical context, reasons over time, plans multi-step procedures, and executes them through interconnected tools and systems.

Its core agentic architecture comprises integrated components: multimodal perception (images such as MRI, CT, and ultrasound; time-series signals such as heart rate, blood pressure, and oxygen saturation; clinical text), planning and reasoning (decomposing clinical goals into executable steps), memory (short-term for the current episode and long-term for extended patient experience), tool use (APIs, hospital systems, diagnostic devices), and environmental interaction with continuous feedback loops. As the agent advances from basic text processing toward patient-centric situational awareness, it gains the ability to detect early deterioration and flag suspected tumors, fractures, hemorrhage, or subtle anomalies for clinician review.

The framework classifies six levels of autonomy: Level 0 (no AI — human-only care as a reference baseline), Level 1 (AI as a suggestive assistant that recommends without executing), and Level 2 (AI with continuous human oversight, where the clinician reviews every action). These lower levels represent the currently dominant state of clinical practice, in which all diagnostic, therapeutic, and operational decisions remain in the hands of healthcare professionals with advisory support from systems.

The higher levels comprise: Level 3 (AI with conditional human backup, intervening when confidence bounds are exceeded), Level 4 (AI with human approval only for proposed actions), and Level 5 (fully autonomous AI that plans, executes, and adapts without any human involvement — a level that remains largely theoretical in healthcare for ethical, legal, and safety reasons). The discussion concludes that balancing autonomy and risk is the essence of safe deployment: the broader the agent's authority, the greater the need for alignment, monitoring, and accountability mechanisms, because healthcare decisions directly affect patient safety and cannot tolerate autonomous execution errors.

III. Architectures and Frameworks

Single-agent systems operate within a closed feedback loop in which one LLM-based agent executes the task end to end, suiting narrow, well-scoped tasks. Examples include enhancing computer-aided diagnosis (CAD) outputs — diagnosis, lesion segmentation, and report generation — by summarizing information in natural language, which improved diagnostic performance and produced patient-friendly interactive systems. Their goal is to translate complex medical information into accurate, readable, patient-friendly documentation while maintaining safety through clinician oversight.

Multi-agent systems are distributed systems of specialized agents collaborating toward shared clinical objectives through structured coordination — central supervision, inter-agent debate, or sequential orchestration. For example, a radiology agent may interpret imaging scans and report findings, which are then evaluated by an oncology agent against clinical guidelines to propose management options before being presented to a human physician for final decision-making and oversight. This reflects a broader shift in radiology from single AI systems toward multi-agent networks capable of autonomous reasoning and coordinated decision-making, where agents build upon one another's outputs. In comparison: single-agent systems are narrow in scope with independent execution, while multi-agent systems handle complex multi-step workflows with structural coordination — at the cost of greater coordination complexity and wider scalability demands.

The review also addresses privacy-preserving architectures and threat mitigation: communication protocols among agents raise security issues requiring zero-trust architectures as a reliable backbone for agentic ecosystems, alongside data protection, ethical review, and controlled dataset distribution. Resilience — the ability to produce consistent results despite disruptions such as broken sensors or network failures — is a must-have trait of clinical systems rather than a bonus, particularly when systems control critical infrastructure or perform medical operations.

IV. Applications

In the clinical domain, research has highlighted the potential of AI-powered agents in autonomous data collection, conversational interaction, clinical task coordination, and multimodal medical data analysis. Examples include the Medical Interview Assistant (MIA), which collects patient history and answers predefined questions; interview-style conversational frameworks for attitude and behavior tracking; a collaborative multi-LLM clinical agent for department guidance, preliminary consultation, and diagnostics; and imaging agents that interpret scans and report findings. Recent studies have also evaluated usability, feasibility, real-world clinical performance, and efficiency gains in healthcare administration.

On the administrative side, agents support clinical documentation, operational coordination, and workflow management, intersecting with broader innovations including trial management and predictive analytics. The MedScrubCrew medical multi-agent framework presents a prominent example, integrating stable matching, knowledge graphs, and LLM agents to automate appointment scheduling and triage classification, demonstrating improved operational efficiency and context-aware allocation. Other works examine the impact of digital transformation initiatives on healthcare operations.

Research agents operate in simulated clinical environments across specialties, modalities, and languages, with systematic evaluation benchmarks for LLM agents. Results show that current LLM agents still struggle in multi-turn scenarios, motivating further research into improving interactive medical decision-making. A Wizard-of-Oz study investigates how data analysts respond to AI planning assistance, characterizing which kinds of suggestions are helpful and how they influence the analysis workflow, initiative, and decision-making.

V. Challenges of Action: Hallucination, Explainability, Security

"Action" is the core challenge: systems that generate recommendations may execute actions with safety-critical clinical consequences. Hallucination rates range from 5% to 30% in healthcare settings, and clinical agents may misinterpret patient information, unit conversions, or dosing instructions, potentially leading to dangerous medication recommendations. One documented failure occurred when an automated system mistranscribed the medication "D10W" as "V10W," triggering incorrect data extraction that could have led to dangerous administration errors. Agents may also provide false reassurance or respond to the "wrong query" when a knowledge source lacks specific information.

Explainability, transparency, and interpretability are non-negotiable requirements in regulated settings: agent decisions must be traceable and justifiable to clinicians, patients, and regulators. Yet the dynamic, adaptive behavior of agentic systems complicates monitoring and control, particularly in multi-agent environments where outputs interact and accumulate. The literature proposes self-verification strategies and detailed action ledgers that record every tool invocation and decision step transparently, ensuring that human accountability remains central to all clinical outcomes.

Agentic AI significantly expands the attack surface because agents autonomously access APIs, tool systems, memory, and external environments. In healthcare, breaches may expose sensitive patient data and even enable remote manipulation of medical devices. Vulnerabilities include prompt injection, adversarial tool manipulation, and multimodal jailbreak attacks, and in multi-agent systems the compromise of one agent can cascade across the whole system. Secure, scalable architectures, prevention of catastrophic forgetting, and continual learning without destabilizing system behavior therefore remain open problems.

VI. Governance, Alignment, and Coordination

Agentic systems raise risks of bias, unfairness, and poor data quality that can contribute to disparities in care delivery, delayed interventions, and eroded trust in AI-assisted healthcare. Multiple fairness-oriented strategies have been proposed to counter these risks, including adversarial debiasing techniques that reduce the influence of protected demographic attributes during model optimization, alongside the development of diverse, representative datasets with fairness-aware fine-tuning to reduce the amplification of discriminatory patterns and improve clinical fairness.

On accountability, ethics, and legal responsibility, the problem of automation bias stands out: clinicians under workload pressure and documentation burden may over-rely on system outputs and endorse incorrect recommendations without sufficient scrutiny. Recent literature calls for an "AI responsibility chain" and harm-traceability frameworks with detailed action ledgers. Challenges of human alignment, control, and monitoring include goal misalignment — when agents optimize proxy objectives differing from the intended clinical outcome, such as prioritizing operational efficiency over patient safety — and monitoring complexity in multi-agent environments, emphasizing the role of the doctor-in-the-loop and evaluation in real-world deployment.

Memory, long-horizon planning, and recovery failures add distinct challenges: emerging approaches transition from short-term episodic memory toward persistent memory architectures, including vector-based stores that preserve patient experience across extended clinical timelines. Coordinating multiple agents introduces instability, communication bottlenecks, and emergent behavior, with documented inter-agent error cascades, debugging complexity, and agent misalignment. Finally, social acceptance and workforce impact raise questions of trust, adoption, and the reshaping of clinical roles around human–AI collaboration.

VII. Future Directions and Review Limitations

The review identifies actionable research directions: scalable architectures and secure, scalable memory with prevention of catastrophic forgetting and continual learning, safe deployment with verification guarantees, universal ethical frameworks with auditability mechanisms and regulatory pathways for agentic systems — especially those with continual learning and tool-use autonomy — integration of agentic intelligence into regulated clinical workflows, and alignment of goals with human values. Given the rapidly evolving nature of agentic AI in healthcare, such reviews should be updated regularly to keep pace with the changing landscape.

The review acknowledges its limitations: some included studies are restricted to English-language text-based LLMs without multimodal agentic loops, post-generative clinical integration and deployment challenges receive limited attention, most studied systems remain at the research-only stage with no real clinical deployment, and some surveys lack coverage of domain-specific medical execution. These limitations precisely locate the gaps that future research must address rather than diminishing the value of the framework presented.

VIII. Conclusions

This review has outlined the emergence of artificial intelligence technology within medicine through predictive and generative approaches toward the use of agentic AI that allows autonomous reasoning, planning, and acting. We defined the medical agent and examined its core architectural components — perception, planning, memory, reasoning, tool use, and environmental interaction — as well as the degrees of autonomy in contemporary solutions, from single-agent to multi-agent medical systems, with a comparative analysis of definition, architecture, clinical scope, and coordination strategy dimensions, its applications across clinical, administrative, and research domains, and safety mechanisms from human oversight to self-verification and regulatory alignment.

Our findings indicate that a large number of AI agents have been introduced in various manuscripts for healthcare applications; however, fully autonomous systems remain challenging to achieve due to limitations in reliability, interpretability, data dependency, and integration within complex clinical workflows. This survey therefore shifts the focus from task-specific model performance to system-level autonomy and workflow orchestration, providing a structured foundation for understanding the design, deployment, governance, and limitations of agentic AI systems — and a roadmap toward safe, trustworthy human–agent collaboration in modern healthcare ecosystems.