When lawyers begin asking questions that no one can clearly answer, a certain kind of tension arises in the room. You can sense it when you walk into any serious healthcare compliance meeting these days. Someone will always bring up the topic of what happens when an AI agent makes the incorrect call, somewhere between the open laptops and the half-finished coffee cups. Not a bug in the software. Not a chart that was misread. An autonomous system that operates mostly outside the bounds of any appropriate regulatory framework, acting with real-world consequences.
Healthcare attorneys are currently in that predicament. Agentic AI, or systems that can make decisions and carry out tasks with little human supervision, is no longer a pipe dream. It is being tested in various health systems for everything from prescription management to triage support, and the legal framework for it is, to put it mildly, lacking.
The founder of Metaverse Law, Lily Li, has been keeping a careful eye on this area and is not overly comforted by the rate of regulatory clarity. Her worry isn’t hypothetical. She has cited instances in which an AI agent mishandles a triage line in the emergency room or improperly refills a prescription; in these cases, the error has a body count rather than just a compliance infraction. The statement alone ought to be keeping hospital general counsels up at night. “Even in situations where the AI agent makes the right medical decision, but a patient does not respond well to treatment,” she has argued, “it is unclear whether existing medical malpractice insurance would cover claims if no licensed physician was involved.”
The state of affairs at the federal level is not very stable. The FDA has been releasing draft guidelines on AI-enabled medical devices that address transparency and bias mitigation throughout device life cycles as part of its “software as a medical device” classification framework. As far as it goes, the work is thoughtful. However, agentic AI frequently completely rejects the device-classification model; it functions more as a decision-maker than a tool, and this distinction is crucial when attempting to assign liability.
As state-level regulations proliferate in disparate, overlapping directions, President Trump’s early executive order on AI effectively destroyed the oversight framework that had been carefully put together under the Biden administration, leaving businesses navigating a looser federal environment.
While not exactly a model, the EU’s strategy provides an intriguing contrast. Healthcare AI is classified as high-risk under the AI Act, which imposes more stringent requirements for data governance and human oversight. An additional layer of interoperability and privacy considerations is added by the European Health Data Space, which was formally adopted earlier this year. It is perhaps more ambitious and more structured. These regulations won’t fully take effect until at least 2027, so whether they’re more practical in practice is still up for debate.
Observing all of this, it’s amazing how rapidly technology has surpassed the organizations designed to control it. Health systems are under tremendous financial strain; administrative overhead is crippling, physician burnout is real, and the promise of AI-driven efficiency is truly compelling. Some hospital systems may have used agentic tools without thoroughly stress-testing the liability exposure, not because they were careless but rather because there was no guidance available for consultation.
Before the regulatory picture becomes clear, health system attorneys must create governance frameworks that are flexible rather than adhere to a set set of rules. This entails recording human oversight procedures at each handoff, mapping every instance in which an AI agent interacts with a clinical decision, and auditing whether current malpractice and liability coverage even considers autonomous AI action. In many instances, the legal exposure in this case has already occurred. There is no gray area in the distance. The industry is already standing on this floor.
