Can We Trust AI With Our Health?
- Claire Lee
- Aug 8
- 3 min read
Introduction
Today, artificial intelligence (AI) algorithms are used in hospitals to diagnose medical conditions more than ever, evolving into highly collaborative tools that consistently outperform humans. Not only are they incredibly capable of detecting disease early, but they can efficiently create personalized treatments for patients. But as healthcare systems continue to adopt these technologies, questions and concerns arise about their impact on human care. AI’s occasional inaccuracies create liability issues for hospitals, and bills are increasingly being introduced to regulate and restrict the use of AI-driven diagnosis.
AI algorithms are undoubtedly becoming an important part of modern healthcare, but their growing use should not be confused with perfection. The technology is only as reliable as the data, testing, oversight, and people behind it. So although AI algorithms are undoubtedly here to stay, it is important to shed light on their potential drawbacks.
Hallucinate Data
One prominent flaw of AI algorithms is their inaccuracies, which can have especially serious consequences in a hospital setting.
In July 2026, former Mayo Clinic research director and AI compliance lead Traci Tamiko Eto filed a lawsuit against Mayo Clinic for allegedly burying serious flaws in an internal AI tool and firing her after she raised the alarm. She alleges that Mayo’s digital assistant tool, MAYA, “mischaracterized outcomes, deleted unfavorable results, and used unauthorized software”. Additionally, the lawsuit claims 10 separate whistleblower reports raised similar concerns, alleging researchers sought to conceal a 67% error rate. In other words, employees working on the AI assistant allegedly knew it could make errors roughly two-thirds of the time.
This is not the first high-profile complaint about medical AI accuracy. In November 2023, UnitedHealth was sued for improperly denying claims using an artificial intelligence algorithm, nH Predict, that critics say has a 90% error rate. According to the complaint, the elderly are “prematurely kicked out of care facilities nationwide or forced to deplete family savings to continue receiving necessary medical care, all because [UnitedHealth's] AI model 'disagrees' with their real live doctors' determinations."
These inaccuracies are particularly concerning, as the high error rates could affect the quality of care patients receive. What’s even more alarming is that hospital workers are continuing to use AI algorithms despite knowing about such a high error rate, which raises questions about whether these algorithms are truly beneficial for patients and hospitals.
Overreliance on AI
Even an accurate AI system can become dangerous if healthcare professionals rely on it too heavily. Physicians could gradually become accustomed to accepting AI recommendations rather than independently evaluating each patient's circumstances. This creates the possibility of "automation bias," in which people place excessive trust in a computer-generated recommendation simply because it came from a technological system. This is especially concerning in complex cases where patients may not fit neatly into patterns found in a database. For example, two people with the same diagnosis may have different medical histories, medications, lifestyles, genetic factors, and treatment responses.
This overreliance may even erode the skills of existing and future doctors. A survey study by Ahmad and colleagues on university learners found that using AI significantly diminishes human decision-making and fosters laziness. Such shifts in mindset and attitudes could take root in current training programs, producing graduates who may go on to lead future training efforts. This could ultimately perpetuate a cycle that leads to the deterioration of expertise, creating an abyss of lost clinical skills.
AI is intended to be an assisting tool for doctors, not a replacement. If doctors rely too heavily on AI, it can cause harm for the patients, considering the tendency of AI to make mistakes.
Liability and Accountability
Another important issue is determining who is responsible when an AI-assisted medical decision harms a patient. If a doctor follows an AI recommendation that turns out to be incorrect, responsibility could potentially involve the physician, hospital, software developer, or another organization. This creates a difficult legal question: Should a doctor be held responsible for trusting a system that the hospital approved? Should the hospital be responsible for deploying an inadequately tested system? Or should the company that developed the algorithm be held accountable?
Liability becomes complicated because hospitals cannot simply shift responsibility to an algorithm; an AI system cannot be sued, disciplined, or held morally accountable in the same way that a human or organization can. Ultimately, healthcare institutions need clear policies establishing who has the authority and responsibility to make the final medical decision.
Taking all these factors into consideration, it is important to view AI-driven diagnosis as a supportive tool rather than a replacement for doctors.
References
Comments