As the new year begins, several new laws regulating artificial intelligence (AI) in the healthcare industry have gone into effect, prompting renewed debate over how the rapidly expanding technology should be used and constrained in medical settings.
Over the past decade, AI has become increasingly embedded in daily life, and healthcare has emerged as one of its most promising applications. With its quick access to a web of medical information, seemingly reliable advice, and assured decision-making, AI tools offer answers without the time, cost, or logistical barriers of seeing a doctor in person.
For many patients, that accessibility has become especially appealing amid growing frustration with the U.S. healthcare system, which, according to Gallup, 70% of Americans view as either in crisis or facing major problems.
“Medicine has been the top of my mind, and AI too, as the top of medicine’s mind, because I’ve been sick. And AI has given me more answers than anything because I’ve had to wait three months to see doctors,” said Kate Large, a patient who uses AI for research on her health.
That experience is far from isolated. According to OpenAI, of its over 800 million regular users, more than 40 million prompt ChatGPT about their healthcare every day, so many that the company recently released ChatGPT Health, a new feature designed specifically to address these inquiries.
“I feel like it’s not very different from before AI, when people would be doing their own online searches anyway. People would always come in with their own research of what they’d read about or looked up on the internet,” said Dr. Lailey Oliva, an internal medicine physician at Sutter Health.
However, unlike traditional internet searches, AI tools often present their information with authority and even a tone of empathy, despite not being licensed medical professionals or even people. When patients are trying to judge the trustworthiness of these chatbot responses, this false sense of credibility can lead to real risks.
“What happens if you are asking ChatGPT to help you write an essay, and it gets some facts wrong? You’ll just correct it; no big deal. But what happens if you’re using ChatGPT to answer a health question and it gets some wrong? That could have really, really big implications on a person’s health,” said Nitya Thakkar, a third-year Ph.D. student in computer science at Stanford who is researching AI in health and medicine. “So I think we’re all realizing as a society that we need to be very careful in the health domain. We need extra guardrails in place because, as great as these models may be, there’s a lot of harm they can cause.”
As AI systems become more sophisticated and more convincing, misinformation and misplaced trust have become major concerns. In response, many of the new bills that have just taken effect, led by California, directly address these issues, targeting how AI is presented and used in medical contexts.
“I think that when you’re asking an AI, you’re automatically assuming that it’s going to be correct, which I think is the point of this legislation: to change that idea. It can make mistakes, like how people also make mistakes, even surgeons, and we need to remember that,” Large said.
As a result, part of the recent legislation, Assembly Bill (AB) 489, is designed to keep patients cautious about AI’s role in their medical lives. It prohibits developers and companies from implying that their AI systems provide professional or licensed medical advice by restricting the use of titles, letters, phrases, or design elements, such as “doctor” or “M.D.,” that could possibly mislead users into believing the chatbots are qualified healthcare providers.
“It’s a good way to remind people that these are models; they’re all working on your computer, they’re not real people, and they’re not doctors who’ve been through four years of medical school, residency, and so much training,” Thakkar said. “There’s a big anthropomorphization of these language models. When the AI speaks to you with empathy and uses ‘I’ statements, you start to interface with it like it’s a real doctor. But these bills will help people remember that it is a machine. It can give you information and help you answer some questions, but at the end of the day, you should really talk with a doctor to get an actual, affirmative medical diagnosis.”
Beyond how AI presents itself to patients, California lawmakers have also moved to regulate what happens behind the scenes. Gov. Gavin Newsom signed AB 2013, another bill that went into effect on Jan. 1, requiring developers to share the data used to train their AI systems. For healthcare-related AI, this would likewise require those developers to show what data supports their clinical evaluations and recommendations.
“I think the thing that we worry about the most here is the much greater potential to harm the human body,” said Michelle Mello, Ph.D., J.D., a Stanford professor of law and of health policy, whose research focuses on ensuring responsible use of AI in healthcare. “It’s one thing to have AI give you bad investment advice, or you don’t get hired because of an AI hiring system, and that’s unfair, but we’re talking about uses of AI that could kill you. So the stakes are really, really high, and our tolerance for how much error we would permit is lower as a result.”
Several other states, including Illinois, Nevada, Utah, and Texas, have passed similar legislation mandating transparency and disclosure for chatbots used in clinical settings.
“They have been very moved by stories. There have now been two or three really in-depth pieces of journalism focusing on teens who have died because they’ve gotten suicide advice or they have become mentally ill interacting with AI, and also some stories about adults who developed addictions to chatbots,” Mello said. “So, although there have not really been any horror stories about over-reliance on a medical AI tool, I think there is still an overall concern. The medical profession itself is lobbying legislatures about this. They don’t like it.”
These resulting state-level efforts come amid growing federal pushback. In late 2025, President Donald Trump issued an executive order seeking to limit what his administration has described as “onerous” state AI regulations, implying that such laws will face legal challenges.
“AI developers really hate the idea of state regulation of their products because it subjects them to potentially 50 different regulatory regimes, and that’s just really hard to manage. So they have been lobbying pretty hard to have Congress, and now the president, say that states cannot make these kinds of regulations,” Mello said. “I understand that concern. It’s no fun to have a patchwork of regulations. But it’s one thing to say that a federal approach would be better; it’s another thing to actually get one because neither our Congress nor our executive branch is remotely interested in trying to regulate AI right now.”
That stance became clearer recently on Jan. 6, when the Food and Drug Administration (FDA) announced it would align with the order by ending its regulation of digital health products and refraining from imposing new restrictions on “decision-supporting tools,” such as AI systems used for health inquiries, as the states have been doing.
“I hesitate when there’s no federal regulation on these things that are extremely impactful on people’s lives,” Large said. “But I’d rather have, at minimum, some kind of regulation, whether it comes from the state, than no regulation, as we’re starting to build something like this.”
FDA Commissioner Marty Makary defended the shift, arguing that deregulation is a necessary change to attract investment and innovation. He said at the conference announcement that the agency needs to move “at Silicon Valley speed.”
Thakkar, however, doesn’t see any of the new state regulations as possible obstacles to her research, maintaining that these safeguards are necessary for AI in healthcare regardless.
“I don’t think that they’re hindering innovation or going to slow it down,” Thakkar said. “I think that there’s definitely a balance between innovation and safety, but I also think we need to remember that this domain is different, so its timescale can’t necessarily be compared to that of regular language models because there are so many greater implications here. Safety is also important in other domains, but the stakes are much lower than here.”
Large echoes her perspective, stressing the need for caution when using AI for personal health insight.
“As a patient and user of AI, it’s extremely important to get the facts right,” Large said. “I’ll always use AI for efficiency since there are such long wait times for traditional medical appointments and procedures, but I will still fact-check its responses regardless of how much AI regulation there may be. For me, it won’t completely replace traditional medicine, but it’s a tool that guides me in advocating for my medical care through preparing me with the right questions to ask.”
As AI continues to expand rapidly in healthcare, its role has shifted from whether it can be used to how it should be controlled. While the technology promises efficiency and access in a strained medical system, it also raises concerns about accuracy and trust, which are especially critical to address in a field where errors can have dangerous consequences.
“Even outside of healthcare, we’ve all heard some pretty concerning anecdotal stories of how AI has really steered people, who might’ve been more vulnerable, down certain roads that are dangerous,” Oliva said. “So why wouldn’t there be a need for regulation? Especially at this juncture, where it seems like AI is starting to take over all these different facets of all of our lives, I feel that it’s better to be proactive on safety, as opposed to waiting for bad things to happen and then backtracking.”
