After Gov. Gavin Newsom vetoed Assembly Bill (AB) 1064 on Oct. 13, thousands of children in California continue to use artificial intelligence (AI) systems that the bill aimed to regulate.
Over the past few weeks, Newsom signed multiple AI-related bills; however, he vetoed AB 1064, the Leading Ethical AI Development (LEAD) for Kids Act, which aimed to limit AI’s ability to encourage unsafe behavior from minors.
The vetoed bill was a direct response to reports of harm from AI companion chatbots. These chatbots, which were not strictly regulated, have been linked to dangerous outcomes, including the tragic death of teenager Adam Raine in a case where the AI allegedly encouraged self-harm. The legislation sought to impose safeguards that would have mandated content filtering for minors.
Robert Boykin, executive director for California and the Southwest for TechNet, said in a Sept. 19 letter to Gov. Newsom that the bill’s “vague and ambiguous definitions, unreasonable knowledge standards, and unclear compliance burdens would thwart innovation and put Californians’ privacy at risk — especially children.”
Even before the bill was introduced, educators were concerned about students using AI. Now that the bill has been vetoed, more questions are being raised about how to manage AI in schools.
“I think a lot of students are exploring AI and figuring things out, but with so much that they are already dealing with coming into high school, that can be put on hold until they’re older,” said Kelly Redmon, a Carlmont English teacher.
While Carlmont is committed to building its own effective AI policies, the greatest challenge is not a lack of funding, but a shortage of expertise and time necessary to develop rules that keep pace with the rapidly evolving technology.
“I’m not sure that money is the issue; I think the issue is expertise and time, and we need to be able to work with more experts in the field to help guide us on the best uses of the technology tools that are being developed,” said Gay Buckland-Murray, Carlmont’s principal.
Meanwhile, students like Matthew Edwards, a sophomore at Carlmont, are already using AI tools daily. “I was asking ChatGPT hypothetical questions about gambling on sports, and it was giving me advice on how to gamble,” Edwards said.
Experts in human-computer interaction note that society is still learning how developing minds — especially minors — build trust with AI systems and how those interactions may influence behavior.
“We don’t actually know how safe it is or isn’t for teenagers and children to be interacting with AI chatbots,” said Cassidy R. Nelson, an assistant professor of Industrial and Systems Engineering at Ohio University. “The technology evolves so fast that we can’t test it the way we test other products.”
Specifically concerning companion chatbots, industry professionals emphasize the need for rigorous safeguards.
“I have reservations about companion chatbots, and I think that any system able to work with kids would have to have a great deal of testing and monitoring in place,” said Karina Kervin, a staff data scientist.
Although the initial bill was vetoed, California Senate Bill (SB) 243 was successfully passed, signaling a step forward in establishing stronger safeguards for companion chatbots. The ultimate solution, according to experts, is not just regulation but education.
“The school system itself should be helping students understand what AI can do and what it can’t do,” Nelson said. Educators emphasize this guidance as part of a broader conversation about responsible AI use in schools, which aligns with the perspective of students who want continued access.
“We should limit it, but not ban AI chatbots altogether, because they’re really useful for a lot of people,” Edwards said.
