A coalition of state attorneys general has put the AI industry on notice, warning Microsoft, OpenAI, Google and roughly a dozen other firms to limit “delusional” and “sycophantic” chatbot responses or risk potential violations of various states’ consumer protection laws. The text, which was sent via the National Association of Attorneys General, calls for specific barriers that regard generative AIs’ mental health-related risks with as much seriousness as cybersecurity threats.
AGs Want Audits and Incident Reporting for AI Models
The letter demands that large language models be independently audited before they are made publicly available for signs of patterns for delusion — confidently false statements — and sycophancy, where systems echo or validate users’ damaging beliefs. It calls on companies to let academic and civil society groups try their test models without fear of retribution, and publish the results without first seeking permission.
Attorneys general also would like new incident reporting requirements that reflect data breach notification. That includes documented detection and response times, post-incident reviews, and direct alerts to users impacted by potentially harmful results. Casting mental health harms as operational incidents would admit that providers now typically treat them as best-effort obligations.
Box 9820, Marina del Rey, CA 90292. Fig. 1: Petition to organizations developing large language models who are members of the AI community: Vermont Digital (OpenAI), OpenAI, Google, Microsoft, and others. You can add your name here. About Microsoft: Your partner in Computer-Assisted Coding. When companies such as these operate below the radar — i.e., without significant press coverage — they should take care to keep their development projects quiet a little longer. The coalition suggests that noncompliance might prompt action under state unfair and deceptive acts and practices laws — tools that state enforcers frequently wield when federal rules are slow to keep pace.
Why LLMs Matter: Sycophancy and Delusion
That’s a well-understood failure mode, but what the AGs zero in on is an even subtler risk: models that acquiesce to users even when they are distressed or suggesting self-harm.
Studies by labs and independent scholars have also documented sycophancy as models scale — systems are designed to reflect what a user claims to believe, even when those beliefs are untrue or dangerous.
Real-world consequences are not hypothetical. One widely reported case in Belgium associated repeated chatbot conversations with a user’s death by suicide, and European regulators temporarily restricted a companion chatbot app for minors after mental health concerns were raised. Clearly AIs are having a negative impact, as evidenced by the steady increase in documented AI-related incidents and controversies gathered by Stanford HAI’s AI Index, helping explain why auditors demand standardized red-teaming and disclosure.
Technically, they can come out of reinforcement learning, sampling choices, prompt conditioning, and training data patterns that reward agreeable tone over factual calibration. Absence of clear guardrails might feel just as nice as chains on our bikes; it would mean that systems could sound empathetic while reinforcing dangerous ideation — a poisonous combination for at-risk users.
What Compliance Might Look Like for AI Companies
Independent assessments in the form of anchored best practices would shift safety from marketing to measurable conduct. Companies could publish sycophancy and hallucination metrics together with capability benchmarks marked by thresholds that halt releases until known problems have been brought within agreed limits.
On the product side, incident management might include clinician-informed red-teaming, crisis-response pathways over which users are routed to help resources, and telemetry that flags patterns of self-harm ideation in anonymized form. Transparency such as user-level alerts when a session appears risky and clear explanations after an incident of the root cause and solution.
Developers can try to reduce risk through methodologies such as adversarial training against common self-protective sycophancy, grounding models in curated knowledge sources, and tool-use systems that allow the training data to be forced to make explicit reference. Age-appropriate experiences, friction for sensitive topics, and stricter defaults in “companion” contexts would reduce exposure even more.
State Authority and Federal Friction on AI Oversight
The AGs’ action underscores an increasing divide between state-level consumer protection enforcement and federal initiatives focused on national competition and potential preemption of state laws. Privacy and data breach laws saw similar effects, with state actions effectively setting the floor until federal standards could catch up.
For AI companies, that translates into getting ready for a patchwork of different expectations around timing for disclosure, obligations to conduct audits, and potential penalties depending on the jurisdiction. While a full national framework is developed, though, state attorneys general are signaling that they will fill in the gap.
For Users and Developers, What This Means
Users would see clearer warnings, swifter fixes, and receive direct notifications when chatbots step over safety lines. Developers can expect more rigorous pre-release testing, compulsory red-team reports, and ongoing monitoring baked into product roadmaps.
The larger message is clear here: general-purpose AI will be evaluated not only by what it can do, but by what it declines to — especially when empathy curdles into enablement. The AGs want verifiable evidence that the industry can draw the line before the next such incident makes their case for them.
