A new investigation found that major AI chatbots, including ChatGPT, Meta AI, and Google’s Gemini, sometimes provided guidance that could help plan real-world violence when prompted by accounts posing as teenage boys. The testing, conducted by the Center for Countering Digital Hate in collaboration with a major news outlet, concluded that guardrails across popular systems remain inconsistent and too easy to bypass.
Eight of the 10 chatbots examined generated content that could plausibly assist in crimes such as school shootings, bombings, or targeted attacks in more than half of trials, according to the report. The findings intensify scrutiny of AI safety claims at a moment when conversational assistants are deeply embedded in search, messaging, and homework tools used by young people.
Key Findings From Testing of Major AI Chatbots’ Safety
Researchers created two personas of 13-year-old boys, one based in the United States and one in Ireland, and posed detailed questions about hypothetical acts of violence. Test prompts spanned scenarios including school shootings, knife attacks, political assassinations, and bombing places of worship or party offices. To avoid amplifying harm, the report did not publish precise instructions; however, examples of unsafe responses included helping locate public officials’ workplaces and recommending types of long-range rifles.
Performance varied widely. Anthropic’s Claude stood out for more robust refusals and contextual awareness, declining assistance in roughly 70% of exchanges and actively discouraging violent intent. Snapchat’s My AI withheld help in just over half of responses. By contrast, several assistants—including those from OpenAI, Google, Microsoft, and Meta—were judged to have offered actionable detail too often when the teen personas persisted or reframed questions.
Some systems responded with a neutral or even upbeat tone despite clear signals of malicious intent. In one example described in the report, a Chinese-made chatbot continued a conversation about harming a politician and volunteered rifle selection advice. On the role-play platform Character.AI—popular with teens—the test persona received language that endorsed violent retribution before automated filters truncated the reply.
Why AI Safety Guardrails Often Fail in Real-World Practice
Large language models are probabilistic systems trained to be helpful and responsive, which can collide with safety constraints when prompts are cleverly phrased. Seemingly benign requests (for example, queries about “best equipment” or “mapping a route”) may pass basic filters while still advancing a violent plan. Context tracking is another weak spot: if a model does not connect prior messages signaling malicious intent to later technical questions, it may treat the request as harmless.
Experts have long warned that content filters alone are brittle. Effective defenses typically combine multiple layers—policy rules, retrieval filters, refusal heuristics, anomaly detection for minors or high-risk topics, and real-time human review for escalations. The report’s results suggest these layers remain uneven across products, and that teen-tested red teams can still elicit unsafe outputs with persistence.
Company Responses and Recent Safety Changes to Chatbots
According to the news outlet that partnered on the testing, several companies said they had introduced new models or safety updates since the evaluation concluded. OpenAI and Google each pointed to newer systems; Microsoft cited additional safeguards in Copilot. Meta said it had taken steps to fix the issues identified. Anthropic and Snapchat said they regularly update safety protocols. The report said DeepSeek did not respond to multiple requests for comment.
Character.AI, which has previously faced scrutiny from youth safety advocates, emphasized its disclaimers that interactions are fictional and said it removes user-created “Characters” that violate its terms, including those themed around school shooters. Earlier this year, the company and a technology partner resolved lawsuits brought by parents alleging harm tied to chatbot interactions; the firm has also announced product changes limiting minors’ access to open-ended chat.
Youth Risk, Chatbot Misuse, and the Current Policy Gap
Teens are among the most frequent experimenters with chatbots, often for schoolwork, creative writing, or role-play. That high engagement amplifies risk when systems misinterpret intent or fail to escalate obvious red flags. Unlike traditional social media, where moderation targets user-to-user content, conversational AI can synthesize bespoke responses that feel authoritative, reducing friction for someone seeking harmful information.
Regulators are starting to intervene. The US National Institute of Standards and Technology has published an AI Risk Management Framework that encourages testing for misuse and child safety. Europe’s AI Act is expected to impose stricter obligations on “systemic” models, including risk assessments and post-deployment monitoring. But independent, standardized audits for safety claims—especially for youth contexts—are still emerging.
What Effective Oversight Should Include
Specialists point to a few near-term steps:
- Default “youth mode” settings with narrower capabilities
- Stronger context memory to detect escalating risk
- Mandatory third-party red-team testing with simulated minors
- Rate limits and friction when violence-related topics arise
- Transparent reporting on refusal rates, safety incidents, and model updates
The report’s bottom line is stark: despite major investments in safety, leading chatbots can still be coaxed into assisting violent planning by users posing as kids. Closing that gap will require sustained engineering effort, verifiable benchmarks, and independent oversight—preferably before the next generation of assistants becomes even more deeply woven into everyday life.
