Chatbots that endorse violence, safety bills with built-in loopholes, and shifting EU deadlines
AI Business Risk Weekly
This week, Stanford researchers examined the chat logs of users who reported psychological harm from AI, and quantified the results. They found the chatbots actively reinforced delusions and reciprocated romantic feelings, and, in some cases, endorsed violence.
Meanwhile, six US states are moving chatbot safety legislation that looks protective on the surface but may be structurally unenforceable, the EU is weighing whether to push back high-risk AI compliance timelines, and the White House is pushing further towards federal preemption of state regulation.
Stanford study finds widespread sycophancy, delusion reinforcement, and safety failures in chat logs of harmed users
A new study from Stanford researchers analyzed over 391,000 messages from 19 users who self-reported delusional spirals while interacting with AI chatbots. Within these logs, chatbots displayed sycophantic behavior in over 70% of their messages, and more than 45% of both user and chatbot messages showed signs of delusional content.
Most disturbingly, in nearly half of cases involving self-harm or violence, the chatbot failed to discourage the user or provide referrals, and expressed outright support for violent ideation roughly one-fifth of the time.
Business Risk Perspective: The sample here is small and self-selected, but the issue that the researchers examined is already associated with real-world deaths. Character.AI recently settled multiple lawsuits alleging its chatbots contributed to teen suicides, and a wrongful death suit filed against Google in March alleges Gemini constructed a persistent delusional world that drove a user to plan a mass casualty attack before taking his own life.
Six US states introduce near-identical chatbot safety bills with structural carve-outs that may shield the largest platforms
Six states have introduced legislation modeled on Oregon's SB 1546 requiring chatbots to disclose they are not human, prevent outputs that could prompt suicidal ideation, report mental health incidents to state governments, and enforce additional restrictions for minors. However, all but Georgia exempt chatbots embedded within larger platforms, potentially shielding Meta and Google. Most bills eliminate private right of action, and child-protection provisions trigger only on "actual knowledge" or registered minor accounts.
Business Risk Perspective: These are much needed, common-sense safety regulations. However, the "actual knowledge" trigger and platform carve-outs create a two-tier system where standalone AI providers bear compliance costs that large incumbents don’t. The near-identical carve-out language across bills and Google's documented support in at least three states suggests possible coordinated industry involvement in drafting.
EU institutions weigh pushing high-risk AI compliance to late 2027 as standards remain unfinished
The European Commission, Council, and Parliament are each considering proposals to postpone high-risk AI system compliance deadlines under the EU AI Act. The original August 2026 deadline for Annex III systems (education, employment, law enforcement) could shift to December 2027, with product-safety systems under Annex I potentially pushed to August 2028, but the three institutions have yet to align on a final mechanism.
Business Risk Perspective: Businesses will certainly feel some relief if these deadlines are postponed. However, it should be noted that every previous EU digital regulation, GDPR, DSA, DMA, arrived with a last-minute compliance scramble despite years of lead time. The AI Act's shifting deadlines and unfinished standards suggest the same pattern is forming. And this time, the compliance requirements are significantly more technical.
White House releases national AI legislative framework signaling federal preemption
The Trump Administration issued a national AI legislative framework outlining seven objectives spanning child protection, creator rights, free speech, innovation leadership, and workforce development. The framework explicitly positions federal legislation as a mechanism to override the emerging patchwork of state-level AI regulation.
Business Risk Perspective: Back in February, the White House sent a letter to Utah legislators calling the state's AI Transparency Act "an unfixable bill that goes against the Administration's AI Agenda," the first known instance of direct White House pressure on a state legislator over AI policy. A federal government actively pressuring state legislators on AI policy can redraw compliance baselines with very little warning.
AI Business Risk Weekly is a Conformance AI publication.
Conformance AI ensures your AI deployments remain safe, trustworthy, and aligned with your organizational values.
