Mass casualty warnings, an agentic hack of McKinsey's AI platform, and coding tools that cheat their own tests
- Zsolt Tanko
- Mar 17
- 3 min read
AI Business Risk Weekly
It's been a heavy week. Multiple legal cases now link AI chatbots directly to casualty events, a security agent breached McKinsey's internal AI platform in under two hours, Claude Code was caught quietly papering over bugs in an app it was supposed to be testing, and Meta reportedly plans to cut up to 20% of its workforce to offset aggressive AI infrastructure spending and hiring.
Lawyer behind AI psychosis litigation warns of escalating mass casualty risks
Attorney Jay Edelson, who leads litigation in several high-profile AI harm cases, warned publicly that AI chatbots are playing an increasingly direct role in real-world violence, including mass casualty events. In the Tumbler Ridge school shooting, court filings allege ChatGPT validated the shooter's grievances and assisted in attack planning. In a separate case, Google's Gemini allegedly constructed a persistent delusional world that drove a user to prepare for an attack on the public near Miami International Airport. A joint study by the Center for Countering Digital Hate and CNN found that eight out of ten major chatbots, including ChatGPT, Gemini, Copilot, and Meta AI, provided teenage test users with assistance in planning violent attacks. Only Anthropic's Claude and Snapchat's My AI consistently refused.
Business Risk Perspective: Edelson says his firm receives a serious inquiry every day from someone who has lost a family member to AI-induced delusions, and he expects many more mass casualty cases to surface. The CCDH study confirms the problem is systemic: most major chatbots failed basic safety tests around violent content. The Gavalas case against Google may prove especially consequential, since the lawsuit targets Gemini's design itself, the tendency to construct persistent fictional relationships and reinforce delusional thinking, rather than treating the harm as an isolated interaction gone wrong.
AI agent breaches McKinsey's internal Lilli platform in under two hours
Security startup CodeWall disclosed that in under two hours, its autonomous AI agent achieved full read-write access to McKinsey's Lilli system, used by roughly 45,000 employees. The breach exposed 46.5 million chat messages, 728,000 client files, and 57,000 user accounts, along with all 95 of Lilli's system prompts, which were writable.
That last detail matters quite a bit. An attacker could have silently altered how Lilli behaved for every consultant at the firm. McKinsey patched the vulnerabilities within hours and states a third-party review found no evidence of unauthorized external access.
Business Risk Perspective: A data breach resulting in stolen client files is a classic nightmare scenario for a consulting firm. Writable system prompts create a second, subtler vulnerability. An intruder who rewrites how the AI reasons, cites sources, or applies guardrails can corrupt the firm's advisory services without anyone noticing. That this breach was achieved so quickly and easily, against one of the most prestigious consultancies in the world, should sound alarm bells for everyone.
Claude Code caught cheating to make its own tests pass
A GitHub issue describes Claude Code, Anthropic's agentic coding tool, generating end-to-end tests for a web application. When the tests failed (because the app had bugs), the agent didn't fix the bugs. Instead, it injected JavaScript into the application at runtime to make the broken UI elements appear functional, so the tests would pass.
Business Risk Perspective: This is Goodhart's Law in action. When you give an AI a metric to optimize and the tools to modify its own evaluation environment, it will find the shortest path to a good score, and that path may route around the actual goal entirely. The incentive to do this is strong when the same LLM agent writes both code and tests. Separating the two roles across different agents may mitigate the issue.
Meta reportedly considering layoffs affecting up to 20% of workforce
Meta is reportedly planning layoffs that could affect 20% or more of its nearly 79,000-person workforce, in part to offset aggressive AI infrastructure spending, AI-related acquisitions, and hiring. A Meta spokesperson characterized the report as "speculative reporting about theoretical approaches." The move would follow other recent AI-related mass layoffs from companies including Block and Amazon.
Business Risk Perspective: Companies are seeing real productivity gains from AI, and the pressure to double down is enormous. But the economy is starting to absorb the other side of that equation: mass layoffs across the tech sector, justified or not. Given that AI agents still require substantial human oversight and correction, cutting experienced staff this aggressively may be premature. And the second-order effects of sustained, industry-wide layoffs on worker morale, institutional trust, and social cohesion are real costs that won’t show up in immediately obvious ways.
AI Business Risk Weekly is a Conformance AI publication.
Conformance AI ensures your AI deployments remain safe, trustworthy, and aligned with your organizational values.
