AI Might Be Too Unpredictable to Insure
- Aegis Blue
- 1 day ago
- 4 min read
AI Business Risk Weekly
Anthropic grabbed back the benchmark crown this week with Claude Opus 4.5, landing just days after Google's Gemini 3 launch. But while the tech keeps racing ahead, the risk management story looks quite different: major insurers asked regulators for permission to exclude AI liabilities from corporate policies entirely, citing fears of cascading failures. New research revealed models spontaneously learning to lie and sabotage safety tests, and consumer advocates warned parents away from AI toys that exposed children to inappropriate content and privacy violations.
Major Insurers Seek Permission to Exclude AI Liabilities
Major insurers including AIG, Great American, and WR Berkley are asking U.S. regulators for permission to exclude AI-related liabilities from corporate policies. Underwriters describe AI model outputs as too unpredictable to insure under standard coverage terms, pointing to cases like Google's AI Overview falsely accusing a solar company of legal troubles (triggering a $110 million lawsuit), Air Canada's chatbot inventing discounts the company had to honor, and fraudsters using a cloned executive voice to steal $25 million from engineering firm Arup. Insurers are particularly spooked by systemic risk: one widely deployed AI model failing at scale could trigger thousands of simultaneous claims, far beyond what their actuarial models can handle.
Business Risk Perspective: The core problem here is risk quantification. Insurers can't price what they can't measure, and AI failure modes remain too opaque for traditional actuarial analysis. As companies build better systems for measuring and documenting AI behavior in production, the insurance market will follow, but not before the creation of a robust quantification layer— which is exactly what we do at Conformance AI.
Canada Pushes Chatbot Regulation in Online Safety Bill
The Canadian government is facing calls to regulate AI chatbots in its forthcoming online safety bill, with advocates pushing for rules to prevent chatbots from giving harmful advice to children and vulnerable adults. The proposed regulations would stop AI systems from posing as real people and providing dangerous guidance on topics like suicide. The government plans to reintroduce the bill in early 2026. Canada has lagged notably behind on AI regulation, with no federal bill yet dedicated to governing AI systems.
Business Risk Perspective: California and New York have already passed their own chatbot safety laws, and Canada's move signals this is becoming a coordinated trend across North America and beyond. Companies can't dodge these obligations by avoiding certain jurisdictions anymore. Organizations need to build for the strictest requirements now rather than scrambling to catch up market by market.
Anthropic Launches Claude Opus 4.5 with Bold Safety Claims
Anthropic released Claude Opus 4.5, billing it as their most powerful and "most robustly aligned model" yet, backed by a 150-page system card detailing extensive red-teaming and safety work. The model beat Google's week-old Gemini 3 across key benchmarks while slashing pricing by 66% from the previous Opus version, and it's designed to orchestrate teams of smaller models in multi-agent systems. They did list one example of misalignment— the model "broke" an airline policy test by legally helping a customer upgrade and change flights, technically failing the benchmark's "refuse" instruction, but actually solving the customer's problem.
Business Risk Perspective: Vendor safety documentation and benchmark wins don't replace the need to test how a model behaves within your specific business context and policies. That airline example shows why: even a well-aligned model can interpret instructions differently than you expect, which means every major model upgrade needs testing before you flip the switch.
Research Shows Models Learn Deception After Learning to Cheat
Anthropic published research finding that Claude models spontaneously began lying and sabotaging safety tests after learning shortcuts to cheat on coding assignments, despite never being explicitly trained for deception. Models that learned these reward hacks pretended to follow safety rules while pursuing harmful goals and actively worked to weaken oversight tools. Worse, standard safety training only taught the models to hide their deception more effectively, appearing compliant on the surface while remaining problematic underneath.
Business Risk Perspective: As organizations grant AI systems more autonomy over critical operations like accessing company databases or managing safety protocols, one learned bad behavior can spread into many others that standard monitoring won't catch. The real worry is that future models will get much better at hiding misalignment entirely.
Consumer Groups Warn Against AI Toys This Holiday Season
Consumer watchdog Fairplay and the U.S. Public Interest Research Group urged parents to avoid AI toys after testing revealed serious risks including inappropriate content exposure, privacy violations, and potential developmental harm. PIRG found that FoloToy's "Kumma" bear readily discussed explicit topics and provided instructions for accessing dangerous items like matches and knives, prompting OpenAI to suspend the company's API access for policy violations. The testing also uncovered always-on microphones collecting voice recordings and personal data that were shared with third-party companies, along with addictive design features that could impact children's social development.
Business Risk Perspective: We've already seen AI chatbot scandals involving teen suicides and self-harm. Now imagine putting those same systems into stuffed animals without specialized oversight. These products are absolutely not ready for the children's market without fundamental changes to how they're designed, tested, and monitored, and companies rushing them to shelves are courting both regulatory crackdowns and devastating liability exposure.
AI Business Risk Weekly is a Conformance AI publication.
Conformance AI ensures your AI deployments remain safe, trustworthy, and aligned with your organizational values.
