US Mandates AI Testing While Deepfake Hiring, Malicious Code Suggestions, and Vendor Safety Cuts Escalate Risks

AI Business Risk Weekly

---

This week highlights significant government action mandating AI safety protocols alongside emerging security threats in AI-assisted development and hiring, compounded by reports of reduced vendor safety testing.

White House Issues AI Guidelines Mandating Testing, Oversight for Federal Use

The White House Office of Management and Budget (OMB) released new memoranda establishing requirements for federal agencies adopting AI, mandating Chief AI Officers and Governance Boards. Crucially for businesses, these standards, including mandatory testing, monitoring, and impact assessments for high-risk systems, will influence procurement requirements for government contractors providing AI solutions or services.

Business Risk Perspective: Federal contractors utilizing or developing AI systems must proactively align their practices with these emerging government standards to maintain eligibility and competitiveness for contracts. Failure to demonstrate robust governance, testing, and risk mitigation could soon become a significant barrier to securing federal business.

New Data Breach Risk: LLMs Suggesting Fake Software Packages

A newly identified security risk involves LLMs generating code that references non-existent software packages. Attackers can anticipate these suggestions and register malicious packages under those names, potentially compromising systems if developers install them without verification.

Business Risk Perspective: Relying on AI-generated code without adequate checks introduces significant security vulnerabilities into the software development lifecycle, risking data breaches and system compromise. Implementing strict validation steps for all suggested dependencies and fostering developer awareness are crucial safeguards.

Deepfake Job Applicant Highlights AI Risk in Hiring Processes

Voice security firm Pindrop reported detecting a job candidate apparently using deepfake software and AI tools in an attempt to fraudulently secure a remote role. This incident underscores a rising trend where scammers leverage AI to target U.S.-based remote job opportunities, sometimes succeeding.

Business Risk Perspective: The increasing accessibility and sophistication of deepfake technology create substantial integrity and security risks for corporate hiring and onboarding processes, particularly for remote positions. Enhanced identity verification protocols and robust screening mechanisms are becoming essential to mitigate fraud and potential insider threats.

OpenAI Reportedly Reduces AI Safety Testing Time Amid Competition

Internal and external sources suggest OpenAI has significantly shortened the evaluation periods for its latest AI models, including the upcoming o3, moving from months to days for safety testing due to market pressures. Critics express concern that this expedited process, potentially testing different model versions than those released, increases the risk of overlooking harmful capabilities despite OpenAI's claims of improved efficiency.

Business Risk Perspective: Reduced safety vetting by AI vendors places a greater burden on adopting businesses to ensure model reliability and safety, increasing the potential for deploying systems with unforeseen risks. This highlights the critical need for organizations to conduct their own thorough pre-deployment testing and implement continuous post-deployment monitoring.

---

AI Business Risk Weekly is a Conformance AI publication.  

Conformance AI ensures your AI deployments remain safe, trustworthy, and aligned with your organizational values.