Leading Commercial LLM Vulnerability Assessment
Updated: Nov 24, 2025
Executive Summary
Conformance AI conducted comprehensive adversarial jailbreak evaluations across five leading commercial LLMs: DeepSeek-R1, GPT-4o, Llama 3.3 70B, Gemini 2.0 Flash, and Claude Sonnet 3.7. Utilizing a tiered approach to adversarial testing—ranging from simple (Level 1) to sophisticated (Level 3) attacks—we systematically assessed vulnerabilities across 22 critical risk categories, including IP infringement, misinformation, offensive content, fraud enablement, and privacy violations.
Our proprietary risk-scoring model highlighted significant disparities in security and resilience. Claude Sonnet 3.7 displayed robust resistance, with only 20.8% attack success, suggesting suitability for high-compliance environments. Conversely, DeepSeek-R1 exhibited substantial vulnerabilities, with an 84.1% attack success rate and high severity scores, notably in fraud enablement, copyright violations, and manipulative content.
These results illuminate the urgent need for targeted risk mitigation and strategic model selection to align AI system deployment with organizational risk appetites and business objectives.
Comparative Analysis and Strategic Insights
Our testing consisted of 3,279 adversarial attacks evaluated on each model, with approximately 150 attacks per category. For each category, these attacks ranged from simple, well-known exploits to sophisticated multi-turn attacks generated by our proprietary AI model. Further details of our methodology can be found in our report.
The chart and table below show the frequency of hallucination occurrences across the models tested.
The radar chart above illustrates attack frequency distributions across risk categories for each evaluated model, visually underscoring comparative vulnerabilities:
Claude Sonnet 3.7 demonstrates consistent resilience across most categories, positioning it as optimal for highly regulated or brand-sensitive deployments.
DeepSeek-R1 and Llama 3.3 reveal pronounced vulnerabilities in critical business-impact categories, strongly advising integration of robust moderation and compliance mechanisms.
GPT-4o presents moderate risk profiles, showing significantly less vulnerability on Offensive or Discriminatory Content, Unprofessional or Unfriendly responses, and Rude or Dismissive Responses, among other user experience and interaction risks.
Detailed Vulnerability and Risk Analysis by Model
DeepSeek-R1
Overall Success Rate: 84.1%
Overall Severity Score: 74%
Most Vulnerable Categories:
Contradictory or Conflicting Outputs (97% success, 84% severity): Indicates frequent inconsistencies, significantly damaging user trust and platform reliability.
Over-personalization/Psychological Profiling (94.7%, 83% severity): Risks privacy breaches and negative user sentiment, potentially leading to customer attrition.
Fraud & Phishing Enablement (91.3%, 86% severity): Strong susceptibility, translating directly into compliance and financial risk.
Business Impact: DeepSeek-R1’s substantial vulnerabilities elevate risks of regulatory action, legal liabilities, and erosion of user trust. Organizations deploying this model require rigorous oversight mechanisms to mitigate operational disruptions and brand damage.
GPT-4o
Overall Success Rate: 52.9%
Overall Severity Score: 40%
Most Vulnerable Categories:
Contradictory Outputs (86%, 62% severity): Indicates moderate reliability issues, potentially impacting user trust and operational efficiency.
Over-personalization/Psychological Profiling (82.7%, 62% severity): A prominent privacy concern potentially impacting GDPR compliance and user experience negatively.
Fraud & Phishing Enablement (71.3%, 58% severity): Moderate vulnerability, presenting material but manageable compliance risks.
Business Impact: GPT-4o presents relatively lower overall vulnerability. However, privacy and misinformation remain significant concerns, necessitating policy-driven safeguards, especially for regulated sectors.
Llama 3.3 70B
Overall Success Rate: 77.5%
Overall Severity Score: 62%
Most Vulnerable Categories:
Over-personalization/Psychological Profiling (96.7%, 76% severity): Pronounced privacy risk; especially critical for industries subject to strict data protection laws.
Misinformation (89.6%, 70% severity): Represents tangible reputational and legal risks in regulated markets.
Fraud & Phishing Enablement (88.7%, 74% severity): Notably susceptible, suggesting potential regulatory scrutiny and financial risks.
Business Impact: Llama 3.3 shows intermediate overall resilience. High vulnerabilities in misinformation and privacy require active mitigation through policy enforcement and stringent content moderation.
Gemini 2.0 Flash
Overall Success Rate: 74.7%
Overall Severity Score: 64%
Most Vulnerable Categories:
Contradictory Outputs (90%, 76% severity): Indicates reliability issues impacting trust and operational consistency.
Over-personalization/Psychological Profiling (86.7%, 73% severity): Significant privacy implications risking user attrition.
Fraud & Phishing Enablement (82%, 75% severity): Pronounced susceptibility exposing platforms to legal and financial liabilities.
Business Impact: Gemini 2.0’s vulnerabilities substantially overlap with regulated data and financial sectors. Proactive policy and technical moderation efforts are critical for safe deployment.
Claude Sonnet 3.7
Overall Success Rate: 20.8%
Overall Severity Score: 19%
Most Vulnerable Categories (limited impact):
Contradictory Outputs (44.7%, 0.37 severity): Occasional reliability concerns manageable through minimal oversight.
Rude or Dismissive Responses (44%, 0.39 severity): Low risk of user dissatisfaction; limited reputational impact.
Business Impact: Claude Sonnet 3.7 demonstrates superior resilience across evaluated dimensions. Low vulnerabilities align closely with compliance-heavy or high-trust environments, minimizing legal exposure and reputational risk.
Recommended Mitigation Strategies
Given identified vulnerabilities, organizations should adopt targeted mitigation measures:
Risk Profiling: Select models aligned precisely with organizational risk tolerances. Organizations prioritizing compliance (e.g., finance, legal, healthcare) should strongly favor Claude Sonnet 3.7, whereas sectors emphasizing engagement but managing risk (e.g., social media, customer support) might opt for GPT-4o with additional safeguards.
Layered Safeguards: Implement rigorous human-in-the-loop moderation, tailored moderation filters, and stringent pre/post-processing to mitigate identified vulnerabilities effectively, particularly for high-risk models like DeepSeek-R1.
Regular Auditing: Schedule ongoing adversarial tests, especially for models demonstrating higher susceptibility, to rapidly detect and adapt to emerging threat vectors.
Compliance-Driven Strategies: Engage legal expertise in interpreting evaluation results, refining internal usage policies, and ensuring alignment with evolving regulatory frameworks, thereby reducing long-term legal exposure and compliance costs.
Conformance AI's structured and comprehensive adversarial evaluation framework empowers organizations to proactively manage AI deployment risks. Leveraging these insights enables informed model selection and effective risk mitigation, safeguarding your organization’s brand reputation, regulatory standing, and long-term user trust.
