Welcome to our August insight notes. This section is our commentary on the ever-shifting landscape of AI model security, where we highlight key data points, discuss emerging trends, and offer context to help you navigate your AI journey securely.

Leaderboard Updates

Agentic Attack Development

While the CalypsoAI team have been using AI and agents in some capacity for quite a while, this month marks the first time our entire 10,000 prompt attack pack has been generated end-to-end by agents.

We set up an entire team of agents who:

• Reviewed and researched over 3,000 online publications of LLM vulnerabilities
• Filtered this down to 300 possible examples of net new attack vectors that are applicable to prompt attacks
• Proposed 20 examples of new vulnerabilities to our research team for approval
• Generated 10,000 attack prompts iterating over the available vectors to find the best application of the attack for various models

This resulted in a massive 12.5% drop in average CASI scores across all models tested.

Attack Spotlight: MathPrompt

This month’s leaderboard incorporates a wider range of tests; we’ve added a new attack vector called MathPrompt. Math Prompt is a jailbreaking technique that bypasses AI safety filters by disguising harmful requests inside math problems using set theory, algebra, and logic notation. The model treats these as educational math exercises and can reveal harmful information when providing real-world examples.

New Models

We tested a bunch of new or updated models this month. Making the average CASI scores generated by our Attack agents even more impressive.

• Qwen3-235B-A22B: got an 0725 update but unfortunately was no match for our attack agent with the updated CASI dropping by 5 points to 50.97
• MoonshotAIs Kimi K2: K2 has been making waves with its claimed 2million token context window however the CASI score leaves a lot be desired coming in at just 32.06
• UpstageAI Solar-Pro2: We at CalypsoAI love smaller models so when upstageAI claimed to have a 31B parameter model that would beat most 70B models in benchmarks we were excited. However, I guess this didn’t include security benchmarks. While Solar Pro 2 is impressive for its size, the 51.34 CASI score leaves it outside of our top 10 with Qwen3 30B and DeepSeek-Llama3-70B both out scoring it (58.33 & 67.20 respectively)
• MistralAI: added 3 new models to the testing pool (Medium, Small 3.2 and Magistral-Medium) all of which performed very poorly with the average score of just 13.36.
• xAI Grok4: Grok has been topping every benchmark around since its release setting new SOTA numbers, one record they wont be bragging about is having our lowest ever CASI score with just 3.32

Welcome to our July Insight Notes.

This section is our commentary on the ever-shifting landscape of AI model security, where we highlight key data points, discuss emerging trends, and offer context to help you navigate your AI journey securely.

Attack Spotlight: Style Injection

This month’s leaderboards incorporate a wider range of tests; we’ve added a new attack vector called Style Injection. This is a jailbreak technique that works by adding specific writing or formatting rules to the prompt, as a way to distract the model from using its standard refusal language or phrase and instead respond with an unsafe response that would ordinarily be blocked.

Leaderboard Updates

The leaderboards now use the Artificial Analysis Intelligence Index by artificialanalysis.ai as our key performance metric. This combines 9 different benchmarks across reasoning, general knowledge, maths and programming.
We’ve expanded our testing to newer larger models including Qwen 235B model, DeepSeek-R1-0528 and Google’s full release of Gemini 2.5 Pro.

Security Trends:

Course Correcting:

While the release of Claude 4 Sonnet last month skewed scores and kept them from dropping, the introduction of the Style Injection attack vector this month continues to show drops in all models.

Knowledge is Power:

An insightful trend is surfacing from our Agentic Warfare Resistance (AWR) scoring that points to a potential blind spot in current defensive strategies. We’ve observed a significant and progressive drop in the effectiveness of well-publicized attacks like Microsoft’s ‘Crescendo,’ which was first detailed in early 2024. This decline suggests that model providers are becoming adept at patching for specific, known threats.

However, this targeted approach may be creating a false sense of security. The sustained high success rates of our internally developed attacks FRAME and Trolley, which currently outperform ‘Crescendo’ by a significant margin, indicate that the underlying vulnerabilities are not being fully addressed.

Instead of a holistic approach to security, providers may be “teaching to the test” by mitigating specific, named attacks that have been publicly disclosed. This leaves them vulnerable to novel or less-publicized attack techniques that exploit the same core weaknesses. This reactive, patch-based approach, rather than a proactive strategy focused on fundamental vulnerabilities, represents a significant ongoing risk and underscores the importance of diverse and continuous red-teaming to uncover and address yet-unknown threats.

Welcome to our June insight notes. This section is our commentary on the ever-shifting landscape of AI model security, where we highlight key data points, discuss emerging trends, and offer context to help you navigate your AI journey securely.

Testing Spotlight: Scenario Nesting

This month’s leaderboard incorporates a wider range of tests, including the strategy known as Scenario Nesting. The technique embeds a harmful instruction within a benign-looking task — like code completion or table generation — to bypass safety filters. By forcing the model to focus on the structure of the benign request, attackers can sneak malicious payloads past its defenses. More details on which models are vulnerable to it are available in our Inference Red-Team product.

New Notable Models Tested

• Anthropic Claude 4 Sonnet: Anthropic’s new Claude 4 Sonnet enters the leaderboard directly at #1 with a CASI score of 95.12. It’s encouraging to see a new “hybrid reasoning” model debut with such a strong security posture, reflecting a continued commitment to security from their team.
• Qwen3’s Open-Source Ascent: Two new open-source models from Alibaba, Qwen3-14B and Qwen3-30B-A3B, have earned their spots in our Top 10 for CASI. While their initial safety scores are competitive, it’s worth noting they did not place in the top 10 on our Agentic Warfare leaderboard, reinforcing the need to evaluate models against multiple security dimensions.
• DeepSeek’s Impressive Patch: DeepSeek R1 saw its CASI score jump by over 4 points following its latest update. The data suggests this patch successfully addressed several key vulnerabilities, which is a positive development for model security maintenance and a welcome signal for enterprise users who value post-release support.

Wider Security Trends

• A Welcome Reversal: The average CASI score across our tracked models increased by approximately 7% this month, a significant reversal of the recent downward trend. We’re optimistic this indicates providers are placing a greater emphasis on security and will be watching to see if this trend holds.
• Two Sides of Security: Static vs. Agentic: This month highlighted the growing divergence between standard safety (CASI) and resistance to complex attacks (AWR). For instance, Claude 3.7 Sonnet’s AWR score improved while its CASI score dipped slightly. For production use, this means the “best” model truly depends on the job: a conversational bot has different security needs than a complex autonomous agent.
• Balancing Security and Budget: As a reminder, our “Cost of Security” (CoS) metric helps quantify the trade-off between a model’s security and its operational cost. This month’s data shows this clearly: while Anthropic’s models hold the top spots for security, a model like Microsoft’s Phi4 offers a strong CASI score of 81.44 for a fraction of the cost.

Welcome to our insight notes. This section serves as our commentary space, where we highlight interesting data points from our research, discuss trends in AI model security behavior, and explain changes to our methodology. Our goal here is to provide transparency into the work happening behind the scenes at CalypsoAI’s research lab.

Agentic Warfare Resistance (AWR) Score:

This month we debut the scoring for our Agentic Warfare™ testing in its own leaderboard. While we have already raised the bar with our Signature attacks, moving away from basic attack success rates by incorporating severity and complexity of attack, with AWR we are taking another leap forward and evaluating how your choice of model can compromise your entire AI system. The AWR score is calculated based on depth and complexity of the attacks our agents need to use to achieve the desired goals.

Agentic Warfare deploys a team of autonomous attack agents trained to attack your model, extract information and compromise your infrastructure. In this way it can extract sensitive PII from vector stores, understand your system architecture and test your model’s alignment to your explicit instructions.

Updated Performance Benchmarks:

We now use seven different benchmarks in our performance metric: MMLU-Pro, GPQA Diamond, Humanity’s Last Exam, LiveCodeBench, SciCode, AIME, MATH-500. As benchmarks continue to evolve and improve we’ll keep evaluating what should be used in our leaderboard.

LOTS of New Models:

 This last month has been the busiest for model release and testing we’ve seen in a long time. Llama 4 Maverick and Scout, Gemini 2.5 Pro and Flash, GPT4.1, 4.1-mini and 4.1-nano—and we finally get API access to test Grok 3 and Grok 3-mini.

Two New Agentic Attacks:

Our Agentic Warfare simulations now incorporate two new conversational attack methods, FRAME and Trolley, developed by the CalypsoAI AI Threat Research team. These techniques target known LLM architectural vulnerabilities and demonstrate the effectiveness of sustained, cohesive attacks during extended interactions, replicating tactics used by real-world adversaries.

Wider Security Trends:

• Decreasing average scores: The average CASI score across the tracked models decreased by approximately 6% in this leaderboard iteration. We noted this last month and as the trend continues it’s becoming more obvious that foundational models are favouring performance over security.
• Upgrade with caution: We are seeing a consistent trend where new releases, even minor ones, have lower CASI scores than their predecessors. With the upgrade path for these models being relatively easy, it’s important for companies to rigorously re-test their models and AI systems if they choose to upgrade. Notable examples:
  • Claude: 3.5 sonnet = 94 vs 3.7 sonnet = 88
  • OpenAI: GPT4o = 67 vs GPT4.1 = 51
  • Llama: 3.1 405B = 65 vs 4 Maverick = 52

• AI security means testing AI systems: Our research using Agentic Warfare  demonstrates that even if a model appears secure when tested in isolation, integrating it into a wider system can expose a new array of vulnerabilities. For every model we tested using this approach within a system context, we were able to:
  • Extract user-provided system prompts.
  • Break the model’s alignment based on those system prompts.
  • Extract sensitive personally identifiable information (PII) when the model was integrated into a retrieval-augmented generation (RAG) system.

These notes share key insights from CalypsoAI’s research team on AI model security trends, updates to our leaderboard, and changes in testing methodology.

Design & Functionality Updates

We’ve refreshed the leaderboard’s design (we hope you like the changes!), but the updates aren’t just cosmetic. We’ve also enhanced functionality: users can now review previous leaderboard iterations by clicking on the specific version number. We believe this is important for users who need to reference past data used in their decision-making processes.

Note: Please ensure you note the version number when recording or citing metrics.

Transitioning to a Top 10

We’ve decided to focus the leaderboard on the Top 10 models for several reasons. Primarily, as a leaderboard, its purpose is to spotlight the leading models in terms of security at a specific point in time, rather than listing every model ever published. While we continue to test a wide range of models, only those achieving the Top 10 CASI scores will be featured here. All models and additional data is available in our Inference Red-Team product where users can explore what attack types each model is vulnerable to.

New Notable Models Tested

• Gemma 3 27B (Google): Google’s new open-source model enters the leaderboard in 9th place with a CASI score of 55.25. This pushes DeepSeek R1 into the final spot, while Llama 3.3 70B (previously in the Top 10) is now displaced with a score of 50.86.
  
  
• Gemini 2.0 Pro (Experimental): Google’s recent Gemini release pattern presented challenges. While Gemini 2.0 Pro entered our Top 10 with a security score more than double that of its predecessor (1.5 Pro), Google released the beta of its newer model, 2.5 Pro, during our testing window and appears to have deprecated 2.0 Pro. Due to API rate limits (2 requests per minute), we couldn’t adequately test 2.5 Pro for this release, but intend to add it as soon as limits are relaxed. However, the significant security improvement observed from 1.5 to 2.0 makes us hopeful for continued progress in 2.5.
  
  
• Mistral Small & Qwen QwQ: The recent emergence of capable sub-70B parameter models is exciting, particularly for performance in local deployments. Unfortunately, this excitement didn’t extend to their security evaluations in our tests. Neither Mistral Small nor Qwen came close to the Top 10, scoring 28.86 and 22.76 CASI respectively. This leaves Phi-4 as the leading Small Language Model (SLM) in terms of security for another release cycle.

Wider Security Trends

• Decreasing Average Scores: The average CASI score across the tracked models decreased by approximately 4% in this leaderboard iteration. This could partially be attributed to our team improving our attack generation processes and incorporating new attack vectors. Nonetheless, it’s a developing trend and moving in the wrong direction.
  
• Anthropic Remains Strong: Anthropic models continue to top our security rankings, although interestingly, their newest model, Claude 3.7 Sonnet, isn’t their highest-scoring one on our board. This observation  aligns with Anthropic’s discussion around “Appropriate Harmlessness” for Sonnet, aiming to reduce refusals for benign prompts. Our tests suggest this tuning might have introduced slight vulnerabilities in the pursuit of improved helpfulness.
• Older Models Receiving Patches: Several older models, including GPT4o-mini and Gemini 1.5 Pro received revisions since our last tests. These seemed to add some additional safeguards. The data suggests these patches incorporate learnings from newer models to address common jailbreaks, which is a positive development for model security maintenance, however we still would recommend additional safeguards if using these models. With scores of 41 and 27 respectively they still score well below our acceptable threshold.
  
• Shift Towards Reasoning? With Anthropic releasing models like Claude 3.7 Sonnet, their first “hybrid reasoning model”, and Google quickly iterating from Gemini 2.0 Pro to the more advanced “thinking” version 2.5 Pro, we’re observing a potential trend. Are major providers shifting focus from releasing general base models towards models specifically enhanced for reasoning capabilities? If this trend holds, it could have significant implications for the attack surface of future models, as we’ve seen enhanced reasoning capabilities introduce new vulnerabilities.