Download Privacy Needle App

Type to search

Cybersecurity

AI Models Agree on Only One Out of 121 Potential Code Flaws

Share
A digital representation of artificial intelligence scanning lines of computer code for security vulnerabilities.

Frontier artificial intelligence models used to audit open-source software have produced vulnerability findings that barely overlap, according to new research. A comparison of three leading models revealed that out of 121 potential security issues, the systems agreed on only a single flaw.

The study, conducted by software security firm Echo, tested Anthropic’s Claude Mythos 5.1, OpenAI’s GPT-5.6 Sol, and Z.ai’s GLM-5.3. Each model was tasked with auditing the same codebases, including two Maven libraries and one npm package, using identical hunting frameworks and validation processes.

The results highlight a significant lack of consensus in AI-driven security analysis. Of the 121 distinct potential issues identified across the models, 83% were flagged by only one model. Only 20 issues were caught by two of the models, and just one was identified by all three.

Varying Performance and Severity Ratings

Performance metrics varied widely across the different platforms. Claude Mythos 5.1 achieved the highest number of validation passes with 25, significantly higher than GLM-5.3 which had nine, and GPT-5.6 Sol with eight.

However, GPT-5.6 Sol demonstrated specific strengths, accounting for six of the seven successful passes on one of the Maven libraries. In contrast, GLM-5.3 found six validated issues on a different library that both of its rivals missed entirely.

The models also disagreed on the severity of the risks they discovered. In one instance involving an npm package with over 350 million monthly downloads, GLM-5.3 classified a flaw as Critical. Claude Mythos rated the same issue as High, while GPT-5.6 Sol did not consider it a bug at all, treating it as documented behaviour. Automated validation eventually rated the risk as Low.

Speed vs. Cost in AI Auditing

The efficiency and cost of the audits showed sharp contrasts. OpenAI’s GPT-5.6 Sol, which was accessed through the Daybreak program for vetted security teams, was the fastest, completing its tasks in approximately 24 minutes. Claude Mythos and GLM-5.3 took considerably longer, at 198 minutes and 174 minutes respectively.

GLM-5.3 proved to be the most economical option, with three audit runs costing roughly $22. This compared to $30 for GPT-5.6 Sol and $303 for Claude Mythos, though Echo noted these figures reflected their specific access terms and rates.

Implications for Security Defenders

The findings suggest that relying on a single AI model for vulnerability discovery could leave organisations exposed to significant blind spots. Echo security researcher Yuval Saggie noted that if an attacker uses one model while a defender relies on another, the defender might never see the vulnerability the attacker intends to exploit.

The initial nine audit runs generated 446 findings, which were narrowed down to 152 after cleanup. Following automated proof-of-concept testing, 42 findings passed. Echo clarified that these 42 were not necessarily unique bugs, as multiple models could receive credit for the same issue.

Echo has not yet released technical details or the names of the affected software projects. This information is being withheld until the respective maintainers have been notified and given the opportunity to address the vulnerabilities.

Watch Our Latest Video
Stay ahead with expert insights on privacy, cybersecurity, artificial intelligence, data protection and compliance.
Privacy Needle AI | AI DPO Assistant & NDPA Compliance Software
Published: November 2, 2026
Daily Privacy News
Cybersecurity Updates
Data Protection Tips
GDPR & NDPA Explained
Tags:
Ikeh James Certified Data Protection Officer (CDPO) | NDPC-Accredited

Ikeh James Ifeanyichukwu is a Certified Data Protection Officer (CDPO) accredited by the Institute of Information Management (IIM) in collaboration with the Nigeria Data Protection Commission (NDPC). With years of experience supporting organizations in data protection compliance, privacy risk management, and NDPA implementation, he is committed to advancing responsible data governance and building digital trust in Africa and beyond. In addition to his privacy and compliance expertise, James is a Certified IT Expert, Data Analyst, and Web Developer, with proven skills in programming, digital marketing, and cybersecurity awareness. He has a background in Statistics (Yabatech) and has earned multiple certifications in Python, PHP, SEO, Digital Marketing, and Information Security from recognized local and international institutions. James has been recognized for his contributions to technology and data protection, including the Best Employee Award at DKIPPI (2021) and the Outstanding Student Award at GIZ/LSETF Skills & Mentorship Training (2019). At Privacy Needle, he leverages his diverse expertise to break down complex data privacy and cybersecurity issues into clear, actionable insights for businesses, professionals, and individuals navigating today’s digital world.

  • 1

You Might also Like

Leave a Reply

Your email address will not be published. Required fields are marked *

  • Rating

This site uses Akismet to reduce spam. Learn how your comment data is processed.