AI Models Agree on Only One Out of 121 Potential Code Flaws
Share
Frontier artificial intelligence models used to audit open-source software have produced vulnerability findings that barely overlap, according to new research. A comparison of three leading models revealed that out of 121 potential security issues, the systems agreed on only a single flaw.
The study, conducted by software security firm Echo, tested Anthropic’s Claude Mythos 5.1, OpenAI’s GPT-5.6 Sol, and Z.ai’s GLM-5.3. Each model was tasked with auditing the same codebases, including two Maven libraries and one npm package, using identical hunting frameworks and validation processes.
The results highlight a significant lack of consensus in AI-driven security analysis. Of the 121 distinct potential issues identified across the models, 83% were flagged by only one model. Only 20 issues were caught by two of the models, and just one was identified by all three.
Varying Performance and Severity Ratings
Performance metrics varied widely across the different platforms. Claude Mythos 5.1 achieved the highest number of validation passes with 25, significantly higher than GLM-5.3 which had nine, and GPT-5.6 Sol with eight.
However, GPT-5.6 Sol demonstrated specific strengths, accounting for six of the seven successful passes on one of the Maven libraries. In contrast, GLM-5.3 found six validated issues on a different library that both of its rivals missed entirely.
The models also disagreed on the severity of the risks they discovered. In one instance involving an npm package with over 350 million monthly downloads, GLM-5.3 classified a flaw as Critical. Claude Mythos rated the same issue as High, while GPT-5.6 Sol did not consider it a bug at all, treating it as documented behaviour. Automated validation eventually rated the risk as Low.
Speed vs. Cost in AI Auditing
The efficiency and cost of the audits showed sharp contrasts. OpenAI’s GPT-5.6 Sol, which was accessed through the Daybreak program for vetted security teams, was the fastest, completing its tasks in approximately 24 minutes. Claude Mythos and GLM-5.3 took considerably longer, at 198 minutes and 174 minutes respectively.
GLM-5.3 proved to be the most economical option, with three audit runs costing roughly $22. This compared to $30 for GPT-5.6 Sol and $303 for Claude Mythos, though Echo noted these figures reflected their specific access terms and rates.
Implications for Security Defenders
The findings suggest that relying on a single AI model for vulnerability discovery could leave organisations exposed to significant blind spots. Echo security researcher Yuval Saggie noted that if an attacker uses one model while a defender relies on another, the defender might never see the vulnerability the attacker intends to exploit.
The initial nine audit runs generated 446 findings, which were narrowed down to 152 after cleanup. Following automated proof-of-concept testing, 42 findings passed. Echo clarified that these 42 were not necessarily unique bugs, as multiple models could receive credit for the same issue.
Echo has not yet released technical details or the names of the affected software projects. This information is being withheld until the respective maintainers have been notified and given the opportunity to address the vulnerabilities.




Leave a Reply