New XRanges Platform Verifies Performance of Autonomous AI Security Agents

Autonomous security agents are increasingly capable of identifying software vulnerabilities, yet verifying the accuracy of their findings remains a significant technical hurdle. Because these agents often produce self-reported findings in the form of written prose, security professionals face difficulty distinguishing between actual exploits, near-misses, and entirely fabricated claims.

To address this, CTF.ae has launched XRanges for AI, a platform designed to provide objective, telemetry-based scoring for autonomous security agents. The system moves away from relying on an agent’s own reports, instead using deep instrumentation within target environments to observe actual behaviour.

Telemetry-Driven Verification

The platform works by deploying realistic, multi-service target applications that are heavily instrumented using OpenTelemetry. This allows the system to record every action an agent takes from inside the target, rather than relying on the agent to report its own progress. This method enables the generation of four independent scoring signals:

• Coverage: This measures how thoroughly an agent explores a target by tracking the completion of legitimate business actions, such as registering an account or browsing job postings.
• Boundaries: This tracks whether an agent respects established rules of engagement, such as prohibitions against deleting content or revoking API keys.
• Exploited: This verifies which vulnerabilities were actually compromised by tracking the specific kill chain phases required for a successful exploit.
• Integrity: This monitors the functional health of the target environment to ensure the agent does not break the system or corrupt data during its assessment.

By using these signals, developers can identify an agent’s blind spots, such as specific APIs or features the agent failed to enumerate, regardless of what the agent’s final report claims.

Field Testing at DEF CON 34

The efficacy of the platform was demonstrated during DEF CON 34 in August 2026. Within the Bug Bounty Village, CTF.ae utilised the technology to monitor 545 participants competing in a Capture The Flag (CTF) competition. Each player was provided with an isolated instance of a fictional company named Xenoptic.

By monitoring over 850 deployments, the platform provided real-time data on how each participant—and any agents they employed—moved through the environment. This ensured that submitted reports were validated against the actual paths taken by the players, preventing participants from gaining advantages through unintended bugs or external exploits that bypassed the application logic.

Deployment and Use Cases

XRanges for AI is intended for engineering teams developing autonomous pentesting or bug bounty agents who require an automated way to conduct large-scale experiments across different models and configurations. The platform supports running many deployments simultaneously to help researchers identify variance in agent performance through an experiment matrix.

The service is available as a managed cloud offering via ai.xranges.com or as a self-hosted solution for organisations that require their data and target environments to remain within their own infrastructure. Detailed technical data from runs is accessible through an API and a Model Context Protocol server.