Most evaluations of artificial intelligence take place in controlled settings, under conditions chosen by the people who built the test. That is convenient and safe, but a narrow test may capture only part of what a system can do. In cybersecurity, an underestimated capability means an underestimated threat, so the gap between what is measured and what is possible matters.
Sai Yeswanth Maturi’s paper, “Crowdsourced Frontier: Unveiling Autonomous Adversarial Cybercapabilities via Open AI Competition,” looks at this problem. Instead of testing autonomous agents in a closed environment, he developed a crowdsourced elicitation framework and used data from two public capture-the-flag competitions to evaluate autonomous AI agents against human teams. The first was the AI vs. Humans CTF, conducted in collaboration with Hack The Box. The second was Cyber Apocalypse, which drew more than 8,000 teams. Both offer standardized challenges, public scoring, and detailed data across cryptography, reverse engineering, binary exploitation, and web exploitation.
The idea behind the approach is that when a large and varied community takes part, many people try their own prompting and tooling methods to get the best results from the agents. Using data from both events, Maturi compared the agents with human teams across different cybersecurity domains. He argues that this kind of open testing can show capabilities a single in-house team would probably miss.
What the results showed
According to the paper, four of the seven agents in the AI vs. Humans event solved 19 of 20 challenges, and the strongest placed in the top 5% of 403 human teams. In Cyber Apocalypse, where the tasks involved more complex interaction with networks and systems, the agents ranked roughly in the top 10% of 8,129 teams, though their solve rate was 32.2%.
To put these numbers in context, the study uses a measure called the 50% task-completion time horizon. It describes how much human effort a task typically takes when an AI can complete it about half the time. Across both datasets, the horizon was about one hour. In other words, the agents could generally handle problems that would take a focused person around sixty minutes. The paper reads this as a sign that AI is approaching human performance in short-term tactical reasoning.
The results differed by domain. The agents did best in cryptography and reverse engineering, which reward systematic search and computational precision. They did worse in web exploitation, where the target keeps changing and the attacker has to adapt. The paper concludes that machines bring speed and endurance, while people bring hypothesis generation, situational awareness, and judgment, and it suggests that mixed human-AI teams could outperform either alone.
Limits and wider implications
The paper also lists limitations. The agents had trouble keeping track of state across long, multi-step challenges. Small changes to prompts or environments could change results significantly. Some of the best performers relied on non-deterministic reasoning, which made their results hard to reproduce.
Maturi draws a broader conclusion from this. If narrow testing understates what offensive AI can do, he argues, open and transparent evaluation could serve as an early-warning tool for policymakers and defenders. Spreading the work across many participants also reduces reliance on any one team’s assumptions, and the paper presents these competitions as a repeatable way to evaluate AI rather than a one-time experiment.
The same question applies outside capture-the-flag events: whether an organization’s own testing is demanding enough to show how an AI system behaves under unexpected conditions. Maturi works on this in his job at Maturion Solutions LLC, where he helps test and validate enterprise conversational AI that interacts with business rules, backend data, and cloud environments. His work there involves finding edge cases and checking that the system behaves reliably.
The approach in both settings is similar: test under pressure, assume the first result may understate what a system can do, and keep people involved where judgment is needed. This matches the paper’s conclusion that automation works best alongside human oversight. As AI systems are given access to customer records and operational platforms, that kind of scrutiny is likely to matter more.
