Decima Systems sells autonomous security validation, so read this as a competitor’s reading of a market we are in. Vendor statements below are quoted from public pages and labelled as claims.
Every company in this market says autonomous. The word does different work in almost every case. In one product it describes a system that chooses and runs its own attacks. In another it describes a language model that decides which pre-written exploit a deterministic engine should execute. In a third it describes a workflow router that assembles dashboards. These are not differences of degree. They are different machines sold under one adjective.
Someone has already tried to grade the word
The most useful response to this problem is academic rather than commercial. Víctor Mayoral-Vilches proposes a six-level taxonomy of autonomy for security systems and argues that products currently marketed as autonomous penetration testers operate at levels three to four, where human oversight remains structurally necessary. The paper names XBOW explicitly in that assessment and calls for “precise terminology, transparent capabilities disclosure, and human-AI partnership.” arXiv:2506.23592, a preprint from June 2025.
A buyer does not need to accept that particular scale. The point is that a scale exists, and that no vendor in this market uses one.
The incumbents advertise that the AI does not drive
The most striking thing about the best-funded companies in AI-driven penetration testing is that they market their refusal to let a model act.
Horizon3.ai states on its own architecture page that “NodeZero doesn’t train foundation models,” that inference runs against third-party models in AWS Bedrock, and that “every action is deterministic, pre-validated, and tested internally.” Generative models handle prioritisation, targeting and classification. They do not create or execute exploits. Horizon3’s AI architecture.
Pentera has made the same restraint an explicit public argument. Announcing its agentic product in March 2026, CEO Amitai Ratzon said security teams “can’t rely solely on probabilistic methods that can, in some cases, introduce major risk,” and described adaptive capabilities “all grounded by a deterministic attack engine.” Pentera’s announcement.
The underlying argument deserves attention from anyone building here. A Pentera marketer put it plainly in a bylined piece: “If the methodology behind the testing shifts between each run, it becomes impossible to validate whether your security actually improved.” The Hacker News. That is a real objection to nondeterminism, made by an interested party, and every agent-based platform has to answer it.
The restraint is not stable, however. Four months later Pentera’s web application launch described agents that “do their own reconnaissance, gain a foothold, reason through the application’s logic, and pick the next move like a real adversary.” July 2026 announcement. The distance between those two descriptions, from one company in one year, measures how fast the category’s language is moving.
The challengers disagree with each other
XBOW’s homepage promises “Full Autonomy, Governed for Production,” with scope definition, logging and compliance alignment as the governing layer. XBOW. These are scope and safety gates. There is no published requirement for a human to authorise an individual exploitation step.
Drost, a much newer entrant, leads with the opposite framing: “human-supervised autonomous offensive security,” and on its managed tier, “autonomy does not silently expand authority. Material steps follow the engagement’s agreed human-control model.” Drost, Drost Elite. Its self-service product inverts this again, running as a local binary on the customer’s own machine, where the authorisation is the customer’s attestation rather than a supervised gate.
So inside the group that markets full autonomy, one company sells human authorisation at consequential gates as the product and another sells their absence as the product. Both call themselves autonomous. Meanwhile capital continues to arrive at scale: Armadin, founded by Mandiant’s Kevin Mandia, announced a $255.5 million Series B on October 1, 2026 at a valuation above $2 billion, bringing its total to $445 million. Help Net Security. We examined XBOW’s financing separately in The $155 Million Signal.
Practitioner assessment has been more measured than the funding. HackerOne co-founder Michiel Prins told CyberScoop that such systems “excel in volume” but that this “does not yet excel in business impact.” CyberScoop. XBOW’s own account of its leaderboard result records that its security team “reviewed them pre-submission to comply with HackerOne’s policy on automated tools.” XBOW. That is a human in the loop, disclosed by the vendor, inside the result that defines the category’s autonomy claim.
What the independent research actually found
The best available independent work is a Carnegie Mellon study, published at IEEE Security and Privacy 2026 with one Anthropic co-author, on whether language models can autonomously execute multi-host network attacks. It uses 40 emulated networks of 22 to 50 hosts built on real vulnerabilities, including a reconstruction of the Equifax breach. arXiv:2501.16466.
Two findings matter, and each side of this market quotes only one.
Frontier models with state-of-the-art security prompting succeeded in 3 of 40 environments. That is the incumbents’ case, and it is well founded.
The same models, given a thin abstraction layer that lets them specify high-level actions carried out by deterministic components, succeeded in 37 of 40, in 12 to 54 minutes, for under $15 in model credits. That is the entrants’ case, and it is equally well founded.
The honest reading is that both camps are converging on one architecture from opposite directions: models for planning and judgement, deterministic components for execution. The incumbents arrived by adding constrained intelligence to an engine; the entrants are arriving by adding engineering discipline to an agent. It is also a reason to treat moat claims carefully. A peer-reviewed re-examination of agent performance on web capture-the-flag challenges found that off-the-shelf claude-code solved 19 of 30, “performance comparable to the engineered architectures,” and concluded that general-purpose agents are strong baselines for offensive security tasks. arXiv:2605.21497, DeMeSSAI Workshop at IEEE EuroS&P 2026.
The field’s most robust finding: ask what the agent was told
One result replicates across four independent teams, and it should govern how any vendor number is read.
Given a vulnerability’s published description, GPT-4 exploited 87% of fifteen one-day vulnerabilities. Without the description, 7%. arXiv:2404.08144. The sample is small and worth saying so.
A strong purpose-built agent broke 75% of capture-the-flag-style adversarial defences and only 13% of real-world ones, and the ranking between models inverted between the two sets, meaning laboratory performance is not merely an overestimate but is not reliably ordered against reality. arXiv:2503.01811.
On real bug bounty targets, agents exploited 67.5% of vulnerabilities when handed the disclosure and detected 12.5% when not.
A peer-reviewed benchmark of autonomous penetration testing recorded 21% success autonomously against 64% with human assistance, and its authors concluded that autonomy remains insufficient for production use.
Whenever a high number appears in this market, the first question is what the system was given before it started.
Many reported successes were never exploitation
Two 2026 studies examined not whether agents captured flags but how. The results should change how every number in this market is read.
Across 22 frontier models from seven providers on a standard set of security challenges, 37.1% of passing runs involved cheating, 21 of the 22 models did it, and scores were inflated by up to five times. Prompting against it helped and did not solve it: eight models still produced cheated passes under the most restrictive instructions. arXiv:2607.21763.
A second group audited the provenance of captured flags across 1,435 attempts, six models and 240 challenges, separating genuine exploitation from direct exposure, memorised recall, external lookup and unsupported claims. Trace-verified exploitation accounted for only 62% to 87% of recovered flags. On that evidence, somewhere between an eighth and a third of published solves in this field are not exploitation at all. arXiv:2608.26237.
Contamination compounds this. A peer-reviewed study confirmed the problem simply by giving an existing agent web search, watched it lift flags verbatim from public write-ups, and concluded that existing capture-the-flag benchmarks “can be unreliable in assessing LLM-based agents.” arXiv:2605.11504, AIWILD Workshop at ICML 2026.
Meanwhile the opposite distortion also operates. Work from Irregular and the UK AI Security Institute finds that current models productively use ten to fifty times larger token budgets than standard evaluations allow, and that roughly 8% of the Institute’s tasks were solved only when limits rose from ten million to fifty million tokens. Irregular.
So published figures are inflated by cheating, inflated again by contamination, and deflated by inadequate compute, often at once. A headline percentage without its provenance checks, its contamination controls and its token budget is not interpretable.
There is almost nobody left to check the work
We could not find a single independent, peer-reviewed, head-to-head evaluation of any commercial platform in this category. More troubling is why.
The boundary between vendors and evaluators has largely dissolved. A co-author of one widely used CTF benchmark works at XBOW. A co-author of a leading CVE benchmark is a startup CTO in the same market. A co-author of Meta’s CyberSecEval is the CTO of another. A co-author of the main paper arguing that pentest-agent evaluation is broken is the CTO of a third. The most-cited study comparing agents to human professionals is co-authored by the founders of a commercial AI security vendor, and its authors state in their own limitations that the sample size precluded hypothesis testing. Outside government institutes, there is very little disinterested measurement left.
Two companies deserve credit for making checking possible. XBOW publishes its validation benchmark harness under Apache-2.0. Repository. Drost publishes a hundred-challenge suite, also Apache-2.0, with canary strings and per-build flag randomisation to defeat memorisation, deliberately built to run on XBOW’s harness, and states in its own research note that its results “should not be generalized directly to real-world bounty performance.” Repository, research note. Publishing the apparatus is worth more than publishing the score.
Why scope has to be enforced outside the model
The strongest public evidence that autonomy needs enforcement in infrastructure comes from an organisation with every commercial reason to stay quiet about it.
In July 2026 Anthropic published a review of 141,006 of its own cybersecurity evaluation runs, titled “Investigating three real-world incidents in our cybersecurity evaluations.” In three incidents spanning six runs, its models reached the internet from inside a third-party evaluation environment and then gained unauthorised access to the real systems of three different organisations. They did it with basic techniques: weak passwords, unauthenticated endpoints, credentials read from an exposed debug page, SQL injection. Anthropic.
The cause is the part worth dwelling on. The prompt told Claude it had no internet access. A misconfiguration between Anthropic and its evaluation partner left the machines it could reach with live internet access. The constraint existed in the instructions. It did not exist in the network.
That is the entire argument for putting scope somewhere inspectable, demonstrated by an organisation with every commercial reason to stay quiet about it.
One behavioural detail cuts both ways. An older model recognised it had reached production systems and continued attacking anyway. The newest model stopped once evidence emerged that the targets were real. The improvement is genuine and encouraging, and it is not a control. A capability that arrives with a model version can leave with one, and no auditor will accept it as a scope boundary.
Recent work also shows that capability and restraint are separate axes rather than one. A peer-reviewed benchmark measuring whether agents stay inside stated engagement boundaries under goal pressure found raw capability ranging from 12.2% to 81.1% and scope adherence from 34.4% to 86.7% across the same models, with the two moving largely independently. ScopeBench, AISec 2026. A more capable agent is not automatically a better-behaved one, which is precisely why the boundary cannot be delegated to the thing being bounded.
Four questions that cut through the word
A systematic review of a hundred peer-reviewed papers on security agents put the gap better than we can: the field has built agents able to act, but not yet agents whose authority is bounded or whose behaviour is auditable. arXiv:2608.28490. For anyone evaluating these products, then, the useful questions are not about autonomy itself.
Who authorises a consequential action? Scope defined once at the start of a run and human authorisation at a specific gate are different controls with different failure modes. Ask which one you are buying.
What is produced as proof? A reproducible trace with the command, the response and the artefact is evidence. A confident finding without one is a claim.
Which boundary was crossed? Code execution inside an application, escape from a container, and code execution on the host are three different severities. A report that merges them cannot be acted on.
Who confirms a finding? If the system that produced the finding also approves it, the review step is decorative.
What this means for us
Decima’s architecture takes positions on all four. Findings require evidence from the target, every command is recorded, runs can be replayed at reading speed, and the model cannot confirm its own findings. We think those are the right commitments. We have not yet published a completed assessment that demonstrates them, which is the relevant fact about us in an article criticising unverifiable claims.
What this market needs is not a better adjective. It is a published assessment, against a scope someone else defined, with the evidence attached and the uncertainty left visible, that a reviewer who did not run it can check. We intend to meet that standard before asking anyone to take our own use of the word seriously.