AI Pentesting vs Human Pentesters: What Each One Finds in 2026

The paper "Comparing AI Agents to Cybersecurity Professionals in Real-World Penetration Testing" (ICLR 2026) is the first published comparison on a live production network rather than a capture-the-flag lab. The setup, as the authors describe it:

• Environment: a university network of about 8,000 hosts across 12 subnets, with the institution's consent and its IT team watching.
• Participants: 10 professional penetration testers, 6 existing AI agents, and ARTEMIS, a multi-agent framework built by the authors.
• Result: ARTEMIS placed second overall. It found 9 valid vulnerabilities with an 82% valid submission rate and finished ahead of 9 of the 10 humans. The existing agents ranked well below it.
• Cost: the authors estimate some ARTEMIS variants at about $18 per hour, against about $60 per hour for professional penetration testers.

The limits matter as much as the headline. It is one network. Active work was capped at about 10 hours over 4 days of access. The IT team approved actions the system flagged, so the agent was not unsupervised. And the authors built the winning agent, which is normal for research but means the result describes what a well-engineered system can do, not what any product on the market does.

• Breadth

  An agent can enumerate and probe every reachable host in the scope on every run. A human team samples, because days are finite.

• Speed and repetition

  The same techniques run the same way at 3 a.m. on a Sunday. That makes results comparable over time, which audits and boards ask for.

• Re-testing

  After a fix, the agent can replay the exact path that worked and show whether it still does. With people, a retest is usually a new booking.

• Chaining known techniques at scale

  Credential reuse, misconfigurations and known CVEs combined across hundreds of hosts is tedious for people and routine for software.

• Cost per hour

  The authors put some agent variants at about $18 an hour against about $60 for a professional. That changes what is affordable: testing every change, not once a year.

• Business logic

  Knowing that a refund flow should never accept a negative amount, or that a role should not see another branch's data, needs an understanding of the business that agents still lack.

• Novel chains

  Creative combinations nobody has written down, especially across application and people, remain a human strength.

• Judgment on impact

  Deciding that a medium finding is critical in this organisation, and explaining why to a board, is context work.

• Social engineering and physical access

  Out of scope for software agents, and often the real entry point.

• Regulator-led testing

  DORA TLPT, TIBER-EU and CBEST require external, threat-led testers. Autonomous testing supports that work between exercises; it does not replace it.

Humans and agents make different mistakes, and a testing program has to control for both.

Agents can claim findings that are not real. A model can misread a response, assume an exploit worked, or describe a vulnerability that a version string only suggests. The study's own numbers show it: even the best agent had about one in five submissions rejected. The control is proof: for every finding, the exact request or exploit, the response, and what was done after access, recorded so it cannot be edited afterwards.

Agents can act outside what you intended. On a live network that is a safety question, not a quality one. The controls are a hard scope, an emergency stop, and approval gates for any action that could disrupt or exfiltrate. Ask any vendor which actions wait for a human, and whether every approval is logged with a name and time. See human-in-the-loop AI security testing.

Humans miss what there was no time for. A yearly test covers a sample of the estate on the day it ran. Anything deployed the week after is untested until next year.

1. 1Let autonomous testing own breadth and frequency. Run autonomous penetration testing continuously or on every significant change, across the whole internal and external scope.
2. 2Gate the risky actions. Choose the autonomy level per campaign: what runs on its own, what waits for approval, what never runs in production.
3. 3Accept only proven findings. No proof of exploitation, no ticket.
4. 4Spend human hours where judgment pays. Business logic, crown-jewel scenarios, social engineering, and validating the top findings the agents surface.
5. 5Keep the regulated tests external. TLPT and similar exercises stay with accredited testers; continuous testing gives them a cleaner starting point.
6. 6Re-test every fix automatically. Close the ticket only when the original path no longer works.

For how the automated families differ (scripted tools, breach and attack simulation, AI agents), see automated penetration testing.

• Ask whether a measurement was blind. Results a vendor produced on public challenge sets it could study are not blind, and none of the commercial vendors publishes an independent, blind measurement of its exploitation rate.
• Ask for one finding end to end on your scope: exploit, response, evidence after access.
• Ask where the model runs and what data it sees. A pentest gives the tool the most sensitive map of your estate.
• Ask what happens when it is wrong. Who reviews findings before they reach you, and how false positives are measured.

The automated pentesting tools comparison applies these questions to ten platforms, and the cost guide covers what they charge.

Can AI replace penetration testers? Not entirely. In the ICLR 2026 study the best agent beat 9 of 10 professionals on findings, but humans still lead on business logic, novel chains and judging impact, and regulator-led tests require human teams. AI replaces the repetitive share of the work, not the judgment.

Is AI pentesting accurate? It depends on the system and on whether findings are proven. The study's best agent had an 82% valid submission rate, and the off-the-shelf agents ranked well below it. Accept only findings that come with proof of exploitation.

Is it safe to run an autonomous AI pentest on production? Only with a hard scope, an emergency stop and human approval for disruptive actions. In the study, the IT team approved flagged actions.

What does an AI pentest cost compared with a human one? The authors estimate some variants of their agent at about $18 per hour, against about $60 per hour for a professional pentester. Commercial platforms are priced by subscription or usage; see the cost guide linked above.

Where does Zero Hunt fit? Zero Hunt is an autonomous AI red team on an on-premise appliance that runs its own models, so test data stays in your network. Operators choose one of five autonomy levels that decide which actions wait for approval, and every finding is backed by Ed25519-signed, hash-chained evidence. It publishes no benchmark numbers yet; its evaluation methodology is in preparation.