How to Evaluate an Autonomous Pentesting Platform: Pilot Plan and Scorecard

Most failed evaluations fail here: the pilot starts before anyone has agreed what a good result is.

1. 1Pick a representative scope. One or two internal segments with real workloads, an Active Directory domain if you run one, and part of the external perimeter. A clean lab hides exactly the problems you are buying the tool to find.
2. 2Write the rules of engagement. Excluded hosts and ranges, testing windows, which actions are never allowed in production, who can stop the test, and how to reach them.
3. 3Plant known weaknesses. In a test segment, set up a handful of issues your team already understands: a reused local admin password, an unpatched service, an overly broad share. They give you a known answer to measure misses against.
4. 4Define success in writing. For example: no action outside scope, every finding proven, invalid findings below a level you choose, most planted weaknesses found, re-test working on fixed issues.
5. 5Define the stop conditions. Any action outside scope, any service disruption that was not approved, or any data leaving the environment ends the pilot.
6. 6Name the people. An owner on your side, the approvers for gated actions, and the analysts who will review findings.

Test safety first, because a platform that cannot be controlled cannot be run often, and frequency is the reason to buy it.

• Hard scope. Put an excluded host inside a scoped range and check that the agent never touches it, and that the attempt, if any, is logged. Scope should be enforced by the platform, not left to the model's judgment.
• Emergency stop. Trigger it in the middle of a run. Every running action should halt, and the log should show when.
• Approval gates. Ask which actions run on their own, which wait for a human, and whether you can change that per campaign. Each approval should record who approved what, and when. See human-in-the-loop AI security testing.
• Rate and impact limits. Check how the platform limits load, lockouts and disruptive techniques on fragile systems such as OT, printers or legacy servers.
• Full action log. You should be able to reconstruct every command the agent ran, on which host, and why.

An autonomous agent can describe a vulnerability that a version string only suggests, or believe an exploit worked when it did not. Proof is what separates a finding from a guess.

• For every finding, ask for: the exact request or technique used, the target's response, and what the agent could do after access, with timestamps.
• Check that evidence cannot be edited afterwards. Signed and chained records let an auditor confirm that the report matches what happened.
• Pick three findings and reproduce them with your own team from the evidence alone. If they cannot, the evidence is not good enough for remediation or for an auditor.
• Check what the evidence contains. Captured credentials and data samples are sensitive: ask where they are stored, who can see them, and how they are deleted at the end.

Count what your analysts accept, not what the dashboard shows.

• Invalid findings. Have your analysts mark every finding as valid, invalid or duplicate. The share that is invalid is the cost you will pay in triage every week. In the ICLR 2026 comparison of AI agents and professional pentesters, even the best agent had about one in five submissions rejected, so ask each vendor how theirs is measured.
• Misses. Compare the results against the weaknesses you planted. A tool that reports a lot but misses your known issues is noisy, not thorough.
• Severity in context. Check whether severity reflects what an attacker could reach from the finding in your estate, not only the CVSS score.
• Duplicates and noise. One root cause should be one finding with many affected hosts, not fifty tickets.

• Attack paths, not lists

  The value of an autonomous red team is chaining: a weak credential to a server, to a domain admin, to a target system. Ask for at least one path shown step by step.

• Internal and external

  Check both, and whether the agent can start from an assumed-breach position on an ordinary workstation.

• Identity

  Active Directory and cloud identity misconfigurations are where real attacks move. See what the agent tests there and how it proves it.

• Black box and gray box

  Run once with no credentials and once with a standard user account; the difference shows how much the agent depends on what you give it.

• What it does not test

  Business logic in custom applications, social engineering and physical access usually stay with people. A vendor that claims everything is a red flag.

The pilot is a few weeks; the tool runs for years. Score the work around it.

• Re-testing. Fix two findings and ask the platform to re-test them. It should replay the original path and show that it no longer works.
• Hand-off. Check the export to your ticketing system and the remediation guidance: is it specific enough for the team that owns the system?
• Reports. Look at the executive summary and the technical report separately, and at mapping to the frameworks you report against, such as NIS2, DORA, PCI DSS or ISO 27001.
• Analyst time. Measure how long triage takes per finding. That number, multiplied by run frequency, is the real staffing cost.
• Deployment and data. Where the models run, what leaves the network and whether it works air-gapped are covered in the 15-question on-premise buyer's checklist.

• No pilot on your network,

  only demos on the vendor's lab or a public challenge set.

• Benchmark scores without method

  A score on challenges the vendor could study in advance is not blind. Ask who ran the measurement and on what.

• Findings without proof,

  or proof limited to screenshots and scanner output.

• Scope that depends on the model

  following instructions, rather than a limit the platform enforces.

• No emergency stop you can trigger yourself,

  or one that takes minutes to act.

• Your data leaves the network

  for inference or analysis without a clear, contractual answer on where it goes.

• "Replaces your pentesters entirely."

  Regulator-led tests such as DORA TLPT still require human teams.

1. 1Week 0: preparation. Agree scope, rules of engagement, success and stop criteria; plant the known weaknesses; install the platform and connect approvers.
2. 2Week 1: safety tests and first run. Test scope enforcement and the emergency stop on purpose, then run a black-box campaign on the external perimeter and one internal segment at the most conservative autonomy level.
3. 3Week 2: depth. Run a gray-box, assumed-breach campaign with a standard user account. Review attack paths and have analysts mark every finding.
4. 4Week 3: fix and re-test. Remediate a few findings and the planted weaknesses, then re-test. Reproduce three findings from the evidence alone.
5. 5Week 4: score and decide. Fill in the scorecard, measure analyst time per finding, and compare against the criteria you wrote in week 0.

To see how autonomous penetration testing works end to end before the pilot, start with the guide. To compare platforms on paper first, use the automated pentesting tools comparison and the OWASP Autonomous Penetration Testing Standard as a common language for requirements.

How long should an autonomous pentest pilot last? Two to four weeks is usually enough: one week for safety tests and a first run, one for a deeper gray-box run, one to fix and re-test, and time to score. Shorter pilots rarely include a re-test, which is where many tools fall short.

Is it safe to run a pilot on production? Only with a platform-enforced scope, an emergency stop you control, and human approval for disruptive actions. Start at the most conservative autonomy level and on segments where an outage would be tolerable.

How do I measure false positives? Have your analysts mark every reported finding as valid, invalid or duplicate, and plant known weaknesses to measure misses. Both numbers matter: a low false-positive rate with many misses is not a good result.

Should I trust vendor benchmark scores? Only if the measurement was blind and run by someone other than the vendor. Results on public challenge sets the vendor could study are not blind. A pilot on your own network is a better guide.

What should the contract say after the pilot? Scope and rules of engagement, who approves gated actions, where data and models live, how evidence is stored and deleted, and the re-test process. The on-premise buyer's checklist linked above lists the data and deployment questions.

Where does Zero Hunt fit? Zero Hunt is an autonomous AI red team on an on-premise appliance that runs its own models, so test data stays in your network. Scope limits and an emergency stop are enforced by the platform, operators choose one of five autonomy levels that decide which actions wait for approval, and every finding is backed by Ed25519-signed, hash-chained evidence. It publishes no benchmark numbers yet; its evaluation methodology is in preparation, and it can be evaluated with the pilot described here.