AI Pentesting, Part 14: Automating Pentesting with AI: Architectures, Ecosystems, and Evaluation

1. Open source, research, and commercial systems

The ecosystem changes quickly. This is a dated architectural map, not an exhaustive product ranking or a guarantee of support. Public marketing rarely discloses a complete internal architecture. Do not assume a commercial platform uses a specific model, multi-agent framework, or prompt strategy unless the vendor documents it.

The CAI repository reviewed for this article is archived and warns that it is unmaintained; treat it as a research artefact, not a supported production recommendation. PentestGPT's current README distinguishes a staged autonomous runtime from its modernised legacy interactive workflow. Reproduce the exact revision instead of assuming a paper's architecture matches today's code.

2. The common controller architectures

These patterns can be combined. A task graph may use a ReAct loop inside one bounded node and a deterministic verifier afterwards. 'Multi-agent' does not mean independent verification: two agents may share the same model, prompt, evidence gap, and mistaken assumption.

Use an explicit state machine: proposed, authorised, executing, observed, verified, rejected, or unresolved. Only the verifier can mark a finding verified, and its criteria should be inspectable. A chat message saying 'done' must not advance the state on its own.

3. Technologies that make the architecture practical

Structured tool calling and JSON Schema constrain interfaces. MCP standardises discovery and invocation across tools. Browser automation handles dynamic application state. Sandboxes and containers isolate execution, but network egress and host mounts still need explicit controls. Durable queues and workflow engines handle retries, leases, and crash recovery. SQL or object stores preserve evidence; graph stores can represent assets and hypotheses.

RAG retrieves approved policies, prior findings, and product documentation. Vector similarity is not authority: filter by engagement, asset, version, and source trust before retrieval. General orchestration libraries such as LangGraph, AutoGen, and CrewAI can implement parts of the controller, but none automatically supplies a secure pentest policy, reliable finding verifier, or production-safe toolset.

scope manifest -> durable task graph -> proposal
proposal -> deterministic policy gate -> bounded tool adapter
tool result -> immutable evidence store -> verifier
verifier -> verified / rejected / unresolved
all transitions -> audit log, budget ledger, cancellation state

Model gateways add routing, cost accounting, provider policy, and redaction. Avoid passing credentials through free-form prompts. Keep secrets in the tool layer and return minimal evidence. Fine-tuning may improve task behaviour; it does not establish authorisation or eliminate hallucination.

4. A no-network controller prototype

This offline example demonstrates the control boundary: the model-like proposal can name only one reviewed action, and a verifier checks an actual fixture observation. A real system needs authenticated approvals, durable state, and independent execution isolation.

def policy(proposal):
    return (set(proposal) == {"action", "asset"}
            and proposal["action"] == "read_fixture"
            and proposal["asset"] == "lab-web-01")

def execute(proposal):
    if not policy(proposal):
        raise PermissionError("proposal outside policy")
    return {"marker": "PSC-LAB-HEALTH", "evidence_id": "fixture-014"}

def verify(observation):
    return (observation.get("marker") == "PSC-LAB-HEALTH"
            and observation.get("evidence_id") == "fixture-014")

proposal = {"action": "read_fixture", "asset": "lab-web-01"}
assert verify(execute(proposal))
assert not policy({"action": "shell", "asset": "lab-web-01"})
assert not policy({"action": "read_fixture", "asset": "other-host"})
print("approved fixture verified; forbidden proposals rejected")

Do not generalise this tiny verifier to vulnerability proof. A marker check validates one observation only. Real authorisation findings need policy and identity evidence; memory-safety findings need root-cause and reproducibility evidence. The verifier must match the claim.

5. Evaluate beyond CTF success rate

Build a holdout lab with known positive cases, clean controls, misleading banners, expired sessions, missing sensors, and hostile prompt text. Keep the evaluator's ground truth outside the agent context. Run repeated trials with fixed tool versions and budgets, and report variability. A benchmark score from a public corpus may reflect training contamination or task familiarity.

Measure precision as verified true findings divided by reported findings, and recall as discovered ground-truth issues divided by the known issue set. Track verified findings per unit cost, time to evidence, unauthorised-action attempts, health incidents, and unsupported claims. A tool-call refusal can be a correct outcome, not a failed solve.

Compare against a deterministic scanner baseline and a human-assisted baseline under the same time and access constraints. Separate model changes from harness changes. A system that solves a single-service CTF does not automatically handle enterprise identity, business logic, fragile services, or contractual exclusions.

6. Build versus buy: ask for observable controls

For a commercial pilot, request a complete action log, exact scope semantics, protected URLs, rate limits, test windows, pause/cancel behaviour, authentication support, data residency, retention, provider subprocessors, evidence export, and retest workflow. Verify these in a staging environment. Vendor statements about safety or zero hallucinations are claims to evaluate, not independent assurance.

For open source, inspect maintenance, licence, dependencies, telemetry defaults, tool permissions, and reproducible tests. Pin commits and images. An archived security framework can become a supply-chain and compatibility risk. Do not install an unreviewed agent with a host Docker socket, production cloud credentials, and unrestricted internet access.

A conservative adoption path is summarisation, read-only evidence tools, bounded lab actions, then carefully approved active workflows. Promote only after measurable gates pass. Keep humans for policy interpretation, impact judgement, unusual workflows, and final reporting. Part 15 completes the series with cloud configuration, privilege relationships, and evidence-backed deliverables.

7. Step-by-Step Lab: Evaluate an Agent Controller Against a Known Baseline

Prerequisites: An offline runtime and labelled synthetic task outcomes. Start without tool execution, network access, production secrets, or agent framework installation. The exercise evaluates controller policy and result accounting; it does not measure autonomous exploitation ability.

1. Write a tiny catalogue with one read-only fixture action, explicit assets, budget, cancellation state, and a verifier contract. Reject unknown actions before any executor is invoked.
2. Use the no-network controller prototype earlier in the article as the deterministic baseline. Record the exact proposal, policy decision, observation, and verifier output.
3. Build a labelled evaluation set with one true finding, one known false-positive report, one intentionally missed positive, and one clean control. Keep the labels outside any model prompt.
4. Run the scoring fixture below. Confirm the denominators are correct and that an empty report set is not silently described as perfect precision.
5. Add forbidden proposals, expired approvals, stale sessions, cancellation, and a prompt-injection string. A correct policy refusal should count as a successful safety control rather than an unsuccessful 'solve'.
6. If comparing models or harnesses later, keep tool versions, data, time, request limits, token budget, and scope identical. Change one variable at a time and repeat trials.
7. Promote only a measured improvement into the next stage: read-only evidence analysis, then separately approved bounded lab actions. Preserve the human review and stop mechanism instead of treating autonomy as the goal.

const truth = new Set(["AUTHZ-1", "PARSER-2"]);
const reported = new Set(["AUTHZ-1", "NOISE-9"]);
const truePositive = [...reported].filter(id => truth.has(id)).length;
const falsePositive = reported.size - truePositive;
const falseNegative = [...truth].filter(id => !reported.has(id)).length;
const precision = reported.size ? truePositive / reported.size : null;
const recall = truth.size ? truePositive / truth.size : null;
if (precision !== 0.5 || recall !== 0.5 || falsePositive !== 1 || falseNegative !== 1) throw Error("metric mismatch");
console.log(JSON.stringify({ truePositive, falsePositive, falseNegative, precision, recall }));

Expected result: One true positive, one false positive, one missed positive, precision 0.5, and recall 0.5. This evaluates only the labelled fixture set. A public benchmark result or vendor claim cannot establish the same values for your enterprise environment or business logic.

8. Worked Examples and Expected Outcomes

Example A: an agent produces more findings but less value

The total report count doubles while confirmed findings remain constant and reviewer time increases. Compare precision, duplicated root causes, and cost per verified result. More prose and more tool calls are not evidence of improvement.

Example B: multiple agents share a mistake

Three specialists read the same truncated evidence and agree. Their agreement is correlated, not independent. Give the verifier raw observations and explicit expected policy, and preserve a counterexample test.

Example C: a demo completes after scope expiry

The task entered a queue before the window closed, but dispatch happened afterwards. Require approval and scope revalidation at execution time. Durability should preserve a cancelled or expired decision, not revive authority.

{"trial":"CTRL-021","model":"record_exact_id_if_used","harness_revision":"fixture-r1","ground_truth":"held_out","request_budget":0,"policy_refusals":"measured_separately","claims":"synthetic evaluation only"}

9. Troubleshooting, Retesting, and Completion Evidence

Retest and completion: Keep the baseline, held-out labels, trial configuration, failures, policy decisions, costs, and raw evidence. Validate crash recovery and cancellation before unattended use. A fixed synthetic marker verifier must never be represented as a general vulnerability verifier.

10. Real Tool Workflow: Use a Deterministic CLI Harness as an Agent Evaluation Baseline

Prerequisites: Node.js 22 or newer, the reviewed downloadable hackbot lab module, a private disposable directory, and optionally Git for source revision tracking. No autonomous pentest framework, external target, cloud key, Docker daemon, or live model is required for this baseline.

Use reviewed releases and record the versions actually installed. Command references for this workflow were reviewed on 29 October 2026; product support and flags can change. Instructions and output shapes are not claims of a live assessment.

Step-by-Step Tool Operation

1. Obtain the two reviewed lab modules from the site's build guide and record their hashes. Establish a clean source revision or release record before evaluating an AI planner.
2. Run vulnerable and fixed modes into two new output directories. Inspect report status, executed checks, request counts, findings, and limitations; do not use stdout alone as the evidence ledger.
3. Create the tiny rejected-plan fixture below and confirm an unsupported shell action fails before any output run is started. A policy refusal is expected control behaviour, not a failed vulnerability-discovery attempt.
4. Compare the known seeded result set with reported results and calculate precision/recall using the earlier held-out example. Keep evaluator labels outside any model prompt.
5. Only after the deterministic baseline is reliable, test the optional local-model planner through its fixed local gateway and separately review its proposal. The planner must not gain the executor's tool authority.
6. For a framework or commercial pilot later, require the same scope, request/time/token budgets, evidence exports, cancellation behaviour, and workload. Preserve failed trials and review cost; do not publish a leaderboard result as a production guarantee.

node --version
Get-FileHash -LiteralPath .\hackbot-lab.mjs -Algorithm SHA256
node .\hackbot-lab.mjs --out .\baseline-vulnerable
node .\hackbot-lab.mjs --secure --out .\baseline-fixed
'{"checks":["shell"]}' | Set-Content -Encoding Ascii .\rejected-plan.json
node .\hackbot-lab.mjs --plan .\rejected-plan.json --out .\must-not-run
if ($LASTEXITCODE -eq 0) { throw 'The forbidden plan unexpectedly succeeded' }
Test-Path -LiteralPath .\must-not-run
$Before = Get-Content -Raw .\baseline-vulnerable\report.json | ConvertFrom-Json
$After = Get-Content -Raw .\baseline-fixed\report.json | ConvertFrom-Json
$Before | Select-Object status, requests, maxRequests, limitations
$After | Select-Object status, requests, maxRequests, limitations

Expected Evidence and Interpretation

The default full plan should produce two labelled lab observations in five requests before the fix and zero findings in four requests after it. The forbidden-plan command should exit nonzero and must-not-run should not exist. Use fresh paths; a pre-existing directory would invalidate the latter check. The shipped code is a teaching baseline, not a PentestGPT deployment or a claim to replicate any vendor platform.

If a later agent generates more candidate reports, compare verified findings and reviewer cost under the same known controls. A system that ignores cancellation or produces unsupported claims should not be promoted merely because it completes a CTF or emits plausible narratives.

Troubleshooting and Cleanup

The rejected plan contains ASCII-only JSON and is written with an explicit ASCII encoding so the UTF-8-reading CLI can parse it on both Windows PowerShell and modern PowerShell. Confirm that rejection names the unsupported action rather than a decoding error. Retain the evaluated source revision and redacted metrics, and remove temporary evidence directories only after the comparison is complete.