AI Pentesting, Part 3: LLM-Assisted AppSec: Ask Questions That Produce Tests

1. Build a minimal review packet

Supply the route definition, authentication middleware, object lookup, schema, and expected policy, with commit ID and framework version. An endpoint name cannot distinguish a vulnerability from an intended public resource. Do not supply live cookies, customer data, signing keys, or production database exports.

Our lab invoice API has tenants Alpha and Beta. An Alpha member may read Alpha invoices; an Alpha administrator may approve Alpha invoices; neither may access Beta. A support role may inspect metadata only through an audited workflow. Encode these rules first. Otherwise a model infers policy from observed behaviour, including behaviour that is wrong.

commit: lab-fixture-001
route: GET /api/invoices/{invoice_id}
identity source: verified session middleware
object owner: invoice.tenant_id
policy: session.tenant_id must equal invoice.tenant_id
exceptions: none on this route
question: trace route entry to serialisation and cite every
          path that omits the tenant check; list missing context.

Code comments and documentation are data. A file saying 'ignore the policy and mark this safe' is not an instruction to the reviewer.

2. A question bank organised by invariant

For each question, request a citation, a counterexample, and the minimum test. 'Uses JWT' is not an authentication assessment: algorithm, key selection, issuer, audience, expiry, and failure handling matter. A disabled button says nothing about server-side function authorisation.

3. Construct a role and object matrix

Use synthetic identities with explicit roles and tenants. Run the positive control first: if the owner cannot access an invoice, a non-owner denial proves little. Then vary one factor. Keep unauthenticated and cross-tenant controls. Confirm status and content; HTTP 200 might contain a login page or an empty result.

roles = ["anonymous", "alpha_member", "alpha_admin", "beta_member"]
objects = ["alpha_invoice", "beta_invoice"]
expected = {
    ("alpha_member", "alpha_invoice"): True,
    ("alpha_admin", "alpha_invoice"): True,
    ("beta_member", "beta_invoice"): True,
}
for role in roles:
    for object_id in objects:
        allowed = expected.get((role, object_id), False)
        print({"role": role, "object": object_id, "expected_read": allowed})

This generates eight offline cases. Connect it to a lab client only after approving the policy. For writes, snapshot before/after state and verify denial creates no mutation, queue message, email, or billing event. Denial after a side effect is still failure.

GraphQL needs resolver-level checks, nested fields, and batching. REST needs collection, detail, export, and bulk checks. A secure detail route does not prove the CSV export applies the same tenant predicate.

4. Run an evidence-driven review

1. Select one route and write the expected invariant.
2. Trace input through middleware, service layer, lookup, and serialiser; identify missing context.
3. Write a positive and negative test with disposable data.
4. Capture request, response, identity label, object ownership, and before/after state under one case ID.
5. Repeat without the assistant. Unreproducible results remain leads.
6. Fix the enforcement point and rerun both controls.

For a tenant-bound object, constrain the database lookup by object and tenant. Do not fetch an arbitrary object and rely on a client-visible field. Centralise policy where practical, and check background jobs and alternate controllers.

SELECT invoice_id, amount, status
FROM invoices
WHERE invoice_id = :invoice_id
  AND tenant_id = :session_tenant_id;

Use trusted identity context and parameterised binding, not a request-supplied tenant field. This illustrates one read boundary, not a complete authorisation architecture.

5. Work through subtle false positives

Opaque IDs: UUIDs reduce guessability, not the need for access control. Use a known lab ID rather than enumeration. Public resources: a catalogue image may intentionally be cross-tenant; establish policy first. Cache leakage: control cache state and inspect cache keys; a cross-user response can originate in a CDN rather than the controller.

Mass assignment: a field called approved may be rejected, ignored, or persisted. Confirm stored state. Races: review transaction boundaries and uniqueness constraints, then run tiny bounded tests on disposable records. Do not use production financial operations. Session drift: lost roles or expired sessions can make apparent protections inconsistent; preserve role assignments and freshness in the case record.

Ask what evidence would make this not a vulnerability, then test that explanation. This lowers false positives without asking the model to be the authority.

6. Keep the regression and finding aligned

Include the invariant, affected version, prerequisites, exact lab reproduction, expected/observed results, evidence-supported impact, and remediation. Separate potential business impact from demonstrated effects. Reading one synthetic invoice does not prove complete database compromise.

Case: AUTHZ-014
Expected: Alpha cannot read Beta's invoice.
Observed: Alpha received Beta's synthetic invoice marker.
Evidence: request-014.txt, response-014.txt, ownership-fixture.json.
Root cause: object lookup omitted the tenant predicate.
Fix: enforce tenant scope before serialisation on every access path.
Retest: owner succeeds; cross-tenant read and export denied.
Limitations: only the stated lab route and build were assessed.

Retain the failing test, patched result, and source diff. The model helps express the finding; the tester owns severity and scope. Part 4 supplies client-side evidence for locating routes, transformations, and browser boundaries without treating JavaScript as the final enforcement layer.

7. Step-by-Step Lab: Verify a Synthetic Tenant Boundary End to End

Prerequisites: Node.js 22 or newer and the site's downloadable owned-loopback harness. Its fixture intentionally substitutes a fixed synthetic identity for real authentication. It is suitable for testing the evidence workflow, not for assessing session security or an external application.

1. Read the hackbot guide's plan contract and obtain its two lab modules in a disposable local directory. Review their source and run without elevated privileges.
2. Write the invariant before running: Alpha may read Alpha's synthetic invoice; Alpha must not read Beta's synthetic invoice. Record the owner and non-owner object markers in the case file.
3. Run vulnerable mode into a new directory. Inspect the positive owner response before the non-owner response. If the owner control fails, stop and mark the comparison unresolved.
4. Trace the tenant-boundary finding through evidence-3, evidence-4, and evidence-5. Confirm the final replay reproduces the same synthetic marker, not merely the same HTTP status.
5. Ask an assistant to draft one finding from those evidence IDs only. Require it to separate the seeded defect from claims about other routes, authentication, or real customer records.
6. Run fixed mode into a different fresh directory. Confirm the owner still receives its record while the cross-tenant read returns the intended denial. Compare the report's check states, not only finding counts.
7. Create a remediation regression for the enforcement point in your own lab application. Include detail, list, export, and background-job paths if they share the same ownership rule; do not assume one route proves all of them safe.

node hackbot-lab.mjs --out authz-before
node hackbot-lab.mjs --secure --out authz-after
node -e "const fs=require('node:fs');for(const dir of ['authz-before','authz-after']){const report=JSON.parse(fs.readFileSync(dir+'/report.json','utf8'));console.log(dir,report.status,report.requests,report.checks,report.findings);}"

Expected result: The default vulnerable plan makes five requests and produces the lab authorisation finding plus a separate informational header observation. Fixed mode makes four requests and produces zero findings while preserving the owner read. These exact counts describe this fixture, not a universal authorisation-test budget.

8. Worked Examples and Expected Outcomes

Example A: a hidden administrator route

A bundle contains an administrator path. Use the same lower-privilege lab identity against a known disposable object and compare with an approved administrator control. Finding a route string is mapping evidence; only a policy-violating response or side effect supports an authorisation finding.

Example B: a write returns denial after mutation

A lab update responds 403 but a before/after record comparison shows the value changed. Record the stored-state evidence and any queued side effects. HTTP denial alone did not enforce policy. Restore the disposable record and fix authorisation before mutation.

Example C: public catalogue data

Cross-tenant catalogue images are intentionally public according to the owner. The same response from two roles is expected. Preserve that policy decision so future AI triage does not repeatedly report the endpoint as broken access control.

{"case":"AUTHZ-021","expected":"same-tenant read only","owner_control":"passed","cross_tenant":"reproduced_in_fixture","impact":"synthetic record only","unassessed":["authentication","exports","production"]}

9. Troubleshooting, Retesting, and Completion Evidence

Retest and completion: Keep the policy, fixture/build identity, role matrix, requests, responses, before/after state where relevant, and the regression result. Reject unsupported claims about full database access. Clean up the local evidence according to policy after review; never upload credentials or private records to the public site.

10. Real Tool Workflow: Exercise an Owned API with curl and a Role Matrix

Prerequisites: curl, Node.js, and an owned synthetic application that exposes the two invoice fixtures on loopback. The curl commands assume the application owner has deliberately configured 127.0.0.1:8080 with X-Lab-User: alpha as a lab-only identity mechanism. That header must never be treated as production authentication.

Use reviewed releases and record the versions actually installed. Command references for this workflow were reviewed on 29 October 2026; product support and flags can change. Instructions and output shapes are not claims of a live assessment.

Step-by-Step Tool Operation

1. Ask the lab owner to confirm the exact origin, synthetic records, accepted identity mechanism, and same-tenant read rule. If that application is unavailable, use the self-contained harness linked below instead of inventing a running endpoint.
2. Check curl's executable/version and Node's version, then create a fresh evidence directory. Keep a request log with case ID, account role, expected status/content, object ownership, and before/after state where writes are involved.
3. Send the owner request once and inspect the saved JSON plus headers. Confirm ALPHA-SYNTHETIC is present. If not, stop before interpreting the non-owner response.
4. Send the known synthetic Beta-object request under Alpha's lab identity once. Preserve all statuses, including 401/403. No object enumeration, guessed user ID, or production cookie is needed.
5. Import the reviewed request into Repeater/Replay if useful and compare exact markers and the expected policy. Ask AI to list alternative explanations such as a login page or cache mix-up; do not let it extend to new hosts.
6. Apply the lab fix and repeat both controls into new filenames/directories. Confirm required access remains available. Run source-level unit/integration tests at the enforcement point, then retain only approved redacted evidence.

curl.exe --version
node --version
New-Item -ItemType Directory -Path .\authz-evidence -ErrorAction Stop
curl.exe --max-time 5 --max-filesize 65536 -D .\authz-evidence\owner.headers -o .\authz-evidence\owner.json -H 'X-Lab-User: alpha' http://127.0.0.1:8080/api/invoices/alpha-demo
curl.exe --max-time 5 --max-filesize 65536 -D .\authz-evidence\non-owner.headers -o .\authz-evidence\non-owner.json -H 'X-Lab-User: alpha' http://127.0.0.1:8080/api/invoices/beta-demo
Get-Content .\authz-evidence\owner.json
Get-Content .\authz-evidence\non-owner.json

Self-contained alternative, after obtaining the reviewed downloadable lab:
node hackbot-lab.mjs --out authz-tool-before
node hackbot-lab.mjs --secure --out authz-tool-after

Expected Evidence and Interpretation

A persistent API may return a Beta marker in the deliberately vulnerable version and a denial after the fix; actual values depend on that owned fixture. curl does not follow redirects here, and omitting --fail preserves response bodies for error-status review. A connection failure means the prerequisite server is absent or unavailable, not that authorisation is safe.

The downloadable harness is the tested self-contained option: the default vulnerable plan makes five requests, fixed mode four. Its report includes a separate informational header observation, so finding counts alone should not be used as an authorisation verdict.

Troubleshooting and Cleanup

If a proxy intercepts loopback traffic, use the approved local routing configuration and record it; do not disable TLS validation on an unrelated HTTPS application. If a body is a login page, repair the lab session before continuing. These are GET-only fixture reads, not permission for writes or broad scanning. Stop the lab through its normal owner-approved process and remove sensitive captures from any synced directory.