AI Pentest Agents Part 3: PentestGPT and the Academic Case for Reasoning, Generation, Parsing

Every other tool in this series is either brand new (PentestCode, barely a week old as of Part 1) or a 2026-era open-source project (PentAGI, covered in Part 4). PentestGPT is different: it started life as a peer-reviewed paper, published at USENIX Security 2024 — “PentestGPT: Evaluating and Harnessing Large Language Models for Automated Penetration Testing,” Deng et al. — two years before the current wave of autonomous agents made the approach fashionable. That head start shows up in how the project talks about itself: it’s the only tool in this series that opens its README with a citation block rather than a feature list.

The original three-module design

The paper’s contribution wasn’t “point an LLM at a target,” which even in 2024 was an obvious enough idea that several groups had already tried variations of it and mostly gotten frustrating, context-losing results. The contribution was splitting the problem into three cooperating modules: reasoning (what should happen next, given everything found so far), generation (turn that reasoning into an actual concrete command or tool call), and parsing (turn the tool’s raw output back into something the reasoning module can use on the next turn). Separating those three responsibilities — rather than asking one prompt to reason, act, and interpret results all at once — is the same underlying insight PentestCode’s engagement-state design leans on in Part 2 of this series: an LLM given one undifferentiated job over a long-running task degrades faster than the same work split across specialized roles with a shared, structured handoff between them. PentestGPT got there first, and did it with three cooperating LLM calls rather than thirteen coordinated agents — a much lighter architecture for the same underlying idea.

What changed by v1.0

The current release describes itself as the “Agentic Upgrade”: a fully autonomous pipeline rather than the original interactive-assistant mode, session persistence so a multi-day engagement survives being closed and reopened, and a Docker-first distribution — make install, make config for first-time API key setup, make connect to enter the container. Multi-model support beyond Anthropic (OpenAI, Gemini, and others) is explicitly listed as still in progress rather than shipped, which is a more candid way to describe a gap than most projects manage.

Authentication options are broader than most of this series: a direct Anthropic API key, Claude’s own OAuth login for subscription holders, OpenRouter for routing to other model providers, or — genuinely useful for anyone wary of API spend on a tool that can run for hours unattended — local models via LM Studio or Ollama, with a router configuration (scripts/ccr-config-template.json) that lets you assign different local models to different roles: a fast default model for general tasks, a stronger reasoning model for the think route, a long-context model for large tool output. That’s a more deliberate take on the same idea AI Part 7 tested informally by hand — that model capability requirements aren’t uniform across a pentest workflow, so routing cheaper or smaller models to easier sub-tasks and reserving a capable model for the steps that need real judgment is a reasonable design, not just a cost-cutting hack. Part 7’s finding that a 7B model fabricated tool output wholesale while a 32B model reasoned about failures correctly is exactly the gap a router like this is trying to manage — put the 7B somewhere it can’t do damage, keep the 32B (or better) on the steps where honesty matters.

The benchmark numbers, and why they’re worth taking seriously

PentestGPT ships 104 benchmarks drawn from XBOW’s own public validation suite — genuinely useful, since it means PentestGPT is measuring itself against the same yardstick the field’s most publicly successful autonomous tool uses, rather than a suite it wrote for itself. Run via python3 run_benchmarks.py --range 1-1 --pattern-flag for a single benchmark or --all for the full set, with --retry-failed and --dry-run options that suggest a project used to running this suite repeatedly rather than once for a launch announcement.

The published results: an 86.5% success rate (90 of 104 benchmarks), averaging $1.11 and a median of $0.42 per successful benchmark, averaging 6.1 minutes and a median of 3.3 minutes. Success rate falls off by difficulty in the direction you’d expect and want to see reported honestly — 91.1% at Level 1, 74.5% at Level 2, 62.5% at Level 3 — rather than a single blended number that hides where the tool actually struggles.

Two things are worth noting about how that compares to XBOW’s own headline figure from Part 1 of this series (a human pentester’s 40 hours against XBOW’s 28 minutes on the same 104-scenario set — an 85x speed difference). First, PentestGPT’s median 3.3 minutes per successful benchmark isn’t directly comparable to XBOW’s total-suite completion time without knowing XBOW’s own per-benchmark breakdown, so treat both as interesting data points rather than a head-to-head. Second, and more useful practically: PentestGPT publishing cost-per-benchmark at all, in both mean and median, is a level of self-reported detail none of the other open tools in this series offer — PentestCode’s README gives a $5–50 per-engagement range with no benchmark suite behind it, and PentAGI’s documentation (Part 4) doesn’t quote cost figures at all.

Telemetry, and the disclosure that matters

PentestGPT collects anonymous usage telemetry via Langfuse by default — session metadata (target type, duration, completion status), which tools got used (not the actual commands run), and that a flag was found (not its content) — with an explicit statement that command outputs, credentials, and flag values are never transmitted, and an opt-out via --no-telemetry or LANGFUSE_ENABLED=false. For a tool that’s going to be pointed at real client engagements, that’s a meaningfully different privacy posture from PentestCode, which doesn’t mention telemetry in its README at all — worth checking directly with each project rather than assuming either behavior if you’re going to run one of these against anything sensitive.

Part 4 moves to PentAGI, the other major open-source entrant, and a genuinely different infrastructure decision: full Docker sandboxing with root-level access to docker.sock, against PentestCode’s root-free single-binary install covered in Part 2.