AI Pentest Agents Part 6: Fingerprinting the Swarm, a Defender's Read on Agent Noise

Parts 2 through 5 of this series covered what PentestCode, PentestGPT, PentAGI, NodeZero, and XBOW actually do. This part asks the question none of their own documentation is built to answer: what does a defender actually see when one of these is running against their network, and can it be told apart from a human doing the same job?

The honest starting point is PentestCode’s own README, quoted in full in Part 2: “not stealthy… doesn’t think about OPSEC by default… fine for authorized tests, not for red team stealth operations.” That’s the only tool in this series that says this about itself in plain language, and it’s worth taking at face value rather than assuming every autonomous agent shares the same blind spot — XBOW, running against production bug-bounty targets with a human review step before submission, plainly has to survive at least basic scrutiny to be useful there at all. But PentestCode’s admission gives a concrete starting hypothesis: if an agent doesn’t reason about stealth, its noise pattern should look different from a human’s, in ways a defender can actually use.

Where the noise comes from

Two specific admissions in PentestCode’s own limitations section point at where that noise would show up. First: it’s “token-hungry,” and “large scans with verbose output make this worse” — which, translated into network behavior, means broader, more exhaustive scans than a time-constrained human tester would typically run, because the agent isn’t paying for its own attention budget the way a person watching the clock is. Second, and more directly useful: it “repeats work… the agent sometimes re-runs tools it already ran,” despite wordlist tracking and state diffs meant to prevent exactly that. A human operator who already scanned a host isn’t going to absent-mindedly re-run the same nmap sweep against it an hour later without a reason. An agent, working from imperfect state tracking across a long autonomous run, might — and that’s a signature: redundant, near-identical tool invocations against the same target, spaced by the agent’s own reasoning cadence rather than by any operational reason to re-check.

The credential-spraying behavior described in Part 1 and Part 2 is the other clear tell. PentestCode’s own description of itself: it sprays every discovered credential against every discovered service on every known host — SMB, WinRM, LDAP, RDP, all of them, “something humans routinely skip” by the project’s own account. That’s presented as a strength (thoroughness a rushed human tester wouldn’t manage), but it’s also a distinctive shape: a single credential appearing in authentication attempts against services and hosts that have no obvious relationship to each other in a short window is a pattern this site’s Pivoting and Tunneling series already covered from the offensive side — flow-level detection of lateral movement doesn’t require decrypting anything, just noticing that traffic shape.

What actually transfers from this site’s existing detection work

Two series already published here are directly relevant, and neither was written with AI agents in mind — which is itself informative, since it means the detection logic doesn’t actually depend on whether a human or an agent is driving.

rst-forensics, covered earlier on this site, classifies TCP reset origin using TTL deltas, IP-ID jumps, and window/MSS fingerprints — none of that cares who or what initiated the connection that got reset. If PentestCode’s redundant tool-run behavior means the same port gets probed and reset multiple times in a session, the classifier doesn’t need to know an LLM was involved to flag the pattern; it just sees more resets, clustered tighter in time, than a single deliberate human check would typically generate.

The Pivoting and Tunneling series’ closing hardening checklist — VDOMs, egress filtering, TLS inspection, FortiAnalyzer flow monitoring, ZTNA — was written against human-operated Chisel and ligolo-ng pivots, and every control in that checklist applies exactly as well against an agent using PentestCode’s tunnel_manage tool to plan the same SSH/Chisel/ligolo-ng tunnels, because tunnel_manage is explicitly described in Part 2 as planning tunnels using the same tools that series already covers, not some novel agent-specific tunneling mechanism. The pivot itself doesn’t get stealthier because an LLM chose to run it — if anything, the credential-spray pattern above makes it noisier.

What genuinely doesn’t transfer, and the honest limit here

None of this amounts to “AI pentest agents are easy to catch.” The mode settings covered in Part 2 — guided, pause-on-finding, scope-checked auto — mean an operator who cares about stealth can dial PentestCode down toward something closer to a supervised, deliberate human pace, at which point the volume-based signatures above stop being reliable. And a tool built specifically for stealth (none of the five covered in this series claim to be) would presumably not exhibit the redundant-scan and blanket-spray behavior PentestCode admits to. The honest conclusion is narrower than “detectable” or “undetectable”: the default, unattended, auto-mode behavior these particular tools ship with today produces a noisier, more exhaustive, more repetitive pattern than a time-constrained human tester typically would, and existing flow- and reset-based detection built for human-operated tooling already catches a meaningful share of that without any AI-specific logic at all. Whether that holds as the field matures past its current beta-labeled state is a different question, and one worth revisiting once these tools start advertising stealth as a feature rather than admitting its absence.

Part 7 puts this series’ claims to an actual test: installing PentestCode against the same Metasploitable2 target already sitting on the Proxmox lab built for AI Part 7, and seeing what a real run — not a described one — actually looks like.