Checking Your Own Public Footprint Part 7: Automating the Whole Audit With SpiderFoot

Parts 2 through 5 were all manual: one tool, one technique, one thing checked at a time. SpiderFoot is the automated version of the same idea, and it’s worth closing on specifically because of how much of this series it actually wraps rather than replaces.

What it is

SpiderFoot is an open-source, MIT-licensed OSINT aggregator, stable at v4.0, with over 200 modules feeding each other in a publisher/subscriber model so that one discovered fact (a subdomain, an email address, an IP) automatically triggers further checks against it. It runs as a self-hosted web UI or entirely from the command line, and most modules don’t need an API key at all; the ones that do mostly have a free tier.

$ wget https://github.com/smicallef/spiderfoot/archive/v4.0.tar.gz
$ tar zxvf v4.0.tar.gz && cd spiderfoot-4.0
$ pip3 install -r requirements.txt
$ python3 ./sf.py -l 127.0.0.1:5001

Point a scan at a domain, an email address, or a username, and it runs its module set against that target, correlating results as they come in against a set of 37 pre-defined YAML rules that flag patterns worth a human looking at, rather than just dumping raw findings into a list.

Modules that map directly onto Parts 2 through 5

This is the part worth actually reading SpiderFoot’s own module list for, because it isn’t a coincidence: SpiderFoot’s own documentation lists Certificate Transparency (via crt.sh) as a module, Shodan as a module, HaveIBeenPwned as a module, Archive.org as a module for “identifies historic versions of interesting files/pages from the Wayback Machine”, and TruffleHog itself as a tool module for scanning associated repositories for secrets. A single SpiderFoot scan is, in large part, an automated version of Parts 2, 4, and 5 of this series, run against every module at once instead of one technique at a time.

What it adds beyond simple aggregation is the correlation layer: an email address discovered via one module gets automatically checked against HaveIBeenPwned by another, a subdomain discovered via crt.sh gets automatically port-scanned and fingerprinted by a third, without you manually feeding one tool’s output into the next tool’s input. That’s the genuine value of an orchestrator over a manual checklist: not new data sources, but automatically wiring the ones that already exist together.

Where it still needs you

Every automated tool this site has covered, from Sn1per’s port-list gaps to Maigret’s false positives two parts ago, ends up needing the same closing caveat, and SpiderFoot is no exception. A correlation rule flagging a pattern is a lead, not a conclusion. A module reporting zero results from HaveIBeenPwned means nothing showed up in that specific database, not that no breach exists anywhere. And a broad scan against your own name or domain is going to surface a lot of noise alongside genuine findings, the same false-positive problem Part 3 walked through with Maigret, just at a larger scale because SpiderFoot is checking far more sources per run.

Treat a SpiderFoot scan as the fast first pass, then use Parts 2 through 5 of this series to manually verify anything it flags as worth a second look. That order, automated pass first, manual verification of the interesting bits, not the other way round, is the same order this site has used for every automated security tool it’s covered.

The checklist

A short version of this entire series, to actually run through periodically, not just read once:

• Run whois (or check via RDAP once your TLD retires WHOIS) against every domain you own. Confirm privacy is actually on, not just remembered as being on.
• Query crt.sh for %.yourdomain.tld and check every subdomain listed against what you actually meant to have public.
• Run Maigret (or a similar tool) against your own real handle, then manually open and verify every result before trusting it.
• Run gitleaks and TruffleHog against every public repository you maintain, full history, not just the current commit.
• Check internetdb.shodan.io for your own public IP addresses.
• Run exiftool against any photo, PDF, or document you’re about to publish, before you publish it, not after.
• Query the Wayback Machine’s CDX API for old WHOIS-lookup snapshots of your domains.
• Check your passwords against the Pwned Passwords range API. It’s free, safe, and never transmits your actual password.
• Fix what you find: registrar privacy toggle, Nominet individual opt-out if applicable, an archive.org removal request for anything stale, and account deletion for anything abandoned.
• Set a reminder to do all of the above again in six to twelve months, or immediately after any registrar transfer.

That’s the whole audit. None of it requires anything you don’t already have access to, and none of it requires trusting a single tool’s output over your own judgment once you’ve looked at what it actually found.