FMG API From Scratch Part 9: The Finished Tool, and Hardening It

Eight parts of small pieces. This one puts them together into a single command, fmg_push.py, and is honest about what it is: a careful tool for pushing, staging and installing CLI scripts to one device at a time.

The code for this part is in the fmg-api-from-scratch repo at tag part-09.

The pieces

Each module exists because of a lesson in an earlier part:

ModuleJobComes from
fmg_client.pyTalk to the API, check every status, never send without a sessionPart 3
fmg_audit.pyJudge the API account against the four fencesPart 4
fmg_redact.pyMask secrets by key name and by value shapePart 4
fmg_discover.pyDescribe an endpoint by shape, never by valuePart 5
fmg_scripts.pyRender locally, push a script, stage itPart 6
fmg_logs.pyPoll the task, read the log by its id, check what changedPart 7
fmg_install.pyBuild the preview diff, and install only what was previewedPart 8
fmg_push.pyThe command that ties them togetherThis part

What it does

python fmg_push.py devices
python fmg_push.py push    --template dns.j2 --vars vars.json --device <device>            # dry run
python fmg_push.py push    --template dns.j2 --vars vars.json --device <device> --apply
python fmg_push.py stage   --script dns__<device> --device <device> \
                           --watch /pm/config/device/<device>/global/system/dns --apply
python fmg_push.py log     --device <device>
python fmg_push.py preview --device <device>
python fmg_push.py install --device <device> --apply

Six commands, and a few rules built into all of them. (Options that apply to every command, such as --env-file and --audit, go before the command name.)

Writes are off unless you say --apply. push, stage and install without it do their checks, print what they would do, then stop. The default is the safe direction. A dry run does log in, read the FortiManager version and the ADOM list, and log out. It makes no write requests, and the tests check that by recording every call a fake FortiManager receives.

Rendered CLI is never printed. Part 6 explained why: after local rendering, the text holds every substituted variable, secrets included. The tool prints the number of lines and the first twelve characters of a SHA-256. If you want to read a log or a preview diff, that is an explicit choice (log --show, or preview), and the tool prints a warning above it that the text contains rendered CLI.

Every action leaves a record, and the record holds no content.

def audit_record(action: str, adom: str, device: str, script: str, content: str = "") -> dict:
    """What to remember about an action. Never the content itself, only its hash and size."""
    return {"ts": datetime.now(timezone.utc).isoformat(timespec="seconds"),
            "action": action, "adom": adom, "device": device, "script": script,
            "sha256": hashlib.sha256(content.encode()).hexdigest() if content else None,
            "lines": content.count("\n") if content else 0}

It is appended, one JSON object per line, to fmg_push_audit.jsonl. Later you can answer “what did we push, when, to which device” and prove that the text you have now is the text that went, by comparing hashes, without the log itself becoming a place secrets live. The same hash goes into the script’s desc on FortiManager, so the two records tie together. Previews and installs are recorded the same way, with the hash of the diff.

It refuses versions it hasn’t been tested on. The first thing it does after login is read /sys/status and compare the major and minor version with a short list:

SUPPORTED = ((7, 6),)          # (major, minor) pairs this tool has been tested against

def check_version(status: dict, supported=SUPPORTED) -> None:
    """Refuse to run against a FortiManager version the tool has not been tested on."""
    have = (status.get("Major"), status.get("Minor"))
    if have not in supported:
        raise SystemExit(f"FortiManager {have[0]}.{have[1]} is not one of the tested versions "
                         f"{list(supported)}; refusing to continue.")

Part 3 pointed out that FortiManager returns the version as separate numbers for exactly this reason. Everything in this series was observed on 7.6.6. A path that moved in 7.6.5, as the script endpoint did, is precisely the sort of thing that changes between minor versions, and the failure would be silent. A tool that stops is better than one that quietly does something different.

A missing template variable stops the push with a plain message. Part 6’s StrictUndefined raises an exception; the tool catches it, says which variable, sends nothing, and exits with status 2.

stage --watch tells you whether anything changed. Give it a Device Database path, and it reads that path before and after, waits for the task, and prints the names of the fields that differ. A failed task exits with status 1, so anything wrapping the tool can tell.

install only sends what preview showed you. That is the whole of Part 8: the preview records a hash, and the install rebuilds the diff, compares, and refuses if it differs.

Running it

These are real results from my lab, with device names, the ADOM name and addresses replaced. The first two commands are read-only.

$ python fmg_push.py devices
<hub-70g>                    os_ver=7  conn_status=2
<hub-vm>                     os_ver=7  conn_status=2
<spoke-1>                    os_ver=7  conn_status=2
<analyzer>                   os_ver=7  conn_status=2

$ python fmg_push.py log --device <analyzer>
No recorded script run for <analyzer>.

Then a push, starting with the dry run. The template is a five-line DNS block, and the variables file supplies the values:

$ python fmg_push.py push --template dns.j2 --vars dns-test.json --device <spoke-1>
Rendered dns.j2 for <spoke-1>: 5 lines, sha256 1c8bb9e25d43
Dry run. Would push script 'dns__<spoke-1>'. Add --apply to do it.

$ python fmg_push.py push --template dns.j2 --vars dns-test.json --device <spoke-1> --apply
Rendered dns.j2 for <spoke-1>: 5 lines, sha256 1c8bb9e25d43
Pushed script 'dns__<spoke-1>' to the ADOM database. No device has been touched.

The hash is the same in the dry run and the real push, which is the point: it identifies the text. With a variable missing from the file the tool stops and exits with status 2, and sends nothing.

Then stage it, with --watch on the DNS object, and read the log by the id that stage’s task implies:

$ python fmg_push.py stage --script dns__<spoke-1> --device <spoke-1> \
      --watch /pm/config/device/<spoke-1>/global/system/dns --apply
Task 138 finished, state 4 (success).
Fields changed in the Device Database: domain

$ python fmg_push.py log --device <spoke-1> --show
log_id 1381  script dns__<spoke-1>  at <date>  (10 lines)
...
 config system dns
 set primary 192.0.2.53
 set secondary 198.51.100.53
 set domain "lab.test"
 end
Running script(dns__<spoke-1>) on DB success

Then preview, and install, as in Part 8. The audit file now has a line for each step, and each line is a hash, a line count and a timestamp. None of them is the DNS block.

The account it runs as

The tool never asks for more than the Part 4 account has. It reads devices, reads and writes script objects, executes scripts against a device’s database, builds install previews and commits an install for devices in its ADOM, and reads task and log records. It does not read administrator records, profiles or global settings, and none of its commands try. If you run the audit from Part 4 against the account the tool uses, that is what you want the result to say.

Testing the safeties

The suite has 93 tests and needs no network. The ones that matter here follow the same rule as the rest of the series: each exists to fail when a safety is removed.

Six of the 93 are the certificate-pinning tests from Part 4. They need the openssl command to make a throwaway certificate, so they skip themselves on a machine without it. On a stock Windows install you will see OK (skipped=6), which is expected.

TestThe safety it guards
A dry-run push makes no write request and prints no contentWrites off by default; content never printed
A missing template variable stops before any request, with status 2StrictUndefined from Part 6
--apply pushes, and neither the output nor the audit file contains the secretContent stays out of both
A dry-run stage sends nothingNothing executes without --apply
A green task with no change says soThe “did anything change” check
A failed task returns status 1Callers can detect failure
An untested version is refusedVersion guard
The audit record has a hash and a size but never contentAudit trail holds no secrets
install with no preview, an old preview or a changed diff is refused, and does not commitThe install guard
install without --apply sends nothingDry run by default
latest_log takes the highest log_id, and never calls the latest endpointsPart 7’s log lookup
Polling stops at a terminal state, and a task that never finishes raisesPart 7’s wait_for_task
Stage’s scope uses vdom global; a stored script with the wrong target is caughtPart 6’s scope and target
A pinned login works, a wrong pin is refused, and the pin survives REQUESTS_CA_BUNDLEPart 4’s certificate pinning

I also broke the tool on purpose, one safety at a time, on a scratch copy: the dry-run check in push and in stage, the audit record storing content, the diff comparison in install, the preview expiry, the cleanup after an install, the global vdom, StrictUndefined, the version guard, the log lookup, the secret-shape check in the redactor, the Super_User failure in the audit, the fingerprint assertion and the cert_verify override in the pin, and two of the audit’s other rules. The suite failed each time, which is the evidence that the tests are looking at the right things. When I removed the timeout from wait_for_task the test for it did not fail, it hung, which is the behaviour the timeout exists to prevent. If you change the tool, do the same to whatever safety you touch.

What it does not do

A hardened tool is one whose limits you know, so here are the ones I know:

  • It handles one device at a time. That’s deliberate for now. A fleet needs a plan for partial failure, which is a different problem.
  • It changes an existing, registered device. It does not create ADOMs, register new devices, build interfaces, routing or SD-WAN, or install a policy package. Taking a new FortiGate to a working site is a bigger job than a safe push, and is what I want to build next on top of this tool.
  • The changed-diff refusal is unit-tested, not field-tested. I proved on the lab that install refuses without a preview. I did not engineer a diff that changes between preview and install on the real box.
  • It doesn’t lock anything. In the default workspace mode nothing stopped a GUI user and the tool editing at once. I found no lock call in the GUI’s own traffic for a normal install, but if you run workspace mode with approvals, that needs revisiting.
  • The variables file is plaintext. The tool keeps secrets out of its output and its audit trail; it does not protect the file you give it. Keep that file outside the repository, readable only by you.
  • The audit file is local and append-only by convention only. Anyone who can edit the file can edit the history. For anything shared, ship the lines to somewhere the tool’s user can’t rewrite.
  • The version list is short. It contains only what I’ve tested, and I’ve tested one release.
  • FortiManager’s own log is separate. Turning on jsonapi-log (Part 4) gives you FortiManager’s view of the account’s activity. Even with it off, the Event Log records configuration changes made through the API, and that is the useful cross-check on this tool’s own record.

Before you point it at a real device

  • The account passes the Part 4 audit, and you have tried to cross each fence
  • TLS is pinned or properly issued
  • The version is on the tested list
  • You’ve run the change as a dry run and read the line count and the hash
  • You’ve staged with --watch and the field names that changed are the ones you expected
  • The log for that run, read by its id, matches
  • You’ve read the preview diff, and it is only what you expected

The last one comes from something I found in my own lab: the preview can contain changes the script never mentioned, so the log of what your script did is not the same thing as the list of what install will send. Part 8 covers it, and the install guard makes the preview the only diff you can install.

What comes next

This series ends with a tool that is safe to point at one device. The natural follow-up is the part I deliberately left out: building a whole FortiGate site from nothing through the API, starting with the ADOM and device registration and ending with an installed policy package, using the same preview-first rules. That needs a lab device registered from scratch and a look at whether FortiManager’s provisioning templates have a usable API, so I’ll write it up once I’ve proved each step.