Text Processing for Network Engineers Part 11: Tracing a Bad ACL Across the Fleet, Start to Finish

This series has covered regex, grep, sed, awk, vim, nano, diff and patch, find and xargs, and the encoding gotchas that undermine all of them. This part is one incident, worked start to finish, using nothing else.

The incident

A maintenance window pushes an updated outbound ACL to 150 branch firewalls, adding a permit line for a new SaaS provider’s IP range ahead of an existing deny catch-all. Two hours later, a subset of branches report the new service is still unreachable. The rest are fine. Config backups from every device, pulled nightly, sit in configs/, one file per hostname, and a fresh backup pull was triggered immediately after the push, so configs/ reflects the post-change state everywhere.

Step 1: confirm the change actually landed everywhere (grep, Part 2)

First question: did the push even apply to every device, or did some silently fail?

grep -rl 'permit tcp any 203.0.113.0 0.0.0.255' configs/ | wc -l

150 devices, 150 matches. The line landed everywhere. That rules out “the push failed on some boxes” and narrows the problem to something about the context the line landed in, not whether it landed at all.

Step 2: find where in the ACL the line actually landed (awk, Part 4)

The push was meant to insert the new permit line immediately before the existing catch-all deny ip any any. If it landed after the deny on some devices instead of before it, the deny would shadow it, ACLs are evaluated top-down, first match wins. Checking line order across every device:

for f in configs/*.cfg; do
  awk '/permit tcp any 203.0.113.0 0.0.0.255/{p=NR} /deny ip any any/{d=NR} END{if (p && d && p>d) print FILENAME}' "$f"
done

That awk one-liner records the line number of the permit and the deny inside each file, and at the end of each file, prints the filename only if the permit’s line number is greater than the deny’s, meaning it landed after the catch-all. It returns exactly the broken subset: 23 devices out of 150. Everywhere else, the line landed in the right place and works correctly.

Step 3: understand why those 23 specifically (diff, Part 8)

Before assuming this is random bad luck, diff one of the broken configs against a known-good one from the same push, to see whether there’s a structural reason:

diff -u configs/branch-fw04.cfg configs/branch-fw12.cfg

The 23 broken devices, it turns out, all had a locally added ACL entry from an earlier, unrelated change, one that had drifted from the standard template and inserted an extra line between the anchor point the push script matched on and the catch-all deny. The automation that generated the push assumed a fixed line offset from a matched anchor line, and that assumption held everywhere except the 23 devices where local drift had already changed the offset. This is the actual root cause, not a partial push failure, a push script that assumed a config shape that wasn’t universally true anymore.

Step 4: build the fix, verified against one file first (sed, Part 3, and diff again)

The fix on each broken device is: remove the misplaced permit line, then re-insert it immediately before the catch-all deny, wherever that deny actually sits in that specific file. A pattern-range sed handles this correctly, because it’s addressed to the deny line’s position in each file individually, not to a fixed line number:

sed -e '/permit tcp any 203.0.113.0 0.0.0.255/d' \
    -e '/deny ip any any/i\
permit tcp any 203.0.113.0 0.0.0.255' \
    configs/branch-fw17.cfg > /tmp/branch-fw17-fixed.cfg
diff -u configs/branch-fw17.cfg /tmp/branch-fw17-fixed.cfg

The first -e deletes the misplaced line wherever it is, the second inserts it immediately before whichever line matches the catch-all deny, and because sed here is pattern-addressed rather than line-number-addressed, it self-corrects for however far the drift had actually moved things on that specific device. The diff at the end is the check from Part 8’s discipline: read the change before trusting it, on the one file, before touching the other 22.

Step 5: apply it to all 23, then verify with grep and awk again

for f in $(grep -rl 'permit tcp any 203.0.113.0 0.0.0.255' configs/ | xargs grep -L 'deny ip any any' -A0 2>/dev/null); do :; done

In practice, at this point the 23 filenames from Step 2 are already known and saved to a list, so the fix loop runs directly against that list rather than re-deriving it:

while read -r f; do
  sed -i.bak -e '/permit tcp any 203.0.113.0 0.0.0.255/d' \
             -e '/deny ip any any/i\
permit tcp any 203.0.113.0 0.0.0.255' \
             "$f"
done < broken-devices.txt

Then the exact Step 2 awk check, re-run against the same 23 files, confirms zero remaining cases where the permit lands after the deny. Every .bak file sed left behind is the rollback path if anything about this needs to be reverted before the next maintenance window.

Step 6: the one device that needed a manual touch (vim, Part 6)

22 of the 23 fixed cleanly with the loop above. The 23rd had a config drift severe enough (a second, conflicting local ACL entry referencing an overlapping range) that the automated fix would have produced a technically-valid but semantically-wrong result. That one gets opened directly:

vim configs/branch-fw41.cfg

:g/permit tcp any 203.0.113.0 0.0.0.255/d clears every existing occurrence first (there were two, one correctly placed and one stray duplicate from an earlier half-applied change), then a single manual insertion above the catch-all deny, checked visually before :wq. This is exactly the case Part 6 and Part 7 both flagged: automation handles the common shape correctly and quickly, and the one device whose history doesn’t match that shape gets the deliberate, manual, single-file treatment instead of being forced through the same loop as everything else.

Step 7: the backup that wouldn’t grep at all (encoding, Part 10)

One config, pulled from a branch device on a much older firmware version, didn’t match any of the greps above, including the initial Step 1 check that should have shown it as compliant. file on it explained why: ASCII text, with CRLF line terminators, a config exported through a path that had, at some point, touched a Windows-based tool. The trailing \r on every line meant nothing anchored with $ in any of the patterns above ever matched. dos2unix on that one file, followed by re-running every check from Step 1 forward, brought it into line with the rest of the fleet.

What the whole incident actually demonstrates

No single tool in this series would have solved this on its own. grep found where the change landed and confirmed the fix afterward. awk answered a structural question, ordering, that grep has no concept of. diff explained why only 23 devices broke, instead of just confirming that they had. sed applied the fix at scale, correctly, because it was pattern-addressed rather than assuming a fixed line number the way the original push script incorrectly did. vim handled the one case that didn’t fit the general pattern. And the encoding check caught the one device that would otherwise have silently stayed broken because every prior step assumed the file was plain Unix text. That’s the actual argument for learning all of these properly rather than memorizing a handful of one-liners: the incident that matters is rarely solved by one tool, it’s solved by knowing which tool answers which specific question, and reaching for the right one at each step.