Bash Patterns Every Network Engineer Should Know
Most network automation starts as bash — a quick SSH loop over a device list, a grep against a config backup, a cron job that pings a list of hosts. It is easy to write bash that appears to work and then breaks in exactly the situations where you need it most: one device in fifty being unreachable, a config line containing a special character, a script run twice concurrently.
This post covers the patterns that hold up under those conditions.
Strict Mode
Every script should start with this:
#!/usr/bin/env bash
set -euo pipefail
IFS=$'\n\t'
set -e— exit immediately if any command fails (non-zero exit code). Without this, a failedsshin the middle of a loop is silently ignored and the script carries on as if nothing happened.set -u— error on undefined variables, rather than silently substituting an empty string. Catches typos like$HSOTinstead of$HOSTimmediately instead of producing a confusing downstream failure.set -o pipefail— a pipeline’s exit code is the last failing command, not just the last command. Without this,ssh host 'bad-command' | grep somethingexits 0 even ifbad-commandfailed, becausegrep’s exit code (whether it matched) is all that is checked by default.IFS=$'\n\t'— restricts word-splitting to newlines and tabs, not the default of space/tab/newline. Prevents filenames or hostnames with spaces from silently splitting into multiple words.
The gap set -e does not close: commands inside if, while, &&/|| chains, or with their exit code already captured are exempt from triggering the exit — this is intentional (otherwise no script could ever check for failure), but it surprises people the first time a “should have failed” command does not stop the script.
Trap for Cleanup
TMPDIR=$(mktemp -d)
trap 'rm -rf "$TMPDIR"' EXIT
trap ... EXIT runs on any exit path — normal completion, set -e triggering, or a Ctrl-C. This is the single most reliable way to guarantee temp files, lock files, or background processes get cleaned up regardless of how the script ends.
# Cleanup that also handles a background process
trap 'kill $(jobs -p) 2>/dev/null; rm -rf "$TMPDIR"' EXIT
Parallel SSH Fan-Out
The naive loop runs sequentially — fine for five hosts, painful for five hundred:
# Sequential — slow
for host in "${hosts[@]}"; do
ssh "$host" "show version" >> output.txt
done
Pattern: background jobs with a concurrency cap
#!/usr/bin/env bash
set -euo pipefail
hosts=(router1 router2 router3 router4 router5 router6 router7 router8)
max_jobs=4
outdir=$(mktemp -d)
trap 'rm -rf "$outdir"' EXIT
for host in "${hosts[@]}"; do
# Wait if we're at the concurrency cap
while (( $(jobs -r -p | wc -l) >= max_jobs )); do
wait -n
done
(
ssh -o ConnectTimeout=5 -o BatchMode=yes "$host" "show version" \
> "$outdir/$host.out" 2> "$outdir/$host.err"
echo "$host: exit $?" >> "$outdir/status.log"
) &
done
wait
cat "$outdir"/*.out
wait -n (bash 4.3+) waits for the next background job to finish, rather than all of them — this is what makes the concurrency cap work without an external tool. Each host’s output and exit status goes to its own file, avoiding interleaved output corruption that happens when multiple backgrounded processes write to the same file descriptor simultaneously.
Pattern: GNU parallel (when available)
printf '%s\n' "${hosts[@]}" \
| parallel -j8 'ssh -o ConnectTimeout=5 {} "show version" > {}.out 2>{}.err'
parallel handles the concurrency cap, output separation, and job tracking that the manual bash version above builds by hand. Worth installing on any host that does this kind of fan-out regularly — apt install parallel / dnf install parallel.
Retry Logic with Backoff
Network operations fail transiently — a device mid-reboot, a flaky management VPN, a brief API rate limit. Retrying immediately often hits the same transient condition; exponential backoff gives the underlying issue time to clear.
retry() {
local max_attempts=5
local delay=1
local attempt=1
local cmd=("$@")
until "${cmd[@]}"; do
if (( attempt >= max_attempts )); then
echo "Failed after $attempt attempts: ${cmd[*]}" >&2
return 1
fi
echo "Attempt $attempt failed, retrying in ${delay}s..." >&2
sleep "$delay"
((attempt++))
((delay *= 2))
done
}
retry curl -sf "https://$FGT_HOST/api/v2/monitor/system/status"
The function captures the command and its arguments as an array (cmd=("$@")), which correctly preserves arguments containing spaces or special characters — a common bug in hand-rolled retry loops that instead store the command as a single string and re-split it with eval.
Structured Output Instead of Parsing Free Text
Resist the urge to grep/awk/sed your way through structured data when a tool already emits a structured format. If the upstream tool supports JSON output, use it and pipe into jq:
# Fragile — breaks the moment column spacing changes
ip addr show eth0 | grep "inet " | awk '{print $2}'
# Robust — survives formatting changes because it doesn't depend on alignment
ip -j addr show eth0 | jq -r '.[0].addr_info[] | select(.family=="inet") | "\(.local)/\(.prefixlen)"'
When the upstream tool has no JSON mode, isolate the parsing into one function so a format change only requires fixing one place:
get_interface_speed() {
local iface="$1"
ethtool "$iface" | awk -F': ' '/Speed:/ {print $2}'
}
Safe Variable Expansion
Always quote variable expansions unless you specifically want word-splitting and glob expansion:
# Wrong — breaks on filenames/hostnames with spaces, and on empty values
for f in $files; do
# Right
for f in "${files[@]}"; do
# Wrong — silently does nothing if $var is empty, rather than erroring
rm -rf $var/*
# Right — fails loudly if var is unset (combined with set -u above)
rm -rf "${var:?var is not set}"/*
The ${var:?error message} syntax is worth knowing on its own — it expands to an error and exits immediately if var is unset or empty, which is exactly the guard you want before any destructive operation like rm -rf.
Argument Parsing That Doesn’t Fall Over
A minimal but robust getopts-based parser:
#!/usr/bin/env bash
set -euo pipefail
usage() {
echo "Usage: $0 -h <host> -c <command> [-t <timeout>]" >&2
exit 1
}
timeout=5
while getopts "h:c:t:" opt; do
case "$opt" in
h) host="$OPTARG" ;;
c) command="$OPTARG" ;;
t) timeout="$OPTARG" ;;
*) usage ;;
esac
done
: "${host:?Missing -h <host>}"
: "${command:?Missing -c <command>}"
ssh -o ConnectTimeout="$timeout" "$host" "$command"
: "${var:?message}" is the same guard pattern as above, used here purely for its side effect (the : no-op command) to validate required arguments exist before continuing.
Logging That’s Actually Useful Later
log() {
local level="$1"; shift
printf '%s [%s] %s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$level" "$*" >&2
}
log INFO "Starting config push to ${#hosts[@]} hosts"
log ERROR "Failed to connect to $host"
Logging to stderr (>&2) rather than stdout keeps log noise separate from the script’s actual data output — critical if the script’s stdout is meant to be piped into something else (jq, a CSV writer, another script).
Putting It Together: A Realistic Audit Script
#!/usr/bin/env bash
set -euo pipefail
IFS=$'\n\t'
hosts=(fw01 fw02 fw03)
max_jobs=3
outdir=$(mktemp -d)
trap 'rm -rf "$outdir"' EXIT
log() { printf '%s [%s] %s\n' "$(date -u +%H:%M:%S)" "$1" "${*:2}" >&2; }
audit_host() {
local host="$1"
local out="$outdir/$host.json"
if ! ssh -o ConnectTimeout=5 -o BatchMode=yes "$host" \
"get system status" > "$out" 2>"$outdir/$host.err"; then
log ERROR "Failed to reach $host"
return 1
fi
log INFO "Collected $host"
}
for host in "${hosts[@]}"; do
while (( $(jobs -r -p | wc -l) >= max_jobs )); do
wait -n
done
audit_host "$host" &
done
wait
log INFO "Audit complete: $(ls "$outdir"/*.json 2>/dev/null | wc -l)/${#hosts[@]} hosts succeeded"
This combines strict mode, trap-based cleanup, a concurrency-capped fan-out, structured per-host output files, and timestamped logging — the patterns above are not abstract advice, they are exactly what separates a script that runs fine in testing from one that survives a 200-device estate with three hosts down for maintenance.
What to Read Next
- Python with Nornir — for estates where Netmiko/NAPALM multi-vendor abstractions and proper task concurrency are worth the overhead over raw bash + SSH
- Linux tc — the shaping tool whose test harness benefits most from the parallel bash patterns above: run netem impairments and
iperf3measurements across multiple links simultaneously