Tagged: troubleshooting
23 posts · browse all tags
-
FortiGate BGP Cookbook Part 2: Verify and Debug
The commands I run after every FortiGate BGP change: read the session, compare received and advertised routes before and after policy, soft-clear safely, triage a session that will not establish, and run BGP debug without leaving it on.
-
Proving the Path Part 1: The Chain, and the Fault We're Chasing
A companion series to Day in the Life of a Packet, framed around diagnosis rather than architecture: one Teams call that stayed choppy long after the dashboards said everything had recovered, and the nine-stage chain I used to prove where the packet actually was.
-
Proving the Path Part 2: Ingress and Session
Before touching SD-WAN or routing, the session table tells you whether the flow you're chasing predates the event you're investigating. For the choppy Teams call, that one field settles more than the dashboard did.
-
Proving the Path Part 3: Policy Route and Routing
The pinned session predates the fault. Before blaming SD-WAN or routing for what's happening now, I need to know what a brand-new flow to the same destination would actually do today, and that's a different question from what the config says on paper.
-
Proving the Path Part 4: SD-WAN Member, SLA, and Rule Selection
The health-check history shows a breach, a failover, and a recovery, all working exactly as designed. Which raises the real question: if SD-WAN did its job correctly, why did the call stay broken?
-
Proving the Path Part 5: Policy/NAT and IPsec
The tunnel the pinned session is using never went down. The policy it matched hasn't changed. Both of those facts are true, and neither of them means the session's path was healthy the whole time it mattered.
-
Proving the Path Part 6: Egress, NPU Offload, and the Verdict
Every stage checked out clean on its own terms. The fix isn't clearing everything and hoping, it's one targeted action once the evidence points at a single cause. Closing the series with the full false-conclusions list and the discipline behind it.
-
Text Processing for Network Engineers Part 11: Tracing a Bad ACL Across the Fleet, Start to Finish
A maintenance-window ACL push goes out to 150 branch firewalls. A subset break. This part ties every tool from the series together, regex, grep, sed, awk, vim, diff, find and xargs, to find the affected devices, understand why only some of them broke, and fix all of them correctly in one pass.
-
Text Processing for Network Engineers Part 10: Encoding, Line Endings, and the Gotchas That Break Your Scripts
Every tool in this series assumes plain Unix text. A config pasted from Windows, pulled over a serial console, or exported by an older device's SNMP agent frequently isn't, and the failure it causes looks exactly like a bug in your pattern instead of what it actually is.
-
Container Networking Deep Dive Part 10: A Full Walkthrough, Chasing a Cross-Node Connectivity Bug
A checkout service scales out, starts failing calls to a payment gateway on a different node, and the failure turns out to be two separate problems stacked on top of each other. A start-to-finish walkthrough using every technique from this series.
-
Container Networking Deep Dive Part 9: Troubleshooting Container Networking
A vendor-neutral methodology for the five failure modes that account for most container networking outages: conntrack exhaustion, VXLAN MTU fragmentation, DNS-in-pod failures, cross-node connectivity breaks, and tcpdump inside a network namespace.
-
OSPF on Linux Part 5: Troubleshooting OSPF on Linux
The closing post: OSPF Deep Dive Part 11's three-vendor show-command table extended to four with FRR's vtysh, then the failure modes that only exist on Linux, ip_forward, a firewall dropping protocol 89, multicast group membership, the wrong network namespace, veth MTU, and tcpdump on the wire.
-
OSPF Deep Dive Part 10: A Vendor-Neutral Troubleshooting Methodology
A structured way to diagnose OSPF problems that works on any vendor, because it's built on the neighbor state machine from Part 1, not on any platform's specific commands: what state is it stuck at, and what does that state rule in or out.
-
OSPF Deep Dive Part 11: Show Commands and the LSDB, Side by Side
The same neighbor table, interface state, and LSDB, read through three different command sets: Cisco IOS/IOS-XE, FortiOS, and Junos, mapped side by side against the CORE-CSR/BRANCH-FGT/EDGE-MX lab from this series.
-
OSPF Deep Dive Part 12: A Multi-Vendor Outage, Start to Finish
A closing walkthrough on the CORE-CSR/BRANCH-FGT/EDGE-MX lab: a complaint with no down interfaces and no failed adjacencies anywhere, a red herring that turns out to be real but irrelevant, and a root cause that traces straight back to Part 3's reference-bandwidth warning.
-
Advanced tcpdump Recipes for the Enterprise Engineer: The Packet Never Lies
Bitwise BPF masking, enterprise recipes for asymmetric routing and retransmission hunting, a safe SSH-to-Wireshark live-streaming setup that won't loop your own session, and a cross-vendor capture map spanning Debian, Cisco IOS, FortiOS, Junos, and VeloCloud.
-
A Day in the Life of a Packet on a 50G FortiGate, Part 1: Ingress, NP7, and the Fast Path
Where the packet is born on a 50G FortiGate. From the wire and DMA, through the NP7 SoC's session cache, IPSA, NTurbo, and the moment a packet either flies through hardware or crosses the bridge into the kernel slow path.
-
A Day in the Life of a Packet on a 50G FortiGate, Part 2: Stateful Inspection, Session Lookup, and Anti-Spoofing
The packet has been punted from the NP7 to the kernel. Now FortiOS does the things ASICs cannot: IP integrity, DoS sensors, RPF, session table lookup, helpers, and the state machine that decides whether this is a brand new flow or one we already know.
-
A Day in the Life of a Packet on a 50G FortiGate, Part 3: Routing, Policy Routes, and SDWAN Service Rules
The packet has a session entry and now needs to know where to go. FortiOS resolves that in a strict order: policy routes, then SDWAN service rules, then the FIB. Each layer has its own logic, its own match criteria, and its own diagnostic surface.
-
A Day in the Life of a Packet on a 50G FortiGate, Part 4: Firewall Policy, NAT, and Security Profiles
Routing told the packet where it's going. Firewall policy decides whether it's allowed, NAT rewrites it, and security profiles inspect it. Inside the iprope chain, central NAT vs policy NAT, VIPs, IP pools, and the flow-vs-proxy UTM pipeline.
-
A Day in the Life of a Packet on a 50G FortiGate, Part 5: Egress, NPU Offload, and the Full Troubleshooting Cookbook
The packet is decided. Now it has to actually leave. Egress shaping, NPU offload re-evaluation, IPsec encap, ARP, transmit. Then a single-page reference of every diagnose, get, and show command from across this series.
-
NSE5 Part 9: Diagnostics and Troubleshooting
Part 9 of the NSE5 study series — covers the FortiManager diagnostic toolbox: device-manager diagnostics, the FGFM tunnel, install-failure forensics, oftpd, packet capture, and the debug commands worth knowing under exam pressure.
-
tcpdump Deep Dive: BPF Filters, Capture Rotation, and Cross-Mapping to FortiGate's diagnose sniffer packet
A practical, command-heavy guide to getting real value out of tcpdump — precise BPF filters, production-grade ring-buffer captures, and a side-by-side mapping to FortiGate's diagnose sniffer packet so you can switch between the two without losing your place.