Text Processing for Network Engineers Part 10: Encoding, Line Endings, and the Gotchas That Break Your Scripts

Every tool this series has covered, grep, sed, awk, vim, diff, assumes the input is plain, consistent Unix text: lines separated by a single \n, characters encoded in a way the tool expects. That assumption breaks more often than it should, and when it does, the failure looks exactly like a bug in your pattern rather than what it actually is.

CRLF vs LF: the one that bites hardest

Unix text uses a single line feed (\n, 0x0A) to end a line. Windows uses a carriage return followed by a line feed (\r\n, 0x0D 0x0A). A config exported, edited, or emailed through anything that touched Windows along the way frequently carries \r\n line endings, and on a Linux box, that trailing \r is invisible in a normal terminal but very much present in the file.

The failure mode: a sed or grep pattern anchored with $ (end of line) stops matching, because the actual end of the line, byte-for-byte, is \r, not the character immediately before it:

grep 'shutdown$' interface-config.txt        # matches nothing, because the real line is "shutdown\r"

There’s no error. It just silently finds zero matches, and the natural conclusion is “my regex is wrong,” when the regex was fine and the file wasn’t what you assumed it was.

Detect it:

file interface-config.txt          # often reports "ASCII text, with CRLF line terminators"
cat -A interface-config.txt | head  # shows every \r as a literal ^M at the end of each line

cat -A (or cat -e, a subset of the same flag) is the fastest way to actually see the hidden character rather than infer its presence from a tool’s failure. Fix it:

dos2unix interface-config.txt          # in place, converts CRLF to LF
sed -i 's/\r$//' interface-config.txt   # the sed equivalent, if dos2unix isn't installed

The sed version works because it explicitly matches a literal \r immediately before the (now correctly recognized) end of line and deletes it, which is the same anchoring concept from Part 1 applied to solving this exact problem rather than being broken by it.

BOM: the three invisible bytes at the start of a file

A byte-order mark (EF BB BF for UTF-8) sometimes appears at the very start of a file saved by Windows-native tools (Notepad in particular, historically), even though UTF-8 doesn’t strictly need one. It’s invisible in most text viewers, but it’s real data sitting before the first character of the file, and it will break a pattern anchored at the very start with ^ on line one specifically, or cause a config parser expecting the first line to be a specific keyword to fail on line one only, every single time.

xxd interface-config.txt | head -1      # look at the raw bytes; EF BB BF at the start is the tell
sed -i '1s/^\xef\xbb\xbf//' interface-config.txt   # strip it from line 1 only

xxd (or hexdump -C) is the tool for “what is actually in this file, at the byte level,” which is the right escalation once cat -A isn’t specific enough to identify the problem.

Character encoding: iconv, for when the bytes aren’t ASCII at all

CRLF and BOM are both still ASCII-compatible problems. A genuine encoding mismatch is different: a config pulled from an older device whose SNMP sysLocation string, or a config comment field, contains a non-ASCII character (an accented name, a currency symbol, anything outside the 0-127 ASCII range) encoded in Latin-1 (ISO-8859-1) rather than UTF-8, will produce mangled or outright rejected characters the moment a UTF-8-expecting tool (which is most of the modern Linux userland by default) tries to read it as text.

file -i interface-config.txt                          # reports the detected encoding
iconv -f ISO-8859-1 -t UTF-8 interface-config.txt -o interface-config-utf8.txt

file -i gives a best-guess at the source encoding; iconv -f <from> -t <to> does the actual conversion. This comes up often enough with older network gear specifically that it’s worth checking file -i as a standing first step whenever a script chokes on a config pull from an unfamiliar or older device, rather than assuming the file is UTF-8 by default and debugging everything else first.

Serial consoles and TFTP transfers: where stray bytes actually come from

Two specific transfer paths are worth naming because they’re common sources of this whole class of problem on network gear specifically, rather than a general Linux quirk: a config captured off a serial console session can pick up terminal control sequences or stray characters from the terminal emulator’s own settings (baud rate mismatches occasionally corrupt individual bytes, and copy-paste through some terminal emulators silently reflows or re-encodes text), and a config pushed or pulled via TFTP, an unauthenticated, unencrypted, historically fragile protocol, has no built-in integrity check at all, so a truncated or corrupted transfer produces a file that looks superficially fine until a specific line fails to parse.

The habit that catches both: after any pull from a device, file and cat -A on the resulting text before trusting it as clean input to any of the tools in this series, the same two-command check used above, applied as routine rather than only when something has already gone wrong.

The standing check

Before debugging a “why doesn’t my pattern match” problem any further, run file and cat -A on the input first. It costs two commands and rules out the entire category of encoding and line-ending problems this part covers, before you spend time doubting a regex that was correct all along. Part 11 puts every tool from this series to work in a single realistic incident: a bad ACL pushed fleet-wide, traced and fixed start to finish using nothing but grep, sed, awk, vim, diff, and the tools around them.