Text Processing for Network Engineers Part 5: cut, sort, uniq, tr, and the Rest of the Supporting Cast

grep, sed, and awk are the three tools everyone eventually learns by name. The tools in this part rarely get top billing, but they’re in almost every real pipeline sitting between those three, doing the small jobs that would otherwise need an awk one-liner of their own.

cut: field extraction without awk’s overhead

When the job is genuinely just “give me column 3,” cut is less to type than awk and makes the intent obvious to the next person reading the script:

cut -d',' -f2 interface-inventory.csv        # field 2, comma-delimited
cut -d':' -f1 /etc/passwd                    # field 1, colon-delimited
cut -c1-15 fixed-width-report.txt            # characters 1 through 15, not field-based at all

That last form, character-position rather than delimiter-based, is the thing cut can do that awk -F can’t do as cleanly: fixed-width legacy output (some vendor show commands still are) where there’s no consistent delimiter, just column position.

sort: the step almost every pipeline needs before uniq

sort alone is unremarkable. sort combined with the right flag turns raw output into a ranked report:

sort -n counts.txt          # numeric sort (default sort is lexical: "10" comes before "9")
sort -rn counts.txt         # numeric, reversed: biggest first
sort -u list.txt            # sort and deduplicate in one pass
sort -k2 -t',' data.csv     # sort by the second comma-delimited field

The -n flag matters more than it looks: without it, sort treats 10 and 9 as strings, and strings sort 10 before 9 because 1 sorts before 9 character by character. Every “why is my sorted list of interface speeds in the wrong order” question has this as the answer.

uniq: counting, but only on adjacent duplicates

uniq removes adjacent duplicate lines, and with -c counts how many times each run of duplicates occurred. The trap: it only looks at adjacent lines, so the input almost always needs a sort first:

sort access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head -10

That’s a top-ten-source-IPs report built entirely from pipeline plumbing: extract the IP field, sort so duplicates are adjacent, count the runs, sort the counts descending, keep the top ten. No awk arithmetic, no Python, just five small tools chained. uniq -d prints only lines that had duplicates (useful for finding repeated config lines that shouldn’t be there), uniq -u prints only lines that had no duplicates (useful for finding the one-off entry in an otherwise uniform list).

tr: character-level substitution and deletion

tr operates on individual characters, not lines or fields, which makes it the right tool for format normalization before something else in the pipeline gets confused by inconsistent characters:

tr 'A-Z' 'a-z' < hostnames.txt                # force lowercase
tr -d '\r' < windows-exported.txt              # strip stray carriage returns (see Part 10)
tr -s ' ' < padded-output.txt                  # squeeze repeated spaces into one
tr ':' ',' < mac-addresses.txt                 # convert colon-separated MACs to comma-separated

tr -s ' ' is the quiet fix behind a huge number of “why doesn’t my awk $2 line up” problems: show command output from network gear is frequently padded with variable amounts of whitespace for alignment, and awk’s default whitespace splitting mostly handles that already, but piping through tr -s ' ' first makes the columns visually predictable too, which matters when you’re eyeballing the output before scripting against it.

column: turning delimited data into an aligned table

column -t takes delimiter-separated input and pads it into aligned columns, purely for human readability:

column -t -s',' interface-inventory.csv

It doesn’t change the data, only the on-screen alignment, so it’s a “last step before you look at it” tool, not something you’d put in the middle of a pipeline feeding another program.

paste: gluing files together side by side

Where cat stacks files vertically, paste merges them horizontally, line by line:

paste hostnames.txt ip-addresses.txt

Given two files, one hostname per line and one IP per line in matching order, that produces a two-column hostname/IP table without writing a loop. It’s a narrow tool, but for combining two separately generated lists (say, a list of device names from inventory and a list of IPs pulled with the greps and awks earlier in this series) it’s the fastest way there.

wc: counting, when a count is the whole answer

wc -l configs/*.cfg          # line count per file, plus a total
grep -c 'interface' running-config.txt   # (a reminder: grep's own -c is often enough on its own)

The worked example: top talkers from a flow log, start to finish

Put five of these together against a raw flow log where each line has a source IP in the first field:

awk '{print $1}' flows.log | sort | uniq -c | sort -rn | head -20 | column -t

Read left to right: pull the source-IP field, sort so duplicates sit next to each other, count each run, sort the counts biggest-first, keep the top twenty, and align the output into a clean table. That’s a complete top-talkers report in one line, built from awk (field extraction), sort (twice, for two different jobs), uniq (counting), and column (formatting), with no single tool doing more than one small job.

This is the actual shape of most real Unix text-processing work: not one clever tool doing everything, but several boring tools each doing one thing, chained with |. Part 6 moves from processing text on the command line to editing it directly, on a box that may have no GUI available at all: vim.