← Back to Linux

How to Use grep, awk and sed: Linux Text Processing Guide

grep, awk, and sed are three powerful command-line utilities that form the backbone of Linux text processing. grep searches for patterns, sed performs stream editing and substitutions, and awk extracts and transforms data from structured text. Together, they solve 90% of text manipulation tasks you'll encounter.

Understanding the Three Tools

These three utilities work differently but often complement each other in pipelines. grep filters lines matching a pattern. sed modifies text in a stream without opening it in an editor. awk is a full programming language designed for processing columnar data. Knowing when to use each tool saves you from writing complex scripts.

grep: Finding Patterns

grep stands for "global regular expression print." It searches through files or standard input, printing lines that match a pattern. The basic syntax is straightforward: grep [options] pattern [file]. If you don't specify a file, grep reads from stdin, making it perfect for pipelines.

Common grep options include -i for case-insensitive matching, -v to invert the match (lines that don't match), -c to count matches, and -n to show line numbers. The -r flag recursively searches directories, which is invaluable for searching codebases.

# Find all lines containing "error" in a log file
grep "error" /var/log/syslog

# Count occurrences of "warning" (case-insensitive)
grep -ci "warning" /var/log/syslog

# Show line numbers for matches
grep -n "failed" /var/log/auth.log

# Find lines NOT containing "success"
grep -v "success" results.txt

# Recursively search all .txt files in a directory
grep -r "TODO" .

grep supports regular expressions by default, but you can use -F for fixed string matching (faster for literal strings) or -E for extended regex. The -E flag lets you use +, ?, and pipe | operators without escaping.

# Find lines starting with "ERROR"
grep "^ERROR" logfile.txt

# Match IP addresses (basic pattern)
grep -E "^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$" ips.txt

# Find either "warning" or "error"
grep -E "warning|error" syslog

# Use multiple patterns from a file
grep -f patterns.txt data.txt

sed: Stream Editing and Substitution

sed is a stream editor that processes text line by line, applying transformations without requiring you to load the entire file into memory. The most common use is substitution with the s command. The syntax is sed 's/old/new/' or sed 's/old/new/g' where g means global (replace all occurrences, not just the first).

By default, sed prints every line. With the -n flag, it only prints lines you explicitly tell it to. This is useful when you want to extract specific lines or modified versions of lines.

# Replace first occurrence of "old" with "new" on each line
sed 's/old/new/' file.txt

# Replace all occurrences of "old" with "new"
sed 's/old/new/g' file.txt

# Replace "old" with "new" only on line 5
sed '5s/old/new/' file.txt

# Replace "old" with "new" on lines 5-10
sed '5,10s/old/new/' file.txt

# Case-insensitive replacement
sed 's/old/new/i' file.txt

sed can also delete lines with the d command. You specify which lines to delete by line number, range, or pattern. The p command prints lines (useful with -n to print only matching lines). You can also insert, append, or change lines entirely.

# Delete all blank lines
sed '/^$/d' file.txt

# Delete lines matching a pattern
sed '/^#/d' config.txt

# Print only lines containing "important" (suppress default output with -n)
sed -n '/important/p' file.txt

# Insert text before matching lines
sed '/ERROR/i ALERT:' logfile.txt

# Append text after matching lines
sed '/\[END\]/a --- End of Section ---' document.txt

sed also supports addresses using regular expressions. For example, /pattern/,/pattern2/ matches from the first pattern to the second pattern on the same or subsequent lines. You can apply multiple operations using the -e flag or by separating commands with semicolons.

# Apply multiple substitutions
sed -e 's/old1/new1/g' -e 's/old2/new2/g' file.txt

# Delete from first "START" to first "END"
sed '/START/,/END/d' file.txt

# Swap two columns (if separated by whitespace)
sed 's/^\([^ ]*\) \([^ ]*\)/\2 \1/' file.txt

awk: Data Extraction and Processing

awk is more powerful than grep and sed because it's a complete programming language. It automatically splits each line into fields, making it ideal for processing structured text like CSV files, logs, or columnar data. The basic syntax is awk 'pattern { action }' file.

awk divides input into records (lines by default) and fields (columns, separated by whitespace or a specified delimiter). You reference fields as ₹85, ₹170, etc., where ₹0 is the entire line. Built-in variables like NR (record number) and NF (number of fields) are automatically available.

# Print the first field of each line
awk '{print ₹85}' file.txt

# Print lines where the third field equals "error"
awk '₹260 == "error" {print}' file.txt

# Print the second and fourth fields
awk '{print ₹170 ₹340}' file.txt

# Print line number and content
awk '{print NR, ₹0}' file.txt

# Count the total number of lines
awk 'END {print NR}' file.txt

awk supports BEGIN and END blocks. BEGIN runs before processing input, useful for setting variables or printing headers. END runs after all input is processed, ideal for printing totals or summaries. This makes awk excellent for generating reports from raw data.

# Sum values in the second column
awk '{sum += ₹170} END {print "Total:", sum}' numbers.txt

# Count occurrences of each unique value in column 1
awk '{count[₹85]++} END {for (word in count) print word, count[word]}' data.txt

# Print header and formatted output
awk 'BEGIN {print "Name\tAge\tCity"} {printf "%-15s %3d %s\n", ₹85 ₹170 ₹260}' people.txt

# Calculate average of column 2
awk '{sum += ₹170; n++} END {print "Average:", sum/n}' data.txt

You can specify a field separator using the -F flag. This is essential for CSV files or any delimited format. awk also supports string functions like substr(), length(), gsub(), and split(), plus mathematical functions.

# Process CSV file (comma-separated values)
awk -F',' '{print ₹170}' data.csv

# Extract substring from field 1, characters 1-5
awk '{print substr(₹85 1, 5)}' file.txt

# Replace all occurrences within a field
awk '{gsub(/old/, "new", ₹170); print}' file.txt

# Get the length of each line
awk '{print length(₹0), ₹0}' file.txt

# Print columns only if they match a condition
awk '₹170 > 100 {print ₹85 ₹170}' data.txt

Combining grep, sed, and awk in Pipelines

The real power emerges when you chain these tools together using pipes. You might use grep to filter relevant lines, sed to clean up formatting, and awk to extract and summarize data. Each tool does one thing well, and together they're unstoppable.

# Find all error lines, count them by type
grep "ERROR" logfile.txt | awk -F': ' '{print ₹170}' | sort | uniq -c

# Extract IP addresses from logs, remove duplicates
grep -oE "[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}" access.log | sort -u

# Extract usernames from /etc/passwd, remove blank lines, count them
cat /etc/passwd | sed '/^$/d' | awk -F':' '{print ₹85}' | wc -l

# Find lines with "warning", extract the timestamp, format them
grep "warning" syslog | sed 's/\[//' | sed 's/\]//' | awk '{print ₹85 ₹170 ₹260}'

# Process Apache logs: extract IP and response code for errors
grep " 5[0-9][0-9] " access.log | awk '{print ₹85 ₹760}' | sort | uniq -c

Performance Considerations

When working with large files, efficiency matters. grep is the fastest for simple pattern matching because it's optimized for that single task. sed processes files sequentially without loading them entirely into RAM. awk uses more memory but handles complex logic efficiently.

For searching specific strings (not patterns), use grep -F instead of regex—it's significantly faster. When processing massive files, avoid collecting all results in memory; use streaming operations and pipes. If you're calling grep/sed/awk repeatedly, consider writing a single awk script instead of multiple pipes.

The -l flag in grep only prints filenames, which is faster than printing matches when you just need to know which files contain a pattern. For very large datasets, consider parallel processing with tools like parallel or GNU xargs combined with these utilities.

Practical Real-World Examples

Let's apply these tools to actual scenarios you'll encounter. Suppose you have an Apache access log and want to find the most common referrers: awk '{print ₹940}' access.log | grep -v "^-$" | sed 's/^"//; s/"$//' | sort | uniq -c | sort -rn | head -10. This extracts the 11th field, filters out empty referrers, removes quotes, counts, sorts by frequency, and shows the top 10.

For extracting user data from a CSV file where you want only active users with a score above 80: awk -F',' '₹260 == "active" && ₹340 > 80 {print ₹85 ₹170}' users.csv. This reads comma-separated values, filters by status and score, and prints name and email.

To clean up a configuration file by removing comments and blank lines: sed '/^#/d; /^$/d' config.conf. Or, to find all files modified in the last day containing a specific error: find . -mtime -1 -type f -exec grep -l "OutOfMemory" {} \;. These patterns scale from simple tasks to complex data processing workflows.