Stage 2 · Tools
Text Processing
Log Analysis in Practice
Parse Nginx/Apache access logs, extract metrics, and summarize errors with shell tools.
Understanding Log Formats
Most web servers use predictable log formats. Nginx and Apache use a combined log format that includes IP, timestamp, request, status code, and user agent. Understanding this structure is key to parsing logs.
192.168.1.1 - alice [15/Jan/2025:10:30:00 +0000] "GET /api/users HTTP/1.1" 200 1234 "https://example.com" "Mozilla/5.0"
| | | | | | | | |
IP - username timestamp method path status size referer user-agentEach field is separated by spaces, except the timestamp which is wrapped in brackets and the request which is in quotes. Understanding this structure makes parsing straightforward.
Parsing Access Logs
The combination of awk, grep, and sort makes log parsing efficient. Here are the essential patterns for extracting information from access logs.
# Extract all IPs
awk '{print $1}' access.log
# Extract unique IPs
awk '{print $1}' access.log | sort -u
# Extract status codes
awk '{print $9}' access.log
# Extract requested paths
awk '{print $7}' access.log
# Extract request method and path
awk '{print $6, $7}' access.log | tr -d '"'awk is the fastest tool for field extraction from logs. $1 is the IP, $9 is the status code, $7 is the path. The exact field numbers depend on your log format.
Extracting Metrics
Once you can parse logs, you can extract meaningful metrics: request counts, response times, error rates, and traffic patterns.
# Requests per IP (top 10)
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -10
# Requests per hour
awk -F'[/: ]' '{print $5":"$6}' access.log | sort | uniq -c
# Status code distribution
awk '{print $9}' access.log | sort | uniq -c | sort -rn
# 404 error pages
awk '$9 == 404 {print $7}' access.log | sort | uniq -c | sort -rn | head -20
# Response size statistics
awk '{sum+=$10; count++} END {print "Avg size:", sum/count, "bytes"}' access.logThese patterns give you immediate insight into your traffic. The status code distribution shows errors. The per-IP count shows heavy users. The 404 list shows broken links.
# Requests per minute (for the last hour)
awk '$4 >="['"$(date -d '1 hour ago' '+%d/%b/%Y:%H')"' access.log |
awk -F'[/: ]' '{print $5":"$6":"$7}' | sort | uniq -c
# Peak traffic hours
awk -F'[/: ]' '{print $6}' access.log | sort | uniq -c | sort -rn
# Traffic over time (hourly)
awk '{print substr($4,2,12)}' access.log | sort | uniq -cTime-based analysis reveals traffic patterns. You can identify peak hours, detect anomalies, and plan capacity. The timestamp parsing varies by log format.
Error Analysis
Error analysis focuses on 4xx and 5xx status codes. Understanding errors helps you fix broken endpoints, identify security issues, and improve user experience.
# All errors (4xx and 5xx)
awk '$9 >= 400 {print $9, $7}' access.log | sort | uniq -c | sort -rn
# 5xx errors (server errors)
awk '$9 >= 500 {print $0}' access.log
# Error rate percentage
awk '{
total++
if ($9 >= 400) errors++
} END {
printf "Error rate: %.2f%%\n", (errors/total)*100
}' access.log
# Errors by IP
awk '$9 >= 400 {print $1}' access.log | sort | uniq -c | sort -rn
# Unique error URLs
awk '$9 >= 400 {print $9, $7}' access.log | sort -uError analysis is the first step in debugging production issues. The error rate percentage tells you the overall health. The per-IP breakdown can reveal bots or attackers.
Ensure your Nginx/Apache uses the combined log format. It includes the Referer and User-Agent fields which are essential for analyzing traffic sources and detecting bots.
Automated Reports
Combining these patterns into automated scripts gives you regular visibility into your application's health. These scripts can run via cron or CI pipelines.
#!/usr/bin/env bash
set -euo pipefail
LOG_FILE="{1:-/var/log/nginx/access.log}"
REPORT_DATE="{2:-$(date -d yesterday '+%d/%b/%Y')}"
echo "=== Daily Report for $REPORT_DATE ==="
echo ""
# Total requests
TOTAL=$(awk -v date="$REPORT_DATE" '$4 ~ date' "$LOG_FILE" | wc -l)
echo "Total requests: $TOTAL"
# Status code breakdown
echo ""
echo "Status codes:"
awk -v date="$REPORT_DATE" '$4 ~ date {print $9}' "$LOG_FILE" |
sort | uniq -c | sort -rn
# Top endpoints
echo ""
echo "Top 10 endpoints:"
awk -v date="$REPORT_DATE" '$4 ~ date {print $7}' "$LOG_FILE" |
sort | uniq -c | sort -rn | head -10
# Error summary
ERRORS=$(awk -v date="$REPORT_DATE" '$4 ~ date && $9 >= 400' "$LOG_FILE" | wc -l)
echo ""
echo "Errors (4xx/5xx): $ERRORS"
if [[ $TOTAL -gt 0 ]]; then
echo "Error rate: $(echo "scale=2; $ERRORS * 100 / $TOTAL" | bc)%"
fiThis script generates a daily report from yesterday's logs. It can be scheduled with cron to run every morning. The -v flag passes shell variables into awk for date filtering.
#!/usr/bin/env bash
set -euo pipefail
LOG_DIR="/var/log/nginx"
KEEP_DAYS=30
# Archive old logs
find "$LOG_DIR" -name "access.log.*.gz" -mtime +$KEEP_DAYS -delete
# Compress uncompressed logs
find "$LOG_DIR" -name "access.log.[0-9]*" ! -name "*.gz" -exec gzip {} \;
echo "Log cleanup completed"Log rotation prevents disk space issues. gzip compresses old logs by 80-90%. Deleting logs older than a retention period keeps disk usage manageable.
Always test your log parsing scripts on a small sample first. A wrong awk field number or regex can silently produce incorrect results. Verify with head -100 access.log | your_script before running on full logs.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.