Stage 3 · Build
Observability, Tracing & Security
perf & Profiling Tools
CPU profiling, flame graphs, and hardware performance counters — understanding where time is spent.
perf Overview
perf is the standard Linux profiling tool. It uses hardware performance counters (PMCs) and software events to measure CPU cycles, cache misses, branch predictions, and more. perf is the foundation of most Linux performance analysis.
# Install perf
sudo apt install linux-tools-$(uname -r)
sudo dnf install perf
# Verify
perf --version
# perf version 6.6
# List available events
sudo perf list | head -20
# List of pre-defined events:
# cpu-cycles [hardware event]
# instructions [hardware event]
# cache-misses [hardware event]
# context-switches [software event]
# page-faults [software event]perf uses the perf_event_open() syscall to access hardware counters. Most events require no kernel modules — they are built into the CPU.
CPU Profiling
# Profile entire system for 10 seconds
sudo perf record -g -a sleep 10
# Profile a specific command
sudo perf record -g ./myapp --args
# Profile a running process
sudo perf record -g -p <pid> sleep 30
# View the report
sudo perf report --no-children -g graph
# Overhead Command Shared Object Symbol
# 45.23% myapp myapp [.] process_data
# 12.34% myapp libc.so.6 [.] __memcpy_avx2The -g flag records call graphs, showing which functions called the hot functions. --no-children hides child function overhead for cleaner output.
Flame Graphs
Flame graphs visualize stack traces as a flame chart. Each rectangle is a function; wider rectangles indicate more time spent. The x-axis shows call frequency, the y-axis shows stack depth.
# Record and generate a flame graph
sudo perf record -g -a sleep 10
sudo perf script | stackcollapse-perf.pl | flamegraph.pl > cpu.svg
# For a specific process
sudo perf record -g -p <pid> sleep 10
sudo perf script | stackcollapse-perf.pl | flamegraph.pl > cpu.svg
# Off-CPU flame graph (time spent sleeping)
sudo offcputime-bpfcc -p <pid> -f 10 | flamegraph.pl > offcpu.svgFlame graphs are the fastest way to identify performance bottlenecks. Look for wide plateaus (hot loops) and tall stacks (deep call chains).
Hardware Counters
# Record specific hardware events
sudo perf record -e cache-misses,instructions,cycles sleep 5
# Event ratio analysis
sudo perf stat -e cycles,instructions,cache-misses,cache-references ./myapp
# 1,234,567,890 cycles
# 456,789,012 instructions # 0.37 insn per cycle
# 12,345,678 cache-misses # 12.34% of cache refs
# Branch prediction
sudo perf stat -e branch-misses,branch-instructions ./myapp
# 5,678,901 branch-misses # 2.34% of all branchesLow IPC (instructions per cycle) indicates the CPU is waiting on memory. High cache-miss rate indicates poor data locality. Use this to guide optimization.
perf stat
# Quick benchmark
perf stat ./myapp
# 1,234,567,890 cycles
# 456,789,012 instructions
# 12,345,678 cache-misses
# 12,345 context-switches
# 1,234 page-faults
# Wall clock: 2.345678901 seconds
# Compare two runs
perf stat -o before.txt ./myapp
# ... make changes ...
perf stat -o after.txt ./myapp
perf diff before.txt after.txtperf stat is the quickest way to get a performance baseline. Compare before/after to validate optimizations.
perf trace
# Trace syscalls (like strace but faster)
sudo perf trace ls
# 0.000 ( 0.005s): ls /bin/ls -> 3
# 0.006 ( 0.001s): close(fd: 3) = 0
# Trace with summary
sudo perf trace -s ls
# Trace specific syscalls
sudo perf trace -e openat,read,write curl http://example.com
# Attach to running process
sudo perf trace -p <pid> -e read,writeperf trace is faster than strace because it uses perf_event_open() instead of ptrace(). Use it when strace overhead is too high.
Start with perf stat for a quick overview, then use perf record -g for detailed profiling. Flame graphs make it easy to spot the hottest code paths.
When perf stat shows high cache-misses and low IPC, the bottleneck is memory access. Profile with perf record -e cache-misses to find the code causing cache thrashing.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.