Stage 3 · Build

Observability, Tracing & Security

bpftrace & BCC

One-liners for latency, throughput, and syscall tracing — instant observability without code.

6 min readLinux Deep DivesBuild

bpftrace Overview

bpftrace is a high-level tracing language for eBPF. It provides a awk-like syntax for writing kernel tracing scripts. bpftrace is ideal for ad-hoc investigation, performance analysis, and debugging.

Install and verify


# Install bpftrace
sudo apt install bpftrace    # Debian/Ubuntu
sudo dnf install bpftrace    # Fedora/RHEL

# Verify installation
bpftrace --version
# bpftrace v0.17.0

# List available probes
sudo bpftrace -l 'tracepoint:*' | head -20
sudo bpftrace -l 'kprobe:*' | grep tcp

bpftrace uses eBPF under the hood but provides a much simpler interface. One-liners can be typed directly in the terminal.

Probe Types

Probe	Syntax	Description
tracepoint	tracepoint:category:event	Stable kernel tracepoints
kprobe	kprobe:function	Kernel function entry
kretprobe	kretprobe:function	Kernel function return
uprobe	uprobe:/path:func	Userspace function entry
profile	profile:hz:997	Periodic sampling
interval	interval:s:1	Timed output
BEGIN/END	BEGIN/END	Script start/end

Powerful One-Liners

bpftrace one-liners


# Trace process execution
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_execve { printf("%s (pid=%d)\n", comm, pid); }'

# Measure syscall latency
sudo bpftrace -e 'kprobe:do_sys_open { @start[tid] = nsecs; } kretprobe:do_sys_open /@start[tid]/ { @us = hist((nsecs - @start[tid]) / 1000); delete(@start[tid]); }'

# Count syscalls by process
sudo bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); }'

# Trace TCP connections
sudo bpftrace -e 'kprobe:tcp_connect { printf("connect: %s\n", comm); }'

# Disk I/O latency histogram
sudo bpftrace -e 'tracepoint:block:block_rq_complete { @usecs = hist(args->io_duration / 1000); }'

bpftrace one-liners are the fastest way to investigate production issues. No code compilation or module loading required.

Writing Scripts

disklatency.bt — Disk I/O latency tracer


#!/usr/bin/env bpftrace
/*
 * Disk I/O latency by process
 */

tracepoint:block:block_rq_issue
{
    @io_start[args->dev, args->sector] = nsecs;
}

tracepoint:block:block_rq_complete
/@io_start[args->dev, args->sector]/
{
    $delta_us = (nsecs - @io_start[args->dev, args->sector]) / 1000;
    @us[comm] = hist($delta_us);
    delete(@io_start[args->dev, args->sector]);
}

END
{
    clear(@io_start);
}

This script tracks I/O latency by correlating block_rq_issue and block_rq_complete tracepoints. The hist() function prints a histogram of latency values.

BCC Tools

BCC (BPF Compiler Collection) provides pre-built tools for common tracing tasks. These tools are ready to use and cover most diagnostic needs.

Essential BCC tools


# Install BCC
sudo apt install bpfcc-tools

# Process execution tracing
sudo execsnoop           # Trace new processes

# TCP connection tracing
sudo tcpconnect          # Trace outgoing connections
sudo tcpaccept           # Trace incoming connections

# Disk I/O
sudo biosnoop             # Trace block I/O
sudo cachestat            # Page cache hit/miss rate

# CPU profiling
sudo profile              # CPU stack profile (flame graph input)

# File system
sudo opensnoop            # Trace file opens
sudo filetop              # Top files by I/O

BCC tools are the quickest way to diagnose production issues. Each tool is a self-contained eBPF program with a user-friendly interface.

Production Usage

Production-safe tracing


# Limit tracing to specific processes
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_read /pid == 1234/ { @bytes = count(); }'

# Use intervals for bounded output
sudo bpftrace -e 'interval:s:1 { printf("CPU: "); runqlat; }'

# Filter by cgroup (container)
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat /cgroup == cgroupid("/sys/fs/cgroup/mycontainer")/ { printf("%s\n", comm); }'

# Limit output rate
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_write { @[comm] = count(); } interval:s:10 { print(@); clear(@); }'

Always filter by PID, cgroup, or time window in production. Unfiltered tracing generates enormous output and adds overhead.

Use profile for CPU flame graphs

sudo profile -f 997 -p <pid> | flamegraph.pl > cpu.svg generates CPU flame graphs. The 997 Hz frequency provides good resolution without excessive overhead.

bpftrace requires root

Most bpftrace operations require root privileges. Use sudo or grant CAP_BPF and CAP_PERFMON capabilities. In containers, mount /sys/kernel/debug for full functionality.

Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.