Stage 3 · Build
Linux Kernel Internals
Kernel Debugging
ftrace, perf, kprobes, and crash dump analysis — tools for diagnosing kernel-level issues.
ftrace Function Tracer
ftrace is the kernel's built-in function tracer. It can trace function calls, schedule events, interrupts, and more. It is the foundation of many tracing tools and provides low-overhead visibility into kernel behavior.
# Trace a specific function
cd /sys/kernel/debug/tracing
# See available tracers
cat available_tracers
# nop function function_graph
# Trace function calls (brief)
echo function > current_tracer
echo 1 > tracing_on
sleep 1
echo 0 > tracing_on
cat trace | head -20
# <...>-1234 [001] d... 12345.678: schedule -> __schedule
# <...>-1234 [001] d... 12345.679: __schedule -> scheduleThe function_graph tracer shows function call duration with indentation. The nop tracer records events without filtering.
# Trace only functions matching a pattern
echo tcp_sendmsg > set_ftrace_filter
echo function > current_tracer
echo 1 > tracing_on
# ... generate network traffic ...
echo 0 > tracing_on
cat trace
# Reset
echo > set_ftrace_filter
echo nop > current_tracerset_ftrace_filter lets you trace only the functions you care about, dramatically reducing overhead and noise.
kprobes and kretprobes
kprobes let you instrument any kernel function at runtime without recompilation. A kprobe fires when the probed function is entered; a kretprobe fires when it returns. This is the basis for dynamic tracing tools like bpftrace.
# Trace when a function is called
sudo bpftrace -e 'kprobe:tcp_sendmsg { printf("tcp_sendmsg called\n"); }'
# Trace with arguments
sudo bpftrace -e 'kprobe:tcp_sendmsg { printf("pid=%d size=%d\n", pid, arg1); }'
# Trace function return
sudo bpftrace -e 'kretprobe:tcp_sendmsg { printf("returned %d\n", retval); }'
# List available kprobe events
cat /sys/kernel/debug/tracing/available_filter_functions | grep tcpkprobes are powerful but must be used carefully. Probing the wrong function or accessing invalid arguments can cause kernel panics. Always use bpftrace or other safe interfaces.
perf for Kernel Analysis
perf is the standard Linux profiling tool. It uses hardware performance counters to measure CPU cycles, cache misses, branch predictions, and more. It can profile both userspace and kernel code.
# Record kernel samples for 10 seconds
sudo perf record -g -a sleep 10
# Show kernel call graph
sudo perf report --no-children -g graph
# Specific event profiling
sudo perf stat -e cache-misses,cache-references,instructions sleep 5
# Trace context switches
sudo perf record -e context-switches -a sleep 5
sudo perf reportperf record captures samples at ~1000 Hz by default. The -g flag records call graphs, showing you which functions called the hot path.
Crash Dump Analysis
When a kernel panic occurs, the crash dump captures the state of memory at the time of failure. Analyzing this dump with the crash utility reveals what went wrong — stack traces, open files, process state, and kernel logs.
# Enable kdump (kernel crash dumping)
sudo systemctl enable kdump
sudo systemctl start kdump
# Check kdump status
cat /sys/kernel/kexec_crash_loaded
# 1 = loaded and ready
# Trigger a test crash (CAREFUL!)
echo c > /proc/sysrq-trigger
# After reboot, analyze the dump
crash /usr/lib/debug/lib/modules/$(uname -r)/vmlinux /var/crash/*/vmcore
# Inside crash:
# bt - backtrace of current task
# log - kernel ring buffer
# ps - process list
# files - open files for current task
# mod - loaded modulesKeep kernel debuginfo packages installed for crash analysis. The vmlinux file with debug symbols is required to get meaningful stack traces.
Debug Kernel Options
The kernel has many compile-time and runtime debug options. Enable them for development and testing, but never in production.
| Option | Effect | Performance Impact |
|---|---|---|
| CONFIG_DEBUG_KERNEL | Enable debug features | Significant |
| CONFIG_KASAN | Kernel Address Sanitizer | 2-3x slowdown |
| CONFIG_PROVE_LOCKING | Lockdep lock validator | Moderate |
| CONFIG_FTRACE | Function tracing | Low when off |
| CONFIG_DEBUG_FS | debugfs interface | Low |
# Check if KASAN is enabled
zgrep KASAN /proc/config.gz
# CONFIG_KASAN is not set (production kernel)
# Check lockdep status
cat /proc/lockdep_stats
# lock-validated: 12345
# Runtime debug parameters
cat /proc/sys/kernel/sched_debug
# 0 = off, 1 = on
# Enable debug for specific subsystem
echo 1 > /sys/kernel/debug/ieee80211/phy0/ath9k/debugdebugfs (/sys/kernel/debug/) provides runtime tuning parameters for many subsystems. Be careful changing these — they can affect system stability.
Common Kernel Issues
Most kernel issues fall into a few categories: memory corruption, lockups, driver bugs, and configuration mistakes. Knowing the symptoms helps you narrow down the root cause.
- Kernel panic on boot — usually initramfs, root device, or driver issue
- Soft lockup — CPU stuck in kernel code for >20 seconds (watchdog)
- Hard lockup — CPU unresponsive, NMI watchdog triggers crash dump
- OOM kills — insufficient memory, check cgroup limits and leaks
- Module load failure — version mismatch, missing symbols, or signing issue
- Corrupted filesystem — run fsck from rescue media
# Check for errors in kernel log
dmesg -l err,crit,alert,emerg | tail -20
# Check for warnings
dmesg -l warn | tail -10
# Check for soft lockups
dmesg | grep -i "lockup"
# Check for OOM kills
dmesg | grep -i "out of memory"
# Check for MCE (Machine Check Exceptions)
dmesg | grep -i "mce"dmesg -l err,crit,alert,emerg filters the kernel log to show only serious errors. Always check these after a reboot to catch boot-time issues.
Install kernel-debuginfo and kernel-debuginfo-common packages. They provide the symbols needed for crash analysis, perf reporting, and meaningful stack traces. Without them, debugging is nearly impossible.
Kernel debug options like KASAN, lockdep, and DEBUG_FS add significant overhead. Only enable them in development or staging environments. Production kernels should be optimized for performance.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.