Printk Should Be Your Last Resort
summary
Argues that printk is a last resort and shows concrete replacements: uses ftrace function_graph to map TX path and finds a 3s mailbox timeout caused by bottom halves being disabled on a uniprocessor, uses tracepoints to follow mailbox stages across PF/AF, uses fprobe to catch a page order 17 vs 13 mismatch traced to 4KB vs 64KB pages, and builds a working eBPF-based memory-leak detector (kernel eBPF + userspace C controller) that quickly finds 10 leaked blocks and one single-block leak in the dummy netdev driver.
AI-written summary · about
chapters (15)
- 0:00printk limitations
- 1:03Tracing vs printk
- 3:56Trace points in kernel
- 6:42Trace points example
- 9:09Function probes (fprobes)
- 10:12Memory allocation with fprobes
- 13:44eBPF for kernel tracing
- 15:12Memory leak detector with eBPF
- 22:11BPFtrace for tracing
- 24:05Q&A on eBPF leaks
- 25:11Timing issues with tracing
- 26:37eBPF for subsystems
- 28:45Q&A on trace points
- 30:51kprobes for debugging
- 33:45Summary
related talks
31mFantastic tracepoints and where to find them
52mLinux Kernel Debugging: Going Beyond Printk Messages
32mIs Your Kernel Being Honest? Understanding & measuring low level bottlenecks
55mLinux Kernel Debug And Profiling Tools
38mThe Next Linux Superpower: eBPF Primer
49mDebugging Methodologies for Realtime Issues in Linux Systems
46mbpftrace internals
35mLinux Tracing updates