Tracepoints & Probes
Linux has three ways to attach instrumentation to running code: static tracepoints (pre-planned hooks in kernel source), dynamic kprobes (attach to any kernel instruction at runtime), and uprobes (attach to any userspace function). Together they give you complete observability of any code path, kernel or userspace, with minimal overhead.
Static Tracepoints
What are static tracepoints and why use them over kprobes?
Static tracepoints are pre-placed hooks in the kernel source — intentionally inserted by kernel developers at meaningful events like "process scheduled", "packet received", "block I/O issued". They're stable across kernel versions (part of the ABI), well-documented, and have lower overhead than kprobes because no instruction modification is needed. Use tracepoints when available; use kprobes when no tracepoint exists.
# List available tracepoints:
ls /sys/kernel/debug/tracing/events/
# block/ kmem/ net/ sched/ syscalls/ tcp/ ...
# See events in a category:
ls /sys/kernel/debug/tracing/events/sched/
# sched_switch sched_wakeup sched_process_exec sched_process_exit
# Enable a tracepoint:
echo 1 > /sys/kernel/debug/tracing/events/sched/sched_switch/enable
# View output:
cat /sys/kernel/debug/tracing/trace
# nginx-1234 [001] 12345.678: sched_switch:
# prev_comm=nginx prev_pid=1234 prev_prio=120 prev_state=S
# next_comm=kworker next_pid=5 next_prio=120
# Disable:
echo 0 > /sys/kernel/debug/tracing/events/sched/sched_switch/enable
# With perf:
perf stat -e sched:sched_switch sleep 5
perf record -e tcp:tcp_retransmit_skb -ag sleep 10
kprobes — Dynamic Kernel Instrumentation
How do kprobes attach to arbitrary kernel functions?
kprobes work by replacing the target instruction with an INT3 (breakpoint) instruction at runtime. When the CPU hits the breakpoint, it calls your probe handler. After the handler runs, execution continues normally. This works on any instruction in the kernel without recompilation. The overhead is roughly one function call per hit.
# Create a kprobe via /sys/kernel/debug/kprobes/:
echo 'p:myprobe tcp_sendmsg' > /sys/kernel/debug/tracing/kprobe_events
# "p" = probe (at function entry), "myprobe" = name, tcp_sendmsg = function
# Enable it:
echo 1 > /sys/kernel/debug/tracing/events/kprobes/myprobe/enable
# View hits:
cat /sys/kernel/debug/tracing/trace | grep myprobe
# Return probe (fires on function exit, can capture return value):
echo 'r:myretprobe tcp_sendmsg $retval' >> /sys/kernel/debug/tracing/kprobe_events
# Remove probes:
echo > /sys/kernel/debug/tracing/kprobe_events
# With perf (easier):
perf probe --add tcp_sendmsg
perf record -e probe:tcp_sendmsg -ag sleep 5
perf report
# List available kernel symbols for probing:
perf probe --list
cat /proc/kallsyms | grep tcp_send
uprobes — Userspace Function Probing
# uprobes attach to userspace functions (any binary or library)
# Find function offset in a binary:
objdump -d /usr/lib/postgresql/14/bin/postgres | grep '<ExecHashJoin>'
# or: nm -D /lib/x86_64-linux-gnu/libssl.so | grep SSL_write
# Add a uprobe (probe on ssl_write entry in OpenSSL):
perf probe -x /lib/x86_64-linux-gnu/libssl.so.3 SSL_write
# Record hits:
perf record -e probe_libssl:SSL_write -ag sleep 10
perf report
# bpftrace with uprobes (much easier):
bpftrace -e 'uprobe:/lib/x86_64-linux-gnu/libssl.so.3:SSL_write {
printf("PID %d wrote %d bytes via SSL\n", pid, arg2);
}'
# Fires every time any process calls SSL_write!
# Trace Python function calls (uprobes on Python interpreter):
bpftrace -e 'uprobe:/usr/bin/python3:PyObject_Call {
printf("%s called a function\n", comm);
}'
Choosing Between Tracepoints, kprobes, and uprobes
| Static Tracepoints | kprobes | uprobes | |
|---|---|---|---|
| Target | Kernel only | Kernel only | Userspace only |
| Stability | Stable ABI | Unstable (function names change) | Depends on binary |
| Overhead | Lowest | Low (one breakpoint) | Low |
| Availability | Only predefined events | Any kernel function | Any userspace function |
| Use when | Event exists as tracepoint | No tracepoint, kernel target | App/library instrumentation |
Frequently Asked Questions
What will I learn here?
This page covers the core concepts and techniques you need to understand the topic and progress confidently to the next lesson.
How should I use this page?
Start with the overview, then follow the section links to deepen your understanding. Use the table of contents on the right to jump to specific sections.
What should I read next?
Use the navigation below to continue to the next lesson or explore related topics.