ftrace
ftrace is the kernel's built-in tracing framework, accessible through /sys/kernel/debug/tracing. No extra tools needed — you configure tracing by writing to files and read results from other files. It can trace any kernel function, measure latency, record function call graphs, and identify scheduling or interrupt issues.
ftrace Setup
# Mount debugfs (usually auto-mounted):
mount -t debugfs none /sys/kernel/debug
# ftrace lives here:
ls /sys/kernel/debug/tracing/
# available_tracers current_tracer trace
# available_events events/ trace_pipe
# tracing_on set_ftrace_filter set_graph_function
# trace_options per_cpu/
# Available tracers:
cat /sys/kernel/debug/tracing/available_tracers
# blk function function_graph hwlat irqsoff nop preemptoff preemptirqsoff wakeup
# Enable/disable tracing:
echo 1 > /sys/kernel/debug/tracing/tracing_on
echo 0 > /sys/kernel/debug/tracing/tracing_on
# Clear the trace buffer:
echo > /sys/kernel/debug/tracing/trace
Function Tracer
# Trace all kernel function calls (very verbose!):
echo function > /sys/kernel/debug/tracing/current_tracer
echo 1 > /sys/kernel/debug/tracing/tracing_on
sleep 1
echo 0 > /sys/kernel/debug/tracing/tracing_on
cat /sys/kernel/debug/tracing/trace | head -30
# tracer: function
# CPU TASK/PID TIMESTAMP FUNCTION
# 1 kworker-45 12345.678 kthread_worker_fn
# 0 nginx-1234 12345.679 vfs_read
# 0 nginx-1234 12345.680 tcp_sendmsg
# Filter to specific function:
echo vfs_read > /sys/kernel/debug/tracing/set_ftrace_filter
echo function > /sys/kernel/debug/tracing/current_tracer
# Filter to functions matching a pattern:
echo '*tcp*' > /sys/kernel/debug/tracing/set_ftrace_filter
# Filter to a specific process (PID):
echo 1234 > /sys/kernel/debug/tracing/set_ftrace_pid
# Remove filter:
echo > /sys/kernel/debug/tracing/set_ftrace_filter
function_graph — Call Graph Tracing
What makes function_graph tracer special?
function_graph shows not just which functions were called, but their complete call graph with nesting and duration. You can see that tcp_sendmsg called ip_queue_xmit which called dev_queue_xmit, and each took how long. This makes it easy to understand the call chain and find where time was spent inside the kernel.
# Use function_graph tracer:
echo function_graph > /sys/kernel/debug/tracing/current_tracer
echo vfs_write > /sys/kernel/debug/tracing/set_graph_function
echo 1 > /sys/kernel/debug/tracing/tracing_on
# ... run your workload ...
echo 0 > /sys/kernel/debug/tracing/tracing_on
cat /sys/kernel/debug/tracing/trace
# tracer: function_graph
# CPU DURATION FUNCTION CALLS
# 0 | | vfs_write() {
# 0 | 0.543 us | rw_verify_area();
# 0 | | ext4_file_write_iter() {
# 0 | | generic_write_checks() {
# 0 | 0.123 us | iov_iter_count();
# 0 | 0.234 us | }
# 0 | 12.34 us | ext4_buffered_write_iter();
# 0 | 13.45 us | }
# 0 | 14.23 us | }
trace-cmd — Easier ftrace Interface
# trace-cmd wraps ftrace in a friendlier interface:
apt install trace-cmd
# Record for 5 seconds:
trace-cmd record -p function_graph -g vfs_write sleep 5
# Report:
trace-cmd report | head -50
# Record specific events:
trace-cmd record -e sched:sched_switch -e irq:irq_handler_entry sleep 3
trace-cmd report | grep irq
# List available events:
trace-cmd list -e | grep tcp
# tcp:tcp_probe
# tcp:tcp_retransmit_skb
# Trace scheduling latency:
trace-cmd record -p preemptoff sleep 1
trace-cmd report | head -20
Latency Tracers
# irqsoff: find what disables interrupts the longest
echo irqsoff > /sys/kernel/debug/tracing/current_tracer
echo 1 > /sys/kernel/debug/tracing/tracing_on
sleep 5
echo 0 > /sys/kernel/debug/tracing/tracing_on
cat /sys/kernel/debug/tracing/trace
# Shows the longest IRQ-disabled section with full call trace
# preemptoff: find what disables preemption the longest
echo preemptoff > /sys/kernel/debug/tracing/current_tracer
# wakeup: measure scheduler wakeup latency (RT task wakeup-to-run)
echo wakeup_rt > /sys/kernel/debug/tracing/current_tracer
# Useful for real-time latency analysis
Frequently Asked Questions
What will I learn here?
This page covers the core concepts and techniques you need to understand the topic and progress confidently to the next lesson.
How should I use this page?
Start with the overview, then follow the section links to deepen your understanding. Use the table of contents on the right to jump to specific sections.
What should I read next?
Use the navigation below to continue to the next lesson or explore related topics.