strace
strace shows you every system call a process makes — every file it opens, every network connection, every memory allocation request. It's the first tool to reach for when a program fails mysteriously, is unexpectedly slow, or you want to understand what it actually does at the OS level.
Basic Usage
# Trace a command from start:
strace ls /tmp
# execve("/usr/bin/ls", ["ls", "/tmp"], envp) = 0
# brk(NULL) = 0x55c... (heap start)
# openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
# mmap(NULL, 12345, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f...
# close(3) = 0
# openat(AT_FDCWD, "/tmp", O_RDONLY|O_DIRECTORY) = 3
# getdents64(3, /* 5 entries */, 32768) = 168
# write(1, "file1.txt file2.log\n", 21) = 21
# exit_group(0) = ?
# Attach to running process:
strace -p PID
# Trace with timestamps:
strace -t ls # clock time
strace -T ls # time spent in each syscall
# Count syscalls (summary, not each call):
strace -c ls
# % time seconds usecs/call calls syscall
# 45.00 0.000450 90 5 read
# 30.00 0.000300 60 5 openat
# ...
Useful Flags
# Follow child processes (fork/exec):
strace -f nginx
# Filter to specific syscalls:
strace -e trace=open,read,write ls
# Filter by category:
strace -e trace=file ls # all file-related syscalls
strace -e trace=network curl # network syscalls only
strace -e trace=process bash # process management
# Output to file (useful for long traces):
strace -o /tmp/trace.txt myapp
# Trace forked children to separate files:
strace -ff -o /tmp/trace myapp
# Creates: /tmp/trace.PID1, /tmp/trace.PID2, etc.
# Combine: full trace with timestamps, children, to file:
strace -ff -tt -o /tmp/trace myapp
Diagnosing Slowness
How do you find which syscall is making a program slow?
Use -T to show time spent in each call, combined with -e to filter relevant categories. Sort the output to find the slowest calls. A program spending 2 seconds in read() on a network socket is waiting for data. One spending 2 seconds in stat() is doing excessive filesystem metadata lookups.
# Find slow syscalls — sort by time spent:
strace -T myapp 2>&1 | grep " <" | sort -t'<' -k2 -n | tail -20
# write(1, "...", 1024) = 1024 <0.000010>
# read(3, "...", 4096) = 4096 <2.034567> ← 2 seconds in read!
# Profile just disk I/O:
strace -T -e trace=read,write,pread64,pwrite64 myapp 2>&1 | \
awk '/ <[0-9]/ {match($0, /<([0-9.]+)>/, a); sum+=a[1]} END {print sum}'
# Find "open" calls that fail (ENOENT = file not found):
strace -e trace=openat myapp 2>&1 | grep ENOENT
# openat(AT_FDCWD, "/etc/myapp.conf", O_RDONLY) = -1 ENOENT (No such file)
# openat(AT_FDCWD, "/usr/local/etc/myapp.conf", O_RDONLY) = -1 ENOENT
# openat(AT_FDCWD, "/home/user/.myapp.conf", O_RDONLY) = 3 ← found it here
strace Overhead Warning
Is strace safe to use on production systems?
strace has significant overhead — typically 5-50x slowdown. It works by injecting ptrace into the target process, which intercepts every syscall via a trap. For debugging a slow startup or diagnosing an error, it's fine. For profiling a production service under load, use perf or eBPF instead — much lower overhead. Never attach strace to a high-throughput production process without understanding the impact.
Frequently Asked Questions
What will I learn here?
This page covers the core concepts and techniques you need to understand the topic and progress confidently to the next lesson.
How should I use this page?
Start with the overview, then follow the section links to deepen your understanding. Use the table of contents on the right to jump to specific sections.
What should I read next?
Use the navigation below to continue to the next lesson or explore related topics.