strace

strace shows you every system call a process makes — every file it opens, every network connection, every memory allocation request. It's the first tool to reach for when a program fails mysteriously, is unexpectedly slow, or you want to understand what it actually does at the OS level.

Basic Usage

# Trace a command from start:
strace ls /tmp
# execve("/usr/bin/ls", ["ls", "/tmp"], envp) = 0
# brk(NULL)                               = 0x55c... (heap start)
# openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
# mmap(NULL, 12345, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f...
# close(3)                                = 0
# openat(AT_FDCWD, "/tmp", O_RDONLY|O_DIRECTORY) = 3
# getdents64(3, /* 5 entries */, 32768)   = 168
# write(1, "file1.txt  file2.log\n", 21)  = 21
# exit_group(0)                           = ?

# Attach to running process:
strace -p PID

# Trace with timestamps:
strace -t ls    # clock time
strace -T ls    # time spent in each syscall

# Count syscalls (summary, not each call):
strace -c ls
# % time  seconds  usecs/call  calls   syscall
# 45.00   0.000450       90    5       read
# 30.00   0.000300       60    5       openat
# ...

Useful Flags

# Follow child processes (fork/exec):
strace -f nginx

# Filter to specific syscalls:
strace -e trace=open,read,write ls

# Filter by category:
strace -e trace=file ls       # all file-related syscalls
strace -e trace=network curl  # network syscalls only
strace -e trace=process bash  # process management

# Output to file (useful for long traces):
strace -o /tmp/trace.txt myapp

# Trace forked children to separate files:
strace -ff -o /tmp/trace myapp
# Creates: /tmp/trace.PID1, /tmp/trace.PID2, etc.

# Combine: full trace with timestamps, children, to file:
strace -ff -tt -o /tmp/trace myapp

Diagnosing Slowness

How do you find which syscall is making a program slow? Use -T to show time spent in each call, combined with -e to filter relevant categories. Sort the output to find the slowest calls. A program spending 2 seconds in read() on a network socket is waiting for data. One spending 2 seconds in stat() is doing excessive filesystem metadata lookups.

# Find slow syscalls — sort by time spent:
strace -T myapp 2>&1 | grep " <" | sort -t'<' -k2 -n | tail -20
# write(1, "...", 1024)   = 1024 <0.000010>
# read(3, "...", 4096)    = 4096 <2.034567>  ← 2 seconds in read!

# Profile just disk I/O:
strace -T -e trace=read,write,pread64,pwrite64 myapp 2>&1 | \
  awk '/ <[0-9]/ {match($0, /<([0-9.]+)>/, a); sum+=a[1]} END {print sum}'

# Find "open" calls that fail (ENOENT = file not found):
strace -e trace=openat myapp 2>&1 | grep ENOENT
# openat(AT_FDCWD, "/etc/myapp.conf", O_RDONLY) = -1 ENOENT (No such file)
# openat(AT_FDCWD, "/usr/local/etc/myapp.conf", O_RDONLY) = -1 ENOENT
# openat(AT_FDCWD, "/home/user/.myapp.conf", O_RDONLY) = 3  ← found it here

strace Overhead Warning

Is strace safe to use on production systems? strace has significant overhead — typically 5-50x slowdown. It works by injecting ptrace into the target process, which intercepts every syscall via a trap. For debugging a slow startup or diagnosing an error, it's fine. For profiling a production service under load, use perf or eBPF instead — much lower overhead. Never attach strace to a high-throughput production process without understanding the impact.