dmesg & Ring Buffer
When something goes wrong at the hardware or kernel level — a disk error, an OOM kill, a driver crash — the kernel writes a message to its ring buffer. dmesg reads that buffer. It's the first place to look when diagnosing hardware issues, kernel panics, driver failures, and mysterious system instability.
The Kernel Ring Buffer
What is a ring buffer and why does the kernel use it?
A ring buffer is a fixed-size circular buffer — when it fills up, new messages overwrite the oldest. The kernel uses a ring buffer for log messages because it can't use dynamic memory allocation during early boot, interrupt handlers, or panic situations. Size is configurable (typical default: 512KB–1MB). On busy systems, old messages can be overwritten — increase the buffer if needed.
# Read kernel messages:
dmesg
# Human-readable timestamps (requires accurate clock):
dmesg -T
# Follow new messages in real-time:
dmesg -w # like tail -f for the ring buffer
# Filter by log level:
dmesg -l err # errors only
dmesg -l warn,err # warnings and errors
dmesg -l emerg,alert,crit,err # all serious
# Log levels (0=emergency to 7=debug):
# 0 emerg - system unusable
# 1 alert - immediate action needed
# 2 crit - critical conditions
# 3 err - error conditions
# 4 warn - warning conditions
# 5 notice - normal but significant
# 6 info - informational
# 7 debug - debug messages
Finding Common Issues in dmesg
# OOM (Out of Memory) kill:
dmesg | grep -i "oom\|killed\|out of memory"
# [12345.678] Out of memory: Kill process 1234 (nginx) score 500 or sacrifice child
# [12345.679] Killed process 1234 (nginx) total-vm:2048MB, anon-rss:512MB, file-rss:128MB
# Disk errors (hardware failure warning!):
dmesg | grep -i "error\|failed\|ata\|scsi\|blk"
# [ 123.456] ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
# [ 123.457] ata1.00: failed command: READ DMA EXT
# [ 123.458] ata1: hard resetting link
# THESE INDICATE DISK FAILURE — check SMART data: smartctl -a /dev/sda
# Network card issues:
dmesg | grep -i "eth\|ens\|enp\|link"
# [ 234.567] eth0: renamed from vethABC
# [ 345.678] eth0: NIC Link is Down
# Memory errors (bad RAM):
dmesg | grep -i "mce\|machine check\|memory error"
# MCE = Machine Check Exception = hardware error from CPU
# Driver failures:
dmesg | grep -i "firmware\|module\|failed to load"
Understanding Timestamps
# Default dmesg timestamps are seconds since boot:
dmesg | head -5
# [ 0.000000] Booting Linux on physical CPU 0x0
# [ 0.000000] Linux version 6.1.0-21-amd64 ...
# [ 0.356789] ACPI: Core revision 20221020
# [ 12.345678] eth0: renamed from veth1234abc
# Convert to human-readable time:
dmesg -T | head -5
# [Mon Jan 15 10:30:00 2024] Booting Linux...
# Calculate when event happened:
# Event at [12345.678] seconds since boot
# Boot time from: date -d "$(uptime -s)" +%s (Unix timestamp)
# Event time = boot_timestamp + 12345 seconds
# Recent messages only (last hour):
dmesg --since "1 hour ago" # (requires util-linux 2.33+)
# Save ring buffer to file (snapshot):
dmesg > /tmp/dmesg-$(date +%Y%m%d-%H%M%S).txt
Persistent Kernel Logs — Beyond the Ring Buffer
# Ring buffer is lost on reboot!
# journald can persist kernel messages:
journalctl -k # kernel messages from current boot
journalctl -k -b -1 # kernel messages from previous boot
journalctl -k -b -2 # two boots ago
# This is how you find OOM kills that caused a reboot:
journalctl -k -b -1 | grep -i "oom\|killed\|panic"
# /var/log/kern.log (traditional syslog):
grep "oom\|error" /var/log/kern.log
# Configure ring buffer size (default varies, usually 512KB):
# In /etc/default/grub:
# GRUB_CMDLINE_LINUX="log_buf_len=16M"
# Rebuild grub: update-grub
# Or at runtime (root only):
dmesg -C # clear the ring buffer (be careful!)
# Check ring buffer size and current usage:
dmesg --buffer-size # or check CONFIG_LOG_BUF_SHIFT in kernel config
Frequently Asked Questions
What will I learn here?
This page covers the core concepts and techniques you need to understand the topic and progress confidently to the next lesson.
How should I use this page?
Start with the overview, then follow the section links to deepen your understanding. Use the table of contents on the right to jump to specific sections.
What should I read next?
Use the navigation below to continue to the next lesson or explore related topics.