OOM Killer Selection
When the OOM killer activates, it doesn't pick randomly. It calculates a "badness score" for every process and kills the one most likely to free the most memory with the least collateral damage.
The OOM Score — 0 to 1000
How does the kernel score processes?
The base score is proportional to RAM usage as a percentage of total RAM: a process using 100% of RAM gets score 1000. Scores are then adjusted by oom_score_adj. Higher score = more likely to be killed.
# See OOM score for any process
cat /proc/PID/oom_score
cat /proc/self/oom_score # your current shell
# Check scores for all processes
for pid in /proc/[0-9]*/oom_score; do
score=$(cat "$pid" 2>/dev/null)
proc=$(cat "${pid%/oom_score}/comm" 2>/dev/null)
echo "$score $proc"
done | sort -rn | head -10
oom_score_adj — Your Control Knob
You can bias the OOM killer's decision by adjusting oom_score_adj for each process. Range: -1000 to +1000.
| Value | Effect | Use case |
|---|---|---|
| -1000 | Never kill this process | systemd (PID 1), sshd — killing these would be catastrophic |
| -500 to -100 | Very unlikely to kill | Your database, critical services |
| 0 | Default — no adjustment | Regular processes |
| +500 to +999 | Very likely to kill | Disposable workers, test processes |
# Set for a running process
echo -500 > /proc/PID/oom_score_adj # protect it
echo 900 > /proc/PID/oom_score_adj # sacrifice it
# Set in systemd service (persistent)
# In /etc/systemd/system/mydb.service:
[Service]
OOMScoreAdjust=-500 # protect database from OOM killer
# After editing:
sudo systemctl daemon-reload
sudo systemctl restart mydb
Reading OOM Kill Events
# Find OOM kills in kernel log
dmesg -T | grep -A 20 "Out of memory"
# Sample output:
# [Sun Jan 15 03:12:44 2024] Out of memory: Kill process 8743 (java) score 872 or sacrifice child
# [Sun Jan 15 03:12:44 2024] Killed process 8743 (java) total-vm:8428756kB, rss:7284256kB, pgtables:16400kB
# In journald
journalctl -k | grep -i "killed process"
The OOM killer killed the wrong process. How do I fix this?
Lower the oom_score_adj of the process you want to protect. Raise it for processes you don't mind losing. Set OOMScoreAdjust in the systemd unit file for persistent configuration. Also consider setting proper memory limits with cgroup memory.max so individual containers/services can't exhaust all RAM.
Frequently Asked Questions
What will I learn here?
This page covers the core concepts and techniques you need to understand the topic and progress confidently to the next lesson.
How should I use this page?
Start with the overview, then follow the section links to deepen your understanding. Use the table of contents on the right to jump to specific sections.
What should I read next?
Use the navigation below to continue to the next lesson or explore related topics.