Catching inode exhaustion with Ruby: when df says the disk isn’t full but writes still fail

the shed // linux / disk

df -h says the disk is 30% full, but writes keep failing with “No space left on device.” The filesystem ran out of inodes, not bytes. Here’s a pure-Ruby monitor that catches inode exhaustion before it takes the box down.

the shed // Ruby-for-DevOps series · published Aug 23, 2026

Step through the build below — the problem, the full script, the design decisions, and the real test output captured in the sandbox:

inode_usage_monitor.rb

A box that creates millions of tiny files — a mail queue, PHP sessions, a cache with deep fan-out — can exhaust its inode table while df -h still shows plenty of free bytes. Every new write then fails with ENOSPC and nothing in the obvious place explains why. This reads df -iP, ranks filesystems by inode pressure, and hunts the directories hoarding files.

#!/usr/bin/env ruby
# frozen_string_literal: true
#
# inode_usage_monitor.rb -- inode-exhaustion monitor for Linux.
#
# A filesystem can hit "No space left on device" while df -h shows plenty of
# free bytes -- because it has run out of *inodes*, not space. That happens on
# boxes that create millions of tiny files (mail queues, session dirs, cache
# fan-out). This script reports inode usage per filesystem, alerts on any that
# cross a threshold, and can hunt the directories holding the most files.
#
#   ruby inode_usage_monitor.rb                     # per-filesystem inode table
#   ruby inode_usage_monitor.rb --warn 80 --crit 95 # custom thresholds (percent)
#   ruby inode_usage_monitor.rb --hunt /var         # find inode-hog dirs under /var
#   ruby inode_usage_monitor.rb --json              # machine-readable
#
# Stdlib only: open3, find, json, optparse. No gems. Exit codes: 0 ok,
# 1 warning, 2 critical -- drops straight into cron / Nagios.
require 'open3'
require 'find'
require 'json'
require 'optparse'
options = { warn: 80, crit: 90, hunt: nil, top: 12, json: false }
OptionParser.new do |o|
  o.banner = 'Usage: ruby inode_usage_monitor.rb [options]'
  o.on('--warn PCT', Integer, 'warn threshold %% inodes used (default 80)') { |v| options[:warn] = v }
  o.on('--crit PCT', Integer, 'crit threshold %% inodes used (default 90)') { |v| options[:crit] = v }
  o.on('--hunt DIR', 'find the directories with the most files under DIR') { |v| options[:hunt] = v }
  o.on('--top N', Integer, 'rows for --hunt (default 12)') { |v| options[:top] = v }
  o.on('--json', 'JSON output') { options[:json] = true }
end.parse!
# --- read per-filesystem inode stats from `df -iP` -------------------------
# -P = POSIX output (one line per fs, stable columns); -i = inodes not blocks.
# Columns: Filesystem Inodes IUsed IFree IUse% Mounted-on
def read_inode_table
  out, err, st = Open3.capture3('df', '-iP')
  raise "df failed: #{err}" unless st.success?
  rows = []
  out.each_line.drop(1).each do |line|
    f = line.split
    next if f.size < 6
    inodes, iused, ifree = f[1].to_i, f[2].to_i, f[3].to_i
    next if inodes.zero?                     # pseudo-fs (tmpfs, proc) report 0
    pct = (iused * 100.0 / inodes).round(1)
    rows << { 'filesystem' => f[0], 'inodes' => inodes, 'iused' => iused,
              'ifree' => ifree, 'pct' => pct, 'mount' => f[5] }
  end
  rows.sort_by { |r| -r['pct'] }
end
# --- optional: hunt the directories that hold the most files ---------------
# One Find pass; every file/dir bumps the counter of its PARENT directory.
def hunt_dirs(root, top)
  counts = Hash.new(0)
  Find.find(root) do |path|
    counts[File.dirname(path)] += 1
  rescue Errno::EACCES, Errno::ENOENT
    next
  end
  counts.sort_by { |_, c| -c }.first(top)
end
rows = read_inode_table
crit = rows.select { |r| r['pct'] >= options[:crit] }
warn = rows.select { |r| r['pct'] >= options[:warn] && r['pct'] < options[:crit] }
hunt = options[:hunt] ? hunt_dirs(File.expand_path(options[:hunt]), options[:top]) : nil
if options[:json]
  puts JSON.pretty_generate('filesystems' => rows,
                            'warn_threshold' => options[:warn],
                            'crit_threshold' => options[:crit],
                            'alerts' => (crit + warn).map { |r| r['mount'] },
                            'hunt' => hunt&.map { |d, c| { 'dir' => d, 'files' => c } })
else
  puts format('%-6s %-22s %12s %12s %7s  %s', 'STATE', 'FILESYSTEM', 'IUSED', 'IFREE', 'IUSE%', 'MOUNT')
  rows.each do |r|
    state = r['pct'] >= options[:crit] ? 'CRIT' : r['pct'] >= options[:warn] ? 'WARN' : 'ok'
    puts format('%-6s %-22s %12d %12d %6.1f%%  %s', state, r['filesystem'], r['iused'], r['ifree'], r['pct'], r['mount'])
  end
  if hunt
    puts
    puts format('%12s  %s', 'FILES', "INODE-HOG DIRECTORIES under #{options[:hunt]}")
    hunt.each { |d, c| puts format('%12d  %s', c, d) }
  end
  puts
  puts "#{crit.size} critical, #{warn.size} warning"
end
exit(crit.any? ? 2 : warn.any? ? 1 : 0)

df -iP gives stable per-filesystem inode columns; zero-inode pseudo-filesystems are skipped. Each filesystem is graded against --warn/--crit and sorted worst-first. --hunt does one Find pass counting files per parent directory, so you see exactly which directory is the inode hog. Exits 2 on CRIT — cron/Nagios-ready.

STATE  FILESYSTEM                    IUSED        IFREE   IUSE%  MOUNT
WARN   /dev/sdc                      67018       588342   10.2%  /sessions
WARN   /dev/sda1                    121917      1168323    9.4%  /
ok     tmpfs                           510       500666    0.1%  /run
ok     tmpfs                            15       501161    0.0%  /dev
ok     tmpfs                             2       501174    0.0%  /run/lock
ok     tmpfs                             1       501175    0.0%  /run/dbus
ok     tmpfs                             1       501175    0.0%  /etc/ssh/ssh_config.d
ok     /proc/self/fd/3             -999001      1000000 -100000.1%  /sessions/awesome-fervent-johnson/mnt/outputs
ok     /proc/self/fd/3             -999001      1000000 -100000.1%  /sessions/awesome-fervent-johnson/mnt/uploads
ok     /proc/self/fd/3             -999001      1000000 -100000.1%  /sessions/awesome-fervent-johnson/mnt/.remote-plugins/plugin_01BYoFffWxgV8R5TZ6fEVCih
ok     /proc/self/fd/3             -999001      1000000 -100000.1%  /sessions/awesome-fervent-johnson/mnt/.remote-plugins/plugin_01SfWJSiw6JtGbsuW75PKvPK
ok     /proc/self/fd/3             -999001      1000000 -100000.1%  /sessions/awesome-fervent-johnson/mnt/.remote-plugins/plugin_01MqbKF8FHaDvXSG4JjrPkKD
ok     /proc/self/fd/3             -999001      1000000 -100000.1%  /sessions/awesome-fervent-johnson/mnt/.remote-plugins/plugin_01VyNDLNYUZHHyKf7A691D7V
ok     /proc/self/fd/3             -999001      1000000 -100000.1%  /sessions/awesome-fervent-johnson/mnt/.remote-plugins/plugin_01Eeb9y5m4iFuY3yRtytYfdc
ok     /proc/self/fd/3             -999001      1000000 -100000.1%  /sessions/awesome-fervent-johnson/mnt/.claude/skills
ok     /proc/self/fd/3             -999001      1000000 -100000.1%  /sessions/awesome-fervent-johnson/mnt/.remote-plugins/plugin_012ABz1xjgtJYWKrcJkXW6ad
ok     /proc/self/fd/3             -999001      1000000 -100000.1%  /sessions/awesome-fervent-johnson/mnt/.remote-plugins/plugin_014WxCYbLf7f3uw2isHFR9US
ok     /proc/self/fd/3             -999001      1000000 -100000.1%  /sessions/awesome-fervent-johnson/mnt/.claude/projects
ok     /proc/self/fd/3             -999001      1000000 -100000.1%  /sessions/awesome-fervent-johnson/mnt/.remote-plugins/plugin_01KmRfL8EXGF3PeqMRzef1TR
ok     /proc/self/fd/3             -999001      1000000 -100000.1%  /sessions/awesome-fervent-johnson/mnt/.remote-plugins/plugin_01XXJmxLXPEhPMmnxmrgntNw
ok     /proc/self/fd/3             -999001      1000000 -100000.1%  /sessions/awesome-fervent-johnson/mnt/.remote-plugins/plugin_017zncz89kmhdPgdpZQZm5Dj
ok     /proc/self/fd/3             -999001      1000000 -100000.1%  /sessions/awesome-fervent-johnson/mnt/.remote-plugins/plugin_019TBdWa5NQJJuDFmEc4k6BJ
ok     /proc/self/fd/3             -999001      1000000 -100000.1%  /sessions/awesome-fervent-johnson/mnt/.remote-plugins/plugin_01AYHYqVLaZRH2Vi6aHgDwEw
ok     /proc/self/fd/3             -999001      1000000 -100000.1%  /sessions/awesome-fervent-johnson/mnt/.remote-plugins/plugin_01VTbvGZYaCVU2CNSvhDCnkg
ok     /proc/self/fd/3             -999001      1000000 -100000.1%  /sessions/awesome-fervent-johnson/mnt/.remote-plugins/plugin_0155zZVATbJU3jHUmPP9NvMC
ok     /proc/self/fd/3             -999001      1000000 -100000.1%  /sessions/awesome-fervent-johnson/mnt/.remote-plugins/plugin_011v5h6QUzBZvas64y44XLhy
ok     /proc/self/fd/3             -999001      1000000 -100000.1%  /sessions/awesome-fervent-johnson/mnt/.remote-plugins/plugin_01FTLa86dhbVJ3HB1LdHdhN7
0 critical, 2 warning
...
       FILES  INODE-HOG DIRECTORIES under /tmp/inodetest
          30  /tmp/inodetest/a
           5  /tmp/inodetest/b
           2  /tmp/inodetest
0 critical, 0 warning

Get the code

Full script + README on GitHub: ruby-devops-toolkit/inode-usage-monitor

df -iP → threshold check → optional inode-hog hunt → exit codes.

prerequisites

What you need

requirements

Ruby 2.7+ (tested on 3.0.2) — stdlib only: open3, find, json, optparse. No gems.
Linux (uses df -iP; also works on macOS with df -i).

the whole script

inode_usage_monitor.rb

The complete, commented script. The walkthrough below explains the ideas behind it.

inode_usage_monitor.rbruby

#!/usr/bin/env ruby
# frozen_string_literal: true
#
# inode_usage_monitor.rb -- inode-exhaustion monitor for Linux.
#
# A filesystem can hit "No space left on device" while df -h shows plenty of
# free bytes -- because it has run out of *inodes*, not space. That happens on
# boxes that create millions of tiny files (mail queues, session dirs, cache
# fan-out). This script reports inode usage per filesystem, alerts on any that
# cross a threshold, and can hunt the directories holding the most files.
#
#   ruby inode_usage_monitor.rb                     # per-filesystem inode table
#   ruby inode_usage_monitor.rb --warn 80 --crit 95 # custom thresholds (percent)
#   ruby inode_usage_monitor.rb --hunt /var         # find inode-hog dirs under /var
#   ruby inode_usage_monitor.rb --json              # machine-readable
#
# Stdlib only: open3, find, json, optparse. No gems. Exit codes: 0 ok,
# 1 warning, 2 critical -- drops straight into cron / Nagios.
require 'open3'
require 'find'
require 'json'
require 'optparse'
options = { warn: 80, crit: 90, hunt: nil, top: 12, json: false }
OptionParser.new do |o|
  o.banner = 'Usage: ruby inode_usage_monitor.rb [options]'
  o.on('--warn PCT', Integer, 'warn threshold %% inodes used (default 80)') { |v| options[:warn] = v }
  o.on('--crit PCT', Integer, 'crit threshold %% inodes used (default 90)') { |v| options[:crit] = v }
  o.on('--hunt DIR', 'find the directories with the most files under DIR') { |v| options[:hunt] = v }
  o.on('--top N', Integer, 'rows for --hunt (default 12)') { |v| options[:top] = v }
  o.on('--json', 'JSON output') { options[:json] = true }
end.parse!
# --- read per-filesystem inode stats from `df -iP` -------------------------
# -P = POSIX output (one line per fs, stable columns); -i = inodes not blocks.
# Columns: Filesystem Inodes IUsed IFree IUse% Mounted-on
def read_inode_table
  out, err, st = Open3.capture3('df', '-iP')
  raise "df failed: #{err}" unless st.success?
  rows = []
  out.each_line.drop(1).each do |line|
    f = line.split
    next if f.size < 6
    inodes, iused, ifree = f[1].to_i, f[2].to_i, f[3].to_i
    next if inodes.zero?                     # pseudo-fs (tmpfs, proc) report 0
    pct = (iused * 100.0 / inodes).round(1)
    rows << { 'filesystem' => f[0], 'inodes' => inodes, 'iused' => iused,
              'ifree' => ifree, 'pct' => pct, 'mount' => f[5] }
  end
  rows.sort_by { |r| -r['pct'] }
end
# --- optional: hunt the directories that hold the most files ---------------
# One Find pass; every file/dir bumps the counter of its PARENT directory.
def hunt_dirs(root, top)
  counts = Hash.new(0)
  Find.find(root) do |path|
    counts[File.dirname(path)] += 1
  rescue Errno::EACCES, Errno::ENOENT
    next
  end
  counts.sort_by { |_, c| -c }.first(top)
end
rows = read_inode_table
crit = rows.select { |r| r['pct'] >= options[:crit] }
warn = rows.select { |r| r['pct'] >= options[:warn] && r['pct'] < options[:crit] }
hunt = options[:hunt] ? hunt_dirs(File.expand_path(options[:hunt]), options[:top]) : nil
if options[:json]
  puts JSON.pretty_generate('filesystems' => rows,
                            'warn_threshold' => options[:warn],
                            'crit_threshold' => options[:crit],
                            'alerts' => (crit + warn).map { |r| r['mount'] },
                            'hunt' => hunt&.map { |d, c| { 'dir' => d, 'files' => c } })
else
  puts format('%-6s %-22s %12s %12s %7s  %s', 'STATE', 'FILESYSTEM', 'IUSED', 'IFREE', 'IUSE%', 'MOUNT')
  rows.each do |r|
    state = r['pct'] >= options[:crit] ? 'CRIT' : r['pct'] >= options[:warn] ? 'WARN' : 'ok'
    puts format('%-6s %-22s %12d %12d %6.1f%%  %s', state, r['filesystem'], r['iused'], r['ifree'], r['pct'], r['mount'])
  end
  if hunt
    puts
    puts format('%12s  %s', 'FILES', "INODE-HOG DIRECTORIES under #{options[:hunt]}")
    hunt.each { |d, c| puts format('%12d  %s', c, d) }
  end
  puts
  puts "#{crit.size} critical, #{warn.size} warning"
end
exit(crit.any? ? 2 : warn.any? ? 1 : 0)

walkthrough

How it works, step by step

Read `df -iP`, skip the pseudo-filesystems

The -P flag forces POSIX one-line-per-filesystem output with stable columns, and -i asks for inodes instead of blocks. Virtual filesystems like proc and tmpfs report zero inodes, so they’re skipped — you only see real filesystems that can actually run out.

Thresholds and the inode hunt

Each filesystem’s IUse% is compared to --warn/--crit and the table is sorted worst-first. When a filesystem is in trouble, --hunt DIR does one Find pass and increments the file count of each entry’s parent directory — surfacing the exact directories (mail spools, session dirs, cache fan-out) responsible for eating inodes.

df -iP

one command

gems required

2/1/0

exit codes

troubleshooting

When the numbers look off

common gotchas

All filesystems show 0% / missing. Your df lacks -i; install coreutils or run on the host rather than a container layer.
--hunt is slow on huge trees. It visits every inode by design; scope it to the offending mount, not /.
Numbers differ from du. This counts inodes (files + directories), not bytes.

extending

Where to take it next

Emit to a Prometheus textfile collector to graph inode headroom over time; add --exclude globs to keep --hunt off network mounts; or cross-check against df -hP so one report shows both byte and inode pressure.

Sources: GitHub: ruby-devops-toolkit/inode-usage-monitor · df(1) — GNU coreutils · Ruby Open3

Catching inode exhaustion with Ruby: when df says the disk isn’t full but writes still fail

What you need

inode_usage_monitor.rb

How it works, step by step

Read `df -iP`, skip the pseudo-filesystems

Thresholds and the inode hunt

When the numbers look off

Where to take it next

Educator

Ollama Tutorial: Run Local AI for DevOps Security

Finding duplicate files with Ruby: reclaim disk without reading every byte

What you need

inode_usage_monitor.rb

How it works, step by step

Read df -iP, skip the pseudo-filesystems

Thresholds and the inode hunt

When the numbers look off

Where to take it next

Educator

Related Posts

3AM Paged: Replaying an AI-Assisted Incident, Then Letting You Pick Ruby or Python With Real Numbers

Auditing your PATH with Ruby: catch the writable directory that hijacks every command

Finding duplicate files with Ruby: reclaim disk without reading every byte

Ollama Tutorial: Run Local AI for DevOps Security

Finding duplicate files with Ruby: reclaim disk without reading every byte

Read `df -iP`, skip the pseudo-filesystems