A mirror that has been running degraded for three weeks is not redundancy, it is a countdown. This Ruby script turns /proc/mdstat into a clear verdict and an exit code your monitoring already understands.
Full script + README on GitHub: ruby-devops-toolkit/mdstat-raid-monitor
Step through the build below:
Linux md RAID fails quietly. When a member disk drops out of a RAID1 or RAID5, the array keeps serving I/O, the filesystem stays mounted, and nothing user-facing changes. The only evidence is a _ in the status bitmap of /proc/mdstat and, if you set it up, an email from mdadm --monitor that lands in a mailbox nobody reads.
We want a check that runs from cron or a monitoring agent, needs no root and no mdadm binary, and reports: which arrays exist, which are degraded, which members failed, and how long a running rebuild or scrub will take. It must exit 0 / 1 / 2 / 3 like a Nagios plugin so it plugs into Icinga, Zabbix, or a plain MAILTO cron job.
#!/usr/bin/env ruby
# frozen_string_literal: true
#
# mdstat_raid_monitor.rb - Linux software RAID (md) health monitor
#
# Parses /proc/mdstat (no root, no mdadm binary required), reports every
# array's level, member disks, failed members, and any running
# resync/recovery/check/reshape with progress and ETA. Exits with a
# Nagios/Icinga-compatible code so it can be dropped straight into cron,
# a systemd timer, or a monitoring check:
#
# 0 = OK all arrays clean
# 1 = WARNING a rebuild/resync/check is in progress (array still fully readable)
# 2 = CRITICAL an array is degraded or has failed members / inactive
# 3 = UNKNOWN /proc/mdstat unreadable or no md arrays found
#
# Usage:
# ruby mdstat_raid_monitor.rb # human-readable report
# ruby mdstat_raid_monitor.rb --json # machine-readable JSON
# ruby mdstat_raid_monitor.rb --file test.txt # parse a saved mdstat (for testing)
# ruby mdstat_raid_monitor.rb --quiet # only print when not OK (cron-friendly)
#
# Ruby >= 2.7, stdlib only.
require 'json'
require 'optparse'
module MdstatMonitor
VERSION = '1.0.0'
MDSTAT_PATH = '/proc/mdstat'
# One entry per array found in /proc/mdstat.
Array_ = Struct.new(
:name, :active, :level, :members, :failed_members, :spare_members,
:total_slots, :working_slots, :status_map, :operation, :progress_pct,
:finish_min, :speed_kbs, :degraded, keyword_init: true
) do
def state
return 'INACTIVE' unless active
return 'DEGRADED' if degraded
return operation.upcase if operation
'CLEAN'
end
end
# Pure parser: takes the text of /proc/mdstat, returns an Array of Array_.
class Parser
# Example header line:
# md0 : active raid1 sdb1[1] sda1[0](F) sdc1[2](S)
HEADER = /\A(md\d+)\s*:\s*(\w+)\s+(?:\((?:auto-)?read-only\)\s+)?(\S+)\s+(.*)\z/.freeze
MEMBER = /(\S+?)\[(\d+)\]((?:\([A-Z]\))*)/.freeze
# Example status line:
# 1953382400 blocks super 1.2 [2/2] [UU]
STATUS = /\[(\d+)\/(\d+)\]\s+\[([U_]+)\]/.freeze
# Example progress line:
# [=====>...............] recovery = 27.4% (535938432/1953382400) finish=118.2min speed=199568K/sec
PROGRESS = /\]\s+(resync|recovery|reshape|check)\s*=\s*([\d.]+)%.*?finish=([\d.]+)min\s+speed=(\d+)K\/sec/.freeze
def parse(text)
arrays = []
current = nil
text.each_line do |raw|
line = raw.rstrip
next if line.empty?
if (m = HEADER.match(line))
current = build_array(m)
arrays << current
next
end
next unless current # skip "Personalities :" and "unused devices:" lines
if (m = STATUS.match(line))
current.total_slots = m[1].to_i
current.working_slots = m[2].to_i
current.status_map = m[3]
current.degraded = m[3].include?('_') || m[2].to_i < m[1].to_i
elsif (m = PROGRESS.match(line))
current.operation = m[1]
current.progress_pct = m[2].to_f
current.finish_min = m[3].to_f
current.speed_kbs = m[4].to_i
elsif line =~ /resync=(PENDING|DELAYED)/
current.operation = "resync-#{Regexp.last_match(1).downcase}"
end
end
arrays
end
private
def build_array(m)
members = m[4].scan(MEMBER).map { |dev, slot, flags| [dev, slot.to_i, flags] }
Array_.new(
name: m[1],
active: m[2] == 'active',
level: m[3],
members: members.map(&:first),
failed_members: members.select { |_, _, f| f.include?('(F)') }.map(&:first),
spare_members: members.select { |_, _, f| f.include?('(S)') }.map(&:first),
total_slots: 0, working_slots: 0, status_map: '',
operation: nil, progress_pct: nil, finish_min: nil, speed_kbs: nil,
degraded: m[2] != 'active'
)
end
end
# Turns parsed arrays into an exit code + summary line.
class Evaluator
OK = 0
WARNING = 1
CRITICAL = 2
UNKNOWN = 3
def evaluate(arrays)
return [UNKNOWN, 'UNKNOWN - no md arrays found'] if arrays.empty?
crit = arrays.select { |a| !a.active || a.degraded || a.failed_members.any? }
warn = arrays.select { |a| a.operation && !crit.include?(a) }
if crit.any?
[CRITICAL, "CRITICAL - #{crit.map { |a| "#{a.name} #{a.state}" }.join(', ')}"]
elsif warn.any?
[WARNING, "WARNING - #{warn.map { |a| "#{a.name} #{a.operation} #{a.progress_pct}%" }.join(', ')}"]
else
[OK, "OK - #{arrays.size} array(s) clean: #{arrays.map(&:name).join(', ')}"]
end
end
end
# Rendering helpers.
class Report
def self.text(arrays, summary)
out = [summary, '']
arrays.each do |a|
out << format('%-6s %-8s %-9s slots %d/%d %s',
a.name, a.level, a.state, a.working_slots, a.total_slots, a.status_map)
out << " members : #{a.members.join(' ')}"
out << " FAILED : #{a.failed_members.join(' ')}" if a.failed_members.any?
out << " spares : #{a.spare_members.join(' ')}" if a.spare_members.any?
if a.progress_pct
eta = a.finish_min >= 60 ? format('%.1fh', a.finish_min / 60) : format('%.0fmin', a.finish_min)
out << format(' %-8s: %.1f%% ETA %s @ %d MB/s', a.operation, a.progress_pct, eta, a.speed_kbs / 1024)
end
end
out.join("\n")
end
def self.json(arrays, code, summary)
JSON.pretty_generate(
status: %w[OK WARNING CRITICAL UNKNOWN][code],
summary: summary,
checked_at: Time.now.utc.iso8601,
arrays: arrays.map { |a| a.to_h.merge(state: a.state) }
)
end
end
def self.run(argv)
opts = { file: MDSTAT_PATH, json: false, quiet: false }
OptionParser.new do |o|
o.banner = 'Usage: mdstat_raid_monitor.rb [--file PATH] [--json] [--quiet]'
o.on('--file PATH', 'Parse this file instead of /proc/mdstat') { |v| opts[:file] = v }
o.on('--json', 'Emit JSON') { opts[:json] = true }
o.on('--quiet', 'Print nothing when status is OK') { opts[:quiet] = true }
o.on('-v', '--version') { puts VERSION; exit 0 }
end.parse!(argv)
text = begin
File.read(opts[:file])
rescue Errno::ENOENT, Errno::EACCES => e
puts "UNKNOWN - cannot read #{opts[:file]}: #{e.message}"
exit Evaluator::UNKNOWN
end
arrays = Parser.new.parse(text)
code, summary = Evaluator.new.evaluate(arrays)
if opts[:json]
puts Report.json(arrays, code, summary)
elsif !(opts[:quiet] && code == Evaluator::OK)
puts Report.text(arrays, summary)
end
exit code
end
end
require 'time'
MdstatMonitor.run(ARGV) if $PROGRAM_NAME == __FILE__
Parser is a pure function over text: four regexes recognise the header line (md1 : active raid5 sdd2[3] ... sda2[0](F)), the status line ([4/3] [_UUU]), and any progress line (recovery = 27.4% ... finish=118.2min speed=199568K/sec). Member flags (F) and (S) become failed_members and spare_members.
Evaluator maps arrays to a severity: inactive, degraded, or any failed member is CRITICAL; a running resync/recovery/check/reshape on an otherwise healthy array is WARNING; everything else is OK. Empty input is UNKNOWN (3), which keeps a server with no md arrays from showing green by accident.
Report renders a text table or --json. --file lets you feed saved fixtures, which is exactly how the output tab was produced inside a container that has no RAID at all.
$ captured from the sandbox test run
$ ruby mdstat_raid_monitor.rb --file fixtures/mdstat_degraded.txt
CRITICAL - md1 DEGRADED
md0 raid1 CLEAN slots 2/2 UU
members : sdb1 sda1
md1 raid5 DEGRADED slots 3/4 _UUU
members : sdd2 sdc2 sdb2 sda2
FAILED : sda2
recovery: 27.4% ETA 2.0h @ 194 MB/s
md2 raid10 CHECK slots 4/4 UUUU
members : sdh1 sdg1 sdf1 sde1
check : 12.0% ETA 4.8h @ 192 MB/s
exit=2
$ ruby mdstat_raid_monitor.rb --file fixtures/mdstat_clean.txt --quiet
exit=0
$ ruby mdstat_raid_monitor.rb --file fixtures/mdstat_degraded.txt --json | head -12
{
"status": "CRITICAL",
"summary": "CRITICAL - md1 DEGRADED",
"checked_at": "2026-09-09T15:39:40Z",
"arrays": [
{
"name": "md0",
"active": true,
"level": "raid1",
"members": [
"sdb1",
"sda1"
...
The real-world problem
Software RAID via mdadm is still everywhere: on-prem hypervisors, NAS boxes, bare-metal database hosts, and the boot mirror on every server that never got a hardware controller. Its weak point is not reliability, it is visibility. The kernel exposes everything through /proc/mdstat, but that file is meant for humans, and most teams only look at it after the second disk dies.
Ruby is a good fit here for three reasons. The parsing is regex-heavy and Ruby’s regex ergonomics are excellent; the script needs zero gems so it runs on any distro’s system Ruby; and wrapping the result in a Struct gives you a clean JSON export for dashboards without a second tool.
The script answers four questions on every run: Which arrays exist and at what level? Are any degraded or inactive? Which specific members failed? Is a rebuild or scrub running, and when will it finish? The first line of output is a one-line summary and the process exit code carries the same verdict.
Prerequisites
- Linux with the md driver loaded (any box that shows arrays in
cat /proc/mdstat). - Ruby 2.7 or newer — tested on Ruby 3.0.2. Stdlib only:
json,optparse,time. - No root required:
/proc/mdstatis world-readable. - Optional: a cron entry or systemd timer, and a Nagios/Icinga/Zabbix agent that understands exit codes 0-3.
The complete script
Everything below is the exact file that ran in the output tab. It is stdlib-only, so there is no Gemfile to install.
#!/usr/bin/env ruby
# frozen_string_literal: true
#
# mdstat_raid_monitor.rb - Linux software RAID (md) health monitor
#
# Parses /proc/mdstat (no root, no mdadm binary required), reports every
# array's level, member disks, failed members, and any running
# resync/recovery/check/reshape with progress and ETA. Exits with a
# Nagios/Icinga-compatible code so it can be dropped straight into cron,
# a systemd timer, or a monitoring check:
#
# 0 = OK all arrays clean
# 1 = WARNING a rebuild/resync/check is in progress (array still fully readable)
# 2 = CRITICAL an array is degraded or has failed members / inactive
# 3 = UNKNOWN /proc/mdstat unreadable or no md arrays found
#
# Usage:
# ruby mdstat_raid_monitor.rb # human-readable report
# ruby mdstat_raid_monitor.rb --json # machine-readable JSON
# ruby mdstat_raid_monitor.rb --file test.txt # parse a saved mdstat (for testing)
# ruby mdstat_raid_monitor.rb --quiet # only print when not OK (cron-friendly)
#
# Ruby >= 2.7, stdlib only.
require 'json'
require 'optparse'
module MdstatMonitor
VERSION = '1.0.0'
MDSTAT_PATH = '/proc/mdstat'
# One entry per array found in /proc/mdstat.
Array_ = Struct.new(
:name, :active, :level, :members, :failed_members, :spare_members,
:total_slots, :working_slots, :status_map, :operation, :progress_pct,
:finish_min, :speed_kbs, :degraded, keyword_init: true
) do
def state
return 'INACTIVE' unless active
return 'DEGRADED' if degraded
return operation.upcase if operation
'CLEAN'
end
end
# Pure parser: takes the text of /proc/mdstat, returns an Array of Array_.
class Parser
# Example header line:
# md0 : active raid1 sdb1[1] sda1[0](F) sdc1[2](S)
HEADER = /\A(md\d+)\s*:\s*(\w+)\s+(?:\((?:auto-)?read-only\)\s+)?(\S+)\s+(.*)\z/.freeze
MEMBER = /(\S+?)\[(\d+)\]((?:\([A-Z]\))*)/.freeze
# Example status line:
# 1953382400 blocks super 1.2 [2/2] [UU]
STATUS = /\[(\d+)\/(\d+)\]\s+\[([U_]+)\]/.freeze
# Example progress line:
# [=====>...............] recovery = 27.4% (535938432/1953382400) finish=118.2min speed=199568K/sec
PROGRESS = /\]\s+(resync|recovery|reshape|check)\s*=\s*([\d.]+)%.*?finish=([\d.]+)min\s+speed=(\d+)K\/sec/.freeze
def parse(text)
arrays = []
current = nil
text.each_line do |raw|
line = raw.rstrip
next if line.empty?
if (m = HEADER.match(line))
current = build_array(m)
arrays << current
next
end
next unless current # skip "Personalities :" and "unused devices:" lines
if (m = STATUS.match(line))
current.total_slots = m[1].to_i
current.working_slots = m[2].to_i
current.status_map = m[3]
current.degraded = m[3].include?('_') || m[2].to_i < m[1].to_i
elsif (m = PROGRESS.match(line))
current.operation = m[1]
current.progress_pct = m[2].to_f
current.finish_min = m[3].to_f
current.speed_kbs = m[4].to_i
elsif line =~ /resync=(PENDING|DELAYED)/
current.operation = "resync-#{Regexp.last_match(1).downcase}"
end
end
arrays
end
private
def build_array(m)
members = m[4].scan(MEMBER).map { |dev, slot, flags| [dev, slot.to_i, flags] }
Array_.new(
name: m[1],
active: m[2] == 'active',
level: m[3],
members: members.map(&:first),
failed_members: members.select { |_, _, f| f.include?('(F)') }.map(&:first),
spare_members: members.select { |_, _, f| f.include?('(S)') }.map(&:first),
total_slots: 0, working_slots: 0, status_map: '',
operation: nil, progress_pct: nil, finish_min: nil, speed_kbs: nil,
degraded: m[2] != 'active'
)
end
end
# Turns parsed arrays into an exit code + summary line.
class Evaluator
OK = 0
WARNING = 1
CRITICAL = 2
UNKNOWN = 3
def evaluate(arrays)
return [UNKNOWN, 'UNKNOWN - no md arrays found'] if arrays.empty?
crit = arrays.select { |a| !a.active || a.degraded || a.failed_members.any? }
warn = arrays.select { |a| a.operation && !crit.include?(a) }
if crit.any?
[CRITICAL, "CRITICAL - #{crit.map { |a| "#{a.name} #{a.state}" }.join(', ')}"]
elsif warn.any?
[WARNING, "WARNING - #{warn.map { |a| "#{a.name} #{a.operation} #{a.progress_pct}%" }.join(', ')}"]
else
[OK, "OK - #{arrays.size} array(s) clean: #{arrays.map(&:name).join(', ')}"]
end
end
end
# Rendering helpers.
class Report
def self.text(arrays, summary)
out = [summary, '']
arrays.each do |a|
out << format('%-6s %-8s %-9s slots %d/%d %s',
a.name, a.level, a.state, a.working_slots, a.total_slots, a.status_map)
out << " members : #{a.members.join(' ')}"
out << " FAILED : #{a.failed_members.join(' ')}" if a.failed_members.any?
out << " spares : #{a.spare_members.join(' ')}" if a.spare_members.any?
if a.progress_pct
eta = a.finish_min >= 60 ? format('%.1fh', a.finish_min / 60) : format('%.0fmin', a.finish_min)
out << format(' %-8s: %.1f%% ETA %s @ %d MB/s', a.operation, a.progress_pct, eta, a.speed_kbs / 1024)
end
end
out.join("\n")
end
def self.json(arrays, code, summary)
JSON.pretty_generate(
status: %w[OK WARNING CRITICAL UNKNOWN][code],
summary: summary,
checked_at: Time.now.utc.iso8601,
arrays: arrays.map { |a| a.to_h.merge(state: a.state) }
)
end
end
def self.run(argv)
opts = { file: MDSTAT_PATH, json: false, quiet: false }
OptionParser.new do |o|
o.banner = 'Usage: mdstat_raid_monitor.rb [--file PATH] [--json] [--quiet]'
o.on('--file PATH', 'Parse this file instead of /proc/mdstat') { |v| opts[:file] = v }
o.on('--json', 'Emit JSON') { opts[:json] = true }
o.on('--quiet', 'Print nothing when status is OK') { opts[:quiet] = true }
o.on('-v', '--version') { puts VERSION; exit 0 }
end.parse!(argv)
text = begin
File.read(opts[:file])
rescue Errno::ENOENT, Errno::EACCES => e
puts "UNKNOWN - cannot read #{opts[:file]}: #{e.message}"
exit Evaluator::UNKNOWN
end
arrays = Parser.new.parse(text)
code, summary = Evaluator.new.evaluate(arrays)
if opts[:json]
puts Report.json(arrays, code, summary)
elsif !(opts[:quiet] && code == Evaluator::OK)
puts Report.text(arrays, summary)
end
exit code
end
end
require 'time'
MdstatMonitor.run(ARGV) if $PROGRAM_NAME == __FILE__
How the code works, step by step
1. Read the source, or a fixture
--file defaults to /proc/mdstat. If the file cannot be read the script prints UNKNOWN and exits 3 instead of raising a stack trace, which is what a monitoring agent expects.
2. Parse headers and members
The HEADER regex captures the array name, the active/inactive word, the RAID level, and the remaining member list. It also tolerates the optional (read-only) and (auto-read-only) markers. The MEMBER regex then pulls device[slot](FLAGS) triples, so sda2[0](F) becomes a failed member and sdc1[2](S) a spare.
3. Parse status and progress
[4/3] [_UUU] means four slots, three working. The script marks the array degraded if either the counts differ or the bitmap contains an underscore. The PROGRESS regex reads the operation name, percentage, finish= minutes and speed= in KB/s. A resync=PENDING or DELAYED line is recorded too so a queued resync still shows as WARNING.
4. Evaluate
Evaluator#evaluate returns [code, summary]. Severity ordering matters: an array that is both degraded and rebuilding is CRITICAL, not WARNING, because the rebuild is not finished yet and a second failure during it is fatal.
5. Report and exit
--quiet suppresses output when everything is OK, so a cron job with MAILTO only emails you when something is wrong. --json emits the full structures for Grafana, a Slack bot, or your CMDB.
Example output
Run against the degraded fixture (a RAID5 with a failed member mid-recovery plus a RAID10 mid-scrub), the summary line and per-array table look like this:
Troubleshooting
- UNKNOWN – no md arrays found on a host that definitely has RAID: you are probably inside a container or a VM where
/proc/mdstatis the host’s empty view. Run it on the host, or pass--filewith a copy. - An array shows CLEAN but mdadm says it is degraded: compare with
mdadm --detail /dev/mdX. Some kernels print the status bitmap on the line after the block count; the parser scans every line so this is normally fine, but send the raw/proc/mdstatas a fixture and adjustSTATUSif your layout differs. - Progress never appears for a scrub: a
checkonly shows a progress bar while it runs. Trigger one withecho check > /sys/block/md0/md/sync_actionto test. - Exit code is always 0 under cron: make sure you are not piping the script into
teeormail; the pipeline’s exit status is the last command’s. Useset -o pipefailor write to a file first. - Everything here was tested in a Linux sandbox using saved fixtures via
--file; the sandbox has no md devices, so the real/proc/mdstatrun correctly returned UNKNOWN (exit 3).
Extending the script
- Add
--warn-eta HOURSto escalate a rebuild that will take longer than your recovery window to CRITICAL. - Emit Prometheus text format (
md_array_degraded{name="md1"} 1) and serve it from the toolkit’sprometheus-exporter. - Cross-reference failed members with
smartctl -Houtput to attach the drive’s serial number to the alert. - Push the JSON to a webhook so a Slack channel gets one message when an array degrades and another when the rebuild finishes.
- Track
speed_kbsover time to spot a rebuild that has stalled because of a second marginal disk.