the shed // linux / storage

A mirror that has been running degraded for three weeks is not redundancy, it is a countdown. This Ruby script turns /proc/mdstat into a clear verdict and an exit code your monitoring already understands.

Get the code

Full script + README on GitHub: ruby-devops-toolkit/mdstat-raid-monitor

Step through the build below:

mdstat_raid_monitor.rb

Linux md RAID fails quietly. When a member disk drops out of a RAID1 or RAID5, the array keeps serving I/O, the filesystem stays mounted, and nothing user-facing changes. The only evidence is a _ in the status bitmap of /proc/mdstat and, if you set it up, an email from mdadm --monitor that lands in a mailbox nobody reads.

We want a check that runs from cron or a monitoring agent, needs no root and no mdadm binary, and reports: which arrays exist, which are degraded, which members failed, and how long a running rebuild or scrub will take. It must exit 0 / 1 / 2 / 3 like a Nagios plugin so it plugs into Icinga, Zabbix, or a plain MAILTO cron job.

#!/usr/bin/env ruby
# frozen_string_literal: true
#
# mdstat_raid_monitor.rb - Linux software RAID (md) health monitor
#
# Parses /proc/mdstat (no root, no mdadm binary required), reports every
# array's level, member disks, failed members, and any running
# resync/recovery/check/reshape with progress and ETA. Exits with a
# Nagios/Icinga-compatible code so it can be dropped straight into cron,
# a systemd timer, or a monitoring check:
#
#   0 = OK        all arrays clean
#   1 = WARNING   a rebuild/resync/check is in progress (array still fully readable)
#   2 = CRITICAL  an array is degraded or has failed members / inactive
#   3 = UNKNOWN   /proc/mdstat unreadable or no md arrays found
#
# Usage:
#   ruby mdstat_raid_monitor.rb                 # human-readable report
#   ruby mdstat_raid_monitor.rb --json          # machine-readable JSON
#   ruby mdstat_raid_monitor.rb --file test.txt # parse a saved mdstat (for testing)
#   ruby mdstat_raid_monitor.rb --quiet         # only print when not OK (cron-friendly)
#
# Ruby >= 2.7, stdlib only.
require 'json'
require 'optparse'
module MdstatMonitor
  VERSION = '1.0.0'
  MDSTAT_PATH = '/proc/mdstat'
  # One entry per array found in /proc/mdstat.
  Array_ = Struct.new(
    :name, :active, :level, :members, :failed_members, :spare_members,
    :total_slots, :working_slots, :status_map, :operation, :progress_pct,
    :finish_min, :speed_kbs, :degraded, keyword_init: true
  ) do
    def state
      return 'INACTIVE' unless active
      return 'DEGRADED' if degraded
      return operation.upcase if operation
      'CLEAN'
    end
  end
  # Pure parser: takes the text of /proc/mdstat, returns an Array of Array_.
  class Parser
    # Example header line:
    #   md0 : active raid1 sdb1[1] sda1[0](F) sdc1[2](S)
    HEADER = /\A(md\d+)\s*:\s*(\w+)\s+(?:\((?:auto-)?read-only\)\s+)?(\S+)\s+(.*)\z/.freeze
    MEMBER = /(\S+?)\[(\d+)\]((?:\([A-Z]\))*)/.freeze
    # Example status line:
    #   1953382400 blocks super 1.2 [2/2] [UU]
    STATUS = /\[(\d+)\/(\d+)\]\s+\[([U_]+)\]/.freeze
    # Example progress line:
    #   [=====>...............]  recovery = 27.4% (535938432/1953382400) finish=118.2min speed=199568K/sec
    PROGRESS = /\]\s+(resync|recovery|reshape|check)\s*=\s*([\d.]+)%.*?finish=([\d.]+)min\s+speed=(\d+)K\/sec/.freeze
    def parse(text)
      arrays = []
      current = nil
      text.each_line do |raw|
        line = raw.rstrip
        next if line.empty?
        if (m = HEADER.match(line))
          current = build_array(m)
          arrays << current
          next
        end
        next unless current # skip "Personalities :" and "unused devices:" lines
        if (m = STATUS.match(line))
          current.total_slots   = m[1].to_i
          current.working_slots = m[2].to_i
          current.status_map    = m[3]
          current.degraded      = m[3].include?('_') || m[2].to_i < m[1].to_i
        elsif (m = PROGRESS.match(line))
          current.operation    = m[1]
          current.progress_pct = m[2].to_f
          current.finish_min   = m[3].to_f
          current.speed_kbs    = m[4].to_i
        elsif line =~ /resync=(PENDING|DELAYED)/
          current.operation = "resync-#{Regexp.last_match(1).downcase}"
        end
      end
      arrays
    end
    private
    def build_array(m)
      members = m[4].scan(MEMBER).map { |dev, slot, flags| [dev, slot.to_i, flags] }
      Array_.new(
        name: m[1],
        active: m[2] == 'active',
        level: m[3],
        members: members.map(&:first),
        failed_members: members.select { |_, _, f| f.include?('(F)') }.map(&:first),
        spare_members:  members.select { |_, _, f| f.include?('(S)') }.map(&:first),
        total_slots: 0, working_slots: 0, status_map: '',
        operation: nil, progress_pct: nil, finish_min: nil, speed_kbs: nil,
        degraded: m[2] != 'active'
      )
    end
  end
  # Turns parsed arrays into an exit code + summary line.
  class Evaluator
    OK = 0
    WARNING = 1
    CRITICAL = 2
    UNKNOWN = 3
    def evaluate(arrays)
      return [UNKNOWN, 'UNKNOWN - no md arrays found'] if arrays.empty?
      crit = arrays.select { |a| !a.active || a.degraded || a.failed_members.any? }
      warn = arrays.select { |a| a.operation && !crit.include?(a) }
      if crit.any?
        [CRITICAL, "CRITICAL - #{crit.map { |a| "#{a.name} #{a.state}" }.join(', ')}"]
      elsif warn.any?
        [WARNING, "WARNING - #{warn.map { |a| "#{a.name} #{a.operation} #{a.progress_pct}%" }.join(', ')}"]
      else
        [OK, "OK - #{arrays.size} array(s) clean: #{arrays.map(&:name).join(', ')}"]
      end
    end
  end
  # Rendering helpers.
  class Report
    def self.text(arrays, summary)
      out = [summary, '']
      arrays.each do |a|
        out << format('%-6s %-8s %-9s slots %d/%d %s',
                      a.name, a.level, a.state, a.working_slots, a.total_slots, a.status_map)
        out << "       members : #{a.members.join(' ')}"
        out << "       FAILED  : #{a.failed_members.join(' ')}" if a.failed_members.any?
        out << "       spares  : #{a.spare_members.join(' ')}" if a.spare_members.any?
        if a.progress_pct
          eta = a.finish_min >= 60 ? format('%.1fh', a.finish_min / 60) : format('%.0fmin', a.finish_min)
          out << format('       %-8s: %.1f%%  ETA %s  @ %d MB/s', a.operation, a.progress_pct, eta, a.speed_kbs / 1024)
        end
      end
      out.join("\n")
    end
    def self.json(arrays, code, summary)
      JSON.pretty_generate(
        status: %w[OK WARNING CRITICAL UNKNOWN][code],
        summary: summary,
        checked_at: Time.now.utc.iso8601,
        arrays: arrays.map { |a| a.to_h.merge(state: a.state) }
      )
    end
  end
  def self.run(argv)
    opts = { file: MDSTAT_PATH, json: false, quiet: false }
    OptionParser.new do |o|
      o.banner = 'Usage: mdstat_raid_monitor.rb [--file PATH] [--json] [--quiet]'
      o.on('--file PATH', 'Parse this file instead of /proc/mdstat') { |v| opts[:file] = v }
      o.on('--json', 'Emit JSON') { opts[:json] = true }
      o.on('--quiet', 'Print nothing when status is OK') { opts[:quiet] = true }
      o.on('-v', '--version') { puts VERSION; exit 0 }
    end.parse!(argv)
    text = begin
      File.read(opts[:file])
    rescue Errno::ENOENT, Errno::EACCES => e
      puts "UNKNOWN - cannot read #{opts[:file]}: #{e.message}"
      exit Evaluator::UNKNOWN
    end
    arrays = Parser.new.parse(text)
    code, summary = Evaluator.new.evaluate(arrays)
    if opts[:json]
      puts Report.json(arrays, code, summary)
    elsif !(opts[:quiet] && code == Evaluator::OK)
      puts Report.text(arrays, summary)
    end
    exit code
  end
end
require 'time'
MdstatMonitor.run(ARGV) if $PROGRAM_NAME == __FILE__

Parser is a pure function over text: four regexes recognise the header line (md1 : active raid5 sdd2[3] ... sda2[0](F)), the status line ([4/3] [_UUU]), and any progress line (recovery = 27.4% ... finish=118.2min speed=199568K/sec). Member flags (F) and (S) become failed_members and spare_members.

Evaluator maps arrays to a severity: inactive, degraded, or any failed member is CRITICAL; a running resync/recovery/check/reshape on an otherwise healthy array is WARNING; everything else is OK. Empty input is UNKNOWN (3), which keeps a server with no md arrays from showing green by accident.

Report renders a text table or --json. --file lets you feed saved fixtures, which is exactly how the output tab was produced inside a container that has no RAID at all.

$ captured from the sandbox test run

$ ruby mdstat_raid_monitor.rb --file fixtures/mdstat_degraded.txt
CRITICAL - md1 DEGRADED
md0    raid1    CLEAN     slots 2/2 UU
       members : sdb1 sda1
md1    raid5    DEGRADED  slots 3/4 _UUU
       members : sdd2 sdc2 sdb2 sda2
       FAILED  : sda2
       recovery: 27.4%  ETA 2.0h  @ 194 MB/s
md2    raid10   CHECK     slots 4/4 UUUU
       members : sdh1 sdg1 sdf1 sde1
       check   : 12.0%  ETA 4.8h  @ 192 MB/s
exit=2
$ ruby mdstat_raid_monitor.rb --file fixtures/mdstat_clean.txt --quiet
exit=0
$ ruby mdstat_raid_monitor.rb --file fixtures/mdstat_degraded.txt --json | head -12
{
  "status": "CRITICAL",
  "summary": "CRITICAL - md1 DEGRADED",
  "checked_at": "2026-09-09T15:39:40Z",
  "arrays": [
    {
      "name": "md0",
      "active": true,
      "level": "raid1",
      "members": [
        "sdb1",
        "sda1"
  ...
Data flow: /proc/mdstat to Parser to Evaluator to exit code

/proc/mdstat -> Parser -> Evaluator -> Report + Nagios-style exit code
01 / context

The real-world problem

Software RAID via mdadm is still everywhere: on-prem hypervisors, NAS boxes, bare-metal database hosts, and the boot mirror on every server that never got a hardware controller. Its weak point is not reliability, it is visibility. The kernel exposes everything through /proc/mdstat, but that file is meant for humans, and most teams only look at it after the second disk dies.

Ruby is a good fit here for three reasons. The parsing is regex-heavy and Ruby’s regex ergonomics are excellent; the script needs zero gems so it runs on any distro’s system Ruby; and wrapping the result in a Struct gives you a clean JSON export for dashboards without a second tool.

The script answers four questions on every run: Which arrays exist and at what level? Are any degraded or inactive? Which specific members failed? Is a rebuild or scrub running, and when will it finish? The first line of output is a one-line summary and the process exit code carries the same verdict.

02 / setup

Prerequisites

You need
  • Linux with the md driver loaded (any box that shows arrays in cat /proc/mdstat).
  • Ruby 2.7 or newer — tested on Ruby 3.0.2. Stdlib only: json, optparse, time.
  • No root required: /proc/mdstat is world-readable.
  • Optional: a cron entry or systemd timer, and a Nagios/Icinga/Zabbix agent that understands exit codes 0-3.
03 / source

The complete script

Everything below is the exact file that ran in the output tab. It is stdlib-only, so there is no Gemfile to install.

mdstat_raid_monitor.rbruby
#!/usr/bin/env ruby
# frozen_string_literal: true
#
# mdstat_raid_monitor.rb - Linux software RAID (md) health monitor
#
# Parses /proc/mdstat (no root, no mdadm binary required), reports every
# array's level, member disks, failed members, and any running
# resync/recovery/check/reshape with progress and ETA. Exits with a
# Nagios/Icinga-compatible code so it can be dropped straight into cron,
# a systemd timer, or a monitoring check:
#
#   0 = OK        all arrays clean
#   1 = WARNING   a rebuild/resync/check is in progress (array still fully readable)
#   2 = CRITICAL  an array is degraded or has failed members / inactive
#   3 = UNKNOWN   /proc/mdstat unreadable or no md arrays found
#
# Usage:
#   ruby mdstat_raid_monitor.rb                 # human-readable report
#   ruby mdstat_raid_monitor.rb --json          # machine-readable JSON
#   ruby mdstat_raid_monitor.rb --file test.txt # parse a saved mdstat (for testing)
#   ruby mdstat_raid_monitor.rb --quiet         # only print when not OK (cron-friendly)
#
# Ruby >= 2.7, stdlib only.
require 'json'
require 'optparse'
module MdstatMonitor
  VERSION = '1.0.0'
  MDSTAT_PATH = '/proc/mdstat'
  # One entry per array found in /proc/mdstat.
  Array_ = Struct.new(
    :name, :active, :level, :members, :failed_members, :spare_members,
    :total_slots, :working_slots, :status_map, :operation, :progress_pct,
    :finish_min, :speed_kbs, :degraded, keyword_init: true
  ) do
    def state
      return 'INACTIVE' unless active
      return 'DEGRADED' if degraded
      return operation.upcase if operation
      'CLEAN'
    end
  end
  # Pure parser: takes the text of /proc/mdstat, returns an Array of Array_.
  class Parser
    # Example header line:
    #   md0 : active raid1 sdb1[1] sda1[0](F) sdc1[2](S)
    HEADER = /\A(md\d+)\s*:\s*(\w+)\s+(?:\((?:auto-)?read-only\)\s+)?(\S+)\s+(.*)\z/.freeze
    MEMBER = /(\S+?)\[(\d+)\]((?:\([A-Z]\))*)/.freeze
    # Example status line:
    #   1953382400 blocks super 1.2 [2/2] [UU]
    STATUS = /\[(\d+)\/(\d+)\]\s+\[([U_]+)\]/.freeze
    # Example progress line:
    #   [=====>...............]  recovery = 27.4% (535938432/1953382400) finish=118.2min speed=199568K/sec
    PROGRESS = /\]\s+(resync|recovery|reshape|check)\s*=\s*([\d.]+)%.*?finish=([\d.]+)min\s+speed=(\d+)K\/sec/.freeze
    def parse(text)
      arrays = []
      current = nil
      text.each_line do |raw|
        line = raw.rstrip
        next if line.empty?
        if (m = HEADER.match(line))
          current = build_array(m)
          arrays << current
          next
        end
        next unless current # skip "Personalities :" and "unused devices:" lines
        if (m = STATUS.match(line))
          current.total_slots   = m[1].to_i
          current.working_slots = m[2].to_i
          current.status_map    = m[3]
          current.degraded      = m[3].include?('_') || m[2].to_i < m[1].to_i
        elsif (m = PROGRESS.match(line))
          current.operation    = m[1]
          current.progress_pct = m[2].to_f
          current.finish_min   = m[3].to_f
          current.speed_kbs    = m[4].to_i
        elsif line =~ /resync=(PENDING|DELAYED)/
          current.operation = "resync-#{Regexp.last_match(1).downcase}"
        end
      end
      arrays
    end
    private
    def build_array(m)
      members = m[4].scan(MEMBER).map { |dev, slot, flags| [dev, slot.to_i, flags] }
      Array_.new(
        name: m[1],
        active: m[2] == 'active',
        level: m[3],
        members: members.map(&:first),
        failed_members: members.select { |_, _, f| f.include?('(F)') }.map(&:first),
        spare_members:  members.select { |_, _, f| f.include?('(S)') }.map(&:first),
        total_slots: 0, working_slots: 0, status_map: '',
        operation: nil, progress_pct: nil, finish_min: nil, speed_kbs: nil,
        degraded: m[2] != 'active'
      )
    end
  end
  # Turns parsed arrays into an exit code + summary line.
  class Evaluator
    OK = 0
    WARNING = 1
    CRITICAL = 2
    UNKNOWN = 3
    def evaluate(arrays)
      return [UNKNOWN, 'UNKNOWN - no md arrays found'] if arrays.empty?
      crit = arrays.select { |a| !a.active || a.degraded || a.failed_members.any? }
      warn = arrays.select { |a| a.operation && !crit.include?(a) }
      if crit.any?
        [CRITICAL, "CRITICAL - #{crit.map { |a| "#{a.name} #{a.state}" }.join(', ')}"]
      elsif warn.any?
        [WARNING, "WARNING - #{warn.map { |a| "#{a.name} #{a.operation} #{a.progress_pct}%" }.join(', ')}"]
      else
        [OK, "OK - #{arrays.size} array(s) clean: #{arrays.map(&:name).join(', ')}"]
      end
    end
  end
  # Rendering helpers.
  class Report
    def self.text(arrays, summary)
      out = [summary, '']
      arrays.each do |a|
        out << format('%-6s %-8s %-9s slots %d/%d %s',
                      a.name, a.level, a.state, a.working_slots, a.total_slots, a.status_map)
        out << "       members : #{a.members.join(' ')}"
        out << "       FAILED  : #{a.failed_members.join(' ')}" if a.failed_members.any?
        out << "       spares  : #{a.spare_members.join(' ')}" if a.spare_members.any?
        if a.progress_pct
          eta = a.finish_min >= 60 ? format('%.1fh', a.finish_min / 60) : format('%.0fmin', a.finish_min)
          out << format('       %-8s: %.1f%%  ETA %s  @ %d MB/s', a.operation, a.progress_pct, eta, a.speed_kbs / 1024)
        end
      end
      out.join("\n")
    end
    def self.json(arrays, code, summary)
      JSON.pretty_generate(
        status: %w[OK WARNING CRITICAL UNKNOWN][code],
        summary: summary,
        checked_at: Time.now.utc.iso8601,
        arrays: arrays.map { |a| a.to_h.merge(state: a.state) }
      )
    end
  end
  def self.run(argv)
    opts = { file: MDSTAT_PATH, json: false, quiet: false }
    OptionParser.new do |o|
      o.banner = 'Usage: mdstat_raid_monitor.rb [--file PATH] [--json] [--quiet]'
      o.on('--file PATH', 'Parse this file instead of /proc/mdstat') { |v| opts[:file] = v }
      o.on('--json', 'Emit JSON') { opts[:json] = true }
      o.on('--quiet', 'Print nothing when status is OK') { opts[:quiet] = true }
      o.on('-v', '--version') { puts VERSION; exit 0 }
    end.parse!(argv)
    text = begin
      File.read(opts[:file])
    rescue Errno::ENOENT, Errno::EACCES => e
      puts "UNKNOWN - cannot read #{opts[:file]}: #{e.message}"
      exit Evaluator::UNKNOWN
    end
    arrays = Parser.new.parse(text)
    code, summary = Evaluator.new.evaluate(arrays)
    if opts[:json]
      puts Report.json(arrays, code, summary)
    elsif !(opts[:quiet] && code == Evaluator::OK)
      puts Report.text(arrays, summary)
    end
    exit code
  end
end
require 'time'
MdstatMonitor.run(ARGV) if $PROGRAM_NAME == __FILE__
04 / walkthrough

How the code works, step by step

1. Read the source, or a fixture

--file defaults to /proc/mdstat. If the file cannot be read the script prints UNKNOWN and exits 3 instead of raising a stack trace, which is what a monitoring agent expects.

2. Parse headers and members

The HEADER regex captures the array name, the active/inactive word, the RAID level, and the remaining member list. It also tolerates the optional (read-only) and (auto-read-only) markers. The MEMBER regex then pulls device[slot](FLAGS) triples, so sda2[0](F) becomes a failed member and sdc1[2](S) a spare.

3. Parse status and progress

[4/3] [_UUU] means four slots, three working. The script marks the array degraded if either the counts differ or the bitmap contains an underscore. The PROGRESS regex reads the operation name, percentage, finish= minutes and speed= in KB/s. A resync=PENDING or DELAYED line is recorded too so a queued resync still shows as WARNING.

4. Evaluate

Evaluator#evaluate returns [code, summary]. Severity ordering matters: an array that is both degraded and rebuilding is CRITICAL, not WARNING, because the rebuild is not finished yet and a second failure during it is fatal.

5. Report and exit

--quiet suppresses output when everything is OK, so a cron job with MAILTO only emails you when something is wrong. --json emits the full structures for Grafana, a Slack bot, or your CMDB.

05 / output

Example output

Run against the degraded fixture (a RAID5 with a failed member mid-recovery plus a RAID10 mid-scrub), the summary line and per-array table look like this:

ruby mdstat_raid_monitor.rb –file fixtures/mdstat_degraded.txt
CRITICAL – md1 DEGRADED
md0 raid1 CLEAN slots 2/2 UU
members : sdb1 sda1
md1 raid5 DEGRADED slots 3/4 _UUU
members : sdd2 sdc2 sdb2 sda2
FAILED : sda2
recovery: 27.4% ETA 2.0h @ 194 MB/s
md2 raid10 CHECK slots 4/4 UUUU
members : sdh1 sdg1 sdf1 sde1
check : 12.0% ETA 4.8h @ 192 MB/s
exit=2
06 / debug

Troubleshooting

When it misbehaves
  • UNKNOWN – no md arrays found on a host that definitely has RAID: you are probably inside a container or a VM where /proc/mdstat is the host’s empty view. Run it on the host, or pass --file with a copy.
  • An array shows CLEAN but mdadm says it is degraded: compare with mdadm --detail /dev/mdX. Some kernels print the status bitmap on the line after the block count; the parser scans every line so this is normally fine, but send the raw /proc/mdstat as a fixture and adjust STATUS if your layout differs.
  • Progress never appears for a scrub: a check only shows a progress bar while it runs. Trigger one with echo check > /sys/block/md0/md/sync_action to test.
  • Exit code is always 0 under cron: make sure you are not piping the script into tee or mail; the pipeline’s exit status is the last command’s. Use set -o pipefail or write to a file first.
  • Everything here was tested in a Linux sandbox using saved fixtures via --file; the sandbox has no md devices, so the real /proc/mdstat run correctly returned UNKNOWN (exit 3).
07 / next

Extending the script

Ideas
  • Add --warn-eta HOURS to escalate a rebuild that will take longer than your recovery window to CRITICAL.
  • Emit Prometheus text format (md_array_degraded{name="md1"} 1) and serve it from the toolkit’s prometheus-exporter.
  • Cross-reference failed members with smartctl -H output to attach the drive’s serial number to the alert.
  • Push the JSON to a webhook so a Slack channel gets one message when an array degrades and another when the rebuild finishes.
  • Track speed_kbs over time to spot a rebuild that has stalled because of a second marginal disk.