RPi5 NVMe Emergency: Filesystem Gone Read-Only

The Situation

You're SSH'd into your Raspberry Pi 5 and something feels wrong. Maybe a Docker container failed to start. Maybe a git pull threw a strange error. Maybe you just noticed this in your logs:

EXT4-fs error (device nvme0n1p2): ext4_journal_check_start: Detected aborted journal
EXT4-fs (nvme0n1p2): Remounting filesystem read-only

Your NVMe drive has remounted itself as read-only. No writes are possible — no Docker containers can write logs, no files can be saved, no configuration changes will stick. Everything that depends on disk writes is broken.

This guide walks through diagnosing the problem, recovering the filesystem, and preventing it from happening again.

Why This Happens

The Linux ext4 filesystem has a built-in safety mechanism: when it detects errors it cannot handle safely — corrupt journal, bad blocks, unexpected power loss during a write — it remounts itself as read-only rather than continuing to write to a potentially damaged filesystem.

This is intentional and correct behavior. A filesystem that keeps writing to a damaged drive risks making the damage worse. Read-only mode is the filesystem saying "something is wrong — I'm protecting your data until you can investigate."

Common causes on the Raspberry Pi 5:

• Unexpected power loss — unplugging the Pi without a proper shutdown
• Power supply issues — insufficient current causing voltage drops during heavy NVMe writes
• Overheating — thermal throttling causing erratic PCIe behavior
• Bad blocks — physical wear or damage on the NVMe drive
• PCIe Gen 3 instability — the Pi 5's Gen 3 PCIe mode is technically out-of-spec and can cause issues on some drives

Step 1 — Confirm the Problem

First, confirm the filesystem is actually read-only and identify which drive is affected:

mount | grep "ro,"

A read-only mount looks like this:

/dev/nvme0n1p2 on / type ext4 (ro,relatime)

The ro flag confirms it. Check the kernel logs for the error that triggered the remount:

dmesg | grep -E "EXT4|nvme|error" | tail -30

Look for lines mentioning ext4_journal_check_start, Detected aborted journal, or Remounting filesystem read-only. These confirm a journal error rather than a hardware failure.

Also check systemd's persistent journal for errors before the remount:

journalctl -p err -b -1 --no-pager | tail -50

The -b -1 flag shows the previous boot — useful for seeing what happened right before the Pi went into read-only mode.

Step 2 — Attempt a Safe Remount

Before running filesystem repair tools, try the least invasive fix first — remounting the filesystem as read-write after flushing pending writes:

sudo mount -o remount,rw /

Then immediately sync any buffered writes to disk:

sync

Test if writes are now possible:

touch /tmp/test_write && echo "Write successful" || echo "Still read-only"

Step 3 — Stop Everything Before Repair

If the remount didn't stick or you want to do a proper repair, stop all processes that might be writing to the filesystem. Docker containers writing logs or database files during a filesystem check can cause additional corruption.

Stop all running containers:

sudo docker stop $(docker ps -q)

Stop the Docker daemon:

sudo systemctl stop docker

Verify nothing critical is still writing:

sudo lsof | grep nvme | grep -v "\.log" | head -20

Step 4 — Run fsck to Repair the Filesystem

fsck (filesystem check) scans the filesystem for errors and repairs them. It must be run on an unmounted or read-only filesystem — running it on an actively mounted read-write filesystem can cause data loss.

Since our filesystem is already read-only, we're in a safe state to run fsck directly. However the safest approach is to boot from a different drive or run fsck from the initramfs at boot time.

Option A — Force fsck on next boot (recommended)

This is the safest method. It tells the system to run fsck automatically before mounting the filesystem on the next boot:

sudo touch /forcefsck
sudo reboot

The Pi will run fsck during boot, repair any errors, and then boot normally. This may take several minutes depending on drive size — the Pi will appear to hang at the boot screen while fsck runs. Be patient and do not interrupt power.

Option B — Run fsck manually (advanced)

If you need to repair without rebooting, you can run fsck on a read-only mounted filesystem. First identify your root partition:

lsblk

Then run fsck on the partition (not the whole drive):

sudo fsck -y /dev/nvme0n1p2

If fsck finds and repairs errors you'll see output like:

/dev/nvme0n1p2: recovering journal
/dev/nvme0n1p2: 47382/1835008 files (0.2% non-contiguous)
/dev/nvme0n1p2: 542918/7340032 blocks

After fsck completes, reboot:

sudo reboot

Step 5 — Verify the Repair

After rebooting, confirm the filesystem is mounted read-write:

mount | grep nvme

It should now show rw instead of ro:

/dev/nvme0n1p2 on / type ext4 (rw,relatime)

Test that writes are working:

touch ~/test_write && rm ~/test_write && echo "Filesystem is healthy"

Restart Docker and verify your containers come back up:

sudo systemctl start docker
docker compose -f /mnt/data/docker-compose/docker-compose.yml up -d
docker ps

Check the filesystem error count — a healthy drive shows zero:

sudo tune2fs -l /dev/nvme0n1p2 | grep -E "Mount count|errors"

Step 6 — Investigate the Root Cause

Fixing the immediate problem is only half the job. If you don't understand why it happened, it will happen again.

Check drive health

sudo apt install smartmontools
sudo smartctl -a /dev/nvme0n1

Look for:

• Reallocated sectors — blocks the drive has marked as bad and replaced from its spare pool. A few is normal; many is a warning sign
• Uncorrectable errors — data that couldn't be read or written correctly
• Power loss count — how many times the drive lost power unexpectedly
• Temperature — NVMe drives throttle and fail early when running hot

Check your power supply

Insufficient power is the most common cause of NVMe issues on the Pi 5. The official Raspberry Pi 5 power supply (27W USB-C) is the minimum recommended. Third-party supplies that can't deliver stable current under load cause voltage drops that the NVMe controller interprets as errors.

Test under load:

sudo apt install stress-ng
stress-ng --cpu 4 --io 2 --vm 1 --vm-bytes 512M --timeout 60s

Watch for filesystem errors during the stress test:

dmesg -w | grep -E "EXT4|nvme|error"

If errors appear under load but not at idle, the power supply is the likely culprit.

Check PCIe stability

If you enabled Gen 3 speeds in Part 1, the Pi 5's technically out-of-spec Gen 3 mode can cause issues on some drives. Test by reverting to Gen 2:

sudo nano /boot/firmware/config.txt

Remove or comment out:

# dtparam=pciex1_gen=3

Reboot and monitor for recurrence. If the problem stops, your drive doesn't play well with Gen 3 on this particular board — stay on Gen 2.

Prevention

Enable automatic filesystem checks

Configure the filesystem to run fsck automatically every 30 mounts or 30 days, whichever comes first:

sudo tune2fs -c 30 -i 30d /dev/nvme0n1p2

Set up UPS or graceful shutdown

Unexpected power loss is the leading cause of filesystem corruption. Even a small UPS (uninterruptible power supply) gives the Pi time to shut down cleanly when power fails.

If a UPS isn't practical, at minimum avoid pulling the power plug. Always shut down properly:

sudo shutdown -h now

Monitor drive health automatically

Add a weekly SMART check to cron:

sudo crontab -e

Add:

0 3 * * 0 smartctl -H /dev/nvme0n1 | grep -v "^#" | mail -s "NVMe Health Check" admin@thedefaultgateway.com

This emails you every Sunday at 3am with the drive's health status.

When fsck Doesn't Fix It

If fsck reports uncorrectable errors, the drive has bad blocks that can't be repaired. At this point:

1. Back up everything immediately while you still have read-only access:

sudo mount -o remount,ro /
rsync -avz /mnt/data/ user@backup-location:/backup/

2. Run a full bad block scan to understand the extent of the damage:

sudo badblocks -v /dev/nvme0n1 > ~/badblocks.txt
cat ~/badblocks.txt

3. Replace the drive if bad blocks are found. NVMe drives don't recover from bad blocks — they get worse over time. The data you can still read today may be unreadable tomorrow.

Quick Reference

Related

• RPi5 Home Server — Part 1
• RPi5 Home Server — Part 2