At 03:27 on October 24, the Btrfs filesystem on the Arch machine filled up.
Over the next hour, several services fell one after another: the personal site’s PostgreSQL backend Aborted, which then stayed stuck at the database system is in recovery mode; WordPress’s MariaDB exited at 04:20, leaving no space left on device behind in Docker; AstrBot exited at 03:42 with exit code 135, ENOSPC on the overlay mount directory; KDE Baloo’s indexing process SIGBUS core dumped in the same minute. AstroDX’s database also reported PANIC: could not write ... global/pg_control: No space left on device during this window, and by the time I took over it had already been rescued by the previous round of operations.
I didn’t start until October 25. Nearly a day had passed, and the first thing wasn’t to restart — it was to decide the order.
Stop first, back up next, start last
These containers were all interrupted by ENOSPC, not broken by bad configuration. Their volumes hold “half-written databases,” and a direct docker start could carry a half-finished state forward as if it were normal. But leaving them stopped forever isn’t a solution either, because with nothing writable, no check can be run.
What actually needs separating is two things:
- What state the data in the volume is in right now;
- Whether it can still serve properly after recovery.
The first must be documented before starting; the second can only be verified after starting. So the order is fixed: stop → offline backup → start as-is → verify data read-only. Not one step is “rebuild.”
Stopping has a cost. iv2-prod-db-1 was still in crash recovery at that point, health checks kept refusing connections, and stopping it meant admitting this round of recovery had failed; but not stopping it meant no consistent pgdata. First docker stop --time 45, then confirm State.Running is false.
For the backup, use the image’s own tar to run a one-shot container, bypassing host paths:
docker run --rm --pull=never --network none --read-only \
--mount type=volume,src=iv2-prod_pgdata,dst=/source,readonly \
--entrypoint tar postgres:18-alpine -czf - -C /source .The --network none, --read-only, and readonly mounts all exist to make it impossible for this read to modify the source volume. As soon as the archive is written, fsync immediately, then decompress the whole thing and read it through to confirm it isn’t a truncated gzip, then use tarfile to count members and bytes, and finally compute a SHA256 and store it in the receipt. 1,887 files, 378,342,728 raw bytes, 101,183,135 bytes after archiving.
Only at this point is starting allowed. Starting isn’t “repair”; it’s handing the state back to the original program: PostgreSQL replays the WAL itself to complete crash recovery, 30 rounds of pg_isready polling wait for it to accept connections, then query pg_is_in_recovery() to confirm it’s already a writable primary. Throughout, no pg_resetwal, no WAL deletion, no initdb, and no overwriting from an old snapshot.
MariaDB took the same path: back up wordpress_db_data, start the original container, then CHECK TABLE all 46 tables one by one — all OK.
Endpoints outlive containers
After bringing those two containers up, wordpress-db-1 wouldn’t start. The reason: when it fell over, Docker cleared the container’s network config, but the endpoint in libnetwork stayed behind, still named ep-.... The same network now had two endpoints with the same name, and the new container couldn’t get the original alias db.
The usual approach is docker network disconnect --force. The problem is that running it against an already stopped container only clears the container’s config; the endpoint stays in the network all the same — it looks like it succeeded, but nothing actually changed. So the script first writes an assertion: after disconnect, re-run network inspect, and stop if the name can still be found.
Reading Moby’s daemon/container_operations.go made it clear that force cleanup works by endpoint name. So it changed to: temporarily rename the target container so its endpoint name no longer collides, force-clear by the original name, rename the container back, and finally reconnect with the original alias. Before and after cleanup, compare the endpoints of every other container in the network item by item — they must match exactly; and the conclusion holds up: only this one endpoint was touched, the network wasn’t deleted, dockerd wasn’t restarted.
After recovery, CHECK TABLE all passed, the container was healthy, the site’s TLS check and HTTP 200, WP REST API 200. The data mount and container ID were the same as before the failure.
That all-zero journal
The full disk also left behind a bad thing unrelated to any service: under /var/log/journal there’s a file of exactly 8 MiB, all zeros, and journalctl reports Bad message when reading it. The trouble is that it isn’t “one broken file” — every user journal scan trips over it, and other logs read badly along with it.
Deleting it is the easiest, but once deleted there’s no evidence left. The wrap-up script did this:
- Use
journalctl --file ... --verifyto confirm it really is unreadable; - Check
/proc/<journald pid>/fdto confirm it isn’t currently open, avoiding moving it while it’s being read; - Copy it to a private directory,
fsync, compute a SHA256 on both sides, which must be equal; - Move it out of the journal scan directory, then
journalctl --verifya full re-verification, which must be 0; loggera marker so new logs can prove the write path works.
The copy is 8,388,608 bytes, SHA256 2daeb1f3..., readable only by root. The original bytes are preserved, but the content can’t be recovered, so the receipt says “preserved” rather than “recovered”.
Checks we didn’t fake, conclusions we didn’t fake
A few places where it’s easy to slip into writing a conclusion:
- A successful logical backup ≠ the indexes are intact. All four business databases
pg_dumpsucceeded, plus Umami and one drill database — six in total;BEGIN → CREATE 独立测试表 → INSERT → SELECT → ROLLBACKwas also done. These prove the table data can be read; they don’t prove every physical index is intact.amcheckisn’t installed in production, so no physical verification was done, and the receipt says so explicitly. - Error counters at zero ≠ the whole disk is healthy. In the end, Btrfs’s write/read/flush/corruption/generation counters were all 0, with about 17.2 GiB free. But that’s not a scrub verdict — just the counters at that moment.
- Post-hoc integrity ≠ no requests were lost. Requests that should have hit disk but didn’t during that hour of full disk can’t be found by inspecting the database schema.
- The service starting ≠ the features were verified. The AstrBot UI returned 200, but I couldn’t send a test message from someone else’s account, so I only wrote “plugin functionality not accepted”.
There’s also a timing gap worth recording separately: Certbot’s renewals for four domains started failing on the morning of 10/23, a day before the disk filled up, and they kept failing after space was freed. Counting it as part of this incident would obscure the real cause, so it’s recorded separately.
Wrapping up
All 19 business containers ended up running; every service interrupted by ENOSPC went back to its original container, original volume, and original alias. The only thing actually confirmed damaged was that journal file — the copy exists, the original isn’t on the scan path.
The cause of the disk filling up wasn’t chased down separately this round. In hindsight, the most expensive thing wasn’t the space — it was the half-day of hesitation over “what to do first”. If we’d restarted first, pgdata would be in a twice-written state, and all the later forensics would be discounted.
