The save bodies have always lived in a Windows storage service, right alongside the game sources in the same bucket. The reason to move them isn’t complicated: that backend can only run on a Windows box with a GUI, and every save of every user is a complete copy, so the same bytes get stored over and over.
The goal is to move the bodies into a shared CAS in object storage: the object name is just <sha256>.bin, across users and across works, keeping only one copy of identical bytes. The business database keeps deciding who owns which copy; the bucket only holds bytes.
Six product files changed, plus a migration nobody wants to do a second time.
Three non-negotiable constraints
- The object name is the content, not the owner. The key must not contain an owner or a work id, otherwise deduplication is gone.
- Deletion is a no-op. Deleting a save only deletes the ownership record in the database; the object stays — because there’s no telling who else references it. No physical GC this round; garbage is preferable.
- Plaintext cap of 32 MiB. Name by SHA256, then verify against the provider’s native SHA1; same name but different content, or a digest mismatch after reading, both fail — never overwrite.
Two more are procedural: credentials are handed over in memory, never landing in source, logs, or the local machine; capacity is explicitly reported as null rather than inventing a fake local free space.
These were made to go red in tests before any code was written: identical bytes across two owners upload only once, deleting one owner leaves the shared body alone, a bad body doesn’t overwrite, a malformed hash is rejected before IO, a scope or read-only key fails before the first upload, 429/503 don’t auto-retry, a digest mismatch on read-back is rejected. After 37 targeted items passed, edge cases were added, and 2 of 22 went red: a 401 from CF shouldn’t clear B2 auth; abnormal non-empty metadata shouldn’t be treated as a missing object. Fixed to 42/42.
The read path gets measured too
After writing, read concurrency was saturated by downloads. So a regular path was added: an existing CAS is read through Cloudflare via a fixed URL — no Authorization, no Cookie, no Range, no query parameters, no following redirects — and on read-back the size, SHA256 and native SHA1 are checked. A version lacking native SHA1 evidence can’t be trusted to the cache; it’s read once from origin by that exact fileId, and the verification record binds only to that version.
An in-process body LRU of 32 MiB / 128 entries / 60 seconds returns an independent Buffer and never touches disk. Existence is re-checked against B2 metadata every time — a cache hit may still be a stale version at the source, or even already deleted.
Offline counters show the effect: re-reading an existing object is 1 CF body fetch and 0 native downloads; 8 concurrent reads coalesce into 1 metadata lookup and 1 CF request. But these are synthetic requests, not a production hit rate.
Cloudflare blocks the way first
The first acceptance run over the real path: auth 200, metadata 200, body 403.
Not permissions. The exact CF-Ray events in the receipt say it plainly: source=botFight, ruleId=bot_fight_mode, action=managed_challenge. The egress is IPv6 2a01:4f9:c013:773a::1, CF took it for a bot, served a challenge page, cf-mitigated: challenge, content-type: text/html, with the body being Just a moment.
There are three ways around it, at different costs:
- Turn off Bot Fight Mode zone-wide — too broad;
- Spoof a browser UA — may stop working tomorrow;
- Add an IP Allow — the smallest, but a free IP Access allow applies to the whole
saop.cczone by IP, skipping WAF and rate limiting for that IP too, with no way to scope it to the resource host.
A plain WAF Skip doesn’t affect BFM, which was confirmed at the time too. IP Allow was chosen in the end: one rule, its id recorded in the receipt, and rollback is deleting that one id. CF echoed the IPv6 back in expanded form, the original script’s text comparison failed once, no POST was resent; a separate read-only GET verified address equivalence, and the rule count matched 0 → 1.
Changing security config itself went beyond the original “don’t touch CF” constraint, so it waited for explicit approval; production stayed on Windows the whole time.
Interrupted uploads are not replayed
The migration covers 33 objects, 1,994,672 bytes, single concurrency, 5-second intervals, stopping immediately on 429 or 503.
Then it hit wall after wall:
- r1 failed on a file access permission up front, without sending a single B2 request;
- r2’s first upload returned 200, then cold auth plus the 5-second wait burned through the 20-second timeout;
- r3 moved auth earlier and the wait ahead of the operation; 6 items passed, then
UND_ERR_SOCKET; - r4 added
Connection: closeto the migration’s HTTP; 20 items passed, and after a 40 KB object read back exactly with 200, the body timed out.
(This is part 1 of 2 of the same post.)
None of the four rounds hit a 429/503. Each round used a fresh task id and never replayed an old task, but objects already uploaded were reused via the existing CAS rather than re-uploaded. There was a gap that had to be closed: the two objects whose upload had been cut off halfway only had a 200 header, with no proof of a complete body; later, in a dedicated gate, full read-back receipts were obtained for them by exact fileId.
Timeouts weren’t just scaled up blindly. The new operationTimeoutMs is an optional integer ranging from 20 to 120 seconds; production still defaults to 20 seconds, and only migration tasks explicitly pass 120. The old implementation failed two tests first; the new 44-item suite is all green.
Freezing Is Not “Pausing for a Moment”
A real cutover has to stop writes and re-verify everything, in a short window. What happens inside that window:
- Re-read every hash in full on Windows and echo this round’s nonce — not just re-stamp the old manifest;
- Run 41 objects through the offline gate and check reference coverage in the production database clone;
- Confirm the manifest is a subset of the existing source manifest.
It took four attempts to settle. The real freeze lasted 265.575 seconds and covered 41 objects, 94 files / 1,996,140 bytes, 76 owner rows, 24 unique hashes, and 100 refs; the switch happened only after everything matched. The live compose was changed to save-cas-b2-20261115-r2, and the only thing that changed was WG_BLOB_STORAGE=b2-cas.
Two layers of protection wrapped the freeze: a 420-second absolute alarm and an independent Postgres watchdog. Releasing both the session and lock counters to zero depended entirely on independent confirmation — missing either one meant stopping.
Three protocol gaps were also closed along the way: assertFreeze used to pass by default with empty arguments, so a caller that forgot to pass a checker still got through; a missing-checker test case was added and failed first, then the code was changed to require an explicit function, ending at 8/8.
Rollback Cannot Just Restore the Database
This is the single most important point of the whole round.
After switching to B2, running restore against the old database or swapping the old compose back would lose the objects uploaded after the switch — the database travels back in time, but the bucket doesn’t. The right move is the reverse: freeze again, read the newly added B2 references back from B2 by exact hash, write them back to Windows, verify them fully, realign both sides, and only then switch back to the old backend.
The rollback script has a few hard rules: prefer reusing the full SHA proofs from this freeze, and read from B2 only for objects missing on Windows; on lock loss or a 429/503, stop immediately without retrying. Before release there was already a full reverse rehearsal with exit 0, and the lock was confirmed released.
Several release attempts also left two lessons behind: in one, the package was missing a static copy — the version number changed but the module body didn’t — and the digest gate caught it on the spot and automatically rolled back to the old code, restoring the live config SHA to its original value; in another, the checker mistook private, no-store on a trusted module for “must contain no-transform”. Neither counted as a success, and both original logs were kept.
Acceptance Boundaries
After release, 31 monitoring points passed over roughly 15 minutes, and web/manager fatals were both 0. The cloud-save group ran real upload, download, history restore, idempotent submit, and two pages of authoritative WebSocket events once each in Chromium, Firefox, and WebKit, always with a freshly created synthetic identity.
The following “doesn’t count” items were still written down:
- Only synthetic accounts were used; no real user saves were involved;
- The game-origin CDN was blocked throughout, and the four engines only got a loading-screen smoke test;
- It isn’t called a full three-engine, four-browser matrix, nor a clear;
- The summary script for the WebKit round still asserts 3 browsers while only 1 item actually ran; the original exit 1 was kept, and a separate read-only summary was bound to the original JSON’s SHA without modifying the old receipt.
Wrapping Up
The original Windows archives, old releases, old assets, and original accounts were all kept. The migration payload only passed through Debian and Windows in memory; it never landed on the Mac, nor on the production machine.
What’s most worth remembering now isn’t how CAS deduplicates, but the line written into the plan: without reverse realignment, you can’t just roll back. A migration that treats “rollback” as a button is already in debt the moment it cuts over.
