A week ago, 55 of the plan's 307 steps were done and the first thirteen stores were writing to Postgres. Since then the rest have followed, the last of them the two that everything else depends on.

What is done

  • Every store in the system now writes to SQLite and to Postgres: 28 databases, plus a dozen small state files, such as the ones that record what has already been sent. SQLite is still the one that gets read.
  • The last wave was the core: the queue that holds incoming work and the register of every signed job. They went last on purpose, once everything around them had a clean record.
  • The soak has started. The first full day after the last wave was clean: every store matched, with no failed writes and no mismatched reads.
  • The read switch is a script now, and I've run it both ways on the sandbox. That gets a post of its own.

What turned up along the way

None of these reached a user. They are the reason the shadow phase exists.

  • A twelve-second gap. One release expected a new column, and for twelve seconds it ran against a database that didn't have it yet. A few writes to the copy failed, and a resync repaired them within the hour. The rule now: a change that only adds a column goes into the database before the code that uses it.
  • A job a third slower. Writing everything twice made one sync job slower by a third, close to its time limit. The fix made it faster than it was before the move began: the full story is here.
  • A false alarm with a real lesson. The check that runs the same reads on both databases flagged a search. The rows were identical, but the columns came back in a different order: one copy of the table had been built fresh, the other followed a table that had grown by additions over the years. Search now names the columns it wants instead of asking for all of them.
  • Counters that had fallen behind. While SQLite hands out the ids, Postgres's own id counters don't move. On the day of the switch they have to be wound forward, or the first new record collides with an old one.

Where it stands

94 of 308 steps are done. Most of that is the database half, which is about two-thirds of the way through. The container half has barely started: a few of its security gates went in early, and the rest waits on a decision about running costs.

What is next

Six more clean nights, a cross-check of the job database against the business's own records, and every weekly job seen running on both sides. Then reads move to Postgres with the rehearsed script, in a window of seconds, with SQLite kept as the copy for a few working days. After that SQLite stops being written, a week runs on Postgres alone, and only then is SQLite retired.

The earlier posts cover how the shadow writes and the nightly compare work and where this stood a week ago.