Every change I make to the system I look after runs about 17,700 tests before it can merge. That takes thirteen minutes. This is what those minutes are for, and the one rule I broke today.

What runs

Seven checks start together. Most are over before I have switched windows:

  • Lint and formatting: 14 seconds.
  • Strict type checks on the core: 17 seconds.
  • The dependency lock against the declared versions: 31 seconds.
  • Secret and security scans: 45 seconds.
  • Building the container image and scanning it: 1 minute 42.
  • The unit suite, 10,357 tests: 8 minutes 6 seconds.
  • The stores' own tests again, 7,341 of them, with Postgres as the main database: 11 minutes 20 seconds.

They run side by side, so the slowest sets the pace. With queueing, that is about thirteen minutes from push to a full set of green ticks.

Why I don't make it faster

The slow one is the point. The system is part way through a move from SQLite to Postgres, and that last job runs every store's tests against the database it will end up on. It catches what a laptop can't: a query SQLite forgives and Postgres rejects, a lock held a moment too long, a column whose order differs on a table that grew over time. Thirteen minutes is cheap against finding one of those in production.

Locally I run only the tests for the area I touched. Sixty tests take four seconds. The full suite on my development machine takes long enough to starve everything else running on it, so the complete run belongs in CI.

The rule I broke

Never merge on yellow or red. Today a small change merged with one red check. I had chained "wait for the checks" and "merge" into a single command, and the failing check was not one the branch requires, so nothing stopped it.

The change only touched a configuration line and some documentation, and the failure turned out to be a flaky test elsewhere: it had failed once on another branch and passed on the next three runs. But "unrelated" is exactly what you say before it isn't. Now the wait counts every check that is not a pass, and the merge only runs when that count is zero.

What the minutes are for

  • Things tests can't check. While today's run went, I confirmed a new access restriction had actually taken effect, by testing it against a resource the account had never touched. Testing one it had touched before would have shown a cached result.
  • Flaky tests are findings, not noise. I wrote down which test, which runs, and passed it to whoever owns that area, instead of pressing re-run.
  • Conflicts. Two small branches edited the same document. The second needed a rebase, which meant another thirteen minutes. Shared documents are worth editing in one branch when the changes land together.
  • This post.

What I took from it

  • The suite is slow because it checks the thing most likely to break. Keep it, and run the narrow tests locally.
  • A merge script should decide from the results, not from the wait finishing.
  • Thirteen minutes, three or four times a day, is most of an hour. Plan the next piece of work into it rather than watching the ticks.