The short answer.
A change to db/prod.sql, the file that prepares the read-optimized tables, used to run the whole data chain: rebuild the tables, publish them to Hugging Face, rebuild the database image, then the two images on top of it. Four such changes on 9 November each took about 1.5 hours from merge to live. The chain now looks at what the push changed. The last change of that day was live 28.5 minutes after its merge.
Where the 90 minutes went.
Measured on one run: build 6 minutes, publish 31, the database image 12, and 5 each for the two images above it. Publish spent 30 of its 31 minutes waiting for a vectors run that only a data change starts. It then committed unchanged tables to Hugging Face, which restarts the dataset viewer there for about 33 minutes.
Build only what changed.
A new first job compares the push with the commit before it. If nothing under data/ changed, the build, publish and database image jobs are skipped, and the read-optimized image is built on the database image already published for that data. When in doubt, on a manual run, a missing commit or a failed comparison, it counts the push as a data change and runs everything. Publish now waits 3 minutes for a vectors run to appear, not 30.
The proof.
We merged a one-line comment in db/prod.sql to measure it. The data run took 9 minutes 34 seconds: 7 seconds to compare, 5 minutes 11 seconds for the read-optimized image and 4 minutes 8 seconds for the all-in-one image. Hugging Face received no commit. The site reported the new build 43 minutes 39 seconds after the merge, and about 25 of those minutes were the timeout described next.
A timeout shorter than the wait inside it.
The deploy's stage job waits up to 30 minutes for Cloudflare to roll out the idle side, but the job itself was cancelled after 15. Rollouts that day took 14 minutes 17 seconds, then 6 minutes 7 seconds, then more than 15, and that deploy needed a manual re-run. The job now gets 40 minutes.
New code never meets the old database.
On 7 November one change to code and data started two deploys: one after the 5-minute checks and one after the 40-minute data build. The first staged the new code on the old database, and its warm-up requests got errors for 24 minutes until the second deploy replaced it. The first fix looked for a data build on the same commit. Its own merge, a CI-only commit that landed 3 minutes after a schema change, slipped past it. A deploy now waits for any data build still running. In both cases only our own warm-up probes saw the errors: visitors were served none that day.
Checks that count as checks.
A file in the repository, livecheck.yaml, says what production must show: the home page, a law page, the search API, the deployed build matching the main branch, and six MCP calls at once. Our own checks and deploy probes identify themselves, and the site leaves them out of its response-time charts. Before, their fast calls would have made the numbers look better than what people get, and one slow warm-up probe on a cold container fired an alert at 2,627 ms that no visitor experienced.