← Project journal

The nightly update runs again, as one pull request

The data had not been updated since 2 November. The update runs again every night at 06:40 Beirut time and keeps one pull request up to date until it is merged. Its first run removed two decrees the Lebanese University no longer publishes.

How to update the data

One pull request, updated every night.

Each run rebuilds the same branch, data/nightly, from the main branch and replaces it, so the open pull request always holds the newest night. When a night's data equals what is already published, the run closes the pull request. Before, every night opened its own pull request, and unmerged nights piled up.

What the first run changed.

The first run, on 4 November, changed 50 data files. It marked 102,808 machine-English pages of rarely read texts as not for search engines and replaced 27 links between texts with 25, following Centre pages that changed between 2 and 4 November. It removed two extradition decrees, 180938 and 186684: since 2 November the Centre has served empty pages for them while its other pages load normally, so we follow the source.

Cloudflare deletes 100 at a time.

Search by meaning keeps one vector per passage in Cloudflare Vectorize. After the first run, the sync had to delete 266 vectors, including the removed decrees' passages, and Cloudflare refused: it takes at most 100 ids per delete request, a limit its documentation does not list. Deletes now go 100 at a time. The index holds 1,547,867 vectors, one for every current passage.

Deploys that recover on their own.

Twice, one ruling page failed with a database error during a full render and stopped it. A page that fails is now tried again after 1 and 4 seconds. A failed deploy is rerun in full, because the 24 parallel jobs of a full render coordinate with each other and one cannot be rerun alone. And a journal entry like this one now renders only the journal's pages and the sitemaps that list them.

Sources

Run it yourself