← Project journal

Article pages now describe themselves to search engines

On 4 November each article page got a description of its own, 92,053 pages that copied another left the sitemaps, and unchanged pages began answering 304.

The short answer.

Two audits of the site on 3 and 4 November found three things a search engine would trip on. Every article of a law had the same description. Tens of thousands of article pages repeated the page of their own short text. And a crawler that already held a page was sent the whole page again. All three were fixed on 4 November.

Every article had one description.

The description is the text a search engine shows under a page's title. Ours listed facts about the law first and the article's words only if room was left, which never happened. So every article of a text shared one description, apart from its number. An Arabic article page now says which article it is, then its subject, then the article's own words.

Subjects from the source's own headings.

The subject is the chapter heading the Lebanese University's text places above the article, with its numbering removed. Article 655 of the Penal Code reads "الاحتيال وسائر ضروب الغش" and Article 547 "القتل قصدا". 43,379 of 389,078 articles have one, 11%, mostly in the codes. No model wrote any of it.

92,053 copies left the sitemaps.

A text of one to three articles shows every article whole on its own page, so each of its article pages repeated that page. Those article pages now ask search engines not to index them and are out of the sitemaps: 289,082 article addresses became 197,029. Readers and links still reach them. Texts of four or more articles are unchanged.

Short descriptions, completed honestly.

A site audit flagged 989 list pages whose descriptions were 72 to 97 characters. Descriptions under 110 characters now gain one of the site's own short lines, whole, and only if it fits within 160. None is cut off with an ellipsis any more.

Unchanged pages answer 304.

A crawler that sent back a page's exact version tag still got the full page. A page served from the Worker's cache skipped the comparison, and the comparison was exact while Cloudflare marks a tag as weak when it compresses. Pages now answer 304 Not Modified and carry a Last-Modified date, so a crawler learns in one small request that a page has not changed.

The home page shows before its script.

The title and search box on the home page waited for a script before they appeared. On a phone, Lighthouse measured 5.8 seconds until the largest element showed, 92% of it waiting to render. They now show without the script.

Sources

Browse the laws