ShapBot was blocked by mistake.
ShapBot is the crawler behind Parallel's AI search API. Since 13 October our site Worker answered it with 429, a line copied from the setup of the old single server, and a Cloudflare rule answered it with 403. We removed both on 4 November.
Search, the API and MCP are open to them.
robots.txt now gives each named AI crawler its own group that allows everything except our internal review pages: search, /api/v1/, /mcp and addresses with query strings. That covers GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, CCBot, ShapBot, DuckAssistBot and MistralAI-User. Search engines keep the old rules and stay out of /api/ and /search/.
The same limit for everyone.
Opening the doors did not remove the rate limit. Search, the API and MCP still allow every client, person or crawler, 30 requests per 10 seconds from one address.
Bing hears about every change.
ChatGPT search reads Bing's index, and 749 of our 1,289 page views on 3 November came from ChatGPT. On 31 October Bing had indexed 3,768 of the 410,503 pages in our sitemaps. On 3 November we sent all 410,503 to IndexNow in 41 batches, and every batch was accepted. Each render now sends the pages it changed to Bing, Yandex, Naver and Seznam.
Topic pages: open to Bing, hidden from Google.
Since 2 November our 30 topic pages had asked every search engine not to index them. Now only Google is asked, with a googlebot meta tag. Bing, which ChatGPT and Copilot search, can index them. They stay out of the sitemaps, which Google reads.