Googlebot Redefined: A Unified Crawler for Google's Internal Platform

Googlebot is not just a standalone application, but one client of Google's shared internal fetching platform. This infrastructure is used by various Google products such as Search, News, Shopping, AdSense, Gemini, and NotebookLM.

The architecture explains behavior that was once considered arbitrary, including rate limiting, retry logic, and robots.txt parsing. These functions now happen at the platform level rather than being rebuilt product by product.

Google published a plain-language overview of its crawl logic in March 2026. Discovery comes from links on known pages, sitemap files, and URL submissions through Search Console. The scheduling queue is where discovered addresses enter, with recrawl frequency following observed change.

The fetch itself runs over HTTP/1.1 by default, but can switch to HTTP/2 when crawling statistics favor it. Servers under strain can push the crawler back with 429, 500, or 503 responses. HTML that survives the fetch goes to the Web Rendering Service, a headless browser that executes JavaScript.

Google cut Googlebot's fetch limit from 15MB to 2MB per URL in February 2026, reducing the ceiling by 86.7%. The cap applies to uncompressed data and counts HTTP request headers. Anything past the line is not fetched, rendered, or indexed.