Google's AI Search Crawler Sparks Concerns Over Fairness and Competition

Google's AI is transforming the web in ways that are both beneficial and detrimental. The company's search crawler, Googlebot, has long been indexing websites to provide users with relevant results. However, its recent strategy of using this same technology to feed its AI model, Gemini, raises concerns about fairness and competition.

Matthew Prince, CEO of Cloudflare, a web security firm that handles traffic for 20% of the web, says Google's bundling of search and scraping systems gives it an unfair advantage. He claims Googlebot sees 3.2 times more of the web than OpenAI's version and 4.8 times more than Microsoft Corp.'s.

This strategy has led to a concerning trend: AI-generated traffic is surpassing human activity on the internet. In 2026, AI agents accounted for over 57% of web traffic compared to around 42% from humans. Prince predicts this divergence will worsen in the coming years, echoing the 'dead internet theory' that machines increasingly create and consume online content.

Google's dominance of the search market makes it a gatekeeper to the web, but its exploitative ambitions are becoming more problematic with AI on the rise. The company has faced pressure from regulators, including the UK's antitrust watchdog, which ordered Google to give website owners a clear choice to block their content from being used for AI products while remaining in search results.

Google is testing a new setting that lets websites opt out of its AI answers without hurting their search rankings. This marks a shift towards greater transparency and control over content. However, fully separating Google's crawler into two distinct systems, one for indexing search and another for AI scraping, would provide more robust solutions for publishers seeking to block AI training.