Why is AI expected to cause such a large increase in internet bot traffic?

Generative AI agents, when tasked with answering a user's query, visit an exponentially larger number of websites than a human would for the same purpose. For instance, an AI might scrape 5,000 websites to research a purchase a human would investigate on just five, creating a thousand-fold increase in traffic for that single task.

Online bot traffic will exceed human traffic by 2027, Cloudflare CEO says

The internet is on track to be dominated by non-human users within the next three years, according to Cloudflare CEO Matthew Prince. Speaking at the SXSW conference, Prince projected that by 2027, traffic from artificial intelligence bots will exceed that of humans. This shift is a direct consequence of the rapid expansion of generative AI, whose models require vast amounts of data from the web to function, fundamentally altering the nature and volume of online activity.

The mechanism behind this surge is the operational scale of AI agents. Prince explained that while a person might visit five websites to research a product, an AI agent performing the same task could visit 5,000 sites, generating a thousand times the traffic and server load. This represents a significant increase from the pre-generative AI era, when bots accounted for roughly 20% of traffic, mostly from established web crawlers like Google's. To accommodate this new class of automated activity, Prince suggests a need for new technologies, such as temporary 'sandboxes' that can be created on-demand for an agent to complete a task and then be dismantled.

This sustained growth in machine-driven traffic poses a distinct challenge to the web's physical infrastructure, including data centers and servers. Unlike the sharp but plateauing traffic spike seen during the COVID-19 pandemic, Prince noted that AI-driven growth is gradual but continuous with no slowdown in sight. This creates both a strain on website operators and a considerable market for infrastructure companies like Cloudflare, which provide services to manage load, block unwanted bots, and ensure site availability. Prince frames this evolution as a core platform shift, comparable to the transition from desktop to mobile, that will redefine how information is consumed online.

The internet's infrastructure, originally built for human interaction, is now being redesigned by the demands of AI agents. This creates immediate economic pressure on website owners to handle the load and a major business opportunity for platforms that can manage, authenticate, and serve this new, dominant class of machine traffic.