A growing number of digital companies are finding ways to counter the bots from artificial intelligence (AI) companies that are sucking content from the web like vampires and feeding them to algorithms, changing the economic model of the internet. Indexers (or crawlers), which examine billions of websites and rank them in search engines, have been around for decades and are essential to the functioning of the Internet.
- If a chatbot makes a mistake, who pays? Company sues major US tech company over AI’s incorrect responses
- Alibaba: Chinese e-commerce giant is preparing a major overhaul of its flagship AI app to rival ChatGPT
Previously, “websites that allowed robots to access their content were getting readers in return: people who could show them ads or sell subscriptions,” explains Kurt Muehmel, head of AI strategy at Dataiku, a company specializing in data management.
But with the advent of generative AI, new robots are crawling websites and feeding chatbots such as OpenAI’s ChatGPT, and Google’s “preview” feature allows users to play content, sometimes in real-time, “without having to visit the original website,” he added. This process “completely destroys the exchange” on which the Internet functions.
One example of this change is the decline in the flow of internet users to Wikipedia due to AI bots (by 8% between 2024 and 2025), and the constant activity of AI bots will also impact the online encyclopedia’s traffic, he warned in an October blog note.
— The fundamental problem is that the new internet economy based on AI will not generate human traffic — Matthew Prince, general director of US digital service provider Cloudflare, explains on the sidelines of the Web Summit in Lisbon.
Content creators, especially news organizations, are experiencing digital infrastructure saturation and reduced advertising revenue because of these robots, a type of digital leech that can view website content hundreds of thousands of times a day.
To combat this phenomenon, Cloudflare, which controls about 20% of the network’s global traffic, implemented systematic blocking of AI bots several months ago.
— It’s like putting up speed limit signs or “no passing” signs — Prince says. — Bad bots may ignore this sign, but we can track them (…) and, if necessary, tighten these rules so that AI companies can’t get through them — he added.
- “Deathbot”: Testing AI robot that can “talk to the dead”
The measure has been applied to more than 10 million websites and has already “attracted the attention of artificial intelligence giants,” the expert said.
Meanwhile, American startup TollBit provides online publishers with tools to stop, monitor, and monetize AI robot traffic.
— The internet is a highway, and we pay a toll for these bots — says Tosit Panigrahi, co-founder and general director of the company.
The company works with more than 5,600 websites, including U.S. media outlets USA Today, Time, and the Associated Press, and allows customers to set their own prices for access to their content.
- To the sound of AI: A dancing robot stole the show at a technology event in Portugal. watch video
While the analytics tools are free for publishers, “we charge AI companies a per-transaction fee for each piece of content they access,” Panigrahi explained.
However, Kurt Muehmel believes that the AI indexer problem cannot be solved with “piecemeal or single-company measures” alone.
— This is an evolution of the entire internet economy, and it will take years — he warns.
Matthew Prince also highlighted this risk, predicting that if no one resolves the situation, “all incentives for content production will disappear.”
— a loss not only for humans who want to consume this content, but also for AI companies that need original content to train their systems — he emphasizes.
About The Author
