Recent findings reveal that AI-powered search engines from companies like OpenAI and Perplexity are drastically reducing referral traffic to news sites and blogs. According to a report by content licensing platform TollBit, shared exclusively with Forbes, these AI search engines send 96% less referral traffic compared to traditional Google search. The report also highlights that AI developers’ scraping of websites has more than doubled in recent months. In the fourth quarter of last year, OpenAI, Perplexity, Meta, and other AI companies scraped websites an average of 2 million times, with each page being scraped about seven times on average. TollBit CEO Toshit Panigrahi noted the significant demand for publisher content, despite the lack of traffic benefits.
OpenAI did not comment on the report, while a Perplexity spokesperson stated the company respects “robots.txt” directives. Meanwhile, businesses reliant on search traffic, like edtech company Chegg, have been negatively impacted. Chegg recently sued Google, alleging its AI-generated summaries included content from its website without attribution, leading to a 49% traffic drop in January year-over-year. Chegg’s CEO Nathan Schultz emphasized the broken social contract between Google and publishers.
Legal battles are mounting as news publishers take action against OpenAI and Perplexity for alleged intellectual property infringement. Travel booking sites like Kayak and TripAdvisor are also concerned about Google’s AI search overviews affecting their traffic. The challenge for publishers lies in identifying and blocking AI scrapers, as many do not properly disclose their bots. Even when blocked, some AI companies like Perplexity continue to scrape sites covertly.
The issue extends beyond traffic loss, as rampant AI scraping is driving up server costs for publishers. To combat this, some publishers are pursuing direct content licensing deals with AI companies. TollBit, for instance, charges AI companies each time they scrape content, working with 500 publishers including TIME, Hearst, and Adweek. As AI continues to evolve, finding sustainable economic models for publishers remains crucial. — news from Forbes