AI Firms Offer Publishers a Simpler Way to Block Bots
Major AI companies have promised publishers a new way to block some web scraping. Today, publishers must add separate blocking commands for every bot that crawls the web to collect data.
Context
Context: AI models are trained on large amounts of web content, and news sites have long complained about bots copying their work. Robots.txt, the standard file that tells bots what not to crawl, is the common tool publishers use today.
Burning Potato score: 70%