166
The New York Times blocks OpenAI’s web crawler (www.theverge.com)
submitted 3 years ago by [M, B] to c/technology@lemmy.world
16 comments fedilink hide all child comments
 

The New York Times blocks OpenAI’s web crawler::The New York Times has officially blocked GPTBot, OpenAI’s web crawler. The outlet’s robot.txt page specifically disallows GPTBot, preventing OpenAI from scraping content from its website to train AI models.

other discussions
110
NYT Disables Open AI bot (www.theverge.com)
submitted 3 years ago by to c/technology@lemmy.world
7 comments fedilink hide all child comments
 

NYT looks like it's updated it's robots.txt file to disallow the Open AI bot from scraping it's data. Pretty interested to see if they just update their user agent string or if they'll respect it