Any script kiddie can scrape the entirety of Lemmy, with the exception of direct/private messages. robots.txt is merely a request, with no enforcement capability.