57
Should we block OpenAI from scrapping the server? (platform.openai.com)
submitted 3 years ago by [M] to c/lemmy_ca_support@lemmy.ca
21 comments fedilink hide all child comments
 

Right now, robots.txt on lemmy.ca is configured this way

User-Agent: *
  Disallow: /login
  Disallow: /login_reset
  Disallow: /settings
  Disallow: /create_community
  Disallow: /create_post
  Disallow: /create_private_message
  Disallow: /inbox
  Disallow: /setup
  Disallow: /admin
  Disallow: /password_change
  Disallow: /search/
  Disallow: /modlog

Would it be a good idea privacy-wise to deny GPTBot from scrapping content from the server?

User-agent: GPTBot
Disallow: /

Thanks!

you are viewing a single comment's thread
view the rest of the comments
[–] 2 points 3 years ago

Strangely, = works the same as == with nginx. It's a very strange config format...

https://nginx.org/en/docs/http/ngx_http_rewrite_module.html#if

  • permalink
  • fedilink
  • source
  • parent