A year of fighting scrapers on my 1.5 million-page website. What I tried, how I failed, and what's working now.

you are viewing a single comment's thread
view the rest of the comments
[–] 2 points 1 day ago (3 children)

I feel you! Same issue here. I created my own solution https://angieguardian.org/. Fully open source and free to use under agpl license.

  • source
  • hideshow 6 child comments
  • [–] 2 points 23 hours ago* (1 child)

    Why would a website owner use this over Anubis. Not bashing it or anything, just curious how it differentiates?

  • source
  • parent
  • hideshow 2 child comments
  • [–] 1 point 23 hours ago

    I wanted to have high performance. See my benchmarks. I'm able to reach 150k-180k reqs/sec allowing me to handle ddos attacks as well. Even under very heavy load. Anubis will fall apart.

    Great question BTW!

  • source
  • parent
  • [–] [S] 0 points 1 day ago (1 child)

    I don't run PatronView, just sharing the blog post from them.

    I tried to access angieguardian.org, but was met with "JavaScript is required to access this site" which basically guaranteed I will never visit it again.

  • source
  • parent
  • hideshow 2 child comments