I like tech but Ai is really spooky.

Having said that, Ai is making a mess of web traffic among all the other things.

How are people handling the massive surges in users that are apparently Ai crawler bots and whatever else, acting like DDoS attacks without being intentionally malicious (assumption), the whole thing is a real headache.

There are many facets to this topic, one also being the proliferation of Ai into everything, and Serach being an obvious one, where the top result is now an AI readout, that is and will kill clicks down the SERP page and changes user behaviours, causing revenue loss already I am sure.

Removing the need to think. Making knowledge access even more frictionless to the point that broad scale cognitive atrophy occurs, a net result, potentially further concentrating knowledge into smaller more powerful control groups (due to the current horsepower required). It's eating our energy supply too. Essentially we are at the BORG stage of the game.

As all should be rather familiar with now, it goes well beyond web traffic, but web traffic is the entry point here because this is where NodeBB's bread and butter is.

I flagged this in multiple places, years back at this point, with multiple warnings, adoption was done in haste, reaction was negative (as usual) and there was no due regard to Pandora's tsunami people thought they could surf safely, but the adoption of the Ai into anything and everything is what has allowed the Ai bot swarms to proliferate. it's a vicious cycle and maybe worse.

I could see and rationalise years ago that Ai itself is a parasitic technology and works in net deficit, it functions at a permanent net-negative (I understand many tech do not see it like this, feel free to posit your rebuttle). Why this is not glaringly obvious, is another point, but a fundamental one at that. There are other fundamental points. This is one that get's less airtime afaict.

Well have at it if you think you can add value to help the whole, server tips, what do you do to mitigate the server load, protect your content from the LLM plunder bots, etc. etc. - try stay on point in web traffic terms, as hard as it it due to the vastness of the consequences and implications before us, and Merry Christmas too while I'm here too! :santa: :christmas_tree:

you are viewing a single comment's thread
view the rest of the comments
[–] 1 point 8 months ago (2 children)

@omega nice! I shall give it a try... Wonder how well it works with federation

  • source
  • parent
  • hideshow 4 child comments
  • [–] [S] 1 point 8 months ago* (last edited 8 months ago)

    @Julian To note, it uses up one of the 5 free WAF/Security rules slots.

    Having looked into it a bit more there seems to be more bot control rules you can invoke with the paid tiers.

    On the free one you have - (cf.client.bot) true/false

    And then the Ai crawled control (managed via own menu option), which nicely groups the various types of AI crawlers which you can allow/block.

    This is 2 slots used, you need to use your remaining 3 wisely.

    It's amazing how much traffic you're going to block with a wildcard approach (if applicable) to blocking all .php request, and I have had to play around with the order a bit to allow the goodbots as such in. I'm not totally sure how the rules fire in order is 100% robust. It might be 99.x% robust. I would like to hear others thoughts/experiences on this.

    I would urgently encourage anyone who is using CF to look at the security analytics, and look at the top countries overall.

    China, Singapore are the top offenders in this user surge wave. Incredible volumes of multi second users can swarm out of nowhere and really eat up server resources. You may find the only solution is a total block but this would be my starting point.

    However, interactive challenge may be as good or nearly as good (but I am too paranoid to be totally convinced, I tested it and I am still not sure and reverted to some hard blocks).

    I think the managed challenge may be out of it's depth here when dealing with origin offenders.

    It is very tricky. However, drilling down using the "filter" option in CF has been a really useful visual aid. I never had to do as much looking until this China/Singapore wave started ramping up.

    Log search is paid feature. You can however apply various filters combined to analytics.

  • source
  • parent