29
Wouldn't a decentralized AI that croud sources compute power from person computers make all of their stupid data centers obsolette? (lemmy.zip)
submitted 4 days ago* (last edited 4 days ago) by to c/showerthoughts@lemmy.world
31 comments fedilink hide all child comments
 

If there was an AI model that ran locally on a PC for basic things and cloud sourced computer power for more complex tasks, all of these data centers would become obsolete in an instant. All of the invested capital and materials wasted in a flash.

Edit: lots of great counter points in the comments.

you are viewing a single comment's thread
view the rest of the comments
[–] 43 points 4 days ago (3 children)

For the local on pc part yes, if you have a decent pc. That is why they want to forbid open source weights models.

For the crowdsourced part not really. You need fast communication between your nodes for this to work, and fast here means really fast. To the point where ram to gpu on the same machine is not fast enough, so network is out of the question here.

  • permalink
  • fedilink
  • source
  • hideshow 6 child comments
    •
  • [–] 3 points 4 days ago (1 child)

    If they would just install $100K compute nodes in every home, that should be enough local high bandwidth to do the job. It will also run about $1000-$1500 per month in electricity costs, but, hey, it's about time to upgrade that grid to be able to quick charge EVs whenever we want, too, isn't it? You might even try using the compute node to make hot water and heat your home in the winter - gonna be kind of a challenge to muffle the fan noise though.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 2 points 3 days ago (1 child)

    Not close to enough for frontier models unfortunately. Ideally you'd want at least a DGX B200 server, costing about 400-500k. It has 8 GPUs each of which has a TDP of 1 kW so at least it would keep your house warm in the winter lol

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 2 points 3 days ago

    Yeah, $100K will run 8 bit quant on the latest open Chinese models. The ex-OpenAI team just released a 975B parameter 16 bit model and the price estimate I got to run that was $650K in hardware (including cooling) and 12,000kWh per month if you actually use it.

  • permalink
  • fedilink
  • source
  • parent
  • [–] [S] -4 points 4 days ago* (last edited 4 days ago) (1 child)

    Many people have fiber now. If it could delegate task fragments based on connection speeds it might be efficient enough for reasonably fast results. I think enough people would choose this over the alternative even if it means slightly slower results. But too much of a disparity and I could see people rejecting it in exchange for what might be percieved as the "premium" experience of the corporate models.

    The big irony is that if the centralized AIs get good enough, they could in theory create the dectralzied model themselves, and may infact see it as a means of self preservation, if AI actually ever gets to that level of intelligence.

    I'd even say an advanced enough AI may eventually do this with all available computer power connected to the web whether we want it to or not at this rate.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 18 points 4 days ago (1 child)

    This vastly underestimates how big and interconnected modern LLM networks are. They are 100s of GBs in size, and transmit GBs of data between layers 100s of times per second. Having to transmit any part of the intermediate state of an LLM over a 1 Gb/s link would degrade the performance of an LLM by over 1000x. Current top-of-the-line datacenter GPUs are running about 4 TB/s total memory bandwidth, a limit LLMs are already pushing up against.

    The kind of clustering you're talking about is being done: between multiple GPUs on the same machine. The slowdown of dropping to the measly 64 Gb/s speeds of the PCIe 5.0x16 bus is a huge performance hit and has to be done very carefully. The various parts of the computation are broken up and shuttled around.

    It's the combination of massively-parallel compute (in the GPU cores) and insanely-fast, fully interconnected RAM that makes generative AI possible. The load characteristics aren't conducive to parallelization.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 2 points 4 days ago

    The load characteristics aren’t conducive to parallelization.

    Not on tiny nodes separated by tiny fiber pipes... as I said elsewhere: a $100K node sucking down $1500/month in electricity could start to serve some useful loads, and provide all the hot water you could ever need.

  • permalink
  • fedilink
  • source
  • parent