Typically you need about 1GB graphics RAM for each billion parameters (i.e. one byte per parameter). This is a 405B parameter model.

Or you could run it via cpu and ram at a much slower rate.

At work we habe a small cluster totalling around 4TB of RAM

It has 4 cooling units, a m3 of PSUs and it must take something like 30 m2 of space

When the 8 bit quants hit, you could probably lease a 128GB system on runpod.

Can you run this in a distributed manner, like with kubernetes and lots of smaller machines?

According to huggingface, you can run a 34B model using 22.4GBs of RAM max. That's a RTX 3090 Ti.

Hmm, I probably have that much distributed across my network... maybe I should look into some way of distributing it across multiple gpu.

Frak, just counted and I only have 270gb installed. Approx 40gb more if I install some of the deprecated cards in any spare pcie slots i can find.

Ypu mean my 4090 isn't good enough 🤣😂

I've run quantized 70B models on CPU with 32 gigs but it is very slow

I regularly run llama3 70b unqantized on two P40s and CPU at like 7tokens/s. It's usable but not very fast.