It hasn't been quantized, then. I've run 70B models on my consumer graphics card at a reasonably good tokens-per-second rate.

I'm curious how local generation goes with potentially dedicated AI extensions using stuff like tensor cores and their own memory instead of hijacking parts of consumer GPUs for this.