351
The AI bubble is so big it's propping up the US economy (for now) (www.bloodinthemachine.com)
submitted 1 year ago by to c/technology@lemmy.world
124 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[–] 7 points 1 year ago (2 children)

I did not understand half of what you've written. But what do I need to get this running on my home PC?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [–] 5 points 1 year ago* (last edited 1 year ago)

    I am referencing this: https://z.ai/blog/glm-4.5

    The full GLM? Basically a 3090 or 4090 and a budget EPYC CPU. Or maybe 2 GPUs on a threadripper system.

    GLM Air? Now this would work on a 16GB+ VRAM desktop, just slap in 96GB+ (maybe 64GB?) of fast RAM. Or the recent Framework desktop, or any mini PC/laptop with the 128GB Ryzen 395 config, or a 128GB+ Mac.

    You’d download the weights, quantize yourself if needed, and run them in ik_llama.cpp (which should get support imminently).

    https://github.com/ikawrakow/ik_llama.cpp/

    But these are…not lightweight models. If you don’t want a homelab, there are better ones that will fit on more typical hardware configs.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 4 points 1 year ago* (last edited 1 year ago) (2 children)

    You can probably just use ollama and import the model.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [–] 3 points 1 year ago (1 child)

    I'm running Qwen 3B and it is seldom useful

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 2 points 1 year ago* (1 child)

    It's too small.

    IDK what your platform is, but have you tried Qwen3 A3B? Or smallthinker 21B?

    https://huggingface.co/PowerInfer/SmallThinker-21BA3B-Instruct

    The speed should be somewhat similar.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 1 point 1 year ago (1 child)

    Qwen3 8B sorry, Idiot spelling. I use it to talk about problems when I have no internet or maxed out on Claude. I can rarely trust it with anything reasoning related, it's faster and easier to do most things myself.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 3 points 1 year ago*

    Yeah, 7B models are just not quite there.

    There are tons of places to get free access to bigger models. I'd suggest Jamba, Kimi, Deepseek Chat, and Google AI Studio, and the new GLM chat app: https://chat.z.ai/

    And depending on your hardware, you can probably run better MoEs at the speed of 8Bs. Qwen3 30B is so much smarter its not even funny, and faster on CPU.

  • permalink
  • fedilink
  • source
  • parent