531
2 in a single week that is crazy (lemmy.ml)
submitted 2 years ago by to c/memes@lemmy.ml
20 comments fedilink hide all child comments
 

blob:https://phtn.app/bce94c48-9b96-4b8e-a4fd-e90166d56ed7

you are viewing a single comment's thread
view the rest of the comments
[–] 2 points 2 years ago (1 child)

Any use for programming? Preferably local hosting only?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 7 points 2 years ago* (2 children)

    I mean, if you have huge GPU, sure. Or at least 12GB free vram or a big Mac.

    Local LLMs for coding is kinda a niche because most people don’t have a 3090 or 7900 lying around, and you really need 12GB+ free VRAM for the models to start being "smart" and even worth using over free LLM APIs, much less cheap paid ones.

    But if you do have the hardware and the time to set a server up, the Deepseek R1 models or the FuseAI merges are great for "slow" answers where the model thinks things out for replying. Qwen 2.5 32B coder is great for quick answers on 24GB VRAM. Arcee 14B is great for 12GB VRAM.

    Sometimes running a small model on a "fast" less vram efficient backend is better for stuff like cursor code completion.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [–] 1 point 2 years ago (1 child)

    Would a 12g 3060 work?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 1 point 2 years ago*

    Yes! Try this model: https://huggingface.co/arcee-ai/Virtuoso-Small-v2

    Or the 14B thinking model: https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-14B

    But for speed and coherence, instead of ollama, I'd recommend running it through Aphrodite or TabbyAPI as a backend, depending if you prioritize speed or long inputs. They both act as generic OpenAI endpoints.

    I'll even step you through it and upload a quantization for your card, if you want, as it looks like there's not a good-sized exl2 on huggingface.

  • permalink
  • fedilink
  • source
  • parent