141
Do you host your own AI? (aussie.zone)
submitted 1 month ago by to c/selfhosted@lemmy.world
202 comments fedilink hide all child comments
 

Do you host your own ML / AI / LLM? What do you use, and what do you use it for?

you are viewing a single comment's thread
view the rest of the comments
[–] 1 point 1 month ago (1 child)

Thanks! I'll look into this. I'm a bit limited at 12GB of VRAM right now.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 1 point 1 month ago*

    A 3060?

    Exllama/TabbyAPI is still worth looking at if you are trying to run a model purely in GPU RAM. It’s easily the most VRAM efficient backend, it just doesn’t support CPU offloading (which is useful for MoEs if you have considerable spare CPU RAM) and more optimized for 4xxx and up Nvidia cards.

    And TabbyAPI has a docker container you can use. Look for “exl3” models on huggingface.

  • permalink
  • fedilink
  • source
  • parent