▲ 141 ▼ Do you host your own AI? (aussie.zone) submitted 1 month ago by SuspiciousCarrot78@aussie.zone to c/selfhosted@lemmy.world 202 comments fedilink hide all child comments Do you host your own ML / AI / LLM? What do you use, and what do you use it for?
[–] SuspiciousCarrot78@aussie.zone [S] 14 points 1 month ago (2 children) Llama.cpp or death! permalink fedilink source parent hideshow 4 child comments replies: [+] tristynalxander@mander.xyz 4 points 1 month ago* (last edited 3 weeks ago) (1 child) [deleted] permalink fedilink source parent hideshow 2 child comments replies: [–] BlackLaZoR@lemmy.world 2 points 1 month ago* I use LMStudio, because it has quality of life improvements like nice GUI and huggingface search engine. Also they have Vulkan backend that at least on 7900XTX is ~10% faster than rocm (on LLama 3 8b Q4_0 it gets 115Tokens/s vs 105 on rocm) permalink fedilink source parent [–] brucethemoose@lemmy.world 3 points 1 month ago* Or exllama! Vllm, sglang, Lorax. Koboldcpp, Aphrodite, text-generation-webui, LM Studio, powerinfer, ktransformers, mlc-LLM, really whatever floats your boat. Just not ollama, specifically. permalink fedilink source parent
[+] tristynalxander@mander.xyz 4 points 1 month ago* (last edited 3 weeks ago) (1 child) [deleted] permalink fedilink source parent hideshow 2 child comments replies: [–] BlackLaZoR@lemmy.world 2 points 1 month ago* I use LMStudio, because it has quality of life improvements like nice GUI and huggingface search engine. Also they have Vulkan backend that at least on 7900XTX is ~10% faster than rocm (on LLama 3 8b Q4_0 it gets 115Tokens/s vs 105 on rocm) permalink fedilink source parent
[–] BlackLaZoR@lemmy.world 2 points 1 month ago* I use LMStudio, because it has quality of life improvements like nice GUI and huggingface search engine. Also they have Vulkan backend that at least on 7900XTX is ~10% faster than rocm (on LLama 3 8b Q4_0 it gets 115Tokens/s vs 105 on rocm) permalink fedilink source parent
[–] brucethemoose@lemmy.world 3 points 1 month ago* Or exllama! Vllm, sglang, Lorax. Koboldcpp, Aphrodite, text-generation-webui, LM Studio, powerinfer, ktransformers, mlc-LLM, really whatever floats your boat. Just not ollama, specifically. permalink fedilink source parent