141
Do you host your own AI? (aussie.zone)
submitted 1 month ago by to c/selfhosted@lemmy.world
202 comments fedilink hide all child comments
 

Do you host your own ML / AI / LLM? What do you use, and what do you use it for?

you are viewing a single comment's thread
view the rest of the comments
[–] 1 point 1 month ago (1 child)

Bought b70 with egpu enclosure and usb4 connection wasn't really planning to actually run anything but now ended up with llama.cpp with openwebui - kids/parents want to/have to use chat, might as well provide local solution than them using industry options. Also started with ollama and Gemma 4 26b a4b - asked it to write script to setup llama.cpp in container.

  • permalink
  • fedilink
  • source
  • hideshow 2 child comments
    •
  • [–] [S] 1 point 1 month ago* (last edited 1 month ago) (1 child)

    Yeah, I've heard the B70 is good bang for buck. My kids love using chat GPT to generate images and I'm aware that there are some really capable local models that can do that as well now - B70 should make short work of it.

    That may be something for me to look at later on if I decide to keep self hosting.

    OTOH, I'm also aware that I may end up building something that they don't actually use. Been there, done that, and I don't want to do it again.

    Actually, on that topic, one interesting use case for me is my youngest one wants to have a YouTube channel.

    So obviously, I'm not going to let her become a YouTuber, but what I'm thinking of doing is providing her my old phone (properly locked down) so that she can video record clips of what she wants.

    Then - have those clips sent automatically to our jellyfin server so it appears like a channel. Code a fake YT plugin so that AI can do likes, positive comments etc.

    It's... work. I dunno...maybe a good enough AI can vibe code the entire project for me.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 1 point 1 month ago

    I've been testing coding capabilities a bit (mostly scripts - so that work done by ai is reproducible).

    1. Context size is very much a required thing along with model capabilities.
    2. Local model can generate good enough script in one shot - but reiterations r crazy
    3. Use git to keep files tracked (easy to revert) and make a modular script - main script calls function a, function b etc where functions are relatively self contained (no need to look at others) when u need new capability add function c
    4. If something need changing- try to do it urself (unless it's whole architectural change, then just start new project)

    As for image gen + chat . as long as chat model + context + image model fits, u should be fine.

  • permalink
  • fedilink
  • source
  • parent