45
Self hosting an LLM for research (lemmy.world)
submitted 2 years ago* (last edited 2 years ago) by to c/selfhosted@lemmy.world
22 comments fedilink hide all child comments
 

I am a teacher and I have a LOT of different literature material that I wish to study, and play around with.

I wish to have a self-hosted and reasonably smart LLM into which I can feed all the textual material I have generated over the years. I would be interested to see if this model can answer some of my subjective course questions that I have set over my exams, or write small paragraphs about the topic I teach.

In terms of hardware, I have an old Lenovo laptop with an NVIDIA graphics card.

P.S: I am not technically very experienced. I run Linux and can do very basic stuff. Never self hosted anything other than LibreTranslate and a pihole!

you are viewing a single comment's thread
view the rest of the comments
[–] 1 point 2 years ago

The easiest way to run local LLMs on older hardware is Llamafile https://github.com/Mozilla-Ocho/llamafile

For non-nvidia GPUs, webgpu is the way to go https://github.com/abi/secret-llama

  • permalink
  • fedilink
  • source
    •