196
Smaug-72B-v0.1: The New Open-Source LLM Roaring to the Top of the Leaderboard (huggingface.co)
submitted 2 years ago by to c/technology@lemmy.world
60 comments fedilink hide all child comments
 

Abacus.ai:

We recently released Smaug-72B-v0.1 which has taken first place on the Open LLM Leaderboard by HuggingFace. It is the first open-source model to have an average score more than 80.

you are viewing a single comment's thread
view the rest of the comments
[–] 2 points 2 years ago (1 child)

I'm currently playing around with the Jan client, which uses the nitro engine. I think I need to read up on it more, because when I set the ngl value to 15 in order to offload 50% to GPU like the Jan guide says, nothing happens. Though that could be an issue specific to Jan.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 2 points 2 years ago (1 child)

    Maybe 50% GPU is already using too much VRAM and it crashes. You could try to set it to 0% GPU and see if that works.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 1 point 2 years ago (2 children)

    I may need to lower it a bit more, yeah. Though when I try to to use offloading, I can see that vram usage doesn't increase at all.

    When I leave the setting at its default 100 value on the other hand, I see vram usage climb until it stops because there isn't enough of it.

    So I guess not all models support offloading?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •