196
Smaug-72B-v0.1: The New Open-Source LLM Roaring to the Top of the Leaderboard (huggingface.co)
submitted 2 years ago by to c/technology@lemmy.world
60 comments fedilink hide all child comments
 

Abacus.ai:

We recently released Smaug-72B-v0.1 which has taken first place on the Open LLM Leaderboard by HuggingFace. It is the first open-source model to have an average score more than 80.

you are viewing a single comment's thread
view the rest of the comments
[–] 11 points 2 years ago (1 child)

It's been discovered that you can reduce the bits per parameter down to 4 or 5 and still get good results. Just saw a paper this morning describing a technique to get down to 2.5 bits per parameter, even, and apparently it 's fine. We'll see if that works out in practice I guess

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 2 points 2 years ago* (2 children)

    I'm more experienced with graphics than ML, but wouldn't that cause a significant increase in computation time, since those aren't native types for arithmetic? Maybe that's not a big problem?

    If you have a link for the paper I'd like to check it out.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [–] 14 points 2 years ago

    My understanding is that the bottleneck for the GPU is moving data into and out of it, not the processing of the data once it's in there. So if you can get the whole model crammed into VRAM it's still faster even if you have to do some extra work unpacking and repacking it during processing time.

    The paper was posted on /r/localLLaMA.

  • permalink
  • fedilink
  • source
  • parent