198
The first GPT-4-class AI model anyone can download has arrived: Llama 405B (arstechnica.com)
submitted 2 years ago by to c/technology@lemmy.world
61 comments fedilink hide all child comments
you are viewing a single comment's thread
view the rest of the comments
[–] 2 points 2 years ago* (last edited 2 years ago) (1 child)

I regularly run llama3 70b unqantized on two P40s and CPU at like 7tokens/s. It's usable but not very fast.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 1 point 2 years ago (2 children)

    so there is no way a 24gb and 64gb can run thing?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
  • [–] 2 points 2 years ago* (1 child)

    My specs because you asked:

    CPU: Intel(R) Xeon(R) E5-2699 v3 (72) @ 3.60 GHz
GPU 1: NVIDIA Tesla P40 [Discrete]
GPU 2: NVIDIA Tesla P40 [Discrete]
GPU 3: Matrox Electronics Systems Ltd. MGA G200EH
Memory: 66.75 GiB / 251.75 GiB (27%)
Swap: 75.50 MiB / 40.00 GiB (0%)
  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 1 point 2 years ago (1 child)

    What are you asking exactly?

    What do you want to run? I assume you have a 24GB GPU and 64GB host RAM?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •