656
The Rule (lemmy.ml)
submitted 2 years ago by to c/196@lemmy.blahaj.zone
60 comments fedilink hide all child comments
 
you are viewing a single comment's thread
view the rest of the comments
[–] 4 points 2 years ago (1 child)

I'm not sure what "FP16/FP8/INT4" means, and where would GTX 4090 fall in those categories, but the VRAM required is respectively 810Gb/403Gb/203Gb. I guess 4090 would fall under the INT4?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 7 points 2 years ago*

    They stand for Floating Point 16-bit, 8-bit and 4 bit respectively. Normal floating point numbers are generally 32 or 64 bits in size, so if you're willing to sacrifice some range, you can save a lot of space used by the model. Oh, and it's about the model rather than the GPU

  • permalink
  • fedilink
  • source
  • parent