106
A 25-Person Startup Built a Chip That Only Runs One AI Model. It's 73 Times Faster Than Nvidia. (dev.to)
submitted 5 months ago by to c/technology@beehaw.org
43 comments fedilink hide all child comments
 

Taalas HC1: 17,000 tokens/sec on Llama 3.1 8B vs Nvidia H200's 233 tokens/sec. 73x faster at one-tenth the power. Each chip runs ONE model, hardwired into the transistors.

you are viewing a single comment's thread
view the rest of the comments
[–] 0 points 5 months ago (1 child)

Then it would stop being 73 times faster than NVIDIA.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 2 points 5 months ago (1 child)

    That doesn’t make sense.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 1 point 5 months ago (1 child)

    If you add levels of indirection, extra transistors and such, it would be surprising to manage to maintain the same level of performance, especially since this design seems to rely on hardwiring to achieve its speed...

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 1 point 5 months ago (1 child)

    Pretty sure the advantage is the AI directly on the chip.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •