106
A 25-Person Startup Built a Chip That Only Runs One AI Model. It's 73 Times Faster Than Nvidia. (dev.to)
submitted 5 months ago by to c/technology@beehaw.org
43 comments fedilink hide all child comments
 

Taalas HC1: 17,000 tokens/sec on Llama 3.1 8B vs Nvidia H200's 233 tokens/sec. 73x faster at one-tenth the power. Each chip runs ONE model, hardwired into the transistors.

other discussions
6
A possible hardware solution for ultra speed (73x faster than H200) self hosted small models that is not dependent on RAM (dev.to)
submitted 5 months ago by to c/localllama@sh.itjust.works
2 comments fedilink hide all child comments
 

Approach hardwires model weights into transistors, and uses older 6nm process. Targetting 70b model sizes (presumably 16 bit) by year end. It should cost much less than a 140gb card. but I don't know details.