You could use the 35B MoE model, tune it a little bit and get much better results. I have a 5060 ti and 70-80 tok/s are the norm
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
replies:
You could use the 35B MoE model, tune it a little bit and get much better results. I have a 5060 ti and 70-80 tok/s are the norm