I am on Gentoo for it, but everything with a decent rocm should work.

Have a look for llama-swap, that handles multi head endpoints.

Also, as you are on a big board, you can quantize yourself, as the BF16 version of qwen has only 72gb.

I will try and post a full writeup next days. But feel free to dm me, if you need some guidance on quantize or more.

I am using this fork currently: https://github.com/charlie12345/ROCmFPX

Stuff happens fast currently, so may be worth to wait a week or two ig you need something super stable, but if you are up for experimenting, that's the way to go