MSCHF made a device called an Alexagate, which jams the microphones using ultrasound and is turned on and off by clapping.
It's a bit expensive, though ($100).
Otherwise, as you mentioned, you can use DNS to block the tracking. NextDNS has a built-in blocklist specifically for Alexa.