It's exhausting having to go into the defensive posture simply for saying I'm interested in a subject that is way over most people's heads.

It's probably more because the 'I'm interested in quantum mechanics' line is such a trope of an affectation that most people can't help but to roll their eyes.

I'm not saying that you aren't genuinely interested, just that that's where it may be coming from.