It sounds like your reasoning is similar to how a lot of people feel about the word 'moist'.

I wouldn't be happy if the descriptor for me was viscerally hurtful to myself.

I think the big difference is the world we live in. Lgbtq+ are being targeted by rightwing fucks all over the world. Talking about how you don't like the word sounds a lot like 'Im just asking what a woman is'. It's not exactly transphobic, but those phrases are often used by people trying to legitimize their hate. For example, take a look at JKRs change in rhetoric.