With most things, I would tend to agree. It's just that with marginalized groups, that "but" is only rarely going to be something that's not just a dismissal of some part of the fundamental issues that make the group marginalized in the first place.

Not sure who has experienced what, but here in the south I have lost track of how many times I've heard things like "black people are great, but...." or "I don't have anything against gays, but..."

There's a way of thinking behind buts when applied to this kind of matter that's different from "I like shrimp, but..."