You're not missing anything, people either think you're defending Rogan by pointing out the insult or they just don't care about the fact that the insult is degrading for peasants.

Calling someone a peasant as a humorous jab is okay, but using it as a serious insult isn't.

Pay no mind to these overly censorious bastards; you say what you want to say. Nothing wrong with being a peasant though, I actually agree with that. But you can use it as an insult if you feel like it.