When one says something, there's always some more information to be gleaned from context. Like yeah, I agree that some people use the phrase "it's just a joke" to be a dipshit, but I'd only bring it up if there was someone is being a dipshit.

By bringing it up in that context, it's giving an implication that you are agreeing with them, and it gives you the bonus of being able to wriggle out and go "oh actually I'm not on their side" once you get criticised.