I think it's also worth remembering that there are always people around who are malicious, and who will always deliberately misconstrue your words for some kind of personal gain - whether from gossip, or from acting morally superior, or just to make themselves feel better by insulting others.

There's not much you can do about that, but it will always happen.

I think I you have to come to terms with it because you can't please all of the people all of the time.