I’ve never understood this.
I’m a Christian, but I don’t make the connection on purposely avoiding greeting someone’s religious celebration.
I have no problem saying ‘happy Diwali’ to my Hindu neighbours. Do you think they feel uncomfortable saying ‘merry Christmas’ to me?