Sure, if people suddenly switched languages and then laughed in my face, I would feel bad.

But if it's like the other comments in this post, and it's a couple having a quick word about a private matter, I wouldn't mind. It's not like I should be a part of that conversation