Fun fact: instead of going invisible, you can mark games as private.

They won't show up as recently played, on your profile, and other people won't see you're in-game, but at least you're not announcing to the world what you're doing.

Conversely, if you're annoyed by others, you can turn those notifications off.