'You people' is typically defined by the attached action. 'You people' are the people who actually do that thing, whatever it is.
If you don't do it, 'you people' is not you.
If you do do it, 'you people' is you.
The point is not about 'who', it's an oppositional statement about the 'what', the action, and implies it's bad, or more importantly that the speaker doesn't approve.
Like: 'You people, always with the child raping.'
If you don't rape children, 'you people' isn't you.
But also, the speaker is implying that child raping is bad, or something they don't approve of, and something they don't do.