Try flipping your process. Instead of working from the full list and taking things out, start from an empty list and add stuff in. If there isn't a good enough reason for it to be there, don't put it in. And if this leaves you with just humans, that's fine.
I'm not removing githyanki from my game. Githyanki were never in my game.