So, that puts the cutoff at 20 million. Most people don't know anyone that rich, but if you ask someone with just a few million, they might, and they'll probably be against it. And guess what, they're going to know a lot of people with just hundreds of thousands, and those are the white picket fence people who vote and donate the most.
It's slanted, not the original question, in that it passes the buck to some distant third party. Politicians don't have the luxury of just ignoring that they have one big interconnected populace to satisfy, and none of them are an island. They raise taxes, and the big narrative is that they raised taxes for no reason, and that can cost an election.
It has nothing to do with it being a good idea or not. It's just politically difficult, and not because of anything spooky. (And FWIW OP was about income taxes; wealth taxes have additional complications)