The only people with the power to decide he is unfit are all in congress, and playing a "my team vs your team" game. I'm assuming they think they'll look bad or weak or like traitors and are not willing to actually do it, or there are simply two distinct realities people live in, and in one of them he is perfectly fit to do the job.