The problem with denying due process to any group is that it can then be denied to anyone simply by claiming they're a member of that group. The implications are horrifying; the government could deport anyone it wants even if they're a citizen with ample proof that they are because the opportunity to show that proof is what the law means when it guarantees the right to due process.
People are frustrated because you don't seem to acknowledge this problem.