the more that instances grow
If an instance grows beyond it's ability to quickly identify and ban Nazis then it's bad at moderating itself and will probably be defederated from by instance admins that are good at moderating.
If the users are driving out Nazis as you say, then it kind of implies that the admins are either OK with them being there, or didn't get to them quickly.