"Ostracizing" the accuser is generally voluntary. There is a difference between "I'm not comfortable working with this person" and leaving, and everyone coming to you and saying "Get out".
The latter is fairly rare to happen to accusers, but it's expected for the accused.