As soon as men try to organize and speak out we get called sexist.

That's simply not true. We have at least one counselling centre in our city that is "boys/young men only" and several "men only" self help groups. I've never heard them being called sexist, on the contrary people generally agree that this is a good thing and we need more of this. And they are certainly not forced to include other genders.

There are obviously not enough initiatives like these. But a blanket statement like yours is false and if you make the claim that men are regularly getting called out as sexist for forming liberal safe spaces you should provide some sources (I'm not denying that it happens, it's just not something I've experienced).

Just look at the boy scouts (ignoring the pedophiles)

The goal of boy scouts wasn't to provide a safe space to explore gender identity or emotions or anything like that. There was no reason to exclude other genders.