My ideological opponents are already silencing speech. I gain nothing by ceding that tool solely to them.
As long as fascists exist they must be silenced. When they seize power, they will not do you the courtesy of allowing you to speak just because you let them. It's naive to think otherwise.