Ah, I'd say that when it's the government, any application of the power of state against speech has to be rigorously prevented. Not just violence.

I'm not claiming perfection for myself, I do hold contradictory beliefs (on the surface anyway, underneath they're extensions of a more complex thought process). But, yeah, there's a point at which intolerance becomes such a threat to a stable society that only eradicating it when it arises is going to allow for stability. Once you get into nazi territory, all bets are off for me.