This stuff is a social contract - if people are free to break the social contract and be intolerant or fuck with peoples' freedoms, it harms peoples' freedom to tolerate that behaviour.

Your argument is akin to saying that using force to stop someone that's currently committing a mass shooting justifies that mass shooting - it's moronic.