EDIT: no, I don't sympathize with nazis (neither I sympathize with those who call everyone nazi when they're losing an argument ;)
If it were that simple, then it would be fine.
But the point is, people just start to label anything that whiffs of a different opinion as "intolerance".