Maybe I missed it being mentioned elsewhere, but I think the writeup I'm familiar fits well with this angle of the discussion. Basically, it says tolerance is a social contract that we're all born into and protected by so long as we uphold our part of the contract (by being tolerant.) If you are intolerant then you break that contract and are no longer protected by it, therefore making intolerance toward you acceptable and not a breach of the contract for others.
(Also, I agree that religions/race/etc are invalid for judging somebody's tolerance)