Not tolerating someone ("the intolerant") makes you, to a certain extent, intolerant yourself. According to your own logic, they then should not tolerate you (the shouldn't "tolerate the intolerant").

Essentially, who is "intolerant" depends on your subjective opinion and cannot be objectively determimed, except if that person accepts all voices to be heard, in that case we could say they are very much tolerant. In any other case, it depends on your opinion.