I admitted there are edge cases.
Then it's not binary.
When you flip a coin, there is a chance that it will land on the side, yet we still use a coin flip for a 50:50 probability scenario because it is close enough.
Absolutely. For day to day life, "there are two outcomes" is safe way to describe coin flips. But given that a coin landing on its side can happen, it's not a binary system. It only becomes binary when we ignore the edge cases. Just like sex...
And that's before we get to the point that there isn't even a single definition of sex that accounts for all scenarios. People can change their legal sex, people can change their morphological sex, "genetic sex" isn't foolproof, as it doesn't always correlate with morphological sexual characteristics, or even gamete production.
Calling sex binary is either a generalisation, or something you want to be true. At no point is it reality of the situation though...