How big does a minority need to be before it's "relevant" enough to be acknowledged and its members' rights respected? People with 4 or 6 fingers exist. People whose chromosomes don't match their physiology exist. People whose gender identity doesn't match their genitals exist. It doesn't matter how many of them there are, because every single one of us is a unique minority of one.

But you asked for numbers, so I'll give you some numbers.

According to this article, around 1.7% of people are intersex, meaning they have physiology that doesn't fit neatly into the common conceptions of male or female. That's close to the number of people with red hair, which is estimated to be 2% of the world population. I have never heard anyone suggest that redheads are too small a percentage to matter.

I think you were asking specifically about chromosomes, though. There's a table in the linked article that breaks down intersex conditions by cause. The first entry is "Non-XX or non-XY (except Turner’s or Klinefelter’s)". This refers to people with XY chromosomes whose bodies developed female characteristics (Swyer syndrome) and people with XX chromosomes whose bodies developed male characteristics (de la Chapelle syndrome). It does not include people with X, XXY, or XO chromosomes. (Those are the next two entries in the table.)

The estimated frequency for this condition is 0.0639 per 100 live births, equivalent to 0.0639% of population. That looks like a really low number, right? Surely not enough to be "relevant"! Except... There are 8.1 billion people on this planet. 0.0639% of 8.1 billion is 5,175,900 people, which is roughly the current population of New Zealand.

Remember, that is only women with XY chromosomes and men with XX chromosomes. If we include all intersex people that number rises to 140 million, which is nearly the population of Russia.