That gender is at least partially a social construct and even if you have XY chromosomes and you've been brought up as a woman you shouldn't be forced to live as a man in case you do a chromosomal test and it turns out you have XY chromosomes.

Imagine the opposite - you're an XX man with a penis and you've lived as a man your whole life and then you do a random chromosomal test and somebody tells you "you see I know you have a penis and I know you're fine being a man but your chromosomes says you're "female" so we have to chop off your dick and you have to grow titties so you can fit into the binaries we've established because it's too hard for others to believe you're a man"

The idea that the use of the term "Man" is at all related to chromosomes, as opposed to how you interact with society. If you were talking about biological sex that'd be a cogent view, but even there there's a ton of nuance.

https://en.m.wikipedia.org/wiki/Sex-determining_region_Y_protein

More important (but still not all-deciding) than the XY phenotype is the presence or absence of SRY, which depending on mutations and faulty crossovers can be present on either X or Y

To oversimplify, every human has an X chromosome and it is necessary to survive. The Y chromosome is optional and codes for some additional traits that make a person male. Chromosomes aren't foolproof and genes and in some cases entire chromosomes can essentially be turned off. However, since we have two of each chromosome this is often not a problem. So, if parts of the Y chromosome turn off, you potentially turn off the mechanisms that make a person male. In which case the person would remain the default of female.