That's basically it afaik. Chess has been around long enough that it was originally man's game, then they started letting women compete but only against other women. The reason why I don't think there's been a significant challenge to it is that with physical competitions often being segregated, I don't think anyone really stopped and thought about the fact that the segregation is pointless. I think they were just like, "competitions are usually segregated by gender, makes sense for chess to be segregated" without stopping to think about why competitions are usually segregated.
However, I'm not a huge chess person. I liked it as a kid but it doesn't even remotely appeal to me anymore. So my info might be wrong, and if there are any chess historians who want to correct me, go for it!