Yeah, this is common belief but it's not completely true. The egg is fertilised by either an X or Y sperm, and will develop for a couple of months as a kind of intersex thing.

What would be ovaries develop into testes, but for example, no fallopian tube of uterus would have been developed.