if she were XY in this scenario, wouldn't that mean that she'd pass that trait along to her kids as well?

I could be wrong, but I don’t believe XY females (Swyer syndrome) produce eggs and thus cannot bear their own children.

But a colorblind XX woman who can bear children would give birth to colorblind sons and, if the father of the child is also colorblind, colorblind daughters.