This is what it has to be, it doesn't make sense any other way.

If there was a fight over reproductive opportunities, and this valley's men took the women of another valley, there would be a mix; some from the locals, some from the other tribe.

If there was a fight over reproductive opportunities and this valley's men lost, there'd be no adult women at all.

If there was a fight for some other reason, you would assume that either a) everyone dies, or b) some mix of men and women died leaving a mix of men and women, all of whom would be locals.

The only way this works is if there's no fight; the men traded with another tribe's men.