We only have a different word for it because of oranges. Prior to that it was just "red."

It would be like if brick-red became so commonly used that people referred to the color as "brick" and people wondered which came first.