Materials and labor are relatively static compared to home costs. A 10% rise in housing costs was like another 15-20 grand in most cases, before housing costs exploded, factoring in inflation.
Compare that to the doubling (or more - my home is over 250% of what I paid) of home prices (tied to lot value) and the difference is stark.
Even assuming a dramatic increase in parts/labor of like 50% of those costs and you're barely hitting on the final value, all things considered.
Space is the problem and building vertically (even just 2-4 stories) is the answer.
If it helps, consider that parts and labor are generally 30-50% of home costs (assuming "normal" values) and even a doubling of that cost is less than the growth of home prices.
By far, the biggest cost increase has been lot value.