Growth in this context isn't measured by physical objects; it's an increase in subjective value. So we've experienced high growth since the stone age, even though we've gotten almost no new stones.

While I agree in practice (and the resultant harm is real), in theory mathematical limits exist. (And this is without appealing to the idea that humans might eventually not be limited to a single planet.)