You don't even have to get high-mindedly preachy, it's a bad take on its own terms, since it hinges on the notion that the population at large are all "dumb" and therefore deserve bad things.

Growth in this context isn't measured by physical objects; it's an increase in subjective value. So we've experienced high growth since the stone age, even though we've gotten almost no new stones.

While I agree in practice (and the resultant harm is real), in theory mathematical limits exist. (And this is without appealing to the idea that humans might eventually not be limited to a single planet.)