well i might be wrong about this but last time i checked all infrastructure of modern society (and i assume that includes houses and cities) takes up about 3% of our land. That is not so much, if you compare it to agriculture which takes up much more (i forgot the exact number). So i would argue that gardens aren't the biggest problem.

Frankly, if you have ten billion people on a planet, of course it's gonna impact the environment. There is no way to avoid that. I wouldn't start nitpicking with whether people can have a garden or not, though. A garden can help people with a fresh source of vegetables which can improve health and wellbeing, and strengthen independence and community-building, maybe, if the garden is shared or vegetables are distributed among neighbours. It also reduces the transport distance for vegetables which saves on emissions. So, a garden can be a meaningful part of human life, i'd argue.