You're looking at it purely from a present-day perspective. Just because the pool of available workers is large enough today to provide for everybody doesn't mean that it will be fifty years into the future. It's not about "infinite growth" it's about providing a consistent standard of living and a fair generational contract that doesn't place an undue burden on future generations.