Net-new luxury housing is good for affordability, because when someone moves into a more expensive unit, that frees up their current more affordable unit.
More to the point, Seattle literally just legalized missing middle housing in most of the city earlier this year. That's good for affordability, but new housing takes time to build. And developers will try to build the most lucrative project they currently can.
Housing is a matter of supply and demand. When you're in a housing shortage, prices will be high and most of the new supply will be luxury. The solution to a housing shortage is to build more housing, period. If you build housing faster than increasing demand from population growth, prices of units will go down. If you build housing and prices stay high, you didn't build enough. Build more. Remove NIMBYs ability to prevent new builds.
Which is not to say that building public housing or other projects to subsidize housing is a bad idea. But it's really, really hard to do that effectively during a housing shortage and solving the shortage is good for everyone except homeowners who wanted to use their equity as their retirement nest egg.