Yup, so take away the ability to grossly profit off the backs of others and allow the scales to balance. There's no reason we can't do both by disincentivizing gouging and slumlording while at the same time increasing the creation of more affordable housing.
Hell, if a sliding-scale of fees against # of properties/profit were implemented they could use the revenue from that to help fund more affordable housing, while discouraging house-hoarding at the same time.