You’re discounting the people who have always lived within their means and so never took on debt.

No I'm not. Those people are unknown quantities, and so also suffer if credit scores go away, because bad borrowers are worse than first-time borrowers, so without credit scores, first-timers will be treated worse.