As someone who highly values spending time just thinking, it does take a level of privilege to do so. If your needs aren't met, or your life isn't safe/secure, letting the mind wander is giving it the ability to latch onto these distressing topics and create loops of anxiety and stress.
As much as I value spending my time thinking, I also value ways of shutting my brain off so that I don't have another panic attack about how I can't afford groceries yet or not knowing if my housing situation is secure. I can't feel like my life is collapsing if I distract myself.
And more people than not are struggling to make ends meet, so I imagine more people than not have stresses they need to put out of mind, just to retain their sanity.