Yeah, I feel like with usage of the RNG control the game affords you and smart drafting it's not that big of a deal outside of maybe a small handful of things.
And honestly, there aren't even all that many red rooms that are super punishing in the late game. It's mainly Archives, Maid's Chambers and Lost & Found that can really punish you. Although Lost & Found set to common can be pretty interesting, if you get it early when you don't have any items it's basically no downside all upside. Actually if I got the chance to play it again I would spec into the Red Room synergy build, I feel like it can be pretty fun and strong.