I did cut back significantly since that initial switch. At one point, I had 1TB of music. Right now it's sitting at ~210MB of mp3s. Quod Libet uses less RAM than Firefox, Thunderbird, or RSSHub, but it is sitting at 4th place on my system.
I don't think there's a way to scale music libraries to these obscene sizes without impacting RAM. Unless you manage it strictly with a file manager and open album folders individually with a lightweight player used only for playback.