First of all, I'm not poor. I come from a poor family but I was blessed with a sharp mind and I've been lucky enough that my hard work has given me a rather good life. Due to my life experience I'm acutely aware of not just how incredibly privileged my life is but also how there are people who start from an even worse position than me and will never get to experience a life where they don't have to worry about whether they can afford to eat, pay rent (let alone own a house) and have clothes on their back. So yeah, excuse me for both having enough empathy to understand the struggles of others and enough privilege to not perpetuate it.
As for awful people, if there's a beggar on the street shifting through trash to find recyclable bottles or anything to make money and someone comes and takes some of their money would you not consider that kind of a person awful? And yet that's what investors do, not with just the poor but with everyone who works for a living. Every cent you make from stocks is a cent you take out of the pocket of someone significantly poorer than you.
I'm not going to tell you stop investing. You want a lavish lifestyle that's how you get a lavish lifestyle. All I'm doing is pointing out that your vanity comes from robbing the poorest of us. If you don't like being called an awful person maybe don't do awful things?