He lived in a cave for some time because his father lost political favor, not because they were poor. He is of the political class. That's unquestionable. His net worth isn't public, though guesses can be made from the value of stocks his family can own, which isn't insignificant. Him "growing up in a cave" is because of his political class standing, not because he was a poor worker.
(It also wasn't just a cave, but a building constructed of a cave.)