I almost left the Fediverse myself after stumbling across them - talking with such people is not what I wanted to spend my time and attention on. Fortunately Kbin.social that I had been on prior to it going defunct and my switching to Lemmy did not promote their posts so heavily (it had, and its fork Mbin still has, its own internal voting system, so its sorting algorithms ignore "upvotes" from the Fediverse and pays attention instead to "favorites" from only other Mbin/Kbin instances, probably for precisely this reason). So I knew that the Fediverse could be better.
The problem is: nobody that I recommend it to irl realizes that - they see the variety of posts from e.g. !memes@lemmy.ml, and think "this is what Lemmy is", and they drop it never to return. And how can I argue otherwise? I myself would rather leave it entirely than put up with constantly having posts promoted everywhere you look from the likes of Donald Trump, Elon Musk, or the same style from hexbear even if they call it "leftist". This, along with the lack of technical development (compared to e.g. Reddit), are the chief impediments to growth that I can see.
But things are steadily improving - e.g. we can do instance blocks now, and e.g. PieFed and Sublinks are coming to join with Mbin and Tesseract to offer alternative codebases to just Lemmy on the Fediverse, which like Kbin's alternative voting scheme may offer solutions to some of the current issues. Therefore as long as I can enjoy sane conversations, e.g. this one with you:-), I will stick with the Fediverse, as it evolves moving forward.