Besides the main implementation synapse being slow, the new implementation dendrite is unfinished but progressing.

But technical standards aside, the hassle of managing encryption keys is too buggy and confusing IMO. That'll deter most people I feel.

I want it to be great but it just isn't there yet.

I personally really like Matrix, but there are a few outstanding complaints about it. The biggest one is that the reference implementation everyone uses by default is known to be bloated and slow, and poor at scaling. Server admins have had a huge challenge of supporting a large amount of data for things like room history, which in the past required propagation to every server hosting every participant. The protocol itself has been described by some developers as overtly complex.

Some of this seems to be improving, particularly with development of a Go-based backend implementation, Dendrite.

I'm not even close to an expert, but from what I have heard, matrix collects quite a bit of metadata depending on the server you are using/federating with.