Meh, I use it. I'll take it over Discord or Telegram any day. But I don't use it for anything that may be sensitive or anything involving IRL people.
It's leaky. I remember all media were uploaded unencrypted and available over https, I don't know if it is still like that. Lots and lots of metadata out in the open. To be searchable you have to give your phone number to a centralized service. The protocol is overly complex, all messages live on all servers of everyone involved in the conversation, lots of duplication, but ActivityPub is like that too and we are on Lemmy...
If I set my own stuff up, I prefer XMPP, and increasingly Simplex. If some project uses matrix, I have an account and will talk to them there.
Overall I'm not a fan, but I don't outright hate it.