One issue is: As rooms are decentralised, if someone posts illegal content and your server downloads it, that could lead to legal trouble, especially as you'd then also host it.
That should only be an issue for media though shouldn't it, since all plain text can be set up to be e2e.
The encryption often fails, leading to the meme "Message cannot be decrypted".
Is that still the case, I have been using matrix for about a year now and I haven't ever had an issue sending and receiving except for a day which the vector.org servers were down. I hear that hosting a synapse server is a pain in the ass, but I haven't gotten around to it yet.