So you want self hosted? And what you mean by changing the encryption? Changing the encryption protocol and encryption algorithm are two different thing. OMEMO is kind of de facto for XMPP. Last I checked it seems doesn't let you choose the cipher suite like TLS 1.2 does. You can spin another if you have the capability to do so but not recommend.

And whatelse you want to tweek?

EDIT: typo