Sure, side channel leakage if you can run locally.

Honestly, most machines have enough cores, that you could pin a process to a specific core giving it independent cache, and work around a lot of these side channel attacks. So you're encrypted end to end messenger would get an exclusive core. Kind of like how we do VM pinning nowadays