Fair question!

If an email address is being used for fraud, they don't need to see the encrypted copy; they can see the copy sent out to other people from that address. So if I send you a message from my Protonmail to your Gmail, the following is true:

Copy @ Protonmail: E2EE.
Copy @ Gmail: NOT E2EE.

There are other, circumstantial ways to tell as well. If you're trying to scam people with DudeBro Cryptocurrency, you necessarily reveal the address you use when you send our your spam or scams. If I send malware from notactuallydiotima@proton.me, the proof that I sent the malware does not require you to see my server stored mail; you can just look at your own copy to see.

Does that make sense?