"Takes seconds" seems like a strange experience. I remember OsmAnd years ago performed like that for me – clearly, painfully loading in the individual tiles. But nowdays, it smoothly transitions between LoD and has no problem smoothly scrubbing over e.g. a major city.

I'm using offline vector maps, for context.

Edit: Trying online tiles again, I'm assuming your problem can be resolved by switching to offline. You can have the offline data update automatically such that you don't have to worry about it. CoMaps and Organic only use offline maps, which is why they're similarly snappy to offline OsmAnd.

This is under Configure Map > Map Source, and you only need to download for the region (usually e.g. a province) that you're using.