Give it a shot again, something changed recently in Proton (I assume) that made Vortex "just work" for me on my Steam Deck. I didn't even need to do any fiddling, I just ran the installer exe from desktop mode using Lutris and whatever Proton was latest, and it installed perfectly. Vortex now runs entirely as expected, even from game mode.