My dumb ass left VPN_AUTO_PORT_FORWARD set as "Tru" in my docker-compose and apparently that made the image assume I MEANT "false" so it was randomizing my port and not matching the VPN to it
Fixed that stupid typo and suddenly I can seed! I. Can. FIGHT!