But if you draw out the velocity vectors, the portal and people would have a would have a positive differential.

Imagine a building comes flying at you and but you barely jump into the window. To you, you're not moving, but everything else around you is. It would be the same principle if a moving portal came flying at you.