IIRC, it was because the tiller for the rudder (with which you steer) was on the right side. (Styrbord in Swedish = steer board) As to why you have port in English, I have no idea. It's babord in Swedish, from bakbord (back board) as when steering, the left side was behind you.

Edit: Apparently it's port because you'd dock with the left side to the port as otherwise you'd crush the rudder, which again was on the right.