Left and Right rely on the orientation of the person. So if I'm looking ahead at you, and you're looking at me, my left is different than yours. However, port and starboard don't change based on position of the viewer but the position of the vessel. Cuts out any ambiguity.

Left and right are referenced to yourself, or relative. Port and starboard are referenced to the ship, or absolute.
It's the same reason people who work in theater say "stage right" and "stage left" instead of just "right" and "left."
Imagine being on a boat for weeks, in the middle of the ocean, with no land in sight; it makes a lot of sense that you would think of locations in terms of the ship, as that is your only reference point.

Historically, boats had rudders (or “steer boards”) along the side of the ship instead of in the middle like we’d see them today. It was always on the right side of the boat, so to avoid smashing your rudder into the dock, you’d dock your ship on the other side. That means it was always the left side that literally faced the port, while starboard faced out into open water to protect that side from damage.

Then you have to say something like "your left or mine?" This removes any ambiguity, and that might matter when one is at sea.

Also, port is red and Starboard is green. Usually boats will colour the sheets to match. It helps to identify the lines when someone tells you to ease the port sheet.