I don't think most people have a seconds clock up on their phone when they set their watch to their phone time (also meaning OP is correct that phone time is base time for people).

That means they aren't syncing the seconds hand and there is ± 60 seconds of error. Also probably some error in ± 1 minute of hand placement error, especially for small watch faces, plus mechanical timing error for all but the high end mechanical watches. There are tons of sources of error. NTP time is accurate to the 10s of milliseconds, so around 3 orders of magnitude difference for most people.