Communication about times would get so much easier, communication about schedules would get so much easier.
Except that it wouldn't. It would make communication about time a culture sensitive topic. Sure, the exact time of day in relation to the position of the sun might get lost with our current system, but if someone tells you "I've slept til 12am" at least you know it was somewhat around noon. Under your new system you'd always have to consider where someone lives.