It really depends, my friend was having a hard time once so in the evening I drove 3 hours to go see them, had a beer with them, then came home. I think it was absolutely worth it. I've also driven that far to buy something that took 15 minutes.

But say, a museum, I'd definitely want to make a day of it if I was driving that far. Maybe more.