Depends what I'm doing. Yardwork or exercise? Change as soon as I'm done. Just sitting around the house in pyjamas? They're fine to wear again.
Two caveats: I don't wear socks to bed, and if I'm wearing socks indoors (I prefer bare feet) I always wear shoes or slippers over them