Not a runner, but for me the feeling is that I can workout for 364 days a year, but if I miss one single day, then the habit is broken and it takes the willpower necessary to move a fucking mountain to get back to the gym after that. It feels like I don't built habits no matter how many times I do it.
It's still necessary to rest, but the fear is legit.