One downside I haven't seen mentioned yet is the hassle of having to laboriously lock up your bike front and back, saddle bag, too, so some freaking degenerate won't steal anything that isn't secured. I've had my rear wheel stolen before (who even DOES that?) and am not looking for a repeat.
I love riding my bike and do most trips with it in the summer (shopping, appointments, gym, etc.)
But I envy motorists sometimes who just get out of their car, push the button on their key and everything's neatly locked up.