I agree with you, but a baby can't read a ballot or pull a lever. Help is always available to anyone who asks, so I suppose we could just eliminate the age requirement altogether and let anyone who is able to register go to the polls.
I would be concerned about a certain type of person trying to make as many little voters as they can crank out, but I suppose some people do that anyway and just wait until they turn 18.