I'd make it from a larger dowel, 1.5" to 2" diameter, with a rounded nosecone and chunky fins from 1/2" stock, so they don't break when it's tossed in the air. That way you could attach the fins with a screws and glue, no fancy milling needed.
Before shaping, drill a hole half-way through for a window port, don't go all the way through or they'll get their finger stuck someday.
Modeling after the v-2 because it looks as stereotypical as a rocket can