You are probably 100% right that a more heterogeneous grind would make a better brew. And tighten steep times. While I try for something around 5-7 minutes it isn't uncommon for it to be 2-20 minutes. It has still been good coffee, way better than any drip. Frech press is ridiculously robust.
At some point I should borrow a great grinder and test out a bunch of grind/steep time combinations to see what I prefer. And compare it to my normal brew. Perhaps there is a light for me to see.