Yeah, I think that's why I didn't have an issue with it. I remember only doing maybe 1 or 2 of the barrels in the water in Skellige. I mostly used the question marks as small distractions as I went from one quest to another. I never focused on clearing them out.

You're right that most of the items weren't that rewarding to get. It's mostly just busy work. Hopefully rebirth keeps that stuff to a minimum and just has a large map with cool locations and unique well written side quests. Maybe a few collectibles scattered around the map to reward exploration too.