As a dev, I think agile works best when there's an ongoing conversation with the users, and I usually have to fight with management to get to speak to those actual users.

Tru dat. Agile product management is not the same as agile project management. Agile Project Management is about the ability to figure and changes things along the lines of the predetermined cost and time path (e.g. figuring out features required along the way), not about the agility to prolong/shorten product value proposition time to market.

Do your scrum-using organization put users at the development process?!? I don't think I've seen any Agile¹ organization doing that.

1 - The one with capital "A", that is an antonym of the one with lower cap "a".