I am not vegan, but simply trying to understand how honey is bad, but as you say “unavoidable collateral damage of agriculture” or not.

Is bad as well, we simply have no good way of avoiding it.

Think about it this way: Beekeeping is bad, agriculture is bad. Can we avoid both? No. But can we avoid at least one of them? Easily so. So let's do that - half a win is better than nothing.

There are many ways agriculture could be less harm, less pesticides, less monotone growing practices, more spread out growing. We do not have to accept these practices to not starve.

I agree, which is why many (if not all) vegans strive to support those more sustainable forms of agriculture. But economic constraints are a real thing for many people. Not everyone can always decide to buy the higher quality produce. If we can - good, let's do that. While and if we can't, same thing with the honey: Can we avoid all the problems at once? No, but at least we can do as best as reasonable possible, so let's do that. That's veganism for many people.

I don’t think honey collecting is worse than agriculture (even of direct plants for human consumption), so I don’t think vegans should discount honey.

Even if it's just 1% worse than agriculture wouldn't we reduce a bit of suffering by replacing it? And I mean it's not even like we need honey for anything. We consume too much sugar anyway. Even if honey is exactly as harmful as sugar cane farming (which is debatable), by omitting it we would save not only agricultural resources but animal exploitation as well. Not consuming it is better than consuming it in terms of animal suffering. Since we don't need to consume it, from a vegan perspective I think it's understandable why that's seen as preferable.