I'd say the issue is that if honey isn't vegan because you're causing harm to bees, isn't most of modern vegetable agriculture at least equally harmful to bees & other insects due to all the pesticides being used?
Or is it just if we directly involve bees, it's bad, but if we inflict greater harm in a less direct way, it's acceptable?