Bacteria and yeast aren't animals. They aren't sentient, they have no perception of the world around them, they don't have feelings. That's the difference. Nearly every animal (yes, even insects) is sentient. We may not understand exactly what it feels like to be a bee (what kinds of emotions they can experience), but it's better to err on the side of not hurting an animal than assume they are mindless little robots.

Given, this is usually not the primary focus of vegan activism. Taking some food from some bees versus raising cows in the pure hellscape that is factory farming... There's a very obvious greater evil happening. Let's not let the minutia of veganism derail from the greater picture.