There are plenty of alternative explanations.

For a few examples:

• In such a huge universe, even if FTL travel is possible, why should we expect Earth to be a common destination?
• It is possible that only species who do not wish to colonize the galaxy will avoid the Great Filter and acquire the technology needed to colonize it.
• Not all space-faring races must be highly populous, while less populous species will encounter less of the issues that would cause a population to collapse before becoming space-faring.
• We could be colonized without recognizing it.
• Colonization itself could be inherently unsustainable.

There is also no reason to limit the discussion to a galaxy. If we assume that an FTL civilization will colonize (in a way that we would recognize), then they could come from any galaxy. Given the expanse of the universe, if such a behavior is common enough that it would stand a chance of succeeding, then it should probably exist already. And yet we do not appear to be colonized. Which is more likely: That FTL intelligences must colonize, or that we are all alone in the universe? Axiomatic reasoning reveals that the latter is statistically much less likely than the former. So it is less likely to successfully explain why we appear to be uncolonized.

If faster than light travel existed, then a species would possibly be able to colonize anything in less than an instant because they'd be traveling backwards in time.

However, it might be possible to change locations faster than light by cutting through spacetime using a wormhole.

Yet, in support of your argument, I think I also remember that perhaps there are ways to warp spacetime around a ship so that locally, the ship is not traveling faster than light, but the warping of spacetime is. I don't understand this concept well enough to have a confident opinion on it tho. For example, is it possible to warp spacetime faster than light without violating general relativity?