it is absurd that paying off a debt can lower your score at all.

I think you're missing an important piece.

The point is that if the credit score is supposed to represent your trustworthiness to a potential lender today

My bolded word added. If you pay off all debt and a month goes by (and your score drops) its because they don't have evidence that you still have the resources to service the future debt.