I’d argue this is too low. I do t know how long it’s been like that but doubt it has risen with inflation

I'm on the fence here. Realistically, if you can show "normal" bills related to wear&tear and maintenance, that gets you out of paying taxes in even the most expensive parts in the country. I live outside of Boston and have never come close to owing a penny on selling my home.

I live in a high cost of living area. Despite this being low percentage growth, my house has gained that much over the 20 years I’ve lived here.

So the problem is the paper trail, and a good real estate attourney can help you maneuver it. What have your expenses been on the house over the last 20 years? They all count. Profit isn't just "selling price minus buying price" in this situation. At least in my state.

what happens to someone retired after owning a house for decades

That does get complicated. To my understanding, normally the law works reasonably well in situations like these, but I don't know all the nuances. It's a "lawyer-up" situation, especially if there's an estate involved. I'm sure there are edge cases, but largely people end up not owing taxes on their home residence even in very pricy areas, unless their gains on it are absurd. If you bought a Boston apartment building when it was cheap and sell now, then you're likely going to be paying taxes on some of that $5M+ windfall.