The people who will be impacted first will be people who own vacant lots and parking lots in and around downtowns. If you're concerned about people getting booted out of their homes, consider Estonia:

Estonia levies an LVT to fund municipalities. It is a state level tax, but 100% of the revenue funds Local Councils. The rate is set by the Local Council within the limits of 0.1–2.5%. It is one of the most important sources of funding for municipalities.[90] LVT is levied on the value of the land only. Few exemptions are available and even public institutions are subject to it. Church sites are exempt, but other land held by religious institutions is not.[90] The tax has contributed to a high rate (~90%)[90] of owner-occupied residences within Estonia, compared to a rate of 67.4% in the United States.[91]

https://en.m.wikipedia.org/wiki/Land_value_tax

In general, LVT should increase overall housing supply, improve affordability, and can be used to reduce other taxes such as property, income, and sales taxes. Most serious proposals I have seen have been to replace property taxes with LVT. These factors should make it easier on average households generally, and also allow them more flexibility to downsize (once your kids have moved out, do you really need a jumbo house all to yourself?), rather than locking you into the only place you can afford.