There are already tax discounts on primary residence, there's a capital gains exemption. Doesn't matter because housing prices still go up too much. And why do they keep going up? Hmm... maybe something about a shortage of housing?
Speculators can't single handedly increase the price because they still need someone to buy or rent the property for a worthwhile price.
There are areas where housing prices stagnate and it's not like some speculators can just go in there and force prices to go up.