When it's about choosing a place to live with my family you're goddamn right it does.
I can not fathom raising a child in the place where I currently live, next to them. My child would learn the most vile words and risk getting beaten just for being too innocent the second they step outside.
You know when else this discrimination would be justified? In hospitals, where a screaming Roma guy that threatens the nurses constantly, yelling and getting in the way of staff, gets the attention on his friend with a mild fever while an innocent dies of blood loss in the same room. They take advantage of our free urgent care while the people that fund said urgent care die next to them. The distinction between them and everyone else is that they bring their entire extended family to threaten the staff into submission. And, since they're not integrated into society...they don't actually fund that urgent care. I know nurses. Nobody does that, ever, except them, according to nurses and lab workers. Ever seen someone busting into a lab? Our nurses have. It's always the Roma, according to them.
In other contexts, no, it's not justified, at least not in general.
Your take is good tho... We need to give them a chance, but it is so, so fucking tiring and hard not to be racist against them...