I've only seen it used in the pejorative sense when there was also a racial factor. So by people who were racist.