Things can undergo a transformation, just because the original usage might have been innocent. If something starts being used by white racists or western imperialists in a racist way, would ya look at that, it’s racist.
Also, it’s covert racism. It gives people plausible deniability. Like you are doing now.