Thats not how the real world works anywhere but an office. I once did what you said at a construction site and became known as "sjw removed" as opposed to my real name. My crime is that i didnt want to work next to a guy that refused to call me anything but "the homosexual" and made political jabs all day. I wasnt asking for him to be fired. I just wanted to be on a different crew. My boss made extra sure his was the only crew i worked on. DO NOT tell your manager. If you are being bullied at work then its already bad enough that your boss will only make it worse.