you can be annoyed about something someone is doing without making some personal judgement about that someone
if my train is late because the train driver had a heart attack im aware that its not the drivers fault, but im still not going to be thrilled about my luck