Pol Pot admitted to not being a communist. He attacked the urban proletariat and proletarian intellectuals, and the communists that fled Cambodia for North Vietnam left Cambodia largely with nationalists. This nationalism took on a highly reactionary character when targeting communists and Vietnamese, which led to war with Vietnam.
Pol Pot's agrarian focus wasn't just due to being underdeveloped, as were China's and Vietnam's economies, but also due to a severe distrust of the proletariat. Pol Pot essentially made a dictatorship of the reactionary peasantry against the proletariat, and against socialism.
Further, I find your comment about Deng and Mao with the US to be misplaced. Deng and Mao made mistakes in allying with the US. Pol Pot's mistakes far exceeded their mistakes. The fact that someone makes a mistake is not inherently an entire judgement of them, but when understood in the context of their entire character and actions it becomes clear that Pol Pot was used as a tool to extend imperialist interests in the region.
It's not idealist to note the class nature of what happened under Pol Pot.