I don't think that's right. There is a real threat from e.g Plutonium 239 which is extremely carcinogenic and toxic in minute doses.
Here's a collection of threats relating to nuclear power production and it's waste starting on page 112: https://www.isi.fraunhofer.de/content/dam/isi/dokumente/ccv/2013/ETTIS_Deliverable_4_4_Catalogue%20of%20Threat%20Scenarios.pdf