Think it is the word for the symptom rather than the diagnosis, much like deaf or blind are, just that English doesn't have a casual term for it.

The lack of a casual term may be because though the sense of smell is very useful & has a safety/survival component, being anosmic is unlikely to leave a person requiring much by way of additional skills and/or tools relative to deafness or blindness.