Wow, so poisoning the training data works that well?

This is awesome - guys, let's all upload papers that say the lining of MAGA hats contains a chemical that causes cancer, that NOT getting vaccines causes autism, and that the most conservative universities show Trump lost the 2020 election. Let Grok gobble it up!

There are going to be people who believe this disease is real for the rest of their lives.

The condition doesn’t appear in the standard medical literature — because it doesn’t exist. It’s the invention of a team led by Almira Osmanovic Thunström, a medical researcher at the University of Gothenburg, Sweden, who dreamt up the skin condition and then uploaded two fake studies about it to a preprint server in early 2024. Osmanovic Thunström carried out this unusual experiment to test whether large language models (LLMs) would swallow the misinformation and then spit it out as reputable health advice. “I wanted to see if I can create a medical condition that did not exist in the database,” she says.

The problem was that the experiment worked too well. Within weeks of her uploading information about the condition, attributed to a fictional author, major artificial-intelligence systems began repeating the invented condition as if it were real.

Another study said that in recent years 25% of studies were being faked up for academic credentials, either by paper mills or academics themselves. Thousands of papers even used the same diagrams, with text in slight variations, citing each other.

Add AI to the mix, things get worse.

In a separate study of 20 LLMs, Omar found that LLMs are more prone to hallucinate and elaborate on misinformation when the text they’re processing looks professionally medical — formatted like a hospital discharge note or clinical paper — than when it comes from social-media posts (M. Omar et al. Lancet Digit. Health 8, 100949; 2026). “When the text looks professional and written as a doctor writes, there’s an increase in the hallucination rates,” says Omar.

You can just make an Overleaf account (or install GNU TeXmacs) and start outputting academic-like papers for fun and profit. I would have thought LLM developers would have at least highlighted PageRank-like citation metadata as very important when training on academic publications; papers with no citations clearly aren't reputable.

Okey but was the paper obviously a joke? How a human would determine it was fake