Yeah, this is a phenomenon called 'confabulation.' You see it with stroke patients too. There's some who feel like it's a more accurate term than 'hallucinations' for when LLMs make shit up these days too.