LLMs break words up into chunks of letters which commonly appear - suffixes like "-tion" and "-ism" are obvious examples. They then predict which chunk comes next based on the ones before, or whether the word will end.

This is very useful for generating sensible-looking text while at the same time correlating concepts associated with different words. However, it also means that the dont really "see" the letters that make up each word, just the chunks of letters, which are stored as mathematical vectors. This is why they struggle so much with analysing the makeup of words.

However, with numbers they generally store each digit individually, so they shouldnt have as much of a problem saying how many 5's are in 1,589,005, for example.

Simply put, LLMs are great at forming sentences but can't do math. Like, any math. If they get 60+21=81 right, it's only going to be because it's textually written somewhere in the training data that 60+21=81. However, it's very unlikely for counting the number of Ts in colonialism to be in there, so it just hallucinates what it thinks is a correct response.

Two very different answers to this question...