You're guessing, you don't actually know that for sure, it seems intuitively correct, but we simply do not know enough about cognition to make that assumption.

Perhaps our ability to reason exclusively comes from our ability to predict, and by scaling up the ability to predict, we become more and more able to reason.

These are guesses, all we have now are guesses, you can say "it doesn't reason" and "it's just autocorrect" all you want, but if that were the case why did scaling it up eventually enable it to perform basic math? Why did scaling it up improve its ability to problemsolve significantly (gpt3 vs gpt4), there's so many unknowns in this field, to just say "nah, can't be, it works differently from us" doesn't mean it can't do the same things as us given enough scale.

So for context, I am an applied mathematician, and I primarily work in neural computation. I have an essentially cursory knowledge of LLMs, their architecture, and the mathematics of how they work.

I hear this argument, that LLMs are glorified autocomplete and merely statistical inference machines and are therefore completely divorced from anything resembling human thought.

I feel the need to point out that not only is there no compelling evidence that any neural computation that humans do anything different from a statistical inference machine, there's actually quite a bit of evidence that that is exactly what real, biological neural networks do.

Now, admittedly, real neurons and real neural networks are way more sophisticated than any deep learning network module, real neural networks are extremely recurrent and extremely nonlinear, with some neural circuits devoted to simply changing how other neural circuits process signals without actually processing said signals on their own. And in the case of humans, several orders of magnitude larger than even the largest LLM.

All that said, it boils down to an insanely powerful statistical machine.

There are questions of motivation and input: we all want to stay alive (ish), avoid pain, and have constant feedback from sensory organs while a LLM just produces what it was supposed to. But in an abstraction the ideas of wants and needs and rewards aren't substantively different from prompts.

Anyway. I agree that modern AI is a poor substitute for real human intelligence, but the fundamental reason is a matter of complexity, not method.

Some reading:

Large scale neural recordings call for new insights to link brain and behavior

A unifying perspective on neural manifolds and circuits for cognition

a comparison of neuronal population dynamics measured with calcium imaging and electrophysiology

I am picking up a hint of the autocompletion you describe, in your writing.

If I teach a real AI about fishing, it should be able to reason about fishing and it shouldn’t need to have read a supplementary knowledge of mankind to do it.

This is a faulty assumption.

In order for you to learn about fishing, you had to learn a shitload about the world. Babies don't come out of the womb able to do such tasks, there is a shitload of prerequisite knowledge in order to fish, it's unfair to expect an ai to do this without prerequisite knowledge.

Furthermore, LLM's have been shown to do many things that aren't in their training data, so the notion that it's a stochastic parrot is also false.

No that's definitely the case. However, Microsoft is now working making LLM's more dependent on several high quality sources. For example: encyclopedias will be more important sources than random reddit posts.

There are still plenty of videos to watch and games to play. We might be running short on books, but there are many other sources of information that aren't accessible to LLMs at the moment.

Also just because the training set contained most of the books, doesn't mean the model itself was large enough to learn from all of them. The more detailed your questions get, the bigger the change it will get them wrong, even if that knowledge should have been in the training set. For example ChatGPT as walkthrough for games is pretty terrible, even so there should be more than enough walkthroughs in the training set to learn from, same for summarizing movies, it will do the most popular ones, but quickly fall apart with anything a little lesser known.

There is of course also the possibility that using the LLM as knowledge store by itself is a bad idea. Humans use books for that, not their brain. So an LLM that is very good at looking things up in a library could answer a lot more without the enormous models size and training cost.

Basically, there are still a ton of unexplored areas, even if we have collected all the digital books.

Why is that a problem?

For example, I've used it to learn the basics of Galois theory, and it worked pretty well.

• The information is stored in the model, do it can tell me the basics
• The interactive nature of taking to LLM actually helped me learn better than just reading.
• And I know enough general math so I can tell the rare occasions (and they indeed were rare) when it makes things up.
• Asking it questions can be better than searching Google, because Google needs exact keywords to find the answer, and the LLM can be more flexible (of course, neither will answer if the answer isn't in the index/training data).

So what if it doesn't understand Galois theory - it could teach it to me well enough. Frankly if it did actually understand it, I'd be worried about slavery.