language model, not an objective reality model

Sort of. It's a generic prediction model. Multi-modal models work the same way as text-only models in this sense.

So do organic brains.

Right now, you are "hallucinating" most of your visual field outside the fovea centralis.

This aspect of your conscious perceptual system is exactly the same kind of high-dimensional interpolation that ML neural networks do.