That is another great question. If it is transformative use of the primary data source, then that is likely illegal, as nobody gave permission for them to transform and process that personal data. If it is not transformative, and it just gives access to the primary source like a search engine on the other hand, then the problem is that if it returns copyrighted data, it is no longer fair use most likely.

That's a good point, that muddies the waters a bit. Makes it hard to say wether it's spouting info from the web or if it's data from the model.

I can't comment on actual legality in this case, but I feel handling personal data like this, even from the open web, in a context where hallucinations are an overwhelming possibility, is still morally wrong. I don't know the GDPR well enough to say wether it covers temporary information like this, but I kinda hope it does.