I agree that they are useful for this. In fact, as a programmer I find them quite useful whenever I need a bit of a guided start on something that otherwise I'd have to trawl the internet to find. Once the LLM has given a pointer it's easier to follow up with appropriate resources. And the LLM is useful for writing code when the code is predictable and you know reasonably precisely what you need, where the LLM really just saves you some typing and you know how to review it for correctness. Outside of these cases you have to be pretty careful how you use them.

But I don't think LLMs are as useful a tool as the business people want them to be. Programming is unusual in that it involves very predictable patterns, and the aim is to find the most appropriate pattern for the task. And software documentation too follows very predictable patterns. Where an LLM has seen the exact same pattern many times, it will be good at producing it on demand. So programming and explaining software is a good use case for LLMs. But not many areas of activity are like this, and when you get out into all the nuance and complexity of other less formal domains, LLMs are so prone to slipping up that they're much less useful.

I've tried getting LLMs to summarize notes for talks on complex topics, and they are not good at it. I've tried getting them to tidy documents and they're not good at it. I've tried getting them to explain complex topics for someone who knows nothing, and they can be good at it but they can also be misleading, and you don't know which one you're getting unless you go to other sources you could have checked in the first place.

So I think they're most useful for a quick orientation on a topic that points you to further sources, or for very highly formalized activities like programming. But they can't be trusted for math or physics or law or medicine or literature or philosophy or complex decision making or psychology or any number of other areas.