LLMs produce code that is functionally error prone while looking reasonable (in the same way that it produces answers that are grammatically correct, correctly spelled, but factually incorrect).

As we all know, fixing bugs in someone else’s code is generally more difficult than writing the code correctly in the 1st place , and that’s going to apply to a LLMs code output just as much as a humans, if not more.

That's assuming they're using one of the generic models like ChatGPT and not something custom they've created specifically to do this.

Edit: they are in fact using their own as per the article

The AI would likely be trained or fine tuned specifically for COBOL. In these very narrow use cases AI can find some things that humans can miss.

Google did this recently on a sorting algorithm and was able to speed it up by 70%: More info here