I've used it to add test coverage. Granted it wasn't for new major features but small features that still necessitated having tests.
IME so far that's what it's best at. As long as there exists tests in the codebase that provide enough of the basic skeleton (i.e. if they already test part of a controller or code that interacts with a DB) to guess what's needed then it can do a decent job. It's still not perfect though and when it comes to inventing bespoke tests that are uniquely different from other tests it needs much more of a helping hand.
A big Bay Area data company recently refactored their old JS codebase from var to let and const. It was done at a scale that is way beyond "senior engineer double-checks LLM to see if code is good."
LLMs are terrible at that in my experience. In what world is refactoring "very local in reasoning"?
Switching libraries/frameworks or switching piecemeal to a new language for a codebase that's already well structured seems like it would be noticeably less costly though.
In a situation where the transformations are explicit and can almost be mechanical with little to no involved reasoning. With those requirements, the LLM just has to recognize the patterns and "translate" them to the new pattern. Which is almost exactly what they were designed to do.
IME so far that's what it's best at. As long as there exists tests in the codebase that provide enough of the basic skeleton (i.e. if they already test part of a controller or code that interacts with a DB) to guess what's needed then it can do a decent job. It's still not perfect though and when it comes to inventing bespoke tests that are uniquely different from other tests it needs much more of a helping hand.