[phatfish]

I suspect is very hard to get a consultant that doesn't hand over a repo of AI generated nonsense right now. For the reason above, they are (usually) under time pressure to deliver, and they can wash their hands of the results at the end of the contract.

I am at 2 for 2 at the moment in the "infrastructure as code" arena (I wasn't involved with choosing to use a consultant, just dealing with the output). Which is an area that AI was supposed to eat for lunch. And it seems like it should, DSLs with a narrower scope seem perfect for an LLM, but I'm not convinced.

I think the issue is, infrastructure DSLs like Terraform or Azure Bicep are distilling down an architecture that has complex interactions and often needs a lot of "inside baseball" knowledge from outside of the code to create a congruent result. Unless you feed a bible of markdown files to the LLM to guide it in the right direction the output goes off the rails fast. The time spend creating the bible might as well be spent creating the code.

Of course there are areas where an LLM will definitely help, like re-factoring, stamping out boiler-plate or even building on a solid base. But attempt to create even a semi-complex architecture from scratch using a few paragraphs of prompt and you are asking for trouble.

The trouble with the consultants I have interacted with is they don't write the bible first, as far as i can tell they just iterate on slop and you end up with multiple 800+ line PowerShell scripts in IaC pipelines and other craziness that is almost impossible to unpick after they have gone.