[fasten]

Thanks for the feedback! We already use AI in the PR to explain whats happening and the best practices to adopt. As for the code remediation part: most LLMs fail to generate the right IaC code thats adapted to your infra because they miss its general context (config, dependencies..). We are building first the deterministic part (the context) and once we have the context our plan is to add the fix/recommendation in the change.

[anAiguy]

How will you be checkiing the quality of the AI recommendations in the your PR. Do you think that using different model ( chatgpt, claude,gemin, qwen) to challenge the recommendation made by another AI could help ?

[fasten]

About having differents models challenging each other, I haven’t seen anything useful yet but I understand where you are going. Might be a future direction

[anAiguy]

I have in mind the following paper. It is called Self-Taught Evaluators (https://arxiv.org/pdf/2408.02666)by Meta . It is interesting as they get big improvements from LLM checking and improving solution. WDYT ? I don't know if you could generate an PR using AI with let's say Claude and then check the quality by using chatgpt or gemini.. I would be interested by knowing if that would provide quality and more trust or the opposite

[emmtold]

Ok focusing on context makes sense but I’d challenge the idea that LLMs inherently fail without it. Some teams have used fine-tuned models or hybrid workflows with partial context to generate useful IaC snippets

[fasten]

Agreed 100%. LLMs are doing solid job at generating IaC but in a context where the person who use them knows what he/she's doing. In our case, remediaiton means an extra level of trust, where your infra is already super sensitive.

[emmtold]

we have used some tools to generate terraform code based on our unmanaged cloud resource for instance and it worked well..

[fasten]

The tools we are aware of will create a 1-to-1 mapping to some code, but very often with hardcoded values because they lack the full context of your infrastructure. This can lead to potential incidents in the future (broken dependencies / visibility). This is at least the way we are approaching it, and why we want to build this "deterministic" part first and then use it as context to the LLMs.