[Quothling]

We've got a rather extensive AI setup through our equity fund and I've setup a group of agents for data architecture at scale. One is the main agent I discuss with and it's setup to know our infrastructure and has access to image generation tools, websearch, hand off agents and other things. I tend to use Opus (4-6 currently) and I find it to be rather great. As you point out it comes with the danger of making mistakes, and again, as you point out, it's not an issue for things I'm an expert on. What I rely on it for, however, is analysing how specific tools would fit into our architecture. In the past you would likely have hired a group of consultants to do this research, but now you can have an AI agent tell you what the advantages and disadvantages of Microsoft Fabric in your setup. Since I don't know the capabilities of Fabric I can't tell if the AI gives me the correct analysis of a Lakehouse and a Warehouse (fabric tools).

What I do to mitigate this is that I have fact checking agents configured to be extremely critical and non-biased on Opus, Gemini and GPT. Which are then handed the entire conversation to review it. Then it's handed off to a Opus agent which is setup to assume everything is wrong. After this, and if I'm convinced something is correct I'll hand the entire thing off to a sonnet agent, which is setup to go through the source material and give me a compiled list of exactly what I'll need to verify.

It's ridicilously effective, but I do wonder how it would work with someone who couldn't challenge to analytic agent on domain knowledge it gets wrong. Because despite knowing our architecture and needs, it'll often make conceptional errors in the "science" (I'm not sure what the English word for this is) of data architecture. Each iteration gets better though, and with the image generation tools, "drawing" the architecture for presentations from c-level to nerds is ridiclously easy.

[stasomatic]

Are you using this agent hive for any repeatable tasks? What you described, superficially, seems like a one off. Genuinely curious.

[Quothling]

I think it depends on what you mean by repeatable tasks. I reuse the critical handoff agents quite a lot since they are basically just set up to help spot bias and errors. I kind of reuse the top agent. I have a few "core" configurations that I can add to. So one will know our network, one will know our data architecture and so on, to keep them a little more focused. So for this specific agent that I described, I'll add a few lines on what I'm considering to the configuration, that I'll not reuse for anything unrelated to Microsoft Fabric. I've tried using these "core" agent configurations as hand-off agents in the past, but it doesn't seem to work well in our setup which is very isolated because we're NIS2 compliant.

I don't usually go back to the original prompt. I've actually done it a few times in regards to the presentation, to get some refined images but usually I'll start a new prompt.

[stasomatic]

In my previous jobby job I needed to pull CSVs out of Tableau, then from an ancient monolithic PHP admin and other sources, then manually merge them, reformat them in G Sheets, pivot this and that and send the report to my supervisor. Initially took 2 hours then down to one, but still senseless busy work. It was the “fault” of the incumbent IT, but if I could turn that hour into a minute… I wouldn’t get a raise, but I’d have more time for something else or nothing. I feel like this is still a scenario for countless many and perhaps the valley of the low hanging fruit. That’s where my question was coming from.

Your firm seems to operate on a higher plane, jealous :)