[sarchertech]

I just finished writing a Kafka consumer to migrate data with heavy AI help. This was basically best case a scenario for AI. It’s throw away greenfield code in a language I know pretty well (go) but haven’t used daily in a decade.

For complicated reasons the whole database is coming through on 1 topic, so I’m doing some fairly complicated parallelization to squeeze out enough performance.

I’d say overall the AI was close to a 2x speed up. It mostly saved me time when I forgot the go syntax for something vs looking it up.

However, there were at least 4 subtle bugs (and many more unsubtle ones) that I think anyone who wasn’t very familiar with Kafka or multithreaded programming would have pushed to prod. As it is, they took me a while to uncover.

On larger longer lived codebases, I’ve seen something closer to a 10-20% improvement.

All of this is using the latest models.

Overall this is at best the kind of productivity boost we got from moving to memory managed languages. Definitely not something that is going to replace engineers with PMs vibe coding anytime soon (based on rate of change I’ve seen over the last 3 years).

My real worry is that this is going to make mid level technical tornadoes, who in my experience are the most damaging kind of programmer, 10x as productive because they won’t know how to spot or care about stopping subtle bugs.

I don’t see how senior and staff engineers are going to be able to keep up with the inevitable flood of reviews.

I also worry about the junior to senior pipeline in a world where it’s even easier to get something up that mostly works—we already have this problem today with copy paste programmers, but we’ve just make copy paste programming even easier.

I think the market will eventually sort this all out, but I worry that it could take decades.

[awfulneutral]

Yeah, the AI-generated bugs are really insidious. I also pushed a couple subtle bugs in some multi-threaded code I had AI write, because I didn't think it through enough. Reviews and tests don't replace the level of scrutiny something gets when it's hand-written. For now, you have to be really careful with what you let AI write, and make sure any bugs will be low impact since there will probably be more than usual.

[skeeter2020]

> I’ve seen something closer to a 10-20% improvement.

The seems to match my experience in "important" work too; a real increase but not essentially changing the essence of software development. Brook's "No Silver Bullet" strikes again...

[electromech]

> My real worry is that this is going to make mid level technical tornadoes...

Yes! Especially in the consulting world, there's a perception that veterans aren't worth the money because younger engineers get things done faster.

I have been the younger engineer scoffing at the veterans, and I have been the veteran desperately trying to get non-technical program managers to understand the nuances of why the quick solution is inadequate.

Big tech will probably sort this stuff out faster, but much of the code that processes our financial and medical records gets written by cheap, warm bodies in 6 month contracts.

All that was a problem before LLMs. Thankfully I'm no longer at a consulting firm. That world must be hell for security-conscious engineers right now.

[murukesh_s]

What about generating testable code? I mean you mentioned detecting subtle bugs in generated code - I too have seen similar - but what if that was found via generated test cases than found by a human reviewers? Of course the test code could have bugs, but I can see a scenario in the future where all we do is review the tests output instead of scrutinising the generated code...

[sarchertech]

And the AI is trained to write plausible output and pass test cases.

Have you ever tried to generate test cases that were immune to a malicious actor trying to pass your test cases? For example if you are trying to automate homework grading?

The AI writing tests needs to understand the likely problem well enough to know to write a test case for it, but there are an infinite amount of subtle bugs for an AI writing code to choose from.

[LgWoodenBadger]

Complicated parallelization? That’s what partitions and consumers/consumer-groups are for!

[sarchertech]

Of course they are, but I’m not controlling the producer.

[LgWoodenBadger]

Producer doesn’t care how many partitions there are, it doesn’t even know about them, unless it wants to use its own partitioning algorithm. You can change the number of partitions on the topic after the fact.

[sarchertech]

In this case it would need to use its own partitioning algorithm because of some specific ordering guarantees we care about.

[LgWoodenBadger]

Then rewrite them to another topic. Nevermind, complex multithreading sounds like the better solution

[sarchertech]

There’s more to it than that. We don’t care about total order even within partitions. Every so often we get a message that must not be sent downstream until some subset of messages have been sent.

So most of the time we’re fine sending 100-200 parallel message batches, but sometimes we need to stop and wait for some batches to complete before sending any more.

We also want to control how hard we hammer specific resources downstream, which don’t correlate with the partitions we’d need. Additionally we want to scale up and scale down the parallelism per each of the previously mentioned resources depending on how fast they are coming in to maximize batch size (while keeping latency low).

There’s of course ways to do this with multiple partitions by having the consumers communicate with each other. But now we have added an additional consumer and topic to the pipeline, and an inter-consumer control system.

It was overall easier to have one consumer read from the existing topic and spawn goroutines, so that we can have more dynamic control, the ability to scale up and down immediately without worrying about rebalancing, and easy communication between threads.