Hacker News new | ask | show | jobs
by _jonas 601 days ago
I'm excited for LLM applications that can setup, monitor/validate, and optimize data pipelines at scale. Seems possible soon given that SQL and most data records aren't intended to be human-friendly
5 comments

Ah, the hubris of youth.
Either this is a joke comment, or you haven't seen https://ludic.mataroa.blog/blog/i-will-fucking-piledrive-you...
Once again. Because clearly this needs to be said loudly and repeatedly.

That is a _technical solution_ to a fundamentally _social_ problem.

And now we have two problems...
Ten problems. Regex is for wusses who fear danger. LLMs are where real men turn.
When LLMs can do the following, they might be able to fix data hell:

- Negotiate with different teams to figure out what a field means

- Be told that a field should be converted from one format to another, but oh wait it's causing errors somewhere downstream because it was told the wrong instructions

- People come to you with some issue about the code you maintain, and you dig enough to realize the root cause is another team's code

- explain how they came up with “12” as the answer.