Hacker News new | ask | show | jobs
by PaulHoule 1528 days ago
Yeah, this is almost the sentiment behind the snarky comment I made that got voted down.

The difference is I am not so pessimistic about those UI-based ETL systems because I did a deep dive on the architecture of those and even worked for a startup that built one. I've seen how the industry's fixation on raw performance has kept ETL systems using a relational model that is hell to program for complex jobs -- I pitched a next generation system to a very acquisitive leader in this space and got shot down because my system wasn't friendly to storage locality and SIMD instructions. (That startup had the right idea of passing 'JSON' documents between the lines and the boxes but never settled exactly what the algebra was for those things, how to close event pipelines in the end properly so you always get the right results, ...)

Those "integration builders" though strike me a way of replacing a 10 line Python script with incredible tedium and "you can't get from here to there" experiences. That leads me to say things like "When I hear the word 'integration' I reach for my gun."