|
|
|
|
|
by skydhash
97 days ago
|
|
There still is. In most enterprises, the tasks are usually to take some data somewhere, transform it to be the intake of another process. Or to made a tweak to an existing process. Most of the other types of problems (information storage, communication, accounting,..) have been solved already (and solved before the digital world). People can see it as grind. But the pleasure comes in solving the meta problem instead of the one in front (the latter always create brittle systems). But I agree that it can becomes hell if there were no care in building the current systems. |
|
I just told it to do it.
It got the “create S3 pre-signed url to upload it to” right. But then it did the naive implementation of download the file and do a bulk upsetting wrong instead of “use the AWS extension to Postgres and upload it to S3”. Once I told it to do that, it knew what to do.
But still I cared about systems and architecture and not whether it decided to use a for loop or while loop.
Knowing that or knowing how best to upload files to Redshift or other data engineering practices aren’t knew or novel