| Having worked with Makefile and pushed it to limits, I'd say there are some deficiencies in the system you may try to tackle: 1. Databases. We ended up calling `psql -f some_script.sql && touch $my_task_name` and tracking changes with touch files. (Putting this into database on views or materialized views proved to be unsustainable.) 2. Datasets. If you just open a sqlite file, it's changed, and GNU Make thinks you must rebuild everything downstream. Datasets are mostly treated as row-order-independent, so hashing them as is does not always work. 3. Very expensive tasks that shouldn't be called always. Like my makefile had a script that parsed a million web pages, going around captchas via Tor, and touching the upstream files was to be avoided -- or if it happened, I had to manually touch the target, to avoid re-running that part. 4. Some targets can be updated and the result will always be new -- e.g. run a query to a live database, or news website. Some may produce the same. Would appreciate if any system has such a distinction. 5. Surprisingly, lots of alternative build systems don't do partial update. They only update every item in the deps graph. If you manage to get any of these right, you'd be praised. |