[villasv]

If they were truly one-off, you wouldn't need to manage them. Categorize and document. SQL queries for analytics go into something like Metabase. Database migrations go into git. Frequent adhoc scripts go into something like Jenkins so you can trigger them easily. So on and so forth. All of those documented somewhere, like the team's Confluence/Notion/whatever.

But don't just leave them at that, do work on eliminating their need. Sisyphean task but necessary nonetheless.

[rolandtshen]

I'm curious about your workflow for reusable scripts with Jenkins. What are the advantages to putting it in Jenkins? How do you set up Jenkins to run these individual scripts?

[villasv]

Jenkins is not the ideal place. We take some scripts and create pipelines that just invoke them. Some of these are configured to be scheduled jobs. We have separate declarative pipelines for each one.

But this isn’t good, we have Airflow and that’s where these things should be. Their stay inside Jenkins is technical debt, but at least is much better than not having it automated. The advantage is that you made an ad-hoc process less unstructured, you can add post-failure alarms, you don’t have to worry about long running scripts being interrupted by shabby notebooks, you don’t need to spread production credentials to every developer and so on.