[brummm]

I don't think Jupyter notebooks should be used for automated jobs. They're great for exploratory stuff but once things are getting fleshed out and cleaned up, one should move to proper python files that can be unit tested and versioned without having to go to crazy lengths...

[_coveredInBees]

To be honest, I've always been puzzled with how extensively people use Jupyter notebooks, despite how deficient they can be in many situations. Don't get me wrong, I have used them a lot and I still use them on occasion, but pretty much the only times I would really reach for them are when I want code + documentation to reside together.

So if I am prototyping things that also would benefit from Markdown + Latex comments/documentation, it is far superior to a script with text comments. The only other situation where I've found them to be pretty useful are when utilizing their interactive widgets to let the end user explore the data in interesting ways.

But most people seem to use it as an IDE when it is quite deficient compared to something like PyCharm. Even for quick prototyping, I find PyCharm to be far more useful because a) I can directly run things in the IPython console, b) Examine variables in the variable display widget, c) Attach a debugger to the console at any time and start debugging with an unmatched debugging experience in Python land, d) Easily have the correct venv be utilized by the IPython console, e) Have outstanding linting + code introspection + autocompletion, f) Have sane git diffs that make it easy to use version control appropriately and frequently unlike with Jupyter notebooks, and probably a bunch of other benefits that come with having access to a powerful IDE. All of these make rapid prototyping much faster than anything I can achieve in a Jupyter notebook.

I've also been pretty unimpressed with the quality of the average Jupyter notebook that I find on Github repos. They encourage dumping everything into global state, rely on state across code cells in non-obvious manners and in general result in ugly scripts that need a lot more work if they had to be refactored into modular packages/modules.

Running automated jobs from Jupyter seems a bit crazy and I hope people stop to think whether that is the appropriate path to take when trying to write automated jobs.

[the_watcher]

The reason Jupyter notebooks are so popular is that people who only learned to code or are learning to code to enable some kind of analysis can easily do it without needing to get much of anything else set up. If all you really need is to read in some data, do some statistical tests, and make some plots, I'd argue it's by far the simplest solution. Many academics don't need much more than this. Similarly, when a data scientist or product analyst is just doing analysis, not much more is needed.

Your comments about what Jupyter notebooks incentivize are definitely true (I'd argue its part of the appeal - people new to programming often don't immediately grok types of state and Jupyter kind of just says "eh, work around it"). I certainly fall victim to it and often wish for a week of "the expectation is that you will go through your notebooks and create modules or packages for them".

I also agree that running automated jobs from Jupyter isn't an optimal solution, but for many companies, if it means a data scientist (whose coding skills are primarily statistical) can get a report into production without a single engineering hour, it's often worth it.

[SmirkingRevenge]

Notebooks are a souped up repl, basically. So they might make sense anytime you find yourself turning to a repl. Its also nice to have that environment persist, so that you can pick up where you left off from another machine.

They also make nice front ends for spark clusters (and/or other big data engines).

I agree with the OP though. I wouldn't automate notebook execution as a part of any workflow.

[mlevental]

as someone that learned pycharm below jupyter I always wondered what people used them for. but your comment actually opened my eyes since very often when I'm implementing a numerical algo i want latex comments (rendered). i wonder how hard it would be to do just that in pycharm natively