What I'm working on right now is a perfect example. Mission: to write a Python program that parses. a file format that has no documentation. Create a folder for the project along with a data directory inside that has some of the files I need to parse. VS Code is my tool of choice, with the Python extension. Use its ability to run Jupyter to play with opening the files and parsing their contents. Experiment with some Jupyter. Get something running. Take my code and paste it into a module. Save the module, which I install locally in develop mode. Import the module into the notebook. Try what I just did against the file to make sure everything works nicely together. Keep going until the file format is completely figured out. Go on to the next challenge.
Frankly I HATE web based notebooks. I do deep learning research, and I write a lot of complex functions to handle pre-processing. And with that there's a lot of debugging I need to do, particularly visual debugging of images. The interactivity of notebooks and the features of an IDE has made my programming experience much better.
I'm curious, what interactivity do you enjoy about notebooks that you find missing in IDEs? I find my situation to be quite the opposite. I love developing/writing code in Pycharm because apart from all the things it excels at (introspection, auto complete, linting, documentation, debugging, etc), it lets me easily prototype things in the interactive python interpreter (usually set to IPython), has an excellent variable viewer and even allows you to attach a debugger to your python interpreter and debug things on-the-fly.
I find myself able to prototype code a lot faster in that setting compared to a Notebook. Even with plots, plotting in Pycharm provides me an interactive plot I can zoom around in, which is not the default behavior in Jupyter. The only situations where I ever use a Notebook is when I need to share a Python "document" that isn't just a python script + results, but rather some form of extensive self-documenting code that could benefit from Markdown integration and co-location of output plots, etc. The other situation is some of their built in animation/interactive widgets (perhaps that is the interactivity you are referring to?). In most other cases, I've found it quite limiting compared to a good IDE like Pycharm/Code.
That being said, I think the efforts on Pycharm and Vscode's part to bridge this gap is commendable and quite interesting.
Microsoft Visual Studio level of Edit and Continue has never been well implemented in open source tooling (with the exception of certain Common Lisp implementations that predate even visual studio, I suspect Turbo Pascal might have this feature too but I have never really used it). This is why most people have to resort to hacks like notebooks and repls with bad UX that spend most of the time spewing errors at the user. It is starting to come back with Flutter, React, and all the livereload stuff but you need a ton of work to get it to work on Python.
If I'm being honest, I haven't really mucked around pycharm. My primary hate really is using a chome/firefox/safari browser window for the notebook interactivity.
Personally, I use Atom with Hydrogen backend, but I develop the files in sch a way that they tend to end up as working for both interactivity and for scripts. So say my train_keras_script.py works both to train when it's deployed on our GPU server, and works as my primary debugging script.
I tend to throw most of my results into a .png output files and .csv's, the latter of which I use for R analysis scripts (and yes I know R Studio has markdown, but I'm just stubborn I suppose ).
In terms of IDE bells and whistles, I think #1 is certainly autocomplete. I will admit that I can be somewhat of a crummy speller. When first learning to program, it caused me a lot of headache trying to debug. Simple syntax checking and autocomplete of variables (especially long variable names that are meant to be descriptive) made me really embrace programming professionally.
The IDE (I also use PyCharms) is great for editing code (search, refactoring, syntax highlight...)
Jupyter notebooks (or JupyterLab in my case) is great for prototyping and data visualisation.
In my work (data engineering and training deep neural nets) I also happen to use both. It's unwieldy, but I can't get rid of either one (pycharms or jupyter lab)
I feel like this is the only sane way to go about things. I really don't understand people using notebooks to actually develop code and train ML/deeplearning models. A real IDE like Pycharm is so much more powerful at debugging, refactoring and in-general, writing good code, structuring it, and not to mention, being able to version control it in a sane manner.
I keep trying VSCode and Jupiter Labs every so often and yet I keep going back to the old Jupiter Notebook web UI. It’s getting better but it’s still not as good. For example, just the way VSCode displays a Pandas dataframe when you do something like .head() just isn’t as nice as Jupiter Notebook’s output. Anecdotally I’ve been developing software for 2 decades from C, Java, all the web crap, using Vim, IDEs, etc. and I _still_ prefer Jupiter Notebooks (not Labs, it sucks) over anything in terms of just how quickly you can iterate. With Vim keystrokes plugin and Black plugin I feel very comfortable.
The problem with notebooks is that they are not searchable and tracking code changes and creating libraries from them is not straight forward.
My approach is to combine the strengths of both IDE and notebooks. I put my code into a package that I edit with PyCharm. I import that code with the `%autoreload 2` magic in Jupyther to load data and test my functions. This way I get data persistency and plotting capabilities of the notebook and editing, etc capabilities of the IDE.
I do something similar, but I also check in my notebooks with all the output cells cleared. It’s not perfect but I can see what changes between commits.
While most of my code is in my modules, I still like having the actual notebook versioned.
I'd compare it loosely to taking notes by hand on a tablet. There is benefit in both, when they are combined it can be a better tool than either alone.