[grp000]

Doing data science in an IDE would be terrible. With a notebook, you get the chance to load the data, view it, clean it where needed, view it again, analyze it, model it and do anything else you need to it. An IDE means that you can't use the previous output to guide your next operation in a direct fashion like you can with a notebook.

[vnpc1]

> With a notebook, you get the chance to load the data, view it, clean it where needed, view it again, analyze it, model it and do anything else you need to it.

In a good data-oriented IDE like RStudio you get to do all of those things and write code which can be saved as plain text and can be version controlled well under git which you can't do well with Jupyter.

R folks have to be the best indicator in this case because they have access to a good IDE and they have good support for Jupyter. Their use is overwhelmingly in plain text files in RStudio, a small portion of rmarkdown notebooks and pretty much no one user R in Jupyter.

[nestorD]

Yes! Rstudio is the one thing I miss most when doing datascience in python.

Notebooks give me some of the interactivity but the experience degrades significantly.

The spyder IDE seem like an okayish replacement but some of the library I use expect you to have html display (within a notebook) to give you full functionalities which is not yet available in spyder.

[tastyminerals]

Have you tried Orange. It has scripting capabilities.

[nestorD]

No but, looking at some screenshot + descriptions, it seems to get me further from the code which does not seem like what I am looking for

Rstudio gives you the experience of a classical IDE + easy data exploration which I found to be productive from the exploratory stages (where I need to see my data and the effect of my code) to the clean-up phase (where I refactor my file).

[bllguo]

as a counterpoint, plenty of R folks are pretty happy doing all of that in Rstudio

[meztez]

It is interesting to see this discussion about notebooks while I'm thinking about all the RStudio users who do all their work inside the IDE and are pretty happy. Notebooks seem like such an inferior tool to me. I'm also extremely bias.

[arminiusreturns]

Also lots of emacs users of org-mode as an awesome notebook.

[kyllo]

I'm an R folk, and I'm even happy doing all of that in Emacs!

[tastroder]

That kind of depends on your process. In many cases pdb (or the debugging interface in your IDE of choice) works just fine for that. It's certainly not "terrible".

After the exploration and preprocessing stage I personally don't see much benefit of the notebook model, training/evaluation and any meaningful visualization takes forever anyway, that means I need to cache and persist intermittent results. With that it doesn't really matter all too much if I work on it in vim&pdb, an IDE, or Jupyter.

[C1sc0cat]

Maybe thinking about the data and what your trying to do before coding might be an idea as well.

[aldanor]

'Thinking about the data' most often requires looking at the data from hundreds of different angles, quickly investigating its properties and statistics, maybe plotting or fitting a few things, checking some hypotheses etc (all of the above code you will most likely throw out after the initial stage).

Same with the results - once you've coded something (perhaps outside of a notebook environment) and obtained results, verifying that they are what you expect is much more efficient to do in a notebook.

[comment_guy]

Maybe you use a notebook I'm completely unfamiliar with, but my experience is that they allow you to write code, run it, and save the results in cells. My IDE does all of that except the saving of partial results part, but this can be done easily by just dumping your precomputed data to disk if you can't recompute it easily. In either case, an IDE gives you get an actual debugger, plus with IntelliJ it has a great data visualization plugins, database viewer, great autocompletion, and integrates with your VCS, etc. What do you do when you need an actual debugger, or need to profile your code? What about documentation for the function you are calling? In my IDE this is a popup, in every notebook I've used, this is a google search.

[aldanor]

I use both PyCharm and JupyterLab on daily basis, typically dealing with multi-gb datasets.

If I'm writing a library or adding new features to one, or writing tests I'll use PyCharm sure thing, otherwise the notebook is a quicker way to sketch prototypes and always have a kernel with preloaded datasets and pre-imported stuff ready at hand. I don't want to wait 10 minutes to just load the data every time I want to check if my new function works well on it at big scale. That's one of the most important bits.

PyCharm is a clear winner at actually writing code that you won't throw in the bin 10 min later, and once you know what to write.

Debugging? Don't remember ever using PyCharm despite the fact it exists... either pudb or python-devtools or something else. I'd just write tests and things start working in the process. And btw you have pdb debugger (some weak version of it) in jupyter if you really need it. Docstrings? Press tab twice in the notebook. Or keep PyCharm open on the side so you can cmd-b. Profiling? Never a pycharm builtin, maybe something like flamegraph but an external tool anyway.