[akater]

> Mathematica is a desktop product, specifically an IDE for the Wolfram Language

Presenting Mathematica (or, rather, FrontEnd?) as IDE is harmful for its reputation among hackers:

1) Workflow differs from one in traditional IDEs, and lots of build tools are simply not there

2) It has different goals, being a bottom-up environment: users start with tiny one-cell programs; IDEs like Eclipse suggest to “create a new project” by default

It's a tool for discovery and improvisation, not for building large programs, with multi-modular source code provided by different independent authors, that is sometimes (often?) written in different languages.

“Improvisations and Experiment Environment” or something like that would be a more adequate description.

It's not that FrontEnd can't be an IDE too. I believe it can be a better IDE than anything else out there but experienced programmers disagree unanimously. Part of it is personal tastes, part is mind being closed by means of classical IDE heritage, part is not-so-rational disgust towards proprietary software, paid software, S.W., etc.

FrontEnd and the language itself encourage users to add features themselves. Those who persist probably end up with the highly personalised product the majority of them could never buy. (+) Still, some features would be universally needed, so one might want to install a package from a public hub, but FrontEnd lacks this feature. (–)

I think hackers would appreciate it more if Mathematica (FrontEnd?) were marketed as emacs-like tool. Still, without a proper package manager (and an open package server, or at least an open standard for package server) it would be a hopeless take. — Though I heard something like package manager is coming. :-)

[taliesinb]

You're right, it is more of an emacs-like tool. Probably in the next year or so we will consolidate some ideas of what an IDE should (or could) be for WL. Some Brett Victor like things should be in there, no doubt...

And yes, we need to get PacletManager into a state where it is easy to use as a package manager. The only thing (and a big, crucial thing) it is missing right now is explicit dependencies, and a solver. If you have ideas about this please offer them (are you on the Mathematica stack exchange?)

[akater]

I'm at mma.sx, yes, and I love the community. :-) https://mathematica.stackexchange.com/users/1859/akater I'm interested in the topic and discuss it sporadically in chat but can never find enough time to give it the attention it deserves. (For now.)

Dependencies hell is a tough nut, yes. Though, every language has this problem, AFAIK, and the more high-level the language is, the more perfect solution it tries to implement, it seems. :-)

But unfortunately I don't have any ideas for now, due simply to lack of empirical evidence: there's yet no stack of independently developed packages meant to interact with each other, to experiment with. One probably can't derive successful solution purely from general [even if expert] understanding of contexts, packages and shadowing in WL; real-life practice should be taken into account which is almost non-present for the time being.

[szhorvat]

The Front End isn't very good at taking the job of an IDE ...

But it is a very good notebook interface (doesn't the notebook concept come from Mathematica originally?). Notebook interfaces are excellent for interactive work (a large part of scientific computing) and improvisational programming. It's a pity that they're only becoming popular recently (IPython). Many people still don't get it what notebooks are really good for. For example, the R community seems to think that a "notebook" is for report generation, and none of the "notebooks" for R allow any interactive work (knitr).

What's good about the notebook interface is the workflows it allows. Even MATLAB adopted "cell mode" a few years ago which enables the same basic workflow (except it does't have the equivalent of output cells, so it's not quite as good).

[akater]

> doesn't the notebook concept come from Mathematica originally?

Yes. It stems naturally from Lispy core of Wolfram language, IMO. Hypothesis: whenever you try to develop a user-friendly environment with a Lisp-like language, given enough effort and being consistent, you end up with notebook interface. :-)

> The Front End isn't very good at taking the job of an IDE ... > But it is a very good notebook interface

Actually, notebook interface is the key here, I believe.

How do people start writing large programs, anyway? They invariably start with small things (~ “evaluating Plot[…] in one cell”). Then they join communities, accept certain practices and fashions, and pick the tools that dominate the community.

Workbench (Eclipse + Mathematica Kernel caller + some build tools) is a universally recommended solution for large programs written in Mathematica. Having no experience with programming, I honestly tried it, used for several months only to come to believe people only recommend it because of fashions and traditions currently dominating the area. There was absolutely no benefit in using it for a [relatively] large project instead of a bunch of notebooks with their cells properly and consistently (and automatically, of course) tagged. I deploy, call unit tests and show project directory tree, with one-word command for each task, it all happens inside FontEnd, and I have no idea why people repeatedly try to float away from notebook interface.

I don't have years of experience, and may be stupid, but I strongly suspect it's only due to old habits, community influence or plain unwillingness to innovate when you have traditional environments at hand.

Multi-language projects could present a challenge, though (given that we have to represent text with boxes and not pure strings), but it can be overcome as well, if only people invested their time and effort.

[szhorvat]

Notebook interfaces are nice for interactive work (I love the concept and advocate for it), but they're no replacement for an IDE for writing packages. I spend most of my time in the notebook interface, but when it comes to writing a polished package, I use an IDE or a separate text editor. (Check out http://www.mathematicaplugin.halirutan.de/)

Having worked on a medium-large project collaboratively (http://matlink.org/), I can't imagine how we could have managed without using plain text files and version control (notebooks currently cannot be efficently used with version control systems).

IDEs are not used just out of habit. They have their uses, and notebooks can't replace them, just as IDEs or simple REPLs can't replace notebooks.

[akater]

> Check out http://www.mathematicaplugin.halirutan.de/

I'm aware of this plugin. That's exactly what I'm talking about. People obviously put lots of thought, effort and care into it. If it was invested in modifying notebooks, I argue, the results wouldn't be less impressive.

The only difference between notebooks and traditional editors is the latter being strings-oriented; the former being trees-oriented.

So, what exactly can a traditional IDE (or a command line, for that matter) do that notebook can't? :-)

> I can't imagine how we could have managed

> without using plain text files and version control

I haven't yet collaborated, that's important, of course. Still, I have plain text files that can be version controlled, and this doesn't seem to have anything to do with interface. I never edit them directly. Programs' text and metadata (comments, preamble, datestamp, sections markup) are all compiled into package files from notebooks. Git will diff as it does usually, so your colleague doesn't have to use tree-oriented editors. There's even a default feature like that in FrontEnd but the functionality is very basic and it's not meant to be extendable.

I don't collaborate, so I don't have a package to notebook converter but there's nothing conceptually difficult about it. You import someone's changes to a local notebook (or to a newly created copy of it): your drafts, experiments, test results and overall interactive canvas all stay, only definitions become modified. Drafts and experiments from collaborators' notebooks could be merged, as well. Notebooks don't have to be a part of version control at all.

Again, it's not conceptually difficult to write a version control aimed not at strings but at cells, or maybe cells with datestamps. If I want to merge notebooks I'll have to implement that. — An ad hoc merger I've written right now is merely 4loc, though.

The question at its lowest level is very simple: what is a better medium to organise data (and code — which is the same, in our case), strings or trees?

[szhorvat]

"You import someone's changes to a local notebook" <-- there are no existing tools that reliably automate that. (I'm aware of the notebook diff utility in the AuthorTools package, but that's not reliable). There are many tools if you're working solely with plain text.