[taliesinb]

Let me correct you: Mathematica is a desktop product, specifically an IDE for the Wolfram Language. It is not a language, though before we had the name "Wolfram Language" it was used to mean that. Other such IDEs are Wolfram Desktop, the Eclipse-based Workbench, and the cloud-based Programming Cloud that is being announced here.

And although I'm biased, I don't think it is unfair for our language to meter access to an expensive resource in the form of real world data. There are complex translation layers that mediate access to a diversity of different databases, real time and static, and merge them to present a nice, clean logical view.

On top of that, cleaning and curating the data is expensive, and it has to be done in perpetuity -- think how often PersonData (http://reference.wolfram.com/language/ref/PersonData.html) needs to be updated, for example. Some data we also have to buy outright.

Should this prevent us building such real-world data into the language? I suspect in 20 years this idea will be common in new languages, and probably free.

For now, it needs to be subsidized, and the basis is fair: we charge Cloud Credits on the basis of the amount of time our servers are kept busy: 0.3 cents per second of computation time on our API servers. As we make our APIs faster, that effective cost will only go down.

[akater]

> Mathematica is a desktop product, specifically an IDE for the Wolfram Language

Presenting Mathematica (or, rather, FrontEnd?) as IDE is harmful for its reputation among hackers:

1) Workflow differs from one in traditional IDEs, and lots of build tools are simply not there

2) It has different goals, being a bottom-up environment: users start with tiny one-cell programs; IDEs like Eclipse suggest to “create a new project” by default

It's a tool for discovery and improvisation, not for building large programs, with multi-modular source code provided by different independent authors, that is sometimes (often?) written in different languages.

“Improvisations and Experiment Environment” or something like that would be a more adequate description.

It's not that FrontEnd can't be an IDE too. I believe it can be a better IDE than anything else out there but experienced programmers disagree unanimously. Part of it is personal tastes, part is mind being closed by means of classical IDE heritage, part is not-so-rational disgust towards proprietary software, paid software, S.W., etc.

FrontEnd and the language itself encourage users to add features themselves. Those who persist probably end up with the highly personalised product the majority of them could never buy. (+) Still, some features would be universally needed, so one might want to install a package from a public hub, but FrontEnd lacks this feature. (–)

I think hackers would appreciate it more if Mathematica (FrontEnd?) were marketed as emacs-like tool. Still, without a proper package manager (and an open package server, or at least an open standard for package server) it would be a hopeless take. — Though I heard something like package manager is coming. :-)

[taliesinb]

You're right, it is more of an emacs-like tool. Probably in the next year or so we will consolidate some ideas of what an IDE should (or could) be for WL. Some Brett Victor like things should be in there, no doubt...

And yes, we need to get PacletManager into a state where it is easy to use as a package manager. The only thing (and a big, crucial thing) it is missing right now is explicit dependencies, and a solver. If you have ideas about this please offer them (are you on the Mathematica stack exchange?)

[akater]

I'm at mma.sx, yes, and I love the community. :-) https://mathematica.stackexchange.com/users/1859/akater I'm interested in the topic and discuss it sporadically in chat but can never find enough time to give it the attention it deserves. (For now.)

Dependencies hell is a tough nut, yes. Though, every language has this problem, AFAIK, and the more high-level the language is, the more perfect solution it tries to implement, it seems. :-)

But unfortunately I don't have any ideas for now, due simply to lack of empirical evidence: there's yet no stack of independently developed packages meant to interact with each other, to experiment with. One probably can't derive successful solution purely from general [even if expert] understanding of contexts, packages and shadowing in WL; real-life practice should be taken into account which is almost non-present for the time being.

[szhorvat]

The Front End isn't very good at taking the job of an IDE ...

But it is a very good notebook interface (doesn't the notebook concept come from Mathematica originally?). Notebook interfaces are excellent for interactive work (a large part of scientific computing) and improvisational programming. It's a pity that they're only becoming popular recently (IPython). Many people still don't get it what notebooks are really good for. For example, the R community seems to think that a "notebook" is for report generation, and none of the "notebooks" for R allow any interactive work (knitr).

What's good about the notebook interface is the workflows it allows. Even MATLAB adopted "cell mode" a few years ago which enables the same basic workflow (except it does't have the equivalent of output cells, so it's not quite as good).

[akater]

> doesn't the notebook concept come from Mathematica originally?

Yes. It stems naturally from Lispy core of Wolfram language, IMO. Hypothesis: whenever you try to develop a user-friendly environment with a Lisp-like language, given enough effort and being consistent, you end up with notebook interface. :-)

> The Front End isn't very good at taking the job of an IDE ... > But it is a very good notebook interface

Actually, notebook interface is the key here, I believe.

How do people start writing large programs, anyway? They invariably start with small things (~ “evaluating Plot[…] in one cell”). Then they join communities, accept certain practices and fashions, and pick the tools that dominate the community.

Workbench (Eclipse + Mathematica Kernel caller + some build tools) is a universally recommended solution for large programs written in Mathematica. Having no experience with programming, I honestly tried it, used for several months only to come to believe people only recommend it because of fashions and traditions currently dominating the area. There was absolutely no benefit in using it for a [relatively] large project instead of a bunch of notebooks with their cells properly and consistently (and automatically, of course) tagged. I deploy, call unit tests and show project directory tree, with one-word command for each task, it all happens inside FontEnd, and I have no idea why people repeatedly try to float away from notebook interface.

I don't have years of experience, and may be stupid, but I strongly suspect it's only due to old habits, community influence or plain unwillingness to innovate when you have traditional environments at hand.

Multi-language projects could present a challenge, though (given that we have to represent text with boxes and not pure strings), but it can be overcome as well, if only people invested their time and effort.

[szhorvat]

Notebook interfaces are nice for interactive work (I love the concept and advocate for it), but they're no replacement for an IDE for writing packages. I spend most of my time in the notebook interface, but when it comes to writing a polished package, I use an IDE or a separate text editor. (Check out http://www.mathematicaplugin.halirutan.de/)

Having worked on a medium-large project collaboratively (http://matlink.org/), I can't imagine how we could have managed without using plain text files and version control (notebooks currently cannot be efficently used with version control systems).

IDEs are not used just out of habit. They have their uses, and notebooks can't replace them, just as IDEs or simple REPLs can't replace notebooks.

[akater]

> Check out http://www.mathematicaplugin.halirutan.de/

I'm aware of this plugin. That's exactly what I'm talking about. People obviously put lots of thought, effort and care into it. If it was invested in modifying notebooks, I argue, the results wouldn't be less impressive.

The only difference between notebooks and traditional editors is the latter being strings-oriented; the former being trees-oriented.

So, what exactly can a traditional IDE (or a command line, for that matter) do that notebook can't? :-)

> I can't imagine how we could have managed

> without using plain text files and version control

I haven't yet collaborated, that's important, of course. Still, I have plain text files that can be version controlled, and this doesn't seem to have anything to do with interface. I never edit them directly. Programs' text and metadata (comments, preamble, datestamp, sections markup) are all compiled into package files from notebooks. Git will diff as it does usually, so your colleague doesn't have to use tree-oriented editors. There's even a default feature like that in FrontEnd but the functionality is very basic and it's not meant to be extendable.

I don't collaborate, so I don't have a package to notebook converter but there's nothing conceptually difficult about it. You import someone's changes to a local notebook (or to a newly created copy of it): your drafts, experiments, test results and overall interactive canvas all stay, only definitions become modified. Drafts and experiments from collaborators' notebooks could be merged, as well. Notebooks don't have to be a part of version control at all.

Again, it's not conceptually difficult to write a version control aimed not at strings but at cells, or maybe cells with datestamps. If I want to merge notebooks I'll have to implement that. — An ad hoc merger I've written right now is merely 4loc, though.

The question at its lowest level is very simple: what is a better medium to organise data (and code — which is the same, in our case), strings or trees?

[NotAtWork]

> Should this prevent us building such real-world data into the language?

You didn't build it in to the language, you connected the language to a standard database via the hosted, proprietary runtime.

I'm not against this in principle - it's clearly a useful thing - but it's hardly revolutionary.

I already pay a company for essentially the same thing (where I use a DSL embedded in a programming language to interact with their database of social media).

Again, I'm not against the idea as I understand it, but the misuse of terminology and generally the way that various people associated with the project talk about it is a huge turn off.

[taliesinb]

Entity is part of the documented language -- we've standardized the representation of millions of real-world things. Looking up properties of Entities and parsing text into Entities depends on Wolfram|Alpha, just like the ServiceExecute function relies on Twitter and Facebook to work, and just like Graphics relies on a front-end executable to actually render.

These aren't libraries as you might call (say) core.logic in Clojure, because Entity and co integrate tightly across the rest of the language, and because they all share the same System namespace. They aren't quite DSLs either, because if they were the entire language would be a DSL -- rather the language is symbolic, which gives it some properties in common with DSLs.

If there is nomenclature that is different or new, it is mostly because the language is different. The only thing I can't defend is the pure in our "pure functions". They're not pure functions, they are lambdas.

[NotAtWork]

Thank you for your reply.

Looking through examples, it still appears very much that domain specific calls were added to the very flexible language typically used in Mathematica to essentially extend computation to these new domains using domain specific widgets. (Also, some exchange data types.)

I'm still having trouble seeing how this is a radical departure from say, embedding a bunch of domain specific tie ins in a Lisp using macros (or similar constructs) to get a similar syntax across them.

I get that the constructs are likely tightly integrated, but I guess I'm still not seeing where it's more than a framework for dealing with particular classes of data running on top of a programming language that targets a hosted run time with a database of facts.

I'm not saying that this isn't a useful thing, and entirely worth it for well curated data. As I mentioned, I pay for essentially that service targeting a database about social media. I'm just trying to see if there's something key to understanding the features that I'm missing.

If you'll excuse one quasi-dig: it can sometimes be hard to see what something really is and what its use cases are through the hype and marketing.

[programnature]

Have you actually used the computable data functionality in Wolfram products?

If you think its a matter of just connecting to a standard database, you are missing the whole point.

People in the data business tell you cleaning and preparing the data is 80% of the work. Wolfram has done that for you. And then taken it farther: integrating it directly into language constructs.

I think SW's post did a poor job of explaining these capabilities, but in some domains this integration totally trounces what is possible in other systems. A great example of this is how GeoGraphics works:

http://reference.wolfram.com/language/ref/GeoGraphics.html

Notice how semantic entities like countries and landmarks are interoperable with the graphics language.

[zimbatm]

I think we're operating on a different definition of "language". Unlike natural languages a programming language has extension points to define new vocabulary, it doesn't mean that all available words are part of the language. Otherwise it's like saying that everything that is in perl's CPAN is part of the language.

Usually there is a distinction that's made where the language is the syntax + the semantic defined by the compiler or the runtime. Then comes the standard library, then comes the user libraries. The "wolfram language" seem to mash everything together into a single thing.

[NotAtWork]

In what way is anything in that example different than say, a DSL implemented in Lisp macros connected to a database?