[tavert]

The reason I ran away from Julia and don't plan on ever using it again, and don't recommend anyone use it outside of academia, is that so much of the community is made up of grad students. So you get a lot of research code and people who have never been professional programmers maintaining most of the ecosystem. Julia Computing is largely made up of people they've hired from the community straight out of grad school.

[johnsea]

I don't see your point of academia and about hiring from the community?

What I see on Github is as professional as it can get. Issues, discussions, triage, review, CI-tests for example.

Maybe you started too early, before Julia was settled? And/or were too over-enthusiastic to begin with? I think Julia had to grow, find the 'correct' solution with e.g. NA/Missing/Nullable. Break things b/c it didn't work out as expected. Postpone things, debugger (maybe?), for more important areas or because base was not stable yet.

Two years ago in a project I hoped that people would switch immediately from R to Julia. But in retrospect it was good they didn't. Julia was not ready for them and too much ecosystem stuff missing/unclear still. (This said, Julia would in principle have been much much better suited for that project).

[tavert]

Things are decent on average, but there's a persistent carelessness and rush to do things without paying attention to the consequences. More in packages than base nowadays, but there's a lot of merging and releasing things immediately without waiting for code review that could have caught mistakes before breaking users.

[simonbyrne]

Out of curiosity: what language has a package ecosystem that in your opinion does do this right?

[tavert]

Large Apache projects, notable widely-used c++ projects like boost, llvm, zmq, cmake, the c++ language standards committee itself, all take their time and rarely if ever release changes/bugfixes immediately. Things go through review, testing, release candidates, and people other than original authors of code provide input before normal users get their hands on anything. The core pydata projects take their time and are cautious about breaking things.

[ScottPJones]

I complained also about the "cowboy" culture I saw among the Julia developers when I first started with it (people making a change directly to master, or merging there own PR without giving time for people around the world to review, or not having a minimum number of qualified reviewers before merging), but those days are gone, and I feel they've matured quite a lot in the past few years in that respect. Some of it I think was simply the great excitement that comes from being able to be so creative with the language, and a rush to get things figured out and nailed down to finally get to v1.0. As far as projects in other languages, I don't really feel it has much to do with the languages themselves, more the type of people that particular project attracts.

[tavert]

Just in the last few months BinDeps was broken by a "deprecation fix" that was completely wrong and using a name that didn't exist, and it got merged and released by the patch author before anyone else could look at it, breaking many downstream packages.

Refactorings and major changes in ZMQ.jl and the web stack similarly get merged and released immediately with zero review, still. This is a major problem.

Features in the base language have been deleted during 0.7-DEV because a single core developer didn't like them, despite multiple other core developers voicing disagreement that the features were useful and removing them was not urgent or necessary.

It's not a development culture I would rely on when products and money and jobs are at stake. Even the startup you were working with abandoned julia, correct?

[petre]

Dlang. But it's a compiled language and you have the option of statically compiling all your dependencies. The package manager is also quite simple and just works.

https://github.com/dlang/dub

[ChrisRackauckas]

SciPy's ODE solver doesn't have a stable contributor who contributes more than than about once a year even though it has many long standing issues, and PyDSTool hasn't had a commit in 2 years and doesn't work on Python 3 (and most of pycont is incomplete...). R's deSolve isn't even in a repository you can track and still hasn't fixed their DDE solver issues even though they directly mention in the docs how it's incorrect. So it's not like other open source software has strong maintenance structures....

[marmaduke]

SciPy solvers are mostly interfaces to existing established solvers, and I’ve not had any problems with them. We’ve also used PyDSTool without problem, and it appears to support Python 3,

https://github.com/robclewley/pydstool/blob/master/README.rs...

If you think these are poorly maintained you should see XPPAUT, a tool still quite widely used.

[Certhas]

It just sounds like your code doesn't require more than the occasional hotloop. That's fine then. There is no reason to leave numba.

If you have anything that requires more complications, numba becomes painful. You seem to somehow insist that your usecase is the only one out there. We are actively developing a scientific simulation library in Julia. The prototype was in Python+numba. The Julia code is vastly simpler, and that is because Julia is not "an interface to LLVM for fast loops". It's a full fledged language with performant abstractions, closures, inline functions, metaprogramming, etc. To get things fast in numba I ended up doing code generation (I talked to the Numba developers, it seemed the only way). Talk about brittle, painful and impossible to generalize.

Now we have Julia code, using sparse matrices in the hot loop is easy, Automatic Differentiation just works, etc...

The correct comparison for Julia is this context is C++, not Python.

[marmaduke]

I’m not insisting I have the only use case, but apart from the examples of traversing language boundaries, I haven’t see a good example of what’s so painful in Numba. What is so challenging that is requires code generation?

[Certhas]

I have a data structure based on which I generate a dynamical behaviour that I want to integrate. So I construct a rhs.

I further want the user of the library be able to pass it new functions that can be integrated into overall dynamical behaviour.

There are different ways to achieve this, the simplest version is with closures. Pass a list of functions, and some parameters and I construct a right hand side function from it. Unfortunately this does not work with numba. What I ended up doing is passing not the function itself but the function text to generate the code of the function to be jited and then eval that. It worked but it was horrible to maintain, and required users to pass function bodies as text witha very specific format.

Now in Julia we will probably eventually transition to a macro based approach, but the simple closure based model just worked.

Previously I had large scale, inhomogeneous right hand side functions that I wanted to jit in numba and that need sparse matrices. So I ended up having to implement sparse matrix algorithms by hand because I can't call scipy.sparse.

Another instance: I implemented a solver for stochastic differential equations with algebraic constraints in numba, partly to be able to use it with numba jited functions and get a complete compiled solver out of it. This already constrained my users to use numba compatible code in their right hand side functions.

In order to get this to work I had to implement a non-linear solver from scratch in numba rather than being able to use scipys excellent set of solvers.

Julia is not a magic silver bullet. Getting the ODE solvers to make full use of sparsity still requires some care and attention. But I simply spend a lot less time on bullshit than before. (so I have more time to spend on HackerNews :P)

I decided to switch over when for one paper I was able to implement a problem using the standard tools and packages available in Julia within half a day. The Python equivalent would have involved using a new library that came with its own DSL, which would have meant rewriting quite a bit of my code to take advantage of it. Easily several days work.

With DifferentialEquations.jl I also could just test half a dozen different numerical algorithms on a problem in a matter of minutes, find out which performed best and use that for MonteCarlo. Saved about a week of computation time on one project alone. That's not a critical amount, nobody cares if the paper comes out a week later or earlier, but it's nice (and I don't waste super computer time). With Python libraries with different DSLs this would have taken considerably longer, and I probably would not have done it. This is the result of having one library and interface rather than a whole bunch, if everyone agreed on scipys ode interface (which just got properly established in scipy 1.0.0) this would be easy in Python as well. But that's also the point that people have been making: Julias design for composition over inheritance makes it convenient to rally around one base package.

I also personally very much like being able to enforce types when I want to. This is a big win for bigger projects for us.

[marmaduke]

> With DifferentialEquations.jl I also could just

yep... I took a look at the DE packages in Julia today, and quite frankly they're much better than the situation in Python, perhaps because of one or more prolific applied mathematicians are making a concerted effort, which is lacking Python? I dunno, but I did recommend my colleagues look at Julia for DEs, for this reason.

That said,

> Pass a list of functions, and some parameters and I construct a right hand side function from it. Unfortunately this does not work with numba.

I'm pretty sure I've done this before with numba, so maybe getting concrete would help, e.g. an Euler step

    def euler(f, dt, nopython=False):
        @numba.jit(nopython=nopython)
        def step(x):
            return x + dt * f(x)

where user can provide regular Python function or a @numba.jit'd function. If a @numba.jit'd function is provided, and nopython=True, this should result in fused machine code. This sort of code gen through nest functions can be done repeatedly for e.g. the time stepping loop.

I've done this for CPU & GPU code for a complex model space (10 neural mass models, 8 coupling functions, 2 integration schemes, N output functions, ...) which, by the above pattern, results in flexible but fast code.

Is this a pattern that captures your use case or not yet?

> implement sparse matrix algorithms by hand because I can't call scipy.sparse.

agreed, this is a surprising omission, which I attribute to not much of the numerical Python community making use of Numba, but could be fixed rapidly.

> constrained my users to use numba compatible code in their right hand side functions

what did you run into that was problematic?

> I had to implement a non-linear solver from scratch in numba rather than being able to use scipys excellent set of solvers

I didn't follow; passing @numba.jit'd functions to scipy is in the Numba guide, so what exactly didn't work?

[mlthoughts2018]

This just sounds like bad software designto me. You are miswanting something overly generic that’s super not needed, and regardless of implementing in any given language, it sounds like it would benefit hugely from taking a more YAGNI approach to it, restricting its genericity based on likely usage (not intended or imagined usage), and either just manually writing stuff for an exhaustive set of use cases, or code genning just those cases and not allowing or encouraging arbitrary code gen of possible other cases.

I love it when libraries limit what can be done with them and document an extremely specific scope they apply to.

When libraries try to be all things to all people, it’s bad. A sophisticatedcode gen tool that enables library authors to choose to do that is a bad thing, not a good thing.

[ChrisRackauckas]

SciPy's solvers cannot handle events which are nearby, most return codes aren't documented, you cannot run the wrapped solvers in stepping control mode, you cannot give it a linear solver routine, etc. So it wraps established solvers but still only gives the very basic solving feature of it, and most of the details that made the methods famous are not actually available from SciPy's interface.

And it wasn't Python 3 for pydstool. It's SciPy 1.0.0. Some of the recent maintenance for this stuff has actually come from the Julia devs though:

https://github.com/robclewley/pydstool/pull/132

[marmaduke]

You mentioned Python 3, not me. Btw, I did look through your DE packages, and they are definitely an amazing contribution not seen in Python; I've recommend to colleagues.

[ChrisRackauckas]

>You mentioned Python 3, not me.

Yeah sorry, I was just acknowledging that I was wrong when I found the PR and noticed the mistake. I guess it come across oddly.

[Accipitriform]

> Julia Computing is largely made up of people they've hired from the community straight out of grad school.

Where do you think most companies get "professional programmers" from, exactly?

Julia's been designed and implemented by some very bright people, and it shows.

[jhayward]

Industry gets professional programmers by hiring people who have been hammering out shipping code in paying products for years, and years, doing support, maintenance, and new product development and research.

Grad students may be brilliant but that does not help give them any insight in to what makes a good ecosystem, toolchain, and feature set good.

[simonbyrne]

We would love to have more professional programmers contribute: unfortunately those 1-based indices put them off.

More seriously: part of the problem does seem to be that Julia does have some significant differences from "traditional" languages (e.g. the concept of a "virtual method" is a bit fuzzy in Julia, what we call a JIT is probably better described as a JAOT, whether it has a "type system", homoiconicity, etc.).

That said, this JuliaCon I have met a lot more people from and classical "programmer" backgrounds. So hopefully that is changing.

[ScottPJones]

I've seen quite an evolution over the past 3.4 years I've been using Julia and the 4 JuliaCon's I've attended so far. Back at the 2015 JuliaCon, a number of us "older" professional programmers felt like we should stage a palace coup, because it did feel like the input of people who had "been around the block" a few times was not really valued. That's changed quite a lot (maybe because in the intervening years many of the core contributors have gotten their Ph.D.s and are having to live off their blood, sweat, and tears (plus lots of joy, to be sure) of producing things with Julia that people will actually pay money for). Yes, it was young and brash, but those awkward years seem to be past, and I feel the future of Julia is quite bright.

[sampo]

How was it with the origin and design with Python, NumPy, Matplotlib, Pandas? Were the people who originated these projects in their time any more professional and seasoned than Julia people are currently?

[jhayward]

Well, if they'd been as brilliant as the GP indicates there would be no need for Julia, would there?

[simonbyrne]

Also, I was actually a postdoc...

[celrod]

I hope that if/as Julia gets adopted in industry, more libraries get written and maintained by professionals. If the language is successful, that may change.

Although AFAIK it hasn't really in the case of R, outside of tidyverse.

As a grad student without a CS background, I don't think I'm qualified to say much more on this.

[baldfat]

The issue isn't "Professionals" it is domain specialist. In the past data specilist didn't have Computer Science specialty. They were awesome in stats and numbers but lacked strong programming skills.

Hadley Wickham is special because he has both the stats, data science AND programming skills.

data.table is also an amazing library. R to me is the most improved language in the history of programming languages over the past 5 years.

Also R allows anyone with basic hackery R skills to create libraries easily and that is why so many of them are not optimal.