[TheRealKing]

Your benchmarks are not concrete and realistic. I have never seen any language but (unsurprisingly) Julia, to compare its performance with the performance of other languages by calling those languages from within the host language. Definitely 15000 times faster than MATLAB as claimed by JuliaComputing can be achieved with such seemingly concrete and realistic benchmarks. LLVM has benefited and learned so much from Fortran and its compilers. But free food and service are always undervalued.

[ChrisRackauckas]

> Definitely 15000 times faster than MATLAB as claimed by JuliaComputing can be achieved with such seemingly concrete and realistic benchmarks.

I believe you're talking about NASA Launch Services engineers claiming Julia's ModelingToolkit simulations outperformed Simulink by 15,000x? That claim was of course not made by Julia Computing or anyone affiliated by Julia Computing, which is pretty clear because the person who makes the claim very clearly describes his affiliation at the beginning of the video. The source is here: https://www.youtube.com/watch?v=tQpqsmwlfY0, at 12:55. You did watch the whole video to understand the application and the caveats etc. instead of just reading the headline and immediately coming to a conclusion, right?

[adgjlsfhk1]

The benchmarks on Julia's website are ones that take around 1 second. The couple nanoseconds of call overhead doesn't matter.

[TheRealKing]

It is not just about call overhead. It is about a whole suite of aggressive optimizations only possible for a whole program. Point to one person or entity in the world who calls SUNDIALS Julia wrapper to bind their C production code to SUNDIALS. If you cannot, you have two options: 1. make your Julia benchmarks concrete and realistic or 2. cease and desist from pointless advocacy of your employer (JuliaComputing) and its benchmarks in public forums.

[ChrisRackauckas]

Sure you can keep moving goal posts. Of course it doesn't make sense to bind a C production code to a C package (SUNDIALS) through Julia. But if you're asking who is using Julia bindings to SUNDIALS as part of a real case, one case that comes to mind is the Sienna power systems dynamics stuff out of NREL (https://www.nrel.gov/analysis/sienna.html). If you look inside of the dynamics part of Sienna you can clearly see IDA being used (https://github.com/NREL-Sienna/PowerSimulationsDynamics.jl). IIRC at a recent Julia meetup in the Benelux region kite model simulations also used it for the same reasons (https://github.com/aenarete/KiteSimulators.jl) which of course is pointing to the open source code organization for Aenarete (http://aenarete.eu/).

The way to find other use cases is to look through the citations. Generally there will be a pattern to it. For cases which reduce to (mass matrix) ODEs FBDF generally (but not always) outperforms CVODE's BDF these days, so those cases have mostly converted over to using the pure Julia solvers. This includes not just ODEs but also other DAEs which are defined through ModelingToolkit, as the index reduction process generates ODEs and generally the ODE form ends up more efficient than using the original DAE form (though not always of course). It's in the fully implicit DAE form that the documentation (as of May 1st 2023, starting somewhere back in 2017 according to the historical docs) recommends using Sundials' IDA as the most efficient method for that case (https://docs.sciml.ai/DiffEqDocs/stable/solvers/dae_solve/) (yes, the docs recommend non-Julia solvers when appropriate. There's more than a few of such recommendations in the documentation). Power systems is such a case with Index-1 DAEs written in the fully implicit form which are difficult in many instances to write in mass matrix form and not already written in ModelingToolkit, hence its use of IDA here. By the same reasoning you can also search around in the citations for other use cases of IDA.

[TheRealKing]

Our discussion will continue as long as one side believes fair, realistic benchmarks are merely moving the goalposts. Your benchmark has a severe fundamental flaw, especially given the tiny reported runtimes. I hope you realize and fix it before other critics (perhaps more credible than an unknown forum contributor) begin to question your programming knowledge or fairness. To address the matter, you must compile/write whole programs in each of the respective languages to enable full compiler/interpreter optimizations. If you use special routines (BLAS/LAPACK, ...), use them everywhere as the respective community does. Apples to Apples.

[sdfghswe]

> before other critics (...) begin to question your programming knowledge

You must be a truly special kind of stupid to be questioning Chris' knowledge.

[TheRealKing]

Typical Julia programmer/employee response when pointed to their flaws. Arrogant, rude, Young and inexperienced, irrational.

[ChrisRackauckas]

What about the other benchmarks on the same site? https://docs.sciml.ai/SciMLBenchmarksOutput/stable/Bio/BCR/ BCR takes about a hundred seconds and is pretty indicative of systems biological models, coming from 1122 ODEs with 24388 terms that describe a stiff chemical reaction network modeling the BCR signaling network from Barua et al. Or the discrete diffusion models https://docs.sciml.ai/SciMLBenchmarksOutput/stable/Jumps/Dif... which are the justification behind the claims in https://www.biorxiv.org/content/10.1101/2022.07.30.502135v1 that the O(1) scaling methods scale better than O(log n) scaling for large enough models? There's lots of benchmarks on that site which show things from small to large. And small models do matter too...

> If you use special routines (BLAS/LAPACK, ...), use them everywhere as the respective community does.

It tests with and with BLAS/LAPACK (which isn't always helpful, which of course you'd see from the benchmarks if you read them). One of the key differences of course though is that there are some pure Julia tools like https://github.com/JuliaLinearAlgebra/RecursiveFactorization... which outperform the respective OpenBLAS/MKL equivalent in many scenarios, and that's one noted factor for the performance boost (and is not trivial to wrap into the interface of the other solvers, so it's not done). There are other benchmarks showing that it's not apples to apples and is instead conservative in many cases, for example https://github.com/SciML/SciPyDiffEq.jl#measuring-overhead showing the SciPyDiffEq handling with the Julia JIT optimizations gives a lower overhead than direct SciPy+Numba, so we use the lower overhead numbers in https://docs.sciml.ai/SciMLBenchmarksOutput/stable/MultiLang....

> you must compile/write whole programs in each of the respective languages to enable full compiler/interpreter optimizations

You do realize that a .so has lower overhead to call from a JIT compiled language than from a static compiled language like C because you can optimize away some of the bindings at the runtime right? https://github.com/dyu/ffi-overhead is a measurement of that, and you see LuaJIT and Julia as faster than C and Fortran here. This shouldn't be surprising because it's pretty clear how that works?

I mean yes, someone can always ask for more benchmarks, but now we have a site that's auto updating tons and tons of ODE benchmarks with ODE systems ranging from size 2 to the thousands, with as many things as we can wrap in as many scenarios as we can wrap. And we don't even "win" all of our benchmarks because unlike for you, these benchmarks aren't for winning but for tracking development (somehow for Hacker News folks they ignore the utility part and go straight to language wars...).

If you have a concrete change you think can improve the benchmarks, then please share it at https://github.com/SciML/SciMLBenchmarks.jl. We'll be happy to make and maintain another.