[ccvannorman]

This is awesome!

I have been warned that LINQ is often inefficient compared to raw C# with the same functionality -- but I have my doubts as the MS C#/.net put quite a lot into LINQ efficiency under the hood. Did you test this against any raytracing setups that don't use LINQ?

[wenc]

LINQ isn't really inefficient as such -- that's not really how I would put it. When used correctly, the performance is pretty good and it buys you elegance and a massive amount of expressive power with relatively little performance tradeoff. I use LINQ everywhere in my code and the maintainability gains are palpable.

It does have overhead, but it shouldn't be avoided for that reason unless you're really trying to squeeze performance gains out of your code. Eric Lippert, the designer of LINQ, has this to say [1].

[1] https://stackoverflow.com/questions/3769989/should-linq-be-a...

[cm2187]

Though my understanding is that if you do something like this:

var a = new double[10000];

var b = a.Select(x=>x*x).ToArray();

I'd expect LINQ to create a List or a similar structure when executing the select, then copy the output to the final array. If you were operating on two arrays directly, I would expect less memory allocations (unless these things get optimized away).

[thecopy]

No. The Select-statement is not evaluated until necessary, i.e. until ToArray() is called.

[0xffff2]

While your comment is true, I'm not sure how it's relevant. `ToArray` is still operating on the `IEnumerable` produced by Select, which doesn't tell you that the Select is itself operating on an array.

The CLR does do some type checking to try to propagate the size if it can, but that's by no means guaranteed.

Edit: In skimming the source available on sourceof.net, it looks to me like the array doesn't actually get propagated far enough, so the construct in GP will in fact incur several unnecessary array copies.

[munchbunny]

I'm curious what you saw. My understanding was that it's mostly generators, i.e. it's done through yields, not extra allocated arrays.

[0xffff2]

It's `ToArray` itself that's the problem. Since it doesn't know how big the array will be in the first place, it has to allocate a small array, start iterating, copy everything into a bigger array each time it runs out of space, and finally allocate an array of exactly the right size and copy everything into that.

You can see the `ToArray` implementation at [0], which defers the implementation to [1]. The implementation checks for ICollection to get the exact size, but the type [2] that `Select` returns doesn't implement ICollection, so `ToArray` has to fall back to the less efficient algorithm.

[0] https://referencesource.microsoft.com/#System.Core/System/Li...

[1] https://referencesource.microsoft.com/#System.Core/System/Li...

[2] https://referencesource.microsoft.com/#System.Core/System/Li...

[wenc]

Ugh I made a mistake -- Eric Lippert isn't the designer of LINQ. That would be Erik Meijer [1].

[1] https://en.wikipedia.org/wiki/Erik_Meijer_(computer_scientis...

[moron4hire]

Of course I'd never use LINQ inside a tight loop in the rendering path of my Unity3D projects, that's not what it is built for, but in my setup and project management code it is a god-send for expressiveness and maintainability.

[munchbunny]

LINQ doesn't optimize that much. If you're using LINQ on a database, you may benefit from the underlying database and driver's optimizations. However, if you're just operating on lists and in-memory data structures, you are probably eating the overhead.

As an example, a .Where(...) on a List<...> object is exactly what it claims to be: a linear pass through the list. Sometimes that's exactly what you need. Other times, it's more elegant and fast enough to represent an operation as a list comprehension. Finally, sometimes you really do need to eke out more performance. It's only the last case where LINQ is a bad idea.

[rejschaap]

> As an example, a .Where(...) on a List<...> object is exactly what it claims to be: a linear pass through the list.

Yes, but it uses deferred execution, which means that it returns enough information to perform the filter, it doesn't really do much until you start to enumerate the list. If, for instance, you continue to do a Take(5) on the result, the filter is only applied until it finds 5 elements that satisfy the filter (or until there are no more elements).

[Analemma_]

The thing about Linq efficiency is that under the hood it’s setting up a state machine with enumerators and continuations to be fully general. It’s about as fully-optimized as it could possibly be given the design constraints, but it will always be less efficient than just iterating over a collection directly.

But the worst performance hits in Linq come when people don’t know how to use it. The biggest culprit is unnecessary materializations, e.g. ToList() or anything that requires traversing the entire enumerator like All(), so if you’re careful to avoid the obvious pitfalls you should be fine.

[jameshart]

That's... not really true of LINQ, specifically. I have seen this assumption from developers before, with regard to LINQ-to-objects, that 'the compiler does clever stuff to optimize LINQ expressions', but... no, not really. It compiles a series of method calls. The JIT might do some optimization. But it's not a query optimizer - it just does what you tell it, and underneath it all there's nothing but an enumerator.MoveNext() getting called repeatedly.

Setting up state machines and continuations is something C# does when you write generators (with yield) and async methods (with await), but LINQ doesn't do that, particularly.

LINQ the language feature in particular doesn't do any of that sort of thing. As a language feature, LINQ is just the 'query comprehension' syntax, which is the 'from x in y where bar select z' coding form. And from the language's point of view that is just a different way of writing y.Where(x => bar).Select(x => z), which it will then try to compile. If it happens to wind up calling generator methods which implement state machines and continuations, or just simple methods that return ordinary objects, LINQ doesn't care, so long as the types all line up.

The only other thing that confuses matters with LINQ is that those lambdas it wants to compile can match method signatures that have delegate types, or Expression<> types, in which case the lambda doesn't get compiled as executable code, but rather as an expression tree literal, which is how Linq to SQL and friends were built (but again, that's not a 'LINQ' feature - you can assign or pass a lambda as an Expression<> yourself without involving 'LINQ'). And THAT is where LINQ gets a lot of its worst reputation, because that turns out to introduce a ton of complexity and failure modes that aren't the fault of the C# language, except in as much as it made it possible for a library to get involved in interpreting your code's syntax tree at runtime, which may have been a mistake.

[davidgl]

You can always use LinqOptimizer to improve performance: http://nessos.github.io/LinqOptimizer/index.html