[pvillega]

To sum it up (feel free to add corrections) one guy put a blog post saying that Parallel collections had many issues, etc etc... and the reality was that he misunderstood them somehow.

As one guy put it, the collections do nice work but you have to know what you are doing (not all work can be broken into fragments that can be computed in parallel). He was kind of expecting otherwise.

Not saying they are perfect, just what happened (more or less)

[jebberjeb]

I used them a bit with some image processing. Was awesome to see all 8 cores pegged w/ one line of code!

[microtonal]

That does not say much, without actually specifying how much data is processed. It's great if you can spread calculations among 8 cores, but if a single-threaded C process provides better performance (let alone a parallel C implementation), what does it say?

Could you provide more data points?

(By the way, in C such loops can typically be parallelized by adding one OpenMP pragma.)

[pivo]

> if a single-threaded C process provides better performance (let alone a parallel C implementation), what does it say?

It doesn't say much until you define what's most important to you. Is it more important that you can use Scala and get your code working more quickly or do you have more time to write and debug a C program and do you really need to do that for performance reasons? Does the code need to function as part of a larger Java/Scala or C application, etc. In other words, and as always, you need to pick to right tool for the job.

The JVM is very fast now days, in my work the issue of performance vs. C never comes up. Obviously there are still valid uses for C these days, but you can get very decent performance from the JVM and that often means you can finish your work faster.

[eternalban]

Java or Scala?

    list.par().map(f);

[dxbydt]

I've found that if the list is very large, using list.par.map(...) hits the memory limit & then simply hangs.

I process upwards of a million facilities ( a facility is the exposure on a commercial loan a bank makes to a client ), trying to forecast their expected loss over 12 future quarters.

I rewrote my code like so :

val facs:List[Facility] = ...populated via jdbc query ///

val N = 1000

facs.grouped(N).foreach( group => {

   group.par.map( fac => expectedloss(fac)

})

Now the outer loop is sequential but the inner loop is parallel. It chugs along, and processes the entire million plus facilities! I get some control via the N. Currently my N is 1000 and all 8 cores running at 100% capacity, but I can slow down the workload with smaller N.

[jebberjeb]

yep, that's what I did!