Hacker News new | ask | show | jobs
by optimali 3539 days ago
edit: local[1] has been updated to local[N], thank you for the update!

Ok thanks, I didn't know that's what "local[1]" did, so the more relevant comparison would be with --master local[30]?

The algorithm took around 500 seconds to train on the NETFLIX dataset on a SINGLE processor, which is good for data as large as 1 billion ratings. - this is from the sequential portion of the test, the parallel portion is the next section.

1 comments

That was a typo in the blog post. If you look at the graph, with more cores, spark gets faster as does Julia. The typo is now fixed.
Thanks for the update. The typo had me misinterpreting things. Now it makes more sense.

Assuming you're part of the team? Keep up the good work.