[Gazoo101]

I think this statement at the end of the article - 'There is a common myth in software development that parallel programming is hard.' - is misleading. Granted the author denotes explicit situations where it is not hard, but if it's applicable in general, then it is hard. Not a common myth.

Is parallel programming hard? Without any further details or specifics, yes it is. It is far harder to conceptualize code instructions executing simultaneously, than one-at-a-time in a sequential order.

[unblough]

> Is parallel programming hard? Without any further details or specifics, yes it is. It is far harder to conceptualize code instructions executing simultaneously, than one-at-a-time in a sequential order.

If I program (map inc [0 1 2 3]) is it really any more difficult to conceptualize the (inc ) function performing on each element sequentially than in parallel?

I think the difficulty of parallel programming is less innate and more two fold:

1) languages often default to sequential so to do async requires introducing additional primitives to the programmer

2) knowing when to effectively use parallel programming

When I have a list or stream that I know has independent elements that require wholly independent calculations then parallel programming is straightforward

Where people get hung up is trying to shoe horn async where it is either unnecessary (performance is equal or worse than sequential) or introduces breaking behavior (the computations are in fact interdependent).

[mgaunard]

Most problems are not embarrassingly parallel.

(Fun fact: I once had someone call HR on me because they didn't know embarrassingly parallel was a technical term, and they thought I was belittling them)

[rfoo]

Prefix scan is not embarrassingly parallel. Yet OP's statement still works when you change it to scanl (+) 0 [0 1 2 3]

[mgaunard]

That requires + to be associative. And scan is one of the core skeletons of parallel skeletons, so obviously if you express everything as parallel skeletons, parallel programming remains manageable.

[Gazoo101]

I agree that if we define the individual instructions to always be wholly independent, then sure, it is more straightforward.

While I'd probably argue that it is still more difficult to conceptualize, the statement we're discussing is presented as broad and general. I'd call it far less misleading if it said something like:

There is a common myth in software development that parallel programming *has* to be hard.

[kortex]

The whole reason async is even a thing is due to slow, side-effect producing operations. Of course pure functions are easy to parallelize.

I don't think folks so much "shoe horn async where it is unnecessary" as the red/blue problem causes async code in most languages to spread.

Or by "async" do you just mean concurrent code? I'm reading "async" to mean lightweight coroutines or similar.

[unblough]

> Or by "async" do you just mean concurrent code? I'm reading "async" to mean lightweight coroutines or similar.

Yeah, my bad, I was utilizing a colloquial definition of a term that has a technical definition in a technical conversation. A lamentation lo the lossyness of language.

I guess I assumed we were talking about something other than in terms of red/blue because I'd argue red/blue's "hard"ness transcends myth to mathematical fact.

[wrsh07]

I don't think this is right. Thinking about operations on matrices is not complex. Defining how a single agent should act on its environment is not complex

When you say "without further details or specifics" you're saying "using my default framework of a c/ c descendent world"

The author's point is that sequential programming is one type of simple programming, but it's not the only type, and it doesn't map easily to modern hardware

[Gazoo101]

The author's article generally focuses on C (and possibly descendant languages), but the phrase I am critical of, does not. Furthermore, I explicitly consider a very broad selection of programming languages (many not C-derived) in my opinion. The author's phrasing, I'd argue, paints the entire concept of parallel programming as not hard.

There's some irony to the fact that you re-interpret my opinion as being very specific to C and (indirectly) posit that - in that specific case - parallel programming is hard, and then yourself go on to select a very specific case where parallel programming is not hard, because some matrix operations are independent.

I agree that there are languages that are explicitly built to make parallel programming easy. But in general, and not just related to c or c descendant languages, parallel programming is hard.

[wrsh07]

My point (and I think the points of others responding to you) is that parallel programming is not always hard. That's also what the author is saying.

The common myth - you're doing parallel programming? That sounds hard

It's not always hard. It really isn't! You don't need to be a genius or an expert to write parallel code.

Maybe where we're getting caught up is Cassie K's comment on ml engineering. You don't need to know how to build a microwave to use a microwave. In the same way, you don't need to be a genius or some deep expert in distributed systems to use abstractions that parallelize your programs

To write a parallel program does not require that you know what a mutex is. It just needs you to understand some simple algebraic (6-8th grade) properties about your functions (and, in fact, for library functions, they can be annotated as associative)

There is a broad spectrum of parallel programs. Somebody using a web server implementation? They've made a parallel application

Somebody running tensorflow or pytorch? Also parallel! Even for simple stuff!

You could be a beginner programmer and be taught to make parallel programs without understanding distributed systems. It's not always hard. It's not generally hard. The complex bits are hard. The simple bits use 8th grade math.

[Gazoo101]

> My point (and I think the points of others responding to you) is that parallel programming is not always hard.

Sure, and even more people commenting appear to be of the mind that it is generally hard.

> That's also what the author is saying.

It's not what author is explicitly saying in the statement I'm addressing if you re-read my original comment. There, the author isn't saying that it's not always hard, they're implying that it 'in general' isn't hard.

From your arguments, it would seem you think anything that actually runs in parallel (regardless of whether it programmed as such) can be considered 'parallel programming' and from that perspective, sure, it is super easy. But with that kind of reasoning, you can argue that anyone who only knows how to drive cars with automatic gears is actually a gear-shifting expert and shifting gears is really easy, because it happens automatically for them.

[ethbr1]

Agreed. The potential state space in parallel processing is a lot larger, which makes it more complex, which makes it harder.

That Erlang exists and people use it successfully does not mean that harder things aren't.

[fanf2]

Concurrent programming - doing lots of different things at once - is hard. It is hard to use concurrent programming infrastructure (processes, threads) to implement parallel algorithms. Parallel programming - using lots of processing elements to work on the same thing at once - is much easier if you have the right abstractions.

[danmaz74]

I wouldn't say that it's hard to conceptualize instructions executing in parallel, but it's hard to coordinate those parallel subtasks in an efficient and correct way - except in some use cases, like eg matrix multiplication.

[grotorea]

Isn't it distant from how humans work? We can't really do parallel, can we? And programming is translating human instructions to computer instructions, and translation is harder between more distant languages.

[wrsh07]

A factory is parallel

Or do you mean an individual can't do things in parallel?

Like.... Pushing all of those grocery carts in a long line is moving them in parallel

Or do you mean processing? Like thinking?