| >It distresses me that we don’t have a language that can do a best effort parallelization of arbitrary loop like code across SIMD, multiple threads, multiple cores and GPU with a small directive. It's called ParaSail. The next closest thing to ParaSail is ... ... literally just Rust. Why? Because ParaSail has completely eliminated pointers, thereby preventing pointer aliasing. It can't be understated that pointer aliasing and alignment are the two biggest bottlenecks preventing autovectorization. The people talking about better compilers, etc, just don't get it. It's not a compiler problem, it's a language semantics problem. Pointer aliasing prevents parallelism full stop. If there is a single memory region and you perform a write to it, you have to assume that the write invalidates all data loaded from the pointers. If you make pointer aliasing illegal, then you have guaranteed that each pointer points to a distinct subset of the global memory, turning each pointer into a pointer to an isolated region. This means you have multiple regions you can write to in parallel. This is crucial, if you do not understand this you don't get parallel programming at all. I mean think about it, this is the difference between having a four toilet bathroom with a single door or four doors. The thing about alignment is not as easy to explain, but here is my attempt at it: If you allow the array to be misaligned at the front, then you have to run scalar code at the front. Same problem if you misalign at the end. If you have a naked pointer and a for loop (think C), then the alignment problem alone precludes autovectorization without a complex scalar preamble. Autovectorization has to assume that the loop length could be anything, meaning it could be less than 8 elements to begin with. If the loop is always a multiple of 8 and always aligned, then the loop can be autovectorized even if the loop only does a single iteration. Of course, after these critical blockages are gone you're still stuck with the problem of having to write branchless/non-diverging code. |
> All of the objects declared in a given scope are associated with a storage region, essentially a local heap. As an object grows, all new storage for it is allocated out of this region. As an object shrinks, the old storage can be immediately released back to this region. When a scope is exited, the entire region is reclaimed. There is no need for asynchronous garbage collection, as garbage never accumulates. Objects may grow in a highly irregular fashion without losing their locality of reference.
> Note that pointers are still used behind the scenes in the ParaSail implementation, but eliminating them from the surface syntax and semantics eliminates the complexity associated with pointers.
This approach seems to come up in many contexts, where a pointer-based address (raw pointer, reference, slice, etc.) is abstracted into a higher-level address key, usually an index integer (essentially a higher-level virtual pointer). The implementation might reallocate under the surface, or manage chunks of data through some sort of paging where the underlying pointers don't change (I was musing about this here [2]).
I guess I'm thinking out loud here, but most dynamic languages (eg. Python) don't expose the pointers or care about invalidation of the addresses, they just happily reallocate. I've barely written parallelized code, is pointer aliasing really one of the biggest roadblocks? It seems like it can be abstracted away fairly easily, even in a pointer-exposing language.
1. https://www.adacore.com/uploads/papers/parasail-pointer-free...
2. https://news.ycombinator.com/item?id=49013285