[malisper]

> Threads does not offer any major performance advantage

This is very not true. When it comes to parallel queries, a process model adds a ton of overhead. You can't pass pointers between processes because the address space is different. This adds a ton of overhead in a bunch of different places. For example when doing a parallel hash join, Postgres will have each worker build a local hash table. Then it will take all the tuples out of the local hash table and copy them through shared memory to the leader who will then construct a new hash table. This duplicates a lot of work as you have to hash the tuples multiple times.

A lot of getting to Clickhouse level performance was making better use of parallelism.

[mort96]

Passing pointers is not significantly faster than passing offsets into a shared memory pool.

[menaerus]

Sorry, what? Passing a pointer is a matter of wrapping the value into the CPU register. OTOH passing an offset into a shared memory is a write to main memory so several magnitudes slower.

[mort96]

Passing a pointer within one thread requires putting the pointer into a register. Passing a SHM offset within one process requires putting the offset into a register.

Passing a pointer between threads requires going through the memory system and letting cache coherence algorithms sort out the data sharing between cores (with or without a futex lock/unlock depending on implementation). Passing a SHM offset between processes requires going through the memory system and letting cache coherence algorithms sort out the data sharing between cores (with or without a context switch to the kernel depending on implementation).

It's not that different.

[jeltz]

Ok ... you know PostgreSQL supports hash tables in shared memory, right? PostgreSQL could in theory share those if we wanted to. The issue is just that coding anything which uses shared memory is a lot of work.

Additionally the reasons PostgreSQL does not offer Clickhouse performance has very little to do with parallelism. PostgreSQL plans to move to threading but the efforts around imporving OLAP performance are almost entirely unrelated.

[comex]

> The issue is just that coding anything which uses shared memory is a lot of work.

Doesn’t that kind of prove the parent’s point though? In theory shared memory can do anything that threads can do. But if in practice some feature doesn’t get implemented in the multi-process design (because shared memory is hard), when it likely would have been implemented in a threaded design, then that’s still an advantage for threads.

[gpm]

Apart from being a lot of work are you really gaining much at that point? Memory corruption can still take down both sides...

[dupontcyborg]

i may be missing context, but shared memory across processes, without ipc?

[Giefo6ah]

There's nothing special about threads vs processes in Linux. mmap works the same, the challenge is to map the same file. You can share a path, pass a file descriptor via fork or unix domain socket, among other techniques.

[LoganDark]

That induces disk I/O overhead (even if it somehow doesn't impact IPC performance)

[drdexebtjl]

The file doesn’t have to be disk-backed.

[LoganDark]

Don't you need something mounted for that?

[outpost_mystic2]

It doesnt. Processes can share memory

[frollogaston]

Being really pedantic here, shared memory is considered IPC, but not the kind you're thinking of. Shared address space, no overhead.

[guenthert]

As long as we're pedantic ... the subject is shared memory. Unless you specify the same, non-null, target address in the call to mmap (and the kernel happens to grant you that mapping on all calling sites), the addresses will be different; the address space is not shared (each mapping might also have different access permissions).

That distinction is important as pointers generally cannot be shared (a problem which can of course be solved with one more indirection ;-) .

[JackSlateur]

"Shared address space, no overhead"

But concurrent access, so synchronization is required (lock or whatever), so overhead :)

[frollogaston]

Yeah that's true, and I'd say threads need synchronization for concurrent access too, but supposedly the options for doing that are faster than what you need to use across processes.