[CyberDildonics]

_I_ want to know what we're talking about

lock free memory based interprocess communication (copied from my first reply)

There is no need to back this memory with files

Again, I'm interested in how you have two processes read and write lock free directly to the same memory if you don't use a memory mapped file.

You have said it isn't necessary, I'm just asking you to back up this claim and explain exactly what you mean.

[jstimpfle]

First, I didn't assume it a requirement to have two processes read() and write() _directly_ to the same memory (I suppose you meant "file region" here). And idk, it might not be a good idea to require that.

Also, you can use normal (non-file-backed) memory to do the necessary synchronization (lock-free or not). I'm still not seeing why the memory should be backed by a file, that's why I was genuinely asking. One reason why it could be practical that I can now see could be for an embedded database like sqlite, but again I'm not sure it would be a good idea. While it would allow for pretty much setup-less synchronization of otherwise uncoordinated processes, it's a fringe application that might be better implemented with one big flock(). And one reason why it could be not a good idea is that it might couple the file format to a particular CPU architecture.

Another big issue I guess is that the atomics actually do have an effect to the underlying file whenever the pages are flushed. What if the computer shuts down unexpectedly? The synchronization affairs aren't cleaned up, yet the original processes are gone.

[CyberDildonics]

that's why I was genuinely asking.

You weren't asking, you were saying it wasn't necessary, which you did in the sentence right before this one:

Also, you can use normal (non-file-backed) memory to do the necessary synchronization (lock-free or not).

Again, this is just a repeated claim, it isn't an explanation. How do you have two processes writing to the same place in memory without memory mapping a file?

have two processes read() and write() _directly_ to the same memory

I didn't say read() and write() I said read and write as in reading and writing with memory addresses. Again, this is all about lock free interprocess communication. You can't write outside your own memory from a process with normal permissions so how do you share memory with another process?

You memory map the same file. This isn't about the file being written to some sort of persistent storage, that happens on the OS level and doesn't interfere with two running processes communicating with each other. The file can be deleted after the last process closes it. It is just a way for the two processes to have memory mapped into their virtual memory space that overlaps with each other.

You need to deal with memory directly so you can use atomics. You need to use atomics so you can avoid locks.

I thought you might have had some other technique that I'm not aware of but it seems now you were making claims without much behind them, which is disappointing.

[pritambaral]

> How do you have two processes writing to the same place in memory without memory mapping a file?

In increasing order of modernity: System V SHM, POSIX SHM, and `mmap(... MAP_SHARED | MAP_ANONYMOUS ...)` (e.g., on Linux).

[CyberDildonics]

These are still memory mapping files using file paths and returning file descriptors as far as I know, which makes sense because you have to have something coordinated between the two processes.

[pritambaral]

> These are still memory mapping files using file paths ...

No, they're not. The entire purpose of MAP_ANONYMOUS is to avoid using files.

Sources:

1. The Linux Kernel source code [1], where it comes with the code comment: "don't use a file".

2. The glibc source code [2], where it comes with the same code comment: "Don't use a file".

3. The Linux man-pages project documentation of mmap [3], where it is documented thus: "The mapping is not backed by any file; its contents are initialized to zero. The fd argument is ignored"

Similarly for SHM, but if you still don't get the point about MAP_ANONYMOUS, I doubt you'll get it for SHM either.

> ... and returning file descriptors

A socket is a file descriptor. An epoll handle is a file descriptor. On modern Linux kernels, a pid handle is a file descriptor. None of them are backed by "files".

> ... because you have to have something coordinated between the two processes.

FDs are not the only things processes can share, even if you go back to the venerable, original Unices, so I don't see what you mean.

----

[1] https://elixir.bootlin.com/linux/0.99.14r/source/include/lin... (the first version it was released in) to https://elixir.bootlin.com/linux/v5.16.1/source/include/uapi... (the latest version)

[2] https://code.woboq.org/userspace/glibc/sysdeps/unix/sysv/lin...

[3] https://man7.org/linux/man-pages/man2/mmap.2.html

[CyberDildonics]

This just goes back to the same question - what do two processes use to map the same memory into their memory space if it isn't a path to a file?

I'm not saying there isn't anything, I'm just seeing an extreme avoidance to an actual answer. The other guy went down a rabbit hole of syncing that memory to storage, which has nothing to do with anything.

[jstimpfle]

> You weren't asking, you were saying it wasn't necessary, which you did in the sentence right before this one:

quoting my OP: " Why do you need lockless atomic updates to a file-backed memory area? Genuinely curious. " . Dude.

> it seems now you were making claims without much behind them, which is disappointing.

Well thank you very much.

I get the feeling we might just be talking about the same thing. Or we might be not, I'm not sure.

> How do you have two processes writing to the same place in memory without memory mapping a file?

> You can't write outside your own memory from a process with normal permissions so how do you share memory with another process?

For example on Linux, use shm_open() + mmap(). This is just an example, and granted it uses a file-like API (shared memory objects show up on /dev/shm on a typical Linux) but it is not "file-backed" (I meant disk backed and this might be the misunderstanding) and in particular it's certainly not mapping the database file. It's just one way on one OS to map the same physical memory into different processes' address spaces.

If this example approach is "file-backed" to you, then so be it but I think you have willfully misread my comments up to here.

    #include <sys/mman.h>
    #include <sys/ipc.h>
    #include <sys/shm.h>
    #include <sys/fcntl.h>
    #include <errno.h>
    #include <string.h>
    #include <stdio.h>
    #include <stdlib.h>
    #include <stdint.h>
    #include <unistd.h>

    int main(int argc, const char **argv)
    {
            int fd = shm_open("/TESTOBJECT", O_CREAT | O_RDWR, 0664);
            if (fd == -1)
            {
                    perror("shm_open()");
                    exit(1);
            }

            if (ftruncate(fd, 4) != 0)
            {
                    perror("ftruncate()");
                    exit(1);
            }

            void *mapping = mmap(NULL, 4, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);

            if (mapping == MAP_FAILED)
            {
                    perror("mmap()");
                    exit(1);
            }

            volatile int32_t *ptr = mapping;

            for (;;)
            {
                    printf("%d\n", (int) *ptr);
                    if (argc > 1)
                        *ptr = rand();
                    sleep(1);
            }

            return 0;
    }

[CyberDildonics]

If this example approach is "file-backed" to you, then so be it but I think you have willfully misread my comments up to here.

You kept making the same claim and I asked you to explain it, then you just made the same claim again.

shm_open("/TESTOBJECT"

That's a file path. Maybe you just wanted to bait an argument by not explaining yourself.

[jstimpfle]

Homework: go back through my comments and identify all the places where I was VERY CLEARLY pointing out that my statement is that no disk-backed file is needed, or where you could reasonably infer this from my use of the term "file-backed", as well as from the general context of the discussion.

> shm_open("/TESTOBJECT"

>>That's a file path

Pedantically, no. It's a name (https://man7.org/linux/man-pages/man3/shm_open.3.html) that identifies a memory object that is only coincidentally also mapped to the file path "/dev/shm/TESTOBJECT" on a typical linux. shm_open() returns an "FD", though.

On Linux, as a sibling poster noted, you could also use mmap(.. MAP_SHARED | MAP_ANONYMOUS, /*fd*/ -1 ...) , which to my knowledge is entirely "file-free" by any meaning of the term "file". But then again, in my understanding this would only work with child processes because that mapping has to be inherited.

On other OSes, there may be completely different APIs to map shared memory that don't involve anything "file" like, either. Quite honestly I can't point you to any because I do only Linux and Windows, but let's just end the discussion here and let's agree that memory != file. I'm angry at myself for wasting another evening fighting a pointless discussion with somebody who would rather argue than try to get my point.

[CyberDildonics]

You conflated files with disks on your own. No one did that for you.

rather argue than try to get my point.

I still don't know what your point is. You have to have something that coordinates between two processes for shared memory interprocess communication and that ends up being file paths for the OS. You asked questions, they were answered and you could have learned something.

The whole point was actually that you can map the same memory into two different processes and use atomics, which is an incredible technique. For some reason you wanted to ignore that and make claims without explanation.

If you didn't want to waste time, you would have explained what you meant or asked questions.