[jeremyjh]

I thought the Go runtime runs foreign code on M:M threads; e.g. when Go time calls foreign code, it dedicates a thread to it. This is so foreign libraries (which are unaware of yielding to the Go scheduler when they do I/O) don't block a thread with multiple goroutines scheduled on it. I do not think Nim code can run "in a goroutine".

[stefantalpalaru]

> I do not think Nim code can run "in a goroutine".

It does. Don't forget that this is gccgo so it is possible to use plain C functions as goroutines. Nim is translated to C and with the help of a macro I convert Nim functions with an arbitrary number of arguments into ones with a single void* arg that gccgo wants for its 'go' keyword implementation:

    extern void* __go_go(void (*f)(void *), void *);

[jeremyjh]

Yes, but when the Go runtime calls that foreign function pointer, it is going to schedule an entire OS thread for its duration isn't it? If it doesn't, then nothing prevents foreign code from blocking other goroutines. How does the Nim code yield the thread for other goroutines, does it have to register a callback?

[stefantalpalaru]

> when the Go runtime calls that foreign function pointer, it is going to schedule an entire OS thread for its duration isn't it?

No.

> How does the Nim code yield the thread for other goroutines, does it have to register a callback?

There are no callbacks. Yielding happens automatically when launching another goroutine, when sending, receiving or selecting on a channel. You can also yield explicitly with go_yield() - the better named equivalent of Go's runtime.Gosched().

It's easier to understand if you realize that all those operations with goroutines and channels end up being done in the Go runtime.

[jeremyjh]

> > when the Go runtime calls that foreign function pointer, it is going to schedule an entire OS thread for its duration isn't it?

>No.

Are you sure? Once a thread enters cgo it is considered blocked and is removed from the thread pool according to these sources [1][2]. I previously found a thread where Ian Lance Taylor explained it more explicitly but I couldn't find that now. Is that not what is happening though when __go_go invokes your function pointer?

I do not understand how the Nim code can live in the segmented stack of a goroutine, nor how the Go runtime could know it is time to grow that stack.

[1] https://groups.google.com/forum/#!searchin/golang-nuts/gorou...

[2]http://stackoverflow.com/questions/27600587/why-my-program-o...

[stefantalpalaru]

> Are you sure?

Yes. See the chinese whispers benchmark with 500000 goroutines and a maximum resident set size of 5.4 GB on amd64. It has the same memory usage (and run time) as the Go version compiled with gccgo.

> Once a thread enters cgo

This has nothing to do with cgo. It's a different mechanism specific to gccgo.

> I do not understand how the Nim code can live in the segmented stack of a goroutine

Good thing you asked. I just ported to Nim the peano.go benchmark described as a "torture test for segmented stacks" and... it failed. The fix was to pass -fsplit-stack to gcc when it compiles the C code generated by nim.

> nor how the Go runtime could know it is time to grow that stack

From what I can tell it's done in __splitstack_makecontext() and friends from GCC's libgo.

[jeremyjh]

Well, cool! Great work :)

[nulltype]

How does it handle making a blocking socket call in a go goroutine?

[stefantalpalaru]

It doesn't, yet.