Wait is this now heap allocating a value in every iteration of every loop? I hope that allocation is optimized out in every case where there isn't a closure over the loop variable?
The fine details resemble the analysis of correctness - all the evidence shows people expect per-iteration semantics with considerable frequency, and don’t rely on per-loop semantics with measurable frequency. But it’s impossible to completely automate that assessment. Likewise, it’s impossible to automatically detect code that will spuriously allocate because of the semantic transition.
Regardless of how the compiler is optimising this, I 100% agree that the old behaviour is unexpected and it’s caught me at least once. Really happy to see this (until recently) unexpected change.
I don't actually use Go, but I have used many other languages where it is like the old behavior. I learned once that I have to build the closure correctly to get the value I want and know now to do it. Don't have any statistics on whether I made that mistake again, but anecdotally I can't remember a case where I have. In their analysis they have found a lot of cases with that mistake, though. So I guess fair enough.
However, I wonder what it will mean if someone who mostly writes Go will now use another language? Will they be more prone to make that mistake?
It's hardly standard behaviour. I mean in Java for example there didn't used to be value types, so everything was a pointer and the effect of this would be the same as the new behaviour in Go.
The only lesson to be learned here is that languages are different. But I think the new Go behaviour is more ergonomic.
In Java you can only close over final variables, so you can't close over the loop variable at all. (Unless that changed since last time I used Java, which - granted - was a long time ago.)
The problem being fixed doesn’t affect only closures, but the body of the for loop itself. So for example taking the address of the loop variable would unexpectedly return the same value for the duration of the loop.
The way to view it is "unless there is syntactic sharing, it is a for loop, same as before". The compiler uses a syntactic test (with little knowledge of control flow or value use) to exclude loops from the change. This excludes most loops.
After the change, escape analysis figures out if the changed iteration variable actually needs heap allocation; in an internal sample of code that was actually buggy (i.e., biased, guaranteed to have at least one loop like this) for 5/6 of the loops escape analysis decided that heap allocation wasn't needed.
The reason this optimization isn't part of the language change proposal is that escape analysis is "behind the curtain"; ignoring performance, a program should behave the same with or without it, and it is removing heap allocations all over the place already. Escape analysis is also extremely difficult to explain exactly, so you would not want it in the spec, and "make escape analysis better" (that is, change it) is one of the prominent items in the bag of things to do for Go.
In fact, you can enable warnings/logs that indicate whether code that is affected by the loopvar experiment results in a stack-allocated or heap-allocated loop variable: https://github.com/golang/go/wiki/LoopvarExperiment#can-i-se...
I imagine that the current workarounds for this issue also end up with heap-allocated variables in many cases.