When you manually allocate memory you are naturally responsible for the lifecycle of that memory. In Go you allocate such memory outside the GC to avoid tracing.
The issue, as Ian pointed out, is that it's hard to pinpoint when GC allocations occur. Language constructs allocate in ways that are not immediately obvious.