Ironically, I do the same with golang at my current employer. Build times are nowhere near as what's hyped. Much closer to Java, and in fact, due to Java's incremental compilation, golang is often slower.
Go has incremental builds as well. I’ve been using it for years and build times are usually a couple seconds; a couple of minutes for huge projects. Are you sure that’s just build time and not some combination of downloading dependencies and/or running tests? How big is your project?