You probably won't be able to match the built-in's performance without a similarly large surface area of unsafe usage. And Go's maintainers won't maintain your unsafe code for you as they do the default, nor consider the impact of other language changes on your micro-optimizations.
https://github.com/golang/go/blob/master/src/runtime/map.go
You also won't have `for k, v := range` iteration, but that is also likely being addressed within the next few versions.
But - no, there's not really any major ergonomic issues to basic lookups given generics these days.