What's even better is that the DLR consists of some pretty tight code. Historically I was using a ConcurrentDictionary containing Funcs compiled using Expression. I've found the DLR is just as fast (for things that you can express with it), with a much lower overhead. I've started using it in dynamic scenarios wherever possible.
Basically, this is going to be about as fast as you can get with the statically-typed underlying runtime.
The codegen creates a cache of delegates (created using expression trees), but uses a structure more suited to the problem than a simple ConcurrentDictionary.
Basically, this is going to be about as fast as you can get with the statically-typed underlying runtime.