I'm pretty sure it was a custom VM: I wasn't on the team, but my own group (Unix) was playing with custom IP stacks with the same goal (reducing latency, increasing determinism).
With collectors like CMS, you can control after what percentage of the GC is filled GC will kick in. I think you can use that get the same effect. This is only for old gen.