Which brings you back to doing an expensive scan of the large off-heap data structure you were trying to avoid.