The Rust code is still only brute force - using suitable persistent acceleration structures you can probably get a 10x again or maybe even 100x, in 2D a kd-tree is really fast for NNs.
So much faster that the allocations for the result will probably be the bottleneck.
You can always speed up an application if you rewrite the used libraries to match your specific use case.