This could also be a great way to check if your optimized version is still necessary with new versions of your language implementation. The more I think about it, the more it seems like a great idea.
I wouldn't generally throw benchmarking into the unit test process, because it can be brittle. What if something else clobbers the CPU on your test machine at just the wrong time.
What you really want from your unit tests is for them to run quickly, so you run them as often as possible, and reliably, so any failure means something is wrong. I'd use the slow implementation just for checking correctness.
But separately, if you want to add a performance benchmarking process (great!), you have a reference baseline implementation for free.
But separately, if you want to add a performance benchmarking process (great!), you have a reference baseline implementation for free.