[menaerus]

TBH I find it difficult to understand:

1. How do you compile the C++ code? E.g. what flags do you use with GCC, clang and MSVC

2. How exactly is C++ binary run through a Rust benchmark suite? FFI?

3. Is it possible for rust code-under-test to be at any advantage here because it is run and built natively from the benchmark written in the rust itself?

What I find questionable though is the actual C++ code found in the benchmark:

1. Exception-handling code around std::sort and std::stable_sort - nobody does that. What problem are you trying to solve with this? Comparators do not throw exceptions.

2. Using function pointers for the comparators - surprising to see such code in C++ benchmark - it's very non-idiomatic and essentially making it impossible for a compiler to inline the code. std::sort is rarely used like that.

3. Passing over some magical third argument to the comparator function - ?

4. Passing over the context to the comparator "function" - I fail to see if "ctx" has been used anywhere?

5. "Making" the comparator function with make_compare_fn which in turn instantiates a lambda that, again, throws an exception from its body.

6. Storing a comparison result - why? You only need to return true or false from a comparator.

7. Modeling rust "panics" by storing a boolean is_panic plus exception-handling code plus throwing exceptions when is_panic is "ON" - why?

8. Confusing exceptions with rust panics - even if you wanted to do so, which still would be an arguable thing to do, std::abort or std::terminate is a replacement for std::panic. Not throwing exceptions and implementing the is_panic logic around it.

9. What is https://github.com/Voultapher/sort-research-rs/blob/main/src... being used for?

I think you're making a bit dishonest representation here for the reasons above. And I have not delved very much into depth nor have I covered all the code but just the fragment of it. Also, it is not quite clear how everything is put together and run because, ideally, you would want to have multiple binaries built with their representative toolchains/scripts _regardless_ of your benchmarking framework. And only then I would want to point the benchmarking framework to the respective binary to run the test. Here, it seems it's the other way around.

[Voultapher]

You claim yourself that "I have not delved very much into depth" yet you make a lot of assumptions here. Here are the key facts, the `_by` functions that contain the complex exception handling stuff are only used for tests and the benchmark ones are as native as possible e.g. `std::sort(data, data + len)`. I pick either gcc or clang based on what gives better performance, and use these settings with the cc crate https://github.com/Voultapher/sort-research-rs/blob/b131ed61... plus sort specific options. It all gets statically linked into one big binary. Also note the C++ code is header only, so one big translation unit per sort. The only disadvantage based on me looking at the generated asm is one additional function call versus possible inlining for the initial part of the sort. This adds a total of ~20 or less cycles to the total time, which in the benchmark is measured in microseconds, where 1 microsecond is ~4900 cycles.

[menaerus]

And yet you ignore majority of my questions above. It was an honest feedback but now I will bite and say that the report looks "good on the paper" but the underlying code is low-quality, very non-idiomatic, without explanations, very difficult to understand on what and how it is being tested, testing/benchmarking methodology etc. Many important pieces are missing and all that poses a question on what genuine intention of this experiment was - was it to actually find a bottleneck in implementations, if any, or was it something else. Now, that you're avoiding to address the feedback, it begins to become obvious.