I'm wondering this too. I haven't heard of retdec being used too much, but it looks very cool. I'll guess that Hex-Rays is better, but I still am interested in the opinion of someone more experienced who has tried retdec.
I'll go one better: I've contributed patches to retdec.
Retdec is ... okay.
On small binaries it's usable. On even average sized windows binaries (a few meg), not really.
Like on things that IDA takes 10-15 minutes and a reasonable amount of memory (like a 7 meg windows binary), retdec can take forever and unlimited amounts of memory.
I started fixing a lot of the memory issues (completely recursive CFG traversal, etc), but there are also very serious algorithmic issues (N^3/N^4 algorithms in the optimizers).
If i disable a lot of the backend optimizers, i can make it work okay.
But then the output is also a lot larger/worse. To be fair: It used to be about 50x bigger than similar IDA output. The latest development version of retdec now has a new backend IR converter, and the output is only 5x-10x bigger than IDA output.
So as a TL;DR: retdec in its default state is unusable for anything but small binaries.
If you understand what is going on, you can get it to work on a lot of binaries as long as you have a ton of memory and time to spare.
Yeah, sorry, as I understand it, it uses Capstone as the disassembler and implements an LLVM lifter over it. It was pretty dumb to describe it as "based on Capstone"; I was just mentally breaking tools down by which CFG recovery system it relied on.