My limited understanding is that there's less cache thrashing (from multiple different workloads scheduled on the same core) and less scheduler overhead (from less overall threads).
Just to add that scheduling overhead goes away with SMT (assuming you don’t oversubscribe), but the sharing of caches and branch prediction logic is still an issue as you point out.