“Book of the New Sun”, beginning with Shadow of the Torturer, is his major work but can be unapproachable at first. The narrator/protagonist is a terrific character, but a IMO difficult one to spend time with.
The Fifth Head of Cerberus is also a great entry point. It’s a set of three interconnected novellas that, for my part at least, were more immediately engaging than BotNS. It’s got all the Wolfe-isms you want: unreliable narrators, unconventional settings, and puzzle-box stories that slowly open themselves.
Despite what the parent comment said I'd still recommend Shadow of The Torturer, the first book in the Book of The New Sun series. If you like the idea of being immersed in a world that you don't fully understand, but gives you the impression that there is a richness of lore behind every minor detail, then you'll enjoy it.
The Fifth Head of Cerberus is also a great entry point. It’s a set of three interconnected novellas that, for my part at least, were more immediately engaging than BotNS. It’s got all the Wolfe-isms you want: unreliable narrators, unconventional settings, and puzzle-box stories that slowly open themselves.