It's all custom; the frontend is React and the backend is Ruby. Our plan is to introduce Markdoc as an intermediary step to improve the authoring experience.
Just to add to that, "normal" markdown (I forget which flavor exactly) is currently used for the text blocks within the Stripe API reference documentation.