XSLT 3.0 can be directed to output HTML5 [0]. However, browsers only implement XSLT 1.0, and as far as I am aware there is no open-source XSLT 3.0 implementation.
Still, it's possible with XSLT 1.0 to produce documents in the common subset of XML and HTML5 ("XHTML5"). It can't produce the usual <!DOCTYPE html> at the top of the document, but it can produce the alternative <!DOCTYPE html SYSTEM "about:legacy-compat">.
On the input side, every XSLT version only accepts valid XML, as far as I am aware.
`xsltproc --html` is an example of HTML input (probably HTML4 parsing rules though?) if you really need it. This is an XSLT 1.0 processor, wrapping libxslt which most browsers use.
As for output, the difference is largely irrelevant for browser purposes since they just want a tree.
I'm not sure how many extensions the browsers allow, but a major part of the reason XSLT 2/3 failed to take off is because libxslt already provides most of the useful features from newer versions as extensions (many via EXSLT-namespaced modules, at least partially supported in browsers - see MDN); what it doesn't do is implement the unnecessary complexity that the Java world loves.
At the time HTML was converted from SGML to XML:
https://en.wikipedia.org/wiki/XHTML
so if you authored XHTML, you could XSLT it. There is also XHTML5, an XML serialization of HTML5. I imagine in the real world there is a great deal of web that is HTML, accepted by browser, but not XML.
As far as I know, HTML5 has diverged from its origins enough that it's neither SGML nor XML. However, given the existence of XHTML5, it might be possible to parse an HTML5 DOM and re-serialize it as XHTML5, and thus it might be possible to take parseable HTML as input to XSLT, albeit with some indirection.
We were going to move on to XHTML after HTML4, for those variants (did it go beyond XHTML1.1?) HTML is XML compliant. That got caught in slow design-by-committee hell though so HTML5 became the defacto standard instead. There is XHTML5 which is an attempt to direct that back towards compliance, but I've never seen it used in the wild.
As I understand XSLT, it takes an XML document as input and an XML document describing the transformation, and produces an XML document as output.
But most HTML in the wild today is not valid XML. There is XHTML as mentioned by a sibling comment but it's rarely used. So if you were to start with an existing base of HTML documents, you couldn't easily add XSLT preprocessing to them. The issue is with the input rather than the output.
If you're using it as a template language for your own pages, you can of course just write it correctly (this is not different than needing to use correct syntax for react code to compile).
If you have someone else's documents, or need to mass convert your own to fix them, there's HTML tidy[0]. This one is quite useful to be able to run XML processing CLI tools on scraped web pages.
But the real power is in delivering XML to the client, not HTML. This lets you work in the domain model directly on the frontend, and use XSLT to transform that into HTML for display. So of course you'd use well-formed XML in that case.
Imagine if you didn't have a distinction between APIs and pages; you just returned the data along with a link to a template that says how to display it and (ideally) a link to a schema definition. Modifying templates for components could be as easy as modifying CSS attributes in the browser console, giving the next generation an easy way to peak and play around with how it all works as they're growing up. We were so close, and then things veered off into the hellworld that is the modern web.
Still, it's possible with XSLT 1.0 to produce documents in the common subset of XML and HTML5 ("XHTML5"). It can't produce the usual <!DOCTYPE html> at the top of the document, but it can produce the alternative <!DOCTYPE html SYSTEM "about:legacy-compat">.
On the input side, every XSLT version only accepts valid XML, as far as I am aware.
[0] https://www.w3.org/TR/xslt-xquery-serialization-30/#html-out...