TeX was the first serious examination of what good typographical layout was and how to codify it. Previously there were many rules of thumb, and lots of examples, and a handful of programs for typesetting (like RUNOFF, which was and is pretty damned ugly, although it beats the pants off a typewriter) but in TeX Knuth created an algorithm for things like mathematical layout, for paragraph layout. I am unaware of anything that has surpassed TeX in this field—programs like Lout use the same algorithm TeX does, just with a less idiosyncratic surface syntax and with a rewritten core.
Strictly speaking it is possible to use fonts other than R, B, I, BI using ".fn" but because the set of fonts is determined by the output device it's very limited.
Much to my surprise GNU groff's font format does actually contain kerning information and ligatures although it's quite primitive and none of the included fonts have characters much outside of Latin-1 (to the point that there's a specialized "EURO" font you have to use if you want a Euro symbol). I would have to go over the older documents I have to be sure but I don't think this was a feature of the original troff.