What macros do you use? I have some simple programs that generate documents to print out, and I use LaTeX and a bunch of packages. It takes hundreds of megabytes of software with all the packages. I've long toyed with using the MOM macros but can't get motivated to rewrite my programs and learn troff.
It used to bother me that groff doesn't support Unicode (as far as I can tell) but then realized that all I write is English so why am I fetishizing Unicode? groff will get me any accented character that I would realistically need.
You can have UTF-8 with groff using the preconv [1] tool, just pass the "-k" or "-Kutf8" paramater. It will preprocess your source and replace Unicode symbols with its groff special character form. Of course, the built-in standard fonts do not support many glyphs but it is no big thing to use a more Unicode friendly font like DejaVu Sans or Noto, check [2].
So this is my setup using MOM macros for nice typesetting.
But I fully agree on your last point, in the end I seldom need the Unicode universe for most of my documents.
Little-known fact: groff (more particularly grotty) does colour, albeit that people have secretly turned this off for you. grotty's own man page is in fact in colour.
It used to bother me that groff doesn't support Unicode (as far as I can tell) but then realized that all I write is English so why am I fetishizing Unicode? groff will get me any accented character that I would realistically need.