There is a cool site with mod lists that are confirmed to be compatible with OpenMW. I'm currently running 'I heart vanilla' which is basically the vanilla game with extra content and some balance and bug fixes. Highly recommend checking it out.
I can't remember which ones I used. Texture mods IIRC worked fine, pretty sure I had hi-res replacement mods for most of the textures in the game. I did some loading art replacer thing so they'd be widescreen and hi-res, but that wasn't so much a mod as replacing some images. I think I may have used some of the "Comes Alive" stuff—I know I used to "back in the day", so I assume I did this time, and I'd probably remember if it hadn't worked. One obscure library mod that I like, that adds an empty library just outside Vivec that you can fill by giving them books to copy, that worked fine. Twin Lamps, fairly certain I used that one.
As I recall, the main things that don't work are graphics overhauls that require patching or wrapping the original executable in some fashion. Usually these do some kind of shader magic to e.g. make the water look nicer.
[EDIT] Oh, and I think I had Herbalism installed. If not that, something similar. A mod that turns plants from containers into pick-ups is one of those don't-play-without-it things, for me.
All the alternatives seem to involved managing a plethora of mini-mods that hopefully work together.. it's really easy to install and it turns the game instantly from a drab/stale/dated look to something a lot busier and lively - at least at a surface level. Maybe it has some issues that I'm just not aware of - I never really got far in playing it
I'd be curious what other people thought of it. It seems to never be mentioned in lists and "hardcore" players seem to not like it for some reason
Also, for certain mods, you do need at least OpenMW 0.48 release candidate [1], a development build [2] (it's below Release Versions), or just build from source. I'm pretty sure it was after 0.47 when they added the ability to add shaders, and configure them using the F2 menu. The modding-openmw site will tell you if you actually need a 0.48 build for a particular mod.
I usually use "I Heart Vanilla" [3] as a base and then pick extra mods from "Expanded Vanilla" [4], which includes the load order if you also want Tamriel Rebuilt [5] (which lets you explore the mainland). The Expansions Integrated, Tribunal Rebalance, Bloodmoon Rebalance, and Expansion Delay are all nice mods too. Makes the expansions really feel like they are part of the base game. When adding extra mods like that, I just use the load ordering they suggest.
One of my favorites (in "Expanded Vanilla") is called Natural Character Growth and Decay [6], so you don't have to worry about min-maxing the level mechanics. It has some options, and I turn off the decay bit for a more relaxing time. It allows you to just play the game as you want.
I love the addition of shaders, and use Zesterer's Volumetric Cloud & Mist Mod for OpenMW [7] as you can have the game keep some of that hazy feeling yet still have distant land and objects. I also use Bloom Linear shader [8]. If you want some other shaders, you can use OMWFX Shaders [9] and/or Zesterer's OpenMW Shader Pack [10]. I have used Zesterer's OpenMW Shader Pack, but you have to edit the config.glsl if you want to set it different from the vanilla preset. The game looks nice with just FXAA, clouds and mist, and linear bloom (very mild). The "Expanded Vanilla" guide under shaders gives a good preset, and then you can change them using the F2 menu in-game.
This all said, it's been awhile since I've properly played last. I take my old list and update it occasionally as mods change or new features are added to OpenMW, thinking I will play through it again one of these days.
As far as I know, downloading and installing each mod is the main method for the modding-openmw lists.
There is a tool to try to automate game modlist installs called Wabbajack [1]. It was briefly discussed on the openmw forums [2]. I'm not sure if anyone has tried further to get the modding-openmw lists working with it. I have never used Wabbajack, so I don't know too much other than it exists. It still seems to have extra requirements according to the post, including a premium NexusMods account if you don't want to manually download everything.
Just as I typed all the above I saw a mention of portmod [3] in the modding-openmw guide about mod managers. Seems it's like Gentoo's portage, but for game mods. I see it has some meta packages for modding-openmw in their ebuild-like repository [4], but I don't think any of the big lists have been updated in awhile. Also, there again seems to be issues with it auto downloading from NexusMods mentioned in the issue tracker. Nevertheless, it looks to be an interesting project.
https://modding-openmw.com/