Action seems like the wrong word. Walking simulators lack any skill requirements. They’re basically just choose your own adventure stories with a small number, or even just one, choice to make, but you navigate with a character movement instead of picking options from a menu.
It's still a bit of a derogatory term, and it doesn't entirely match up to history. Text adventure games, point & click adventure games and so on were very similar in gameplay mechanics to many games that get this moniker.
Many of them do incorporate enough puzzle elements that the game consists of more than just taping down the up arrow. Often, the term feels like it attempts to reduce the term "game" to something more narrow than what the art form does and has encompassed, historically. Not all games require mechanical skill. Not all games are what you expect them to be. Etc.