Is that true for the majority? While I don't play myself, it seems a part of the appeal of fantasy sports (at least football) is the drafting/management of your team.
There's traditional fantasy sports, which take place generally over the course of the season, and then there's daily fantasy, which takes place on a given day or weekend. The former is generally played among groups of friends, usually with money involved, but primarily as a recreational group activity. The latter is played primarily for the purpose of winning money against strangers on the internet. I've played both and while the actual draft event is my favorite part when playing in my league with friends, it's far less enjoyable for me in daily fantasy sports. Daily fantasy is technically a skill game, but it really feels like gambling.
Traditional fantasy sports, where you draft a team and manage them in a set league over a season or multiple seasons, is very different and will still exist because, as you say, it can be fun.
What DraftKings and FanDuel provide is basically a loophole in the federal law which made gambling via fantasy sports legal. You draft a new team every day, and gamble that your team will do better on that particular day than other people. It's kind of a pain in the butt to do, when what you really want is to just watch the games and root for one team...