That was my opinion for a while as well. It depends on your audience but about 50% of hits to my site come from one mobile device or another and in the end I felt I had to put the effort in to support them.
What is this "audience" you speak of? I've run a blog for 25 years. First, it was FrontPage, then a custom PHP site, then a custom Rails app, and now Wordpress. The only thing that hits it is web crawlers. I do it because I must.
Ha! I run a simple hit tracker and most of my posts get single digit hits a year. I rent a server to run my blog and write a blog to have something for my server to do.