For people with good vision, a 4K 6" screen held 18" from the eyes will look better than a 1080p screen, all other factors being equivalent. Held further than about 18", 4K won't make a a difference for most people.
For a 60" TV that sits 10 feet away, you're not going to see much difference. Move that TV to 4 feet away, and you'll appreciate the extra pixels.
For virtual reality, and other applications that put the screen much closer to your eyes, the extra pixels will make a huge difference.
I completely agree if you talk about virtual reality, but here we're talking about VP9 or HVEC compressed video, which will also need to be resampled as the display isn't 4k but something in between 1080 and 4k.
At 20/20 vision, we can resolve ~300 pixels/inch at 10-12 inches from the eye. [1]
For a 6" screen, that works out to ~1800 pixels. So maybe not 4k, but the next step down from 4k generally is 1080p, which is less than the figure I gave above.
So yes, there is a point.
(And that's not getting into other factors either, such as panning and stroboscopic effects.)
Though one thing I will still mention is panning. While you're panning, at sufficiently high framerates you're still effectively having 1 pixel for every 2. (Look at what happens when you draw a one-pixel-width horizontal line. In general, with non-integer position, it'll take up 2 pixels high, in some proportion.)
Still not an argument for quite as high as 4k though.
What's the point of the Nexus 6 having 1440 vertical pixels in landscape mode? Nexus 6's 1440 pixels is a third of the way between 1080p and 4k. So, presumably, scaled down 4k video would look better on it than scaled up 1080p video when the device is held closely.
A very high pixel density makes a real difference when you're looking at printed text and, in general, images with very sharp lines.
That's not what VP9 or HVEC are good for anyway, so I would bet that scaling down 4k video or scaling up 1080p (for the same bitrate) won't make any noticeable difference.
But of course that's just a matter of opinion unless we do a blind test with real use cases.
For a 60" TV that sits 10 feet away, you're not going to see much difference. Move that TV to 4 feet away, and you'll appreciate the extra pixels.
For virtual reality, and other applications that put the screen much closer to your eyes, the extra pixels will make a huge difference.
It's all about the viewing distance.