I bet it's just a cost/benefit analysis. An API is a way to get more eyeballs by motivating 3rd party developers to integrate and publicise your service. HN does not need that: it has enough traffic as it is, and given the target audience, you would see an instant proliferation of half-assed apps hammering its endpoints. So it would be an additional cost for no real benefit.
The current situation (PG and friends optimise a basic but very accessible website, and a handful of third parties build APIs on top) is much more manageable.
I don't believe that HN restricts or discourages the scraping of HN content in any way... Other than the restrictions here: https://news.ycombinator.com/robots.txt
If you have a fabulous idea for how to use the data contained on this site, I'm sure everyone will be impressed and interested to see it.
The current situation (PG and friends optimise a basic but very accessible website, and a handful of third parties build APIs on top) is much more manageable.