Can someone explain to me why weather data is seemingly hard to get/provide? Isn’t data from the US government (weather.gov) in the public domain and completely free? What did Dark Sky and similar provide on top?
The answer is that it is not hard to get. The problem is that you are asking web developers not weather scientists.
The US provides free updated every 15 minute forecast for an absurd number of weather variables for every 3km square of the US in a nice GRIB file package. It is the gold standard on the free domain. Want a real api? Be prepared to pay real money. All Darksky did on top of this was interpolate that forecast data across time and the map.
Without an actual meteorologist on hand to guide the process, the forecast is probably going to be bad. NOAA and the weather community are just fine with the bar being high for entry. A proliferation of low effort/poor forecast web sites would just lead to them removing the realtime data from the public domain.
Relative to the prediction side (which Dark Sky is arguably most known for): Every day, NOAA collects billions of data points (20 TB) of data about the planet's weather, and throws it into their supercomputers to make a forecast. NOAA has the 49th and 50th fastest computers in the world. In fact, the two major applications of the original supercomputers were nuclear physics and the weather.
They make it all available at https://www.noaa.gov/nodd/datasets, but as noted - since the data is so incredibly large and NOAAs already running stats on it... you can't really outcompute them.
[There are very very few CS applications with both a) more computing power and b) more real world impact than predicting weather and hurricanes. Maybe cancer research.]
For me, it was the history data. I could check if it snowed in the last couple days in a new location and whether it melted yet. With most weather apps, the furthest back I can see is midnight, which does not even answer the question of whether there are going to be puddles/snow/slush/ice on the ground that formed since the time I went to sleep.
The main issue I ran into with the NWS API was a lack of stable conditions codes/icons (something like WMO codes that could be easily mapped). They encode some weird dynamic image URL that renders a summary of the weather, but isn't machine readable. I think they're deprecating it anyway. Otherwise, it had pretty much everything through one endpoint or another.
I agree this data should be free and easy to access from the government.
Personally, I've found weather.gov APIs hard to work with. There's no geocoding included, and you can't even use lat, lon directly. But the bigger dealbreaker is that I have about a ~10-20% success rate querying any given endpoint, where sometimes I just get "An unexpected problem has occurred." And this is an automated twice-a-day query, so I don't think I'm hitting any rate limits...
You can use lat,lon with the points endpoint: https://www.weather.gov/documentation/services-web-api , then properties.forecast and properties.forecastHourly endpoints from there to get daily and hourly forecasts. I don't have enough experience with it to comment on rate limiting, though.
It collects raw weather mode data and redistributes weather forecasts with simple APIs
Briefsky is also using it :)