[Waterluvian]

A few thoughts from a geographer (be prepared to shoot me):

- Basically every comment is wowed by this, but nobody questions what the accuracy is. I, too, can Krig interpolated surfaces to any resolution.

- off-nadir view doesn't seem to offer much

- We've been dealing with janky tile loading for like 20 years now. I really hope we'll get a much smoother approach for viewing these tiles as they load. The dissolve transition hides it a bit, but makes the data uncomfortable to view when playing the timeseries.

- I'm deeply curious about the Picnic data layer. Can someone share the ArcGIS/QGIS model for that one? =D

[samvher]

This seems to be a thing with weather models more generally. Somewhat relatedly, I've spent quite a bit of time evaluating weather models for use in India and Africa, and while predictions are easy to find, validation results for the predictions are very hard to find. And when you do find them, the results are pretty poor, with many models performing worse than if you would say "predict temperature on date X to be the average observed temperature on the same date in the past 10 years". But people still sell (and buy) these predictions!

Weather predictions seem to be accepted quite uncritically. Perhaps people have a lot of confidence in the smart people that built these predictions (a bit like how AI predictions can sometimes be accepted uncritically).

[Waterluvian]

100% agree. Scientists and engineers all know that you must provide validation results, accuracy/uncertainty calculations, etc. or your data is just a pretty guess. I think weather forecast models are so commoditized and useful for laypersons that we've UX'd all of the complexity (scrutinizing the data) out of the product. The most scrutiny I ever see are people discussing what "Probability of Precipitation" values really mean.

My grad thesis advisor encouraged me to actually get the Environment Canada models and learn how to run them (they're in FORTRAN). I could never make them spit out data consistent with what EC publishes. That's probably on me, but it was a real eye-opener to this whole domain's complexity.

[duanem]

I've been working with weather models for 10 years and I often get asked "How accurate is X?" or "Which model is more accurate?" Many people think "accuracy" is a single number or a single thing - it is more complex than this and depends on your needs.

This chapter on Numerical Weather Predictions [0] is great, especially the section on "Forecast Quality and Verification" (p777). The eye-opener for me was "Binary/Categorical Event". An example of a binary event is rain, one model could predict rain correctly but a second model might not predict the rain at all. This doesn't mean the second model was completely wrong, it still predicted the rain but it predicted the rain passing further to the south.

[0] https://www.eoas.ubc.ca/books/Practical_Meteorology/mse3/Ch2...

I've also noticed some model are better than other at predicting one phenomena while other models might be better in certain regions. For example, many people report that Canada's GDPS is better at higher latitudes whereas NOAA's GFS is better at equatorial regions.

One final note, just because someone is solving an WRF model without verifying the results, doesn't mean it's wrong. Many numerical techniques and physical models within WRF have been validated analytical and experimental models. But it is also true that someone can naively setup a WRF model that gives bad results.

I use a 900m WRF model that predicts the wind shadow around an island and we use it to find the best beach for a picnic - and it works. But this same model predicts the general pattern of rain but it doesn't get the start and stop time of rain correct.

People get fixated on accuracy as a single thing and use it as a single basis for argument but to take a quote from the chapter [0] above "One of the least useful measures of quality is forecast accuracy" (ref. p777, Forecast Quality and Verification, third paragraph).

[CalChris]

> other models might be better in certain regions

The US Navy's COAMPS model is good for littoral regions.

[carabiner]

Meteoblue was dramatically more accurate in Chamonix last spring than the GFS.

[duanem]

You have to be careful you aren't comparing apples to oranges. You might be looking at the Meteoblue MOS (statistically corrected) predictions which might be based on their regional weather simulation. This regional simulation might be nested in a larger global model, probably from ECMWF. If you compare this ECMWF model to GFS, then you are comparing apples with apples.

I find global models like GFS are great for understanding the large scale weather systems. The regional high-resolution models, which are usually nested in a global model, give better definition of local weather phenomena like wind shadows or cooler temperatures in valleys.

Dues to averaging, weather simulations usually have a bias error in temperature predictions. These errors are corrected using statistics (look up Model-Output-Statistics) but is hyper-local, i.e., you loose the big picture. This is probably what you're looking at with Meteoblue.

[lbrindze]

Given this is in the SF bay there a number of high quality observations that you can use to validate the forecast skill (unlike India and Africa). I have not bothered doing this here since… well that’s too much like my day job.

I’m always excited to see new forecast products, generally. If I were to guess (as an above comment did) it looks like they are applying some dynamic downscaling on top of either a custom WRF model (expensive and complicated) or more likely already available weather model data like the HRRR, which still would represent a 10x resolution increase.

I’m more curious what the refresh rate is. Anyone can get a super accurate forecast for the next 3 hours that takes 10 hours to run, but at that point it’s no longer a forecast by the time the data is available.

I still think that windy has set the standards as far as modern weather visualization goes. Not saying everything has to be particles but other things (like the inclusion of isobars) is really clean and not trivial to execute.

Either way this has definitely piqued my interests and I will be keeping an eye on it, their advisory board looks legit (at least in the meteorology end)

[mturmon]

The website claims to be using DL which may mean less of a model-centric approach? The expertise of the people at the top of the organization, on this problem, seems a little thin, TBH. And, no stated validation results at all? Without such details, this is just marketing.

It would be interesting to see how this behaves for longer prediction times and across a range of difficult forcing conditions off the ocean in the BA.

[lbrindze]

I agree, this generally left me feeling skeptical. I know of Luca Delle Monache on the advisory team, through colleagues who have researched under him at Scripps and they spoke highly of him. But yes, there is a lot left to the imagination here.

With regards to the sfbay specifically I used to work with a fairly high resolution wind model for the bay (this was a more traditional dynamic based simulation) and it worked pretty well overall, but every time a storm blew through it would crash. This ultimately had to do with the relatively steep terrain in the bay specifically (and the physics configurations we were using in the actual model).

Even if they are using DL they still need initial and boundary conditions. As I said there are a ton of weather stations around so I could imagine a DL type approach that looked at terrain elevation, and recent + historical observations to initialize a forecast, but I still imagine that boundary conditions would have to be provided by nesting this in a larger model somehow. Then again, I'm not a DL expert at all so there are probably some newer stuff in this field that I'm just out of date on.

Its really expensive to run your own dynamic forecast model, at a refresh rate acceptable for an actual forecast, at this resolution. That's why I suspected its taking existing weather models and downscaling them with DL techniques, but I can't really know just by looking.

[mturmon]

(For clarity, I was referring to the company leadership proper, not the advisory team.)

[johmathe]

We are currently integrating with Forecast Watch a 3rd party that analyses and compare various forecasting systems [1]. Please stay tuned until we integrate our APIs. I will be updating this thread when it is ready.

[1] https://www.forecastwatch.com/

[hadlock]

Few thoughts as a sailor in the SF Bay area.

- Accuracy seems at least somewhat correct.That wedge shape you see in the late afternoon sailors call "the wind engine". Local sailing magazine Lattitude 38 has a special PDF that talks about doing a sailing trip around the bay accounting for this local wind phenomenon.

Correct stuff:

- The SF waterfront, out to the edge of the piers is mostly calm which is correct

- Berkely, Oakland, Emeryville getting blasted late afternoon is correct

- Back side of treasure island, immediately to the east is much lower than the west side, particularly near clipper cove

- Vast majority of alameda estuary is dead calm, that's correct for this time of year

- There's a big blast of wind between Daly City and OAK international where there's a gap in the mountains

Weird stuff:

- Most noticable, is the wind is still strong up to and south of the bay bridge. The bay bridge has been described by many as "a wall" when it comes to the wind. There's a drop off but it's not in line with the bay bridge. At all. at least 45 degrees off from true wind speed.

- There's a very windy patch between golden gate coast guard station and belvedere, it's usually really patchy wind here but I guess if the wind direction is just right it'll blow there

- Pointe Bonita (lighthouse on the west side of gg bridge about 2 miles, north coast) they are modeling the gap in the rocks there and you can see it funnel through which is neat

It's a cool visualization though, gives you a great idea of where the wind is, and more importantly where it won't be. There are a bunch of races that start in the bay and head south towards Santa Cruz and Monterey so it's nice to better visualize where the wind just dies off on the coast as it skips over the mountains.

Anyone who wants to see what the wind is like in the bay I recommend reaching out to YRA.org they can put you in touch with a boat who needs crew most likely. There are races 4-5 days a week through november all around the bay. It is modeling a distinct drop off of wind speed on the south side of the bay

[_yb2s]

Also, as a person with experience sailing on the Bay, this model immediately seems unusually accurate. It shows, for example, the Angel Island wind shadow moving around correctly as it does during the day, which I have never seen in another model.

Could anyone with more understanding of meteorology (or OP) please explain what is different about this model vs say the ECMWF model that you can see in apps like Windy, that are supposedly great, but just don't seem to get these features right? Those models are incredibly bad when dealing with the unusual local geographic features on the bay. What resolution are they operating at?

[johmathe]

Thanks for the feedback :) The highest resolution models you can currently find on windy is around 3x3 km (HRRR) and this resolution is unfortunately too big to capture fine terrain and water features. 300x300 gives you 100 times more data points to work with.

[_yb2s]

How often are you planning to re-run this model? Can I convince you to rerun it before this weekend? I'm in a situation to get you a lot of new followers/users if I had a new model for Sat/Sun.

[johmathe]

The model is currently ran every day. Hope you were able to use it for this weekend :)

[counters]

It's purely spatial resolution and the representation of topography that high-resolution simulation permits. Any NWP model like WRF or WRF-LES will produce topographic-driven wind fields with high fidelity pretty much out of the box, without any customization required. The result may be pretty, but it's really nothing of note from a weather modeling perspective.

ECMWF's model is 9km spatial resolution, so Angel Island probably doesn't even show up in the model domain.

[johmathe]

Very valuable feedback - thank you!

[wmeredith]

Thanks for chiming in. I can make an animation say whatever I want. I am very curious about the accuracy.

Unrelated: The site breaks my back button in Chrome, which is an unforgivable UI sin.

[johmathe]

Thanks for the feedback. We will have the back button fixed quickly :)

[thot_experiment]

Same thing in firefox, and pleeease have a units switch. Would be infinitely more useful if I could view the site in the units my brain normally works in rather than having to think about the conversion all the time. Very cool site otherwise, also very cool bay area right now.

[johmathe]

All fixed up and deployed.

[thot_experiment]

<3 thank you for the quick fix! no unit switch yet right, or am I missing it?

[johmathe]

It will come within the next few days - please stay tuned. Being from Europe I am more used to Celsius myself :)

[_Microft]

I had been wondering if that was actually something special because I thought I remembered that the German Meteorological Service ("Deutscher Wetterdienst") offered accurate forecasts on a sub-kilometer grid for years already. At least if you are ready to spend money on that because that service is not free, so maybe that's the innovation here.

[namibj]

Where did you get the existence of sub-km (horizontal) grid forecasts from? Thus page https://www.dwd.de/EN/research/weatherforecasting/num_modell... contains the following sentence: "The operational NWP models of DWD currently employ horizontal grid mesh widths between 2.8 and 13 km."

The convective cell tracking for nowcasting seems finer, IMO reasonable as it's about predicting watersheds down to <10km² area flash-flooding and causing the local creek to swell to actually dangerous levels/requiring partial evacuation of a valley.

[s1artibartfast]

>Basically every comment is wowed by this, but nobody questions what the accuracy is

I am very skeptical. Does the San Mateo bridge really block 10 knot winds for the entire south bay? Similarly, the land temperatures all seem the same close to sea level.

[hadlock]

Tall bridges do weird things to the wind. I can confirm the bay bridge at surface level, there is functionally no wind for about half a mile downwind from it. Just glassy smooth.

[s1artibartfast]

Most of the san mateo bridge is quite low, especially the stretch crossing the bay. This is why I was so shocked it had a wind shadow 10+ miles

[hadlock]

I'm not seeing much of a wind shadow for that bridge, particularly at 4pm Friday. Maybe they updated the model already. Most of the onshore windflow begins after 11am goes from the cold (high pressure) pacific through the gg bridge, wraps around the east side of angel island and north past Richmond and Vallejo towards the hot (low pressure) central valley. South of SFO silicon valley is surrounded by tall geographic features and there's not much path to hotter (low pressure) zones so it's unusual to see high winds there unless there's a special offshore wind event coming from the south (most often in the winter).