I've used grid with IE11. You need to add some extra prefixing noise to your css. The way you identify grid elements is different and it sucks to duplicate the layout information.
It doesn't with IE10 though. And there's no way to fall back to flex or another column system.
Which means, if you need to support IE10, you need a whole parallel set of styles. There's no graceful degradation when it comes to overall site layout, unless you're willing to serve the single-column mobile version to older browsers.
One scenario I can think of is a non-IE user telling an IE user to go to a particular website and click the button that's at the top of the right sidebar. For the IE user, it might be a stacked layout where the sidebar actually appears under the main content. I would call this a nuisance more so than a major issue, but nevertheless it is something to consider.
I think the average computer user knows that websites often look different on computers vs. phones, so they might be confused when a website looks different than normal on a computer. Like I said, I would only call this a nuisance. Certainly not a big deal.
I think it depends of if you need trafic from those browser or not.
Because if you do, you'll have to maintain several version, or increase your time on working on it to do gracefull degradation (which is of course a good way to do, but more expensive)
Which means, if you need to support IE10, you need a whole parallel set of styles. There's no graceful degradation when it comes to overall site layout, unless you're willing to serve the single-column mobile version to older browsers.