Outliers are likely to drag the averages down, not up. The only really substantial global outlier events are volcanic eruptions, which cause a net cooling. Everything else (hurricanes and the like) is just heat moving around
Once we recognize that we're talking about local rather than global there are many local outliers that increase temperatures, countering your argument that volcanic eruptions are the only global outliers.
And that means their measurement contributes to the noise when attempting to identify the long-term causal trend. Taking the median is the correct step. One doesn’t measure the output and assume it’s the input.