The decrease from 2012 to 2020 was much less and generally not statistically significant. Further 2012 was the all time high, you'd expect some regression to the mean.
That's how all time highs work, yes. And it didn't fall to 1980s levels, 2020 was tied for second highest all time in math and was 1 point behind second highest all time in reading.
It's an all time high because it started declining after that. You'd expect some regression, but you don't expect it to fall back to 1980s levels.