The decrease from 2012 to 2020 was much less and generally not statistically significant. Further 2012 was the all time high, you'd expect some regression to the mean.
That's how all time highs work, yes. And it didn't fall to 1980s levels, 2020 was tied for second highest all time in math and was 1 point behind second highest all time in reading.