Being better aligned with astronomical events makes it more correct. The purpose of the calendar is to map the year, the orbit of the Earth around the Sun. The drift in the Julian calendar was not intended. It is a flaw.
I would say that the purpose of the calendar is tracking time/days in general. Astronomical events are just used as a tool to do that. Earlier calendars were lunar-based instead of solar-based, as keeping synchronized with moon is easier.
The solar cycle is extremely important for premodern agricultural societies, since it allows predicting growing and harvesting seasons. If you're going by the Julian calendar and the Autumn equinox is falling on September 10, there's probably going to be confusion as to when the harvest should occur. In the case of the Gregorian Calendar, Catholic countries rely on the Spring Equinox to schedule Easter, and when it started occurring well before March 21, this made it increasingly difficult to synchronize the religious and secular calendars.
While it's somewhat true, most of the people in premodern agricultural societies couldn't read (especially those concerned with when the harvest should happen), and probably couldn't care less about dates in the calendar. Even today, weather and actual crop lifecycle plays a larger role in agriculture than particular dates.
Even if people were tracking dates, adjusting for a couple of days every 200 years wouldn't be that hard: nobody would remember the good old times when we did the harvest on September 22nd in 1234, and now we do them on September 20th in 1434.
Finally, matching up with astronomical events would sometimes put sidereal year (and day) at the forefront: a day that's ~4 minutes shorter than the solar day making the tropical year Gregorian calendar is based on. Things get murky quite quickly once you start going down that path of what "correct" really is.
Note that in the Gregorian calendar, Spring equinox in 2021 and 2022 fell or falls on March 20th. It's only pretty good when averaged out over a 400-year cycle.
Basically, all of these calendar systems are attempts to "square the circle": find something resembling the least common multiple of non-integer values (solar day length and tropical year length), and then try to mix in a bunch of events observed in a different coordinate system (to overly simplify it, all the night stuff is "sidereal").
So we get back to what is really "useful"?
If you don't care about knowing how many days ago, or on what date in the proleptic calendar of your choice something happened in the past just from the date inscribed on it (eg. imagine a letter dated January 5th, 1605), you would certainly be fine with just dropping 10-13 days somewhere along the way. I can, however, understand when someone thinks it's easier to be off from astronomical events for a few weeks to avoid all that administrative trouble, for instance. However, the biggest practical problem today would be that everyone else has written those 10-13 days off, so it's probably easiest to switch too, especially in the global world we've got today.
But there is nothing intrinsically better in the Gregorian calendar that makes it win on all counts. It's just another agreed-upon approximation.
What about the drift in the Gregorian calendar? Solar day and tropical year do not have a least common multiple.
Earth's rotation is — arguably — better measured against the stars, so a sidereal day is more "correct", yet it would not map to our daily routine at all.
So I believe the question is not what is more "correct" (we've long established that we are dealing with approximations at best, Julian calendar included: it already had leap years), but what is more useful? And that's what has driven adoption of the Gregorian calendar in most of the world, but I am fine if somewhere it's more useful not to have to worry about did some days just disappear at some point in the past.
I mean the answer is pretty simple: Neither is perfect, but the Gregorian calendar is objectively more correct. It has a small amount of drift, but the drift is less than the Julian calendar. This also makes it more useful, because it will be able to predict solstices and equinoxes (and other dates needed for scheduling planting and harvesting of crops) more accurately than the Julian calendar. Its overwhelming dominance is evidence that not many people care about the ~2 weeks that disappeared a couple hundred years ago.
You missed my point: where continuous dates are useful, it's ok not to care about two weeks of discrepancy even today. Basically, I am saying is that the biggest advantage of the Gregorian calendar today is its prevalance: all the other things matter less.
You also seem to be overstating the importance of exact dates when it comes to agriculture: my experience is that +-13 days does not make a practical difference, especially if it slowly accumulates (it's not like you would suddenly have to do the harvest 13 days later from one year to the next — you actuslly had to do it 10-13 days "early" once the calendar was switched, and it didn't make a difference even then).
If human civilization continued using the Julian calendar, we wouldn't have been any worse off: nothing points at it that we would have been. I am not saying that Gregorian calendar is "worse" at all (though those born on Feb 29 might beg to disagree when they go 8 years between birthdays 2096-2104 :)), just that where it's better does not matter much.
And here we have the root of the disconnect between wall-clock time and stopwatch time.
In short: How long it takes to cook an egg doesn't change because someone inserted a leap second.
Of course, when people try to do the obvious thing, and measure time durations using a wall clock, that's when the fun begins. In that world, "a day from now" is not a consistent number of seconds in the future, due to the aforementioned leap seconds, daylight savings time, and, potentially, time zone shifts if the person doing the measuring is traveling, or is (or, perhaps, was) in a particularly "interesting" jurisdiction.