[yuliyp]

Keeping a calendar is fairly simple (requires knowing writing). Observing which day the solstice is on is also fairly simple (requires a sundial: the shortest/longest shadow at noon are the solstices. If you're in the tropics just observe when the sun passes directly overhead instead). Observing that the solstice moves by a day every 4 years isn't too hard by analyzing those observations.

[amluto]

The tricky part seems like it would be that one needs precision quite a bit better than one day. So either the measurement can be done over the course of many years or the time of the solstice needs to be determined to excellent precision.

[yuliyp]

It requires someone to at some point remember that "Hey when I was a kid 20 years ago the solstice was on December 22 but now it's on December 17, what happened?" and then go looking at historical records to generate a trend line with a slope of 0.25 days / year. The measurement doesn't have to be at one point.

[amluto]

Sure, but it needs an extra digit or two do do as well as the Gregorian calendar. Doable, but longer records are needed.

[yuliyp]

The adjustment to 365.25 came during Roman times. The Gregorian adjustment came after well over a thousand years to realize the drift since Julius Caesar's time.