Harvest festivals were religious- or at least tied up in religion.
You can't just take a core element of a heavy intertwined, complex system without collateral damage. It's like deleting a file in a spaghetti codebase. Half the codebase interacts with any given file in some way or other.
You don't get that in secular society.