The setting is "Automatically rearrange Spaces based on most recent use" which explains why the behaviour felt so intermittent.