That still results in a physical change to the building, even if it is only temporary (or not, if the building material is unusually porous).
The test that I remember seeing applied in the past concerns "intention distruction or defacement". As I understand it, alterations (graffiti, breaking a window, removing ornamentation, etc.) that alters the physical object or space can apply.
You're using the word in a different sense than the person you're responding to. They're talking about the standard for charging someone for a criminal act.
As far as I know, where I have lived most(Western Australia) it is not considered vandalism.
This is the third result on Google for "vandalism chalk"(though it seems to have some mixed messages):
https://itsartlaw.com/2013/09/02/california-court-rules-that...