One can use any finite field. GF(256) is popular, since it allows computers to split one byte at a time. Plus, there are many implementations of GF(256) operations, since the same field is also used in AES. In GF(256), the addition operation is indeed XOR, as you write.
If your field size is a prime number, the maths become simpler, since you can use "regular" integer addition and multiplication modulo the field size.
If your field size is a prime number, the maths become simpler, since you can use "regular" integer addition and multiplication modulo the field size.
For psst, there is some information here about why we chose GF(5): https://github.com/Sjlver/psst/blob/main/docs/design-choices...