Cryptographic hash functions are designed to resist existing attacks, so you'll want an understanding of differential & linear cryptanalysis, as well as a variety of algebraic attacks.
I don't know of a good textbook on the subject, so you might find yourself searching keywords on https://eprint.iacr.org/