https://en.wikipedia.org/wiki/Multigrid_method
Unless of course your matrix has a prohibitively complicated structure.