I've been searching for this myself for some time, but either I don't know the proper search terms, or the knowledge is scattered such that the terminology isn't fixed.
This would be some sort of "constraint satisfaction" algorithm (ie, the same basic approach as used in Computer Algebra Systems(CAS) ). There are various approaches to these kinds of problems: someone above mentioned a backtracking algorithm.