[ionforce]

Sounds very much like a genetic algorithm/evolutionary computation.

[f-]

Sure, it's even called that on the project page :-) It just uses an interesting fitness function that knows nothing about the underlying data format - essentially, "improve the edge coverage in this black-box binary".