Isn't the whole claim to fame for AFL that it largely mitigates or avoids that problem by tracking branch coverage so it doesn't waste time permuting the input in ways that don't change the program behavior meaningfully?
AFL is great for most file formats (e.g. ELF), but probably not suitable for video formats like H.264, which uses complex encoding even for things as simple as frame width/height in the header (see things like ue(v) and CAVLC).
It will take ages for AFL to generate a valid H.264 NALU that isn't rejected outright.
How do you know *ahead of time* which mutation of input will result in new path in code? You don’t. What you can do deduplicate possible inputs for mutation based on the branches taken/path.
It will take ages for AFL to generate a valid H.264 NALU that isn't rejected outright.