I created something very similar. It is a PNG and bootable x86 MBR polyglot. When executed, it rewrites the PNG's image data, to convey the program's output.
To be honest there is no singular intended solution, I deliberately designed it so that there were multiple approaches that could make sense (especially in combination). Intended solutions include but are not limited to:
- Guessing by hand.
- Bruteforce by programatically feeding input into QEMU (as in that writeup).
- Bruteforce by only emulating the "important" code, via something like Unicorn.
- Bruteforce by reimplementing the algorithm in another language (There are some tricks you can use to make it go fast - maybe I should write those up...)
- Reducing the search space by grepping for likely keywords.
- Bruteforce of remaining bytes of the RC4 key (also as in that writeup).