[_kst_]

As the hint explains, it's a combination of PDP-11 and VAX machine code, set up so that either system will run its own code and ignore the foreign code.

You can extract a few ASCII strings from the data. As the hint says: "Can you guess what is printed? We knew you couldn't! :-)"

The ASCII strings I found were "vax", "pdp", "str", "write", and " :-)".

[cptnapalm]

So not only did they write something baffling, they also wrote it so that it was semi-portable.