[EliRivers]

Xadoc's advice above is good; unit tests. I work with poorly documented protocols that have been implemented "around the theme of the protocol" by hardware from a variety of suppliers, and this is how we work out its quirks.

A battery of unit tests, starting with the simplest functions it offers, and thence upwards into more complicated tests (i.e. chained calls of the presented functions) where we track what internal state we think the system should have at that point in the tests and interrogate it to discover what internal state it really does have.

[mytailorisrich]

These are exploratory tests rather than unit tests, but your point stands.