[NBJack]

A few ways, if I understand them correctly. On-site people you employ (if you can afford it) auditing the process, connectivity to the HSM to record every key assigned (makes it easy to trace duplicated keys to their origin), and ideally assigning these keys at the very last possible step after the rest is assembled (to minimize 'defective' parts wandering off and being assembled elsewhere).

[swiftcoder]

Yeah, it's important that your own people are doing regular inspections of the production line, keys get burned at the final station (use temp keys for QA stages before that point), and you retain software control of the HSM.

You probably aren't going to do this whole rigmarole for small runs of devices like in the OP.