Exactly - if the diamond mines are honest then you don't really need to trust the middlemen. You could also see how many diamonds a certain mine was claiming to produce, and if that number seemed disproportionate to the size of the mine, people might stop treating diamonds coming from that mine.