[lookforr]

nice article. a few questions here

1. scalability: does your system ingest multiple documents in parallel? if so, how often do you observe over-segmentation, if any?

2. thresholds: how did you set the thresholds at various parts of the systems?

[jrfinkel]

1. We do ingest multiple documents in parallel, but we also have a system in place for merging clusters, so it ends up not being a problem. 2. Pure hackery. Tweak, look at data, repeat.

[felipehummel]

Can you elaborate on how you merge clusters? Many times a clusters will be just single-doc clusters, how to check if you need to merge two single-doc clusters? (it seems like the same process that was done in the first place to create the clusters: doc x doc similarity)

[jrfinkel]

Merging is pretty simple, and could probably use a little more TLC. The way we do it is that when we get a new document in the system, if the similarity score is above some threshold for documents in two different clusters we will consider merging those clusters. We then make the yes/no decision by comparing random documents from both clusters and averaging the scores, but the threshold we use here is a bit lower than the non-merging decisions (since we have the additional information of this new document doing a good job of linking the clusters).

[lookforr]

this doesn't work in the case where there are two (and only two) similar documents get ingested into the system as new singleton clusters at the same time; this case is very rare so it is not a big issue to you, i guess.