[chiscript]

I see... There are other solutions that claim to use A.I to train or generate models for their their apps. I'm not really sure how effective they are though. That said, however, Flookup can help you flag dupes quite well. There are many ways I tried to shore up the fact that no algorithm is a one-size-fits all solution. For example, Flookup allows you to dictate what stop words to remove or combine lookup variables for more specificity or even return the next best match in case the first one isn't to your liking. All this makes it quite malleable and usable for a case like yours.

[throw_14JAS]

Thanks for replying! I don't tend to do that type of work anymore, but I'm still stoked to see a solution to the problem I had frequently. I think there's a great service to be built (and maybe it's yours!) that deduplicates data.

Specific models might be an interesting addon. Address parsing, normalization, and deduplication (with potential covariates like phone number, email address, etc.) is a massive pain in the ass for any data engineer who works with sales or marketing folks. Their databases (CRMs) are awful -- it was always a chore to clean these up, but measurably saved money (imagine you mail physical cards, and only want 1 per customer... but you have 5 different contacts at that company for 3 unique individuals).

I would have paid for a deduplication service -- say, quarterly batches at somewhere >$500/quarter for e.g. 20-50k contacts.

The 1-size-fits-all isn't really a value add for me, that wasn't so much my issue. For other target users, I can see that use -- for them, the interface is the value add. Especially if you can read/write Excel files directly.

Stop words aren't something I used in my deduplication efforts. How many of your users request or use this? What kind of stop words do you want to exclude from comparing two entries? I would be worried that stopwords still carry information: "The Store" versus "Store" might be significant.

[chiscript]

I hope that service is mine. I'm really pushing to get more users on board this year.

You have interesting insights, especially about the value deduplication offers. In fact, apps similar to mine charge in the thousands for an annual subscription. I didn't know how much of a pain it was until I released my first paid version last year.

As for the stop words: A significant number of my users either asked for it or presented me with issues that could be solved by removing certain irrelevant text. I don't track usage patterns and such so I added the feature based on this feedback, fully trusting that the users could identify the stop words correctly.

Some of these words might be anything from TLDs to definite articles like "the"; it all depends on the user. Sometimes it's a simple word like "very" or "best", etc., and, based on the data they've shared with me, it looks like the feature was a welcome addition to Flookup.