Hacker News new | ask | show | jobs
by long11l 1890 days ago
having worked behind the scenes I'll tell you a secret. this isn't even new and the data is quite frankly pretty boring

The one good thing about mass data collection is that the data eventually becomes pretty useless, your eyes would bleed just looking it after a while.
1 comments
jokoon 1890 days ago
Useless?

If you give that data to machine learning, it can still give interesting insights, assuming you have the product, shop type, time of purchase, profile of the buyer, etc.

I would have thought marketing people would die for this kind of data quality and quantity.

link
cosmie 1890 days ago
The issue is with the `quality` aspect. There are three different levels of credit card transactions[1].

Most consumer purchases will end up being a level 1 transaction, where the card network doesn't see much more than the transaction amount and bare minimum of details required to process that transaction.

The date is based on the transaction date, which may or may not be the actual time of sale. Many transactions involve placing an initial authorization/hold, and actually making a charge later on (when an item ships, for example).

The merchant data is likely to be unhelpful, as well. For example, the MCC for Amazon is 'Bookstores', which would be wholly unhelpful as a shop type. And while you can do some data cleaning for well known exceptions, the miscategorization issue holds true for pretty much any business that sells more than a single type of something.

So you don't know the specific product that was bought, you may not know the actual time of purchase, the merchant details are general and potentially unhelpful for categorization (or even misleading).

And even the buyer data isn't all that helpful - because of the way credit cards get used, it's more useful for a "household" level profile moreso than an individual's profile.

In general it ends up being far less useful than you'd expect. Level 2 transactions provide a bit more detail, and Level 3 transactions provide line-item breakouts of a transaction. But those are primarily used by vendors servicing businesses and governments. So the detailed transactions aren't consumer-related, and the consumer-related transactions end up too coarse to do much with.

[1] https://tidalcommerce.com/learn/what-is-level-3-data

[2] https://developer.wepay.com/api/reference/mcc

link