Hacker News new | ask | show | jobs
by logvol 3308 days ago
Is the data freshness a joke? 2 DAYS to get data into the data warehouse? Stitch, Fivetran, Segment, and more can do way better than that without any internal hooks.... :/

> Additional processing time–up to 48 hours–is required to make your account’s transactional data available in Sigma. This means that it does not reflect your account’s most recent data and should be considered a couple days behind. The Sigma interface displays the date and time of the last update to your data.

4 comments

I've seen plenty of reporting pipelines that are that slow over the years. If this was built on the cheap, so it just uses existing pipelines, and instead of working through streaming, it regenerates the world every night, a 48 max failure with some is not out of the question once you add some CYA magic. That would make this pretty cheap to make: Some website work, boxes for queries, and some security work to make sure data from other customers doesn't leak.

Given how Stripe seems to build products in a lean way, it'd not surprise me if they are just launching like this and measure customer reaction. If the main reason it doesn't get the traction they want is the 48 lag, they'll just rewrite their reporting pipeline to use streaming, and the product gets faster for free.

For decision making, hopefully you're looking at more than the past two days.
Does that have anything to do with transaction settlement or just pure lag?

I can understand an async mode of updates being partially behind but two days seems excessively laggy without a business reason for it.

It's payments. Miss a single row and customers will balk and scream at the missing $.45 that they think they are owed. Better to buffer with a longer SLA until you feel very confident that your pipeline is bulletproof before you try to approach 24 hours or less.