Hacker News new | ask | show | jobs
by lawrjone 1373 days ago
Author of the article here, and thought it's worth noting on:

> No non-productive environment can help you: They don't have the same data or they do have the same data but the statistic sampling was slightly different.

GoCardless still has a massive Postgres database (10TB or there-abouts) and only managed to scale it by investing heavily in tooling that helps developers work with it safely.

One example is https://github.com/gocardless/draupnir, a tool to create instances of production datasets very quickly (just `eval $(draupnir new); psql` and you have a mini production in ~3s) so you could try things like adding indexes, tweaking the plan settings (`set enable_seq_scan='off'`) and reshaping the data to see how your planner behaved.

I think it's very doable, though the planner still has blindspots. I had a side project to add linear correlation statistics to the planner that I abandoned when I stopped working with big Postgres databases, but that's an example of statistics that Postgres just doesn't track but lead to these pathological edge cases.

I'd rather have the clever planner than not, though. I've a healthy appreciation for the heavy lifting Postgres can do for you.

2 comments

Very true. Postgres is a good product, using it in prod without much maintenance is very doable, and the planner is a good thing that regularly outsmarts me with a strategy that is better than what I had in mind. I am sitting on a 1TB db with it right now, expected to grow to ~10TB in the next few years. I like what postgres brings to the table for me.

But it's not all roses and rainbows. Draupnir seem cool, but it can't help you avoid the problem, only fix it faster.

At the core, there is a trade off here: Performance for predictability. You see the same thing in compilers, JITs and sometimes even processor cores. There is an optimizer in these things, that works 99% of the time, and makes things clearly better than human effort alone allows. But once in a while it guesses wrong, and you fall off a performance cliff.

Meanwhile, predictability is valuable in production, even to the point you might want to trade a serious dent in your performance for it.

Wow. Gocardless has over five years of engineering invested in scaling Postgres, and it was a on going development to enable them to "work with it safely"? At what point did operating safety concerns arise? I'm absolutely no fan of Oracle business practices, but this sounds like a story that sales can dine on for years, notwithstanding my private conviction that I could negotiate Exadata for less than the gross salaries excluding employer contributions.
Of course GoCardless have to invest engineering effort into scaling the technology they use.

The company doubles in size every year, with totally different products and use patterns.

If you're under the impression that purchasing an Oracle database would mean GC can let go of their engineers then I have a bridge to sell you!

As an escapee from Oracle land: I am not at all advocating purchasing it. Postgres in general is great, even if Oracle has some features that are better. The enterprise class insanity of the Oracle world is literally a cause of developer depressions in some cases. Run away.
Hi,

However do you make it that I have claimed that a Oracle license would make your team redundant?

I certainly didn't suggest anything like that.

But please forgive me for being reflexive, has your response accidentally been more revealing than intended about duplicating technologies that isn't AS clear as it might be from the context?

Nobody gets to let go any good database team this side of sanity for whatever reason.

Nevertheless I didn't phrase my comment as thoroughly as I probably should have:

"Working safely with" any asset class data store just shouldn't ever be a question without immediate answers. On going development for the same pirates, doesn't provide executive management with solid answers to "define safety issues present future and potentially retrospectively debugging any failure".

10TB primary dataset isn't considerable amount of production in valuable chain scale of store. Default not a large database on any Oracle installation.

DBMSs require administrative rigors and procedures as well as ideally in depth theory of operations and definitely codebase development skills with the engineering and management system itself is extremely desirable.

However, I can't help thinking that here is a potential case of taking those undoubted talents to directly create proprietary variants of Postgres, which is only going to develop technical debts and future increase in nominally normative support costs.

In other words, I think that your talents have been inappropriately unleashed. You're brain surgeons and everything looks like a brain to you?

Unfortunately and obviously the mere mention of Oracle is liable to create greater difficulty in the creation of openly equitable technical discussions. Oracle management managed to perfect this awful disassociation effect I'm convinced purely for stress testing potential customers often in a gaighting style / hazing sales process . Check out the lady Oracle sales executive who files suit in California every year just to try and negotiate payments of some approximate order to her contractual commission deal. Not many smaller companies get much beyond that not inconsiderable corporate culture clash. I used to joke that Oracle sales cycles were a super proving ground for whether you have a growth business model or not. Because account growth is the keys to sales commissions and simultaneously your easiest leverage for getting big discounts. (Start from 50% before anyone says anything, kept our business afloat)

Ultimately I am unable to understand the point you're making, because you claim inseparable effects from tangential issues and not the smallest misrepresentation of my argument.

Being pedantic, neither of us ought to have used the word "purchase" in relation to licensing nigh inseparable from support and other fat margins. You can still buy a per socket license for RDB , however, and obtain the x64 40% socket discount if you can get VMS X64 running. Sometime soon I'm going to risk indicating that we've renewed interest in such a installation.

The problem I have with the engineering path you took is merely that for the scale and growth outlined, off the shelf solutions absolutely exist and some are thoroughly honed for optimal low administration and even lights out running. We're all potentially caught up in the limitations of early start up scaling of essential computer services, when ad hoc OSS wrangling definitely can sound more attractive than months failing to even understand the small talk spoken by the whole freaking teams turning up to sell you big company shrink wrap software. I'm going round this once again, and have cut myself the budget for dumping all non novel problems onto the most tried solutions. In other end, being ruthless to only spend engineering resources on strategic advantage absolutely can encompass a large license deal or two, but ironically whilst wanting to get rid of Oracle (together with historical deployment and tuning sins) is just that much more attractive a fictional moral campaign than holy war against teaching development teams about transactions on payroll. I'm cynical indeed, but I hope the circumstances are more clear now?