[Hydrocharged]

Back then, when we were surveying the landscape, PostgreSQL was definitely in widespread use, but we were seeing MySQL being used in more professional/non-hobby spaces. Nowadays that's not necessarily the case, and we are seeing shops that were once MySQL-only now adopt PostgreSQL for some of their newer projects. With Dolt supporting MySQL and DoltgreSQL supporting PostgreSQL, we're hoping to appeal to both audiences.

[manojlds]

Let's put it this way - someone who would use Dolt would have been using Postgres in 2019.

[Hydrocharged]

The adoption of Dolt has not pointed to this being the case. Many of our (paying) customers have not had PostgreSQL setups, and only around last year have we started to run into potential customers who truly needed a PostgreSQL variant of Dolt. Dolt and DoltgreSQL are open source and will forever be as we believe in true open source, however as a business, we prioritize the features that our customers need.

Either way, with DoltgreSQL now in development, we will now have solutions for both sides! (Other popular databases with their own syntax, such as Oracle and Microsoft, are not planned for the future at this time)

[Terretta]

> The adoption of Dolt has not pointed to this being the case.

That's incredibly self-fulfilling. Counter status quo signals self-select out before even talking with status quo vendors.

[Terretta]

This reply suggests you are able to see point-in-time among your self-selected subset of the world, but not able to see trend-through-time across the overall.

It's as if driving by rear view mirror based on noticing cars around you, rather than having a traffic helicopter high overhead looking across all the roads and ahead.

Inability to see difference in trending versus point in time is how "incumbents" end up having their lunch eaten. Looking at point in time instead of trend, the challenger isn't a threat until they've crossed over, and by then, well, you've lost 5 years. Of course only looking ahead, you might never get anywhere most people want to go today, and you have to get paid. So the trick is always considering both!

All that said, strong pivot now and we'll be checking your take out.

[cyanydeez]

probably has more to do with people counting the number of mature projects vs greenfield development

by sheer numbers, MySQL is probably still dominant but if projects are starting today it's likely postgresql

[whizzter]

Why a fork? Couldn't you just add a separate front-end? Will the underlying storage format stay the same so one can flip over or do you want to get closer to PG semantics?

[Hydrocharged]

This isn't a fork of PostgreSQL, it's a completely bespoke database solution. Its only tie to PostgreSQL is that we've chosen to appear as a PostgreSQL server to clients. If a user didn't use any versioning features, then the goal is that they should be unable to tell that they're not on an actual PostgreSQL server.

The versioning features are an important distinction though. Dolt (production ready, MySQL protocol) and DoltgreSQL (pre-alpha, PostgreSQL protocol) are built specifically to address the lack of versioning support in databases, and gaining these versioning features is as easy as swapping out the database you are using for Dolt and DoltgreSQL (once it's finished). MySQL and PostgreSQL are written using C/C++, while Dolt and DoltgreSQL are using Go, so there is no shared code. The storage format is implemented using prolly trees (https://docs.dolthub.com/architecture/storage-engine/prolly-...), which are based on merkle trees (used by Git and Bitcoin), so there is no overlap with any existing database solutions.

[samsk]

Hmm, when choosing database solution for a new project I'm not selecting sql dialect (like postgresql sql), but stability and ecosystem. So having this as an extension to postgresql, and possibility to combine it with other extensions it would be great, but reimplementation is a no go here.

And I can not use it for existing projects, because again extensions, and I surely don't want to findout how your implementation differs from the mainstream postgres...

[Hydrocharged]

Knowing that extensions are very important to you is great feedback for us. As we hear more about what users' requirements are, it helps us better plan for the future.

Regarding any differences from mainstream Postgres, you can look to how we've handled Dolt, which is production-ready. It targets MySQL, just as DoltgreSQL targets PostgreSQL, and it recently achieved 99.99% correctness according to a set of roughly 6 million tests (https://www.dolthub.com/blog/2023-10-11-four-9s-correctness/). This test is not a definitive stance that we are exactly 99.99% the same as MySQL, but it's a good general guide to how we approach our compatibility, and how serious we are in that regard.

Sadly though, the full versioning capabilities would not work as an extension to Postgres. We looked into it before we settled on our current approach. I talk a bit about it in the blog post as well. To truly allow versioning in the same capacity that Git does for source code, it required us to either fork Postgres and spend years reimplementing all of the work that we've done in Dolt just to get to where we are today, or choose the path that gets something out quickly, and allows us to have the very conversation that we're having right now.

That was actually the very reason for deciding to host an announcement that we're working on it. Many people have said that they'd like Dolt but for Postgres, however they've not said whether they need the Postgres binary specifically, the ecosystem, the syntax/wire protocol, etc. This announcement gives us the opportunity to receive that feedback.

[dotancohen]

Where are the 0.01% differences? When I'm trying to commit transactions? During select? Just not supporting some esoteric stored function syntax?

And most importantly, how does Dolt compare under heavy load, at the limits of server memory or bandwidth or CPU or disk thoroughput? You can assume an SSD and a multicore processor for purposes of answering.

Thank you very much. You are competing in a field where trust is extremely difficult to acquire - and the consequences to a lead dev for choosing Dolt[greSQL] could end his career. Nobody ever got fired for choosing the incumbent, as variations on the saying go.

[Hydrocharged]

The sqllogictests are better used as a judgement of how accurate our results are to MySQL's results given a very large range of statements (around 6 million tests). They are not, however, indicative of the entire feature set. That's to say, there are a few things that are not yet supported in Dolt, however they are the more esoteric features. Things like transactions, etc. are what I would consider "core" SQL, and we are very keen on making sure those work exactly as you'd expect

There are two key points regarding performance that I'd like to mention, and that's that Dolt uses higher-than-expected disk, and we also use more memory than MySQL by comparison. The disk usage is due to our optimization of speed at the sacrifice of disk accretion via temporary storage, which we've decided is a fair trade off considering disk is very cheap (and we're working on making this tunable so users can decide on speed vs disk efficiency). Memory usage is a bit more complex, and I'm not the employee to comment too much about it, but both of these issues are being worked on to reduce their impact. With that in mind, our performance is comparable to MySQL as long as the machine limits are not being reached, however I'd expect us to be a bit slower once those limits are reached, simply due to the extra complexity that we're managing.

Lastly, trust is something that can only be built over time. In 10 years, I'm sure that there will be no doubt of our stability, and at that time I can see Dolt and DoltgreSQL becoming the de-facto databases used for relational storage. Of course, I may be a bit biased :)

Trust is a

[tomhallett]

It sounds like question is: why fork Dolt to make DoltgreSQL?

[Hydrocharged]

Dolt was built with MySQL in mind, and we're creating DoltgreSQL with PostgreSQL in mind. We've gotten interest in a "Dolt for Postgres", and so we're finally starting development on that exact thing.

[dotancohen]

Is there any compatibility between the underlying storage systems of Dolt and DoltgreSQL? Yes, the interface is slightly different, but is the storage (partially) compatible? Why aren't the two SQL dialects different interfaces to the same underlying storage?

The reason that I am asking is because I have a hard time trusting a newcomer to the database competition. And for purposes off discussion, DoltgreSQL will be a newcomer for the first ten years of its existence. Sharing the underlying storage model with Dolt (still a newcomer, sorry) would greatly increase my confidence in it.

[Hydrocharged]

You can think of Dolt as having 4 layers to it. From the top-most interface to the bottom:

Vitess (Handles Wire Protocol)

go-mysql-server (GMS, Implementation of MySQL featureset)

Dolt GMS interface

Storage Engine

DoltgreSQL, right now, shares the storage engine, interface, and a portion of GMS. This is primarily to get something up and running so that we're able to further iterate on a "working" product (i.e. something we can set up testing for). I envision that GMS will split into three, with a MySQL & PostgreSQL core interacting with a base SQL core. Dolt's GMS interface would then adapt to the base SQL core interface. The storage engine is more-or-less set in stone.