[tptacek]

"Questionable technology decisions". You're trying to dunk, but I don't think you understand where the hoop is. Their technology decisions have panned out Tailscale well. We should all be so fortunate. This isn't Twitter with the "fail whale"; the only reason you know about any of this stuff is because they wrote about it. They ran their entire service with a JSON file backend for 18 months, and switched from it to etcd without you even knowing about it.

[jeffffff]

i really don't know a ton about this product or team but it sounds like if they had used aurora mysql or aurora postgres in the first place then there would be nothing to write a blog post about because it would've just worked and kept working. they say they want to avoid vendor lock-in but if the vendor became a real issue they'd be doing their first migration instead of being on v3 already. additionally, their bespoke solution relies on s3, which is also a vendor-specific technology, so it seems like they haven't avoided vendor lock in? i've seen many cases of developers doing more work to avoid vendor lock-in than it would take to replatform if it ended up being a necessity, and this really feels like that looking at it from the outside. i'd understand this better if mysql or postgres couldn't solve their problem, but that is not the case here, and i can't wrap my head around a company who is ok with their devs reinventing a very good wheel 3 times when the obvious choice would've worked fine the first time. it seems like they are successful in spite of these decisions, not because of these decisions.

https://mcfunley.com/choose-boring-technology

[tptacek]

You should start by learning more about the product, and then tell them they should use Aurora for all their backing store.

[hokus_p2]

>> it seems like they are successful in spite of these decisions, not because of these decisions

That is my conclusion, too.

[mrkurt]

The thing about startup decisions is: most of them are "wrong". Or they start right and become wrong later. Successful startups aren't successful in spite of their wrong decisions, they're successful because they can change them very quickly (then write a blog post about it and get more customers).

There's also a strong correlation between people who look at things from weird angles and also build good products. Why are you surprised the people who invented an entirely new way of doing VPNs also don't cargo cult database storages?

[jeffffff]

it's not cargo culting if it works, and they even say in the article that mysql or postgres would've worked

maybe next they will stop cargo culting operating systems and switch to SerenityOS?

[tptacek]

They're talking about running many instances of MySQL locally, not hooking all their systems up to RDS.

[jeffffff]

no they aren't

Tailscale’s coordination server, our “control plane”, has become known as CONTROL. It’s currently a single Go process on a single VM.

source: https://tailscale.com/blog/an-unlikely-database-migration/

[bobnamob]

The s3 api is rapidly becoming a distributed storage standard. Building your product around it is hardly lock-in these days.

AWS does offer one (very reliable) implementation but they’re very definitely not a monopoly.

[yawaramin]

Sounds like you need to do a lot more research before commenting on this? It's pretty easy to find in the Litestream docs that it replicates to S3, Google Cloud Storage, Azure, and other options even including SFTP. In fact by off-loading the storage integration details to Litestream, the Tailscale people now get seamless storage vendor independence almost for free.

This is what a really smart and future-proof solution looks like.

[jeffffff]

or they could have used mysql or postgres hosted by aws, gcp, azure, etc? i would put money on this not being the last time they change databases

[tptacek]

Again: this is an infrastructure component, running on many machines internally. It's not a Rails app. We use SQLite in very similar circumstances across hundreds of machines. Having all those machines schlepping all their reads back to RDS would not only be untenably slow, but it would also make the whole system less stable. I don't think you've really thought through the design at all, and you should before making comments like these.

I'd put money on this not being the last database change too! But not for the same reasons you would.

[mwcampbell]

Unless I'm misunderstanding something, it seems to me that Tailscale's controller, discussed in the previous blog post [1], is like a typical CRUD API server, currently only running on one machine (they're not using Litestream's work-in-progress live read replicas), not something widely distributed like Fly's service discovery infrastructure.

I'm not joining the "just use RDS" crowd, but it looks to me like using a managed DB service like RDS would have been a reasonable decision for this application.

[1]: https://tailscale.com/blog/an-unlikely-database-migration/

[yawaramin]

I'll take that money quite easily ;-)

[IshKebab]

They're not questionable because they didn't work, they're questionable because they were so difficult to run that they had to migrate twice.

You could absolutely run a web site using Brainfuck without any failures or any customers realising it. Doesn't mean it's not a questionable decision!

[hokus_p2]

The etcd solution was highly questionable. Actually, a prime example for reinventing the wheel, but worse.