[kiwicopple]

hey hn, supabase ceo here

this one is a long-time coming and it's a continuation of our acquisition of Logflare[0]. Since the acquisition we've be open-source-ing the server, which you can find here: https://github.com/Logflare/logflare

Logflare handles about 1.5 billion log-events everyday on supabase. It's built with Elixir and has no problems with that workload.

This is really just the start of the Logflare updates. All logs are currently ingested into BigQuery, and we are adding support for Clickhouse and other backends (focusing primarily on open source backend). Over time Logflare will function very much like an open source Sentry alternative, where you can ingest data from various sources.

The team will be around if you have any questions about the technical implementation

[0] acquision: https://supabase.com/blog/supabase-acquires-logflare

[maxloh]

Seems that a lot of Supabase's project ls written in Elixir, which is not a very popular choice.

What is the reason behind choosing Elixir?

[kiwicopple]

The tools written in elixir are this Logging server, our Realtime server, and a Postgres Pooler which we launched on HN this weekend [0]

The decision is centered more around the BEAM [1], the virtual machine which powers Elixir/Erlang. It was designed for high-availability systems and it's very efficient at managing large numbers of processes and threads.

In the context of these tools, it's useful for handling open connections and spiky workloads.

edit: Also worth mentioning - we don't care too much about the language. We aim to use the right tool for the job, or preferably an existing open source tool. A good example of this is PostgREST which is written in Haskell. The only person on our team who knows Haskell is Steve (the maintainer). We don't mind because it's the right tool for the job, it's open source, and it has an amazing community.

[0] Pooler: https://news.ycombinator.com/item?id=35501718

[1] BEAM: https://blog.lelonek.me/elixir-on-erlang-vm-demystified-3205...

[doublesocket]

What's the contingency plan if Steve gets hit by a bus?

(very much hoping he doesn't)

[kiwicopple]

We have steve locked in a safe-room.

But we also sponsor other members of the community to work on PostgREST[0]. We don't promote the paid sponsorship much. Instead, we find people who are already doing the work and then offer them sponsorship to continue doing it.

[0] sponsorship: https://opencollective.com/supabase-postgrest

[gregwebs]

There’s more than 1 developer out there that knows Haskell and SQL. You just run into issues if you want those developers to sit in your office next to you.

[kiwicopple]

> sit in your office next to you

you comment is probably metaphorical, but it's worth pointing out how we have de-risked this: supabase is fully distributed. We look for the right person no matter where they live. We're currently 65 people in more than 20 countries

[ThePhysicist]

Sentry is open-source.

[mdaniel]

That is no longer true, it is source available, becoming open source after a 3 year embargo period: https://github.com/getsentry/sentry/blob/23.3.1/LICENSE#L19

[kiwicopple]

Simply drawing a comparison with that comment, so that readers know the direction of the product.

I'll leave my comment above unedited, so that your comment makes sense - but perhaps I shouldn't have said "alternative", and simply left it a "Over time Logflare will function very much like Sentry."

[csunbird]

Sorry for off topic question, but do you plan on doing anything about the egress prices?

I do understand that you are hosting your stuff on AWS, which is subject to AWS egress fees, but the egress fee is just way too much for any serious project (imho)

[kiwicopple]

At this stage we're only using AWS for our infra. I wish we could lower the costs, but would be expensive for us to cover the AWS egress charges for all of our customers.

> but the egress fee is just way too much for any serious project

I assume you're not hosting your serious projects on AWS? Do you host them in another cloud provider?

[monlockandkey]

Why not look into other cloud providers such as Digital Ocean? Much cheaper for compute and bandwidth.

[kiwicopple]

we are definitely looking at other cloud providers, and slowly moving our infra away from AWS. I don't have any timeline on that - it's hard to migrate all of our databases, while also keeping up with the features that our community are demanding.

[swyx]

thank you thank you. the longer i spend as a dev the more i have infinite appreciation for better logging.

this is asking a lot but could we get a breakdown as to what you think are relevant decision factors for choice of logging server? (eg vs logstash/kibana, splunk and i guess people DIY with clickhouse but we can effectively ignore that)

[lyziinc]

Hi I’m one of the logflare devs and I work on observability at Supabase.

Great question. To directly address some of the tools you mentioned:

- Logstash is the transport and transformation portion (along with Filbert) in the elastic stack, and it performs the same functions as vector. It is out of scope for Logflare, which focuses on acting as a centralised server to point all your logging pipelines to.

- Kibana is the visualisation layer of the elastic stack. For Supabase, this functionality is taken over by the Supabase Studio, and the reporting capabilities will eventually converge to compete to match APM services like sentry etc.

- Splunk’s core is not open source, and is very much geared to large contract enterprise customers. Their main product is also much more geared towards visualisation as opposed to bare log analysis.

When it comes to a logging server/service, you’d consider the following factors:

- Cost. Logging is quite expensive, and the way that Logflare leverages BQ (and in the future, other OLAP engines) cuts down storage costs greatly

- Reliability. The last thing that you would want is for your application to take high load and go down, but you’re unable to debug it because the high traffic led to high log load and subsequently took down your o11y server. Logflare is built on the BEAM and can handle high loads without breaking a sweat. We’ve handled over 10x average load for ingestion spikes and Logflare just chugs along.

- Querying capabilities. Storing logs isn’t enough, you need to effectively debug and aggregate your logs for insights. This incurs both querying costs and additional complexity in the sense that your storage mechanism must be able to handle such complex queries without breaking the bank. Logflare performs optimisations for these, performing table partitioning and caching to make sure costs are kept low. This allows Supabase to expose all logging data to users and perform joins and filters within the Logs Explorer to their hearts’ content.

[nextaccountic]

> - Reliability. The last thing that you would want is for your application to take high load and go down, but you’re unable to debug it because the high traffic led to high log load and subsequently took down your o11y server. Logflare is built on the BEAM and can handle high loads without breaking a sweat. We’ve handled over 10x average load for ingestion spikes and Logflare just chugs along.

Can logflare scale out into multiple containers / vms / machines? Is Supabase currently deploying something like autoscaling with kubernetes or something?

[lyziinc]

For the Logflare infra, we manage a cluster of around 6 VMs. Not a very complex setup, and no need for k8s as we have a monolith architecture. We also don't scale horizontally as much as the cross-node chatter increases with each additional node.

The supabase infra is much more complex, however.

[swyx]

those last 4 bullets were what i needed. thank you!

[dale_glass]

Since you mentioned Sentry, do you have any plans to support crash reporting as well?

[kiwicopple]

yes, ideally through existing libraries as much as possible (for example, Logflare currently supports pino: https://github.com/Logflare/pino-logflare).