[pauldix]

InfluxDB creator here. I've actually been working on this one myself and am excited to answer any questions. The project is InfluxDB IOx (short for iron oxide, pronounced eye-ox). Among other things, it's an in-memory columnar database with object storage as the persistence layer.

[spenczar5]

You are the CTO of a significantly-sized tech company. How do you find the time to work on this, and how do you justify your use of your own time, versus hiring and grooming a team, like more traditional CTOs might?

I hope this doesn’t come across as confrontational; I find this inspiring.

[pauldix]

Not confrontational at all. In short, it's taken me years to be able to do this. Last year we finally got our engineering organization to a point where I no longer have any direct reports. Instead, everyone rolls up to our VP of engineering. In most mid-sized startups (like ours), the VP of engineering is the person that manages most of the process and generally makes sure the trains run on time.

My view of the CTO role is that it's focused on more long term efforts or the product generally (in more technical companies). So that's what I've been doing for much of this year. With the caveat that I'm also the founder and on the board so there is still about 10 hours a week of executive meetings/management, reviewing of other people's writing & product efforts and working with our biggest customers and prospects.

All that being said, I'm not just working on this because it's what I want to do and it's what excites me (which it does). I also think it's my highest point of leverage within the company. Very few people have the same view and in depth experience in this problem domain. Most of the people I know of that do are either founding and running competitive offerings or they're working on similar projects at Google, AWS, or Azure and getting paid significantly more than we can afford to pay a single engineer.

I think this is one of our most important efforts right now and the best way I know to make it successful is to be working on it in depth. I'm not the best programmer on the team or the smartest, but I have some depth of experience that gives me a clear vision on what it should be able to do and what tradeoffs we should be making.

My role on this effort is basically as the product person and the tech lead. In this case the tech lead isn't the manager of people on the team. We try to have a multiple paths for advancement in engineering and one of them tech focused individual contributor.

Of course, I view this project as necessary, but not sufficient for our overall success. That's why most of our engineering team is working on our legacy products and the continued forward development of the overall platform.

[davidkell]

The architecture you've proposed here has many overlaps with q/KDB (or Jd, Shakti), which does not (AFAIK) have a decent open source alternative. Looking forward to see what you build!

[kthielen]

Re: no open source alternative to kdb

I built this for Morgan Stanley (the same place where kdb originally came from) and put it to use in some major high-volume/low-latency trading systems where a significant fraction of US equity trades execute every day:

https://github.com/Morgan-Stanley/hobbes/

[davidkell]

How is Hobbes support for relational workloads? My favourite feature of KDB is mixing SQL-like queries with list manipulations. I see support for tables in the Hobbes docs [0], but an array of tuples is row-oriented whereas KDB is column-oriented (flipped dict of lists). Am I missing something?

Nice job btw.

[0] https://hobbes.readthedocs.io/en/latest/language/types.html#...

[kthielen]

Yes similar things are done in hobbes, although with array comprehension syntax instead of SQL. Grouping and joins are done with regular functions. I think that this approach scales better than fitting it all into one query syntax (though I don’t feel strongly about the syntax).

Yes you could say it’s row oriented and not column oriented. It’s possible to arrange data in column vectors, and that does get better performance in some situations. I have some ideas to unify the two approaches (so either layout can be decided, ad-hoc), but need a better approach to dependent types.

[davidkell]

Thanks, appreciate the response.

[snark42]

Depending on your need for KDB, old school InfluxDB, OpenTSBD, SciDB and Promethius are open source alternatives.

[valyala]

There is also VictoriaMetrics open source alternative with similar architecture and comparable performance - see https://github.com/VictoriaMetrics/VictoriaMetrics/

[davidkell]

What's the secret of the performance? Especially interested to see that it is written in Go.

[AndrewKemendo]

I am bullish on better time-series systems, but why would I use InfluxDB IOx over Kafka?

I can choose whatever memory I have available for each topic/partition to do a query - or KSQL to transform topics.

edit: I should say that I can recognize a few benefits from columnar memory and lower level management with Rust, but it would be a huge infrastructure/tooling shift for a regular CRUD API shop or one that has already invested in any eventing system IMO.

[ddorian43]

We are talking 10x+ less storage/servers when working on columnar storage.

[valyala]

Time series databases and Kafka are intended for very different workloads.

Kafka is used as distributed message queue. I.e. a set of apps produce opaque messages and send them to Kafka queues, while another set of apps consume these messages from Kafka queues.

Time series databases are used for storing and querying time series data. There are no Kafka-like queues in time series databases. Every time series has a key, which uniquely identifies this time series, and a set of (timestamp, value) tuples usually ordered by timestamp. Time series key is usually composed of a name plus a set of ("key"=>"value") labels. These labels are used for filtering and grouping of time series data during queries. There are various time series databases optimized for various workloads. For example, VictoriaMetrics [1] is optimized for monitoring and observability of IT infrastructure, app-level metrics, industrial telemetry and other IoT cases.

As you can see, there is little sense in comparing Kafka with time series databases :)

[1] https://victoriametrics.com/

[hansvm]

Last time I checked out VictoriaMetrics it had something akin to a memory leak when writing evenly to large numbers of distinct keys -- attempting to cache all the data in RAM and not freeing it even under high memory pressure (it's been awhile, but iirc it had a hard-coded 1hr expiry). Does that sound like a behavior that would still exist in the current design?

[AndrewKemendo]

You seem to be making an assumption that I'm not going to (or shouldn't) use Kafka as a database for my time series data - when in fact that is how we use it.

I store and query messages in certain Kafka Topics - it's not all our data, but it is a non-trivial amount of it.

[rubiquity]

In that sense, InfluxDB will start to have a decent amount of overlap with something like Snowflake or Redshift.

[pauldix]

Yes, this will likely be the case, but it's not a specific goal. We're focusing our efforts right now on the InfluxDB time series use cases we see most. More general data warehousing work isn't our focus, but we expect to pick up quite a bit of that along the way as we develop and as the underlying Arrow tools develop.

[mhall119]

Snowflake was actually specifically mentioned as an example of who else is doing this in the blog post

[valyala]

It looks like InfluxDB is going to target general-purpose analytical workloads. It would be interesting to look at how InfluxDB will compete with ClickHouse in this space.

[kremi]

What are the advantages of Influx-iOX compared to Apache Druid? There seems to be much overlap in feature set and target use cases: seperate storage from compute, using parquet for columnar storage, good at analytical workloads on timeseries.

[valyala]

It looks like Influx-IOx and Apache Druid have many common base building blocks. The main difference is that Influx-IOx is optimized for time series workloads, while Apache Druid is optimized for analytical workloads. While time series workloads can be treated as analytical workloads in most cases, time series databases usually provide specialized query languages such as Flux, InfluxQL, PromQL [1] or MetricsQL [2] that simplify typical queries over time series data.

P.S. Apache Druid should be compared to ClickHouse [3] or similar analytical databases.

[1] https://valyala.medium.com/promql-tutorial-for-beginners-9ab...

[2] https://victoriametrics.github.io/MetricsQL.html

[3] https://clickhouse.tech/

[harikb]

Would this imply that you will be targeting in-memory analytics only?

Offtopic: I have been following your company and team closely during many Gophercons. Good luck to your team ... although a bit sad to see you go the Rust route :P

[pauldix]

For now the execution is in-memory only. Over time we want to be able to execute against Parquet files on disk.

However, for large scale analytics on huge data sets where you're scanning all of it, we'll likely push you to EMR or something like that. The nice part is that those big data systems can execute directly against the Parquet files in object storage that form the basis of our durability.

This is part of the bet we're making compatibility with a larger data processing ecosystem.

Almost all of our use cases fall within what you can compute in memory on a single system (which is pretty much up to 1TB).

[silvester23]

Do you have any plans for supporting versioned time series? As in "show me the time series as it was on 2020-11-11-08T:00:00:00Z"? Most of our time series are forecasts that change over time and that would be super helpful.

Anyway, thanks for your work!

[e-dard]

Hi, one of the creators of IOx here. Unlike InfluxDB where you can only do efficient range based queries on one time value, in IOx you will be able to apply predicates efficiently to any column. That means you can have several columns for different times (e.g., time created, forecast time, expire time..).

You will be able to do efficient range queries on any of these. Most use-cases will work best when you partition your data into time chunks, which commonly would be the time you inserted the samples into the database (created time in your case I guess).

[cammil]

I don't think the time that the forecast is for should be the main index. Rather the time that the forecast is created should be the main index. In general I think the time index is about what happened at a particular point in time.

[silvester23]

Interesting idea. How would you model the forecast time in this case? I'm not an expert but to my knowledge, you can only have one time index in InfluxDB, right?

[cammil]

I'm not sure about InfluxDB at all, sorry if I gave that impression. I have however designed and built data architecture for various forecasting software. "all data you perform analytics on is time series data" (from the article) rings so true with me.

However you need to consider that in this sense, it's about what is happening at any given moment in time. In your case, the forecasts are created at a point in time, and that "should" be their index speaking purely from a time-is-the-ultimate-index perspective.

Just some random questions that might help you answer your own questions: 1. Are there multiple forecasts made at the same time, for the same period in the future, but by different systems / algorithms / hyperparameters? Would you not want to keep multiple, and in that regard, what would be the "latest" forecast? 2. If your latest forecast is necessarily the most accurate, why would you need to keep previous versions in the same database?

None of this might be relevant to you, and maybe you'll only get the benefits of InfluxDB by using the forecast time as the index. But I thought I'd give you my thoughts just in case it helps you.

[jacques_chester]

I think you're on the edge of describing bitemporal databases. You have a range which represents "time over which this fact is asserted to be true" and a second range for "time during which the assertion was recorded as valid". These typically get called "valid time" and "transaction time".

You can express forecasts this way. The valid time range is the range for which you assert a given forecast is going to be true. The transaction time range is the time during which you held that forecast to be the current forecast.

Using transaction time you can then reconstruct your state of belief from any point in time. Using valid time you can make assertions about any fact in ranges over the past, present or future.

I think Snodgrass's Developing Time-Oriented Database Applications in SQL[0] is still the best overall introduction, though the SQL used is fairly dated (ha). The relevant Wikipedia entry is OK too[1].

[0] https://www2.cs.arizona.edu/~rts/tdbbook.pdf

[1] https://en.wikipedia.org/wiki/Temporal_database

[kungito]

Can't you just have another time key?

[silvester23]

Can I? That would be great. I thought InfluxDB did not support that but I might be wrong, I'm not really an expert.

[Yoric]

What's it best suited for?

[pauldix]

Right now it's just a project in development, so nothing yet. But it's supposed to be for time series data. This could be metrics, events, or any kind of semi-structured data that fits into tables where you want to ask questions about it over time.

It also defines rules for replication (push and pull), subscriptions to subsets of the data, and processing data as it arrives via a scripting engine.

Some of these features will arrive before others. Right now we want to make it work well for the data InfluxDB is currently good at (metrics, sensor data) and also work well for high cardinality data like events and tracing data.

[jstrong]

do you expect any of your work on this to end up as rust libraries that can be used on their own?

[pauldix]

Yes, absolutely. There are the Arrow libraries in Rust that we're already contributing to. DataFusion, for example is an SQL execution engine.

We'll be publishing crates for parsing InfluxDB Line Protocol, Reading InfluxDB TSM files (for conversion into Parquet and other formats), and client libraries as well.

The entire project itself will also be published as a crate. So you can use any part of the server in your own project.

[jstrong]

cool. interested to see the TSM library.

if it's any use to you, here's an unpublished rust influx client crate I've been using for a few years. the main point of interest is an ergonomic macro to construct measurements, `measure!`. https://github.com/jonathanstrong/influx-writer