Hacker News new | ask | show | jobs
by dmpetrov 589 days ago
Lance is just a data format. Lance DB might be more comparable to DataChain.

DataChain focuses on data transformation and versioning, whereas LanceDB appears to be more about retrieving and serving data. Both designed for multimodal use cases.

From technical side: Lance has it's own data format and DB engine while DataChain utilizes existing DB engines (SQLite in open-source and ClickHouse/BigQuery in SaaS).

In SaaS, DataChain has analytics features including data lineage tracking and visualization for PDFs, videos, and annotated images (e.g., bounding boxes, poses). I'm curious to understand the unique value of LanceDB's SaaS — insight would be helpful!

You could think of it as OLTP (Lance) versus OLAP (DataChain) for multimodal data, though this analogy may not be perfect.
1 comments
m0sth8 589 days ago
How about daft https://github.com/Eventual-Inc/Daft - also looks like a new multimodal dataframe framework
link
jaychia 588 days ago
One of the maintainers of Daft here.

Just dug through the datachain codebase to understand a little more. I think while both projects have a Dataframe interface, they're very different projects!

Datachain seems to operate more on the orchestration layer, running Python libraries such as PIL and requests (for making API calls) and relying on an external database engine (SQLite or BigQuery/Clickhouse) for the actual compute.

Daft is an actual data engine. Essentially, it's "multimodal BigQuery/Clickhouse". We've built out a lot of our own data system functionality such as custom Rust-defined multimodal data structures, kernels to work on multimodal types, a query optimizer, distributed joins etc.

In non-technical terms, I think this means that Datachain really is more of a "DBT" which orchestrates compute over an existing engine, whereas Daft is the actual compute/data engine that runs the workload. A project such as Datachain could actually run on top of Daft, which can handle the compute and I/O operations necessary to execute the requested workload.

link
dmpetrov 589 days ago
Good question! I’m not so familiar with it.

It looks like Daft is closer to Lance with it’s own data format and engine. But I’d appreciate more insights from users or the creators.

link