[austin-cheney]

I am going to write an original in-memory database in JavaScript. I hate SQL and believe I can write something that executes faster than existing solutions while also feeling natural to JavaScript: storage and search via objects and arrays.

[Horos]

Interesting project. A few questions that came to mind: How do you handle GC pressure at scale? V8's hidden classes make homogeneous object arrays fast, but the per-object overhead adds up — 100K entries is already 6-8 MB of metadata alone, and major GC pauses become unpredictable. What's the persistence story? The moment you serialize to IndexedDB or OPFS, the "native structures" advantage disappears. Have you looked at columnar formats to keep it fast? How do you handle compound queries without a planner? Something like "age > 30 AND city = 'Paris' ORDER BY name" needs index selection strategy, otherwise you're back to full scans. The part I find most compelling is reactive queries — define a filter, then as objects land in the store (from DOM extraction, a WebSocket, whatever), results update incrementally via Proxy interception. No re-scan. That's not really a database, it's a live dataflow layer. Concrete example: a browser extension that extracts product data from whatever page you're on. Each page dumps heterogeneous objects into the store. A reactive query like "items where price < 50 and source contains 'amazon'" updates in real time as you browse. No server, no SQL, just JS objects flowing through live filters. That would be genuinely useful and hard to do well with existing tools.

[austin-cheney]

I have not gotten far enough for that kind of load testing. I am working on this thing, but its still incomplete. My experience with GC related issues is that frequency of calls is more of a concern than size of the calls. So, I would have to monitor for memory spikes, which I can do from Node but not so much from the browser.

[Horos]

Quick question before going further: is this an exercise in language internals, or do you have a concrete use case in mind?

Asking because the answer changes the architecture significantly. If you're targeting live in-page data — extracting objects from the DOM as you browse, filtering them reactively — you may not need storage at all.

A Proxy-based observation layer gives you reactive queries without allocating anything new: the objects already exist in the tab's heap, you're just watching them mutate. No GC pressure, no persistence headaches, no query planner needed. That covers most of what you described: "items where price < 50 updates as you browse" is an event subscription with pattern matching, not a database problem.

The cases where you actually need storage — and therefore need to think about heap budgets, GC, serialization, query planning — are narrower:

Cross-session persistence (you want the data after the tab closes) Cross-tab aggregation (comparing prices across multiple open tabs simultaneously) Queries over historical data (not just what's on screen now, but what you saw across 20 pages of browsing)

Those are real storage problems.

But they're also the cases where you're competing with IndexedDB, OPFS, and SQLite WASM — and "I hate SQL" stops being enough of a reason to rebuild from scratch. What's the actual workflow you're trying to support?

[austin-cheney]

It is mostly experimental, but there is a very tiny valid use case.

I have a strictly personal application at: https://github.com/prettydiff/aphorio

In that project I have various data artifacts stored in memory that I am constantly having to query in various ways:

* sockets per server

* servers and their sockets

* ports in use and by what

* docker containers and their active state

* various hardware and OS data lists

Currently all this data is just objects attached to a big object and all defined by TypeScript interfaces. I am storing the information just fine, but getting the information I need for a particular task and in task's format requires a variety of different logic and object definitions in the form of internal services.

Instead it would be nice to have information stores that contain all I need in the way a SQL database does with tables. Except I hate the SQL language, and its not as fast as you would think. Last I saw the fastest SQL based database is SQLite and its really only 3x faster than using primitive read/write streams to the file system. I can do must faster by not dicking around with language constructs.

My proposal is to store the database in something that vaguely resembles a database table but is really just JavaScript objects/arrays in memory as part of the application's current runtime and that can return artifacts in either object or array format. Query statements would be a format of JavaScript objects. I could have a table for server data, socket data, port data, and each record links back to records in other tables as necessary, kind of like SQL foreign keys. So in short: stores, functions to do all the work, and storage format that can be either objects or arrays and take both objects and arrays as queries.

The reason I want to store the data as both objects and arrays is a performance hack discovered by Paul Heckel in 1978. The stores would actually be a collection of objects against a unique primary key that can be reference as though it were an array.

[Horos]

Three real risks in your current approach before anything else. Shared references mutate silently — in JS, objects passed between your "tables" are aliases, not copies, so a mutation in one place propagates everywhere with no transaction and no rollback. No atomicity — Node is single-threaded but async I/O means two callbacks can interleave writes on the same structure with no guarantee a multi-step update lands cleanly. And everything disappears on crash — for socket/port/container state that's probably fine since it's observable from the system anyway, but you have no history.

That said, you may not need to leave your stack at all. V8's native Map is already a key-value store — O(1) reads, no overhead, typed in TypeScript. Your "tables" are just Maps and cross-referencing is composite string keys:

sockets.set(serverId:{serverId}: serverId:{socketId} , socketData). No library, no dependency, no SQL. This covers your use case as described.

If you want ACID transactions and persistence without SQL, look at lmdb-js — a Node binding on LMDB, the fastest embedded KV store in existence, zero-copy reads, used in production for 20 years. Your tables become named databases, your records are typed values, your cross-references are composite keys. Same mental model you're building, with 20 years of correctness guarantees underneath.

What's the actual reason for building from scratch rather than using native Map for the in-memory case?