[Horos]

I've set a fully async patern. blobs chunked into sqlite shards.

It's a blind fire n forget go worker danse.

wich can be hold as monitoreed or scale as multiple instances if needed by simple parameters.

Basicaly, It's a job as librairy patern.

If you dont need real time, its bulletproof and very llm friendly.

and a good token saver by the batching abilities.

[leandot]

Curious about more details about this setup?

[Horos]

The "job as library" pattern is simple: instead of wiring jobs into main or a framework, you split into 3 things.

Your queue is a struct with New(db) — it knows submit, poll, complete, fail, nothing else.

Your worker is another struct that loops on the queue and dispatches to handlers registered via RegisterHandler("type", fn). Your handlers are pure functions (ctx,payload) → (result, error) carried by a dependency struct.

Main just assembles: open DB, create queue, create worker, register handlers, call worker.Start(ctx). Result: each handler is unit-testable without the worker or network, the worker is reusable across any pipeline, and lifecycle is controlled by a simple context.Cancel().

Bonus: here the queue is a SQLite table with atomic poll (BEGIN IMMEDIATE), zero external infra.

The whole "framework" is 500 lines of readable Go, not an opaque DSL. TL;DR: every service is a library with New() + Start(ctx), the binary is just an assembler.

The "all in connectivity" pattern means every capability in your system — embeddings, document extraction, replication, MCP tools — is called through one interface: router.Call(ctx,"service", payload).

The router looks up a SQLite routes table to decide how to fulfill that call: in-memory function (local), HTTP POST (http), QUIC stream (quic), MCP tool (mcp), vector embedding (embed), DB replication (dbsync), or silent no-op (noop).

You code everything as local function calls — monolith. When you need to split a service out, you UPDATE one row in the routes table, the watcher picks it up via PRAGMA data_version, and the next call goes remote.

Zero code change, zero restart. Built-in circuit breaker, retry with backoff, fallback-to-local on remote failure, SSRF guard.

The caller never knows where the work happens.

That's the "job as library" pattern: the boundary between monolith and microservices is a config row, not an architecture decision.

https://github.com/hazyhaar/pkg/tree/main/connectivity

[Horos]

had a look ?