[toddh]

Interesting, i didn't see how security works? Is there backpressure on message senders? Any ordering guarantees? Are messages queued so activated objects can reconstruct state? Can passivation warmth be controlled? Can objects support multiple threads? Can objects move? Failover?

[kentonv]

Great questions.

> how security works?

Messages can only be sent to Durable Objects from other Workers. To send a message, you must configure the sending Worker with a "Durable Object Namespace Binding". Currently, we only permit workers on the same account to bind to a namespace. Without the binding, there's no way to talk to Durable Objects in that namespace.

> Is there backpressure on message senders?

Currently, the only message type is HTTP (including WebSocket). There is indeed backpressure on the HTTP request/response bodies and WebSocket streams.

In fact, this is exactly why we added streaming flow control to Cap'n Proto: https://capnproto.org/news/2020-04-23-capnproto-0.8.html

We plan to support other formats for messaging in the future.

> Any ordering guarantees?

Since each object is single-threaded, any block of code that doesn't contain an `await` statement is guaranteed to execute atomically. Any put()s to durable storage will be ordered according to when put() was invoked (even though it's an async method that you have to `await`.)

When sending messages to a Durable Object, two messages sent with the same stub will be delivered in order, i.e.:

    let stub = OBJECT_NAMESPACE.get(id);
    let promise1 = stub.fetch(request1);
    let promise2 = stub.fetch(request2);
    await promise1;
    await promise2;

If you have heard of a concept called "E-order" (from capability-based security and the E programming language designed by Mark Miller), we try to follow that wherever possible.

> Are messages queued so activated objects can reconstruct state?

No. The only state that is durable is what you explicitly store using the storage interface that is passed to the object's constructor. We don't attempt to reconstruct live object state. We thought about it, but there's a lot of tricky problems with that... maybe someday.

If the machine hosting an object randomly dies mid-request, the client will get an exception thrown from `stub.fetch()` and will have to retry (with a new stub; the existing stub is permanently disconnected per e-order). In capability-based terms, this is CapTP-style, not Ken-style.

> Can passivation warmth be controlled?

Sorry, I don't know what that means.

> Can objects support multiple threads?

No, each object is intentionally single-threaded. It's up to the app to replicate objects if needed, though we might add built-in features to simplify this in the future.

> Can objects move?

This is a big part of the plan -- objects will transparently migrate between datacenters to be close to whatever is talking to them. It's not fully implemented yet, but the pieces are there, we just need to write some more code. This will be done before coming out of beta.

> Failover?

If a machine goes down, we automatically move the object to a different machine. If a colo goes down, we will automatically move to another colo. We still have a little bit of missing code for colo failover -- the data is replicated already, but we haven't fully implemented the live failover just yet. Again, that'll happen before we exit beta.

[ec109685]

Around the ordering guarantee, are there tricky edge cases if objects are moving between machines in the face of network partitions or some other system instability? Where request1 and request2 hit two different instances of the object, but request1 persists at the end instead of request2?

[kentonv]

Those are tricky edge cases for us to solve, yes, but that's our job, not yours. :)

[ec109685]

:) More of a question of what tradeoffs you're using to solve them.

[toddh]

> Can passivation warmth be controlled?

A variant of the cold start problem. How long after all the messages for an object drain is it passivated? Can you keep it pinned in memory?

Another question. Can objects contain relationships that are themselves references to other Durable Objects? Say a simple reference, or a list, or DAGs?

[kentonv]

The system will evict the live object when it has been idle for some period. If there are connections still open to the object, it won't be evicted (unless it exceeds resource limits, etc.).

As always, "cold starts" with Workers are very fast, usually imperceptible to a human. Also, multiple objects may be hosted in the same isolate; when instantiating in an existing isolate, the only "cold start" overhead is whatever you write in your class constructor.

> Can objects contain relationships that are themselves references to other Durable Objects?

Yes, by storing the object IDs.

[toddh]

Are they garbage collected? Do you use tombstones?

[kentonv]

No, no GC, at least at present. If you store any durable state, you have to delete it explicitly, or it stays forever.

I am interested in the idea of objects whose IDs are never revealed to the app, but to which references can be stored inside other objects. Then we could do GC... and it would be a true capability system.