[techdragon]

Might seem like odd questions, but is there a plan to handle the "too much data" case? and how protocol agnostic can this be? This seems really cool, and I can think of some strait forward uses for it ... but working on a satellite project where communications are 75% of the hassle, this sort of thing seems like it could be very useful...

If it can deal with all the "weird" (its very normal for space, just not used anywhere else) batched radio communication standards from the Consultative Committee for Space Data Systems (CCSDS) then intermittent communications aren't as big of a problem.

However it will also need to handle "old data that can be deleted client side". A satellite will generate a lot of data that it doesn't need to keep but will want to transfer to the ground and keep on the ground, telemetry and tracking metrics, system statuses etc. One way is to keep all this buffered on the satellite, and transmit to the ground, then you can throw it away, etc.. But having a synchronised store is a much simpler programming model, you push important data into the store and let it look after itself, and simple code is easier to make more reliable and reliability is key for anything in aerospace. However it can't result in the remote device being "full", it's a 100% non-starter. The client/remote end has to be able to throw away old/expired data while the central store is able to keep that data, without requiring awkward secondary systems polling the latest state on the ground side to save current data into a separate system to store historical data, that would just be shifting the complexity around.

[bigfish24]

Adam (cofounder) - great question! Yes, this situation is very common for our use-cases. Ditto supports the concept of `eviction` [1]. For example, imagine a restaurant that wants to store all of today's orders in their devices, but then evict the data when they close out at end of day (and data is synced into Cloud backend). Eviction is a core concept because we consider devices using Ditto's SDK a "Small peer" that should only have the data it wants - queries define which data is synced and you can always evict anything unneeded. Conversely, the server version of Ditto is a "Big Peer" that is a cluster designed for scalability and fault tolerance, so it wants to capture everything. Another way to put it is selfish vs. greedy replication.

To help users with eviction, we are adding support for dynamic queries, for example, using "now" for a date in the query to aid in creating TTL logic. In addition, to built-in TTL as well.

Satellite comms are definitely a great fit for another reason, since Ditto's replication is "delta-based" it will only sync differences. Our protocol is designed to overlay on top of custom arbitrary links, including unreliable ones. Not familiar with CCSDS, but we have explored Link16, and even built-in we replicate over BLE GATT.

Hope that helps and perhaps we can get Ditto into space!

[1] https://docs.ditto.live/ios/common/concepts/syncing-data#evi...

[techdragon]

Thanks for the reply! I had a very quick look at the documentation and didn’t see the eviction information. That’s excellent to know it’s already a built in capability. As for space, don’t worry if you’ve never heard of CCSDS, it’s just yet another standards group for various things, the key thing is that it’s delta based and already designed to handle non-TCP-IP based data transfer mechanisms. One of the big challenges I’ve found with bringing Linux and other modern software into an environment where “normal” (you can get a 24/7 IP link, but it’s unusual for satellite to maintain an open link like that) space communication is that a lot of useful modern open source tools is just not designed for a non IP based networking environment, so much so that you dig into things like delay tolerant networks and you’ll find mechanisms for trying to fake TCP-IP over the top of them just to help accomodate this software. Having a mechanism to replicate important data on the basis of just a transmission mechanism agnostic “packet” of delta update data, would be a huge boon for low cost satellite systems.

Im definitely going to have to dig in deeper and check out the rust stack! I’ll be sure to get in touch about the various CCSDS data transmission related stuff once I get there, since if you’ve looked at Link16 I’m sure you’re probably interested in adding more aerospace communications link standards/mechanisms.