[sporksmith]

Shadow developer here! Disclaimer - I joined the project a little over a year ago. I'm not as well versed in some of the related work as my colleagues, but I can take a shot -

Mininet looks cool!

It looks like Mininet lets programs run pretty close to natively, using network namespaces to pipe data over its simulated network, and some other lightweight techniques for isolation and some extent of reproducibility (e.g. using the fifo scheduler).

Shadow intercepts the libc API via LD_PRELOAD, allowing finer grained control over the process's view of the world. In particular it lets us emulate time - we can run simulations with tens of thousands of endpoints on a single physical machine; even if we don't have the CPU power to run it in real-time, the simulation behaves as if we do. Conversely, small simulations can be run faster than real-time.

The down-side is that it's more difficult to support arbitrary software, since there are various ways of "escaping" LD_PRELOAD, and we don't support the full libc API. We're in the process of switching to a ptrace-based approach that is more robust, at the cost of some additional runtime overhead.

> Not sure exactly the extent of networking you can simulate with this toolset, but mininet lets you put entire simulated autonomous systems on a single laptop and you can mess around with BGP policies.

It sounds like mininet has a lower-level model of the network itself. Shadow simulates links between endpoints, including bandwidth limitations and drop rates. It doesn't model individual routers along the path, so yeah I don't think you could use it to simulate BGP.

[exikyut]

A question: I tried poking around with `tc` (qdisc) a while back to limit speed and emulate latency, and I found the implementation seemed to simulate delay by waiting a bunch of time then dumping all the data at once, for example if I asked for 5s of latency I'd be sitting at 0 bytes/sec for 5 seconds then it'd be like INCOMING and I'd get a giant pile of data all at once.

I do absolutely have horrendously little networking experience :) so I was probably fiddling with the knobs wrong (and fiddling with wrong expectations from the mechanics), but I didn't see any tunables for ramp-up et al.

For a while I've been idly curious how to iterate on applications that need to respond favorably to everything from 10Gbps and "what latency?" to flaky 3.5G with "give up" latency and worse jitter. At the end of the day I know I need real networks to play with, but I've been curious what software will get me the closest approximation in terms of microsecond-to-microsecond network behavior.

TBH this question is me being somewhat lazy :) - I'm not yet ready to start working on the above ideas, so I'm yet to actually sit down and do some basic prerequisite research... but I thought I'd jump on the opportunity to ask while I could.

[sporksmith]

Sorry, I'm not well versed in the network model; most of my work has been on the new ptrace-based syscall interposition. That sure sounds like a bug, though.

We recently added a discussion section to Shadow's GH repo: https://github.com/shadow/shadow/discussions. That'd be a good place for your questions. More knowledgeable folks are pretty responsive there, but aren't on HN :)

[exikyut]

Apologies for the ambiguity and lack of context - to clarify, I fiddled around with raw tc/qdisc (part of Linux's network stack) a while back, and was wondering if Shadow had more accurate timing simulation.

The discussion section sounds like exactly the place to follow up on this sort of thing, thanks for the reference. (Once I verify what I'm looking for is actually exotic :P)

Thanks for replying!

[littlestymaar]

> We're in the process of switching to a ptrace-based approach that is more robust, at the cost of some additional runtime overhead.

Wouldn't eBPF be a better alternative?

[sporksmith]

We've started looking into eBPF a bit - IIUC eBPF by itself doesn't give us the ability to service or arbitrarily manipulate the traced process's syscalls.

We have recently learned of an interesting technique that dettrace [1] uses of combining seccomp with an eBPF filter and ptrace. Instead of generating a ptrace-stop for every syscall (as we do now, using PTRACE_SYSEMU), they use a seccomp policy with an eBPF filter, s.t. a ptrace-stop is only generated for syscalls that violate the policy, allowing them to emulate the result of those syscalls. syscalls that don't violate the policy are allowed to execute natively, saving a lot of overhead.

[1]: https://github.com/dettrace/dettrace

This works great for them since they want to emulate a relatively small subset of syscalls. In our case we want to emulate most syscalls, so it's not as clear-cut of a win. We have found though that if we use an LD_PRELOAD'd shim in the target process to intercept syscalls and then service them via IPC, that's substantially faster than catching them with ptrace. That runs back into the problems with LD_PRELOAD in general of there being various ways of missing syscalls. but, we may be able to use that technique along with ptrace+seccomp+ebpf to intercept any syscalls that we'd otherwise miss. The seccomp technique would allow us to exempt the syscalls that our shim itself is making to do the IPC.

[littlestymaar]

Thanks for the detailed answer!