[kettleballroll]

> in this case it takes 4-8 threads or so for KySync to match Zsync's performance.

Assuming zsynch is single threaded, in an apples/apples comparison (1 thread) zsynch wins by over 10x. I can't help but wonder how much faster kysynch would be if they focused more on single threaded performance. The way things stand, kysynch's implementation sounds rather wasteful I'm terms of CPU resources, in spite of what this announcement claims. Still, kudos on this impressive project!

[kbenson]

Given that it runs client side pairing with plain rsync servers, meant to efficiently support mass file distribution, using multiple cores but less efficiently to gain in overall performance seems a good tradeoff.

One thing most client/home systems have in abundance most the time is spare cores.

[tialaramex]

Dozens of them?

The huge speed-ups for KySync are when it has 16+ threads on a 96 core system. But if my computer has two cores, it is not in fact going to efficiently run 16 CPU banging threads, and actually it might choke badly and I should rather run the dual thread KySync, which is much less impressive.

Some problems in this area (data synchronization) look embarrassingly parallel, which means that sure, if you can run more at once you win, but, there's no interesting problem, you could also run four copies of Zsync, "embarrassingly parallel" means the concurrency problem is trivial, in this case each of those four maybe handles a separate file or region of the file, it's embarrassing because all the clever techniques you learned in class aren't important, the problem is so easy.

I can see it wasn't the point of this work, but from an inquisitiveness point of view my question even after just one chart is, "What happens if we run this on a much more modest VM?". Do the extra threads trample on each other? Are they automatically suppressed?

[kbenson]

> The huge speed-ups for KySync are when it has 16+ threads on a 96 core system.

Only if you cherry pick the results and look only at 128MB files. At 512MB files, there is parity at 4 threads. At 2GB files there is almost parity at 2 threads, and significant gains at 4 threads.

> But if my computer has two cores

Then your CPU wasn't sold to you in recent years, as entry level desktop CPUs ship with 4 cores now and have for a couple years.

That said, if you're actually in this situation, then you just run zsync if it's faster. The existence of kysync isn't going to make it disappear or prevent you from using it, any more than zsync prevented rsync from being used.

I'm not sure the point you're trying to make. A new implementation of an existing protocol came out that allows for speed gains if certain criteria are met, which includes throwing more compute at it. It feels like you think this is a bad thing for some reason?

[kyotov]

check out https://kyall.notion.site/kyall/KySync-v1-1-dd9931f330f24146...

[kettleballroll]

Awesome improvements, congrats!

[stingraycharles]

I wonder whether they have an explanation for this. Probably more overhead in the abstractions necessary for parallel processing?

[boris]

I can't say anything specifically about this case, but generally, sometimes the whole architecture has to be changed in order to support concurrency (especially in the "millions of mutexes" kinds of problems). And such alternative architectures are often inherently inefficient in serial execution.

[kyotov]

Instead of an explanation, we decided to address some of the reasons :)

https://kyall.notion.site/KySync-v1-1-dd9931f330f241469d3e60...

Enjoy!