Unfortunately you will not get any speed boost in that case, in fact it will be slower having to fetch all the remote side data for small file changes.
Fair enough. It does work for my simple use case: rsync student coursework to the remarkable, mark coursework on the remarkable, rsync back. There aren't many small file changes and USB is fast enough. The real bottleneck is the marking but I'm not sure if I want a package manager to help me with that!