File transfers: fix performance regression due to syncing

[vjr]

Fixes the slowdown introduced by #2828 to fix #2818 while preserving the intent of keeping users informed that their file transfers are in progress so they don't inadvertently unplug their external/removable storage.

The progress dialog now shows "finishing copy" or "finishing move" briefly at the end while the transfer is being synced.

Tested copy/move between fast/slow internal/external storage - fast nvme ssds and slow usb sticks.

[jeremypw]

@vjr Performance aside, I think main feels more natural with slow devices. With this PR, moving one large file onto a slow drive immediately shows progress of 100% (transfer to cache) but the progress window stays open for a long time showing "Finishing ..." but giving no indication of how long it will take. In main you get a better idea of the true rate of progress. For slow drives syncing often makes little performance difference, whereas for SSDs it makes a more significant difference. Not sure how to solve this in a simple manner - its probably not worth spending too much time on tbh. I'll maybe revisit after porting copy/move to Vala.

[jeremypw]

@vjr I notice your first solution was to limit syncing on a time basis which seems reasonable (together with syncing after transferring the last file). What was the reason you tried a different solution?

[vjr]

The previous commit throttling in between file transfers had essentially the same slowdown effect.

@jeremypw that throttling commit (i intentionally pushed it to save the code if needed later) didn't improve speeds, so i went with the sync-only-at-end commit. the "finishing copy..." progress dialog is caused by kernel cache for smaller file sizes (i think it may be 2gb IINM) but yes it's boring to look at even though it still lets a user know transfer is still in progress so they don't unplug.

i can try the following approaches to see which ones (if any) work to fix the transfer speeds while avoiding showing "finishing copy..." for removables:

• use a const for some time duration and/or data size like the existing "time required for accurate transfer rate" const.
• check the drive.is_removable flag and only then do the throttle or somehow disable kernel buffer caching.

if nothing works out, yes, we can revisit at a later time in another PR.

[jeremypw]

I would be in favour of only syncing removeable drives tbh. The performance hit is then less important. Disabling caching for those drives would be easier than syncing if it can be done programmatically.

[jeremypw]

Looks like disabling caching can only be done with a udev rule or in fstab. Unfortunately GLib.MountMountFlags doesn't include a sync option (or any other options!).

[vjr]

ok thanks i'll look at using the is_removable check.

[vjr]

@jeremypw see if the latest changes work well enough in your assessment?

The last commit 017efba is to avoid syncs for faster removable storage, like the 256GB SATA SSD I have connected via a USB adapter which does about 350 MB/s reads and 200 MB/s writes with rated read/write for this interface up to 500 (or even 600) MB/s.

This particular commit is to not lose such faster devices' speed advantage for users. There are even faster external storage like one NVMe-to-USB-type-C adapter I have that can do 1000 MB/s read/write so we don't sync here.

This PR is now a trade-off (some threshold to pick, currently 300 MiB/s) which should work in most cases but there may be exceptions where it effectively reverses the sync decision such as for very fast removable storage like an NVMe enclosure or very slow internal storage like older HDDs.

This is only for removable storage, all internal storage do not sync at all, fixing the transfer speed regression.

[jeremypw]

Apart from the nitpicking comments this seems to cause no regressions and should improve performance for internal devices.

[jeremypw]

There is still a performance hit on slow devices when transferring large numbers of very small files but this is a corner case that can be addressed separately. The transfer rate is about 10% of that if a single large file.

There is no need to sync at the end of each file if they are only 100 bytes, but should probably sync every X% of the total number of files (or X% of the total number of bytes) so as to keep the progress realistic.