It is not a bad idea to be able to switch source and destination, but that is usually design for sync tools (ex. rclone).
Backup usually designed to archive local data somewhere else. And NAS is one exception, but it usually exposes itself as shared drive, hence local source works. But from performance and open files reasons, I’d run backup command locally on NAS itself.
Indeed one way to do backups is to archive data from local to remote. Yet, that would require backup software such as duplicati to run from all devices, or use a different backup system to pull the data out of the devices first. Whereas if remote to local backups are also supported, that would:
Allow backups to be made from locations that do not allow interactive shell.
Allow backups from low resource devices that do offer SFTP but not much else.
Relieve the need for installing and maintaining software on all devices.
PS We are talking about backup, not synchronization indeed.
Most backup software (and I am talking about traditional ones - Veritas/Symantec, Bacula, etc.) use local agents and central server design for a reason - it is possible to control endpoint state, preventing machine from sleep, verify file hash, copy open files…
Without local agent you loose this flexibility. You can’t (in most cases) copy open files, machine can sleep, and to check hash to verify if file have changed, you’ll need to send entire file over the wire.
Those are the reasons why you do not see this architecture often.
If you are talking Unix/Linux - you can always mount remote SSH as a folder (SSHFS) and use any backup tool as it it is local (all issues still apply). If your remote is Windows - just install SSH server - there are couple of free ones.
And keep in mind - SFTP performance is not up to SMB or even HTTP. in many cases.
That sounds like with DT you can’t have two computers backing up to the same account at the same storage provider. Is that correct? Or does it just mean that DT can’t have concurrent backups to the same folder in that account?
From reading the source code for Duplicacy, it appears that they actually build a table of all known blocks and keeps this in memory. We had that option some time ago, but it was a real memory hog, so I removed it.
I have just tested with a small backup (~2000 blocks), and a block lookup cache had a tiny positive effect on the backup speed. But it is possible that this speedup is much more pronounced with larger backups, so I made a canary build with the option --use-block-cache. If it turns out that it really does improve performance without ridiculous memory overhead, I will convert it to a --disable-block-cache option.
If @dgcom and @jl_678 have a test setup, I would like to know the performance difference from the --use-block-cache option. I am aware of another performance issue related to how a file’s previous metadata (size, timestamp, attributes) is fetched, so it is possible that this is what really makes the difference, but maybe the --use-block-cache can help shed light on that.
If that is how it works, it would require that the destination supports some kind of “If-Modified-Since” approach.
Without actually running Duplicacy, it does appear that it lists all remote blocks first, and keeps them in memory:
I think it works by having a copy (maybe without contents) of the remote store, in the local filesystem:
Some of the complexity in Duplicati is there to handle big backups. If you have a file of 1TB, you need 300mb of raw hash data (with 100kb blocks). To avoid a blowup in memory and large file-lists, Duplicati keeps it in database and uses “blocks og blocks”.
But if you have a smaller backup, this should not perform worse.
I tested Duplicati 184.108.40.206 Canary and added test results to the spreadsheet.
This time I used locally attached USB 2.0 drive and used single up/down thread for CY compare.
And CY still noticeably faster, even with single thread.
I do not see much of a change between --use-block-cache="true" and not setting this option.
On backup, time is actually spent reading source files and there is no CPU bottleneck.
I’ll see if I can dig more into why TI is much slower in comparable configuration.
Some more testing did not reveal any real help from --use-block-cache="true"
The only way I was able to speed up backup was to enable --synchronous-upload="true"
Restore with --no-local-db="true" is a bit faster compared to two separate operations.
I ditched VSS since it may take variable time and also measured compression and encryption impact on the backup.
Here are some of the results:
00:08:49.995 --synchronous-upload="true" --no-backend-verification="true" - COMMON for below
00:07:23.040 --no-encryption="true" --use-block-cache="true"
00:06:08.722 --zip-compression-level=1 --use-block-cache="true"
00:05:04.746 --zip-compression-level=1 --no-encryption="true" --use-block-cache="true"
00:01:52.741 --no-local-db="true" --no-local-blocks="true --skip-restore-verification="true" - COMMON for below
00:00:18.345 DB repair
00:01:32.149 --dbpath=<repaireddb> --use-block-cache="true" --no-local-blocks="true --skip-restore-verification="true"
These results are still noticeably worse than CY
As suggested before, I’ll try backup of very large files instead of many small ones.
Thanks @dgcom for trying that out, much appreciated.
I am a bit surprised that TI is that much slower, and I guessed at maybe the in-memory lookup table was reason, but it should not matter a lot, since most lookups will fail (the block is new), and the database is using a log(n) lookup time anyway. Your results show that the database is indeed fast enough (at least on an SSD).
Compared to CY there are not many differences, so I think TI should be able to reach similar speeds.
CY stores all blocks “as-is” on the remote store (in some cases using folders to reduce the number of files in a single folder).
TI stores files inside archives to reduce the number of remote files and requests.
CY keeps a cache of the remote data locally on-disk.
TI keeps a cache/lookup of the remote data in a database.
CY uses a flexible block width (content defined chunking), TI uses a fixed block width and a block-of-blocks.
They both use SHA256 as the hashing algorithm.
I see a big jump when you lower the compression level, so maybe the bottleneck is really the zip compression.
The speedup from --synchronous-upload is a bit strange, as it just “pauses” the backup while there is an upload happening.
Yes, there should be no huge difference in performance if designs are close enough. And again - let me say that this testing is not fully conclusive yet - I need to run similar tests on larger data source.
This level of compression impact is strange - the was plenty of CPU headroom. Testing is done on i5-3570 3.40GHz and I haven’t seen single core pegged and compression should utilize hyper-threading pretty well.
For backups, --zip-compression-level=1 is not bad actually if performance is a concern. And if backing up already compressed files, it may be actually recommended.
The behavior of --synchronous-upload="true" was a surprise for me as well and I remember looking at disk I/O chart and seeing better utilization when it was set to true. I am pretty sure I haven’t screwed up testing, but will re-test again.
I have some ideas on what profiling I can do on this, but first I want to try similar tests on a much bigger set of very large files - 15Gb with Windows Server 2016 iso, some .Net app memory dump Virtual Box vmdk file with CentOs and couple of rar files… Largest file is 5.5Gb, most files are non-compressible.
You’re doing such a great job so I don’t want to ask too much, but if you don’t mind, could you post your updated results directly in the actual post? It would make it much easier for people to find and read those results and they would be preserved as long as this forum exists.
@kenkendk - I will try to point temporary folder to another local drive and see if it changes the behavior… I also have to point out, that source and temp are located on relatively slow disk… This creates a lot of variables and still does not explain why CY is at least twice as fast. We should definitely try and find the bottleneck.
@tophee - I prefer working with real spreadsheet, of course - it allows much better formatting, calculations, notes, etc…
However, I will see if I can post brief summaries here and re-link to the Google Spreadsheet for detailed data.
I am not always record data systematically - quick tests may benefit from embedded tables.