Duplicati 2 vs. Duplicacy 2

@dgcom you are right. Duplcacy does not support SFTP as a source (contrary to what I understood from reading their website), I’ve just checked with the Linux CLI and windows gui clients. Why do you think making source and destination interchangeable is not a good idea?

I’m open to better idea’s. I’m assuming I’m missing something here. How can one:

  1. set up 1 single system (e.g. NAS, home/SMB server) to store backups from locations that only allow SSH/SFTP (e.g. shared webhosting, friends, multiple computers))
  2. backup collected data securely to remote (cloud) locations

It is not a bad idea to be able to switch source and destination, but that is usually design for sync tools (ex. rclone).
Backup usually designed to archive local data somewhere else. And NAS is one exception, but it usually exposes itself as shared drive, hence local source works. But from performance and open files reasons, I’d run backup command locally on NAS itself.

Indeed one way to do backups is to archive data from local to remote. Yet, that would require backup software such as duplicati to run from all devices, or use a different backup system to pull the data out of the devices first. Whereas if remote to local backups are also supported, that would:

  1. Allow backups to be made from locations that do not allow interactive shell.
  2. Allow backups from low resource devices that do offer SFTP but not much else.
  3. Relieve the need for installing and maintaining software on all devices.

PS We are talking about backup, not synchronization indeed.

Most backup software (and I am talking about traditional ones - Veritas/Symantec, Bacula, etc.) use local agents and central server design for a reason - it is possible to control endpoint state, preventing machine from sleep, verify file hash, copy open files…
Without local agent you loose this flexibility. You can’t (in most cases) copy open files, machine can sleep, and to check hash to verify if file have changed, you’ll need to send entire file over the wire.
Those are the reasons why you do not see this architecture often.
If you are talking Unix/Linux - you can always mount remote SSH as a folder (SSHFS) and use any backup tool as it it is local (all issues still apply). If your remote is Windows - just install SSH server - there are couple of free ones.
And keep in mind - SFTP performance is not up to SMB or even HTTP. in many cases.

1 Like

That sounds like with DT you can’t have two computers backing up to the same account at the same storage provider. Is that correct? Or does it just mean that DT can’t have concurrent backups to the same folder in that account?

From reading the source code for Duplicacy, it appears that they actually build a table of all known blocks and keeps this in memory. We had that option some time ago, but it was a real memory hog, so I removed it.

I have just tested with a small backup (~2000 blocks), and a block lookup cache had a tiny positive effect on the backup speed. But it is possible that this speedup is much more pronounced with larger backups, so I made a canary build with the option --use-block-cache. If it turns out that it really does improve performance without ridiculous memory overhead, I will convert it to a --disable-block-cache option.

If @dgcom and @jl_678 have a test setup, I would like to know the performance difference from the --use-block-cache option. I am aware of another performance issue related to how a file’s previous metadata (size, timestamp, attributes) is fetched, so it is possible that this is what really makes the difference, but maybe the --use-block-cache can help shed light on that.

Version with the --use-block-cache switch is here (2.0.2.6): Releases · duplicati/duplicati · GitHub

If that is how it works, it would require that the destination supports some kind of “If-Modified-Since” approach.

Without actually running Duplicacy, it does appear that it lists all remote blocks first, and keeps them in memory:

I think it works by having a copy (maybe without contents) of the remote store, in the local filesystem:

Some of the complexity in Duplicati is there to handle big backups. If you have a file of 1TB, you need 300mb of raw hash data (with 100kb blocks). To avoid a blowup in memory and large file-lists, Duplicati keeps it in database and uses “blocks og blocks”.

But if you have a smaller backup, this should not perform worse.

I commented on the issue on github.

I will test this new build and provide the results. I’ll use SMB share for the destination - that should remove extra variables from testing.

Interesting. I’d like to test this, but how often you’ll have to backup 1Tb file? Would this difference be really noticeable on a bunch of, let’s say, 4Gb files?

Here’s where a “scan your files and suggest settings” profiling “wizard” might be useful. :slight_smile:

I tested Duplicati 2.0.2.6 Canary and added test results to the spreadsheet.
This time I used locally attached USB 2.0 drive and used single up/down thread for CY compare.
And CY still noticeably faster, even with single thread.

I do not see much of a change between --use-block-cache="true" and not setting this option.
On backup, time is actually spent reading source files and there is no CPU bottleneck.

I’ll see if I can dig more into why TI is much slower in comparable configuration.

EDIT
Some more testing did not reveal any real help from --use-block-cache="true"
The only way I was able to speed up backup was to enable --synchronous-upload="true"
Restore with --no-local-db="true" is a bit faster compared to two separate operations.
I ditched VSS since it may take variable time and also measured compression and encryption impact on the backup.
Here are some of the results:

Backup
00:08:49.995    --synchronous-upload="true" --no-backend-verification="true" - COMMON for below
00:08:42.335    --use-block-cache="true" 
00:07:23.040    --no-encryption="true"  --use-block-cache="true"
00:06:08.722    --zip-compression-level=1 --use-block-cache="true"
00:05:04.746    --zip-compression-level=1  --no-encryption="true" --use-block-cache="true"

Restore
00:01:52.741    --no-local-db="true" --no-local-blocks="true --skip-restore-verification="true" - COMMON for below
00:01:45.409    --use-block-cache="true"

00:00:18.345    DB repair

00:01:32.149    --dbpath=<repaireddb> --use-block-cache="true" --no-local-blocks="true --skip-restore-verification="true" 

These results are still noticeably worse than CY :frowning:

As suggested before, I’ll try backup of very large files instead of many small ones.

Thanks @dgcom for trying that out, much appreciated.

I am a bit surprised that TI is that much slower, and I guessed at maybe the in-memory lookup table was reason, but it should not matter a lot, since most lookups will fail (the block is new), and the database is using a log(n) lookup time anyway. Your results show that the database is indeed fast enough (at least on an SSD).

Compared to CY there are not many differences, so I think TI should be able to reach similar speeds.

CY stores all blocks “as-is” on the remote store (in some cases using folders to reduce the number of files in a single folder).
TI stores files inside archives to reduce the number of remote files and requests.

CY keeps a cache of the remote data locally on-disk.
TI keeps a cache/lookup of the remote data in a database.

CY uses a flexible block width (content defined chunking), TI uses a fixed block width and a block-of-blocks.
They both use SHA256 as the hashing algorithm.

I see a big jump when you lower the compression level, so maybe the bottleneck is really the zip compression.
The speedup from --synchronous-upload is a bit strange, as it just “pauses” the backup while there is an upload happening.

3 Likes

Hi!

Yes, there should be no huge difference in performance if designs are close enough. And again - let me say that this testing is not fully conclusive yet - I need to run similar tests on larger data source.

This level of compression impact is strange - the was plenty of CPU headroom. Testing is done on i5-3570 3.40GHz and I haven’t seen single core pegged and compression should utilize hyper-threading pretty well.
For backups, --zip-compression-level=1 is not bad actually if performance is a concern. And if backing up already compressed files, it may be actually recommended.

The behavior of --synchronous-upload="true" was a surprise for me as well and I remember looking at disk I/O chart and seeing better utilization when it was set to true. I am pretty sure I haven’t screwed up testing, but will re-test again.

I have some ideas on what profiling I can do on this, but first I want to try similar tests on a much bigger set of very large files - 15Gb with Windows Server 2016 iso, some .Net app memory dump Virtual Box vmdk file with CentOs and couple of rar files… Largest file is 5.5Gb, most files are non-compressible.

Will update this thread with the results.

That could explain it perhaps. If the disk is reading two different files (source and compressed file), maybe that reduces the throughput. But it should not matter as much as your results indicate.

I look very much forward to the results of this!

You’re doing such a great job so I don’t want to ask too much, but if you don’t mind, could you post your updated results directly in the actual post? It would make it much easier for people to find and read those results and they would be preserved as long as this forum exists.

It can be done via copy and paste: Here's a very easy way of creating tables on this forum

1 Like

What? Then why am I wasting my time hitting spaces all over the place to get my posts to line up??? :smiley:

Let’s see…

DT DC Feature
x * Free (*DC command line free for individual users, else per user or computer fee)
x x Client side software
Server side software

Oh, yeahhh…much better…

1 Like

@kenkendk - I will try to point temporary folder to another local drive and see if it changes the behavior… I also have to point out, that source and temp are located on relatively slow disk… This creates a lot of variables and still does not explain why CY is at least twice as fast. We should definitely try and find the bottleneck.

@tophee - I prefer working with real spreadsheet, of course - it allows much better formatting, calculations, notes, etc…
However, I will see if I can post brief summaries here and re-link to the Google Spreadsheet for detailed data.
I am not always record data systematically - quick tests may benefit from embedded tables.

@kenkendk, maybe that could be the default compression level if the majority of files in a block are compressed files such as zip or jpeg or docx?

Are you proposing a per-dblock (archive) compression setting based on the actual archive contents?

1 Like

Basically, I don’t know what I’m talking about :zipper_mouth_face:

I was going to propose the lower compression level for certain file types but then realised that duplicati is not zipping individual files and so I came up with this majority thing :smirk:

All compressed files are not re-compressed. There is a file (default_compressed_extensions.txt) which contains all extensions that are known to be un-compressable:

Content from files in that list are sent to the zip archive without compression, so that should not matter.

2 Likes