Yeah, I would live multiple destinations from a single source as well. And the way it sounds like Duplicacy does it’s block handling (with remove file names vs. a local DB) is actually pretty cool and fits PERFECTLY with multiple destination features.
BTW: A good way of showing your appreciation for a post is to like it: just press the button under the post.
In the #support category, if you asked the original question, you can also mark an answer as the accepted answer which solved your problem using the tick-box button you see under each reply.
All of this also helps the forum software distinguish interesting from less interesting posts when compiling summary emails.
Done. Hint: you can flag your post and ask for it to be turned into a wiki.
I find this one quite fascinating:
Multiple clients can back up to the same storage at the same time
Added it to the OP.
Oh, just realized that @dgcom also mentiined it above. I’m not sure I understand all of what you’re saying, though. Are you implying that there should be a
* for duplicati?
Well, in this case I’d say Duplicati should get an
x since the Feature is that multiple CLIENTS can back up to the same destination at the same time - which I’m pretty sure Duplicati can do (unles the destination somehow restricts concurrent logins in some way).
If we were talking about a SINGLE client backing running multiple backups to the same destination at the same time (maybe you’ve got an hourly Documents backup but only a daily Photos backup that both happen to fire at the same time) then I’d say Duplicati should stay empty.
As far as I can tell a single backup job can be multi-threaded, but only one backup job (or any tray based command line) runs at any one time. Note that I could be wrong here as the tray/web UI doesnt’ seem to be designed to show multiple tasks running at the same time.
Yes, that it can do. But my understanding is that duplicacy not only allows multiple backups to the same destination but to the same archive, including block level deduplication so that if a copy of the same file exists on multiple machines, it will only be uploaded once. That is pretty fascinating! Or am I misunderstanding something?
This is my source:
However, if concurrent access is required, an unreferenced chunk can’t be trivially removed, because of the possibility that a backup procedure in progress may reference the same chunk. The ongoing backup procedure, still unknown to the deletion procedure, may have already encountered that chunk during its file scanning phase, but decided not to upload the chunk again since it already exists in the file storage.
Based on that quote and how Duplicacy appears to use the file names themselves to identify blocks then it would make sense that it “natively” supports cross source de-duplication.
We know that at present Duplicati does NOT support that (though it has been mentioned before that it could) so perhaps a new Feature for “Cross source de-duplication” or “Cross-backup de-duplication” should be added with blank for Duplicati and
? for Duplicacy until somebody can verify our assumptions?
As far as I understand, it is not recommended to have different clients and different jobs to backup to the same location (folder) because it can cause conflicts…
I don’t want to annoy anyone, but I think I found another little thing to bug people about:
If you refer to some other post on this forum, it would be great if you could provide a link to that post.
The link will automatically be visible from both sides (i.e. there will also be a link back from the post you’re linking to). Those links will make it much easier for people to navigate the forum and find relevant information.
I have been testing both with Backblaze B2 and am still trying to sort out which I will adopt. One big difference is speed/deduplication. They take two different approaches. Duplicacy by default relies on file size/time stamp differencing unless you specify the “-hash” tag which is not the default while Duplicati chops files into much smaller pieces for more efficient data reduction. These differences make the two products hard to compare. Here is the results from a test uploading 3GB of actual data with both solutions.
Duplicacy: Upload time ~45 minutes
Duplicati: Upload time 1 hour, 9 minutes.
Duplicacy: ~30 seconds
Duplicati: ~2:40 seconds
In short Duplicacy seems much faster although Duplicati is storing about 200MB less data. To be 100% clear, I am running this on a small single board computer running Ubuntu dedicated to this purpose, and so compute power, memory and bandwidth are limited. (Your performance is likely much faster.)
I really like the Duplicati GUI and there is no GUI for Duplicacy on Linux; however, I find the performance difference significant.
I will continue my tests and welcome any feedback.
Am I right to understand that the default behavior for duplicati is to not rely on timestamps and examine file contents on each backup? Is there an option just to rely on timestamps?
A bit of background - in my long term plan, I have two data sets that I plan to backup -
Logging/Records data set (relatively small 10mb but frequently changing)
Archive data set (large ~200Gb but low frequent changes, usually just additional files and extremely rare for file edits)
If there is an option for timestamps, then I suppose the backup refresh of the latter would be much much faster (and less wear on hard disk but that is super minor)
Take a look at
--check-filetime-only, that might do what you want.
BTW, is there any documentation for the advanced options? If not, I guess I need to search the Google Group…
Not sure on process on joining out team for helping out development of duplicati - but (depending on scope) I am happy to volunteer to maintain documentation on this (and others?). I am imagining a list of the options with some explanations etc.
Any ideas on who to PM for this?
Please, run the following in the command line:
Duplicati.CommandLine.exe help advanced
That said: I don’t even think you need to bug anyone about this (let alone ask for permission). You can simply create a topic in the #howto category and update it whenever appropriate.
That would be fantastic help and much appreciated!
Duplicati uses indicators (metadata, filesize, timestamp) to guess if a file has been changed. If one of these indicators have changed, the file contents are scanned.
As noted by @JonMikelV you can use
--check-filetime-only to make the timestamp the only factor. You can also use
--disable-filetime-check to always check file contents.
I didn’t know Duplicacy before, and am impressed with the simple concept which looks really elegant and fail safe!
I just tested it and was impressed with the performance/speed too. But the GUI is still really simple and a lot of featues are missing in it (e.g. just one backup job and one source directory is supported). So maybe we should include this in the table above?
It sounds like multiple are supported via the CLI?
yes, I was just talking about the GUI
The way things handled in duplicay cli aligns more with Unix/Linux based OSes - you have to choose a single folder, which will be a root of your source… You can do this at root of the filesystem or you can create a special folder and then link all your source folders there. This also works in Windows with symlinks or junctions. This requires much more housekeeping compared to usual way of specifying backup sources.