Duplicati looked for me like the perfect Backup solution:
Lots of options
Lots of protocols supported
Lots of Cloud services supported
encryption
versioning
I started backing up 500Gb of files to JottaCloud. This took some days. Sometimes duplicati crashed and I had to restart the process.
After that I tried to start the restore proccess. And here the catastrophe started:
Duplicati needed 8 hours on my QNAP to create its database
Duplicati is not able to download the backuped files in parallel and jottacloud is insanly slow with one single download
My NAS restarted during the restore (my fault) and I found no option to resume the restore process there it stoped
I downloaded the files via JottaCloud Software (6 streams in parallel)
The files are now stored locally
I restarted Duplicati and the restore process from local files
I took again hours for the database and then Duplicati crashed by itself
and again no way to continue the restore
All in all Duplicati has the following flaws:
The Web Gui is reacting very slowly or sometimes didn’t react at all
No usable logs are created in case of a crash (or I am too dump to find them)
Restore process can’t be resumed
My actual status is that I am not able to restore my files or have to wait for days due to the slow jottacloud download.
I have to be sure that in case of data loss my Backup software is able to recover the backup. Duplicati seems not to be able to handle basic tasks like resuming the restore process (or I am again to dump to find the option to do so).
Also the performance is weak: taking hours for rebuilding the database of a 500GB restore is really much.
I know that the work invested in the developement of this software is done in your spare time and duplicati is free. Nevertheless you endanger your users by advertising the software as backup solution:
Backup files and folders with strong AES-256 encryption. Save space with incremental backups and data deduplication. Run backups on any machine through the web-based interface or via command line interface. Duplicati has a built-in scheduler and auto-updater.
I am already a technician and I am not able to recover my files, even if they are saved locally. “Normal” users will not do a restore test and they will trust that in case of error duplicati can restore the files.
I would be happy if you proove me wrong and show me that I made mistakes why all this happend.
Actually I am a bit depressed because at the beginning I really liked Duplicati.
From what I am seeing, there are multiple levels you’re mentioning here.
First, what version are you using? There are known bugs with 2.0.3.6-canary, so I strongly suggest going back to 2.0.3.5-canary.
500GB is a good sized backup for any software, much less trying to run it to/from a remote server over an internet connection that is limited in speed.
Many people fail to realize that this is not just a simple file copy. You may be able to direct copy files from remote server to local drive and get it done in a few hours, but with backup programs such as this (and others), they have to sift through all those files, add the details of file name, size, type, markers where it is on the source and the backup location, as well as the full database scan when a restore is needed as well as updating the database during a backup.
I personally have been testing Duplicati for my job and we deal with several TB worth of data and backups. The old backup software we use typically takes 3-5 days to do a single full weekly backup within the local network (all connected via gigabit or 10Gbit connections). If the server goes down the backup is usually lost and it takes several days to resync everything just on the server side not including backups. So that means some employees may end up with an older version of the file depending on where their marker is pointed to.
I have Duplicati setup to backup the core 2TB worth of data (data server via 1Gbit network to testing backup server with 6TB local storage), split among 5 daily backup processes for specific “drives”, and having them staggered across 12 hours (first starts around 6PM, last starts around 6AM). Once the first initial backup is done, some of these backup processes take 45 minutes to process 500-600GB worth of very few files changed, others take 3 hours to process 120GB worth of files with a lot of changes. The backup server is 8 core Xeon at 2.0GHz, 32GB RAM, 6TB storage space.
I found a first initial backup with my specific conditions ran approximately 10GB per hour. Since you’re running over a much slower internet connection, this will likely take longer. My 1Gbit LAN connection means 125MB/s, but a 50Mbit internet connection means at best 6.25 MB/s. Even in a best case scenario taking Duplicati out of the loop, 500GB at 6MB/s means 26 hours just for straight file copy. Add in the overhead of other items using the network connection, the server processing time of all those files, and you’re still looking at 5-10GB per hour at best.
So my best suggestion is split it up, point the source to specific folders or groups of folders, and backup 50-100GB at a time. If you have lots of folders immediately under the top level folder then it may be time for some maintenance and cleanup at that level as well to consolidate those into related folders. This will help regardless of what backup software you use.
500GB is a good sized backup for any software, much less trying to run it to/from a remote server over an internet connection that is limited in speed.
Correct but I also tried the restore with local files. And this also failed. You are reffering to network issues but I am not able to restore files from a local duplicati backup. And due to missing entries in the logs I can not do anything against it.
I cannot comment regarding that since restore has always worked for me.
One key I have found though is if Duplicati is currently running a backup, it will not run the restore job until after the backup job has finished. It is not immediate. If that backup job takes hours or even a few days, it has to finish that before the restore will do its thing.
There has to be at least one good full backup before restore works, otherwise the database is incomplete and it errors out.
I gave one of my backups another try: duplicati is now running on my windows laptop (ssd / core i7 / 8gb ram)
This time everything went very fast and it already could recover half the storage (40GB recovered from 80).
so it looks like that there is a problem with duplicati on arm devices. if anybody wants to troubleshoot it feel free to contact me
This restore is also running from local drives. All restore tests from the internet (sftp and jottacloud) are not usable due to missing parallel downloads. The download would take days…
I’m glad it’s working well for you on at least one of your machines.
From what I can tell, this seems to be the general experience - it either works great (which is the norm) or you have lots of difficulties. We haven’t figured out why some setups run into a bunch of issue while the majority have no problems (that they’ve told us about, anyway).
I think there are a few people running Duplicati on some ARM devices (RasperryPi, for example), but if I recall correctly the performance is pretty slow.
Parallel transfers are in the works, but require a big code change to be supported. That started with version 2.0.3.6 canary, but we’re still ironing out the bugs related to so much code having to be updated.
I too am using duplicati with Jottacloud (not sure how many of us are there). Recently my home windows server ran into issues after a windows update (CLASSIC!) so I started the process of restoring the image backup file that was backed up to Jottacloud using duplicati (I could have also restored the backup files using the local drives attached to my server but this was a good test case). I installed duplicati on my laptop, entered all the info to restore the files straight from jottacloud and ran into issues (this was a large restore…i think like 300gb or something like that). After the restore “finished”, none of the data was actually restored and got an error for basically every single file (can add the specific details later). I didn’t bother trying to further troubleshoot the issues since it took so long for the restore to even fail. Not really duplicati’s fault for taking so long since I think the download\upload speed for Jottacloud is quite gimped when using duplicati (I find I can get 20mbps down when using the jottacloud app) however a little frustrating to have to wait so long for a message indicating nothing was restored. I then proceeded to use the jottacloud app to download all the encrypted files first and perform the restore locally which worked without any issues at all.
Anyway it was all in vain since apparently the version of software I was using to perform my image backups doesn’t really work for my use case (windows server 2016 essentials running as a DC) so I ultimately ended up having to rebuild the OS drive from scratch This however gave me a new opportunity to test the “adoption” process of duplicati on the rebuilt machine (don’t really know how else to put it since that’s what crashplan called it and that’s what I’m familiar with). That process went very smoothly, which consisted of recreating the backup jobs I had previously made (learned my lesson and now have the configurations exported and backed up), running each backup job for the first time, it failing, running a database repair and then running the backup again.
Anyway, as a new user to duplicati, and Jottacloud for that matter (just started using them like 3 months ago), I really do like it.
Hi @ahslan, welcome to the forum - and thanks for sharing!
I think there’s another topic somewhere where it’s been discussed possibly including the backup job in the backup itself which could help smooth out situations like this.
Just to clarify, that backup to Jottacloud worked, but the restore DIRECT from them failed - though a copy to local drives & restore from that was fine?
Correct. The backup to them worked, and the restore directly from them didn’t but once the backup files were downloaded to a local drive, the restore from the local files worked perfectly. Sorry for the confusion.
This was the same procedure which I used. Duplicati take a long time (5 days) to download the files because of one single download stream. The software uses 6 streams and the download was 6x faster in total.
Nevertheless a restore which tooks five days and can not be resumed in case of error is no option for me.
This however gave me a new opportunity to test the “adoption” process of duplicati on the rebuilt machine (don’t really know how else to put it since that’s what crashplan called it and that’s what I’m familiar with). That process went very smoothly, which consisted of recreating the backup jobs I had previously made (learned my lesson and now have the configurations exported and backed up), running each backup job for the first time, it failing, running a database repair and then running the backup again.