Note: Starting new topic that better articulates my issue than topic “Slow Duplicati Bakcup on Mac OS”
When running an initial backup, sometimes the backup hangs. Here is a current example from a 250GB backup I have been trying to run over the last two days. Notice that the is a put operation happening every minute or so and then at 12:24pm everything suddenly stops until I find the job stalled 9 hrs later and I restart it. Then again at 2:33am everything suddenly stops until I find the job stalled 5 hrs later and I restart it.
· Aug 24, 2018 7:23 AM: put duplicati-bebef6a70140540f0a451fa1c88ae22a5.dblock.zip.aes
· Aug 24, 2018 7:23 AM: put duplicati-ib85f75fffaa949fca3e74fbe8386823d.dindex.zip.aes
· Aug 24, 2018 7:21 AM: put duplicati-b259d360a40b94ecba5d61af42e387ef9.dblock.zip.aes
· Aug 24, 2018 7:19 AM: list (Note: Found job hung for 5 hours and restarted here)
· Aug 24, 2018 2:33 AM: put duplicati-bfeff4ce7aef34f1b81804c1e35108039.dblock.zip.aes
· Aug 24, 2018 2:33 AM: put duplicati-i921580fc30be4d9e904eb003d5906297.dindex.zip.aes
· Aug 24, 2018 2:31 AM: put duplicati-bfc972f064b95470fa6a322b429ca53ae.dblock.zip.aes Skipped many similar entries in log file
· Aug 23, 2018 9:21 PM: put duplicati-bf58dc4c89ea94bc189b65d4c04bd43ab.dblock.zip.aes
· Aug 23, 2018 9:21 PM: put duplicati-20180823T133346Z.dlist.zip.aes
· Aug 23, 2018 9:21 PM: list (Note: Found job hung for 9 hours and restarted here)
· Aug 23, 2018 12:24 PM: put duplicati-b8e3099c1a63347cf9807e53c31745ecd.dblock.zip.aes
· Aug 23, 2018 12:24 PM: put duplicati-i912b9849ef714ee5a2f695b2642fdf58.dindex.zip.aes
· Aug 23, 2018 12:20 PM: put duplicati-b7afdd487882949c287d6d7e4feea5549.dblock.zip.aes
· Aug 23, 2018 12:20 PM: put duplicati-icba676b16d2a48bbbbc220af16989c2a.dindex.zip.aes Skipped many similar entries in log file
· Aug 23, 2018 9:38 AM: put duplicati-ia9289638f25c4c229fbdd8d9233616ef.dindex.zip.aes
· Aug 23, 2018 9:37 AM: put duplicati-bbc390eb59a7a4e578e39dd44a5eb9868.dblock.zip.aes
· Aug 23, 2018 9:37 AM: put duplicati-i6679f8324e7748538701e2f2fe4ca918.dindex.zip.aes
· Aug 23, 2018 9:35 AM: put duplicati-b14d69631ea0c4115b14f156d0268083a.dblock.zip.aes
· Aug 23, 2018 9:35 AM: put duplicati-id2c71fd39350452380361910be1a99b5.dindex.zip.aes
· Aug 23, 2018 9:33 AM: put duplicati-bd7fcb7e0917f47ff91bbec9cd67b5000.dblock.zip.aes
· Aug 23, 2018 9:33 AM: list (Note: Started job here)
Notes:
-I am running Duplicati on 5 macs with 13 different backup sets and I have seen this effect on 2 macs with 5 different backup sets.
-Backing up to B2
-All default options selected. No advanced setting other than URL reports.
-The current job is on a computer with a SSD.
I suspect this is not MacOS specific as it sounds very similar to the issue below where a connectivity interruption at just the right time during a “put” call seems to cause Duplicati to wait forever for the (already failed due to connection problem) put to finish. Unfortunately, as far far as I know, there isn’t a solution to that one yet.
Does this (short connectivity drop) sound like something that might be happening in your case?
I could be. I am operating over wifi so connectivity interruptions are certainly feasible and the initial backup, which takes days, provides a lot of opportunity for an interruption over that time span. I am unsure how to know if there was a connectivity issue, however.
I’m having the same issue on my Mac backing up to my home server over SFTP. I’m not sure how to see the logs you have posted @Mark_Daubenmier but I’m happy to provide any information I can to help debug.
@Mark_Daubenmier did you ever get your backup working? My initial backup has been churning for over a month now without making any progress. I have had to repair the database several times and even deleted and recreated the job once.
Eventually I got through the initial backup and things have been more stable since then, but not trouble free. I did stop the job, restart my computer, and restart the job many times during the initial backup. I could never figure out exactly why it would hang, but @JonMikeIV commented that it could be due to a connectivity issue. I never found a way to demonstrate a connectivity issue when the job was stalled. Sorry I can’t be of more help, Mark
I installed duplicati yesterday. After one false start I successfully backed up my home directory, about 60GB, to an external SSD. Today, I tried the same thing using box.com as my destination. It hung on my first three attempts.
A few hours ago I created a new box.com backup… still, basically, my Home directory but I excluded it’s largest directories (photos, cloud services, music, etc). Its total size was down to just under 10GB and, it completed successfully.
I’d love to backup my complete media library, over 1TB to the cloud but I don’t know if I have the patience to endure the initial upload especially if I can expect to encounter errors like the one you’ve described.
The macbookair that I’m running Duplicati on is not my main machine. These backups are not critical. I just want to see if Duplicati is reliable and trouble-free enough for my “real” data.
I had 750GB to back up and ended up breaking it up into 6 different backup jobs. This was helpful since when any job completed, it then worked better from that point going forward. Also, I was then able to set different rules for each. So, for example, my desktop backs up frequently and my long term photo storage less frequently. Yet, getting through the initial backup was problematic. It took a lot of monitoring and restarts over multiple weeks.
I have tried frequent restarts when it gets hung up, but every few days it starts over at “counting files”. I was assuming it was doing that because too many files changed from the time it started counting, but maybe it is a different issue. Did either of you guys see that @Mark_Daubenmier@curtis?
Yesterday the incremental backup to box.com worked flawlessly. My backup size is only about 10GB. I changed its config by by excluding a directory whose files caused errors in duplicati’s logs. After the change the backup wouldn’t run until I performed a database repair. After the repair operation I manually ran a successful backup. A few hours later a regularly scheduled backup ran successfully. Both completed without errors.
I mentioned in my first post that, for now, I am only testing duplicati. I’m testing other software too. On my main machine I use Backblaze for cloud backups. I installed it on my macbookair yesterday. As part of the installation process a dialog box prompted me to open the Security & Privacy preference pane and allow Full Disk Access for Backblaze. While there I added Duplicati to the Full Disk Access list too.
Hopefully you all don’t care about this because your backups are working fine, but just in case…
I know this is a bit stale but “counting files” is just indexing ALL the files (less any exclusions) in your source folders. It’s normal for it to “start over” with each backup run.
This list is then compared to what Duplicati has already backed up.
Anything “missing” from Duplicati is assumed new and thus added
Anything with a change in timestamp is assumed updated and rescanned
Anything in Duplicati but NOT the sure source folders is assumed deleted and flagged for removal from backups (depending on retention policy)
Note that a path change (rename or move) looks to Duplicati like a delete AND an add so will result in a re-hashing of the files but since all the blocks have already been backed up there shouldn’t be anything uploaded (other than the new file path, date stamp, size, etc.
