I am a new user of Duplicati. The product looks great I am super willing to have it working. I quickly went through more or less similar cases on the forum but could not find a solution.
I downloaded it today and did my first backup on an external HDD (around 250 GB of data, I use windows 10 and the web UI of Duplicati in Chrome). Everything went fine until almost the end. Now the program has been stuck for one hour trying to back up a pdf file of 7 kB at a theoretical 10 MB/s speed.
I could not find the “live” log often mentioned in the forum. What I can access is the “remote” log (?). The last line is a bit older than one hour and says :
{“Size”:47198333,“Hash”:"/dv7nhvRwhek7A7orICT6dRTi8w7mJryptJ7TD4BcVc="}
Is there something to do to solve the problem ? Let me know if more info are needed.
The Live log is accessible in the web UI main menu. Click About → Show log → Live and set the dropdown to Verbose. You should see activity here if a job is running (most recent events will be on the top).
Thanks a lot for the answer. I found the place you indicate (it mentions a server so I thought it was an irrelevant place to look at in first place). So now it is on verbose but there is nothing to read so I guess nothing is happening.
Otherwise the situation is still the same, the green bar indicate the backup is dealing with one ultimate light file at 5 MB/s.
Just to say I more or less solved the problem. I stopped the backup and restarted it. It went much faster (around 1h, same amount of total data and no recent modification) and it ended in a successful backup. No idea what happened in first place.
One thing now puzzles me. I added a folder containing 5 pictures in one of the locations I am backing up and relaunched a backup to see how long it takes to do a backup after a small modification. It still took 50 minutes. I remember using Time Machine back in the days and, if only a small modification was done, the new backup was really fast. So I am wondering if I set correctly by backup ?
Subsequent backups should be pretty fast. Only modified files are processed. If you were to run the backup again right now (with presumably no modifications to the source files since the last backup), how long does it take?
Thanks again for the support. It took again one hour even though nothing had changed. Based on the info in the green bar, the program is considering the whole amount of data and browsing through it entirely to check possible differences. It counts up to to total amount and then count down to 0 while it browses through the data. Live log indicates things like :
Checking file for changes
Including path as no filters matched
File has only metadata changes
I also have 3 remaining warmings, 2 related to two corrupted folders I don’t manage to delete and one that I don’t understand about a hidden file I cannot find in windows explorer.
Finally these data are also synced on a cloud so maybe it complicates the process for Duplicati.
Locally I was seeing kind of the same on lan from a computer the other day. Apparently, one of the cable ends needed to be taken out and put back in. Went from 100 link speed to 1,000 after I did that.
If the external drive is connect via USB, that would be different. Could be the cache is saturated fully or something. Who knows. Lots of annoying situations are possible anymore.
Were you getting a lot of these? Normally this would be something like the timestamp on a modified file.
Files that are not changed should just be seen as not changed, then skipped over in the file processing.
The <job> → Show log → General log should show examined files that got as far as opening for a look:
I did not check a lot of pages of the live log but it seemed that yes, a lot of these messages were there. The pattern was these 3 messages I pasted, one after the other, repeated many many times. Below is the mini report from the General log. It seems indeed that the program goes through almost everything at each backup even though nothing was modified (at least actively by me).
That seems like an unexpectedly large number of files seen as modified (given your view of change).
That’s almost all your files. Maybe we can try to isolate it down to the details on the lines that you got.
Could you post a sample, redacting path if you like? Here are two of mine from the test backup I ran:
First is a text file, opened in notepad then just saved. Explorer also showed Date modified change, however at least it gives local time (Duplicati looks like UTC). How do the Explorer timestamps look?
Another possibility is if you use usn-policy Advanced option, there’s a bug on the latest Windows version.
If you have done advanced work like moving backups or recreating database, other oddities may occur…
I’m not sure what else might be changing things. Some antivirus programs hide some information in files. Are you using any specifically-installed AV? Windows comes with an AV, but some people install another.
Probably good next step is to post your line to see what it says in the details, then maybe look in Explorer.
If it’s an easier test environment, you can set up a small test backup to look into the nature of the problem.
First thing : I did something my wife usually never let me do, I took an initiative. I re-ran my backup but excluding temporary files, hidden files and system files (I have no personal interest in them, I included them initially because I had no opinion on the question). And it went really fast as expected when everything is fine ! So Windows is messing up with some of these files. So that’s great news. I have a super functional backup now, that’s awesome.
Regarding my personal use I am happy as it is now but if it is of interest for you, I am also happy to keep the investigations going. So to answer your questions, I did not change anything regarding usn-policy. It’s not something you go through when you set a normal backup with the gui so I did not touch it. It is used in its default mode.
I have an AV running on my machine, McAfee EndPoint security. I did not install it myself, this is the laptop my employer provided me with.
I am a bit confused with the things you suggest to do, maybe I got it wrong. What I did is a mini backup of a text file like you (and including the hidden, temporary and system files). I put it within the bigger space I am backing up with the big backup so that the same kind of potential problems apply. The backup goes super fast. I re-ran it 3 times. Here are the last 12 messages of the verbose live log (from the very last backup). MS explorer does not indicate that the backup modifies the file (not sure I understood the suggestions linked to the explorer). Let me know if something different would be interesting.
Have a nice weekend !
Mar 12, 2021 6:19 PM: The operation Backup has completed
Mar 12, 2021 6:19 PM: Backend event: Get - Completed: duplicati-b46033e1c0e344826b465c632a33d7f01.dblock.zip.aes (1005 bytes)
Mar 12, 2021 6:19 PM: Backend event: Get - Started: duplicati-b46033e1c0e344826b465c632a33d7f01.dblock.zip.aes (1005 bytes)
Mar 12, 2021 6:19 PM: Backend event: Get - Completed: duplicati-i39b09166c2c942fca9d42cac452f1d58.dindex.zip.aes (925 bytes)
Mar 12, 2021 6:19 PM: Backend event: Get - Started: duplicati-i39b09166c2c942fca9d42cac452f1d58.dindex.zip.aes (925 bytes)
Mar 12, 2021 6:19 PM: Backend event: Get - Completed: duplicati-20210312T171840Z.dlist.zip.aes (989 bytes)
Mar 12, 2021 6:19 PM: Backend event: Get - Started: duplicati-20210312T171840Z.dlist.zip.aes (989 bytes)
Mar 12, 2021 6:19 PM: Backend event: List - Completed: (3 bytes)
Mar 12, 2021 6:19 PM: Backend event: List - Started: ()
Mar 12, 2021 6:19 PM: No remote filesets were deleted
Mar 12, 2021 6:19 PM: removing temp files, as no data needs to be uploaded
Mar 12, 2021 6:19 PM: Skipped checking file, because no metadata was updated C:\Users\Marti126\OneDrive - Universiteit Utrecht\Admin\test4backup.txt
I am a bit puzzled. I re-ran the backup unticking the exclusion options one after the other to see what category of files was making problems (temporary, hidden, system). And now that every box is unticked (the 3 types of files are included, as initially) it is still fast and nice. So nothing more to debug even when trying ! If it comes back at some point and is of interest for you I can come back and post the lines we were talking about,
Anyways thanks a lot. Your product is great, works fine, looks awesome, you provide a lot of help, that’s just amazing. I’ll advertise Duplicati around me the best I can.
Thanks for the kind words. The product is still not as stable as desired (so it’s still considered a Beta), however the remaining issues are being chipped away as fast as the available volunteers can do so…
