I’m afraid I’ll have to add another topic about problems with repair (I’ll add some others at the end of the post for reference).
I have not been very successful so far in getting duplicati to work for me. Recently, I number of possible sources of problems have been eliminated, including
changing from ACD to pcloud as a backend
setting up an entirely new backup job
installing 2.0.2.15_experimental_2018-01-03
But not even even the initial backup of 9-10 GB completed. It just throws the usual error:
error whilst running , found 243 files that are missing from the remote storage, please run repair.
I don’t understand how hundreds of files can repeatedly “disappear” from the storage, across different backup jobs and storage backends. I’m suspecting that they are not missing but duplicati falsely believes it has uploaded them. But that’s another issue that I don’t intend to discuss here.
So when I then do run repair, it seems to freeze, sooner or later. The progress bar is stuck and there are no more new entries in the logs. Same thing for two different backup jobs, run independently of each other several days apart:
The Duplicati.Server process still continuously uses about 15-17% CPU so it might still be doing something, but I doubt it since there is neither disk nor network activity: .
I only waited a bit over 24 hours before I killed the process, so I can’t say what would have happened if I waited, say, a week.
I guess the next step is “delete and recreate”?
For anyone with repair problems which don’t match mine, here are some other relevant topics:
So after waiting another 24 hours, I did precisely that: delete and recreate. This is what I got:
The target URL is correct and there are 245 files using 5.94 GB in that path.
Maybe this can lead us to the source of all the misery? Why is duplicati not seeing those files? (no wonder it complains that files are missing!) How to find out?
UPDATE: I Tried to run the backup job after the above error and got this:
So I ran repair and delete again but got the first error again, and so on and so forth.
I don’t even understand why it says “The database repair was attempted” when I did not attempt a repair but performed (I believe, since I was not told otherwise) a “Delete and recreate”.
I believe a database repair is basically recreating just the parts of the database that are having issues. So that would make “Delete and recreate” the same as manually deleting the database and clicking “Repair”.
The reason the repair (or recreate) won’t work is that those are done by downloading dlist (or dblock) files from the destination and putting their contents into the local database.
If NO remote files are found, then there’s nothing on which to base the repair / recreate.
If SOME remote files are found, then the repair / recreate can happen. For a recreate, it should run fine (though any contents of the missing files will be gone). For a repair, you may then get an error about files missing from the destination (at which point a purge process will likely be needed to clear the database of references to the no-longer-existing files).
If Duplicati sees files from another job at the destination then it should report unexpected files found so unless there’s a communication error between the destination and Duplicati (for some reason the destination is telling Duplicati there are no files to be found) the issue is most likely some hard to see typo making the actual destination folder and the one setup in the job be different from each other.
I’ve had issues with the usual things like upper case “i” lower case “L” (aka “I” != “l”), upper / lower case “o” vs. “0” (zero), unnoticed spaces, and (depending on destination) case differences and even leading slashes, tilda’s, etc.
I usually find them by copy/pasting the paths from the destination and Duplicati into a text editor so I can see them side by side.
Another thing I’ve tried is re-doing the “Test connection” process to see if it reports needing to create the folder again.
I guess this is the crucial point. In other words, you are saying that the delete and recreate operation actually failed, even though Duplicati did not produce any error. So, as a side product of this topic, I suppose Duplicati should produce an error in this situation (@kenkendk).
And, while we’re at it, here is another idea: I think it might be helpful if Duplicati would always write some file into the backend, basically first thing after setting up a backup job. That file would serve as a test for whether Duplicati actually sees anything in the backend. If it encounters misding files and doesn’t see that file, it knows this is not just about missing files, this is “something bigger” and can react accordingly (i.e. not tell the user to do a repair).
So, back to the actual problem: why is Duplicati not seeing the remote files?
I can double check that when I’m back at the computer, but I’m pretty certain that this cannot be the reason in this case because this is an entirely new backup job (for which a brand new folder was created in the backend) and all the above mentioned files that are there, have undoubtedly been uploaded by that very job. And the job has not been edited whatsoever, so the paths basically have to be correct…
One thing you could try would be to change the job destination to a new folder and run “Test connection” so Duplicati creates the new folder. Then manually move the files from the old to new folder and see what the backup has to say.
The idea of disallowing a recreate / repair if no files are found makes sense to me. Assuming it’s not already doing that (but with a bug) perhaps @kenkendk has an opinion on that.
So the result was as expected: path is okay. I even accessed the storage via webdav using windows explorer in order to see whether it might have something to do with pclouds webdav access. But whichever way I check, the files are clearly there, and whatever I do with duplicati, it says they are not:
Here is the config of duplicati:
And here is the folder accessed via webdav:
Well, to start with, “Test connection” doesn’t create the folder if it is missing and I believe it never did. - No, wait, I think it used to throw an error about path not existing or something but then it had nevertheless created the folder. In any case, when I just changed the folder to PC_D2 it gave an error and when I went to the storage directly to create the folder manually, it wasn’t there, so Duplicati did not create it.
I tried both renaming the existing folder and creating a new folder (and renaming the path in the duplicati config accordingly) but to no avail. Duplicati does not see the files. At this point, I am actually pretty sure that this is a bug.
And there is also a bug in the “delete and repair” button because, as reported earlier, clicking that button repeatedly fails/freezes but when I press “delete” and then “repair”, I at least get out of the vicious circle of failed repair attempt with the suggestion of deleting and repairing. When I do delete and repair in separate steps, it at least starts recreating the database (at least it says so in the progress bar) but then complains that there are no files in the storage.
This is a long shot, but perhaps the parsing of the path differs between backup and repair.
Is it stated somewhere that WebDAV paths should be written with backslashes? Perhaps try writing the paths with forward slashes and see if restore starts working. And again, sorry if I’m way off track here
Bingo! Looks like your long shot hit the nail on the head! Forward slash seems to work.
So, @kenkendk, I guess the main two lessons from this odyssee are:
Duplicati accepted the backslash path for uploading and successfully uploaded 246 files but when it came to reading those files, it was no longer able to handle the backslashes while at the same time giving positive results for “Test connection”. So either it should not accept backslashes at all, or when it accepts them, coherently transfer them to forward slashes (if that is what the backend expects).
I have just had a similar issue, repair just getting stuck. It appears that my local sqlite database got messed - it was claiming to be open (even after a restart). To provide this, I moved to the location of the database and moved it via the command line (adding .orig) after it’s name and then ran the repair process again. This time, all over and done with in a couple of hours. It is clearly worth checking to see if the local database is incorrectly locked if you repair seems to take for ages.
Thanks for sharing your experience with this issue. I’m not aware of any other database locking problems being reported, but there have been some rewrites in that area lately so something might have been missed.
What version of Duplicati were you using when you ran into this and on what operating system (MacOS, Linux, Windows, etc.)?
.
