Thanks for the tips – I’ll throw my scripting cap on later and start to pull it apart.
Regarding list-broken-files versus affected, try described test to see if those two ever disagree.
You were having troubles with list-broken-files, and I’m not sure affected was tried for that case.
If you can find a disagreement, then some of the later checks will help when you file the isse about that.
Regarding WebDAV upload integrity issues, feel free to work with verification, but another useful check would look into some jobs’ Complete log for RetryAttempts to see how easily the files are uploaded.
Just because it only took one try doesn’t prove it got there OK, so you can go to the list at backup end (see my example) to click it to see if it saw that file right after backup. Especially handy for missing files. Corrupted files might show a good list however there’s not a whole lot of information in the listing, e.g.:
{“Name”:“duplicati-b77d8569eee444e419a4865312f82dd70.dblock.zip.aes”,“LastAccess”:“2021-02-01T23:43:26.24Z”,“LastModification”:“2021-02-01T23:43:26.24Z”,“Size”:52371869,“IsFolder”:false},
Above is a file to Google Drive, but the format should be similar. If size or time changes later, at least the listing looked OK once. I’d note, though, that file presence and size (but not contents) are checked often.
Your missing file should have been noticed at start of any backup unless you set no-backend-verification.
Finding old list operations in the GUI is hard, so I was trying to get either a DB bug report posted or you going into the DB read-only to browse around. Browsing my own DB, the above-mentioned dblock is like:
Remotevolume table
ID OperationID Name Type Size Hash State VerificationCount DeleteGraceTime
17315 1990 duplicati-b77d8569eee444e419a4865312f82dd70.dblock.zip.aes Blocks 52371869 LQ4rHLxj455A/IHCKx/cWSdEr5BE1xF+gpOtsm3oCw8= Verified 58 0
RemoteOperation table, searching in Path:
ID OperationID Timestamp Operation Path Data
48042 1990 1612222899 put duplicati-b77d8569eee444e419a4865312f82dd70.dblock.zip.aes {"Size":52371869,"Hash":"LQ4rHLxj455A/IHCKx/cWSdEr5BE1xF+gpOtsm3oCw8="}
48054 1990 1612223039 get duplicati-b77d8569eee444e419a4865312f82dd70.dblock.zip.aes {"Size":52371869,"Hash":"LQ4rHLxj455A/IHCKx/cWSdEr5BE1xF+gpOtsm3oCw8="}
UNIX timestamps above are probably backup upload, and verification after backup.
1612222899 Monday, February 1, 2021 6:41:39 PM GMT-05:00
1612223039 Monday, February 1, 2021 6:43:59 PM GMT-05:00
Remote file time seen at server will likely be the put time plus maybe some delay.
RemoteOperation table, searching in Data:
ID OperationID Timestamp Operation Path Data
48049 1990 1612223023 list [ ... ]
Data is a big JSON array containing what should look familiar from GUI:
{"Name":"duplicati-b77d8569eee444e419a4865312f82dd70.dblock.zip.aes","LastAccess":"2021-02-01T23:43:26.24Z","LastModification":"2021-02-01T23:43:26.24Z","Size":52371869,"IsFolder":false}
So this saves having to flip through GUI, and also traces file history.
Got a new one – backup goes thru all of steps with maybe a handful of errors, then tries to upload the verification json and blows up
(423) locked
I appreciate all of your help, but I believe the root cause of my grief was a failed array two weeks ago. I will start making fresh backups from scratch on Tuesday whenever I can grab the machines. I’ll move the ‘bad’ backups to an external disk in case I need to pull something from several months ago. Hopefully, the remake count will be less than a third of my total load.
I’ll probably keep playing around with it, but at some point, it’s time to stop cutting bait and get your hook in the water. But please if you have any brilliant ideas you’d like me to try (in the spirit of helping the developers), I’m game.
I did find a forum post (https://forum.duplicati.com/t/backup-unusable-duplicati-tells-me-to-repair-but-repairing-doesnt-work/4209) that had a lot of the same problems I did, especially in regards to the list-broken and purge-broken not returning anything.
Thanks
Rick
At least this one is something that a Google search finds that other Nextcloud systems are giving.
There are even lots at help.nextcloud.com. I’m not reading them, and never did find much on 503.
Nextcloud was said to have “raid controller”? If this system is ill, maybe it’s causing HTTP errors?
Regardless, I hope your fresh backup plan works, but I worry about the health of Nextcloud server.
If it needs more debugging, you have ideas, and there are more too, but rare errors are still hard…
Question:
Running the python script locally on my nextcloud server takes about an hour to complete on user LIN.
Logging onto LIN and running commandline test with all and -full-remote-verification takes 1/4th of that time
And as I understand the full-remote part, it’s going beyond the hash check. So why is there such a time differential?
In addition, the python check shows 6 missing files. The commandline test all (full-verification) shows no such problems.
Since the commandline test is also doing a hash check, why aren’t the missing files flagged?
Is this a huge problem, or will a simple db rebuild fix the issue?
Thanks
Rick
Different implementations in different languages on different systems?
I’d have guessed local Python would win the race, but seemingly not…
Are you sure you ran it when Duplicati was idle? I would have expected agreement, unless those files happen to be in a state where one check forgives and the other doesn’t. What is state from looking at duplicati-verification.json or browsing the database’s Remotevolume table in DB Browser for SQLite?