My computer crashed, and I am trying to restore my files from the Duplicati backup to Google Drive.
When I recreate the database, it takes over 30+ hours, and then it throws error “Recreate database has missing blocks and 8 broken filelists”.
So, I tried to do a “direct restore from backup files” with the option to generate a detailed log file (–log-file= --log-file-log-level=Information) and when I look at the log I find something quite astonishing: I see it fails to find several dblock files with “The requested file does not exist” error:
2022-11-13 13:06:56 +09 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-bd8844ced934a4767a212fe6b7a344b8c.dblock.zip.aes (49.91 MB)
2022-11-13 13:07:23 +09 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Failed: duplicati-bd8844ced934a4767a212fe6b7a344b8c.dblock.zip.aes (49.91 MB)
2022-11-13 13:07:23 +09 - [Warning-Duplicati.Library.Main.Operation.RecreateDatabaseHandler-FailedRebuildingWithFile]: Failed to use information from duplicati-bd8844ced934a4767a212fe6b7a344b8c.dblock.zip.aes to rebuild database: The requested file does not exist
Duplicati.Library.Interface.FileMissingException: The requested file does not exist
at Duplicati.Library.Main.AsyncDownloader.AsyncDownloaderEnumerator.AsyncDownloadedFile.get_TempFile()
at Duplicati.Library.Main.Operation.RecreateDatabaseHandler.DoRun(LocalDatabase dbparent, Boolean updating, IFilter filter, NumberedFilterFilelistDelegate filelistfilter, BlockVolumePostProcessor blockprocessor)
So, I login to Google Drive, and I search for that same file name (“duplicati-bd8844ced934a4767a212fe6b7a344b8c.dblock.zip.aes”), and I do find it and I can download it.
So, I guess the question is: what could be causing Duplicati to not find the file, even though it exist?
Any help to troubleshoot this would be greatly appreciated.
(Note: this is a dedicated Google Drive account for backup purposes, so nobody could have possibly messed up with its contents)
PS> $FileNameOK=“duplicati-bcde8b9309dd64cdaa66b48c1223eb5cb.dblock.zip.aes”
PS> $FileNameNG=“duplicati-bd8844ced934a4767a212fe6b7a344b8c.dblock.zip.aes”
PS> & ‘C:\Program Files\Duplicati 2\Duplicati.CommandLine.BackendTool.exe’ GET googledrive://0Backup/Duplicati $FileNameOK --authid=$AuthID
PS> & ‘C:\Program Files\Duplicati 2\Duplicati.CommandLine.BackendTool.exe’ GET googledrive://0Backup/Duplicati $FileNameNG --authid=$AuthID
Command failed: The requested file does not exist
As you can see, the first one works, the 2nd doesn’t.
I can find both of them in Google Drive, inside the correct folder (0Backup/Duplicati).
Both of them have been created by Duplicati (as I can confirm in activity history), and have the exact same sharing settings in Google Drive.
It seems like it’s caused by some obscure Google Drive authorization issue… I am guessing some obscure issue with Oauth scopes or something? (but it makes no sense, since all files have been created by Duplicati, so he should be able to see them all, or none of them, but not just some…)
I tried to generate a “full access” AuthID by using https://duplicati-oauth-handler.appspot.com/ and selecting Google Drive (Full access) login [ This deviates from what’s proposed by default by the GUI, which is a “limited” AuthID ].
That did the trick. The new full access AuthID can see and download both files. I’m now recreating the DB and hoping that this time it will be fully successful.
This solution, if it works, will be OK for me because, as I mentioned, I have a dedicated Google Drive account for the backup, so I basically don’t care about restricting access to only certain files.
However, it’s surely not a good solution for most other users. Therefore it could be interesting to understand the root cause. I’m willing to help the Duplicati team figure it out.
For instance, I compared the output of Duplicati.CommandLine.BackendTool.exe’ LIST for both AuthID, and found that:
Limited AuthID lists only 3746 files
Full access AuthID lists 4146 files
I compared both file lists using WinMerge, but I could not see any pattern for the missing files in the limited AuthID. Files of all types (dindex, dblock) seem to be missing randomly across all the previous backup versions. I find it very hard to make any sense out of it.
If there’s any other information I can provide to help improve this please let me know.
while I am not an user of Google Drive, I have occasionally seen reports that at some point, Google has decided to police the files stored on Google Drive.
Maybe in your ‘bad’ files the algorithm has detected that the encrypted content is matching some forbidden information ? After all, if you make a million monkeys encrypt random data, at some point some resulting file will have in it something looking like coherent text.
According to this information, the ‘owner’ could still see them, but not the restricted access.
The owner was supposed to be informed by mail that the algorithm had done its thing.
Thanks for your reply. I think that’s not what’s going on here, because with the new “full access” AuthID my restore was successful without any errors.
The previous really strange Google (Team Drive) permission failure I recall here ended like this:
Maybe yours (while strange) can be studied in Google web UI. Do the file details show creator?
above is from January, but the latest GUI here doesn’t show what created it now. Does yours?
I get nervous about relying on full access because Google had been planning to take it away…
It seemed to me like this would break lots of use cases, which might be why it’s not done - yet.
Ever since I upgraded perms to manager, we have not had a single failure, and have done several successful test restores as well. The folder creator showed the backup account that we used to get the api key.
Sorry for my (very) late reply! It’s been a very rough month…
@genitguy3 : could you teach me how to upgrade perms to manager? This could be better than what I did as workaround perhaps?
@ts678 : I confirm one month ago I could see both files (OK and NG case) exactly like in your screenshot (owned by me, modified by me, opened by me, created with Duplicati). Now, as you also pointed out, Google seems to have changed something and I can no longer see “Created with”:
For our scenario, we have a shared drive used only for backups. In google drive web version, on the left, find the shared drive that will be the destination. Right click on it and goto ‘manage members’. Once in that menu, you can change perms to the account to manager. When complete, that account will have manager to all files and folders. This is the only way I was able to do it.
If you don’t see ‘manage members’, you don’t have rights to perform this action.
Thanks for your reply, which I somehow missed (apologies for that!).
Just to provide a conclusion on this topic:
I think you are using a business Google Workspace account, which gives you “Shared Drives” function. The solution you propose is indeed good for that scenario.
In my case, I am using a personal Google account (with additional Google One storage purchased for my backups), so I guess the only solution is to create a dedicated Google Account for the backup, and give Duplicati “Full Access” to it… It’s far from being an ideal solution (too much authorization), but it’s restricted to just the backup account. I’ve done that 2 years ago and since then I never experienced this issue anymore.
