Several Days and No Restore / Feedback

Cirrus22 · November 24, 2018, 1:55pm

Let me prefix this with I came from CrashPlan. After the initial Crashplan backup, restoring anything was simple, you just went to the folder/file and restored it.

I have about 80Gb going to a Dropbox nightly backup. The actual backup seems to have worked fine. I’m try to restore the files (I’ve tried on a Win 10 VM and a High Sierra Mac)> I’m using I believe the latest version.

Look I understand this is free, but I don’t want to be a test pilot. I just want the most basic backup and restore. I’ve been trying to restore files/folder now for several days (what a complete an utter waste of time). I ‘though’ I had restored the data, but no, after 24 hours all I had was an empty folder structure. So here’s some feedback.

Yes I know things take a long time, but how about some feedback, it is absolutely ridiculous to be sitting around with a single message (say Recreating database), the progress bars go to 50% and sit there - utter waste of time, with the user left wondering whether the system is dead or is something happening
For God’s sake, disable the buttons on the UI, coding 101, you accidentally click one of the left hand buttons and your last 12 hours of fruitless waiting is gone (and if it isn’t there’s no obvious way to get it back / continue)
Put the OAUTH token in an encrypted INI file / Registry etc etc. Of the dozens of times I’ve had to start this software getting the Token every time (yes I put it in a file eventually) is a PITA
Again, the only way I know anything is happening is that the fans on my MBP are spinning out, that is really poor interaction with the software and the user, the user should know at all time that things are happening and what is happening
And now, an hour or so after the database was recreated and I have managed to select folders, the system is off in la la land again supposedly building a partial database and I get the progress bar sitting there about 2mm wide doing nothing, I only assume something is happening from the noise the fans are making and the fact that mono-sgen32 is way over 100% CPU

I’ve seen many posts with people who are trying to restore or repair some fairly large backup sets and just go on waiting for days, literally. I felt quite good when I got the initial backups sorted to Dropbox, that was an excellent feature. I checked daily to ensure encrypted files were being written, ie. the incremental backups and they were. That was, as it should be, a fire and forget.

The restore data OTOH is an absolute nightmare. To my mind, I have little confidence that I can (relatively simply) restore my meagre folders. I think the UI for the most part is clean and simple, but fails miserably when it come to informing the user what if anything is happening and has the user guessing whether to leave the system for several more hours or kill it and start again. Simple as it was, I never had any issues like this with CrashPlan, it just mounted the backup (quickly and effectively) and, in an Explorer like interface, I simply copied (in real time) the files I wanted. I didn’t have to guess whether the files were restored or not. Even things like TimeMachine, if doing a big restore, have a progress bar that is at least meaningful.

OK rant mode off. My user experience is poor, and I’m the first to admit I’m no guru, but the process should be absolutely straight forward with feedback and confirmation:

Enter/verify credentials
Select backup set (should be quite quick, it isn’t)
Select Files/Folders if not full restore (should be quite quick - it isn’t)
Select destination
Duplicati tells you what it’s doing etc etc so you can see precisely where it’s up tp etc (no it doesn’t)

Once I have the data I need, if ever, I will cease using Duplicati, it’s too risky and should anything go really wrong, I don’t have much confidence in the product doing the two absolutely fundamental things: 1. Backing up my data - yes positive experience; and 2. restoring my data - No absolutely negative experience.

And just to reinforce, a couple of hours of creating/recreating the temporary database, the proagressbars are pegged at about 2mm, there is nothing in the destination folder, I have no feedback AT ALL about what’s going on - utterly utterly frustrating - all I can do is let the software churn and ‘hope’ something is happening because there is no evidence that anything other than fan spin is going on…

Kevin

Pectojin · November 24, 2018, 3:28pm

Hi Cirrus22,

I’m sory to hear you’ve had a bad experience, but I’m happy you got in touch to let us know what can be improved

I would definitely advice against using Duplicati at this time if you need production ready software. It’s in beta because we’ve stil got some way to go before we can guarantee a good service.

I recognise this sentiment. We have a progress bar, but it’s not always explicit enough during processes that take a long time. Improving this is indeed important.

Do you mean that during the restore you would lose progress if you click away from the restore screen? It should still continue the restore progress even if you go way or close the tab.
Perhaps the issue is that there is no list of previous/currently running tasks, making it hard to tell?

It’s saved in the database so you should only have to save it once when configuring the job. In the case that you’re restoring without a job, then it has to be configured every time, but is that case happening often?

Yes, it’s definitely working on it. If you already got to select folders/files, and nothing is missing in the backend data, then it will restore successfully. But I agree this part can be unintuitive because it may take a long time on less powerful CPU’s and in that time it’s hard to tell how it’s actually progressing.

Cirrus22 · November 25, 2018, 1:33am

Hi Rune,

Thank you for your point by point response. That post was 03:00am and it was a bit of a rant. I’ve been trying to recover some data for several days. The MBP (with with Fusion VM or natively) seems to go off for hours and I think the restore has failed so I abort and start again (it probably hasn’t). I think it actually was working the first time. After 20++ hours it was scanning for whatever blocks (where it is now I might add). I killed it because there was no sign of progress.

Again, there are two big issues for me:

The complete and utter lack of feedback. A progress bar is useless for an activity that takes 20 hours, there is NO sign of life. And you cannot always go by the fans going berserk. When searching for files to restore was happening there was dead silence, the system, for all intents and purposes, was completely hung. It wasn’t, but that’s the problem.
Yes, I know it does some nice things, but the times are simply ridiculous. And I know I should expect some delays, but Duplicati is OTT. For example the latest attempt I have started at or about 0230am -ish. It is now my time 1315, almost 11 hours later, the system has been stuck on Scanning local blocks for 5 hours. There is, again, no indication that it is progressing I simply get this screen:

That’s how the screen has been, nothing else - no movement, no messages etc etc. Would you think anything was actually happening or that the software was hung? Me, I’d pick the latter.

And, once again, the folder structure of the folders I elected to restore is there, but empty and the MBP fans are running at top speed (or 2rpm below, but often above), but no other indication I should let things happen:

To be fair, in a Time machine restore from a local disk I can restore a 250Gb ++ VM in about 2-3 hours. Admittedly this isn’t via Dropbox over the net, but at least I have a meaningful progress bar and it does finish. I have restored my complete system (800Gb) in about 6-7 hours IIRC and finding and restoring an individual file or folder is instantaneous.

It’s saved in the database so you should only have to save it once when configuring the job. In the case that you’re restoring without a job, then it has to be configured every time, but is that case happening often?

I could be wrong, but I don’t think power is the issue, my MBP isn’t the fastest, it’s a 2015 15” Retina with 1Tb SSD, i7, 16Gb RAM, running High Sierra, but it should be adequate.

Thanks for getting back…

warwickmm · November 25, 2018, 6:00pm

Sorry to hear about your experience. You make some good points.

Can you provide the exact version of Duplicati that you’re running?

Cirrus22 · November 26, 2018, 1:50am

Hi Warwick,

Can you provide the exact version of Duplicati that you’re running?

It’s the macOS version: duplicati 2.0.3.3_beta_2018-04-02

And just an update:

The Scanning Local Blocks (or whatever it’s called) just went on and on and on for 20++ hours, fans at 100%. I finally killed the restore, there was no feedback WRT what was happening. I restarted the restore with the (well disguised) option to NOT scan local blocks.

QUESTION: I am restoring a Server 2012 R2 system to a MAC (macOS High Sierra). Why would anyone think there was likely to be any similarity and why would you choose to scan the MAC to even try and find like files?

If I restore to the same point then fine might be worth a try, but even then, the time to scan takes longer than actually doing the restore. That is some severely flawed logic. I could have restored the files I wanted in seriously less time than attempting to scan for local blocks. Why not put that option front and centre just like the preserve permissions.

After the restart of the restore, yet again tries to construct/rebuild the database. As I think I said previously, why not, when the backup is being performed do you not associate a database record that has the layout of the backup with the date/time of the backup. Then if I do a restore I can get the structure with one or two database reads without wasting hours to rebuild it?

And finally there is data being pulled down from Dropbox, frankly I just didn’t think I was ever going to get there. And now, Again, the perennial problem of providing user feedback. Apart from the fans going bezerk (and actually seeing data in the location I am restoring to) there is no indication that the software hasn’t hung/died/collapsed.

I am left with hour upon hour of this screen, nothing changes, nothing updates just nothing:

So from my perspective, great, I’m getting backup restored. On the downside, it took me several days to get here….

Thanks for taking an interest.

warwickmm · November 26, 2018, 2:39am

Does anything show up in the server logs? The Live - Profiling logs are the most verbose ones. This doesn’t replace a more informative progress indicator, but may provide some indication as to what your current restore process is doing.

https://duplicati.readthedocs.io/en/latest/03-using-the-graphical-user-interface/#viewing-the-duplicati-server-logs

Cirrus22 · November 26, 2018, 4:08am

Hi Warwick,

The actual, physical restore is finished (in about a quarter of the time it took to not finish scanning local blocks), see my point about disabling this by default and presenting an option to enable during the restore.

Does anything show up in the server logs? The Live - Profiling logs are the most verbose ones. This doesn’t replace a more informative progress indicator, but may provide some indication as to what your current restore process is doing.

I did enable logging and log level profiling, but I cannot say I have ever looked at the logs (my bad). According to the documentation it says:

If you want to see what is happening in the background in real time, click the Live button. This is disabled by default to preserve system resources. Choose one of the levels Error, Warning, Information or Profiling. Error will only display events indicating that something goes wrong, Profiling lists about every single event that occurs. Clicking on an event also reveals detailed information about that event.

And I suspect what I wanted was the ‘live’ button, but I do not recall seeing it. I have started another restore and I can see no ‘live’ option presented nor any live button and I do not see any option to view the logs. There are things that may make perfect sense if you are familiar with the software, but if a noob, then not obvious at all.

Look, thanks for the interest and support. I have my data back so I can at least try and rebuild my server (and that’s a good thing), so Duplicati indeed worked as it should (eventually). There was some pilot error, but I think justifiably so……

warwickmm · November 27, 2018, 2:55am

I’m glad you were able to eventually recover your data. Hopefully we can work out the kinks to make the process more intuitive and seamless.

There are also discussions on how to improve the presentation of the logs. For now, to get to the live logs, go to the Main page -> About -> Show log.

corn · July 31, 2021, 3:39pm

Were there supposed to be any updates to the “progress bar”? A “Progress bar” which does not show “progress” is a useless bar:

Ive left Duplicati running on my machine for 3 days trying to restore from google drive to a hard drive plugged into my machine via USB. A lot of my files have been restored to the hard drive but I need to know if ALL are restored. Without a progress bar or some kind of feedback, I don’t know if the restore really is complete. The following is the Log Data from the Server:

The "Jul 28, 2021 12:45 AM: Failed while executing “List” with id: 1 " has the following details:

Mono.Data.Sqlite.SqliteException (0x80004005): The database file is locked

database is locked
at Mono.Data.Sqlite.SQLite3.Step (Mono.Data.Sqlite.SqliteStatement stmt) [0x000a4] in <03f5dc00126848489ffe244e568a5e02>:0
at Mono.Data.Sqlite.SqliteDataReader.NextResult () [0x00129] in <03f5dc00126848489ffe244e568a5e02>:0
at Mono.Data.Sqlite.SqliteDataReader…ctor (Mono.Data.Sqlite.SqliteCommand cmd, System.Data.CommandBehavior behave) [0x00051] in <03f5dc00126848489ffe244e568a5e02>:0
at (wrapper remoting-invoke-with-check) Mono.Data.Sqlite.SqliteDataReader…ctor(Mono.Data.Sqlite.SqliteCommand,System.Data.CommandBehavior)
at Mono.Data.Sqlite.SqliteCommand.ExecuteReader (System.Data.CommandBehavior behavior) [0x00006] in <03f5dc00126848489ffe244e568a5e02>:0
at Mono.Data.Sqlite.SqliteCommand.ExecuteDbDataReader (System.Data.CommandBehavior behavior) [0x00000] in <03f5dc00126848489ffe244e568a5e02>:0
at System.Data.Common.DbCommand.System.Data.IDbCommand.ExecuteReader () [0x00000] in :0
at Duplicati.Library.Main.Database.ExtensionMethods.ExecuteScalarInt64 (System.Data.IDbCommand self, System.Boolean writeLog, System.String cmd, System.Int64 defaultvalue, System.Object values) [0x00061] in :0
at Duplicati.Library.Main.Database.ExtensionMethods.ExecuteScalarInt64 (System.Data.IDbCommand self, System.String cmd, System.Int64 defaultvalue, System.Object values) [0x00000] in :0
at Duplicati.Library.Main.Database.LocalDatabase…ctor (System.Data.IDbConnection connection, System.String operation) [0x0005e] in :0
at Duplicati.Library.Main.Database.LocalDatabase…ctor (System.String path, System.String operation, System.Boolean shouldclose) [0x00007] in :0
at Duplicati.Library.Main.Database.LocalListDatabase…ctor (System.String path) [0x00000] in :0
at Duplicati.Library.Main.Operation.ListFilesHandler.Run (System.Collections.Generic.IEnumerable1[T] filterstrings, Duplicati.Library.Utility.IFilter compositefilter) [0x00075] in <e60bc008dd1b454d861cfacbdd3760b9>:0 at Duplicati.Library.Main.Controller+<>c__DisplayClass21_0.<List>b__0 (Duplicati.Library.Main.ListResults result) [0x0001c] in <e60bc008dd1b454d861cfacbdd3760b9>:0 at Duplicati.Library.Main.Controller.RunAction[T] (T result, System.String[x]& paths, Duplicati.Library.Utility.IFilter& filter, System.Action1[T] method) [0x0026f] in :0
at Duplicati.Library.Main.Controller.RunAction[T] (T result, Duplicati.Library.Utility.IFilter& filter, System.Action1[T] method) [0x00007] in <e60bc008dd1b454d861cfacbdd3760b9>:0 at Duplicati.Library.Main.Controller.List (System.Collections.Generic.IEnumerable1[T] filterstrings, Duplicati.Library.Utility.IFilter filter) [0x00021] in :0
at Duplicati.Server.Runner.Run (Duplicati.Server.Runner+IRunnerData data, System.Boolean fromQueue) [0x00374] in <156011ea63b34859b4073abdbf0b1573>:0

The "Jul 28, 2021 12:45 AM: Request for http://localhost:8200/api/v1/backup/1/filesets gave error " has the following details:
Mono.Data.Sqlite.SqliteException (0x80004005): The database file is locked

database is locked
at Mono.Data.Sqlite.SQLite3.Step (Mono.Data.Sqlite.SqliteStatement stmt) [0x000a4] in <03f5dc00126848489ffe244e568a5e02>:0
at Mono.Data.Sqlite.SqliteDataReader.NextResult () [0x00129] in <03f5dc00126848489ffe244e568a5e02>:0
at Mono.Data.Sqlite.SqliteDataReader…ctor (Mono.Data.Sqlite.SqliteCommand cmd, System.Data.CommandBehavior behave) [0x00051] in <03f5dc00126848489ffe244e568a5e02>:0
at (wrapper remoting-invoke-with-check) Mono.Data.Sqlite.SqliteDataReader…ctor(Mono.Data.Sqlite.SqliteCommand,System.Data.CommandBehavior)
at Mono.Data.Sqlite.SqliteCommand.ExecuteReader (System.Data.CommandBehavior behavior) [0x00006] in <03f5dc00126848489ffe244e568a5e02>:0
at Mono.Data.Sqlite.SqliteCommand.ExecuteDbDataReader (System.Data.CommandBehavior behavior) [0x00000] in <03f5dc00126848489ffe244e568a5e02>:0
at System.Data.Common.DbCommand.System.Data.IDbCommand.ExecuteReader () [0x00000] in :0
at Duplicati.Library.Main.Database.ExtensionMethods.ExecuteScalarInt64 (System.Data.IDbCommand self, System.Boolean writeLog, System.String cmd, System.Int64 defaultvalue, System.Object values) [0x00061] in :0
at Duplicati.Library.Main.Database.ExtensionMethods.ExecuteScalarInt64 (System.Data.IDbCommand self, System.String cmd, System.Int64 defaultvalue, System.Object values) [0x00000] in :0
at Duplicati.Library.Main.Database.LocalDatabase…ctor (System.Data.IDbConnection connection, System.String operation) [0x0005e] in :0
at Duplicati.Library.Main.Database.LocalDatabase…ctor (System.String path, System.String operation, System.Boolean shouldclose) [0x00007] in :0
at Duplicati.Library.Main.Database.LocalListDatabase…ctor (System.String path) [0x00000] in :0
at Duplicati.Library.Main.Operation.ListFilesHandler.Run (System.Collections.Generic.IEnumerable1[T] filterstrings, Duplicati.Library.Utility.IFilter compositefilter) [0x00075] in <e60bc008dd1b454d861cfacbdd3760b9>:0 at Duplicati.Library.Main.Controller+<>c__DisplayClass21_0.<List>b__0 (Duplicati.Library.Main.ListResults result) [0x0001c] in <e60bc008dd1b454d861cfacbdd3760b9>:0 at Duplicati.Library.Main.Controller.RunAction[T] (T result, System.String[]& paths, Duplicati.Library.Utility.IFilter& filter, System.Action1[T] method) [0x0026f] in :0
at Duplicati.Library.Main.Controller.RunAction[T] (T result, Duplicati.Library.Utility.IFilter& filter, System.Action1[T] method) [0x00007] in <e60bc008dd1b454d861cfacbdd3760b9>:0 at Duplicati.Library.Main.Controller.List (System.Collections.Generic.IEnumerable1[T] filterstrings, Duplicati.Library.Utility.IFilter filter) [0x00021] in :0
at Duplicati.Server.Runner.Run (Duplicati.Server.Runner+IRunnerData data, System.Boolean fromQueue) [0x00668] in <156011ea63b34859b4073abdbf0b1573>:0
at Duplicati.Server.WebServer.RESTMethods.Backup.ListFileSets (Duplicati.Server.Serialization.Interface.IBackup backup, Duplicati.Server.WebServer.RESTMethods.RequestInfo info) [0x000a4] in <156011ea63b34859b4073abdbf0b1573>:0
at Duplicati.Server.WebServer.RESTMethods.Backup.GET (System.String key, Duplicati.Server.WebServer.RESTMethods.RequestInfo info) [0x001fd] in <156011ea63b34859b4073abdbf0b1573>:0
at Duplicati.Server.WebServer.RESTHandler.DoProcess (Duplicati.Server.WebServer.RESTMethods.RequestInfo info, System.String method, System.String module, System.String key) [0x00146] in <156011ea63b34859b4073abdbf0b1573>:0

The "Jul 28, 2021 12:45 AM: Reporting error gave error " has the following details:
System.ObjectDisposedException: Can not write to a closed TextWriter.
at System.IO.StreamWriter.Flush (System.Boolean flushStream, System.Boolean flushEncoder) [0x0000b] in <12b418a7818c4ca0893feeaaf67f1e7f>:0
at System.IO.StreamWriter.Flush () [0x00006] in <12b418a7818c4ca0893feeaaf67f1e7f>:0
at Duplicati.Server.WebServer.RESTHandler.DoProcess (Duplicati.Server.WebServer.RESTMethods.RequestInfo info, System.String method, System.String module, System.String key) [0x003bc] in <156011ea63b34859b4073abdbf0b1573>:0

ts678 · August 1, 2021, 5:46pm

Welcome to the forum @corn

Restoring files from a backup has a picture. You might not be done, but is it working or hanging?

It’s not as friendly as a progress bar, but check About → Show log → Live → Verbose for action.

You can also look for recently changed files in target. Time stamps aren’t set until end of restore.

After restoring file contents, then file attributes, it verifies file contents, then tells you that it’s done.

The Jul 28, 2021 12:45 AM errors don’t have any context. When did the current restore start?
Seemingly it ran for awhile, built at least some files, partially or completely. What’s time on latest?

Progress bar does not show meaningful information during restore #3646

No progress indiction during a restore

corn · August 3, 2021, 5:33pm

I have since tried restoring twice. On both occasions, the restore operation “finished” pretty quickly (~6 hrs) but not all files were restored (I know this because the Source size, according to Duplicati, is 79.73 GB and the directory to where I am restoring is 75.2 GB).

Last time I tried restoring (in the previous post), I clicked on the “Restore” tab and went from there with “no-local-blocks” checked and “Restore read/write permissions” checked.

These next 2 times, I left “no-local-blocks” unchecked, “Restore read/write permissions” unchecked, and I restored by clicking on the drop-down for my backup directory in the “Home” tab and clicked “Restore files …”

Restore 2:

Restore 3:

The complete logs for each restore are in the following zip:
RestoreLogs.zip (5.7 KB)

Also, out of curiosity, why do the "complete Log"s (that ive attatched) only show on the order of tens of warnings and errors when the pictures above show on the order of 10,000 warnings and errors?

ts678 · August 3, 2021, 7:59pm

“Complete log” is complete in the sense that it’s not broken out for convenience (and simplified) as in the earlier sections that are shown. It’s also complete in that it’s all that is stored in the backup job database.

If you want a complete log to scroll by, try About → Show log → Live → Warning, but you’d likely want log-file=<path> if you want all the very many warnings and errors. You might find a better pattern to the issue.

Using what you have so far (limited to 20 lines…) is a start. For folder creation errors, how deep did it go?

Failing to create the folder structure is certainly going to interfere with restoring the files into that structure.
Was this Linux backup and restore? /media/BACKUP_DIR is the USB drive, I suppose. What filesystem?

The “Invalid argument” on the folder creates sound more like path or character set issue than permission, however I’ll still ask what user Duplicati (not the web browser for access) is running as. Something failed.
About → System info → UserName shows this, if you’re not sure.

no-local-blocks should only change things if any original source paths are there at restore time. Are they?

--no-local-blocks = false
Duplicati will attempt to use data from source files to minimize the amount of downloaded data. Use this option to skip this optimization and only use remote data.

You should be able to get a better block-level view of the building of the restored files if you log at Verbose, however the log does get larger. Perhaps you can do a smaller-than-all restore or even one troubled file…

Looking closer, you did have a few errors that might be permissions rather than path issues, for example

Error message was: Access to the path \"/media/BACKUP_DIR/.android/avd/doo1.avd/read-snapshot.txt\" is denied."

How much of that path exists? Can you think of any reason why the user Duplicati uses is being denied?

corn · August 3, 2021, 9:41pm

Using what you have so far (limited to 20 lines…) is a start. For folder creation errors, how deep did it go?

What do you mean by “how deep did it go”?

Failing to create the folder structure is certainly going to interfere with restoring the files into that structure.
Was this Linux backup and restore? /media/BACKUP_DIR is the USB drive, I suppose. What filesystem?

Yes, this is a restore of my Linux laptop’s home directory. I backed-up my home directory to google drive and tried restoring it to a hard-disk drive attached via a USB cable to (the same) Linux laptop. The hard-disk drive is formatted as FAT. The original home directory, which is backed up, is in a partition that is formatted as Ext4. That Ext4 partition is part of an SSD inside of my laptop.

The “Invalid argument” on the folder creates sound more like path or character set issue than permission, however I’ll still ask what user Duplicati (not the web browser for access) is running as. Something failed.
About → System info → UserName shows this, if you’re not sure.

UserName : MY_USERNAME (redacted for obvious reasons)

I tracked down this offending directory and deleted it on my local linux machine. Seems to be some useless java preferences directory.

no-local-blocks should only change things if any original source paths are there at restore time. Are they?

The original source paths SHOULD be there since I am restoring to a hard-disk drive which is connected via USB to my Linux machine whose home directory makes up the backup files.

Looking closer, you did have a few errors that might be permissions rather than path issues, for example

Error message was: Access to the path \"/media/BACKUP_DIR/.android/avd/doo1.avd/read-snapshot.txt\" is denied."

How much of that path exists? Can you think of any reason why the user Duplicati uses is being denied?

The file example you gave (read-snapshot.txt) is owned by root and has 644 permissions. Looking back at previous back up operations I got the following warnings:

2021-08-01 20:44:14 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: /home/MY_USERNAME/.android/avd/doo1.avd/snapshots/default_boot/textures.bin
2021-08-01 20:44:14 -04 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-PathProcessingError]: Failed to process path: /home/MY_USERNAME/.android/avd/doo1.avd/snapshots/default_boot/textures.bin

I tried cd’ing into .android/avd/doo1.avd/snapshots/default_boot but as a non-root user, I was not able to.

Is there any way to run the duplicati restore and backup processes as root so we dont have to worry about file permission errors?

Also, I think I ran a database repair operation a few days ago but im not sure…

Thank you so much for all of your help. Let me know what other information I should provide to help diagnose.

ts678 · August 3, 2021, 11:17pm

“deep” means the number of folders, e.g. reading path from left to right. When did path stop working?
Actually, I can sort of see this in some of the 20 messages where it starts failing then lower paths fail.
Failed to create folder: \"/media/BACKUP_DIR/.local/share/Trash/files/cd ../\", message: Invalid argument"
might have been the first, but messages are incomplete so I don’t know if the files level got created.

FAT has character restrictions, and so might have choked on this style of (I think) Java-encoded string:

Failed to create folder: \"/media/BACKUP_DIR/.java/.userPrefs/_!()!bw\"i!'8!d!\"i!(`!a@\"s!'@!~@\"y!#4!^@\"h!'k!bg\"'!()!}@\"t!'`=/\", message: Invalid argument"

I suppose you can try getting that quoted for a shell to see if you can make such a folder name by hand.

Does that mean you deleted a current directory like the above, or it actually was created by the restore?

I wasn’t sure if there was a problem with the system. If system is alive and well and has room, restore to Linux filesystem instead of FAT and see if it does better. Linux filesystems have few forbidden characters.
Having restored (might need further work…), see if you can cp --archive or something onto USB drive.

Yes. It should be all set to run as a systemd service as root, but it will probably take some systemctl work.
Installing Duplicati on Linux covers it, but maybe not ideally according to Part of installation fails on Ubuntu.

If you know systemctl it’s probably something like systemctl status duplicati to see where you are, then maybe an enable and start if you were starting from nothing, but because you’re already running as non-root user, you probably want to stop Duplicati and move some databases. Currently they’re located in ~/.config/Duplicati and ~ will be ~root after you’re root. I would suggest DB copy rather than moves initially, because I don’t have an exact Linux procedure. You can probably do this step by step too. Copying current Duplicati-server.sqlite in your directory (after Duplicati shutdown) to root area should let you start Duplicati, however it will use per-job databases in your directory. This should work, but what if that ever gets deleted?

To move all your job databases after getting up as root, you can probably use the Database screen to edit Local database path to new ~root location, then click the newly-blue Move existing database button.

Migrating from User to Service install on Windows has a similar method. Directions also describe how to retain the Tray Icon if you have one now and like it. Because the root start-at-boot Duplicati is just a server that doesn’t put the Tray Icon on your desktop, you can still start Tray Icon, but tell it --no-hosted-server.

Another migration caveat is that if your current backup source was done in the User data section (e.g. My Documents), you might need to change that to something in Computer so that it’s not dependent on a user.

corn · August 5, 2021, 3:17pm

Ok, I now tried using duplicati by launching it from the command-line with “sudo duplicati” . Doing so allowed me to successfully create a full backup (no warning or errors) which is great. However, whats not-so-great is that I still have warnings and errors when trying to restore (see log and picture below).

I am hesitant to backup directly to my linux partition in case things get messed up, so I will reformat my harddrive as ext4 and try to restore to there.

Complete Log:
{
“RestoredFiles”: 497490,
“SizeOfRestoredFiles”: 63406912101,
“RestoredFolders”: 72941,
“RestoredSymlinks”: 0,
“PatchedFiles”: 0,
“DeletedFiles”: 0,
“DeletedFolders”: 0,
“DeletedSymlinks”: 0,
“MainOperation”: “Restore”,
“RecreateDatabaseResults”: null,
“ParsedResult”: “Error”,
“Version”: “2.0.6.3 (2.0.6.3_beta_2021-06-17)”,
“EndTime”: “2021-08-05T13:44:38.59762Z”,
“BeginTime”: “2021-08-04T15:14:46.011481Z”,
“Duration”: “22:29:52.5861390”,
“MessagesActualLength”: 1438,
“WarningsActualLength”: 66237,
“ErrorsActualLength”: 21533,
“Messages”: [
“2021-08-04 11:14:46 -04 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Restore has started”,
“2021-08-04 11:14:46 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started: ()”,
“2021-08-04 11:15:16 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed: (2.02 KB)”,
“2021-08-04 11:15:16 -04 - [Information-Duplicati.Library.Main.Database.LocalRestoreDatabase-SearchingBackup]: Searching backup 0 (8/4/2021 1:58:46 AM) …”,
“2021-08-05 03:37:21 -04 - [Information-Duplicati.Library.Main.Operation.RestoreHandler-RemoteFileCount]: 714 remote files are required to restore”,
“2021-08-05 03:37:21 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-b8ee9f977cbb8474eb6b2a5857fac09d5.dblock.zip.aes (49.94 MB)”,
“2021-08-05 03:37:26 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-b8ee9f977cbb8474eb6b2a5857fac09d5.dblock.zip.aes (49.94 MB)”,
“2021-08-05 03:37:26 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-b32a935362baa40a8981aa037806bd663.dblock.zip.aes (49.91 MB)”,
“2021-08-05 03:37:30 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-b32a935362baa40a8981aa037806bd663.dblock.zip.aes (49.91 MB)”,
“2021-08-05 03:37:30 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-bfc02f8232ea0470bb58eee236e187603.dblock.zip.aes (49.92 MB)”,
“2021-08-05 03:37:35 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-bfc02f8232ea0470bb58eee236e187603.dblock.zip.aes (49.92 MB)”,
“2021-08-05 03:37:35 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-b342a68582dd94b00b5b06ae11136edb8.dblock.zip.aes (49.91 MB)”,
“2021-08-05 03:37:40 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-b342a68582dd94b00b5b06ae11136edb8.dblock.zip.aes (49.91 MB)”,
“2021-08-05 03:37:40 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-bbced73a5c0ca4078a5f03c5d5de1d68a.dblock.zip.aes (49.92 MB)”,
“2021-08-05 03:37:44 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-bbced73a5c0ca4078a5f03c5d5de1d68a.dblock.zip.aes (49.92 MB)”,
“2021-08-05 03:37:44 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-bc690044ebc4c4fe8a85845ef55ae9706.dblock.zip.aes (49.97 MB)”,
“2021-08-05 03:37:49 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-bc690044ebc4c4fe8a85845ef55ae9706.dblock.zip.aes (49.97 MB)”,
“2021-08-05 03:37:49 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-b81781fe1765a44469c4f3a8f75bf925d.dblock.zip.aes (49.98 MB)”,
“2021-08-05 03:37:53 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-b81781fe1765a44469c4f3a8f75bf925d.dblock.zip.aes (49.98 MB)”,
“2021-08-05 03:37:53 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-b82788e5e47844312b3ec513add15d86a.dblock.zip.aes (49.98 MB)”
],
“Warnings”: [
“2021-08-04 11:16:41 -04 - [Warning-Duplicati.Library.Main.Operation.RestoreHandler-FolderCreateFailed]: Failed to create folder: “/media/BACKUP_DIR/MY_USERNAME/.local/share/Trash/files/cd …/”, message: Invalid argument”,
“2021-08-04 11:16:41 -04 - [Warning-Duplicati.Library.Main.Operation.RestoreHandler-FolderCreateFailed]: Failed to create folder: “/media/BACKUP_DIR/MY_USERNAME/.local/share/Trash/files/cd …/.metadata/”, message: Invalid argument”,
“2021-08-04 11:16:41 -04 - [Warning-Duplicati.Library.Main.Operation.RestoreHandler-FolderCreateFailed]: Failed to create folder: “/media/BACKUP_DIR/MY_USERNAME/.local/share/Trash/files/cd …/.metadata/.mylyn/”, message: Invalid argument”,
“2021-08-04 11:16:41 -04 - [Warning-Duplicati.Library.Main.Operation.RestoreHandler-FolderCreateFailed]: Failed to create folder: “/media/BACKUP_DIR/MY_USERNAME/.local/share/Trash/files/cd …/.metadata/.mylyn/contexts/”, message: Invalid argument”,
“2021-08-04 11:16:41 -04 - [Warning-Duplicati.Library.Main.Operation.RestoreHandler-FolderCreateFailed]: Failed to create folder: “/media/BACKUP_DIR/MY_USERNAME/.local/share/Trash/files/cd …/.metadata/.mylyn/.taskListIndex/”, message: Invalid argument”,
“2021-08-04 11:16:41 -04 - [Warning-Duplicati.Library.Main.Operation.RestoreHandler-FolderCreateFailed]: Failed to create folder: “/media/BACKUP_DIR/MY_USERNAME/.local/share/Trash/files/cd …/.metadata/.plugins/”, message: Invalid argument”,
“2021-08-04 11:16:41 -04 - [Warning-Duplicati.Library.Main.Operation.RestoreHandler-FolderCreateFailed]: Failed to create folder: “/media/BACKUP_DIR/MY_USERNAME/.local/share/Trash/files/cd …/.metadata/.plugins/org.eclipse.ui.workbench/”, message: Invalid argument”,
“2021-08-04 11:16:41 -04 - [Warning-Duplicati.Library.Main.Operation.RestoreHandler-FolderCreateFailed]: Failed to create folder: “/media/BACKUP_DIR/MY_USERNAME/.local/share/Trash/files/cd …/.metadata/.plugins/org.eclipse.tips.ide/”, message: Invalid argument”,
“2021-08-04 11:16:41 -04 - [Warning-Duplicati.Library.Main.Operation.RestoreHandler-FolderCreateFailed]: Failed to create folder: “/media/BACKUP_DIR/MY_USERNAME/.local/share/Trash/files/cd …/.metadata/.plugins/org.eclipse.mylyn.bugzilla.core/”, message: Invalid argument”,
“2021-08-04 11:16:41 -04 - [Warning-Duplicati.Library.Main.Operation.RestoreHandler-FolderCreateFailed]: Failed to create folder: “/media/BACKUP_DIR/MY_USERNAME/.local/share/Trash/files/cd …/.metadata/.plugins/org.eclipse.core.resources/”, message: Invalid argument”,
“2021-08-04 11:16:41 -04 - [Warning-Duplicati.Library.Main.Operation.RestoreHandler-FolderCreateFailed]: Failed to create folder: “/media/BACKUP_DIR/MY_USERNAME/.local/share/Trash/files/cd …/.metadata/.plugins/org.eclipse.core.resources/.history/”, message: Invalid argument”,
“2021-08-04 11:16:41 -04 - [Warning-Duplicati.Library.Main.Operation.RestoreHandler-FolderCreateFailed]: Failed to create folder: “/media/BACKUP_DIR/MY_USERNAME/.local/share/Trash/files/cd …/.metadata/.plugins/org.eclipse.core.resources/.root/”, message: Invalid argument”,
“2021-08-04 11:16:41 -04 - [Warning-Duplicati.Library.Main.Operation.RestoreHandler-FolderCreateFailed]: Failed to create folder: “/media/BACKUP_DIR/MY_USERNAME/.local/share/Trash/files/cd …/.metadata/.plugins/org.eclipse.core.resources/.root/.indexes/”, message: Invalid argument”,
“2021-08-04 11:16:41 -04 - [Warning-Duplicati.Library.Main.Operation.RestoreHandler-FolderCreateFailed]: Failed to create folder: “/media/BACKUP_DIR/MY_USERNAME/.local/share/Trash/files/cd …/.metadata/.plugins/org.eclipse.core.resources/.safetable/”, message: Invalid argument”,
“2021-08-04 11:16:41 -04 - [Warning-Duplicati.Library.Main.Operation.RestoreHandler-FolderCreateFailed]: Failed to create folder: “/media/BACKUP_DIR/MY_USERNAME/.local/share/Trash/files/cd …/.metadata/.plugins/org.eclipse.mylyn.context.core/”, message: Invalid argument”,
“2021-08-04 11:16:41 -04 - [Warning-Duplicati.Library.Main.Operation.RestoreHandler-FolderCreateFailed]: Failed to create folder: “/media/BACKUP_DIR/MY_USERNAME/.local/share/Trash/files/cd …/.metadata/.plugins/org.eclipse.mylyn.context.core/contexts/”, message: Invalid argument”,
“2021-08-04 11:16:41 -04 - [Warning-Duplicati.Library.Main.Operation.RestoreHandler-FolderCreateFailed]: Failed to create folder: “/media/BACKUP_DIR/MY_USERNAME/.local/share/Trash/files/cd …/.metadata/.plugins/org.eclipse.ui.intro/”, message: Invalid argument”,
“2021-08-04 11:16:41 -04 - [Warning-Duplicati.Library.Main.Operation.RestoreHandler-FolderCreateFailed]: Failed to create folder: “/media/BACKUP_DIR/MY_USERNAME/.local/share/Trash/files/cd …/.metadata/.plugins/org.eclipse.oomph.setup/”, message: Invalid argument”,
“2021-08-04 11:16:41 -04 - [Warning-Duplicati.Library.Main.Operation.RestoreHandler-FolderCreateFailed]: Failed to create folder: “/media/BACKUP_DIR/MY_USERNAME/.local/share/Trash/files/cd …/.metadata/.plugins/org.eclipse.jdt.ui/”, message: Invalid argument”,
“2021-08-04 11:16:41 -04 - [Warning-Duplicati.Library.Main.Operation.RestoreHandler-FolderCreateFailed]: Failed to create folder: “/media/BACKUP_DIR/MY_USERNAME/.local/share/Trash/files/cd …/.metadata/.plugins/org.eclipse.oomph.setup.ui/”, message: Invalid argument”
],
“Errors”: [
“2021-08-05 05:08:22 -04 - [Error-Duplicati.Library.Main.Operation.RestoreHandler-RestoreFileFailed]: Failed to restore empty file: “/media/BACKUP_DIR/MY_USERNAME/.local/share/Trash/files/cd …/.metadata/.lock”. Error message was: Invalid argument”,
“2021-08-05 05:08:22 -04 - [Error-Duplicati.Library.Main.Operation.RestoreHandler-RestoreFileFailed]: Failed to restore empty file: “/media/BACKUP_DIR/MY_USERNAME/.local/share/Trash/files/cd …/.metadata/.mylyn/.taskListIndex/write.lock”. Error message was: Invalid argument”,
“2021-08-05 05:08:22 -04 - [Error-Duplicati.Library.Main.Operation.RestoreHandler-RestoreFileFailed]: Failed to restore empty file: “/media/BACKUP_DIR/MY_USERNAME/.local/share/Trash/files/cd …/.metadata/.plugins/org.eclipse.jdt.core/index.db”. Error message was: Invalid argument”,
“2021-08-05 06:15:32 -04 - [Error-Duplicati.Library.Main.Operation.RestoreHandler-RestoreFileFailed]: Failed to restore file: “/media/BACKUP_DIR/MY_USERNAME/Documents/Development/EclipseWorkspace/SOME_LIBRARY/documentation/cpp/Filesystem_8h_source.html”. Error message was: Failed to restore file: “/media/BACKUP_DIR/MY_USERNAME/Documents/Development/EclipseWorkspace/SOME_LIBRARY/documentation/cpp/Filesystem_8h_source.html”. File hash is 3ftIi21o+zbp3qU4gqwFyo5ts/z1TXQGpuPfOGxoRLo=, expected hash is NBk03KbHmR6kuI3mO+kannYShJyVdgec7j4tUdZXjdQ=”,
“2021-08-05 06:17:51 -04 - [Error-Duplicati.Library.Main.Operation.RestoreHandler-RestoreFileFailed]: Failed to restore file: “/media/BACKUP_DIR/MY_USERNAME/Documents/Development/EclipseWorkspace/SOME_LIBRARY/usr/include/linux/netfilter_ipv4/ipt_ttl.h”. Error message was: Failed to restore file: “/media/BACKUP_DIR/MY_USERNAME/Documents/Development/EclipseWorkspace/SOME_LIBRARY/usr/include/linux/netfilter_ipv4/ipt_ttl.h”. File hash is 77iJ3g1GhVPFQqawOT0LFGXqjD1OqJrf8soTJeERJQ4=, expected hash is SmXeVnorZZwyU4sLB9pUXK8rMOLPAmwgaAwAa9lR33I=”,
“2021-08-05 06:17:51 -04 - [Error-Duplicati.Library.Main.Operation.RestoreHandler-RestoreFileFailed]: Failed to restore file: “/media/BACKUP_DIR/MY_USERNAME/Documents/Development/EclipseWorkspace/SOME_LIBRARY/usr/include/linux/netfilter_ipv4/ipt_TTL.h”. Error message was: Failed to restore file: “/media/BACKUP_DIR/MY_USERNAME/Documents/Development/EclipseWorkspace/SOME_LIBRARY/usr/include/linux/netfilter_ipv4/ipt_TTL.h”. File hash is 77iJ3g1GhVPFQqawOT0LFGXqjD1OqJrf8soTJeERJQ4=, expected hash is 5z/3XMBYvKYUldDrNCU8XN1nkIuUsGvtrkhcESnWkXo=”,
“2021-08-05 06:17:51 -04 - [Error-Duplicati.Library.Main.Operation.RestoreHandler-RestoreFileFailed]: Failed to restore file: “/media/BACKUP_DIR/MY_USERNAME/Documents/Development/EclipseWorkspace/SOME_LIBRARY/usr/include/linux/netfilter_ipv4/ipt_ecn.h”. Error message was: Failed to restore file: “/media/BACKUP_DIR/MY_USERNAME/Documents/Development/EclipseWorkspace/SOME_LIBRARY/usr/include/linux/netfilter_ipv4/ipt_ecn.h”. File hash is C+PWlt4gGoJtfHBCdUu1pwLPnDj5YKBJArlbG+B1ATk=, expected hash is RbyXh85XX8S37UuYDzxuME8lXXBqwYrU/Nd7MqV14yM=”,
“2021-08-05 06:17:52 -04 - [Error-Duplicati.Library.Main.Operation.RestoreHandler-RestoreFileFailed]: Failed to restore file: “/media/BACKUP_DIR/MY_USERNAME/Documents/Development/EclipseWorkspace/SOME_LIBRARY/usr/include/linux/netfilter/xt_CONNMARK.h”. Error message was: Failed to restore file: “/media/BACKUP_DIR/MY_USERNAME/Documents/Development/EclipseWorkspace/SOME_LIBRARY/usr/include/linux/netfilter/xt_CONNMARK.h”. File hash is p9Pl7vch/5o/irMlpTw8gftYlJNyAwUVrrWwUxoq1Xs=, expected hash is 4V2W/oDH2+mA4GIJIBYeZnr4OZM5nny7dPMJVwiOogQ=”,
“2021-08-05 06:17:52 -04 - [Error-Duplicati.Library.Main.Operation.RestoreHandler-RestoreFileFailed]: Failed to restore file: “/media/BACKUP_DIR/MY_USERNAME/Documents/Development/EclipseWorkspace/SOME_LIBRARY/usr/include/linux/netfilter/xt_rateest.h”. Error message was: Failed to restore file: “/media/BACKUP_DIR/MY_USERNAME/Documents/Development/EclipseWorkspace/SOME_LIBRARY/usr/include/linux/netfilter/xt_rateest.h”. File hash is LTvnp1mf++auFVHSw3yP2XWOguL1XpMW41q4VBmKP/0=, expected hash is J496bW8i45LviWWwF+FSOQPRQjbBjFqEE2xTicr8/Vg=”,
“2021-08-05 06:17:52 -04 - [Error-Duplicati.Library.Main.Operation.RestoreHandler-RestoreFileFailed]: Failed to restore file: “/media/BACKUP_DIR/MY_USERNAME/Documents/Development/EclipseWorkspace/SOME_LIBRARY/usr/include/linux/netfilter/xt_DSCP.h”. Error message was: Failed to restore file: “/media/BACKUP_DIR/MY_USERNAME/Documents/Development/EclipseWorkspace/SOME_LIBRARY/usr/include/linux/netfilter/xt_DSCP.h”. File hash is c+zn9Dxq3+ClU1HS15XKZWvmqpjzsJ2ZyIGf/yte9ZE=, expected hash is 1vq4LDggWt3UD0UV1MEIG1Hu++VJxpBH2n84sd/GzrU=”,
“2021-08-05 06:17:52 -04 - [Error-Duplicati.Library.Main.Operation.RestoreHandler-RestoreFileFailed]: Failed to restore file: “/media/BACKUP_DIR/MY_USERNAME/Documents/Development/EclipseWorkspace/SOME_LIBRARY/usr/include/linux/netfilter/xt_TCPMSS.h”. Error message was: Failed to restore file: “/media/BACKUP_DIR/MY_USERNAME/Documents/Development/EclipseWorkspace/SOME_LIBRARY/usr/include/linux/netfilter/xt_TCPMSS.h”. File hash is iQODhstSceaKRiI39L/4T/9Gcyd5VYjeNaDU0dWqffI=, expected hash is 2zvPSdKp9xOG+u4nvkCwIcwNNwPF1DXD5G3OXsYFlcQ=”,
“2021-08-05 06:17:52 -04 - [Error-Duplicati.Library.Main.Operation.RestoreHandler-RestoreFileFailed]: Failed to restore file: “/media/BACKUP_DIR/MY_USERNAME/Documents/Development/EclipseWorkspace/SOME_LIBRARY/usr/include/linux/netfilter/xt_mark.h”. Error message was: Failed to restore file: “/media/BACKUP_DIR/MY_USERNAME/Documents/Development/EclipseWorkspace/SOME_LIBRARY/usr/include/linux/netfilter/xt_mark.h”. File hash is dQejWzaKWlOGnCeFJ+5S3/eb/Wa0MHq9A6sCIyvKL+E=, expected hash is 6qwdMND1lanMFgqFVde47HydPcN2645tKezDJxrAKaI=”,
“2021-08-05 06:17:52 -04 - [Error-Duplicati.Library.Main.Operation.RestoreHandler-RestoreFileFailed]: Failed to restore file: “/media/BACKUP_DIR/MY_USERNAME/Documents/Development/EclipseWorkspace/SOME_LIBRARY/usr/include/linux/netfilter/xt_RATEEST.h”. Error message was: Failed to restore file: “/media/BACKUP_DIR/MY_USERNAME/Documents/Development/EclipseWorkspace/SOME_LIBRARY/usr/include/linux/netfilter/xt_RATEEST.h”. File hash is LTvnp1mf++auFVHSw3yP2XWOguL1XpMW41q4VBmKP/0=, expected hash is NeJFfwzSDB7ZJNpMjBtuy940wVF1Qm/3p8wk8lFomtI=”,
“2021-08-05 06:17:52 -04 - [Error-Duplicati.Library.Main.Operation.RestoreHandler-RestoreFileFailed]: Failed to restore file: “/media/BACKUP_DIR/MY_USERNAME/Documents/Development/EclipseWorkspace/SOME_LIBRARY/usr/include/linux/netfilter/xt_MARK.h”. Error message was: Failed to restore file: “/media/BACKUP_DIR/MY_USERNAME/Documents/Development/EclipseWorkspace/SOME_LIBRARY/usr/include/linux/netfilter/xt_MARK.h”. File hash is dQejWzaKWlOGnCeFJ+5S3/eb/Wa0MHq9A6sCIyvKL+E=, expected hash is 4GYjd6bGc+jy5vPHU5QXkkkClQouSnU6c5QMyHF4m9w=”,
“2021-08-05 06:17:52 -04 - [Error-Duplicati.Library.Main.Operation.RestoreHandler-RestoreFileFailed]: Failed to restore file: “/media/BACKUP_DIR/MY_USERNAME/Documents/Development/EclipseWorkspace/SOME_LIBRARY/usr/include/linux/netfilter_ipv6/ip6t_HL.h”. Error message was: Failed to restore file: “/media/BACKUP_DIR/MY_USERNAME/Documents/Development/EclipseWorkspace/SOME_LIBRARY/usr/include/linux/netfilter_ipv6/ip6t_HL.h”. File hash is 9tHU48A9/wQesAuqf+SL212hS9Gw5yAafd5B7uSkQpA=, expected hash is aofzvXZaHV5z3oDJKJeHKt3ouq0pU9bpM5/mqjDnLwc=”,
“2021-08-05 06:55:08 -04 - [Error-Duplicati.Library.Main.Operation.RestoreHandler-RestoreFileFailed]: Failed to restore file: “/media/BACKUP_DIR/MY_USERNAME/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share/doc/rust/html/std/self.k.html”. Error message was: Failed to restore file: “/media/BACKUP_DIR/MY_USERNAME/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share/doc/rust/html/std/self.k.html”. File hash is q3q5L74ikCFYiIO4P25HUmdsYb9+b1OT0+SEYlifT38=, expected hash is P/5Ag4ioegiYFga+35ASkM18OS/UVR9n6RBcVCsywiY=”,
“2021-08-05 06:55:09 -04 - [Error-Duplicati.Library.Main.Operation.RestoreHandler-RestoreFileFailed]: Failed to restore file: “/media/BACKUP_DIR/MY_USERNAME/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share/doc/rust/html/std/keyword.self.html”. Error message was: Failed to restore file: “/media/BACKUP_DIR/MY_USERNAME/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share/doc/rust/html/std/keyword.self.html”. File hash is zgANuOycuSg8zUtJmY7AmnqGhXdyyv/c+KclLjnPUUI=, expected hash is NRBoqlpS1ymLPWJ4ueEkbTccy++nYwAZ9o/tjDleh/I=”,
“2021-08-05 06:58:33 -04 - [Error-Duplicati.Library.Main.Operation.RestoreHandler-RestoreFileFailed]: Failed to restore file: “/media/BACKUP_DIR/MY_USERNAME/FOO/cpp/current/include/LICENSE.txt”. Error message was: Failed to restore file: “/media/BACKUP_DIR/MY_USERNAME/FOO/cpp/current/include/LICENSE.txt”. File hash is u0zG592IMxI2fegy1+K5IXs2mYTCYE7xx8yT0Xlk7GQ=, expected hash is gFrdY1HXi+gWSN09RBsJk3a7jaozqL+lsaaGoOBOJsA=”,
“2021-08-05 07:00:12 -04 - [Error-Duplicati.Library.Main.Operation.RestoreHandler-RestoreFileFailed]: Failed to restore file: “/media/BACKUP_DIR/MY_USERNAME/FOO/2020/SOME_LIBRARY/documentation/cpp/Filesystem_8h_source.html”. Error message was: Failed to restore file: “/media/BACKUP_DIR/MY_USERNAME/FOO/2020/SOME_LIBRARY/documentation/cpp/Filesystem_8h_source.html”. File hash is 3ftIi21o+zbp3qU4gqwFyo5ts/z1TXQGpuPfOGxoRLo=, expected hash is NBk03KbHmR6kuI3mO+kannYShJyVdgec7j4tUdZXjdQ=”,
“2021-08-05 07:04:08 -04 - [Error-Duplicati.Library.Main.Operation.RestoreHandler-RestoreFileFailed]: Failed to restore file: “/media/BACKUP_DIR/MY_USERNAME/FOO/2020/SOME_LIBRARY/usr/include/linux/netfilter_ipv4/ipt_ttl.h”. Error message was: Failed to restore file: “/media/BACKUP_DIR/MY_USERNAME/FOO/2020/SOME_LIBRARY/usr/include/linux/netfilter_ipv4/ipt_ttl.h”. File hash is 77iJ3g1GhVPFQqawOT0LFGXqjD1OqJrf8soTJeERJQ4=, expected hash is SmXeVnorZZwyU4sLB9pUXK8rMOLPAmwgaAwAa9lR33I=”
],
“BackendStatistics”: {
“RemoteCalls”: 717,
“BytesUploaded”: 0,
“BytesDownloaded”: 37393761314,
“FilesUploaded”: 0,
“FilesDownloaded”: 714,
“FilesDeleted”: 0,
“FoldersCreated”: 0,
“RetryAttempts”: 2,
“UnknownFileSize”: 0,
“UnknownFileCount”: 0,
“KnownFileCount”: 2071,
“KnownFileSize”: 54323144011,
“LastBackupDate”: “2021-08-03T21:58:46-04:00”,
“BackupListCount”: 1,
“TotalQuotaSpace”: 2199023255552,
“FreeQuotaSpace”: 1807278163884,
“AssignedQuotaSpace”: -1,
“ReportedQuotaError”: false,
“ReportedQuotaWarning”: false,
“MainOperation”: “Restore”,
“ParsedResult”: “Success”,
“Version”: “2.0.6.3 (2.0.6.3_beta_2021-06-17)”,
“EndTime”: “0001-01-01T00:00:00”,
“BeginTime”: “2021-08-04T15:14:46.011508Z”,
“Duration”: “00:00:00”,
“MessagesActualLength”: 0,
“WarningsActualLength”: 0,
“ErrorsActualLength”: 0,
“Messages”: null,
“Warnings”: null,
“Errors”: null
}
}

On another note, I got the following interesting (negative) value in the progress bar when I tried to backup a few days ago:

Xavron · August 5, 2021, 4:52pm

That’s definitely known and nothing to be concerned about. Its like putting ugly clothes on.

corn · August 5, 2021, 8:09pm

Ok I was able to do a full restore without warnings or errors to an ext4 partition.

I think the keys to success was:

launching the duplicati browser GUI application from CLI using “sudo duplicati” Doing this helped create a backup without warnings or errors. Previously, I started duplicati by double-clicking on the application icon in my ubuntu Activities. Creating backups like this led to file permission warnings (since I’m not the root user) which later turned into errors when I tried to restore those files.
restoring to an ext4 INSTEAD OF Fat formatted partitions. This must’ve helped with those pesky “Invalid argument” restore errors when duplicati tried creating files with filenames containing FAT-banned characters

Thank you all for your help and guidance. On a final note, is there any way (through a checksum or CRC or something) to confirm that my restored data is identical to the data that is backed-up in the cloud (google drive in my case?)

ts678 · August 6, 2021, 12:37am

There is always a check that restored files are identical to the source data at the time backup was done.
Failures of the hash check will get the sort of “Failed to restore file” “File hash” “expected hash” you got.

github.com

duplicati/duplicati/blob/265d202a1f99f12f7c5bcf8b9e0deac50ac799c3/Duplicati/Library/Main/Operation/RestoreHandler.cs#L468-L479


      
          Logging.Log.WriteVerboseMessage(LOGTAG, "TestFileIntegrity", "Testing restored file integrity: {0}", file.Path);

                                      

          string key;

          long size;

          using(var fs = SystemIO.IO_OS.FileOpenRead(file.Path))

          {

              size = fs.Length;

              key = Convert.ToBase64String(filehasher.ComputeHash(fs));

          }

              

          if (key != file.Hash)

              throw new Exception(string.Format("Failed to restore file: \"{0}\". File hash is {1}, expected hash is {2}", file.Path, key, file.Hash));

Basically, quiet is good. You don’t want to see any “Failed to restore” like the above. If you really want, you can log at Verbose level to log the “TestFileIntegrity” messages, which I think “should” name all of the files.

If you’re super ambitious, you could probably write a script to sanity-check that list against list of The FIND command with the version of interest. It will only show the approximate size though. Some sample output:

Listing contents 0 (8/5/2021 7:40:00 PM):
C:\PortableApps\Notepad++Portable\ 
C:\PortableApps\Notepad++Portable\App\ 
C:\PortableApps\Notepad++Portable\App\AppInfo\ 
C:\PortableApps\Notepad++Portable\App\AppInfo\Launcher\ 
C:\PortableApps\Notepad++Portable\App\AppInfo\Launcher\Custom.nsh (555 bytes)
C:\PortableApps\Notepad++Portable\App\AppInfo\Launcher\Notepad++Portable.ini (2.67 KB)