Temp database created during a restore - anyway to save that?

I have a restore that is still downloading dblock files after almost 3 days. I know it is creating a temp database for the restore process to use (since I’m restoring ‘*’). Once I finish the restore, I’d like to keep the database so that if a failure happens again, it doesn’t take 3 days to build up another temp DB. The repair command takes a --dbpath option, but the restore does not. Since the first part of the restore is building up the DB, I sure would like to know how to preserve that time and work.

A database (not just this one) shouldn’t be reused unless it’s been kept current with the backup it’s with.
Duplicati keeps its database current as it does backup and such, but I’m not sure that’s the design here.

Ordinarily one would have all operations use the same database. A restore does accept --dbpath option, which you can use after you do a repair with --dbpath option. Using implicit dbpath for all is also possible.

Using stale database can hurt restore due to new or moved data. Repair may delete newer backup data.

This exercise was for the first rebuild in your canary scenario (which can take days), before removing the dindex files and rebuilding again.

I had a backup that was in the middle of rebuilding under the beta when you showed me your dindex solution and I wanted to preserve that time already spent.

Your dindex instructions worked great on linux, but could not get the same results trying it under windows.

Here’s my rather lengthy attempt:

*** First I made a copy of my good DB, results shown here. ***

C:\ProgramData\Duplicati\data>dir

Directory of C:\ProgramData\Duplicati\data

02/24/2021 02:00 PM 9,269,248 xxx.yyy.sqlite

02/24/2021 02:00 PM 9,269,248 xxx.yyy.sqlite.original

02/24/2021 03:47 PM control_dir_v2

02/25/2021 03:03 AM 110,592 Duplicati-server.sqlite

*** Then I moved the .dindex files (not shown) and checked that they were no longer where they would be expected ***

W:\xxx.yyy\files>dir .dindex

Directory of W:\xxx.yyy\files

File Not Found

*** Then I ran the repair command, specifying the good DB. As I understand your previous post, this should

have rebuilt the .dindex files using the good DB ***

C:\ProgramData\Duplicati\data>“c:\program files\duplicati 2\duplicati.commandline.exe” repair “webdavs://safeskies1.ddns.net/remote.php/dav/files/xxx.yyy?auth-username=xxx.yyy&auth-password=xxx.yyy” --dbpath=c:\programdata\duplicati\data\xxx.yyy.sqlite --passphrase=xx --console-log-level=profiling --console-log-filter=-.Database. --number-of-retries=1 --retry-delay=1s

C:\Program Files\Duplicati 2\win-tools\gpg.exe

The operation Repair has started

Starting - Running Repair

Starting - RemoteOperationList

Backend event: List - Started: ()

Listing remote folder …

Backend event: List - Completed: (170 bytes) ***** looks like the backend list is not working….this should be a bigger number, correct? *****

RemoteOperationList took 0:00:00:00.572

Destination and database are synchronized, not making any changes ***** regardless of the backend list issue, how can the DB and destination think they are in sync? *****

Starting - RemoteOperationTerminate

RemoteOperationTerminate took 0:00:00:00.000

Running Repair took 0:00:00:01.904

Unhandled Exception: System.AggregateException: One or more errors occurred. —> System.IO.IOException: The pipe has been ended.

at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)

at System.IO.FileStream.FlushOSBuffer()

at System.Threading.Tasks.Task.Execute()

— End of stack trace from previous location where exception was thrown —

at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

at System.IO.StreamWriter.d__68.MoveNext()

— End of stack trace from previous location where exception was thrown —

at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

at System.IO.StreamWriter.d__55.MoveNext()

— End of stack trace from previous location where exception was thrown —

at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

at Duplicati.Library.AutoUpdater.UpdaterManager.<>c__DisplayClass58_1.<b__1>d.MoveNext()

— End of inner exception stack trace —

at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)

at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)

at Duplicati.Library.AutoUpdater.UpdaterManager.RunFromMostRecentSpawn(MethodInfo method, String cmdargs, AutoUpdateStrategy defaultstrategy)

at Duplicati.CommandLine.Program.Main(String args)

C:\ProgramData\Duplicati\data>

*** this whole process ran about 3 seconds and did nothing other than stack trace. It did say that destination

and database are synchronized, which isn’t possible since all of the .dindex files are missing ***

I’ve never been any good at reading windows stack traces, so maybe you can shed some light on this one.

Thanks

Rick

The number is possibly fine (you would have to look), but I think the unit should be files. Generic printer.

How does what looks like Windows Command Prompt operation below to describe and prove “missing”:

affect what Duplicati sees when looking at what I thought was a Nextcloud server which is not Windows?

Can you provide a URL or subject line for that? I can’t find it. Please try to supply context for every topic…

Thanks for responding

The webdav reference does indeed point to a nextcloud installation on safeskies1.ddns.net which is a machine on my local network. The linux path is /media/nc-data/xxx.yyy/files. I’m also running samba so the ‘W’ drive shown in my windows command box points to the exact same location as does the webdav command. Sorry for that bit of confusion.

The dindex hack was actually from Doc Watson here: https://forum.duplicati.com/t/very-slow-database-recreation/10827/8 I wasn’t paying attention initially who was responding – sorry.

In regards to the 170 bytes, a working repair for the same account also shows 170 bytes (don’t know where this number is derived from), but then goes into “rebuild database started, downloading 17 filelists” whereas the failure goes straight to “Destination and database are synchronized, not making any changes”

Thanks

Rick

1 Like

Then a next question is what’s webdav listing? Nextcloud is apparently not a simple server, such as an FTP server might be, reporting directly from the filesystem you altered. It uses a database to track files.

How to make nextcloud aware of added files possibly also applies to making it aware of removed files?

How do files look if you look in some Nextcloud web UI, use a third-party WebDAV client, use Duplicati’s Duplicati.CommandLine.BackendTool.exe, or find a job database to click a Show log → Remote list?

As mentioned, 170 bytes should say 170 files, and I think it’s whatever Nextcloud is reporting as present.

And (drum-roll please!) the backend tool exposed my issue – the index files will still being listed by the webdav frontend. Did the research on the occ command, ran a scan and the files did indeed disappear from the backend tool list. Running repair is now doing what it was supposed to, rebuilding the dindex files per the Doc’s suggestion.

So the only question I have left is the stack trace that seems to appear after most (if not all) windows commands – even the backend command did the full list and then stack traced:

C:\Program Files\Duplicati 2>duplicati.commandline.backendtool list “webdavs://safeskies1.ddns.net/remote.php/dav/files/bruce.rick?auth-username=bruce.rick&auth-password=bruce.rick#9mm”

Name Dir/File LastChange Size

duplicati-20200513T194440Z.dlist.zip.aes File 5/13/2020 2:41:09 PM 231.50 KB

.

.{bunch o’ lines eliminated for brevity}

.

duplicati-if19183afca9d4f298d44fc8a14e92aa8.dindex.zip.aes File 5/13/2020 2:32:17 PM 44.65 KB

duplicati-if408051ddec2439585885700a33552c1.dindex.zip.aes File 5/13/2020 1:12:23 PM 51.09 KB

duplicati-if623f01c2a1a460ea9a8ddd529c9dc33.dindex.zip.aes File 5/13/2020 2:06:15 PM 50.90 KB

duplicati-verification.json File 2/25/2021 2:00:48 PM 34.89 KB

Unhandled Exception: System.AggregateException: One or more errors occurred. —> System.IO.IOException: The pipe has been ended.

at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)

at System.IO.FileStream.FlushOSBuffer()

at System.Threading.Tasks.Task.Execute()

— End of stack trace from previous location where exception was thrown —

at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

at System.IO.StreamWriter.d__68.MoveNext()

— End of stack trace from previous location where exception was thrown —

at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

at System.IO.StreamWriter.d__55.MoveNext()

— End of stack trace from previous location where exception was thrown —

at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

at Duplicati.Library.AutoUpdater.UpdaterManager.<>c__DisplayClass58_1.<b__1>d.MoveNext()

— End of inner exception stack trace —

at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)

at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)

at Duplicati.Library.AutoUpdater.UpdaterManager.RunFromMostRecentSpawn(MethodInfo method, String cmdargs, AutoUpdateStrategy defaultstrategy)

at Duplicati.CommandLine.BackendTool.Program.Main(String args)

Since the stack trace does not appear to be affecting the completion of the commands, it’s more a curiosity for me than a life-or-death.

Thanks again for pointing me in the right direction!

Rick