Check rebuild DB operation progress

1

Hi,

Following my issues with latest canary version, I was looking to downgrade back to 2.0.2.
It imply DB schema changes, and so DB need to be rebuild.

The DB rebuild operation has begun 3 days ago on my B2 backup set. It seems it,s still running, but not sure of the progress.
My B2 account show thousands of daily class C transactions, and the sqlite seems to continue to be updated.

But the sqlite file is only 40MB for now, vs 700MB before the downgrade + delete.

I can’t get logs via the GUI due to:

The database file is locked, database is locked

  • Is the rebuild operation still in progress ?
  • How long should it be ? my backup set is around 400GB
  • How to get logs of the operation ? with the cli ?

I would like to fix it, and not restart once again the backup from scratch :roll_eyes:

Rgds,

2

Yeah, the logs being logged while jobs are running is kind of a pain.

Is it running?

Have you tried using the main menu “About” → “System info” page and scrolling down to “Server state properties” where you should find the most recent commands dynamically appearing in the lastPgEvent line? That should at least confirm that the rebuild is in progress and not hung.

When will it be done?

I did a rebuild test a few months ago (My experience with Database Rebuild / Recreate) and basically found that on an 835MB sqlite database that took 42 minutes to REPAIR I had to way 4 days, 5 hours, 37 minutes for a RECREATE (and this is all via local SFTP).

If you look around you’ll see other users with week long recreate times - so yes, rebuilds / recreates are known to be slow sometimes (though we haven’t narrowed down exactly why yet).

How do I see logs while it’s running?

At present Duplicati doesn’t send much logging data to the console by default. Pretty much your only option is to start over and include the --log-file parameter (possibly with --log-level=profiling for pre-2.0.3.2 or --log-file-log-level=profiling for 2.0.3.2+) then just look at the log file.

Definitely the preferred way to go. :slight_smile:

3 
lastEventId : 5527
lastDataUpdateId : 7
lastNotificationUpdateId : 2
estimatedPauseEnd : 0001-01-01T00:00:00
activeTask : {"Item1":3,"Item2":"1"}
programState : Running
lastErrorMessage :
connectionState : connected
xsfrerror : false
connectionAttemptTimer : 0
failedConnectionAttempts : 0
lastPgEvent : {"BackupID":"1","TaskID":3,"BackendAction":"Get","BackendPath":"duplicati-i45278c51997043339cae78c20248ec60.dindex.zip.aes","BackendFileSize":44541,"BackendFileProgress":0,"BackendSpeed":-1,"BackendIsBlocking":false,"CurrentFilename":null,"CurrentFilesize":0,"CurrentFileoffset":0,"Phase":"Recreate_Running","OverallProgress":0.329290926,"ProcessedFileCount":0,"ProcessedFileSize":0,"TotalFileCount":0,"TotalFileSize":0,"StillCounting":false}
updaterState : Waiting
updatedVersion :
updateReady : false
updateDownloadProgress : 0
proposedSchedule : []
schedulerQueueIds : []
pauseTimeRemain : 0

And now to lastEventId : 5528, seems running.

Last time, B2 had issues last time; so might explain why it hanged.

Why the rebuild process is so slow ? Just need to cross dlist from remote and local files ?

And how may we end with more than 113,230 daily transaction (Class C Transactions on B2) for only 15.000 remote files ? Not used to what are those class C transactions…
Sound’s like something is looping forever.

4

After 6 days, the rebuild operation just disapeared, DB clearly not rebuilded, and the backup set not working.
There is even no logs from all the operation after May 16, which doesn’t help.
The only working point is the increased bill for B2 due to class C transactions.

I don’t know what other options are available to restore the backup, that’s quite disappointing :disappointed_relieved:

5

That’s very odd - usually Duplicati has SOMETHING to report about failures unless the computer was powered down / rebooted (which has been known to happen with Microsoft’s annoying automatic updates).

Most likely this is due to some less-than-optimal database designs/processes. Some of these are being addressed, but aren’t currently ready to go live.

Unfortunately, I don’t know much (anything) about B2 transaction types so between that and the rebuild issue I’m going to see if maybe @Pectojin has some thoughts.