DB recreation takes forever

Yep, that works too. I think I saw another user talking about setting up a 2nd backup job with the sole purpose of backing up the 1st job’s database file.

If you’re storing in the cloud, that makes sense as the de-duplication will cut down on data transfer. The drawback is that if you end up needing to restore the 1st job you’ll first have to restore the 2nd job to get the DB to then use on the 1st job.

It’s not a big deal - having a small backup of just the database file(s) should restore very quickly, but it’s another step to keep in mind in case of disaster recovery.

You got the point.
I use duplicati for a 12 tb backup on google drive (4 jobs)
Adding a 5th job with only the db file make me secure with versioning…

How about the recreation speed? Why is taking so long?

Is about my 4tb film collection…and the db was approximately 8 GB…

I haven’t look specifically at the database recreate process but my guess is there’s just a lot of database processing going on. Remember, this process isn’t just restoring a file from a remote destination - it’s reading through every index file on the destination and adding the entire history of every file contents in 100k (by default) blocks of data.

With a 12TB data source chopped into 100k blocks that’s 128.8 million block records being recreated in the local database for the initial backup. Then add to that however many blocks changed with each new file or historical version of an existing file.

Once some of the more commonly used functionality has been updated and finalized (like the pending improvement in restore browsing speed) I’m hoping time can be found to review the database recreate process for performance (as well as reporting) improvements.

1 Like

Understood…
Another question…if the recreation jobs stops at half I have to restart from the beginning?

Hmm…that I’m not sure about.

My guess is that one a recreate is started a database will have been created at which point a Repair should be enough to continue from any interruption point, but @kenkendk would know better.

Edit: Note that my guess is WRONG and it appears once a “The database was attempted repaired, but the repair did not complete.” message is received a Recreate action is necessary to recover.

so we wait a answer from @kenkendk!
thx

uppp @kenkendk can you give me a feedback?

Is this variable available in the current beta release 2.0.2.1? Is there a full list of variables documented somewhere?

@Bazzu85, have a little patience - kenkendk generally only has time to hit the forums once or twice a week.


@drwtsn32, I don’t know of any list but pretty much any parameter is exposed with a DUPLICATI__ prefix (yes, two ending underscores) and any “-” replaced with “_”.

This post might help:

1 Like

I ended up putting a ‘set > filename’ in my post-backup script so I could see all the values. here they are in case others are curious. (Some values have been redacted.) This was from version 2.0.2.1.

DUPLICATI__backup_name=Default
DUPLICATI__blocksize=102400b
DUPLICATI__block_hash_algorithm=SHA256
DUPLICATI__compression_module=zip
DUPLICATI__dblock_size=50mb
DUPLICATI__dbpath=C:\Users\xxx\AppData\Local\Duplicati\HYVSNQPDJL.sqlite
DUPLICATI__disable_module=console-password-input
DUPLICATI__encryption_module=aes
DUPLICATI__EVENTNAME=AFTER
DUPLICATI__file_hash_algorithm=SHA256
DUPLICATI__LOCALPATH=xxx
DUPLICATI__main_action=Backup
DUPLICATI__OPERATIONNAME=Backup
DUPLICATI__passphrase=xxx
DUPLICATI__REMOTEURL=xxx
DUPLICATI__RESULTFILE=C:\Users\xxx\AppData\Local\Temp\dup-604d7ae1-6ace-4aca-ae2a-2fc05a4f8ef5
DUPLICATI__run_script_after=C:\Users\xxx\AppData\Local\Duplicati\BackupDb.cmd
DUPLICATI__send_mail_from=xxx
DUPLICATI__send_mail_level=Success,Warning,Error,Fatal
DUPLICATI__send_mail_password=xxx
DUPLICATI__send_mail_subject=Duplicati %OPERATIONNAME% report for xxx-%backup-name%: %PARSEDRESULT%
DUPLICATI__send_mail_to=xxx
DUPLICATI__send_mail_url=xxx
DUPLICATI__send_mail_username=xxx
DUPLICATI__snapshot_policy=On
DUPLICATI__throttle_download=
DUPLICATI__throttle_upload=
DUPLICATI__usn_policy=On
1 Like

Thanks for that list!

However, I should note that the list generated from set won’t include ones with no value, so what you got is probably everything you have values for in the particular job on which you ran the set command.

So there are a lot of optional/advanced/less-commonly-used parameters not represented in the above list.

Good to know! There were a few listed that had no values set (throttle up/down for instance).

So my worries about recreation stop at half work are reality…
Today my server stop the process for a power failure…

Trying to start a repair give error…

What can I do?

Apparently my guess above was wrong and you will indeed have to start over. :frowning:

My convenience is to start over with a new job for the 4tb film backup…and doing a database backup every time i manually complete the job for fix the db…

Unfortunately yes… The recreate is not clever enough to figure out it could just continue.

Sad but true… and this time I got way behind :cry:

I resolved re-starting the backup from scratch…
and doing a backup of all my db every day…

I do the same thing! A “DB’s backup” right after the backup runs…