Timespan to rebuild a database?

Support
21

Hello @it-ruedi59

I don’t know if you are still here, but if your restore is still going on, I think now that you should consider disaster recovery. I am currently struggling to decode some hairy code in Duplicati and I’m beginning to suspect that some corner case, namely huge files and missing index files, may not have been regression tested and so could be bugged currently.
As disaster recovery bypasses completely index rebuilding, it may have a better chance of completing before the end of time. Hope this helps a bit.

1 Like
22

I don’t know the timeframe desired but I assume fast is desired, so I agree prior path looked real slow.

Disaster Recovery will suggest several methods. Restoring files if your Duplicati installation is lost is a Direct restore from backup files which still builds a database, but just for the latest version, so should move a little faster and might possibly avoid some problem only present in some older backup.

This temporary DB is not accessible for regular use, so is a poor way to go if one wants to continue to backup. I think one can get a permanent one-version database with --version on a repair command.

Your database recreating problem can of course be avoided if you can get the old one, or source files.

Duplicati.CommandLine.RecoveryTool.exe has no database, but does take steps to build its own index from the dblock files that it will download. Needed space is nearly the size now needed at destination.

I’m not sure how fast it runs on a big restore, and I’m not sure how well it handles a cross-platform one.
If it runs out of memory for this big backup, there appears to be a --build-index-with-files option.
Build RecoveryTool index in memory by default #4061

Somebody found it doesn’t restore empty files. That’s been fixed by gpatel-fr but is not in a release yet.

Further out are several tools that don’t use Duplicati code at all (so no need for .NET Framework, etc.).
Independent restore program is an older one in Python.

0-byte files are not restored with RecoveryTool #4884 mentioned a new variant of an older tool in Rust.
Rust Duplicati Restore and I have no experience with either Rust tool, nor do I know how they operate.

1 Like
23

Yes, I’m still alive. The restore process is also still alive but slow as a snake is a formula 1 racer. I will cancel it soon, because take a look here to a snippet from the Verbose Live Log:

Bildschirm­foto 2023-01-31 um 18.35.12
Bildschirm­foto 2023-01-31 um 18.35.121932×1006 123 KB

Once apon in time it will be finished. Maybe gpatel-fr is right with “before time ends” :joy:
His other joke was “hairy code”…I’m loosing my last hairs in the last days.

This was the first thing I tried. This one is slow, too, and I saw somewhere in the forum or support that the temporary build database will be thrown away after the restore is completed. For that I decided to import the configuration hoping the process will run many more faster.

A Disaster Recovery using Duplicati will end in a disaster, that’s a sad fact.

And I could damn the so-called “IT-specialist”, who thought a disaster will never happens. His testing area was this poor customer, he never tested that scenario before. The complete setup of the server was (maybe) useful for a “children’s room datacenter” but not for a company’s professional needs.

I’m very angry about that and feel a little bit of helplessness.

I’m not angry about Duplicati and the team behind.

The thing I’ve done is a repair of the structure of the “damaged” ext4 filesystem, expand the volume a bit and restore some missing or damaged files. Next step is to clean up non-NTFS-chars railing blanks and dots -the original source was a macOS Server -which has had no trouble to allow “forbidden” chars- from folder- and filenames.

In the next days a new server hardware will arrive and then we will copying the data from the Debian Linux to a VMware based Windows Server 2019.

All in all it was a high-costly and hard way of learning things about server setups (how not). but nevertheless some kind of learning some interesting things. About machines. About human.

1 Like
24

I don’t see what could make you think so. The temporary database used in this case is useless after the restore, but that should not be your first concern.
Duplicati will restore what is possible to restore, it’s slower than a restore with a functional database, but not that slow, there is no risk of quadratic time. You will not be able to use the backup after that that’s true, but data will be recovered.

1 Like
25

Beware that this tool works but does not restore the file attributes so it’s a desperate last measure. If needed I can provide the necessary changes, though. I did not submit a PR because it’s not exactly a pressing concern, I did not test them very seriously and the current state of PR merging is what you know it is.

1 Like
26

For sure. But not with such a large backup in a day or two. Rebuilding the database took 2 days now and got 8 volumes done of 1044. The only thing what I could test is to copy the original Duplicati database from the Debian server to the Windows 10 PC, which hosts the Duplicati software.

1 Like
27

Not a fact. I test database recreate each hour. Excessive, but I look for bugs.
More reasonable would be at least one DR test, and I’m sorry you had none.
That would have given some idea of DR procedures and time (barring bugs).

Good practices for well-maintained backups has my thoughts on that issue…

A mission-critical system should IMO have multiple backups, not simply one.
Any software may have bugs, and Duplicati is still in Beta partly due to them.
This is somewhat aside from the scaling-to-large-backups issue that was hit.

Your main scaling problem is the --blocksize is giving far too many blocks,
which is an open issue linked above, but we’re not sure how to tell everyone.
Some could have been found in testing, but DB recreate testing wasn’t done.

You “might” have lost a dindex file, which caused dblock downloads to occur,
however a backup initially done on an earlier Duplicati might have other ways.

Sorry about the rough ride. I think Duplicati is OK for small less critical backup
where price might also be an issue, but sounds a bad choice for your usage…

It’s still a database recreate, so has risk of a scaling issue. How did below go?

Did it finish the partial temporary database creation, then was slow in restore?
Alternatively, if slow in DB recreate, then no need to time a --version=0 repair.
There’s also the normal-slow in the first 70% (also improvable with blocksize).
This part worked in the full database recreate, then got slower in the last 30%.

After DB (permanent or temporary), restore will still be slower than a file copy.
There’s a lot more work to do, building the files back up from individual blocks
where the advantage of that is that it’s a compact way to store many versions.

That would completely bypass the need to recreate. You can try it if you like…

1 Like
28

For your understanding, I’m not a newbie to this business.

Please don’t understand me wrong. It’s not about you or the team behind Duplicati, I’m angry about so-called “IT-Specialist” who installed and configures the backups. I’m the one who bites the dogs because I had troubles with the setup for a 3TB large data-mountain. :disappointed_relieved:

My hope was to restore/recover the damaged server in less than 48 hours.

You’re absolutely right! The customer has multiple backups: on a NAS and on S3-Storage, planned was a fullbackup every weekend. All made with Duplicati.

Basically no problem. If you have anytime seen a Mainframe from Siemens or IBM, you will know a little bit more about rough rides.

Yes, this is the reality - Duplicati is well done for smaller business and I’ve seen companies with an amount of 300 up to 500GB of files for which Duplicati would be a well-selected choice.

This is not new to me, this is normal. But the full-backup takes 5-6 hours, a restore should be done in approximately 15 hours (1 to 3). Or not?

I will do so and let you know what happens.

2 Likes
29

You can watch live log detail (but not as excessive as Profiling) at Verbose level, watching it make files, downloading a dblock file and then dropping blocks from it wherever some restored file could use them.

Watching or logging at Warning level would be good because there’s a chance something may happen.

If you want a minor speedup, use no-local-blocks to stop its hoping it can grab blocks without download.

Duplicati will attempt to use data from source files to minimize the amount of downloaded data. Use this option to skip this optimization and only use remote data.

1 Like
30

This means, I could/should set this option before start a restore job?

1 Like
31

Very optional, but without it it will spend time looking for old source paths. I assume you have none.
Because this is a regular restore, you can just put it into job Advanced options along with any log.

If I’m wrong about your restore system being empty of original files, then you’d probably NOT add it
because block-stealing locally might be worth it. This is more useful for version fixes than all-empty.

EDIT:

I called the speedup “minor”, but that’s just a guess on how many source files you have to check for.
We’d probably like to see some file blocks get written soon, so this is one step that one can remove.

EDIT 2:

There also an unavoidable step of checking existing target files in hope there’s just a tweak needed.
You’ll possibly notice some or all of these phases go by in the status bar at the top of the web page:

EDIT 3:

The next-to-last Verify is a SHA-256 hash check against the original backup, so it should be reliable.
Unfortunately, your previous work has been stuck in the next-to-first step working on database build.
We’d like to get past that and into some of the other interesting parts that hopefully will be smoother.

Thinking about the local block check again, database has Linux paths, so Windows won’t find them.
Migrating from one OS to another is a different hurdle. Let’s just try to get some files back right now.

Attribute setting is at the end, so during restore you’ll see bits and pieces of files being reassembled.
Actually, you’ll probably not get attribute setting because attributes are Linux and you’re in Windows.

Regardless, it will be nice to see it get a little further while you’re waiting for new equipment to arrive.

EDIT 4:

If there are any particular folders you want first, you can do them, if restore hasn’t gotten too far now.

1 Like
32

Hello again @it-ruedi59

If you could be so good to access at some point the database recreated by the attempt at restoration, and report the result of the following query:

select count(*) from blocklisthash;

it could be helpful (I have to stress that you are not owing me anything)

The reason is that I am really surprised by the high number of blocks that are handled in this wretched loop (1044) and I wonder if all the entries in blocklisthash are included or only a part, if all are concerned it could be some problem in code handling, if only a part, possibly more something really bad on the backend.

1 Like
33

The restore is cancelled - no progress while rebuilding the database. I found another way to recover the files. I had a working folder on a NAS to clean up filenames which doesn’t meet Windows criteria (like slashes, dots, blanks at the end and many more) and newer files from a regular Duplicati backup, which I could recover on the “repaired old” Debian server.

The new equipement is already here and the restore was startet on a Win10Pro VM - first goal was to restore the current project files.

For the moment I’ve a lack of time to support your request. After the new server is filled with data I could do so. I hope you understand this priority. But I’ll try to support your needs and wishes - it’s interest only.

1 Like
34

As a comment on this thread that could be another way to argue that Duplicati is not reliable, I have tried on a toy backup I use to test the software to delete all index files.
It was not a lot of files (9) and I got after running repair 8 messages
Pass 2 of 3, processing blocklist volume xx of 8
In this case, the message was xx of 1044.
It would be very interesting to investigate the backend state in this case.
With my toy disaster, the following query:

select type, count(*) from Remotevolume group by type;
Blocks|8
Files|4

shows clearly the problem (no index file, should have one for every block). What would this query show in this particular problematic case could be also interesting.

1 Like
35

Creating a bug report is a nice way to be able to investigate at leisure, but sometimes people won’t produce them (perhaps too difficult or too worrisome?). There might also be a 4 GB .zip size limit.

  --zip-compression-zip64 (Boolean): Toggles Zip64 support
    The zip64 format is required for files larger than 4GiB, use this flag to
    toggle it
    * default value: False

Alternatively, sometimes people will be able and willing to run queries, e.g. in DB Browser for SQLite.

To build on that idea, one can look at IndexBlockLink table. In my non-authoritative view, ideal is a match between Blocks (dblock files), Index (dindex files) and rows in IndexBlockLink table that pairs them, however adding a qualifier that Remotevolume shows State as Verified is safest, as sometimes others creep in. For example, I think Deleted is by-design there for awhile after a delete, which are often done.

Things that make you go, “Uh-oh.” kicked off a nice discussion, but someone would need to do a deeper code dive to figure out which which sort of mismatches (if any) cause immediate pains (e.g. in Recreate), and which can set the stage for later ones via some weird indirect mechanism. Because my code diving and SQL working are not the best, I run a DB recreation after hourly backup, and monitor its result code.

If it goes bad, I keep an enormous Profiling log and 30 versions of the database to help trace its history, however even a new proven “badness” check would be a nice enhancement to DB verifications arsenal, attempting to get early warning of cases that might want to read dblock files when recreating database.

Thanks for poking at this. There are also some pre-poked open issues that are in the bug fixing backlog.

1 Like
36

I had some more fun with it. Without intending it, I used an unrelated database for my toy recovery and ran repair. Well, result bad. Deleted several block and list files. After I deleted on purpose index files, there was not much left to save. I hope you don’t think I’m complaining, just exploring (not on purpose this time) how doing very moronic stuff can be destructive for Duplicati. Using a bad database and running repair seems to be very destructive. As in ‘near total destruction’ :slight_smile: . Possible room for improvement if there were not more important problems to solve.

Currently still thinking about this case, I think that if (as it seems to be) lots of index files were not available, that means that, given that the remote block size was raised to 150 Mb and the total size was 3 TB, it could mean that indexes were spread on quite a lot of big data files (with parallel uploads info is not contained in one file, it’s put on all 4 files equally) and so to recreate the necessary information, maybe a very significant part of these 3 TB was needed to recreate the missing info in the database. This could already be a significant part of the time wasted, and if I understand correctly the MV was rather underpowered to boot so the network performance could suffer. Trouble is, to recreate the database you need maybe 2 TB, but after that, to restore data this data needs to be downloaded again. Index files matter a lot for performance even if they are functionally redondant.

1 Like
37

There are a few milder-than-that ones in the Issues of repair with a stale database from an image restore or with a two machine use case. Neither is legal, but prevention and detection could be improved. Until a better way is devised, I suggested looking at dlist files. Anything impossibly new per DB is a red flag…

I don’t know if it would have to be lots, but someone needs to read C# and SQL (or test…) to decide that.

Uploading is after making of files, so shouldn’t influence packing of dblock. I don’t think I follow this idea.

Right, they’re what allow database recreation without reading dblock files to figure out where things live.

index-file-policy is a configuration control, but by default the dindex has not only a list of block hashes in associated dblock, but also contains, in list folder, redundant blocklist information to avoid dblock read.
The dlist file doesn’t enumerate all the blocks for a file. For multi-block file, it indirects through a blocklist, some of whose blocks might be in a different dblock file (deduplication). Recreate has to find the blocks.

Processing a large file in blocklists: gives an example of a blocklist hash, used to locate the blocklist, which lives in a blockset. Source file data and metadata are also blockset uses, and blocks are in dblock.

Database rebuild was an attempt to try to sort this out. Channel Pipeline was too, but with different focus.

1 Like
38

Thank you for all the toughts, hints and tipps and this very enlightening discussion.

Earlier in this discussion ts678 had an idea to copy the existing Duplicati database to the new Windows destination - but it could end in trouble and tears :wink:

Now my intension is to setup a standalone Debian Linux with Samba on board, but not attached to an Active Directory, only for restore proposes. I would then copy the ~user/.config/Duplicati from the “old” server to this new one and importing the belonging configuration file …

This server would only be the restore target with much less HDD resources and maybe 8GB of RAM. What do you think about that? Could it work?

By the way: the RAM-Size…before the crash the Debian Server had 16, then 32 and at least 64GB of RAM, because Duplicati Backup or Restore takes every RAM it got and didn’t release it. For sure after the restore - would you have a look into the code why this happens? After a reboot RAM size was normal at 3-4GB with the Duplicati Server running on it.

1 Like
39

I am still struggling to understand the recreate database code base, but what is certain is that the current code has no good performance to handle a missing index file, and worse information necessary is not reused for a subsequent missing index file. In short, current code is not fit for a restore when there are a lot of missing index files, and that seems to be exactly your case (that’s why I asked a database query on the remotevolume table, to confirm it).

If this is the case, you will not get a decent recovery time with standard restore, only disaster recovery will do.

1 Like
40

Not my words, but thanks for the smiley. You did take the better-proven path of a same-OS restore.
Changing source OS shows how it looks putting a Windows DB on Linux. Other way “should” work.

Don’t expect any special treatment for the NTFS ACL setup. There’s no mapping from Linux perms.
Based on the above test, it seemed like at least modification timestamps got set going in other way.

For only doing a Restore, you don’t even need to get more than a dummy job with right Destination.
Copy the database in and go, but take care not to do a backup or other destination-changing things.

I’m not sure I follow the situation. You got new large (?) server. Is small Debian one for initial restore?
Are you trying to make a way to get a few older files if needed from old maybe-damaged Destination?
Knowing the scale of your restore ambition might help explain my following confusion over the config.

Why would 8 GB be enough now if 64 GB wasn’t enough before. Was growth long-term or very fast?
Regardless, nobody’s going to be able to find it short-term even if explained, but it sounds pretty odd.
There’s also a chance that Windows behavior will differ because it’s using .NET Framework not mono.

1 Like