2.0.4.5 upgrade - but now scheduled backup missing


#1

Win10 Pro, Running the BETA releases, standard install (not service).

Backups go to OneDrive (Office365 Home package)

Backups seem to have been working well. Three PCs in an office.

Today I get around to doing the updates of the application. So I quit the application, download and install on top. All works well on two out of three computers

Update settings to OneDriveV2. Backups have resumed.

BUT I now get to the last computer… updated the software… and now when I go to start it there are no backups listed as scheduled.

Is there a simple way I can rebuild the old backup to let it continue? And no, I have not exported any config.

Are there any settings I can re-import? Everything is still sitting in OneDrive. And in *%APPDATA%\Local* I see I have a backup 20190116063617.sqlite which I assume is from this new isntall. I also see a 85,600KB ICBYVMIOZA.sqlite file which I guess is relevant.

Can I rebuild? Or is this the lesson to export the configs?

(To be honest, it won’t be too much hassle to delete and restart this one backup… but would be good to learn how to fix)


#2

There might not be a rebuild of the backup itself to do, depending on whether a backup was attempted after Duplicati’s mysterious update issue. You can check OneDrive file dates, and the date on ICBYVMIOZA.sqlite.

Configuration information is almost entirely in Duplicati-server.sqlite, not in the backup (or its own database).

You can’t let it continue on OneDrive (unless Microsoft brings it back), but you could downgrade then try the upgrade again, or at least downgrade and see if you can get Duplicati to output config info, e.g. from Export.

Downgrading / reverting to a lower version talks about sqlite backups made before database version upgrade (which you get with 2.0.4.5), how to downgrade by identifying and restoring the backup databases (which also raises consistency issues with the destination, if backups have been run, unknown to the restored database).

If you decide to downgrade temporarily by doing something like moving your update folder elsewhere (see the above document for where the upgrade might be), then your now-older Duplicati should refuse to operate with newer database formats. You can use this to your advantage because I think it will stop an actual backup run, while letting you look at the job configuration (which is in Duplicati-server.sqlite normally, and maybe in one of the smaller sqlite files with the backup name – anything big is probably from a former backup job, not server).

Basically, the 85 MB file is probably the job database, and if backup 20190116063617.sqlite is a few hundred KB, it might be your former Duplicati-server.sqlite which you could try using as that (after renaming current), if you decide to try to downgrade, start Duplicati Server or Tray Icon, and note (or maybe even export) your job.

Looking into the current Duplicati-server.sqlite (if there is one) might shed light on what went wrong, but would need --unencrypted-database set on the Server or TrayIcon before DB Browser for SQLite or similar can look.


#3

I’ll read through those downgrade options. See if I can dig out of this.

You are confusing me a little though with this statement:

Don’t know if I follow you on that bit. The OneDrive to OneDriveV2 change has worked perfect on the other two computers. Hardly even a hiccup as I swapped the credentials over for the destination, new AuthID, and the backup just picked up where it left off. So I am kinda expecting that to happen on this one too - once I work out where the settings have gone.

As you say, the backup files are there. And the 85MB looks like the job database. Though, worryingly, the “backup 20190116063617.sqlite” is the same 56KB as the Duplicati-server.sqlite file. I did try a sneaky stop Duplicati, rename backup, start again… but no joy.

I’ll see if Windows own File History has an older copy of that sqlite file too. (Nope, no joy :frowning: )

I’d want to avoid running one of those never ending repairs though… be quicker to kill and restart.

-=-=-

Have read that thread a bit more now, and it lost me again. I have too much knowledge missing on the workings of Duplicati. Trying to follow a thread like that is tricky as it jumps around and assumes I know things that I don’t.

The main problem I have is not knowing the original choice of paths I was backing up to that location. And no backup was kept during the upgrade - this could be because the setttings had already been lost. Now I dig around in the backup data I don’t see any encouraging file stamps on the dblock.zip files. Mainly all back in August when it was initially installed. Though newer times on the dlist.zip.aes files.


#4

Actually… this brings up a worrying thought for me.

Does that mean there is no way to “adopt a backup”? Or open a backup on a different computer?

What happens if this building is flooded? If the three computers end up underwater? (Something VERY likely in this property) How would I go around retrieving data if all I have in my hands are the backup files on the OneDrive server and my password?

I thought there was a way of recovering from that stage? Have I missed a concept of this backup software? Or just not read the right page of the help files? :thinking:


#5

Sorry, let me try again. There are too many moving parts to this, so it’s easy to get different ones confused.

Question was vague. I spoke about downgrading Duplicati, but downgrading would not do OneDrive v2, and continuation on OneDrive (original version) is not possible because Microsoft retired the API, They’d require Duplicati 2.0.4.5 to do backup, but you can continue with the old destination data, and the old configuration (assuming we can get it, for example by temporary downgrade in order to get the config, not do the backup).

Basically you need to move forward on new Duplicati with old config, which ideally would have just happened. We’re apparently still trying to determine whether problem was at upgrade, or if backups had stopped earlier.

That would have avoided current chase, but configuration is clearly not supposed to disappear on upgrades. Losing it through machine loss is certainly possible. Restore can be done if you kept other records, but some details (for example source file selection) may be large enough to not be in, for example, a password keeper.

Restoring files if your Duplicati installation is lost shows how one might deal with a disaster, but it needs some information to get going. If all you have is your OneDrive authentication information, you might also need the Duplicati passphrase. It’s not stored with the encrypted files because that could make encryption pointless…

For any backup, one should test at least a sample restore, and even better is a direct restore to see if you’ve got the right information to do that in an actual disaster. An export of the backup configuration is supposed to also work (saving you some manual typing), but there might currently be a problem with extra option dashes:

Restore from Backup configuration

No and no, but neither is as easy as with some backup providers that have custom-built features to do them.

Backups are typically meant to be used from one computer. You don’t ever want two backing up to the same destination. It’s possible to grab files from a backup when at another computer by use of the same restores mentioned above, but it’s a bit risky because you get closer to the two-computers-on-one-destination failure.

Adopting a backup is most quickly done by moving files, including databases, though backup databases can be (sometimes very slowly) recreated from destination files, assuming you have the necessary configuration. That would be similar to the direct restore’s needs. It builds a database too but it’s only a “partial temporary”.

Migration to new PC - faq wanted is a useful request, had a planned solution, and we haven’t heard back yet.

In my view, the main Duplicati page at https://www.duplicati.com/ shows some of the key concepts behind why things are not as easy as they might be. The “Many backends” means one can’t count on advanced backend features that pre-packaged backup systems attached to specific storage may use, e.g. for backup adoptions. “Strong encryption” is taken seriously, causing passphrase to be stored separately from the files it encrypted. There might be ease-of-use aids later (Duplicati is young), but flexibility and security will probably remain key.

What exactly did you do? If you think you found a 56KB file that used to be Duplicati-server.sqlite, you’d move any current one aside, copy backup 20190116063617.sqlite to Duplicati-server.sqlite, start Duplicati, and see if it sees any backups. If you started it without a downgrade first, it should upgrade the database version, and wind up with the same configuration. Sometimes backups and other things show up slowly in the web UI, so a look at something else such as About --> System info to see if anything shows up as unknown may be useful.

You can take a guess at that by seeing what’s present on direct restore unless the config was super-precise. You can also go down the --unencrypted-database path, and find it in the Source table in an SQLite browser.
Duplicati.GUI.TrayIcon.exe explains the options. If you have the default automatic-start-at-logon you have no special options. You can either copy the shortcut and add --unencrypted-database temporarily, or Quit the Duplicati tray icon, then re-launch it manually from a Command Prompt with --unencrypted-database added.

Sounds odd and worrisome. Ordinarily you’d get updated dblock, dindex, and dlist files on every backup run. Did anyone have a look at job logs on that system, or reporting (e.g. emails) to see if backups were running?


#6

I’ll slice and dice my reply as I can see a few confused tangents have drifted in due to my shorthand with the question. And then my confusing supplementals. Thanks for this detailed response and I’ll be back shortly to get my head around it fully (need more coffee). :slight_smile:

Edit added at end of the day: I have also waffled too much during these replies that have been scattered through my day. Please don’t think you need to reply to all of the babblings of this loonatic. :rofl:

Call it “feedback” and also you have taught me some handy steps here. Thanks a lot for your details - it sent me off to find the right solution. And shows how resilient the backup data is even when badly setup.


#7

Thanks for your wide and detailed reply.

Recovering this specific backup
I think in this case it sounds like the config may have been lost a while back. There are just not enough different files on the server to have run multiple times like it should have. Especially when I compare it to the other successfully working backups.

The config in the %appdata% is clearly gone. The single backup is just the fresh database at 56KB

So already I have decided to restart this backup. I’m keeping the data for now to experiment with and see if I can bring it back to life (time allowing). But the fact I would have to download it all 60GB from OneDrive to reconnect it locally adds to a faff factor on this one that will not be worth the time required.

I now fully understand what you mean about OneDriveV1 and OneDriveV2. Even if I did downgrade to the old version I can’t use the old backup destination as Microsoft won’t let Duplicati talk that way due to the retired API.

Log Files and Emails
This is an area where I need to improve. As the logfiles are currently unreadable to a normal human, I haven’t been setting up the email to check them. Instead I am manually keeping an eye on a few of the running systems to learn about the issues that occur. Then trying to untangle the errors I see in a log.

The logs are currently very hard to skim read. So much developer info in there. This is one of my lessons from this event.

The passwords and Restoring files if installation is lost
I should also be clear - I do have all my Encryption Keys. I just don’t remember exactly which folders I had chosen on this PC to backup. I am about to start reading through the “Restoring files if your Duplicati installation is lost” link. (Once I get access to that PC again…). I can probably make a good guess as to what the folders should have been. This is going to be a handy lesson on restoring.

(And yes - I have done test recoveries… so that side is fine)

The Configs
I’ve also now learnt I need to export the configs after setting Duplicati up.


#8

Restoring that orphaned data
https://duplicati.readthedocs.io/en/latest/03-using-the-graphical-user-interface/#restoring-files-if-your-duplicati-installation-is-lost

Now this has been a worthwhile read. See - disasters bring good news. This does confirm to me that if a PC does explode, as long as I can get at the folder of backed up data and I have the Encryption Password then I’ll be able to recover the files.

I may even partially test that one on these files later…

So maybe I can’t “adopt a backup” and continue it… but at least I can open a backup on a different computer to recover the files.

That is the important step I needed to confirm to myself. That is what I meant by “had I missed the concept”.

Many backends
One of the strengths I have already see is that data can be lifted and moved. And swaps between SMB, FTP, OneDrive, Local USB, etc really don’t phase Duplicati. That impresses me and why I am here in the betas - even if I hit the odd issue.

The two above steps - either just getting a FULL recovery from backup data without config files. OR the ability to migrate a config between backup locations. That is a big plus which I have briefly misunderstood.


#9

Specifics of what I tried
Just to confirm a few points…

I did not attempt to install a downgraded old version.

What I specifically tried with the database is as you describe. I move aside the Duplicati-server.sqlite and copied backup 20190116063617.sqlite to Duplicati-server.sqlite.

One extra step was to do a “repair install” of Duplicati latest beta to make sure the correct files installed.

I then started Duplicati - but still nothing showing to backup.

I can also now understand what you mean by using a Full Restore to rebuild the config paths based on what was restored. Good knowledge for another day.

But it is the last point… the fact that what is on the server clearly shows me that there are not exactly many versions in the backup. This lets me think that the config vanished months ago. I’d put it down to PEBCAK and am not exactly concerned.


#10

More sagas from the UK South Coast…

Restoring from those “orphaned” files
As I have lost all config on the PC, I have now attempted a “Direct restore from backup files…”. And that works beautifully. I can see all the “building databases” messages as it works through the process of putting things back together for me. And a little heap of test files appeared at the end.

VERY happy with that.:grin::sunglasses: Panic over.

I assume if I am going to do a real restore it would be quicker to manually copy the files from OneDrive to the local folders and recover locally. i.e. get the network part out of the way in one go, and let Duplicati do all file work on local faster storage?

i.e. if I am pulling 35GB of data out of a backup store of 60GB then I assume the overheads of doing a restore bit by bit over the network from the cloud is slower than just pulling it all back to local and restoring locally.

With the exact specifics of this case I can see backups were running happy until October 17th and then all ceased… can’t be sure what killed that, but this is why I’m now going to get the email logs flowing next.

Also this shows me there is now no need to attempt to “reattach” this backup as I can dig into this one if needed.