Duplicati no longer launching on Mac - database version?


#15

Sorry to hear that - and glad to hear you did back up your json file.

Let us know if you run into any issues with the database repair - and thanks for the detail about the process you had to go through. Hopefully we can align it with some other reports and figure out why it happened.


#16

Duplicati is not longer launching on a Mac. No upgrades were made. Running Duplicati 2.0.3.3_beta_2018-04-02 and Mac High Sierra 10.13.6. I am not an experienced user, although I am running Duplicati on 6 different Macs. Any suggestions on what I should do to try to find out why it will no longer launch?


#17

Is this the same Mac you’ve had issues on twice before or is it a different machine?

Are you running tray-icon or service/daemon?

Do you get any console messages if you run from command line?

Have you checked for an error log in %LOCALAPPDATA%Duplicati\crashlog.txt? (Usually created only on super catastrophic errors like not being able to write to the database logs.)


#18

I finally got together with my son on whose computer this issue was occurring. We still can’t launch Duplicati. Here are the answers to your questions. Any suggestions on how to proceed?

  1. It is a different Mac than the one that had issues from before.
  2. On the mac, the main Duplicati app is automatically launched at startup.
  3. Have not tried to launch from the command line. I have not used the command line tool before. I would need to learn how to do that.
  4. There is an error file that gets created when you try to launch! Here it is:
    System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. —> System.Exception: A serious error occurred in Duplicati: System.Exception: Failed to create, open or upgrade the database.
    Error message: The database disk image is malformed
    database disk image is malformed
    at Duplicati.Server.Program.GetDatabaseConnection (System.Collections.Generic.Dictionary2[TKey,TValue] commandlineOptions) [0x0023b] in <670b0e4a9ae144208688fcb192d20146>:0 at Duplicati.Server.Program.RealMain (System.String[] _args) [0x002bb] in <670b0e4a9ae144208688fcb192d20146>:0 ---> System.Exception: Failed to create, open or upgrade the database. Error message: The database disk image is malformed database disk image is malformed at Duplicati.Server.Program.GetDatabaseConnection (System.Collections.Generic.Dictionary2[TKey,TValue] commandlineOptions) [0x0023b] in <670b0e4a9ae144208688fcb192d20146>:0
    at Duplicati.Server.Program.RealMain (System.String _args) [0x002bb] in <670b0e4a9ae144208688fcb192d20146>:0
    — End of inner exception stack trace —
    at Duplicati.GUI.TrayIcon.HostedInstanceKeeper…ctor (System.String args) [0x00093] in :0
    at Duplicati.GUI.TrayIcon.Program.RealMain (System.String _args) [0x002a8] in :0
    at (wrapper managed-to-native) System.Reflection.MonoMethod.InternalInvoke(System.Reflection.MonoMethod,object,object,System.Exception&)
    at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object parameters, System.Globalization.CultureInfo culture) [0x00032] in :0
    — End of inner exception stack trace —
    at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object parameters, System.Globalization.CultureInfo culture) [0x00048] in :0
    at System.Reflection.MethodBase.Invoke (System.Object obj, System.Object parameters) [0x00000] in :0
    at Duplicati.Library.AutoUpdater.UpdaterManager.RunMethod (System.Reflection.MethodInfo method, System.String args) [0x00000] in <9ce928d8db224e1f8330a80e119b5dbe>:0

VACUUM - performance improvement is huge
#19

Looks like somehow the database (probably the main one) got corrupted.

I’d suggest moving any databases not named “duplicati-server.sqlite” to another folder and see if it starts.

If not, move the server one ou5 and try again. If that works, move any non-server ones back.

Once we’ve narrowed down the bad database(s) we can look at repairing it replacing.


#20

Just connected with my son (and his laptop) today. We followed this suggestion and Duplicati started. We had moved out three .sqlite files and one .backup file. I then moved the files back in and restarted the computer and Duplicati and Duplicati still started, which was good, but the backup sets did not show up in the GUI.

What might be a good next step to see if we can get the backup sets to show up again?


#21

The backup sets are stored in the duplicati-server.sqlite for (I think… There might be a.json for involved as well).

When Duplicati starter up it will create a black one if none is found. Do you have an older version of that database you could try copying into the folder?

You can always manually recreate the jobs and point the to the old databases - the truck is knowing which database was for which job.

The .backup file indicates a database was updated or repaired at some point. You might be able to use that if it’s not too old - but that’s a fairly hands on approach.

@kenkendk, how hard would it be to serialize the job (maybe with global settings} and store it in the job database to simplify this type of recovery?


#22

I think so too. Unfortunately it was put aside earlier because The database disk image is malformed.
Database disk image is malformed was one case where the damage was assessed then manually repaired, however that’s a rather advanced effort. Some Internet postings also say that an export then import can fix, however what was done at the link above was more. You could attempt an SQLite repair if you would like…

A relatively simple task would be to see if the damaged Duplicati-server.sqlite will open in an SQLite viewer. Information in the Backup table includes the given name for each backup, and the path to its job database. Information in the Source table includes the folders and files that had been chosen to go into each backup. There’s also schedule information and more, for your use as aids to manually rebuilding the configurations.

DB Browser for SQLite is a tool that you can install to see whether it can read the old Duplicati-server.sqlite.

The best case would be if you have exported and saved the backup configuration yourself. If so, just import.

Old databases with random names can be inspected by looking inside using Direct restore from backup files. Maybe your configuration is short enough to just recreate. For help, see above on old Duplicati-server.sqlite.


#23

Great tip, thanks! 


#24

Or maybe not a great tip. Works well for seeing what’s at the destination but I’m still looking for the UI to plug in a database. I suspect it doesn’t exist because a job database caches destination info, and has little config. One could reverse-engineer the source file configuration from the backup it produced, but it’s rather indirect. The inspection could be done by opening the randomly-named database in a viewer, or using Direct restore.

How to test which database belongs to which job? talks about this, and also about configuration backup idea.


#25

Hi, There are various tools that we can use to try to figure out what is going on with the databases. (suggest to make a copy of duplicati-server.sqlite before proceeding and work only on the copy.)

Open the Duplicati-server.sqlite copy in dbeaver (the tool I usually use), DB Browser which @ts687 mentioned or another such tool. then can you please issue the following command and paste the results?

pragma
integrity_check;

In most SQL tools you type the commands and press ‘ctrl’/‘cmd’ + ‘enter’ to execute them.


#26

Thanks for the suggestion. I downloaded DB Browser and tried to open the Duplicati-server.sqlite file and it could not be opened. The error message said that the database disk image was malformed.

So, do I trash the online backup and start over or is there a way to save me from having to backup all of my files again to the cloud?


#27

That’s kind of… alarming. How on earth could the database get into such a bad state that it cannot even be opened? I don’t want to alarm you but is there any evidence of a disk or filesystem issue?

What does file(1) identify ‘Duplicati-server.sqlite’ as? I belive that the traditional POSIX file command is installed with macintoshes, so E.G.

file /path/to/Duplicati-server.sqlite

Please paste the exact output.

At this point the only thing that I can suggest is seeing if the sqlite3 client tool can open it. It may be that the gui tool is doing some kind of processing which trips this error before we have time to get to a worksheet and enter commands. For example, looking at the screenshot, it looks like there are “database structure” and “browse data” tabs. IN order to meaningfully load them, it will parse the database to some way, probably by issuing SELECT commands against the system catalogue. In this case, it may ‘trip’ over the source of ‘malformity’ and die ungracefully.

Maybe we will get lucky!

If you want to try the sqlite3 tool, please download and unzip the SQLITE3 Precompiled Binaries for Mac OSX from SQLite Download Page Alternatively, a tool such as Homebrew may be used but I am not familiar with this.

Then extract and open a terminal in the location the binaries are extracted to and enter
./sqlite3 /path/to/copy of Duplicati-server.sqlite "PRAGMA integrity_check"

Again, I advise using a copy of the file.

Please paste the exact output. Even if they are vague.


#28

I am not a dev, but I had a thought:

Theres’ a button on the UI that exports the job config as JSON

On each restart, the enterprise application that I work with ingests the latest version of its main configuration properties file into a special table called CONFIG_REGISTRY

maybe there could be, in the main and job-specific databases a table like:

CREATE TABLE config_registry ( 
    id           INTEGER PRIMARY KEY,
    digest       TEXT, -- md5sum of the JSON
    job_name     TEXT, -- denormalized reference to the 'name' of the job
    dtmodified   TEXT, -- Sqlite does not support dates
    config_json  TEXT  -- that exported JSON mentioned above
);

CREATE UNIQUE INDEX config_reg_digest on config_registry (digest);

On each startup, the app does the JSON export, gets the MD5 and inserts into this table with the ON CONFLICT DO NOTHING option

This can be kept from growing out of control rather easily by also doing

DELETE FROM config_registry WHERE id 
                           NOT IN ( SELECT id FROM config_registry
                                    WHERE job_name = :1 
                                    ORDER BY dtmodified DESC
                                    LIMIT 10 );

There would need to be some way to handle renaming of jobs, however.


#29

Have you tested the actual backup? Backups don’t need to be listed to be available. Try direct restore and see how the backup looks that way. Try a small restore. If all looks sound, then the challenge is to work out how to hook everything back together. Now that you have a DB browser, you can look through *.sqlite files, especially in their File table, to see if you can find something that matches what you think your backup had. You can also sanity-check it by file date. If it looks right, make sure you keep it around, and you can slide it underneath new job creation in Duplicati by copying it into the path Duplicati says on job Database screen. With only one backup and three regular .sqlite files, opening the four would probably get something of use.

Job databases can also be recreated from destination files (see Recreate button) but it can be very slow…

@Ingmyv makes a good point that there’s a chance there’s a hardware problem in action, and getting up after a disk replacement (which we can hope we don’t have) would be a similar process, starting maybe on direct restore to get some things back, then getting the job database rebuilt with Recreate plus some time. You’re in a somewhat nicer position because you might still have all your sqlite files except for the server’s.


#30

If I recall correctly, when I tried using DB Browser in Linux and tried to open my .sqlite file when connect with a read-only user I got that error.

I ended up copying the .sqlite file to a local writeable folder and opening the copy from three w/out issue.

Do you mean where in the UI do you point an existing job to a different database? If so, it is rather un-intuative as you need to use the job Data base menu then not just enter but actually CHANGE the existing database path which will then activate the buttons beneath the field.
image


[Solved] After importing configuration and running task is it ok to "Rapair"?
Restore Backup with only dblock and dindex files
#31

No. The context for that was direct restore, and for a moment I thought it could look inside random databases, but then I went to look for the actual web UI and didn’t see a place to enter its path. Your tip on specifying the database for a job was good though. I’ve poked at that UI for awhile, and its behavior was hard to understand, which definitely slows one down when handling important things like databases. Ideally it would be VERY clear.


#32

Agreed - so you’re probably gonna love this… :slight_smile:

When doing a direct restore, you should be able to use “Edit as text” on one of the Advanced Options sections (such as in step 1 or 2) and manually add the --dbpath= parameter.

I’m not positive (and not sure how to easily test), but I think that will tell it to use that database instead of creating an interim one based on the destination files.


#33

Not working here. Step 1 doesn’t use it (I’ve had trouble before here, trying to slide options in), and step 2 uses it then gives a red Error box at screen bottom saying “The version(s) being updated to, already exists”. Profiling log showed it did a SELECT for the newest timestamp, presumably found it there already, and then got upset…

Apologies to others on this thread for rambling a little, while waiting for @Mark_Daubenmier news on backups. Seems like we now have several options to re-attach the old destination to its old database, if it can be located.


#34

Sorry for the long delay. I ran out of time because I was moving. I ended up just deleting everything and starting over. This got things up and running again, but I lost the opportunity to track down what was wrong for the educational benefit.