Command list reports 2 backup versions more than overview

mja47 · May 31, 2021, 12:02pm

Installed version is 2.0.6.1 on Linux (openSUSE 15.2) with mono 6.8.0. Backups to a cloud based WebDAV destination works fine.

Backup overview in Duplicati UI reports 17 versions but command line command list reports 19 (0-18). Backup uses 4 source folders and all show this list (German localization):

0	: 30.05.2021 22:31:57  - 
1	: 29.05.2021 22:12:38  - 
2	: 23.05.2021 22:32:36  - 
3	: 10.05.2021 22:19:01  - 
4	: 18.04.2021 22:38:53  - 
5	: 02.04.2021 14:27:19  - 
6	: 20.03.2021 22:06:05  - 
7	: 21.02.2021 22:50:49  - 
8	: 24.01.2021 22:49:24  - 
9	: 07.01.2021 21:55:13  - 
10	: 30.12.2020 22:10:00  - 
11	: 21.12.2020 21:18:49  - 
12	: 13.12.2020 18:03:34  - 
13	: 06.12.2020 08:13:58  - 
14	: 25.11.2020 13:50:49  - 
15	: 12.11.2020 22:20:13  - 
16	: 04.11.2020 22:27:47  - 
17	: 05.10.2020 17:27:52  - 
18	: 31.08.2020 21:50:58  -

Why 19 here and 17 shown in overview?

Other question, probably related to this: overview reports #5 as the last successfull one.

EDIT:
New backup yesterday evening. Now list shows 20 entries (0-19), duplicati ui still reports 17 versions.

ts678 · June 1, 2021, 5:52pm

Can you say whether or not 2.0.5.1 had the same behavior? Unfortunately, some logs only last 30 days, otherwise I’d ask how the logs looked around the spots that differ. You can still look for the spots though.
If you have any other records (e.g. emails), you could also look around the ones that vary, for any clues.

Please add in opinions from home page in the backup entry, and for latest job Complete log. Example:

Backup:2.98 GB / 40 Versions

“BackupListCount”: 40

I sometimes suspect partial backups, but they seem OK here. They are marked on Restore dropdown:

Running the list command on showed 40 as well, although the partial ones weren’t flagged as partial.

If need be, we can get into looking into databases with sqlitebrowser, or creating a database bug report.
I’d prefer to start with simpler things first. Possibly simple too is – how many dlist files are in backup?

If your WebDAV tools aren’t up to the task, you can also do <backup> → Show log → Remote and click latest list command that you see. That will give you a file listing which hopefully matches some count.

mja47 · June 1, 2021, 7:24pm

Thank you.

Simple things first:
Webdav: folder lists 20 files with extension dlist.zip.aes.

JSON log data of latest backup contains
[…]
“BackendStatistics”:{
[…],
“BackupListCount”: 20,
[…]
}

Restore: no entries marked “partial”.

Your question regarding 2.0.5.1: sorry, I don’t remember.

I have sqlitebrowser installed.

ts678 · June 1, 2021, 9:19pm

The majority opinion is 20 then (which doesn’t prove anything, but suggests where to look for the loss).

If you would like, you can open your Database (read-only if given a choice, or make a copy then open).
The Fileset table will probably have 20 rows, one for each backup version, maybe all IsFullBackup.
I’m not sure what reason there would be to not show everything. Did you locate the missing versions?

A compare is easier with GUI and list, as DB Timestamp needs converting, e.g. by EpochConverter.
Version 0 is the most recent, so you can sort by Timestamp and get the browser line number minus 1.
My oldest is now July 2, number 40 in the Restore dropdown on 2.0.5.114 Canary. Its DB entry is this:

Your time zone: Thursday, July 2, 2020 7:40:02 AM GMT-04:00 DST

I’m not familiar with HTTP protocol use, but a couple of ways to see what gets sent are packet capture (getting even more technical) or browser web developer tools. Below is Firefox Restore screen in F12:

and on the right, the Response shows me my now 41 versions numbered 0 through 40 per dropdown:

[
	{
		"Version": 0,
		"IsFullBackup": 1,
		"Time": "2021-06-01T16:40:01-04:00",
		"FileCount": -1,
		"FileSizes": -1
	},
	{
		"Version": 1,
		"IsFullBackup": 1,
		"Time": "2021-06-01T15:40:03-04:00",
		"FileCount": -1,
		"FileSizes": -1
	},
	{
		"Version": 2,
		"IsFullBackup": 1,
		"Time": "2021-06-01T14:41:20-04:00",
		"FileCount": -1,
		"FileSizes": -1
	},
	{
		"Version": 3,
		"IsFullBackup": 1,
		"Time": "2021-06-01T13:40:00-04:00",
		"FileCount": -1,
		"FileSizes": -1
	},
	{
		"Version": 4,
		"IsFullBackup": 1,
		"Time": "2021-06-01T12:41:08-04:00",
		"FileCount": -1,
		"FileSizes": -1
	},
	{
		"Version": 5,
		"IsFullBackup": 1,
		"Time": "2021-06-01T11:40:11-04:00",
		"FileCount": -1,
		"FileSizes": -1
	},
	{
		"Version": 6,
		"IsFullBackup": 1,
		"Time": "2021-06-01T10:40:00-04:00",
		"FileCount": -1,
		"FileSizes": -1
	},
	{
		"Version": 7,
		"IsFullBackup": 1,
		"Time": "2021-06-01T09:40:00-04:00",
		"FileCount": -1,
		"FileSizes": -1
	},
	{
		"Version": 8,
		"IsFullBackup": 1,
		"Time": "2021-06-01T08:40:00-04:00",
		"FileCount": -1,
		"FileSizes": -1
	},
	{
		"Version": 9,
		"IsFullBackup": 1,
		"Time": "2021-06-01T07:40:14-04:00",
		"FileCount": -1,
		"FileSizes": -1
	},
	{
		"Version": 10,
		"IsFullBackup": 1,
		"Time": "2021-06-01T06:44:33-04:00",
		"FileCount": -1,
		"FileSizes": -1
	},
	{
		"Version": 11,
		"IsFullBackup": 1,
		"Time": "2021-06-01T06:22:46-04:00",
		"FileCount": -1,
		"FileSizes": -1
	},
	{
		"Version": 12,
		"IsFullBackup": 1,
		"Time": "2021-05-31T21:40:02-04:00",
		"FileCount": -1,
		"FileSizes": -1
	},
	{
		"Version": 13,
		"IsFullBackup": 1,
		"Time": "2021-05-31T20:40:00-04:00",
		"FileCount": -1,
		"FileSizes": -1
	},
	{
		"Version": 14,
		"IsFullBackup": 1,
		"Time": "2021-05-31T19:40:00-04:00",
		"FileCount": -1,
		"FileSizes": -1
	},
	{
		"Version": 15,
		"IsFullBackup": 1,
		"Time": "2021-05-31T18:40:01-04:00",
		"FileCount": -1,
		"FileSizes": -1
	},
	{
		"Version": 16,
		"IsFullBackup": 1,
		"Time": "2021-05-31T17:40:12-04:00",
		"FileCount": -1,
		"FileSizes": -1
	},
	{
		"Version": 17,
		"IsFullBackup": 1,
		"Time": "2021-05-31T14:40:00-04:00",
		"FileCount": -1,
		"FileSizes": -1
	},
	{
		"Version": 18,
		"IsFullBackup": 1,
		"Time": "2021-05-30T14:40:00-04:00",
		"FileCount": -1,
		"FileSizes": -1
	},
	{
		"Version": 19,
		"IsFullBackup": 1,
		"Time": "2021-05-29T14:40:00-04:00",
		"FileCount": -1,
		"FileSizes": -1
	},
	{
		"Version": 20,
		"IsFullBackup": 1,
		"Time": "2021-05-28T13:40:01-04:00",
		"FileCount": -1,
		"FileSizes": -1
	},
	{
		"Version": 21,
		"IsFullBackup": 1,
		"Time": "2021-05-27T12:40:02-04:00",
		"FileCount": -1,
		"FileSizes": -1
	},
	{
		"Version": 22,
		"IsFullBackup": 1,
		"Time": "2021-05-26T11:40:02-04:00",
		"FileCount": -1,
		"FileSizes": -1
	},
	{
		"Version": 23,
		"IsFullBackup": 1,
		"Time": "2021-05-25T11:40:02-04:00",
		"FileCount": -1,
		"FileSizes": -1
	},
	{
		"Version": 24,
		"IsFullBackup": 0,
		"Time": "2021-05-22T13:58:23-04:00",
		"FileCount": -1,
		"FileSizes": -1
	},
	{
		"Version": 25,
		"IsFullBackup": 0,
		"Time": "2021-05-22T13:16:35-04:00",
		"FileCount": -1,
		"FileSizes": -1
	},
	{
		"Version": 26,
		"IsFullBackup": 1,
		"Time": "2021-05-18T07:40:00-04:00",
		"FileCount": -1,
		"FileSizes": -1
	},
	{
		"Version": 27,
		"IsFullBackup": 1,
		"Time": "2021-05-10T20:40:00-04:00",
		"FileCount": -1,
		"FileSizes": -1
	},
	{
		"Version": 28,
		"IsFullBackup": 1,
		"Time": "2021-04-19T12:40:01-04:00",
		"FileCount": -1,
		"FileSizes": -1
	},
	{
		"Version": 29,
		"IsFullBackup": 0,
		"Time": "2021-04-02T21:40:02-04:00",
		"FileCount": -1,
		"FileSizes": -1
	},
	{
		"Version": 30,
		"IsFullBackup": 1,
		"Time": "2021-03-14T14:40:01-04:00",
		"FileCount": -1,
		"FileSizes": -1
	},
	{
		"Version": 31,
		"IsFullBackup": 1,
		"Time": "2021-02-06T10:40:04-05:00",
		"FileCount": -1,
		"FileSizes": -1
	},
	{
		"Version": 32,
		"IsFullBackup": 1,
		"Time": "2021-01-01T10:40:04-05:00",
		"FileCount": -1,
		"FileSizes": -1
	},
	{
		"Version": 33,
		"IsFullBackup": 1,
		"Time": "2020-11-26T07:44:53-05:00",
		"FileCount": -1,
		"FileSizes": -1
	},
	{
		"Version": 34,
		"IsFullBackup": 1,
		"Time": "2020-10-17T20:40:00-04:00",
		"FileCount": -1,
		"FileSizes": -1
	},
	{
		"Version": 35,
		"IsFullBackup": 0,
		"Time": "2020-10-14T06:40:02-04:00",
		"FileCount": -1,
		"FileSizes": -1
	},
	{
		"Version": 36,
		"IsFullBackup": 0,
		"Time": "2020-10-03T08:15:13-04:00",
		"FileCount": -1,
		"FileSizes": -1
	},
	{
		"Version": 37,
		"IsFullBackup": 0,
		"Time": "2020-10-03T07:59:12-04:00",
		"FileCount": -1,
		"FileSizes": -1
	},
	{
		"Version": 38,
		"IsFullBackup": 0,
		"Time": "2020-10-03T07:54:46-04:00",
		"FileCount": -1,
		"FileSizes": -1
	},
	{
		"Version": 39,
		"IsFullBackup": 1,
		"Time": "2020-09-13T17:16:28-04:00",
		"FileCount": -1,
		"FileSizes": -1
	},
	{
		"Version": 40,
		"IsFullBackup": 1,
		"Time": "2020-07-02T07:40:02-04:00",
		"FileCount": -1,
		"FileSizes": -1
	}
]

I can’t think of a reason why a backup wouldn’t be shown. A developer might need to go study the code.
Because the list command has some not in GUI, what happens if you try to list that version’s files?

Harder to start this in sqlitebrowser is to copy Fileset ID to FilesetID filtering header in FilesetEntry table. File counts don’t change hugely from one to the next, but it may matter if your missing versions are odd.

I suppose one easy thing to try is to do a browser hard refresh (often Control-F5), but it likely won’t help.

mja47 · June 2, 2021, 6:18am

SQLITE database, table fileset contains 20 records, all marked 1 in column IsFullBackup.

To clarify: I don’t miss versions? Choosing Restore in the GUI offers 20 versions (numbered from 0 (latest) to 19 (oldest)).

I’m just a bit confused and worried because GUI overview (actually) reports 17 versions and last successful one month ago which is definitely wrong.

Using Firefox F12 Developer tools I did find request “backup” which did contain JSON data. Here’s extract (same data masked with xxxx):
[
{
“IsUnencryptedOrPassphraseStored”: true,
“Schedule”: null,
“Backup”: {
“ID”: “1”,
“Name”: “xxxx”,
“Description”: “xxxx.”,
“Tags”: [ ],
“TargetURL”: “webdavs://xxxx”,
“DBPath”: “xxxx”,
“Sources”: null,
“Settings”: null,
“Filters”: null,
“Metadata”: {
“LastBackupDate”: “20210402T122719Z”,
“BackupListCount”: “17”,
[…]

My conclusion: Some kind of logic error in duplicati code.

ts678 · June 2, 2021, 2:17pm

Sorry, I see you mentioned 5 : 02.04.2021 14:27:19 as last successful one earlier, and I missed that.
Although it’s more like 2 months ago now, is that still the latest version shown on home page for backup?
I see mine seems to be updating, assuming I’m now understanding where to look. I assume you mean at:

which is supposed to be updated on every successful GUI backup, but is not for CLI or for other changes.
Do recent backups’ Show log show logs at the right time, with green dots and no seeming issues inside?

If so, then you likely did a backup. You can test it by trying a small Restore (good practice in any case).

Backup job configuration and some result statistics are in Duplicati-server.sqlite, which you can look at.
The BackupID is in the Backup table and GUI URLs, e.g. localhost:8200/ngax/index.html#/log/1

Also check your About → Show log → Stored to see if there are any errors happening after backup itself.
There are post-backup actions such as Reporting options and run-script-after that sometimes hit errors, however the fact that you are getting backups at least suggests that nothing is getting permanently stuck.

You can set up one or both of those to see if you can find further information, but I’m not sure it’s worth it.
What’s a bit easier is to log ending of backup, e.g. About → Show log → Live → Verbose for any oddities.

You can also check the modification time of Duplicati-server.sqlite in your Database folder. Mine’s at 9:43 which matches the time of the job log. If yours isn’t changing, can you change it any other way, e.g. using Add backup to make a new small backup? If it changes, then you can schedule the backup for more test.

mja47 · June 2, 2021, 5:45pm

I guess this is the point:

During march I prepared cli backup with automatic shutdown of my workstation.

And this backup is definitely the last issued from GUI:

Thank you. My initial question is answered and it’s clarified why GUI presents different information than list command.

Is this explained somewhere?

ts678 · June 2, 2021, 6:17pm

You might be able to put that in a GUI run-script-after, although I do get a little nervous about the DB.
I “think” that DB writes are over with by the time the script runs, but you could check DB timestamp.

CLI jobs are intentionally independent of each other and of the server. The manual could clarify this.
It comes up fairly often on forum. Suggestions are seemingly accepted by author at GitHub project.
It’s confusing because things that are in per-backup DB get update, but things in server DB do not.
One solution mentioned at the above link is https://github.com/Pectojin/duplicati-client

One trick that did get in begins

Instead of composing the complete backup command yourself, including all advanced options, you can create a backup job in the Graphical User Interface"

then Export As Command-line for compatible CLI work, but (unsaid) don’t expect all GUI to update.
The reason the per-backup info updates is because the export gave its –dbpath to CLI to do its work.
Per-backup DB contains not only info on the backup data, but is also where the per-backup logs are.

mja47 · June 2, 2021, 8:27pm

Thanks.

ALL backups performed regardless of issued by GUI or cli are listed in output of duplicati list command.
My backup script uses option --dbpath to point to duplicatis sqlite database. That’s the reason why all backups are known in database.
Shutdown command is called AFTER backup script has terminated. shutdown then waits (by default) 1 minute to start shutdown process.

Here is script template (individual option values masked with xxxx):

#!/bin/bash
# run this script with sudo and nohup to be able to shutdown!
# check effective id first
if [ -n "$SUDO_USER" ]; then

    # perform backup as user xxxx
    sudo -u xxxx \
    duplicati-cli backup \
    "webdavs:xxxx" \
    /home/xxxx/Dokumente/ /home/xxxx/Bilder/ \
    --backup-name=xxxx \
    --dbpath=/home/xxxx/.config/Duplicati/xxxx.sqlite \
    --encryption-module=aes \
    --compression-module=zip \
    --dblock-size=50mb \
    --passphrase="xxxx" \
    --retention-policy="3M:1D,1Y:1W,10Y:1M" \
    --run-script-before=xxxx.sh \
    --disable-module=console-password-input

    /sbin/shutdown
else
    echo "Please run this script with sudo!";
fi

The remaining question is: What is missing for GUI to report actual number of valid backups and date of last backup operation?

BTW: my duplicati sqlite database does not show table “Metadata”. Table Version shows entry id=1, version=11.

ts678 · June 2, 2021, 9:12pm

Probably the code to reach in the job database and pull it out. It’s probably easier when server runs job.
There’s quite a bit of other data for a GUI job that you haven’t asked about, though some is less visibile:

If you like, you can add a request to the pile of issues, and maybe a developer will look more someday.
Depending on how the code is partitioned, it may be easier or harder for requested info to be retrieved.
There might also need to be some sort of timed poll, as server does not know that the CLI job has run.
Ultimately, you might prefer an integrated system, but there isn’t one, and that was the original intent…

EDIT:

Server DB is version 6. You’re in the job database. Look in the server database, Duplicati-server.sqlite.

mja47 · June 3, 2021, 6:01am

Ok, thank you. Now it’s clear to me.

duplicati-cli backup performs backup and populates session database provided by option --dbath
duplicati-cli does NOT update duplicati server database

That’s the reason why GUI (frontent of duplicati server) summary is not aware of backups performed by duplicati-cli. Duplicati-cli list uses session database as source and lists ALL backups logged in session database.