Deleting Files on B2 not working because old DB not found

Support
1

Hi Overthere,
have been reading through some of the related articles but couldn’t make out my special problem.I had to reset my jobs a while back and have created new DBs in the process.

Because nowadays I run into the same B2 behavior as mentioned here Deleting files - excessive download I tried to delete older backups via commandline in the backend which starts fine but ends with DatabaseDoesNotExist referring to the old DB.

When trying to compact I get nearly the same error:

ErrorID: DatabaseDoesNotExist
Database file does not exist: C:\Windows\system32\config\systemprofile\AppData\Local\Duplicati\QWVNCTWPSD.sqlite
Return code: 100

I assume the reference comes from the files stored on B2 and I’ve got no idea how to get rid of them. Someone evt. got a hint for me?

2

Hello

the command line should include the appropriate database reference in the --dbpath= parameter.

1 Like
3

@gpatel-fr to my knowledge Delete command works without dbpath. Where did you get this info?

And the problem seems to be exactly that, there are old backup-files lingering around on the remote which I cannot adress because ot the DB-error. I don’t know which DBs originally were involved with these backups.

4

It would have been good to have noted them, but folder they’re in defaults to a user profile folder.

C:\Users\<Duplicati user account>\AppData\Local\Duplicati\ for a regular user install.
C:\Windows\system32\config\systemprofile\AppData\Local\Duplicati for service install.
One just above is if Windows service runs as SYSTEM (default) and is protected from most users.

What were you and are you now running as what user? Is commandline GUI or Windows prompt?
An ordinary user name doesn’t matter, but SYSTEM is special and somehow you’re referencing it.
Switching between GUI and GUI Commandline and true CLI can be confusing for verious reasons.

The files stored on B2 neither know nor need to know where your local database is. That’s local.

It cannot work without a database. A dbpath is one of many ways to say where that is. More here.

Which behavior? I assume you read enough to find that downloads are a normal part of compact.

EDIT:

We’ll see if a description of that will be needed, but one way to delete database and remote is below.
Note that the default is to delete the database but not the remote (which makes disasters less likely).

image
image1158×448 29.5 KB

5

Most commands (if not all) can work without a database. When it’s the case, the data from the backend is used: it works but it’s slower.
In your case, it seems that Duplicati has generated a path for a database from it’s complicated rules and afterwards did not find it. It’s rather baffling to be honest, because the code seems to imply that the database’s existence is checked before selecting a path. Could it be a rights problem ?
Anyway it’s always possible to explicitly specify a path, and if you have such a confirmed database somewhere, it would be interesting to do it.

6

There’s lots of SQL, and SQL needs a database (which doesn’t imply it needs a --dbpath).
You can sometimes set --console-log-level=information to see commands that work.

find says

The operation List has started
No local database, accessing remote store

compare says

The operation ListChanges has started
No local database, accessing remote store

delete and compact complain about the missing database, either my --dbpath one or implied one,
as recorded for that destination in C:\Users\<user>\Local\AppData\Duplicati\dbconfig.json.

We’ve been waiting several days for @Ralf to get some information back about the run environment which might save us from having to guess how situations arose, e.g. why is it in the SYSTEM profile?
GUI Commandline in a Windows service install might do that, and deleting a job can delete database.
See image above of Delete screen for a backup. Was that used with Delete the local database?

is very vague. Maybe all steps aren’t recalled, but I’m just asking for general info, including state now.
The more that can be said, the better, otherwise the answers can’t be tailored around known situation.

7

To try another angle, in hopes of getting a response, what is current situation in general for new jobs?

Best case would be everything’s great, destination is separate, and you wish to delete old job storage.
That can be done from B2 UI. If you’re trying to keep some old versions, situation gets more complex. Hearing “I had to reset my jobs” made me wonder whether there was an issue with old jobs. How so?

Missing database without old issues may be fixable on database screen, but gets B2 download fees, potentially heavy depending on what the old issue (if any) was, and this information is not stated yet.
There’s also no point in fixing old jobs if goal is to delete old jobs and all their data. What is the goal?

What jobs are in the GUI now (if you use it)? You went to some sort of commandline for some reason.

8

@ts678 @gpatel-fr thx for your hints and sorry for the delay which happens when life gets in the way.

I try to give the infos as good as I can:

It would have been good to have noted them, but folder they’re in defaults to a user profile folder.

C:\Users\<Duplicati user account>\AppData\Local\Duplicati\ for a regular user install.
C:\Windows\system32\config\systemprofile\AppData\Local\Duplicati for service install.

As you can see from the error-message it’s running as service, puttin’ the DBs in the later path.

In that folder I found the actual DBs and a json-File which I opened just out of curiosity. Therein I found references to the DB in the error-message. (Don’t know how I missed these because I already searched for the DB-name in file-contents :frowning: )

But I noticed that all other DBs mentioned in the json are also obsolete ones so I deleted the json and tried a “list-broken-files” as test but which ran into the same “db not found”, and recreated the json in the process.

I had to reset my jobs a while back and have created new DBs in the process.

Sorry not to be clear: With “reset” I ment “DBmaintenance delete and recreate” in the GUI. I havn’t touched the remote files.

what is current situation in general for new jobs?

They are running fine and seem to be ok

There’s also no point in fixing old jobs if goal is to delete old jobs and all their data. What is the goal?

Hm, that’s the real question and I must recollect:

  • My original intention was to reduce storage space on B2
  • B2 ran into the transactions limit with the actual Duplicati threshold of 25. As mentioned in other posts on that matter I reduced threshold to 10.
  • In that process I thought I could get rid of the most fragmented remote-files when deleting the oldest versions and to speed up the migration to the “less than 10%”-rule
  • Which than ran into the “db-not-founds” which than triggered my “why the heck do I have non-existant DBs refererenced?” leading to my post here

I hope this description is understandable?

9

I don’t see what could be the use of this json file in the appdata\local\duplicati directory. I don’t see one in my Windows test setup.
From what you say, I think that the ‘delete’ part of ‘delete and recreate’ only succeeded in part. Now if your database don’t exist, it could be the reason for your problem. Confirmed, when having deleted (moved) the database for the job, trying to compact generates an error with a non existing database name.
So you need to recreate the database.

10

It’s filled in some gaps, but left some gaps…

At first I thought you had old jobs and new jobs, but now I hear it was simply job database recreate.
For a given job that’s running fine, GUI has database, and this version of commandline knows that:

Using the Command line tools from within the Graphical User Interface
will put DB path from database screen into the dbpath option on the screen, so command will know.

If you open an OS terminal to go true CLI, you’re on your own to get the options right, or use Export.
Not giving --dbpath will assign a database path which is stored in dbconfig.json as noted above.
This is a convenience equivalent to how the GUI remembers the database for a job, except true CLI
doesn’t have jobs, so the next best thing is to tie database to backup based on destination (at least).
True CLI is by design entirely independent of GUI. To slightly tie them together, you’d give a dbpath.

Are you running true CLI without dbpath? That will make one with an entry for destination and DB, e.g.:

I think realization of a lack of database assigns one and records assignment, but filling the DB is later.
GUI is that way anyway. As soon as you make job, database screen shows path for first backup to fill.

Or use the right database, which would be the GUI one which works fine. Don’t recreate from true CLI simply for deletes. That would change the CLI database and the destination, and surprise the GUI DB.

Some points are still unclear, but if the goal is to delete version from working GUI job, do a GUI delete.

Here’s what’s unclear:

Based on being old, or what? The GUI delete and recreate would not have left the old database around.
Working in true CLI some time back could have left an old database untouched since last true CLI work.

There are others ways to get that path error. Is this in GUI Commandline or true CLI? If CLI, what user?

11

Yes obsolete like in cannot be found anymore on the disk and are not referenced in any job.

Working in true CLI some time back could have left an old database untouched since last true CLI work.

I agree but havn’t ever bothered with CLI. Therefore I would assume that recreating the DBs in GUI should remove any mention of the older GUI DBs. That’s why I thought it might have got sth to do with the remote-files.

There are others ways to get that path error. Is this in GUI Commandline or true CLI? If CLI, what user?

As said above no CLI so far. The duplicati-service is running as user System as by default.

What’you think, would it help to clear things up if I delete/recreate all DBs once again?

12

Then in GUI Commandline, if you’re prone to remove inapplicable options, don’t remove dbpath.
Possibly that’s how you got dbconfig.json working. I might also misunderstand how that works.

I suggest looking at path in Database screen, using GUI Commandline, verifying dbpath shows it,
then leaving it alone and trying desired delete. It should use the GUI database just like GUI does,
successfully according to you on GUI operations, so I assume that GUI has a good database now.