Stuck on *.sqlite-journal file that disappeared

#1

Backup has been stuck for several days. Ended backup with Task Manager and restarted. Things were chugging along fine and then the speed started to slow down noticeably. Right now it is stuck on the following file:

Current file: C:\Users\XXXXX\AppData\Local\Duplicati\VCHJHDUSPI.sqlite-journal

Funny thing is, when I looked for the file with File Explorer, it was less than 100 kb in size. I looked back later and the file wasn’t even there anymore even though Duplicati was still hung up on it. What is the sqlite-journal file anyway? Is it some kind of a temp file that shouldn’t even be backed up?

#2

Yeah, definitely avoid backing up the Duplicati databases directly. There is no value in doing so (as they can be regenerated from the backend data).

A future version of Duplicati will blacklist these files from backup. But for now just don’t include them.

#3

2.1. Rollback Journals describes it, and as you can see it’s quite dynamic, which might be one way it leads to your situation. I suspect Duplicati still has an occasional problem with certain file change patterns during backup, however the situation has not been reproducible by me – even backing up the Duplicati databases.

What would be excellent is to find how to reproduce the hang for the developers so that it can be looked at. Yours has already been cleared, and field debug can be hard. Does your problem happen easily or rarely?

Creating a bug report and posting it with the hang time is one way that past hang can maybe be looked at. Chances are that there’s not much in the way of other logs, but if you can find any, those would also help.

Always exclude job-specific database files from backup #4390 was in progress before a delay to see if we could understand the hang better. You can see some discussion of the pro and con of doing workarounds.

Do you think we ought to just do it (and ideally, document it some, but where) while looking for root cause?

One do-it-yourself workaround is to use snapshot-policy to use VSS after setting up Administrator privilege.

#4

I went back and looked and the only way that I can see that Duplicati should have even tried to backup that file was if I had checked my Windows - C: drive which I haven’t. Why is it even trying to back it up?

#5

I have stopped and restarted the backup several times and it continues to stop on the same file EVERYTIME. I’m wondering about if it may be because of the way that I installed Duplicati on my laptop. When it installed, I had it installed in:

C:\Program Files\Duplicati 2

Could it be that because I did it that way when it is backing up the User Data\Application Data that it goes to that folder that I created to hold the program files? If that is the issue, how should I properly install Duplicati on my laptop?

#6

I guess that’s as reproducible a problem as one can get (which is good), but other things confuse me.

I’m confused. C:\Program Files\Duplicati 2 is the installation spot, you don’t create it, and typically there’s no Duplicati data in there unless you use –portable-mode to make a data folder there to hold databases.

Typically, databases are in C:\Users\<user>\AppData\Local\Duplicati, unless it’s Windows service install.

Could you say more about how you installed? Installs can be either quite simple, or elaborate user setups.

You can also look at your job’s Database page to see where the database is for that job. Is it as expected?

Original post had path above, which looks like a Duplicati database, so how did the question below arise?

You don’t create the AppData folder either, and when I hear term “program files” I think C:\Program Files".

Please clarify. You can also get a log by About --> Show log --> Live --> Verbose to see where program is getting its files from. It “should” use your Source data configuration. If it’s not, something seems very off.

A more permanent and somewhat nicer log is possible with log-file=<path> and log-file-log-level=verbose.

EDIT: After figuring out the basics of the situation, a higher log level might help, but they can get enormous. Verbose level does show some paths, which might be sensitive, but for a small log maybe you can redact.

How are these repeat stops done? Are they hard stops, such as killing Duplicati (not good, but sometimes needed). Regardless, I think restart tries to not repeat backup already done, but pick up where interrupted.

#7

When I originally installed Duplicati on my laptop, all of the files installed in the root directory of C:. When that happened, I deleted all of the files and created a new folder in C:\Program Files\ named “Duplicati 2” and then reinstalled Duplicati and directed the install to that folder.

There are three folders and a couple of files at C:\Users\user>\AppData\Local\Duplicati but there is a LOT more folders and files at the installation location, C:\Program Files\Duplicati 2. It appears that this is where the databases are as there are as there are a LOT of .dll files.

They are not hard stops. When it gets to that file, the transfer to the hard drive stops, processor usage by the program goes to zero and the Google Chrome Duplicati interface stops with the number of files remaining, GB’s remaining and the last transfer speed showing. In order to get it working again, I have to use Task Master to stop the program and restart it. While I do have ability to select “Stop after Current File” or “Stop Now”, neither of these commands do anything.

Will run it again and look at the verbose log live.

#8

Installing Duplicati on Windows shows the install, assuming you did it that way. Are you saying that the:

image

either didn’t come up that way, or did but wasn’t respected, so you had to Browse? I’ve never tried that…

.dll file location does not determine the databases location. .dll files are program code and belong there.

If you want to know where the databases are, look at your Database screen, or look for the .sqlite files.

That’s a hard stop. You’re simply killing it… Proper shutdown is right click on tray icon, then select Quit, and this would be after “Stop after current file”, but those don’t work, so you hard stop, and risk damage.
Presumably you mean “Task Manager”, but if a “Task Master” exists, I expect it also only does hard stop.

Thank you.