Sorry, I thought I was clear. Yes, this database has lost configs. It is not a new database.
I did not make any changes to startup parameters, or alter any settings. I run it as vanilla as I can. That said, is it possible those were set in the docker image?
This is what my settings look like. I definitely havenāt changed them:
I only ever remember seeing this popup when I deleted and restarted everything weeks ago. It definitely does not appear when the configs disappear.
So those are two points saying that itās not like the database itself disappeared totally.
The point suggesting it did is that your re-entry started at ID 1. Backup table design is
and you can probably see the same layout in the browserās Database structure tab.
SQLite Autoincrement explains how AUTOINCREMENT works. Iām not an SQL expert:
If the AUTOINCREMENT keyword appears after INTEGER PRIMARY KEY, that changes the automatic ROWID assignment algorithm to prevent the reuse of ROWIDs over the lifetime of the database. In other words, the purpose of AUTOINCREMENT is to prevent the reuse of ROWIDs from previously deleted rows.
sounds like it asks explicitly for non-reuse, so itās odd that ID 1 was reused, maybe from:
I suppose itās possible that somethingās bad in the database. If so, a new one might help.
You could test like I did, adding some dummy job, deleting, adding, seeing ID not reused.
Weāve also discussed some things to keep track of, whichever way you decide to pursue.
This is how my set up stands right now: 3 jobs, all manual, all have just run as expected. Iāll keep monitoring and report back if / when the jobs either disappear, or remain.
ts678, are you a Duplicati dev, or work for Duplicati?
Not really a Duplicati dev. Just a volunteer. Duplicati exists only through those, and we could use more.
Did you try to reattach the old databases? I suppose with three you might need a map (from database screen, while configs are still there) of which file goes with which job. You can also just put a DB into a test job simply for purpose of entering Restore to look over its backups. Donāt actually change backup.
If any job database got damaged, was it at the same time as server database? Thatād be a major clue.
which is good, as it answers the question of whether loss coincides with a scheduled job run, but does it coincide with anything? Although the server might occasionally poll for updates (and we probably saw its timestamp in the server database), things are, I think, very quiet when nothing is being asked to happen.
You could keep an eye on your file names, sizes, and timestamps to watch for any unexpected changes.
At the moment, thereās a lot of conflicting information, but speculating further might be all thatās available, somewhat limited by expertise gaps. If any readers are expert in anything discussed, feel free to jump in.
The 56 KB .fuse_hidden file, according to Google, is a file that was deleted while open. Ordinarily, Linux just makes the file disappear from its directory while still used as usual by the process thatās still using it.
From the name, I guessed this was a FUSE filesystem thing, and found mergerfs talking about its usage and future plans. If I move page to a newer Ubuntu version, it looks like various versions do indeed vary.
Do you know of any FUSE (or perhaps any UnionFS) around. I think Docker might use several varieties. Problems at filesystem level might be one way to roll back Duplicati-server.sqlite to some earlier content, and 56 KB happens to be the size of a freshly-made no-configs-done Duplicati-server.sqlite on Windows. The fileās date and time info might be relevant, although would 12:27 AM be a time youāre usually active?
Instead of keeping manual records, you could just type a bash loop or something to periodically ls -la into a log file. You might even be able to copy the actual Duplicati-server.sqlite unless locking stops that. Maybe at least knowing time will lead to what happened then, ultimately leading to how it lost the config.
Happy monitoring.
Well, I really appreciate all the help youāve given me thus far.
1 Like
You may be on to something. I checked this morning and the Duplicati-server.sqlite has been reset back to a 56 KB near-empty database (ie all backup jobs have disappeared), with the old timestamp from when the database was originally created. Plus I now have a new .fuse file.
The 3 backup databases appear to be untouched.
Looks like Iāve got some FUSE research to do. Do any folks on the forum have this expertise?
Interestingly the configs are being held somewhere because I opened up the Duplicati home page in a different browser (expecting to see a blank page) but instead I see all 3 config jobs
I am also able to run the backup jobs manually without issue (see new timestamp on GODYHLUWZT.sqlite)
and if that applies here, and knowing that Duplicati holds the file open while server is up (I canāt find previous fuser test by you quickly, but if it wasnāt as root or under sudo, it might have missed a use) presumably Duplicati just happily uses the deleted-by-something file unaware itās not in file directory.
FWIW Date modified is 05-Apr-23 10:29 AM ā identical to what Duplicati-server.sqlite reverted to.
What might be special about that time? You can also check the ls -lc date from the Linux view and compare ā that time is a better indicator of when a file changed, even if modification time was written.
The file is meant to detect multiple Duplicati trying to use the same database area ā which wonāt work. Windows exclusive file locking keeps it from being read, but possibly SMB or direct Linux read will do.
EDIT:
If you can read it, thereās probably a PID in it (presumably Duplicatiās PID in the container PID space).
Iāve done something stupid!!!
Full confession: I spun up another Duplicati instance in Docker, on port 8300, to conduct more tests BUT I forgot to change the Docker config directory, so overwrote the Duplicati-server.sqlite (and likely contaminated other files) in the original config directory.
For now, I have recreated the backup jobs in a brand new instance and put them each on a schedule. Iāll check again in the morning to see if the Duplicati-server.sqlite has been reset. Idiot!!
Well, so far so good. The jobs have all run successfully over 2 nights and the Duplicati-serve.sqlite is still intact!
The only difference with this new instance is the directory name for the Docker configs - the old one was names āduplicatiā and the new one āduplicati2ā.
I wonder if the fuse subsystem got snarled up and was interacting with files in the original āduplicatiā folder and causing the database to revert to itās original, unpopulated version. There are no .fuse entries in the new āduplicati2ā config folder.
Iāll keep monitoringā¦
FWIW Apr 5 10:29 AM Duplicati-server.sqlite is in the earliest Linux and Windows screenshots too.
So for awhile you had two containers (probably each using FUSE) pointing at the same host folder.
I donāt know what to expect in terms of how they would step on each other, but apparently they did.
I also tested some with Duplicati on Windows and Linux, and one-per-area seemed to be enforced.
Possibly it needs Docker to evade enforcement, but TBH this isnāt something Iām set up to look into.
Perhaps you could file an Issue when itās well-enough understood to write exact steps to reproduce.
Yeah, I think you are on to something. When I switched between the docker instances I reused the same config folder. It was only my last attempt where (by my stupidity) I HAD to create a new config folder on the host, that seems to have resolved the issue so far, at least.
Thank you so much for all your help. I really appreciate the time you have taken. Iāll continue to monitor and report back in a few days.
Cheers for now
1 Like
Itās been 7 days now, 4 backup jobs have been running on their individual nightly schedules and all is well!
I suspect ts678 was correct when identifying the .fuse (ie hidden) files as being the culprit. They must have been left over from a failed setup. My solution was to use a brand new config folder for Docker, rather than use a cleaned out, existing config folder.
1 Like