Keep getting System.AggregateException and System.IO.FileNotFoundException. Need help

I’m pretty close to lost. Might need to gather some logs on how this breaks. Meanwhile I misinterpreted this:

(sorry for thinking out loud – you don’t need all the details, but perhaps they will help somebody somehow)

“No filelists found on the remote destination” made me think of the no-files-seen issue, but you proved that files were seen by list. Looking up the message text (thanks for providing it), this was actually from code attempting to recreate the database from backend files, but found it had no dlist (list of files) to start from. Because the dlist contains backup details, it can’t be finalized for uploading until the very end of the backup.

The 3 usual get operations are sampled from remote files known to the database, but no DB may mean no samples. The question becomes – where did the database go? Unfortunately, I don’t do Docker, but think it sometimes becomes an issue of where to keep persistent data. Keeping it outside the container solves the problem of losing it all when you replace the container with a new version, but it needs some special steps.

Duplicati on Synology via Docker?! is a writeup by the person who you worked with on your previous issue.
Docker container sqlite location is an older discussion.
Can’t repair database on synology was one where a 512 MB Synology couldn’t do Recreate due to lack of memory. That initial crash was due to lack of temporary file space. tempdir and other settings control that.

Because you’re on a small system (what memory size?) that can run a small backup but not a larger one, possibly there is some sort of resource issue that arises. I don’t know how it leads to the original message whose source is not known, but appears to be the result of previous problems (which a log might capture).

Duplicati uses temporary file space (I guess from your error yours is at /tmp – is that inside the container?) heavily for accumulating information and staging files for upload to destination. Can you watch free space?

Can you check the job Database tab to see its path, then figure out where that really is? Watch free space. You can also watch the database itself. Especially from a clean start, I’d expect it to start small then grow.

I’m not sure why it’s in the recreate code. I think in some cases it does a repair before backup if it feels the need (and this probably can be seen in the log), however (per repair link), a recreate is due to no database. Specific issue of no filelist is probably because it never got far enough in initial backup to upload the dlist.

A log file was suggested earlier to try to see what got the 504 from NextCloud. That log level was at retry, which is good because it doesn’t show private info like paths, however there are higher levels if necessary.

You can possibly gather some information on any stuck-here situation like the “No filelists found” (is there a file with dlist in its name in the backup destination area) this way, or I suppose you could just try to restart by deleting the database (Delete button) and the corresponding NextCloud files using some manual delete.

I’d be happy if @drwtsn32 has any thoughts on setting up Docker on Synology, and how to check issues.

“No filelists found on the remote destination” made me think of the no-files-seen issue, but you proved that files were seen by list . Looking up the message text (thanks for providing it), this was actually from code attempting to recreate the database from backend files, but found it had no dlist (list of files) to start from. Because the dlist contains backup details, it can’t be finalized for uploading until the very end of the backup.

The 3 usual get operations are sampled from remote files known to the database, but no DB may mean no samples. The question becomes – where did the database go? Unfortunately, I don’t do Docker, but think it sometimes becomes an issue of where to keep persistent data. Keeping it outside the container solves the problem of losing it all when you replace the container with a new version, but it needs some special steps.

This actually made me understand the process more, but i still have no idea what caused the problem. Thanks for thinking out loud.

I have mounted the container /config folder to a folder on my NAS. I think i remember seeing .sqlite files in the folder but it is currently completely empty. Is that normal? I still have an old linuxserver/duplicati container which i haven’t used anymore since switching to duplicati/duplicati but when looking in the config folder for the old linuxserver/duplicati i do see a bunch of config files with .sqlite files.

Because you’re on a small system (what memory size?) that can run a small backup but not a larger one, possibly there is some sort of resource issue that arises. I don’t know how it leads to the original message whose source is not known, but appears to be the result of previous problems (which a log might capture).

My Synology NAS has 8GB RAM with an Intel Pentium N3710. Usually there’s only 35% of RAM is use, and the Duplicati container is not limited in resource usage.

Duplicati uses temporary file space (I guess from your error yours is at /tmp – is that inside the container?) heavily for accumulating information and staging files for upload to destination. Can you watch free space?

I have tried mounting and not mounting the /tmp file to the host filesystem because i had the same idea with free space. With both options everything that’s been going on still happens. I have about 9TB free on the NAS so it shouldn’t be running out of space when it’s mounted, and since it does the same thing when it’s not mounted i guess it’s also not a space issue in that case. (I’m not sure where the files are when they are not mounted)

Can you check the job Database tab to see its path, then figure out where that really is? Watch free space. You can also watch the database itself. Especially from a clean start, I’d expect it to start small then grow.

I do see a problem here. All databases for each backup say they are in /data/Duplicati, which is not a path i have mounted to the host and thus must be getting cleared every time the container restarts, and i have done that a couple times when trying to solve problems. I will try to mount /data to the host filesystem now and see if that makes a difference. You’ll hear about this on my next reply.

Can it be that linuxserver/duplicati uses /config and duplicati/duplicati uses /data? I had mounted /config, not /data. If this is really the cause of all of these issues (not sure how this would cause a 504) i’m going to be very frustrated.

Now that i have mounted /data to the host filesystem, my backup tasks are gone completely. That kind of sucks, but i guess i can set them up again with the exact same settings. This hasn’t happened before on container restarts so maybe the /data folder never got cleared like i said i thought had happened before.

Edit: I also suddenly don’t have an encryption option anymore?? It just says “No Encryption” instead of the 256 bit AES encryption that i used before. What can cause this??

Edit Edit: The encryption option is back after a container restart. I am so confused.

A log file was suggested earlier to try to see what got the 504 from NextCloud. That log level was at retry , which is good because it doesn’t show private info like paths, however there are higher levels if necessary.

I’m sorry i must have read over this. I’ll add the log options to the backup task and see what that produces.

You can possibly gather some information on any stuck-here situation like the “No filelists found” (is there a file with dlist in its name in the backup destination area) this way, or I suppose you could just try to restart by deleting the database (Delete button) and the corresponding NextCloud files using some manual delete.

I cannot find a file with ‘dlist’ in it’s name on the 3TB backup, but i do see one on the 300GB backup.

Edit edit edit (3 hours later): i let the 300GB backup recreate the database and it suddenly says theres 15k files locally missing. That can’t be right so im even more confused than before now. I guess i’ll redo that backup, and if it says the same for the 3TB backup i guess i’ll redo that one too. I have no clue what’s going on at this point.

Yeah, the two docker images are quite different in where they place data. Personally I would stick with the official image but you’ll definitely need to update your mappings if you haven’t done so already.

I also have this running on a Synology NAS, as @ts678 mentioned. I have not seen the issues you describe but I also use a different back end.

Answered at least tentatively (before issue reverted to unsolved) in the topic I linked.

I can’t do Docker specifics, but I did click on the links and they still appear to say what the post said.

Configurations are in Duplicati-server.sqlite which is usually right next to the per-backup files.
Getting everything that you need onto the host, if possible, would save you some redoing of things…
I’m not sure where your browser is, but it can export configurations to files (keep for safety anyway).

I’m not sure about that one, but some of the UI takes awhile to populate data. System info is that way.
Sometimes fields will say something like “Unknown” for things that aren’t yet known from server data.

That makes sense because the 300 GB finished, while the 3 TB hasn’t finished even its initial backup.

I can’t find an error message like that, even loosening and using a regular expression. Can you quote?

Hello, thanks for sticking with me.

Answered at least tentatively (before issue reverted to unsolved) in the topic I linked.

Thank you. I had originally started with linuxserver/duplicati before someone on this forum told me to switch to duplicati/duplicati and apparently i had not realised that the mappings are different. That is a stupid mistake on my end.

Configurations are in Duplicati-server.sqlite which is usually right next to the per-backup files.
Getting everything that you need onto the host, if possible, would save you some redoing of things…

Unfortunately it looked like everything was cleared once i mounted the /data folder. I unmounted it to try and see if my configuration was back, but it wasnt.

It was

Duplicati.Library.Interface.UserInformationException: Found 15774 remote files that are not recorded in local storage, please run repair

I have deleted the remote 300GB backup though and am redoing the entire thing. I just want to be safe. I don’t fancy redoing the 3TB though, that 1TB it’s done so far takes like 2 weeks. If it says the same message on that one i’ll report back. I can start it after the 300GB backup is done which will probably be tomorrow or the day after. It’ll have to recreate database which will take a couple hours and then i’ll see what it has to say. Thanks for the help so far, i appreciate it a lot.

The easiest way to get that is to lose the DB, which then thinks there should be no remote files.
You can see if the destination has 15774. If so, that’s probably it. This can happen if one deletes
the DB on purpose to try to get a fresh start (solution is also deleting the backup files manually).
In your case, the disappearing DB issues might have caused a new DB to see the old backup…

EDIT:

If the goal is to recreate the DB from backup (to avoid backup uploads), use Database Recreate.
Generally this should be pretty fast, but if information is missing it can wind up in a big download.

I thought it meant that there are remote files found which are not in local storage, and when i click repair it’ll download the files from the remote backup and put them in my local storage. That’s why i didnt want to press repair, i didn’t want to mess up my local files.

So what do you suggest doing on the 1TB backup task once the 300GB current one is done? Do i press repair on the message since i’ll probably get it if i understand right? Or do i go to Database -> Repair there? Is it the same repair button? Will pressing repair on the error not edit my local files, but only repair the database?

If local files means source files, Repair won’t touch those. It’s a DB process to reconcile with the backup, however there’s a known issue when an old DB is restored (e.g. from an image backup) and Repair runs.

Apparently the stale DB is a convincing enough DB that agreement with remote is made by deleting new files from remote. Not good. I just did Repair to a totally new empty DB, and it rebuilt the DB from remote. Using the Recreate button deletes the old DB first, so there’s no chance of accidental damage to remote.

This is the one that got to 1 TB and never finished (and never made its dlist, so it won’t recreate) I believe.
It’s also the original debug target, but it takes two weeks to get to the 1 TB. If there’s anything left of its DB then looking inside may be possible. If the DB is gone, and no log files exist, then all there is is your recall.

You can see if the DB in the Database page has a plausible size. if it’s below 1 MB it’s probably too empty. You can also see if you can spot any history on anything in the job’s Show log General or Remote pages. Finding older history would mean there’s something to look at or work on. If it’s extremely new, that’s bad.

Duplicati can ordinarily resume an interrupted backup from where it left off, but yours was getting an error. Possibly the problems in Docker contributed, but I guess first step is to see if there’s any sign of a DB left.

I do have an experimental method that “might” be able to make use of your backup files even if DB is gone, however I’d prefer if the original DB is still there, as there’s a chance it has clues. A big log might do better, however it’d be nice if we didn’t have to wait 2 weeks for something to fail before having some information.

I suspect the Repair button on the 1 TB partial will wind up in “No filelists found on the remote destination”, because I think Duplicati runs repair internally before the backup if needed, and that might go into recreate.

The REPAIR command

Tries to repair the backup. If no local database is found or the database is empty, the database is re-created with data from the storage. If the database is in place but the remote storage is corrupt, the remote storage gets repaired with local data (if available).

but there’s a lot of uncertainty about how much of a database existed at what times during all the testing.

If local files means source files, Repair won’t touch those. It’s a DB process to reconcile with the backup, however there’s a known issue when an old DB is restored (e.g. from an image backup) and Repair runs.
Apparently the stale DB is a convincing enough DB that agreement with remote is made by deleting new files from remote. Not good. I just did Repair to a totally new empty DB, and it rebuilt the DB from remote. Using the Recreate button deletes the old DB first, so there’s no chance of accidental damage to remote.

Well i redid this entire backup because i thought the message meant something else than it did, so i guess that works too.

This is the one that got to 1 TB and never finished (and never made its dlist, so it won’t recreate) I believe.

Yes i’m sorry, i called it the 1TB backup task but it should have been the 3TB backup task.

You can see if the DB in the Database page has a plausible size. if it’s below 1 MB it’s probably too empty. You can also see if you can spot any history on anything in the job’s Show log General or Remote pages. Finding older history would mean there’s something to look at or work on. If it’s extremely new, that’s bad.

I don’t think there’s anything left of the DB. My entire Duplicati got cleared when i mounted the /data folder. When cllicking on the job and then show log, it says “Failed to connect: SQLite error no such table: LogData”.

I suspect the Repair button on the 1 TB partial will wind up in “No filelists found on the remote destination”, because I think Duplicati runs repair internally before the backup if needed, and that might go into recreate.

I can try this to see if the message occurs. The 300GB backup task is at 80GB left, it should be done later today.

So you suggest trying to repair, and if that doesn’t work i come back here to try the experimental method you mentioned?

Also i just wanted to mention again that i’m very grateful for your help.

The 3TB backup task, as expected, returns a “No filelists found on the remote destination”. Going to Database -> Delete & Repair does the same, as you expected.

What is the experimental method that might work? Does it affect data integrity at all? Obviously i can’t have that, but if it doesn’t and it’s only experimental because it’s not guaranteed to work, then i’m willing to try it.

I just hope that if it doesn’t work for some reason and i have to redo the entire backup, i don’t get stuck at the same point.

It’s described here. It’s basically making a dummy dlist with no files listed, to get Recreate to populate block information into the DB so that blocks already uploaded are reattached to any files that have those blocks…

It’s basically replacing one file in a .zip file. It sounds like you encrypt, but using AES Crypt can do that part.

It’s just a Recreate helper, and risks were discussed earlier. Basically, if you mean source data you’re fine.

Recreate button, if blue, will delete the DB thus it can’t think of deleting remote files (which are less critical than source data, and in fact if this doesn’t work we’re probably going to have to throw away the partial…).

If DB is not present at all, I think Recreate won’t go blue, but Repair button will turn into database recreate.

Assuming this loads block info successfully, you’d want to watch some of the files that go by on the home screen or live log at verbose level, and at least do a test restore of those to see if they backed up properly.

So none of this should endanger source files, but it’d take more doing to get convinced that backup is fine. What I’d probably do if you’re up for trying to save two weeks of uploads is do some more testing myself…

One advantage of the two-weeks-then-fail plan is it could generate a log that might show what went wrong. Going that route should probably bump blocksize up from its default 100KB, because 3 TB of 100 KB gets 30 million blocks for the database to track, and it slows down operations (Recreate shows it quite heavily).

Especially given video which doesn’t deduplicate well, something like 5 MB blocksize might be reasonable.
Choosing sizes in Duplicati gets into this some. A big note is blocksize on existing backup cannot change, which possibly will drive this effort into a 1 TB reupload just to get the blocksize more appropriate for 3 TB.

But this other path should be pretty fast to play with, and we might learn something without a 2 week wait:

Trying to read current upload back into a DB might be interesting to see if it even starts, fails soon, or does another two weeks. Is network simple (e.g. typical home network) with no boxes in the middle to time out?

Trying to debug something that takes so long to fail is certainly awkward, and I don’t know where it’ll land…

Thank you for the writeup, but reading this and considering how valuable the source data is, and how important it will be to have a proper backup solution that i can rely on for years to come, and how i have absolutely no idea where to start with the described process, i think i will just redo the entire thing and hope it goes well this time.

Going that route should probably bump blocksize up from its default 100KB
Especially given video which doesn’t deduplicate well, something like 5 MB blocksize might be reasonable.

You mean i should change the blocksize on the task to 5 MB so there will be less database entries? That sounds like a good plan actually.

Choosing a large value will cause a larger overhead on file changes,

I’m not entirely sure what this means.

Is there anything else i need to do before restarting the task from scratch? Any flags i need to add to the task? I have the following flags on it at the moment:

auto-cleanup: Yes
blocksize: 5MByte
log-file: /data/video-log
log-file-log-level: Retry
upload-verification-file: Yes
zip-compression-level: 9

Trying to read current upload back into a DB might be interesting to see if it even starts, fails soon, or does another two weeks. Is network simple (e.g. typical home network) with no boxes in the middle to time out?

Yes typical home network on both sides.

Trying to debug something that takes so long to fail is certainly awkward, and I don’t know where it’ll land…

I know but i don’t think i have another option.

Also before i start it, is there a way to know which file was being processed at the moment an error occurred? I’ve been wondering if it has been happening on one specific file or not.

That was just the rough writeup, to gauge interest. There are other steps, but ultimately it’s still an experiment, and just the blocksize change means you might wind up with a performance problem.

Please get a log as described earlier. If you prefer, using verbose level is more informative without completely overwhelming size (like profiling will make), but you’d have to sanitize before posting it.
Logs at retry level might be enough, and are likely postable. OTOH I’d sure hate to repeat this drill.

While it’s backing up, you could watch the DB grow, and you should see dup-* files in temp area.
Most will probably be transient files that are created, uploaded, then deleted. Some are long-term.

I’m curious what type "Could not find file “/tmp/dup-6f255783-2945-47fe-8786-8f3f19ece462” was,
however the naming doesn’t distinguish. There are some extreme measures to take to gather info,
however they don’t scale up. I’ve used Sysinternals Process Monitor to watch temp file action, and
also used a batch file to copy all dup-* files to another folder in case I wanted to look at one later…
If you have extra TBs of space available, that might work, but in any case we’d still want the log file.

For anything very valuable, two backups (done differently) is a good idea. Software is never perfect.
Do these videos have valuable old-version info? If it’s all just-the-latest, maybe direct copies will do.
That would give you more choice on how to get the copies, although 100 GB size may pose issues. Duplicati’s strength include deduplication and compression to keep multiple versions compactly, but
video is good at defeating both of those. Unless you edit and want to undo, versions may be overkill.
OTOH versions are handy if ransomware clobbers things. You don’t want to clobber backup as well.

Yes, and I picked 5 MB because of my initial test results that showed a slowdown at 1 million entries.
There’s not much solid data, except that large backups get slow in their SQL queries and Recreates.
Currently the blocks are inserted one at a time, and recreate has to do every one of them that way…
Backups are better at (sometimes) hiding the slowness because they start with an existing database.

How does Duplicati Resume After Interruption? is an old post from the Duplicati author suggesting not using --auto-cleanup, but possibly that being set explains why repair ran (and maybe went to recreate).
That also describes the experiment that we’re not trying, which is to try to patch up DB for continuation.

Initial backup is especially troublesome (as you saw) due to no dlist file, so one way to proceed is to do smaller subset backup initially (maybe most important first?) to get at least the initial backup in the files.
If something breaks later, at least you can recreate the DB, but we hope the DB loss issue is over now.

Processing is parallel. At higher log levels, you first see the file getting past the exclude and other filters. Actually reading through the file is (I think) log-file-silent. Data blocks collect in temp files, then upload as filled to 50 MB (default dblock-size a.k.a. Remote volume size on the options screen), but they queue…
asynchronous-upload-limit controls how many queue. It’s a good question, but difficult to answer without even much understanding of whether it’s related to a source file, a file-made-for-upload, or the uploading.

What’s interesting is that SpillCollectorProcess is trying to do something. Ordinarily its job is to collect the leftovers after concurrent file processors have finished their work of filling dblock files. The end isn’t even. Your backup was seemingly nowhere near the end so I’m not sure why the code is running where it looks.

Here’s an example Verbose log of a small backup. I’m posting lines with three backticks above and below which help with the formatting (and scrolling), but for a really big file you can zip it up and drag to window.

2020-10-06 16:03:48 -04 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Backup has started
2020-10-06 16:03:51 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started:  ()
2020-10-06 16:03:51 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed:  ()
2020-10-06 16:03:51 -04 - [Verbose-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-IncludingSourcePath]: Including source path: C:\backup source\length1.txt
2020-10-06 16:03:51 -04 - [Verbose-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-IncludingSourcePath]: Including source path: C:\backup source\short.txt
2020-10-06 16:03:51 -04 - [Verbose-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-IncludingSourcePath]: Including source path: C:\backup source\length1.txt
2020-10-06 16:03:51 -04 - [Verbose-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-IncludingSourcePath]: Including source path: C:\backup source\short.txt
2020-10-06 16:03:51 -04 - [Verbose-Duplicati.Library.Main.Operation.Backup.FilePreFilterProcess.FileEntry-CheckFileForChanges]: Checking file for changes C:\backup source\length1.txt, new: True, timestamp changed: True, size changed: True, metadatachanged: True, 10/4/2020 1:42:38 AM vs 1/1/0001 12:00:00 AM
2020-10-06 16:03:51 -04 - [Verbose-Duplicati.Library.Main.Operation.Backup.FilePreFilterProcess.FileEntry-CheckFileForChanges]: Checking file for changes C:\backup source\short.txt, new: True, timestamp changed: True, size changed: True, metadatachanged: True, 10/4/2020 6:56:47 PM vs 1/1/0001 12:00:00 AM
2020-10-06 16:03:51 -04 - [Verbose-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-NewFile]: New file C:\backup source\length1.txt
2020-10-06 16:03:51 -04 - [Verbose-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-NewFile]: New file C:\backup source\short.txt
2020-10-06 16:03:51 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-b2c52d2a1185c4cd280e6f6b14133f540.dblock.zip (1.11 KB)
2020-10-06 16:03:51 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-b2c52d2a1185c4cd280e6f6b14133f540.dblock.zip (1.11 KB)
2020-10-06 16:03:51 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-20201006T200351Z.dlist.zip (748 bytes)
2020-10-06 16:03:51 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-ib2eaa999e8d44fe08d64af2e88704c82.dindex.zip (688 bytes)
2020-10-06 16:03:51 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-20201006T200351Z.dlist.zip (748 bytes)
2020-10-06 16:03:51 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-ib2eaa999e8d44fe08d64af2e88704c82.dindex.zip (688 bytes)
2020-10-06 16:03:51 -04 - [Verbose-Duplicati.Library.Main.Database.LocalDeleteDatabase-FullyDeletableCount]: Found 0 fully deletable volume(s)
2020-10-06 16:03:51 -04 - [Verbose-Duplicati.Library.Main.Database.LocalDeleteDatabase-SmallVolumeCount]: Found 1 small volumes(s) with a total size of 1.11 KB
2020-10-06 16:03:51 -04 - [Verbose-Duplicati.Library.Main.Database.LocalDeleteDatabase-WastedSpaceVolumes]: Found 0 volume(s) with a total of 0.00% wasted space (0 bytes of 285 bytes)
2020-10-06 16:03:51 -04 - [Information-Duplicati.Library.Main.Database.LocalDeleteDatabase-CompactReason]: Compacting not required
2020-10-06 16:03:51 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started:  ()
2020-10-06 16:03:51 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed:  (3 bytes)
2020-10-06 16:03:52 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-20201006T200351Z.dlist.zip (748 bytes)
2020-10-06 16:03:52 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-20201006T200351Z.dlist.zip (748 bytes)
2020-10-06 16:03:52 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-ib2eaa999e8d44fe08d64af2e88704c82.dindex.zip (688 bytes)
2020-10-06 16:03:52 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-ib2eaa999e8d44fe08d64af2e88704c82.dindex.zip (688 bytes)
2020-10-06 16:03:52 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-b2c52d2a1185c4cd280e6f6b14133f540.dblock.zip (1.11 KB)
2020-10-06 16:03:52 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-b2c52d2a1185c4cd280e6f6b14133f540.dblock.zip (1.11 KB)

I have to be honest, i’m getting slightly overwhelmed with information now.

Ok, so:

I’m ready to start the 3TB backup process from scratch and hope everything goes better now, for whatever reason even though the only thing that has changed is the data/ folder, since i feel like this is really the only chance i got. I have put logging on verbose on the task and put block size to 5MByte.

What do you suggest putting “Remote volume size” on? All the tiny 50 MB files were getting slightly difficult to search through when you asked me to find a file in them, and that was with only 1TB done. Is leaving them it on 50 okay? Or should i increase it to decrease the amount of files on the remote system?

Software is never perfect.
Do these videos have valuable old-version info? If it’s all just-the-latest, maybe direct copies will do.
That would give you more choice on how to get the copies, although 100 GB size may pose issues. Duplicati’s strength include deduplication and compression to keep multiple versions compactly, but
video is good at defeating both of those. Unless you edit and want to undo, versions may be overkill.
OTOH versions are handy if ransomware clobbers things. You don’t want to clobber backup as well.

It’s just the latest honestly and i have thought about direct copies too, however versioning with randomware seems like a plus, and the fact that if i delete a file it’ll still be in my backup for a couple of versions before it’s completely gone seems good too. I also wanted to compress as much as possible but i do see now that the compression is pretty minimal, understandable.

For anything very valuable, two backups (done differently) is a good idea.

Yes i do know that, i’ll be looking into another backup option too but just having this one is already a million times better than none at all.

Duplicati author suggesting not using --auto-cleanup, but possibly that being set explains why repair ran (and maybe went to recreate).

“Cleanup” sounded like a positive thing that’s kind of why i added it. I have removed it from both tasks now.

I did read the rest of your post but i’m not sure what to say to all the information. Thanks for explaining and i’ll keep everything in mind. I’ll be waiting for an answer before i start the 3TB task again.

The “Choosing sizes” document talks about that some. There aren’t really any hard-and-fast rules on this.
Especially if you use versions of files (which may or may not fit video), restoring a version may take many dblock files, as the updates will certainly land in a different file (and non-updated parts use original dblock). Compacting can mix blocks around too. If you delete files and create waste space, rearranging gets done.

Basically, larger remote volume size means possibly slower restores. It might not matter if this is just DR. When doing a restore of everything, all dblock files will likely download. For limited restores, it may matter.

Download speed is also a factor. If fast, larger volume might not hurt as much (if NextCloud can keep up).
If lots of smallish files are annoying, and your connection is fairly fast and fairly reliable (we will see when something finishes and “RetryAttempts” in the log is seen – you can keep checking your working backup), increasing this to 500 MB might be reasonable if you don’t mind the possibility of restores getting slower…

Because you have “just the latest”, if you also have few deletes, then probably you’ll just keep on using the original upload dblocks, and not have to go chasing around for updates, so larger volumes will slow less…

EDIT:

Large volumes increase the chance of timeouts. What might be nice is the small initial backup I proposed, then a restore to see if that seems to be working, then either add gradually or go for broke with rest of files.

Download speed is 500 Mbit/s, but upload speed at the backup server is rather slow at 20 Mbit/s. I’m already fearing the day i need to restore a backup with that slow upload speed at the other end.

You named 500 MB which was actually a lot higher than i originally had in mind. I was thinking about doubling or tripling the default 50 MB.

I honestly simply had absolutely no idea what to do, so i just chose a value inbetween your thoughts and my thoughts: 250 MB.

Before starting the backup i also went into the configuration of the Nextcloud server (the server the backups are running to) and i increased the maximum execution time to the max (3600 seconds) to try and prevent the 504 errors, since these are timeouts. I thought that might help.

The backup has been running for about 5 days now and i should say i vastly underestimated the speed. I said 1TB takes about a week but it has done about 1.5TB now in these 5 days. The total backup size to be precise is 2.8TB, and currently it’s on 1.2TB remaining. The previous 504 errors usually occurred around 1.8TB remaining, and since it has already passed that point my hopes are high. I don’t want to speak too soon, but i have a feeling everything might work out this time :). I’ll report back once the backup is fully complete.

Should be fine assuming no timeouts. Value is quite arbitrary and depends on too many things to optimize. Changing it later is possible, but affects only new volumes (which could also be produced by compacting).

One other thing that can happen (that ideally should not happen) is the earlier mentioned Recreate (which ideally is never needed) can download the whole backup as opposed to just its small dlist and dindex files. Runnng in the 90% to 100% range on the progress bar is going to be slow. Live log at Verbose shows info.

Carefully testing occasionally by moving the database aside temporarily to see if Repair can rebuild it fast would give you early alert of a possible problem on a necessary rebuild, but can’t help if source drive dies.

Of course things can never go my way lmao.

Just as the backup was done i got the following error: “The remote server returned an error: (423) Locked.”.

Verbose logging was on. The log file has 102k lines. The last 300 ish lines (because i think those are interesting) can be found here: 2020-10-13 20:53:09 +00 - [Verbose-Duplicati.Library.Main.Operation.Backup.FileB - Pastebin.com

Once i again unfortunately i do not know what to do in this situation.

As far as i can tell, for some reason it says the file /video/2002/2002-07-28 Spain/108-0894_MVI.AVI has been edited since the last time it checked the file but i can confirm no one edited the file. It’s a 3.2MB Video file with last modified date 2002. I asked the other person who could have done anything to the files but they say they haven’t been doing anything. I do not understand what is going on.

Also i noticed it is backing up files from #recycle and @eadir folders. I should probaly exclude those because those are system folders from the NAS. Do i just go to Source Data -> Filters -> Press “Exclude Folder” and enter @eadir ? Is it that simple? Should i do this while problem solving or wait for a succesful backup first?

I can see that it tried 5 times to delete a file called duplicati-idea09091bb4e43699ae1bb7c1359de07.dindex.zip.aes and each time it got a 423 Locked error. At the time of this error (2020-10-13 21:36:21) i do not see anything in the logs at the remote server.

What do you suggest doing? Maybe just try run the backup task again?

It does look done, or pretty close. I see the dlist file put up at end of backup, and the verification file too.

There might be some cleanup code trying to delete files that didn’t make it up, so were retried under a different name. That’s the design, and the file that got the upload error gets a delete attempt right then.

Maybe there’s another try at the end. Can you search the whole log for early history of the named file?
Database should also have some records, if you’re willing to go browsing the RemoteOperation table.

Do you control the remote server? I wonder if rebooting either it or local system would unlock the file?
Less disruptive than reboot if remote is shared may be WebDAV restart. You can also research error.
I’m not immediately seeing Duplicati locking files, but I wonder if the server can’t delete during upload.

Got lines that show that? What I see for that path is that it looks like a newly backed up file, not an edit:

2020-10-13 20:53:13 +00 - [Verbose-Duplicati.Library.Main.Operation.Backup.FilePreFilterProcess.FileEntry-CheckFileForChanges]: Checking file for changes /source/video/2002/2002-07-28 Spain/108-0894_MVI.AVI, new: True, timestamp changed: True, size changed: True, metadatachanged: True, 08/05/2002 20:32:44 vs 01/01/0001 00:00:00
2020-10-13 20:53:14 +00 - [Verbose-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-NewFile]: New file /source/video/2002/2002-07-28 Spain/108-0894_MVI.AVI

I don’t know the entire backup history and whether it used to work, but at the moment incomplete backups could very easily leave some files not backed up. If not backed up before (check old backups) they’re new.

Does it normally show delete operations I wonder? Or maybe it only shows ones that happen. No idea.

Look in early logs for duplicati-idea09091bb4e43699ae1bb7c1359de07.dindex.zip.aes operations.

Maybe look on remote for file date, to see whether it was even uploaded in the run with good logs.

If you can get technical, browse Database read-only to search via RemoteOperation table header.
Timestamp is in UNIX form. Convert with whatever converter you like, e.g. using EpochConverter.

If the file has early history of not uploading, so needs a deletion, need is remembered in database Remotevolume table having Deleting as State. Delete will be tried again at start of next backup.

I’m not sure it will fare any better next time unless some things get restarted to unlock WebDAV…

You can also research 423 error, including a special search for behaviors of your particular server.

Or maybe it cleared up on its own, and next backup log will show it being deleted without an error.

@eaDir is specifically discussed in the Filters article here, but let’s try to get this backup fixed first.
There’s also a bug in the GUI filter builder, so using its three-dot menu Edit as text does better.

I think i misread the logs.

Yes i did do this but i also wanted to wait for your reply because you’re the expert here :). It seems like the backend (Nextcloud) puts a lock on files when they’re in use, and somehow Duplicati interferred with it’s own lock.

I reran the task and it completed in about a minute. I think i will run a “verify” task now to verify the integrity of the backup. Does that seem like a good idea?

Also i noticed that the compression - as expected - literally only saved 2 GB on 2.7TB. Does turning compression off entirely for that task specifically increase upload speed OR decrease cpu usage at all? To maybe completely skip an attempt at compression on each file. Also is this still something i can do while i already made one successful backup? I did read there’s a list of default extensions that Duplicati will not compress and .mp4 and .avi is the list.

Once again i want to make clear that i’m extremely grateful for your help and once i get everything going good and i have done a couple more successful restore tests i will absolutely consider donating to Duplicati. The support i have been getting on this free piece of software has been hundreds of times better than some support for some paid software.