I have encountered an issue with Duplicati 2.1.0.5 that I can’t explain. One of my backup jobs is stuck.
The job ran at 10:00PM, on schedule. After the job was done, it appears to have decided to run the Compaction process. The backup set for this job is about 350 GB, and my upload speed is only 10 Mbps, so when this happens it generally takes a while.
The process appears to have run normally, downloading and uploading until about 3:20AM, where it just … stops. The progress bar says “Deleting unwanted files”, but nothing seems to be happening anymore. I didn’t notice anything had happened until my monitoring solution reported that all my backup jobs hadn’t reported in lately.
Not really. You would need to have a debugger attached to figure out where all the threads are.
It is a bit strange though, because the delete step is fairly straightforward with no concurrency.
You can look at the verbose logs, but they do not get generated unless you view them (for performance reasons), so most likely nothing will be there if the system just hangs.
What version of Duplicati, operating system and storage type are you using?
Using the latest Stable version (2.1.0.5) on Windows 10 Pro. The storage back end is Microsoft OneDrive v2.
This job has also started behaving in another strange way. When the job got stuck, I eventually terminated the Duplicati process and then restarted it. When the job in question finishes, it still wants to compact, but the compaction process is behaving strangely now. Usually during compact it will follow a process like this: Get, Get, Get, Put, Delete, Delete, Get, Get, Get, Put, Delete, etc. as it pulls down files, repacks them, uploads them, and then deletes the originals.
Now, when it compacts it is downloading and uploading, but there are never any deletions. I eventually stopped the job and disabled compaction as it had been running for a very long time, and I didn’t have time a the moment to play with it.
I checked through the logs, and there is nothing to indicate something is wrong… I have just never seen it behave like that before.
The way the compacting looks now, is that it will wait until the end before it deletes files it does not need. I assume it has been that way for a long time, but I think @ts678 was also pointing out that this was new?
was the same early conclusion that was solved after hearing deletes are at end.
is my concern about what that means. This hits super hard if remote volume size change causes compact. Dealing with full destination was tough even before this. Previously if one could free up a tiny space, compact could make more. No more.
The behavior seems new to me - I had assumed that the compaction process had always been get/put/delete.
Is there a technical reason to leave all the deletions to the end? I suppose it is safer by some measures… but as @ts678 says, it can make dealing with space-constrained storage quite a bit more difficult.
I think that was the logic. Avoid complicating things by removing files before we know we have a good state. But… we know we have uploaded everything needed when we mark the old file for deletion, so it should be safe.
