I just ran a new small Sia test batch. I notice specifically that the dblock it downloads for verification at the end of the backup job is not removed after the job completes. Also other temp files seem to be left behind, though I’m less sure where those residuals are from. BTW, I created a new/clean temp file folder just for this backup set.
Forum Request: can we have a category for “Bug Reports” since this is a bit tangential to “Support”? I suppose I could file bug reports on Git, but unless that’s absolutely mandatory I’d rather concentrate my activity here mainly.
I was running a job just to test, looking at the temp files folder in windows and watching the Information-level log output in Duplicati. It did the Get during the verification phase, downloaded the dblock (creating the temp file), completed verification, and the temp file didn’t go away. I refreshed / checked again later just to make sure, but it was still there.
As an aside, it looks like there are several quirks (on Sia’s side I assume) that cause potential issues with Duplicati, which will need careful stress testing (and hopefully improvements in Sia’s client, as well as maybe some necessary tweaks in Duplicati to handle them).
One of the bigger ones i’ve observed several times now:
wallOfText
after being up for several hours Sia will just… fail to upload anything anymore, and any new uploads stall at 0.0%. The only way to get uploads to resume again is to restart the Sia client. This is a problem with Duplicati because as soon as the Sia client is shut down, Duplicati immediately starts retrying uploads of renamed dblock files. Somehow in a test batch I was running yesterday I ended up with a dblock still in duplicati’s database but it had uploaded 0%, and the only way to fix that was to do a purge-broken-files. Obviously the biggest issue here is Sia failing, but there may be some things Duplicati can do more carefully to avoid the issue with orphaned, non-uploaded dblock files making it into the indexes and/or database.
That should not really be possible. Duplicati will wait until the upload is complete, before marking the file as “uploaded”. If something stops before this, the file will just be removed from the database, and deleted should it show up at the backend.
As I’ve mentioned before, the Sia daemon seems to become unstable (or something) after several hours of running, and at the start of a random upload it will stall at 0% uploaded (and would continue to sit at 0% for 12 hours if left alone). The only way to abort within Duplicati is to do the “stop now” option, but the 0% uploaded file is not removed from Sia, and when trying to run a subsequent backup job, will cause issues in Duplicati (esp since it seems Duplicati notices the file’s presence, but doesn’t notice it’s empty).
Yes, that could explain it. If the file has the correct size when listing it, Duplicati will think it has successfully uploaded, and promote it to a real file. If it later goes away, you have the problem.
I’m not sure how Sia reports filesizes to the outside - I wouldn’t be too surprised if the file SIZE is reported at 100% from the beginning of upload, even if it’s at 0.1x redundancy (or 0.0x in my case) - so to handle this duplicati might need to check the redundancy level instead of the size when determining whether a remote file was completed. Also it’d be great if after being stuck for a while on an upload, there was a way to cleanly abort the duplicati job without having to kill the task through task manager - not even “stop now” works for certain phases of the backup job.