I’ve noticed that, in some cases, the file referenced by DUPLICATI__RESULTFILE is deleted before I can use it in my script. The script is configured to be executed after backup, but I see the next:
When backup was finishing I saw in /tmp the next files:
but when the script started to run, the file referenced by DUPLICATI__RESULTFILE had been deleted (in this case DUPLICATI__RESULTFILE=dup-2a807a00-90dc-48d3-ac23-3e172a5882c7)
I guess the file is created just after the backup, and then the script is started after the post-backup operations such as “Waiting for uploading … Deleting unwanted files … Verifying”. If the timeout of the file expires, the script doesnt find the file.
How can increase this timeout or apply any workaround to avoid these intermittent failures?
Does the file exist when your script starts and then disappear during execution or is it gone before your script even starts?
My initial guesses would be:
this is a side effect of the multi-threading rewrite (the ‘cleanup files’ thread triggers before the ‘run scripts’ thread)
the ‘clean up files’ step happens BEFORE an interim step that doesn’t always fire (such as retention policy compaction) after which scripts are run
In both cases, I don’t think there’s a timeout involved - it’s just code running at times other than expected so I don’t know what there’s a simple setting to get around the issue.
Hi @JonMikelV .
the file DUPLICATI__RESULTFILE is often deleted before the script starts. Indeed, the script is taking only an instant to be executed completely.
I suppose the sequence is this (looking at the status bar and monitoring the file existence):
copy of files
db/backup verification starts. In this point, the file DUPLICATI__RESULTFILE exists
db/backup verification finishes. In this point, the file DUPLICATI__RESULTFILE may exist or not
Execution of script
I’ve configured different backups and notice this issue happens in backups of large number of changing files in which the ‘verification’ takes too long. As you said I can be the cleanup threads and I also dismiss the timeout as a solution.
Then, a approach could be a different organization in the involved processes. Here a couple of ideas:
Option 1: the script should be executed immediately after the copy of files, just when the DUPLICATI__RESULTFILE is generated
Option 2: DUPLICATI__RESULTFILE should be created after the backup verification process(once all the backup processes have finished)
I’m guessing option 1 won’t be viable because there could be a result from the verification that people want to process.
Process wise, option 2 makes sense however code wise the verification file & validation don’t really go together.
I’d prefer to see the threading issue fixed as my guess is this is behind a few other reported problems as well.
Unfortunately, I don’t have any time to look into the code but it we can come up with a repeatable failure process we can post it on GitHub in case somebody else has time to dig into it.