Duplicati stuck on Compact when Getting/Putting from OneDrive and results in missing files

Support
16

Here’s the very slightly updated version of the file changer. This one is called random_file3.py:

#!/usr/bin/python
import os
import random
import argparse
import sys

parser = argparse.ArgumentParser()
parser.add_argument('path')
parser.add_argument('--block-size', type=int, default=102400, help='size of each block of this file')
parser.add_argument('--change-size', type=int, default=2, help='size of change at start of a block')
parser.add_argument('--percent', type=int, default=10, help='percent of blocks to get changed')
parser.add_argument('--create', type=int, help='create file with this many blocks', dest='blocks')
args = parser.parse_args()

if args.blocks is not None:
    size = args.block_size * args.blocks
    fd = os.open(args.path, os.O_WRONLY | os.O_BINARY | os.O_CREAT | os.O_TRUNC)
    os.ftruncate(fd, size)
    blocks = args.blocks
elif os.path.exists(args.path):
    fd = os.open(args.path, os.O_WRONLY | os.O_BINARY)
    size = os.path.getsize(args.path)
    blocks = size // args.block_size
else:
    print('Please supply path of an existing file, or use --create to set up a new file.', file=sys.stderr)
    sys.exit(1)

changes = blocks * args.percent // 100

for index in random.sample(range(blocks), k=changes):
    os.lseek(fd, index * args.block_size, os.SEEK_SET)
    os.write(fd, random.randbytes(args.change_size))

There are some subtleties to its use. In the test script below, I initialize to fully random data because otherwise the blocks compress too well, and I need size in order to get the compact churning data…

I did a second run at 50 MB to see if behavior stablized, and it had.

Downloaded 20 file(s) with a total size of 978.77 MB, deleted 40 file(s) with a total size of 979.41 MB, and compacted to 36 file(s) with a size of 880.84 MB, which reduced storage by 4 file(s) and 98.57 MB

My first run at 1 MB after initial backup creation (I commented out the whole-file create for later runs) is

Downloaded 646 file(s) with a total size of 632.23 MB, deleted 1292 file(s) with a total size of 633.07 MB, and compacted to 994 file(s) with a size of 534.26 MB, which reduced storage by 298 file(s) and 98.82 MB

and I guess I’ll increase dblock-size as you suggest, to see if anything in this smallish backup will hang.

set DUPLICATI=C:\Program Files\Duplicati 2\Duplicati.CommandLine.exe
set RANDOM=C:\bin\random_file3.py
set FILE=C:\tmp\onedrive_compact\file.bin
set LOG=C:\tmp\onedrive_compact\profiling.log

rem "%RANDOM%" --create 10000 --percent 100 --change-size=102400 "%FILE%"

:loop
"%RANDOM%" --percent 10 --change-size 4 "%FILE%"
"%DUPLICATI%" backup "onedrivev2://Duplicati Backups/onedrive_compact?authid=REDACTED" "%FILE%" --no-encryption --keep-versions=1 --threshold=5 --console-log-level=information --log-file="%LOG%" --log-file-log-level=profiling --profile-all-database-queries --dblock-size=1MB
rem timeout /t 10
rem goto loop

I took out the timeout and loop because I’m running this by hand, increasing dblock-size after each run.
I can increase from 1 GB to 3.3, but maybe I’ll first continue prior plan while pondering what you posted.
Good progress I guess. BTW if you ever go to huge dblock-size, you need larger http-operation-timeout.

1 Like