Db rebuild with rclone od4b doesn't work

Support
1

Hello,

It’s it a well known issue of database rebuild taking days for large backup sets. But this time it’s will rclone & od4b.

The reason od4b with rclone backend is to bypass 5k files limit per directory in od.

Database rebuild starts & rclone process starts & closes then doesn’t return back to task manager. Duplicati UI stays at Re-creating database for days doing nothing.

The only way to proceed is:
rclone -v -P sync od4b: backups D:\backups
Change destination path & rebuild database.

Tried with rclone 1.42, 1.44 Duplicati 2.0.3.14 & 2.0.3.11

Thanks…

2

Give me a few days to verify. I’m actually moving to od4b with rclone but a repair is ongoing at the original location, which should finish in a couple of days. Will let you know my results.

3

There was hope (and some test evidence) in ‘No files were found at the remote location’ using Onedrive for business that the 5000 limit was removed when Duplicati switched od4b to the Graph API in 2.0.3.4 in April. Are you still seeing issues arise at 5000 files when using a recent enough OneDrive For Business release?

I have no immediate idea on the rclone aspect (I don’t have a configuration like that), but do you know if the rebuild problem depends on large destination size or file count, or can it be reproduced on a small backup?

Possibly another way to test would be Restore → Direct restore from backup files, which makes a small DB.

If there’s a problem with rclone as a backend, there should be some evidence, maybe in About → Show log → Live → Information (which should be enough to show transfers of backend files, or the lack thereof…) or –log-file with –log-file-log-level=Information (or even higher, such as Profiling, if lots of output is tolerable…).

4

I am using 2.0.4.3 beta I see one drive V2 but od4b looks the same.:neutral_face:

5

I had no issues with rclone od4b on over 6000 files on od4b.

Latest beta duplicati and latest rclone stable.

6

Backup the database and attempt to rebuild. It fails here.

7

Please share exact error message together with export to command line of your config.

8

It worked this time :angry: something must have been wrong with my internet…

Mine looks like.
Local repro local
Remote repro od4b
Path backups

I had a few more questions

How to give path to config file at non default location
How about giving a encrypted rclone config path

Thanks…

9

Would Environment Variables help? Rclone can be configured entirely using environment variables. These can be used to set defaults for options or config file entries.

For detailed information on rclone usage, you might get faster and better answers in the rclone forum.

EDIT:

Configuration Encryption mentions RCLONE_CONFIG_PASS. I don’t use rclone, but maybe that’s the trick.

10

Try with the option --rclone-option= which you can set at the remote configuration.

11

Duplicati’s Rclone page (which I neglected to read :blush:) describes --rclone-option. Thanks to @Wim_Jansen