I’m using Duplicati 2.0.5.1 beta and uploading to OneDrive for Business. When my Internet connection was disrupted, upon resuming the backup Duplicati displays “Verifying backend data …” and then fails with error The attempted operation is prohibited because it exceeds the list view threshold enforced by the administrator
As I’ve already backed up a large amount of data taking several weeks over my slow Internet connection - is it possible to resume? Or something I can fix as the Office 365 admin?
[ version Duplicati ver 2.0.5.1_beta_2020-01-18 | Windows ver 10.0.18363.815 ]
I believe this is a 5,000 item limit with OneDrive. Here’s another recent thread on the topic. You probably need to start over and use a larger volume size.
Mentioned there is that some people found “Microsoft OneDrive v2” helped, if they were failing on “Microsoft OneDrive for Business” at 5000 (which is a famous limit the underlying SharePoint has).
I think it’s a fixed limit unless you have local SharePoint, but I kind of suspect you’re entirely remote.
There’s lots written about this on the web though. If you’re lucky, Graph will avoid it however it does.
Thanks for the welcome and insights ts678 and drwtsn32! I really appreciate your quick responses
The 5000 limit is very unfortunate and I wish I checked this before I began. Perhaps a sizing tool or warning can be incorporated into future builds of Duplicati when connecting to O365?
I am entirely remote and using SharePoint Online’s Onedrive for Business
So let’s say I was backing up a maximum of 1TB of data, I should use chunks of 5000 x 204.8MB or larger? (Perhaps use 250MB+ to be safe). This leads me to my next question, does Duplicati have any limit (documented or otherwise) on the size of the chunks?
I’ll read up about Microsoft OneDrive v2 to see if it’s an option - thanks for the tip!
Choosing sizes in Duplicati gives some background if you pursue that route. The size that can prevent you from having more than 5000 is the –dblock-size which defaults to 50 MB. What I personally would consider the chunk size is –blocksize and defaults to 100 KB, which OneDrive won’t see, but Duplicati will, so there gets to be a huge amount of block tracking, and things such as database operations can become very slow. There are no hard-and-fast rules, and not really much systematic testing, but some like to use --blocksize of 1 MB for large backups. --dblock-size is the “Remote volume size” of screen 5 Options. Despite its warning, people sometimes think it’s the entire size of remote storage, so set it to higher values than you would while simply trying to upload your blocks in larger dblock bundles. You’ll probably hit a OneDrive limit first. I see some web citations that they have a 15 GB limit on the file size. Zip-compression-zip64 clarification suggests that Duplicati’s dblock size limit is roughly 16 EiB per this.
