BackBlaze is reporting large daily downloads when using Duplicati

At the moment I am testing out Dupicati with BackBlaze’s B2 cloud storage. I have a free account with BackBlaze that allows for 10GB of storage. Currently I have less than 1GB of data; just small data\Doc files. Today, after about 30 days of using Duplicati with BackBlaze (incremental backup once per day), I received a message from BackBlaze that Dupicati is exceeding the 1GB daily download limit for a free account.

Why would Dupicati download 1GB of data when there less than 1GB of storage being used?
And why would Dupicati be downloading the entire backed up data during the incremental backup process?

Currently the daily incremental backup is less than 1mb.

using version 2.0.4.5_beta_2018-11-28

Two reasons duplicati may download data:

  • Auto volume compaction feature which tries to save space on the remote side
  • Auto volume verification feature which randomly tests a remote volume to ensure it is valid (default is to test 1 volume after each backup)

Both of these settings can be adjusted but there are pros and cons.

Are you using the default volume size of 50MB or did you change that? Actually even if you didn’t change it, a 50MB volume test per day would exceed 1GB in 20 days.

Thanks… I believe you have identified the source of the problem. I need to do some more reading to better understand how to configure it correctly.

Note: 1GB download limit is per day.

I am just going through the testing phase to flush out issues like this one before I start backing up massive amounts of data to B2.

Short term goal: Backup 10GB data to B2 cloud staying within free data limits. Document files only. No large media files. Tweak backup config as needed.

Long Term goal: Backup about 10TB of data to B2 in the most cost effective manner. One example, I have a 1TB monthly data cap on my internet connection, thus 10TB needs to be done over time.

Oh, 1GB per day… I misread that!
Are you using the default volume size of 50MB?

I have reset it back to 50. I had set it much larger thinking it was MAX size of the cloud storage, thus when it was doing a verification it appears it was downloading everything and hitting the daily 1GB download limit.

I have since read the documentation explaining that setting.

Thanks

The COMPACT command describes three configuration options that are currently available for tweaks.

Compact - Limited / Partial suggested an algorithm change. Reply was a half-done idea that –threshold documentation and code look like it’s used two ways. Using separate controls might give better control, however with a more complex UI. Someone could take a closer look at the compact code than I took…

threshold
--threshold = 25
As files are changed, some data stored at the remote destination may not be required. This option controls how much wasted space the destination can contain before being reclaimed. This value is a percentage used on each volume and the total storage.

For the time being, though, lowering --threshold in search of more-often-but-smaller compacts may help.

Small increments (adding files as you go) also does better, if an error occurs somewhere along the way.

Choosing sizes in Duplicati would be worth reading. For a backup that large, larger –blocksize can avoid performance issues from tracking lots of 100KB blocks. Deduplication will be worse (does that matter?).