# Newbie: What is the most cost effective way to backup 12+ terabytes windows folder of data daily to AWS Deep Archive? **URL:** \ **Category:** How-To\ **Created:** [April 29, 2024, 3:15pm UTC](https://forum.duplicati.com/t/newbie-what-is-the-most-cost-effective-way-to-backup-12-terabytes-windows-folder-of-data-daily-to-aws-deep-archive/17830 "2024-04-29T15:15:58Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![sul](https://forum.duplicati.com/letter_avatar_proxy/v4/letter/s/4da419/32.png) [@sul](https://forum.duplicati.com/u/sul)\ **Post date:** [April 29, 2024, 3:15pm UTC](https://forum.duplicati.com/t/newbie-what-is-the-most-cost-effective-way-to-backup-12-terabytes-windows-folder-of-data-daily-to-aws-deep-archive/17830/1 "2024-04-29T15:15:58Z")
Hi Folks, I am new to Duplicati. I have a file server running windows standard 2022. If at all possible, I would love to use Duplicati to incrementally backup a specific folder on my F: Drive to AWS Deep Archive on a daily basis. I understand the initial backup can be a full backup and all subsequent backups would be incremental. Also, I do not anticipate needing to retrieve data from deep storage (possibly for years) as it is only for safekeeping in case of an emergency. I have not purchased AWS storage as yet but I can do as soon as I can be sure the above is indeed possible. I assume this is not an uncommon thing to do. However, unfortunately I am not finding a clear cut set of instructions to backup incrementally to AWS Deep Archive. BTW: I currently practice the backup operation above using LTO7 tapes using a different backup software. My current cost for tapes is less than $120 per year on the average. So, I’m hoping I can replicate a similar scheme using Duplicati and AWS Deep Archive. I appreciate any advice. Thanks! -sul. ---
**Author:** ![drwtsn32](https://forum.duplicati.com/user_avatar/forum.duplicati.com/drwtsn32/32/2591_2.png) [@drwtsn32](https://forum.duplicati.com/u/drwtsn32)\ **Post date:** [April 29, 2024, 4:38pm UTC](https://forum.duplicati.com/t/newbie-what-is-the-most-cost-effective-way-to-backup-12-terabytes-windows-folder-of-data-daily-to-aws-deep-archive/17830/2 "2024-04-29T16:38:00Z")
> [@sul](#): > > My current cost for tapes is less than $120 per year on the average. So, I’m hoping I can replicate a similar scheme using Duplicati and AWS Deep Archive. You won’t beat $120/year. I just looked at AWS pricing and the cost for deep archive is $0.00099/GB/month. For 12TiB this would be $12.16/month, or about $146/year. This doesn’t even include the transaction costs associated with S3, not to mention restore (retrieval) costs which are relatively high for this tier storage. Besides the cost, there are complications and challenges when using Duplicati with archive tiers that are not “online.” ---
**Author:** ![sul](https://forum.duplicati.com/letter_avatar_proxy/v4/letter/s/4da419/32.png) [@sul](https://forum.duplicati.com/u/sul)\ **Post date:** [April 29, 2024, 6:48pm UTC](https://forum.duplicati.com/t/newbie-what-is-the-most-cost-effective-way-to-backup-12-terabytes-windows-folder-of-data-daily-to-aws-deep-archive/17830/3 "2024-04-29T18:48:10Z")
drwtsn32, thank you for your honest feedback. I can certainly live with the higher costs since I would potentially do away with some labor like physical management of tapes and maintenance of on-premise tape hardware. However, I think Duplicati’s challenges with archive tiers could be a deal breaker. I wonder–is it that Duplicati is not yet mature enough to robustly handle (e.g., with minimal, low cost S3 transactions) “Deep Archive” backups in a manner that I desire (i.e., daily incremental backup to deep archive)? Or more generally, is Deep Archive such a beast that perhaps the way I intend to use it is just not possible by any backup software–even commercial software–without breaking the bank? Appreciate any insight. Thank you! -sul. ---
**Author:** ![drwtsn32](https://forum.duplicati.com/user_avatar/forum.duplicati.com/drwtsn32/32/2591_2.png) [@drwtsn32](https://forum.duplicati.com/u/drwtsn32)\ **Post date:** [April 29, 2024, 9:43pm UTC](https://forum.duplicati.com/t/newbie-what-is-the-most-cost-effective-way-to-backup-12-terabytes-windows-folder-of-data-daily-to-aws-deep-archive/17830/4 "2024-04-29T21:43:05Z")
Duplicati can work with archive-tier storage, but you must change some default options. See this post for more information: > [@Amazon Glacier Best Practices?](https://forum.duplicati.com/t/amazon-glacier-best-practices/13330/6): > > Glacier doesn’t allow immediate read access to the objects. As such, Duplicati fails when it tries testing the files. I personally advise against using archive tier storage. (You can get hot storage at about the same cost as Glacier from other cloud providers such as Backblaze B2 or Wasabi.) Google does have an archive tier that allows for near immediate access, so that’s another option. If you really, really want to use Glacier, then you should set these options: --no-auto-compact = true --n… And this thread: > [@Up-to-date on glacier usage](https://forum.duplicati.com/t/up-to-date-on-glacier-usage/14659/1): > > I went through nearly all threads I could find and most are outdated, what’s the current method if one wants to use (scaleway-) glacier for backups? I assume --no-backend-verification is necessary, but what about the rest, any other configs needed for duplicati to blindly trust the remote? any of the custom URLs necessary still? how would one attempt a restore; say I restore it to hot-storage first, would I have to set that up as a separate remote first and then point duplicati at it? or i… Also for a 12TiB backup you’ll want to either change the default deduplication “block size”, or break up the backups into smaller sets, or both. > [@Choosing sizes in Duplicati](https://forum.duplicati.com/t/choosing-sizes-in-duplicati/17683): > > Choosing Sizes in Duplicati All options in Duplicati are chosen to fit a wide range of users, such that as few as possible of the users need to change settings. Some of these options are related to sizes of various elements. Choosing these options optimally is a balance between different usage scenarios and has different tradeoffs. This documents explains what these tradeoffs are and how to choose those that fit a specific backup best. The block size As Duplicati makes backups with blocks, aka… ---
**Author:** ![ts678](https://forum.duplicati.com/letter_avatar_proxy/v4/letter/t/8491ac/32.png) [@ts678](https://forum.duplicati.com/u/ts678)\ **Post date:** [April 29, 2024, 11:02pm UTC](https://forum.duplicati.com/t/newbie-what-is-the-most-cost-effective-way-to-backup-12-terabytes-windows-folder-of-data-daily-to-aws-deep-archive/17830/5 "2024-04-29T23:02:43Z")
Thanks for jumping in. I don’t have anything AWS, and I personally worry about things like testing backups. Because this one sounds like emergency-only use, I’m wondering how one will know that it’s working well. Downloading everything for a test will cost over $1000 I think – isn’t [usual S3 egress](https://aws.amazon.com/s3/pricing/) around $90/terabyte? Trying to arrange files so dlist and dindex stay in hot storage can test Recreate, but it might need a dblock. You were at one time doing sync of a local backup on a NAS, where at least one could test local copy well. Additionally this would allow things like version deletes and compacting of wasted space from it to happen. Later on, you were pursuing direct backup from Duplicati, and liked it. Did the good view [here](https://forum.duplicati.com/t/duplicati-and-s3-glacier/13323/16) grow worse? ---
**Author:** ![ts678](https://forum.duplicati.com/letter_avatar_proxy/v4/letter/t/8491ac/32.png) [@ts678](https://forum.duplicati.com/u/ts678)\ **Post date:** [April 29, 2024, 11:14pm UTC](https://forum.duplicati.com/t/newbie-what-is-the-most-cost-effective-way-to-backup-12-terabytes-windows-folder-of-data-daily-to-aws-deep-archive/17830/6 "2024-04-29T23:14:43Z")
> [@sul](#): > > Also, I do not anticipate needing to retrieve data from deep storage (possibly for years) as it is only for safekeeping in case of an emergency. Does that mean this is a secondary backup or are you just planning on restore almost never happening? How important is your data? Usually important data gets more than one backup. One-plus-sync isn’t two, however it allows not only testing, but restoring more often that years apart. It can also cut remote space. You gave a focus on cost though. Lowest cost might mean less protection – along with other drawbacks. ---
**Author:** ![sul](https://forum.duplicati.com/letter_avatar_proxy/v4/letter/s/4da419/32.png) [@sul](https://forum.duplicati.com/u/sul)\ **Post date:** [May 1, 2024, 2:49pm UTC](https://forum.duplicati.com/t/newbie-what-is-the-most-cost-effective-way-to-backup-12-terabytes-windows-folder-of-data-daily-to-aws-deep-archive/17830/7 "2024-05-01T14:49:15Z")
drwtsn32, This is all great information-- and I sincerely appreciate that you took the time to curate the relevant articles in your reply. I’ll try to navigate through it all. Being an old fashioned tape-backup guy, there is certainly a lot to learn about the nuances of backing up a 12+ terabytes to deep archive. From an outsider looking in, it sounds like if someone doesn’t know all the “tricks” (block size, verification, etc.), it could significantly affect monthly storage costs. Hopefully I’ll have gathered enough confidence to move forward with S3 deep archive or an alternative provider in a way that is not too much more expensive than tapes. Thanks again! -sul. ---
**Author:** ![Gloomfrost](https://forum.duplicati.com/user_avatar/forum.duplicati.com/gloomfrost/32/9242_2.png) [@Gloomfrost](https://forum.duplicati.com/u/Gloomfrost)\ **Post date:** [May 1, 2024, 9:32pm UTC](https://forum.duplicati.com/t/newbie-what-is-the-most-cost-effective-way-to-backup-12-terabytes-windows-folder-of-data-daily-to-aws-deep-archive/17830/8 "2024-05-01T21:32:48Z")
Hi all, I was actually about to make a very similar post about a similar situation. I have a server with a 35TB data set, 8.4 million files (works out to about 440kb average file size). We also have 65% in deduplication savings using the windows server dedup. With B2, our actual remote size is 20TB as of last week. We’ve been using Duplicati for years and it’s worked fairly well (about 4hrs for incremental backups) to B2, but lately we’ve been getting issues with zip64 4GB streams which I recently updated, and also the backup is now taking almost 16 hours (spiked recently for some reason). We’re using a remote volume size of 500MB but I missed the block size recommendation so we were at 100kb the whole time which I am assuming is the culprit. I’ve decided it’s time to update block size so before I dump a new 20TB+ volume to B2, I was wondering if there are any other variables to adjust, if any, other than what I already have for extremely large datasets? auto-vacuum X auto-vacuum-interval 1 month zip-compression-zip64 true blocksize 5MB remote volume size 500MB Besides splitting the backup into multiple backups which will not be possible in our environment, is there anything else recommended? We have a symmetrical fiber gigabit connection, and the data is on a RAID5 SSD volume. The system has a 1TB primary NVME drive, with 40GB RAM and a modern i5. ---
**Author:** ![ts678](https://forum.duplicati.com/letter_avatar_proxy/v4/letter/t/8491ac/32.png) [@ts678](https://forum.duplicati.com/u/ts678)\ **Post date:** [May 1, 2024, 11:01pm UTC](https://forum.duplicati.com/t/newbie-what-is-the-most-cost-effective-way-to-backup-12-terabytes-windows-folder-of-data-daily-to-aws-deep-archive/17830/9 "2024-05-01T23:01:05Z")
Welcome to the forum @Gloomfrost You hit most of the highlights, although I usually like to keep backups below 1 million blocks, which means blocksize of 20 MB or so, but file count blows the budget, as a file has at least two blocks (one metadata). Performance analysis at this size has been scarce so far due to limits of equipment and people, however another person (probably with a faster system, and yours sounds pretty fast) thinks it can do more blocks. Regardless of where the speed drop off really happens, any larger blocksize will almost surely move it out. Lots of files and blocks also get larger databases, which means overflowing the tiny default 2 MB memory cache in SQLite. It doesn’t show up in drive accesses (which I notice immediately on a mechanical drive), however I can see the OS level read rate start to go super high, e.g. in Task Manager or [Process Explorer](https://learn.microsoft.com/en-us/sysinternals/downloads/process-explorer). > [@Gloomfrost](#): > > I was wondering if there are any other variables to adjust > [@How to recover from aborted/stuck/crashed backup run?](https://forum.duplicati.com/t/how-to-recover-from-aborted-stuck-crashed-backup-run/17818/13): > > > [@How to recover from aborted/stuck/crashed backup run?](https://forum.duplicati.com/t/how-to-recover-from-aborted-stuck-crashed-backup-run/17818/12): > > > > for cache size there exists an environment variable I could modify. If I know its name and which value format it expects, I can test increasing the cache size. > > `CUSTOMSQLITEOPTIONS_DUPLICATI` is its name. Was `EXPERIMENTAL_CUSTOMSQLITEOPTIONS_DUPLICATI` and you can probably find several usages under one or the other in the forum and possibly GitHub Issues. > > `set CUSTOMSQLITEOPTIONS_DUPLICATI=cache_size=-200000` is how a batch file gets 200 MB of cache. > Command Prompt and manual launch can also be done. Changing environment variables for a Windows service is awkward compared with a Linux service where it’s very easy. Windows might need a global set. however the increased blocksize will probably reduce database size and access stress. Just mentioning it because you asked for other adjustments. As usual, the exact need if you go this route would probably be experimental. If the SQL gets too slow, or you see tons of file access (Process Monitor can detail those) to database or its journal of a file whose name starts with `etilqs` (likely in Temp), more cache might help it… > [@Really frustrated with Duplicati's lack of feedback](https://forum.duplicati.com/t/really-frustrated-with-duplicatis-lack-of-feedback/17783/4): > > I think it was a good choice some 10+ years ago because it keeps the memory usage low and performs relatively fast B+ tree lookups. But it does not scale well with +1TiB backups which are more common today. I am aware of the scaling issues, and intend to work on it soon-ish. is a recent SQLite comment. I know the default blocksize is being raised times-ten, but I don’t know what else might be cooking. Possibly this quote will pull out some more comment, or maybe it won’t. We’ll see.