Newbie: What is the most cost effective way to backup 12+ terabytes windows folder of data daily to AWS Deep Archive?

Welcome to the forum @Gloomfrost

You hit most of the highlights, although I usually like to keep backups below 1 million blocks, which means blocksize of 20 MB or so, but file count blows the budget, as a file has at least two blocks (one metadata).

Performance analysis at this size has been scarce so far due to limits of equipment and people, however another person (probably with a faster system, and yours sounds pretty fast) thinks it can do more blocks.

Regardless of where the speed drop off really happens, any larger blocksize will almost surely move it out.

Lots of files and blocks also get larger databases, which means overflowing the tiny default 2 MB memory cache in SQLite. It doesn’t show up in drive accesses (which I notice immediately on a mechanical drive), however I can see the OS level read rate start to go super high, e.g. in Task Manager or Process Explorer.

however the increased blocksize will probably reduce database size and access stress. Just mentioning it because you asked for other adjustments. As usual, the exact need if you go this route would probably be experimental. If the SQL gets too slow, or you see tons of file access (Process Monitor can detail those) to database or its journal of a file whose name starts with etilqs (likely in Temp), more cache might help it…

is a recent SQLite comment. I know the default blocksize is being raised times-ten, but I don’t know what else might be cooking. Possibly this quote will pull out some more comment, or maybe it won’t. We’ll see.

1 Like