Really grateful for all these replies everyone - some really great ideas! The most obvious solution to me would be to house a RPi or similar with a 10TB HDD attached at a friends office or house - however my current job is going away and the friend I trust most to host the data turns out to have purchases a metered Internet plan which isn’t going to work. I clearly need more friends!
On a separate note, I notice that a 5 user (or more) plan on G Suite Business offers unlimited Google Drive storage for all users. I don’t particularly want to pay for a 5 user plan myself, but what’s to stop 5 people or so banding together and setting up a G Suite organisation - costs would be around US$12 per month per user as I understand it, plus a cheap domain name.
Am I missing a trick? That seems like a really obvious low cost solution, and Duplicati supports it. Any thoughts?
Duplicati uses strong AES-256 encryption to protect your backups. It is designed following the TNO principle: Trust No One. For instance, all data is encrypted locally before it is transferred to the remote storage system. The password/key to your backup never leaves your computer.
Maybe it is cheaper to pay the better internet connection for your friend? For $12 more than what he currently pays, he should already get something that you can work with. May be a win-win
Also, you can do the first upload of the data with the RPi at your place over LAN and afterwards move the RPi it your friend. Then it is not that much data that goes over his wire.
Maybe we should start a network here where people set up storage backends for each other. Where are you located?
Trust should not be the problem, given the encryption. The only thing is that you may have to back up to multiple untrusted backends in order to have good chances that when you need the backup, one of them is there.
@crazy4chrissi that’s a really great idea - I would be happy to work with someone else in a mutually beneficial arrangement. I’m just outside London in the UK.
I agree about encryption - my “trust issue” is more about my non-technical friends who would unplug the raspberry pi (or whatever the chosen solution is) to plug in a vacuum cleaner or something…
@ts678 - really good points - yes I did some reading last night and found out all about the ebay market for G Suite accounts, and how they were all getting disabled. So I can see how that would all go a bit “sideways”.
As I mentioned to @crazy4chrissi, I’m actually fine with the encryption - my trust issues are people who would turn the box off because the lights annoy them, or to plug in an appliance - I have some really lovely friends, but not many who are truly technical…
If online doesn’t work, maybe a carry-it-yourself rotation of 2 or 3 drives to friend or future job would do? This can hurt the “off-site” goal unless 2/3 off-site-at-a-time is enough or you buy lots of drives to rotate.
This would probably be best done by dividing your 10TB into smaller chunks, one per destination, which would be good anyway because a single 10TB backup with Duplicati might call for special care such as Choosing sizes in Duplicati so that performance (including that of recreate) doesn’t die from the scaling. There’s also a question of how much of the 10TB backup you want to lose at once if the backup breaks.
Some people also try to divide backup source based on use, to try to get online backup of changed files, while letting old files get a different backup. I don’t know if this fits your usage pattern, and it can be tricky.
Filter: older than x days would help with this plan, but such a filter isn’t built-in yet, so you use scripting… Possibly this would work poorly with VM images anyway (scattered changes). Can you backup in guest?
In the realm of emerging technologies that try to undercut traditional cloud storage from companies that purchase their own drives, there’s decentralized storage such as Sia, Storj, and others that rent storage from those who have extra, and resell it (in a more presentable more-available form) to those needing it. This might bring price down from $5-$6/TB/month for inexpensive traditional storage such as Backblaze B2 or Wasabi, to $2/TB/month, however cryptocurrency (and possible speculation) is typically involved. That $2 price is from their own web site, however I found other numbers on other sites that track pricing.
Sia Decentralized Cloud is the Duplicati setting, however forum reliability reports haven’t been that good, possibly due to the sometimes-there-sometimes-not nature of their backends in spite of the redundancy.
Very few companies want to lose money by providing storage below their costs. One that’s still willing is Backblaze with their own client. The catch is it’s aimed at flat-rate backup for users who favor simplicity. Large users (like you) who refer less technical friends (like yours) with less data average out acceptably. Client has some other catches that technical users might not like, such as not supporting server OSs…
