Announcement: syncosync - backup appliance

[steviehs1]

After several years of development I am proud to announce the availability of the syncosync backup appliance. It is Free and Open Source Software and could be useful for everybody who wants to combine local and remote backups with low costs and very fast disaster recovery capabilities. We are a small group of developers and are using the prototypes with Duplicati since > 4 years.

The name of the project is “syncosync” the starting point to understand more is https://syncosync.org

So the idea is to have a Raspberry Pi4 with a decent big enough HDD at your place and exactly the same thing at a friends place (your backup buddy). The HDDs are (normally) partitioned equally which means, that 50% of the disk space is for you, the other 50% are for your buddy.

You are putting your backups via Duplicati on your local syncosync box using encryption, ssh+rsync and the rest is done by syncosync: the backups are synced to the buddies syncosync box and so they are always and automatically held also remote. This is also the reason, why you do not need any raid for your local backup. The remote HDD is the mirror of your local HDD.

When the maximum disaster has happened: you lost all your local data, you can visit your backup buddy, grab his syncosync box, buy a new laptop and restore your data with the speed of the local network. Alternatively - if you have the time - you can restore from your buddy via remote restore.

The whole syncosync system is operated via WebUI, there is no need for any commandline setup - which is possible though as there is a separation between core and UI. The backup per system is based on separated backup accounts, so your whole family or small company could use one syncosync system.

Right now the UI is available in English and German, we hope other languages will happen soon, there is a weblate project set up to support that.

Please give syncosync a try, we think it is a perfect alternative to cloud services and also a much better automation of backups than remote manually stored harddisk drives. There is a forum, the source code is on gitlab, the translation is on weblate, manuals are available in English right now.

So, please give it a try. This is the first public announcement, there may be questions, glitches… but overall it is working stable. If you have any questions, please let me know.

[drwtsn32]

Nice work, looks like a useful complement to Duplicati.

[steviehs1]

Thx. I hope many people will give it a try…

[ts678]

Looks like a nice new way to rearrange the usual cost/speed/disaster-resiliency tradeoffs differently.
Thanks for doing this as a multi-backup-platform booster, and congratulations on your initial launch.

You also have the potential to cut daytime congestion, or maybe even avoid fast malware damages.
Both of those issues have forum posts but only awkward solutions. Your system might offer options.

On the downside of sync delay, the files at the remote site might be a bit inconsistent at a given time.
Duplicati.CommandLine.RecoveryTool.exe is more robust, but I hope normal methods typically work.

@steviehs1

ssh+rsync

Which Duplicati Storage Providers work? Because you do SSH, does SFTP work (instead of rsync)?

@steviehs1

The remote HDD is the mirror of your local HDD.

If true at a fairly deep level, including administrative setup, having it used by either buddy is real nice, however removal of the syncosync might bother a backup buddy if that means no access for too long.
If mirror is very deep, maybe a disk clone or even (adding costs) redundant or spare parts might help.

Apologies if I’m getting ahead of current flexibility, but it does seem like there’s a lot of potential to this.

[steviehs1]

@ts678

Looks like a nice new way to rearrange the usual cost/speed/disaster-resiliency tradeoffs differently.
Thanks for doing this as a multi-backup-platform booster, and congratulations on your initial launch.

Thanks, that was also the initial idea to be a) cheaper (after approx 1-2 years you have the break even) and b) recovery-faster.

@ts678

On the downside of sync delay, the files at the remote site might be a bit inconsistent at a given time.
Duplicati.CommandLine.RecoveryTool.exe is more robust, but I hope normal methods typically work.

While a full backup could of course need some time to sync incremental backups happen in a reasonable time frame. We have also added some bandwidth restriction settings for day/night, so syncosync does not eat up your internet while you need it.

@ts678

Which Duplicati Storage Providers work? Because you do SSH, does SFTP work (instead of rsync)?

yes, it is SFTP

@steviehs1

The remote HDD is the mirror of your local HDD.

@ts678

If true at a fairly deep level, including administrative setup, having it used by either buddy is real nice, however removal of the syncosync might bother a backup buddy if that means no access for too long.
If mirror is very deep, maybe a disk clone or even (adding costs) redundant or spare parts might help.

Hmm… I do not get you right at this point. Both sync partners have a syncosync system running 24/7. So “normally” the remote system should be always available. And your sync partner should be interested in having it running for the same reason as you do. Still if the sync is not happening you get a mail which informs you about the situation - in the worst case you have to find a new - more reliable - sync buddy. This is also the social reason that none of the two buddies complains about the system and power consumption costs - which are really low anyways: they are equal for both parties.

@ts678

Apologies if I’m getting ahead of current flexibility, but it does seem like there’s a lot of potential to this.

Oh, we have many additional ideas about possible features… so as long as there are users for it

[ts678]

@steviehs1

I do not get you right at this point. Both sync partners have a syncosync system running 24/7.

That’s the normal case. In the disaster case, I’m not clear on what gear is where and what it can do.

@steviehs1

When the maximum disaster has happened: you lost all your local data, you can visit your backup buddy, grab his syncosync box, buy a new laptop and restore your data with the speed of the local network. Alternatively - if you have the time - you can restore from your buddy via remote restore.

The remote restore at end would be one option to avoid having to hang around buddy’s home awhile. Alternatively if both buddies can share surviving syncosync, start big restore to laptop and return later when it’s done. This works worse if it’s a desktop. There, it might be easier to move data to computer.

With the “grab his syncosync box” (as opposed to connecting over a LAN), it sounds like grab-and-go (perhaps to a temporary location and a more leisurely restore, perhaps with most important files first).

Impact to backup buddy would be less if syncosync stayed in place, and a clone of data drive moved.
This might be slower than taking the syncosync away, and would need new Pi able to adopt old drive.
I don’t know how closely the Pi is attached to its particular drive. Less attachment increases flexibility.

I assume you can’t sync without two working syncosync, but you might be able to create a drive-to-go which (if time allows) could plug into a new (or spare) Pi for the fastest possible return to normal case.

@steviehs1

Oh, we have many additional ideas about possible features

Although there are occasional exceptions, typically the idea is a lot easier than implementing the idea.
I’m just poking a little to understand the various available scenarios for recovery then return to normal.

[steviehs1]

@ts678

With the “grab his syncosync box” (as opposed to connecting over a LAN), it sounds like grab-and-go (perhaps to a temporary location and a more leisurely restore, perhaps with most important files first).

Yes, that’s the main idea of the so called “Disaster Swap”: you grab the box from your buddy, buy a new desktop, laptop whatever, go to your place and restore the most important data. This should be possible in a few hours of course depending on size. The risk for your buddy is minimal, as the probability to lose data in exactly the same time is rather low (or there is some other real tough stuff happening…).

If you do not want your buddy to give away his box for this time you could either clone his disk (with any mean you have… ) and set up a new syncosync box from this disk and swap it to disaster mode or just bring him a new empty syncosync box and let him do a remote recovery over LAN, which still needs some time…

… so from my understanding of risks of losing data, I’d say in the - hopefully very rare case - of a Disaster, giving away your syncosync box is acceptable. Of course all the buddies backup data is not readable and would not be touched.

@ts678

I don’t know how closely the Pi is attached to its particular drive. Less attachment increases flexibility.

The syncosync system is on a micro SD card. When you plug in a different syncosync written HDD, this is detected and you will be asked if the configuration from this HDD should be restored to the system.

@ts678

I assume you can’t sync without two working syncosync, but you might be able to create a drive-to-go which (if time allows) could plug into a new (or spare) Pi for the fastest possible return to normal case.

Such a mirroring function should not be too complicated to be implemented in the actual solution… maybe worth thinking about it as RPi’s are so hard to get these days

@ts678

Although there are occasional exceptions, typically the idea is a lot easier than implementing the idea.
I’m just poking a little to understand the various available scenarios for recovery then return to normal.

Did I mention, that you can make the Disaster Swap mode persistent. After that this box looks like your old box. And you can buy a new box for your buddy and do a remote restore… or you can Disaster Swap from that again and make it persistent… ok, I see, I have not added any chapters for disaster recovery in the manual so far… will do soon.

[ts678]

@steviehs1

I have not added any chapters for disaster recovery in the manual so far… will do soon.

Thanks! That would help people better understand in advance what they may use later on.
The system sounds quite flexible, but (as Duplicati has found) sometimes that’s confusing.

[steviehs1]

Sorry for the delay in answering, but now find a new chapter in the syncosync: Administrator Manual describing the operation modes.

I have also changed the website a little bit, so that the firefights are not the first picture to be seen - for psychological reasons

Please let me know when there are still open points to clarify…

[mr-flibble]

Hello, thanks you for this project, I have “Crashplan P2P backup” feeling from it

One question: Maybe requirements for public IP address should be mention in FAQ or elsewhere? Only mention I found was in port forwarding section

When I first study your project, I was very exited, because i thought that project will work for everyone - even for people without own public IP address - like old Crashplan, but it seems that something like WIKI: TCP hole punching is not part of project, which is understandable but still sad : )

[steviehs1]

@mr-flibble

Hello, thanks you for this project, I have “Crashplan P2P backup” feeling from it

Yes, that was basically the idea. For cost saving but also for faster disaster recovery.

@mr-flibble

One question: Maybe requirements for public IP address should be mention in FAQ or elsewhere? Only mention I found was in port forwarding section

yes, I will add this to the FAQ. We also have to work for IPv6(lite) support…

@mr-flibble

When I first study your project, I was very exited, because i thought that project will work for everyone - even for people without own public IP address - like old Crashplan, but it seems that something like WIKI: TCP hole punching is not part of project, which is understandable but still sad : )

At the point now, we were first working on supporting all the basic features - mainly the recovery stuff was very important to implement…
We have since some time the plan to evaluate NAT hole punching or even use libnice and TCP over WebRTC, but it seems not that easy. Also it could be, that for some of these technologies we would need at least a kind of a public arbitration server. This again could lead to privacy concerns for some people, so it should be at max. an alternative way to the existing. What I would try to avoid by any mean is the transmission of data through a proxy server. This would be a real bottleneck.

How is e.g. your IP access to understand, what could work for you?

[steviehs1]

Ok, I added a prerequisites chapter to the FAQ.