Sia backup gateway appliance


#1

INTRODUCTION
This HOWTO describes the process of creating a virtual or physical server that is a gateway to the Sia network. Here are some objectives:

  • Single wallet requirement (vs. needing to have a wallet on every backed up computer)
  • Headless deployment (vs. needing more resources for GUI)
  • Simultaneous client backups

DISCLAIMER: This HOWTO describes deployment in a controlled lab environment. Any use outside of such environment is at your own risk and, at a minimum, should consider common-sense security measures which are not the focus of this topic. As always, the 3-2-1 rule applies in backup design. You should have three copies of your data, including production, two of which should reside on different media and at least one of which resides offsite. It may NOT be wise to consider this deployment as your only offsite solution! A wiser choice would probably be to consider it a second offsite solution or even an archive offsite solution that augments your existing 3-2-1 deployment!

OVERVIEW
Here is a high-level diagram of the deployment design:

I will list my personal deployment below. Please know that none of this is a recommendation; it is just what I had available to me for testing. You are definitely encouraged to improve upon it with your own design choices!

  • Host hardware: Dell PowerEdge R410 (1U server w/ 16 cores and 64GB RAM)
  • Host OS: Linux Mint 18.3 Sylvia 64bit
  • Hypervisor: Oracle VirtualBox 5.2.12 r122591
  • Sia server: Ubuntu Server 18.04 LTS VM w/ 4 cores and 8GB RAM
  • Client machines: A mix of VMs (on the same host), laptops, etc. – all Linux OS

NOTE: I have not tested this deployment with client machines that run a Windows OS. The reason why this deployment works is because the client machines AND the Sia gateway appliance can all access the /media/cache folder. I am not sure how this would work with Windows OS’s because obviously a CIFS mount will not have the same logical path, i.e. /media/cache, but rather some drive letter such as M:\duplicati-cache. More on this later.

Here is a low-level diagram of my deployment design:

REQUIREMENTS
Sia server configuration:

  • Must have a cache volume (i.e. /media/cache) which can be internal storage or a mounted iSCSI volume
  • Must share cache volume via Samba
  • Must run socat to listen on external 8000 and forward to internal 9980
  • Must run siad which listens internally on 9980
  • Must have wallet unlocked (I use siac to do this and to check status on uploads etc)

Client(s) configuration:

  • Must have Sia server’s cache volume mounted via CIFS/Samba at same location (i.e. /media/cache); in other words, both server and client(s) must have the same local path to the same volume
  • Duplicati must be installed (GUI or headless is fine) and global tempdir must reference local path to cache volume (i.e. /media/cache); it is recommended to append a UID, such as client’s hostname, to this (i.e. /media/cache/myclient1); this allows the clients to keep their Duplicati files separate from each other
  • Duplicati jobs must be configured to use port 8000
  • Duplicati job dblock (upload) sizes can be whatever, but I used 1GB as per what I found others recommended for Sia blockchain

These requirements will be explained in more detail below.

STEPS
This is a rough outline of some steps you could follow to deploy a similar Sia backup gateway appliance. Since most people probably do not have a virtual lab available, I will actually outline the process using spare computers instead.


SIA BACKUP GATEWAY APPLIANCE PREP

  1. Choose an old computer to be your Sia backup gateway appliance

    NOTE: You will need a good bit of space for the Sia blockchain AND for the Duplicati cached backup files (the chunks that Duplicati creates prior to uploading). Therefore, it is a good idea to use a 2nd hard drive OR an attached USB drive (permanently mounted, of course) OR (in the case of a virtual server) a dedicated virtual disk. Technically, you could even carve out an iSCSI LUN from a NAS and attach it to the appliance for this purpose, but I will not go into detail on this. The basic idea is have plenty of space available! And “plenty of space” is qualified by at least enough for the Sia blockchain + the size of your backup job data (which is hard to estimate because of compression, I know).

    Example: I have a 30GB dedicated volume for Sia and a 100G dedicated volume for my Duplicati cache
    Screenshot%20at%202018-08-18%2011-05-06

  2. Install Ubuntu Server (or workstation if you want a GUI) and update packages
    sudo apt update && sudo apt upgrade -y && sudo reboot

  3. Install SSH, Samba, Socat, unzip, and screen; if you are running headless and want a nice web UI, also install webmin

  4. Confirm that you have access to the appliance either by SSH or using a GUI

  5. Not a bad idea to drop the firewall, at least until you get everything working; then you could create rules to lock down all but what you need open


SIA BACKUP GATEWAY APPLIANCE CONFIG
We’re going to create two mount points, one for the Sia blockchain and our files and one for the Duplicati cache for all our clients. You can see the previous section point 1 for ideas on the volume itself.

  1. Create a mount point for Sia at /media/sia
  2. Create a mount point for the Duplicati cache at /media/cache
  3. Grant permissions on both folders to whatever account you plan to run Sia under

Example: Volumes
Screenshot%20at%202018-08-18%2011-01-29

Example: fstab

  1. Download siad (the Sia Daemon package, not the Sia UI)
  2. Extract siad package to /media/sia

Example: (ignore the different domain name than previous screenshots; I stood up a new VM for cleaner examples)
Screenshot%20at%202018-08-18%2012-07-14

  1. cd into that folder and run: ./siad --sia-directory ../blockchain &
    This will create /media/sia/blockchain/ and point Sia daemon to it. The Sia daemon will start in the background. (Press Enter a 2nd time on your keyboard to get back to the command line after you’ve run a process with an & symbol at the end.)

  2. Now in the same folder, type: ./siac
    You should see Progress (estimated): and some percentage. This is the percentage of your blockchain download. You can continue with the instructions below, but you should wait until this is 100% before you actually unlock your wallet or try to perform a backup. NOTE: This could take a day or more to download, depending on your Internet speed and other factors!

While the blockchain is downloading, let’s proceed with some additional setup.

The Sia daemon only listens on localhost (i.e. internally) for security reasons. I tried various ways around this, but the solution that proved the best in the end was a simple forward with the socat program.

  1. Type: socat tcp-listen:8000,reuseaddr,fork tcp:localhost:9980 &
    This will cause socat to run as a background process and forward any inbound connections on port 8000 to the local Sia daemon that is listening by default on port 9980.

FYI: If you want to bring either background process to the foreground, just type: fg followed by the number (1 for the siad background process we created in 6 above or 2 for the socat process we created in 8 above). Then if you want to send it to the background again, press CTRL+Z (to suspend it) and then bg followed by the number again. And even BETTER way to handle multiple processes is to use the screen program. But you can look that up and learn how to use it on your own!

So at this point, we have siad running in the background and downloading the blockchain. And we have socat running in the background, forwarding any traffic from external port 8000 to internal port 9980.

Next, we need to setup our Duplicati cache folder and Samba.

The /media/cache folder is a central repository that both the Sia daemon AND the Duplicati clients need to see and use. Basically, the way it works is that the Duplicati client stages its dblocks (upload files) there and then instructs the Sia daemon to upload them. Obviously the Sia daemon can access the files locally at the /media/cache folder. However, we need to use Samba to give the client machines (where Duplicati is installed) access to the same location at the same path, i.e. /media/cache that is local to them. Hopefully this will become more clear as you go through the steps below.

  1. Add the following lines to the end of your /etc/samba/smb.conf file

    [duplicati-cache]
    comment = Duplicati backup cache
    path = /media/cache
    available = yes
    browseable = yes
    public = no
    read only = no
    guest ok = no
    writeable = yes

  2. Restart Samba by typing: sudo systemctl restart smbd

Now one last thing on the Sia Backup Gateway Appliance before we proceed to the Duplicati client machines: It is important that the files created in this /media/cache folder are read/writeable for BOTH the user account under which siad is running AND the client machines on which this Samba share will be mounted. The way I handled this was just making sure I was using the same Linux username (jwilmoth) and UID and group (jwilmoth) and GUID. However, you may want to do something different.

NOTE: As mentioned at the first, I have not tested this deployment with client machines that run a Windows OS. The reason why this deployment works is because the client machines AND the Sia gateway appliance can all access the /media/cache folder. I am not sure how this would work with Windows OS’s because obviously a CIFS mount will not have the same logical path, i.e. /media/cache, but rather some drive letter such as M:\duplicati-cache.


DUPLICATI CLIENT MACHINE (LINUX) CONFIG

  1. Create the folder /media/cache with same permissions as the previous section, step 3
  2. Mount the CIFS share in the previous section, step 9 at this folder

Example fstab entry: //wil-sia-srv/duplicati-cache /media/cache cifs users,user=jwilmoth,uid=1000,gid=1000,username=jwilmoth,password=[your_password] 0 0

(replace [your_password] above with the password used to access the share)

  1. Create a test file in /media/cache and verify that you can see it and have read/write access to it from the Sia gateway appliance at the same location

  2. Install Duplicati

  3. Access Duplicati at http://localhost:8200/ngax/index.html

  4. Click Settings and under Default options > Add advanced option… locate tempdir under the Core options section (easiest to find if you just start typing)

  5. Set the tempdir path to /media/cache/[client name] where [client name] is either the hostname of your client computer or something to uniquely identify it from other client computers; the idea that every client computer will have its own sub-folder under /media/cache where its instance of Duplicati can cache files

For example:

This will configure this client’s Duplicati instance to cache the dblock files at this location that is actually a CIFS mount of the Sia gateway appliance’s volume which means that both this client AND the Sia gateway appliance can see the same folder and have the same logical path to it.

You can repeat steps 1-7 above for each client you wish to backup.

NOTE: I personally run Duplicati on my client machines as a systemd service so that it starts up automatically. I also configure the web UI to be remote accessible so that I can access all the backup software instances from my main computer. It would be awesome to one day have a software or appliance that provides a central pane of glass for multiple Duplicati instances across multiple clients, but I do not know if one exists at this time.

Now it’s time to configure your job! However, remember that we must NOT run the job until the last step. So with this in mind, proceed to setup your jobs as per below.

  1. In the Duplicati web UI, add a backup…
  2. On the General step, set the following:

Name: whatever you want
Encryption: No encryption
The reason why we disable encryption is because Sia already encrypts the data. It is not recommended to double-encrypt it.

  1. On the Destination step, set the following:

Storage Type: Sia Decentralized Cloud
Server: the hostname/DNS name or IP of your Sia gateway appliance and the port 8000 that we setup using socat!
Folder path: if you plan to have multiple jobs, then splice them out into sub-folders; for example: /backups/job1
Server password: leave empty
Minimum redundancy: Choose your setting after reading the explanation below the box; for critical jobs, I set mine to 3

Example:
Screenshot%20at%202018-08-18%2014-38-00

  1. Click the Test connection button; you should get a “Connection worked!” if your Sia daemon is reachable via socat
  2. On the Source Data step, choose your folders/files to backup
  3. On the Schedule step, create your schedule
  4. On the Options step, set the upload volume size to 1GB; you can play around with this setting, based on your change data size etc, but I have read that this is a good average size; this just means backup larger than 1GB will be broken into 1GB files for upload
  5. Click Save and ignore the warning about No encryption

DO NOT RUN THE BACKUP YET!! We still have to make sure the blockchain has been fully downloaded, and you have to unlock your wallet.


(CONT.) SIA BACKUP GATEWAY APPLIANCE CONFIG
Back on the appliance, let’s check the progress of the blockchain download.

  1. In the folder where you extracted the siad package and ran siac, run ./siac again
  2. Check the Progress (estimated): percentage to see if the blockchain has been downloaded completely; if not, you have to give it more time

Once the blockchain has been fully sync’d, you can proceed below:

  1. Unlock your Sia wallet: ./siac wallet unlock
  2. Check your Sia wallet balance: ./siac wallet
  3. If you don’t have a Confirmed Balance, you need to get some SiaCoin! Reference Guide to Buying Siacoins (SC) - SiaSetup and then come back here
  4. If you do have a Confirmed Balance, check that you have Contracts: ./siac renter contracts
  5. If you don’t have any Contracts, check that you have set an Allowance: ./siac renter allowance
  6. If you don’t have an Allowance, read about setting an Allowance to decide what you want to set at Ubuntu Manpage: ./siac-renter-setallowance - Set the allowance and then come back here
  7. Create your allowance: ./siac renter setallowance and the amount and period

Example: ./siac renter setallowance 100SC 1w
If you want to spend a maximum of 100 SiaCoin in a week

At this point, the Sia daemon will start forming contracts for you. This can take one or more hours!

  1. Check how many Conracts you have at any given time: ./siac renter contracts | wc -l
  2. If you have 30 or more Contracts, you can start your Duplicati backup!

Duplicati Auth Issues with Sia-UI Network issues since 1.3.4
#2

Hi TitaniumCoder477,
Many thanks for this.
I have set up a new VM (Ubuntu 18.04) when I saw your post. I’m currently downloading the blockain again (strange because I pointed to the version I had downloaded a few days ago and it appears to start again).
One question: I’m using Duplicati version 2.0.3.3_beta_2018-04-02. Do you use another version, and is it mandatory - to your knowledge - to use it?
I get in Duplicati in the ‘Backup destination’ page, the error “Failed to connect: Input string was not in a correct format.”
I had an error with socat, it appears I had a space issue. Not sure if I copied it from this ‘how to’ but may be worth knowing for others.
If you have any ideas… in the meantime I’ll wait for the blockchain to finish downloading and will reboot everything afterwords.
Tugdual


#3

Hey tugdualenligne,
I also use Duplicati - 2.0.3.3_beta_2018-04-02. I think I have used others as well with successful results. So I do not think the version is the problem. So you are getting 'Failed to connect: Input string was not in a correct format" error message on the Backup Destination page? Are you using correct IP:port format? (or you can use hostname:port as well) If so, I would suggest first troubleshooting from a network perspective. For instance, make sure your client VM (the one with Duplicati install) can ping same IP or hostname of Sia VM from Terminal. Then, once you verify ping is OK, install nmap as well and run nmap [IP or hostname] to run a port scan against your Sia VM. If socat is running correctly, you should see open port 8000. If you can ping OK and nmap shows port OK but you still get error, then problem is likely socat. Maybe it is not forwarding correctly to internal port 9980. Or, maybe siad is not listening correctly on internal port 9980.

Example


#4

Thanks
I can ping the Sia VM from the Duplicati one. The Duplicati NMAP gives port 8000 open.
However Sia NMAP does not show the 9980 port but only the 8000. Not sure this is normal?
In Duplicati’s Backup destination, I’m using 192.168.0.6:8000 (.6 being my Sia VM). I still get the error message.
So, as you said, error shall come from socat or siad.
A netstat command gives this:
tcp 0 0 0.0.0.0:8000 0.0.0.0:* LISTEN 633/socat
tcp 0 0 127.0.0.1:9980 0.0.0.0:* LISTEN 561/./siad
Not sure what to do next ;-(


#5

Correct. Port 9980 does not need to be LAN accessible; it only needs to be localhost (127.0.0.1) accessible because the local siad process will need to connect to it. The purpose of socat is to be a running process that takes LAN connections on 9980 and passes them to the siad process listening on 127.0.0.1:9980. So I think you have that setup correctly.

Try running socat under sudo or root. I just noticed that I am running that process under root at the moment, though I seem to recall running it under non-root without a problem too. Let me know if that makes a difference.

Also, I just want to clarify… You are running socat on the Sia VM, right? Both siad and socat must be running on the Sia VM.


#6

Thanks.
I tried running socat under sudo (and I confirm it is run in the Sia VM) but it does not change anything: same error message.
I tried installing another Duplicati instance in the Sia VM to see if direct access ease the thing: unfortunately I have the exact same error message in the Backup destination page: “Failed to connect: Input string was not in a correct format.”
Out of ideas from here :wink:


#7

Go ahead and post here exactly what you have on Destination tab, i.e. copy & paste what you have for Storage Type, Server, Folder Path, and Minimum Redundancy. Or post screenshot. I will probably spot the issue. Or, if you have managed to skip to the end and Save, then use drop down arrow on job > Advanced > Commandline > and copy & paste here what you have for Target URL.


#8

Thanks TitaniumC,
Here’s the Target URL:
sia://192.168.0.6:8000//BD?sia-password=&sia-targetpath=%2FBD&sia-redundancy=1.5


#9

Interesting. This is what mine looks like:

sia://wil-sia-srv:8000//backups/gideon/jamesprofile?sia-password&sia-targetpath=%2Fbackups%2Fgideon%2Fjamesprofile&sia-redundancy=3

Yours looks essentially the same except for the '=' after sia-password. Not sure if that has anything to do with the problem or not.


#10

TitaniumCoder477,
I know this is crazy but apparently the issue comes from the redundancy parameter: setting it to 2 (instead of 1.5) works! The connection now works. Can’t believe it, to be honest.
However, for one first test job, I’m always thrown errors to repair the database. No matter what I do, it continues…
Error 1:
“Found 3 files that are missing from the remote storage, please run repair” and so on, no repair operation actually cures this

And for another backup job, “Object reference not set to an instance of an object”
with detailed error message:
Error 2:
" Fatal error
System.NullReferenceException: Object reference not set to an instance of an object at Duplicati.Library.Main.BackendManager.List () [0x00049] in <ae134c5a9abb455eb7f06c134d211773>:0 at Duplicati.Library.Main.Operation.FilelistProcessor.RemoteListAnalysis (Duplicati.Library.Main.BackendManager backend, Duplicati.Library.Main.Options options, Duplicati.Library.Main.Database.LocalDatabase database, Duplicati.Library.Main.IBackendWriter log, System.String protectedfile) [0x0000d] in <ae134c5a9abb455eb7f06c134d211773>:0 at Duplicati.Library.Main.Operation.FilelistProcessor.VerifyRemoteList (Duplicati.Library.Main.BackendManager backend, Duplicati.Library.Main.Options options, Duplicati.Library.Main.Database.LocalDatabase database, Duplicati.Library.Main.IBackendWriter log, System.String protectedfile) [0x00000] in <ae134c5a9abb455eb7f06c134d211773>:0 at Duplicati.Library.Main.Operation.BackupHandler.PreBackupVerify (Duplicati.Library.Main.BackendManager backend, System.String protectedfile) [0x00066] in <ae134c5a9abb455eb7f06c134d211773>:0

And with this job i got also this error message: “Previous volume not finished, call FinishVolume before starting a new volume”.
Detailed error 3 log is:
" Failed to process path: /sharedfolders/Films/3 hommes à abattre.m4v

Mono.Data.Sqlite.SqliteException (0x80004005): Abort due to constraint violation UNIQUE constraint failed: BlocklistHash.BlocksetID, BlocklistHash.Index at Mono.Data.Sqlite.SQLite3.Reset (Mono.Data.Sqlite.SqliteStatement stmt) [0x00096] in <fe9fd999cd9f407db94500dce293e66f>:0 at Mono.Data.Sqlite.SQLite3.Step (Mono.Data.Sqlite.SqliteStatement stmt) [0x00046] in <fe9fd999cd9f407db94500dce293e66f>:0 at Mono.Data.Sqlite.SqliteDataReader.NextResult () [0x00129] in <fe9fd999cd9f407db94500dce293e66f>:0 at Mono.Data.Sqlite.SqliteDataReader…ctor (Mono.Data.Sqlite.SqliteCommand cmd, System.Data.CommandBehavior behave) [0x00051] in <fe9fd999cd9f407db94500dce293e66f>:0 at (wrapper remoting-invoke-with-check) Mono.Data.Sqlite.SqliteDataReader:.ctor (Mono.Data.Sqlite.SqliteCommand,System.Data.CommandBehavior) at Mono.Data.Sqlite.SqliteCommand.ExecuteReader (System.Data.CommandBehavior behavior) [0x00006] in <fe9fd999cd9f407db94500dce293e66f>:0 at Mono.Data.Sqlite.SqliteCommand.ExecuteNonQuery () [0x00000] in <fe9fd999cd9f407db94500dce293e66f>:0 at Duplicati.Library.Main.Database.LocalBackupDatabase.AddBlockset (System.String filehash, System.Int64 size, System.Int32 blocksize, System.Collections.Generic.IEnumerable1[T] hashes, System.Collections.Generic.IEnumerable1[T] blocklistHashes, System.Int64& blocksetid, System.Data.IDbTransaction transaction) [0x000d7] in <ae134c5a9abb455eb7f06c134d211773>:0 at Duplicati.Library.Main.Operation.BackupHandler.AddFileToOutput (Duplicati.Library.Main.BackendManager backend, System.String filename, System.Int64 size, System.DateTime lastmodified, Duplicati.Library.Main.IMetahash metadata, System.Collections.Generic.IEnumerable1[T] hashlist, System.String filehash, System.Collections.Generic.IEnumerable1[T] blocklisthashes) [0x0000a] in <ae134c5a9abb455eb7f06c134d211773>:0 at Duplicati.Library.Main.Operation.BackupHandler.HandleFilesystemEntry (Duplicati.Library.Snapshots.ISnapshotService snapshot, Duplicati.Library.Main.BackendManager backend, System.String path, System.IO.FileAttributes attributes) [0x005c3] in <ae134c5a9abb455eb7f06c134d211773>:0

All in all, it seems also quite unstable - the siad process crashed several times. I’ve added RAM to the VM: from 1 Gb to 3Gb which is my max). I had another siad crash with this new VM set-up while trying to run a new backup, with error 2 and 3 above.

Many thanks.


#11

Glad you found the root issue!

As for the stability, I too experienced instability when using anything less than 8GB of RAM on my Sia VM. I probably didn’t need to give it four cores, but I was also testing concurrent jobs too. The only instability I currently have with it is when my Internet connection is interrupted. It doesn’t like that very much and sometimes requires repairing or rebuilding the database.

My personal opinion is that it should definitely be only a tertiary/archive augmentation to an existing robust backup solution. Or, it should at least be used with data that perhaps one wishes to offload to free up space locally BUT doesn’t really care to lose if that were to happen.


#12

You’re right. My impression nowadays is that it is very far from being a usable solution (not even talking mainstream here)…
Quite disappointed. Will follow the project to see if improvements will come in