INTRODUCTION
This HOWTO describes the process of creating a virtual or physical server that is a gateway to the Sia network. Here are some objectives:
- Single wallet requirement (vs. needing to have a wallet on every backed up computer)
- Headless deployment (vs. needing more resources for GUI)
- Simultaneous client backups
DISCLAIMER: This HOWTO describes deployment in a controlled lab environment. Any use outside of such environment is at your own risk and, at a minimum, should consider common-sense security measures which are not the focus of this topic. As always, the 3-2-1 rule applies in backup design. You should have three copies of your data, including production, two of which should reside on different media and at least one of which resides offsite. It may NOT be wise to consider this deployment as your only offsite solution! A wiser choice would probably be to consider it a second offsite solution or even an archive offsite solution that augments your existing 3-2-1 deployment!
OVERVIEW
Here is a high-level diagram of the deployment design:
I will list my personal deployment below. Please know that none of this is a recommendation; it is just what I had available to me for testing. You are definitely encouraged to improve upon it with your own design choices!
- Host hardware: Dell PowerEdge R410 (1U server w/ 16 cores and 64GB RAM)
- Host OS: Linux Mint 18.3 Sylvia 64bit
- Hypervisor: Oracle VirtualBox 5.2.12 r122591
- Sia server: Ubuntu Server 18.04 LTS VM w/ 4 cores and 8GB RAM
- Client machines: A mix of VMs (on the same host), laptops, etc. â all Linux OS
NOTE: I have not tested this deployment with client machines that run a Windows OS. The reason why this deployment works is because the client machines AND the Sia gateway appliance can all access the /media/cache folder. I am not sure how this would work with Windows OSâs because obviously a CIFS mount will not have the same logical path, i.e. /media/cache, but rather some drive letter such as M:\duplicati-cache. More on this later.
Here is a low-level diagram of my deployment design:
REQUIREMENTS
Sia server configuration:
- Must have a cache volume (i.e. /media/cache) which can be internal storage or a mounted iSCSI volume
- Must share cache volume via Samba
- Must run socat to listen on external 8000 and forward to internal 9980
- Must run siad which listens internally on 9980
- Must have wallet unlocked (I use siac to do this and to check status on uploads etc)
Client(s) configuration:
- Must have Sia serverâs cache volume mounted via CIFS/Samba at same location (i.e. /media/cache); in other words, both server and client(s) must have the same local path to the same volume
- Duplicati must be installed (GUI or headless is fine) and global tempdir must reference local path to cache volume (i.e. /media/cache); it is recommended to append a UID, such as clientâs hostname, to this (i.e. /media/cache/myclient1); this allows the clients to keep their Duplicati files separate from each other
- Duplicati jobs must be configured to use port 8000
- Duplicati job dblock (upload) sizes can be whatever, but I used 1GB as per what I found others recommended for Sia blockchain
These requirements will be explained in more detail below.
STEPS
This is a rough outline of some steps you could follow to deploy a similar Sia backup gateway appliance. Since most people probably do not have a virtual lab available, I will actually outline the process using spare computers instead.
SIA BACKUP GATEWAY APPLIANCE PREP
-
Choose an old computer to be your Sia backup gateway appliance
NOTE: You will need a good bit of space for the Sia blockchain AND for the Duplicati cached backup files (the chunks that Duplicati creates prior to uploading). Therefore, it is a good idea to use a 2nd hard drive OR an attached USB drive (permanently mounted, of course) OR (in the case of a virtual server) a dedicated virtual disk. Technically, you could even carve out an iSCSI LUN from a NAS and attach it to the appliance for this purpose, but I will not go into detail on this. The basic idea is have plenty of space available! And âplenty of spaceâ is qualified by at least enough for the Sia blockchain + the size of your backup job data (which is hard to estimate because of compression, I know).
Example: I have a 30GB dedicated volume for Sia and a 100G dedicated volume for my Duplicati cache
-
Install Ubuntu Server (or workstation if you want a GUI) and update packages
sudo apt update && sudo apt upgrade -y && sudo reboot -
Install SSH, Samba, Socat, unzip, and screen; if you are running headless and want a nice web UI, also install webmin
-
Confirm that you have access to the appliance either by SSH or using a GUI
-
Not a bad idea to drop the firewall, at least until you get everything working; then you could create rules to lock down all but what you need open
SIA BACKUP GATEWAY APPLIANCE CONFIG
Weâre going to create two mount points, one for the Sia blockchain and our files and one for the Duplicati cache for all our clients. You can see the previous section point 1 for ideas on the volume itself.
- Create a mount point for Sia at
/media/sia - Create a mount point for the Duplicati cache at
/media/cache - Grant permissions on both folders to whatever account you plan to run Sia under
Example: Volumes
Example: fstab
- Download siad (the Sia Daemon package, not the Sia UI)
- Extract siad package to
/media/sia
Example: (ignore the different domain name than previous screenshots; I stood up a new VM for cleaner examples)
-
cd into that folder and run:
./siad --sia-directory ../blockchain &
This will create/media/sia/blockchain/and point Sia daemon to it. The Sia daemon will start in the background. (Press Enter a 2nd time on your keyboard to get back to the command line after youâve run a process with an & symbol at the end.) -
Now in the same folder, type:
./siac
You should seeProgress (estimated):and some percentage. This is the percentage of your blockchain download. You can continue with the instructions below, but you should wait until this is 100% before you actually unlock your wallet or try to perform a backup. NOTE: This could take a day or more to download, depending on your Internet speed and other factors!
While the blockchain is downloading, letâs proceed with some additional setup.
The Sia daemon only listens on localhost (i.e. internally) for security reasons. I tried various ways around this, but the solution that proved the best in the end was a simple forward with the socat program.
- Type:
socat tcp-listen:8000,reuseaddr,fork tcp:localhost:9980 &
This will cause socat to run as a background process and forward any inbound connections on port 8000 to the local Sia daemon that is listening by default on port 9980.
FYI: If you want to bring either background process to the foreground, just type: fg followed by the number (1 for the siad background process we created in 6 above or 2 for the socat process we created in 8 above). Then if you want to send it to the background again, press CTRL+Z (to suspend it) and then bg followed by the number again. And even BETTER way to handle multiple processes is to use the screen program. But you can look that up and learn how to use it on your own!
So at this point, we have siad running in the background and downloading the blockchain. And we have socat running in the background, forwarding any traffic from external port 8000 to internal port 9980.
Next, we need to setup our Duplicati cache folder and Samba.
The /media/cache folder is a central repository that both the Sia daemon AND the Duplicati clients need to see and use. Basically, the way it works is that the Duplicati client stages its dblocks (upload files) there and then instructs the Sia daemon to upload them. Obviously the Sia daemon can access the files locally at the /media/cache folder. However, we need to use Samba to give the client machines (where Duplicati is installed) access to the same location at the same path, i.e. /media/cache that is local to them. Hopefully this will become more clear as you go through the steps below.
-
Add the following lines to the end of your /etc/samba/smb.conf file
[duplicati-cache]
comment = Duplicati backup cache
path = /media/cache
available = yes
browseable = yes
public = no
read only = no
guest ok = no
writeable = yes -
Restart Samba by typing:
sudo systemctl restart smbd
Now one last thing on the Sia Backup Gateway Appliance before we proceed to the Duplicati client machines: It is important that the files created in this /media/cache folder are read/writeable for BOTH the user account under which siad is running AND the client machines on which this Samba share will be mounted. The way I handled this was just making sure I was using the same Linux username (jwilmoth) and UID and group (jwilmoth) and GUID. However, you may want to do something different.
NOTE: As mentioned at the first, I have not tested this deployment with client machines that run a Windows OS. The reason why this deployment works is because the client machines AND the Sia gateway appliance can all access the /media/cache folder. I am not sure how this would work with Windows OSâs because obviously a CIFS mount will not have the same logical path, i.e. /media/cache, but rather some drive letter such as M:\duplicati-cache.
DUPLICATI CLIENT MACHINE (LINUX) CONFIG
- Create the folder
/media/cachewith same permissions as the previous section, step 3 - Mount the CIFS share in the previous section, step 9 at this folder
Example fstab entry: //wil-sia-srv/duplicati-cache /media/cache cifs users,user=jwilmoth,uid=1000,gid=1000,username=jwilmoth,password=[your_password] 0 0
(replace [your_password] above with the password used to access the share)
-
Create a test file in
/media/cacheand verify that you can see it and have read/write access to it from the Sia gateway appliance at the same location -
Install Duplicati
-
Access Duplicati at http://localhost:8200/ngax/index.html
-
Click Settings and under Default options > Add advanced option⌠locate tempdir under the Core options section (easiest to find if you just start typing)
-
Set the tempdir path to
/media/cache/[client name]where [client name] is either the hostname of your client computer or something to uniquely identify it from other client computers; the idea that every client computer will have its own sub-folder under/media/cachewhere its instance of Duplicati can cache files
For example:
This will configure this clientâs Duplicati instance to cache the dblock files at this location that is actually a CIFS mount of the Sia gateway applianceâs volume which means that both this client AND the Sia gateway appliance can see the same folder and have the same logical path to it.
You can repeat steps 1-7 above for each client you wish to backup.
NOTE: I personally run Duplicati on my client machines as a systemd service so that it starts up automatically. I also configure the web UI to be remote accessible so that I can access all the backup software instances from my main computer. It would be awesome to one day have a software or appliance that provides a central pane of glass for multiple Duplicati instances across multiple clients, but I do not know if one exists at this time.
Now itâs time to configure your job! However, remember that we must NOT run the job until the last step. So with this in mind, proceed to setup your jobs as per below.
- In the Duplicati web UI, add a backupâŚ
- On the General step, set the following:
Name: whatever you want
Encryption: No encryption
The reason why we disable encryption is because Sia already encrypts the data. It is not recommended to double-encrypt it.
- On the Destination step, set the following:
Storage Type: Sia Decentralized Cloud
Server: the hostname/DNS name or IP of your Sia gateway appliance and the port 8000 that we setup using socat!
Folder path: if you plan to have multiple jobs, then splice them out into sub-folders; for example: /backups/job1
Server password: leave empty
Minimum redundancy: Choose your setting after reading the explanation below the box; for critical jobs, I set mine to 3
Example:
- Click the Test connection button; you should get a âConnection worked!â if your Sia daemon is reachable via socat
- On the Source Data step, choose your folders/files to backup
- On the Schedule step, create your schedule
- On the Options step, set the upload volume size to 1GB; you can play around with this setting, based on your change data size etc, but I have read that this is a good average size; this just means backup larger than 1GB will be broken into 1GB files for upload
- Click Save and ignore the warning about No encryption
DO NOT RUN THE BACKUP YET!! We still have to make sure the blockchain has been fully downloaded, and you have to unlock your wallet.
(CONT.) SIA BACKUP GATEWAY APPLIANCE CONFIG
Back on the appliance, letâs check the progress of the blockchain download.
- In the folder where you extracted the siad package and ran siac, run
./siacagain - Check the
Progress (estimated):percentage to see if the blockchain has been downloaded completely; if not, you have to give it more time
Once the blockchain has been fully syncâd, you can proceed below:
- Unlock your Sia wallet:
./siac wallet unlock - Check your Sia wallet balance:
./siac wallet - If you donât have a Confirmed Balance, you need to get some SiaCoin! Reference Guide to Buying Siacoins (SC) - SiaSetup and then come back here
- If you do have a Confirmed Balance, check that you have Contracts:
./siac renter contracts - If you donât have any Contracts, check that you have set an Allowance:
./siac renter allowance - If you donât have an Allowance, read about setting an Allowance to decide what you want to set at Ubuntu Manpage: ./siac-renter-setallowance - Set the allowance and then come back here
- Create your allowance:
./siac renter setallowanceand the amount and period
Example: ./siac renter setallowance 100SC 1w
If you want to spend a maximum of 100 SiaCoin in a week
At this point, the Sia daemon will start forming contracts for you. This can take one or more hours!
- Check how many Conracts you have at any given time:
./siac renter contracts | wc -l - If you have 30 or more Contracts, you can start your Duplicati backup!
