Replace current backup workflow with Duplicati

How-To
1

Hello and good evening everyone :wave:t2:

I’m want to replace my current backup workflow with OSS components and for this, I tend to use Duplicati but need a little more understanding of how it would look like.

This is my current backup setting:

  • Devices
    • Android client
    • NAS
    • External hard disks
  • Android backup
    • Every second day, the Android device syncs different folders from the smartphone to the NAS with a dedicated user who also has access to a specific backup directory on the NAS.
    • Sync happens over WebDAV (HTTPS) and is one-way (only from smartphone to NAS).
  • NAS
    • There are different user directories which are internally encrypted with a vendor implemenation (ASUSTOR Shared Folders).
  • External hard drives
    • Hard disk 1: contains most recent backup (e. g. 2023-06), which is a single VeraCrypt container
    • Hard disk 2: contains three older backups (e. g. 2023-03, 2023-04 and 2023-05), each a single VeraCrypt container
  • Once a month, I do the following:
    • Copy the most recent backup (e. g. 2023-06) from hard disk 1 to 2, mount it with VeraCrypt, check some files whether they are okay, unmount and delete the oldest backup (e. g. 2023-03).
    • Create a new backup (e. g. 2023-07) on hard disk 1 and delete the previous one (e. g. 2023-06).
    • A file sync tools accesses all the different user directories on the NAS via SFTP (because it does not support WebDAV over HTTPS) with a dedicated read-only user and copies the files into the new VeraCrypt backup (e. g. 2023-07) on hard disk 1.
  • Every once in a while, when visiting my family, I take hard disk 1 with me and copy the latest VeraCrypt backup on another hard disk (because I don’t want their network to be accessible via the internet/the hard disk always attached to the router).

As you can see, this involves many different steps, apps, etc. and although I’ve automated a lot, is is still a bit annoying.

Now if I understand it correctly, with Duplicati, it could look like this:

  • Install Duplicati on both the NAS (ASUSTOR supports this ootb) and Android smartphone.
  • Configure the phone to sync the relevant files every two days to the NAS.
  • Configure hard disks 1 and 2 as Duplicati storages.
  • Once a month, I attach the two hard disks to the NAS and trigger a Duplicati backup.

What remains unclear to me:

  1. Can I always create full, non-incremental backups to keep my per-month structure (e. g. 2023-03, 2023-04 and 2023-05)?
  2. How does a backup look like on a file system level? I’ve read that Duplicati seems to be block-based, i. e. are there many tiny files on the disks?
  3. When copying the files on the third hard disk at my parents house, how can I easily verify that the Duplicati backup is okay without actually restoring the backup? So far I could easily mount the VeraCrypt file and randomly check some files - not elegant, but pragmatic and easy.
  4. Can I somehow access the files coming from my smartphone onto the NAS via Duplicati from within a file explorer? Imagine the following: I take a picture with the smartphone and sync it via Duplicati onto the NAS. Now on my Windows laptop, I want to browse the images (or other files) without actually syncing them onto my laptop - at the moment I can simply browse the UNC path of the NAS where the images are synced to.

Thank you in advance for taking the time to answer my rather longish question :slightly_smiling_face:

2

Welcome to the forum @mu88

Duplicati is not available for Android, AFAIK. The download page shows what Duplicati offers.
Occasionally a third party might package it up for something else, but Android seems unlikely.
Maybe good news is it also makes little sense, if the plan was to backup the Duplicati backup.

Overview

Duplicati is not:
A file synchronization program.
Duplicati is a block based backup solution. Files are split up in small chunks of data (blocks), which are optionally encrypted and compressed before they are sent to the backup location. In backup location, Duplicati uploads not original files but files that contain blocks of original files and other necessary data that allows Duplicati to restore stored files to its original form by restoration process. This block based backup system allows features like file versioning and deduplication. If you need to be able to access your files directly from the backup location, you will need file synchronization software, not block based backup software like Duplicati.

It’s a folder with default 50 MB dblock files, each with a dindex. A dlist per backup gives contents.

How the backup process works

That’s very much going against the intention, although you could probably force a full if you desired it.

Features

Incremental backups
Duplicati performs a full backup initially. Afterwards, Duplicati updates the initial backup by adding the changed data only. That means, if only tiny parts of a huge file have changed, only those tiny parts are added to the backup. This saves time and space and the backup size usually grows slowly.

How were you going to make sure the files you copied from disk 1 were fine before you copied them?
If they were fine before copy, then various software is available to make sure copy matches originals.

To verify files with Duplicati, random restore is an option, but it runs best on original Duplicati system because otherwise local database needs to be built, but testing that is good to test disaster recovery.

If goal is simply to test file integrity, upload-verification-file can upload a file to verify the backup’s files using a provided Python or PowerShell script, however that doesn’t attempt to check any source files.

The TEST command can be a little more thorough, but you need Duplicati and an available database.

Nothing beats restore (preferably with no-local-blocks) and occasional test that database rebuilds OK.

That’s because it’s sync. Duplicati is not. You would need to restore the files before you can read them.

3

Thank you very much for your in-depth explanation :muscle:t2: looks like it is not what I am looking for at the moment, but maybe later with different needs