Duplicati on Synology via Docker?!

You shouldn’t copy it while the container (and therefore Duplicati) is running. Stop the container, then copy the Duplicati-server.sqlite file, then start the container back up. See if that works better for you…

Do that first before the container is running. It should ideally be done before the container is run for the first time, I’m not sure if it would cause problems if you try to do it after there is data present in the container at that location. Feel free to delete the container and start over, Docker makes it easy to experiment!

/data/Duplicati can be mapped after the container is created. If the externally mapped folder is empty then the internal container files will appear there. I did try to stop the container, copy over the database and then restart the container, but it just kept crashing. So I now set up everything manually again. Bad luck.

Just when I thought that I could finally start the backup another culprit turned up. The hidden/system folders “@eadir” that contain thousands of thumbnail files are not marked as system/hidden inside the Docker folder map. This means I would have to back these up, too, because it’s impractical to go through dozens of image subfolders to manually remove them from backup. But that’s tens of thousands more files that increase backup size from 200 to 300 gb.

Duplicati needs an Exclude option that allows to exclude files and folder based on names.

For comparison I installed Mono 5 and Duplicati for Synology (Canary) again. Turns out that the Synology version does not identify the @eadir folders as hidden/system either.

So I guess it’s back to backing up via Windows PC over SMB share. Too bad.

I wouldn’t have guessed it would behave that way. Very interesting…

You can definitely exclude based on folder name. I’ll look into this, I don’t think I noticed my Duplicati backing up the @eadir folders. I’ll update later after I check.

Duplicati on my NAS doesn’t back up @eadir folders. I wasn’t sure why until I checked my filtering. I had added these a couple years back and forgot about it. You can add similar filters on the Source Data page of the backup set config:

Capture

Stupid me. I was looking under “Exclude” for something like this, instead of looking under “Filter”. Of course it would still be preferable if Duplicati properly detected these folders as system, hidden and temp, like it does on Windows.

The initial backup is about 30% through with uploading the files. I may cancel the upload and then try to move it to the NAS to see how well that works with unfinished backups. Main problem is that the source path is different, because on the NAS there is only “My Documents” and “Home” under “User Data”. I did not find a way to add my own shares there.

With the Docker version I cannot use my NAS’ IP as source, with the Synology version I could. That way the backup could be moved more easily. I can set the container to use the “host” network instead of the bridged one, that way the container has access to the NAS IP. Any known drawbacks to that approach?

Unfortunately the Synology version (Canary + Mono 5) only worked yesterday (needed two tries at the beginning, though), today it keeps crashing after a fresh install. No idea why.

Hm, curiously the Docker version of Duplicati does not allow backslash IP paths’: "The path must be an absolute path, i.e. it must start with a forward slash ‘/’ ".

On the Windows version this works, I can add something like “\\192.168.x.y\share\folder”.

Is there a way to throttle an ongoing upload from a Docker container (or Synology NAS)? On Windows I can do that with 3rd party software (CfosSpeed), but to do it from within Duplicati it seems that I have to first cancel the ongoing upload, change the throttle option and restart again.

It won’t because Linux doesn’t work like Windows in this regard. Linux has no concept of a “system” attribute, and it considers a file “hidden” if its name starts with a dot. Not sure what you mean by temp files - do you mean a third DOS/Windows attribute called “archive”? Linux has no equivalent for that either.

You can find everything under “Computer” in the backup job Source Data tab. For docker version of Duplicati you are presented with a view of the filesystem that the container sees, which isn’t the same as the host NAS device. The only folders on the host that you’ll see in the docker view of the filesystem are folders you’ve mapped to the container in the Docker configuration.

I’m not sure what you mean by this. You should be selecting your source folders under “Computer” in the Source Data tab. Host mode networking in Docker is unrelated.

Again I’m not really sure what you are trying to do here. But Duplicati is correct. Linux doesn’t use backslashes like Windows does. The path separator character is the forward slash.

In general I would not back up data via CIFS/SMB shares. It’s better to install Duplicati on the local machine/device and back up the normal local drive letters (Windows) or paths (Linux).

I’m not familiar with the throttling feature of Duplicati, as I control/prioritize my outbound internet traffic a different way. But docker should be no different in this regard than any other install of Duplicati. I don’t recall if you can change the throttling on the fly, or if it doesn’t go into effect until it starts the next file.

Synology DSM seems to consider folder hidden and system that start with a “@”. Not only do these folders not show up in File Station, but they also don’t show up in SMB shares. Because of the latter I did not have to filter the “/@eadir” in the Windows Duplicati job. Only once I moved that over to Synology/Docker did I notice the extra files.

Now that I know about it I can handle it manually, but the Windows version is easier to handle in this regard. The Windows version also handles the “#Recycle” folders as hidden and thus excludes them when the corresponding option is set in the backup job.

Not sure what you mean by temp files - do you mean the third DOS/Windows attribute called “archive”? Linux has no equivalent for that either.

The Windows version offers the following “Exclude” options:

Hidden files, System files, Temporary files, Files larger than:

I assume that “Temporary files” automatically excludes files based on certain file-name features, like .tmp extension or files beginning with a ~ and the like. Unfortunately I did not find a documentation on that yet.

You can find everything under “Computer” in the backup job Source Data tab. For docker version of Duplicati you are presented with a view of the filesystem that the container sees, which isn’t the same as the host NAS device. The only folders on the host that you’ll see in the docker view of the filesystem are folders you’ve mapped to the container in the Docker configuration.

Yes, of course. But everything under “Computer” is a hard-coded path inside the backup job. On Windows everything under “User Data” has the advantage that its path can be changed externally. When Duplicati is set to backup “My Documents” or “My Pictures” then I can change the path of these folders in Windows and Duplicati will still back them up without having to change the backup job.

Everything under “Computer” looks different on different computers and even more so on different operating-systems. On Linux/Synology you see shares (like /photo), on Windows you see drive-letters (like P:).

Because of this I opted for the third alternative, adding a “path directly”. On Windows I added "\nas-ip\share\folder" directly, which points to the very same location as “//photo/pictures/” or "P:\pictures" or “My Pictures” would. But when I move the job over to another PC in the same network it would work out of the box using the same NAS IP, instead of me having to change the source first.

Why is this relevant? Because changing the source in Duplicati likely leads Duplicati to identify all files as “new” files, even though the different Duplicati “Source” points to the very same NAS source folders. I am still in the process of testing this, but realize that the behavior may change with any new Duplicati version anyway.

I’m not sure what you mean by this. You should be selecting your source folders under “Computer” in the Source Data tab. Host mode networking in Docker is unrelated.

“Host” mode networking makes the Docker container accessible under the same sub-network address as the Synology NAS. Instead of an isolated 172.17.0.0/16 network you can access it via your local sub-net (usually 192.168.x.0/24). This way the container should reach the NAS under the very same IP as any other Duplicati installation outside the container would (like 192.168.x.y) and thus the backup job can be moved in and out of containers without having to change the “Source”.

Using the container via “HOST” network also exposes the Duplicati web UI via port 8200 on the host IP.

Again I’m not really sure what you are trying to do here. But Duplicati is correct. Linux doesn’t use backslashes like Windows does. The path separator character is the forward slash.

I am trying to add a path that includes an IP address pointing to a SMB share. Said SMB share is on the very same NAS that the container is working on, but since the container does not know how to access SMB shares I cannot use this. It was worth a try, though.

In general I would not back up data via CIFS/SMB shares. It’s better to install Duplicati on the local machine/device and back up the normal local drive letters (Windows) or paths (Linux).

The drawback of that approach is that each backup job is only valid on the machine that created it. If you want to move backup jobs between machines without starting the whole backup/upload over there likely are more or less painful ways of doing it right. I am trying different possibilities to find a “best practice” for when the need arises in the future.

I’m not familiar with the throttling feature of Duplicati, as I control/prioritize my outbound internet traffic a different way. But docker should be no different in this regard than any other install of Duplicati. I don’t recall if you can change the throttling on the fly, or if it doesn’t go into effect until it starts the next file.

It seems that the backup job has to be restarted for Duplicati throttling to come into effect, regardless of whether it’s set up in the job or general settings.

Synology’s “Traffic Control” can throttle Docker containers that use the BRIDGE interface, containers via HOST interface may work, too, if the specific port is set to be throttled. I have to test yet if this can be changed while an active upload is running.

On my Windows PC I am using a software called “CfosSpeed” that allows to set up both priorities and bandwidth throttling. Unfortunately using priorities does not help much with Duplicati’s upload to Sharepoint, I have to use throttling, else my occasional online gaming traffic is stalled (despite Duplicati being low and the game being high in priorities). That’s still a local control, though, while I would have to first log into the NAS in order to change that on the fly. Not much more work, just different.

Turns out that Duplicati does not seem to throttle on its own at all.

I just set “throttle-upload” to 1500 kb/s, both via job and general settings before restarting a job that was cancelled before. Upload still happens at the full 3.9 mb/s bandwidth of my internet connection.

That’s DSM doing it in File Station and their Samba layer. When you run native Linux software, it doesn’t go through that layer and you see all the files as you would on a normal Linux system.

Ok, I understand what you are saying. I still think it’s preferable to back up files on the NAS with a Duplicati instance running on that NAS, instead of doing it over the network. But if it works for you then it should be fine.

If Duplicati thinks a file is new, it will reprocess it. But with its deduplication engine, it won’t need to save file contents as new blocks of data on the back end.

But that happens anyway with the default networking model. Docker does a sort of NAT between the host and the container’s internal IP address. Port 8200 on my NAS gets mapped to the internal container.

Also in either networking mode your container can access the NAS or other network resources by its normal LAN address.

There are special circumstances where host mode makes sense, but for almost all containers it is not required. It should not be required in the use case you are outlining here, but you’re probably not hurting anything by using it. It would become problematic if you wanted to run multiple docker containers that would otherwise want to use the same ports though. The NAT/port mapping layer helps mitigate that. If this is your only container you plan to run and it doesn’t conflict with anything on the host, you probably won’t see any downsides to host mode.

On Linux the convention is to use forward slashes always, even if you are referring to an SMB or CIFS share. For example this mounts a share on my Linux PC:

mount //nas/Public /mnt

That being said, I don’t know if it would work to type a UNC path in directly to Duplicati on Linux. Normally SMB shares need to be mounted before accessed.

That’s an interesting use case and not one I’ve thought about. I just set up Duplicati on each machine that has data to protect and don’t move it around like that. I see from your latest posts that this is what you are trying to do and now I understand why you are having more questions and issues. You have a more complex setup than I do and I never had to deal with these challenges…

That could be… I don’t use throttling myself in any application.

I use QoS and traffic shaping at my gateway/router level to prevent low priority traffic (backup traffic, etc) from using all my upstream bandwidth. It can use it all if nothing else is going on, but as soon as normal or high priority traffic wants to use upstream the low priority instantly gets throttled back. Works great but was trickier to set up!

Good luck with your backup adventures, I hope all the quirks get worked out!

Thanks for all the information and responses. I should clarify that I am mostly testing the limits and limitations of Duplicati and its various installation bases. For example, I do not plan on moving backups jobs around, I am testing beforehand what happens when I am forced to do so and what method is the least painful and success-promising. You could say that I am putting Duplicati in jeopardy in order to know what happens before it happens really happens. Well… that on top of the real hurdles and issues I am stumbling over while setting up my personal backup. :wink:

Yes, it seems so. I am not a Linux person, even though I do manage to spontaneously find my way around SSH and the file-system. So having tens of thousands of files backed up on the Synology/Linux based machine that did not show up on the Windows machine came as a surprise and I mostly just shared the experience here.

That being said, if Duplicati is able to identify “system” and “temporary files” on a Windows system then it’s not unreasonable (for the user) to expect that it also does to on a Synology system. I know the limitation now and how to workaround it via filters (thanks again), Duplicati still should learn about this for its Synology build, though.

Both have their pros and cons. Generally I am trying to get Duplicati running properly on the NAS and will likely prefer that solution in the future. But I have to test and be prepared beforehand what happens when the NAS, Duplicati, Docker or whatever fails one day. It’s important to know what obstacles happen when jobs are transferred from one to another system (and maybe OS), and it helps to know how to create a backup job properly so that can transfer easier.

The reprocessing is still an unnecessary hurdle that can take quite a lot of time, especially when it’s bottle-necked by network bandwidth or (lack of) CPU optimizations. If Duplicati deems a file as new because the “Source” setting of its corresponding backup job changed then we want to keep changes to that setting minimal. Using an IP instead of a named share/folder seemed like a possible way to do that.

I know, but thought to share my experience with using HOST instead of BRIDGE.

I did not expect this to work over the BRIDGE network, but I just tried pinging several local LAN resources out of a CentOS container and it works over both BRIDGE and HOST networks. Good to know.

The Duplicati docker still cannot access the SMB share via IP address, so this solution is out when Docker is used. The Synology version of Duplicati should be able to do that, but my last try ended in constant crashes again (despite it working just a day before).

That’s what the software CfosSpeed is supposed to do on my Windows PC. Unfortunately it does not work properly with the combination of Duplicati upload + gaming traffic. My router offers QoS, too, but the remote port for the Sharepoint upload is 443 and I would prefer not having to put all SSL traffic on low priority.

Yeah I wouldn’t want to do that either. If your router has the ability to check QoS tags on individual packets - instead of classifying by tcp port - then you may have an alternate solution available. Windows PCs can tag packets coming from certain processes with QoS flags so that the router knows they are low priority traffic. That’s probably a discussion for a different thread though :wink:

I don’t think that my router offers that, so Synology’s Traffic Control will have to be enough. Those QoS tags would have to come from the Docker container or Synology NAS once successfully move Duplicate to the NAS.

It can identify single Docker containers when they use the BRIDGE network, so that will help to throttle manually for long uploads.

Unfortunately those filters do not work. Alternative I tried to use “Exclude directories whose name include” “@eaDir”, this seems to work for all those sub-folders going by that name.

*Edit: I first claimed that “Exclude directories…” did not work, but it does. I accidentally used an upper-case “E” instead of lower-case.

Strange…they work for me. Duplicati is definitely not backing up those folders.

Where are you using those filters? Docker instance of Duplicati?

Sorry, my mistake. I just tried your filters again, but this time looking specifically for lower-case. When I re-tried before I did not notice that I still used upper-case E with your filter. Again, stupid me. Using lower-case “e” works with both filter-types.

Thanks again for the hint!

I am too used to case-insensitive Windows, which I admittedly prefer over case-sensitive systems.

Yep, that case-sensitive filesystem will getcha… glad you figured it out!