Are any of these known issues? (Xavron June 2021 edition)

1

2.0.6.1_beta_2021-05-03

Instead of posting a bunch of issues out of annoyance in git will try this first this time around… Duplicati is just unusable outside of development…

1. [SOLVED] Source becomes incorrect with imported backup configurations

EDIT:
SOLUTION… The following that the exported backup config.json has in it produces this problem:

“Sources”: [
“%HOME%”
]

And needs to be full paths to work correctly. At this time all previous backup configs with the above are broken! This is an export from 2.0.0.7. Users with encrypted backup configs where this happened are burned.

Import complains about repair. Mind you that this DB backup is not entirely the same of the export. Repair has no errors, backup has no errors but source changes.

This happened anyway even with a backup configuration that originally made the backup files but a little out of date.

 Source after repair and backup:
 Source:
     45.68 MB
 Backup:
     11.70 GB / 2 Versions
 
Source on import: 
Source:
     20.00 GB

Found 482 remote files that are not recorded in local storage, please run repair

Backup is of home but full path home vs user data home. Source data UI folder browser shows incorrect folders… as is eg folderA has no sub folders in it when it should be showing that it does.

Is this some known issue with import/export or maybe with the export required to be fresh? I don’t remember that being the case.

2. Possibly same or not same as above issue

There might be a cache issue as the import of the backup configuration might improve a little after a reboot once attempting another backup of same name.

Not sure of this one. It could be nothing but a reboot did have an obvious effect here.

3. [SOLVED see #10] Duplicati creates new media folder of same name of usb drive but with one char difference

Edit for solution:
Original line for this is misleading. Duplicati creates the original path and overrides the drive in /run/media so when the drive gets mounted, that switches to incremental and has the +1 char. Could use one more look.

eg /run/media/user/3B33-8E29/duplicati where 3B33-8E29 is the default UUID of the usb driver

Duplicati on import/repair or whatever causes the following to happen:

/run/media/user/3B33-8E29/duplicati
/run/media/user/3B33-8E291/duplicati

And the backups go to the one that’s not the usb drive but simply a folder on the local OS drive.

I noticed my drive was then defaulting to 3B33-8E291. Removed 3B33-8E29 and Duplicati recreated it and put the backups there again. Reboot (or remount) should solve it until Duplicati does it again. And usb drive is now back to 3B33-8E29.

4. Renaming a file while the backup (possibly only initial) is running…

Causes backups to fail to run until user manually fixes it and it is a badly named error about a path not existing when its because of a file - System.IO.IOException: Backup aborted since the source path /home/user/some-file does not exist.

5. Selecting not full path home for source data along with import/repair/delete repeatedly can cause Duplicati to erroneously use a different source directory of what appears to be only the Local database path and then be in a loop of backing up incorrect files!

Can’t find any good logs for this.

6. Unmounting usb drive used for backup and then trying to backup results in db please run repair popup…

Probably a bad way to go :wink:

7. Let’s do it! #6 repair leads to no apparent repair

The forever run and repair popup loop.

8. #6 & 7 leads to crash and burn!

Mount drive and Duplicati can no longer repair db…

Log data...
Only contains the following repeated on each attempt:
list
[

]

9. #8 then delete and recreate db leads to further crash and burn…

Popup: No files were found at the remote location, perhaps the target url is incorrect?

Destination’s test connection works. Remote files are there. Logs are empty except for one entry of what’s found in #8.

10. #9 then delete only the local db (not the files) then re-import the backup config leads to Duplicati saying, “I don’t need the original device anymore.”…

Run the backup leads to #3!

/run/media/user/3B33-8E29 - is now where Duplicati is putting the backup files (is where the usb drive was)
/run/media/user/3B33-8E291 - is now the usb drive

Other backup software still finds the drive just fine.

2

Works fine here on Windows and Linux, at least export and import on same system. Expands fine.
But see below for a non-Import-related issue on folder expansion, if Duplicati is root. Is yours root?
Please also post information on your operating system name and version, and your mono version.

There was never such a release, although this is used some places as a placeholder. From my file:

"CreatedByVersion": "2.0.6.2",
...
    "Sources": [
      "%HOME%"
    ],
...
  "DisplayNames": {
    "%HOME%": "Home"
  }

For those not wanting to open up an export (encrypted ones are decryptable with AES Crypt or included SharpAESCrypt.exe), the best way to ask an installed system its version is to look on the About screen.

Steps (what are yours, preferably reproducible on a single system version instead of requiring multiple)?:

Start Duplicati as a regular user on Linux Mint 19.3 (Ubuntu 18.04 base), with mono --version 6.12.0.122.
On Source screen, check only Home under User data. Rest of backup is pretty vanilla. No schedule set.
Export to file.
Add backup → Import from a file → (select) → Import → check Source screen. It expands, nothing runs.

Nothing’s supposed to run when I click Import AFAIK, though scheduled jobs can start running after Save.

If, however, I run as root, I can see only My Documents and Home under User folders. I need to use the Show hidden folders option to expand those. That happens even without Import, and seems suspect, however 2.0.5.1 is the same way and I can’t quickly find anybody has filed a GitHub issue asking about it.

I don’t think Duplicati gets into the mounting. When exactly does the drive get mounted? Mounting is quite involved, and I believe it’s done with software such as udisksd from the udisks project. Some software is responsible for the /dev/disk/by-* links. A UUID is 128 bits. You have 32. It might be volume serial number.

allow-missing-source can cover rare cases. If seeing this a lot, snapshot-policy can capture a stable view.

Drive is unmounted. Repair needs information from the backup. The local database is a cache of that info.

It just needs a list to work to pass the connection test. The list doesn’t need files. It could be initial backup.

Which “there” given some apparent folder confusion? It “should” be looking at exactly what Destination screen has in Folder path. If it comes up with a tree view, you can click Manually type path to see it.

3

Correct, it doesn’t mount anything. Instead Duplicati creates the exact path of the previously mounted device (and uses it) which overrides that device when it is mounted by the user or system.

Duplicati is clearly in the wrong.

If the device doesn’t exist (unmounted) Duplicati shouldn’t be trying to repair the db.

Duplicati is clearly in the wrong.

That’s what the file says. I made it a few months or less ago so it should have been a newer version but I have no way of knowing anymore. Can only go based on what it says.

That explains nothing about why its missing files on import. It goes from 20GB source to 45MB source.

I know there’s a lot of information above. None of it can be skipped.

“%HOME%” replaced with full paths clearly puts it back to 20GB source. Duplicati is in the wrong either on import or for exporting using “%HOME%” when it shouldn’t have, or otherwise %HOME% wouldn’t be working correctly.

There’s no other reason for that.

  backup config file shows:
  "TargetFilesCount": "619",
  "TargetSizeString": "14.15 GB",
  "SourceFilesSize": "21476780159",
  "SourceFilesCount": "83900",
  "SourceSizeString": "20.00 GB"

%HOME% = 45MB… showing hidden files should not fix that or even be a reason for it when they are not hidden in the first place.

I would hope to not go back and forth as to all these issues not being bugs. They happened. They are bugs on my system. I see them. I ran into them. They exist.

I guarantee you others are running into them or will.

In time I can even prove it via code. First step is reproducing them and getting the mess untangled. That was a lot of problems I ran into while trying to look into another problem not found here which I couldn’t follow through on because of all the problems I got hit with.

Just because you’re not seeing it doesn’t mean it doesn’t happen. You should know that.

Duplicati is complex with the amount of code needing to be exact in many situations. That’s a lot of combinations where things can go wrong.

4

Again, I don’t think Duplicati is particularly aware of mounts, but if you mean it creates the exact path of its configured destination folder, this is probably a feature for most cases on most backends, avoidable using

disable-autocreate-folder

--disable-autocreate-folder = false
If Duplicati detects that the target folder is missing, it will create it automatically. Activate this option to prevent automatic folder creation.

Assuming that’s accurate, a question would be how Duplicati gets there and sees a missing folder, and attempts to fix that (but if I understand correctly, making an actual folder interferes with system later on).
Plugging in the drive before letting Duplicati probe for an existing backup folder should stop autocreates.

For safety, you can alternatively probe in run-script-before, and use the script exit code to prevent a run:

because if the drive isn’t there, and Duplicati creates the folder, that’s bad, and if it doesn’t, that’s bad too.

What file, and where in it? I’d be surprised if export CreatedByVersion said 2.0.0.7. Other spots, less so.

When you put a big list of issues in a post, you need to match responses to issues. This response was to

The change in source size was always a little unclear because so little was said about sequence of steps.
Now I’m told that the one quote that I thought might have been a sequence was presented in reverse order.
Top to bottom is “repair and backup” “import” and “please run repair”. There are also the disclaimers about

so I’m wondering what combination of things was brought in. There’s also a probable missing line after

because the Backup statistics should also have been imported, assuming you checked the box below

image

and if you didn’t, the Source data shouldn’t be there. The metadata stats are as old as the config export. Completing a backup updates metadata about the backup. How big do you think the source should be?
Make sure you don’t have filters or links to throw things off, then you can ask your file manager, du, etc.

Your job log (fresh one after a successful backup) should give you a summary of what it sees, such as

image

If you think it’s wrong, you can see details using About → Show log → Live → Verbose, or set up options
log-file=<path> log-file-log-level=verbose to see how individual files are filtered out or opened (if so, why).

2021-06-13 10:42:02 -04 - [Verbose-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-IncludingPath]: Including path as no filters matched: C:\PortableApps\Notepad++Portable\Data\Config\backup\webpages.txt@2021-06-11_152639
2021-06-13 10:42:02 -04 - [Verbose-Duplicati.Library.Main.Operation.Backup.FilePreFilterProcess.FileEntry-CheckFileForChanges]: Checking file for changes C:\PortableApps\Notepad++Portable\Data\Config\backup\webpages.txt@2021-06-11_152639, new: False, timestamp changed: True, size changed: True, metadatachanged: True, 6/13/2021 1:47:56 PM vs 6/13/2021 1:14:20 PM

It fixes the display for me, and it might be a bug (as hinted already) that it’s needed. The chronology was too unclear (and still is) to understand exactly what was done at what time, and with what age of configs and database. Is this a new system? System moves work best with recent config and database moves.

This wasn’t said before either. The reference to 20.00 GB was on the Import where it’s an outdated value.
What does “puts it back” mean? I’m less interested in stale config metadata, but interested in the backup.
To track down details, somebody will have to look at backup and post-backup logs that I’m talking about…

The Issues template requests “Steps to reproduce”, and ideally they require little-to-no special equipment, and are precise enough that any developer (or non-developer) can easily run step-by-step to see problem.

I ask the same here, but this forum is not a bug tracker. If you can narrow this down to essentials, file with specific steps and ideally with lots of helpful info. There are far, far too few developers, and this might help.

Understanding it at that level is great. If you get that far, it would be wonderful if you could fix it, otherwise it might wind up in the queue. Duplicati progress is entirely reliant on volunteers, and there are far too few…

I suggest you first prove it (or at least look at it) with logs, and probably a specific test sequence that’s run.

That’s exactly why steps to reproduce should say at least one, otherwise fishing for the bad case is tough.

5

I believe version 2.0.0.7 is what’s reported when running the TrayIcon project from an IDE. I’m guessing you were debugging something at the time you exported that configuration. Unfortunately, this could mean that there were local code changes being tested, so like @ts678 mentioned above it would be helpful if you could outline some steps to easily reproduce the issues.

2 Likes
6

Thanks for the comment. I was actually thinking of you if it comes down to hot-plugging a drive on Linux.
I don’t have a physical Linux system to plug anything into, and VirtualBox doesn’t seem well-suited for it.
I would be happy if this could be pulled apart a little. Maybe the %HOME% issue is separate from USB?

BTW

might be where the missing folder create happens, and the plan seems reasonable (but if not, use option).

7

Its there And needs to be full paths to work correctly. That was there before your 1st post :slightly_smiling_face:

I know there’s a lot of info and it isn’t easy to understand it all. I was tearing into the problems so perhaps its not as clear as it could be. Though when I look at it, its clear.

This entire point is based off of every bit of text found with it. The original backup source was about 20GB. The export was done with a backup source of about 20GB.

The import only reaches 45MB unless I manually remove %HOME% and replace that will full paths.

If what warickmm says… there could have been a moment in time when I did the export and they had it broken at the time.

Really I don’t remember if it was from source code or from official or even which version.

So I will have to find if it can be reproduced in a backup config file where on import the source size isn’t correct. That will be the next step on that one.

Thank you to both of you :slightly_smiling_face:

8

Yep, that’s the version number I see in all the binaries that I compile myself using Visual Studio.

1 Like
9

I suppose the question is if %HOME% should include all user home files or if its meaning is something else?

This is reproducible already by having only %HOME% in the backup config file with Duplicati rpm pulled from the Duplicati’s website which is what I used with all the problems above.

The %HOME% could have slipped into the backup config incorrectly during a source build but that would mean its usage is something different than the user’s home folder and all things in it since the source size is only 45MB and full path is roughly 20GB with no other difference.

At this time with Duplicati rpm pulled from Duplicati website the following using a new backup creates %HOME% in an export backup config file and does not use the user home folder

home
home800×292 18.4 KB

So I’m confused as to what should be happening. I fully expect it to use the user home folder and include other things in it.

I try to restore from this and restore is also broken.

10

Search for 20 in your browser, from top of page going down. There is no early 20 there, which is my claim.

First reference is the “little out of date” import: One plausible cause of bad initial size is due to stale data…

Source on import: 
Source:
     20.00 GB

How are you launching Duplicati? I can get a problem with HOME if the environment variable is mis-aimed.
For example, sudo duplicati has a completely different set of backups than sudo --login duplicati which changes the HOME environment variable and goes to a different $HOME/.config/Duplicati/duplicati-server.sqlite. Although it’s not the config locator code, you can see how special folders get expanded, with

and I’m wondering, regarding the GUI expansion issue, if the hidden-folder problem is because all my root folders are hidden in the UNIX sense, at least for the ls command by having a dot in front, so ls needs -a:

# ls -1 ~root
# ls -a1 ~root
.
..
.aptitude
.bash_history
.bashrc
.cache
.config
.dbus
.lesshst
.local
.profile
.recently-used
.synaptic

I think it follows the HOME environment variable, with possible exclusions set out on the Source screen.
Just be careful about the environment variable setup. The files it looks at should be very clear even by a look at the Restore tree (no need to even look at logs). If you got a wrong size backup, see what’s in it…

11

Launching from the browser http://localhost:8200/ngax/index.html

I got that from somewhere as to how to open Duplicati from the browser. No idea where.

Is it not correct?

Looks like I got it from here possibly Using the Graphical User Interface - Duplicati 2 User's Manual

That still would be an issue however since one can get full paths that work.

12

It might be correct but it’s also not a Duplicati launch. It’s just connecting a browser to a launched Duplicati. Somehow you need to get Duplicati going, for example you might launch manually with /usr/bin/duplicati, or through systemctl start duplicati, or through telling the distro to launch duplicati at the time you login.

Here’s more info on the demo of the UID and HOME variable getting out of sync due to my bad use of sudo

# ps -ef | grep -i duplicati
root     18426 31642  0 16:39 pts/11   00:00:00 sudo duplicati
root     18427 18426  0 16:39 pts/11   00:00:00 Duplicati /usr/lib/duplicati/Duplicati.GUI.TrayIcon.exe
root     18431 18427  0 16:39 pts/11   00:00:05 /usr/bin/mono-sgen /usr/lib/duplicati/Duplicati.GUI.TrayIcon.exe
root     18626 31718  0 17:10 pts/12   00:00:00 grep --color=auto -i duplicati
# ps -p 18431 e | cat - | less

and even though the process is running as root, the e to ps dumps the environment, and HOME is still me.
I don’t know if you did anything like this, but the point is that %HOME% depends on $HOME being set right.

13

Yeah, I see starting Duplicati from terminal has that home use with all files found. However that creates http://localhost:8300 when the duplicati.service is using http://localhost:8200.

In order to open Duplicati that way I have to disable duplicati.service or there are two instances running with different backups.

How do I get into Duplicati with the duplicati.service in use and avoid this?

Really I think this should be by default working appropriately. Shouldn’t have to play around like this :wink:

It should also be good to have Duplicati be aware of the %HOME% path and use or hide if its not the user’s /home path. The way it is isn’t good. Bad experience.

14

Service got there first and took 8200. Having multiple Duplicati on a system can become confusing.

Duplicati components

The server tries to start listening on TCP port 8200. If this port is unavailable (because of another running Duplicati server instance or another process that uses port 8200), port 8300 will be tried, increasing until an unused port is found.

If by “service” you mean systemd service, then probably you’re running as root with HOME as /root, which would probably be a dramatic different set of files than the HOME that you get if started as you.

About → System info has a UserName line where you can see the user, but HOME isn’t directly seen.
Usually it matches, unless you use sudo or some such thing to change the user but not environment.

15

Yes, but how do I access Duplicati interface? Command line only when using duplicati.server?

There has to be a way to get to it from the browser and have it always running even on reboot.

I could add a .desktop file to autostart but that isn’t found in the Duplicati manual.

I install from rpm and Duplicati doesn’t automatically run on reboot does it? As far as I know that wasn’t a thing.

I know you’re typing but I think this is going into the wrong direction. The issue is %HOME% not being correct. Since full paths work with duplicati.service, fixing %HOME% by hiding it or other may be the correct answer. Instead of trying to conform how Duplicati runs to fix it. It can be obvious even in code that my system uses /home/user as the %HOME% path instead of /root.

Duplicati works fine otherwise as far as finding files when using duplicati.service and the interface works fine also. The multiple instances could be fixed as well though and maybe that’s warranted here though.

By the way, though we may bash heads at times, thanks for hanging in there :wink:

16

With Duplicati.GUI.TrayIcon, you can double-click or use right-click menu to get it to open in the browser.
I don’t use Linux much, but systemctl start duplicati seems to launch Server not TrayIcon, so I suppose one needs to know the URL. Some people may run a TrayIcon with --no-hosted-server to have something nicer to drive. I suspect TrayIcon isn’t the systemd default because not everything has a GUI
(e.g. headless servers), and even if they have a GUI, it’s not clear that a background process can use it.

There is a concern that setting up a Windows service with TrayIcon is hard. Same may be true for Linux.
Duplicati has multiple components, and things like a Windows service or systemd server can’t get to UI.
I don’t know if there’s a good way to improve all of this, but whatever it is would need someone to build it.

systemd does this part. It will run scheduled backup whether or not you open a browser to interact with it.

The Linux installation instructions leave a lot to be desired, as noted in Part of installation fails on Ubuntu, which may or may not result in any changes in the manual. Somebody has to either write, or help author.

Windows fares slightly better on the TrayIcon setup. See Configuring the Duplicati Tray Icon in Windows, which is somewhat odd because the default Windows installer sets up TrayIcon including internal server, started at login. Many people are happy with this. Linux is somewhat more do-it-yourself about setting up.

I had thought one needed to systemctl enable duplicati, or at least the manual now suggests it.

Which do you want? Please check your Duplicati environment as shown to see what $HOME it sees.
I can’t think offhand how systemd would set things up wrong. Are you sure that’s start from systemd?
systemd status duplicati should show if it started it. systemd restart duplicati to be sure…

17

Its definitely systemd using :8200 and with %HOME% not going to /home/user

Suppose Duplicati needs an environment variable set with systemd to switch home to /home/user.

Sure, some people argue Linux means you get broken things fix it yourself but there are a lot of things that you install and it works correctly and as expected. Just because its Linux doesn’t mean you should get the half working half diy stuff :smile:

I have no problem doing things but I expect an application to work correctly on install. It shouldn’t be all broken fix it yourself try to figure it out when you install it. That’s not good with any software on any OS last time I checked.

18

A possible solution for %HOME% with systemd…

sudo gedit /etc/default/duplicati
HOME=/home/user
systemctl restart duplicati.service

Home becomes /home/user. Import using %HOME% works correctly.

Other possibilities exist but that does solve that. Duplicati devs can either add a note for that, do nothing and not care, or change something with Duplicati. Its not my project so I can’t make that decision or I would.

19

OK, so what you want is to run as root but have HOME be /home/user.

I guess your workaround sets that up. There’s nothing in the export that says what HOME was at the time.

A remaining mystery to me is why I’m getting two special folders instead of none when run under systemd which sets up a very bare set of environment variables. Maybe mono gives HOME even if it’s not even set, which I suppose is possible by getting it from the /etc/passwd file. That might help out for some programs.

I’m not familiar with GUI internals, but tcpdump -i lo -A port 8200 did capture the following traffic to it:

  {
    "text": "My Documents",
    "id": "%MY_DOCUMENTS%",
    "cls": "folder",
    "iconCls": "x-tree-icon-special",
    "check": false,
    "leaf": false,
    "resolvedpath": "/root",
    "hidden": false,
    "symlink": false
  },
  {
    "text": "Home",
    "id": "%HOME%",
    "cls": "folder",
    "iconCls": "x-tree-icon-special",
    "check": false,
    "leaf": false,
    "resolvedpath": "/root",
    "hidden": false,
    "symlink": false
  },

and I’m guessing that’s what caused those two (and no others) to be shown. I could test more, but I won’t.
An easier place to get the same results is on About → System info where one can see the special folders:

SpecialFolders : [{“ID”:“%MY_DOCUMENTS%”,“Path”:“/root”},{“ID”:“%HOME%”,“Path”:“/root”}]

When started as a non-root user, I get five special folders on such a list and on the Source tree in the GUI.

There might be an equivalent issue on Windows, when Duplicati runs as SYSTEM service, but I don’t plan an investigation now because this has already taken a lot of time. At least we did learn something, I think…

20

You access the web GUI by running the command duplicati. It launches the “tray icon”, which in turn launches a service process with user’s privileges. On Windows, you get it (optionally?) running during the installation, but on Linux, you have to run the command yourself and you have to add the command manually to “Startup Application Preferences” too. Unfortunately all that is missing form the installation instructions. Apparently the assumption is that on Linux, everybody wants to run Duplicati as root. I don’t.

Anyway, even if you set up a service that runs as root, you can still use the web UI with that service; you just need to add some more options to the command, as ts678 pointed out.