When I run a backup from gui everything looks fine, zero new files. If I run the same backup from a script with Duplicati.CommandLine it finds 8000 new files. That is about a quarter of all files.
Is that with a current Export as Command-line? If not, you can export and look for differences.
User might also matter. For GUI, the user used would be in About → System info → UserName.
How many times can it do that? If it does it repeatedly on different runs, then this gets weirder…
The GUI seems to have a bit more in stats than CommandLine does. If they share DB, compare Source files stats in the job log between a GUI run and a CommandLine run. Any difference?
Any changed or deleted files? If not, you might not even have a new Restore version made for it.
8000 new files should be doing some uploads, which should be visible in the job Complete log.
I did compare the parameters as suggested. They are identical and they share same database.
I stopped using the command line but it happens after every single gui backup.
It should never be changes (8000 files) immediately after a gui backup.
This sounds slightly different than the original.
Are you now saying GUI after GUI gets 8000?
Does immediately mean it’s only immediately?
Or are you saying GUI after GUI is fine, but CLI after GUI isn’t?
If you want any comments on why, please check logs as asked.
That is the case. Just to add that only first cli after gui identifies 8000 changes.
This now says "8000 changes. The original post said “8000 new files”. There’s a difference.
CLI displays stats like:
Files added: 0
Files deleted: 0
Files changed: 1
GUI stats for the same run (visible because job database is shared) are:
If it’s not really a new files problem but a changed files one, forum topic below sounds similar, however I don’t think it was a GUI versus CLI thing. It was also Docker on NAS. What is this?
Duration of backup: 00:22:19.9983977
Remote files: 766
Remote size: 31.406 GB
Total remote quota: 931.512 GB
Available remote quota: 182.352 GB
Files added: 8177
Files deleted: 0
Files changed: 0
Data uploaded: 0 bytes
Data downloaded: 0 bytes
Backup completed successfully!
It supposed to take 2min and add no files.
I guess this is you also filing a GitHub issue – and pointing to another involving changed files.
I’m not positive that the GUI and CLI reports are made the same way, but I would expect that.
Data uploaded: 0 bytes seems wrong unless new files are false alarms, as changed were.
You can see log lines in the cited issue showing verbose-level statements on files of interest.
Those were coming out with new: False but metadatachanged: True. Maybe yours differ?
Unfortunately getting log needs CLI-after-GUI run, e.g. with --console-log-level=verbose.
Maybe the developer will have some other ideas while you consider whether or not to do that.
Seemingly it just wastes 20 minutes, but does no harm beyond that.
that is because of --dry-run
Seemingly it just wastes 20 minutes, but does no harm beyond that.
It’s not only that. I no longer have confidence that the cli backup works.
You can check the job log of an actual CLI-after-GUI run, to see if it uploaded anything.
They should be pretty easy to identify, assuming that GUI stats show issue of CLI stats.
Complete log will have a section like example below:
"BackendStatistics": {
"RemoteCalls": 23,
"BytesUploaded": 113793776,
"BytesDownloaded": 53430983,
"FilesUploaded": 8,
Has CLI backup done anything that impacted the GUI backup done after odd CLI result?
Has either GUI or CLI backup actually had to backup new or changed files with 2.1.0.4?
Although it might obscure an upgrade problem, note that another problem was solved by
What was running before, and on what platform? The mono precision bugs were in 2019.
was recently done, and fixes at least one case that went wrong. I don’t know if it’s this case…
If it turns out that CLI actually did something, such as create a new version or do upload, then Create bug report and sharing it via a posted link might shed some clues without a CLI run.
If nothing at all resulted (i.e. no new version made), then the database will have less to look at.
Again, the developer might have some other ideas.
with log verbose I can see thousands of entries like this:
Checking file for changes E:\Data\Invest\data\tmp\fws-553361_20240222070132.xml, new: True, timestamp changed: True, size changed: True, metadatachanged: True, 2024-02-21 12:25:19 PM vs 0001-01-01 12:00:00 AM
or like this
Checking file for changes E:\Users\****\Documents\Samsung\SmartSwitch\backup\SM-A505W\SM-A505W_18076321955\SM-A505W_20240421154354\APKFILE\atws.app.png, new: True, timestamp changed: True, size changed: True, metadatachanged: True, 2024-04-21 7:46:50 PM vs 0001-01-01 12:00:00 AM
These are actually excluded from gui and cli with
--exclude="E:\Data\Invest\data\tmp\\"
--exclude="E:\Users\****\Documents\Samsung\\"'
Importantly, the filters are processed in the order they are supplied
so simply using an exclude might not be the total story to the filtering.
You can take your entire filter option set and try it in The TEST-FILTERS command if you like. Filters frequently are confusing, but I haven’t heard anyone (yet) show they broke in 2.1.0.4…
So lack of filtering might be due to config, but those “Checking file for changes” sure look odd.
Your filters look right to me for CLI. GUI filter would be very different (view in three-dot menu):
-C:\tmp\ is an example. CLI has --exclude= for -, double quotes, doubles last backslash.
Maybe you can post your entire set of filters (redact lightly, preferably leaving all punctuation)?
Or use test-filters, maybe saving output to a text file if it gets too big for scrolling terminal.
I am using same script with same filters for about 8 years, always worked until this upgrade.
The issue seems to be filters related. One thing I noticed is that --exclude in parameter file, definitely does not work. Identical --exclude in the command line seems to work. And this is the exclude I get from gui export > As Command-line.
BTW, using parameters in a parameter file seems to be hit and miss. Many of the params do not work and there is absolutely zero doc about what can be included and what not. Also, there is no way to actually see which params are actually used / loaded from file.
The help says:
--parameters-file (Path): Path to a file with parameters
Use this option to store some or all of the options given to the commandline client. The file must be a plain text
file, and UTF-8 encoding is preferred. Each line in the file should be of the format --option=value. Use the
special options --source and --target to override the localpath and the remote destination uri, respectively. The
options in this file take precedence over the options provided on the commandline. You cannot specify filters in
both the file and on the commandline. Instead, you can use the special --replace-filter, --append-filter, or
--prepend-filter options to specify filters inside the parameter file. Each filter must be prefixed with either a
+ or a -, and multiple filters must be joined with ;.
* aliases: --parameter-file, --parameterfile
This is on Windows. On other systems, the join between filters is probably colon, not semicolon.
When did parameters-file enter the picture here? I don’t think GUI even supports it (just CLI).
EDIT 1:
Do you have any feel for if the “Strange behavior after upgrade to 2.1.0.4” is an upgrade issue or (simpler to chase) a 2.1.0.4 issue that doesn’t require starting on a prior version then upgrading?
Simple test would be – can you make a very plain small new backup that shows same problem? Basically, in terms of troubleshooting, exact complete steps to repro issues for the devs can help.
EDIT 2:
As opposed to, say, saying “hit or miss”. Something specific with few redactions is always better.
There is an open issue I can quickly find on parameters-file. You can see what you think about it.
The help does not say that --exclude should not be in the parameters file. Also tried --replace-filter with colon and you can imagine the result when the some of the filters start with e:\ .
Yes. It appears it just says don’t do it if you also did it on the command line.
This is Windows, right? The help says use a semicolon, which is the path separator on Windows.
Has parameters-file been part of this from the start? It once was an Export As Command-line.
I don’t have filter on cmd line and (being on windows) I only tried colon after your suggestion.
Just to be clear. Neither --replace-filter nor --exclude work in parameter files without any other filters in cmd line.
There was no such suggestion for Windows, only explanation of how non-Windows would go:
EDIT 1:
I tested Linux. It said what I thought it would (use colon). Some of the help text is context-sensitive.
--parameters-file (Path): Path to a file with parameters
Use this option to store some or all of the options given to the
commandline client. The file must be a plain text file, and UTF-8
encoding is preferred. Each line in the file should be of the format
--option=value. Use the special options --source and --target to override
the localpath and the remote destination uri, respectively. The options
in this file take precedence over the options provided on the
commandline. You cannot specify filters in both the file and on the
commandline. Instead, you can use the special --replace-filter,
--append-filter, or --prepend-filter options to specify filters inside
the parameter file. Each filter must be prefixed with either a + or a -,
and multiple filters must be joined with :.
* aliases: --parameter-file, --parameterfile
anyway, neither --replace-filter (colon or semi) nor --exclude work in parameter files with no any other filters in cmd line.
OK, that’s simple enough that I can probably test it, but did you look through the existing issue?
I’m still interested in knowing if parameters-file somehow is in original report, but I’ll divert…