- [x] I have searched open and closed issues for duplicates.
----------------…------------------------
## Environment info
- **Duplicati version**: 2.0.4.34 canary
- **Operating system**: Windows 10 Version 1903
- **Backend**: Local folder
## Description
Various issues around partial backup. Rather than file separate issues now, this is a collection of issues.
["Stop after current file" with --usn-policy may never back up remaining files #3971](https://github.com/duplicati/duplicati/issues/3971) was separately filed, and possibly has a different cause than the ones here which are primarily on how backups are now split into full and partial, with record-keeping in the database and backup, and attempts to make things like retention work right by hiding partial backups after a full exists. This breaks things because some code is going by the view-after-hiding, some is going with the raw view, and translating between may break.
[Fix 'stop after current file' #3836](https://github.com/duplicati/duplicati/pull/3836#issuecomment-546687741) has technical info. This issue is more of a hopefully-reproducible tour.
* Partial backup produces a Warning result without indication of why. Unknown if this is intentional.
* Partial backup Repair regenerates missing dlist file without a `fileset` file to show "partial" status.
* Partial backup marking is on all Direct restore versions, whether that version is really full or partial.
* Hidden partial backups hold backend storage forever.
* Hidden partial backups cannot be referenced by user.
* Hidden partial backups might not be reachable by Duplicati, unless direct queries to DB are used.
* Hidden partial backups break Duplicati when direct DB queries are translated to version for users.
One is probably in the [number translator](https://github.com/duplicati/duplicati/commit/2826dbe6f04497bcfdb88a9da96ea7e65338f055) for "Unexpected difference in fileset" errors (not tested).
* There's no telling how much other code is now (or may someday be) broken due to the dual map. Having only one way to map Fileset ID to its version (and vice versa) just didn't have this problem.
## Steps to reproduce
1. Make a C:\stop test folder holding two Duplicati executables zip files. Mine are about 14 MB each.
2. Slow with Remote Volume Size at 1 MB, --throttle-download and --throttle-upload at 100 KByte/s
3. Set Backup retention to keep 1 backup.
4. To spot when to Stop, and see other info, open a tab to About --> Show log --> Live --> Verbose
5. "Run now" Backup, go to live log tab, wait for uploading to begin, then do "Stop after current file"
6. "Run now" Backup. There will be a third backup later, but partial-then-full is a good starting point.
* Look at the job log icon and full log for the first backup which was the partial.
- **Actual result**:
Icon is a yellow exclamation point (Warning I think), and log has one `"ParsedResult": "Warning",`
- **Expected result**:
Not sure which I prefer, but if it's going to do a Warning, there should be some way to know why.
* Test Commandline `purge` with arguments set to some file -- it doesn't seem to make a difference.
- **Actual result**:
```
Listing remote folder ...
System.InvalidProgramException: Fileset was reported with id 1, but could not be found?
at Duplicati.Library.Main.Operation.PurgeFilesHandler.DoRun(LocalPurgeDatabase db, IFilter filter, Action`3 filtercommand, Single pgoffset, Single pgspan)
at Duplicati.Library.Main.Operation.PurgeFilesHandler.Run(IFilter filter)
at Duplicati.Library.Main.Controller.RunAction[T](T result, String[]& paths, IFilter& filter, Action`1 method)
at Duplicati.Library.Main.Controller.PurgeFiles(IFilter filter)
at Duplicati.CommandLine.Commands.PurgeFiles(TextWriter outwriter, Action`1 setup, List`1 args, Dictionary`2 options, IFilter filter)
at Duplicati.CommandLine.Program.ParseCommandLine(TextWriter outwriter, Action`1 setup, Boolean& verboseErrors, String[] args)
at Duplicati.CommandLine.Program.RunCommandLine(TextWriter outwriter, TextWriter errwriter, Action`1 setup, String[] args)
Return code: 100
```
- **Expected result**:
Not certain, but the id 1 would be the first partial backup which is now hidden and not user-visible.
Keeping an eye on the DB (especially the Fileset table) using an SQLite DB browser could be useful.
* Test Commandline `affected` with arguments set to file name of oldest dblock file, i.e. from partial.
- **Actual result**:
```
System.Collections.Generic.KeyNotFoundException: The given key was not present in the dictionary.
at System.ThrowHelper.ThrowKeyNotFoundException()
at System.Collections.Generic.Dictionary`2.get_Item(TKey key)
at Duplicati.Library.Main.Database.LocalListAffectedDatabase.<GetFilesets>d__5.MoveNext()
at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
at System.Linq.OrderedEnumerable`1.<GetEnumerator>d__1.MoveNext()
at System.Linq.Enumerable.Any[TSource](IEnumerable`1 source)
at Duplicati.CommandLine.Commands.<>c__DisplayClass3_0.<Affected>b__0(IListAffectedResults res)
at Duplicati.Library.Main.Operation.ListAffected.Run(List`1 args, Action`1 callback)
at Duplicati.Library.Main.Controller.RunAction[T](T result, String[]& paths, IFilter& filter, Action`1 method)
at Duplicati.Library.Main.Controller.ListAffected(List`1 args, Action`1 callback)
at Duplicati.CommandLine.Commands.Affected(TextWriter outwriter, Action`1 setup, List`1 args, Dictionary`2 options, IFilter filter)
at Duplicati.CommandLine.Program.ParseCommandLine(TextWriter outwriter, Action`1 setup, Boolean& verboseErrors, String[] args)
at Duplicati.CommandLine.Program.RunCommandLine(TextWriter outwriter, TextWriter errwriter, Action`1 setup, String[] args)
Return code: 100
```
- **Expected result**:
Proper report.
* Test Commandline `delete` with arguments --version=1
- **Actual result**:
```
Listing remote folder ...
No filesets matched the criteria
Return code: 0
```
- **Expected result**:
Maybe actual was expected. --version=1 would be initial partial which exists but you can't get to it.
* Test Commandline `find` with arguments * and --all-versions to see if "all versions" does just that.
- **Actual result**:
```
Listing contents 0 (11/9/2019 7:05:30 PM):
C:\stop test\
C:\stop test\duplicati-2.0.4.12_canary_2019-01-16.zip (14.44 MB)
C:\stop test\duplicati-2.0.4.13_canary_2019-01-29.zip (14.44 MB)
Return code: 0
```
- **Expected result**:
Per [the current manual](https://duplicati.readthedocs.io/en/latest/04-using-duplicati-from-the-command-line/#the-find-command), "Searches in all backup sets, instead of just searching the latest." No longer.
* Do the third backup after replacing the two zip files with a short .txt file. The one I used was 1 byte.
- **Actual result**:
```
Source:1 bytes
Backup:14.49 MB / 1 Version
```
- **Expected result**:
It should have released the space from the first partial, but that still remains and is hanging onto it.
The Duplicati user can't get to these to delete them, and it appears Duplicati retention can't either.
As additional backups happen and replace old blocks, these hidden partials become a storage leak.
* Look at things in a start down Direct restore from backup folder, to see what versions it shows you.
- **Actual result**:
Version 0 is the current full with 1 byte file, but is mis-called partial. Version 1 is the original partial.
- **Expected result**:
Correct label of partial. Direct restore says that everything is partial, whether it actually is, or is not.
* Look inside both dlist zip files to see `fileset` file is there, delete dlists, then run a Database Repair.
- **Actual result**:
dlist files are regenerated without `fileset` file.
- **Expected result**:
dlist files are regenerated with correct `fileset` file with correct `IsFullBackup` variable in its JSON.
* Run Database `Recreate (delete and Repair)`, then look to see what `Restore files` will offer you.
- **Actual result**:
Version 0 is the 1 byte file full backup. Version 1 is the original partial. Neither one is marked partial.
- **Expected result**:
This test used the bad regenerated dlist files. Good ones work OK. Unhiding partials may be useful, should one desperately need to get to one, with no other way (but it's sure a strange workaround).
## Screenshots
## Debug log
