Duplicati keeps storing files in preservation hold library of OneDrive

Hi,
I seem to have an issue backing up to OneDrivev2 target. When testing the connection in the duplicati config then it will correctly connect and also create directories if the given path does not exist. But actually running the backup I found that the duplicati backup files have not been updated for a couple of months. Hunting around I finally found where duplicati is now storing its files. It all ends up in the “Preservation Hold Library” instead of the user files. Is there some setting missing that will tell the OneDrive API to place the duplicati files in the correct location instead of just dumping them in the Peservation Hold Library?

It is a OneDrive Business account I’m connecting to via the v2 API

It’d be nice if somebody who has such a setup helped out, but from what I can see, the preservation hold library is where deleted files go, provided whoever administers retention policy has set that up to happen.

Learn about retention policies for SharePoint and OneDrive

The easiest way to test if that’s what you’re seeing is when a duplicati*dlist* file shows up. It contains the backup’s date and time in UTC. Does your Options screen 5 Backup retention allow backup deletes?

I’m finding no evidence that it’s even possible to store directly to the preservation hold library, but if some backup manages to do it, the dlist file should have the date and time in UTC of when that backup started.

The Duplicati home screen also has your backup size and number of versions. Hopefully it’s reasonable, however I’m not sure why you’re not seeing files as expected. Have you done browser refresh or switch?

Job’s Show log --> Remote at start and end of backup should show a list operation. Click it to see what Duplicati heard is up there (the whole set of files) based on the directory listing from the remote OneDrive.

Hi,
I’ll try to answer the questions as best as I can.
Backup retention is set to smart. Not sure what you mean with if the options allow deletion. I assume (I know first mistake always) it does since it works on all my other ssh-fs targets.

So I just ran another backup which finishes successfully against the OneDrive traget after just about 2 mins. Given that the source data is about 150 GB and I’m triggering it behind an ADSL 1 Mbit upload connection I doubt that any backup could have been done.

Before triggering the backup the target location contains these dlist files:

duplicati-20200219T131235Z.dlist.zip.aes
duplicati-20200401T065153Z.dlist.zip.aes
duplicati-20200401T071539Z.dlist.zip.aes
duplicati-20200401T094922Z.dlist.zip.aes
duplicati-20200401T213006Z.dlist.zip.aes

and the Preservation Hold Library contains these duplicati files:

duplicati-b4832419b07944b38ba6ac45ffc70a491.dblock.zip_767EF7E2-01B5-4ED1-A5FA-C2840ECC29A62020-06-12T01-15-15.aes
duplicati-b6f93936b1559413e9bb460f99ccf23d8.dblock.zip_3E31A6DE-835B-4587-A233-4CF931F44CA12020-06-12T00-37-01.aes
duplicati-b8e1660ad190c4cbcbd21ec016f4875fe.dblock.zip_007D4D9F-91D4-445E-B1FD-BEEAA2FD37382020-06-11T05-52-45.aes
duplicati-i5ae782defbb740e2872e7547fa957fa3.dindex.zip_C91455A3-0544-434B-A83C-A134642155452020-06-12T00-37-01.aes
duplicati-id95e3e45972447589040ff0788b07fc2.dindex.zip_2EA27E09-A0E9-42AF-8B36-2FB968B1C0BD2020-06-12T01-15-18.aes
duplicati-ieb4c46be5dea4ec1aa98ed4242dd56c2.dindex.zip_50FA593F-07A8-4DF2-8AFE-AC1FDDDDDB9E2020-06-11T05-52-45.aes

The date pointing to my test runs from last week.
After Duplicati rant its backup lightning fast in 2 mins and claiming success the target location does not contain any updated dlist file but the Preservation Hold Library now contains two new file in addition to the one listed above:

duplicati-bf6394f8a50ce41a0924cc85e1f1f3363.dblock.zip_422A6973-96C1-4693-9DC0-87869685592E2020-06-16T00-41-35.aes
duplicati-icf07abf59032459e89dbc8237d36f1fb.dindex.zip_75A196E1-9731-45E6-91A4-DA8D19D30CE62020-06-16T00-41-36.aes

I’m attaching the Duplicati log file for that run in hope that it can give some additional pointers.
Duplicati.zip (2.0 KB)

More or less, anything other than “Keep all backups”. Your setting deletes backups as their age grows.
File deletion causes OneDrive to move the file to preservation hold library, and insert that time in Name.
.

The lack of timestamps from Duplicati is why I was asking to see a dlist file, with Duplicati’s timestamp.
There is usually no dlist file and no backup unless some source changes, but Duplicati sees no source:

  "DeletedFiles": 0,
  "DeletedFolders": 0,
  "ModifiedFiles": 0,
  "ExaminedFiles": 0,
  "OpenedFiles": 0,
  "AddedFiles": 0,

Below is a job log from a tiny test backup with no content changes, but at least it’s examining a few files:

image

The reason I have a backup at all (despite my general claim earlier about backups requiring changes) is:

which you can try setting to see if you can get at least a dlist where it should be. But where’s the source?

As for why files are storing into the preservation hold library, the log you posted shows these files deleted:

  "FilesDeleted": 2,

and the reason they were deleted is shown as:

    "2020-06-12 10:15:37 +02 - [Information-Duplicati.Library.Main.Database.LocalDeleteDatabase-CompactReason]: Compacting because there are 1 fully deletable volume(s)",
    "2020-06-12 10:15:37 +02 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Started: duplicati-b4832419b07944b38ba6ac45ffc70a491.dblock.zip.aes (845 bytes)",
    "2020-06-12 10:15:38 +02 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Completed: duplicati-b4832419b07944b38ba6ac45ffc70a491.dblock.zip.aes (845 bytes)",
    "2020-06-12 10:15:38 +02 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Started: duplicati-id95e3e45972447589040ff0788b07fc2.dindex.zip.aes (893 bytes)",
    "2020-06-12 10:15:38 +02 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Completed: duplicati-id95e3e45972447589040ff0788b07fc2.dindex.zip.aes (893 bytes)",
    "2020-06-12 10:15:39 +02 - [Information-Duplicati.Library.Main.Operation.CompactHandler-CompactResults]: Deleted 2 files, which reduced storage by 1.70 KB",

Let’s look for them in the preservation hold library listing:

duplicati-b4832419b07944b38ba6ac45ffc70a491.dblock.zip_767EF7E2-01B5-4ED1-A5FA-C2840ECC29A62020-06-12T01-15-15.aes
duplicati-id95e3e45972447589040ff0788b07fc2.dindex.zip_2EA27E09-A0E9-42AF-8B36-2FB968B1C0BD2020-06-12T01-15-18.aes

Time stamp in the name underwent major hours adjustment (due to time zone?) though minute matches.

The time stamp seems very poorly documented (that I can see), but below is one article that describes it:

How to set a Retention Policy on a SharePoint site

Retention in Office 365. The new way. gave a few points on the feature. There’s also the site I cited earlier.

No real backup because no source files found. Make sure your setting is right, you didn’t exclude it all, etc.
About → Show log → Live → Verbose or –log-file –log-file-log-level=verbose will show source scanning.

EDIT:

Your two latest additions to the preservation hold library weren’t commented on because the log given was from June 12. You can do the same check in the relevant log, or see delete using Information level logging.

@ts678 Just like to let you know that I appreciate your detailed feedback. I will look into it as soon as I can but will probably now take a couple of weeks for me. Still it was definitely not in vane as I will want to get my OneDrive backup behaving again.

@ts678 So finally after a looong time I started looking into it closer.

TL;DR: Make sure to not have “*” in your exclusion filters :man_facepalming:

I also tried to use the SharePoint backend as an alternative which created the same result of backup not doing anything. Then the final straw was using rclone to mount OneDrive locally and running duplicati against that one. The result was the same. Backup finished quickly without the transfer of any files. When looking through all the advanced settings I also unfolded the exclusion list and there was one entry with "*". I have no idea who this ended up there since I’ve always only used the select with click method to exclude files to do backup on. Anyway, it now finally does the backup as it should so this issue is not an issue at all and sorry for wasting precious time.

1 Like