Endless loop of deleting unwanted files/System.ArgumentNullException path


My current problem

After the task finished, it remains a very long time in the step “Deleting unwanted files”


First I thought it hangs, but even after seval hours there is ressource usage like CPU on the system. It also seems to delete files on OneDrive on every run:

The marked one was just created a few seconds ago, cause I was able to watch in realtime what seems to happen: Hours after staying in “Deleting unwanted files”, I got a popup that the connection was gone and the UI trys to reconnect in I guess 15 seconds (Screenshot removed for post limit)

I wondered what’s the reason for that. I thought it was a server-side restart, but the container is running since three days:

# docker-ps | grep duplicati
duplicati_duplicati_1          Up 3 days             3 days ago          linuxserver/duplicati:latest

After the waiting time, it looks like the job is started again: I see_Backup will be started_ and the Nextcloud data taskname in the header (screenshot removed for post limit)

Then it verifies the backend data, processes the files on the server and see that there are no (or just a few small) changes, so this process is finished after just a few minutes. But now it stucks again in Deleting unwanted files.

This seems to happen in an endless loop, I looked from time to time on that backup and it’s the second time now I could watch the described disconnect-process with restart in realtime. According to the OneDrive recycle bin screenshot above, just today since 0AM there were 23 files deleted. Since everyone except the first one at 02:01 AM are 12,9MB in size and have a newer timestamp in the filename, I guess its the same file which he first uploads and later deletes.

I noticed that the Duplicati container logs shows a exception very often:

[ERROR] FATAL UNHANDLED EXCEPTION: System.ArgumentNullException: Value cannot be null.
Parameter name: path
  at System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share, System.Int32 bufferSize, System.Boolean anonymous, System.IO.FileOptions options) [0x00014] in <254335e8c4aa42e3923a8ba0d5ce8650>:0
  at System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share) [0x00000] in <254335e8c4aa42e3923a8ba0d5ce8650>:0
  at (wrapper remoting-invoke-with-check) System.IO.FileStream..ctor(string,System.IO.FileMode,System.IO.FileAccess,System.IO.FileShare)
  at Duplicati.Library.Main.Volumes.VolumeReaderBase.LoadCompressor (System.String compressor, System.String file, Duplicati.Library.Main.Options options, System.IO.Stream& stream) [0x00000] in <8f1de655bd1240739a78684d845cecc8>:0
  at Duplicati.Library.Main.Volumes.VolumeReaderBase..ctor (System.String compressor, System.String file, Duplicati.Library.Main.Options options) [0x00007] in <8f1de655bd1240739a78684d845cecc8>:0
  at Duplicati.Library.Main.Volumes.IndexVolumeReader..ctor (System.String compressor, System.String file, Duplicati.Library.Main.Options options, System.Int64 hashsize) [0x00000] in <8f1de655bd1240739a78684d845cecc8>:0
  at Duplicati.Library.Main.BackendManager.RenameFileAfterError (Duplicati.Library.Main.BackendManager+FileEntryItem item) [0x0018a] in <8f1de655bd1240739a78684d845cecc8>:0
  at Duplicati.Library.Main.BackendManager.ThreadRun () [0x002de] in <8f1de655bd1240739a78684d845cecc8>:0
  at System.Threading.ThreadHelper.ThreadStart_Context (System.Object state) [0x00014] in <254335e8c4aa42e3923a8ba0d5ce8650>:0
  at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00071] in <254335e8c4aa42e3923a8ba0d5ce8650>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00000] in <254335e8c4aa42e3923a8ba0d5ce8650>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state) [0x0002b] in <254335e8c4aa42e3923a8ba0d5ce8650>:0
  at System.Threading.ThreadHelper.ThreadStart () [0x00008] in <254335e8c4aa42e3923a8ba0d5ce8650>:0

It was much more often in the logs, I just put out a few for better readability and to fit the thread limit here. It seems to be always the same, and it appears after the crash. So I guess the following happens:

During cleanup, something went wrong which cause the System.ArgumentNullException. Since it’s unhandled, the application inside the Docker container is re-started quickly. It seems so quick that Docker doesn’t notice it and things the container is still running. This causes the connection abort in the UI. Since Duplicati notices that it was exited before the Nextcloud data task was finished, it re-starts the task and everything starts from the beginning.

My setup

I have a selfhosted Nextcloud instance with currently 632GB of data in the filesystem. My account is the biggest with 423G, the other space is spread across some other users (family/friends). It’s running on Docker and I’d like to use Duplicat to backup the data directory securely encrypted to OneDrive personal. I also have some other things to backup in several tasks, from 8GB to 40GB. These smaller ones works well, but the Nextcloud data task never worked without issues yet.

docker-compose.yml from Duplicati

version: '2'
    image: linuxserver/duplicati
    mem_limit: 4G
      - duplicati-config:/config
      - /storage/duplicati-temp:/backups
      - /storage/duplicati-temp/tmp:/tmp

      - /storage:/source:ro
      - /etc/docker/container:/src/docker-container:ro
      - /vms/docker-lib/volumes/:/src/docker-volumes:ro
      - my-net
      #- PUID=1000
      #- PGID=1000
      - TZ=Europe/Berlin
      - PUID=0
      - PGID=0

      - "traefik.port=8200"
      - "traefik.frontend.rule=Host:my.domain"
      - "traefik.frontend.entryPoints=http"

    external: true


I made sure to pull the latest version avaliable.

Exported job config

  "CreatedByVersion": "",
  "Schedule": {
    "ID": 3,
    "Tags": [
    "Time": "2021-03-02T03:30:00Z",
    "Repeat": "1D",
    "LastRun": "2021-03-01T10:50:20Z",
    "Rule": "AllowedWeekDays=Monday,Tuesday,Wednesday,Thursday,Friday,Saturday,Sunday",
    "AllowedDays": [
  "Backup": {
    "ID": "3",
    "Name": "Nextcloud-Daten",
    "Description": "",
    "Tags": [],
    "TargetURL": "onedrivev2://backups/nas-duplicati/nextcloud-data/",
    "DBPath": "/config/XLGOTEDXNE.sqlite",
    "Sources": [
    "Settings": [
        "Filter": "",
        "Name": "encryption-module",
        "Value": "aes",
        "Argument": null
        "Filter": "",
        "Name": "compression-module",
        "Value": "zip",
        "Argument": null
        "Filter": "",
        "Name": "dblock-size",
        "Value": "2GB",
        "Argument": null
        "Filter": "",
        "Name": "retention-policy",
        "Value": "1W:1D,4W:1W,12M:1M",
        "Argument": null
    "Filters": [],
    "Metadata": {
      "LastErrorDate": "20210301T105043Z",
      "LastErrorMessage": "Found 1 files that are missing from the remote storage, please run repair",
      "LastBackupDate": "20210301T105130Z",
      "BackupListCount": "4",
      "TotalQuotaSpace": "1104880336896",
      "FreeQuotaSpace": "119831723888",
      "AssignedQuotaSpace": "-1",
      "TargetFilesSize": "874866981263",
      "TargetFilesCount": "891",
      "TargetSizeString": "814,78 GB",
      "SourceFilesSize": "672284328220",
      "SourceFilesCount": "138763",
      "SourceSizeString": "626,11 GB",
      "LastBackupStarted": "20210301T105130Z",
      "LastBackupFinished": "20210301T110756Z",
      "LastBackupDuration": "00:16:26.0346620",
      "LastCompactDuration": "00:00:53.9963860",
      "LastCompactStarted": "20210101T043856Z",
      "LastCompactFinished": "20210101T043950Z"
    "IsTemporary": false
  "DisplayNames": {
    "/source/nextcloud_data-html/": "nextcloud_data-html"

As you can see, I changed the blocksize to 2GB. I thought this make sense, because I have a large amount of data and a relatively fast upload speed (40Mbit/s). And I guess that too much files could potentially cause problems. I’ll now try to reduce it to 1GB, which is recommended in Choosing Sizes in Duplicati • Duplicati

Another noticeble thing is, that my ISP do a reconnect every 24 hours. By the large amount of data, I think that Duplicati will be interupted at least one time during a full backup. I don’t know if that’s a problem, I’d expect that Duplicati detect this since it’s not a uncommon scanario, and continue the download, or if this is not possible re-start the volume upload.


My first problem after the first run was, that there seems to be some inconsistences in the avaliabla data vs the expected:

Finally I purged the broken files and this leads me to the situation where I’m currently: The backup itself seems to work, but it takes long on “Deleting unwanted files” and then starts an endless loop.

Where is the problem?

I’ll now try to reduce the volume size to 1GB and maybe even to 512MB if 1GB doesn’t help, maybe many Duplicati users have much smaller sizes and I ran into issues by choosing larger ones. Maybe I also have to delete the existing server backup and re-upload everything.

I made this thread for ideas how to fix it, and on the other side to get feedback if I’m doing something wrong. Since I had a lot of problems with the Nextcloud data backup task, I think more and more that I’m doing something wrong.
Or is Duplicati not designed for such a large amount of data?
Is increasing the block size to better use my connection speed maybe not as good as it sounds from my understanding?

I set the volume size to 1GB, then I got

2021-03-02 22:33:54 +01 - [Error-Duplicati.Library.Main.Operation.TestHandler-FailedToProcessFile]: Failed to process file duplicati-b2b2a1d2ec5aa4ae5a3e67edef675fb37.dblock.zip.aes

So I reduced it to 512MB, same result. A file test worked. After I ran repair, it successfully worked:


I hope that the backup is consistent now and it will run from now without errors.

To take a guess (lacking logs now), http-operation-timeout of 100 seconds for OneDrive might not be enough for the large remote volumes even on your connection, and you might have needed to raise it.

If you want to set up a test backup, watch About → Show log → Live → Retry and see how that looks. Connection speed also doesn’t equal transfer speed. There can be other limitations, e.g. at the server.

“Deleting unwanted files” is based on the Backup retention setting that you set in Options screen 5. Compacting files at the backend may run if version deletions obsoleted enough of the backup contents. Watching the logs, you can see the downloads of the overly empty dblock files, and uploads of the new.

Looking at files, dlist files represent versions, and deletion should be predictable per the rules you set, and this too can be watched in the log. Retry level should be enough to get the highlights of the process.

Compacting is less predictable because it depends so much on the data, but you can see this in log too.

If it’s not out for long, the upload or download retries should ride through it. If need be, adjust number-of-retries and retry-delay. For every backup that completes, its Complete log shows the RetryAttempts

To be pedantic (but for a reason), you’re increasing dblock-size but blocksize remains at default 100 KB which is kind of small for that amount of backup. The database gets slow tracking all of the small blocks.

You might want to bump it to 1 MB or something if you ever recreate the backup. Can’t change it midway. This should make any future disaster recovery or database Recreate go faster. It’s good to testing these.