New to duplicati, slow backup and low ram/cpu usage, and size quota

Support
#1

Hi all, so I was originally a crashplan user, I then gave ARQ backup a try, however it had issues with completely filling my target backup drive, and the same happens with Duplicacy, so I’m giving you guys a shot, as syncthing is running great, but has limitations in terms of versioning being full file copies. I love that it monitors the file system like crashplan did, and I’m sad that this doesn’t but hopefully it makes up for it in scan speed compared to ARQ and Duplicacy.

Now, my problem is it’s taking quite a while to process files. I have two Xeon X5670’s, not blazingly fast, but plenty of threads for Duplicati to run, as well as 72gb of ram, with 60% of that currently in use. I have the AES file size set to 1GB

Duplicati however is only using at most so far, 14% of my cpu, usually sitting at 5%, and less than 200mb of ram.

It seems oddly low to me, and maybe this will change as it consistently reaches larger files, but I was wondering if there are options somewhere that I need to configure to allow it to really stretch it’s legs and take as much as it wants in terms of system resources.

Now, I am running on a hard drive that’s currently stuck at SATA II so that won’t help I’m sure, but I feel like it could be doing more. I read that multithreading was added around a year ago, maybe that’s more in use with simultaneous backup operations? for now I’m just doing one to see how the size compares with Duplicati’s de-duplication and compression, and to test adopting a backup that was once a mapped network drive and will later be a remote STFP server.

I am also concerned about Duplicati eventually filling my drive completely and being unable to delete old backups, I want to keep 50-100gb free, however I don’t know if setting the remote storage quota to 5400gb on each individual backup will mean that they all think they have that much space available, or if it will be a cumulative backup size, and it will be able to manage old backups and keep rolling out newer ones with the smart backup setting.

Edit: Another question, does de-duplication take place globally? I’m setting up a backup for multiple drives individually, and so I’m not sure if it would apply to them all or only in each individual backup. There shouldn’t be many duplicate files, but I’d like to dodge potential large ones.

#2

Welcome to the forum!

Looks like those are 6 core/12 thread CPUs. Check out some of the --concurrency options. Maybe increasing them can help improve throughput. I personally don’t have experience with adjusting them so can’t really comment.

The default is 4 threads for block hashers and compressors… if you have 24 threads total between your two CPUs, then 14% sounds about right.

Also, where are you writing the backup data to? If it’s cloud storage, your upload bandwidth may be a bottleneck. But it sounds like you may be backing it up to a local disk.

If you did fill up your backup drive, I don’t see why that would stop you from being able to delete old backup versions.

I have never messed around with the quota options, but according to the description --quota-size isn’t used if the backend reports its size to Duplicati directly. --quota-warning-threshold should work and the default is 10%. If you set that in your global options it will apply to all jobs.

No, it is per backup job. Each backup job must write to a unique location. Don’t set up multiple backup jobs to write to the exact same folder in your backup location or you’ll corrupt your backups.

For better deduplication you may want to consider having fewer backup jobs. On most of my PCs I only have a single backup job defined. On some other machines I do set up multiple backup jobs but it’s usually only because I want different retention or a different deduplication chunk size.

#3

I’m writing my backup to a shared network drive right now, so uploads are over a gigabit connection, however later on it will be sftp over the internet.

I’m not sure why the drive being completely full stopped the other programs from being able to delete files, didn’t make sense to me either but when I deleted old files manually they were able to perform prune operations properly.

I don’t believe that the remote size is reported by the SFTP software I have but maybe duplicati supports that sort of thing better.

Shame that the de-duplication is per backup job, though that’s a minimal issue. Also, I won’t be doing multiple backup jobs to the same remote folder, I saw how it was uploading files and so I’ll make a folder for each drive. Maybe doing everything as one backup would be a good idea but I feel like it might be slightly harder to manage, though this software does do a nice job of loading the file tree quickly, even if I don’t like the default viewing order being alphabetical showing files above folders.

Also, holy crap, I don’t know if this is actually correct, I don’t have loads of text documents or anything, but my first hard drive backup is complete, and while syncthing made a 1:1 image that was 645gb, duplicatis backup of this is only 91gb? that’s insane, if something didn’t go wrong, that’s smaller than ARQ or Duplicacy from what I recall.

#4

Double check the job results and make sure it didn’t skip anything and that it actually processed all 645GB.

#5

Yeah, I’m browsing around it right now, I do have lots of images but even with absolutely ridiculous compression it seems a tad suspect. The only errors I had were accessing the system volume information folder. I need to poke around more though to see if it went through everything.

The file search on Duplicati is pretty nice too compared to the speed of others.

I’m also wondering about the ability to add more hard drives to existing backups, as I go, since I’ll be getting more of them later on, I don’t know if I can add those to an existing bulk backup.

#6

Hmm, so no issues verifying or running the backup again, however it seems odd that it examined 0 source files apparently and in the output log it says for the known file count it’s 185, but there should be a little over 138,800 files.

Sorry, not sure how to do a scroll-able code style paste of this so it’s not so long.

Blockquote
{
“MainOperation”: “Test”,
“VerificationsActualLength”: 3,
“Verifications”: [
{
“Key”: “duplicati-20200223T084231Z.dlist.zip.aes”,
“Value”:
},
{
“Key”: “duplicati-i02393b6d56dd40da83ed63a821207c1b.dindex.zip.aes”,
“Value”:
},
{
“Key”: “duplicati-b69b4ce74b2484fa2875159a0c0397c41.dblock.zip.aes”,
“Value”:
}
],
“ParsedResult”: “Success”,
“Version”: “2.0.5.1 (2.0.5.1_beta_2020-01-18)”,
“EndTime”: “2020-02-23T21:19:34.2957872Z”,
“BeginTime”: “2020-02-23T21:18:40.9829065Z”,
“Duration”: “00:00:53.3128807”,
“MessagesActualLength”: 9,
“WarningsActualLength”: 0,
“ErrorsActualLength”: 0,
“Messages”: [
“2020-02-23 15:18:41 -06 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Test has started”,
“2020-02-23 15:18:45 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started: ()”,
“2020-02-23 15:18:45 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed: (185 bytes)”,
“2020-02-23 15:18:45 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-20200223T084231Z.dlist.zip.aes (5.95 MB)”,
“2020-02-23 15:18:45 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-20200223T084231Z.dlist.zip.aes (5.95 MB)”,
“2020-02-23 15:18:45 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-i02393b6d56dd40da83ed63a821207c1b.dindex.zip.aes (1.02 MB)”,
“2020-02-23 15:18:45 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-i02393b6d56dd40da83ed63a821207c1b.dindex.zip.aes (1.02 MB)”,
“2020-02-23 15:18:53 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-b69b4ce74b2484fa2875159a0c0397c41.dblock.zip.aes (1,023.91 MB)”,
“2020-02-23 15:19:26 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-b69b4ce74b2484fa2875159a0c0397c41.dblock.zip.aes (1,023.91 MB)”
],
“Warnings”: ,
“Errors”: ,
“BackendStatistics”: {
“RemoteCalls”: 4,
“BytesUploaded”: 0,
“BytesDownloaded”: 1080947655,
“FilesUploaded”: 0,
“FilesDownloaded”: 3,
“FilesDeleted”: 0,
“FoldersCreated”: 0,
“RetryAttempts”: 0,
“UnknownFileSize”: 0,
“UnknownFileCount”: 0,
“KnownFileCount”: 185,
“KnownFileSize”: 98112264469,
“LastBackupDate”: “2020-02-23T02:42:31-06:00”,
“BackupListCount”: 1,
“TotalQuotaSpace”: 6001039241216,
“FreeQuotaSpace”: 5202969399296,
“AssignedQuotaSpace”: 4831838208000,
“ReportedQuotaError”: false,
“ReportedQuotaWarning”: false,
“MainOperation”: “Test”,
“ParsedResult”: “Success”,
“Version”: “2.0.5.1 (2.0.5.1_beta_2020-01-18)”,
“EndTime”: “0001-01-01T00:00:00”,
“BeginTime”: “2020-02-23T21:18:40.9829065Z”,
“Duration”: “00:00:00”,
“MessagesActualLength”: 0,
“WarningsActualLength”: 0,
“ErrorsActualLength”: 0,
“Messages”: null,
“Warnings”: null,
“Errors”: null
}
}

The initial full backup shows accurate numbers for file counts and sizes however, here is the log. Not sure if there’s anywhere else to poke around.

Blockquote
{
“DeletedFiles”: 0,
“DeletedFolders”: 0,
“ModifiedFiles”: 0,
“ExaminedFiles”: 139053,
“OpenedFiles”: 139053,
“AddedFiles”: 139053,
“SizeOfModifiedFiles”: 0,
“SizeOfAddedFiles”: 692179331976,
“SizeOfExaminedFiles”: 692179331976,
“SizeOfOpenedFiles”: 692179331976,
“NotProcessedFiles”: 0,
“AddedFolders”: 5266,
“TooLargeFiles”: 0,
“FilesWithError”: 0,
“ModifiedFolders”: 0,
“ModifiedSymlinks”: 0,
“AddedSymlinks”: 0,
“DeletedSymlinks”: 0,
“PartialBackup”: false,
“Dryrun”: false,
“MainOperation”: “Backup”,
“CompactResults”: null,
“VacuumResults”: null,
“DeleteResults”: {
“DeletedSetsActualLength”: 0,
“DeletedSets”: ,
“Dryrun”: false,
“MainOperation”: “Delete”,
“CompactResults”: null,
“ParsedResult”: “Success”,
“Version”: “2.0.5.1 (2.0.5.1_beta_2020-01-18)”,
“EndTime”: “2020-02-23T14:36:34.3524943Z”,
“BeginTime”: “2020-02-23T14:36:21.7907872Z”,
“Duration”: “00:00:12.5617071”,
“MessagesActualLength”: 0,
“WarningsActualLength”: 0,
“ErrorsActualLength”: 0,
“Messages”: null,
“Warnings”: null,
“Errors”: null,
“BackendStatistics”: {
“RemoteCalls”: 190,
“BytesUploaded”: 98112264469,
“BytesDownloaded”: 1080708471,
“FilesUploaded”: 185,
“FilesDownloaded”: 3,
“FilesDeleted”: 0,
“FoldersCreated”: 0,
“RetryAttempts”: 0,
“UnknownFileSize”: 0,
“UnknownFileCount”: 0,
“KnownFileCount”: 185,
“KnownFileSize”: 98112264469,
“LastBackupDate”: “2020-02-23T02:42:31-06:00”,
“BackupListCount”: 1,
“TotalQuotaSpace”: 6001039241216,
“FreeQuotaSpace”: 5202970066944,
“AssignedQuotaSpace”: 4831838208000,
“ReportedQuotaError”: false,
“ReportedQuotaWarning”: false,
“MainOperation”: “Backup”,
“ParsedResult”: “Success”,
“Version”: “2.0.5.1 (2.0.5.1_beta_2020-01-18)”,
“EndTime”: “0001-01-01T00:00:00”,
“BeginTime”: “2020-02-23T08:42:31.4244036Z”,
“Duration”: “00:00:00”,
“MessagesActualLength”: 0,
“WarningsActualLength”: 0,
“ErrorsActualLength”: 0,
“Messages”: null,
“Warnings”: null,
“Errors”: null
}
},
“RepairResults”: null,
“TestResults”: {
“MainOperation”: “Test”,
“VerificationsActualLength”: 3,
“Verifications”: [
{
“Key”: “duplicati-20200223T084231Z.dlist.zip.aes”,
“Value”:
},
{
“Key”: “duplicati-ie123b19ec0714e7fa145321ccb33f09a.dindex.zip.aes”,
“Value”:
},
{
“Key”: “duplicati-b3516a1f5d865426d9d127ebdc5e10cbc.dblock.zip.aes”,
“Value”:
}
],
“ParsedResult”: “Success”,
“Version”: “2.0.5.1 (2.0.5.1_beta_2020-01-18)”,
“EndTime”: “2020-02-23T14:36:58.5688558Z”,
“BeginTime”: “2020-02-23T14:36:34.9785305Z”,
“Duration”: “00:00:23.5903253”,
“MessagesActualLength”: 0,
“WarningsActualLength”: 0,
“ErrorsActualLength”: 0,
“Messages”: null,
“Warnings”: null,
“Errors”: null,
“BackendStatistics”: {
“RemoteCalls”: 190,
“BytesUploaded”: 98112264469,
“BytesDownloaded”: 1080708471,
“FilesUploaded”: 185,
“FilesDownloaded”: 3,
“FilesDeleted”: 0,
“FoldersCreated”: 0,
“RetryAttempts”: 0,
“UnknownFileSize”: 0,
“UnknownFileCount”: 0,
“KnownFileCount”: 185,
“KnownFileSize”: 98112264469,
“LastBackupDate”: “2020-02-23T02:42:31-06:00”,
“BackupListCount”: 1,
“TotalQuotaSpace”: 6001039241216,
“FreeQuotaSpace”: 5202970066944,
“AssignedQuotaSpace”: 4831838208000,
“ReportedQuotaError”: false,
“ReportedQuotaWarning”: false,
“MainOperation”: “Backup”,
“ParsedResult”: “Success”,
“Version”: “2.0.5.1 (2.0.5.1_beta_2020-01-18)”,
“EndTime”: “0001-01-01T00:00:00”,
“BeginTime”: “2020-02-23T08:42:31.4244036Z”,
“Duration”: “00:00:00”,
“MessagesActualLength”: 0,
“WarningsActualLength”: 0,
“ErrorsActualLength”: 0,
“Messages”: null,
“Warnings”: null,
“Errors”: null
}
},
“ParsedResult”: “Warning”,
“Version”: “2.0.5.1 (2.0.5.1_beta_2020-01-18)”,
“EndTime”: “2020-02-23T14:36:58.5768571Z”,
“BeginTime”: “2020-02-23T08:42:31.4244036Z”,
“Duration”: “05:54:27.1524535”,
“MessagesActualLength”: 387,
“WarningsActualLength”: 4,
“ErrorsActualLength”: 0,
“Messages”: [
“2020-02-23 02:42:31 -06 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Backup has started”,
“2020-02-23 02:42:32 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started: ()”,
“2020-02-23 02:42:32 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed: ()”,
“2020-02-23 02:51:09 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-be27ecf968da34af09747d81888f2ca96.dblock.zip.aes (1,023.93 MB)”,
“2020-02-23 02:51:10 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-be705797a1ddc4cbeb3e3066da0ebd022.dblock.zip.aes (1,023.95 MB)”,
“2020-02-23 02:51:19 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-b712da0e4605242b3820f29539fd2a339.dblock.zip.aes (1,023.92 MB)”,
“2020-02-23 02:51:30 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-b4bc1d73934684566ae0a23f5bea2d58c.dblock.zip.aes (1,023.92 MB)”,
“2020-02-23 02:51:53 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-be27ecf968da34af09747d81888f2ca96.dblock.zip.aes (1,023.93 MB)”,
“2020-02-23 02:51:53 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-be705797a1ddc4cbeb3e3066da0ebd022.dblock.zip.aes (1,023.95 MB)”,
“2020-02-23 02:53:32 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-b712da0e4605242b3820f29539fd2a339.dblock.zip.aes (1,023.92 MB)”,
“2020-02-23 02:53:32 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-b4bc1d73934684566ae0a23f5bea2d58c.dblock.zip.aes (1,023.92 MB)”,
“2020-02-23 02:53:33 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-iccb37aaf3e444b6ab3ad1ad983a40467.dindex.zip.aes (655.34 KB)”,
“2020-02-23 02:53:33 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-i9fcbb002cb65464c8b008dc51fa9c60b.dindex.zip.aes (553.75 KB)”,
“2020-02-23 02:54:24 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-iccb37aaf3e444b6ab3ad1ad983a40467.dindex.zip.aes (655.34 KB)”,
“2020-02-23 02:54:24 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-i9fcbb002cb65464c8b008dc51fa9c60b.dindex.zip.aes (553.75 KB)”,
“2020-02-23 02:54:26 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-i3cd4a286bf164b1d88847c7a8c76c15d.dindex.zip.aes (726.01 KB)”,
“2020-02-23 02:54:26 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-i9569545caf1147669cc4dc13c521176e.dindex.zip.aes (1.04 MB)”,
“2020-02-23 02:54:26 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-b1ee004f0d53b45dba07a47babd14d819.dblock.zip.aes (1,023.99 MB)”,
“2020-02-23 02:54:26 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-b914c16729c6c46ec9c8801e3582cfb3d.dblock.zip.aes (1,023.96 MB)”,
“2020-02-23 02:54:28 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-i3cd4a286bf164b1d88847c7a8c76c15d.dindex.zip.aes (726.01 KB)”
],
“Warnings”: [
“2020-02-23 02:42:32 -06 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: D:\System Volume Information\”,
“2020-02-23 02:42:32 -06 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: D:\System Volume Information\”,
“2020-02-23 03:08:55 -06 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: D:\System Volume Information\”,
“2020-02-23 03:08:55 -06 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: D:\System Volume Information\”
],
“Errors”: ,
“BackendStatistics”: {
“RemoteCalls”: 190,
“BytesUploaded”: 98112264469,
“BytesDownloaded”: 1080708471,
“FilesUploaded”: 185,
“FilesDownloaded”: 3,
“FilesDeleted”: 0,
“FoldersCreated”: 0,
“RetryAttempts”: 0,
“UnknownFileSize”: 0,
“UnknownFileCount”: 0,
“KnownFileCount”: 185,
“KnownFileSize”: 98112264469,
“LastBackupDate”: “2020-02-23T02:42:31-06:00”,
“BackupListCount”: 1,
“TotalQuotaSpace”: 6001039241216,
“FreeQuotaSpace”: 5202970066944,
“AssignedQuotaSpace”: 4831838208000,
“ReportedQuotaError”: false,
“ReportedQuotaWarning”: false,
“MainOperation”: “Backup”,
“ParsedResult”: “Success”,
“Version”: “2.0.5.1 (2.0.5.1_beta_2020-01-18)”,
“EndTime”: “0001-01-01T00:00:00”,
“BeginTime”: “2020-02-23T08:42:31.4244036Z”,
“Duration”: “00:00:00”,
“MessagesActualLength”: 0,
“WarningsActualLength”: 0,
“ErrorsActualLength”: 0,
“Messages”: null,
“Warnings”: null,
“Errors”: null
}
}

#7

Holy crap, well investigating with windirstat, I have a directory backing up savegames from a multiplayer server I was hosting, and those would all be test, and christ, that folder is 538gb, I had no idea that it got that big, I wonder if I forgot a background task running or something. That would be super easily compressed.

Also, woo, I just found half a terabyte of storage space to free up! That means overall I have around 195gb on this drive to properly back up.

#8

Yep it does look like it process all 645GiB of data. You can also check the basic summary on the Duplicati Web UI for a quick idea of how much data it is protecting:

Capture

Source should match the size of files on your disk.
Backup is how much it takes on the back end including all versions.

#9

Duplicati expects a single location for the backup destination. So probably what I would do if you outgrow one drive and buy a second is to move some backups entirely to the second drive. Different jobs can have different targets.

#10

Ah sorry I was a little unclear, I should explain my setup better.

I have a rackmount HP DL380 G6 and the current drive cage of 8 drives is full, however I have a second drive cage and controller cards on the way, so I can expand up to 16 as money allows.

If I set up a backup job for the entire machine with all current drives, I was wondering if I can then alter it to add the 9th drive into the existing backup set when I physically add it into the server, or if I’d need to make an individual backup for each new drive as I add them in.

That being said since I’ll need more remote storage anyhow I might just make a backup set for the whole left drive cage, and later one for the whole right cage, and have those go to two individual hard drives remotely, of course I’d still need to add physical disks to the right drive cage backup, if that makes sense.

edit: also not sure if duplicati should give an estimate for file recovery downloads, or a progress bar, it currently just says it’s downloading, but no speed or progress, but it is pulling files.

#11

Oh yes, you can always add new folders or drives to the Source Data page of the backup job configuration. You don’t need to make a new backup set. I thought you were talking about adding a disk to the backup target.

#12

Well, this is being annoying, it looks like duplicati totally filled my C drive (with about 40gb of free space) and errored out, and now I can’t repair the database as it first said “unexpected update count 0”, then “System.IO.InvalidDataException: Found 2 file(s) with missing blocklist hashes” and now I can’t repair or delete and recreate the database because there are apparently no file lists.

Uhg. What sort of hard drive space do I need for a database of ~5.4tb? Maybe it just screwed up for some other reason or something, I don’t know but I guess I’ll start over from scratch and try adding the drives one at a time instead of all at once? Should have been through a decent chunk of this by now.

#13

Was it the sqlite database? How big did you see it get?

#14

I wasn’t monitoring it closely, when I checked my server the OS drive briefly showed a a red bar in file explorer before it fully loaded. The logs were also being iffy about loading in the web ui for some reason, not sure why but when I clicked “show” on the error, or tried to view the log from the backup and not through the about page it didn’t load.