I’m trying to backup /var/; when I add it, my backup size suddenly balloons to 100+TB. My total local storage is about 1.9TB total.
I’ve tried excluding gvfs, as I figured that was the culprit. But no luck there.
I assume this a link issue of some sort, but I don’t know how to debug.
You might be experiencing the same issue as described here:
Unfortunately, we have yet to determine what the cause is as it’s difficult to reproduce. What operating system and filesystem are you running? On my Arch Linux system with ext4, I can select /var as a source and it reports the size correctly. Most people that have this issue appear to be using some copy-on-write filesystem, but at least one user is using ext4.
I wonder if your file manager can measure directory sizes, starting with /var? If huge, go into subdirectories until the cause is found. On Windows, there are also tools to do this. TreeSize Alternatives for Linux has Linux ideas.
It turned out that my Linux Mint system came with baobab a.k.a. Disk Usage Analyzer, and mentioned in that list.
A non-GUI tool is du which has numerous options for how links should be treated. I’m not sure about the others.
Files with sizes larger then the file system could be sparse files, but even for that, 100TB seems like a whole lot.
Another possibility is that Duplicati’s counting is off. If this backup finishes anytime soon, that’s likely the case…
I’m running Arch Linux. ext4 file system.
du -sh ./ says 11 gigs.
sudo du -sh ./*/ has an interesting entry. However, it shouldn’t be affecting me due to my excludes:
du: cannot access './run/user/1000/gvfs': Permission denied
Per the Arch wiki, this should be excluded from backups. GVFS = gnome virtual file system. I’ve got an exclude there to account for this.
I killed the backup when it got that big. I’ll give it a shot now and see what happens.
Ok, I did a (partial) backup of /var/, adding one subdirectory at a time.
Partial because I only got 3 directories in before things ballooned.
It was /var/lib that caused the size to balloon. du says it’s 5.6G. Duplicati says adds 256TB. But it uploads in a few mins, so I’m pretty sure it’s not uploading 256TB.
Also worth noting that the source size (displayed under the profile name) seems appropriate.
If the high size sticks on future backups (might be hard to tell because it might flash by too fast, but starting a new test backup would solve that), you could test subdirectories of /var/lib to isolate this further. Rather than wade through 70 (on my system – yours may have more) subdirectories, you could getting a suspects list by
sudo find /var/lib ! -type f ! -type d -exec ls -lnd {} ; | less
basically looking for any folders containing something not a plain old directory or file, and testing there first…
Simpler but indirect would be to add --apparent-size to du. Here I play with a nearly 16TB spare file I created:
$ du -sh
4.0K .
$ du -sh --apparent-size
16T
Although you probably don’t have an actual sparse file, but there could be similar filesystem oddities at play.
I’m going to wait for this backup to finish before I start making test backups (this current backup is my only true backup; true local backup hard drive is in the mail…)
However, I did just du -sh -apparent-size the entire directory, folder-by-folder. Nothing suspicious that I could find.
One thing that does seem odd is that it displays 256.25TB. Note that the actual size of my source is 256GB, which accounts for the 0.25TB.
Occasionally (after uploading one set of some recursive links or something, maybe), it displays 128.XXTB.
Is it just a weird coincidence that it’s 256 and 128 (maybe next would be 64 and 32)? Or is it just a factor of 1000 error? EDIT: Nope on the 1000x. I increased my source files to a total of 285G. Duplicati is now displaying 256.28.
@funnyguy Did you ever solve this issue?
Looks like the ticket #3536 in github was never fixed.
I can’t belive that no one else is affected and pushing this issue.
Welcome!
Have you customized the symlink policy option?
Hi, you mean setting symlink-policy = Store?
No change. Still counting to 400 GB what should be around 30 GB.
Do the following two commands produce the same output?
$ du /var
$ du /var -L
No, -L produces more.
Mostly in /var/run.
Is that number from something like du /var? If not, what? Context for the various numbers would help.
Does the backup log Examined under Source Files show about 400 GB, 30 GB, or something else?
What about home screen Source and Backup size/versions (versions may increase size, of course)?
I’m assuming this is just a plain backup of /var. Yours is larger somehow. Mine is only around half a GB, ignoring the Docker stuff. Have you looked into the Restore tree to see if you see anything unexpected?
Depending on how much space, time, and maybe bandwidth you want to spend, Restore to a different folder would let you look around to see if, for example, restore restores symlinks, or things they point to.
Download amount might not be that bad, because by default source blocks are used if they’re available.
I am running Ubuntu on a bootable 64GB USB Stick.
Any full backup of the complete ubuntu install that shows me 400 GB of counted size is sooo wrong!!
Performing Backups should be straight forward without spending hours of investigating errors and checking that everything works. Not talking about the effort in testing restore. I lost confidence in Duplicati.
This is taking me more time than I expected. So I will create image backups of the stick running a different host os (Win10) on my PC instead.
Thanks for everyone who tried to help. Seems that duplicati is nice for for backups of personal folders only, but my intention was to have incremental system snapshots.
Best Regard & Thanks again
limex
It can be completely expected if sparse files exist, and /var has one of the common ones, /var/log/lastlog
lastlog - reports the most recent login of all users or of a given user
It is a sparse file, so its size on the disk is usually much smaller than the one shown by " ls -l " (which can indicate a really big file if you have in passwd users with a high UID). You can display its real size with " ls -s ".
This may or may not be what’s going on. I note that du has an option that will get size with holes included:
–apparent-size
… it may be larger due to holes in (‘sparse’) files
Duplicati would back up the holes, but unless file extension calls for no compression, they compress well.
For more about this, Google search for:
“/var/log/lastlog” “size”
or
“/var/log/lastlog” “backup”
or
“/var/log/lastlog” “exclude”
Assuming this means a direct image of the blocks without interpretation as a filesystem, that should work.
You can backup the images to get versions if you like, however they might not block-deduplicate very well.
EDIT 1:
Here’s how to put a 100 GB file on my 19 GB (3 GB left) file system with roughly zero impact on its space:
$ df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 19G 15G 3.0G 84% /
$ dd if=/dev/zero of=sparse_file conv=sparse bs=1M count=102400
102400+0 records in
102400+0 records out
107374182400 bytes (107 GB, 100 GiB) copied, 60.1291 s, 1.8 GB/s
$ df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 19G 15G 3.0G 84% /
$ du -h sparse_file
0 sparse_file
$ du -h --apparent-size sparse_file
100G sparse_file
$
EDIT 2:
If the du with and without --apparent-size shows a possible sparse file situation, it’s possible to find files.
Then it sounds like you have some symlinks in there. What does this show?
$ ls -l `find /var -type l`
This will show you all symlinks. Are any of them pointing back to a parent directory, which may lead to endless looping if followed?
Yeah Duplicati is not the right choice if you want full system backups (like image level or bare metal DR backups). Good luck on your quest!