I come to you because I would like to have a Duplicati use to make Backup of some of my folders in my NAS.
I know that Duplicati is made to do versionning, but in my case it would use too much disk space, I would like to make 1 Backup for 1 folder with this configuration suite:
1 Times all days at 5am and it simply delete the old one.
Example:
Backup created on November 15 at 5am
Backup created on November 16 at 5am > Removes the backup of November 15
Backup created on November 17 at 5am > Removes the backup of November 16
Every day, so on.
Basically a simple incremental backup rule like rsync could do, it seems simple but none of the configurations I tried to work for this use case, is it mandatory to do versionning?
I already tried with the predefined rules that removes the older backup of 1 day when the new one is created, but it never worked, a backup was created on October 28, we are the 15 November, major changes were made to the original folder (folder size divided by 3), and the backup size is always the same as the one created on October 28th, as if it had never created another.
Thank you and sorry if my question is stupid or inconvenient.
Incremental backups
Duplicati performs a full backup initially. Afterwards, Duplicati updates the initial backup by adding the changed data only. That means, if only tiny parts of a huge file have changed, only those tiny parts are added to the backup. This saves time and space and the backup size usually grows slowly.
Deduplication
Duplicati analyzes the content of the files and stores data blocks. Due to that, Duplicati will find duplicate files and similar content and store them only once in the backup. As Duplicati analyzes the content of files it can handle situations very well if files and folders are moved or renamed. As the content does not change, the next backup will be tiny.
As shown where? I don’t think there’s any per-version size, as data is shared when possible.
On home screen, what is the version count? On Restore menu, do you see the new version?
As backup size on home screen is all the versions, you might just be in a steady size pattern.
You can look in the Complete log in your job log at “BytesUploaded”. Unless a compact ran, uploads are basically your changes, and a file (with date in name) that says what’s in backup.
There should be as many of these “dlist” files as there are versions, and if you keep 1 version, previous dlist will be deleted after new one is uploaded at end of backup. At some time it does
and you can look in your job logs to see what those are doing. Total space used may fluctuate.
Old version is not normally deleted until new one is made, as new one might use data from old. Additionally, versions are specific time points, unlike rsync which is half-done in middle of a run.
A file synchronization program.
Duplicati is a block based backup solution. Files are split up in small chunks of data (blocks), which are optionally encrypted and compressed before they are sent to the backup location. In backup location, Duplicati uploads not original files but files that contain blocks of original files and other necessary data that allows Duplicati to restore stored files to its original form by restoration process. This block based backup system allows features like file versioning and deduplication. If you need to be able to access your files directly from the backup location, you will need file synchronization software, not block based backup software like Duplicati.
Thank you very much for taking the time to answer me, it’s very clear, I understand better how it works now.
In my case as I have specified, my first backup was done on October 28, at that time my folder weigh 3gb (1.8gb for the Duplicati backup), meanwhile I had a bigre-organization of my folder and the mismatch of the content was moving to a new folder, the inital folder weighs only 45mo from October 30.
on the date of November 15, the Backup Duplicati always makes 1.8gb and when I try to restore the backup as you advise me I only have the date of October 28 that in the choices of the backup.
I assume that no Backup was created after the 28th of October, date of the first Backup.
I must have made a mistake in the versioning parameters.
Do these both refer to the same folder before file re-org? Is new folder inside or outside original? Moving files to different subfolders inside a backed up folder should have no effect on space use.
It sounds like your Nov 15 backup didn’t finish successfully. Do you recall? Is there a job log? Checking About → Show log → Stored for Nov 15 messages is another place you could look. Destination drive is as well. Any files there from Nov 15? You can sort by date. The dlist file should be roughly the latest. If you have just Nov 15 dblock and dindex, maybe you did fill up, however that should have been logged, and fill should also be apparent using Windows tools.
EDIT:
I was assuming you meant a local disk drive when I talked about checking, but is 1.8 gigabytes actually capable of getting it almost full? That’s far smaller than, for example, recent hard drive.
No it is another folder that has nothing to do with the original one, not a sub-folder
No there is nothing, unfortunately I have deleted the configuration so this backup does not exist anymore.
No I’m talking about a local hard drive, 1.8GB will not fill my disk, it’s as an example because for now it is the only folder that I backup with Duplicati, it is my folder “Tutorial” so mainly text files, I already do my backups with Rsync, but I start with this folder as a test phase, then I intend to replace all the folders synchronized with Rsync to pass them on Duplicati, but for that I must first understand its operation, That’s why I started with a small file only.
What should happen when you move files out of a backed up folder to one that’s not is the job log Source Files summary should show less source, with the files you moved considered Deleted, however because the purpose of a backup is normally to preserve files as instructed, deleted files remain available until your chosen retention policy removes that version. Compact could run then, reclaiming wasted space to reduce the destination size. The settings for compact are configurable.
Note that some logs are not in the job, e.g. server log is About → Show log → Stored – for all jobs.
Unless there is massive data churn on all folders (some people do have this), the size growth with more than 1 version should be limited as explained, because a second version shares first’s data.
Yes we agree it is the whole principle of the backup, but precisely this is what alerted me because I like to indicate in my post, I want only 1 single version that is updated every day at the same time, so as soon as the next day of the files moved from the source, my backup was no longer censer make 1.7go but 45mo, I left him more than a week in doubt but it is still stay at this size there.
Yes I have error logs dated November 15, but I don’t think this is related to the initial problem because when I had this problem before coming here I tried to fix it by myself by changing the retention rules, I removed the default rule of “If at least one more recent backup is found, all backups prior to that date are deleted > 1 Day” and I created a custom rule with “1D:1D” but this did not work I run the backup to test and I still had errors, I can show them to you if you want but they do not concern the initial problem on the default rule.
Yes I understood this concept thanks to you and I thank you, it motivates me even more to switch to Duplicati because I find its operation awesome, but I really do not master it at the moment, I must absolutely understand its operation and fix the problem of the folder test before, I have made a new backup, it is 3.8go, I have to set well on “If at least one more recent backup is found, all backups prior to that date are deleted > 1 Day”, I will voluntarily add a large file of 3.2Go (Debian ISO), logically, tomorrow after 5am the backup should weigh 7gb, then I will delete this file again, in 2 days it should go back to 3.8gb - I am good?
The folder layout was not clear. Previously the 45mo one sounded like one files moved to per:
If above 45mo a source folder size or a Duplicati Destination size? Or are both 45 somehow?
I also don’t know the term “censer make”, but if I ignore it you now complain backup is small?
This tells me nothing without knowing the backup history, source activity, and logs you deleted.
There are too many unanswerable questions, including why “there is nothing” from the backup.
You might need to test more carefully, looking at job logs and available restores, to learn more.
What sort of error? If that was a backup, it might explain the “there is nothing” you wrote before.
That’s not the default rule. You can see this by start down “Add backup”. Don’t save, just notice:
I’m not even sure what setup can configure your rule. Regardless, you replaced whatever it had:
That is pretty strange if that’s all there is, or is there more to that string after that part you show? Admittedly your goal is a little unusual, but you’re asking for a backup to exist for one single day.
All errors are welcome. Redact any private info if need be. Errors are there to say what’s wrong.
Given your topic subject of “Only 1 Backup”, I was expecting that’s what you had configured, so
Can you post a screenshot of the setting? I can’t comment much on next part without knowing it.
Or say exactly what retention dropdown has, and exactly what you specified for any typed parts.
OK, destination size. Following this so far.
Roughly right. Compression might reduce that a little.
Your job log should show the Source size shrink down to initial size (if no other source change).
Did you say how often you do backup? I’m searching for “daily” and “hour” and not seeing them.
The “go back” is not always exact, as data between versions can be shared, however the added image probably doesn’t have much sharing. Probably the Compact part job log will show a lot of completely deleted files. This is faster than having to repack still-relevant blocks to upload again.
Retention setting and backup rates are unclear, but if you ask to retain one backup, then the ISO removal and backup should show some drop right away, as after backup you should have only a single backup after things settle. Those things (view the job log) include version delete and likely compact, as such a large drop in size should exceed the default config, so waste gets reclaimed.
I’ve mentioned the Source statistics in the job log. For delete and compact info, expand sections:
and you can also keep an eye on the home screen versions and sizes, and peek at Restore lists.
Although I’m giving an example of only 1 backup (because it was asked), I’m hoping more will fit.
Yes it’s true that I don’t give all the configuration sorry, so I’ll show you everything so you can understand what I target, and if my settings match well to what I’m looking for.
As I have stated, I want a backup to be created every day at 5am, and that when a new backup is created, the old one is automatically deleted, no versionning therefore, I consider versioning but later and this does not concern all the folders and it is necessary above all that I understand the exact functioning of Duplicati before I start in the versioning.
I did a test with my E-book folder, yesterday I created the folder it made 3.8gb, I started the backup task immediately on it and the backup folder made 3.6gb (with compression if I remember well). around 6pm, I added the iso of 3.2go and I did not launch anything, I let it do automation at 5am to see if it will copy the new file add.
Today I check and indeed he added the new file, the new backup now makes 7.11gb.
Now I will delete this file again, and logically tomorrow after his 5am task, the backup folder should show 3.6gb again (that’s what I’m looking for).
I think @ts678 has covered it, but here is another way of explaining it:
Duplicati stores “versions” (i.e., snapshot-ish versions) with deduplication on the contents.
Due to the deduplication, the backups only grow with the size of the new data (roughly).
When you delete an old version, you are only deleting the description of the version, but the contents are still there and may be used by other files/versions.
When you delete versions there can be data that is no longer referenced by any version, and this is essentially wasted space. However, since Duplicati stored data in compressed+encrypted remote volumes, it is possible that parts of the remote volume is in use, and parts are unused.
To fix this issue, Duplicati runs the “compact” operation, where it downloads all remote volumes with wasted space above a threshold, and combines them into new volumes (if a volume contains only unused data, it is naturally not downloaded).
Because the compact operation is time consuming and does not offer anything other than reduced space, it is only triggered when it is deemed relevant. You can control this with the advanced option
--threshold=25
The --threshold option defines how many percentage of the volume should be unused before it is collected. If 0 or 1 remote volumes can be compacted nothing is done. You can set it to 1, to have an agressive collection of remote data, if that fits your use case.
(The compacting is also triggered when a sufficient number of “small” volumes are stored, to reduce the overall number of remote volumes, but that is not relevant for your case).
Sorry for the late answer I have a lot of work lately,
I come back to the result of my test: "Today I checked and indeed it added the new file, the new backup is now 7.11gb.
Now I will delete this file again, and logically tomorrow after its 5am task, the backup folder should show 3.6gb again (that’s what I’m looking for)."
and indeed the folder is down to 3.6gb, the backup has worked well.
What is strange is that I did exactly the same configuration on the first one that caused my problem and my coming to the forum, but it didn’t work.
I will keep this configuration and adapt the source and destination folders according to the folders I want to save.
Thank you very much for your help @kenkendk and especially @ts678 that I had to make crazy with all my questions, you are great and kind.
