Incremental Differential Backups that include removal of data from source? Is there an easy way OR what if I do it manually?

Support
1

Part of what finally drove me to implement a backup solution (other than the fear of loss) was that I store a lot of videos on my machine and run out of disk space fast. I wanted a way to automate transferring the videos to the cloud (google cloud backup) where I could remove the files from my machine locally, add more new ones, then have those new ones added to the lump in the cloud. I’m guessing that fits the “Incremental Cumulative” model?

So I have 2 questions:

1: Can Duplicati be configured to remove the local files after backup; and, in the future, just rinse and repeat with any new files found in that location?

2: If the answer to 1 is no, then what if I manually delete the files locally that were backed up to the cloud? Will Duplicati freak out the next time it tries to back up a few new files placed in that location (ie: hundred or thousands of errors/warnings and maybe even job failure)?

Thanks.
I really want to find a way to do this cause I need to git rid of some data from local to make space but I don’t want to lose it.

2

No. Duplicati is a backup program, meaning it keeps another copy. It’s not meant to delete the original.
There may be special-purpose programs claiming huge reliability and good searches for your archive.

Duplicati takes views at specific times, so finding a long-ago-deleted file is guessing (or your records).
In addition, videos don’t compress or deduplicate well, and sometimes you don’t even need versioning.

I’m not clear on this concern. Deleting and adding files in a folder is a completely normal scenario, right?
If you replace a file with one having the same name, the hard job of figuring out what is what gets harder.

Deleted files are removed from the backup when Backup retention on Options screen 5 says to drop.
Keeping versions forever is possible (but operations slow as backup grows – larger blocksize may help).

Unless you see major benefits to features that Duplicati adds, you could just copy a file at a time directly.

Move files to cloud storage deleting the local after verification

sounds like what you ask for (plus maybe some record-keeping, that you might be on your own for doing).
Rclone syncs your files to cloud storage is the page I got the above quoted text from. Does that help you?

3

As long as the software doesn’t try to make the destination match the source no matter what (eg: deleting files from the destination because they are no longer found in the source).

My concern was just that. That I would lose data at the destination… that when it was deleted locally (source) it would be removed remotely (destination).

The configuration section I think is being referred to has the following options in a drop down menu…

  • Keep all backups
  • Delete backups that are older than
  • Keep a specific number of backups
  • Smart backup retention
  • Custom backup retention

…none of which include “Drop”. But I think I get what you are saying. Currently that backup is set to the first option “Keep all backups”; and, based on the info below the configuration that says “Nothing will be deleted. The backup size will grow with each change”, I’m figuring that if I delete the files from the source they will still persist at the destination even after the next backup.

^ The reason I make a point of saying that is I am literally not deleting anything until I am absolutely clear whether that is correct or not - whether that “Keep all backups” option means the destination data will continue to persist even after deleting from the source - I don’t want to find out by losing data.

Just that the process is automated so I don’t have worry about it getting done at regular intervals or having to repeat the steps .

Ultimately I do have a need for both a regular sort of backup (where the destination is maintained as identical to the source even if it means deleting things at the destination) - as well as - just moving files from source to destination (what I asked about here). I would love if that can be done all in one application (rather than 2 or more) and be very simple to use/implement (like Duplicati is easy to use).

Thanks for mentioning the other option too. I’ll check out what that is.

In the mean time I would love to hear a definite confirmation (or negation) of whether my current configuration is going to lead to files in the destination being deleted or not → with the “Keep all backups” option that I have selected. This way I would know if I can safely delete those files now before I brick my laptop.

Much appreciated.
Jake

4

Overview says

Duplicati is not:

  • A file synchronization program.

Duplicati is a block based backup solution.

(if it were a sync program then it could mirror deletions – backup programs like Duplicati are different.
A particular point-in-time backup version will match the source of course, but it’s supposed to match)

“Drop”, “Delete”, “Don’t keep” (after a specific number of backups) – all control retention/deletion result.

Correct because you still have a backup version that is holding the file, but you might have to go find it.
Note that “persist” doesn’t mean you have a file of that name sitting there. It’s a special backup format.
I’m trying to get you to not use those, because long-term format support is a risk for long-term archive.

Duplicati will not intentionally delete it, but your data risks permanent irrrecoverable loss if Duplicati fails.
This is why I’m trying very hard to not have you delete original files. What if you want them in 20 years?

Archival storage should use extremely standard formats, not relying on any particular software package. While Duplicati does offer a Python script to restore without Duplicati, Python also changes periodically.

Although cloud storage always has some risk of the vendor going away, Google seems pretty safe until someone gets in your account and deletes everything, or your credit card fails thus your data is deleted.

There’s really no substitute for actual backups (multiple file copies) if you truly seek to avoid losing data.

I have no product recommendations. Duplicati is backup, and poorly suited to archive-and-delete usage.
Add a global search for all versions of backup feature request would make finding files a little bit simpler, however I still think you’d have a hard time finding any software guaranteeing perfect archiving forever…

Records retention is essential to businesses (sometimes required by law). I think costly software exists. Periodic migrations may be required as technologies change over the decades. Time frames matter too.

From a restore point of view, you should (until something breaks or too much time passes) be able to get source files back from the backup files thanks to “Keep all backups”. I still advise against doing it this way. “Should” is not always “will”, and if you hang around the forum, you can see things do sometimes go bad. Duplicati in its current Beta form is very good for many people, but to give it your one copy forever? Risky.

If brick means because weird breakages will happen when disk fills, Duplicati can also suffer such breaks.
In terms of “safely”, I don’t think your plan is particularly safe, except for off-site protects against drive loss. Only you can decide how much risk you want to take with your data. I’ve laid out my thinking extensively…

As a side note on Google, if you use Google Drive and allow it to sync the remote Duplicati backup, it adds further local storage usage but it can be configured. Google Cloud Storage is a different Google offering…

If your computer is teetering on the verge of collapse, you can try some lower-risk things to free up space, such as perhaps buying a USB hard drive to temporarily free some space so you can run long-term plans.
You could also upload some big files to Google using its web UI, or some GUI client. Such software exists.

1 Like
5

@ts678

Very educational. Really appreciate the time you took to explain these things to me. It may not be perfect but I think I have a plan. :slight_smile: