How to backup files older than X days


#1

Hello, I thought an interesting use case. I want to backup only files that are older than 14 days.
So Duplicati will ignore (filter out) files files younger than 14 days and backup only older files.

Is there any way to do it in Duplicati?

I cant find any date related filter, so the only thing I can think of is external script to set attributes to files and use “exclude-files-attributes” in Duplicati
But we already have “skip-files-larger-than” so maybe “skip-files-younger-than” should exist too? :slight_smile:

Thanks for the ideas.


Filter question
#2

I like the idea - especially as it could be a good way to avoid backing up temp or in-use files since they’re usually pretty fresh. :slight_smile:

One problem scenario I see is this:

  1. file is created thus younger than skip limit and not backed up
  2. file eventually gets old enough to be backed up and is
  3. file is updated making it younger than skip limit and not only is it not backed up, but the it’s flagged in the backup run as if it had been deleted
  4. file eventually gets old enough to be backed up and is

So the major tripping points off the top of my head include:

  • depending on retention policies you could lose a lot of your backup coverage
  • if you go to restore files and happen to select a backup that occurred between 3 and 4 above then you wont get ANY version of the file in question restored (because Duplicati saw it was deleted).

I think this could be avoided if instead of excluding a file because it was too young instead flagged it as “unchanged”, but that brings about it’s own potential (likely less damaging) set of situations to ponder. :thinking:


#3

Maybe this is doable with help of --exclude-files-attributes

description says:

Use this option to exclude files with certain attributes. Use a comma separated list of attribute names to specify more than one. Possible values are: ReadOnly , Hidden , System , Directory , Archive , Device , Normal , Temporary , SparseFile , ReparsePoint , Compressed , Offline , NotContentIndexed , Encrypted , IntegrityStream , NoScrubData .

So in theory, I can create daily script which will be set Archive attribute for new file and remove it from older files. And Duplicati will skip new files with Archive attribute and back up older files.

Can you think of any drawbacks to this approach? Thanks


#4

I don’t even think you’d need the first part. As far as I know, Windows still sets the Archive flag automatically on new files or when files are modified. You’d have to just script the clearing of that attribute for files that are old enough.


#5

Thanks @drwtsn32 ! I just test it with help of Everything and you’r right, all my files have Archive bit set.

So I can set up --exclude-files-attributes=Archive witch which excludes all my files from backup set.

Then I can run PowerShell script like this everyday to remove Archive attribute from files - and Duplicati will backup them.

$limit = (Get-Date).AddDays(-30)
$path = “C:\Folder”
Get-ChildItem -Path $path -Force | Where-Object { !$.PSIsContainer -and $.CreationTime -lt $limit } | Clear-ItemProperty -Name attributes

Did I forget something? Thanks


#6

I solved it, maybe this help’s someone.

This script will search all files with “Archive” attribute and remove all attributes (so not only Archive, but also System, Invisible…)
Removing only Archive attribute would be unnecessarily more complicated.

I added parameters to backup job
--run-script-before-required=C:\DuplicatiScripts\BAT_calling_PowerShell.bat

--exclude-files-attributes=Archive

content of BAT_calling_PowerShell.bat file

powershell -ExecutionPolicy Bypass -File “C:\DuplicatiScripts\PowerShellScript.ps1”

Then I created PowerShell file PowerShellScript.ps1” with content:

$limit = (Get-Date).AddDays(-30)
$path = “C:\MyFolder”

#view files or number
#Get-ChildItem -Path $path -Force -Recurse | Where-Object { !$.PSIsContainer -and $.CreationTime -lt $limit -and $.Attributes -eq ‘Archive’ } | Measure-Object | %{$.Count}

#Set attribute Arhive for older than the $limit.
Get-ChildItem -Path $path -Force -Recurse | Where-Object { !$.PSIsContainer -and $.CreationTime -lt $limit -and $_.Attributes -eq ‘Archive’ } | Clear-ItemProperty -Name attributes

When backup job is ran, script is executed, archive attribute is cleared for old files and Duplicati will backup them.
Thanks @drwtsn32 bringing me to this idea and helped me :hearts:

@JonMikelV Can you please move this topick to How-To - Duplicati category?


#7

It’s not too bad and I think it’s important not to clear other attributes. Here’s a small change to your script:

Get-ChildItem -Recurse -Force -Attributes 'Archive' | Where-Object { $_.CreationTime -lt (Get-Date).AddDays(-30) } | ForEach-Object { $_ | Set-ItemProperty -Name Attributes -Value ($_.Attributes -bxor 'Archive') }

It is also more efficient to have the check for Archive attribute in the Get-ChildItem command itself, so there are less objects to send through the pipeline.