Purge clarification

ccmgr · February 27, 2022, 9:49pm

I have a backup that contains a database backup that’s not VSS compliant. As a result I create a zip file backup. These zip files get removed every 30 days and I retain the one from a month end. These zip files are in a duplicati backup source with other files and folders

As a result of backend storage issue I now want to purge the zip files that are no longer in the source from the duplicati backup.

From reading docs if i purge using *.zip it will remove all zip files from backups. Is this correct ?

Can I pass a list of files to purge from a txt file ?

Anyone have sql query that will list files in backup database but not currently in the current source ?

Ger

drwtsn32 · February 28, 2022, 5:00pm

Yes, that is correct.

You can restrict what data is purged by using more path information in your wildcard. For example purging D:\path\*.zip would only delete zip files in D:\path or one of its subfolders.

Also you can limit the purge operation to only certain backup snapshot versions.

No, not that I’m aware of but if you can craft your purge request to be a more specific wildcard as I mentioned above, maybe that would help. Note that you aren’t limited to just one wildcard/include. You can do more than one in a single purge operation.

Might I suggest a better solution in the future is to separate these zip files so they are in their own backup job. That way you can use a different retention length and avoid needing to do any purges.

Lastly, please be very careful about running the purge command. Do it with a test backup set if possible so you can get familiar with how it works. The command line options must be crafted very carefully.

ccmgr · March 9, 2022, 12:01pm

Rod

Thanks for the clarification. So as these zip files where in two directories I decided to remove them from backup and create a separate backup. And then purge. However when I try to purge I am getting error with orphan files.

Backup is running with no issue. I have tried to resolve the issue with the --auto-cleanup = true on a backup but no luck.

List-Broken-Files returns no broken file sets and repair database has no errors.

I had a look at the source and this is a list of entries in the file table that are no longer in a file set. In an effort to reduce this backup I had previously changed the rotation from keep-all to smart rotation. This ran without error and removed versions and was running without issue for several weeks. I suspect these entries are as a result of version removal.

Is there anyway to clear this error other then recreating the database ?

Ger

ts678 · March 9, 2022, 10:04pm

Nice job. Are you a developer? Duplicati team needs those, or even people who can look at source.

github.com

duplicati/duplicati/blob/fd1e687ac8a5a872fd15b88633c3f95c00a9e4f9/Duplicati/Library/Main/Operation/PurgeFilesHandler.cs#L78-L80


      
          var orphans = db.CountOrphanFiles(null);

          if (orphans != 0)

              throw new UserInformationException(string.Format("Unable to start the purge process as there are {0} orphan file(s)", orphans), "CannotPurgeWithOrphans");

called

github.com

duplicati/duplicati/blob/fd1e687ac8a5a872fd15b88633c3f95c00a9e4f9/Duplicati/Library/Main/Database/LocalPurgeDatabase.cs#L51-L59


      
          internal long CountOrphanFiles(System.Data.IDbTransaction transaction)

          {

              using (var cmd = m_connection.CreateCommand(transaction))

              using (var rd = cmd.ExecuteReader(@"SELECT COUNT(*) FROM ""FileLookup"" WHERE ""ID"" NOT IN (SELECT DISTINCT ""FileID"" FROM ""FilesetEntry"")"))

                  if (rd.Read())

                      return rd.ConvertValueToInt64(0, 0);

                  else

                      return 0;

          }

Its relative that I found with a text search for some of the above SQL to see if there was fixup code:

github.com

duplicati/duplicati/blob/fd1e687ac8a5a872fd15b88633c3f95c00a9e4f9/Duplicati/Library/Main/Database/LocalDeleteDatabase.cs#L87-L94


      
          //Then we delete anything that is no longer being referenced

          cmd.ExecuteNonQuery(@"DELETE FROM ""FilesetEntry"" WHERE ""FilesetID"" NOT IN (SELECT DISTINCT ""ID"" FROM ""Fileset"")");

          cmd.ExecuteNonQuery(@"DELETE FROM ""ChangeJournalData"" WHERE ""FilesetID"" NOT IN (SELECT DISTINCT ""ID"" FROM ""Fileset"")");

          cmd.ExecuteNonQuery(@"DELETE FROM ""FileLookup"" WHERE ""ID"" NOT IN (SELECT DISTINCT ""FileID"" FROM ""FilesetEntry"") ");

          cmd.ExecuteNonQuery(@"DELETE FROM ""Metadataset"" WHERE ""ID"" NOT IN (SELECT DISTINCT ""MetadataID"" FROM ""FileLookup"") ");

          cmd.ExecuteNonQuery(@"DELETE FROM ""Blockset"" WHERE ""ID"" NOT IN (SELECT DISTINCT ""BlocksetID"" FROM ""FileLookup"" UNION SELECT DISTINCT ""BlocksetID"" FROM ""Metadataset"") ");

          cmd.ExecuteNonQuery(@"DELETE FROM ""BlocksetEntry"" WHERE ""BlocksetID"" NOT IN (SELECT DISTINCT ""ID"" FROM ""Blockset"") ");

          cmd.ExecuteNonQuery(@"DELETE FROM ""BlocklistHash"" WHERE ""BlocksetID"" NOT IN (SELECT DISTINCT ""ID"" FROM ""Blockset"") ");

This is in DropFilesetsFromTable which is run in PurgeFilesHandler.cs (unfortunately after test) and

github.com

duplicati/duplicati/blob/fd1e687ac8a5a872fd15b88633c3f95c00a9e4f9/Duplicati/Library/Main/Operation/DeleteHandler.cs#L110-L113


      
          if (versionsToDelete.Count > 0)

              Logging.Log.WriteInformationMessage(LOGTAG, "DeleteRemoteFileset", "Deleting {0} remote fileset(s) ...", versionsToDelete.Count);

          

          var lst = db.DropFilesetsFromTable(versionsToDelete.Select(x => x.Time).ToArray(), transaction).ToArray();

Can you either read more to see if you think just deleting a version will fix the bad reference, or test it, keeping in mind that this is highly experimental? I tried a tiny test backup, and it seemed to do the job.
Rather than delete any existing version, I backed up a new one just to delete it as version=0 (newest).

ccmgr · March 24, 2022, 12:27pm

Good spot. Creating a new version and then deleting it removed the orphan files from the Database

Not a developer but as a System Admin I have goods scripting knowledge and can read code to debug at times

Will probably look to help on this project once I retire in the next year or so.