Is Duplicati 2 ready for production?

61

It’s a bit sad that there are no developers willing to help :frowning:
Maybe you need to take care of marketing more, collect more subsidies to pay helpers, etc …? I just mean make it louder about Duplicati in the world :wink:

Do you have any experience with Vorta/Borg or Kopia?
I wonder if the Duplicati + Vorta (or Kopia) scenario is a good idea…

That’s why I’m reheating the old thread to ask if the authors of the negative posts have solved their Duplicati problems or changed the backup application

This is a bit like judging the reliability of cars. When you walk into a Bentley or Rolls-Royce workshop, you’ll think “oh my God, so many broken Bentleys! = I won’t buy a Bentley”:wink:

Here, however, I have the impression that the data does not contain backups that cannot be restored (because they are broken, but you don’t know about it)

I am only considering FOSS solutions.
And they don’t even have to Free (I always send a donation because I consider it fair-play). But they must be Open Source.

62

You can try, but the forum doesn’t email all past posters automatically, so you rely on right people noticing.

I wish there was a restore stat of any sort, but there isn’t. My assumption is it’s worked well enough to keep using it to backup. Couple that with an assumption that failure gets posted here, and try to draw inferences.

Generally before getting to the point where it’s concluded that a backup cannot be restored, we lead people through Disaster Recovery and if all else fails there’s the Duplicati.CommandLine.RecoveryTool.exe which gets run (or even mentioned) quite rarely given the number of backups. You can search for uses yourself…

This does not mean that recovering before that point is easy – it can be difficult and take forum assistance.

63

I opened this thread, thus I should probably answer too.
I installed Duplicati 2 on two windows systems to create online backups to an off-site location via scp.
One user has stopped using it, mainly because it was using to much (well, all off the little available) internet upstream bandwidth and online TV was suffering from this. Clearly not a problem of Duplicati 2, I assume other online backup solutions would have similar problems :-).
The other system is running with an “acceptable” failure rate. I fails to backup every few months and needs then manual attention (recreate local database, IIRC).
To be honest, I did not try a restore since some time, but I will do so now, as I have the backup disk right here by coincidence (I never considered restore via home DSL connections, my plan was always to go to the backup site in case of a failure).
UPDATE Restore worked, not super fast, but it worked, as far as I can tell.

64

This is exactly one of the pain points. But even worse is the silent corruption.

Did you do the restore test right? Ie. without the local database, as example to another computer. One of the mail rage points is that the test works, as long as you have the original data, because it can recover using local blocks… It’s just like the compression trojan horse apps,. which compress data very efficiently. And it works perfectly, until you’ll delete the original files. Yep, it didn’t compress anything at all, it just referenced the original files. - I’m actually very worried that people will find out this the hard way. The issue isn’t probably technically big, as I’ve posted in the another post. But it’s still total show stopper.

65

In my case, I will create a fresh full backup for the simple reason that the backup medium is reporting uncorrectable sectors. I’m not taking any risk there.

Backup was done on a windows computer and saved to a remote machine using SCP. For maintenance reasons, I have this “remote” machine currently at my home and used this to perform a restore test.
For the restore test, I installed Duplicati on my linux Desktop and did a full restore of a backup point about 1 month in the past. Data was transferred on a local LAN, speed was not that great, a bit more than 10MByte/sec. CPU-Usage on my linux computer was about 300% on an old Opteron with 8 Threads. I was positively surprised by the fact that it is multi-threaded :slight_smile:

1 Like
66

Any idea why it drastically decreased compared to 2 months ago ? Any FOTM tool (ie Kopia) or more precision about this ? Thank you !

67

I’m not seeing that. The monthly graph by compression is nice because almost all is in .zip, so there’s no need to total the categories manually. This went up a bit from 3.99 to 4.31 million from 2021-11 to 2022-01.

One anomaly is the monthly by operating system, where Linux dropped in 2021-10. My guess is this if from Old Let’s Encrypt Root Certificate Expiration and OpenSSL 1.0.2, so inability to return some usage reports.

Because client-side fixes can be hard, I suggested a server certificate change. I haven’t heard it being done.
https://github.com/duplicati/usage-reporter should be the code. I know there are details the graphs don’t get.

I don’t follow the “FOTM tool (ie Kopia)” comment, but I don’t have access to server or any further precision.

68

Update, finally. I’ve reset ALL backup sets and updated ALL clients to latest version. I’ll now run full restore test on ALL systems. After that I’ll keep running automated restore tests as normal. Then we’ll see if things are broken, or not. Some of the older backup sets could have been partially broken for a very very long time, even if restore worked. As I said, it was obviously bad code with earlier versions.

Yet even with the latest version the restore sometimes is absolutely ridiculously slow. There’s a very bad code smell at least 10 miles far. But that might have been caused by the older versions leaving some gunk or broken files in the directories.

As I said, restore often gave code 2. Which mean that the backup was broken, but it was still able to restore it. Which obviously shouldn’t happen.

69

I hope you had adjusted blocksize on large backups to keep blocks in the backup below a few million blocks. It makes big difference to database performance. Blocksize change does require a fresh start.

I first thought this followed on from the paragraph above it, but now I’m thinking it’s the earlier history, especially since first paragraph spoke in future tense. Third paragraph might also be on prior results?

I can test the theory if you like, but it looks like it doesn’t even take an error, merely a warning, to get 2. Personally, I think Duplicati sometimes over-warns, but if you keep seeing these, what warning is this?

70

Yes, I sometimes get warnings on restores merely because I have the destination folder open in an Explorer window.

71

–dblock-size=“1GB” --blocksize=“64MB”

The ridiculous / insanely slow restore usually starts with “Registering missing blocks”… And then you know you can throw out all wishful thinking about RTO out of the window. Because it can take well, days or weeks as mentioned here earlier.

Related the the previous statement, starts with registering missing blocks … And if you’re able to wait through the code smell and insanely slow single threaded code (which btw is NOT limited by I/O just burning CPU cycles) then you’ll end up with code 2, if you’re lucky. But Code 100 is also likely result and restore failure after that.

It’s not over warning, if it’s up to random change if backup restore will ever complete / be successful.

But let’s hope the newer version is less likely to constantly corrupt data. We’ll be seeing. As mentioned. Today I’m testing all the backups and confirming that the backup sets are now all good. Then I’ll just keep monitoring the failure rate. If there’s the classic Duplicati silent backup set corruption problem, I’ve been so frustrated with. Hopefully the latest canary version does handle the situation better. Also with compact. At least I’m hopeful because now it seems to at least at times, to start recovery correctly if aborted. As well as do some recovery steps which were missing earlier.

Ref: FTP delete not atomic / verified / transactional (?)

I’ve also updated absolutely all related software, Duplicati, .NET, Windows versions, back-end server software, absolutely all possible related things and so on.

72

I did huge job updating all clients, checking all configuration parameters, upgrading all server side software to the latest version. Making sure that configuration and hardware all are in good shape.

Yet …

Testing and repairing says everything is 100% good and there are no problems…

Examined 24 files and found no errors

And then restore fails:

Duplicati return code: 100
Duplicati output:
Restore started at 24.3.2022 13.00.43
  Listing remote folder ...
  Downloading file (37,18 KT) ...
  Downloading file (36,53 KT) ...
  Downloading file (36,89 KT) ...
  Downloading file (37,17 KT) ...
  Downloading file (36,68 KT) ...
  Downloading file (36,92 KT) ...
  Downloading file (36,56 KT) ...
  Downloading file (36,73 KT) ...
  Downloading file (40,72 KT) ...
  Downloading file (36,83 KT) ...
  Downloading file (36,68 KT) ...
  Downloading file (36,86 KT) ...
  Downloading file (1,86 MT) ...
  Downloading file (401,62 KT) ...
  Downloading file (2,92 MT) ...
  Downloading file (484,73 KT) ...
  Downloading file (487,17 KT) ...
  Downloading file (593,56 KT) ...

Duplicati error output:

ErrorID: DatabaseIsBrokenConsiderPurge
Recreated database has missing blocks and 3 broken filelists. Consider using "list-broken-files" and "purge-broken-files" to purge broken data from the remote store and the database.

And from verbose log, interestingly no actual error message is logged. It just fails after this point.
2022-03-24 13.00.55 +02 - [Information-Duplicati.Library.Main.Operation.RecreateDatabaseHandler-RecreateCompletedCheckingDatabase]: Recreate completed, verifying the database consistency

100% proof that the program is systemically and logically broken and does not work as intended. Yet, this is not the first time, I’ve very clearly indicated and pointed this same thing out.

… I’ve got two similar backup sets right now, both with same issue, 100% good and valid but totally unrestorable …

To sum it up: Again, the program self-sabotages data, gives users very dangerous disinformation and then lets them down when the backup is actually needed.

I’ve now triple checked the situation, and I’m 100 certain. Latest version, test and repair both pass… Restore fails… That’s just so … bleep bleep bleep !!!

I just keep wonder what the point of developing all kind of nice ahem, pseudo cute frosting features is, if the essential key ingredients of the product are totally spoiled?

73

I have been unable to reproduce your problem. I’ve done a full direct restore from backup files a few times without issue.

I am still not sure what is different about your particular setup, but it definitely seems unreliable. Are you still using the same destination target and protocol as before?

74

I’m redoing this test myself (again)… mine had no issue passing that point:

...creating temp database....
Mar 24, 2022 8:15 AM: Recreate completed, verifying the database consistency 
Mar 24, 2022 8:15 AM: Recreate completed, and consistency checks completed, marking database as complete
Mar 24, 2022 8:15 AM: The operation Repair has completed
...starting actual restore...
1 Like
75

I also agree and yet also disagree. It is broken in some ways. Once you figure out how to workaround that then it works great.

I have a computer where Duplicati has a code issue that it doesn’t handle and the way its coded freezes its UI. Backups would become corrupted. The computer wouldn’t suspend. Once I spent many hours debugging its code, I found a solution that on that computer didn’t backup the local Duplicati DB files and all was good. This problem never occurred on this computer I’m typing from.

The code is complex that you cannot always pinpoint the actual problem from the logs so anyone who tries may misdiagnose the actual issue.

Duplicati definitely absolutely has code issues and it needs to be hardened. Those issues are not exactly simple to deal with so it may be a number of years (or never if they never do).

I do not recommend it as your only backup application. You should use multiple as any of them can fail. But if you use multiple then its not that bad. As the only one then I’d absolutely choose another way at this time.

76

I shared some numbers with you in private. I did test 100% of the backups during a week, and I found issues in some. Sometimes it needs a specific edge or corner case to get triggered. Works most of time but not always. My classic failure example is like, having a file where there’s a monotonic counter 1, 2, 3 … When I update the file, I open it truncate it and write a new value. It can easily work millions or billions of times, until the the exact moment is hit where the process is stopped, disk becomes full after the truncate and before the write, or the system is shutdown / loses power before the data is flushed to persistent storage or someone launches two parallel processes by accident. The code is still broken logically, because it doesn’t cover this case. And yes, this is a issue I’ve had with my own code. :wink:

But let’s focus how we can get forward from this, and hopefully isolate the problem and get it fixed (!).

77

Presumably you are testing what GUI calls “Direct restore from backup files”, otherwise Recreate would not even be running. Your reported error output (which I had not spotted before) comes from this check:

which unfortunately seems like it has a ton of SQL behind it, but probably the SQL can be snagged from the profiling log during that run (it should be just before the error), and put into the database in question, however a DB recreation for restore makes a partial temporary database. Code comment sounds like it might still be hanging around somewhere. Not sure offhand just where – check tmp and usual DB areas.

If restore can’t recreate the database, you’d probably have a repair issue too, if a database did not exist.
I’m not sure if there would be any debug advantage though, but broken database might be easier to find, which might make it easier to provide by sanitized database bug report, if we decide to try that idea next.

If you can find the SQL (which probably looks easy until one looks at BROKEN_FILE_IDS), it may help.
DB Browser for SQLite will let you or someone with original or sanitized database look more at problem.
Error message itself is pretty much lacking in details, but I guess there’s only so much one can write up.

and it feels very strange getting in this deep on this particular thread, so feel free to start another topic.