HELP?! Trying to use Duplicati, but something's always broken!


#1

I’ve been trying to use duplicati for a while now to replace CrashPlan as a cross-offsite backup solution between my Parent’s house and mine. I’ve gotten everything setup so that it should work…but something’s nearly always broken with one of the systems or another.

Layout:
1.) Primary Home Server - linux based (unRAID) - Runs duplicati in a docker container.
Has 2 backups configured - one is to a local mount point/file share that exists one one singular backup drive. This tends to run fine. The other is offsite to an ubuntu server at my parent’s house. The endpoint is a Minio docker container (Amazon S3 compatible storage). This endpoint gives me nothing but trouble. Hasn’t worked since 6/15/2018. Connection failures I understand and can usually fix those, but even when I test and it’s working I can’t seem to get it to do anything. Recently I’ve been getting an error that says to repair the database. I attempt that and all it does is sit on “Starting…”. I’m trying a recreate now and it just says “Recreating database …” IS THERE NO WAY TO SEE WHAT THE @Q#$%& IT’S DOING?!?!?!* Logs show N.O.T.H.I.N.G. in the webUI - tcpdump on my host only shows a few packets every few minutes going from my server out to their IP.
2.) My Desktop - Windows 10 - Backs up to minio on my server and on my parent’s server. Biggest complaint here is Duplicati doesn’t seem to want to launch at startup so I went months without a backup because I didn’t know it wasn’t running. Anyway to fix this that I’m missing? Backs up to the same endpoints as 1. Not sure if it has any better luck right now or not…will let you know when the current “verify backend data” finishes. Though it does say that the last offsite backup took 23:23:10 (W.T.F?!)
3.) Parent’s server - Ubuntu 18.04.01 - running the same Minio docker I have on mine for an S3 endpoint. This isn’t actually running duplicati…it’s just an endpoint.
4.) Mom’s desktop - Windows 10 - Honestly, I haven’t checked up on this in a while. I need to go visit and check it out. I know it was having trouble with remote backups to my server. I believe it was backing up to their server just fine.

Notes:
Minio endpoints are accessed via LetsEncrypt docker containers that are handling https certs for all internet traffic to/from the boxes. Both are accessible at a dynamicDNS entry via web browser from the opposite location.
Original backup from my server and desktop were done with my parent’s server on my local network then it was moved offsite and the connection updated.
When data transfers were occurring they were SLOW AS *&%^! I’ll need to get connections re-established to verify speeds but it was in the Kbps (measured mine to theirs). I have ~ 100Mbps down and about 15Mbps up on a good day for connection speeds. They’ve got 60 down and ?? up…shouldn’t be far off mine…we’re both using the same provider and about 15-20 miles apart geographically.

Now my desktop is running verification against my main server…it says it’s 100% progress, but the top bar still says “Running…” and nothing appears to be happening. Is there ANY way to tell if this is doing something!!!

Are there any really good guides for ensuring your setup is correct? Sorry for ranting but I’m getting really worried that I can’t get good backups in place for two households. With all the family memories being digital today I need to get this in place.


#2

Hello @NeoMatrixJR and welcome to the forum!

Quite a lot of stuff going on there, but let me see if I can help address one of your frustrations at least a little.

The web UI (as is typical for such “user friendly” interfaces, leaves something to be desired when debugging, however one can also open it in another tab, go to About --> Show log -> Live and crank the logging way up. Profiling level can be overwhelming, but Information should get you at least more insight into what’s going on.

There are a lot of options to control logging. On newer Duplicati canary versions (not yet in Beta I think), you specify separate settings for console logging and log file logging. You can look over the logging options here.

Another way to have a traditional line-by-line view of the activity is to use the job’s Commandline menu option with advanced options set to the desired verbosity level. Use console-log-level on the newer canary versions.

One Commandline caveat is that if you do something besides backup, the command line might need change.

Even the increased logging might not get you everything you might want, but might as well get what’s there…


#3

Does that mean you installed it as service, expect it to start at boot, but find after boot that service is stopped? Does it start manually? On some systems (such as mine…), service on “Automatic (Delayed Start)” avoids this. There’s a technical explanation for this (involving updates, do you have any?), but first let’s learn the situation.


#4

hmmm… Well, I could have sworn it was installed as a service as it used to start at boot, but I can’t find it in my services list. Sorry…I wasn’t sure what mechanism it was supposed to use to start at boot (startup item, service, etc.) Is it possible a windows update could have removed the service? Is there a way to re-register the service?
Also, biggest “here” means just with that machine…


#5

I wouldn’t have thought a monthly or even the twice-yearly Windows update would remove the service, but the twice-yearly one can clobber Duplicati configuration and database files if they are kept in the SYSTEM profile.

Migrating from User to Service install on Windows is one place which discusses this, and what to do about it… Although it calls itself a migration guide, info is also useful on first install. Or are you migrating from tray icon?

I know that there are other issues, but I thought I’d start where I could. Some others may need log information.

One other thing I could mention is that Linux distro versions of Mono seem to be a chronic problem, but there aren’t enough specifics here yet to advise an upgrade, unless you want to try it because it often works better.

Do you have anything else going to the Minio remotely, or between houses using any other communications?


#6

ok, so it turns out I do have a shortcut to the duplicati.gui.trayicon in my personal startup folder…it just apparently never actually starts up :frowning:

I made the mistake once of setting my dockerized duplicati to canary and updating it, letting a backup run, then updating the docker which downgraded my version back to stable causing my backups to have a version mismatch. I’m not sure I’m fond of attempting a mono upgrade inside a docker only to have it revert later…though I suppose I could set the docker to not auto-update and just do duplicati updates manually… I will consider this.

Nothing else hits minio on the servers. Mine gets hit only by my mom’s desktop and my desktop, theirs gets hit by her desktop, my server, and my desktop. I’d have to double check schedules, but I was pretty sure they should be scheduled far enough apart to avoid concurrency issues…though not with one task taking 23hrs!


#7

Started looking into this…is there a preferred version to upgrade to or one that’s been tried/discussed? It looks like it’s currently running the latest stable by default.


#8

Generally the advice seems to be to use the latest, and it seems to fix odd Duplicati issues and not break Linux, although if you have concern about this, please use your judgment. I know the latest fixed an odd TLS failure…

SSL TLS support in Mono gets into some other discussion that may or may not apply. Personally I didn’t need it.


#9

Sorry to hear about the troubles you’re having.

I’ve got Duplicati running in a Docker container on unRAID and haven’t had any issues with mono, but then I’m using local and Box.com destinations so maybe they don’t have there latest SSL certificates.

Are you using the LinuxServer or Duplicati container?

I set up a Minio destination but never really got it working so moved to SFTP.

With the off-site support you seem to be doing you might want to consider a reporting aggregation tool like Dup-Report so you have a better idea of what might not be running.


#10

Running the linuxserver.io container. Will look into the reporting. Was planning on doing this if it ever stabilized.


#11

I used the linuxserver.io container until an official Duplicati one (duplicati/duplicati) came out a few months ago.

Since the local destination is working fine I’m going to assume your container isn’t the problem. Just to check if it’s a connectivity issue, maybe you could set up a Minio container on the same box and use that as a destination for a few runs.