Recreating database hanging for days

[enviouselitist]

@ts678

This is an interesting find. How did you determine they’re UTF-8 and not Latin-1 (ISO 8859-1) or similar?

I used python, though apparently it’s not definitive. The characters are Korean.

chardet.detect(os.fsencode(filename))
{'encoding': 'utf-8', 'confidence': 0.99, 'language': ''}

I should correct myself, I found the foreign characters from a later SELECT statement finding surrounding blocks, since the block in question is of course missing association (from your SELECT statement).

SELECT * FROM File WHERE BlocksetID BETWEEN #### AND ####

It’s these neighboring files have the foreign characters. But important point is that plenty these files with these characters were backed up correctly, so maybe the issue has nothing to do with it. There is a gap of 18 BlocksetIDs that are missing in the File view, 1 of which contains the block causing the issue.

@ts678

Are these files with the foreign characters seemingly recently added, e.g. if you look at time with ls -lc.?
Keeping them around would allow further testing after current crisis passes to see how they break things.
Are these your files, or do you need to consult with someone else to move them, let you take a copy, etc.?

Indeed, the creation date of these is exactly right after the last successful backup time. They are actually from a google takeout archive of all their products, so there are a ton of new files introduced. Hopefully, since the BlocksetID looks somewhat linear with respect to directories and filenames, the actual files can be inferred, though the ordering does not match my OS’s alphabetical listing. But I am open to doing more testing with these files to see if it breaks things.

@ts678

It’s not hanging there. It’s saying the SELECT completed, and saying how long it took.
SELECT like this is very common. It would be useful to see context to it, for example:

Good point. I did mean to say “after” not “on” and clarified this in my 1st reply.

@ts678

That’s why you need to look earlier to see if it’s in the middle of processing dindex files, or did the last one.
There shouldn’t be a hang either way, but at least it would be nice to find out where it was when it stopped.

Ah, so the issue was due to using the most verbose profiling output as mentioned, the web interface was flooded, and it only keeps some small amount of buffer. That was entirely filled with the similar SELECT statements, which as you mention, aren’t very helpful. I added an output logging file and can now see more useful messages.

Running while logging to an external file now shows me everything. I was getting a surprising amount of Google Drive 403 errors. I had previously thought this meant upload/download quota, but right when I saw that message I checked and verified I can download and upload without error. I’m thinking this also means transactions per time window quota as well. Other processes are accessing this same google drive at the same time. EDIT: I did stop all other processes and still get intermittent 403 Forbidden occasionally for some reason, but it succeeds after a few retries.

It hasn’t gotten to the point of hanging yet, but I’m guessing when exhausting the retry count and retry delays (mine was set to 25 retries, 10s delay), duplicati does not error out correctly. Based on this theory, I increased the retries to 30 with 30s of delay and restarted the database recreation and it is going strong.

Using the Profiling timer was probably a mistake, the profiling output log is up to 20 GBs now, though at least it’s being compressed by the filesystem.

If this completes successfully, I can see how backing up the files with foreign characters goes as well as see if I can reproduce the hanging scenario by changing the retry delay / limit.