Alternative FTP failure?

While testing I’ve found that the AlternativeFTP/FluentFTP is failing on the verify portion within the PutAsync method.

I’m not sure if the issue is with FluentFTP or the FTP server which I have setup locally using FileZilla Server.

What happens in PutAsync is that after a file is uploaded a listing of the ftp server is pulled. Except that the listing does not always contain the just uploaded file. Retries will occur but will all eventually fail for the same reason.

I can resolve the problem by just placing Thread.Sleep(3000) in between the upload and the verify portion.

Has anyone encountered this issue or inconsistent backups with AlternativeFTP?

Error: uploaded but the returned size was 0 was a recent forum report, which I found by looking for –disable-upload-verify, which I suggested after looking at the code. FTP has it too though. I don’t know why it’s better. One course of action might be to see if the newer FluentFTP code does better. It’s urgently needed anyway, because the current one breaks on current usage going to UNIX-based servers (changes the “line endings”).

EDIT:

FTP problem described here and it was one of the reasons there’s no beta yet – didn’t want this regression. Whether or not the latest FluentFTP actually fixes it is not 100% clear, so the earlier the tests of it, the better.

Thanks for the quick research! I’ll test ftp to see if it behaves like the aftp backend.

Is the alternativeftp supposed to replace the ftp backend once it works or are both to remain?

AFAIK both are to remain. Sometimes one works and the other doesn’t (can go either way) so choice is good, plus yanking away things that currently exist and are used is not a good thing to do to the user base. Stats at:

https://usage-reporter.duplicati.com/

FTP usage is pretty high (despite all of FTP’s inherent flaws) and for whatever reason higher than Alternative.

You might have noticed that the report I cited used Filezilla Server too, and I was asking for info from its log…

I just ran a simple console app using FluentFTP and FileZillaServer to test the interaction between them.

I was able to transfer with no problem. I’m thinking the issue not with those but Duplicati is somehow ending the transfer before it is finished since adding my 3 second delay also makes Duplicati transfer the files correctly.

With the other thread about the same type of hash mismatch but for SMB I’m now wondering if there is an overall issue of backend transfers being stopped earlier at times.

I need to look into it further.

(I’ve created a WIP PR in place for DEV discussion and at least a workaround, if not a permanent fix; PR3866)

I need assistance from other devs on this.

I can only replicate the problem when not in debug mode. This is AFTP against a local FileZillaServer.

So far I’ve tracked it to line 30 in BlockVolumeWriter.cs and m_compression will be null when the failure occurs.

The path of the code is that SpillCollectorProcess.cs line 116 will call AddBlock() in BlockVolumeWriter.cs and the m_compression will be null.

When run in debug mode or I place a 3 second delay on after the aftp PutAsync upload, then m_compression will not be null when/if the AddBlock() method gets called, depending on the spill.

p.s. I’m still trying to understand the whole ‘spill’ thing going on. If anyone is able to explain it I’d appreciate it.

Secondly, and maybe just as important a clue, before getting to the issue above I first was hitting an exception with a temp file missing. In TempFile.cs I commented out the .Delete(m_path); down in the Dispose and I was able to get to the exception outlined above. so there seems to be an issue with the temp files getting removed too early… possibly these issues are both realted to something ending too early and releasing an object(s).

I’m still trying to understand it too. Let me suggest some things which may or may not be actually correct.

# CoCoL - Concurrent Communications Library from the Duplicati lead developer is used lots in Duplicati:

The answer to why such a complex scheme was added (first beta was 2.0.4.5) is probably concurrency.
Lots of work gets parallelized, work doesn’t stop nice and even, so something must tidy up the leftovers:

I once began trying to trace the flow of things through channels and code, but haven’t had time in awhile…

If anybody knows of a good design document for that (also seeking one for transactions, please point to it).

Thanks. I’ve spent hours tracing the flow to try and understand how things are interacting. I’m thinking the AFTP issue is now resolved and that the errors in the spill portion were just a result of the AFTP backend not completing. I’m going to keep the ‘spill’ portion in mind though if we see issues with things like data corruption.