Could not find file "/var/folders..."; The socket has been shut down

Support
1

I’m trying to back up a Macbook Pro (10.15.7) to S3, and I get the error below. I haven’t found anything similar except maybe [this] (Failed ... "Backup" with id: 3 (Mac, Backblaze B2)). However, that post hasn’t helped me figure out what’s happening and how to fix it.

It’s under 12000 files, around 40 GB total.

For what it’s worth, I’d had a similar error on another Mac, which fixed itself when I excluded an erroneously included folder (which happened to be synced with dropbox) from the backup source. Not sure if that is relevant, but it’s the only other data point I have.

I’d appreciate any and all thoughts on what could be happening, or what experiments I could run that would throw light on what’s happening. Many thanks!

27 Nov 2020 09:59: Failed while executing "Backup" with id: 1
System.AggregateException: One or more errors occurred. (Could not find file "/var/folders/6l/0m065l1s0mj34kjj0546hfv80000gn/T/dup-0fa35334-b0c5-4d8a-9c52-6269a3c354cf" (Could not find file "/var/folders/6l/0m065l1s0mj34kjj0546hfv80000gn/T/dup-0fa35334-b0c5-4d8a-9c52-6269a3c354cf") (One or more errors occurred. (The socket has been shut down))) ---> System.AggregateException: Could not find file "/var/folders/6l/0m065l1s0mj34kjj0546hfv80000gn/T/dup-0fa35334-b0c5-4d8a-9c52-6269a3c354cf" (Could not find file "/var/folders/6l/0m065l1s0mj34kjj0546hfv80000gn/T/dup-0fa35334-b0c5-4d8a-9c52-6269a3c354cf") (One or more errors occurred. (The socket has been shut down)) ---> System.IO.FileNotFoundException: Could not find file "/var/folders/6l/0m065l1s0mj34kjj0546hfv80000gn/T/dup-0fa35334-b0c5-4d8a-9c52-6269a3c354cf"
  at System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share, System.Int32 bufferSize, System.Boolean anonymous, System.IO.FileOptions options) [0x0019e] in <b18e5763ec3f41e68a8fe16d7f5e0bee>:0 
  at System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share) [0x00000] in <b18e5763ec3f41e68a8fe16d7f5e0bee>:0 
  at (wrapper remoting-invoke-with-check) System.IO.FileStream..ctor(string,System.IO.FileMode,System.IO.FileAccess,System.IO.FileShare)
  at Duplicati.Library.Main.Volumes.VolumeReaderBase.LoadCompressor (System.String compressor, System.String file, Duplicati.Library.Main.Options options, System.IO.Stream& stream) [0x00000] in <8f1de655bd1240739a78684d845cecc8>:0 
  at Duplicati.Library.Main.Volumes.VolumeReaderBase..ctor (System.String compressor, System.String file, Duplicati.Library.Main.Options options) [0x00007] in <8f1de655bd1240739a78684d845cecc8>:0 
  at Duplicati.Library.Main.Volumes.BlockVolumeReader..ctor (System.String compressor, System.String file, Duplicati.Library.Main.Options options) [0x00010] in <8f1de655bd1240739a78684d845cecc8>:0 
  at Duplicati.Library.Main.Operation.Backup.SpillCollectorProcess+<>c__DisplayClass0_0.<Run>b__0 (<>f__AnonymousType11`2[<Input>j__TPar,<Output>j__TPar] self) [0x0023c] in <8f1de655bd1240739a78684d845cecc8>:0 
  at CoCoL.AutomationExtensions.RunTask[T] (T channels, System.Func`2[T,TResult] method, System.Boolean catchRetiredExceptions) [0x000d5] in <9a758ff4db6c48d6b3d4d0e5c2adf6d1>:0 
  at Duplicati.Library.Main.Operation.BackupHandler.RunMainOperation (System.Collections.Generic.IEnumerable`1[T] sources, Duplicati.Library.Snapshots.ISnapshotService snapshot, Duplicati.Library.Snapshots.UsnJournalService journalService, Duplicati.Library.Main.Operation.Backup.BackupDatabase database, Duplicati.Library.Main.Operation.Backup.BackupStatsCollector stats, Duplicati.Library.Main.Options options, Duplicati.Library.Utility.IFilter sourcefilter, Duplicati.Library.Utility.IFilter filter, Duplicati.Library.Main.BackupResults result, Duplicati.Library.Main.Operation.Common.ITaskReader taskreader, System.Int64 filesetid, System.Int64 lastfilesetid, System.Threading.CancellationToken token) [0x0035f] in <8f1de655bd1240739a78684d845cecc8>:0 
  at Duplicati.Library.Main.Operation.BackupHandler.RunAsync (System.String[] sources, Duplicati.Library.Utility.IFilter filter, System.Threading.CancellationToken token) [0x00a12] in <8f1de655bd1240739a78684d845cecc8>:0 
   --- End of inner exception stack trace ---
  at Duplicati.Library.Main.Operation.BackupHandler.RunAsync (System.String[] sources, Duplicati.Library.Utility.IFilter filter, System.Threading.CancellationToken token) [0x01033] in <8f1de655bd1240739a78684d845cecc8>:0 
   --- End of inner exception stack trace ---
  at CoCoL.ChannelExtensions.WaitForTaskOrThrow (System.Threading.Tasks.Task task) [0x0005d] in <9a758ff4db6c48d6b3d4d0e5c2adf6d1>:0 
  at Duplicati.Library.Main.Operation.BackupHandler.Run (System.String[] sources, Duplicati.Library.Utility.IFilter filter, System.Threading.CancellationToken token) [0x00009] in <8f1de655bd1240739a78684d845cecc8>:0 
  at Duplicati.Library.Main.Controller+<>c__DisplayClass14_0.<Backup>b__0 (Duplicati.Library.Main.BackupResults result) [0x0004b] in <8f1de655bd1240739a78684d845cecc8>:0 
  at Duplicati.Library.Main.Controller.RunAction[T] (T result, System.String[]& paths, Duplicati.Library.Utility.IFilter& filter, System.Action`1[T] method) [0x0026f] in <8f1de655bd1240739a78684d845cecc8>:0 
  at Duplicati.Library.Main.Controller.Backup (System.String[] inputsources, Duplicati.Library.Utility.IFilter filter) [0x00074] in <8f1de655bd1240739a78684d845cecc8>:0 
  at Duplicati.Server.Runner.Run (Duplicati.Server.Runner+IRunnerData data, System.Boolean fromQueue) [0x00349] in <c5f097a49c0a4f1fb0f93cf3f5f218b1>:0 
---> (Inner Exception #0) System.AggregateException: Could not find file "/var/folders/6l/0m065l1s0mj34kjj0546hfv80000gn/T/dup-0fa35334-b0c5-4d8a-9c52-6269a3c354cf" (Could not find file "/var/folders/6l/0m065l1s0mj34kjj0546hfv80000gn/T/dup-0fa35334-b0c5-4d8a-9c52-6269a3c354cf") (One or more errors occurred. (The socket has been shut down)) ---> System.IO.FileNotFoundException: Could not find file "/var/folders/6l/0m065l1s0mj34kjj0546hfv80000gn/T/dup-0fa35334-b0c5-4d8a-9c52-6269a3c354cf"
  at System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share, System.Int32 bufferSize, System.Boolean anonymous, System.IO.FileOptions options) [0x0019e] in <b18e5763ec3f41e68a8fe16d7f5e0bee>:0 
  at System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share) [0x00000] in <b18e5763ec3f41e68a8fe16d7f5e0bee>:0 
  at (wrapper remoting-invoke-with-check) System.IO.FileStream..ctor(string,System.IO.FileMode,System.IO.FileAccess,System.IO.FileShare)
  at Duplicati.Library.Main.Volumes.VolumeReaderBase.LoadCompressor (System.String compressor, System.String file, Duplicati.Library.Main.Options options, System.IO.Stream& stream) [0x00000] in <8f1de655bd1240739a78684d845cecc8>:0 
  at Duplicati.Library.Main.Volumes.VolumeReaderBase..ctor (System.String compressor, System.String file, Duplicati.Library.Main.Options options) [0x00007] in <8f1de655bd1240739a78684d845cecc8>:0 
  at Duplicati.Library.Main.Volumes.BlockVolumeReader..ctor (System.String compressor, System.String file, Duplicati.Library.Main.Options options) [0x00010] in <8f1de655bd1240739a78684d845cecc8>:0 
  at Duplicati.Library.Main.Operation.Backup.SpillCollectorProcess+<>c__DisplayClass0_0.<Run>b__0 (<>f__AnonymousType11`2[<Input>j__TPar,<Output>j__TPar] self) [0x0023c] in <8f1de655bd1240739a78684d845cecc8>:0 
  at CoCoL.AutomationExtensions.RunTask[T] (T channels, System.Func`2[T,TResult] method, System.Boolean catchRetiredExceptions) [0x000d5] in <9a758ff4db6c48d6b3d4d0e5c2adf6d1>:0 
  at Duplicati.Library.Main.Operation.BackupHandler.RunMainOperation (System.Collections.Generic.IEnumerable`1[T] sources, Duplicati.Library.Snapshots.ISnapshotService snapshot, Duplicati.Library.Snapshots.UsnJournalService journalService, Duplicati.Library.Main.Operation.Backup.BackupDatabase database, Duplicati.Library.Main.Operation.Backup.BackupStatsCollector stats, Duplicati.Library.Main.Options options, Duplicati.Library.Utility.IFilter sourcefilter, Duplicati.Library.Utility.IFilter filter, Duplicati.Library.Main.BackupResults result, Duplicati.Library.Main.Operation.Common.ITaskReader taskreader, System.Int64 filesetid, System.Int64 lastfilesetid, System.Threading.CancellationToken token) [0x0035f] in <8f1de655bd1240739a78684d845cecc8>:0 
  at Duplicati.Library.Main.Operation.BackupHandler.RunAsync (System.String[] sources, Duplicati.Library.Utility.IFilter filter, System.Threading.CancellationToken token) [0x00a12] in <8f1de655bd1240739a78684d845cecc8>:0 
   --- End of inner exception stack trace ---
  at Duplicati.Library.Main.Operation.BackupHandler.RunAsync (System.String[] sources, Duplicati.Library.Utility.IFilter filter, System.Threading.CancellationToken token) [0x01033] in <8f1de655bd1240739a78684d845cecc8>:0 
---> (Inner Exception #0) System.IO.FileNotFoundException: Could not find file "/var/folders/6l/0m065l1s0mj34kjj0546hfv80000gn/T/dup-0fa35334-b0c5-4d8a-9c52-6269a3c354cf"
File name: '/var/folders/6l/0m065l1s0mj34kjj0546hfv80000gn/T/dup-0fa35334-b0c5-4d8a-9c52-6269a3c354cf'
  at System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share, System.Int32 bufferSize, System.Boolean anonymous, System.IO.FileOptions options) [0x0019e] in <b18e5763ec3f41e68a8fe16d7f5e0bee>:0 
  at System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share) [0x00000] in <b18e5763ec3f41e68a8fe16d7f5e0bee>:0 
  at (wrapper remoting-invoke-with-check) System.IO.FileStream..ctor(string,System.IO.FileMode,System.IO.FileAccess,System.IO.FileShare)
  at Duplicati.Library.Main.Volumes.VolumeReaderBase.LoadCompressor (System.String compressor, System.String file, Duplicati.Library.Main.Options options, System.IO.Stream& stream) [0x00000] in <8f1de655bd1240739a78684d845cecc8>:0 
  at Duplicati.Library.Main.Volumes.VolumeReaderBase..ctor (System.String compressor, System.String file, Duplicati.Library.Main.Options options) [0x00007] in <8f1de655bd1240739a78684d845cecc8>:0 
  at Duplicati.Library.Main.Volumes.BlockVolumeReader..ctor (System.String compressor, System.String file, Duplicati.Library.Main.Options options) [0x00010] in <8f1de655bd1240739a78684d845cecc8>:0 
  at Duplicati.Library.Main.Operation.Backup.SpillCollectorProcess+<>c__DisplayClass0_0.<Run>b__0 (<>f__AnonymousType11`2[<Input>j__TPar,<Output>j__TPar] self) [0x0023c] in <8f1de655bd1240739a78684d845cecc8>:0 
  at CoCoL.AutomationExtensions.RunTask[T] (T channels, System.Func`2[T,TResult] method, System.Boolean catchRetiredExceptions) [0x000d5] in <9a758ff4db6c48d6b3d4d0e5c2adf6d1>:0 
  at Duplicati.Library.Main.Operation.BackupHandler.RunMainOperation (System.Collections.Generic.IEnumerable`1[T] sources, Duplicati.Library.Snapshots.ISnapshotService snapshot, Duplicati.Library.Snapshots.UsnJournalService journalService, Duplicati.Library.Main.Operation.Backup.BackupDatabase database, Duplicati.Library.Main.Operation.Backup.BackupStatsCollector stats, Duplicati.Library.Main.Options options, Duplicati.Library.Utility.IFilter sourcefilter, Duplicati.Library.Utility.IFilter filter, Duplicati.Library.Main.BackupResults result, Duplicati.Library.Main.Operation.Common.ITaskReader taskreader, System.Int64 filesetid, System.Int64 lastfilesetid, System.Threading.CancellationToken token) [0x0035f] in <8f1de655bd1240739a78684d845cecc8>:0 
  at Duplicati.Library.Main.Operation.BackupHandler.RunAsync (System.String[] sources, Duplicati.Library.Utility.IFilter filter, System.Threading.CancellationToken token) [0x00a12] in <8f1de655bd1240739a78684d845cecc8>:0 <---

---> (Inner Exception #1) System.AggregateException: One or more errors occurred. (The socket has been shut down) ---> System.Net.Sockets.SocketException: The socket has been shut down
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.<Run>b__13_0 (<>f__AnonymousType12`1[<Input>j__TPar] self) [0x00673] in <8f1de655bd1240739a78684d845cecc8>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.<Run>b__13_0 (<>f__AnonymousType12`1[<Input>j__TPar] self) [0x0089e] in <8f1de655bd1240739a78684d845cecc8>:0 
  at CoCoL.AutomationExtensions.RunTask[T] (T channels, System.Func`2[T,TResult] method, System.Boolean catchRetiredExceptions) [0x000d5] in <9a758ff4db6c48d6b3d4d0e5c2adf6d1>:0 
   --- End of inner exception stack trace ---
---> (Inner Exception #0) System.Net.Sockets.SocketException (0x80004005): The socket has been shut down
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.<Run>b__13_0 (<>f__AnonymousType12`1[<Input>j__TPar] self) [0x00673] in <8f1de655bd1240739a78684d845cecc8>:0 
  at Duplicati.Library.Main.Operation.Backup.BackendUploader.<Run>b__13_0 (<>f__AnonymousType12`1[<Input>j__TPar] self) [0x0089e] in <8f1de655bd1240739a78684d845cecc8>:0 
  at CoCoL.AutomationExtensions.RunTask[T] (T channels, System.Func`2[T,TResult] method, System.Boolean catchRetiredExceptions) [0x000d5] in <9a758ff4db6c48d6b3d4d0e5c2adf6d1>:0 <---
<---
<---
2

Welcome to the forum @csk

If you do a Google search

“SpillCollectorProcess” “VolumeReaderBase” “Could not find file”

you’ll see some other mysterious failures where a file that should be in a temporary file folder isn’t there.
Why is Mac OSX temp. directory in weird path may explain your path. I don’t have a Mac. Had to search.
Assuming this failed near end of backup, it would be trying to consolidate leftovers from parallel threads attempting to fill dblock files to their dblock-size (often set by Remote volume size on Options screen)

Sometimes the problem gets fixed by solving some errors Duplicati sees with the backend. Can you run log-file=<path> and log-file-log-level=retry to at least get some view of whether remote operations failed?
SocketException from BackendUploader looks like it might be one. You can also raise number-of-retries.
If a smaller backup works, or if raising retries as a test helps, look at RetryAttempts. in Complete log.

You can also go as close as possible to default, test with a smaller backup, test Minio version of S3, etc.

image

3

Thank you, @ts678 - much appreciated!

I will work through all the suggested tasks shortly, but the only response I have right now is that this is not near the end of the backup - in fact, if I remember correctly, it was getting stuck on the second file.

I also am pretty much at default; the only setting I remember changing was “Use SSL” in the S3 configuration step (but I also tried running it without SSL, and that failed in the exact same way).

I will work through the other suggested steps soon, and provide an update as soon as I can. Thanks!

4

Hello again,

I got to work on the machine in question again, and the news is both good and bad - good because the backup now works, bad because I don’t know why.

The only thing that changed was an OS update to 11.0.1. All other settings were the same - default everywhere, except checking the “Use SSL” box while configuring S3. The backup predictably failed before, and predictably works after. Unfortunately, pinpointing what about the OS update made it work is well beyond my capabilities.