Upgraded 32bit RaspiOS to Bullseye, backup jobs hang with errors "Failed to load assembly ..... error message: Invalid Image"

Support
1

I needed to update one of my Raspberry Pi4 servers to Bullseye as I needed some newer libraries for the new release of Zabbix 6.2 amongst other things, and now Duplicati (2.0.6.104_canary_2022-06-15) fails to go past starting a backup. The whole service starts and I can access the web GUI, but any jobs I start never get past “starting” and after a certain amount of time forces the service to restart and the job to retry.

In the logs when I start the job it shows the following errors:

{"ClassName":"System.BadImageFormatException","Message":"Invalid Image","Data":null,"InnerException":null,"HelpURL":null,"StackTraceString":"  at (wrapper managed-to-native) 
System.Reflection.Assembly.LoadFile_internal(string,System.Threading.StackCrawlMark&)\n  at System.Reflection.Assembly.LoadFile (System.String path, System.Security.Policy.Evidence 
securityEvidence) [0x00033] in <12b418a7818c4ca0893feeaaf67f1e7f>:0 \n  at System.Reflection.Assembly.LoadFile (System.String path) [0x00000] in <12b418a7818c4ca0893feeaaf67f1e7f>:0 \n  
at 
Duplicati.Library.DynamicLoader.DynamicLoader`1[T].FindInterfaceImplementors (System.String[] additionalfolders) [0x00085] in <0f2eba409abd4b7ba27aae5e0bea8106>:0 
","RemoteStackTraceString":null,"RemoteStackIndex":0,"ExceptionMethod":null,"HResult":-2147024885,"Source":null,"BadImageFormat_FileName":"/usr/lib/duplicati/System.Globalization.Extensions.dl

l","BadImageFormat_FusionLog":null}

{"ClassName":"System.BadImageFormatException","Message":"Invalid Image","Data":null,"InnerException":null,"HelpURL":null,"StackTraceString":"  at (wrapper managed-to-native) 
System.Reflection.Assembly.LoadFile_internal(string,System.Threading.StackCrawlMark&)\n  at System.Reflection.Assembly.LoadFile (System.String path, System.Security.Policy.Evidence 
securityEvidence) [0x00033] in <12b418a7818c4ca0893feeaaf67f1e7f>:0 \n  at System.Reflection.Assembly.LoadFile (System.String path) [0x00000] in <12b418a7818c4ca0893feeaaf67f1e7f>:0 \n  
at 
Duplicati.Library.DynamicLoader.DynamicLoader`1[T].FindInterfaceImplementors (System.String[] additionalfolders) [0x00085] in <0f2eba409abd4b7ba27aae5e0bea8106>:0 
","RemoteStackTraceString":null,"RemoteStackIndex":0,"ExceptionMethod":null,"HResult":-2147024885,"Source":null,"BadImageFormat_FileName":"/usr/lib/duplicati/System.Net.Http.dll","BadImageForm

at_FusionLog":null}

{"ClassName":"System.BadImageFormatException","Message":"Invalid Image","Data":null,"InnerException":null,"HelpURL":null,"StackTraceString":"  at (wrapper managed-to-native) 
System.Reflection.Assembly.LoadFile_internal(string,System.Threading.StackCrawlMark&)\n  at System.Reflection.Assembly.LoadFile (System.String path, System.Security.Policy.Evidence 
securityEvidence) [0x00033] in <12b418a7818c4ca0893feeaaf67f1e7f>:0 \n  at System.Reflection.Assembly.LoadFile (System.String path) [0x00000] in <12b418a7818c4ca0893feeaaf67f1e7f>:0 \n  
at 
Duplicati.Library.DynamicLoader.DynamicLoader`1[T].FindInterfaceImplementors (System.String[] additionalfolders) [0x00085] in <0f2eba409abd4b7ba27aae5e0bea8106>:0 
","RemoteStackTraceString":null,"RemoteStackIndex":0,"ExceptionMethod":null,"HResult":-2147024885,"Source":null,"BadImageFormat_FileName":"/usr/lib/duplicati/System.Threading.Overlapped.dll","

BadImageFormat_FusionLog":null}

{"ClassName":"System.BadImageFormatException","Message":"Invalid Image","Data":null,"InnerException":null,"HelpURL":null,"StackTraceString":"  at (wrapper managed-to-native) 
System.Reflection.Assembly.LoadFile_internal(string,System.Threading.StackCrawlMark&)\n  at System.Reflection.Assembly.LoadFile (System.String path, System.Security.Policy.Evidence 
securityEvidence) [0x00033] in <12b418a7818c4ca0893feeaaf67f1e7f>:0 \n  at System.Reflection.Assembly.LoadFile (System.String path) [0x00000] in <12b418a7818c4ca0893feeaaf67f1e7f>:0 \n  
at 
Duplicati.Library.DynamicLoader.DynamicLoader`1[T].FindInterfaceImplementors (System.String[] additionalfolders) [0x00085] in <0f2eba409abd4b7ba27aae5e0bea8106>:0 
","RemoteStackTraceString":null,"RemoteStackIndex":0,"ExceptionMethod":null,"HResult":-2147024885,"Source":null,"BadImageFormat_FileName":"/usr/lib/duplicati/System.IO.Compression.dll","BadIma

geFormat_FusionLog":null}

I tried uninstalling Duplicati along with cleaning up Mono, then reinstalling but it’s still the same.

Any ideas?

2

Small additional update: while digging around I found that Mono is quite old with RaspiOS, v6.8 vs v6.12 on the Mono site, so I followed their directions for adding their repo for Mono. Afterwards I uninstalled both Duplicati and Mono (mono-devel) from the machine and re-installed both, but it’s still the same:

mono-devel/stable-raspbianbuster,now 6.12.0.182-0xamarin1+raspbian10b1 all [installed,automatic]

And yes I know the above is for “Buster”, but I wanted to try it anyway.

3

I opened a Github issue: Upgraded 32bit RaspiOS to Bullseye, service restarts during backup jobs with errors “Failed to load assembly ….. error message: Invalid Image” · Issue #4787 · duplicati/duplicati · GitHub

4

I don’t have a Pi4 to test this, so I’ll just launch a few ideas: usually when something that works don’t work in some case, it’s always interesting to look at what is special. As I have seen posts reporting success with Pi4, I’d think that it’s not Pi4 itself. Could it be that it’s a 32 bits OS ? PI4 is a 64 (sort of) system, so why did you install a 32 bit OS ? could it be because it’s a 1 Gb Pi4, so you wanted to limit the memory footprint ? If yes, maybe the compilation could get bad because the memory is too tight (and before you upgrading to bullseye, it was working because the apps where less heavy).
A way to test this could be to stop all other applications and test Duplicati having all resources of the system.
Another possible reason could be a bug in 32 bits versions of Mono. After all, 32 bits systems are on the way out so will get less work. And Mono itself is on the way out. Did you search through the hum, 2100 issues of the Mono project ?

5

Because official 64bit did not exist when I got my Pi4 soon after release, even now the default using the Pi Imager is still 32bit, it’s also a 4GB RPI4.

It’s running less resources now than before I upgraded as I needed to remove Unifi before I could upgrade to Bullseye.

I did a general search of Mono project and nothing appears.

6

yeah, first issue I found was this:

uninspiring to say the least.

Now it gives me an idea, could you export the backup as command line and run it from a console window ? maybe it could bring an insight, say crash in a more detailed way :slight_smile:

7

New update, I went ahead and updated another similar RPI4 and afterwards even though everything else was running like before, Duplicati would get into this job loop of completing but crashing the service and therefore restarting the same backup over and over again.

I then remembered I had also tried using the latest build direct from Mono, and after that and rebooting, this time the backup are now working, although the service continues to report the message from the second post. So I did the same i.e. just updating the OS because I had already tried the later Mono, and now it too seems to be fine.

So bascially, I needed to use a more recent Mono v6.12 than the default v6.8 that comes with 32bit RPiOS Bullseye.