# Plans to update docker image?

**URL:** <https://forum.duplicati.com/t/plans-to-update-docker-image/16019>\
**Category:** Support\
**Created:** [March 29, 2023, 6:25pm UTC](https://forum.duplicati.com/t/plans-to-update-docker-image/16019 "2023-03-29T18:25:39Z")\
**Posts on this page:** 18\
**Page:** 1

<div class="post-metadata">

**Author:** ![Bubblesaway](https://forum.duplicati.com/letter_avatar_proxy/v4/letter/b/fbc32d/32.png) [@Bubblesaway](https://forum.duplicati.com/u/Bubblesaway)\
**Post date:** [March 29, 2023, 6:25pm UTC](https://forum.duplicati.com/t/plans-to-update-docker-image/16019/1 "2023-03-29T18:25:39Z")

</div>

Are there any plans on fixing the Docker image’s certificate authentication problems? As documented:

- [TrustFailure error when using Backblaze B2 - #16 by DarkArc](https://forum.duplicati.com/t/trustfailure-error-when-using-backblaze-b2/13488/16)
- [Failed to connect: Error: TrustFailure (Authentication failed, see inner exception.) - #14 by yarcod](https://forum.duplicati.com/t/failed-to-connect-error-trustfailure-authentication-failed-see-inner-exception/13512/14)
- [Http send report errors / duplicati-monitoring - #15 by ts678](https://forum.duplicati.com/t/http-send-report-errors-duplicati-monitoring/13157/15)

the Docker image has a bad/out-of-date certificate that causes any connections to Backblaze B2 to fail. The forum posts above have a solution to this. Is that fix ever going to get merged into the official docker image?

Reason I ask is that I’m trying to install on TrueNAS SCALE using TrueCharts, but have run into this issue. The docker container file-system is read-only, so I can’t perform the above fix. This issue has been [brought up to TrueCharts](https://github.com/truecharts/charts/issues/2317), but they’re response is that the official docker image needs to be fixed. The image hasn’t been touched since 2021.

I should note that I’m referring to the “latest” docker images, not the canary images.

Edit: Looks like there’s a GH issue regarding this as well: ["CERTIFICATE\_VERIFY\_FAILED" on all connection attempts · Issue #4721 · duplicati/duplicati · GitHub](https://github.com/duplicati/duplicati/issues/4721)  
Doesn’t look like anything was done about it though.

---

<div class="post-metadata">

**Author:** ![ts678](https://forum.duplicati.com/letter_avatar_proxy/v4/letter/t/8491ac/32.png) [@ts678](https://forum.duplicati.com/u/ts678)\
**Post date:** [March 29, 2023, 10:32pm UTC](https://forum.duplicati.com/t/plans-to-update-docker-image/16019/2 "2023-03-29T22:32:37Z")

</div>

> [@Bubblesaway](#):
>
> Are there any plans

You are in the process of missing the current planning cycle which gathered some pull requests:

[2023 release planning](https://forum.duplicati.com/t/2023-release-planning/15808) leading to  
[Preview of security and bugfix release Canary 105](https://forum.duplicati.com/t/preview-of-security-and-bugfix-release-canary-105/15925)  
which as of last post is in a build and release issue after a time of testing the combined changes.

> [@Bubblesaway](#):
>
> The docker container file-system is read-only

That seems a bit non-standard. Did you set that up, did you verify it by going in and testing, etc.?  
[Docker Objects](https://docs.docker.com/get-started/overview/#docker-objects) says final layer is read-write. [docker run](https://docs.docker.com/engine/reference/commandline/run/) can set read-only, but may break things.

> [@Bubblesaway](#):
>
> I should note that I’m referring to the “latest” docker images, not the canary images.

Last Beta was [2.0.6.3\_beta\_2021-06-17](https://github.com/duplicati/duplicati/releases/tag/v2.0.6.3-2.0.6.3_beta_2021-06-17), so Docker isn’t being singled out. Volunteers are scarce.  
Do you know Docker well enough to say if [Dockerfile](https://github.com/duplicati/duplicati/blob/master/Installer/Docker/context/Dockerfile) change could do the steps of a [workaround](https://forum.duplicati.com/t/error-with-docker-install/15884/2)?  
It might be possible to squeak a build change in. I don’t know if anyone has tested the Docker yet.

EDIT:

I have asked at the above workaround link to find minimal steps. Alternatively, you could help find.  
You would need to be able to run the image without TrueCharts. Anybody else here willing to test?

After having found what we think is Dockerfile fix, it would be good if a pre-Canary test is possible.  
I’m not sure if Docker build is in the preview linked above. If so, that suggests this path is possible.

---

<div class="post-metadata">

**Author:** ![Bubblesaway](https://forum.duplicati.com/letter_avatar_proxy/v4/letter/b/fbc32d/32.png) [@Bubblesaway](https://forum.duplicati.com/u/Bubblesaway)\
**Post date:** [March 29, 2023, 10:59pm UTC](https://forum.duplicati.com/t/plans-to-update-docker-image/16019/3 "2023-03-29T22:59:40Z")

</div>

> [@ts678](#):
>
> > [@Bubblesaway](#):
> >
> > The docker container file-system is read-only
> 
> That seems a bit non-standard. Did you set that up, did you verify it by going in and testing, etc.?  
> [Docker Objects](https://docs.docker.com/get-started/overview/#docker-objects) says final layer is read-write. [docker run](https://docs.docker.com/engine/reference/commandline/run/) can set read-only, but may break things.

I didn’t set that up, but that seems to be the way that TrueNAS SCALE/TrueCharts/Helm set up the container. My experience is from sshing into the container itself and finding that I can’t do any of the workaround steps due to read-only filesystem, even as root.

> [@ts678](#):
>
> > [@Bubblesaway](#):
> >
> > I should note that I’m referring to the “latest” docker images, not the canary images.
> 
> Last Beta was [2.0.6.3\_beta\_2021-06-17](https://github.com/duplicati/duplicati/releases/tag/v2.0.6.3-2.0.6.3_beta_2021-06-17), so Docker isn’t being singled out. Volunteers are scarce.

Putting quotes around “latest” was to specify the `latest` tag in Docker, not as a backhanded complaint about how old it is. I get how open-source projects can be, and while I can’t contribute much (if any) in time or skills, I do donate to the project.

> [@ts678](#):
>
> Do you know Docker well enough to say if [Dockerfile](https://github.com/duplicati/duplicati/blob/master/Installer/Docker/context/Dockerfile) change could do the steps of a [workaround](https://forum.duplicati.com/t/error-with-docker-install/15884/2)?  
> It might be possible to squeak a build change in. I don’t know if anyone has tested the Docker yet.

Unfortunately my knowledge of Docker is limited to mostly conceptual ideas and pattern-matching files. I was actually looking at my migration to TrueNAS SCALE as a “Let’s see how to use containers”. However, in the `RUN` section, I don’t see why running the workaround steps in there wouldn’t solve it.

I was next kind of planning on circumventing the TrueCharts part and running the Docker image straight, so I might be able to play around at that stage and see what I can test with (assuming I can get that running…).

---

<div class="post-metadata">

**Author:** ![Bubblesaway](https://forum.duplicati.com/letter_avatar_proxy/v4/letter/b/fbc32d/32.png) [@Bubblesaway](https://forum.duplicati.com/u/Bubblesaway)\
**Post date:** [March 29, 2023, 11:05pm UTC](https://forum.duplicati.com/t/plans-to-update-docker-image/16019/4 "2023-03-29T23:05:14Z")

</div>

> [@ts678](#):
>
> I have asked at the above workaround link to find minimal steps. Alternatively, you could help find.  
> You would need to be able to run the image without TrueCharts. Anybody else here willing to test?
> 
> After having found what we think is Dockerfile fix, it would be good if a pre-Canary test is possible.  
> I’m not sure if Docker build is in the preview linked above. If so, that suggests this path is possible.

I think minimal steps can be found in this post: [Http send report errors / duplicati-monitoring - #16 by hairlesshobo](https://forum.duplicati.com/t/http-send-report-errors-duplicati-monitoring/13157/16)

> [@Http send report errors / duplicati-monitoring](https://forum.duplicati.com/t/http-send-report-errors-duplicati-monitoring/13157/16):
>
> ```plaintext
> ##!!! use at your own risk.. I do NOT suggest deleting the offending cert 
> ##!!! but instead moving it to another location in case you need to put 
> ##!!! things back the way they were for some reason.
> 
> # update apt
> apt-get update
> 
> # make sure the latest ca certs are installed
> apt-get install ca-certificates
> 
> # remove the offending root cert from the root store
> mv /usr/share/ca-certificates/mozilla/DST_Root_CA_X3.crt ~/DST_Root_CA_X3.crt 
> 
> # update the root ca store.. this automatically updated mono for me.
> update-ca-certificates
> 
> ```

But it might also be the case that moving to a different base OS might fix the issue, since the root problem is “old” certificates.

It might also be the case that this isn’t even a problem on Canary builds, be it by a change to Duplicati’s code (doubtful) or by just an update to the image (my theory is it does work). Again, something I can test (once I figure out how access the Duplicati interface after running the container).

---

<div class="post-metadata">

**Author:** ![ts678](https://forum.duplicati.com/letter_avatar_proxy/v4/letter/t/8491ac/32.png) [@ts678](https://forum.duplicati.com/u/ts678)\
**Post date:** [March 29, 2023, 11:10pm UTC](https://forum.duplicati.com/t/plans-to-update-docker-image/16019/5 "2023-03-29T23:10:21Z")

</div>

> [@Bubblesaway](#):
>
> But it might also be the case that moving to a different base OS might fix the issue

This one sure isn’t going to happen in the last days before release. Maybe someday.  
If you read my workaround link, we now have confirmation of what was true minimal.  
I don’t know why anything `apt` is useful, especially since Debian didn’t fix the cert…  
I don’t run Docker, but I would have thought updating the OS wasn’t always advised.

---

<div class="post-metadata">

**Author:** ![Bubblesaway](https://forum.duplicati.com/letter_avatar_proxy/v4/letter/b/fbc32d/32.png) [@Bubblesaway](https://forum.duplicati.com/u/Bubblesaway)\
**Post date:** [March 29, 2023, 11:21pm UTC](https://forum.duplicati.com/t/plans-to-update-docker-image/16019/6 "2023-03-29T23:21:57Z")

</div>

> [@ts678](#):
>
> This one sure isn’t going to happen in the last days before release. Maybe someday.

Yeah, no worries there. Not expecting anyone to move mountains or anything. Though I hope maybe a little bit sooner than “someday”. Assuming we can get a working fix, I hope next release would make sense.

> [@ts678](#):
>
> I don’t know why anything `apt` is useful, especially since Debian didn’t fix the cert…

Might just be the case of debugging for awhile, finding a solution, but not trying to get the minimal response. Or trying to cover every possible use case in ensuring that the solution is in the correct state. A “don’t try and get too clever” type mindset.

---

<div class="post-metadata">

**Author:** ![ts678](https://forum.duplicati.com/letter_avatar_proxy/v4/letter/t/8491ac/32.png) [@ts678](https://forum.duplicati.com/u/ts678)\
**Post date:** [March 29, 2023, 11:28pm UTC](https://forum.duplicati.com/t/plans-to-update-docker-image/16019/7 "2023-03-29T23:28:58Z")

</div>

> [@Bubblesaway](#):
>
> Though I hope maybe a little bit sooner than “someday”.

For your original situation that needs a Beta, those have never been very frequent.  
The lack of developer volunteers has been extreme (any out there?), only recently  
improving enough to at least pull in some existing PRs, hopefully leading to a Beta.  
On that note, let me see what @gpatel-fr thinks of a Docker build change or a test.

---

<div class="post-metadata">

**Author:** ![Bubblesaway](https://forum.duplicati.com/letter_avatar_proxy/v4/letter/b/fbc32d/32.png) [@Bubblesaway](https://forum.duplicati.com/u/Bubblesaway)\
**Post date:** [March 30, 2023, 2:28pm UTC](https://forum.duplicati.com/t/plans-to-update-docker-image/16019/8 "2023-03-30T14:28:38Z")

</div>

Working on TrueNAS SCALE, I can confirm that doing:

```bash
root@ix-duplicati-docker-ix-chart-c7f4cdf5d-gfs8g:/$ rm /usr/share/ca-certificates/mozilla/DST_Root_CA_X3.crt
root@ix-duplicati-docker-ix-chart-c7f4cdf5d-gfs8g:/$ update-ca-certificates
Updating certificates in /etc/ssl/certs...
W: /usr/share/ca-certificates/mozilla/DST_Root_CA_X3.crt not found, but listed in /etc/ca-certificates.conf.
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
Updating Mono key store
Mono Certificate Store Sync - version 6.12.0.122
Populate Mono certificate store from a concatenated list of certificates.
Copyright 2002, 2003 Motus Technologies. Copyright 2004-2008 Novell. BSD licensed.

Importing into legacy system store:
I already trust 137, your new list has 136
1 previously trusted certificates were removed.
Certificate removed: O=Digital Signature Trust Co., CN=DST Root CA X3
Import process completed.

Importing into BTLS system store:
I already trust 137, your new list has 136
1 previously trusted certificates were removed.
Certificate removed: O=Digital Signature Trust Co., CN=DST Root CA X3
Import process completed.
Done
done.

```

does remove the TrustCertification error.

Following [this post](https://forum.duplicati.com/t/trustfailure-error-when-using-backblaze-b2/13488/10), this can be easily confirmed from command-line via:

```bash
root@ix-duplicati-docker-ix-chart-c7f4cdf5d-gfs8g:/$ csharp -e 'new System.Net.WebClient ().DownloadString ("https://api.backblazeb2.com")'
"<!DOCTYPE html>
<html lang="en">
<head>

<script src="https://cdn.cookielaw.org/scripttemplates/otSDKStub.js" type="text/javascript" charset="UTF-8" data-domain-script="c2b991fa-af6b-41eb-a5e8-4d9878afe4d8"></script>
<style type="text/css">
[.........]

```

The above is old news, but I’ve confirmed this is the case for both the beta/latest and canary docker images. I’ll now try and see if I can make changes to the `Dockerfile` to automatically fix the issue in build.

---

<div class="post-metadata">

**Author:** ![gpatel-fr](https://forum.duplicati.com/letter_avatar_proxy/v4/letter/g/7993a0/32.png) [@gpatel-fr](https://forum.duplicati.com/u/gpatel-fr)\
**Post date:** [March 30, 2023, 3:21pm UTC](https://forum.duplicati.com/t/plans-to-update-docker-image/16019/9 "2023-03-30T15:21:52Z")

</div>

> [@ts678](#):
>
> Docker build change

I am in no way a Docker expert, what I know is that when I manage to get a working canary install in the hands of the project owner, the build installer script should build and push a Docker image to Docker hub (it’s intended as such at least).

I have downloaded the latest mono image (6-slim) and ‘openssl s\_client -connect [api.backblazeb2.com:443](http://api.backblazeb2.com:443)’ seems to work all right inside the container, so it should be good (If I manage to get the installer working before Buster going obsolete, or ChatGptX achieving the singularity, whichever comes first)

---

<div class="post-metadata">

**Author:** ![Bubblesaway](https://forum.duplicati.com/letter_avatar_proxy/v4/letter/b/fbc32d/32.png) [@Bubblesaway](https://forum.duplicati.com/u/Bubblesaway)\
**Post date:** [March 30, 2023, 3:44pm UTC](https://forum.duplicati.com/t/plans-to-update-docker-image/16019/10 "2023-03-30T15:44:25Z")

</div>

I’m even less of a SSL/security expert than I am a Docker expert, but I think the “root” issue is that, yes `openssl` works, but [mono uses `BoringSSL`](https://forum.duplicati.com/t/trustfailure-error-when-using-backblaze-b2/13488/5).

Can you try `csharp -e 'new System.Net.WebClient ().DownloadString ("https://api.backblazeb2.com")'` in your test? Seems like that’s the way of testing how Duplicati interfaces with backblaze.

---

<div class="post-metadata">

**Author:** ![ts678](https://forum.duplicati.com/letter_avatar_proxy/v4/letter/t/8491ac/32.png) [@ts678](https://forum.duplicati.com/u/ts678)\
**Post date:** [March 30, 2023, 4:03pm UTC](https://forum.duplicati.com/t/plans-to-update-docker-image/16019/11 "2023-03-30T16:03:53Z")

</div>

> [@Bubblesaway](#):
>
> yes `openssl` works

It depends on the version. Buster has OpenSSL 1.1.1n according to repology. Stretch has 1.0.2u.

[Old Let’s Encrypt Root Certificate Expiration and OpenSSL 1.0.2](https://www.openssl.org/blog/blog/2021/09/13/LetsEncryptRootCertExpire/) (OpenSSL Blog explains issue)

> [@gpatel-fr](#):
>
> seems to work all right inside the container

You can directly check the openssl version (which might explain why it works) and check certificate.  
The suggested test with `csharp` would be best. I think that command was in Docker, at one point…

---

<div class="post-metadata">

**Author:** ![Bubblesaway](https://forum.duplicati.com/letter_avatar_proxy/v4/letter/b/fbc32d/32.png) [@Bubblesaway](https://forum.duplicati.com/u/Bubblesaway)\
**Post date:** [March 30, 2023, 4:09pm UTC](https://forum.duplicati.com/t/plans-to-update-docker-image/16019/12 "2023-03-30T16:09:32Z")

</div>

Success! Applying the following change to `context/Dockerfile`:

```plaintext
14c14,16
- cert-sync /etc/ssl/certs/ca-certificates.crt
---
+ cert-sync /etc/ssl/certs/ca-certificates.crt && \
+ rm /usr/share/ca-certificates/mozilla/DST_Root_CA_X3.crt && \
+ update-ca-certificates

```

has backblaze connecting successfully in TrueNAS SCALE (and presumably elsewhere). That was the only change needed as well.

Edit: Note I did test a build of the Docker container without any modifications to verify the behavior before implementing the above fix.

---

<div class="post-metadata">

**Author:** ![Bubblesaway](https://forum.duplicati.com/letter_avatar_proxy/v4/letter/b/fbc32d/32.png) [@Bubblesaway](https://forum.duplicati.com/u/Bubblesaway)\
**Post date:** [March 30, 2023, 4:10pm UTC](https://forum.duplicati.com/t/plans-to-update-docker-image/16019/13 "2023-03-30T16:10:19Z")

</div>

> [@ts678](#):
>
> The suggested test with `csharp` would be best. I think that command was in Docker, at one point…

Yeah, I was able to run the `csharp` command inside the Docker without having to do anything.

---

<div class="post-metadata">

**Author:** ![gpatel-fr](https://forum.duplicati.com/letter_avatar_proxy/v4/letter/g/7993a0/32.png) [@gpatel-fr](https://forum.duplicati.com/u/gpatel-fr)\
**Post date:** [March 30, 2023, 4:20pm UTC](https://forum.duplicati.com/t/plans-to-update-docker-image/16019/14 "2023-03-30T16:20:38Z")

</div>

> [@Bubblesaway](#):
>
> backblaze connecting successfully

Yes, that’s it. Only ChatGpt could explain why deleting an obsolete certificate can make a connection succeed but it’s as you say.

---

<div class="post-metadata">

**Author:** ![ts678](https://forum.duplicati.com/letter_avatar_proxy/v4/letter/t/8491ac/32.png) [@ts678](https://forum.duplicati.com/u/ts678)\
**Post date:** [March 30, 2023, 4:22pm UTC](https://forum.duplicati.com/t/plans-to-update-docker-image/16019/15 "2023-03-30T16:22:55Z")

</div>

The theory is that Debian Buster feels no pressing need to fix the cert because it doesn’t hurt them.  
The question is whether Mono Project (who got hurt) has slid a cert fix into their image. It’d be nice.  
OTOH ever since Microsoft bought them, their attention has been on `.NET`. Sad, but not surprising.

If it turns out we have to do our own two RUN lines, maybe do them right after we take their image?  
Our Dockerfile seems to use different sections for different things, which is why I’m suggesting that.  
Lines that begin with a # are comments, if it seems reasonable to put something in on the change…

---

<div class="post-metadata">

**Author:** ![Bubblesaway](https://forum.duplicati.com/letter_avatar_proxy/v4/letter/b/fbc32d/32.png) [@Bubblesaway](https://forum.duplicati.com/u/Bubblesaway)\
**Post date:** [March 30, 2023, 5:03pm UTC](https://forum.duplicati.com/t/plans-to-update-docker-image/16019/16 "2023-03-30T17:03:42Z")

</div>

> [@gpatel-fr](#):
>
> Only ChatGpt could explain why deleting an obsolete certificate can make a connection succeed but it’s as you say.

[This post](https://forum.duplicati.com/t/http-send-report-errors-duplicati-monitoring/13157/19) is talking about [duplicati-monitoring.com](http://duplicati-monitoring.com), but they determined that the certificate is in the chain of certification somewhere. I think we could go through the same process if we really felt like it for backblaze.

---

<div class="post-metadata">

**Author:** ![Bubblesaway](https://forum.duplicati.com/letter_avatar_proxy/v4/letter/b/fbc32d/32.png) [@Bubblesaway](https://forum.duplicati.com/u/Bubblesaway)\
**Post date:** [March 30, 2023, 5:06pm UTC](https://forum.duplicati.com/t/plans-to-update-docker-image/16019/17 "2023-03-30T17:06:00Z")

</div>

> [@ts678](#):
>
> Lines that begin with a # are comments, if it seems reasonable to put something in on the change…

I think that’d be wise. Otherwise those who don’t know would be confused AF as to why we’re deleting this one seemingly random certificate.

---

<div class="post-metadata">

**Author:** ![ts678](https://forum.duplicati.com/letter_avatar_proxy/v4/letter/t/8491ac/32.png) [@ts678](https://forum.duplicati.com/u/ts678)\
**Post date:** [March 30, 2023, 5:37pm UTC](https://forum.duplicati.com/t/plans-to-update-docker-image/16019/18 "2023-03-30T17:37:57Z")

</div>

The OpenSSL blog explains how the failure works. Let’s Encrypt also has a nice summary of the issue.

[Production Chain Changes](https://community.letsencrypt.org/t/production-chain-changes/150739) explains and links to how they saved old Android by chaining to an expired certificate – because Android doesn’t mind and it actually has some standards support behind the idea.

There are two versions of `ISRG Root X1`, but its default uses the Android-supporting chain-to- expired.

There’s an alternative chain with self-signed `ISRG Root X1` that relies on that being installed on the OS. While waiting for [everyone to catch up](https://letsencrypt.org/docs/certificate-compatibility/), they save Android and break mono or OpenSSL 1.0.2 which was initial BoringSSL code source, but not the current one. Mono won’t upgrade, so still breaks on old cert…

Removing the old cert forces the code to realize it actually has the cert in its trust store, so it then works.

OpenSSL explains that OpenSSL 1.0.2 “always prefers the untrusted chain”, following it to expired end.

EDIT to add an example as shown by an openssl:

```plaintext
Certificate chain
 0 s:CN = backblazeb2.com
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
 2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   i:O = Digital Signature Trust Co., CN = DST Root CA X3

```

EDIT 2:

[Google BoringSSL commit](https://boringssl.googlesource.com/boringssl/+/8f5eb80b810ff63d14ad3535cb16f7cb8271a4f5) also explains the multiple paths, and how one can prefer the trusted path.  
[Enable X509\_V\_FLAG\_TRUSTED\_FIRST flag in BoringSSL #21233](https://github.com/mono/mono/issues/21233) is Mono team declining to do it.  
[https://valid-isrgrootx1.letsencrypt.org/](https://valid-isrgrootx1.letsencrypt.org/) is a test site for the self-signed `ISRG Root X1`, that looks like:

```plaintext
Certificate chain
 0 s:CN = valid-isrgrootx1.letsencrypt.org
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1

```

Sometimes people call this one the short chain. After the cross-signed cert expires in 2024, maybe this will be what’s used. If so, old Android stops working, and mono/Duplicati may start working once again.