Suddenly started crashing - mono-sgen pthread_mutex_lock.c

Support
21

Not much in common. Can anybody comment if these could be the culprit? I would doubt the rpm items.

    Upgrade  koji-1.29.1-1.fc35.noarch                               @updates		    Upgrade       koji-1.29.1-3.fc35.noarch                                       @updates
    Upgrade  python3-koji-1.29.1-1.fc35.noarch                       @updates		    Upgrade       python3-koji-1.29.1-3.fc35.noarch                               @updates
    Upgrade  python3-rpm-4.17.1-2.fc35.x86_64                        @updates		    Upgrade       python3-rpm-4.17.1-3.fc35.x86_64                                @updates
    Upgrade  rpm-4.17.1-2.fc35.x86_64                                @updates		    Upgrade       rpm-4.17.1-3.fc35.x86_64                                        @updates
    Upgrade  rpm-build-4.17.1-2.fc35.x86_64                          @updates		    Upgrade       rpm-build-4.17.1-3.fc35.x86_64                                  @updates
    Upgrade  rpm-build-libs-4.17.1-2.fc35.x86_64                     @updates		    Upgrade       rpm-build-libs-4.17.1-3.fc35.x86_64                             @updates
    Upgrade  rpm-libs-4.17.1-2.fc35.x86_64                           @updates		    Upgrade       rpm-libs-4.17.1-3.fc35.x86_64                                   @updates
    Upgrade  rpm-plugin-selinux-4.17.1-2.fc35.x86_64                 @updates		    Upgrade       rpm-plugin-selinux-4.17.1-3.fc35.x86_64                         @updates
    Upgrade  rpm-plugin-systemd-inhibit-4.17.1-2.fc35.x86_64         @updates		    Upgrade       rpm-plugin-systemd-inhibit-4.17.1-3.fc35.x86_64                 @updates
    Upgrade  rpm-sign-libs-4.17.1-2.fc35.x86_64                      @updates		    Upgrade       rpm-sign-libs-4.17.1-3.fc35.x86_64                              @updates
22

all these packages are related to rpm - koji is the Fedora rpm building utility. It should not have any impact on Duplicati. Normally. Well, given that one of these packages is a selinux plugin, I’d say that there is a caveat here. I have seen report of bad or badly configured selinux plugins having unfortunate side effects even in permissive mode.
Anyway, the mere presence of these packages on your system is a mark of customized setups. These systems are not out-of-the-box distro Linux - and as such could exhibit very particular behaviour, difficult to debug generally. If this kind of problem don’t happen on a developer’s box, the only chance of solving it could be by trying to uninstall things more or less randomly until it works.

23

Using coredumpctl -1 info, I see getpwuid in only the following:

                Module libcom_err.so.2 with build-id c980b4303c51332b16b98f799a158445eae81aa0
                Stack trace of thread 269982:
                #0  0x00007fcfe148ec4c __pthread_kill_implementation (libc.so.6 + 0x8ec4c)
                #1  0x00007fcfe143e9c6 raise (libc.so.6 + 0x3e9c6)
                #2  0x00007fcfe14287f4 abort (libc.so.6 + 0x287f4)
                #3  0x000056385fa2de0e mono_post_native_crash_handler (mono-sgen + 0x2de0e)
                #4  0x000056385fa6b923 mono_handle_native_crash (mono-sgen + 0x6b923)
                #5  0x000056385fabcbeb sigabrt_signal_handler (mono-sgen + 0xbcbeb)
                #6  0x00007fcfe143ea70 __restore_rt (libc.so.6 + 0x3ea70)
                #7  0x00007fcfe148ec4c __pthread_kill_implementation (libc.so.6 + 0x8ec4c)
                #8  0x00007fcfe143e9c6 raise (libc.so.6 + 0x3e9c6)
                #9  0x00007fcfe14287f4 abort (libc.so.6 + 0x287f4)
                #10 0x00007fcfe142871b __assert_fail_base.cold (libc.so.6 + 0x2871b)
                #11 0x00007fcfe1437576 __assert_fail (libc.so.6 + 0x37576)
                #12 0x00007fcfe14902b0 __pthread_mutex_lock@GLIBC_2.2.5 (libc.so.6 + 0x902b0)
                #13 0x00007fcfe1bd8736 sss_nss_mc_get_ctx (libnss_sss.so.2 + 0x3736)
                #14 0x00007fcfe1bd91fe sss_nss_mc_getpwuid (libnss_sss.so.2 + 0x41fe)
                #15 0x00007fcfe1bdb036 _nss_sss_getpwuid_r (libnss_sss.so.2 + 0x6036)
                #16 0x00007fcfe14dc631 getpwuid_r@@GLIBC_2.2.5 (libc.so.6 + 0xdc631)
                #17 0x00007fcfd541b905 Mono_Posix_Syscall_getpwuid_r (libMonoPosixHelper.so + 0x1b905)
                #18 0x00000000413c5c79 n/a (n/a + 0x0)
                #19 0x00000000413c4418 n/a (n/a + 0x0)
                #20 0x0000000041500fab n/a (n/a + 0x0)
                #21 0x00000000414fffa0 n/a (n/a + 0x0)
                #22 0x00000000414ff3bc n/a (n/a + 0x0)
                #23 0x00000000414fef73 n/a (n/a + 0x0)
                #24 0x00000000414fca20 n/a (n/a + 0x0)
                #25 0x00007fcfd792b6a1 System_Runtime_CompilerServices_AsyncMethodBuilderCore_MoveNextRunner_InvokeMoveNext_object (mscorlib.dll.so + 0x32b6a1)
                #26 0x00007fcfd7790fca System_Threading_ExecutionContext_RunInternal_System_Threading_ExecutionContext_System_Threading_ContextCallback_object_bool (mscorlib.dll.so + 0x190fca)
                #27 0x00007fcfd7790dd3 System_Threading_ExecutionContext_Run_System_Threading_ExecutionContext_System_Threading_ContextCallback_object_bool (mscorlib.dll.so + 0x190dd3)
                #28 0x00007fcfd792b54a System_Runtime_CompilerServices_AsyncMethodBuilderCore_MoveNextRunner_Run (mscorlib.dll.so + 0x32b54a)
                #29 0x00007fcfd77cdf1b System_Threading_Tasks_AwaitTaskContinuation_RunOrScheduleAction_System_Action_bool_System_Threading_Tasks_Task_ (mscorlib.dll.so + 0x1cdf1b)
                #30 0x00007fcfd77c4fa3 System_Threading_Tasks_Task_FinishContinuations (mscorlib.dll.so + 0x1c4fa3)
                #31 0x00007fcfd77c3541 System_Threading_Tasks_Task_FinishStageThree (mscorlib.dll.so + 0x1c3541)
                #32 0x00007fcfd77b79e6 System_Threading_Tasks_Task_1_TResult_REF_TrySetResult_TResult_REF (mscorlib.dll.so + 0x1b79e6)
                #33 0x00007fcfd7929ca3 System_Runtime_CompilerServices_AsyncTaskMethodBuilder_1_TResult_REF_SetResult_TResult_REF (mscorlib.dll.so + 0x329ca3)
                #34 0x00000000411cc0db n/a (n/a + 0x0)
                #35 0x00007fcfd792b6a1 System_Runtime_CompilerServices_AsyncMethodBuilderCore_MoveNextRunner_InvokeMoveNext_object (mscorlib.dll.so + 0x32b6a1)
                #36 0x00007fcfd7790fca System_Threading_ExecutionContext_RunInternal_System_Threading_ExecutionContext_System_Threading_ContextCallback_object_bool (mscorlib.dll.so + 0x190fca)
                #37 0x00007fcfd7790dd3 System_Threading_ExecutionContext_Run_System_Threading_ExecutionContext_System_Threading_ContextCallback_object_bool (mscorlib.dll.so + 0x190dd3)
                #38 0x00007fcfd792b54a System_Runtime_CompilerServices_AsyncMethodBuilderCore_MoveNextRunner_Run (mscorlib.dll.so + 0x32b54a)
                #39 0x00007fcfd77cdf1b System_Threading_Tasks_AwaitTaskContinuation_RunOrScheduleAction_System_Action_bool_System_Threading_Tasks_Task_ (mscorlib.dll.so + 0x1cdf1b)
                #40 0x00007fcfd77c4fa3 System_Threading_Tasks_Task_FinishContinuations (mscorlib.dll.so + 0x1c4fa3)
                #41 0x00007fcfd77c3541 System_Threading_Tasks_Task_FinishStageThree (mscorlib.dll.so + 0x1c3541)
                #42 0x00007fcfd77b79e6 System_Threading_Tasks_Task_1_TResult_REF_TrySetResult_TResult_REF (mscorlib.dll.so + 0x1b79e6)
                #43 0x00007fcfd77a4b0f System_Threading_Tasks_TaskCompletionSource_1_TResult_REF_TrySetResult_TResult_REF (mscorlib.dll.so + 0x1a4b0f)
                #44 0x00007fcfd77a4b5f System_Threading_Tasks_TaskCompletionSource_1_TResult_REF_SetResult_TResult_REF (mscorlib.dll.so + 0x1a4b5f)
                #45 0x0000000041306593 n/a (n/a + 0x0)
                #46 0x00007fcfd7790dd3 System_Threading_ExecutionContext_Run_System_Threading_ExecutionContext_System_Threading_ContextCallback_object_bool (mscorlib.dll.so + 0x190dd3)
                #47 0x00007fcfd7799738 System_Threading_QueueUserWorkItemCallback_System_Threading_IThreadPoolWorkItem_ExecuteWorkItem (mscorlib.dll.so + 0x199738)
                #48 0x00007fcfd77978ba System_Threading_ThreadPoolWorkQueue_Dispatch (mscorlib.dll.so + 0x1978ba)
                #49 0x00007fcfd779955d System_Threading__ThreadPoolWaitCallback_PerformWaitCallback (mscorlib.dll.so + 0x19955d)
                #50 0x000000004123e137 n/a (n/a + 0x0)
                #51 0x000056385fa3850a mono_jit_runtime_invoke (mono-sgen + 0x3850a)
                #52 0x000056385fc3b06c do_runtime_invoke (mono-sgen + 0x23b06c)
                #53 0x000056385fc68aa9 try_invoke_perform_wait_callback (mono-sgen + 0x268aa9)
                #54 0x000056385fcb9dcb worker_thread (mono-sgen + 0x2b9dcb)
                #55 0x000056385fc6793b start_wrapper_internal (mono-sgen + 0x26793b)
                #56 0x00007fcfe148ce2d start_thread (libc.so.6 + 0x8ce2d)
                #57 0x00007fcfe15121b0 __clone3 (libc.so.6 + 0x1121b0)
24

Can we escalate this to somebody who can read coredumps or stack traces? We do not seem to be getting any traction here, and I have already been a week without a backup.

25

Any chance Python 2 was removed? The updated to Python 3 occurred for both users and the timing fits.

Possibly similar (but certainly a different manifestation) to what has happened with MacOS after 12.3.

@jimkd1yv your issue has been escalated to the top of the forms list (for the moment), which is as good as it gets BTW.

26

Nope, both still on my Fedora 36 box:

# ls -l /usr/bin/python[23]
lrwxrwxrwx. 1 root root  9 Jun 17 04:38 /usr/bin/python2 -> python2.7
lrwxrwxrwx. 1 root root 10 Aug  3 13:20 /usr/bin/python3 -> python3.10

I also booted to single user mode, unmounted my user folders and forced a disk check on each of them. The very next backup attempt failed within seconds.

27

Also still on my Fedora 35 systems

ls -l /usr/bin/python[23]
lrwxrwxrwx. 1 root root 9 Jun 16 21:54 /usr/bin/python2 → python2.7
lrwxrwxrwx. 1 root root 10 Jun 9 20:07 /usr/bin/python3 → python3.10

28

This does not give any new information.

What could (possibly) be more interesting is to examine the file accessed when the crash happens, and if there are concurrent access in another thread. If you are unable to do that yourself, there remains the possibility of exposing the core file to people that could (maybe) find something more, but it would also compromise all your Duplicati secrets. It’s awkward to do that without the expert having a proper contract.

Anyway, the test that you did using a VM seems to show that when using a clean OS install, the problem does not occur. I am unclear if you used a clean new target or not (you should have created a new target)

If this clean install in a VM has the same update level than your problematic setup (that is, if you have an up to date VM), this shows that your normal setup has something very special in it that is not compatible with Duplicati. This kind of problem is difficult to solve, and even more remotely.

29

Why are there ten packages listed later in the current topic here compared to big other-topic list?

Above is about 103 Upgraded packages, plus other changes. One that’s particularly interesting is:

sssd-2.7.3-1.fc35.x86_64

because of

SSSD 2.7.3 Release Notes

All SSSD client libraries (nss, pam, etc) won’t serialize requests anymore by default, i.e. requests from multiple threads can be executed in parallel. Old behavior (serialization) can be enabled by setting environment variable “SSS_LOCKFREE” to “NO”.

SSSD Architecture

suggests that client libraries from the project include libnss_sss.so, used by clients such as glibc.

Anybody know SSSD well? Anybody know DNF well? There appear to be ways to undo or rollback.

History Command (dnf.readthedocs.io)

although from what I found from Google, it’s uncertain, and is subject to what old versions are kept.
Basically, what’s the Fedora debug/recovery/workaround when one updates it and it breaks things?

EDIT 1:

I suppose one might try ls -lc /lib64/libnss_sss.so.2 to see if time is near the breakage time.
If that’s not it, maybe dnf fishing can give other suspects that changed around when Duplicati broke.

EDIT 2:

Troubleshooting Basics (sssd.io) shows how to use debug logs and other things. Maybe it can help?
Chasing issues deep below Duplicati (all the way on the far side of mono) will need what we can find.

Ultimately, to get a fix from some other project will need a report to it, and preferably a nice test case, preferably (typically) something direct to make it as obvious as possible how the library had been run.

Regarding backing out changes, I don’t know how well Fedora puts up with that, but repology.org has:

image

Given that 2.7.3 may be very recent, there might be few dependencies on it from other projects so far.
I’m picking on sssd partly because of their 2.7.3 release note, and because that’s the last step shown before pthread_mutex_lock seemingly declared something was wrong (what an assert typically does).

30

@ts678 You could be a carpenter - it seems you’ve hit the nail on the head! :partying_face:
Both the constantly failing Duplicati jobs have succeeded.

I checked my dnf logs, and see that in fact, on 4 August, the various sssd packages had been upgraded:

[root@fedora log]# dnf history info 1600 | grep sss
    Upgrade       libsss_autofs-2.7.3-1.fc35.x86_64                         @updates
    Upgraded      libsss_autofs-2.7.1-2.fc35.x86_64                         @@System
    Upgrade       libsss_certmap-2.7.3-1.fc35.x86_64                        @updates
    Upgraded      libsss_certmap-2.7.1-2.fc35.x86_64                        @@System
    Upgrade       libsss_idmap-2.7.3-1.fc35.x86_64                          @updates
    Upgraded      libsss_idmap-2.7.1-2.fc35.x86_64                          @@System
    Upgrade       libsss_nss_idmap-2.7.3-1.fc35.x86_64                      @updates
    Upgraded      libsss_nss_idmap-2.7.1-2.fc35.x86_64                      @@System
    Upgrade       libsss_sudo-2.7.3-1.fc35.x86_64                           @updates
    Upgraded      libsss_sudo-2.7.1-2.fc35.x86_64                           @@System
    Upgrade       python3-sssdconfig-2.7.3-1.fc35.noarch                    @updates
    Upgraded      python3-sssdconfig-2.7.1-2.fc35.noarch                    @@System
    Upgrade       sssd-2.7.3-1.fc35.x86_64                                  @updates
    Upgraded      sssd-2.7.1-2.fc35.x86_64                                  @@System
    Upgrade       sssd-ad-2.7.3-1.fc35.x86_64                               @updates
    Upgraded      sssd-ad-2.7.1-2.fc35.x86_64                               @@System
    Upgrade       sssd-client-2.7.3-1.fc35.x86_64                           @updates
    Upgraded      sssd-client-2.7.1-2.fc35.x86_64                           @@System
    Upgrade       sssd-common-2.7.3-1.fc35.x86_64                           @updates
    Upgraded      sssd-common-2.7.1-2.fc35.x86_64                           @@System
    Upgrade       sssd-common-pac-2.7.3-1.fc35.x86_64                       @updates
    Upgraded      sssd-common-pac-2.7.1-2.fc35.x86_64                       @@System
    Upgrade       sssd-idp-2.7.3-1.fc35.x86_64                              @updates
    Upgraded      sssd-idp-2.7.1-2.fc35.x86_64                              @@System
    Upgrade       sssd-ipa-2.7.3-1.fc35.x86_64                              @updates
    Upgraded      sssd-ipa-2.7.1-2.fc35.x86_64                              @@System
    Upgrade       sssd-krb5-2.7.3-1.fc35.x86_64                             @updates
    Upgraded      sssd-krb5-2.7.1-2.fc35.x86_64                             @@System
    Upgrade       sssd-krb5-common-2.7.3-1.fc35.x86_64                      @updates
    Upgraded      sssd-krb5-common-2.7.1-2.fc35.x86_64                      @@System
    Upgrade       sssd-ldap-2.7.3-1.fc35.x86_64                             @updates
    Upgraded      sssd-ldap-2.7.1-2.fc35.x86_64                             @@System
    Upgrade       sssd-nfs-idmap-2.7.3-1.fc35.x86_64                        @updates
    Upgraded      sssd-nfs-idmap-2.7.1-2.fc35.x86_64                        @@System
    Upgrade       sssd-proxy-2.7.3-1.fc35.x86_64                            @updates
    Upgraded      sssd-proxy-2.7.1-2.fc35.x86_64                            @@System
[root@fedora log]#

You’ll notice those were Fedora 35 packages, and in trying to fix this lot I’d upgraded to Fedora 36, which hadn’t helped. My test VM ran Fedora 36, which I first didn’t patch, then later patched, and it continued working - BUT it was using SSHFS to mount the folders I’m backing up via SSH, so I guess the file access mechanism was different. In my real working environment, the source files are all local filesystem components.

So based on the questions from @ts678 I ran the following (I had tried it with glibc, but not with sssd … so close): dnf downgrade sssd-client
(I had worked out the specific package name from rpm -q --whatprovides /lib64/libnss_sss.so.2 )
Here is what changed:

[root@fedora log]# dnf history info 1624
Transaction ID : 1624
Begin time     : Mon 22 Aug 2022 04:17:20 PM SAST
Begin rpmdb    : fb8905094e07a76b4a7bb433ad4bcbe8ee06c8ebf2b41752e89a9bbadf7cc5ec
End time       : Mon 22 Aug 2022 04:17:27 PM SAST (7 seconds)
End rpmdb      : 0977187d56015941f4c0ffc3516b21db28df35d29c05ce810cad1a9ef5a03bf3
User           : Louis van Dyk <louis>
Return-Code    : Success
Releasever     : 36
Command Line   : downgrade sssd-client
Comment        : 
Packages Altered:
    Downgrade  libipa_hbac-2.7.0-1.fc36.x86_64      @fedora
    Downgraded libipa_hbac-2.7.3-1.fc36.x86_64      @@System
    Downgrade  libsss_autofs-2.7.0-1.fc36.x86_64    @fedora
    Downgraded libsss_autofs-2.7.3-1.fc36.x86_64    @@System
    Downgrade  libsss_certmap-2.7.0-1.fc36.x86_64   @fedora
    Downgraded libsss_certmap-2.7.3-1.fc36.x86_64   @@System
    Downgrade  libsss_idmap-2.7.0-1.fc36.x86_64     @fedora
    Downgraded libsss_idmap-2.7.3-1.fc36.x86_64     @@System
    Downgrade  libsss_nss_idmap-2.7.0-1.fc36.x86_64 @fedora
    Downgraded libsss_nss_idmap-2.7.3-1.fc36.x86_64 @@System
    Downgrade  libsss_sudo-2.7.0-1.fc36.x86_64      @fedora
    Downgraded libsss_sudo-2.7.3-1.fc36.x86_64      @@System
    Downgrade  sssd-2.7.0-1.fc36.x86_64             @fedora
    Downgraded sssd-2.7.3-1.fc36.x86_64             @@System
    Downgrade  sssd-ad-2.7.0-1.fc36.x86_64          @fedora
    Downgraded sssd-ad-2.7.3-1.fc36.x86_64          @@System
    Downgrade  sssd-client-2.7.0-1.fc36.x86_64      @fedora
    Downgraded sssd-client-2.7.3-1.fc36.x86_64      @@System
    Downgrade  sssd-common-2.7.0-1.fc36.x86_64      @fedora
    Downgraded sssd-common-2.7.3-1.fc36.x86_64      @@System
    Downgrade  sssd-common-pac-2.7.0-1.fc36.x86_64  @fedora
    Downgraded sssd-common-pac-2.7.3-1.fc36.x86_64  @@System
    Downgrade  sssd-idp-2.7.0-1.fc36.x86_64         @fedora
    Downgraded sssd-idp-2.7.3-1.fc36.x86_64         @@System
    Downgrade  sssd-ipa-2.7.0-1.fc36.x86_64         @fedora
    Downgraded sssd-ipa-2.7.3-1.fc36.x86_64         @@System
    Downgrade  sssd-krb5-2.7.0-1.fc36.x86_64        @fedora
    Downgraded sssd-krb5-2.7.3-1.fc36.x86_64        @@System
    Downgrade  sssd-krb5-common-2.7.0-1.fc36.x86_64 @fedora
    Downgraded sssd-krb5-common-2.7.3-1.fc36.x86_64 @@System
    Downgrade  sssd-ldap-2.7.0-1.fc36.x86_64        @fedora
    Downgraded sssd-ldap-2.7.3-1.fc36.x86_64        @@System
    Downgrade  sssd-nfs-idmap-2.7.0-1.fc36.x86_64   @fedora
    Downgraded sssd-nfs-idmap-2.7.3-1.fc36.x86_64   @@System
    Downgrade  sssd-proxy-2.7.0-1.fc36.x86_64       @fedora
    Downgraded sssd-proxy-2.7.3-1.fc36.x86_64       @@System
[root@fedora log]#

@jimkd1yv could you also run dnf downgrade sssd-client and see if it fixes your problem?

So assuming this is the cause of it, where do we go from here? Do I log this as a bug with Fedora?

31

I don’t know your distro, but if it’s not a rolling release, it’s supposed to not introduce incompatible behaviour for a given release, so if you can confirm that the library is the culprit, yes IMO you could open a bug.
To remove the necessity to hold packages, you could maybe try to set an environment variable in the systemd job, for example like this:

sudo systemctl edit duplicati

[Service]
Environment="SSS_LOCKFREE=NO”
32

Thanks for that. I didn’t know systemd well enough to try the workaround from the sssd release note. Determining whether or not it works would perhaps lead us to the easiest possible Duplicati way out.

It would also provide more weight (unless someone feels up to writing a test case in C or something) wherever this issue goes to. I would assume that Fedora has an interest in knowing what they broke, however ultimately they may need to take it to the sssd project. They might be used to such things…
I don’t know what their policy is, but if you’re a Linux distro, you probably get reports on bits you ship.

Maybe you’re all set up for that? If so, it seems like it would be worth at least making them aware of it.

Reporting bugs) is how sssd.io prefers their reports. It’d be best to know if environment variable helps. Presumably they put it there because they were trying to give people a way out if change broke things.

Now that we’re on the way to understanding, issue or general Google searches might also find things.

Thanks! I was trying to do searches earlier, making my way through the layers. Finally found sssd clue.

33

I indeed do have ls -lc /lib64/libnss_sss.so.2, and it’s date is near the breakage time.
However, when I ran stat on this file, it had not been accessed at all for 3 days before the most recent failed backup attempt.

34

Successful workaround:

I took a chance of not having a successful backup for a week.

I upgraded from Fedora 35 to Fedora 36, and now Duplicati works correctly.

Again I ran stat /lib64/libnss_sss.so.2, but it shows no access since the upgrade last night. I have ran 3 duplicati backups today, but not using that file.

35

I logged a bug with Fedora. Mine was a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=2111582
It has already been fixed upstream and will reflect in the repositories in due course.

Thank you to EVERYONE who had input into solving this.

36

Be careful about drawing conclusions from access time unless other files (maybe try libc) update on use.
mount(8) — Linux manual page shows many ways to disable or restrict the update. Some are by default, either the way the OS sets up its mount (mine set relatime for me) or said to be defaulted in the kernel.

Running pmap on the PIDs (fairly easy) might be some proof, but strace on the library path may be best.

# which sleep
/bin/sleep
# ldd /bin/sleep
	linux-vdso.so.1 (0x00007ffdc61ca000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f3d818b4000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f3d81eae000)
# strace -t -y -P /lib/x86_64-linux-gnu/libc.so.6 /bin/sleep 10
strace: Requested path '/lib/x86_64-linux-gnu/libc.so.6' resolved into '/lib/x86_64-linux-gnu/libc-2.27.so'
16:03:37 openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3</lib/x86_64-linux-gnu/libc-2.27.so>
16:03:37 read(3</lib/x86_64-linux-gnu/libc-2.27.so>, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\240\35\2\0\0\0\0\0"..., 832) = 832
16:03:37 fstat(3</lib/x86_64-linux-gnu/libc-2.27.so>, {st_mode=S_IFREG|0755, st_size=2030928, ...}) = 0
16:03:37 mmap(NULL, 4131552, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3</lib/x86_64-linux-gnu/libc-2.27.so>, 0) = 0x7f19b2e23000
16:03:37 mmap(0x7f19b320a000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3</lib/x86_64-linux-gnu/libc-2.27.so>, 0x1e7000) = 0x7f19b320a000
16:03:37 close(3</lib/x86_64-linux-gnu/libc-2.27.so>) = 0
^Z
[1]+  Stopped                 strace -t -y -P /lib/x86_64-linux-gnu/libc.so.6 /bin/sleep 10
# ps | grep sleep
 6471 pts/4    00:00:00 sleep
# pmap 6471 | grep libc
00007f19b2e23000   1948K r-x-- libc-2.27.so
00007f19b300a000   2048K ----- libc-2.27.so
00007f19b320a000     16K r---- libc-2.27.so
00007f19b320e000      8K rw--- libc-2.27.so
# ls -lu /lib/x86_64-linux-gnu/libc.so.6
lrwxrwxrwx 1 root root 12 Aug 23 14:52 /lib/x86_64-linux-gnu/libc.so.6 -> libc-2.27.so
# ls -lu /lib/x86_64-linux-gnu/libc-2.27.so
-rwxr-xr-x 1 root root 2030928 Aug 23 14:52 /lib/x86_64-linux-gnu/libc-2.27.so
# date
Tue Aug 23 16:04:53 EDT 2022
# df /bin
Filesystem     1K-blocks     Used Available Use% Mounted on
/dev/sda1       20509264 14589192   4855216  76% /
# grep sda1 /proc/mounts
/dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0
#

It might stop working when you update, because it will pick up the same troublesome library 35 got.

image

but if it breaks, that will give a chance to test dnf downgrade or the environment variable approach.

Or maybe waiting is an option:

Thanks for doing that and following through. This was quite an effort by its participants.