On Tuesday, July 18, 2023 10:56:12 AM EDT Wyll Ingersoll wrote:
Every night at midnight, our ceph-mgr daemons open up ssh connections to the other nodes and then leaves them open. Eventually they become zombies. I cannot figure out what module is causing this or how to turn it off. If left unchecked over days/weeks, the zombie ssh connections just keep growing, the only way to clear them is to restart ceph-mgr services.
Any idea what is causing this or how it can be disabled?
Example:
ceph 1350387 1350373 7 Jul17 ? 01:19:39 /usr/bin/ceph-mgr -n mgr.mon03 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix
ceph 1350548 1350387 0 Jul17 ? 00:00:01 ssh -C -F /tmp/cephadm-conf-d0khggdz -i /tmp/cephadm-identity-onf2msju -o ServerAliveInterval=7 -o ServerAliveCountMax=3 xxx@10.4.1.11 sudo python
[...snip...]
Is this cluster on pacific? The module in question is likely to be `cephadm` but the cephadm ssh backend has been changed and the team assumes problems like this no longer occur. Hope that helps!