Found unknown daemon type ceph-exporter on host after upgrade to 17.2.7
Hello! I see [WRN] Found unknown daemon type ceph-exporter on host for all 3 ceph servers in logs, after upgrade to 17.2.7 from 17.2.6 in dashboard and cephadm ['--image', 'quay.io/ceph/ceph@sha256:1fcdbead4709a7182047f8ff9726e0f17b0b209aaa6656c5c8b2339b818e70bb', '--timeout', '895', 'ls'] 2023-11-07 16:15:37,531 7fddb699b740 WARNING version for unknown daemon type ceph-exporter in cephadm.log There were no such messages before. I guess this will not create any problem, but is there any way to fix this?
Sorry, there ended up being an issue with the ceph-exporter when it was backported to quincy so it was removed for 17.2.7 . You should be able to do `ceph orch rm ceph-exporter` if there is a "ceph-exporter" service in "ceph orch ls" output. If it's not in the "ceph orch ls" output, I believe cephadm will just remove the ceph-exporter daemons on its own and the warnings will clear out. It shouldn't cause any issues with the rest of the cluster I believe. On Tue, Nov 7, 2023 at 8:10 AM Dmitry Melekhov <dm@belkam.com> wrote:
Hello!
I see
[WRN] Found unknown daemon type ceph-exporter on host
for all 3 ceph servers in logs, after upgrade to 17.2.7 from 17.2.6 in dashboard
and
cephadm ['--image', ' quay.io/ceph/ceph@sha256:1fcdbead4709a7182047f8ff9726e0f17b0b209aaa6656c5c8b2339b818e70bb',
'--timeout', '895', 'ls'] 2023-11-07 16:15:37,531 7fddb699b740 WARNING version for unknown daemon type ceph-exporter
in
cephadm.log
There were no such messages before.
I guess this will not create any problem, but is there any way to fix this?
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
07.11.2023 22:51, Adam King пишет:
Sorry, there ended up being an issue with the ceph-exporter when it was backported to quincy so it was removed for 17.2.7 . You should be able to do `ceph orch rm ceph-exporter` if there is a "ceph-exporter" service in "ceph orch ls" output. If it's not in the "ceph orch ls" output, I believe cephadm will just remove the ceph-exporter daemons on its own and the warnings will clear out. It shouldn't cause any issues with the rest of the cluster I believe.
It is not in output: [root@vmx1 ~]# ceph orch ls NAME PORTS RUNNING REFRESHED AGE PLACEMENT alertmanager ?:9093,9094 1/1 7m ago 3M count:1 crash 3/3 8m ago 3M * grafana ?:3000 1/1 7m ago 3M count:1 mds.cephfs 3/3 8m ago 3M count:3 mgr 2/2 8m ago 3M count:2 mon 3/3 8m ago 3M vmx1;vmx2;vmx3;count:3 node-exporter ?:9100 3/3 8m ago 3M * osd 3 8m ago - <unmanaged> osd.all-available-devices 15 8m ago 3M * prometheus ?:9095 1/1 7m ago 3M count:1 [root@vmx1 ~]# but it is still running after several hours, and I still have warnings in logs: cephadm ['--image', 'quay.io/ceph/ceph@sha256:1fcdbead4709a7182047f8ff9726e0f17b0b209aaa6656c5c8b2339b818e70bb', '--timeout', '895', 'ls'] 2023-11-08 06:53:01,463 7f4e063a3740 WARNING version for unknown daemon type ceph-exporter and looks like it is started at latest server reboot: [root@vmx1 ceph]# podman ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 00a7bef5e61b quay.io/ceph/ceph@sha256:11e9bcdd06cfd4f6b0fef3da5c2a3f0f87d7a48c043080d6f5d02f7010ad72eb -n client.ceph-ex... 3 weeks ago Up 3 weeks ceph-8e911cca-248a-11ee-8971-84160c1b4d0c-ceph-exporter-vmx1 other containers are startet 17 hours ago during ceph upgrade: c28fdaa53e77 quay.io/ceph/ceph@sha256:1fcdbead4709a7182047f8ff9726e0f17b0b209aaa6656c5c8b2339b818e70bb -n mgr.vmx1.cfbiw... 17 hours ago Up 17 hours ceph-8e911cca-248a-11ee-8971-84160c1b4d0c-mgr-vmx1-cfbiwh So, looks like cephadm will not stop ceph-exporter. Could you tell me, how can I correctly stop it and prevent from start on boot, if it is not disabled already ? I'm very new in podman, really I know almost nothing about it right now :-( Thank you!
On Tue, Nov 7, 2023 at 8:10 AM Dmitry Melekhov <dm@belkam.com> wrote:
Hello!
I see
[WRN] Found unknown daemon type ceph-exporter on host
for all 3 ceph servers in logs, after upgrade to 17.2.7 from 17.2.6 in dashboard
and
cephadm ['--image', 'quay.io/ceph/ceph@sha256:1fcdbead4709a7182047f8ff9726e0f17b0b209aaa6656c5c8b2339b818e70bb <http://quay.io/ceph/ceph@sha256:1fcdbead4709a7182047f8ff9726e0f17b0b209aaa6656c5c8b2339b818e70bb>',
'--timeout', '895', 'ls'] 2023-11-07 16:15:37,531 7fddb699b740 WARNING version for unknown daemon type ceph-exporter
in
cephadm.log
There were no such messages before.
I guess this will not create any problem, but is there any way to fix this?
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
btw, as I see ceph-exporter is started by /etc/systemd/system/ceph-8e911cca-248a-11ee-8971-84160c1b4d0c.target.wants in this target there is ceph-8e911cca-248a-11ee-8971~c@ceph-exporter.vmx2.service the same is on other nodes can I just stop this service on each server and remove it from ceph-8e911cca-248a-11ee-8971-84160c1b4d0c.target.wants ? or I have to do this somehow by using cephadm ? Thank you! 08.11.2023 07:07, Dmitry Melekhov пишет:
07.11.2023 22:51, Adam King пишет:
Sorry, there ended up being an issue with the ceph-exporter when it was backported to quincy so it was removed for 17.2.7 . You should be able to do `ceph orch rm ceph-exporter` if there is a "ceph-exporter" service in "ceph orch ls" output. If it's not in the "ceph orch ls" output, I believe cephadm will just remove the ceph-exporter daemons on its own and the warnings will clear out. It shouldn't cause any issues with the rest of the cluster I believe.
It is not in output:
[root@vmx1 ~]# ceph orch ls NAME PORTS RUNNING REFRESHED AGE PLACEMENT alertmanager ?:9093,9094 1/1 7m ago 3M count:1 crash 3/3 8m ago 3M * grafana ?:3000 1/1 7m ago 3M count:1 mds.cephfs 3/3 8m ago 3M count:3 mgr 2/2 8m ago 3M count:2 mon 3/3 8m ago 3M vmx1;vmx2;vmx3;count:3 node-exporter ?:9100 3/3 8m ago 3M * osd 3 8m ago - <unmanaged> osd.all-available-devices 15 8m ago 3M * prometheus ?:9095 1/1 7m ago 3M count:1 [root@vmx1 ~]#
but it is still running after several hours, and I still have warnings in logs:
cephadm ['--image', 'quay.io/ceph/ceph@sha256:1fcdbead4709a7182047f8ff9726e0f17b0b209aaa6656c5c8b2339b818e70bb', '--timeout', '895', 'ls'] 2023-11-08 06:53:01,463 7f4e063a3740 WARNING version for unknown daemon type ceph-exporter
and looks like it is started at latest server reboot:
[root@vmx1 ceph]# podman ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 00a7bef5e61b quay.io/ceph/ceph@sha256:11e9bcdd06cfd4f6b0fef3da5c2a3f0f87d7a48c043080d6f5d02f7010ad72eb -n client.ceph-ex... 3 weeks ago Up 3 weeks ceph-8e911cca-248a-11ee-8971-84160c1b4d0c-ceph-exporter-vmx1
other containers are startet 17 hours ago during ceph upgrade:
c28fdaa53e77 quay.io/ceph/ceph@sha256:1fcdbead4709a7182047f8ff9726e0f17b0b209aaa6656c5c8b2339b818e70bb -n mgr.vmx1.cfbiw... 17 hours ago Up 17 hours ceph-8e911cca-248a-11ee-8971-84160c1b4d0c-mgr-vmx1-cfbiwh
So, looks like cephadm will not stop ceph-exporter.
Could you tell me, how can I correctly stop it and prevent from start on boot, if it is not disabled already ?
I'm very new in podman, really I know almost nothing about it right now :-(
Thank you!
On Tue, Nov 7, 2023 at 8:10 AM Dmitry Melekhov <dm@belkam.com> wrote:
Hello!
I see
[WRN] Found unknown daemon type ceph-exporter on host
for all 3 ceph servers in logs, after upgrade to 17.2.7 from 17.2.6 in dashboard
and
cephadm ['--image', 'quay.io/ceph/ceph@sha256:1fcdbead4709a7182047f8ff9726e0f17b0b209aaa6656c5c8b2339b818e70bb <http://quay.io/ceph/ceph@sha256:1fcdbead4709a7182047f8ff9726e0f17b0b209aaa6656c5c8b2339b818e70bb>',
'--timeout', '895', 'ls'] 2023-11-07 16:15:37,531 7fddb699b740 WARNING version for unknown daemon type ceph-exporter
in
cephadm.log
There were no such messages before.
I guess this will not create any problem, but is there any way to fix this?
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Seems you're right. The normal code to remove stray daemons doesn't seem to work with unknown types. For cleanup, I found on each node that this worked [root@vm-01 ~]# cephadm ls --no-detail | grep ceph-exporter "name": "ceph-exporter.vm-01", "systemd_unit": "ceph-7875b090-7e49-11ee-a5a9-525400a17001@ceph-exporter.vm-01" [root@vm-01 ~]# [root@vm-01 ~]# systemctl stop ceph-7875b090-7e49-11ee-a5a9-525400a17001@ceph-exporter.vm-01 [root@vm-01 ~]# [root@vm-01 ~]# rm -rf /var/lib/ceph/7875b090-7e49-11ee-a5a9-525400a17001/ceph-exporter.* so essentially stopping the systemd unit and then removing the daemon dir in /var/lib/ceph/<fsid>/. Removing the dir in /var/lib/ceph/<fsid>/ removes the warning in the log and stopping the systemd unit removes the actual container process. On Wed, Nov 8, 2023 at 8:51 AM Sake <ceph@paulusma.eu> wrote:
Hi Adam,
This also happens on our cluster where ceph-exporter got deployed. Cephadm isn't removing any daemons and the logs are getting spammed about this :(
What actions do we need to take to completely remove ceph-exporter daemons?
Best regards, Sake
On 7 Nov 2023 19:51, Adam King <adking@redhat.com> wrote:
Sorry, there ended up being an issue with the ceph-exporter when it was backported to quincy so it was removed for 17.2.7 . You should be able to do `ceph orch rm ceph-exporter` if there is a "ceph-exporter" service in "ceph orch ls" output. If it's not in the "ceph orch ls" output, I believe cephadm will just remove the ceph-exporter daemons on its own and the warnings will clear out. It shouldn't cause any issues with the rest of the cluster I believe.
On Tue, Nov 7, 2023 at 8:10 AM Dmitry Melekhov <dm@belkam.com> wrote:
Hello!
I see
[WRN] Found unknown daemon type ceph-exporter on host
for all 3 ceph servers in logs, after upgrade to 17.2.7 from 17.2.6 in dashboard
and
cephadm ['--image', '
quay.io/ceph/ceph@sha256:1fcdbead4709a7182047f8ff9726e0f17b0b209aaa6656c5c8b2339b818e70bb',
'--timeout', '895', 'ls'] 2023-11-07 16:15:37,531 7fddb699b740 WARNING version for unknown daemon type ceph-exporter
in
cephadm.log
There were no such messages before.
I guess this will not create any problem, but is there any way to fix
this?
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Sake, could you, please, try this and inform me about result? Unfortunately I will not able to do this tomorrow :-( 08.11.2023 19:52, Adam King пишет:
Seems you're right. The normal code to remove stray daemons doesn't seem to work with unknown types. For cleanup, I found on each node that this worked
[root@vm-01 ~]# cephadm ls --no-detail | grep ceph-exporter "name": "ceph-exporter.vm-01", "systemd_unit": "ceph-7875b090-7e49-11ee-a5a9-525400a17001@ceph-exporter.vm-01" [root@vm-01 ~]# [root@vm-01 ~]# systemctl stop ceph-7875b090-7e49-11ee-a5a9-525400a17001@ceph-exporter.vm-01 [root@vm-01 ~]# [root@vm-01 ~]# rm -rf /var/lib/ceph/7875b090-7e49-11ee-a5a9-525400a17001/ceph-exporter.*
so essentially stopping the systemd unit and then removing the daemon dir in /var/lib/ceph/<fsid>/. Removing the dir in /var/lib/ceph/<fsid>/ removes the warning in the log and stopping the systemd unit removes the actual container process.
I guess we also have to remove to disable daemon by systemctl and remove symlink in /etc/systemd ...
On Wed, Nov 8, 2023 at 8:51 AM Sake <ceph@paulusma.eu> wrote:
Hi Adam,
This also happens on our cluster where ceph-exporter got deployed. Cephadm isn't removing any daemons and the logs are getting spammed about this :(
What actions do we need to take to completely remove ceph-exporter daemons?
Best regards, Sake
On 7 Nov 2023 19:51, Adam King <adking@redhat.com> wrote:
Sorry, there ended up being an issue with the ceph-exporter when it was backported to quincy so it was removed for 17.2.7 . You should be able to do `ceph orch rm ceph-exporter` if there is a "ceph-exporter" service in "ceph orch ls" output. If it's not in the "ceph orch ls" output, I believe cephadm will just remove the ceph-exporter daemons on its own and the warnings will clear out. It shouldn't cause any issues with the rest of the cluster I believe.
On Tue, Nov 7, 2023 at 8:10 AM Dmitry Melekhov <dm@belkam.com> wrote:
Hello!
I see
[WRN] Found unknown daemon type ceph-exporter on host
for all 3 ceph servers in logs, after upgrade to 17.2.7 from 17.2.6 in dashboard
and
cephadm ['--image', '
quay.io/ceph/ceph@sha256:1fcdbead4709a7182047f8ff9726e0f17b0b209aaa6656c5c8b2339b818e70bb',
'--timeout', '895', 'ls'] 2023-11-07 16:15:37,531 7fddb699b740 WARNING version for unknown daemon type ceph-exporter
in
cephadm.log
There were no such messages before.
I guess this will not create any problem, but is there any way to fix this?
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Unfortunately I also can't test this tomorrow, Friday it will be the first thing I will do. Best regards, Sake
Op 08-11-2023 17:08 CET schreef Dmitry Melekhov <dm@belkam.com>:
Sake, could you, please, try this and inform me about result?
Unfortunately I will not able to do this tomorrow :-(
08.11.2023 19:52, Adam King пишет:
Seems you're right. The normal code to remove stray daemons doesn't seem to work with unknown types. For cleanup, I found on each node that this worked
[root@vm-01 ~]# cephadm ls --no-detail | grep ceph-exporter "name": "ceph-exporter.vm-01", "systemd_unit": "ceph-7875b090-7e49-11ee-a5a9-525400a17001@ceph-exporter.vm-01" [root@vm-01 ~]# [root@vm-01 ~]# systemctl stop ceph-7875b090-7e49-11ee-a5a9-525400a17001@ceph-exporter.vm-01 [root@vm-01 ~]# [root@vm-01 ~]# rm -rf /var/lib/ceph/7875b090-7e49-11ee-a5a9-525400a17001/ceph-exporter.*
so essentially stopping the systemd unit and then removing the daemon dir in /var/lib/ceph/<fsid>/. Removing the dir in /var/lib/ceph/<fsid>/ removes the warning in the log and stopping the systemd unit removes the actual container process.
I guess we also have to remove to disable daemon by systemctl and remove symlink in /etc/systemd ...
On Wed, Nov 8, 2023 at 8:51 AM Sake <ceph@paulusma.eu> wrote:
Hi Adam,
This also happens on our cluster where ceph-exporter got deployed. Cephadm isn't removing any daemons and the logs are getting spammed about this :(
What actions do we need to take to completely remove ceph-exporter daemons?
Best regards, Sake
On 7 Nov 2023 19:51, Adam King <adking@redhat.com> wrote:
Sorry, there ended up being an issue with the ceph-exporter when it was backported to quincy so it was removed for 17.2.7 . You should be able to do `ceph orch rm ceph-exporter` if there is a "ceph-exporter" service in "ceph orch ls" output. If it's not in the "ceph orch ls" output, I believe cephadm will just remove the ceph-exporter daemons on its own and the warnings will clear out. It shouldn't cause any issues with the rest of the cluster I believe.
On Tue, Nov 7, 2023 at 8:10 AM Dmitry Melekhov <dm@belkam.com> wrote:
Hello!
I see
[WRN] Found unknown daemon type ceph-exporter on host
for all 3 ceph servers in logs, after upgrade to 17.2.7 from 17.2.6 in dashboard
and
cephadm ['--image', '
quay.io/ceph/ceph@sha256:1fcdbead4709a7182047f8ff9726e0f17b0b209aaa6656c5c8b2339b818e70bb',
'--timeout', '895', 'ls'] 2023-11-07 16:15:37,531 7fddb699b740 WARNING version for unknown daemon type ceph-exporter
in
cephadm.log
There were no such messages before.
I guess this will not create any problem, but is there any way to fix this?
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Found some time and the following cleared most references to ceph-exporter. There are only some files left in /var/lib/containers/storage/overlay/. HOSTSTOCLEAN="host1 host2 host3" for HOST in ${HOSTSTOCLEAN}; do ssh -o StrictHostKeyChecking=accept-new ${HOST} ' sudo systemctl disable ceph-<cluster_id>@ceph-exporter.$(hostname -s).service; \ sudo systemctl stop ceph-<cluster_id>@ceph-exporter.$(hostname -s).service; \ sudo rm -rf /var/lib/ceph/<cluster_id>/ceph-exporter.$(hostname -s); \ sudo rm -rf /var/lib/ceph/<cluster_id>/custom_config_files/ceph-exporter.$(hostname -s); \ sudo podman system prune -a -f; \ sudo podman volume prune -f; \ ' done
Op 08-11-2023 18:13 CET schreef Sake Ceph <ceph@paulusma.eu>:
Unfortunately I also can't test this tomorrow, Friday it will be the first thing I will do.
Best regards, Sake
Op 08-11-2023 17:08 CET schreef Dmitry Melekhov <dm@belkam.com>:
Sake, could you, please, try this and inform me about result?
Unfortunately I will not able to do this tomorrow :-(
08.11.2023 19:52, Adam King пишет:
Seems you're right. The normal code to remove stray daemons doesn't seem to work with unknown types. For cleanup, I found on each node that this worked
[root@vm-01 ~]# cephadm ls --no-detail | grep ceph-exporter "name": "ceph-exporter.vm-01", "systemd_unit": "ceph-7875b090-7e49-11ee-a5a9-525400a17001@ceph-exporter.vm-01" [root@vm-01 ~]# [root@vm-01 ~]# systemctl stop ceph-7875b090-7e49-11ee-a5a9-525400a17001@ceph-exporter.vm-01 [root@vm-01 ~]# [root@vm-01 ~]# rm -rf /var/lib/ceph/7875b090-7e49-11ee-a5a9-525400a17001/ceph-exporter.*
so essentially stopping the systemd unit and then removing the daemon dir in /var/lib/ceph/<fsid>/. Removing the dir in /var/lib/ceph/<fsid>/ removes the warning in the log and stopping the systemd unit removes the actual container process.
I guess we also have to remove to disable daemon by systemctl and remove symlink in /etc/systemd ...
On Wed, Nov 8, 2023 at 8:51 AM Sake <ceph@paulusma.eu> wrote:
Hi Adam,
This also happens on our cluster where ceph-exporter got deployed. Cephadm isn't removing any daemons and the logs are getting spammed about this :(
What actions do we need to take to completely remove ceph-exporter daemons?
Best regards, Sake
On 7 Nov 2023 19:51, Adam King <adking@redhat.com> wrote:
Sorry, there ended up being an issue with the ceph-exporter when it was backported to quincy so it was removed for 17.2.7 . You should be able to do `ceph orch rm ceph-exporter` if there is a "ceph-exporter" service in "ceph orch ls" output. If it's not in the "ceph orch ls" output, I believe cephadm will just remove the ceph-exporter daemons on its own and the warnings will clear out. It shouldn't cause any issues with the rest of the cluster I believe.
On Tue, Nov 7, 2023 at 8:10 AM Dmitry Melekhov <dm@belkam.com> wrote:
Hello!
I see
[WRN] Found unknown daemon type ceph-exporter on host
for all 3 ceph servers in logs, after upgrade to 17.2.7 from 17.2.6 in dashboard
and
cephadm ['--image', '
quay.io/ceph/ceph@sha256:1fcdbead4709a7182047f8ff9726e0f17b0b209aaa6656c5c8b2339b818e70bb',
'--timeout', '895', 'ls'] 2023-11-07 16:15:37,531 7fddb699b740 WARNING version for unknown daemon type ceph-exporter
in
cephadm.log
There were no such messages before.
I guess this will not create any problem, but is there any way to fix this?
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
09.11.2023 13:44, Sake Ceph пишет:
Found some time and the following cleared most references to ceph-exporter. There are only some files left in /var/lib/containers/storage/overlay/.
HOSTSTOCLEAN="host1 host2 host3"
for HOST in ${HOSTSTOCLEAN}; do ssh -o StrictHostKeyChecking=accept-new ${HOST} ' sudo systemctl disable ceph-<cluster_id>@ceph-exporter.$(hostname -s).service; \ sudo systemctl stop ceph-<cluster_id>@ceph-exporter.$(hostname -s).service; \ sudo rm -rf /var/lib/ceph/<cluster_id>/ceph-exporter.$(hostname -s); \ sudo rm -rf /var/lib/ceph/<cluster_id>/custom_config_files/ceph-exporter.$(hostname -s); \ sudo podman system prune -a -f; \ sudo podman volume prune -f; \ ' done
Thank you very much. I'll do the same tomorrow...
Op 08-11-2023 18:13 CET schreef Sake Ceph <ceph@paulusma.eu>:
Unfortunately I also can't test this tomorrow, Friday it will be the first thing I will do.
Best regards, Sake
Op 08-11-2023 17:08 CET schreef Dmitry Melekhov <dm@belkam.com>:
Sake, could you, please, try this and inform me about result?
Unfortunately I will not able to do this tomorrow :-(
08.11.2023 19:52, Adam King пишет:
Seems you're right. The normal code to remove stray daemons doesn't seem to work with unknown types. For cleanup, I found on each node that this worked
[root@vm-01 ~]# cephadm ls --no-detail | grep ceph-exporter "name": "ceph-exporter.vm-01", "systemd_unit": "ceph-7875b090-7e49-11ee-a5a9-525400a17001@ceph-exporter.vm-01" [root@vm-01 ~]# [root@vm-01 ~]# systemctl stop ceph-7875b090-7e49-11ee-a5a9-525400a17001@ceph-exporter.vm-01 [root@vm-01 ~]# [root@vm-01 ~]# rm -rf /var/lib/ceph/7875b090-7e49-11ee-a5a9-525400a17001/ceph-exporter.*
so essentially stopping the systemd unit and then removing the daemon dir in /var/lib/ceph/<fsid>/. Removing the dir in /var/lib/ceph/<fsid>/ removes the warning in the log and stopping the systemd unit removes the actual container process. I guess we also have to remove to disable daemon by systemctl and remove symlink in /etc/systemd ...
On Wed, Nov 8, 2023 at 8:51 AM Sake <ceph@paulusma.eu> wrote:
Hi Adam,
This also happens on our cluster where ceph-exporter got deployed. Cephadm isn't removing any daemons and the logs are getting spammed about this :(
What actions do we need to take to completely remove ceph-exporter daemons?
Best regards, Sake
On 7 Nov 2023 19:51, Adam King <adking@redhat.com> wrote:
Sorry, there ended up being an issue with the ceph-exporter when it was backported to quincy so it was removed for 17.2.7 . You should be able to do `ceph orch rm ceph-exporter` if there is a "ceph-exporter" service in "ceph orch ls" output. If it's not in the "ceph orch ls" output, I believe cephadm will just remove the ceph-exporter daemons on its own and the warnings will clear out. It shouldn't cause any issues with the rest of the cluster I believe.
On Tue, Nov 7, 2023 at 8:10 AM Dmitry Melekhov <dm@belkam.com> wrote:
Hello!
I see
[WRN] Found unknown daemon type ceph-exporter on host
for all 3 ceph servers in logs, after upgrade to 17.2.7 from 17.2.6 in dashboard
and
cephadm ['--image', '
quay.io/ceph/ceph@sha256:1fcdbead4709a7182047f8ff9726e0f17b0b209aaa6656c5c8b2339b818e70bb',
'--timeout', '895', 'ls'] 2023-11-07 16:15:37,531 7fddb699b740 WARNING version for unknown daemon type ceph-exporter
in
cephadm.log
There were no such messages before.
I guess this will not create any problem, but is there any way to fix this? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
Adam King
-
Dmitry Melekhov
-
Sake
-
Sake Ceph