Re: osd removal leaves 'stray daemon'
Hi, We're encountering this unexpected behavior as well. This tracker [1] was created 4 months ago. Regards, Frédéric. [1] https://tracker.ceph.com/issues/67018 ----- Le 6 Déc 22, à 8:41, Holger Naundorf naundorf@rz.uni-kiel.de a écrit :
Hello, a mgr failover did not change the situation - the osd still shows up in the 'ceph node ls' - I assume that this is more or less 'working as intended' as I did ask for the OSD to be kept in the CRUSH map to be replacd later - but as we are still not so experienced with Ceph here I wanted to get some input from other sites.
Regards, Holger
On 30.11.22 16:28, Adam King wrote:
I typically don't see this when I do OSD replacement. If you do a mgr failover ("ceph mgr fail") and wait a few minutes does this still show up? The stray daemon/host warning is roughly equivalent to comparing the daemons in `ceph node ls` and `ceph orch ps` and seeing if there's anything in the former but not the latter. Sometimes I have seen the mgr will have some out of data info in the node ls and a failover will refresh it.
On Fri, Nov 25, 2022 at 6:07 AM Holger Naundorf <naundorf@rz.uni-kiel.de <mailto:naundorf@rz.uni-kiel.de>> wrote:
Hello, I have a question about osd removal/replacement:
I just removed an osd where the disk was still running but had read errors, leading to failed deep scrubs - as the intent is to replace this as soon as we manage to get a spare I removed it with the '--replace' flag:
# ceph orch osd rm 224 --replace
After all placement groups are evacuated I now have 1 osd down/out and showing as 'destroyed':
# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF (...) 214 hdd 14.55269 osd.214 up 1.00000 1.00000 224 hdd 14.55269 osd.224 destroyed 0 1.00000 234 hdd 14.55269 osd.234 up 1.00000 1.00000 (...)
All as expected - but now the health check complains that the (destroyed) osd is not managed:
# ceph health detail HEALTH_WARN 1 stray daemon(s) not managed by cephadm [WRN] CEPHADM_STRAY_DAEMON: 1 stray daemon(s) not managed by cephadm stray daemon osd.224 on host ceph19 not managed by cephadm
Is this expected behaviour and I have to live with the yellow check until we get a replacement disk and recreate the osd or did something not finish correctly?
Regards, Holger
-- Dr. Holger Naundorf Christian-Albrechts-Universität zu Kiel Rechenzentrum / HPC / Server und Storage Tel: +49 431 880-1990 Fax: +49 431 880-1523 naundorf@rz.uni-kiel.de <mailto:naundorf@rz.uni-kiel.de> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
-- Dr. Holger Naundorf Christian-Albrechts-Universität zu Kiel Rechenzentrum / HPC / Server und Storage Tel: +49 431 880-1990 Fax: +49 431 880-1523 naundorf@rz.uni-kiel.de
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
You can get this sort of behaviour because different Ceph subsystems get their information from different places instead of having an authoritative source of information. Specifically, Ceph may look directly at: A) Its configuration database B) Systemd units running on the OSD host C) Containers running ceph modules. The problem is especially likely if you've managed to end up running the same OSD number as both legacy (/var/lib/ceph/osd.x) and Manager (/var/lib/ceph/{fsid}/ceph). If you have a dual-defined OSD, the cleanest approach seems to be to stop ceph on the bad machine and manually delete the /var/lib/ceph/osd.x directory. You may need to delete a systemd unit file for that OSD. You cannot delete the systemd unit for Managed OSDs. It's dynamically created when the system comes up and will simply re-create itself. Which is why it's easier to purge the artefacts of a legacy OSD. Tim On 11/7/24 10:28, Frédéric Nass wrote:
Hi,
We're encountering this unexpected behavior as well. This tracker [1] was created 4 months ago.
Regards, Frédéric.
[1] https://tracker.ceph.com/issues/67018
----- Le 6 Déc 22, à 8:41, Holger Naundorf naundorf@rz.uni-kiel.de a écrit :
Hello, a mgr failover did not change the situation - the osd still shows up in the 'ceph node ls' - I assume that this is more or less 'working as intended' as I did ask for the OSD to be kept in the CRUSH map to be replacd later - but as we are still not so experienced with Ceph here I wanted to get some input from other sites.
Regards, Holger
On 30.11.22 16:28, Adam King wrote:
I typically don't see this when I do OSD replacement. If you do a mgr failover ("ceph mgr fail") and wait a few minutes does this still show up? The stray daemon/host warning is roughly equivalent to comparing the daemons in `ceph node ls` and `ceph orch ps` and seeing if there's anything in the former but not the latter. Sometimes I have seen the mgr will have some out of data info in the node ls and a failover will refresh it.
On Fri, Nov 25, 2022 at 6:07 AM Holger Naundorf <naundorf@rz.uni-kiel.de <mailto:naundorf@rz.uni-kiel.de>> wrote:
Hello, I have a question about osd removal/replacement:
I just removed an osd where the disk was still running but had read errors, leading to failed deep scrubs - as the intent is to replace this as soon as we manage to get a spare I removed it with the '--replace' flag:
# ceph orch osd rm 224 --replace
After all placement groups are evacuated I now have 1 osd down/out and showing as 'destroyed':
# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF (...) 214 hdd 14.55269 osd.214 up 1.00000 1.00000 224 hdd 14.55269 osd.224 destroyed 0 1.00000 234 hdd 14.55269 osd.234 up 1.00000 1.00000 (...)
All as expected - but now the health check complains that the (destroyed) osd is not managed:
# ceph health detail HEALTH_WARN 1 stray daemon(s) not managed by cephadm [WRN] CEPHADM_STRAY_DAEMON: 1 stray daemon(s) not managed by cephadm stray daemon osd.224 on host ceph19 not managed by cephadm
Is this expected behaviour and I have to live with the yellow check until we get a replacement disk and recreate the osd or did something not finish correctly?
Regards, Holger
-- Dr. Holger Naundorf Christian-Albrechts-Universität zu Kiel Rechenzentrum / HPC / Server und Storage Tel: +49 431 880-1990 Fax: +49 431 880-1523 naundorf@rz.uni-kiel.de <mailto:naundorf@rz.uni-kiel.de> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
-- Dr. Holger Naundorf Christian-Albrechts-Universität zu Kiel Rechenzentrum / HPC / Server und Storage Tel: +49 431 880-1990 Fax: +49 431 880-1523 naundorf@rz.uni-kiel.de
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Tim, I see what you're referring to, but it doesn't apply since there's actually **no** stray daemons, that is **no** ghost process on any hosts trying to start. Here we're talking about unexpected behavior, most likely a bug. Regards, Frédéric. ----- Le 7 Nov 24, à 21:08, Tim Holloway timh@mousetech.com a écrit :
You can get this sort of behaviour because different Ceph subsystems get their information from different places instead of having an authoritative source of information.
Specifically, Ceph may look directly at:
A) Its configuration database
B) Systemd units running on the OSD host
C) Containers running ceph modules.
The problem is especially likely if you've managed to end up running the same OSD number as both legacy (/var/lib/ceph/osd.x) and Manager (/var/lib/ceph/{fsid}/ceph).
If you have a dual-defined OSD, the cleanest approach seems to be to stop ceph on the bad machine and manually delete the /var/lib/ceph/osd.x directory. You may need to delete a systemd unit file for that OSD.
You cannot delete the systemd unit for Managed OSDs. It's dynamically created when the system comes up and will simply re-create itself. Which is why it's easier to purge the artefacts of a legacy OSD.
Tim
On 11/7/24 10:28, Frédéric Nass wrote:
Hi,
We're encountering this unexpected behavior as well. This tracker [1] was created 4 months ago.
Regards, Frédéric.
[1] https://tracker.ceph.com/issues/67018
----- Le 6 Déc 22, à 8:41, Holger Naundorf naundorf@rz.uni-kiel.de a écrit :
Hello, a mgr failover did not change the situation - the osd still shows up in the 'ceph node ls' - I assume that this is more or less 'working as intended' as I did ask for the OSD to be kept in the CRUSH map to be replacd later - but as we are still not so experienced with Ceph here I wanted to get some input from other sites.
Regards, Holger
On 30.11.22 16:28, Adam King wrote:
I typically don't see this when I do OSD replacement. If you do a mgr failover ("ceph mgr fail") and wait a few minutes does this still show up? The stray daemon/host warning is roughly equivalent to comparing the daemons in `ceph node ls` and `ceph orch ps` and seeing if there's anything in the former but not the latter. Sometimes I have seen the mgr will have some out of data info in the node ls and a failover will refresh it.
On Fri, Nov 25, 2022 at 6:07 AM Holger Naundorf <naundorf@rz.uni-kiel.de <mailto:naundorf@rz.uni-kiel.de>> wrote:
Hello, I have a question about osd removal/replacement:
I just removed an osd where the disk was still running but had read errors, leading to failed deep scrubs - as the intent is to replace this as soon as we manage to get a spare I removed it with the '--replace' flag:
# ceph orch osd rm 224 --replace
After all placement groups are evacuated I now have 1 osd down/out and showing as 'destroyed':
# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF (...) 214 hdd 14.55269 osd.214 up 1.00000 1.00000 224 hdd 14.55269 osd.224 destroyed 0 1.00000 234 hdd 14.55269 osd.234 up 1.00000 1.00000 (...)
All as expected - but now the health check complains that the (destroyed) osd is not managed:
# ceph health detail HEALTH_WARN 1 stray daemon(s) not managed by cephadm [WRN] CEPHADM_STRAY_DAEMON: 1 stray daemon(s) not managed by cephadm stray daemon osd.224 on host ceph19 not managed by cephadm
Is this expected behaviour and I have to live with the yellow check until we get a replacement disk and recreate the osd or did something not finish correctly?
Regards, Holger
-- Dr. Holger Naundorf Christian-Albrechts-Universität zu Kiel Rechenzentrum / HPC / Server und Storage Tel: +49 431 880-1990 Fax: +49 431 880-1523 naundorf@rz.uni-kiel.de <mailto:naundorf@rz.uni-kiel.de> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
-- Dr. Holger Naundorf Christian-Albrechts-Universität zu Kiel Rechenzentrum / HPC / Server und Storage Tel: +49 431 880-1990 Fax: +49 431 880-1523 naundorf@rz.uni-kiel.de
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I admit I don't follow what the exact problem is, but I wanted to point out that as long as there are ANY OSD metadata files on a machine, some (but not always all) ceph commands will consider there to be an OSD there. To completely eradicate an OSD, I believe that (per Eugen Block) you also have to set the OSD's crush weights to 0. I had a heck of a time flushing out crud a few months back. I learned a lot. Whether I wanted to or not. Tim On 11/7/24 16:57, Frédéric Nass wrote:
Hi Tim,
I see what you're referring to, but it doesn't apply since there's actually **no** stray daemons, that is **no** ghost process on any hosts trying to start.
Here we're talking about unexpected behavior, most likely a bug.
Regards, Frédéric.
----- Le 7 Nov 24, à 21:08, Tim Holloway timh@mousetech.com a écrit :
You can get this sort of behaviour because different Ceph subsystems get their information from different places instead of having an authoritative source of information.
Specifically, Ceph may look directly at:
A) Its configuration database
B) Systemd units running on the OSD host
C) Containers running ceph modules.
The problem is especially likely if you've managed to end up running the same OSD number as both legacy (/var/lib/ceph/osd.x) and Manager (/var/lib/ceph/{fsid}/ceph).
If you have a dual-defined OSD, the cleanest approach seems to be to stop ceph on the bad machine and manually delete the /var/lib/ceph/osd.x directory. You may need to delete a systemd unit file for that OSD.
You cannot delete the systemd unit for Managed OSDs. It's dynamically created when the system comes up and will simply re-create itself. Which is why it's easier to purge the artefacts of a legacy OSD.
Tim
On 11/7/24 10:28, Frédéric Nass wrote:
Hi,
We're encountering this unexpected behavior as well. This tracker [1] was created 4 months ago.
Regards, Frédéric.
[1] https://tracker.ceph.com/issues/67018
----- Le 6 Déc 22, à 8:41, Holger Naundorf naundorf@rz.uni-kiel.de a écrit :
Hello, a mgr failover did not change the situation - the osd still shows up in the 'ceph node ls' - I assume that this is more or less 'working as intended' as I did ask for the OSD to be kept in the CRUSH map to be replacd later - but as we are still not so experienced with Ceph here I wanted to get some input from other sites.
Regards, Holger
On 30.11.22 16:28, Adam King wrote:
I typically don't see this when I do OSD replacement. If you do a mgr failover ("ceph mgr fail") and wait a few minutes does this still show up? The stray daemon/host warning is roughly equivalent to comparing the daemons in `ceph node ls` and `ceph orch ps` and seeing if there's anything in the former but not the latter. Sometimes I have seen the mgr will have some out of data info in the node ls and a failover will refresh it.
On Fri, Nov 25, 2022 at 6:07 AM Holger Naundorf <naundorf@rz.uni-kiel.de <mailto:naundorf@rz.uni-kiel.de>> wrote:
Hello, I have a question about osd removal/replacement:
I just removed an osd where the disk was still running but had read errors, leading to failed deep scrubs - as the intent is to replace this as soon as we manage to get a spare I removed it with the '--replace' flag:
# ceph orch osd rm 224 --replace
After all placement groups are evacuated I now have 1 osd down/out and showing as 'destroyed':
# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF (...) 214 hdd 14.55269 osd.214 up 1.00000 1.00000 224 hdd 14.55269 osd.224 destroyed 0 1.00000 234 hdd 14.55269 osd.234 up 1.00000 1.00000 (...)
All as expected - but now the health check complains that the (destroyed) osd is not managed:
# ceph health detail HEALTH_WARN 1 stray daemon(s) not managed by cephadm [WRN] CEPHADM_STRAY_DAEMON: 1 stray daemon(s) not managed by cephadm stray daemon osd.224 on host ceph19 not managed by cephadm
Is this expected behaviour and I have to live with the yellow check until we get a replacement disk and recreate the osd or did something not finish correctly?
Regards, Holger
-- Dr. Holger Naundorf Christian-Albrechts-Universität zu Kiel Rechenzentrum / HPC / Server und Storage Tel: +49 431 880-1990 Fax: +49 431 880-1523 naundorf@rz.uni-kiel.de <mailto:naundorf@rz.uni-kiel.de> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
-- Dr. Holger Naundorf Christian-Albrechts-Universität zu Kiel Rechenzentrum / HPC / Server und Storage Tel: +49 431 880-1990 Fax: +49 431 880-1523 naundorf@rz.uni-kiel.de
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, I added some more logs to the bug tracker [1]. Could this be related to the 60s (hard coded) limit in def _check_for_strays(self) [2]? Regards, Frédéric. [1] https://tracker.ceph.com/issues/67018 [2] https://github.com/ceph/ceph/blob/f55fc4599a6c0da0f4bd2f3ecd2122e603ad94dd/s... ----- Le 7 Nov 24, à 16:28, Frédéric Nass frederic.nass@univ-lorraine.fr a écrit :
Hi,
We're encountering this unexpected behavior as well. This tracker [1] was created 4 months ago.
Regards, Frédéric.
[1] https://tracker.ceph.com/issues/67018
----- Le 6 Déc 22, à 8:41, Holger Naundorf naundorf@rz.uni-kiel.de a écrit :
Hello, a mgr failover did not change the situation - the osd still shows up in the 'ceph node ls' - I assume that this is more or less 'working as intended' as I did ask for the OSD to be kept in the CRUSH map to be replacd later - but as we are still not so experienced with Ceph here I wanted to get some input from other sites.
Regards, Holger
On 30.11.22 16:28, Adam King wrote:
I typically don't see this when I do OSD replacement. If you do a mgr failover ("ceph mgr fail") and wait a few minutes does this still show up? The stray daemon/host warning is roughly equivalent to comparing the daemons in `ceph node ls` and `ceph orch ps` and seeing if there's anything in the former but not the latter. Sometimes I have seen the mgr will have some out of data info in the node ls and a failover will refresh it.
On Fri, Nov 25, 2022 at 6:07 AM Holger Naundorf <naundorf@rz.uni-kiel.de <mailto:naundorf@rz.uni-kiel.de>> wrote:
Hello, I have a question about osd removal/replacement:
I just removed an osd where the disk was still running but had read errors, leading to failed deep scrubs - as the intent is to replace this as soon as we manage to get a spare I removed it with the '--replace' flag:
# ceph orch osd rm 224 --replace
After all placement groups are evacuated I now have 1 osd down/out and showing as 'destroyed':
# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF (...) 214 hdd 14.55269 osd.214 up 1.00000 1.00000 224 hdd 14.55269 osd.224 destroyed 0 1.00000 234 hdd 14.55269 osd.234 up 1.00000 1.00000 (...)
All as expected - but now the health check complains that the (destroyed) osd is not managed:
# ceph health detail HEALTH_WARN 1 stray daemon(s) not managed by cephadm [WRN] CEPHADM_STRAY_DAEMON: 1 stray daemon(s) not managed by cephadm stray daemon osd.224 on host ceph19 not managed by cephadm
Is this expected behaviour and I have to live with the yellow check until we get a replacement disk and recreate the osd or did something not finish correctly?
Regards, Holger
-- Dr. Holger Naundorf Christian-Albrechts-Universität zu Kiel Rechenzentrum / HPC / Server und Storage Tel: +49 431 880-1990 Fax: +49 431 880-1523 naundorf@rz.uni-kiel.de <mailto:naundorf@rz.uni-kiel.de> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
-- Dr. Holger Naundorf Christian-Albrechts-Universität zu Kiel Rechenzentrum / HPC / Server und Storage Tel: +49 431 880-1990 Fax: +49 431 880-1523 naundorf@rz.uni-kiel.de
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (2)
-
Frédéric Nass
-
Tim Holloway