Hi, what's the current ceph status? Wasn't there a bug in early Reef versions preventing upgrades if there were removed OSDs in the queue? But IIRC, the cephadm module would crash. Can you check ceph config-key get mgr/cephadm/osd_remove_queue And then I would check the mgr log, maybe set it to a higher debug level to see what's blocking it. Zitat von Michel Jouvin <michel.jouvin@ijclab.in2p3.fr>:

Hi,

I tried to restart all the mgrs (we have 3, 1 active, 2 standby) by executing 3 times the `ceph mgr fail`, no impact. I don't really understand why I get these stray daemons after doing a 'ceph orch osd rm --replace` but I think I have always seen this. I tried to mute rather than disable the stray daemon check but it doesn't help either. And I find strange this message every 10s about one of the destroyed OSD and only one, reporting it is down and already destroyed and saying it'll zap it (I think I didn't add --zap when I removed it as the underlying disk is dead).

I'm completely stuck with this upgrade and I don't remember having this kind of problems in previous upgrades with cephadm... Any idea where to look for the cause and/or how to fix it?

Best regards,

Michel

Le 24/04/2025 à 23:34, Michel Jouvin a écrit :

...

Hi,

I'm trying to upgrade a (cephadm) cluster from 18.2.2 to 18.2.6, using 'ceph orch upgrade'. When I enter the command 'ceph orch upgrade start --ceph-version 18.2.6', I receive a message saying that the upgrade has been initiated, with a similar message in the logs but nothing happens after this. 'ceph orch upgrade status' says:

-------

[root@ijc-mon1 ~]# ceph orch upgrade status {     "target_image": "quay.io/ceph/ceph:v18.2.6",     "in_progress": true,     "which": "Upgrading all daemon types on all hosts",     "services_complete": [],     "progress": "",     "message": "",     "is_paused": false } -------

The first time I entered the command, the cluster status was HEALTH_WARN because of 2 stray daemons (caused by destroyed OSDs, rm --replace). I set mgr/cephadm/warn_on_stray_daemons to false to ignore these 2 daemons, the cluster is now HEALTH_OK but it doesn't help. Following a Red Hat KB entry, I tried to failover the mgr, stopped an restarted the upgrade but without any improvement. I have not seen anything in the logs, except that there is an INF entry every 10s about the destroyed OSD saying:

------

2025-04-24T21:30:54.161988+0000 mgr.ijc-mon1.yyfnhz (mgr.55376028) 14079 : cephadm [INF] osd.253 now down 2025-04-24T21:30:54.162601+0000 mgr.ijc-mon1.yyfnhz (mgr.55376028) 14080 : cephadm [INF] Daemon osd.253 on dig-osd4 was already removed 2025-04-24T21:30:54.164440+0000 mgr.ijc-mon1.yyfnhz (mgr.55376028) 14081 : cephadm [INF] Successfully destroyed old osd.253 on dig-osd4; ready for replacement 2025-04-24T21:30:54.164536+0000 mgr.ijc-mon1.yyfnhz (mgr.55376028) 14082 : cephadm [INF] Zapping devices for osd.253 on dig-osd4 -----

The message seems to be only for one of the 2 destroyed OSDs since I restarted the mgr. May this be the cause for the stucked upgrade? What can I do for fixing this?

Thanks in advance for any hint. Best regards,

Michel

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Do you see anything in the mgr log? To get fresh logs I would cancel the upgrade (ceph orch upgrade stop) and then try again. A workaround could be to manually upgrade the mgr daemons by changing their unit.run file, but that would be my last resort. Btwm did you stop and start the upgrade after failing the mgr as well? Zitat von Michel Jouvin <michel.jouvin@ijclab.in2p3.fr>:

Eugen,

Thanks for the hint. Here is the osd_remove_queue:

[root@ijc-mon1 ~]# ceph config-key get mgr/cephadm/osd_remove_queue|jq [   {     "osd_id": 253,     "started": true,     "draining": false,     "stopped": false,     "replace": true,     "force": false,     "zap": true,     "hostname": "dig-osd4",     "drain_started_at": null,     "drain_stopped_at": null,     "drain_done_at": "2025-04-15T14:09:30.521534Z",     "process_started_at": "2025-04-15T14:09:14.091592Z"   },   {     "osd_id": 381,     "started": true,     "draining": false,     "stopped": false,     "replace": true,     "force": false,     "zap": false,     "hostname": "dig-osd6",     "drain_started_at": "2025-04-23T11:56:09.864724Z",     "drain_stopped_at": null,     "drain_done_at": "2025-04-25T06:53:03.678729Z",     "process_started_at": "2025-04-23T11:56:05.924923Z"   } ]

It is not empty the two stray daemons are listed. Not sure it these entries are expected as I specified --replace... A similar issue was reported in https://tracker.ceph.com/issues/67018 so before Reef but the cause may be different. Still not clear for me how to get out of this, except may be replacing the OSDs but this will take some time...

Best regards,

Michel

Le 27/04/2025 à 10:21, Eugen Block a écrit :

...

Hi,

what's the current ceph status? Wasn't there a bug in early Reef versions preventing upgrades if there were removed OSDs in the queue? But IIRC, the cephadm module would crash. Can you check

ceph config-key get mgr/cephadm/osd_remove_queue

And then I would check the mgr log, maybe set it to a higher debug level to see what's blocking it.

Zitat von Michel Jouvin <michel.jouvin@ijclab.in2p3.fr>:

...

Hi,

I tried to restart all the mgrs (we have 3, 1 active, 2 standby) by executing 3 times the `ceph mgr fail`, no impact. I don't really understand why I get these stray daemons after doing a 'ceph orch osd rm --replace` but I think I have always seen this. I tried to mute rather than disable the stray daemon check but it doesn't help either. And I find strange this message every 10s about one of the destroyed OSD and only one, reporting it is down and already destroyed and saying it'll zap it (I think I didn't add --zap when I removed it as the underlying disk is dead).

I'm completely stuck with this upgrade and I don't remember having this kind of problems in previous upgrades with cephadm... Any idea where to look for the cause and/or how to fix it?

Best regards,

Michel

Le 24/04/2025 à 23:34, Michel Jouvin a écrit :

...

Hi,

I'm trying to upgrade a (cephadm) cluster from 18.2.2 to 18.2.6, using 'ceph orch upgrade'. When I enter the command 'ceph orch upgrade start --ceph-version 18.2.6', I receive a message saying that the upgrade has been initiated, with a similar message in the logs but nothing happens after this. 'ceph orch upgrade status' says:

-------

[root@ijc-mon1 ~]# ceph orch upgrade status {     "target_image": "quay.io/ceph/ceph:v18.2.6",     "in_progress": true,     "which": "Upgrading all daemon types on all hosts",     "services_complete": [],     "progress": "",     "message": "",     "is_paused": false } -------

The first time I entered the command, the cluster status was HEALTH_WARN because of 2 stray daemons (caused by destroyed OSDs, rm --replace). I set mgr/cephadm/warn_on_stray_daemons to false to ignore these 2 daemons, the cluster is now HEALTH_OK but it doesn't help. Following a Red Hat KB entry, I tried to failover the mgr, stopped an restarted the upgrade but without any improvement. I have not seen anything in the logs, except that there is an INF entry every 10s about the destroyed OSD saying:

------

2025-04-24T21:30:54.161988+0000 mgr.ijc-mon1.yyfnhz (mgr.55376028) 14079 : cephadm [INF] osd.253 now down 2025-04-24T21:30:54.162601+0000 mgr.ijc-mon1.yyfnhz (mgr.55376028) 14080 : cephadm [INF] Daemon osd.253 on dig-osd4 was already removed 2025-04-24T21:30:54.164440+0000 mgr.ijc-mon1.yyfnhz (mgr.55376028) 14081 : cephadm [INF] Successfully destroyed old osd.253 on dig-osd4; ready for replacement 2025-04-24T21:30:54.164536+0000 mgr.ijc-mon1.yyfnhz (mgr.55376028) 14082 : cephadm [INF] Zapping devices for osd.253 on dig-osd4 -----

The message seems to be only for one of the 2 destroyed OSDs since I restarted the mgr. May this be the cause for the stucked upgrade? What can I do for fixing this?

Thanks in advance for any hint. Best regards,

Michel

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Have you increased the debug level for the mgr? It would surprise me if stray daemons would really block an upgrade. But debug logs might reveal something. And if it can be confirmed that the strays really block the upgrade, you could either remove the OSDs entirely (they are already drained) to continue upgrading, or create a tracker issue to report this and wait for instructions. Zitat von Michel Jouvin <michel.jouvin@ijclab.in2p3.fr>:

Hi Eugen,

Yes I stopped and restarted the upgrade several times already, in particular after failing over the mgr. And the only messages related are the upgrade started and upgrade canceled ones. Nothing related to an error or a crash...

For me the question is why do I have stray daemons after removing OSD. IMO it is unexpected as these daemons are not there anymore. I can understand that stray daemons prevent the upgrade to start if they are really strayed... And it would be nice if cephadm was giving a message about why the upgrade does not really start despite its status is "in progress"...

Best regards,

Michel Sent from my mobile Le 28 avril 2025 07:27:44 Eugen Block <eblock@nde.ag> a écrit :

...

Do you see anything in the mgr log? To get fresh logs I would cancel the upgrade (ceph orch upgrade stop) and then try again. A workaround could be to manually upgrade the mgr daemons by changing their unit.run file, but that would be my last resort. Btwm did you stop and start the upgrade after failing the mgr as well?

Zitat von Michel Jouvin <michel.jouvin@ijclab.in2p3.fr>:

...

Eugen,

Thanks for the hint. Here is the osd_remove_queue:

[root@ijc-mon1 ~]# ceph config-key get mgr/cephadm/osd_remove_queue|jq [ { "osd_id": 253, "started": true, "draining": false, "stopped": false, "replace": true, "force": false, "zap": true, "hostname": "dig-osd4", "drain_started_at": null, "drain_stopped_at": null, "drain_done_at": "2025-04-15T14:09:30.521534Z", "process_started_at": "2025-04-15T14:09:14.091592Z" }, { "osd_id": 381, "started": true, "draining": false, "stopped": false, "replace": true, "force": false, "zap": false, "hostname": "dig-osd6", "drain_started_at": "2025-04-23T11:56:09.864724Z", "drain_stopped_at": null, "drain_done_at": "2025-04-25T06:53:03.678729Z", "process_started_at": "2025-04-23T11:56:05.924923Z" } ]

It is not empty the two stray daemons are listed. Not sure it these entries are expected as I specified --replace... A similar issue was reported in https://tracker.ceph.com/issues/67018 so before Reef but the cause may be different. Still not clear for me how to get out of this, except may be replacing the OSDs but this will take some time...

Best regards,

Michel

Le 27/04/2025 à 10:21, Eugen Block a écrit :

...

Hi,

what's the current ceph status? Wasn't there a bug in early Reef versions preventing upgrades if there were removed OSDs in the queue? But IIRC, the cephadm module would crash. Can you check

ceph config-key get mgr/cephadm/osd_remove_queue

And then I would check the mgr log, maybe set it to a higher debug level to see what's blocking it.

Zitat von Michel Jouvin <michel.jouvin@ijclab.in2p3.fr>:

...

Hi,

I tried to restart all the mgrs (we have 3, 1 active, 2 standby) by executing 3 times the `ceph mgr fail`, no impact. I don't really understand why I get these stray daemons after doing a 'ceph orch osd rm --replace` but I think I have always seen this. I tried to mute rather than disable the stray daemon check but it doesn't help either. And I find strange this message every 10s about one of the destroyed OSD and only one, reporting it is down and already destroyed and saying it'll zap it (I think I didn't add --zap when I removed it as the underlying disk is dead).

I'm completely stuck with this upgrade and I don't remember having this kind of problems in previous upgrades with cephadm... Any idea where to look for the cause and/or how to fix it?

Best regards,

Michel

Le 24/04/2025 à 23:34, Michel Jouvin a écrit :

...

Hi,

I'm trying to upgrade a (cephadm) cluster from 18.2.2 to 18.2.6, using 'ceph orch upgrade'. When I enter the command 'ceph orch upgrade start --ceph-version 18.2.6', I receive a message saying that the upgrade has been initiated, with a similar message in the logs but nothing happens after this. 'ceph orch upgrade status' says:

-------

[root@ijc-mon1 ~]# ceph orch upgrade status { "target_image": "quay.io/ceph/ceph:v18.2.6", "in_progress": true, "which": "Upgrading all daemon types on all hosts", "services_complete": [], "progress": "", "message": "", "is_paused": false } -------

The first time I entered the command, the cluster status was HEALTH_WARN because of 2 stray daemons (caused by destroyed OSDs, rm --replace). I set mgr/cephadm/warn_on_stray_daemons to false to ignore these 2 daemons, the cluster is now HEALTH_OK but it doesn't help. Following a Red Hat KB entry, I tried to failover the mgr, stopped an restarted the upgrade but without any improvement. I have not seen anything in the logs, except that there is an INF entry every 10s about the destroyed OSD saying:

------

2025-04-24T21:30:54.161988+0000 mgr.ijc-mon1.yyfnhz (mgr.55376028) 14079 : cephadm [INF] osd.253 now down 2025-04-24T21:30:54.162601+0000 mgr.ijc-mon1.yyfnhz (mgr.55376028) 14080 : cephadm [INF] Daemon osd.253 on dig-osd4 was already removed 2025-04-24T21:30:54.164440+0000 mgr.ijc-mon1.yyfnhz (mgr.55376028) 14081 : cephadm [INF] Successfully destroyed old osd.253 on dig-osd4; ready for replacement 2025-04-24T21:30:54.164536+0000 mgr.ijc-mon1.yyfnhz (mgr.55376028) 14082 : cephadm [INF] Zapping devices for osd.253 on dig-osd4 -----

The message seems to be only for one of the 2 destroyed OSDs since I restarted the mgr. May this be the cause for the stucked upgrade? What can I do for fixing this?

Thanks in advance for any hint. Best regards,

Michel

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

I just tried this on a single-node virtual test cluster, deployed it with 18.2.2. Then I removed one OSD with --replace flag (no --zap, otherwise it would redeploy the OSD on that VM). Then I also see the stray daemon warning, but the upgrade from 18.2.2 to 18.2.6 finished successfully. That's why I don't think the stray daemon is the root cause here. I would suggest to scan monitor and cephadm logs as well. After the upgrade to 18.2.6 the stray warning cleared, btw. Zitat von Michel Jouvin <michel.jouvin@ijclab.in2p3.fr>:

Eugen,

As said in a previous message, I found a tracker issue with a similar problem: https://tracker.ceph.com/issues/67018, even if the cause may be different as it is in older versions than me. For some reasons the sequence of messages every 10s is now back on the 2 OSDs:

2025-04-28T10:00:28.226741+0200 mgr.dig-mon1.fownxo [INF] osd.253 now down 2025-04-28T10:00:28.227249+0200 mgr.dig-mon1.fownxo [INF] Daemon osd.253 on dig-osd4 was already removed 2025-04-28T10:00:28.228929+0200 mgr.dig-mon1.fownxo [INF] Successfully destroyed old osd.253 on dig-osd4; ready for replacement 2025-04-28T10:00:28.228994+0200 mgr.dig-mon1.fownxo [INF] Zapping devices for osd.253 on dig-osd4 2025-04-28T10:00:39.132028+0200 mgr.dig-mon1.fownxo [INF] osd.381 now down 2025-04-28T10:00:39.132599+0200 mgr.dig-mon1.fownxo [INF] Daemon osd.381 on dig-osd6 was already removed 2025-04-28T10:00:39.133424+0200 mgr.dig-mon1.fownxo [INF] Successfully destroyed old osd.381 on dig-osd6; ready for replacement

except that the "Zapping.." message is not present for the second OSD...

I tried to increase the mgr log verbosity with 'ceph tell mgr.dig-mon1.fownxo config set debug_mgr 20/20' and there stop/start the upgrade without any additonal message displayed.

Michel

Le 28/04/2025 à 09:20, Eugen Block a écrit :

...

Have you increased the debug level for the mgr? It would surprise me if stray daemons would really block an upgrade. But debug logs might reveal something. And if it can be confirmed that the strays really block the upgrade, you could either remove the OSDs entirely (they are already drained) to continue upgrading, or create a tracker issue to report this and wait for instructions.

Zitat von Michel Jouvin <michel.jouvin@ijclab.in2p3.fr>:

...

Hi Eugen,

Yes I stopped and restarted the upgrade several times already, in particular after failing over the mgr. And the only messages related are the upgrade started and upgrade canceled ones. Nothing related to an error or a crash...

For me the question is why do I have stray daemons after removing OSD. IMO it is unexpected as these daemons are not there anymore. I can understand that stray daemons prevent the upgrade to start if they are really strayed... And it would be nice if cephadm was giving a message about why the upgrade does not really start despite its status is "in progress"...

Best regards,

Michel Sent from my mobile Le 28 avril 2025 07:27:44 Eugen Block <eblock@nde.ag> a écrit :

...

Do you see anything in the mgr log? To get fresh logs I would cancel the upgrade (ceph orch upgrade stop) and then try again. A workaround could be to manually upgrade the mgr daemons by changing their unit.run file, but that would be my last resort. Btwm did you stop and start the upgrade after failing the mgr as well?

Zitat von Michel Jouvin <michel.jouvin@ijclab.in2p3.fr>:

...

Eugen,

Thanks for the hint. Here is the osd_remove_queue:

[root@ijc-mon1 ~]# ceph config-key get mgr/cephadm/osd_remove_queue|jq [  {    "osd_id": 253,    "started": true,    "draining": false,    "stopped": false,    "replace": true,    "force": false,    "zap": true,    "hostname": "dig-osd4",    "drain_started_at": null,    "drain_stopped_at": null,    "drain_done_at": "2025-04-15T14:09:30.521534Z",    "process_started_at": "2025-04-15T14:09:14.091592Z"  },  {    "osd_id": 381,    "started": true,    "draining": false,    "stopped": false,    "replace": true,    "force": false,    "zap": false,    "hostname": "dig-osd6",    "drain_started_at": "2025-04-23T11:56:09.864724Z",    "drain_stopped_at": null,    "drain_done_at": "2025-04-25T06:53:03.678729Z",    "process_started_at": "2025-04-23T11:56:05.924923Z"  } ]

It is not empty the two stray daemons are listed. Not sure it these entries are expected as I specified --replace... A similar issue was reported in https://tracker.ceph.com/issues/67018 so before Reef but the cause may be different. Still not clear for me how to get out of this, except may be replacing the OSDs but this will take some time...

Best regards,

Michel

Le 27/04/2025 à 10:21, Eugen Block a écrit :

...

Hi,

what's the current ceph status? Wasn't there a bug in early Reef versions preventing upgrades if there were removed OSDs in the queue? But IIRC, the cephadm module would crash. Can you check

ceph config-key get mgr/cephadm/osd_remove_queue

And then I would check the mgr log, maybe set it to a higher debug level to see what's blocking it.

Zitat von Michel Jouvin <michel.jouvin@ijclab.in2p3.fr>:

> Hi, > > I tried to restart all the mgrs (we have 3, 1 active, 2 standby) > by executing 3 times the `ceph mgr fail`, no impact. I don't > really understand why I get these stray daemons after doing a > 'ceph orch osd rm --replace` but I think I have always seen this. > I tried to mute rather than disable the stray daemon check but it > doesn't help either. And I find strange this message every 10s > about one of the destroyed OSD and only one, reporting it is down > and already destroyed and saying it'll zap it (I think I didn't > add --zap when I removed it as the underlying disk is dead). > > I'm completely stuck with this upgrade and I don't remember having > this kind of problems in previous upgrades with cephadm... Any > idea where to look for the cause and/or how to fix it? > > Best regards, > > Michel > > Le 24/04/2025 à 23:34, Michel Jouvin a écrit : >> Hi, >> >> I'm trying to upgrade a (cephadm) cluster from 18.2.2 to 18.2.6, >> using 'ceph orch upgrade'. When I enter the command 'ceph orch >> upgrade start --ceph-version 18.2.6', I receive a message saying >> that the upgrade has been initiated, with a similar message in >> the logs but nothing happens after this. 'ceph orch upgrade >> status' says: >> >> ------- >> >> [root@ijc-mon1 ~]# ceph orch upgrade status >> { >>    "target_image": "quay.io/ceph/ceph:v18.2.6", >>    "in_progress": true, >>    "which": "Upgrading all daemon types on all hosts", >>    "services_complete": [], >>    "progress": "", >>    "message": "", >>    "is_paused": false >> } >> ------- >> >> The first time I entered the command, the cluster status was >> HEALTH_WARN because of 2 stray daemons (caused by destroyed OSDs, >> rm --replace). I set mgr/cephadm/warn_on_stray_daemons to false >> to ignore these 2 daemons, the cluster is now HEALTH_OK but it >> doesn't help. Following a Red Hat KB entry, I tried to failover >> the mgr, stopped an restarted the upgrade but without any >> improvement. I have not seen anything in the logs, except that >> there is an INF entry every 10s about the destroyed OSD saying: >> >> ------ >> >> 2025-04-24T21:30:54.161988+0000 mgr.ijc-mon1.yyfnhz >> (mgr.55376028) 14079 : cephadm [INF] osd.253 now down >> 2025-04-24T21:30:54.162601+0000 mgr.ijc-mon1.yyfnhz >> (mgr.55376028) 14080 : cephadm [INF] Daemon osd.253 on dig-osd4 >> was already removed >> 2025-04-24T21:30:54.164440+0000 mgr.ijc-mon1.yyfnhz >> (mgr.55376028) 14081 : cephadm [INF] Successfully destroyed old >> osd.253 on dig-osd4; ready for replacement >> 2025-04-24T21:30:54.164536+0000 mgr.ijc-mon1.yyfnhz >> (mgr.55376028) 14082 : cephadm [INF] Zapping devices for osd.253 >> on dig-osd4 >> ----- >> >> The message seems to be only for one of the 2 destroyed OSDs >> since I restarted the mgr. May this be the cause for the stucked >> upgrade? What can I do for fixing this? >> >> Thanks in advance for any hint. Best regards, >> >> Michel >> > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Hi Michel, You need to turn on cephadm debugging as described here [1] in the documentation $ ceph config set mgr mgr/cephadm/log_to_cluster_level debug and then look for any hints with $ ceph -W cephadm --watch-debug or $ tail -f /var/log/ceph/$(ceph fsid)/ceph.cephadm.log (on the active MGR) when you start/stop the upgrade. Regards, Frédéric. [1] https://docs.ceph.com/en/reef/cephadm/operations/ ----- Le 28 Avr 25, à 12:52, Michel Jouvin michel.jouvin@ijclab.in2p3.fr a écrit :

Eugen,

Thanks for doing the test. I scanned all logs and cannot find anything except the message mentioned displayed every 10s about the removed OSDs that led me to think there is something not exactly as expected... No clue what...

Michel Sent from my mobile Le 28 avril 2025 12:43:23 Eugen Block <eblock@nde.ag> a écrit :

...

I just tried this on a single-node virtual test cluster, deployed it with 18.2.2. Then I removed one OSD with --replace flag (no --zap, otherwise it would redeploy the OSD on that VM). Then I also see the stray daemon warning, but the upgrade from 18.2.2 to 18.2.6 finished successfully. That's why I don't think the stray daemon is the root cause here. I would suggest to scan monitor and cephadm logs as well. After the upgrade to 18.2.6 the stray warning cleared, btw.

Zitat von Michel Jouvin <michel.jouvin@ijclab.in2p3.fr>:

...

Eugen,

As said in a previous message, I found a tracker issue with a similar problem: https://tracker.ceph.com/issues/67018, even if the cause may be different as it is in older versions than me. For some reasons the sequence of messages every 10s is now back on the 2 OSDs:

2025-04-28T10:00:28.226741+0200 mgr.dig-mon1.fownxo [INF] osd.253 now down 2025-04-28T10:00:28.227249+0200 mgr.dig-mon1.fownxo [INF] Daemon osd.253 on dig-osd4 was already removed 2025-04-28T10:00:28.228929+0200 mgr.dig-mon1.fownxo [INF] Successfully destroyed old osd.253 on dig-osd4; ready for replacement 2025-04-28T10:00:28.228994+0200 mgr.dig-mon1.fownxo [INF] Zapping devices for osd.253 on dig-osd4 2025-04-28T10:00:39.132028+0200 mgr.dig-mon1.fownxo [INF] osd.381 now down 2025-04-28T10:00:39.132599+0200 mgr.dig-mon1.fownxo [INF] Daemon osd.381 on dig-osd6 was already removed 2025-04-28T10:00:39.133424+0200 mgr.dig-mon1.fownxo [INF] Successfully destroyed old osd.381 on dig-osd6; ready for replacement

except that the "Zapping.." message is not present for the second OSD...

I tried to increase the mgr log verbosity with 'ceph tell mgr.dig-mon1.fownxo config set debug_mgr 20/20' and there stop/start the upgrade without any additonal message displayed.

Michel

Le 28/04/2025 à 09:20, Eugen Block a écrit :

...

Have you increased the debug level for the mgr? It would surprise me if stray daemons would really block an upgrade. But debug logs might reveal something. And if it can be confirmed that the strays really block the upgrade, you could either remove the OSDs entirely (they are already drained) to continue upgrading, or create a tracker issue to report this and wait for instructions.

Zitat von Michel Jouvin <michel.jouvin@ijclab.in2p3.fr>:

...

Hi Eugen,

Yes I stopped and restarted the upgrade several times already, in particular after failing over the mgr. And the only messages related are the upgrade started and upgrade canceled ones. Nothing related to an error or a crash...

For me the question is why do I have stray daemons after removing OSD. IMO it is unexpected as these daemons are not there anymore. I can understand that stray daemons prevent the upgrade to start if they are really strayed... And it would be nice if cephadm was giving a message about why the upgrade does not really start despite its status is "in progress"...

Best regards,

Michel Sent from my mobile Le 28 avril 2025 07:27:44 Eugen Block <eblock@nde.ag> a écrit :

...

Do you see anything in the mgr log? To get fresh logs I would cancel the upgrade (ceph orch upgrade stop) and then try again. A workaround could be to manually upgrade the mgr daemons by changing their unit.run file, but that would be my last resort. Btwm did you stop and start the upgrade after failing the mgr as well?

Zitat von Michel Jouvin <michel.jouvin@ijclab.in2p3.fr>:

> Eugen, > > Thanks for the hint. Here is the osd_remove_queue: > > [root@ijc-mon1 ~]# ceph config-key get mgr/cephadm/osd_remove_queue|jq > [ > { > "osd_id": 253, > "started": true, > "draining": false, > "stopped": false, > "replace": true, > "force": false, > "zap": true, > "hostname": "dig-osd4", > "drain_started_at": null, > "drain_stopped_at": null, > "drain_done_at": "2025-04-15T14:09:30.521534Z", > "process_started_at": "2025-04-15T14:09:14.091592Z" > }, > { > "osd_id": 381, > "started": true, > "draining": false, > "stopped": false, > "replace": true, > "force": false, > "zap": false, > "hostname": "dig-osd6", > "drain_started_at": "2025-04-23T11:56:09.864724Z", > "drain_stopped_at": null, > "drain_done_at": "2025-04-25T06:53:03.678729Z", > "process_started_at": "2025-04-23T11:56:05.924923Z" > } > ] > > It is not empty the two stray daemons are listed. Not sure it these > entries are expected as I specified --replace... A similar issue was > reported in https://tracker.ceph.com/issues/67018 so before Reef but > the cause may be different. Still not clear for me how to get out of > this, except may be replacing the OSDs but this will take some time... > > Best regards, > > Michel > > Le 27/04/2025 à 10:21, Eugen Block a écrit : >> Hi, >> >> what's the current ceph status? Wasn't there a bug in early Reef >> versions preventing upgrades if there were removed OSDs in the >> queue? But IIRC, the cephadm module would crash. Can you check >> >> ceph config-key get mgr/cephadm/osd_remove_queue >> >> And then I would check the mgr log, maybe set it to a higher debug >> level to see what's blocking it. >> >> Zitat von Michel Jouvin <michel.jouvin@ijclab.in2p3.fr>: >> >>> Hi, >>> >>> I tried to restart all the mgrs (we have 3, 1 active, 2 standby) >>> by executing 3 times the `ceph mgr fail`, no impact. I don't >>> really understand why I get these stray daemons after doing a >>> 'ceph orch osd rm --replace` but I think I have always seen this. >>> I tried to mute rather than disable the stray daemon check but it >>> doesn't help either. And I find strange this message every 10s >>> about one of the destroyed OSD and only one, reporting it is down >>> and already destroyed and saying it'll zap it (I think I didn't >>> add --zap when I removed it as the underlying disk is dead). >>> >>> I'm completely stuck with this upgrade and I don't remember having >>> this kind of problems in previous upgrades with cephadm... Any >>> idea where to look for the cause and/or how to fix it? >>> >>> Best regards, >>> >>> Michel >>> >>> Le 24/04/2025 à 23:34, Michel Jouvin a écrit : >>>> Hi, >>>> >>>> I'm trying to upgrade a (cephadm) cluster from 18.2.2 to 18.2.6, >>>> using 'ceph orch upgrade'. When I enter the command 'ceph orch >>>> upgrade start --ceph-version 18.2.6', I receive a message saying >>>> that the upgrade has been initiated, with a similar message in >>>> the logs but nothing happens after this. 'ceph orch upgrade >>>> status' says: >>>> >>>> ------- >>>> >>>> [root@ijc-mon1 ~]# ceph orch upgrade status >>>> { >>>> "target_image": "quay.io/ceph/ceph:v18.2.6", >>>> "in_progress": true, >>>> "which": "Upgrading all daemon types on all hosts", >>>> "services_complete": [], >>>> "progress": "", >>>> "message": "", >>>> "is_paused": false >>>> } >>>> ------- >>>> >>>> The first time I entered the command, the cluster status was >>>> HEALTH_WARN because of 2 stray daemons (caused by destroyed OSDs, >>>> rm --replace). I set mgr/cephadm/warn_on_stray_daemons to false >>>> to ignore these 2 daemons, the cluster is now HEALTH_OK but it >>>> doesn't help. Following a Red Hat KB entry, I tried to failover >>>> the mgr, stopped an restarted the upgrade but without any >>>> improvement. I have not seen anything in the logs, except that >>>> there is an INF entry every 10s about the destroyed OSD saying: >>>> >>>> ------ >>>> >>>> 2025-04-24T21:30:54.161988+0000 mgr.ijc-mon1.yyfnhz >>>> (mgr.55376028) 14079 : cephadm [INF] osd.253 now down >>>> 2025-04-24T21:30:54.162601+0000 mgr.ijc-mon1.yyfnhz >>>> (mgr.55376028) 14080 : cephadm [INF] Daemon osd.253 on dig-osd4 >>>> was already removed >>>> 2025-04-24T21:30:54.164440+0000 mgr.ijc-mon1.yyfnhz >>>> (mgr.55376028) 14081 : cephadm [INF] Successfully destroyed old >>>> osd.253 on dig-osd4; ready for replacement >>>> 2025-04-24T21:30:54.164536+0000 mgr.ijc-mon1.yyfnhz >>>> (mgr.55376028) 14082 : cephadm [INF] Zapping devices for osd.253 >>>> on dig-osd4 >>>> ----- >>>> >>>> The message seems to be only for one of the 2 destroyed OSDs >>>> since I restarted the mgr. May this be the cause for the stucked >>>> upgrade? What can I do for fixing this? >>>> >>>> Thanks in advance for any hint. Best regards, >>>> >>>> Michel >>>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users@ceph.io >>> To unsubscribe send an email to ceph-users-leave@ceph.io >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Hi, I had no time to really investigate more our problem yesterday. But I realized one issue that may explain the problem with osd.253: the underlying disk is so dead that it is no longer visible by the OS. Probably I added --zap when I did the 'ceph orch osd rm' and thus it is trying to do the zapping, fails as it doesn't find the disk and retries indefinitely... I remain a little bit surprise that this zapping error is not reported (without the traceback) at the INFO level and requires DEBUG to be seen but it is a detail. I'm surprised that Ceph is not giving up on zapping if it cannot access the device or did I miss something and there is a way to stop this process? May be it is a corner case that has been fixed/improved since 18.2.2... Anyway, the question remains: is there a way out of this problem (that seems the only reason for the upgrade not really starting) apart from getting the replacement device? Best regards, Michel Le 28/04/2025 à 18:19, Michel Jouvin a écrit :

Hi Frédéric,

Thanks for the command. I'm always looking at the wrong page of the doc! I looked at https://docs.ceph.com/en/latest/rados/troubleshooting/log-and-debug/ that list the Ceph subsystem and their default log level but there is no mention of cephadm there... After enabling cephadm debug log level and restarting the upgrade, I got the messages below. The only thing strange points to the problem with osd.253 where it tries to zap the device that was probably already zapped and thus cannot find the LV volume associated with osd.253. There is not really any other messages saying the impact on the upgrade but I guess it is the reason. What do you think ? And is there any way to fix it, other than replacing the OSD?

Best regards,

Michel

--------------------- cephadm debug level log -------------------------

2025-04-28T17:32:12.713746+0200 mgr.dig-mon1.fownxo [INF] Upgrade: Started with target quay.io/ceph/ceph:v18.2.6 2025-04-28T17:32:14.822030+0200 mgr.dig-mon1.fownxo [DBG] Refreshed host dig-osd4 devices (23) 2025-04-28T17:32:14.822550+0200 mgr.dig-mon1.fownxo [DBG] Finding OSDSpecs for host: <dig-osd4> 2025-04-28T17:32:14.822614+0200 mgr.dig-mon1.fownxo [DBG] Generating OSDSpec previews for [] 2025-04-28T17:32:14.822695+0200 mgr.dig-mon1.fownxo [DBG] Loading OSDSpec previews to HostCache for host <dig-osd4> 2025-04-28T17:32:14.985257+0200 mgr.dig-mon1.fownxo [DBG] mon_command: 'config generate-minimal-conf' -> 0 in 0.005s 2025-04-28T17:32:15.262102+0200 mgr.dig-mon1.fownxo [DBG] mon_command: 'auth get' -> 0 in 0.277s 2025-04-28T17:32:15.262751+0200 mgr.dig-mon1.fownxo [DBG] Combine hosts with existing daemons [] + new hosts.... (very long line)

2025-04-28T17:32:15.416491+0200 mgr.dig-mon1.fownxo [DBG] _update_paused_health 2025-04-28T17:32:17.314607+0200 mgr.dig-mon1.fownxo [DBG] mon_command: 'osd df' -> 0 in 0.064s 2025-04-28T17:32:17.637526+0200 mgr.dig-mon1.fownxo [DBG] mon_command: 'osd df' -> 0 in 0.320s 2025-04-28T17:32:17.645703+0200 mgr.dig-mon1.fownxo [DBG] 2 OSDs are scheduled for removal: [osd.381, osd.253] 2025-04-28T17:32:17.661910+0200 mgr.dig-mon1.fownxo [DBG] mon_command: 'osd df' -> 0 in 0.011s 2025-04-28T17:32:17.667068+0200 mgr.dig-mon1.fownxo [DBG] mon_command: 'osd safe-to-destroy' -> 0 in 0.002s 2025-04-28T17:32:17.667117+0200 mgr.dig-mon1.fownxo [DBG] cmd: osd safe-to-destroy returns: 2025-04-28T17:32:17.667164+0200 mgr.dig-mon1.fownxo [DBG] running cmd: osd down on ids [osd.381] 2025-04-28T17:32:17.667854+0200 mgr.dig-mon1.fownxo [DBG] mon_command: 'osd down' -> 0 in 0.001s 2025-04-28T17:32:17.667908+0200 mgr.dig-mon1.fownxo [INF] osd.381 now down 2025-04-28T17:32:17.668446+0200 mgr.dig-mon1.fownxo [INF] Daemon osd.381 on dig-osd6 was already removed 2025-04-28T17:32:17.669534+0200 mgr.dig-mon1.fownxo [DBG] mon_command: 'osd destroy-actual' -> 0 in 0.001s 2025-04-28T17:32:17.669675+0200 mgr.dig-mon1.fownxo [DBG] cmd: osd destroy-actual returns: 2025-04-28T17:32:17.669789+0200 mgr.dig-mon1.fownxo [INF] Successfully destroyed old osd.381 on dig-osd6; ready for replacement 2025-04-28T17:32:17.669874+0200 mgr.dig-mon1.fownxo [DBG] Removing osd.381 from the queue. 2025-04-28T17:32:17.680411+0200 mgr.dig-mon1.fownxo [DBG] mon_command: 'osd df' -> 0 in 0.010s 2025-04-28T17:32:17.685141+0200 mgr.dig-mon1.fownxo [DBG] mon_command: 'osd safe-to-destroy' -> 0 in 0.002s 2025-04-28T17:32:17.685190+0200 mgr.dig-mon1.fownxo [DBG] cmd: osd safe-to-destroy returns: 2025-04-28T17:32:17.685234+0200 mgr.dig-mon1.fownxo [DBG] running cmd: osd down on ids [osd.253] 2025-04-28T17:32:17.685710+0200 mgr.dig-mon1.fownxo [DBG] mon_command: 'osd down' -> 0 in 0.000s 2025-04-28T17:32:17.685759+0200 mgr.dig-mon1.fownxo [INF] osd.253 now down 2025-04-28T17:32:17.686186+0200 mgr.dig-mon1.fownxo [INF] Daemon osd.253 on dig-osd4 was already removed 2025-04-28T17:32:17.687068+0200 mgr.dig-mon1.fownxo [DBG] mon_command: 'osd destroy-actual' -> 0 in 0.001s 2025-04-28T17:32:17.687102+0200 mgr.dig-mon1.fownxo [DBG] cmd: osd destroy-actual returns: 2025-04-28T17:32:17.687141+0200 mgr.dig-mon1.fownxo [INF] Successfully destroyed old osd.253 on dig-osd4; ready for replacement 2025-04-28T17:32:17.687176+0200 mgr.dig-mon1.fownxo [INF] Zapping devices for osd.253 on dig-osd4 2025-04-28T17:32:17.687508+0200 mgr.dig-mon1.fownxo [DBG] _run_cephadm : command = ceph-volume 2025-04-28T17:32:17.687554+0200 mgr.dig-mon1.fownxo [DBG] _run_cephadm : args = ['--', 'lvm', 'zap', '--osd-id', '253', '--destroy'] 2025-04-28T17:32:17.687637+0200 mgr.dig-mon1.fownxo [DBG] osd container image quay.io/ceph/ceph@sha256:798f1b1e71ca1bbf76c687d8bcf5cd3e88640f044513ae55a0fb571502ae641f 2025-04-28T17:32:17.687677+0200 mgr.dig-mon1.fownxo [DBG] args: --image quay.io/ceph/ceph@sha256:798f1b1e71ca1bbf76c687d8bcf5cd3e88640f044513ae55a0fb571502ae641f --timeout 895 ceph-volume --fsid f5195e24-158c-11ee-b338-5ced8c61b074 -- lvm zap --osd-id 253 --destroy 2025-04-28T17:32:17.687733+0200 mgr.dig-mon1.fownxo [DBG] Running command: which python3 2025-04-28T17:32:17.731474+0200 mgr.dig-mon1.fownxo [DBG] Running command: /usr/bin/python3 /var/lib/ceph/f5195e24-158c-11ee-b338-5ced8c61b074/cephadm.2b9d7d139a9cb40289f2358faf49a109fc297c0a258bde893227c262c30bca8d --image quay.io/ceph/ceph@sha256:798f1b1e71ca1bbf76c687d8bcf5cd3e88640f044513ae55a0fb571502ae641f --timeout 895 ceph-volume --fsid f5195e24-158c-11ee-b338-5ced8c61b074 -- lvm zap --osd-id 253 --destroy 2025-04-28T17:32:20.406723+0200 mgr.dig-mon1.fownxo [DBG] code: 1 2025-04-28T17:32:20.406764+0200 mgr.dig-mon1.fownxo [DBG] err: Inferring config /var/lib/ceph/f5195e24-158c-11ee-b338-5ced8c61b074/config/ceph.conf Non-zero exit code 1 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:798f1b1e71ca1bbf76c687d8bcf5cd3e88640f044513ae55a0fb571502ae641f -e NODE_NAME=dig-osd4 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f5195e24-158c-11ee-b338-5ced8c61b074:/var/run/ceph:z -v /var/log/ceph/f5195e24-158c-11ee-b338-5ced8c61b074:/var/log/ceph:z -v /var/lib/ceph/f5195e24-158c-11ee-b338-5ced8c61b074/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f5195e24-158c-11ee-b338-5ced8c61b074/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpgtvcw4gk:/etc/ceph/ceph.conf:z quay.io/ceph/ceph@sha256:798f1b1e71ca1bbf76c687d8bcf5cd3e88640f044513ae55a0fb571502ae641f lvm zap --osd-id 253 --destroy /usr/bin/podman: stderr Traceback (most recent call last): /usr/bin/podman: stderr   File "/usr/sbin/ceph-volume", line 11, in <module> /usr/bin/podman: stderr     load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() /usr/bin/podman: stderr   File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ /usr/bin/podman: stderr     self.main(self.argv) /usr/bin/podman: stderr   File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc /usr/bin/podman: stderr     return f(*a, **kw) /usr/bin/podman: stderr   File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main /usr/bin/podman: stderr     terminal.dispatch(self.mapper, subcommand_args) /usr/bin/podman: stderr   File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch /usr/bin/podman: stderr     instance.main() /usr/bin/podman: stderr   File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main /usr/bin/podman: stderr     terminal.dispatch(self.mapper, self.argv) /usr/bin/podman: stderr   File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch /usr/bin/podman: stderr     instance.main() /usr/bin/podman: stderr   File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 403, in main /usr/bin/podman: stderr     self.zap_osd() /usr/bin/podman: stderr   File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root /usr/bin/podman: stderr     return func(*a, **kw) /usr/bin/podman: stderr   File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 301, in zap_osd /usr/bin/podman: stderr     devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) /usr/bin/podman: stderr   File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices /usr/bin/podman: stderr     '%s' % osd_id or osd_fsid) /usr/bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 253 Traceback (most recent call last):   File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main     return _run_code(code, main_globals, None,   File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code     exec(code, run_globals)   File "/var/lib/ceph/f5195e24-158c-11ee-b338-5ced8c61b074/cephadm.2b9d7d139a9cb40289f2358faf49a109fc297c0a258bde893227c262c30bca8d/__main__.py", line 10700, in <module>   File "/var/lib/ceph/f5195e24-158c-11ee-b338-5ced8c61b074/cephadm.2b9d7d139a9cb40289f2358faf49a109fc297c0a258bde893227c262c30bca8d/__main__.py", line 10688, in main   File "/var/lib/ceph/f5195e24-158c-11ee-b338-5ced8c61b074/cephadm.2b9d7d139a9cb40289f2358faf49a109fc297c0a258bde893227c262c30bca8d/__main__.py", line 2445, in _infer_config   File "/var/lib/ceph/f5195e24-158c-11ee-b338-5ced8c61b074/cephadm.2b9d7d139a9cb40289f2358faf49a109fc297c0a258bde893227c262c30bca8d/__main__.py", line 2361, in _infer_fsid   File "/var/lib/ceph/f5195e24-158c-11ee-b338-5ced8c61b074/cephadm.2b9d7d139a9cb40289f2358faf49a109fc297c0a258bde893227c262c30bca8d/__main__.py", line 2473, in _infer_image   File "/var/lib/ceph/f5195e24-158c-11ee-b338-5ced8c61b074/cephadm.2b9d7d139a9cb40289f2358faf49a109fc297c0a258bde893227c262c30bca8d/__main__.py", line 2348, in _validate_fsid   File "/var/lib/ceph/f5195e24-158c-11ee-b338-5ced8c61b074/cephadm.2b9d7d139a9cb40289f2358faf49a109fc297c0a258bde893227c262c30bca8d/__main__.py", line 6970, in command_ceph_volume   File "/var/lib/ceph/f5195e24-158c-11ee-b338-5ced8c61b074/cephadm.2b9d7d139a9cb40289f2358faf49a109fc297c0a258bde893227c262c30bca8d/__main__.py", line 2136, in call_throws RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:798f1b1e71ca1bbf76c687d8bcf5cd3e88640f044513ae55a0fb571502ae641f -e NODE_NAME=dig-osd4 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f5195e24-158c-11ee-b338-5ced8c61b074:/var/run/ceph:z -v /var/log/ceph/f5195e24-158c-11ee-b338-5ced8c61b074:/var/log/ceph:z -v /var/lib/ceph/f5195e24-158c-11ee-b338-5ced8c61b074/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f5195e24-158c-11ee-b338-5ced8c61b074/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpgtvcw4gk:/etc/ceph/ceph.conf:z quay.io/ceph/ceph@sha256:798f1b1e71ca1bbf76c687d8bcf5cd3e88640f044513ae55a0fb571502ae641f lvm zap --osd-id 253 --destroy 2025-04-28T17:32:20.409316+0200 mgr.dig-mon1.fownxo [DBG] serve loop sleep

-----------------------

Le 28/04/2025 à 14:00, Frédéric Nass a écrit :

...

Hi Michel,

You need to turn on cephadm debugging as described here [1] in the documentation

$ ceph config set mgr mgr/cephadm/log_to_cluster_level debug

and then look for any hints with

$ ceph -W cephadm --watch-debug

or

$ tail -f /var/log/ceph/$(ceph fsid)/ceph.cephadm.log      (on the active MGR)

when you start/stop the upgrade.

Regards, Frédéric.

[1] https://docs.ceph.com/en/reef/cephadm/operations/

----- Le 28 Avr 25, à 12:52, Michel Jouvin michel.jouvin@ijclab.in2p3.fr a écrit :

...

Eugen,

Thanks for doing the test. I scanned all logs and cannot find anything except the message mentioned displayed every 10s about the removed OSDs that led me to think there is something not exactly as expected... No clue what...

Michel Sent from my mobile Le 28 avril 2025 12:43:23 Eugen Block <eblock@nde.ag> a écrit :

...

I just tried this on a single-node virtual test cluster, deployed it with 18.2.2. Then I removed one OSD with --replace flag (no --zap, otherwise it would redeploy the OSD on that VM). Then I also see the stray daemon warning, but the upgrade from 18.2.2 to 18.2.6 finished successfully. That's why I don't think the stray daemon is the root cause here. I would suggest to scan monitor and cephadm logs as well. After the upgrade to 18.2.6 the stray warning cleared, btw.

Zitat von Michel Jouvin <michel.jouvin@ijclab.in2p3.fr>:

...

Eugen,

As said in a previous message, I found a tracker issue with a similar problem: https://tracker.ceph.com/issues/67018, even if the cause may be different as it is in older versions than me. For some reasons the sequence of messages every 10s is now back on the 2 OSDs:

2025-04-28T10:00:28.226741+0200 mgr.dig-mon1.fownxo [INF] osd.253 now down 2025-04-28T10:00:28.227249+0200 mgr.dig-mon1.fownxo [INF] Daemon osd.253 on dig-osd4 was already removed 2025-04-28T10:00:28.228929+0200 mgr.dig-mon1.fownxo [INF] Successfully destroyed old osd.253 on dig-osd4; ready for replacement 2025-04-28T10:00:28.228994+0200 mgr.dig-mon1.fownxo [INF] Zapping devices for osd.253 on dig-osd4 2025-04-28T10:00:39.132028+0200 mgr.dig-mon1.fownxo [INF] osd.381 now down 2025-04-28T10:00:39.132599+0200 mgr.dig-mon1.fownxo [INF] Daemon osd.381 on dig-osd6 was already removed 2025-04-28T10:00:39.133424+0200 mgr.dig-mon1.fownxo [INF] Successfully destroyed old osd.381 on dig-osd6; ready for replacement

except that the "Zapping.." message is not present for the second OSD...

I tried to increase the mgr log verbosity with 'ceph tell mgr.dig-mon1.fownxo config set debug_mgr 20/20' and there stop/start the upgrade without any additonal message displayed.

Michel

Le 28/04/2025 à 09:20, Eugen Block a écrit :

...

Have you increased the debug level for the mgr? It would surprise me if stray daemons would really block an upgrade. But debug logs might reveal something. And if it can be confirmed that the strays really block the upgrade, you could either remove the OSDs entirely (they are already drained) to continue upgrading, or create a tracker issue to report this and wait for instructions.

Zitat von Michel Jouvin <michel.jouvin@ijclab.in2p3.fr>:

> Hi Eugen, > > Yes I stopped and restarted the upgrade several times already, in > particular after failing over the mgr. And the only messages > related are the upgrade started and upgrade canceled ones. Nothing > related to an error or a crash... > > For me the question is why do I have stray daemons after removing > OSD. IMO it is unexpected as these daemons are not there anymore. > I can understand that stray daemons prevent the upgrade to start > if they are really strayed... And it would be nice if cephadm was > giving a message about why the upgrade does not really start > despite its status is "in progress"... > > Best regards, > > Michel > Sent from my mobile > Le 28 avril 2025 07:27:44 Eugen Block <eblock@nde.ag> a écrit : > >> Do you see anything in the mgr log? To get fresh logs I would >> cancel >> the upgrade (ceph orch upgrade stop) and then try again. >> A workaround could be to manually upgrade the mgr daemons by >> changing >> their unit.run file, but that would be my last resort. Btwm did >> you >> stop and start the upgrade after failing the mgr as well? >> >> Zitat von Michel Jouvin <michel.jouvin@ijclab.in2p3.fr>: >> >>> Eugen, >>> >>> Thanks for the hint. Here is the osd_remove_queue: >>> >>> [root@ijc-mon1 ~]# ceph config-key get >>> mgr/cephadm/osd_remove_queue|jq >>> [ >>>   { >>>     "osd_id": 253, >>>     "started": true, >>>     "draining": false, >>>     "stopped": false, >>>     "replace": true, >>>     "force": false, >>>     "zap": true, >>>     "hostname": "dig-osd4", >>>     "drain_started_at": null, >>>     "drain_stopped_at": null, >>>     "drain_done_at": "2025-04-15T14:09:30.521534Z", >>>     "process_started_at": "2025-04-15T14:09:14.091592Z" >>>   }, >>>   { >>>     "osd_id": 381, >>>     "started": true, >>>     "draining": false, >>>     "stopped": false, >>>     "replace": true, >>>     "force": false, >>>     "zap": false, >>>     "hostname": "dig-osd6", >>>     "drain_started_at": "2025-04-23T11:56:09.864724Z", >>>     "drain_stopped_at": null, >>>     "drain_done_at": "2025-04-25T06:53:03.678729Z", >>>     "process_started_at": "2025-04-23T11:56:05.924923Z" >>>   } >>> ] >>> >>> It is not empty the two stray daemons are listed. Not sure it >>> these >>> entries are expected as I specified --replace... A similar >>> issue was >>> reported in https://tracker.ceph.com/issues/67018 so before >>> Reef but >>> the cause may be different. Still not clear for me how to get >>> out of >>> this, except may be replacing the OSDs but this will take some >>> time... >>> >>> Best regards, >>> >>> Michel >>> >>> Le 27/04/2025 à 10:21, Eugen Block a écrit : >>>> Hi, >>>> >>>> what's the current ceph status? Wasn't there a bug in early Reef >>>> versions preventing upgrades if there were removed OSDs in the >>>> queue? But IIRC, the cephadm module would crash. Can you check >>>> >>>> ceph config-key get mgr/cephadm/osd_remove_queue >>>> >>>> And then I would check the mgr log, maybe set it to a higher >>>> debug >>>> level to see what's blocking it. >>>> >>>> Zitat von Michel Jouvin <michel.jouvin@ijclab.in2p3.fr>: >>>> >>>>> Hi, >>>>> >>>>> I tried to restart all the mgrs (we have 3, 1 active, 2 >>>>> standby) >>>>> by executing 3 times the `ceph mgr fail`, no impact. I don't >>>>> really understand why I get these stray daemons after doing a >>>>> 'ceph orch osd rm --replace` but I think I have always seen >>>>> this. >>>>> I tried to mute rather than disable the stray daemon check >>>>> but it >>>>> doesn't help either. And I find strange this message every 10s >>>>> about one of the destroyed OSD and only one, reporting it is >>>>> down >>>>> and already destroyed and saying it'll zap it (I think I didn't >>>>> add --zap when I removed it as the underlying disk is dead). >>>>> >>>>> I'm completely stuck with this upgrade and I don't remember >>>>> having >>>>> this kind of problems in previous upgrades with cephadm... Any >>>>> idea where to look for the cause and/or how to fix it? >>>>> >>>>> Best regards, >>>>> >>>>> Michel >>>>> >>>>> Le 24/04/2025 à 23:34, Michel Jouvin a écrit : >>>>>> Hi, >>>>>> >>>>>> I'm trying to upgrade a (cephadm) cluster from 18.2.2 to >>>>>> 18.2.6, >>>>>> using 'ceph orch upgrade'. When I enter the command 'ceph orch >>>>>> upgrade start --ceph-version 18.2.6', I receive a message >>>>>> saying >>>>>> that the upgrade has been initiated, with a similar message in >>>>>> the logs but nothing happens after this. 'ceph orch upgrade >>>>>> status' says: >>>>>> >>>>>> ------- >>>>>> >>>>>> [root@ijc-mon1 ~]# ceph orch upgrade status >>>>>> { >>>>>>     "target_image": "quay.io/ceph/ceph:v18.2.6", >>>>>>     "in_progress": true, >>>>>>     "which": "Upgrading all daemon types on all hosts", >>>>>>     "services_complete": [], >>>>>>     "progress": "", >>>>>>     "message": "", >>>>>>     "is_paused": false >>>>>> } >>>>>> ------- >>>>>> >>>>>> The first time I entered the command, the cluster status was >>>>>> HEALTH_WARN because of 2 stray daemons (caused by destroyed >>>>>> OSDs, >>>>>> rm --replace). I set mgr/cephadm/warn_on_stray_daemons to >>>>>> false >>>>>> to ignore these 2 daemons, the cluster is now HEALTH_OK but it >>>>>> doesn't help. Following a Red Hat KB entry, I tried to >>>>>> failover >>>>>> the mgr, stopped an restarted the upgrade but without any >>>>>> improvement. I have not seen anything in the logs, except that >>>>>> there is an INF entry every 10s about the destroyed OSD >>>>>> saying: >>>>>> >>>>>> ------ >>>>>> >>>>>> 2025-04-24T21:30:54.161988+0000 mgr.ijc-mon1.yyfnhz >>>>>> (mgr.55376028) 14079 : cephadm [INF] osd.253 now down >>>>>> 2025-04-24T21:30:54.162601+0000 mgr.ijc-mon1.yyfnhz >>>>>> (mgr.55376028) 14080 : cephadm [INF] Daemon osd.253 on >>>>>> dig-osd4 >>>>>> was already removed >>>>>> 2025-04-24T21:30:54.164440+0000 mgr.ijc-mon1.yyfnhz >>>>>> (mgr.55376028) 14081 : cephadm [INF] Successfully destroyed >>>>>> old >>>>>> osd.253 on dig-osd4; ready for replacement >>>>>> 2025-04-24T21:30:54.164536+0000 mgr.ijc-mon1.yyfnhz >>>>>> (mgr.55376028) 14082 : cephadm [INF] Zapping devices for >>>>>> osd.253 >>>>>> on dig-osd4 >>>>>> ----- >>>>>> >>>>>> The message seems to be only for one of the 2 destroyed OSDs >>>>>> since I restarted the mgr. May this be the cause for the >>>>>> stucked >>>>>> upgrade? What can I do for fixing this? >>>>>> >>>>>> Thanks in advance for any hint. Best regards, >>>>>> >>>>>> Michel >>>>>> >>>>> _______________________________________________ >>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>> >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users@ceph.io >>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users@ceph.io >>> To unsubscribe send an email to ceph-users-leave@ceph.io >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Eugen, Thanks, I forgot that operation started with the orchestrator can be stopped. You were right: stopping the 'osd rm' was enough to unblock the upgrade. I am not completely sure what is the consequence on the replace flag: I have the feeling it has been lost somehow as the OSD is no longer listed by 'ceph orch osd rm status' and 'ceph -s' reports now one OSD down and 1 stray daemon instead of 2 stray daemons. Michel Le 30/04/2025 à 09:24, Eugen Block a écrit :

You can stop the osd removal:

ceph orch osd rm stop <OSD_ID>

I'm not entirely sure what the orchestrator will do except for clearing the pending state, and since the OSDs are already marked as destroyed in the crush tree, I wouldn't expect anything weird. But it's worth a try, I guess.

Zitat von Michel Jouvin <michel.jouvin@ijclab.in2p3.fr>:

...

Hi,

I had no time to really investigate more our problem yesterday. But I realized one issue that may explain the problem with osd.253: the underlying disk is so dead that it is no longer visible by the OS. Probably I added --zap when I did the 'ceph orch osd rm' and thus it is trying to do the zapping, fails as it doesn't find the disk and retries indefinitely... I remain a little bit surprise that this zapping error is not reported (without the traceback) at the INFO level and requires DEBUG to be seen but it is a detail. I'm surprised that Ceph is not giving up on zapping if it cannot access the device or did I miss something and there is a way to stop this process?

May be it is a corner case that has been fixed/improved since 18.2.2... Anyway, the question remains: is there a way out of this problem (that seems the only reason for the upgrade not really starting) apart from getting the replacement device?

Best regards,

Michel

Le 28/04/2025 à 18:19, Michel Jouvin a écrit :

...

Hi Frédéric,

Thanks for the command. I'm always looking at the wrong page of the doc! I looked at https://docs.ceph.com/en/latest/rados/troubleshooting/log-and-debug/ that list the Ceph subsystem and their default log level but there is no mention of cephadm there... After enabling cephadm debug log level and restarting the upgrade, I got the messages below. The only thing strange points to the problem with osd.253 where it tries to zap the device that was probably already zapped and thus cannot find the LV volume associated with osd.253. There is not really any other messages saying the impact on the upgrade but I guess it is the reason. What do you think ? And is there any way to fix it, other than replacing the OSD?

Best regards,

Michel

--------------------- cephadm debug level log -------------------------

2025-04-28T17:32:12.713746+0200 mgr.dig-mon1.fownxo [INF] Upgrade: Started with target quay.io/ceph/ceph:v18.2.6 2025-04-28T17:32:14.822030+0200 mgr.dig-mon1.fownxo [DBG] Refreshed host dig-osd4 devices (23) 2025-04-28T17:32:14.822550+0200 mgr.dig-mon1.fownxo [DBG] Finding OSDSpecs for host: <dig-osd4> 2025-04-28T17:32:14.822614+0200 mgr.dig-mon1.fownxo [DBG] Generating OSDSpec previews for [] 2025-04-28T17:32:14.822695+0200 mgr.dig-mon1.fownxo [DBG] Loading OSDSpec previews to HostCache for host <dig-osd4> 2025-04-28T17:32:14.985257+0200 mgr.dig-mon1.fownxo [DBG] mon_command: 'config generate-minimal-conf' -> 0 in 0.005s 2025-04-28T17:32:15.262102+0200 mgr.dig-mon1.fownxo [DBG] mon_command: 'auth get' -> 0 in 0.277s 2025-04-28T17:32:15.262751+0200 mgr.dig-mon1.fownxo [DBG] Combine hosts with existing daemons [] + new hosts.... (very long line)

2025-04-28T17:32:15.416491+0200 mgr.dig-mon1.fownxo [DBG] _update_paused_health 2025-04-28T17:32:17.314607+0200 mgr.dig-mon1.fownxo [DBG] mon_command: 'osd df' -> 0 in 0.064s 2025-04-28T17:32:17.637526+0200 mgr.dig-mon1.fownxo [DBG] mon_command: 'osd df' -> 0 in 0.320s 2025-04-28T17:32:17.645703+0200 mgr.dig-mon1.fownxo [DBG] 2 OSDs are scheduled for removal: [osd.381, osd.253] 2025-04-28T17:32:17.661910+0200 mgr.dig-mon1.fownxo [DBG] mon_command: 'osd df' -> 0 in 0.011s 2025-04-28T17:32:17.667068+0200 mgr.dig-mon1.fownxo [DBG] mon_command: 'osd safe-to-destroy' -> 0 in 0.002s 2025-04-28T17:32:17.667117+0200 mgr.dig-mon1.fownxo [DBG] cmd: osd safe-to-destroy returns: 2025-04-28T17:32:17.667164+0200 mgr.dig-mon1.fownxo [DBG] running cmd: osd down on ids [osd.381] 2025-04-28T17:32:17.667854+0200 mgr.dig-mon1.fownxo [DBG] mon_command: 'osd down' -> 0 in 0.001s 2025-04-28T17:32:17.667908+0200 mgr.dig-mon1.fownxo [INF] osd.381 now down 2025-04-28T17:32:17.668446+0200 mgr.dig-mon1.fownxo [INF] Daemon osd.381 on dig-osd6 was already removed 2025-04-28T17:32:17.669534+0200 mgr.dig-mon1.fownxo [DBG] mon_command: 'osd destroy-actual' -> 0 in 0.001s 2025-04-28T17:32:17.669675+0200 mgr.dig-mon1.fownxo [DBG] cmd: osd destroy-actual returns: 2025-04-28T17:32:17.669789+0200 mgr.dig-mon1.fownxo [INF] Successfully destroyed old osd.381 on dig-osd6; ready for replacement 2025-04-28T17:32:17.669874+0200 mgr.dig-mon1.fownxo [DBG] Removing osd.381 from the queue. 2025-04-28T17:32:17.680411+0200 mgr.dig-mon1.fownxo [DBG] mon_command: 'osd df' -> 0 in 0.010s 2025-04-28T17:32:17.685141+0200 mgr.dig-mon1.fownxo [DBG] mon_command: 'osd safe-to-destroy' -> 0 in 0.002s 2025-04-28T17:32:17.685190+0200 mgr.dig-mon1.fownxo [DBG] cmd: osd safe-to-destroy returns: 2025-04-28T17:32:17.685234+0200 mgr.dig-mon1.fownxo [DBG] running cmd: osd down on ids [osd.253] 2025-04-28T17:32:17.685710+0200 mgr.dig-mon1.fownxo [DBG] mon_command: 'osd down' -> 0 in 0.000s 2025-04-28T17:32:17.685759+0200 mgr.dig-mon1.fownxo [INF] osd.253 now down 2025-04-28T17:32:17.686186+0200 mgr.dig-mon1.fownxo [INF] Daemon osd.253 on dig-osd4 was already removed 2025-04-28T17:32:17.687068+0200 mgr.dig-mon1.fownxo [DBG] mon_command: 'osd destroy-actual' -> 0 in 0.001s 2025-04-28T17:32:17.687102+0200 mgr.dig-mon1.fownxo [DBG] cmd: osd destroy-actual returns: 2025-04-28T17:32:17.687141+0200 mgr.dig-mon1.fownxo [INF] Successfully destroyed old osd.253 on dig-osd4; ready for replacement 2025-04-28T17:32:17.687176+0200 mgr.dig-mon1.fownxo [INF] Zapping devices for osd.253 on dig-osd4 2025-04-28T17:32:17.687508+0200 mgr.dig-mon1.fownxo [DBG] _run_cephadm : command = ceph-volume 2025-04-28T17:32:17.687554+0200 mgr.dig-mon1.fownxo [DBG] _run_cephadm : args = ['--', 'lvm', 'zap', '--osd-id', '253', '--destroy'] 2025-04-28T17:32:17.687637+0200 mgr.dig-mon1.fownxo [DBG] osd container image quay.io/ceph/ceph@sha256:798f1b1e71ca1bbf76c687d8bcf5cd3e88640f044513ae55a0fb571502ae641f 2025-04-28T17:32:17.687677+0200 mgr.dig-mon1.fownxo [DBG] args: --image quay.io/ceph/ceph@sha256:798f1b1e71ca1bbf76c687d8bcf5cd3e88640f044513ae55a0fb571502ae641f --timeout 895 ceph-volume --fsid f5195e24-158c-11ee-b338-5ced8c61b074 -- lvm zap --osd-id 253 --destroy 2025-04-28T17:32:17.687733+0200 mgr.dig-mon1.fownxo [DBG] Running command: which python3 2025-04-28T17:32:17.731474+0200 mgr.dig-mon1.fownxo [DBG] Running command: /usr/bin/python3 /var/lib/ceph/f5195e24-158c-11ee-b338-5ced8c61b074/cephadm.2b9d7d139a9cb40289f2358faf49a109fc297c0a258bde893227c262c30bca8d --image quay.io/ceph/ceph@sha256:798f1b1e71ca1bbf76c687d8bcf5cd3e88640f044513ae55a0fb571502ae641f --timeout 895 ceph-volume --fsid f5195e24-158c-11ee-b338-5ced8c61b074 -- lvm zap --osd-id 253 --destroy 2025-04-28T17:32:20.406723+0200 mgr.dig-mon1.fownxo [DBG] code: 1 2025-04-28T17:32:20.406764+0200 mgr.dig-mon1.fownxo [DBG] err: Inferring config /var/lib/ceph/f5195e24-158c-11ee-b338-5ced8c61b074/config/ceph.conf Non-zero exit code 1 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:798f1b1e71ca1bbf76c687d8bcf5cd3e88640f044513ae55a0fb571502ae641f -e NODE_NAME=dig-osd4 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f5195e24-158c-11ee-b338-5ced8c61b074:/var/run/ceph:z -v /var/log/ceph/f5195e24-158c-11ee-b338-5ced8c61b074:/var/log/ceph:z -v /var/lib/ceph/f5195e24-158c-11ee-b338-5ced8c61b074/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f5195e24-158c-11ee-b338-5ced8c61b074/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpgtvcw4gk:/etc/ceph/ceph.conf:z quay.io/ceph/ceph@sha256:798f1b1e71ca1bbf76c687d8bcf5cd3e88640f044513ae55a0fb571502ae641f lvm zap --osd-id 253 --destroy /usr/bin/podman: stderr Traceback (most recent call last): /usr/bin/podman: stderr   File "/usr/sbin/ceph-volume", line 11, in <module> /usr/bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() /usr/bin/podman: stderr   File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ /usr/bin/podman: stderr     self.main(self.argv) /usr/bin/podman: stderr   File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc /usr/bin/podman: stderr     return f(*a, **kw) /usr/bin/podman: stderr   File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main /usr/bin/podman: stderr     terminal.dispatch(self.mapper, subcommand_args) /usr/bin/podman: stderr   File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch /usr/bin/podman: stderr     instance.main() /usr/bin/podman: stderr   File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main /usr/bin/podman: stderr     terminal.dispatch(self.mapper, self.argv) /usr/bin/podman: stderr   File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch /usr/bin/podman: stderr     instance.main() /usr/bin/podman: stderr   File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 403, in main /usr/bin/podman: stderr     self.zap_osd() /usr/bin/podman: stderr   File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root /usr/bin/podman: stderr     return func(*a, **kw) /usr/bin/podman: stderr   File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 301, in zap_osd /usr/bin/podman: stderr     devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) /usr/bin/podman: stderr   File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices /usr/bin/podman: stderr     '%s' % osd_id or osd_fsid) /usr/bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 253 Traceback (most recent call last):   File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main     return _run_code(code, main_globals, None,   File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code     exec(code, run_globals)   File "/var/lib/ceph/f5195e24-158c-11ee-b338-5ced8c61b074/cephadm.2b9d7d139a9cb40289f2358faf49a109fc297c0a258bde893227c262c30bca8d/__main__.py", line 10700, in <module>   File "/var/lib/ceph/f5195e24-158c-11ee-b338-5ced8c61b074/cephadm.2b9d7d139a9cb40289f2358faf49a109fc297c0a258bde893227c262c30bca8d/__main__.py", line 10688, in main   File "/var/lib/ceph/f5195e24-158c-11ee-b338-5ced8c61b074/cephadm.2b9d7d139a9cb40289f2358faf49a109fc297c0a258bde893227c262c30bca8d/__main__.py", line 2445, in _infer_config   File "/var/lib/ceph/f5195e24-158c-11ee-b338-5ced8c61b074/cephadm.2b9d7d139a9cb40289f2358faf49a109fc297c0a258bde893227c262c30bca8d/__main__.py", line 2361, in _infer_fsid   File "/var/lib/ceph/f5195e24-158c-11ee-b338-5ced8c61b074/cephadm.2b9d7d139a9cb40289f2358faf49a109fc297c0a258bde893227c262c30bca8d/__main__.py", line 2473, in _infer_image   File "/var/lib/ceph/f5195e24-158c-11ee-b338-5ced8c61b074/cephadm.2b9d7d139a9cb40289f2358faf49a109fc297c0a258bde893227c262c30bca8d/__main__.py", line 2348, in _validate_fsid   File "/var/lib/ceph/f5195e24-158c-11ee-b338-5ced8c61b074/cephadm.2b9d7d139a9cb40289f2358faf49a109fc297c0a258bde893227c262c30bca8d/__main__.py", line 6970, in command_ceph_volume   File "/var/lib/ceph/f5195e24-158c-11ee-b338-5ced8c61b074/cephadm.2b9d7d139a9cb40289f2358faf49a109fc297c0a258bde893227c262c30bca8d/__main__.py", line 2136, in call_throws RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:798f1b1e71ca1bbf76c687d8bcf5cd3e88640f044513ae55a0fb571502ae641f -e NODE_NAME=dig-osd4 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f5195e24-158c-11ee-b338-5ced8c61b074:/var/run/ceph:z -v /var/log/ceph/f5195e24-158c-11ee-b338-5ced8c61b074:/var/log/ceph:z -v /var/lib/ceph/f5195e24-158c-11ee-b338-5ced8c61b074/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f5195e24-158c-11ee-b338-5ced8c61b074/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpgtvcw4gk:/etc/ceph/ceph.conf:z quay.io/ceph/ceph@sha256:798f1b1e71ca1bbf76c687d8bcf5cd3e88640f044513ae55a0fb571502ae641f lvm zap --osd-id 253 --destroy 2025-04-28T17:32:20.409316+0200 mgr.dig-mon1.fownxo [DBG] serve loop sleep

-----------------------

Le 28/04/2025 à 14:00, Frédéric Nass a écrit :

...

Hi Michel,

You need to turn on cephadm debugging as described here [1] in the documentation

$ ceph config set mgr mgr/cephadm/log_to_cluster_level debug

and then look for any hints with

$ ceph -W cephadm --watch-debug

or

$ tail -f /var/log/ceph/$(ceph fsid)/ceph.cephadm.log (on the active MGR)

when you start/stop the upgrade.

Regards, Frédéric.

[1] https://docs.ceph.com/en/reef/cephadm/operations/

----- Le 28 Avr 25, à 12:52, Michel Jouvin michel.jouvin@ijclab.in2p3.fr a écrit :

...

Eugen,

Thanks for doing the test. I scanned all logs and cannot find anything except the message mentioned displayed every 10s about the removed OSDs that led me to think there is something not exactly as expected... No clue what...

Michel Sent from my mobile Le 28 avril 2025 12:43:23 Eugen Block <eblock@nde.ag> a écrit :

...

I just tried this on a single-node virtual test cluster, deployed it with 18.2.2. Then I removed one OSD with --replace flag (no --zap, otherwise it would redeploy the OSD on that VM). Then I also see the stray daemon warning, but the upgrade from 18.2.2 to 18.2.6 finished successfully. That's why I don't think the stray daemon is the root cause here. I would suggest to scan monitor and cephadm logs as well. After the upgrade to 18.2.6 the stray warning cleared, btw.

Zitat von Michel Jouvin <michel.jouvin@ijclab.in2p3.fr>:

> Eugen, > > As said in a previous message, I found a tracker issue with a > similar problem: https://tracker.ceph.com/issues/67018, even if the > cause may be different as it is in older versions than me. For some > reasons the sequence of messages every 10s is now back on the 2 > OSDs: > > 2025-04-28T10:00:28.226741+0200 mgr.dig-mon1.fownxo [INF] > osd.253 now down > 2025-04-28T10:00:28.227249+0200 mgr.dig-mon1.fownxo [INF] Daemon > osd.253 on dig-osd4 was already removed > 2025-04-28T10:00:28.228929+0200 mgr.dig-mon1.fownxo [INF] > Successfully destroyed old osd.253 on dig-osd4; ready for > replacement > 2025-04-28T10:00:28.228994+0200 mgr.dig-mon1.fownxo [INF] Zapping > devices for osd.253 on dig-osd4 > 2025-04-28T10:00:39.132028+0200 mgr.dig-mon1.fownxo [INF] > osd.381 now down > 2025-04-28T10:00:39.132599+0200 mgr.dig-mon1.fownxo [INF] Daemon > osd.381 on dig-osd6 was already removed > 2025-04-28T10:00:39.133424+0200 mgr.dig-mon1.fownxo [INF] > Successfully destroyed old osd.381 on dig-osd6; ready for > replacement > > except that the "Zapping.." message is not present for the > second OSD... > > I tried to increase the mgr log verbosity with 'ceph tell > mgr.dig-mon1.fownxo config set debug_mgr 20/20' and there > stop/start > the upgrade without any additonal message displayed. > > Michel > > Le 28/04/2025 à 09:20, Eugen Block a écrit : >> Have you increased the debug level for the mgr? It would surprise >> me if stray daemons would really block an upgrade. But debug logs >> might reveal something. And if it can be confirmed that the strays >> really block the upgrade, you could either remove the OSDs >> entirely >> (they are already drained) to continue upgrading, or create a >> tracker issue to report this and wait for instructions. >> >> Zitat von Michel Jouvin <michel.jouvin@ijclab.in2p3.fr>: >> >>> Hi Eugen, >>> >>> Yes I stopped and restarted the upgrade several times already, in >>> particular after failing over the mgr. And the only messages >>> related are the upgrade started and upgrade canceled ones. >>> Nothing >>> related to an error or a crash... >>> >>> For me the question is why do I have stray daemons after removing >>> OSD. IMO it is unexpected as these daemons are not there anymore. >>> I can understand that stray daemons prevent the upgrade to start >>> if they are really strayed... And it would be nice if cephadm was >>> giving a message about why the upgrade does not really start >>> despite its status is "in progress"... >>> >>> Best regards, >>> >>> Michel >>> Sent from my mobile >>> Le 28 avril 2025 07:27:44 Eugen Block <eblock@nde.ag> a écrit : >>> >>>> Do you see anything in the mgr log? To get fresh logs I would >>>> cancel >>>> the upgrade (ceph orch upgrade stop) and then try again. >>>> A workaround could be to manually upgrade the mgr daemons by >>>> changing >>>> their unit.run file, but that would be my last resort. Btwm >>>> did you >>>> stop and start the upgrade after failing the mgr as well? >>>> >>>> Zitat von Michel Jouvin <michel.jouvin@ijclab.in2p3.fr>: >>>> >>>>> Eugen, >>>>> >>>>> Thanks for the hint. Here is the osd_remove_queue: >>>>> >>>>> [root@ijc-mon1 ~]# ceph config-key get >>>>> mgr/cephadm/osd_remove_queue|jq >>>>> [ >>>>>   { >>>>>     "osd_id": 253, >>>>>     "started": true, >>>>>     "draining": false, >>>>>     "stopped": false, >>>>>     "replace": true, >>>>>     "force": false, >>>>>     "zap": true, >>>>>     "hostname": "dig-osd4", >>>>>     "drain_started_at": null, >>>>>     "drain_stopped_at": null, >>>>>     "drain_done_at": "2025-04-15T14:09:30.521534Z", >>>>>     "process_started_at": "2025-04-15T14:09:14.091592Z" >>>>>   }, >>>>>   { >>>>>     "osd_id": 381, >>>>>     "started": true, >>>>>     "draining": false, >>>>>     "stopped": false, >>>>>     "replace": true, >>>>>     "force": false, >>>>>     "zap": false, >>>>>     "hostname": "dig-osd6", >>>>>     "drain_started_at": "2025-04-23T11:56:09.864724Z", >>>>>     "drain_stopped_at": null, >>>>>     "drain_done_at": "2025-04-25T06:53:03.678729Z", >>>>>     "process_started_at": "2025-04-23T11:56:05.924923Z" >>>>>   } >>>>> ] >>>>> >>>>> It is not empty the two stray daemons are listed. Not sure >>>>> it these >>>>> entries are expected as I specified --replace... A similar >>>>> issue was >>>>> reported in https://tracker.ceph.com/issues/67018 so before >>>>> Reef but >>>>> the cause may be different. Still not clear for me how to >>>>> get out of >>>>> this, except may be replacing the OSDs but this will take >>>>> some time... >>>>> >>>>> Best regards, >>>>> >>>>> Michel >>>>> >>>>> Le 27/04/2025 à 10:21, Eugen Block a écrit : >>>>>> Hi, >>>>>> >>>>>> what's the current ceph status? Wasn't there a bug in early >>>>>> Reef >>>>>> versions preventing upgrades if there were removed OSDs in the >>>>>> queue? But IIRC, the cephadm module would crash. Can you check >>>>>> >>>>>> ceph config-key get mgr/cephadm/osd_remove_queue >>>>>> >>>>>> And then I would check the mgr log, maybe set it to a >>>>>> higher debug >>>>>> level to see what's blocking it. >>>>>> >>>>>> Zitat von Michel Jouvin <michel.jouvin@ijclab.in2p3.fr>: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I tried to restart all the mgrs (we have 3, 1 active, 2 >>>>>>> standby) >>>>>>> by executing 3 times the `ceph mgr fail`, no impact. I don't >>>>>>> really understand why I get these stray daemons after doing a >>>>>>> 'ceph orch osd rm --replace` but I think I have always >>>>>>> seen this. >>>>>>> I tried to mute rather than disable the stray daemon check >>>>>>> but it >>>>>>> doesn't help either. And I find strange this message every >>>>>>> 10s >>>>>>> about one of the destroyed OSD and only one, reporting it >>>>>>> is down >>>>>>> and already destroyed and saying it'll zap it (I think I >>>>>>> didn't >>>>>>> add --zap when I removed it as the underlying disk is dead). >>>>>>> >>>>>>> I'm completely stuck with this upgrade and I don't >>>>>>> remember having >>>>>>> this kind of problems in previous upgrades with cephadm... >>>>>>> Any >>>>>>> idea where to look for the cause and/or how to fix it? >>>>>>> >>>>>>> Best regards, >>>>>>> >>>>>>> Michel >>>>>>> >>>>>>> Le 24/04/2025 à 23:34, Michel Jouvin a écrit : >>>>>>>> Hi, >>>>>>>> >>>>>>>> I'm trying to upgrade a (cephadm) cluster from 18.2.2 to >>>>>>>> 18.2.6, >>>>>>>> using 'ceph orch upgrade'. When I enter the command 'ceph >>>>>>>> orch >>>>>>>> upgrade start --ceph-version 18.2.6', I receive a message >>>>>>>> saying >>>>>>>> that the upgrade has been initiated, with a similar >>>>>>>> message in >>>>>>>> the logs but nothing happens after this. 'ceph orch upgrade >>>>>>>> status' says: >>>>>>>> >>>>>>>> ------- >>>>>>>> >>>>>>>> [root@ijc-mon1 ~]# ceph orch upgrade status >>>>>>>> { >>>>>>>>     "target_image": "quay.io/ceph/ceph:v18.2.6", >>>>>>>>     "in_progress": true, >>>>>>>>     "which": "Upgrading all daemon types on all hosts", >>>>>>>>     "services_complete": [], >>>>>>>>     "progress": "", >>>>>>>>     "message": "", >>>>>>>>     "is_paused": false >>>>>>>> } >>>>>>>> ------- >>>>>>>> >>>>>>>> The first time I entered the command, the cluster status was >>>>>>>> HEALTH_WARN because of 2 stray daemons (caused by >>>>>>>> destroyed OSDs, >>>>>>>> rm --replace). I set mgr/cephadm/warn_on_stray_daemons to >>>>>>>> false >>>>>>>> to ignore these 2 daemons, the cluster is now HEALTH_OK >>>>>>>> but it >>>>>>>> doesn't help. Following a Red Hat KB entry, I tried to >>>>>>>> failover >>>>>>>> the mgr, stopped an restarted the upgrade but without any >>>>>>>> improvement. I have not seen anything in the logs, except >>>>>>>> that >>>>>>>> there is an INF entry every 10s about the destroyed OSD >>>>>>>> saying: >>>>>>>> >>>>>>>> ------ >>>>>>>> >>>>>>>> 2025-04-24T21:30:54.161988+0000 mgr.ijc-mon1.yyfnhz >>>>>>>> (mgr.55376028) 14079 : cephadm [INF] osd.253 now down >>>>>>>> 2025-04-24T21:30:54.162601+0000 mgr.ijc-mon1.yyfnhz >>>>>>>> (mgr.55376028) 14080 : cephadm [INF] Daemon osd.253 on >>>>>>>> dig-osd4 >>>>>>>> was already removed >>>>>>>> 2025-04-24T21:30:54.164440+0000 mgr.ijc-mon1.yyfnhz >>>>>>>> (mgr.55376028) 14081 : cephadm [INF] Successfully >>>>>>>> destroyed old >>>>>>>> osd.253 on dig-osd4; ready for replacement >>>>>>>> 2025-04-24T21:30:54.164536+0000 mgr.ijc-mon1.yyfnhz >>>>>>>> (mgr.55376028) 14082 : cephadm [INF] Zapping devices for >>>>>>>> osd.253 >>>>>>>> on dig-osd4 >>>>>>>> ----- >>>>>>>> >>>>>>>> The message seems to be only for one of the 2 destroyed OSDs >>>>>>>> since I restarted the mgr. May this be the cause for the >>>>>>>> stucked >>>>>>>> upgrade? What can I do for fixing this? >>>>>>>> >>>>>>>> Thanks in advance for any hint. Best regards, >>>>>>>> >>>>>>>> Michel >>>>>>>> >>>>>>> _______________________________________________ >>>>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>>> >>>>>> _______________________________________________ >>>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>> _______________________________________________ >>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>> >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users@ceph.io >>>> To unsubscribe send an email to ceph-users-leave@ceph.io >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Frédéric, My situation is a bit different I think. I had two malfunctionning OSDs that I removed with `ceph orch osd rm --replace --zap`: 1 was really dead and no longer seen by the OS (osd.253) and the other one with a lot of HW errors but still there (osd.381). Both have been successfully marked as destroyed in the CRUSH map. Just I didn't realize that cephadm was retrying every 10s to zap osd.253, getting an error as the disk could not be found. Looking at the removal status this morning(the removal was done ~2 weeks ago) with 'ceph orch osd rm status', I got: OSD  HOST      STATE                    PGS  REPLACE  FORCE ZAP    DRAIN STARTED AT 253  dig-osd4  done, waiting for purge    0  True     False  True 381  dig-osd6  done, waiting for purge    0  True     False False  2025-04-23 11:56:09.864724+00:00 I don't know what the status "waiting to purge" means... but we can see that cephadm considers that the drain never started for osd.253, as the device was unavailable I guess... What happens with dig-osd6 is less clear to me but may be the consequence that at some point the disk freed by the initial rm was picked up by cephadm and readded as the replacement OSD because we forgot to set the osd.all-available-devices service to unmanaged. The drain started on Apr 23 is the second rm I did after fixing osd.all-available-devices service. For this second attempt, I didn't specify --zap, not sure why ( a mistake!). I have the feeling but I may be wrong, that 'ceph osd destroy' will not help as they are already marked destroyed in the CRUSH map... I'm wondering wether I should do 'ceph orch osd rm stop 381'  as I did for 253 or whether it will impact the replacement later. Or said in a different way, is the replace flag something managed by cephadm and requiring the OSD to stay in the "rm queue" until the replacement is done? Best regards, Michel Le 30/04/2025 à 10:50, Frédéric Nass a écrit :

Hi Michel,

I've seen this recently on Reef (OSD stuck in the rm queue with the orchestrator tryng to zap a device that had already been zapped).

I could reproduce this a few times by deleting a batch of OSDs running on the same node. The whole 'ceph orch osd rm' process would stop progressing when trying to remove the ~8th OSD. I suspect that ceph-volume or the orchestrator is misinformed at some point that the device has already been zapped, looping over and over trying to remove this device that doesn't exist anymore.

I think you should now run 'ceph osd destroy <OSD_ID> --yes-i-really-mean-it'.

Regards, Frédéric.

----- Le 30 Avr 25, à 10:28, Michel Jouvin michel.jouvin@ijclab.in2p3.fr a écrit :

...

Eugen,

Thanks, I forgot that operation started with the orchestrator can be stopped. You were right: stopping the 'osd rm' was enough to unblock the upgrade. I am not completely sure what is the consequence on the replace flag: I have the feeling it has been lost somehow as the OSD is no longer listed by 'ceph orch osd rm status' and 'ceph -s' reports now one OSD down and 1 stray daemon instead of 2 stray daemons.

Michel

Le 30/04/2025 à 09:24, Eugen Block a écrit :

...

You can stop the osd removal:

ceph orch osd rm stop <OSD_ID>

I'm not entirely sure what the orchestrator will do except for clearing the pending state, and since the OSDs are already marked as destroyed in the crush tree, I wouldn't expect anything weird. But it's worth a try, I guess.

Zitat von Michel Jouvin <michel.jouvin@ijclab.in2p3.fr>:

...

Hi,

I had no time to really investigate more our problem yesterday. But I realized one issue that may explain the problem with osd.253: the underlying disk is so dead that it is no longer visible by the OS. Probably I added --zap when I did the 'ceph orch osd rm' and thus it is trying to do the zapping, fails as it doesn't find the disk and retries indefinitely... I remain a little bit surprise that this zapping error is not reported (without the traceback) at the INFO level and requires DEBUG to be seen but it is a detail. I'm surprised that Ceph is not giving up on zapping if it cannot access the device or did I miss something and there is a way to stop this process?

May be it is a corner case that has been fixed/improved since 18.2.2... Anyway, the question remains: is there a way out of this problem (that seems the only reason for the upgrade not really starting) apart from getting the replacement device?

Best regards,

Michel

Le 28/04/2025 à 18:19, Michel Jouvin a écrit :

...

Hi Frédéric,

Thanks for the command. I'm always looking at the wrong page of the doc! I looked at https://docs.ceph.com/en/latest/rados/troubleshooting/log-and-debug/ that list the Ceph subsystem and their default log level but there is no mention of cephadm there... After enabling cephadm debug log level and restarting the upgrade, I got the messages below. The only thing strange points to the problem with osd.253 where it tries to zap the device that was probably already zapped and thus cannot find the LV volume associated with osd.253. There is not really any other messages saying the impact on the upgrade but I guess it is the reason. What do you think ? And is there any way to fix it, other than replacing the OSD?

Best regards,

Michel

--------------------- cephadm debug level log -------------------------

2025-04-28T17:32:12.713746+0200 mgr.dig-mon1.fownxo [INF] Upgrade: Started with target quay.io/ceph/ceph:v18.2.6 2025-04-28T17:32:14.822030+0200 mgr.dig-mon1.fownxo [DBG] Refreshed host dig-osd4 devices (23) 2025-04-28T17:32:14.822550+0200 mgr.dig-mon1.fownxo [DBG] Finding OSDSpecs for host: <dig-osd4> 2025-04-28T17:32:14.822614+0200 mgr.dig-mon1.fownxo [DBG] Generating OSDSpec previews for [] 2025-04-28T17:32:14.822695+0200 mgr.dig-mon1.fownxo [DBG] Loading OSDSpec previews to HostCache for host <dig-osd4> 2025-04-28T17:32:14.985257+0200 mgr.dig-mon1.fownxo [DBG] mon_command: 'config generate-minimal-conf' -> 0 in 0.005s 2025-04-28T17:32:15.262102+0200 mgr.dig-mon1.fownxo [DBG] mon_command: 'auth get' -> 0 in 0.277s 2025-04-28T17:32:15.262751+0200 mgr.dig-mon1.fownxo [DBG] Combine hosts with existing daemons [] + new hosts.... (very long line)

2025-04-28T17:32:15.416491+0200 mgr.dig-mon1.fownxo [DBG] _update_paused_health 2025-04-28T17:32:17.314607+0200 mgr.dig-mon1.fownxo [DBG] mon_command: 'osd df' -> 0 in 0.064s 2025-04-28T17:32:17.637526+0200 mgr.dig-mon1.fownxo [DBG] mon_command: 'osd df' -> 0 in 0.320s 2025-04-28T17:32:17.645703+0200 mgr.dig-mon1.fownxo [DBG] 2 OSDs are scheduled for removal: [osd.381, osd.253] 2025-04-28T17:32:17.661910+0200 mgr.dig-mon1.fownxo [DBG] mon_command: 'osd df' -> 0 in 0.011s 2025-04-28T17:32:17.667068+0200 mgr.dig-mon1.fownxo [DBG] mon_command: 'osd safe-to-destroy' -> 0 in 0.002s 2025-04-28T17:32:17.667117+0200 mgr.dig-mon1.fownxo [DBG] cmd: osd safe-to-destroy returns: 2025-04-28T17:32:17.667164+0200 mgr.dig-mon1.fownxo [DBG] running cmd: osd down on ids [osd.381] 2025-04-28T17:32:17.667854+0200 mgr.dig-mon1.fownxo [DBG] mon_command: 'osd down' -> 0 in 0.001s 2025-04-28T17:32:17.667908+0200 mgr.dig-mon1.fownxo [INF] osd.381 now down 2025-04-28T17:32:17.668446+0200 mgr.dig-mon1.fownxo [INF] Daemon osd.381 on dig-osd6 was already removed 2025-04-28T17:32:17.669534+0200 mgr.dig-mon1.fownxo [DBG] mon_command: 'osd destroy-actual' -> 0 in 0.001s 2025-04-28T17:32:17.669675+0200 mgr.dig-mon1.fownxo [DBG] cmd: osd destroy-actual returns: 2025-04-28T17:32:17.669789+0200 mgr.dig-mon1.fownxo [INF] Successfully destroyed old osd.381 on dig-osd6; ready for replacement 2025-04-28T17:32:17.669874+0200 mgr.dig-mon1.fownxo [DBG] Removing osd.381 from the queue. 2025-04-28T17:32:17.680411+0200 mgr.dig-mon1.fownxo [DBG] mon_command: 'osd df' -> 0 in 0.010s 2025-04-28T17:32:17.685141+0200 mgr.dig-mon1.fownxo [DBG] mon_command: 'osd safe-to-destroy' -> 0 in 0.002s 2025-04-28T17:32:17.685190+0200 mgr.dig-mon1.fownxo [DBG] cmd: osd safe-to-destroy returns: 2025-04-28T17:32:17.685234+0200 mgr.dig-mon1.fownxo [DBG] running cmd: osd down on ids [osd.253] 2025-04-28T17:32:17.685710+0200 mgr.dig-mon1.fownxo [DBG] mon_command: 'osd down' -> 0 in 0.000s 2025-04-28T17:32:17.685759+0200 mgr.dig-mon1.fownxo [INF] osd.253 now down 2025-04-28T17:32:17.686186+0200 mgr.dig-mon1.fownxo [INF] Daemon osd.253 on dig-osd4 was already removed 2025-04-28T17:32:17.687068+0200 mgr.dig-mon1.fownxo [DBG] mon_command: 'osd destroy-actual' -> 0 in 0.001s 2025-04-28T17:32:17.687102+0200 mgr.dig-mon1.fownxo [DBG] cmd: osd destroy-actual returns: 2025-04-28T17:32:17.687141+0200 mgr.dig-mon1.fownxo [INF] Successfully destroyed old osd.253 on dig-osd4; ready for replacement 2025-04-28T17:32:17.687176+0200 mgr.dig-mon1.fownxo [INF] Zapping devices for osd.253 on dig-osd4 2025-04-28T17:32:17.687508+0200 mgr.dig-mon1.fownxo [DBG] _run_cephadm : command = ceph-volume 2025-04-28T17:32:17.687554+0200 mgr.dig-mon1.fownxo [DBG] _run_cephadm : args = ['--', 'lvm', 'zap', '--osd-id', '253', '--destroy'] 2025-04-28T17:32:17.687637+0200 mgr.dig-mon1.fownxo [DBG] osd container image quay.io/ceph/ceph@sha256:798f1b1e71ca1bbf76c687d8bcf5cd3e88640f044513ae55a0fb571502ae641f 2025-04-28T17:32:17.687677+0200 mgr.dig-mon1.fownxo [DBG] args: --image quay.io/ceph/ceph@sha256:798f1b1e71ca1bbf76c687d8bcf5cd3e88640f044513ae55a0fb571502ae641f --timeout 895 ceph-volume --fsid f5195e24-158c-11ee-b338-5ced8c61b074 -- lvm zap --osd-id 253 --destroy 2025-04-28T17:32:17.687733+0200 mgr.dig-mon1.fownxo [DBG] Running command: which python3 2025-04-28T17:32:17.731474+0200 mgr.dig-mon1.fownxo [DBG] Running command: /usr/bin/python3 /var/lib/ceph/f5195e24-158c-11ee-b338-5ced8c61b074/cephadm.2b9d7d139a9cb40289f2358faf49a109fc297c0a258bde893227c262c30bca8d --image quay.io/ceph/ceph@sha256:798f1b1e71ca1bbf76c687d8bcf5cd3e88640f044513ae55a0fb571502ae641f --timeout 895 ceph-volume --fsid f5195e24-158c-11ee-b338-5ced8c61b074 -- lvm zap --osd-id 253 --destroy 2025-04-28T17:32:20.406723+0200 mgr.dig-mon1.fownxo [DBG] code: 1 2025-04-28T17:32:20.406764+0200 mgr.dig-mon1.fownxo [DBG] err: Inferring config /var/lib/ceph/f5195e24-158c-11ee-b338-5ced8c61b074/config/ceph.conf Non-zero exit code 1 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:798f1b1e71ca1bbf76c687d8bcf5cd3e88640f044513ae55a0fb571502ae641f -e NODE_NAME=dig-osd4 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f5195e24-158c-11ee-b338-5ced8c61b074:/var/run/ceph:z -v /var/log/ceph/f5195e24-158c-11ee-b338-5ced8c61b074:/var/log/ceph:z -v /var/lib/ceph/f5195e24-158c-11ee-b338-5ced8c61b074/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f5195e24-158c-11ee-b338-5ced8c61b074/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpgtvcw4gk:/etc/ceph/ceph.conf:z quay.io/ceph/ceph@sha256:798f1b1e71ca1bbf76c687d8bcf5cd3e88640f044513ae55a0fb571502ae641f lvm zap --osd-id 253 --destroy /usr/bin/podman: stderr Traceback (most recent call last): /usr/bin/podman: stderr   File "/usr/sbin/ceph-volume", line 11, in <module> /usr/bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() /usr/bin/podman: stderr   File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ /usr/bin/podman: stderr     self.main(self.argv) /usr/bin/podman: stderr   File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc /usr/bin/podman: stderr     return f(*a, **kw) /usr/bin/podman: stderr   File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main /usr/bin/podman: stderr     terminal.dispatch(self.mapper, subcommand_args) /usr/bin/podman: stderr   File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch /usr/bin/podman: stderr     instance.main() /usr/bin/podman: stderr   File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main /usr/bin/podman: stderr     terminal.dispatch(self.mapper, self.argv) /usr/bin/podman: stderr   File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch /usr/bin/podman: stderr     instance.main() /usr/bin/podman: stderr   File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 403, in main /usr/bin/podman: stderr     self.zap_osd() /usr/bin/podman: stderr   File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root /usr/bin/podman: stderr     return func(*a, **kw) /usr/bin/podman: stderr   File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 301, in zap_osd /usr/bin/podman: stderr     devices = find_associated_devices(self.args.osd_id, self.args.osd_fsid) /usr/bin/podman: stderr   File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", line 88, in find_associated_devices /usr/bin/podman: stderr     '%s' % osd_id or osd_fsid) /usr/bin/podman: stderr RuntimeError: Unable to find any LV for zapping OSD: 253 Traceback (most recent call last):   File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main     return _run_code(code, main_globals, None,   File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code     exec(code, run_globals)   File "/var/lib/ceph/f5195e24-158c-11ee-b338-5ced8c61b074/cephadm.2b9d7d139a9cb40289f2358faf49a109fc297c0a258bde893227c262c30bca8d/__main__.py", line 10700, in <module>   File "/var/lib/ceph/f5195e24-158c-11ee-b338-5ced8c61b074/cephadm.2b9d7d139a9cb40289f2358faf49a109fc297c0a258bde893227c262c30bca8d/__main__.py", line 10688, in main   File "/var/lib/ceph/f5195e24-158c-11ee-b338-5ced8c61b074/cephadm.2b9d7d139a9cb40289f2358faf49a109fc297c0a258bde893227c262c30bca8d/__main__.py", line 2445, in _infer_config   File "/var/lib/ceph/f5195e24-158c-11ee-b338-5ced8c61b074/cephadm.2b9d7d139a9cb40289f2358faf49a109fc297c0a258bde893227c262c30bca8d/__main__.py", line 2361, in _infer_fsid   File "/var/lib/ceph/f5195e24-158c-11ee-b338-5ced8c61b074/cephadm.2b9d7d139a9cb40289f2358faf49a109fc297c0a258bde893227c262c30bca8d/__main__.py", line 2473, in _infer_image   File "/var/lib/ceph/f5195e24-158c-11ee-b338-5ced8c61b074/cephadm.2b9d7d139a9cb40289f2358faf49a109fc297c0a258bde893227c262c30bca8d/__main__.py", line 2348, in _validate_fsid   File "/var/lib/ceph/f5195e24-158c-11ee-b338-5ced8c61b074/cephadm.2b9d7d139a9cb40289f2358faf49a109fc297c0a258bde893227c262c30bca8d/__main__.py", line 6970, in command_ceph_volume   File "/var/lib/ceph/f5195e24-158c-11ee-b338-5ced8c61b074/cephadm.2b9d7d139a9cb40289f2358faf49a109fc297c0a258bde893227c262c30bca8d/__main__.py", line 2136, in call_throws RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:798f1b1e71ca1bbf76c687d8bcf5cd3e88640f044513ae55a0fb571502ae641f -e NODE_NAME=dig-osd4 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/f5195e24-158c-11ee-b338-5ced8c61b074:/var/run/ceph:z -v /var/log/ceph/f5195e24-158c-11ee-b338-5ced8c61b074:/var/log/ceph:z -v /var/lib/ceph/f5195e24-158c-11ee-b338-5ced8c61b074/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/f5195e24-158c-11ee-b338-5ced8c61b074/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpgtvcw4gk:/etc/ceph/ceph.conf:z quay.io/ceph/ceph@sha256:798f1b1e71ca1bbf76c687d8bcf5cd3e88640f044513ae55a0fb571502ae641f lvm zap --osd-id 253 --destroy 2025-04-28T17:32:20.409316+0200 mgr.dig-mon1.fownxo [DBG] serve loop sleep

-----------------------

Le 28/04/2025 à 14:00, Frédéric Nass a écrit :

...

Hi Michel,

You need to turn on cephadm debugging as described here [1] in the documentation

$ ceph config set mgr mgr/cephadm/log_to_cluster_level debug

and then look for any hints with

$ ceph -W cephadm --watch-debug

or

$ tail -f /var/log/ceph/$(ceph fsid)/ceph.cephadm.log (on the active MGR)

when you start/stop the upgrade.

Regards, Frédéric.

[1] https://docs.ceph.com/en/reef/cephadm/operations/

----- Le 28 Avr 25, à 12:52, Michel Jouvin michel.jouvin@ijclab.in2p3.fr a écrit :

> Eugen, > > Thanks for doing the test. I scanned all logs and cannot find > anything > except the message mentioned displayed every 10s about the removed > OSDs > that led me to think there is something not exactly as expected... > No clue > what... > > Michel > Sent from my mobile > Le 28 avril 2025 12:43:23 Eugen Block <eblock@nde.ag> a écrit : > >> I just tried this on a single-node virtual test cluster, deployed it >> with 18.2.2. Then I removed one OSD with --replace flag (no --zap, >> otherwise it would redeploy the OSD on that VM). Then I also see the >> stray daemon warning, but the upgrade from 18.2.2 to 18.2.6 finished >> successfully. That's why I don't think the stray daemon is the root >> cause here. I would suggest to scan monitor and cephadm logs as >> well. >> After the upgrade to 18.2.6 the stray warning cleared, btw. >> >> >> Zitat von Michel Jouvin <michel.jouvin@ijclab.in2p3.fr>: >> >>> Eugen, >>> >>> As said in a previous message, I found a tracker issue with a >>> similar problem: https://tracker.ceph.com/issues/67018, even if the >>> cause may be different as it is in older versions than me. For some >>> reasons the sequence of messages every 10s is now back on the 2 >>> OSDs: >>> >>> 2025-04-28T10:00:28.226741+0200 mgr.dig-mon1.fownxo [INF] >>> osd.253 now down >>> 2025-04-28T10:00:28.227249+0200 mgr.dig-mon1.fownxo [INF] Daemon >>> osd.253 on dig-osd4 was already removed >>> 2025-04-28T10:00:28.228929+0200 mgr.dig-mon1.fownxo [INF] >>> Successfully destroyed old osd.253 on dig-osd4; ready for >>> replacement >>> 2025-04-28T10:00:28.228994+0200 mgr.dig-mon1.fownxo [INF] Zapping >>> devices for osd.253 on dig-osd4 >>> 2025-04-28T10:00:39.132028+0200 mgr.dig-mon1.fownxo [INF] >>> osd.381 now down >>> 2025-04-28T10:00:39.132599+0200 mgr.dig-mon1.fownxo [INF] Daemon >>> osd.381 on dig-osd6 was already removed >>> 2025-04-28T10:00:39.133424+0200 mgr.dig-mon1.fownxo [INF] >>> Successfully destroyed old osd.381 on dig-osd6; ready for >>> replacement >>> >>> except that the "Zapping.." message is not present for the >>> second OSD... >>> >>> I tried to increase the mgr log verbosity with 'ceph tell >>> mgr.dig-mon1.fownxo config set debug_mgr 20/20' and there >>> stop/start >>> the upgrade without any additonal message displayed. >>> >>> Michel >>> >>> Le 28/04/2025 à 09:20, Eugen Block a écrit : >>>> Have you increased the debug level for the mgr? It would surprise >>>> me if stray daemons would really block an upgrade. But debug logs >>>> might reveal something. And if it can be confirmed that the strays >>>> really block the upgrade, you could either remove the OSDs >>>> entirely >>>> (they are already drained) to continue upgrading, or create a >>>> tracker issue to report this and wait for instructions. >>>> >>>> Zitat von Michel Jouvin <michel.jouvin@ijclab.in2p3.fr>: >>>> >>>>> Hi Eugen, >>>>> >>>>> Yes I stopped and restarted the upgrade several times already, in >>>>> particular after failing over the mgr. And the only messages >>>>> related are the upgrade started and upgrade canceled ones. >>>>> Nothing >>>>> related to an error or a crash... >>>>> >>>>> For me the question is why do I have stray daemons after removing >>>>> OSD. IMO it is unexpected as these daemons are not there anymore. >>>>> I can understand that stray daemons prevent the upgrade to start >>>>> if they are really strayed... And it would be nice if cephadm was >>>>> giving a message about why the upgrade does not really start >>>>> despite its status is "in progress"... >>>>> >>>>> Best regards, >>>>> >>>>> Michel >>>>> Sent from my mobile >>>>> Le 28 avril 2025 07:27:44 Eugen Block <eblock@nde.ag> a écrit : >>>>> >>>>>> Do you see anything in the mgr log? To get fresh logs I would >>>>>> cancel >>>>>> the upgrade (ceph orch upgrade stop) and then try again. >>>>>> A workaround could be to manually upgrade the mgr daemons by >>>>>> changing >>>>>> their unit.run file, but that would be my last resort. Btwm >>>>>> did you >>>>>> stop and start the upgrade after failing the mgr as well? >>>>>> >>>>>> Zitat von Michel Jouvin <michel.jouvin@ijclab.in2p3.fr>: >>>>>> >>>>>>> Eugen, >>>>>>> >>>>>>> Thanks for the hint. Here is the osd_remove_queue: >>>>>>> >>>>>>> [root@ijc-mon1 ~]# ceph config-key get >>>>>>> mgr/cephadm/osd_remove_queue|jq >>>>>>> [ >>>>>>>   { >>>>>>>     "osd_id": 253, >>>>>>>     "started": true, >>>>>>>     "draining": false, >>>>>>>     "stopped": false, >>>>>>>     "replace": true, >>>>>>>     "force": false, >>>>>>>     "zap": true, >>>>>>>     "hostname": "dig-osd4", >>>>>>>     "drain_started_at": null, >>>>>>>     "drain_stopped_at": null, >>>>>>>     "drain_done_at": "2025-04-15T14:09:30.521534Z", >>>>>>>     "process_started_at": "2025-04-15T14:09:14.091592Z" >>>>>>>   }, >>>>>>>   { >>>>>>>     "osd_id": 381, >>>>>>>     "started": true, >>>>>>>     "draining": false, >>>>>>>     "stopped": false, >>>>>>>     "replace": true, >>>>>>>     "force": false, >>>>>>>     "zap": false, >>>>>>>     "hostname": "dig-osd6", >>>>>>>     "drain_started_at": "2025-04-23T11:56:09.864724Z", >>>>>>>     "drain_stopped_at": null, >>>>>>>     "drain_done_at": "2025-04-25T06:53:03.678729Z", >>>>>>>     "process_started_at": "2025-04-23T11:56:05.924923Z" >>>>>>>   } >>>>>>> ] >>>>>>> >>>>>>> It is not empty the two stray daemons are listed. Not sure >>>>>>> it these >>>>>>> entries are expected as I specified --replace... A similar >>>>>>> issue was >>>>>>> reported in https://tracker.ceph.com/issues/67018 so before >>>>>>> Reef but >>>>>>> the cause may be different. Still not clear for me how to >>>>>>> get out of >>>>>>> this, except may be replacing the OSDs but this will take >>>>>>> some time... >>>>>>> >>>>>>> Best regards, >>>>>>> >>>>>>> Michel >>>>>>> >>>>>>> Le 27/04/2025 à 10:21, Eugen Block a écrit : >>>>>>>> Hi, >>>>>>>> >>>>>>>> what's the current ceph status? Wasn't there a bug in early >>>>>>>> Reef >>>>>>>> versions preventing upgrades if there were removed OSDs in the >>>>>>>> queue? But IIRC, the cephadm module would crash. Can you check >>>>>>>> >>>>>>>> ceph config-key get mgr/cephadm/osd_remove_queue >>>>>>>> >>>>>>>> And then I would check the mgr log, maybe set it to a >>>>>>>> higher debug >>>>>>>> level to see what's blocking it. >>>>>>>> >>>>>>>> Zitat von Michel Jouvin <michel.jouvin@ijclab.in2p3.fr>: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> I tried to restart all the mgrs (we have 3, 1 active, 2 >>>>>>>>> standby) >>>>>>>>> by executing 3 times the `ceph mgr fail`, no impact. I don't >>>>>>>>> really understand why I get these stray daemons after doing a >>>>>>>>> 'ceph orch osd rm --replace` but I think I have always >>>>>>>>> seen this. >>>>>>>>> I tried to mute rather than disable the stray daemon check >>>>>>>>> but it >>>>>>>>> doesn't help either. And I find strange this message every >>>>>>>>> 10s >>>>>>>>> about one of the destroyed OSD and only one, reporting it >>>>>>>>> is down >>>>>>>>> and already destroyed and saying it'll zap it (I think I >>>>>>>>> didn't >>>>>>>>> add --zap when I removed it as the underlying disk is dead). >>>>>>>>> >>>>>>>>> I'm completely stuck with this upgrade and I don't >>>>>>>>> remember having >>>>>>>>> this kind of problems in previous upgrades with cephadm... >>>>>>>>> Any >>>>>>>>> idea where to look for the cause and/or how to fix it? >>>>>>>>> >>>>>>>>> Best regards, >>>>>>>>> >>>>>>>>> Michel >>>>>>>>> >>>>>>>>> Le 24/04/2025 à 23:34, Michel Jouvin a écrit : >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> I'm trying to upgrade a (cephadm) cluster from 18.2.2 to >>>>>>>>>> 18.2.6, >>>>>>>>>> using 'ceph orch upgrade'. When I enter the command 'ceph >>>>>>>>>> orch >>>>>>>>>> upgrade start --ceph-version 18.2.6', I receive a message >>>>>>>>>> saying >>>>>>>>>> that the upgrade has been initiated, with a similar >>>>>>>>>> message in >>>>>>>>>> the logs but nothing happens after this. 'ceph orch upgrade >>>>>>>>>> status' says: >>>>>>>>>> >>>>>>>>>> ------- >>>>>>>>>> >>>>>>>>>> [root@ijc-mon1 ~]# ceph orch upgrade status >>>>>>>>>> { >>>>>>>>>>     "target_image": "quay.io/ceph/ceph:v18.2.6", >>>>>>>>>>     "in_progress": true, >>>>>>>>>>     "which": "Upgrading all daemon types on all hosts", >>>>>>>>>>     "services_complete": [], >>>>>>>>>>     "progress": "", >>>>>>>>>>     "message": "", >>>>>>>>>>     "is_paused": false >>>>>>>>>> } >>>>>>>>>> ------- >>>>>>>>>> >>>>>>>>>> The first time I entered the command, the cluster status was >>>>>>>>>> HEALTH_WARN because of 2 stray daemons (caused by >>>>>>>>>> destroyed OSDs, >>>>>>>>>> rm --replace). I set mgr/cephadm/warn_on_stray_daemons to >>>>>>>>>> false >>>>>>>>>> to ignore these 2 daemons, the cluster is now HEALTH_OK >>>>>>>>>> but it >>>>>>>>>> doesn't help. Following a Red Hat KB entry, I tried to >>>>>>>>>> failover >>>>>>>>>> the mgr, stopped an restarted the upgrade but without any >>>>>>>>>> improvement. I have not seen anything in the logs, except >>>>>>>>>> that >>>>>>>>>> there is an INF entry every 10s about the destroyed OSD >>>>>>>>>> saying: >>>>>>>>>> >>>>>>>>>> ------ >>>>>>>>>> >>>>>>>>>> 2025-04-24T21:30:54.161988+0000 mgr.ijc-mon1.yyfnhz >>>>>>>>>> (mgr.55376028) 14079 : cephadm [INF] osd.253 now down >>>>>>>>>> 2025-04-24T21:30:54.162601+0000 mgr.ijc-mon1.yyfnhz >>>>>>>>>> (mgr.55376028) 14080 : cephadm [INF] Daemon osd.253 on >>>>>>>>>> dig-osd4 >>>>>>>>>> was already removed >>>>>>>>>> 2025-04-24T21:30:54.164440+0000 mgr.ijc-mon1.yyfnhz >>>>>>>>>> (mgr.55376028) 14081 : cephadm [INF] Successfully >>>>>>>>>> destroyed old >>>>>>>>>> osd.253 on dig-osd4; ready for replacement >>>>>>>>>> 2025-04-24T21:30:54.164536+0000 mgr.ijc-mon1.yyfnhz >>>>>>>>>> (mgr.55376028) 14082 : cephadm [INF] Zapping devices for >>>>>>>>>> osd.253 >>>>>>>>>> on dig-osd4 >>>>>>>>>> ----- >>>>>>>>>> >>>>>>>>>> The message seems to be only for one of the 2 destroyed OSDs >>>>>>>>>> since I restarted the mgr. May this be the cause for the >>>>>>>>>> stucked >>>>>>>>>> upgrade? What can I do for fixing this? >>>>>>>>>> >>>>>>>>>> Thanks in advance for any hint. Best regards, >>>>>>>>>> >>>>>>>>>> Michel >>>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>>>>> _______________________________________________ >>>>>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>>>> _______________________________________________ >>>>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>>> _______________________________________________ >>>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users@ceph.io >>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users@ceph.io >>> To unsubscribe send an email to ceph-users-leave@ceph.io >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io

I’ve had a similar experience with Reef, trying to destroy an improperly deployed OSD on a viable drive. I had to ceph-volume lvm zap to get past the purge and at no point was the OSD marked as destroyed.

On Apr 30, 2025, at 6:15 AM, Eugen Block <eblock@nde.ag> wrote:

Right, 'ceph osd destroy' most likely won't help you here. The --replace flag is only to mark an OSD as destroyed (so it will reuse its ID after replacing the drive). You wrote that stopping osd rm for 253 unblocked the upgrade, so the cluster is currently upgrading?

To clear the pending state, I would stop rm for the other OSD as well since it's already out and down anyway. You can always zap a drive, either directly on the host with:

cephadm ceph-volume lvm zap --destroy /dev/sdX

Or using the orchestrator:

orch device zap <hostname> <path> [--force]

But just to clarify, OSD.381 is already the replacement disk for a previously failed drive? If you zap it, the orchestrator would try to apply any matching spec and create a new OSD, probably with ID 381 again.

Zitat von Michel Jouvin <michel.jouvin@ijclab.in2p3.fr>:

...

Frédéric,

My situation is a bit different I think. I had two malfunctionning OSDs that I removed with `ceph orch osd rm --replace --zap`: 1 was really dead and no longer seen by the OS (osd.253) and the other one with a lot of HW errors but still there (osd.381). Both have been successfully marked as destroyed in the CRUSH map. Just I didn't realize that cephadm was retrying every 10s to zap osd.253, getting an error as the disk could not be found. Looking at the removal status this morning(the removal was done ~2 weeks ago) with 'ceph orch osd rm status', I got:

OSD HOST STATE PGS REPLACE FORCE ZAP DRAIN STARTED AT 253 dig-osd4 done, waiting for purge 0 True False True 381 dig-osd6 done, waiting for purge 0 True False False 2025-04-23 11:56:09.864724+00:00

I don't know what the status "waiting to purge" means... but we can see that cephadm considers that the drain never started for osd.253, as the device was unavailable I guess... What happens with dig-osd6 is less clear to me but may be the consequence that at some point the disk freed by the initial rm was picked up by cephadm and readded as the replacement OSD because we forgot to set the osd.all-available-devices service to unmanaged. The drain started on Apr 23 is the second rm I did after fixing osd.all-available-devices service. For this second attempt, I didn't specify --zap, not sure why ( a mistake!).

I have the feeling but I may be wrong, that 'ceph osd destroy' will not help as they are already marked destroyed in the CRUSH map...

I'm wondering wether I should do 'ceph orch osd rm stop 381' as I did for 253 or whether it will impact the replacement later. Or said in a different way, is the replace flag something managed by cephadm and requiring the OSD to stay in the "rm queue" until the replacement is done?

Best regards,

Michel

...

Le 30/04/2025 à 10:50, Frédéric Nass a écrit : Hi Michel,

I've seen this recently on Reef (OSD stuck in the rm queue with the orchestrator tryng to zap a device that had already been zapped).

I could reproduce this a few times by deleting a batch of OSDs running on the same node. The whole 'ceph orch osd rm' process would stop progressing when trying to remove the ~8th OSD. I suspect that ceph-volume or the orchestrator is misinformed at some point that the device has already been zapped, looping over and over trying to remove this device that doesn't exist anymore.

I think you should now run 'ceph osd destroy <OSD_ID> --yes-i-really-mean-it'.

Regards, Frédéric.

----- Le 30 Avr 25, à 10:28, Michel Jouvin michel.jouvin@ijclab.in2p3.fr a écrit :

...

Eugen,

Thanks, I forgot that operation started with the orchestrator can be stopped. You were right: stopping the 'osd rm' was enough to unblock the upgrade. I am not completely sure what is the consequence on the replace flag: I have the feeling it has been lost somehow as the OSD is no longer listed by 'ceph orch osd rm status' and 'ceph -s' reports now one OSD down and 1 stray daemon instead of 2 stray daemons.

Michel

Le 30/04/2025 à 09:24, Eugen Block a écrit :

...

You can stop the osd removal:

ceph orch osd rm stop <OSD_ID>

I'm not entirely sure what the orchestrator will do except for clearing the pending state, and since the OSDs are already marked as destroyed in the crush tree, I wouldn't expect anything weird. But it's worth a try, I guess.

Zitat von Michel Jouvin <michel.jouvin@ijclab.in2p3.fr>:

...

Hi,

I had no time to really investigate more our problem yesterday. But I realized one issue that may explain the problem with osd.253: the underlying disk is so dead that it is no longer visible by the OS. Probably I added --zap when I did the 'ceph orch osd rm' and thus it is trying to do the zapping, fails as it doesn't find the disk and retries indefinitely... I remain a little bit surprise that this zapping error is not reported (without the traceback) at the INFO level and requires DEBUG to be seen but it is a detail. I'm surprised that Ceph is not giving up on zapping if it cannot access the device or did I miss something and there is a way to stop this process?

May be it is a corner case that has been fixed/improved since 18.2.2... Anyway, the question remains: is there a way out of this problem (that seems the only reason for the upgrade not really starting) apart from getting the replacement device?

Best regards,

Michel

Le 28/04/2025 à 18:19, Michel Jouvin a écrit : > Hi Frédéric, > > Thanks for the command. I'm always looking at the wrong page of the > doc! I looked at > https://docs.ceph.com/en/latest/rados/troubleshooting/log-and-debug/ > that list the Ceph subsystem and their default log level but there > is no mention of cephadm there... After enabling cephadm debug log > level and restarting the upgrade, I got the messages below. The only > thing strange points to the problem with osd.253 where it tries to > zap the device that was probably already zapped and thus cannot find > the LV volume associated with osd.253. There is not really any other > messages saying the impact on the upgrade but I guess it is the > reason. What do you think ? And is there any way to fix it, other > than replacing the OSD? > > Best regards, > > Michel > > --------------------- cephadm debug level log ------------------------- > > 2025-04-28T17:32:12.713746+0200 mgr.dig-mon1.fownxo [INF] Upgrade: > Started with target quay.io/ceph/ceph:v18.2.6 > 2025-04-28T17:32:14.822030+0200 mgr.dig-mon1.fownxo [DBG] Refreshed > host dig-osd4 devices (23) > 2025-04-28T17:32:14.822550+0200 mgr.dig-mon1.fownxo [DBG] Finding > OSDSpecs for host: <dig-osd4> > 2025-04-28T17:32:14.822614+0200 mgr.dig-mon1.fownxo [DBG] Generating > OSDSpec previews for [] > 2025-04-28T17:32:14.822695+0200 mgr.dig-mon1.fownxo [DBG] Loading > OSDSpec previews to HostCache for host <dig-osd4> > 2025-04-28T17:32:14.985257+0200 mgr.dig-mon1.fownxo [DBG] > mon_command: 'config generate-minimal-conf' -> 0 in 0.005s > 2025-04-28T17:32:15.262102+0200 mgr.dig-mon1.fownxo [DBG] > mon_command: 'auth get' -> 0 in 0.277s > 2025-04-28T17:32:15.262751+0200 mgr.dig-mon1.fownxo [DBG] Combine > hosts with existing daemons [] + new hosts.... (very long line) > > 2025-04-28T17:32:15.416491+0200 mgr.dig-mon1.fownxo [DBG] > _update_paused_health > 2025-04-28T17:32:17.314607+0200 mgr.dig-mon1.fownxo [DBG] > mon_command: 'osd df' -> 0 in 0.064s > 2025-04-28T17:32:17.637526+0200 mgr.dig-mon1.fownxo [DBG] > mon_command: 'osd df' -> 0 in 0.320s > 2025-04-28T17:32:17.645703+0200 mgr.dig-mon1.fownxo [DBG] 2 OSDs are > scheduled for removal: [osd.381, osd.253] > 2025-04-28T17:32:17.661910+0200 mgr.dig-mon1.fownxo [DBG] > mon_command: 'osd df' -> 0 in 0.011s > 2025-04-28T17:32:17.667068+0200 mgr.dig-mon1.fownxo [DBG] > mon_command: 'osd safe-to-destroy' -> 0 in 0.002s > 2025-04-28T17:32:17.667117+0200 mgr.dig-mon1.fownxo [DBG] cmd: osd > safe-to-destroy returns: > 2025-04-28T17:32:17.667164+0200 mgr.dig-mon1.fownxo [DBG] running > cmd: osd down on ids [osd.381] > 2025-04-28T17:32:17.667854+0200 mgr.dig-mon1.fownxo [DBG] > mon_command: 'osd down' -> 0 in 0.001s > 2025-04-28T17:32:17.667908+0200 mgr.dig-mon1.fownxo [INF] osd.381 > now down > 2025-04-28T17:32:17.668446+0200 mgr.dig-mon1.fownxo [INF] Daemon > osd.381 on dig-osd6 was already removed > 2025-04-28T17:32:17.669534+0200 mgr.dig-mon1.fownxo [DBG] > mon_command: 'osd destroy-actual' -> 0 in 0.001s > 2025-04-28T17:32:17.669675+0200 mgr.dig-mon1.fownxo [DBG] cmd: osd > destroy-actual returns: > 2025-04-28T17:32:17.669789+0200 mgr.dig-mon1.fownxo [INF] > Successfully destroyed old osd.381 on dig-osd6; ready for replacement > 2025-04-28T17:32:17.669874+0200 mgr.dig-mon1.fownxo [DBG] Removing > osd.381 from the queue. > 2025-04-28T17:32:17.680411+0200 mgr.dig-mon1.fownxo [DBG] > mon_command: 'osd df' -> 0 in 0.010s > 2025-04-28T17:32:17.685141+0200 mgr.dig-mon1.fownxo [DBG] > mon_command: 'osd safe-to-destroy' -> 0 in 0.002s > 2025-04-28T17:32:17.685190+0200 mgr.dig-mon1.fownxo [DBG] cmd: osd > safe-to-destroy returns: > 2025-04-28T17:32:17.685234+0200 mgr.dig-mon1.fownxo [DBG] running > cmd: osd down on ids [osd.253] > 2025-04-28T17:32:17.685710+0200 mgr.dig-mon1.fownxo [DBG] > mon_command: 'osd down' -> 0 in 0.000s > 2025-04-28T17:32:17.685759+0200 mgr.dig-mon1.fownxo [INF] osd.253 > now down > 2025-04-28T17:32:17.686186+0200 mgr.dig-mon1.fownxo [INF] Daemon > osd.253 on dig-osd4 was already removed > 2025-04-28T17:32:17.687068+0200 mgr.dig-mon1.fownxo [DBG] > mon_command: 'osd destroy-actual' -> 0 in 0.001s > 2025-04-28T17:32:17.687102+0200 mgr.dig-mon1.fownxo [DBG] cmd: osd > destroy-actual returns: > 2025-04-28T17:32:17.687141+0200 mgr.dig-mon1.fownxo [INF] > Successfully destroyed old osd.253 on dig-osd4; ready for replacement > 2025-04-28T17:32:17.687176+0200 mgr.dig-mon1.fownxo [INF] Zapping > devices for osd.253 on dig-osd4 > 2025-04-28T17:32:17.687508+0200 mgr.dig-mon1.fownxo [DBG] > _run_cephadm : command = ceph-volume > 2025-04-28T17:32:17.687554+0200 mgr.dig-mon1.fownxo [DBG] > _run_cephadm : args = ['--', 'lvm', 'zap', '--osd-id', '253', > '--destroy'] > 2025-04-28T17:32:17.687637+0200 mgr.dig-mon1.fownxo [DBG] osd > container image > quay.io/ceph/ceph@sha256:798f1b1e71ca1bbf76c687d8bcf5cd3e88640f044513ae55a0fb571502ae641f > 2025-04-28T17:32:17.687677+0200 mgr.dig-mon1.fownxo [DBG] args: > --image > quay.io/ceph/ceph@sha256:798f1b1e71ca1bbf76c687d8bcf5cd3e88640f044513ae55a0fb571502ae641f > --timeout 895 ceph-volume --fsid > f5195e24-158c-11ee-b338-5ced8c61b074 -- lvm zap --osd-id 253 --destroy > 2025-04-28T17:32:17.687733+0200 mgr.dig-mon1.fownxo [DBG] Running > command: which python3 > 2025-04-28T17:32:17.731474+0200 mgr.dig-mon1.fownxo [DBG] Running > command: /usr/bin/python3 > /var/lib/ceph/f5195e24-158c-11ee-b338-5ced8c61b074/cephadm.2b9d7d139a9cb40289f2358faf49a109fc297c0a258bde893227c262c30bca8d > --image > quay.io/ceph/ceph@sha256:798f1b1e71ca1bbf76c687d8bcf5cd3e88640f044513ae55a0fb571502ae641f > --timeout 895 ceph-volume --fsid > f5195e24-158c-11ee-b338-5ced8c61b074 -- lvm zap --osd-id 253 --destroy > 2025-04-28T17:32:20.406723+0200 mgr.dig-mon1.fownxo [DBG] code: 1 > 2025-04-28T17:32:20.406764+0200 mgr.dig-mon1.fownxo [DBG] err: > Inferring config > /var/lib/ceph/f5195e24-158c-11ee-b338-5ced8c61b074/config/ceph.conf > Non-zero exit code 1 from /usr/bin/podman run --rm --ipc=host > --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume > --privileged --group-add=disk --init -e > CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:798f1b1e71ca1bbf76c687d8bcf5cd3e88640f044513ae55a0fb571502ae641f > -e NODE_NAME=dig-osd4 -e CEPH_USE_RANDOM_NONCE=1 -e > CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v > /var/run/ceph/f5195e24-158c-11ee-b338-5ced8c61b074:/var/run/ceph:z > -v > /var/log/ceph/f5195e24-158c-11ee-b338-5ced8c61b074:/var/log/ceph:z > -v > /var/lib/ceph/f5195e24-158c-11ee-b338-5ced8c61b074/crash:/var/lib/ceph/crash:z > -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v > /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v > /run/lock/lvm:/run/lock/lvm -v > /var/lib/ceph/f5195e24-158c-11ee-b338-5ced8c61b074/selinux:/sys/fs/selinux:ro > -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v > /tmp/ceph-tmpgtvcw4gk:/etc/ceph/ceph.conf:z > quay.io/ceph/ceph@sha256:798f1b1e71ca1bbf76c687d8bcf5cd3e88640f044513ae55a0fb571502ae641f > lvm zap --osd-id 253 --destroy > /usr/bin/podman: stderr Traceback (most recent call last): > /usr/bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in > <module> > /usr/bin/podman: stderr load_entry_point('ceph-volume==1.0.0', > 'console_scripts', 'ceph-volume')() > /usr/bin/podman: stderr File > "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in > __init__ > /usr/bin/podman: stderr self.main(self.argv) > /usr/bin/podman: stderr File > "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line > 59, in newfunc > /usr/bin/podman: stderr return f(*a, **kw) > /usr/bin/podman: stderr File > "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in > main > /usr/bin/podman: stderr terminal.dispatch(self.mapper, > subcommand_args) > /usr/bin/podman: stderr File > "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line > 194, in dispatch > /usr/bin/podman: stderr instance.main() > /usr/bin/podman: stderr File > "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", > line 46, in main > /usr/bin/podman: stderr terminal.dispatch(self.mapper, self.argv) > /usr/bin/podman: stderr File > "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line > 194, in dispatch > /usr/bin/podman: stderr instance.main() > /usr/bin/podman: stderr File > "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", > line 403, in main > /usr/bin/podman: stderr self.zap_osd() > /usr/bin/podman: stderr File > "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line > 16, in is_root > /usr/bin/podman: stderr return func(*a, **kw) > /usr/bin/podman: stderr File > "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", > line 301, in zap_osd > /usr/bin/podman: stderr devices = > find_associated_devices(self.args.osd_id, self.args.osd_fsid) > /usr/bin/podman: stderr File > "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/zap.py", > line 88, in find_associated_devices > /usr/bin/podman: stderr '%s' % osd_id or osd_fsid) > /usr/bin/podman: stderr RuntimeError: Unable to find any LV for > zapping OSD: 253 > Traceback (most recent call last): > File "/usr/lib64/python3.9/runpy.py", line 197, in > _run_module_as_main > return _run_code(code, main_globals, None, > File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code > exec(code, run_globals) > File > "/var/lib/ceph/f5195e24-158c-11ee-b338-5ced8c61b074/cephadm.2b9d7d139a9cb40289f2358faf49a109fc297c0a258bde893227c262c30bca8d/__main__.py", > line 10700, in <module> > File > "/var/lib/ceph/f5195e24-158c-11ee-b338-5ced8c61b074/cephadm.2b9d7d139a9cb40289f2358faf49a109fc297c0a258bde893227c262c30bca8d/__main__.py", > line 10688, in main > File > "/var/lib/ceph/f5195e24-158c-11ee-b338-5ced8c61b074/cephadm.2b9d7d139a9cb40289f2358faf49a109fc297c0a258bde893227c262c30bca8d/__main__.py", > line 2445, in _infer_config > File > "/var/lib/ceph/f5195e24-158c-11ee-b338-5ced8c61b074/cephadm.2b9d7d139a9cb40289f2358faf49a109fc297c0a258bde893227c262c30bca8d/__main__.py", > line 2361, in _infer_fsid > File > "/var/lib/ceph/f5195e24-158c-11ee-b338-5ced8c61b074/cephadm.2b9d7d139a9cb40289f2358faf49a109fc297c0a258bde893227c262c30bca8d/__main__.py", > line 2473, in _infer_image > File > "/var/lib/ceph/f5195e24-158c-11ee-b338-5ced8c61b074/cephadm.2b9d7d139a9cb40289f2358faf49a109fc297c0a258bde893227c262c30bca8d/__main__.py", > line 2348, in _validate_fsid > File > "/var/lib/ceph/f5195e24-158c-11ee-b338-5ced8c61b074/cephadm.2b9d7d139a9cb40289f2358faf49a109fc297c0a258bde893227c262c30bca8d/__main__.py", > line 6970, in command_ceph_volume > File > "/var/lib/ceph/f5195e24-158c-11ee-b338-5ced8c61b074/cephadm.2b9d7d139a9cb40289f2358faf49a109fc297c0a258bde893227c262c30bca8d/__main__.py", > line 2136, in call_throws > RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host > --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume > --privileged --group-add=disk --init -e > CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:798f1b1e71ca1bbf76c687d8bcf5cd3e88640f044513ae55a0fb571502ae641f > -e NODE_NAME=dig-osd4 -e CEPH_USE_RANDOM_NONCE=1 -e > CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v > /var/run/ceph/f5195e24-158c-11ee-b338-5ced8c61b074:/var/run/ceph:z > -v > /var/log/ceph/f5195e24-158c-11ee-b338-5ced8c61b074:/var/log/ceph:z > -v > /var/lib/ceph/f5195e24-158c-11ee-b338-5ced8c61b074/crash:/var/lib/ceph/crash:z > -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v > /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v > /run/lock/lvm:/run/lock/lvm -v > /var/lib/ceph/f5195e24-158c-11ee-b338-5ced8c61b074/selinux:/sys/fs/selinux:ro > -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v > /tmp/ceph-tmpgtvcw4gk:/etc/ceph/ceph.conf:z > quay.io/ceph/ceph@sha256:798f1b1e71ca1bbf76c687d8bcf5cd3e88640f044513ae55a0fb571502ae641f > lvm zap --osd-id 253 --destroy > 2025-04-28T17:32:20.409316+0200 mgr.dig-mon1.fownxo [DBG] serve loop > sleep > > ----------------------- > > > Le 28/04/2025 à 14:00, Frédéric Nass a écrit : >> Hi Michel, >> >> You need to turn on cephadm debugging as described here [1] in the >> documentation >> >> $ ceph config set mgr mgr/cephadm/log_to_cluster_level debug >> >> and then look for any hints with >> >> $ ceph -W cephadm --watch-debug >> >> or >> >> $ tail -f /var/log/ceph/$(ceph fsid)/ceph.cephadm.log (on the >> active MGR) >> >> when you start/stop the upgrade. >> >> Regards, >> Frédéric. >> >> [1] https://docs.ceph.com/en/reef/cephadm/operations/ >> >> ----- Le 28 Avr 25, à 12:52, Michel Jouvin >> michel.jouvin@ijclab.in2p3.fr a écrit : >> >>> Eugen, >>> >>> Thanks for doing the test. I scanned all logs and cannot find >>> anything >>> except the message mentioned displayed every 10s about the removed >>> OSDs >>> that led me to think there is something not exactly as expected... >>> No clue >>> what... >>> >>> Michel >>> Sent from my mobile >>> Le 28 avril 2025 12:43:23 Eugen Block <eblock@nde.ag> a écrit : >>> >>>> I just tried this on a single-node virtual test cluster, deployed it >>>> with 18.2.2. Then I removed one OSD with --replace flag (no --zap, >>>> otherwise it would redeploy the OSD on that VM). Then I also see the >>>> stray daemon warning, but the upgrade from 18.2.2 to 18.2.6 finished >>>> successfully. That's why I don't think the stray daemon is the root >>>> cause here. I would suggest to scan monitor and cephadm logs as >>>> well. >>>> After the upgrade to 18.2.6 the stray warning cleared, btw. >>>> >>>> >>>> Zitat von Michel Jouvin <michel.jouvin@ijclab.in2p3.fr>: >>>> >>>>> Eugen, >>>>> >>>>> As said in a previous message, I found a tracker issue with a >>>>> similar problem: https://tracker.ceph.com/issues/67018, even if the >>>>> cause may be different as it is in older versions than me. For some >>>>> reasons the sequence of messages every 10s is now back on the 2 >>>>> OSDs: >>>>> >>>>> 2025-04-28T10:00:28.226741+0200 mgr.dig-mon1.fownxo [INF] >>>>> osd.253 now down >>>>> 2025-04-28T10:00:28.227249+0200 mgr.dig-mon1.fownxo [INF] Daemon >>>>> osd.253 on dig-osd4 was already removed >>>>> 2025-04-28T10:00:28.228929+0200 mgr.dig-mon1.fownxo [INF] >>>>> Successfully destroyed old osd.253 on dig-osd4; ready for >>>>> replacement >>>>> 2025-04-28T10:00:28.228994+0200 mgr.dig-mon1.fownxo [INF] Zapping >>>>> devices for osd.253 on dig-osd4 >>>>> 2025-04-28T10:00:39.132028+0200 mgr.dig-mon1.fownxo [INF] >>>>> osd.381 now down >>>>> 2025-04-28T10:00:39.132599+0200 mgr.dig-mon1.fownxo [INF] Daemon >>>>> osd.381 on dig-osd6 was already removed >>>>> 2025-04-28T10:00:39.133424+0200 mgr.dig-mon1.fownxo [INF] >>>>> Successfully destroyed old osd.381 on dig-osd6; ready for >>>>> replacement >>>>> >>>>> except that the "Zapping.." message is not present for the >>>>> second OSD... >>>>> >>>>> I tried to increase the mgr log verbosity with 'ceph tell >>>>> mgr.dig-mon1.fownxo config set debug_mgr 20/20' and there >>>>> stop/start >>>>> the upgrade without any additonal message displayed. >>>>> >>>>> Michel >>>>> >>>>> Le 28/04/2025 à 09:20, Eugen Block a écrit : >>>>>> Have you increased the debug level for the mgr? It would surprise >>>>>> me if stray daemons would really block an upgrade. But debug logs >>>>>> might reveal something. And if it can be confirmed that the strays >>>>>> really block the upgrade, you could either remove the OSDs >>>>>> entirely >>>>>> (they are already drained) to continue upgrading, or create a >>>>>> tracker issue to report this and wait for instructions. >>>>>> >>>>>> Zitat von Michel Jouvin <michel.jouvin@ijclab.in2p3.fr>: >>>>>> >>>>>>> Hi Eugen, >>>>>>> >>>>>>> Yes I stopped and restarted the upgrade several times already, in >>>>>>> particular after failing over the mgr. And the only messages >>>>>>> related are the upgrade started and upgrade canceled ones. >>>>>>> Nothing >>>>>>> related to an error or a crash... >>>>>>> >>>>>>> For me the question is why do I have stray daemons after removing >>>>>>> OSD. IMO it is unexpected as these daemons are not there anymore. >>>>>>> I can understand that stray daemons prevent the upgrade to start >>>>>>> if they are really strayed... And it would be nice if cephadm was >>>>>>> giving a message about why the upgrade does not really start >>>>>>> despite its status is "in progress"... >>>>>>> >>>>>>> Best regards, >>>>>>> >>>>>>> Michel >>>>>>> Sent from my mobile >>>>>>> Le 28 avril 2025 07:27:44 Eugen Block <eblock@nde.ag> a écrit : >>>>>>> >>>>>>>> Do you see anything in the mgr log? To get fresh logs I would >>>>>>>> cancel >>>>>>>> the upgrade (ceph orch upgrade stop) and then try again. >>>>>>>> A workaround could be to manually upgrade the mgr daemons by >>>>>>>> changing >>>>>>>> their unit.run file, but that would be my last resort. Btwm >>>>>>>> did you >>>>>>>> stop and start the upgrade after failing the mgr as well? >>>>>>>> >>>>>>>> Zitat von Michel Jouvin <michel.jouvin@ijclab.in2p3.fr>: >>>>>>>> >>>>>>>>> Eugen, >>>>>>>>> >>>>>>>>> Thanks for the hint. Here is the osd_remove_queue: >>>>>>>>> >>>>>>>>> [root@ijc-mon1 ~]# ceph config-key get >>>>>>>>> mgr/cephadm/osd_remove_queue|jq >>>>>>>>> [ >>>>>>>>> { >>>>>>>>> "osd_id": 253, >>>>>>>>> "started": true, >>>>>>>>> "draining": false, >>>>>>>>> "stopped": false, >>>>>>>>> "replace": true, >>>>>>>>> "force": false, >>>>>>>>> "zap": true, >>>>>>>>> "hostname": "dig-osd4", >>>>>>>>> "drain_started_at": null, >>>>>>>>> "drain_stopped_at": null, >>>>>>>>> "drain_done_at": "2025-04-15T14:09:30.521534Z", >>>>>>>>> "process_started_at": "2025-04-15T14:09:14.091592Z" >>>>>>>>> }, >>>>>>>>> { >>>>>>>>> "osd_id": 381, >>>>>>>>> "started": true, >>>>>>>>> "draining": false, >>>>>>>>> "stopped": false, >>>>>>>>> "replace": true, >>>>>>>>> "force": false, >>>>>>>>> "zap": false, >>>>>>>>> "hostname": "dig-osd6", >>>>>>>>> "drain_started_at": "2025-04-23T11:56:09.864724Z", >>>>>>>>> "drain_stopped_at": null, >>>>>>>>> "drain_done_at": "2025-04-25T06:53:03.678729Z", >>>>>>>>> "process_started_at": "2025-04-23T11:56:05.924923Z" >>>>>>>>> } >>>>>>>>> ] >>>>>>>>> >>>>>>>>> It is not empty the two stray daemons are listed. Not sure >>>>>>>>> it these >>>>>>>>> entries are expected as I specified --replace... A similar >>>>>>>>> issue was >>>>>>>>> reported in https://tracker.ceph.com/issues/67018 so before >>>>>>>>> Reef but >>>>>>>>> the cause may be different. Still not clear for me how to >>>>>>>>> get out of >>>>>>>>> this, except may be replacing the OSDs but this will take >>>>>>>>> some time... >>>>>>>>> >>>>>>>>> Best regards, >>>>>>>>> >>>>>>>>> Michel >>>>>>>>> >>>>>>>>> Le 27/04/2025 à 10:21, Eugen Block a écrit : >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> what's the current ceph status? Wasn't there a bug in early >>>>>>>>>> Reef >>>>>>>>>> versions preventing upgrades if there were removed OSDs in the >>>>>>>>>> queue? But IIRC, the cephadm module would crash. Can you check >>>>>>>>>> >>>>>>>>>> ceph config-key get mgr/cephadm/osd_remove_queue >>>>>>>>>> >>>>>>>>>> And then I would check the mgr log, maybe set it to a >>>>>>>>>> higher debug >>>>>>>>>> level to see what's blocking it. >>>>>>>>>> >>>>>>>>>> Zitat von Michel Jouvin <michel.jouvin@ijclab.in2p3.fr>: >>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> I tried to restart all the mgrs (we have 3, 1 active, 2 >>>>>>>>>>> standby) >>>>>>>>>>> by executing 3 times the `ceph mgr fail`, no impact. I don't >>>>>>>>>>> really understand why I get these stray daemons after doing a >>>>>>>>>>> 'ceph orch osd rm --replace` but I think I have always >>>>>>>>>>> seen this. >>>>>>>>>>> I tried to mute rather than disable the stray daemon check >>>>>>>>>>> but it >>>>>>>>>>> doesn't help either. And I find strange this message every >>>>>>>>>>> 10s >>>>>>>>>>> about one of the destroyed OSD and only one, reporting it >>>>>>>>>>> is down >>>>>>>>>>> and already destroyed and saying it'll zap it (I think I >>>>>>>>>>> didn't >>>>>>>>>>> add --zap when I removed it as the underlying disk is dead). >>>>>>>>>>> >>>>>>>>>>> I'm completely stuck with this upgrade and I don't >>>>>>>>>>> remember having >>>>>>>>>>> this kind of problems in previous upgrades with cephadm... >>>>>>>>>>> Any >>>>>>>>>>> idea where to look for the cause and/or how to fix it? >>>>>>>>>>> >>>>>>>>>>> Best regards, >>>>>>>>>>> >>>>>>>>>>> Michel >>>>>>>>>>> >>>>>>>>>>> Le 24/04/2025 à 23:34, Michel Jouvin a écrit : >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> I'm trying to upgrade a (cephadm) cluster from 18.2.2 to >>>>>>>>>>>> 18.2.6, >>>>>>>>>>>> using 'ceph orch upgrade'. When I enter the command 'ceph >>>>>>>>>>>> orch >>>>>>>>>>>> upgrade start --ceph-version 18.2.6', I receive a message >>>>>>>>>>>> saying >>>>>>>>>>>> that the upgrade has been initiated, with a similar >>>>>>>>>>>> message in >>>>>>>>>>>> the logs but nothing happens after this. 'ceph orch upgrade >>>>>>>>>>>> status' says: >>>>>>>>>>>> >>>>>>>>>>>> ------- >>>>>>>>>>>> >>>>>>>>>>>> [root@ijc-mon1 ~]# ceph orch upgrade status >>>>>>>>>>>> { >>>>>>>>>>>> "target_image": "quay.io/ceph/ceph:v18.2.6", >>>>>>>>>>>> "in_progress": true, >>>>>>>>>>>> "which": "Upgrading all daemon types on all hosts", >>>>>>>>>>>> "services_complete": [], >>>>>>>>>>>> "progress": "", >>>>>>>>>>>> "message": "", >>>>>>>>>>>> "is_paused": false >>>>>>>>>>>> } >>>>>>>>>>>> ------- >>>>>>>>>>>> >>>>>>>>>>>> The first time I entered the command, the cluster status was >>>>>>>>>>>> HEALTH_WARN because of 2 stray daemons (caused by >>>>>>>>>>>> destroyed OSDs, >>>>>>>>>>>> rm --replace). I set mgr/cephadm/warn_on_stray_daemons to >>>>>>>>>>>> false >>>>>>>>>>>> to ignore these 2 daemons, the cluster is now HEALTH_OK >>>>>>>>>>>> but it >>>>>>>>>>>> doesn't help. Following a Red Hat KB entry, I tried to >>>>>>>>>>>> failover >>>>>>>>>>>> the mgr, stopped an restarted the upgrade but without any >>>>>>>>>>>> improvement. I have not seen anything in the logs, except >>>>>>>>>>>> that >>>>>>>>>>>> there is an INF entry every 10s about the destroyed OSD >>>>>>>>>>>> saying: >>>>>>>>>>>> >>>>>>>>>>>> ------ >>>>>>>>>>>> >>>>>>>>>>>> 2025-04-24T21:30:54.161988+0000 mgr.ijc-mon1.yyfnhz >>>>>>>>>>>> (mgr.55376028) 14079 : cephadm [INF] osd.253 now down >>>>>>>>>>>> 2025-04-24T21:30:54.162601+0000 mgr.ijc-mon1.yyfnhz >>>>>>>>>>>> (mgr.55376028) 14080 : cephadm [INF] Daemon osd.253 on >>>>>>>>>>>> dig-osd4 >>>>>>>>>>>> was already removed >>>>>>>>>>>> 2025-04-24T21:30:54.164440+0000 mgr.ijc-mon1.yyfnhz >>>>>>>>>>>> (mgr.55376028) 14081 : cephadm [INF] Successfully >>>>>>>>>>>> destroyed old >>>>>>>>>>>> osd.253 on dig-osd4; ready for replacement >>>>>>>>>>>> 2025-04-24T21:30:54.164536+0000 mgr.ijc-mon1.yyfnhz >>>>>>>>>>>> (mgr.55376028) 14082 : cephadm [INF] Zapping devices for >>>>>>>>>>>> osd.253 >>>>>>>>>>>> on dig-osd4 >>>>>>>>>>>> ----- >>>>>>>>>>>> >>>>>>>>>>>> The message seems to be only for one of the 2 destroyed OSDs >>>>>>>>>>>> since I restarted the mgr. May this be the cause for the >>>>>>>>>>>> stucked >>>>>>>>>>>> upgrade? What can I do for fixing this? >>>>>>>>>>>> >>>>>>>>>>>> Thanks in advance for any hint. Best regards, >>>>>>>>>>>> >>>>>>>>>>>> Michel >>>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>>>>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>>>>>>> _______________________________________________ >>>>>>>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>>>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>>>>>> _______________________________________________ >>>>>>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>>>>> _______________________________________________ >>>>>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>>> _______________________________________________ >>>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>> _______________________________________________ >>>>> ceph-users mailing list -- ceph-users@ceph.io >>>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users@ceph.io >>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users@ceph.io >>> To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Hm, I thought there was an excerpt from the osd tree, but apparently not? Could you then please confirm that the OSDs are in fact marked as destroyed in the osd tree? Zitat von Anthony D'Atri <anthony.datri@gmail.com>:

...

I'm not entirely sure what the orchestrator will do except for clearing the pending state, and since the OSDs are already marked as destroyed in the crush tree,

Do we know that they are? The thread shows some log messages, but not unless I’m missing it evidence that they were marked. When I ran into a similar issue recently, they were not marked destroyed in the CRUSH tree.

Sounds great that the upgrade went through!

May be it should be better documented that you should not zap a device intended for definitive removal if you don't have osd.all-available-devices service placement was set to unmanaged...

I'd rather vote for not considering osd.all-available-devices as something for production. ;-) I only use that in small test clusters to quickly set up OSDs without having to deal with specs. But in production clusters, I like to have full control over OSD creation. Just my opinion. :-) Zitat von Michel Jouvin <michel.jouvin@ijclab.in2p3.fr>:

Hi,

Thanks for all the feedback and suggestions. Summary of the summary: after stopping the removal for the OSD waiting to be zapped (because of the no longer available disk), the upgrade started immediately and ran well. The cluster is now running 18.2.6! And as said previously by Eugen, I confirm that in 18.2.6, removed OSDs are no longer considered stray daemons. I still have the feeling that Ceph could give more useful information if:

- a cephadm message at INFO level (and visible with 'ceph orch upgrade status' would report that the upgrade cannot proceed because of described reason. This information could be given once, a few minutes after entering the upgrade command is no daemon has been upgraded yet, for example.

- a message at INFO level was informing that the zap operation failed (suggesting to use DEBUG level for more information)

About Anthony's last question, yes the 2 OSDs were destroyed as showed by:

# ceph osd tree|grep destroyed 253    hdd    16.37108                  osd.253 destroyed         0  1.00000 381    hdd    16.37108                  osd.381 destroyed         0  1.00000

@Eugen regarding what I said about osd.381 being picked up by Ceph to replace the failed osd.381 OSD, I think it is the conjunction of the fact that osd.all-available-devices service placement was not set to unmanaged (something we tend to do normally but as we add a few servers recently we changed it and forgot to set it back to unmanaged) and that in the initial removal I zapped the device. Because of this, the device appeared to be free for use... May be it should be better documented that you should not zap a device intended for definitive removal if you don't have osd.all-available-devices service placement was set to unmanaged...

Thanks again. Best regards,

Michel

Le 30/04/2025 à 15:41, Eugen Block a écrit :

...

Hm, I thought there was an excerpt from the osd tree, but apparently not? Could you then please confirm that the OSDs are in fact marked as destroyed in the osd tree?

Zitat von Anthony D'Atri <anthony.datri@gmail.com>:

...

...

I'm not entirely sure what the orchestrator will do except for clearing the pending state, and since the OSDs are already marked as destroyed in the crush tree,

Do we know that they are?  The thread shows some log messages, but not unless I’m missing it evidence that they were marked. When I ran into a similar issue recently, they were not marked destroyed in the CRUSH tree.

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io