Hi Anthony, Seeing it since last after noon. It is same with mgr services as , "ceph -s" is reporting only TWO instead of THREE Also mon and mgr shows " is_active: false" see below. # ceph orch ps --daemon_type=mgr NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mgr.a001s016.ctmoay a001s016 *:8443 running (18M) 3m ago 23M 206M - 16.2.5 6e73176320aa 169cafcbbb99 mgr.a001s017.bpygfm a001s017 *:8443 running (19M) 3m ago 23M 332M - 16.2.5 6e73176320aa 97257195158c mgr.a001s018.hcxnef a001s018 *:8443 running (20M) 3m ago 23M 113M - 16.2.5 6e73176320aa 21ba5896cee2 # ceph orch ls --service_name=mgr NAME PORTS RUNNING REFRESHED AGE PLACEMENT mgr 3/3 3m ago 23M a001s016;a001s017;a001s018;count:3 # ceph orch ps --daemon_type=mon --format=json-pretty [ { "container_id": "8484a912f96a", "container_image_digests": [ docker.io/ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586<mailto:docker.io/ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586> ], "container_image_id": "6e73176320aaccf3b3fb660b9945d0514222bd7a83e28b96e8440c630ba6891f", "container_image_name": docker.io/ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586<mailto:docker.io/ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586>, "created": "2024-03-31T23:55:16.164155Z", "daemon_id": "a001s016", "daemon_type": "mon", "hostname": "a001s016", "is_active": false, <== why is it false "last_refresh": "2024-04-01T19:38:30.929014Z", "memory_request": 2147483648, "memory_usage": 761685606, "ports": [], "service_name": "mon", "started": "2024-03-31T23:55:16.268266Z", "status": 1, "status_desc": "running", "version": "16.2.5" }, Thank you, Anantha From: Anthony D'Atri <aad@dreamsnake.net> Sent: Monday, April 1, 2024 12:25 PM To: Adiga, Anantha <anantha.adiga@intel.com> Cc: ceph-users@ceph.io Subject: Re: [ceph-users] ceph status not showing correct monitor services a001s017.bpygfm(active, since 13M), standbys: a001s016.ctmoay Looks like you just had an mgr failover? Could be that the secondary mgr hasn't caught up with current events.

Hi Eugen, Yes that is it. OSDs were restarted since mon a001s017 was reporting is low on available space. How to update the mon map to add mon.a001s016 as it is already online? And how to update mgr map to include standby mgr.a001s018 as it is also running. ceph mon dump dumped monmap epoch 6 epoch 6 fsid 604d56db-2fab-45db-a9ea-c418f9a8cca8 last_changed 2024-03-31T23:54:18.692983+0000 created 2021-09-30T16:15:12.884602+0000 min_mon_release 16 (pacific) election_strategy: 1 0: [v2:10.45.128.28:3300/0,v1:10.45.128.28:6789/0] mon.a001s018 1: [v2:10.45.128.27:3300/0,v1:10.45.128.27:6789/0] mon.a001s017 Thank you, Anantha -----Original Message----- From: Eugen Block <eblock@nde.ag> Sent: Monday, April 1, 2024 1:10 PM To: ceph-users@ceph.io Subject: [ceph-users] Re: ceph status not showing correct monitor services Maybe it’s just not in the monmap? Can you show the output of: ceph mon dump Did you do any maintenance (apparently OSDs restarted recently) and maybe accidentally removed a MON from the monmap? Zitat von "Adiga, Anantha" <anantha.adiga@intel.com>:

Hi Anthony,

Seeing it since last after noon. It is same with mgr services as , "ceph -s" is reporting only TWO instead of THREE

Also mon and mgr shows " is_active: false" see below.

# ceph orch ps --daemon_type=mgr NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mgr.a001s016.ctmoay a001s016 *:8443 running (18M) 3m ago 23M 206M - 16.2.5 6e73176320aa 169cafcbbb99 mgr.a001s017.bpygfm a001s017 *:8443 running (19M) 3m ago 23M 332M - 16.2.5 6e73176320aa 97257195158c mgr.a001s018.hcxnef a001s018 *:8443 running (20M) 3m ago 23M 113M - 16.2.5 6e73176320aa 21ba5896cee2

# ceph orch ls --service_name=mgr NAME PORTS RUNNING REFRESHED AGE PLACEMENT mgr 3/3 3m ago 23M a001s016;a001s017;a001s018;count:3

# ceph orch ps --daemon_type=mon --format=json-pretty

[ { "container_id": "8484a912f96a", "container_image_digests": [

docker.io/ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586<mailto:docker.io/ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586> ], "container_image_id": "6e73176320aaccf3b3fb660b9945d0514222bd7a83e28b96e8440c630ba6891f", "container_image_name": docker.io/ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586<mailto:docker.io/ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586>, "created": "2024-03-31T23:55:16.164155Z", "daemon_id": "a001s016", "daemon_type": "mon", "hostname": "a001s016", "is_active": false, <== why is it false "last_refresh": "2024-04-01T19:38:30.929014Z", "memory_request": 2147483648, "memory_usage": 761685606, "ports": [], "service_name": "mon", "started": "2024-03-31T23:55:16.268266Z", "status": 1, "status_desc": "running", "version": "16.2.5" },

Thank you, Anantha

From: Anthony D'Atri <aad@dreamsnake.net> Sent: Monday, April 1, 2024 12:25 PM To: Adiga, Anantha <anantha.adiga@intel.com> Cc: ceph-users@ceph.io Subject: Re: [ceph-users] ceph status not showing correct monitor services

a001s017.bpygfm(active, since 13M), standbys: a001s016.ctmoay

Looks like you just had an mgr failover? Could be that the secondary mgr hasn't caught up with current events. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Thank you. I will try the export and import method first. Thank you, Anantha -----Original Message----- From: Eugen Block <eblock@nde.ag> Sent: Monday, April 1, 2024 1:57 PM To: Adiga, Anantha <anantha.adiga@intel.com> Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Re: ceph status not showing correct monitor services I have two approaches in mind, first one (and preferred) would be to edit the mon spec to first remove mon.a001s016 and have a clean state. Get the current spec with: ceph orch ls mon --export > mon-edit.yaml Edit the spec file so that mon.a001s016 is not part of it, then apply: ceph orch apply -i mon-edit.yaml This should remove the mon.a001s016 daemon. Then wait a few minutes or so (until the daemon is actually gone, check locally on the node with 'cephadm ls' and in /var/lib/ceph/<FSID>/removed) and add it back to the spec file, then apply again. I would expect a third MON to be deployed. If that doesn't work for some reason you'll need to inspect logs to find the root cause. The second approach would be to remove and add the daemon manually: ceph orch daemon rm mon.a001s016 Wait until it's really gone, then add it: ceph orch daemon add mon a001s016 Not entirely sure about the daemon add mon command, you might need to provide something else, I'm typing this by heart. Zitat von "Adiga, Anantha" <anantha.adiga@intel.com>:

Hi Eugen,

Yes that is it. OSDs were restarted since mon a001s017 was reporting is low on available space. How to update the mon map to add mon.a001s016 as it is already online? And how to update mgr map to include standby mgr.a001s018 as it is also running.

ceph mon dump dumped monmap epoch 6 epoch 6 fsid 604d56db-2fab-45db-a9ea-c418f9a8cca8 last_changed 2024-03-31T23:54:18.692983+0000 created 2021-09-30T16:15:12.884602+0000 min_mon_release 16 (pacific) election_strategy: 1 0: [v2:10.45.128.28:3300/0,v1:10.45.128.28:6789/0] mon.a001s018 1: [v2:10.45.128.27:3300/0,v1:10.45.128.27:6789/0] mon.a001s017

Thank you, Anantha

-----Original Message----- From: Eugen Block <eblock@nde.ag> Sent: Monday, April 1, 2024 1:10 PM To: ceph-users@ceph.io Subject: [ceph-users] Re: ceph status not showing correct monitor services

Maybe it’s just not in the monmap? Can you show the output of:

ceph mon dump

Did you do any maintenance (apparently OSDs restarted recently) and maybe accidentally removed a MON from the monmap?

Zitat von "Adiga, Anantha" <anantha.adiga@intel.com>:

...

Hi Anthony,

Seeing it since last after noon. It is same with mgr services as , "ceph -s" is reporting only TWO instead of THREE

Also mon and mgr shows " is_active: false" see below.

# ceph orch ps --daemon_type=mgr NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mgr.a001s016.ctmoay a001s016 *:8443 running (18M) 3m ago 23M 206M - 16.2.5 6e73176320aa 169cafcbbb99 mgr.a001s017.bpygfm a001s017 *:8443 running (19M) 3m ago 23M 332M - 16.2.5 6e73176320aa 97257195158c mgr.a001s018.hcxnef a001s018 *:8443 running (20M) 3m ago 23M 113M - 16.2.5 6e73176320aa 21ba5896cee2

# ceph orch ls --service_name=mgr NAME PORTS RUNNING REFRESHED AGE PLACEMENT mgr 3/3 3m ago 23M a001s016;a001s017;a001s018;count:3

# ceph orch ps --daemon_type=mon --format=json-pretty

[ { "container_id": "8484a912f96a", "container_image_digests": [

docker.io/ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586<mailto:docker.io/ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586> ], "container_image_id": "6e73176320aaccf3b3fb660b9945d0514222bd7a83e28b96e8440c630ba6891f", "container_image_name": docker.io/ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586<mailto:docker.io/ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586>, "created": "2024-03-31T23:55:16.164155Z", "daemon_id": "a001s016", "daemon_type": "mon", "hostname": "a001s016", "is_active": false, <== why is it false "last_refresh": "2024-04-01T19:38:30.929014Z", "memory_request": 2147483648, "memory_usage": 761685606, "ports": [], "service_name": "mon", "started": "2024-03-31T23:55:16.268266Z", "status": 1, "status_desc": "running", "version": "16.2.5" },

Thank you, Anantha

From: Anthony D'Atri <aad@dreamsnake.net> Sent: Monday, April 1, 2024 12:25 PM To: Adiga, Anantha <anantha.adiga@intel.com> Cc: ceph-users@ceph.io Subject: Re: [ceph-users] ceph status not showing correct monitor services

a001s017.bpygfm(active, since 13M), standbys: a001s016.ctmoay

Looks like you just had an mgr failover? Could be that the secondary mgr hasn't caught up with current events. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

root@a001s016:/var/run/ceph/604d56db-2fab-45db-a9ea-c418f9a8cca8# ceph tell mon.a001s016 mon_status Error ENOENT: problem getting command descriptions from mon.a001s016 a001s016 is outside quorum see below root@a001s016:/var/run/ceph/604d56db-2fab-45db-a9ea-c418f9a8cca8# ceph tell mon.a001s017 mon_status { "name": "a001s017", "rank": 1, "state": "peon", "election_epoch": 162, "quorum": [ 0, 1 ], "quorum_age": 79938, "features": { "required_con": "2449958747317026820", "required_mon": [ "kraken", "luminous", "mimic", "osdmap-prune", "nautilus", "octopus", "pacific", "elector-pinging" ], "quorum_con": "4540138297136906239", "quorum_mon": [ "kraken", "luminous", "mimic", "osdmap-prune", "nautilus", "octopus", "pacific", "elector-pinging" ] }, "outside_quorum": [], "extra_probe_peers": [ { "addrvec": [ { "type": "v2", "addr": "10.45.128.26:3300", "nonce": 0 }, { "type": "v1", "addr": "10.45.128.26:6789", "nonce": 0 } ] } ], "sync_provider": [], "monmap": { "epoch": 6, "fsid": "604d56db-2fab-45db-a9ea-c418f9a8cca8", "modified": "2024-03-31T23:54:18.692983Z", "created": "2021-09-30T16:15:12.884602Z", "min_mon_release": 16, "min_mon_release_name": "pacific", "election_strategy": 1, "disallowed_leaders: ": "", "stretch_mode": false, "features": { "persistent": [ "kraken", "luminous", "mimic", "osdmap-prune", "nautilus", "octopus", "pacific", "elector-pinging" ], "optional": [] }, "mons": [ { "rank": 0, "name": "a001s018", "public_addrs": { "addrvec": [ { "type": "v2", "addr": "10.45.128.28:3300", "nonce": 0 }, { "type": "v1", "addr": "10.45.128.28:6789", "nonce": 0 } ] }, "addr": "10.45.128.28:6789/0", "public_addr": "10.45.128.28:6789/0", "priority": 0, "weight": 0, "crush_location": "{}" }, { "rank": 1, "name": "a001s017", "public_addrs": { "addrvec": [ { "type": "v2", "addr": "10.45.128.27:3300", "nonce": 0 }, { "type": "v1", "addr": "10.45.128.27:6789", "nonce": 0 } ] }, "addr": "10.45.128.27:6789/0", "public_addr": "10.45.128.27:6789/0", "priority": 0, "weight": 0, "crush_location": "{}" } ] }, "feature_map": { "mon": [ { "features": "0x3f01cfb9fffdffff", "release": "luminous", "num": 1 } ], "mds": [ { "features": "0x3f01cfb9fffdffff", "release": "luminous", "num": 3 } ], "osd": [ { "features": "0x3f01cfb9fffdffff", "release": "luminous", "num": 15 } ], "client": [ { "features": "0x2f018fb86aa42ada", "release": "luminous", "num": 50 }, { "features": "0x2f018fb87aa4aafe", "release": "luminous", "num": 40 }, { "features": "0x3f01cfb8ffedffff", "release": "luminous", "num": 1 }, { "features": "0x3f01cfb9fffdffff", "release": "luminous", "num": 72 } ] }, "stretch_mode": false } root@a001s016:/var/run/ceph/604d56db-2fab-45db-a9ea-c418f9a8cca8# ceph mon dump dumped monmap epoch 6 epoch 6 fsid 604d56db-2fab-45db-a9ea-c418f9a8cca8 last_changed 2024-03-31T23:54:18.692983+0000 created 2021-09-30T16:15:12.884602+0000 min_mon_release 16 (pacific) election_strategy: 1 0: [v2:10.45.128.28:3300/0,v1:10.45.128.28:6789/0] mon.a001s018 1: [v2:10.45.128.27:3300/0,v1:10.45.128.27:6789/0] mon.a001s017 root@a001s016:/var/run/ceph/604d56db-2fab-45db-a9ea-c418f9a8cca8# -----Original Message----- From: Adiga, Anantha Sent: Monday, April 1, 2024 3:20 PM To: Adiga, Anantha <anantha.adiga@intel.com>; Eugen Block <eblock@nde.ag> Cc: ceph-users@ceph.io Subject: RE: [ceph-users] Re: ceph status not showing correct monitor services Both methods are not update the mon map, is there a way to inject mon.a001s016 into the current mon map? # ceph mon dump dumped monmap epoch 6 epoch 6 fsid 604d56db-2fab-45db-a9ea-c418f9a8cca8 last_changed 2024-03-31T23:54:18.692983+0000 created 2021-09-30T16:15:12.884602+0000 min_mon_release 16 (pacific) election_strategy: 1 0: [v2:10.45.128.28:3300/0,v1:10.45.128.28:6789/0] mon.a001s018 1: [v2:10.45.128.27:3300/0,v1:10.45.128.27:6789/0] mon.a001s017 # ceph tell mon.a001s016 mon_status Error ENOENT: problem getting command descriptions from mon.a001s016 # ceph tell mon.a001s016 mon_status Error ENOENT: problem getting command descriptions from mon.a001s016 # ceph tell mon.a001s017 mon_status { "name": "a001s017", "rank": 1, "state": "peon", "election_epoch": 162, "quorum": [ 0, 1 ], "quorum_age": 69551, "features": { .............. .............. # ceph orch ls --service_name=mon --export > mon3.yml service_type: mon service_name: mon placement: count: 3 hosts: - a001s016 - a001s017 - a001s018 # cp mon3.yml mon2.yml # vi mon2.yml #cat mon2.yml service_type: mon service_name: mon placement: count: 2 hosts: - a001s017 - a001s018 # ceph orch apply -i mon2.yml --dry-run WARNING! Dry-Runs are snapshots of a certain point in time and are bound to the current inventory setup. If any on these conditions changes, the preview will be invalid. Please make sure to have a minimal timeframe between planning and applying the specs. #################### SERVICESPEC PREVIEWS #################### +---------+------+--------+--------------+ |SERVICE |NAME |ADD_TO |REMOVE_FROM | +---------+------+--------+--------------+ |mon |mon | |mon.a001s016 | +---------+------+--------+--------------+ ################ OSDSPEC PREVIEWS ################ +---------+------+------+------+----+-----+ |SERVICE |NAME |HOST |DATA |DB |WAL | +---------+------+------+------+----+-----+ +---------+------+------+------+----+-----+ # ceph orch ls --service_name=mon --refresh NAME PORTS RUNNING REFRESHED AGE PLACEMENT mon 3/3 5m ago 18h a001s016;a001s017;a001s018;count:3 # ceph orch ps --refresh | grep mon mon.a001s016 a001s016 running (21h) 2s ago 21h 734M 2048M 16.2.5 6e73176320aa 8484a912f96a mon.a001s017 a001s017 running (18h) 2s ago 21h 976M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 2s ago 2y 1164M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s016 a001s016 running (21h) 37s ago 21h 734M 2048M 16.2.5 6e73176320aa 8484a912f96a mon.a001s017 a001s017 running (18h) 37s ago 21h 977M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 38s ago 2y 1166M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch apply -i mon2.yml Scheduled mon update... # ceph orch ps --refresh | grep mon mon.a001s016 a001s016 running (21h) 21s ago 21h 734M 2048M 16.2.5 6e73176320aa 8484a912f96a mon.a001s017 a001s017 running (18h) 20s ago 21h 962M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 21s ago 2y 1156M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s017 a001s017 running (18h) 23s ago 21h 962M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 24s ago 2y 1156M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s017 a001s017 running (18h) 27s ago 21h 962M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 0s ago 2y 1154M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s017 a001s017 running (18h) 2s ago 21h 960M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 3s ago 2y 1154M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s017 a001s017 running (18h) 5s ago 21h 960M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 0s ago 2y 1154M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s017 a001s017 running (18h) 2s ago 21h 962M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 2s ago 2y 1154M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ls /var/lib/ceph/604d56db-2fab-45db-a9ea-c418f9a8cca8/ cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931 keepalived.nfs.nfs.a001s016.ofqyjl osd.1 osd.22 osd.4 crash mds.cephfs.a001s016.vtyivn osd.10 osd.24 osd.7 crash.a001s016 mgr.a001s016.ctmoay osd.13 osd.27 removed haproxy.nfs.nfs.a001s016.vpjtxt nfs.nfs.0.0.a001s016.wumnjs osd.15 osd.30 rgw.ceph.a001s016.gjyanl home node-exporter.a001s016 osd.18 osd.33 # ceph mon dump dumped monmap epoch 6 epoch 6 fsid 604d56db-2fab-45db-a9ea-c418f9a8cca8 last_changed 2024-03-31T23:54:18.692983+0000 created 2021-09-30T16:15:12.884602+0000 min_mon_release 16 (pacific) election_strategy: 1 0: [v2:10.45.128.28:3300/0,v1:10.45.128.28:6789/0] mon.a001s018 1: [v2:10.45.128.27:3300/0,v1:10.45.128.27:6789/0] mon.a001s017 # cat mon3.yml service_type: mon service_name: mon placement: count: 3 hosts: - a001s016 - a001s017 - a001s018 # ceph orch apply -i mon3.yml --dry-run WARNING! Dry-Runs are snapshots of a certain point in time and are bound to the current inventory setup. If any on these conditions changes, the preview will be invalid. Please make sure to have a minimal timeframe between planning and applying the specs. #################### SERVICESPEC PREVIEWS #################### +---------+------+----------+-------------+ |SERVICE |NAME |ADD_TO |REMOVE_FROM | +---------+------+----------+-------------+ |mon |mon |a001s016 | | +---------+------+----------+-------------+ ################ OSDSPEC PREVIEWS ################ +---------+------+------+------+----+-----+ |SERVICE |NAME |HOST |DATA |DB |WAL | +---------+------+------+------+----+-----+ +---------+------+------+------+----+-----+ # ceph orch apply -i mon3.yml Scheduled mon update... # ceph orch ps --refresh | grep mon mon.a001s016 a001s016 starting - - - 2048M <unknown> <unknown> <unknown> mon.a001s017 a001s017 running (18h) 2s ago 21h 960M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 3s ago 2y 1152M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s016 a001s016 running (6s) 0s ago 6s 14.4M 2048M 16.2.5 6e73176320aa dcd4705c2069 mon.a001s017 a001s017 running (18h) 8s ago 21h 960M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 9s ago 2y 1152M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s016 a001s016 running (10s) 4s ago 10s 14.4M 2048M 16.2.5 6e73176320aa dcd4705c2069 mon.a001s017 a001s017 running (18h) 12s ago 21h 960M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 13s ago 2y 1152M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ls /var/lib/ceph/604d56db-2fab-45db-a9ea-c418f9a8cca8/ cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931 mgr.a001s016.ctmoay osd.15 osd.4 crash mon.a001s016 osd.18 osd.7 crash.a001s016 nfs.nfs.0.0.a001s016.wumnjs osd.22 removed haproxy.nfs.nfs.a001s016.vpjtxt node-exporter.a001s016 osd.24 rgw.ceph.a001s016.gjyanl home osd.1 osd.27 keepalived.nfs.nfs.a001s016.ofqyjl osd.10 osd.30 mds.cephfs.a001s016.vtyivn osd.13 osd.33 # ceph mon dump dumped monmap epoch 6 epoch 6 fsid 604d56db-2fab-45db-a9ea-c418f9a8cca8 last_changed 2024-03-31T23:54:18.692983+0000 created 2021-09-30T16:15:12.884602+0000 min_mon_release 16 (pacific) election_strategy: 1 0: [v2:10.45.128.28:3300/0,v1:10.45.128.28:6789/0] mon.a001s018 1: [v2:10.45.128.27:3300/0,v1:10.45.128.27:6789/0] mon.a001s017 # ceph -s cluster: id: 604d56db-2fab-45db-a9ea-c418f9a8cca8 health: HEALTH_OK services: mon: 2 daemons, quorum a001s018,a001s017 (age 18h) mgr: a001s016.ctmoay(active, since 109m), standbys: a001s017.bpygfm mds: 1/1 daemons up, 2 standby osd: 36 osds: 36 up (since 2h), 36 in (since 2y) rgw: 3 daemons active (3 hosts, 1 zones) data: volumes: 1/1 healthy pools: 43 pools, 1633 pgs objects: 51.64M objects, 77 TiB usage: 119 TiB used, 132 TiB / 252 TiB avail pgs: 1633 active+clean io: client: 356 MiB/s rd, 633 MiB/s wr, 506 op/s rd, 755 op/s wr # ceph orch ps --refresh | grep mon mon.a001s016 a001s016 running (91s) 7s ago 91s 14.5M 2048M 16.2.5 6e73176320aa dcd4705c2069 mon.a001s017 a001s017 running (18h) 7s ago 21h 978M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 8s ago 2y 1163M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ls | grep mon mon 3/3 9s ago 114s a001s016;a001s017;a001s018;count:3 # ceph log last cephadm 2024-04-01T18:56:29.631337+0000 mgr.a001s017.bpygfm (mgr.60881489) 18014760 : cephadm [INF] Schedule restart daemon osd.6 2024-04-01T18:57:05.983577+0000 mgr.a001s017.bpygfm (mgr.60881489) 18014780 : cephadm [INF] Schedule restart daemon osd.9 2024-04-01T18:57:17.919236+0000 mgr.a001s017.bpygfm (mgr.60881489) 18014788 : cephadm [INF] Schedule restart daemon osd.12 2024-04-01T18:57:59.823635+0000 mgr.a001s017.bpygfm (mgr.60881489) 18014811 : cephadm [INF] Schedule restart daemon osd.16 2024-04-01T18:58:26.039310+0000 mgr.a001s017.bpygfm (mgr.60881489) 18014826 : cephadm [INF] Schedule restart daemon osd.19 2024-04-01T18:58:57.278955+0000 mgr.a001s017.bpygfm (mgr.60881489) 18014844 : cephadm [INF] Schedule restart daemon osd.21 2024-04-01T18:59:36.656834+0000 mgr.a001s017.bpygfm (mgr.60881489) 18014867 : cephadm [INF] Schedule restart daemon osd.25 2024-04-01T19:00:03.272925+0000 mgr.a001s017.bpygfm (mgr.60881489) 18014882 : cephadm [INF] Schedule restart daemon osd.28 2024-04-01T19:00:28.348138+0000 mgr.a001s017.bpygfm (mgr.60881489) 18014897 : cephadm [INF] Schedule restart daemon osd.31 2024-04-01T19:00:56.297517+0000 mgr.a001s017.bpygfm (mgr.60881489) 18014913 : cephadm [INF] Schedule restart daemon osd.34 2024-04-01T19:38:21.552347+0000 mgr.a001s017.bpygfm (mgr.60881489) 18016056 : cephadm [INF] Schedule restart daemon mgr.a001s017.bpygfm 2024-04-01T19:38:21.603564+0000 mgr.a001s017.bpygfm (mgr.60881489) 18016057 : cephadm [INF] Failing over to other MGR 2024-04-01T21:21:24.414308+0000 mgr.a001s016.ctmoay (mgr.61827136) 3096 : cephadm [INF] Saving service mon spec with placement a001s017;a001s018;count:2 2024-04-01T21:25:02.815151+0000 mgr.a001s016.ctmoay (mgr.61827136) 3211 : cephadm [INF] Saving service mon spec with placement a001s017;a001s018;count:2 2024-04-01T21:25:02.844721+0000 mgr.a001s016.ctmoay (mgr.61827136) 3212 : cephadm [INF] Safe to remove mon.a001s016: not in monmap (['a001s018', 'a001s017']) 2024-04-01T21:25:02.845059+0000 mgr.a001s016.ctmoay (mgr.61827136) 3213 : cephadm [INF] Removing monitor a001s016 from monmap... 2024-04-01T21:25:02.846401+0000 mgr.a001s016.ctmoay (mgr.61827136) 3214 : cephadm [INF] Removing daemon mon.a001s016 from a001s016 2024-04-01T21:25:06.870895+0000 mgr.a001s016.ctmoay (mgr.61827136) 3218 : cephadm [INF] Removing key for mon. 2024-04-01T21:26:27.861600+0000 mgr.a001s016.ctmoay (mgr.61827136) 3266 : cephadm [INF] Saving service mon spec with placement a001s016;a001s017;a001s018;count:3 2024-04-01T21:26:38.247516+0000 mgr.a001s016.ctmoay (mgr.61827136) 3273 : cephadm [INF] Saving service mon spec with placement a001s016;a001s017;a001s018;count:3 2024-04-01T21:26:44.494080+0000 mgr.a001s016.ctmoay (mgr.61827136) 3277 : cephadm [INF] Deploying daemon mon.a001s016 on a001s016 # cat /var/lib/ceph/604d56db-2fab-45db-a9ea-c418f9a8cca8/mon.a001s016/unit.image docker.io/ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586 # ceph orch ps --daemon_type=mon NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mon.a001s016 a001s016 running (16m) 3m ago 16m 16.3M 2048M 16.2.5 6e73176320aa dcd4705c2069 mon.a001s017 a001s017 running (18h) 4m ago 21h 960M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 4m ago 2y 1155M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch daemon redeploy mon.a001s016 docker.io/ceph/daemon:latest-pacific Scheduled to redeploy mon.a001s016 on host 'a001s016' # ceph orch ps --daemon_type=mon --refresh NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mon.a001s016 a001s016 running (13s) 7s ago 18m 14.1M 2048M 16.2.5 6e73176320aa 1917b1edf4ab mon.a001s017 a001s017 running (18h) 6m ago 21h 960M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 6m ago 2y 1155M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 ============================================================================================= The second method also does not update ceph mon map: #ceph mon dump dumped monmap epoch 6 epoch 6 fsid 604d56db-2fab-45db-a9ea-c418f9a8cca8 last_changed 2024-03-31T23:54:18.692983+0000 created 2021-09-30T16:15:12.884602+0000 min_mon_release 16 (pacific) election_strategy: 1 0: [v2:10.45.128.28:3300/0,v1:10.45.128.28:6789/0] mon.a001s018 1: [v2:10.45.128.27:3300/0,v1:10.45.128.27:6789/0] mon.a001s017 # ceph orch ps --refresh | grep mon mon.a001s016 a001s016 running (81s) 63s ago 81s 14.2M 2048M 16.2.5 6e73176320aa 0c0ec88139e1 mon.a001s017 a001s017 running (19h) 62s ago 22h 965M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 63s ago 2y 1157M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch daemon stop mon.a001s016 Scheduled to stop mon.a001s016 on host 'a001s016' # ceph orch ps --refresh | grep mon mon.a001s016 a001s016 running (96s) 6s ago 96s 14.4M 2048M 16.2.5 6e73176320aa 0c0ec88139e1 mon.a001s017 a001s017 running (19h) 5s ago 22h 968M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 6s ago 2y 1157M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s016 a001s016 running (98s) 7s ago 98s 14.4M 2048M 16.2.5 6e73176320aa 0c0ec88139e1 mon.a001s017 a001s017 running (19h) 7s ago 22h 968M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 8s ago 2y 1157M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s016 a001s016 running (100s) 1s ago 100s 14.4M 2048M 16.2.5 6e73176320aa 0c0ec88139e1 mon.a001s017 a001s017 running (19h) 9s ago 22h 968M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 10s ago 2y 1157M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s016 a001s016 running (106s) 7s ago 106s 14.4M 2048M 16.2.5 6e73176320aa 0c0ec88139e1 mon.a001s017 a001s017 running (19h) 15s ago 22h 968M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 16s ago 2y 1157M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s016 a001s016 stopped 0s ago 108s - 2048M <unknown> <unknown> <unknown> mon.a001s017 a001s017 running (19h) 17s ago 22h 968M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 0s ago 2y 1160M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s016 a001s016 stopped 2s ago 111s - 2048M <unknown> <unknown> <unknown> mon.a001s017 a001s017 running (19h) 2s ago 22h 972M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 3s ago 2y 1160M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch daemon rm mon.a001s016 Error EINVAL: must pass --force to REMOVE daemon with potentially PRECIOUS DATA for mon.a001s016 # ceph orch daemon rm mon.a001s016 --force Removed mon.a001s016 from host 'a001s016' # ceph orch ps --refresh | grep mon mon.a001s017 a001s017 running (19h) 30s ago 22h 976M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 0s ago 2y 1166M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 #ceph orch ps --refresh | grep mon mon.a001s017 a001s017 running (19h) 2s ago 22h 982M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 3s ago 2y 1166M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s016 a001s016 starting - - - 2048M <unknown> <unknown> <unknown> mon.a001s017 a001s017 running (19h) 6s ago 22h 982M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 7s ago 2y 1166M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s016 a001s016 running (8s) 2s ago 8s 14.4M 2048M 16.2.5 6e73176320aa 39db8cfba7e1 mon.a001s017 a001s017 running (19h) 1s ago 22h 987M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 2s ago 2y 1171M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph mon dump dumped monmap epoch 6 epoch 6 fsid 604d56db-2fab-45db-a9ea-c418f9a8cca8 last_changed 2024-03-31T23:54:18.692983+0000 created 2021-09-30T16:15:12.884602+0000 min_mon_release 16 (pacific) election_strategy: 1 0: [v2:10.45.128.28:3300/0,v1:10.45.128.28:6789/0] mon.a001s018 1: [v2:10.45.128.27:3300/0,v1:10.45.128.27:6789/0] mon.a001s017 # -----Original Message----- From: Adiga, Anantha <anantha.adiga@intel.com> Sent: Monday, April 1, 2024 2:01 PM To: Eugen Block <eblock@nde.ag> Cc: ceph-users@ceph.io Subject: [ceph-users] Re: ceph status not showing correct monitor services Thank you. I will try the export and import method first. Thank you, Anantha -----Original Message----- From: Eugen Block <eblock@nde.ag> Sent: Monday, April 1, 2024 1:57 PM To: Adiga, Anantha <anantha.adiga@intel.com> Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Re: ceph status not showing correct monitor services I have two approaches in mind, first one (and preferred) would be to edit the mon spec to first remove mon.a001s016 and have a clean state. Get the current spec with: ceph orch ls mon --export > mon-edit.yaml Edit the spec file so that mon.a001s016 is not part of it, then apply: ceph orch apply -i mon-edit.yaml This should remove the mon.a001s016 daemon. Then wait a few minutes or so (until the daemon is actually gone, check locally on the node with 'cephadm ls' and in /var/lib/ceph/<FSID>/removed) and add it back to the spec file, then apply again. I would expect a third MON to be deployed. If that doesn't work for some reason you'll need to inspect logs to find the root cause. The second approach would be to remove and add the daemon manually: ceph orch daemon rm mon.a001s016 Wait until it's really gone, then add it: ceph orch daemon add mon a001s016 Not entirely sure about the daemon add mon command, you might need to provide something else, I'm typing this by heart. Zitat von "Adiga, Anantha" <anantha.adiga@intel.com>:

Hi Eugen,

Yes that is it. OSDs were restarted since mon a001s017 was reporting is low on available space. How to update the mon map to add mon.a001s016 as it is already online? And how to update mgr map to include standby mgr.a001s018 as it is also running.

ceph mon dump dumped monmap epoch 6 epoch 6 fsid 604d56db-2fab-45db-a9ea-c418f9a8cca8 last_changed 2024-03-31T23:54:18.692983+0000 created 2021-09-30T16:15:12.884602+0000 min_mon_release 16 (pacific) election_strategy: 1 0: [v2:10.45.128.28:3300/0,v1:10.45.128.28:6789/0] mon.a001s018 1: [v2:10.45.128.27:3300/0,v1:10.45.128.27:6789/0] mon.a001s017

Thank you, Anantha

-----Original Message----- From: Eugen Block <eblock@nde.ag> Sent: Monday, April 1, 2024 1:10 PM To: ceph-users@ceph.io Subject: [ceph-users] Re: ceph status not showing correct monitor services

Maybe it’s just not in the monmap? Can you show the output of:

ceph mon dump

Did you do any maintenance (apparently OSDs restarted recently) and maybe accidentally removed a MON from the monmap?

Zitat von "Adiga, Anantha" <anantha.adiga@intel.com>:

...

Hi Anthony,

Seeing it since last after noon. It is same with mgr services as , "ceph -s" is reporting only TWO instead of THREE

Also mon and mgr shows " is_active: false" see below.

# ceph orch ps --daemon_type=mgr NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mgr.a001s016.ctmoay a001s016 *:8443 running (18M) 3m ago 23M 206M - 16.2.5 6e73176320aa 169cafcbbb99 mgr.a001s017.bpygfm a001s017 *:8443 running (19M) 3m ago 23M 332M - 16.2.5 6e73176320aa 97257195158c mgr.a001s018.hcxnef a001s018 *:8443 running (20M) 3m ago 23M 113M - 16.2.5 6e73176320aa 21ba5896cee2

# ceph orch ls --service_name=mgr NAME PORTS RUNNING REFRESHED AGE PLACEMENT mgr 3/3 3m ago 23M a001s016;a001s017;a001s018;count:3

# ceph orch ps --daemon_type=mon --format=json-pretty

[ { "container_id": "8484a912f96a", "container_image_digests": [

docker.io/ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586<mailto:docker.io/ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586> ], "container_image_id": "6e73176320aaccf3b3fb660b9945d0514222bd7a83e28b96e8440c630ba6891f", "container_image_name": docker.io/ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586<mailto:docker.io/ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586>, "created": "2024-03-31T23:55:16.164155Z", "daemon_id": "a001s016", "daemon_type": "mon", "hostname": "a001s016", "is_active": false, <== why is it false "last_refresh": "2024-04-01T19:38:30.929014Z", "memory_request": 2147483648, "memory_usage": 761685606, "ports": [], "service_name": "mon", "started": "2024-03-31T23:55:16.268266Z", "status": 1, "status_desc": "running", "version": "16.2.5" },

Thank you, Anantha

From: Anthony D'Atri <aad@dreamsnake.net> Sent: Monday, April 1, 2024 12:25 PM To: Adiga, Anantha <anantha.adiga@intel.com> Cc: ceph-users@ceph.io Subject: Re: [ceph-users] ceph status not showing correct monitor services

a001s017.bpygfm(active, since 13M), standbys: a001s016.ctmoay

Looks like you just had an mgr failover? Could be that the secondary mgr hasn't caught up with current events. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

You can add a mon manually to the monmap, but that requires a downtime of the mons. Here's an example [1] how to modify the monmap (including network change which you don't need, of course). But that would be my last resort, first I would try to find out why the MON fails to join the quorum. What is that mon.a001s016 logging, and what are the other two logging? Do you have another host where you could place a mon daemon to see if that works? [1] https://docs.ceph.com/en/latest/rados/operations/add-or-rm-mons/#example-pro... Zitat von "Adiga, Anantha" <anantha.adiga@intel.com>:

# ceph mon stat e6: 2 mons at {a001s017=[v2:10.45.128.27:3300/0,v1:10.45.128.27:6789/0],a001s018=[v2:10.45.128.28:3300/0,v1:10.45.128.28:6789/0]}, election epoch 162, leader 0 a001s018, quorum 0,1 a001s018,a001s017

# ceph orch ps | grep mon mon.a001s016 a001s016 running (3h) 6m ago 3h 527M 2048M 16.2.5 6e73176320aa 39db8cfba7e1 mon.a001s017 a001s017 running (22h) 47s ago 1h 993M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 48s ago 2y 1167M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54

# ceph mgr stat { "epoch": 1130365, "available": true, "active_name": "a001s016.ctmoay", "num_standby": 1 }

# ceph orch ps | grep mgr mgr.a001s016.ctmoay a001s016 *:8443 running (18M) 109s ago 23M 518M - 16.2.5 6e73176320aa 169cafcbbb99 mgr.a001s017.bpygfm a001s017 *:8443 running (19M) 5m ago 23M 501M - 16.2.5 6e73176320aa 97257195158c mgr.a001s018.hcxnef a001s018 *:8443 running (20M) 5m ago 23M 113M - 16.2.5 6e73176320aa 21ba5896cee2

# ceph orch ls --service_name=mgr --export service_type: mgr service_name: mgr placement: count: 3 hosts: - a001s016 - a001s017 - a001s018

# ceph orch ls --service_name=mon --export service_type: mon service_name: mon placement: count: 3 hosts: - a001s016 - a001s017 - a001s018

-----Original Message----- From: Adiga, Anantha Sent: Monday, April 1, 2024 6:06 PM To: Eugen Block <eblock@nde.ag> Cc: ceph-users@ceph.io Subject: RE: [ceph-users] Re: ceph status not showing correct monitor services

# ceph tell mon.a001s016 mon_status Error ENOENT: problem getting command descriptions from mon.a001s016

a001s016 is outside quorum see below

# ceph tell mon.a001s017 mon_status { "name": "a001s017", "rank": 1, "state": "peon", "election_epoch": 162, "quorum": [ 0, 1 ], "quorum_age": 79938, "features": { "required_con": "2449958747317026820", "required_mon": [ "kraken", "luminous", "mimic", "osdmap-prune", "nautilus", "octopus", "pacific", "elector-pinging" ], "quorum_con": "4540138297136906239", "quorum_mon": [ "kraken", "luminous", "mimic", "osdmap-prune", "nautilus", "octopus", "pacific", "elector-pinging" ] }, "outside_quorum": [], "extra_probe_peers": [ { "addrvec": [ { "type": "v2", "addr": "10.45.128.26:3300", "nonce": 0 }, { "type": "v1", "addr": "10.45.128.26:6789", "nonce": 0 } ] } ], "sync_provider": [], "monmap": { "epoch": 6, "fsid": "604d56db-2fab-45db-a9ea-c418f9a8cca8", "modified": "2024-03-31T23:54:18.692983Z", "created": "2021-09-30T16:15:12.884602Z", "min_mon_release": 16, "min_mon_release_name": "pacific", "election_strategy": 1, "disallowed_leaders: ": "", "stretch_mode": false, "features": { "persistent": [ "kraken", "luminous", "mimic", "osdmap-prune", "nautilus", "octopus", "pacific", "elector-pinging" ], "optional": [] }, "mons": [ { "rank": 0, "name": "a001s018", "public_addrs": { "addrvec": [ { "type": "v2", "addr": "10.45.128.28:3300", "nonce": 0 }, { "type": "v1", "addr": "10.45.128.28:6789", "nonce": 0 } ] }, "addr": "10.45.128.28:6789/0", "public_addr": "10.45.128.28:6789/0", "priority": 0, "weight": 0, "crush_location": "{}" }, { "rank": 1, "name": "a001s017", "public_addrs": { "addrvec": [ { "type": "v2", "addr": "10.45.128.27:3300", "nonce": 0 }, { "type": "v1", "addr": "10.45.128.27:6789", "nonce": 0 } ] }, "addr": "10.45.128.27:6789/0", "public_addr": "10.45.128.27:6789/0", "priority": 0, "weight": 0, "crush_location": "{}" } ] }, "feature_map": { "mon": [ { "features": "0x3f01cfb9fffdffff", "release": "luminous", "num": 1 } ], "mds": [ { "features": "0x3f01cfb9fffdffff", "release": "luminous", "num": 3 } ], "osd": [ { "features": "0x3f01cfb9fffdffff", "release": "luminous", "num": 15 } ], "client": [ { "features": "0x2f018fb86aa42ada", "release": "luminous", "num": 50 }, { "features": "0x2f018fb87aa4aafe", "release": "luminous", "num": 40 }, { "features": "0x3f01cfb8ffedffff", "release": "luminous", "num": 1 }, { "features": "0x3f01cfb9fffdffff", "release": "luminous", "num": 72 } ] }, "stretch_mode": false }

# ceph mon dump dumped monmap epoch 6 epoch 6 fsid 604d56db-2fab-45db-a9ea-c418f9a8cca8 last_changed 2024-03-31T23:54:18.692983+0000 created 2021-09-30T16:15:12.884602+0000 min_mon_release 16 (pacific) election_strategy: 1 0: [v2:10.45.128.28:3300/0,v1:10.45.128.28:6789/0] mon.a001s018 1: [v2:10.45.128.27:3300/0,v1:10.45.128.27:6789/0] mon.a001s017 root@a001s016:/var/run/ceph/604d56db-2fab-45db-a9ea-c418f9a8cca8#

-----Original Message----- From: Adiga, Anantha Sent: Monday, April 1, 2024 3:20 PM To: Adiga, Anantha <anantha.adiga@intel.com>; Eugen Block <eblock@nde.ag> Cc: ceph-users@ceph.io Subject: RE: [ceph-users] Re: ceph status not showing correct monitor services

Both methods are not update the mon map, is there a way to inject mon.a001s016 into the current mon map?

# ceph mon dump dumped monmap epoch 6 epoch 6 fsid 604d56db-2fab-45db-a9ea-c418f9a8cca8 last_changed 2024-03-31T23:54:18.692983+0000 created 2021-09-30T16:15:12.884602+0000 min_mon_release 16 (pacific) election_strategy: 1 0: [v2:10.45.128.28:3300/0,v1:10.45.128.28:6789/0] mon.a001s018 1: [v2:10.45.128.27:3300/0,v1:10.45.128.27:6789/0] mon.a001s017

# ceph tell mon.a001s016 mon_status Error ENOENT: problem getting command descriptions from mon.a001s016

# ceph tell mon.a001s016 mon_status Error ENOENT: problem getting command descriptions from mon.a001s016

# ceph tell mon.a001s017 mon_status { "name": "a001s017", "rank": 1, "state": "peon", "election_epoch": 162, "quorum": [ 0, 1 ], "quorum_age": 69551, "features": { .............. ..............

# ceph orch ls --service_name=mon --export > mon3.yml service_type: mon service_name: mon placement: count: 3 hosts: - a001s016 - a001s017 - a001s018

# cp mon3.yml mon2.yml # vi mon2.yml #cat mon2.yml service_type: mon service_name: mon placement: count: 2 hosts: - a001s017 - a001s018

# ceph orch apply -i mon2.yml --dry-run WARNING! Dry-Runs are snapshots of a certain point in time and are bound to the current inventory setup. If any on these conditions changes, the preview will be invalid. Please make sure to have a minimal timeframe between planning and applying the specs. #################### SERVICESPEC PREVIEWS #################### +---------+------+--------+--------------+ |SERVICE |NAME |ADD_TO |REMOVE_FROM | +---------+------+--------+--------------+ |mon |mon | |mon.a001s016 | +---------+------+--------+--------------+ ################ OSDSPEC PREVIEWS ################ +---------+------+------+------+----+-----+ |SERVICE |NAME |HOST |DATA |DB |WAL | +---------+------+------+------+----+-----+ +---------+------+------+------+----+-----+

# ceph orch ls --service_name=mon --refresh NAME PORTS RUNNING REFRESHED AGE PLACEMENT mon 3/3 5m ago 18h a001s016;a001s017;a001s018;count:3

# ceph orch ps --refresh | grep mon mon.a001s016 a001s016 running (21h) 2s ago 21h 734M 2048M 16.2.5 6e73176320aa 8484a912f96a mon.a001s017 a001s017 running (18h) 2s ago 21h 976M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 2s ago 2y 1164M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s016 a001s016 running (21h) 37s ago 21h 734M 2048M 16.2.5 6e73176320aa 8484a912f96a mon.a001s017 a001s017 running (18h) 37s ago 21h 977M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 38s ago 2y 1166M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54

# ceph orch apply -i mon2.yml Scheduled mon update... # ceph orch ps --refresh | grep mon mon.a001s016 a001s016 running (21h) 21s ago 21h 734M 2048M 16.2.5 6e73176320aa 8484a912f96a mon.a001s017 a001s017 running (18h) 20s ago 21h 962M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 21s ago 2y 1156M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s017 a001s017 running (18h) 23s ago 21h 962M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 24s ago 2y 1156M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s017 a001s017 running (18h) 27s ago 21h 962M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 0s ago 2y 1154M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s017 a001s017 running (18h) 2s ago 21h 960M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 3s ago 2y 1154M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s017 a001s017 running (18h) 5s ago 21h 960M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 0s ago 2y 1154M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s017 a001s017 running (18h) 2s ago 21h 962M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 2s ago 2y 1154M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54

# ls /var/lib/ceph/604d56db-2fab-45db-a9ea-c418f9a8cca8/ cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931 keepalived.nfs.nfs.a001s016.ofqyjl osd.1 osd.22 osd.4 crash mds.cephfs.a001s016.vtyivn osd.10 osd.24 osd.7 crash.a001s016 mgr.a001s016.ctmoay osd.13 osd.27 removed haproxy.nfs.nfs.a001s016.vpjtxt nfs.nfs.0.0.a001s016.wumnjs osd.15 osd.30 rgw.ceph.a001s016.gjyanl home node-exporter.a001s016 osd.18 osd.33

# ceph mon dump dumped monmap epoch 6 epoch 6 fsid 604d56db-2fab-45db-a9ea-c418f9a8cca8 last_changed 2024-03-31T23:54:18.692983+0000 created 2021-09-30T16:15:12.884602+0000 min_mon_release 16 (pacific) election_strategy: 1 0: [v2:10.45.128.28:3300/0,v1:10.45.128.28:6789/0] mon.a001s018 1: [v2:10.45.128.27:3300/0,v1:10.45.128.27:6789/0] mon.a001s017

# cat mon3.yml service_type: mon service_name: mon placement: count: 3 hosts: - a001s016 - a001s017 - a001s018

# ceph orch apply -i mon3.yml --dry-run WARNING! Dry-Runs are snapshots of a certain point in time and are bound to the current inventory setup. If any on these conditions changes, the preview will be invalid. Please make sure to have a minimal timeframe between planning and applying the specs. #################### SERVICESPEC PREVIEWS #################### +---------+------+----------+-------------+ |SERVICE |NAME |ADD_TO |REMOVE_FROM | +---------+------+----------+-------------+ |mon |mon |a001s016 | | +---------+------+----------+-------------+ ################ OSDSPEC PREVIEWS ################ +---------+------+------+------+----+-----+ |SERVICE |NAME |HOST |DATA |DB |WAL | +---------+------+------+------+----+-----+ +---------+------+------+------+----+-----+

# ceph orch apply -i mon3.yml Scheduled mon update... # ceph orch ps --refresh | grep mon mon.a001s016 a001s016 starting - - - 2048M <unknown> <unknown> <unknown> mon.a001s017 a001s017 running (18h) 2s ago 21h 960M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 3s ago 2y 1152M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s016 a001s016 running (6s) 0s ago 6s 14.4M 2048M 16.2.5 6e73176320aa dcd4705c2069 mon.a001s017 a001s017 running (18h) 8s ago 21h 960M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 9s ago 2y 1152M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s016 a001s016 running (10s) 4s ago 10s 14.4M 2048M 16.2.5 6e73176320aa dcd4705c2069 mon.a001s017 a001s017 running (18h) 12s ago 21h 960M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 13s ago 2y 1152M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54

# ls /var/lib/ceph/604d56db-2fab-45db-a9ea-c418f9a8cca8/ cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931 mgr.a001s016.ctmoay osd.15 osd.4 crash mon.a001s016 osd.18 osd.7 crash.a001s016 nfs.nfs.0.0.a001s016.wumnjs osd.22 removed haproxy.nfs.nfs.a001s016.vpjtxt node-exporter.a001s016 osd.24 rgw.ceph.a001s016.gjyanl home osd.1 osd.27 keepalived.nfs.nfs.a001s016.ofqyjl osd.10 osd.30 mds.cephfs.a001s016.vtyivn osd.13 osd.33

# ceph mon dump dumped monmap epoch 6 epoch 6 fsid 604d56db-2fab-45db-a9ea-c418f9a8cca8 last_changed 2024-03-31T23:54:18.692983+0000 created 2021-09-30T16:15:12.884602+0000 min_mon_release 16 (pacific) election_strategy: 1 0: [v2:10.45.128.28:3300/0,v1:10.45.128.28:6789/0] mon.a001s018 1: [v2:10.45.128.27:3300/0,v1:10.45.128.27:6789/0] mon.a001s017

# ceph -s cluster: id: 604d56db-2fab-45db-a9ea-c418f9a8cca8 health: HEALTH_OK

services: mon: 2 daemons, quorum a001s018,a001s017 (age 18h) mgr: a001s016.ctmoay(active, since 109m), standbys: a001s017.bpygfm mds: 1/1 daemons up, 2 standby osd: 36 osds: 36 up (since 2h), 36 in (since 2y) rgw: 3 daemons active (3 hosts, 1 zones)

data: volumes: 1/1 healthy pools: 43 pools, 1633 pgs objects: 51.64M objects, 77 TiB usage: 119 TiB used, 132 TiB / 252 TiB avail pgs: 1633 active+clean

io: client: 356 MiB/s rd, 633 MiB/s wr, 506 op/s rd, 755 op/s wr

# ceph orch ps --refresh | grep mon mon.a001s016 a001s016 running (91s) 7s ago 91s 14.5M 2048M 16.2.5 6e73176320aa dcd4705c2069 mon.a001s017 a001s017 running (18h) 7s ago 21h 978M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 8s ago 2y 1163M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ls | grep mon mon 3/3 9s ago 114s a001s016;a001s017;a001s018;count:3

# ceph log last cephadm 2024-04-01T18:56:29.631337+0000 mgr.a001s017.bpygfm (mgr.60881489) 18014760 : cephadm [INF] Schedule restart daemon osd.6 2024-04-01T18:57:05.983577+0000 mgr.a001s017.bpygfm (mgr.60881489) 18014780 : cephadm [INF] Schedule restart daemon osd.9 2024-04-01T18:57:17.919236+0000 mgr.a001s017.bpygfm (mgr.60881489) 18014788 : cephadm [INF] Schedule restart daemon osd.12 2024-04-01T18:57:59.823635+0000 mgr.a001s017.bpygfm (mgr.60881489) 18014811 : cephadm [INF] Schedule restart daemon osd.16 2024-04-01T18:58:26.039310+0000 mgr.a001s017.bpygfm (mgr.60881489) 18014826 : cephadm [INF] Schedule restart daemon osd.19 2024-04-01T18:58:57.278955+0000 mgr.a001s017.bpygfm (mgr.60881489) 18014844 : cephadm [INF] Schedule restart daemon osd.21 2024-04-01T18:59:36.656834+0000 mgr.a001s017.bpygfm (mgr.60881489) 18014867 : cephadm [INF] Schedule restart daemon osd.25 2024-04-01T19:00:03.272925+0000 mgr.a001s017.bpygfm (mgr.60881489) 18014882 : cephadm [INF] Schedule restart daemon osd.28 2024-04-01T19:00:28.348138+0000 mgr.a001s017.bpygfm (mgr.60881489) 18014897 : cephadm [INF] Schedule restart daemon osd.31 2024-04-01T19:00:56.297517+0000 mgr.a001s017.bpygfm (mgr.60881489) 18014913 : cephadm [INF] Schedule restart daemon osd.34 2024-04-01T19:38:21.552347+0000 mgr.a001s017.bpygfm (mgr.60881489) 18016056 : cephadm [INF] Schedule restart daemon mgr.a001s017.bpygfm 2024-04-01T19:38:21.603564+0000 mgr.a001s017.bpygfm (mgr.60881489) 18016057 : cephadm [INF] Failing over to other MGR 2024-04-01T21:21:24.414308+0000 mgr.a001s016.ctmoay (mgr.61827136) 3096 : cephadm [INF] Saving service mon spec with placement a001s017;a001s018;count:2 2024-04-01T21:25:02.815151+0000 mgr.a001s016.ctmoay (mgr.61827136) 3211 : cephadm [INF] Saving service mon spec with placement a001s017;a001s018;count:2 2024-04-01T21:25:02.844721+0000 mgr.a001s016.ctmoay (mgr.61827136) 3212 : cephadm [INF] Safe to remove mon.a001s016: not in monmap (['a001s018', 'a001s017']) 2024-04-01T21:25:02.845059+0000 mgr.a001s016.ctmoay (mgr.61827136) 3213 : cephadm [INF] Removing monitor a001s016 from monmap... 2024-04-01T21:25:02.846401+0000 mgr.a001s016.ctmoay (mgr.61827136) 3214 : cephadm [INF] Removing daemon mon.a001s016 from a001s016 2024-04-01T21:25:06.870895+0000 mgr.a001s016.ctmoay (mgr.61827136) 3218 : cephadm [INF] Removing key for mon. 2024-04-01T21:26:27.861600+0000 mgr.a001s016.ctmoay (mgr.61827136) 3266 : cephadm [INF] Saving service mon spec with placement a001s016;a001s017;a001s018;count:3 2024-04-01T21:26:38.247516+0000 mgr.a001s016.ctmoay (mgr.61827136) 3273 : cephadm [INF] Saving service mon spec with placement a001s016;a001s017;a001s018;count:3 2024-04-01T21:26:44.494080+0000 mgr.a001s016.ctmoay (mgr.61827136) 3277 : cephadm [INF] Deploying daemon mon.a001s016 on a001s016

# cat /var/lib/ceph/604d56db-2fab-45db-a9ea-c418f9a8cca8/mon.a001s016/unit.image docker.io/ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586 # ceph orch ps --daemon_type=mon NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mon.a001s016 a001s016 running (16m) 3m ago 16m 16.3M 2048M 16.2.5 6e73176320aa dcd4705c2069 mon.a001s017 a001s017 running (18h) 4m ago 21h 960M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 4m ago 2y 1155M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54

# ceph orch daemon redeploy mon.a001s016 docker.io/ceph/daemon:latest-pacific Scheduled to redeploy mon.a001s016 on host 'a001s016'

# ceph orch ps --daemon_type=mon --refresh NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mon.a001s016 a001s016 running (13s) 7s ago 18m 14.1M 2048M 16.2.5 6e73176320aa 1917b1edf4ab mon.a001s017 a001s017 running (18h) 6m ago 21h 960M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 6m ago 2y 1155M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54

=============================================================================================

The second method also does not update ceph mon map:

#ceph mon dump dumped monmap epoch 6 epoch 6 fsid 604d56db-2fab-45db-a9ea-c418f9a8cca8 last_changed 2024-03-31T23:54:18.692983+0000 created 2021-09-30T16:15:12.884602+0000 min_mon_release 16 (pacific) election_strategy: 1 0: [v2:10.45.128.28:3300/0,v1:10.45.128.28:6789/0] mon.a001s018 1: [v2:10.45.128.27:3300/0,v1:10.45.128.27:6789/0] mon.a001s017

# ceph orch ps --refresh | grep mon mon.a001s016 a001s016 running (81s) 63s ago 81s 14.2M 2048M 16.2.5 6e73176320aa 0c0ec88139e1 mon.a001s017 a001s017 running (19h) 62s ago 22h 965M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 63s ago 2y 1157M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54

# ceph orch daemon stop mon.a001s016 Scheduled to stop mon.a001s016 on host 'a001s016' # ceph orch ps --refresh | grep mon mon.a001s016 a001s016 running (96s) 6s ago 96s 14.4M 2048M 16.2.5 6e73176320aa 0c0ec88139e1 mon.a001s017 a001s017 running (19h) 5s ago 22h 968M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 6s ago 2y 1157M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s016 a001s016 running (98s) 7s ago 98s 14.4M 2048M 16.2.5 6e73176320aa 0c0ec88139e1 mon.a001s017 a001s017 running (19h) 7s ago 22h 968M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 8s ago 2y 1157M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s016 a001s016 running (100s) 1s ago 100s 14.4M 2048M 16.2.5 6e73176320aa 0c0ec88139e1 mon.a001s017 a001s017 running (19h) 9s ago 22h 968M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 10s ago 2y 1157M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s016 a001s016 running (106s) 7s ago 106s 14.4M 2048M 16.2.5 6e73176320aa 0c0ec88139e1 mon.a001s017 a001s017 running (19h) 15s ago 22h 968M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 16s ago 2y 1157M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s016 a001s016 stopped 0s ago 108s - 2048M <unknown> <unknown> <unknown> mon.a001s017 a001s017 running (19h) 17s ago 22h 968M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 0s ago 2y 1160M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s016 a001s016 stopped 2s ago 111s - 2048M <unknown> <unknown> <unknown> mon.a001s017 a001s017 running (19h) 2s ago 22h 972M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 3s ago 2y 1160M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54

# ceph orch daemon rm mon.a001s016 Error EINVAL: must pass --force to REMOVE daemon with potentially PRECIOUS DATA for mon.a001s016

# ceph orch daemon rm mon.a001s016 --force Removed mon.a001s016 from host 'a001s016'

# ceph orch ps --refresh | grep mon mon.a001s017 a001s017 running (19h) 30s ago 22h 976M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 0s ago 2y 1166M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 #ceph orch ps --refresh | grep mon mon.a001s017 a001s017 running (19h) 2s ago 22h 982M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 3s ago 2y 1166M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s016 a001s016 starting - - - 2048M <unknown> <unknown> <unknown> mon.a001s017 a001s017 running (19h) 6s ago 22h 982M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 7s ago 2y 1166M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s016 a001s016 running (8s) 2s ago 8s 14.4M 2048M 16.2.5 6e73176320aa 39db8cfba7e1 mon.a001s017 a001s017 running (19h) 1s ago 22h 987M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 2s ago 2y 1171M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54

# ceph mon dump dumped monmap epoch 6 epoch 6 fsid 604d56db-2fab-45db-a9ea-c418f9a8cca8 last_changed 2024-03-31T23:54:18.692983+0000 created 2021-09-30T16:15:12.884602+0000 min_mon_release 16 (pacific) election_strategy: 1 0: [v2:10.45.128.28:3300/0,v1:10.45.128.28:6789/0] mon.a001s018 1: [v2:10.45.128.27:3300/0,v1:10.45.128.27:6789/0] mon.a001s017

#

-----Original Message----- From: Adiga, Anantha <anantha.adiga@intel.com> Sent: Monday, April 1, 2024 2:01 PM To: Eugen Block <eblock@nde.ag> Cc: ceph-users@ceph.io Subject: [ceph-users] Re: ceph status not showing correct monitor services

Thank you. I will try the export and import method first.

Thank you, Anantha

-----Original Message----- From: Eugen Block <eblock@nde.ag> Sent: Monday, April 1, 2024 1:57 PM To: Adiga, Anantha <anantha.adiga@intel.com> Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Re: ceph status not showing correct monitor services

I have two approaches in mind, first one (and preferred) would be to edit the mon spec to first remove mon.a001s016 and have a clean state. Get the current spec with:

ceph orch ls mon --export > mon-edit.yaml

Edit the spec file so that mon.a001s016 is not part of it, then apply:

ceph orch apply -i mon-edit.yaml

This should remove the mon.a001s016 daemon. Then wait a few minutes or so (until the daemon is actually gone, check locally on the node with 'cephadm ls' and in /var/lib/ceph/<FSID>/removed) and add it back to the spec file, then apply again. I would expect a third MON to be deployed. If that doesn't work for some reason you'll need to inspect logs to find the root cause.

The second approach would be to remove and add the daemon manually:

ceph orch daemon rm mon.a001s016

Wait until it's really gone, then add it:

ceph orch daemon add mon a001s016

Not entirely sure about the daemon add mon command, you might need to provide something else, I'm typing this by heart.

Zitat von "Adiga, Anantha" <anantha.adiga@intel.com>:

...

Hi Eugen,

Yes that is it. OSDs were restarted since mon a001s017 was reporting is low on available space. How to update the mon map to add mon.a001s016 as it is already online? And how to update mgr map to include standby mgr.a001s018 as it is also running.

ceph mon dump dumped monmap epoch 6 epoch 6 fsid 604d56db-2fab-45db-a9ea-c418f9a8cca8 last_changed 2024-03-31T23:54:18.692983+0000 created 2021-09-30T16:15:12.884602+0000 min_mon_release 16 (pacific) election_strategy: 1 0: [v2:10.45.128.28:3300/0,v1:10.45.128.28:6789/0] mon.a001s018 1: [v2:10.45.128.27:3300/0,v1:10.45.128.27:6789/0] mon.a001s017

Thank you, Anantha

-----Original Message----- From: Eugen Block <eblock@nde.ag> Sent: Monday, April 1, 2024 1:10 PM To: ceph-users@ceph.io Subject: [ceph-users] Re: ceph status not showing correct monitor services

Maybe it’s just not in the monmap? Can you show the output of:

ceph mon dump

Did you do any maintenance (apparently OSDs restarted recently) and maybe accidentally removed a MON from the monmap?

Zitat von "Adiga, Anantha" <anantha.adiga@intel.com>:

...

Hi Anthony,

Seeing it since last after noon. It is same with mgr services as , "ceph -s" is reporting only TWO instead of THREE

Also mon and mgr shows " is_active: false" see below.

# ceph orch ps --daemon_type=mgr NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mgr.a001s016.ctmoay a001s016 *:8443 running (18M) 3m ago 23M 206M - 16.2.5 6e73176320aa 169cafcbbb99 mgr.a001s017.bpygfm a001s017 *:8443 running (19M) 3m ago 23M 332M - 16.2.5 6e73176320aa 97257195158c mgr.a001s018.hcxnef a001s018 *:8443 running (20M) 3m ago 23M 113M - 16.2.5 6e73176320aa 21ba5896cee2

# ceph orch ls --service_name=mgr NAME PORTS RUNNING REFRESHED AGE PLACEMENT mgr 3/3 3m ago 23M a001s016;a001s017;a001s018;count:3

# ceph orch ps --daemon_type=mon --format=json-pretty

[ { "container_id": "8484a912f96a", "container_image_digests": [

docker.io/ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586<mailto:docker.io/ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586> ], "container_image_id": "6e73176320aaccf3b3fb660b9945d0514222bd7a83e28b96e8440c630ba6891f", "container_image_name": docker.io/ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586<mailto:docker.io/ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586>, "created": "2024-03-31T23:55:16.164155Z", "daemon_id": "a001s016", "daemon_type": "mon", "hostname": "a001s016", "is_active": false, <== why is it false "last_refresh": "2024-04-01T19:38:30.929014Z", "memory_request": 2147483648, "memory_usage": 761685606, "ports": [], "service_name": "mon", "started": "2024-03-31T23:55:16.268266Z", "status": 1, "status_desc": "running", "version": "16.2.5" },

Thank you, Anantha

From: Anthony D'Atri <aad@dreamsnake.net> Sent: Monday, April 1, 2024 12:25 PM To: Adiga, Anantha <anantha.adiga@intel.com> Cc: ceph-users@ceph.io Subject: Re: [ceph-users] ceph status not showing correct monitor services

a001s017.bpygfm(active, since 13M), standbys: a001s016.ctmoay

Looks like you just had an mgr failover? Could be that the secondary mgr hasn't caught up with current events. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Hi Eugen, Noticed this in the config dump: Why only "mon.a001s016 " listed? And this is the one that is not listed in "ceph -s" mon advanced auth_allow_insecure_global_id_reclaim false mon advanced auth_expose_insecure_global_id_reclaim false mon advanced mon_compact_on_start true mon.a001s016 basic container_image docker.io/ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586 * mgr advanced mgr/cephadm/container_image_base docker.io/ceph/daemon mgr advanced mgr/cephadm/container_image_node_exporter docker.io/prom/node-exporter:v0.17.0 cluster: id: 604d56db-2fab-45db-a9ea-c418f9a8cca8 health: HEALTH_OK services: mon: 2 daemons, quorum a001s018,a001s017 (age 45h) mgr: a001s016.ctmoay(active, since 28h), standbys: a001s017.bpygfm mds: 1/1 daemons up, 2 standby osd: 36 osds: 36 up (since 29h), 36 in (since 2y) rgw: 3 daemons active (3 hosts, 1 zones) var lib mon unit.image ==================== a001s016: # cat /var/lib/ceph/604d56db-2fab-45db-a9ea-c418f9a8cca8/mon.a001s016/unit.image docker.io/ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586 a001s017: # cat /var/lib/ceph/604d56db-2fab-45db-a9ea-c418f9a8cca8/mon.a001s017/unit.image docker.io/ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586 a001s018: # cat /var/lib/ceph/604d56db-2fab-45db-a9ea-c418f9a8cca8/mon.a001s018/unit.image docker.io/ceph/daemon:latest-pacific ceph image tag, digest from docker inspect of: ceph/daemon latest-pacific 6e73176320aa 2 years ago 1.27GB ================================================================================================== a001s016: "Id": "sha256:6e73176320aaccf3b3fb660b9945d0514222bd7a83e28b96e8440c630ba6891f", "RepoTags": [ "ceph/daemon:latest-pacific" "RepoDigests": [ "ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586" a001s017: "Id": "sha256:6e73176320aaccf3b3fb660b9945d0514222bd7a83e28b96e8440c630ba6891f", "RepoTags": [ "ceph/daemon:latest-pacific" "RepoDigests": [ "ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586" a001s018: "Id": "sha256:6e73176320aaccf3b3fb660b9945d0514222bd7a83e28b96e8440c630ba6891f", "RepoTags": [ "ceph/daemon:latest-pacific" "RepoDigests": [ "ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586" -----Original Message----- From: Adiga, Anantha Sent: Tuesday, April 2, 2024 10:42 AM To: Eugen Block <eblock@nde.ag> Cc: ceph-users@ceph.io Subject: RE: [ceph-users] Re: ceph status not showing correct monitor services Hi Eugen, Currently there are only three nodes, but I can add a node to the cluster and check it out. I will take a look at the mon logs Thank you, Anantha -----Original Message----- From: Eugen Block <eblock@nde.ag> Sent: Tuesday, April 2, 2024 12:19 AM To: Adiga, Anantha <anantha.adiga@intel.com> Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Re: ceph status not showing correct monitor services You can add a mon manually to the monmap, but that requires a downtime of the mons. Here's an example [1] how to modify the monmap (including network change which you don't need, of course). But that would be my last resort, first I would try to find out why the MON fails to join the quorum. What is that mon.a001s016 logging, and what are the other two logging? Do you have another host where you could place a mon daemon to see if that works? [1] https://docs.ceph.com/en/latest/rados/operations/add-or-rm-mons/#example-pro... Zitat von "Adiga, Anantha" <anantha.adiga@intel.com>:

# ceph mon stat e6: 2 mons at {a001s017=[v2:10.45.128.27:3300/0,v1:10.45.128.27:6789/0],a001s018=[v2 :10.45.128.28:3300/0,v1:10.45.128.28:6789/0]}, election epoch 162, leader 0 a001s018, quorum 0,1 a001s018,a001s017

# ceph orch ps | grep mon mon.a001s016 a001s016 running (3h) 6m ago 3h 527M 2048M 16.2.5 6e73176320aa 39db8cfba7e1 mon.a001s017 a001s017 running (22h) 47s ago 1h 993M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 48s ago 2y 1167M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54

# ceph mgr stat { "epoch": 1130365, "available": true, "active_name": "a001s016.ctmoay", "num_standby": 1 }

# ceph orch ps | grep mgr mgr.a001s016.ctmoay a001s016 *:8443 running (18M) 109s ago 23M 518M - 16.2.5 6e73176320aa 169cafcbbb99 mgr.a001s017.bpygfm a001s017 *:8443 running (19M) 5m ago 23M 501M - 16.2.5 6e73176320aa 97257195158c mgr.a001s018.hcxnef a001s018 *:8443 running (20M) 5m ago 23M 113M - 16.2.5 6e73176320aa 21ba5896cee2

# ceph orch ls --service_name=mgr --export service_type: mgr service_name: mgr placement: count: 3 hosts: - a001s016 - a001s017 - a001s018

# ceph orch ls --service_name=mon --export service_type: mon service_name: mon placement: count: 3 hosts: - a001s016 - a001s017 - a001s018

-----Original Message----- From: Adiga, Anantha Sent: Monday, April 1, 2024 6:06 PM To: Eugen Block <eblock@nde.ag> Cc: ceph-users@ceph.io Subject: RE: [ceph-users] Re: ceph status not showing correct monitor services

# ceph tell mon.a001s016 mon_status Error ENOENT: problem getting command descriptions from mon.a001s016

a001s016 is outside quorum see below

# ceph tell mon.a001s017 mon_status { "name": "a001s017", "rank": 1, "state": "peon", "election_epoch": 162, "quorum": [ 0, 1 ], "quorum_age": 79938, "features": { "required_con": "2449958747317026820", "required_mon": [ "kraken", "luminous", "mimic", "osdmap-prune", "nautilus", "octopus", "pacific", "elector-pinging" ], "quorum_con": "4540138297136906239", "quorum_mon": [ "kraken", "luminous", "mimic", "osdmap-prune", "nautilus", "octopus", "pacific", "elector-pinging" ] }, "outside_quorum": [], "extra_probe_peers": [ { "addrvec": [ { "type": "v2", "addr": "10.45.128.26:3300", "nonce": 0 }, { "type": "v1", "addr": "10.45.128.26:6789", "nonce": 0 } ] } ], "sync_provider": [], "monmap": { "epoch": 6, "fsid": "604d56db-2fab-45db-a9ea-c418f9a8cca8", "modified": "2024-03-31T23:54:18.692983Z", "created": "2021-09-30T16:15:12.884602Z", "min_mon_release": 16, "min_mon_release_name": "pacific", "election_strategy": 1, "disallowed_leaders: ": "", "stretch_mode": false, "features": { "persistent": [ "kraken", "luminous", "mimic", "osdmap-prune", "nautilus", "octopus", "pacific", "elector-pinging" ], "optional": [] }, "mons": [ { "rank": 0, "name": "a001s018", "public_addrs": { "addrvec": [ { "type": "v2", "addr": "10.45.128.28:3300", "nonce": 0 }, { "type": "v1", "addr": "10.45.128.28:6789", "nonce": 0 } ] }, "addr": "10.45.128.28:6789/0", "public_addr": "10.45.128.28:6789/0", "priority": 0, "weight": 0, "crush_location": "{}" }, { "rank": 1, "name": "a001s017", "public_addrs": { "addrvec": [ { "type": "v2", "addr": "10.45.128.27:3300", "nonce": 0 }, { "type": "v1", "addr": "10.45.128.27:6789", "nonce": 0 } ] }, "addr": "10.45.128.27:6789/0", "public_addr": "10.45.128.27:6789/0", "priority": 0, "weight": 0, "crush_location": "{}" } ] }, "feature_map": { "mon": [ { "features": "0x3f01cfb9fffdffff", "release": "luminous", "num": 1 } ], "mds": [ { "features": "0x3f01cfb9fffdffff", "release": "luminous", "num": 3 } ], "osd": [ { "features": "0x3f01cfb9fffdffff", "release": "luminous", "num": 15 } ], "client": [ { "features": "0x2f018fb86aa42ada", "release": "luminous", "num": 50 }, { "features": "0x2f018fb87aa4aafe", "release": "luminous", "num": 40 }, { "features": "0x3f01cfb8ffedffff", "release": "luminous", "num": 1 }, { "features": "0x3f01cfb9fffdffff", "release": "luminous", "num": 72 } ] }, "stretch_mode": false }

# ceph mon dump dumped monmap epoch 6 epoch 6 fsid 604d56db-2fab-45db-a9ea-c418f9a8cca8 last_changed 2024-03-31T23:54:18.692983+0000 created 2021-09-30T16:15:12.884602+0000 min_mon_release 16 (pacific) election_strategy: 1 0: [v2:10.45.128.28:3300/0,v1:10.45.128.28:6789/0] mon.a001s018 1: [v2:10.45.128.27:3300/0,v1:10.45.128.27:6789/0] mon.a001s017 root@a001s016:/var/run/ceph/604d56db-2fab-45db-a9ea-c418f9a8cca8#

-----Original Message----- From: Adiga, Anantha Sent: Monday, April 1, 2024 3:20 PM To: Adiga, Anantha <anantha.adiga@intel.com>; Eugen Block <eblock@nde.ag> Cc: ceph-users@ceph.io Subject: RE: [ceph-users] Re: ceph status not showing correct monitor services

Both methods are not update the mon map, is there a way to inject mon.a001s016 into the current mon map?

# ceph mon dump dumped monmap epoch 6 epoch 6 fsid 604d56db-2fab-45db-a9ea-c418f9a8cca8 last_changed 2024-03-31T23:54:18.692983+0000 created 2021-09-30T16:15:12.884602+0000 min_mon_release 16 (pacific) election_strategy: 1 0: [v2:10.45.128.28:3300/0,v1:10.45.128.28:6789/0] mon.a001s018 1: [v2:10.45.128.27:3300/0,v1:10.45.128.27:6789/0] mon.a001s017

# ceph tell mon.a001s016 mon_status Error ENOENT: problem getting command descriptions from mon.a001s016

# ceph tell mon.a001s016 mon_status Error ENOENT: problem getting command descriptions from mon.a001s016

# ceph tell mon.a001s017 mon_status { "name": "a001s017", "rank": 1, "state": "peon", "election_epoch": 162, "quorum": [ 0, 1 ], "quorum_age": 69551, "features": { .............. ..............

# ceph orch ls --service_name=mon --export > mon3.yml service_type: mon service_name: mon placement: count: 3 hosts: - a001s016 - a001s017 - a001s018

# cp mon3.yml mon2.yml # vi mon2.yml #cat mon2.yml service_type: mon service_name: mon placement: count: 2 hosts: - a001s017 - a001s018

# ceph orch apply -i mon2.yml --dry-run WARNING! Dry-Runs are snapshots of a certain point in time and are bound to the current inventory setup. If any on these conditions changes, the preview will be invalid. Please make sure to have a minimal timeframe between planning and applying the specs. #################### SERVICESPEC PREVIEWS #################### +---------+------+--------+--------------+ |SERVICE |NAME |ADD_TO |REMOVE_FROM | +---------+------+--------+--------------+ |mon |mon | |mon.a001s016 | +---------+------+--------+--------------+ ################ OSDSPEC PREVIEWS ################ +---------+------+------+------+----+-----+ |SERVICE |NAME |HOST |DATA |DB |WAL | +---------+------+------+------+----+-----+ +---------+------+------+------+----+-----+

# ceph orch ls --service_name=mon --refresh NAME PORTS RUNNING REFRESHED AGE PLACEMENT mon 3/3 5m ago 18h a001s016;a001s017;a001s018;count:3

# ceph orch ps --refresh | grep mon mon.a001s016 a001s016 running (21h) 2s ago 21h 734M 2048M 16.2.5 6e73176320aa 8484a912f96a mon.a001s017 a001s017 running (18h) 2s ago 21h 976M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 2s ago 2y 1164M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s016 a001s016 running (21h) 37s ago 21h 734M 2048M 16.2.5 6e73176320aa 8484a912f96a mon.a001s017 a001s017 running (18h) 37s ago 21h 977M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 38s ago 2y 1166M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54

# ceph orch apply -i mon2.yml Scheduled mon update... # ceph orch ps --refresh | grep mon mon.a001s016 a001s016 running (21h) 21s ago 21h 734M 2048M 16.2.5 6e73176320aa 8484a912f96a mon.a001s017 a001s017 running (18h) 20s ago 21h 962M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 21s ago 2y 1156M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s017 a001s017 running (18h) 23s ago 21h 962M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 24s ago 2y 1156M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s017 a001s017 running (18h) 27s ago 21h 962M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 0s ago 2y 1154M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s017 a001s017 running (18h) 2s ago 21h 960M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 3s ago 2y 1154M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s017 a001s017 running (18h) 5s ago 21h 960M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 0s ago 2y 1154M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s017 a001s017 running (18h) 2s ago 21h 962M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 2s ago 2y 1154M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54

# ls /var/lib/ceph/604d56db-2fab-45db-a9ea-c418f9a8cca8/ cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931 keepalived.nfs.nfs.a001s016.ofqyjl osd.1 osd.22 osd.4 crash mds.cephfs.a001s016.vtyivn osd.10 osd.24 osd.7 crash.a001s016 mgr.a001s016.ctmoay osd.13 osd.27 removed haproxy.nfs.nfs.a001s016.vpjtxt nfs.nfs.0.0.a001s016.wumnjs osd.15 osd.30 rgw.ceph.a001s016.gjyanl home node-exporter.a001s016 osd.18 osd.33

# ceph mon dump dumped monmap epoch 6 epoch 6 fsid 604d56db-2fab-45db-a9ea-c418f9a8cca8 last_changed 2024-03-31T23:54:18.692983+0000 created 2021-09-30T16:15:12.884602+0000 min_mon_release 16 (pacific) election_strategy: 1 0: [v2:10.45.128.28:3300/0,v1:10.45.128.28:6789/0] mon.a001s018 1: [v2:10.45.128.27:3300/0,v1:10.45.128.27:6789/0] mon.a001s017

# cat mon3.yml service_type: mon service_name: mon placement: count: 3 hosts: - a001s016 - a001s017 - a001s018

# ceph orch apply -i mon3.yml --dry-run WARNING! Dry-Runs are snapshots of a certain point in time and are bound to the current inventory setup. If any on these conditions changes, the preview will be invalid. Please make sure to have a minimal timeframe between planning and applying the specs. #################### SERVICESPEC PREVIEWS #################### +---------+------+----------+-------------+ |SERVICE |NAME |ADD_TO |REMOVE_FROM | +---------+------+----------+-------------+ |mon |mon |a001s016 | | +---------+------+----------+-------------+ ################ OSDSPEC PREVIEWS ################ +---------+------+------+------+----+-----+ |SERVICE |NAME |HOST |DATA |DB |WAL | +---------+------+------+------+----+-----+ +---------+------+------+------+----+-----+

# ceph orch apply -i mon3.yml Scheduled mon update... # ceph orch ps --refresh | grep mon mon.a001s016 a001s016 starting - - - 2048M <unknown> <unknown> <unknown> mon.a001s017 a001s017 running (18h) 2s ago 21h 960M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 3s ago 2y 1152M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s016 a001s016 running (6s) 0s ago 6s 14.4M 2048M 16.2.5 6e73176320aa dcd4705c2069 mon.a001s017 a001s017 running (18h) 8s ago 21h 960M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 9s ago 2y 1152M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s016 a001s016 running (10s) 4s ago 10s 14.4M 2048M 16.2.5 6e73176320aa dcd4705c2069 mon.a001s017 a001s017 running (18h) 12s ago 21h 960M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 13s ago 2y 1152M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54

# ls /var/lib/ceph/604d56db-2fab-45db-a9ea-c418f9a8cca8/ cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931 mgr.a001s016.ctmoay osd.15 osd.4 crash mon.a001s016 osd.18 osd.7 crash.a001s016 nfs.nfs.0.0.a001s016.wumnjs osd.22 removed haproxy.nfs.nfs.a001s016.vpjtxt node-exporter.a001s016 osd.24 rgw.ceph.a001s016.gjyanl home osd.1 osd.27 keepalived.nfs.nfs.a001s016.ofqyjl osd.10 osd.30 mds.cephfs.a001s016.vtyivn osd.13 osd.33

# ceph mon dump dumped monmap epoch 6 epoch 6 fsid 604d56db-2fab-45db-a9ea-c418f9a8cca8 last_changed 2024-03-31T23:54:18.692983+0000 created 2021-09-30T16:15:12.884602+0000 min_mon_release 16 (pacific) election_strategy: 1 0: [v2:10.45.128.28:3300/0,v1:10.45.128.28:6789/0] mon.a001s018 1: [v2:10.45.128.27:3300/0,v1:10.45.128.27:6789/0] mon.a001s017

# ceph -s cluster: id: 604d56db-2fab-45db-a9ea-c418f9a8cca8 health: HEALTH_OK

services: mon: 2 daemons, quorum a001s018,a001s017 (age 18h) mgr: a001s016.ctmoay(active, since 109m), standbys: a001s017.bpygfm mds: 1/1 daemons up, 2 standby osd: 36 osds: 36 up (since 2h), 36 in (since 2y) rgw: 3 daemons active (3 hosts, 1 zones)

data: volumes: 1/1 healthy pools: 43 pools, 1633 pgs objects: 51.64M objects, 77 TiB usage: 119 TiB used, 132 TiB / 252 TiB avail pgs: 1633 active+clean

io: client: 356 MiB/s rd, 633 MiB/s wr, 506 op/s rd, 755 op/s wr

# ceph orch ps --refresh | grep mon mon.a001s016 a001s016 running (91s) 7s ago 91s 14.5M 2048M 16.2.5 6e73176320aa dcd4705c2069 mon.a001s017 a001s017 running (18h) 7s ago 21h 978M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 8s ago 2y 1163M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ls | grep mon mon 3/3 9s ago 114s a001s016;a001s017;a001s018;count:3

# ceph log last cephadm 2024-04-01T18:56:29.631337+0000 mgr.a001s017.bpygfm (mgr.60881489) 18014760 : cephadm [INF] Schedule restart daemon osd.6 2024-04-01T18:57:05.983577+0000 mgr.a001s017.bpygfm (mgr.60881489) 18014780 : cephadm [INF] Schedule restart daemon osd.9 2024-04-01T18:57:17.919236+0000 mgr.a001s017.bpygfm (mgr.60881489) 18014788 : cephadm [INF] Schedule restart daemon osd.12 2024-04-01T18:57:59.823635+0000 mgr.a001s017.bpygfm (mgr.60881489) 18014811 : cephadm [INF] Schedule restart daemon osd.16 2024-04-01T18:58:26.039310+0000 mgr.a001s017.bpygfm (mgr.60881489) 18014826 : cephadm [INF] Schedule restart daemon osd.19 2024-04-01T18:58:57.278955+0000 mgr.a001s017.bpygfm (mgr.60881489) 18014844 : cephadm [INF] Schedule restart daemon osd.21 2024-04-01T18:59:36.656834+0000 mgr.a001s017.bpygfm (mgr.60881489) 18014867 : cephadm [INF] Schedule restart daemon osd.25 2024-04-01T19:00:03.272925+0000 mgr.a001s017.bpygfm (mgr.60881489) 18014882 : cephadm [INF] Schedule restart daemon osd.28 2024-04-01T19:00:28.348138+0000 mgr.a001s017.bpygfm (mgr.60881489) 18014897 : cephadm [INF] Schedule restart daemon osd.31 2024-04-01T19:00:56.297517+0000 mgr.a001s017.bpygfm (mgr.60881489) 18014913 : cephadm [INF] Schedule restart daemon osd.34 2024-04-01T19:38:21.552347+0000 mgr.a001s017.bpygfm (mgr.60881489) 18016056 : cephadm [INF] Schedule restart daemon mgr.a001s017.bpygfm 2024-04-01T19:38:21.603564+0000 mgr.a001s017.bpygfm (mgr.60881489) 18016057 : cephadm [INF] Failing over to other MGR 2024-04-01T21:21:24.414308+0000 mgr.a001s016.ctmoay (mgr.61827136) 3096 : cephadm [INF] Saving service mon spec with placement a001s017;a001s018;count:2 2024-04-01T21:25:02.815151+0000 mgr.a001s016.ctmoay (mgr.61827136) 3211 : cephadm [INF] Saving service mon spec with placement a001s017;a001s018;count:2 2024-04-01T21:25:02.844721+0000 mgr.a001s016.ctmoay (mgr.61827136) 3212 : cephadm [INF] Safe to remove mon.a001s016: not in monmap (['a001s018', 'a001s017']) 2024-04-01T21:25:02.845059+0000 mgr.a001s016.ctmoay (mgr.61827136) 3213 : cephadm [INF] Removing monitor a001s016 from monmap... 2024-04-01T21:25:02.846401+0000 mgr.a001s016.ctmoay (mgr.61827136) 3214 : cephadm [INF] Removing daemon mon.a001s016 from a001s016 2024-04-01T21:25:06.870895+0000 mgr.a001s016.ctmoay (mgr.61827136) 3218 : cephadm [INF] Removing key for mon. 2024-04-01T21:26:27.861600+0000 mgr.a001s016.ctmoay (mgr.61827136) 3266 : cephadm [INF] Saving service mon spec with placement a001s016;a001s017;a001s018;count:3 2024-04-01T21:26:38.247516+0000 mgr.a001s016.ctmoay (mgr.61827136) 3273 : cephadm [INF] Saving service mon spec with placement a001s016;a001s017;a001s018;count:3 2024-04-01T21:26:44.494080+0000 mgr.a001s016.ctmoay (mgr.61827136) 3277 : cephadm [INF] Deploying daemon mon.a001s016 on a001s016

# cat /var/lib/ceph/604d56db-2fab-45db-a9ea-c418f9a8cca8/mon.a001s016/unit.i mage docker.io/ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a 2cb7ff7668f776b61b9d586 # ceph orch ps --daemon_type=mon NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mon.a001s016 a001s016 running (16m) 3m ago 16m 16.3M 2048M 16.2.5 6e73176320aa dcd4705c2069 mon.a001s017 a001s017 running (18h) 4m ago 21h 960M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 4m ago 2y 1155M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54

# ceph orch daemon redeploy mon.a001s016 docker.io/ceph/daemon:latest-pacific Scheduled to redeploy mon.a001s016 on host 'a001s016'

# ceph orch ps --daemon_type=mon --refresh NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mon.a001s016 a001s016 running (13s) 7s ago 18m 14.1M 2048M 16.2.5 6e73176320aa 1917b1edf4ab mon.a001s017 a001s017 running (18h) 6m ago 21h 960M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 6m ago 2y 1155M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54

====================================================================== =======================

The second method also does not update ceph mon map:

#ceph mon dump dumped monmap epoch 6 epoch 6 fsid 604d56db-2fab-45db-a9ea-c418f9a8cca8 last_changed 2024-03-31T23:54:18.692983+0000 created 2021-09-30T16:15:12.884602+0000 min_mon_release 16 (pacific) election_strategy: 1 0: [v2:10.45.128.28:3300/0,v1:10.45.128.28:6789/0] mon.a001s018 1: [v2:10.45.128.27:3300/0,v1:10.45.128.27:6789/0] mon.a001s017

# ceph orch ps --refresh | grep mon mon.a001s016 a001s016 running (81s) 63s ago 81s 14.2M 2048M 16.2.5 6e73176320aa 0c0ec88139e1 mon.a001s017 a001s017 running (19h) 62s ago 22h 965M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 63s ago 2y 1157M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54

# ceph orch daemon stop mon.a001s016 Scheduled to stop mon.a001s016 on host 'a001s016' # ceph orch ps --refresh | grep mon mon.a001s016 a001s016 running (96s) 6s ago 96s 14.4M 2048M 16.2.5 6e73176320aa 0c0ec88139e1 mon.a001s017 a001s017 running (19h) 5s ago 22h 968M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 6s ago 2y 1157M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s016 a001s016 running (98s) 7s ago 98s 14.4M 2048M 16.2.5 6e73176320aa 0c0ec88139e1 mon.a001s017 a001s017 running (19h) 7s ago 22h 968M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 8s ago 2y 1157M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s016 a001s016 running (100s) 1s ago 100s 14.4M 2048M 16.2.5 6e73176320aa 0c0ec88139e1 mon.a001s017 a001s017 running (19h) 9s ago 22h 968M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 10s ago 2y 1157M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s016 a001s016 running (106s) 7s ago 106s 14.4M 2048M 16.2.5 6e73176320aa 0c0ec88139e1 mon.a001s017 a001s017 running (19h) 15s ago 22h 968M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 16s ago 2y 1157M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s016 a001s016 stopped 0s ago 108s - 2048M <unknown> <unknown> <unknown> mon.a001s017 a001s017 running (19h) 17s ago 22h 968M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 0s ago 2y 1160M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s016 a001s016 stopped 2s ago 111s - 2048M <unknown> <unknown> <unknown> mon.a001s017 a001s017 running (19h) 2s ago 22h 972M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 3s ago 2y 1160M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54

# ceph orch daemon rm mon.a001s016 Error EINVAL: must pass --force to REMOVE daemon with potentially PRECIOUS DATA for mon.a001s016

# ceph orch daemon rm mon.a001s016 --force Removed mon.a001s016 from host 'a001s016'

# ceph orch ps --refresh | grep mon mon.a001s017 a001s017 running (19h) 30s ago 22h 976M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 0s ago 2y 1166M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 #ceph orch ps --refresh | grep mon mon.a001s017 a001s017 running (19h) 2s ago 22h 982M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 3s ago 2y 1166M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s016 a001s016 starting - - - 2048M <unknown> <unknown> <unknown> mon.a001s017 a001s017 running (19h) 6s ago 22h 982M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 7s ago 2y 1166M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s016 a001s016 running (8s) 2s ago 8s 14.4M 2048M 16.2.5 6e73176320aa 39db8cfba7e1 mon.a001s017 a001s017 running (19h) 1s ago 22h 987M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 2s ago 2y 1171M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54

# ceph mon dump dumped monmap epoch 6 epoch 6 fsid 604d56db-2fab-45db-a9ea-c418f9a8cca8 last_changed 2024-03-31T23:54:18.692983+0000 created 2021-09-30T16:15:12.884602+0000 min_mon_release 16 (pacific) election_strategy: 1 0: [v2:10.45.128.28:3300/0,v1:10.45.128.28:6789/0] mon.a001s018 1: [v2:10.45.128.27:3300/0,v1:10.45.128.27:6789/0] mon.a001s017

#

-----Original Message----- From: Adiga, Anantha <anantha.adiga@intel.com> Sent: Monday, April 1, 2024 2:01 PM To: Eugen Block <eblock@nde.ag> Cc: ceph-users@ceph.io Subject: [ceph-users] Re: ceph status not showing correct monitor services

Thank you. I will try the export and import method first.

Thank you, Anantha

-----Original Message----- From: Eugen Block <eblock@nde.ag> Sent: Monday, April 1, 2024 1:57 PM To: Adiga, Anantha <anantha.adiga@intel.com> Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Re: ceph status not showing correct monitor services

I have two approaches in mind, first one (and preferred) would be to edit the mon spec to first remove mon.a001s016 and have a clean state. Get the current spec with:

ceph orch ls mon --export > mon-edit.yaml

Edit the spec file so that mon.a001s016 is not part of it, then apply:

ceph orch apply -i mon-edit.yaml

This should remove the mon.a001s016 daemon. Then wait a few minutes or so (until the daemon is actually gone, check locally on the node with 'cephadm ls' and in /var/lib/ceph/<FSID>/removed) and add it back to the spec file, then apply again. I would expect a third MON to be deployed. If that doesn't work for some reason you'll need to inspect logs to find the root cause.

The second approach would be to remove and add the daemon manually:

ceph orch daemon rm mon.a001s016

Wait until it's really gone, then add it:

ceph orch daemon add mon a001s016

Not entirely sure about the daemon add mon command, you might need to provide something else, I'm typing this by heart.

Zitat von "Adiga, Anantha" <anantha.adiga@intel.com>:

...

Hi Eugen,

Yes that is it. OSDs were restarted since mon a001s017 was reporting is low on available space. How to update the mon map to add mon.a001s016 as it is already online? And how to update mgr map to include standby mgr.a001s018 as it is also running.

ceph mon dump dumped monmap epoch 6 epoch 6 fsid 604d56db-2fab-45db-a9ea-c418f9a8cca8 last_changed 2024-03-31T23:54:18.692983+0000 created 2021-09-30T16:15:12.884602+0000 min_mon_release 16 (pacific) election_strategy: 1 0: [v2:10.45.128.28:3300/0,v1:10.45.128.28:6789/0] mon.a001s018 1: [v2:10.45.128.27:3300/0,v1:10.45.128.27:6789/0] mon.a001s017

Thank you, Anantha

-----Original Message----- From: Eugen Block <eblock@nde.ag> Sent: Monday, April 1, 2024 1:10 PM To: ceph-users@ceph.io Subject: [ceph-users] Re: ceph status not showing correct monitor services

Maybe it’s just not in the monmap? Can you show the output of:

ceph mon dump

Did you do any maintenance (apparently OSDs restarted recently) and maybe accidentally removed a MON from the monmap?

Zitat von "Adiga, Anantha" <anantha.adiga@intel.com>:

...

Hi Anthony,

Seeing it since last after noon. It is same with mgr services as , "ceph -s" is reporting only TWO instead of THREE

Also mon and mgr shows " is_active: false" see below.

# ceph orch ps --daemon_type=mgr NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mgr.a001s016.ctmoay a001s016 *:8443 running (18M) 3m ago 23M 206M - 16.2.5 6e73176320aa 169cafcbbb99 mgr.a001s017.bpygfm a001s017 *:8443 running (19M) 3m ago 23M 332M - 16.2.5 6e73176320aa 97257195158c mgr.a001s018.hcxnef a001s018 *:8443 running (20M) 3m ago 23M 113M - 16.2.5 6e73176320aa 21ba5896cee2

# ceph orch ls --service_name=mgr NAME PORTS RUNNING REFRESHED AGE PLACEMENT mgr 3/3 3m ago 23M a001s016;a001s017;a001s018;count:3

# ceph orch ps --daemon_type=mon --format=json-pretty

[ { "container_id": "8484a912f96a", "container_image_digests": [

docker.io/ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586<mailto:docker.io/ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586> ], "container_image_id": "6e73176320aaccf3b3fb660b9945d0514222bd7a83e28b96e8440c630ba6891f", "container_image_name": docker.io/ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586<mailto:docker.io/ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586>, "created": "2024-03-31T23:55:16.164155Z", "daemon_id": "a001s016", "daemon_type": "mon", "hostname": "a001s016", "is_active": false, <== why is it false "last_refresh": "2024-04-01T19:38:30.929014Z", "memory_request": 2147483648, "memory_usage": 761685606, "ports": [], "service_name": "mon", "started": "2024-03-31T23:55:16.268266Z", "status": 1, "status_desc": "running", "version": "16.2.5" },

Thank you, Anantha

From: Anthony D'Atri <aad@dreamsnake.net> Sent: Monday, April 1, 2024 12:25 PM To: Adiga, Anantha <anantha.adiga@intel.com> Cc: ceph-users@ceph.io Subject: Re: [ceph-users] ceph status not showing correct monitor services

a001s017.bpygfm(active, since 13M), standbys: a001s016.ctmoay

Looks like you just had an mgr failover? Could be that the secondary mgr hasn't caught up with current events. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

removed the config setting for mon. a001s016. Here it is # ceph config get mon container_image docker.io/ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586 # ceph config get osd container_image docker.io/ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586 # ceph config get mgr mgr/cephadm/container_image_base docker.io/ceph/daemon Thank you, Anantha -----Original Message----- From: Eugen Block <eblock@nde.ag> Sent: Wednesday, April 3, 2024 12:27 AM To: Adiga, Anantha <anantha.adiga@intel.com> Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Re: ceph status not showing correct monitor services I have no idea what you did there ;-) I would remove that config though and rather configure the ceph image globally, there have been several issues when cephadm tries to launch daemons with different ceph versions. Although in your case it looks like they are actually the same images according to the digest (and also in the ceph orch ps output). But it might cause some trouble anyway, so I'd recommend to remove the individual config for mon.a001s016 and only use the global config. Can you add these outputs (mask sensitive data)? ceph config get mon container_image ceph config get osd container_image ceph config get mgr mgr/cephadm/container_image_base Zitat von "Adiga, Anantha" <anantha.adiga@intel.com>:

Hi Eugen,

Noticed this in the config dump: Why only "mon.a001s016 " listed? And this is the one that is not listed in "ceph -s"

mon advanced auth_allow_insecure_global_id_reclaim false mon advanced auth_expose_insecure_global_id_reclaim false mon advanced mon_compact_on_start true mon.a001s016 basic container_image

docker.io/ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586 * mgr advanced mgr/cephadm/container_image_base docker.io/ceph/daemon mgr advanced mgr/cephadm/container_image_node_exporter docker.io/prom/node-exporter:v0.17.0

cluster: id: 604d56db-2fab-45db-a9ea-c418f9a8cca8 health: HEALTH_OK

services: mon: 2 daemons, quorum a001s018,a001s017 (age 45h) mgr: a001s016.ctmoay(active, since 28h), standbys: a001s017.bpygfm mds: 1/1 daemons up, 2 standby osd: 36 osds: 36 up (since 29h), 36 in (since 2y) rgw: 3 daemons active (3 hosts, 1 zones)

var lib mon unit.image ====================

a001s016: # cat /var/lib/ceph/604d56db-2fab-45db-a9ea-c418f9a8cca8/mon.a001s016/unit.i mage docker.io/ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a 2cb7ff7668f776b61b9d586

a001s017: # cat /var/lib/ceph/604d56db-2fab-45db-a9ea-c418f9a8cca8/mon.a001s017/unit.i mage docker.io/ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a 2cb7ff7668f776b61b9d586 a001s018: # cat /var/lib/ceph/604d56db-2fab-45db-a9ea-c418f9a8cca8/mon.a001s018/unit.i mage docker.io/ceph/daemon:latest-pacific

ceph image tag, digest from docker inspect of: ceph/daemon latest-pacific 6e73176320aa 2 years ago 1.27GB ====================================================================== ============================ a001s016: "Id": "sha256:6e73176320aaccf3b3fb660b9945d0514222bd7a83e28b96e8440c630ba6891f", "RepoTags": [ "ceph/daemon:latest-pacific" "RepoDigests": [

"ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586"

a001s017: "Id": "sha256:6e73176320aaccf3b3fb660b9945d0514222bd7a83e28b96e8440c630ba6891f", "RepoTags": [ "ceph/daemon:latest-pacific" "RepoDigests": [

"ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586"

a001s018: "Id": "sha256:6e73176320aaccf3b3fb660b9945d0514222bd7a83e28b96e8440c630ba6891f", "RepoTags": [ "ceph/daemon:latest-pacific" "RepoDigests": [

"ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586"

-----Original Message----- From: Adiga, Anantha Sent: Tuesday, April 2, 2024 10:42 AM To: Eugen Block <eblock@nde.ag> Cc: ceph-users@ceph.io Subject: RE: [ceph-users] Re: ceph status not showing correct monitor services

Hi Eugen,

Currently there are only three nodes, but I can add a node to the cluster and check it out. I will take a look at the mon logs

Thank you, Anantha

-----Original Message----- From: Eugen Block <eblock@nde.ag> Sent: Tuesday, April 2, 2024 12:19 AM To: Adiga, Anantha <anantha.adiga@intel.com> Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Re: ceph status not showing correct monitor services

You can add a mon manually to the monmap, but that requires a downtime of the mons. Here's an example [1] how to modify the monmap (including network change which you don't need, of course). But that would be my last resort, first I would try to find out why the MON fails to join the quorum. What is that mon.a001s016 logging, and what are the other two logging? Do you have another host where you could place a mon daemon to see if that works?

[1] https://docs.ceph.com/en/latest/rados/operations/add-or-rm-mons/#examp le-procedure

Zitat von "Adiga, Anantha" <anantha.adiga@intel.com>:

...

# ceph mon stat e6: 2 mons at {a001s017=[v2:10.45.128.27:3300/0,v1:10.45.128.27:6789/0],a001s018=[v 2 :10.45.128.28:3300/0,v1:10.45.128.28:6789/0]}, election epoch 162, leader 0 a001s018, quorum 0,1 a001s018,a001s017

# ceph orch ps | grep mon mon.a001s016 a001s016 running (3h) 6m ago 3h 527M 2048M 16.2.5 6e73176320aa 39db8cfba7e1 mon.a001s017 a001s017 running (22h) 47s ago 1h 993M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 48s ago 2y 1167M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54

# ceph mgr stat { "epoch": 1130365, "available": true, "active_name": "a001s016.ctmoay", "num_standby": 1 }

# ceph orch ps | grep mgr mgr.a001s016.ctmoay a001s016 *:8443 running (18M) 109s ago 23M 518M - 16.2.5 6e73176320aa 169cafcbbb99 mgr.a001s017.bpygfm a001s017 *:8443 running (19M) 5m ago 23M 501M - 16.2.5 6e73176320aa 97257195158c mgr.a001s018.hcxnef a001s018 *:8443 running (20M) 5m ago 23M 113M - 16.2.5 6e73176320aa 21ba5896cee2

# ceph orch ls --service_name=mgr --export service_type: mgr service_name: mgr placement: count: 3 hosts: - a001s016 - a001s017 - a001s018

# ceph orch ls --service_name=mon --export service_type: mon service_name: mon placement: count: 3 hosts: - a001s016 - a001s017 - a001s018

-----Original Message----- From: Adiga, Anantha Sent: Monday, April 1, 2024 6:06 PM To: Eugen Block <eblock@nde.ag> Cc: ceph-users@ceph.io Subject: RE: [ceph-users] Re: ceph status not showing correct monitor services

# ceph tell mon.a001s016 mon_status Error ENOENT: problem getting command descriptions from mon.a001s016

a001s016 is outside quorum see below

# ceph tell mon.a001s017 mon_status { "name": "a001s017", "rank": 1, "state": "peon", "election_epoch": 162, "quorum": [ 0, 1 ], "quorum_age": 79938, "features": { "required_con": "2449958747317026820", "required_mon": [ "kraken", "luminous", "mimic", "osdmap-prune", "nautilus", "octopus", "pacific", "elector-pinging" ], "quorum_con": "4540138297136906239", "quorum_mon": [ "kraken", "luminous", "mimic", "osdmap-prune", "nautilus", "octopus", "pacific", "elector-pinging" ] }, "outside_quorum": [], "extra_probe_peers": [ { "addrvec": [ { "type": "v2", "addr": "10.45.128.26:3300", "nonce": 0 }, { "type": "v1", "addr": "10.45.128.26:6789", "nonce": 0 } ] } ], "sync_provider": [], "monmap": { "epoch": 6, "fsid": "604d56db-2fab-45db-a9ea-c418f9a8cca8", "modified": "2024-03-31T23:54:18.692983Z", "created": "2021-09-30T16:15:12.884602Z", "min_mon_release": 16, "min_mon_release_name": "pacific", "election_strategy": 1, "disallowed_leaders: ": "", "stretch_mode": false, "features": { "persistent": [ "kraken", "luminous", "mimic", "osdmap-prune", "nautilus", "octopus", "pacific", "elector-pinging" ], "optional": [] }, "mons": [ { "rank": 0, "name": "a001s018", "public_addrs": { "addrvec": [ { "type": "v2", "addr": "10.45.128.28:3300", "nonce": 0 }, { "type": "v1", "addr": "10.45.128.28:6789", "nonce": 0 } ] }, "addr": "10.45.128.28:6789/0", "public_addr": "10.45.128.28:6789/0", "priority": 0, "weight": 0, "crush_location": "{}" }, { "rank": 1, "name": "a001s017", "public_addrs": { "addrvec": [ { "type": "v2", "addr": "10.45.128.27:3300", "nonce": 0 }, { "type": "v1", "addr": "10.45.128.27:6789", "nonce": 0 } ] }, "addr": "10.45.128.27:6789/0", "public_addr": "10.45.128.27:6789/0", "priority": 0, "weight": 0, "crush_location": "{}" } ] }, "feature_map": { "mon": [ { "features": "0x3f01cfb9fffdffff", "release": "luminous", "num": 1 } ], "mds": [ { "features": "0x3f01cfb9fffdffff", "release": "luminous", "num": 3 } ], "osd": [ { "features": "0x3f01cfb9fffdffff", "release": "luminous", "num": 15 } ], "client": [ { "features": "0x2f018fb86aa42ada", "release": "luminous", "num": 50 }, { "features": "0x2f018fb87aa4aafe", "release": "luminous", "num": 40 }, { "features": "0x3f01cfb8ffedffff", "release": "luminous", "num": 1 }, { "features": "0x3f01cfb9fffdffff", "release": "luminous", "num": 72 } ] }, "stretch_mode": false }

# ceph mon dump dumped monmap epoch 6 epoch 6 fsid 604d56db-2fab-45db-a9ea-c418f9a8cca8 last_changed 2024-03-31T23:54:18.692983+0000 created 2021-09-30T16:15:12.884602+0000 min_mon_release 16 (pacific) election_strategy: 1 0: [v2:10.45.128.28:3300/0,v1:10.45.128.28:6789/0] mon.a001s018 1: [v2:10.45.128.27:3300/0,v1:10.45.128.27:6789/0] mon.a001s017 root@a001s016:/var/run/ceph/604d56db-2fab-45db-a9ea-c418f9a8cca8#

-----Original Message----- From: Adiga, Anantha Sent: Monday, April 1, 2024 3:20 PM To: Adiga, Anantha <anantha.adiga@intel.com>; Eugen Block <eblock@nde.ag> Cc: ceph-users@ceph.io Subject: RE: [ceph-users] Re: ceph status not showing correct monitor services

Both methods are not update the mon map, is there a way to inject mon.a001s016 into the current mon map?

# ceph mon dump dumped monmap epoch 6 epoch 6 fsid 604d56db-2fab-45db-a9ea-c418f9a8cca8 last_changed 2024-03-31T23:54:18.692983+0000 created 2021-09-30T16:15:12.884602+0000 min_mon_release 16 (pacific) election_strategy: 1 0: [v2:10.45.128.28:3300/0,v1:10.45.128.28:6789/0] mon.a001s018 1: [v2:10.45.128.27:3300/0,v1:10.45.128.27:6789/0] mon.a001s017

# ceph tell mon.a001s016 mon_status Error ENOENT: problem getting command descriptions from mon.a001s016

# ceph tell mon.a001s016 mon_status Error ENOENT: problem getting command descriptions from mon.a001s016

# ceph tell mon.a001s017 mon_status { "name": "a001s017", "rank": 1, "state": "peon", "election_epoch": 162, "quorum": [ 0, 1 ], "quorum_age": 69551, "features": { .............. ..............

# ceph orch ls --service_name=mon --export > mon3.yml service_type: mon service_name: mon placement: count: 3 hosts: - a001s016 - a001s017 - a001s018

# cp mon3.yml mon2.yml # vi mon2.yml #cat mon2.yml service_type: mon service_name: mon placement: count: 2 hosts: - a001s017 - a001s018

# ceph orch apply -i mon2.yml --dry-run WARNING! Dry-Runs are snapshots of a certain point in time and are bound to the current inventory setup. If any on these conditions changes, the preview will be invalid. Please make sure to have a minimal timeframe between planning and applying the specs. #################### SERVICESPEC PREVIEWS #################### +---------+------+--------+--------------+ |SERVICE |NAME |ADD_TO |REMOVE_FROM | +---------+------+--------+--------------+ |mon |mon | |mon.a001s016 | +---------+------+--------+--------------+ ################ OSDSPEC PREVIEWS ################ +---------+------+------+------+----+-----+ |SERVICE |NAME |HOST |DATA |DB |WAL | +---------+------+------+------+----+-----+ +---------+------+------+------+----+-----+

# ceph orch ls --service_name=mon --refresh NAME PORTS RUNNING REFRESHED AGE PLACEMENT mon 3/3 5m ago 18h a001s016;a001s017;a001s018;count:3

# ceph orch ps --refresh | grep mon mon.a001s016 a001s016 running (21h) 2s ago 21h 734M 2048M 16.2.5 6e73176320aa 8484a912f96a mon.a001s017 a001s017 running (18h) 2s ago 21h 976M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 2s ago 2y 1164M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s016 a001s016 running (21h) 37s ago 21h 734M 2048M 16.2.5 6e73176320aa 8484a912f96a mon.a001s017 a001s017 running (18h) 37s ago 21h 977M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 38s ago 2y 1166M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54

# ceph orch apply -i mon2.yml Scheduled mon update... # ceph orch ps --refresh | grep mon mon.a001s016 a001s016 running (21h) 21s ago 21h 734M 2048M 16.2.5 6e73176320aa 8484a912f96a mon.a001s017 a001s017 running (18h) 20s ago 21h 962M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 21s ago 2y 1156M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s017 a001s017 running (18h) 23s ago 21h 962M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 24s ago 2y 1156M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s017 a001s017 running (18h) 27s ago 21h 962M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 0s ago 2y 1154M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s017 a001s017 running (18h) 2s ago 21h 960M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 3s ago 2y 1154M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s017 a001s017 running (18h) 5s ago 21h 960M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 0s ago 2y 1154M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s017 a001s017 running (18h) 2s ago 21h 962M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 2s ago 2y 1154M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54

# ls /var/lib/ceph/604d56db-2fab-45db-a9ea-c418f9a8cca8/ cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931 keepalived.nfs.nfs.a001s016.ofqyjl osd.1 osd.22 osd.4 crash mds.cephfs.a001s016.vtyivn osd.10 osd.24 osd.7 crash.a001s016 mgr.a001s016.ctmoay osd.13 osd.27 removed haproxy.nfs.nfs.a001s016.vpjtxt nfs.nfs.0.0.a001s016.wumnjs osd.15 osd.30 rgw.ceph.a001s016.gjyanl home node-exporter.a001s016 osd.18 osd.33

# ceph mon dump dumped monmap epoch 6 epoch 6 fsid 604d56db-2fab-45db-a9ea-c418f9a8cca8 last_changed 2024-03-31T23:54:18.692983+0000 created 2021-09-30T16:15:12.884602+0000 min_mon_release 16 (pacific) election_strategy: 1 0: [v2:10.45.128.28:3300/0,v1:10.45.128.28:6789/0] mon.a001s018 1: [v2:10.45.128.27:3300/0,v1:10.45.128.27:6789/0] mon.a001s017

# cat mon3.yml service_type: mon service_name: mon placement: count: 3 hosts: - a001s016 - a001s017 - a001s018

# ceph orch apply -i mon3.yml --dry-run WARNING! Dry-Runs are snapshots of a certain point in time and are bound to the current inventory setup. If any on these conditions changes, the preview will be invalid. Please make sure to have a minimal timeframe between planning and applying the specs. #################### SERVICESPEC PREVIEWS #################### +---------+------+----------+-------------+ |SERVICE |NAME |ADD_TO |REMOVE_FROM | +---------+------+----------+-------------+ |mon |mon |a001s016 | | +---------+------+----------+-------------+ ################ OSDSPEC PREVIEWS ################ +---------+------+------+------+----+-----+ |SERVICE |NAME |HOST |DATA |DB |WAL | +---------+------+------+------+----+-----+ +---------+------+------+------+----+-----+

# ceph orch apply -i mon3.yml Scheduled mon update... # ceph orch ps --refresh | grep mon mon.a001s016 a001s016 starting - - - 2048M <unknown> <unknown> <unknown> mon.a001s017 a001s017 running (18h) 2s ago 21h 960M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 3s ago 2y 1152M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s016 a001s016 running (6s) 0s ago 6s 14.4M 2048M 16.2.5 6e73176320aa dcd4705c2069 mon.a001s017 a001s017 running (18h) 8s ago 21h 960M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 9s ago 2y 1152M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s016 a001s016 running (10s) 4s ago 10s 14.4M 2048M 16.2.5 6e73176320aa dcd4705c2069 mon.a001s017 a001s017 running (18h) 12s ago 21h 960M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 13s ago 2y 1152M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54

# ls /var/lib/ceph/604d56db-2fab-45db-a9ea-c418f9a8cca8/ cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931 mgr.a001s016.ctmoay osd.15 osd.4 crash mon.a001s016 osd.18 osd.7 crash.a001s016 nfs.nfs.0.0.a001s016.wumnjs osd.22 removed haproxy.nfs.nfs.a001s016.vpjtxt node-exporter.a001s016 osd.24 rgw.ceph.a001s016.gjyanl home osd.1 osd.27 keepalived.nfs.nfs.a001s016.ofqyjl osd.10 osd.30 mds.cephfs.a001s016.vtyivn osd.13 osd.33

# ceph mon dump dumped monmap epoch 6 epoch 6 fsid 604d56db-2fab-45db-a9ea-c418f9a8cca8 last_changed 2024-03-31T23:54:18.692983+0000 created 2021-09-30T16:15:12.884602+0000 min_mon_release 16 (pacific) election_strategy: 1 0: [v2:10.45.128.28:3300/0,v1:10.45.128.28:6789/0] mon.a001s018 1: [v2:10.45.128.27:3300/0,v1:10.45.128.27:6789/0] mon.a001s017

# ceph -s cluster: id: 604d56db-2fab-45db-a9ea-c418f9a8cca8 health: HEALTH_OK

services: mon: 2 daemons, quorum a001s018,a001s017 (age 18h) mgr: a001s016.ctmoay(active, since 109m), standbys: a001s017.bpygfm mds: 1/1 daemons up, 2 standby osd: 36 osds: 36 up (since 2h), 36 in (since 2y) rgw: 3 daemons active (3 hosts, 1 zones)

data: volumes: 1/1 healthy pools: 43 pools, 1633 pgs objects: 51.64M objects, 77 TiB usage: 119 TiB used, 132 TiB / 252 TiB avail pgs: 1633 active+clean

io: client: 356 MiB/s rd, 633 MiB/s wr, 506 op/s rd, 755 op/s wr

# ceph orch ps --refresh | grep mon mon.a001s016 a001s016 running (91s) 7s ago 91s 14.5M 2048M 16.2.5 6e73176320aa dcd4705c2069 mon.a001s017 a001s017 running (18h) 7s ago 21h 978M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 8s ago 2y 1163M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ls | grep mon mon 3/3 9s ago 114s a001s016;a001s017;a001s018;count:3

# ceph log last cephadm 2024-04-01T18:56:29.631337+0000 mgr.a001s017.bpygfm (mgr.60881489) 18014760 : cephadm [INF] Schedule restart daemon osd.6 2024-04-01T18:57:05.983577+0000 mgr.a001s017.bpygfm (mgr.60881489) 18014780 : cephadm [INF] Schedule restart daemon osd.9 2024-04-01T18:57:17.919236+0000 mgr.a001s017.bpygfm (mgr.60881489) 18014788 : cephadm [INF] Schedule restart daemon osd.12 2024-04-01T18:57:59.823635+0000 mgr.a001s017.bpygfm (mgr.60881489) 18014811 : cephadm [INF] Schedule restart daemon osd.16 2024-04-01T18:58:26.039310+0000 mgr.a001s017.bpygfm (mgr.60881489) 18014826 : cephadm [INF] Schedule restart daemon osd.19 2024-04-01T18:58:57.278955+0000 mgr.a001s017.bpygfm (mgr.60881489) 18014844 : cephadm [INF] Schedule restart daemon osd.21 2024-04-01T18:59:36.656834+0000 mgr.a001s017.bpygfm (mgr.60881489) 18014867 : cephadm [INF] Schedule restart daemon osd.25 2024-04-01T19:00:03.272925+0000 mgr.a001s017.bpygfm (mgr.60881489) 18014882 : cephadm [INF] Schedule restart daemon osd.28 2024-04-01T19:00:28.348138+0000 mgr.a001s017.bpygfm (mgr.60881489) 18014897 : cephadm [INF] Schedule restart daemon osd.31 2024-04-01T19:00:56.297517+0000 mgr.a001s017.bpygfm (mgr.60881489) 18014913 : cephadm [INF] Schedule restart daemon osd.34 2024-04-01T19:38:21.552347+0000 mgr.a001s017.bpygfm (mgr.60881489) 18016056 : cephadm [INF] Schedule restart daemon mgr.a001s017.bpygfm 2024-04-01T19:38:21.603564+0000 mgr.a001s017.bpygfm (mgr.60881489) 18016057 : cephadm [INF] Failing over to other MGR 2024-04-01T21:21:24.414308+0000 mgr.a001s016.ctmoay (mgr.61827136) 3096 : cephadm [INF] Saving service mon spec with placement a001s017;a001s018;count:2 2024-04-01T21:25:02.815151+0000 mgr.a001s016.ctmoay (mgr.61827136) 3211 : cephadm [INF] Saving service mon spec with placement a001s017;a001s018;count:2 2024-04-01T21:25:02.844721+0000 mgr.a001s016.ctmoay (mgr.61827136) 3212 : cephadm [INF] Safe to remove mon.a001s016: not in monmap (['a001s018', 'a001s017']) 2024-04-01T21:25:02.845059+0000 mgr.a001s016.ctmoay (mgr.61827136) 3213 : cephadm [INF] Removing monitor a001s016 from monmap... 2024-04-01T21:25:02.846401+0000 mgr.a001s016.ctmoay (mgr.61827136) 3214 : cephadm [INF] Removing daemon mon.a001s016 from a001s016 2024-04-01T21:25:06.870895+0000 mgr.a001s016.ctmoay (mgr.61827136) 3218 : cephadm [INF] Removing key for mon. 2024-04-01T21:26:27.861600+0000 mgr.a001s016.ctmoay (mgr.61827136) 3266 : cephadm [INF] Saving service mon spec with placement a001s016;a001s017;a001s018;count:3 2024-04-01T21:26:38.247516+0000 mgr.a001s016.ctmoay (mgr.61827136) 3273 : cephadm [INF] Saving service mon spec with placement a001s016;a001s017;a001s018;count:3 2024-04-01T21:26:44.494080+0000 mgr.a001s016.ctmoay (mgr.61827136) 3277 : cephadm [INF] Deploying daemon mon.a001s016 on a001s016

# cat /var/lib/ceph/604d56db-2fab-45db-a9ea-c418f9a8cca8/mon.a001s016/unit. i mage docker.io/ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5 a 2cb7ff7668f776b61b9d586 # ceph orch ps --daemon_type=mon NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mon.a001s016 a001s016 running (16m) 3m ago 16m 16.3M 2048M 16.2.5 6e73176320aa dcd4705c2069 mon.a001s017 a001s017 running (18h) 4m ago 21h 960M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 4m ago 2y 1155M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54

# ceph orch daemon redeploy mon.a001s016 docker.io/ceph/daemon:latest-pacific Scheduled to redeploy mon.a001s016 on host 'a001s016'

# ceph orch ps --daemon_type=mon --refresh NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mon.a001s016 a001s016 running (13s) 7s ago 18m 14.1M 2048M 16.2.5 6e73176320aa 1917b1edf4ab mon.a001s017 a001s017 running (18h) 6m ago 21h 960M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 6m ago 2y 1155M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54

===================================================================== = =======================

The second method also does not update ceph mon map:

#ceph mon dump dumped monmap epoch 6 epoch 6 fsid 604d56db-2fab-45db-a9ea-c418f9a8cca8 last_changed 2024-03-31T23:54:18.692983+0000 created 2021-09-30T16:15:12.884602+0000 min_mon_release 16 (pacific) election_strategy: 1 0: [v2:10.45.128.28:3300/0,v1:10.45.128.28:6789/0] mon.a001s018 1: [v2:10.45.128.27:3300/0,v1:10.45.128.27:6789/0] mon.a001s017

# ceph orch ps --refresh | grep mon mon.a001s016 a001s016 running (81s) 63s ago 81s 14.2M 2048M 16.2.5 6e73176320aa 0c0ec88139e1 mon.a001s017 a001s017 running (19h) 62s ago 22h 965M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 63s ago 2y 1157M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54

# ceph orch daemon stop mon.a001s016 Scheduled to stop mon.a001s016 on host 'a001s016' # ceph orch ps --refresh | grep mon mon.a001s016 a001s016 running (96s) 6s ago 96s 14.4M 2048M 16.2.5 6e73176320aa 0c0ec88139e1 mon.a001s017 a001s017 running (19h) 5s ago 22h 968M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 6s ago 2y 1157M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s016 a001s016 running (98s) 7s ago 98s 14.4M 2048M 16.2.5 6e73176320aa 0c0ec88139e1 mon.a001s017 a001s017 running (19h) 7s ago 22h 968M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 8s ago 2y 1157M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s016 a001s016 running (100s) 1s ago 100s 14.4M 2048M 16.2.5 6e73176320aa 0c0ec88139e1 mon.a001s017 a001s017 running (19h) 9s ago 22h 968M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 10s ago 2y 1157M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s016 a001s016 running (106s) 7s ago 106s 14.4M 2048M 16.2.5 6e73176320aa 0c0ec88139e1 mon.a001s017 a001s017 running (19h) 15s ago 22h 968M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 16s ago 2y 1157M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s016 a001s016 stopped 0s ago 108s - 2048M <unknown> <unknown> <unknown> mon.a001s017 a001s017 running (19h) 17s ago 22h 968M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 0s ago 2y 1160M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s016 a001s016 stopped 2s ago 111s - 2048M <unknown> <unknown> <unknown> mon.a001s017 a001s017 running (19h) 2s ago 22h 972M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 3s ago 2y 1160M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54

# ceph orch daemon rm mon.a001s016 Error EINVAL: must pass --force to REMOVE daemon with potentially PRECIOUS DATA for mon.a001s016

# ceph orch daemon rm mon.a001s016 --force Removed mon.a001s016 from host 'a001s016'

# ceph orch ps --refresh | grep mon mon.a001s017 a001s017 running (19h) 30s ago 22h 976M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 0s ago 2y 1166M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 #ceph orch ps --refresh | grep mon mon.a001s017 a001s017 running (19h) 2s ago 22h 982M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 3s ago 2y 1166M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s016 a001s016 starting - - - 2048M <unknown> <unknown> <unknown> mon.a001s017 a001s017 running (19h) 6s ago 22h 982M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 7s ago 2y 1166M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54 # ceph orch ps --refresh | grep mon mon.a001s016 a001s016 running (8s) 2s ago 8s 14.4M 2048M 16.2.5 6e73176320aa 39db8cfba7e1 mon.a001s017 a001s017 running (19h) 1s ago 22h 987M 2048M 16.2.5 6e73176320aa e5e5cb6c256c mon.a001s018 a001s018 running (5w) 2s ago 2y 1171M 2048M 16.2.5 6e73176320aa 7d2bb6d41f54

# ceph mon dump dumped monmap epoch 6 epoch 6 fsid 604d56db-2fab-45db-a9ea-c418f9a8cca8 last_changed 2024-03-31T23:54:18.692983+0000 created 2021-09-30T16:15:12.884602+0000 min_mon_release 16 (pacific) election_strategy: 1 0: [v2:10.45.128.28:3300/0,v1:10.45.128.28:6789/0] mon.a001s018 1: [v2:10.45.128.27:3300/0,v1:10.45.128.27:6789/0] mon.a001s017

#

-----Original Message----- From: Adiga, Anantha <anantha.adiga@intel.com> Sent: Monday, April 1, 2024 2:01 PM To: Eugen Block <eblock@nde.ag> Cc: ceph-users@ceph.io Subject: [ceph-users] Re: ceph status not showing correct monitor services

Thank you. I will try the export and import method first.

Thank you, Anantha

-----Original Message----- From: Eugen Block <eblock@nde.ag> Sent: Monday, April 1, 2024 1:57 PM To: Adiga, Anantha <anantha.adiga@intel.com> Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Re: ceph status not showing correct monitor services

I have two approaches in mind, first one (and preferred) would be to edit the mon spec to first remove mon.a001s016 and have a clean state. Get the current spec with:

ceph orch ls mon --export > mon-edit.yaml

Edit the spec file so that mon.a001s016 is not part of it, then apply:

ceph orch apply -i mon-edit.yaml

This should remove the mon.a001s016 daemon. Then wait a few minutes or so (until the daemon is actually gone, check locally on the node with 'cephadm ls' and in /var/lib/ceph/<FSID>/removed) and add it back to the spec file, then apply again. I would expect a third MON to be deployed. If that doesn't work for some reason you'll need to inspect logs to find the root cause.

The second approach would be to remove and add the daemon manually:

ceph orch daemon rm mon.a001s016

Wait until it's really gone, then add it:

ceph orch daemon add mon a001s016

Not entirely sure about the daemon add mon command, you might need to provide something else, I'm typing this by heart.

Zitat von "Adiga, Anantha" <anantha.adiga@intel.com>:

...

Hi Eugen,

Yes that is it. OSDs were restarted since mon a001s017 was reporting is low on available space. How to update the mon map to add mon.a001s016 as it is already online? And how to update mgr map to include standby mgr.a001s018 as it is also running.

ceph mon dump dumped monmap epoch 6 epoch 6 fsid 604d56db-2fab-45db-a9ea-c418f9a8cca8 last_changed 2024-03-31T23:54:18.692983+0000 created 2021-09-30T16:15:12.884602+0000 min_mon_release 16 (pacific) election_strategy: 1 0: [v2:10.45.128.28:3300/0,v1:10.45.128.28:6789/0] mon.a001s018 1: [v2:10.45.128.27:3300/0,v1:10.45.128.27:6789/0] mon.a001s017

Thank you, Anantha

-----Original Message----- From: Eugen Block <eblock@nde.ag> Sent: Monday, April 1, 2024 1:10 PM To: ceph-users@ceph.io Subject: [ceph-users] Re: ceph status not showing correct monitor services

Maybe it’s just not in the monmap? Can you show the output of:

ceph mon dump

Did you do any maintenance (apparently OSDs restarted recently) and maybe accidentally removed a MON from the monmap?

Zitat von "Adiga, Anantha" <anantha.adiga@intel.com>:

...

Hi Anthony,

Seeing it since last after noon. It is same with mgr services as , "ceph -s" is reporting only TWO instead of THREE

Also mon and mgr shows " is_active: false" see below.

# ceph orch ps --daemon_type=mgr NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mgr.a001s016.ctmoay a001s016 *:8443 running (18M) 3m ago 23M 206M - 16.2.5 6e73176320aa 169cafcbbb99 mgr.a001s017.bpygfm a001s017 *:8443 running (19M) 3m ago 23M 332M - 16.2.5 6e73176320aa 97257195158c mgr.a001s018.hcxnef a001s018 *:8443 running (20M) 3m ago 23M 113M - 16.2.5 6e73176320aa 21ba5896cee2

# ceph orch ls --service_name=mgr NAME PORTS RUNNING REFRESHED AGE PLACEMENT mgr 3/3 3m ago 23M a001s016;a001s017;a001s018;count:3

# ceph orch ps --daemon_type=mon --format=json-pretty

[ { "container_id": "8484a912f96a", "container_image_digests": [

docker.io/ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586<mailto:docker.io/ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586> ], "container_image_id": "6e73176320aaccf3b3fb660b9945d0514222bd7a83e28b96e8440c630ba6891f", "container_image_name": docker.io/ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586<mailto:docker.io/ceph/daemon@sha256:261bbe628f4b438f5bf10de5a8ee05282f2697a5a2cb7ff7668f776b61b9d586>, "created": "2024-03-31T23:55:16.164155Z", "daemon_id": "a001s016", "daemon_type": "mon", "hostname": "a001s016", "is_active": false, <== why is it false "last_refresh": "2024-04-01T19:38:30.929014Z", "memory_request": 2147483648, "memory_usage": 761685606, "ports": [], "service_name": "mon", "started": "2024-03-31T23:55:16.268266Z", "status": 1, "status_desc": "running", "version": "16.2.5" },

Thank you, Anantha

From: Anthony D'Atri <aad@dreamsnake.net> Sent: Monday, April 1, 2024 12:25 PM To: Adiga, Anantha <anantha.adiga@intel.com> Cc: ceph-users@ceph.io Subject: Re: [ceph-users] ceph status not showing correct monitor services

a001s017.bpygfm(active, since 13M), standbys: a001s016.ctmoay

Looks like you just had an mgr failover? Could be that the secondary mgr hasn't caught up with current events. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io