Hi, just add the host back to the cluster with 'ceph orch host add ...'. If it still has the cephadm pub key, the orchestrator would deploy the missing mon daemon (depending on your actual mon spec) and a couple of other services. In the current state, there won't be any recovery because you have only two hosts but your crush rule requires three. There are ways to recover anyway, for example by editing the rule or reduce the pool size to 2, but I would only do that in a test cluster. Depending on what the host removal actually removed, you might be able to just reintegrate the OSDs (ceph cephadm osd active <host>) and recovery will kick in. In case the OSD keyrings are gone, you can import them, they should be still present on the removed OSD host. Regards, Eugen Zitat von Kirby Haze <kirbyhaze01@gmail.com>:
I have a 3 node test 17.2.7 cluster, and I decided to power one of the hosts down which contained a mon, 6 osds and a standby mgr. Then I used the `ceph orch host rm <host> --force --rm` to remove the host (after powering it down).
All of this looks expected except the logs after removing the host. These pools have size 3 min_size 2. Actually I'm unsure what would happen if I were even to successfully add back the host as those objects would be degraded. Besides the error from cephadm, what is ceph thinking when I pulled the plug then removed that host? Am I guaranteed some type of recovery if I do successfully add back that host?
---- Before removing host
root@ceph-test-2:/# ceph -s cluster: id: fca870d8-e431-11ef-8000-bc2411363b7d health: HEALTH_OK
services: mon: 3 daemons, quorum ceph-test-2,ceph-test-3,ceph-test-4 (age 29m) mgr: ceph-test-2.vbjhdq(active, since 31m), standbys: ceph-test-4.jjubsa osd: 18 osds: 18 up (since 22m), 18 in (since 28m) rgw: 1 daemon active (1 hosts, 1 zones)
data: pools: 5 pools, 129 pgs objects: 223 objects, 581 KiB usage: 5.2 GiB used, 8.8 TiB / 8.8 TiB avail pgs: 129 active+clean
root@ceph-test-2:/# ceph orch host ls HOST ADDR LABELS STATUS ceph-test-2 10.0.0.52 _admin,rgw ceph-test-3 10.0.0.53 ceph-test-4 10.0.0.54
------------- After removing host
# ceph orch host rm ceph-test-4 --offline --force Removed offline host 'ceph-test-4'
# ceph orch ps mon.ceph-test-4 ceph-test-4 stopped osd.1 ceph-test-4 error osd.7 ceph-test-4 error .... ....
# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 5.85956 root default -5 2.92978 host ceph-test-2 2 hdd 0.48830 osd.2 up 1.00000 1.00000 4 hdd 0.48830 osd.4 up 1.00000 1.00000 8 hdd 0.48830 osd.8 up 1.00000 1.00000 11 hdd 0.48830 osd.11 up 1.00000 1.00000 14 hdd 0.48830 osd.14 up 1.00000 1.00000 16 hdd 0.48830 osd.16 up 1.00000 1.00000 -3 2.92978 host ceph-test-3 0 hdd 0.48830 osd.0 up 1.00000 1.00000 3 hdd 0.48830 osd.3 up 1.00000 1.00000 6 hdd 0.48830 osd.6 up 1.00000 1.00000 9 hdd 0.48830 osd.9 up 1.00000 1.00000 12 hdd 0.48830 osd.12 up 1.00000 1.00000 15 hdd 0.48830 osd.15 up 1.00000 1.00000
root@ceph-test-2:/rootfs/root# ceph -s cluster: id: fca870d8-e431-11ef-8000-bc2411363b7d health: HEALTH_WARN 6 failed cephadm daemon(s) Degraded data redundancy: 145/669 objects degraded (21.674%), 24 pgs degraded, 71 pgs undersized
services: mon: 2 daemons, quorum ceph-test-2,ceph-test-3 (age 50m) mgr: ceph-test-2.vbjhdq(active, since 2h), standbys: ceph-test-3.wzmioq osd: 12 osds: 12 up (since 51m), 12 in (since 2h); 58 remapped pgs rgw: 1 daemon active (1 hosts, 1 zones)
data: pools: 5 pools, 129 pgs objects: 223 objects, 581 KiB usage: 3.5 GiB used, 5.9 TiB / 5.9 TiB avail pgs: 145/669 objects degraded (21.674%) 75/669 objects misplaced (11.211%) 54 active+clean+remapped 47 active+undersized 24 active+undersized+degraded 4 active+clean
progress: Global Recovery Event (50m) [================............] (remaining: 37m)
The cephadm logs show this as well
2025-02-06T06:27:46.936+0000 7f85026a4700 -1 log_channel(cephadm) log [ERR] : auth get failed: failed to find osd.7 in keyring retval: -2 Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/serve.py", line 1002, in _check_daemons self.mgr._daemon_action(daemon_spec, action=action) File "/usr/share/ceph/mgr/cephadm/module.py", line 2136, in _daemon_action daemon_spec) File "/usr/share/ceph/mgr/cephadm/services/cephadmservice.py", line 494, in generate_config extra_ceph_config=daemon_spec.ceph_conf) File "/usr/share/ceph/mgr/cephadm/services/cephadmservice.py", line 520, in get_config_and_keyring 'entity': entity, File "/usr/share/ceph/mgr/mgr_module.py", line 1593, in check_mon_command raise MonCommandFailed(f'{cmd_dict["prefix"]} failed: {r.stderr} retval: {r.retval}') mgr_module.MonCommandFailed: auth get failed: failed to find osd.7 in keyring retval: -2 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io