Hi Dev, The config-history keys related to node3 are expected to remain even for nodes that you remove, as these keys are not cleaned up upon node deletion. These keys are used by the 'ceph config log' command to list all configuration changes that have occurred over time. The 'ceph status' shows 132 PGs undersized - certainly all related to EC3+2 pools - and 685 other PGs related to replicated pools. With 5 nodes and 22 OSDs per node, that's 132*5/(95+22) = 5 PGs per OSD for EC pools and 685*3/(95+22) = 17 PGs per OSD for replicated pool. With such a low PG per OSD ratio, it is very likely that some pools are not utilizing all OSDs. You might want to increase the number of PGs in these pools to better balance I/O and improve performance. Additionally, ensure that you do not have an even number of MONs. Regards, Frédéric. ----- Le 25 Jan 25, à 20:34, Devender Singh <devender@netskrt.io> a écrit :
Hello Fredreic
Thanks for your reply, Yes I also faced this issue after draining and removing of the node. So used the same command and remove “original_weight” using ceph config-key get mgr/cephadm/osd_remove_queue and injected file again. Which resolved the orch issue.
“ Error ENOENT: Module not found - ceph orch commands stoppd working
ceph config-key get mgr/cephadm/osd_remove_queue > osd_remove_queue.json
Then only remove the "original_weight" key from that json and upload it back to the config-key store:
ceph config-key set mgr/cephadm/osd_remove_queue -i osd_remove_queue_modified.json
Then fail the mgr:
ceph mgr fail ”
But now issue is, my cluster showing objects misplaced, whereas I had 5 nodes with host failure domain with R3 pool (size 3 and min 2), EC with 3+2.
# ceph -s
cluster:
id: 384d7590-d018-11ee-b74c-5b2acfe0b35c
health: HEALTH_WARN
Degraded data redundancy: 2848547/29106793 objects degraded (9.787%), 105 pgs degraded, 132 pgs undersized
services:
mon: 4 daemons, quorum node1,node5,node4,node2 (age 12h)
mgr: node1.cvknae(active, since 12h), standbys: node4.foomun
mds: 2/2 daemons up, 2 standby
osd: 95 osds: 95 up (since 16h), 95 in (since 21h); 124 remapped pgs
rgw: 2 daemons active (2 hosts, 1 zones)
data:
volumes: 2/2 healthy
pools: 18 pools, 817 pgs
objects: 6.06M objects, 20 TiB
usage: 30 TiB used, 302 TiB / 332 TiB avail
pgs: 2848547/29106793 objects degraded (9.787%)
2617833/29106793 objects misplaced (8.994%)
561 active+clean
124 active+clean+remapped
105 active+undersized+degraded
27 active+undersized
io:
client: 1.4 MiB/s rd, 4.0 MiB/s wr, 25 op/s rd, 545 op/s wr
And when using 'ceph config-key ls’ it’s showing old node and osd’s.
# ceph config-key ls|grep -i 03n
"config-history/135/+osd/host:node3/osd_memory_target",
"config-history/14990/+osd/host:node3/osd_memory_target",
"config-history/14990/-osd/host:node3/osd_memory_target",
"config-history/15003/+osd/host:node3/osd_memory_target",
"config-history/15003/-osd/host:node3/osd_memory_target",
"config-history/15016/+osd/host:node3/osd_memory_target",
"config-history/15016/-osd/host:node3/osd_memory_target",
"config-history/15017/+osd/host:node3/osd_memory_target",
"config-history/15017/-osd/host:node3/osd_memory_target",
"config-history/15022/+osd/host:node3/osd_memory_target",
"config-history/15022/-osd/host:node3/osd_memory_target",
"config-history/15024/+osd/host:node3/osd_memory_target",
"config-history/15024/-osd/host:node3/osd_memory_target",
"config-history/15025/+osd/host:node3/osd_memory_target",
"config-history/15025/-osd/host:node3/osd_memory_target",
"config-history/153/+osd/host:node3/osd_memory_target",
"config-history/153/-osd/host:node3/osd_memory_target",
"config-history/165/+mon.node3/container_image",
"config-history/171/-mon.node3/container_image",
"config-history/176/+client.crash.node3/container_image",
"config-history/182/-client.crash.node3/container_image",
"config-history/4276/+osd/host:node3/osd_memory_target",
"config-history/4276/-osd/host:node3/osd_memory_target",
"config-history/433/+client.ceph-exporter.node3/container_image",
"config-history/439/-client.ceph-exporter.node3/container_image",
"config-history/459/+osd/host:node3/osd_memory_target",
"config-history/459/-osd/host:node3/osd_memory_target",
"config-history/465/+osd/host:node3/osd_memory_target",
"config-history/465/-osd/host:node3/osd_memory_target",
"config-history/4867/+osd/host:node3/osd_memory_target",
"config-history/4867/-osd/host:node3/osd_memory_target",
"config-history/4878/+mon.node3/container_image",
"config-history/4884/-mon.node3/container_image",
"config-history/4889/+client.crash.node3/container_image",
"config-history/4895/-client.crash.node3/container_image",
"config-history/5139/+mds.k8s-dev-cephfs.node3.iebxqn/container_image",
"config-history/5142/-mds.k8s-dev-cephfs.node3.iebxqn/container_image",
"config-history/5150/+client.ceph-exporter.node3/container_image",
"config-history/5156/-client.ceph-exporter.node3/container_image",
"config-history/5179/+osd/host:node3/osd_memory_target",
"config-history/5179/-osd/host:node3/osd_memory_target",
"config-history/5183/+client.rgw.sea-dev.node3.betyqd/rgw_frontends",
"config-history/5189/+osd/host:node3/osd_memory_target",
"config-history/5189/-osd/host:node3/osd_memory_target",
"config-history/6929/-client.rgw.sea-dev.node3.betyqd/rgw_frontends",
"config-history/6933/+osd/host:node3/osd_memory_target",
"config-history/6933/-osd/host:node3/osd_memory_target",
"config-history/9710/+osd/host:node3/osd_memory_target",
"config-history/9710/-osd/host:node3/osd_memory_target",
"config/osd/host:node3/osd_memory_target”,
Regards
Dev
On Jan 25, 2025, at 4:39 AM, Frédéric Nass <frederic.nass@univ-lorraine.fr> wrote:
Hi, I've seen this happening on a test cluster after draining a host that also had a MGR service. Can you check if Eugen's solution here [1] helps in your case ? And maybe investigate 'ceph config-key ls' for any issues in config keys ?
Regards, Frédéric.
[1] [ https://www.google.com/url?q=https://www.spinics.net/lists/ceph-users/msg83667.html&source=gmail-imap&ust=1738413580000000&usg=AOvVaw3Zk70LrQ6SrLX02gJ7Cowl | https://www.spinics.net/lists/ceph-users/msg83667.html ]
De : Devender Singh <devender@netskrt.io> Envoyé : samedi 25 janvier 2025 06:27 À : Fnu Virender Kumar Cc: ceph-users Objet : [ceph-users] Re: Error ENOENT: Module not found
Thanks for you reply… but those command not working as its an always module..but strange still showing error,
# ceph mgr module enable orchestrator module 'orchestrator' is already enabled (always-on)
# ceph orch set backend — returns successfully…
# # ceph orch ls Error ENOENT: No orchestrator configured (try `ceph orch set backend`)
Its revolving between same error..
Root Cause: I removed a hosts and its odd’s and after some time above error started automatically.
Earlier in the had 5 nodes but now 4.. Cluster is showing unclean pg but not doing anything..
But big error is Error ENOENT:
Regards Dev
On Jan 24, 2025, at 4:59 PM, Fnu Virender Kumar <virenderk@uchicago.edu> wrote:
Did you try
Ceph mgr module enable orchestrator Ceph orch set backend Ceph orch ls
Check the mgr service daemon as well Ceph -s
Regards Virender From: Devender Singh <devender@netskrt.io> Sent: Friday, January 24, 2025 6:34:43 PM To: ceph-users <ceph-users@ceph.io> Subject: [ceph-users] Error ENOENT: Module not found
Hello all
Any quick fix for …
root@sea-devnode1:~# ceph orch ls Error ENOENT: Module not found
Regards Dev _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io