What does your OSD service specification look like? Did your db/wall device show as having free space prior to the OSD creation? On Tue, Jan 31, 2023, at 04:01, mailing-lists wrote:
OK, the OSD is filled again. In and Up, but it is not using the nvme WAL/DB anymore.
And it looks like the lvm group of the old osd is still on the nvme drive. I come to this idea, because the two nvme drives still have 9 lvm groups each. 18 groups but only 17 osd are using the nvme (shown in dashboard).
Do you have a hint on how to fix this?
Best
Ken
On 30.01.23 16:50, mailing-lists wrote:
oph wait,
i might have been too impatient:
1/30/23 4:43:07 PM[INF]Deploying daemon osd.232 on ceph-a1-06
1/30/23 4:42:26 PM[INF]Found osd claims for drivegroup dashboard-admin-1661788934732 -> {'ceph-a1-06': ['232']}
1/30/23 4:42:26 PM[INF]Found osd claims -> {'ceph-a1-06': ['232']}
1/30/23 4:42:19 PM[INF]Found osd claims -> {'ceph-a1-06': ['232']}
1/30/23 4:41:01 PM[INF]Found osd claims for drivegroup dashboard-admin-1661788934732 -> {'ceph-a1-06': ['232']}
1/30/23 4:41:01 PM[INF]Found osd claims -> {'ceph-a1-06': ['232']}
1/30/23 4:41:01 PM[INF]Found osd claims -> {'ceph-a1-06': ['232']}
1/30/23 4:41:00 PM[INF]Found osd claims -> {'ceph-a1-06': ['232']}
1/30/23 4:39:34 PM[INF]Found osd claims for drivegroup dashboard-admin-1661788934732 -> {'ceph-a1-06': ['232']}
1/30/23 4:39:34 PM[INF]Found osd claims -> {'ceph-a1-06': ['232']}
1/30/23 4:39:34 PM[INF]Found osd claims -> {'ceph-a1-06': ['232']}
Although, it doesnt show the NVME as wal/db yet, but i will let it proceed to a clear state until i do anything further.
On 30.01.23 16:42, mailing-lists wrote:
root@ceph-a2-01:/# ceph osd destroy 232 --yes-i-really-mean-it destroyed osd.232
OSD 232 shows now as destroyed and out in the dashboard.
root@ceph-a1-06:/# ceph-volume lvm zap /dev/sdm --> Zapping: /dev/sdm --> --destroy was not specified, but zapping a whole device will remove the partition table Running command: /usr/bin/dd if=/dev/zero of=/dev/sdm bs=1M count=10 conv=fsync stderr: 10+0 records in 10+0 records out stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0675647 s, 155 MB/s --> Zapping successful for: <Raw Device: /dev/sdm>
root@ceph-a2-01:/# ceph orch device ls
ceph-a1-06 /dev/sdm hdd TOSHIBA_X_X 16.0T 21m ago *locked*
It shows locked and is not automatically added now, which is good i think? otherwise it would probably be a new osd 307.
root@ceph-a2-01:/# ceph orch osd rm status No OSD remove/replace operations reported
root@ceph-a2-01:/# ceph orch osd rm 232 --replace Unable to find OSDs: ['232']
Unfortunately it is still not replacing.
It is so weird, i tried this procedure exactly in my virtual ceph environment and it just worked. The real scenario is acting up now. -.-
Do you have more hints for me?
Thank you for your help so far!
Best
Ken
On 30.01.23 15:46, David Orman wrote:
The 'down' status is why it's not being replaced, vs. destroyed, which would allow the replacement. I'm not sure why --replace lead to that scenario, but you will probably need to mark it destroyed for it to be replaced.
https://docs.ceph.com/en/latest/rados/operations/add-or-rm-osds/#replacing-a... has instructions on the non-orch way of doing that. You only need 1/2.
You should look through your logs to see what happened that the OSD was marked down and not destroyed. Obviously, make sure you understand ramifications before running any commands. :)
David
On Mon, Jan 30, 2023, at 04:24, mailing-lists wrote:
# ceph orch osd rm status No OSD remove/replace operations reported # ceph orch osd rm 232 --replace Unable to find OSDs: ['232']
It is not finding 232 anymore. It is still shown as down and out in the Ceph-Dashboard.
pgs: 3236 active+clean
This is the new disk shown as locked (because unzapped at the moment).
# ceph orch device ls
ceph-a1-06 /dev/sdm hdd TOSHIBA_X_X 16.0T 9m ago locked
Best
Ken
On 29.01.23 18:19, David Orman wrote:
What does "ceph orch osd rm status" show before you try the zap? Is your cluster still backfilling to the other OSDs for the PGs that were on the failed disk?
David
On Fri, Jan 27, 2023, at 03:25, mailing-lists wrote: > Dear Ceph-Users, > > i am struggling to replace a disk. My ceph-cluster is not > replacing the > old OSD even though I did: > > ceph orch osd rm 232 --replace > > The OSD 232 is still shown in the osd list, but the new hdd will be > placed as a new OSD. This wouldnt mind me much, if the OSD was also > placed on the bluestoreDB / NVME, but it doesn't. > > > My steps: > > "ceph orch osd rm 232 --replace" > > remove the failed hdd. > > add the new one. > > Convert the disk within the servers bios, so that the node can have > direct access on it. > > It shows up as /dev/sdt, > > enter maintenance mode > > reboot server > > drive is now /dev/sdm (which the old drive had) > > "ceph orch device zap node-x /dev/sdm" > > A new OSD is placed on the cluster. > > > Can you give me a hint, where did I take a wrong turn? Why is the > disk > not being used as OSD 232? > > > Best > > Ken > > _______________________________________________ > ceph-users mailing list --ceph-users@ceph.io > To unsubscribe send an email toceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
ceph-users mailing list --ceph-users@ceph.io To unsubscribe send an email toceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io