- ceph-users - lists.ceph.io

Re: Integrating OSDs from a previous Ceph installation
by Jacek Rużyczka 31 May '26

31 May '26

Hi, and here, everything ends for me: ceph-objectstore-tool --data-path */var/lib/ceph/osd/ceph-**0* --no-mon-config --op update-mon-db --mon-store-path /tmp/mon-store As all hosts have died, I no longer have the necessary /var/lib/ceph/osd folder. And: I've tried to bring up a new cluster with cephadm bootstrap --mon-ip $MYIP --initial-config /etc/ceph/ ceph.conf~~, which itself was the config file containing the original ID, namely, 8aad3073-39a1-11f1-bf6e-f2704a1efa9b, but this was apparently ignored by cephadm. My new cluster now has this ID: add010c1-5cfa-11f1-8af8-ceccf52b4a0f So: Have my data become inaccessible?

2 1

Re: The mail threading on this list is broken
by Adam Kraitman 31 May '26

31 May '26

Thank you for the detailed analysis, the threading issue you've described is a known problem with the current mailing list infrastructure. We're planning to migrate all mailing lists to Groups.io, and this issue will be resolved as part of that migration. We'll share more details and the migration timeline once everything is finalized On Sun, May 31, 2026 at 3:08 PM Kai Stian Olstad via ceph-users < ceph-users(a)ceph.io> wrote: > Hi > > For several month the mail threading on this list broken and making the > list hard to read. > It can easily be seen by looking at the archive, in this link for about > 9 mount(as of writing) > > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/latest?count=10&pa… > > Subjects all have nice subject without "Re:" for replay, and on the > right you see how may people(person icon) in the thread and number of > post(talking bubble) in the thread. > > But if we look at more recent posts like this one > > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/latest?count=10&pa… > you see a lot of "Re:" in the subject and most of them have one person > and only one post because the threading is broken. > > Take this recent mail "Weird behavior for 2 OSDs in our cluster" > The fist mail by Wannes Smet > > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/62B63BH33F3… > when i check my mail client this mail has following ID. > "Message-ID: > < > 0100019e7c77595f-ffb4d04b-1771-46cf-8b60-10795bc6a1c5-000000(a)email.amazonses.com > >" > > The reply from Eugen Block > > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/E63Z3JR7DMS… > the header says this is a reply to different ID > "In-Reply-To: > < > 0100019e7c7979e5-8d4d96e6-1c93-4a76-a5cd-efdf25ada853-000000(a)email.amazonses.com > >" > > When I search the header of the mail from Wannes Smet I find no such ID > that is in the In-Reply-To. > > It looks to me that the mailing-list server is sending different > "Messages-ID" to the members on the list. > This should be the same or the threading breaks as we can se. > > Can this be fixed please? > > -- > Kai Stian Olstad > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

The mail threading on this list is broken
by Kai Stian Olstad 31 May '26

31 May '26

Hi For several month the mail threading on this list broken and making the list hard to read. It can easily be seen by looking at the archive, in this link for about 9 mount(as of writing) https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/latest?count=10&pa… Subjects all have nice subject without "Re:" for replay, and on the right you see how may people(person icon) in the thread and number of post(talking bubble) in the thread. But if we look at more recent posts like this one https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/latest?count=10&pa… you see a lot of "Re:" in the subject and most of them have one person and only one post because the threading is broken. Take this recent mail "Weird behavior for 2 OSDs in our cluster" The fist mail by Wannes Smet https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/62B63BH33F3… when i check my mail client this mail has following ID. "Message-ID: <0100019e7c77595f-ffb4d04b-1771-46cf-8b60-10795bc6a1c5-000000(a)email.amazonses.com>" The reply from Eugen Block https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/E63Z3JR7DMS… the header says this is a reply to different ID "In-Reply-To: <0100019e7c7979e5-8d4d96e6-1c93-4a76-a5cd-efdf25ada853-000000(a)email.amazonses.com>" When I search the header of the mail from Wannes Smet I find no such ID that is in the In-Reply-To. It looks to me that the mailing-list server is sending different "Messages-ID" to the members on the list. This should be the same or the threading breaks as we can se. Can this be fixed please? -- Kai Stian Olstad

1 0

Re: Weird behavior for 2 OSDs in our cluster
by Eugen Block 31 May '26

31 May '26

Hi, the MGR doesn't always report the correct PG status, so don't rely on that too much. Sometimes it's necessary to restart primary OSDs for stuck PGs, although a repeer could have been sufficient. Your Ceph clients had to refresh their osdmap, that's when they notice that there had been down OSDs. It's not a real-time log in this case, no need to worry. It's a common question though, I think we also asked it 8 to 10 years ago. ;-) Regards, Eugen Zitat von Wannes Smet via ceph-users <ceph-users(a)ceph.io>: > Hi, > > I'm running a Ceph cluster 19.2.2, 23 nodes, 152 OSDs, cephadm > deployed. Most SAS SSDs, 12 NVMe SSDs. > > Yesterday we experienced a total power failure and everything went > down hard. Also our Ceph cluster. There were a couple of things, but > this stood out after it got back up: > > [ERR] OSD_UNREACHABLE: 2 osds(s) are not reachable > osd.53's public address is not in '192.168.11.0/24' subnet > osd.86's public address is not in '192.168.11.0/24' subnet > > ceph -s did not say reduced data {availability,redundancy} which is > a bit "off", given that both OSDs are in separate hosts, failure > domain=host. There must have been PGs with less than 3 replicas and > also PGs with just one replica left? > > So I manually restarted those OSDs with systemctl , a recovery > process started and all our VMs, "magically" started booting now. > I'm also surprised that the recovery process only started when those > OSDs got back up. > > I didn't make too much of the above, but now this morning, I'm > looking at the kernel ring buffer of our PVE nodes and I notice the > logs below. Just a single "blip". All at the same time on all of our > PVE nodes (ceph clients): > > [Sat May 30 22:03:46 2026] libceph > (e8020818-2100-11f0-8a12-9cdc71772100 e179035): osd53 down > [Sat May 30 22:03:46 2026] libceph > (e8020818-2100-11f0-8a12-9cdc71772100 e179050): osd53 up > [Sat May 30 22:03:46 2026] libceph > (e8020818-2100-11f0-8a12-9cdc71772100 e179057): osd86 down > [Sat May 30 22:03:46 2026] libceph > (e8020818-2100-11f0-8a12-9cdc71772100 e179074): osd86 up > > I don't see anything weird in the Ceph cluster itself, neither in > the log files of the ODS. > > I'm not sure what to make from this. Why would this happen and what > would you do? > > Thanks for your insights, > > Wannes Smet > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Integrating OSDs from a previous Ceph installation
by Eugen Block 31 May '26

31 May '26

Again, please don't drop the list from your responses. I played around a bit with a single-host cluster, created three OSDs, added a pool and copied a few bytes of data into it. Then I attached the drives to a different host where I bootstrapped a fresh cluster with the previous FSID. But the OSDs can't be activated, unfortunately. I think the main problem is this: orchestrator._interface.OrchestratorError: osd.0 not in osdmap I'm not sure if the procedure to recover from OSDs will help here, if I have the time, I'm gonna try. Or another approach could be to create new OSDs in the new cluster you built (if you have the resources) and then use ceph-objectstore-tool export/import to get the data onto the new OSDs. But that's also an advanced procedure, you require experience to accomplish that. I still think the easiest approach would have been to reduce the monmap to 1 and start from there, but it's too late for that now. Zitat von Jacek Rużyczka <jacek.ruzyczka(a)googlemail.com>: > But then I'd have to destroy the old Ceph cluster, right? > > Am Sa., 30. Mai 2026 um 11:15 Uhr schrieb Eugen Block via ceph-users < > ceph-users(a)ceph.io>: > >> I haven't tried it myself yet, but you could bootstrap a fresh cluster >> with the flag "--fsid", excerpt from the 'cephadm bootstrap -h' command: >> >> --fsid FSID cluster FSID >> >> and set the FSID your OSDs show. But again, I have no idea if that's >> gonna work. >> >> Zitat von Eugen Block <eblock(a)nde.ag>: >> >> > You're not gonna be able to reactivate those OSDs because they were >> > built with a different cluster FSID: >> > >> > Your OSDs: >> > >> >> cluster fsid 8aad3073-39a1-11f1-bf6e-f2704a1efa9b >> > >> > Your new cluster: >> > >> >> Inferring fsid 98e04296-5b5a-11f1-84cf-ceccf52b4a0f >> > >> > And if you have a cephadm-managed cluster, you shouldn't use >> > ceph-volume directly but via cephadm [0]: >> > >> > ceph cephadm osd activate <node> >> > >> > But this won't help you if the cluster's fsid differs as mentioned >> > above. I think you might be able to recover if you recreate the >> > monmap by using the OSDs [1]. But this would mean you'd need to tear >> > down your MONs, it's a difficult situation, I can't tell if it's >> > gonna work that way. The procedure itself has been verified several >> > times on this list, but again, it's hard to tell. >> > >> > [0] >> > >> https://docs.ceph.com/en/latest/cephadm/services/osd/#activate-existing-osds >> > [1] >> > >> https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-mon/#… >> > >> > >> > Zitat von Jacek Rużyczka via ceph-users <ceph-users(a)ceph.io>: >> > >> >> Hi, >> >> >> >> After being forced to reinstall the O/S on all my Ceph nodes, I'm >> trying to >> >> reactivate the OSDs, whose data lie on separate drives. The OSDs are >> >> recognised: >> >> >> >> mixtile@blade3n3:~$ sudo sudo ceph-volume lvm list >> >> [sudo] password for mixtile: >> >> >> >> >> >> ====== osd.1 ======= >> >> >> >> [block] >> >> /dev/ceph-5b8338a0-9246-48f5-9cfd-e1bbfa1cc199/osd-block-ccba3dd >> >> 2-0503-440f-b7c2-47e8d8e85253 >> >> >> >> block device >> >> /dev/ceph-5b8338a0-9246-48f5-9cfd-e1bbfa1cc199/o >> >> sd-block-ccba3dd2-0503-440f-b7c2-47e8d8e85253 >> >> block uuid 21ftde-FdBB-s4DM-Qeq7-TU4R-vjY7-6VXLoW >> >> cephx lockbox secret >> >> cluster fsid 8aad3073-39a1-11f1-bf6e-f2704a1efa9b >> >> cluster name ceph >> >> crush device class >> >> encrypted 0 >> >> osd fsid ccba3dd2-0503-440f-b7c2-47e8d8e85253 >> >> osd id 1 >> >> osdspec affinity all-available-devices >> >> type block >> >> vdo 0 >> >> with tpm 0 >> >> devices /dev/nvme0n1 >> >> >> >> *But: *When trying to link the OSD to the all-new Ceph installation >> (I've >> >> made several Google searches with contradictory results. This is the >> last >> >> one.), I get an error: >> >> >> >> mixtile@blade3n3:~$ sudo cephadm shell >> >> Inferring fsid 98e04296-5b5a-11f1-84cf-ceccf52b4a0f >> >> Inferring config >> >> /var/lib/ceph/98e04296-5b5a-11f1-84cf-ceccf52b4a0f/mon.blade3n3 >> >> /config >> >> Not using image >> >> 'sha256:af0c5903e901e329adabe219dfc8d0c3efc1f05102a753902f33ee16 >> >> c26b6cee' as it's not in list of non-dangling images with ceph=True >> label >> >> Unable to find image 'quay.io/ceph/ceph:v19' locally >> >> v19: Pulling from ceph/ceph >> >> Digest: >> >> sha256:af0c5903e901e329adabe219dfc8d0c3efc1f05102a753902f33ee16c26b6cee >> >> Status: Downloaded newer image for quay.io/ceph/ceph:v19 >> >> >> >> root@blade3n3:/usr/bin# cd >> >> >> >> root@blade3n3:~# ceph-volume lvm activate --all >> >> Running command: /usr/bin/ceph-authtool --gen-print-key >> >> Running command: /usr/bin/ceph-authtool --gen-print-key >> >> --> Activating OSD ID 1 FSID ccba3dd2-0503-440f-b7c2-47e8d8e85253 >> >> Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-1 >> >> Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 >> >> Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph >> prime-osd-dir >> >> --dev >> >> >> /dev/ceph-5b8338a0-9246-48f5-9cfd-e1bbfa1cc199/osd-block-ccba3dd2-0503-440f-b7c2-47e8d8e85253 >> >> --path /var/lib/ceph/osd/ceph-1 --no-mon-config >> >> stderr: 2026-05-29T17:28:36.854+0000 ffffbd2de040 -1 bdev(0xaaab21623800 >> >> /dev/c >> >> >> eph-5b8338a0-9246-48f5-9cfd-e1bbfa1cc199/osd-block-ccba3dd2-0503-440f-b7c2-47e8d >> >> 8e85253) open stat got: (1) Operation not permitted >> >> stderr: failed to read label for >> >> /dev/ceph-5b8338a0-9246-48f5-9cfd-1bbfa1cc199 >> >> /osd-block-ccba3dd2-0503-440f-b7c2-47e8d8e85253: (1) Operation not >> >> permitted >> >> --> RuntimeError: command returned non-zero exit status: 1 >> >> >> >> This puzzles me because I'm already root, but the Bluestore tool won't >> let >> >> me reactivate the OSD? The old OSD is now full of valuable production >> data. >> >> And: Isn't it possible to use ceph orch to perform this operation? >> >> >> >> Regards >> >> Jacek Rużyczka >> >> _______________________________________________ >> >> ceph-users mailing list -- ceph-users(a)ceph.io >> >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>

1 1

Weird behavior for 2 OSDs in our cluster
by Wannes Smet 31 May '26

31 May '26

Hi, I'm running a Ceph cluster 19.2.2, 23 nodes, 152 OSDs, cephadm deployed. Most SAS SSDs, 12 NVMe SSDs. Yesterday we experienced a total power failure and everything went down hard. Also our Ceph cluster. There were a couple of things, but this stood out after it got back up: [ERR] OSD_UNREACHABLE: 2 osds(s) are not reachable osd.53's public address is not in '192.168.11.0/24' subnet osd.86's public address is not in '192.168.11.0/24' subnet ceph -s did not say reduced data {availability,redundancy} which is a bit "off", given that both OSDs are in separate hosts, failure domain=host. There must have been PGs with less than 3 replicas and also PGs with just one replica left? So I manually restarted those OSDs with systemctl , a recovery process started and all our VMs, "magically" started booting now. I'm also surprised that the recovery process only started when those OSDs got back up. I didn't make too much of the above, but now this morning, I'm looking at the kernel ring buffer of our PVE nodes and I notice the logs below. Just a single "blip". All at the same time on all of our PVE nodes (ceph clients): [Sat May 30 22:03:46 2026] libceph (e8020818-2100-11f0-8a12-9cdc71772100 e179035): osd53 down [Sat May 30 22:03:46 2026] libceph (e8020818-2100-11f0-8a12-9cdc71772100 e179050): osd53 up [Sat May 30 22:03:46 2026] libceph (e8020818-2100-11f0-8a12-9cdc71772100 e179057): osd86 down [Sat May 30 22:03:46 2026] libceph (e8020818-2100-11f0-8a12-9cdc71772100 e179074): osd86 up I don't see anything weird in the Ceph cluster itself, neither in the log files of the ODS. I'm not sure what to make from this. Why would this happen and what would you do? Thanks for your insights, Wannes Smet

1 0

Re: Integrating OSDs from a previous Ceph installation
by Eugen Block 30 May '26

30 May '26

You're not gonna be able to reactivate those OSDs because they were built with a different cluster FSID: Your OSDs: > cluster fsid 8aad3073-39a1-11f1-bf6e-f2704a1efa9b Your new cluster: > Inferring fsid 98e04296-5b5a-11f1-84cf-ceccf52b4a0f And if you have a cephadm-managed cluster, you shouldn't use ceph-volume directly but via cephadm [0]: ceph cephadm osd activate <node> But this won't help you if the cluster's fsid differs as mentioned above. I think you might be able to recover if you recreate the monmap by using the OSDs [1]. But this would mean you'd need to tear down your MONs, it's a difficult situation, I can't tell if it's gonna work that way. The procedure itself has been verified several times on this list, but again, it's hard to tell. [0] https://docs.ceph.com/en/latest/cephadm/services/osd/#activate-existing-osds [1] https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-mon/#… Zitat von Jacek Rużyczka via ceph-users <ceph-users(a)ceph.io>: > Hi, > > After being forced to reinstall the O/S on all my Ceph nodes, I'm trying to > reactivate the OSDs, whose data lie on separate drives. The OSDs are > recognised: > > mixtile@blade3n3:~$ sudo sudo ceph-volume lvm list > [sudo] password for mixtile: > > > ====== osd.1 ======= > > [block] > /dev/ceph-5b8338a0-9246-48f5-9cfd-e1bbfa1cc199/osd-block-ccba3dd > 2-0503-440f-b7c2-47e8d8e85253 > > block device > /dev/ceph-5b8338a0-9246-48f5-9cfd-e1bbfa1cc199/o > sd-block-ccba3dd2-0503-440f-b7c2-47e8d8e85253 > block uuid 21ftde-FdBB-s4DM-Qeq7-TU4R-vjY7-6VXLoW > cephx lockbox secret > cluster fsid 8aad3073-39a1-11f1-bf6e-f2704a1efa9b > cluster name ceph > crush device class > encrypted 0 > osd fsid ccba3dd2-0503-440f-b7c2-47e8d8e85253 > osd id 1 > osdspec affinity all-available-devices > type block > vdo 0 > with tpm 0 > devices /dev/nvme0n1 > > *But: *When trying to link the OSD to the all-new Ceph installation (I've > made several Google searches with contradictory results. This is the last > one.), I get an error: > > mixtile@blade3n3:~$ sudo cephadm shell > Inferring fsid 98e04296-5b5a-11f1-84cf-ceccf52b4a0f > Inferring config > /var/lib/ceph/98e04296-5b5a-11f1-84cf-ceccf52b4a0f/mon.blade3n3 > /config > Not using image > 'sha256:af0c5903e901e329adabe219dfc8d0c3efc1f05102a753902f33ee16 > c26b6cee' as it's not in list of non-dangling images with ceph=True label > Unable to find image 'quay.io/ceph/ceph:v19' locally > v19: Pulling from ceph/ceph > Digest: > sha256:af0c5903e901e329adabe219dfc8d0c3efc1f05102a753902f33ee16c26b6cee > Status: Downloaded newer image for quay.io/ceph/ceph:v19 > > root@blade3n3:/usr/bin# cd > > root@blade3n3:~# ceph-volume lvm activate --all > Running command: /usr/bin/ceph-authtool --gen-print-key > Running command: /usr/bin/ceph-authtool --gen-print-key > --> Activating OSD ID 1 FSID ccba3dd2-0503-440f-b7c2-47e8d8e85253 > Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-1 > Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 > Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir > --dev > /dev/ceph-5b8338a0-9246-48f5-9cfd-e1bbfa1cc199/osd-block-ccba3dd2-0503-440f-b7c2-47e8d8e85253 > --path /var/lib/ceph/osd/ceph-1 --no-mon-config > stderr: 2026-05-29T17:28:36.854+0000 ffffbd2de040 -1 bdev(0xaaab21623800 > /dev/c > eph-5b8338a0-9246-48f5-9cfd-e1bbfa1cc199/osd-block-ccba3dd2-0503-440f-b7c2-47e8d > 8e85253) open stat got: (1) Operation not permitted > stderr: failed to read label for > /dev/ceph-5b8338a0-9246-48f5-9cfd-1bbfa1cc199 > /osd-block-ccba3dd2-0503-440f-b7c2-47e8d8e85253: (1) Operation not > permitted > --> RuntimeError: command returned non-zero exit status: 1 > > This puzzles me because I'm already root, but the Bluestore tool won't let > me reactivate the OSD? The old OSD is now full of valuable production data. > And: Isn't it possible to use ceph orch to perform this operation? > > Regards > Jacek Rużyczka > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 1

Short survey for anyone running 15-30TB SSD OSDs
by Anthony D'Atri 29 May '26

29 May '26

* Or even 60 TB SSD OSDs * Which Ceph release? * Are you using the bitmap or hybrid bluefs / BlueStore allocators? * Are you serving CephFS from them? * Autotuning or static BlueStore cache / osd_memory_target values? * What’s the RSS like on your OSDs? I’m seeing some unexpected behavior and would like some context from others’ experience. I suggest replying privately so as to not flood the list, and I’ll summarize. This will help me directly as well as evolve the hardware recommendations in the docs. Thanks. Anthony D’Atri Ceph Ambassador, Doc lead, etc.

1 0

Re: tentacle 20.2.2 QE validation status
by Adam Emerson 29 May '26

29 May '26

On 29/05/2026, Yuri Weinstein via dev wrote: > We are still missing several approvals (rgw, rbd, orch, ceph-volume) > Please review ASAP > TIA RGW approved. > > On Tue, May 26, 2026 at 8:06 AM Yuri Weinstein <yweinste(a)redhat.com> wrote: > > > Details of this release are summarized here: > > > > https://tracker.ceph.com/issues/75878#note-6 > > > > Release Notes - TBD > > LRC upgrade - TBD > > > > Seeking approvals/reviews for: > > > > smoke: PASSED > > > > rados: Radek, Laura, Afreen, Travis > > rgw: Adam Emerson > > fs: Venky > > orch/ceph-volume: Adam King & Guillaume > > rbd: Ilya > > krbd: Ilya > > nvmeof: Aviv, Vallari > > > > upgrade: Patrick > > > > upgrade-clients:client-upgrade-reef-tentacle-tentacle: TBD > > upgrade-clients:client-upgrade-squid-tentacle-tentacle: TBD > > > > perf-basic: PASSED > > > > crimson-rados: N/A > > > > powercycle: PASSED > > > > TIA > > -- Adam Emerson https://people.redhat.com/~aemerson/ Principal Software Engineer, IBM Storage they/them/their aemerson(a)redhat.com IRC: Aemerson@{RedHat,OFTC} 0x80F7544B90EDBFB9 E707 86BA 0C1B 62CC 152C 7C12 80F7 544B 90ED BFB9

1 0

Integrating OSDs from a previous Ceph installation
by Jacek Rużyczka 29 May '26

29 May '26

Hi, After being forced to reinstall the O/S on all my Ceph nodes, I'm trying to reactivate the OSDs, whose data lie on separate drives. The OSDs are recognised: mixtile@blade3n3:~$ sudo sudo ceph-volume lvm list [sudo] password for mixtile: ====== osd.1 ======= [block] /dev/ceph-5b8338a0-9246-48f5-9cfd-e1bbfa1cc199/osd-block-ccba3dd 2-0503-440f-b7c2-47e8d8e85253 block device /dev/ceph-5b8338a0-9246-48f5-9cfd-e1bbfa1cc199/o sd-block-ccba3dd2-0503-440f-b7c2-47e8d8e85253 block uuid 21ftde-FdBB-s4DM-Qeq7-TU4R-vjY7-6VXLoW cephx lockbox secret cluster fsid 8aad3073-39a1-11f1-bf6e-f2704a1efa9b cluster name ceph crush device class encrypted 0 osd fsid ccba3dd2-0503-440f-b7c2-47e8d8e85253 osd id 1 osdspec affinity all-available-devices type block vdo 0 with tpm 0 devices /dev/nvme0n1 *But: *When trying to link the OSD to the all-new Ceph installation (I've made several Google searches with contradictory results. This is the last one.), I get an error: mixtile@blade3n3:~$ sudo cephadm shell Inferring fsid 98e04296-5b5a-11f1-84cf-ceccf52b4a0f Inferring config /var/lib/ceph/98e04296-5b5a-11f1-84cf-ceccf52b4a0f/mon.blade3n3 /config Not using image 'sha256:af0c5903e901e329adabe219dfc8d0c3efc1f05102a753902f33ee16 c26b6cee' as it's not in list of non-dangling images with ceph=True label Unable to find image 'quay.io/ceph/ceph:v19' locally v19: Pulling from ceph/ceph Digest: sha256:af0c5903e901e329adabe219dfc8d0c3efc1f05102a753902f33ee16c26b6cee Status: Downloaded newer image for quay.io/ceph/ceph:v19 root@blade3n3:/usr/bin# cd root@blade3n3:~# ceph-volume lvm activate --all Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph-authtool --gen-print-key --> Activating OSD ID 1 FSID ccba3dd2-0503-440f-b7c2-47e8d8e85253 Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-1 Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-5b8338a0-9246-48f5-9cfd-e1bbfa1cc199/osd-block-ccba3dd2-0503-440f-b7c2-47e8d8e85253 --path /var/lib/ceph/osd/ceph-1 --no-mon-config stderr: 2026-05-29T17:28:36.854+0000 ffffbd2de040 -1 bdev(0xaaab21623800 /dev/c eph-5b8338a0-9246-48f5-9cfd-e1bbfa1cc199/osd-block-ccba3dd2-0503-440f-b7c2-47e8d 8e85253) open stat got: (1) Operation not permitted stderr: failed to read label for /dev/ceph-5b8338a0-9246-48f5-9cfd-1bbfa1cc199 /osd-block-ccba3dd2-0503-440f-b7c2-47e8d8e85253: (1) Operation not permitted --> RuntimeError: command returned non-zero exit status: 1 This puzzles me because I'm already root, but the Bluestore tool won't let me reactivate the OSD? The old OSD is now full of valuable production data. And: Isn't it possible to use ceph orch to perform this operation? Regards Jacek Rużyczka

1 0