Paul, unfortunately there is no way to recover an OSD which lost its DB volume. All the metadata is kept on this volume so it's impossible to recover object structure after such a failure. Some user data pieces could be found on main device but they're completely unstructured. Not to mention that some Ceph apps (e.g. RGW or CephFS) depend severely on OMAP metadata which is kept purely in DB. And yes - adding new DB volume implies legacy DB data is still accessible, e.g. original DB data collocates user one at main device. But once you had dedicated DB volume and it's lost that's not the case any more. Thanks, Igor On 23.04.2025 14:29, Paul Browne wrote:

Hi Igor,

Ah OK; So in the failure mode of, for instance, 1 failed dedicated SSD/NVMe based dedicated DB devices serving 1 OSD, is there going to be no real way to replace that failed DB device alone without also replacing/wiping the OSD and then having the OSD back-fill objects again from cluster replicas?

I had been looking into e.g.

ceph-bluestore-tool <https://docs.ceph.com/en/latest/man/8/ceph-bluestore-tool/> bluefs-bdev-new-db <https://docs.ceph.com/en/latest/man/8/ceph-bluestore-tool/>

, which seemed to offer some way forward around this.

But it may be the case that this only offers a way forward to replace an older/smaller/slower but still working DB device to a replacement/larger/faster one. And if the old DB device is actually failed and un-usable, then the whole DB+OSD needs replacing?

Here we've lost too many DB devices and by extension too many blocked OSDs to ever recover this Ceph cluster, I fear...

Thanks, Paul

******************* Paul Browne Research Computing Platforms University Information Services Roger Needham Building JJ Thompson Avenue University of Cambridge Cambridge United Kingdom E-Mail: pfb29@cam.ac.uk <mailto:pfb29@cam.ac.uk> Tel: 0044-1223-746548 ******************* ------------------------------------------------------------------------ *From:* Igor Fedotov <igor.fedotov@croit.io> *Sent:* 23 April 2025 09:35 *To:* Paul Browne <pfb29@cam.ac.uk>; ceph-users@ceph.io <ceph-users@ceph.io> *Subject:* Re: [ceph-users] Cluster recovery: DC power failure killed OSD node BlueStore block.DB devices Hi Paul,

have I got your idea correct - you're trying to attach new empty DB volumes to existing OSDs in an attempt to recover these OSDs, right? And original SSD drives which kept DB volumes are physically dead?

If so than IMO it's a way to nowhere, "recovered" OSDs wouldn't run without original metadata. This is rather a waste of time..

Thanks,

Igor

...

Hi ceph-users,

We recently suffered a total power failure at our main DC; fortunately, our production Ceph cluster emerged unscathed but a smaller Ceph cluster came back with the majority of its dedicated SSD BlueStore block.DB devices dead (but its HDD OSD devices unharmed). This cluster underpinned a small OpenStack cloud, so it would be

On 23.04.2025 0:11, Paul Browne wrote: preferable to recover it rather than writing it off.

...

In terms of deployment tooling, this ailing Ceph cluster is a fairly

standard Red Hat Ceph Storage 7 (so Quincy) cephadm deployed cluster, with the main wrinkle about it being that both the DB and HDD OSD devices make use of the cephadm supported LVM->LUKS layering above the BlueStore devices.

...

The dead BlueStore block.DB devices are of course blocking the

surviving HDD OSD daemons (in cephadm deployed containers) from coming up cleanly and so the Ceph cluster status is currently very degraded (attached status for the ugly picture)

...

I've kicked around some ideas of recovering the dead DB devices and

restarting down+out OSDs by;

...

  * Manually partitioning replacement SSDs into new DB device

partitions+LVs

...

   * Installing the same LUKS keys on them retrieved from the Ceph config DB, matching up against which OSD is on which OSD host.

    * Manually changing over device links for OSDs to their DB device with "ceph-bluestore-tool bluefs-bdev-new-db" or similar     * Try restarting OSDs with updated links to new LVM->LUKS->block.DB devices

This approach seems highly messy and subject to needing to extract a lot of information error-free from dumps of ceph volume lvm, list in order to exactly match extant OSD UUIDs to newly created DB devicemapper devices.

Is there going to be some smarter/better/faster way to non-destructively recover these intact HDD OSDs which have links to dead block.DB devices, using native cephadm tooling rather than getting so low-level as all the above?

Many thanks for any advice,

******************* Paul Browne Research Computing Platforms University Information Services Roger Needham Building JJ Thompson Avenue University of Cambridge Cambridge United Kingdom E-Mail: pfb29@cam.ac.uk<mailto:pfb29@cam.ac.uk <mailto:pfb29@cam.ac.uk>> Tel: 0044-1223-746548 ******************* _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Hi Frédéric, The failed SSD disks seem to be quite dead unfortunately, not visible to the OS and also marked as dead in the node iDRAC BMC. We haven't tried moving them to a different node to test though, I can try that. In this power event we lost all of the SSD devices on 2 out of 3 OSD nodes in the cluster (it was a small testing cluster) and half of them on the 3rd OSD node. So the vast majority of OSDs can't start here and the overall cluster state is extremely degraded. So if there is state contained within the old, dead DB devices that can't be directly replaced with the instantiation of new replacement DB devices, then it's looking like we've just lost too many DB devices in one foul swoop to ever recover this Ceph cluster, despite the OSD HDDs all being clean+untouched by the power event. I had been hoping that the DB state was more ephemeral than it seems to be, and so instantiation of new DB devices mapped to the correct OSD devices (via LUKS key) would allow for restarting the down+out OSD devices. But that's increasingly looking to not be possible, from updates on this thread. ******************* Paul Browne Research Computing Platforms University Information Services Roger Needham Building JJ Thompson Avenue University of Cambridge Cambridge United Kingdom E-Mail: pfb29@cam.ac.uk<mailto:pfb29@cam.ac.uk> Tel: 0044-1223-746548 ******************* ________________________________ From: Frédéric Nass <frederic.nass@univ-lorraine.fr> Sent: 23 April 2025 11:24 To: Paul Browne <pfb29@cam.ac.uk> Cc: ceph-users <ceph-users@ceph.io> Subject: Re: [ceph-users] Cluster recovery: DC power failure killed OSD node BlueStore block.DB devices Hi Paul, Could you provide more details about the 'SSD BlueStore block.DB devices dead' issue? Are these devices not seen or seen as defective at the hardware level (through iLO, iDrac, etc.)? Or are they visible to the operating system but their associated OSDs are failing to start? If you can't bring these RocksDB devices back online, associated OSDs will be permanently dead. Regards, Frédéric. ----- Le 22 Avr 25, à 23:11, Paul Browne pfb29@cam.ac.uk a écrit :

Hi ceph-users,

We recently suffered a total power failure at our main DC; fortunately, our production Ceph cluster emerged unscathed but a smaller Ceph cluster came back with the majority of its dedicated SSD BlueStore block.DB devices dead (but its HDD OSD devices unharmed). This cluster underpinned a small OpenStack cloud, so it would be preferable to recover it rather than writing it off.

In terms of deployment tooling, this ailing Ceph cluster is a fairly standard Red Hat Ceph Storage 7 (so Quincy) cephadm deployed cluster, with the main wrinkle about it being that both the DB and HDD OSD devices make use of the cephadm supported LVM->LUKS layering above the BlueStore devices.

The dead BlueStore block.DB devices are of course blocking the surviving HDD OSD daemons (in cephadm deployed containers) from coming up cleanly and so the Ceph cluster status is currently very degraded (attached status for the ugly picture)

I've kicked around some ideas of recovering the dead DB devices and restarting down+out OSDs by;

  * Manually partitioning replacement SSDs into new DB device partitions+LVs    * Installing the same LUKS keys on them retrieved from the Ceph config DB,    matching up against which OSD is on which OSD host.

* Manually changing over device links for OSDs to their DB device with "ceph-bluestore-tool bluefs-bdev-new-db" or similar * Try restarting OSDs with updated links to new LVM->LUKS->block.DB devices

This approach seems highly messy and subject to needing to extract a lot of information error-free from dumps of ceph volume lvm, list in order to exactly match extant OSD UUIDs to newly created DB devicemapper devices.

Is there going to be some smarter/better/faster way to non-destructively recover these intact HDD OSDs which have links to dead block.DB devices, using native cephadm tooling rather than getting so low-level as all the above?

Many thanks for any advice,

******************* Paul Browne Research Computing Platforms University Information Services Roger Needham Building JJ Thompson Avenue University of Cambridge Cambridge United Kingdom E-Mail: pfb29@cam.ac.uk<mailto:pfb29@cam.ac.uk> Tel: 0044-1223-746548 ******************* _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Hi Paul, I'm continuing this thread here, following Anthony's insightful remarks and offer to help. I find it hard to believe that enterprise-grade NVMe drives would fail during a power outage, unless there's an issue with the NVMe or HBA firmware. I recommend opening a support case with DELL, HPE, or whatever manufacturer made your server. Before doing that, try these troubleshooting steps: - Shut down the server completely - Disconnect all power cables for at least 10 minutes - Restart the server (this might resolve temporary discovery issues during boot) If the drives reappear during startup, you may need to 'import' them in the boot process. Watch for a message on the console prompting you to do this. If these steps don't help, try upgrading all firmware on the server. I've seen 'dead' DELL SSDs (Toshiba) come back to life after a firmware upgrade, even when marked as dead in iDrac. See [1] for details. Ultimately, your best course of action is to open a support case with your hardware manufacturer. Regards, Frédéric. [1] https://www.spinics.net/lists/ceph-users/msg78647.html ----- Le 23 Avr 25, à 15:37, Anthony D'Atri anthony.datri@gmail.com a écrit :

...

The failed SSD disks seem to be quite dead unfortunately, not visible to the OS and also marked as dead in the node iDRAC BMC.

I’ve found that iDRAC’s view of SSDs is sometimes … imperfect, but not visible to the OS is telling.

If you could send me

storcli64 /c0 show termlog >/var/tmp/termlog.txt # or perccli64 storcli64 /c0 show all

I’d love to take a look and see if the HBA has any additional information.

One possible though unlikely scenario is that the lost drives had a firmware flaw but surviving drives had a newer revision

...

We haven't tried moving them to a different node to test though, I can try that.

In this power event we lost all of the SSD devices on 2 out of 3 OSD nodes in the cluster (it was a small testing cluster) and half of them on the 3rd OSD node.

So the vast majority of OSDs can't start here and the overall cluster state is extremely degraded.

So if there is state contained within the old, dead DB devices that can't be directly replaced with the instantiation of new replacement DB devices, then it's looking like we've just lost too many DB devices in one foul swoop to ever recover this Ceph cluster, despite the OSD HDDs all being clean+untouched by the power event.

I had been hoping that the DB state was more ephemeral than it seems to be, and so instantiation of new DB devices mapped to the correct OSD devices (via LUKS key) would allow for restarting the down+out OSD devices. But that's increasingly looking to not be possible, from updates on this thread.

******************* Paul Browne Research Computing Platforms University Information Services Roger Needham Building JJ Thompson Avenue University of Cambridge Cambridge United Kingdom E-Mail: pfb29@cam.ac.uk<mailto:pfb29@cam.ac.uk> Tel: 0044-1223-746548 ******************* ________________________________ From: Frédéric Nass <frederic.nass@univ-lorraine.fr> Sent: 23 April 2025 11:24 To: Paul Browne <pfb29@cam.ac.uk> Cc: ceph-users <ceph-users@ceph.io> Subject: Re: [ceph-users] Cluster recovery: DC power failure killed OSD node BlueStore block.DB devices

Hi Paul,

Could you provide more details about the 'SSD BlueStore block.DB devices dead' issue?

Are these devices not seen or seen as defective at the hardware level (through iLO, iDrac, etc.)? Or are they visible to the operating system but their associated OSDs are failing to start? If you can't bring these RocksDB devices back online, associated OSDs will be permanently dead.

Regards, Frédéric.

----- Le 22 Avr 25, à 23:11, Paul Browne pfb29@cam.ac.uk a écrit :

...

Hi ceph-users,

We recently suffered a total power failure at our main DC; fortunately, our production Ceph cluster emerged unscathed but a smaller Ceph cluster came back with the majority of its dedicated SSD BlueStore block.DB devices dead (but its HDD OSD devices unharmed). This cluster underpinned a small OpenStack cloud, so it would be preferable to recover it rather than writing it off.

In terms of deployment tooling, this ailing Ceph cluster is a fairly standard Red Hat Ceph Storage 7 (so Quincy) cephadm deployed cluster, with the main wrinkle about it being that both the DB and HDD OSD devices make use of the cephadm supported LVM->LUKS layering above the BlueStore devices.

The dead BlueStore block.DB devices are of course blocking the surviving HDD OSD daemons (in cephadm deployed containers) from coming up cleanly and so the Ceph cluster status is currently very degraded (attached status for the ugly picture)

I've kicked around some ideas of recovering the dead DB devices and restarting down+out OSDs by;

  * Manually partitioning replacement SSDs into new DB device partitions+LVs    * Installing the same LUKS keys on them retrieved from the Ceph config DB,    matching up against which OSD is on which OSD host.

* Manually changing over device links for OSDs to their DB device with "ceph-bluestore-tool bluefs-bdev-new-db" or similar * Try restarting OSDs with updated links to new LVM->LUKS->block.DB devices

This approach seems highly messy and subject to needing to extract a lot of information error-free from dumps of ceph volume lvm, list in order to exactly match extant OSD UUIDs to newly created DB devicemapper devices.

Is there going to be some smarter/better/faster way to non-destructively recover these intact HDD OSDs which have links to dead block.DB devices, using native cephadm tooling rather than getting so low-level as all the above?

Many thanks for any advice,

******************* Paul Browne Research Computing Platforms University Information Services Roger Needham Building JJ Thompson Avenue University of Cambridge Cambridge United Kingdom E-Mail: pfb29@cam.ac.uk<mailto:pfb29@cam.ac.uk> Tel: 0044-1223-746548 ******************* _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

I have seen drives work absolutely fine past their "100% used" indicator in smart, but on a power cycle they flat out refuse to enumerate on the bus. The team that ran into this was lucky enough to catch it on one machine first so they could grab the data before rebooting the other hosts. This was also many years ago so I hope their firmware does something different now.

In recent years the trend is to NOT hard-disable at the end of rated PE cycles, but this is still on a SKU-by-SKU basis.

What does the health look like on the remaining drives? How long were the dead ones in service?

That SKU is from 2018, but was rated at 10DWPD so I suspect it’s not a lifetime issue as such, perhaps firmware.

-paul

--

Paul Mezzanini Platform Engineer III Research Computing Rochester Institute of Technology

________________________________________ From: Frédéric Nass <frederic.nass@univ-lorraine.fr> Sent: Wednesday, April 23, 2025 10:09 AM To: Paul Browne Cc: ceph-users; Anthony D'Atri Subject: [ceph-users] Re: Cluster recovery: DC power failure killed OSD node BlueStore block.DB devices

Hi Paul,

I'm continuing this thread here, following Anthony's insightful remarks and offer to help.

I find it hard to believe that enterprise-grade NVMe drives would fail during a power outage, unless there's an issue with the NVMe or HBA firmware. I recommend opening a support case with DELL, HPE, or whatever manufacturer made your server.

Before doing that, try these troubleshooting steps:

- Shut down the server completely - Disconnect all power cables for at least 10 minutes - Restart the server (this might resolve temporary discovery issues during boot)

If the drives reappear during startup, you may need to 'import' them in the boot process. Watch for a message on the console prompting you to do this. If these steps don't help, try upgrading all firmware on the server.

I've seen 'dead' DELL SSDs (Toshiba) come back to life after a firmware upgrade, even when marked as dead in iDrac. See [1] for details.

Ultimately, your best course of action is to open a support case with your hardware manufacturer.

Regards, Frédéric.

[1] https://www.spinics.net/lists/ceph-users/msg78647.html

----- Le 23 Avr 25, à 15:37, Anthony D'Atri anthony.datri@gmail.com a écrit :

...

...

The failed SSD disks seem to be quite dead unfortunately, not visible to the OS and also marked as dead in the node iDRAC BMC.

I’ve found that iDRAC’s view of SSDs is sometimes … imperfect, but not visible to the OS is telling.

If you could send me

storcli64 /c0 show termlog >/var/tmp/termlog.txt # or perccli64 storcli64 /c0 show all

I’d love to take a look and see if the HBA has any additional information.

One possible though unlikely scenario is that the lost drives had a firmware flaw but surviving drives had a newer revision

...

We haven't tried moving them to a different node to test though, I can try that.

In this power event we lost all of the SSD devices on 2 out of 3 OSD nodes in the cluster (it was a small testing cluster) and half of them on the 3rd OSD node.

So the vast majority of OSDs can't start here and the overall cluster state is extremely degraded.

So if there is state contained within the old, dead DB devices that can't be directly replaced with the instantiation of new replacement DB devices, then it's looking like we've just lost too many DB devices in one foul swoop to ever recover this Ceph cluster, despite the OSD HDDs all being clean+untouched by the power event.

I had been hoping that the DB state was more ephemeral than it seems to be, and so instantiation of new DB devices mapped to the correct OSD devices (via LUKS key) would allow for restarting the down+out OSD devices. But that's increasingly looking to not be possible, from updates on this thread.

******************* Paul Browne Research Computing Platforms University Information Services Roger Needham Building JJ Thompson Avenue University of Cambridge Cambridge United Kingdom E-Mail: pfb29@cam.ac.uk<mailto:pfb29@cam.ac.uk> Tel: 0044-1223-746548 ******************* ________________________________ From: Frédéric Nass <frederic.nass@univ-lorraine.fr> Sent: 23 April 2025 11:24 To: Paul Browne <pfb29@cam.ac.uk> Cc: ceph-users <ceph-users@ceph.io> Subject: Re: [ceph-users] Cluster recovery: DC power failure killed OSD node BlueStore block.DB devices

Hi Paul,

Could you provide more details about the 'SSD BlueStore block.DB devices dead' issue?

Are these devices not seen or seen as defective at the hardware level (through iLO, iDrac, etc.)? Or are they visible to the operating system but their associated OSDs are failing to start? If you can't bring these RocksDB devices back online, associated OSDs will be permanently dead.

Regards, Frédéric.

----- Le 22 Avr 25, à 23:11, Paul Browne pfb29@cam.ac.uk a écrit :

...

Hi ceph-users,

We recently suffered a total power failure at our main DC; fortunately, our production Ceph cluster emerged unscathed but a smaller Ceph cluster came back with the majority of its dedicated SSD BlueStore block.DB devices dead (but its HDD OSD devices unharmed). This cluster underpinned a small OpenStack cloud, so it would be preferable to recover it rather than writing it off.

In terms of deployment tooling, this ailing Ceph cluster is a fairly standard Red Hat Ceph Storage 7 (so Quincy) cephadm deployed cluster, with the main wrinkle about it being that both the DB and HDD OSD devices make use of the cephadm supported LVM->LUKS layering above the BlueStore devices.

The dead BlueStore block.DB devices are of course blocking the surviving HDD OSD daemons (in cephadm deployed containers) from coming up cleanly and so the Ceph cluster status is currently very degraded (attached status for the ugly picture)

I've kicked around some ideas of recovering the dead DB devices and restarting down+out OSDs by;

  * Manually partitioning replacement SSDs into new DB device partitions+LVs    * Installing the same LUKS keys on them retrieved from the Ceph config DB,    matching up against which OSD is on which OSD host.

* Manually changing over device links for OSDs to their DB device with "ceph-bluestore-tool bluefs-bdev-new-db" or similar * Try restarting OSDs with updated links to new LVM->LUKS->block.DB devices

This approach seems highly messy and subject to needing to extract a lot of information error-free from dumps of ceph volume lvm, list in order to exactly match extant OSD UUIDs to newly created DB devicemapper devices.

Is there going to be some smarter/better/faster way to non-destructively recover these intact HDD OSDs which have links to dead block.DB devices, using native cephadm tooling rather than getting so low-level as all the above?

Many thanks for any advice,

******************* Paul Browne Research Computing Platforms University Information Services Roger Needham Building JJ Thompson Avenue University of Cambridge Cambridge United Kingdom E-Mail: pfb29@cam.ac.uk<mailto:pfb29@cam.ac.uk> Tel: 0044-1223-746548 ******************* _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

---

ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io