[18.2.4 Reef] Using RAID1 as Metadata Device in Ceph – Risks and Recommendations
Hi all! I’ve modified Ceph to allow using a RAID1 array as the metadata device. Specifically, I've updated ceph_volume/util/device.py and ceph_volume/util/disk.py to recognize the "raid1" device type as valid device for OSDs and metadata storage. The changes are listed below. My questions: - What are the risks or issues which may arise from using RAID1 for metadata? (e.g., performance, reliability, data integrity) - Why this could be a bad idea, or in what scenarios could it be beneficial? My test setup: - Ceph version: 18.2.4 Reef (stable) - OS: Ubuntu 22.04.5 LTS - Hardware: 1 host, RAID1 array on two NVMe SSDs (/dev/nvme0n1 and /dev/nvme0n2, 19.98 GiB) via mdadm for block.db, 4 HDDs for OSD data - Additional details: Tested on a small cluster on VMs; no noticeable performance changes so far, but I’m concerned about long-term implications. Code changes: diff --git a/src/ceph-volume/ceph_volume/util/device.py b/src/ceph-volume/ceph_volume/util/device.py index 1b52774d1a1..148a8e326f8 100644 --- a/src/ceph-volume/ceph_volume/util/device.py +++ b/src/ceph-volume/ceph_volume/util/device.py @@ -237,7 +237,7 @@ class Device(object): self.disk_api = dev device_type = dev.get('TYPE', '') # always check is this is an lvm member - valid_types = ['part', 'disk', 'mpath'] + valid_types = ['part', 'disk', 'mpath', 'raid1'] if allow_loop_devices(): valid_types.append('loop') if device_type in valid_types: @@ -489,7 +489,7 @@ class Device(object): elif self.blkid_api: api = self.blkid_api if api: - valid_types = ['disk', 'device', 'mpath'] + valid_types = ['disk', 'device', 'mpath', 'raid1'] if allow_loop_devices(): valid_types.append('loop') return self.device_type in valid_types diff --git a/src/ceph-volume/ceph_volume/util/disk.py b/src/ceph-volume/ceph_volume/util/disk.py index 2984c391d06..5dafd6dc62a 100644 --- a/src/ceph-volume/ceph_volume/util/disk.py +++ b/src/ceph-volume/ceph_volume/util/disk.py @@ -362,7 +362,7 @@ def is_device(dev): TYPE = lsblk(dev).get('TYPE') if TYPE: - return TYPE in ['disk', 'mpath'] + return TYPE in ['disk', 'mpath', 'raid1'] # fallback to stat return _stat_is_device(os.lstat(dev).st_mode) and not is_partition(dev) Any feedback is really appreciated as all of us care about not to lost our precious bytes :)
I’ve modified Ceph to allow using a RAID1 array as the metadata device. Specifically, I've updated ceph_volume/util/device.py and ceph_volume/util/disk.py to recognize the "raid1" device type as valid device for OSDs and metadata storage.
IMHO this isn't the right layer for this. An admin wishing to mirror the offload device should do so (via MD or (sigh) an HBA) and present that device in the OSD spec. ymmv.
The changes are listed below.
My questions: - What are the risks or issues which may arise from using RAID1 for metadata? (e.g., performance, reliability, data integrity) - Why this could be a bad idea, or in what scenarios could it be beneficial?
Conventional wisdom has favored instead offloading fewer OSDs to each SSD to reduce write amp and the blast radius.
My test setup:
- Ceph version: 18.2.4 Reef (stable) - OS: Ubuntu 22.04.5 LTS - Hardware: 1 host, RAID1 array on two NVMe SSDs (/dev/nvme0n1 and /dev/nvme0n2, 19.98 GiB)
That's an unusual size, and as you write, that appears to be an MD array of two namespaces on the same NVMe device. Is that what you intended? What SSD model is this? Is it serving purposes other than OSD WAL+DB offload?
via mdadm for block.db, 4 HDDs for OSD data - Additional details: Tested on a small cluster on VMs; no noticeable performance changes so far, but I’m concerned about long-term implications.
Honestly IMHO the economics and hassle would favor an all-NVMe chassis and monolithic NVMe OSDs. Especially when you consider the cost of the HBA.
Code changes:
diff --git a/src/ceph-volume/ceph_volume/util/device.py b/src/ceph-volume/ceph_volume/util/device.py index 1b52774d1a1..148a8e326f8 100644 --- a/src/ceph-volume/ceph_volume/util/device.py +++ b/src/ceph-volume/ceph_volume/util/device.py @@ -237,7 +237,7 @@ class Device(object): self.disk_api = dev device_type = dev.get('TYPE', '') # always check is this is an lvm member - valid_types = ['part', 'disk', 'mpath'] + valid_types = ['part', 'disk', 'mpath', 'raid1'] if allow_loop_devices(): valid_types.append('loop') if device_type in valid_types:
@@ -489,7 +489,7 @@ class Device(object): elif self.blkid_api: api = self.blkid_api if api: - valid_types = ['disk', 'device', 'mpath'] + valid_types = ['disk', 'device', 'mpath', 'raid1'] if allow_loop_devices(): valid_types.append('loop') return self.device_type in valid_types
diff --git a/src/ceph-volume/ceph_volume/util/disk.py b/src/ceph-volume/ceph_volume/util/disk.py index 2984c391d06..5dafd6dc62a 100644 --- a/src/ceph-volume/ceph_volume/util/disk.py +++ b/src/ceph-volume/ceph_volume/util/disk.py @@ -362,7 +362,7 @@ def is_device(dev):
TYPE = lsblk(dev).get('TYPE') if TYPE: - return TYPE in ['disk', 'mpath'] + return TYPE in ['disk', 'mpath', 'raid1']
# fallback to stat return _stat_is_device(os.lstat(dev).st_mode) and not is_partition(dev)
Any feedback is really appreciated as all of us care about not to lost our precious bytes :) _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
IMHO this isn't the right layer for this. An admin wishing to mirror the offload device should do so (via MD or (sigh) an HBA) and present that device in the OSD spec. ymmv.
Hello Anthony! The idea behind is to keep osds (with data on HDD) running in case of meta device goes down. For sure, my test lab with md Raid1 on 2 NS within the same NVME device is just to bring the idea and it has no sense at all in real world. In prod I mean to use, for instance, nvme0n1 and nvme1n1 united into raid1. And the problem is that ceph-volume tries to get blkid, which is obviosly none on /dev/md127. That is my intention to modify the code.
Conventional wisdom has favored instead offloading fewer OSDs to each SSD to reduce write amp and the blast radius.
Indeed, but I use fast SSD devices for wal and db. BTW, don't you like HBA's? Cuz you did sigh when mentioned HBA?:) Why? Cuz of price?
IMHO this isn't the right layer for this. An admin wishing to mirror the offload device should do so (via MD or (sigh) an HBA) and present that device in the OSD spec. ymmv.
Hello Anthony! The idea behind is to keep osds (with data on HDD) running in case of meta device goes down.
Of course, but remember that Ceph stores data across multiple OSDs and hosts for just that reason. This isn't IMHO the most effective place to spend money. You're effectively doing RAID on RAID.
For sure, my test lab with md Raid1 on 2 NS within the same NVME device is just to bring the idea and it has no sense at all in real world. In prod I mean to use, for instance, nvme0n1 and nvme1n1 united into raid1. And the problem is that ceph-volume tries to get blkid, which is obviosly none on /dev/md127. That is my intention to modify the code.
Conventional wisdom has favored instead offloading fewer OSDs to each SSD to reduce write amp and the blast radius.
Indeed, but I use fast SSD devices for wal and db.
You're still burning the SSD endurance twice as quickly.
BTW, don't you like HBA's? Cuz you did sigh when mentioned HBA?:) Why? Cuz of price?
I've seen a tri-mode RAID HBA in use that had a list price of USD 2000 when it was purchased. All it was doing was mirroring the boot drives, then it failed. The system would have cost less and been more reliable were it all-NVMe with no HBA. RAID HBAs are an anachronism, they're fussy, almost nobody monitors them, and they are money better spent elsewhere.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Of course, but remember that Ceph stores data across multiple OSDs and hosts for just that reason.
Yes, but when nvme with 5 (just fot instance) OSD's db+wal on it goes down - all 5 OSDs go down as well. Keeping in mind, that each of those OSDs are 16TB HDDs, in turn it brings alive massive recovery byteflow. This is what I'd like to avoid. One member of md raid dies - no worries, we have another one which (in theory) should substitute the dead one in-flight.
You're still burning the SSD endurance twice as quickly. Indeed,you are right! But in my opinion it is better to hold additional costs of a couple of new nvme disks instead of making clients to suffer.
The system would have cost less and been more reliable were it all-NVMe with no HBA.
in my case I need thick and cheap cold storage and all-flash cluster cannot beat the price :(
[Disclaimer: AI-assisted answer] There is an interesting quirk in Linux RAID1 implementation in connection to data races, documented in the manual page, https://man7.org/linux/man-pages/man4/md.4.html, under the heading "Scrubbing and mismatches". When the md driver initiates the writeback of a page, it asks both RAID members independently and simultaneously to write that page to their persistent storage. If the page has changed during the writeback (e.g., if disk 0 has serviced the write operation before the data has changed, and disk 1 did so after the change), there is a chance that the two drives might have received different data, thus creating an internal inconsistency. This is common when using RAID1 drives for swap, but in theory, any use of mmap(2) can also trigger this, especially in cases of writeback triggered by memory pressure as opposed to an explicit msync(). This is relevant, as RocksDB uses mmap(). It usually means (at least for the md driver) that the software does not care which of the copies is correct, as it changed the data in flight. Yet, it might cause non-repeatable reads afterward, as some read operation can be serviced from disk 0 and some from disk 1. If the RAID array does not have a write-intent bitmap (the upstream default is to have an interactive recommendation to create it, with "N" being the default answer), such inconsistencies can survive indefinitely after a power failure. I don't know whether RocksDB has an implicit assumption that rereads of data that was being written during a power failure yield the same results. If you want to avoid this effect, please use RAID5, not RAID1. You still can create RAID5 with two disks. The difference is that, unlike RAID1, the kernel will make a copy of the to-be-written memory area, and then write that copy (guaranteed to be stable during the write operation) to the disks. On Thu, Sep 11, 2025 at 5:19 PM Alex from North <service.plant@ya.ru> wrote:
Of course, but remember that Ceph stores data across multiple OSDs and hosts for just that reason.
Yes, but when nvme with 5 (just fot instance) OSD's db+wal on it goes down - all 5 OSDs go down as well. Keeping in mind, that each of those OSDs are 16TB HDDs, in turn it brings alive massive recovery byteflow. This is what I'd like to avoid. One member of md raid dies - no worries, we have another one which (in theory) should substitute the dead one in-flight.
You're still burning the SSD endurance twice as quickly. Indeed,you are right! But in my opinion it is better to hold additional costs of a couple of new nvme disks instead of making clients to suffer.
The system would have cost less and been more reliable were it all-NVMe with no HBA.
in my case I need thick and cheap cold storage and all-flash cluster cannot beat the price :( _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Alexander Patrakov
participants (3)
-
Alex from North
-
Alexander Patrakov
-
Anthony D'Atri