- ceph-users - lists.ceph.io

Re: Recommended approach for hybrid devices
by Anthony D'Atri 20 Mar '26

20 Mar '26

> >> My understanding is a manual device add approach, while possible, removes >> the benefits of orch (creating "unmanaged" devices) and should be avoided. >> But would it even be possible for orch to be fed a yaml with the NVMes >> operating as both a data and db device? > > Check if the NVMe devices support multiple namespaces: Don't they all? > > https://narasimhan-v.github.io/2020/06/12/Managing-NVMe-Namespaces.html > > This way the kernel sees two block devices for each NVMe and the orchestrator is able to provision an OSD on one and the RocksDBs for the hybrid OSDs on the other. I've wondered for years how namespaces in pragmatic terms differ from partitions. > > Regards > -- > Robert Sander > Linux Consultant > > Heinlein Consulting GmbH > Schwedter Str. 8/9b, 10119 Berlin > > https://www.heinlein-support.de > > Tel: +49 30 405051 - 0 > Fax: +49 30 405051 - 19 > > Amtsgericht Berlin-Charlottenburg - HRB 220009 B > Geschäftsführer: Peer Heinlein - Sitz: Berlin > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Re: Recommended approach for hybrid devices
by Robert Sander 20 Mar '26

20 Mar '26

Hi, Am 19.03.26 um 10:18 AM schrieb Ryan Sleeth via ceph-users: > My understanding is a manual device add approach, while possible, removes > the benefits of orch (creating "unmanaged" devices) and should be avoided. > But would it even be possible for orch to be fed a yaml with the NVMes > operating as both a data and db device? Check if the NVMe devices support multiple namespaces: https://narasimhan-v.github.io/2020/06/12/Managing-NVMe-Namespaces.html This way the kernel sees two block devices for each NVMe and the orchestrator is able to provision an OSD on one and the RocksDBs for the hybrid OSDs on the other. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

Re: tentacle 20.2.1 QE validation status
by Vallari Agrawal 20 Mar '26

20 Mar '26

nvmeof passed based on https://pulpito.ceph.com/yuriw-2026-03-06_16:22:59-nvmeof-tentacle-release-… (which does not include revert PR https://github.com/ceph/ceph/pull/67750, please let me know if we need to re-review after it's merged) thanks, Vallari On Tue, Mar 17, 2026 at 2:28 AM Yuri Weinstein via Dev <dev(a)ceph.io> wrote: > Only dashboard, rbd and krbd were approved. > > Patrick is working on https://github.com/ceph/ceph/pull/67750 > <https://qcflzmq6.r.us-east-1.awstrack.me/L0/https:%2F%2Fgithub.com%2Fceph%2…> > revert > > I also added tentacle-p2p run and it needs reviews. > > Dev leads, pls review/approve the run results. > > On Mon, Mar 9, 2026 at 11:44 AM Yuri Weinstein <yweinste(a)redhat.com> > wrote: > >> Details of this release are summarized here: >> >> https://tracker.ceph.com/issues/74838#note-4 >> <https://qcflzmq6.r.us-east-1.awstrack.me/L0/https:%2F%2Ftracker.ceph.com%2F…> >> >> Release Notes - TBD >> LRC upgrade - TBD >> >> Seeking approvals/reviews for: >> >> smoke: PASSED >> >> rados: Radek, Laura, Afreen, Travis >> rgw: Adam Emerson >> fs: Venky >> orch/ceph-volume: Adam King & Guillaume >> rbd: Ilya >> krbd: Ilya (still running) >> nvmeof: Aviv, Vallari >> >> upgrade/squid-x: Laura >> upgrade/reef-x: Laura >> upgrade/telemetry-upgrade: Laura >> >> upgrade/tentacle-p2p: N/A >> >> upgrade-clients:client-upgrade-reef-tentacle-tentacle: PASSED >> upgrade-clients:client-upgrade-squid-tentacle-tentacle: PASSED >> >> perf-basic: PASSED >> >> crimson-rados: N/A >> >> powercycle: PASSED >> >> TIA >> > _______________________________________________ > Dev mailing list -- dev(a)ceph.io > To unsubscribe send an email to dev-leave(a)ceph.io >

1 0

Re: Recommended approach for hybrid devices
by Anthony D'Atri 20 Mar '26

20 Mar '26

> I have a test Tentacle cluster that is 9 nodes each with 8x 20 TB HDDs and > 2x 2 TB enterprise NVMes. Ensure the firmware is up to date on each! > My initial plan was to split the NVMes into 5x > 300 GB partitions: 1 partition for data and 4 for db (to be paired with 4 > HDDs creating hybrid devices). My hope was that for performance, the one > NVMe data partition could then be used for 4x replicated SSD-only > metadata pools. What SSD and HDD SKUs exactly please? > The hybrid devices would be for 4+2 EC cephfs. Depending on your workload, you might find that the WAL+DB offload doesn't get you much. > > Previously I had tested this successfully on Squid using a series of manual > device adds, but on Tentacle it is seeing pre-created LVM partitions as in > use by the file system where Squid did not (ceph orch device ls ceph01). > > My understanding is a manual device add approach, while possible, removes > the benefits of orch (creating "unmanaged" devices) and should be avoided. > But would it even be possible for orch to be fed a yaml with the NVMes > operating as both a data and db device? If you specify the paths explicitly, that should work. > The obvious answer is to have a third NVMe per node operating as data for > the 4x replicated metadata pool, but I am not sure if my budget permits > that. What's your file workload like? You might consider making the OSD partition larger and the WAL+DB offload partitions smaller.

2 3

Re: updating RBD client auth parameters
by Eugen Block 19 Mar '26

19 Mar '26

Hi, that's exactly where we landed this week with our own 11 year old cluster. We tried to access a home directory on a fresh openSUSE Leap 16 system via CephFS and failed due to an old auth cap for that user. We did migrate a couple of rbd clients over the years to the "profile rbd" syntax. But we still have some leftovers, so this is gonna take some time. I usually create new clients and compare them (like you did), then take some uncritical client and update its caps. If nobody complains, I adapt it on other clients as well. But I'm not aware of any documented update strategy, and at the same time I expect backwards compatibility for such clients. Our specific issue was an "allow r path=/" statement which MDS rejects if the user is supposed to only have access to a subdirectory. It still works with 6.4 kernel but failed with kernel 6.12. Regards, Eugen Zitat von Jan Kasprzak via ceph-users <ceph-users(a)ceph.io>: > Hello, Ceph users, > > is there a recommended way how to keep ceph users' capabilities up-to-date? > I have a many years old cluster originally based on Hammer, gradually updated > up to Tentacle, and I noticed that users created for rbd access long time ago > according to recommendations of that time differ in their capabilities: > > # ceph auth ls > [...] > client.one > key: XXX...== > caps: [mgr] allow r > caps: [mon] allow r, allow command "osd blacklist" > caps: [osd] allow class-read object_prefix rbd_children, allow rwx pool=one > client.two > key: YYY...== > caps: [mgr] allow r > caps: [mon] allow r > caps: [osd] allow class-read object_prefix rbd_children, allow rwx pool=two > client.three > key: ZZZ...== > caps: [mgr] profile rbd pool=three > caps: [mon] profile rbd > caps: [osd] profile rbd pool=three > > are the above setting equivalent on Tentacle? Can the older users > "client.one" > and "client.two" be safely updated to the same settings as the > "client.three"? > > Thanks! > > -Yenya > > -- > | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | > | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | > I don't like Python; its lack of inline, anonymous, multi-statement > functions makes me sad. --Eric Wastl > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

updating RBD client auth parameters
by Jan Kasprzak 19 Mar '26

19 Mar '26

Hello, Ceph users, is there a recommended way how to keep ceph users' capabilities up-to-date? I have a many years old cluster originally based on Hammer, gradually updated up to Tentacle, and I noticed that users created for rbd access long time ago according to recommendations of that time differ in their capabilities: # ceph auth ls [...] client.one key: XXX...== caps: [mgr] allow r caps: [mon] allow r, allow command "osd blacklist" caps: [osd] allow class-read object_prefix rbd_children, allow rwx pool=one client.two key: YYY...== caps: [mgr] allow r caps: [mon] allow r caps: [osd] allow class-read object_prefix rbd_children, allow rwx pool=two client.three key: ZZZ...== caps: [mgr] profile rbd pool=three caps: [mon] profile rbd caps: [osd] profile rbd pool=three are the above setting equivalent on Tentacle? Can the older users "client.one" and "client.two" be safely updated to the same settings as the "client.three"? Thanks! -Yenya -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | I don't like Python; its lack of inline, anonymous, multi-statement functions makes me sad. --Eric Wastl

1 0

Re: Merging two clusters
by Anthony D'Atri 19 Mar '26

19 Mar '26

I’ve used rbd-mirror to do this sort of thing in the past. It’s straightforward with unattached volumes. I had a homebrew harness that for attached images would handle a reboot of the client with reattach. > On Mar 19, 2026, at 8:24 AM, Nico Schottelius via ceph-users <ceph-users(a)ceph.io> wrote: > > > Hello, > > I was wondering if there is any (recent, in general) method to merge to > clusters. We have several older nautilus based clusters that are > natively deployed. We also have several newer rook/kubernetes/squid > based clusters. > > In each data center we have both, alongside each other. While the > clusters are running in separate networks, we could open the firewalls > between them for a merge/migration. > > My question is, is there any reasonable way of merging an old nautilus > cluster into another existing squid cluster? > > In theory, what would be amazing is something like a fluid process in > which the newer cluster could retrieve items from the older cluster and > once it has them, decommission them from the old cluster. > > The live migration on > https://docs.ceph.com/en/latest/rbd/rbd-live-migration/ seems to go a > bit into the direction, but I am not sure if that works in our case. > > We are looking at migrating around 500 rbd images in total. > > Any hints appreciated! > > Best regards, > > Nico > > -- > Sustainable and modern Infrastructures by ungleich.ch > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Merging two clusters
by Burkhard Linke 19 Mar '26

19 Mar '26

Hi, we used the rbd live migration in the past to split up a cluster. The processed pools were used for image and volumes in our openstack cloud. Live migration mostly works, but if the RBD image is in use, you have to add some extra effort. Cinder for examples stores the mon information in its database, and this has to be updated. I used a script to migrate a RBD image, pause the instance, adjust the database entries, and live migrate the instance itself to another host to update the qemu/libvirt information. If the image is not actively used migration is trivial; if the image is attached to a stopped virtual machine migration is also trivial, but database entries have to be updated. The rather old nautilus clusters might be problem, so if you want to try rbd live migration you need to test this first. Not sure whether the script still exists, but I can try to dig it out if necessary. Best regards, Burkhard Linke On 3/19/26 13:28, Nico Schottelius via ceph-users wrote: > Hello, > > I was wondering if there is any (recent, in general) method to merge to > clusters. We have several older nautilus based clusters that are > natively deployed. We also have several newer rook/kubernetes/squid > based clusters. > > In each data center we have both, alongside each other. While the > clusters are running in separate networks, we could open the firewalls > between them for a merge/migration. > > My question is, is there any reasonable way of merging an old nautilus > cluster into another existing squid cluster? > > In theory, what would be amazing is something like a fluid process in > which the newer cluster could retrieve items from the older cluster and > once it has them, decommission them from the old cluster. > > The live migration on > https://docs.ceph.com/en/latest/rbd/rbd-live-migration/ seems to go a > bit into the direction, but I am not sure if that works in our case. > > We are looking at migrating around 500 rbd images in total. > > Any hints appreciated! > > Best regards, > > Nico > > -- > Sustainable and modern Infrastructures by ungleich.ch > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Merging two clusters
by Nico Schottelius 19 Mar '26

19 Mar '26

Hello, I was wondering if there is any (recent, in general) method to merge to clusters. We have several older nautilus based clusters that are natively deployed. We also have several newer rook/kubernetes/squid based clusters. In each data center we have both, alongside each other. While the clusters are running in separate networks, we could open the firewalls between them for a merge/migration. My question is, is there any reasonable way of merging an old nautilus cluster into another existing squid cluster? In theory, what would be amazing is something like a fluid process in which the newer cluster could retrieve items from the older cluster and once it has them, decommission them from the old cluster. The live migration on https://docs.ceph.com/en/latest/rbd/rbd-live-migration/ seems to go a bit into the direction, but I am not sure if that works in our case. We are looking at migrating around 500 rbd images in total. Any hints appreciated! Best regards, Nico -- Sustainable and modern Infrastructures by ungleich.ch

1 0

Recommended approach for hybrid devices
by Ryan Sleeth 19 Mar '26

19 Mar '26

I have a test Tentacle cluster that is 9 nodes each with 8x 20 TB HDDs and 2x 2 TB enterprise NVMes. My initial plan was to split the NVMes into 5x 300 GB partitions: 1 partition for data and 4 for db (to be paired with 4 HDDs creating hybrid devices). My hope was that for performance, the one NVMe data partition could then be used for 4x replicated SSD-only metadata pools. The hybrid devices would be for 4+2 EC cephfs. Previously I had tested this successfully on Squid using a series of manual device adds, but on Tentacle it is seeing pre-created LVM partitions as in use by the file system where Squid did not (ceph orch device ls ceph01). My understanding is a manual device add approach, while possible, removes the benefits of orch (creating "unmanaged" devices) and should be avoided. But would it even be possible for orch to be fed a yaml with the NVMes operating as both a data and db device? The obvious answer is to have a third NVMe per node operating as data for the 4x replicated metadata pool, but I am not sure if my budget permits that. -- Ryan Sleeth

1 0