- ceph-users - lists.ceph.io

Re: tentacle 20.2.1 QE validation status
by Ilya Dryomov 09 Mar '26

09 Mar '26

On Mon, Mar 9, 2026 at 7:50 PM Yuri Weinstein via Dev <dev(a)ceph.io> wrote: > > Details of this release are summarized here: > > https://tracker.ceph.com/issues/74838#note-4 > > Release Notes - TBD > LRC upgrade - TBD > > Seeking approvals/reviews for: > > smoke: PASSED > > rados: Radek, Laura, Afreen, Travis > rgw: Adam Emerson > fs: Venky > orch/ceph-volume: Adam King & Guillaume > rbd: Ilya Hi Yuri, Could you please do another rerun for rbd? > krbd: Ilya (still running) krbd approved. Thanks, Ilya

1 0

CSC Meeting Notes: 2026-03-09
by Adam Emerson 09 Mar '26

09 Mar '26

Ceph Steering Committee Meeting 9 March 2026 * [Ernesto/AnthonyM]: Github Plan migration to Enterprise Account. - Upgrading Github to enterprise is generally pretty straightforward - The main issue is permissions with github actions - Benjamin Sternthal from the LF can help audit and get things ready. - The main issue is default permissions and needing to update actions to specify them. - [Ilya] We don't have all that many actions, so some temporary breakage probably wouldn't be the end of the world. - [Benjamin Sternthal] Probably the best thing to do is pick a time and work back from there. - The banner from Github suggests it's going to happen automatically eventually, and it's better if we do it on our timeline. - LF Requested the upgrade about a week ago, it usually comes in two weeks, so it should be soon. - #github-enterprise-upgrade is open for anyone wanting to be involved. * [Anthony Middleton] CSC Election https://docs.google.com/document/d/12zmuG6qVWU33dlcR4aXkCG2z4MdAGT8A3F36ofn… - Continuing from previous conversations, this is a planning document for how the election could go. - Lots of questions about voting. - There seems to be a consensus about having up to three co-chairs. If we want to have a vote about that, Anthony's happy and Benjamin Sternthal says there's no LF objection. - This election is happening now because we didn't have an election on 2025, so there will be another election later this year as well. - We could have an election about leadership now, and handle other matters around October/November. - Do we need to move forward with the leadership? Or is there more that needs done soon? - [Greg] We need to clean up our rolls. Anthony has a list of who voted in the 2024/2025 elections. - We can make a list of people who didn't vote in the last two elections and vote about removing them. - [Benjamin Sternthal] Now would be a good time to think about how to run things, conduct votes, etc. - [Dan van der Ster] We could do concrete procedural changes in pull requests, put our procedural documents on github, so we can iterate and vote on them. That's how we did it in the past with our governance rules. - [Matt V] ARIN/RIPE revokes voting for orgs that haven't voted in two years, which sounds similar. - Mark Nelson and Anthony will work on a draft. We've talked about auto-expiration, an annual event where people have to affirm their voting status (but there were concerns the note might be missed), or we vote every year on pruning our rolls. - [Ilya] We have to vote once a year to rotate council members. We could have whoever's organizing that vote propose removal of inactive members (based on some rules) as part of the same ballot. If we have to vote every year anyway, then doing it then will go much more smoothly. - [Dan van der Ster] This is how we did it last time, we just didn't have a specific procedure for determining inactive members. - [Anthony Middleton] Who will represent the CSC to the quarterly meeting of the Ceph Governing Board at the LF? That could be one of the leadership people or any member of the CSC. - [Dan van der Ster] The CEC can decide among themselves who will be the voting representative/single point of contact. - [Benjamin Sternthal] There's a Voting Representative and this organization decides who it is. The whole CSC is invited to the quarterly meetings with the Governing Board. - [Anthony Middleton] So the vote will be about leadership and the removal process for inactive members. Then there will be another election in October or November. * [Mark] Do we have a policy on which QA runs should pass before PRs are merged? continue discussion from last meeting. - Mark remembered Ilya saying there was a policy somewhere. - Ilya qualified that he was saying the general consensus, not a policy. Components scheduling test runs, with merges happening only after test runs are finished and reviewed. - The point was that you can do your own run, but it should be a coordinated process where the relevant component team reviews the results, since not everyone is up to speed on all of the ongoing issues. - [Mark] We know what the expectaiton from the leads is, but don't have anything written down. - [Greg] Someone should just write something up and put up a PR - [Laura Flores] It should probably be component-by-component. Component leads should write down their own process. - Adam E will get it started and the component leads can fill it in. * [Radoslaw] Conclusion on "Should the `reef` branch be opened for merging" question. - Moving to next week since Patrick isn't here * [Anthony Middleton] CSC and Ceph Governing Board Q1 meeting, Monday March 30th 1 pm EDT - [Greg] The agenda looks very long and that is not a one hour meeting - Anthony Middleton will edit the agenda, moderate the call, etc. - This meeting will be led by the governing board this quarter, next quarter the CSC will be. * [Patrick] Shall we archive ceph/ceph-deploy and other vestiges? - Bumped to next week since Patrick isn't here. * [Yaarit] Rocky 10 update - We are working on merging the patches for having Rock 10 in main and the Python 3.12 changes. - Shortly after, we would like to change the base image to be Rocky 10 instead of CentOS 9. - Will have a separate discussion with David G, but wants to ask all the component leads to review and fix the test branches. - We saw an unreasonable amount of failures last week, and we're currently looking into that. - Once everything looks green, we hope to be able to merge the changes. - There are pending issues for CephFS and RBD as well.

1 0

Re: RAW USE in 'df' - no pools
by Janne Johansson 07 Mar '26

07 Mar '26

I think it counts overhead "lost" to preallocated space and databases which occur even before a single pool is in place. Den lör 7 mars 2026 kl 09:28 skrev listy via ceph-users <ceph-users(a)ceph.io>: > > Hi guys. > > What is 'RAW USE' in _df_ while there is no pool using given > _osd_? > Like so: > > -> $ ceph osd df tree | egrep '(osd\.(3|2|4)\ |ID)' > ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA > OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME > 2 MONERO 0.39059 1.00000 400 GiB 660 MiB 14 MiB > 31 KiB 646 MiB 399 GiB 0.16 0.04 0 up > osd.2 > 3 MONERO 0.39059 1.00000 400 GiB 1.5 GiB 15 MiB > 467 KiB 1.4 GiB 399 GiB 0.36 0.08 1 up > osd.3 > 4 MONERO 0.39059 1.00000 400 GiB 567 MiB 15 MiB > 50 KiB 553 MiB 399 GiB 0.14 0.03 1 up > osd.4 > > I have crush rule which is mean to - at least how I get it - > separate osds: cluster is 3x nodes with 150GB + 400GB drives > on each node - in case this matter, separation is suppose to > be between different size drives. > Should _raw use_ & _data_ not be always _zero_? > > many thanks, L. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io -- May the most significant bit of your life be positive.

1 0

Set Up Ceph certmgr CA as SubCA of an internal RootCA
by Kleinknecht_ch＠web.de 07 Mar '26

07 Mar '26

Hello cephusers Does anyone have experience setting up the Ceph certmgr CA as a SubCA of an internal RootCA? I haven't found anything in the documentation yet. Greetings Christoph

1 0

Re: RAW USE in 'df' - no pools
by Eugen Block 07 Mar '26

07 Mar '26

Hi, each OSD has a DB/WAL device which is on the main data device if not separated on faster drives. And since there are two PGs, there is at least one pool, I assume the .mgr pool which is created by default. Regards, Eugen Zitat von listy via ceph-users <ceph-users(a)ceph.io>: > Hi guys. > > What is 'RAW USE' in _df_ while there is no pool using given _osd_? > Like so: > > -> $ ceph osd df tree | egrep '(osd\.(3|2|4)\ |ID)' > ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP > META AVAIL %USE VAR PGS STATUS TYPE NAME > 2 MONERO 0.39059 1.00000 400 GiB 660 MiB 14 MiB 31 KiB > 646 MiB 399 GiB 0.16 0.04 0 up osd.2 > 3 MONERO 0.39059 1.00000 400 GiB 1.5 GiB 15 MiB 467 KiB > 1.4 GiB 399 GiB 0.36 0.08 1 up osd.3 > 4 MONERO 0.39059 1.00000 400 GiB 567 MiB 15 MiB 50 KiB > 553 MiB 399 GiB 0.14 0.03 1 up osd.4 > > I have crush rule which is mean to - at least how I get it - > separate osds: cluster is 3x nodes with 150GB + 400GB drives on each > node - in case this matter, separation is suppose to be between > different size drives. > Should _raw use_ & _data_ not be always _zero_? > > many thanks, L. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

RAW USE in 'df' - no pools
by listy 07 Mar '26

07 Mar '26

Hi guys. What is 'RAW USE' in _df_ while there is no pool using given _osd_? Like so: -> $ ceph osd df tree | egrep '(osd\.(3|2|4)\ |ID)' ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME 2 MONERO 0.39059 1.00000 400 GiB 660 MiB 14 MiB 31 KiB 646 MiB 399 GiB 0.16 0.04 0 up osd.2 3 MONERO 0.39059 1.00000 400 GiB 1.5 GiB 15 MiB 467 KiB 1.4 GiB 399 GiB 0.36 0.08 1 up osd.3 4 MONERO 0.39059 1.00000 400 GiB 567 MiB 15 MiB 50 KiB 553 MiB 399 GiB 0.14 0.03 1 up osd.4 I have crush rule which is mean to - at least how I get it - separate osds: cluster is 3x nodes with 150GB + 400GB drives on each node - in case this matter, separation is suppose to be between different size drives. Should _raw use_ & _data_ not be always _zero_? many thanks, L.

1 0

Re: rbd export/export-diff of group snaps
by Ilya Dryomov 06 Mar '26

06 Mar '26

On Fri, Mar 6, 2026 at 8:04 AM Sascha Lucas via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi Ilya, > > On Thu, 5 Mar 2026, Ilya Dryomov via ceph-users wrote: > > > To see the image-level snapshots that comprise a group snapshot taken > > with "rbd group snap create", run "rbd snap ls --all". > > > > There is no support for cloning from a group snapshot directly but > > cloning from the corresponding image-level snapshots is possible with > > "rbd clone <image name> --snap-id <snap id>" syntax (instead of the > > usual "rbd clone <image name>@<snap name>"). As a workaround one can > > do that for each member image and then create a new group out of the > > resulting clone images. The new "clone" group wouldn't get linked to > > the original "parent" group like clone images get linked to their > > parent images though. > > Thanks for this valuable information. I was able to clone each individual > image from the group snapshot. Also found this IBM documentation[1]. > > > Unfortunately there is no support for exporting a group snapshot > > directly (you already saw the unfinished PR) and even being able to > > export the corresponding image-level snapshots individually is pending > > on https://tracker.ceph.com/issues/72410. As a workaround one can > > clone from an image-level snapshot and then export the clone image. > > Also confirming that exporting the cloned images works well. But with this > workaround there seems no chance for export-diff between two group > snapshots? Correct -- https://tracker.ceph.com/issues/72410 is needed for that. Thanks, Ilya

1 0

Question about caching with Ceph
by Ewen Chan 06 Mar '26

06 Mar '26

Hi there. I am looking to potentially evaluate migrating my storage systems from ZFS to Ceph and my question is, is there something like ZFS' adaptive replacement cache (ARC) on Ceph? I am asking because I am looking to potentially deploy four nodes with six HGST 6 TB SATA HDDs in each node, where each HDD will become a Ceph OSD. My concern is that without something like ZFS' ARC, if it has to read the data from disk every single time, that it can be really slow, especially for repeat reads. I found this old thread talking a little bit about this: https://ceph-users.ceph.narkive.com/vPnNl3zj/cephfs-and-caching#, but as noted, it is a rather old thread by now, so I am wondering what is the state of caching with Ceph? I understand that cache tiering has been deprecated due to the lack of a maintainer. It is also my understanding (via the above mentioned older thread) that ZFS ARC works a little differently than the "general" linux cache. Your help in educating me how this works in 2026, is greatly appreciated. Thank you. Sincerely ~alpha

1 0

Re: rbd export/export-diff of group snaps
by Sascha Lucas 06 Mar '26

06 Mar '26

Hi Ilya, On Thu, 5 Mar 2026, Ilya Dryomov via ceph-users wrote: > To see the image-level snapshots that comprise a group snapshot taken > with "rbd group snap create", run "rbd snap ls --all". > > There is no support for cloning from a group snapshot directly but > cloning from the corresponding image-level snapshots is possible with > "rbd clone <image name> --snap-id <snap id>" syntax (instead of the > usual "rbd clone <image name>@<snap name>"). As a workaround one can > do that for each member image and then create a new group out of the > resulting clone images. The new "clone" group wouldn't get linked to > the original "parent" group like clone images get linked to their > parent images though. Thanks for this valuable information. I was able to clone each individual image from the group snapshot. Also found this IBM documentation[1]. > Unfortunately there is no support for exporting a group snapshot > directly (you already saw the unfinished PR) and even being able to > export the corresponding image-level snapshots individually is pending > on https://tracker.ceph.com/issues/72410. As a workaround one can > clone from an image-level snapshot and then export the clone image. Also confirming that exporting the cloned images works well. But with this workaround there seems no chance for export-diff between two group snapshots? > See my reply to Matthias -- involving qemu-guest-agent to freeze the > filesystems before taking the snapshot is actually a superior option, > so I wouldn't refer to it as a workaround. Yes, immediately after sending, I realised that "workaround" is not the right wording. Of cause is fs-freezing the next right step into more consistent snapshots. Thanks, Sascha. [1] https://www.ibm.com/docs/en/storage-ceph/9.9.0?topic=snapshot-cloning-block…

1 0

Re: rbd export/export-diff of group snaps
by Ilya Dryomov 05 Mar '26

05 Mar '26

On Thu, Mar 5, 2026 at 6:14 PM Sascha Lucas via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi Matthias, > > On Thu, 5 Mar 2026, Matthias Riße via ceph-users wrote: > > > I ran into this issue too and was quite puzzled. There is this open PR which > > suggests that group snapshot exports are unsupported as of now: > > https://github.com/ceph/ceph/pull/40808 > > Thanks for your fast reply. I was afraid of that the group feature is not > yet much usable for: no export, no clone, no mirror. Just snap/rollback > works. The IBM version supports group mirror[1]. But I don't know if this > feature will stay an exclusive enterprise one? > > The mentioned PR is old and outdated. Seems not a prominent feature ... Hi Sascha, To see the image-level snapshots that comprise a group snapshot taken with "rbd group snap create", run "rbd snap ls --all". There is no support for cloning from a group snapshot directly but cloning from the corresponding image-level snapshots is possible with "rbd clone <image name> --snap-id <snap id>" syntax (instead of the usual "rbd clone <image name>@<snap name>"). As a workaround one can do that for each member image and then create a new group out of the resulting clone images. The new "clone" group wouldn't get linked to the original "parent" group like clone images get linked to their parent images though. Unfortunately there is no support for exporting a group snapshot directly (you already saw the unfinished PR) and even being able to export the corresponding image-level snapshots individually is pending on https://tracker.ceph.com/issues/72410. As a workaround one can clone from an image-level snapshot and then export the clone image. > > > The best thing I could figure out to get consistent snapshots with multiple > > rbd's is to fsfreeze my filesystems (using libvirt, so domfs{freeze,thaw} in > > my case) and make individual snapshots of the rbd images. > > Yes, I already considered this "workaround", but I'm not using libvirt, so > have to talk to qemu-guest-agent my self. See my reply to Matthias -- involving qemu-guest-agent to freeze the filesystems before taking the snapshot is actually a superior option, so I wouldn't refer to it as a workaround. Thanks, Ilya

1 0