- ceph-users - lists.ceph.io

Re: Mounting CephFS on Ceph nodes themselves
by Stefan Möding 29 Jul '26

29 Jul '26

Marek Szuba via ceph-users <ceph-users(a)ceph.io> writes: > A few weeks ago I came across in Tentacle documentation on docs.ceph.org > (and OF COURSE now that I'm trying to quote it, I cannot find it any > more) a mention that CephFS should not be mounted on systems serving > Ceph because of there being a risk of harmful interference in the > kernel. The context suggested this pertained to one-node Ceph clusters, > that said I do not believe that was explicitly stated. Therefore, I very > much hope someone will be able to enlighten me whether: Maybe it was this page: https://docs.ceph.com/en/squid/rados/troubleshooting/troubleshooting-pg/ -- Stefan

1 0

Ceph Governing Board Meetings 2026-07-28
by Anthony Middleton 28 Jul '26

28 Jul '26

Hey Ceph friends, I hope you're doing well. Starting now, a portion of our Ceph Governing Board meetings will be recorded and made publicly available each month. This aims to increase transparency and encourage more community participation. You can watch the latest meeting and see the minutes attached below: https://www.youtube.com/watch?v=gPyu9KLVD64. As highlighted in the meeting, there are several committees you can get involved in, like Hardware, Ceph Days, Ceph Ambassadors, and Community Outreach. Feel free to reach out to me if you're interested in connecting with any of the committee leaders. Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

Re: v20.2.3 tentacle QE validation status
by Kamoltat Sirivadhna 28 Jul '26

28 Jul '26

RADOS Approved On Tue, Jul 28, 2026 at 1:00 PM Adam Emerson via ceph-users < ceph-users(a)ceph.io> wrote: > On 27/07/2026, Yuri Weinstein via dev wrote: > > Details of this release are summarized here: > > https://tracker.ceph.com/issues/77256#note-5 > > > > Release Notes - TBD > > LRC upgrade - TBD > > > > *PLEASE ADD YOUR COMMENTS/APPROVALS TO THE TRACKER!* > > > > Seeking approvals/reviews for: > > > > smoke: PASSED > > > > rados: Kamoltat, Afreen, Travis > > > > rgw: Adam Emerson > > RGW Approved. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > -- Kamoltat (Junior) Sirivadhna (HE/HIM) Senior SoftWare Engineer - IBM Ceph Storage ksirivad(a)redhat.com T: (857) <(919)716-5348>253-8927

1 0

Re: Mounting CephFS on Ceph nodes themselves
by Gregory Farnum 28 Jul '26

28 Jul '26

On Tue, Jul 28, 2026 at 3:05 AM Janne Johansson via ceph-users <ceph-users(a)ceph.io> wrote: > > I think the idea was that if you mount cephfs on a box running OSDs or a > mon daemon, and it gets into some kind of trouble like running out of > memory or so, then both the client side and the "server" side might be > competing for the same resources that aren't there, making it worse. This is a classic problem in network filesystems: 1) The kernel runs out of memory and needs to flush out buffered writes to free up space 2) The client sends out a network message, which comes back in to the server 3) The server needs to allocate memory to handle the incoming network message 4) ...stuck RBD and CephFS are susceptible to this, if you mount the kernels on the same hosts as the OSDs. But the odds of hitting it are a lot lower on modern systems with their RAM capacities than in the past. -Greg > > If you have separate OSD hosts and they are having issues, then cephfs > clients will be slow, but not much more than that, and if clients are > having resource dips, a separate cluster doesn't mind at all. I think that > is where the non-recommendation comes from, in case you do combine them, it > will be hard to get out of this situation, any cpu/ram given to the client > side would cause more IO to be requested, from the server side that doesn't > have it. In any case it might make it lots harder to diagnose which end > actually caused the issue if both suffer at the same time. > > > > Den tis 28 juli 2026 kl 14:45 skrev Marek Szuba via ceph-users < > ceph-users(a)ceph.io>: > > > Dear fellow Cephies, > > > > A few weeks ago I came across in Tentacle documentation on docs.ceph.org > > (and OF COURSE now that I'm trying to quote it, I cannot find it any > > more) a mention that CephFS should not be mounted on systems serving > > Ceph because of there being a risk of harmful interference in the > > kernel. The context suggested this pertained to one-node Ceph clusters, > > that said I do not believe that was explicitly stated. Therefore, I very > > much hope someone will be able to enlighten me whether: > > > > * this is indeed still the case with Tentacle, > > > > * it applies to one-node clusters only or all Ceph clusters, > > > > * it only applies to CephFS mounts using the kernel driver, or ceph-fuse > > as well. > > > > Thank you in advance! > > > > -- > > MS > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > -- > May the most significant bit of your life be positive. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: Mounting CephFS on Ceph nodes themselves
by Malte Stroem 28 Jul '26

28 Jul '26

Hi, you can mount it on the Ceph nodes. You can mount it everywhere. Best, Malte On 7/28/26 14:48, Marek Szuba via ceph-users wrote: > Dear fellow Cephies, > > A few weeks ago I came across in Tentacle documentation on docs.ceph.org > (and OF COURSE now that I'm trying to quote it, I cannot find it any > more) a mention that CephFS should not be mounted on systems serving > Ceph because of there being a risk of harmful interference in the > kernel. The context suggested this pertained to one-node Ceph clusters, > that said I do not believe that was explicitly stated. Therefore, I very > much hope someone will be able to enlighten me whether: > > * this is indeed still the case with Tentacle, > > * it applies to one-node clusters only or all Ceph clusters, > > * it only applies to CephFS mounts using the kernel driver, or ceph-fuse > as well. > > Thank you in advance! >

1 0

Re: v20.2.3 tentacle QE validation status
by Adam Emerson 28 Jul '26

28 Jul '26

On 27/07/2026, Yuri Weinstein via dev wrote: > Details of this release are summarized here: > https://tracker.ceph.com/issues/77256#note-5 > > Release Notes - TBD > LRC upgrade - TBD > > *PLEASE ADD YOUR COMMENTS/APPROVALS TO THE TRACKER!* > > Seeking approvals/reviews for: > > smoke: PASSED > > rados: Kamoltat, Afreen, Travis > > rgw: Adam Emerson RGW Approved.

1 0

Re: Mounting CephFS on Ceph nodes themselves
by Janne Johansson 28 Jul '26

28 Jul '26

I think the idea was that if you mount cephfs on a box running OSDs or a mon daemon, and it gets into some kind of trouble like running out of memory or so, then both the client side and the "server" side might be competing for the same resources that aren't there, making it worse. If you have separate OSD hosts and they are having issues, then cephfs clients will be slow, but not much more than that, and if clients are having resource dips, a separate cluster doesn't mind at all. I think that is where the non-recommendation comes from, in case you do combine them, it will be hard to get out of this situation, any cpu/ram given to the client side would cause more IO to be requested, from the server side that doesn't have it. In any case it might make it lots harder to diagnose which end actually caused the issue if both suffer at the same time. Den tis 28 juli 2026 kl 14:45 skrev Marek Szuba via ceph-users < ceph-users(a)ceph.io>: > Dear fellow Cephies, > > A few weeks ago I came across in Tentacle documentation on docs.ceph.org > (and OF COURSE now that I'm trying to quote it, I cannot find it any > more) a mention that CephFS should not be mounted on systems serving > Ceph because of there being a risk of harmful interference in the > kernel. The context suggested this pertained to one-node Ceph clusters, > that said I do not believe that was explicitly stated. Therefore, I very > much hope someone will be able to enlighten me whether: > > * this is indeed still the case with Tentacle, > > * it applies to one-node clusters only or all Ceph clusters, > > * it only applies to CephFS mounts using the kernel driver, or ceph-fuse > as well. > > Thank you in advance! > > -- > MS > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- May the most significant bit of your life be positive.

1 0

Mounting CephFS on Ceph nodes themselves
by Marek Szuba 28 Jul '26

28 Jul '26

Dear fellow Cephies, A few weeks ago I came across in Tentacle documentation on docs.ceph.org (and OF COURSE now that I'm trying to quote it, I cannot find it any more) a mention that CephFS should not be mounted on systems serving Ceph because of there being a risk of harmful interference in the kernel. The context suggested this pertained to one-node Ceph clusters, that said I do not believe that was explicitly stated. Therefore, I very much hope someone will be able to enlighten me whether: * this is indeed still the case with Tentacle, * it applies to one-node clusters only or all Ceph clusters, * it only applies to CephFS mounts using the kernel driver, or ceph-fuse as well. Thank you in advance! -- MS

1 0

Re: Ec pool min size
by GLE, Vivien 28 Jul '26

28 Jul '26

>That configuration is optimal for maximizing recovery times and >write latency: since each EC set will span 8 hosts of 9, every >time a host becames unavailable virtually all PGs will become >degraded, and even with all host available any write will >involve nearly all hosts. >This kind of optimal configuration is popular with those who >think that the original design assumption of Ceph of having >many, many more hosts and OSDs than the replication factor or >stripe width was wasteful and cowardly. :-) Isn't a bigger k+m a good things for performance if network and hardware not the bottleneck ? >Is there any particular reason why the failure domain is host >but pods with 3 hosts get shut down? I feel that it will give the same result as pods failure domain with the gain of loosing 3 hosts >Depends for what. But it is difficult to say without knowing the >expected data and workload profiles. Will be mostly bloc (virtual machine and kubernetes pods) >I might opine that in most cases it is as unimportant as having >'min_size=2' with 'size=3'. :-) >"Safely" and 'k=5' with 'min_size=5' is an interesting >association :-). Asking just in case but already knew the answer I guess >Perhaps 'k=2,m=2' or for the very brave 'k=4,m=2'. But again it >all depends on expected data and workload profiles. Just downsize the profile to 4+3 as this cluster is in test phase (I want to keep this failure host domain and I believe this is the best to do ?) ________________________________ De : Peter Grandi via ceph-users <ceph-users(a)ceph.io> Envoyé : lundi 27 juillet 2026 12:15:47 À : list Linux fs Ceph Objet : [ceph-users] Re: Ec pool min size > We have a 9 hosts (3 per pod) cluster with 1 pool in EC 5+3 > (host domain failure), That configuration is optimal for maximizing recovery times and write latency: since each EC set will span 8 hosts of 9, every time a host becames unavailable virtually all PGs will become degraded, and even with all host available any write will involve nearly all hosts. This kind of optimal configuration is popular with those who think that the original design assumption of Ceph of having many, many more hosts and OSDs than the replication factor or stripe width was wasteful and cowardly. :-) > I expected that shutting down 1 pod Is there any particular reason why the failure domain is host but pods with 3 hosts get shut down? > wouldn't affect our pg but I just discovered that the EC pool > is in min_size 6 while trying it. Is min_size 6 important in > this case ? Depends for what. But it is difficult to say without knowing the expected data and workload profiles. I might opine that in most cases it is as unimportant as having 'min_size=2' with 'size=3'. :-) > Could I safely lower it to 5 "Safely" and 'k=5' with 'min_size=5' is an interesting association :-). > or using another EC k+m would suit better? Perhaps 'k=2,m=2' or for the very brave 'k=4,m=2'. But again it all depends on expected data and workload profiles. _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Re: Ec pool min size
by Janne Johansson 28 Jul '26

28 Jul '26

> > > >That configuration is optimal for maximizing recovery times and > >write latency: since each EC set will span 8 hosts of 9, every > >time a host becames unavailable virtually all PGs will become > >degraded, and even with all host available any write will > >involve nearly all hosts. > > >This kind of optimal configuration is popular with those who > >think that the original design assumption of Ceph of having > >many, many more hosts and OSDs than the replication factor or > >stripe width was wasteful and cowardly. :-) > > > Isn't a bigger k+m a good things for performance if network and hardware > not the bottleneck ? > In some sense, but it also matters how many hosts you have, so it's more about "how close to degraded am I when everything is normal?" If you have 6+3 with only 10 hosts or 6+3 with 100 hosts matters a lot if one,two or three hosts accidentally goes down. So it's not only about if "6+3 is better than 4+2", but also how large part of your hosts this covers. -- May the most significant bit of your life be positive.

1 0