- ceph-users - lists.ceph.io

Re: CSC minutes 2026-03-16
by Martin Wege 18 Mar '26

18 Mar '26

On Mon, Mar 16, 2026 at 6:46 PM Takeshi Nishimura via ceph-users <ceph-users(a)ceph.io> wrote: > > On Mon, Mar 16, 2026 at 4:41 PM Laura Flores via Dev <dev(a)ceph.io> wrote: > > > > 2026/03/16 > > > > [DanV] drafted up a process for escalation/dispute resolution here: https://github.com/ceph/ceph/pull/67724 - incorporated feedback from greg and joe > ... > > Ceph on Windows discussion > > Weekly meeting was set up on the Community Calendar on Tuesdays > > Alexandre Jardon, Andy Herron involved > > > > Will come back to CSC next week to ask opinion on plan of action > > Goal will be to improve understanding/direction around Windows > > Could you please consider getting Roland Mainz > <roland.mainz(a)nrubsig.org> involved? > > Mr Mainz wrote ms-nfs41-client (link: > https://lore.kernel.org/linux-nfs/CANH4o6M1J2FW1dgvA4JHz+EXzMsyguuAT_TfbUy5… > and https://github.com/kofemann/ms-nfs41-client) and should have > sufficient expertise to provide comments. ms-nfs41-client is a Windows NFSv4.2 filesystem client. How should it be able to help with Ceph? Thanks, Martin

1 0

Re: reef 18.2.8 QE validation status
by Yuri Weinstein 17 Mar '26

17 Mar '26

This point release is ready We need to finish reviewing and approving release notes and add notable changes. https://github.com/ceph/ceph/pull/67746 On Tue, Mar 10, 2026 at 8:53 AM Yuri Weinstein via ceph-users < ceph-users(a)ceph.io> wrote: > Patrick approved fs and we will continue with this release > > On Wed, Feb 25, 2026 at 7:43 AM Patrick Donnelly <pdonnell(a)redhat.com> > wrote: > > > The `fs` and `powercycle` suites will need rerun with [1]. I believe > > you're already doing that but I'm documenting that for the list. > > > > I'm going to look into the upgrade issues today [2]. > > > > [1] https://github.com/ceph/ceph/pull/67495 > > [2] https://tracker.ceph.com/issues/75034 > > > > On Mon, Feb 23, 2026 at 4:29 PM Yuri Weinstein via ceph-users > > <ceph-users(a)ceph.io> wrote: > > > > > > Patrick, > > > > > > The only outstanding issues are the FS approval and upgrade issues, > > > reported by Laura. > > > > > > What do you suggest we do next here? > > > > > > On Mon, Feb 23, 2026 at 8:54 AM Adam King via Dev <dev(a)ceph.io> wrote: > > > > > > > orch approved. Run looks ugly due to us no having a proper ignorelist > > for > > > > the orch suite on reef, but the actual results are okay > > > > > > > > On Sat, Nov 29, 2025 at 12:09 PM Yuri Weinstein via ceph-users < > > > > ceph-users(a)ceph.io> wrote: > > > > > > > >> Rerun scheduled for krbd > > > >> > > > >> Still waiting for replies from dev leads > > > >> > > > >> On Mon, Nov 24, 2025 at 6:24 AM Ilya Dryomov <idryomov(a)gmail.com> > > wrote: > > > >> > > > >> > On Fri, Nov 21, 2025 at 12:04 AM Yuri Weinstein < > > yweinste(a)redhat.com> > > > >> > wrote: > > > >> > > > > > >> > > Note: this is an attempt to test and publish post EOL reef > point > > > >> release > > > >> > > > > > >> > > Details of this release are summarized here: > > > >> > > > > > >> > > https://tracker.ceph.com/issues/73906#note-1 > > > >> < > > > https://qcflzmq6.r.us-east-1.awstrack.me/L0/https:%2F%2Ftracker.ceph.com%2F… > > > > > > >> > > Release Notes - TBD > > > >> > > LRC upgrade - N/A > > > >> > > > > > >> > > distro => jammy focal centos9 windows bookworm > > > >> > > > > > >> > > Seeking approvals/reviews for: > > > >> > > > > > >> > > smoke - Laura approved? > > > >> > > > > > >> > > rados - Radek, Laura approved? Travis? Nizamudeen? Adam King > > > >> approved? > > > >> > > (rados is also running on build 2 == build 1 + > > > >> > > https://github.com/ceph/ceph/pull/66056 > > > >> < > > > https://qcflzmq6.r.us-east-1.awstrack.me/L0/https:%2F%2Fgithub.com%2Fceph%2… > > > > > > >> per Igor's request) > > > >> > > > > > >> > > rgw - Adam E approved? > > > >> > > > > > >> > > fs - Venky approved? > > > >> > > > > > >> > > orch - Adam King approved? > > > >> > > > > > >> > > rbd - Ilya approved? > > > >> > > krbd - Ilya approved? > > > >> > > > > >> > rbd approved, krbd needs a rerun as all jobs failed while > > > >> > going through cephlab.yml playbook (not related to RBD): > > > >> > > > > >> > ['Error mounting /var/lib/ceph: mount: /var/lib/ceph: wrong fs > type, > > > >> > bad option, bad superblock on /dev/mapper/vg_nvme-lv_5, missing > > > >> > codepage or helper program, or other error.'] > > > >> > > > > >> > Thanks, > > > >> > > > > >> > Ilya > > > >> > > > > >> > > > > >> _______________________________________________ > > > >> ceph-users mailing list -- ceph-users(a)ceph.io > > > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > >> > > > > _______________________________________________ > > > > Dev mailing list -- dev(a)ceph.io > > > > To unsubscribe send an email to dev-leave(a)ceph.io > > > > > > > _______________________________________________ > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > > > > -- > > Patrick Donnelly, Ph.D. > > He / Him / His > > Red Hat Partner Engineer > > IBM, Inc. > > GPG: 19F28A586F808C2402351B93C3301A3E258DD79D > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: cephfs quota samba export
by ronny.lippold 17 Mar '26

17 Mar '26

hi burkhard, i found that way and build my own script for that. but thanks, its good to get some experience. sometimes not easy, to find the right way ... had some issues with rbd|cephfs-mirroring. so, very good information for that money ;) i also used that for the samba dfree command script. its now working fine. little notice ... map a network drive and there is no "X GB of Y GB free" line. right click, rename the network drive and voila ... there it is ... funny. regards, ronny Am 2026-03-17 12:40, schrieb Burkhard Linke via ceph-users: > Hi, > > > just my 0.02 euro: > > > if you want to collect usage information on the cluster, you can use > the recursive stats feature in cephfs: > > $ getfattr -n ceph.dir.rbytes /vol/adm > getfattr: Removing leading '/' from absolute path names > # file: vol/adm > ceph.dir.rbytes="1199125900626" > > $ getfattr -n ceph.dir.rfiles /vol/adm > getfattr: Removing leading '/' from absolute path names > # file: vol/adm > ceph.dir.rfiles="309017" > > > We use a script to generate the corresponding values in a prometheus > compatible format, and use script-exporter for invoking the script > remotely. Results are stored in prometheus/thanos and visualized with > grafana. > > > This will not show correct values for users, but you can also export > the current quotas (also just xattrs), and use prometheus or grafana > alerting if certain thresholds are reached. > > > Best regards, > > Burkhard > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Cephadm staggered upgrade --limit parameter not always applied
by Eugen Block 17 Mar '26

17 Mar '26

Hi, I did a quick search on tracker but couldn't find anything related. A customer reported this, and I can confirm the behaviour on a lab cluster. I usually perform a staggered upgrade with --daemon-types and --limit, but not for all daemon types. So I haven't stumbled across this yet myself, but our customer did (I can reproduce with --daemon-types as well). They upgraded from latest Reef to latest Squid and reported that despite providing the --limit parameter, the mons were all upgraded. So I tried to reproduce, and the behaviour is not really clear to me, I'll try to clarify. # Start with MGRs reef1:~ # ceph orch upgrade start --image quay.io/ceph/ceph:v19.2.3 --services mgr --limit 1 Upgrading MGRs with limit works, but it doesn't reflect in the MGR log. Usually, I expect a line like this: ...[cephadm INFO root] Hit upgrade limit of 1. Stopping upgrade But there is no such line in the logs. Then I upgrade the rest of the MGRs. # Continue with MONs reef1:~ # ceph orch upgrade start --image quay.io/ceph/ceph:v19.2.3 --services mon --limit 1 This gets even weirder, the orchestrator upgrades 2 out of 3 MONs. And again, no such line in the log (Hit upgrade limit). What I noticed was a MGR respawn after the first MON had been upgraded successfully. Maybe some state of the upgrade progress gets lost during the respawn? I then upgraded the remaining MON. Then ceph-crash is upgraded successfully. # Upgrade OSD reef1:~ # ceph orch upgrade start --image quay.io/ceph/ceph:v19.2.3 --services osd.osd.standalone --limit 1 And this is the first service that actually reports the upgrade limit: 2026-03-05T15:47:17.108+0000 7f5bfdc73640 0 [cephadm INFO cephadm.upgrade] Upgrade: Updating osd.0 (1/1) 2026-03-05T15:47:17.108+0000 7f5bfdc73640 0 log_channel(cephadm) log [INF] : Upgrade: Updating osd.0 (1/1) 2026-03-05T15:47:32.518+0000 7f5bfdc73640 0 [cephadm INFO root] Hit upgrade limit of 1. Stopping upgrade 2026-03-05T15:47:32.518+0000 7f5bfdc73640 0 log_channel(cephadm) log [INF] : Hit upgrade limit of 1. Stopping upgrade 2026-03-05T15:47:47.395+0000 7f5bfdc73640 0 [cephadm INFO cephadm.upgrade] Upgrade: Setting container_image for all nvmeof 2026-03-05T15:47:47.395+0000 7f5bfdc73640 0 log_channel(cephadm) log [INF] : Upgrade: Setting container_image for all nvmeof 2026-03-05T15:47:47.492+0000 7f5bfdc73640 0 [cephadm INFO cephadm.upgrade] Upgrade: Finalizing container_image settings 2026-03-05T15:47:47.493+0000 7f5bfdc73640 0 log_channel(cephadm) log [INF] : Upgrade: Finalizing container_image settings 2026-03-05T15:47:47.667+0000 7f5bfdc73640 0 [cephadm INFO cephadm.upgrade] Upgrade: Complete! 2026-03-05T15:47:47.667+0000 7f5bfdc73640 0 log_channel(cephadm) log [INF] : Upgrade: Complete! This is really irritating and inconsistent: If the orchestrator does honor --limit with other services than OSD, why isn't that visible in the logs? And what's with the MONs? Why 2 out of 3? Any pointers appreciated! Not sure which Ceph versions might be affected by this, I'll try out a couple more upgrade paths. Thanks, Eugen

1 2

Re: cephfs quota samba export
by Burkhard Linke 17 Mar '26

17 Mar '26

Hi, just my 0.02 euro: if you want to collect usage information on the cluster, you can use the recursive stats feature in cephfs: $ getfattr -n ceph.dir.rbytes /vol/adm getfattr: Removing leading '/' from absolute path names # file: vol/adm ceph.dir.rbytes="1199125900626" $ getfattr -n ceph.dir.rfiles /vol/adm getfattr: Removing leading '/' from absolute path names # file: vol/adm ceph.dir.rfiles="309017" We use a script to generate the corresponding values in a prometheus compatible format, and use script-exporter for invoking the script remotely. Results are stored in prometheus/thanos and visualized with grafana. This will not show correct values for users, but you can also export the current quotas (also just xattrs), and use prometheus or grafana alerting if certain thresholds are reached. Best regards, Burkhard

1 0

Re: cephfs quota samba export
by ronny.lippold 17 Mar '26

17 Mar '26

only windows explorer will not show the space value i see name, letter and an usage diagram. mouse over shows me the correct value. but i miss "90mb free of 120mb". if you do not have an spontaneous idea ... we can close here its not samba list. thanks to all the people here. great to find community help from real users. regards, ronny Am 2026-03-17 07:45, schrieb Nemesiz via ceph-users: > I recommend to use external script or program for storage report. > > "dfree" https://www.samba.org/samba/docs/current/man-html/smb.conf.5.html#DFREECOMM… > > > On 2026-03-17 08:27, ronny.lippold via ceph-users wrote: >> many thanks ... did not found something in the archive, will search >> again. >> >> i found out, it looks like an samba issue >> >> i set up: >> data/users/user1 75M >> >> i have 2 shares in samba >> users -> path=data/users >> user1 -> path=data/users/user1 >> >> if i mount //server/users/user1 -> quota is not shown >> (//server/users/user1 5.5T 3.8G 5.5T 1% /mnt) >> >> if i mount the //server/user1 -> quota is shown correctly >> (//server/user1 72M 36M 36M 50% /mnt) >> >> looks bad to me, don't see an solution. >> >> >> regards, >> ronny >> >> >> >> Am 2026-03-16 16:05, schrieb Chris Palmer: >>> My next question was whether you were trying to apply quotas at >>> multiple points between the root and the target, and it appears you >>> are. IIRC there were discussions about that on this list some years >>> ago. I'm not sure that it is supported, and I don't really see how it >>> could be. However I'm not an expert in this area, so perhaps someone >>> else can chime in. >>> >>> On 16/03/2026 14:22, ronny.lippold wrote: >>>> update, burns my head down :) >>>> >>>> first level is working fine, under >>>> /mnt/pve/cephfs/shares/FOLDER/data/ >>>> server1 >>>> server2 >>>> users >>>> (in samba and smb client) >>>> >>>> one step down, not >>>> /mnt/pve/cephfs/shares/FOLDER/data/users >>>> username >>>> >>>> if i mount via ceph on the samba server >>>> /mnt/pve/cephfs/shares/FOLDER/data/users/username >>>> i see the correct quota size >>>> SRV(a)ID.cephfs=//shares/FOLDER/data/users/username 72M 0 72M >>>> 0% /srv/samba/data/username >>>> >>>> if i mount this via smb client no >>>> //SERVER/data/users/username 5.5T 3.8G 5.4T 1% /mnt3 >>>> >>>> >>>> ronny >>>> >>>> >>>> Am 2026-03-16 12:44, schrieb Chris Palmer via ceph-users: >>>>> Hi Ronny >>>>> >>>>> I use the very straightforward method of having a linux VM that >>>>> kernel-mounts the cephfs filesystem via fstab at /mnt/cephfs. I >>>>> apply an 8TB quota to one of the top-level directories >>>>> /mnt/cephfs/data2 (not the root of /mnt/cephfs as I mistakenly said >>>>> earlier!). Samba then exports /mnt/cephfs/data2/x, >>>>> /mnt/cephfs/data2/y etc just via the VM's directory tree. Each of >>>>> the exports (some with different access permissions etc specified >>>>> in samba) all correctly share the quota set on data2. >>>>> >>>>> This is a low-usage cluster, especially via samba, and has always >>>>> just worked. So I've never had any reason to experiment with vfs >>>>> etc. >>>>> >>>>> Regards, Chris >>>>> >>>>> On 16/03/2026 11:07, ronny.lippold wrote: >>>>>> hi chris and thanks for answer. >>>>>> >>>>>> do you use samba vfs ceph/ceph_new or mount cephfs on samba server >>>>>> and export that? >>>>>> what do you mean, "cephfs root though" >>>>>> >>>>>> i would like to have something like that >>>>>> >>>>>> data/users/user1 10g >>>>>> data/users/user2 5g >>>>>> data/users/user3 20g >>>>>> data/users/user4 10g >>>>>> data/share1 100g >>>>>> data/share2 200g >>>>>> ... >>>>>> >>>>>> regards, >>>>>> ronny >>>>>> >>>>>> >>>>>> Am 2026-03-16 11:56, schrieb Chris Palmer via ceph-users: >>>>>>> I do use cephfs exported via samba. My quota is simply done at >>>>>>> the cephfs root though. That works fine with Win10 & Win11 hosts >>>>>>> - they see the correct total/used/available space. I haven't >>>>>>> tried multiple quotas or setting further down the tree though. >>>>>>> >>>>>>> Regards, Chris >>>>>>> >>>>>>> On 16/03/2026 10:20, ronny.lippold via ceph-users wrote: >>>>>>>> hi, hope you can help me. >>>>>>>> >>>>>>>> i like to export an cephfs via samba an ctdb. >>>>>>>> its all working fine via kernel cephfs client. >>>>>>>> >>>>>>>> i set now quota to the users home dir. >>>>>>>> quota also working fine. >>>>>>>> >>>>>>>> but now, the problem ... on the client machine, i do not see the >>>>>>>> quota space. >>>>>>>> mostly, i see the complete cluster, or on windows i see nothing >>>>>>>> about disk usage. >>>>>>>> >>>>>>>> it would be important for us and the user, to se the remaining >>>>>>>> space. >>>>>>>> >>>>>>>> samba vfs ceph and ceph_new would not work ... i always got: >>>>>>>> cephmount_mount_fs: [CEPH] mount failed: user_id='XXX' >>>>>>>> fsname='XXX' No such file or directorymake_connection_snum: >>>>>>>> SMB_VFS_CONNECT for service 'XXX' at 'PATH' failed: No such file >>>>>>>> or directory >>>>>>>> >>>>>>>> does anybody use cephfs with samba an quota? >>>>>>>> running out of ideas, so an help is welcome. >>>>>>>> >>>>>>>> regards, >>>>>>>> ronny >>>>>>>> >>>>>>> _______________________________________________ >>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>> _______________________________________________ >>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: cephfs quota samba export
by ronny.lippold 17 Mar '26

17 Mar '26

great, thanks ... danke :) what a "hack" ... i love it. dfree command is working fine. have a great day, ronny Am 2026-03-17 07:45, schrieb Nemesiz via ceph-users: > I recommend to use external script or program for storage report. > > "dfree" https://www.samba.org/samba/docs/current/man-html/smb.conf.5.html#DFREECOMM… > > > On 2026-03-17 08:27, ronny.lippold via ceph-users wrote: >> many thanks ... did not found something in the archive, will search >> again. >> >> i found out, it looks like an samba issue >> >> i set up: >> data/users/user1 75M >> >> i have 2 shares in samba >> users -> path=data/users >> user1 -> path=data/users/user1 >> >> if i mount //server/users/user1 -> quota is not shown >> (//server/users/user1 5.5T 3.8G 5.5T 1% /mnt) >> >> if i mount the //server/user1 -> quota is shown correctly >> (//server/user1 72M 36M 36M 50% /mnt) >> >> looks bad to me, don't see an solution. >> >> >> regards, >> ronny >> >> >> >> Am 2026-03-16 16:05, schrieb Chris Palmer: >>> My next question was whether you were trying to apply quotas at >>> multiple points between the root and the target, and it appears you >>> are. IIRC there were discussions about that on this list some years >>> ago. I'm not sure that it is supported, and I don't really see how it >>> could be. However I'm not an expert in this area, so perhaps someone >>> else can chime in. >>> >>> On 16/03/2026 14:22, ronny.lippold wrote: >>>> update, burns my head down :) >>>> >>>> first level is working fine, under >>>> /mnt/pve/cephfs/shares/FOLDER/data/ >>>> server1 >>>> server2 >>>> users >>>> (in samba and smb client) >>>> >>>> one step down, not >>>> /mnt/pve/cephfs/shares/FOLDER/data/users >>>> username >>>> >>>> if i mount via ceph on the samba server >>>> /mnt/pve/cephfs/shares/FOLDER/data/users/username >>>> i see the correct quota size >>>> SRV(a)ID.cephfs=//shares/FOLDER/data/users/username 72M 0 72M >>>> 0% /srv/samba/data/username >>>> >>>> if i mount this via smb client no >>>> //SERVER/data/users/username 5.5T 3.8G 5.4T 1% /mnt3 >>>> >>>> >>>> ronny >>>> >>>> >>>> Am 2026-03-16 12:44, schrieb Chris Palmer via ceph-users: >>>>> Hi Ronny >>>>> >>>>> I use the very straightforward method of having a linux VM that >>>>> kernel-mounts the cephfs filesystem via fstab at /mnt/cephfs. I >>>>> apply an 8TB quota to one of the top-level directories >>>>> /mnt/cephfs/data2 (not the root of /mnt/cephfs as I mistakenly said >>>>> earlier!). Samba then exports /mnt/cephfs/data2/x, >>>>> /mnt/cephfs/data2/y etc just via the VM's directory tree. Each of >>>>> the exports (some with different access permissions etc specified >>>>> in samba) all correctly share the quota set on data2. >>>>> >>>>> This is a low-usage cluster, especially via samba, and has always >>>>> just worked. So I've never had any reason to experiment with vfs >>>>> etc. >>>>> >>>>> Regards, Chris >>>>> >>>>> On 16/03/2026 11:07, ronny.lippold wrote: >>>>>> hi chris and thanks for answer. >>>>>> >>>>>> do you use samba vfs ceph/ceph_new or mount cephfs on samba server >>>>>> and export that? >>>>>> what do you mean, "cephfs root though" >>>>>> >>>>>> i would like to have something like that >>>>>> >>>>>> data/users/user1 10g >>>>>> data/users/user2 5g >>>>>> data/users/user3 20g >>>>>> data/users/user4 10g >>>>>> data/share1 100g >>>>>> data/share2 200g >>>>>> ... >>>>>> >>>>>> regards, >>>>>> ronny >>>>>> >>>>>> >>>>>> Am 2026-03-16 11:56, schrieb Chris Palmer via ceph-users: >>>>>>> I do use cephfs exported via samba. My quota is simply done at >>>>>>> the cephfs root though. That works fine with Win10 & Win11 hosts >>>>>>> - they see the correct total/used/available space. I haven't >>>>>>> tried multiple quotas or setting further down the tree though. >>>>>>> >>>>>>> Regards, Chris >>>>>>> >>>>>>> On 16/03/2026 10:20, ronny.lippold via ceph-users wrote: >>>>>>>> hi, hope you can help me. >>>>>>>> >>>>>>>> i like to export an cephfs via samba an ctdb. >>>>>>>> its all working fine via kernel cephfs client. >>>>>>>> >>>>>>>> i set now quota to the users home dir. >>>>>>>> quota also working fine. >>>>>>>> >>>>>>>> but now, the problem ... on the client machine, i do not see the >>>>>>>> quota space. >>>>>>>> mostly, i see the complete cluster, or on windows i see nothing >>>>>>>> about disk usage. >>>>>>>> >>>>>>>> it would be important for us and the user, to se the remaining >>>>>>>> space. >>>>>>>> >>>>>>>> samba vfs ceph and ceph_new would not work ... i always got: >>>>>>>> cephmount_mount_fs: [CEPH] mount failed: user_id='XXX' >>>>>>>> fsname='XXX' No such file or directorymake_connection_snum: >>>>>>>> SMB_VFS_CONNECT for service 'XXX' at 'PATH' failed: No such file >>>>>>>> or directory >>>>>>>> >>>>>>>> does anybody use cephfs with samba an quota? >>>>>>>> running out of ideas, so an help is welcome. >>>>>>>> >>>>>>>> regards, >>>>>>>> ronny >>>>>>>> >>>>>>> _______________________________________________ >>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>> _______________________________________________ >>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

problem with mgr prometheus module
by Dario Graña 17 Mar '26

17 Mar '26

Hi all! I'm running ceph quincy 17.2.7 in a cluster. On monday I updated the OS to AlmaLinux 9.3 to 9.4, since then grafana shows "No Data" message in all ceph related fields but, for example, the nodes information is still fine (Host Detail Dashboard). I have redeployed the mgr service with cephadm, disabled and re-enabled mgr prometheus module , but nothing changed. Digging into the problem, I accessed the prometheus interface. When I access prometheus, and found this error[image: Screen Shot 2024-06-04 at 15.22.37.png] When I access the node shown as down, it reports 503 Service Unavailable No cached data available yet Traceback (most recent call last): File "/lib/python3.6/site-packages/cherrypy/_cprequest.py", line 638, in respond self._do_respond(path_info) File "/lib/python3.6/site-packages/cherrypy/_cprequest.py", line 697, in _do_respond response.body = self.handler() File "/lib/python3.6/site-packages/cherrypy/lib/encoding.py", line 219, in __call__ self.body = self.oldhandler(*args, **kwargs) File "/lib/python3.6/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__ return self.callable(*self.args, **self.kwargs) File "/usr/share/ceph/mgr/prometheus/module.py", line 1751, in metrics return self._metrics(_global_instance) File "/usr/share/ceph/mgr/prometheus/module.py", line 1762, in _metrics raise cherrypy.HTTPError(503, 'No cached data available yet') cherrypy._cperror.HTTPError: (503, 'No cached data available yet') I checked the mgr prometheus address and port [ceph: root@ceph-admin01 /]# ceph config get mgr mgr/prometheus/server_addr :: [ceph: root@ceph-admin01 /]# ceph config get mgr mgr/prometheus/server_port 9283 It seems to be ok. When I check the master manager node for the port, I found [root@ceph-hn01 ~]# netstat -natup | grep 9283 tcp6 0 0 :::9283 :::* LISTEN 2453/ceph-mgr tcp6 0 0 192.168.97.51:9283 192.168.97.60:36130 ESTABLISHED 2453/ceph-mgr I don't understand why it is showing as IPv6, the node doesn't have a dual stack. I also tried to use a newer version of the prometheus container image, the 1.6.0, but it keeps reporting the same, so I rolled it back to the original one. Has anyone experienced an issue like this? Where can I look for more information about it? Thanks in advance. Regards. -- Dario Graña PIC (Port d'Informació Científica) Campus UAB, Edificio D E-08193 Bellaterra, Barcelona http://www.pic.es Avis - Aviso - Legal Notice: http://legal.ifae.es

2 2

Re: problem with mgr prometheus module
by JOUSSE Adrien 17 Mar '26

17 Mar '26

Hello, Quite an hold thread, but maybe this will help someone. We had the same issue on our Ceph cluster. It all came down to an erasure code profile with a missing setting. In the logs of ceph-mgr (journalctl -xru ceph-mgr@<host>.service), we saw the following error: KeyError: 'm' description = f"ec:{profile['k']}+{profile['m']}" in /usr/share/ceph/mgr/prometheus/module.py. One of our erasure code profile was missing the setting "m" (which is the number of parity chunks for a chunk of data, if I understood correctly). You can see your erasure code profile using : ceph osd erasure-code-profile ls And the settings of each code profile with : ceph osd erasure-code-profile get <name> We deployed a simple fix by editing the module.py file to handle this scenario. Replace (around l.1250) if profile: description = f"ec:{profile['k']}+{profile['m']}" With if profile: if profile.get("m", None) is not None: description = f"ec:{profile['k']}+{profile['m']}" else: description = f"ec:{profile['k']}" Adrien [Site du CEA]<https://www.cea.fr/> Adrien JOUSSE Ingénieur systèmes Unix Direction de la Recherche Fondamentale (DRF) DRF/IRFU/DEDIP//LIS Centre de Saclay [Facebook]<https://fr-fr.facebook.com/cea.pageofficielle> [Youtube] <https://www.youtube.com/channel/UCpTMWY8NRQby8zXUKhHznfg> [Twitter] <https://twitter.com/CEA_Officiel> [LinkedIn] <https://fr.linkedin.com/company/cea> [Instagram] <https://www.instagram.com/cea_officiel/>

1 0

Re: cephfs quota samba export
by Nemesiz 17 Mar '26

17 Mar '26

I recommend to use external script or program for storage report. "dfree" https://www.samba.org/samba/docs/current/man-html/smb.conf.5.html#DFREECOMM… On 2026-03-17 08:27, ronny.lippold via ceph-users wrote: > many thanks ... did not found something in the archive, will search > again. > > i found out, it looks like an samba issue > > i set up: > data/users/user1 75M > > i have 2 shares in samba > users -> path=data/users > user1 -> path=data/users/user1 > > if i mount //server/users/user1 -> quota is not shown > (//server/users/user1 5.5T 3.8G 5.5T 1% /mnt) > > if i mount the //server/user1 -> quota is shown correctly > (//server/user1 72M 36M 36M 50% /mnt) > > looks bad to me, don't see an solution. > > > regards, > ronny > > > > Am 2026-03-16 16:05, schrieb Chris Palmer: >> My next question was whether you were trying to apply quotas at >> multiple points between the root and the target, and it appears you >> are. IIRC there were discussions about that on this list some years >> ago. I'm not sure that it is supported, and I don't really see how it >> could be. However I'm not an expert in this area, so perhaps someone >> else can chime in. >> >> On 16/03/2026 14:22, ronny.lippold wrote: >>> update, burns my head down :) >>> >>> first level is working fine, under >>> /mnt/pve/cephfs/shares/FOLDER/data/ >>> server1 >>> server2 >>> users >>> (in samba and smb client) >>> >>> one step down, not >>> /mnt/pve/cephfs/shares/FOLDER/data/users >>> username >>> >>> if i mount via ceph on the samba server >>> /mnt/pve/cephfs/shares/FOLDER/data/users/username >>> i see the correct quota size >>> SRV(a)ID.cephfs=//shares/FOLDER/data/users/username 72M 0 72M >>> 0% /srv/samba/data/username >>> >>> if i mount this via smb client no >>> //SERVER/data/users/username 5.5T 3.8G 5.4T 1% /mnt3 >>> >>> >>> ronny >>> >>> >>> Am 2026-03-16 12:44, schrieb Chris Palmer via ceph-users: >>>> Hi Ronny >>>> >>>> I use the very straightforward method of having a linux VM that >>>> kernel-mounts the cephfs filesystem via fstab at /mnt/cephfs. I >>>> apply an 8TB quota to one of the top-level directories >>>> /mnt/cephfs/data2 (not the root of /mnt/cephfs as I mistakenly said >>>> earlier!). Samba then exports /mnt/cephfs/data2/x, >>>> /mnt/cephfs/data2/y etc just via the VM's directory tree. Each of >>>> the exports (some with different access permissions etc specified >>>> in samba) all correctly share the quota set on data2. >>>> >>>> This is a low-usage cluster, especially via samba, and has always >>>> just worked. So I've never had any reason to experiment with vfs etc. >>>> >>>> Regards, Chris >>>> >>>> On 16/03/2026 11:07, ronny.lippold wrote: >>>>> hi chris and thanks for answer. >>>>> >>>>> do you use samba vfs ceph/ceph_new or mount cephfs on samba server >>>>> and export that? >>>>> what do you mean, "cephfs root though" >>>>> >>>>> i would like to have something like that >>>>> >>>>> data/users/user1 10g >>>>> data/users/user2 5g >>>>> data/users/user3 20g >>>>> data/users/user4 10g >>>>> data/share1 100g >>>>> data/share2 200g >>>>> ... >>>>> >>>>> regards, >>>>> ronny >>>>> >>>>> >>>>> Am 2026-03-16 11:56, schrieb Chris Palmer via ceph-users: >>>>>> I do use cephfs exported via samba. My quota is simply done at >>>>>> the cephfs root though. That works fine with Win10 & Win11 hosts >>>>>> - they see the correct total/used/available space. I haven't >>>>>> tried multiple quotas or setting further down the tree though. >>>>>> >>>>>> Regards, Chris >>>>>> >>>>>> On 16/03/2026 10:20, ronny.lippold via ceph-users wrote: >>>>>>> hi, hope you can help me. >>>>>>> >>>>>>> i like to export an cephfs via samba an ctdb. >>>>>>> its all working fine via kernel cephfs client. >>>>>>> >>>>>>> i set now quota to the users home dir. >>>>>>> quota also working fine. >>>>>>> >>>>>>> but now, the problem ... on the client machine, i do not see the >>>>>>> quota space. >>>>>>> mostly, i see the complete cluster, or on windows i see nothing >>>>>>> about disk usage. >>>>>>> >>>>>>> it would be important for us and the user, to se the remaining >>>>>>> space. >>>>>>> >>>>>>> samba vfs ceph and ceph_new would not work ... i always got: >>>>>>> cephmount_mount_fs: [CEPH] mount failed: user_id='XXX' >>>>>>> fsname='XXX' No such file or directorymake_connection_snum: >>>>>>> SMB_VFS_CONNECT for service 'XXX' at 'PATH' failed: No such file >>>>>>> or directory >>>>>>> >>>>>>> does anybody use cephfs with samba an quota? >>>>>>> running out of ideas, so an help is welcome. >>>>>>> >>>>>>> regards, >>>>>>> ronny >>>>>>> >>>>>> _______________________________________________ >>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0