December 2025 - ceph-users

Re: SMB on ARM
by Anthony D'Atri 09 Dec '25

09 Dec '25

> Don’t suppose there’s a way to update that for the correct architecture? And or to document it if it isn’t going to be possible? I defer to the nonpariel JM re daemons / services available on ARM, but FWIW I've added a note to the docs: https://github.com/ceph/ceph/pull/66560/files -- aad

2 1

Re: dashboard dial tcp connection refused
by Nizamudeen A 09 Dec '25

09 Dec '25

Hi, Was that cluster ever configured with ssl before? Maybe the initial alertmanager webhook config had the ssl enabled url added to it and its now failing to detect it. Have you tried to redeploy the alertmanager to probably refresh its config? It might be worth checking the alertmanager configuration to see what urls its pointing to. Also it might help more if you can share some info about your cluster like the version it is in right now. I remember there were some bugs around the standby server and alertmanager configuration but they seem to have been fixed in some version of ceph. Regards, Nizam On Mon, Dec 8, 2025 at 5:22 PM lejeczek via ceph-users <ceph-users(a)ceph.io> wrote: > Hi guys. > > I've no _ssl_ used by _dashboard_ > > -> $ ceph mgr services > { > "dashboard": "http://10.1.1.63:8080/", > "prometheus": "http://10.1.1.63:9283/" > } > -> $ ceph config get mgr mgr/dashboard/ssl > false > > and the 'standby' servers logs: > ... > Dec 08 11:33:39 > ceph-9f4f9dba-72c7-11f0-8052-525400519d29-alertmanager-podster1[2503]: > ts=2025-12-08T11:33:39.781Z caller=dispatch.go:353 > level=error component=dispatcher msg="Notify for alerts > failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify > retry canceled after 7 attempts: Post \"<redacted>\": dial > tcp 10.1.1.63:8443: connect: connection refused; > ceph-dashboard/webhook[1]: notify retry canceled after 7 > attempts: Post \"<redacted>\": dial tcp 10.1.1.61:8443: > connect: connection refused" > Dec 08 11:33:39 > ceph-9f4f9dba-72c7-11f0-8052-525400519d29-alertmanager-podster1[2503]: > ts=2025-12-08T11:33:39.781Z caller=notify.go:848 level=warn > component=dispatcher receiver=ceph-dashboard > integration=webhook[0] > aggrGroup="{}/{}:{alertname=\"CephHealthWarning\"}" > msg="Notify attempt failed, will retry later" attempts=1 > err="Post \"<redacted>\": dial tcp 10.1.1.63:8443: connect: > connection refused" > ... > with the snippet of the logs above _ceph_ says only: > -> $ ceph health detail > HEALTH_WARN mon podster1 is low on available space > [WRN] MON_DISK_LOW: mon podster1 is low on available space > mon.podster1 has 12% avail > > nothing listens on 8443 anywhere, which is as it should be - > why standby server wants that - would anybody know? > 10.1.1.61 is standby's own IP. > > many thanks, L. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- Nizamudeen A Sr. Software Engineer - IBM Partner Engineer IBM and Red Hat Ceph Storage Red Hat <https://www.redhat.com/> <https://www.redhat.com/>

1 0

Size or IOPS for DB+WAL?
by Andrew Klaassen 09 Dec '25

09 Dec '25

Hi, I'm looking at converting an old Nutanix+VMware cluster to Ceph+Proxmox. It'll host a mix of a couple of hundred VMs - web servers, Active Directory, Elasticsearch clusters, postgres and mysql databases, software deployment servers, log forwarders, some stuff that I don't even know what it does - basically a grab bag of workloads. It'll be a 9 node cluster. CPU+RAM is a mix from 12 core Intel Haswell with 512GB to 18 core Intel Ice Lake with 1TB. NICs will be two dual-port 10GbE (total four ports) per node. Disks per node will be a mix of: HDD: 2x12TB + 1x8TB SSD: 2x3.84TB HDD: 1x12TB + 2x6TB + 1x8TB SSD: 1x7.68TB This will leave me with one slot in each node for a DB+WAL drive. Is either of these a clear winner? Smaller and faster, or bigger and slower? Micron 7450 MTFDKCB960TFR - 960GB - 85K random write IOPS - 2% of raw storage KIOXIA KCD6XLUL3T84 - 3.84TB - 60K random write IOPS - 8% of raw storage If there is a clear winner, are there any setup or tuning options I should know about to take full advantage of them? Thanks. Andrew

1 0

smartctl issues on squid (ubuntu 24.04)
by Nathan Butcher 08 Dec '25

08 Dec '25

Hi, We've been running squid 19.2.3 on Ubuntu 24.04.3 LTS We encountered some issues where the Ceph dashboard was unable to get SMART data from OSDs under the Device Health tab. At first we were getting -22 errors from that tab suggesting that smartctl was under version 7. That wasn't the case at all. System logging determined that there was an apparmor profile called unix-chkpwd which was producing errors as ceph tried to authenticate into the pod to run smartctl. We had to disable that profile in order to get that to work. We had to set the profile to complain mode using aa-complain (and even then Ubuntu had some bugs getting that command to work) Our OSD node has a HPE smart array card on it which ensures the kernel sees all drives on it as scsi devices, whether in HBA or RAID mode. From within the OSD pod, it seems that smartctl isn't able to autodetect the type of drives it is trying to access, possibly because attributed from udev and getting blocked somehow. We needed the "-d scsi" command to work, but the ceph code doesn't allow any customisation or flexibility in how smartctl is called. In the end I had to write a wrapper script to inject the -d scsi command around smartctl in order to get it to work within the confines of the OSD pod. Then the wrapper had to be distributed across every OSD pod. It's a real messy hack. So a couple of things: How can we control how the OSD pods supposed to present allowing smartctl to work but without compromising on security? Is this just an ubuntu apparmor thing that ubuntu should deal with? I can see that there is a separate apparmor profile applied to pods. I'm sure we're not the only ones who have hit this roadblock. In any case, I hope someone else having the same issues find these workarounds useful Nathan Butcher Senior Systems Engineer 1300 852 110 Chat on Teams nbutcher(a)ct.com.au We are passionate about service excellence. If you would like to contact my Team Leader for any reason, their name is Tim Sandy, email address is tsandy(a)ct.com.au and phone number is (03) 9998 6377. We’re 100% Carbon Neutral and ISO 14001 certified. Check out our Environmental Sustainability pledge here. Adelaide | Brisbane | Canberra | Darwin | Hobart | Melbourne | Perth | Sydney Disclaimer This email message is confidential and may be privileged. Unauthorised use, copying or distribution of any part of this email including attachments is prohibited. If you are not the intended recipient, please forward the email to infosec(a)ct.com.au and delete the original. Any advice contained in this email is of a general nature only and has been prepared without considering your relevant personal circumstances. To the extent permitted by law, we exclude (and where the law does not permit exclusion, limit to the extent permitted by law) all liability for any direct, indirect and consequential losses, damages and expenses incurred in any way (including but not limited to that arising from negligence), connected with any use or access to or any reliance on information contained in this email or any attachments. Centorrino Technologies Pty Ltd (ABN: 83 606 931 524). All Rights Reserved.

1 0

Re: reef 18.2.8 QE validation status
by Yuri Weinstein 08 Dec '25

08 Dec '25

This test branch was built loner then 2.5 weeks ago I have to rebuild it first and then will schedule krbd On Mon, Dec 8, 2025 at 1:39 PM Ilya Dryomov via ceph-users < ceph-users(a)ceph.io> wrote: > On Mon, Dec 8, 2025 at 8:50 PM Ilya Dryomov <idryomov(a)gmail.com> wrote: > > > > On Mon, Dec 8, 2025 at 5:00 PM Venky Shankar via ceph-users > > <ceph-users(a)ceph.io> wrote: > > > > > > Hi Yuri, > > > > > > On Fri, Nov 21, 2025 at 4:34 AM Yuri Weinstein <yweinste(a)redhat.com> > wrote: > > > > > > > > Note: this is an attempt to test and publish post EOL reef point > release > > > > > > > > Details of this release are summarized here: > > > > > > > > https://tracker.ceph.com/issues/73906#note-1 > > > > Release Notes - TBD > > > > LRC upgrade - N/A > > > > > > > > distro => jammy focal centos9 windows bookworm > > > > > > > > Seeking approvals/reviews for: > > > > > > > > smoke - Laura approved? > > > > > > > > rados - Radek, Laura approved? Travis? Nizamudeen? Adam King > approved? > > > > (rados is also running on build 2 == build 1 + > > > > https://github.com/ceph/ceph/pull/66056 per Igor's request) > > > > > > > > rgw - Adam E approved? > > > > > > > > fs - Venky approved? > > > > > > There are lot many failures in the fs suite resembling > > > > > > Error mounting /var/lib/ceph: mount: /var/lib/ceph: wrong fs > > > type, bad option ... > > > > > > which is an issue with the testing kernel. Adding Alex who was looking > > > into this issue. > > > > > > As far as the run validation is concerned, it's risky to ignore the > > > large number of failures. We have two options: > > > > > > - fix the testing kernel issue and rerun > > > - rerun with `for-linus` branch (with `-k for-linus` in the > > > teuthology-suite schedule command) > > > > > > The latter would be using an older kernel branch, but I think it > > > should suffice for this case. > > > > Hi Venky, > > > > With your report of the fs suite being affected as well, I carved out > > some time to take a look. > > > > I think it's happening because XFS maintainers disabled support for V4 > > filesystems that was deprecated 5 years ago -- as of 6.18 kernel it's > > no longer compiled in by default. The proper fix would be to create an > > image with/move to V5 on-disk format; I have opened [1] to try to > > achieve that. We can also re-enable V4 on-disk format in our kernel > > builds as a workaround. > > > > [1] https://github.com/ceph/ceph-cm-ansible/pull/799 > > This should be fixed now. Yuri, please schedule a rerun for krbd. > > Thanks, > > Ilya > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: reef 18.2.8 QE validation status
by Ilya Dryomov 08 Dec '25

08 Dec '25

On Mon, Dec 8, 2025 at 5:00 PM Venky Shankar via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi Yuri, > > On Fri, Nov 21, 2025 at 4:34 AM Yuri Weinstein <yweinste(a)redhat.com> wrote: > > > > Note: this is an attempt to test and publish post EOL reef point release > > > > Details of this release are summarized here: > > > > https://tracker.ceph.com/issues/73906#note-1 > > Release Notes - TBD > > LRC upgrade - N/A > > > > distro => jammy focal centos9 windows bookworm > > > > Seeking approvals/reviews for: > > > > smoke - Laura approved? > > > > rados - Radek, Laura approved? Travis? Nizamudeen? Adam King approved? > > (rados is also running on build 2 == build 1 + > > https://github.com/ceph/ceph/pull/66056 per Igor's request) > > > > rgw - Adam E approved? > > > > fs - Venky approved? > > There are lot many failures in the fs suite resembling > > Error mounting /var/lib/ceph: mount: /var/lib/ceph: wrong fs > type, bad option ... > > which is an issue with the testing kernel. Adding Alex who was looking > into this issue. > > As far as the run validation is concerned, it's risky to ignore the > large number of failures. We have two options: > > - fix the testing kernel issue and rerun > - rerun with `for-linus` branch (with `-k for-linus` in the > teuthology-suite schedule command) > > The latter would be using an older kernel branch, but I think it > should suffice for this case. Hi Venky, With your report of the fs suite being affected as well, I carved out some time to take a look. I think it's happening because XFS maintainers disabled support for V4 filesystems that was deprecated 5 years ago -- as of 6.18 kernel it's no longer compiled in by default. The proper fix would be to create an image with/move to V5 on-disk format; I have opened [1] to try to achieve that. We can also re-enable V4 on-disk format in our kernel builds as a workaround. [1] https://github.com/ceph/ceph-cm-ansible/pull/799 Thanks, Ilya

1 1

Re: SMB on ARM
by John Mulligan 08 Dec '25

08 Dec '25

On Monday, December 8, 2025 10:02:56 AM Eastern Standard Time Daniel Brown via ceph-users wrote: > Upgraded to tentacle over the weekend - mostly a smooth process. Had one > node that was a bit pokey and had to manually download the ceph container > image, but other than that, all good. > > I did try to setup SMB — unfortunately, my cluster is running on Arm64 > machines, and the SMB setup appears to be trying to start an Amd64 > container. Didn’t see a lot of logging, but after some digging around under > /var/lib/ceph/[my cluster id here]/… > > > $ cat unit.image > > quay.io/samba.org/samba-server:ceph20-centos-amd64 > > > Don’t suppose there’s a way to update that for the correct architecture? And > or to document it if it isn’t going to be possible? Thanks for trying the SMB support out. Unfortunately, the samba container images [1] do not support ARM64 at this time. We worked on it a while ago but got pulled off of that onto other tasks and never really resumed working on arm support. I have filed an issue to revisit arm64 support for samba container images on that project. [1] https://github.com/samba-in-kubernetes/samba-container

1 0

Re: SMB on ARM
by Robert Sander 08 Dec '25

08 Dec '25

Hi, Am 08.12.25 um 4:02 PM schrieb Daniel Brown via ceph-users: > > I did try to setup SMB The SMB service and the SMB MGR module are far from being production ready. I only get errors when trying to use them on AMD64. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

Re: How to fix BLUESTORE_FREE_FRAGMENTATION:
by Anthony D'Atri 08 Dec '25

08 Dec '25

> On Dec 8, 2025, at 4:07 AM, Olivier Delcourt via ceph-users <ceph-users(a)ceph.io> wrote: > > > >> Le 8 déc. 2025 à 09:36, Robert Sander via ceph-users <ceph-users(a)ceph.io> a écrit : >> >> Hi, >> >> Am 08.12.25 um 9:32 AM schrieb Nico Schottelius via ceph-users: >> >>> since migrating to Squid (19.2.3), I get the following health warning: >>> [WRN] BLUESTORE_FREE_FRAGMENTATION: 1 OSD(s) >>> osd.25 0.806301 How large is this OSD? >>> However while this is a warning, there seems to be no documented way of >>> fixing it. >>> What do I do with this warning / how to defrag the bluestore filesystem? >> >> Even the IBM documentation recommends to contact the IBM support in this case: https://www.ibm.com/docs/en/storage-ceph/8.1.0?topic=bluestore-fragmentatio… >> >> My best bet is to just recreate the OSD. > > Before that, what about trying to empty the OSD ( ceph osd reweight XXX 0) and when empty, put it back to business ( ceph osd reweight XXX 1) ? I suspect that would do the trick, but would suggest instead "crush reweigh" so that the balancer doesn't get confused.

1 0

SMB on ARM
by Daniel Brown 08 Dec '25

08 Dec '25

Upgraded to tentacle over the weekend - mostly a smooth process. Had one node that was a bit pokey and had to manually download the ceph container image, but other than that, all good. I did try to setup SMB — unfortunately, my cluster is running on Arm64 machines, and the SMB setup appears to be trying to start an Amd64 container. Didn’t see a lot of logging, but after some digging around under /var/lib/ceph/[my cluster id here]/… $ cat unit.image quay.io/samba.org/samba-server:ceph20-centos-amd64 Don’t suppose there’s a way to update that for the correct architecture? And or to document it if it isn’t going to be possible?

1 0

dashboard dial tcp connection refused
by lejeczek 08 Dec '25

08 Dec '25

Hi guys. I've no _ssl_ used by _dashboard_ -> $ ceph mgr services { "dashboard": "http://10.1.1.63:8080/", "prometheus": "http://10.1.1.63:9283/" } -> $ ceph config get mgr mgr/dashboard/ssl false and the 'standby' servers logs: ... Dec 08 11:33:39 ceph-9f4f9dba-72c7-11f0-8052-525400519d29-alertmanager-podster1[2503]: ts=2025-12-08T11:33:39.781Z caller=dispatch.go:353 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"<redacted>\": dial tcp 10.1.1.63:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"<redacted>\": dial tcp 10.1.1.61:8443: connect: connection refused" Dec 08 11:33:39 ceph-9f4f9dba-72c7-11f0-8052-525400519d29-alertmanager-podster1[2503]: ts=2025-12-08T11:33:39.781Z caller=notify.go:848 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] aggrGroup="{}/{}:{alertname=\"CephHealthWarning\"}" msg="Notify attempt failed, will retry later" attempts=1 err="Post \"<redacted>\": dial tcp 10.1.1.63:8443: connect: connection refused" ... with the snippet of the logs above _ceph_ says only: -> $ ceph health detail HEALTH_WARN mon podster1 is low on available space [WRN] MON_DISK_LOW: mon podster1 is low on available space mon.podster1 has 12% avail nothing listens on 8443 anywhere, which is as it should be - why standby server wants that - would anybody know? 10.1.1.61 is standby's own IP. many thanks, L.

1 0

Re: radosgw-19.2.3 STS AssumeRoleWithWebIdentity and OIDC namespace
by Boris 08 Dec '25

08 Dec '25

Hi Pritha, we are using keycloak and keystone to federate users. Users are able to create keystone projects and use the EC2 credentials to authenticate with radosgw. We also have a panel where a user can access their buckets. Typical cloud provider stuff. The panel is a client side JS application that used to fetch all the EC2 credentials from keystone and use the first set to do authenticate with radosgw. But because of security concerns we don't want keystone to be public reachable. So we implemented STS with AssumeRoleWithWebIdentity. The user authenticates via keycloak and receives some temporary credentials. We've set a custom user attribute that contains the and map this attribute to the token claim with sub. This attribute is just a string field which will get populated by a middleware, when the user selects the project. We've tried to create the bucket with the STS credentials and access them with the EC2 credentials and vice versa. This was working with 19.2.2 when the keystone und sts uid matched. The buckets that are created via the STS credentials belong to the UID "$oidc$KEYSTONE_PROJECT_ID" while the buckets created via EC2 credentials belong to the UID "KEYSTONE_PROJECT_ID". We also tested this with a local user. We will gather the logs later today. But those basically say "Authentication worked, you don't have access to the bucket of the other uid". I've created a tracker for this problem, where I will add the logs later: https://tracker.ceph.com/issues/74131 Here is how we proceeded with our initial setup that worked with 19.2.2: https://tracker.ceph.com/issues/74131?issue_count=3&issue_position=1&next_i… Am Mo., 8. Dez. 2025 um 07:08 Uhr schrieb Pritha Srivastava via ceph-users < ceph-users(a)ceph.io>: > Hi Boris, > > Is it possible for you to elaborate on what problem you are facing? like > which credentials you used to create the bucket? and check rgw logs and see > what the error is, when you try to access the bucket? > > Thanks, > Pritha > > On Fri, Dec 5, 2025 at 5:19 PM Boris <bb(a)kervyn.de> wrote: > > > Hi, > > since the last squid update (19.2.3) we have the problem, that the STS > > integration does not work anymore. (relevant tracker: > > https://tracker.ceph.com/issues/69924) > > > > Now there is a user with the $oidc$ prefix with the same suffix as the > > original user name. For example: > > > > >"$oidc$e0a0eed4f6a64c9cad70b69625dccba8", > > >"e0a0eed4f6a64c9cad70b69625dccba8", > > > > and both user are treated as different users in radosgw. > > > > We user STS with AssumeRoleWithWebIdentity, to make bucket management > > possible via a website, and didn't want to fetch all the access > credentials > > just to pick one and authenticate with that. > > > > Now when we create a bucket in the panel, we don't have access via the > > normal EC2 credentials anymore. > > > > I still think I am holding it wrong. > > This is the role and policy we've set: > > > > > # radosgw-admin role get --role-name=S3Access > > > { > > > "RoleId": "49d0d470-dc7a-4ffe-8db3-4f40cb82ebfd", > > > "RoleName": "S3Access", > > > "Path": "/", > > > "Arn": "arn:aws:iam:::role/S3Access", > > > "CreateDate": "2025-08-12T08:31:11.761Z", > > > "MaxSessionDuration": 3600, > > > "AssumeRolePolicyDocument": > > > "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":{\"Federated\":[\"arn:aws:iam:::oidc-provider/vv.xx.yy.zz:8443/realms/snc-customera\"]},\"Action\":[\"sts:AssumeRoleWithWebIdentity\"],\"Condition\":{\"StringEquals\":{\"vv.xx.yy.zz:8443/realms/snc-customera:app_id\":\"account\"}}}]}", > > > "PermissionPolicies": [ > > > { > > > "PolicyName": "Policy1", > > > "PolicyValue": > > > "{\"Version\":\"2012-10-17\",\"Statement\":{\"Effect\":\"Allow\",\"Action\":\"s3:*\",\"Resource\":\"arn:aws:s3:::*\"}}" > > > } > > > ] > > > } > > > > Best wishes > > Boris > > -- > > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im > > groÃƒ¼en Saal. > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

1 0

rgw index auto resharding not acting
by Johan Thomsen 08 Dec '25

08 Dec '25

Hi For some reason, the auto-resharding is not doing anything, even though it reports fill status over 100%. See below: $ radosgw-admin bucket limit check --rgw-zonegroup=k1-objectstore --rgw-zone=k1-objectstore 2025-12-08T08:45:57.143+0000 7f8921528900 0 ERROR: current period 43e92ddb-d0d2-4fbc-b18b-dc87126a7664 does not contain zone id 1da0f602-9397-4997-8e76-3e3425468746 2025-12-08T08:45:57.166+0000 7f8921528900 0 period (43e92ddb-d0d2-4fbc-b18b-dc87126a7664 does not have zone 1da0f602-9397-4997-8e76-3e3425468746 configured [ { "user_id": "k1-ceph-user", "buckets": [ { "bucket": "k1-snapshots", "tenant": "", "num_objects": 5766564, "num_shards": 97, "objects_per_shard": 59449, "fill_status": "OVER 114%" } ] } Here's a dump of related config: rgw_dynamic_resharding true rgw_max_dynamic_shards 1999 rgw_max_objs_per_shard 50000 rgw_md_log_max_shards 64 rgw_objexp_hints_num_shards 127 rgw_override_bucket_index_max_shards 1031 rgw_reshard_batch_size 64 rgw_reshard_bucket_lock_duration 360 rgw_reshard_max_aio 128 rgw_reshard_num_logs 16 rgw_reshard_thread_interval 600 rgw_safe_max_objects_per_shard 52000 rgw_shard_warning_threshold 90.000000 rgw_usage_max_shards 32 Does anyone have an idea why it is not auto-resharding to a value below at least 52k objs per shard?

1 0

Re: How to fix BLUESTORE_FREE_FRAGMENTATION:
by Olivier Delcourt 08 Dec '25

08 Dec '25

> Le 8 déc. 2025 à 09:36, Robert Sander via ceph-users <ceph-users(a)ceph.io> a écrit : > > Hi, > > Am 08.12.25 um 9:32 AM schrieb Nico Schottelius via ceph-users: > >> since migrating to Squid (19.2.3), I get the following health warning: >> [WRN] BLUESTORE_FREE_FRAGMENTATION: 1 OSD(s) >> osd.25 0.806301 >> However while this is a warning, there seems to be no documented way of >> fixing it. >> What do I do with this warning / how to defrag the bluestore filesystem? > > Even the IBM documentation recommends to contact the IBM support in this case: https://www.ibm.com/docs/en/storage-ceph/8.1.0?topic=bluestore-fragmentatio… > > My best bet is to just recreate the OSD. Before that, what about trying to empty the OSD ( ceph osd reweight XXX 0) and when empty, put it back to business ( ceph osd reweight XXX 1) ? Olivier

1 0

Re: How to fix BLUESTORE_FREE_FRAGMENTATION:
by Robert Sander 08 Dec '25

08 Dec '25

Hi, Am 08.12.25 um 9:32 AM schrieb Nico Schottelius via ceph-users: > since migrating to Squid (19.2.3), I get the following health warning: > > [WRN] BLUESTORE_FREE_FRAGMENTATION: 1 OSD(s) > osd.25 0.806301 > > However while this is a warning, there seems to be no documented way of > fixing it. > > What do I do with this warning / how to defrag the bluestore filesystem? Even the IBM documentation recommends to contact the IBM support in this case: https://www.ibm.com/docs/en/storage-ceph/8.1.0?topic=bluestore-fragmentatio… My best bet is to just recreate the OSD. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

How to fix BLUESTORE_FREE_FRAGMENTATION:
by Nico Schottelius 08 Dec '25

08 Dec '25

Good morning, since migrating to Squid (19.2.3), I get the following health warning: [WRN] BLUESTORE_FREE_FRAGMENTATION: 1 OSD(s) osd.25 0.806301 However while this is a warning, there seems to be no documented way of fixing it. What do I do with this warning / how to defrag the bluestore filesystem? BR, Nico -- Sustainable and modern Infrastructures by ungleich.ch

1 0

radosgw-19.2.3 STS AssumeRoleWithWebIdentity and OIDC namespace
by Boris 08 Dec '25

08 Dec '25

Hi, since the last squid update (19.2.3) we have the problem, that the STS integration does not work anymore. (relevant tracker: https://tracker.ceph.com/issues/69924) Now there is a user with the $oidc$ prefix with the same suffix as the original user name. For example: >"$oidc$e0a0eed4f6a64c9cad70b69625dccba8", >"e0a0eed4f6a64c9cad70b69625dccba8", and both user are treated as different users in radosgw. We user STS with AssumeRoleWithWebIdentity, to make bucket management possible via a website, and didn't want to fetch all the access credentials just to pick one and authenticate with that. Now when we create a bucket in the panel, we don't have access via the normal EC2 credentials anymore. I still think I am holding it wrong. This is the role and policy we've set: > # radosgw-admin role get --role-name=S3Access > { > "RoleId": "49d0d470-dc7a-4ffe-8db3-4f40cb82ebfd", > "RoleName": "S3Access", > "Path": "/", > "Arn": "arn:aws:iam:::role/S3Access", > "CreateDate": "2025-08-12T08:31:11.761Z", > "MaxSessionDuration": 3600, > "AssumeRolePolicyDocument": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":{\"Federated\":[\"arn:aws:iam:::oidc-provider/vv.xx.yy.zz:8443/realms/snc-customera\"]},\"Action\":[\"sts:AssumeRoleWithWebIdentity\"],\"Condition\":{\"StringEquals\":{\"vv.xx.yy.zz:8443/realms/snc-customera:app_id\":\"account\"}}}]}", > "PermissionPolicies": [ > { > "PolicyName": "Policy1", > "PolicyValue": "{\"Version\":\"2012-10-17\",\"Statement\":{\"Effect\":\"Allow\",\"Action\":\"s3:*\",\"Resource\":\"arn:aws:s3:::*\"}}" > } > ] > } Best wishes Boris -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

2 1

Re: ceph mon disable_stretch_mode...no valid command found!
by Maximilian Kaya 06 Dec '25

06 Dec '25

It's solved! the ceph reef documentation was ahead of its capability. -- BR Max > On 5. Dec 2025, at 12:37, Maximilian Kaya via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi, > due to ceph reef documentation I need to run only this command but I am unable to run this to exit the stretched cluster: > ceph mon disable_stretch_mode distributed_cluster_rule --yes-i-really-mean-it > ... > no valid command found; 10 closest matches: > ... > Error EINVAL: invalid command > … > > what do I am missing ? I have ceph 18.2.7 with 5 mons and I would like to extend datacenter by adding dc3 with same number of racks,hosts and osds to get back to repsize 3 using crush rule distributed_cluster_rule: > > rule distributed_cluster_rule { > id 3 > type replicated > step take default > step choose firstn 0 type datacenter > step chooseleaf firstn 1 type rack > step emit > } > .. > #ceph mon dump > epoch 87 > fsid 583647c2-4603-11ef-9941-b360866a782f > min_mon_release 18 (reef) > election_strategy: 1 > stretch_mode_enabled 1 > tiebreaker_mon mon3 > disallowed_leaders mon3 > 0: [v2:172.16.1.1:3300/0,v1:172.16.1.1:6789/0] mon.mon1; crush_location {datacenter=dc1} > 1: [v2:172.16.1.2:3300/0,v1:172.16.1.2:6789/0] mon.mon2; crush_location {datacenter=dc2} > 2: [v2:172.16.1.3:3300/0,v1:172.16.1.3:6789/0] mon.mon3; crush_location {datacenter=quorum_room} > 3: [v2:172.16.1.4:3300/0,v1:172.16.1.4:6789/0] mon.mon4; crush_location {datacenter=dc2} > 4: [v2:172.16.1.5:3300/0,v1:172.16.1.5:6789/0] mon.mon5; crush_location {datacenter=dc1} > > > is there any other way to exit stretched cluster? any help would be appreciated. > > BR > Max > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Tentacle 20.2.0 NFS ganesh and mgmt-gateway - The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8)
by Salatiel Filho 05 Dec '25

05 Dec '25

Are there any plans to make a linux/arm64/v8 container image for quay.io/ceph/keepalived:2.2.4, quay.io/ceph/haproxy:2.3 and quay.io/ceph/nginx:sclorg-nginx-126 ? Currently it is not possible to start a HA NFS Ganesh in ceph or even the management gateway on ARM devices. They all complain about: WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested Atenciosamente/Kind regards, Salatiel

1 0

Re: Fwd: On your post: "[Tentacle 20.2.0]: Inconsistent pg's after enabling ec optimisation flag"
by Jeff Bailey 05 Dec '25

05 Dec '25

I've seen OSD's crash when running an fstrim on an RBD with an EC data pool with the EC optimizations enabled. Always two OSD's. It's a 8+2 pool but I don't know if the "2"s are a coincidence or not. Other than that, I haven't seen any OSD crashes with EC optimizations. We had the same size mismatches that went away after migrating the data. On 12/5/2025 10:34 AM, Reto Gysi via ceph-users wrote: > Hi Bill > > Am Fr., 5. Dez. 2025 um 10:02 Uhr schrieb Bill Scales < > bill_scales(a)uk.ibm.com>: > >> Hi Reto, >> >> Sorry to hear about your problems with turning on ec optimizations. I've >> led the team that developed this feature, so we are keen to help understand >> what happened to your system. >> >> Your configuration looks fine as far as being supported with ec >> optimizations. The daemons (mons, osds and mgr) need to be running tentacle >> code to use this feature, there is no requirement to update any of your >> clients. >> >> Do you have any more examples of the inconsistent objects logs that you >> can share with me? >> >> There was a bug that we fixed late in the development cycle which was >> scrubbing incorrectly reporting a size mismatch for objects written prior >> to turning on optimizations. This is because prior to turning on >> optimizations objects are padded to a multiple of the stripe_width, >> afterwards objects are no longer padded. The scrub code was occasionally >> getting confused and incorrectly reporting an inconsistency. In this case >> the scrub was reporting false positive errors - there was nothing wrong >> with the data and no problems accessing the data. I notice the log you >> shared in the email was for a size mismatch, I'm interested whether all the >> logs were for mismatched sizes. We will do some further work to confirm >> that the fix is in the 20.2.0 tentacle release and that there are no >> problems with the back port. >> > As far I could see the errors where all about the mismatched sizes. > > >> You also mention that you saw some OSD crashes. Do you have any further >> information about these? >> > I just could cause another crash with starting up a windows VM, and doing a > WIndows file history backup to a drive that is on an RBD Image on pool > rbd_ecpool where I had the allow_ec_optimization flag enabled. > > <disk type="network" device="disk"> > <driver name="qemu" type="raw" cache="writethrough" io="threads" > discard="unmap" detect_zeroes="unmap"/> > <auth username="admin"> > <secret type="ceph" uuid="878b0bc5-c471-4ec6-a92a-f65282ffbdf6"/> > </auth> > <source protocol="rbd" name="rbd/game_windows_backup_drive" index="3"> > <host name="zephir" port="6789"/> > </source> > <target dev="vdd" bus="virtio"/> > <alias name="virtio-disk3"/> > <address type="pci" domain="0x0000" bus="0x00" slot="0x02" > function="0x0"/> > </disk> > > root@zephir:~# ceph -s > cluster: > id: 27923302-87a5-11ec-ac5b-976d21a49941 > health: HEALTH_WARN > 2 osds down > Reduced data availability: 49 pgs inactive > Degraded data redundancy: 15722192/79848340 objects degraded > (19.690%), 246 pgs degraded, 247 pgs undersized > > services: > mon: 3 daemons, quorum zephir,debian,raspi (age 20h) [leader: > zephir] > mgr: zephir.enywvy(active, since 21h), standbys: debian.nusuye > mds: 3/3 daemons up, 3 standby > osd: 18 osds: 16 up (since 5m), 18 in (since 3d); 8 remapped > pgs > cephfs-mirror: 2 daemons active (2 hosts) > rbd-mirror: 2 daemons active (2 hosts) > rgw: 1 daemon active (1 hosts, 1 zones) > tcmu-runner: 5 portals active (2 hosts) > > data: > volumes: 3/3 healthy > pools: 25 pools, 450 pgs > objects: 17.52M objects, 62 TiB > usage: 116 TiB used, 63 TiB / 179 TiB avail > pgs: 10.889% pgs not active > 15722192/79848340 objects degraded (19.690%) > 197 active+undersized+degraded > 183 active+clean > 49 undersized+degraded+peered > 15 active+clean+scrubbing > 5 active+clean+scrubbing+deep > 1 active+undersized > > io: > client: 1.7 KiB/s rd, 1023 B/s wr, 1 op/s rd, 0 op/s wr > > root@zephir:~# ceph osd status > ID HOST USED AVAIL WR OPS WR DATA RD OPS RD DATA STATE > 0 debian 11.2T 5289G 0 0 0 0 exists,up > 1 zephir 11.6T 4846G 0 0 0 0 exists > 2 zephir 10.7T 5827G 0 0 2 4 exists,up > 3 zephir 11.0T 5497G 0 0 0 0 exists > 4 zephir 159G 1376G 0 0 2 0 exists,up > 5 zephir 120G 1415G 0 0 0 0 exists,up > 6 debian 11.3T 5200G 0 0 1 0 exists,up > 7 zephir 305G 1230G 0 0 0 0 exists,up > 8 zephir 10.9T 5594G 0 0 0 0 exists,up > 9 zephir 11.9T 4616G 0 0 0 0 exists,up > 10 zephir 235G 1300G 0 0 0 0 exists,up > 11 zephir 223G 1312G 0 0 0 0 exists,up > 12 zephir 11.0T 5479G 0 0 0 0 exists,up > 13 debian 118G 631G 0 0 3 8 exists,up > 14 debian 345G 1190G 0 0 0 0 exists,up > 15 debian 11.4T 5079G 0 0 0 0 exists,up > 16 debian 554G 3029G 0 0 3 8 exists,up > 17 debian 12.5T 5874G 0 0 1 0 exists,up > root@zephir:~# > > I attached the log of osd.1 to > https://filebin.net/jwvs6kuqrc7hx8id > > Best Regards, > > Reto > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Fwd: On your post: "[Tentacle 20.2.0]: Inconsistent pg's after enabling ec optimisation flag"
by Reto Gysi 05 Dec '25

05 Dec '25

Hi Bill Am Fr., 5. Dez. 2025 um 10:02 Uhr schrieb Bill Scales < bill_scales(a)uk.ibm.com>: > Hi Reto, > > Sorry to hear about your problems with turning on ec optimizations. I've > led the team that developed this feature, so we are keen to help understand > what happened to your system. > > Your configuration looks fine as far as being supported with ec > optimizations. The daemons (mons, osds and mgr) need to be running tentacle > code to use this feature, there is no requirement to update any of your > clients. > > Do you have any more examples of the inconsistent objects logs that you > can share with me? > > There was a bug that we fixed late in the development cycle which was > scrubbing incorrectly reporting a size mismatch for objects written prior > to turning on optimizations. This is because prior to turning on > optimizations objects are padded to a multiple of the stripe_width, > afterwards objects are no longer padded. The scrub code was occasionally > getting confused and incorrectly reporting an inconsistency. In this case > the scrub was reporting false positive errors - there was nothing wrong > with the data and no problems accessing the data. I notice the log you > shared in the email was for a size mismatch, I'm interested whether all the > logs were for mismatched sizes. We will do some further work to confirm > that the fix is in the 20.2.0 tentacle release and that there are no > problems with the back port. > As far I could see the errors where all about the mismatched sizes. > > You also mention that you saw some OSD crashes. Do you have any further > information about these? > I just could cause another crash with starting up a windows VM, and doing a WIndows file history backup to a drive that is on an RBD Image on pool rbd_ecpool where I had the allow_ec_optimization flag enabled. <disk type="network" device="disk"> <driver name="qemu" type="raw" cache="writethrough" io="threads" discard="unmap" detect_zeroes="unmap"/> <auth username="admin"> <secret type="ceph" uuid="878b0bc5-c471-4ec6-a92a-f65282ffbdf6"/> </auth> <source protocol="rbd" name="rbd/game_windows_backup_drive" index="3"> <host name="zephir" port="6789"/> </source> <target dev="vdd" bus="virtio"/> <alias name="virtio-disk3"/> <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0"/> </disk> root@zephir:~# ceph -s cluster: id: 27923302-87a5-11ec-ac5b-976d21a49941 health: HEALTH_WARN 2 osds down Reduced data availability: 49 pgs inactive Degraded data redundancy: 15722192/79848340 objects degraded (19.690%), 246 pgs degraded, 247 pgs undersized services: mon: 3 daemons, quorum zephir,debian,raspi (age 20h) [leader: zephir] mgr: zephir.enywvy(active, since 21h), standbys: debian.nusuye mds: 3/3 daemons up, 3 standby osd: 18 osds: 16 up (since 5m), 18 in (since 3d); 8 remapped pgs cephfs-mirror: 2 daemons active (2 hosts) rbd-mirror: 2 daemons active (2 hosts) rgw: 1 daemon active (1 hosts, 1 zones) tcmu-runner: 5 portals active (2 hosts) data: volumes: 3/3 healthy pools: 25 pools, 450 pgs objects: 17.52M objects, 62 TiB usage: 116 TiB used, 63 TiB / 179 TiB avail pgs: 10.889% pgs not active 15722192/79848340 objects degraded (19.690%) 197 active+undersized+degraded 183 active+clean 49 undersized+degraded+peered 15 active+clean+scrubbing 5 active+clean+scrubbing+deep 1 active+undersized io: client: 1.7 KiB/s rd, 1023 B/s wr, 1 op/s rd, 0 op/s wr root@zephir:~# ceph osd status ID HOST USED AVAIL WR OPS WR DATA RD OPS RD DATA STATE 0 debian 11.2T 5289G 0 0 0 0 exists,up 1 zephir 11.6T 4846G 0 0 0 0 exists 2 zephir 10.7T 5827G 0 0 2 4 exists,up 3 zephir 11.0T 5497G 0 0 0 0 exists 4 zephir 159G 1376G 0 0 2 0 exists,up 5 zephir 120G 1415G 0 0 0 0 exists,up 6 debian 11.3T 5200G 0 0 1 0 exists,up 7 zephir 305G 1230G 0 0 0 0 exists,up 8 zephir 10.9T 5594G 0 0 0 0 exists,up 9 zephir 11.9T 4616G 0 0 0 0 exists,up 10 zephir 235G 1300G 0 0 0 0 exists,up 11 zephir 223G 1312G 0 0 0 0 exists,up 12 zephir 11.0T 5479G 0 0 0 0 exists,up 13 debian 118G 631G 0 0 3 8 exists,up 14 debian 345G 1190G 0 0 0 0 exists,up 15 debian 11.4T 5079G 0 0 0 0 exists,up 16 debian 554G 3029G 0 0 3 8 exists,up 17 debian 12.5T 5874G 0 0 1 0 exists,up root@zephir:~# I attached the log of osd.1 to https://filebin.net/jwvs6kuqrc7hx8id Best Regards, Reto >

1 0

ceph mon disable_stretch_mode...no valid command found!
by Maximilian Kaya 05 Dec '25

05 Dec '25

Hi, due to ceph reef documentation I need to run only this command but I am unable to run this to exit the stretched cluster: ceph mon disable_stretch_mode distributed_cluster_rule --yes-i-really-mean-it ... no valid command found; 10 closest matches: ... Error EINVAL: invalid command … what do I am missing ? I have ceph 18.2.7 with 5 mons and I would like to extend datacenter by adding dc3 with same number of racks,hosts and osds to get back to repsize 3 using crush rule distributed_cluster_rule: rule distributed_cluster_rule { id 3 type replicated step take default step choose firstn 0 type datacenter step chooseleaf firstn 1 type rack step emit } .. #ceph mon dump epoch 87 fsid 583647c2-4603-11ef-9941-b360866a782f min_mon_release 18 (reef) election_strategy: 1 stretch_mode_enabled 1 tiebreaker_mon mon3 disallowed_leaders mon3 0: [v2:172.16.1.1:3300/0,v1:172.16.1.1:6789/0] mon.mon1; crush_location {datacenter=dc1} 1: [v2:172.16.1.2:3300/0,v1:172.16.1.2:6789/0] mon.mon2; crush_location {datacenter=dc2} 2: [v2:172.16.1.3:3300/0,v1:172.16.1.3:6789/0] mon.mon3; crush_location {datacenter=quorum_room} 3: [v2:172.16.1.4:3300/0,v1:172.16.1.4:6789/0] mon.mon4; crush_location {datacenter=dc2} 4: [v2:172.16.1.5:3300/0,v1:172.16.1.5:6789/0] mon.mon5; crush_location {datacenter=dc1} is there any other way to exit stretched cluster? any help would be appreciated. BR Max

1 0

Re: reef 18.2.8 QE validation status
by Travis Nielsen 04 Dec '25

04 Dec '25

The latest Reef tests are passing for Rook. Travis On Mon, Dec 1, 2025 at 7:18 AM Ilya Dryomov via ceph-users < ceph-users(a)ceph.io> wrote: > On Sat, Nov 29, 2025 at 6:08 PM Yuri Weinstein <yweinste(a)redhat.com> > wrote: > > > > Rerun scheduled for krbd > > It's still the same. I'd suggest keeping an eye on the run before > "admitting" it for a possible approval -- something that is completely > red with not a single job reaching the point where the actual test > starts clearly can't be approved. > > Thanks, > > Ilya > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: On your post: "[Tentacle 20.2.0]: Inconsistent pg's after enabling ec optimisation flag"
by Reto Gysi 04 Dec '25

04 Dec '25

Hi David Sorry I forgot to answer your mail. I both wanted to do some tests first and then also got the flu. The current situation on my small "cluster" is: - After disabling any load on the affected pool , after enough times ceph pg repair and node restarts the inconsistent pg errors/warnings disappeared and the cluster came back into a clean state. - However as soon I started some qemu virtual machines (both a linux and a windows vm) I got crashing osds again and inconsistent pg's. I thought I saw it was mentioned that ec_optimizations would require any client library modifications. But maybe that is not the case for qemu? I haven't tested that yet. - writing to an ext4 fs on an RBD image mounted directly via mount -t ceph on one of the cluster nodes didn't lead to nearl immediate /to very soon osd crashes and inconsistent pg's. - As for now I copied most of the images back to an erasure coded pool without ec_optimization support enabled and there I can run the VM images again with qemu without osd crashing and inconsistent pgs. Regading your questions: - What EC scheme do you use? We are using 6+2 The erasure coded pool I've enabled ec_optimization on used jerasure 3+2 with lz4 compression. - Do you use cephadm? Yes, the cluster has been deployed from the beginning with cephadm - Do you run on via kubernetes? No it's running on baremetal with debian 13 as OS (debian 12 at start of year) - What is the operating system that you are using? Debian 13. The libs on the debian base system are from 18.2.7+ds-1 - What was the version you updated from? I've upgraded from 19.2.3 to 20.2.0. I've ran the cluster without any issues (asides from some dashboard bugs) for a week, before I tried to enable allow_ec_optimization flag on one of the erasure coded pools - What version did you use to create the pool? The original pool was created under 17.2.6 (quincy) back in 2023 - The effected pool, is it hard drives? Yes the pool is hard drives Regards Reto Am Do., 4. Dez. 2025 um 15:21 Uhr schrieb David Walter <david.walter(a)cern.ch >: > Hi Reto, > > I would be very thankful for a reply since it seems we have to solve > this on our own. > Do you have any new insights into the problem? > > Things that could help are: > > - What EC scheme do you use? We are using 6+2 > - Do you use cephadm? > - Do you run on via kubernetes? > - What is the operating system that you are using? > - What was the version you updated from? > - What version did you use to create the pool? > - The effected pool, is it hard drives? > > Best, > > David > > On 12/1/25 07:59, David Walter wrote: > > Hi Reto, > > > > I've seen your post on the ceph users list, unfortunately I was not > > able to reply on the thread or post something myself. > > > > I have the same problem. I was upgrading from ceph squid 19.2.3 to > > ceph tentacle 20.2.0 and all was good first. > > But when I enabled ec optimizations I get scrubbing errors like this: > > > > HEALTH_ERR 88909311 scrub errors; Possible data damage: 117 pgs > > inconsistent > > [ERR] OSD_SCRUB_ERRORS: 88909311 scrub errors > > [ERR] PG_DAMAGED: Possible data damage: 117 pgs inconsistent > > > > and I was unable repairing them. Some errors disappear after > > restarting all osd daemons but they come back after another scrubbing. > > I don't have any indication that data is lost so far. But the cluster > > is under pressure and daemons crash, so it's in a critical state. > > > > When running "rados list-inconsistent-obj <PG>" one some PGs I get > > errors like these: > > > > > {"osd":62,"primary":false,"shard":4,"errors":["size_mismatch_info","obj_size_info_mismatch"],"size":700416,"object_info":{"oid":{"oid":"1000793c33a.000000dd","key":"","snapid":-2,"hash":2213519396,"max":0,"pool":4,"namespace":""},"version":"43143'3253925","prior_version":"0'0","last_reqid":"client.1822828.0:160762343","user_version":3253925,"size":4194304,"mtime":"2025-11-10T22:13:15.257031-0500","local_mtime":"2025-11-10T22:14:44.294560-0500","lost":0,"flags":["dirty","data_digest"],"truncate_seq":0,"truncate_size":0,"data_digest":"0x7e22aafe","omap_digest":"0xffffffff","expected_object_size":0,"expected_write_size":0,"alloc_hint_flags":0,"manifest":{"type":0},"watchers":{}}},{"osd":65,"primary":false,"shard":3,"errors":[],"size":700416} > > > > > > > Did you find out more about this issue or how do you plan to move > > forward? > > > > Best, > > > > David > > >

1 0

Re: [ceph-users] Re: Stuck v2 notification migration
by Lázár György 04 Dec '25

04 Dec '25

Thank you so much! This worked. This issue kept me busy for about a week, since the rados objects backing the old pubsub framework are not really documented. I'm impressed that I got a highly technical and correct answer so quickly. Since other people might be using multisite the same way we did (with creating the same topic on both sites), this might be worth mentioning in the docs somewhere. Regards, György Lázár Csütörtök, December 04, 2025 15:30 CET dátummal, Casey Bodley via ceph-users <ceph-users(a)ceph.io> ezt írta: > (cc Yuval and Ali) > > the problem is that v1 topics did not support multisite, so i'm > guessing you had to recreate those same topics on each zone. the v2 > migration code can only run on the primary zone, so multisite can't > automatically reconcile those v1 topics on your secondary zone > > > WARNING: topic migration in process. please try again later > > you're seeing this warning on the secondary zone because the v1 topics > still exist there. that's also why "radosgw-admin topic list" is only > showing the v1 topics > > all v1 topics were stored in a single rados object per tenant, named > "pubsub.{tenant}" in the site2.rgw.log pool. if you're not using > tenants, that would just be "pubsub." > > since the v2 topic metadata has already replicated, it should be safe > to delete any of those v1 rados objects manually to complete that v2 > migration: > > 1. first make a local backup: > > $ rados -p site2.rgw.log get pubsub. pubsub.bak > > 2. delete the rados object: > > $ rados -p site2.rgw.log rm pubsub. > > 3. verify that v2 topics are now visible to radosgw-admin: > > $ radosgw-admin topic list > > 4. restart radosgw instances that are still caching the v1 topic metadata > > repeat steps 1 and 2 for other tenant names as needed. if necessary, > you can restore the v1 metadata with: > > $ rados -p site2.rgw.log put pubsub. pubsub.bak > > > On Thu, Dec 4, 2025 at 6:08 AM Lázár György via ceph-users > <ceph-users(a)ceph.io> wrote: > > > > Hello, > > I have an object multisite setup. I'm on squid 19.2.3 (upgraded, not a greenfield deployment - both clusters are on the same builds), so I upgraded to notification_v2 as well. However, while the primary zone converted the notifications and topics to v2 easily, the secondary zone seems to have issues with this (topics and notifications are not synced to the secondary, but the rados objects are synced, shown via rados -p site2.rgw.log -N notif ls. The radosgw-admin topic list command also gives back the v1 format of the json output on the secondary zone, while having the v2 json output on the primary). What's (almost) certain is, that one of the clusters (the primary) wants to use v2, and the other the v1 notification style, and mixing these two causes issues (topics randomly breaking, etc). Also, since 19.2.3, in theory, if I create a topic on the secondary zone, the operation should be forwarded to the primary zone. While this happens, the topic creation still fails with an 503 error, while the topic still gets created. The rgw on the secondary site after this half-failed operation also spits out logs like: > > rgw debug 2025-11-26T11:51:14.241+0000 7ffa54948640 1 req 12442569609481506669 0.000000000s sns:pubsub_topic_create ERROR: topic not found > > rgw debug 2025-11-26T11:51:14.254+0000 7ffa2e8fc640 1 req 12442569609481506669 0.013000983s sns:pubsub_topic_create WARNING: topic migration in process. please try again later > > I tried to create another multisite setup (same clusters, different realm), and there the notification v2 scheme works without an issue. I think if I would delete the secondary zone and re-create it (including the pools), it would fix the issue, but since that operation has a relatively large blast radius (especially if something goes wrong, but there are some services that use only the secondary zone, so they should be redirected beforehand), I'm a bit hesitant to do it (before the christmas holidays). I also tried searching for these error messages on google, with no luck. I also tried searching for these error messages in the RGW source code, and still no luck. I can find them in the compiled RGW binary at least. > > > > Any help would be greatly appreciated! I would be also very happy if I wouldn't have to nuke the secondary zone. > > > > Redards, > > György Lázár > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: bucket/user usage metrics (size, objects count)
by Konstantin Shalygin 04 Dec '25

04 Dec '25

Hi, https://github.com/blemmenes/radosgw_usage_exporter k Sent from my iPhone > On 4 Dec 2025, at 17:47, Mindaugas Mikučionis via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi, > > Running squid. What is best way to expose metrics per bucket/user usage > (size per bucket, object count per bucket, size per user, object count per > user) ? > As i see ceph-exporter exposes only per bucket/user ops counter metrics. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: [ceph-users] bucket/user usage metrics (size, objects count)
by Lázár György 04 Dec '25

04 Dec '25

Hi! Not sure about "the best way", but we are running this (with a few custom modifications): https://github.com/blemmenes/radosgw_usage_exporter Regards, György Lázár Csütörtök, December 04, 2025 15:47 CET dátummal, Mindaugas Mikučionis via ceph-users <ceph-users(a)ceph.io> ezt írta: > Hi, > > Running squid. What is best way to expose metrics per bucket/user usage > (size per bucket, object count per bucket, size per user, object count per > user) ? > As i see ceph-exporter exposes only per bucket/user ops counter metrics. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

bucket/user usage metrics (size, objects count)
by Mindaugas Mikučionis 04 Dec '25

04 Dec '25

Hi, Running squid. What is best way to expose metrics per bucket/user usage (size per bucket, object count per bucket, size per user, object count per user) ? As i see ceph-exporter exposes only per bucket/user ops counter metrics.

1 0

Re: Updating cephadm-managed Tentacle cluster from RC to production
by Michael Worsham 04 Dec '25

04 Dec '25

So I take it that you're all good now? -- Michael ________________________________ From: Marek Szuba via ceph-users <ceph-users(a)ceph.io> Sent: Wednesday, December 3, 2025 12:01 PM To: ceph-users(a)ceph.io <ceph-users(a)ceph.io> Subject: [ceph-users] Re: Updating cephadm-managed Tentacle cluster from RC to production This is an external email. Please take care when clicking links or opening attachments. When in doubt, check with the Help Desk or Security. On 2025-12-03 15:57, Marek Szuba via ceph-users wrote: > Well, things have got interesting now. The above appears to have worked > for 4 out of 5 mgr daemons - but now most "ceph orch" commands give me > the error "Error ENOENT: Module not found". I can also see many > cephadm-related "module not found" error in system logs. Progress! Having carefully searched through all the logs, I discovered the module was unable to load due to a spurious field in certificate JSON. A quick look at the relevant bit of Python code then allowed me to identify the field "editable", present in all six cert/key config keys, as the culprit. Having carefully edited and reset the offending JSON, I was then able to successfully run 'ceph orch upgrade start --ceph-version 20.2.0' (which redid the mgr and mon upgrade - that was intentional, in case the manual redeployment with new image missed something). This has just successfully completed. @Michael: I did see that bit of the documentation, I just had no clue which config keys exactly to examine and what to look for in them. Fortunately it turned out that some of the operations resulting in "Module not found" also crashed the active mgr (interestingly enough, it wasn't all such operations - Ceph only noted two mgr crashes), and I was able to find the traceback in crash metadata. -- MS _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments.

1 0

Re: Stuck v2 notification migration
by Casey Bodley 04 Dec '25

04 Dec '25

(cc Yuval and Ali) the problem is that v1 topics did not support multisite, so i'm guessing you had to recreate those same topics on each zone. the v2 migration code can only run on the primary zone, so multisite can't automatically reconcile those v1 topics on your secondary zone > WARNING: topic migration in process. please try again later you're seeing this warning on the secondary zone because the v1 topics still exist there. that's also why "radosgw-admin topic list" is only showing the v1 topics all v1 topics were stored in a single rados object per tenant, named "pubsub.{tenant}" in the site2.rgw.log pool. if you're not using tenants, that would just be "pubsub." since the v2 topic metadata has already replicated, it should be safe to delete any of those v1 rados objects manually to complete that v2 migration: 1. first make a local backup: $ rados -p site2.rgw.log get pubsub. pubsub.bak 2. delete the rados object: $ rados -p site2.rgw.log rm pubsub. 3. verify that v2 topics are now visible to radosgw-admin: $ radosgw-admin topic list 4. restart radosgw instances that are still caching the v1 topic metadata repeat steps 1 and 2 for other tenant names as needed. if necessary, you can restore the v1 metadata with: $ rados -p site2.rgw.log put pubsub. pubsub.bak On Thu, Dec 4, 2025 at 6:08 AM Lázár György via ceph-users <ceph-users(a)ceph.io> wrote: > > Hello, > I have an object multisite setup. I'm on squid 19.2.3 (upgraded, not a greenfield deployment - both clusters are on the same builds), so I upgraded to notification_v2 as well. However, while the primary zone converted the notifications and topics to v2 easily, the secondary zone seems to have issues with this (topics and notifications are not synced to the secondary, but the rados objects are synced, shown via rados -p site2.rgw.log -N notif ls. The radosgw-admin topic list command also gives back the v1 format of the json output on the secondary zone, while having the v2 json output on the primary). What's (almost) certain is, that one of the clusters (the primary) wants to use v2, and the other the v1 notification style, and mixing these two causes issues (topics randomly breaking, etc). Also, since 19.2.3, in theory, if I create a topic on the secondary zone, the operation should be forwarded to the primary zone. While this happens, the topic creation still fails with an 503 error, while the topic still gets created. The rgw on the secondary site after this half-failed operation also spits out logs like: > rgw debug 2025-11-26T11:51:14.241+0000 7ffa54948640 1 req 12442569609481506669 0.000000000s sns:pubsub_topic_create ERROR: topic not found > rgw debug 2025-11-26T11:51:14.254+0000 7ffa2e8fc640 1 req 12442569609481506669 0.013000983s sns:pubsub_topic_create WARNING: topic migration in process. please try again later > I tried to create another multisite setup (same clusters, different realm), and there the notification v2 scheme works without an issue. I think if I would delete the secondary zone and re-create it (including the pools), it would fix the issue, but since that operation has a relatively large blast radius (especially if something goes wrong, but there are some services that use only the secondary zone, so they should be redirected beforehand), I'm a bit hesitant to do it (before the christmas holidays). I also tried searching for these error messages on google, with no luck. I also tried searching for these error messages in the RGW source code, and still no luck. I can find them in the compiled RGW binary at least. > > Any help would be greatly appreciated! I would be also very happy if I wouldn't have to nuke the secondary zone. > > Redards, > György Lázár > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Stuck v2 notification migration
by Lázár György 04 Dec '25

04 Dec '25

Hello, I have an object multisite setup. I'm on squid 19.2.3 (upgraded, not a greenfield deployment - both clusters are on the same builds), so I upgraded to notification_v2 as well. However, while the primary zone converted the notifications and topics to v2 easily, the secondary zone seems to have issues with this (topics and notifications are not synced to the secondary, but the rados objects are synced, shown via rados -p site2.rgw.log -N notif ls. The radosgw-admin topic list command also gives back the v1 format of the json output on the secondary zone, while having the v2 json output on the primary). What's (almost) certain is, that one of the clusters (the primary) wants to use v2, and the other the v1 notification style, and mixing these two causes issues (topics randomly breaking, etc). Also, since 19.2.3, in theory, if I create a topic on the secondary zone, the operation should be forwarded to the primary zone. While this happens, the topic creation still fails with an 503 error, while the topic still gets created. The rgw on the secondary site after this half-failed operation also spits out logs like: rgw debug 2025-11-26T11:51:14.241+0000 7ffa54948640 1 req 12442569609481506669 0.000000000s sns:pubsub_topic_create ERROR: topic not found rgw debug 2025-11-26T11:51:14.254+0000 7ffa2e8fc640 1 req 12442569609481506669 0.013000983s sns:pubsub_topic_create WARNING: topic migration in process. please try again later I tried to create another multisite setup (same clusters, different realm), and there the notification v2 scheme works without an issue. I think if I would delete the secondary zone and re-create it (including the pools), it would fix the issue, but since that operation has a relatively large blast radius (especially if something goes wrong, but there are some services that use only the secondary zone, so they should be redirected beforehand), I'm a bit hesitant to do it (before the christmas holidays). I also tried searching for these error messages on google, with no luck. I also tried searching for these error messages in the RGW source code, and still no luck. I can find them in the compiled RGW binary at least. Any help would be greatly appreciated! I would be also very happy if I wouldn't have to nuke the secondary zone. Redards, György Lázár

1 0

Re: 500 - Internal Server Error ceph RGW Dashboard
by Nizamudeen A 04 Dec '25

04 Dec '25

Hi Vivien, the full_user_id is a key that exists in the latest rgw code (tentacle) but not present in the squid. https://github.com/ceph/ceph/pull/55451 But in the mgr code, the changes are made to the API and they exists in your cluster now but as the rgw is older the API cannot find the key which is why its raising the error here. So I think once you update the rgw, this should go away. Regards, Nizam On Wed, Dec 3, 2025 at 3:29 PM GLE, Vivien via ceph-users < ceph-users(a)ceph.io> wrote: > Hi, > > > I'm trying S3 on my cluster with a RGW in VM. My cluster is currently in > v20.2.0 and RGW is in v19.2.1. S3 is working fine the only problem is that > if I try to search (via RGW VM) for anything user relative (bucket or user > management) on the object ceph dashboard this search leads to 500 - > internal server error with those logs in the mgr > > > 2025-12-03T09:47:06.545+0000 700619b22640 0 [dashboard INFO request] > [::ffff:172.16.112.15:60786] [GET] [500] [0.155s] [admin] [513.0B] > /api/rgw/user/dashboard > 2025-12-03T09:47:06.585+0000 70061a323640 0 [dashboard ERROR exception] > Internal Server Error > Traceback (most recent call last): > File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 47, in > dashboard_exception_handler > return handler(*args, **kwargs) > File "/lib/python3.9/site-packages/cherrypy/_cpdispatch.py", line 54, in > __call__ > return self.callable(*self.args, **self.kwargs) > File "/usr/share/ceph/mgr/dashboard/controllers/_base_controller.py", > line 264, in inner > ret = func(*args, **kwargs) > File "/usr/share/ceph/mgr/dashboard/controllers/_rest_controller.py", > line 193, in wrapper > return func(*vpath, **params) > File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 929, in get > return self._get(uid, daemon_name=daemon_name, stats=stats) > File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 941, in > _get > result['uid'] = result['full_user_id'] > KeyError: 'full_user_id' > > I found a workaround with deploying a container RGW via cephadm v20.2.0 > and using it to manage my bucket and user. > > Does radosgw needs to be at the same version as the ceph cluster ? Or is > it a bug ? > > Thanks > > Vivien > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > -- Nizamudeen A Sr. Software Engineer - IBM Partner Engineer IBM and Red Hat Ceph Storage Red Hat <https://www.redhat.com/> <https://www.redhat.com/>

1 0

Overriding service templates
by Stolte, Felix 04 Dec '25

04 Dec '25

Hi guys, the ganesha.conf file renedered by the nfs service is missing some parameters i would like to use. Is it possible to override the jinja2 template used for rendering the file with my own? Greetings Felix ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------ Forschungszentrum Jülich GmbH 52425 Jülich Sitz der Gesellschaft: Jülich Eingetragen im Handelsregister des Amtsgerichts Düren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Stefan Müller Geschäftsführung: Prof. Dr. Astrid Lambrecht (Vorsitzende), Dr. Stephanie Bauer (stellv. Vorsitzende), Prof. Dr. Ir. Pieter Jansens, Prof. Dr. Laurens Kuipers ------------------------------------------------------------------------------------------------ ------------------------------------------------------------------------------------------------

1 0

[Tentacle 20.2.0]: Size mismatch errors after enabling EC optimization
by David Walter 03 Dec '25

03 Dec '25

Hi, I've recently upgraded from squid 19.2.3 to tentacle 20.2.0 on our 10 Alma 9.6 machines and first everything was good. When I enabled the EC optimizations via ceph osd pool set submit_ec_hdd allow_ec_optimizations true Errors started to appear, basically after each scrubbing, like these: HEALTH_ERR 88909311 scrub errors; Possible data damage: 117 pgs inconsistent [ERR] OSD_SCRUB_ERRORS: 88909311 scrub errors [ERR] PG_DAMAGED: Possible data damage: 117 pgs inconsistent and I was unable repairing them. Some errors disappear after restarting all osd daemons but they come back after another scrubbing. When running "rados list-inconsistent-obj <PG>" on some PGs I get errors like these: {"osd":62,"primary":false,"shard":4,"errors":["size_mismatch_info","obj_size_info_mismatch"],"size":700416,"object_info":{"oid":{"oid":"1000793c33a.000000dd","key":"","snapid":-2,"hash":2213519396,"max":0,"pool":4,"namespace":""},"version":"43143'3253925","prior_version":"0'0","last_reqid":"client.1822828.0:160762343","user_version":3253925,"size":4194304,"mtime":"2025-11-10T22:13:15.257031-0500","local_mtime":"2025-11-10T22:14:44.294560-0500","lost":0,"flags":["dirty","data_digest"],"truncate_seq":0,"truncate_size":0,"data_digest":"0x7e22aafe","omap_digest":"0xffffffff","expected_object_size":0,"expected_write_size":0,"alloc_hint_flags":0,"manifest":{"type":0},"watchers":{}}},{"osd":65,"primary":false,"shard":3,"errors":[],"size":700416} So it looks like there is some size mismatch. I don't have any indication that data is lost so far. But the cluster is under pressure and daemons crash, so it's in a critical state. OSDs crash with errors like [ERR] : 4.cc shard 63(5) soid 4:33394f7f:::10001e60e03.00000000:head : candidate size 311296 info size 307200 mismatch [ERR] : 4.cc shard 63(5) soid 4:33394f7f:::10001e60e03.00000000:head : candidate size 311296 info size 307200 mismatch ... again, there is some size mismatch. I'm pretty sure this is the same error than what was mentioned in this thread: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/F3JYHK4INKS… But there has been no reply yet. Does anyone have an idea what can be done to solve this? Or what I can try to get more information? We don't have copies of most of the data and it would be bad if we loose some. Best regards and many thanks for your help, David

1 0

Re: Updating cephadm-managed Tentacle cluster from RC to production
by Marek Szuba 03 Dec '25

03 Dec '25

On 2025-12-03 15:57, Marek Szuba via ceph-users wrote: > Well, things have got interesting now. The above appears to have worked > for 4 out of 5 mgr daemons - but now most "ceph orch" commands give me > the error "Error ENOENT: Module not found". I can also see many > cephadm-related "module not found" error in system logs. Progress! Having carefully searched through all the logs, I discovered the module was unable to load due to a spurious field in certificate JSON. A quick look at the relevant bit of Python code then allowed me to identify the field "editable", present in all six cert/key config keys, as the culprit. Having carefully edited and reset the offending JSON, I was then able to successfully run 'ceph orch upgrade start --ceph-version 20.2.0' (which redid the mgr and mon upgrade - that was intentional, in case the manual redeployment with new image missed something). This has just successfully completed. @Michael: I did see that bit of the documentation, I just had no clue which config keys exactly to examine and what to look for in them. Fortunately it turned out that some of the operations resulting in "Module not found" also crashed the active mgr (interestingly enough, it wasn't all such operations - Ceph only noted two mgr crashes), and I was able to find the traceback in crash metadata. -- MS

1 0

Re: Updating cephadm-managed Tentacle cluster from RC to production
by Michael Worsham 03 Dec '25

03 Dec '25

Check this from the Ceph documentation on it... https://docs.ceph.com/en/latest/cephadm/upgrade/ Scroll down to the "Module Not Found" section. -- M ________________________________ From: Marek Szuba via ceph-users <ceph-users(a)ceph.io> Sent: Wednesday, December 3, 2025 10:51:57 AM To: ceph-users(a)ceph.io <ceph-users(a)ceph.io> Subject: [ceph-users] Re: Updating cephadm-managed Tentacle cluster from RC to production This is an external email. Please take care when clicking links or opening attachments. When in doubt, check with the Help Desk or Security. On 2025-12-03 14:06, Michael Worsham via ceph-users wrote: > ceph orch daemon redeploy <daemon_name> --image quay.io/ceph/ceph:v20.2.0 Well, things have got interesting now. The above appears to have worked for 4 out of 5 mgr daemons - but now most "ceph orch" commands give me the error "Error ENOENT: Module not found". I can also see many cephadm-related "module not found" error in system logs. Update a few minutes later: have updated the last mgr + all the mons to 20.2.0 by manually updating the container image in their respective unit.run files. Now all ten daemons are on stable Tentacle and the Ceph cluster seems happy - but cephadm still doesn't work. -- MS _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments.

1 0

Re: [dashboard] bug when adding a label to a host?
by Afreen Misbah 03 Dec '25

03 Dec '25

Hi, Thanks for reporting , you can track it here https://tracker.ceph.com/issues/74078. On Wed, Dec 3, 2025 at 9:55 PM Robert Sander via ceph-users < ceph-users(a)ceph.io> wrote: > Hi, > > Am 03.12.25 um 5:22 PM schrieb Bailey Allison via ceph-users: > > > > Double check the labels in the terminal. I thought I had the same issue > > when adding a label but it was more of a visual thing in the UI, all of > > the labels were still there. > > I had the cephadm log running while performing the editing in the > dashboard. And this told me that it just removed two existing labels. > > I did not add an existing label to the host but created a completely new > one. This empties the list of assigned labels somehow. The UI for that > dialogue really does not look very good. The UX was better in the older > dashboard. > > Regards > -- > Robert Sander > Linux Consultant > > Heinlein Consulting GmbH > Schwedter Str. 8/9b, 10119 Berlin > > https://www.heinlein-support.de > > Tel: +49 30 405051 - 0 > Fax: +49 30 405051 - 19 > > Amtsgericht Berlin-Charlottenburg - HRB 220009 B > Geschäftsführer: Peer Heinlein - Sitz: Berlin > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- Afreen

1 0

Re: [dashboard] bug when adding a label to a host?
by Robert Sander 03 Dec '25

03 Dec '25

Hi, Am 03.12.25 um 5:22 PM schrieb Bailey Allison via ceph-users: > > Double check the labels in the terminal. I thought I had the same issue > when adding a label but it was more of a visual thing in the UI, all of > the labels were still there. I had the cephadm log running while performing the editing in the dashboard. And this told me that it just removed two existing labels. I did not add an existing label to the host but created a completely new one. This empties the list of assigned labels somehow. The UI for that dialogue really does not look very good. The UX was better in the older dashboard. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

Re: [dashboard] bug when adding a label to a host?
by Bailey Allison 03 Dec '25

03 Dec '25

Hi Robert, Double check the labels in the terminal. I thought I had the same issue when adding a label but it was more of a visual thing in the UI, all of the labels were still there. Regards, Bailey Allison Service Team Lead 45Drives, Ltd. 866-594-7199 x868 On 2025-12-03 12:13, Robert Sander via ceph-users wrote: > Hi, > > I have a test cluster running on 20.2.0 with the cephadm orchestrator. > > The host edit form in the dashboard has been changed. > > When I try to add a new label the existing labels get removed. > > Is this a known issue? > > Regards

1 0

[dashboard] bug when adding a label to a host?
by Robert Sander 03 Dec '25

03 Dec '25

Hi, I have a test cluster running on 20.2.0 with the cephadm orchestrator. The host edit form in the dashboard has been changed. When I try to add a new label the existing labels get removed. Is this a known issue? Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

Re: Updating cephadm-managed Tentacle cluster from RC to production
by Marek Szuba 03 Dec '25

03 Dec '25

On 2025-12-03 14:06, Michael Worsham via ceph-users wrote: > ceph orch daemon redeploy <daemon_name> --image quay.io/ceph/ceph:v20.2.0 Well, things have got interesting now. The above appears to have worked for 4 out of 5 mgr daemons - but now most "ceph orch" commands give me the error "Error ENOENT: Module not found". I can also see many cephadm-related "module not found" error in system logs. Update a few minutes later: have updated the last mgr + all the mons to 20.2.0 by manually updating the container image in their respective unit.run files. Now all ten daemons are on stable Tentacle and the Ceph cluster seems happy - but cephadm still doesn't work. -- MS

1 0

Config mds_session_blocklist_on_evict ignored
by Eugen Block 03 Dec '25

03 Dec '25

Hi, I'm hunting down a CephFS issue where mds caps become stale for no apparent reason. I might write another email about that when I don't find a solution. But while debugging that, I also tried to prevent clients from being blocklisted when evicted, so I set this: ceph04:~ # ceph config set mds mds_session_blocklist_on_evict false According to the config help, it can be changed at runtime, and it is applied: mds01:~ # cephadm shell -- ceph daemon mds.cephfs.mds01 config get mds_session_blocklist_on_evict ... { "mds_session_blocklist_on_evict": "false" } But after triggering the misbehaving client, it is still blocklisted: 2025-12-03T13:34:52.498+0000 7f5f6cd2b640 1 mds.0.45600 Evicting (and blocklisting) client session 1822981126 (v1:IP:0/1026509947) 2025-12-03T13:34:52.498+0000 7f5f6cd2b640 0 log_channel(cluster) log [INF] : Evicting (and blocklisting) client session 1822981126 (v1:IP:0/1026509947) ceph04:~ # ceph osd blocklist ls IP:0/1026509947 2025-12-03T14:34:52.503570+0000 listed 1 entries Is this a know bug or something? This is a cephadm managed cluster version 19.2.3. Thanks for any pointers! Eugen

1 0

Re: Updating cephadm-managed Tentacle cluster from RC to production
by Michael Worsham 03 Dec '25

03 Dec '25

I haven't tried to downgrade, but perhaps the following steps might help. ceph orch daemon redeploy <daemon_name> --image quay.io/ceph/ceph:v20.2.0 -or- ceph config set global container_image quay.io/ceph/ceph:v20.2.0 Then do each daemon individually ceph orch apply mgr --image quay.io/ceph/ceph:v20.2.0 Get Outlook for Android<https://aka.ms/AAb9ysg> ________________________________ From: Marek Szuba via ceph-users <ceph-users(a)ceph.io> Sent: Wednesday, December 3, 2025 8:50:43 AM To: ceph-users(a)ceph.io <ceph-users(a)ceph.io> Subject: [ceph-users] Re: Updating cephadm-managed Tentacle cluster from RC to production This is an external email. Please take care when clicking links or opening attachments. When in doubt, check with the Help Desk or Security. On 2025-12-03 01:53, Michael Worsham via ceph-users wrote: > What does it show if you do 'ceph versions'? "ceph version 20.3.0-2957-g62bcf65e (62bcf65e8c0995783bb3e368909716346874ad62) tentacle (dev - RelWithDebInfo)" for all daemons. > Also, there is a way that you can upgrade individual daemons via "staggered" upgrade... > > "ceph orch upgrade --daemon-types <type1,Type2, etc.>" I have in fact already tried that, initially limiting the upgrade to mgr and mon daemons. Same as without the restriction - with "--ceph-version 20.2.0" the upgrade fails immediately with "ceph cannot downgrade to a dev release", with "--image quay.io/ceph/ceph:v20.2.0" the "cannot downgrade" appears in the upgrade status. -- MS _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments.

1 0

Re: Updating cephadm-managed Tentacle cluster from RC to production
by Marek Szuba 03 Dec '25

03 Dec '25

On 2025-12-03 01:53, Michael Worsham via ceph-users wrote: > What does it show if you do 'ceph versions'? "ceph version 20.3.0-2957-g62bcf65e (62bcf65e8c0995783bb3e368909716346874ad62) tentacle (dev - RelWithDebInfo)" for all daemons. > Also, there is a way that you can upgrade individual daemons via "staggered" upgrade... > > "ceph orch upgrade --daemon-types <type1,Type2, etc.>" I have in fact already tried that, initially limiting the upgrade to mgr and mon daemons. Same as without the restriction - with "--ceph-version 20.2.0" the upgrade fails immediately with "ceph cannot downgrade to a dev release", with "--image quay.io/ceph/ceph:v20.2.0" the "cannot downgrade" appears in the upgrade status. -- MS

1 0

Migrating krbd -> librbd
by Boris 03 Dec '25

03 Dec '25

Hey, we are thinking to migrate from krbd to librbd. Anybody using librbd in larger setups? Any things we should consider on the storage nodes if we want to have 10k VMs via librbd?

1 0

500 - Internal Server Error ceph RGW Dashboard
by GLE, Vivien 03 Dec '25

03 Dec '25

Hi, I'm trying S3 on my cluster with a RGW in VM. My cluster is currently in v20.2.0 and RGW is in v19.2.1. S3 is working fine the only problem is that if I try to search (via RGW VM) for anything user relative (bucket or user management) on the object ceph dashboard this search leads to 500 - internal server error with those logs in the mgr 2025-12-03T09:47:06.545+0000 700619b22640 0 [dashboard INFO request] [::ffff:172.16.112.15:60786] [GET] [500] [0.155s] [admin] [513.0B] /api/rgw/user/dashboard 2025-12-03T09:47:06.585+0000 70061a323640 0 [dashboard ERROR exception] Internal Server Error Traceback (most recent call last): File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 47, in dashboard_exception_handler return handler(*args, **kwargs) File "/lib/python3.9/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__ return self.callable(*self.args, **self.kwargs) File "/usr/share/ceph/mgr/dashboard/controllers/_base_controller.py", line 264, in inner ret = func(*args, **kwargs) File "/usr/share/ceph/mgr/dashboard/controllers/_rest_controller.py", line 193, in wrapper return func(*vpath, **params) File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 929, in get return self._get(uid, daemon_name=daemon_name, stats=stats) File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 941, in _get result['uid'] = result['full_user_id'] KeyError: 'full_user_id' I found a workaround with deploying a container RGW via cephadm v20.2.0 and using it to manage my bucket and user. Does radosgw needs to be at the same version as the ceph cluster ? Or is it a bug ? Thanks Vivien

1 0

Re: Updating cephadm-managed Tentacle cluster from RC to production
by Michael Worsham 03 Dec '25

03 Dec '25

What does it show if you do 'ceph versions'? Also, there is a way that you can upgrade individual daemons via "staggered" upgrade... "ceph orch upgrade --daemon-types <type1,Type2, etc.>" Example types: mgr, mon, crash, osd, mds, rgw, rbd-mirror, and cephfs-mirror NOTE: Enforced upgrade order is: mgr -> mon -> crash -> osd -> mds -> rgw -> rbd-mirror -> cephfs-mirror -- Michael ________________________________ From: Marek Szuba via ceph-users <ceph-users(a)ceph.io> Sent: Tuesday, December 2, 2025 6:46 PM To: ceph-users(a)ceph.io <ceph-users(a)ceph.io> Subject: [ceph-users] Updating cephadm-managed Tentacle cluster from RC to production This is an external email. Please take care when clicking links or opening attachments. When in doubt, check with the Help Desk or Security. Hello, We have got a cephadm-managed test Ceph cluster created with a Tentacle Release Candidate, RC1 if memory serves me right. With 20.2.0 having officially been releases mid-November, we would ideally like to bring this cluster into production without data loss - none of the data already present on the OSDs is irreplaceable but we would prefer to retain snapshot history we have established during the test phase. Unfortunately, it turns out that the naive approach of having "ceph orch upgrade" upgrade the cluster to a production release does not work - if we run it with "--ceph-version 20.2.0" it immediately fails with the error "cannot downgrade to a dev version", "--image quay.io/ceph/ceph:v20.2.0" on the other hand appears to start successfully but gets immediately paused, with the same error message appearing in the upgrade status report. The "downgrade" bit is likely related to the fact that according to "ceph versions", the Ceph version we have presently got running is 20.3.0-2957-g62bcf65e (despite the fact the cephadm package we used clearly having been labelled 20.1.something). The "to a dev version" bit, however, has got me stumped given the long-established Ceph has been using the "even minor versions are production" convention for ages. Any thoughts on how we can get this cluster upgraded to a production release without wiping everything and starting from scratch? Thanks in advance! PS. cephadm on the admin node has already been upgraded to 20.2.0. -- MS _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments.

1 0

Updating cephadm-managed Tentacle cluster from RC to production
by Marek Szuba 02 Dec '25

02 Dec '25

Hello, We have got a cephadm-managed test Ceph cluster created with a Tentacle Release Candidate, RC1 if memory serves me right. With 20.2.0 having officially been releases mid-November, we would ideally like to bring this cluster into production without data loss - none of the data already present on the OSDs is irreplaceable but we would prefer to retain snapshot history we have established during the test phase. Unfortunately, it turns out that the naive approach of having "ceph orch upgrade" upgrade the cluster to a production release does not work - if we run it with "--ceph-version 20.2.0" it immediately fails with the error "cannot downgrade to a dev version", "--image quay.io/ceph/ceph:v20.2.0" on the other hand appears to start successfully but gets immediately paused, with the same error message appearing in the upgrade status report. The "downgrade" bit is likely related to the fact that according to "ceph versions", the Ceph version we have presently got running is 20.3.0-2957-g62bcf65e (despite the fact the cephadm package we used clearly having been labelled 20.1.something). The "to a dev version" bit, however, has got me stumped given the long-established Ceph has been using the "even minor versions are production" convention for ages. Any thoughts on how we can get this cluster upgraded to a production release without wiping everything and starting from scratch? Thanks in advance! PS. cephadm on the admin node has already been upgraded to 20.2.0. -- MS

1 0

Re: reef 18.2.8 QE validation status
by Afreen Misbah 01 Dec '25

01 Dec '25

Dashboard approved On Sat, Nov 29, 2025 at 10:47 PM Yuri Weinstein via ceph-users < ceph-users(a)ceph.io> wrote: > Rerun scheduled for krbd > > Still waiting for replies from dev leads > > On Mon, Nov 24, 2025 at 6:24 AM Ilya Dryomov <idryomov(a)gmail.com> wrote: > > > On Fri, Nov 21, 2025 at 12:04 AM Yuri Weinstein <yweinste(a)redhat.com> > > wrote: > > > > > > Note: this is an attempt to test and publish post EOL reef point > release > > > > > > Details of this release are summarized here: > > > > > > https://tracker.ceph.com/issues/73906#note-1 > > > Release Notes - TBD > > > LRC upgrade - N/A > > > > > > distro => jammy focal centos9 windows bookworm > > > > > > Seeking approvals/reviews for: > > > > > > smoke - Laura approved? > > > > > > rados - Radek, Laura approved? Travis? Nizamudeen? Adam King approved? > > > (rados is also running on build 2 == build 1 + > > > https://github.com/ceph/ceph/pull/66056 per Igor's request) > > > > > > rgw - Adam E approved? > > > > > > fs - Venky approved? > > > > > > orch - Adam King approved? > > > > > > rbd - Ilya approved? > > > krbd - Ilya approved? > > > > rbd approved, krbd needs a rerun as all jobs failed while > > going through cephlab.yml playbook (not related to RBD): > > > > ['Error mounting /var/lib/ceph: mount: /var/lib/ceph: wrong fs type, > > bad option, bad superblock on /dev/mapper/vg_nvme-lv_5, missing > > codepage or helper program, or other error.'] > > > > Thanks, > > > > Ilya > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- Afreen

1 0

Test e-mail after updating TXT records
by Adam Kraitman 01 Dec '25

01 Dec '25

-- Adam Kraitman Systems Administrator Ceph Engineering IRC: akraitma

1 0

Removing a specific object.
by quaglio＠bol.com.br 01 Dec '25

01 Dec '25

Hi, I needed to remove a disk from a specific pool and now I'm facing a problem that I can't solve. There is a pool that has 1 "unfound" object. Furthermore, recovery is not happening for that PG because of this object. pg 3.792 is active+forced_recovery+recovery_unfound+undersized+degraded+remapped, acting [69], 1 unfound There is no problem removing this object. How do I delete it? Thanks Rafael.

1 1