- ceph-users - lists.ceph.io

smartctl issues on squid (ubuntu 24.04)
by Nathan Butcher 08 Dec '25

08 Dec '25

Hi, We've been running squid 19.2.3 on Ubuntu 24.04.3 LTS We encountered some issues where the Ceph dashboard was unable to get SMART data from OSDs under the Device Health tab. At first we were getting -22 errors from that tab suggesting that smartctl was under version 7. That wasn't the case at all. System logging determined that there was an apparmor profile called unix-chkpwd which was producing errors as ceph tried to authenticate into the pod to run smartctl. We had to disable that profile in order to get that to work. We had to set the profile to complain mode using aa-complain (and even then Ubuntu had some bugs getting that command to work) Our OSD node has a HPE smart array card on it which ensures the kernel sees all drives on it as scsi devices, whether in HBA or RAID mode. From within the OSD pod, it seems that smartctl isn't able to autodetect the type of drives it is trying to access, possibly because attributed from udev and getting blocked somehow. We needed the "-d scsi" command to work, but the ceph code doesn't allow any customisation or flexibility in how smartctl is called. In the end I had to write a wrapper script to inject the -d scsi command around smartctl in order to get it to work within the confines of the OSD pod. Then the wrapper had to be distributed across every OSD pod. It's a real messy hack. So a couple of things: How can we control how the OSD pods supposed to present allowing smartctl to work but without compromising on security? Is this just an ubuntu apparmor thing that ubuntu should deal with? I can see that there is a separate apparmor profile applied to pods. I'm sure we're not the only ones who have hit this roadblock. In any case, I hope someone else having the same issues find these workarounds useful Nathan Butcher Senior Systems Engineer 1300 852 110 Chat on Teams nbutcher(a)ct.com.au We are passionate about service excellence. If you would like to contact my Team Leader for any reason, their name is Tim Sandy, email address is tsandy(a)ct.com.au and phone number is (03) 9998 6377. We’re 100% Carbon Neutral and ISO 14001 certified. Check out our Environmental Sustainability pledge here. Adelaide | Brisbane | Canberra | Darwin | Hobart | Melbourne | Perth | Sydney Disclaimer This email message is confidential and may be privileged. Unauthorised use, copying or distribution of any part of this email including attachments is prohibited. If you are not the intended recipient, please forward the email to infosec(a)ct.com.au and delete the original. Any advice contained in this email is of a general nature only and has been prepared without considering your relevant personal circumstances. To the extent permitted by law, we exclude (and where the law does not permit exclusion, limit to the extent permitted by law) all liability for any direct, indirect and consequential losses, damages and expenses incurred in any way (including but not limited to that arising from negligence), connected with any use or access to or any reliance on information contained in this email or any attachments. Centorrino Technologies Pty Ltd (ABN: 83 606 931 524). All Rights Reserved.

1 0

Re: reef 18.2.8 QE validation status
by Yuri Weinstein 08 Dec '25

08 Dec '25

This test branch was built loner then 2.5 weeks ago I have to rebuild it first and then will schedule krbd On Mon, Dec 8, 2025 at 1:39 PM Ilya Dryomov via ceph-users < ceph-users(a)ceph.io> wrote: > On Mon, Dec 8, 2025 at 8:50 PM Ilya Dryomov <idryomov(a)gmail.com> wrote: > > > > On Mon, Dec 8, 2025 at 5:00 PM Venky Shankar via ceph-users > > <ceph-users(a)ceph.io> wrote: > > > > > > Hi Yuri, > > > > > > On Fri, Nov 21, 2025 at 4:34 AM Yuri Weinstein <yweinste(a)redhat.com> > wrote: > > > > > > > > Note: this is an attempt to test and publish post EOL reef point > release > > > > > > > > Details of this release are summarized here: > > > > > > > > https://tracker.ceph.com/issues/73906#note-1 > > > > Release Notes - TBD > > > > LRC upgrade - N/A > > > > > > > > distro => jammy focal centos9 windows bookworm > > > > > > > > Seeking approvals/reviews for: > > > > > > > > smoke - Laura approved? > > > > > > > > rados - Radek, Laura approved? Travis? Nizamudeen? Adam King > approved? > > > > (rados is also running on build 2 == build 1 + > > > > https://github.com/ceph/ceph/pull/66056 per Igor's request) > > > > > > > > rgw - Adam E approved? > > > > > > > > fs - Venky approved? > > > > > > There are lot many failures in the fs suite resembling > > > > > > Error mounting /var/lib/ceph: mount: /var/lib/ceph: wrong fs > > > type, bad option ... > > > > > > which is an issue with the testing kernel. Adding Alex who was looking > > > into this issue. > > > > > > As far as the run validation is concerned, it's risky to ignore the > > > large number of failures. We have two options: > > > > > > - fix the testing kernel issue and rerun > > > - rerun with `for-linus` branch (with `-k for-linus` in the > > > teuthology-suite schedule command) > > > > > > The latter would be using an older kernel branch, but I think it > > > should suffice for this case. > > > > Hi Venky, > > > > With your report of the fs suite being affected as well, I carved out > > some time to take a look. > > > > I think it's happening because XFS maintainers disabled support for V4 > > filesystems that was deprecated 5 years ago -- as of 6.18 kernel it's > > no longer compiled in by default. The proper fix would be to create an > > image with/move to V5 on-disk format; I have opened [1] to try to > > achieve that. We can also re-enable V4 on-disk format in our kernel > > builds as a workaround. > > > > [1] https://github.com/ceph/ceph-cm-ansible/pull/799 > > This should be fixed now. Yuri, please schedule a rerun for krbd. > > Thanks, > > Ilya > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: reef 18.2.8 QE validation status
by Ilya Dryomov 08 Dec '25

08 Dec '25

On Mon, Dec 8, 2025 at 5:00 PM Venky Shankar via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi Yuri, > > On Fri, Nov 21, 2025 at 4:34 AM Yuri Weinstein <yweinste(a)redhat.com> wrote: > > > > Note: this is an attempt to test and publish post EOL reef point release > > > > Details of this release are summarized here: > > > > https://tracker.ceph.com/issues/73906#note-1 > > Release Notes - TBD > > LRC upgrade - N/A > > > > distro => jammy focal centos9 windows bookworm > > > > Seeking approvals/reviews for: > > > > smoke - Laura approved? > > > > rados - Radek, Laura approved? Travis? Nizamudeen? Adam King approved? > > (rados is also running on build 2 == build 1 + > > https://github.com/ceph/ceph/pull/66056 per Igor's request) > > > > rgw - Adam E approved? > > > > fs - Venky approved? > > There are lot many failures in the fs suite resembling > > Error mounting /var/lib/ceph: mount: /var/lib/ceph: wrong fs > type, bad option ... > > which is an issue with the testing kernel. Adding Alex who was looking > into this issue. > > As far as the run validation is concerned, it's risky to ignore the > large number of failures. We have two options: > > - fix the testing kernel issue and rerun > - rerun with `for-linus` branch (with `-k for-linus` in the > teuthology-suite schedule command) > > The latter would be using an older kernel branch, but I think it > should suffice for this case. Hi Venky, With your report of the fs suite being affected as well, I carved out some time to take a look. I think it's happening because XFS maintainers disabled support for V4 filesystems that was deprecated 5 years ago -- as of 6.18 kernel it's no longer compiled in by default. The proper fix would be to create an image with/move to V5 on-disk format; I have opened [1] to try to achieve that. We can also re-enable V4 on-disk format in our kernel builds as a workaround. [1] https://github.com/ceph/ceph-cm-ansible/pull/799 Thanks, Ilya

1 1

Re: SMB on ARM
by John Mulligan 08 Dec '25

08 Dec '25

On Monday, December 8, 2025 10:02:56 AM Eastern Standard Time Daniel Brown via ceph-users wrote: > Upgraded to tentacle over the weekend - mostly a smooth process. Had one > node that was a bit pokey and had to manually download the ceph container > image, but other than that, all good. > > I did try to setup SMB — unfortunately, my cluster is running on Arm64 > machines, and the SMB setup appears to be trying to start an Amd64 > container. Didn’t see a lot of logging, but after some digging around under > /var/lib/ceph/[my cluster id here]/… > > > $ cat unit.image > > quay.io/samba.org/samba-server:ceph20-centos-amd64 > > > Don’t suppose there’s a way to update that for the correct architecture? And > or to document it if it isn’t going to be possible? Thanks for trying the SMB support out. Unfortunately, the samba container images [1] do not support ARM64 at this time. We worked on it a while ago but got pulled off of that onto other tasks and never really resumed working on arm support. I have filed an issue to revisit arm64 support for samba container images on that project. [1] https://github.com/samba-in-kubernetes/samba-container

1 0

Re: SMB on ARM
by Robert Sander 08 Dec '25

08 Dec '25

Hi, Am 08.12.25 um 4:02 PM schrieb Daniel Brown via ceph-users: > > I did try to setup SMB The SMB service and the SMB MGR module are far from being production ready. I only get errors when trying to use them on AMD64. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

Re: How to fix BLUESTORE_FREE_FRAGMENTATION:
by Anthony D'Atri 08 Dec '25

08 Dec '25

> On Dec 8, 2025, at 4:07 AM, Olivier Delcourt via ceph-users <ceph-users(a)ceph.io> wrote: > > > >> Le 8 déc. 2025 à 09:36, Robert Sander via ceph-users <ceph-users(a)ceph.io> a écrit : >> >> Hi, >> >> Am 08.12.25 um 9:32 AM schrieb Nico Schottelius via ceph-users: >> >>> since migrating to Squid (19.2.3), I get the following health warning: >>> [WRN] BLUESTORE_FREE_FRAGMENTATION: 1 OSD(s) >>> osd.25 0.806301 How large is this OSD? >>> However while this is a warning, there seems to be no documented way of >>> fixing it. >>> What do I do with this warning / how to defrag the bluestore filesystem? >> >> Even the IBM documentation recommends to contact the IBM support in this case: https://www.ibm.com/docs/en/storage-ceph/8.1.0?topic=bluestore-fragmentatio… >> >> My best bet is to just recreate the OSD. > > Before that, what about trying to empty the OSD ( ceph osd reweight XXX 0) and when empty, put it back to business ( ceph osd reweight XXX 1) ? I suspect that would do the trick, but would suggest instead "crush reweigh" so that the balancer doesn't get confused.

1 0

SMB on ARM
by Daniel Brown 08 Dec '25

08 Dec '25

Upgraded to tentacle over the weekend - mostly a smooth process. Had one node that was a bit pokey and had to manually download the ceph container image, but other than that, all good. I did try to setup SMB — unfortunately, my cluster is running on Arm64 machines, and the SMB setup appears to be trying to start an Amd64 container. Didn’t see a lot of logging, but after some digging around under /var/lib/ceph/[my cluster id here]/… $ cat unit.image quay.io/samba.org/samba-server:ceph20-centos-amd64 Don’t suppose there’s a way to update that for the correct architecture? And or to document it if it isn’t going to be possible?

1 0

dashboard dial tcp connection refused
by lejeczek 08 Dec '25

08 Dec '25

Hi guys. I've no _ssl_ used by _dashboard_ -> $ ceph mgr services { "dashboard": "http://10.1.1.63:8080/", "prometheus": "http://10.1.1.63:9283/" } -> $ ceph config get mgr mgr/dashboard/ssl false and the 'standby' servers logs: ... Dec 08 11:33:39 ceph-9f4f9dba-72c7-11f0-8052-525400519d29-alertmanager-podster1[2503]: ts=2025-12-08T11:33:39.781Z caller=dispatch.go:353 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"<redacted>\": dial tcp 10.1.1.63:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"<redacted>\": dial tcp 10.1.1.61:8443: connect: connection refused" Dec 08 11:33:39 ceph-9f4f9dba-72c7-11f0-8052-525400519d29-alertmanager-podster1[2503]: ts=2025-12-08T11:33:39.781Z caller=notify.go:848 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] aggrGroup="{}/{}:{alertname=\"CephHealthWarning\"}" msg="Notify attempt failed, will retry later" attempts=1 err="Post \"<redacted>\": dial tcp 10.1.1.63:8443: connect: connection refused" ... with the snippet of the logs above _ceph_ says only: -> $ ceph health detail HEALTH_WARN mon podster1 is low on available space [WRN] MON_DISK_LOW: mon podster1 is low on available space mon.podster1 has 12% avail nothing listens on 8443 anywhere, which is as it should be - why standby server wants that - would anybody know? 10.1.1.61 is standby's own IP. many thanks, L.

1 0

Re: radosgw-19.2.3 STS AssumeRoleWithWebIdentity and OIDC namespace
by Boris 08 Dec '25

08 Dec '25

Hi Pritha, we are using keycloak and keystone to federate users. Users are able to create keystone projects and use the EC2 credentials to authenticate with radosgw. We also have a panel where a user can access their buckets. Typical cloud provider stuff. The panel is a client side JS application that used to fetch all the EC2 credentials from keystone and use the first set to do authenticate with radosgw. But because of security concerns we don't want keystone to be public reachable. So we implemented STS with AssumeRoleWithWebIdentity. The user authenticates via keycloak and receives some temporary credentials. We've set a custom user attribute that contains the and map this attribute to the token claim with sub. This attribute is just a string field which will get populated by a middleware, when the user selects the project. We've tried to create the bucket with the STS credentials and access them with the EC2 credentials and vice versa. This was working with 19.2.2 when the keystone und sts uid matched. The buckets that are created via the STS credentials belong to the UID "$oidc$KEYSTONE_PROJECT_ID" while the buckets created via EC2 credentials belong to the UID "KEYSTONE_PROJECT_ID". We also tested this with a local user. We will gather the logs later today. But those basically say "Authentication worked, you don't have access to the bucket of the other uid". I've created a tracker for this problem, where I will add the logs later: https://tracker.ceph.com/issues/74131 Here is how we proceeded with our initial setup that worked with 19.2.2: https://tracker.ceph.com/issues/74131?issue_count=3&issue_position=1&next_i… Am Mo., 8. Dez. 2025 um 07:08 Uhr schrieb Pritha Srivastava via ceph-users < ceph-users(a)ceph.io>: > Hi Boris, > > Is it possible for you to elaborate on what problem you are facing? like > which credentials you used to create the bucket? and check rgw logs and see > what the error is, when you try to access the bucket? > > Thanks, > Pritha > > On Fri, Dec 5, 2025 at 5:19 PM Boris <bb(a)kervyn.de> wrote: > > > Hi, > > since the last squid update (19.2.3) we have the problem, that the STS > > integration does not work anymore. (relevant tracker: > > https://tracker.ceph.com/issues/69924) > > > > Now there is a user with the $oidc$ prefix with the same suffix as the > > original user name. For example: > > > > >"$oidc$e0a0eed4f6a64c9cad70b69625dccba8", > > >"e0a0eed4f6a64c9cad70b69625dccba8", > > > > and both user are treated as different users in radosgw. > > > > We user STS with AssumeRoleWithWebIdentity, to make bucket management > > possible via a website, and didn't want to fetch all the access > credentials > > just to pick one and authenticate with that. > > > > Now when we create a bucket in the panel, we don't have access via the > > normal EC2 credentials anymore. > > > > I still think I am holding it wrong. > > This is the role and policy we've set: > > > > > # radosgw-admin role get --role-name=S3Access > > > { > > > "RoleId": "49d0d470-dc7a-4ffe-8db3-4f40cb82ebfd", > > > "RoleName": "S3Access", > > > "Path": "/", > > > "Arn": "arn:aws:iam:::role/S3Access", > > > "CreateDate": "2025-08-12T08:31:11.761Z", > > > "MaxSessionDuration": 3600, > > > "AssumeRolePolicyDocument": > > > "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":{\"Federated\":[\"arn:aws:iam:::oidc-provider/vv.xx.yy.zz:8443/realms/snc-customera\"]},\"Action\":[\"sts:AssumeRoleWithWebIdentity\"],\"Condition\":{\"StringEquals\":{\"vv.xx.yy.zz:8443/realms/snc-customera:app_id\":\"account\"}}}]}", > > > "PermissionPolicies": [ > > > { > > > "PolicyName": "Policy1", > > > "PolicyValue": > > > "{\"Version\":\"2012-10-17\",\"Statement\":{\"Effect\":\"Allow\",\"Action\":\"s3:*\",\"Resource\":\"arn:aws:s3:::*\"}}" > > > } > > > ] > > > } > > > > Best wishes > > Boris > > -- > > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im > > groÃƒ¼en Saal. > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

1 0

rgw index auto resharding not acting
by Johan Thomsen 08 Dec '25

08 Dec '25

Hi For some reason, the auto-resharding is not doing anything, even though it reports fill status over 100%. See below: $ radosgw-admin bucket limit check --rgw-zonegroup=k1-objectstore --rgw-zone=k1-objectstore 2025-12-08T08:45:57.143+0000 7f8921528900 0 ERROR: current period 43e92ddb-d0d2-4fbc-b18b-dc87126a7664 does not contain zone id 1da0f602-9397-4997-8e76-3e3425468746 2025-12-08T08:45:57.166+0000 7f8921528900 0 period (43e92ddb-d0d2-4fbc-b18b-dc87126a7664 does not have zone 1da0f602-9397-4997-8e76-3e3425468746 configured [ { "user_id": "k1-ceph-user", "buckets": [ { "bucket": "k1-snapshots", "tenant": "", "num_objects": 5766564, "num_shards": 97, "objects_per_shard": 59449, "fill_status": "OVER 114%" } ] } Here's a dump of related config: rgw_dynamic_resharding true rgw_max_dynamic_shards 1999 rgw_max_objs_per_shard 50000 rgw_md_log_max_shards 64 rgw_objexp_hints_num_shards 127 rgw_override_bucket_index_max_shards 1031 rgw_reshard_batch_size 64 rgw_reshard_bucket_lock_duration 360 rgw_reshard_max_aio 128 rgw_reshard_num_logs 16 rgw_reshard_thread_interval 600 rgw_safe_max_objects_per_shard 52000 rgw_shard_warning_threshold 90.000000 rgw_usage_max_shards 32 Does anyone have an idea why it is not auto-resharding to a value below at least 52k objs per shard?

1 0