- ceph-users - lists.ceph.io

Re: radosgw-19.2.3 STS AssumeRoleWithWebIdentity and OIDC namespace
by Boris 08 Dec '25

08 Dec '25

Hi Pritha, we are using keycloak and keystone to federate users. Users are able to create keystone projects and use the EC2 credentials to authenticate with radosgw. We also have a panel where a user can access their buckets. Typical cloud provider stuff. The panel is a client side JS application that used to fetch all the EC2 credentials from keystone and use the first set to do authenticate with radosgw. But because of security concerns we don't want keystone to be public reachable. So we implemented STS with AssumeRoleWithWebIdentity. The user authenticates via keycloak and receives some temporary credentials. We've set a custom user attribute that contains the and map this attribute to the token claim with sub. This attribute is just a string field which will get populated by a middleware, when the user selects the project. We've tried to create the bucket with the STS credentials and access them with the EC2 credentials and vice versa. This was working with 19.2.2 when the keystone und sts uid matched. The buckets that are created via the STS credentials belong to the UID "$oidc$KEYSTONE_PROJECT_ID" while the buckets created via EC2 credentials belong to the UID "KEYSTONE_PROJECT_ID". We also tested this with a local user. We will gather the logs later today. But those basically say "Authentication worked, you don't have access to the bucket of the other uid". I've created a tracker for this problem, where I will add the logs later: https://tracker.ceph.com/issues/74131 Here is how we proceeded with our initial setup that worked with 19.2.2: https://tracker.ceph.com/issues/74131?issue_count=3&issue_position=1&next_i… Am Mo., 8. Dez. 2025 um 07:08 Uhr schrieb Pritha Srivastava via ceph-users < ceph-users(a)ceph.io>: > Hi Boris, > > Is it possible for you to elaborate on what problem you are facing? like > which credentials you used to create the bucket? and check rgw logs and see > what the error is, when you try to access the bucket? > > Thanks, > Pritha > > On Fri, Dec 5, 2025 at 5:19 PM Boris <bb(a)kervyn.de> wrote: > > > Hi, > > since the last squid update (19.2.3) we have the problem, that the STS > > integration does not work anymore. (relevant tracker: > > https://tracker.ceph.com/issues/69924) > > > > Now there is a user with the $oidc$ prefix with the same suffix as the > > original user name. For example: > > > > >"$oidc$e0a0eed4f6a64c9cad70b69625dccba8", > > >"e0a0eed4f6a64c9cad70b69625dccba8", > > > > and both user are treated as different users in radosgw. > > > > We user STS with AssumeRoleWithWebIdentity, to make bucket management > > possible via a website, and didn't want to fetch all the access > credentials > > just to pick one and authenticate with that. > > > > Now when we create a bucket in the panel, we don't have access via the > > normal EC2 credentials anymore. > > > > I still think I am holding it wrong. > > This is the role and policy we've set: > > > > > # radosgw-admin role get --role-name=S3Access > > > { > > > "RoleId": "49d0d470-dc7a-4ffe-8db3-4f40cb82ebfd", > > > "RoleName": "S3Access", > > > "Path": "/", > > > "Arn": "arn:aws:iam:::role/S3Access", > > > "CreateDate": "2025-08-12T08:31:11.761Z", > > > "MaxSessionDuration": 3600, > > > "AssumeRolePolicyDocument": > > > "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":{\"Federated\":[\"arn:aws:iam:::oidc-provider/vv.xx.yy.zz:8443/realms/snc-customera\"]},\"Action\":[\"sts:AssumeRoleWithWebIdentity\"],\"Condition\":{\"StringEquals\":{\"vv.xx.yy.zz:8443/realms/snc-customera:app_id\":\"account\"}}}]}", > > > "PermissionPolicies": [ > > > { > > > "PolicyName": "Policy1", > > > "PolicyValue": > > > "{\"Version\":\"2012-10-17\",\"Statement\":{\"Effect\":\"Allow\",\"Action\":\"s3:*\",\"Resource\":\"arn:aws:s3:::*\"}}" > > > } > > > ] > > > } > > > > Best wishes > > Boris > > -- > > Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im > > groÃƒ¼en Saal. > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

1 0

rgw index auto resharding not acting
by Johan Thomsen 08 Dec '25

08 Dec '25

Hi For some reason, the auto-resharding is not doing anything, even though it reports fill status over 100%. See below: $ radosgw-admin bucket limit check --rgw-zonegroup=k1-objectstore --rgw-zone=k1-objectstore 2025-12-08T08:45:57.143+0000 7f8921528900 0 ERROR: current period 43e92ddb-d0d2-4fbc-b18b-dc87126a7664 does not contain zone id 1da0f602-9397-4997-8e76-3e3425468746 2025-12-08T08:45:57.166+0000 7f8921528900 0 period (43e92ddb-d0d2-4fbc-b18b-dc87126a7664 does not have zone 1da0f602-9397-4997-8e76-3e3425468746 configured [ { "user_id": "k1-ceph-user", "buckets": [ { "bucket": "k1-snapshots", "tenant": "", "num_objects": 5766564, "num_shards": 97, "objects_per_shard": 59449, "fill_status": "OVER 114%" } ] } Here's a dump of related config: rgw_dynamic_resharding true rgw_max_dynamic_shards 1999 rgw_max_objs_per_shard 50000 rgw_md_log_max_shards 64 rgw_objexp_hints_num_shards 127 rgw_override_bucket_index_max_shards 1031 rgw_reshard_batch_size 64 rgw_reshard_bucket_lock_duration 360 rgw_reshard_max_aio 128 rgw_reshard_num_logs 16 rgw_reshard_thread_interval 600 rgw_safe_max_objects_per_shard 52000 rgw_shard_warning_threshold 90.000000 rgw_usage_max_shards 32 Does anyone have an idea why it is not auto-resharding to a value below at least 52k objs per shard?

1 0

Re: How to fix BLUESTORE_FREE_FRAGMENTATION:
by Olivier Delcourt 08 Dec '25

08 Dec '25

> Le 8 déc. 2025 à 09:36, Robert Sander via ceph-users <ceph-users(a)ceph.io> a écrit : > > Hi, > > Am 08.12.25 um 9:32 AM schrieb Nico Schottelius via ceph-users: > >> since migrating to Squid (19.2.3), I get the following health warning: >> [WRN] BLUESTORE_FREE_FRAGMENTATION: 1 OSD(s) >> osd.25 0.806301 >> However while this is a warning, there seems to be no documented way of >> fixing it. >> What do I do with this warning / how to defrag the bluestore filesystem? > > Even the IBM documentation recommends to contact the IBM support in this case: https://www.ibm.com/docs/en/storage-ceph/8.1.0?topic=bluestore-fragmentatio… > > My best bet is to just recreate the OSD. Before that, what about trying to empty the OSD ( ceph osd reweight XXX 0) and when empty, put it back to business ( ceph osd reweight XXX 1) ? Olivier

1 0

Re: How to fix BLUESTORE_FREE_FRAGMENTATION:
by Robert Sander 08 Dec '25

08 Dec '25

Hi, Am 08.12.25 um 9:32 AM schrieb Nico Schottelius via ceph-users: > since migrating to Squid (19.2.3), I get the following health warning: > > [WRN] BLUESTORE_FREE_FRAGMENTATION: 1 OSD(s) > osd.25 0.806301 > > However while this is a warning, there seems to be no documented way of > fixing it. > > What do I do with this warning / how to defrag the bluestore filesystem? Even the IBM documentation recommends to contact the IBM support in this case: https://www.ibm.com/docs/en/storage-ceph/8.1.0?topic=bluestore-fragmentatio… My best bet is to just recreate the OSD. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

How to fix BLUESTORE_FREE_FRAGMENTATION:
by Nico Schottelius 08 Dec '25

08 Dec '25

Good morning, since migrating to Squid (19.2.3), I get the following health warning: [WRN] BLUESTORE_FREE_FRAGMENTATION: 1 OSD(s) osd.25 0.806301 However while this is a warning, there seems to be no documented way of fixing it. What do I do with this warning / how to defrag the bluestore filesystem? BR, Nico -- Sustainable and modern Infrastructures by ungleich.ch

1 0

radosgw-19.2.3 STS AssumeRoleWithWebIdentity and OIDC namespace
by Boris 08 Dec '25

08 Dec '25

Hi, since the last squid update (19.2.3) we have the problem, that the STS integration does not work anymore. (relevant tracker: https://tracker.ceph.com/issues/69924) Now there is a user with the $oidc$ prefix with the same suffix as the original user name. For example: >"$oidc$e0a0eed4f6a64c9cad70b69625dccba8", >"e0a0eed4f6a64c9cad70b69625dccba8", and both user are treated as different users in radosgw. We user STS with AssumeRoleWithWebIdentity, to make bucket management possible via a website, and didn't want to fetch all the access credentials just to pick one and authenticate with that. Now when we create a bucket in the panel, we don't have access via the normal EC2 credentials anymore. I still think I am holding it wrong. This is the role and policy we've set: > # radosgw-admin role get --role-name=S3Access > { > "RoleId": "49d0d470-dc7a-4ffe-8db3-4f40cb82ebfd", > "RoleName": "S3Access", > "Path": "/", > "Arn": "arn:aws:iam:::role/S3Access", > "CreateDate": "2025-08-12T08:31:11.761Z", > "MaxSessionDuration": 3600, > "AssumeRolePolicyDocument": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":{\"Federated\":[\"arn:aws:iam:::oidc-provider/vv.xx.yy.zz:8443/realms/snc-customera\"]},\"Action\":[\"sts:AssumeRoleWithWebIdentity\"],\"Condition\":{\"StringEquals\":{\"vv.xx.yy.zz:8443/realms/snc-customera:app_id\":\"account\"}}}]}", > "PermissionPolicies": [ > { > "PolicyName": "Policy1", > "PolicyValue": "{\"Version\":\"2012-10-17\",\"Statement\":{\"Effect\":\"Allow\",\"Action\":\"s3:*\",\"Resource\":\"arn:aws:s3:::*\"}}" > } > ] > } Best wishes Boris -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

2 1

Re: ceph mon disable_stretch_mode...no valid command found!
by Maximilian Kaya 06 Dec '25

06 Dec '25

It's solved! the ceph reef documentation was ahead of its capability. -- BR Max > On 5. Dec 2025, at 12:37, Maximilian Kaya via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi, > due to ceph reef documentation I need to run only this command but I am unable to run this to exit the stretched cluster: > ceph mon disable_stretch_mode distributed_cluster_rule --yes-i-really-mean-it > ... > no valid command found; 10 closest matches: > ... > Error EINVAL: invalid command > … > > what do I am missing ? I have ceph 18.2.7 with 5 mons and I would like to extend datacenter by adding dc3 with same number of racks,hosts and osds to get back to repsize 3 using crush rule distributed_cluster_rule: > > rule distributed_cluster_rule { > id 3 > type replicated > step take default > step choose firstn 0 type datacenter > step chooseleaf firstn 1 type rack > step emit > } > .. > #ceph mon dump > epoch 87 > fsid 583647c2-4603-11ef-9941-b360866a782f > min_mon_release 18 (reef) > election_strategy: 1 > stretch_mode_enabled 1 > tiebreaker_mon mon3 > disallowed_leaders mon3 > 0: [v2:172.16.1.1:3300/0,v1:172.16.1.1:6789/0] mon.mon1; crush_location {datacenter=dc1} > 1: [v2:172.16.1.2:3300/0,v1:172.16.1.2:6789/0] mon.mon2; crush_location {datacenter=dc2} > 2: [v2:172.16.1.3:3300/0,v1:172.16.1.3:6789/0] mon.mon3; crush_location {datacenter=quorum_room} > 3: [v2:172.16.1.4:3300/0,v1:172.16.1.4:6789/0] mon.mon4; crush_location {datacenter=dc2} > 4: [v2:172.16.1.5:3300/0,v1:172.16.1.5:6789/0] mon.mon5; crush_location {datacenter=dc1} > > > is there any other way to exit stretched cluster? any help would be appreciated. > > BR > Max > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Tentacle 20.2.0 NFS ganesh and mgmt-gateway - The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8)
by Salatiel Filho 05 Dec '25

05 Dec '25

Are there any plans to make a linux/arm64/v8 container image for quay.io/ceph/keepalived:2.2.4, quay.io/ceph/haproxy:2.3 and quay.io/ceph/nginx:sclorg-nginx-126 ? Currently it is not possible to start a HA NFS Ganesh in ceph or even the management gateway on ARM devices. They all complain about: WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested Atenciosamente/Kind regards, Salatiel

1 0

Re: Fwd: On your post: "[Tentacle 20.2.0]: Inconsistent pg's after enabling ec optimisation flag"
by Jeff Bailey 05 Dec '25

05 Dec '25

I've seen OSD's crash when running an fstrim on an RBD with an EC data pool with the EC optimizations enabled. Always two OSD's. It's a 8+2 pool but I don't know if the "2"s are a coincidence or not. Other than that, I haven't seen any OSD crashes with EC optimizations. We had the same size mismatches that went away after migrating the data. On 12/5/2025 10:34 AM, Reto Gysi via ceph-users wrote: > Hi Bill > > Am Fr., 5. Dez. 2025 um 10:02 Uhr schrieb Bill Scales < > bill_scales(a)uk.ibm.com>: > >> Hi Reto, >> >> Sorry to hear about your problems with turning on ec optimizations. I've >> led the team that developed this feature, so we are keen to help understand >> what happened to your system. >> >> Your configuration looks fine as far as being supported with ec >> optimizations. The daemons (mons, osds and mgr) need to be running tentacle >> code to use this feature, there is no requirement to update any of your >> clients. >> >> Do you have any more examples of the inconsistent objects logs that you >> can share with me? >> >> There was a bug that we fixed late in the development cycle which was >> scrubbing incorrectly reporting a size mismatch for objects written prior >> to turning on optimizations. This is because prior to turning on >> optimizations objects are padded to a multiple of the stripe_width, >> afterwards objects are no longer padded. The scrub code was occasionally >> getting confused and incorrectly reporting an inconsistency. In this case >> the scrub was reporting false positive errors - there was nothing wrong >> with the data and no problems accessing the data. I notice the log you >> shared in the email was for a size mismatch, I'm interested whether all the >> logs were for mismatched sizes. We will do some further work to confirm >> that the fix is in the 20.2.0 tentacle release and that there are no >> problems with the back port. >> > As far I could see the errors where all about the mismatched sizes. > > >> You also mention that you saw some OSD crashes. Do you have any further >> information about these? >> > I just could cause another crash with starting up a windows VM, and doing a > WIndows file history backup to a drive that is on an RBD Image on pool > rbd_ecpool where I had the allow_ec_optimization flag enabled. > > <disk type="network" device="disk"> > <driver name="qemu" type="raw" cache="writethrough" io="threads" > discard="unmap" detect_zeroes="unmap"/> > <auth username="admin"> > <secret type="ceph" uuid="878b0bc5-c471-4ec6-a92a-f65282ffbdf6"/> > </auth> > <source protocol="rbd" name="rbd/game_windows_backup_drive" index="3"> > <host name="zephir" port="6789"/> > </source> > <target dev="vdd" bus="virtio"/> > <alias name="virtio-disk3"/> > <address type="pci" domain="0x0000" bus="0x00" slot="0x02" > function="0x0"/> > </disk> > > root@zephir:~# ceph -s > cluster: > id: 27923302-87a5-11ec-ac5b-976d21a49941 > health: HEALTH_WARN > 2 osds down > Reduced data availability: 49 pgs inactive > Degraded data redundancy: 15722192/79848340 objects degraded > (19.690%), 246 pgs degraded, 247 pgs undersized > > services: > mon: 3 daemons, quorum zephir,debian,raspi (age 20h) [leader: > zephir] > mgr: zephir.enywvy(active, since 21h), standbys: debian.nusuye > mds: 3/3 daemons up, 3 standby > osd: 18 osds: 16 up (since 5m), 18 in (since 3d); 8 remapped > pgs > cephfs-mirror: 2 daemons active (2 hosts) > rbd-mirror: 2 daemons active (2 hosts) > rgw: 1 daemon active (1 hosts, 1 zones) > tcmu-runner: 5 portals active (2 hosts) > > data: > volumes: 3/3 healthy > pools: 25 pools, 450 pgs > objects: 17.52M objects, 62 TiB > usage: 116 TiB used, 63 TiB / 179 TiB avail > pgs: 10.889% pgs not active > 15722192/79848340 objects degraded (19.690%) > 197 active+undersized+degraded > 183 active+clean > 49 undersized+degraded+peered > 15 active+clean+scrubbing > 5 active+clean+scrubbing+deep > 1 active+undersized > > io: > client: 1.7 KiB/s rd, 1023 B/s wr, 1 op/s rd, 0 op/s wr > > root@zephir:~# ceph osd status > ID HOST USED AVAIL WR OPS WR DATA RD OPS RD DATA STATE > 0 debian 11.2T 5289G 0 0 0 0 exists,up > 1 zephir 11.6T 4846G 0 0 0 0 exists > 2 zephir 10.7T 5827G 0 0 2 4 exists,up > 3 zephir 11.0T 5497G 0 0 0 0 exists > 4 zephir 159G 1376G 0 0 2 0 exists,up > 5 zephir 120G 1415G 0 0 0 0 exists,up > 6 debian 11.3T 5200G 0 0 1 0 exists,up > 7 zephir 305G 1230G 0 0 0 0 exists,up > 8 zephir 10.9T 5594G 0 0 0 0 exists,up > 9 zephir 11.9T 4616G 0 0 0 0 exists,up > 10 zephir 235G 1300G 0 0 0 0 exists,up > 11 zephir 223G 1312G 0 0 0 0 exists,up > 12 zephir 11.0T 5479G 0 0 0 0 exists,up > 13 debian 118G 631G 0 0 3 8 exists,up > 14 debian 345G 1190G 0 0 0 0 exists,up > 15 debian 11.4T 5079G 0 0 0 0 exists,up > 16 debian 554G 3029G 0 0 3 8 exists,up > 17 debian 12.5T 5874G 0 0 1 0 exists,up > root@zephir:~# > > I attached the log of osd.1 to > https://filebin.net/jwvs6kuqrc7hx8id > > Best Regards, > > Reto > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Fwd: On your post: "[Tentacle 20.2.0]: Inconsistent pg's after enabling ec optimisation flag"
by Reto Gysi 05 Dec '25

05 Dec '25

Hi Bill Am Fr., 5. Dez. 2025 um 10:02 Uhr schrieb Bill Scales < bill_scales(a)uk.ibm.com>: > Hi Reto, > > Sorry to hear about your problems with turning on ec optimizations. I've > led the team that developed this feature, so we are keen to help understand > what happened to your system. > > Your configuration looks fine as far as being supported with ec > optimizations. The daemons (mons, osds and mgr) need to be running tentacle > code to use this feature, there is no requirement to update any of your > clients. > > Do you have any more examples of the inconsistent objects logs that you > can share with me? > > There was a bug that we fixed late in the development cycle which was > scrubbing incorrectly reporting a size mismatch for objects written prior > to turning on optimizations. This is because prior to turning on > optimizations objects are padded to a multiple of the stripe_width, > afterwards objects are no longer padded. The scrub code was occasionally > getting confused and incorrectly reporting an inconsistency. In this case > the scrub was reporting false positive errors - there was nothing wrong > with the data and no problems accessing the data. I notice the log you > shared in the email was for a size mismatch, I'm interested whether all the > logs were for mismatched sizes. We will do some further work to confirm > that the fix is in the 20.2.0 tentacle release and that there are no > problems with the back port. > As far I could see the errors where all about the mismatched sizes. > > You also mention that you saw some OSD crashes. Do you have any further > information about these? > I just could cause another crash with starting up a windows VM, and doing a WIndows file history backup to a drive that is on an RBD Image on pool rbd_ecpool where I had the allow_ec_optimization flag enabled. <disk type="network" device="disk"> <driver name="qemu" type="raw" cache="writethrough" io="threads" discard="unmap" detect_zeroes="unmap"/> <auth username="admin"> <secret type="ceph" uuid="878b0bc5-c471-4ec6-a92a-f65282ffbdf6"/> </auth> <source protocol="rbd" name="rbd/game_windows_backup_drive" index="3"> <host name="zephir" port="6789"/> </source> <target dev="vdd" bus="virtio"/> <alias name="virtio-disk3"/> <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0"/> </disk> root@zephir:~# ceph -s cluster: id: 27923302-87a5-11ec-ac5b-976d21a49941 health: HEALTH_WARN 2 osds down Reduced data availability: 49 pgs inactive Degraded data redundancy: 15722192/79848340 objects degraded (19.690%), 246 pgs degraded, 247 pgs undersized services: mon: 3 daemons, quorum zephir,debian,raspi (age 20h) [leader: zephir] mgr: zephir.enywvy(active, since 21h), standbys: debian.nusuye mds: 3/3 daemons up, 3 standby osd: 18 osds: 16 up (since 5m), 18 in (since 3d); 8 remapped pgs cephfs-mirror: 2 daemons active (2 hosts) rbd-mirror: 2 daemons active (2 hosts) rgw: 1 daemon active (1 hosts, 1 zones) tcmu-runner: 5 portals active (2 hosts) data: volumes: 3/3 healthy pools: 25 pools, 450 pgs objects: 17.52M objects, 62 TiB usage: 116 TiB used, 63 TiB / 179 TiB avail pgs: 10.889% pgs not active 15722192/79848340 objects degraded (19.690%) 197 active+undersized+degraded 183 active+clean 49 undersized+degraded+peered 15 active+clean+scrubbing 5 active+clean+scrubbing+deep 1 active+undersized io: client: 1.7 KiB/s rd, 1023 B/s wr, 1 op/s rd, 0 op/s wr root@zephir:~# ceph osd status ID HOST USED AVAIL WR OPS WR DATA RD OPS RD DATA STATE 0 debian 11.2T 5289G 0 0 0 0 exists,up 1 zephir 11.6T 4846G 0 0 0 0 exists 2 zephir 10.7T 5827G 0 0 2 4 exists,up 3 zephir 11.0T 5497G 0 0 0 0 exists 4 zephir 159G 1376G 0 0 2 0 exists,up 5 zephir 120G 1415G 0 0 0 0 exists,up 6 debian 11.3T 5200G 0 0 1 0 exists,up 7 zephir 305G 1230G 0 0 0 0 exists,up 8 zephir 10.9T 5594G 0 0 0 0 exists,up 9 zephir 11.9T 4616G 0 0 0 0 exists,up 10 zephir 235G 1300G 0 0 0 0 exists,up 11 zephir 223G 1312G 0 0 0 0 exists,up 12 zephir 11.0T 5479G 0 0 0 0 exists,up 13 debian 118G 631G 0 0 3 8 exists,up 14 debian 345G 1190G 0 0 0 0 exists,up 15 debian 11.4T 5079G 0 0 0 0 exists,up 16 debian 554G 3029G 0 0 3 8 exists,up 17 debian 12.5T 5874G 0 0 1 0 exists,up root@zephir:~# I attached the log of osd.1 to https://filebin.net/jwvs6kuqrc7hx8id Best Regards, Reto >

1 0