- ceph-users - lists.ceph.io

Re: How to fix BLUESTORE_FREE_FRAGMENTATION:
by Olivier Delcourt 08 Dec '25

08 Dec '25

> Le 8 déc. 2025 à 09:36, Robert Sander via ceph-users <ceph-users(a)ceph.io> a écrit : > > Hi, > > Am 08.12.25 um 9:32 AM schrieb Nico Schottelius via ceph-users: > >> since migrating to Squid (19.2.3), I get the following health warning: >> [WRN] BLUESTORE_FREE_FRAGMENTATION: 1 OSD(s) >> osd.25 0.806301 >> However while this is a warning, there seems to be no documented way of >> fixing it. >> What do I do with this warning / how to defrag the bluestore filesystem? > > Even the IBM documentation recommends to contact the IBM support in this case: https://www.ibm.com/docs/en/storage-ceph/8.1.0?topic=bluestore-fragmentatio… > > My best bet is to just recreate the OSD. Before that, what about trying to empty the OSD ( ceph osd reweight XXX 0) and when empty, put it back to business ( ceph osd reweight XXX 1) ? Olivier

1 0

Re: How to fix BLUESTORE_FREE_FRAGMENTATION:
by Robert Sander 08 Dec '25

08 Dec '25

Hi, Am 08.12.25 um 9:32 AM schrieb Nico Schottelius via ceph-users: > since migrating to Squid (19.2.3), I get the following health warning: > > [WRN] BLUESTORE_FREE_FRAGMENTATION: 1 OSD(s) > osd.25 0.806301 > > However while this is a warning, there seems to be no documented way of > fixing it. > > What do I do with this warning / how to defrag the bluestore filesystem? Even the IBM documentation recommends to contact the IBM support in this case: https://www.ibm.com/docs/en/storage-ceph/8.1.0?topic=bluestore-fragmentatio… My best bet is to just recreate the OSD. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

How to fix BLUESTORE_FREE_FRAGMENTATION:
by Nico Schottelius 08 Dec '25

08 Dec '25

Good morning, since migrating to Squid (19.2.3), I get the following health warning: [WRN] BLUESTORE_FREE_FRAGMENTATION: 1 OSD(s) osd.25 0.806301 However while this is a warning, there seems to be no documented way of fixing it. What do I do with this warning / how to defrag the bluestore filesystem? BR, Nico -- Sustainable and modern Infrastructures by ungleich.ch

1 0

radosgw-19.2.3 STS AssumeRoleWithWebIdentity and OIDC namespace
by Boris 08 Dec '25

08 Dec '25

Hi, since the last squid update (19.2.3) we have the problem, that the STS integration does not work anymore. (relevant tracker: https://tracker.ceph.com/issues/69924) Now there is a user with the $oidc$ prefix with the same suffix as the original user name. For example: >"$oidc$e0a0eed4f6a64c9cad70b69625dccba8", >"e0a0eed4f6a64c9cad70b69625dccba8", and both user are treated as different users in radosgw. We user STS with AssumeRoleWithWebIdentity, to make bucket management possible via a website, and didn't want to fetch all the access credentials just to pick one and authenticate with that. Now when we create a bucket in the panel, we don't have access via the normal EC2 credentials anymore. I still think I am holding it wrong. This is the role and policy we've set: > # radosgw-admin role get --role-name=S3Access > { > "RoleId": "49d0d470-dc7a-4ffe-8db3-4f40cb82ebfd", > "RoleName": "S3Access", > "Path": "/", > "Arn": "arn:aws:iam:::role/S3Access", > "CreateDate": "2025-08-12T08:31:11.761Z", > "MaxSessionDuration": 3600, > "AssumeRolePolicyDocument": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":{\"Federated\":[\"arn:aws:iam:::oidc-provider/vv.xx.yy.zz:8443/realms/snc-customera\"]},\"Action\":[\"sts:AssumeRoleWithWebIdentity\"],\"Condition\":{\"StringEquals\":{\"vv.xx.yy.zz:8443/realms/snc-customera:app_id\":\"account\"}}}]}", > "PermissionPolicies": [ > { > "PolicyName": "Policy1", > "PolicyValue": "{\"Version\":\"2012-10-17\",\"Statement\":{\"Effect\":\"Allow\",\"Action\":\"s3:*\",\"Resource\":\"arn:aws:s3:::*\"}}" > } > ] > } Best wishes Boris -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

2 1

Re: ceph mon disable_stretch_mode...no valid command found!
by Maximilian Kaya 06 Dec '25

06 Dec '25

It's solved! the ceph reef documentation was ahead of its capability. -- BR Max > On 5. Dec 2025, at 12:37, Maximilian Kaya via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi, > due to ceph reef documentation I need to run only this command but I am unable to run this to exit the stretched cluster: > ceph mon disable_stretch_mode distributed_cluster_rule --yes-i-really-mean-it > ... > no valid command found; 10 closest matches: > ... > Error EINVAL: invalid command > … > > what do I am missing ? I have ceph 18.2.7 with 5 mons and I would like to extend datacenter by adding dc3 with same number of racks,hosts and osds to get back to repsize 3 using crush rule distributed_cluster_rule: > > rule distributed_cluster_rule { > id 3 > type replicated > step take default > step choose firstn 0 type datacenter > step chooseleaf firstn 1 type rack > step emit > } > .. > #ceph mon dump > epoch 87 > fsid 583647c2-4603-11ef-9941-b360866a782f > min_mon_release 18 (reef) > election_strategy: 1 > stretch_mode_enabled 1 > tiebreaker_mon mon3 > disallowed_leaders mon3 > 0: [v2:172.16.1.1:3300/0,v1:172.16.1.1:6789/0] mon.mon1; crush_location {datacenter=dc1} > 1: [v2:172.16.1.2:3300/0,v1:172.16.1.2:6789/0] mon.mon2; crush_location {datacenter=dc2} > 2: [v2:172.16.1.3:3300/0,v1:172.16.1.3:6789/0] mon.mon3; crush_location {datacenter=quorum_room} > 3: [v2:172.16.1.4:3300/0,v1:172.16.1.4:6789/0] mon.mon4; crush_location {datacenter=dc2} > 4: [v2:172.16.1.5:3300/0,v1:172.16.1.5:6789/0] mon.mon5; crush_location {datacenter=dc1} > > > is there any other way to exit stretched cluster? any help would be appreciated. > > BR > Max > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Tentacle 20.2.0 NFS ganesh and mgmt-gateway - The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8)
by Salatiel Filho 05 Dec '25

05 Dec '25

Are there any plans to make a linux/arm64/v8 container image for quay.io/ceph/keepalived:2.2.4, quay.io/ceph/haproxy:2.3 and quay.io/ceph/nginx:sclorg-nginx-126 ? Currently it is not possible to start a HA NFS Ganesh in ceph or even the management gateway on ARM devices. They all complain about: WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested Atenciosamente/Kind regards, Salatiel

1 0

Re: Fwd: On your post: "[Tentacle 20.2.0]: Inconsistent pg's after enabling ec optimisation flag"
by Jeff Bailey 05 Dec '25

05 Dec '25

I've seen OSD's crash when running an fstrim on an RBD with an EC data pool with the EC optimizations enabled. Always two OSD's. It's a 8+2 pool but I don't know if the "2"s are a coincidence or not. Other than that, I haven't seen any OSD crashes with EC optimizations. We had the same size mismatches that went away after migrating the data. On 12/5/2025 10:34 AM, Reto Gysi via ceph-users wrote: > Hi Bill > > Am Fr., 5. Dez. 2025 um 10:02 Uhr schrieb Bill Scales < > bill_scales(a)uk.ibm.com>: > >> Hi Reto, >> >> Sorry to hear about your problems with turning on ec optimizations. I've >> led the team that developed this feature, so we are keen to help understand >> what happened to your system. >> >> Your configuration looks fine as far as being supported with ec >> optimizations. The daemons (mons, osds and mgr) need to be running tentacle >> code to use this feature, there is no requirement to update any of your >> clients. >> >> Do you have any more examples of the inconsistent objects logs that you >> can share with me? >> >> There was a bug that we fixed late in the development cycle which was >> scrubbing incorrectly reporting a size mismatch for objects written prior >> to turning on optimizations. This is because prior to turning on >> optimizations objects are padded to a multiple of the stripe_width, >> afterwards objects are no longer padded. The scrub code was occasionally >> getting confused and incorrectly reporting an inconsistency. In this case >> the scrub was reporting false positive errors - there was nothing wrong >> with the data and no problems accessing the data. I notice the log you >> shared in the email was for a size mismatch, I'm interested whether all the >> logs were for mismatched sizes. We will do some further work to confirm >> that the fix is in the 20.2.0 tentacle release and that there are no >> problems with the back port. >> > As far I could see the errors where all about the mismatched sizes. > > >> You also mention that you saw some OSD crashes. Do you have any further >> information about these? >> > I just could cause another crash with starting up a windows VM, and doing a > WIndows file history backup to a drive that is on an RBD Image on pool > rbd_ecpool where I had the allow_ec_optimization flag enabled. > > <disk type="network" device="disk"> > <driver name="qemu" type="raw" cache="writethrough" io="threads" > discard="unmap" detect_zeroes="unmap"/> > <auth username="admin"> > <secret type="ceph" uuid="878b0bc5-c471-4ec6-a92a-f65282ffbdf6"/> > </auth> > <source protocol="rbd" name="rbd/game_windows_backup_drive" index="3"> > <host name="zephir" port="6789"/> > </source> > <target dev="vdd" bus="virtio"/> > <alias name="virtio-disk3"/> > <address type="pci" domain="0x0000" bus="0x00" slot="0x02" > function="0x0"/> > </disk> > > root@zephir:~# ceph -s > cluster: > id: 27923302-87a5-11ec-ac5b-976d21a49941 > health: HEALTH_WARN > 2 osds down > Reduced data availability: 49 pgs inactive > Degraded data redundancy: 15722192/79848340 objects degraded > (19.690%), 246 pgs degraded, 247 pgs undersized > > services: > mon: 3 daemons, quorum zephir,debian,raspi (age 20h) [leader: > zephir] > mgr: zephir.enywvy(active, since 21h), standbys: debian.nusuye > mds: 3/3 daemons up, 3 standby > osd: 18 osds: 16 up (since 5m), 18 in (since 3d); 8 remapped > pgs > cephfs-mirror: 2 daemons active (2 hosts) > rbd-mirror: 2 daemons active (2 hosts) > rgw: 1 daemon active (1 hosts, 1 zones) > tcmu-runner: 5 portals active (2 hosts) > > data: > volumes: 3/3 healthy > pools: 25 pools, 450 pgs > objects: 17.52M objects, 62 TiB > usage: 116 TiB used, 63 TiB / 179 TiB avail > pgs: 10.889% pgs not active > 15722192/79848340 objects degraded (19.690%) > 197 active+undersized+degraded > 183 active+clean > 49 undersized+degraded+peered > 15 active+clean+scrubbing > 5 active+clean+scrubbing+deep > 1 active+undersized > > io: > client: 1.7 KiB/s rd, 1023 B/s wr, 1 op/s rd, 0 op/s wr > > root@zephir:~# ceph osd status > ID HOST USED AVAIL WR OPS WR DATA RD OPS RD DATA STATE > 0 debian 11.2T 5289G 0 0 0 0 exists,up > 1 zephir 11.6T 4846G 0 0 0 0 exists > 2 zephir 10.7T 5827G 0 0 2 4 exists,up > 3 zephir 11.0T 5497G 0 0 0 0 exists > 4 zephir 159G 1376G 0 0 2 0 exists,up > 5 zephir 120G 1415G 0 0 0 0 exists,up > 6 debian 11.3T 5200G 0 0 1 0 exists,up > 7 zephir 305G 1230G 0 0 0 0 exists,up > 8 zephir 10.9T 5594G 0 0 0 0 exists,up > 9 zephir 11.9T 4616G 0 0 0 0 exists,up > 10 zephir 235G 1300G 0 0 0 0 exists,up > 11 zephir 223G 1312G 0 0 0 0 exists,up > 12 zephir 11.0T 5479G 0 0 0 0 exists,up > 13 debian 118G 631G 0 0 3 8 exists,up > 14 debian 345G 1190G 0 0 0 0 exists,up > 15 debian 11.4T 5079G 0 0 0 0 exists,up > 16 debian 554G 3029G 0 0 3 8 exists,up > 17 debian 12.5T 5874G 0 0 1 0 exists,up > root@zephir:~# > > I attached the log of osd.1 to > https://filebin.net/jwvs6kuqrc7hx8id > > Best Regards, > > Reto > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Fwd: On your post: "[Tentacle 20.2.0]: Inconsistent pg's after enabling ec optimisation flag"
by Reto Gysi 05 Dec '25

05 Dec '25

Hi Bill Am Fr., 5. Dez. 2025 um 10:02 Uhr schrieb Bill Scales < bill_scales(a)uk.ibm.com>: > Hi Reto, > > Sorry to hear about your problems with turning on ec optimizations. I've > led the team that developed this feature, so we are keen to help understand > what happened to your system. > > Your configuration looks fine as far as being supported with ec > optimizations. The daemons (mons, osds and mgr) need to be running tentacle > code to use this feature, there is no requirement to update any of your > clients. > > Do you have any more examples of the inconsistent objects logs that you > can share with me? > > There was a bug that we fixed late in the development cycle which was > scrubbing incorrectly reporting a size mismatch for objects written prior > to turning on optimizations. This is because prior to turning on > optimizations objects are padded to a multiple of the stripe_width, > afterwards objects are no longer padded. The scrub code was occasionally > getting confused and incorrectly reporting an inconsistency. In this case > the scrub was reporting false positive errors - there was nothing wrong > with the data and no problems accessing the data. I notice the log you > shared in the email was for a size mismatch, I'm interested whether all the > logs were for mismatched sizes. We will do some further work to confirm > that the fix is in the 20.2.0 tentacle release and that there are no > problems with the back port. > As far I could see the errors where all about the mismatched sizes. > > You also mention that you saw some OSD crashes. Do you have any further > information about these? > I just could cause another crash with starting up a windows VM, and doing a WIndows file history backup to a drive that is on an RBD Image on pool rbd_ecpool where I had the allow_ec_optimization flag enabled. <disk type="network" device="disk"> <driver name="qemu" type="raw" cache="writethrough" io="threads" discard="unmap" detect_zeroes="unmap"/> <auth username="admin"> <secret type="ceph" uuid="878b0bc5-c471-4ec6-a92a-f65282ffbdf6"/> </auth> <source protocol="rbd" name="rbd/game_windows_backup_drive" index="3"> <host name="zephir" port="6789"/> </source> <target dev="vdd" bus="virtio"/> <alias name="virtio-disk3"/> <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0"/> </disk> root@zephir:~# ceph -s cluster: id: 27923302-87a5-11ec-ac5b-976d21a49941 health: HEALTH_WARN 2 osds down Reduced data availability: 49 pgs inactive Degraded data redundancy: 15722192/79848340 objects degraded (19.690%), 246 pgs degraded, 247 pgs undersized services: mon: 3 daemons, quorum zephir,debian,raspi (age 20h) [leader: zephir] mgr: zephir.enywvy(active, since 21h), standbys: debian.nusuye mds: 3/3 daemons up, 3 standby osd: 18 osds: 16 up (since 5m), 18 in (since 3d); 8 remapped pgs cephfs-mirror: 2 daemons active (2 hosts) rbd-mirror: 2 daemons active (2 hosts) rgw: 1 daemon active (1 hosts, 1 zones) tcmu-runner: 5 portals active (2 hosts) data: volumes: 3/3 healthy pools: 25 pools, 450 pgs objects: 17.52M objects, 62 TiB usage: 116 TiB used, 63 TiB / 179 TiB avail pgs: 10.889% pgs not active 15722192/79848340 objects degraded (19.690%) 197 active+undersized+degraded 183 active+clean 49 undersized+degraded+peered 15 active+clean+scrubbing 5 active+clean+scrubbing+deep 1 active+undersized io: client: 1.7 KiB/s rd, 1023 B/s wr, 1 op/s rd, 0 op/s wr root@zephir:~# ceph osd status ID HOST USED AVAIL WR OPS WR DATA RD OPS RD DATA STATE 0 debian 11.2T 5289G 0 0 0 0 exists,up 1 zephir 11.6T 4846G 0 0 0 0 exists 2 zephir 10.7T 5827G 0 0 2 4 exists,up 3 zephir 11.0T 5497G 0 0 0 0 exists 4 zephir 159G 1376G 0 0 2 0 exists,up 5 zephir 120G 1415G 0 0 0 0 exists,up 6 debian 11.3T 5200G 0 0 1 0 exists,up 7 zephir 305G 1230G 0 0 0 0 exists,up 8 zephir 10.9T 5594G 0 0 0 0 exists,up 9 zephir 11.9T 4616G 0 0 0 0 exists,up 10 zephir 235G 1300G 0 0 0 0 exists,up 11 zephir 223G 1312G 0 0 0 0 exists,up 12 zephir 11.0T 5479G 0 0 0 0 exists,up 13 debian 118G 631G 0 0 3 8 exists,up 14 debian 345G 1190G 0 0 0 0 exists,up 15 debian 11.4T 5079G 0 0 0 0 exists,up 16 debian 554G 3029G 0 0 3 8 exists,up 17 debian 12.5T 5874G 0 0 1 0 exists,up root@zephir:~# I attached the log of osd.1 to https://filebin.net/jwvs6kuqrc7hx8id Best Regards, Reto >

1 0

ceph mon disable_stretch_mode...no valid command found!
by Maximilian Kaya 05 Dec '25

05 Dec '25

Hi, due to ceph reef documentation I need to run only this command but I am unable to run this to exit the stretched cluster: ceph mon disable_stretch_mode distributed_cluster_rule --yes-i-really-mean-it ... no valid command found; 10 closest matches: ... Error EINVAL: invalid command … what do I am missing ? I have ceph 18.2.7 with 5 mons and I would like to extend datacenter by adding dc3 with same number of racks,hosts and osds to get back to repsize 3 using crush rule distributed_cluster_rule: rule distributed_cluster_rule { id 3 type replicated step take default step choose firstn 0 type datacenter step chooseleaf firstn 1 type rack step emit } .. #ceph mon dump epoch 87 fsid 583647c2-4603-11ef-9941-b360866a782f min_mon_release 18 (reef) election_strategy: 1 stretch_mode_enabled 1 tiebreaker_mon mon3 disallowed_leaders mon3 0: [v2:172.16.1.1:3300/0,v1:172.16.1.1:6789/0] mon.mon1; crush_location {datacenter=dc1} 1: [v2:172.16.1.2:3300/0,v1:172.16.1.2:6789/0] mon.mon2; crush_location {datacenter=dc2} 2: [v2:172.16.1.3:3300/0,v1:172.16.1.3:6789/0] mon.mon3; crush_location {datacenter=quorum_room} 3: [v2:172.16.1.4:3300/0,v1:172.16.1.4:6789/0] mon.mon4; crush_location {datacenter=dc2} 4: [v2:172.16.1.5:3300/0,v1:172.16.1.5:6789/0] mon.mon5; crush_location {datacenter=dc1} is there any other way to exit stretched cluster? any help would be appreciated. BR Max

1 0

Re: reef 18.2.8 QE validation status
by Travis Nielsen 04 Dec '25

04 Dec '25

The latest Reef tests are passing for Rook. Travis On Mon, Dec 1, 2025 at 7:18 AM Ilya Dryomov via ceph-users < ceph-users(a)ceph.io> wrote: > On Sat, Nov 29, 2025 at 6:08 PM Yuri Weinstein <yweinste(a)redhat.com> > wrote: > > > > Rerun scheduled for krbd > > It's still the same. I'd suggest keeping an eye on the run before > "admitting" it for a possible approval -- something that is completely > red with not a single job reaching the point where the actual test > starts clearly can't be approved. > > Thanks, > > Ilya > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0