- ceph-users - lists.ceph.io

Re: squid 19.2.4 QE validation status
by Redouane Kachach 21 Apr '26

21 Apr '26

Orch approved On Mon, Apr 20, 2026 at 8:45 AM Guillaume ABRIOUX via dev <dev(a)ceph.io> wrote: > Hello Yuri, > > ceph-volume approved. > > > Thanks! > > -- > > Guillaume Abrioux > > Software Engineer > *From: *Yuri Weinstein via ceph-users <ceph-users(a)ceph.io> > *Date: *Monday, 13 April 2026 at 21:25 > *To: *dev <dev(a)ceph.io>, ceph-users <ceph-users(a)ceph.io> > *Subject: *[EXTERNAL] [ceph-users] squid 19.2.4 QE validation status > > Details of this release are summarized here: > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__tracker.ceph.com_issue… > > Release Notes - TBD > LRC upgrade - N/A > > Seeking approvals/reviews for: > > smoke: PASSED > > rados: Radek, Laura, Afreen, Travis > rgw: Adam Emerson > fs: Venky > orch/ceph-volume: Redouane & Guillaume > rbd: Ilya > krbd: Ilya > > upgrade/squid-x: Laura, Patrick > upgrade/reef-x: Laura, Patrick > upgrade/telemetry-upgrade: Laura, Patrick > upgrade/squid-p2p: Laura, Patrick > > upgrade-clients:client-upgrade-reef-squid-squid: PASSED > upgrade-clients:client-upgrade-quincy-squid-squid: PASSED > > perf-basic: PASSED > crimson-rados: N/A > powercycle: PASSED > > TIA > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > Unless otherwise stated above: > > Compagnie IBM France > Siège Social : 17, avenue de l'Europe, 92275 Bois-Colombes Cedex > RCS Nanterre 552 118 465 > Forme Sociale : S.A.S. > Capital Social : 664 614 175,50 € > SIRET : 552 118 465 03644 - Code NAF 6203Z >

1 0

Re: ceph 19.2.3 health POOL_TOO_MANY_PGS / POOL_TOO_FEW_PGS ping-pong
by Anthony D'Atri 20 Apr '26

20 Apr '26

> The only reason setting mode=off is the observed ping-pong. Ah, I didn't catch that you were actively experiencing that. > The index pool is small (3 GiB stored) and PG merge/split is fast (~3 minutes). Right after adjusting pg_num (to 32/64) the health warning flips into the opposite, several times, back and forth. Do you set the mode to on for all pools? Does every pool use a CRUSH rule that specifies a device class? > The SSD-OSD's size is only 32 GiB, 4 per host and 3 hosts. The small size and count was chosen out of unawareness (pre-created LVs). Remind me please, how many OSDs total? The index pool does benefit from more vs larger OSDs, though if you're shaving that device class with other purposes it can be nuanced. If > Could this be something that confuses the autoscaler? For sure the autoscaler isn't great at very small clusters in some cases, but I wouldn't expected flapping. > >> What you're thinking of is the per-pool threshold. Historically this has defaulted to 3.0 : the autoscaler won't make a change until the calculated value varies by more than a factor of 3 from the current setting. I think of this as hysteresis, but there's probably a better term. There's been some fairly recent work to do this more surgically, which I _think_ removes the per-pool threshold, but in any event, I believe that recent autoscaler releases are fairly immune to this flapping. > > Thanks for clarification on BIAS and threshold. Yes I meant threshold, but looks like threshold is obsolete long ago (can't find it in the docs[1])? Curious, could be that we never documented it upstream. IBM FWIW documents this: https://www.ibm.com/docs/en/storage-ceph/9.9.0?topic=groups-configuring-pla… (IBM 9 is Tentacle, 8 is Squid) > At least for me the autoscaler seems not immune to flapping. What's your `ceph df`? Do you set any of the knobs to non-default values? I would think that your buckets.data pool would show a higher RATIO, but the bulk flag on it may be a factor. I've yet to fully wrap my head around how the bulk flag actually works, but it might be interesting to set it on the index pool too to see what happens with the autoscaler enabled. You might try # ceph osd pool set default.rgw.buckets.index pg_num_min 64 > > Thanks, Sascha. > > [1] https://docs.ceph.com/en/latest/rados/operations/pools/#setting-pool-values > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: ceph 19.2.3 health POOL_TOO_MANY_PGS / POOL_TOO_FEW_PGS ping-pong
by Sascha Lucas 20 Apr '26

20 Apr '26

Hi Anthony, On Sat, 18 Apr 2026, Anthony D'Atri via ceph-users wrote: > You might just periodically set them to warn, capture the info, and turn > it back off. But I do have to ask, is there a specific reason you > aren't using the autoscaler? It's much better than it used to be, and > there are really great improvements in the pipe. The only reason setting mode=off is the observed ping-pong. The index pool is small (3 GiB stored) and PG merge/split is fast (~3 minutes). Right after adjusting pg_num (to 32/64) the health warning flips into the opposite, several times, back and forth. The SSD-OSD's size is only 32 GiB, 4 per host and 3 hosts. The small size and count was chosen out of unawareness (pre-created LVs). Cloud this be something that confuses the autoscaler? > What you're thinking of is the per-pool threshold. Historically this > has defaulted to 3.0 : the autoscaler won't make a change until the > calculated value varies by more than a factor of 3 from the current > setting. I think of this as hysteresis, but there's probably a better > term. There's been some fairly recent work to do this more surgically, > which I _think_ removes the per-pool threshold, but in any event, I > believe that recent autoscaler releases are fairly immune to this > flapping. Thanks for clarification on BIAS and threshold. Yes I meant threshold, but looks like threshold is obsolete long ago (can't find it in the docs[1])? At least for me the autoscaler seems not immune to flapping. Thanks, Sascha. [1] https://docs.ceph.com/en/latest/rados/operations/pools/#setting-pool-values

1 0

Re: Tentacle, hybrid-OSD bug and making space for block.db
by Marek Szuba 20 Apr '26

20 Apr '26

On 2026-04-17 15:18, Anthony D'Atri wrote: > I suspect that the answer might be something ilke: > > * Use the tool to migrate to the slow device > * Compact it there (if the tool doesn't compact as a function of the migration) > * Migrate it to a smaller offload slice Judging from Igor's response, looks like this is indeed the way to go here. Oh well, fortunately we aren't short on space just yet so I think we'll save ourselves the hassle and just keep waiting for whichever Tentacle release ends up incorporating the fix. Thank you for your suggestions! -- MS

1 0

Re: Tentacle cluster: health OK, no recovery traffic, 24% objects misplaced
by Marek Szuba 20 Apr '26

20 Apr '26

Dear everyone, Update, on Friday we decided to apply blunt force to the problem (this still being a prototype cluster for the time being) and started rebooting nodes, one by one, again. After the second rebooted node had come back online the misplaced percentage jumped by about 10 percent but we finally began to see 'ceph -s' report recovery traffic - and when I checked this morning all PGs were back to normal. Just in case this is helps anyone figure out what did happen, so that we can avoid this / fix it faster in the future: * CRUSH: we do group our hosts into buckets corresponding to the server rooms they are located in (of which there are three). Apart from that, nothing non-default; * EC pools - yes, we have got some of these. See below. * pool and CRUSH-rule details: pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 6 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 545 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 50.00 pool 2 'cephfs.fs1.meta' replicated size 3 min_size 2 crush_rule 6 object_hash rjenkins pg_num 16 pgp_num 16 autoscale_mode on last_change 548 lfor 0/0/153 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 3.02 pool 3 'cephfs.fs1.data' replicated size 3 min_size 2 crush_rule 6 object_hash rjenkins pg_num 1024 pgp_num 1024 autoscale_mode on last_change 38714 lfor 0/0/201 flags hashpspool,selfmanaged_snaps,bulk stripe_width 0 application cephfs read_balance_score 1.08 pool 4 '.nfs' replicated size 3 min_size 2 crush_rule 6 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 546 lfor 0/0/233 flags hashpspool stripe_width 0 application nfs read_balance_score 2.99 pool 5 'cephfs.fs1.repl1' replicated size 3 min_size 2 crush_rule 6 object_hash rjenkins pg_num 512 pgp_num 512 autoscale_mode on last_change 38714 lfor 0/0/264 flags hashpspool,selfmanaged_snaps,bulk stripe_width 0 application cephfs read_balance_score 1.13 pool 8 '.rgw.root' replicated size 3 min_size 2 crush_rule 6 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 20219 lfor 0/0/569 flags hashpspool stripe_width 0 application rgw read_balance_score 4.52 pool 9 'default.rgw.log' replicated size 3 min_size 2 crush_rule 6 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 20221 lfor 0/0/571 flags hashpspool stripe_width 0 application rgw read_balance_score 2.99 pool 10 'default.rgw.control' replicated size 3 min_size 2 crush_rule 6 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 20226 lfor 0/0/571 flags hashpspool stripe_width 0 application rgw read_balance_score 4.49 pool 11 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 6 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 20231 lfor 0/0/573 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw read_balance_score 2.99 pool 15 '.smb' replicated size 3 min_size 2 crush_rule 6 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 20233 lfor 0/0/6089 flags hashpspool stripe_width 0 application smb read_balance_score 4.50 pool 16 'cephfs.fs1.ec1' erasure profile isa_rsv_4p2_host_hdd size 6 min_size 5 crush_rule 4 object_hash rjenkins pg_num 128 pgp_num 128 autoscale_mode on last_change 38714 lfor 0/0/9461 flags hashpspool,ec_overwrites,selfmanaged_snaps,bulk stripe_width 16384 fast_read 1 application cephfs Where the relevant CRUSH rules are: * 4 - erasure, on hard drives only, host-level failure domain * 6 - replicated, on NVMe drives only, room-level failure domain -- MS

1 0

Re: squid 19.2.4 QE validation status
by Guillaume ABRIOUX 20 Apr '26

20 Apr '26

Hello Yuri, ceph-volume approved. Thanks! -- Guillaume Abrioux Software Engineer From: Yuri Weinstein via ceph-users <ceph-users(a)ceph.io> Date: Monday, 13 April 2026 at 21:25 To: dev <dev(a)ceph.io>, ceph-users <ceph-users(a)ceph.io> Subject: [EXTERNAL] [ceph-users] squid 19.2.4 QE validation status Details of this release are summarized here: https://urldefense.proofpoint.com/v2/url?u=https-3A__tracker.ceph.com_issue… Release Notes - TBD LRC upgrade - N/A Seeking approvals/reviews for: smoke: PASSED rados: Radek, Laura, Afreen, Travis rgw: Adam Emerson fs: Venky orch/ceph-volume: Redouane & Guillaume rbd: Ilya krbd: Ilya upgrade/squid-x: Laura, Patrick upgrade/reef-x: Laura, Patrick upgrade/telemetry-upgrade: Laura, Patrick upgrade/squid-p2p: Laura, Patrick upgrade-clients:client-upgrade-reef-squid-squid: PASSED upgrade-clients:client-upgrade-quincy-squid-squid: PASSED perf-basic: PASSED crimson-rados: N/A powercycle: PASSED TIA _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io Unless otherwise stated above: Compagnie IBM France Siège Social : 17, avenue de l'Europe, 92275 Bois-Colombes Cedex RCS Nanterre 552 118 465 Forme Sociale : S.A.S. Capital Social : 664 614 175,50 € SIRET : 552 118 465 03644 - Code NAF 6203Z

1 0

Re: ceph 19.2.3 health POOL_TOO_MANY_PGS / POOL_TOO_FEW_PGS ping-pong
by Anthony D'Atri 18 Apr '26

18 Apr '26

> Hi Anthony, > > On Thu, 16 Apr 2026, Anthony D'Atri via ceph-users wrote: > >> I might suggest either enabling the autoscaler or setting the mode to off. > > I confirm, setting the autoscaler mode from warn to off removes the health warning. Until now I had always used mode=warn as a last resort for the absentminded admin (i.e. cluster expansion / OSD addition). That's one of the reasons the PG autoscaler was implemented, it is very, very easy to neglect the need to recalculate, and computers should serve US after all. > What I miss from previews ceph versions is, that autoscale-status shows the suggested PG count in the column "NEW PG_NUM", like shown in the docs[1]. You might just periodically set them to warn, capture the info, and turn it back off. But I do have to ask, is there a specific reason you aren't using the autoscaler? It's much better than it used to be, and there are really great improvements in the pipe. > And regarding the ping-pong with TOO_MANY / TOO_FEW. I understand that BIAS=4.0 for the index pool should prevent oscillating suggestions? The BIAS value means "give this pool more PGs than the amount of data would otherwise indicate". You'll see it on RGW index and some other metadata pools, which benefit from more sharding than does bulk data. What you're thinking of is the per-pool threshold. Historically this has defaulted to 3.0 : the autoscaler won't make a change until the calculated value varies by more than a factor of 3 from the current setting. I think of this as hysteresis, but there's probably a better term. There's been some fairly recent work to do this more surgically, which I _think_ removes the per-pool threshold, but in any event, I believe that recent autoscaler releases are fairly immune to this flapping. > > Thanks, Sascha. > > [1] https://docs.ceph.com/en/latest/rados/operations/placement-groups/#viewing-… > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

April Ceph Tech Talk
by Anthony Middleton 17 Apr '26

17 Apr '26

The next Ceph Tech Talk is on Wednesday, April 22, at 9 am PDT / 16:00 UTC, and is titled “MAAS as a Backend: Provisioning Infrastructure for Teuthology Suites,” a technical deep dive into how MaaS is now integrated with Teuthology to provision and manage bare-metal infrastructure on demand. Invite is on the Ceph Community Calendar: https://t.ly/CephTechTalks If you want to hear a topic at an upcoming session, email me at amiddleton(a)linuxfoundation.org. Best, ------- Anthony Middleton Community Manager, Ceph Foundation amiddleton(a)linuxfoundation.org LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

Re: Tentacle, hybrid-OSD bug and making space for block.db
by Igor Fedotov 17 Apr '26

17 Apr '26

Hi Marek, the only way to do that is to migrate data from DB volume(s) to main ones using ceph-volume(which will remove DB ones as well) and reallocate/reattach them back. Thanks, Igor On 4/17/2026 5:11 PM, Marek Szuba via ceph-users wrote: > Hello again, > > Having just tried to add additional hard drives to our production Ceph > cluster (i.e. not the one currently experiencing problems with misplaced > PGs), I have been forcibly reminded of the "orch puts OSD block.db and > data on the same device" orchestrator bug in Squid and Tentacle. > > With the relevant ticket having only reached "pending backport" stage 2 > days ago, I figured I would just follow the workaround of assembling the > new OSDs by hand rather than wait for whichever Tentacle patch release > ends up incorporating the fix. There is, however, a snag - turns out the > orchestrator has already allocated all the space on the DB SSDs to > existing OSDs! > > Is it possible to manually shrink block.db LVs so that I can fit another > one of them in? Having had a cursory look at documentation, seems one > can use ceph-bluestore-tool to _extend_ OSD storage - but have yet to > see any mention of the possibility of shrinking it. > > Thanks in advance. >

1 0

Re: squid 19.2.4 QE validation status
by Laura Flores 17 Apr '26

17 Apr '26

The RADOS suite is approved: https://tracker.ceph.com/projects/rados/wiki/SQUID#v1924-httpstrackercephco… Upgrade suites are pending the inclusion of https://tracker.ceph.com/issues/76028, and will need to be rerun. Thanks, Laura Laura Flores Software Engineer; Ceph Storage 1 708 738 8804 Mobile She/Her/Hers IBM ________________________________ From: Laura Flores via dev <dev(a)ceph.io> Sent: Thursday, April 16, 2026 4:08 PM To: Yuri Weinstein <yweinste(a)redhat.com>; Travis Nielsen <tnielsen(a)redhat.com> Cc: dev <dev(a)ceph.io>; ceph-users <ceph-users(a)ceph.io> Subject: [EXTERNAL] RE: squid 19.2.4 QE validation status I reviewed the RADOS results. Will consult with Radek and update this thread by tomorrow. Thanks, Laura Laura Flores Software Engineer; Ceph Storage 1 708 738 8804 Mobile She/Her/Hers IBM From: Travis Nielsen via dev <dev@ ceph. io> Sent: I reviewed the RADOS results. Will consult with Radek and update this thread by tomorrow. Thanks, Laura Laura Flores Software Engineer; Ceph Storage 1 708 738 8804 Mobile She/Her/Hers IBM ________________________________ From: Travis Nielsen via dev <dev(a)ceph.io> Sent: Thursday, April 16, 2026 12:56 PM To: Yuri Weinstein <yweinste(a)redhat.com> Cc: dev <dev(a)ceph.io>; ceph-users <ceph-users(a)ceph.io> Subject: [EXTERNAL] Re: squid 19.2.4 QE validation status Approved for Rook, the daily CI against the squid devel tag is passing. Travis On Thu, Apr 16, 2026 at 9: 01 AM Yuri Weinstein via dev <dev@ ceph. io> wrote: We got only rbd, krbd approved. Still seeking approvals On Mon, Apr 13, 2026 at Approved for Rook, the daily CI against the squid devel tag is passing. Travis On Thu, Apr 16, 2026 at 9:01 AM Yuri Weinstein via dev <dev(a)ceph.io<mailto:dev@ceph.io>> wrote: We got only rbd, krbd approved. Still seeking approvals On Mon, Apr 13, 2026 at 12:24 PM Yuri Weinstein <yweinste(a)redhat.com<mailto:yweinste@redhat.com>> wrote: Details of this release are summarized here: https://tracker.ceph.com/issues/75029#note-5<https://urldefense.proofpoint.com/v2/url?u=https-3A__tracker.ceph.com_issue…> Release Notes - TBD LRC upgrade - N/A Seeking approvals/reviews for: smoke: PASSED rados: Radek, Laura, Afreen, Travis rgw: Adam Emerson fs: Venky orch/ceph-volume: Redouane & Guillaume rbd: Ilya krbd: Ilya upgrade/squid-x: Laura, Patrick upgrade/reef-x: Laura, Patrick upgrade/telemetry-upgrade: Laura, Patrick upgrade/squid-p2p: Laura, Patrick upgrade-clients:client-upgrade-reef-squid-squid: PASSED upgrade-clients:client-upgrade-quincy-squid-squid: PASSED perf-basic: PASSED crimson-rados: N/A powercycle: PASSED TIA

1 0