- ceph-users - lists.ceph.io

Re: Tentacle cluster: health OK, no recovery traffic, 24% objects misplaced
by Anthony D'Atri 17 Apr '26

17 Apr '26

What does `ceph osd tree` look like? How many hosts? `ceph osd crush rule dump` `ceph osd pool ls detail` I'm wondering if your CRUSH rule for at least one pool requires more failure domains than the cluster provides. Also, in past releases there have been issues with the mclock scheduler on HDD and especially EC pools, improved by reverting to wpq. I thought Tentacle improved this but have not tested.

1 0

Re: Tentacle cluster: health OK, no recovery traffic, 24% objects misplaced
by Malte Stroem 17 Apr '26

17 Apr '26

Thanks! What about your pools? Show us: ceph osd pool ls detail Do you use erasure coded pools? What about your crush rules? Did you set up something special inside? Best, Malte On 4/17/26 15:44, Marek Szuba via ceph-users wrote: > Many thank to those who have responded! Will comment on everything in > this message, for brevity. > > > On 2026-04-14 17:00, Eugen Block via ceph-users wrote: > >> First thing coming to mind is: ceph mgr fail >> >> If you already did that something else is going on. > > I have indeed already tried it, several times in fact. Seems to have no > effect. > > > On 2026-04-14 17:41, tobias tempel wrote: > >> you write "all the daemons which should be up and running ARE up and >> running. In particular, all the expected OSDs appear to be >> operational". Does that mean, that not all OSDs are running? > No, all OSDs are expected to be running at present so what I meant by > the above was that all were - indeed, still are - up. Should have made > it more clear, apologies. > > > On 2026-04-14 18:05, Malte Stroem wrote: > > > add storage to the cluster. > > Is this expected to help if the cluster is nowhere near full? According > to 'ceph osd df' even the most heavily utilised OSDs are only about 20% > full, and most of them are half that or less. > > > Please show us: > > > > ceph -s > > As of a few minutes ago, this shows: > > cluster: > id: (REDACTED) > health: HEALTH_OK > > services: > mon: 5 daemons, quorum cdev01,cdev02,cdev03,cdev06,cdev08(age 3d) > [leader: cdev01] > mgr: cdev04.lmasuw(active, since 2d), standbys: cdev01.dgatgl, > cdev08.nxaiqd > mds: 1/1 daemons up, 2 standby > osd: 88 osds: 88 up (since 2d), 88 in (since 9w); 1448 remapped pgs > rgw: 6 daemons active (3 hosts, 1 zones) > > data: > volumes: 1/1 healthy > pools: 13 pools, 2641 pgs > objects: 98.57M objects, 27 TiB > usage: 105 TiB used, 846 TiB / 951 TiB avail > pgs: 76576200/314890284 objects misplaced (24.318%) > 1448 active+clean+remapped > 1189 active+clean > 2 active+clean+scrubbing+deep > 2 active+clean+scrubbing > > io: > client: 986 KiB/s rd, 0 B/s wr, 120 op/s rd, 0 op/s wr > > > Nb. The number of misplaced objects HAS in fact decreased slightly (by > about 0.3%) since two days ago so something IS happening, if extremely > slowly. >

1 0

Tentacle, hybrid-OSD bug and making space for block.db
by Marek Szuba 17 Apr '26

17 Apr '26

Hello again, Having just tried to add additional hard drives to our production Ceph cluster (i.e. not the one currently experiencing problems with misplaced PGs), I have been forcibly reminded of the "orch puts OSD block.db and data on the same device" orchestrator bug in Squid and Tentacle. With the relevant ticket having only reached "pending backport" stage 2 days ago, I figured I would just follow the workaround of assembling the new OSDs by hand rather than wait for whichever Tentacle patch release ends up incorporating the fix. There is, however, a snag - turns out the orchestrator has already allocated all the space on the DB SSDs to existing OSDs! Is it possible to manually shrink block.db LVs so that I can fit another one of them in? Having had a cursory look at documentation, seems one can use ceph-bluestore-tool to _extend_ OSD storage - but have yet to see any mention of the possibility of shrinking it. Thanks in advance. -- MS

1 0

Re: Tentacle cluster: health OK, no recovery traffic, 24% objects misplaced
by Marek Szuba 17 Apr '26

17 Apr '26

Many thank to those who have responded! Will comment on everything in this message, for brevity. On 2026-04-14 17:00, Eugen Block via ceph-users wrote: > First thing coming to mind is: ceph mgr fail > > If you already did that something else is going on. I have indeed already tried it, several times in fact. Seems to have no effect. On 2026-04-14 17:41, tobias tempel wrote: > you write "all the daemons which should be up and running ARE up and > running. In particular, all the expected OSDs appear to be > operational". Does that mean, that not all OSDs are running? No, all OSDs are expected to be running at present so what I meant by the above was that all were - indeed, still are - up. Should have made it more clear, apologies. On 2026-04-14 18:05, Malte Stroem wrote: > add storage to the cluster. Is this expected to help if the cluster is nowhere near full? According to 'ceph osd df' even the most heavily utilised OSDs are only about 20% full, and most of them are half that or less. > Please show us: > > ceph -s As of a few minutes ago, this shows: cluster: id: (REDACTED) health: HEALTH_OK services: mon: 5 daemons, quorum cdev01,cdev02,cdev03,cdev06,cdev08(age 3d) [leader: cdev01] mgr: cdev04.lmasuw(active, since 2d), standbys: cdev01.dgatgl, cdev08.nxaiqd mds: 1/1 daemons up, 2 standby osd: 88 osds: 88 up (since 2d), 88 in (since 9w); 1448 remapped pgs rgw: 6 daemons active (3 hosts, 1 zones) data: volumes: 1/1 healthy pools: 13 pools, 2641 pgs objects: 98.57M objects, 27 TiB usage: 105 TiB used, 846 TiB / 951 TiB avail pgs: 76576200/314890284 objects misplaced (24.318%) 1448 active+clean+remapped 1189 active+clean 2 active+clean+scrubbing+deep 2 active+clean+scrubbing io: client: 986 KiB/s rd, 0 B/s wr, 120 op/s rd, 0 op/s wr Nb. The number of misplaced objects HAS in fact decreased slightly (by about 0.3%) since two days ago so something IS happening, if extremely slowly. -- MS

1 0

Re: ceph 19.2.3 health POOL_TOO_MANY_PGS / POOL_TOO_FEW_PGS ping-pong
by Sascha Lucas 17 Apr '26

17 Apr '26

Hi Anthony, On Thu, 16 Apr 2026, Anthony D'Atri via ceph-users wrote: > I might suggest either enabling the autoscaler or setting the mode to off. I confirm, setting the autoscaler mode from warn to off removes the health warning. Until now I had always used mode=warn as a last resort for the absentminded admin (i.e. cluster expansion / OSD addition). What I miss from previews ceph versions is, that autoscale-status shows the suggested PG count in the column "NEW PG_NUM", like shown in the docs[1]. And regarding the ping-pong with TOO_MANY / TOO_FEW. I understand that BIAS=4.0 for the index pool should prevent oscillating suggestions? Thanks, Sascha. [1] https://docs.ceph.com/en/latest/rados/operations/placement-groups/#viewing-…

1 0

Re: squid 19.2.4 QE validation status
by Laura Flores 16 Apr '26

16 Apr '26

I reviewed the RADOS results. Will consult with Radek and update this thread by tomorrow. Thanks, Laura Laura Flores Software Engineer; Ceph Storage 1 708 738 8804 Mobile She/Her/Hers IBM ________________________________ From: Travis Nielsen via dev <dev(a)ceph.io> Sent: Thursday, April 16, 2026 12:56 PM To: Yuri Weinstein <yweinste(a)redhat.com> Cc: dev <dev(a)ceph.io>; ceph-users <ceph-users(a)ceph.io> Subject: [EXTERNAL] Re: squid 19.2.4 QE validation status Approved for Rook, the daily CI against the squid devel tag is passing. Travis On Thu, Apr 16, 2026 at 9: 01 AM Yuri Weinstein via dev <dev@ ceph. io> wrote: We got only rbd, krbd approved. Still seeking approvals On Mon, Apr 13, 2026 at Approved for Rook, the daily CI against the squid devel tag is passing. Travis On Thu, Apr 16, 2026 at 9:01 AM Yuri Weinstein via dev <dev(a)ceph.io<mailto:dev@ceph.io>> wrote: We got only rbd, krbd approved. Still seeking approvals On Mon, Apr 13, 2026 at 12:24 PM Yuri Weinstein <yweinste(a)redhat.com<mailto:yweinste@redhat.com>> wrote: Details of this release are summarized here: https://tracker.ceph.com/issues/75029#note-5<https://urldefense.proofpoint.com/v2/url?u=https-3A__tracker.ceph.com_issue…> Release Notes - TBD LRC upgrade - N/A Seeking approvals/reviews for: smoke: PASSED rados: Radek, Laura, Afreen, Travis rgw: Adam Emerson fs: Venky orch/ceph-volume: Redouane & Guillaume rbd: Ilya krbd: Ilya upgrade/squid-x: Laura, Patrick upgrade/reef-x: Laura, Patrick upgrade/telemetry-upgrade: Laura, Patrick upgrade/squid-p2p: Laura, Patrick upgrade-clients:client-upgrade-reef-squid-squid: PASSED upgrade-clients:client-upgrade-quincy-squid-squid: PASSED perf-basic: PASSED crimson-rados: N/A powercycle: PASSED TIA

1 0

Re: squid 19.2.4 QE validation status
by Travis Nielsen 16 Apr '26

16 Apr '26

Approved for Rook, the daily CI against the squid devel tag is passing. Travis On Thu, Apr 16, 2026 at 9:01 AM Yuri Weinstein via dev <dev(a)ceph.io> wrote: > We got only rbd, krbd approved. > > Still seeking approvals > > On Mon, Apr 13, 2026 at 12:24 PM Yuri Weinstein <yweinste(a)redhat.com> > wrote: > >> Details of this release are summarized here: >> >> https://tracker.ceph.com/issues/75029#note-5 >> >> Release Notes - TBD >> LRC upgrade - N/A >> >> Seeking approvals/reviews for: >> >> smoke: PASSED >> >> rados: Radek, Laura, Afreen, Travis >> rgw: Adam Emerson >> fs: Venky >> orch/ceph-volume: Redouane & Guillaume >> rbd: Ilya >> krbd: Ilya >> >> upgrade/squid-x: Laura, Patrick >> upgrade/reef-x: Laura, Patrick >> upgrade/telemetry-upgrade: Laura, Patrick >> upgrade/squid-p2p: Laura, Patrick >> >> upgrade-clients:client-upgrade-reef-squid-squid: PASSED >> upgrade-clients:client-upgrade-quincy-squid-squid: PASSED >> >> perf-basic: PASSED >> crimson-rados: N/A >> powercycle: PASSED >> >> TIA >> >

1 0

Re: squid 19.2.4 QE validation status
by Adam Emerson 16 Apr '26

16 Apr '26

On 13/04/2026, Yuri Weinstein via dev wrote: > Details of this release are summarized here: > > https://tracker.ceph.com/issues/75029#note-5 > > Release Notes - TBD > LRC upgrade - N/A > > Seeking approvals/reviews for: > > smoke: PASSED > > rados: Radek, Laura, Afreen, Travis > rgw: Adam Emerson RGW Approved. Thank you!

1 0

Re: ceph 19.2.3 health POOL_TOO_MANY_PGS / POOL_TOO_FEW_PGS ping-pong
by Anthony D'Atri 16 Apr '26

16 Apr '26

Hmmm, I thought this was fixed by Squid. I've seen Pool wombats has 32 placement groups, should have 32 with Pacific. I might suggest either enabling the autoscaler or setting the mode to off. When enabling, setting mon_max_pg_per_osd 500 mon_target_pg_per_osd 300 will likely give you reasonable results. > On Apr 16, 2026, at 10:07 AM, Sascha Lucas via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi, > > having a small ceph-19.2.3 cluster for RGW workload. index-pool is on separate SSD class. PG autoscaler is enabled in warning mode only. > > Cluster health is: > > # ceph health detail > HEALTH_WARN 1 pools have too many placement groups > [WRN] POOL_TOO_MANY_PGS: 1 pools have too many placement groups > Pool default.rgw.buckets.index has 64 placement groups, should have 32 > > When merging PGs down to 32, the opposite is shown: > > # ceph health detail > HEALTH_WARN 1 pools have too few placement groups > [WRN] POOL_TOO_FEW_PGS: 1 pools have too few placement groups > Pool default.rgw.buckets.index has 32 placement groups, should have 64 > > It seems not the autoscaler who raises health warning here (NEW PG_NUM is empty): > > # ceph osd pool autoscale-status > POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO EFFECTIVE RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE BULK > .mgr 23812k 3.0 384.0G 0.0002 1.0 1 warn False > .rgw.root 20480 3.0 384.0G 0.0000 1.0 32 warn False > default.rgw.log 23905k 3.0 384.0G 0.0002 1.0 32 warn False > default.rgw.control 0 3.0 384.0G 0.0000 1.0 32 warn False > default.rgw.meta 575.2k 3.0 384.0G 0.0000 4.0 32 warn False > default.rgw.buckets.index 3017M 3.0 384.0G 0.0230 4.0 64 warn False > default.rgw.buckets.non-ec 0 3.0 384.0G 0.0000 1.0 32 warn False > default.rgw.buckets.data 10678G 3.0 374.2T 0.0836 1.0 1024 warn True > > Anyone knowing from where this health warning is coming and how to stop the ping-pong? > > Thanks, Sascha. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

ceph 19.2.3 health POOL_TOO_MANY_PGS / POOL_TOO_FEW_PGS ping-pong
by Sascha Lucas 16 Apr '26

16 Apr '26

Hi, having a small ceph-19.2.3 cluster for RGW workload. index-pool is on separate SSD class. PG autoscaler is enabled in warning mode only. Cluster health is: # ceph health detail HEALTH_WARN 1 pools have too many placement groups [WRN] POOL_TOO_MANY_PGS: 1 pools have too many placement groups Pool default.rgw.buckets.index has 64 placement groups, should have 32 When merging PGs down to 32, the opposite is shown: # ceph health detail HEALTH_WARN 1 pools have too few placement groups [WRN] POOL_TOO_FEW_PGS: 1 pools have too few placement groups Pool default.rgw.buckets.index has 32 placement groups, should have 64 It seems not the autoscaler who raises health warning here (NEW PG_NUM is empty): # ceph osd pool autoscale-status POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO EFFECTIVE RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE BULK .mgr 23812k 3.0 384.0G 0.0002 1.0 1 warn False .rgw.root 20480 3.0 384.0G 0.0000 1.0 32 warn False default.rgw.log 23905k 3.0 384.0G 0.0002 1.0 32 warn False default.rgw.control 0 3.0 384.0G 0.0000 1.0 32 warn False default.rgw.meta 575.2k 3.0 384.0G 0.0000 4.0 32 warn False default.rgw.buckets.index 3017M 3.0 384.0G 0.0230 4.0 64 warn False default.rgw.buckets.non-ec 0 3.0 384.0G 0.0000 1.0 32 warn False default.rgw.buckets.data 10678G 3.0 374.2T 0.0836 1.0 1024 warn True Anyone knowing from where this health warning is coming and how to stop the ping-pong? Thanks, Sascha.

1 0