- ceph-users - lists.ceph.io

Re: An issue during expanding an cluster - one osd getting close to nearfull
by Eugen Block 01 Jul '26

01 Jul '26

Hi, I would try to avoid adding a host with only few OSDs if other hosts are equipped differently. Although I haven't checked in a while I don't expect the behavior to have changed in recent versions. So here's my theory: If you add a new host with only one or two OSDs, Ceph can assign many more PGs than average to the first up OSD in order to comply with failure domain (probably host). Same goes for the last OSD during shut down. We had a related case four years ago when we tried to rebuild OSD nodes in a customer cluster. Since the customer cluster was still on default values for mon_max_pg_per_osd and osd_max_pg_per_osd_hard_ratio, the (too many) assigned PGs led to inactive PGs: > maybe_wait_for_max_pg withhold creation of pg 75.56s16: 750 >= 750 When more OSDs were created on that host (ceph-volume does that sequentially), the issue resolved eventually, except for some inactive PGs, failing the MGR resolved those as well. Since you have around 360 PGs per OSD in average, I assume you increased mon_max_pg_per_osd (default 250), maybe also osd_max_pg_per_osd_hard_ratio (default 3), you most likely dodged the "inactive PG" bullet. So my recommendation would be to rather add the hosts with all OSDs at the same time but with crush weight 0. That way you have time to inspect the outcome (for example with dedicated DB/WAL devices you can check if they were created according to the specs). Then reweight the OSDs to the correct value, some users do that gradually to prevent an overload. Note the recent discussion about osd heartbeats when adding multiple hosts at once: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/RHVPS247M4E… Regards, Eugen Zitat von Robert Lukan via ceph-users <ceph-users(a)ceph.io>: > Hi everyone, > > We are currently in the middle of migrating our infrastructure to a > Proxmox/Ceph stack and ran into a concerning edge case while expanding the > cluster. I am hoping to get some advice on the safest way to handle this > without risking a nearfull OSD. > > *The Topology & Situation:* > > *Existing:* 5 hosts (proxmox-01 to proxmox-05), each with 8x 7TB SSDs. > > *Expansion:* Added a new host (proxmox-08) with 2x 7TB SSDs (osd.40 and > osd.41). > > *Cluster Average:* ~360 PGs per OSD, ~60% utilization. > > *Ceph Version:* Squid 19.2.3 > > The last time we were expanding the cluster, we had issues when expanding > it with many OSDs at the same time(it was a memory problem on a host), so > we decided to be more careful and add 2 disks at the time. > > During the backfill phase, we noticed osd.40 catching significantly more > PGs than osd.41. We paused the migration by setting the nobackfill flag to > prevent a potential nearfull situation. > > *The Current State (ceph osd df):* > > osd.40: 423 PGs, 72.56% used, > > osd.41: 307 PGs, 51.87% used, > > We had 218 PGs sitting in active+remapped+backfill_wait. At that stage we > have paused the backfill. > > If we were to unset nobackfill, the incoming data for those 218 PGs would > risk pushing osd.40 past the 85% nearfull threshold. However, our > balancer refused to generate any optimization plans. It reports: > "optimize_result": > "Unable to find further optimization... or distribution is already perfect" > > We "understand" why this is happening: the balancer is evaluating only when > backfilling is done. However, it is completely ignoring the transient acting > state, which was heavily overloading osd.40 > > *Our Temporary Workaround:* To prevent the potential issue, we have > temporarily set the reweight of osd.40 to 0 (effectively draining it out) > so we can safely lift the nobackfill flag and let the cluster settle > without hitting capacity limits. > > *My Questions for the Community:* > > What is the standard best practice for handling this "small bucket" > transient imbalance when adding a small number of OSDs to a new host? Or is > this just a bad idea in general ? > > Is there a way to force the built-in upmap balancer to respect the > transient/acting capacity of an OSD so it doesn't allow a drive to go > towards 85% during a migration? I understand we were not there, but being > 12 % away was "close" enough. > > One of the ideas was to reweight the offending OSD, to remove a few PGs, so > it would not be so close to nearfull situation. But this approach is > tedious and requires "baby-sitting". > > I guess this might be a good candidate for upmap remap script, but we were > not "brave" enough to run in it a production environment the first time. > > Thanks, > > Robert > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Grafana dashboards warnings after upgrade to Squid 19.2.4
by Sake 01 Jul '26

01 Jul '26

Probably something not going correctly, because I just upgraded to 19.2.4 and Grafana is using the grafana image (in the past this was ceph-grafana) version 12.3.1. On 1 July 2026 11:13:02 CEST, Eugen Block via ceph-users <ceph-users(a)ceph.io> wrote: >Hi, > >I just upgraded our production cluster to 19.2.4 (from 19.2.3), there were no issues reported by our users, so all good. But I noticed that many graphs now show different warnings or strange labels: > >- "This panel requires Angular (deprecated)." on multiple graphs >- "PromQL info: metric might not be a counter, name does not end in _total/_sum/_count/_bucket: "ceph_pool_wr" (1:6)" also on multiple graphs for different metrics >- "OSD Capacity Utilization" just shows: "RangeError: invalid array length" >- The "Pool Overview" graph has column names like "Value #F", "Value #G" and so on. And the values within don't make sense either. > >I have uploaded screenshots: > >https://www.pasteboard.co/A34Zcy9cqHe5.png >https://www.pasteboard.co/F_vkpeBjGGWH.png >https://www.pasteboard.co/mkCtlBN3zS6v.png >https://www.pasteboard.co/T_ByD_gCpslA.png > >I compared the prometheus and grafana versions we have currently running with a fresh 19.2.4 bootstrapped cluster and the versions are identical: > >ceph orch ps --daemon-type grafana --format json | jq -r '.[].version' >10.4.0 > >ceph orch ps --daemon-type prometheus --format json | jq -r '.[].version' >2.51.0 > >Is there anything I'm missing? > >Thanks! >Eugen >_______________________________________________ >ceph-users mailing list -- ceph-users(a)ceph.io >To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Grafana dashboards warnings after upgrade to Squid 19.2.4
by Chris Palmer 01 Jul '26

01 Jul '26

From Tentacle 20.2.1: ceph orch ps --daemon-type grafana --format json | jq -r '.[].version' 12.3.1 ceph orch ps --daemon-type prometheus --format json | jq -r '.[].version' 3.6.0 I noticed a big improvement moving from squid to tentacle. A different set of things wrong, but a much smaller set! Chris On 01/07/2026 10:46, Daniel Brown via ceph-users wrote: > Not probably immediately related to your issue, but Grafana just release version 13.1 in June. That means that 12.4 becomes the earlier “supported” version. 11.x and 10.x are a couple years old and unsupported now. Less specially an issue here, as Ceph is using the OSS version, not the Enterprise version of Grafana, but might be time for the project to consider revving which Grafana image is used. > > > -- > Dan Brown > >> On Jul 1, 2026, at 05:13, Eugen Block via ceph-users<ceph-users(a)ceph.io> wrote: >> >> ceph orch ps --daemon-type grafana --format json | jq -r '.[].version' >> 10.4.0 >> >> ceph orch ps --daemon-type prometheus --format json | jq -r '.[].version' >> 2.51.0 >> > _______________________________________________ > ceph-users mailing list --ceph-users(a)ceph.io > To unsubscribe send an email toceph-users-leave(a)ceph.io

1 0

Re: Grafana dashboards warnings after upgrade to Squid 19.2.4
by Daniel Brown 01 Jul '26

01 Jul '26

Not probably immediately related to your issue, but Grafana just release version 13.1 in June. That means that 12.4 becomes the earlier “supported” version. 11.x and 10.x are a couple years old and unsupported now. Less specially an issue here, as Ceph is using the OSS version, not the Enterprise version of Grafana, but might be time for the project to consider revving which Grafana image is used. -- Dan Brown > On Jul 1, 2026, at 05:13, Eugen Block via ceph-users <ceph-users(a)ceph.io> wrote: > > ceph orch ps --daemon-type grafana --format json | jq -r '.[].version' > 10.4.0 > > ceph orch ps --daemon-type prometheus --format json | jq -r '.[].version' > 2.51.0 >

1 0

Grafana dashboards warnings after upgrade to Squid 19.2.4
by Eugen Block 01 Jul '26

01 Jul '26

Hi, I just upgraded our production cluster to 19.2.4 (from 19.2.3), there were no issues reported by our users, so all good. But I noticed that many graphs now show different warnings or strange labels: - "This panel requires Angular (deprecated)." on multiple graphs - "PromQL info: metric might not be a counter, name does not end in _total/_sum/_count/_bucket: "ceph_pool_wr" (1:6)" also on multiple graphs for different metrics - "OSD Capacity Utilization" just shows: "RangeError: invalid array length" - The "Pool Overview" graph has column names like "Value #F", "Value #G" and so on. And the values within don't make sense either. I have uploaded screenshots: https://www.pasteboard.co/A34Zcy9cqHe5.png https://www.pasteboard.co/F_vkpeBjGGWH.png https://www.pasteboard.co/mkCtlBN3zS6v.png https://www.pasteboard.co/T_ByD_gCpslA.png I compared the prometheus and grafana versions we have currently running with a fresh 19.2.4 bootstrapped cluster and the versions are identical: ceph orch ps --daemon-type grafana --format json | jq -r '.[].version' 10.4.0 ceph orch ps --daemon-type prometheus --format json | jq -r '.[].version' 2.51.0 Is there anything I'm missing? Thanks! Eugen

1 0

Re: squid 19.2.5 QE validation status
by Adam Emerson 30 Jun '26

30 Jun '26

On 29/06/2026, Yuri Weinstein via dev wrote: > Details of this release are summarized here: > > https://tracker.ceph.com/issues/76983#note-2 > > Release Notes - TBD > LRC upgrade - N/A > > Note: please reply to this email with comments/approvals and also add your > comments to the tracker above. > > Seeking approvals/reviews for: > > smoke: PASSED > > rados: Radek, Laura, Nitzan, Afreen, Travis > rgw: Adam Emerson RGW Approved. The one failure is known and due to a mistake in a backported test case.

1 0

An issue during expanding an cluster - one osd getting close to nearfull
by Robert Lukan 29 Jun '26

29 Jun '26

Hi everyone, We are currently in the middle of migrating our infrastructure to a Proxmox/Ceph stack and ran into a concerning edge case while expanding the cluster. I am hoping to get some advice on the safest way to handle this without risking a nearfull OSD. *The Topology & Situation:* *Existing:* 5 hosts (proxmox-01 to proxmox-05), each with 8x 7TB SSDs. *Expansion:* Added a new host (proxmox-08) with 2x 7TB SSDs (osd.40 and osd.41). *Cluster Average:* ~360 PGs per OSD, ~60% utilization. *Ceph Version:* Squid 19.2.3 The last time we were expanding the cluster, we had issues when expanding it with many OSDs at the same time(it was a memory problem on a host), so we decided to be more careful and add 2 disks at the time. During the backfill phase, we noticed osd.40 catching significantly more PGs than osd.41. We paused the migration by setting the nobackfill flag to prevent a potential nearfull situation. *The Current State (ceph osd df):* osd.40: 423 PGs, 72.56% used, osd.41: 307 PGs, 51.87% used, We had 218 PGs sitting in active+remapped+backfill_wait. At that stage we have paused the backfill. If we were to unset nobackfill, the incoming data for those 218 PGs would risk pushing osd.40 past the 85% nearfull threshold. However, our balancer refused to generate any optimization plans. It reports: "optimize_result": "Unable to find further optimization... or distribution is already perfect" We "understand" why this is happening: the balancer is evaluating only when backfilling is done. However, it is completely ignoring the transient acting state, which was heavily overloading osd.40 *Our Temporary Workaround:* To prevent the potential issue, we have temporarily set the reweight of osd.40 to 0 (effectively draining it out) so we can safely lift the nobackfill flag and let the cluster settle without hitting capacity limits. *My Questions for the Community:* What is the standard best practice for handling this "small bucket" transient imbalance when adding a small number of OSDs to a new host? Or is this just a bad idea in general ? Is there a way to force the built-in upmap balancer to respect the transient/acting capacity of an OSD so it doesn't allow a drive to go towards 85% during a migration? I understand we were not there, but being 12 % away was "close" enough. One of the ideas was to reweight the offending OSD, to remove a few PGs, so it would not be so close to nearfull situation. But this approach is tedious and requires "baby-sitting". I guess this might be a good candidate for upmap remap script, but we were not "brave" enough to run in it a production environment the first time. Thanks, Robert

1 0

Re: squid 19.2.5 QE validation status
by Travis Nielsen 29 Jun '26

29 Jun '26

Rook tests against squid devel is passing, so no concerns from my perspective. Travis On Mon, Jun 29, 2026 at 8:29 AM Yuri Weinstein via ceph-users < ceph-users(a)ceph.io> wrote: > Details of this release are summarized here: > > https://tracker.ceph.com/issues/76983#note-2 > > Release Notes - TBD > LRC upgrade - N/A > > Note: please reply to this email with comments/approvals and also add your > comments to the tracker above. > > Seeking approvals/reviews for: > > smoke: PASSED > > rados: Radek, Laura, Nitzan, Afreen, Travis > rgw: Adam Emerson > fs: Venky > orch/ceph-volume: Redouane & Guillaume > rbd: Ilya > krbd: Ilya > > upgrade: Laura, Patrick (see https://tracker.ceph.com/issues/77569, > https://github.com/ceph/ceph/pull/69791) > > upgrade-clients:client-upgrade-reef-squid-squid: PASSED > upgrade-clients:client-upgrade-quincy-squid-squid: PASSED > > perf-basic: PASSED > crimson-rados: N/A > powercycle: PASSED > > TIA > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

Re: "cephadm shell" container missing permissions?
by mailing-lists 29 Jun '26

29 Jun '26

Thanks, that worked flawlessly! On 29.06.26 18:17, Eugen Block via ceph-users wrote: > You need to specify the daemon: > > cephadm shell --name osd.X > > Then the device should be visible within the shell. > And please don’t use ceph-volume on an orchestrated cluster outside of > the shell. This will most likely cause issues. > > Zitat von mailing-lists via ceph-users <ceph-users(a)ceph.io>: > >> Hey Cephers, >> >> The big picture is, that I am trying to migrate a DB to a different >> DB-Device. I used to do this many times on Ubuntu 22.04 and >> Reef. With Ubuntu 24.04 and Squid however, I am facing a "permission >> denied" scenario, that I do not understand. It seems like the >> "cephadm shell" container is missing permissions to interact with >> /dev/ceph-* lvs. >> >> Our setup is containerized with Docker and deployed with cephadm. >> >> >> From within "cephadm shell": >> >> "ls -alh >> /dev/ceph-0bafc014-2bc6-4ecf-96a2-80b2dd414ad2/osd-block-91ae2955-577a-4722-8548-5e78372f065 >> >> ls: cannot access >> '/dev/ceph-0bafc014-2bc6-4ecf-96a2-80b2dd414ad2/osd-block-91ae2955-577a-4722-8548-5e78372f0650': >> No such file or directory" >> >> From root on the same host: >> "ls -alh >> /dev/ceph-0bafc014-2bc6-4ecf-96a2-80b2dd414ad2/osd-block-91ae2955-577a-4722-8548-5e78372f0650 >> lrwxrwxrwx 1 root root 7 Jun 25 17:12 >> /dev/ceph-0bafc014-2bc6-4ecf-96a2-80b2dd414ad2/osd-block-91ae2955-577a-4722-8548-5e78372f0650 >> -> ../dm-3" >> >> The container just cannot "see" the lv-device. >> >> Something like >> >> "ceph-volume lvm activate 1 91ae2955-577a-4722-8548-5e78372f0650" >> >> will then just fail with: >> ``` >> Running command: /usr/bin/ceph-authtool --gen-print-key >> Running command: /usr/bin/ceph-authtool --gen-print-key >> Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 >> Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph >> prime-osd-dir --dev >> /dev/ceph-0bafc014-2bc6-4ecf-96a2-80b2dd414ad2/osd-block-91ae2955-577a-4722-8548-5e78372f0650 >> --path /var/lib/ceph/osd/ceph-1 --no-mon-config >> stderr: failed to read label for >> /dev/ceph-0bafc014-2bc6-4ecf-96a2-80b2dd414ad2/osd-block-91ae2955-577a-4722-8548-5e78372f0650: >> (1) *Operation not permitted* >> stderr: 2026-06-29T15:57:48.848+0000 7e8959bbca80 -1 >> bdev(0x5d2610965800 >> /dev/ceph-0bafc014-2bc6-4ecf-96a2-80b2dd414ad2/osd-block-91ae2955-577a-4722-8548-5e78372f0650) >> open stat got: (1) Operation not permitted >> --> RuntimeError: command returned non-zero exit status: 1 >> >> ``` >> >> Has anyone had this problem and solved it? >> >> Is the recommended way to install ceph utilities like ceph-volume on >> the host instead using the toolkit from the cephadm shell? >> >> >> Best >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: "cephadm shell" container missing permissions?
by happy.otter659＠passmail.net 29 Jun '26

29 Jun '26

Hi, I did have this issue but not while migrating a db. Try removing the bluestore tool and any other ceph-* packages that you installed manually, except for cephadm. Then maybe a reatart might be necessary for that node or reatarting the docker daemon. Hope this helps Cheers Best regards - mit freundlichen Gruessen, Christian Frech -------- Original Message -------- On Monday, 06/29/26 at 18:08 mailing-lists via ceph-users <ceph-users(a)ceph.io> wrote: Hey Cephers, The big picture is, that I am trying to migrate a DB to a different DB-Device. I used to do this many times on Ubuntu 22.04 and Reef. With Ubuntu 24.04 and Squid however, I am facing a "permission denied" scenario, that I do not understand. It seems like the "cephadm shell" container is missing permissions to interact with /dev/ceph-* lvs. Our setup is containerized with Docker and deployed with cephadm. From within "cephadm shell": "ls -alh /dev/ceph-0bafc014-2bc6-4ecf-96a2-80b2dd414ad2/osd-block-91ae2955-577a-4722-8548-5e78372f065 0 ls: cannot access '/dev/ceph-0bafc014-2bc6-4ecf-96a2-80b2dd414ad2/osd-block-91ae2955-577a-4722-8548-5e78372f0650': No such file or directory" From root on the same host: "ls -alh /dev/ceph-0bafc014-2bc6-4ecf-96a2-80b2dd414ad2/osd-block-91ae2955-577a-4722-8548-5e78372f0650 lrwxrwxrwx 1 root root 7 Jun 25 17:12 /dev/ceph-0bafc014-2bc6-4ecf-96a2-80b2dd414ad2/osd-block-91ae2955-577a-4722-8548-5e78372f0650 -> ../dm-3" The container just cannot "see" the lv-device. Something like "ceph-volume lvm activate 1 91ae2955-577a-4722-8548-5e78372f0650" will then just fail with: ``` Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-0bafc014-2bc6-4ecf-96a2-80b2dd414ad2/osd-block-91ae2955-577a-4722-8548-5e78372f0650 --path /var/lib/ceph/osd/ceph-1 --no-mon-config stderr: failed to read label for /dev/ceph-0bafc014-2bc6-4ecf-96a2-80b2dd414ad2/osd-block-91ae2955-577a-4722-8548-5e78372f0650: (1) *Operation not permitted* stderr: 2026-06-29T15:57:48.848+0000 7e8959bbca80 -1 bdev(0x5d2610965800 /dev/ceph-0bafc014-2bc6-4ecf-96a2-80b2dd414ad2/osd-block-91ae2955-577a-4722-8548-5e78372f0650) open stat got: (1) Operation not permitted --> RuntimeError: command returned non-zero exit status: 1 ``` Has anyone had this problem and solved it? Is the recommended way to install ceph utilities like ceph-volume on the host instead using the toolkit from the cephadm shell? Best _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0