January 2025 - ceph-users

who build RPM package
by Tony Liu 09 Jan '25

09 Jan '25

Hi, I wonder which team is building Ceph RPM packages for CentOS Stream 9? I see Reef RPM packages in [1] and [2]. For example, ceph-18.2.4-0.el9.x86_64.rpm in [1] while ceph-18.2.4-1.el9.x86_64.rpm and -2 in [2]. Are those packages built by the same or different teams? What's the difference, -1 and -2 are newer than -0? I see Ceph container image is built with packages from [1]. If I install Ceph on CentOS Stream 9, which repo should I use? [1] http://download.ceph.com/rpm-18.2.4/el9/x86_64/ [2] https://mirror.stream.centos.org/SIGs/9-stream/storage/x86_64/ceph-reef/ Thanks! Tony

2 1

OSDs won't come back after upgrade
by Jorge Garcia 09 Jan '25

09 Jan '25

Hello, I'm going down the long and winding road of upgrading our ceph clusters from mimic to the latest version. This has involved slowly going up one release at a time. I'm now going from octopus to pacific, which also involves upgrading the OS on the host systems from Centos 7 to Rocky 9. I first upgraded the monitors and managers, and those upgraded with no problems. Now I'm upgrading the OSD servers, and I ran into some issues that caused the first system to be down for a couple of days. I finally got it back up, and got all the OSDs ready to come back online, but whenever I try to bring the OSDs back up, they start running for a bit, and it looks like the cluster is recovering and catching up, but then the OSDs all go down again. The logs show some messages like: received signal: Interrupt from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 osd.10 254568 *** Got signal Interrupt *** osd.10 254568 *** Immediate shutdown (osd_fast_shutdown=true) *** osd.10 254568 prepare_to_stop starting shutdown I found this thread: https://www.spinics.net/lists/ceph-users/msg75628.html which seems to be something similar, and they claim that the cluster needs to be restarted many times in order for the OSDs to catch up to the current epoch. I have restarted the OSDs many times, and now it's gotten to a spot where there doesn't seem to be any progress. My questions are: Is this the right solution? Is there a way of seeing if some progress is happening with the OSDs? Is there something else I should be trying? Thanks for any help! Jorge

2 1

Protection of WAL during spillover on implicitly colocated db/wal devices
by Wesley Dillingham 09 Jan '25

09 Jan '25

I have a question for bluestore experts about the ability for an OSD to continue to make use of the WAL during spillover events when the WAL and DB were implicitly collocated as described here: https://docs.ceph.com/en/reef/rados/configuration/bluestore-config-ref/#dev… " whenever a DB device is specified but an explicit WAL device is not, the WAL will be implicitly colocated with the DB on the faster device." During DB spillover (happen because fast device's LV is out of space) does the WAL have some guaranteed allocation of space in the LV / assurance that it can use a fixed amount of space when in contention with the db or does the bluestore-OSD no longer have the ability to use the WAL during spillover? I understand an admin could implement some workarounds to logically isolate and protect a wal and db on the same physical media but I am curious if such a mechanism exists already in the bluestore OSD itself. Thanks for the clarification. Respectfully, *Wes Dillingham* LinkedIn <http://www.linkedin.com/in/wesleydillingham> wes(a)wesdillingham.com

4 3

Experimental upgrade of a Cephadm-managed Squid cluster to Ubuntu Noble (walk-through and RFC)
by Florian Haas 09 Jan '25

09 Jan '25

Hi everyone, as part of keeping our Ceph course updated, we recently went through the *experimental* process of upgrading a Cephadm-managed cluster from Ubuntu Jammy to Noble. Note that at this point there are no community-built Ceph packages that are available for Noble, though there *are* packages available directly from the Ubuntu mirrors. We figured that this shouldn't make too big of a difference, considering the Ceph deployment is all containerised anyway. We did eventually manage to successfully upgrade the cluster, with minimal disruption. That said, DO NOT consider the rest of this message to be an instruction for how to do this in production. Rather, it's meant to highlight some quirks that — I think — still need ironing out. Again: for the time being, DO NOT RUN THIS IN PRODUCTION, please and thank you. :) What's included here is a walkthrough of the process, and I would ask for feedback on this walkthrough. Specifically, if someone notices an error or omission in this process, please point them out. Each one of these steps needs to be repeated for every node in the cluster. It assumes a working Cephadm-managed cluster on Ubuntu Jammy with Podman, running Ceph Squid. All commands should be run as root. == 0. If the host you're about to upgrade is a MON, ensure that your orchestrator placement policy allows for the mon service to move to a different host. (This is to ensure that throughout your OS upgrade, you always have 3 MONs available.) 1. Disable orchestrator scheduling for the affected node: "ceph orch host label add <host> _no_schedule". 2. Wait for services (except OSDs) to be redeployed. 3. Check with "ceph orch host ok-to-stop <host>" to see whether the host reports "presumed safe to stop". (May require failing a mgr instance.) 4. Set the noout flag for that host: "ceph osd set-group noout <host>". 5. Check with "ceph health detail" to see if it says "host <host> has flags noout". 6. Shell into the host. 7. Make sure the host is fully updated on Jammy: "apt update && apt upgrade -y". Reboot the host if necessary. 8. Stop all remaining Ceph services (at this point, this should only be OSDs) with "systemctl stop ceph.target". 9. If your host has the cephadm package installed from a Ceph community repo, manually create the "/var/lib/cephadm" directory. This is required by the Ubuntu cephadm package, which during the OS upgrade will be cross-upgraded from a different package source. Failure to create this directory will cause this package upgrade to fail. I presume that this issue will go away whenever the Ceph community repos start including packages for Noble. 10. Run "do-release-upgrade" confirming all steps *except* the automated reboot at the end (opt out of that). 11. Remove the MongoDB_Compass profile from the /etc/apparmor.d directory. (If left in place, it will cause CEPHADM_REFRESH_FAILED errors.) 12. Remove any legacy ceph-osd packages, if installed on the host: "apt remove ceph-osd". (Failure to do so will keep your OSD containers from starting.) 13. Reboot the host. 14. Re-enable orchestrator scheduling with "ceph orch host label rm <host> _no_schedule". 15. Unset the noout flag for the host: "ceph osd unset-group noout <host>". 16. Check overall cluster health with "ceph -s" and "ceph health detail" before proceeding to the next node. == Once carried out for each Cephadm-managed Ubuntu node, you should have a reasonable expection of having a working Ceph/Noble test cluster. Again: do not run this in production, however, feedback on the above steps is much appreciated. Please let us know your thoughts. Cheers, Florian

3 7

Random ephemeral pinning, what happens to sub-tree under pin root dir
by Frank Schilder 09 Jan '25

09 Jan '25

Dear all, I have a question about random ephemeral pinning that I can't find an answer in the docs to. Question first and some background later. Docs checked for any version from octopus up to latest. Our version for applying random ephemeral pinning is pacific. What I would like to configure on a subtree is this: Enable random ephemeral pinning at the root of the tree, say, /cephfs/root: setfattr -n ceph.dir.pin.random -v 0.0001 /cephfs/root Will this have the following effects: A) The root of the tree /cephfs/root is ephemerally pinned to a rank according to a consistent hash of its inode number. B) Any descendant sub-directory may be ephemerally pinned 0.01 percent of the time to a rank according to a consistent hash of its inode number. The important difference to the docs is point A. I don't want to have *any* subdir under the root /cephfs/root *not pinned* to an MDS. The docs only talk about descendant sub-dirs, but the root is here important too because if it is not pinned it will create a large number of unpinned dirfrags that float around with expensive exportdir operations that pinning is there to avoid in the first place. My questions are: 1) What does random ephemeral pinning do to the sub-tree root? Is it pinned or not? 2) If it doesn't pin the root, does this work as intended or will it pin everything to rank 1: setfattr -n ceph.dir.pin.random -v 0.001 /cephfs/root setfattr -n ceph.dir.pin -v 1 /cephfs/root Background: We use cephfs as a home file system for an HPC cluster and are in exactly in the situation of the example for distributed ephemeral pinning (https://docs.ceph.com/en/latest/cephfs/multimds/?highlight=ephemeral+pin#se…) *except* that the home-dirs of users differ dramatically in size. We were not pinning at first and this lead to our MDSes go crazy due to the load balancer moving dirfrags around all the time. This "load balancing" was itself responsible for 90-95% (!!!) of the total MDS load. After moving to octopus we simulated distributed ephemeral pinning with a cron job that assigned home-dirs in a round robin fashion to MDS ranks that had the least pins. This immediately calmed down our entire MDS cluster (8 active and 4 stand-by) and user experience improved dramatically. MDS load dropped from 125-150% (idle load!!) to about 20-25% per MDS and memory usage stabilized as well. The easy way forward would be to replace our manual distribution with distributed ephemeral pinning of /home (in octopus this was experimental, after our recent upgrade to pacific we can use the built-in distribution). However, as stated above, the size of home-dirs differs to a degree that chunking up the file system into equally-sized sub-dir trees would be better than distributing entire home dir trees over ranks. Users with very large sub-trees might get spread out over more than one rank. This is what random ephemeral pinning seems to be there for and I would like to chunk our entire filesystem up into sub-trees of size 10000-100000 directory fragments and distribute these over the MDSes. However, this only works if the root and with it the first sub-tree is also pinned. Note that this is not a problem with distributed ephemeral pinning, because this policy pins *all* *immediate* children of the pin root and, therefore, does not create free-floating directory fragments. I would be grateful if someone could shed light on the question whether or not the pin root of random ephemeral pinning is itself pinned or not. Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 2

cephadm rollout behavior and post adoption issues
by Nima AbolhassanBeigi 08 Jan '25

08 Jan '25

Hi everybody, I'm fairly new to cephadm. I'm trying to get some hands-on experience. I have a test cluster consisting of: 3 Monitor/Manager nodes. 6 OSD nodes 3 RGW nodes Pacific version containerized deployment using: quay.io/ceph/daemon:v6.0.11-stable-6.0-pacific-centos-stream8 I've adopted this cluster using the cephadm adopt playbook in the ceph-ansible project. infrastructure-playbooks/cephadm-adopt.yml I set the default container_image to a certain image and I checked the configuration database and it was set to the correct image. The adoption process for monitor, manager, and RGW daemons (redeployment in the case of RGWs) used the value for container_image, but the OSDs used the old image they were using before they were adopted. container_image value: quay.io/ceph/ceph:v16.2.13 I was wondering what could be the reason for the image not changing for OSDs. After the adoption, I wanted to mount the client.admin.keyring to monitor daemons. I used the extra_container_args service specification and added the volume. After I applied the mon service spec file to the cluster using ceph orch apply -i <filename>, I witnessed something odd. I read that cephadm has a rollout-like behavior for redeploying daemons. But the delay between the redeploys was 4 seconds and there was a period when all the monitor daemons were in the starting state. Is there any mechanism to increase the delay between the redeployments? Is there a way to enforce the redeployments to only 1 daemon? I've provided some logs, at the end of the message, that I think can be helpful. Thanks in advance. Regards Nima AbolhassanBeigi NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID mon.stg-pacific-mon1 stg-pacific-mon1 starting - - - 6144M <unknown> <unknow n> mon.stg-pacific-mon2 stg-pacific-mon2 starting - - - 6144M <unknown> <unknow n> mon.stg-pacific-mon3 stg-pacific-mon3 starting - - - 6144M <unknown> <unknon w> Every 1.0s: ceph orch ps --service_name=mon --refresh stg-pacific-mon1: Sun Dec 29 15:52:15 2024 NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID mon.stg-pacific-mon1 stg-pacific-mon1 starting - - - 6144M <unknown> <unknow n> mon.stg-pacific-mon2 stg-pacific-mon2 starting - - - 6144M <unknown> <unknow n> mon.stg-pacific-mon3 stg-pacific-mon3 starting - - - 6144M <unknown> <unknow n> Every 1.0s: ceph orch ps --service_name=mon --refresh stg-pacific-mon1: Sun Dec 29 15:52:41 2024 NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mon.stg-pacific-mon1 stg-pacific-mon1 running (37s) 2s ago 6d 30.2M 6144M 16.2.13 9e29efbb67d5 367a7f3bac31 mon.stg-pacific-mon2 stg-pacific-mon2 running (33s) 2s ago 11d 30.5M 6144M 16.2.13 9e29efbb67d5 43828e2453da mon.stg-pacific-mon3 stg-pacific-mon3 running (29s) 2s ago 11d 24.8M 6144M 16.2.13 9e29efbb67d5 d698a809370f 2024-12-29T12:18:16.974710+0000 mgr.stg-pacific-mon1 (mgr.173193) 322363 : cephadm [INF] Saving service mon spec with placement count:7;label:mons 2024-12-29T12:19:44.140240+0000 mgr.stg-pacific-mon1 (mgr.173193) 322545 : cephadm [INF] Saving service mon spec with placement count:3;label:mons 2024-12-29T12:19:47.497916+0000 mgr.stg-pacific-mon1 (mgr.173193) 322555 : cephadm [INF] Redeploying mon.stg-pacific-mon1, (container cli args changed) . . . 2024-12-29T12:19:47.507683+0000 mgr.stg-pacific-mon1 (mgr.173193) 322556 : cephadm [INF] Deploying daemon mon.stg-pacific-mon1 on stg-pacific-mon1 2024-12-29T12:19:50.970499+0000 mgr.stg-pacific-mon1 (mgr.173193) 322562 : cephadm [INF] Redeploying mon.stg-pacific-mon2, (container cli args changed) . . . 2024-12-29T12:19:50.982244+0000 mgr.stg-pacific-mon1 (mgr.173193) 322563 : cephadm [INF] Deploying daemon mon.stg-pacific-mon2 on stg-pacific-mon2 2024-12-29T12:19:55.774118+0000 mgr.stg-pacific-mon1 (mgr.173193) 322573 : cephadm [INF] Redeploying mon.stg-pacific-mon3, (container cli args changed) . . . 2024-12-29T12:19:55.782550+0000 mgr.stg-pacific-mon1 (mgr.173193) 322574 : cephadm [INF] Deploying daemon mon.stg-pacific-mon3 on stg-pacific-mon3 2024-12-29T12:21:56.620507+0000 mgr.stg-pacific-mon1 (mgr.173193) 322830 : cephadm [INF] Saving service mon spec with placement count:3;label:mons 2024-12-29T12:22:02.198770+0000 mgr.stg-pacific-mon1 (mgr.173193) 322844 : cephadm [INF] Redeploying mon.stg-pacific-mon1, (container cli args changed) . . . 2024-12-29T12:22:02.210484+0000 mgr.stg-pacific-mon1 (mgr.173193) 322845 : cephadm [INF] Deploying daemon mon.stg-pacific-mon1 on stg-pacific-mon1 2024-12-29T12:22:05.654926+0000 mgr.stg-pacific-mon1 (mgr.173193) 322851 : cephadm [INF] Redeploying mon.stg-pacific-mon2, (container cli args changed) . . . 2024-12-29T12:22:05.667782+0000 mgr.stg-pacific-mon1 (mgr.173193) 322852 : cephadm [INF] Deploying daemon mon.stg-pacific-mon2 on stg-pacific-mon2 2024-12-29T12:22:10.332756+0000 mgr.stg-pacific-mon1 (mgr.173193) 322862 : cephadm [INF] Redeploying mon.stg-pacific-mon3, (container cli args changed) . . . 2024-12-29T12:22:10.344298+0000 mgr.stg-pacific-mon1 (mgr.173193) 322863 : cephadm [INF] Deploying daemon mon.stg-pacific-mon3 on stg-pacific-mon3

1 0

Cephfs path based restricition without cephx
by Rok Jaklič 08 Jan '25

08 Jan '25

Hi, is it possible somehow to restrict client in cephfs to subdirectory without cephx enabled? We do not have any auth requirements enabled in ceph. auth cluster required = none auth service required = none auth client required = none Kind regards, Rok

2 2

check Nova keyring file
by Michel Niyoyita 07 Jan '25

07 Jan '25

Hello Team, I am integrating Openstack 2023.2 with ceph cluster reef running on ubuntu 22.05, I have configured openstack global as follow: ceph_nova_keyring: "ceph.client.nova.keyring" ceph_nova_user: "nova" ceph_nova_pool_name: "vms" I have copied nova keyring to from /etc/ceph to /etc/kolla/config/nova ; /etc/kolla/config/nova$ ls -l total 16 -rw-r--r-- 1 stack stack 64 Jan 7 13:12 ceph.client.cinder.keyring -rw-r--r-- 1 stack stack 62 Jan 7 15:11 ceph.client.nova.keyring -rw-r--r-- 1 stack stack 283 Jan 7 14:51 ceph.conf -rw-rw-r-- 1 stack stack 182 Jan 7 15:14 nova.conf (kolla-ansible) stack@open-deployment:/etc/kolla/config/nova$ I have created nova user in ceph : sudo ceph auth get-or-create client.nova mon 'profile rbd' osd 'profile rbd pool=vms, profile rbd pool=images' mgr 'profile rbd pool=vms' -o /etc/ceph/ceph.client.nova.keyring But while deploying I am facing the following issue : TASK [nova-cell : Check nova keyring file] ******************************************************************************************** fatal: [compute1 -> localhost]: FAILED! => {"msg": "No file was found when using first_found."} fatal: [compute2 -> localhost]: FAILED! => {"msg": "No file was found when using first_found."} fatal: [compute3 -> localhost]: FAILED! => {"msg": "No file was found when using first_found."} Kindly help to solve the issue , it takes long time without solution. previously was not occured on 2023.1 Best Regards Michel

1 0

How to configure prometheus password in ceph dashboard.
by s.dhivagar.cse＠gmail.com 07 Jan '25

07 Jan '25

We have configured basic authentication in Prometheus, so how to set username and password in ceph dashboard.

3 2

Re: Many misplaced PG's, full OSD's and a good amount of manual intervention to keep my Ceph cluster alive.
by Bruno Gomes Pessanha 06 Jan '25

06 Jan '25

+ceph-users On Mon, 6 Jan 2025 at 15:55, Bruno Gomes Pessanha <bruno.pessanha(a)gmail.com> wrote: > Old-style legacy override reweights don’t mesh well with the balancer. >> Best to leave them at 1.00. >> > Makes sense. I'll leave it at 1.0 and won't touch it again. > > Please send `ceph osd crush rule dump `. And `ceph osd dump | grep pool` > > > # ceph osd crush rule dump > [ > { > "rule_id": 0, > "rule_name": "replicated_rule", > "type": 1, > "steps": [ > { > "op": "take", > "item": -1, > "item_name": "default" > }, > { > "op": "chooseleaf_firstn", > "num": 0, > "type": "host" > }, > { > "op": "emit" > } > ] > }, > { > "rule_id": 1, > "rule_name": "cephfs.cephfs01.data", > "type": 3, > "steps": [ > { > "op": "set_chooseleaf_tries", > "num": 5 > }, > { > "op": "set_choose_tries", > "num": 100 > }, > { > "op": "take", > "item": -2, > "item_name": "default~ssd" > }, > { > "op": "choose_indep", > "num": 0, > "type": "osd" > }, > { > "op": "emit" > } > ] > } > ] > > # ceph osd dump | grep pool > pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash > rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 17721 flags > hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr > read_balance_score 75.00 > pool 2 '.nfs' replicated size 3 min_size 2 crush_rule 0 object_hash > rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 17721 lfor > 0/0/110 flags hashpspool stripe_width 0 application nfs read_balance_score > 7.50 > pool 3 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash > rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 17721 lfor > 0/0/125 flags hashpspool stripe_width 0 application rgw read_balance_score > 4.99 > pool 4 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 > object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change > 17721 lfor 0/0/125 flags hashpspool stripe_width 0 application rgw > read_balance_score 4.99 > pool 5 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 > object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change > 17721 lfor 0/0/126 flags hashpspool stripe_width 0 application rgw > read_balance_score 4.99 > pool 6 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 > object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change > 17721 lfor 0/0/128 flags hashpspool stripe_width 0 pg_autoscale_bias 4 > application rgw read_balance_score 7.53 > pool 7 'default.rgw.buckets.index' replicated size 3 min_size 2 crush_rule > 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change > 17721 lfor 0/0/222 flags hashpspool stripe_width 0 pg_autoscale_bias 4 > application rgw read_balance_score 4.97 > pool 10 'default.rgw.buckets.data' replicated size 3 min_size 2 crush_rule > 0 object_hash rjenkins pg_num 1024 pgp_num 1024 autoscale_mode on > last_change 17721 lfor 0/0/1044 flags hashpspool,bulk stripe_width 0 > pg_num_max 2048 application rgw read_balance_score 1.80 > pool 11 'default.rgw.buckets.non-ec' replicated size 3 min_size 2 > crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on > last_change 17721 lfor 0/0/319 flags hashpspool stripe_width 0 application > rgw read_balance_score 5.01 > pool 12 'cephfs.cephfs01.data' erasure profile 8k2m size 10 min_size 9 > crush_rule 1 object_hash rjenkins pg_num 144 pgp_num 16 pg_num_target 1024 > pgp_num_target 1024 autoscale_mode warn last_change 17721 lfor 0/0/16936 > flags hashpspool,ec_overwrites,selfmanaged_snaps,bulk stripe_width 32768 > pg_num_max 1024 application cephfs > pool 13 'cephfs.cephfs01.metadata' replicated size 4 min_size 2 crush_rule > 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change > 17721 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 > recovery_priority 5 application cephfs read_balance_score 80.00 > > On Sun, 5 Jan 2025 at 15:12, Anthony D'Atri <anthony.datri(a)gmail.com> > wrote: > >> >> >> What reweighs have been set for the top OSDs (ceph osd df tree)? >> >> >> > Right now they are all at 1.0. I had to lower them to something close to >> > 0.2 in order to free up space but I changed them back to 1.0. Should I >> > lower them while the backfill is happening? >> >> Old-style legacy override reweights don’t mesh well with the balancer. >> Best to leave them at 1.00. >> >> 0.2 is pretty extreme, back in the day I rarely went below 0.8. >> >> >> ``` >> >> "optimize_result": "Too many objects (0.355160 > 0.050000) are >> misplaced; >> >> try again late >> >> ``` >> >> That should clear. The balancer doesn’t want to stir up trouble if the >> cluster already has a bunch of backfill / recovery going on. Patience! >> >> >> default.rgw.buckets.data 10 1024 197 TiB 133.75M 592 TiB 93.69 >> >> 13 TiB >> >> default.rgw.buckets.non-ec 11 32 78 MiB 1.43M 17 GiB >> >> That’s odd that the data pool is that full but the others aren’t. >> >> Please send `ceph osd crush rule dump `. And `ceph osd dump | grep pool` >> >> >> >> >> >> I also tried changing the following but it does not seem to persist: >> >> Could be an mclock thing. >> >> >> 1. Why I ended up with so many misplaced PG's since there were no >> changes >> >> on the cluster: number of osd's, hosts, etc. >> >> Probably a result of the autoscaler splitting PGs or of some change to >> CRUSH rules such that some data can’t be placed. >> >> >> 2. Is it ok to change the target_max_misplaced_ratio to something >> higher >> >> than .05 so the autobalancer would work and I wouldn't have to >> constantly >> >> rebalance the osd's manually? >> >> I wouldn’t, that’s a symptom not the disease. >> >> Bruno >> >> _______________________________________________ >> >> ceph-users mailing list -- ceph-users(a)ceph.io >> >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >> >> >> >> >> >> >> > >> > -- >> > Bruno Gomes Pessanha >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users(a)ceph.io >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io >> > > > -- > Bruno Gomes Pessanha > -- Bruno Gomes Pessanha

1 0

Re: Many misplaced PG's, full OSD's and a good amount of manual intervention to keep my Ceph cluster alive.
by Bruno Gomes Pessanha 06 Jan '25

06 Jan '25

+ceph-users On Mon, 6 Jan 2025 at 15:50, Bruno Gomes Pessanha <bruno.pessanha(a)gmail.com> wrote: > `ceph osd df`, look at the PGS column. Could be you’re hitting the limit >> on some OSDs. It’s odd to stop at a non power of 2. >> > > # ceph osd df > ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META > AVAIL %USE VAR PGS STATUS > 8 ssd 13.97069 1.00000 14 TiB 9.4 TiB 9.4 TiB 9.8 MiB 21 > GiB 4.6 TiB 67.09 1.07 63 up > 18 ssd 13.97069 1.00000 14 TiB 8.2 TiB 8.2 TiB 8.5 MiB 19 > GiB 5.7 TiB 59.02 0.94 54 up > 28 ssd 13.97069 1.00000 14 TiB 8.1 TiB 8.1 TiB 1.5 GiB 19 > GiB 5.9 TiB 58.02 0.93 54 up > 37 ssd 13.97069 1.00000 14 TiB 13 TiB 13 TiB 84 MiB 31 > GiB 1.2 TiB 91.14 1.45 135 up > 45 ssd 13.97069 1.00000 14 TiB 9.2 TiB 9.2 TiB 542 MiB 22 > GiB 4.8 TiB 65.79 1.05 66 up > 56 ssd 13.97069 1.00000 14 TiB 8.3 TiB 8.3 TiB 5.0 MiB 19 > GiB 5.7 TiB 59.19 0.94 56 up > 66 ssd 13.97069 1.00000 14 TiB 8.9 TiB 8.8 TiB 358 MiB 20 > GiB 5.1 TiB 63.39 1.01 54 up > 77 ssd 13.97069 1.00000 14 TiB 8.7 TiB 8.7 TiB 3.7 MiB 19 > GiB 5.3 TiB 62.18 0.99 53 up > 6 ssd 13.97069 1.00000 14 TiB 8.3 TiB 8.3 TiB 355 MiB 18 > GiB 5.6 TiB 59.67 0.95 52 up > 16 ssd 13.97069 1.00000 14 TiB 7.3 TiB 7.3 TiB 572 MiB 18 > GiB 6.7 TiB 52.05 0.83 47 up > 26 ssd 13.97069 1.00000 14 TiB 8.6 TiB 8.6 TiB 921 MiB 20 > GiB 5.4 TiB 61.55 0.98 57 up > 36 ssd 13.97069 1.00000 14 TiB 13 TiB 13 TiB 1.2 GiB 31 > GiB 1.2 TiB 91.45 1.46 141 up > 46 ssd 13.97069 1.00000 14 TiB 8.1 TiB 8.1 TiB 419 MiB 19 > GiB 5.9 TiB 57.77 0.92 57 up > 54 ssd 13.97069 1.00000 14 TiB 9.0 TiB 8.9 TiB 1011 MiB 20 > GiB 5.0 TiB 64.10 1.02 65 up > 64 ssd 13.97069 1.00000 14 TiB 7.9 TiB 7.9 TiB 962 MiB 19 > GiB 6.0 TiB 56.87 0.91 59 up > 74 ssd 13.97069 1.00000 14 TiB 7.2 TiB 7.1 TiB 879 MiB 17 > GiB 6.8 TiB 51.23 0.82 49 up > 3 ssd 13.97069 1.00000 14 TiB 9.1 TiB 9.1 TiB 2.0 GiB 20 > GiB 4.8 TiB 65.45 1.04 71 up > 12 ssd 13.97069 1.00000 14 TiB 8.4 TiB 8.4 TiB 433 MiB 19 > GiB 5.6 TiB 59.92 0.96 61 up > 22 ssd 13.97069 1.00000 14 TiB 8.8 TiB 8.8 TiB 501 MiB 20 > GiB 5.2 TiB 63.01 1.01 66 up > 31 ssd 13.97069 1.00000 14 TiB 8.1 TiB 8.1 TiB 932 MiB 18 > GiB 5.8 TiB 58.20 0.93 53 up > 41 ssd 13.97069 1.00000 14 TiB 8.7 TiB 8.6 TiB 627 MiB 20 > GiB 5.3 TiB 61.96 0.99 54 up > 51 ssd 13.97069 1.00000 14 TiB 6.9 TiB 6.9 TiB 1.4 GiB 16 > GiB 7.1 TiB 49.18 0.78 43 up > 61 ssd 13.97069 1.00000 14 TiB 13 TiB 13 TiB 1000 MiB 31 > GiB 1.4 TiB 89.99 1.44 134 up > 71 ssd 13.97069 1.00000 14 TiB 8.1 TiB 8.0 TiB 553 MiB 19 > GiB 5.9 TiB 57.63 0.92 51 up > 4 ssd 13.97069 1.00000 14 TiB 7.1 TiB 7.1 TiB 525 MiB 15 > GiB 6.9 TiB 50.87 0.81 47 up > 15 ssd 13.97069 1.00000 14 TiB 8.7 TiB 8.7 TiB 606 MiB 19 > GiB 5.2 TiB 62.55 1.00 55 up > 23 ssd 13.97069 1.00000 14 TiB 7.9 TiB 7.8 TiB 423 MiB 18 > GiB 6.1 TiB 56.32 0.90 49 up > 34 ssd 13.97069 1.00000 14 TiB 10 TiB 10 TiB 551 MiB 24 > GiB 3.5 TiB 74.79 1.19 72 up > 44 ssd 13.97069 1.00000 14 TiB 13 TiB 13 TiB 77 MiB 28 > GiB 1.4 TiB 89.90 1.43 132 up > 59 ssd 13.97069 1.00000 14 TiB 9.1 TiB 9.0 TiB 510 MiB 20 > GiB 4.9 TiB 64.91 1.04 61 up > 69 ssd 13.97069 1.00000 14 TiB 9.7 TiB 9.7 TiB 1.0 GiB 22 > GiB 4.2 TiB 69.62 1.11 71 up > 79 ssd 13.97069 1.00000 14 TiB 7.2 TiB 7.1 TiB 1.0 GiB 17 > GiB 6.8 TiB 51.26 0.82 47 up > 9 ssd 13.97069 1.00000 14 TiB 8.2 TiB 8.2 TiB 1.1 MiB 18 > GiB 5.8 TiB 58.81 0.94 49 up > 19 ssd 13.97069 1.00000 14 TiB 9.2 TiB 9.2 TiB 421 MiB 21 > GiB 4.8 TiB 65.68 1.05 63 up > 29 ssd 13.97069 1.00000 14 TiB 8.8 TiB 8.7 TiB 511 MiB 21 > GiB 5.2 TiB 62.68 1.00 61 up > 39 ssd 13.97069 1.00000 14 TiB 8.4 TiB 8.4 TiB 556 MiB 20 > GiB 5.6 TiB 60.15 0.96 58 up > 49 ssd 13.97069 1.00000 14 TiB 7.0 TiB 7.0 TiB 2.9 MiB 15 > GiB 7.0 TiB 49.92 0.80 45 up > 58 ssd 13.97069 1.00000 14 TiB 7.2 TiB 7.1 TiB 432 MiB 16 > GiB 6.8 TiB 51.22 0.82 49 up > 67 ssd 13.97069 1.00000 14 TiB 7.4 TiB 7.4 TiB 1.1 GiB 17 > GiB 6.6 TiB 52.80 0.84 47 up > 78 ssd 13.97069 1.00000 14 TiB 7.7 TiB 7.7 TiB 574 MiB 19 > GiB 6.3 TiB 54.94 0.88 55 up > 0 ssd 13.97069 1.00000 14 TiB 8.2 TiB 8.2 TiB 455 MiB 19 > GiB 5.8 TiB 58.66 0.94 58 up > 10 ssd 13.97069 1.00000 14 TiB 8.5 TiB 8.5 TiB 439 MiB 19 > GiB 5.4 TiB 61.07 0.97 48 up > 21 ssd 13.97069 1.00000 14 TiB 13 TiB 13 TiB 646 MiB 30 > GiB 1.3 TiB 90.87 1.45 135 up > 32 ssd 13.97069 1.00000 14 TiB 8.6 TiB 8.6 TiB 3.2 MiB 19 > GiB 5.4 TiB 61.60 0.98 56 up > 42 ssd 13.97069 1.00000 14 TiB 13 TiB 13 TiB 582 MiB 33 > GiB 1.3 TiB 90.78 1.45 133 up > 52 ssd 13.97069 1.00000 14 TiB 7.4 TiB 7.4 TiB 1.4 GiB 17 > GiB 6.6 TiB 52.74 0.84 47 up > 62 ssd 13.97069 1.00000 14 TiB 7.4 TiB 7.3 TiB 1.5 GiB 16 > GiB 6.6 TiB 52.68 0.84 49 up > 72 ssd 13.97069 1.00000 14 TiB 7.7 TiB 7.7 TiB 1.2 GiB 17 > GiB 6.3 TiB 55.20 0.88 53 up > 1 ssd 13.97069 1.00000 14 TiB 8.1 TiB 8.0 TiB 1.5 GiB 19 > GiB 5.9 TiB 57.73 0.92 64 up > 11 ssd 13.97069 1.00000 14 TiB 8.4 TiB 8.4 TiB 422 MiB 19 > GiB 5.5 TiB 60.37 0.96 62 up > 20 ssd 13.97069 1.00000 14 TiB 7.7 TiB 7.7 TiB 405 MiB 17 > GiB 6.3 TiB 55.02 0.88 52 up > 30 ssd 13.97069 1.00000 14 TiB 8.1 TiB 8.1 TiB 494 MiB 19 > GiB 5.9 TiB 57.84 0.92 49 up > 40 ssd 13.97069 1.00000 14 TiB 9.2 TiB 9.2 TiB 376 MiB 22 > GiB 4.7 TiB 66.17 1.06 74 up > 50 ssd 13.97069 1.00000 14 TiB 8.3 TiB 8.3 TiB 1.0 GiB 20 > GiB 5.7 TiB 59.43 0.95 54 up > 60 ssd 13.97069 1.00000 14 TiB 13 TiB 13 TiB 2.1 GiB 32 > GiB 1.2 TiB 91.27 1.46 136 up > 70 ssd 13.97069 1.00000 14 TiB 8.3 TiB 8.3 TiB 498 MiB 19 > GiB 5.6 TiB 59.64 0.95 56 up > 2 ssd 13.97069 1.00000 14 TiB 13 TiB 13 TiB 1.1 GiB 31 > GiB 1.2 TiB 91.62 1.46 133 up > 14 ssd 13.97069 1.00000 14 TiB 9.2 TiB 9.2 TiB 9.8 MiB 22 > GiB 4.7 TiB 66.05 1.05 60 up > 24 ssd 13.97069 1.00000 14 TiB 7.4 TiB 7.4 TiB 426 MiB 17 > GiB 6.5 TiB 53.13 0.85 50 up > 33 ssd 13.97069 1.00000 14 TiB 8.9 TiB 8.9 TiB 413 MiB 20 > GiB 5.1 TiB 63.54 1.01 56 up > 43 ssd 13.97069 1.00000 14 TiB 8.2 TiB 8.2 TiB 529 MiB 18 > GiB 5.8 TiB 58.58 0.93 58 up > 53 ssd 13.97069 1.00000 14 TiB 7.6 TiB 7.6 TiB 533 MiB 17 > GiB 6.4 TiB 54.29 0.87 52 up > 63 ssd 13.97069 1.00000 14 TiB 8.5 TiB 8.5 TiB 569 MiB 19 > GiB 5.4 TiB 61.20 0.98 58 up > 73 ssd 13.97069 1.00000 14 TiB 8.7 TiB 8.7 TiB 536 MiB 20 > GiB 5.3 TiB 62.14 0.99 63 up > 5 ssd 13.97069 1.00000 14 TiB 8.3 TiB 8.3 TiB 1.3 GiB 19 > GiB 5.7 TiB 59.46 0.95 53 up > 13 ssd 13.97069 1.00000 14 TiB 8.6 TiB 8.6 TiB 923 MiB 21 > GiB 5.4 TiB 61.60 0.98 49 up > 25 ssd 13.97069 1.00000 14 TiB 7.4 TiB 7.4 TiB 4.3 MiB 16 > GiB 6.6 TiB 53.03 0.85 46 up > 35 ssd 13.97069 1.00000 14 TiB 7.4 TiB 7.3 TiB 397 MiB 17 > GiB 6.6 TiB 52.70 0.84 45 up > 47 ssd 13.97069 1.00000 14 TiB 9.0 TiB 9.0 TiB 10 MiB 21 > GiB 4.9 TiB 64.74 1.03 60 up > 57 ssd 13.97069 1.00000 14 TiB 7.1 TiB 7.1 TiB 418 MiB 17 > GiB 6.8 TiB 51.13 0.82 50 up > 68 ssd 13.97069 1.00000 14 TiB 13 TiB 13 TiB 590 MiB 30 > GiB 1.2 TiB 91.28 1.46 141 up > 76 ssd 13.97069 1.00000 14 TiB 7.2 TiB 7.2 TiB 976 MiB 16 > GiB 6.7 TiB 51.70 0.82 46 up > 7 ssd 13.97069 1.00000 14 TiB 7.8 TiB 7.8 TiB 2.6 MiB 18 > GiB 6.1 TiB 56.03 0.89 49 up > 17 ssd 13.97069 1.00000 14 TiB 7.2 TiB 7.2 TiB 443 MiB 16 > GiB 6.8 TiB 51.34 0.82 43 up > 27 ssd 13.97069 1.00000 14 TiB 8.5 TiB 8.4 TiB 8.6 MiB 21 > GiB 5.5 TiB 60.51 0.97 54 up > 38 ssd 13.97069 1.00000 14 TiB 7.9 TiB 7.9 TiB 1005 MiB 18 > GiB 6.0 TiB 56.88 0.91 52 up > 48 ssd 13.97069 1.00000 14 TiB 7.6 TiB 7.5 TiB 2.0 MiB 17 > GiB 6.4 TiB 54.08 0.86 45 up > 55 ssd 13.97069 1.00000 14 TiB 13 TiB 13 TiB 509 MiB 30 > GiB 1.2 TiB 91.50 1.46 137 up > 65 ssd 13.97069 1.00000 14 TiB 7.7 TiB 7.7 TiB 901 MiB 17 > GiB 6.3 TiB 55.07 0.88 51 up > 75 ssd 13.97069 1.00000 14 TiB 8.5 TiB 8.5 TiB 6.0 MiB 19 > GiB 5.5 TiB 60.66 0.97 58 up > TOTAL 1.1 PiB 701 TiB 699 TiB 49 GiB 1.6 > TiB 417 TiB 62.70 > MIN/MAX VAR: 0.78/1.46 STDDEV: 11.76 > > We’re a community! That curve is way gentler than it used to be. >> > 🙌👍 > > > On Sun, 5 Jan 2025 at 14:59, Anthony D'Atri <anthony.datri(a)gmail.com> > wrote: > >> >> >> Do you use the autoscalar or did you trigger a manual PG increment of >> the >> >> pool? >> > >> > The pool had autoscale enabled until 2 days ago when I thought it was >> > better to change things manually in order to have a more deterministic >> > result. Yes, I wanted to increase from "1" to something like "1024" but >> it >> > looks like it was capped to the 144 no matter what I do: >> >> `ceph osd df`, look at the PGS column. Could be you’re hitting the limit >> on some OSDs. It’s odd to stop at a non power of 2. >> >> > Is it correct to say that every PG/OSD change can potentially cause data >> > misplacements, unbalanced osd's and long backfills? I'll be way more >> > careful before tuning it if that's the case. >> >> The autoscaler will usually only bump pg_num for a pool when the value is >> <half what it thinks it should be. >> >> I suggest setting the ‘bulk’ flag one pool at a time to effectively >> pre-split PGs as if the pool were full of data already. Ignore .mgr. >> Start with the pools with the fewest , let the cluster settle between each >> adjustment. That way the autoscaler will only make changes if cluster >> topology changes. >> >> >> > Thank you both so much! It definitely helped me to understand Ceph >> better. >> > It is kind of a steep curve :). >> >> We’re a community! That curve is way gentler than it used to be. > > > > -- > Bruno Gomes Pessanha > -- Bruno Gomes Pessanha

1 0

Problems with autoscaler (overlapping roots) after changing the pool class
by Massimo Sgaravatto 06 Jan '25

06 Jan '25

Dear all I have just changed the crush rule for all the replicated pools in the following way: ceph osd crush rule create-replicated replicated_hdd default host hdd ceph osd pool set <poolname> crush_rule replicated_hdd See also this [*] thread Before applying this change, these pools were all using the replicated_ruleset rule where the class is not specified. I am noticing now a problem with the autoscaler: "ceph osd pool autoscale-status" doesn't report any output and the mgr log complains about overlapping roots: [pg_autoscaler ERROR root] pool xyz has overlapping roots: {-18, -1} Indeed: # ceph osd crush tree --show-shadow ID CLASS WEIGHT TYPE NAME -18 hdd 1329.26501 root default~hdd -17 hdd 329.14154 rack Rack11-PianoAlto~hdd -15 hdd 54.56085 host ceph-osd-04~hdd 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... ... -1 1329.26501 root default -7 329.14154 rack Rack11-PianoAlto -8 54.56085 host ceph-osd-04 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... I have already read about this behavior but I have no clear ideas how to fix the problem. I read somewhere that the problem happens when there are rules that force some pools to only use one class and there are also pools which does not make any distinction between device classes All the replicated pools are using the replicated_hdd pool but I also have some EC pools which are using a profile where the class is not specified. As far I understand, I can't force these pools to use only the hdd class: according to the doc I can't change this profile specifying the hdd class (or at least the change wouldn't be applied to the existing EC pools) Any suggestions ? The crush map is available at https://cernbox.cern.ch/s/gIyjbQbmoTFHCrr, if you want to have a look Many thanks, Massimo [*] https://www.mail-archive.com/ceph-users@ceph.io/msg18534.html

4 9

recovery a downed/inaccessible pg
by Nick Anderson 02 Jan '25

02 Jan '25

Hello ceph user-community, We have a reef (18.2.2) cluster with an erasure coded pool of k=8, m=4 that has a pg that is currently down with the following: ``` WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive, 1 pg incomplete pg 2.699 is remapped+incomplete, acting [NONE,NONE,NONE,NONE,NONE,353,488,333,408,282,167,145] (reducing pool i1-sea-rbd-01-data min_size from 9 may help; search ceph.com/docs for 'incomplete') ``` Timeline leading up to our current state noted above. We had an issue with a host that had a flappy network and during our efforts to restore the network some prolonged OSD heartbeats not reachable seems to have caused two OSDs to go down/out. The OSD containers were stuck in a restart loop and eventually were auto-out'd from the cluster. Restarting the affected OSD containers had no effect and the pg was stuck in a "down+remapped state." ``` [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive, 1 pg down pg 2.699 is down+remapped, acting [NONE,NONE,NONE,NONE,613,353,488,333,408,282,167,145] ``` Looking over the `pg query 2.699` we could the following: ``` "blocked": "peering is blocked due to down osds", "down_osds_we_would_probe": [ 38, 120 ], "peering_blocked_by": [ { "osd": 38, "current_lost_at": 90605, "comment": "starting or marking this osd lost may let us proceed" }, { "osd": 120, "current_lost_at": 0, "comment": "starting or marking this osd lost may let us proceed" }, ``` With that output and reading ceph documentation[1], we decided to mark osd.38 as lost (`ceph osd lost 38`). Within a couple minutes, a 3rd OSD went down in osd.613 and exhibited the same restart loop behavior. Leaving us with the following state from a ` ceph pg query 2.699` ``` "blocked": "peering is blocked due to down osds", "down_osds_we_would_probe": [ 38, 120, 613 ], "peering_blocked_by": [ { "osd": 38, "current_lost_at": 90605, "comment": "starting or marking this osd lost may let us proceed" }, { "osd": 120, "current_lost_at": 0, "comment": "starting or marking this osd lost may let us proceed" }, { "osd": 613, "current_lost_at": 0, "comment": "starting or marking this osd lost may let us proceed" } ] }, { "name": "Started", "enter_time": "2024-12-21T17:13:18.435769+0000" } ], "agent_state": {} ``` After sleeping on it and letting the cluster recover from another failed OSD, we decided to mark all three as lost (osd.38, osd.120, and osd.613). Unfortunately, this caused two additional OSDs (osd.226 and osd.339) to go down. This changed the pg state from `down+remapped` to`remapped+incomplete`. We now have 5 down/out OSDs and are wondering if there is anyone to recover this PG? This one PG has caused our 1.6 PiB to be inaccessible.. We've gleaned some documentation[2] involving exporting the affected pg data with ceph-objectstore-tool and importing into a new OSD. Is this our only route to possibly recover this pg and its data? The strange thing is the OSDs underlying disk seems fine in that the ceph volume-group/LVM are intact. There are 0 issues with the disks from what I can glean or any indication of disk issues (the last 3 disks that were outed were part of a successful recovery/backfill with no issues). I tested the two original outed OSDs and the underlying disks all pass smartctl short tests and have seen no indication of hardware issues with these spinning disks. Looking over the ceph documentation, it looks like there is a newer tool that is cephadm/bluestore friendly in ceph-bluestore-tool[3] that does the same job as ceph-objectstore-tool? We are hoping there is a way to recover this PG and appreciate any advice. Currently our production cluster is down and are looking for recovery avenues. - Nick Links: [1] https://docs.ceph.com/en/reef/rados/troubleshooting/troubleshooting-pg/#pla… [2] https://www.croit.io/blog/how-to-recover-inactive-pgs-using-ceph-objectstor… [3] https://docs.ceph.com/en/reef/man/8/ceph-bluestore-tool/

3 2