- ceph-users - lists.ceph.io

Changing crush map result in > 100% objects degraded
by Kasper Rasmussen 21 Jan '25

21 Jan '25

Hi community Please help me understand what is going on. I have a ceph (Reef) test cluster with the following crushmap ceph osd crush tree ID CLASS WEIGHT TYPE NAME -1 12.00000 root default -7 3.00000 host ksr-ceph-osd1 0 hdd 1.00000 osd.0 6 hdd 1.00000 osd.6 10 hdd 1.00000 osd.10 -9 3.00000 host ksr-ceph-osd2 3 hdd 1.00000 osd.3 7 hdd 1.00000 osd.7 11 hdd 1.00000 osd.11 -5 3.00000 host ksr-ceph-osd3 2 hdd 1.00000 osd.2 5 hdd 1.00000 osd.5 9 hdd 1.00000 osd.9 -3 3.00000 host ksr-ceph-osd4 1 hdd 1.00000 osd.1 4 hdd 1.00000 osd.4 8 hdd 1.00000 osd.8 -11 0 rack rack1 -13 0 rack rack2 -15 0 rack rack3 -17 0 rack rack4 Ceph status is like: cluster: id: 8a174287-42f8-43b6-9973-f174110b508b health: HEALTH_OK services: mon: 5 daemons, quorum ksr-ceph-mon2,ksr-ceph-mon3,ksr-ceph-mon1,ksr-ceph-mon5,ksr-ceph-mon4 (age 3h) mgr: ksr-ceph-mon1(active, since 3h), standbys: ksr-ceph-mon2, ksr-ceph-mon3 mds: 2/2 daemons up, 3 standby osd: 12 osds: 12 up (since 3h), 12 in (since 6d) data: volumes: 2/2 healthy pools: 5 pools, 129 pgs objects: 578 objects, 154 MiB usage: 1.0 GiB used, 599 GiB / 600 GiB avail pgs: 129 active+clean I then run: ceph osd set norecover ceph osd set nobackfill ceph osd set norebalance ceph osd crush move ksr-ceph-osd1 rack=rack1 ceph osd crush move ksr-ceph-osd2 rack=rack2 ceph osd crush move ksr-ceph-osd3 rack=rack3 ceph osd crush move ksr-ceph-osd4 rack=rack4 resulting in the following crush tree ceph osd crush tree ID CLASS WEIGHT TYPE NAME -1 12.00000 root default -11 3.00000 rack rack1 -7 3.00000 host ksr-ceph-osd1 0 hdd 1.00000 osd.0 6 hdd 1.00000 osd.6 10 hdd 1.00000 osd.10 -13 3.00000 rack rack2 -9 3.00000 host ksr-ceph-osd2 3 hdd 1.00000 osd.3 7 hdd 1.00000 osd.7 11 hdd 1.00000 osd.11 -15 3.00000 rack rack3 -5 3.00000 host ksr-ceph-osd3 2 hdd 1.00000 osd.2 5 hdd 1.00000 osd.5 9 hdd 1.00000 osd.9 -17 3.00000 rack rack4 -3 3.00000 host ksr-ceph-osd4 1 hdd 1.00000 osd.1 4 hdd 1.00000 osd.4 8 hdd 1.00000 osd.8 And ceph status is like: cluster: id: 8a174287-42f8-43b6-9973-f174110b508b health: HEALTH_WARN nobackfill,norebalance,norecover flag(s) set Degraded data redundancy: 2701/1734 objects degraded (155.767%), 55 pgs degraded services: mon: 5 daemons, quorum ksr-ceph-mon2,ksr-ceph-mon3,ksr-ceph-mon1,ksr-ceph-mon5,ksr-ceph-mon4 (age 3h) mgr: ksr-ceph-mon1(active, since 3h), standbys: ksr-ceph-mon2, ksr-ceph-mon3 mds: 2/2 daemons up, 3 standby osd: 12 osds: 12 up (since 3h), 12 in (since 6d); 22 remapped pgs flags nobackfill,norebalance,norecover data: volumes: 2/2 healthy pools: 5 pools, 129 pgs objects: 578 objects, 154 MiB usage: 1.0 GiB used, 599 GiB / 600 GiB avail pgs: 2701/1734 objects degraded (155.767%) 479/1734 objects misplaced (27.624%) 70 active+clean 34 active+recovery_wait+degraded 20 active+recovery_wait+undersized+degraded+remapped 3 active+recovering 1 active+recovery_wait+remapped 1 active+recovery_wait+degraded+remapped No crush rules have been changed on the pools. All pools has the default replicated_rule ceph osd crush rule dump replicated_rule { "rule_id": 0, "rule_name": "replicated_rule", "type": 1, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] } Questions 1 - Why do this result in such a high - objects degraded - percentage? 2 - Why do PGs get undersized? All in all this behavior does not make sense to me - I expect nothing to happen basically since I've only implemented some buckets in the map, no rules have changed - so I reach out in hope that someone can explain me the logic.

4 6

Emergency support request for ceph MDS trouble shooting
by Frank Schilder 20 Jan '25

20 Jan '25

Dear all, this is a request to companies/consultants with development experience on ceph for a contract to help us out of our current file system outage. If you can offer help, please send a PM directly back to me. Short description with links to what we found out already: We experience a total outage of access to our ceph file system due to MDS rank 2 getting hung on startup. It does not crash. This rank has .mds_cache.num_strays=99446248 and the initial processing of stray items leads to a producer-consumer deadlock that prevents this rank from becoming responsive. On startup approximately 37Mio stray items can be scheduled for purging before the dead lock occurs. The cluster is healthy otherwise. Ceph version is 16.2.15 (618f440892089921c3e944a991122ddc44e60516) pacific (stable). We have deployed a rescue server with large amount of RAM and swap that we can make accessible remotely. On this rescue server we have an MDS up on rank 2 for trouble shooting and this is all working fine. Related bug reports with more details: https://tracker.ceph.com/issues/69547 https://www.spinics.net/lists/ceph-users/msg85394.html https://www.spinics.net/lists/ceph-users/msg85480.html Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

1 1

Notes from CSC Weekly 2025-01-20
by Dan van der Ster 20 Jan '25

20 Jan '25

Hi all, The CSC Weekly (formerly known as the CLT Weekly) was shortened today due to a US holiday. All planned agenda items were postponed to next week. There was however one technical point to be communicated today: Radek would like all component leads to review PR https://github.com/ceph/ceph/pull/60104, which touches bufferlists across the stack. Otherwise, also please note that the etherpad for the CSC Weekly has been moved to https://pad.ceph.com/p/csc-weekly-minutes Cheers, Dan

1 0

ceph orch ls --refresh
by Alex Hussein-Kershaw (HE/HIM) 20 Jan '25

20 Jan '25

Hi Folks, Curious about the --refresh argument on this command (and similar). From experience so far, and a bit of a code read, it seems that: * Passing --refresh will kick the serve loop to run. * It won't wait for it to complete, so data returned may not be the latest. * Re-running the "ceph orch ls" command soon after may or may not return the latest data, depending on if the serve loop has completed. * The serve loop will run once every N seconds anyway (looks like 60 for me, which I assume is the default?) so data shouldn't be more than a minute out of date in normal operation. Is that an accurate understanding? I'm looking to use this command in some scripts so would like to be sure I understand what I am doing. Thanks, Alex

1 0

Help needed: s3cmd set ACL command possess S3 error: 400 (InvalidArgument) in squid ceph version.
by Saif Mohammad 20 Jan '25

20 Jan '25

Hello Community, We are trying to set ACL for one of the objects by s3cmd tool within the buckets to be public by using the command as follows but we are unable to set it in squid ceph version, however the same was done in the reef version, we were successfully able to set it public. Please let me know if we have any solutions for it. $ s3cmd setacl s3://<my-bucket>/logo.png --acl-public ERROR: S3 error: 400 (InvalidArgument): Invalid group uri Thanks, Mohammad Saif

2 2

Adding Rack to crushmap - Rebalancing multiple PB of data - advice/experience
by Kasper Rasmussen 20 Jan '25

20 Jan '25

I'm managing a ceph cluster with +1K OSDs distributed accross 56 host. Untill now the crush rule used is the default replicated rule, but I want to change that in order to implement failure domain on rack level. Ceph version: Pacific 16.2.15 All pools(RBD and CephFS) currently use the default replicated_rule All OSD hosts has 25G network, and spinning disks(HDD) MON DBs is on NVMEs Workload is 24x7x365 The builtin balancer is disabled, and has been for a long time - Instead balancing has been done by a cron job executing - ceph osd reweight-by-utilization 112 0.05 30 Current plan is to * Disable rebalancing and backfilling by executing - ceph osd set norebalance; ceph osd set nobackfill; * Add all 7 Rack to crushmap and distribute the hosts (8 in each.) by using the built in commands like - * ceph osd crush add-bucket rack<#> rack root=default * ceph osd crush move osd-host<#> rack=rack<#> * Create the new rack split rule with command * ceph osd crush rule create-replicated rack_split default rack * Set the rule across all my pools * for p in $(ceph osd lspools | cut -d' ' -f 2) ; do echo $p $(ceph osd pool set $p crush_rule rack_split) ; done * I will probably also be using upmap-remapped.py here.. * Finally enable rebalancing, and backfilling - ceph osd unset norebalance; ceph osd unset nobackfill; However I'm concerned with the amount of data that needs to be rebalanced, since the cluster holds multiple PB, and I'm looking for review of/input for my plan, as well as words of advice/experience from someone who has been in a similar situation. Also I've seen some weird behavior - where Pacific(16) seems to do something different from Quincy(17) - In a test cluster I've tested the plan - On Pasific: Data is marked as "degraded", and not misplaced as expected. I also see above 2000% degraded data (but that might be another issue) On Quincy: Data is marked as misplaced - which seems correct. All experience and/or input will be greatly appreciated.

4 7

weekend maintenance to bot+bridge slack/irc/discord
by Alvaro Soto 19 Jan '25

19 Jan '25

Hi All, This weekend, I'll be moving the bot from the primary node, so expect some hiccups in the bridge and log bot. Cheers. -- Alvaro Soto *Note: My work hours may not be your work hours. Please do not feel the need to respond during a time that is not convenient for you.* ---------------------------------------------------------- Great people talk about ideas, ordinary people talk about things, small people talk... about other people.

1 1

squid 19.2.1 RC QE validation status
by Yuri Weinstein 18 Jan '25

18 Jan '25

Details of this release are summarized here: https://tracker.ceph.com/issues/69234#note-1 Release Notes - TBD LRC upgrade - TBD Gibba upgrade -TBD Please provide tracks for failures so we avoid duplicates. Seeking approvals/reviews for: rados - Radek, Laura rgw - Eric, Adam E fs - Venky orch - Adam King rbd, krbd - Ilya quincy-x, reef-x - Laura, Neha powercycle - Brad crimson-rados - Matan, Samuel ceph-volume - Guillaume

14 33

Non existing host in maintenance
by Dominique Ramaekers 17 Jan '25

17 Jan '25

Hi, I have removed a host (hvs004) that was in maintenance. The system disk of this host had failed, so removed the host hvs004 in ceph; replaced the system disk; erased all the osd-disks and reinstalled the host as hvs005. Resulting a cluster status in waring that doesn’t goes away: health: HEALTH_WARN 1 host is in maintenance mode Removal is done by “ceph orch host rm hvs004 --offline –force” in cephadm shell. How can I correct this false warning? Some more info: root@hvs001:/# ceph orch host ls HOST ADDR LABELS STATUS hvs001 xxx.xxx.xxx.xxx _admin hvs002 xxx.xxx.xxx.xxx _admin hvs003 xxx.xxx.xxx.xxx _admin hvs005 xxx.xxx.xxx.xxx _admin 4 hosts in cluster root@hvs001:/# ceph health detail HEALTH_WARN 1 host is in maintenance mode [WRN] HOST_IN_MAINTENANCE: 1 host is in maintenance mode hvs004 is in maintenance :-/ Help is greatly apreciated…

2 4

Cephadm: Specifying RGW Certs & Keys By Filepath
by Alex Hussein-Kershaw (HE/HIM) 17 Jan '25

17 Jan '25

Hi Folks, Looking for some advice on RGW service specs and Cephadm. I've read the docs here: RGW Service — Ceph Documentation<https://docs.ceph.com/en/reef/cephadm/services/rgw/> Using a service spec I can deploy a RGW: service_type: rgw service_id: '25069123' service_name: rgw.25069123 placement: hosts: - raynor-sc-1 extra_container_args: - -v - /etc/pki:/etc/pki:ro - -v - /etc/ssl/:/etc/ssl:ro spec: rgw_frontend_port: 7480 rgw_realm: geored_realm rgw_zone: siteA rgw_zonegroup: geored_zg ssl: true rgw_frontend_extra_args: - "ssl_certificate=/etc/ssl/certs/server.crt" - "ssl_private_key=/etc/ssl/private/server.key" But it fails to start because my "rgw_frontends" config ends up as this. Notice the two entries of "ssl_certificate". beast ssl_port=7480 ssl_certificate=config://rgw/cert/rgw.25069123 ssl_certificate=/etc/ssl/certs/server.crt ssl_private_key=/etc/ssl/private/server.key That causes it to fail to start: debug 2025-01-16T15:26:54.219+0000 7f65bcfdc840 0 deferred set uid:gid to 167:167 (ceph:ceph) debug 2025-01-16T15:26:54.219+0000 7f65bcfdc840 0 ceph version 19.2.0 (16063ff2022298c9300e49a547a16ffda59baf13) squid (stable), process radosgw, pid 7 debug 2025-01-16T15:26:54.219+0000 7f65bcfdc840 0 framework: beast debug 2025-01-16T15:26:54.219+0000 7f65bcfdc840 0 framework conf key: ssl_port, val: 7480 debug 2025-01-16T15:26:54.219+0000 7f65bcfdc840 0 framework conf key: ssl_certificate, val: config://rgw/cert/rgw.25069123 debug 2025-01-16T15:26:54.219+0000 7f65bcfdc840 0 framework conf key: ssl_certificate, val: /etc/ssl/certs/server.crt debug 2025-01-16T15:26:54.219+0000 7f65bcfdc840 0 framework conf key: ssl_private_key, val: /etc/ssl/private/server.key debug 2025-01-16T15:26:54.520+0000 7f65bcfdc840 -1 LDAP not started since no server URIs were provided in the configuration. debug 2025-01-16T15:26:54.531+0000 7f65bcfdc840 0 framework: beast debug 2025-01-16T15:26:54.531+0000 7f65bcfdc840 0 framework conf key: ssl_certificate, val: config://rgw/cert/$realm/$zone.crt debug 2025-01-16T15:26:54.531+0000 7f65bcfdc840 0 framework conf key: ssl_private_key, val: config://rgw/cert/$realm/$zone.key debug 2025-01-16T15:26:54.597+0000 7f65bcfdc840 0 starting handler: beast debug 2025-01-16T15:26:54.601+0000 7f65bcfdc840 -1 ssl_certificate was not found: rgw/cert/rgw.25069123 debug 2025-01-16T15:26:54.602+0000 7f65bcfdc840 -1 no ssl_certificate configured for ssl_port debug 2025-01-16T15:26:54.602+0000 7f65bcfdc840 -1 ERROR: failed initializing frontend debug 2025-01-16T15:26:54.602+0000 7f65bcfdc840 -1 ERROR: initialize frontend fail, r = 22 Correcting that config manually: $ ceph config set client.rgw.25069123.raynor-sc-1.qiatgr rgw_frontends "beast ssl_port=7480 ssl_certificate=/etc/ssl/certs/server.crt ssl_private_key=/etc/ssl/private/server.key" Then allows the RGW to start: debug 2025-01-16T15:28:17.391+0000 7fba6e85a840 0 deferred set uid:gid to 167:167 (ceph:ceph) debug 2025-01-16T15:28:17.391+0000 7fba6e85a840 0 ceph version 19.2.0 (16063ff2022298c9300e49a547a16ffda59baf13) squid (stable), process radosgw, pid 7 debug 2025-01-16T15:28:17.391+0000 7fba6e85a840 0 framework: beast debug 2025-01-16T15:28:17.391+0000 7fba6e85a840 0 framework conf key: ssl_port, val: 7480 debug 2025-01-16T15:28:17.391+0000 7fba6e85a840 0 framework conf key: ssl_certificate, val: /etc/ssl/certs/server.crt debug 2025-01-16T15:28:17.391+0000 7fba6e85a840 0 framework conf key: ssl_private_key, val: /etc/ssl/private/server.key debug 2025-01-16T15:28:17.767+0000 7fba6e85a840 -1 LDAP not started since no server URIs were provided in the configuration. debug 2025-01-16T15:28:17.781+0000 7fba6e85a840 0 framework: beast debug 2025-01-16T15:28:17.781+0000 7fba6e85a840 0 framework conf key: ssl_certificate, val: config://rgw/cert/$realm/$zone.crt debug 2025-01-16T15:28:17.781+0000 7fba6e85a840 0 framework conf key: ssl_private_key, val: config://rgw/cert/$realm/$zone.key debug 2025-01-16T15:28:17.860+0000 7fba6e85a840 0 starting handler: beast debug 2025-01-16T15:28:17.908+0000 7fba6e85a840 0 set uid:gid to 167:167 (ceph:ceph) debug 2025-01-16T15:28:17.912+0000 7fba6e85a840 1 mgrc service_daemon_register rgw.643295 metadata {arch=x86_64,ceph_release=squid,ceph_version=ceph version 19.2.0 (16063ff2022298c9300e49a547a16ffda59baf13) squid (stable),ceph_version_short=19.2.0,container_hostname=raynor-sc-1,container_image=ceph/squid:v19.2.0,cpu=Intel(R) Xeon(R) CPU E5-2683 v3 @ 2.00GHz,distro=centos,distro_description=CentOS Stream 9,distro_version=9,frontend_config#0=beast ssl_port=7480 ssl_certificate=/etc/ssl/certs/server.crt ssl_private_key=/etc/ssl/private/server.key,frontend_type#0=beast,hostname=raynor-sc-1,id=25069123.raynor-sc-1.qiatgr,kernel_description=#1 SMP Thu Dec 19 20:58:14 EST 2024,kernel_version=5.4.288-1.el8.elrepo.x86_64,mem_swap_kb=0,mem_total_kb=12251748,num_handles=1,os=Linux,pid=7,realm_id=c99973d7-27cb-4abd-8391-bfbe4fa928be,realm_name=geored_realm,zone_id=30816d1d-e315-471b-bc84-6cdc4f6f1408,zone_name=siteA,zonegroup_id=c109a213-6638-4028-b21b-8bc9fcb49cd3,zonegroup_name=geored_zg} Specifying the certificate by file path I think is the right choice for me as: * I don't have a domain, I have a per host certificate covering the IP of the host. Each is different, signed by a self-signed trusted CA. * I don't want to put the hardcoded key into the service spec, as suggested in the docs, as it appears in the mgr logs. My org policy (sensibly) doesn't allow for secrets to appear in logs. * It's just generally more hassle to manage keys on a per VM basis, rather than a static file path. * It will also allow me to not manage RGW services on a per host basis, but use the same spec for all and a count=1 placement, which would be lovely. So my question - is there a way around this such that I can specify the key and certificate by file path and not have to manually change the config to make it work? Thanks, Alex

2 10