- ceph-users - lists.ceph.io

OSD stuck during a two-OSD drain
by Nicola Mori 20 Dec '24

20 Dec '24

Dear Ceph users, I'm upgrading some disks of my cluster (Squid 19.2.0 managed by cephadm, in which basically I have only a 6+2 EC pool over 12 hosts). To speed up the operations I issued a ceph orch osd rm --replace for two OSDs in two different hosts; the drain started for both and for one OSD finished smoothly and it is now in destroyed state. But for the second OSD it stopped with a single PG remaining to be moved away before the OSD is completely drained: # ceph orch osd rm status OSD HOST STATE PGS REPLACE FORCE ZAP DRAIN STARTED AT 31 rokanan draining 1 True False False 2024-12-19 08:57:36.458704+00:00 and there is no backfill activity going on, even i f the PG is labeled as backfilling : # ceph -s cluster: id: b1029256-7bb3-11ec-a8ce-ac1f6b627b45 health: HEALTH_WARN 52 pgs not deep-scrubbed in time (muted: OSD_SLOW_PING_TIME_BACK OSD_SLOW_PING_TIME_FRONT) services: mon: 5 daemons, quorum bofur,fili,aka,bifur,romolo (age 7d) mgr: fili.olevnm(active, since 18h), standbys: bofur.tklnrn, bifur.htimkf mds: 2/2 daemons up, 1 standby osd: 124 osds: 123 up (since 4h), 122 in (since 22h); 1 remapped pgs data: volumes: 1/1 healthy pools: 3 pools, 529 pgs objects: 27.11M objects, 78 TiB usage: 104 TiB used, 162 TiB / 266 TiB avail pgs: 53120/216457202 objects misplaced (0.025%) 302 active+clean 178 active+clean+scrubbing 48 active+clean+scrubbing+deep 1 active+remapped+backfilling Is all of the above normal? I guessed that maybe only one destroyed OSD at once can exist in the cluster, and that after replacing its disk and recreating it the drain for the second one would resume and finish, is this plausible? Thanks, Nicola

2 2

Cephadm multi zone rgw_dns_name setting
by Huseyin Cotuk 20 Dec '24

20 Dec '24

Hello Cephers, I am running a multi-site Ceph cluster with multiple zones in both regions. Specifically, site A has two zones named tr-ist-01 and tr-ist-02. The first zone (tr-ist-01) is used for local object storage needs and not synced with the remote site. The second zone (tr-ist-02) is used for multi-site and synced with remote site. When using cephadm with single zone radosgw, the fqdn can be set as follows. ceph config set global rgw_dns_name {rgw_service_fqdn} My question is, how can I set the fqdn of each radosgw service in a multiple zone environment? Rather than setting a global rgw_dns_name, is it possible to set this setting for each zone? It is possible to set this for each pod with specific podname but it can change in runtime. Any help will be appreciated. Thanks in advance Best regards, Huseyin Cotuk hcotuk(a)gmail.com

2 2

[RGW] multisite sync, stall recovering shards
by Gilles Mocellin 19 Dec '24

19 Dec '24

Hi there ! Being suspicious about replication between two clusters, I've done a radosgw-admin data sync init on the secondary zone. Since then, after a lot of activity, I'm stuck with recovering shards, nothing moves. Incremental sync still work. Wondering if I had a bad state also on the primary, I also did a data sync init on the primary... And now, it's also stuck with recovering shard ! In sync error list, I can find some "failed to sync bucket instance: (125) Operation canceled" errors I also tried to rewrite some buckets chown in those errors, but nothing changes. Strange, in those errors, the objects names are not real objects, example : "name": "replic_cfn_rec/cfb0047:aefd400 3-1866-4b16-b1b3-2f308075cd1c.20298566.4:11[0]" I wonder what is this ending ":10[0]". I also tried to remove stale instances, but nothing. I've still not retry a data sync init on secondary, perhaps I should, but the generated activity is impactfull. Can we reduce that resync activity priority ? ah, My primary cluster is on Reef 18.2.4, the secondary still on 18.2.2 (needs OS upgrade, Ubuntu 18.04). -- Gilles -- Gilles

1 0

(no subject)
by Gilles Mocellin 19 Dec '24

19 Dec '24

Hi there ! Being suspicious about replication between two clusters, I've done a radosgw-admin data sync init on the secondary zone. Since then, after a lot of activity, I'm stuck with recovering shards, nothing moves. Incremental sync still work. Wondering if I had a bad state also on the primary, I also did a data sync init on the primary... And now, it's also stuck with recovering shard ! In sync error list, I can find some "failed to sync bucket instance: (125) Operation canceled" errors I also tried to rewrite some buckets chown in those errors, but nothing changes. Strange, in those errors, the objects names are not real objects, example : "name": "replic_cfn_rec/cfb0047:aefd400 3-1866-4b16-b1b3-2f308075cd1c.20298566.4:11[0]" I wonder what is this ending ":10[0]". I also tried to remove stale instances, but nothing. I've still not retry a data sync init on secondary, perhaps I should, but the generated activity is impactfull. Can we reduce that resync activity priority ? ah, My primary cluster is on Reef 18.2.4, the secondary still on 18.2.2 (needs OS upgrade, Ubuntu 18.04). -- Gilles

1 0

January 2025 Ceph Meetup in Berlin, Germany and Frankfurt/Main, Germany - interested people welcome !
by Matthias Muench 19 Dec '24

19 Dec '24

Hi people in Germany, we run two meetup groups in Berlin and Frankfurt with meetings nearly quarterly. For each location, we provide also a quick introduction to Ceph before the main event. Currently, we have a new set of appointments. You can join us by RSVP to the events: - Ceph Frankfurt: 21.01.2025 in Frankfurt/Main, Germany Introduction: https://www.meetup.com/ceph-frankfurt/events/305161885/ Main meetup: https://www.meetup.com/ceph-frankfurt/events/305161680/ - Ceph Berlin: 20.01.2025 in Berlin, Germany Introduction: https://www.meetup.com/ceph-berlin/events/305161635/ Main meetup: https://www.meetup.com/ceph-berlin/events/305161618/ To get the invites also for any future meetings, one could also join the meetup groups: - Ceph Frankfurt: https://www.meetup.com/de-DE/ceph-frankfurt/ - Ceph Berlin: https://www.meetup.com/de-DE/ceph-berlin/ Happy to see you soon, -matt -- —————————————————— Matthias Muench Principal Specialist Solution Architect, OpenShift EMEA Storage Specialist matthias.muench(a)redhat.com Phone: +49-160-92654111 Red Hat GmbH Technopark II Werner-von-Siemens-Ring 12 85630 Grasbrunn Germany _______________________________________________________________________ Red Hat GmbH, Registered seat: Werner von Siemens Ring 12, D-85630 Grasbrunn, Germany Commercial register: Amtsgericht Muenchen/Munich, HRB 153243, Managing Directors: Ryan Barnhart, Charles Cachera, Michael O'Neill, Amy Ross

1 0

MONs not trimming
by Janek Bevendorff 19 Dec '24

19 Dec '24

Hi all, We moved our Ceph cluster to a new data centre about three months ago, which completely changed its physical topology. I changed the CRUSH map accordingly so that the CRUSH location matches the physical location again and the cluster has been rebalancing ever since. Due to capacity limits, the rebalancing requires rather frequent manual reweights so that individual OSDs don't run full. We started at around 65% of remapped PGs and are now down to 12% (it goes a little faster now that the bulk is done). Unfortunately, the MONs refuse to trim their stores while there are remapped PGs in the cluster, so their disk usage has increased gradually from 600MB to 17GB. I assume the rebalancing will finish before we run out of disk space, but the MON restart times have become unsustainably long and compacts now take up to 20 minutes or so. I'm a bit worried that at some point the MONs will start becoming unstable or worse: the stores may get corrupted. Is there anything that I can do to safely (!) get the MON stores back to a more manageable size even without all PGs being active+clean? I would like to forgo any potential disaster, especially before the holidays. Thanks Janek

6 13

CRC Bad Signature when using KRBD
by jsterr＠deckplosion.de 19 Dec '24

19 Dec '24

Hello Ceph Users, * Problem: we get the following errors when using krbd, we are using rbd for vms. * Workaround: by switching to librbd the errors disappear. * Software: ** Kernel: 6.8.8-2 (parameters: intel_iommu=on iommu=pt pcie_aspm.policy=performance) ** Ceph: 18.2.2 Description/Details: Errors from using krbd with ceph. Side-effects: [Wed Aug 21 03:04:17 2024] libceph: read_partial_message 0000000015af2284 data crc 1221767919 != exp. 282251377 [Wed Aug 21 03:04:17 2024] libceph: read_partial_message 0000000066b200ab data crc 3817026135 != exp. 3925662391 [Wed Aug 21 03:04:17 2024] libceph: osd15 (1)10.1.4.13:6836 bad crc/signature [Wed Aug 21 03:04:17 2024] libceph: osd13 (1)10.1.4.13:6809 bad crc/signature [Wed Aug 21 03:04:21 2024] libceph: read_partial_message 000000008a131738 data crc 2612835980 != exp. 917302924 [Wed Aug 21 03:04:21 2024] libceph: read_partial_message 000000005160776b data crc 2965872045 != exp. 563323792 [Wed Aug 21 03:04:21 2024] libceph: osd15 (1)10.1.4.13:6836 bad crc/signature [Wed Aug 21 03:04:21 2024] libceph: osd6 (1)10.1.4.12:6843 bad crc/signature [Wed Aug 21 03:06:44 2024] libceph: read_partial_message 000000007e548354 data crc 1265032637 != exp. 2426281931 [Wed Aug 21 03:06:44 2024] libceph: osd0 (1)10.1.4.11:6835 bad crc/signature [Wed Aug 21 03:06:44 2024] libceph: read_partial_message 000000009214d802 data crc 2596010853 != exp. 1221875667 [Wed Aug 21 03:06:44 2024] libceph: osd10 (1)10.1.4.12:6809 bad crc/signature [Wed Aug 21 03:06:47 2024] libceph: read_partial_message 000000000f9edc73 data crc 1326019705 != exp. 3079604517 [Wed Aug 21 03:06:47 2024] libceph: osd3 (1)10.1.4.11:6803 bad crc/signature [Wed Aug 21 03:06:50 2024] libceph: read_partial_message 000000004769da61 data crc 3421275194 != exp. 4183754554 [Wed Aug 21 03:06:50 2024] libceph: osd8 (1)10.1.4.12:6835 bad crc/signature [Wed Aug 21 03:06:51 2024] libceph: read_partial_message 0000000044db9a59 data crc 2603270708 != exp. 4150529351 [Wed Aug 21 03:06:51 2024] libceph: osd14 (1)10.1.4.13:6806 bad crc/signature Description/Details 2: vms get problems with buffer i/o errors on rbd-backed virtual disks: Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#211 timing out command, waited 180s Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#39 timing out command, waited 180s Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#211 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=885s Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#39 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=847s Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#211 Sense Key: Aborted Command [current] Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#211 Add. Sense: I/O process terminated Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#39 Sense Key: Aborted Command [current] Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#39 Add. Sense: I/0 process terminated Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#211 CDB: Write(10) 2a 00 34 87 48 08 00 00 08 00 Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#39 CDB: Write (10) 2a 00 34 81 52 70 00 00 58 00 Aug 24 03:16:01 de-vlix-dbix-01 kernel: I/O error, dev sdb, sector 881281032 op 0x1: (WRITE) flags 0x800 phys_seg 1 prio class 0 Aug 24 03:16:01 de-vlix-dbix-01 kernel: I/O error, dev sdb, sector 880890480 op 0x1: (WRITE) flags 0x103000 phys_seg 11 prio class 0 Aug 24 03:16:01 de-vlix-dbix-01 kernel: EXT4-fs warning (device sdb1): ext4_end_bio:343: I/0 error 10 writing to inode 27525908 starting Aug 24 03:16:01 de-vlix-dbix-01 kernel: Buffer I/O error on dev sdbl, logical block 110111054, lost async page write Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#212 timing out command, waited 180s Aug 24 03:16:01 de-vlix-dbix-01 kernel: buffer_io_error: 21 callbacks suppressed Aug 24 03:16:01 de-vlix-dbix-01 kernel: Buffer 1/0 error on device sdbl, logical block 110159873 Aug 24 03:16:01 de-vlix-dbix-01 kernel: Buffer 1/0 error on dev sdbl, logical block 110111055, lost async page write Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#212 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=875s Aug 24 03:16:01 de-vlix-dbix-01 kernel: Buffer I/O error on dev sdb1, logical block 110111056, lost async page write Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#212 Sense Key: Aborted Command [current] Aug 24 03:16:01 de-vlix-dbix-01 kernel: Buffer I/O error on dev sdbl, logical block 110111057, lost async page write Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#212 Add. Sense: I/0 process terminated Aug 24 03:16:01 de-vlix-dbix-01 kernel: Buffer I/O error on dev sdb1, logical block 110111058, lost async page write Aug 24 03:16:01 de-vlix-dbix-01 kernel: sd 0:0:0:1: [sdb] tag#212 CDB: Write(10) 2a 00 34 87 48 10 00 00 08 00 Aug 24 03:16:01 de-vlix-dbix-01 kernel: Buffer I/O error on dev sdbl, logical block 110111059, lost async page write Aug 24 03:16:01 de-vlix-dbix-01 kernel: I/O error, dev sdb, sector 881281040 op 0x1: (WRITE) flags 0x800 phys_seg 1 prio class 0 Aug 24 03:16:01 de-vlix-dbix-01 kernel: Buffer I/O error on dev sdbl, logical block 110111060, lost async page write Thanks for helping out. Greetings Jonas

3 4

Squid: deep scrub issues
by Laimis Juzeliūnas 19 Dec '24

19 Dec '24

Hello Ceph community, Wanted to highlight one observation and gather any Squid users having similar experiences. Since upgrading to 19.2.0 (from 18.4.0) we have observed that pg deep scrubbing times have drastically increased. Some pgs take 2-5 days to complete deep scrubbing while others increase to 20+ days. This causes the deep scrubbing queue to fill up and the cluster almost constantly has 'pgs not deep-scrubbed in time' alerts. We have on average 67 pgs/osd: running on 15TB hdd disks this results in 200GB-ish pgs. While fairly large - these pgs did not cause such increase in deep scrubs when on Reef. "ceph pg dump | grep 'deep scrubbing for'" will always have a few entries of quite morbid scrubs like the following: 7.3e 121289 0 0 0 0 225333247207 0 0 127 0 127 active+clean+scrubbing+deep 2024-11-13T09:37:42.549418+0000 490179'5220664 490179:23902923 [268,27,122] 268 [268,27,122] 268 483850'5203141 2024-11-02T11:33:57.835277+0000 472713'5197481 2024-10-11T04:30:00.639763+0000 0 21873 deep scrubbing for 1169147s 34.247 62618 0 0 0 0 179797964677 0 0 101 50 101 active+clean+scrubbing+deep 2024-11-05T06:27:52.288785+0000 490179'22729571 490179:80672442 [34,97,25] 34 [34,97,25] 34 481331'22436869 2024-10-23T16:06:50.092439+0000 471395'22289914 2024-10-07T19:29:26.115047+0000 0 204864 deep scrubbing for 1871733s Not pointing any fingers but Squid release had "better scrub scheduling" announced. Though this is not scheduling directly, but maybe this change had any impact causing such behaviour? Scrubbing configurations: ceph config get osd | grep scrub global advanced osd_deep_scrub_interval 2678400.000000 global advanced osd_deep_scrub_large_omap_object_key_threshold 500000 global advanced osd_max_scrubs 5 global advanced osd_scrub_auto_repair true global advanced osd_scrub_max_interval 2678400.000000 global advanced osd_scrub_min_interval 172800.000000 Cluster details (backfilling expected and caused by some manual reweights): cluster: id: 96df99f6-fc1a-11ea-90a4-6cb3113cb732 health: HEALTH_WARN 24 pgs not deep-scrubbed in time services: mon: 5 daemons, quorum ceph-node004,ceph-node003,ceph-node001,ceph-node005,ceph-node002 (age 4d) mgr: ceph-node001.hgythj(active, since 11d), standbys: ceph-node002.jphtvg mds: 20/20 daemons up, 12 standby osd: 384 osds: 384 up (since 25h), 384 in (since 5d); 5 remapped pgs rbd-mirror: 2 daemons active (2 hosts) rgw: 64 daemons active (32 hosts, 1 zones) data: volumes: 1/1 healthy pools: 14 pools, 8681 pgs objects: 758.42M objects, 1.5 PiB usage: 4.6 PiB used, 1.1 PiB / 5.7 PiB avail pgs: 275177/2275254543 objects misplaced (0.012%) 6807 active+clean 989 active+clean+scrubbing+deep 880 active+clean+scrubbing 5 active+remapped+backfilling io: client: 37 MiB/s rd, 59 MiB/s wr, 1.72k op/s rd, 439 op/s wr recovery: 70 MiB/s, 38 objects/s One thread of other users experiencing same 19.2.0 prolonged deep scrub issues: https://www.reddit.com/r/ceph/comments/1guynak/strange_issue_where_scrubdee… Any hints or help would be greately appreciated! Thanks in advance, Laimis J. laimis.juzeliunas(a)oxylabs.io

6 13

Re: Squid Manager Daemon: balancer crashing orchestrator and dashboard
by Alexander Closs 18 Dec '24

18 Dec '24

Just chiming in to say this also affected our cluster, same symptoms and a temporary fix of disabling the balancer. Happy to add my cluster's logs to the issue, though I suspect they'll look the same as Laimis' cluster. -Alex MIT CSAIL

7 7

pgs not deep-scrubbed in time
by Jan Kasprzak 18 Dec '24

18 Dec '24

Hello, Ceph users, a question/problem related to deep scrubbing: I have a HDD-based Ceph 18 cluster currently with 34 osds and 600-ish pgs. In order to avoid latency peaks which apparently correlate with HDD being 100 % busy for several hours during a deep scrub, I wanted to relax the scrubbing frequency and concurrency. Six days ago I have modified the following config parameters: ceph config set osd osd_scrub_max_interval 2592000 # was 604800 ceph config set osd osd_deep_scrub_interval 2592000 # was 604800 ceph config set osd osd_max_scrubs 1 # was 3 The intervals are four times longer and max_scrubs is three times lower, so the total scrubbing load should _decrease_. However, just several hours after the config change, my cluster went to HEALTH_WARN with "XX pgs not deep-scrubbed in time". I thought this was something temporary, but six days later, the number of pgs not scrubbed in time still grows - now it says 58. In "ceph -s" there are about 6-8 pgs active+clean+scrubbing, and 4-6 active+clean+scrubbing+deep all the time, so scrubbing still happens. According to "ceph pg dump" a deep scrub of a pg takes about 9000 seconds. It seems all pgs are scheduled to be scrubbed in the next two days: # for i in `seq 18 24`; do echo -n "2024-12-$i "; ceph pg dump 2>/dev/null | grep -c "scheduled @ 2024-12-$i"; done 2024-12-18 152 2024-12-19 422 2024-12-20 7 2024-12-21 0 2024-12-22 0 2024-12-23 0 2024-12-24 0 The positive thing is that latency-wise it helped: with at most one pg being deep-scrubbed per OSD at any time, the utilization of the HDD never grows near 100 %, it stays at ~60 % when deep scrubbing is in progress on that OSD. Is there any other config parameter which I should modify together with the above three parameters? Thanks! -Yenya -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall

2 2