- ceph-users - lists.ceph.io

Severe Latency Issues in Ceph Cluster
by Ramin Najjarbashi 04 Mar '25

04 Mar '25

Hi We are currently facing severe latency issues in our Ceph cluster, particularly affecting read and write operations. At times, write operations completely stall, leading to significant service degradation. Below is a detailed breakdown of the issue, our observations, and the mitigation steps we have taken so far. We would greatly appreciate any insights or suggestions. Incident Timeline & Observations: Users reported extreme slowness in write and read operations. Network Issues: Packet drops and a high number of TCP retransmits were identified. TCP Connection Delays: Many TCP connections were stuck in the TIME-WAIT state. Initial Suspicions: We first suspected high list_bucket requests on RGW, but log analysis disproved this. Resharding Efforts: Some high-latency buckets were resharded, but this had minimal impact. HTTP 503 Errors: Indicating request overload and service instability. Earlier Related Incidents: Network Congestion: Routing issues previously caused congestion in the Provision network. RGW Downtime: Several RGWs went down, requiring removal from HAProxy. NIC Troubleshooting: A faulty NIC was replaced, but errors persisted. Troubleshooting & Mitigation Attempts: Network & Traffic Management: Investigated logs and blocked unnecessary list_bucket requests. Disabled HAProxy to isolate Ceph internal traffic → Ceph internals worked fine, but external requests still caused issues. RGW-Specific Troubleshooting: Restarted RGW instances → Temporary 20-minute improvement, but latency returned. Found an RGW node with incomplete configurations → Fixed, but issue resurfaced. TCP & Network Investigations: Adjusted tcp_mem settings → No substantial improvement. packet analysis showed a high number of duplicate packets (DUP ACKs). Observed high TCP retransmit errors and many TIME-WAIT connections. Assistance Requested: Could high TCP retransmits & TIME-WAIT connections indicate a deeper network issue affecting RGW writes? Are there recommended debugging techniques for tracing TCP-related issues in a Ceph RGW environment? Would tweaking KeepAlived settings help in case of incorrect VIP failover behavior? Could an underlying Ceph metadata issue be disproportionately affecting write operations? Any guidance, recommendations, or debugging steps would be immensely helpful. Thank you for your time and support! Best regards, Ramin

7 8

When 18.2.5 will be released?
by wojiaowugen 04 Mar '25

04 Mar '25

Hi, everyone, It's an honor to ask questions here. can I ask when 18.2.5 will be released?

2 1

Replace OSD while cluster is recovering?
by Gustavo Garcia Rondina 04 Mar '25

04 Mar '25

Hello list, We have a Ceph cluster (17.2.6 quincy) with 2 admin nodes and 6 storage nodes, each storage node connected to a JBOD enclosure. Each enclosure houses 28 HDD disks of 18 TB size, totaling 168 OSDs. The pool that houses the majority of the data is erasure-coded (4+2). We have recently had one disk failure, which brought one OSD down: # ceph osd tree | grep down 2 hdd 16.49579 osd.2 down 0 1.00000 This OSD is out of the cluster, but we haven't replaced it physically yet. The problem that we are facing is that the cluster was not in the best shape when this OSD failed. Currently we have the following: ################################################ cluster: id: <redacted> health: HEALTH_ERR 1026 scrub errors Possible data damage: 18 pgs inconsistent 2137 pgs not deep-scrubbed in time 2137 pgs not scrubbed in time services: mon: 5 daemons, quorum xyz-admin1,xyz-admin2,xyz-osd1,xyz-osd2,xyz-osd3 (age 17M) mgr: xyz-admin2.sipadf(active, since 17M), standbys: xyz-admin1.nwaovh mds: 2/2 daemons up, 2 standby osd: 168 osds: 167 up (since 44h), 167 in (since 6w); 220 remapped pgs data: volumes: 2/2 healthy pools: 9 pools, 2137 pgs objects: 448.54M objects, 1.0 PiB usage: 1.6 PiB used, 1.1 PiB / 2.7 PiB avail pgs: 134404830/2676514497 objects misplaced (5.022%) 1902 active+clean 191 active+remapped+backfilling 26 active+remapped+backfill_wait 15 active+clean+inconsistent 2 active+remapped+inconsistent+backfilling 1 active+remapped+inconsistent+backfill_wait io: recovery: 597 MiB/s, 252 objects/s progress: Global Recovery Event (6w) [=========================...] (remaining: 5d) ################################################ I have noticed the number of active+clean increasing (was ~1750 two days ago), and objects misplaced very slowly decreasing. My question is, should I wait until recovery is complete, then repair the 18 damaged pg, and only then replace the disk? My thinking is that replacing the disk will trigger more backfilling which will slow down the recovering even more. Another question, should I disable scrubbing while the recovery is not finalized? Thank you for any insights you may be able to provide! - Gustavo

4 5

March 3rd Ceph Steering Committee Meeting Notes
by Laura Flores 03 Mar '25

03 Mar '25

Hi all, Here is a summary of what was discussed at today's CSC Meeting: - Deciding on the best way for a community vote on the name for U release - Tentacle Kickoff PR merged - https://tracker.ceph.com/issues/69953 being investigated by fs team - v18.2.5 status: Working on including a few outstanding fixes Thanks, Laura -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

1 0

[Cephfs] Can't get snapshot under a subvolume
by ceph＠gurkan.in 02 Mar '25

02 Mar '25

Hello all, We're getting a "Operation not permitted" error while trying to create a snapshot on the client. It is somehow related to previously-asked Pacific issue mentioned here: https://www.spinics.net/lists/ceph-users/msg67908.html We are on squid (19.2.1) and the given workaround seems to be only temporary. Anyway, here are some details about the issue: Cephx side: ``` client.mount-update-production key: *** caps: [mds] allow rws fsname=production-cephfs path=/volumes/_nogroup/update-production caps: [mon] allow r fsname=production-cephfs caps: [osd] allow rw tag cephfs data=production-cephfs ``` FS flags: ``` ~> sudo ceph fs lsflags production-cephfs joinable allow_snaps allow_multimds_snaps ``` I got the path via following command: ``` ~> sudo ceph fs subvolume getpath production-cephfs update-production /volumes/_nogroup/update-production/e155ba30-d201-4653-be3f-86533324ee5f ``` And mounted to client with mentioned key: ``` ~> mount | grep ceph mount-update-production@00000000-0000-0000-0000-000000000000.production-cephfs=/volumes/_nogroup/update-production/e155ba30-d201-4653-be3f-86533324ee5f on /opt/aptly/.aptly type ceph (rw,noatime,name=mount-update-production,secret=<hidden>,acl,mon_addr=[multiple mon addresses here]) ``` Everything works flawlessly. Yet, when we want to get a snapshot: ``` /opt/aptly/.aptly/pool/9c/0b ~> sudo mkdir .snap/test mkdir: cannot create directory ‘.snap/test’: Operation not permitted ``` If I mount the whole FS with admin rights and run `setfattr -n ceph.dir.subvolume -v 0` for _all_ directories of following path one by one: /volumes/_nogroup/update-production/e155ba30-d201-4653-be3f-86533324ee5f, then it is solving the issue. But it does not last; not sure what is triggering it again, but if I try snapshotting anything next day, getting the same "Operation not permitted" error again. Is there something I can try, or am I missing something obvious? Thanks, Gürkan

3 5

Schrödinger's Server
by Tim Holloway 01 Mar '25

01 Mar '25

Ack. Another fine mess. I was trying to clean things up and the process of tossing around OSD's kept getting me reports of slow responses and hanging PG operations. This is Ceph Pacific, by the way. I found a deprecated server that claimed to have an OSD even though it didn't show in either "ceph osd tree" or the dashboard OSD list. I suspect that a lot of the grief came from it attempting to use resources that weren't always seen as resources. I shut down the server's OSD (removed the daemon using ceph orch), then foolishly deleted the server from the inventory without doing a drain first. Now cephadmin hates me (key not found), and there are still an MDS and MON listed as ceph orch ls daemons even after I powered the host off. I cannot do a ceph orch daemon delete because there's no longer an IP address available to the daemon delete, and I cannot clear the cephadmin queue: [ERR] MGR_MODULE_ERROR: Module 'cephadm' has failed: 'ceph07.internal.mousetech.com' Any suggestions?

3 9

Reef: draining host with mclock quicker than expected
by Eugen Block 01 Mar '25

01 Mar '25

Hi *, for a change, I just wanted to share some positive experience with mclock. On a customer cluster (recently upgraded to Reef), I drained one host with 23 HDD OSDs (1,1 TB size). And to my surprise, it took less than an hour until all PGs were backfilled. The OSDs weren't too full, I believe something around 1,6 TB had to be moved away from this host. But the recovery rates were quite nice, especially compared to what other users on Reef have been reporting. I saw values between several hundred MB to 2 or 3 GB per second. Most of the configs are default, especially with regards to mclock there haven't been any changes. Although the general recommendation is still to switch back to wpq, I haven't done that yet in this cluster. I usually wait until I encounter any issue pointing to mclock. But since this was a positive experience, I'll keep mclock enabled until otherwise required. Have a nice weekend! Eugen

1 0

Request for Assistance: OSDS Stability Issues Post-Upgrade to Ceph Quincy 17.2.8
by Aref Akhtari 28 Feb '25

28 Feb '25

Dear Ceph Community, I hope this message finds you well. I am reaching out to seek assistance regarding a stability issue we have encountered after upgrading our Ceph cluster from version Pacific 16.2.3 to Quincy 17.2.8. Following the upgrade, we have observed that several of our Object Storage Daemons (OSDs) are experiencing erratic behavior. These OSDs frequently exhibit a "flapping" condition, where they unexpectedly go down and then come back up. This issue has predominantly affected the recently upgraded OSDs within the cluster. Upon reviewing the logs from the affected OSDs, we encountered the following messages: > 2025-02-03T08:34:09.769+0000 7f0f11390780 -1 > bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 > 2025-02-03T08:38:22.920+0000 7feb9dd44780 -1 > bluestore::NCB::__restore_allocator::No Valid allocation info on disk > (empty file) In an attempt to resolve the issue, we executed the ceph-bluestore-tool fsck and repair commands. Although these commands executed successfully, they did not rectify the problem at hand. Additionally, we have captured the following crash information from the ceph logs: > ceph crash info > 2025-02-03T09:19:08.749233Z_9e2800fb-77f6-46cb-8087-203ea15a2039 > { > "assert_condition": "log.t.seq == log.seq_live", > "assert_file": > "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/17.2.8/rpm/el9/BUILD/c > eph-17.2.8/src/os/bluestore/BlueFS.cc", > "assert_func": "uint64_t BlueFS::_log_advance_seq()", > "assert_line": 3029, > "assert_msg": > "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/17.2.8/rpm/el9/BUILD/ce > ph-17.2.8/src/os/bluestore/BlueFS.cc: In function 'uint64_t > BlueFS::_log_advance_seq()' thread 7ff983564640 time > 2025-02-03T09:19:08.738781+0000\n/home/jenkins-build/build/workspace/ceph-bu > ild/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/17.2.8/rpm/el9/BUILD/ceph-17.2.8/src/os/bluestore/BlueFS.cc: > 3029: FAILED ceph_assert > (log.t.seq == log.seq_live)\n", > "assert_thread_name": "bstore_kv_sync", > "backtrace": [ > "/lib64/libc.so.6(+0x3e730) [0x7ff9930f5730]", > "/lib64/libc.so.6(+0x8bbdc) [0x7ff993142bdc]", > "raise()", > "abort()", > "(ceph::__ceph_assert_fail(char const*, char const*, int, char > const*)+0x179) [0x55882dfb7fdd]", > "/usr/bin/ceph-osd(+0x36b13e) [0x55882dfb813e]", > "/usr/bin/ceph-osd(+0x9cff3b) [0x55882e61cf3b]", > "(BlueFS::_flush_and_sync_log_jump_D(unsigned long)+0x4e) > [0x55882e6291ee]", > "(BlueFS::_compact_log_async_LD_LNF_D()+0x59b) [0x55882e62e8fb]", > "/usr/bin/ceph-osd(+0x9f2b15) [0x55882e63fb15]", > "(BlueFS::fsync(BlueFS::FileWriter*)+0x1b9) [0x55882e631989]", > "/usr/bin/ceph-osd(+0x9f4889) [0x55882e641889]", > "/usr/bin/ceph-osd(+0xd74cd5) [0x55882e9c1cd5]", > "(rocksdb::WritableFileWriter::SyncInternal(bool)+0x483) > [0x55882eade393]", > "(rocksdb::WritableFileWriter::Sync(bool)+0x120) [0x55882eae0b60]", > "(rocksdb::DBImpl::WriteToWAL(rocksdb::WriteThread::WriteGroup > const&, rocksdb::log::Writer*, unsigned long*, bool, bool, unsigned > long)+0x337) [0x55882ea00ab7]", > "(rocksdb::DBImpl::WriteImpl(rocksdb::WriteOptions const&, > rocksdb::WriteBatch*, rocksdb::WriteCallback*, unsigned long*, unsigned > long, bool, unsigned long*, unsigned long, rocksdb > ::PreReleaseCallback*)+0x1935) [0x55882ea07675]", > "(rocksdb::DBImpl::Write(rocksdb::WriteOptions const&, > rocksdb::WriteBatch*)+0x35) [0x55882ea077c5]", > "(RocksDBStore::submit_common(rocksdb::WriteOptions&, > std::shared_ptr<KeyValueDB::TransactionImpl>)+0x83) [0x55882e992593]", > "(RocksDBStore::submit_transaction_sync(std::shared_ptr<KeyValueDB::TransactionImpl>)+0x99) > [0x55882e992ee9]", > "(BlueStore::_kv_sync_thread()+0xf64) [0x55882e578e24]", > "/usr/bin/ceph-osd(+0x8afb81) [0x55882e4fcb81]", > "/lib64/libc.so.6(+0x89e92) [0x7ff993140e92]", > "/lib64/libc.so.6(+0x10ef20) [0x7ff9931c5f20]" > ], > "ceph_version": "17.2.8", > "crash_id": > "2025-02-03T09:19:08.749233Z_9e2800fb-77f6-46cb-8087-203ea15a2039", > "entity_name": "osd.211", > "os_id": "centos", > "os_name": "CentOS Stream", > "os_version": "9", > "os_version_id": "9", > "process_name": "ceph-osd", > "stack_sig": > "ba90de24e2beba9c6a75249a4cce7c533987ca5127cfba5b835a3456174d6080", > "timestamp": "2025-02-03T09:19:08.749233Z", > "utsname_hostname": "afra-osd18", > "utsname_machine": "x86_64", > "utsname_release": "5.15.0-119-generic", > "utsname_sysname": "Linux", > "utsname_version": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024" > } The above crash logs highlight an assertion failure in the BlueFS component, specifically within the function BlueFS::_log_advance_seq(). Despite our efforts to analyze and resolve the issue, we have reached an impasse. For completeness, we have verified the health of our disks using smartctl, and they have all been deemed healthy. We kindly request guidance from the community on how to address this issue or any recommended steps for deeper diagnostics. We appreciate your support and expertise during this troubleshooting process. Thank you for your attention and assistance. Best regards, Aref Akhtari

2 1

Replace OSD while cluster is recovering?
by grondina＠uchicago.edu 28 Feb '25

28 Feb '25

Hello list, We have a Ceph cluster (v17.2.6 quincy) with 3 admin nodes and 6 storage nodes, each connected to a JBOD enclosure. Each enclosure houses 28 HDD disks of size 18 TB, totaling 168 OSDs. The pool that houses the majority of the data is erasure-coded (4+2). We have recently had one disk failure, which brought one OSD down: # ceph osd tree | grep down 2 hdd 16.49579 osd.2 down 0 1.00000 This OSD is out of the cluster, but we haven't replaced it physically yet. The problem that we are facing is that the cluster was not in the best shape when this OSD failed. Currently we have the following: ################ cluster: id: <redacted> health: HEALTH_ERR 1026 scrub errors Possible data damage: 18 pgs inconsistent 2122 pgs not deep-scrubbed in time 2122 pgs not scrubbed in time services: mon: 5 daemons, quorum xyz-admin1,xyz-admin2,xyz-osd1,xyz-osd2,xyz-osd3 (age 17M) mgr: xyz-admin2.sipadf(active, since 17M), standbys: xyz-admin1.nwaovh mds: 2/2 daemons up, 2 standby osd: 168 osds: 167 up (since 40h), 167 in (since 6w); 226 remapped pgs data: volumes: 2/2 healthy pools: 9 pools, 2122 pgs objects: 448.54M objects, 1.0 PiB usage: 1.6 PiB used, 1.1 PiB / 2.7 PiB avail pgs: 133905796/2676514497 objects misplaced (5.003%) 1880 active+clean 201 active+remapped+backfilling 23 active+remapped+backfill_wait 16 active+clean+inconsistent 1 active+remapped+inconsistent+backfill_wait 1 active+remapped+inconsistent+backfilling io: recovery: 703 MiB/s, 281 objects/s progress: Global Recovery Event (6w) [=========================...] (remaining: 5d) ################ I have noticed the number of active+clean increasing, and objects misplaced very slowly decreasing. My question is, should I wait until recovery is complete, repair the 18 damaged pg, and then replace the disk? My thinking is that replacing the disk will trigger more backfilling which will slow down the recovering even more. Another question, should I disable scrubbing while the recovery is not finalized? Thank you for any insights you may be able to provide! - Gustavo

1 0

Squid: Grafana host-details shows total number of OSDs
by Eugen Block 28 Feb '25

28 Feb '25

Hi, as always, I looked briefly into the tracker but couldn't find anything related (yet). Not sure if this has been reported or not, but after I upgraded a customer cluster to Squid (19.2.0), Grafana was also upgraded to version 9.4.12. And now the host details show all 126 OSDs for each host instead of the number per host. The respecitve query is simply: "expr": "count(sum by (ceph_daemon) (ceph_osd_metadata{cluster=~\"$cluster\", }))", To me it looks like there's something missing after the comma, this query for example shows the actual number of OSDs on that host: count(sum by (ceph_daemon) (ceph_osd_metadata{cluster=~"9bafee50-775c-11eb-9ed6-4c52620b498e",hostname="ses01"})) If this has been already reported, could anyone point me to the according tracker? If not, I'd be happy to create a new one. Thanks! Eugen

2 2