- ceph-users - lists.ceph.io

OSD crc errors: Faulty SSD?
by Roland Giesler 27 Aug '25

27 Aug '25

I have relatively new Samsung Enterprise NVMe in a node that is generating the following error: 2025-08-26T15:56:43.870+0200 7fe8ac968700 0 bad crc in data 3326000616 != exp 1246001655 fromv1:192.168.131.4:0/1799093090 2025-08-26T16:03:54.757+0200 7fe8ad96a700 0 bad crc in data 3195468789 != exp 4291467912 fromv1:192.168.131.3:0/315398791 2025-08-26T16:17:34.160+0200 7fe8ad96a700 0 bad crc in data 1471079732 != exp 1408597599 fromv1:192.168.131.3:0/315398791 2025-08-26T16:33:34.035+0200 7fe8ad96a700 0 bad crc in data 724234454 != exp 3110238891 fromv1:192.168.131.3:0/315398791 2025-08-26T16:36:34.265+0200 7fe8ad96a700 0 bad crc in data 96649884 != exp 3724606899 fromv1:192.168.131.3:0/315398791 2025-08-26T16:40:34.395+0200 7fe8ad96a700 0 bad crc in data 1554359919 != exp 1420125995 fromv1:192.168.131.3:0/315398791 2025-08-26T16:54:18.323+0200 7fe8ad169700 0 bad crc in data 362320144 != exp 1850249930 fromv1:192.168.131.1:0/1316652062 This is ceph osd.40. More details from the log: 2025-08-26T17:00:15.013+0200 7fe8a06fe700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1756220415016487, "job": 447787, "event": "table_file_deletion", "file_number": 389377} 2025-08-26T17:00:15.013+0200 7fe8a06fe700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1756220415016839, "job": 447787, "event": "table_file_deletion", "file_number": 389359} 2025-08-26T17:00:15.013+0200 7fe8a06fe700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1756220415017245, "job": 447787, "event": "table_file_deletion", "file_number": 389342} 2025-08-26T17:00:15.013+0200 7fe8a06fe700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1756220415017633, "job": 447787, "event": "table_file_deletion", "file_number": 389276} 2025-08-26T17:00:15.041+0200 7fe8a06fe700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1756220415047481, "job": 447787, "event": "table_file_deletion", "file_number": 389254} 2025-08-26T17:00:15.045+0200 7fe8a06fe700 4 rocksdb: (Original Log Time 2025/08/26-17:00:15.047592) [db/db_impl/db_impl_compaction_flush.cc:2818] Compaction nothing to do 2025-08-26T17:04:35.776+0200 7fe899ee2700 4 rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2025-08-26T17:04:35.776+0200 7fe899ee2700 4 rocksdb: [db/db_impl/db_impl.cc:903] ** DB Stats ** Uptime(secs): 24012063.5 total, 600.0 interval Cumulative writes: 10G writes, 37G keys, 10G commit groups, 1.0 writes per commit group, ingest: 19073.99 GB, 0.81 MB/s Cumulative WAL: 10G writes, 4860M syncs, 2.14 writes per sync, written: 19073.99 GB, 0.81 MB/s Cumulative stall: 00:00:0.000H:M:S, 0.0 percent Interval writes: 159K writes, 546K keys, 159K commit groups, 1.0 writes per commit group, ingest: 202.96 MB, 0.34 MB/s Interval WAL: 159K writes, 76K syncs, 2.10 writes per sync, written: 0.20 MB, 0.34 MB/s Interval stall: 00:00:0.000H:M:S, 0.0 percent ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 0/0 0.00 KB 0.0 0.0 0.0 0.0 2.4 2.4 0.0 1.0 0.0 25.1 98.33 49.99 19074 0.005 0 0 L1 2/0 137.69 MB 1.0 299.7 2.4 297.3 297.9 0.7 0.0 123.5 59.6 59.2 5152.28 4745.40 4769 1.080 8130M 47M L2 7/0 410.80 MB 0.2 1.3 0.2 1.1 1.2 0.1 0.3 5.8 79.4 72.1 17.00 15.59 4 4.250 32M 3822K Sum 9/0 548.48 MB 0.0 301.0 2.6 298.4 301.5 3.1 0.3 125.0 58.5 58.6 5267.61 4810.98 23847 0.221 8162M 51M Int 0/0 0.00 KB 0.0 0.1 0.0 0.1 0.1 0.0 0.0 3194.3 48.0 47.9 2.87 2.72 2 1.437 5633K 2173 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 301.0 2.6 298.4 299.1 0.7 0.0 0.0 59.6 59.3 5169.28 4761.00 4773 1.083 8162M 51M High 0/0 0.00 KB 0.0 0.0 0.0 0.0 2.4 2.4 0.0 0.0 0.0 25.1 98.33 49.99 19073 0.005 0 0 User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 32.8 0.00 0.00 1 0.001 0 0 This gets repeated many times until finally all seems well again. This a few minutes later the problem repeats itself. Is this a faulty ssd causing this? Environment: # pveversion: pve-manager/7.4-19/f98bf8d4 (running kernel: 5.15.131-2-pve) # ceph version 17.2.7 (29dffbfe59476a6bb5363cf5cc629089b25654e3) quincy (stable)

3 2

Preventing device zapping while replacing faulty drive (Squid 19.2.2)
by Dmitrijs Demidovs 27 Aug '25

27 Aug '25

Hi List! We have Squid 19.2.2. It is cephadm/docker based deployment (recently upgraded from Pacific 16.2.15). We are using 8 SAS drives for Block and 2 SSD drives for DB on every OSD Host. Problem: One of the SAS Block drives failed on OSD Host and we need to replace it. When our Ceph cluster was running on Pacific, we usually performed drive replacement using this steps: 1) Edit -> MARK osd.xx as OUT [re-balancing starts, wait until it is completed] 2) Edit -> MARK osd.xx as DOWN 3) Edit -> DELETE osd.xx [put check on "Preserve OSD ID" and "Yes, I am sure"] 4) Edit -> DESTROY osd.xx 5) Edit -> PURGE osd.xx [re-balancing starts, wait until it is completed] 6) Set "noout" and "norebalance" flags. Put OSD Host in maintenance mode. Shutdown OSD Host. Replace failed drive. Start OSD Host. 7) Wipe old DB Logical Volume (LV) [dd if=/dev/zero of=/dev/ceph-xxx/osd-db-xxx bs=1M count=10 conv=fsync]. 8) Wipe new Block disk. Destroy old DB LV. Wait for automatic discovery and creation of new osd.xx instance. In Pacific 16.2.15, after execution of PURGE command, Ceph just removed old osd.xx instance from cluster without deletion/zapping of DB and Block LVs. Now in Squid 19.2.2 we see what Ceph behaves differently. Execution of step 3 (Edit -> DELETE osd.xx) automatically executes DESTROY and PURGE, and after that Ceph automatically performs zapping and deletion of DB LV and Block LV! And after that it's automatic discovery finds "clean" SAS disk + free space on SSD drive and happily forms new osd.xx instance form failed drive what we need to replace :) Questions: 1) What is correct procedure how-to replace failed Block drive in Ceph Squid 19.2.2? 2) Is it possible to disable Zapping? 3) Is it possible to temporary disable automatic discovery of new drives for OSD service? P.S. Here is our Pacement Specification for OSD service: [ceph: root@ceph-mon12 /]# ceph orch ls osd osd.dashboard-admin-1633624229976 --export service_type: osd service_id: dashboard-admin-1633624229976 service_name: osd.dashboard-admin-1633624229976 placement: host_pattern: '*' spec: data_devices: rotational: true db_devices: rotational: false db_slots: 4 filter_logic: AND objectstore: bluestore Logs from Ceph: 12/8/25 08:54 AM [INF] Cluster is now healthy 12/8/25 08:54 AM [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 11/121226847 objects degraded (0.000%), 1 pg degraded) 12/8/25 08:54 AM [WRN] Health check update: Degraded data redundancy: 11/121226847 objects degraded (0.000%), 1 pg degraded (PG_DEGRADED) 12/8/25 08:54 AM [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 12/8/25 08:54 AM [WRN] Health check failed: Degraded data redundancy: 12/121226847 objects degraded (0.000%), 2 pgs degraded (PG_DEGRADED) 12/8/25 08:54 AM [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 12/8/25 08:54 AM [INF] osd.33 [v2:10.10.10.105:6824/297218036,v1:10.10.10.105:6825/297218036] boot 12/8/25 08:54 AM [WRN] OSD bench result of 1909.305644 IOPS is not within the threshold limit range of 50.000000 IOPS and 500.000000 IOPS for osd.33. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd]. 12/8/25 08:53 AM [INF] Deploying daemon osd.33 on ceph-osd15 12/8/25 08:53 AM [INF] Found osd claims for drivegroup dashboard-admin-1633624229976 -> {'ceph-osd15': ['33']} 12/8/25 08:53 AM [INF] Found osd claims -> {'ceph-osd15': ['33']} 12/8/25 08:53 AM [INF] Detected new or changed devices on ceph-osd15 12/8/25 08:52 AM [INF] Successfully zapped devices for osd.33 on ceph-osd15 12/8/25 08:52 AM [INF] Zapping devices for osd.33 on ceph-osd15 12/8/25 08:52 AM [INF] Successfully destroyed old osd.33 on ceph-osd15; ready for replacement 12/8/25 08:52 AM [INF] Successfully removed osd.33 on ceph-osd15 12/8/25 08:52 AM [INF] Removing key for osd.33 12/8/25 08:52 AM [INF] Removing daemon osd.33 from ceph-osd15 -- ports []

4 3

cannot reach telemetry.ceph.com -> HEALTH_WARN after cluster upgrade to 19.2.3
by Björn Lässig 27 Aug '25

27 Aug '25

Hi, Does anyone know who to contact to report problems with telemetry.ceph.com? After a reasonably successful update of my cluster, I received a HEALTH_WARN [Telemetry requires re-opt-in] even though I had already re-approved telemetry. While trying to understand the problem, I stumbled across the fact that although the host telemetry.ceph.com has an IPv4 and an IPv6 address, I cannot reach it because IPv6 only has a dropping firewall or something similar attached to it. Now, I can configure a proxy that also reaches the legacy Internet, but it also tried to reach IPv6 first, so that currently no IPv6-enabled Ceph cluster should be visible in the telemetry. My workaround would now be to overlay telemetry.ceph.com on the DNS cache... but that's just a hack. ----- raw shell output for reproduction ----- > host telemetry.ceph.com telemetry.ceph.com has address 8.43.84.137 telemetry.ceph.com has IPv6 address 2620:52:3:3:201a:4aff:feab:8409 > telnet 2620:52:3:3:201a:4aff:feab:8409 80 Trying 2620:52:3:3:201a:4aff:feab:8409... telnet: Unable to connect to remote host: Connection timed out > telnet 8.43.84.137 80 Trying 8.43.84.137... Connected to 8.43.84.137. Escape character is '^]'. get / HTTP/1.0 HTTP/1.1 302 Found Location: https://telemetry.ceph.com/

2 2

Add previously OSD to a new installation of a 3 nodes Proxmox CEPH
by Gilberto Ferreira 27 Aug '25

27 Aug '25

Hi I have 3 nodes Proxmox Cluster with CEPH, and after a crash, I have to reinstall Proxmox from scratch, along with Ceph. OSD are intact. I already did ceph-volume lvm activate --all and the OSD appears with ceph-volum lvm list and I got a folder with the name of the OSD under /var/lib/ceph/osd. However is not appear in ceph osd tree or ceph -s or even in the web gui. Is there any way to re-add this OSD to Proxmox CEPH? Thanks a lot for any help. Best Regards --- Gilbert

3 19

MGR does not generate prometheus config for ceph-exporter
by Adam Prycki 27 Aug '25

27 Aug '25

Hello, I have issue with multiple of our clusters. It doesn't seem to be versions specific and just happens on our clusters which have been running for a while and have been through some version upgrades. In current cephadm versions prometheus is configured via service discovery from MGR. Relevant prometheus config looks like this: - job_name: 'ceph-exporter' honor_labels: true http_sd_configs: - url: http://X.X.X.X:8765/sd/prometheus/sd-config?service=ceph-exporter Issue is that mgr doesn't return any hosts. Just an empty array []. Prometheus doesn't scrap ceph-exporter and some of our grafana graphs are empty. :/ Other URLs in this config file work fine and generate valid sd configs. Has anyone else experienced this? Does anyone know how to fix this? Best regards Adam Prycki

2 1

Ceph Steering Committee Meeting Minutes for 2025-08-25
by Patrick Donnelly 26 Aug '25

26 Aug '25

Short meeting yesterday: * [Yuri] Tentacle Update - [Laura] maybe an NVMe blocker PR - https://github.com/ceph/ceph/pull/64779 + [Ilya] Does not believe this is a blocker. * [Patrick] Cloud spend has been increasing: we need to move more services local. Probably will be done next year after the lab infrastructure is relocated. -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

NVME-oF only with quay.ceph.io/ceph-ci/ceph:squid-nvmeof ?
by Robert Sander 25 Aug '25

25 Aug '25

Hi, I have created a cluster with the container image quay.ceph.io/ceph-ci/ceph:squid-nvmeof and deployed an NVMe-oF service. This was successful. I am able to connect from a Linux host with the nvme-cli. I do see /dev/nvme0n1 and can fdisk it and format /dev/nvme0n1p1, mount the filesystem and run some IO on it. I can then stop one of the 4 NVMe-oF daemons that are running as listeners and the iniator re-connects to another one. Great. Is it true that currently this image is the only one where NVMe-oF works? The version is reported as 19.2.0-935-g0006f0dd. How much does this image deviate from the quay.io/ceph/ceph:v19.2.3? Does the whole cluster need to run with quay.ceph.io/ceph-ci/ceph:squid-nvmeof or is it sufficient to run the NVMe-oF service with quay.io/ceph/nvmeof:1.4 ? Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

Large file downloads (~2TB) from CephFS drop at ~100 GB
by Mevludin Blazevic 25 Aug '25

25 Aug '25

Hello all, I am experiencing a strange issue with downloads from a CephFS-backed environment and would like to ask if anyone has seen similar behavior. When downloading large files (via wget or even directly through Firefox), the transfer speed is fine until the file size reaches roughly 100 GB. At that point, the session gets disconnected. This happens consistently, independent of the bandwidth available (e.g., whether the link runs at ~80 Mbit/s or ~230 Mbit/s). The only difference is how quickly the 100 GB “border” is reached. After the disconnect, the download can usually be resumed, but the cutoff is very repeatable. Is there a configuration that needs to be done on the ceph site? Best, Mevludin

4 6

After disk failure not deep scrubbed pgs started to increase (ceph quincy)
by Szabo, Istvan (Agoda) 25 Aug '25

25 Aug '25

Hi, After a disk failure, the number of PGs not deep scrubbed started to increase (Ceph Quincy). Here is an example PG: https://gist.githubusercontent.com/Badb0yBadb0y/afe9b26fbae72c79c9430585bf4… Normally, when this happens, I observe some stuck operations and increased latency on certain OSDs (restarting the affected OSD usually resolves the issue, and the not deep-scrubbed PGs get scrubbed within a day). However, this time I don’t see any obvious problems. Could you please help review the attached not deep-scrubbed PG and let me know if you notice anything unusual? For context, the disk was re-added yesterday and rebalance is still in progress, but the not deep-scrubbed alerts started appearing before the disk was added back. Thank you. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 4

For posterity: cct->_conf->osd_fast_shutdown_timeout OSD errors / Run Full Recovery from ONodes (might take a while) during Reef 18.2.1 to 18.2.7 upgrade
by Anthony D'Atri 25 Aug '25

25 Aug '25

Posting this for posterity, in case someone runs into it down the line and finds it in the archives when trying to figure out what the heck is going on. On a Reef 18.2.1 cluster, when nodes reboot, some OSDs experience the below: "assert_msg": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.2.1/rpm/el8/BUILD/ceph-18.2.1/src/osd/OSD.cc: In function 'int OSD::shutdown()' thread 7f1286b61700 time 2025-08-24T19:57:36.343925+0000\n/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.2.1/rpm/el8/BUILD/ceph-18.2.1/src/osd/OSD.cc: 4495: FAILED ceph_assert(end_time - start_time_func < cct->_conf->osd_fast_shutdown_timeout)\n”, which seems to lead to the below at OSD startup: 2025-08-25T00:04:29.669+0000 7faa68c20740 1 freelist _read_cfg 2025-08-25T00:04:29.881+0000 7faa68c20740 1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2025-08-25T00:04:29.881+0000 7faa68c20740 0 bluestore(/var/lib/ceph/osd/ceph-29) _init_alloc::NCB::restore_allocator() failed! Run Full Recovery from ONodes (might take a while) … After some research my understanding is that the root cause is addressed in 18.2.6. My sense is that after this 18.2.1 cluster is completely on 18.2.7 the problem should fade. During the upgrade, however, the instances of it complicate the process and require archiving crashes, waiting an hour or two (20TB spinners) for the abovedescribed full recovery at OSD startup, etc. I’ve found that doubling the default timeout: ceph config set global osd_fast_shutdown_timeout 30 is making a dramatic difference. After setting the above, the upgrade is progressing nicely as expected.

1 0