- ceph-users - lists.ceph.io

April 21st CSC Meeting Notes
by Adam King 21 Apr '25

21 Apr '25

[Matt] required github checks (and make check) instability (and long run times) greatly hurt developer productivity - thanks for renewed attention to this from several folks - can we break up make check into component wise portions - can we take ceph api test out of the CI checks - it also runs in teuthology - more stable there - should discuss with Ernesto [Venky] Next squid/reef point release? - Yuri to open 18.2.7 and 19.2.3 thread - PTLs can review and decide what they need in these releases - Reef EOL is August 2025 - should plan for final point release around then - T release may complicate planning [Yehuda] cephx changes update - planned to be completed by the end of May - Should not block initial T release - will want to backport this [Mike] Infra update: Shaman updated / chacra is next - Are there any other infra projects that need updating? - nothing suggested

1 0

Dashboard lies to me
by Tim Holloway 21 Apr '25

21 Apr '25

Although I've had this problem since at least Pacific, I'm still seeing it on Reef. After much pain and suffering (covered elsewhere), I got my Prometheus services deployed as intended, Ceph health OK, green across the board. However, over the weekend, the dreaded "CephMgrPrometheusModuleInactive" alert has returned to the Dashboard. "The mgr/prometheus module at dell02.mousetech.com:9283 is unreachable." It's a blatant lie. I still get "Ceph HEALTH_OK". All monitor status command show everything running. Checking ports on the host says it's listening. More to the point, I can send my desktop browser to http://dell02.mousetech.com:9283 and get a page that will allow me to see the metrics. So everyone can see it but the Dashboard! I did have some issues when the other prometheus host couldn't resolve the hostname, but I fixed that for all ceph hosts and it was green for days. Now the error is back. Restarting Prometheus didn't help. How is the Dashboard hallucinating this??? Tim

3 7

Upgrade from 18.2.4 to 19.x or even 20
by e3gh75＠gmail.com 21 Apr '25

21 Apr '25

Hello, I have dumb question but hopefully simple question. In the cephadm documentation it states you can upgrade from a point release to another point release without problem, 15.2.2 to 15.2.3. What about jumping up a version or two, like from 18.2.4 to 19.2.2? What have others experience been? Any advice? Thank you for your time.

5 4

Question about compatibility of librdb and backend cluster
by 段世博 21 Apr '25

21 Apr '25

Hi everyone, I would like to ask about the compatibility of librdb versions. If the client version (librbd) used is 17.2.8, but the backend cluster is 16.2.15 or 15.2.17, or even lower versions, can it be accessed normally? Any reply will be very much appreciated.

2 1

Re: Preflight Playbook
by Frédéric Nass 18 Apr '25

18 Apr '25

Do you mean RHCS 5 to 6? The recommended and documented upgrade path is actually RHCS 5 to 7 after upgrading RHCS 5 nodes to RHEL 9. Did you upgrade cephadm and cephadm-ansible packages before running the preflight playbook? Regards, Frédéric. ________________________________ De : Alex <mr.alexey(a)gmail.com> Envoyé : mercredi 16 avril 2025 20:03 À : Frédéric Nass Cc: ceph-users Objet : Re: [ceph-users] Preflight Playbook I was upgrading from Ceph 5 to 6 on RHEL 9. Looking at the dnf logs it seems that i was running crun 1.8.7 and now on 1.16.1 Thanks.

2 8

ceph mon out of quorom and cephfs degraded
by Amudhan P 17 Apr '25

17 Apr '25

Hi, I am having 2 problems with my Ceph version 16.2.6 (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable) deployed thru cephadm. First issue :- 1 out 3 mon service went out of quorum . When restarted service it comes normal but after a few minutes in ceph watch log it reports slow ops and mon goes out of quorum. Node where this mon service failed had one weird thing that I could see 40% of wait in the top command. But I don't see any error in dmesg or anything related to drive IO error. Below are the logs that were printed in ceph watch command. 2025-04-16T09:30:00.000393+0530 mon.strg-node2 [WRN] [WRN] MON_DOWN: 1/3 mons down, quorum strg-node2,strg-node3 2025-04-16T09:30:00.000416+0530 mon.strg-node2 [WRN] mon.strg-node1 (rank 0) addr [v2:10.0.103.1:3300/0,v1:10.0.103.1:6789/0] is down (out of quorum) For now this is not appearing again. Second issue Cephfs degraded :- I have 2 MDS services running in 2 different nodes. Both are in a stopped state. when running Ceph -s command cluster: id: 7b3a4952-2131-11ec-94ce-0cc47a5ec98a health: HEALTH_WARN 2 failed cephadm daemon(s) 1 filesystem is degraded insufficient standby MDS daemons available services: mon: 3 daemons, quorum strg-node1,strg-node2,strg-node3 (age 4h) mgr: strg-node2.unyimy(active, since 4h), standbys: strg-node1.ivkfid mds: 1/1 daemons up osd: 32 osds: 32 up (since 4h), 32 in (since 10w) data: volumes: 0/1 healthy, 1 recovering pools: 3 pools, 321 pgs objects: 15.49M objects, 54 TiB usage: 109 TiB used, 66 TiB / 175 TiB avail pgs: 321 active+clean Volume shows recovering but there wasn't any progress till now even manual start mds service fails again. In Ceph -s command under services it shows mds up no any mds service is running. Below is a log snip from one of the mds service. -25> 2025-04-16T09:59:29.954+0000 7f43d0874700 1 mds.0.journaler.pq(ro) _finish_read_head loghead(trim 13967032320, ex> Apr 16 15:29:30 strg-node4 bash[7566]: debug -24> 2025-04-16T09:59:29.954+0000 7f43d0874700 1 mds.0.journaler.pq(ro) probing for end of the log Apr 16 15:29:30 strg-node4 bash[7566]: debug -23> 2025-04-16T09:59:29.954+0000 7f43d9085700 10 monclient: get_auth_request con 0x562856a17400 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug -22> 2025-04-16T09:59:29.954+0000 7f43d8884700 10 monclient: get_auth_request con 0x562856a17c00 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug -21> 2025-04-16T09:59:29.974+0000 7f43cf071700 1 mds.0.journaler.mdlog(ro) recover start Apr 16 15:29:30 strg-node4 bash[7566]: debug -20> 2025-04-16T09:59:29.974+0000 7f43cf071700 1 mds.0.journaler.mdlog(ro) read_head Apr 16 15:29:30 strg-node4 bash[7566]: debug -19> 2025-04-16T09:59:29.974+0000 7f43cf071700 4 mds.0.log Waiting for journal 0x200 to recover... Apr 16 15:29:30 strg-node4 bash[7566]: debug -18> 2025-04-16T09:59:29.974+0000 7f43d8083700 10 monclient: get_auth_request con 0x562856a25000 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug -17> 2025-04-16T09:59:29.998+0000 7f43d0874700 1 mds.0.journaler.pq(ro) _finish_probe_end write_pos = 13968309289 (hea> Apr 16 15:29:30 strg-node4 bash[7566]: debug -16> 2025-04-16T09:59:29.998+0000 7f43d0874700 4 mds.0.purge_queue operator(): open complete Apr 16 15:29:30 strg-node4 bash[7566]: debug -15> 2025-04-16T09:59:29.998+0000 7f43d0874700 1 mds.0.journaler.pq(ro) set_writeable Apr 16 15:29:30 strg-node4 bash[7566]: debug -14> 2025-04-16T09:59:29.998+0000 7f43cf872700 1 mds.0.journaler.mdlog(ro) _finish_read_head loghead(trim 189741504921> Apr 16 15:29:30 strg-node4 bash[7566]: debug -13> 2025-04-16T09:59:29.998+0000 7f43cf872700 1 mds.0.journaler.mdlog(ro) probing for end of the log Apr 16 15:29:30 strg-node4 bash[7566]: debug -12> 2025-04-16T09:59:30.002+0000 7f43d9085700 10 monclient: get_auth_request con 0x562856a25c00 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug -11> 2025-04-16T09:59:30.098+0000 7f43cf872700 1 mds.0.journaler.mdlog(ro) _finish_probe_end write_pos = 1897428915052> Apr 16 15:29:30 strg-node4 bash[7566]: debug -10> 2025-04-16T09:59:30.098+0000 7f43cf071700 4 mds.0.log Journal 0x200 recovered. Apr 16 15:29:30 strg-node4 bash[7566]: debug -9> 2025-04-16T09:59:30.098+0000 7f43cf071700 4 mds.0.log Recovered journal 0x200 in format 1 Apr 16 15:29:30 strg-node4 bash[7566]: debug -8> 2025-04-16T09:59:30.098+0000 7f43cf071700 2 mds.0.127506 Booting: 1: loading/discovering base inodes Apr 16 15:29:30 strg-node4 bash[7566]: debug -7> 2025-04-16T09:59:30.098+0000 7f43cf071700 0 mds.0.cache creating system inode with ino:0x100 Apr 16 15:29:30 strg-node4 bash[7566]: debug -6> 2025-04-16T09:59:30.098+0000 7f43cf071700 0 mds.0.cache creating system inode with ino:0x1 Apr 16 15:29:30 strg-node4 bash[7566]: debug -5> 2025-04-16T09:59:30.098+0000 7f43d8884700 10 monclient: get_auth_request con 0x562856a25800 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug -4> 2025-04-16T09:59:30.098+0000 7f43d8083700 10 monclient: get_auth_request con 0x562856a5dc00 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug -3> 2025-04-16T09:59:30.146+0000 7f43cf872700 2 mds.0.127506 Booting: 2: replaying mds log Apr 16 15:29:30 strg-node4 bash[7566]: debug -2> 2025-04-16T09:59:30.146+0000 7f43cf872700 2 mds.0.127506 Booting: 2: waiting for purge queue recovered Apr 16 15:29:30 strg-node4 bash[7566]: debug -1> 2025-04-16T09:59:30.146+0000 7f43d9085700 10 monclient: get_auth_request con 0x562856a25400 auth_method 0 Apr 16 15:29:30 strg-node4 bash[7566]: debug 0> 2025-04-16T09:59:30.230+0000 7f43ce06f700 -1 *** Caught signal (Segmentation fault) ** Apr 16 15:29:30 strg-node4 bash[7566]: in thread 7f43ce06f700 thread_name:md_log_replay Apr 16 15:29:30 strg-node4 bash[7566]: ceph version 16.2.6 (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable) Apr 16 15:29:30 strg-node4 bash[7566]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f43dd293b20] Apr 16 15:29:30 strg-node4 bash[7566]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x8ec7a00) [0x7f43e6ef3a00] Apr 16 15:29:30 strg-node4 bash[7566]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Not sure what caused the issue. I couldn't find any resources to fix this issue. Need help from someone to bring the ceph cluster online.

2 6

HEALTH_ERR: 1 MDSs report damaged metadata - damage_type=dentry
by Christophe DIARRA 17 Apr '25

17 Apr '25

Hello, After an electrical maintenance I restarted our ceph cluster but it remains in an unhealthy state: HEALTH_ERR 1 MDSs report damaged metadata. How to repair this damaged metadata ? To bring down the cephfs cluster I unmounted the fs from the client first and then did: ceph fs set cfs_irods_test down true To bring up the cephfs cluster I did: ceph fs set cfs_irods_test down false Fortunately the cfs_irods_test fs is almost empty and is a fs for tests.The ceph cluster is not in production yet. Following is the current status: [root@mon-01 ~]# ceph health detail HEALTH_ERR 1 MDSs report damaged metadata *[ERR] MDS_DAMAGE: 1 MDSs report damaged metadata mds.cfs_irods_test.mon-03.vlmeuz(mds.0): Metadata damage detected* [root@mon-01 ~]# ceph -s cluster: id: b87276e0-1d92-11ef-a9d6-507c6f66ae2e health: HEALTH_ERR 1 MDSs report damaged metadata services: mon: 3 daemons, quorum mon-01,mon-03,mon-02 (age 17h) mgr: mon-02.mqaubn(active, since 17h), standbys: mon-03.gvywio, mon-01.xhxqdi mds: 1/1 daemons up, 2 standby osd: 368 osds: 368 up (since 17h), 368 in (since 3w) data: volumes: 1/1 healthy pools: 10 pools, 4353 pgs objects: 1.25M objects, 3.9 TiB usage: 417 TiB used, 6.4 PiB / 6.8 PiB avail pgs: 4353 active+clean [root@mon-01 ~]# ceph fs ls name: cfs_irods_test, metadata pool: cfs_irods_md_test, data pools: [cfs_irods_def_test cfs_irods_data_test ] [root@mon-01 ~]# ceph mds stat cfs_irods_test:1 {0=cfs_irods_test.mon-03.vlmeuz=up:active} 2 up:standby [root@mon-01 ~]# ceph fs status cfs_irods_test - 0 clients ============== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active cfs_irods_test.mon-03.vlmeuz Reqs: 0 /s 12 15 14 0 POOL TYPE USED AVAIL cfs_irods_md_test metadata 11.4M 34.4T cfs_irods_def_test data 0 34.4T cfs_irods_data_test data 0 4542T STANDBY MDS cfs_irods_test.mon-01.hitdem cfs_irods_test.mon-02.awuygq MDS version: ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable) [root@mon-01 ~]# [root@mon-01 ~]# ceph tell mds.cfs_irods_test:0 damage ls 2025-04-17T10:23:31.849+0200 7f4b87fff640 0 client.86181 ms_handle_reset on v2:130.84.80.10:6800/3218663047 2025-04-17T10:23:31.866+0200 7f4b87fff640 0 client.86187 ms_handle_reset on v2:130.84.80.10:6800/3218663047 [ { *"damage_type": "dentry",* "id": 241447932, "ino": 1, "frag": "*", "dname": "testdir2", "snap_id": "head", "path": "/testdir2" }, { *"damage_type": "dentry"*, "id": 2273238993, "ino": 1, "frag": "*", "dname": "testdir1", "snap_id": "head", "path": "/testdir1" } ] [root@mon-01 ~]# Any help will be appreciated, Thanks, Christophe

1 0

CephFS Snapshot Mirroring
by Vladimir Cvetkovic 16 Apr '25

16 Apr '25

Hi everyone, We have two remote clusters and are trying to set up snapshot mirroring. Our directories are in a syncing state but never seem to actually sync the snapshots. Snapshot mirroring seems way to slow for large file systems - does anyone have a working setup? Thanks!

4 5

reef 18.2.6 hotfix QE validation status
by Yuri Weinstein 16 Apr '25

16 Apr '25

Details of this release are summarized here: https://tracker.ceph.com/issues/70938#note-1 Release Notes - TBD LRC upgrade - N/A Seeking approvals/reviews for: smoke - same as in 18.2.5 rados - Radek, Laura approved? orch - Adam King, Guillaume approved? This release has two PRs: https://github.com/ceph/ceph/pull/62791 https://github.com/ceph/ceph/pull/62814 and has limited testing (only three suits above)

3 2

Preflight Playbook
by Alex 16 Apr '25

16 Apr '25

Good morning. Should the Preflight playbook maybe update `crun` as well? I was updating the Ceph version yesterday and it wouldn't work until I updated `crun`. Thanks.

2 2