- ceph-users - lists.ceph.io

18.2.2: Upgrade not starting (ceph orch upgrade)
by Michel Jouvin 30 Apr '25

30 Apr '25

Hi, I'm trying to upgrade a (cephadm) cluster from 18.2.2 to 18.2.6, using 'ceph orch upgrade'. When I enter the command 'ceph orch upgrade start --ceph-version 18.2.6', I receive a message saying that the upgrade has been initiated, with a similar message in the logs but nothing happens after this. 'ceph orch upgrade status' says: ------- [root@ijc-mon1 ~]# ceph orch upgrade status { "target_image": "quay.io/ceph/ceph:v18.2.6", "in_progress": true, "which": "Upgrading all daemon types on all hosts", "services_complete": [], "progress": "", "message": "", "is_paused": false } ------- The first time I entered the command, the cluster status was HEALTH_WARN because of 2 stray daemons (caused by destroyed OSDs, rm --replace). I set mgr/cephadm/warn_on_stray_daemons to false to ignore these 2 daemons, the cluster is now HEALTH_OK but it doesn't help. Following a Red Hat KB entry, I tried to failover the mgr, stopped an restarted the upgrade but without any improvement. I have not seen anything in the logs, except that there is an INF entry every 10s about the destroyed OSD saying: ------ 2025-04-24T21:30:54.161988+0000 mgr.ijc-mon1.yyfnhz (mgr.55376028) 14079 : cephadm [INF] osd.253 now down 2025-04-24T21:30:54.162601+0000 mgr.ijc-mon1.yyfnhz (mgr.55376028) 14080 : cephadm [INF] Daemon osd.253 on dig-osd4 was already removed 2025-04-24T21:30:54.164440+0000 mgr.ijc-mon1.yyfnhz (mgr.55376028) 14081 : cephadm [INF] Successfully destroyed old osd.253 on dig-osd4; ready for replacement 2025-04-24T21:30:54.164536+0000 mgr.ijc-mon1.yyfnhz (mgr.55376028) 14082 : cephadm [INF] Zapping devices for osd.253 on dig-osd4 ----- The message seems to be only for one of the 2 destroyed OSDs since I restarted the mgr. May this be the cause for the stucked upgrade? What can I do for fixing this? Thanks in advance for any hint. Best regards, Michel

4 22

manual squid update procedure
by Christopher Durham 30 Apr '25

30 Apr '25

Hi, The non-cephadm update procedure to update reef to squid (for rpm-based clusters) here: https://docs.ceph.com/en/latest/releases/squid/#upgrading-non-cephadm-clust… suggests that your monitors/mds/radosgw/osds are all on separate servers. While perhaps theyshould be that is not possible at current. If for example, I have a mon and an mds on a single server, I really don't want to updatejust a mon as the mds and mon may use common libraries. As such, I am thinking that to do an update in this scenario I do the following set noout set max_mds to for my filesystem, and note the remaining active mds disable standby_replay Then, for each of the servers running an mds (but not the remaining active mds): 1. stop all ceph daemons (mon,mds,radosgw,osd) and do the update, reboot and/or restart the mon and mds 2. Proceed to all the other mds servers except the last one that is on reef3. Finish all other remaining mon updates as in #1 4. Then when the last server with a reef mds is left stop it and let a standby squid mds take over and then do the update and either restart all the daemons or reboot 5. if there are other osd-only nodes, update them and restart the osds Then, afterwards, allow standby replay, reset max_mds to what it was before, unset noout, require squid for osds. Does this make sense? Am I forgetting something? I know setting max_mds 1 is important but want to be sure I have not forgotten anything. Thanks. -Chris

2 1

Memory usage
by Dominique Ramaekers 29 Apr '25

29 Apr '25

Hi, Housekeeping... I was cleaning up my snapshots and was flattening clones... Suddenly I ran out of memory on my nodes! 4 node cluster with each 10 ssd osd's with total storage size 25TiB. Each node has about 45 GiB of free (available) memory in normal operation. After flattening several images and removing dozens of snapshots, the free memory probably was already lower than the usual 45GiB. I was flattening an image of 75GiB and 2 out of 4 nodes ran out of memory. One node even automatically killed processes randomly to free up memory... After regaining control over the system, I put that node in maintenance and rebooted this node. After reboot the free mem was like 70Gib. Over night all of the nodes were back on the usual 45GiB. Today I checked the free mem of each node: 45GiB free. So I flattened again an image of 75GiB. And yes, the free mem dropped from 45GiB to 5GiB really fast! Is there a way to avoid this behavior of the cluster? Greetings, Dominique.

5 5

v18.2.6 Reef released
by Yuri Weinstein 29 Apr '25

29 Apr '25

We're happy to announce the sixth backport (hotfix) release in the Reef series. https://ceph.io/en/news/blog/2025/v18-2-6-reef-released/ Notable Changes --------------- * ceph-volume: A bug related to cryptsetup version handling has been fixed. Related tracker: https://tracker.ceph.com/issues/66393 * RADOS: A bug related to IPv6 support is now fixed. Related tracker: https://tracker.ceph.com/issues/67517 Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-18.2.6.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: ff498e17d264a1a4d588c361cbce9cc65afa2327

3 2

Re: write latency increase after upgrade from pacific to quincy
by Nima AbolhassanBeigi 29 Apr '25

29 Apr '25

Hi Enrico We were aware of the change on the OSD scheduler before upgrading and changed it back to wpq. We put off the change of this option for another time. However, to make sure this wasn't the root cause of the problem, we set the norecover flag on the cluster, and I suspect that if the recovery process were the reason behind this issue, it would have been mitigated. I think the rocksdb compaction_on_deletion option is what caused the problem, but as, for the time being, we can't put our theory to the test, I'm not quite sure it is what caused it. The reason we suspect so is because I checked the OSD logs and used a Python script that parses the compaction logs and gives useful information about all the compaction operations that happened on the OSD in that log file. I compared the extracted data from logs between two OSDs, one had the option set to true and the other set to false. The number of compactions on the second OSD was almost half of what it was on the first one, and as we had the experience of rocksdb compactions affecting the latency of our OSDs, especially the SSD ones, we came to this conclusion. The compaction data ( https://docs.google.com/document/d/1qXDAHvJnfnOPN4ZevdGbgHmsMse3UpL0gZfSvVi… ) *Do you think this is possible?* And to answer your question about the kind of workload, it is quite diverse. We serve both RBD and S3 clients. Our clients have different workloads, and I cannot describe their behaviour in a certain pattern. This might be a bad idea, and we probably should break the clusters down into several more specific-purpose clusters, but that's a problem for another day. If you have any specific questions in mind that I should answer, *don't hesitate to ask.* And by the increase in the number of compactions on our OSDs, after the option was set to true, we could say that the workload could have high delete requests, which I think is what causes the high number of tombstones. Regards, Nima On Tue, Apr 29, 2025 at 3:40 PM Nima AbolhassanBeigi < nima.abolhassanbeigi(a)gmail.com> wrote: > Hi Enrico > We were aware of the change on the OSD scheduler before upgrading and > changed it back to wpq. We put off the change of this option for another > time. However, to make sure this wasn't the root cause of the problem, we > set the norecover flag on the cluster, and I suspect that if the recovery > process were the reason behind this issue, it would have been mitigated. > > I think the rocksdb compaction_on_deletion option is what caused the > problem, but as, for the time being, we can't put our theory to the test, > I'm not quite sure it is what caused it. > The reason we suspect so is because I checked the OSD logs and used a > Python script that parses the compaction logs and gives useful information > about all the compaction operations that happened on the OSD in that log > file. > I compared the extracted data from logs between two OSDs, one had the > option set to true and the other set to false. The number of compactions on > the second OSD was almost half of what it was on the first one, and as we > had the experience of rocksdb compactions affecting the latency of our > OSDs, especially the SSD ones, we came to this conclusion. > The compaction data ( > https://docs.google.com/document/d/1qXDAHvJnfnOPN4ZevdGbgHmsMse3UpL0gZfSvVi… > ) > *Do you think this is possible?* > > And to answer your question about the kind of workload, it is quite > diverse. We serve both RBD and S3 clients. Our clients have different > workloads, and I cannot describe their behaviour in a certain pattern. This > might be a bad idea, and we probably should break the clusters down into > several more specific-purpose clusters, but that's a problem for another > day. > If you have any specific questions in mind that I should answer, *don't > hesitate to ask.* And by the increase in the number of compactions on our > OSDs, after the option was set to true, we could say that the workload > could have high delete requests, which I think is what causes the high > number of tombstones. > > Regards, > Nima > > On Tue, Apr 29, 2025 at 2:33 PM Enrico Bocchi <enrico.bocchi(a)cern.ch> > wrote: > >> Hello Nima, >> >> Unsure if you have found the root cause of the problem in the meantime> >> From the top of my head, if any useful: >> - Quincy changes the default scheduler from wpq to mclock >> - The default number of scrubs on each osd is increased from 1 to 3 >> - There's a new rocksdb compact_on_deletion option that triggers >> compaction more frequently according to number of tombstones over a >> sliding window. >> >> The latter, however, is very workload dependent. What type of workload >> does the cluster serve? >> >> Cheers, >> Enrico >> >> >> On 4/15/25 12:13, Nima AbolhassanBeigi wrote: >> > Hello again >> > We have done some investigating. Since our first message did not have >> any >> > information or context, let me add some new information. >> > >> > I looked through our dashboards and the write latency panels in the OSD >> > Overview dashboard. >> > The latency for write operations had increased significantly, but in >> > distinction, the physical write more or less stayed the same; however, >> the >> > write process operations and write prepare operations have increased >> > dramatically. >> > >> > In the end, the RocksDB tunings helped our situation and decreased the >> > latency more or less to the pre-upgrade amount. >> > We changed the default parameters to the following: >> > >> compression=kNoCompression,max_write_buffer_number=128,min_write_buffer_number_to_merge=16,compaction_style=kCompactionStyleLevel,write_buffer_size=8388608,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0 >> > This seems to have done the trick, however, we are still worried that we >> > haven't found the root cause of this problem and only mitigated the >> > symptoms. >> > >> > Another probably important information is that the bluefs allocator for >> our >> > OSDs is not the default hybrid and is set to the bitmap option. In >> > addition, the bluestore allocation block score of these OSDs is, for >> most >> > of them, almost 0.9, which I know is terrible, but what I don't >> understand >> > is why this problem occurred post-upgrade. Could it be the terrible >> block >> > scores? >> > >> > Another interesting change was the memory usage of each OSD >> post-upgrade, >> > which dropped nearly in half, but the Node's memory cache/buffer usage >> > increased. >> > >> > I've put the Grafana panels screenshots in this Google Doc >> > < >> https://docs.google.com/document/d/1DSf4MJoze_BTSWAJWoYyQetA33P6fwpMavYXPdA… >> > >> > . >> > We started our upgrade process on March 6 and then again on March 7. We >> > tuned the RocksDB parameters on March 9th, as the screenshots show, >> which >> > drops the latency. >> > >> > We are still looking for the root cause, and I could use all the help I >> can >> > get to try to find it, and it is much appreciated. >> > >> > Regards >> > Nima >> > >> > On Mon, Apr 7, 2025 at 3:27 PM Nima AbolhassanBeigi < >> > nima.abolhassanbeigi(a)gmail.com> wrote: >> > >> >> Hi dear ceph community >> >> >> >> We have encountered an issue with our ceph cluster after upgrading from >> >> v16.2.13 to v17.2.7. >> >> The issue is that the write latency on OSDs has increased significantly >> >> and doesn't seem to plummet back down. >> >> The average write latency has almost doubled, and this has happened >> since >> >> we upgraded the OSDs. >> >> >> >> If anybody could help figure this out. >> >> >> >> Kind regards >> >> Nima >> >> >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users(a)ceph.io >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> -- >> Enrico Bocchi >> CERN European Laboratory for Particle Physics >> IT - Storage & Data Management - General Storage Services >> Mailbox: G20500 - Office: 31-2-010 >> 1211 Genève 23 >> Switzerland >> >>

1 0

crush rule: is it valid to use a non root element for the root parameter?
by Michel Jouvin 29 Apr '25

29 Apr '25

Hi, We have use case where we had like to restrict some pools to a subset of the OSDs located in a particular section of the crush map hierarchy (OSDs backed up by UPS/Diesel). We tried to define for these (replica 3) pools a specific crush rule with the root paramater defined to a specific row (which contains 3 OSD servers with #10 OSD each). At the beginning it worked but after some time (probably after doing a reweight on the OSDs in this row to reduce the number of PGs from other pools), a few PGs are active+clean+remapped and 1 is undersized. 'ceph osd pg dump|grep remapped' gives an output similar to the following one for each remapped PG: 20.1ae 648 0 0 648 0 374416846 0 0 438 1404 438 active+clean+remapped 2025-04-16T07:19:40.507778+0000 43117'1018433 48443:1131738 [70,58] 70 [70,58,45] 70 43117'1018433 2025-04-16T07:19:40.507443+0000 43117'1018433 2025-04-16T07:19:40.507443+0000 0 15 periodic scrub scheduled @ 2025-04-17T19:18:23.470846+0000 648 0 We can see that we currently have 3 replica but that Ceph would like to move to 2... (the undersized PG has currently only 2 replica for an unknown reason, probably the same). Is it wrong trying to do what we did, i.e. using a row for the crush rule root parameter? If not, where could we find more information about the cause? Thanks in advance for any help. Best regards, Michel --------------------- Crush rule used ----------------- { "rule_id": 2, "rule_name": "ha-replicated_ruleset", "type": 1, "steps": [ { "op": "take", "item": -22, "item_name": "row-01~hdd" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] } ------------------- Beginning of the CRUSH tree ------------------- ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 843.57141 root default -19 843.57141 datacenter bat.206 -21 283.81818 row row-01 -15 87.32867 host cephdevel-76079 1 hdd 7.27739 osd.1 up 0.50000 1.00000 2 hdd 7.27739 osd.2 up 0.50000 1.00000 14 hdd 7.27739 osd.14 up 0.50000 1.00000 39 hdd 7.27739 osd.39 up 0.50000 1.00000 40 hdd 7.27739 osd.40 up 0.50000 1.00000 41 hdd 7.27739 osd.41 up 0.50000 1.00000 42 hdd 7.27739 osd.42 up 0.50000 1.00000 43 hdd 7.27739 osd.43 up 0.50000 1.00000 44 hdd 7.27739 osd.44 up 0.50000 1.00000 45 hdd 7.27739 osd.45 up 0.50000 1.00000 46 hdd 7.27739 osd.46 up 0.50000 1.00000 47 hdd 7.27739 osd.47 up 0.50000 1.00000 -3 94.60606 host cephdevel-76154 49 hdd 7.27739 osd.49 up 0.50000 1.00000 50 hdd 7.27739 osd.50 up 0.50000 1.00000 51 hdd 7.27739 osd.51 up 0.50000 1.00000 66 hdd 7.27739 osd.66 up 0.50000 1.00000 67 hdd 7.27739 osd.67 up 0.50000 1.00000 68 hdd 7.27739 osd.68 up 0.50000 1.00000 69 hdd 7.27739 osd.69 up 0.50000 1.00000 70 hdd 7.27739 osd.70 up 0.50000 1.00000 71 hdd 7.27739 osd.71 up 0.50000 1.00000 72 hdd 7.27739 osd.72 up 0.50000 1.00000 73 hdd 7.27739 osd.73 up 0.50000 1.00000 74 hdd 7.27739 osd.74 up 0.50000 1.00000 75 hdd 7.27739 osd.75 up 0.50000 1.00000 -4 101.88345 host cephdevel-76204 48 hdd 7.27739 osd.48 up 0.50000 1.00000 52 hdd 7.27739 osd.52 up 0.50000 1.00000 53 hdd 7.27739 osd.53 up 0.50000 1.00000 54 hdd 7.27739 osd.54 up 0.50000 1.00000 56 hdd 7.27739 osd.56 up 0.50000 1.00000 57 hdd 7.27739 osd.57 up 0.50000 1.00000 58 hdd 7.27739 osd.58 up 0.50000 1.00000 59 hdd 7.27739 osd.59 up 0.50000 1.00000 60 hdd 7.27739 osd.60 up 0.50000 1.00000 61 hdd 7.27739 osd.61 up 0.50000 1.00000 62 hdd 7.27739 osd.62 up 0.50000 1.00000 63 hdd 7.27739 osd.63 up 0.50000 1.00000 64 hdd 7.27739 osd.64 up 0.50000 1.00000 65 hdd 7.27739 osd.65 up 0.50000 1.00000 -23 203.16110 row row-02 -13 87.32867 host cephdevel-76213 27 hdd 7.27739 osd.27 up 1.00000 1.00000 28 hdd 7.27739 osd.28 up 1.00000 1.00000 29 hdd 7.27739 osd.29 up 1.00000 1.00000 30 hdd 7.27739 osd.30 up 1.00000 1.00000 31 hdd 7.27739 osd.31 up 1.00000 1.00000 32 hdd 7.27739 osd.32 up 1.00000 1.00000 33 hdd 7.27739 osd.33 up 1.00000 1.00000 34 hdd 7.27739 osd.34 up 1.00000 1.00000 35 hdd 7.27739 osd.35 up 1.00000 1.00000 36 hdd 7.27739 osd.36 up 1.00000 1.00000 37 hdd 7.27739 osd.37 up 1.00000 1.00000 38 hdd 7.27739 osd.38 up 1.00000 1.00000 ......

3 5

18.2.6 upgrade OSDs fail to mount
by Marco Pizzolo 29 Apr '25

29 Apr '25

Hello Everyone, I'm upgrading from 18.2.4 to 18.2.6, and I have a 4-node cluster with 8 NVMe's per node. Each NVMe is split into 2 OSDs. The upgrade went through the mgr, mon, crash and began upgrading OSDs. The OSDs it was upgrading were not coming back online. I tried rebooting, and no luck. journalctl -xe shows the following: ░░ The unit docker-02cb79ef9a657cdaa26b781966aa6d2f1d5e54cdc9efa6c5ff1f0e98c3a866e4.scope has successfully entered the 'dead' state. Apr 29 06:24:09 prdhcistonode01 dockerd[2967]: time="2025-04-29T06:24:09.282073583-04:00" level=info msg="ignoring event" container=76c56ddd668015de0022bfa2527060e64a9513> Apr 29 06:24:09 prdhcistonode01 containerd[2797]: time="2025-04-29T06:24:09.282129114-04:00" level=info msg="shim disconnected" id=76c56ddd668015de0022bfa2527060e64a95137> Apr 29 06:24:09 prdhcistonode01 containerd[2797]: time="2025-04-29T06:24:09.282219664-04:00" level=warning msg="cleaning up after shim disconnected" id=76c56ddd668015de00> Apr 29 06:24:09 prdhcistonode01 containerd[2797]: time="2025-04-29T06:24:09.282242484-04:00" level=info msg="cleaning up dead shim" Apr 29 06:24:09 prdhcistonode01 bash[23886]: debug 2025-04-29T10:24:09.287+0000 7f6961ae9740 1 mClockScheduler: set_osd_capacity_params_from_config: osd_bandwidth_cost_p> Apr 29 06:24:09 prdhcistonode01 bash[23886]: debug 2025-04-29T10:24:09.287+0000 7f6961ae9740 0 osd.3:0.OSDShard using op scheduler mclock_scheduler, cutoff=196 Apr 29 06:24:09 prdhcistonode01 bash[23886]: debug 2025-04-29T10:24:09.287+0000 7f6961ae9740 1 bdev(0x56046b4c8000 /var/lib/ceph/osd/ceph-3/block) open path /var/lib/cep> Apr 29 06:24:09 prdhcistonode01 containerd[2797]: time="2025-04-29T06:24:09.292047607-04:00" level=warning msg="cleanup warnings time=\"2025-04-29T06:24:09-04:00\" level=> Apr 29 06:24:09 prdhcistonode01 dockerd[2967]: time="2025-04-29T06:24:09.292163618-04:00" level=info msg="ignoring event" container=02cb79ef9a657cdaa26b781966aa6d2f1d5e54> Apr 29 06:24:09 prdhcistonode01 containerd[2797]: time="2025-04-29T06:24:09.292216428-04:00" level=info msg="shim disconnected" id=02cb79ef9a657cdaa26b781966aa6d2f1d5e54c> Apr 29 06:24:09 prdhcistonode01 containerd[2797]: time="2025-04-29T06:24:09.292277279-04:00" level=warning msg="cleaning up after shim disconnected" id=02cb79ef9a657cdaa2> Apr 29 06:24:09 prdhcistonode01 containerd[2797]: time="2025-04-29T06:24:09.292291949-04:00" level=info msg="cleaning up dead shim" Apr 29 06:24:09 prdhcistonode01 bash[23886]: debug 2025-04-29T10:24:09.287+0000 7f6961ae9740 1 bdev(0x56046b4c8000 /var/lib/ceph/osd/ceph-3/block) open size 640122932428> Apr 29 06:24:09 prdhcistonode01 bash[23886]: debug 2025-04-29T10:24:09.287+0000 7f6961ae9740 -1 bluestore(/var/lib/ceph/osd/ceph-3) _set_cache_sizes bluestore_cache_meta_> Apr 29 06:24:09 prdhcistonode01 bash[23886]: debug 2025-04-29T10:24:09.287+0000 7f6961ae9740 1 bdev(0x56046b4c8000 /var/lib/ceph/osd/ceph-3/block) close Apr 29 06:24:09 prdhcistonode01 containerd[2797]: time="2025-04-29T06:24:09.303385220-04:00" level=warning msg="cleanup warnings time=\"2025-04-29T06:24:09-04:00\" level=> Apr 29 06:24:09 prdhcistonode01 bash[24158]: debug 2025-04-29T10:24:09.307+0000 7f2c10403740 1 mClockScheduler: set_osd_capacity_params_from_config: osd_bandwidth_cost_p> Apr 29 06:24:09 prdhcistonode01 bash[24158]: debug 2025-04-29T10:24:09.307+0000 7f2c10403740 0 osd.0:0.OSDShard using op scheduler mclock_scheduler, cutoff=196 Apr 29 06:24:09 prdhcistonode01 bash[23144]: debug 2025-04-29T10:24:09.307+0000 7f12f08c5740 -1 osd.15 0 OSD:init: unable to mount object store Apr 29 06:24:09 prdhcistonode01 bash[23144]: debug 2025-04-29T10:24:09.307+0000 7f12f08c5740 -1 ** ERROR: osd init failed: (22) Invalid argument Apr 29 06:24:09 prdhcistonode01 bash[24158]: debug 2025-04-29T10:24:09.307+0000 7f2c10403740 1 bdev(0x55d5e45f0000 /var/lib/ceph/osd/ceph-0/block) open path /var/lib/cep> Apr 29 06:24:09 prdhcistonode01 bash[24158]: debug 2025-04-29T10:24:09.307+0000 7f2c10403740 1 bdev(0x55d5e45f0000 /var/lib/ceph/osd/ceph-0/block) open size 640122932428> Apr 29 06:24:09 prdhcistonode01 bash[24158]: debug 2025-04-29T10:24:09.307+0000 7f2c10403740 -1 bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes bluestore_cache_meta_> Apr 29 06:24:09 prdhcistonode01 bash[24158]: debug 2025-04-29T10:24:09.307+0000 7f2c10403740 1 bdev(0x55d5e45f0000 /var/lib/ceph/osd/ceph-0/block) close Apr 29 06:24:09 prdhcistonode01 bash[24328]: debug 2025-04-29T10:24:09.363+0000 7f30b83b1740 1 mClockScheduler: set_osd_capacity_params_from_config: osd_bandwidth_cost_p> Apr 29 06:24:09 prdhcistonode01 bash[24328]: debug 2025-04-29T10:24:09.363+0000 7f30b83b1740 0 osd.8:0.OSDShard using op scheduler mclock_scheduler, cutoff=196 Apr 29 06:24:09 prdhcistonode01 bash[24328]: debug 2025-04-29T10:24:09.363+0000 7f30b83b1740 1 bdev(0x555f40688000 /var/lib/ceph/osd/ceph-8/block) open path /var/lib/cep> Apr 29 06:24:09 prdhcistonode01 bash[24328]: debug 2025-04-29T10:24:09.363+0000 7f30b83b1740 1 bdev(0x555f40688000 /var/lib/ceph/osd/ceph-8/block) open size 640122932428> Apr 29 06:24:09 prdhcistonode01 bash[24328]: debug 2025-04-29T10:24:09.363+0000 7f30b83b1740 -1 bluestore(/var/lib/ceph/osd/ceph-8) _set_cache_sizes bluestore_cache_meta_> Apr 29 06:24:09 prdhcistonode01 bash[24328]: debug 2025-04-29T10:24:09.363+0000 7f30b83b1740 1 bdev(0x555f40688000 /var/lib/ceph/osd/ceph-8/block) close Apr 29 06:24:09 prdhcistonode01 systemd[1]: ceph-fbc38f5c-a3a6-11ea-805c-3b954db9ce7a(a)osd.12.service: Main process exited, code=exited, status=1/FAILURE Any help you can offer would be greatly appreciated. This is running in docker: Client: Docker Engine - Community Version: 24.0.7 API version: 1.43 Go version: go1.20.10 Git commit: afdd53b Built: Thu Oct 26 09:08:01 2023 OS/Arch: linux/amd64 Context: default Server: Docker Engine - Community Engine: Version: 24.0.7 API version: 1.43 (minimum version 1.12) Go version: go1.20.10 Git commit: 311b9ff Built: Thu Oct 26 09:08:01 2023 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.6.25 GitCommit: d8f198a4ed8892c764191ef7b3b06d8a2eeb5c7f runc: Version: 1.1.10 GitCommit: v1.1.10-0-g18a0cb0 docker-init: Version: 0.19.0 GitCommit: de40ad0 Thanks, Marco

2 5

While update minor version of Ceph cluster with ceph-ansible
by dhivagar selvam 29 Apr '25

29 Apr '25

Hi.. We are facing the below error while update minor version from 18.2.4 to 18.2.6 using rolling-update playbook. TASK [ceph-facts : Set_fact _monitor_addresses - ipv4] ************************************************************************************************************************************************************ task path: /root/ceph-ansible/roles/ceph-facts/tasks/set_monitor_address.yml:2 fatal: [ceph-mont1]: FAILED! => msg: |- The task includes an option with an undefined variable. The error was: 'ansible.vars.hostvars.HostVarsVars object' has no attribute 'public_network'. 'ansible.vars.hostvars.HostVarsVars object' has no attribute 'public_network' The error appears to be in '/root/ceph-ansible/roles/ceph-facts/tasks/set_monitor_address.yml': line 2, column 3, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: --- - name: Set_fact _monitor_addresses - ipv4 ^ here fatal: [ceph-mont2]: FAILED! => msg: |- The task includes an option with an undefined variable. The error was: 'ansible.vars.hostvars.HostVarsVars object' has no attribute 'public_network'. 'ansible.vars.hostvars.HostVarsVars object' has no attribute 'public_network' The error appears to be in '/root/ceph-ansible/roles/ceph-facts/tasks/set_monitor_address.yml': line 2, column 3, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: --- - name: Set_fact _monitor_addresses - ipv4 ^ here fatal: [ceph-mont3]: FAILED! => msg: |- The task includes an option with an undefined variable. The error was: 'ansible.vars.hostvars.HostVarsVars object' has no attribute 'public_network'. 'ansible.vars.hostvars.HostVarsVars object' has no attribute 'public_network' The error appears to be in '/root/ceph-ansible/roles/ceph-facts/tasks/set_monitor_address.yml': line 2, column 3, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: --- - name: Set_fact _monitor_addresses - ipv4 ^ here fatal: [ceph-mont4]: FAILED! => msg: |- The task includes an option with an undefined variable. The error was: 'ansible.vars.hostvars.HostVarsVars object' has no attribute 'public_network'. 'ansible.vars.hostvars.HostVarsVars object' has no attribute 'public_network' The error appears to be in '/root/ceph-ansible/roles/ceph-facts/tasks/set_monitor_address.yml': line 2, column 3, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: --- - name: Set_fact _monitor_addresses - ipv4 ^ here fatal: [ceph-osdt1]: FAILED! => msg: |- The task includes an option with an undefined variable. The error was: 'ansible.vars.hostvars.HostVarsVars object' has no attribute 'public_network'. 'ansible.vars.hostvars.HostVarsVars object' has no attribute 'public_network' The error appears to be in '/root/ceph-ansible/roles/ceph-facts/tasks/set_monitor_address.yml': line 2, column 3, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: --- - name: Set_fact _monitor_addresses - ipv4 ^ here fatal: [ceph-osdt2]: FAILED! => msg: |- The task includes an option with an undefined variable. The error was: 'ansible.vars.hostvars.HostVarsVars object' has no attribute 'public_network'. 'ansible.vars.hostvars.HostVarsVars object' has no attribute 'public_network' The error appears to be in '/root/ceph-ansible/roles/ceph-facts/tasks/set_monitor_address.yml': line 2, column 3, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: --- - name: Set_fact _monitor_addresses - ipv4 ^ here fatal: [ceph-mdst1]: FAILED! => msg: |- The task includes an option with an undefined variable. The error was: 'ansible.vars.hostvars.HostVarsVars object' has no attribute 'public_network'. 'ansible.vars.hostvars.HostVarsVars object' has no attribute 'public_network' The error appears to be in '/root/ceph-ansible/roles/ceph-facts/tasks/set_monitor_address.yml': line 2, column 3, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: --- - name: Set_fact _monitor_addresses - ipv4 ^ here fatal: [ceph-mdst2]: FAILED! => msg: |- The task includes an option with an undefined variable. The error was: 'ansible.vars.hostvars.HostVarsVars object' has no attribute 'public_network'. 'ansible.vars.hostvars.HostVarsVars object' has no attribute 'public_network' The error appears to be in '/root/ceph-ansible/roles/ceph-facts/tasks/set_monitor_address.yml': line 2, column 3, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: --- - name: Set_fact _monitor_addresses - ipv4 ^ here

1 0

Subject: Ceph MGR k8sevents module - community feedback wanted
by John Mulligan 28 Apr '25

28 Apr '25

Some Background: https://tracker.ceph.com/issues/64213 In particular, the python cryptography module uses PyO3 and on some versions of python, Ceph, and a new enough cryptography module the mgr may fail to start certain modules with an error like: PyO3 modules may only be initialized once per interpreter process A few of us have been looking into the problem and have come up with a workaround for the Ceph dashboard. The dashboard makes occasional use of cryptography and other libraries that also use cryptography. However, this use is fairly easy to encapsulate and not called to frequently. The cephadm module uses a ssh library that more heavily uses cryptography and is the module that will have direct access. That leaves two other modules that make use of cryptography. The 'rook' module and 'k8sevents' module. Speaking with some of the other Ceph developers we think that the 'rook' module is being used actively on rook deployments to enable the dashboard when orchestrated with rook. However, we are unsure if anyone is using the 'k8sevents' module at all. --- To help us better understand the status of the k8sevents module we'd like to ask the community if anyone is successfully using the k8sevents module. This will help us understand if this module is something we might be able to ultimately remove. Note that only one module of either k8sevents and rook could run on systems with newer cryptography libraries - so if we kept both we'd have to figure out a way to make the two modules that use kubernetes libraries compatible - and that may be complicated. To summarize: please let us know if you are using the k8sevents module, that it works for you and accomplishes something useful. Feel free to also let us know if you're using rook and the 'rook' module with-or-without the k8smodule. Thanks!

1 0

Re: ceph mon out of quorom and cephfs degraded
by Eugen Block 28 Apr '25

28 Apr '25

Was there any issue in your network or anything? Something that would explain the MDS crash? I would recommend to scan syslog, dmesg etc. for anything suspicious. If you don't find anything, you might need to go through https://docs.ceph.com/en/latest/cephfs/disaster-recovery-experts/ to get your CephFS back up. Don't forget to create a backup of the journals. Read the instructions carefully, you might not need to do all the mentioned steps. I would start with (for both journals where applicable): cephfs-journal-tool journal export backup.bin cephfs-journal-tool event recover_dentries summary cephfs-journal-tool [--rank=<fs_name>:{mds-rank|all}] journal reset --yes-i-really-really-mean-it cephfs-table-tool all reset session But again, read the instructions carefully. This procedure is usually the last resort, I would first try to find out what happened causing this situation. Zitat von Amudhan P <amudhan83(a)gmail.com>: > I don't think I have a memory issue. > Sorry, for sending log as file Pastebin is not working due to size > limitation. > > Crash info > >> ceph crash info >> 2025-04-17T08:50:35.931485Z_0109f173-206c-471e-afac-c0d3e0aa2785 >> { >> "backtrace": [ >> "/lib64/libpthread.so.0(+0x12b20) [0x7fbd42bbdb20]", >> "/usr/lib64/ceph/libceph-common.so.2(+0x8ec7a00) [0x7fbd4c81da00]" >> ], >> "ceph_version": "16.2.6", >> "crash_id": >> "2025-04-17T08:50:35.931485Z_0109f173-206c-471e-afac-c0d3e0aa2785", >> "entity_name": "mds.mummasstrg.strg-node1.gchapr", >> "os_id": "centos", >> "os_name": "CentOS Linux", >> "os_version": "8", >> "os_version_id": "8", >> "process_name": "ceph-mds", >> "stack_sig": >> "5238fe1be0b82b479b10ee7d17b5ad3182cdc93ede581af63d627a472a4fcf9e", >> "timestamp": "2025-04-17T08:50:35.931485Z", >> "utsname_hostname": "strg-node1", >> "utsname_machine": "x86_64", >> "utsname_release": "5.10.0-8-amd64", >> "utsname_sysname": "Linux", >> "utsname_version": "#1 SMP Debian 5.10.46-5 (2021-09-23)" >> } >> > > On Thu, Apr 17, 2025 at 2:55 PM Eugen Block <eblock(a)nde.ag> wrote: > >> Oh right, I just noticed you had the status in your first message. Is >> there any chance the MDS is out of memory? Can you paste a complete >> (fresh) startup log on some pastebin or so? Also, a complete 'ceph >> crash info <crash>' could be useful. >> >> Zitat von Amudhan P <amudhan83(a)gmail.com>: >> >> > I have run the status and stat command below is the output. >> > >> > ceph -s >> > >> > cluster: >> > id: 7b3a4952-2131-11ec-94ce-0cc47a5ec98a >> > health: HEALTH_WARN >> > 2 failed cephadm daemon(s) >> > 1 filesystem is degraded >> > insufficient standby MDS daemons available >> > 7 daemons have recently crashed >> > >> > services: >> > mon: 3 daemons, quorum strg-node1,strg-node2,strg-node3 (age 20h) >> > mgr: strg-node2.unyimy(active, since 20h), standbys: >> strg-node1.ivkfid >> > mds: 1/1 daemons up >> > osd: 32 osds: 32 up (since 20h), 32 in (since 10w) >> > >> > data: >> > volumes: 0/1 healthy, 1 recovering >> > pools: 3 pools, 321 pgs >> > objects: 15.49M objects, 54 TiB >> > usage: 109 TiB used, 66 TiB / 175 TiB avail >> > pgs: 317 active+clean >> > 4 active+clean+scrubbing+deep >> > >> > >> > ceph mds stat >> > mumstrg:1/1 {0=mumstrg.strg-node1.gchapr=up:replay(laggy or crashed)} >> > >> > ceph osd lspools >> > 1 device_health_metrics >> > 2 cephfs.mumstrg.meta >> > 3 cephfs.mumstrg.data >> > >> > >> > >> > On Thu, Apr 17, 2025 at 10:33 AM Eugen Block <eblock(a)nde.ag> wrote: >> > >> >> What’s your overall Ceph status? It says data pool 3 not found. >> >> >> >> Zitat von Amudhan P <amudhan83(a)gmail.com>: >> >> >> >> > There are few more logs in MDS. I have highlighted few log lines >> which I >> >> am >> >> > not sure what it is. >> >> > >> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -79> >> >> > 2025-04-16T14:43:59.170+0000 7f74c6ecd780 5 asok(0x560a2c44e000) >> >> > register_command dump inode hook 0x560a2c354580 >> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -78> >> >> > 2025-04-16T14:43:59.170+0000 7f74c6ecd780 5 asok(0x560a2c44e000) >> >> > register_command exit hook 0x560a2c354580 >> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -77> >> >> > 2025-04-16T14:43:59.170+0000 7f74c6ecd780 5 asok(0x560a2c44e000) >> >> > register_command respawn hook 0x560a2c354580 >> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -76> >> >> > 2025-04-16T14:43:59.170+0000 7f74c6ecd780 5 asok(0x560a2c44e000) >> >> > register_command heap hook 0x560a2c354580 >> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -75> >> >> > 2025-04-16T14:43:59.170+0000 7f74b5030700 1 >> >> mds.mumstrg.strg-node3.xhxbwx >> >> > Updating MDS map to version 127517 f >> >> > rom mon.2 >> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -74> >> >> > 2025-04-16T14:43:59.170+0000 7f74c6ecd780 5 asok(0x560a2c44e000) >> >> > register_command cpu_profiler hook 0x560a2c35458 >> >> > >> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -73> >> >> > 2025-04-16T14:43:59.170+0000 7f74b302c700 5 >> >> > mds.beacon.mumstrg.strg-node3.xhxbwx Sending beacon up:boot seq 1 >> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -72> >> >> > 2025-04-16T14:43:59.170+0000 7f74b302c700 10 monclient: >> _send_mon_message >> >> > to mon.strg-node3 at v2:10.0.103.3:3300/ >> >> > >> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -71> >> >> > 2025-04-16T14:43:59.254+0000 7f74b5030700 1 >> >> mds.mumstrg.strg-node3.xhxbwx >> >> > Updating MDS map to version 127518 f >> >> > rom mon.2 >> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -70> >> >> > 2025-04-16T14:43:59.254+0000 7f74b5030700 10 monclient: _renew_subs >> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -69> >> >> > 2025-04-16T14:43:59.254+0000 7f74b5030700 10 monclient: >> _send_mon_message >> >> > to mon.strg-node3 at v2:10.0.103.3:3300/ >> >> > >> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -68> >> >> > 2025-04-16T14:43:59.254+0000 7f74b5030700 4 mds.0.purge_queue >> >> operator(): >> >> > data pool 3 not found in OSDMap >> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -67> >> >> > 2025-04-16T14:43:59.254+0000 7f74b5030700 5 asok(0x560a2c44e000) >> >> > register_command objecter_requests hook 0x560a2c >> >> > 3544c0 >> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -66> >> >> > 2025-04-16T14:43:59.254+0000 7f74b5030700 10 monclient: _renew_subs >> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -65> >> >> > 2025-04-16T14:43:59.254+0000 7f74b5030700 10 monclient: >> _send_mon_message >> >> > to mon.strg-node3 at v2:10.0.103.3:3300/ >> >> > >> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -64> >> >> > 2025-04-16T14:43:59.254+0000 7f74b5030700 10 log_channel(cluster) >> >> > update_config to_monitors: true to_syslog: false >> >> > syslog_facility: daemon prio: info to_graylog: false graylog_host: >> >> > 127.0.0.1 graylog_port: 12201) >> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -63> >> >> > 2025-04-16T14:43:59.254+0000 7f74b5030700 4 mds.0.purge_queue >> >> operator(): >> >> > data pool 3 not found in OSDMap >> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -62> >> >> > 2025-04-16T14:43:59.254+0000 7f74b5030700 4 mds.0.0 handle_osd_map >> epoch >> >> > 0, 0 new blocklist entries >> >> > Apr 16 20:13:59 strg-node3 bash[59114]: debug -61> >> >> > 2025-04-16T14:43:59.254+0000 7f74b5030700 1 mds.0.127518 >> handle_mds_map >> >> i >> >> > am now mds.0.127518 >> >> > >> >> >> Apr 16 20:13:59 strg-node3 bash[59114]: debug -60> >> >> >> 2025-04-16T14:43:59.254+0000 7f74b5030700 1 mds.0.127518 >> handle_mds_map >> >> >> state change up:boot --> up:replay >> >> >> Apr 16 20:13:59 strg-node3 bash[59114]: debug -59> >> >> >> 2025-04-16T14:43:59.254+0000 7f74b5030700 5 >> >> >> mds.beacon.mummasstrg.strg-node3.xhxbwx set_want_state: up:boot -> >> >> up:replay >> >> >> Apr 16 20:13:59 strg-node3 bash[59114]: debug -58> >> >> >> 2025-04-16T14:43:59.254+0000 7f74b5030700 1 mds.0.127518 >> replay_start >> >> >> *Apr 16 20:13:59 strg-node3 bash[59114]: debug -57> >> >> >> 2025-04-16T14:43:59.254+0000 7f74b5030700 1 mds.0.127518 waiting >> for >> >> >> osdmap 45749 (which blocklists prior instance)* >> >> >> Apr 16 20:13:59 strg-node3 bash[59114]: debug -56> >> >> >> 2025-04-16T14:43:59.254+0000 7f74b5030700 10 monclient: >> >> _send_mon_message >> >> >> to mon.strg-node3 at v2:10.0.103.3:3300/0 >> >> >> *Apr 16 20:13:59 strg-node3 bash[59114]: debug -55> >> >> >> 2025-04-16T14:43:59.254+0000 7f74b5030700 4 mds.0.purge_queue >> >> operator(): >> >> >> data pool 3 not found in OSDMap* >> >> >> >> >> > >> >> > >> >> > On Thu, Apr 17, 2025 at 7:06 AM Amudhan P <amudhan83(a)gmail.com> >> wrote: >> >> > >> >> >> Eugen, >> >> >> >> >> >> This is the output for the command >> >> >> cephfs-journal-tool --rank=mumstrg:all --journal=purge_queue journal >> >> >> inspect >> >> >> Overall journal integrity: OK >> >> >> cephfs-journal-tool --rank=mumstrg:all --journal=mdlog journal >> inspect >> >> >> Overall journal integrity: OK >> >> >> >> >> >> On Thu, Apr 17, 2025 at 2:59 AM Eugen Block <eblock(a)nde.ag> wrote: >> >> >> >> >> >>> I think either your mdlog or the purge_queue journal is corrupted: >> >> >>> >> >> >>> 2025-04-16T09:59:30.146+0000 7f43cf872700 2 mds.0.127506 Booting: >> 2: >> >> >>> waiting for purge queue recovered >> >> >>> Apr 16 15:29:30 strg-node4 bash[7566]: debug -1> >> >> >>> 2025-04-16T09:59:30.146+0000 7f43d9085700 10 monclient: >> >> get_auth_request >> >> >>> con 0x562856a25400 auth_method 0 >> >> >>> Apr 16 15:29:30 strg-node4 bash[7566]: debug 0> >> >> >>> 2025-04-16T09:59:30.230+0000 7f43ce06f700 -1 *** Caught signal >> >> >>> (Segmentation fault) ** >> >> >>> Apr 16 15:29:30 strg-node4 bash[7566]: in thread 7f43ce06f700 >> >> >>> thread_name:md_log_replay >> >> >>> >> >> >>> Can you paste the output of this command? >> >> >>> >> >> >>> cephfs-journal-tool --rank={YOUR_CEPH_FS}:all --journal=purge_queue >> >> >>> journal inspect >> >> >>> cephfs-journal-tool --rank={YOUR_CEPH_FS}:all --journal=mdlog >> journal >> >> >>> inspect >> >> >>> >> >> >>> I expect one or more damaged entries. Check this thread for more >> >> details: >> >> >>> >> >> >>> https://www.spinics.net/lists/ceph-users/msg80124.html >> >> >>> >> >> >>> You should try to backup the journal, but in my case that wasn't >> >> >>> possible, so I had no other choice than resetting it. >> >> >>> >> >> >>> Zitat von Amudhan P <amudhan83(a)gmail.com>: >> >> >>> >> >> >>> > Hi, >> >> >>> > >> >> >>> > I am having 2 problems with my Ceph version 16.2.6 >> >> >>> > (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable) >> deployed >> >> >>> thru >> >> >>> > cephadm. >> >> >>> > >> >> >>> > First issue :- >> >> >>> > 1 out 3 mon service went out of quorum . >> >> >>> > When restarted service it comes normal but after a few minutes in >> >> ceph >> >> >>> > watch log it reports slow ops and mon goes out of quorum. >> >> >>> > Node where this mon service failed had one weird thing that I >> could >> >> see >> >> >>> 40% >> >> >>> > of wait in the top command. But I don't see any error in dmesg or >> >> >>> anything >> >> >>> > related to drive IO error. >> >> >>> > Below are the logs that were printed in ceph watch command. >> >> >>> > >> >> >>> > 2025-04-16T09:30:00.000393+0530 mon.strg-node2 [WRN] [WRN] >> MON_DOWN: >> >> 1/3 >> >> >>> > mons down, quorum strg-node2,strg-node3 >> >> >>> > 2025-04-16T09:30:00.000416+0530 mon.strg-node2 [WRN] >> >> mon.strg-node1 >> >> >>> > (rank 0) addr [v2:10.0.103.1:3300/0,v1:10.0.103.1:6789/0] is down >> >> (out >> >> >>> of >> >> >>> > quorum) >> >> >>> > >> >> >>> > For now this is not appearing again. >> >> >>> > >> >> >>> > >> >> >>> > Second issue Cephfs degraded :- >> >> >>> > I have 2 MDS services running in 2 different nodes. Both are in a >> >> >>> stopped >> >> >>> > state. >> >> >>> > when running Ceph -s command >> >> >>> > >> >> >>> > cluster: >> >> >>> > id: 7b3a4952-2131-11ec-94ce-0cc47a5ec98a >> >> >>> > health: HEALTH_WARN >> >> >>> > 2 failed cephadm daemon(s) >> >> >>> > 1 filesystem is degraded >> >> >>> > insufficient standby MDS daemons available >> >> >>> > >> >> >>> > services: >> >> >>> > mon: 3 daemons, quorum strg-node1,strg-node2,strg-node3 (age >> 4h) >> >> >>> > mgr: strg-node2.unyimy(active, since 4h), standbys: >> >> >>> strg-node1.ivkfid >> >> >>> > mds: 1/1 daemons up >> >> >>> > osd: 32 osds: 32 up (since 4h), 32 in (since 10w) >> >> >>> > >> >> >>> > data: >> >> >>> > volumes: 0/1 healthy, 1 recovering >> >> >>> > pools: 3 pools, 321 pgs >> >> >>> > objects: 15.49M objects, 54 TiB >> >> >>> > usage: 109 TiB used, 66 TiB / 175 TiB avail >> >> >>> > pgs: 321 active+clean >> >> >>> > >> >> >>> > Volume shows recovering but there wasn't any progress till now >> even >> >> >>> manual >> >> >>> > start mds service fails again. In Ceph -s command under services >> it >> >> >>> shows >> >> >>> > mds up no any mds service is running. >> >> >>> > >> >> >>> > Below is a log snip from one of the mds service. >> >> >>> > >> >> >>> > >> >> >>> > -25> 2025-04-16T09:59:29.954+0000 7f43d0874700 1 >> >> >>> > mds.0.journaler.pq(ro) _finish_read_head loghead(trim 13967032320, >> >> ex> >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -24> >> >> >>> > 2025-04-16T09:59:29.954+0000 7f43d0874700 1 >> mds.0.journaler.pq(ro) >> >> >>> probing >> >> >>> > for end of the log >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -23> >> >> >>> > 2025-04-16T09:59:29.954+0000 7f43d9085700 10 monclient: >> >> get_auth_request >> >> >>> > con 0x562856a17400 auth_method 0 >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -22> >> >> >>> > 2025-04-16T09:59:29.954+0000 7f43d8884700 10 monclient: >> >> get_auth_request >> >> >>> > con 0x562856a17c00 auth_method 0 >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -21> >> >> >>> > 2025-04-16T09:59:29.974+0000 7f43cf071700 1 >> >> mds.0.journaler.mdlog(ro) >> >> >>> > recover start >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -20> >> >> >>> > 2025-04-16T09:59:29.974+0000 7f43cf071700 1 >> >> mds.0.journaler.mdlog(ro) >> >> >>> > read_head >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -19> >> >> >>> > 2025-04-16T09:59:29.974+0000 7f43cf071700 4 mds.0.log Waiting for >> >> >>> journal >> >> >>> > 0x200 to recover... >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -18> >> >> >>> > 2025-04-16T09:59:29.974+0000 7f43d8083700 10 monclient: >> >> get_auth_request >> >> >>> > con 0x562856a25000 auth_method 0 >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -17> >> >> >>> > 2025-04-16T09:59:29.998+0000 7f43d0874700 1 >> mds.0.journaler.pq(ro) >> >> >>> > _finish_probe_end write_pos = 13968309289 (hea> >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -16> >> >> >>> > 2025-04-16T09:59:29.998+0000 7f43d0874700 4 mds.0.purge_queue >> >> >>> operator(): >> >> >>> > open complete >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -15> >> >> >>> > 2025-04-16T09:59:29.998+0000 7f43d0874700 1 >> mds.0.journaler.pq(ro) >> >> >>> > set_writeable >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -14> >> >> >>> > 2025-04-16T09:59:29.998+0000 7f43cf872700 1 >> >> mds.0.journaler.mdlog(ro) >> >> >>> > _finish_read_head loghead(trim 189741504921> >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -13> >> >> >>> > 2025-04-16T09:59:29.998+0000 7f43cf872700 1 >> >> mds.0.journaler.mdlog(ro) >> >> >>> > probing for end of the log >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -12> >> >> >>> > 2025-04-16T09:59:30.002+0000 7f43d9085700 10 monclient: >> >> get_auth_request >> >> >>> > con 0x562856a25c00 auth_method 0 >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -11> >> >> >>> > 2025-04-16T09:59:30.098+0000 7f43cf872700 1 >> >> mds.0.journaler.mdlog(ro) >> >> >>> > _finish_probe_end write_pos = 1897428915052> >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -10> >> >> >>> > 2025-04-16T09:59:30.098+0000 7f43cf071700 4 mds.0.log Journal >> 0x200 >> >> >>> > recovered. >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -9> >> >> >>> > 2025-04-16T09:59:30.098+0000 7f43cf071700 4 mds.0.log Recovered >> >> journal >> >> >>> > 0x200 in format 1 >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -8> >> >> >>> > 2025-04-16T09:59:30.098+0000 7f43cf071700 2 mds.0.127506 >> Booting: 1: >> >> >>> > loading/discovering base inodes >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -7> >> >> >>> > 2025-04-16T09:59:30.098+0000 7f43cf071700 0 mds.0.cache creating >> >> system >> >> >>> > inode with ino:0x100 >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -6> >> >> >>> > 2025-04-16T09:59:30.098+0000 7f43cf071700 0 mds.0.cache creating >> >> system >> >> >>> > inode with ino:0x1 >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -5> >> >> >>> > 2025-04-16T09:59:30.098+0000 7f43d8884700 10 monclient: >> >> get_auth_request >> >> >>> > con 0x562856a25800 auth_method 0 >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -4> >> >> >>> > 2025-04-16T09:59:30.098+0000 7f43d8083700 10 monclient: >> >> get_auth_request >> >> >>> > con 0x562856a5dc00 auth_method 0 >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -3> >> >> >>> > 2025-04-16T09:59:30.146+0000 7f43cf872700 2 mds.0.127506 >> Booting: 2: >> >> >>> > replaying mds log >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -2> >> >> >>> > 2025-04-16T09:59:30.146+0000 7f43cf872700 2 mds.0.127506 >> Booting: 2: >> >> >>> > waiting for purge queue recovered >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug -1> >> >> >>> > 2025-04-16T09:59:30.146+0000 7f43d9085700 10 monclient: >> >> get_auth_request >> >> >>> > con 0x562856a25400 auth_method 0 >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: debug 0> >> >> >>> > 2025-04-16T09:59:30.230+0000 7f43ce06f700 -1 *** Caught signal >> >> >>> > (Segmentation fault) ** >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: in thread 7f43ce06f700 >> >> >>> > thread_name:md_log_replay >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: ceph version 16.2.6 >> >> >>> > (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable) >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: 1: >> >> >>> /lib64/libpthread.so.0(+0x12b20) >> >> >>> > [0x7f43dd293b20] >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: 2: >> >> >>> > /usr/lib64/ceph/libceph-common.so.2(+0x8ec7a00) [0x7f43e6ef3a00] >> >> >>> > Apr 16 15:29:30 strg-node4 bash[7566]: NOTE: a copy of the >> >> executable, >> >> >>> or >> >> >>> > `objdump -rdS <executable>` is needed to interpret this. >> >> >>> > >> >> >>> > >> >> >>> > Not sure what caused the issue. I couldn't find any resources to >> fix >> >> >>> this >> >> >>> > issue. >> >> >>> > Need help from someone to bring the ceph cluster online. >> >> >>> > _______________________________________________ >> >> >>> > ceph-users mailing list -- ceph-users(a)ceph.io >> >> >>> > To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >>> >> >> >>> >> >> >>> _______________________________________________ >> >> >>> ceph-users mailing list -- ceph-users(a)ceph.io >> >> >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >>> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>

2 6