- ceph-users - lists.ceph.io

MDS stuck in rejoin
by Frank Schilder 09 Nov '23

09 Nov '23

Hi all, we had a client with the warning "[WRN] MDS_CLIENT_OLDEST_TID: 1 clients failing to advance oldest client/flush tid". I looked at the client and there was nothing going on, so I rebooted it. After the client was back, the message was still there. To clean this up I failed the MDS. Unfortunately, the MDS that took over is remained stuck in rejoin without doing anything. All that happened in the log was: [root@ceph-10 ceph]# tail -f ceph-mds.ceph-10.log 2023-07-20T15:54:29.147+0200 7fedb9c9f700 1 mds.2.896604 rejoin_start 2023-07-20T15:54:29.161+0200 7fedb9c9f700 1 mds.2.896604 rejoin_joint_start 2023-07-20T15:55:28.005+0200 7fedb9c9f700 1 mds.ceph-10 Updating MDS map to version 896614 from mon.4 2023-07-20T15:56:00.278+0200 7fedb9c9f700 1 mds.ceph-10 Updating MDS map to version 896615 from mon.4 [...] 2023-07-20T16:02:54.935+0200 7fedb9c9f700 1 mds.ceph-10 Updating MDS map to version 896653 from mon.4 2023-07-20T16:03:07.276+0200 7fedb9c9f700 1 mds.ceph-10 Updating MDS map to version 896654 from mon.4 After some time I decided to give another fail a try and, this time, the replacement daemon went to active state really fast. If I have a message like the above, what is the clean way of getting the client clean again (version: 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable))? Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 16

HDD cache
by Peter 09 Nov '23

09 Nov '23

Hi All, I note that HDD cluster commit delay improves after i turn off HDD cache. However, i also note that not all HDDs are able to turn off the cache. special I found that two HDD with same model number, one can turn off, anther doesn't. i guess i have my system config or something different setting with two HDDs. Below is my command to turn off the HDD cache. root@lahost008:~# sdparm --set WCE=0 /dev/sdb /dev/sdb: ATA ST10000NE000-3AP EN01 root@lahost008:~# cat /sys/block/sdb/queue/write_cache write through I also tried sdparm to run "sdparm --set WCE=0 /dev/sdb" I got the same result. Anyone experienced this can advise? Thanks a lot

4 4

Memory footprint of increased PG number
by Nicola Mori 09 Nov '23

09 Nov '23

Dear Ceph user, I'm wondering how much an increase of PG number would impact on the memory occupancy of OSD daemons. In my cluster I currently have 512 PGs and I would like to increase it to 1024 to mitigate some disk occupancy issues, but having machines with low amount of memory (down to 24 GB for 16 OSDs) I fear this could kill my cluster. Is it possible to evaluate the relative increase in OSD memory footprint when doubling the number of PGs (hopefully not a linear scaling)? Or is there a way to experiment without crashing everything? Thank you.

2 2

High iowait when using Ceph NVME
by Huy Nguyen 09 Nov '23

09 Nov '23

Hi, Currently, I'm testing Ceph v17.2.7 with NVMe. When mapping an rbd image to physical compute host, "fio bs=4k iodepth=128 randwrite" give 150k IOPS. I have a VM that located within the compute host, and the fio give ~40k IOPS and with 50 %iowait. I know there is a bottleneck, I'm not sure where it is though I have tried to enabled iothread in virsh but nothing change. Anyone have any ideal? Thanks in advance.

1 0

Found unknown daemon type ceph-exporter on host after upgrade to 17.2.7
by Dmitry Melekhov 09 Nov '23

09 Nov '23

Hello! I see [WRN] Found unknown daemon type ceph-exporter on host for all 3 ceph servers in logs, after upgrade to 17.2.7 from 17.2.6 in dashboard and cephadm ['--image', 'quay.io/ceph/ceph@sha256:1fcdbead4709a7182047f8ff9726e0f17b0b209aaa6656c5c8b2339b818e70bb', '--timeout', '895', 'ls'] 2023-11-07 16:15:37,531 7fddb699b740 WARNING version for unknown daemon type ceph-exporter in cephadm.log There were no such messages before. I guess this will not create any problem, but is there any way to fix this?

4 9

Crush map & rule
by Albert Shih 09 Nov '23

09 Nov '23

Hi everyone, I'm totally newbie with ceph, so sorry if I'm asking some stupid question. I'm trying to understand how the crush map & rule work, my goal is to have two groups of 3 servers, so I'm using “row” bucket ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 59.38367 root default -15 59.38367 zone City -17 29.69183 row primary -3 9.89728 host server1 0 ssd 3.49309 osd.0 up 1.00000 1.00000 1 ssd 1.74660 osd.1 up 1.00000 1.00000 2 ssd 1.74660 osd.2 up 1.00000 1.00000 3 ssd 2.91100 osd.3 up 1.00000 1.00000 -5 9.89728 host server2 4 ssd 1.74660 osd.4 up 1.00000 1.00000 5 ssd 1.74660 osd.5 up 1.00000 1.00000 6 ssd 2.91100 osd.6 up 1.00000 1.00000 7 ssd 3.49309 osd.7 up 1.00000 1.00000 -7 9.89728 host server3 8 ssd 3.49309 osd.8 up 1.00000 1.00000 9 ssd 1.74660 osd.9 up 1.00000 1.00000 10 ssd 2.91100 osd.10 up 1.00000 1.00000 11 ssd 1.74660 osd.11 up 1.00000 1.00000 -19 29.69183 row secondary -9 9.89728 host server4 12 ssd 1.74660 osd.12 up 1.00000 1.00000 13 ssd 1.74660 osd.13 up 1.00000 1.00000 14 ssd 3.49309 osd.14 up 1.00000 1.00000 15 ssd 2.91100 osd.15 up 1.00000 1.00000 -11 9.89728 host server5 16 ssd 1.74660 osd.16 up 1.00000 1.00000 17 ssd 1.74660 osd.17 up 1.00000 1.00000 18 ssd 3.49309 osd.18 up 1.00000 1.00000 19 ssd 2.91100 osd.19 up 1.00000 1.00000 -13 9.89728 host server6 20 ssd 1.74660 osd.20 up 1.00000 1.00000 21 ssd 1.74660 osd.21 up 1.00000 1.00000 22 ssd 2.91100 osd.22 up 1.00000 1.00000 and I want to create a some rules, first I like to have a rule «replica» (over host) inside the «row» primary a rule «erasure» (over host) inside the «row» primary but also two crush rule between primary/secondary, meaning I like to have a replica (with only 1 copy of course) of pool from “row” primary to secondary. How can I achieve that ? Regards -- Albert SHIH 🦫 🐸 mer. 08 nov. 2023 18:37:54 CET

2 3

Question about PG mgr/balancer/crush_compat_metrics
by bryansoong21＠gmail.com 08 Nov '23

08 Nov '23

Hello, We are using a Ceph Pacific (16.2.10) cluster and enabled the balancer module, but the usage of some OSDs keeps growing and reached up to mon_osd_nearfull_ratio, which we use 85% by default, and we think the balancer module should do some balancer work. So I checked our balancer configuration and found that our "crush_compat_metrics" is set to "pgs,objects,bytes", and this three values are used in src.pybind.mgr.balancer.module.Module.calc_eval. However, when doing the actual balance task, only the first key is used to do the auto balance, in src.pybind.mgr.balancer.module.Module.do_crush_compat: metrics = self.get_module_option('crush_compat_metrics').split(',') key = metrics[0] # balancing using the first score metric My concern is, any reason why we calculate the balancing using the three items but only do the balance using the first one? Thanks.

2 1

Ceph Leadership Team Weekly Meeting Minutes 2023-11-08
by Patrick Donnelly 08 Nov '23

08 Nov '23

Hello all, Here are the minutes from today's meeting. - New time for CDM APAC to increase participation - 9.30 - 11.30 pm PT seems like the most popular based on https://doodle.com/meeting/participate/id/aM9XGZ3a/vote - One more week for more feedback; please ask more APAC folks to suggest their preferred times. - [Ernesto] Revamp Ansible/Ceph-Ansible for non-containerized users? - open nebula / proxmox - solicit maintainers for ceph-ansible on the ML - 18.2.1 - yuri: approval email sent out a few days ago; waiting on some approvals - Blocker: - https://tracker.ceph.com/issues/63391 - lab upgrades (Laura will help Yuri coordinate) - Next Pacific release being worked on in background by Yuri. - https://pad.ceph.com/p/pacific_16.2.15 - Try v16.2.15 milestone to help prune PRs - https://github.com/ceph/ceph/milestone/17 - [Nizam] Ceph News Ticker - Ceph Dashboard - Notify when new release is available (display changelogs) - Display important ceph events - CVEs, critical bug fixes - Maybe newly added blog posts or informations regarding the upcoming group meetings? - User + Dev meeting next week - Topics include migration between EC profiles and challenges related to RGW zone replication - Casey can attend end of meeting - open nebula folks planning to do webinar; looking for speakers -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

ceph storage pool error
by necoe0147＠gmail.com 08 Nov '23

08 Nov '23

Ceph 3 clusters are running and the 3rd cluster gave an error, it is currently offline. I want to get all the remaining data in 2 clusters. Instead of fixing ceph, I just want to save the data. How can I access this data and connect to the pool? Can you help me?1 and 2 clusters are working. I want to view my data from them and then transfer them to another place. How can I do this? I have never used Ceph before.

2 1

Ceph OSD reported Slow operations
by prabhav＠cdac.in 08 Nov '23

08 Nov '23

In a production setup of 36 OSDs( SAS disks) totalling 180 TB allocated to a single Ceph Cluster with 3 monitors and 3 managers. There were 830 volumes and VMs created in Openstack with Ceph as a backend. On Sep 21, users reported slowness in accessing the VMs. Analysing the logs lead us to problem with SAS , Network congestion and Ceph configuration( as all default values were used). We updated the Network from 1Gbps to 10Gbps for public and cluster networking. There was no change. The ceph benchmark performance showed that 28 OSDs out of 36 OSDs reported very low IOPS of 30 to 50 while the remaining showed 300+ IOPS. We gradually started reducing the load on the ceph cluster and now the volumes count is 650. Now the slow operations has gradually reduced but I am aware that this is not the solution. Ceph configuration is updated with increasing the osd_journal_size to 10 GB, osd_max_backfills = 1 osd_recovery_max_active = 1 osd_recovery_op_priority = 1 bluestore_cache_trim_max_skip_pinned=10000 After one month, now we faced another issue with Mgr daemon stopped in all 3 quorums and 16 OSDs went down. From the ceph-mon,ceph-mgr.log could not get the reason. Please guide me as its a production setup

3 3