- ceph-users - lists.ceph.io

RGW multisite slowness issue due to the "304 Not Modified" responses on primary zone
by praveenkumargpk17＠gmail.com 01 May '24

01 May '24

Hi, We have 2 clusters (v18.2.1) primarily used for RGW which has over 2+ billion RGW objects. They are also in multisite configuration totaling to 2 zones and we've got around 2 Gbps of bandwidth dedicated (P2P) for the multisite traffic. We see that using "radosgw-admin sync status" on the zone 2, all the 128 shards are recovering and unfortunately there is very less data transfer from primary zone ie., the link utilization is barely 100 Mbps / 2 Gbps. Our objects are quite small as well like avg. of 1 MB in size. On further inspection, we noticed the rgw access the logs at primary site are mostly yielding "304 Not Modified" for RGWs at site-2. Is this expected? Here are some of the logs (information is redacted) root@host-04:~# tail -f /var/log/haproxy-msync.log Feb 12 05:06:51 host-04 haproxy[971171]: 10.1.85.14:33730 [12/Feb/2024:05:06:51.047] https~ backend/host-04-msync 0/0/0/2/2 304 143 - - ---- 56/55/1/0/0 0/0 "GET /bucket1/object1.jpg?rgwx-zonegroup=71dceb3d-3092-4dc6-897f-a9abf60c9972&rgwx-prepend-metadata=true&rgwx-sync-manifest&rgwx-sync-cloudtiered&rgwx-skip-decrypt&rgwx-if-not-replicated-to=a8204ce2-b69e-4d90-bca1-93edd05a1a29%3Abucket1%3A8b96aea5-c763-40a3-8430-efd67cff0c62.20010.7 HTTP/1.1" Feb 12 05:06:51 host-04 haproxy[971171]: 10.1.85.14:59730 [12/Feb/2024:05:06:51.048] https~ backend/host-04-msync 0/0/0/2/2 304 143 - - ---- 56/55/3/1/0 0/0 "GET /bucket1/object91.jpg?rgwx-zonegroup=71dceb3d-3092-4dc6-897f-a9abf60c9972&rgwx-prepend-metadata=true&rgwx-sync-manifest&rgwx-sync-cloudtiered&rgwx-skip-decrypt&rgwx-if-not-replicated-to=a8204ce2-b69e-4d90-bca1-93edd05a1a29%3Abucket1%3A8b96aea5-c763-40a3-8430-efd67cff0c62.20010.7 HTTP/1.1" We also took a look at our Grafana instance and out of 1000 requests / second, 200 requests are getting "200 OK" and 800 requests are getting "304 Not Modified". Sync threads are run on only 2 rgw daemons per zone and are behind a Load Balancer. "# radosgw-admin sync error list" also contains around 20 errors which are mostly automatically recoverable. As we understand, does it mean that RGW multisite sync logs in the log pool are yet to be generated or some sort? Please provide us some insights and let us know how to resolve this. Thanks, Praveen

4 4

Re: Stuck OSD service specification - can't remove
by Wang Jie 30 Apr '24

30 Apr '24

Hello David, did you resolve it? I have the same problem for rgw. I upgraded from N to P. Regards, Jie

1 0

Remove an OSD with hardware issue caused rgw 503
by Mary Zhang 30 Apr '24

30 Apr '24

Hi, We recently removed an osd from our Cepth cluster. Its underlying disk has a hardware issue. We use command: ceph orch osd rm osd_id --zap During the process, sometimes ceph cluster enters warning state with slow ops on this osd. Our rgw also failed to respond to requests and returned 503. We restarted rgw daemon to make it work again. But the same failure occured from time to time. Eventually we noticed that rgw 503 error is a result of osd slow ops. Our cluster has 18 hosts and 210 OSDs. We expect remove an osd with hardware issue won't impact cluster performance & rgw availbility. Is our expectation reasonable? What's the best way to handle osd with hardware failures? Thank you in advance for any comments or suggestions. Best Regards, Mary Zhang

4 11

Ceph Squid released?
by Robert Sander 30 Apr '24

30 Apr '24

Hi, https://www.linuxfoundation.org/press/introducing-ceph-squid-the-future-of-… Does the LF know more than the mailing list? Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

4 6

SPDK with cephadm and reef
by R A 30 Apr '24

30 Apr '24

Hello Community, is there a guide / documentation how to configure spdk with cephadm (running in containers) in reef? BR

1 0

Re: MDS Behind on Trimming...
by Erich Weiler 29 Apr '24

29 Apr '24

Hi Xiubo, Nevermind I was wrong, most the blocked ops were 12 hours old. Ug. I restarted the MDS daemon to clear them. I just reset to having one active MDS instead of two, let's see if that makes a difference. I am beginning to think it may be impossible to catch the logs that matter here. I feel like sometimes the blocked ops are just waiting because of load and sometimes they are waiting because they are stuck. But, it's really hard to tell which, without waiting a while. But, I can't wait while having debug turned on because my root disks (which are 150 GB large) fill up with debug logs in 20 minutes. So it almost seems that unless I could somehow store many TB of debug logs we won't be able to catch this. Let's see how having one MDS helps. Or maybe I actually need like 4 MDSs because the load is too high for only one or two. I don't know. Or maybe it's the lock issue you've been working on. I guess I can test the lock order fix when it's available to test. -erich On 4/19/24 7:26 AM, Erich Weiler wrote: > So I woke up this morning and checked the blocked_ops again, there were > 150 of them. But the age of each ranged from 500 to 4300 seconds. So > it seems as if they are eventually being processed. > > I wonder if we are thinking about this in the wrong way? Maybe I should > be *adding* MDS daemons because my current ones are overloaded? > > Can a single server hold multiple MDS daemons? Right now I have three > physical servers each with one MDS daemon on it. > > I can still try reducing to one. And I'll keep an eye on blocked ops to > see if any get to a very old age (and are thus wedged). > > -erich > > On 4/18/24 8:55 PM, Xiubo Li wrote: >> Okay, please try it to set only one active mds. >> >> >> On 4/19/24 11:54, Erich Weiler wrote: >>> We have 2 active MDS daemons and one standby. >>> >>> On 4/18/24 8:52 PM, Xiubo Li wrote: >>>> BTW, how man active mds you are using ? >>>> >>>> >>>> On 4/19/24 10:55, Erich Weiler wrote: >>>>> OK, I'm sure I caught it in the right order this time, the logs >>>>> should definitely show when the blocked/slow requests start. Check >>>>> out these logs and dumps: >>>>> >>>>> http://hgwdev.gi.ucsc.edu/~weiler/ >>>>> >>>>> It's a 762 MB tarball but it uncompresses to 16 GB. >>>>> >>>>> -erichll >>>>> >>>>> >>>>> On 4/18/24 6:57 PM, Xiubo Li wrote: >>>>>> Okay, could you try this with 18.2.0 ? >>>>>> >>>>>> I just double it was introduce by: >>>>>> >>>>>> commit e610179a6a59c463eb3d85e87152ed3268c808ff >>>>>> Author: Patrick Donnelly <pdonnell(a)redhat.com> >>>>>> Date: Mon Jul 17 16:10:59 2023 -0400 >>>>>> >>>>>> mds: drop locks and retry when lock set changes >>>>>> >>>>>> An optimization was added to avoid an unnecessary gather on >>>>>> the inode >>>>>> filelock when the client can safely get the file size without >>>>>> also >>>>>> getting issued the requested caps. However, if a retry of >>>>>> getattr >>>>>> is necessary, this conditional inclusion of the inode filelock >>>>>> can cause lock-order violations resulting in deadlock. >>>>>> >>>>>> So, if we've already acquired some of the inode's locks then >>>>>> we must >>>>>> drop locks and retry. >>>>>> >>>>>> Fixes: https://tracker.ceph.com/issues/62052 >>>>>> Fixes: c822b3e2573578c288d170d1031672b74e02dced >>>>>> Signed-off-by: Patrick Donnelly <pdonnell(a)redhat.com> >>>>>> (cherry picked from commit >>>>>> b5719ac32fe6431131842d62ffaf7101c03e9bac) >>>>>> >>>>>> >>>>>> On 4/19/24 09:54, Erich Weiler wrote: >>>>>>> I'm on 18.2.1. I think I may have gotten the timing off on the >>>>>>> logs and dumps so I'll try again. Just really hard to capture >>>>>>> because I need to kind of be looking at it in real time to >>>>>>> capture it. Hang on, lemme see if I can get another capture... >>>>>>> >>>>>>> -erich >>>>>>> >>>>>>> On 4/18/24 6:35 PM, Xiubo Li wrote: >>>>>>>> >>>>>>>> BTW, which ceph version you are using ? >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 4/12/24 04:22, Erich Weiler wrote: >>>>>>>>> BTW - it just happened again, I upped the debugging settings as >>>>>>>>> you instructed and got more dumps (then returned the debug >>>>>>>>> settings to normal). >>>>>>>>> >>>>>>>>> Attached are the new dumps. >>>>>>>>> >>>>>>>>> Thanks again, >>>>>>>>> erich >>>>>>>>> >>>>>>>>> On 4/9/24 9:00 PM, Xiubo Li wrote: >>>>>>>>>> >>>>>>>>>> On 4/10/24 11:48, Erich Weiler wrote: >>>>>>>>>>>>> Dos that mean it could be the locker order bug >>>>>>>>>>>>> (https://tracker.ceph.com/issues/62123) as Xiubo suggested? >>>>>>>>>>>> >>>>>>>>>>>> I have raised one PR to fix the lock order issue, if >>>>>>>>>>>> possible please have a try to see could it resolve this issue. >>>>>>>>>>> >>>>>>>>>>> Thank you! Yeah, this issue is happening every couple days >>>>>>>>>>> now. It just happened again today and I got more MDS dumps. >>>>>>>>>>> If it would help, let me know and I can send them! >>>>>>>>>>> >>>>>>>>>> Once this happen if you could enable the mds debug logs will >>>>>>>>>> be better: >>>>>>>>>> >>>>>>>>>> debug mds = 20 >>>>>>>>>> >>>>>>>>>> debug ms = 1 >>>>>>>>>> >>>>>>>>>> And then provide the debug logs together with the MDS dumps. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> I assume if this fix is approved and backported it will then >>>>>>>>>>> appear in like 18.2.3 or something? >>>>>>>>>>> >>>>>>>>>> Yeah, it will be backported after being well tested. >>>>>>>>>> >>>>>>>>>> - Xiubo >>>>>>>>>> >>>>>>>>>>> Thanks again, >>>>>>>>>>> erich >>>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >>

2 3

ceph recipe for nfs exports
by Roberto Maggi ＠ Debian 29 Apr '24

29 Apr '24

Hi you all, I'm almost new to ceph and I'm understanding, day by day, why the official support is so expansive :) I setting up a ceph nfs network cluster whose recipe can be found here below. ####################### --> cluster creation cephadm bootstrap --mon-ip 10.20.20.81 --cluster-network 10.20.20.0/24 --fsid $FSID --initial-dashboard-user adm \ --initial-dashboard-password 'Hi_guys' --dashboard-password-noupdate --allow-fqdn-hostname --ssl-dashboard-port 443 \ --dashboard-crt /etc/ssl/wildcard.it/wildcard.it.crt --dashboard-key /etc/ssl/wildcard.it/wildcard.it.key \ --allow-overwrite --cleanup-on-failure cephadm shell --fsid $FSID -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring cephadm add-repo --release reef && cephadm install ceph-common --> adding hosts and set labels for IP in $(grep ceph /etc/hosts | awk '{print $1}') ; do ssh-copy-id -f -i /etc/ceph/ceph.pub root@$IP ; done ceph orch host add cephstage01 10.20.20.81 --labels _admin,mon,mgr,prometheus,grafana ceph orch host add cephstage02 10.20.20.82 --labels _admin,mon,mgr,prometheus,grafana ceph orch host add cephstage03 10.20.20.83 --labels _admin,mon,mgr,prometheus,grafana ceph orch host add cephstagedatanode01 10.20.20.84 --labels osd,nfs,prometheus ceph orch host add cephstagedatanode02 10.20.20.85 --labels osd,nfs,prometheus ceph orch host add cephstagedatanode03 10.20.20.86 --labels osd,nfs,prometheus --> network setup and daemons deploy ceph config set mon public_network 10.20.20.0/24,192.168.7.0/24 ceph orch apply mon --placement="cephstage01:10.20.20.81,cephstage02:10.20.20.82,cephstage03:10.20.20.83" ceph orch apply mgr --placement="cephstage01:10.20.20.81,cephstage02:10.20.20.82,cephstage03:10.20.20.83" ceph orch apply prometheus --placement="cephstage01:10.20.20.81,cephstage02:10.20.20.82,cephstage03:10.20.20.83,cephstagedatanode01:10.20.20.84,cephstagedatanode02:10.20.20.85,cephstagedatanode03:10.20.20.86" ceph orch apply grafana --placement="cephstage01:10.20.20.81,cephstage02:10.20.20.82,cephstage03:10.20.20.83,cephstagedatanode01:10.20.20.84,cephstagedatanode02:10.20.20.85,cephstagedatanode03:10.20.20.86" ceph orch apply node-exporter ceph orch apply alertmanager ceph config set mgr mgr/cephadm/secure_monitoring_stack true --> disks and osd setup for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; do ssh root@$IP "hostname && wipefs -a -f /dev/sdb&& wipefs -a -f /dev/sdc"; done ceph config set mgr mgr/cephadm/device_enhanced_scan true for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; doceph orch device ls --hostname=$IP --wide --refresh ; done for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; doceph orch device zap $IP /dev/sdb; done for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; doceph orch device zap $IP /dev/sdc ; done for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; doceph orch daemon add osd $IP:/dev/sdb ; done for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; doceph orch daemon add osd $IP:/dev/sdc ; done --> ganesha nfs cluster ceph mgr module enable nfs ceph fs volume create vol1 ceph nfs cluster create nfs-cephfs "cephstagedatanode01,cephstagedatanode02,cephstagedatanode03" --ingress --virtual-ip 192.168.7.80 --ingress-mode default ceph nfs export create cephfs --cluster-id nfs-cephfs --pseudo-path /mnt --fsname vol1 --> nfs mount mount -t nfs -o nfsvers=4.1,proto=tcp 192.168.7.80:/mnt /mnt/ceph is my recipe correct? the cluster is set up by 3 mon/mgr nodes and 3 osd/nfs nodes, on the latters I installed one 3tb ssd, for the data, and one 300gb ssd for the journaling but my problems are : - Although I can mount the export I can't write on it - I can't understand how to use the sdc disks for journaling - I can't understand the concept of "pseudo path" here below you can find the json output of the exports --> check ceph nfs export ls nfs-cephfs ceph nfs export info nfs-cephfs /mnt ------------------------------------ json file --------- { "export_id": 1, "path": "/", "cluster_id": "nfs-cephfs", "pseudo": "/mnt", "access_type": "RW", "squash": "none", "security_label": true, "protocols": [ 4 ], "transports": [ "TCP" ], "fsal": { "name": "CEPH", "user_id": "nfs.nfs-cephfs.1", "fs_name": "vol1" }, "clients": [] } ------------------------------------ Thanks in advance Rob

5 7

which grafana version to use with 17.2.x ceph version
by Osama Elswah 29 Apr '24

29 Apr '24

Hi, in quay.io I can find a lot of grafana versions for ceph (https://quay.io/repository/ceph/grafana?tab=tags) how can I find out which version should be used when I upgrade my cluster to 17.2.x ? Can I simply take the latest grafana version? Or is there a specfic grafana version I need to use?

3 2

Impact of large PG splits
by Eugen Block 29 Apr '24

29 Apr '24

Hi, I'm trying to estimate the possible impact when large PGs are splitted. Here's one example of such a PG: PG_STAT OBJECTS BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG UP 86.3ff 277708 414403098409 0 0 3092 3092 [187,166,122,226,171,234,177,163,155,34,81,239,101,13,117,8,57,111] Their main application is RGW on EC (currently 1024 PGs on 240 OSDs), 8TB HDDs backed by SSDs. There are 6 RGWs running behind HAProxies. It took me a while to convince them to do a PG split and now they're trying to assess how big the impact could be. The fullest OSD is already at 85% usage, the least filled one at 59%, so there is definitely room for a better balancing which, will be necessary until the new hardware arrives. The current distribution is around 100 PGs per OSD which usually would be fine, but since the PGs are that large only a few PGs difference have a huge impact on the OSD utilization. I'm targeting 2048 PGs for that pool for now, probably do another split when the new hardware has been integrated. Any comments are appreciated! Eugen

6 16

Re: MDS crash
by Alexey GERASIMOV 28 Apr '24

28 Apr '24

Colleagues, thank you for the advice to check the operability of MGRs. In fact, it is strange also: we checked our nodes for the network issues (ip connectivity, sockets, ACL, DNS) and find nothing wrong - but suddenly just the restart of all MGRs solved the problem with stale PGs and with ceph commands hang! So, we are at the start point again - ceph is working except MDS daemons crash. But now we see some additional errors in MDS logs when try to start the daemon: dir 0x1000dd10fa0 object missing on disk; some files may be lost (/volumes/csi/csi-vol-2eb40f89-f2e1-11ee-b657-3aa98da4c4a6/1080803d-1277-4ad8-ae80-a004bd3a5699/gallery/pc-12083932925583528732) dir 0x1000dd10f9d object missing on disk; some files may be lost (/volumes/csi/csi-vol-2eb40f89-f2e1-11ee-b657-3aa98da4c4a6/1080803d-1277-4ad8-ae80-a004bd3a5699/cadserver-filevault/project-files/661fb14d341d3746ea5c2a8f I promiced to create the bug, so will do it later a bit. But should I try to do something more from my side also? What I did exactly last time: cephfs-journal-tool journal reset cephfs-table-tool all reset session cephfs-data-scan scan_extents cephfs-data-scan scan_inodes cephfs-data-scan scan_links cephfs-data-scan cleanup And one more question: is it possible to access to cephfs content directly, without MDS?

2 1