- ceph-users - lists.ceph.io

MDS behind on trimming every 4-5 weeks causing issue for ceph filesystem
by Akash Warkhade 20 May '24

20 May '24

Hi, We are using rook-ceph with operator 1.10.8 and ceph 17.2.5. we are using ceph filesystem with 4 mds i.e 2 active & 2 standby MDS every 3-4 weeks filesystem is having issue i.e in ceph status we can see below warnings warnings : 2 MDS reports slow requests 2 MDS Behind on Trimming mds.myfs-a(mds.1) : behind on trimming (6378/128) max_segments:128, num_segments: 6378 mds.myfs-c(mds.1): behind on trimming (6560/128) max_segments:128, num_segments: 6560 to fix it, we have to restart all MDS pods one by one. this is happening every 4-5 weeks. We have seen many ceph issues related to it on ceph tracker and many people are suggesting to increase mds_cache_memory_limit currently for our cluster *mds_cache_memory_limit* is set to default 4GB *mds_log_max_segments* is set to default 128 Should we increase *mds_cache_memory_limit* to 8GB from default 4GB or is there any solution to fix this issue permanently?

2 4

Ceph Squid release / release candidate timeline?
by Peter Sabaini 17 May '24

17 May '24

Hi, is there a ballpark timeline for a Squid release candidate / release? I'm aware of this pad that tracks blockers, is that still accurate or should I be looking at another resource? https://pad.ceph.com/p/squid-upgrade-failures Thanks! peter.

1 0

Reef RGWs stop processing requests
by Iain Stott 17 May '24

17 May '24

Hi, We are running 3 clusters in multisite. All 3 were running Quincy 17.2.6 and using cephadm. We upgraded one of the secondary sites to Reef 18.2.1 a couple of weeks ago and were planning on doing the rest shortly afterwards. We run 3 RGW daemons on separate physical hosts behind an external HAProxy HA pair for each cluster. Since we upgrade to Reef we have had issues with the RGWs stopping processing requests. We can see that they don't crash as they still have entries in the logs about syncing, but as far as request processing goes, they just stop. While debugging this we have 1 of the 3 RGWs running a Quincy image, and this has never had an issue where it stops processing requests. Any Reef containers we deploy have always stopped within 48Hrs of being deployed. We have tried Reef versions 18.2.1, 18.2.2 and 18.1.3 and all exhibit the same issue. We are running podman 4.6.1 on Centos 8 with kernel 4.18.0-513.24.1.el8_9.x86_64. We have enabled debug logs for the RGWs but we have been unable to find anything in them that would shed light on the cause. We are just wondering if anyone had any ideas on what could be causing this or how to debug it further? Thanks Iain Iain Stott OpenStack Engineer Iain.Stott(a)thg.com [THG Ingenuity Logo]<https://www.thg.com> www.thg.com<https://www.thg.com/> [LinkedIn]<https://www.linkedin.com/company/thgplc/?originalSubdomain=uk> [Instagram] <https://www.instagram.com/thg> [X] <https://twitter.com/thgplc?lang=en>

1 0

RadosGW Multisite Zonegroup redirect
by rabe＠telemaxx.de 17 May '24

17 May '24

Hi folks, I am currently in the process of building a Multisite Cluster (v18.2.2) with two Ceph clusters (cluster1 and cluster2). Each Cluster is its own Zone and has its own zonegroup. The idea is to sync metadata but not data. The user should select the zonegroup (region) via the location constraint. I figured out how to set up the two zonegroups but now the following problem appears: When i create a bucket via the cluster2 rgw with the cluster1 zonegroup as a target my minio client shows the following error: mc mb test/test1 --region cluster1 mc: <ERROR> Unable to make bucket `test/test1`. The specified location-constraint is not valid the cluster1 rgw shows the following error: s3:create_bucket location constraint (cluster1) doesn't match zonegroup (cluster2) the cluster2 rgw shows no logs When i do the same with the aws cli i get no error but the rgw seems to ignore the location constraint and just creates the bucket in cluster2: aws s3api --endpoint http://cluster2 --profile test create-bucket --bucket test2 --region cluster1 When I copy a file to a bucket in cluster2 via the cluster1 rgw i get the following error on the cluster1 rgw: op->ERRORHANDLER: err_no=-2024 new_err_no=-2024 The Period is synced on both clusters and the rgws are restarted: { "id": "7189ba3c-c7de-4ea2-8d14-307e4355a9e3", "epoch": 6, "predecessor_uuid": "0253dec1-d804-470f-92b6-90a64ffb5548", "sync_status": [], "period_map": { "id": "7189ba3c-c7de-4ea2-8d14-307e4355a9e3", "zonegroups": [ { "id": "4daa58d4-3eb5-4e59-bffe-e929f64c70b5", "name": "cluster1", "api_name": "cluster1", "is_master": true, "endpoints": [ "http://172.31.4.12:80" ], "hostnames": [ "172.31.4.12", "172.31.4.16" ], "hostnames_s3website": [], "master_zone": "ae3aa0e9-c580-410a-8ed6-2f01657a06f8", "zones": [ { "id": "ae3aa0e9-c580-410a-8ed6-2f01657a06f8", "name": "cluster1-a", "endpoints": [ "http://172.31.4.12:80" ], "log_meta": false, "log_data": false, "bucket_index_max_shards": 11, "read_only": false, "tier_type": "", "sync_from_all": true, "sync_from": [], "redirect_zone": "", "supported_features": [ "compress-encrypted", "resharding" ] } ], "placement_targets": [ { "name": "default-placement", "tags": [], "storage_classes": [ "STANDARD" ] } ], "default_placement": "default-placement", "realm_id": "5e8f4256-cf89-40db-b2a9-bb717f980b4f", "sync_policy": { "groups": [] }, "enabled_features": [ "resharding" ] }, { "id": "8d1d8050-0dea-4c5a-9bc3-d6631ef132f8", "name": "cluster2", "api_name": "cluster2", "is_master": false, "endpoints": [ "http://172.31.4.16:80" ], "hostnames": [ "172.31.4.12", "172.31.4.16" ], "hostnames_s3website": [], "master_zone": "2a882ccc-b540-4de2-abe3-f026fccb9aef", "zones": [ { "id": "2a882ccc-b540-4de2-abe3-f026fccb9aef", "name": "cluster2-a", "endpoints": [ "http://172.31.4.16:80" ], "log_meta": false, "log_data": false, "bucket_index_max_shards": 11, "read_only": false, "tier_type": "", "sync_from_all": true, "sync_from": [], "redirect_zone": "", "supported_features": [ "compress-encrypted", "resharding" ] } ], "placement_targets": [ { "name": "default-placement", "tags": [], "storage_classes": [ "STANDARD" ] } ], "default_placement": "default-placement", "realm_id": "5e8f4256-cf89-40db-b2a9-bb717f980b4f", "sync_policy": { "groups": [] }, "enabled_features": [ "resharding" ] } ], "short_zone_ids": [ { "key": "2a882ccc-b540-4de2-abe3-f026fccb9aef", "val": 4229298211 }, { "key": "ae3aa0e9-c580-410a-8ed6-2f01657a06f8", "val": 1754305595 } ] }, "master_zonegroup": "4daa58d4-3eb5-4e59-bffe-e929f64c70b5", "master_zone": "ae3aa0e9-c580-410a-8ed6-2f01657a06f8", "period_config": { "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "user_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "user_ratelimit": { "max_read_ops": 0, "max_write_ops": 0, "max_read_bytes": 0, "max_write_bytes": 0, "enabled": false }, "bucket_ratelimit": { "max_read_ops": 0, "max_write_ops": 0, "max_read_bytes": 0, "max_write_bytes": 0, "enabled": false }, "anonymous_ratelimit": { "max_read_ops": 0, "max_write_ops": 0, "max_read_bytes": 0, "max_write_bytes": 0, "enabled": false } }, "realm_id": "5e8f4256-cf89-40db-b2a9-bb717f980b4f", "realm_name": "testrealm", "realm_epoch": 2 } According to my understanding this setup should work but I probably missed some config setting. Any idea? Thanks in advance!

1 0

Write issues on CephFS mounted with root_squash
by Nicola Mori 17 May '24

17 May '24

Dear Ceph users, I'm trying to export a CephFS with the root_squash option. This is the client configuration: client.wizardfs_rootsquash key: XXXXXXXXXXXX caps: [mds] allow rw fsname=wizardfs root_squash caps: [mon] allow r fsname=wizardfs caps: [osd] allow rw tag cephfs data=wizardfs I can mount it flawlessly on several machines using the kernel driver, but when a machine writes on it then the content seems fine from the writing machine but it's not actually written on disk since other machines just see an empty file: [12:43 mori@stryke ~]$ echo test > /wizard/ceph/software/el9/test [12:43 mori@stryke ~]$ ll /wizard/ceph/software/el9/test -rw-r--r-- 1 mori wizard 5 mag 15 12:43 /wizard/ceph/software/el9/test [12:43 mori@stryke ~]$ cat /wizard/ceph/software/el9/test test [12:43 mori@stryke ~]$ [mori@fili ~]$ ll /wizard/ceph/software/el9/test -rw-r--r--. 1 mori 1014 0 May 15 06:43 /wizard/ceph/software/el9/test [mori@fili ~]$ cat /wizard/ceph/software/el9/test [mori@fili ~]$ Unmounting and then remounting on "stryke" the file is seen as empty, so I guess that the content shown just after the write is only a cache effect and nothing is effectively written on disk. I checked the posix permissions on the folder and I have rw rights from both the machines. All of the above using Ceph 18.2.2 on the cluster (deployed with cephadm) and both the machines. Machine "fili" has kernel 5.14.0 while "stryke" has 6.8.9. The same issue happens consistently also in the reverse direction (writing from "fili" and reading from "stryke"), and also with other machines. Removing the squash_root option the problem vanishes. I don't know what might

4 7

Low rate Call girls service In Mehrauli Delhi l Just hire me | 8800733197
by Call Girls 16 May '24

16 May '24

1 0

Reminder: User + Dev Monthly Meetup rescheduled to May 23rd
by Laura Flores 15 May '24

15 May '24

Hi all, Those of you subscribed to the Ceph Community Calendar may have already gotten an update that the meeting was rescheduled, but I wanted to send a reminder here as well. The meeting will be held at the usual time a week from now on May 23rd. If you haven't already, please take the new survey we put out! Results will be discussed at the meeting next week: https://www.meetup.com/ceph-user-group/events/300883526/ Thanks, Laura Meeting Details + Survey link here: https://www.meetup.com/ceph-user-group/events/300883526/ -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

1 0

ceph dashboard reef 18.2.2 radosgw
by Christopher Durham 15 May '24

15 May '24

Hello, I am uisng 18.2.2 on Rocky 8 Linux. I am getting http error 500 whe trying to hit the ceph dashboard on reef 18.2.2 when trying to look at any of the radosgw pages. I tracked this down to /usr/share/ceph/mgr/dashboard/controllers/rgw.py It appears to parse the metadata for a given radosgw server improperly. In my varoous rgw ceph.conf entries, I have: rgw frontends = beast ssl_endpoint=0.0.0.0 ssl_certificate=/path/to/pem_with_cert_and_key but, rgw.py pulls the metadata for each server, and it is looking for 'port=' in the metadata for each server. When it doesn't find it based on line 147 in rgw.py, the ceph-mgr logs throwan exception which the manager proper catches and returns a 500. Would changing my frontends definition work? Is this known? I have had the frontends definition for awhile prior to my reef upgrade. Thanks -Chris

3 3

ceph tell mds.0 dirfrag split - syntax of the "frag" argument
by Alexander E. Patrakov 15 May '24

15 May '24

Hello, In the context of https://tracker.ceph.com/issues/64298, I decided to do something manually. In the help output of "ceph tell" for an MDS, I found these possibly useful commands: dirfrag ls <path>: List fragments in directory dirfrag merge <path> <frag>: De-fragment directory by path dirfrag split <path> <frag> <bits:int>: Fragment directory by path They accept the "frag" argument that is underdocumented. In the testsuite, they are used, and it seems like this argument accepts some notation containing a slash, which is also produced as "str" by "dirfrag ls". Can anyone explain the meaning of the parts before and after the slash? What is the relation between the accepted values for "dirfrag split" and the output of "dirfrag ls" - do I just feed the fragment from "dirfrag ls" to "dirfrag split" as-is? Is running "dirfrag split" manually safe on a production cluster? Thanks in advance. -- Alexander Patrakov

1 0

Problem with take-over-existing-cluster.yml playbook
by vladimir franciz blando 15 May '24

15 May '24

I know that only a few are using this script but just trying my luck here if someone has the same issue as mine. But first, who has successfully used this script and what version did you use? Im using this guide on my test environment ( https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/3/html/i… ) Error encountered --- TASK [Generate ceph configuration file] ********************************************************************** *********************************************************************************** fatal: [vladceph-1]: FAILED! => msg: '''_monitor_addresses'' is undefined. ''_monitor_addresses'' is undefined' fatal: [vladceph-3]: FAILED! => msg: '''_monitor_addresses'' is undefined. ''_monitor_addresses'' is undefined' fatal: [vladceph-2]: FAILED! => msg: '''_monitor_addresses'' is undefined. ''_monitor_addresses'' is undefined' --- Regards, Vlad Blando <https://about.me/vblando>

4 9