- ceph-users - lists.ceph.io

slow mds requests with random read test
by Ben 07 Jun '23

07 Jun '23

Hi, We are performing couple performance tests on CephFS using fio. fio is run in k8s pod and 3 pods will be up running mounting the same pvc to CephFS volume. Here is command line for random read: fio -direct=1 -iodepth=128 -rw=randread -ioengine=libaio -bs=4k -size=1G -numjobs=5 -runtime=500 -group_reporting -directory=/tmp/cache -name=Rand_Read_Testing_$BUILD_TIMESTAMP The random read is performed very slow. Here is the cluster log from dashboard: 5/30/23 8:13:16 PM [INF] Health check cleared: MDS_SLOW_REQUEST (was: 1 MDSs report slow requests) 5/30/23 8:13:16 PM [INF] Health check cleared: MDS_SLOW_METADATA_IO (was: 1 MDSs report slow metadata IOs) 5/30/23 8:13:16 PM [INF] MDS health message cleared (mds.?): 1 slow metadata IOs are blocked > 30 secs, oldest blocked for 33 secs 5/30/23 8:13:16 PM [INF] MDS health message cleared (mds.?): 1 slow requests are blocked > 30 secs 5/30/23 8:13:14 PM [WRN] Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 5/30/23 8:13:13 PM [INF] MDS health message cleared (mds.?): 1 slow requests are blocked > 30 secs 5/30/23 8:13:08 PM [WRN] Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 5/30/23 8:13:08 PM [WRN] Health check failed: 1 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 5/30/23 8:13:08 PM [WRN] slow request 34.213327 seconds old, received at 2023-05-30T12:12:33.951399+0000: client_request(client.270564:1406144 getattr pAsLsXsFs #0x700000103d0 2023-05-30T12:12:33.947323+0000 caller_uid=0, caller_gid=0{}) currently failed to rdlock, waiting 5/30/23 8:13:08 PM [WRN] 1 slow requests, 1 included below; oldest blocked for > 34.213328 secs 5/30/23 8:13:07 PM [WRN] slow request 33.169703 seconds old, received at 2023-05-30T12:12:33.952078+0000: peer_request:client.270564:1406144 currently dispatched 5/30/23 8:13:07 PM [WRN] 1 slow requests, 1 included below; oldest blocked for > 33.169704 secs 5/30/23 8:13:04 PM [INF] Cluster is now healthy 5/30/23 8:13:04 PM [INF] Health check cleared: MDS_SLOW_REQUEST (was: 1 MDSs report slow requests) 5/30/23 8:13:04 PM [INF] Health check cleared: MDS_SLOW_METADATA_IO (was: 1 MDSs report slow metadata IOs) 5/30/23 8:13:04 PM [INF] MDS health message cleared (mds.?): 9 slow metadata IOs are blocked > 30 secs, oldest blocked for 45 secs 5/30/23 8:13:04 PM [INF] MDS health message cleared (mds.?): 2 slow requests are blocked > 30 secs 5/30/23 8:12:57 PM [WRN] 2 slow requests, 0 included below; oldest blocked for > 44.954377 secs 5/30/23 8:12:52 PM [WRN] 2 slow requests, 0 included below; oldest blocked for > 39.954313 secs 5/30/23 8:12:48 PM [WRN] Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 5/30/23 8:12:47 PM [WRN] slow request 34.935921 seconds old, received at 2023-05-30T12:12:12.185614+0000: client_request(client.270564:1406139 create #0x7000001045b/atomic7966567911433736706tmp 2023-05-30T12:12:12.182999+0000 caller_uid=0, caller_gid=0{}) currently submit entry: journal_and_reply 5/30/23 8:12:47 PM [WRN] slow request 34.954254 seconds old, received at 2023-05-30T12:12:12.167281+0000: client_request(client.270564:1406138 rename #0x70000010457/build.xml #0x70000010457/atomic6590865221269854506tmp 2023-05-30T12:12:12.162999+0000 caller_uid=0, caller_gid=0{}) currently submit entry: journal_and_reply 5/30/23 8:12:47 PM [WRN] 2 slow requests, 2 included below; oldest blocked for > 34.954254 secs 5/30/23 8:12:44 PM [WRN] Health check failed: 1 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 5/30/23 8:12:41 PM [INF] Cluster is now healthy 5/30/23 8:12:41 PM [INF] Health check cleared: MDS_SLOW_REQUEST (was: 1 MDSs report slow requests) 5/30/23 8:12:41 PM [INF] MDS health message cleared (mds.?): 1 slow requests are blocked > 30 secs 5/30/23 8:12:40 PM [INF] Health check cleared: MDS_SLOW_METADATA_IO (was: 1 MDSs report slow metadata IOs) 5/30/23 8:12:40 PM [INF] MDS health message cleared (mds.?): 1 slow metadata IOs are blocked > 30 secs, oldest blocked for 38 secs However, random write test is performing very good. Any suggestions on the problem? Thanks, Ben

2 3

RGW: perf dump. What is "objecter-0x55b63c38fb80" ?
by Rudenko Aleksandr 07 Jun '23

07 Jun '23

Hi guys, In perf dump of RGW instance I have two similar sections. First one: "objecter": { "op_active": 0, "op_laggy": 0, "op_send": 38816, "op_send_bytes": 199927218, "op_resend": 0, "op_reply": 38816, "oplen_avg": { "avgcount": 38816, "sum": 90408 }, "op": 38816, "op_r": 12624, "op_w": 26192, "op_rmw": 0, "op_pg": 0, … } Second one: "objecter-0x55b63c38fb80": { "op_active": 0, "op_laggy": 0, "op_send": 5540, "op_send_bytes": 217343, "op_resend": 0, "op_reply": 5540, "oplen_avg": { "avgcount": 5540, "sum": 5636 }, "op": 5540, "op_r": 680, "op_w": 4860, "op_rmw": 0, "op_pg": 0, … } What is 0x55b63c38fb80 ? I try to monitor ‘op_active’ metric, but this metric refreshes only in ‘objecter-0x55b63c38fb80’ section and always 0 in ‘objecter’ section and it’s so difficult to monitor this metric because this id is dynamic and will be changed on next rgw restart.

1 0

Orchestration seems not to work
by Thomas Widhalm 07 Jun '23

07 Jun '23

Hi, I'm in the process of upgrading my cluster from 17.2.5 to 17.2.6 but the following problem existed when I was still everywhere on 17.2.5 . I had a major issue in my cluster which could be solved with a lot of your help and even more trial and error. Right now it seems that most is already fixed but I can't rule out that there's still some problem hidden. The very issue I'm asking about started during the repair. When I want to orchestrate the cluster, it logs the command but it doesn't do anything. No matter if I use ceph dashboard or "ceph orch" in "cephadm shell". I don't get any error message when I try to deploy new services, redeploy them etc. The log only says "scheduled" and that's it. Same when I change placement rules. Usually I use tags. But since they don't work anymore, too, I tried host and umanaged. No success. The only way I can actually start and stop containers is via systemctl from the host itself. When I run "ceph orch ls" or "ceph orch ps" I see services I deployed for testing being deleted (for weeks now). Ans especially a lot of old MDS are listed as "error" or "starting". The list doesn't match reality at all because I had to start them by hand. I tried "ceph mgr fail" and even a complete shutdown of the whole cluster with all nodes including all mgs, mds even osd - everything during a maintenance window. Didn't change anything. Could you help me? To be honest I'm still rather new to Ceph and since I didn't find anything in the logs that caught my eye I would be thankful for hints how to debug. Cheers, Thomas -- http://www.widhalm.or.at GnuPG : 6265BAE6 , A84CB603 Threema: H7AV7D33 Telegram, Signal: widhalmt(a)widhalm.or.at

3 19

Encryption per user Howto
by huxiaoyu＠horebdata.cn 07 Jun '23

07 Jun '23

Dear Ceph folks, Recently one of our clients approached us with a request on encrpytion per user, i.e. using individual encrytion key for each user and encryption files and object store. Does anyone know (or have experience) how to do with CephFS and Ceph RGW? Any suggestionns or comments are highly appreciated, best regards, Samuel huxiaoyu(a)horebdata.cn

8 27

change user root to non-root after deploy cluster by cephadm
by farhad kh 07 Jun '23

07 Jun '23

Hi guys I deployed the ceph cluster with cephadm and root user, but I need to change the user to a non-root user And I did these steps: 1- Created a non-root user on all hosts with access without password and sudo `$USER_NAME ALL = (root) NOPASSWD:ALL` 2- Generated a SSH key pair and use ssh-copy-it to add all hosts ` ssh-keygen (accept the default file name and leave the passphrase empty) ssh-copy-id USER_NAME@HOST_NAME ` 3 - ceph cephadm set-user <user>But I get "Error EINVAL: ssh connection to root@hostname failed" error How to deal with this issue? What should be done to change the user to non-root?

2 1

Non cephadm cluster upgrade from octopus to quincy
by Szabo, Istvan (Agoda) 07 Jun '23

07 Jun '23

Hi, I don't find any documentation for this upgrade process. Is there anybody who has already done it yet? Is the normal apt-get update method works? Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0

reef v18.1.0 QE Validation status
by Yuri Weinstein 07 Jun '23

07 Jun '23

Details of this release are summarized here: https://tracker.ceph.com/issues/61515#note-1 Release Notes - TBD Seeking approvals/reviews for: rados - Neha, Radek, Travis, Ernesto, Adam King (we still have to merge https://github.com/ceph/ceph/pull/51788 for the core) rgw - Casey fs - Venky orch - Adam King rbd - Ilya krbd - Ilya upgrade/octopus-x - deprecated upgrade/pacific-x - known issues, Ilya, Laura? upgrade/reef-p2p - N/A clients upgrades - not run yet powercycle - Brad ceph-volume - in progress Please reply to this email with approval and/or trackers of known issues/PRs to address them. gibba upgrade was done and will need to be done again this week. LRC upgrade TBD TIA

7 9

Re: Workload Separation in Ceph RGW Cluster - Recommended or Not?
by Ramin Najjarbashi 06 Jun '23

06 Jun '23

Thank you for your response and for raising an important question regarding the potential bottlenecks within the RGW or the overall Ceph cluster. I appreciate your insight and would like to provide more information about the issues I have been experiencing. In my deployment, RGW instances 17-20 have been encountering problems such as hanging or returning errors, including "failed to read header: The socket was closed due to a timeout" and "res_query() failed." These issues have led to disruptions and congestions within the cluster. The index pool is indeed placed on a large number of NVMe SSDs to ensure fast access and efficient indexing of data. The number of Placement Groups (PGs) allocated for the index pool is also configured to be sufficient for the workload On Tue, Jun 6, 2023 at 21:27 Anthony D'Atri <anthony.datri(a)gmail.com> wrote: > Do you have reason to believe that your bottlenecks are within RGW not > within the cluster? > > e.g. is your index pool on a large number of NVMe SSDs with sufficient > PGs? Is your bucket data on SSD as well? > > > On Jun 6, 2023, at 13:52, Ramin Najjarbashi <ramin.najarbashi(a)gmail.com> > wrote: > > I would like to seek your insights and recommendations regarding the > practice of workload separation in a Ceph RGW (RADOS Gateway) cluster. I > have been facing challenges with large queues in my deployment and would > appreciate your expertise in determining whether workload separation is a > recommended approach or not. > > >

1 0

Workload Separation in Ceph RGW Cluster - Recommended or Not?
by Ramin Najjarbashi 06 Jun '23

06 Jun '23

Hi I would like to seek your insights and recommendations regarding the practice of workload separation in a Ceph RGW (RADOS Gateway) cluster. I have been facing challenges with large queues in my deployment and would appreciate your expertise in determining whether workload separation is a recommended approach or not. In my current Ceph cluster, I have 20 RGW instances. Client requests are directed to RGW1-16, while RGW17-20 are dedicated to administrative tasks and backend usage. However, I have been encountering errors and congestion issues due to the accumulation of large queues within the RGW instances. Considering the above scenario, I would like to inquire about your opinions on workload separation as a potential solution. Specifically, I am interested in knowing whether workload separation is recommended in a Ceph RGW cluster. To address the queue congestion and improve performance, my proposed solution includes separating the RGW instances based on their specific purposes. This entails allocating dedicated instances for client requests, backend usage, administrative tasks, metadata synchronization with other zone groups, garbage collection (GC), and lifecycle (LC) operations. I kindly request your feedback and insights on the following points: 1. Is workload separation considered a recommended practice in Ceph RGW deployments? 2. What are the potential benefits and drawbacks of workload separation in terms of performance, resource utilization, and manageability? 3. Are there any specific considerations or best practices to keep in mind while implementing workload separation in a Ceph RGW cluster? 4. Can you share your experiences or any references/documentation that highlight successful implementations of workload separation in Ceph RGW deployments? I truly value your expertise and appreciate your time and effort in providing guidance on this matter. Your insights will contribute significantly to optimizing the performance and stability of my Ceph RGW cluster.

1 0

Quincy release -Swift integration with Keystone
by fsbiz＠yahoo.com 06 Jun '23

06 Jun '23

Hi folks, My ceph cluster with Quincy and Rocky9 is up and running. But I'm having issues with swift authenticating with keystone. Was wondering if I'm missed anything in the configuration. From the debug logs below, it appears that radosgw is still trying to authenticate with Swift instead of Keystone. Any pointers will be appreciated. thanks, Here is my configuration. # ceph config dump | grep rgw client advanced debug_rgw 20/20 client advanced rgw_keystone_accepted_roles admin,user * client advanced rgw_keystone_admin_domain Default * client advanced rgw_keystone_admin_password <secret> * client advanced rgw_keystone_admin_project service * client advanced rgw_keystone_admin_user ceph-ks-svc * client advanced rgw_keystone_api_version 3 client advanced rgw_keystone_implicit_tenants false * client advanced rgw_keystone_token_cache_size 0 client basic rgw_keystone_url <Identity URL> * client advanced rgw_s3_auth_use_keystone true client advanced rgw_swift_account_in_url true client basic rgw_thread_pool_size 512 client.rgw.s_rgw.dev-ipp1-u1-control01.ojmddc basic rgw_frontends beast port=7480 * client.rgw.s_rgw.dev-ipp1-u1-control02.adnjrx basic rgw_frontends beast port=7480 Here's the debug log. If I interpret it correctly, it is trying to do a swift authentication and failing. Am I missing any configuration for Keystone based authentication ? Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: beast: 0x7fddeb8e7710: 10.117.53.10 - - [03/Jun/2023:18:47:03.060 +0000] "GET /swift/v1/AUTH_c668ed224e434c88a9e0fce125056112?format=json HTTP/1.1" 401 119 - "openstacksdk/0.52.0 keystoneauth1/4.0.0 python-requests/2.22.0 CPython/3.8.10" - latency=0.000000000s Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: HTTP_ACCEPT=*/* Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: HTTP_ACCEPT_ENCODING=gzip, deflate Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: HTTP_CONNECTION=close Jun 03 11:47:03 dev-ipp1-u1-control02.radosgw[2802861]: HTTP_HOST=dev-ipp1-u1-object-store Jun 03 11:47:03 dev-ipp1-u1-control02radosgw[2802861]: HTTP_USER_AGENT=openstacksdk/0.52.0 keystoneauth1/4.0.0 python-requests/2.22.0 CPython/3.8.10 Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: HTTP_VERSION=1.1 Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: HTTP_X_AUTH_TOKEN=gAAAAABke4qn779UQ_XMz0EDL3P3TgjBQsGG6p-MNhviJxLZTuMTnTDmpT5Yfi9UpgO_T3LOOsPjQAw6zoMUIaC22wPeryp5x-UumB3XwXOWp-qSXLbuN3b9oj_Qg5kCZWA0waWNRHzQ1mwtlEmmpTgvTXbU5V1ym6hEBOn6Q3RWhn34Hj3cF9o Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: HTTP_X_FORWARDED_FOR=10.117.148.3 Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: QUERY_STRING=format=json Jun 03 11:47:03 dev-ipp1-u1-control02.radosgw[2802861]: REMOTE_ADDR=10.117.53.10 Jun 03 11:47:03 dev-ipp1-u1-control02.radosgw[2802861]: REQUEST_METHOD=GET Jun 03 11:47:03 dev-ipp1-u1-control02.radosgw[2802861]: REQUEST_URI=/swift/v1/AUTH_c668ed224e434c88a9e0fce125056112?format=json Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: SCRIPT_URI=/swift/v1/AUTH_c668ed224e434c88a9e0fce125056112 Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: SERVER_PORT=7480 Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: ====== starting new request req=0x7fddeb8e7710 ===== Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s initializing for trans_id = tx000003991cfc5c1791f95-00647b8aa7-30c56-default Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s rgw api priority: s3=8 s3website=7 Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s host=dev-ipp1-u1-object-store Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s final domain/bucket subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 s->info.domain= s->info.request_uri=/swift/v1/AUTH_c668ed224e434c88a9e0fce125056112 Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s name: format val: json Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s ver=v1 first= req= Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s handler=29RGWHandler_REST_Service_SWIFT Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s getting op 0 Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s get_system_obj_state: rctx=0x7fddeb8e6790 obj=default.rgw.log:script.prerequest. state=0x55f743b97720 s->prefetch_data=0 Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s cache get: name=default.rgw.log++script.prerequest. : hit (negative entry) Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s swift:list_buckets scheduling with throttler client=3 cost=1 Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s swift:list_buckets op=29RGWListBuckets_ObjStore_SWIFT Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s swift:list_buckets verifying requester Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s swift:list_buckets rgw::auth::swift::DefaultStrategy: trying rgw::auth::swift::TempURLEngine Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s swift:list_buckets rgw::auth::swift::TempURLEngine denied with reason=-13 Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s swift:list_buckets rgw::auth::swift::DefaultStrategy: trying rgw::auth::swift::SignedTokenEngine Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s swift:list_buckets rgw::auth::swift::SignedTokenEngine denied with reason=-1 Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s swift:list_buckets rgw::auth::swift::DefaultStrategy: trying rgw::auth::swift::SwiftAnonymousEngine Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s swift:list_buckets rgw::auth::swift::SwiftAnonymousEngine denied with reason=-1 Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s swift:list_buckets Failed the auth strategy, reason=-1 Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: failed to authorize request Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s op->ERRORHANDLER: err_no=-1 new_err_no=-1 Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s get_system_obj_state: rctx=0x7fddeb8e6790 obj=default.rgw.log:script.postrequest. state=0x55f743b97960 s->prefetch_data=0 Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s cache get: name=default.rgw.log++script.postrequest. : hit (negative entry) Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s swift:list_buckets op status=0 Jun 03 11:47:03 dev-ipp1-u1-control02 radosgw[2802861]: req 4148325180046385045 0.000000000s swift:list_buckets http status=401

2 1