May 2023 - ceph-users - lists.ceph.io

Telemetry and Redmine sync
by Yaarit Hatuka 16 May '23

16 May '23

Hi everyone, Over this weekend we will run a sync between telemetry crashes and Redmine tracker issues. This might affect your inbox, depending on your Redmine email notification setup. You can set up filters for these emails to skip your inbox. Thanks, Yaarit

2 1

cephadm and remoto package
by Shashi Dahal 15 May '23

15 May '23

Hi, I followed this documentation: https://docs.ceph.com/en/pacific/cephadm/adoption/ This is the error I get when trying to enable cephadm. ceph mgr module enable cephadm Error ENOENT: module 'cephadm' reports that it cannot run on the active manager daemon: loading remoto library:No module named 'remoto' (pass --force to force enablement) When I import remoto, it imports just fine. OS is ubuntu 20.04 focal -- Cheers, Shashi

1 0

Quincy Ceph-orchestrator and multipath SAS
by Deep Dish 15 May '23

15 May '23

Hello, I have a few hosts about to add into a cluster that have a multipath storage config for SAS devices. Is this supported on Quincy, and how would ceph-orchestrator and / or ceph-volume handle multipath storage? Here's a snip of lsblk output of a host in question: # lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS ... sdc 8:32 0 9.1T 0 disk └─mpathh 253:4 0 9.1T 0 mpath sdd 8:48 0 9.1T 0 disk └─mpathi 253:5 0 9.1T 0 mpath sde 8:64 0 7.3T 0 disk └─mpathj 253:6 0 7.3T 0 mpath sdf 8:80 0 7.3T 0 disk └─mpathl 253:7 0 7.3T 0 mpath sdg 8:96 0 7.2T 0 disk └─mpathk 253:8 0 7.2T 0 mpath sdh 8:112 0 7.3T 0 disk └─mpathe 253:9 0 7.3T 0 mpath sdi 8:128 0 7.3T 0 disk └─mpathg 253:10 0 7.3T 0 mpath sdj 8:144 0 7.3T 0 disk └─mpathf 253:11 0 7.3T 0 mpath sdk 8:160 0 7.3T 0 disk └─mpathc 253:12 0 7.3T 0 mpath sdl 8:176 0 7.3T 0 disk └─mpathd 253:13 0 7.3T 0 mpath ...

2 1

Octopus on Ubuntu 20.04.6 LTS with kernel 5
by Szabo, Istvan (Agoda) 15 May '23

15 May '23

Hi, In octopus documentation we can see kernel 4 as recommended, however we've changed our test cluster yesterday from centos 7 / 8 to Ubuntu 20.04.6 LTS with kernel 5.4.0-148 and seems working, I just want to make sure before I move to prod there isn't any caveats. Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

4 5

Upgrade Ceph cluster + radosgw from 14.2.18 to latest 15
by viplanghe6＠gmail.com 15 May '23

15 May '23

Hi, I want to upgrade my old Ceph cluster + Radosgw from v14 to v15. But I'm not using cephadm and I'm not sure how to limit errors as much as possible during the upgrade process? Here is my upgrade steps: Firstly, upgrade from 14.2.18 to 14.2.22 (latest nautilus version) Then, upgrade it from 14.2.22 to 15.2.17 (latest octopus version) service restart order: - restart monitors (sleep ~10) - restart managers (sleep ~5) - restart metadata OSDs (sleep ~30) - restart data OSDs (sleep ~30) - restart radosgw (sleep ~20) Is there anything wrong with these steps? Does 15.2.17 is the most stable version of Octopus? Thanks.

3 7

RGW: Using Lua script to monitor storage class put operations
by viplanghe6＠gmail.com 14 May '23

14 May '23

I have a lua script that read Storageclass header of any put request (as I understand): local function isempty(input) return input == nil or input == '' end if Request.RGWOp == 'put_obj' then RGWDebugLog("Put_Obj with StorageClass: " .. Request.HTTP.StorageClass ) end Then apply the script: radosgw-admin script put --infile=storage_class.lua --context=preRequest Then put an object using boto3: s3.Object(bucket, 'test-object').put(Body="0"*100, ACL='public-read', StorageClass='COLD') But I can't find any log about it in radosgw log? grep -i Lua /var/log/ceph/ceph-client.rgw.ceph1.log Anyone have any ideal? Thanks

2 1

Unable to Retrieve Usage Logs in Ceph RGW with Boto3
by Ramin Najjarbashi 13 May '23

13 May '23

Hi I am currently working on retrieving user usage and usage logs from Ceph RGW using Boto3. While I am able to successfully obtain user usage and all bucket usage with the following model and code: ```python params = { "start-date": "2023-05-13T00:00:00Z", "end-date": "2023-05-13T23:59:59Z", } print(json.dumps(s3_client.get_usage_stats(**params), indent=2)) ``` ```json { "version": 1.0, "merge": { "operations": { "GetUsageStats": { "name": "GetUsageStats", "http": { "method": "GET", "requestUri": "/?usage&start-date={start-date}&end-date={end-date}", "responseCode": 200 }, "input": { "shape": "GetUsageLogInput" }, "output": { "shape": "GetUsageStatsOutput" }, "documentationUrl": " https://docs.ceph.com/docs/master/radosgw/s3/serviceops#get-usage-stats", "documentation": "Get usage stats for the user" } }, "shapes": { "GetUsageLogInput": { "type": "structure", "members": { "start-date": { "shape": "Timestamp", "location": "uri", "documentation": "The start date and time for the log entries to retrieve." }, "end-date": { "shape": "Timestamp", "location": "uri", "documentation": "The end date and time for the log entries to retrieve." } }, "documentation": "Input shape for the GetUsageLog operation." }, "Timestamp": { "type": "string", "documentation": "A timestamp in ISO 8601 format." }, "GetUsageStatsOutput": { "type": "structure", "members": { "Summary": { "shape": "UsageStatsSummary", "documentation": "Summary of usage stats." }, "CapacityUsed": { "shape": "CapacityUsedUsage", "documentation": "Usage stats for capacity used." } } }, "CapacityUsedUsage": { "type": "structure", "members": { "User": { "shape": "UserUsageEntries", "documentation": "Usage stats for the user." } } }, "UserUsageEntries": { "type": "structure", "members": { "Buckets": { "shape": "BucketsUsageEntries", "documentation": "Usage stats for buckets." } } }, "BucketsUsageEntries": { "type": "structure", "members": { "Entry": { "shape": "UsageEntries", "documentation": "List of usage entries for buckets." } } }, "UsageEntries": { "type": "list", "member": { "shape": "BucketEntry" }, "documentation": "List of usage entries." }, "BucketEntry": { "type": "structure", "members": { "Bucket": { "shape": "String", "documentation": "Name of the bucket." }, "Bytes_Rounded": { "shape": "Bytes_Rounded", "documentation": "Number of bytes (rounded)." }, "Bytes": { "shape": "Bytes", "documentation": "Number of bytes." } } }, "UsageStatsSummary": { "type": "structure", "members": { "QuotaMaxBytes": { "shape": "QuotaMaxBytes", "documentation": "Maximum quota in bytes." }, "QuotaMaxBuckets": { "shape": "QuotaMaxBuckets", "documentation": "Maximum quota for buckets." }, "QuotaMaxObjCount": { "shape": "QuotaMaxObjCount", "documentation": "Maximum quota for object count." }, "QuotaMaxBytesPerBucket": { "shape": "QuotaMaxBytesPerBucket", "documentation": "Maximum quota in bytes per bucket." }, "QuotaMaxObjCountPerBucket": { "shape": "QuotaMaxObjCountPerBucket", "documentation": "Maximum quota for object count per bucket." }, "TotalBytes": { "shape": "TotalBytes", "documentation": "Total number of bytes." }, "TotalBytesRounded": { "shape": "TotalBytesRounded", "documentation": "Total number of bytes (rounded)." }, "TotalEntries": { "shape": "TotalEntries", "documentation": "Total number of entries." } } }, "BytesReceived": { "type": "integer", "documentation": "Number of bytes received." }, "QuotaMaxBytes": { "type": "integer", "documentation": "Maximum quota in bytes." }, "QuotaMaxBuckets": { "type": "integer", "documentation": "Maximum quota for buckets." }, "QuotaMaxObjCount": { "type": "integer", "documentation": "Maximum quota for object count." }, "QuotaMaxBytesPerBucket": { "type": "integer", "documentation": "Maximum quota in bytes per bucket." }, "QuotaMaxObjCountPerBucket": { "type": "integer", "documentation": "Maximum quota for object count per bucket." }, "TotalBytesRounded": { "type": "integer", "documentation": "Total number of bytes (rounded)." }, "TotalBytes": { "type": "integer", "documentation": "Total number of bytes." }, "TotalEntries": { "type": "integer", "documentation": "Total number of entries." }, "String": { "type": "string", "documentation": "A string value." }, "Bytes_Rounded": { "type": "integer", "documentation": "Number of bytes (rounded)." }, "Bytes": { "type": "integer", "documentation": "Number of bytes." } }, "documentation": "Model for retrieving usage statistics." } } ``` and this is my output: ``` "Summary": { "QuotaMaxBytes": -1, "QuotaMaxBuckets": 1000, "QuotaMaxObjCount": -1, "QuotaMaxBytesPerBucket": -1, "QuotaMaxObjCountPerBucket": -1, "TotalBytes": 52696, "TotalBytesRounded": 53248, "TotalEntries": 1 }, "CapacityUsed": { "User": { "Buckets": { "Entry": [ { "Bucket": "opa-test", "Bytes_Rounded": 53248, "Bytes": 52696 }, { "Bucket": "ramfrom", "Bytes_Rounded": 0, "Bytes": 0 } } } } } ``` The issue I am facing is that the usage logs are not being displayed. Upon investigating the code in the send_response() function of the RGWGetUsage_ObjStore_S3 class, I found the following conditions: ```cpp if (show_log_sum) { formatter->open_array_section("Summary"); map<string, rgw_usage_log_entry>::iterator siter; for (siter = summary_map.begin(); siter != summary_map.end(); ++siter) { const rgw_usage_log_entry& entry = siter->second; formatter->open_object_section("User"); formatter->dump_string("User", siter->first); dump_usage_categories_info(formatter, entry, &categories); rgw_usage_data total_usage; entry.sum(total_usage, categories); formatter->open_object_section("Total"); encode_json("BytesSent", total_usage.bytes_sent, formatter); encode_json("BytesReceived", total_usage.bytes_received, formatter); encode_json("Ops", total_usage.ops, formatter); encode_json("SuccessfulOps", total_usage.successful_ops, formatter); formatter->close_section(); // total formatter->close_section(); // user } ``` According to the code, the show_log_sum parameter must be true to display the summary log entries, and I have confirmed that it is indeed set to true. Additionally, the show_log_entries parameter must also be true to retrieve the usage log entries, and I have verified that it is set to true as well. I am using Ceph version 16.2.12, and I have set the RGW debug level to 99. Here is a snippet of the log output: ```log 2023-05-13T20:39:17.111+0330 7f346da3d700 1 ====== starting new request req= 0x7f3504a0e620 ===== 2023-05-13T20:39:17.147+0330 7f346da3d700 2 req 11854699226730900750 0.035999894s initializing for trans_id = tx00000a4845a234864ad0e-00645fc43d-fb27-default 2023-05-13T20:39:17.283+0330 7f346da3d700 10 req 11854699226730900750 0.171999499s rgw api priority: s3=8 s3website=7 2023-05-13T20:39:17.287+0330 7f346da3d700 10 req 11854699226730900750 0.175999492s host=s3.abrak.stage.test.com 2023-05-13T20:39:17.291+0330 7f346da3d700 20 req 11854699226730900750 0.179999471s subdomain= domain=s3.abrak.stage.test.com in_hosted_domain=1 in_hosted_domain_s3website=0 2023-05-13T20:39:17.291+0330 7f346da3d700 20 req 11854699226730900750 0.179999471s final domain/bucket subdomain= domain=s3.abrak.stage.test.com in_hosted_domain=1 in_hosted_domain_s3website=0 s->info.domain=s3.abrak. stage.test.com s->info.request_uri=/ 2023-05-13T20:39:17.315+0330 7f346da3d700 10 req 11854699226730900750 0.207999393s meta>> HTTP_X_AMZ_CONTENT_SHA256 2023-05-13T20:39:17.319+0330 7f346da3d700 10 req 11854699226730900750 0.207999393s meta>> HTTP_X_AMZ_DATE 2023-05-13T20:39:17.319+0330 7f346da3d700 10 req 11854699226730900750 0.207999393s x>> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2023-05-13T20:39:17.319+0330 7f346da3d700 10 req 11854699226730900750 0.207999393s x>> x-amz-date:20230513T170916Z 2023-05-13T20:39:17.347+0330 7f346da3d700 10 req 11854699226730900750 0.235999316s name: usage val: 2023-05-13T20:39:17.347+0330 7f346da3d700 10 req 11854699226730900750 0.235999316s name: start-date val: 2023-05-13T00:00:00Z 2023-05-13T20:39:17.347+0330 7f346da3d700 10 req 11854699226730900750 0.235999316s name: end-date val: 2023-05-13T23:59:59Z 2023-05-13T20:39:17.355+0330 7f346da3d700 20 req 11854699226730900750 0.243999273s get_handler handler=26RGWHandler_REST_Service_S3 2023-05-13T20:39:17.383+0330 7f346da3d700 10 req 11854699226730900750 0.271999210s handler=26RGWHandler_REST_Service_S3 2023-05-13T20:39:17.383+0330 7f346da3d700 2 req 11854699226730900750 0.271999210s getting op 0 2023-05-13T20:39:17.423+0330 7f346da3d700 10 req 11854699226730900750 0.311999112s s3:get_self_usage scheduling with throttler client=3 cost=1 2023-05-13T20:39:17.431+0330 7f346da3d700 10 req 11854699226730900750 0.319999069s s3:get_self_usage op=23RGWGetUsage_ObjStore_S3 2023-05-13T20:39:17.431+0330 7f346da3d700 2 req 11854699226730900750 0.319999069s s3:get_self_usage verifying requester 2023-05-13T20:39:17.435+0330 7f346da3d700 20 req 11854699226730900750 0.323999047s s3:get_self_usage rgw::auth::StrategyRegistry::s3_main_strategy_t: trying rgw::auth::s3::AWSAuthStrategy 2023-05-13T20:39:17.447+0330 7f346da3d700 20 req 11854699226730900750 0.335999012s s3:get_self_usage rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3:: S3AnonymousEngine 2023-05-13T20:39:17.451+0330 7f346da3d700 20 req 11854699226730900750 0.339999020s s3:get_self_usage rgw::auth::s3::S3AnonymousEngine denied with reason=-1 2023-05-13T20:39:17.451+0330 7f346da3d700 20 req 11854699226730900750 0.339999020s s3:get_self_usage rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3:: LocalEngine 2023-05-13T20:39:17.455+0330 7f346da3d700 10 req 11854699226730900750 0.343998998s v4 signature format = 2e4a95626b3633439d13f5769d6a0be142c30823192837c3599fbb3f758d7c40 2023-05-13T20:39:17.463+0330 7f346da3d700 10 req 11854699226730900750 0.351998985s v4 credential format = OSDDH3XY9FPGHJ07O7JP/20230513/us-east-1/s3/ aws4_request 2023-05-13T20:39:17.463+0330 7f346da3d700 10 req 11854699226730900750 0.351998985s access key id = OSDDH3XY9FPGHJ07O7JP 2023-05-13T20:39:17.463+0330 7f346da3d700 10 req 11854699226730900750 0.351998985s credential scope = 20230513/us-east-1/s3/aws4_request 2023-05-13T20:39:17.471+0330 7f346da3d700 10 req 11854699226730900750 0.359998941s canonical headers format = host:s3.abrak.stage.test.com x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date:20230513T170916Z 2023-05-13T20:39:17.479+0330 7f346da3d700 10 req 11854699226730900750 0.367998898s payload request hash = e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2023-05-13T20:39:17.483+0330 7f346da3d700 10 req 11854699226730900750 0.371998906s canonical request = GET / end-date=2023-05-13T23%3A59%3A59Z&start-date=2023-05-13T00%3A00%3A00Z&usage= host:s3.abrak.stage.test.com x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date:20230513T170916Z host;x-amz-content-sha256;x-amz-date e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2023-05-13T20:39:17.483+0330 7f346da3d700 10 req 11854699226730900750 0.371998906s canonical request hash = 060e8a249e9f3708393aa7d2cfde262976bdd4d10216be12ca74ae7961e3994f 2023-05-13T20:39:17.483+0330 7f346da3d700 10 req 11854699226730900750 0.371998906s string to sign = AWS4-HMAC-SHA256 20230513T170916Z 20230513/us-east-1/s3/aws4_request 060e8a249e9f3708393aa7d2cfde262976bdd4d10216be12ca74ae7961e3994f 2023-05-13T20:39:17.511+0330 7f346da3d700 10 req 11854699226730900750 0.399998844s date_k = fe25d3b269a34fa9c86caf0b35d6a46791ddd7fbcd0650a212fb7528a2dce06d 2023-05-13T20:39:17.511+0330 7f346da3d700 10 req 11854699226730900750 0.399998844s region_k = 03b05d2d23e0b150ae09d8baf554e8a8bc34887951f47b5925dead931c7faeaa 2023-05-13T20:39:17.511+0330 7f346da3d700 10 req 11854699226730900750 0.399998844s service_k = d10551f64ed49675cde1f1b609ee5b54b513b077e2c00198ade53b689dee17df 2023-05-13T20:39:17.511+0330 7f346da3d700 10 req 11854699226730900750 0.399998844s signing_k = c35cb1ca41d5cc1c5a49429c50afe3c840d0b67e81df0faf8122d979294098e7 2023-05-13T20:39:17.511+0330 7f346da3d700 10 req 11854699226730900750 0.399998844s generated signature = 2e4a95626b3633439d13f5769d6a0be142c30823192837c3599fbb3f758d7c40 2023-05-13T20:39:17.511+0330 7f346da3d700 15 req 11854699226730900750 0.399998844s s3:get_self_usage string_to_sign=AWS4-HMAC-SHA256 20230513T170916Z 20230513/us-east-1/s3/aws4_request 060e8a249e9f3708393aa7d2cfde262976bdd4d10216be12ca74ae7961e3994f 2023-05-13T20:39:17.511+0330 7f346da3d700 15 req 11854699226730900750 0.399998844s s3:get_self_usage server signature= 2e4a95626b3633439d13f5769d6a0be142c30823192837c3599fbb3f758d7c40 2023-05-13T20:39:17.511+0330 7f346da3d700 15 req 11854699226730900750 0.399998844s s3:get_self_usage client signature= 2e4a95626b3633439d13f5769d6a0be142c30823192837c3599fbb3f758d7c40 2023-05-13T20:39:17.511+0330 7f346da3d700 15 req 11854699226730900750 0.399998844s s3:get_self_usage compare=0 2023-05-13T20:39:17.519+0330 7f346da3d700 20 req 11854699226730900750 0.407998830s s3:get_self_usage rgw::auth::s3::LocalEngine granted access 2023-05-13T20:39:17.519+0330 7f346da3d700 20 req 11854699226730900750 0.407998830s s3:get_self_usage rgw::auth::s3::AWSAuthStrategy granted access 2023-05-13T20:39:17.519+0330 7f346da3d700 2 req 11854699226730900750 0.407998830s s3:get_self_usage normalizing buckets and tenants 2023-05-13T20:39:17.519+0330 7f346da3d700 10 req 11854699226730900750 0.407998830s s->object=<NULL> s->bucket= 2023-05-13T20:39:17.519+0330 7f346da3d700 2 req 11854699226730900750 0.407998830s s3:get_self_usage init permissions 2023-05-13T20:39:17.535+0330 7f346da3d700 20 req 11854699226730900750 0.423998743s s3:get_self_usage get_system_obj_state: rctx=0x7f3504a0ca28 obj=default.rgw. meta:users.uid:development state=0x56448667e0a0 s->prefetch_data=0 2023-05-13T20:39:17.535+0330 7f346da3d700 10 req 11854699226730900750 0.423998743s s3:get_self_usage cache get: name=default.rgw.meta+users.uid+development : hit (requested=0x6, cached=0x17) 2023-05-13T20:39:17.539+0330 7f346da3d700 20 req 11854699226730900750 0.427998751s s3:get_self_usage get_system_obj_state: s->obj_tag was set empty 2023-05-13T20:39:17.539+0330 7f346da3d700 20 req 11854699226730900750 0.427998751s s3:get_self_usage Read xattr: user.rgw.idtag 2023-05-13T20:39:17.539+0330 7f346da3d700 10 req 11854699226730900750 0.427998751s s3:get_self_usage cache get: name=default.rgw.meta+users.uid+development : hit (requested=0x3, cached=0x17) 2023-05-13T20:39:17.679+0330 7f346da3d700 2 req 11854699226730900750 0.567998350s s3:get_self_usage recalculating target 2023-05-13T20:39:17.679+0330 7f346da3d700 2 req 11854699226730900750 0.567998350s s3:get_self_usage reading permissions 2023-05-13T20:39:17.687+0330 7f346da3d700 2 req 11854699226730900750 0.575998306s s3:get_self_usage init op 2023-05-13T20:39:17.687+0330 7f346da3d700 2 req 11854699226730900750 0.575998306s s3:get_self_usage verifying op mask 2023-05-13T20:39:17.695+0330 7f346da3d700 20 req 11854699226730900750 0.583998263s s3:get_self_usage required_mask= 1 user.op_mask=7 2023-05-13T20:39:17.695+0330 7f346da3d700 2 req 11854699226730900750 0.583998263s s3:get_self_usage verifying op permissions 2023-05-13T20:39:17.695+0330 7f346da3d700 2 req 11854699226730900750 0.583998263s s3:get_self_usage verifying op params 2023-05-13T20:39:17.695+0330 7f346da3d700 2 req 11854699226730900750 0.583998263s s3:get_self_usage pre-executing 2023-05-13T20:39:17.695+0330 7f346da3d700 2 req 11854699226730900750 0.583998263s s3:get_self_usage executing 2023-05-13T20:39:21.023+0330 7f343c1da700 2 req 11854699226730900750 3.911988497s s3:get_self_usage completing 2023-05-13T20:39:21.067+0330 7f343c1da700 30 AccountingFilter::send_status: e=0, sent=17, total=0 2023-05-13T20:39:21.071+0330 7f343c1da700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2023-05-13T20:39:21.075+0330 7f343c1da700 30 AccountingFilter:: send_chunked_transfer_encoding: e=0, sent=28, total=0 2023-05-13T20:39:21.079+0330 7f343c1da700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2023-05-13T20:39:21.107+0330 7f343b1d8700 30 AccountingFilter:: complete_header: e=0, sent=156, total=0 2023-05-13T20:39:21.107+0330 7f343b1d8700 30 AccountingFilter::set_account: e=1 2023-05-13T20:39:21.199+0330 7f3457a11700 30 AccountingFilter::send_body: e= 1, sent=4051, total=0 2023-05-13T20:39:21.199+0330 7f3457a11700 30 AccountingFilter:: complete_request: e=1, sent=5, total=4051 2023-05-13T20:39:21.199+0330 7f3457a11700 30 req 11854699226730900750 4.087987900s log_usage: bucket_name= tenant=, bytes_sent=4056, bytes_received=0, success= 1 2023-05-13T20:39:21.199+0330 7f3457a11700 2 req 11854699226730900750 4.087987900s s3:get_self_usage op status=0 2023-05-13T20:39:21.199+0330 7f3457a11700 2 req 11854699226730900750 4.087987900s s3:get_self_usage http status=200 2023-05-13T20:39:21.219+0330 7f3457a11700 1 ====== req done req= 0x7f3504a0e620 op status=0 http_status=200 latency=4.103988171s ====== 2023-05-13T20:39:21.227+0330 7f3457a11700 1 beast: 0x7f3504a0e620: 192.168. 200.1 - development [13/May/2023:20:39:17.059 +0330] "GET /?usage&start-date=2023-05-13T00%3A00%3A00Z&end-date=2023-05-13T23%3A59%3A59Z HTTP/1.0" 200 4056 - "Boto3/1.26.131 Python/3.10.11 Darwin/22.4.0 Botocore/1.29.131" - latency=4.103988171s ``` I would greatly appreciate any guidance or insights into why I am unable to retrieve the usage logs. How can I ensure that the usage logs are included in the response? Thank you for your assistance! P.S. I have also verified the response using tools like Wireshark and tcpdump, and I can confirm that the responses received are consistent with the output I provided earlier.

1 0

MDS "newly corrupt dentry" after patch version upgrade
by Janek Bevendorff 12 May '23

12 May '23

Hi, After a patch version upgrade from 16.2.10 to 16.2.12, our rank 0 MDS fails start start. After replaying the journal, it just crashes with [ERR] : MDS abort because newly corrupt dentry to be committed: [dentry #0x1/storage [2,head] auth (dversion lock) Immediately after the upgrade, I had it running shortly, but then it decided to crash for unknown reasons and I cannot get it back up. We have five ranks in total, the other four seem to be fine. I backed up the journal and tried to run cephfs-journal-tool --rank=cephfs.storage:0 event recover_dentries summary, but it never finishes only eats up a lot of RAM. I stopped it after an hour and 50GB RAM. Resetting the journal makes the MDS crash with a missing inode error on another top-level directory, so I re-imported the backed-up journal. Is there any way to recover from this without rebuilding the whole file system? Thanks Janek Here's the full crash log: May 02 16:16:53 xxx077 ceph-mds[3047358]: -29> 2023-05-02T16:16:52.761+0200 7f51f878b700 1 mds.0.1711712 Finished replaying journal May 02 16:16:53 xxx077 ceph-mds[3047358]: -28> 2023-05-02T16:16:52.761+0200 7f51f878b700 1 mds.0.1711712 making mds journal writeable May 02 16:16:53 xxx077 ceph-mds[3047358]: -27> 2023-05-02T16:16:52.761+0200 7f51f878b700 1 mds.0.journaler.mdlog(ro) set_writeable May 02 16:16:53 xxx077 ceph-mds[3047358]: -26> 2023-05-02T16:16:52.761+0200 7f51f878b700 2 mds.0.1711712 i am not alone, moving to state resolve May 02 16:16:53 xxx077 ceph-mds[3047358]: -25> 2023-05-02T16:16:52.761+0200 7f51f878b700 3 mds.0.1711712 request_state up:resolve May 02 16:16:53 xxx077 ceph-mds[3047358]: -24> 2023-05-02T16:16:52.761+0200 7f51f878b700 5 mds.beacon.xxx077 set_want_state: up:replay -> up:resolve May 02 16:16:53 xxx077 ceph-mds[3047358]: -23> 2023-05-02T16:16:52.761+0200 7f51f878b700 5 mds.beacon.xxx077 Sending beacon up:resolve seq 15 May 02 16:16:53 xxx077 ceph-mds[3047358]: -22> 2023-05-02T16:16:52.761+0200 7f51f878b700 10 monclient: _send_mon_message to mon.xxx056 at v2:141.54.133.56:3300/0 May 02 16:16:53 xxx077 ceph-mds[3047358]: -21> 2023-05-02T16:16:53.113+0200 7f51fef98700 10 monclient: tick May 02 16:16:53 xxx077 ceph-mds[3047358]: -20> 2023-05-02T16:16:53.113+0200 7f51fef98700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2023-05-02T16:16:23.118186+0200) May 02 16:16:53 xxx077 ceph-mds[3047358]: -19> 2023-05-02T16:16:53.373+0200 7f51fff9a700 1 mds.xxx077 Updating MDS map to version 1711713 from mon.1 May 02 16:16:53 xxx077 ceph-mds[3047358]: -18> 2023-05-02T16:16:53.373+0200 7f51fff9a700 1 mds.0.1711712 handle_mds_map i am now mds.0.1711712 May 02 16:16:53 xxx077 ceph-mds[3047358]: -17> 2023-05-02T16:16:53.373+0200 7f51fff9a700 1 mds.0.1711712 handle_mds_map state change up:replay --> up:resolve May 02 16:16:53 xxx077 ceph-mds[3047358]: -16> 2023-05-02T16:16:53.373+0200 7f51fff9a700 1 mds.0.1711712 resolve_start May 02 16:16:53 xxx077 ceph-mds[3047358]: -15> 2023-05-02T16:16:53.373+0200 7f51fff9a700 1 mds.0.1711712 reopen_log May 02 16:16:53 xxx077 ceph-mds[3047358]: -14> 2023-05-02T16:16:53.373+0200 7f51fff9a700 1 mds.0.1711712 recovery set is 1,2,3,4 May 02 16:16:53 xxx077 ceph-mds[3047358]: -13> 2023-05-02T16:16:53.373+0200 7f51fff9a700 1 mds.0.1711712 recovery set is 1,2,3,4 May 02 16:16:53 xxx077 ceph-mds[3047358]: -12> 2023-05-02T16:16:53.373+0200 7f5202fa0700 10 monclient: get_auth_request con 0x5574fe74c400 auth_method 0 May 02 16:16:53 xxx077 ceph-mds[3047358]: -11> 2023-05-02T16:16:53.373+0200 7f52037a1700 10 monclient: get_auth_request con 0x5574fe40fc00 auth_method 0 May 02 16:16:53 xxx077 ceph-mds[3047358]: -10> 2023-05-02T16:16:53.373+0200 7f520279f700 10 monclient: get_auth_request con 0x5574f932fc00 auth_method 0 May 02 16:16:53 xxx077 ceph-mds[3047358]: -9> 2023-05-02T16:16:53.373+0200 7f520279f700 10 monclient: get_auth_request con 0x5574ffce2000 auth_method 0 May 02 16:16:53 xxx077 ceph-mds[3047358]: -8> 2023-05-02T16:16:53.377+0200 7f5202fa0700 5 mds.beacon.xxx077 received beacon reply up:resolve seq 15 rtt 0.616008 May 02 16:16:53 xxx077 ceph-mds[3047358]: -7> 2023-05-02T16:16:53.393+0200 7f51fff9a700 5 mds.xxx077 handle_mds_map old map epoch 1711713 <= 1711713, discarding May 02 16:16:53 xxx077 ceph-mds[3047358]: -6> 2023-05-02T16:16:53.393+0200 7f51fff9a700 5 mds.xxx077 handle_mds_map old map epoch 1711713 <= 1711713, discarding May 02 16:16:53 xxx077 ceph-mds[3047358]: -5> 2023-05-02T16:16:53.393+0200 7f51fff9a700 5 mds.xxx077 handle_mds_map old map epoch 1711713 <= 1711713, discarding May 02 16:16:53 xxx077 ceph-mds[3047358]: -4> 2023-05-02T16:16:53.393+0200 7f51fff9a700 5 mds.xxx077 handle_mds_map old map epoch 1711713 <= 1711713, discarding May 02 16:16:53 xxx077 ceph-mds[3047358]: -3> 2023-05-02T16:16:53.545+0200 7f51fff9a700 -1 mds.0.cache.den(0x1 storage) newly corrupt dentry to be committed: [dentry #0x1/storage [2,head] auth (dversion lock) v=78956500 ino=0x10000000000 state=1610612736 | inodepin=1 dirty=1 0x5574f932db80] May 02 16:16:53 xxx077 ceph-mds[3047358]: -2> 2023-05-02T16:16:53.545+0200 7f51fff9a700 -1 log_channel(cluster) log [ERR] : MDS abort because newly corrupt dentry to be committed: [dentry #0x1/storage [2,head] auth (dversion lock) v=78956500 ino=0x10000000000 state=1610612736 | inodepin=1 dirty=1 0x5574f932db80] May 02 16:16:53 xxx077 ceph-mds[3047358]: -1> 2023-05-02T16:16:53.549+0200 7f51fff9a700 -1 /build/ceph-16.2.12/src/mds/CDentry.cc: In function 'bool CDentry::check_corruption(bool)' thread 7f51fff9a700 time 2023-05-02T16:16:53.549536+0200 /build/ceph-16.2.12/src/mds/CDentry.cc: 697: ceph_abort_msg("abort() called") ceph version 16.2.12 (5a2d516ce4b134bfafc80c4274532ac0d56fc1e2) pacific (stable) 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe0) [0x7f52054c3495] 2: (CDentry::check_corruption(bool)+0x86b) [0x5574f7e3a91b] 3: (EMetaBlob::add_dir_context(CDir*, int)+0x507) [0x5574f7f9f9d7] 4: (MDCache::create_subtree_map()+0x13e1) [0x5574f7d20dc1] 5: (MDLog::_journal_segment_subtree_map(MDSContext*)+0x4d) [0x5574f7f2949d] 6: (MDLog::_submit_entry(LogEvent*, MDSLogContextBase*)+0x206) [0x5574f7f29866] 7: (MDCache::log_leader_commit(metareqid_t)+0x277) [0x5574f7ccc3c7] 8: (MDCache::finish_committed_leaders()+0x87) [0x5574f7ccd0d7] 9: (MDCache::maybe_resolve_finish()+0x78) [0x5574f7d38358] 10: (MDCache::handle_resolve(boost::intrusive_ptr<MMDSResolve const> const&)+0x1e02) [0x5574f7d454c2] 11: (MDCache::dispatch(boost::intrusive_ptr<Message const> const&)+0x144) [0x5574f7d47bd4] 12: (MDSRank::handle_message(boost::intrusive_ptr<Message const> const&)+0x733) [0x5574f7bac793] 13: (MDSRank::_dispatch(boost::intrusive_ptr<Message const> const&, bool)+0x643) [0x5574f7bcacb3] 14: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr<Message const> const&)+0x5c) [0x5574f7bcb34c] 15: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x1d6) [0x5574f7b9f226] 16: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0x460) [0x7f5205714020] 17: (DispatchQueue::entry()+0x58f) [0x7f52057118bf] 18: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f52057df261] 19: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f5205205609] 20: clone() May 02 16:16:53 xxx077 ceph-mds[3047358]: 0> 2023-05-02T16:16:53.553+0200 7f51fff9a700 -1 *** Caught signal (Aborted) ** in thread 7f51fff9a700 thread_name:ms_dispatch ceph version 16.2.12 (5a2d516ce4b134bfafc80c4274532ac0d56fc1e2) pacific (stable) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f5205211420] 2: gsignal() 3: abort() 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1af) [0x7f52054c3564] 5: (CDentry::check_corruption(bool)+0x86b) [0x5574f7e3a91b] 6: (EMetaBlob::add_dir_context(CDir*, int)+0x507) [0x5574f7f9f9d7] 7: (MDCache::create_subtree_map()+0x13e1) [0x5574f7d20dc1] 8: (MDLog::_journal_segment_subtree_map(MDSContext*)+0x4d) [0x5574f7f2949d] 9: (MDLog::_submit_entry(LogEvent*, MDSLogContextBase*)+0x206) [0x5574f7f29866] 10: (MDCache::log_leader_commit(metareqid_t)+0x277) [0x5574f7ccc3c7] 11: (MDCache::finish_committed_leaders()+0x87) [0x5574f7ccd0d7] 12: (MDCache::maybe_resolve_finish()+0x78) [0x5574f7d38358] 13: (MDCache::handle_resolve(boost::intrusive_ptr<MMDSResolve const> const&)+0x1e02) [0x5574f7d454c2] 14: (MDCache::dispatch(boost::intrusive_ptr<Message const> const&)+0x144) [0x5574f7d47bd4] 15: (MDSRank::handle_message(boost::intrusive_ptr<Message const> const&)+0x733) [0x5574f7bac793] 16: (MDSRank::_dispatch(boost::intrusive_ptr<Message const> const&, bool)+0x643) [0x5574f7bcacb3] 17: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr<Message const> const&)+0x5c) [0x5574f7bcb34c] 18: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x1d6) [0x5574f7b9f226] 19: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0x460) [0x7f5205714020] 20: (DispatchQueue::entry()+0x58f) [0x7f52057118bf] 21: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f52057df261] 22: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f5205205609] 23: clone() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. May 02 16:16:53 xxx077 ceph-mds[3047358]: --- logging levels --- May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 5 none May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 1 lockdep May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 1 context May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 1 crush May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 mds May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 mds_balancer May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 mds_locker May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 mds_log May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 mds_log_expire May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 mds_migrator May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 1 buffer May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 1 timer May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 1 filer May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 1 striper May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 1 objecter May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 5 rados May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 5 rbd May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 5 rbd_mirror May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 5 rbd_replay May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 5 rbd_pwl May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 5 journaler May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 5 objectcacher May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 5 immutable_obj_cache May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 5 client May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 osd May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 5 optracker May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 5 objclass May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 3 filestore May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 3 journal May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 0 ms May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 mon May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/10 monc May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 paxos May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 5 tp May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 auth May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 crypto May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 1 finisher May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 1 reserver May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 heartbeatmap May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 perfcounter May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 rgw May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 rgw_sync May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/10 civetweb May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 javaclient May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 asok May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 1 throttle May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 0 refs May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 compressor May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 bluestore May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 bluefs May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 3 bdev May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 kstore May 02 16:16:53 xxx077 ceph-mds[3047358]: 4/ 5 rocksdb May 02 16:16:53 xxx077 ceph-mds[3047358]: 4/ 5 leveldb May 02 16:16:53 xxx077 ceph-mds[3047358]: 4/ 5 memdb May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 fuse May 02 16:16:53 xxx077 ceph-mds[3047358]: 2/ 5 mgr May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 mgrc May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 dpdk May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 eventtrace May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 prioritycache May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 5 test May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 5 cephfs_mirror May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 5 cephsqlite May 02 16:16:53 xxx077 ceph-mds[3047358]: 99/99 (syslog threshold) May 02 16:16:53 xxx077 ceph-mds[3047358]: -2/-2 (stderr threshold) May 02 16:16:53 xxx077 ceph-mds[3047358]: --- pthread ID / name mapping for recent threads --- May 02 16:16:53 xxx077 ceph-mds[3047358]: 139990037739264 / May 02 16:16:53 xxx077 ceph-mds[3047358]: 139990054524672 / May 02 16:16:53 xxx077 ceph-mds[3047358]: 139990062917376 / MR_Finisher May 02 16:16:53 xxx077 ceph-mds[3047358]: 139990079702784 / PQ_Finisher May 02 16:16:53 xxx077 ceph-mds[3047358]: 139990096488192 / ms_dispatch May 02 16:16:53 xxx077 ceph-mds[3047358]: 139990130059008 / ceph-mds May 02 16:16:53 xxx077 ceph-mds[3047358]: 139990146844416 / safe_timer May 02 16:16:53 xxx077 ceph-mds[3047358]: 139990163629824 / ms_dispatch May 02 16:16:53 xxx077 ceph-mds[3047358]: 139990180415232 / io_context_pool May 02 16:16:53 xxx077 ceph-mds[3047358]: 139990197200640 / admin_socket May 02 16:16:53 xxx077 ceph-mds[3047358]: 139990205593344 / msgr-worker-2 May 02 16:16:53 xxx077 ceph-mds[3047358]: 139990213986048 / msgr-worker-1 May 02 16:16:53 xxx077 ceph-mds[3047358]: 139990222378752 / msgr-worker-0 May 02 16:16:53 xxx077 ceph-mds[3047358]: 139990239524736 / ceph-mds May 02 16:16:53 xxx077 ceph-mds[3047358]: max_recent 10000 May 02 16:16:53 xxx077 ceph-mds[3047358]: max_new 10000 May 02 16:16:53 xxx077 ceph-mds[3047358]: log_file /var/lib/ceph/crash/2023-05-02T14:16:53.555508Z_0b05f5cb-130c-4979-95ca-1ba7f31cf7e5/log May 02 16:16:53 xxx077 ceph-mds[3047358]: --- end dump of recent events --- -- Bauhaus-Universität Weimar Bauhausstr. 9a, R308 99423 Weimar, Germany Phone: +49 3643 58 3577 www.webis.de

3 9

Radosgw multisite replication issues
by Tarrago, Eli (RIS-BCT) 12 May '23

12 May '23

Good Afternoon, I am experiencing an issue where east-1 is no longer able to replicate from west-1, however, after a realm pull, west-1 is now able to replicate from east-1. In other words: West <- Can Replicate <- East West -> Cannot Replicate -> East After confirming the access and secret keys are identical on both sides, I restarted all radosgw services. Here is the current status of the cluster below. Thank you for your help, Eli Tarrago root@east01:~# radosgw-admin zone get { "id": "ddd66ab8-0417-46ee-a53b-043352a63f93", "name": "rgw-east", "domain_root": "rgw-east.rgw.meta:root", "control_pool": "rgw-east.rgw.control", "gc_pool": "rgw-east.rgw.log:gc", "lc_pool": "rgw-east.rgw.log:lc", "log_pool": "rgw-east.rgw.log", "intent_log_pool": "rgw-east.rgw.log:intent", "usage_log_pool": "rgw-east.rgw.log:usage", "roles_pool": "rgw-east.rgw.meta:roles", "reshard_pool": "rgw-east.rgw.log:reshard", "user_keys_pool": "rgw-east.rgw.meta:users.keys", "user_email_pool": "rgw-east.rgw.meta:users.email", "user_swift_pool": "rgw-east.rgw.meta:users.swift", "user_uid_pool": "rgw-east.rgw.meta:users.uid", "otp_pool": "rgw-east.rgw.otp", "system_key": { "access_key": "PxxxxxxxxxxxxxxxxW", "secret_key": "Hxxxxxxxxxxxxxxxx6" }, "placement_pools": [ { "key": "default-placement", "val": { "index_pool": "rgw-east.rgw.buckets.index", "storage_classes": { "STANDARD": { "data_pool": "rgw-east.rgw.buckets.data" } }, "data_extra_pool": "rgw-east.rgw.buckets.non-ec", "index_type": 0 } } ], "realm_id": "98e0e391-16fb-48da-80a5-08437fd81789", "notif_pool": "rgw-east.rgw.log:notif" } root@west01:~# radosgw-admin zone get { "id": "b2a4a31c-1505-4fdc-b2e0-ea07d9463da1", "name": "rgw-west", "domain_root": "rgw-west.rgw.meta:root", "control_pool": "rgw-west.rgw.control", "gc_pool": "rgw-west.rgw.log:gc", "lc_pool": "rgw-west.rgw.log:lc", "log_pool": "rgw-west.rgw.log", "intent_log_pool": "rgw-west.rgw.log:intent", "usage_log_pool": "rgw-west.rgw.log:usage", "roles_pool": "rgw-west.rgw.meta:roles", "reshard_pool": "rgw-west.rgw.log:reshard", "user_keys_pool": "rgw-west.rgw.meta:users.keys", "user_email_pool": "rgw-west.rgw.meta:users.email", "user_swift_pool": "rgw-west.rgw.meta:users.swift", "user_uid_pool": "rgw-west.rgw.meta:users.uid", "otp_pool": "rgw-west.rgw.otp", "system_key": { "access_key": "PxxxxxxxxxxxxxxW", "secret_key": "Hxxxxxxxxxxxxxx6" }, "placement_pools": [ { "key": "default-placement", "val": { "index_pool": "rgw-west.rgw.buckets.index", "storage_classes": { "STANDARD": { "data_pool": "rgw-west.rgw.buckets.data" } }, "data_extra_pool": "rgw-west.rgw.buckets.non-ec", "index_type": 0 } } ], "realm_id": "98e0e391-16fb-48da-80a5-08437fd81789", "notif_pool": "rgw-west.rgw.log:notif" east01:~# radosgw-admin metadata sync status { "sync_status": { "info": { "status": "init", "num_shards": 0, "period": "", "realm_epoch": 0 }, "markers": [] }, "full_sync": { "total": 0, "complete": 0 } } west01:~# radosgw-admin metadata sync status { "sync_status": { "info": { "status": "sync", "num_shards": 64, "period": "44b6b308-e2d8-4835-8518-c90447e7b55c", "realm_epoch": 3 }, "markers": [ { "key": 0, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 46, "pos": 0, "timestamp": "0.000000", "realm_epoch": 3 } }, #### goes on for a long time… { "key": 63, "val": { "state": 1, "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000", "realm_epoch": 3 } } ] }, "full_sync": { "total": 46, "complete": 46 } } east01:~# radosgw-admin sync status realm 98e0e391-16fb-48da-80a5-08437fd81789 (rgw-blobs) zonegroup 0e0faf4e-39f5-402e-9dbb-4a1cdc249ddd (EastWestceph) zone ddd66ab8-0417-46ee-a53b-043352a63f93 (rgw-east) metadata sync no sync (zone is master) 2023-04-20T19:03:13.388+0000 7f25fa036c80 0 ERROR: failed to fetch datalog info data sync source: b2a4a31c-1505-4fdc-b2e0-ea07d9463da1 (rgw-west) failed to retrieve sync info: (13) Permission denied west01:~# radosgw-admin sync status realm 98e0e391-16fb-48da-80a5-08437fd81789 (rgw-blobs) zonegroup 0e0faf4e-39f5-402e-9dbb-4a1cdc249ddd (EastWestceph) zone b2a4a31c-1505-4fdc-b2e0-ea07d9463da1 (rgw-west) metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master data sync source: ddd66ab8-0417-46ee-a53b-043352a63f93 (rgw-east) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 16 shards behind shards: [5,56,62,65,66,70,76,86,87,94,104,107,111,113,120,126] oldest incremental change not applied: 2023-04-20T19:02:48.783283+0000 [5] east01:~# radosgw-admin zonegroup get { "id": "0e0faf4e-39f5-402e-9dbb-4a1cdc249ddd", "name": "EastWestceph", "api_name": "EastWestceph", "is_master": "true", "endpoints": [ http://east01.example.net:8080, http://east02.example.net:8080, http://east03.example.net:8080, http://west01.example.net:8080, http://west02.example.net:8080, http://west03.example.net:8080 ], "hostnames": [ "eastvip.example.net", "westvip.example.net" ], "hostnames_s3website": [], "master_zone": "ddd66ab8-0417-46ee-a53b-043352a63f93", "zones": [ { "id": "b2a4a31c-1505-4fdc-b2e0-ea07d9463da1", "name": "rgw-west", "endpoints": [ http://west01.example.net:8080, http://west02.example.net:8080, http://west03.example.net:8080 ], "log_meta": "false", "log_data": "true", "bucket_index_max_shards": 0, "read_only": "false", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" }, { "id": "ddd66ab8-0417-46ee-a53b-043352a63f93", "name": "rgw-east", "endpoints": [ http://east01.example.net:8080, http://east02.example.net:8080, http://east03.example.net:8080 ], "log_meta": "false", "log_data": "true", "bucket_index_max_shards": 0, "read_only": "false", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" } ], "placement_targets": [ { "name": "default-placement", "tags": [], "storage_classes": [ "STANDARD" ] } ], "default_placement": "default-placement", "realm_id": "98e0e391-16fb-48da-80a5-08437fd81789", "sync_policy": { "groups": [] } } ________________________________ The information contained in this e-mail message is intended only for the personal and confidential use of the recipient(s) named above. This message may be an attorney-client communication and/or work product and as such is privileged and confidential. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail, and delete the original message.

4 6

client isn't responding to mclientcaps(revoke), pending pAsLsXsFsc issued pAsLsXsFsc
by Frank Schilder 12 May '23

12 May '23

Hi all, I think we might be hitting a known problem (https://tracker.ceph.com/issues/57244) I don't want to fail the mds yet, because we have troubles with older kclients that miss the mds restart and hold on to cache entries referring to the killed instance, leading to hanging jobs on our HPC cluster. I have seen this issue before and there was a process in D-state that dead-locked itself. Usually, killing this process succeeded and resolved the issue. However, this time I can't find such a process. The tracker mentions that one can delete the file/folder. I have the inode number, but really don't want to start a find on a 1.5PB file system. Is there a better way to find what path is causing the issue (ask the MDS directly, look at a cache dump, or similar)? Is there an alternative to deletion or MDS fail? Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

4 8

docker restarting lost all managers accidentally
by Ben 12 May '23

12 May '23

Hi, This cluster is deployed by cephadm 17.2.5,containerized. It ends up in this(no active mgr): [root@8cd2c0657c77 /]# ceph -s cluster: id: ad3a132e-e9ee-11ed-8a19-043f72fb8bf9 health: HEALTH_WARN 6 hosts fail cephadm check no active mgr 1/3 mons down, quorum h18w,h19w Degraded data redundancy: 781908/2345724 objects degraded (33.333%), 101 pgs degraded, 209 pgs undersized services: mon: 3 daemons, quorum h18w,h19w (age 19m), out of quorum: h15w mgr: no daemons active (since 5h) mds: 1/1 daemons up, 1 standby osd: 9 osds: 6 up (since 5h), 6 in (since 5h) rgw: 2 daemons active (2 hosts, 1 zones) data: volumes: 1/1 healthy pools: 8 pools, 209 pgs objects: 781.91k objects, 152 GiB usage: 312 GiB used, 54 TiB / 55 TiB avail pgs: 781908/2345724 objects degraded (33.333%) 108 active+undersized 101 active+undersized+degraded I checked the h20w, there is a manager container running with log: debug 2023-05-10T12:43:23.315+0000 7f5e152ec000 0 monclient(hunting): authenticate timed out after 300 debug 2023-05-10T12:48:23.318+0000 7f5e152ec000 0 monclient(hunting): authenticate timed out after 300 debug 2023-05-10T12:53:23.318+0000 7f5e152ec000 0 monclient(hunting): authenticate timed out after 300 debug 2023-05-10T12:58:23.319+0000 7f5e152ec000 0 monclient(hunting): authenticate timed out after 300 debug 2023-05-10T13:03:23.319+0000 7f5e152ec000 0 monclient(hunting): authenticate timed out after 300 debug 2023-05-10T13:08:23.319+0000 7f5e152ec000 0 monclient(hunting): authenticate timed out after 300 debug 2023-05-10T13:13:23.319+0000 7f5e152ec000 0 monclient(hunting): authenticate timed out after 300 any idea to get a mgr up running again through cephadm? Thanks, Ben

2 2

Lua scripting in the rados gateway
by Thomas Bennett 11 May '23

11 May '23

Hi ceph users, I've been trying out the lua scripting for the rados gateway (thanks Yuval). As in my previous email I mentioned that there is an error when trying to load the luasocket module. However, I thought it was a good time to report on my progress. My 'hello world' example below is called *test.lua* below includes the following checks: 1. Can I write to the debug log? 2. Can I use the lua socket package to do something stupid but intersting, like connect to a webservice? Before you continue reading this, you might need to know that I run all ceph processes in a *CentOS Stream release 8 *container deployed using ceph orchestrator running *Ceph v17.2.5*, so please view the information below in that context. For anyone looking for a reference, I suggest going to the ceph lua rados gateway documentation at radosgw/lua-scripting <https://docs.ceph.com/en/quincy/radosgw/lua-scripting/>. There are two new switches you need to know about in the radosgw-admin: - *script* -> loading your lua script - *script-package* -> loading supporting packages for your script - e.i. luasocket in this case. For a basic setup, you'll need to have a few dependencies in your containers: - cephadm container: requires luarocks (I've checked the code - it runs a luarocks search command) - radosgw container: requires luarocks, gcc, make, m4, wget (wget just in case). To achieve the above, I updated the container image for our running system. I needed to do this because I needed to redeploy the rados gateway container to inject the lua script packages into the radosgw runtime process. This will start with a fresh container based on the global config *container_image* setting on your running system. For us this is currently captured in *quay.io/tsolo/ceph:v17.2.5-3 <http://quay.io/tsolo/ceph:v17.2.5-3>* and included the following exta steps (including installing the lua dev from an rpm because there is no centos package in yum): yum install luarocks gcc make wget m4 rpm -i https://rpmfind.net/linux/centos/8-stream/PowerTools/x86_64/os/Packages/lua… You will notice that I've included a compiler and compiler support into the image. This is because luarocks on the radosgw to compile luasocket (the package I want to install). This will happen at start time when the radosgw is restarted from ceph orch. In the cephadm container I still need to update our cephadm shell so I need to install luarocks by hand: yum install luarocks Then set thew updated image to use: ceph config set global container_image quay.io/tsolo/ceph:v17.2.5-3 I now create a file called: *test.lua* in the cephadm container. This contains the following lines to write to the log and then do a get request to google. This is not practical in production, but it serves the purpose of testing the infrastructure: RGWDebugLog("Tsolo start lua script") local LuaSocket = require("socket") client = LuaSocket.connect("google.com", 80) client:send("GET / HTTP/1.0\r\nHost: google.com\r\n\r\n") while true do s, status, partial = client:receive('*a') RGWDebugLog(s or partial) if status == "closed" then break end end client:close() RGWDebugLog("Tsolo stop lua") Next I run: radosgw-admin script-package add --package=luasocket --allow-compilation And then list the added package to make sure it is there: radosgw-admin script-package list Note - at this point the radosgw has not been modified, it must first be restarted. Then I put the *test.lua *script into the pre request context: radosgw-admin script put --infile=test.lua --context=preRequest You also need to raise the debug log level on the running rados gateway: ceph daemon /var/run/ceph/ceph-client.rgw.xxx.xxx-cms1.xxxxx.x.xxxxxxxxxxxxxx.asok config set debug_rgw 20 Inside the radosgw container I apply my fix (as per previous email): cp -ru /tmp/luarocks/client.rgw.xxxxxx.xxx-xxxx-xxxx.pcoulb/lib64/* /tmp/luarocks/client.rgw.xxxxxx.xxx-xxxx-xxxx.pcoulb/lib/ Outside on the host running the radosgw-admin container I follow the journalctl for the radosgw container (to get the logs): journalctl -fu ceph-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx@rgw. xxx.xxx-cms1.xxxxx.x.xxxxxxxxxxxxxx.service Then I run an s3cmd to put data in via the rados gateway and check the journalctl logs and see: Apr 25 20:54:47 brp-ceph-cms1 radosgw[60901]: Lua INFO: Tsolo start lua Apr 25 20:54:47 brp-ceph-cms1 radosgw[60901]: Lua INFO: HTTP/1.0 301 Moved Permanently Apr 25 20:54:47 brp-ceph-cms1 radosgw[60901]: Lua INFO: Apr 25 20:54:47 brp-ceph-cms1 radosgw[60901]: Lua INFO: Tsolo stop lua Apr 25 20:54:47 brp-ceph-cms1 radosgw[60901]: Lua INFO: Tsolo start lua Apr 25 20:54:48 brp-ceph-cms1 radosgw[60901]: Lua INFO: HTTP/1.0 301 Moved Permanently Apr 25 20:54:48 brp-ceph-cms1 radosgw[60901]: Lua INFO: Apr 25 20:54:48 brp-ceph-cms1 radosgw[60901]: Lua INFO: Tsolo stop lua So the script worked :) If you want to see where the luarocks libraries have been installed, look in the /tmp/ directory of the radosgw container after you redeploy it and you will find the content in /tmp/luarocks. Conclusions: There was a bit to figure out to get this working, but now that I've got this simple test working I think there is a lot more to look into and discover and use w.r.t. this powerful tool. Cheers, Tom

2 5

multisite sync and multipart uploads
by Yixin Jin 11 May '23

11 May '23

Hi guys, With Quincy release, does anyone know how multisite sync deals with multipart uploads? I mean those part objects of some incomplete multipart uploads. Are those objects also sync-ed over either with full-sync or incremental sync? I did a quick experiment and notice that these objects are not sync-ed over. Is it intentional or is there a defect of it? Thanks, Yixin

2 1

multisite synchronization and multipart uploads
by Yixin Jin 11 May '23

11 May '23

Hi folks,

1 0

Recovering from OSD with corrupted DB
by Jessica Sol 11 May '23

11 May '23

Hey all, I recently had two OSDs fail. The first one I just removed and expected replication to fix for me. Replication froze and I restarted the OSDs after seeing heartbeat failures. It allowed replication to resume but one of the OSD's RocksDB became corrupted, showing this error when I try to bring it up: rocksdb: verify_sharding unable to list column families: Corruption: CURRENT file does not end with newline bluestore(/var/lib/ceph/osd/ceph-1) _open_db erroring opening db: osd.1 0 OSD:init: unable to mount object store ** ERROR: osd init failed: (5) Input/output error I found a relevant bug report[1] with a workaround, but it requires ceph-bluestore-tool to have bluefs-import, which was only added in 16.2.11, I'm on 16.2.9 right now. Is it safe to do a patch upgrade from 16.2.9 to 16.2.11 without having an OSD running? Or an alternative way of running the tool without upgrading? Is the tool even safe to run on data from 16.2.9? Thanks, Jess [1] https://tracker.ceph.com/issues/47330

1 0

Protected snapshot deleted when RBD image is deleted
by Work Ceph 10 May '23

10 May '23

Hello guys, We have a doubt regarding snapshot management, when a protected snapshot is created, should it be deleted when its RBD image is removed from the system? If not, how can we list orphaned snapshots in a pool?

2 3

Ceph Leadership Team Meeting Minutes - May 10, 2023
by Ernesto Puerta 10 May '23

10 May '23

Hi Cephers, These are the topics that we just covered in today's meeting: - *Issues recording our meetings in Jitsi (Mike Perez)* - David Orman suggested using a self-hosted Jitsi instance: https://jitsi.github.io/handbook/docs/devops-guide/. Tested on a single container, with 4 cores for 4-5 attendees a call. - Help needed from Dan Mick & Adam Kraitman. - Ilya: this might scale for smaller meetings (dailies), but not for the larger ones (CDM, User-Devs, etc.). - *Reef release status review* - Josh: CentOS 9 blocked by Dashboard/Cephadm Python deps missing in CentOS 9. Casey: It's blocking teuthology testing. - Already discussed in devs mailing list a couple of months ago, for Quincy + Centos 9 (the missing packages would be the same). - Ken Dreyer preferred to keep the legacy approach (distro packages) instead of embedding Python deps. - Casey, Matt & Ernesto to resume that discussion with Ken Dreyer. - Other issues: - Radek: - Performance issue with RocksDB config. - Mismatching client-server features during upgrade. msgr encoder-decoder issues, introduced feature bit for Squid ( https://github.com/ceph/ceph/commit/1049d3e5eff0b7fa4fc9e5853494cb21c10b290a) Performance concerns. To be further discussed at Mark's Perf meeting (incl. Yuval) - Paul Cuzner perf testing Reef: higher CPU usage in Reef vs Quincy (more time spent in RocksDB get calls) - Target GA remains this June. Missing CentOS 9 packages becoming a blocker issue for the release. - *Perf regression in Pacific vs Nautilus (David Orman)* - https://tracker.ceph.com/issues/58530 - Mark: Missing change in upstream Rocksdb project. - David Orman: Is this degradation still happening in newer Rocksdb versions (reef)? Mark: No reason to think otherwise. Kind Regards, Ernesto

1 0

How can I use not-replicated pool (replication 1 or raid-0)
by mhnx 10 May '23

10 May '23

Hello. I have a 10 node cluster. I want to create a non-replicated pool (replication 1) and I want to ask some questions about it: Let me tell you my use case: - I don't care about losing data, - All of my data is JUNK and these junk files are usually between 1KB to 32MB. - These files will be deleted in 5 days. - Writable space and I/O speed is more important. - I have high Write/Read/Delete operations, minimum 200GB a day. I'm afraid that, in any failure, I won't be able to access the whole cluster. Losing data is okay but I have to ignore missing files, remove the data from the cluster and continue with existing data and while doing this, I want to be able to write new data to the cluster. My questions are: 1- To reach this goal do you have any recommendations? 2- With this setup, what potential problems do you have in mind? 3- I think Erasure Coding is not a choice because of the performance problems and slow file deletion. With this I/O need EC will miss files and leaks may happen (I've seen before on Nautilus). 4- You read my needs, is there a better way to do this? Thank you for the answers. Best regards.

3 8

Re: 17.2.6 Dashboard/RGW Signature Mismatch
by Ondřej Kukla 10 May '23

10 May '23

Hello, I have found out that the issue seems to be in this change - https://github.com/ceph/ceph/pull/47207 When I’ve commented out the change and replaced it with the previous value the dashboard works as expected. Ondrej

1 0

Re: 17.2.6 Dashboard/RGW Signature Mismatch
by Ondřej Kukla 10 May '23

10 May '23

Hello, I have found out that the issue seems to be in this change - https://github.com/ceph/ceph/pull/47207 When I’ve commented out the change and replaced it with the previous value the dashboard works as expected. Ondrej

1 0

non root deploy ceph 17.2.5 failed
by Ben 10 May '23

10 May '23

Hi, with following command: sudo cephadm --docker bootstrap --mon-ip 10.1.32.33 --skip-monitoring-stack --ssh-user deployer the user deployer has passwordless sudo configuration. I can see the error below: debug 2023-05-04T12:46:43.268+0000 7fc5ddc2e700 0 [cephadm ERROR cephadm.ssh] Unable to write szhyf-xx1d002-hx15w:/var/lib/ceph/ad3a132e-e9ee-11ed-8a19-043f72fb8bf9/cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e: scp: /tmp/var/lib/ceph/ad3a132e-e9ee-11ed-8a19-043f72fb8bf9/cephadm.059bfc99f5cf36ed881f2494b104711faf4cbf5fc86a9594423cc105cafd9b4e.new: Permission denied Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/ssh.py", line 222, in _write_remote_file await asyncssh.scp(f.name, (conn, tmp_path)) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 922, in scp await source.run(srcpath) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 458, in run self.handle_error(exc) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in handle_error raise exc from None File "/lib/python3.6/site-packages/asyncssh/scp.py", line 456, in run await self._send_files(path, b'') File "/lib/python3.6/site-packages/asyncssh/scp.py", line 438, in _send_files self.handle_error(exc) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in handle_error raise exc from None File "/lib/python3.6/site-packages/asyncssh/scp.py", line 434, in _send_files await self._send_file(srcpath, dstpath, attrs) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 365, in _send_file await self._make_cd_request(b'C', attrs, size, srcpath) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 343, in _make_cd_request self._fs.basename(path)) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 224, in make_request raise exc Any ideas on this? Thanks, Ben

3 10

cephadm docker registry
by Satish Patel 10 May '23

10 May '23

Folks, I am trying to install ceph on 10 node clusters and planning to use cephadm. My question is if next year i will add new nodes to this cluster then what docker image version cephadm will use to add new nodes? Are there any local registry can i create one to copy images locally? How does cephadm control images?

2 1

v16.2.13 Pacific released
by Yuri Weinstein 10 May '23

10 May '23

We're happy to announce the 13th backport release in the Pacific series. https://ceph.io/en/news/blog/2023/v16-2-13-pacific-released/ Notable Changes --------------- * CEPHFS: Rename the `mds_max_retries_on_remount_failure` option to `client_max_retries_on_remount_failure` and move it from mds.yaml.in to mds-client.yaml.in because this option was only used by MDS client from its birth. * `ceph mgr dump` command now outputs `last_failure_osd_epoch` and `active_clients` fields at the top level. Previously, these fields were output under `always_on_modules` field. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-16.2.13.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: 5378749ba6be3a0868b51803968ee9cde4833a3e

3 2

ceph status is warning with "266 pgs not deep-scrubbed in time"
by Louis Koo 10 May '23

10 May '23

ceph version is: v16.2.10 But I had close the ration with "ceph config set mon mon_warn_pg_not_deep_scrubbed_ratio 0" show the prints: [root@smd-node01 deeproute]# ceph config get mon mon_warn_pg_not_deep_scrubbed_ratio 0.000000

1 0

Change in DMARC handling for the list
by Dan Mick 09 May '23

09 May '23

Several users have complained for some time that our DMARC/DKIM handling is not correct. I've recently had time to go study DMARC, DKIM, SPF, SRS, and other tasty morsels of initialisms, and have thus made a change to how Mailman handles DKIM signatures for the list: If a domain advertises that it will reject or quarantine messages that fail DKIM (through its DMARC policy in the DNS text record _dmarc.<domain>), the message will be rewritten to be "From" ceph.io, and SPF should be correct. I do not know if it will regenerate a DKIM signature in that case for what is now it's own message. The From: address will say something like "From Original Sender via ceph-users <ceph-users(a)ceph.io> so it's somewhat clear who first sent the message, and Reply-To will be set to Original Sender. Again, this will only happen for senders from domains that advertise a strict DMARC policy. This does not include gmail.com, surprisingly. Let me know if you notice anything that seems to have gotten worse. Next on the list is to investigate if DKIM-signing outbound messages, or at least ones that don't already have an ARC-Seal, is appropriate and/or workable.

2 1

ERROR: Distro uos version 20 not supported
by Ben 09 May '23

09 May '23

Hi, This seems not very relevant since all ceph components are running in containers. Any ideas to get over this issue? Any other ideas or suggestions on this kind of deployment? sudo ./cephadm --image 10.21.22.1:5000/ceph:v17.2.5-20230316 --docker bootstrap --mon-ip 10.21.22.1 --skip-monitoring-stack Creating directory /etc/ceph for ceph.conf Verifying podman|docker is present... Verifying lvm2 is present... Verifying time synchronization is in place... No time sync service is running; checked for ['chrony.service', 'chronyd.service', 'systemd-timesyncd.service', 'ntpd.service', 'ntp.service', 'ntpsec.service', 'openntpd.service'] ERROR: Distro uos version 20 not supported uname -a Linux aaaa 4.19.0-91.82.42.uelc20.x86_64 #1 SMP Sat May 15 13:50:04 CST 2021 x86_64 x86_64 x86_64 GNU/Linux Thank you in advance Ben

1 1

pg deep-scrub issue
by Peter 08 May '23

08 May '23

Dear all, I am writing to seek your assistance in resolving an issue with my Ceph cluster. Currently, the cluster is experiencing a problem where the number of Placement Groups (PGs) that need to undergo deepscrub and regular scrub cannot be completed in a timely manner. I have noticed that these PGs appear to be concentrated on a single OSD. I am seeking your guidance on how to address this issue and would appreciate any insights or suggestions you may have. Please find attached the Ceph health detail output for your reference. HEALTH_WARN 13 pgs not deep-scrubbed in time; 7 pgs not scrubbed in time [WRN] PG_NOT_DEEP_SCRUBBED: 13 pgs not deep-scrubbed in time pg 4.426 not deep-scrubbed since 2023-04-22T10:00:21.529716+0800 pg 4.f0 not deep-scrubbed since 2023-04-22T04:55:17.868881+0800 pg 4.b9 not deep-scrubbed since 2023-04-22T16:47:25.219603+0800 pg 4.87 not deep-scrubbed since 2023-04-22T20:01:02.508600+0800 pg 4.31 not deep-scrubbed since 2023-04-23T00:27:39.299893+0800 pg 4.5b9 not deep-scrubbed since 2023-04-19T21:03:47.041934+0800 pg 4.68a not deep-scrubbed since 2023-04-21T19:52:39.251293+0800 pg 4.6a4 not deep-scrubbed since 2023-04-22T16:20:51.078431+0800 pg 4.6ec not deep-scrubbed since 2023-04-21T11:20:33.661595+0800 pg 4.7a4 not deep-scrubbed since 2023-04-20T22:30:44.506420+0800 pg 4.7a2 not deep-scrubbed since 2023-04-16T12:05:56.586205+0800 pg 4.7b4 not deep-scrubbed since 2023-04-17T15:50:10.595292+0800 pg 4.7c8 not deep-scrubbed since 2023-04-19T15:10:12.673655+0800 [WRN] PG_NOT_SCRUBBED: 7 pgs not scrubbed in time pg 4.31e not scrubbed since 2023-04-24T13:34:26.103257+0800 pg 4.5b9 not scrubbed since 2023-04-24T07:20:53.891175+0800 pg 4.68a not scrubbed since 2023-04-24T03:37:58.070854+0800 pg 4.7a4 not scrubbed since 2023-04-24T02:55:25.912789+0800 pg 4.7b4 not scrubbed since 2023-04-24T10:04:46.889422+0800 pg 4.7c8 not scrubbed since 2023-04-24T13:36:07.284271+0800 pg 4.7d2 not scrubbed since 2023-04-24T14:47:19.365551+0800 Peter

3 3

16.2.13 pacific QE validation status
by Yuri Weinstein 08 May '23

08 May '23

Details of this release are summarized here: https://tracker.ceph.com/issues/59542#note-1 Release Notes - TBD Seeking approvals for: smoke - Radek, Laura rados - Radek, Laura rook - Sébastien Han cephadm - Adam K dashboard - Ernesto rgw - Casey rbd - Ilya krbd - Ilya fs - Venky, Patrick upgrade/octopus-x (pacific) - Laura (look the same as in 16.2.8) upgrade/pacific-p2p - Laura powercycle - Brad (SELinux denials) ceph-volume - Guillaume, Adam K Thx YuriW

10 26

rgw service fails to start with zone not found
by Adiga, Anantha 08 May '23

08 May '23

Hi, An existing multisite configuration was removed. But the radosgw services still see the old zone name and fail to start. journalctl -u ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e(a)rgw.default.default.fl31ca104ja0201.ninovs ........... May 08 16:10:48 fl31ca104ja0201 bash[3964341]: debug 2023-05-08T16:10:48.897+0000 7f2f47634740 0 deferred set uid:gid to 167:167 (ceph:ceph) May 08 16:10:48 fl31ca104ja0201 bash[3964341]: debug 2023-05-08T16:10:48.897+0000 7f2f47634740 0 ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989> May 08 16:10:48 fl31ca104ja0201 bash[3964341]: debug 2023-05-08T16:10:48.897+0000 7f2f47634740 0 framework: beast May 08 16:10:48 fl31ca104ja0201 bash[3964341]: debug 2023-05-08T16:10:48.897+0000 7f2f47634740 0 framework conf key: port, val: 8080 May 08 16:10:48 fl31ca104ja0201 bash[3964341]: debug 2023-05-08T16:10:48.897+0000 7f2f47634740 1 radosgw_Main not setting numa affinity May 08 16:10:48 fl31ca104ja0201 bash[3964341]: debug 2023-05-08T16:10:48.901+0000 7f2f47634740 1 rgw_d3n: rgw_d3n_l1_local_datacache_enabled=0 May 08 16:10:48 fl31ca104ja0201 bash[3964341]: debug 2023-05-08T16:10:48.901+0000 7f2f47634740 1 D3N datacache enabled: 0 May 08 16:10:48 fl31ca104ja0201 bash[3964341]: debug 2023-05-08T16:10:48.917+0000 7f2f47634740 0 rgw main: ERROR: could not find zone (fl2site2) May 08 16:10:48 fl31ca104ja0201 bash[3964341]: debug 2023-05-08T16:10:48.917+0000 7f2f47634740 0 rgw main: ERROR: failed to start notify service> May 08 16:10:48 fl31ca104ja0201 bash[3964341]: debug 2023-05-08T16:10:48.917+0000 7f2f47634740 0 rgw main: ERROR: failed to init services (ret=(> May 08 16:10:48 fl31ca104ja0201 bash[3964341]: debug 2023-05-08T16:10:48.921+0000 7f2f47634740 -1 Couldn't init storage provider (RADOS) May 08 16:10:49 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e(a)rgw.default.default.fl31ca104ja0201.ninovs.service: Main pr> May 08 16:10:49 fl31ca104ja0201 systemd[1]: ceph-d0a3b6e0-d2c3-11ed-be05-a7a3a1d7a87e(a)rgw.default.default.fl31ca104ja0201.ninovs.service: Failed Here is the current configuration root@fl31ca104ja0201:/# radosgw-admin period get { "id": "729f7cef-6340-4750-b3ae-9164177c0df3", "epoch": 1, "predecessor_uuid": "1d124ad5-57b0-41de-8def-823bd40f72aa", "sync_status": [], "period_map": { "id": "729f7cef-6340-4750-b3ae-9164177c0df3", "zonegroups": [ { "id": "21b8306c-be43-4567-a0c5-74ab69937535", "name": "default", "api_name": "default", "is_master": "true", "endpoints": [], "hostnames": [], "hostnames_s3website": [], "master_zone": "ff468492-8a14-414b-a8cf-7c20ab699af3", "zones": [ { "id": "ff468492-8a14-414b-a8cf-7c20ab699af3", "name": "default", "endpoints": [], "log_meta": "false", "log_data": "false", "bucket_index_max_shards": 11, "read_only": "false", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" } ], "placement_targets": [ { "name": "default-placement", "tags": [], "storage_classes": [ "STANDARD" ] } ], "default_placement": "default-placement", "realm_id": "8999e04c-17a4-4150-8845-cecd6672d312", "sync_policy": { "groups": [] } } ], "short_zone_ids": [ { "key": "ff468492-8a14-414b-a8cf-7c20ab699af3", "val": 3883136521 } ] }, "master_zonegroup": "21b8306c-be43-4567-a0c5-74ab69937535", "master_zone": "ff468492-8a14-414b-a8cf-7c20ab699af3", "period_config": { "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "user_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "user_ratelimit": { "max_read_ops": 0, "max_write_ops": 0, "max_read_bytes": 0, "max_write_bytes": 0, "enabled": false }, "bucket_ratelimit": { "max_read_ops": 0, "max_write_ops": 0, "max_read_bytes": 0, "max_write_bytes": 0, "enabled": false }, "anonymous_ratelimit": { "max_read_ops": 0, "max_write_ops": 0, "max_read_bytes": 0, "max_write_bytes": 0, "enabled": false } }, "realm_id": "8999e04c-17a4-4150-8845-cecd6672d312", "realm_name": "default", "realm_epoch": 2 } root@fl31ca104ja0201:/# radosgw-admin realm list { "default_info": "8999e04c-17a4-4150-8845-cecd6672d312", "realms": [ "default" ] } root@fl31ca104ja0201:/# radosgw-admin zonegroup list { "default_info": "21b8306c-be43-4567-a0c5-74ab69937535", "zonegroups": [ "default" ] } root@fl31ca104ja0201:/# radosgw-admin zone list { "default_info": "", "zones": [ "default" ] } Where is the radosgw service getting the old zone info from? How to fix this configuration? Thank you, Anantha

3 4

Ceph Host offline after performing dnf upgrade on RHEL 8.7 host
by Mevludin Blazevic 08 May '23

08 May '23

Hi all, after I performed a minor RHEL package upgrade (8.7 -> 8.7) in one of our Ceph hosts, I get a Ceph warning describing that cephadm "Can't communicate with remote host `...`, possibly because python3 is not installed there: [Errno 12] Cannot allocate memory, although Python3 is installed. However, I figured out that these packages were updated: Installing: kernel 4.18.0-425.19.2.el8_7 kernel-core 4.18.0-425.19.2.el8_7 kernel-modules 4.18.0-425.19.2.el8_7 Upgrading: NetworkManager 1:1.40.0-6.el8_7 NetworkManager-config-server 1:1.40.0-6.el8_7 NetworkManager-initscripts-updown 1:1.40.0-6.el8_7 NetworkManager-libnm 1:1.40.0-6.el8_7 NetworkManager-team 1:1.40.0-6.el8_7 NetworkManager-tui 1:1.40.0-6.el8_7 authselect 1.2.5-2.el8_7 authselect-compat 1.2.5-2.el8_7 authselect-libs 1.2.5-2.el8_7 bpftool 4.18.0-425.19.2.el8_7 buildah 1:1.27.3-1.module+el8.7.0+17824+66a0202b cockpit-podman 53-1.module+el8.7.0+17824+66a0202b conmon 3:2.1.4-1.module+el8.7.0+17824+66a0202b container-selinux 2:2.189.0-1.module+el8.7.0+17824+66a0202b containernetworking-plugins 1:1.1.1-3.module+el8.7.0+17824+66a0202b containers-common 2:1-46.module+el8.7.0+17824+66a0202b criu 3.15-3.module+el8.7.0+17824+66a0202b curl 7.61.1-25.el8_7.3 dbus 1:1.12.8-23.el8_7.1 dbus-common 1:1.12.8-23.el8_7.1 dbus-daemon 1:1.12.8-23.el8_7.1 dbus-libs 1:1.12.8-23.el8_7.1 dbus-tools 1:1.12.8-23.el8_7.1 device-mapper-multipath 0.8.4-28.el8_7.3 device-mapper-multipath-libs 0.8.4-28.el8_7.3 dhcp-client 12:4.3.6-48.el8_7.1 dhcp-common 12:4.3.6-48.el8_7.1 dhcp-libs 12:4.3.6-48.el8_7.1 dracut 049-218.git20221019.el8_7 dracut-config-rescue 049-218.git20221019.el8_7 dracut-network 049-218.git20221019.el8_7 dracut-squash 049-218.git20221019.el8_7 emacs-filesystem 1:26.1-7.el8_7.1 epel-release 8-19.el8 expat 2.2.5-10.el8_7.1 fuse-overlayfs 1.9-1.module+el8.7.0+17824+66a0202b gnutls 3.6.16-6.el8_7 grub2-common 1:2.02-142.el8_7.3 grub2-pc 1:2.02-142.el8_7.3 grub2-pc-modules 1:2.02-142.el8_7.3 grub2-tools 1:2.02-142.el8_7.3 grub2-tools-efi 1:2.02-142.el8_7.3 grub2-tools-extra 1:2.02-142.el8_7.3 grub2-tools-minimal 1:2.02-142.el8_7.3 insights-client 3.1.7-9.el8_7 iptables 1.8.4-23.el8_7.1 iptables-ebtables 1.8.4-23.el8_7.1 iptables-libs 1.8.4-23.el8_7.1 iwl100-firmware 39.31.5.1-111.el8_7.1 iwl1000-firmware 1:39.31.5.1-111.el8_7.1 iwl105-firmware 18.168.6.1-111.el8_7.1 iwl135-firmware 18.168.6.1-111.el8_7.1 iwl2000-firmware 18.168.6.1-111.el8_7.1 iwl2030-firmware 18.168.6.1-111.el8_7.1 iwl3160-firmware 1:25.30.13.0-111.el8_7.1 iwl5000-firmware 8.83.5.1_1-111.el8_7.1 iwl5150-firmware 8.24.2.2-111.el8_7.1 iwl6000-firmware 9.221.4.1-111.el8_7.1 iwl6000g2a-firmware 18.168.6.1-111.el8_7.1 iwl6000g2b-firmware 18.168.6.1-111.el8_7.1 iwl6050-firmware 41.28.5.1-111.el8_7.1 iwl7260-firmware 1:25.30.13.0-111.el8_7.1 kernel-tools 4.18.0-425.19.2.el8_7 kernel-tools-libs 4.18.0-425.19.2.el8_7 kmod-kvdo 6.2.7.17-88.el8_7 kpartx 0.8.4-28.el8_7.3 libblkid 2.32.1-39.el8_7 libcurl 7.61.1-25.el8_7.3 libertas-usb8388-firmware 2:20220726-111.git150864a4.el8_7 libfdisk 2.32.1-39.el8_7 libgcc 8.5.0-16.el8_7 libgomp 8.5.0-16.el8_7 libipa_hbac 2.7.3-4.el8_7.3 libksba 1.3.5-9.el8_7 libmount 2.32.1-39.el8_7 libnfsidmap 1:2.3.3-57.el8_7.1 libslirp 4.4.0-1.module+el8.7.0+17824+66a0202b libsmartcols 2.32.1-39.el8_7 libsmbclient 4.16.4-6.el8_7 libsolv 0.7.20-4.el8_7 libsss_autofs 2.7.3-4.el8_7.3 libsss_certmap 2.7.3-4.el8_7.3 libsss_idmap 2.7.3-4.el8_7.3 libsss_nss_idmap 2.7.3-4.el8_7.3 libsss_sudo 2.7.3-4.el8_7.3 libstdc++ 8.5.0-16.el8_7 libtasn1 4.13-4.el8_7 libuuid 2.32.1-39.el8_7 libwbclient 4.16.4-6.el8_7 libxml2 2.9.7-15.el8_7.1 linux-firmware 20220726-111.git150864a4.el8_7 nss 3.79.0-11.el8_7 nss-softokn 3.79.0-11.el8_7 nss-softokn-freebl 3.79.0-11.el8_7 nss-sysinit 3.79.0-11.el8_7 nss-util 3.79.0-11.el8_7 openssh 8.0p1-17.el8_7 openssh-clients 8.0p1-17.el8_7 openssh-server 8.0p1-17.el8_7 openssl 1:1.1.1k-9.el8_7 openssl-libs 1:1.1.1k-9.el8_7 platform-python 3.6.8-48.el8_7.1 platform-python-setuptools 39.2.0-6.el8_7.1 podman 3:4.2.0-8.module+el8.7.0+17824+66a0202b podman-catatonit 3:4.2.0-8.module+el8.7.0+17824+66a0202b python3-libs 3.6.8-48.el8_7.1 python3-libxml2 2.9.7-15.el8_7.1 python3-perf 4.18.0-425.19.2.el8_7 python3-setuptools 39.2.0-6.el8_7.1 python3-setuptools-wheel 39.2.0-6.el8_7.1 python3-sssdconfig 2.7.3-4.el8_7.3 rhc 1:0.2.1-12.el8_7 rsync 3.1.3-19.el8_7.1 runc 1:1.1.4-1.module+el8.7.0+17824+66a0202b samba-client-libs 4.16.4-6.el8_7 samba-common 4.16.4-6.el8_7 samba-common-libs 4.16.4-6.el8_7 selinux-policy 3.14.3-108.el8_7.2 selinux-policy-targeted 3.14.3-108.el8_7.2 slirp4netns 1.2.0-2.module+el8.7.0+17824+66a0202b sos 4.5.1-3.el8 sqlite 3.26.0-17.el8_7 sqlite-libs 3.26.0-17.el8_7 sssd 2.7.3-4.el8_7.3 sssd-ad 2.7.3-4.el8_7.3 sssd-client 2.7.3-4.el8_7.3 sssd-common 2.7.3-4.el8_7.3 sssd-common-pac 2.7.3-4.el8_7.3 sssd-ipa 2.7.3-4.el8_7.3 sssd-kcm 2.7.3-4.el8_7.3 sssd-krb5 2.7.3-4.el8_7.3 sssd-krb5-common 2.7.3-4.el8_7.3 sssd-ldap 2.7.3-4.el8_7.3 sssd-nfs-idmap 2.7.3-4.el8_7.3 sssd-proxy 2.7.3-4.el8_7.3 sudo 1.8.29-8.el8_7.1 systemd 239-68.el8_7.4 systemd-libs 239-68.el8_7.4 systemd-pam 239-68.el8_7.4 systemd-udev 239-68.el8_7.4 tar 2:1.30-6.el8_7.1 tzdata 2023c-1.el8 util-linux 2.32.1-39.el8_7 util-linux-user 2.32.1-39.el8_7 zlib 1.2.11-21.el8_7 Is something odd with the updated Python3 packages? Best, Mevludin

1 1

Os changed to Ubuntu, device class not shown
by Szabo, Istvan (Agoda) 08 May '23

08 May '23

Hi, We have an octopus cluster where we want to move from centos to Ubuntu, after activate all the osd, class is not shown in ceph osd tree. However ceph-volume list shows the crush device class :/ Should I just add it or? ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0

Testing to simulate real-world usage scenarios
by Michal Strnad 07 May '23

07 May '23

Hi everyone, how do you test features that are used by your users before updating/upgrading to a newer version of Ceph? For example, if you have users or customers using RBD images with special sets of permissions/caps, or users/customers using S3 with versioning... None of these should be broken by the new code. Do you use any framework, set of scripts, or anything else? So far, I have only found Teuthology for integration tests and then for S3 [https://github.com/ceph/s3-tests] Thank you Michal

1 0

splitting client and replication traffic
by Justin Ryan 06 May '23

06 May '23

hiya folks! i am excited to be learning about ceph in my homelab, where i have a couple trios of machines service, currently, a cephfs, but i’m looking fw to using ceph for block devices for vms, etc.. my phyiscal machines have multiple network interfaces, one trio of them are ARM machines with up to 4x SFP+ 10GBe interfaces. I have a couple of 10GBe switches - one which has exactly three ports I can connect these nodes to for general distribution to the rest of the network, and another… I would like to configure my Ceph cluster to replicate over a different network interface than it serves clients over.. Is this a common configuration, and/or can anyone provide me some guidance?! Thanks in advance! Best! J -- Justin Alan Ryan

2 1

Unable to restart mds - mds crashes almost immediately after finishing recovery
by Emmanuel Jaep 05 May '23

05 May '23

Hi, I just inherited a ceph storage. Therefore, my level of confidence with the tool is certainly less than ideal. We currently have an mds server that refuses to come back online. While reviewing the logs, I can see that, upon mds start, the recovery goes well: ``` -10> 2023-05-03T08:12:43.632+0200 7f345d00b700 1 mds.4.2638711 cluster recovered. 12: (MDCache::_open_ino_traverse_dir(inodeno_t, MDCache::open_ino_info_t&, int)+0xbf) [0x558323d602df] ``` However, rights after this message, ceph handles a couple of clients request: ``` -9> 2023-05-03T08:12:43.632+0200 7f345d00b700 4 mds.4.2638711 set_osd_epoch_barrier: epoch=249241 -8> 2023-05-03T08:12:43.632+0200 7f3459003700 2 mds.4.cache Memory usage: total 2739784, rss 2321188, heap 348412, baseline 315644, 0 / 765023 inodes have caps, 0 caps, 0 caps per inode -7> 2023-05-03T08:12:43.688+0200 7f3458802700 4 mds.4.server handle_client_request client_request(client.108396030:57271 lookup #0x70001516236/012385530.npy 2023-05-02T20:37:19.675666+0200 RETRY=6 caller_uid=135551, caller_gid=11157{0,4,27,11157,}) v5 -6> 2023-05-03T08:12:43.688+0200 7f3458802700 4 mds.4.server handle_client_request client_request(client.104073212:5109945 readdir #0x70001516236 2023-05-02T20:36:29.517066+0200 RETRY=6 caller_uid=180090, caller_gid=11157{0,4,27,11157,}) v5 -5> 2023-05-03T08:12:43.688+0200 7f3458802700 4 mds.4.server handle_client_request client_request(client.104288735:3008344 readdir #0x70001516236 2023-05-02T20:36:29.520801+0200 RETRY=6 caller_uid=135551, caller_gid=11157{0,4,27,11157,}) v5 -4> 2023-05-03T08:12:43.688+0200 7f3458802700 4 mds.4.server handle_client_request client_request(client.8558540:46306346 readdir #0x700019ba15e 2023-05-01T21:26:34.303697+0200 RETRY=49 caller_uid=0, caller_gid=0{}) v2 -3> 2023-05-03T08:12:43.688+0200 7f3458802700 4 mds.4.server handle_client_request client_request(client.96913903:2156912 create #0x1000b37db9a/street-photo-3.png 2023-05-01T17:27:37.454042+0200 RETRY=59 caller_uid=271932, caller_gid=30034{}) v2 -2> 2023-05-03T08:12:43.688+0200 7f345d00b700 5 mds.icadmin006 handle_mds_map old map epoch 2638715 <= 2638715, discarding ``` and crashes: ``` -1> 2023-05-03T08:12:43.692+0200 7f345d00b700 -1 /build/ceph-16.2.10/src/mds/Server.cc: In function 'void Server::handle_client_open(MDRequestRef&)' thread 7f345d00b700 time 2023-05-03T08:12:43.694660+0200 /build/ceph-16.2.10/src/mds/Server.cc: 4240: FAILED ceph_assert(cur->is_auth()) ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f3462533d65] 2: /usr/lib/ceph/libceph-common.so.2(+0x265f6d) [0x7f3462533f6d] 3: (Server::handle_client_open(boost::intrusive_ptr<MDRequestImpl>&)+0x1834) [0x558323c89f04] 4: (Server::handle_client_openc(boost::intrusive_ptr<MDRequestImpl>&)+0x28f) [0x558323c925ef] 5: (Server::dispatch_client_request(boost::intrusive_ptr<MDRequestImpl>&)+0xa45) [0x558323cc3575] 6: (MDCache::dispatch_request(boost::intrusive_ptr<MDRequestImpl>&)+0x3d) [0x558323d7460d] 7: (MDSContext::complete(int)+0x61) [0x558323f68681] 8: (MDCache::_open_remote_dentry_finish(CDentry*, inodeno_t, MDSContext*, bool, int)+0x3e) [0x558323d3edce] 9: (C_MDC_OpenRemoteDentry::finish(int)+0x3e) [0x558323de6cce] 10: (MDSContext::complete(int)+0x61) [0x558323f68681] 11: (MDCache::open_ino_finish(inodeno_t, MDCache::open_ino_info_t&, int)+0xcf) [0x558323d5ff2f] 12: (MDCache::_open_ino_traverse_dir(inodeno_t, MDCache::open_ino_info_t&, int)+0xbf) [0x558323d602df] 13: (MDSContext::complete(int)+0x61) [0x558323f68681] 14: (MDSRank::_advance_queues()+0x88) [0x558323c23c38] 15: (MDSRank::_dispatch(boost::intrusive_ptr<Message const> const&, bool)+0x1fa) [0x558323c24a1a] 16: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr<Message const> const&)+0x5e) [0x558323c254fe] 17: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x1d6) [0x558323bfd906] 18: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0x460) [0x7f34627854e0] 19: (DispatchQueue::entry()+0x58f) [0x7f3462782d7f] 20: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f346284eee1] 21: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f3462278609] 22: clone() 0> 2023-05-03T08:12:43.700+0200 7f345d00b700 -1 *** Caught signal (Aborted) ** in thread 7f345d00b700 thread_name:ms_dispatch ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x143c0) [0x7f34622843c0] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1ad) [0x7f3462533dc0] 5: /usr/lib/ceph/libceph-common.so.2(+0x265f6d) [0x7f3462533f6d] 6: (Server::handle_client_open(boost::intrusive_ptr<MDRequestImpl>&)+0x1834) [0x558323c89f04] 7: (Server::handle_client_openc(boost::intrusive_ptr<MDRequestImpl>&)+0x28f) [0x558323c925ef] 8: (Server::dispatch_client_request(boost::intrusive_ptr<MDRequestImpl>&)+0xa45) [0x558323cc3575] 9: (MDCache::dispatch_request(boost::intrusive_ptr<MDRequestImpl>&)+0x3d) [0x558323d7460d] 10: (MDSContext::complete(int)+0x61) [0x558323f68681] 11: (MDCache::_open_remote_dentry_finish(CDentry*, inodeno_t, MDSContext*, bool, int)+0x3e) [0x558323d3edce] 12: (C_MDC_OpenRemoteDentry::finish(int)+0x3e) [0x558323de6cce] 13: (MDSContext::complete(int)+0x61) [0x558323f68681] 14: (MDCache::open_ino_finish(inodeno_t, MDCache::open_ino_info_t&, int)+0xcf) [0x558323d5ff2f] 15: (MDCache::_open_ino_traverse_dir(inodeno_t, MDCache::open_ino_info_t&, int)+0xbf) [0x558323d602df] 16: (MDSContext::complete(int)+0x61) [0x558323f68681] 17: (MDSRank::_advance_queues()+0x88) [0x558323c23c38] 18: (MDSRank::_dispatch(boost::intrusive_ptr<Message const> const&, bool)+0x1fa) [0x558323c24a1a] 19: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr<Message const> const&)+0x5e) [0x558323c254fe] 20: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x1d6) [0x558323bfd906] 21: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0x460) [0x7f34627854e0] 22: (DispatchQueue::entry()+0x58f) [0x7f3462782d7f] 23: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f346284eee1] 24: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f3462278609] 25: clone() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 rbd_pwl 0/ 5 journaler 0/ 5 objectcacher 0/ 5 immutable_obj_cache 0/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 0 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 1 reserver 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 rgw_sync 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 fuse 2/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace 1/ 5 prioritycache 0/ 5 test 0/ 5 cephfs_mirror 0/ 5 cephsqlite -2/-2 (syslog threshold) -1/-1 (stderr threshold) --- pthread ID / name mapping for recent threads --- 139862749464320 / 139862757857024 / md_submit 139862766249728 / 139862774642432 / MR_Finisher 139862791427840 / PQ_Finisher 139862799820544 / mds_rank_progr 139862808213248 / ms_dispatch 139862841784064 / ceph-mds 139862858569472 / safe_timer 139862875354880 / ms_dispatch 139862892140288 / io_context_pool 139862908925696 / admin_socket 139862917318400 / msgr-worker-2 139862925711104 / msgr-worker-1 139862934103808 / msgr-worker-0 139862951257984 / ceph-mds max_recent 10000 max_new 10000 log_file /var/log/ceph/floki-mds.icadmin006.log --- end dump of recent events --- ``` How could I troubleshoot that further? Thanks in advance for your help, Emmanuel

3 3

osd pause
by Thomas Bennett 05 May '23

05 May '23

Hi, FYI - This might be pedantic, but there does not seem to be any difference between using these two sets of commands: - ceph osd pause / ceph osd unpause - ceph osd set pause / ceph osd unset pause I can see that they both set/unset the pauserd,pausewr flags, but since they don't report anything else, I assume they do exactly the same thing. I also assumed it only stopped reads/writes to the OSDs, but I found this openattic post <https://openattic.org/posts/how-to-do-a-ceph-cluster-maintenanceshutdown/> which had this comment: Pausing the cluster means that you can't see when OSDs come back up again and no map update will happen. I didn't know that but it seems pretty useful knowing. Pausing is mentioned when posts talking about shutting down Ceph cluster for maintenance but often it's added as optional. Does anyone know what is the original intended purpose of pausing and when/why would you use it? Also - can I assume that pause will complete any current write/read ops on OSDs before pausing? Thanks, Tom

1 0

rbd map: corrupt full osdmap (-22) when
by Kamil Madac 05 May '23

05 May '23

Hi, We deployed pacific cluster 16.2.12 with cephadm. We experience following error during rbd map: [Wed May 3 08:59:11 2023] libceph: mon2 (1)[2a00:da8:ffef:1433::]:6789 session established [Wed May 3 08:59:11 2023] libceph: another match of type 1 in addrvec [Wed May 3 08:59:11 2023] libceph: corrupt full osdmap (-22) epoch 200 off 1042 (000000009876284d of 000000000cb24b58-0000000080b70596) [Wed May 3 08:59:11 2023] osdmap: 00000000: 08 07 7d 10 00 00 09 01 5d 09 00 00 a2 22 3b 86 ..}.....]....";. [Wed May 3 08:59:11 2023] osdmap: 00000010: e4 f5 11 ed 99 ee 47 75 ca 3c ad 23 c8 00 00 00 ......Gu.<.#.... [Wed May 3 08:59:11 2023] osdmap: 00000020: 21 68 4a 64 98 d2 5d 2e 84 fd 50 64 d9 3a 48 26 !hJd..]...Pd.:H& [Wed May 3 08:59:11 2023] osdmap: 00000030: 02 00 00 00 01 00 00 00 00 00 00 00 1d 05 71 01 ..............q. .... Linux Kernel is 6.1.13 and the important thing is that we are using ipv6 addresses for connection to ceph nodes. We were able to map rbd from client with kernel 5.10, but in prod environment we are not allowed to use that kernel. What could be the reason for such behavior on newer kernels and how to troubleshoot it? Here is output of ceph osd dump: # ceph osd dump epoch 200 fsid a2223b86-e4f5-11ed-99ee-4775ca3cad23 created 2023-04-27T12:18:41.777900+0000 modified 2023-05-02T12:09:40.642267+0000 flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit crush_version 34 full_ratio 0.95 backfillfull_ratio 0.9 nearfull_ratio 0.85 require_min_compat_client luminous min_compat_client jewel require_osd_release pacific stretch_mode_enabled false pool 1 'device_health_metrics' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 183 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr_devicehealth pool 2 'idp' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 48 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd max_osd 3 osd.0 up in weight 1 up_from 176 up_thru 182 down_at 172 last_clean_interval [170,171) [v2:[2a00:da8:ffef:1431::]:6800/805023868,v1:[2a00:da8:ffef:1431::]:6801/805023868,v2: 0.0.0.0:6802/805023868,v1:0.0.0.0:6803/805023868] [v2:[2a00:da8:ffef:1431::]:6804/805023868,v1:[2a00:da8:ffef:1431::]:6805/805023868,v2: 0.0.0.0:6806/805023868,v1:0.0.0.0:6807/805023868] exists,up e8fd0ee2-ea63-4d02-8f36-219d36869078 osd.1 up in weight 1 up_from 136 up_thru 182 down_at 0 last_clean_interval [0,0) [v2:[2a00:da8:ffef:1432::]:6800/2172723816,v1:[2a00:da8:ffef:1432::]:6801/2172723816,v2: 0.0.0.0:6802/2172723816,v1:0.0.0.0:6803/2172723816] [v2:[2a00:da8:ffef:1432::]:6804/2172723816,v1:[2a00:da8:ffef:1432::]:6805/2172723816,v2: 0.0.0.0:6806/2172723816,v1:0.0.0.0:6807/2172723816] exists,up 0b7b5628-9273-4757-85fb-9c16e8441895 osd.2 up in weight 1 up_from 182 up_thru 182 down_at 178 last_clean_interval [123,177) [v2:[2a00:da8:ffef:1433::]:6800/887631330,v1:[2a00:da8:ffef:1433::]:6801/887631330,v2: 0.0.0.0:6802/887631330,v1:0.0.0.0:6803/887631330] [v2:[2a00:da8:ffef:1433::]:6804/887631330,v1:[2a00:da8:ffef:1433::]:6805/887631330,v2: 0.0.0.0:6806/887631330,v1:0.0.0.0:6807/887631330] exists,up 21f8d0d5-6a3f-4f78-96c8-8ec4e4f78a01 Thank you. -- Kamil Madac

3 5

Re: 17.2.6 fs 'ls' ok, but 'cat' 'operation not permitted' puzzle
by Harry G Coin 05 May '23

05 May '23

This problem of inaccessible file systems post upgrade by other than client.admin date back from v14 carries on through v17. It also applies to any case of specifying other than the default pool names for new file systems. Solved because Curt remembered link on this list. (Thanks Curt!) Here's what the official ceph docs ought have provided, for others who hit this. YMMV: IF you have ceph file systems which have data and meta data pools that were specified in the 'ceph fs new' command (meaning not left to the defaults which create them for you), OR you have an existing ceph file system and are upgrading to a new major version of ceph THEN for the documented 'ceph fs authorize...' commands to do as documented (and avoid strange 'operation not permitted' errors when doing file I/O or similar security related problems for all but such as the client.admin user), you must first run: ceph osd pool application set <your metadata pool name> cephfs metadata <your ceph fs filesystem name> and ceph osd pool application set <your data pool name> cephfs data <your ceph fs filesystem name> Otherwise when the OSD's get a request to read or write data (not the directory info, but file data) they won't know which ceph file system name to look up, nevermind the names you may have chosen for the pools, as the 'defaults' themselves changed in the major releases, from data pool=fsname metadata pool=fsname_metadata to data pool=fsname.data and metadata pool=fsname.meta as the ceph revisions came and went. Any setup that just used 'client.admin' for all mounts didn't see the problem as the admin key gave blanket permission. A temporary 'fix' is to change mount requests to the 'client.admin' and associated key. A less drastic but still half-fix is to change the osd cap for your user to just 'caps osd = "allow rw" and delete "tag cephfs data=...." The only documentation I could find for this upgrade security-related ceph-ending catastrophe was in the NFS, not cephfs docs: https://docs.ceph.com/en/latest/cephfs/nfs/ and the Genius level much appreciated pointer from Curt here: On 5/2/23 14:21, Curt wrote: > This thread might be of use, it's an older version of ceph 14, but > might still apply, > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/23FDDSYBCDV… > ? > > On Tue, May 2, 2023 at 11:06 PM Harry G Coin <hgcoin(a)gmail.com> wrote: > > In 17.2.6 is there a security requirement that pool names > supporting a > ceph fs filesystem match the filesystem name.data for the data and > name.meta for the associated metadata pool? (multiple file systems > are > enabled) > > I have filesystems from older versions with the data pool name > matching > the filesystem and appending _metadata for that, > > and even older filesystems with the pool name as in 'library' and > 'library_metadata' supporting a filesystem called 'libraryfs' > > The pools all have the cephfs tag. > > But using the documented: > > ceph fs authorize libraryfs client.basicuser / rw > > command allows the root user to mount and browse the library > directory > tree, but fails with 'operation not permitted' when even reading > any file. > > However, changing the client.basicuser osd auth to 'allow rw' > instead of > 'allow rw tag...' allows normal operations. > > So: > > [client.basicuser] > key = <key stuff>== > caps mds = "allow rw fsname=libraryfs" > caps mon = "allow r fsname=libraryfs" > caps osd = "allow rw" > > works, but the same with > > caps osd = "allow rw tag cephfs data=libraryfs" > > leads to the 'operation not permitted' on read, or write or any > actual > access. > > It remains a puzzle. Help appreciated! > > Were there upgrade instructions about that, any help pointing me > to them? > > Thanks > > Harry Coin > Rock Stable Systems > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

3 2

CephFS Scrub Questions
by Chris Palmer 04 May '23

04 May '23

Hi Grateful if someone could clarify some things about CephFS Scrubs: 1) Am I right that a command such as "ceph tell mds.cephfs:0 scrub start / recursive" only triggers a forward scrub (not a backward scrub)? 2) I couldn't find any reference to forward scrubs being done automatically and was wondering whether I should do them using cron? But then I saw an undated (but I think a little elderly) presentation by Greg Farnum that states that "forward scrub...runs continuously in the background". Is that still correct (for Quincy), and if so what controls the frequency? 3) Are backward scrubs always manual, using the 3 cephfs-data-scan phases? 4) Are regular backward scrubs recommended, or only if there is indication of a problem? (With due regard to the amount of time they may take...) Thanks for any advice. Regards, Chris

2 1

Radosgw: ssl_private_key could not find the file even if it existed
by viplanghe6＠gmail.com 04 May '23

04 May '23

The radosgw has been configured like this: [client.rgw.ceph1] host = ceph1 rgw_frontends = beast port=8080 ssl_port=443 ssl_certificate=/root/ssl/ca.crt ssl_private_key=/root/ssl/ca.key #rgw_frontends = beast port=8080 ssl_port=443 ssl_certificate=/root/ssl/ca.crt ssl_private_key=config://rgw/cert/default/ca.key admin_socket = /var/run/ceph/ceph-client.rgw.ceph1 but I'm getting this error: failed to add ssl_private_key=/root/ssl/ca.key: No such file or directory I also tried to import the key into ceph db and provided the path with config://, but it doesn't work too. Anyone have any ideal? Thanks

2 1

Frequent calling monitor election
by Frank Schilder 04 May '23

04 May '23

Hi all, our monitors have enjoyed democracy since the beginning. However, I don't share a sudden excitement about voting: 2/9/23 4:42:30 PM[INF]overall HEALTH_OK 2/9/23 4:42:30 PM[INF]mon.ceph-01 is new leader, mons ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 in quorum (ranks 0,1,2,3,4) 2/9/23 4:42:26 PM[INF]mon.ceph-01 calling monitor election 2/9/23 4:42:26 PM[INF]mon.ceph-26 calling monitor election 2/9/23 4:42:26 PM[INF]mon.ceph-25 calling monitor election 2/9/23 4:42:26 PM[INF]mon.ceph-02 calling monitor election 2/9/23 4:40:00 PM[INF]overall HEALTH_OK 2/9/23 4:30:00 PM[INF]overall HEALTH_OK 2/9/23 4:24:34 PM[INF]overall HEALTH_OK 2/9/23 4:24:34 PM[INF]mon.ceph-01 is new leader, mons ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 in quorum (ranks 0,1,2,3,4) 2/9/23 4:24:29 PM[INF]mon.ceph-01 calling monitor election 2/9/23 4:24:29 PM[INF]mon.ceph-02 calling monitor election 2/9/23 4:24:29 PM[INF]mon.ceph-03 calling monitor election 2/9/23 4:24:29 PM[INF]mon.ceph-01 calling monitor election 2/9/23 4:24:29 PM[INF]mon.ceph-26 calling monitor election 2/9/23 4:24:29 PM[INF]mon.ceph-25 calling monitor election 2/9/23 4:24:29 PM[INF]mon.ceph-02 calling monitor election 2/9/23 4:24:04 PM[INF]overall HEALTH_OK 2/9/23 4:24:03 PM[INF]mon.ceph-01 is new leader, mons ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 in quorum (ranks 0,1,2,3,4) 2/9/23 4:23:59 PM[INF]mon.ceph-01 calling monitor election 2/9/23 4:23:59 PM[INF]mon.ceph-02 calling monitor election 2/9/23 4:20:00 PM[INF]overall HEALTH_OK 2/9/23 4:10:00 PM[INF]overall HEALTH_OK 2/9/23 4:00:00 PM[INF]overall HEALTH_OK 2/9/23 3:50:00 PM[INF]overall HEALTH_OK 2/9/23 3:43:13 PM[INF]overall HEALTH_OK 2/9/23 3:43:13 PM[INF]mon.ceph-01 is new leader, mons ceph-01,ceph-02,ceph-03,ceph-25,ceph-26 in quorum (ranks 0,1,2,3,4) 2/9/23 3:43:08 PM[INF]mon.ceph-01 calling monitor election 2/9/23 3:43:08 PM[INF]mon.ceph-26 calling monitor election 2/9/23 3:43:08 PM[INF]mon.ceph-25 calling monitor election We moved a switch from one rack to another and after the switch came beck up, the monitors frequently bitch about who is the alpha. How do I get them to focus more on their daily duties again? Thanks for any help! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

4 9

pg upmap primary
by Nguetchouang Ngongang Kevin 04 May '23

04 May '23

Hello, I have a question, when happened when i delete a pg on which i set a particular osd as primary using the pg-upmap-primary command ? -- Nguetchouang Ngongang Kevin ENS de Lyon https://perso.ens-lyon.fr/kevin.nguetchouang/

2 1

回复：ceph-users Digest, Vol 107, Issue 20
by 余晟佐 04 May '23

04 May '23

help ------------------ 原始邮件 ------------------ 发件人: "ceph-users" <ceph-users-request(a)ceph.io>; 发送时间: 2023年5月4日(星期四) 下午4:40 收件人: "ceph-users"<ceph-users(a)ceph.io>; 主题: ceph-users Digest, Vol 107, Issue 20 Send ceph-users mailing list submissions to ceph-users(a)ceph.io To subscribe or unsubscribe via email, send a message with subject or body 'help' to ceph-users-request(a)ceph.io You can reach the person managing the list at ceph-users-owner(a)ceph.io When replying, please edit your Subject line so it is more specific than "Re: Contents of ceph-users digest..." Today's Topics:    1. Initialization timeout, failed to initialize (Vitaly Goot)    2. Re: MDS crash on FAILED ceph_assert(cur->is_auth())       (Peter van Heusden)    3. Re: MDS "newly corrupt dentry" after patch version upgrade       (Janek Bevendorff)    4. Best practice for expanding Ceph cluster (huxiaoyu(a)horebdata.cn)    5. Re: 16.2.13 pacific QE validation status (Guillaume Abrioux) ---------------------------------------------------------------------- Date: Thu, 04 May 2023 01:50:12 -0000 From: "Vitaly Goot" <vitaly.goot(a)gmail.com> Subject: [ceph-users] Initialization timeout, failed to initialize To: ceph-users(a)ceph.io Message-ID: <168316501216.1713.9594013921879975501@mailman-web> Content-Type: text/plain; charset="utf-8" playing with MULTI-SITE zones for CEPH Object Gateway ceph version: 17.2.5 my setup: 3 zone multi-site; 3-way full sync mode; each zone has 3 machines -> RGW+MON+OSD running load test:  3000 concurrent uploads of 1M object after about 3-4 minutes of load RGW machine get stuck, on 2 zone out of 3 RGW is not responding (e.g. curl $RGW:80) attempt to restart RGW ends up with `Initialization timeout, failed to initialize` here is a backtrace from gdb with a backtrace where it hangs after restart: (gdb) inf thr   Id   Target Id                                           Frame * 1    Thread 0x7fa7d3abbcc0 (LWP 30791) "radosgw"         futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7ffc7f7a2438) at ../sysdeps/nptl/futex-internal.h:183 ... (gdb) bt #0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7ffc7f7a2438) at ../sysdeps/nptl/futex-internal.h:183 #1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7ffc7f7a2488, cond=0x7ffc7f7a2410) at pthread_cond_wait.c:508 #2  __pthread_cond_wait (cond=cond@entry=0x7ffc7f7a2410, mutex=0x7ffc7f7a2488) at pthread_cond_wait.c:647 #3  0x00007fa7d7097e42 in ceph::condition_variable_debug::wait (this=this@entry=0x7ffc7f7a2410, lock=...) at ../src/common/mutex_debug.h:148 #4  0x00007fa7d7953cba in ceph::condition_variable_debug::wait<librados::IoCtxImpl::operate(const object_t&, ObjectOperation*, ceph::real_time*, int)::<lambda()> > (pred=..., lock=..., this=0x7ffc7f7a2410) at ../src/librados/IoCtxImpl.cc:672 #5  librados::IoCtxImpl::operate (this=this@entry=0x558347c21010, oid=..., o=0x558347e12310, pmtime=<optimized out>, flags=<optimized out>) at ../src/librados/IoCtxImpl.cc:672 #6  0x00007fa7d792bd55 in librados::v14_2_0::IoCtx::operate (this=this@entry=0x558347e44760, oid="notify.0", o=o@entry=0x7ffc7f7a2690, flags=flags@entry=0) at ../src/librados/librados_cxx.cc:1536 #7  0x00007fa7d9490ad1 in rgw_rados_operate (dpp=<optimized out>, ioctx=..., oid="notify.0", op=op@entry=0x7ffc7f7a2690, y=..., flags=0) at ../src/rgw/rgw_tools.cc:277 #8  0x00007fa7d9627e0f in RGWSI_RADOS::Obj::operate (this=this@entry=0x558347e44710, dpp=<optimized out>, op=op@entry=0x7ffc7f7a2690, y=..., flags=flags@entry=0) at ../src/rgw/services/svc_rados.h:112 #9  0x00007fa7d96209a5 in RGWSI_Notify::init_watch (this=this@entry=0x558347c49530, dpp=<optimized out>, y=...) at ../src/rgw/services/svc_notify.cc:214 #10 0x00007fa7d962161b in RGWSI_Notify::do_start (this=0x558347c49530, y=..., dpp=<optimized out>) at ../src/rgw/services/svc_notify.cc:277 #11 0x00007fa7d8f17bcf in RGWServiceInstance::start (this=0x558347c49530, y=..., dpp=<optimized out>) at ../src/rgw/rgw_service.cc:331 #12 0x00007fa7d8f1a260 in RGWServices_Def::init (this=this@entry=0x558347de90a0, cct=<optimized out>, have_cache=<optimized out>, raw=raw@entry=false, run_sync=<optimized out>, y=..., dpp=<optimized out>) at /usr/include/c++/9/bits/unique_ptr.h:360 #13 0x00007fa7d8f1cc40 in RGWServices::do_init (this=this@entry=0x558347de90a0, _cct=<optimized out>, have_cache=<optimized out>, raw=raw@entry=false, run_sync=<optimized out>, y=..., dpp=<optimized out>) at ../src/rgw/rgw_service.cc:284 #14 0x00007fa7d92a7b1f in RGWServices::init (dpp=<optimized out>, y=..., run_sync=<optimized out>, have_cache=<optimized out>, cct=<optimized out>, this=0x558347de90a0) at ../src/rgw/rgw_service.h:153 #15 RGWRados::init_svc (this=this@entry=0x558347de8dc0, raw=raw@entry=false, dpp=<optimized out>) at ../src/rgw/rgw_rados.cc:1380 #16 0x00007fa7d930f241 in RGWRados::initialize (this=0x558347de8dc0, dpp=<optimized out>) at ../src/rgw/rgw_rados.cc:1400 #17 0x00007fa7d944f85f in RGWRados::initialize (dpp=<optimized out>, _cct=0x558347c6a320, this=<optimized out>) at ../src/rgw/rgw_rados.h:586 #18 StoreManager::init_storage_provider (dpp=<optimized out>, dpp@entry=0x7ffc7f7a2e90, cct=cct@entry=0x558347c6a320, svc="rados", use_gc_thread=use_gc_thread@entry=true, use_lc_thread=use_lc_thread@entry=true, quota_threads=quota_threads@entry=true, run_sync_thread=true, run_reshard_thread=true, use_cache=true,     use_gc=true) at ../src/rgw/rgw_sal.cc:55 #19 0x00007fa7d8e7367a in StoreManager::get_storage (use_gc=true, use_cache=true, run_reshard_thread=true, run_sync_thread=true, quota_threads=true, use_lc_thread=true, use_gc_thread=true, svc="rados", cct=0x558347c6a320, dpp=0x7ffc7f7a2e90) at /usr/include/c++/9/bits/basic_string.h:267 #20 radosgw_Main (argc=<optimized out>, argv=<optimized out>) at ../src/rgw/rgw_main.cc:372 #21 0x0000558347883f56 in main (argc=<optimized out>, argv=<optimized out>) at ../src/rgw/radosgw.cc:12 (gdb) #0  futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7ffc7f7a2438) at ../sysdeps/nptl/futex-internal.h:183 #1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7ffc7f7a2488, cond=0x7ffc7f7a2410) at pthread_cond_wait.c:508 #2  __pthread_cond_wait (cond=cond@entry=0x7ffc7f7a2410, mutex=0x7ffc7f7a2488) at pthread_cond_wait.c:647 #3  0x00007fa7d7097e42 in ceph::condition_variable_debug::wait (this=this@entry=0x7ffc7f7a2410, lock=...) at ../src/common/mutex_debug.h:148 #4  0x00007fa7d7953cba in ceph::condition_variable_debug::wait<librados::IoCtxImpl::operate(const object_t&, ObjectOperation*, ceph::real_time*, int)::<lambda()> > (pred=..., lock=..., this=0x7ffc7f7a2410) at ../src/librados/IoCtxImpl.cc:672 #5  librados::IoCtxImpl::operate (this=this@entry=0x558347c21010, oid=..., o=0x558347e12310, pmtime=<optimized out>, flags=<optimized out>) at ../src/librados/IoCtxImpl.cc:672 #6  0x00007fa7d792bd55 in librados::v14_2_0::IoCtx::operate (this=this@entry=0x558347e44760, oid="notify.0", o=o@entry=0x7ffc7f7a2690, flags=flags@entry=0) at ../src/librados/librados_cxx.cc:1536 #7  0x00007fa7d9490ad1 in rgw_rados_operate (dpp=<optimized out>, ioctx=..., oid="notify.0", op=op@entry=0x7ffc7f7a2690, y=..., flags=0) at ../src/rgw/rgw_tools.cc:277 #8  0x00007fa7d9627e0f in RGWSI_RADOS::Obj::operate (this=this@entry=0x558347e44710, dpp=<optimized out>, op=op@entry=0x7ffc7f7a2690, y=..., flags=flags@entry=0) at ../src/rgw/services/svc_rados.h:112 #9  0x00007fa7d96209a5 in RGWSI_Notify::init_watch (this=this@entry=0x558347c49530, dpp=<optimized out>, y=...) at ../src/rgw/services/svc_notify.cc:214 #10 0x00007fa7d962161b in RGWSI_Notify::do_start (this=0x558347c49530, y=..., dpp=<optimized out>) at ../src/rgw/services/svc_notify.cc:277 #11 0x00007fa7d8f17bcf in RGWServiceInstance::start (this=0x558347c49530, y=..., dpp=<optimized out>) at ../src/rgw/rgw_service.cc:331 #12 0x00007fa7d8f1a260 in RGWServices_Def::init (this=this@entry=0x558347de90a0, cct=<optimized out>, have_cache=<optimized out>, raw=raw@entry=false, run_sync=<optimized out>, y=..., dpp=<optimized out>) at /usr/include/c++/9/bits/unique_ptr.h:360 #13 0x00007fa7d8f1cc40 in RGWServices::do_init (this=this@entry=0x558347de90a0, _cct=<optimized out>, have_cache=<optimized out>, raw=raw@entry=false, run_sync=<optimized out>, y=..., dpp=<optimized out>) at ../src/rgw/rgw_service.cc:284 #14 0x00007fa7d92a7b1f in RGWServices::init (dpp=<optimized out>, y=..., run_sync=<optimized out>, have_cache=<optimized out>, cct=<optimized out>, this=0x558347de90a0) at ../src/rgw/rgw_service.h:153 #15 RGWRados::init_svc (this=this@entry=0x558347de8dc0, raw=raw@entry=false, dpp=<optimized out>) at ../src/rgw/rgw_rados.cc:1380 #16 0x00007fa7d930f241 in RGWRados::initialize (this=0x558347de8dc0, dpp=<optimized out>) at ../src/rgw/rgw_rados.cc:1400 #17 0x00007fa7d944f85f in RGWRados::initialize (dpp=<optimized out>, _cct=0x558347c6a320, this=<optimized out>) at ../src/rgw/rgw_rados.h:586 #18 StoreManager::init_storage_provider (dpp=<optimized out>, dpp@entry=0x7ffc7f7a2e90, cct=cct@entry=0x558347c6a320, svc="rados", use_gc_thread=use_gc_thread@entry=true, use_lc_thread=use_lc_thread@entry=true, quota_threads=quota_threads@entry=true, run_sync_thread=true, run_reshard_thread=true, use_cache=true,     use_gc=true) at ../src/rgw/rgw_sal.cc:55 #19 0x00007fa7d8e7367a in StoreManager::get_storage (use_gc=true, use_cache=true, run_reshard_thread=true, run_sync_thread=true, quota_threads=true, use_lc_thread=true, use_gc_thread=true, svc="rados", cct=0x558347c6a320, dpp=0x7ffc7f7a2e90) at /usr/include/c++/9/bits/basic_string.h:267 #20 radosgw_Main (argc=<optimized out>, argv=<optimized out>) at ../src/rgw/rgw_main.cc:372 #21 0x0000558347883f56 in main (argc=<optimized out>, argv=<optimized out>) at ../src/rgw/radosgw.cc:12 Any suggestion on what can be a problem and how to reset RGW so it will be able to start normally? ------------------------------ Date: Thu, 4 May 2023 09:13:56 +0200 From: Peter van Heusden <pvh(a)sanbi.ac.za> Subject: [ceph-users] Re: MDS crash on FAILED ceph_assert(cur->is_auth()) Cc: ceph-users(a)ceph.io Message-ID: <CAK1reXhEDjfKuLmuyus0RT09mwecRmP=LGLcoSKWeZ+pu+YXJQ(a)mail.gmail.com> Content-Type: text/plain; charset="UTF-8" Hi Emmaneul It was a while ago, but as I recall I evicted all clients and that allowed me to restart the MDS servers. There was something clearly "broken" in how at least one of the clients was interacting with the system. Peter On Thu, 4 May 2023 at 07:18, Emmanuel Jaep <emmanuel.jaep(a)gmail.com> wrote: > Hi, > > did you finally figure out what happened? > I do have the same behavior and we can't get the mds to start again... > > Thanks, > > Emmanuel > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > ------------------------------ Date: Thu, 4 May 2023 09:15:38 +0200 From: Janek Bevendorff <janek.bevendorff(a)uni-weimar.de> Subject: [ceph-users] Re: MDS "newly corrupt dentry" after patch version upgrade To: Patrick Donnelly <pdonnell(a)redhat.com> Cc: ceph-users <ceph-users(a)ceph.io> Message-ID: <591f410c-aabc-72af-36d0-478ce8d09028(a)uni-weimar.de> Content-Type: text/plain; charset=UTF-8; format=flowed After running the tool for 11 hours straight, it exited with the following exception: Traceback (most recent call last):    File "/home/webis/first-damage.py", line 156, in <module>      traverse(f, ioctx)    File "/home/webis/first-damage.py", line 84, in traverse      for (dnk, val) in it:    File "rados.pyx", line 1389, in rados.OmapIterator.__next__    File "rados.pyx", line 318, in rados.decode_cstr UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 8: invalid start byte Does that mean that the last inode listed in the output file is corrupt? Any way I can fix it? The output file has 14 million lines. We have about 24.5 million objects in the metadata pool. Janek On 03/05/2023 14:20, Patrick Donnelly wrote: > On Wed, May 3, 2023 at 4:33 AM Janek Bevendorff > <janek.bevendorff(a)uni-weimar.de> wrote: >> Hi Patrick, >> >>> I'll try that tomorrow and let you know, thanks! >> I was unable to reproduce the crash today. Even with >> mds_abort_on_newly_corrupt_dentry set to true, all MDS booted up >> correctly (though they took forever to rejoin with logs set to 20). >> >> To me it looks like the issue has resolved itself overnight. I had run a >> recursive scrub on the file system and another snapshot was taken, in >> case any of those might have had an effect on this. It could also be the >> case that the (supposedly) corrupt journal entry has simply been >> committed now and hence doesn't trigger the assertion any more. Is there >> any way I can verify this? > You can run: > > https://github.com/ceph/ceph/blob/main/src/tools/cephfs/first-damage.py > > Just do: > > python3 first-damage.py --memo run.1 <meta pool> > > No need to do any of the other steps if you just want a read-only check. > -- Bauhaus-Universität Weimar Bauhausstr. 9a, R308 99423 Weimar, Germany Phone: +49 3643 58 3577 www.webis.de ------------------------------ Date: Thu, 4 May 2023 10:38:30 +0200 From: "huxiaoyu(a)horebdata.cn" <huxiaoyu(a)horebdata.cn> Subject: [ceph-users] Best practice for expanding Ceph cluster To: ceph-users <ceph-users(a)ceph.io> Message-ID: <985B22F000D9DE88+2023050410382589829410(a)horebdata.cn> Content-Type: text/plain;       charset="us-ascii" Dear Ceph folks, I am writing to ask for advice on best practice of expanding ceph cluster. We are running an 8-node Ceph cluster and RGW, and would like to add another 10 node, each of which have 10x 12TB HDD. The current 8-node has ca. 400TB user data. I am wondering whether to add 10 nodes at one shot and let the cluster to rebalance, or divide into 5 steps, each of which add 2 nodes and rebalance step by step?  I do not know what would be the advantages or disadvantages with the one shot scheme vs 5 bataches of adding 2 nodes step-by-step. Any suggestions, experience sharing or advice are highly appreciated. thanks a lot in advance, Samuel huxiaoyu(a)horebdata.cn ------------------------------ Date: Thu, 4 May 2023 10:40:02 +0200 From: Guillaume Abrioux <gabrioux(a)redhat.com> Subject: [ceph-users] Re: 16.2.13 pacific QE validation status To: Laura Flores <lflores(a)redhat.com> Cc: Yuri Weinstein <yweinste(a)redhat.com>, Radoslaw Zarzynski <rzarzyns(a)redhat.com>, dev <dev(a)ceph.io>, ceph-users <ceph-users(a)ceph.io> Message-ID: <CANqTTH5ba9qf3xStCcCZr24n5GPyq0Eeimw3Seha1MZ6wna5nA(a)mail.gmail.com> Content-Type: text/plain; charset="UTF-8" ceph-volume approved https://jenkins.ceph.com/job/ceph-volume-test/553/ On Wed, 3 May 2023 at 22:43, Guillaume Abrioux <gabrioux(a)redhat.com> wrote: > The failure seen in ceph-volume tests isn't related. > That being said, it needs to be fixed to have a better view of the current > status. > > On Wed, 3 May 2023 at 21:00, Laura Flores <lflores(a)redhat.com> wrote: > >> upgrade/octopus-x (pacific) is approved. Went over failures with Adam >> King and it was decided they are not release blockers. >> >> On Wed, May 3, 2023 at 1:53 PM Yuri Weinstein <yweinste(a)redhat.com> >> wrote: >> >>> upgrade/octopus-x (pacific) - Laura >>> ceph-volume - Guillaume >>> >>> + 2 PRs are the remaining issues >>> >>> Josh FYI >>> >>> On Wed, May 3, 2023 at 11:50 AM Radoslaw Zarzynski <rzarzyns(a)redhat.com> >>> wrote: >>> > >>> > rados approved. >>> > >>> > Big thanks to Laura for helping with this! >>> > >>> > On Thu, Apr 27, 2023 at 11:21 PM Yuri Weinstein <yweinste(a)redhat.com> >>> wrote: >>> > > >>> > > Details of this release are summarized here: >>> > > >>> > > https://tracker.ceph.com/issues/59542#note-1 >>> > > Release Notes - TBD >>> > > >>> > > Seeking approvals for: >>> > > >>> > > smoke - Radek, Laura >>> > > rados - Radek, Laura >>> > >   rook - Sébastien Han >>> > >   cephadm - Adam K >>> > >   dashboard - Ernesto >>> > > >>> > > rgw - Casey >>> > > rbd - Ilya >>> > > krbd - Ilya >>> > > fs - Venky, Patrick >>> > > upgrade/octopus-x (pacific) - Laura (look the same as in 16.2.8) >>> > > upgrade/pacific-p2p - Laura >>> > > powercycle - Brad (SELinux denials) >>> > > ceph-volume - Guillaume, Adam K >>> > > >>> > > Thx >>> > > YuriW >>> > > _______________________________________________ >>> > > Dev mailing list -- dev(a)ceph.io >>> > > To unsubscribe send an email to dev-leave(a)ceph.io >>> > >>> _______________________________________________ >>> Dev mailing list -- dev(a)ceph.io >>> To unsubscribe send an email to dev-leave(a)ceph.io >>> >> >> >> -- >> >> Laura Flores >> >> She/Her/Hers >> >> Software Engineer, Ceph Storage <https://ceph.io> >> >> Chicago, IL >> >> lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> >> M: +17087388804 >> >> >> _______________________________________________ >> Dev mailing list -- dev(a)ceph.io >> To unsubscribe send an email to dev-leave(a)ceph.io >> > > > -- > > *Guillaume Abrioux*Senior Software Engineer > -- *Guillaume Abrioux*Senior Software Engineer ------------------------------ Subject: Digest Footer _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s ------------------------------ End of ceph-users Digest, Vol 107, Issue 20 *******************************************

1 0

MDS crash on FAILED ceph_assert(cur->is_auth())
by Peter van Heusden 04 May '23

04 May '23

I am running Ceph 15.2.13 on CentOS 7.9.2009 and recently my MDS servers have started failing with the error message In function 'void Server::handle_client_open(MDRequestRef&)' thread 7f0ca9908700 time 2021-06-28T09:21:11.484768+0200 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/15.2.13/rpm/el7/BUILD/ceph-15.2.13/src/mds/Server.cc: 4149: FAILED ceph_assert(cur->is_auth()) Complete log is: https://gist.github.com/pvanheus/4da555a6de6b5fa5e46cbf74f5500fbd ceph status output is: # ceph status cluster: id: ed7b2c16-b053-45e2-a1fe-bf3474f90508 health: HEALTH_WARN 30 OSD(s) experiencing BlueFS spillover insufficient standby MDS daemons available 1 MDSs report slow requests 2 mgr modules have failed dependencies 4347046/326505282 objects misplaced (1.331%) 6 nearfull osd(s) 23 pgs not deep-scrubbed in time 23 pgs not scrubbed in time 8 pool(s) nearfull services: mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 22m) mgr: ceph-mon1(active, since 11w), standbys: ceph-mon2, ceph-mon3 mds: SANBI_FS:2 {0=ceph-mon1=up:active(laggy or crashed),1=ceph-mon2=up:stopping} osd: 54 osds: 54 up (since 2w), 54 in (since 11w); 50 remapped pgs data: pools: 8 pools, 833 pgs objects: 42.37M objects, 89 TiB usage: 159 TiB used, 105 TiB / 264 TiB avail pgs: 4347046/326505282 objects misplaced (1.331%) 782 active+clean 49 active+clean+remapped 1 active+clean+scrubbing+deep 1 active+clean+remapped+scrubbing io: client: 29 KiB/s rd, 427 KiB/s wr, 37 op/s rd, 48 op/s wr When restarting a MDS it goes through states replace, reconnect, resolve and finally sets itself to active before this crash happens. Any advice on what to do? Thanks, Peter P.S. apologies if you received this email more than once - I have had some trouble figuring out the correct mailing list to use.

3 5

Initialization timeout, failed to initialize
by Vitaly Goot 04 May '23

04 May '23

playing with MULTI-SITE zones for CEPH Object Gateway ceph version: 17.2.5 my setup: 3 zone multi-site; 3-way full sync mode; each zone has 3 machines -> RGW+MON+OSD running load test: 3000 concurrent uploads of 1M object after about 3-4 minutes of load RGW machine get stuck, on 2 zone out of 3 RGW is not responding (e.g. curl $RGW:80) attempt to restart RGW ends up with `Initialization timeout, failed to initialize` here is a backtrace from gdb with a backtrace where it hangs after restart: (gdb) inf thr Id Target Id Frame * 1 Thread 0x7fa7d3abbcc0 (LWP 30791) "radosgw" futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7ffc7f7a2438) at ../sysdeps/nptl/futex-internal.h:183 ... (gdb) bt #0 futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7ffc7f7a2438) at ../sysdeps/nptl/futex-internal.h:183 #1 __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7ffc7f7a2488, cond=0x7ffc7f7a2410) at pthread_cond_wait.c:508 #2 __pthread_cond_wait (cond=cond@entry=0x7ffc7f7a2410, mutex=0x7ffc7f7a2488) at pthread_cond_wait.c:647 #3 0x00007fa7d7097e42 in ceph::condition_variable_debug::wait (this=this@entry=0x7ffc7f7a2410, lock=...) at ../src/common/mutex_debug.h:148 #4 0x00007fa7d7953cba in ceph::condition_variable_debug::wait<librados::IoCtxImpl::operate(const object_t&, ObjectOperation*, ceph::real_time*, int)::<lambda()> > (pred=..., lock=..., this=0x7ffc7f7a2410) at ../src/librados/IoCtxImpl.cc:672 #5 librados::IoCtxImpl::operate (this=this@entry=0x558347c21010, oid=..., o=0x558347e12310, pmtime=<optimized out>, flags=<optimized out>) at ../src/librados/IoCtxImpl.cc:672 #6 0x00007fa7d792bd55 in librados::v14_2_0::IoCtx::operate (this=this@entry=0x558347e44760, oid="notify.0", o=o@entry=0x7ffc7f7a2690, flags=flags@entry=0) at ../src/librados/librados_cxx.cc:1536 #7 0x00007fa7d9490ad1 in rgw_rados_operate (dpp=<optimized out>, ioctx=..., oid="notify.0", op=op@entry=0x7ffc7f7a2690, y=..., flags=0) at ../src/rgw/rgw_tools.cc:277 #8 0x00007fa7d9627e0f in RGWSI_RADOS::Obj::operate (this=this@entry=0x558347e44710, dpp=<optimized out>, op=op@entry=0x7ffc7f7a2690, y=..., flags=flags@entry=0) at ../src/rgw/services/svc_rados.h:112 #9 0x00007fa7d96209a5 in RGWSI_Notify::init_watch (this=this@entry=0x558347c49530, dpp=<optimized out>, y=...) at ../src/rgw/services/svc_notify.cc:214 #10 0x00007fa7d962161b in RGWSI_Notify::do_start (this=0x558347c49530, y=..., dpp=<optimized out>) at ../src/rgw/services/svc_notify.cc:277 #11 0x00007fa7d8f17bcf in RGWServiceInstance::start (this=0x558347c49530, y=..., dpp=<optimized out>) at ../src/rgw/rgw_service.cc:331 #12 0x00007fa7d8f1a260 in RGWServices_Def::init (this=this@entry=0x558347de90a0, cct=<optimized out>, have_cache=<optimized out>, raw=raw@entry=false, run_sync=<optimized out>, y=..., dpp=<optimized out>) at /usr/include/c++/9/bits/unique_ptr.h:360 #13 0x00007fa7d8f1cc40 in RGWServices::do_init (this=this@entry=0x558347de90a0, _cct=<optimized out>, have_cache=<optimized out>, raw=raw@entry=false, run_sync=<optimized out>, y=..., dpp=<optimized out>) at ../src/rgw/rgw_service.cc:284 #14 0x00007fa7d92a7b1f in RGWServices::init (dpp=<optimized out>, y=..., run_sync=<optimized out>, have_cache=<optimized out>, cct=<optimized out>, this=0x558347de90a0) at ../src/rgw/rgw_service.h:153 #15 RGWRados::init_svc (this=this@entry=0x558347de8dc0, raw=raw@entry=false, dpp=<optimized out>) at ../src/rgw/rgw_rados.cc:1380 #16 0x00007fa7d930f241 in RGWRados::initialize (this=0x558347de8dc0, dpp=<optimized out>) at ../src/rgw/rgw_rados.cc:1400 #17 0x00007fa7d944f85f in RGWRados::initialize (dpp=<optimized out>, _cct=0x558347c6a320, this=<optimized out>) at ../src/rgw/rgw_rados.h:586 #18 StoreManager::init_storage_provider (dpp=<optimized out>, dpp@entry=0x7ffc7f7a2e90, cct=cct@entry=0x558347c6a320, svc="rados", use_gc_thread=use_gc_thread@entry=true, use_lc_thread=use_lc_thread@entry=true, quota_threads=quota_threads@entry=true, run_sync_thread=true, run_reshard_thread=true, use_cache=true, use_gc=true) at ../src/rgw/rgw_sal.cc:55 #19 0x00007fa7d8e7367a in StoreManager::get_storage (use_gc=true, use_cache=true, run_reshard_thread=true, run_sync_thread=true, quota_threads=true, use_lc_thread=true, use_gc_thread=true, svc="rados", cct=0x558347c6a320, dpp=0x7ffc7f7a2e90) at /usr/include/c++/9/bits/basic_string.h:267 #20 radosgw_Main (argc=<optimized out>, argv=<optimized out>) at ../src/rgw/rgw_main.cc:372 #21 0x0000558347883f56 in main (argc=<optimized out>, argv=<optimized out>) at ../src/rgw/radosgw.cc:12 (gdb) #0 futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7ffc7f7a2438) at ../sysdeps/nptl/futex-internal.h:183 #1 __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7ffc7f7a2488, cond=0x7ffc7f7a2410) at pthread_cond_wait.c:508 #2 __pthread_cond_wait (cond=cond@entry=0x7ffc7f7a2410, mutex=0x7ffc7f7a2488) at pthread_cond_wait.c:647 #3 0x00007fa7d7097e42 in ceph::condition_variable_debug::wait (this=this@entry=0x7ffc7f7a2410, lock=...) at ../src/common/mutex_debug.h:148 #4 0x00007fa7d7953cba in ceph::condition_variable_debug::wait<librados::IoCtxImpl::operate(const object_t&, ObjectOperation*, ceph::real_time*, int)::<lambda()> > (pred=..., lock=..., this=0x7ffc7f7a2410) at ../src/librados/IoCtxImpl.cc:672 #5 librados::IoCtxImpl::operate (this=this@entry=0x558347c21010, oid=..., o=0x558347e12310, pmtime=<optimized out>, flags=<optimized out>) at ../src/librados/IoCtxImpl.cc:672 #6 0x00007fa7d792bd55 in librados::v14_2_0::IoCtx::operate (this=this@entry=0x558347e44760, oid="notify.0", o=o@entry=0x7ffc7f7a2690, flags=flags@entry=0) at ../src/librados/librados_cxx.cc:1536 #7 0x00007fa7d9490ad1 in rgw_rados_operate (dpp=<optimized out>, ioctx=..., oid="notify.0", op=op@entry=0x7ffc7f7a2690, y=..., flags=0) at ../src/rgw/rgw_tools.cc:277 #8 0x00007fa7d9627e0f in RGWSI_RADOS::Obj::operate (this=this@entry=0x558347e44710, dpp=<optimized out>, op=op@entry=0x7ffc7f7a2690, y=..., flags=flags@entry=0) at ../src/rgw/services/svc_rados.h:112 #9 0x00007fa7d96209a5 in RGWSI_Notify::init_watch (this=this@entry=0x558347c49530, dpp=<optimized out>, y=...) at ../src/rgw/services/svc_notify.cc:214 #10 0x00007fa7d962161b in RGWSI_Notify::do_start (this=0x558347c49530, y=..., dpp=<optimized out>) at ../src/rgw/services/svc_notify.cc:277 #11 0x00007fa7d8f17bcf in RGWServiceInstance::start (this=0x558347c49530, y=..., dpp=<optimized out>) at ../src/rgw/rgw_service.cc:331 #12 0x00007fa7d8f1a260 in RGWServices_Def::init (this=this@entry=0x558347de90a0, cct=<optimized out>, have_cache=<optimized out>, raw=raw@entry=false, run_sync=<optimized out>, y=..., dpp=<optimized out>) at /usr/include/c++/9/bits/unique_ptr.h:360 #13 0x00007fa7d8f1cc40 in RGWServices::do_init (this=this@entry=0x558347de90a0, _cct=<optimized out>, have_cache=<optimized out>, raw=raw@entry=false, run_sync=<optimized out>, y=..., dpp=<optimized out>) at ../src/rgw/rgw_service.cc:284 #14 0x00007fa7d92a7b1f in RGWServices::init (dpp=<optimized out>, y=..., run_sync=<optimized out>, have_cache=<optimized out>, cct=<optimized out>, this=0x558347de90a0) at ../src/rgw/rgw_service.h:153 #15 RGWRados::init_svc (this=this@entry=0x558347de8dc0, raw=raw@entry=false, dpp=<optimized out>) at ../src/rgw/rgw_rados.cc:1380 #16 0x00007fa7d930f241 in RGWRados::initialize (this=0x558347de8dc0, dpp=<optimized out>) at ../src/rgw/rgw_rados.cc:1400 #17 0x00007fa7d944f85f in RGWRados::initialize (dpp=<optimized out>, _cct=0x558347c6a320, this=<optimized out>) at ../src/rgw/rgw_rados.h:586 #18 StoreManager::init_storage_provider (dpp=<optimized out>, dpp@entry=0x7ffc7f7a2e90, cct=cct@entry=0x558347c6a320, svc="rados", use_gc_thread=use_gc_thread@entry=true, use_lc_thread=use_lc_thread@entry=true, quota_threads=quota_threads@entry=true, run_sync_thread=true, run_reshard_thread=true, use_cache=true, use_gc=true) at ../src/rgw/rgw_sal.cc:55 #19 0x00007fa7d8e7367a in StoreManager::get_storage (use_gc=true, use_cache=true, run_reshard_thread=true, run_sync_thread=true, quota_threads=true, use_lc_thread=true, use_gc_thread=true, svc="rados", cct=0x558347c6a320, dpp=0x7ffc7f7a2e90) at /usr/include/c++/9/bits/basic_string.h:267 #20 radosgw_Main (argc=<optimized out>, argv=<optimized out>) at ../src/rgw/rgw_main.cc:372 #21 0x0000558347883f56 in main (argc=<optimized out>, argv=<optimized out>) at ../src/rgw/radosgw.cc:12 Any suggestion on what can be a problem and how to reset RGW so it will be able to start normally?

1 0

test
by Dan Mick 04 May '23

04 May '23

making some DMARC-related changes. Ignore this please.

1 0

To drain unmanaged OSDs by an oversight
by Ackermann, Christoph 03 May '23

03 May '23

Hello List, i made a mistake, draining a host instead of entering it into Maintenance Mode (for OS reboot). :-/ After "Stop Drain" and restore of original "crush reweight" values, so far everything looks fine. cluster: health: HEALTH_OK services: [..] osd: 79 osds: 78 up (since 3h), 78 in (since 6w); 166 remapped pgs [..] And some minor objects being misplaced.. But I can't remove/solve this annoying message "deleting" on these host OSDs. [image: Auswahl_2023-05-03_14-17.png] Does someone have a hint for me? Thanks, Christoph

1 0

MAnual Upgrade from Octopus Ubuntu 18.04 to Quincy 20.04
by Iban Cabrillo 03 May '23

03 May '23

Dear Cephers, We are planing the dist upgrade from Octopus to Quincy in the next weeks. The first step is the linux version upgrade from Ubuntu 18.04 to Ubuntu 20.04 from some big ODS servers runnign this OS version. we just have a look at ( Upgrading non-cephadm clusters [ https://ceph.io/en/news/blog/2022/v17-2-0-quincy-released/#upgrading-non-ce… | ¶ ] ): https://ceph.io/en/news/blog/2022/v17-2-0-quincy-released/ Is any advise or suggestion before to start the procedure ? regards, I -- ================================================================ Ibán Cabrillo Bartolomé Instituto de Física de Cantabria (IFCA-CSIC) Santander, Spain Tel: +34942200969/+34669930421 Responsible for advanced computing service (RSC) ========================================================================================= ========================================================================================= All our suppliers must know and accept IFCA policy available at: https://confluence.ifca.es/display/IC/Information+Security+Policy+for+Exter… ==========================================================================================

1 0

Flushing stops as copy-from message being throttled
by lingu2008 03 May '23

03 May '23

Hi all, On one server with a cache tier on Samsung PM983 SSDs for an EC base tier on HDDs, I find the cache tier stops flushing or evicting when the cache tier is near full. With quite some gdb-debugging, I find the problem may be with the throttling mechanism. When the write traffic is high, the cache tier quickly fills its maximum request count and throttles further requests. Then flush stops because copy-from requests are throttled by the cache tier OSD. Ironically, the 256 requests already accepted by the cache tier cannot proceed, either, because the cache tier is full and cannot flush/evict. While we may advise cache tier should not go full, this deadlock situation is not entirely comprehensible to me because a full cache usually can flush/evict as long as the base tier has space. I wonder whether there has been some specific reasons for this behavior. My test environment is with version 15.2.17 but the code in 17.2.2 appears to handle this part of logic in the same way. Cheers, lin

2 1

RBD mirroring, asking for clarification
by wodel youchi 03 May '23

03 May '23

Hi, When using rbd mirroring, the mirroring concerns the images only, not the whole pool? So, we don't need to have a dedicated pool in the destination site to be mirrored, the only obligation is that the mirrored pools must have the same name. In other words, We create two pools with the same name, one on the source site the other on the destination site, we create the mirror link (one way or two ways replication), then we choose what images to sync. Both pools can be used simultaneously on both sites, it's the mirrored images that cannot be used simultaneously, only promoted ones. Is this correct? Regards.

2 5

architecture help (iscsi, rbd, backups?)
by Angelo Höngens 02 May '23

02 May '23

Hey guys and girls, I'm working on a project to build storage for one of our departments, and I want to ask you guys and girls for input on the high-level overview part. It's a long one, I hope you read along and comment. SUMMARY I made a plan last year to build a 'storage solution' including ceph and some windows VM's to expose the data over SMB to clients. A year later I finally have the hardware, built a ceph cluster, and I'm doing tests. Ceph itself runs great, but when I wanted to start exposing the data using iscsi to our VMware farm, I ran into some issues. I know the iscsi gateways will introduce some new performance bottlenecks, but I'm seeing really slow performance, still working on that. But then I ran into the warning on the iscsi gateway page: "The iSCSI gateway is in maintenance as of November 2022. This means that it is no longer in active development and will not be updated to add new features.". Wait, what? Why!? What does this mean? Does this mean that iSCSI is now 'feature complete' and will still be supported the next 5 years, or will it be deprecated in the future? I tried searching, but couldn't find any info on the decision and the roadmap. My goal is to build a future-proof setup, and using deprecated components should not be part of that of course. If the iscsi gateway will still be supported the next few years and I can iron out the performance issues, I can still go on with my original plan. If not, I have to go back to the drawing board. And maybe you guys would advise me to take another route anyway. GOALS My goals/considerations are: - we want >1PB of storage capacity for cheap (on a tight budget) for research data. Most of it is 'store once, read sometimes'. <1% of the data is 'hot'. - focus is on capacity, but it would be nice to have > 200MB/s of sequential write/read performance and not 'totally suck' on random i/o. Yes, not very well quantified, but ah. Sequential writes are most important. - end users all run Windows computers (mostly VDI's) and a lot of applications require SMB shares. - security is a big thing, we want really tight ACL's, specific monitoring agents, etc. - our data is incredibly important to us, we still want the 3-2-1 backup rule. Primary storage solution, a second storage solution in a different place, and some of the data that is not reproducible is also written to tape. We also want to be protected from ransomware or user errors (so no direct replication to the second storage). - I like open source, reliability, no fork-lift upgrades, no vendor lock-in, blah, well, I'm on the ceph list here, no need to convince you guys ;) - We're hiring a commercial company to do ceph maintenance and support for when I'm on leave or leaving the company, but they won't support clients, backup software, etc, so I want something as simple as possible. We do have multiple Windows/VMware admins, but no other real linux guru's. THE INITIAL PLAN Given these considerations, I ordered two identical clusters, each consisting of 3 monitor nodes and 8 osd nodes, Each osd node has 2 ssd's and 10 capacity disks (EC 4:2 for the data), and each node is connected using a 2x25Gbps bond. Ceph is running like a charm. Now I just have to think about exposing the data to end users, and I've been testing different setups. My original plan was to expose for example 10x100TB rbd images using iSCSI to our VMware farm, formatting the luns with VMFS6, and run for example 2 Windows file servers per datastore on that with a single DFS namespace to end users. Then backup the file servers using our existing Veeam infrastructure to RGW running on the second cluster with an immutable bucket. This way we would have easily defined security boundaries: the clients can only reach the file servers, the file servers only see their local VMDK's, ESX only sees the luns on the iSCSI target, etc. When a file server would be compromised, it would have no access to ceph. We have easy incremental backups, immutability for ransomware protection, etc. And the best part is that the ceph admin can worry about ceph, the vmware admin can focus on ESX, VMFS and all the vmware stuff, and the Windows admins can focus on the Windows boxes, Windows-specific ACLS and tools and Veeam backups and stuff. CURRENT SITUATION I'm building out this plan now, but I'm running into issues with iSCSI. Are any of you doing something similar? What is your iscsi performance compared to direct rbd? In regard to performance: If I take 2 test windows VM's, I put one on an iSCSI datastore and another with direct rbd access using the windows rbd driver, I create a share on those boxes and push data to it, I see different results (of course). Copying some iso images over SMB to the 'windows vm running direct rbd' I see around 800MB/s write, and 200MB/s read, which is pretty okay. When I send data to the 'windows vm running on top of iscsi' it starts writing at around 350MB/s, but after like 10-20 seconds drops to 100MB/s and won't go faster. Reads are anywhere from 40MB/s to 80MB/s, which is not really acceptable. Another really viable and performant scenario would be to have the Windows file servers connect to rbd directly with the windows rbd driver. It seems to work well, it's fast, and you don't have the bottleneck that the iscsi gateway creates. But I see this driver is still in beta. Is anyone using this in production? What are your experiences? We would miss out on the separation of layers and thus have less security, but at the same time, it really increases efficiency and performance. And if I use rbd, then vmware won't see the storage, and I cannot do an image backup using veeam. I could of course do backups of the rbd images, using tools like restic or backy to rgw running on the second cluster with immutable buckets. What are your experiences? Is it easy to do differential backups of lots of 50TB rbd images? Change rate is usually like 0.005% per day or something ;) By the way, we also thought about CephFS, but we have some complex stuff going on with extended ACL's that I don't think will play nice with CephFS, and I think it's a lot more complex to backup CephFS than block images. If you made it here, thank you for your time! I hope you can share thoughts on my questions! Angelo.

6 8

[multisite] "bucket sync status" takes a while
by Yixin Jin 02 May '23

02 May '23

Hi folks, With a multi-site environment, when I create a bucket-level sync policy with a symmetric flow between the master zone and another zone, "bucket sync status" immediately shows that the sync is now enabled in the master zone. But it takes a while for it to show that in the other zone. I tried "period pull" at the other zone and "period push" at the master zone. Neither seem to make a difference. Is there a way to speed up this process? Thanks, Yixin

1 0

17.2.6 fs 'ls' ok, but 'cat' 'operation not permitted' puzzle
by Harry G Coin 02 May '23

02 May '23

In 17.2.6 is there a security requirement that pool names supporting a ceph fs filesystem match the filesystem name.data for the data and name.meta for the associated metadata pool? (multiple file systems are enabled) I have filesystems from older versions with the data pool name matching the filesystem and appending _metadata for that, and even older filesystems with the pool name as in 'library' and 'library_metadata' supporting a filesystem called 'libraryfs' The pools all have the cephfs tag. But using the documented: ceph fs authorize libraryfs client.basicuser / rw command allows the root user to mount and browse the library directory tree, but fails with 'operation not permitted' when even reading any file. However, changing the client.basicuser osd auth to 'allow rw' instead of 'allow rw tag...' allows normal operations. So: [client.basicuser] key = <key stuff>== caps mds = "allow rw fsname=libraryfs" caps mon = "allow r fsname=libraryfs" caps osd = "allow rw" works, but the same with caps osd = "allow rw tag cephfs data=libraryfs" leads to the 'operation not permitted' on read, or write or any actual access. It remains a puzzle. Help appreciated! Were there upgrade instructions about that, any help pointing me to them? Thanks Harry Coin Rock Stable Systems

1 0

Memory leak in MGR after upgrading to pacific.
by Gary Molenkamp 02 May '23

02 May '23

Good morning, After upgrading from Octopus (15.2.17) to Pacific (16.2.12) two days ago, I'm noticing that the MGR daemons keep failing over to standby and then back every 24hrs. Watching the output of 'ceph orch ps' I can see that the memory consumption of the mgr is steadily growing until it becomes unresponsive. When the mgr becomes unresponsive, tasks such as RESTful calls start to fail, and the standby eventually takes over after ~20 minutes. I've included a log of memory consumption (in 10 minute intervals) at the end of this message. While the cluster recovers during this issue, the loss of usage data during the outage, and the fact its occurring is problematic. Any assistance would be appreciated. Note, this is a cluster that has been upgraded from an original jewel based ceph using filestore, through bluestore conversion, container conversion, and now to Pacific. The data below shows memory use with three mgr modules enabled: cephadm, restful, iostat. By disabling iostat, I can reduce the rate of memory consumption increasing to about 200MB/hr. Thanks Gary. Wed Apr 26 12:50:08 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (3h) 7m ago 3h 1099M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (16h) 7m ago 16h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 13:00:09 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (3h) 6m ago 3h 1161M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (16h) 6m ago 16h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 13:10:10 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (3h) 6m ago 3h 1230M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (16h) 6m ago 16h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 13:20:11 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (4h) 6m ago 4h 1250M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (16h) 6m ago 16h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 13:30:12 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (4h) 5m ago 4h 1318M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (16h) 5m ago 16h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 13:40:13 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (4h) 5m ago 4h 1379M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (17h) 5m ago 17h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 13:50:13 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (4h) 5m ago 4h 1442M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (17h) 5m ago 17h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 14:00:14 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (4h) 4m ago 4h 1498M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (17h) 4m ago 17h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 14:10:15 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (4h) 4m ago 4h 1554M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (17h) 4m ago 17h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 14:20:16 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (5h) 4m ago 5h 1617M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (17h) 3m ago 17h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 14:30:17 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (5h) 3m ago 5h 1677M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (17h) 3m ago 17h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 14:40:18 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (5h) 3m ago 5h 1735M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (18h) 3m ago 18h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 14:50:19 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (5h) 2m ago 5h 1806M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (18h) 2m ago 18h 383M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 15:00:20 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (5h) 2m ago 5h 1873M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (18h) 2m ago 18h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 15:10:21 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (5h) 2m ago 5h 1920M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (18h) 2m ago 18h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 15:20:21 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (6h) 114s ago 6h 1981M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (18h) 113s ago 18h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 15:30:22 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (6h) 95s ago 6h 2048M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (18h) 95s ago 18h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 15:40:23 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (6h) 80s ago 6h 2121M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (19h) 80s ago 19h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 15:50:24 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (6h) 53s ago 6h 2170M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (19h) 53s ago 19h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 16:00:25 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (6h) 35s ago 6h 2238M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (19h) 34s ago 19h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 16:10:26 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (6h) 19s ago 6h 2266M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (19h) 19s ago 19h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 16:20:27 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (7h) 10m ago 7h 2266M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (19h) 10m ago 19h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 16:30:27 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (7h) 9m ago 7h 2321M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (19h) 9m ago 19h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 16:40:28 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (7h) 9m ago 7h 2382M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (20h) 9m ago 20h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 16:50:29 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (7h) 9m ago 7h 2449M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (20h) 9m ago 20h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 17:00:30 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (7h) 8m ago 7h 2505M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (20h) 8m ago 20h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 17:10:31 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (7h) 8m ago 7h 2562M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (20h) 8m ago 20h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 17:20:32 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (8h) 8m ago 8h 2628M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (20h) 8m ago 20h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 17:30:33 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (8h) 7m ago 8h 2692M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (20h) 7m ago 20h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 17:40:34 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (8h) 7m ago 8h 2761M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (21h) 7m ago 21h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 17:50:34 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (8h) 7m ago 8h 2811M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (21h) 7m ago 21h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 18:00:35 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (8h) 6m ago 8h 2873M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (21h) 6m ago 21h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 18:10:36 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (8h) 6m ago 8h 2923M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (21h) 6m ago 21h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 18:20:37 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (9h) 6m ago 9h 2995M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (21h) 6m ago 21h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 18:30:38 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (9h) 5m ago 9h 3055M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (21h) 5m ago 21h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 18:40:39 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (9h) 5m ago 9h 3110M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (22h) 5m ago 22h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 18:50:40 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (9h) 5m ago 9h 3166M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (22h) 5m ago 22h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 19:00:41 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (9h) 4m ago 9h 3244M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (22h) 4m ago 22h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 19:10:42 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (9h) 4m ago 9h 3289M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (22h) 4m ago 22h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 19:20:42 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (10h) 4m ago 10h 3368M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (22h) 4m ago 22h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 19:30:43 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (10h) 3m ago 10h 3437M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (22h) 3m ago 22h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 19:40:44 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (10h) 3m ago 10h 3496M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (23h) 3m ago 23h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 19:50:45 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (10h) 3m ago 10h 3546M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (23h) 3m ago 23h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 20:00:46 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (10h) 2m ago 10h 3627M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (23h) 2m ago 23h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 20:10:47 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (10h) 2m ago 10h 3637M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (23h) 2m ago 23h 384M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 20:20:48 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (11h) 92s ago 11h 492M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (23h) 91s ago 23h 418M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 20:31:55 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (11h) 51s ago 11h 373M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (23h) 51s ago 23h 520M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 20:41:56 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (11h) 32s ago 11h 374M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (0h) 32s ago 0h 585M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 20:51:57 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (11h) 17s ago 11h 376M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (0h) 17s ago 0h 647M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 21:01:58 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (11h) 10m ago 11h 376M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (0h) 10m ago 0h 647M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 21:11:58 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (11h) 9m ago 11h 378M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (0h) 9m ago 0h 698M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 21:21:59 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (12h) 9m ago 12h 379M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (0h) 9m ago 0h 758M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 21:32:00 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (12h) 9m ago 12h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (0h) 9m ago 0h 825M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 21:42:01 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (12h) 8m ago 12h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (1h) 8m ago 1h 895M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 21:52:02 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (12h) 8m ago 12h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (1h) 8m ago 1h 948M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 22:02:03 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (12h) 8m ago 12h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (1h) 8m ago 1h 1010M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 22:12:04 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (12h) 7m ago 12h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (1h) 7m ago 1h 1071M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 22:22:05 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (13h) 7m ago 13h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (1h) 7m ago 1h 1132M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 22:32:06 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (13h) 7m ago 13h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (1h) 7m ago 1h 1183M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 22:42:06 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (13h) 6m ago 13h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (2h) 6m ago 2h 1244M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 22:52:07 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (13h) 6m ago 13h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (2h) 6m ago 2h 1311M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 23:02:08 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (13h) 6m ago 13h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (2h) 6m ago 2h 1380M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 23:12:09 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (13h) 5m ago 13h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (2h) 5m ago 2h 1441M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 23:22:10 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (14h) 5m ago 14h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (2h) 5m ago 2h 1508M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 23:32:11 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (14h) 5m ago 14h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (2h) 5m ago 2h 1541M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 23:42:12 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (14h) 4m ago 14h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (3h) 4m ago 3h 1610M - 16.2.12 921c4e969fff 6fed006895e8 Wed Apr 26 23:52:13 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (14h) 4m ago 14h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (3h) 4m ago 3h 1674M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 00:02:14 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (14h) 4m ago 14h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (3h) 4m ago 3h 1725M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 00:12:14 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (14h) 3m ago 14h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (3h) 3m ago 3h 1794M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 00:22:15 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (15h) 3m ago 15h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (3h) 3m ago 3h 1844M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 00:32:16 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (15h) 3m ago 15h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (3h) 3m ago 3h 1917M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 00:42:17 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (15h) 2m ago 15h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (4h) 2m ago 4h 2002M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 00:52:18 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (15h) 2m ago 15h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (4h) 2m ago 4h 2056M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 01:02:19 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (15h) 2m ago 15h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (4h) 2m ago 4h 2113M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 01:12:20 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (15h) 92s ago 15h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (4h) 91s ago 4h 2173M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 01:22:21 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (16h) 73s ago 16h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (4h) 72s ago 4h 2201M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 01:32:22 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (16h) 57s ago 16h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (4h) 57s ago 4h 2264M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 01:42:22 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (16h) 29s ago 16h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (5h) 29s ago 5h 2324M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 01:52:23 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (16h) 10s ago 16h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (5h) 10s ago 5h 2387M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 02:02:24 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (16h) 10m ago 16h 380M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (5h) 10m ago 5h 2387M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 02:12:25 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (16h) 9m ago 16h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (5h) 9m ago 5h 2442M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 02:22:26 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (17h) 9m ago 17h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (5h) 9m ago 5h 2502M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 02:32:27 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (17h) 9m ago 17h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (5h) 9m ago 5h 2552M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 02:42:28 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (17h) 8m ago 17h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (6h) 8m ago 6h 2614M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 02:52:28 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (17h) 8m ago 17h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (6h) 8m ago 6h 2681M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 03:02:29 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (17h) 8m ago 17h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (6h) 8m ago 6h 2748M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 03:12:30 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (17h) 7m ago 17h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (6h) 7m ago 6h 2810M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 03:22:31 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (18h) 7m ago 18h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (6h) 7m ago 6h 2867M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 03:32:32 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (18h) 7m ago 18h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (6h) 7m ago 6h 2916M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 03:42:33 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (18h) 6m ago 18h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (7h) 6m ago 7h 3024M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 03:52:34 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (18h) 6m ago 18h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (7h) 6m ago 7h 3070M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 04:02:35 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (18h) 6m ago 18h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (7h) 6m ago 7h 3127M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 04:12:36 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (18h) 5m ago 18h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (7h) 5m ago 7h 3183M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 04:22:36 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (19h) 5m ago 19h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (7h) 5m ago 7h 3215M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 04:32:37 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (19h) 4m ago 19h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (7h) 4m ago 7h 3290M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 04:42:38 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (19h) 4m ago 19h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (8h) 4m ago 8h 3342M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 04:52:39 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (19h) 4m ago 19h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (8h) 4m ago 8h 3392M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 05:02:40 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (19h) 3m ago 19h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (8h) 3m ago 8h 3451M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 05:12:41 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (19h) 3m ago 19h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (8h) 3m ago 8h 3516M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 05:22:42 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (20h) 3m ago 20h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (8h) 3m ago 8h 3566M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 05:32:43 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (20h) 2m ago 20h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (8h) 2m ago 8h 3632M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 05:42:44 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (20h) 2m ago 20h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (9h) 2m ago 9h 3696M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 05:52:44 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (20h) 2m ago 20h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (9h) 2m ago 9h 3760M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 06:02:45 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (20h) 115s ago 20h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (9h) 114s ago 9h 3813M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 06:12:46 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (20h) 99s ago 20h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (9h) 99s ago 9h 3869M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 06:22:47 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (21h) 72s ago 21h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (9h) 72s ago 9h 3923M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 06:32:48 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (21h) 54s ago 21h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (9h) 53s ago 9h 3990M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 06:42:49 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (21h) 38s ago 21h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (10h) 38s ago 10h 4040M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 06:52:50 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (21h) 11s ago 21h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (10h) 10s ago 10h 4097M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 07:02:51 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (21h) 10m ago 21h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (10h) 10m ago 10h 4097M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 07:12:51 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (21h) 9m ago 21h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (10h) 9m ago 10h 4157M - 16.2.12 921c4e969fff 6fed006895e8 Thu Apr 27 07:22:52 EDT 2023 mgr.controller04.chhvsy controller04.mydomain running (22h) 9m ago 22h 381M - 16.2.12 921c4e969fff e898a9606028 mgr.storage02.jnvnrm storage02.mydomain *:8003 running (10h) 9m ago 10h 4213M - 16.2.12 921c4e969fff 6fed006895e8 -- Gary Molenkamp Science Technology Services Systems Administrator University of Western Ontario molenkam(a)uwo.ca http://sts.sci.uwo.ca (519) 661-2111 x86882 (519) 661-3566

1 1

Balancing Reads in Ceph
by Alan Nair 02 May '23

02 May '23

Hi. I am currently using Ceph for replicated storage to store many objects across 5 nodes with 3x replication. When I generate ~1000 read requests to a single object, they all get serviced by the same primary OSD. I would like to balance the reads across the replicas. So I use the following: auto read_op = rados_create_read_op(); rados_read_op_read(read_op, offset, outSize, buffer, &bytes_read, &prval); err = rados_read_op_operate(read_op, pool->ioctx, keyName.c_str(), LIBRADOS_OPERATION_BALANCE_READS); However, this does not seem to balance the reads across replicas. I do not see what I am doing wrong in the above code. Could you please guide me on this? ceph-mon and ceph-osd are run on Ubuntu 22.04 installed via apt-get update ceph ceph-mds ceph-volume If I should ask this question somewhere else, please point me in the right direction. Thanks and regards, Alan. The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Is e buidheann carthannais a th' ann an Oilthigh Dhùn Èideann, clàraichte an Alba, àireamh clàraidh SC005336.

1 0

PVE CEPH OSD heartbeat show
by Peter 02 May '23

02 May '23

Dear all, We are experiencing with Ceph after deploying it by PVE with the network backed by a 10G Cisco switch with VPC feature on. We are encountering a slow OSD heartbeat and have not been able to identify any network traffic issues. Upon checking, we found that the ping is around 0.1ms, and there is occasional 2% packet loss when using flood ping, but not consistently. We also noticed a large number of UDP port 5405 packets and the 'corosync' process utilizing a significant amount of CPU. When running the 'ceph -s' command, we observed a slow OSD heartbeat on the back and front, with the longest latency being 2250.54ms. We suspect that this may be a network issue, but we are unsure of how Ceph detects such long latency. Additionally, we are wondering if a 2% packet loss can significantly affect Ceph's performance and even cause the OSD process to fail sometimes. We have heard about potential issues with rockdb 6 causing OSD process failures, and we are curious about how to check the rockdb version. Furthermore, we are wondering how severe traffic package loss and latency must be to cause OSD process crashes, and how the monitoring system determines that an OSD is offline. We would greatly appreciate any assistance or insights you could provide on these matters. Thanks,

3 4

Re: cephfs - max snapshot limit?
by MARTEL Arnaud 02 May '23

02 May '23

Hi Venky, > Also, at one point the kclient wasn't able to handle more than 400 snapshots (per file system), but we have come a long way from that and that is not a constraint right now. Does it mean that there is no more limit to the number of snapshots per filesystem? And, if not, do you know what is the max number of snapshots now (per filesystem) ?? Cheers, Arnaud

2 1

[multisite] Resetting an empty bucket
by Yixin Jin 01 May '23

01 May '23

Hi folks, Armed with bucket-specific sync policy feature, I found that we could move objects of a bucket between zones. It is migration via sync followed by object removal at the source. This allows us to better utilize available capacities in different clusters/zones. However, to achieve this, we need a way to reset an empty bucket so that it can serve as a destination for a migration after it serves as a source before. ceph/rgw currently doesn't seem to be able to do that. So I create a feature request for it https://tracker.ceph.com/issues/59593 My own prototype shows that this feature is fairly simply to implement and works well for bucket migration. Cheers, Yixin

2 1

Ceph recovery
by wodel youchi 01 May '23

01 May '23

Hi, When creating a ceph cluster, a failover domain is created, and by default it uses host as a minimal domain, that domain can be modified to chassis, or rack, ...etc. My question is : Suppose I have three osd nodes, my replication is 3 and my failover domain is host, which means that each copy of data is stored on a different node. What happens when one node crashes, does Ceph use the remaining free space on the other two to create the third copy, or the ceph cluster will run in degraded mode, like a RAID5 which lost a disk. Regards.

2 2

Nearly 1 exabyte of Ceph storage
by Yaarit Hatuka 01 May '23

01 May '23

Hi everyone, We are excited to share with you the latest statistics from our Ceph public telemetry dashboards <https://telemetry-public.ceph.com/>. With almost 2,500 clusters reporting nearly 1 exabyte of total storage capacity, we can see a steady growth rate <https://telemetry-public.ceph.com/d/ZFYuv1qWz/telemetry?viewPanel=19&orgId=…> since the telemetry service was launched. One of the things telemetry helps us to understand is version adoption rate. See, for example, the trend of Quincy <https://telemetry-public.ceph.com/d/ZFYuv1qWz/telemetry?viewPanel=28&orgId=…> deployments in the community. We can also learn that the average cluster size grows over time <https://telemetry-public.ceph.com/d/ZFYuv1qWz/telemetry?viewPanel=33&orgId=1>. Ceph telemetry is on an opt-in basis. You can opt-in with: `ceph telemetry on` Learn more here <https://docs.ceph.com/en/latest/mgr/telemetry/#enabling-telemetry>. Help us cross the exabyte mark by opting-in today! Learn more about the latest developments around Telemetry <https://sched.co/1JKZ2> at the upcoming Cephalocon. Thank you for your contribution, Yaarit [image: ceph_telemetry_dashboards.jpg]

2 3

RGW Lua - cancel request
by Ondřej Kukla 01 May '23

01 May '23

Hello, Lately I’ve been playing with Lua scripting on top of RGW. I would like to implement a request blocking based on bucket name -> when there is a dot in a bucket name return error code and a message that this name is invalid. Here is the code I was able to came up with. if string.find(Request.HTTP.URI, '%.') then Request.Response.HTTPStatusCode = 400 Request.Response.HTTPStatus = “InvalidBucketName" Request.Response.Message = “Dots in bucket name are not allowed." end This works fine, but the request for creating a bucket would be processed and the bucket will be created. I thought about a dirty workaround with setting the Request.Bucket.Name to a bucket that already exists but it seems that this field is not writable in Quincy. Is there a way to block the request from processing? Any help is much appreciated. Kind regards, Ondrej

3 3