- ceph-users - lists.ceph.io

User + Dev Monthly Meeting happening this Thursday
by Laura Flores 16 May '23

16 May '23

Hi Ceph Users, The User + Dev Monthly Meeting is coming up next week on *Thursday, May 17th* *@* *14:00 UTC* (time conversions below). See meeting details at the bottom of this email. Please add any topics you'd like to discuss to the agenda: https://pad.ceph.com/p/ceph-user-dev-monthly-minutes See you there, Laura Flores Meeting link: https://meet.jit.si/ceph-user-dev-monthly Time Conversions: UTC: Wednesday, May 17, 14:00 UTC Mountain View, CA, US: Wednesday, May 17, 7:00 PDT Phoenix, AZ, US: Wednesday, May 17, 7:00 MST Denver, CO, US: Wednesday, May 17, 8:00 MDT Huntsville, AL, US: Wednesday, May 17, 9:00 CDT Raleigh, NC, US: Wednesday, May 17, 10:00 EDT London, England: Wednesday, May 17, 15:00 BST Paris, France: Wednesday, May 17, 16:00 CEST Helsinki, Finland: Wednesday, May 17, 17:00 EEST Tel Aviv, Israel: Wednesday, May 17, 17:00 IDT Pune, India: Wednesday, May 17, 19:30 IST Brisbane, Australia: Thursday, May 18, 0:00 AEST Singapore, Asia: Wednesday, May 17, 22:00 +08 Auckland, New Zealand: Thursday, May 18, 2:00 NZST -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

1 1

rbd mirror snapshot trash
by Eugen Block 16 May '23

16 May '23

Good morning, I would be grateful if anybody could shed some light on this, I can't reproduce it in my lab clusters so I was hoping for the community. A customer has 2 clusters with rbd mirroring (snapshots) enabled, it seems to work fine, they have regular checks and the images on the remote site are synced correctly. While investigating a different issue (could be related though) we noticed that there are quite a lot of snapshots in the trash namespace but for unkown reasons they are not purged. This is some example output from the remote site (the primary site looks similar): ---snip--- # rbd snap ls --all <pool>/<image> SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 23962608 42c2e4c3-e1ab-480f-9d29-aab5555c751b 20 GiB Tue Sep 20 20:33:18 2022 trash (.mirror.non_primary.61ef3a1e-6e5b-4147-ac62-552e4776dd70.ef345a74-2585-48ac-8e25-09b15c20c877) 96796437 24cc96d4-51c6-447e-870d-1545d8ec9308 20 GiB Tue Apr 18 12:15:28 2023 trash (.mirror.non_primary.61ef3a1e-6e5b-4147-ac62-552e4776dd70.a6ae46e4-cbb5-4264-b2e4-6b565820aa16) 110025019 .mirror.non_primary.61ef3a1e-6e5b-4147-ac62-552e4776dd70.0fc73e88-6671-4ba8-a118-8fcbcd422c22 20 GiB Mon May 15 16:27:39 2023 mirror (non-primary peer_uuids:[] 18ac37b9-a6c6-4f64-ba43-a1f0d02b3f96:115986586 copied) ---snip--- They're on latest Octopus and use all config defaults. There are more than 500 images in that pool with a snapshot schedule of 3 minutes for each images. A couple of questions arised and hopefully at least some of them can be answered: - Why are there snapshots in the trash namespace from September 2022, how can they be cleaned up and why aren't they cleaned up automatically? Not all images have trash entries though. - Could disabling mirroring for those images help getting rid of the trash, then enable mirroring again? - Is the default osd_max_trimming_pgs = 2 a bottleneck here? They have a week old sst files in the store.db, leading to long startup duration for MONs after reboot/restart. Would increasing that value help getting rid of the trash entries and maybe also trim the mon store? - Regarding the general snapshot mirroring procedure, the default of rbd_mirroring_max_mirroring_snapshots is 5, but I assume that the number of active snapshots would only grow in case of a disruption between those sites, correct? If the sync works there's no need to keep 5 snapshots, right? I'm still looking into these things myself but I'd appreciate anyone chiming in here. Thanks! Eugen

2 2

Telemetry and Redmine sync
by Yaarit Hatuka 15 May '23

15 May '23

Hi everyone, Over this weekend we will run a sync between telemetry crashes and Redmine tracker issues. This might affect your inbox, depending on your Redmine email notification setup. You can set up filters for these emails to skip your inbox. Thanks, Yaarit

2 1

cephadm and remoto package
by Shashi Dahal 15 May '23

15 May '23

Hi, I followed this documentation: https://docs.ceph.com/en/pacific/cephadm/adoption/ This is the error I get when trying to enable cephadm. ceph mgr module enable cephadm Error ENOENT: module 'cephadm' reports that it cannot run on the active manager daemon: loading remoto library:No module named 'remoto' (pass --force to force enablement) When I import remoto, it imports just fine. OS is ubuntu 20.04 focal -- Cheers, Shashi

1 0

Quincy Ceph-orchestrator and multipath SAS
by Deep Dish 15 May '23

15 May '23

Hello, I have a few hosts about to add into a cluster that have a multipath storage config for SAS devices. Is this supported on Quincy, and how would ceph-orchestrator and / or ceph-volume handle multipath storage? Here's a snip of lsblk output of a host in question: # lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS ... sdc 8:32 0 9.1T 0 disk └─mpathh 253:4 0 9.1T 0 mpath sdd 8:48 0 9.1T 0 disk └─mpathi 253:5 0 9.1T 0 mpath sde 8:64 0 7.3T 0 disk └─mpathj 253:6 0 7.3T 0 mpath sdf 8:80 0 7.3T 0 disk └─mpathl 253:7 0 7.3T 0 mpath sdg 8:96 0 7.2T 0 disk └─mpathk 253:8 0 7.2T 0 mpath sdh 8:112 0 7.3T 0 disk └─mpathe 253:9 0 7.3T 0 mpath sdi 8:128 0 7.3T 0 disk └─mpathg 253:10 0 7.3T 0 mpath sdj 8:144 0 7.3T 0 disk └─mpathf 253:11 0 7.3T 0 mpath sdk 8:160 0 7.3T 0 disk └─mpathc 253:12 0 7.3T 0 mpath sdl 8:176 0 7.3T 0 disk └─mpathd 253:13 0 7.3T 0 mpath ...

2 1

Octopus on Ubuntu 20.04.6 LTS with kernel 5
by Szabo, Istvan (Agoda) 15 May '23

15 May '23

Hi, In octopus documentation we can see kernel 4 as recommended, however we've changed our test cluster yesterday from centos 7 / 8 to Ubuntu 20.04.6 LTS with kernel 5.4.0-148 and seems working, I just want to make sure before I move to prod there isn't any caveats. Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

4 5

Upgrade Ceph cluster + radosgw from 14.2.18 to latest 15
by viplanghe6＠gmail.com 15 May '23

15 May '23

Hi, I want to upgrade my old Ceph cluster + Radosgw from v14 to v15. But I'm not using cephadm and I'm not sure how to limit errors as much as possible during the upgrade process? Here is my upgrade steps: Firstly, upgrade from 14.2.18 to 14.2.22 (latest nautilus version) Then, upgrade it from 14.2.22 to 15.2.17 (latest octopus version) service restart order: - restart monitors (sleep ~10) - restart managers (sleep ~5) - restart metadata OSDs (sleep ~30) - restart data OSDs (sleep ~30) - restart radosgw (sleep ~20) Is there anything wrong with these steps? Does 15.2.17 is the most stable version of Octopus? Thanks.

3 7

RGW: Using Lua script to monitor storage class put operations
by viplanghe6＠gmail.com 14 May '23

14 May '23

I have a lua script that read Storageclass header of any put request (as I understand): local function isempty(input) return input == nil or input == '' end if Request.RGWOp == 'put_obj' then RGWDebugLog("Put_Obj with StorageClass: " .. Request.HTTP.StorageClass ) end Then apply the script: radosgw-admin script put --infile=storage_class.lua --context=preRequest Then put an object using boto3: s3.Object(bucket, 'test-object').put(Body="0"*100, ACL='public-read', StorageClass='COLD') But I can't find any log about it in radosgw log? grep -i Lua /var/log/ceph/ceph-client.rgw.ceph1.log Anyone have any ideal? Thanks

2 1

Unable to Retrieve Usage Logs in Ceph RGW with Boto3
by Ramin Najjarbashi 13 May '23

13 May '23

Hi I am currently working on retrieving user usage and usage logs from Ceph RGW using Boto3. While I am able to successfully obtain user usage and all bucket usage with the following model and code: ```python params = { "start-date": "2023-05-13T00:00:00Z", "end-date": "2023-05-13T23:59:59Z", } print(json.dumps(s3_client.get_usage_stats(**params), indent=2)) ``` ```json { "version": 1.0, "merge": { "operations": { "GetUsageStats": { "name": "GetUsageStats", "http": { "method": "GET", "requestUri": "/?usage&start-date={start-date}&end-date={end-date}", "responseCode": 200 }, "input": { "shape": "GetUsageLogInput" }, "output": { "shape": "GetUsageStatsOutput" }, "documentationUrl": " https://docs.ceph.com/docs/master/radosgw/s3/serviceops#get-usage-stats", "documentation": "Get usage stats for the user" } }, "shapes": { "GetUsageLogInput": { "type": "structure", "members": { "start-date": { "shape": "Timestamp", "location": "uri", "documentation": "The start date and time for the log entries to retrieve." }, "end-date": { "shape": "Timestamp", "location": "uri", "documentation": "The end date and time for the log entries to retrieve." } }, "documentation": "Input shape for the GetUsageLog operation." }, "Timestamp": { "type": "string", "documentation": "A timestamp in ISO 8601 format." }, "GetUsageStatsOutput": { "type": "structure", "members": { "Summary": { "shape": "UsageStatsSummary", "documentation": "Summary of usage stats." }, "CapacityUsed": { "shape": "CapacityUsedUsage", "documentation": "Usage stats for capacity used." } } }, "CapacityUsedUsage": { "type": "structure", "members": { "User": { "shape": "UserUsageEntries", "documentation": "Usage stats for the user." } } }, "UserUsageEntries": { "type": "structure", "members": { "Buckets": { "shape": "BucketsUsageEntries", "documentation": "Usage stats for buckets." } } }, "BucketsUsageEntries": { "type": "structure", "members": { "Entry": { "shape": "UsageEntries", "documentation": "List of usage entries for buckets." } } }, "UsageEntries": { "type": "list", "member": { "shape": "BucketEntry" }, "documentation": "List of usage entries." }, "BucketEntry": { "type": "structure", "members": { "Bucket": { "shape": "String", "documentation": "Name of the bucket." }, "Bytes_Rounded": { "shape": "Bytes_Rounded", "documentation": "Number of bytes (rounded)." }, "Bytes": { "shape": "Bytes", "documentation": "Number of bytes." } } }, "UsageStatsSummary": { "type": "structure", "members": { "QuotaMaxBytes": { "shape": "QuotaMaxBytes", "documentation": "Maximum quota in bytes." }, "QuotaMaxBuckets": { "shape": "QuotaMaxBuckets", "documentation": "Maximum quota for buckets." }, "QuotaMaxObjCount": { "shape": "QuotaMaxObjCount", "documentation": "Maximum quota for object count." }, "QuotaMaxBytesPerBucket": { "shape": "QuotaMaxBytesPerBucket", "documentation": "Maximum quota in bytes per bucket." }, "QuotaMaxObjCountPerBucket": { "shape": "QuotaMaxObjCountPerBucket", "documentation": "Maximum quota for object count per bucket." }, "TotalBytes": { "shape": "TotalBytes", "documentation": "Total number of bytes." }, "TotalBytesRounded": { "shape": "TotalBytesRounded", "documentation": "Total number of bytes (rounded)." }, "TotalEntries": { "shape": "TotalEntries", "documentation": "Total number of entries." } } }, "BytesReceived": { "type": "integer", "documentation": "Number of bytes received." }, "QuotaMaxBytes": { "type": "integer", "documentation": "Maximum quota in bytes." }, "QuotaMaxBuckets": { "type": "integer", "documentation": "Maximum quota for buckets." }, "QuotaMaxObjCount": { "type": "integer", "documentation": "Maximum quota for object count." }, "QuotaMaxBytesPerBucket": { "type": "integer", "documentation": "Maximum quota in bytes per bucket." }, "QuotaMaxObjCountPerBucket": { "type": "integer", "documentation": "Maximum quota for object count per bucket." }, "TotalBytesRounded": { "type": "integer", "documentation": "Total number of bytes (rounded)." }, "TotalBytes": { "type": "integer", "documentation": "Total number of bytes." }, "TotalEntries": { "type": "integer", "documentation": "Total number of entries." }, "String": { "type": "string", "documentation": "A string value." }, "Bytes_Rounded": { "type": "integer", "documentation": "Number of bytes (rounded)." }, "Bytes": { "type": "integer", "documentation": "Number of bytes." } }, "documentation": "Model for retrieving usage statistics." } } ``` and this is my output: ``` "Summary": { "QuotaMaxBytes": -1, "QuotaMaxBuckets": 1000, "QuotaMaxObjCount": -1, "QuotaMaxBytesPerBucket": -1, "QuotaMaxObjCountPerBucket": -1, "TotalBytes": 52696, "TotalBytesRounded": 53248, "TotalEntries": 1 }, "CapacityUsed": { "User": { "Buckets": { "Entry": [ { "Bucket": "opa-test", "Bytes_Rounded": 53248, "Bytes": 52696 }, { "Bucket": "ramfrom", "Bytes_Rounded": 0, "Bytes": 0 } } } } } ``` The issue I am facing is that the usage logs are not being displayed. Upon investigating the code in the send_response() function of the RGWGetUsage_ObjStore_S3 class, I found the following conditions: ```cpp if (show_log_sum) { formatter->open_array_section("Summary"); map<string, rgw_usage_log_entry>::iterator siter; for (siter = summary_map.begin(); siter != summary_map.end(); ++siter) { const rgw_usage_log_entry& entry = siter->second; formatter->open_object_section("User"); formatter->dump_string("User", siter->first); dump_usage_categories_info(formatter, entry, &categories); rgw_usage_data total_usage; entry.sum(total_usage, categories); formatter->open_object_section("Total"); encode_json("BytesSent", total_usage.bytes_sent, formatter); encode_json("BytesReceived", total_usage.bytes_received, formatter); encode_json("Ops", total_usage.ops, formatter); encode_json("SuccessfulOps", total_usage.successful_ops, formatter); formatter->close_section(); // total formatter->close_section(); // user } ``` According to the code, the show_log_sum parameter must be true to display the summary log entries, and I have confirmed that it is indeed set to true. Additionally, the show_log_entries parameter must also be true to retrieve the usage log entries, and I have verified that it is set to true as well. I am using Ceph version 16.2.12, and I have set the RGW debug level to 99. Here is a snippet of the log output: ```log 2023-05-13T20:39:17.111+0330 7f346da3d700 1 ====== starting new request req= 0x7f3504a0e620 ===== 2023-05-13T20:39:17.147+0330 7f346da3d700 2 req 11854699226730900750 0.035999894s initializing for trans_id = tx00000a4845a234864ad0e-00645fc43d-fb27-default 2023-05-13T20:39:17.283+0330 7f346da3d700 10 req 11854699226730900750 0.171999499s rgw api priority: s3=8 s3website=7 2023-05-13T20:39:17.287+0330 7f346da3d700 10 req 11854699226730900750 0.175999492s host=s3.abrak.stage.test.com 2023-05-13T20:39:17.291+0330 7f346da3d700 20 req 11854699226730900750 0.179999471s subdomain= domain=s3.abrak.stage.test.com in_hosted_domain=1 in_hosted_domain_s3website=0 2023-05-13T20:39:17.291+0330 7f346da3d700 20 req 11854699226730900750 0.179999471s final domain/bucket subdomain= domain=s3.abrak.stage.test.com in_hosted_domain=1 in_hosted_domain_s3website=0 s->info.domain=s3.abrak. stage.test.com s->info.request_uri=/ 2023-05-13T20:39:17.315+0330 7f346da3d700 10 req 11854699226730900750 0.207999393s meta>> HTTP_X_AMZ_CONTENT_SHA256 2023-05-13T20:39:17.319+0330 7f346da3d700 10 req 11854699226730900750 0.207999393s meta>> HTTP_X_AMZ_DATE 2023-05-13T20:39:17.319+0330 7f346da3d700 10 req 11854699226730900750 0.207999393s x>> x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2023-05-13T20:39:17.319+0330 7f346da3d700 10 req 11854699226730900750 0.207999393s x>> x-amz-date:20230513T170916Z 2023-05-13T20:39:17.347+0330 7f346da3d700 10 req 11854699226730900750 0.235999316s name: usage val: 2023-05-13T20:39:17.347+0330 7f346da3d700 10 req 11854699226730900750 0.235999316s name: start-date val: 2023-05-13T00:00:00Z 2023-05-13T20:39:17.347+0330 7f346da3d700 10 req 11854699226730900750 0.235999316s name: end-date val: 2023-05-13T23:59:59Z 2023-05-13T20:39:17.355+0330 7f346da3d700 20 req 11854699226730900750 0.243999273s get_handler handler=26RGWHandler_REST_Service_S3 2023-05-13T20:39:17.383+0330 7f346da3d700 10 req 11854699226730900750 0.271999210s handler=26RGWHandler_REST_Service_S3 2023-05-13T20:39:17.383+0330 7f346da3d700 2 req 11854699226730900750 0.271999210s getting op 0 2023-05-13T20:39:17.423+0330 7f346da3d700 10 req 11854699226730900750 0.311999112s s3:get_self_usage scheduling with throttler client=3 cost=1 2023-05-13T20:39:17.431+0330 7f346da3d700 10 req 11854699226730900750 0.319999069s s3:get_self_usage op=23RGWGetUsage_ObjStore_S3 2023-05-13T20:39:17.431+0330 7f346da3d700 2 req 11854699226730900750 0.319999069s s3:get_self_usage verifying requester 2023-05-13T20:39:17.435+0330 7f346da3d700 20 req 11854699226730900750 0.323999047s s3:get_self_usage rgw::auth::StrategyRegistry::s3_main_strategy_t: trying rgw::auth::s3::AWSAuthStrategy 2023-05-13T20:39:17.447+0330 7f346da3d700 20 req 11854699226730900750 0.335999012s s3:get_self_usage rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3:: S3AnonymousEngine 2023-05-13T20:39:17.451+0330 7f346da3d700 20 req 11854699226730900750 0.339999020s s3:get_self_usage rgw::auth::s3::S3AnonymousEngine denied with reason=-1 2023-05-13T20:39:17.451+0330 7f346da3d700 20 req 11854699226730900750 0.339999020s s3:get_self_usage rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3:: LocalEngine 2023-05-13T20:39:17.455+0330 7f346da3d700 10 req 11854699226730900750 0.343998998s v4 signature format = 2e4a95626b3633439d13f5769d6a0be142c30823192837c3599fbb3f758d7c40 2023-05-13T20:39:17.463+0330 7f346da3d700 10 req 11854699226730900750 0.351998985s v4 credential format = OSDDH3XY9FPGHJ07O7JP/20230513/us-east-1/s3/ aws4_request 2023-05-13T20:39:17.463+0330 7f346da3d700 10 req 11854699226730900750 0.351998985s access key id = OSDDH3XY9FPGHJ07O7JP 2023-05-13T20:39:17.463+0330 7f346da3d700 10 req 11854699226730900750 0.351998985s credential scope = 20230513/us-east-1/s3/aws4_request 2023-05-13T20:39:17.471+0330 7f346da3d700 10 req 11854699226730900750 0.359998941s canonical headers format = host:s3.abrak.stage.test.com x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date:20230513T170916Z 2023-05-13T20:39:17.479+0330 7f346da3d700 10 req 11854699226730900750 0.367998898s payload request hash = e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2023-05-13T20:39:17.483+0330 7f346da3d700 10 req 11854699226730900750 0.371998906s canonical request = GET / end-date=2023-05-13T23%3A59%3A59Z&start-date=2023-05-13T00%3A00%3A00Z&usage= host:s3.abrak.stage.test.com x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 x-amz-date:20230513T170916Z host;x-amz-content-sha256;x-amz-date e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 2023-05-13T20:39:17.483+0330 7f346da3d700 10 req 11854699226730900750 0.371998906s canonical request hash = 060e8a249e9f3708393aa7d2cfde262976bdd4d10216be12ca74ae7961e3994f 2023-05-13T20:39:17.483+0330 7f346da3d700 10 req 11854699226730900750 0.371998906s string to sign = AWS4-HMAC-SHA256 20230513T170916Z 20230513/us-east-1/s3/aws4_request 060e8a249e9f3708393aa7d2cfde262976bdd4d10216be12ca74ae7961e3994f 2023-05-13T20:39:17.511+0330 7f346da3d700 10 req 11854699226730900750 0.399998844s date_k = fe25d3b269a34fa9c86caf0b35d6a46791ddd7fbcd0650a212fb7528a2dce06d 2023-05-13T20:39:17.511+0330 7f346da3d700 10 req 11854699226730900750 0.399998844s region_k = 03b05d2d23e0b150ae09d8baf554e8a8bc34887951f47b5925dead931c7faeaa 2023-05-13T20:39:17.511+0330 7f346da3d700 10 req 11854699226730900750 0.399998844s service_k = d10551f64ed49675cde1f1b609ee5b54b513b077e2c00198ade53b689dee17df 2023-05-13T20:39:17.511+0330 7f346da3d700 10 req 11854699226730900750 0.399998844s signing_k = c35cb1ca41d5cc1c5a49429c50afe3c840d0b67e81df0faf8122d979294098e7 2023-05-13T20:39:17.511+0330 7f346da3d700 10 req 11854699226730900750 0.399998844s generated signature = 2e4a95626b3633439d13f5769d6a0be142c30823192837c3599fbb3f758d7c40 2023-05-13T20:39:17.511+0330 7f346da3d700 15 req 11854699226730900750 0.399998844s s3:get_self_usage string_to_sign=AWS4-HMAC-SHA256 20230513T170916Z 20230513/us-east-1/s3/aws4_request 060e8a249e9f3708393aa7d2cfde262976bdd4d10216be12ca74ae7961e3994f 2023-05-13T20:39:17.511+0330 7f346da3d700 15 req 11854699226730900750 0.399998844s s3:get_self_usage server signature= 2e4a95626b3633439d13f5769d6a0be142c30823192837c3599fbb3f758d7c40 2023-05-13T20:39:17.511+0330 7f346da3d700 15 req 11854699226730900750 0.399998844s s3:get_self_usage client signature= 2e4a95626b3633439d13f5769d6a0be142c30823192837c3599fbb3f758d7c40 2023-05-13T20:39:17.511+0330 7f346da3d700 15 req 11854699226730900750 0.399998844s s3:get_self_usage compare=0 2023-05-13T20:39:17.519+0330 7f346da3d700 20 req 11854699226730900750 0.407998830s s3:get_self_usage rgw::auth::s3::LocalEngine granted access 2023-05-13T20:39:17.519+0330 7f346da3d700 20 req 11854699226730900750 0.407998830s s3:get_self_usage rgw::auth::s3::AWSAuthStrategy granted access 2023-05-13T20:39:17.519+0330 7f346da3d700 2 req 11854699226730900750 0.407998830s s3:get_self_usage normalizing buckets and tenants 2023-05-13T20:39:17.519+0330 7f346da3d700 10 req 11854699226730900750 0.407998830s s->object=<NULL> s->bucket= 2023-05-13T20:39:17.519+0330 7f346da3d700 2 req 11854699226730900750 0.407998830s s3:get_self_usage init permissions 2023-05-13T20:39:17.535+0330 7f346da3d700 20 req 11854699226730900750 0.423998743s s3:get_self_usage get_system_obj_state: rctx=0x7f3504a0ca28 obj=default.rgw. meta:users.uid:development state=0x56448667e0a0 s->prefetch_data=0 2023-05-13T20:39:17.535+0330 7f346da3d700 10 req 11854699226730900750 0.423998743s s3:get_self_usage cache get: name=default.rgw.meta+users.uid+development : hit (requested=0x6, cached=0x17) 2023-05-13T20:39:17.539+0330 7f346da3d700 20 req 11854699226730900750 0.427998751s s3:get_self_usage get_system_obj_state: s->obj_tag was set empty 2023-05-13T20:39:17.539+0330 7f346da3d700 20 req 11854699226730900750 0.427998751s s3:get_self_usage Read xattr: user.rgw.idtag 2023-05-13T20:39:17.539+0330 7f346da3d700 10 req 11854699226730900750 0.427998751s s3:get_self_usage cache get: name=default.rgw.meta+users.uid+development : hit (requested=0x3, cached=0x17) 2023-05-13T20:39:17.679+0330 7f346da3d700 2 req 11854699226730900750 0.567998350s s3:get_self_usage recalculating target 2023-05-13T20:39:17.679+0330 7f346da3d700 2 req 11854699226730900750 0.567998350s s3:get_self_usage reading permissions 2023-05-13T20:39:17.687+0330 7f346da3d700 2 req 11854699226730900750 0.575998306s s3:get_self_usage init op 2023-05-13T20:39:17.687+0330 7f346da3d700 2 req 11854699226730900750 0.575998306s s3:get_self_usage verifying op mask 2023-05-13T20:39:17.695+0330 7f346da3d700 20 req 11854699226730900750 0.583998263s s3:get_self_usage required_mask= 1 user.op_mask=7 2023-05-13T20:39:17.695+0330 7f346da3d700 2 req 11854699226730900750 0.583998263s s3:get_self_usage verifying op permissions 2023-05-13T20:39:17.695+0330 7f346da3d700 2 req 11854699226730900750 0.583998263s s3:get_self_usage verifying op params 2023-05-13T20:39:17.695+0330 7f346da3d700 2 req 11854699226730900750 0.583998263s s3:get_self_usage pre-executing 2023-05-13T20:39:17.695+0330 7f346da3d700 2 req 11854699226730900750 0.583998263s s3:get_self_usage executing 2023-05-13T20:39:21.023+0330 7f343c1da700 2 req 11854699226730900750 3.911988497s s3:get_self_usage completing 2023-05-13T20:39:21.067+0330 7f343c1da700 30 AccountingFilter::send_status: e=0, sent=17, total=0 2023-05-13T20:39:21.071+0330 7f343c1da700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2023-05-13T20:39:21.075+0330 7f343c1da700 30 AccountingFilter:: send_chunked_transfer_encoding: e=0, sent=28, total=0 2023-05-13T20:39:21.079+0330 7f343c1da700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2023-05-13T20:39:21.107+0330 7f343b1d8700 30 AccountingFilter:: complete_header: e=0, sent=156, total=0 2023-05-13T20:39:21.107+0330 7f343b1d8700 30 AccountingFilter::set_account: e=1 2023-05-13T20:39:21.199+0330 7f3457a11700 30 AccountingFilter::send_body: e= 1, sent=4051, total=0 2023-05-13T20:39:21.199+0330 7f3457a11700 30 AccountingFilter:: complete_request: e=1, sent=5, total=4051 2023-05-13T20:39:21.199+0330 7f3457a11700 30 req 11854699226730900750 4.087987900s log_usage: bucket_name= tenant=, bytes_sent=4056, bytes_received=0, success= 1 2023-05-13T20:39:21.199+0330 7f3457a11700 2 req 11854699226730900750 4.087987900s s3:get_self_usage op status=0 2023-05-13T20:39:21.199+0330 7f3457a11700 2 req 11854699226730900750 4.087987900s s3:get_self_usage http status=200 2023-05-13T20:39:21.219+0330 7f3457a11700 1 ====== req done req= 0x7f3504a0e620 op status=0 http_status=200 latency=4.103988171s ====== 2023-05-13T20:39:21.227+0330 7f3457a11700 1 beast: 0x7f3504a0e620: 192.168. 200.1 - development [13/May/2023:20:39:17.059 +0330] "GET /?usage&start-date=2023-05-13T00%3A00%3A00Z&end-date=2023-05-13T23%3A59%3A59Z HTTP/1.0" 200 4056 - "Boto3/1.26.131 Python/3.10.11 Darwin/22.4.0 Botocore/1.29.131" - latency=4.103988171s ``` I would greatly appreciate any guidance or insights into why I am unable to retrieve the usage logs. How can I ensure that the usage logs are included in the response? Thank you for your assistance! P.S. I have also verified the response using tools like Wireshark and tcpdump, and I can confirm that the responses received are consistent with the output I provided earlier.

1 0

MDS "newly corrupt dentry" after patch version upgrade
by Janek Bevendorff 12 May '23

12 May '23

Hi, After a patch version upgrade from 16.2.10 to 16.2.12, our rank 0 MDS fails start start. After replaying the journal, it just crashes with [ERR] : MDS abort because newly corrupt dentry to be committed: [dentry #0x1/storage [2,head] auth (dversion lock) Immediately after the upgrade, I had it running shortly, but then it decided to crash for unknown reasons and I cannot get it back up. We have five ranks in total, the other four seem to be fine. I backed up the journal and tried to run cephfs-journal-tool --rank=cephfs.storage:0 event recover_dentries summary, but it never finishes only eats up a lot of RAM. I stopped it after an hour and 50GB RAM. Resetting the journal makes the MDS crash with a missing inode error on another top-level directory, so I re-imported the backed-up journal. Is there any way to recover from this without rebuilding the whole file system? Thanks Janek Here's the full crash log: May 02 16:16:53 xxx077 ceph-mds[3047358]: -29> 2023-05-02T16:16:52.761+0200 7f51f878b700 1 mds.0.1711712 Finished replaying journal May 02 16:16:53 xxx077 ceph-mds[3047358]: -28> 2023-05-02T16:16:52.761+0200 7f51f878b700 1 mds.0.1711712 making mds journal writeable May 02 16:16:53 xxx077 ceph-mds[3047358]: -27> 2023-05-02T16:16:52.761+0200 7f51f878b700 1 mds.0.journaler.mdlog(ro) set_writeable May 02 16:16:53 xxx077 ceph-mds[3047358]: -26> 2023-05-02T16:16:52.761+0200 7f51f878b700 2 mds.0.1711712 i am not alone, moving to state resolve May 02 16:16:53 xxx077 ceph-mds[3047358]: -25> 2023-05-02T16:16:52.761+0200 7f51f878b700 3 mds.0.1711712 request_state up:resolve May 02 16:16:53 xxx077 ceph-mds[3047358]: -24> 2023-05-02T16:16:52.761+0200 7f51f878b700 5 mds.beacon.xxx077 set_want_state: up:replay -> up:resolve May 02 16:16:53 xxx077 ceph-mds[3047358]: -23> 2023-05-02T16:16:52.761+0200 7f51f878b700 5 mds.beacon.xxx077 Sending beacon up:resolve seq 15 May 02 16:16:53 xxx077 ceph-mds[3047358]: -22> 2023-05-02T16:16:52.761+0200 7f51f878b700 10 monclient: _send_mon_message to mon.xxx056 at v2:141.54.133.56:3300/0 May 02 16:16:53 xxx077 ceph-mds[3047358]: -21> 2023-05-02T16:16:53.113+0200 7f51fef98700 10 monclient: tick May 02 16:16:53 xxx077 ceph-mds[3047358]: -20> 2023-05-02T16:16:53.113+0200 7f51fef98700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2023-05-02T16:16:23.118186+0200) May 02 16:16:53 xxx077 ceph-mds[3047358]: -19> 2023-05-02T16:16:53.373+0200 7f51fff9a700 1 mds.xxx077 Updating MDS map to version 1711713 from mon.1 May 02 16:16:53 xxx077 ceph-mds[3047358]: -18> 2023-05-02T16:16:53.373+0200 7f51fff9a700 1 mds.0.1711712 handle_mds_map i am now mds.0.1711712 May 02 16:16:53 xxx077 ceph-mds[3047358]: -17> 2023-05-02T16:16:53.373+0200 7f51fff9a700 1 mds.0.1711712 handle_mds_map state change up:replay --> up:resolve May 02 16:16:53 xxx077 ceph-mds[3047358]: -16> 2023-05-02T16:16:53.373+0200 7f51fff9a700 1 mds.0.1711712 resolve_start May 02 16:16:53 xxx077 ceph-mds[3047358]: -15> 2023-05-02T16:16:53.373+0200 7f51fff9a700 1 mds.0.1711712 reopen_log May 02 16:16:53 xxx077 ceph-mds[3047358]: -14> 2023-05-02T16:16:53.373+0200 7f51fff9a700 1 mds.0.1711712 recovery set is 1,2,3,4 May 02 16:16:53 xxx077 ceph-mds[3047358]: -13> 2023-05-02T16:16:53.373+0200 7f51fff9a700 1 mds.0.1711712 recovery set is 1,2,3,4 May 02 16:16:53 xxx077 ceph-mds[3047358]: -12> 2023-05-02T16:16:53.373+0200 7f5202fa0700 10 monclient: get_auth_request con 0x5574fe74c400 auth_method 0 May 02 16:16:53 xxx077 ceph-mds[3047358]: -11> 2023-05-02T16:16:53.373+0200 7f52037a1700 10 monclient: get_auth_request con 0x5574fe40fc00 auth_method 0 May 02 16:16:53 xxx077 ceph-mds[3047358]: -10> 2023-05-02T16:16:53.373+0200 7f520279f700 10 monclient: get_auth_request con 0x5574f932fc00 auth_method 0 May 02 16:16:53 xxx077 ceph-mds[3047358]: -9> 2023-05-02T16:16:53.373+0200 7f520279f700 10 monclient: get_auth_request con 0x5574ffce2000 auth_method 0 May 02 16:16:53 xxx077 ceph-mds[3047358]: -8> 2023-05-02T16:16:53.377+0200 7f5202fa0700 5 mds.beacon.xxx077 received beacon reply up:resolve seq 15 rtt 0.616008 May 02 16:16:53 xxx077 ceph-mds[3047358]: -7> 2023-05-02T16:16:53.393+0200 7f51fff9a700 5 mds.xxx077 handle_mds_map old map epoch 1711713 <= 1711713, discarding May 02 16:16:53 xxx077 ceph-mds[3047358]: -6> 2023-05-02T16:16:53.393+0200 7f51fff9a700 5 mds.xxx077 handle_mds_map old map epoch 1711713 <= 1711713, discarding May 02 16:16:53 xxx077 ceph-mds[3047358]: -5> 2023-05-02T16:16:53.393+0200 7f51fff9a700 5 mds.xxx077 handle_mds_map old map epoch 1711713 <= 1711713, discarding May 02 16:16:53 xxx077 ceph-mds[3047358]: -4> 2023-05-02T16:16:53.393+0200 7f51fff9a700 5 mds.xxx077 handle_mds_map old map epoch 1711713 <= 1711713, discarding May 02 16:16:53 xxx077 ceph-mds[3047358]: -3> 2023-05-02T16:16:53.545+0200 7f51fff9a700 -1 mds.0.cache.den(0x1 storage) newly corrupt dentry to be committed: [dentry #0x1/storage [2,head] auth (dversion lock) v=78956500 ino=0x10000000000 state=1610612736 | inodepin=1 dirty=1 0x5574f932db80] May 02 16:16:53 xxx077 ceph-mds[3047358]: -2> 2023-05-02T16:16:53.545+0200 7f51fff9a700 -1 log_channel(cluster) log [ERR] : MDS abort because newly corrupt dentry to be committed: [dentry #0x1/storage [2,head] auth (dversion lock) v=78956500 ino=0x10000000000 state=1610612736 | inodepin=1 dirty=1 0x5574f932db80] May 02 16:16:53 xxx077 ceph-mds[3047358]: -1> 2023-05-02T16:16:53.549+0200 7f51fff9a700 -1 /build/ceph-16.2.12/src/mds/CDentry.cc: In function 'bool CDentry::check_corruption(bool)' thread 7f51fff9a700 time 2023-05-02T16:16:53.549536+0200 /build/ceph-16.2.12/src/mds/CDentry.cc: 697: ceph_abort_msg("abort() called") ceph version 16.2.12 (5a2d516ce4b134bfafc80c4274532ac0d56fc1e2) pacific (stable) 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe0) [0x7f52054c3495] 2: (CDentry::check_corruption(bool)+0x86b) [0x5574f7e3a91b] 3: (EMetaBlob::add_dir_context(CDir*, int)+0x507) [0x5574f7f9f9d7] 4: (MDCache::create_subtree_map()+0x13e1) [0x5574f7d20dc1] 5: (MDLog::_journal_segment_subtree_map(MDSContext*)+0x4d) [0x5574f7f2949d] 6: (MDLog::_submit_entry(LogEvent*, MDSLogContextBase*)+0x206) [0x5574f7f29866] 7: (MDCache::log_leader_commit(metareqid_t)+0x277) [0x5574f7ccc3c7] 8: (MDCache::finish_committed_leaders()+0x87) [0x5574f7ccd0d7] 9: (MDCache::maybe_resolve_finish()+0x78) [0x5574f7d38358] 10: (MDCache::handle_resolve(boost::intrusive_ptr<MMDSResolve const> const&)+0x1e02) [0x5574f7d454c2] 11: (MDCache::dispatch(boost::intrusive_ptr<Message const> const&)+0x144) [0x5574f7d47bd4] 12: (MDSRank::handle_message(boost::intrusive_ptr<Message const> const&)+0x733) [0x5574f7bac793] 13: (MDSRank::_dispatch(boost::intrusive_ptr<Message const> const&, bool)+0x643) [0x5574f7bcacb3] 14: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr<Message const> const&)+0x5c) [0x5574f7bcb34c] 15: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x1d6) [0x5574f7b9f226] 16: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0x460) [0x7f5205714020] 17: (DispatchQueue::entry()+0x58f) [0x7f52057118bf] 18: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f52057df261] 19: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f5205205609] 20: clone() May 02 16:16:53 xxx077 ceph-mds[3047358]: 0> 2023-05-02T16:16:53.553+0200 7f51fff9a700 -1 *** Caught signal (Aborted) ** in thread 7f51fff9a700 thread_name:ms_dispatch ceph version 16.2.12 (5a2d516ce4b134bfafc80c4274532ac0d56fc1e2) pacific (stable) 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f5205211420] 2: gsignal() 3: abort() 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1af) [0x7f52054c3564] 5: (CDentry::check_corruption(bool)+0x86b) [0x5574f7e3a91b] 6: (EMetaBlob::add_dir_context(CDir*, int)+0x507) [0x5574f7f9f9d7] 7: (MDCache::create_subtree_map()+0x13e1) [0x5574f7d20dc1] 8: (MDLog::_journal_segment_subtree_map(MDSContext*)+0x4d) [0x5574f7f2949d] 9: (MDLog::_submit_entry(LogEvent*, MDSLogContextBase*)+0x206) [0x5574f7f29866] 10: (MDCache::log_leader_commit(metareqid_t)+0x277) [0x5574f7ccc3c7] 11: (MDCache::finish_committed_leaders()+0x87) [0x5574f7ccd0d7] 12: (MDCache::maybe_resolve_finish()+0x78) [0x5574f7d38358] 13: (MDCache::handle_resolve(boost::intrusive_ptr<MMDSResolve const> const&)+0x1e02) [0x5574f7d454c2] 14: (MDCache::dispatch(boost::intrusive_ptr<Message const> const&)+0x144) [0x5574f7d47bd4] 15: (MDSRank::handle_message(boost::intrusive_ptr<Message const> const&)+0x733) [0x5574f7bac793] 16: (MDSRank::_dispatch(boost::intrusive_ptr<Message const> const&, bool)+0x643) [0x5574f7bcacb3] 17: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr<Message const> const&)+0x5c) [0x5574f7bcb34c] 18: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x1d6) [0x5574f7b9f226] 19: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0x460) [0x7f5205714020] 20: (DispatchQueue::entry()+0x58f) [0x7f52057118bf] 21: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f52057df261] 22: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f5205205609] 23: clone() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. May 02 16:16:53 xxx077 ceph-mds[3047358]: --- logging levels --- May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 5 none May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 1 lockdep May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 1 context May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 1 crush May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 mds May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 mds_balancer May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 mds_locker May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 mds_log May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 mds_log_expire May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 mds_migrator May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 1 buffer May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 1 timer May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 1 filer May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 1 striper May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 1 objecter May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 5 rados May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 5 rbd May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 5 rbd_mirror May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 5 rbd_replay May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 5 rbd_pwl May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 5 journaler May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 5 objectcacher May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 5 immutable_obj_cache May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 5 client May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 osd May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 5 optracker May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 5 objclass May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 3 filestore May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 3 journal May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 0 ms May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 mon May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/10 monc May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 paxos May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 5 tp May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 auth May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 crypto May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 1 finisher May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 1 reserver May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 heartbeatmap May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 perfcounter May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 rgw May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 rgw_sync May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/10 civetweb May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 javaclient May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 asok May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 1 throttle May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 0 refs May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 compressor May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 bluestore May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 bluefs May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 3 bdev May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 kstore May 02 16:16:53 xxx077 ceph-mds[3047358]: 4/ 5 rocksdb May 02 16:16:53 xxx077 ceph-mds[3047358]: 4/ 5 leveldb May 02 16:16:53 xxx077 ceph-mds[3047358]: 4/ 5 memdb May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 fuse May 02 16:16:53 xxx077 ceph-mds[3047358]: 2/ 5 mgr May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 mgrc May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 dpdk May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 eventtrace May 02 16:16:53 xxx077 ceph-mds[3047358]: 1/ 5 prioritycache May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 5 test May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 5 cephfs_mirror May 02 16:16:53 xxx077 ceph-mds[3047358]: 0/ 5 cephsqlite May 02 16:16:53 xxx077 ceph-mds[3047358]: 99/99 (syslog threshold) May 02 16:16:53 xxx077 ceph-mds[3047358]: -2/-2 (stderr threshold) May 02 16:16:53 xxx077 ceph-mds[3047358]: --- pthread ID / name mapping for recent threads --- May 02 16:16:53 xxx077 ceph-mds[3047358]: 139990037739264 / May 02 16:16:53 xxx077 ceph-mds[3047358]: 139990054524672 / May 02 16:16:53 xxx077 ceph-mds[3047358]: 139990062917376 / MR_Finisher May 02 16:16:53 xxx077 ceph-mds[3047358]: 139990079702784 / PQ_Finisher May 02 16:16:53 xxx077 ceph-mds[3047358]: 139990096488192 / ms_dispatch May 02 16:16:53 xxx077 ceph-mds[3047358]: 139990130059008 / ceph-mds May 02 16:16:53 xxx077 ceph-mds[3047358]: 139990146844416 / safe_timer May 02 16:16:53 xxx077 ceph-mds[3047358]: 139990163629824 / ms_dispatch May 02 16:16:53 xxx077 ceph-mds[3047358]: 139990180415232 / io_context_pool May 02 16:16:53 xxx077 ceph-mds[3047358]: 139990197200640 / admin_socket May 02 16:16:53 xxx077 ceph-mds[3047358]: 139990205593344 / msgr-worker-2 May 02 16:16:53 xxx077 ceph-mds[3047358]: 139990213986048 / msgr-worker-1 May 02 16:16:53 xxx077 ceph-mds[3047358]: 139990222378752 / msgr-worker-0 May 02 16:16:53 xxx077 ceph-mds[3047358]: 139990239524736 / ceph-mds May 02 16:16:53 xxx077 ceph-mds[3047358]: max_recent 10000 May 02 16:16:53 xxx077 ceph-mds[3047358]: max_new 10000 May 02 16:16:53 xxx077 ceph-mds[3047358]: log_file /var/lib/ceph/crash/2023-05-02T14:16:53.555508Z_0b05f5cb-130c-4979-95ca-1ba7f31cf7e5/log May 02 16:16:53 xxx077 ceph-mds[3047358]: --- end dump of recent events --- -- Bauhaus-Universität Weimar Bauhausstr. 9a, R308 99423 Weimar, Germany Phone: +49 3643 58 3577 www.webis.de

3 9