August 2024 - ceph-users

could not find secret_id
by xiaowenhao111 22 Apr '26

22 Apr '26

2 1

problem with mgr prometheus module
by Dario Graña 17 Mar '26

17 Mar '26

Hi all! I'm running ceph quincy 17.2.7 in a cluster. On monday I updated the OS to AlmaLinux 9.3 to 9.4, since then grafana shows "No Data" message in all ceph related fields but, for example, the nodes information is still fine (Host Detail Dashboard). I have redeployed the mgr service with cephadm, disabled and re-enabled mgr prometheus module , but nothing changed. Digging into the problem, I accessed the prometheus interface. When I access prometheus, and found this error[image: Screen Shot 2024-06-04 at 15.22.37.png] When I access the node shown as down, it reports 503 Service Unavailable No cached data available yet Traceback (most recent call last): File "/lib/python3.6/site-packages/cherrypy/_cprequest.py", line 638, in respond self._do_respond(path_info) File "/lib/python3.6/site-packages/cherrypy/_cprequest.py", line 697, in _do_respond response.body = self.handler() File "/lib/python3.6/site-packages/cherrypy/lib/encoding.py", line 219, in __call__ self.body = self.oldhandler(*args, **kwargs) File "/lib/python3.6/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__ return self.callable(*self.args, **self.kwargs) File "/usr/share/ceph/mgr/prometheus/module.py", line 1751, in metrics return self._metrics(_global_instance) File "/usr/share/ceph/mgr/prometheus/module.py", line 1762, in _metrics raise cherrypy.HTTPError(503, 'No cached data available yet') cherrypy._cperror.HTTPError: (503, 'No cached data available yet') I checked the mgr prometheus address and port [ceph: root@ceph-admin01 /]# ceph config get mgr mgr/prometheus/server_addr :: [ceph: root@ceph-admin01 /]# ceph config get mgr mgr/prometheus/server_port 9283 It seems to be ok. When I check the master manager node for the port, I found [root@ceph-hn01 ~]# netstat -natup | grep 9283 tcp6 0 0 :::9283 :::* LISTEN 2453/ceph-mgr tcp6 0 0 192.168.97.51:9283 192.168.97.60:36130 ESTABLISHED 2453/ceph-mgr I don't understand why it is showing as IPv6, the node doesn't have a dual stack. I also tried to use a newer version of the prometheus container image, the 1.6.0, but it keeps reporting the same, so I rolled it back to the original one. Has anyone experienced an issue like this? Where can I look for more information about it? Thanks in advance. Regards. -- Dario Graña PIC (Port d'Informació Científica) Campus UAB, Edificio D E-08193 Bellaterra, Barcelona http://www.pic.es Avis - Aviso - Legal Notice: http://legal.ifae.es

2 2

OpenSSL security update for Octopus container?
by Robert Sander 02 Oct '25

02 Oct '25

Hi, I am running a Ceph Octopus cluster in version 15.2.10 setup with the cephadm orchestrator and official Docker hub container images. Yesterday an important security fix was released for libssl and is already packaged for all major distributions. I tried to run "ceph orch upgrade check docker.io/ceph/ceph:v15" but it tells me that the containers do not need to be upgraded. How will this security fix of OpenSSL be deployed in a timely manner to users of the Ceph container images? Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 93818 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 1

RBD Mirror - Failed to unlink peer
by Scott Cairns 01 Aug '25

01 Aug '25

Hi, Following the introduction of an additional node to our Ceph cluster, we've started to see unlink errors when taking a rbd mirror snapshot. We've had RBD mirroring configured for over a year now and it's been working flawlessly, however after we created OSD's on a new node we've receiving the following error: librbd::mirror::snapshot::CreatePrimaryRequest: 0x7f60c80056f0 handle_unlink_peer: failed to unlink peer: (2) No such file or directory This seemed to appear on around 3 of 150 snapshots on the first night and over the weeks has progressed to almost every snapshot. What's odd, is that the snapshot appears to be taken without any issues and does mirror to the DR site - we can see the snapshot ID taken on the source side is mirrored to the destination side when checking the rbd snap ls, and we've tested promoting an image on the DR site to ensure the snapshot does include up to date data, which it does. I can't see any other errors generated when the snapshot is taken to identify what file/directory isn't found - everything appears to be working okay it's just generating an error during the snapshot. I've also tried disabling mirroring on the disk and re-enabling however it doesn't appear to make any difference - there's no error on the initial mirror image, or the first snapshot taken after that, but every subsequent snapshot shows the error again. Any ideas? Thanks, Scott ________________________________ The content of this e-mail and any attachment is confidential and intended solely for the use of the individual to whom it is addressed. Any views or opinions presented are solely those of the author and do not necessarily represent those of Tecnica Limited. If you have received this e-mail in error please notify the sender. Any use, dissemination, forwarding, printing, or copying of this e-mail or any attachments thereto, in whole or part, without permission is strictly prohibited. Tecnica Limited Registered office: 5 Castle Court, Carnegie Campus, Dunfermline, Fife, KY11 8PB. Registered in Scotland No. SC250307. VAT No. 827 5110 42. This footnote also confirms that this email message has been swept for the presence of computer viruses.

4 8

Patching Ceph cluster
by Michael Worsham 25 Apr '25

25 Apr '25

What is the proper way to patch a Ceph cluster and reboot the servers in said cluster if a reboot is necessary for said updates? And is it possible to automate it via Ansible? This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments.

6 14

Migrating from S3 to Ceph RGW (Cloud Sync Module)
by James McClune 10 Apr '25

10 Apr '25

Hello Ceph User Community, I currently have a large Amazon S3 environment with terabytes of data spread over dozens of buckets. I'm looking to migrate from Amazon S3 to an on-site Ceph cluster using the RGW. I'm trying to figure out the most efficient way to achieve this. Looking through the documentation, I found articles related to the cloud sync module, released in Mimic ( https://docs.ceph.com/en/latest/radosgw/cloud-sync-module/) I also watched a video on the cloud sync module as well. It *sounds* like this is the functionality I'm looking for. Given I'm moving away from Amazon S3, I'm really just looking for a one-way replication between the buckets (i.e. Provide an Amazon S3 access key/secret which is associated to the buckets and the same for the Ceph environment, so object data can be replicated one-to-one, without creating ad-hoc tooling). Once the data is replicated from S3 to Ceph, I plan on modifying my boto connection objects to use the new Ceph environment. Is what I'm describing feasible with the cloud sync module? Just looking for some affirmation, given I'm not well versed in Ceph's RGW, especially around multi-site configurations. Thanks, Jimmy

7 8

Problem while upgrade 17.2.6 to 17.2.7
by Jean-Marc FONTANA 21 Feb '25

21 Feb '25

Hello everyone, We operate two clusters that we installed with ceph-deploy in Nautilus version on Debian 10. We use them for external S3 storage (owncloud) and rbd disk images.We had them upgraded to Octopus and Pacific versions on Debian 11 and recently converted them to cephadm and upgraded to Quincy (17.2.6). As we now have the orchestrator, we tried updating to 17.2.7 using the command# ceph orch upgrade start --image quay.io/ceph/ceph:v17.2.7 Everything went well, both clusters work perfectly for our use, except that the Rados gateway configuration is no longer accessible from the dashboard with the following error messageError connecting to Object Gateway: RGW REST API failed request with status code 404. We tried a few solutions found on the internet (reset rgw credentials, restart rgw adnd mgr, reenable dashboard, ...), unsuccessfully. Does somebody have an idea ? Best regards, Jean-Marc Fontana

5 9

RGW Lifecycle Problem (Reef)
by Chris Palmer 19 Feb '25

19 Feb '25

Reef 18.2.2, package install on Centos 9. This is a very straightforward production cluster, 2 RGW hosts, no multisite. 4 buckets have lifecycle policies: $ radosgw-admin lc list [ { "bucket": ":aaaaa:1d84db34-ed2b-400a-842e-344cdfa3deed.261076466.1", "shard": "lc.3", "started": "Mon, 15 Jul 2024 00:00:00 GMT", "status": "COMPLETE" }, { "bucket": ":bbbbb:1d84db34-ed2b-400a-842e-344cdfa3deed.304826873.1", "shard": "lc.3", "started": "Mon, 15 Jul 2024 00:00:00 GMT", "status": "COMPLETE" }, { "bucket": ":cccc:1d84db34-ed2b-400a-842e-344cdfa3deed.828249990.3", "shard": "lc.29", "started": "Thu, 01 Jan 1970 00:00:00 GMT", "status": "COMPLETE" }, { "bucket": ":dddd:1d84db34-ed2b-400a-842e-344cdfa3deed.547988508.2", "shard": "lc.29", "started": "Thu, 01 Jan 1970 00:00:00 GMT", "status": "UNINITIAL" } ] Symptoms are: - aaaaa & bbbbb work & list above as expected - ccccc works, but the "started" date is never updated - ddddd never fires, unless run manually (the listing above is 1 week after the policy was removed and cleanly reapplied, but with no manual run) The first three buckets are small, but ddddd is larger (6.3 TB, 3.2 M RGW objects). The policy for ddddd transitions objects to a different storage class. However this never happens unless I run the lifecycle manually. Then it works for that manual run, but never fires automatically each night. IIRC after a manual run the status updates to something else but I can't remember what, but I think the started date remains unset. Each night the following log entries can be seen in the ceph-client.rgw log on both RGW hosts: 2024-07-15T00:00:00.162+0000 7f1e44715640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.11 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:00.163+0000 7f1e44715640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:00.248+0000 7f1e46719640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.18 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:00.248+0000 7f1e46719640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:00.312+0000 7f1e46719640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.3 head last stored at Mon Jul 15 00:00:00 2024 2024-07-15T00:00:00.386+0000 7f1e42711640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.3 head last stored at Mon Jul 15 00:00:00 2024 2024-07-15T00:00:00.399+0000 7f1e42711640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.18 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:00.399+0000 7f1e42711640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:00.420+0000 7f1e42711640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.11 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:00.421+0000 7f1e42711640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:00.435+0000 7f1e42711640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.29 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:00.435+0000 7f1e42711640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:02.382+0000 7f1e46719640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.11 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:02.382+0000 7f1e46719640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:02.392+0000 7f1e46719640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.29 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:02.392+0000 7f1e46719640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:02.718+0000 7f1e44715640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.3 head last stored at Mon Jul 15 00:00:00 2024 2024-07-15T00:00:02.718+0000 7f1e44715640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:02.728+0000 7f1e44715640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.29 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:02.728+0000 7f1e44715640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:02.791+0000 7f1e44715640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.18 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:02.791+0000 7f1e44715640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 In case it is relevant: for the RGW-related pools, index is R3, data is EC2+2 and the pool for the storage class we've moving objects to is EC4+2. Any ideas how to proceed with diagnosis/fix? Thanks, Chris

1 2

purging already destroyed OSD leads to degraded and misplaced objects?
by tobias tempel 10 Jan '25

10 Jan '25

Dear Cephers, reorganizing one of our clusters i'm removing some hosts from it, taking "out" all OSDs on these hosts and waiting until all PGs are fine. After stopping and destroying all OSDs on one host i notice, that "purge" of such destroyed OSDs temporarily leads to degraded and misplaced objects (tried "reweight 0" as well, same picture) . Why that? I would - due to my limited experience - expect, that completely removing an already destroyed OSD could not have any effect on object placement at all. This effect yet seems not to be a real problem, as recovery works fine ... but perhaps it indicates some other issue. All of that takes place at Pacific 16.2.14 AlmaLinux 8.9, yes i know, there's some work to do. Perhaps you can give me a hint, where to look, for an explanation? Thank you cheers, toBias -- Tobias Tempel Deutsches Elektronen-Synchrotron - IT Notkestr. 85, 22607 Hamburg, Germany email: tobias.tempel(a)desy.de

4 7

Problems with autoscaler (overlapping roots) after changing the pool class
by Massimo Sgaravatto 06 Jan '25

06 Jan '25

Dear all I have just changed the crush rule for all the replicated pools in the following way: ceph osd crush rule create-replicated replicated_hdd default host hdd ceph osd pool set <poolname> crush_rule replicated_hdd See also this [*] thread Before applying this change, these pools were all using the replicated_ruleset rule where the class is not specified. I am noticing now a problem with the autoscaler: "ceph osd pool autoscale-status" doesn't report any output and the mgr log complains about overlapping roots: [pg_autoscaler ERROR root] pool xyz has overlapping roots: {-18, -1} Indeed: # ceph osd crush tree --show-shadow ID CLASS WEIGHT TYPE NAME -18 hdd 1329.26501 root default~hdd -17 hdd 329.14154 rack Rack11-PianoAlto~hdd -15 hdd 54.56085 host ceph-osd-04~hdd 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... ... -1 1329.26501 root default -7 329.14154 rack Rack11-PianoAlto -8 54.56085 host ceph-osd-04 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... I have already read about this behavior but I have no clear ideas how to fix the problem. I read somewhere that the problem happens when there are rules that force some pools to only use one class and there are also pools which does not make any distinction between device classes All the replicated pools are using the replicated_hdd pool but I also have some EC pools which are using a profile where the class is not specified. As far I understand, I can't force these pools to use only the hdd class: according to the doc I can't change this profile specifying the hdd class (or at least the change wouldn't be applied to the existing EC pools) Any suggestions ? The crush map is available at https://cernbox.cern.ch/s/gIyjbQbmoTFHCrr, if you want to have a look Many thanks, Massimo [*] https://www.mail-archive.com/ceph-users@ceph.io/msg18534.html

4 9

multifs and snapshots
by Dmitry Melekhov 12 Nov '24

12 Nov '24

Hello! There is note here https://docs.ceph.com/en/reef/dev/cephfs-snapshots/ about multiply filesystems and snapshots: If each FS gets its own pool things probably work, but this isn’t tested and may not be true. Is it still untested ? Thank you!

3 4

Help needed to configure erasure coding LRC plugin
by Michel Jouvin 16 Oct '24

16 Oct '24

Hi, As discussed in another thread (Crushmap rule for multi-datacenter erasure coding), I'm trying to create an EC pool spanning 3 datacenters (datacenters are present in the crushmap), with the objective to be resilient to 1 DC down, at least keeping the readonly access to the pool and if possible the read-write access, and have a storage efficiency better than 3 replica (let say a storage overhead <= 2). In the discussion, somebody mentioned LRC plugin as a possible jerasure alternative to implement this without tweaking the crushmap rule to implement the 2-step OSD allocation. I looked at the documentation (https://docs.ceph.com/en/latest/rados/operations/erasure-code-lrc/) but I have some questions if someone has experience/expertise with this LRC plugin. I tried to create a rule for using 5 OSDs per datacenter (15 in total), with 3 (9 in total) being data chunks and others being coding chunks. For this, based of my understanding of examples, I used k=9, m=3, l=4. Is it right? Is this configuration equivalent, in terms of redundancy, to a jerasure configuration with k=9, m=6? The resulting rule, which looks correct to me, is: -------- { "rule_id": 6, "rule_name": "test_lrc_2", "ruleset": 6, "type": 3, "min_size": 3, "max_size": 15, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -4, "item_name": "default~hdd" }, { "op": "choose_indep", "num": 3, "type": "datacenter" }, { "op": "chooseleaf_indep", "num": 5, "type": "host" }, { "op": "emit" } ] } ------------ Unfortunately, it doesn't work as expected: a pool created with this rule ends up with its pages active+undersize, which is unexpected for me. Looking at 'ceph health detail` output, I see for each page something like: pg 52.14 is stuck undersized for 27m, current state active+undersized, last acting [90,113,2147483647,103,64,147,164,177,2147483647,133,58,28,8,32,2147483647] For each PG, there is 3 '2147483647' entries and I guess it is the reason of the problem. What are these entries about? Clearly it is not OSD entries... Looks like a negative number, -1, which in terms of crushmap ID is the crushmap root (named "default" in our configuration). Any trivial mistake I would have made? Thanks in advance for any help or for sharing any successful configuration? Best regards, Michel

4 24

rgw connection resets
by Nathan Gleason 10 Oct '24

10 Oct '24

Hello, I’ve noticed what I think may be strange behavior from some rgw transfers. It only seems to be with larger multipart mp4 files. Once a connection comes into rgw it is accepted and processed, data begins to transfer, then I see "write_data failed: Connection reset by peer” and "s3:get_obj iterate_obj() failed with -104”. The client then reconnects and the transfer seems to be ok. This doesn’t appear to happen just downloading a regular file. This is happening to both my production and test cluster. I’m not seeing errors anywhere else, the network looks good and the clusters are healthy. I’m in the process of collecting some tcpdumps but figured I would ask if anyone knows why this is occurring. We run haproxy in front of the rgw. For troubleshooting purposes we have a basic haproxy config to just one rgw. We have tested with multiple browsers. Ceph 17.2.5 - cluster built and managed with cephadm Ubuntu 20.04.6 LTS Log: 2024-01-03T17:40:23.916+0000 7f646033d700 1 ====== starting new request req=0x7f640ca95730 ===== 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s initializing for trans_id = tx000003ec1e4b4dc157efd-0065959c07-104216-default 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s rgw api priority: s3=8 s3website=7 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s host=a2-west.usacld.net 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s final domain/bucket subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 s->info.domain= s->info.request_uri=/n athan:tshoot/testfile.mp4 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s get_handler handler=22RGWHandler_REST_Obj_S3 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s handler=22RGWHandler_REST_Obj_S3 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s getting op 0 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s get_system_obj_state: rctx=0x7f640ca947c0 obj=default.rgw.log:script.prerequest. state=0x55fa31f882e0 s->prefetch_data=0 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s cache get: name=default.rgw.log++script.prerequest. : hit (negative entry) 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj scheduling with throttler client=2 cost=1 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj op=21RGWGetObj_ObjStore_S3 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj verifying requester 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rgw::auth::StrategyRegistry::s3_main_strategy_t: trying rgw::auth::s3::AWSAuthStrategy 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::S3AnonymousEngine 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rgw::auth::s3::S3AnonymousEngine granted access 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rgw::auth::s3::AWSAuthStrategy granted access 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj normalizing buckets and tenants 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s->object=testfile.mp4 s->bucket=nathan/tshoot 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj init permissions 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_system_obj_state: rctx=0x7f640ca94200 obj=default.rgw.meta:root:nathan/tshoot state=0x55fa31f882e0 s->prefetch _data=0 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj cache get: name=default.rgw.meta+root+nathan/tshoot : hit (requested=0x16, cached=0x17) 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_system_obj_state: s->obj_tag was set empty 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj cache get: name=default.rgw.meta+root+nathan/tshoot : hit (requested=0x11, cached=0x17) 2024-01-03T17:40:23.916+0000 7f646033d700 15 req 4522146966294331133 0.000000000s s3:get_obj decode_policy Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner> <ID>nathan$testuser</ID><DisplayName>Nathan API User</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>nathan$testuser </ID><DisplayName>Nathan API User</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy> 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWSI_User_RADOS::read_user_info(): anonymous user 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj recalculating target 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj reading permissions 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_obj_state: rctx=0x7f640ca94ad0 obj=tshoot:testfile.mp4 state=0x55fa31082de8 s->prefetch_data=1 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj WARNING: blocking librados call 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj manifest: total_size = 88811314 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_obj_state: setting s->obj_tag to _0OUNxgsSsqTAs9Z6xh1SOKpJZLV_Vos 2024-01-03T17:40:23.916+0000 7f646033d700 15 req 4522146966294331133 0.000000000s s3:get_obj decode_policy Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner> <ID>nathan$testuser</ID><DisplayName>Nathan API User</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="Group"><URI>http://acs.amazonaws.c om/groups/global/AllUsers</URI></Grantee><Permission>READ</Permission></Grant><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>nathan$testuser</ID><DisplayNa me>Nathan API User</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy> 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj init op 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_system_obj_state: rctx=0x7f640ca94488 obj=default.rgw.meta:users.uid:nathan$testuser state=0x55fa31f882e0 s->p refetch_data=0 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj cache get: name=default.rgw.meta+users.uid+nathan$testuser : hit (requested=0x16, cached=0x17) 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_system_obj_state: s->obj_tag was set empty 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr: user.rgw.idtag 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj cache get: name=default.rgw.meta+users.uid+nathan$testuser : hit (requested=0x13, cached=0x17) 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj verifying op mask 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj required_mask= 1 user.op_mask=7 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj verifying op permissions 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj -- Getting permissions begin with perm_mask=49 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj Searching permissions for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=anonymous, acct_n ame=, subuser=, perm_mask=15, is_admin=0) mask=49 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj Searching permissions for uid=anonymous 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj Permissions for user not found 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj Searching permissions for group=1 mask=49 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj Found permission: 1 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj -- Getting permissions done for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=anonymous, acct_name=, subuser=, perm_mask=15, is_admin=0), owner=nathan$testuser, perm=1 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=anonymous, acct_name=, subuser=, perm_mask =15, is_admin=0) requested perm (type)=1, policy perm=1, user_perm_mask=15, acl perm=1 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj verifying op params 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj pre-executing 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj check rate limiting 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj executing 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_obj_state: rctx=0x7f640ca94ad0 obj=tshoot:testfile.mp4 state=0x55fa31082de8 s->prefetch_data=1 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.acl 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.content_type 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.etag 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.idtag 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.manifest 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.pg_ver 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.source_zone 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.tail_tag 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.x-amz-content-sha256 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.x-amz-date 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.x-amz-meta-s3cmd-attrs 2024-01-03T17:40:23.916+0000 7f646033d700 15 req 4522146966294331133 0.000000000s Encryption mode: 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_obj_state: rctx=0x7f640ca94ad0 obj=tshoot:testfile.mp4 state=0x55fa31082de8 s->prefetch_data=1 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__multipart_testfile.mp4.2~XSiTvBBtaU4 134dQCusv3kMP7YjDBk7.1 obj-ofs=0 read_ofs=0 len=4194304 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=4194304 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): result: ofs=4194304 stripe_ofs=4194304 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134 dQCusv3kMP7YjDBk7.1_1 obj-ofs=4194304 read_ofs=0 len=4194304 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=8388608 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): result: ofs=8388608 stripe_ofs=8388608 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134 dQCusv3kMP7YjDBk7.1_2 obj-ofs=8388608 read_ofs=0 len=4194304 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=12582912 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): result: ofs=12582912 stripe_ofs=12582912 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134 dQCusv3kMP7YjDBk7.1_3 obj-ofs=12582912 read_ofs=0 len=3145728 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=16777216 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): result: ofs=15728640 stripe_ofs=15728640 part_ofs=15728640 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__multipart_testfile.mp4.2~XSiTvBBtaU4 134dQCusv3kMP7YjDBk7.2 obj-ofs=15728640 read_ofs=0 len=4194304 2024-01-03T17:40:23.924+0000 7f6442301700 20 req 4522146966294331133 0.007999974s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.924+0000 7f6442301700 20 req 4522146966294331133 0.007999974s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=19922944 part_ofs=15728640 rule->part_size=15728640 2024-01-03T17:40:23.924+0000 7f6442301700 20 req 4522146966294331133 0.007999974s s3:get_obj RGWObjManifest::operator++(): result: ofs=19922944 stripe_ofs=19922944 part_ofs=15728640 rule->part_size=15728640 2024-01-03T17:40:23.924+0000 7f6442301700 20 req 4522146966294331133 0.007999974s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134 dQCusv3kMP7YjDBk7.2_1 obj-ofs=19922944 read_ofs=0 len=4194304 2024-01-03T17:40:23.932+0000 7f643d2f7700 20 req 4522146966294331133 0.015999949s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.932+0000 7f643d2f7700 20 req 4522146966294331133 0.015999949s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=24117248 part_ofs=15728640 rule->part_size=15728640 2024-01-03T17:40:23.932+0000 7f643d2f7700 20 req 4522146966294331133 0.015999949s s3:get_obj RGWObjManifest::operator++(): result: ofs=24117248 stripe_ofs=24117248 part_ofs=15728640 rule->part_size=15728640 2024-01-03T17:40:23.932+0000 7f643d2f7700 20 req 4522146966294331133 0.015999949s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134 dQCusv3kMP7YjDBk7.2_2 obj-ofs=24117248 read_ofs=0 len=4194304 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_status: e=0, sent=17, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_content_length: e=0, sent=26, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::complete_header: e=0, sent=460, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::set_account: e=1 2024-01-03T17:40:24.068+0000 7f642a2d1700 4 write_data failed: Connection reset by peer 2024-01-03T17:40:24.068+0000 7f642a2d1700 0 req 4522146966294331133 0.151999518s s3:get_obj iterate_obj() failed with -104 2024-01-03T17:40:24.156+0000 7f642eada700 2 req 4522146966294331133 0.239999235s s3:get_obj completing 2024-01-03T17:40:24.156+0000 7f642eada700 20 req 4522146966294331133 0.239999235s get_system_obj_state: rctx=0x7f640ca947c0 obj=default.rgw.log:script.postrequest.nathan state=0x55fa32424760 s->prefetch_data =0 2024-01-03T17:40:24.156+0000 7f642eada700 10 req 4522146966294331133 0.239999235s cache get: name=default.rgw.log++script.postrequest.nathan : hit (negative entry) 2024-01-03T17:40:24.156+0000 7f642eada700 30 AccountingFilter::complete_request: e=1, sent=0, total=0 2024-01-03T17:40:24.156+0000 7f642eada700 2 req 4522146966294331133 0.239999235s s3:get_obj op status=-104 2024-01-03T17:40:24.156+0000 7f642eada700 2 req 4522146966294331133 0.239999235s s3:get_obj http status=200 2024-01-03T17:40:24.156+0000 7f642eada700 1 ====== req done req=0x7f640ca95730 op status=-104 http_status=200 latency=0.239999235s ====== 2024-01-03T17:40:24.156+0000 7f642eada700 1 beast: 0x7f640ca95730: 10.30.0.196 - anonymous [03/Jan/2024:17:40:23.916 +0000] "GET /nathan:tshoot/testfile.mp4 HTTP/1.1" 200 0 - "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" - latency=0.239999235s 2024-01-03T17:40:24.156+0000 7f642eada700 20 failed to read header: end of stream 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_ACCEPT=*/* 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_ACCEPT_ENCODING=identity;q=1, *;q=0 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_ACCEPT_LANGUAGE=en-US,en;q=0.7 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_HOST=a2-west.usacld.net 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_RANGE=bytes=0- 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_REFERER=https://a2-west.usacld.net/nathan:tshoot/testfile.mp4 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_CH_UA="Not_A Brand";v="8", "Chromium";v="120", "Brave";v="120" 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_CH_UA_MOBILE=?0 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_CH_UA_PLATFORM="macOS" 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_FETCH_DEST=video 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_FETCH_MODE=no-cors 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_FETCH_SITE=same-origin 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_GPC=1 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_VERSION=1.1 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_X_FORWARDED_PORT=443 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_X_FORWARDED_PROTO=https 2024-01-03T17:40:24.160+0000 7f6503483700 20 REMOTE_ADDR=10.30.0.196 2024-01-03T17:40:24.160+0000 7f6503483700 20 REQUEST_METHOD=GET 2024-01-03T17:40:24.160+0000 7f6503483700 20 REQUEST_URI=/nathan:tshoot/testfile.mp4 2024-01-03T17:40:24.160+0000 7f6503483700 20 SCRIPT_URI=/nathan:tshoot/testfile.mp4 2024-01-03T17:40:24.160+0000 7f6503483700 20 SERVER_PORT=8000 2024-01-03T17:40:24.160+0000 7f6503483700 1 ====== starting new request req=0x7f640ca95730 ===== Thanks, Nathan

2 1

Not all Bucket Shards being used
by Christian Kugler 01 Oct '24

01 Oct '24

Hi, I have trouble with large OMAP files in a cluster in the RGW index pool. Some background information about the cluster: There is CephFS and RBD usage on the main cluster but for this issue I think only S3 is interesting. There is one realm, one zonegroup with two zones which have a bidirectional sync set up. Since this does not allow for autoresharding we have to do it by hand in this cluster – looking forward to Reef! From the logs: cluster 2023-07-17T22:59:03.018722+0000 osd.75 (osd.75) 623978 : cluster [WRN] Large omap object found. Object: 34:bcec3016:::.dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.5:head PG: 34.680c373d (34.5) Key count: 962091 Size (bytes): 277963182 The offending bucket looks like this: # radosgw-admin bucket stats \ | jq '.[] | select(.marker =="3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9") |"\(.num_shards) \(.usage["rgw.main"].num_objects)"' -r 131 9463833 Last week the number of objects was about 12 million. Which is why I reshareded the offending bucket twice, I think. Once to 129 and the second time to 131 because I wanted some leeway (or lieway? scnr, Sage). Unfortunately, even after a week the objects were still to big (the log line above is quite recent), so I looked into it again. # rados -p raum.rgw.buckets.index ls \ |grep .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9 \ |sort -V .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.0 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.1 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.2 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.3 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.4 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.5 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.6 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.7 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.8 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.9 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.10 # rados -p raum.rgw.buckets.index ls \ |grep .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9 \ |sort -V \ |xargs -IOMAP sh -c \ 'rados -p raum.rgw.buckets.index listomapkeys OMAP | wc -l' 1013854 1011007 1012287 1011232 1013565 998262 1012777 1012713 1012230 1010690 997111 Apparently, only 11 shards are in use. This would explain why the "Key usage" (from the log line) is about ten times higher than I would expect. How can I deal with this issue? One thing I could try to fix this would be to reshard to a lower number, but I am not sure if there are any risks associated with "downsharding". After that I could reshard to something like 97. Or I could directly "downshard" to 97. Also, the second zone has a similar problem, but as the error messsage lets me know, this would be a bad idea. Will it just take more time until the sharding is transferred to the seconds zone? Best, Christian Kugler

2 5

device_health_metrics pool automatically recreated
by Eugen Block 29 Sep '24

29 Sep '24

Hi, I just looked into one customer cluster that we upgraded some time ago from Octopus to Quincy (17.2.6) and I'm wondering why there are still both pools, "device_health_metrics" and ".mgr". According to the docs [0], it's supposed to be renamed: > Prior to Quincy, the devicehealth module created a > device_health_metrics pool to store device SMART statistics. With > Quincy, this pool is automatically renamed to be the common manager > module pool. Now only .mgr has data while device_health_metrics is empty, but it has a newer ID: ses01:~ # ceph df | grep -E "device_health|.mgr" .mgr 1 1 68 MiB 18 204 MiB 0 254 TiB device_health_metrics 15 1 0 B 0 0 B 0 254 TiB On a test cluster (meanwhile upgraded to latest Reef) I see the same: ceph01:~ # ceph df | grep -E "device_health_metrics|.mgr" .mgr 38 1 577 KiB 2 1.7 MiB 0 71 GiB device_health_metrics 45 1 0 B 0 0 B 0 71 GiB Since there are still many users who haven't upgraded to >= Quincy yet, this should be clarified/fixed. I briefly checked tracker.ceph.com, but didn't find anything related to this. I'm currently trying to reproduce it on a one-node test cluster which I upgraded from Pacific to Quincy, but no results yet, only that the renaming was successful. But for the other clusters I don't have enough logs to find out how/why the device_health_metrics pool had been recreated. Thanks, Eugen [0] https://docs.ceph.com/en/latest/mgr/administrator/#module-pool

2 3

Old MDS container version when: Ceph orch apply mds
by opositorvlc＠mail.com 28 Sep '24

28 Sep '24

Hi All, I migrated my CEPH 18.2.2 cluster from a non cephadm configuration. All goes fine except MDS service was deployed in a old version: 17.0.0 I'm trying to deploy MDS daemons using ceph orch but CEPH always download an old MDS image from docker. How could I deploy the MDS service in the same 18.2.2 version that the rest of services? [root@master1 ~]# ceph orch apply mds datafs --placement="2 master1 master2" [root@master1 ~]# ceph orch ps NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mds.datafs.master1.gcpovr master1 running (36m) 6m ago 36m 37.2M - 17.0.0-7183-g54142666 75e3d7089cea 96682779c7ad mds.datafs.master2.oqaxuy master2 running (36m) 6m ago 36m 33.1M - 17.0.0-7183-g54142666 75e3d7089cea a9a647f87c83 mgr.master master1 running (16h) 6m ago 17h 448M - 18.2.2 3c937764e6f5 70f06fa05b70 mgr.master2 master2 running (16h) 6m ago 17h 524M - 18.2.2 3c937764e6f5 2d0d5376d8b3 mon.master master1 running (16h) 6m ago 17h 384M 2048M 18.2.2 3c937764e6f5 66a65017ce29 mon.master2 master2 running (16h) 6m ago 17h 380M 2048M 18.2.2 3c937764e6f5 51d783a9e36c osd.0 osd00 running (16h) 3m ago 17h 432M 4096M 18.2.2 3c937764e6f5 fedff66f5ed2 osd.1 osd00 running (16h) 3m ago 17h 475M 4096M 18.2.2 3c937764e6f5 24e24a1a22e6 osd.2 osd00 running (16h) 3m ago 17h 516M 4096M 18.2.2 3c937764e6f5 ccd05451b739 osd.3 osd00 running (16h) 3m ago 17h 454M 4096M 18.2.2 3c937764e6f5 f6d8f13c8aaf osd.4 master1 running (16h) 6m ago 17h 525M 4096M 18.2.2 3c937764e6f5 a2dcf9f1a9b7 osd.5 master2 running (16h) 6m ago 17h 331M 4096M 18.2.2 3c937764e6f5 b0011e8561a4 [root@master1 ~]# ceph orch ls NAME PORTS RUNNING REFRESHED AGE PLACEMENT mds.datafs 2/2 6m ago 46s master1;master2;count:2 mgr 2/0 6m ago - <unmanaged> mon 2/0 6m ago - <unmanaged> osd 6 6m ago - <unmanaged> [root@master1 ~]# ceph versions { "mon": { "ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable)": 2 }, "mgr": { "ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable)": 2 }, "osd": { "ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable)": 6 }, "mds": { "ceph version 17.0.0-7183-g54142666 (54142666e5705ced88e3e2d91ddc0ff29867a362) quincy (dev)": 2 }, "overall": { "ceph version 17.0.0-7183-g54142666 (54142666e5705ced88e3e2d91ddc0ff29867a362) quincy (dev)": 2, "ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable)": 10 } } [root@master1 ~]# podman images REPOSITORY TAG IMAGE ID CREATED SIZE quay.io/ceph/ceph v18.2.2 3c937764e6f5 7 weeks ago 1.28 GB quay.io/ceph/ceph v18 3c937764e6f5 7 weeks ago 1.28 GB registry.access.redhat.com/ubi8 latest c70d72aaebb4 3 months ago 212 MB quay.io/ceph/ceph v16 0d668911f040 23 months ago 1.27 GB quay.io/ceph/ceph-grafana 8.3.5 dad864ee21e9 2 years ago 571 MB quay.io/prometheus/prometheus v2.33.4 514e6a882f6e 2 years ago 205 MB quay.io/prometheus/node-exporter v1.3.1 1dbe0e931976 2 years ago 22.3 MB quay.io/prometheus/alertmanager v0.23.0 ba2b418f427c 2 years ago 58.9 MB docker.io/ceph/daemon-base latest-master-devel 75e3d7089cea 2 years ago 1.29 GB

3 2

Numa pinning best practices
by Szabo, Istvan (Agoda) 15 Sep '24

15 Sep '24

Hi, Haven't really found a proper descripton in case of 2 socket how to pin osds to numa node, only this: https://tracker.ceph.com/projects/ceph/wiki/Tuning_for_All_Flash_Deployment… Tuning for All Flash Deployments - Ceph - Ceph <https://tracker.ceph.com/projects/ceph/wiki/Tuning_for_All_Flash_Deployment…> Redmine tracker.ceph.com Is there anybody have some good how to on this topic? Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

5 5

squid 19.2.0 QE validation status
by Yuri Weinstein 12 Sep '24

12 Sep '24

Details of this release are summarized here: https://tracker.ceph.com/issues/67779#note-1 Release Notes - TBD Gibba upgrade -TBD LRC upgrade - TBD It was decided and agreed upon that there would be limited testing for this release, given it is based on 19.1.1 rather than a full rebase. Seeking approvals/reviews for: (some reruns are still in progress) rgw - Eric, Adam E fs - Venky orch - Adam King quincy-x - Laura, Neha reef-x - Laura, Neha Pls let me know if any tests were missed from this list.

7 18

lifecycle policy on non-replicated buckets
by Christopher Durham 12 Sep '24

12 Sep '24

Hi, I have a multisite system with two sites on 18.2.2, on Rocky 8. I have set up a sync policy to allow replication between sites. I have also createda policy for a given bucket that prevents replication on that given bucket. This allworks just fine, and objects I create in that bucket on side A do not get replicatedto side B, and objects I create in that bucket on side B do not get replicated to side A, butreplication for all other buckets works fine. This is great. But I still want to have a lifecycle policy that deletes objects after say, 4 days. If I create and install the json for this policy, via s3api put-bucket-lifecycle-configuration then only on the master side, side A, do objects get deleted after 4 days. Any objects on side B never get deleted. What am I doing wrong? Thanks -Chris

2 4

Successfully using dm-cache
by Michael Lipp 12 Sep '24

12 Sep '24

Just in case anybody is interested: Using dm-cache works and boosts performance -- at least for my use case. The "challenge" was to get 100 (identical) Linux-VMs started on a three node hyperconverged cluster. The hardware is nothing special, each node has a Supermicro server board with a single CPU with 24 cores and 4 x 4 TB hard disks. And there's that extra 1 TB NVMe... I know that the general recommendation is to use the NVMe for WAL and metadata, but this didn't seem appropriate for my use case and I'm still not quite sure about failure scenarios with this configuration. So instead I made each drive a logical volume (managed by an OSD) and added 85 GiB NVMe to each LV as read-only cache. Each VM uses as system disk an RBD based on a snapshot from the master image. The idea was that with this configuration, all VMs should share most (actually almost all) of the data on their system disk and this data should be available from the cache. Well, it works. When booting the 100 VMs, almost all read operations are satisfied from the cache. So I get close to NVMe speed but have payed for conventional hard drives only (well, SSDs aren't that much more expensive nowadays, but the hardware is 4 years old). So, nothing sophisticated, but as I couldn't find anything about this kind of setup, it might be of interest nevertheless. - Michael

13 27

RGW sync gets stuck every day
by Olaf Seibert 12 Sep '24

12 Sep '24

Hi all, we have some Ceph clusters with RGW replication between them. It seems that in the last month at least, it gets stuck at around the same time ~every day. Not 100% the same time, and also not 100% of the days, but in the more recent days seem to happen more, and for longer. With "stuck" I mean that the "oldest incremental change not applied" is getting 5 or more minutes old, and not changing. In the past this seemed to resolve itself in a short time, but recently it didn't. It remained stuck at the same place for several hours. Also, on several different occasions I noticed that the shard number in question was the same. We are using Ceph 18.2.2, image id 719d4c40e096. The output on one end looks like this (I redacted out some of the data because I don't know how much of the naming would be sensitive information): root@zone2:/# radosgw-admin sync status --rgw-realm backup realm xxxxxxxx-xxxx-xxxx-xxxx-8ddf4576ebab (backup) zonegroup xxxxxxxx-xxxx-xxxx-xxxx-58af9051e063 (backup) zone xxxxxxxx-xxxx-xxxx-xxxx-e1223ae425a4 (zone2-backup) current time 2024-08-04T10:22:00Z zonegroup features enabled: resharding disabled: compress-encrypted metadata sync no sync (zone is master) data sync source: xxxxxxxx-xxxx-xxxx-xxxx-e8db1c51b705 (zone1-backup) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 3 shards behind shards: [30,90,95] oldest incremental change not applied: 2024-08-04T10:05:54.015403+0000 [30] while on the other side it looks ok (not more than half a minute behind): root@zone1:/# radosgw-admin sync status --rgw-realm backup realm xxxxxxxx-xxxx-xxxx-xxxx-8ddf4576ebab (backup) zonegroup xxxxxxxx-xxxx-xxxx-xxxx-58af9051e063 (backup) zone xxxxxxxx-xxxx-xxxx-xxxx-e8db1c51b705 (zone1-backup) current time 2024-08-04T10:23:05Z zonegroup features enabled: resharding disabled: compress-encrypted metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master data sync source: xxxxxxxx-xxxx-xxxx-xxxx-e1223ae425a4 (zone2-backup) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 4 shards behind shards: [89,92,95,98] oldest incremental change not applied: 2024-08-04T10:22:53.175975+0000 [95] With some experimenting, we found that redeploying the RGWs on this side resolves the situation: "ceph orch redeploy rgw.zone1-backup". The shards go into "Recovering" state and after a short time it is "caught up with source" as well. Redeploying stuff seems like a much too big hammer to get things going again. Surely there must be something more reasonable? Also, any ideas about how we can find out what is causing this? It may be that some customer has some job running every 24 hours, but that shouldn't cause the replication to get stuck. Thanks in advance, -- Olaf Seibert Site Reliability Engineer SysEleven GmbH Boxhagener Straße 80 10245 Berlin T +49 30 233 2012 0 F +49 30 616 7555 0 https://www.syseleven.de https://www.linkedin.com/company/syseleven-gmbh/ Current system status always at: https://www.syseleven-status.net/ Company headquarters: Berlin Registered court: AG Berlin Charlottenburg, HRB 108571 Berlin Managing directors: Andreas Hermann, Jens Ihlenfeld, Norbert Müller, Jens Plogsties

3 4

v19.1.1 Squid RC1 released
by Yuri Weinstein 06 Sep '24

06 Sep '24

This is the second release candidate for Squid. Feature highlights: RGW Fixed a regression in bucket ownership for Keystone users and implicit tenants. The User Accounts feature unlocks several new AWS-compatible IAM APIs for the self-service management of users, keys, groups, roles, policy and more. RADOS BlueStore has been optimized for better performance in snapshot-intensive workloads. BlueStore RocksDB LZ4 compression is now enabled by default to improve average performance and "fast device" space usage. Other improvements include more flexible EC configurations, an OpTracker to help debug mgr module issues, and better scrub scheduling. Dashboard * Rearranged Navigation Layout: The navigation layout has been reorganized for improved usability and easier access to key features. CephFS Improvements * Support for managing CephFS snapshots and clones, as well as snapshot schedule management * Manage authorization capabilities for CephFS resources * Helpers on mounting a CephFS volume RGW Improvements * Support for managing bucket policies * Add/Remove bucket tags * ACL Management * Several UI/UX Improvements to the bucket form Monitoring: Grafana dashboards are now loaded into the container at runtime rather than building a grafana image with the grafana dashboards. Official Ceph grafana images can be found in quay.io/ceph/grafana * Monitoring: RGW S3 Analytics: A new Grafana dashboard is now available, enabling you to visualize per bucket and user analytics data, including total GETs, PUTs, Deletes, Copies, and list metrics. Crimson/Seastore Crimson's first tech preview release! Supporting RBD workloads on Replicated pools. For more information please visit: https://ceph.io/en/news/crimson If any of our community members would like to help us with performance investigations or regression testing of the Squid release candidate, please feel free to provide feedback via email or in https://pad.ceph.com/p/squid_scale_testing. For more active discussions, please use the #ceph-at-scale slack channel in ceph-storage.slack.com. * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-19.1.1.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: 1d9f35852eef16b81614e38a05cf88b505cc142b

2 2

[RGW] Radosgw instances hang for a long time while doing realm update
by Huy Nguyen 05 Sep '24

05 Sep '24

Hi community, I'm using ceph v18.2.4. Each time I commit a period, all of my radosgw instances paused for about 30s which is kinda long, and make my object service down during that time. how to avoid the down time? Is there better/safer practice? Thanks Here is radosgw logs each time I did a period commit. You can see it was paused for a while since the last request 2024-08-23T22:43:32.152+0700 7f4594d2f700 1 ====== starting new request req=0x7f42dcfbf730 ===== 2024-08-23T22:43:32.152+0700 7f438b11c700 1 ====== req done req=0x7f42dcfbf730 op status=0 http_status=200 latency=0.000000000s ====== 2024-08-23T22:43:32.152+0700 7f438b11c700 1 beast: 0x7f42dcfbf730: 172.24.9.101 - anonymous [23/Aug/2024:22:43:32.152 +0700] "GET / HTTP/1.0" 200 231 - "check_http/v2.2 (monitoring-plugins 2.2)" - latency=0.000000000s 2024-08-23T22:44:02.778+0700 7f42e1fca700 1 rgw realm reloader: Pausing frontends for realm update... 2024-08-23T22:44:02.778+0700 7f42e1fca700 1 rgw realm reloader: Frontends paused 2024-08-23T22:44:03.358+0700 7f42e1fca700 1 rgw realm reloader: driver closed 2024-08-23T22:44:03.497+0700 7f42e1fca700 1 rgw realm reloader: Creating new driver 2024-08-23T22:44:03.497+0700 7f42e1fca700 1 mgrc service_daemon_register rgw.5055090799 metadata {arch=x86_64,ceph_release=reef,ceph_version=ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable),ceph_version_short=18.2.4.... 2024-08-23T22:44:03.497+0700 7f42e1fca700 1 rgw realm reloader: Finishing initialization of new driver 2024-08-23T22:44:03.497+0700 7f42e1fca700 1 rgw realm reloader: - REST subsystem init 2024-08-23T22:44:03.497+0700 7f42e1fca700 1 rgw realm reloader: - usage subsystem init 2024-08-23T22:44:03.497+0700 7f42e1fca700 1 rgw realm reloader: Resuming frontends with new realm configuration. 2024-08-23T22:44:06.135+0700 7f4637674700 1 ====== starting new request req=0x7f42dcfbf730 ===== 2024-08-23T22:44:06.136+0700 7f4637674700 -1 res_query() failed 2024-08-23T22:44:06.136+0700 7f4637674700 1 ====== req done req=0x7f42dcfbf730 op status=0 http_status=405 latency=0.000999984s ====== 2024-08-23T22:44:06.136+0700 7f4637674700 1 beast: 0x7f42dcfbf730: 172.24.17.21 - - [23/Aug/2024:22:44:06.135 +0700] "GET /admin HTTP/1.1" 405 133 - "curl/7.61.1" - latency=0.000999984s

2 1

Re: The snaptrim queue of PGs has not decreased for several days.
by Eugen Block 05 Sep '24

05 Sep '24

Hi (please don't drop the ML from your responses), > All PGs of pool cephfs are affected and they are in all OSDs then just pick a random one and check if anything stands out. I'm not sure if you mentioned it already, did you also try restarting OSDs? > Oh, not yesterday. I do it now, then I compat all osds with nostrim. > Do I add OSDs? Let's wait for the other results first (compaction, reducing defaults, OSD restart). If that doesn't change anything, I would probably try to add three more OSDs. I assume you have three hosts? Zitat von Giovanna Ratini <giovanna.ratini(a)uni-konstanz.de>: > Hello Eugen, > > Am 20.08.2024 um 09:44 schrieb Eugen Block: >> You could also look into the historic_ops of the primary OSD for >> one affected PG: > > All PGs of pool cephfs are affected and they are in all OSDs :-( > >> Did you reduce the default values I mentioned? > > Oh, not yesterday. I do it now, then I compat all osds with nostrim. > > Do I add OSDs? > > Regars, > > Gio > > >> >> ceph tell osd.<OSD_ID> dump_historic_ops_by_duration >> >> But I'm not sure if that can actually help here. There are plenty >> of places to look at, you could turn on debug logs on one primary >> OSD and inspect the output. >> >> I just get the feeling that this is one of the corner cases with >> too few OSDs, although the cluster load seems to be low. >> >> Zitat von Giovanna Ratini <giovanna.ratini(a)uni-konstanz.de>: >> >>> Hello Eugen, >>> >>> yesterday after stop and go of snaptrim the queue decrease a >>> little and then remain blocked. >>> They didn't grow and didn't decrease. >>> >>> Is that good or bad? >>> >>> >>> Am 19.08.2024 um 15:43 schrieb Eugen Block: >>>> There's a lengthy thread [0] where several approaches are >>>> proposed. The worst is a OSD recreation, but that's the last >>>> resort, of course. >>>> >>>> What's are the current values for these configs? >>>> >>>> ceph config get osd osd_pg_max_concurrent_snap_trims >>>> ceph config get osd osd_max_trimming_pgs >>>> >>>> Maybe decrease them to 1 each while the nosnaptrim flag is set, >>>> then unset it. You could also try online (and/or offline osd >>>> compaction) before unsetting the flag. Are the OSD processes >>>> utilizing an entire CPU? >>>> >>>> [0] https://www.spinics.net/lists/ceph-users/msg75626.html >>>> >>>> Zitat von Giovanna Ratini <giovanna.ratini(a)uni-konstanz.de>: >>>> >>>>> Hallo Eugen, >>>>> >>>>> yes, the load is for now not too much. >>>>> >>>>> I stop the snap and now this is the output. No changes in the queue. >>>>> >>>>> root@kube-master02:~# k ceph -s >>>>> Info: running 'ceph' command with args: [-s] >>>>> cluster: >>>>> id: 3a35629a-6129-4daf-9db6-36e0eda637c7 >>>>> health: HEALTH_WARN >>>>> nosnaptrim flag(s) set >>>>> 32 pgs not deep-scrubbed in time >>>>> 32 pgs not scrubbed in time >>>>> >>>>> services: >>>>> mon: 3 daemons, quorum bx,bz,ca (age 30h) >>>>> mgr: a(active, since 29h), standbys: b >>>>> mds: 1/1 daemons up, 1 hot standby >>>>> osd: 6 osds: 6 up (since 21h), 6 in (since 6d) >>>>> flags nosnaptrim >>>>> >>>>> data: >>>>> volumes: 1/1 healthy >>>>> pools: 4 pools, 97 pgs >>>>> objects: 4.21M objects, 2.5 TiB >>>>> usage: 7.7 TiB used, 76 TiB / 84 TiB avail >>>>> pgs: 65 active+clean >>>>> 32 active+clean+snaptrim_wait >>>>> >>>>> io: >>>>> client: 7.4 MiB/s rd, 7.9 MiB/s wr, 11 op/s rd, 35 op/s wr >>>>> >>>>> Am 19.08.2024 um 14:54 schrieb Eugen Block: >>>>>> What happens when you disable snaptrimming entirely? >>>>>> >>>>>> ceph osd set nosnaptrim >>>>>> >>>>>> So the load on your cluster seems low, but are the OSDs heavily >>>>>> utilized? Have you checked iostat? >>>>>> >>>>>> Zitat von Giovanna Ratini <giovanna.ratini(a)uni-konstanz.de>: >>>>>> >>>>>>> Hello Eugen, >>>>>>> >>>>>>> *root@kube-master02:~# k ceph -s* >>>>>>> >>>>>>> Info: running 'ceph' command with args: [-s] >>>>>>> cluster: >>>>>>> id: 3a35629a-6129-4daf-9db6-36e0eda637c7 >>>>>>> health: HEALTH_WARN >>>>>>> 32 pgs not deep-scrubbed in time >>>>>>> 32 pgs not scrubbed in time >>>>>>> >>>>>>> services: >>>>>>> mon: 3 daemons, quorum bx,bz,ca (age 13h) >>>>>>> mgr: a(active, since 13h), standbys: b >>>>>>> mds: 1/1 daemons up, 1 hot standby >>>>>>> osd: 6 osds: 6 up (since 5h), 6 in (since 5d) >>>>>>> >>>>>>> data: >>>>>>> volumes: 1/1 healthy >>>>>>> pools: 4 pools, 97 pgs >>>>>>> objects: 4.20M objects, 2.5 TiB >>>>>>> usage: 7.7 TiB used, 76 TiB / 84 TiB avail >>>>>>> pgs: 65 active+clean >>>>>>> 20 active+clean+snaptrim_wait >>>>>>> 12 active+clean+snaptrim >>>>>>> >>>>>>> io: >>>>>>> client: 3.5 MiB/s rd, 3.6 MiB/s wr, 6 op/s rd, 12 op/s wr >>>>>>> >>>>>>> If I understand the documentation correctly, I will never have >>>>>>> a scrub unless the PGs (Placement Groups) are active and clean. >>>>>>> >>>>>>> All 32 PGs of the CephFS pool have been in this status for >>>>>>> several days: >>>>>>> >>>>>>> * 20 active+clean+snaptrim_wait >>>>>>> * 12 active+clean+snaptrim" >>>>>>> >>>>>>> Today, I restarted the MON, MGR, and MDS, but no changes in >>>>>>> the growing. >>>>>>> >>>>>>> Am 18.08.2024 um 18:39 schrieb Eugen Block: >>>>>>>> Can you share the current ceph status? Are the OSDs reporting >>>>>>>> anything suspicious? How is the disk utilization? >>>>>>>> >>>>>>>> Zitat von Giovanna Ratini <giovanna.ratini(a)uni-konstanz.de>: >>>>>>>> >>>>>>>>> More information: >>>>>>>>> >>>>>>>>> The snaptrim take a lot of time but the he objects_trimmed are "0" >>>>>>>>> >>>>>>>>> "objects_trimmed": 0, >>>>>>>>> "snaptrim_duration": 500.58076017500002, >>>>>>>>> >>>>>>>>> It could explain, why the queue are growing up.. >>>>>>>>> >>>>>>>>> >>>>>>>>> Am 17.08.2024 um 14:37 schrieb Giovanna Ratini: >>>>>>>>>> Hello again, >>>>>>>>>> >>>>>>>>>> I checked the pgs dump. Snapshot grow up >>>>>>>>>> >>>>>>>>>> Query für PG: 3.12 >>>>>>>>>> { >>>>>>>>>> "snap_trimq": >>>>>>>>>> "[5b974~3b,5cc3a~1,5cc3c~1,5cc3e~1,5cc40~1,5cd83~1,5cd85~1,5cd87~1,5cd89~1,5cecc~1,5cece~4,5ced3~2,5cf72~1,5cf74~4,5cf79~a2,5d0b8~1,5d0bb~1,5d0bd~a5,5d1f9~2,5d204~a5,5d349~a7,5d48e~3,5d493~a4,5d5d7~a7,5d71e~a3,5d7c2~3,5d860~1,5d865~4,5d86a~a2,5d9aa~1,5d9ac~1,5d9ae~a5,5daf3~a5,5db9a~2,5dc3a~a5,5dce1~1,5dce3~1,5dd81~a7,5dec8~a7,5e00f~a7,5e156~a8,5e29d~1,5e29f~a7,5e3e6~a8,5e52e~a6,5e5d6~2,5e676~a6,5e71e~2,5e7be~a9,5e907~a5,5e9ad~3,5ea50~a7,5eaf9~1,5eafb~1,5eb99~a7,5ec42~2,5ece2~a7,5ed8a~2,5ee2b~a9,5ef74~a7,5f01c~1,5f0bd~a1,5f15f~1,5f161~1,5f163~1,5f167~1,5f206~a1,5f2a8~1,5f2aa~1,5f2ac~1,5f2ae~1,5f34f~a1,5f3f1~1,5f3f3~1,5f3f5~1,5f3f7~1,5f499~a1,5f53b~1,5f53d~1,5f53f~1,5f541~1,5f5e3~a1,5f685~1,5f687~1,5f689~1,5f68d~1,5f72d~a1,5f7cf~1,5f7d1~1,5f7d3~1]", >>>>>>>>>> * "snap_trimq_len": 5421,* >>>>>>>>>> "state": "active+clean+snaptrim", >>>>>>>>>> "epoch": 734130, >>>>>>>>>> >>>>>>>>>> Query für PG: 3.12 >>>>>>>>>> { >>>>>>>>>> "snap_trimq": >>>>>>>>>> "[5b976~39,5ba53~1,5ba56~a0,5cc3a~1,5cc3c~1,5cc3e~1,5cc40~1,5cd83~1,5cd85~1,5cd87~1,5cd89~1,5cecc~1,5cece~4,5ced3~2,5cf72~1,5cf74~4,5cf79~a2,5d0b8~1,5d0bb~1,5d0bd~a5,5d1f9~2,5d204~a5,5d349~a7,5d48e~3,5d493~a4,5d5d7~a7,5d71e~a3,5d7c2~3,5d860~1,5d865~4,5d86a~a2,5d9aa~1,5d9ac~1,5d9ae~a5,5daf3~a5,5db9a~2,5dc3a~a5,5dce1~1,5dce3~1,5dd81~a7,5dec8~a7,5e00f~a7,5e156~a8,5e29d~1,5e29f~a7,5e3e6~a8,5e52e~a6,5e5d6~2,5e676~a6,5e71e~2,5e7be~a9,5e907~a5,5e9ad~3,5ea50~a7,5eaf9~1,5eafb~1,5eb99~a7,5ec42~2,5ece2~a7,5ed8a~2,5ee2b~a9,5ef74~a7,5f01c~1,5f0bd~a1,5f15f~1,5f161~1,5f163~1,5f167~1,5f206~a1,5f2a8~1,5f2aa~1,5f2ac~1,5f2ae~1,5f34f~a1,5f3f1~1,5f3f3~1,5f3f5~1,5f3f7~1,5f499~a1,5f53b~1,5f53d~1,5f53f~1,5f541~1,5f5e3~a1,5f685~1,5f687~1,5f689~1,5f68d~1,5f72d~a1,5f7cf~1,5f7d1~1,5f7d3~1,5f875~a1]", >>>>>>>>>> * "snap_trimq_len": 5741,* >>>>>>>>>> "state": "active+clean+snaptrim", >>>>>>>>>> "epoch": 734240, >>>>>>>>>> "up": [ >>>>>>>>>> >>>>>>>>>> Do you know the way to see if the snaptim "process" works? >>>>>>>>>> >>>>>>>>>> Best Regard >>>>>>>>>> >>>>>>>>>> Gio >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Am 17.08.2024 um 12:59 schrieb Giovanna Ratini: >>>>>>>>>>> Hello Eugen, >>>>>>>>>>> >>>>>>>>>>> thank you for your answer. >>>>>>>>>>> >>>>>>>>>>> I restarted all the kube-ceph nodes one after the other. >>>>>>>>>>> Nothing has changed. >>>>>>>>>>> >>>>>>>>>>> ok, I deactivate the snap ... : ceph fs snap-schedule deactivate / >>>>>>>>>>> >>>>>>>>>>> Is there a way to see how many snapshots will be deleted per hour? >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> >>>>>>>>>>> Gio >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Am 17.08.2024 um 10:12 schrieb Eugen Block: >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> have you tried to fail the mgr? Sometimes the PG stats >>>>>>>>>>>> are not correct. You could also temporarily disable >>>>>>>>>>>> snapshots to see if things settle down. >>>>>>>>>>>> >>>>>>>>>>>> Zitat von Giovanna Ratini <giovanna.ratini(a)uni-konstanz.de>: >>>>>>>>>>>> >>>>>>>>>>>>> Hello all, >>>>>>>>>>>>> >>>>>>>>>>>>> We use Ceph (v18.2.2) and Rook (1.14.3) as the CSI for a >>>>>>>>>>>>> Kubernetes environment. Last week, we had a problem with >>>>>>>>>>>>> the MDS falling behind on trimming every 4-5 days >>>>>>>>>>>>> (GitHub issue link). We resolved the issue using the >>>>>>>>>>>>> steps outlined in the GitHub issue. >>>>>>>>>>>>> >>>>>>>>>>>>> We have 3 hosts (I know, I need to increase this as soon >>>>>>>>>>>>> as possible, and I will!) and 6 OSDs. After running the >>>>>>>>>>>>> commands: >>>>>>>>>>>>> >>>>>>>>>>>>> ceph config set mds mds_dir_max_commit_size 80, >>>>>>>>>>>>> >>>>>>>>>>>>> ceph fs fail <fs_name>, and >>>>>>>>>>>>> >>>>>>>>>>>>> ceph fs set <fs_name> joinable true, >>>>>>>>>>>>> >>>>>>>>>>>>> After that, the snaptrim queue for our PGs has stopped >>>>>>>>>>>>> decreasing. All PGs of our CephFS are in either >>>>>>>>>>>>> active+clean+snaptrim_wait or active+clean+snaptrim >>>>>>>>>>>>> states. For example, the PG 3.12 is in the >>>>>>>>>>>>> active+clean+snaptrim state, and its snap_trimq_len was >>>>>>>>>>>>> 4077 yesterday but has increased to 4538 today. >>>>>>>>>>>>> >>>>>>>>>>>>> I increased the osd_snap_trim_priority to 10 (ceph >>>>>>>>>>>>> config set osd osd_snap_trim_priority 10), but it didn't >>>>>>>>>>>>> help. Only the PGs of our CephFS have this problem. >>>>>>>>>>>>> >>>>>>>>>>>>> Do you have any ideas on how we can resolve this issue? >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks in advance, >>>>>>>>>>>>> Giovanna >>>>>>>>>>>>> p.s. I'm not a ceph expert :-). >>>>>>>>>>>>> Faulkener asked me for more information, so here it is: >>>>>>>>>>>>> MDS Memory: 11GB >>>>>>>>>>>>> mds_cache_memory_limit: 11,811,160,064 bytes >>>>>>>>>>>>> >>>>>>>>>>>>> root@kube-master02:~# ceph fs snap-schedule status / >>>>>>>>>>>>> { >>>>>>>>>>>>> "fs": "rook-cephfs", >>>>>>>>>>>>> "subvol": null, >>>>>>>>>>>>> "path": "/", >>>>>>>>>>>>> "rel_path": "/", >>>>>>>>>>>>> "schedule": "3h", >>>>>>>>>>>>> "retention": {"h": 24, "w": 4}, >>>>>>>>>>>>> "start": "2024-05-05T00:00:00", >>>>>>>>>>>>> "created": "2024-05-05T17:28:18", >>>>>>>>>>>>> "first": "2024-05-05T18:00:00", >>>>>>>>>>>>> "last": "2024-08-15T18:00:00", >>>>>>>>>>>>> "last_pruned": "2024-08-15T18:00:00", >>>>>>>>>>>>> "created_count": 817, >>>>>>>>>>>>> "pruned_count": 817, >>>>>>>>>>>>> "active": true >>>>>>>>>>>>> } >>>>>>>>>>>>> I do not understand if the snapshots in the PGs are >>>>>>>>>>>>> correlated with the snapshots on CephFS. Until we >>>>>>>>>>>>> encountered the issue with the "MDS falling behind on >>>>>>>>>>>>> trimming every 4-5 days," we didn't have any problems >>>>>>>>>>>>> with snapshots. >>>>>>>>>>>>> >>>>>>>>>>>>> Could someone explain me this or send me to the documentation? >>>>>>>>>>>>> Thank you >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>>>>>>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>>>>>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>>>>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>>>>>>> _______________________________________________ >>>>>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>>>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>>>>>> _______________________________________________ >>>>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>>>> >>>>>>> -- >>>>>>> Giovanna Ratini >>>>>>> Mail:ratini@dbvis.inf.uni-konstanz.de >>>>>>> Phone: +49 (0) 7531 88 - 4550 >>>>>>> >>>>>>> Technical Support >>>>>>> Data Analysis and Visualization Group >>>>>>> Department of Computer and Information Science >>>>>>> University of Konstanz (Box 78) >>>>>>> Universitätsstr. 10 >>>>>>> 78457 Konstanz, Germany >>>>>>> _______________________________________________ >>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>>> >>>>> -- >>>>> Giovanna Ratini >>>>> Mail:ratini@dbvis.inf.uni-konstanz.de >>>>> Phone: +49 (0) 7531 88 - 4550 >>>>> >>>>> Technical Support >>>>> Data Analysis and Visualization Group >>>>> Department of Computer and Information Science >>>>> University of Konstanz (Box 78) >>>>> Universitätsstr. 10 >>>>> 78457 Konstanz, Germany >>>>> _______________________________________________ >>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>> >>>> >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> >>> -- >>> Giovanna Ratini >>> Mail:ratini@dbvis.inf.uni-konstanz.de >>> Phone: +49 (0) 7531 88 - 4550 >>> >>> Technical Support >>> Data Analysis and Visualization Group >>> Department of Computer and Information Science >>> University of Konstanz (Box 78) >>> Universitätsstr. 10 >>> 78457 Konstanz, Germany >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > -- > Giovanna Ratini > Mail:ratini@dbvis.inf.uni-konstanz.de > Phone: +49 (0) 7531 88 - 4550 > > Technical Support > Data Analysis and Visualization Group > Department of Computer and Information Science > University of Konstanz (Box 78) > Universitätsstr. 10 > 78457 Konstanz, Germany

4 11

MDS cache always increasing
by Sake Ceph 04 Sep '24

04 Sep '24

I hope someone can help us with a MDS caching problem. Ceph version 18.2.4 with cephadm container deployment. Question 1: For me it's not clear how much cache/memory you should allocate for the MDS. Is this based on the number of open files, caps or something else? Question 2/Problem: At the moment we have MDS nodes with 32 GB of memory and a configured cache limit of 20 GB. There are 4 MDS nodes: 2 active and 2 in standby-replay mode (with max_mds set at 2 of course). We pinned top directories to specific ranks, so the balancer isn't used. The memory usage is for the most part increasing, sometimes a little dip with couple hundred MB's freed. After all the memory is consumed, SWAP gets used. This results in a couple of hundred MB's of freed memory, but not much. When eventually the SWAP runs out and the memory is full, the MDS service stops and the cluster logs show: 1. no beacon from mds 2. marking mds up:active laggy 3. replacing mds 4. MDS daemon <daemon> is removed because it is dead or otherwise unavailable For example: we have the top folder app2 and app4 which is pinnend to rank 1. Folder app2 is always accessed by 4 clients (application servers), the same happens with folder app4. Folder app2 is 3 times larger than folder app4 (last time I checked, don't wanna do a du at the moment). After a couple of hours the memory usage of the MDS server stays around 18% (Grafana shows a flatline for 7 hours). At night the 9the client connects and makes first a backup with rsync of the latest snapshot folder of app2 and afterwards the same happens for folder app4 with a pause for 5 minutes. When the backup starts, the memory increases to 70% and stays at 70% after the backup of app2 is completed. 5 minutes later the memory starts increases again with the start of the backup of folder app4. When the backup is done, it's at 78% and stays there for the rest of the day. Why isn't the memory usage decreasing after the rsync is completed? Is there a memory leak with the MDS service? Ps. I have some small log files/Grafana screenshots, not sure how to share. Kind regards, Sake

3 13

ceph-ansible installation error
by Michel Niyoyita 02 Sep '24

02 Sep '24

Dear team , I configuring ceph cluster using ceph-ansible , ubuntu OS 20.04 . my previous production cluster have been configured using the same configuration and it is working perfectly , now I am trying to build the new cluster using the same configurations , but I am facing the following errors: ansible-playbook site.yml Traceback (most recent call last): File "/usr/bin/ansible-playbook", line 66, in <module> from ansible.utils.display import Display, initialize_locale ImportError: cannot import name 'initialize_locale' from 'ansible.utils.display' (/usr/local/lib/python3.8/dist-packages/ansible/utils/display.py) Can someone help to solve the issue? Best regards Michel

6 11

How many MDS & MON servers are required
by s.dhivagar.cse＠gmail.com 01 Sep '24

01 Sep '24

Hi, We are using raw cephfs data 182 TB replica 2 and single MDS seemed to regularly run around 4002 req/s, So how many MDS & MON servers are required? Also mentioned current ceph cluster servers Client : 60 MDS: 3 (2 Active + 1 Standby) MON: 4 MGR: 3 (1 Active + 2 Standby) OSD: 52 PG : auto scale

2 1

How to know is ceph is ready?
by Alfredo Rezinovsky 30 Aug '24

30 Aug '24

I have a proxmox cluster using an external CEPH cluster. Sometimes due to blackouts the servers need to restart. If proxmox starts before CEPH is ready the VMs fail to boot. I want to add a dependency in proxmox to wait for ceph to be ready. I can work with a HEALTH_WARN as long the RBD pool is usable. ceph status exit status doesn´t helps Should I grep for "pgs not active" in ceph status or for "inactive" pgs in ceph health or is there something more direct to know if everything is alright? -- Alfrenovsky

5 6

ceph orch host drain daemon type
by Eugen Block 30 Aug '24

30 Aug '24

Hi, is there anything on the road map to be able to choose a specific daemon type to be entirely removed from a host instead of all cephadm managed daemons? I just did a quick search in tracker and github, but it may be "hidden" somewhere else. I was thinking about colocated daemons on a host, for example MON, MGR, OSDs, node-exporter, crash. That's quite common, but if I just wanted to drain all OSDs (maybe mark them as "destroyed" in order to replace the drives), the usual 'ceph orch host drain <host>' command would remove all daemons. That seems unnecessary if I'm going to add the OSDs back. Since there are a couple of other daemon types which can be deployed multiple times per host, e. g. MDS, RGW, it doesn't only make sense for OSDs but for other daemons as well. And those other daemons usually have some cryptic suffix, we wouldn't need that in order to get rid of them, it doesn't save that much time, but it could be a nice enhancement. Should I create a tracker issue in the enhancement section for that? Thanks! Eugen

3 6

squid release codename
by Alfredo Rezinovsky 30 Aug '24

30 Aug '24

I think is a very bad idea to name a release with the name of the most popular http cache. It will difficult googling. -- Alfrenovsky

13 15

how to distinguish SED/OPAL and non SED/OPAL disks in orchestrator?
by Boris 29 Aug '24

29 Aug '24

Hi, is it possible to somehow distinguish self encrypting drives from drives that are lacking the support in the orchestrator. So I don't encrypt them twice :) -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

3 3

Found a way to clean ceph device ls
by Alfredo Rezinovsky 28 Aug '24

28 Aug '24

Ceph device ls shows every disk. Even removed ones with ceph config-key ls I can find the old disk key and remove it with ceph config-key rm -- Alfrenovsky

1 0

Cannot remove bucket due to missing placement rule
by sinan＠turka.nl 28 Aug '24

28 Aug '24

I created a new placement target/pool. I don't have the exact commands anymore, but something similar to: --- $ radosgw-admin zonegroup placement add \ --rgw-zonegroup default \ --placement-id temporary $ radosgw-admin zone placement add \ --rgw-zone default \ --placement-id temporary \ --data-pool default.rgw.temporary.data \ --index-pool default.rgw.temporary.index \ --data-extra-pool default.rgw.temporary.non-ec --- I created a new bucket to use the "default.rgw.temporary.data" as its data pool. Without removing the bucket, I removed the placement target/rule. I realised that I still had a bucket, so tried to remove the bucket: --- $ radosgw-admin bucket rm --bucket=tempbucket 2024-08-26T14:12:08.722+0200 7f6c88a73fc0 0 could not find placement rule temporary within zonegroup 2024-08-26T14:12:08.722+0200 7f6c88a73fc0 0 ERROR: int RGWRados::Bucket::List::list_objects_unordered(const DoutPrefixProvider*, int64_t, std::vector<rgw_bucket_dir_entry>*, std::map<std::__cxx11::basic_string<char>, bool>*, bool*, optional_yield) cls_bucket_list_unordered returned -22 for :tempbucket[d9c26db8-925f-4c6c-838d-6e886ec345ca.694015388.44]) --- So I tried to remove the bucket in a different way: --- $ radosgw-admin metadata rm bucket:tempbucket $ radosgw-admin bucket rm --bucket=tempbucket --- This looked good. I thought the bucket was removed, but my rgw containers are crashing when trying to trim (so it seems the bucket is not really removed): --- debug -2> 2024-08-26T12:33:21.291+0000 7eff1bc70700 0 could not find placement rule temporary within zonegroup debug -1> 2024-08-26T12:33:21.291+0000 7eff1bc70700 0 ERROR: open_bucket_index_shard() returned ret=-22 debug 0> 2024-08-26T12:33:21.295+0000 7eff1bc70700 -1 *** Caught signal (Segmentation fault) ** in thread 7eff1bc70700 thread_name:sync-log-trim ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable) 1: (()+0x12ce0) [0x7eff4530cce0] 2: (()+0x97b01) [0x7eff4f8a6b01] 3: (librados::v14_2_0::IoCtx::aio_operate(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, librados::v14_2_0::AioCompletion*, librados::v14_2_0::ObjectWriteOperation*)+0x74) [0x7eff4f87f3b4] 4: (RGWRadosBILogTrimCR::send_request()+0x1c7) [0x7eff5029fb07] 5: (RGWSimpleCoroutine::state_send_request()+0x13) [0x7eff50297483] 6: (RGWSimpleCoroutine::operate()+0xac) [0x7eff5029ca5c] 7: (RGWCoroutinesStack::operate(RGWCoroutinesEnv*)+0x67) [0x7eff5029a287] 8: (RGWCoroutinesManager::run(std::__cxx11::list<RGWCoroutinesStack*, std::allocator<RGWCoroutinesStack*> >&)+0x271) [0x7eff5029b0b1] 9: (RGWSyncLogTrimThread::process()+0x200) [0x7eff503636d0] 10: (RGWRadosThread::Worker::entry()+0x176) [0x7eff5032bc26] 11: (()+0x81ca) [0x7eff453021ca] 12: (clone()+0x43) [0x7eff43948dd3] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- I can recreate (if required for the fix) the metadata info, since I have the output of "radosgw-admin metadata get bucket:tempbucket > tempbucket_metadata.json" by using: --- $ radosgw-admin metadata put bucket:tempbucket < tempbucket_metadata.json --- What is the best way to recover from this? Recreate the placement rule? Or are there other options? I just want to get rid of the bucket, so my rgw containers don't crash while trying to trim this bucket. Thanks!

2 3

Persistent Bucket Notification Reconfiguring with CreateTopic
by Alex Hussein-Kershaw (HE/HIM) 28 Aug '24

28 Aug '24

Hi folks, Currently running a test Ceph cluster on 19.1.0 and seeing some odd behaviour using the CreateTopic API. Initially I call the CreateTopic API, and all looks well: $ rga topic list { "topics": [ { "topics": [ { "owner": "ahk2", "name": "ahk2", "dest": { "push_endpoint": "http://1.2.3.6:4321", "push_endpoint_args": "", "push_endpoint_topic": "ahk2", "stored_secret": false, "persistent": true, "persistent_queue": ":ahk2", "time_to_live": "None", "max_retries": "None", "retry_sleep_duration": "None" }, "arn": "arn:aws:sns:geored_zg::ahk2", "opaqueData": "", "policy": "" } ] } ] } I call it again, in an identical manner: $ rga topic list { "topics": [ { "topics": [ { "owner": "ahk2", "name": "ahk2", "dest": { "push_endpoint": "http://1.2.3.6:4321", "push_endpoint_args": "", "push_endpoint_topic": "ahk2", "stored_secret": false, "persistent": true, "persistent_queue": "", "time_to_live": "None", "max_retries": "None", "retry_sleep_duration": "None" }, "arn": "arn:aws:sns:geored_zg::ahk2", "opaqueData": "", "policy": "" } ] } ] } Notice that the "persistent_queue" field is gone. I now hit errors trying to use my topic, for example: $ rga topic stats --topic ahk2 This topic does not have a persistent queue. Suspect this might be a bug in the RC. Happy to open a tracker if so. Unless I'm doing something stupid...? Best Wishes, Alex

1 1

Cephfs client capabilities
by wiz.chen＠gmail.com 27 Aug '24

27 Aug '24

Hi All, How to restrict a user that cannot create volume, subvolumegroup, subvolume of cephfs. This user just can access(mount) a subvolume only. Thanks in advance Yufan Chen

3 2

cephadm basic questions: image config, OS reimages
by Matthew Vernon 27 Aug '24

27 Aug '24

Hi, I've some experience with Ceph, but haven't used cephadm much before, and am trying to configure a pair of reef clusters with cephadm. A couple of newbie questions, if I may: * cephadm shell image I'm in an isolated environment, so pulling from a local repository. I bootstrapped OK with cephadm --image docker-registry.wikimedia.org/ceph bootstrap ... And that worked nicely, but if I want to run cephadm shell (to do any sort of admin), then I have to specify cephadm --image docker-registry.wikimedia.org/ceph shell (otherwise it just hangs failing to talk to quay.io). I found the docs, which refer to setting lots of other images, but not the one that cephadm uses: https://docs.ceph.com/en/reef/cephadm/install/#deployment-in-an-isolated-en… I found an old tracker in this area: https://tracker.ceph.com/issues/47274 ...but is there a good way to arrange for cephadm to use the already-downloaded image without having to remember to specify --image each time? * OS reimages We do OS upgrades by reimaging the server (which doesn't touch the storage disks); on an old-style deployment you could then use ceph-volume to re-start the OSDs and away you went; how does one do this in a cephadm cluster? [I presume involves telling cephadm to download a new image for podman to use and suchlike] Would the process be smoother if we arranged to leave /var/lib/ceph intact between reimages? Thanks, Matthew

3 4

cephfs client capabilities
by YuFan Chen 27 Aug '24

27 Aug '24

Hi all, How to restrict an user that cannot create volume, subvolume group, subvolume of cephfs? This user only can access(mount) a subvolume. Thanks in advance Yufan Chen

1 0

Connecting A Client To 2 Different Ceph Clusters
by duluxoz 27 Aug '24

27 Aug '24

Hi All, I haven't been able to find anything in the doco or on the web about this. I'm blessed with having available to me 2 different Ceph Clusters within our organisation (a "regular" Ceph Cluster and a Hyper-converged Proxmox Ceph Cluster). Both Ceph Clusters have a CephFS system on them (regular_cephfs and proxmox_cephfs). Obviously each Ceph Cluster has its own set of Manager, Monitor, and MDS Nodes, and each CephFS has its own user(s)/key(s). Also obviously, I've got the all the relevant key files install on my client in the /etc/ceph folder. How do I set up the ceph.conf file(s) on my clients so that I can use /etc/fstab to connect to each CephFS or each Ceph Cluster? My existing /etc/fstab file (extract of): ~~~ regular_cephfs_user@.regular_cephfs=/folder1 /mnt/folder1 ceph noatime,_netdev 0 0 proxmox_cephfs_user@.proxmox_cephfs=/folder2 /mnt/folder2 ceph noatime,_netdev 0 0 ~~~ Thanks in advance Dulux-Oz

2 4

Do you need to use a dedicated server for the MON service?
by Phong Tran Thanh 27 Aug '24

27 Aug '24

Hi Ceph users I am designing a CEPH system with 6 servers running full NVMe. Do I need to use 3 separate servers to run the MON services that communicate with OpenStack, or should I integrate the MON services into the OSD servers? What is the recommendation? Thank you. -- Email: tranphong079(a)gmail.com Skype: tranphong079

6 8

Snaptrim issue after nautilus to octopus upgrade
by Özkan Göksu 26 Aug '24

26 Aug '24

Hello folks. We have a ceph cluster and we have 2000+ RBD drives on 20 nodes. We upgraded the cluster from 14.2.16 to 15.2.14 and after the upgrade we started to see snap trim issues. Without the "nosnaptrim" flag, the system is not usable right now. I think the problem is because of the omap conversion at Octopus upgrade. Note that the first time each OSD starts, it will do a format conversion to improve the accounting for “omap” data. This may take a few minutes to as much as a few hours (for an HDD with lots of omap data). You can disable this automatic conversion with: What should I do to solve this problem? Thanks.

3 9

[PSA] New git version tag: v19.3.0
by Patrick Donnelly 26 Aug '24

26 Aug '24

TL;DR: we need an intermediate tag, v19.3.0, before v20.0.0 (tentacle) to get upgrade tests from squid -> main working. It is not a real release and should/will not be built/used. Right now upgrade tests are failing [1] from the squid RC to the main branch. This is because teuthology refuses to do an apparent downgrade (v19.1.X -> v19.0.0) during upgrade testing. I'm proposing an intermediate band-aid solution of creating a v19.3.0 **unsigned** tag on the current version of the main branch which would only exist in the git repository (possibly for only a short time). This version is chosen because it'll be greater than any squid release (v19.2.X) or release candidate (v19.1.X) using our current release schema while also not changing the major version (v19 -> v20). Changing the major version is an involved process touching many parts of Ceph and historically takes some time to merge. (I will be starting that process in the coming weeks.) [1] https://tracker.ceph.com/issues/67335 -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

tracing in ceph - tentacle release
by Yuval Lifshitz 26 Aug '24

26 Aug '24

Dear Community, jaeger based tracing was introduced to Ceph in the "reef" release, and a major improvement is done in the "squid" release, where we introduced end2end tracing between the RGW and the OSD. In "tentacle" we want to improve tracing further. Initial discussion was done as part of the RGW CDS (although this is not an RGW specific topic). Recording could be found here: https://www.youtube.com/watch?v=t3tT360Eub0&t=24m40s . Some ideas on where we want to take tracing in the upcoming releases are summarized here: https://gist.github.com/yuvalif/52032debe4065d8fb2ddc06308ed94d9 Would appreciate any feedback suggestions and requests on that topic! Yuval

1 0

Bluefs spillover
by Ruben Bosch 26 Aug '24

26 Aug '24

Hi all, ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable) We are working on marking out OSDs on a host with EC4+2. The OSDs are HDDs. The OSDs have a separate DB on an NVMe disk. All the operations take ages. After some time we see BLUEFS_SPILLOVER. Telling the mentioned OSDs to compact sometimes helps, but not always. The OSDs have plenty space remaining in the db but the spillover does not disappear. [WRN] BLUEFS_SPILLOVER: 2 OSD(s) experiencing BlueFS spillover osd.91 spilled over 141 MiB metadata from 'db' device (15 GiB used of 50 GiB) to slow device osd.106 spilled over 70 MiB metadata from 'db' device (12 GiB used of 50 GiB) to slow device Has anyone seen similar behavior before and have they found a workaround or solution? Kind regards, Ruben Bosch

2 4

Is ceph-qa list still under administrated?
by Fred Liu 26 Aug '24

26 Aug '24

Hello, Just by chance, there are administrators here. Thank you. Fred

1 0

Paid support options?
by Paul Mezzanini 24 Aug '24

24 Aug '24

We've reached the point of our ceph cluster's life where it make sense to have an outside vendor in the passenger seat with us. I already know of a few but I see value in having a thread consolidating this information so I'm leaving it open ended. The two main questions I'm asking are: What vendors offer paid ceph support? Do they have specific requirements? (e.g. must run their version of ceph vs community, must be containerized vs bare metal) Thanks -paul -- Paul Mezzanini Platform Engineer III Research Computing Rochester Institute of Technology

10 9

Re: Snaptrim issue after nautilus to octopus upgrade
by Boris 23 Aug '24

23 Aug '24

Ah crap, I didn't reply to the list :D tldr for the rest of the list: move to next release because of bad bug https://tracker.ceph.com/issues/53729#note-65 I would go to pacific and then wait some time so the pglog_dups get cleaned up. Pacific and quincy are really nice releases that worked very well for us. Am Fr., 23. Aug. 2024 um 18:08 Uhr schrieb Özkan Göksu <ozkangksu(a)gmail.com >: > Hello Boris. Thank you so much for the recommendation. > > Actually my intention is jumping next releases until I reach Quincy. > What do you think about this? Am I gonna face with any problem while > jumping Octopus to Pacific and Quincy after ? > > Best regards. > > Boris <bb(a)kervyn.de>, 23 Ağu 2024 Cum, 18:56 tarihinde şunu yazdı: > >> Hi, I would suggest to move fast to the next release. >> There was a bug that still haunts my sleep that is known as the PGLOG_DUP >> bug. >> >> You can check if you are suffering from it here: >> https://tracker.ceph.com/issues/53729#note-65 >> >> Basically the pglog dups would not be cleaned anymore. And snaptrim seems >> to trigger that. The pacific release got the fix. >> >> Cheers >> Boris >> >> Am Fr., 23. Aug. 2024 um 16:48 Uhr schrieb Özkan Göksu < >> ozkangksu(a)gmail.com>: >> >>> Hello folks. >>> >>> We have a ceph cluster and we have 2000+ RBD drives on 20 nodes. >>> >>> We upgraded the cluster from 14.2.16 to 15.2.14 and after the upgrade we >>> started to see snap trim issues. >>> Without the "nosnaptrim" flag, the system is not usable right now. >>> >>> I think the problem is because of the omap conversion at Octopus upgrade. >>> >>> Note that the first time each OSD starts, it will do a format conversion >>> to >>> improve the accounting for “omap” data. This may take a few minutes to as >>> much as a few hours (for an HDD with lots of omap data). You can disable >>> this automatic conversion with: >>> >>> What should I do to solve this problem? >>> >>> Thanks. >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> >> >> >> -- >> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im >> groÃƒ¼en Saal. >> > -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

2 1

Pull failed on cluster upgrade
by Nicola Mori 22 Aug '24

22 Aug '24

Dear Ceph users, during an upgrade from 18.2.2 to 18.2.4 the image pull from Dockerhub failed on one machine running a monitor daemon, while it succeeded on the previous ones. # ceph orch upgrade status { "target_image": "snack14/ceph-wizard@sha256:b1994328eb078778abdba0a17a7cf7b371e7d95ee1d543bd987892941ceb91c1", "in_progress": true, "which": "Upgrading all daemon types on all hosts", "services_complete": [ "mgr" ], "progress": "3/152 daemons upgraded", "message": "Error: UPGRADE_FAILED_PULL: Upgrade: failed to pull target image", "is_paused": true } # ceph health detail [WRN] UPGRADE_FAILED_PULL: Upgrade: failed to pull target image failed to pull snack14/ceph-wizard@sha256:b1994328eb078778abdba0a17a7cf7b371e7d95ee1d543bd987892941ceb91c1 on host aka On the affected host I can see the image has been correctly pulled, I can even delete and re-pull it manually: $ sudo docker pull snack14/ceph-wizard@sha256:b1994328eb078778abdba0a17a7cf7b371e7d95ee1d543bd987892941ceb91c1 docker.io/snack14/ceph-wizard@sha256:b1994328eb078778abdba0a17a7cf7b371e7d95ee1d543bd987892941ceb91c1: Pulling from snack14/ceph-wizard c4df4d1fcd03: Pull complete 676ca14fffd6: Pull complete cde0e2cfc7c9: Pull complete Digest: sha256:b1994328eb078778abdba0a17a7cf7b371e7d95ee1d543bd987892941ceb91c1 Status: Downloaded newer image for snack14/ceph-wizard@sha256:b1994328eb078778abdba0a17a7cf7b371e7d95ee1d543bd987892941ceb91c1 docker.io/snack14/ceph-wizard@sha256:b1994328eb078778abdba0a17a7cf7b371e7d95ee1d543bd987892941ceb91c1 so I believe I didn't hit Dockerhub pull limits. Yet the upgrade fails again in the same way when I resume it. I noticed that the docker logs show these warnings in coincidence with a failed pull triggered by the upgrade: Aug 05 18:24:28 aka dockerd[1564]: time="2024-08-05T18:24:28.087629058+02:00" level=warning msg="canonical references cannot be resolved: snack14/ceph-wizard@sha256:b1994328eb078778abdba0a17a7cf7b371e7d95ee1d543bd987892941ceb91c1" Aug 05 18:24:29 aka dockerd[1564]: time="2024-08-05T18:24:29.332143728+02:00" level=warning msg="reference for unknown type: " digest="sha256:b1994328eb078778abdba0a17a7cf7b371e7d95ee1d543bd987892941ceb91c1" remote="docker.io/snack14/ceph-wizard@sha256:b1994328eb078778> but only the second one appears when doing a successful manual pull, so I suspect only the first one is representative of the problem. The docker version on the affected machine is 26.1.3. Thanks in advance for any help, Nicola

8 12

Unable to recover cluster, error: unable to read magic from mon data
by RIT Computer Science House 22 Aug '24

22 Aug '24

Hello, Our cluster has become unresponsive after a teammate's work on the cluster. We are currently unable to get the full story on what he did to fully understand what is going on, and the only error we are able to see in any of the logs is the following: 2024-08-20T03:12:34.183+0000 7f3670246b80 -1 unable to read magic from mon data Any help would be greatly appreciated! I can provide any information necessary to help debugging. Thanks, Tyler System Administrator @ Computer Science House

2 3

Ceph XFS deadlock with Rook
by Raphaël Ducom 21 Aug '24

21 Aug '24

Hi I'm reaching out to check on the status of the XFS deadlock issue with RBD in hyperconverged environments, as detailed in Ceph tracker issue #43910 ( https://tracker.ceph.com/issues/43910?tab=history) It looks like there hasn’t been much activity on this for a while, and I'm wondering if there have been any updates or if it’s just been lost in the issue volume. The issue is a deadlock when using XFS, leading Rook to recommend using EXT4 instead. However, kernel 5.6 introduced PR_SET_IO_FLUSHER, which should allow Ceph to handle this scenario better. Versions lacking PR_SET_IO_FLUSHER are either already EOL or soon will be, with the exception of kernel 5.4. While EXT4 works according to the Ceph documentation, the limitations with XATTR on EXT4 are still a concern. It’s a bit unfortunate that Rook has to move away from XFS where Ceph recommends it. The issue on the Rook side : https://github.com/rook/rook/issues/3132#issuecomment-580508760 Could anyone provide an update on this Ceph issue or suggest how we might push it forward? Any insights would be really appreciated, as this impacts the broader use of XFS in hyperconverged Ceph setups deployed with Rook. Thanks in advance for your help.

2 1

weird outage of ceph
by Simon Oosthoek 21 Aug '24

21 Aug '24

Hi We had a really weird outage today of ceph and I wonder how it came about. The problem seems to have started around midnight, I still need to look if it was to the extend I found it in this morning or if it grew more gradually, but when I found it several osd servers had most or all osd processes down, to the point where our EC 8+3 buckets didn't work anymore. Restarting the servers or the services turned out to be the way to quickly recover from this. I see some of our OSDs are coming close to (but not quite) 80-85% full, There are many times when I've seen an overfull error lead to cascading and catastrophic failures. I suspect this may have been one of them. Which brings me to another question, why is our balancer doing so badly at balancing the OSDs? It's configured with upmap mode and it should work great with the amount of PGs per OSD we have, but it is letting some OSD's reach 80% full and others not yet 50% full (we're just over 61% full in total). The current health status is: HEALTH_WARN Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull pg 30.3fc is active+remapped+backfill_wait+backfill_toofull, acting [66,105,124,113,89,132,206,242,179] I've started reweighting again, because the balancer is not doing it's job in our cluster for some reason... Below is our dashboard overview, you can see the start and recovery in the 24h graph... Cheers /Simon [image: image.png] -- I'm using my gmail.com address, because the gmail.com dmarc policy is "none", some mail servers will reject this (microsoft?) others will instead allow this when I send mail to a mailling list which has not yet been configured to send mail "on behalf of" the sender, but rather do a kind of "forward". The latter situation causes dkim/dmarc failures and the dmarc policy will be applied. see https://wiki.list.org/DEV/DMARC for more details

4 5

Re: Cephfs mds node already exists crashes mds
by Tarrago, Eli (RIS-BCT) 21 Aug '24

21 Aug '24

Here is the backtrace from a ceph crash ceph crash info '2024-08-20T16:07:39.319197Z_8bcdf3df-f9b5-451a-b971-16f8190ab351' { "assert_condition": "!p", "assert_file": "/build/ceph-18.2.4/src/mds/MDCache.cc", "assert_func": "void MDCache::add_inode(CInode*)", "assert_line": 251, "assert_msg": "/build/ceph-18.2.4/src/mds/MDCache.cc: In function 'void MDCache::add_inode(CInode*)' thread 7f0551248700 time 2024-08-20T12:07:39.313490-0400\n/build/ceph-18.2.4/src/mds/MDCache.cc: 251: FAILED ceph_assert(!p)\n", "assert_thread_name": "ms_dispatch", "backtrace": [ "/lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f055736a420]", "gsignal()", "abort()", "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x182) [0x7f05576e12c9]", "/usr/lib/ceph/libceph-common.so.2(+0x32d42b) [0x7f05576e142b]", "(MDCache::add_inode(CInode*)+0x348) [0x55f4f007b7e8]", "(Server::prepare_new_inode(boost::intrusive_ptr<MDRequestImpl>&, CDir*, inodeno_t, unsigned int, file_layout_t const*)+0x7b3) [0x55f4f00023a3]", "(Server::handle_client_mkdir(boost::intrusive_ptr<MDRequestImpl>&)+0x1cd) [0x55f4f000546d]", "(MDSContext::complete(int)+0x5f) [0x55f4f02b72cf]", "(void finish_contexts<std::vector<MDSContext*, std::allocator<MDSContext*> > >(ceph::common::CephContext*, std::vector<MDSContext*, std::allocator<MDSContext*> >&, int)+0x90) [0x55f4eff5c730]", "(MDSCacheObject::finish_waiting(unsigned long, int)+0x5c) [0x55f4f02c7dbc]", "(Locker::eval_gather(SimpleLock*, bool, bool*, std::vector<MDSContext*, std::allocator<MDSContext*> >*)+0x12dc) [0x55f4f016fe2c]", "(Locker::dispatch(boost::intrusive_ptr<Message const> const&)+0x16c) [0x55f4f0183b1c]", "(MDSRank::_dispatch(boost::intrusive_ptr<Message const> const&, bool)+0x5a6) [0x55f4eff6b1a6]", "(MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr<Message const> const&)+0x5c) [0x55f4eff6b79c]", "(MDSDaemon::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x1bf) [0x55f4eff5530f]", "(Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0x468) [0x7f055795c448]", "(DispatchQueue::entry()+0x657) [0x7f05579599e7]", "(DispatchQueue::DispatchThread::entry()+0x11) [0x7f0557a20b11]", "/lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f055735e609]", "clone()" ], "ceph_version": "18.2.4", "crash_id": "2024-08-20T16:07:39.319197Z_8bcdf3df-f9b5-451a-b971-16f8190ab351", "entity_name": "mds.mds0202", "os_id": "ubuntu", "os_name": "Ubuntu", "os_version": "20.04.6 LTS (Focal Fossa)", "os_version_id": "20.04", "process_name": "ceph-mds", "stack_sig": "e51039f21ed17c3220d021c313d5af656ee59d9ccfb3e6d6f991bcf245dbf000", "timestamp": "2024-08-20T16:07:39.319197Z", "utsname_hostname": "mds02", "utsname_machine": "x86_64", "utsname_release": "5.4.0-190-generic", "utsname_sysname": "Linux", "utsname_version": "#210-Ubuntu SMP Fri Jul 5 17:03:38 UTC 2024" } ________________________________ The information contained in this e-mail message is intended only for the personal and confidential use of the recipient(s) named above. This message may be an attorney-client communication and/or work product and as such is privileged and confidential. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail, and delete the original message.

3 2

squid 19.1.1 RC QE validation status
by Yuri Weinstein 20 Aug '24

20 Aug '24

Details of this release are summarized here: https://tracker.ceph.com/issues/67340#note-1 Release Notes - N/A LRC upgrade - N/A Gibba upgrade -TBD Seeking approvals/reviews for: rados - Radek, Laura (https://github.com/ceph/ceph/pull/59020 is being tested and will be cherry-picked when ready) rgw - Eric, Adam E fs - Venky orch - Adam King rbd, krbd - Ilya quincy-x, reef-x - Laura, Neha powercycle - Brad crimson-rados - Matan, Samuel ceph-volume - Guillaume Pls let me know if any tests were missed from this list.

9 22

Cephfs mds node already exists crashes mds
by Tarrago, Eli (RIS-BCT) 20 Aug '24

20 Aug '24

Good Morning Ceph Users, I’m currently engaged in troubleshooting an issue and I wanted to post here to get some feedback. If there is no response or feedback that this looks like a bug, then I’ll write up a bug report. Cluster: Reef 18.2.4 Ubuntu 20.04 ceph -s cluster: id: 93e49b2e-2b56-4faa-xxxx-88ceb615daac health: HEALTH_WARN 1 filesystem is degraded 49 large omap objects 1 MDSs report oversized cache 1 MDSs report slow requests 152 pgs not deep-scrubbed in time 14 daemons have recently crashed services: mon: 3 daemons, quorum mon01,mon03,mon02 (age 2w) mgr: mgr01(active, since 2w), standbys: mgr02, mgr03 mds: 16/16 daemons up, 1 standby, 1 hot standby osd: 170 osds: 167 up (since 3d), 165 in (since 3d) rgw: 3 daemons active (3 hosts, 1 zones) The error message received when mds crashes are: Aug 18 09:14:32 node02 ceph-mds[4098736]: log_channel(cluster) log [ERR] : loaded dup inode 0x100009e4175 [2,head] v12 at /volume/db2/RDP/sql02/innodb/20240817.103010/idm_dw/tmp/text-base, but inode 0x100009e417> Aug 18 09:14:32 node02 ceph-mds[4098736]: log_channel(cluster) log [ERR] : loaded dup inode 0x100009e4176 [2,head] v10 /volume/db2/RDP/sql02/innodb/20240817.103010/idm_dw/tmp/props, but inode 0x100009e4176.he> Current status of cephfs for this volume ceph fs status cephfs - 93 clients ====== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 clientreplay mds0206 126M 126M 123k 73 1 active mds0102 Reqs: 0 /s 10 13 11 0 2 active mds0204 Reqs: 0 /s 12.9k 12.9k 63.8k 58 3 active mds0203 Reqs: 0 /s 12.9k 12.9k 63.9k 56 4 active mds0207 Reqs: 0 /s 12.7k 12.7k 63.8k 7 5 active mds0201 Reqs: 0 /s 12.8k 12.8k 63.8k 3 6 active mds0106 Reqs: 0 /s 12.8k 12.8k 63.8k 9 7 active mds0105 Reqs: 0 /s 12.8k 12.8k 63.8k 4 8 active mds0205 Reqs: 0 /s 12.4k 12.4k 61.7k 1 9 active mds0208 Reqs: 0 /s 12.5k 12.5k 61.7k 0 10 active mds0107 Reqs: 0 /s 12.5k 12.5k 61.7k 1 11 active mds0104 Reqs: 0 /s 12.4k 12.5k 61.7k 0 12 active mds0101 Reqs: 0 /s 12.5k 12.5k 61.7k 0 13 active mds0103 Reqs: 0 /s 12.5k 12.5k 61.7k 0 14 active mds0108 Reqs: 0 /s 12.4k 12.4k 61.7k 0 POOL TYPE USED AVAIL cephfs_metadata metadata 232G 325T cephfs_data data 439T 325T This error cause mds0206, rank0 to restart. This happened 14 times in the last 48 hours. ________________________________ The information contained in this e-mail message is intended only for the personal and confidential use of the recipient(s) named above. This message may be an attorney-client communication and/or work product and as such is privileged and confidential. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail, and delete the original message.

1 0

CephFS troubleshooting
by Eugenio Tampieri 20 Aug '24

20 Aug '24

Hello, I'm writing to troubleshoot an otherwise functional Ceph quincy cluster that has issues with cephfs. I cannot mount it with ceph-fuse (it gets stuck), and if I mount it with NFS I can list the directories but I cannot read or write anything. Here's the output of ceph -s cluster: id: 3b92e270-1dd6-11ee-a738-000c2937f0ec health: HEALTH_WARN mon ceph-storage-a is low on available space 1 daemons have recently crashed too many PGs per OSD (328 > max 250) services: mon: 5 daemons, quorum ceph-mon-a,ceph-storage-a,ceph-mon-b,ceph-storage-c,ceph-storage-d (age 105m) mgr: ceph-storage-a.ioenwq(active, since 106m), standbys: ceph-mon-a.tiosea mds: 1/1 daemons up, 2 standby osd: 4 osds: 4 up (since 104m), 4 in (since 24h) rbd-mirror: 2 daemons active (2 hosts) rgw: 2 daemons active (2 hosts, 1 zones) data: volumes: 1/1 healthy pools: 13 pools, 481 pgs objects: 231.83k objects, 648 GiB usage: 1.3 TiB used, 1.8 TiB / 3.1 TiB avail pgs: 481 active+clean io: client: 1.5 KiB/s rd, 8.6 KiB/s wr, 1 op/s rd, 0 op/s wr Best regards, Eugenio Tampieri

2 1

The snaptrim queue of PGs has not decreased for several days.
by Giovanna Ratini 20 Aug '24

20 Aug '24

Hello all, We use Ceph (v18.2.2) and Rook (1.14.3) as the CSI for a Kubernetes environment. Last week, we had a problem with the MDS falling behind on trimming every 4-5 days (GitHub issue link). We resolved the issue using the steps outlined in the GitHub issue. We have 3 hosts (I know, I need to increase this as soon as possible, and I will!) and 6 OSDs. After running the commands: ceph config set mds mds_dir_max_commit_size 80, ceph fs fail <fs_name>, and ceph fs set <fs_name> joinable true, After that, the snaptrim queue for our PGs has stopped decreasing. All PGs of our CephFS are in either active+clean+snaptrim_wait or active+clean+snaptrim states. For example, the PG 3.12 is in the active+clean+snaptrim state, and its snap_trimq_len was 4077 yesterday but has increased to 4538 today. I increased the osd_snap_trim_priority to 10 (ceph config set osd osd_snap_trim_priority 10), but it didn't help. Only the PGs of our CephFS have this problem. Do you have any ideas on how we can resolve this issue? Thanks in advance, Giovanna p.s. I'm not a ceph expert :-). Faulkener asked me for more information, so here it is: MDS Memory: 11GB mds_cache_memory_limit: 11,811,160,064 bytes root@kube-master02:~# ceph fs snap-schedule status / { "fs": "rook-cephfs", "subvol": null, "path": "/", "rel_path": "/", "schedule": "3h", "retention": {"h": 24, "w": 4}, "start": "2024-05-05T00:00:00", "created": "2024-05-05T17:28:18", "first": "2024-05-05T18:00:00", "last": "2024-08-15T18:00:00", "last_pruned": "2024-08-15T18:00:00", "created_count": 817, "pruned_count": 817, "active": true } I do not understand if the snapshots in the PGs are correlated with the snapshots on CephFS. Until we encountered the issue with the "MDS falling behind on trimming every 4-5 days," we didn't have any problems with snapshots. Could someone explain me this or send me to the documentation? Thank you

3 15

Bug with Cephadm module osd service preventing orchestrator start
by Benjamin Huth 20 Aug '24

20 Aug '24

Hey there, so I went to upgrade my ceph from 18.2.2 to 18.2.4 and have encountered a problem with my managers. After they had been upgraded, my ceph orch module broke because the cephadm module would not load. This obviously halted the update because you can't really update without the orchestrator. Here are the logs related to why the cephadm module fails to start: https://pastebin.com/SzHbEDVA and the relevent part here: "backtrace": [ " File \\"/usr/share/ceph/mgr/cephadm/module.py\\", line 591, in __init__\\n self.to_remove_osds.load_from_store()", " File \\"/usr/share/ceph/mgr/cephadm/services/osd.py\\", line 918, in load_from_store\\n osd_obj = OSD.from_json(osd, rm_util=self.rm_util)", " File \\"/usr/share/ceph/mgr/cephadm/services/osd.py\\", line 783, in from_json\\n return cls(**inp)", "TypeError: __init__() got an unexpected keyword argument 'original_weight'" ] Unfortunately, I am at a loss to what passes this the original weight argument. I have attempted to migrate back to 18.2.2 and successfully redeployed a manager of that version, but it also has the same issue with the cephadm module. I believe this may be because I recently started several OSD drains, then canceled them, causing this to manifest once the managers restarted. I went through a good bit of the source and found the module at fault: https://github.com/ceph/ceph/blob/e0dd396793b679922e487332a2a4bc48e024a42f/… as well as the PR that caused the issue: https://github.com/ceph/ceph/commit/ba7fac074fb5ad072fcad10862f75c0a26a7591d I unfortunately am not familiar enough with the ceph source to find the ceph-config values I need to delete or smart enough to fix this myself. Any help would be super appreciated. Thanks!

2 7

Prometheus and "404" error on console
by Tim Holloway 19 Aug '24

19 Aug '24

Although I'm seeing this in Pacific, it appears to be a perennial issue with no well-documented solution. The dashboard home screen is flooded with popups saying "404 - Not Found Could not reach Prometheus's API on http://ceph1234.mydomain.com:9095/api/v1 " If I was a slack-jawed PHB casually wandering into the operations center and saw that, I'd probably doubt that Ceph was a good product decision. The "404" indicates that the Prometheus server is running and accepting requests, but all the "404" says is that whatever requests it's receiving are meaningless to it. Presumably either some handler needs to be jacked into Prometheus or the sender needs to be notified to desist. Unfortunately, "404" doesn't provide any clues. Ideally, ceph should log something more specific by default, but failing that, if anyone knows how to shut it up (cleanly), I'd appreciate knowing! Tim

2 2

CLT meeting notes August 19th 2024
by Adam King 19 Aug '24

19 Aug '24

- [travisn] Arm64 OSDs crashing on v18.2.4, need a fix in v18.2.5 - https://tracker.ceph.com/issues/67213 - tcmalloc issue, solved by rebuilding the gperftools package - Travis to reach out to Rongqi Sun about the issue - moving away from tcmalloc would probably cause performance regressions and is therefore undesirable - [zdover] - ceph-users list moderation minute - the whole Python mess seems to be fixed. Release branches on docs.ceph.com are now updating. - [venky] - CVE testing process/documentation - https://docs.ceph.com/en/reef/dev/release-process/?highlight=ceph%20release… - does not include the testing procedure - currently, even when pointed towards branch on private repo, tries to pull packages from ceph-ci - Venky and Zac Dover to coordinate on creating the docs - should this documentation be publicly available? - only once CI/build issues that require the builds being in public places are fixed - just in a google doc for now - [Eric I] - How do people feel that Read the Docs is super-imposing ads over our docs? So far I've only seen one for www.ethicalads.io . - comes with Read the Docs, not enough of an issue to justify moving away from it - [DanV] - Cephalocon CFP is closed : 175 submissions! (for around 50 sessions) - Program Committee reviewing until the end of August. - On-site Dev Summit being planned for Dec 3, will go live here: https://indico.cern.ch/e/ceph-developer-summit - 3-4 rooms at CERN IT for dev whiteboarding. power users session in the afternoon. - If you still want to submit a CFP, talk to Dan ASAP - still looking for additional sponsors - 19.1.1 - almost ready, need ceph-volume approval and Gibba upgrade - 17.2.8 - mark PRs if you need them in the release - Yuri to start testing soon - core team to potentially do scrub of backport PRs

1 0

cephadm module fails to load with "got an unexpected keyword argument"
by Alex Sanderson 19 Aug '24

19 Aug '24

Hi everyone, I recently upgraded from Quincy to Reef v18.2.4 and my dashboard and mgr systems have been broken since. Since the upgrade I was slowly removing and zapping osd's that still had the 64k "bluestore_bdev_block_size" and decided to have a look at the dashboard problem. I restarted the mgrs one at a time and they showed in status that they working but actually the cephadm module was failing. The systems were all upgraded to 18 via orch from 17.2.7 and are running the official docker images. This is the error message: debug 2024-08-13T10:08:11.736+0000 7fd30dbe0640 -1 mgr load Failed to construct class in 'cephadm' debug 2024-08-13T10:08:11.736+0000 7fd30dbe0640 -1 mgr load Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/module.py", line 619, in __init__ self.to_remove_osds.load_from_store() File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 924, in load_from_store osd_obj = OSD.from_json(osd, rm_util=self.rm_util) File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 789, in from_json return cls(**inp) TypeError: __init__() got an unexpected keyword argument 'original_weight' debug 2024-08-13T10:08:11.736+0000 7fd30dbe0640 -1 mgr operator() Failed to run module in active mode ('cephadm') The config-key responsible was mgr/cephadm/osd_remove_queue This is what it looked like before. After removing the original_weight field and setting the variable again, the cephadm module loads and orch works. It seems like a bug. [{"osd_id": 89, "started": true, "draining": true, "stopped": false, "replace": false, "force": true, "zap": true, "hostname": "goanna", "original_weight": 0.930999755859375, "drain_started_at": "2024-08-12T13:21:04.458019Z", "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2024-08-12T13:20:40.021185Z"}, {"osd_id": 37, "started": true, "draining": true, "stopped": false, "replace": false, "force": true, "zap": true, "hostname": "gsceph1osd05", "original_weight": 4, "drain_started_at": "2024-08-10T06:30:37.569931Z", "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2024-08-10T06:30:19.729143Z"}, {"osd_id": 47, "started": true, "draining": true, "stopped": false, "replace": false, "force": true, "zap": true, "hostname": "gsceph1osd07", "original_weight": 4, "drain_started_at": "2024-08-10T09:54:49.132830Z", "drain_stopped_at": null, "drain_done_at": null, "process_started_at": "2024-08-10T09:54:34.367655Z"}] I thought I should put this out there in case anyone else was having a weird issue with a keyword argument problem. It did not fix the problem with the dashboard, still working on that. Alex

2 1

memory leak in mds?
by Dario Graña 19 Aug '24

19 Aug '24

Hi all, We’re experiencing an issue with CephFS. I think we are facing this issue <https://tracker.ceph.com/issues/64977>. The main symptom is that the MDS starts using a lot of memory within a few minutes and finally it gets killed by OS (Out Of Memory). Sometimes it happens once a week and sometimes 2 times a day. We are running ceph quincy 17.2.7 on both the cluster and clients. I have read through some emails on the mailing list about it, but I didn't find a workaround. Does anyone have any suggestions? Thanks in advance. -- Dario Graña PIC (Port d'Informació Científica) Campus UAB, Edificio D E-08193 Bellaterra, Barcelona http://www.pic.es Avis - Aviso - Legal Notice: http://legal.ifae.es

3 4

Re: ceph device ls missing disks
by Alfredo Rezinovsky 18 Aug '24

18 Aug '24

No, I don´t monitor drivers in non-CEPH nodes. All my non-CEPH nodes are disposables. Only CEPH has data I can't lose El jue, 15 ago 2024 a las 20:49, Anthony D'Atri (<anthony.datri(a)gmail.com>) escribió: > Do you monitor OS drives on non-Ceph nodes? > > > > > On Aug 15, 2024, at 8:17 AM, Alfredo Rezinovsky <alfrenovsky(a)gmail.com> > wrote: > > > > With: "ceph device ls" I can see which physical disks are used by which > > daemons. > > > > OSD data and wal disks are ok. > > > > MON appears using operating system disks, or the disks where > /var/lib/ceph > > is mounted, also OK. > > > > The problem is that non monitor nodes operating system disks are missing > > when they are really being used by all the daemons in the node. > > > > The problem with this is that there are no monitoring for those disks > when > > they are essential and failure points for all the daemons running in > those > > nodes. > > > > I think the OS devices should appear used by all the daemons having > > essential information in those disks. > > > > -- > > Alfrenovsky > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > -- Alfrenovsky

1 0

Identify laggy PGs
by Boris 18 Aug '24

18 Aug '24

Hi, currently we encouter laggy PGs and I would like to find out what is causing it. I suspect it might be one or more failing OSDs. We had flapping OSDs and I synced one out, which helped with the flapping, but it doesn't help with the laggy ones. Any tooling to identify or count PG performance and map that to OSDs? -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

5 10

orch adoption and disk encryption without cephx?
by Boris 18 Aug '24

18 Aug '24

Hi, I have some legacy clusters that I can not move to cephx due to customer workload. (We have a plan to move everything iterative to a new cluster, but that might still take a lot time). I would like to adopt the orchestrator and use the ceph disk encryption feature. Is this possible without using cephx? It is only rbd workload. I ask, because I remember that RGW actually required cephx after some update, because there might be certificates in the mon db. And it took me some time to find the problem and the solution back then. I would like to not encounter this with rbd workload :) Cheers Boris

1 0

Accidentally created systemd units for OSDs
by Dan O'Brien 17 Aug '24

17 Aug '24

I was [poorly] following the instructions for migrating the wal/db to an SSD https://docs.clyso.com/blog/ceph-volume-create-wal-db-on-separate-device-fo… and I didn't add the '--no-systemd' when I did 'ceph-volume lvm activate' command (3 f***ing times). The result is that I've "twinned" 3 of my OSDs: There's a container version managed by cephadm and there's an instantiated systemd unit that runs directly. Surprisingly, this has not done a lot of damage, but it does result in the dashboard reporting 3 failed cephadm daemons when the "native" OSDs start before the containerized ones. I've disabled the systemd units for ceph-osd(a)9.service, ceph-osd(a)11.service and ceph-osd(a)25.service, but I'd like to remove them completely. I will eventually badger The Google into giving me an answer, but could someone tell me what I need to do? The semester starts soon and I don't really have the bandwidth for this right now. Thanks in advance. I will forever be in your debt. (Seriously, I'm ready to give you a kidney, if you need it.)

4 8

Ceph Logging Configuration and "Large omap objects found"
by Janek Bevendorff 16 Aug '24

16 Aug '24

Hi, I have a bunch of long-standing struggles with the way Ceph handles logging and I cannot figured out how to solve them. These issues are basically the following: - The log config options are utterly confusing and very badly documented - Mon file logs are spammed with DBG-level cluster logs, no matter what I configure - syslog/journald gets only part of the messages, but file logs must be turned off due to the spam above - "audit" channel logs cannot be configured at all From this, my following needs and questions arise, perhaps you can help: - I want to receive cluster and audit logs on the MONs with level "warn" or at most "info" - I want everything to go to journald - Where to cluster logs go if "clog_to_monitors" is off? - What's the difference between the "mon_cluster_log_*" and "clog_*" settings? - What the hell does "mon_cluster_log_to_syslog_facility" do and what does "audit=local0" mean or do? A very annoying symptom of the wonky logging config is that I cannot debug the infamous "LARGE_OMAP_OBJECTS" warning. It says "Search the cluster log for 'Large omap object found' for more details.", but I cannot do that, because without enabling the file-logging flood gates, I never receive the required cluster log info at the monitors and there seems to be no other way to debug this than to grep the cluster log (why??). My current log config is the following: global advanced clog_to_monitors true global basic err_to_syslog true global basic log_to_file false global basic log_to_stderr false global basic log_to_syslog true mon advanced mon_cluster_log_file /dev/null mon advanced mon_cluster_log_to_file false mon advanced mon_cluster_log_to_stderr false mon advanced mon_cluster_log_to_syslog true mon advanced mon_cluster_log_to_syslog_facility daemon mon advanced mon_cluster_log_to_syslog_level warn Any help solving this conundrum is much appreciated. Thanks! Janek

2 10

Last Call for Cephalocon T-Shirt Contest
by Noah Lehman 15 Aug '24

15 Aug '24

Hi Ceph community, The deadline for the Cephalocon t-shirt design contest is tomorrow, Friday August 16th. You don’t need to be an artist or a graphics designer, all we ask is that it be original art! If you'd like to submit a proposal for the official t-shirt of Cephalocon 2024, you can do so here <https://ceph.io/en/news/blog/2024/cephalocon24-tshirt-contest/>. Best, Noah

1 0

ceph device ls missing disks
by Alfredo Rezinovsky 15 Aug '24

15 Aug '24

With: "ceph device ls" I can see which physical disks are used by which daemons. OSD data and wal disks are ok. MON appears using operating system disks, or the disks where /var/lib/ceph is mounted, also OK. The problem is that non monitor nodes operating system disks are missing when they are really being used by all the daemons in the node. The problem with this is that there are no monitoring for those disks when they are essential and failure points for all the daemons running in those nodes. I think the OS devices should appear used by all the daemons having essential information in those disks. -- Alfrenovsky

1 0

Cephadm Upgrade Issue
by Alex Hussein-Kershaw (HE/HIM) 14 Aug '24

14 Aug '24

Hi Folks, I'm prototyping the upgrade process for our Ceph Clusters. I've adopted the Cluster following the docs, that works nicely 🙂 I then load my docker image into a locally running container registry, as I'm in a disconnected environment. I have a test Cluster with 3 VMs and no data, adopted at Octopus and upgrading to Pacific. I'm running a MON, MGR, MDS and OSD on each VM. I then attempt to upgrade: ceph orch upgrade start --image localhost:5000/ceph/pacific:v16.2.15 Lots of logs below, but the summary appears to be that we initially fail to upgrade the managers and get into a bad state. It looks like there is some confusion in manager naming, and we end up with two managers on each machine instead of one. Eventually Ceph reports a health warning: $ ceph -s cluster: id: e773d9c2-6d8d-4413-8e8f-e38f248f5959 health: HEALTH_ERR 1 failed cephadm daemon(s) Module 'cephadm' has failed: 'cephadm' That does seem to eventually clean its self up and the upgrade appears to have completed ("ceph versions" shows everything on Pacific), but it feels a bit bumpy. Hoping someone has some guidance here. The containers on one host during upgrade are shown below. Notice I somehow have two managers, where the names are a single character different (a "-" replaced with a "."): $ docker ps | grep mgr 2143b6f0e0e6 localhost:5000/ceph/pacific:v16.2.15 "/usr/bin/ceph-mgr -…" About a minute ago Up About a minute ceph-e773d9c2-6d8d-4413-8e8f-e38f248f5959-mgr.raynor-sc-2 59c8cfddac64 ceph-daemon:v5.0.12-stable-5.0-octopus-centos-8 "/usr/bin/ceph-mgr -…" 14 minutes ago Up 14 minutes ceph-e773d9c2-6d8d-4413-8e8f-e38f248f5959-mgr-raynor-sc-2 In the output of "ceph -w" I see this sort of stuff: 2024-08-14T13:45:13.003405+0000 mon.raynor-sc-1 [INF] Manager daemon raynor-sc-3 is now available 2024-08-14T13:45:23.179699+0000 mon.raynor-sc-1 [ERR] Health check failed: Module 'cephadm' has failed: 'cephadm' (MGR_MODULE_ERROR) 2024-08-14T13:45:22.372376+0000 mgr.raynor-sc-3 [ERR] Unhandled exception from module 'cephadm' while running on mgr.raynor-sc-3: 'cephadm' 2024-08-14T13:45:24.761961+0000 mon.raynor-sc-1 [INF] Active manager daemon raynor-sc-3 restarted 2024-08-14T13:45:24.766395+0000 mon.raynor-sc-1 [INF] Activating manager daemon raynor-sc-3 2024-08-14T13:45:31.800989+0000 mon.raynor-sc-1 [INF] Manager daemon raynor-sc-3 is now available 2024-08-14T13:45:32.874227+0000 mon.raynor-sc-1 [INF] Health check cleared: MGR_MODULE_ERROR (was: Module 'cephadm' has failed: 'cephadm') 2024-08-14T13:45:32.874269+0000 mon.raynor-sc-1 [INF] Cluster is now healthy 2024-08-14T13:45:33.664602+0000 mon.raynor-sc-1 [INF] Active manager daemon raynor-sc-3 restarted 2024-08-14T13:45:33.671809+0000 mon.raynor-sc-1 [INF] Activating manager daemon raynor-sc-3 2024-08-14T13:45:34.050292+0000 mon.raynor-sc-1 [INF] Manager daemon raynor-sc-3 is now available 2024-08-14T13:45:38.260385+0000 mon.raynor-sc-1 [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-08-14T13:45:43.462665+0000 mgr.raynor-sc-3 [ERR] Unhandled exception from module 'cephadm' while running on mgr.raynor-sc-3: 'cephadm' 2024-08-14T13:45:44.770711+0000 mon.raynor-sc-1 [ERR] Health check failed: Module 'cephadm' has failed: 'cephadm' (MGR_MODULE_ERROR) 2024-08-14T13:45:45.668379+0000 mon.raynor-sc-1 [INF] Active manager daemon raynor-sc-3 restarted 2024-08-14T13:45:45.673206+0000 mon.raynor-sc-1 [INF] Activating manager daemon raynor-sc-3 2024-08-14T13:45:45.673316+0000 mon.raynor-sc-1 [INF] Active manager daemon raynor-sc-3 restarted 2024-08-14T13:45:45.689515+0000 mon.raynor-sc-1 [INF] Active manager daemon raynor-sc-3 restarted 2024-08-14T13:45:45.694315+0000 mon.raynor-sc-1 [INF] Activating manager daemon raynor-sc-3 2024-08-14T13:45:47.671192+0000 mon.raynor-sc-1 [INF] Active manager daemon raynor-sc-3 restarted 2024-08-14T13:45:47.674805+0000 mon.raynor-sc-1 [INF] Activating manager daemon raynor-sc-3 2024-08-14T13:45:47.675037+0000 mon.raynor-sc-1 [INF] Active manager daemon raynor-sc-3 restarted 2024-08-14T13:45:47.697264+0000 mon.raynor-sc-1 [INF] Active manager daemon raynor-sc-3 restarted 2024-08-14T13:45:47.700886+0000 mon.raynor-sc-1 [INF] Activating manager daemon raynor-sc-3 And in the output of "ceph -W cephadm" I see: 2024-08-14T13:40:32.214742+0000 mgr.raynor-sc-1 [INF] Upgrade: First pull of localhost:5000/ceph/pacific:v16.2.15 2024-08-14T13:40:34.108767+0000 mgr.raynor-sc-1 [INF] Upgrade: Target is localhost:5000/ceph/pacific:v16.2.15 with id 3c4eff6082ae7530e7eda038765ce400beb1bc1b8df67dffb45910eb45b06b2c 2024-08-14T13:40:34.112388+0000 mgr.raynor-sc-1 [INF] Upgrade: Checking mgr daemons... 2024-08-14T13:40:34.112722+0000 mgr.raynor-sc-1 [INF] Upgrade: Need to upgrade myself (mgr.raynor-sc-1) 2024-08-14T13:40:35.456432+0000 mgr.raynor-sc-1 [INF] It is presumed safe to stop ['mgr.raynor-sc-2'] 2024-08-14T13:40:35.456620+0000 mgr.raynor-sc-1 [INF] Upgrade: It is presumed safe to stop ['mgr.raynor-sc-2'] 2024-08-14T13:40:35.456771+0000 mgr.raynor-sc-1 [INF] Upgrade: Redeploying mgr.raynor-sc-2 2024-08-14T13:40:35.481790+0000 mgr.raynor-sc-1 [INF] Deploying daemon mgr.raynor-sc-2 on raynor-sc-2 2024-08-14T13:42:37.608895+0000 mgr.raynor-sc-1 [INF] refreshing raynor-sc-3 facts 2024-08-14T13:42:39.744098+0000 mgr.raynor-sc-1 [INF] refreshing raynor-sc-1 facts 2024-08-14T13:42:40.081740+0000 mgr.raynor-sc-1 [INF] refreshing raynor-sc-2 facts 2024-08-14T13:42:40.937375+0000 mgr.raynor-sc-1 [INF] Applying drive group all-available-devices on host raynor-sc-1... 2024-08-14T13:42:40.937732+0000 mgr.raynor-sc-1 [INF] Applying drive group all-available-devices on host raynor-sc-2... 2024-08-14T13:42:40.938079+0000 mgr.raynor-sc-1 [INF] Applying drive group all-available-devices on host raynor-sc-3... 2024-08-14T13:42:46.226231+0000 mgr.raynor-sc-1 [INF] Upgrade: Target is localhost:5000/ceph/pacific:v16.2.15 with id 3c4eff6082ae7530e7eda038765ce400beb1bc1b8df67dffb45910eb45b06b2c 2024-08-14T13:42:46.229604+0000 mgr.raynor-sc-1 [INF] Upgrade: Checking mgr daemons... 2024-08-14T13:42:46.229727+0000 mgr.raynor-sc-1 [INF] Upgrade: Need to upgrade myself (mgr.raynor-sc-1) 2024-08-14T13:42:47.696110+0000 mgr.raynor-sc-1 [INF] It is presumed safe to stop ['mgr.raynor-sc-3'] My take on this is that we first start to attempt upgrade of the mgr on raynor-sc-2, and don't seem to detect it's not quite worked. Interestingly it seems that there is a two minute gap between the deploying line and the following line, I wonder if something is failing to come up and we're proceeding after a timer expires? Any pointers are much appreciated. Many thanks, Alex

3 5

Bug with Cephadm module osd service preventing orchestrator start
by benjaminmhuth＠gmail.com 14 Aug '24

14 Aug '24

Hey there, so I went to upgrade my ceph from 18.2.2 to 18.2.4 and have encountered a problem with my managers. After they had been upgraded, my ceph orch module broke because the cephadm module would not load. This obviously halted the update because you can't really update without the orchestrator. Here are the logs related to why the cephadm module fails to start: https://pastebin.com/SzHbEDVA and the relevent part here: "backtrace": [ " File \"/usr/share/ceph/mgr/cephadm/module.py\", line 591, in __init__\n self.to_remove_osds.load_from_store()", " File \"/usr/share/ceph/mgr/cephadm/services/osd.py\", line 918, in load_from_store\n osd_obj = OSD.from_json(osd, rm_util=self.rm_util)", " File \"/usr/share/ceph/mgr/cephadm/services/osd.py\", line 783, in from_json\n return cls(**inp)", "TypeError: __init__() got an unexpected keyword argument 'original_weight'" ] Unfortunately, I am at a loss to what passes this the original weight argument. I have attempted to migrate back to 18.2.2 and successfully redeployed a manager of that version, but it also has the same issue with the cephadm module. I believe this may be because I recently started several OSD drains, then canceled them, causing this to manifest once the managers restarted. I've attempted to dig into this myself a bit and found the associated function in the ceph repo: https://github.com/ceph/ceph/blob/e0dd396793b679922e487332a2a4bc48e024a42f/… My issue may also have to do with this PR: https://github.com/ceph/ceph/commit/ba7fac074fb5ad072fcad10862f75c0a26a7591d I unfortunately do not have enough familiarity with the source to see where these values are being set, or to know what keys to remove from the ceph config. Hopefully someone who knows about this will see this and take a look for me! Thanks!

1 0

The snaptrim queue of PGs has not decreased for several days.
by Giovanna Ratini 14 Aug '24

14 Aug '24

Hello all, We use Ceph (v18.2.2) and Rook (1.14.3) as the CSI for a Kubernetes environment. Last week, we had a problem with the MDS falling behind on trimming every 4-5 days (GitHub issue link <https://github.com/rook/rook/issues/14220>). We resolved the issue using the steps outlined in the GitHub issue. We have 3 hosts (I know, I need to increase this as soon as possible, and I will!) and 6 OSDs. After running the commands: |ceph config set mds mds_dir_max_commit_size 80|, | | |ceph fs fail <fs_name>|, and | | |ceph fs set <fs_name> joinable true|, After that, the snaptrim queue for our PGs has stopped decreasing. All PGs of our CephFS are in either |active+clean+snaptrim_wait| or |active+clean+snaptrim| states. For example, the PG |3.12| is in the |active+clean+snaptrim| state, and its |snap_trimq_len| was 4077 yesterday but has increased to 4538 today. I increased the |osd_snap_trim_priority| to 10 (|ceph config set osd osd_snap_trim_priority 10|), but it didn't help. Only the PGs of our CephFS have this problem. Do you have any ideas on how we can resolve this issue? Thanks in advance, Giovanna

1 0

rbd du USED greater than PROVISIONED
by Murilo Morais 14 Aug '24

14 Aug '24

Good morning everyone! I am confused about the listing of the total amount used on a volume. It says that more than the amount provisioned is being used. The image contains a snapshot. Below is the output of the "rbd du" command: user@abc2:~# rbd info osr1_volume_ssd/volume-6e5f90ac-78e9-465e-8705-a4d476ebc019 rbd image 'volume-6e5f90ac-78e9-465e-8705-a4d476ebc019': size 1000 GiB in 256000 objects order 22 (4 MiB objects) snapshot_count: 1 id: 3ec2fef4cd0ad8 block_name_prefix: rbd_data.3ec2fef4cd0ad8 format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten op_features: flags: create_timestamp: Fri Jan 5 22:48:45 2024 access_timestamp: Wed Aug 14 10:46:06 2024 modify_timestamp: Wed Aug 14 10:47:02 2024 user@abc2:~# rbd du osr1_volume_ssd/volume-6e5f90ac-78e9-465e-8705-a4d476ebc019 NAME PROVISIONED USED volume-6e5f90ac-78e9-465e-8705-a4d476ebc019@snapshot-23af41b0-ca7b-45df-991f-ffbe95512dbe 1000 GiB 710 GiB volume-6e5f90ac-78e9-465e-8705-a4d476ebc019 1000 GiB 404 GiB <TOTAL> 1000 GiB 1.1 TiB In this case, is that really the case or is it some kind of BUG? For the record, we are still on version 17.2.3.

2 2

Cephadm and the "--data-dir" Argument
by Alex Hussein-Kershaw (HE/HIM) 14 Aug '24

14 Aug '24

Hi Folks, I'm trying to use the --data-dir argument of cephadm when bootstrapping a Storage Cluster. It looks like exactly what I need, where my use case is that I want to data files onto a persistent disk, such that I can below away my VMs while retaining the files. Everything looks good and the bootstrap command completes. For reference I am running this command: "sudo cephadm --image "ceph/squid:v19.1.0" --docker --data-dir /cephconfig/var/lib/ceph bootstrap --mon-ip 10.235.22.23 --ssh-user qs-admin --ssh-private-key /home/qs-admin/.ssh/id_rsa --ssh-public-key /home/qs-admin/.ssh/id_rsa.pub --output-dir /cephconfig/etc/ceph --skip-dashboard --skip-monitoring-stack --skip-pull --config my.conf" However, when I then try to continue with the deployment of my Storage Cluster, I find that I can't authenticate with the monitors. I run the suggested command to drop into a cephadm shell which then can't speak to the Storage Cluster. For example: $ ceph -s 2024-08-12T10:47:07.862+0000 7f998e59c640 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2,1] [errno 13] RADOS permission denied (error connecting to the cluster) In the MON logs at the same time I can see: "cephx server client.admin: unexpected key: req.key=2c62e1471f111d12 expected_key=d18ce06d18f116b4" In the systemd unit files created I see: ... ExecStart=/bin/bash /var/lib/ceph/64415fba-58b0-11ef-9d27-005056014e4f/%i/unit.run ExecStop=-/bin/bash -c 'bash /var/lib/ceph/64415fba-58b0-11ef-9d27-005056014e4f/%i/unit.stop' ExecStopPost=-/bin/bash /var/lib/ceph/64415fba-58b0-11ef-9d27-005056014e4f/%i/unit.poststop ... Which does not contain my data directory. Looking at the source template it appears that it should: ceph/src/cephadm/cephadmlib/templates/ceph.service.j2 at 616fbc1b181ce15e49281553b35ca215d2aa1053 · ceph/ceph (github.com)<https://github.com/ceph/ceph/blob/616fbc1b181ce15e49281553b35ca215d2aa1053/…> Manually modifying the unit file, reloading systemd and restarting the mon makes the authentication issue go away, although cephadm seems to be periodically rewriting my file and undoing the changes. Is there a templating bug in here? I note that there are no other variables being templated from the ctx in this jinja2 template so it seems likely it is broken. Many thanks, Alex

2 4

Upgrading RGW before cluster?
by Thomas Byrne - STFC UKRI 14 Aug '24

14 Aug '24

Hi all, The Ceph documentation has always recommended upgrading RGWs last when doing a upgrade. Is there a reason for this? As they're mostly just RADOS clients you could imagine the order doesn't matter as long as the cluster and RGW major versions are compatible. Our basic testing has shown no obvious issues (with Pacific RGWs and a Nautilus cluster FWIW). I'm asking because in our case it would be handy to upgrade our gateway infrastructure first, not for any new RGW features, but just for scheduling the operations. Is this a terrible idea? If it helps, this is the first step in a Nautilus -> Pacific -> Reef plan, with no cephadm. Thanks, Tom

4 3

Bluestore issue using 18.2.2
by Marianne Spiller 14 Aug '24

14 Aug '24

Hi, I am trying to gather experience on a Ceph STAGE cluster; it consists of virtual machines - which is not perfect, I know. The VMs are running Debian 12 and podman-4.3.1. There is practically no load on this Ceph - there is just one client using the storage, and it makes no noise. So this is what happened: * "During data consistency checks (scrub), at least one PG has been flagged as being damaged or inconsistent." * so I listed them (["2.3","2.58"]) * and tried to repair ("ceph pg repair 2.3", "ceph pg repair 2.58") * they both went well (resulting in "pgs: 129 active+clean"), but the cluster keeped its "HEALTH_WARN" state ("Too many repaired reads on 1 OSDs") * so I googled for this message; and the only thing I found was to restart the OSD to get rid of this message and - more important - the cluster WARN state ("ceph orch daemon restart osd.3") * after the restart, my cluster was still in WARN state - and complained about "2 PGs has been flagged as being damaged or inconsistent" - but other PGs on other OSDs * I "ceph pg repair"ed them, too, and the cluster's state was WARN afterwards, again ("Too many repaired reads on 1 OSDs") * when I restarted the OSD ("ceph orch daemon restart osd.2"), the crash occured; Ceph marked this OSD "down" and "out" and suspected a hardware issue, while the OSD HDDs in fact are QEMU "harddisks" * I can't judge whether it's a serious bug or just due to my non-optimal STAGE setup, so I'll attach the gzipped log of osd.2 I need help to understand what happened and how to prevent it in the future. What ist this "Too many repaired reads" and how to deal with it? Thanks a lot for reading, Marianne

4 5

Snapshot getting stuck
by Torkil Svensgaard 14 Aug '24

14 Aug '24

Ceph version 18.2.1. We have a nightly backup job snapshotting and exporting all RBDs used for libvirt VMs. Since a couple of weeks ago we've seen one or more getting stuck like this on 3 occasions, so intermittently: " zoperator@yggdrasil:~/backup$ cat /mnt/scratch/personal/zoperator/slurm-2888600.out Creating snap: 10% complete...2024-08-07T10:35:24.687+0200 7f5a2a44b640 0 --2- 172.21.14.135:0/3311921296 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f59fc002060 0x7f59fc0094b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-08-07T10:36:02.064+0200 7f5a2a44b640 0 --2- 172.21.14.135:0/3311921296 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59fc00a2b0 0x7f59fc015010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-08-07T10:37:38.191+0200 7f5a23fff640 0 --2- 172.21.14.135:0/3311921296 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5a0c063ea0 0x7f5a0c066320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-08-07T10:38:16.677+0200 7f5a28c48640 -1 librbd::ImageWatcher: 0x7f5a100076d0 image watch failed: 140024590898336, (107) Transport endpoint is not connected 2024-08-07T10:38:16.677+0200 7f5a28c48640 -1 librbd::Watcher: 0x7f5a100076d0 handle_error: handle=140024590898336: (107) Transport endpoint is not connected " The VM is also throwing stack traces from stuck I/O. Every VM seen affected by this map the RBD through multiple firewalls so that is likely a factor. Hypervisor <-> Palo Alto firewall <-> OpenBSD firewall <-> Ceph Any ideas? I haven't found anything in the ceph logs yet. Mvh. Torkil -- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil(a)drcmr.dk

3 3

All MDS's Crashed, Failed Assert
by m＠silvenga.com 14 Aug '24

14 Aug '24

I'm looking for guidance around how to recover after all MDS continue to crash with a failed assert during journal replay (no MON damage). Context: So I've been working through failed MDS for the past day, likely caused by a large snaptrim operation that caused the cluster to grind to a halt. After evicting all clients and restarting the MDS's (it appears the clients were overwhelming the MDS's). The MDS are failing to start with: debug -1> 2024-07-24T18:44:52.674+0000 7f7878c22700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.2.2/rpm/el8/BUILD/ceph-18.2.2/src/osdc/Journaler.cc: In function 'bool Journaler::try_read_entry(ceph::bufferlist&)' thread 7f7878c22700 time 2024-07-24T18:44:52.676027+0000 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.2.2/rpm/el8/BUILD/ceph-18.2.2/src/osdc/Journaler.cc: 1256: FAILED ceph_assert(start_ptr == read_pos) ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f788aa32e15] 2: /usr/lib64/ceph/libceph-common.so.2(+0x2a9fdb) [0x7f788aa32fdb] 3: (Journaler::try_read_entry(ceph::buffer::v15_2_0::list&)+0x132) [0x55555847ef32] 4: (MDLog::_replay_thread()+0xda) [0x555558436bea] 5: (MDLog::ReplayThread::entry()+0x11) [0x5555580e52d1] 6: /lib64/libpthread.so.0(+0x81ca) [0x7f78897d81ca] 7: clone() debug 0> 2024-07-24T18:44:52.674+0000 7f7878c22700 -1 *** Caught signal (Aborted) ** in thread 7f7878c22700 thread_name:md_log_replay ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable) 1: /lib64/libpthread.so.0(+0x12d20) [0x7f78897e2d20] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f788aa32e6f] 5: /usr/lib64/ceph/libceph-common.so.2(+0x2a9fdb) [0x7f788aa32fdb] 6: (Journaler::try_read_entry(ceph::buffer::v15_2_0::list&)+0x132) [0x55555847ef32] 7: (MDLog::_replay_thread()+0xda) [0x555558436bea] 8: (MDLog::ReplayThread::entry()+0x11) [0x5555580e52d1] 9: /lib64/libpthread.so.0(+0x81ca) [0x7f78897d81ca] 10: clone() Normally, there's three MDS are deployed, 1 active, one on hot standby. The cluster seems to believe that any restarted MDS is attempting to replay, but systemd reports an immediate crashed with a SIGABRT. ceph mds stat cephfs:1/1 {0=cephfs.sm1.esxjag=up:replay(laggy or crashed)} Redeploying the MDS's also continue to crash (suggesting a bad journal?)

3 2

Data recovery after resharding mishap
by Gauvain Pocentek 14 Aug '24

14 Aug '24

Hello list, We have made a mistake and dynamically resharded a bucket in a multi-site RGW setup running Quincy (support for this has been added in Reef). So we have now ~200 million objects still stored in the rados cluster, but completely removed from the bucket index (basically ceph has created a new index for the bucket). We would really like to recover these objects, but we are facing a few issues with our ideas. Any help would be appreciated. The main problem we face is that the rados objects contain binary data, where we expected json data. The RGW is configured to use zlib compression, so we think it can be the reason (although using zlib to decompress doesn't work). Has anyone already faced this, and managed to recover the data from rados objects? Another idea we have is to update the new bucket index to inject the old data. This looks possible as the object marker hasn't changed. We also have access to the old indexes/shards, so we could get all the omap key/value pairs and inject them in the new index. Has anyone been mad enough to try this by any chance? Any other idea to recover the data would help of course. Thank you! Gauvain

1 0

Fwd: [community] [OpenInfra Event Update] The CFP For OpenInfra Days NA is now open!
by Alvaro Soto 14 Aug '24

14 Aug '24

Hello Community!!!! I want to share this with you in case anybody wants to send anything to the CFP \o/ Feel free to reach out in case of questions; I hope to see you there. Cheers! ---------- Forwarded message --------- From: Helena Spease <helena(a)openinfra.dev> Date: Tue, Jul 16, 2024 at 1:46 PM Subject: [community] [OpenInfra Event Update] The CFP For OpenInfra Days NA is now open! To: <community(a)lists.openinfra.dev> Hi everyone, The CFP for the OpenInfra Days North America is now open! Submit your talk before the deadline on August 12. https://docs.google.com/forms/d/e/1FAIpQLSeFQBL1qGNegDWDHCVTLtu_wLf06w7JJkc… Pre-Event Timeline: - August 12: CFP ends - August 22: CFP notification of acceptance - August 26: Program announced ———OPENINFRA DAYS NORTH AMERICA——— October 15 & 16, 2024 IU Indianapolis Campus Center in Indianapolis, Indiana - Learn more: https://ittraining.iu.edu/explore-topics/titles/oid-iu/index.html - Sponsor: https://openinfra.dev/events/sponsorship#OIDNA ———OPENINFRA SUMMIT ASIA——— September 3 & 4, 2024 Suwon, South Korea - Learn more: https://2024.openinfraasia.org/ - Sponsor: https://2024.openinfraasia.org/sponsors/ - Schedule: https://2024.openinfraasia.org/a/schedule This schedule is not complete and only reflects OpenInfra Summit Asia sessions. OCP Summit Asia, Ceph Days Asia and OpenInfra Forum sessions are still being added. Session times are subject to change. Cheers, Helena helena(a)openinfra.dev Community Programs Coordinator The OpenInfra Foundation _______________________________________________ Community mailing list -- community(a)lists.openinfra.dev To unsubscribe send an email to community-leave(a)lists.openinfra.dev -- Alvaro Soto *Note: My work hours may not be your work hours. Please do not feel the need to respond during a time that is not convenient for you.* ---------------------------------------------------------- Great people talk about ideas, ordinary people talk about things, small people talk... about other people.

1 1

Announcing go-ceph v0.29.0
by Sven Anderson 13 Aug '24

13 Aug '24

We are happy to announce another release of the go-ceph API library. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.29.0 The library includes bindings that aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. The library also includes additional APIs that can be used to administer cephfs, rbd, and rgw subsystems. There are already a few consumers of this library in the wild, including the ceph-csi project. Sven

1 0

RBD Journaling seemingly getting stuck for some VMs after upgrade to Octopus
by Oliver Freyermuth 13 Aug '24

13 Aug '24

Dear Cephalopodians, we've successfully operated a "good old" Mimic cluster with primary RBD images, replicated via journaling to a "backup cluster" with Octopus, for the past years (i.e. one-way replication). We've now finally gotten around upgrading the cluster with the primary images to Octopus (and plan to upgrade further in the near future). After the upgrade, all MON+MGR-OSD+rbd_mirror daemons are running 15.2.17. We run three rbd-mirror daemons which all share the following client with auth in the "backup" cluster, to which they write: client.rbd_mirror_backup caps: [mon] profile rbd-mirror caps: [osd] profile rbd and the following shared client with auth in the "primary cluster" from which they are reading: client.rbd_mirror caps: [mon] profile rbd caps: [osd] profile rbd i.e. the same auth as described in the docs[0]. Checking on the primary cluster, we get: # rbd mirror pool status health: UNKNOWN daemon health: UNKNOWN image health: OK images: 288 total 288 replaying For some reason, some values are "unknown" here. But mirroring seems to work, as checking on the backup cluster reveals, see for example: # rbd mirror image status zabbix-test.example.com-disk2 zabbix-test.example.com-disk2: global_id: 1bdcb981-c1c5-4172-9583-be6a6cd996ec state: up+replaying description: replaying, {"bytes_per_second":8540.27,"entries_behind_primary":0,"entries_per_second":1.8,"non_primary_position":{"entry_tid":869176,"object_number":504,"tag_tid":1},"primary_position":{"entry_tid":11143,"object_number":7,"tag_tid":1}} service: rbd_mirror_backup on rbd-mirror002.example.com last_update: 2024-08-12 09:53:17 However, we do in some seemingly random cases see that journals are never advanced on the primary cluster — staying with the example above, on the primary cluster I find the following: # rbd journal status --image zabbix-test.physik.uni-bonn.de-disk2 minimum_set: 1 active_set: 126 registered clients: [id=, commit_position=[positions=[[object_number=7, tag_tid=1, entry_tid=11143], [object_number=6, tag_tid=1, entry_tid=11142], [object_number=5, tag_tid=1, entry_tid=11141], [object_number=4, tag_tid=1, entry_tid=11140]]], state=connected] [id=52b80bb0-a090-4f7d-9950-c8691ed8fee9, commit_position=[positions=[[object_number=505, tag_tid=1, entry_tid=869181], [object_number=504, tag_tid=1, entry_tid=869180], [object_number=507, tag_tid=1, entry_tid=869179], [object_number=506, tag_tid=1, entry_tid=869178]]], state=connected] As you can see, the minimum_set was not advanced. As can be seen in "mirror image status", it shows the strange output that non_primary_position seems much more advanced than primary_position. This seems to happen "at random" for only a few volumes... There are no other active clients apart from the actual VM (libvirt+qemu). As a quick fix, to purge journals piling up over and over, we've only found the "solution" to temporarily disable and then re-enable journaling for affected VM disks, which can be identified by: for A in $(rbd ls); do echo -n "$A: "; rbd --format=json journal status --image $A | jq '.active_set - .minimum_set'; done Any idea what is going wrong here? This did not happen with the primary cluster running Mimic and the backup cluster running Octopus before, and also did not happen when both were running Mimic. We plan to switch to snapshot-based mirroring in the future anyways, but it would be good to understand this strange issue in any case. Cheers, Oliver [0] https://docs.ceph.com/en/octopus/rbd/rbd-mirroring/ -- Oliver Freyermuth Universität Bonn Physikalisches Institut, Raum 1.047 Nußallee 12 53115 Bonn -- Tel.: +49 228 73 2367 Fax: +49 228 73 7869 --

2 6

Multi-Site sync error with multipart objects: Resource deadlock avoided
by Tino Lehnig 13 Aug '24

13 Aug '24

Hi, We've been trying to set up multi-site sync on two test VMs before rolling things out on actual production hardware. Both are running Ceph 18.2.4 deployed via cephadm. Host OS is Debian 12, container runtime is podman (switched from Debian 11 and docker.io, same error there). There is only one RGW daemon on each site. Ceph config is pretty much defaults. One thing I did change was setting rgw_relaxed_region_enforcement to true because the zonegroup got renamed from "default" during the switch to multi-site using the dashboard's assistant. There's nothing special like server-side encryption either. Our end goal is to replicate all RGW data from our current cluster to a new one. The Multi-Site configuration itself went pretty smoothly through the dashboard and pre-existing data started syncing right away. Unfortunately, not all objects made it. To be precise, none of the larger objects over the multipart threshold got synced. This is consistent for newly uploaded multipart objects as well. Curiously, it's working fine in the other direction, i.e. multipart uploads from the secondary zone do get synced to the master. Here are some relevant logs: From `radosgw-admin sync error list`: { "shard_id": 26, "entries": [ { "id": "1_1722598249.479766_23730.1", "section": "data", "name": "foobar/new:5160b406-4428-4fdc-9c5d-5ec9fe9404c0.12564119.3:7/logstash_1%3a8.12.2-1_amd64.deb", "timestamp": "2024-08-02T11:30:49.479766Z", "info": { "source_zone": "5160b406-4428-4fdc-9c5d-5ec9fe9404c0", "error_code": 35, "message": "failed to sync object(35) Resource deadlock avoided" } } ] }, From RGW on the receiving end: Aug 02 13:30:49 dev-ceph-single bash[754387]: debug 2024-08-02T11:30:49.474+0000 7f3a6243e640 0 rgw async rados processor: store->fetch_remote_obj() returned r=-35 Aug 02 13:30:49 dev-ceph-single bash[754387]: debug 2024-08-02T11:30:49.474+0000 7f3a36b7b640 2 req 7168648379339657593 0.000000000s :list_data_changes_log normalizing buckets and tenants Aug 02 13:30:49 dev-ceph-single bash[754387]: debug 2024-08-02T11:30:49.474+0000 7f3a36b7b640 2 req 7168648379339657593 0.003999872s :list_data_changes_log init permissions Aug 02 13:30:49 dev-ceph-single bash[754387]: debug 2024-08-02T11:30:49.478+0000 7f3a36b7b640 2 req 7168648379339657593 0.003999872s :list_data_changes_log recalculating target Aug 02 13:30:49 dev-ceph-single bash[754387]: debug 2024-08-02T11:30:49.478+0000 7f3a36b7b640 2 req 7168648379339657593 0.003999872s :list_data_changes_log reading permissions Aug 02 13:30:49 dev-ceph-single bash[754387]: debug 2024-08-02T11:30:49.478+0000 7f3a36b7b640 2 req 7168648379339657593 0.003999872s :list_data_changes_log init op Aug 02 13:30:49 dev-ceph-single bash[754387]: debug 2024-08-02T11:30:49.478+0000 7f3a36b7b640 2 req 7168648379339657593 0.003999872s :list_data_changes_log verifying op mask Aug 02 13:30:49 dev-ceph-single bash[754387]: debug 2024-08-02T11:30:49.478+0000 7f3a36b7b640 2 req 7168648379339657593 0.003999872s :list_data_changes_log verifying op permissions Aug 02 13:30:49 dev-ceph-single bash[754387]: debug 2024-08-02T11:30:49.478+0000 7f3a36b7b640 2 overriding permissions due to system operation Aug 02 13:30:49 dev-ceph-single bash[754387]: debug 2024-08-02T11:30:49.478+0000 7f3a36b7b640 2 req 7168648379339657593 0.003999872s :list_data_changes_log verifying op params Aug 02 13:30:49 dev-ceph-single bash[754387]: debug 2024-08-02T11:30:49.478+0000 7f3a5241e640 0 RGW-SYNC:data:sync:shard[28]:entry[foobar/new:5160b406-4428-4fdc-9c5d-5ec9fe9404c0.12564119.3:7[0]]:bucket_sync_sources[source=foobar:new[5160b406-4428-4fdc-9c5d-5ec9fe9404c0.12564119.3]):7:source_zone=5160b406-4428-4fdc-9c5d-5ec9fe9404c0]:bucket[foobar/new:5160b406-4428-4fdc-9c5d-5ec9fe9404c0.12564119.3<-foobar/new:5160b406-4428-4fdc-9c5d-5ec9fe9404c0.12564119.3:7]:inc_sync[foobar/new:5160b406-4428-4fdc-9c5d-5ec9fe9404c0.12564119.3:7]:entry[logstash_1%3a8.12.2-1_amd64.deb]: ERROR: failed to sync object: foobar/new:5160b406-4428-4fdc-9c5d-5ec9fe9404c0.12564119.3:7/logstash_1%3a8.12.2-1_amd64.deb And from the sender: Aug 02 13:30:49 test-ceph-single bash[885118]: debug 2024-08-02T11:30:49.476+0000 7f0acfdb2640 1 ====== req done req=0x7f0ab50e4710 op status=-104 http_status=200 latency=0.419986606s ====== Aug 02 13:30:49 test-ceph-single bash[885118]: debug 2024-08-02T11:30:49.476+0000 7f0ba9f66640 2 req 5943847843579143466 0.000000000s initializing for trans_id = tx00000527cca1f3381a52a-0066acc369-c052e6-eu2 Aug 02 13:30:49 test-ceph-single bash[885118]: debug 2024-08-02T11:30:49.476+0000 7f0acfdb2640 1 beast: 0x7f0ab50e4710: 10.139.0.151 - synchronization-user [02/Aug/2024:11:30:49.056 +0000] "GET /foobar%3Anew/logstash_1%253a8.12.2-1_amd64.deb?rgwx-zonegroup=9c1ee979-4362-45a1-ae70-2a83a30ea9fc&rgwx-prepend-metadata=true&rgwx-sync-manifest&rgwx-sync-cloudtiered&rgwx-skip-decrypt&rgwx-if-not-replicated-to=a0fab4b8-ec26-4a11-85dd-abab2e3205fa%3Afoobar%2Fnew%3A5160b406-4428-4fdc-9c5d-5ec9fe9404c0.12564119.3 HTTP/1.1" 200 138413732 - - - latency=0.419986606s Aug 02 13:30:49 test-ceph-single bash[885118]: debug 2024-08-02T11:30:49.476+0000 7f0ba9f66640 2 req 5943847843579143466 0.000000000s getting op 0 Aug 02 13:30:49 test-ceph-single bash[885118]: debug 2024-08-02T11:30:49.476+0000 7f0ba9f66640 2 req 5943847843579143466 0.000000000s :list_metadata_log verifying requester They all keep running into the same error: "failed to sync object(35) Resource deadlock avoided" Any ideas? Thanks! -- Mit freundlichem Gruß // Best regards, Tino Lehnig Cloud Architect Contabo GmbH Aschauer Straße 32a 81549 München https://contabo.com E-Mail: tino.lehnig(a)contabo.de Amtsgericht München HRB 180722 Vertretungsberechtigte Geschäftsführer: Dr. Christian Böing & Thomas Schimmel

1 1

Stable and fastest ceph version for RBD cluster.
by Özkan Göksu 12 Aug '24

12 Aug '24

Hello folks! I built a cluster in 2020 and it has been working great with Nautilus 14.2.16 for the past 4 years. I have 1000++ RBD drives for VM's running on Samsung MZ7LH3T8HMLT drives. Now I want to upgrade the ceph version with a fresh installation and I want to take your opinion on which version will be the best choice for me. I want to upgrade it once and I won't touch it again minimum of 2 years. Does anyone have RBD performance comparison from nautilus, octopus, pacific and quincy? I just want to learn the important changes and benefits of this upgrade. Best regards.

2 1

Important Community Updates [Ceph Developer Summit, Cephalocon]
by Noah Lehman 12 Aug '24

12 Aug '24

Hi Ceph community, I have two important updates to share with you about the Ceph Developer Summit and Cephalocon 20204. *Ceph Developer Summit* The Ceph Developer Summit has been extended until August 20th. Find everything you need to know about the event, including the program and other important details, here <https://ceph.io/en/community/events/2024/ceph-developer-summit-tentacle/>. *Cephalocon 2024* The CFP deadline to submit a speaking proposal has been extended until August 18th. If you're interested in participating as a speaker at the event, you can send in a proposal here <https://ceph.io/en/community/events/2024/cephalocon-2024/>. Thanks everyone! Noah

1 0

Search for a professional service to audit a CephFS infrastructure
by Fabien Sirjean 12 Aug '24

12 Aug '24

Hello, In a professional context, I'm looking for someone with strong CephFS expertise to help us audit our infrastructure. We prefer an on-site audit, but are open to working remotely, and can provide any documentation or information required. Please note that we are not currently in a blocking situation (the storage is functional and meets requirements), but we would like to anticipate an increase in workload and identify bottlenecks and areas for improvement. Our infrastructure, located in Grenoble (France), consists of 12 servers in three server rooms, each server containing 60 HDD of 18.2 TiB, for a total of ~12 PiB raw and 4 PiB net (replicated size 3). We currently store 1.72 PiB (net) of data, in a single CephFS filesystem with ~300 clients (linux kernel). A specific pool is dedicated to MDS, backed with NVMe OSDs. Network is 2 x 25Gbps per server and 2 x 100Gbps between the three server rooms, with jumbo frames for the replication network. This CephFS storage houses the scientific data produced by some fifty fundamental research instruments, as well as the analysis of these data. Our questions mainly concern the configuration of the MDS (currently 6 active MDS + 6 standby-replay), such as : - Best number of MDS for our workload - Pinning MDS to specific parts of the filesystem ? - MDS cache size - CephFS clients performances - ... If you'd like to keep in touch (especially commercially), don't hesitate to contact me at my work address: sirjean(a)ill.fr Cheers, -- Fabien Sirjean Head of IT Infrastructures (DPT/SI/INFRA) Institut Laue Langevin (ILL) +33 (0)4 76 20 76 46 / +33 (0)6 62 47 52 80

1 0

Re: Please guide us inidentifyingthecauseofthedata miss in EC pool
by Best Regards 12 Aug '24

12 Aug '24

Hi, Frédéric Thanks for your advice and suggestions. The failure to identify the root cause of the data loss has a certain impact on subsequent improvement measures.min_size does indeed need to be changed to K+1. We will also reevaluate the disaster recovery situation to better handle extreme scenarios. Thank you again. Best regards. Best Regards wu_chulin(a)qq.com Best Regards Original Email From:"Frédéric Nass"< frederic.nass(a)univ-lorraine.fr >; Sent Time:2024/8/9 16:33 To:"Best Regards"< wu_chulin(a)qq.com >; Cc recipient:"ceph-users"< ceph-users(a)ceph.io >; Subject:Re: Re:Re: Re:[ceph-users] Re: Please guide us inidentifyingthecauseofthedata miss in EC pool Hi Chulin, When it comes to data consistency, it's generally admitted that Ceph is an undefeated master. Considering the very few (~100) rados objects that were completely lost (data and metadata) and the fact that you're using colocated HDD OSDs with volatile disk buffers caching rocksdb metadata and Bluestore data and metadata, I doubt that volatile disk buffers weren't involved in the data loss, whatever the logs say or don't say about which of the 6 over 9 OSDs were in the acting set at the moment of the power outage. Unless you're ok with facing data loss again, I'd advise you fix the initial design flaws if you can. Like stop using non-persistent cache / buffers along the IO path, raise mon_size to k+1 and reconsider data placement in regards to risks of network partitioning, power outage, fire. Also, considering the ceph status, make sure you don't run out of disk space. Best regards, Frédéric. De : Best Regards <wu_chulin(a)qq.com> Envoyé : jeudi 8 août 2024 11:32 À : Frédéric Nass Cc: ceph-users Objet : Re:Re: Re:[ceph-users] Re: Please guide us inidentifying thecauseofthedata miss in EC pool Hi，Frédéric Nass Sorry, I may not have expressed it clearly before. The epoch and OSD up/down timeline was extracted and merged from the 9 OSD logs. I analyzed the PG (9.11b6) peering process. OSD 494, 1169, 1057 fully recorded the down/up of other OSDs. I also checked the logs of the other 6 OSDs. The role conversion during peering was expected and no abnormalities were found. I also checked the status of the monitor. One of the 5 monitors lost power and was powered on after about 40 minutes. The log showed that its rank value was relatively large and it did not become the leader. Let's talk about the fault domain. The fault domain we set is the host level, but in fact all hosts are distributed in 2 buildings, but the original designer did not consider the fault level of the building. In this case, the OSD may have a brain split, but from the log, it does not happen. Best regards. Best Regards wu_chulin(a)qq.com Best Regards Original Email From:"Frédéric Nass"< frederic.nass(a)univ-lorraine.fr >; Sent Time:2024/8/8 15:40 To:"Best Regards"< wu_chulin(a)qq.com >; Cc recipient:"ceph-users"< ceph-users(a)ceph.io >; Subject:Re: Re:[ceph-users] Re: Please guide us inidentifying thecauseofthedata miss in EC pool ceph osd pause is a lot of constraints from an operational perspective. :-) host uptime and service running time is a thing. But it doesn't mean that these 3 OSDs were in the acting set when the power outage occured. Since OSDs 494, 1169 and 1057 did not crash, I assume they're in the same failure domain. Is that right? Being isolated along with their local MON(s) from other MONs and other 6 OSDs, there's a fair chance that any of the 6 other OSDs in other failure domains took the lead, sent 5 chunks around and acknowledged the write to RGW client. Then all of them crashed. Your thoughts? Frédéric. De : Best Regards <wu_chulin(a)qq.com> Envoyé : jeudi 8 août 2024 09:16 À : Frédéric Nass Cc: ceph-users Objet : Re:[ceph-users] Re: Please guide us inidentifying thecause ofthedata miss in EC pool Hi, Frédéric Nass Yes. I checked the host running time where the OSD is located and the OSD service running time.These were stopped when I executed `ceph-object-tool`. They were running before. Because we need to maintain the hardware frequently (the hardware is quite old), min_size is set to the lowest value. When a failure occurs, we set the read and write pause flag. During the failure, there is no PUT action on the S3 keys to which these objects belong. Best Regards Best Regards wu_chulin(a)qq.com Best Regards Original Email From:"Frédéric Nass"< frederic.nass(a)univ-lorraine.fr >; Sent Time:2024/8/8 14:40 To:"Best Regards"< wu_chulin(a)qq.com >; Cc recipient:"ceph-users"< ceph-users(a)ceph.io >; Subject:[ceph-users] Re: Please guide us inidentifying thecause ofthedata miss in EC pool Hi Chulin, Are you 100% sure that 494, 1169 and 1057 (that did not restart) were in the acting set at the exact moment the power outage occured?  I'm asking because min_size 6 would have allowed the data to be written to eventually 6 crashing OSDs. Bests, Frédéric. ________________________________ De : Best Regards Envoyé : jeudi 8 août 2024 08:10 À : Frédéric Nass Cc: ceph-users Objet : Re:Re: Re:Re: Re:Re: Re:Re: [ceph-users] Please guide us inidentifying thecause ofthedata miss in EC pool Hi， Frédéric Nass Thank you for your continued attention and guidance. Let's analyze and verify this issue from different perspectives. The reason why we did not stop the investigation is that we tried to find other ways to avoid the losses caused by this sudden failure. Turning off the disk cache is the last option, of course, this operation will only be carried out after finding definite evidence. I also have a question that among the 9 OSDs, some have not been restarted. In theory, these OSDs should retain the object info(metadata,pg_log,etc.), even if the object cannot be recovered. I sorted out the OSD booting log where the object should be located and the PG peering process: OSD 494/1169/1057 has been in the running state, and osd.494 was the primary of the acting_set during the failure. However, no record of the object was found using `ceph-object-tool --op list or --op log` in, so the loss of data due to disk cache loss does not seem to be a complete explanation (perhaps there is some processing logic that we have not paid attention to). Best Regards, Woo wu_chulin(a)qq.com Best Regards Original Email From:"Frédéric Nass"< frederic.nass(a)univ-lorraine.fr >; Sent Time:2024/8/8 4:01 To:"wu_chulin"< wu_chulin(a)qq.com >; Subject:Re: Re:Re: Re:Re: Re:Re: [ceph-users] Please guide us inidentifying thecause ofthedata miss in EC pool Hey Chulin, Looks clearer now. Non-persistent cache for KV metadata and Bluestore metadata certainly explains how data was lost without the cluster even noticing. What's unexpected is data staying for so long in the disks buffers and not being written to persistent sectors at all. Anyways, thank you for sharing your use case and investigation. It was nice chatting with you. If you can, share this in the ceph-user list. It will for sure benefit everyone in the community. Best regards, Frédéric. PS : Note that using min_size >= k + 1 on EC pools is recommended (so as min_size >= 2 on rep X3 pools) because you don't want to write data without any parity chunks. De : wu_chulin(a)qq.com Envoyé : mercredi 7 août 2024 11:30 À : Frédéric Nass Objet : Re:Re: Re:Re: Re:Re: [ceph-users] Please guide us in identifying thecause ofthedata miss in EC pool Hi, Yes, after the file -> object -> PG -> OSD correspondence is found, the object record can be found on the specified OSD using the command `ceph-objectore-tool --op list ` The pool min_size is 6 The business department reported more than 30, but we proactively screened out more than 100. The upload time of the lost files was mainly distributed about 3 hours before the failure, and these files were successfully downloaded after being uploaded (RGW log). One OSD corresponds to one disk, and no separate space is allocated for WAL/DB. The HDD cache is the default (SATA is enabled by default), and the hard disk cache has not been forcibly turned off due to performance issues. The loss of OSD data due to the loss of hard disk cache was our initial inference, and the initial explanation provided to the business department was the same. When the cluster was restored, ceph reported 12 unfound objects, which is acceptable. After all, most devices were powered off abnormally, and it is difficult to ensure the integrity of all data. Up to now, our team have not located how the data was lost. In the past, when the hard disk hardware was damaged, either the OSD could not start because of damaged key data, or some objects were read incorrectly after the OSD started, which could be repaired. Now deep-scrub cannot find the problem, which may be related to the loss (or deletion) of object metadata. After all, deep-scrub needs the object list of the current PG. If those 9 OSDs do not have the object metadata information, deep-scrub does not know the existence of this object. wu_chulin(a)qq.com wu_chulin(a)qq.com Original Email From:"Frédéric Nass"< frederic.nass(a)univ-lorraine.fr >; Sent Time:2024/8/6 20:40 To:"wu_chulin"< wu_chulin(a)qq.com >; Subject:Re: Re:Re: Re:Re: [ceph-users] Please guide us in identifying thecause ofthedata miss in EC pool That's interesting. Have you tried to correlate any existing retrievable object to PG id and OSD mapping in order to verify the presence of each of these object's shards using ceph-objectore-tool on each one of its acting OSDs, for a previously and successfully written S3 object. This would help verify that the command you've run trying to find the missing shard was good. Also what min_size is this pool using?    How many S3 objects like these were reported missing by the business unit? Have you or they made an inventory of unretrievable / missing objects? Are WAL / RocksDB collocated on HDDs only OSDs or are these OSDs using SSDs for WAL / RocksDB? Did you disable HDD buffers (also known as disk cache) as... HDD buffers are non-persistent. I know nothing about the intensity of your workloads but if your looking for a few tens or a few hundreds of unwritten s3 objects, there might be some situation with non-persistent cache (like volatile disk buffers) were ceph would consider the data to be written when it was actually not at the moment of the power outage, due to the use of non-persistent disk buffers. Especially if you kept writing data with less shards (min_size) than k+1 (no parity at all). That sound like a possibility. Also what I'm thinking right is... If you can identify which shard over 9 is wrong, then you may use ceph-objectore-tool or ceph-kvstore-tool to destroy this particular shard, then deep-scrub the PG so to detect the inexistent shard and have it rebuilt. Never tried this myself though. Best regards, Frédéric. De : wu_chulin(a)qq.com Envoyé : mardi 6 août 2024 12:15 À : Frédéric Nass Objet : Re:Re: Re:Re: [ceph-users] Please guide us in identifying the cause ofthedata miss in EC pool Hi,  Thank you for your attention to this matter. 1. Manually executing deep-scrub PG did not report any errors. I checked the OSD logs and did not see any errors detected or fixed by the OSD. Ceph health was also normal, and the OS where the OSD was located did not report any IO type errors. 2. At first, I also suspected whether the objects were distributed on these OSDs before the failure. I used ceph osd getmap 3. Our Ceph version is 13.2.10, which does not have pg autoscaler; lifecycle policies are not set. 4. We have 90+ hosts, most of which are Dell R720xd, most of the hard disks are 3.5 inches/5400 rpm/10TB Western Digital/, and most of the controllers are PERC H330 Mini; this is the current cluster status: cluster: id: f990db28-9604-4d49-9733-b17155887e3b health: HEALTH_OK services:   mon: 5 daemons, quorum cz-ceph-01,cz-ceph-02,cz-ceph-03,cz-ceph-07,cz-ceph-13   mgr: cz-ceph-01(active), standbys: cz-ceph-03, cz-ceph-02   osd: 1172 osds: 1172 up, 1172 in    rgw: 9 daemons active  data:    pools: 16 pools, 25752 pgs    objects: 2.23 G objects, 6.1 PiB    usage: 9.4 PiB used, 2.5 PiB / 12 PiB avail    pgs:      25686 active+clean                 64 active+clean+scrubbing+deep+repair                 2 active+clean+scrubbing+deep Best regards. Original Email From:"Frédéric Nass"< frederic.nass(a)univ-lorraine.fr >; Sent Time:2024/8/6 15:34 To:"wu_chulin"< wu_chulin(a)qq.com >; Subject:Re: Re:Re: [ceph-users] Please guide us in identifying the cause ofthedata miss in EC pool Hi, Did the deep-scrub report any errors? (any inconsistencies should show errors after deep-scrubbing the PG.)   We're these errors fixed by the PG repair? Is is possible that you looked for the wrong PG or OSD when trying to list these objects with ceph-objectstore-tool? Were the PG autoscaler running at that time? Are you using S3 lifecycle policies that could have move this object to another placement pool and so another PG? Can you give details about this cluster? Hardware, disks, controller, etc. Cheers, Frédéric. De : wu_chulin(a)qq.com Envoyé : lundi 5 août 2024 10:09 À : Frédéric Nass Objet : Re:Re: [ceph-users] Please guide us in identifying the cause of thedata miss in EC pool Hi， Thank you for your reply. I apologize for the omission in the previous email. Please disregard the previous email and refer to this one instead.  After the failure, we executed the repair and deep-scrub command on some of the PGs that lost data, and the status was active+clean after completion, but the object still could not be retrieved. Our erasure code parameters are k=6, m=3. Theoretically, the data of three OSDs lost due to power failure should be recoverable. However, we stopped nine OSDs and exported the object list, but could not find the lost object information. What puzzled us was that some OSDs were not powered off and were still running, but their object lists did not have the information too. Best regards. wu_chulin(a)qq.com wu_chulin(a)qq.com Original Email From:"Frédéric Nass"< frederic.nass(a)univ-lorraine.fr >; Sent Time:2024/8/3 15:11 To:"wu_chulin"< wu_chulin(a)qq.com >;"ceph-users"< ceph-users(a)ceph.io >; Subject:Re: [ceph-users] Please guide us in identifying the cause of thedata miss in EC pool Hi, First thing that comes to mind when it comes to data unavailability or inconsistencies after a power outage is that some dirty data may have been lost along the IO path before reaching persistent storage. This can happen with non enterprise grade SSDs using non-persistent cache or with HDDs disk buffer if left enabled for example. With that said, have you tried to deep-scrub the PG from which you can't retrieve data? What's the status of this PG now? Did it recover? Regards, Frédéric. De : wu_chulin(a)qq.com Envoyé : mercredi 31 juillet 2024 05:49 À : ceph-users Objet : [ceph-users] Please guide us in identifying the cause of the data miss in EC pool Dear Ceph team:&nbsp; &nbsp; &nbsp;On July 13th at 4:55 AM, our Ceph cluster experienced a significant power outage in the data center, causing a large number of OSDs to power off and restart (total: 1172, down: 821). Approximately two hours later, all OSDs successfully started, and the cluster resumed its services. However, around 6 PM, the business department reported that some files, which had been successfully written (via the RGW service), were failing to download, and the number of such files was quite significant. Consequently, we began a series of investigations: 1. The incident occurred at 04:55. At 05:01, we executed noout, nobackfill, and norecover. At 06:22, we executed `ceph osd pause`. By 07:23, all OSDs were UP&amp;IN, and subsequently, we executed `ceph osd unpause`. 2. We randomly selected a problematic file and attempted to download it via the S3 API. The RGW returned "No such key". 3. The RGW logs showed op status=-2, http status=200. We also checked the upload logs, which indicated 2024-07-13 04:19:20.052, op status=0, http_status=200. 4. We set debug_rgw=20 and attempted to download the file again. It was found that a 4M chunk(this file is 64M) failed to get. 5. Using rados get for this chunk returned: "No such file or directory". 6. Setting debug_osd=20, we observed get_object_context: obc NOT found in cache. 7. Setting debug_bluestore=20, we saw get_onode oid xxx, key xxx != '0xfffffffffffffffeffffffffffffffff'o'. 8. We stopped the primary OSD and tried to get the file again, but the result was the same. The object’s corresponding PG state was active+recovery_wait+degraded. 9. Using ceph-objectstore-tool --op list &amp;&amp; --op log, we could not find the object information. The ceph-kvstore-tool rocksdb command also did not reveal anything new. 10. If an OSD had lost data, the PG state should have been unfound or inconsistency. 11. We started reanalyzing the startup logs of the OSDs related to the PG. The pool was set to erasure-code 6-3 with 9 OSDs. Six of these OSDs had restarted, and after peering, the PG state became ACTIVE. 12. We divided the lost files, and the upload time was before the failure occurred. The earliest upload time was around 1 am, and the successful upload records could be found in the RGW log 13. We have submitted an issue on the Ceph issue tracker:&nbsp;https://tracker.ceph.com/issues/66942, it includes the original logs needed for troubleshooting. However, four days have passed without any response. In desperation, we are sending this email, hoping that someone from the Ceph team can guide us as soon as possible. We are currently in a difficult situation and hope you can provide guidance. Thank you. Best regards. wu_chulin(a)qq.com wu_chulin(a)qq.com _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Any way to put the rate limit on rbd flatten operation?
by Henry lol 12 Aug '24

12 Aug '24

Hello, AFAIK, massive rx/tx occurs on the client side for the flatten operation. so, I want to control the network rate limit or predict the network bandwidth it will consume. Is there any way to do that?

2 1

[Cephalocon 2024] CFP Closes TOMORROW
by Patrick Donnelly 10 Aug '24

10 Aug '24

Cephalocon is the premier yearly event that brings together the global community of operators, developers, and researchers for Ceph, the open source distributed storage system. The CFP closes tomorrow, 2024-08-11. Please submit your talk proposals: https://ceph.io/en/community/events/2024/cephalocon-2024/ Regards, -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

Re: Please guide us inidentifying thecauseofthedata miss in EC pool
by Best Regards 09 Aug '24

09 Aug '24

Hi，Frédéric Nass Sorry, I may not have expressed it clearly before. The epoch and OSD up/down timeline was extracted and merged from the 9 OSD logs. I analyzed the PG (9.11b6) peering process. OSD 494, 1169, 1057 fully recorded the down/up of other OSDs. I also checked the logs of the other 6 OSDs. The role conversion during peering was expected and no abnormalities were found. I also checked the status of the monitor. One of the 5 monitors lost power and was powered on after about 40 minutes. The log showed that its rank value was relatively large and it did not become the leader. Let's talk about the fault domain. The fault domain we set is the host level, but in fact all hosts are distributed in 2 buildings, but the original designer did not consider the fault level of the building. In this case, the OSD may have a brain split, but from the log, it does not happen. Best regards. Best Regards wu_chulin(a)qq.com Best Regards Original Email From:"Frédéric Nass"< frederic.nass(a)univ-lorraine.fr >; Sent Time:2024/8/8 15:40 To:"Best Regards"< wu_chulin(a)qq.com >; Cc recipient:"ceph-users"< ceph-users(a)ceph.io >; Subject:Re: Re:[ceph-users] Re: Please guide us inidentifying thecauseofthedata miss in EC pool ceph osd pause is a lot of constraints from an operational perspective. :-) host uptime and service running time is a thing. But it doesn't mean that these 3 OSDs were in the acting set when the power outage occured. Since OSDs 494, 1169 and 1057 did not crash, I assume they're in the same failure domain. Is that right? Being isolated along with their local MON(s) from other MONs and other 6 OSDs, there's a fair chance that any of the 6 other OSDs in other failure domains took the lead, sent 5 chunks around and acknowledged the write to RGW client. Then all of them crashed. Your thoughts? Frédéric. De : Best Regards <wu_chulin(a)qq.com> Envoyé : jeudi 8 août 2024 09:16 À : Frédéric Nass Cc: ceph-users Objet : Re:[ceph-users] Re: Please guide us inidentifying thecause ofthedata miss in EC pool Hi, Frédéric Nass Yes. I checked the host running time where the OSD is located and the OSD service running time.These were stopped when I executed `ceph-object-tool`. They were running before. Because we need to maintain the hardware frequently (the hardware is quite old), min_size is set to the lowest value. When a failure occurs, we set the read and write pause flag. During the failure, there is no PUT action on the S3 keys to which these objects belong. Best Regards Best Regards wu_chulin(a)qq.com Best Regards Original Email From:"Frédéric Nass"< frederic.nass(a)univ-lorraine.fr >; Sent Time:2024/8/8 14:40 To:"Best Regards"< wu_chulin(a)qq.com >; Cc recipient:"ceph-users"< ceph-users(a)ceph.io >; Subject:[ceph-users] Re: Please guide us inidentifying thecause ofthedata miss in EC pool Hi Chulin, Are you 100% sure that 494, 1169 and 1057 (that did not restart) were in the acting set at the exact moment the power outage occured?  I'm asking because min_size 6 would have allowed the data to be written to eventually 6 crashing OSDs. Bests, Frédéric. ________________________________ De : Best Regards Envoyé : jeudi 8 août 2024 08:10 À : Frédéric Nass Cc: ceph-users Objet : Re:Re: Re:Re: Re:Re: Re:Re: [ceph-users] Please guide us inidentifying thecause ofthedata miss in EC pool Hi， Frédéric Nass Thank you for your continued attention and guidance. Let's analyze and verify this issue from different perspectives. The reason why we did not stop the investigation is that we tried to find other ways to avoid the losses caused by this sudden failure. Turning off the disk cache is the last option, of course, this operation will only be carried out after finding definite evidence. I also have a question that among the 9 OSDs, some have not been restarted. In theory, these OSDs should retain the object info(metadata,pg_log,etc.), even if the object cannot be recovered. I sorted out the OSD booting log where the object should be located and the PG peering process: OSD 494/1169/1057 has been in the running state, and osd.494 was the primary of the acting_set during the failure. However, no record of the object was found using `ceph-object-tool --op list or --op log` in, so the loss of data due to disk cache loss does not seem to be a complete explanation (perhaps there is some processing logic that we have not paid attention to). Best Regards, Woo wu_chulin(a)qq.com Best Regards Original Email From:"Frédéric Nass"< frederic.nass(a)univ-lorraine.fr >; Sent Time:2024/8/8 4:01 To:"wu_chulin"< wu_chulin(a)qq.com >; Subject:Re: Re:Re: Re:Re: Re:Re: [ceph-users] Please guide us inidentifying thecause ofthedata miss in EC pool Hey Chulin, Looks clearer now. Non-persistent cache for KV metadata and Bluestore metadata certainly explains how data was lost without the cluster even noticing. What's unexpected is data staying for so long in the disks buffers and not being written to persistent sectors at all. Anyways, thank you for sharing your use case and investigation. It was nice chatting with you. If you can, share this in the ceph-user list. It will for sure benefit everyone in the community. Best regards, Frédéric. PS : Note that using min_size >= k + 1 on EC pools is recommended (so as min_size >= 2 on rep X3 pools) because you don't want to write data without any parity chunks. De : wu_chulin(a)qq.com Envoyé : mercredi 7 août 2024 11:30 À : Frédéric Nass Objet : Re:Re: Re:Re: Re:Re: [ceph-users] Please guide us in identifying thecause ofthedata miss in EC pool Hi, Yes, after the file -> object -> PG -> OSD correspondence is found, the object record can be found on the specified OSD using the command `ceph-objectore-tool --op list ` The pool min_size is 6 The business department reported more than 30, but we proactively screened out more than 100. The upload time of the lost files was mainly distributed about 3 hours before the failure, and these files were successfully downloaded after being uploaded (RGW log). One OSD corresponds to one disk, and no separate space is allocated for WAL/DB. The HDD cache is the default (SATA is enabled by default), and the hard disk cache has not been forcibly turned off due to performance issues. The loss of OSD data due to the loss of hard disk cache was our initial inference, and the initial explanation provided to the business department was the same. When the cluster was restored, ceph reported 12 unfound objects, which is acceptable. After all, most devices were powered off abnormally, and it is difficult to ensure the integrity of all data. Up to now, our team have not located how the data was lost. In the past, when the hard disk hardware was damaged, either the OSD could not start because of damaged key data, or some objects were read incorrectly after the OSD started, which could be repaired. Now deep-scrub cannot find the problem, which may be related to the loss (or deletion) of object metadata. After all, deep-scrub needs the object list of the current PG. If those 9 OSDs do not have the object metadata information, deep-scrub does not know the existence of this object. wu_chulin(a)qq.com wu_chulin(a)qq.com Original Email From:"Frédéric Nass"< frederic.nass(a)univ-lorraine.fr >; Sent Time:2024/8/6 20:40 To:"wu_chulin"< wu_chulin(a)qq.com >; Subject:Re: Re:Re: Re:Re: [ceph-users] Please guide us in identifying thecause ofthedata miss in EC pool That's interesting. Have you tried to correlate any existing retrievable object to PG id and OSD mapping in order to verify the presence of each of these object's shards using ceph-objectore-tool on each one of its acting OSDs, for a previously and successfully written S3 object. This would help verify that the command you've run trying to find the missing shard was good. Also what min_size is this pool using?    How many S3 objects like these were reported missing by the business unit? Have you or they made an inventory of unretrievable / missing objects? Are WAL / RocksDB collocated on HDDs only OSDs or are these OSDs using SSDs for WAL / RocksDB? Did you disable HDD buffers (also known as disk cache) as... HDD buffers are non-persistent. I know nothing about the intensity of your workloads but if your looking for a few tens or a few hundreds of unwritten s3 objects, there might be some situation with non-persistent cache (like volatile disk buffers) were ceph would consider the data to be written when it was actually not at the moment of the power outage, due to the use of non-persistent disk buffers. Especially if you kept writing data with less shards (min_size) than k+1 (no parity at all). That sound like a possibility. Also what I'm thinking right is... If you can identify which shard over 9 is wrong, then you may use ceph-objectore-tool or ceph-kvstore-tool to destroy this particular shard, then deep-scrub the PG so to detect the inexistent shard and have it rebuilt. Never tried this myself though. Best regards, Frédéric. De : wu_chulin(a)qq.com Envoyé : mardi 6 août 2024 12:15 À : Frédéric Nass Objet : Re:Re: Re:Re: [ceph-users] Please guide us in identifying the cause ofthedata miss in EC pool Hi,  Thank you for your attention to this matter. 1. Manually executing deep-scrub PG did not report any errors. I checked the OSD logs and did not see any errors detected or fixed by the OSD. Ceph health was also normal, and the OS where the OSD was located did not report any IO type errors. 2. At first, I also suspected whether the objects were distributed on these OSDs before the failure. I used ceph osd getmap 3. Our Ceph version is 13.2.10, which does not have pg autoscaler; lifecycle policies are not set. 4. We have 90+ hosts, most of which are Dell R720xd, most of the hard disks are 3.5 inches/5400 rpm/10TB Western Digital/, and most of the controllers are PERC H330 Mini; this is the current cluster status: cluster: id: f990db28-9604-4d49-9733-b17155887e3b health: HEALTH_OK services:   mon: 5 daemons, quorum cz-ceph-01,cz-ceph-02,cz-ceph-03,cz-ceph-07,cz-ceph-13   mgr: cz-ceph-01(active), standbys: cz-ceph-03, cz-ceph-02   osd: 1172 osds: 1172 up, 1172 in    rgw: 9 daemons active  data:    pools: 16 pools, 25752 pgs    objects: 2.23 G objects, 6.1 PiB    usage: 9.4 PiB used, 2.5 PiB / 12 PiB avail    pgs:      25686 active+clean                 64 active+clean+scrubbing+deep+repair                 2 active+clean+scrubbing+deep Best regards. Original Email From:"Frédéric Nass"< frederic.nass(a)univ-lorraine.fr >; Sent Time:2024/8/6 15:34 To:"wu_chulin"< wu_chulin(a)qq.com >; Subject:Re: Re:Re: [ceph-users] Please guide us in identifying the cause ofthedata miss in EC pool Hi, Did the deep-scrub report any errors? (any inconsistencies should show errors after deep-scrubbing the PG.)   We're these errors fixed by the PG repair? Is is possible that you looked for the wrong PG or OSD when trying to list these objects with ceph-objectstore-tool? Were the PG autoscaler running at that time? Are you using S3 lifecycle policies that could have move this object to another placement pool and so another PG? Can you give details about this cluster? Hardware, disks, controller, etc. Cheers, Frédéric. De : wu_chulin(a)qq.com Envoyé : lundi 5 août 2024 10:09 À : Frédéric Nass Objet : Re:Re: [ceph-users] Please guide us in identifying the cause of thedata miss in EC pool Hi， Thank you for your reply. I apologize for the omission in the previous email. Please disregard the previous email and refer to this one instead.  After the failure, we executed the repair and deep-scrub command on some of the PGs that lost data, and the status was active+clean after completion, but the object still could not be retrieved. Our erasure code parameters are k=6, m=3. Theoretically, the data of three OSDs lost due to power failure should be recoverable. However, we stopped nine OSDs and exported the object list, but could not find the lost object information. What puzzled us was that some OSDs were not powered off and were still running, but their object lists did not have the information too. Best regards. wu_chulin(a)qq.com wu_chulin(a)qq.com Original Email From:"Frédéric Nass"< frederic.nass(a)univ-lorraine.fr >; Sent Time:2024/8/3 15:11 To:"wu_chulin"< wu_chulin(a)qq.com >;"ceph-users"< ceph-users(a)ceph.io >; Subject:Re: [ceph-users] Please guide us in identifying the cause of thedata miss in EC pool Hi, First thing that comes to mind when it comes to data unavailability or inconsistencies after a power outage is that some dirty data may have been lost along the IO path before reaching persistent storage. This can happen with non enterprise grade SSDs using non-persistent cache or with HDDs disk buffer if left enabled for example. With that said, have you tried to deep-scrub the PG from which you can't retrieve data? What's the status of this PG now? Did it recover? Regards, Frédéric. De : wu_chulin(a)qq.com Envoyé : mercredi 31 juillet 2024 05:49 À : ceph-users Objet : [ceph-users] Please guide us in identifying the cause of the data miss in EC pool Dear Ceph team:&nbsp; &nbsp; &nbsp;On July 13th at 4:55 AM, our Ceph cluster experienced a significant power outage in the data center, causing a large number of OSDs to power off and restart (total: 1172, down: 821). Approximately two hours later, all OSDs successfully started, and the cluster resumed its services. However, around 6 PM, the business department reported that some files, which had been successfully written (via the RGW service), were failing to download, and the number of such files was quite significant. Consequently, we began a series of investigations: 1. The incident occurred at 04:55. At 05:01, we executed noout, nobackfill, and norecover. At 06:22, we executed `ceph osd pause`. By 07:23, all OSDs were UP&amp;IN, and subsequently, we executed `ceph osd unpause`. 2. We randomly selected a problematic file and attempted to download it via the S3 API. The RGW returned "No such key". 3. The RGW logs showed op status=-2, http status=200. We also checked the upload logs, which indicated 2024-07-13 04:19:20.052, op status=0, http_status=200. 4. We set debug_rgw=20 and attempted to download the file again. It was found that a 4M chunk(this file is 64M) failed to get. 5. Using rados get for this chunk returned: "No such file or directory". 6. Setting debug_osd=20, we observed get_object_context: obc NOT found in cache. 7. Setting debug_bluestore=20, we saw get_onode oid xxx, key xxx != '0xfffffffffffffffeffffffffffffffff'o'. 8. We stopped the primary OSD and tried to get the file again, but the result was the same. The object’s corresponding PG state was active+recovery_wait+degraded. 9. Using ceph-objectstore-tool --op list &amp;&amp; --op log, we could not find the object information. The ceph-kvstore-tool rocksdb command also did not reveal anything new. 10. If an OSD had lost data, the PG state should have been unfound or inconsistency. 11. We started reanalyzing the startup logs of the OSDs related to the PG. The pool was set to erasure-code 6-3 with 9 OSDs. Six of these OSDs had restarted, and after peering, the PG state became ACTIVE. 12. We divided the lost files, and the upload time was before the failure occurred. The earliest upload time was around 1 am, and the successful upload records could be found in the RGW log 13. We have submitted an issue on the Ceph issue tracker:&nbsp;https://tracker.ceph.com/issues/66942, it includes the original logs needed for troubleshooting. However, four days have passed without any response. In desperation, we are sending this email, hoping that someone from the Ceph team can guide us as soon as possible. We are currently in a difficult situation and hope you can provide guidance. Thank you. Best regards. wu_chulin(a)qq.com wu_chulin(a)qq.com _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

RGW: HEAD ok but GET fails
by Eugen Block 09 Aug '24

09 Aug '24

Hi, I'm trying to help a customer with a RGW question, maybe someone here can help me out. Their S3 application reports errors every now and then, and it is complaining about missing objects. This is what the RGW logs: [08/Aug/2024:08:23:47.540 +0000] "HEAD /hchsarchiv/20240622221326-20540623-aeaa962adadf5bc92050823dd03039197987f9d16f70c793599e361b6a5910c83941a0ceb3c7bfccb0a8ecbae212c701958d8a316b4fb172a54040b26b3a2508 HTTP/1.1" 200 0 - "aws-sdk-dotnet-45/3.5.9.7 aws-sdk-dotnet-core/3.5.3.7 .NET_Runtime/4.0 .NET_Framework/4.0 OS/Microsoft_Windows_NT_10.0.14393.0 ClientSync" - latency=0.003999992s [08/Aug/2024:08:23:47.552 +0000] "GET /hchsarchiv/20240622221326-20540623-aeaa962adadf5bc92050823dd03039197987f9d16f70c793599e361b6a5910c83941a0ceb3c7bfccb0a8ecbae212c701958d8a316b4fb172a54040b26b3a2508 HTTP/1.1" 404 242 - "aws-sdk-dotnet-45/3.5.9.7 aws-sdk-dotnet-core/3.5.3.7 .NET_Runtime/4.0 .NET_Framework/4.0 OS/Microsoft_Windows_NT_10.0.14393.0 ClientSync" bytes=0-2097151 latency=0.003999992s So apparently, it can successfully query the HEAD, but the GET request shows 404. We can confirm that the queried object indeed doesn't exist in the data pool. But the object metadata must have been written successfully, apparently. Unfortunately, we don't have enough logs to find the corresponding PUT request, they just increased the retention days for logrotate to be able to inspect when it happens the next time. But my question is, should they see some metadata in the listomapkeys/listomapvals output in the index pool? The docs [0] state this about Index Transactions: > Because the head objects are stored in different rados objects than > the bucket indices, we can’t update both atomically with a single > rados operation. In order to satisfy the Consistency Guarantee for > listing operations, we have to coordinate these two object writes > using a three-step bucket index transaction: > > 1. Prepare a transaction on its bucket index object. > 2. Write or delete the head object. > 3. Commit the transaction on the bucket index object (or cancel the > transaction if step 2 fails). > > Object writes and deletes may race with each other, so a given > object may have more than one prepared transaction at a time. RGW > considers an object entry to be ‘pending’ if there are any > outstanding transactions, or ‘completed’ otherwise. Could this be such a race condition which "just happens" from time to time? Or can this somehow be prevented from happening? Because right now the clenaup process is a bit complicated application-wise. I'm not the most experienced RGW user, so I'd be grateful for any pointers here. Thanks! Eugen [0] https://docs.ceph.com/en/reef/dev/radosgw/bucket_index/#index-transaction

2 2

Possible regression? Kernel cephfs >= 6.10 cpu hangup
by caskd 09 Aug '24

09 Aug '24

Hello fellow ceph users, i've noticed that since i've upgraded to kernel version 6.10 i've been occasionally having system lockups with dmesg showing cephfs in the kernel trace. Due to the circumstances of the lockups, i've so far been unable to store these trace logs, but i will attempt to do so in a more isolated and controlled environment. The symptom seems to occur during file syncs (flushing), more specifically it has happened 2 times while saving a file from vim (create new file, flush to file, overwrite old file). Is anyone aware of changes that may have caused such a problem before i start digging deeper? Have a nice day -- Alex D. RedXen System & Infrastructure Administration https://redxen.eu/

1 1

(belated) CLT notes
by Gregory Farnum 08 Aug '24

08 Aug '24

Hi folks, the CLT met on Monday August 5. We discussed a few topics: * The mailing lists are a problem to moderate right now with a huge increase in spam. We have two problems: 1) the moderation system's web front-end apparently isn't operational. That's getting fixed. 2) The moderation is a big load regardless. Casey asked if we want to move to the Linux Foundation-managed list system; Josh will take that to the board since it involves some cost. * Squid Release. 19.1.1 testing is making good progress. * (Likely) Final quincy release 17.2.8 is next up in the queue. There isn't a lot targeted for it right now, so please apply the explicit 17.2.8 milestone or set needs-qa on quincy-labeled PRs if you want Yuri to help move them forward. * The big topic is centos8 stream, in two ways: 1) We dropped building packages for it in the middle of the Reef stable release stream because our build system can't handle the way they closed the release. (Packages are still available at archival URLs, but we aren't set up to use those and probably can't access them the same way we normally do.) This has understandably upset some people. Can we fix it? 2) We had many developers spend a lot of time unexpectedly updating our QA systems when 8.stream disappeared, and it's still an ongoing problem for things like upgrade testing of backports. Is Centos stream a reliable base for our container images, package building, and test infrastructure? Should we move elsewhere? We need more information and a wider discussion, so this was also a topic at yesterday's CDM. We will continue to discuss and coordinate with other interest groups and keep you informed. -Greg

1 0

Cephadm: unable to copy ceph.conf.new
by Magnus Larsen 08 Aug '24

08 Aug '24

Hi Ceph-users! Ceph version: ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) Using cephadm to orchestrate the Ceph cluster I’m running into https://tracker.ceph.com/issues/59189, which is fixed in next version—quincy 17.2.7—via https://github.com/ceph/ceph/pull/50906 But I am unable to upgrade to the fixed version because of that bug When I try to upgrade (using “ceph orch upgrade start –image internal_mirror/ceph:v17.2.7”), we see the same error message: executing _write_files((['dkcphhpcadmin01', 'dkcphhpcmgt028', 'dkcphhpcmgt029', 'dkcphhpcmgt031', 'dkcphhpcosd033', 'dkcphhpcosd034', 'dkcphhpcosd035', 'dkcphhpcosd036', 'dkcphhpcosd037', 'dkcphhpcosd038', 'dkcphhpcosd039', 'dkcphhpcosd040', 'dkcphhpcosd041', 'dkcphhpcosd042', 'dkcphhpcosd043', 'dkcphhpcosd044'],)) failed. Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/ssh.py", line 240, in _write_remote_file conn = await self._remote_connection(host, addr) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 922, in scp await source.run(srcpath) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 458, in run self.handle_error(exc) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in handle_error raise exc from None File "/lib/python3.6/site-packages/asyncssh/scp.py", line 456, in run await self._send_files(path, b'') File "/lib/python3.6/site-packages/asyncssh/scp.py", line 438, in _send_files self.handle_error(exc) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in handle_error raise exc from None File "/lib/python3.6/site-packages/asyncssh/scp.py", line 434, in _send_files await self._send_file(srcpath, dstpath, attrs) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 365, in _send_file await self._make_cd_request(b'C', attrs, size, srcpath) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 343, in _make_cd_request self._fs.basename(path)) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 224, in make_request raise exc asyncssh.sftp.SFTPFailure: scp: /tmp/etc/ceph/ceph.conf.new: Permission denied During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/utils.py", line 79, in do_work return f(*arg) File "/usr/share/ceph/mgr/cephadm/serve.py", line 1088, in _write_files self._write_client_files(client_files, host) File "/usr/share/ceph/mgr/cephadm/serve.py", line 1107, in _write_client_files self.mgr.ssh.write_remote_file(host, path, content, mode, uid, gid) File "/usr/share/ceph/mgr/cephadm/ssh.py", line 261, in write_remote_file self.mgr.wait_async(self._write_remote_file( File "/usr/share/ceph/mgr/cephadm/module.py", line 615, in wait_async return self.event_loop.get_result(coro) File "/usr/share/ceph/mgr/cephadm/ssh.py", line 56, in get_result return asyncio.run_coroutine_threadsafe(coro, self._loop).result() File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result return self.__get_result() File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result raise self._exception File "/usr/share/ceph/mgr/cephadm/ssh.py", line 249, in _write_remote_file logger.exception(msg) orchestrator._interface.OrchestratorError: Unable to write dkcphhpcmgt028:/etc/ceph/ceph.conf: scp: /tmp/etc/ceph/ceph.conf.new: Permission denied We were thinking about removing the keyring from the Ceph orchestrator (https://docs.ceph.com/en/latest/cephadm/operations/#putting-a-keyring-under…) which would then make Ceph not try to copy over a new ceph.conf, alleviating the problem (https://docs.ceph.com/en/latest/cephadm/operations/#client-keyrings-and-con…) but in doing so, Ceph will kindly remove the key from all nodes (https://docs.ceph.com/en/latest/cephadm/operations/#disabling-management-of…) leaving us without the admin keyring. So that doesn’t sound like a path we want to take :S Does anybody know how to get around this issue, so I can get to version where the issue fixed for good? Thanks, Magnus

3 3

Re: Please guide us inidentifying thecause ofthedata miss in EC pool
by Best Regards 08 Aug '24

08 Aug '24

Hi, Frédéric Nass Yes. I checked the host running time where the OSD is located and the OSD service running time.These were stopped when I executed `ceph-object-tool`. They were running before. Because we need to maintain the hardware frequently (the hardware is quite old), min_size is set to the lowest value. When a failure occurs, we set the read and write pause flag. During the failure, there is no PUT action on the S3 keys to which these objects belong. Best Regards Best Regards wu_chulin(a)qq.com Best Regards Original Email From:"Frédéric Nass"< frederic.nass(a)univ-lorraine.fr >; Sent Time:2024/8/8 14:40 To:"Best Regards"< wu_chulin(a)qq.com >; Cc recipient:"ceph-users"< ceph-users(a)ceph.io >; Subject:[ceph-users] Re: Please guide us inidentifying thecause ofthedata miss in EC pool Hi Chulin, Are you 100% sure that 494, 1169 and 1057 (that did not restart) were in the acting set at the exact moment the power outage occured?  I'm asking because min_size 6 would have allowed the data to be written to eventually 6 crashing OSDs. Bests, Frédéric. ________________________________ De : Best Regards Envoyé : jeudi 8 août 2024 08:10 À : Frédéric Nass Cc: ceph-users Objet : Re:Re: Re:Re: Re:Re: Re:Re: [ceph-users] Please guide us inidentifying thecause ofthedata miss in EC pool Hi， Frédéric Nass Thank you for your continued attention and guidance. Let's analyze and verify this issue from different perspectives. The reason why we did not stop the investigation is that we tried to find other ways to avoid the losses caused by this sudden failure. Turning off the disk cache is the last option, of course, this operation will only be carried out after finding definite evidence. I also have a question that among the 9 OSDs, some have not been restarted. In theory, these OSDs should retain the object info(metadata,pg_log,etc.), even if the object cannot be recovered. I sorted out the OSD booting log where the object should be located and the PG peering process: OSD 494/1169/1057 has been in the running state, and osd.494 was the primary of the acting_set during the failure. However, no record of the object was found using `ceph-object-tool --op list or --op log` in, so the loss of data due to disk cache loss does not seem to be a complete explanation (perhaps there is some processing logic that we have not paid attention to). Best Regards, Woo wu_chulin(a)qq.com Best Regards Original Email From:"Frédéric Nass"< frederic.nass(a)univ-lorraine.fr >; Sent Time:2024/8/8 4:01 To:"wu_chulin"< wu_chulin(a)qq.com >; Subject:Re: Re:Re: Re:Re: Re:Re: [ceph-users] Please guide us inidentifying thecause ofthedata miss in EC pool Hey Chulin, Looks clearer now. Non-persistent cache for KV metadata and Bluestore metadata certainly explains how data was lost without the cluster even noticing. What's unexpected is data staying for so long in the disks buffers and not being written to persistent sectors at all. Anyways, thank you for sharing your use case and investigation. It was nice chatting with you. If you can, share this in the ceph-user list. It will for sure benefit everyone in the community. Best regards, Frédéric. PS : Note that using min_size >= k + 1 on EC pools is recommended (so as min_size >= 2 on rep X3 pools) because you don't want to write data without any parity chunks. De : wu_chulin(a)qq.com Envoyé : mercredi 7 août 2024 11:30 À : Frédéric Nass Objet : Re:Re: Re:Re: Re:Re: [ceph-users] Please guide us in identifying thecause ofthedata miss in EC pool Hi, Yes, after the file -> object -> PG -> OSD correspondence is found, the object record can be found on the specified OSD using the command `ceph-objectore-tool --op list ` The pool min_size is 6 The business department reported more than 30, but we proactively screened out more than 100. The upload time of the lost files was mainly distributed about 3 hours before the failure, and these files were successfully downloaded after being uploaded (RGW log). One OSD corresponds to one disk, and no separate space is allocated for WAL/DB. The HDD cache is the default (SATA is enabled by default), and the hard disk cache has not been forcibly turned off due to performance issues. The loss of OSD data due to the loss of hard disk cache was our initial inference, and the initial explanation provided to the business department was the same. When the cluster was restored, ceph reported 12 unfound objects, which is acceptable. After all, most devices were powered off abnormally, and it is difficult to ensure the integrity of all data. Up to now, our team have not located how the data was lost. In the past, when the hard disk hardware was damaged, either the OSD could not start because of damaged key data, or some objects were read incorrectly after the OSD started, which could be repaired. Now deep-scrub cannot find the problem, which may be related to the loss (or deletion) of object metadata. After all, deep-scrub needs the object list of the current PG. If those 9 OSDs do not have the object metadata information, deep-scrub does not know the existence of this object. wu_chulin(a)qq.com wu_chulin(a)qq.com Original Email From:"Frédéric Nass"< frederic.nass(a)univ-lorraine.fr >; Sent Time:2024/8/6 20:40 To:"wu_chulin"< wu_chulin(a)qq.com >; Subject:Re: Re:Re: Re:Re: [ceph-users] Please guide us in identifying thecause ofthedata miss in EC pool That's interesting. Have you tried to correlate any existing retrievable object to PG id and OSD mapping in order to verify the presence of each of these object's shards using ceph-objectore-tool on each one of its acting OSDs, for a previously and successfully written S3 object. This would help verify that the command you've run trying to find the missing shard was good. Also what min_size is this pool using?    How many S3 objects like these were reported missing by the business unit? Have you or they made an inventory of unretrievable / missing objects? Are WAL / RocksDB collocated on HDDs only OSDs or are these OSDs using SSDs for WAL / RocksDB? Did you disable HDD buffers (also known as disk cache) as... HDD buffers are non-persistent. I know nothing about the intensity of your workloads but if your looking for a few tens or a few hundreds of unwritten s3 objects, there might be some situation with non-persistent cache (like volatile disk buffers) were ceph would consider the data to be written when it was actually not at the moment of the power outage, due to the use of non-persistent disk buffers. Especially if you kept writing data with less shards (min_size) than k+1 (no parity at all). That sound like a possibility. Also what I'm thinking right is... If you can identify which shard over 9 is wrong, then you may use ceph-objectore-tool or ceph-kvstore-tool to destroy this particular shard, then deep-scrub the PG so to detect the inexistent shard and have it rebuilt. Never tried this myself though. Best regards, Frédéric. De : wu_chulin(a)qq.com Envoyé : mardi 6 août 2024 12:15 À : Frédéric Nass Objet : Re:Re: Re:Re: [ceph-users] Please guide us in identifying the cause ofthedata miss in EC pool Hi,  Thank you for your attention to this matter. 1. Manually executing deep-scrub PG did not report any errors. I checked the OSD logs and did not see any errors detected or fixed by the OSD. Ceph health was also normal, and the OS where the OSD was located did not report any IO type errors. 2. At first, I also suspected whether the objects were distributed on these OSDs before the failure. I used ceph osd getmap 3. Our Ceph version is 13.2.10, which does not have pg autoscaler; lifecycle policies are not set. 4. We have 90+ hosts, most of which are Dell R720xd, most of the hard disks are 3.5 inches/5400 rpm/10TB Western Digital/, and most of the controllers are PERC H330 Mini; this is the current cluster status: cluster: id: f990db28-9604-4d49-9733-b17155887e3b health: HEALTH_OK services:   mon: 5 daemons, quorum cz-ceph-01,cz-ceph-02,cz-ceph-03,cz-ceph-07,cz-ceph-13   mgr: cz-ceph-01(active), standbys: cz-ceph-03, cz-ceph-02   osd: 1172 osds: 1172 up, 1172 in    rgw: 9 daemons active  data:    pools: 16 pools, 25752 pgs    objects: 2.23 G objects, 6.1 PiB    usage: 9.4 PiB used, 2.5 PiB / 12 PiB avail    pgs:      25686 active+clean                 64 active+clean+scrubbing+deep+repair                 2 active+clean+scrubbing+deep Best regards. Original Email From:"Frédéric Nass"< frederic.nass(a)univ-lorraine.fr >; Sent Time:2024/8/6 15:34 To:"wu_chulin"< wu_chulin(a)qq.com >; Subject:Re: Re:Re: [ceph-users] Please guide us in identifying the cause ofthedata miss in EC pool Hi, Did the deep-scrub report any errors? (any inconsistencies should show errors after deep-scrubbing the PG.)   We're these errors fixed by the PG repair? Is is possible that you looked for the wrong PG or OSD when trying to list these objects with ceph-objectstore-tool? Were the PG autoscaler running at that time? Are you using S3 lifecycle policies that could have move this object to another placement pool and so another PG? Can you give details about this cluster? Hardware, disks, controller, etc. Cheers, Frédéric. De : wu_chulin(a)qq.com Envoyé : lundi 5 août 2024 10:09 À : Frédéric Nass Objet : Re:Re: [ceph-users] Please guide us in identifying the cause of thedata miss in EC pool Hi， Thank you for your reply. I apologize for the omission in the previous email. Please disregard the previous email and refer to this one instead.  After the failure, we executed the repair and deep-scrub command on some of the PGs that lost data, and the status was active+clean after completion, but the object still could not be retrieved. Our erasure code parameters are k=6, m=3. Theoretically, the data of three OSDs lost due to power failure should be recoverable. However, we stopped nine OSDs and exported the object list, but could not find the lost object information. What puzzled us was that some OSDs were not powered off and were still running, but their object lists did not have the information too. Best regards. wu_chulin(a)qq.com wu_chulin(a)qq.com Original Email From:"Frédéric Nass"< frederic.nass(a)univ-lorraine.fr >; Sent Time:2024/8/3 15:11 To:"wu_chulin"< wu_chulin(a)qq.com >;"ceph-users"< ceph-users(a)ceph.io >; Subject:Re: [ceph-users] Please guide us in identifying the cause of thedata miss in EC pool Hi, First thing that comes to mind when it comes to data unavailability or inconsistencies after a power outage is that some dirty data may have been lost along the IO path before reaching persistent storage. This can happen with non enterprise grade SSDs using non-persistent cache or with HDDs disk buffer if left enabled for example. With that said, have you tried to deep-scrub the PG from which you can't retrieve data? What's the status of this PG now? Did it recover? Regards, Frédéric. De : wu_chulin(a)qq.com Envoyé : mercredi 31 juillet 2024 05:49 À : ceph-users Objet : [ceph-users] Please guide us in identifying the cause of the data miss in EC pool Dear Ceph team:&nbsp; &nbsp; &nbsp;On July 13th at 4:55 AM, our Ceph cluster experienced a significant power outage in the data center, causing a large number of OSDs to power off and restart (total: 1172, down: 821). Approximately two hours later, all OSDs successfully started, and the cluster resumed its services. However, around 6 PM, the business department reported that some files, which had been successfully written (via the RGW service), were failing to download, and the number of such files was quite significant. Consequently, we began a series of investigations: 1. The incident occurred at 04:55. At 05:01, we executed noout, nobackfill, and norecover. At 06:22, we executed `ceph osd pause`. By 07:23, all OSDs were UP&amp;IN, and subsequently, we executed `ceph osd unpause`. 2. We randomly selected a problematic file and attempted to download it via the S3 API. The RGW returned "No such key". 3. The RGW logs showed op status=-2, http status=200. We also checked the upload logs, which indicated 2024-07-13 04:19:20.052, op status=0, http_status=200. 4. We set debug_rgw=20 and attempted to download the file again. It was found that a 4M chunk(this file is 64M) failed to get. 5. Using rados get for this chunk returned: "No such file or directory". 6. Setting debug_osd=20, we observed get_object_context: obc NOT found in cache. 7. Setting debug_bluestore=20, we saw get_onode oid xxx, key xxx != '0xfffffffffffffffeffffffffffffffff'o'. 8. We stopped the primary OSD and tried to get the file again, but the result was the same. The object’s corresponding PG state was active+recovery_wait+degraded. 9. Using ceph-objectstore-tool --op list &amp;&amp; --op log, we could not find the object information. The ceph-kvstore-tool rocksdb command also did not reveal anything new. 10. If an OSD had lost data, the PG state should have been unfound or inconsistency. 11. We started reanalyzing the startup logs of the OSDs related to the PG. The pool was set to erasure-code 6-3 with 9 OSDs. Six of these OSDs had restarted, and after peering, the PG state became ACTIVE. 12. We divided the lost files, and the upload time was before the failure occurred. The earliest upload time was around 1 am, and the successful upload records could be found in the RGW log 13. We have submitted an issue on the Ceph issue tracker:&nbsp;https://tracker.ceph.com/issues/66942, it includes the original logs needed for troubleshooting. However, four days have passed without any response. In desperation, we are sending this email, hoping that someone from the Ceph team can guide us as soon as possible. We are currently in a difficult situation and hope you can provide guidance. Thank you. Best regards. wu_chulin(a)qq.com wu_chulin(a)qq.com _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Re: Please guide us inidentifying thecause ofthedata miss in EC pool
by Best Regards 08 Aug '24

08 Aug '24

Hi， Frédéric Nass Thank you for your continued attention and guidance. Let's analyze and verify this issue from different perspectives. The reason why we did not stop the investigation is that we tried to find other ways to avoid the losses caused by this sudden failure. Turning off the disk cache is the last option, of course, this operation will only be carried out after finding definite evidence. I also have a question that among the 9 OSDs, some have not been restarted. In theory, these OSDs should retain the object info(metadata,pg_log,etc.), even if the object cannot be recovered. I sorted out the OSD booting log where the object should be located and the PG peering process: OSD 494/1169/1057 has been in the running state, and osd.494 was the primary of the acting_set during the failure. However, no record of the object was found using `ceph-object-tool --op list or --op log` in, so the loss of data due to disk cache loss does not seem to be a complete explanation (perhaps there is some processing logic that we have not paid attention to). Best Regards, Woo wu_chulin(a)qq.com Best Regards Original Email From:"Frédéric Nass"< frederic.nass(a)univ-lorraine.fr >; Sent Time:2024/8/8 4:01 To:"wu_chulin"< wu_chulin(a)qq.com >; Subject:Re: Re:Re: Re:Re: Re:Re: [ceph-users] Please guide us inidentifying thecause ofthedata miss in EC pool Hey Chulin, Looks clearer now. Non-persistent cache for KV metadata and Bluestore metadata certainly explains how data was lost without the cluster even noticing. What's unexpected is data staying for so long in the disks buffers and not being written to persistent sectors at all. Anyways, thank you for sharing your use case and investigation. It was nice chatting with you. If you can, share this in the ceph-user list. It will for sure benefit everyone in the community. Best regards, Frédéric. PS : Note that using min_size >= k + 1 on EC pools is recommended (so as min_size >= 2 on rep X3 pools) because you don't want to write data without any parity chunks. De : wu_chulin(a)qq.com Envoyé : mercredi 7 août 2024 11:30 À : Frédéric Nass Objet : Re:Re: Re:Re: Re:Re: [ceph-users] Please guide us in identifying thecause ofthedata miss in EC pool Hi, Yes, after the file -> object -> PG -> OSD correspondence is found, the object record can be found on the specified OSD using the command `ceph-objectore-tool --op list ` The pool min_size is 6 The business department reported more than 30, but we proactively screened out more than 100. The upload time of the lost files was mainly distributed about 3 hours before the failure, and these files were successfully downloaded after being uploaded (RGW log). One OSD corresponds to one disk, and no separate space is allocated for WAL/DB. The HDD cache is the default (SATA is enabled by default), and the hard disk cache has not been forcibly turned off due to performance issues. The loss of OSD data due to the loss of hard disk cache was our initial inference, and the initial explanation provided to the business department was the same. When the cluster was restored, ceph reported 12 unfound objects, which is acceptable. After all, most devices were powered off abnormally, and it is difficult to ensure the integrity of all data. Up to now, our team have not located how the data was lost. In the past, when the hard disk hardware was damaged, either the OSD could not start because of damaged key data, or some objects were read incorrectly after the OSD started, which could be repaired. Now deep-scrub cannot find the problem, which may be related to the loss (or deletion) of object metadata. After all, deep-scrub needs the object list of the current PG. If those 9 OSDs do not have the object metadata information, deep-scrub does not know the existence of this object. wu_chulin(a)qq.com wu_chulin(a)qq.com Original Email From:"Frédéric Nass"< frederic.nass(a)univ-lorraine.fr >; Sent Time:2024/8/6 20:40 To:"wu_chulin"< wu_chulin(a)qq.com >; Subject:Re: Re:Re: Re:Re: [ceph-users] Please guide us in identifying thecause ofthedata miss in EC pool That's interesting. Have you tried to correlate any existing retrievable object to PG id and OSD mapping in order to verify the presence of each of these object's shards using ceph-objectore-tool on each one of its acting OSDs, for a previously and successfully written S3 object. This would help verify that the command you've run trying to find the missing shard was good. Also what min_size is this pool using?    How many S3 objects like these were reported missing by the business unit? Have you or they made an inventory of unretrievable / missing objects? Are WAL / RocksDB collocated on HDDs only OSDs or are these OSDs using SSDs for WAL / RocksDB? Did you disable HDD buffers (also known as disk cache) as... HDD buffers are non-persistent. I know nothing about the intensity of your workloads but if your looking for a few tens or a few hundreds of unwritten s3 objects, there might be some situation with non-persistent cache (like volatile disk buffers) were ceph would consider the data to be written when it was actually not at the moment of the power outage, due to the use of non-persistent disk buffers. Especially if you kept writing data with less shards (min_size) than k+1 (no parity at all). That sound like a possibility. Also what I'm thinking right is... If you can identify which shard over 9 is wrong, then you may use ceph-objectore-tool or ceph-kvstore-tool to destroy this particular shard, then deep-scrub the PG so to detect the inexistent shard and have it rebuilt. Never tried this myself though. Best regards, Frédéric. De : wu_chulin(a)qq.com Envoyé : mardi 6 août 2024 12:15 À : Frédéric Nass Objet : Re:Re: Re:Re: [ceph-users] Please guide us in identifying the cause ofthedata miss in EC pool Hi,  Thank you for your attention to this matter. 1. Manually executing deep-scrub PG did not report any errors. I checked the OSD logs and did not see any errors detected or fixed by the OSD. Ceph health was also normal, and the OS where the OSD was located did not report any IO type errors. 2. At first, I also suspected whether the objects were distributed on these OSDs before the failure. I used ceph osd getmap <epoch_before_failure> and got "Error ENOENT: there is no map for epoch xxxxx". I am not sure if there are other ways to get the osdmap before the failure. After the failure, all OSDs joined the cluster, and the number was large (1000+). It was a bit difficult to find the OSD corresponding to the object without osdmap, so we didn't go into this direction. When the business department reported that the files could not be downloaded, I first opened the RGW debug log, found the mapping between file_name (64M) and oid (4M), then found the corresponding OSD according to ceph osd map <pool> <oid>, and then opened the OSD debug log to find the error information. 3. Our Ceph version is 13.2.10, which does not have pg autoscaler; lifecycle policies are not set. 4. We have 90+ hosts, most of which are Dell R720xd, most of the hard disks are 3.5 inches/5400 rpm/10TB Western Digital/, and most of the controllers are PERC H330 Mini; this is the current cluster status: cluster: id: f990db28-9604-4d49-9733-b17155887e3b health: HEALTH_OK services:   mon: 5 daemons, quorum cz-ceph-01,cz-ceph-02,cz-ceph-03,cz-ceph-07,cz-ceph-13   mgr: cz-ceph-01(active), standbys: cz-ceph-03, cz-ceph-02   osd: 1172 osds: 1172 up, 1172 in    rgw: 9 daemons active  data:    pools: 16 pools, 25752 pgs    objects: 2.23 G objects, 6.1 PiB    usage: 9.4 PiB used, 2.5 PiB / 12 PiB avail    pgs:      25686 active+clean                 64 active+clean+scrubbing+deep+repair                 2 active+clean+scrubbing+deep Best regards. Original Email From:"Frédéric Nass"< frederic.nass(a)univ-lorraine.fr >; Sent Time:2024/8/6 15:34 To:"wu_chulin"< wu_chulin(a)qq.com >; Subject:Re: Re:Re: [ceph-users] Please guide us in identifying the cause ofthedata miss in EC pool Hi, Did the deep-scrub report any errors? (any inconsistencies should show errors after deep-scrubbing the PG.)   We're these errors fixed by the PG repair? Is is possible that you looked for the wrong PG or OSD when trying to list these objects with ceph-objectstore-tool? Were the PG autoscaler running at that time? Are you using S3 lifecycle policies that could have move this object to another placement pool and so another PG? Can you give details about this cluster? Hardware, disks, controller, etc. Cheers, Frédéric. De : wu_chulin(a)qq.com Envoyé : lundi 5 août 2024 10:09 À : Frédéric Nass Objet : Re:Re: [ceph-users] Please guide us in identifying the cause of thedata miss in EC pool Hi， Thank you for your reply. I apologize for the omission in the previous email. Please disregard the previous email and refer to this one instead.  After the failure, we executed the repair and deep-scrub command on some of the PGs that lost data, and the status was active+clean after completion, but the object still could not be retrieved. Our erasure code parameters are k=6, m=3. Theoretically, the data of three OSDs lost due to power failure should be recoverable. However, we stopped nine OSDs and exported the object list, but could not find the lost object information. What puzzled us was that some OSDs were not powered off and were still running, but their object lists did not have the information too. Best regards. wu_chulin(a)qq.com wu_chulin(a)qq.com Original Email From:"Frédéric Nass"< frederic.nass(a)univ-lorraine.fr >; Sent Time:2024/8/3 15:11 To:"wu_chulin"< wu_chulin(a)qq.com >;"ceph-users"< ceph-users(a)ceph.io >; Subject:Re: [ceph-users] Please guide us in identifying the cause of thedata miss in EC pool Hi, First thing that comes to mind when it comes to data unavailability or inconsistencies after a power outage is that some dirty data may have been lost along the IO path before reaching persistent storage. This can happen with non enterprise grade SSDs using non-persistent cache or with HDDs disk buffer if left enabled for example. With that said, have you tried to deep-scrub the PG from which you can't retrieve data? What's the status of this PG now? Did it recover? Regards, Frédéric. De : wu_chulin(a)qq.com Envoyé : mercredi 31 juillet 2024 05:49 À : ceph-users Objet : [ceph-users] Please guide us in identifying the cause of the data miss in EC pool Dear Ceph team:&nbsp; &nbsp; &nbsp;On July 13th at 4:55 AM, our Ceph cluster experienced a significant power outage in the data center, causing a large number of OSDs to power off and restart (total: 1172, down: 821). Approximately two hours later, all OSDs successfully started, and the cluster resumed its services. However, around 6 PM, the business department reported that some files, which had been successfully written (via the RGW service), were failing to download, and the number of such files was quite significant. Consequently, we began a series of investigations: 1. The incident occurred at 04:55. At 05:01, we executed noout, nobackfill, and norecover. At 06:22, we executed `ceph osd pause`. By 07:23, all OSDs were UP&amp;IN, and subsequently, we executed `ceph osd unpause`. 2. We randomly selected a problematic file and attempted to download it via the S3 API. The RGW returned "No such key". 3. The RGW logs showed op status=-2, http status=200. We also checked the upload logs, which indicated 2024-07-13 04:19:20.052, op status=0, http_status=200. 4. We set debug_rgw=20 and attempted to download the file again. It was found that a 4M chunk(this file is 64M) failed to get. 5. Using rados get for this chunk returned: "No such file or directory". 6. Setting debug_osd=20, we observed get_object_context: obc NOT found in cache. 7. Setting debug_bluestore=20, we saw get_onode oid xxx, key xxx != '0xfffffffffffffffeffffffffffffffff'o'. 8. We stopped the primary OSD and tried to get the file again, but the result was the same. The object’s corresponding PG state was active+recovery_wait+degraded. 9. Using ceph-objectstore-tool --op list &amp;&amp; --op log, we could not find the object information. The ceph-kvstore-tool rocksdb command also did not reveal anything new. 10. If an OSD had lost data, the PG state should have been unfound or inconsistency. 11. We started reanalyzing the startup logs of the OSDs related to the PG. The pool was set to erasure-code 6-3 with 9 OSDs. Six of these OSDs had restarted, and after peering, the PG state became ACTIVE. 12. We divided the lost files, and the upload time was before the failure occurred. The earliest upload time was around 1 am, and the successful upload records could be found in the RGW log 13. We have submitted an issue on the Ceph issue tracker:&nbsp;https://tracker.ceph.com/issues/66942, it includes the original logs needed for troubleshooting. However, four days have passed without any response. In desperation, we are sending this email, hoping that someone from the Ceph team can guide us as soon as possible. We are currently in a difficult situation and hope you can provide guidance. Thank you. Best regards. wu_chulin(a)qq.com wu_chulin(a)qq.com _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

mds damaged with preallocated inodes that are inconsistent with inotable
by zxcs 08 Aug '24

08 Aug '24

HI, Experts, we are running a cephfs with V16.2.*, and has multi active mds. Currently, we are hitting a mds fs cephfs mds.* id damaged. and this mds always complain “client *** loaded with preallocated inodes that are inconsistent with inotable” and the mds always suicide during replay. Could anyone please help here ? We really need you shed some light! Thanks lot ! xz

2 1

Can you return orphaned objects to a bucket?
by motaharesdq＠gmail.com 08 Aug '24

08 Aug '24

Hello everyone, Is there any way to return orphaned objects to a bucket? A large bucket was accidentally corrupted and emptied, but its objects do exist in rados. bi list and bucket check returns empty list and bucket fix don't do anything as expected. How can we index associated rados objects back to the bucket?

3 3

Re: Cephadm: unable to copy ceph.conf.new
by Eugen Block 07 Aug '24

07 Aug '24

Hi, please don't drop the ML from your response. Is this the first upgrade you're attempting or did previous upgrades work with the current config? > I wonder if can generate a new ssh configuration for the root user, > and then use that to upgrade to the fixed version. > The permissions will then be owned by root, which means we can't use > the ceph user, no? I do remember having an issue with non-root user on a customer cluster, but IIRC it was because of insufficient sudo permissions. In the end, they switched to root user, and there haven't been any issues since, at least nobody reported anything to me. Do you mind sharing your sudo config for the ceph user? Thanks, Eugen Zitat von Magnus Larsen <magnusfynbo(a)hotmail.com>: > Hi, > > We do have client-keyring with the label: > # ceph orch client-keyring ls > ENTITY PLACEMENT MODE OWNER PATH > client.admin label:_admin rw------- 0:0 > /etc/ceph/ceph.client.admin.keyring > > And the SSH-config is also correct (verified just now) - though we > use ceph as the user, not the default root, > which works normally, except that we can't upgrade until we get the > fix in... which is in the next upgrade :< > > I wonder if can generate a new ssh configuration for the root user, > and then use that to upgrade to the fixed version. > The permissions will then be owned by root, which means we can't use > the ceph user, no? > > ref: https://docs.ceph.com/en/octopus/cephadm/operations/#ssh-configuration > > Thanks! > Magnus Larsen > > ________________________________ > Fra: Eugen Block <eblock(a)nde.ag> > Sendt: 7. august 2024 09:15 > Til: ceph-users(a)ceph.io <ceph-users(a)ceph.io> > Emne: [ceph-users] Re: Cephadm: unable to copy ceph.conf.new > > Hi, > > I commented a similar issue a couple of months ago: > > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/IQX2VXA6QQ… > > Can you check if that applies to your cluster? > > Zitat von Magnus Larsen <magnusfynbo(a)hotmail.com>: > >> Hi Ceph-users! >> >> Ceph version: ceph version 17.2.6 >> (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) >> Using cephadm to orchestrate the Ceph cluster >> >> I’m running into https://tracker.ceph.com/issues/59189, which is >> fixed in next version—quincy 17.2.7—via >> https://github.com/ceph/ceph/pull/50906 >> >> But I am unable to upgrade to the fixed version because of that bug >> >> When I try to upgrade (using “ceph orch upgrade start –image >> internal_mirror/ceph:v17.2.7”), we see the same error message: >> executing _write_files((['dkcphhpcadmin01', 'dkcphhpcmgt028', >> 'dkcphhpcmgt029', 'dkcphhpcmgt031', 'dkcphhpcosd033', >> 'dkcphhpcosd034', 'dkcphhpcosd035', 'dkcphhpcosd036', >> 'dkcphhpcosd037', 'dkcphhpcosd038', 'dkcphhpcosd039', >> 'dkcphhpcosd040', 'dkcphhpcosd041', 'dkcphhpcosd042', >> 'dkcphhpcosd043', 'dkcphhpcosd044'],)) failed. Traceback (most >> recent call last): File "/usr/share/ceph/mgr/cephadm/ssh.py", line >> 240, in _write_remote_file conn = await >> self._remote_connection(host, addr) File >> "/lib/python3.6/site-packages/asyncssh/scp.py", line 922, in scp >> await source.run(srcpath) File >> "/lib/python3.6/site-packages/asyncssh/scp.py", line 458, in run >> self.handle_error(exc) File >> "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in >> handle_error raise exc from None File >> "/lib/python3.6/site-packages/asyncssh/scp.py", line 456, in run >> await self._send_files(path, b'') File >> "/lib/python3.6/site-packages/asyncssh/scp.py", line 438, in >> _send_files self.handle_error(exc) File >> "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in >> handle_error raise exc from None File >> "/lib/python3.6/site-packages/asyncssh/scp.py", line 434, in >> _send_files await self._send_file(srcpath, dstpath, attrs) File >> "/lib/python3.6/site-packages/asyncssh/scp.py", line 365, in >> _send_file await self._make_cd_request(b'C', attrs, size, srcpath) >> File "/lib/python3.6/site-packages/asyncssh/scp.py", line 343, in >> _make_cd_request self._fs.basename(path)) File >> "/lib/python3.6/site-packages/asyncssh/scp.py", line 224, in >> make_request raise exc asyncssh.sftp.SFTPFailure: scp: >> /tmp/etc/ceph/ceph.conf.new: Permission denied During handling of >> the above exception, another exception occurred: Traceback (most >> recent call last): File "/usr/share/ceph/mgr/cephadm/utils.py", line >> 79, in do_work return f(*arg) File >> "/usr/share/ceph/mgr/cephadm/serve.py", line 1088, in _write_files >> self._write_client_files(client_files, host) File >> "/usr/share/ceph/mgr/cephadm/serve.py", line 1107, in >> _write_client_files self.mgr.ssh.write_remote_file(host, path, >> content, mode, uid, gid) File "/usr/share/ceph/mgr/cephadm/ssh.py", >> line 261, in write_remote_file >> self.mgr.wait_async(self._write_remote_file( File >> "/usr/share/ceph/mgr/cephadm/module.py", line 615, in wait_async >> return self.event_loop.get_result(coro) File >> "/usr/share/ceph/mgr/cephadm/ssh.py", line 56, in get_result return >> asyncio.run_coroutine_threadsafe(coro, self._loop).result() File >> "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result >> return self.__get_result() File >> "/lib64/python3.6/concurrent/futures/_base.py", line 384, in >> __get_result raise self._exception File >> "/usr/share/ceph/mgr/cephadm/ssh.py", line 249, in >> _write_remote_file logger.exception(msg) >> orchestrator._interface.OrchestratorError: Unable to write >> dkcphhpcmgt028:/etc/ceph/ceph.conf: scp: >> /tmp/etc/ceph/ceph.conf.new: Permission denied >> >> We were thinking about removing the keyring from the Ceph >> orchestrator >> (https://docs.ceph.com/en/latest/cephadm/operations/#putting-a-keyring-under…) >> which would then make Ceph not try to copy over a new ceph.conf, >> alleviating the problem >> (https://docs.ceph.com/en/latest/cephadm/operations/#client-keyrings-and-con…) >> but in doing so, Ceph will kindly remove the key from all nodes >> (https://docs.ceph.com/en/latest/cephadm/operations/#disabling-management-of…) >> leaving us without the admin keyring. So that doesn’t sound like a >> path we want to take :S >> >> Does anybody know how to get around this issue, so I can get to >> version where the issue fixed for good? >> >> Thanks, >> Magnus >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 2

What's the best way to add numerous OSDs?
by Fabien Sirjean 07 Aug '24

07 Aug '24

Hello everyone, We need to add 180 20TB OSDs to our Ceph cluster, which currently consists of 540 OSDs of identical size (replicated size 3). I'm not sure, though: is it a good idea to add all the OSDs at once? Or is it better to add them gradually? The idea is to minimize the impact of rebalancing on the performance of CephFS, which is used in production. Thanks in advance for your opinions and feedback 🙂 Wishing you a great summer, Fabien

4 3

RGW bucket notifications stop working after a while and blocking requests
by Florian Schwab 06 Aug '24

06 Aug '24

Hi, we just set up 2 new ceph clusters (using rook). To do some processing of the user activity we configured a topic that sends events to Kafka. After 5-12 hours this stops working with a 503 SlowDown response: debug 2024-08-02T09:17:58.205+0000 7ff4359ad700 1 req 13681579273117692719 0.005000019s ERROR: failed to reserve notification on queue: private.rgw. error: -28 First thought would be that the queue is full but up to this point see messages coming into Kafka and without much activity on the RGW itself (only a few requests against the S3 API) so it can’t be a load issue. What helps is to remove the notification configuration on the buckets (put-bucket-notification-configuration). If we directly re-add the previous notification configuration it also continuous working for a few hours before failing again with the same error/behaviour. We haven’t been able to reproduce this if we disable persistence for the topic so it looks like it is related to the persistence option - otherwise there would be also no queuing of the event for sending to Kafka. This also suggests that the issue is not with Kafka - this is also what we suspected first e.g. it can’t handle the amount of messages etc. Does anyone else have or had this issue and found the cause or a suggestion on how to best continue debugging? Are there detailed metrics etc. on the size and usage of the event queue? Here is the configuration for the topic and for a bucket: $ radosgw-admin topic list { "topics": [ { "user": "", "name": "private.rgw", "dest": { "push_endpoint": "kafka://rgw-sasl-kafka-user:XXX@kafka-kafka-bootstrap.kafka.svc:9094/private.rgw?sasl.mechanism=SCRAM-SHA-512&mechanism=SCRAM-SHA-512", "push_endpoint_args": "OpaqueData=&Version=2010-03-31&kafka-ack-level=broker&persistent=false&push-endpoint=kafka://rgw-sasl-kafka-user:XXX@kafka-kafka-bootstrap.kafka.svc:9094/private.rgw?sasl.mechanism=SCRAM-SHA-512&mechanism=SCRAM-SHA-512&use-ssl=true&verify-ssl=true", "push_endpoint_topic": "private.rgw", "stored_secret": true, "persistent": true }, "arn": "arn:aws:sns:ceph-objectstore::private.rgw", "opaqueData": "" } ] } $ aws s3api get-bucket-notification-configuration --bucket=XXX { "TopicConfigurations": [ { "Id": “my-id", "TopicArn": "arn:aws:sns:ceph-objectstore::private.rgw", "Events": [ "s3:ObjectCreated:*", "s3:ObjectRemoved:*" ] } ] } Thank you for any input to solve this! Cheers, Florian

2 3

Ceph Developer Summit (Tentacle) Aug 12-19
by Noah Lehman 06 Aug '24

06 Aug '24

Hi Ceph users, The next Ceph Developer Summit is happening virtually from August 12 – 19, 2024 adn we want to see you there. The focus of the summit will include planning around our next release, Tentacle, and everyone in our community is welcome to participate! Learn more and RSVP here: https://ceph.io/en/community/events/2024/ceph-developer-summit-tentacle/ Best, Noah

1 0