July 2024 - ceph-users - lists.ceph.io

could not find secret_id
by xiaowenhao111 22 Apr '26

22 Apr '26

2 1

problem with mgr prometheus module
by Dario Graña 17 Mar '26

17 Mar '26

Hi all! I'm running ceph quincy 17.2.7 in a cluster. On monday I updated the OS to AlmaLinux 9.3 to 9.4, since then grafana shows "No Data" message in all ceph related fields but, for example, the nodes information is still fine (Host Detail Dashboard). I have redeployed the mgr service with cephadm, disabled and re-enabled mgr prometheus module , but nothing changed. Digging into the problem, I accessed the prometheus interface. When I access prometheus, and found this error[image: Screen Shot 2024-06-04 at 15.22.37.png] When I access the node shown as down, it reports 503 Service Unavailable No cached data available yet Traceback (most recent call last): File "/lib/python3.6/site-packages/cherrypy/_cprequest.py", line 638, in respond self._do_respond(path_info) File "/lib/python3.6/site-packages/cherrypy/_cprequest.py", line 697, in _do_respond response.body = self.handler() File "/lib/python3.6/site-packages/cherrypy/lib/encoding.py", line 219, in __call__ self.body = self.oldhandler(*args, **kwargs) File "/lib/python3.6/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__ return self.callable(*self.args, **self.kwargs) File "/usr/share/ceph/mgr/prometheus/module.py", line 1751, in metrics return self._metrics(_global_instance) File "/usr/share/ceph/mgr/prometheus/module.py", line 1762, in _metrics raise cherrypy.HTTPError(503, 'No cached data available yet') cherrypy._cperror.HTTPError: (503, 'No cached data available yet') I checked the mgr prometheus address and port [ceph: root@ceph-admin01 /]# ceph config get mgr mgr/prometheus/server_addr :: [ceph: root@ceph-admin01 /]# ceph config get mgr mgr/prometheus/server_port 9283 It seems to be ok. When I check the master manager node for the port, I found [root@ceph-hn01 ~]# netstat -natup | grep 9283 tcp6 0 0 :::9283 :::* LISTEN 2453/ceph-mgr tcp6 0 0 192.168.97.51:9283 192.168.97.60:36130 ESTABLISHED 2453/ceph-mgr I don't understand why it is showing as IPv6, the node doesn't have a dual stack. I also tried to use a newer version of the prometheus container image, the 1.6.0, but it keeps reporting the same, so I rolled it back to the original one. Has anyone experienced an issue like this? Where can I look for more information about it? Thanks in advance. Regards. -- Dario Graña PIC (Port d'Informació Científica) Campus UAB, Edificio D E-08193 Bellaterra, Barcelona http://www.pic.es Avis - Aviso - Legal Notice: http://legal.ifae.es

2 2

OpenSSL security update for Octopus container?
by Robert Sander 02 Oct '25

02 Oct '25

Hi, I am running a Ceph Octopus cluster in version 15.2.10 setup with the cephadm orchestrator and official Docker hub container images. Yesterday an important security fix was released for libssl and is already packaged for all major distributions. I tried to run "ceph orch upgrade check docker.io/ceph/ceph:v15" but it tells me that the containers do not need to be upgraded. How will this security fix of OpenSSL be deployed in a timely manner to users of the Ceph container images? Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 93818 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 1

RBD Mirror - Failed to unlink peer
by Scott Cairns 01 Aug '25

01 Aug '25

Hi, Following the introduction of an additional node to our Ceph cluster, we've started to see unlink errors when taking a rbd mirror snapshot. We've had RBD mirroring configured for over a year now and it's been working flawlessly, however after we created OSD's on a new node we've receiving the following error: librbd::mirror::snapshot::CreatePrimaryRequest: 0x7f60c80056f0 handle_unlink_peer: failed to unlink peer: (2) No such file or directory This seemed to appear on around 3 of 150 snapshots on the first night and over the weeks has progressed to almost every snapshot. What's odd, is that the snapshot appears to be taken without any issues and does mirror to the DR site - we can see the snapshot ID taken on the source side is mirrored to the destination side when checking the rbd snap ls, and we've tested promoting an image on the DR site to ensure the snapshot does include up to date data, which it does. I can't see any other errors generated when the snapshot is taken to identify what file/directory isn't found - everything appears to be working okay it's just generating an error during the snapshot. I've also tried disabling mirroring on the disk and re-enabling however it doesn't appear to make any difference - there's no error on the initial mirror image, or the first snapshot taken after that, but every subsequent snapshot shows the error again. Any ideas? Thanks, Scott ________________________________ The content of this e-mail and any attachment is confidential and intended solely for the use of the individual to whom it is addressed. Any views or opinions presented are solely those of the author and do not necessarily represent those of Tecnica Limited. If you have received this e-mail in error please notify the sender. Any use, dissemination, forwarding, printing, or copying of this e-mail or any attachments thereto, in whole or part, without permission is strictly prohibited. Tecnica Limited Registered office: 5 Castle Court, Carnegie Campus, Dunfermline, Fife, KY11 8PB. Registered in Scotland No. SC250307. VAT No. 827 5110 42. This footnote also confirms that this email message has been swept for the presence of computer viruses.

4 8

Patching Ceph cluster
by Michael Worsham 25 Apr '25

25 Apr '25

What is the proper way to patch a Ceph cluster and reboot the servers in said cluster if a reboot is necessary for said updates? And is it possible to automate it via Ansible? This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments.

6 14

Migrating from S3 to Ceph RGW (Cloud Sync Module)
by James McClune 10 Apr '25

10 Apr '25

Hello Ceph User Community, I currently have a large Amazon S3 environment with terabytes of data spread over dozens of buckets. I'm looking to migrate from Amazon S3 to an on-site Ceph cluster using the RGW. I'm trying to figure out the most efficient way to achieve this. Looking through the documentation, I found articles related to the cloud sync module, released in Mimic ( https://docs.ceph.com/en/latest/radosgw/cloud-sync-module/) I also watched a video on the cloud sync module as well. It *sounds* like this is the functionality I'm looking for. Given I'm moving away from Amazon S3, I'm really just looking for a one-way replication between the buckets (i.e. Provide an Amazon S3 access key/secret which is associated to the buckets and the same for the Ceph environment, so object data can be replicated one-to-one, without creating ad-hoc tooling). Once the data is replicated from S3 to Ceph, I plan on modifying my boto connection objects to use the new Ceph environment. Is what I'm describing feasible with the cloud sync module? Just looking for some affirmation, given I'm not well versed in Ceph's RGW, especially around multi-site configurations. Thanks, Jimmy

7 8

Problem while upgrade 17.2.6 to 17.2.7
by Jean-Marc FONTANA 21 Feb '25

21 Feb '25

Hello everyone, We operate two clusters that we installed with ceph-deploy in Nautilus version on Debian 10. We use them for external S3 storage (owncloud) and rbd disk images.We had them upgraded to Octopus and Pacific versions on Debian 11 and recently converted them to cephadm and upgraded to Quincy (17.2.6). As we now have the orchestrator, we tried updating to 17.2.7 using the command# ceph orch upgrade start --image quay.io/ceph/ceph:v17.2.7 Everything went well, both clusters work perfectly for our use, except that the Rados gateway configuration is no longer accessible from the dashboard with the following error messageError connecting to Object Gateway: RGW REST API failed request with status code 404. We tried a few solutions found on the internet (reset rgw credentials, restart rgw adnd mgr, reenable dashboard, ...), unsuccessfully. Does somebody have an idea ? Best regards, Jean-Marc Fontana

5 9

RGW Lifecycle Problem (Reef)
by Chris Palmer 19 Feb '25

19 Feb '25

Reef 18.2.2, package install on Centos 9. This is a very straightforward production cluster, 2 RGW hosts, no multisite. 4 buckets have lifecycle policies: $ radosgw-admin lc list [ { "bucket": ":aaaaa:1d84db34-ed2b-400a-842e-344cdfa3deed.261076466.1", "shard": "lc.3", "started": "Mon, 15 Jul 2024 00:00:00 GMT", "status": "COMPLETE" }, { "bucket": ":bbbbb:1d84db34-ed2b-400a-842e-344cdfa3deed.304826873.1", "shard": "lc.3", "started": "Mon, 15 Jul 2024 00:00:00 GMT", "status": "COMPLETE" }, { "bucket": ":cccc:1d84db34-ed2b-400a-842e-344cdfa3deed.828249990.3", "shard": "lc.29", "started": "Thu, 01 Jan 1970 00:00:00 GMT", "status": "COMPLETE" }, { "bucket": ":dddd:1d84db34-ed2b-400a-842e-344cdfa3deed.547988508.2", "shard": "lc.29", "started": "Thu, 01 Jan 1970 00:00:00 GMT", "status": "UNINITIAL" } ] Symptoms are: - aaaaa & bbbbb work & list above as expected - ccccc works, but the "started" date is never updated - ddddd never fires, unless run manually (the listing above is 1 week after the policy was removed and cleanly reapplied, but with no manual run) The first three buckets are small, but ddddd is larger (6.3 TB, 3.2 M RGW objects). The policy for ddddd transitions objects to a different storage class. However this never happens unless I run the lifecycle manually. Then it works for that manual run, but never fires automatically each night. IIRC after a manual run the status updates to something else but I can't remember what, but I think the started date remains unset. Each night the following log entries can be seen in the ceph-client.rgw log on both RGW hosts: 2024-07-15T00:00:00.162+0000 7f1e44715640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.11 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:00.163+0000 7f1e44715640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:00.248+0000 7f1e46719640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.18 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:00.248+0000 7f1e46719640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:00.312+0000 7f1e46719640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.3 head last stored at Mon Jul 15 00:00:00 2024 2024-07-15T00:00:00.386+0000 7f1e42711640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.3 head last stored at Mon Jul 15 00:00:00 2024 2024-07-15T00:00:00.399+0000 7f1e42711640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.18 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:00.399+0000 7f1e42711640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:00.420+0000 7f1e42711640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.11 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:00.421+0000 7f1e42711640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:00.435+0000 7f1e42711640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.29 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:00.435+0000 7f1e42711640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:02.382+0000 7f1e46719640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.11 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:02.382+0000 7f1e46719640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:02.392+0000 7f1e46719640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.29 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:02.392+0000 7f1e46719640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:02.718+0000 7f1e44715640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.3 head last stored at Mon Jul 15 00:00:00 2024 2024-07-15T00:00:02.718+0000 7f1e44715640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:02.728+0000 7f1e44715640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.29 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:02.728+0000 7f1e44715640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 2024-07-15T00:00:02.791+0000 7f1e44715640 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.18 head last stored at Mon Nov 13 00:00:00 2023 2024-07-15T00:00:02.791+0000 7f1e44715640 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 In case it is relevant: for the RGW-related pools, index is R3, data is EC2+2 and the pool for the storage class we've moving objects to is EC4+2. Any ideas how to proceed with diagnosis/fix? Thanks, Chris

1 2

purging already destroyed OSD leads to degraded and misplaced objects?
by tobias tempel 10 Jan '25

10 Jan '25

Dear Cephers, reorganizing one of our clusters i'm removing some hosts from it, taking "out" all OSDs on these hosts and waiting until all PGs are fine. After stopping and destroying all OSDs on one host i notice, that "purge" of such destroyed OSDs temporarily leads to degraded and misplaced objects (tried "reweight 0" as well, same picture) . Why that? I would - due to my limited experience - expect, that completely removing an already destroyed OSD could not have any effect on object placement at all. This effect yet seems not to be a real problem, as recovery works fine ... but perhaps it indicates some other issue. All of that takes place at Pacific 16.2.14 AlmaLinux 8.9, yes i know, there's some work to do. Perhaps you can give me a hint, where to look, for an explanation? Thank you cheers, toBias -- Tobias Tempel Deutsches Elektronen-Synchrotron - IT Notkestr. 85, 22607 Hamburg, Germany email: tobias.tempel(a)desy.de

4 7

Problems with autoscaler (overlapping roots) after changing the pool class
by Massimo Sgaravatto 06 Jan '25

06 Jan '25

Dear all I have just changed the crush rule for all the replicated pools in the following way: ceph osd crush rule create-replicated replicated_hdd default host hdd ceph osd pool set <poolname> crush_rule replicated_hdd See also this [*] thread Before applying this change, these pools were all using the replicated_ruleset rule where the class is not specified. I am noticing now a problem with the autoscaler: "ceph osd pool autoscale-status" doesn't report any output and the mgr log complains about overlapping roots: [pg_autoscaler ERROR root] pool xyz has overlapping roots: {-18, -1} Indeed: # ceph osd crush tree --show-shadow ID CLASS WEIGHT TYPE NAME -18 hdd 1329.26501 root default~hdd -17 hdd 329.14154 rack Rack11-PianoAlto~hdd -15 hdd 54.56085 host ceph-osd-04~hdd 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... ... -1 1329.26501 root default -7 329.14154 rack Rack11-PianoAlto -8 54.56085 host ceph-osd-04 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... I have already read about this behavior but I have no clear ideas how to fix the problem. I read somewhere that the problem happens when there are rules that force some pools to only use one class and there are also pools which does not make any distinction between device classes All the replicated pools are using the replicated_hdd pool but I also have some EC pools which are using a profile where the class is not specified. As far I understand, I can't force these pools to use only the hdd class: according to the doc I can't change this profile specifying the hdd class (or at least the change wouldn't be applied to the existing EC pools) Any suggestions ? The crush map is available at https://cernbox.cern.ch/s/gIyjbQbmoTFHCrr, if you want to have a look Many thanks, Massimo [*] https://www.mail-archive.com/ceph-users@ceph.io/msg18534.html

4 9

multifs and snapshots
by Dmitry Melekhov 12 Nov '24

12 Nov '24

Hello! There is note here https://docs.ceph.com/en/reef/dev/cephfs-snapshots/ about multiply filesystems and snapshots: If each FS gets its own pool things probably work, but this isn’t tested and may not be true. Is it still untested ? Thank you!

3 4

Help needed to configure erasure coding LRC plugin
by Michel Jouvin 16 Oct '24

16 Oct '24

Hi, As discussed in another thread (Crushmap rule for multi-datacenter erasure coding), I'm trying to create an EC pool spanning 3 datacenters (datacenters are present in the crushmap), with the objective to be resilient to 1 DC down, at least keeping the readonly access to the pool and if possible the read-write access, and have a storage efficiency better than 3 replica (let say a storage overhead <= 2). In the discussion, somebody mentioned LRC plugin as a possible jerasure alternative to implement this without tweaking the crushmap rule to implement the 2-step OSD allocation. I looked at the documentation (https://docs.ceph.com/en/latest/rados/operations/erasure-code-lrc/) but I have some questions if someone has experience/expertise with this LRC plugin. I tried to create a rule for using 5 OSDs per datacenter (15 in total), with 3 (9 in total) being data chunks and others being coding chunks. For this, based of my understanding of examples, I used k=9, m=3, l=4. Is it right? Is this configuration equivalent, in terms of redundancy, to a jerasure configuration with k=9, m=6? The resulting rule, which looks correct to me, is: -------- { "rule_id": 6, "rule_name": "test_lrc_2", "ruleset": 6, "type": 3, "min_size": 3, "max_size": 15, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -4, "item_name": "default~hdd" }, { "op": "choose_indep", "num": 3, "type": "datacenter" }, { "op": "chooseleaf_indep", "num": 5, "type": "host" }, { "op": "emit" } ] } ------------ Unfortunately, it doesn't work as expected: a pool created with this rule ends up with its pages active+undersize, which is unexpected for me. Looking at 'ceph health detail` output, I see for each page something like: pg 52.14 is stuck undersized for 27m, current state active+undersized, last acting [90,113,2147483647,103,64,147,164,177,2147483647,133,58,28,8,32,2147483647] For each PG, there is 3 '2147483647' entries and I guess it is the reason of the problem. What are these entries about? Clearly it is not OSD entries... Looks like a negative number, -1, which in terms of crushmap ID is the crushmap root (named "default" in our configuration). Any trivial mistake I would have made? Thanks in advance for any help or for sharing any successful configuration? Best regards, Michel

4 24

rgw connection resets
by Nathan Gleason 10 Oct '24

10 Oct '24

Hello, I’ve noticed what I think may be strange behavior from some rgw transfers. It only seems to be with larger multipart mp4 files. Once a connection comes into rgw it is accepted and processed, data begins to transfer, then I see "write_data failed: Connection reset by peer” and "s3:get_obj iterate_obj() failed with -104”. The client then reconnects and the transfer seems to be ok. This doesn’t appear to happen just downloading a regular file. This is happening to both my production and test cluster. I’m not seeing errors anywhere else, the network looks good and the clusters are healthy. I’m in the process of collecting some tcpdumps but figured I would ask if anyone knows why this is occurring. We run haproxy in front of the rgw. For troubleshooting purposes we have a basic haproxy config to just one rgw. We have tested with multiple browsers. Ceph 17.2.5 - cluster built and managed with cephadm Ubuntu 20.04.6 LTS Log: 2024-01-03T17:40:23.916+0000 7f646033d700 1 ====== starting new request req=0x7f640ca95730 ===== 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s initializing for trans_id = tx000003ec1e4b4dc157efd-0065959c07-104216-default 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s rgw api priority: s3=8 s3website=7 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s host=a2-west.usacld.net 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s final domain/bucket subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 s->info.domain= s->info.request_uri=/n athan:tshoot/testfile.mp4 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s get_handler handler=22RGWHandler_REST_Obj_S3 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s handler=22RGWHandler_REST_Obj_S3 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s getting op 0 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s get_system_obj_state: rctx=0x7f640ca947c0 obj=default.rgw.log:script.prerequest. state=0x55fa31f882e0 s->prefetch_data=0 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s cache get: name=default.rgw.log++script.prerequest. : hit (negative entry) 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj scheduling with throttler client=2 cost=1 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj op=21RGWGetObj_ObjStore_S3 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj verifying requester 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rgw::auth::StrategyRegistry::s3_main_strategy_t: trying rgw::auth::s3::AWSAuthStrategy 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::S3AnonymousEngine 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rgw::auth::s3::S3AnonymousEngine granted access 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rgw::auth::s3::AWSAuthStrategy granted access 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj normalizing buckets and tenants 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s->object=testfile.mp4 s->bucket=nathan/tshoot 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj init permissions 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_system_obj_state: rctx=0x7f640ca94200 obj=default.rgw.meta:root:nathan/tshoot state=0x55fa31f882e0 s->prefetch _data=0 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj cache get: name=default.rgw.meta+root+nathan/tshoot : hit (requested=0x16, cached=0x17) 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_system_obj_state: s->obj_tag was set empty 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj cache get: name=default.rgw.meta+root+nathan/tshoot : hit (requested=0x11, cached=0x17) 2024-01-03T17:40:23.916+0000 7f646033d700 15 req 4522146966294331133 0.000000000s s3:get_obj decode_policy Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner> <ID>nathan$testuser</ID><DisplayName>Nathan API User</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>nathan$testuser </ID><DisplayName>Nathan API User</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy> 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWSI_User_RADOS::read_user_info(): anonymous user 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj recalculating target 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj reading permissions 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_obj_state: rctx=0x7f640ca94ad0 obj=tshoot:testfile.mp4 state=0x55fa31082de8 s->prefetch_data=1 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj WARNING: blocking librados call 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj manifest: total_size = 88811314 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_obj_state: setting s->obj_tag to _0OUNxgsSsqTAs9Z6xh1SOKpJZLV_Vos 2024-01-03T17:40:23.916+0000 7f646033d700 15 req 4522146966294331133 0.000000000s s3:get_obj decode_policy Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner> <ID>nathan$testuser</ID><DisplayName>Nathan API User</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="Group"><URI>http://acs.amazonaws.c om/groups/global/AllUsers</URI></Grantee><Permission>READ</Permission></Grant><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>nathan$testuser</ID><DisplayNa me>Nathan API User</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy> 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj init op 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_system_obj_state: rctx=0x7f640ca94488 obj=default.rgw.meta:users.uid:nathan$testuser state=0x55fa31f882e0 s->p refetch_data=0 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj cache get: name=default.rgw.meta+users.uid+nathan$testuser : hit (requested=0x16, cached=0x17) 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_system_obj_state: s->obj_tag was set empty 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr: user.rgw.idtag 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj cache get: name=default.rgw.meta+users.uid+nathan$testuser : hit (requested=0x13, cached=0x17) 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj verifying op mask 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj required_mask= 1 user.op_mask=7 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj verifying op permissions 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj -- Getting permissions begin with perm_mask=49 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj Searching permissions for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=anonymous, acct_n ame=, subuser=, perm_mask=15, is_admin=0) mask=49 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj Searching permissions for uid=anonymous 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj Permissions for user not found 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj Searching permissions for group=1 mask=49 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj Found permission: 1 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj -- Getting permissions done for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=anonymous, acct_name=, subuser=, perm_mask=15, is_admin=0), owner=nathan$testuser, perm=1 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=anonymous, acct_name=, subuser=, perm_mask =15, is_admin=0) requested perm (type)=1, policy perm=1, user_perm_mask=15, acl perm=1 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj verifying op params 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj pre-executing 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj check rate limiting 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj executing 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_obj_state: rctx=0x7f640ca94ad0 obj=tshoot:testfile.mp4 state=0x55fa31082de8 s->prefetch_data=1 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.acl 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.content_type 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.etag 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.idtag 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.manifest 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.pg_ver 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.source_zone 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.tail_tag 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.x-amz-content-sha256 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.x-amz-date 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.x-amz-meta-s3cmd-attrs 2024-01-03T17:40:23.916+0000 7f646033d700 15 req 4522146966294331133 0.000000000s Encryption mode: 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_obj_state: rctx=0x7f640ca94ad0 obj=tshoot:testfile.mp4 state=0x55fa31082de8 s->prefetch_data=1 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__multipart_testfile.mp4.2~XSiTvBBtaU4 134dQCusv3kMP7YjDBk7.1 obj-ofs=0 read_ofs=0 len=4194304 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=4194304 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): result: ofs=4194304 stripe_ofs=4194304 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134 dQCusv3kMP7YjDBk7.1_1 obj-ofs=4194304 read_ofs=0 len=4194304 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=8388608 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): result: ofs=8388608 stripe_ofs=8388608 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134 dQCusv3kMP7YjDBk7.1_2 obj-ofs=8388608 read_ofs=0 len=4194304 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=12582912 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): result: ofs=12582912 stripe_ofs=12582912 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134 dQCusv3kMP7YjDBk7.1_3 obj-ofs=12582912 read_ofs=0 len=3145728 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=16777216 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): result: ofs=15728640 stripe_ofs=15728640 part_ofs=15728640 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__multipart_testfile.mp4.2~XSiTvBBtaU4 134dQCusv3kMP7YjDBk7.2 obj-ofs=15728640 read_ofs=0 len=4194304 2024-01-03T17:40:23.924+0000 7f6442301700 20 req 4522146966294331133 0.007999974s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.924+0000 7f6442301700 20 req 4522146966294331133 0.007999974s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=19922944 part_ofs=15728640 rule->part_size=15728640 2024-01-03T17:40:23.924+0000 7f6442301700 20 req 4522146966294331133 0.007999974s s3:get_obj RGWObjManifest::operator++(): result: ofs=19922944 stripe_ofs=19922944 part_ofs=15728640 rule->part_size=15728640 2024-01-03T17:40:23.924+0000 7f6442301700 20 req 4522146966294331133 0.007999974s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134 dQCusv3kMP7YjDBk7.2_1 obj-ofs=19922944 read_ofs=0 len=4194304 2024-01-03T17:40:23.932+0000 7f643d2f7700 20 req 4522146966294331133 0.015999949s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.932+0000 7f643d2f7700 20 req 4522146966294331133 0.015999949s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=24117248 part_ofs=15728640 rule->part_size=15728640 2024-01-03T17:40:23.932+0000 7f643d2f7700 20 req 4522146966294331133 0.015999949s s3:get_obj RGWObjManifest::operator++(): result: ofs=24117248 stripe_ofs=24117248 part_ofs=15728640 rule->part_size=15728640 2024-01-03T17:40:23.932+0000 7f643d2f7700 20 req 4522146966294331133 0.015999949s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134 dQCusv3kMP7YjDBk7.2_2 obj-ofs=24117248 read_ofs=0 len=4194304 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_status: e=0, sent=17, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_content_length: e=0, sent=26, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::complete_header: e=0, sent=460, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::set_account: e=1 2024-01-03T17:40:24.068+0000 7f642a2d1700 4 write_data failed: Connection reset by peer 2024-01-03T17:40:24.068+0000 7f642a2d1700 0 req 4522146966294331133 0.151999518s s3:get_obj iterate_obj() failed with -104 2024-01-03T17:40:24.156+0000 7f642eada700 2 req 4522146966294331133 0.239999235s s3:get_obj completing 2024-01-03T17:40:24.156+0000 7f642eada700 20 req 4522146966294331133 0.239999235s get_system_obj_state: rctx=0x7f640ca947c0 obj=default.rgw.log:script.postrequest.nathan state=0x55fa32424760 s->prefetch_data =0 2024-01-03T17:40:24.156+0000 7f642eada700 10 req 4522146966294331133 0.239999235s cache get: name=default.rgw.log++script.postrequest.nathan : hit (negative entry) 2024-01-03T17:40:24.156+0000 7f642eada700 30 AccountingFilter::complete_request: e=1, sent=0, total=0 2024-01-03T17:40:24.156+0000 7f642eada700 2 req 4522146966294331133 0.239999235s s3:get_obj op status=-104 2024-01-03T17:40:24.156+0000 7f642eada700 2 req 4522146966294331133 0.239999235s s3:get_obj http status=200 2024-01-03T17:40:24.156+0000 7f642eada700 1 ====== req done req=0x7f640ca95730 op status=-104 http_status=200 latency=0.239999235s ====== 2024-01-03T17:40:24.156+0000 7f642eada700 1 beast: 0x7f640ca95730: 10.30.0.196 - anonymous [03/Jan/2024:17:40:23.916 +0000] "GET /nathan:tshoot/testfile.mp4 HTTP/1.1" 200 0 - "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" - latency=0.239999235s 2024-01-03T17:40:24.156+0000 7f642eada700 20 failed to read header: end of stream 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_ACCEPT=*/* 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_ACCEPT_ENCODING=identity;q=1, *;q=0 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_ACCEPT_LANGUAGE=en-US,en;q=0.7 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_HOST=a2-west.usacld.net 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_RANGE=bytes=0- 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_REFERER=https://a2-west.usacld.net/nathan:tshoot/testfile.mp4 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_CH_UA="Not_A Brand";v="8", "Chromium";v="120", "Brave";v="120" 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_CH_UA_MOBILE=?0 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_CH_UA_PLATFORM="macOS" 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_FETCH_DEST=video 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_FETCH_MODE=no-cors 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_FETCH_SITE=same-origin 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_GPC=1 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_VERSION=1.1 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_X_FORWARDED_PORT=443 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_X_FORWARDED_PROTO=https 2024-01-03T17:40:24.160+0000 7f6503483700 20 REMOTE_ADDR=10.30.0.196 2024-01-03T17:40:24.160+0000 7f6503483700 20 REQUEST_METHOD=GET 2024-01-03T17:40:24.160+0000 7f6503483700 20 REQUEST_URI=/nathan:tshoot/testfile.mp4 2024-01-03T17:40:24.160+0000 7f6503483700 20 SCRIPT_URI=/nathan:tshoot/testfile.mp4 2024-01-03T17:40:24.160+0000 7f6503483700 20 SERVER_PORT=8000 2024-01-03T17:40:24.160+0000 7f6503483700 1 ====== starting new request req=0x7f640ca95730 ===== Thanks, Nathan

2 1

Not all Bucket Shards being used
by Christian Kugler 01 Oct '24

01 Oct '24

Hi, I have trouble with large OMAP files in a cluster in the RGW index pool. Some background information about the cluster: There is CephFS and RBD usage on the main cluster but for this issue I think only S3 is interesting. There is one realm, one zonegroup with two zones which have a bidirectional sync set up. Since this does not allow for autoresharding we have to do it by hand in this cluster – looking forward to Reef! From the logs: cluster 2023-07-17T22:59:03.018722+0000 osd.75 (osd.75) 623978 : cluster [WRN] Large omap object found. Object: 34:bcec3016:::.dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.5:head PG: 34.680c373d (34.5) Key count: 962091 Size (bytes): 277963182 The offending bucket looks like this: # radosgw-admin bucket stats \ | jq '.[] | select(.marker =="3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9") |"\(.num_shards) \(.usage["rgw.main"].num_objects)"' -r 131 9463833 Last week the number of objects was about 12 million. Which is why I reshareded the offending bucket twice, I think. Once to 129 and the second time to 131 because I wanted some leeway (or lieway? scnr, Sage). Unfortunately, even after a week the objects were still to big (the log line above is quite recent), so I looked into it again. # rados -p raum.rgw.buckets.index ls \ |grep .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9 \ |sort -V .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.0 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.1 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.2 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.3 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.4 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.5 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.6 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.7 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.8 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.9 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.10 # rados -p raum.rgw.buckets.index ls \ |grep .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9 \ |sort -V \ |xargs -IOMAP sh -c \ 'rados -p raum.rgw.buckets.index listomapkeys OMAP | wc -l' 1013854 1011007 1012287 1011232 1013565 998262 1012777 1012713 1012230 1010690 997111 Apparently, only 11 shards are in use. This would explain why the "Key usage" (from the log line) is about ten times higher than I would expect. How can I deal with this issue? One thing I could try to fix this would be to reshard to a lower number, but I am not sure if there are any risks associated with "downsharding". After that I could reshard to something like 97. Or I could directly "downshard" to 97. Also, the second zone has a similar problem, but as the error messsage lets me know, this would be a bad idea. Will it just take more time until the sharding is transferred to the seconds zone? Best, Christian Kugler

2 5

Old MDS container version when: Ceph orch apply mds
by opositorvlc＠mail.com 28 Sep '24

28 Sep '24

Hi All, I migrated my CEPH 18.2.2 cluster from a non cephadm configuration. All goes fine except MDS service was deployed in a old version: 17.0.0 I'm trying to deploy MDS daemons using ceph orch but CEPH always download an old MDS image from docker. How could I deploy the MDS service in the same 18.2.2 version that the rest of services? [root@master1 ~]# ceph orch apply mds datafs --placement="2 master1 master2" [root@master1 ~]# ceph orch ps NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mds.datafs.master1.gcpovr master1 running (36m) 6m ago 36m 37.2M - 17.0.0-7183-g54142666 75e3d7089cea 96682779c7ad mds.datafs.master2.oqaxuy master2 running (36m) 6m ago 36m 33.1M - 17.0.0-7183-g54142666 75e3d7089cea a9a647f87c83 mgr.master master1 running (16h) 6m ago 17h 448M - 18.2.2 3c937764e6f5 70f06fa05b70 mgr.master2 master2 running (16h) 6m ago 17h 524M - 18.2.2 3c937764e6f5 2d0d5376d8b3 mon.master master1 running (16h) 6m ago 17h 384M 2048M 18.2.2 3c937764e6f5 66a65017ce29 mon.master2 master2 running (16h) 6m ago 17h 380M 2048M 18.2.2 3c937764e6f5 51d783a9e36c osd.0 osd00 running (16h) 3m ago 17h 432M 4096M 18.2.2 3c937764e6f5 fedff66f5ed2 osd.1 osd00 running (16h) 3m ago 17h 475M 4096M 18.2.2 3c937764e6f5 24e24a1a22e6 osd.2 osd00 running (16h) 3m ago 17h 516M 4096M 18.2.2 3c937764e6f5 ccd05451b739 osd.3 osd00 running (16h) 3m ago 17h 454M 4096M 18.2.2 3c937764e6f5 f6d8f13c8aaf osd.4 master1 running (16h) 6m ago 17h 525M 4096M 18.2.2 3c937764e6f5 a2dcf9f1a9b7 osd.5 master2 running (16h) 6m ago 17h 331M 4096M 18.2.2 3c937764e6f5 b0011e8561a4 [root@master1 ~]# ceph orch ls NAME PORTS RUNNING REFRESHED AGE PLACEMENT mds.datafs 2/2 6m ago 46s master1;master2;count:2 mgr 2/0 6m ago - <unmanaged> mon 2/0 6m ago - <unmanaged> osd 6 6m ago - <unmanaged> [root@master1 ~]# ceph versions { "mon": { "ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable)": 2 }, "mgr": { "ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable)": 2 }, "osd": { "ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable)": 6 }, "mds": { "ceph version 17.0.0-7183-g54142666 (54142666e5705ced88e3e2d91ddc0ff29867a362) quincy (dev)": 2 }, "overall": { "ceph version 17.0.0-7183-g54142666 (54142666e5705ced88e3e2d91ddc0ff29867a362) quincy (dev)": 2, "ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable)": 10 } } [root@master1 ~]# podman images REPOSITORY TAG IMAGE ID CREATED SIZE quay.io/ceph/ceph v18.2.2 3c937764e6f5 7 weeks ago 1.28 GB quay.io/ceph/ceph v18 3c937764e6f5 7 weeks ago 1.28 GB registry.access.redhat.com/ubi8 latest c70d72aaebb4 3 months ago 212 MB quay.io/ceph/ceph v16 0d668911f040 23 months ago 1.27 GB quay.io/ceph/ceph-grafana 8.3.5 dad864ee21e9 2 years ago 571 MB quay.io/prometheus/prometheus v2.33.4 514e6a882f6e 2 years ago 205 MB quay.io/prometheus/node-exporter v1.3.1 1dbe0e931976 2 years ago 22.3 MB quay.io/prometheus/alertmanager v0.23.0 ba2b418f427c 2 years ago 58.9 MB docker.io/ceph/daemon-base latest-master-devel 75e3d7089cea 2 years ago 1.29 GB

3 2

Numa pinning best practices
by Szabo, Istvan (Agoda) 15 Sep '24

15 Sep '24

Hi, Haven't really found a proper descripton in case of 2 socket how to pin osds to numa node, only this: https://tracker.ceph.com/projects/ceph/wiki/Tuning_for_All_Flash_Deployment… Tuning for All Flash Deployments - Ceph - Ceph <https://tracker.ceph.com/projects/ceph/wiki/Tuning_for_All_Flash_Deployment…> Redmine tracker.ceph.com Is there anybody have some good how to on this topic? Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

5 5

lifecycle policy on non-replicated buckets
by Christopher Durham 12 Sep '24

12 Sep '24

Hi, I have a multisite system with two sites on 18.2.2, on Rocky 8. I have set up a sync policy to allow replication between sites. I have also createda policy for a given bucket that prevents replication on that given bucket. This allworks just fine, and objects I create in that bucket on side A do not get replicatedto side B, and objects I create in that bucket on side B do not get replicated to side A, butreplication for all other buckets works fine. This is great. But I still want to have a lifecycle policy that deletes objects after say, 4 days. If I create and install the json for this policy, via s3api put-bucket-lifecycle-configuration then only on the master side, side A, do objects get deleted after 4 days. Any objects on side B never get deleted. What am I doing wrong? Thanks -Chris

2 4

Successfully using dm-cache
by Michael Lipp 12 Sep '24

12 Sep '24

Just in case anybody is interested: Using dm-cache works and boosts performance -- at least for my use case. The "challenge" was to get 100 (identical) Linux-VMs started on a three node hyperconverged cluster. The hardware is nothing special, each node has a Supermicro server board with a single CPU with 24 cores and 4 x 4 TB hard disks. And there's that extra 1 TB NVMe... I know that the general recommendation is to use the NVMe for WAL and metadata, but this didn't seem appropriate for my use case and I'm still not quite sure about failure scenarios with this configuration. So instead I made each drive a logical volume (managed by an OSD) and added 85 GiB NVMe to each LV as read-only cache. Each VM uses as system disk an RBD based on a snapshot from the master image. The idea was that with this configuration, all VMs should share most (actually almost all) of the data on their system disk and this data should be available from the cache. Well, it works. When booting the 100 VMs, almost all read operations are satisfied from the cache. So I get close to NVMe speed but have payed for conventional hard drives only (well, SSDs aren't that much more expensive nowadays, but the hardware is 4 years old). So, nothing sophisticated, but as I couldn't find anything about this kind of setup, it might be of interest nevertheless. - Michael

13 27

cephadm basic questions: image config, OS reimages
by Matthew Vernon 27 Aug '24

27 Aug '24

Hi, I've some experience with Ceph, but haven't used cephadm much before, and am trying to configure a pair of reef clusters with cephadm. A couple of newbie questions, if I may: * cephadm shell image I'm in an isolated environment, so pulling from a local repository. I bootstrapped OK with cephadm --image docker-registry.wikimedia.org/ceph bootstrap ... And that worked nicely, but if I want to run cephadm shell (to do any sort of admin), then I have to specify cephadm --image docker-registry.wikimedia.org/ceph shell (otherwise it just hangs failing to talk to quay.io). I found the docs, which refer to setting lots of other images, but not the one that cephadm uses: https://docs.ceph.com/en/reef/cephadm/install/#deployment-in-an-isolated-en… I found an old tracker in this area: https://tracker.ceph.com/issues/47274 ...but is there a good way to arrange for cephadm to use the already-downloaded image without having to remember to specify --image each time? * OS reimages We do OS upgrades by reimaging the server (which doesn't touch the storage disks); on an old-style deployment you could then use ceph-volume to re-start the OSDs and away you went; how does one do this in a cephadm cluster? [I presume involves telling cephadm to download a new image for podman to use and suchlike] Would the process be smoother if we arranged to leave /var/lib/ceph intact between reimages? Thanks, Matthew

3 4

All MDS's Crashed, Failed Assert
by m＠silvenga.com 14 Aug '24

14 Aug '24

I'm looking for guidance around how to recover after all MDS continue to crash with a failed assert during journal replay (no MON damage). Context: So I've been working through failed MDS for the past day, likely caused by a large snaptrim operation that caused the cluster to grind to a halt. After evicting all clients and restarting the MDS's (it appears the clients were overwhelming the MDS's). The MDS are failing to start with: debug -1> 2024-07-24T18:44:52.674+0000 7f7878c22700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.2.2/rpm/el8/BUILD/ceph-18.2.2/src/osdc/Journaler.cc: In function 'bool Journaler::try_read_entry(ceph::bufferlist&)' thread 7f7878c22700 time 2024-07-24T18:44:52.676027+0000 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.2.2/rpm/el8/BUILD/ceph-18.2.2/src/osdc/Journaler.cc: 1256: FAILED ceph_assert(start_ptr == read_pos) ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f788aa32e15] 2: /usr/lib64/ceph/libceph-common.so.2(+0x2a9fdb) [0x7f788aa32fdb] 3: (Journaler::try_read_entry(ceph::buffer::v15_2_0::list&)+0x132) [0x55555847ef32] 4: (MDLog::_replay_thread()+0xda) [0x555558436bea] 5: (MDLog::ReplayThread::entry()+0x11) [0x5555580e52d1] 6: /lib64/libpthread.so.0(+0x81ca) [0x7f78897d81ca] 7: clone() debug 0> 2024-07-24T18:44:52.674+0000 7f7878c22700 -1 *** Caught signal (Aborted) ** in thread 7f7878c22700 thread_name:md_log_replay ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable) 1: /lib64/libpthread.so.0(+0x12d20) [0x7f78897e2d20] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f788aa32e6f] 5: /usr/lib64/ceph/libceph-common.so.2(+0x2a9fdb) [0x7f788aa32fdb] 6: (Journaler::try_read_entry(ceph::buffer::v15_2_0::list&)+0x132) [0x55555847ef32] 7: (MDLog::_replay_thread()+0xda) [0x555558436bea] 8: (MDLog::ReplayThread::entry()+0x11) [0x5555580e52d1] 9: /lib64/libpthread.so.0(+0x81ca) [0x7f78897d81ca] 10: clone() Normally, there's three MDS are deployed, 1 active, one on hot standby. The cluster seems to believe that any restarted MDS is attempting to replay, but systemd reports an immediate crashed with a SIGABRT. ceph mds stat cephfs:1/1 {0=cephfs.sm1.esxjag=up:replay(laggy or crashed)} Redeploying the MDS's also continue to crash (suggesting a bad journal?)

3 2

Fwd: [community] [OpenInfra Event Update] The CFP For OpenInfra Days NA is now open!
by Alvaro Soto 14 Aug '24

14 Aug '24

Hello Community!!!! I want to share this with you in case anybody wants to send anything to the CFP \o/ Feel free to reach out in case of questions; I hope to see you there. Cheers! ---------- Forwarded message --------- From: Helena Spease <helena(a)openinfra.dev> Date: Tue, Jul 16, 2024 at 1:46 PM Subject: [community] [OpenInfra Event Update] The CFP For OpenInfra Days NA is now open! To: <community(a)lists.openinfra.dev> Hi everyone, The CFP for the OpenInfra Days North America is now open! Submit your talk before the deadline on August 12. https://docs.google.com/forms/d/e/1FAIpQLSeFQBL1qGNegDWDHCVTLtu_wLf06w7JJkc… Pre-Event Timeline: - August 12: CFP ends - August 22: CFP notification of acceptance - August 26: Program announced ———OPENINFRA DAYS NORTH AMERICA——— October 15 & 16, 2024 IU Indianapolis Campus Center in Indianapolis, Indiana - Learn more: https://ittraining.iu.edu/explore-topics/titles/oid-iu/index.html - Sponsor: https://openinfra.dev/events/sponsorship#OIDNA ———OPENINFRA SUMMIT ASIA——— September 3 & 4, 2024 Suwon, South Korea - Learn more: https://2024.openinfraasia.org/ - Sponsor: https://2024.openinfraasia.org/sponsors/ - Schedule: https://2024.openinfraasia.org/a/schedule This schedule is not complete and only reflects OpenInfra Summit Asia sessions. OCP Summit Asia, Ceph Days Asia and OpenInfra Forum sessions are still being added. Session times are subject to change. Cheers, Helena helena(a)openinfra.dev Community Programs Coordinator The OpenInfra Foundation _______________________________________________ Community mailing list -- community(a)lists.openinfra.dev To unsubscribe send an email to community-leave(a)lists.openinfra.dev -- Alvaro Soto *Note: My work hours may not be your work hours. Please do not feel the need to respond during a time that is not convenient for you.* ---------------------------------------------------------- Great people talk about ideas, ordinary people talk about things, small people talk... about other people.

1 1

Can you return orphaned objects to a bucket?
by motaharesdq＠gmail.com 08 Aug '24

08 Aug '24

Hello everyone, Is there any way to return orphaned objects to a bucket? A large bucket was accidentally corrupted and emptied, but its objects do exist in rados. bi list and bucket check returns empty list and bucket fix don't do anything as expected. How can we index associated rados objects back to the bucket?

3 3

Cephadm Offline Bootstrapping Issue
by Alex Hussein-Kershaw (HE/HIM) 05 Aug '24

05 Aug '24

Hi, I'm hitting an issue doing an offline install of Ceph 18.2.2 using cephadm. Long output below... any advice is appreciated. Looks like we don't managed to add admin labels (but also trying with --skip-admin results in a similar health warning). Subsequently trying to add an OSD fails quietly, I assume because cephadm is unhappy. Thanks, Alex $ sudo cephadm --image "ceph/ceph:v18.2.2" --docker bootstrap --mon-ip `hostname -I` --skip-pull --ssh-user qs-admin --ssh-private-key /home/qs-admin/.ssh/id_rsa --ssh-public-key /home/qs-admin/.ssh/id_rsa.pub --skip-dashboard Verifying ssh connectivity using standard pubkey authentication ... Adding key to qs-admin@localhost authorized_keys... key already in qs-admin@localhost authorized_keys... Verifying podman|docker is present... Verifying lvm2 is present... Verifying time synchronization is in place... Unit chronyd.service is enabled and running Repeating the final host check... docker (/usr/bin/docker) is present systemctl is present lvcreate is present Unit chronyd.service is enabled and running Host looks OK Cluster fsid: 65bee110-3ae6-11ef-a1de-005056013d88 Verifying IP 10.235.22.8 port 3300 ... Verifying IP 10.235.22.8 port 6789 ... Mon IP `10.235.22.8` is in CIDR network `10.235.16.0/20` Mon IP `10.235.22.8` is in CIDR network `10.235.16.0/20` Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network Ceph version: ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable) Extracting ceph user uid/gid from container image... Creating initial keys... Creating initial monmap... Creating mon... Waiting for mon to start... Waiting for mon... mon is available Assimilating anything we can from ceph.conf... Generating new minimal ceph.conf... Restarting the monitor... Setting public_network to 10.235.16.0/20 in mon config section Wrote config to /etc/ceph/ceph.conf Wrote keyring to /etc/ceph/ceph.client.admin.keyring Creating mgr... Verifying port 0.0.0.0:9283 ... Verifying port 0.0.0.0:8765 ... Verifying port 0.0.0.0:8443 ... Waiting for mgr to start... Waiting for mgr... mgr not available, waiting (1/15)... mgr not available, waiting (2/15)... mgr not available, waiting (3/15)... mgr not available, waiting (4/15)... mgr not available, waiting (5/15)... mgr is available Enabling cephadm module... Waiting for the mgr to restart... Waiting for mgr epoch 5... mgr epoch 5 is available Setting orchestrator backend to cephadm... Using provided ssh keys... Adding key to qs-admin@localhost authorized_keys... key already in qs-admin@localhost authorized_keys... Adding host starlight-1... Deploying mon service with default placement... Deploying mgr service with default placement... Deploying crash service with default placement... Deploying ceph-exporter service with default placement... Deploying prometheus service with default placement... Deploying grafana service with default placement... Deploying node-exporter service with default placement... Deploying alertmanager service with default placement... Enabling client.admin keyring and conf on hosts with "admin" label Non-zero exit code 5 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=ceph/ceph:v18.2.2 -e NODE_NAME=starlight-1 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/65bee110-3ae6-11ef-a1de-005056013d88:/var/log/ceph:z -v /tmp/ceph-tmpxbngx708:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp94g7iyn2:/etc/ceph/ceph.conf:z ceph/ceph:v18.2.2 orch client-keyring set client.admin label:_admin /usr/bin/ceph: stderr Error EIO: Module 'cephadm' has experienced an error and cannot handle commands: ContainerInspectInfo(image_id='3c937764e6f5de1131b469dc69f0db09f8bd55cf6c983482cde518596d3dd0e5', ceph_version='ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable)', repo_digests=['']) Unable to set up "admin" label; assuming older version of Ceph Saving cluster configuration to /var/lib/ceph/65bee110-3ae6-11ef-a1de-005056013d88/config directory Enabling autotune for osd_memory_target You can access the Ceph CLI as following in case of multi-cluster or non-default config: sudo /usr/sbin/cephadm shell --fsid 65bee110-3ae6-11ef-a1de-005056013d88 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring Or, if you are only running a single cluster on this host: sudo /usr/sbin/cephadm shell Please consider enabling telemetry to help improve Ceph: ceph telemetry on For more information see: https://docs.ceph.com/en/latest/mgr/telemetry/ Bootstrap complete. ]$ sudo docker exec ceph-1b19e642-3ae5-11ef-b4e4-005056013d88-mon-starlight-1 ceph -s cluster: id: 1b19e642-3ae5-11ef-b4e4-005056013d88 health: HEALTH_ERR Module 'cephadm' has failed: ContainerInspectInfo(image_id='3c937764e6f5de1131b469dc69f0db09f8bd55cf6c983482cde518596d3dd0e5', ceph_version='ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable)', repo_digests=['']) OSD count 0 < osd_pool_default_size 3 services: mon: 1 daemons, quorum starlight-1 (age 2m) mgr: starlight-1.yhqrry(active, since 107s) osd: 0 osds: 0 up, 0 in data: pools: 0 pools, 0 pgs objects: 0 objects, 0 B usage: 0 B used, 0 B / 0 B avail pgs:

4 5

Ceph MDS failing because of corrupted dentries in lost+found after update from 17.2.7 to 18.2.0
by Justin Lee 03 Aug '24

03 Aug '24

After we updated our ceph cluster from 17.2.7 to 18.2.0 the MDS kept being marked as damaged and stuck in up:standby with these errors in the log. debug -12> 2024-07-14T21:22:19.962+0000 7f020cf3a700 1 mds.0.cache.den(0x4 1000b3bcfea) loaded already corrupt dentry: [dentry #0x1/lost+found/1000b3bcfea [head,head] rep(a)0.0 NULL (dversion lock) pv=0 v=2 ino=(nil) state=0 0x558ca63b6500] debug -11> 2024-07-14T21:22:19.962+0000 7f020cf3a700 10 mds.0.cache.dir(0x4) go_bad_dentry 1000b3bcfea these log lines are repeated a bunch of times in our MDS logs, all on dentries that are within the lost+found directory. After reading this mailing list post <https://www.spinics.net/lists/ceph-users/msg77325.html>, we tried setting ceph config set mds mds_go_bad_corrupt_dentry false. This seemed to successfully circumvent the issue, however, after a few seconds our MDS crashes. Our 3 MDS are now stuck in a cycle of active -> crash -> standby -> back to active. Because of this our actual ceph fs is extremely laggy. We read here <https://docs.ceph.com/en/latest/releases/reef/#cephfs> that reef now makes it possible to delete the lost+found directory, which might solve our problem, but it is inaccessible, to cd, ls, rm, etc. Has anyone seen this type of issue or know how to solve it? Thanks!

2 4

ceph pg stuck active+remapped+backfilling
by Jorge Garcia 02 Aug '24

02 Aug '24

We were having an OSD reporting lots of errors, so I tried to remove it by doing: ceph orch osd rm 139 --zap It started moving all the data. Eventually, we got to the point that there's only 1 pg backfilling, but that seems to be stuck now. I think it may be because, in the process, another OSD (103) started reporting errors, too. The pool is erasure k:5 m:2, so it should still be OK. I don't see any progress happening on the backfill, and ceph -s has been reporting "69797/925734949 objects misplaced (0.008%)" for days now. How can I get it to finish the backfilling, or at least find out why it's not working? ceph version: quincy Thanks! Jorge

2 1

Difficulty importing bluestore OSDs from the old cluster (bad fsid) - OSD does not start
by Vinícius Barreto 02 Aug '24

02 Aug '24

Hello everybody! I am fascinated by Ceph, but now I live in moments of terror and despair. I'm using ceph 18.2.2 (reef) and at the moment We need to import 4 OSDs from an old cluster (which was removed by accident). In short, we suspect that the cause and solution for this case lies in the information in the OSD log below: (all OSD daemons repeat the same information) After importing I receive the following information in the "journalctl -u ceph-osd@2" log: --- Jul 17 20:09:55 pxm3 ceph-osd[647313]: in thread 702342c006c0 thread_name:ms_dispatch Jul 17 20:09:55 pxm3 ceph-osd[647313]: ceph version 18.2.2 (e9fe820e7fffd1b7cde143a9f77653b73fcec748) reef (stable) Jul 17 20:09:55 pxm3 ceph-osd[647313]: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x3c050) [0x70235785b050] Jul 17 20:09:55 pxm3 ceph-osd[647313]: 2: /lib/x86_64-linux-gnu/libc.so.6(+0x8ae2c) [0x7023578a9e2c] Jul 17 20:09:55 pxm3 ceph-osd[647313]: 3: gsignal() Jul 17 20:09:55 pxm3 ceph-osd[647313]: 4: abort() Jul 17 20:09:55 pxm3 ceph-osd[647313]: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x18a) [0x62edaeab47> Jul 17 20:09:55 pxm3 ceph-osd[647313]: 6: (OSD::handle_osd_map(MOSDMap*)+0x384a) [0x62edaec0aeca] Jul 17 20:09:55 pxm3 ceph-osd[647313]: 7: (OSD::ms_dispatch(Message*)+0x62) [0x62edaec0b332] Jul 17 20:09:55 pxm3 ceph-osd[647313]: 8: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0xc1) [0x62edaf5eea51] Jul 17 20:09:55 pxm3 ceph-osd[647313]: 9: (DispatchQueue::entry()+0x6cf) [0x62edaf5ed53f] Jul 17 20:09:55 pxm3 ceph-osd[647313]: 10: (DispatchQueue::DispatchThread::entry()+0xd) [0x62edaf40fd5d] Jul 17 20:09:55 pxm3 ceph-osd[647313]: 11: /lib/x86_64-linux-gnu/libc.so.6(+0x89134) [0x7023578a8134] Jul 17 20:09:55 pxm3 ceph-osd[647313]: 12: /lib/x86_64-linux-gnu/libc.so.6(+0x1097dc) [0x7023579287dc] Jul 17 20:09:55 pxm3 ceph-osd[647313]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Jul 17 20:09:55 pxm3 ceph-osd[647313]: -142> 2024-07-17T20:09:54.845-0300 702356d616c0 -1 osd.2 71 log_to_monitors true Jul 17 20:09:55 pxm3 ceph-osd[647313]: -2> 2024-07-17T20:09:55.362-0300 702342c006c0 -1 osd.2 71 ERROR: bad fsid? i have 5514a69a-46ba-4a44-bb56-8d3109c6c9e0 and inc has f4466e33-b57d-4d68-9909-346> Jul 17 20:09:55 pxm3 ceph-osd[647313]: -1> 2024-07-17T20:09:55.366-0300 702342c006c0 -1 ./src/osd/OSD.cc: In function 'void OSD::handle_osd_map(MOSDMap*)' thread 702342c006c0 time 2024-07-17T20:09:5> Jul 17 20:09:55 pxm3 ceph-osd[647313]: ./src/osd/OSD.cc: 8098: ceph_abort_msg("bad fsid") Jul 17 20:09:55 pxm3 ceph-osd[647313]: ceph version 18.2.2 (e9fe820e7fffd1b7cde143a9f77653b73fcec748) reef (stable) --- Above in the OSDs daemon log, we see the "bad fsid" error which indicates that the OSDs are trying to connect to a different Ceph cluster than the one they were initially configured for. Each Ceph cluster has a unique identifier, the FSID, and if the OSDs detect a different FSID, they cannot connect correctly. Note: I believe this is the root cause of the problem and the solution. Problem: OSD has fsid from the old cluster. This would be the solution, change the ceph_fsid (fsid) of the OSDs to the fsid of the new cluster, or recreate a new cluster using the fsid that is recorded in the OSD metadata, that is, recreate the cluster with the fsid of the old cluster? would it be possible?) I also opened a topic for this case on the proxmox forum and inserted a lot of data about this scenario there: https://forum.proxmox.com/threads/ceph-cluster-rebuild-import-bluestore-osd… ----- ~# cat /etc/ceph/ceph.conf [global] auth_client_required = cephx auth_cluster_required = cephx auth_service_required = cephx cluster_network = 192.168.0.2/24 fsid = f4466e33-b57d-4d68-9909-3468afd9e5c2 mon_allow_pool_delete = true mon_host = 192.168.0.2 192.168.0.3 192.168.0.1 ms_bind_ipv4 = true ms_bind_ipv6 = false osd_pool_default_min_size = 2 osd_pool_default_size = 3 public_network = 192.168.0.0/24 [client] # keyring = /etc/pve/priv/$cluster.$name.keyring keyring = /etc/pve/priv/ceph.client.admin.keyring #[client.crash] # keyring = /etc/pve/ceph/$cluster.$name.keyring [client.crash] key = AQAl95NmlvL0HRAAovpivsfHqqokmO0vqIR5Lg== [client.admin] key = AQAk95NmSjMdORAAiAHkTSSMquKkBAGpALjwQA== caps mds = "allow *" caps mgr = "allow *" caps mon = "allow *" caps osd = "allow *" [mon.pxm1] public_addr = 192.168.0.1 [mon.pxm2] public_addr = 192.168.0.2 [mon.pxm3] public_addr = 192.168.0.3 --- root@pxm3:~# ceph fsid f4466e33-b57d-4d68-9909-3468afd9e5c2 --- ~# ceph -s cluster: id: f4466e33-b57d-4d68-9909-3468afd9e5c2 health: HEALTH_WARN mon pxm1 is low on available space 4 osds down 2 hosts (4 osds) down 1 root (4 osds) down 170 daemons have recently crashed services: mon: 3 daemons, quorum pxm2,pxm3,pxm1 (age 3h) mgr: pxm2(active, since 3h), standbys: pxm1, pxm3 osd: 4 osds: 0 up, 4 in (since 2h) data: pools: 0 pools, 0 pgs objects: 0 objects, 0 B usage: 0 B used, 0 B / 0 B avail pgs: --- ~# cephadm ls [ { "style": "legacy", "name": "osd.0", "fsid": "5514a69a-46ba-4a44-bb56-8d3109c6c9e0", "systemd_unit": "ceph-osd@0", "enabled": true, "state": "error", "host_version": "18.2.2" }, { "style": "legacy", "name": "osd.1", "fsid": "5514a69a-46ba-4a44-bb56-8d3109c6c9e0", "systemd_unit": "ceph-osd@1", "enabled": true, "state": "error", "host_version": "18.2.2" }, { "style": "legacy", "name": "mon.pxm2", "fsid": "f4466e33-b57d-4d68-9909-3468afd9e5c2", "systemd_unit": "ceph-mon@pxm2", "enabled": true, "state": "running", "host_version": "18.2.2" }, { "style": "legacy", "name": "mgr.pxm2", "fsid": "f4466e33-b57d-4d68-9909-3468afd9e5c2", "systemd_unit": "ceph-mgr@pxm2", "enabled": true, "state": "running", "host_version": "18.2.2" } ] root@pxm3:~# cephadm ls [ { "style": "legacy", "name": "osd.3", "fsid": "5514a69a-46ba-4a44-bb56-8d3109c6c9e0", "systemd_unit": "ceph-osd@3", "enabled": true, "state": "error", "host_version": "18.2.2" }, { "style": "legacy", "name": "osd.2", "fsid": "5514a69a-46ba-4a44-bb56-8d3109c6c9e0", "systemd_unit": "ceph-osd@2", "enabled": true, "state": "error", "host_version": "18.2.2" }, { "style": "legacy", "name": "mon.pxm3", "fsid": "f4466e33-b57d-4d68-9909-3468afd9e5c2", "systemd_unit": "ceph-mon@pxm3", "enabled": true, "state": "running", "host_version": "18.2.2" }, { "style": "legacy", "name": "mgr.pxm3", "fsid": "f4466e33-b57d-4d68-9909-3468afd9e5c2", "systemd_unit": "ceph-mgr@pxm3", "enabled": true, "state": "running", "host_version": "18.2.2" } ] --- To import the OSDs, we first increased the epoch of the new cluster, repeating the commands below until the cluster's epoch count number was greater than the OSDs' epoch number: Bash: ceph osd set noin ceph osd set noout ceph osd set noup ceph osd set nodown ceph osd set norebalance ceph osd set nobackfill ceph osd unset noin ceph osd unset noout ceph osd unset noup ceph osd unset nodown ceph osd unset norebalance ceph osd unset nobackfill --- OSD volumes: Code: pxm2: -> OSD.0: /dev/ceph-1740d41a-2ae7-4c4d-820f-ec3702e3ba90/osd-block-39f9b32f-c6e7-4b3f-b7f0-9b11a5832aaa -> OSD.1: /dev/ceph-ad425d70-4aa3-419a-997f-f3a4082c9904/osd-block-bb4df480-2b9b-4604-a44d-6151d5c0cb33 pxm3: -> OSD.2: /dev/ceph-94682b88-d09c-4eab-9170-c6d31eac79e6/osd-block-3f6756d6-e64b-4c60-9ac2-305c0e71cc51 -> OSD.3: /dev/ceph-d5ffd027-8289-4a1c-9378-6687d9f950ad/osd-block-eece9fc9-44d6-460b-aced-572c79a98be8 --- ceph-bluestore-tool show-label: -> osb.0: Bash: root@pxm2:~# ceph-bluestore-tool show-label --path /var/lib/ceph/osd/ceph-0 inferring bluefs devices from bluestore path { "/var/lib/ceph/osd/ceph-0/block": { "osd_uuid": "39f9b32f-c6e7-4b3f-b7f0-9b11a5832aaa", "size": 2000397795328, "btime": "2024-07-07T21:32:30.861509-0300", "description": "main", "bfm_blocks": "488378368", "bfm_blocks_per_key": "128", "bfm_bytes_per_block": "4096", "bfm_size": "2000397795328", "bluefs": "1", "ceph_fsid": "5514a69a-46ba-4a44-bb56-8d3109c6c9e0", "ceph_version_when_created": "ceph version 18.2.2 (e9fe820e7fffd1b7cde143a9f77653b73fcec748) reef (stable)", "created_at": "2024-07-08T00:32:32.459102Z", "kv_backend": "rocksdb", "magic": "ceph osd volume v026", "mkfs_done": "yes", "osd_key": "AQCdM4tmHR5tLRAA5AikqvQyMqOoH5MnL8Qdtg==", "ready": "ready", "require_osd_release": "18", "whoami": "0" } } -> osd.1: Bash: root@pxm2:~# ceph-bluestore-tool show-label --path /var/lib/ceph/osd/ceph-1 inferring bluefs devices from bluestore path { "/var/lib/ceph/osd/ceph-1/block": { "osd_uuid": "bb4df480-2b9b-4604-a44d-6151d5c0cb33", "size": 2000397795328, "btime": "2024-07-07T21:32:43.729638-0300", "description": "main", "bfm_blocks": "488378368", "bfm_blocks_per_key": "128", "bfm_bytes_per_block": "4096", "bfm_size": "2000397795328", "bluefs": "1", "ceph_fsid": "5514a69a-46ba-4a44-bb56-8d3109c6c9e0", "ceph_version_when_created": "ceph version 18.2.2 (e9fe820e7fffd1b7cde143a9f77653b73fcec748) reef (stable)", "created_at": "2024-07-08T00:32:45.577456Z", "kv_backend": "rocksdb", "magic": "ceph osd volume v026", "mkfs_done": "yes", "osd_key": "AQCqM4tmTy87JxAAJVK1NokBDjdKSe+Z8OjwMA==", "ready": "ready", "require_osd_release": "18", "whoami": "1" } } -> osd.2 Bash: root@pxm3:~# ceph-bluestore-tool show-label --path /var/lib/ceph/osd/ceph-2 inferring bluefs devices from bluestore path { "/var/lib/ceph/osd/ceph-2/block": { "osd_uuid": "3f6756d6-e64b-4c60-9ac2-305c0e71cc51", "size": 2000397795328, "btime": "2024-07-07T21:33:07.812888-0300", "description": "main", "bfm_blocks": "488378368", "bfm_blocks_per_key": "128", "bfm_bytes_per_block": "4096", "bfm_size": "2000397795328", "bluefs": "1", "ceph_fsid": "5514a69a-46ba-4a44-bb56-8d3109c6c9e0", "ceph_version_when_created": "ceph version 18.2.2 (e9fe820e7fffd1b7cde143a9f77653b73fcec748) reef (stable)", "created_at": "2024-07-08T00:33:09.404317Z", "kv_backend": "rocksdb", "magic": "ceph osd volume v026", "mkfs_done": "yes", "osd_key": "AQDCM4tmu++vLRAAuJOXjTuEHR9VsKz7ShVEPg==", "ready": "ready", "require_osd_release": "18", "whoami": "2" } } -> osd.3 Bash: root@pxm3:~# ceph-bluestore-tool show-label --path /var/lib/ceph/osd/ceph-3 inferring bluefs devices from bluestore path { "/var/lib/ceph/osd/ceph-3/block": { "osd_uuid": "eece9fc9-44d6-460b-aced-572c79a98be8", "size": 2000397795328, "btime": "2024-07-07T21:33:25.725294-0300", "description": "main", "bfm_blocks": "488378368", "bfm_blocks_per_key": "128", "bfm_bytes_per_block": "4096", "bfm_size": "2000397795328", "bluefs": "1", "ceph_fsid": "5514a69a-46ba-4a44-bb56-8d3109c6c9e0", "ceph_version_when_created": "ceph version 18.2.2 (e9fe820e7fffd1b7cde143a9f77653b73fcec748) reef (stable)", "created_at": "2024-07-08T00:33:27.323085Z", "kv_backend": "rocksdb", "magic": "ceph osd volume v026", "mkfs_done": "yes", "osd_key": "AQDUM4tmagOEKBAAeAfZXcyU1naRkqIE5iVOfw==", "ready": "ready", "require_osd_release": "18", "whoami": "3" } } --- root@pxm2:~# cat /var/lib/ceph/osd/ceph-0/ceph_fsid 5514a69a-46ba-4a44-bb56-8d3109c6c9e0 root@pxm2:~# cat /var/lib/ceph/osd/ceph-1/ceph_fsid 5514a69a-46ba-4a44-bb56-8d3109c6c9e0 root@pxm3:~# cat /var/lib/ceph/osd/ceph-2/ceph_fsid 5514a69a-46ba-4a44-bb56-8d3109c6c9e0 root@pxm3:~# cat /var/lib/ceph/osd/ceph-3/ceph_fsid 5514a69a-46ba-4a44-bb56-8d3109c6c9e0 --- ~# ceph daemon osd.0 status no valid command found; 10 closest matches: 0 1 2 abort assert bluefs debug_inject_read_zeros bluefs files list bluefs stats bluestore allocator dump block bluestore allocator fragmentation block admin_socket: invalid command --- ~# ceph osd info osd.0 osd.0 down in weight 1 up_from 0 up_thru 0 down_at 0 last_clean_interval [0,0) exists 39f9b32f-c6e7-4b3f-b7f0-9b11a5832aaa --- ~# ceph osd status ID HOST USED AVAIL WR OPS WR DATA RD OPS RD DATA STATE 0 0 0 0 0 0 0 exists 1 0 0 0 0 0 0 exists 2 0 0 0 0 0 0 exists 3 0 0 0 0 0 0 exists root@pxm3:~# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 7.27759 root default -2 3.63879 host pxm2 0 1.81940 osd.0 down 1.00000 1.00000 1 1.81940 osd.1 down 1.00000 1.00000 -5 3.63879 host pxm3 3 1.81940 osd.3 down 1.00000 1.00000 2 ssd 1.81940 osd.2 down 1.00000 1.00000 --- ~# ceph osd dump epoch 164 fsid f4466e33-b57d-4d68-9909-3468afd9e5c2 created 2024-07-14T13:04:53.354056-0300 modified 2024-07-17T20:24:17.550123-0300 flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit crush_version 63 full_ratio 0.95 backfillfull_ratio 0.9 nearfull_ratio 0.85 require_min_compat_client luminous min_compat_client jewel require_osd_release reef stretch_mode_enabled false max_osd 4 osd.0 down in weight 1 up_from 0 up_thru 0 down_at 0 last_clean_interval [0,0) exists 39f9b32f-c6e7-4b3f-b7f0-9b11a5832aaa osd.1 down in weight 1 up_from 0 up_thru 0 down_at 0 last_clean_interval [0,0) exists bb4df480-2b9b-4604-a44d-6151d5c0cb33 osd.2 down in weight 1 up_from 0 up_thru 0 down_at 0 last_clean_interval [0,0) exists 3f6756d6-e64b-4c60-9ac2-305c0e71cc51 osd.3 down in weight 1 up_from 0 up_thru 0 down_at 0 last_clean_interval [0,0) exists eece9fc9-44d6-460b-aced-572c79a98be8 blocklist 192.168.0.1:0/1377959162 expires 2024-07-18T17:16:21.635340-0300 blocklist 192.168.0.1:6800/49511 expires 2024-07-18T17:16:21.635340-0300 blocklist 192.168.0.1:0/3150227222 expires 2024-07-18T17:16:21.635340-0300 blocklist 192.168.0.2:0/1759809784 expires 2024-07-18T00:56:24.333969-0300 blocklist 192.168.0.1:0/3747372149 expires 2024-07-18T17:16:21.635340-0300 blocklist 192.168.0.1:6801/49511 expires 2024-07-18T17:16:21.635340-0300 blocklist 192.168.0.2:0/1472036822 expires 2024-07-18T00:56:24.333969-0300 blocklist 192.168.0.2:0/571633622 expires 2024-07-18T00:56:24.333969-0300 blocklist 192.168.0.2:6801/10543 expires 2024-07-18T00:56:24.333969-0300 blocklist 192.168.0.2:6800/10543 expires 2024-07-18T00:56:24.333969-0300 --- ~# ceph osd metadata [ { "id": 0 }, { "id": 1 }, { "id": 2 }, { "id": 3 } ] --- ~# cat /var/lib/ceph/osd/ceph-0/keyring [osd.0] key = AQCdM4tmHR5tLRAA5AikqvQyMqOoH5MnL8Qdtg== root@pxm2:~# cat /var/lib/ceph/osd/ceph-1/keyring [osd.1] key = AQCqM4tmTy87JxAAJVK1NokBDjdKSe+Z8OjwMA== root@pxm3:~# cat /var/lib/ceph/osd/ceph-2/keyring [osd.2] key = AQDCM4tmu++vLRAAuJOXjTuEHR9VsKz7ShVEPg== root@pxm3:~# cat /var/lib/ceph/osd/ceph-3/keyring [osd.3] key = AQDUM4tmagOEKBAAeAfZXcyU1naRkqIE5iVOfw== --- ~# ceph auth list osd.0 key: AQCKQJhmMYWeEhAAosrE8Ff+1kZbKcroi22TvQ== caps: [mon] allow profile osd caps: [osd] allow * osd.1 key: AQCTQJhm2uLvAxAAC3uIcRk9d0sxLJgIxcivtw== caps: [mon] allow profile osd caps: [osd] allow * osd.2 key: AQDCM4tmu++vLRAAuJOXjTuEHR9VsKz7ShVEPg== caps: [mon] allow profile osd caps: [osd] allow * osd.3 key: AQBYMZhmmlaFDRAAtoy4//XweaI94OvrPV1aiQ== caps: [mon] allow profile osd caps: [osd] allow * client.admin key: AQAk95NmSjMdORAAiAHkTSSMquKkBAGpALjwQA== caps: [mds] allow * caps: [mgr] allow * caps: [mon] allow * caps: [osd] allow * client.bootstrap-mds key: AQAl95NmWuofFRAAhV7/f1PykW/KlcB8Rede9w== caps: [mon] allow profile bootstrap-mds client.bootstrap-mgr key: AQAl95NmmPQfFRAAG2cQNiWQgdx0Rvr7ZFmuhg== caps: [mon] allow profile bootstrap-mgr client.bootstrap-osd key: AQAl95Nmr/0fFRAA4TahG5PCbZIsltFgkRNEgA== caps: [mon] allow profile bootstrap-osd client.bootstrap-rbd key: AQAl95Nm0gYgFRAAWBvqfphKk62InqY9x0ijHg== caps: [mon] allow profile bootstrap-rbd client.bootstrap-rbd-mirror key: AQAl95Nmxg8gFRAAomFkp299Ca04NwGpfbSZRg== caps: [mon] allow profile bootstrap-rbd-mirror client.bootstrap-rgw key: AQAl95Nm3RwgFRAA/QoIemXPFv5Gs1/PWfJuYw== caps: [mon] allow profile bootstrap-rgw client.crash key: AQAl95NmlvL0HRAAovpivsfHqqokmO0vqIR5Lg== caps: [mgr] profile crash caps: [mon] profile crash mgr.pxm1 key: AQDm+JNmtJ3CCRAAl/wZaT6z12LCfAghvm6s4w== caps: [mds] allow * caps: [mon] allow profile mgr caps: [osd] allow * mgr.pxm2 key: AQAm95Nm6+dmABAA6405D8ROtNpJf6iVhMegQA== caps: [mds] allow * caps: [mon] allow profile mgr caps: [osd] allow * mgr.pxm3 key: AQDp95Nm3ZbLDxAAWc1zLaVOhE0wMxLJCL5IGg== caps: [mds] allow * caps: [mon] allow profile mgr caps: [osd] allow *

2 1

rbd-mirror keeps crashing
by armsby＠armsby.dk 02 Aug '24

02 Aug '24

Hi everyone, I've been running rbd-mirror between my old Ceph system (16.2.10) and my new system (18.2.2). I'm using journaling mode on a pool that contains 7,500 images. Everything was running perfectly until it processed about 5,608 images. Now, it keeps crashing with the following message: 2024-07-19T05:49:32.425+0000 7f582b3fd6c0 0 set uid:gid to 167:167 (ceph:ceph) 2024-07-19T05:49:32.425+0000 7f582b3fd6c0 0 ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable), process rbd-mirror, pid 7 2024-07-19T05:49:32.429+0000 7f582b3fd6c0 1 mgrc service_daemon_register rbd-mirror.3606956688 metadata {arch=x86_64, ceph_release=pacific, ceph_version=ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable), ceph_version_short=16.2.10, container_hostname=mon-001, container_image=quay.io/ceph/ceph@sha256:2b68483bcd050472a18e73389c0e1f3f70d34bb7abf733f692e88c935ea0a6bd, cpu=Intel(R) Xeon(R) Gold 6134 CPU @ 3.20GHz, distro=centos, distro_description=CentOS Stream 8, distro_version=8, hostname=mon-001, id=mon-001.lcqrti, instance_id=3606956688, kernel_description=#1 SMP Mon Jul 18 17:42:52 UTC 2022, kernel_version=4.18.0-408.el8.x86_64, mem_swap_kb=4194300, mem_total_kb=131393360, os=Linux} 2024-07-19T05:50:28.305+0000 7f5812582700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.10/rpm/el8/BUILD/ceph-16.2.10/src/common/Thread.cc: In function 'void Thread::create(const char*, size_t)' thread 7f5812582700 time 2024-07-19T05:50:28.303536+0000 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.10/rpm/el8/BUILD/ceph-16.2.10/src/common/Thread.cc: 165: FAILED ceph_assert(ret == 0) ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f58218b6de8] 2: /usr/lib64/ceph/libceph-common.so.2(+0x277002) [0x7f58218b7002] 3: /usr/lib64/ceph/libceph-common.so.2(+0x362fd7) [0x7f58219a2fd7] 4: (CommonSafeTimer<std::mutex>::init()+0x1fe) [0x7f58219a963e] 5: (journal::Journaler::Threads::Threads(ceph::common::CephContext*)+0x2fc) [0x55c9b33c6ddc] 6: (journal::Journaler::Journaler(librados::v14_2_0::IoCtx&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, journal::Settings const&, journal::CacheManagerHandler*)+0x50) [0x55c9b33c6f10] 7: (librbd::Journal<librbd::ImageCtx>::get_tag_owner(librados::v14_2_0::IoCtx&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, librbd::asio::ContextWQ*, Context*)+0x19f) [0x55c9b2fa65af] 8: (librbd::mirror::GetInfoRequest<librbd::ImageCtx>::get_journal_tag_owner()+0x210) [0x55c9b31869f0] 9: (librbd::mirror::GetInfoRequest<librbd::ImageCtx>::handle_get_mirror_image(int)+0x8c8) [0x55c9b3189d78] 10: /lib64/librados.so.2(+0xa8546) [0x7f582aedb546] 11: /lib64/librados.so.2(+0xc17e5) [0x7f582aef47e5] 12: /lib64/librados.so.2(+0xc3742) [0x7f582aef6742] 13: /lib64/librados.so.2(+0xc914a) [0x7f582aefc14a] 14: /lib64/libstdc++.so.6(+0xc2ba3) [0x7f581fb03ba3] 15: /lib64/libpthread.so.0(+0x81ca) [0x7f5820cec1ca] 16: clone() Has anyone encountered a similar problem or have any insight into what might be causing this crash? Thanks in advance for your help.

2 1

Help with osd spec needed
by Kristaps Cudars 02 Aug '24

02 Aug '24

3 nodes each: 3 hdd – 21G 1 ssd – 80G Create osd containing block_data with block_db size 15G located on ssd- This par works Create block_data osd on remaining space 35G in ssd- This part is not working ceph orch apply osd -i /path/to/osd_spec.yml service_type: osd service_id: osd_spec_hdd placement: host_pattern: '*' spec: block_db_size: 15G db_slots: 3 data_devices: rotational: 1 db_devices: rotational: 0 --- service_type: osd service_id: odd_spec_ssd placement: host_pattern: '*' spec: data_devices: rotational: 0

2 1

ceph 18.2.4 on el8?
by Christopher Durham 31 Jul '24

31 Jul '24

Hi, I see that 18.2.4 is out, in rpm for el9 at: http://download.ceph.com/rpm-18.2.4/ Are there any plans for an '8' version? One of my clusters is not yet ready to update to Rocky 9. We will update to 9 moving forward but this itme around it would be good to have a Rocky 8 version. Thanks! cbd

2 2

Public Swift bucket with Openstack Keystone integration - not working in quincy/reef
by Bartosz Bezak 31 Jul '24

31 Jul '24

Hi, Similar case as with previously fixed https://tracker.ceph.com/issues/48382 - https://github.com/ceph/ceph/pull/47308. Confirmed on Cephadm deployed Ceph 18.2.2/17.2.7 with Openstack Antelope/Yoga. I’m getting "404 NoSuchBucket" error with public buckets. Enabled with Swift/Keystone integration - everything else works fine. With rgw_swift_account_in_url = true and proper endpoints: "https://rgw.test/swift/v1/AUTH_%(project_id)s" ticking public access in horizon properly sets ACL on the bucket according to swift client: swift -v stat test-bucket URL: https://rgw.test/swift/v1/AUTH_daksjhdkajdshda/testbucket Auth Token: Account: AUTH_daksjhdkajdshda Container: testbucket Objects: 1 Bytes: 1021036 Read ACL: .r:*,.rlistings Write ACL: Sync To: Sync Key: X-Timestamp: 1710947159.41219 X-Container-Bytes-Used-Actual: 1024000 X-Storage-Policy: default-placement X-Storage-Class: STANDARD Last-Modified: Thu, 21 Mar 2024 10:30:05 GMT X-Trans-Id: tx00000092ac12312312312-1231231231-1701e5-default X-Openstack-Request-Id: tx00000092ac12312312312-1231231231-1701e5-default Accept-Ranges: bytes Content-Type: text/plain; charset=utf-8 however still getting 404 NoSuchBucket error Could someone using the latest version of Ceph with Swift/Keystone integration please test public buckets? Thank you. Best regards, Bartosz Bezak

3 3

ceph-mgr memory problems 16.2.15
by Тимур Мухаметов 31 Jul '24

31 Jul '24

I have faced a similar problem that is described here https://tracker.ceph.com/issues/59580#change-270387 upon update to 16.2.15. I have a question for developers. Are you planning to fix this problem? Thank you in advance.

1 0

Ceph on Ubuntu 24.04 - Arm64
by Daniel Brown 31 Jul '24

31 Jul '24

Is there any workable solution for running Ceph on Ubuntu 24.04 on Arm64? I’ve tried about every package install method I could think of, short of compiling it myself. I’m aiming for a “cephadm” install using docker containers. It be super nice if there was a working “noble” install of reef 18.2.4 for Arm64 made available on download.ceph.com Ubuntu suggests using their “noble-proposed” in this thread: https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/2063456?comments=all But it ends up trying to pull down an amd64 container for my arm64 machine: sudo cephadm shell ceph -s Inferring fsid b7105258-638a-11ee-b38c-bf980c3a0ec9 Inferring config /etc/ceph/ceph.conf This is a development version of cephadm. For information regarding the latest stable release: https://docs.ceph.com/docs/reef/cephadm/install Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint stat --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:main -e NODE_NAME=hc-36c0ddecd544cb8d quay.ceph.io/ceph-ci/ceph:main -c %u %g /var/lib/ceph stat: stderr Unable to find image 'quay.ceph.io/ceph-ci/ceph:main' locally stat: stderr main: Pulling from ceph-ci/ceph stat: stderr 43735fbce921: Pulling fs layer stat: stderr 7679c4b05544: Pulling fs layer stat: stderr 43735fbce921: Verifying Checksum stat: stderr 43735fbce921: Download complete stat: stderr 43735fbce921: Pull complete stat: stderr 7679c4b05544: Verifying Checksum stat: stderr 7679c4b05544: Download complete stat: stderr 7679c4b05544: Pull complete stat: stderr Digest: sha256:e7e652f9de2825d155a993aecfce4872bbb0c497a4342127081b529aec0528f5 stat: stderr Status: Downloaded newer image for quay.ceph.io/ceph-ci/ceph:main stat: stderr WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested stat: stderr [FATAL tini (7)] exec stat failed: Exec format error ERROR: Failed to extract uid/gid for path /var/lib/ceph: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint stat --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:main -e NODE_NAME=hc-36c0ddecd544cb8d quay.ceph.io/ceph-ci/ceph:main -c %u %g /var/lib/ceph

3 2

[RGW] Resharding in multi-zonegroup env cause data loss
by Huy Nguyen 31 Jul '24

31 Jul '24

Hi, My lab setup two Ceph clusters as 2 zonegroups in a realm. Ceph version 16.2.13 As I know, if a bucket is created on secondary zonegroup, the master zonegroup only know about its metadata, not data. So after resharding command is run on master (to avoid inconsistent metadata), the bucket is synced back to secondary which mean it will have no data, just like in master. Is this a bug or I'm doing it wrong? Is there any other way to reshard a bucket in non-master zonegroup? Thanks

2 1

Please guide us in identifying the cause of the data miss in EC pool
by wu_chulin＠qq.com 31 Jul '24

31 Jul '24

Dear Ceph team:     On July 13th at 4:55 AM, our Ceph cluster experienced a significant power outage in the data center, causing a large number of OSDs to power off and restart (total: 1172, down: 821). Approximately two hours later, all OSDs successfully started, and the cluster resumed its services. However, around 6 PM, the business department reported that some files, which had been successfully written (via the RGW service), were failing to download, and the number of such files was quite significant. Consequently, we began a series of investigations: 1. The incident occurred at 04:55. At 05:01, we executed noout, nobackfill, and norecover. At 06:22, we executed `ceph osd pause`. By 07:23, all OSDs were UP&IN, and subsequently, we executed `ceph osd unpause`. 2. We randomly selected a problematic file and attempted to download it via the S3 API. The RGW returned "No such key". 3. The RGW logs showed op status=-2, http status=200. We also checked the upload logs, which indicated 2024-07-13 04:19:20.052, op status=0, http_status=200. 4. We set debug_rgw=20 and attempted to download the file again. It was found that a 4M chunk(this file is 64M) failed to get. 5. Using rados get for this chunk returned: "No such file or directory". 6. Setting debug_osd=20, we observed get_object_context: obc NOT found in cache. 7. Setting debug_bluestore=20, we saw get_onode oid xxx, key xxx != '0xfffffffffffffffeffffffffffffffff'o'. 8. We stopped the primary OSD and tried to get the file again, but the result was the same. The object’s corresponding PG state was active+recovery_wait+degraded. 9. Using ceph-objectstore-tool --op list && --op log, we could not find the object information. The ceph-kvstore-tool rocksdb command also did not reveal anything new. 10. If an OSD had lost data, the PG state should have been unfound or inconsistency. 11. We started reanalyzing the startup logs of the OSDs related to the PG. The pool was set to erasure-code 6-3 with 9 OSDs. Six of these OSDs had restarted, and after peering, the PG state became ACTIVE. 12. We divided the lost files, and the upload time was before the failure occurred. The earliest upload time was around 1 am, and the successful upload records could be found in the RGW log 13. We have submitted an issue on the Ceph issue tracker: https://tracker.ceph.com/issues/66942, it includes the original logs needed for troubleshooting. However, four days have passed without any response. In desperation, we are sending this email, hoping that someone from the Ceph team can guide us as soon as possible. We are currently in a difficult situation and hope you can provide guidance. Thank you. Best regards. wu_chulin(a)qq.com wu_chulin(a)qq.com

1 0

Re: 18.2.4 regression: 'diskprediction_local' has failed: No module named 'sklearn'
by thymus_03fumbler＠icloud.com 30 Jul '24

30 Jul '24

Same issue here

1 0

RBD Stuck Watcher
by Reid Guyett 30 Jul '24

30 Jul '24

Hi, I have a small script in a Docker container we use for a type of CRUD test to monitor availability. The script uses Python librbd/librados and is launched by Telegraf input.exec. It does the following: 1. Creates an rbd image 2. Writes a small amount of data to the rbd 3. Reads the data from the rbd 4. Deletes the rbd 5. Closes connections It works great for 99% of the time but there is a small chance that something happens and the script takes too long (1 min) to complete and it is killed. I don't have logging to know which step it happens at yet but will be adding some. Regardless when the script is killed, sometimes the watcher on the rbd isn't going away. I use the same RBD name for each test and try to clean up the rbd if it exists prior to starting the next test but when the watcher is stuck, it can't. The only way to cleanup the watcher is to restart the primary osd for the rbd_header. Blocklist and restarting the container free the watcher. When I look at the status of the image I can see the watcher. # rbd -p pool status crud-image Watchers: watcher=<ipaddr>:0/3587274006 client.1053762394 cookie=140375838755648 Lookup up primary OSD # rbd -p pool info crud-image | grep id id: cf235ae95099cb # ceph osd map pool rbd_header.cf235ae95099cb osdmap e332984 pool 'pool' (1) object 'rbd_header.cf235ae95099cb' -> pg 1.a76f353e (1.53e) -> up ([7,66,176], p7) acting ([7,66,176], p7) Checking watchers on primary OSD does NOT list rbd_header.cf235ae95099cb # ceph tell osd.7 dump_watchers [ { "namespace": "", "object": "rbd_header.70fa4f9b5c2cf8", "entity_name": { "type": "client", "num": 998139266 }, "cookie": 140354859197312, "timeout": 30, "entity_addr_t": { "type": "v1", "addr": "<ipaddr>:0", "nonce": 2665188958 } } ] Is this a bug somewhere? I expect that if my script is killed it's watcher should die out within a minute. New runs of the script would result in new watcher/client/cookie ids. Thanks! Reid

2 2

Re: A problem with the CEPH PG state getting stuck
by Eugen Block 30 Jul '24

30 Jul '24

Hi, your cli output is barely readable, although it probably is not that relevant here. Apparently, it's an EC pool you're referring to? A pg repair tries to repair inconsistent objects, see [1] for more details. I don't really know how to explain "repeer", I'm also not a dev, so maybe someone from the ceph team can explain it better. But from how I understand it, a temporary new mapping for the primary OSD is created and that would trigger something like a refresh, I guess: // map to just primary; it will map back to what it wants pending_inc.new_pg_temp[pgid] = { primary } So this doesn't really affect your PG content wise, I think, it's just a refresh. Why your PGs get stuck (is it always the same? Is a specific OSD involved in all of the cases?) is difficut to answer without knowing so little about your cluster. Is the overall cluster health status okay? Are your OSDs (slow or fast drives?) highly utilized? [1] https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-pg/#m… Zitat von 苏察哈尔灿 <2644294460(a)qq.com>: > My ceph cluster sometimes gets stuck in the active+clean+snaptrim > state when doing regular snapshot deletion, and the corresponding pg > does not change for a long time. As follows: > > > 27.7c     14350        >  0          0      >   0  38073876582            > 0           0   2864  >     3000        > active+clean+snaptrim     9h  43777'677286  > 43777:1018248    [52,55,20,36,29,91,9,63,14,2]p52  >   [52,55,20,36,29,91,9,63,14,2]p52  > 2024-07-21T13:57:12.602903+0000  > 2024-07-20T11:02:06.953985+0000          >          52  queued for scrub  >                   >                   >           > 27.c1     14055        >  0          0      >   0  37408096205            > 0           0   2754  >     3000        > active+clean+snaptrim     9h  43777'676887  >  43777:952875    >  [0,21,39,62,26,58,41,86,66,2]p0    >  [0,21,39,62,26,58,41,86,66,2]p0  > 2024-07-22T02:26:43.470883+0000  > 2024-07-20T05:09:23.763918+0000          >          44  periodic scrub scheduled > @ 2024-07-23T07:46:50.084316+0000      > 27.19b    14711         0  >         0        0  > 38926389125            0    >        0   2765      > 3000        active+clean+snaptrim    >  9h  43777'698478   43777:849574    >   [5,7,29,14,61,10,22,0,19,37]p5      > [5,7,29,14,61,10,22,0,19,37]p5  > 2024-07-22T03:28:39.036189+0000  > 2024-07-17T14:00:43.082766+0000          >          40  periodic scrub scheduled > @ 2024-07-23T12:30:57.488823+0000      > 27.1a3    14323         0  >         0        0  > 37918899397            0    >        0   2930      > 3000        active+clean+snaptrim    >  9h  43777'675713   43777:943324  >  [52,4,10,46,20,69,49,39,44,30]p52  >  [52,4,10,46,20,69,49,39,44,30]p52  > 2024-07-21T11:15:16.442506+0000  > 2024-07-20T04:10:45.391550+0000          >          50  queued for scrub  >                 > > > No change for 9 hours. Then, after I typed the command "ceph pg > repeer 27.7c", the corresponding pg state was restored to normal. I > don't know what this "repeer" command is for, will it have any > effect on pg? What's the difference between "repeer" and "ceph pg > repair"? Then, why does the pg get stuck so often? Thanks for your > help! > > > My ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) > quincy (stable) > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Is it possible to replace the md5 etags with the blake3 etags?
by jmusial 30 Jul '24

30 Jul '24

Answer very much appreciated. Thank you, Jan Musial

1 0

[RGW] Setup 2 zones within a cluster does not sync data
by Huy Nguyen 30 Jul '24

30 Jul '24

Hi, Currently I'm testing replication ability between 2 zones in the same cluster. But only metadata is synced, not the data. I checked endpoints, system_key,... all good. If anyone have any idea, please guide me to resolve this situation. Thanks The radosgw shows this log on both side, primary and secondary: rgw rados thread: ERROR: failed to run sync ... RGW-SYNC:data:sync:shard[54]: ERROR: failed to read remote data log info: ret=-5 RGW-SYNC:data:sync:shard[90]: ERROR: failed to read remote data log info: ret=-5 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -5 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -5 Here is my configure (it might be long): zonegroup hn which contains hn-1 and hn-2, each zone have its own endpoint and radosgw instance on two different node: { "id": "36c29ef1-332b-4d6e-a341-ecc9cc2ef281", "epoch": 4, "predecessor_uuid": "e1981515-7355-4029-b26c-6be3de943be0", "sync_status": [], "period_map": { "id": "36c29ef1-332b-4d6e-a341-ecc9cc2ef281", "zonegroups": [ { "id": "9ced5b3b-5322-4ec9-862c-521bfff85c91", "name": "hn", "api_name": "hn", "is_master": true, "endpoints": [ "http://10.3.55.66:8080" ], "hostnames": [], "hostnames_s3website": [], "master_zone": "2760e65c-b9d1-415a-929e-f5de2098d5cf", "zones": [ { "id": "0c5aac15-c899-46aa-bff9-353bf17f9602", "name": "hn-2", "endpoints": [ "http://10.3.53.107:8080" ], "log_meta": false, "log_data": true, "bucket_index_max_shards": 11, "read_only": false, "tier_type": "", "sync_from_all": true, "sync_from": [], "redirect_zone": "", "supported_features": [ "compress-encrypted", "resharding" ] }, { "id": "2760e65c-b9d1-415a-929e-f5de2098d5cf", "name": "hn-1", "endpoints": [ "http://10.3.55.66:8080" ], "log_meta": false, "log_data": true, "bucket_index_max_shards": 11, "read_only": false, "tier_type": "", "sync_from_all": true, "sync_from": [], "redirect_zone": "", "supported_features": [ "compress-encrypted", "resharding" ] } ], "placement_targets": [ { "name": "default-placement", "tags": [], "storage_classes": [ "STANDARD" ] } ], "default_placement": "default-placement", "realm_id": "cb7337a4-5b38-41de-9763-ef7e22a79e17", "sync_policy": { "groups": [] }, "enabled_features": [ "resharding" ] } ], "short_zone_ids": [ { "key": "0c5aac15-c899-46aa-bff9-353bf17f9602", "val": 3534260942 }, { "key": "2760e65c-b9d1-415a-929e-f5de2098d5cf", "val": 1656529937 } ] }, "master_zonegroup": "9ced5b3b-5322-4ec9-862c-521bfff85c91", "master_zone": "2760e65c-b9d1-415a-929e-f5de2098d5cf", "period_config": { "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "user_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, ...ratelimit... }, "realm_id": "cb7337a4-5b38-41de-9763-ef7e22a79e17", "realm_name": "ss-realm", "realm_epoch": 2 } zone hn-1: { "id": "2760e65c-b9d1-415a-929e-f5de2098d5cf", "name": "hn-1", "domain_root": "hn-1.rgw.meta:root", "control_pool": "hn-1.rgw.control", "gc_pool": "hn-1.rgw.log:gc", "lc_pool": "hn-1.rgw.log:lc", "log_pool": "hn-1.rgw.log", "intent_log_pool": "hn-1.rgw.log:intent", "usage_log_pool": "hn-1.rgw.log:usage", "roles_pool": "hn-1.rgw.meta:roles", "reshard_pool": "hn-1.rgw.log:reshard", "user_keys_pool": "hn-1.rgw.meta:users.keys", "user_email_pool": "hn-1.rgw.meta:users.email", "user_swift_pool": "hn-1.rgw.meta:users.swift", "user_uid_pool": "hn-1.rgw.meta:users.uid", "otp_pool": "hn-1.rgw.otp", "system_key": { "access_key": "LIW8M2OTOCOHA41OJUGO", "secret_key": "KxD3fIWrW5GMpaY1ynCp10j2ccCP3fkfCcgyNBDT" }, "placement_pools": [ { "key": "default-placement", "val": { "index_pool": "hn-1.rgw.buckets.index", "storage_classes": { "STANDARD": { "data_pool": "hn-1.rgw.buckets.data" } }, "data_extra_pool": "hn-1.rgw.buckets.non-ec", "index_type": 0, "inline_data": true } } ], "realm_id": "cb7337a4-5b38-41de-9763-ef7e22a79e17", "notif_pool": "hn-1.rgw.log:notif" } zone hn-2: { "id": "0c5aac15-c899-46aa-bff9-353bf17f9602", "name": "hn-2", "domain_root": "hn-2.rgw.meta:root", "control_pool": "hn-2.rgw.control", "gc_pool": "hn-2.rgw.log:gc", "lc_pool": "hn-2.rgw.log:lc", "log_pool": "hn-2.rgw.log", "intent_log_pool": "hn-2.rgw.log:intent", "usage_log_pool": "hn-2.rgw.log:usage", "roles_pool": "hn-2.rgw.meta:roles", "reshard_pool": "hn-2.rgw.log:reshard", "user_keys_pool": "hn-2.rgw.meta:users.keys", "user_email_pool": "hn-2.rgw.meta:users.email", "user_swift_pool": "hn-2.rgw.meta:users.swift", "user_uid_pool": "hn-2.rgw.meta:users.uid", "otp_pool": "hn-2.rgw.otp", "system_key": { "access_key": "LIW8M2OTOCOHA41OJUGO", "secret_key": "KxD3fIWrW5GMpaY1ynCp10j2ccCP3fkfCcgyNBDT" }, "placement_pools": [ { "key": "default-placement", "val": { "index_pool": "hn-2.rgw.buckets.index", "storage_classes": { "STANDARD": { "data_pool": "hn-2.rgw.buckets.data" } }, "data_extra_pool": "hn-2.rgw.buckets.non-ec", "index_type": 0, "inline_data": true } } ], "realm_id": "cb7337a4-5b38-41de-9763-ef7e22a79e17", "notif_pool": "hn-2.rgw.log:notif" }

3 8

Documentation for meaning of "tag cephfs" in OSD caps
by Petr Bena 29 Jul '24

29 Jul '24

Hello In https://docs.ceph.com/en/latest/cephfs/client-auth/ we can find that ceph fs authorize cephfs_a client.foo / r /bar rw Results in client.foo key: *key* caps: [mds] allow r, allow rw path=/bar caps: [mon] allow r caps: [osd] allow rw tag cephfs data=cephfs_a What is this "tag cephfs" thing? It seems like some undocumented black magic to me, since I can't find anything that documents it. Can someone explain how it works under the hood? What does it expand to? What does it limit and how?

4 3

0 slow ops message stuck for down+out OSD
by Frank Schilder 29 Jul '24

29 Jul '24

Hi all, we had a failing disk (with slow ops) and I shut down the OSD. Its status down+out. However, I still see this message stuck in the output of ceph status and ceph health detail: 0 slow ops, oldest one blocked for 70 sec, osd.183 has slow ops I believe there was a case about that some time ago, but I can't find it. How can I get rid of this stuck warning? Our cluster is octopus latest. Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 4

snaptrim not making progress
by Frank Schilder 29 Jul '24

29 Jul '24

Hi all, our cluster is octopus latest. We seem to have a problem with snaptrim. On a pool for HDD RBD images I observed today that all PGs are either in state snaptrim or snaptrim_wait. It looks like the snaptrim process does not actually make any progress. There is no CPU activity by these OSDs indicating they would do snaptrimming (usuallu they would at least use 50% CPU as shown in top). I also don't see anything in the OSD logs. For our VMs we run daily snapshot rotation and snaptrim usually finishes within a few minutes. We had a VM with disks on that pool cause an error due to a hanging virsh domfsfreeze command. This, however, is routine, we see this happening every now and then without any follow-up issues. I'm wondering now if we might have hit a race for the first time. Is there anything on an RBD image or pool that could block snaptrim from starting or progressing? Thanks for any pointers! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

1 2

v18.2.4 Reef released
by Yuri Weinstein 29 Jul '24

29 Jul '24

We're happy to announce the 4th release in the Reef series. An early build of this release was accidentally exposed and packaged as 18.2.3 by the Debian project in April. That 18.2.3 release should not be used. The official release was re-tagged as v18.2.4 to avoid further confusion. v18.2.4 container images, now based on CentOS 9, may be incompatible on older kernels (e.g., Ubuntu 18.04) due to differences in thread creation methods. Users upgrading to v18.2.4 container images with older OS versions may encounter crashes during `pthread_create`. For workarounds, refer to the related tracker. However, we recommend upgrading your OS to avoid this unsupported combination. Related tracker: https://tracker.ceph.com/issues/66989 We recommend users to update to this release. For detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/en/news/blog/2024/v18-2-4-reef-released/ Notable Changes --------------- * RADOS: This release fixes a bug (https://tracker.ceph.com/issues/61948) where pre-reef clients were allowed to connect to the `pg-upmap-primary` (https://docs.ceph.com/en/reef/rados/operations/read-balancer/) interface despite users having set `require-min-compat-client=reef`, leading to an assert in the osds and mons. You are susceptible to this bug in reef versions prior to 18.2.3 if 1) you are using an osdmap generated via the offline osdmaptool with the `--read` option or 2) you have explicitly generated pg-upmap-primary mappings with the CLI command. Please note that the fix is minimal and does not address corner cases such as adding a mapping in the middle of an upgrade or in a partially upgraded cluster (related trackers linked in https://tracker.ceph.com/issues/61948) As such, we recommend removing any existing pg-upmap-primary mappings until remaining issues are addressed in future point releases. See https://tracker.ceph.com/issues/61948#note-32 for instructions on how to remove existing pg-upmap-primary mappings. * RBD: When diffing against the beginning of time (`fromsnapname == NULL`) in fast-diff mode (`whole_object == true` with `fast-diff` image feature enabled and valid), diff-iterate is now guaranteed to execute locally if exclusive lock is available. This brings a dramatic performance improvement for QEMU live disk synchronization and backup use cases. * RADOS: `get_pool_is_selfmanaged_snaps_mode` C++ API has been deprecated due to being prone to false negative results. It's safer replacement is `pool_is_in_selfmanaged_snaps_mode`. * RBD: The option ``--image-id`` has been added to `rbd children` CLI command, so it can be run for images in the trash. Related tracker: https://tracker.ceph.com/issues/65393 Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph_18.2.4.orig.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: e7ad5345525c7aa95470c26863873b581076945d

7 9

How to specify id on newly created OSD with Ceph Orchestrator
by Iztok Gregori 29 Jul '24

29 Jul '24

Hi, Yesterday I had to replace a couple of failed/failing OSDs using Ceph Orchestrator (on Ceph 18.2.1), but I've made a mistake and I removed them without the "--replace" option. I wasn't able to find the correct option (if any) to specify an id for newly created OSD. The closest one was "osd_id_claims", but if I understood correctly is used to claim an OSD id which is still in the crush map (as destroyed). Am I missing something obvious or with Ceph orchestrator there are non way to specify an id during the OSD creation? Cheers Iztok P.S. At the end I (re)added the OSD in the correct order, starting with the lowest id device. -- Iztok Gregori ICT Systems and Services Elettra - Sincrotrone Trieste S.C.p.A. http://www.elettra.eu

4 6

Stuck in remapped state?
by Tim Holloway 27 Jul '24

27 Jul '24

I was in the middle of tuning my OSDs when lightning blew me off the Internet. Had to wait 5 days for my ISP to send a tech and replace a fried cable. In the mean time, among other things. I had some serious time drift between servers thanks to the OS upgrades replacing NTP with chrony and me not having thought to re-establish a master in-house timeserver. Ceph tried really hard to keep up with all that, but eventually it was just too much. Now I've got an offline filesystem and apparently it's stuck trying to get back online again. The forensics: [ceph: root@www7 /]# ceph -s cluster: id: 278fcd86-0861-11ee-a7df-9c5c8e86cf8f health: HEALTH_WARN failed to probe daemons or devices 1 filesystem is degraded 1/3 mons down, quorum www7,ceph03 services: mon: 3 daemons, quorum www7,ceph03 (age 2m), out of quorum: dell02 mgr: ceph08.tlocfi(active, since 58m), standbys: dell02.odtbqw, www7.rxagfn mds: 1/1 daemons up, 1 standby osd: 7 osds: 7 up (since 12h), 7 in (since 18h); 308 remapped pgs rgw: 2 daemons active (2 hosts, 1 zones) data: volumes: 0/1 healthy, 1 recovering pools: 22 pools, 681 pgs objects: 125.10k objects, 36 GiB usage: 91 GiB used, 759 GiB / 850 GiB avail pgs: 47772/369076 objects misplaced (12.944%) 373 active+clean 308 active+clean+remapped io: client: 170 B/s rd, 0 op/s rd, 0 op/s wr [ceph: root@www7 /]# ceph health detail HEALTH_WARN 1 filesystem is degraded; 25 client(s) laggy due to laggy OSDs [WRN] FS_DEGRADED: 1 filesystem is degraded fs ceefs is degraded [WRN] MDS_CLIENTS_LAGGY: 25 client(s) laggy due to laggy OSDs mds.ceefs.www7.drnuyi(mds.0): Client 14019719 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14124385 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14144243 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14144375 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14224103 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14224523 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14234194 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14234545 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14236841 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14237837 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14238536 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14244124 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14264236 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14266870 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14294170 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14294434 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14296012 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14304212 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14316057 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14318379 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14325518 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14328956 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14334283 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14336104 is laggy; not evicted because some OSD(s) is/are laggy mds.ceefs.www7.drnuyi(mds.0): Client 14374237 is laggy; not evicted because some OSD(s) is/are laggy [ceph: root@www7 /]# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 2.79994 root default -25 0.15999 host ceph01 1 hdd 0.15999 osd.1 up 0.15999 1.00000 -28 1.15999 host ceph03 3 hdd 0.15999 osd.3 up 0.15999 1.00000 5 hdd 1.00000 osd.5 up 1.00000 1.00000 -9 0.15999 host ceph06 2 hdd 0.15999 osd.2 up 0.15999 1.00000 -3 0.15999 host ceph07 6 hdd 0.15999 osd.6 up 0.15999 1.00000 -6 1.00000 host ceph08 4 hdd 1.00000 osd.4 up 1.00000 1.00000 -7 0.15999 host www7 0 hdd 0.15999 osd.0 up 0.15999 1.00000 [ceph: root@www7 /]# ceph pg stat 681 pgs: 373 active+clean, 308 active+clean+remapped; 36 GiB data, 91 GiB used, 759 GiB / 850 GiB avail; 255 B/s rd, 0 op/s; 47772/369073 objects misplaced (12.944%)

1 2

18.2.4 regression: 'diskprediction_local' has failed: No module named 'sklearn'
by Harry G Coin 27 Jul '24

27 Jul '24

Upgraded to 18.2.4 yesterday. Healthy cluster reported a few minutes after the upgrade completed. Next morning, this: # ceph health detail HEALTH_ERR Module 'diskprediction_local' has failed: No module named 'sklearn' [ERR] MGR_MODULE_ERROR: Module 'diskprediction_local' has failed: No module named 'sklearn' Module 'diskprediction_local' has failed: No module named 'sklearn' Searching found this was a problem several years ago, then resolved, now returned. Any ideas? Harry Coin

2 2

Reef 18.2.4 EL8 packages ?
by Noe P. 26 Jul '24

26 Jul '24

Hi all, I find this 18.2.4 release highly confusing. Can anyone please confirm that EL8 packages are no more supported ? The small remark in the release notes qa/distros: remove centos 8 from supported distros (pr#57932, Guillaume Abrioux, Casey Bodley, Adam King, Laura Flores) is a bit disturbing, in particular between two minor releases. If yes, is the reason the missing support for EL8 distro kernels ? We could build the packages locally, but there is no v18.2.4 tag in git. Regards, Noe

3 3

Dashboard error on 18.2.4 when listing block images
by Dan O'Brien 26 Jul '24

26 Jul '24

I just yesterday updated by Ceph cluster from Quincy (17.2.7) to Reef (18.2.4, purely by accident; I didn't realize I was getting it THE EXACT SECOND it was pushed out). The upgrade resolved an issue I was having with HTTP 500 errors on the RGW UI, but seems to have created an issue in the dashboard with the Block / Images screen. I see errors that come up as alerts: 0 - Unknown Error * Http failure response for api/prometheus: 0 Unknown Error * Http failure reponse for api/prometheus/rules: 0 Unknown Error * Http failure response for api/summary: 0 Unknown Error All the other menu options on the dashboard are working. I thought it was a UI issue, but having read the notes on 18.2.4, I wonder if I've hit a bug, or if there's something wonky with the RBD images in some of our pools. We're using Ceph mostly for OpenStack. The syslog on the mgr server shows an error and traceback, when I try to use the Block / Images menu link. Notably I see Jul 25 09:13:15 ceph00.cecnet.gmu.edu ceph-mgr[2293091]: librbd::DiffIterate: fast diff enabled just before the crash, and there's mention of this function in the release notes. I'm going to try and use the rbd command and query some of the pools and images to see if I can narrow down specifically what's going on. If anyone's got some suggestions, I'd love to hear them. I can drop additional information from the logs if needed. I don't THINK there were errors prior to the upgrade, but I can't say for sure since I've been working in other parts of the Ceph UI. Full traceback below Jul 25 09:13:15 ceph00.cecnet.gmu.edu ceph-mgr[2293091]: asok(0x56360092c000) register_command rbd cache flush images-pubos/144ebab3-b2ee-4331-9d41-8505bcc4e19b hook 0x56361144ffc0 Jul 25 09:13:15 ceph00.cecnet.gmu.edu ceph-mgr[2293091]: asok(0x56360092c000) register_command rbd cache invalidate images-pubos/144ebab3-b2ee-4331-9d41-8505bcc4e19b hook 0x56361144ffc0 Jul 25 09:13:15 ceph00.cecnet.gmu.edu ceph-mgr[2293091]: librbd::ImageCtx: 0x5636176bd000: disabling zero-copy writes Jul 25 09:13:15 ceph00.cecnet.gmu.edu ceph-mgr[2293091]: librbd::io::Dispatcher: 0x563615cf1170 register_dispatch: dispatch_layer=3 Jul 25 09:13:15 ceph00.cecnet.gmu.edu ceph-mgr[2293091]: librbd::cache::WriteAroundObjectDispatch: 0x56360ce5efc0 init: Jul 25 09:13:15 ceph00.cecnet.gmu.edu ceph-mgr[2293091]: librbd::io::Dispatcher: 0x56360bd64900 register_dispatch: dispatch_layer=1 Jul 25 09:13:15 ceph00.cecnet.gmu.edu ceph-mgr[2293091]: librbd::io::SimpleSchedulerObjectDispatch: 0x5636164b9e00 SimpleSchedulerObjectDispatch: ictx=0x5636176bd000 Jul 25 09:13:15 ceph00.cecnet.gmu.edu ceph-mgr[2293091]: librbd::io::SimpleSchedulerObjectDispatch: 0x5636164b9e00 init: Jul 25 09:13:15 ceph00.cecnet.gmu.edu ceph-mgr[2293091]: librbd::io::Dispatcher: 0x56360bd64900 register_dispatch: dispatch_layer=5 Jul 25 09:13:15 ceph00.cecnet.gmu.edu ceph-mgr[2293091]: librbd::io::WriteBlockImageDispatch: 0x5636134e46e0 block_writes: 0x5636176bd000, num=1 Jul 25 09:13:15 ceph00.cecnet.gmu.edu ceph-mgr[2293091]: librbd::io::Dispatcher: 0x563615cf1170 shut_down_dispatch: dispatch_layer=3 Jul 25 09:13:15 ceph00.cecnet.gmu.edu ceph-mgr[2293091]: librbd::io::WriteBlockImageDispatch: 0x5636134e46e0 unblock_writes: 0x5636176bd000, num=0 Jul 25 09:13:15 ceph00.cecnet.gmu.edu ceph-mgr[2293091]: librbd::io::WriteBlockImageDispatch: 0x5636134e46e0 block_writes: 0x5636176bd000, num=1 Jul 25 09:13:15 ceph00.cecnet.gmu.edu ceph-mgr[2293091]: librbd::io::WriteBlockImageDispatch: 0x5636134e46e0 unblock_writes: 0x5636176bd000, num=0 Jul 25 09:13:15 ceph00.cecnet.gmu.edu ceph-mgr[2293091]: librbd::DiffIterate: fast diff enabled Jul 25 09:13:15 ceph00.cecnet.gmu.edu ceph-mgr[2293091]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4/rpm/el9/BUILD/ceph-18.2.4/src/librbd/api/DiffIterate.cc: In function 'int librbd::api::DiffIterate<ImageCtxT>::execute() [with ImageCtxT = librbd::ImageCtx]' thread 7fd5897ac640 time 2024-07-25T13:13:15.549430+0000 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.4/rpm/el9/BUILD/ceph-18.2.4/src/librbd/api/DiffIterate.cc: 341: FAILED ceph_assert(object_diff_state.size() == end_object_no - start_object_no) ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x12e) [0x7fd6f5b0104d] 2: /usr/lib64/ceph/libceph-common.so.2(+0x16b20b) [0x7fd6f5b0120b] 3: /lib64/librbd.so.1(+0x193403) [0x7fd6e5ce2403] 4: /lib64/librbd.so.1(+0x51ada7) [0x7fd6e6069da7] 5: rbd_diff_iterate2() 6: /lib64/python3.9/site-packages/rbd.cpython-39-x86_64-linux-gnu.so(+0x630bc) [0x7fd6e62f40bc] 7: /lib64/libpython3.9.so.1.0(+0x11d7a1) [0x7fd6f661e7a1] 8: PyVectorcall_Call() 9: /lib64/python3.9/site-packages/rbd.cpython-39-x86_64-linux-gnu.so(+0x44d50) [0x7fd6e62d5d50] 10: _PyObject_MakeTpCall() 11: /lib64/libpython3.9.so.1.0(+0x125133) [0x7fd6f6626133] 12: _PyEval_EvalFrameDefault() 13: /lib64/libpython3.9.so.1.0(+0x10ec35) [0x7fd6f660fc35] 14: _PyFunction_Vectorcall() 15: /lib64/libpython3.9.so.1.0(+0x125031) [0x7fd6f6626031] 16: _PyEval_EvalFrameDefault() 17: /lib64/libpython3.9.so.1.0(+0x11cb73) [0x7fd6f661db73] 18: /lib64/libpython3.9.so.1.0(+0x125031) [0x7fd6f6626031] 19: _PyEval_EvalFrameDefault() 20: /lib64/libpython3.9.so.1.0(+0x11cb73) [0x7fd6f661db73] 21: /lib64/libpython3.9.so.1.0(+0x125031) [0x7fd6f6626031] 22: _PyEval_EvalFrameDefault() 23: /lib64/libpython3.9.so.1.0(+0x10ec35) [0x7fd6f660fc35] 24: _PyFunction_Vectorcall() 25: /lib64/libpython3.9.so.1.0(+0x125031) [0x7fd6f6626031] 26: _PyEval_EvalFrameDefault() 27: /lib64/libpython3.9.so.1.0(+0x10ec35) [0x7fd6f660fc35] 28: _PyFunction_Vectorcall() 29: /lib64/libpython3.9.so.1.0(+0x125031) [0x7fd6f6626031] 30: _PyEval_EvalFrameDefault() 31: /lib64/libpython3.9.so.1.0(+0x10ec35) [0x7fd6f660fc35] Jul 25 09:13:15 ceph00.cecnet.gmu.edu ceph-mgr[2293091]: *** Caught signal (Aborted) ** in thread 7fd5897ac640 thread_name:dashboard ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable) 1: /lib64/libc.so.6(+0x3e6f0) [0x7fd6f54aa6f0] 2: /lib64/libc.so.6(+0x8b94c) [0x7fd6f54f794c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7fd6f5b010a7] 6: /usr/lib64/ceph/libceph-common.so.2(+0x16b20b) [0x7fd6f5b0120b] 7: /lib64/librbd.so.1(+0x193403) [0x7fd6e5ce2403] 8: /lib64/librbd.so.1(+0x51ada7) [0x7fd6e6069da7] 9: rbd_diff_iterate2() 10: /lib64/python3.9/site-packages/rbd.cpython-39-x86_64-linux-gnu.so(+0x630bc) [0x7fd6e62f40bc] 11: /lib64/libpython3.9.so.1.0(+0x11d7a1) [0x7fd6f661e7a1] 12: PyVectorcall_Call() 13: /lib64/python3.9/site-packages/rbd.cpython-39-x86_64-linux-gnu.so(+0x44d50) [0x7fd6e62d5d50] 14: _PyObject_MakeTpCall() 15: /lib64/libpython3.9.so.1.0(+0x125133) [0x7fd6f6626133] 16: _PyEval_EvalFrameDefault() 17: /lib64/libpython3.9.so.1.0(+0x10ec35) [0x7fd6f660fc35] 18: _PyFunction_Vectorcall() 19: /lib64/libpython3.9.so.1.0(+0x125031) [0x7fd6f6626031] 20: _PyEval_EvalFrameDefault() 21: /lib64/libpython3.9.so.1.0(+0x11cb73) [0x7fd6f661db73] 22: /lib64/libpython3.9.so.1.0(+0x125031) [0x7fd6f6626031] 23: _PyEval_EvalFrameDefault() 24: /lib64/libpython3.9.so.1.0(+0x11cb73) [0x7fd6f661db73] 25: /lib64/libpython3.9.so.1.0(+0x125031) [0x7fd6f6626031] 26: _PyEval_EvalFrameDefault() 27: /lib64/libpython3.9.so.1.0(+0x10ec35) [0x7fd6f660fc35] 28: _PyFunction_Vectorcall() 29: /lib64/libpython3.9.so.1.0(+0x125031) [0x7fd6f6626031] 30: _PyEval_EvalFrameDefault() 31: /lib64/libpython3.9.so.1.0(+0x10ec35) [0x7fd6f660fc35] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

2 7

ceph fs authorize doesn't work correctly / Operation not permitted
by Rudenko Aleksandr 26 Jul '24

26 Jul '24

Hi, I have two ceph clusters on 16.2.13 and one CephFS in each. I can see only one difference between FSs: the second one FS has two data pools and one of the root directories is pinned to the second pool. In the first one FS we have only one default data pool. And, when I do: ceph fs authorize fs1 client.XXX /root-dir/some-dir rw I have next permissions in first and second FSs: [client.XXX] key = AQCmbp9myJ0pBhAAnoCBxvKbOfyGH5vs0g2QhQ== caps mds = "allow rw fsname=fs1 path=/root-dir/some-dir" caps mon = "allow r fsname=fs1" caps osd = "allow rw tag cephfs data=fs1" and zero problem in the first FS, the user can mount and write new files. But, on the second FS user can't write files. Operation not permitted. In client logs I can see: check_pool_perm on pool 15 ns need AsFw, but no write perm If I update caps in second FS like this: ceph auth caps client.XXX mds 'allow rw fsname=fs1, allow rw fsname=fs1 path=/root-dir/some-dir' mon 'allow r fsname=fs1' osd 'allow rw pool=fs1_data' and caps will looks like: [client.XXX] key = AQA2e6JmeNrWABAAoIiBstYFSr4/UZ3N1vIrHg== caps mds = "allow rw fsname=fs1, allow rw fsname=fs1 path=/root-dir/some-dir" caps mon = "allow r fsname=fs1" caps osd = "allow rw pool=fs1_data" Everything goes well. So, I have two questions: 1. Where is the problem with "allow rw tag cephfs data=fs1", how can I debug this? 2. What is the difference between "allow rw pool=fs1_data"? Can I use this caps format instead of ceph fs authorize

1 0

[RGW] radosgw does not respond after some time after upgrade from pacific to quincy
by Rok Jaklič 25 Jul '24

25 Jul '24

Hi, we've just updated from pacific(16.2.15) to quincy(17.2.7) and everything seems to work, however after some time radosgw stops responding and we have to restart it. At first look, it seems that radosgw stops responding sometimes during recovery. Does this maybe have to do something with mclock https://docs.ceph.com/en/quincy/rados/configuration/mclock-config-ref/ ? ceph.conf looks something like this: -------- [global] fsid = ... mon initial members = x1,x2 mon host = # public network = 192.168.0.0/24 auth cluster required = none auth service required = none auth client required = none ms_mon_client_mode = crc osd journal size = 1024 osd pool default size = 3 osd pool default min size = 2 osd pool default pg num = 128 osd pool default pgp num = 128 osd crush chooseleaf type = 1 [osd] osd_scrub_begin_hour = 18 osd_scrub_end_hour = 6 osd_class_update_on_start = false osd_scrub_during_recovery = false #scrub during recovery osd_scrub_max_interval = 1209600 osd_deep_scrub_interval = 1209600 osd_max_scrubs = 3 osd_scrub_load_threshold = 1 [client.radosgw.mon2] host = mon2 # keyring = /etc/ceph/ceph.client.radosgw.keyring log_file = /var/log/ceph/client.radosgw.mon2.log rgw_dns_name = ... rgw_frontends = "beast port=4444" rgw_max_put_param_size = 15728640 rgw_crypt_require_ssl = false rgw_max_concurrent_requests = 2048 -------- We have nginx in front of rgw which upstream is set to client.radosgw.mon2 port 4444. Kind regards, Rok

2 3

Ceph Upgrade 18.2.2 -> 18.2.4 fails | Fatal glibc error: CPU does not support x86-64-v2 on virtualized hosts
by Björn Lässig 25 Jul '24

25 Jul '24

Hi there, Last week I upgraded from 17.2.7 to 18.2.2 and the cluster worked fine except for a few bugs about IPv6. But it is working and cluster.health is HEALTH_OK. Hoping to fix some of these minor IPv6 issues, I tried upgrading to 18.2.4. I see 2 errors: The first is that the upgrade fails to: { "target_image": "quay.io/ceph/ceph@sha256:6ac7f923aa1d23b43248ce0ddec7e1388855ee3d00813b52c3172b0b23b37906", "in_progress": true, "which": "Upgrading all daemon types on all hosts", "services_complete": [], "progress": "0/89 daemons upgraded", "message": "Error: UPGRADE_FAILED_PULL: Upgrade: failed to pull target image", "is_paused": true } The error message is misleading as: # podman images --digests REPOSITORY TAG DIGEST IMAGE ID CREATED SIZE quay.io/ceph/ceph <none> sha256:6ac7f923aa1d23b43248ce0ddec7e1388855ee3d00813b52c3172b0b23b37906 2bc0b0f4375d 33 hours ago 1.25 GB […] the target image was pulled without problems. (podman pull runs without problems) The error occures while starting the image: Jul 25 09:16:18 cephmgr1 podman[1171476]: 2024-07-25 09:16:18.752832293 +0200 CEST m=+0.047066303 container create 4368eba845dcbc6f3cb7fe8265fc7044d91208f87c25cbf1abe6c3d0447b468e (image=quay.io/ceph/ceph@sha256:6ac7f923aa1d23b43248ce0ddec7e1388855ee3d00813b52c3172b0b23b37906, name=reverent_hypatia, org.label-schema.build-date=20240716, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=HEAD, org.label-schema.license=GPLv2, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.33.8, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=-18.2.4, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, ceph=True) Jul 25 09:16:18 cephmgr1 systemd[1]: Started libpod-conmon-4368eba845dcbc6f3cb7fe8265fc7044d91208f87c25cbf1abe6c3d0447b468e.scope. Jul 25 09:16:18 cephmgr1 systemd[1]: Started libpod-4368eba845dcbc6f3cb7fe8265fc7044d91208f87c25cbf1abe6c3d0447b468e.scope - libcrun container. Jul 25 09:16:18 cephmgr1 podman[1171476]: 2024-07-25 09:16:18.820823054 +0200 CEST m=+0.115057064 container init 4368eba845dcbc6f3cb7fe8265fc7044d91208f87c25cbf1abe6c3d0447b468e (image=quay.io/ceph/ceph@sha256:6ac7f923aa1d23b43248ce0ddec7e1388855ee3d00813b52c3172b0b23b37906, name=reverent_hypatia, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=-18.2.4, org.label-schema.build-date=20240716, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.33.8, GIT_BRANCH=HEAD, RELEASE=HEAD, ceph=True, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True) Jul 25 09:16:18 cephmgr1 podman[1171476]: 2024-07-25 09:16:18.826093069 +0200 CEST m=+0.120327079 container start 4368eba845dcbc6f3cb7fe8265fc7044d91208f87c25cbf1abe6c3d0447b468e (image=quay.io/ceph/ceph@sha256:6ac7f923aa1d23b43248ce0ddec7e1388855ee3d00813b52c3172b0b23b37906, name=reverent_hypatia, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=-18.2.4, RELEASE=HEAD, org.label-schema.build-date=20240716, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.33.8) Jul 25 09:16:18 cephmgr1 podman[1171476]: 2024-07-25 09:16:18.726428757 +0200 CEST m=+0.020662767 image pull quay.io/ceph/ceph@sha256:6ac7f923aa1d23b43248ce0ddec7e1388855ee3d00813b52c3172b0b23b37906 Jul 25 09:16:18 cephmgr1 podman[1171476]: 2024-07-25 09:16:18.82623823 +0200 CEST m=+0.120472250 container attach 4368eba845dcbc6f3cb7fe8265fc7044d91208f87c25cbf1abe6c3d0447b468e (image=quay.io/ceph/ceph@sha256:6ac7f923aa1d23b43248ce0ddec7e1388855ee3d00813b52c3172b0b23b37906, name=reverent_hypatia, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-18.2.4, RELEASE=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240716, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.33.8, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux <gabrioux(a)redhat.com>) Jul 25 09:16:18 cephmgr1 reverent_hypatia[1171489]: Fatal glibc error: CPU does not support x86-64-v2 Jul 25 09:16:18 cephmgr1 systemd[1]: libpod-4368eba845dcbc6f3cb7fe8265fc7044d91208f87c25cbf1abe6c3d0447b468e.scope: Deactivated successfully. When starting the container for 18.2.4. a glibc error occurs. 18.2.2. to 18.2.4 is a minor upgrade and should run on same hardware but this is not hardware: Both mgr and 3 of 5 monitor daemons run on virtual machines on a proxmox (qemu-based virtualizaton) which has the default CPU "kvm64". This means for the qemu options: kvm ... -smp 1,sockets=1,cores=4,maxcpus=4 \ -device kvm64-x86_64-cpu,id=cpu2,socket-id=0,core-id=1,thread-id=0\ -device kvm64-x86_64-cpu,id=cpu3,socket-id=0,core-id=2,thread-id=0\ -device kvm64-x86_64-cpu,id=cpu4,socket-id=0,core-id=3,thread-id=0 Which CPU type should I choose for my VMs for this libc? regards Björn Lässig

2 4

Subscribe
by Dan O'Brien 25 Jul '24

25 Jul '24

2 2

Release 18.2.4
by Alfredo Rezinovsky 24 Jul '24

24 Jul '24

Ceph dashboard offers me to upgrade to v18.2.4. I can't find any information on 18.2.4. The is no 18.2.4 in https://docs.ceph.com/en/latest/releases/ Not a tag in https://github.com/ceph/ceph I don´t understand why there's a 18.2.4 image and what's in it. -- Alfrenovsky

3 2

ingress for mgr service
by farhad kh 24 Jul '24

24 Jul '24

It is very good to be able to use Ingress for service manager This possibility of high availability for Ed Will this feature be added in the next version or should it still be implemented manually?

1 0

[MDS] Pacific memory leak
by Adrien Georget 23 Jul '24

23 Jul '24

Hi, For the last 2 months, our MDS is frequently switching to another because of a sudden memory leak. The host has 128G RAM and most of the time the MDS occupies ~20% of memory. And in less than 3 minutes it increases to 100% and crashs with tcmalloc: allocation failed. We tried to run heap stats / perf dump on the host but we couldn't find any reasons why the memory used by the MDS exploses so quickly. MDS log available here : https://filesender.renater.fr/?s=download&token=c1e60c3c-7f02-4f1e-b23e-f5b… Any idea what could lead to this memory leak? Anything we can try to understand what happens or prevent this? We use Pacific 16.2.14. Cheers, Adrien |cluster: id: 86cd8a68-7649-11ed-b2be-5cba2c7fdb30 health: HEALTH_OK services: mon: 5 daemons, quorum cccephadm40,cccephadm42,cccephadm41,cccephadm43,cccephadm44 (age 3d) mgr: cccephadm40.ucqhkr(active, since 105m), standbys: cccephadm41.osnxsd mds: 4/4 daemons up, 1 standby osd: 720 osds: 720 up (since 9d), 720 in (since 7w) rgw: 1 daemon active (1 hosts, 1 zones) data: volumes: 1/1 healthy pools: 12 pools, 8801 pgs objects: 1.25G objects, 2.7 PiB usage: 3.5 PiB used, 2.6 PiB / 6.1 PiB avail pgs: 8731 active+clean 59 active+clean+scrubbing+deep 11 active+clean+scrubbing io: client: 2.1 GiB/s rd, 1.2 GiB/s wr, 3.63k op/s rd, 17.47k op/s wr ceph fs status cephfs_astro - 1628 clients ============ RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active cephfs_astro.cccephadm41.nbgvxk Reqs: 7205 /s 11.2M 11.1M 708k 2782k 1 active cephfs_astro.cccephadm42.ieexgj Reqs: 812 /s 11.6M 11.5M 554k 1366k 2 active cephfs_astro.cccephadm40.aafhps Reqs: 1 /s 11.4M 11.3M 1346k 462k 3 active cephfs_astro.cccephadm44.jodfcx Reqs: 3 /s 11.4M 11.4M 1094k 29.3k POOL TYPE USED AVAIL cephfs_metadata metadata 484G 13.5T cephfs_default data 0 13.5T cephfs_EC_data data 3463T 1830T cephfs_ssd data 5314G 13.5T STANDBY MDS cephfs_astro.cccephadm43.wetlrp MDS version: ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable) |

2 2

A problem with the CEPH PG state getting stuck
by 苏察哈尔灿 23 Jul '24

23 Jul '24

My ceph cluster sometimes gets stuck in the active+clean+snaptrim state when doing regular snapshot deletion, and the corresponding pg does not change for a long time. As follows: 27.7c     14350         0          0        0  38073876582            0           0   2864      3000        active+clean+snaptrim     9h  43777'677286  43777:1018248    [52,55,20,36,29,91,9,63,14,2]p52    [52,55,20,36,29,91,9,63,14,2]p52  2024-07-21T13:57:12.602903+0000  2024-07-20T11:02:06.953985+0000                   52  queued for scrub                                                27.c1     14055         0          0        0  37408096205            0           0   2754      3000        active+clean+snaptrim     9h  43777'676887   43777:952875     [0,21,39,62,26,58,41,86,66,2]p0     [0,21,39,62,26,58,41,86,66,2]p0  2024-07-22T02:26:43.470883+0000  2024-07-20T05:09:23.763918+0000                   44  periodic scrub scheduled @ 2024-07-23T07:46:50.084316+0000      27.19b    14711         0          0        0  38926389125            0           0   2765      3000        active+clean+snaptrim     9h  43777'698478   43777:849574      [5,7,29,14,61,10,22,0,19,37]p5      [5,7,29,14,61,10,22,0,19,37]p5  2024-07-22T03:28:39.036189+0000  2024-07-17T14:00:43.082766+0000                   40  periodic scrub scheduled @ 2024-07-23T12:30:57.488823+0000      27.1a3    14323         0          0        0  37918899397            0           0   2930      3000        active+clean+snaptrim     9h  43777'675713   43777:943324   [52,4,10,46,20,69,49,39,44,30]p52   [52,4,10,46,20,69,49,39,44,30]p52  2024-07-21T11:15:16.442506+0000  2024-07-20T04:10:45.391550+0000                   50  queued for scrub                  No change for 9 hours. Then, after I typed the command "ceph pg repeer 27.7c", the corresponding pg state was restored to normal. I don't know what this "repeer" command is for, will it have any effect on pg? What's the difference between "repeer" and "ceph pg repair"? Then, why does the pg get stuck so often? Thanks for your help! My ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)

1 0

cephadm rgw ssl certificate config
by Eugen Block 22 Jul '24

22 Jul '24

Hi, I came across [1] and wanted to try to have all certificates/keys in one file. But it appears that the validation happens only against the first cert. So what I did was to concatenate all certs/keys into one file, then added that to ceph: ceph config-key set rgw/cert/rgw.realm.zone -i all-hosts.cert.pem ceph orch redeploy rgw The daemons pick up the config correctly and start, but running a curl only succeeds for the first in the list: # first host ok curl https://host3.dmain:443 <?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult... # second host fails curl -v https://host4.domain:443 ... * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 * ALPN, server did not agree to a protocol * Server certificate: * subject: [...] CN=host3.domain; emailAddress=[...] * start date: Jul 18 09:59:07 2024 GMT * expire date: Aug 17 09:59:07 2025 GMT * subjectAltName does not match host4.domain * SSL: no alternative certificate subject name matches target host name 'host4.domain' This is still on Pacific 16.2.13 (testing something for a customer). What I tried in a Reef cluster was to set the certs per host by mapping the cert directory into the containers, then changing this config: ceph config set client.rgw.realm.hostname.suffix rgw_frontends "beast ssl_port=443 ssl_certificate=/etc/ceph/mycert.pem" And after restarting the daemon, it seems to work. So my question is, how do you deal with per-host certificates and rgw? Any comments are appreciated. Thanks! Eugen [1] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/F2GXMK6PE2…

2 3

Ceph 19 Squid released?
by Nicola Mori 21 Jul '24

21 Jul '24

Dear Ceph users, on quay.io I see available images for 19.1.0. Yet I can't find any public release announcement, and on this page: https://docs.ceph.com/en/latest/releases/ version 19 is still not mentioned at all. So what's going on? Nicola

2 1

PSA: Bringing up an OSD really, really fast
by Tim Holloway 20 Jul '24

20 Jul '24

While running Ceph in a VM may not be considered the most efficient mode of operation, it's still fairly popular. I've put together a small project that can spin up an OSD VM in just a few minutes. Whereas Ceph given free rein will gladly acquire resources with all the enthusiasm as a sprig of mint in damp earth, what I've developed is more useful for a small farm with very irregular resources such as my own. And for research purposes! The project has 2 components: one of which gets deployed on the intended VM host, the other being an Ansible role and model playbook. Full instructions are provided. And incidentally, the Ansible role should be able to run against a base OS host, even though it was developed for VMs. Comments and suggestions welcome. https://gogs.mousetech.com/mtsinc7/instant_osd

1 0

Ceph filesystems deadlocking and freezing
by Domhnall McGuigan 19 Jul '24

19 Jul '24

Hi, we're trying to run our database package with CephFS as the backing store and we're seeing constant freezes on filesystem accesses, apparently due to deadlocks in Ceph (all versions tested on 5.x kernels, 16.2.6 up to 18.2.2). I've made an issue on the tracker for it here with the details I've found so far: https://tracker.ceph.com/issues/67008 But if that's the wrong place to raise it, or if there are any additional details I could provide, please let me know. *********************************************************************************************************************************************************************** This email, its contents and any files attached are a confidential communication and are intended only for the named addressees indicated in the message. If you are not the named addressee or if you have received this email in error, you may not, without the consent of First Derivatives, copy, use or rely on any information or attachments in any way. Please notify the sender by return email and delete it from your email system. Unless separately agreed, First Derivatives does not accept any responsibility for the accuracy or completeness of the contents of this email or its attachments. Please note that any views, opinion or advice contained in this communication are those of the sending individual and not those of First Derivatives and First Derivatives shall have no liability whatsoever in relation to this communication (or its content) unless separately agreed. ***********************************************************************************************************************************************************************

1 0

Cephadm has a small wart
by Tim Holloway 19 Jul '24

19 Jul '24

I've been setting up a cookbook OSD creation process and as I walked through the various stages, I noted that the /etc/redhat-release file said "CentOS Stream 8". I panicked, because IBM has pulled the Ceph archives for CentOS 8 and nuked the machine, then rebuilt it with more attention to detail. I consistently was seeing the correct redhat-release until I installed cephadm and ran cephadm shell. Cephadm is running a chroot jail, it seems, and it's plugging in the wrong value for that file. Presumably from its internal image instead of copying the actual file. The false release string does not leak outside of cephadm, though. So far this appears to be only important as a possible confusion point, but it is of concern since the repositories in /etc/yum.repos.d download from repositories whose URLs are derived from the OS release that YUM (dnf) thinks it's running under and I'd hate to find out the hard way that something got confused. Probably rpm and yum/dnf shouldn't be available from within cephadm, but since cephadm pulls stuff internally that may not be possible. This was seen on AlmaLinux 9.4, cephadm version 16.2.15.

3 4

v19.1.0 Squid RC0 released
by Yuri Weinstein 19 Jul '24

19 Jul '24

This is the first release candidate for Squid. Feature highlights: RGW The User Accounts feature unlocks several new AWS-compatible IAM APIs for the self-service management of users, keys, groups, roles, policy and more. RADOS BlueStore has been optimized for better performance in snapshot-intensive workloads. BlueStore RocksDB LZ4 compression is now enabled by default to improve average performance and "fast device" space usage. Other improvements include more flexible EC configurations, an OpTracker to help debug mgr module issues, and better scrub scheduling. Dashboard * Rearranged Navigation Layout: The navigation layout has been reorganized for improved usability and easier access to key features. CephFS Improvements * Support for managing CephFS snapshots and clones, as well as snapshot schedule management * Manage authorization capabilities for CephFS resources * Helpers on mounting a CephFS volume RGW Improvements * Support for managing bucket policies * Add/Remove bucket tags * ACL Management * Several UI/UX Improvements to the bucket form Monitoring: Grafana dashboards are now loaded into the container at runtime rather than building a grafana image with the grafana dashboards. Official Ceph grafana images can be found in quay.io/ceph/grafana * Monitoring: RGW S3 Analytics: A new Grafana dashboard is now available, enabling you to visualize per bucket and user analytics data, including total GETs, PUTs, Deletes, Copies, and list metrics. Crimson/Seastore Crimson's first tech preview release! Supporting RBD workloads on Replicated pools. For more information please visit: https://ceph.io/en/news/crimson If any of our community members would like to help us with performance investigations or regression testing of the Squid release candidate, please feel free to provide feedback via email or in https://pad.ceph.com/p/squid_scale_testing. For more active discussions, please use the #ceph-at-scale slack channel in ceph-storage.slack.com. * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-19.1.0.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: 9025b9024baf597d63005552b5ee004013630404

3 2

How to detect condition for offline compaction of RocksDB?
by Rudenko Aleksandr 19 Jul '24

19 Jul '24

Hi, We have a big Ceph cluster (RGW case) with a lot of big buckets 10-500M objects with 31-1024 shards and a lot of io generated by many clients. Index pool placed on enterprise SSDs. We have about 120 SSDs (replication 3) and about 90Gb of OMAP data on each drive. About 75 PGs on each SSD for now. I think 75 is not enough for this amount of data, but I’m not sure that it is critical in our case. The problem: For the last few weeks, we can see big degradation of some SSD OSDs. We can see a lot of Ops (150-256 in perf dump) for a long time and we can see high avg ops time like 1-3 seconds (this metric is based on dump_historic_ops_by_duration and our avg calculation). These numbers are very unusual for our deployment. And it makes a big impact on our customers. For now, we make Offline compaction of ‘degraded’ OSDs and after compaction, we see that all PGs of this OSD are returned on this OSD (because recover is disabled during compaction) and we see that OSD works perfect for some time, all our metrics are dropped down for few weeks or only few days… And, I think the problem is in the rocksdb database which grows by levels and slows down requests. And I have few question about compaction: 1. Why can't automatic compaction manage this on its own? 2. How can I see RocksDB levels usage or some program metric which can be used as a condition for manual compacting? Because our metrics like request latency, OSD Ops count, OSD avg slow ops time are not 100% relative to rocksdb internal state. We try to use perf dump, bluestore.kv_xxx_lat metrics, but I think they are absolutely useless. Because we can see higher values after compaction and restart OSD then these metrics were before compaction, lol. We try to see OSD logs with debug_rocksdb=4, but we can’t understand this output compare it between good and bad OSDs: Uptime(secs): 3610809.5 total, 600.0 interval Flush(GB): cumulative 50.519, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 475.34 GB write, 0.13 MB/s write, 453.07 GB read, 0.13 MB/s read, 3346.3 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [p-0] ** ** Compaction Stats [p-1] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 0/0 0.00 KB 0.0 0.0 0.0 0.0 52.0 52.0 0.0 1.0 0.0 155.2 343.25 176.27 746 0.460 0 0 L1 4/0 204.00 MB 0.8 96.7 52.0 44.7 96.2 51.4 0.0 1.8 164.6 163.6 601.87 316.34 237 2.540 275M 1237K L2 34/0 2.09 GB 1.0 176.8 43.6 133.2 175.3 42.1 7.6 4.0 147.6 146.3 1226.72 590.20 535 2.293 519M 2672K L3 79/0 4.76 GB 0.3 163.6 38.9 124.7 140.7 16.1 8.8 3.6 152.6 131.3 1097.53 443.60 432 2.541 401M 79M L4 316/0 19.96 GB 0.1 0.9 0.4 0.5 0.8 0.3 19.7 2.0 151.1 131.1 5.94 2.53 4 1.485 2349K 385K Sum 433/0 27.01 GB 0.0 438.0 134.9 303.0 465.0 161.9 36.1 8.9 136.9 145.4 3275.31 1528.94 1954 1.676 1198M 83M Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 and this log is not usable as ‘metric’. Our Ceph version is: 16.2.13 and we use default bluestore_rocksdb_options.

3 12

Converting/Migrating EC pool to a replicated pool
by Burkhard Linke 19 Jul '24

19 Jul '24

Hi, we recently expanded a ceph cluster with a second site and want to distribute data accordingly. Two replicate should be present on one site, one replicate on the other site. This works well for replicated pools, but not for EC pool, e.g. the RGW data pool. I've added a second (replicated) storage class to RGW and moved most of the data to it via lifecycle operations. Now about 670 GB are left in the EC pool, mostly the head objects which cannot be moved. I'm looking for the best method to migrate the existing EC pool to a replicated pool, keeping the downtime a small as possible. So far two methods seem to be feasible: 1. Cache tiering Setup cache tiering with the EC pool as frontend pool and a replicated target pool as backend pool. Set cache mode to proxy and evict all objects from the EC pool the replicated pool. This will reduce the downtime to a minimum (stopping all RGW instances, changes the pool name in the placement definition, restart instance). But I'm not sure whether a EC pool can acts as frontend pool at all. And cache tiering is deprecated, so there might be other downfalls. 2. Export / Import Stop all RGW instances, export the EC pool's content with "rados export", and import it into the target pool. This should be a safe method, but will require a significant longer downtime (probably 24+ hours). Are there any other options? I also evaluated working bucket wise by creating a second placement set, disable access to a bucket, move all objects belonging to that bucket from one pool to the other (identifiable via the bucket's marker), change the placement setting of the bucket and reenable access. Unfortunately I've didn't found a good method to temporarily disable bucket access. Best regards, Burkhard Linke

3 2

Small issue with perms
by Albert Shih 18 Jul '24

18 Jul '24

Hi everyone. After my upgrade from 17.2.7 to 18.2.2 I notice after each time I restart I got a issue with perm on /var/lib/ceph/FSID/crash after the restart the owner/group are on nobody and I got Error scraping /var/lib/ceph/crash: [Errno 13] Permission denied: '/var/lib/ceph/crash' I need to do a chown ceph:ceph on the directory. But on the next reboot the dir return to nobody. Anyway to fix (beside some cron + chown) this minor bug ? Regards -- Albert SHIH 🦫 🐸 Observatoire de Paris France Heure locale/Local time: jeu. 18 juil. 2024 09:53:40 CEST

3 9

pg's stuck activating on osd create
by Richard Bade 18 Jul '24

18 Jul '24

Hi Everyone, I had an issue last night when I was bringing online some osds that I was rebuilding. When the osds created and came online 15pgs got stuck in activating. The first osd (osd.112) seemed to come online ok, but the second one (osd.113) triggered the issue. All the pgs in activating included osd.112 in the pg map and I resolved it by doing pg-upmap-items to map the pg back from osd.112 to where it currently was but it was painful having 10min of stuck i/o os an rbd pool with vm's running. Some details about the cluster: Pacific 16.2.15, upgraded from Nautilus fairly recently and Luminos back in the past. All osds were rebuilt on bluestore in Nautilus, as were the mons. The disks in question are Intel DC P4510 8TB nvme. I'm rebuilding them as I had previously had 4x2TB osd's per disk and now wanted to consolidate down to one osd per disk. There's around 300 osd's in the pool with 16384 pgs which means that the 2TB osds had 157pgs on them. However this means that the 8TB osds have 615pgs on them and I'm wondering if this is maybe the cause of the problem. There are no warnings about too many pgs per osd in the logs or ceph status. I have the default value of 250 for mon_max_pg_per_osd and default value of 3.0 for osd_max_pg_per_osd_hard_ratio. My plan is to reduce the number of pgs in the pool but I want to understand and prove what happened here. Is it likely I've hit pg overdose protection? If I have, how would I tell as I can't see anything in the cluster logs. Thanks, Rich

2 1

Heads up: New Ceph images require x86-64-v2 and possibly a qemu config change for virtual servers
by Dan van der Ster 18 Jul '24

18 Jul '24

Hey all, The upcoming community Ceph container images will be based on CentOS 9. In our Clyso CI testing lab we learned that el9-based images won't run on some (default) qemu VMs. Where our el8-based images run well, our new el9-based images get: error during bootstrap: Fatal glibc error: CPU does not support x86-64-v2 This is because RHEL 9 (and clones) added a hard requirement on x86-64-v2: * https://srobb.net/rhel9.html * https://forums.rockylinux.org/t/el9-will-require-x86-64-v2-support/5311 * https://developers.redhat.com/blog/2021/01/05/building-red-hat-enterprise-l… * https://access.redhat.com/solutions/6833751 The reason for the failure in our testing is that some of our virtual nodes use qemu's default CPU emulation (qemu64), which does not support x86-64-v2. The solution for these nodes is to switch to "host" or something else listed at https://www.qemu.org/docs/master/system/i386/cpu.html I would encourage cephadm/Rook users with daemons on VM hosts to verify their CPU modes on their VMs prior to any upgrades, and make the necessary changes to the hypervisor configuration ahead of time. Best, Dan -- Dan van der Ster CTO Clyso GmbH p: +49 89 215252722 | a: Vancouver, Canada w: https://clyso.com | e: dan.vanderster(a)clyso.com

3 2

RBD images can't be mapped anymore
by Daniele Rimoldi 18 Jul '24

18 Jul '24

Dear All, in the last few days we've been facing a strange problem with RBD mapping in our 5 host cluster. The cluster is running since 12 months and 2 weeks ago was updated from Quincy to Reef with no problems. Saturday, we decided to shut-down one of the 5 nodes di insert a test NVME drive, we have failure domain set at host level. Before this operation OSD flags were set (nodown, noout, norebalance... etc). With one node down, the cluster continued working correctly with one exception. Many RBD images mapped on various clients stopped working. This happened across various type of clients so both our external Proxmox cluster and our windows machines lost these RBD mapped devices. After bringing back node 5 to the cluster, the problem is still present. To make things even stranger, cluster is in "HEALTH OK" state and there are no apparent issues on OSDs. We then noticed that not all RBD images were lost, but only those created in pools placed on HDD class devices. There are a number rbd bench --io-type write test_hdd --pool=testhddof images in SSD class pools that are not affected by the problem, so we momentary moved (cloned) most important images to SSD pools to have them back and working. Now I'm seeking community help on how to investigate the problem. We did create a new HDD class pool with a new image for test purposes, but it can't be mapped. In a few tests, mapping in windows succeded but then, the device was immediately removed because of a stale connection and lack of communication with the device. Testing from a cluster node with: rbd bench --io-type write test_hdd --pool=testhdd works perfectly so OSDs and cluster seems to be fine... we are quite lost at this. Any suggestion on what to check? Thanks in advance for your help! Regards, Daniele

2 1

Re: Unable to mount with 18.2.2
by David C. 18 Jul '24

18 Jul '24

Albert, The network is ok. However, strangely, the osd and mds did not activate msgr v2 (msgr v2 was activated on mon). It is possible to bypass by adding the "ms_mode=legacy" option but you need to find out why msgr v2 is not activated Le mar. 16 juil. 2024 à 15:18, Albert Shih <Albert.Shih(a)obspm.fr> a écrit : > Le 16/07/2024 à 15:04:05+0200, David C. a écrit > Hi, > > > > > I think it's related to your network change. > > I though about it but in that case why the old (and before upgrade) server > works ? > > > Can you send me the return of "ceph report" ? > > Nothing related to the old subnet. (see attach file) > > Regards. > > JAS > -- > Albert SHIH 🦫 🐸 > Observatoire de Paris > France > Heure locale/Local time: > mar. 16 juil. 2024 15:14:21 CEST >

4 8

Large amount of empty objects in unused cephfs data pool
by Petr Bena 18 Jul '24

18 Jul '24

I created a cephfs using mgr dashboard, which created two pools: cephfs.fs.meta and cephfs.fs.data We are using custom provisioning for user defined volumes (users provide yaml manifests with definition of what they want) which creates dedicated data pools for them, so cephfs.fs.data is never used for anything, it's literally empty despite dashboard reporting about 120kb in there (probably metadata files from new subvolume API that we are using). I wanted to decrease number of PGs in that unused cephfs.fs.data pool and to my surprise I received warning about uneved object distribution (too many objects per PG). So I dived deeper and figured out that there are in fact many objects in seemingly empty cephfs.fs.data: PROD [root@ceph-drc-mgmt ~]# rados -p cephfs.fs.data ls | head 100002a3b4a.00000000 100001a1876.00000000 100001265b5.00000000 100002af216.00000000 100004e07ec.00000000 1000053a31f.00000000 10000455214.00000000 100003e4c36.00000000 1000049e91a.00000000 100005d0bc7.00000000 When I tried dumping any of those objects - they are empty, 0 bytes each of them. But there is over 7 millions of them: PROD [root@ceph-drc-mgmt ~]# rados -p cephfs.fs.data ls | wc -l 7260394 Why is that unused pool containing 7 million empty objects? Is that some kind of bug in MDS? It's 18.2.2 Thanks

2 1

Mount cephfs
by filip Mutterer 18 Jul '24

18 Jul '24

In the official guide I found this example to mount Cephfs: mount -t ceph name@.fs_name=/ /mnt/mycephfs -o mon_addr=1.2.3.4 How to adapt to part "name@.fs_name=/" to my setup. Which command do I have to run in cephadm shell to pick up the details?

2 1

Failing update check when using quay.io mirror
by Marianne Spiller 17 Jul '24

17 Jul '24

Hi, I'm about setting up a Ceph stage cluster using `cephadm` and `podman`. As I'm not allowed to connect to the outside, our Artifactory keeps the needed quay.io packages mirrored. What I did: * put a config `/etc/containers/registries.conf.d` for `prefix=quay.io` on the first node * created an Artifactory user with permissions for the quay.io packages * bootstrapped the cluster using `registry-url`, `registry-username` and `registry-password` An `/etc/ceph/podman-auth.json` got created; the bootstrapped cluster is up & running (reef, 18.2.2). But the "New version" check does not work, and it floods the logs with messages. **Lots** of messages. **Of course**, a direct connection to quay.io will lead to a `connection refused` - that's why it should use the mirror. So: why does the update check ignore my mirror settings? And what can I do to make it work? Thanks a lot for any kind of an idea, Marianne ``` 7/17/24 7:45:47 PM [ERR] Cannot get tags from url 'https://quay.io/v2/ceph/ceph/tags/list': HTTPSConnectionPool(host='quay.io', port=443): Max retries exceeded with url: /v2/ceph/ceph/tags/list (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused',)) Traceback (most recent call last): File "/lib/python3.6/site-packages/urllib3/connection.py", line 162, in _new_conn (self._dns_host, self.port), self.timeout, **extra_kw) File "/lib/python3.6/site-packages/urllib3/util/connection.py", line 80, in create_connection raise err File "/lib/python3.6/site-packages/urllib3/util/connection.py", line 70, in create_connection sock.connect(sa) ConnectionRefusedError: [Errno 111] Connection refused During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen chunked=chunked) File "/lib/python3.6/site-packages/urllib3/connectionpool.py", line 344, in _make_request self._validate_conn(conn) File "/lib/python3.6/site-packages/urllib3/connectionpool.py", line 844, in _validate_conn conn.connect() File "/lib/python3.6/site-packages/urllib3/connection.py", line 315, in connect conn = self._new_conn() File "/lib/python3.6/site-packages/urllib3/connection.py", line 171, in _new_conn self, "Failed to establish a new connection: %s" % e) urllib3.exceptions.NewConnectionError: : Failed to establish a new connection: [Errno 111] Connection refused During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/lib/python3.6/site-packages/requests/adapters.py", line 449, in send timeout=timeout File "/lib/python3.6/site-packages/urllib3/connectionpool.py", line 639, in urlopen _stacktrace=sys.exc_info()[2]) File "/lib/python3.6/site-packages/urllib3/util/retry.py", line 399, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='quay.io', port=443): Max retries exceeded with url: /v2/ceph/ceph/tags/list (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused',)) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/registry.py", line 43, in get_tags r = requests.get(url, headers=headers) File "/lib/python3.6/site-packages/requests/api.py", line 75, in get return request('get', url, params=params, **kwargs) File "/lib/python3.6/site-packages/requests/api.py", line 60, in request return session.request(method=method, url=url, **kwargs) File "/lib/python3.6/site-packages/requests/sessions.py", line 535, in request resp = self.send(prep, **send_kwargs) File "/lib/python3.6/site-packages/requests/sessions.py", line 648, in send r = adapter.send(request, **kwargs) File "/lib/python3.6/site-packages/requests/adapters.py", line 516, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPSConnectionPool(host='quay.io', port=443): Max retries exceeded with url: /v2/ceph/ceph/tags/list (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused',)) The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/upgrade.py", line 283, in upgrade_ls ls = reg.get_tags(bare_image) File "/usr/share/ceph/mgr/cephadm/registry.py", line 46, in get_tags raise ValueError(msg) from e ValueError: Cannot get tags from url 'https://quay.io/v2/ceph/ceph/tags/list': HTTPSConnectionPool(host='quay.io', port=443): Max retries exceeded with url: /v2/ceph/ceph/tags/list (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused',)) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 137, in wrapper return OrchResult(f(*args, **kwargs)) File "/usr/share/ceph/mgr/cephadm/module.py", line 3251, in upgrade_ls return self.upgrade.upgrade_ls(image, tags, show_all_versions) File "/usr/share/ceph/mgr/cephadm/upgrade.py", line 285, in upgrade_ls raise OrchestratorError(f'{e}') orchestrator._interface.OrchestratorError: Cannot get tags from url 'https://quay.io/v2/ceph/ceph/tags/list': HTTPSConnectionPool(host='quay.io', port=443): Max retries exceeded with url: /v2/ceph/ceph/tags/list (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused',)) ```

2 1

Multisite stuck data shard recovery after bucket deletion
by Adam Prycki 17 Jul '24

17 Jul '24

Hello, I'm testing multisite sync on reef 18.2.2, cephadm and ubuntu 22.04. Right now I'm testing symmetrical sync policy making backup to read-only zone. My sync policy allows for replication and I enable replication via put-bucket-replication. My multisite setup fails at seemingly basic operation. My test looks like this: 1. create bucket 2. upload some data to bucket 3. wait for replication to copy some of the data 4. run `rclone purge` on the bucket in master zone while replication is in progress. All data and bucket itself are deleted. I've tested this on normal secondary zone and archive zone. It seems that bucket is deleted so quickly that replication gets stuck. Buckets are gone from both zones but data sync shard still tries to replicate them Example of a recovering shard. { "shard_id": 100, "marker": { "status": "full-sync", "marker": "", "next_step_marker": "", "total_entries": 0, "pos": 0, "timestamp": "0.000000" }, "pending_buckets": [ "bucket6:58642236-4f66-46f5-b863-1d6a8667c4c3.61059.5:9", "bucket6:58642236-4f66-46f5-b863-1d6a8667c4c3.61059.7:9" ], "recovering_buckets": [ "bucket6:58642236-4f66-46f5-b863-1d6a8667c4c3.61059.7:9[0]" ], "current_time": "2024-07-17T13:23:11Z" } In this case there are 2 pending buckets because I've reused the bucket name. The only semi-automatic solution I've found is to recreate bucket with the same name and wait for recovering shards to disappear. Is there any way to make ceph clean up these stuck shards automatically? Best regards Adam Prycki

1 0

Separated multisite sync and user traffic, doable?
by Szabo, Istvan (Agoda) 17 Jul '24

17 Jul '24

Hi, Could that cause any issue if the endpoints defined in the zonegroups are not in the endpoint list behind haproxy? The question is mainly about the role of the endpoint servers in the zonegroup list. Their role is the sync only or something else also? This would be the scenario, could it work? * * I have 3 mon/mgr server and 15 OSD * RGWs on the mon/mgr would be in the zonegroup definition like this "zones": [ { "id": "61c9sdf40-fdsd-4sdd-9rty9-ed56jda41817", "name": "dc", "endpoints": [ "http://mon1:8080", "http://mon2:8080", "http://mon3:8080" ], * However for user traffic I'd use an haproxy endpoint with the 15 OSD node rgws (each osd node 1x). Ty ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 1

RGW Multisite with a Self-Signed CA
by Alex Hussein-Kershaw (HE/HIM) 17 Jul '24

17 Jul '24

Hi ceph-users! I'm going through the process of migrating to use cephadm for my clusters. Previously I used ceph-ansible. My question is essentially "How can I configure RGW multisite with self-signed certificates with cephadm?". I have prototyped the migration and redeployed RGWs. Everything on my adopted site is running the latest version of Reef (18.2.2). The remote site is using Octopus as historically deployed with ceph-ansible. My RGWs on both sites are up and I can make requests to them, but they are failing to sync, and "radosgw-admin sync status" shows a generic input/output error. Taking some network capture I can see that the TLS handshake is failing with "Unknown CA", so it looks like the RGWs don't trust my self-signed certificate, I suppose that's not a surprise. However, I can't work out how to establish the trust. I've tried mounting in the /etc/pki directory from the machine it's running on into the RGW containers, which does contain the self-signed CA, but I still see errors in my multi-site sync. I did notice that I can curl from within the containers successfully to the remote HTTPS RGWs after this though, so it did do something. Where do the RGWs infer which CAs to trust from? I should also mention that when stepping all the RGWs down to HTTP the sync works with no issues (I was nervous about the Reef to Octopus pairing, but it seems fine). Kind regards, Alex

2 4

Scheduled cluster maintenance tasks
by Szabo, Istvan (Agoda) 17 Jul '24

17 Jul '24

Hi, Trying to figure out what is the trigger time for scrubbing and garbage collection but in the config related to these operations are not straightforward. I'm still looking for my daily laggy pg with slow ops around the same time and what I've found that some cleaning triggered around that time, however due to scrubbing running 0-24 based on config, GC running in theory hourly, lifecycle I'm not sure the schedule either, don't know what is the exact task. Haven't really found proper metrics about these jobs either. What I've tried is Frank S scrub report but that one no info about schedules, it has information about intervals how many scrubbed etc ... So if there is some proper documentation about it, please share with me. Thank you in advance. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0

Node Exporter keep failing while upgrading cluster in Air-gapped ( isolated environment ).
by Saif Mohammad 16 Jul '24

16 Jul '24

Hello, We are facing an issue with node-exporter entering an error state while upgrading our cluster in an air-gapped environment. Specifically, we are upgrading from quincyv17.2.0 to reefv18.2.2. To facilitate this upgrade, we have set up a custom repository on a separate machine within the same network and pushed required images to this private repository. Here are the images that we have pushed: root@custom-registry:~# docker images REPOSITORY TAG IMAGE ID CREATED SIZE 192.168.1.10:5000/ceph/ceph v18.2.2 3c937764e6f5 7 weeks ago 1.25GB 192.168.1.10:5000/ceph/ceph-grafana 9.4.7 954c08fa6188 7 months ago 633MB 192.168.1.10:5000/prometheus/prometheus v2.43.0 a07b618ecd1d 16 months ago 234MB 192.168.1.10:5000/prometheus/alertmanager v0.25.0 c8568f914cd2 19 months ago 65.1MB 192.168.1.10:5000/prometheus/node-exporter v1.5.0 0da6a335fe13 19 months ago 22.5MB Since we have configured insecure private registry, we added following lines in "/etc/containers/registries.conf" file on each node of the cluster. [[registry]] location = "192.168.1.10:5000" insecure = true Ceph image upgradation done by the command (ceph orch upgrade start --image 192.168.1.10:5000/ceph/ceph:v18.2.2) but we encountered issues from the node-exporter image, which fails to start and remains in an error state. For the remaining images ( except ceph ) we used "ceph-config" command. We are here now getting the new images values by executing the following command that seems to be default ones. root@node-01:~# ceph config get mgr mgr/cephadm/container_image_node_exporter quay.io/prometheus/node-exporter:v1.5.0 Does an internet connection is required to pull these images (Monitoring stack components images)? I am unsure because whenever we attempt to run the following command, node-exporter enters an error state and not coming into the running state even after redeploy. root@node-01:~# ceph config set mgr mgr/cephadm/container_image_node_exporter quay.io/prometheus/node-exporter:v1.5.0 Please advise the best approach to upgrade the cluster including all images in Air-gapped environment? Any guidance on resolving this issue would be appreciated. Regards, Mohammad Saif

2 3

Schödinger's OSD
by Tim Holloway 16 Jul '24

16 Jul '24

Incidentally, I just noticed that my phantom host isn't completely gone. It's not in the host list, either command-line or dashboard, but it does list (with no assets) as a host under "ceph osd tree". --- More seriously, I've been having problems with OSDs that report as being both up and down at the same time. This is on 2 new hosts. One host saw this when I made it the _admin host. The other caught it because it's running in a VM with the OSD mapped out as an imported disk and the host os managed to flip which drive was sda and which was sbd, resulting in having to delete and re- define the OSD in the VM. But now the OSD on this VM reports as "UP/IN" on the dashboard while it's "error" on "ceph orch ps" and on the actual vbox, the osd container fails on startup. viz: ul 12 20:06:48 ceph05.internal.mousetech.com ceph-278fcd86-0861-11ee- a7df-9c5c8e86cf8f-osd-4[4017]: debug 2024-07-12T20:06:48.056+0000 7fc17dfb9380 -1 bdev(0x55e4853c4800 /var/lib/ceph/osd/ceph-4/block) open open got: (16) De> Jul 12 20:06:48 ceph05.internal.mousetech.com ceph-278fcd86-0861-11ee- a7df-9c5c8e86cf8f-osd-4[4017]: debug 2024-07-12T20:06:48.056+0000 7fc17dfb9380 -1 osd.4 0 OSD:init: unable to mount object store Jul 12 20:06:48 ceph05.internal.mousetech.com ceph-278fcd86-0861-11ee- a7df-9c5c8e86cf8f-osd-4[4017]: debug 2024-07-12T20:06:48.056+0000 7fc17dfb9380 -1 ** ERROR: osd init failed: (16) Device or resource busy Note that truncated message above reads bdev(0x55e4853c4800 /var/lib/ceph/osd/ceph-4/block) open open got: (16) Device or resource busy Rebooting doesn't help. Nor does freeing us resources and stopping/starting processes manually. The problem eventually cleared up spontaneously on the admin box, but I have no idea why. --- Also noted that now the OSD on the admin box shows in ceph orch ps as "stopped", though again, the dashboard lists it as "UP/IN". Here's what systemctl thinks about it: systemctl status ceph-osd(a)5.service ● ceph-osd(a)5.service - Ceph object storage daemon osd.5 Loaded: loaded (/usr/lib/systemd/system/ceph-osd@.service; enabled-runtime; preset: disabled) Active: active (running) since Fri 2024-07-12 16:45:51 EDT; 1min 40s ago Process: 8511 ExecStartPre=/usr/libexec/ceph/ceph-osd-prestart.sh - -cluster ${CLUSTER} --id 5 (code=exited, status=0/SUCCESS) Main PID: 8517 (ceph-osd) Tasks: 70 Memory: 478.6M CPU: 3.405s CGroup: /system.slice/system-ceph\x2dosd.slice/ceph-osd(a)5.service └─8517 /usr/bin/ceph-osd -f --cluster ceph --id 5 -- setuser ceph --setgroup ceph Jul 12 16:45:51 dell02.mousetech.com systemd[1]: Starting Ceph object storage daemon osd.5... Jul 12 16:45:51 dell02.mousetech.com systemd[1]: Started Ceph object storage daemon osd.5. Jul 12 16:45:51 dell02.mousetech.com ceph-osd[8517]: 2024-07- 12T16:45:51.642-0400 7f2bd440c140 -1 Falling back to public interface Jul 12 16:45:58 dell02.mousetech.com ceph-osd[8517]: 2024-07- 12T16:45:58.352-0400 7f2bd440c140 -1 osd.5 34161 log_to_monitors {default=true} Jul 12 16:45:59 dell02.mousetech.com ceph-osd[8517]: 2024-07- 12T16:45:59.206-0400 7f2bcbbf0640 -1 osd.5 34161 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory The actual container is not running. Ceph version, incidentally, is 16.2.15. Except for that one node that apparently didn't move up from Octupus (I'll be nuking that one shortly).

3 11

Unable to mount with 18.2.2
by Albert Shih 16 Jul '24

16 Jul '24

Hi everyone My cluster ceph run currently 18.2.2 and ceph -s say everything are OK root@cthulhu1:/var/lib/ceph/crash# ceph -s cluster: id: 9c5bb196-c212-11ee-84f3-c3f2beae892d health: HEALTH_OK services: mon: 5 daemons, quorum cthulhu1,cthulhu5,cthulhu3,cthulhu4,cthulhu2 (age 4d) mgr: cthulhu1.yhgean(active, since 4d), standbys: cthulhu3.ylmosn, cthulhu5.hqiarz, cthulhu4.odtqjw, cthulhu2.ynvnob mds: 1/1 daemons up, 4 standby osd: 370 osds: 370 up (since 4d), 370 in (since 3M) data: volumes: 1/1 healthy pools: 4 pools, 259 pgs objects: 333.68M objects, 279 TiB usage: 423 TiB used, 5.3 PiB / 5.8 PiB avail pgs: 226 active+clean 19 active+clean+scrubbing+deep 14 active+clean+scrubbing I got 3 clients cephfs : 2 with debian 12 + 18.2.2 1 with Debian 11 + 17.2.7 The Debian 11 client work fine, I try to umount the cephfs and remount it and it's working The first Debian 12 + 18.2.2 are a upgrade from Debian 11 + 17.2.7 and before the update the mount was working, after the upgrade I'm unable to mount the cephfs The second Debian 12 is a fresh install and I'm also unable to mount the cephfs. I check the network and don't see any firewall problem. On the client when I try mount they take few minutes to answer me mount error: no mds server is up or the cluster is laggy on the client I can see : Jul 16 14:10:43 Debian12-1 kernel: [ 860.636012] ceph: corrupt mdsmap Jul 16 14:23:37 Debian12-2 kernel: [11497.406652] ceph: corrupt mdsmap I try to google this error but all I can find are the situation when ceph -s say it's in big trouble. On my case not only ceph -s say everything are ok but the Debian 11 are able to umount/mount/umount/mount/write Any clue or debugging method ? Regards -- Albert SHIH 🦫 🐸 Observatoire de Paris France Heure locale/Local time: mar. 16 juil. 2024 14:26:47 CEST

3 3

Large omap in index pool even if properly sharded and not "OVER"
by Szabo, Istvan (Agoda) 16 Jul '24

16 Jul '24

Hi, I have a pretty big bucket which sharded with 1999 shard so in theory can hold close to 200m objects (199.900.000). Currently it has 54m objects. Bucket limit check looks also good: "bucket": ""xyz, "tenant": "", "num_objects": 53619489, "num_shards": 1999, "objects_per_shard": 26823, "fill_status": "OK" This is the bucket id: "id": "9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1" This is the log lines: 2024-06-27T10:41:05.679870+0700 osd.261 (osd.261) 9643 : cluster [WRN] Large omap object found. Object: 26:e433e65c:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.151:head PG: 26.3a67cc27 (26.427) Key count: 236919 Size (bytes): 89969920 2024-06-27T10:43:35.557835+0700 osd.89 (osd.89) 9000 : cluster [WRN] Large omap object found. Object: 26:31ff4df1:::.dir.9213182a-14ba-48ad-bde9-289a1c0c0de8.2479481907.1.726:head PG: 26.8fb2ff8c (26.78c) Key count: 236495 Size (bytes): 95560458 Tried to deep scrub the affected pgs, tried to deep-scrub the mentioned osds in the log but didn't help. Why? What I'm missing? Thank you in advance for your help. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

5 14

How to detect condition for offline compaction of RocksDB?
by Александр Руденко 16 Jul '24

16 Jul '24

Hi, We have a big Ceph cluster (RGW case) with a lot of big buckets 10-500M objects with 31-1024 shards and a lot of io generated by many clients. Index pool placed on enterprise SSDs. We have about 120 SSDs (replication 3) and about 90Gb of OMAP data on each drive. About 75 PGs on each SSD for now. I think 75 is not enough for this amount of data, but I’m not sure that it is critical in our case. The problem: For the last few weeks, we can see big degradation of some SSD OSDs. We can see a lot of Ops (150-256 in perf dump) for a long time and we can see high avg ops time like 1-3 seconds (this metric is based on dump_historic_ops_by_duration and our avg calculation). These numbers are very unusual for our deployment. And it makes a big impact on our customers. For now, we make Offline compaction of ‘degraded’ OSDs and after compaction, we see that all PGs of this OSD are returned on this OSD (because recover is disabled during compaction) and we see that OSD works perfect for some time, all our metrics are dropped down for few weeks or only few days… And, I think the problem is in the rocksdb database which grows by levels and slows down requests. And I have few question about compaction: 1. Why can't automatic compaction manage this on its own? 2. How can I see RocksDB levels usage or some program metric which can be used as a condition for manual compacting? Because our metrics like request latency, OSD Ops count, OSD avg slow ops time are not 100% relative to rocksdb internal state. We try to use perf dump, bluestore.kv_xxx_lat metrics, but I think they are absolutely useless. Because we can see higher values after compaction and restart OSD then these metrics were before compaction, lol. We try to see OSD logs with debug_rocksdb=4, but we can’t understand this output compare it between good and bad OSDs: Uptime(secs): 3610809.5 total, 600.0 interval Flush(GB): cumulative 50.519, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 475.34 GB write, 0.13 MB/s write, 453.07 GB read, 0.13 MB/s read, 3346.3 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [p-0] ** ** Compaction Stats [p-1] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 0/0 0.00 KB 0.0 0.0 0.0 0.0 52.0 52.0 0.0 1.0 0.0 155.2 343.25 176.27 746 0.460 0 0 L1 4/0 204.00 MB 0.8 96.7 52.0 44.7 96.2 51.4 0.0 1.8 164.6 163.6 601.87 316.34 237 2.540 275M 1237K L2 34/0 2.09 GB 1.0 176.8 43.6 133.2 175.3 42.1 7.6 4.0 147.6 146.3 1226.72 590.20 535 2.293 519M 2672K L3 79/0 4.76 GB 0.3 163.6 38.9 124.7 140.7 16.1 8.8 3.6 152.6 131.3 1097.53 443.60 432 2.541 401M 79M L4 316/0 19.96 GB 0.1 0.9 0.4 0.5 0.8 0.3 19.7 2.0 151.1 131.1 5.94 2.53 4 1.485 2349K 385K Sum 433/0 27.01 GB 0.0 438.0 134.9 303.0 465.0 161.9 36.1 8.9 136.9 145.4 3275.31 1528.94 1954 1.676 1198M 83M Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 and this log is not usable as ‘metric’. Our Ceph version is: 16.2.13 and we use default bluestore_rocksdb_options.

1 0

Apt update failing?
by Daniel Brown 15 Jul '24

15 Jul '24

1 0

when calling the CreateTopic operation: Unknown
by farhad kh 15 Jul '24

15 Jul '24

hi, i want to use ceph bucket notification . i try to created topic with below command but get error when used kafka with user/password how can i solved this problem ? my syntax have any problem? https://www.ibm.com/docs/en/storage-ceph/7?topic=management-creating-bucket… https://docs.ceph.com/en/latest/radosgw/notifications/ # aws sns create-topic --name storage --profile=test-notifications --endpoint-url=http://192.168.115.59 --attributes=file://topic.json --debug ##### topic.json: {"push-endpoint": "kafka://usr-test:p@ssw#0rd@kafka.test:9092","verify-ssl": "False", "kafka-ack-level": "broker", "persistent":"true"} ##### error is :---------------- content-type:application/x-www-form-urlencoded; charset=utf-8 host:192.168.115.59 x-amz-date:20240712T120700Z content-type;host;x-amz-date 6f328bc5b0736f23ae2cdf68ccffe1a45c705dd1636f61b999350ae18f8d5ad1 2024-07-12 12:07:00,439 - MainThread - botocore.auth - DEBUG - StringToSign: AWS4-HMAC-SHA256 20240712T120700Z 20240712/podspace/sns/aws4_request 38b7d8721abdd98c214ea763d9dcc324fcbc5982990353140f6b734455e9c01e 2024-07-12 12:07:00,439 - MainThread - botocore.auth - DEBUG - Signature: 99a0f7d40bc011c09dfe7c6aed4bae7f739df52278e23bd4f16fdb1888c001b7 2024-07-12 12:07:00,440 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url= http://192.168.115.59/, headers={'Content-Type': b'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': b'aws-cli/1.18.156 Python/3.6.8 Linux/5.4.17-2136.326.6.el8uek.x86_64 botocore/1.18.15', 'X-Amz-Date': b'20240712T120700Z', 'Authorization': b'AWS4-HMAC-SHA256 Credential=2MRG0TXJFISQAIVE47F1/20240712/podspace/sns/aws4_request, SignedHeaders=content-type;host;x-amz-date, Signature=99a0f7d40bc011c09dfe7c6aed4bae7f739df52278e23bd4f16fdb1888c001b7', 'Content-Length': '371'}> 2024-07-12 12:07:00,441 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTP connection (1): 192.168.115.59:80 2024-07-12 12:07:00,446 - MainThread - urllib3.connectionpool - DEBUG - http://192.168.115.59:80 "POST / HTTP/1.1" 400 206 2024-07-12 12:07:00,447 - MainThread - botocore.parsers - DEBUG - Response headers: {'content-length': '206', 'x-amz-request-id': 'tx00000c7a58038e67798c0-0066911c64-19910-site1', 'accept-ranges': 'bytes', 'content-type': 'application/xml', 'date': 'Fri, 12 Jul 2024 12:07:00 GMT'} 2024-07-12 12:07:00,447 - MainThread - botocore.parsers - DEBUG - Response body: b'<?xml version="1.0" encoding="UTF-8"?><Error><Code>InvalidArgument</Code><Message></Message><RequestId>tx00000c7a58038e67798c0-0066911c64-19910-site1</RequestId><HostId>19910-site1-podspace</HostId></Error>' 2024-07-12 12:07:00,448 - MainThread - botocore.hooks - DEBUG - Event needs-retry.sns.CreateTopic: calling handler <botocore.retryhandler.RetryHandler object at 0x7f35e0a7f860> 2024-07-12 12:07:00,448 - MainThread - botocore.retryhandler - DEBUG - No retry needed. 2024-07-12 12:07:00,449 - MainThread - awscli.clidriver - DEBUG - Exception caught in main() Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/awscli/clidriver.py", line 217, in main return command_table[parsed_args.command](remaining, parsed_args) File "/usr/lib/python3.6/site-packages/awscli/clidriver.py", line 358, in __call__ return command_table[parsed_args.operation](remaining, parsed_globals) File "/usr/lib/python3.6/site-packages/awscli/clidriver.py", line 530, in __call__ call_parameters, parsed_globals) File "/usr/lib/python3.6/site-packages/awscli/clidriver.py", line 650, in invoke client, operation_name, parameters, parsed_globals) File "/usr/lib/python3.6/site-packages/awscli/clidriver.py", line 662, in _make_client_call **parameters) File "/usr/lib/python3.6/site-packages/botocore/client.py", line 357, in _api_call return self._make_api_call(operation_name, kwargs) File "/usr/lib/python3.6/site-packages/botocore/client.py", line 676, in _make_api_call raise error_class(parsed_response, operation_name) botocore.exceptions.ClientError: An error occurred (Unknown) when calling the CreateTopic operation: Unknown 2024-07-12 12:07:00,449 - MainThread - awscli.clidriver - DEBUG - Exiting with rc 255 An error occurred (Unknown) when calling the CreateTopic operation: Unknown

2 1

Re: CephFS metadata pool size
by Lars Köppel 15 Jul '24

15 Jul '24

Hello everyone, short update to this problem. The zapped OSD is rebuilt and it has now 1.9 TiB (the expected size ~50%). The other 2 OSDs are now at 2.8 respectively 3.2 TiB. They jumped up and down a lot but the higher one has now also reached 'nearfull' status. How is this possible? What is going on? Does anyone have a solution how to fix this without zapping the OSD? Best regards, Lars [image: ariadne.ai Logo] Lars Köppel Developer Email: lars.koeppel(a)ariadne.ai Phone: +49 6221 5993580 <+4962215993580> ariadne.ai (Germany) GmbH Häusserstraße 3, 69115 Heidelberg Amtsgericht Mannheim, HRB 744040 Geschäftsführer: Dr. Fabian Svara https://ariadne.ai

5 19

Ceph osd df including block.db size
by Dyweni - Ceph-Users 14 Jul '24

14 Jul '24

Hi, I realize this comes from an end-of-life release (Mimic), but I believe it still applies to current releases. I've just upgraded from Luminous to Mimic and noticed that my 'ceph osd df' output now includes the block.db sizes. This is pushing all OSD's into nearfull status (~70% used in Luminous, ~85% used in Mimic). Each OSD is 1TB HDD (block) and 1TB NVME (block.db). I'd like to understand two things: 1. Why is the reasoning behind including the block.db size/usage in 'ceph osd df'? 2. Is there a way to exclude block.db size/usage from 'ceph osd df'? I've looked through the ceph git comments and noticed the following: * b97f3d6761208a1d317db3d0dba8371181e0b49f (8/21/2016, v11.0.0/master), Sage added: + // include dedicated db, too, if that isn't the shared device. + if (bluefs_shared_bdev != BlueFS::BDEV_DB) { + buf->available += bluefs->get_free(BlueFS::BDEV_DB); + buf->total += bluefs->get_total(BlueFS::BDEV_DB); + } * f798b13a043523e1677abdcb3df0d9ced8774098 (1/31/2018, v13.0.1/master), Igor removed: - // include dedicated db, too, if that isn't the shared device. - if (bluefs_shared_bdev != BlueFS::BDEV_DB) { - buf->total += bluefs->get_total(BlueFS::BDEV_DB); - } * 266d1649cf1d57ab32a71a39347bb418a4490c78 (2/20/2018, v13.0.1/master), Sage re-added: + if (bluefs) { + // include dedicated db, too, if that isn't the shared device. + if (bluefs_shared_bdev != BlueFS::BDEV_DB) { + buf->total += bluefs->get_total(BlueFS::BDEV_DB); } I've checked the main branch (as of right now, commit d622325c17bb98672ad22e60a372becfbeb04cff) and the calculation still exists: File: src/os/bluestore/BlueStore.cc Line: 11480 // include dedicated db, too, if that isn't the shared device. if (bluefs_layout.shared_bdev != BlueFS::BDEV_DB) { buf->total += bluefs->get_total(BlueFS::BDEV_DB); } Thanks, Dyweni

1 0

ceph orch osd rm --zap --replace leaves cluster in odd state
by Matthew Vernon 14 Jul '24

14 Jul '24

Hi, I want to prepare a failed disk for replacement. I did: ceph orch osd rm 35 --zap --replace and it's now in the state "Done, waiting for purge", with 0 pgs, and REPLACE and ZAP set to true. It's been like this for some hours, and now my cluster is unhappy: [WRN] CEPHADM_STRAY_DAEMON: 1 stray daemon(s) not managed by cephadm stray daemon osd.35 on host moss-be1002 not managed by cephadm (the OSD is down & out) ...and also neither the disk nor the relevant NVME LV has been zapped. I have my OSDs deployed via a spec: service_type: osd service_id: rrd_single_NVMe placement: label: "NVMe" spec: data_devices: rotational: 1 db_devices: model: "NVMe" And before issuing the ceph orch osd rm I set that to be unmanaged (ceph orch set-unmanaged osd.rrd_single_NVMe), as obviously I don't want ceph to just try and re-make a new OSD on the sad disk. I'd expected from the docs[0] that what I did would leave me with a system ready for the failed disk to be swapped (and that I could then mark osd.rrd_single_NVMe as managed again, and a new OSD built), including removing/wiping the NVME lv so it can be removed. What did I do wrong? I don't much care about the OSD id (but obviously it's neater to not just incrementally increase OSD numbers every time a disk died), but I thought that telling ceph orch not to make new OSDs then using ceph orch osd rm to zap the disk and NVME lv would have been the way to go... Thanks, Matthew [0] https://docs.ceph.com/en/reef/cephadm/services/osd/#replacing-an-osd

3 3

Help needed please ! Filesystem became read-only !
by nbarbier＠deltaonline.net 14 Jul '24

14 Jul '24

Hello, First of all, thanks for reading my message. I set up a Ceph version 18.2.2 cluster with 4 nodes, everything went fine for a while, but after copying some files, the storage showed a warning status and the following message : "HEALTH_WARN: 1 MDSs are read only mds.PVE-CZ235007SH(mds.0): MDS in read-only mode". The logs are showing : Jun 03 08:20:41 PVE-CZ235007SH ceph-mds[1329868]: -9999> 2024-06-03T07:57:17.589+0200 77250fc006c0 -1 log_channel(cluster) log [ERR] : failed to store backtrace on ino 0x1000000039c object, pool 5, errno -2 Jun 03 08:20:41 PVE-CZ235007SH ceph-mds[1329868]: -9998> 2024-06-03T07:57:17.589+0200 77250fc006c0 -1 mds.0.189541 unhandled write error (2) No such file or directory, force readonly... After googling for a while, I did not find a hint to understand more precisely the root cause. Any help would we greatly appreciated, or even a link to post this request elsewhere if this is not the place to. Please find below additional details if needed. Thanks a lot ! Nicolas --- # ceph osd dump [...] pool 5 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 292 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 4.51 [...] # ceph osd lspools 1 .mgr 4 cephfs_data 5 cephfs_metadata 18 ec-pool-001-data 19 ec-pool-001-metadata # ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 633 TiB 633 TiB 61 GiB 61 GiB 0 TOTAL 633 TiB 633 TiB 61 GiB 61 GiB 0 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 119 MiB 31 357 MiB 0 200 TiB cephfs_data 4 32 71 KiB 8.38k 240 KiB 0 200 TiB cephfs_metadata 5 32 329 MiB 6.56k 987 MiB 0 200 TiB ec-pool-001-data 18 32 42 GiB 15.99k 56 GiB 0 451 TiB ec-pool-001-metadata 19 32 0 B 0 0 B 0 200 TiB # ceph status cluster: id: f16f53e1-7028-440f-bf48-f99912619c33 health: HEALTH_WARN 1 MDSs are read only services: mon: 4 daemons, quorum PVE-CZ235007SG,PVE-CZ2341016V,PVE-CZ235007SH,PVE-CZ2341016T (age 35h) mgr: PVE-CZ235007SG(active, since 2d), standbys: PVE-CZ235007SH, PVE-CZ2341016T, PVE-CZ2341016V mds: 1/1 daemons up, 3 standby osd: 48 osds: 48 up (since 2d), 48 in (since 3d) data: volumes: 1/1 healthy pools: 5 pools, 129 pgs objects: 30.97k objects, 42 GiB usage: 61 GiB used, 633 TiB / 633 TiB avail pgs: 129 active+clean

4 6

Questions about the usage of space in Ceph
by 2644294460＠qq.com 13 Jul '24

13 Jul '24

My Ceph cluster has a CephFS file system, using an erasure-code data pool (k=8, m=2), which has used 14TiB of space. My CephFS has 19 subvolumes, and each subvolume automatically creates a snapshot every day and keeps it for 3 days. The problem is that when I manually calculate the disk space usage of each subvolume directory in CephFS, the total amount is only 8.4TiB. I don't know why this is happening. Do snapshots take up a lot of space? ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) Thank you all!

2 1

Repurposing some Dell R750s for Ceph
by Drew Weaver 12 Jul '24

12 Jul '24

Hello, We would like to repurpose some Dell PowerEdge R750s for a Ceph cluster. Currently the servers have one H755N RAID controller for each 8 drives. (2 total) I have been asking their technical support what needs to happen in order for us to just rip out those raid controllers and cable the backplane directly to the motherboard/PCIe lanes and they haven't been super enthusiastic about helping me. I get it just buy another 50 servers, right? No big deal. I have the diagrams that show how each configuration should be connected, I think I just need the right cable(s), my question is has anyone done this work before and was it worth it? Also bonus if anyone has an R750 that has the drives directly connected to the backplane and can find the part number of the cable that connects the backplane to the motherboard I would greatly appreciate that part number. My sales guys are "having a hard time locating it". Thanks, -Drew

7 13

Help with Mirroring
by Dave Hall 12 Jul '24

12 Jul '24

Hello. I would like to use mirroring to facilitate migrating from an existing Nautilus cluster to a new cluster running Reef. RIght now I'm looking at RBD mirroring. I have studied the RBD Mirroring section of the documentation, but it is unclear to me which commands need to be issued on each cluster and, for commands that have both clusters as arguments, when to specify site-a where vs. site-b. Another concern: Both the old and new cluster internally have the default name 'Ceph' - when I set up the second cluster I saw no obvious reason to change from the default. If these will cause a problem with mirroring, is there a workaround? In the long run I will also be migrating a bunch of RGW data. If there are advantages to using mirroring for this I'd be glad to know. (BTW, the plan is to gradually decommission the systems from the old cluster and add them to the new cluster. In this context, I am looking to enable and disable mirroring on specific RBD images and RGW buckets as the client workload is migrated from accessing the old cluster to accessing the new. Thanks. -Dave -- Dave Hall Binghamton University kdhall(a)binghamton.edu

5 4

cephadm for Ubuntu 24.04
by Stefan Kooman 12 Jul '24

12 Jul '24

Hi, Is it possible to only build "cephadm", so not the other ceph packages / daemons? Or can we think about a way to have cephadm packages build for all supported mainstream linux releases during the supported lifetime of a Ceph release: i.e. debian, Ubuntu LTS, CentOS Stream? I went ahead and upgraded one of our (pre-prod) ceph nodes to Ubuntu 24.04 LTS. As it is managed by cephadm and running containers this should be no problem, right? Right. Except that there are is no "cephadm" package (or any other ceph package) for Ubuntu 24.04 from Ceph repository. So, then just build those packages yourself, right? Right. After some python pyyaml 6.0.0 / cython issues in monitoring/ceph-mixin/requirements-alerts.txt and requirements-lint.txt, fixed by bumping pyyaml to 6.0.1) and a hard python 3.10 requirement (I bumped that to 3.12) I ran into an issue with "arrow_ext" breaking the build process which I'm currently trying to figure out ... I know supporting Ceph packages for a bunch of distros is a lot of work, but having _just_ cephadm available on a wider range of platforms, would really help here. It helps avoid upgrading both Ceph and the OS at the same time. This allows the use of the latest (kernel) OS improvements, all while not touching Ceph. Thanks, Gr. Stefan

7 11

reef 18.2.3 QE validation status
by Yuri Weinstein 12 Jul '24

12 Jul '24

Details of this release are summarized here: https://tracker.ceph.com/issues/65393#note-1 Release Notes - TBD LRC upgrade - TBD Seeking approvals/reviews for: smoke - infra issues, still trying, Laura PTL rados - Radek, Laura approved? Travis? Nizamudeen? rgw - Casey approved? fs - Venky approved? orch - Adam King approved? krbd - Ilya approved powercycle - seems fs related, Venky, Brad PTL ceph-volume - will require https://github.com/ceph/ceph/pull/56857/commits/63fe3921638f1fb7fc065907a9e… Guillaume is fixing it. TIA

10 29

AssumeRoleWithWebIdentity in RGW with Azure AD
by Ryan Rempel 12 Jul '24

12 Jul '24

I'm trying to setup the OIDC provider for RGW so that I can have roles that can be assumed by people logging into their regular Azure AD identities. The client I'm planning to use is Cyberduck – it seems like one of the few GUI S3 clients that manages the OIDC login process in a way that could work for relatively naive users. I've gotten a fair ways down the road. I've been able to configure Cyberduck so that it performs the login with Azure AD, gets an identity token, and then sends it to Ceph to engage with the AssumeRoleWithWebIdentity process. However, I then get an error, which shows up in the Ceph rgw logs like this: 2024-07-08T17:18:09.749+0000 7fb2d7845700 0 req 15967124976712370684 1.284013867s sts:assume_role_web_identity Signature validation failed: evp verify final failed: 0 error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding I turned the logging for rgw up to 20 to see if I could follow along to see how much of the process succeeds and learn more about what fails. I can then see logging messages from this file in the source code: https://github.com/ceph/ceph/blob/08d7ff952d78d1bbda04d5ff7e3db1e733301072/… We get to WebTokenEngine::get_from_jwt, and it logs the JWT payload in a way that seems to be as expected. The logs then indicate that a request is sent to the /.well-known/openid-configuration endpoint that appears to be appropriate for the issuer of the JWT. The logs eventually indicate what looks like a successful and appropriate response to that. The logs then show that a request is sent to the jwks_uri that is indicated in the openid-configuration document. The response to that is logged, and it appears to be appropriate. We then get some logging starting with "Certificate is", so it looks like we're getting as far as WebTokenEngine::validate_signature. So, several things appear to have happened successfully – we've loading the OIDC provider that corresponds to the iss, and we've found a client ID that corresponds to what I registered when I configured things. (This is why I say we appear to be a fair ways down the road – a lot of this is working). It looks as though what's happening in the code now is that it's iterating through the certificates given in the jwks_uri content. There are 6 certificates listed, but the code only gets as far as the first one. Looking at the code, what appears to be happening is that, among the various certificates in the jwks_uri, it's finding the first one which matches a thumbprint registered with Ceph (that is, which I registered with Ceph). This must be succeeding (for the first certificate), because the "Signature validation failed" logging comes later. So, the code does verify that the thumbprint of the first certificate matches one of the thumbprints I registered with Ceph for this OIDC provider. We then get to a part of the code where it tries to verify the JWT using the certificate, with jwt::verify. Given what gets logged ("Signature validateion failed: ", this must be throwing an exception. The thing I find surprising about this is that there really isn't any reason to think that the first certificate listed in the jwks_uri content is going to be the certificate used to sign the JWT. If I understand JWT correctly, it's appropriate to sign the JWT with any of the certificates listed in the jwks_uri content. Furthermore, the JWT header includes a reference to the kid, so it's possible for Ceph to know exactly which certificate the JWT purports to be signed by. And, Ceph knows that there might be multiple thumbprints, because we can register 5. So, the logic of trying the first valid certificate in x5c and then stopping if it fails seems broken, actually. I suppose what I could do as a workaround is try to figure out whether Azure AD is consistently using the same kid to sign the JWTs for me, and then only register that thumbprint with Ceph. Then, Ceph would actually choose the correct certificate (as the others wouldn't match a thumbprint I registered). I may try this – in part, just to verify what I think is happening. But it would be awfully fragile – I don't believe there is any requirement in JWT to just use one of the certificates listed in x5c. An alternative would be to try rewriting the code to apply a different kind of logic. The way it ought to work (it seems to me) is something like this: * Get the openid_configuration, and get the jwks stuff from the jwks_uri (which Ceph does already). * Look at the header of the JWT to see which kid it purports to be signed by. * Find the certificate that corresponds to that kid (from the jwks_uri content) * Validate the JWT with that certificate. That ought to work, at least given what I'm seeing. (But, I'm not a JWT expert, so I don't know whether there is something unusual in how Azure AD generates JWT's and handles the jwks_uri content). Anyway, I'm curious whether anyone else has been trying to get this to work with Azure AD, and whether they have run into similar problems. And, of course, whether I appear to be misunderstanding anything about how this is supposed to work. Ryan Rempel Director of Information Technology Canadian Mennonite University

2 3

[RFC][UADK integration][Acceleration of zlib compressor]
by Rongqi Sun 11 Jul '24

11 Jul '24

Hi Ceph community, UADK is an open source accelerator framework, the kernel support part is UACCE <https://docs.kernel.org/misc-devices/uacce.html>, which has been merged in Kernel for several years, targeting to provide Shared Virtual Addressing (SVA) between accelerators and processes. UADK provider users with a unified programming interface to efficiently harness the capabilities of hardware accelerators. It furnishes users with fundamental library and driver support. Now UADK can offer the hardware accelerator engine(e.g: Kunpeng KAE), Arm SVE and Crypto Extension CPU. UADK <https://github.com/Linaro/uadk> is hosted by Linaro. UADK now has already supports different communities for compressor and encryption, such as OpenSSL, DPDK and SPDK, so now, we would like to bring it to Ceph for Acceleration of zlib compressor. As first step, we can see that, 1. save almost 50% cpu usage compared with no-isal compression in RBD 4M workload 2. save almost 40% cpu usage compared with no-isal compression in RGW put op (4M) workload The PR <https://github.com/ceph/ceph/pull/58336> is under review, welcome any comments or reviews. Have a nice day~ Rongqi Sun

2 1

July's User + Developer Monthly Meeting
by Noah Lehman 11 Jul '24

11 Jul '24

Hi Ceph users, July's User + Developer Monthly meeting will be happening Wednesday, July 24th at 10AM EDT. We look forward to seeing you there! Event details: https://hubs.la/Q02GgVNb0 Best, Noah

1 0

Changing ip addr
by Albert Shih 11 Jul '24

11 Jul '24

Hi everyone I just change the subnet of my cluster. The cephfs part seem to working well. But I got many error with Jul 11 10:08:35 hostname ceph-*************** ts=2024-07-11T08:08:35.364Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://OLD_IP:8765/sd/prometheus/sd-config?service=alertmanager\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" I didn't find which service does the discovery and was unable to change the URL. I try a ceph config-key dump |grep OLD_IP and didn't find it. So where this information are store ? Regards JAS -- Albert SHIH 🦫 🐸 Observatoire de Paris France Heure locale/Local time: jeu. 11 juil. 2024 10:22:58 CEST

2 5

Multi site sync details
by Huseyin Cotuk 11 Jul '24

11 Jul '24

Hello Cephers, I am wondering whether it is possible to get the number of objects that are not synced yet in a multi site radosgw configuration. "radosgw-admin sync status” gives the number of shards that are behind. Similarly “radosgw-admin bucket sync status —bucket {bucket_name}” also gives the status of the shards belong to the given bucket. I just want to get the number of objects that are not synced yet. The only way I found is to get bucket stats from each zone and compare the number of objects. While one of the zone the following output (only related part), "usage": { "rgw.main": { "size": 20896350288253, "size_actual": 22604324229120, "size_utilized": 20896350288253, "size_kb": 20406592079, "size_kb_actual": 22074535380, "size_kb_utilized": 20406592079, "num_objects": 749099512 } }, the other zone gives the output below. "usage": { "rgw.main": { "size": 20896350967086, "size_actual": 22604325040128, "size_utilized": 20896350967086, "size_kb": 20406592742, "size_kb_actual": 22074536172, "size_kb_utilized": 20406592742, "num_objects": 749099571 } }, I look for the difference of number of objects between two zones ( 749099571 - 749099512 = 59 objects). But I am not sure about these stats that they provide the number of objects that are already synced and related to its own zone. The metadata of the multisite radosgw is hold on the master zone. Does anybody clarify that the number of objects on the bucket stats query belongs to every zone’s own bucket or multi site shared metadata? I have doubts that the difference between the bucket stats output from each zone may be a result of command’s execution time. Any clue will be appreciated. Best regards, Huseyin Cotuk hcotuk(a)gmail.com

1 0

use of db_slots in DriveGroup specification?
by Robert Sander 11 Jul '24

11 Jul '24

Hi, what is the purpose of the db_slots attribute in a DriveGroup specification? My interpretation of the documentation is that I can define how many OSDs use one db device. https://docs.ceph.com/en/reef/cephadm/services/osd/#additional-options "db_slots - How many OSDs per DB device" The default for the cephadm orchestrator is to create as many DB volumes on a DB device as needed for the number of OSDs. In a scenario where there are empty slots for HDDs and an existing DB device should not be used fully "db_slots" could be used. But even if db_slots is in an OSD service spec the orchestrator will only create as many DB volumes as there are HDDs currently available. There is a discussion from 2021 where the use of "block_db_size" and "limit" is suggested: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/6EVOYOHS3BT… Shouldn't db_slots make that easier? Is this a bug in the orchestrator? Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 220009 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

2 2

IT Consulting Firms with Ceph and Hashicorp Vault expertise?
by Michael Worsham 11 Jul '24

11 Jul '24

I am in need of a list of IT consulting firms that can set up high-availability Vault and also configure the Ceph Object Gateway to use SSE-S3 with Vault. -- Michael This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments.

2 2

High RAM usage for OSDs
by Work Ceph 10 Jul '24

10 Jul '24

Hello guys, We are running Ceph Octopus on Ubuntu 18.04. We noticed that some OSDs are using more than 16GiB of RAM. However, the option "osd_memory_target" is set to 4GiB. The OSDs are SSDs and have 2TiB in size each. Have you guys seen such behavior? Are we missing some other configuration or parameter? Or, maybe, is it some memory leak?

1 0