May 2024 - ceph-users - lists.ceph.io

could not find secret_id
by xiaowenhao111 22 Apr '26

22 Apr '26

2 1

OpenSSL security update for Octopus container?
by Robert Sander 02 Oct '25

02 Oct '25

Hi, I am running a Ceph Octopus cluster in version 15.2.10 setup with the cephadm orchestrator and official Docker hub container images. Yesterday an important security fix was released for libssl and is already packaged for all major distributions. I tried to run "ceph orch upgrade check docker.io/ceph/ceph:v15" but it tells me that the containers do not need to be upgraded. How will this security fix of OpenSSL be deployed in a timely manner to users of the Ceph container images? Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 93818 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 1

RBD Mirror - Failed to unlink peer
by Scott Cairns 01 Aug '25

01 Aug '25

Hi, Following the introduction of an additional node to our Ceph cluster, we've started to see unlink errors when taking a rbd mirror snapshot. We've had RBD mirroring configured for over a year now and it's been working flawlessly, however after we created OSD's on a new node we've receiving the following error: librbd::mirror::snapshot::CreatePrimaryRequest: 0x7f60c80056f0 handle_unlink_peer: failed to unlink peer: (2) No such file or directory This seemed to appear on around 3 of 150 snapshots on the first night and over the weeks has progressed to almost every snapshot. What's odd, is that the snapshot appears to be taken without any issues and does mirror to the DR site - we can see the snapshot ID taken on the source side is mirrored to the destination side when checking the rbd snap ls, and we've tested promoting an image on the DR site to ensure the snapshot does include up to date data, which it does. I can't see any other errors generated when the snapshot is taken to identify what file/directory isn't found - everything appears to be working okay it's just generating an error during the snapshot. I've also tried disabling mirroring on the disk and re-enabling however it doesn't appear to make any difference - there's no error on the initial mirror image, or the first snapshot taken after that, but every subsequent snapshot shows the error again. Any ideas? Thanks, Scott ________________________________ The content of this e-mail and any attachment is confidential and intended solely for the use of the individual to whom it is addressed. Any views or opinions presented are solely those of the author and do not necessarily represent those of Tecnica Limited. If you have received this e-mail in error please notify the sender. Any use, dissemination, forwarding, printing, or copying of this e-mail or any attachments thereto, in whole or part, without permission is strictly prohibited. Tecnica Limited Registered office: 5 Castle Court, Carnegie Campus, Dunfermline, Fife, KY11 8PB. Registered in Scotland No. SC250307. VAT No. 827 5110 42. This footnote also confirms that this email message has been swept for the presence of computer viruses.

4 8

Migrating from S3 to Ceph RGW (Cloud Sync Module)
by James McClune 10 Apr '25

10 Apr '25

Hello Ceph User Community, I currently have a large Amazon S3 environment with terabytes of data spread over dozens of buckets. I'm looking to migrate from Amazon S3 to an on-site Ceph cluster using the RGW. I'm trying to figure out the most efficient way to achieve this. Looking through the documentation, I found articles related to the cloud sync module, released in Mimic ( https://docs.ceph.com/en/latest/radosgw/cloud-sync-module/) I also watched a video on the cloud sync module as well. It *sounds* like this is the functionality I'm looking for. Given I'm moving away from Amazon S3, I'm really just looking for a one-way replication between the buckets (i.e. Provide an Amazon S3 access key/secret which is associated to the buckets and the same for the Ceph environment, so object data can be replicated one-to-one, without creating ad-hoc tooling). Once the data is replicated from S3 to Ceph, I plan on modifying my boto connection objects to use the new Ceph environment. Is what I'm describing feasible with the cloud sync module? Just looking for some affirmation, given I'm not well versed in Ceph's RGW, especially around multi-site configurations. Thanks, Jimmy

7 8

Problem while upgrade 17.2.6 to 17.2.7
by Jean-Marc FONTANA 21 Feb '25

21 Feb '25

Hello everyone, We operate two clusters that we installed with ceph-deploy in Nautilus version on Debian 10. We use them for external S3 storage (owncloud) and rbd disk images.We had them upgraded to Octopus and Pacific versions on Debian 11 and recently converted them to cephadm and upgraded to Quincy (17.2.6). As we now have the orchestrator, we tried updating to 17.2.7 using the command# ceph orch upgrade start --image quay.io/ceph/ceph:v17.2.7 Everything went well, both clusters work perfectly for our use, except that the Rados gateway configuration is no longer accessible from the dashboard with the following error messageError connecting to Object Gateway: RGW REST API failed request with status code 404. We tried a few solutions found on the internet (reset rgw credentials, restart rgw adnd mgr, reenable dashboard, ...), unsuccessfully. Does somebody have an idea ? Best regards, Jean-Marc Fontana

5 9

purging already destroyed OSD leads to degraded and misplaced objects?
by tobias tempel 10 Jan '25

10 Jan '25

Dear Cephers, reorganizing one of our clusters i'm removing some hosts from it, taking "out" all OSDs on these hosts and waiting until all PGs are fine. After stopping and destroying all OSDs on one host i notice, that "purge" of such destroyed OSDs temporarily leads to degraded and misplaced objects (tried "reweight 0" as well, same picture) . Why that? I would - due to my limited experience - expect, that completely removing an already destroyed OSD could not have any effect on object placement at all. This effect yet seems not to be a real problem, as recovery works fine ... but perhaps it indicates some other issue. All of that takes place at Pacific 16.2.14 AlmaLinux 8.9, yes i know, there's some work to do. Perhaps you can give me a hint, where to look, for an explanation? Thank you cheers, toBias -- Tobias Tempel Deutsches Elektronen-Synchrotron - IT Notkestr. 85, 22607 Hamburg, Germany email: tobias.tempel(a)desy.de

4 7

Problems with autoscaler (overlapping roots) after changing the pool class
by Massimo Sgaravatto 06 Jan '25

06 Jan '25

Dear all I have just changed the crush rule for all the replicated pools in the following way: ceph osd crush rule create-replicated replicated_hdd default host hdd ceph osd pool set <poolname> crush_rule replicated_hdd See also this [*] thread Before applying this change, these pools were all using the replicated_ruleset rule where the class is not specified. I am noticing now a problem with the autoscaler: "ceph osd pool autoscale-status" doesn't report any output and the mgr log complains about overlapping roots: [pg_autoscaler ERROR root] pool xyz has overlapping roots: {-18, -1} Indeed: # ceph osd crush tree --show-shadow ID CLASS WEIGHT TYPE NAME -18 hdd 1329.26501 root default~hdd -17 hdd 329.14154 rack Rack11-PianoAlto~hdd -15 hdd 54.56085 host ceph-osd-04~hdd 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... ... -1 1329.26501 root default -7 329.14154 rack Rack11-PianoAlto -8 54.56085 host ceph-osd-04 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... I have already read about this behavior but I have no clear ideas how to fix the problem. I read somewhere that the problem happens when there are rules that force some pools to only use one class and there are also pools which does not make any distinction between device classes All the replicated pools are using the replicated_hdd pool but I also have some EC pools which are using a profile where the class is not specified. As far I understand, I can't force these pools to use only the hdd class: according to the doc I can't change this profile specifying the hdd class (or at least the change wouldn't be applied to the existing EC pools) Any suggestions ? The crush map is available at https://cernbox.cern.ch/s/gIyjbQbmoTFHCrr, if you want to have a look Many thanks, Massimo [*] https://www.mail-archive.com/ceph-users@ceph.io/msg18534.html

4 9

multifs and snapshots
by Dmitry Melekhov 12 Nov '24

12 Nov '24

Hello! There is note here https://docs.ceph.com/en/reef/dev/cephfs-snapshots/ about multiply filesystems and snapshots: If each FS gets its own pool things probably work, but this isn’t tested and may not be true. Is it still untested ? Thank you!

3 4

Help needed to configure erasure coding LRC plugin
by Michel Jouvin 16 Oct '24

16 Oct '24

Hi, As discussed in another thread (Crushmap rule for multi-datacenter erasure coding), I'm trying to create an EC pool spanning 3 datacenters (datacenters are present in the crushmap), with the objective to be resilient to 1 DC down, at least keeping the readonly access to the pool and if possible the read-write access, and have a storage efficiency better than 3 replica (let say a storage overhead <= 2). In the discussion, somebody mentioned LRC plugin as a possible jerasure alternative to implement this without tweaking the crushmap rule to implement the 2-step OSD allocation. I looked at the documentation (https://docs.ceph.com/en/latest/rados/operations/erasure-code-lrc/) but I have some questions if someone has experience/expertise with this LRC plugin. I tried to create a rule for using 5 OSDs per datacenter (15 in total), with 3 (9 in total) being data chunks and others being coding chunks. For this, based of my understanding of examples, I used k=9, m=3, l=4. Is it right? Is this configuration equivalent, in terms of redundancy, to a jerasure configuration with k=9, m=6? The resulting rule, which looks correct to me, is: -------- { "rule_id": 6, "rule_name": "test_lrc_2", "ruleset": 6, "type": 3, "min_size": 3, "max_size": 15, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -4, "item_name": "default~hdd" }, { "op": "choose_indep", "num": 3, "type": "datacenter" }, { "op": "chooseleaf_indep", "num": 5, "type": "host" }, { "op": "emit" } ] } ------------ Unfortunately, it doesn't work as expected: a pool created with this rule ends up with its pages active+undersize, which is unexpected for me. Looking at 'ceph health detail` output, I see for each page something like: pg 52.14 is stuck undersized for 27m, current state active+undersized, last acting [90,113,2147483647,103,64,147,164,177,2147483647,133,58,28,8,32,2147483647] For each PG, there is 3 '2147483647' entries and I guess it is the reason of the problem. What are these entries about? Clearly it is not OSD entries... Looks like a negative number, -1, which in terms of crushmap ID is the crushmap root (named "default" in our configuration). Any trivial mistake I would have made? Thanks in advance for any help or for sharing any successful configuration? Best regards, Michel

4 24

rgw connection resets
by Nathan Gleason 10 Oct '24

10 Oct '24

Hello, I’ve noticed what I think may be strange behavior from some rgw transfers. It only seems to be with larger multipart mp4 files. Once a connection comes into rgw it is accepted and processed, data begins to transfer, then I see "write_data failed: Connection reset by peer” and "s3:get_obj iterate_obj() failed with -104”. The client then reconnects and the transfer seems to be ok. This doesn’t appear to happen just downloading a regular file. This is happening to both my production and test cluster. I’m not seeing errors anywhere else, the network looks good and the clusters are healthy. I’m in the process of collecting some tcpdumps but figured I would ask if anyone knows why this is occurring. We run haproxy in front of the rgw. For troubleshooting purposes we have a basic haproxy config to just one rgw. We have tested with multiple browsers. Ceph 17.2.5 - cluster built and managed with cephadm Ubuntu 20.04.6 LTS Log: 2024-01-03T17:40:23.916+0000 7f646033d700 1 ====== starting new request req=0x7f640ca95730 ===== 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s initializing for trans_id = tx000003ec1e4b4dc157efd-0065959c07-104216-default 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s rgw api priority: s3=8 s3website=7 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s host=a2-west.usacld.net 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s final domain/bucket subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 s->info.domain= s->info.request_uri=/n athan:tshoot/testfile.mp4 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s get_handler handler=22RGWHandler_REST_Obj_S3 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s handler=22RGWHandler_REST_Obj_S3 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s getting op 0 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s get_system_obj_state: rctx=0x7f640ca947c0 obj=default.rgw.log:script.prerequest. state=0x55fa31f882e0 s->prefetch_data=0 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s cache get: name=default.rgw.log++script.prerequest. : hit (negative entry) 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj scheduling with throttler client=2 cost=1 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj op=21RGWGetObj_ObjStore_S3 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj verifying requester 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rgw::auth::StrategyRegistry::s3_main_strategy_t: trying rgw::auth::s3::AWSAuthStrategy 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::S3AnonymousEngine 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rgw::auth::s3::S3AnonymousEngine granted access 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rgw::auth::s3::AWSAuthStrategy granted access 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj normalizing buckets and tenants 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s->object=testfile.mp4 s->bucket=nathan/tshoot 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj init permissions 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_system_obj_state: rctx=0x7f640ca94200 obj=default.rgw.meta:root:nathan/tshoot state=0x55fa31f882e0 s->prefetch _data=0 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj cache get: name=default.rgw.meta+root+nathan/tshoot : hit (requested=0x16, cached=0x17) 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_system_obj_state: s->obj_tag was set empty 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj cache get: name=default.rgw.meta+root+nathan/tshoot : hit (requested=0x11, cached=0x17) 2024-01-03T17:40:23.916+0000 7f646033d700 15 req 4522146966294331133 0.000000000s s3:get_obj decode_policy Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner> <ID>nathan$testuser</ID><DisplayName>Nathan API User</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>nathan$testuser </ID><DisplayName>Nathan API User</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy> 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWSI_User_RADOS::read_user_info(): anonymous user 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj recalculating target 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj reading permissions 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_obj_state: rctx=0x7f640ca94ad0 obj=tshoot:testfile.mp4 state=0x55fa31082de8 s->prefetch_data=1 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj WARNING: blocking librados call 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj manifest: total_size = 88811314 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_obj_state: setting s->obj_tag to _0OUNxgsSsqTAs9Z6xh1SOKpJZLV_Vos 2024-01-03T17:40:23.916+0000 7f646033d700 15 req 4522146966294331133 0.000000000s s3:get_obj decode_policy Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner> <ID>nathan$testuser</ID><DisplayName>Nathan API User</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="Group"><URI>http://acs.amazonaws.c om/groups/global/AllUsers</URI></Grantee><Permission>READ</Permission></Grant><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>nathan$testuser</ID><DisplayNa me>Nathan API User</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy> 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj init op 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_system_obj_state: rctx=0x7f640ca94488 obj=default.rgw.meta:users.uid:nathan$testuser state=0x55fa31f882e0 s->p refetch_data=0 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj cache get: name=default.rgw.meta+users.uid+nathan$testuser : hit (requested=0x16, cached=0x17) 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_system_obj_state: s->obj_tag was set empty 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr: user.rgw.idtag 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj cache get: name=default.rgw.meta+users.uid+nathan$testuser : hit (requested=0x13, cached=0x17) 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj verifying op mask 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj required_mask= 1 user.op_mask=7 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj verifying op permissions 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj -- Getting permissions begin with perm_mask=49 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj Searching permissions for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=anonymous, acct_n ame=, subuser=, perm_mask=15, is_admin=0) mask=49 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj Searching permissions for uid=anonymous 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj Permissions for user not found 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj Searching permissions for group=1 mask=49 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj Found permission: 1 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj -- Getting permissions done for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=anonymous, acct_name=, subuser=, perm_mask=15, is_admin=0), owner=nathan$testuser, perm=1 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=anonymous, acct_name=, subuser=, perm_mask =15, is_admin=0) requested perm (type)=1, policy perm=1, user_perm_mask=15, acl perm=1 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj verifying op params 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj pre-executing 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj check rate limiting 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj executing 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_obj_state: rctx=0x7f640ca94ad0 obj=tshoot:testfile.mp4 state=0x55fa31082de8 s->prefetch_data=1 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.acl 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.content_type 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.etag 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.idtag 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.manifest 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.pg_ver 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.source_zone 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.tail_tag 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.x-amz-content-sha256 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.x-amz-date 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.x-amz-meta-s3cmd-attrs 2024-01-03T17:40:23.916+0000 7f646033d700 15 req 4522146966294331133 0.000000000s Encryption mode: 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_obj_state: rctx=0x7f640ca94ad0 obj=tshoot:testfile.mp4 state=0x55fa31082de8 s->prefetch_data=1 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__multipart_testfile.mp4.2~XSiTvBBtaU4 134dQCusv3kMP7YjDBk7.1 obj-ofs=0 read_ofs=0 len=4194304 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=4194304 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): result: ofs=4194304 stripe_ofs=4194304 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134 dQCusv3kMP7YjDBk7.1_1 obj-ofs=4194304 read_ofs=0 len=4194304 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=8388608 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): result: ofs=8388608 stripe_ofs=8388608 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134 dQCusv3kMP7YjDBk7.1_2 obj-ofs=8388608 read_ofs=0 len=4194304 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=12582912 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): result: ofs=12582912 stripe_ofs=12582912 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134 dQCusv3kMP7YjDBk7.1_3 obj-ofs=12582912 read_ofs=0 len=3145728 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=16777216 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): result: ofs=15728640 stripe_ofs=15728640 part_ofs=15728640 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__multipart_testfile.mp4.2~XSiTvBBtaU4 134dQCusv3kMP7YjDBk7.2 obj-ofs=15728640 read_ofs=0 len=4194304 2024-01-03T17:40:23.924+0000 7f6442301700 20 req 4522146966294331133 0.007999974s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.924+0000 7f6442301700 20 req 4522146966294331133 0.007999974s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=19922944 part_ofs=15728640 rule->part_size=15728640 2024-01-03T17:40:23.924+0000 7f6442301700 20 req 4522146966294331133 0.007999974s s3:get_obj RGWObjManifest::operator++(): result: ofs=19922944 stripe_ofs=19922944 part_ofs=15728640 rule->part_size=15728640 2024-01-03T17:40:23.924+0000 7f6442301700 20 req 4522146966294331133 0.007999974s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134 dQCusv3kMP7YjDBk7.2_1 obj-ofs=19922944 read_ofs=0 len=4194304 2024-01-03T17:40:23.932+0000 7f643d2f7700 20 req 4522146966294331133 0.015999949s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.932+0000 7f643d2f7700 20 req 4522146966294331133 0.015999949s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=24117248 part_ofs=15728640 rule->part_size=15728640 2024-01-03T17:40:23.932+0000 7f643d2f7700 20 req 4522146966294331133 0.015999949s s3:get_obj RGWObjManifest::operator++(): result: ofs=24117248 stripe_ofs=24117248 part_ofs=15728640 rule->part_size=15728640 2024-01-03T17:40:23.932+0000 7f643d2f7700 20 req 4522146966294331133 0.015999949s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134 dQCusv3kMP7YjDBk7.2_2 obj-ofs=24117248 read_ofs=0 len=4194304 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_status: e=0, sent=17, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_content_length: e=0, sent=26, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::complete_header: e=0, sent=460, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::set_account: e=1 2024-01-03T17:40:24.068+0000 7f642a2d1700 4 write_data failed: Connection reset by peer 2024-01-03T17:40:24.068+0000 7f642a2d1700 0 req 4522146966294331133 0.151999518s s3:get_obj iterate_obj() failed with -104 2024-01-03T17:40:24.156+0000 7f642eada700 2 req 4522146966294331133 0.239999235s s3:get_obj completing 2024-01-03T17:40:24.156+0000 7f642eada700 20 req 4522146966294331133 0.239999235s get_system_obj_state: rctx=0x7f640ca947c0 obj=default.rgw.log:script.postrequest.nathan state=0x55fa32424760 s->prefetch_data =0 2024-01-03T17:40:24.156+0000 7f642eada700 10 req 4522146966294331133 0.239999235s cache get: name=default.rgw.log++script.postrequest.nathan : hit (negative entry) 2024-01-03T17:40:24.156+0000 7f642eada700 30 AccountingFilter::complete_request: e=1, sent=0, total=0 2024-01-03T17:40:24.156+0000 7f642eada700 2 req 4522146966294331133 0.239999235s s3:get_obj op status=-104 2024-01-03T17:40:24.156+0000 7f642eada700 2 req 4522146966294331133 0.239999235s s3:get_obj http status=200 2024-01-03T17:40:24.156+0000 7f642eada700 1 ====== req done req=0x7f640ca95730 op status=-104 http_status=200 latency=0.239999235s ====== 2024-01-03T17:40:24.156+0000 7f642eada700 1 beast: 0x7f640ca95730: 10.30.0.196 - anonymous [03/Jan/2024:17:40:23.916 +0000] "GET /nathan:tshoot/testfile.mp4 HTTP/1.1" 200 0 - "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" - latency=0.239999235s 2024-01-03T17:40:24.156+0000 7f642eada700 20 failed to read header: end of stream 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_ACCEPT=*/* 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_ACCEPT_ENCODING=identity;q=1, *;q=0 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_ACCEPT_LANGUAGE=en-US,en;q=0.7 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_HOST=a2-west.usacld.net 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_RANGE=bytes=0- 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_REFERER=https://a2-west.usacld.net/nathan:tshoot/testfile.mp4 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_CH_UA="Not_A Brand";v="8", "Chromium";v="120", "Brave";v="120" 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_CH_UA_MOBILE=?0 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_CH_UA_PLATFORM="macOS" 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_FETCH_DEST=video 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_FETCH_MODE=no-cors 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_FETCH_SITE=same-origin 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_GPC=1 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_VERSION=1.1 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_X_FORWARDED_PORT=443 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_X_FORWARDED_PROTO=https 2024-01-03T17:40:24.160+0000 7f6503483700 20 REMOTE_ADDR=10.30.0.196 2024-01-03T17:40:24.160+0000 7f6503483700 20 REQUEST_METHOD=GET 2024-01-03T17:40:24.160+0000 7f6503483700 20 REQUEST_URI=/nathan:tshoot/testfile.mp4 2024-01-03T17:40:24.160+0000 7f6503483700 20 SCRIPT_URI=/nathan:tshoot/testfile.mp4 2024-01-03T17:40:24.160+0000 7f6503483700 20 SERVER_PORT=8000 2024-01-03T17:40:24.160+0000 7f6503483700 1 ====== starting new request req=0x7f640ca95730 ===== Thanks, Nathan

2 1

Not all Bucket Shards being used
by Christian Kugler 01 Oct '24

01 Oct '24

Hi, I have trouble with large OMAP files in a cluster in the RGW index pool. Some background information about the cluster: There is CephFS and RBD usage on the main cluster but for this issue I think only S3 is interesting. There is one realm, one zonegroup with two zones which have a bidirectional sync set up. Since this does not allow for autoresharding we have to do it by hand in this cluster – looking forward to Reef! From the logs: cluster 2023-07-17T22:59:03.018722+0000 osd.75 (osd.75) 623978 : cluster [WRN] Large omap object found. Object: 34:bcec3016:::.dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.5:head PG: 34.680c373d (34.5) Key count: 962091 Size (bytes): 277963182 The offending bucket looks like this: # radosgw-admin bucket stats \ | jq '.[] | select(.marker =="3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9") |"\(.num_shards) \(.usage["rgw.main"].num_objects)"' -r 131 9463833 Last week the number of objects was about 12 million. Which is why I reshareded the offending bucket twice, I think. Once to 129 and the second time to 131 because I wanted some leeway (or lieway? scnr, Sage). Unfortunately, even after a week the objects were still to big (the log line above is quite recent), so I looked into it again. # rados -p raum.rgw.buckets.index ls \ |grep .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9 \ |sort -V .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.0 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.1 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.2 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.3 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.4 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.5 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.6 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.7 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.8 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.9 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.10 # rados -p raum.rgw.buckets.index ls \ |grep .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9 \ |sort -V \ |xargs -IOMAP sh -c \ 'rados -p raum.rgw.buckets.index listomapkeys OMAP | wc -l' 1013854 1011007 1012287 1011232 1013565 998262 1012777 1012713 1012230 1010690 997111 Apparently, only 11 shards are in use. This would explain why the "Key usage" (from the log line) is about ten times higher than I would expect. How can I deal with this issue? One thing I could try to fix this would be to reshard to a lower number, but I am not sure if there are any risks associated with "downsharding". After that I could reshard to something like 97. Or I could directly "downshard" to 97. Also, the second zone has a similar problem, but as the error messsage lets me know, this would be a bad idea. Will it just take more time until the sharding is transferred to the seconds zone? Best, Christian Kugler

2 5

Numa pinning best practices
by Szabo, Istvan (Agoda) 15 Sep '24

15 Sep '24

Hi, Haven't really found a proper descripton in case of 2 socket how to pin osds to numa node, only this: https://tracker.ceph.com/projects/ceph/wiki/Tuning_for_All_Flash_Deployment… Tuning for All Flash Deployments - Ceph - Ceph <https://tracker.ceph.com/projects/ceph/wiki/Tuning_for_All_Flash_Deployment…> Redmine tracker.ceph.com Is there anybody have some good how to on this topic? Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

5 5

Successfully using dm-cache
by Michael Lipp 12 Sep '24

12 Sep '24

Just in case anybody is interested: Using dm-cache works and boosts performance -- at least for my use case. The "challenge" was to get 100 (identical) Linux-VMs started on a three node hyperconverged cluster. The hardware is nothing special, each node has a Supermicro server board with a single CPU with 24 cores and 4 x 4 TB hard disks. And there's that extra 1 TB NVMe... I know that the general recommendation is to use the NVMe for WAL and metadata, but this didn't seem appropriate for my use case and I'm still not quite sure about failure scenarios with this configuration. So instead I made each drive a logical volume (managed by an OSD) and added 85 GiB NVMe to each LV as read-only cache. Each VM uses as system disk an RBD based on a snapshot from the master image. The idea was that with this configuration, all VMs should share most (actually almost all) of the data on their system disk and this data should be available from the cache. Well, it works. When booting the 100 VMs, almost all read operations are satisfied from the cache. So I get close to NVMe speed but have payed for conventional hard drives only (well, SSDs aren't that much more expensive nowadays, but the hardware is 4 years old). So, nothing sophisticated, but as I couldn't find anything about this kind of setup, it might be of interest nevertheless. - Michael

13 27

cephadm basic questions: image config, OS reimages
by Matthew Vernon 27 Aug '24

27 Aug '24

Hi, I've some experience with Ceph, but haven't used cephadm much before, and am trying to configure a pair of reef clusters with cephadm. A couple of newbie questions, if I may: * cephadm shell image I'm in an isolated environment, so pulling from a local repository. I bootstrapped OK with cephadm --image docker-registry.wikimedia.org/ceph bootstrap ... And that worked nicely, but if I want to run cephadm shell (to do any sort of admin), then I have to specify cephadm --image docker-registry.wikimedia.org/ceph shell (otherwise it just hangs failing to talk to quay.io). I found the docs, which refer to setting lots of other images, but not the one that cephadm uses: https://docs.ceph.com/en/reef/cephadm/install/#deployment-in-an-isolated-en… I found an old tracker in this area: https://tracker.ceph.com/issues/47274 ...but is there a good way to arrange for cephadm to use the already-downloaded image without having to remember to specify --image each time? * OS reimages We do OS upgrades by reimaging the server (which doesn't touch the storage disks); on an old-style deployment you could then use ceph-volume to re-start the OSDs and away you went; how does one do this in a cephadm cluster? [I presume involves telling cephadm to download a new image for podman to use and suchlike] Would the process be smoother if we arranged to leave /var/lib/ceph intact between reimages? Thanks, Matthew

3 4

Public Swift bucket with Openstack Keystone integration - not working in quincy/reef
by Bartosz Bezak 31 Jul '24

31 Jul '24

Hi, Similar case as with previously fixed https://tracker.ceph.com/issues/48382 - https://github.com/ceph/ceph/pull/47308. Confirmed on Cephadm deployed Ceph 18.2.2/17.2.7 with Openstack Antelope/Yoga. I’m getting "404 NoSuchBucket" error with public buckets. Enabled with Swift/Keystone integration - everything else works fine. With rgw_swift_account_in_url = true and proper endpoints: "https://rgw.test/swift/v1/AUTH_%(project_id)s" ticking public access in horizon properly sets ACL on the bucket according to swift client: swift -v stat test-bucket URL: https://rgw.test/swift/v1/AUTH_daksjhdkajdshda/testbucket Auth Token: Account: AUTH_daksjhdkajdshda Container: testbucket Objects: 1 Bytes: 1021036 Read ACL: .r:*,.rlistings Write ACL: Sync To: Sync Key: X-Timestamp: 1710947159.41219 X-Container-Bytes-Used-Actual: 1024000 X-Storage-Policy: default-placement X-Storage-Class: STANDARD Last-Modified: Thu, 21 Mar 2024 10:30:05 GMT X-Trans-Id: tx00000092ac12312312312-1231231231-1701e5-default X-Openstack-Request-Id: tx00000092ac12312312312-1231231231-1701e5-default Accept-Ranges: bytes Content-Type: text/plain; charset=utf-8 however still getting 404 NoSuchBucket error Could someone using the latest version of Ceph with Swift/Keystone integration please test public buckets? Thank you. Best regards, Bartosz Bezak

3 3

ceph orch osd rm --zap --replace leaves cluster in odd state
by Matthew Vernon 14 Jul '24

14 Jul '24

Hi, I want to prepare a failed disk for replacement. I did: ceph orch osd rm 35 --zap --replace and it's now in the state "Done, waiting for purge", with 0 pgs, and REPLACE and ZAP set to true. It's been like this for some hours, and now my cluster is unhappy: [WRN] CEPHADM_STRAY_DAEMON: 1 stray daemon(s) not managed by cephadm stray daemon osd.35 on host moss-be1002 not managed by cephadm (the OSD is down & out) ...and also neither the disk nor the relevant NVME LV has been zapped. I have my OSDs deployed via a spec: service_type: osd service_id: rrd_single_NVMe placement: label: "NVMe" spec: data_devices: rotational: 1 db_devices: model: "NVMe" And before issuing the ceph orch osd rm I set that to be unmanaged (ceph orch set-unmanaged osd.rrd_single_NVMe), as obviously I don't want ceph to just try and re-make a new OSD on the sad disk. I'd expected from the docs[0] that what I did would leave me with a system ready for the failed disk to be swapped (and that I could then mark osd.rrd_single_NVMe as managed again, and a new OSD built), including removing/wiping the NVME lv so it can be removed. What did I do wrong? I don't much care about the OSD id (but obviously it's neater to not just incrementally increase OSD numbers every time a disk died), but I thought that telling ceph orch not to make new OSDs then using ceph orch osd rm to zap the disk and NVME lv would have been the way to go... Thanks, Matthew [0] https://docs.ceph.com/en/reef/cephadm/services/osd/#replacing-an-osd

3 3

reef 18.2.3 QE validation status
by Yuri Weinstein 12 Jul '24

12 Jul '24

Details of this release are summarized here: https://tracker.ceph.com/issues/65393#note-1 Release Notes - TBD LRC upgrade - TBD Seeking approvals/reviews for: smoke - infra issues, still trying, Laura PTL rados - Radek, Laura approved? Travis? Nizamudeen? rgw - Casey approved? fs - Venky approved? orch - Adam King approved? krbd - Ilya approved powercycle - seems fs related, Venky, Brad PTL ceph-volume - will require https://github.com/ceph/ceph/pull/56857/commits/63fe3921638f1fb7fc065907a9e… Guillaume is fixing it. TIA

10 29

cephadm discovery service certificate absent after upgrade.
by Nicolas FOURNIL 09 Jul '24

09 Jul '24

Hello, I've just fresh upgrade from Quincy to Reef and my graphs are now blank... after investigations, it seems that discovery service is not working because of no certificate : # ceph orch sd dump cert Error EINVAL: No certificate found for service discovery Maybe an upgrade issue ? Is there a way to generate or replace the certificate properly ? Regards Nicolas F.

3 8

Fixing BlueFS spillover (pacific 16.2.14)
by Chris Dunlop 09 Jul '24

09 Jul '24

Hi, tl;dr why are my osds still spilling? I've recently upgraded to 16.2.14 from 16.2.9 and started receiving bluefs spillover warnings (due to the "fix spillover alert" per the 16.2.14 release notes). E.g. from 'ceph health detail', the warning on one of these (there are a few): osd.76 spilled over 128 KiB metadata from 'db' device (56 GiB used of 60 GiB) to slow device This is a 15T HDD with only a 60G SSD for the db so it's not surprising it spilled as it's way below the recommendation for rbd usage at db size 1-2% of the storage size. There was some spare space on the db ssd so I increased the size of the db LV up over 400G and did an bluefs-bdev-expand. However, days later, I'm still getting the spillover warning for that osd, including after running a manual compact: # ceph tell osd.76 compact See attached perf-dump-76 for the perf dump output: # cephadm enter --name 'osd.76' ceph daemon 'osd.76' perf dump" | jq -r '.bluefs' In particular, if my understanding is correct, that's telling me the db available size is 487G (i.e. the LV expand worked), of which it's using 59G, and there's 128K spilled to the slow device: "db_total_bytes": 512309059584, # 487G "db_used_bytes": 63470305280, # 59G "slow_used_bytes": 131072, # 128K A "bluefs stats" also says the db is using 128K of slow storage (although perhaps it's getting the info from the same place as the perf dump?): # ceph tell osd.76 bluefs stats 1 : device size 0x7747ffe000 : using 0xea6200000(59 GiB) 2 : device size 0xe8d7fc00000 : using 0x6554d689000(6.3 TiB) RocksDBBlueFSVolumeSelector Usage Matrix: DEV/LEV WAL DB SLOW * * REAL FILES LOG 0 B 10 MiB 0 B 0 B 0 B 8.8 MiB 1 WAL 0 B 2.5 GiB 0 B 0 B 0 B 751 MiB 8 DB 0 B 56 GiB 128 KiB 0 B 0 B 50 GiB 842 SLOW 0 B 0 B 0 B 0 B 0 B 0 B 0 TOTAL 0 B 58 GiB 128 KiB 0 B 0 B 0 B 850 MAXIMUMS: LOG 0 B 22 MiB 0 B 0 B 0 B 18 MiB WAL 0 B 3.9 GiB 0 B 0 B 0 B 1.0 GiB DB 0 B 71 GiB 282 MiB 0 B 0 B 62 GiB SLOW 0 B 0 B 0 B 0 B 0 B 0 B TOTAL 0 B 74 GiB 282 MiB 0 B 0 B 0 B >> SIZE << 0 B 453 GiB 14 TiB I had a look at the "DUMPING STATS" output in the logs bug I don't know how to interpret it. I did try calculating the total of the sizes on the "Sum" lines but that comes to 100G so I don't know what that all means. See attached log-stats-76. I also tried "ceph-kvstore-tool bluestore-kv ... stats": $ { cephadm unit --fsid $clusterid --name osd.76 stop cephadm shell --fsid $clusterid --name osd.76 -- ceph-kvstore-tool bluestore-kv /var/lib/ceph/osd/ceph-76 stats cephadm unit --fsid $clusterid --name osd.76 start } Output attached as bluestore-kv-stats-76. I can't see anything interesting in there, although again I don't really know how to interpret it. So... why is this osd db still spilling onto slow storage, and how do I fix things so it's no longer using the slow storage? And a bonus issue... on another osd that hasn't yet been resized (i.e. again with a grossly undersized 60G db on SSD with a 15T HDD) I'm also getting a spillover warning. The "bluefs stats" seems to be saying the db is NOT currently spilling (i.e. "0 B" the DB/SLOW position in the matrix), but there's "something" currently using 59G on the slow device: $ ceph tell osd.85 bluefs stats 1 : device size 0xeffffe000 : using 0x3a3900000(15 GiB) 2 : device size 0xe8d7fc00000 : using 0x7aea7434000(7.7 TiB) RocksDBBlueFSVolumeSelector Usage Matrix: DEV/LEV WAL DB SLOW * * REAL FILES LOG 0 B 10 MiB 0 B 0 B 0 B 7.4 MiB 1 WAL 0 B 564 MiB 0 B 0 B 0 B 132 MiB 2 DB 0 B 11 GiB 0 B 0 B 0 B 8.1 GiB 177 SLOW 0 B 3.0 GiB 59 GiB 0 B 0 B 56 GiB 898 TOTAL 0 B 13 GiB 59 GiB 0 B 0 B 0 B 1072 MAXIMUMS: LOG 0 B 24 MiB 0 B 0 B 0 B 20 MiB WAL 0 B 2.8 GiB 0 B 0 B 0 B 1.0 GiB DB 0 B 22 GiB 448 KiB 0 B 0 B 18 GiB SLOW 0 B 3.3 GiB 62 GiB 0 B 0 B 62 GiB TOTAL 0 B 27 GiB 62 GiB 0 B 0 B 0 B >> SIZE << 0 B 57 GiB 14 TiB Is there anywhere that describes how to interpret this output, and specifically, what stuff is going into the SLOW row? Seemingly there's 898 "files" there, but not LOG, WAL or DB files - so what are they? Cheers, Chris

4 7

Pacific 16.2.15 `osd noin`
by Zakhar Kirpichenko 08 Jul '24

08 Jul '24

Hi, I'm adding a few OSDs to an existing cluster, the cluster is running with `osd noout,noin`: cluster: id: 3f50555a-ae2a-11eb-a2fc-ffde44714d86 health: HEALTH_WARN noout,noin flag(s) set Specifically `noin` is documented as "prevents booting OSDs from being marked in". But freshly added OSDs were immediately marked `up` and `in`: services: ... osd: 96 osds: 96 up (since 5m), 96 in (since 6m); 338 remapped pgs flags noout,noin # ceph osd tree in | grep -E "osd.11|osd.12|osd.26" 11 hdd 9.38680 osd.11 up 1.00000 1.00000 12 hdd 9.38680 osd.12 up 1.00000 1.00000 26 hdd 9.38680 osd.26 up 1.00000 1.00000 Is this expected behavior? Do I misunderstand the purpose of the `noin` option? Best regards, Zakhar

5 7

[rbd mirror] integrity of journal-based image mirror
by Tony Liu 24 Jun '24

24 Jun '24

Hi, Say, the source image is being updated and data is mirrored to destination continuously. Suddenly, networking of source is down and destination will be promoted and used to restore the VM. Is that going to cause any FS issue and, for example, fsck needs to be invoked to check and repair FS? Is there any integrity check during journal-based mirror to avoid "partial" update caused by networking issue? Any insight from dev or experiences from users is appreciated. Thanks! Tony

2 2

Ceph osd df tree takes a long time to respond
by Huy Nguyen 21 Jun '24

21 Jun '24

Ceph version 14.2.7 Ceph osd df tree command take long time than usual but I can't find out what is the reason? The monitor node still has plenty of available RAM and CPU resources. I checked the monitor and mgr log but nothing seems useful. I checked an older cluster in version 13.2.10 but Ceph osd df tree still responded normally, so that was not version related. Anyone have any idea? Thanks

2 2

Correct Migration Workflow Replicated -> Erasure Code
by Mac Wynkoop 19 Jun '24

19 Jun '24

Hi Everyone, So, I'm in the process of trying to migrate our rgw.buckets.data pool from a replicated rule pool to an erasure coded pool. I've gotten the EC pool set up, good EC profile and crush ruleset, pool created successfully, but when I go to "rados cppool xxx.rgw.buckets.data xxx.rgw.buckets.data.new", I get this error after it transfers 4GB of data: error copying object: (2) No such file or directory error copying pool xxx.rgw.buckets.data => xxx.rgw.buckets.data.new: (2) No such file or directory Is "rados cppool" still the blessed way to do the migration, or has something better/not deprecated been developed that I can use? Thanks, Mac

4 4

Re: MDS crashes to damaged metadata
by Patrick Donnelly 10 Jun '24

10 Jun '24

On Thu, Dec 15, 2022 at 9:32 AM Stolte, Felix <f.stolte(a)fz-juelich.de> wrote: > > Hi Patrick, > > we used your script to repair the damaged objects on the weekend and it went smoothly. Thanks for your support. > > We adjusted your script to scan for damaged files on a daily basis, runtime is about 6h. Until thursday last week, we had exactly the same 17 Files. On thursday at 13:05 a snapshot was created and our active mds crashed once at this time (snapshot was created): > > 2022-12-08T13:05:48.919+0100 7f440afec700 -1 /build/ceph-16.2.10/src/mds/ScatterLock.h: In function 'void ScatterLock::set_xlock_snap_sync(MDSContext*)' thread 7f440afec700 time 2022-12-08T13:05:48.921223+0100 > /build/ceph-16.2.10/src/mds/ScatterLock.h: 59: FAILED ceph_assert(state LOCK_XLOCK || state LOCK_XLOCKDONE) > > 12 Minutes lates the unlink_local error crashes appeared again. This time with a new file. During debugging we noticed a MTU mismatch between MDS (1500) and client (9000) with cephfs kernel mount. The client is also creating the snapshots via mkdir in the .snap directory. > > We disabled snapshot creation for now, but really need this feature. I uploaded the mds logs of the first crash along with the information above to https://tracker.ceph.com/issues/38452 > > I would greatly appreciate it, if you could answer me the following question: > > Is the Bug related to our MTU Mismatch? We fixed the MTU Issue going back to 1500 on all nodes in the ceph public network on the weekend also. I doubt it. > If you need a debug level 20 log of the ScatterLock for further analysis, i could schedule snapshots at the end of our workdays and increase the debug level 5 Minutes arround snap shot creation. This would be very helpful! -- Patrick Donnelly, Ph.D. He / Him / His Principal Software Engineer Red Hat, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

2 4

tuning for backup target cluster
by Lukasz Borek 10 Jun '24

10 Jun '24

Hi Everyone, I'm putting together a HDD cluster with an ECC pool dedicated to the backup environment. Traffic via s3. Version 18.2, 7 OSD nodes, 12 * 12TB HDD + 1NVME each, 4+2 ECC pool. Wondering if there is some general guidance for startup setup/tuning in regards to s3 object size. Files are read from fast storage (SSD/NVME) and written to s3. Files sizes are 10MB-1TB, so it's not standard s3. traffic. Backup for big files took hours to complete. My first shot would be to increase default bluestore_min_alloc_size_hdd, to reduce the number of stored objects, but I'm not sure if it's a good direccion? Any other parameters worth checking to support such a traffic pattern? Thanks! -- Łukasz

4 13

Are we logging IRC channels?
by Alvaro Soto 08 Jun '24

08 Jun '24

Hi Community!!! Are we logging IRC channels? I ask this because a lot of people only use Slack, and the Slack we use doesn't have a subscription, so messages are lost after 90 days (I believe) I believe it's important to keep track of the technical knowledge we see each day over IRC+Slack Cheers! -- Alvaro Soto *Note: My work hours may not be your work hours. Please do not feel the need to respond during a time that is not convenient for you.* ---------------------------------------------------------- Great people talk about ideas, ordinary people talk about things, small people talk... about other people.

5 9

Rebalance OSDs after adding disks?
by tpDev Tester 06 Jun '24

06 Jun '24

Hi, I have been curiuos about ceph for long time and now I started to experiment to find out, how it works. The idea I like most is, that ceph can provide growing storage without the need to move from storage x to storage y on consumer side. I started with a 3 node cluster where each node got one OSD (2x 500GB, 1x 1TB). I created the pool and the filesystem, mounted the filesystem and filled it with data. All three OSDs filled up evenly and when the pool reached 70%, I added two 1TB OSDs (each on the node with 500GB). Now I expected some activity to rebalance the fill level of the OSDs, but nothing special happened. The new OSDs got some data, but the 500GB OSDs ran into 95%, the pool reached 100%, everything got stuck with 2 OSDs filled up just 20% and left 80% free space? I searched the documentation how to initiate the rebalance/redistribution by hand but was unable to find anything. Can someone please point me to the docs how I can expand the capacity of the pool without such problems. Thanks in advance Thomas

2 2

Error EINVAL: check-host failed - Failed to add host
by isnraju26＠yahoo.com 05 Jun '24

05 Jun '24

Hello, Please help: CEPH cluster using docker. using below command for the bootstrap with provided key and pub cephadm -v bootstrap --mon-ip <ip> --allow-overwrite --ssh-private-key id_ed25519 --ssh-public-key id_ed25519.pub able to ssh directly with id_ed25519 key. RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v18 -e NODE_NAME=<hostname> -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/85c044a8-1d82-11ef-9ea1-a73759ab75e5:/var/log/ceph:z -v /tmp/ceph-tmp1sw6a5s0:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpethxlwxr:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v18 orch host add <hostname> <ip>: Error EINVAL: check-host failed: Unable to write <host>:/var/lib/ceph/85c044a8-1d82-11ef-9ea1-a73759ab75e5/cephadm.2b9d7d139a9cb40289f2358faf49a109fc297c0a258bde893227c262c30bca8d: Session request failed also validated below and it was successful: > ceph cephadm get-ssh-config > ssh_config > ceph config-key get mgr/cephadm/ssh_identity_key > key > ssh -F ssh_config -i key root@<hostname> --- root@<hostname>:~/.ssh# ceph orch host add <hostname> <ip> Error EINVAL: check-host failed: Unable to write <hostname>:/var/lib/ceph/fe8ecd30-1da2-11ef-9ea1-a73759ab75e5/cephadm.2b9d7d139a9cb40289f2358faf49a109fc297c0a258bde893227c262c30bca8d: Session request failed root@<hostname>:~/.ssh# Thanks, Surya

2 5

stretched cluster new pool and second pool with nvme
by ronny.lippold 04 Jun '24

04 Jun '24

hi ... running against the wall, i need your help, again. our test stretched cluster is running fine. now i have 2 questions. whats the right way to add another pool? create pool with 4/2 and use the rule for the stretched mode, finished? the exsisting pools were automaticly set to 4/2 after "ceph mon enable_stretch_mode". the second question, we want to use ssd and nvme together. so, we need to have a second pool for class nvme. i don't know, how to setup a second crush rule for the nvme class. i thought, that i need to filter with 2 rules for the classes. is that correct? thanks for help, ronny

3 7

About placement group scrubbing state
by Phong Tran Thanh 03 Jun '24

03 Jun '24

Hi everyone, I want to know about the scrubbing state placement group, my cluster has too many pg in state scrubbing and it's increasing over time, maybe scrubbing takes too long. [image: image.png] Cluster is not problem I'm using reef version root@n1s1:~# ceph health detail HEALTH_OK I want to ask, if the scrubbing state of pg is too long, what is the matter with my cluster? scrubbing error or another problem? Thank to the community ---------------------------------------------------------------------------- *Tran Thanh Phong* Email: tranphong079(a)gmail.com Skype: tranphong079

2 1

Reef: RGW Multisite object fetch limits
by Jayanth Reddy 01 Jun '24

01 Jun '24

Hello Community, We've two zones with Reef (v18.2.1) and trying to sync over 2 billion RGW objects to the secondary zone. We've added a fresh secondary zone and each zones have 2 RGW dedicated daemons (behind LB) each and only for multisite; whereas others don't run sync threads. Strange thing is that during the sync, the master zone LB records only 1k to 1.5k HTTP requests per second from secondary zone RGWs and not more than that. This behaviour leaves us with less link utilization. I don't see any noticeable issues with the link, cluster or RGW daemons on both ends. We've already increased rgw_data_sync_spawn_window, rgw_bucket_sync_spawn_window, rgw_meta_sync_spawn_window values to more than the defaults and don't seem to improve. Has someone noticed this behaviour of secondary zone RGW daemons fetching only around 1k to 1.5k objects per second? Regards, Jayanth

2 3

rgw mgr module not shipped? (in reef at least)
by Matthew Vernon 01 Jun '24

01 Jun '24

Hi, As far as I can tell, the rgw mgr module is not shipped in the published reef Debian packages (nor, I suspect, the ubuntu ones, but I've not actually checked). Is there a reason why it couldn't just be added to ceph-mgr-modules-core ? That contains quite a large number of modules already, and the rgw one is effectively one small python file, I think... I'm using 18.2.2. Thanks, Matthew

2 1

How to create custom container that exposes a listening port?
by Robert Sander 31 May '24

31 May '24

Hi, with the custom container service it is possible to deploy other services with the cephadm orchestrator: https://docs.ceph.com/en/reef/cephadm/services/custom-container/ The application inside the container wants to listen to a port (say 8080/tcp). How do I tell podman to publish this port? I tried adding this to the service specification: extra_container_args: - "--publish 8080/tcp" but this results in this error: Error EINVAL: ServiceSpec: __init__() got an unexpected keyword argument 'extra_container_args' https://docs.ceph.com/en/reef/cephadm/services/#extra-container-arguments Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 1

unknown PGs after adding hosts in different subtree
by Eugen Block 31 May '24

31 May '24

Hi, I got into a weird and unexpected situation today. I added 6 hosts to an existing Pacific cluster (16.2.13, 20 existing OSD hosts across 2 DCs). The hosts were added to the root=default subtree, their designated location is one of two datacenters underneath the default root. Nothing unusual, I believe many people use different subtrees to organize their cluster, as do we in our own (and haven't seen above issue yet). The main application is RGW, the main pool is erasure-coded (k=7, m=11). The crush rule looks like this: rule rule-ec-k7m11 { id 1 type erasure min_size 3 max_size 18 step set_chooseleaf_tries 5 step set_choose_tries 100 step take default class hdd step choose indep 2 type datacenter step chooseleaf indep 9 type host step emit } After almost all peering had finished the status showed 6 inactive + peering PGs for a while. I had to fail the mgr because it didn't report correct stats anymore, then it showed 16 unknown PGs. Their application noticed the (unexpected) disruption, after putting the hosts into their designated crush bucket (datacenter) the situation resolved. But I can't make any sense of it, I tried to reproduce it in my lab environment (Quincy), but to no avail. In my tests it behaves as expected, after new OSDs become active there are remapped PGs, but nothing happens until I add them to their designated location. I know I could have prevented that with either osd_crush_initial_weight = 0, then move the crush buckets, then reweight, or by adding the crush buckets first, but usually I don't need to bother about these things. Does anyone have an explanation? I'd appreciate any comments. Thanks! Eugen

3 16

Ceph Reef v18.2.3 - release date?
by Peter Razumovsky 30 May '24

30 May '24

Hello! We're waiting brand new minor 18.2.3 due to https://github.com/ceph/ceph/pull/56004. Why? Timing in our work is a tough thing. Could you kindly share an estimation of 18.2.3 release timeframe? It is 16 days passed from original tag creation so I want to understand when it will be released for upgrade time planning.

2 1

CephFS HA: mgr finish mon failed to return metadata for mds
by kvesligaj＠cs.hr 30 May '24

30 May '24

Hi, we have a stretched cluster (Reef 18.2.1) with 5 nodes (2 nodes on each side + witness). You can se our daemon placement below. [admin] ceph-admin01 labels="['_admin', 'mon', 'mgr']" [nodes] [DC1] ceph-node01 labels="['mon', 'mgr', 'mds', 'osd']" ceph-node02 labels="['mon', 'rgw', 'mds', 'osd']" [DC2] ceph-node03 labels="['mon', 'mgr', 'mds', 'osd']" ceph-node04 labels="['mon', 'rgw', 'mds', 'osd']" We have been testing CephFS HA and noticed when we have active MDS (we have two active MDS daemons at all times) and active MGR (MGR is either on admin node or in one of the DC's) in one DC and when we shutdown that site (DC) we have a problem when one of the MDS metadata can't be retrieved thus showing in logs as: "mgr finish mon failed to return metadata for mds" After we turn that site back on the problem persists and metadata of MDS in question can't be retrieved with "ceph mds metadata" After I manually fail MDS daemon in question with "ceph mds fail" the problem is solved and I can retrieve MDS metadata. My question is, would this be related to the following bug (https://tracker.ceph.com/issues/63166) - I can see that it is showed as backported to 18.2.1 but I can't find it in release notes for Reef. Second question is should this work in current configuration at all as MDS and MGR are both at the same moment disconnected from the rest of the cluster? And final question would be what would be the solution here and is there any loss of data when this happens? Any help is appreciated.

2 1

Help needed! First MDs crashing, then MONs. How to recover ?
by Noe P. 30 May '24

30 May '24

Hi, we ran into a bigger problem today with our ceph cluster (Quincy, Alma8.9). We have 4 filesystems and a total of 6 MDs, the largest fs having two ranks assigned (i.e. one standby). Since we often have the problem of MDs lagging behind, we restart the MDs occasionally. Helps ususally, the standby taking over. Today however, the restart didn't work and the rank 1 MDs started to crash for unclear reasons. Rank 0 seemed ok. We decided at some point to go back to one rank by settings max_mds to 1. Due to the permanent crashing, the rank1 didn't stop however, and at some point we set it to failed and the fs not joinable. At this point it looked like this: fs_cluster - 716 clients ========== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active cephmd6a Reqs: 0 /s 13.1M 13.1M 1419k 79.2k 1 failed POOL TYPE USED AVAIL fs_cluster_meta metadata 1791G 54.2T fs_cluster_data data 421T 54.2T with rank1 still being listed. The next attempt was to remove that failed ceph mds rmfailed fs_cluster:1 --yes-i-really-mean-it which, after a short while brought down 3 out of five MONs. They keep crashing shortly after restart with stack traces like this: ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) 1: /lib64/libpthread.so.0(+0x12cf0) [0x7ff8813adcf0] 2: gsignal() 3: abort() 4: /lib64/libstdc++.so.6(+0x9009b) [0x7ff8809bf09b] 5: /lib64/libstdc++.so.6(+0x9654c) [0x7ff8809c554c] 6: /lib64/libstdc++.so.6(+0x965a7) [0x7ff8809c55a7] 7: /lib64/libstdc++.so.6(+0x96808) [0x7ff8809c5808] 8: /lib64/libstdc++.so.6(+0x92045) [0x7ff8809c1045] 9: (MDSMonitor::maybe_resize_cluster(FSMap&, int)+0xa9e) [0x55f05d9a5e8e] 10: (MDSMonitor::tick()+0x18a) [0x55f05d9b18da] 11: (MDSMonitor::on_active()+0x2c) [0x55f05d99a17c] 12: (Context::complete(int)+0xd) [0x55f05d76c56d] 13: (void finish_contexts<std::__cxx11::list<Context*, std::allocator<Context*> > >(ceph::common::CephContext*, std::__cxx11::list<Context*, std::allocator<Context*> >&, int)+0x9d) [0x55f05 d799d7d] 14: (Paxos::finish_round()+0x74) [0x55f05d8c5c24] 15: (Paxos::dispatch(boost::intrusive_ptr<MonOpRequest>)+0x41b) [0x55f05d8c7e5b] 16: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x123e) [0x55f05d76a2ae] 17: (Monitor::_ms_dispatch(Message*)+0x406) [0x55f05d76a976] 18: (Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x5d) [0x55f05d79b3ed] 19: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0x478) [0x7ff88367fed8] 20: (DispatchQueue::entry()+0x50f) [0x7ff88367d31f] 21: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff883747381] 22: /lib64/libpthread.so.0(+0x81ca) [0x7ff8813a31ca] 23: clone() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. The MDSMonitor::maybe_resize_cluster somehow suggests a connection to the above MDs operation. Does anyone have an idea how to get this cluster back together again ? Like manually fixing the MD ranks ? 'fs get' can still be called in short moments where enough MONs are reachable: fs_name fs_cluster epoch 3065486 flags 13 allow_snaps allow_multimds_snaps created 2022-08-26T15:55:07.186477+0200 modified 2024-05-28T12:21:59.294364+0200 tableserver 0 root 0 session_timeout 60 session_autoclose 300 max_file_size 4398046511104 required_client_features {} last_failure 0 last_failure_osd_epoch 1777109 compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} max_mds 1 in 0,1 up {0=911794623} failed damaged stopped 2,3 data_pools [32] metadata_pool 33 inline_data disabled balancer standby_count_wanted 1 [mds.cephmd6a{0:911794623} state up:active seq 22777 addr [v2:10.13.5.6:6800/189084355,v1:10.13.5.6:6801/189084355] compat {c=[1],r=[1],i=[7ff]}] Regards, Noe

2 1

MDS Abort druing FS scrub
by Malcolm Haak 30 May '24

30 May '24

When running a cephfs scrub the MDS will crash with the following backtrace -1> 2024-05-25T09:00:23.028+1000 7ef2958006c0 -1 /usr/src/debug/ceph/ceph-18.2.2/src/mds/MDSRank.cc: In function 'void MDSRank::abort(std::string_view)' thread 7ef2958006c0 time 2024-05-25T09:00:23.031373+1000 /usr/src/debug/ceph/ceph-18.2.2/src/mds/MDSRank.cc: 938: ceph_abort_msg("abort() called") ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable) 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xd8) [0x7ef2a3a1ea65] 2: (MDSRank::abort(std::basic_string_view<char, std::char_traits<char> >)+0x89) [0x5b97f9c811b9] 3: (CDentry::check_corruption(bool)+0x4e3) [0x5b97f9ef2583] 4: (EMetaBlob::add_primary_dentry(EMetaBlob::dirlump&, CDentry*, CInode*, unsigned char)+0x59) [0x5b97f9d554a9] 5: (Locker::scatter_writebehind(ScatterLock*)+0x2e9) [0x5b97f9e775c9] 6: (Locker::simple_sync(SimpleLock*, bool*)+0x16a) [0x5b97f9e7db4a] 7: (Locker::file_eval(ScatterLock*, bool*)+0x139) [0x5b97f9e84299] 8: (Locker::_drop_locks(MutationImpl*, std::set<CInode*, std::less<CInode*>, std::allocator<CInode*> >*, bool)+0x1b1) [0x5b97f9e96221] 9: (Locker::drop_locks(MutationImpl*, std::set<CInode*, std::less<CInode*>, std::allocator<CInode*> >*)+0x7f) [0x5b97f9e9726f] 10: (MDCache::repair_inode_stats_work(boost::intrusive_ptr<MDRequestImpl>&)+0x8f7) [0x5b97f9dd2dd7] 11: (MDCache::repair_inode_stats(CInode*)+0x90) [0x5b97f9df61e0] 12: /usr/bin/ceph-mds(+0x460cb8) [0x5b97f9f58cb8] 13: (Continuation::_continue_function(int, int)+0x1c3) [0x5b97f9f6a903] 14: /usr/bin/ceph-mds(+0x46efa1) [0x5b97f9f66fa1] 15: (Continuation::_continue_function(int, int)+0x1c3) [0x5b97f9f6a903] 16: (Continuation::Callback::finish(int)+0x14) [0x5b97f9f6aa14] 17: (Context::complete(int)+0xd) [0x5b97f9c75bfd] 18: (MDSContext::complete(int)+0x5f) [0x5b97f9fd2b0f] 19: (MDSIOContextBase::complete(int)+0x38c) [0x5b97f9fd2ffc] 20: (Finisher::finisher_thread_entry()+0x175) [0x7ef2a3adac95] 21: /usr/lib/libc.so.6(+0x8b55a) [0x7ef2a32a955a] 22: /usr/lib/libc.so.6(+0x108a3c) [0x7ef2a3326a3c] This is after running through the "Advanced recovery procedure" I assume I need a flag or something on the MDS to allow it to keep running during said scan?

2 1

How to setup NVMeoF?
by Robert Sander 30 May '24

30 May '24

Hi, I am trying to follow the documentation at https://docs.ceph.com/en/reef/rbd/nvmeof-target-configure/ to deploy an NVMe over Fabric service. Step 2b of the configuration section is currently the showstopper. First the command says: error: the following arguments are required: --host-name/-t Then it tells me (after adding --host-name): error: unrecognized arguments: --gateway-name XXX and when I remove --gateway-name the error is: both gateway_name and traddr or neither must be specified So I am stuck in a kind of a loop here. Is there a working description for NVMe over TCP available? Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

5 6

RBD-Images are not shown in the Dashbord: Failed to execute RBD [errno 19] error generating diff from snapshot None
by Maximilian Dauer 30 May '24

30 May '24

Dear Community, I hope, you can guide me to solve this error or I can assist in solving a bug: RBD-Images are not shown in my Dashbord. - When accessing the dashboard page (block -> images) no images were listed and the error "Failed to execute RBD [errno 19] error generating diff from snapshot None" is shown. - When asking for the list via cli by "rbd ls" the test-image is shown. - There are no snapshots listed when asking for "rbd snap ls --all test". - Back to the ui: I can create and use a new image via UI, this will be shown via cli, but not in the ui. - The cluster manages several RBD, RGW and Cephfs pools successfully with ~13TiB used capacity out of ~18TiB. **Environment details** - Ceph-Cluster with 7 Debian 12 nodes and 3 Ubuntu 22.04 nodes at latest patch level. - Cluster-Version: 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable). - Error persists since Quincy. - Orchestrator: cephadm. - Dashboard is accessed via haproxy, but it's the same behavior when accessing it directly via ip. - All nodes are updated and rebooted. - Dashboard was removed and reactivated. **Steps to reproduce ** I need your help to find the root cause. **Logs** Health status is ok. I can't find useful logs except the attached error message shown in the Dashboard ("ailed to execute RBD [errno 19] error generating diff from snapshot None"). Do you have an idea, where to search for further useful logs? Last but not least: Thanks for this great piece of software! All the best Max

1 0

Missing ceph data
by Prabu GJ 30 May '24

30 May '24

Hi Team, We are using Ceph Octopus version with a total disk size of 136 TB, configured with two replicas. Currently, our usage is 57 TB, and the available size is 5.3 TB. An incident occurred yesterday where around 3 TB of data was deleted automatically. Upon analysis, we couldn't find the reason for the deletion. All OSDs are functioning properly and actively running. We have 3 MDS , we try to restarted all MDS services. Is there any solution to recover those data. Can anyone please help us find the issue? cluster: id: 0d605d58-5caf-4f76-b6bd-e12402a22296 health: HEALTH_WARN insufficient standby MDS daemons available 5 nearfull osd(s) 3 pool(s) nearfull 1 pool(s) have non-power-of-two pg_num services: mon: 4 daemons, quorum download-mon3,download-mon4,download-mon1,download-mon2 (age 14h) mgr: download-mon2(active, since 14h), standbys: download-mon1, download-mon3 mds: integdownload:2 {0=download-mds3=up:active,1=download-mds1=up:active} osd: 39 osds: 39 up (since 16h), 39 in (since 4d) data: pools: 3 pools, 1087 pgs objects: 71.76M objects, 51 TiB usage: 105 TiB used, 31 TiB / 136 TiB avail pgs: 1087 active+clean io: client: 414 MiB/s rd, 219 MiB/s wr, 513 op/s rd, 1.22k op/s wr ================================================================ ID HOST USED AVAIL WR OPS WR DATA RD OPS RD DATA STATE 0 download-osd1 2995G 581G 14 4785k 6 6626k exists,up 1 download-osd2 2578G 998G 84 3644k 18 10.1M exists,up 2 download-osd3 3093G 483G 17 5114k 5 4152k exists,nearfull,up 3 download-osd4 2757G 819G 12 996k 2 4107k exists,up 4 download-osd5 2889G 687G 28 3355k 20 8660k exists,up 5 download-osd6 2448G 1128G 183 3312k 10 9435k exists,up 6 download-osd7 2814G 762G 7 1667k 4 6354k exists,up 7 download-osd8 2872G 703G 14 1672k 15 10.5M exists,up 8 download-osd9 2577G 999G 10 6615k 3 6960k exists,up 9 download-osd10 2651G 924G 16 4736k 3 7378k exists,up 10 download-osd11 2889G 687G 15 4810k 6 8980k exists,up 11 download-osd12 2912G 664G 11 2516k 2 4106k exists,up 12 download-osd13 2785G 791G 74 4643k 11 3717k exists,up 13 download-osd14 3150G 426G 214 6133k 4 7389k exists,nearfull,up 14 download-osd15 2728G 848G 11 4959k 4 6603k exists,up 15 download-osd16 2682G 894G 13 3170k 3 2503k exists,up 16 download-osd17 2555G 1021G 53 2183k 7 5058k exists,up 17 download-osd18 3013G 563G 18 3497k 3 4427k exists,up 18 download-osd19 2924G 651G 24 3534k 12 10.4M exists,up 19 download-osd20 3003G 573G 19 5149k 3 2531k exists,up 20 download-osd21 2757G 819G 16 3707k 9 9816k exists,up 21 download-osd22 2576G 999G 15 2526k 8 7739k exists,up 22 download-osd23 2758G 818G 13 4412k 16 7125k exists,up 23 download-osd24 2862G 714G 18 4424k 6 5787k exists,up 24 download-osd25 2792G 783G 16 1972k 9 9749k exists,up 25 download-osd26 2397G 1179G 14 4296k 9 12.0M exists,up 26 download-osd27 2308G 1267G 8 3149k 22 6280k exists,up 27 download-osd29 2732G 844G 12 3357k 3 7372k exists,up 28 download-osd28 2814G 761G 11 476k 5 3316k exists,up 29 download-osd30 3069G 507G 15 9043k 17 5628k exists,nearfull,up 30 download-osd31 2660G 916G 15 841k 14 7798k exists,up 31 download-osd32 2037G 1539G 10 1153k 15 3719k exists,up 32 download-osd33 3116G 460G 20 7704k 12 9041k exists,nearfull,up 33 download-osd34 2847G 728G 19 5788k 4 9014k exists,up 34 download-osd35 3088G 488G 17 7178k 7 5730k exists,nearfull,up 35 download-osd36 2414G 1161G 27 2017k 14 7612k exists,up 36 download-osd37 2760G 815G 17 4292k 5 10.6M exists,up 37 download-osd38 2679G 897G 12 2610k 5 10.0M exists,up 38 download-osd39 3013G 563G 18 1804k 7 9235k exists,up Regards Prabu GJ

2 1

How to recover from an MDs rank in state 'failed'
by Noe P. 30 May '24

30 May '24

Hi, after our desaster yesterday, it seems that we got our MONs back. One of the filesystems, however, seems in a strange state: % ceph fs status .... fs_cluster - 782 clients ========== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active cephmd6a Reqs: 5 /s 13.2M 13.2M 1425k 51.4k 1 failed POOL TYPE USED AVAIL fs_cluster_meta metadata 3594G 53.5T fs_cluster_data data 421T 53.5T .... STANDBY MDS cephmd6b cephmd4b MDS version: ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) % ceph fs dump .... Filesystem 'fs_cluster' (3) fs_name fs_cluster epoch 3068261 flags 12 joinable allow_snaps allow_multimds_snaps created 2022-08-26T15:55:07.186477+0200 modified 2024-05-29T12:43:30.606431+0200 tableserver 0 root 0 session_timeout 60 session_autoclose 300 max_file_size 4398046511104 required_client_features {} last_failure 0 last_failure_osd_epoch 1777109 compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} max_mds 2 in 0,1 up {0=911794623} failed damaged stopped 2,3 data_pools [32] metadata_pool 33 inline_data disabled balancer standby_count_wanted 1 [mds.cephmd6a{0:911794623} state up:active seq 44701 addr [v2:10.13.5.6:6800/189084355,v1:10.13.5.6:6801/189084355] compat {c=[1],r=[1],i=[7ff]}] We would like to get rid of the failed rank 1 (without crashing the MONs) and have a 2nd MD from the standbys step in . Anyone have an idea how to do this ? I'm a bit reluctant to try 'ceph mds rmfailed', as this seems to have triggered the MONs to crash. Regards, Noe

3 3

We are using ceph octopus environment. For client can we use ceph quincy?
by s.dhivagar.cse＠gmail.com 30 May '24

30 May '24

We are using ceph octopus environment. For client can we use ceph quincy?

2 1

MDs stuck in rejoin with '[ERR] : loaded dup inode'
by Noe P. 30 May '24

30 May '24

Hi, I'm still unable to get our filesystem back. I now have this: fs_cluster - 0 clients ========== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 rejoin cephmd4b 90.0k 89.4k 14.7k 0 1 rejoin cephmd6b 105k 105k 21.3k 0 2 failed POOL TYPE USED AVAIL fs_cluster_meta metadata 288G 55.2T fs_cluster_data data 421T 55.2T Still cannot get rid of the 3rd failed rank. But the other two currently stay in state rejoin forever. After all clients were stopped, the log complains about a 'dup inode': 2024-05-30T07:59:46.252+0200 7f2fe9146700 -1 log_channel(cluster) log [ERR] : loaded dup inode 0x1001710ea1d [12bc6a,head] v1432525092 at /homes/YYY/ZZZ/.bash_history-21032.tmp, but inode 0x1001710ea1d.head v1432525109 already exists at /homes/YYY/ZZZ/.bash_history Questions: - Is there a way to scan/repair the metadata without any MD in 'active' state ? - Is there a way to remove (or otherwise fix) the inode in question given the above inode number ? - Is the state 'rejoin' due to the inode error or because of that 3rd rank ? Regard, N.

1 0

Ceph data got missed
by prabu.jawahar＠gmail.com 30 May '24

30 May '24

We are using Ceph Octopus version with a total disk size of 136 TB, configured with two replicas. Currently, our usage is 57 TB, and the available size is 5.3 TB. An incident occurred yesterday where around 3 TB of data was deleted automatically. Upon analysis, we couldn't find the reason for the deletion. All OSDs are functioning properly and actively running. We have 3 MDS , we try to restarted all MDS services. Is there any solution to recover those data. Can anyone please help us find the issue? cluster: id: 0d605d58-5caf-4f76-b6bd-e12402a22296 health: HEALTH_WARN insufficient standby MDS daemons available 5 nearfull osd(s) 3 pool(s) nearfull 1 pool(s) have non-power-of-two pg_num services: mon: 4 daemons, quorum download-mon3,download-mon4,download-mon1,download-mon2 (age 14h) mgr: download-mon2(active, since 14h), standbys: download-mon1, download-mon3 mds: integdownload:2 {0=download-mds3=up:active,1=download-mds1=up:active} osd: 39 osds: 39 up (since 16h), 39 in (since 4d) data: pools: 3 pools, 1087 pgs objects: 71.76M objects, 51 TiB usage: 105 TiB used, 31 TiB / 136 TiB avail pgs: 1087 active+clean io: client: 414 MiB/s rd, 219 MiB/s wr, 513 op/s rd, 1.22k op/s wr ================================================================ ID HOST USED AVAIL WR OPS WR DATA RD OPS RD DATA STATE 0 download-osd1 2995G 581G 14 4785k 6 6626k exists,up 1 download-osd2 2578G 998G 84 3644k 18 10.1M exists,up 2 download-osd3 3093G 483G 17 5114k 5 4152k exists,nearfull,up 3 download-osd4 2757G 819G 12 996k 2 4107k exists,up 4 download-osd5 2889G 687G 28 3355k 20 8660k exists,up 5 download-osd6 2448G 1128G 183 3312k 10 9435k exists,up 6 download-osd7 2814G 762G 7 1667k 4 6354k exists,up 7 download-osd8 2872G 703G 14 1672k 15 10.5M exists,up 8 download-osd9 2577G 999G 10 6615k 3 6960k exists,up 9 download-osd10 2651G 924G 16 4736k 3 7378k exists,up 10 download-osd11 2889G 687G 15 4810k 6 8980k exists,up 11 download-osd12 2912G 664G 11 2516k 2 4106k exists,up 12 download-osd13 2785G 791G 74 4643k 11 3717k exists,up 13 download-osd14 3150G 426G 214 6133k 4 7389k exists,nearfull,up 14 download-osd15 2728G 848G 11 4959k 4 6603k exists,up 15 download-osd16 2682G 894G 13 3170k 3 2503k exists,up 16 download-osd17 2555G 1021G 53 2183k 7 5058k exists,up 17 download-osd18 3013G 563G 18 3497k 3 4427k exists,up 18 download-osd19 2924G 651G 24 3534k 12 10.4M exists,up 19 download-osd20 3003G 573G 19 5149k 3 2531k exists,up 20 download-osd21 2757G 819G 16 3707k 9 9816k exists,up 21 download-osd22 2576G 999G 15 2526k 8 7739k exists,up 22 download-osd23 2758G 818G 13 4412k 16 7125k exists,up 23 download-osd24 2862G 714G 18 4424k 6 5787k exists,up 24 download-osd25 2792G 783G 16 1972k 9 9749k exists,up 25 download-osd26 2397G 1179G 14 4296k 9 12.0M exists,up 26 download-osd27 2308G 1267G 8 3149k 22 6280k exists,up 27 download-osd29 2732G 844G 12 3357k 3 7372k exists,up 28 download-osd28 2814G 761G 11 476k 5 3316k exists,up 29 download-osd30 3069G 507G 15 9043k 17 5628k exists,nearfull,up 30 download-osd31 2660G 916G 15 841k 14 7798k exists,up 31 download-osd32 2037G 1539G 10 1153k 15 3719k exists,up 32 download-osd33 3116G 460G 20 7704k 12 9041k exists,nearfull,up 33 download-osd34 2847G 728G 19 5788k 4 9014k exists,up 34 download-osd35 3088G 488G 17 7178k 7 5730k exists,nearfull,up 35 download-osd36 2414G 1161G 27 2017k 14 7612k exists,up 36 download-osd37 2760G 815G 17 4292k 5 10.6M exists,up 37 download-osd38 2679G 897G 12 2610k 5 10.0M exists,up 38 download-osd39 3013G 563G 18 1804k 7 9235k exists,up

1 0

Error EINVAL: check-host failed - Failed to add host
by Suryanarayana Raju 30 May '24

30 May '24

Hello, Please help: CEPH cluster using docker. using below command for the bootstrap with provided key and pub cephadm -v bootstrap --mon-ip <ip> --allow-overwrite --ssh-private-key id_ed25519 --ssh-public-key id_ed25519.pub able to ssh directly with id_ed25519 key. RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v18 -e NODE_NAME=<hostname> -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/85c044a8-1d82-11ef-9ea1-a73759ab75e5:/var/log/ceph:z -v /tmp/ceph-tmp1sw6a5s0:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpethxlwxr:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v18 orch host add <hostname> <ip>: Error EINVAL: check-host failed: Unable to write <host>:/var/lib/ceph/85c044a8-1d82-11ef-9ea1-a73759ab75e5/cephadm.2b9d7d139a9cb40289f2358faf49a109fc297c0a258bde893227c262c30bca8d: Session request failed also validated below and it was successful: > ceph cephadm get-ssh-config > ssh_config > ceph config-key get mgr/cephadm/ssh_identity_key > key > ssh -F ssh_config -i key root@<hostname> --- root@<hostname>:~/.ssh# ceph orch host add <hostname> <ip> Error EINVAL: check-host failed: Unable to write <hostname>:/var/lib/ceph/fe8ecd30-1da2-11ef-9ea1-a73759ab75e5/cephadm.2b9d7d139a9cb40289f2358faf49a109fc297c0a258bde893227c262c30bca8d: Session request failed root@<hostname>:~/.ssh# Thanks, Surya

2 1

Unable to Install librados2 18.2.0 on RHEL 7 from Ceph Repository
by abdel.douichi＠gmail.com 30 May '24

30 May '24

The Ceph repository at https://download.ceph.com/ does not seem to have the librados2 package version 18.2.0 for RHEL 7. The directory <b>https://download.ceph.com/rpm-18.2.0/el7/</b> is empty, and the specific package URL https://download.ceph.com/rpm-18.2.0/el7/x86_64/librados2-18.2.0-0.el7.x86_… returns a 404 error.

2 1

MDS crashing
by Johan 29 May '24

29 May '24

Hi, I have a small cluster with 11 osds and 4 filesystems. Each server (Debian 11, ceph 17.2.7) usually run several services. After troubles with a host with OSD:s I removed the OSD:s and let the cluster repair it self (x3 replica). After a while it returned to a healthy state and everything was well. This might not be important for what followed, but I mention it just in case. A couple of days later a mds-services gave a health-warning. First one was (2024-05-28 10:02) mds.cloudfs.stugan6.ywuomz(mds.0): 1 slow requests are blocked > 30 secs followed by filesystem being degraded (2024-05-28 10:22) fs cloudfs is degraded the other filesystems have been marked degraded from time to time but later cleared. at 2024-05-28 10:28 daemon mds.mediafs.stugan7.zzxavs on stugan7 is in error state daemon mds.cloudfs.stugan7.cmjbun on stugan7 is in error state at 2024-05-28 10:33 daemon mds.cloudfs.stugan4.qxwzox on stugan4 is in error state daemon mds.cloudfs.stugan5.hbkkad on stugan5 is in error state daemon mds.oxylfs.stugan7.iazekf on stugan7 is in error state MDS-services went on crashing ... I put the osd:s on pause and nodown,noout,nobackfill,norebalance,norecover flags, but at present only the flags as I have tried to get the system up and running again. While the osd:s were paused, I could 'clear up' the mess and remove all services in error state. The monitors and managers seems to function well. I could also start getting the mds-services running again. BUT, when I removed the pause from the osd:s the mds-services once again started to go inte error state. Now I have removed the mds-label from all ceph servers and therefore it has calmed down. But if I let the services be recreated the crashes will start over again. If I check the filesystem (I have marked them down for now) status the cloudfs is strange... oxylfs - 0 clients ====== POOL TYPE USED AVAIL oxylfs_metadata metadata 154M 20.8T oxylfs_data data 1827G 20.8T cloudfs - 0 clients ======= RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 replay(laggy) backupfs.stugan6.bgcltx 0 0 0 0 POOL TYPE USED AVAIL cloudfs_metadata metadata 337M 20.8T cloudfs_data data 356G 20.8T mediafs - 0 clients ======= POOL TYPE USED AVAIL mediafs_metadata metadata 66.0M 20.8T mediafs_data data 2465G 20.8T backupfs - 0 clients ======== POOL TYPE USED AVAIL backupfsnew_metadata metadata 221M 20.8T backupfsnew_data data 8740G 20.8T MDS version: ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) Why is the mds (in error state) for the backupfs-filesystem shown with the cloudfs-filesystem? Now... Is there a way back to normal? /Johan

1 0

Re: Lousy recovery for mclock and reef
by Kai Stian Olstad 29 May '24

29 May '24

On 24.05.2024 21:07, Mazzystr wrote: > I did the obnoxious task of updating ceph.conf and restarting all my > osds. > > ceph --admin-daemon /var/run/ceph/ceph-osd.*.asok config get > osd_op_queue > { > "osd_op_queue": "wpq" > } > > I have some spare memory on my target host/osd and increased the target > memory of that OSD to 10 Gb and restarted. No effect observed. In > fact > mem usage on the host is stable so I don't think the change took effect > even with updating ceph.conf, restart and a direct asok config set. > target > memory value is confirmed to be set via asok config get > > Nothing has helped. I still cannot break the 21 MiB/s barrier. > > Does anyone have any more ideas? For recovery you can adjust the following. osd_max_backfills default is 1, in my system I get the best performance with 3 and wpq. The following I have not adjusted myself, but you can try. osd_recovery_max_active is default to 3. osd_recovery_op_priority is default to 3, a lower number increases the priority for recovery. All of them can be runtime adjusted. -- Kai Stian Olstad

8 11

Bug Found in Reef Releases - Action Required for pg-upmap-primary Interface Users
by Laura Flores 29 May '24

29 May '24

Dear Ceph Users, We have discovered a bug with the pg-upmap-primary interface (related to the offline read balancer [1]) that affects all Reef releases. In all Reef versions, users are required to set `require-min-compat-client=reef` in order to use the pg-upmap-primary interface to prevent pre-reef clients from connecting and not understanding the new interface. We found this setting is simply not enforced [2], which leads to miscommunication between older and newer peers or, depending on version, to an assert in the mons and/or osds [3]. However, the fundamental precondition is making use of the new `pg-upmap-primary` feature. If you have not yet upgraded to v18.2.2, we recommend that you refrain from upgrading to v18.2.2 until a later version is out with a fix. We also recommend removing any existing pg-upmap-primary mappings to prevent hitting the assert [3], as well as to prevent any miscommunication between older and newer peers about pg primaries [2]. Remove mappings by: $ `ceph osd dump` For each pg_upmap_primary entry in the above output: $ `ceph osd rm-pg-upmap-primary <pgid>` If you have already upgraded to v18.2.2, your cluster is more likely to hit the osd/mon assert [3] when you set a `pg-upmap-primary` mapping (this would involve explicitly setting a mapping via the osdmaptool or the CLI command). As long as you refrain from setting any pg-upmap-primary mappings, your cluster will NOT be affected by [3]. Follow the trackers below for further updates. 1. pg-upmap-primary documentation: https://docs.ceph.com/en/reef/rados/operations/read-balancer/ 2. mon, osd, *: require-min-compat-client is not really honored - https://tracker.ceph.com/issues/66260 3. Failed assert "pg_upmap_primaries.empty()" in the read balancer - https://tracker.ceph.com/issues/61948 Thanks, Laura Flores -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

3 2

Rocky 8 to Rocky 9 upgrade and ceph without data loss
by Christopher Durham 28 May '24

28 May '24

I have both a small test cluster and a larger production cluster. They are (were, for the test cluster) running Rocky Linux 8.9. They are both updated originally from Pacific, currently at reef 18.2.2.These are all rpm installs. It has come time to consider upgrades to Rocky 9.3. As there is no officially supported upgrade procedure from Rocky 8 to Rocky 9, I wanted to document my procedure for the test cluster(it worked!) before I move on to my production cluster, and verify that what I have done makes sense and ask a few questions. As such, this involves a reinstall of the OS. For my test cluster, my mon nodes also have all the other services: ceph-mon, ceph-mds, ceph-radosgw and ceph-mgr. ALL of my OSDs are LVMs, and they each have wal devices, also as LVMs. My aim was to preserve these across the reinstall of the OS, and I did. To be clear,all the OSDs are not on the same physical disk as the OS and the mon/mgr/mds/radosgw, and most are grouped on their own node. First, I set noout, norecover, nobackfill # ceph osd set noout# ceph osd set norecover# ceph osd set nobackfill On one of the mon nodes, I did the following, to save off the monmap, osdmap, and crushmap in case something went wildly wrong # systemctl stop ceph-mon.target# ceph-mon -i <short servername> --extract-monmap /tmp/monmap.bin# systemcl start ceph-mon.target# ceph osd getcrushmap -o /tmp/crush.bin# ceph osd getmap -o /tmp/osd.bin Then on EVERY node, both mons and osd nodes, I tarred up /etc/ceph and /var/lib/ceph # tar cvf /tmp/varlibceph.tar /etc/ceph /var/lib/ceph I then saved off each tar file on each node to a location not being reinstalled, as such I had a tarfile per osd node and also one for each mon node. I also saved off the monmap, crushmap and osdmaps created above tothe same location. I then went sequentially through the mon/mds/radosgw/mgr servers, one at a time. As I am using kickstart/cobbler, I told kickstartto ignore all disks EXCEPT the one that the original 8.9 OS was installed on. For this I had to use the following in the kickstart file. ignoredisk --only-use=disk/by-path/pci-id-of-root-disk I did this because, at least on my hardware, Rocky 9 reorders drive letters sda, sdb, sdc, etc based on recognition order on *every* boot, which mightoverwrite an OSD LVM if I wasn't careful enough. Note that while I group some commands that follow below, I only did the mon nodes sequentially, whille I grouped together osd nodes and did those groups allat the same time based on my failure domain for my crushmap. After the Rocky 9 reinstall and a reboot: I then installed the appropriate el9 18.2.2 ceph packages $ dnf install ceph-mon (mon nodes)# dnf install ceph-mds (mds nodes) # dnf install ceph-radosgw (radosgw nodes)# dnf install ceph-mgr (mgr nodes)# dnf install ceph-osd (osd nodes) For the various nodes, once the reinstall was done to Rocky 9, I re-enabled the firewall # firewall-cmd --add-service ceph-mon --permanent ( for mon nodes)# firewall-cmd --add-service ceph --permanent (for osd nodes)# firewall-cmd --add-service https --permanent (for radosgw servers I am running them on port 443, with certs)# systemctl restart firewalld and then of course restarted firewalld I then restored /etc/ceph and /var/lib/ceph for each node from the individual tar file backups per node. For the non-OSD nodes, I re-enabled services: # systemctl enable ceph-mon(a)short-name-of-server.service (mon nodes)# systemctl enable ceph-mds(a)short-name-of-server.service (mds nodes)# systemctl enable ceph-radosgw(a)rgw.short-name-of-server.service (radosgw servers)# systemctl enable ceph-mgr(a)short-name-of-server.service (mgr nodes) For the OSD nodes, "ceph-volume lvm list" shows this for each OSD, even after reinstallingto Rocky 9 (which is good evidence that I installed on the proper disk and did not overwritean OSD: # ceph-volume lvm list block device /dev/ceph-c5b97619-4184-4637-9b82-2575004dba41/osd-block-abb210f3-52cf-4d4b-8745-126dc57287da block uuid lmxbj0-zgcB-5MwI-wpsf-9yut-3VlG-p9DoFL cephx lockbox secret cluster fsid 0f3b6c81-3814-4dc5-a19e-f7307543e56c cluster name ceph crush device class encrypted 0 osd fsid abb210f3-52cf-4d4b-8745-126dc57287da osd id 7 osdspec affinity type block vdo 0 devices /dev/sdj I had to enable units of the form: # systemctl enable ceph-lvm-$osd-id-$osd-fsid.service For the above osd lvm, this would be: # systemctl enable ceph-volume(a)lvm-7-abb210f3-52cf-4d4b-8745-126dc57287da.service So If I had 10 OSD LVMs on a given node, I have to enable 10 services of the form above. After reboot of either a mon or osd node (depending on which server I am doing), all comes up just fine. My questions: 1. Did I do anything wrong?2. Did I miss something?3. is there an official way to do this that is different from my way?4. Will anything I did (or did not) do bite me when I later do an update say to 18.2.6 or 19? 4. Is there an 'official' way to back up things like /etc/ceph and /var/lib/ceph that would be '*better**? Of course, after I update all the OSDs, I take off noout, norecover, nobackfill # ceph osd unset nobackfill# ceph osd unset norecover# ceph osd unset noout Any thoughts would be appreciated. Again I am updating from 18.2.2 on Rocky 8 to the same release (but different rpms), 18.2.2 on Rocky 9. -Chris

1 0

OSD processes crashes on repair 'unexpected clone'
by Thomas Björklund 28 May '24

28 May '24

Hi, We have an old cluster with 3 nodes running ceph version 15.2.17. We have a PG in state active+clean+inconsistent which we are unable to repair. It's an RBD pool in use by kubernetes. The earliest indication of the issue comes from ceph-osd.4.log on one of the nodes: *2024-05-23T14:34:38.003+0200 7f00b907c700 0 log_channel(cluster) log [DBG] : 20.4 repair starts2024-05-23T14:34:46.095+0200 7f00b907c700 -1 log_channel(cluster) log [ERR] : 20.4 shard 4 20:20a41d01:stc::rbd_data.44feb146358046.0000000000007737:38e : missing2024-05-23T14:34:46.095+0200 7f00b907c700 -1 log_channel(cluster) log [ERR] : 20.4 shard 16 20:20a41d01:stc::rbd_data.44feb146358046.0000000000007737:38e : missing2024-05-23T14:34:46.095+0200 7f00b907c700 -1 log_channel(cluster) log [ERR] : repair 20.4 20:20a41d01:stc::rbd_data.44feb146358046.0000000000007737:38e : is an unexpected clone2024-05-23T14:34:46.095+0200 7f00b907c700 -1 osd.4 pg_epoch: 30768 pg[20.4( v 30768'359661769 (30767'359657067,30768'359661769] local-lis/les=30660/30665 n=10628 ec=28562/20971 lis/c=30660/30660 les/c/f=30665/30665/0 sis=30660) [4,10,16] r=0 lpr=30660 luod=30768'359661760 crt=30768'359661769 lcod 30768'359661759 mlcod 30768'359661759 active+clean+scrubbing+deep+inconsistent+repair REQ_SCRUB] _scan_snaps no head for 20:20a41d01:stc::rbd_data.44feb146358046.0000000000007737:38e (have 20:20a42186:stc::rbd_data.44feb146358046.00000000000425b6:head)2024-05-23T14:35:27.712+0200 7f00b5074700 -1 log_channel(cluster) log [ERR] : 20.4 repair 1 missing, 0 inconsistent objects2024-05-23T14:35:27.712+0200 7f00b5074700 -1 log_channel(cluster) log [ERR] : 20.4 repair 3 errors, 2 fixed2024-05-23T14:35:27.720+0200 7f00b5074700 -1 ./src/osd/PrimaryLogPG.cc: In function 'int PrimaryLogPG::recover_missing(const hobject_t&, eversion_t, int, PGBackend::RecoveryHandle*)' thread 7f00b5074700 time 2024-05-23T14:35:27.718485+0200./src/osd/PrimaryLogPG.cc: 11540: FAILED ceph_assert(head_obc) ceph version 15.2.17 (542df8d06ef24dbddcf4994db16bcc4c89c9ec2d) octopus (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x131) [0x55a740013abb] 2: (()+0x9d5c46) [0x55a740013c46] 3: (PrimaryLogPG::recover_missing(hobject_t const&, eversion_t, int, PGBackend::RecoveryHandle*)+0x6cf) [0x55a7401fabbf] 4: (PrimaryLogPG::recover_primary(unsigned long, ThreadPool::TPHandle&)+0xf7e) [0x55a74023b16e] 5: (PrimaryLogPG::start_recovery_ops(unsigned long, ThreadPool::TPHandle&, unsigned long*)+0x5ee) [0x55a740246b4e] 6: (OSD::do_recovery(PG*, unsigned int, unsigned long, ThreadPool::TPHandle&)+0x295) [0x55a7400c4c85] 7: (ceph::osd::scheduler::PGRecovery::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x19) [0x55a74031c229] 8: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xf13) [0x55a7400e13c3] 9: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x41a) [0x55a74072ae6a] 10: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55a74072d410] 11: (()+0x7ea7) [0x7f00d59e2ea7] 12: (clone()+0x3f) [0x7f00d556ba6f]2024-05-23T14:35:27.724+0200 7f00b5074700 -1 *** Caught signal (Aborted) ** in thread 7f00b5074700 thread_name:tp_osd_tp ceph version 15.2.17 (542df8d06ef24dbddcf4994db16bcc4c89c9ec2d) octopus (stable) 1: (()+0x13140) [0x7f00d59ee140] 2: (gsignal()+0x141) [0x7f00d54a8ce1] 3: (abort()+0x123) [0x7f00d5492537] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x55a740013b05] 5: (()+0x9d5c46) [0x55a740013c46] 6: (PrimaryLogPG::recover_missing(hobject_t const&, eversion_t, int, PGBackend::RecoveryHandle*)+0x6cf) [0x55a7401fabbf] 7: (PrimaryLogPG::recover_primary(unsigned long, ThreadPool::TPHandle&)+0xf7e) [0x55a74023b16e] 8: (PrimaryLogPG::start_recovery_ops(unsigned long, ThreadPool::TPHandle&, unsigned long*)+0x5ee) [0x55a740246b4e] 9: (OSD::do_recovery(PG*, unsigned int, unsigned long, ThreadPool::TPHandle&)+0x295) [0x55a7400c4c85] 10: (ceph::osd::scheduler::PGRecovery::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x19) [0x55a74031c229] 11: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xf13) [0x55a7400e13c3] 12: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x41a) [0x55a74072ae6a] 13: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55a74072d410] 14: (()+0x7ea7) [0x7f00d59e2ea7] 15: (clone()+0x3f) [0x7f00d556ba6f] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.* Every time we try to start a repair on 20.4 all three OSD crashes (4, 10 & 16) *2024-05-28T11:14:30.573+0200 7f4d3405a700 0 log_channel(cluster) log [DBG] : 20.4 repair starts2024-05-28T11:14:39.481+0200 7f4d30052700 -1 log_channel(cluster) log [ERR] : 20.4 shard 10 soid 20:20a41d01:stc::rbd_data.44feb146358046.0000000000007737:38e : candidate had a read error2024-05-28T11:14:39.481+0200 7f4d30052700 -1 log_channel(cluster) log [ERR] : 20.4 soid 20:20a41d01:stc::rbd_data.44feb146358046.0000000000007737:38e : repairing object info data_digest2024-05-28T11:14:39.481+0200 7f4d30052700 -1 log_channel(cluster) log [ERR] : 20.4 soid 20:20a41d01:stc::rbd_data.44feb146358046.0000000000007737:38e : data_digest 0x43d61c5d != data_digest 0x867fac72 from auth oi 20:20a41d01:stc::rbd_data.44feb146358046.0000000000007737:38e(29034'359526115 osd.4.0:147412411 dirty|data_digest|omap_digest s 4194304 uv 287854203 dd 867fac72 od ffffffff alloc_hint [0 0 0])2024-05-28T11:14:39.481+0200 7f4d30052700 -1 log_channel(cluster) log [ERR] : 20.4 shard 16 20:20a41d01:stc::rbd_data.44feb146358046.0000000000007737:38e : missing2024-05-28T11:14:39.481+0200 7f4d30052700 -1 log_channel(cluster) log [ERR] : repair 20.4 20:20a41d01:stc::rbd_data.44feb146358046.0000000000007737:38e : is an unexpected clone2024-05-28T11:14:39.489+0200 7f4d30052700 -1 ./src/osd/PG.cc: In function 'void PG::update_snap_map(const std::vector<pg_log_entry_t>&, ObjectStore::Transaction&)' thread 7f4d30052700 time 2024-05-28T11:14:39.487768+0200./src/osd/PG.cc: 1225: FAILED ceph_assert(r == 0) ceph version 15.2.17 (542df8d06ef24dbddcf4994db16bcc4c89c9ec2d) octopus (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x131) [0x55c8140ebabb] 2: (()+0x9d5c46) [0x55c8140ebc46] 3: (PG::update_snap_map(std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> > const&, ceph::os::Transaction&)+0x6f8) [0x55c814241d28] 4: (PrimaryLogPG::log_operation(std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> > const&, std::optional<pg_hit_set_history_t> const&, eversion_t const&, eversion_t const&, eversion_t const&, bool, ceph::os::Transaction&, bool)+0x230) [0x55c814358510] 5: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr<PGTransaction, std::default_delete<PGTransaction> >&&, eversion_t const&, eversion_t const&, std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> > const&, std::optional<pg_hit_set_history_t>&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr<OpRequest>)+0x6cd) [0x55c8144df8ad] 6: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x53d) [0x55c8142b250d] 7: (PrimaryLogPG::simple_opc_submit(std::unique_ptr<PrimaryLogPG::OpContext, std::default_delete<PrimaryLogPG::OpContext> >)+0x9b) [0x55c8142b411b] 8: (PrimaryLogPG::scrub_snapshot_metadata(ScrubMap&, std::map<hobject_t, std::pair<std::optional<unsigned int>, std::optional<unsigned int> >, std::less<hobject_t>, std::allocator<std::pair<hobject_t const, std::pair<std::optional<unsigned int>, std::optional<unsigned int> > > > > const&)+0x24b1) [0x55c8142dc0e1] 9: (PG::scrub_compare_maps()+0x18db) [0x55c81424beeb] 10: (PG::chunky_scrub(ThreadPool::TPHandle&)+0x14a) [0x55c81424eb1a] 11: (ceph::osd::scheduler::PGScrub::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x1a) [0x55c8143f41fa] 12: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xf13) [0x55c8141b93c3] 13: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x41a) [0x55c814802e6a] 14: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55c814805410] 15: (()+0x7ea7) [0x7f4d509c0ea7] 16: (clone()+0x3f) [0x7f4d50549a6f]2024-05-28T11:14:39.497+0200 7f4d30052700 -1 *** Caught signal (Aborted) ** in thread 7f4d30052700 thread_name:tp_osd_tp ceph version 15.2.17 (542df8d06ef24dbddcf4994db16bcc4c89c9ec2d) octopus (stable) 1: (()+0x13140) [0x7f4d509cc140] 2: (gsignal()+0x141) [0x7f4d50486ce1] 3: (abort()+0x123) [0x7f4d50470537] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x55c8140ebb05] 5: (()+0x9d5c46) [0x55c8140ebc46] 6: (PG::update_snap_map(std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> > const&, ceph::os::Transaction&)+0x6f8) [0x55c814241d28] 7: (PrimaryLogPG::log_operation(std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> > const&, std::optional<pg_hit_set_history_t> const&, eversion_t const&, eversion_t const&, eversion_t const&, bool, ceph::os::Transaction&, bool)+0x230) [0x55c814358510] 8: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr<PGTransaction, std::default_delete<PGTransaction> >&&, eversion_t const&, eversion_t const&, std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> > const&, std::optional<pg_hit_set_history_t>&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr<OpRequest>)+0x6cd) [0x55c8144df8ad] 9: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x53d) [0x55c8142b250d] 10: (PrimaryLogPG::simple_opc_submit(std::unique_ptr<PrimaryLogPG::OpContext, std::default_delete<PrimaryLogPG::OpContext> >)+0x9b) [0x55c8142b411b] 11: (PrimaryLogPG::scrub_snapshot_metadata(ScrubMap&, std::map<hobject_t, std::pair<std::optional<unsigned int>, std::optional<unsigned int> >, std::less<hobject_t>, std::allocator<std::pair<hobject_t const, std::pair<std::optional<unsigned int>, std::optional<unsigned int> > > > > const&)+0x24b1) [0x55c8142dc0e1] 12: (PG::scrub_compare_maps()+0x18db) [0x55c81424beeb] 13: (PG::chunky_scrub(ThreadPool::TPHandle&)+0x14a) [0x55c81424eb1a] 14: (ceph::osd::scheduler::PGScrub::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x1a) [0x55c8143f41fa] 15: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xf13) [0x55c8141b93c3] 16: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x41a) [0x55c814802e6a] 17: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55c814805410] 18: (()+0x7ea7) [0x7f4d509c0ea7] 19: (clone()+0x3f) [0x7f4d50549a6f] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.* We are unsure on how to rectify the issue and get the cluster healthy again. Best regards, Thomas Björklund

1 1

Call Us - 8264348440 , Call Girls in Sarai Kale Khan - We brings offer model independent genuine call girls/Women Seeking Men Justdial Delhi escorts our high class luxury and premium call girl agency service We are the best leading delhi escorts service providers in delhi who can provide you with all kinds of hot and sexy female escorts near Oyo 5-star hotels. Suppose you want a unique and memorable experience with a hot and beautiful call girl. in that case, this is the perfect place for you.
by bedraj poudeel 28 May '24

28 May '24

1 0

Safe method to perform failback for RBD on one way mirroring.
by Saif Mohammad 28 May '24

28 May '24

Hello Everyone We have Clusters in production with the following configuration: Cluster-A : quincy v17.2.5 Cluster-B : quincy v17.2.5 All images in a pool have the snapshot feature enabled and are mirrored. Each site has 3 daemons. We're testing disaster recovery with one-way mirroring in our block device mirroring setup. On the primary site (Cluster-A), we have Ceph clients attached to it , and couple of images are present there. These images are replicated to the secondary site (Cluster-B). During testing we've successfully conducted failovers, with all resources accessible from Cluster-B once the Ceph client is attached there on secondary site. However, during failback (restoring the primary site), we've encountered an issue. Data that was pushed from the secondary site seems to be deleted, while data originally present only on the primary site remains intact. Here are the steps we took during failback: - Detached the client from Cluster-B. - Ensured that on Cluster-B, "mirroring primary" is set to true, and on Cluster-A, it's set to false. - Demoted the images on Cluster-B and promoted the images on Cluster-A. After performing these steps, our images went into an error state, and they started syncing from Cluster-A to Cluster-B. However, in a failback scenario, the direction should be from Cluster-B to Cluster-A. We are not sure where we are making mistakes. Could anybody please advise on the correct procedure for failback in one-way mirroring and the safest way to execute it to avoid impacting our data. Regards, Mohammad Saif

2 1

does the RBD client block write when the Watcher times out?
by Yuma Ogami 28 May '24

28 May '24

Hello. I'm currently verifying the behavior of RBD on failure. I'm wondering about the consistency of RBD images after network failures. As a result of my investigation, I found that RBD sets a Watcher to RBD image if a client mounts this volume to prevent multiple mounts. In addition, I found that if the client is isolated from the network for a long time, the Watcher is released. However, the client still mounts this image. In this situation, if another client can also mount this image and the image is writable from both clients, data corruption occurs. Could you tell me whether this is a realistic scenario? I tested the following case on which the watcher was released by hand and detected data corruption. 1. Release the Watcher on the node (A) that mounts the RBD using the `ceph osd blocklist add` command 2. Another node (B) mounts the RBD volume. 3. Unblock using the `ceph osd blocklist rm` command 4. Write from node (B) (write successfully) 5. Write from node (A) (can be written successfully from the application's point of view. In fact, the write fails) 6. Write content at node (A) is lost. In this case, I released the watcher by hand to emulate the timeout due to network failure. It's because I couldn't emulate real network failure in this test. I considered using exclusive lock and restricting access to those from a single node. However, we gave up on that as blocking writes entirely would make snapshots non-functional. The version of Ceph we are using is v17.2.6. Best regards, Yuma.

4 4

Ceph ARM providing storage for x86
by filip Mutterer 27 May '24

27 May '24

Is this known to be working: Setting up the Ceph Cluster with ARM and then use the storage with X86 Machines for example LXC, Docker and KVM? Is this possible? Greetings filip

3 2

RBD Mirror - Failed to unlink peer
by scott.cairns＠tecnica-ltd.co.uk 27 May '24

27 May '24

I've had rbd mirror deployed for over a year copying images from Site A to Site B for DR. After introducing an additional node into the cluster a few weeks back and creating OSD's on this node (no mon/mgr) we've started to see snapshots throwing an error 'librbd::mirror::snapshot::CreatePrimaryRequest: 0x7f24fc001bf0 handle_unlink_peer: failed to unlink peer: (2) No such file or directory' However, the snapshot still completes (provides a Snapshot ID) and RBD mirror still pulls the snapshot on the remote end, so it appears to be working as expected, however every snapshot is generating this error. I can't see anything else logged when this error is generated, it's progressively got worse since the start (the first day we saw around 2 errors on 150 snapshots, it's now about 145 errors on 150 snapshots) The only other reference I can see to this is from 2 years ago and was related to a bug which was fixed a while ago. I'm running Ceph 17.2.7. Any idea what file/directory it's having issues with? Thanks.

1 0

RBD Mirror - implicit snapshot cleanup
by scott.cairns＠tecnica-ltd.co.uk 27 May '24

27 May '24

With RBD Mirroring configured from Site A to Site B, we typically see 2 snapshots on the source side (current and 1 previous) and 1 snapshot on the destination side (most recent snapshot from source) - this has been the case for over a year now. When testing a DR scenario (i.e. demoting the primary, promoting the secondary, testing, resyncing, demoting the secondary, promoting the primary) we then result in a lot more snapshots on each side - currently around 6 on the source side (demoted image, then current and 4 previous) and 4 on the destination side (original snapshot, then snapshot from the promote, demote, then the current snapshot from source). Is it possible to clear these existing snapshots, or is there some sort of schedule that'll automatically purge them, so that we return to 2 snapshots on source and 1 snapshot on destination? Thanks.

1 0

ceph orch issue: lsblk: /dev/vg_osd/lvm_osd: not a block device
by duluxoz 27 May '24

27 May '24

Hi All, Is the following a bug or some other problem (I can't tell) :-) Brand new Ceph (Reef v18.2.3) install on Rocky Linux v9.4 - basically, its a brand new box. Ran the following commands (in order; no issues until final command): 1. pvcreate /dev/sda6 2. vgcreate vg_osd /dev/sda6 3. lvcreate -l 100%VG -n lv_osd vg_osd 4. cephadmbootstrap--mon-ip192.168.0.20 5. ceph orch daemon add osd ceph1:/dev/vg_osd/lvm_osd Received a whole bunch of error info on the console; the two relevant lines (as far as I can tell) are: * /usr/bin/podman: stderr stderr: lsblk: /dev/vg_osd/lvm_osd: not a block device * RuntimeError: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:257b3f5140c11b51fd710ffdad6213ed53d74146f464a51717262d156daef553 -e NODE_NAME=ceph1 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=None -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/477045f4-1b34-11ef-9a30-0800274c7359:/var/run/ceph:z -v /var/log/ceph/477045f4-1b34-11ef-9a30-0800274c7359:/var/log/ceph:z -v /var/lib/ceph/477045f4-1b34-11ef-9a30-0800274c7359/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/477045f4-1b34-11ef-9a30-0800274c7359/selinux:/sys/fs/selinux:ro -v /:/rootfs -v /etc/hosts:/etc/hosts:ro -v /tmp/ceph-tmpe_krhtt8:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp_47jsxdp:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:257b3f5140c11b51fd710ffdad6213ed53d74146f464a51717262d156daef553 lvm batch --no-auto /dev/vg_osd/lvm_osd --yes --no-systemd I had a look around the Net and couldn't find anything relevant. This post (https://github.com/rook/rook/issues/4967) talks about a similar issue using Rook, but I'm not using Rook but cephadm. Any help in resolving this (or confirming it is a bug) would be greatly appreciated - thanks in advance. Cheers Dulux-Oz

4 6

Problem in changing monitor address and public_network
by farhad kh 27 May '24

27 May '24

Hello, according to ceph own document and the article that I sent the link to, I tried to change the address of the ceph machines and its public network. But the guarantee that I had to set the machines with the new address(ceph orch host set-addr opcrgfpsksa0101 10.248.35.213) , the command was not executed and the managers were trying to connect to the same previous addresses. I have run my cluster with a non-root user and i change adress from 10.56.12.0/22 to 10.248.35/24. i used ceph version 17.2.6 and cephadm. Here, I don't have a configuration for the cluster_network and all communications are on the public_network. Also, I have sent the specifications and logs that I think are useful Are there any ideas and solutions to fix this problem and the cause of it? -------------- debug 2024-05-26T17:28:33.191+0000 7ffb15579700 0 log_channel(cephadm) log [INF] : Filtered out host opcrgfpsksa0101: does not belong to mon public_network(s): 10.248.35.0/24, host network(s): 10.56.12.0/22,172.17.0.0/16 debug 2024-05-26T17:28:33.191+0000 7ffb15579700 0 [cephadm INFO cephadm.serve] Filtered out host opcrgfpsksa0103: does not belong to mon public_network(s): 10.248.35.0/24, host network(s): 10.56.12.0/22,172.17.0.0/16 debug 2024-05-26T17:28:33.191+0000 7ffb15579700 0 log_channel(cephadm) log [INF] : Filtered out host opcrgfpsksa0103: does not belong to mon public_network(s): 10.248.35.0/24, host network(s): 10.56.12.0/22,172.17.0.0/16 debug 2024-05-26T17:28:33.192+0000 7ffb15579700 0 [cephadm INFO cephadm.serve] Filtered out host opcmrfpsksa0101: does not belong to mon public_network(s): 10.248.35.0/24, host network(s): 10.56.12.0/22,172.17.0.0/16 debug 2024-05-26T17:28:33.192+0000 7ffb15579700 0 log_channel(cephadm) log [INF] : Filtered out host opcmrfpsksa0101: does not belong to mon public_network(s): 10.248.35.0/24, host network(s): 10.56.12.0/22,172.17.0.0/16 debug 2024-05-26T17:28:33.192+0000 7ffb15579700 0 [progress WARNING root] complete: ev da5c20ec-e9df-490f-804d-182d02f0324e does not exist debug 2024-05-26T17:28:33.192+0000 7ffb15579700 0 [progress WARNING root] complete: ev 82392bf5-4940-4252-9aaf-aa4758c00ead does not exist debug 2024-05-26T17:28:33.212+0000 7ffb15579700 0 [progress WARNING root] complete: ev 1d64774f-dc6d-46c8-8f96-21d147f4b053 does not exist debug 2024-05-26T17:28:33.212+0000 7ffb15579700 0 [progress WARNING root] complete: ev 45322c73-8ecb-4471-b41c-0e279805dd0b does not exist debug 2024-05-26T17:28:35.182+0000 7ffb584e1700 0 log_channel(cluster) log [DBG] : pgmap v232: 1185 pgs: 1185 unknown; 0 B data, 0 B used, 0 B / 0 B avail debug 2024-05-26T17:28:37.183+0000 7ffb584e1700 0 log_channel(cluster) log [DBG] : pgmap v233: 1185 pgs: 1185 unknown; 0 B data, 0 B used, 0 B / 0 B avail debug 2024-05-26T17:28:38.609+0000 7ffb4f4df700 0 log_channel(audit) log [DBG] : from='client.14874149 -' entity='client.admin' cmd=[{"prefix": "orch host set-addr", "hostname": "opcrgfpsksa0101", "addr": "10.248.35.213", "target": ["mon-mgr", ""]}]: dispatch debug 2024-05-26T17:28:39.184+0000 7ffb584e1700 0 log_channel(cluster) log [DBG] : pgmap v234: 1185 pgs: 1185 unknown; 0 B data, 0 B used, 0 B / 0 B avail ------------ ceph orch host ls HOST ADDR LABELS STATUS opcmrfpsksa0101 10.56.12.216 _admin mon osd opcpmfpsksa0101 10.56.12.204 rgw opcpmfpsksa0103 10.56.12.205 rgw opcpmfpsksa0105 10.56.12.206 rgw opcrgfpsksa0101 10.56.12.213 _admin mon osd opcrgfpsksa0103 10.56.12.214 _admin mon osd opcsdfpsksa0101 10.56.12.207 osd opcsdfpsksa0103 10.56.12.208 osd opcsdfpsksa0105 10.56.12.209 osd 9 hosts in cluster ------------------------ NAME PORTS RUNNING REFRESHED AGE PLACEMENT alertmanager ?:9093,9094 3/3 8m ago 11M count:3 ceph-exporter 9/9 8m ago 7w * crash 9/9 8m ago 11M * grafana ?:3000 3/3 8m ago 11d count:3;label:mon ingress.rgw.k8s 10.56.12.215:80,1967 4/4 8m ago 11d count:2;label:rgw mds.k8s-cephfs 3/3 8m ago 11M count:3 mgr 2/2 7m ago 11d count:2;label:mon mon 3/3 8m ago 7M count:3;label:mon node-exporter ?:9100 9/9 8m ago 7w * osd 8 8m ago - <unmanaged> osd.dashboard-admin-1695638488579 37 8m ago 8M * prometheus ?:9095 3/3 8m ago 7w count:3;label:mon rgw.k8s ?:8080 3/3 8m ago 5M count:3;label:rgw

1 0

Lousy recovery for mclock and reef
by Mazzystr 24 May '24

24 May '24

Hi all, Goodness I'd say it's been at least 3 major releases since I had to do a recovery. I have disks with 60-75,000 power_on_hours. I just updated from Octopus to Reef last month and I'm hit with 3 disk failures and the mclock ugliness. My recovery is moving at a wondrous 21 mb/sec after some serious hacking. It started out at 9 mb/sec. My hosts are showing minimal cpu use. normal mem use. 0-6% disk business. Load is minimal so processes aren't blocked by disk io. I tried the changing all the sleeps and recovery_max and setting osd_mclock_profile high_recovery_ops to no change in performance. Does anyone have any suggestions to improve performance? Thanks, /Chris C

2 5

Reef RGWs stop processing requests
by Iain Stott 24 May '24

24 May '24

Hi, We are running 3 clusters in multisite. All 3 were running Quincy 17.2.6 and using cephadm. We upgraded one of the secondary sites to Reef 18.2.1 a couple of weeks ago and were planning on doing the rest shortly afterwards. We run 3 RGW daemons on separate physical hosts behind an external HAProxy HA pair for each cluster. Since we upgrade to Reef we have had issues with the RGWs stopping processing requests. We can see that they don't crash as they still have entries in the logs about syncing, but as far as request processing goes, they just stop. While debugging this we have 1 of the 3 RGWs running a Quincy image, and this has never had an issue where it stops processing requests. Any Reef containers we deploy have always stopped within 48Hrs of being deployed. We have tried Reef versions 18.2.1, 18.2.2 and 18.1.3 and all exhibit the same issue. We are running podman 4.6.1 on Centos 8 with kernel 4.18.0-513.24.1.el8_9.x86_64. We have enabled debug logs for the RGWs but we have been unable to find anything in them that would shed light on the cause. We are just wondering if anyone had any ideas on what could be causing this or how to debug it further? Thanks Iain Iain Stott OpenStack Engineer Iain.Stott(a)thg.com [THG Ingenuity Logo]<https://www.thg.com> www.thg.com<https://www.thg.com/> [LinkedIn]<https://www.linkedin.com/company/thgplc/?originalSubdomain=uk> [Instagram] <https://www.instagram.com/thg> [X] <https://twitter.com/thgplc?lang=en>

2 2

Cephadm quincy 17.2.5 always shows slowops in all OSDs and Ceph orch stuck
by pahrialtkj＠gmail.com 24 May '24

24 May '24

Hi all, I'm running a quincy 17.2.5 ceph cluster, with 3 nodes that each have 3 disks with replica size 3 min_size 2, my cluster was running fine before, and suddenly it can't read and write because ceph slowops on all osd. I tried restarting osd, then degraded and misplaced data appeared, and slow ops reappeared. Any suggestions guys to solve this issues ? # ceph -s cluster: id: bb6590a9-155c-4830-aea7-ef4b5a098466 health: HEALTH_ERR 2 failed cephadm daemon(s) 1/282019 objects unfound (0.000%) noout flag(s) set Reduced data availability: 79 pgs inactive, 47 pgs peering Possible data damage: 1 pg recovery_unfound Degraded data redundancy: 26340/838103 objects degraded (3.143%), 16 pgs degraded, 11 pgs undersized 302 slow ops, oldest one blocked for 226773 sec, daemons [osd.2,osd.3,osd.4,osd.6,osd.7] have slow ops. services: mon: 3 daemons, quorum r2c2,r2c3,r2c1 (age 2h) mgr: r2c3(active, since 15m), standbys: r2c2, r2c1.ivwibd osd: 9 osds: 9 up (since 9m), 9 in (since 3d); 30 remapped pgs flags noout data: pools: 12 pools, 329 pgs objects: 282.02k objects, 1.3 TiB usage: 4.9 TiB used, 94 TiB / 99 TiB avail pgs: 24.924% pgs not active 26340/838103 objects degraded (3.143%) 5411/838103 objects misplaced (0.646%) 1/282019 objects unfound (0.000%) 240 active+clean 31 peering 23 activating 16 remapped+peering 7 activating+undersized+degraded+remapped 2 activating+remapped 2 active+undersized+degraded+remapped+backfilling 2 active+recovering+degraded 2 activating+degraded 1 active+recovery_unfound+undersized+degraded+remapped 1 activating+degraded+remapped 1 active+clean+scrubbing+deep 1 active+recovery_wait+undersized+degraded+remapped io: client: 12 KiB/s rd, 1.3 KiB/s wr, 7 op/s rd, 1 op/s wr Thanks, Pahrial

1 0

User + Dev Meetup Tomorrow!
by Laura Flores 24 May '24

24 May '24

Hi all, The User + Dev Meetup will be held tomorrow at 10:00 AM EDT. We will be discussing the results of the latest survey, and users who attend will have the opportunity to provide additional feedback in real time. See you there! Laura Flores Meeting Details: https://www.meetup.com/ceph-user-group/events/300883526/ -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

3 4

quincy rgw with ceph orch and two realms only get answers from first realm
by Boris 24 May '24

24 May '24

Hi, we are currently in the process of adopting the main s3 cluster to orchestrator. We have two realms (one for us and one for the customer). The old config worked fine and depending on the port I requested, I got different x-amz-request-id header back: x-amz-request-id: tx00000307170ac0d734ab4-0066508120-94aa0f66-eu-central-1 x-amz-request-id: tx00000a9d8fc1821bbe258-00665081d1-949b2447-eu-customer-1 I then deployed the orchestrator config to other systems and checked, but now I get always an answer from the eu-central-1 zone and never from the customer zone. The test is very simple and works fine for the old rgw config, but fails for the new orchestrator config: curl [IPv6]:7481/somebucket -v All rgw instances are running on ceph version 17.2.7. Configs attached below. The old ceph.conf looked like this: [client.eu-central-1-s3db1-old] rgw_frontends = beast endpoint=[::]:7480 rgw_region = eu rgw_zone = eu-central-1 rgw_dns_name = example.com rgw_dns_s3website_name = eu-central-1.example.com [client.eu-customer-1-s3db1] rgw_frontends = beast endpoint=[::]:7481 rgw_region = eu-customer rgw_zone = eu-customer-1 rgw_dns_name = s3.customer.domain rgw_dns_s3website_name = eu-central-1.s3.customer.domain And this is the new service.yaml service_type: rgw service_id: ab12 placement: label: rgw config: debug_rgw: 0 rgw_thread_pool_size: 2048 rgw_dns_name: example.com rgw_dns_s3website_name: eu-central-1.example.com rgw_enable_gc_threads: false spec: rgw_frontend_port: 7480 rgw_frontend_type: beast rgw_realm: company rgw_zone: eu-central-1 rgw_zonegroup: eu --- service_type: rgw service_id: customer placement: label: rgw config: debug_rgw: 0 rgw_dns_name: s3.customer.domain rgw_dns_s3website_name: eu-central-1.s3.customer.domain rgw_enable_gc_threads: false spec: rgw_frontend_port: 7481 rgw_frontend_type: beast rgw_realm: customer rgw_zone: eu-customer-1 rgw_zonegroup: eu-customer -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

1 1

cephadm bootstraps cluster with bad CRUSH map(?)
by Matthew Vernon 24 May '24

24 May '24

Hi, I'm probably Doing It Wrong here, but. My hosts are in racks, and I wanted ceph to use that information from the get-go, so I tried to achieve this during bootstrap. This has left me with a single sad pg: [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive pg 1.0 is stuck inactive for 33m, current state unknown, last acting [] ceph osd tree shows that CRUSH picked up my racks OK, eg. -3 45.11993 rack B4 -2 45.11993 host moss-be1001 1 hdd 3.75999 osd.1 up 1.00000 1.00000 But root seems empty: -1 0 root default and if I decompile the crush map, indeed: # buckets root default { id -1 # do not change unnecessarily id -14 class hdd # do not change unnecessarily # weight 0.00000 alg straw2 hash 0 # rjenkins1 } which does indeed look empty, whereas I have rack entries that contain the relevant hosts. And the replication rule: rule replicated_rule { id 0 type replicated step take default step chooseleaf firstn 0 type rack step emit } I passed this config to bootstrap with --config: [global] osd_crush_chooseleaf_type = 3 and an initial spec file with host entries like this: service_type: host hostname: moss-be1001 addr: 10.64.16.40 location: rack: B4 labels: - _admin - NVMe Once the cluster was up I used an osd spec file that looked like: service_type: osd service_id: rrd_single_NVMe placement: label: "NVMe" spec: data_devices: rotational: 1 db_devices: model: "NVMe" I could presumably fix this up by editing the crushmap (to put the racks into the default bucket), but what did I do wrong? Was this not a reasonable thing to want to do with cephadm? I'm running ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable) Thanks, Matthew

3 9

Help with deep scrub warnings
by Nicola Mori 24 May '24

24 May '24

Dear Ceph users, in order to reduce the deep scrub load on my cluster I set the deep scrub interval to 2 weeks, and tuned other parameters as follows: # ceph config get osd osd_deep_scrub_interval 1209600.000000 # ceph config get osd osd_scrub_sleep 0.100000 # ceph config get osd osd_scrub_load_threshold 0.300000 # ceph config get osd osd_deep_scrub_randomize_ratio 0.100000 # ceph config get osd osd_scrub_min_interval 259200.000000 # ceph config get osd osd_scrub_max_interval 1209600.000000 In my admittedly poor knowledge of Ceph's deep scrub procedures, these settings should spread the deep scrub operations in two weeks instead of the default one week, lowering the scrub frequency and the related load. But I'm currently getting warnings like: [WRN] PG_NOT_DEEP_SCRUBBED: 56 pgs not deep-scrubbed in time pg 3.1e1 not deep-scrubbed since 2024-02-22T00:22:55.296213+0000 pg 3.1d9 not deep-scrubbed since 2024-02-20T03:41:25.461002+0000 pg 3.1d5 not deep-scrubbed since 2024-02-20T09:52:57.334058+0000 pg 3.1cb not deep-scrubbed since 2024-02-20T03:30:40.510979+0000 . . . I don't understand the first one, since the deep scrub interval should be two weeks so I don''t expect warnings for PGs which have been deep-scrubbed less than 14 days ago (at the moment I'm writing it's Tue Mar 5 07:39:07 UTC 2024). Moreover, I don't understand why the deep scrub for so many PGs is lagging behind. Is there something wrong in my settings? Thanks in advance for any help, Nicola

4 4

Traefik front end with RGW
by Reid Guyett 23 May '24

23 May '24

Hello, We are considering moving from Nginx to Traefik as the frontend for our RGW services. Prior to putting into production I ran it through s3-tests and noticed that all of the tests involving metadata (x-amz-meta-*) are failing because they are expected to be lowercase (test_s3.py::test_object_set_get_metadata_none_to_good - KeyError: 'meta1'). Traefik / Go converts these to have the first letter uppercase (X-Amz-Meta-Key). Poking around I found https://github.com/traefik/traefik/issues/466 and it points out RFC 2616 which says these should be case insensitive. AWS S3 says "User-defined metadata is a set of key-value pairs. Amazon S3 stores user-defined metadata keys in lowercase." over at https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingMetadata.html. Is anybody using Traefik for their frontend to RGW? Do you mitigate this if so and how? Thanks for any input. Reid

1 0

Best practice regarding rgw scaling
by Szabo, Istvan (Agoda) 23 May '24

23 May '24

Hi, Wonder what is the best practice to scale RGW, increase the thread numbers or spin up more gateways? * Let's say I have 21000 connections on my haproxy * I have 3 physical gateway servers so let's say each of them need to server 7000 connections This means with 512 thread pool size each of them needs 13 gateway altogether 39 in the cluster. or 3 gateway and each 8192 rgw thread? Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

3 2

Status of 18.2.3
by Sake Ceph 23 May '24

23 May '24

I was wondering what happened to the release of 18.2.3? Validation started on April 13th and as far as I know there have been a couple of builds and some extra bug fixes. Is there a way to follow a release or what is holding it back? Normally I wouldn't ask about a release and just wait, but I really need some fixes of this release. Kind regards, Sake

2 2

rgw can't find zone
by stephan.budach＠jvm.com 23 May '24

23 May '24

I am running two Ceph clusters on 18.2.2. On both clusters, I initially configured rgw using the following config from a yaml file: service_type: rgw service_id: s3.jvm.gh79 placement: label: rgw count_per_host: 1 spec: rgw_realm: jvm rgw_zone: gh79 ssl: true rgw_frontend_port: 9443 rgw_frontend_type: beast rgw_frontend_ssl_certificate: | Note, that I didn't configure anything else regarding realms, zonegroups or zones. I then modified the tags on 3 of my cephnodes to include rgw and the services got installed, configured and started. I am pretty sure, that I did this after I updated Ceph to 18.2.2. A couple of days ago, one of my cephnodes, hosting a rgw rebootet and the rgw service didn't come up again. When I looked at the logs I saw that the rgw service had an error when looking up the zone - which seems non-existent at this point: Mai 23 12:43:00 bceph06 radosgw[3608419]: deferred set uid:gid to 167:167 (ceph:ceph) Mai 23 12:43:00 bceph06 radosgw[3608419]: ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable), process radosgw, pid 2 Mai 23 12:43:00 bceph06 radosgw[3608419]: framework: beast Mai 23 12:43:00 bceph06 radosgw[3608419]: framework conf key: ssl_port, val: 9443 Mai 23 12:43:00 bceph06 radosgw[3608419]: framework conf key: ssl_certificate, val: config://rgw/cert/rgw.s3.jvm.gh79 Mai 23 12:43:00 bceph06 radosgw[3608419]: init_numa not setting numa affinity Mai 23 12:43:01 bceph06 radosgw[3608419]: rgw main: ERROR: could not find zone (gh79) Mai 23 12:43:01 bceph06 radosgw[3608419]: rgw main: ERROR: failed to start notify service ((2) No such file or directory Mai 23 12:43:01 bceph06 radosgw[3608419]: rgw main: ERROR: failed to init services (ret=(2) No such file or directory) Mai 23 12:43:01 bceph06 ceph-5ccaf98c-ec2a-11ee-8293-bc2411f9733d-rgw-s3-jvm-gh79-bceph06-rwkczj[3608415]: 2024-05-23T10:43:01.014+0000 7fe0b87b3a80 -1 Couldn't init storage provider (RADOS) Mai 23 12:43:01 bceph06 radosgw[3608419]: Couldn't init storage provider (RADOS) The rados zone configs look like this at the moment: root@cephnode01:/# radosgw-admin zone list { "default_info": "e67d13fc-8343-4c88-9b1f-a8286f46f44a", "zones": [ "default" ] } So, why did it work in the first place and what would be the easiest way of resolving this?

1 0

Re: Pacific 16.2.15 and ceph-volume no-longer creating LVM on block.db partition
by Bruno Canning 23 May '24

23 May '24

I should add, we are not using cephadm. From: Bruno Canning Sent: 23 May 2024 11:36 To: ceph-users(a)ceph.io Subject: Pacific 16.2.15 and ceph-volume no-longer creating LVM on block.db partition Hi Folks, After recently upgrading from 16.2.13 to 16.2.15, when I run: ceph-volume lvm create --data /dev/sda --block.db /dev/nvme0n1p1 to create a new OSD after replacement of a failed disk, ceph-volume no-longer creates a volume group/logical volume on the block.db partition*. This is the old behaviour we saw whilst running Octopus but we have not seen this since our upgrade to Pacific until now. Any ideas why this is happening? Nothing is jumping out of the release notes, but they do mention a lot of work was done on the LVM code in ceph-volume since 16.2.13. Is this now a configuration setting somewhere? Background: Our storage nodes are 60 HDDs per chassis with two 2TB NVMe drives, each containing 30 partitions used as block.db storage, one partition and one HDD per OSD. We are running Ubuntu 20.04 and Ceph Pacific 16.2.15, just upgraded from 16.2.13. We are using the official Ubuntu packages. *so /var/lib/ceph/osd/ceph-1090/block.db points to /dev/nvme0n1p1 and not /dev/ceph-11992480-4ea6-4c1a-80b0-e025a66539b2/osd-db-52655aa3-9102-4d92-8a63-f2c93eed984f which would be on /dev/nvme0n1p1 ---------------------------------------------------------------------- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is Wellcome Sanger Institute, Wellcome Genome Campus, Hinxton, CB10 1SA.

1 0

Pacific 16.2.15 and ceph-volume no-longer creating LVM on block.db partition
by Bruno Canning 23 May '24

23 May '24

Hi Folks, After recently upgrading from 16.2.13 to 16.2.15, when I run: ceph-volume lvm create --data /dev/sda --block.db /dev/nvme0n1p1 to create a new OSD after replacement of a failed disk, ceph-volume no-longer creates a volume group/logical volume on the block.db partition*. This is the old behaviour we saw whilst running Octopus but we have not seen this since our upgrade to Pacific until now. Any ideas why this is happening? Nothing is jumping out of the release notes, but they do mention a lot of work was done on the LVM code in ceph-volume since 16.2.13. Is this now a configuration setting somewhere? Background: Our storage nodes are 60 HDDs per chassis with two 2TB NVMe drives, each containing 30 partitions used as block.db storage, one partition and one HDD per OSD. We are running Ubuntu 20.04 and Ceph Pacific 16.2.15, just upgraded from 16.2.13. We are using the official Ubuntu packages. *so /var/lib/ceph/osd/ceph-1090/block.db points to /dev/nvme0n1p1 and not /dev/ceph-11992480-4ea6-4c1a-80b0-e025a66539b2/osd-db-52655aa3-9102-4d92-8a63-f2c93eed984f which would be on /dev/nvme0n1p1 ---------------------------------------------------------------------- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is Wellcome Sanger Institute, Wellcome Genome Campus, Hinxton, CB10 1SA.

1 0

cephfs-data-scan orphan objects while mds active?
by Olli Rajala 22 May '24

22 May '24

Hi, I suspect that I have some orphan objects on a data pool after quite haphazardly evicting and removing a cache pool after deleting 17TB of files from cephfs. I have forward scrubbed the mds and the filesystem is in clean state. This is a production system and I'm curious if it would be safe to run cephfs-data-scan scan_extents and scan_inodes while the fs is online? Does it help if I give a custom tag while forward scrubbing and then use --filter-tag on the backward scans? ...or is there some other way to check and cleanup orphans? tnx, --------------------------- Olli Rajala - Lead TD Anima Vitae Ltd. www.anima.fi ---------------------------

2 4

Re: How network latency affects ceph performance really with NVME only storage?
by Stefan Bauer 22 May '24

22 May '24

Hi Anthony and others, thank you for your reply. To be honest, I'm not even looking for a solution, i just wanted to ask if latency affects the performance at all in my case and how others handle this ;) One of our partners delivered a solution with a latency-optimized profile for tuned-daemon. Now the latency is much better: apt install tuned tuned-adm profile network-latency # ping 10.1.4.13 PING 10.1.4.13 (10.1.4.13) 56(84) bytes of data. 64 bytes from 10.1.4.13: icmp_seq=1 ttl=64 time=0.047 ms 64 bytes from 10.1.4.13: icmp_seq=2 ttl=64 time=0.028 ms 64 bytes from 10.1.4.13: icmp_seq=3 ttl=64 time=0.025 ms 64 bytes from 10.1.4.13: icmp_seq=4 ttl=64 time=0.020 ms 64 bytes from 10.1.4.13: icmp_seq=5 ttl=64 time=0.023 ms 64 bytes from 10.1.4.13: icmp_seq=6 ttl=64 time=0.026 ms 64 bytes from 10.1.4.13: icmp_seq=7 ttl=64 time=0.024 ms 64 bytes from 10.1.4.13: icmp_seq=8 ttl=64 time=0.023 ms 64 bytes from 10.1.4.13: icmp_seq=9 ttl=64 time=0.033 ms 64 bytes from 10.1.4.13: icmp_seq=10 ttl=64 time=0.021 ms ^C --- 10.1.4.13 ping statistics --- 10 packets transmitted, 10 received, 0% packet loss, time 9001ms rtt min/avg/max/mdev = 0.020/0.027/0.047/0.007 ms Am 21.05.24 um 15:08 schrieb Anthony D'Atri: > Check the netmask on your interfaces, is it possible that you're sending inter-node traffic up and back down needlessly? > >> On May 21, 2024, at 06:02, Stefan Bauer <sb(a)plzk.de> wrote: >> >> Dear Users, >> >> i recently setup a new ceph 3 node cluster. Network is meshed between all nodes (2 x 25G with DAC). >> Storage is flash only (Kioxia 3.2 TBBiCS FLASH 3D TLC, KCMYXVUG3T20) >> >> The latency with ping tests between the nodes shows: >> >> # ping 10.1.3.13 >> PING 10.1.3.13 (10.1.3.13) 56(84) bytes of data. >> 64 bytes from 10.1.3.13: icmp_seq=1 ttl=64 time=0.145 ms >> 64 bytes from 10.1.3.13: icmp_seq=2 ttl=64 time=0.180 ms >> 64 bytes from 10.1.3.13: icmp_seq=3 ttl=64 time=0.180 ms >> 64 bytes from 10.1.3.13: icmp_seq=4 ttl=64 time=0.115 ms >> 64 bytes from 10.1.3.13: icmp_seq=5 ttl=64 time=0.110 ms >> 64 bytes from 10.1.3.13: icmp_seq=6 ttl=64 time=0.120 ms >> 64 bytes from 10.1.3.13: icmp_seq=7 ttl=64 time=0.124 ms >> 64 bytes from 10.1.3.13: icmp_seq=8 ttl=64 time=0.140 ms >> 64 bytes from 10.1.3.13: icmp_seq=9 ttl=64 time=0.127 ms >> 64 bytes from 10.1.3.13: icmp_seq=10 ttl=64 time=0.143 ms >> 64 bytes from 10.1.3.13: icmp_seq=11 ttl=64 time=0.129 ms >> --- 10.1.3.13 ping statistics --- >> 11 packets transmitted, 11 received, 0% packet loss, time 10242ms >> rtt min/avg/max/mdev = 0.110/0.137/0.180/0.022 ms >> >> >> On another cluster i have much better values, with 10G SFP+ and fibre-cables: >> >> 64 bytes from large-ipv6-ip: icmp_seq=42 ttl=64 time=0.081 ms >> 64 bytes from large-ipv6-ip: icmp_seq=43 ttl=64 time=0.078 ms >> 64 bytes from large-ipv6-ip: icmp_seq=44 ttl=64 time=0.084 ms >> 64 bytes from large-ipv6-ip: icmp_seq=45 ttl=64 time=0.075 ms >> 64 bytes from large-ipv6-ip: icmp_seq=46 ttl=64 time=0.071 ms >> 64 bytes from large-ipv6-ip: icmp_seq=47 ttl=64 time=0.081 ms >> 64 bytes from large-ipv6-ip: icmp_seq=48 ttl=64 time=0.074 ms >> 64 bytes from large-ipv6-ip: icmp_seq=49 ttl=64 time=0.085 ms >> 64 bytes from large-ipv6-ip: icmp_seq=50 ttl=64 time=0.077 ms >> 64 bytes from large-ipv6-ip: icmp_seq=51 ttl=64 time=0.080 ms >> 64 bytes from large-ipv6-ip: icmp_seq=52 ttl=64 time=0.084 ms >> 64 bytes from large-ipv6-ip: icmp_seq=53 ttl=64 time=0.084 ms >> ^C >> --- long-ipv6-ip ping statistics --- >> 53 packets transmitted, 53 received, 0% packet loss, time 53260ms >> rtt min/avg/max/mdev = 0.071/0.082/0.111/0.006 ms >> >> If i want best performance, does the latency difference matter at all? Should i change DAC to SFP-transceivers wwith fibre-cables to improve overall ceph performance or is this nitpicking? >> >> Thanks a lot. >> >> Stefan >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io -- Mit freundlichen Grüßen Stefan Bauer Schulstraße 5 83308 Trostberg 0179-1194767

2 3

Cephfs over internet
by Marcus 22 May '24

22 May '24

Hi all, Due to so many reasons (political, heating problems, lack of space aso.) we have to plan for our ceph cluster to be hosted externaly. The planned version to setup is reef. Reading up on documentation we found that it was possible to run in secure mode. Our ceph.conf file will state both v1 and v2 addresses for mons: mon host = [v2:4.3.2.1:3300/0,v1:4.3.2.1:6789/0] [v2:4.3.2.2:3300/0,v1:4.3.2.2:6789/0] [v2:4.3.2.3:3300/0,v1:4.3.2.3:6789/0] Then changing the following configuration options to only secure: ms_cluster_mode = secure ms_service_mode = secure ms_client_mode = secure ms_mon_cluster_mode = secure ms_mon_service_mode = secure ms_mon_client_mode = secure Then I remounted cephfs on the clients on our test cluster, but still the fs would mount on ports 6789. I thought that the above secure config change would "force" the mount on port 3300 and v2. Mounting with option ms_mode=secure, did the trick. Is that the way cephfs is working that you explicit have to specify secure mode? I thought that cephfs clients would use the secure mode with these settings, but maybe I am wrong? Of cause we also plan to limit the firewalls on servers so only the specific subnet will be able to connect and mount cephfs. From my understanding from the documenation this would be the way to set this up with ceph exposed to internet. Is there something that we are missing or something that would make the setup more secure? Many thanks in advance Marcus

11 18

Problems adding a new host via orchestration.
by Gary Molenkamp 22 May '24

22 May '24

Happy Friday all. I was hoping someone could point me in the right direction or clarify any limitations that could be impacting an issue I am having. I'm struggling to add a new set of hosts to my ceph cluster using cephadm and orchestration. When trying to add a host: "ceph orch host add <hostname> 172.31.102.41 --labels _admin" returns: "Error EINVAL: Can't communicate with remote host `172.31.102.41`, possibly because python3 is not installed there: [Errno 12] Cannot allocate memory" I've verified that the ceph ssh key works to the remote host, host's name matches that returned from `hostname`, python3 is installed, and "/usr/sbin/cephadm prepare-host" on the new hosts returns "host is ok". In addition, the cluster ssh key works between hosts and the existing hosts are able to ssh in using the ceph key. The existing ceph cluster is Pacific release using docker based containerization on RockyLinux8 base OS. The new hosts are RockyLinux9 based, with the cephadm being installed from Quincy release: ./cephadm add-repo --release quincy ./cephadm install I did try installing cephadm from the Pacific release by changing the repo to el8, but that did not work either. Is there a limitation is mixing RL8 and RL9 container hosts under Pacific? Does this same limitation exist under Quincy? Is there a python version dependency? The reason for RL9 on the new hosts is to stage upgrading the OS's for the cluster. I did this under Octopus for moving from Centos7 to RL8. Thanks and I appreciate any feedback/pointers. Gary I've added the log trace here in case that helps (from `ceph log last cephadm`) 2024-02-02T14:22:32.610048+0000 mgr.storage01.oonvfl (mgr.441023307) 4957871 : cephadm [ERR] Can't communicate with remote host `172.31.102.41`, possibly because python3 is not installed there: [Errno 12] Cannot allocate memory Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/serve.py", line 1524, in _remote_connection conn, connr = self.mgr._get_connection(addr) File "/usr/share/ceph/mgr/cephadm/module.py", line 1370, in _get_connection sudo=True if self.ssh_user != 'root' else False) File "/lib/python3.6/site-packages/remoto/backends/__init__.py", line 35, in __init__ self.gateway = self._make_gateway(hostname) File "/lib/python3.6/site-packages/remoto/backends/__init__.py", line 46, in _make_gateway self._make_connection_string(hostname) File "/lib/python3.6/site-packages/execnet/multi.py", line 133, in makegateway io = gateway_io.create_io(spec, execmodel=self.execmodel) File "/lib/python3.6/site-packages/execnet/gateway_io.py", line 121, in create_io io = Popen2IOMaster(args, execmodel) File "/lib/python3.6/site-packages/execnet/gateway_io.py", line 21, in __init__ self.popen = p = execmodel.PopenPiped(args) File "/lib/python3.6/site-packages/execnet/gateway_base.py", line 184, in PopenPiped return self.subprocess.Popen(args, stdout=PIPE, stdin=PIPE) File "/lib64/python3.6/subprocess.py", line 729, in __init__ restore_signals, start_new_session) File "/lib64/python3.6/subprocess.py", line 1295, in _execute_child restore_signals, start_new_session, preexec_fn) OSError: [Errno 12] Cannot allocate memory During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/serve.py", line 1528, in _remote_connection raise execnet.gateway_bootstrap.HostNotFound(msg) execnet.gateway_bootstrap.HostNotFound: Can't communicate with remote host `172.31.102.41`, possibly because python3 is not installed there: [Errno 12] Cannot allocate memory The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 125, in wrapper return OrchResult(f(*args, **kwargs)) File "/usr/share/ceph/mgr/cephadm/module.py", line 2709, in apply results.append(self._apply(spec)) File "/usr/share/ceph/mgr/cephadm/module.py", line 2574, in _apply return self._add_host(cast(HostSpec, spec)) File "/usr/share/ceph/mgr/cephadm/module.py", line 1517, in _add_host ip_addr = self._check_valid_addr(spec.hostname, spec.addr) File "/usr/share/ceph/mgr/cephadm/module.py", line 1498, in _check_valid_addr error_ok=True, no_fsid=True) File "/usr/share/ceph/mgr/cephadm/serve.py", line 1326, in _run_cephadm with self._remote_connection(host, addr) as tpl: File "/lib64/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/usr/share/ceph/mgr/cephadm/serve.py", line 1558, in _remote_connection raise OrchestratorError(msg) from e orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.31.102.41`, possibly because python3 is not installed there: [Errno 12] Cannot allocate memory -- Gary Molenkamp Science Technology Services Systems Engineer University of Western Ontario molenkam(a)uwo.ca http://sts.sci.uwo.ca (519) 661-2111 x86882 (519) 661-3566

5 9

rbd-mirror failed to query services: (13) Permission denied
by Stefan Kooman 21 May '24

21 May '24

Hi, We are testing rbd-mirroring. There seems to be a permission error with the rbd-mirror user. Using this user to query the mirror pool status gives: failed to query services: (13) Permission denied And results in the following output: health: UNKNOWN daemon health: UNKNOWN image health: OK images: 3 total 2 replaying 1 stopped So, this command: rbd --id rbd-mirror mirror pool status rbd So basically the health and daemon health cannot be obtained due to permission errors, but status about images can. When the command is run with admin permissions the health and daemon health are returned without issue. I tested this on Reef 18.2.2. Is this expected behavior? If not, I will create a tracker ticket for it. Gr. Stefan

3 5

dkim on this mailing list
by Marc 21 May '24

21 May '24

Just to confirm if I am messing up my mailserver configs. But currently all messages from this mailing list should generate a dkim pass status?

2 1

Please discuss about Slow Peering
by 서민우 21 May '24

21 May '24

Env: - OS: Ubuntu 20.04 - Ceph Version: Octopus 15.0.0.1 - OSD Disk: 2.9TB NVMe - BlockStorage (Replication 3) Symptom: - Peering when OSD's node up is very slow. Peering speed varies from PG to PG, and some PG may even take 10 seconds. But, there is no log for 10 seconds. - I checked the effect of client VM's. Actually, Slow queries of mysql occur at the same time. There are Ceph OSD logs of both Best and Worst. Best Peering Case (0.5 Seconds) 2024-04-11T15:32:44.693+0900 7f108b522700 1 osd.7 pg_epoch: 27368 pg[6.8] state<Start>: transitioning to Primary 2024-04-11T15:32:45.165+0900 7f108f52a700 1 osd.7 pg_epoch: 27371 pg[6.8] state<Started/Primary/Peering>: Peering, affected_by_map, going to Reset 2024-04-11T15:32:45.165+0900 7f108f52a700 1 osd.7 pg_epoch: 27371 pg[6.8] start_peering_interval up [7,6,11] -> [6,11], acting [7,6,11] -> [6,11], acting_primary 7 -> 6, up_primary 7 -> 6, role 0 -> -1, features acting 2024-04-11T15:32:45.165+0900 7f108f52a700 1 osd.7 pg_epoch: 27377 pg[6.8] state<Start>: transitioning to Primary 2024-04-11T15:32:45.165+0900 7f108f52a700 1 osd.7 pg_epoch: 27377 pg[6.8] start_peering_interval up [6,11] -> [7,6,11], acting [6,11] -> [7,6,11], acting_primary 6 -> 7, up_primary 6 -> 7, role -1 -> 0, features acting Worst Peering Case (11.6 Seconds) 2024-04-11T15:32:45.169+0900 7f108b522700 1 osd.7 pg_epoch: 27377 pg[30.20] state<Start>: transitioning to Stray 2024-04-11T15:32:45.169+0900 7f108b522700 1 osd.7 pg_epoch: 27377 pg[30.20] start_peering_interval up [0,1] -> [0,7,1], acting [0,1] -> [0,7,1], acting_primary 0 -> 0, up_primary 0 -> 0, role -1 -> 1, features acting 2024-04-11T15:32:46.173+0900 7f108b522700 1 osd.7 pg_epoch: 27378 pg[30.20] state<Start>: transitioning to Stray 2024-04-11T15:32:46.173+0900 7f108b522700 1 osd.7 pg_epoch: 27378 pg[30.20] start_peering_interval up [0,7,1] -> [0,7,1], acting [0,7,1] -> [0,1], acting_primary 0 -> 0, up_primary 0 -> 0, role 1 -> -1, features acting 2024-04-11T15:32:57.794+0900 7f108b522700 1 osd.7 pg_epoch: 27390 pg[30.20] state<Start>: transitioning to Stray 2024-04-11T15:32:57.794+0900 7f108b522700 1 osd.7 pg_epoch: 27390 pg[30.20] start_peering_interval up [0,7,1] -> [0,7,1], acting [0,1] -> [0,7,1], acting_primary 0 -> 0, up_primary 0 -> 0, role -1 -> 1, features acting *I wish to know about* - Why some PG's take 10 seconds until Peering finishes. - Why Ceph log is quiet during peering. - Is this symptom intended in Ceph. *And please give some advice,* - Is there any way to improve peering speed? - Or, Is there a way to not affect the client when peering occurs? P.S - I checked the symptoms in the following environments. -> Octopus Version, Reef Version, Cephadm, Ceph-Ansible

3 7

How network latency affects ceph performance really with NVME only storage?
by Stefan Bauer 21 May '24

21 May '24

Dear Users, i recently setup a new ceph 3 node cluster. Network is meshed between all nodes (2 x 25G with DAC). Storage is flash only (Kioxia 3.2 TBBiCS FLASH 3D TLC, KCMYXVUG3T20) The latency with ping tests between the nodes shows: # ping 10.1.3.13 PING 10.1.3.13 (10.1.3.13) 56(84) bytes of data. 64 bytes from 10.1.3.13: icmp_seq=1 ttl=64 time=0.145 ms 64 bytes from 10.1.3.13: icmp_seq=2 ttl=64 time=0.180 ms 64 bytes from 10.1.3.13: icmp_seq=3 ttl=64 time=0.180 ms 64 bytes from 10.1.3.13: icmp_seq=4 ttl=64 time=0.115 ms 64 bytes from 10.1.3.13: icmp_seq=5 ttl=64 time=0.110 ms 64 bytes from 10.1.3.13: icmp_seq=6 ttl=64 time=0.120 ms 64 bytes from 10.1.3.13: icmp_seq=7 ttl=64 time=0.124 ms 64 bytes from 10.1.3.13: icmp_seq=8 ttl=64 time=0.140 ms 64 bytes from 10.1.3.13: icmp_seq=9 ttl=64 time=0.127 ms 64 bytes from 10.1.3.13: icmp_seq=10 ttl=64 time=0.143 ms 64 bytes from 10.1.3.13: icmp_seq=11 ttl=64 time=0.129 ms --- 10.1.3.13 ping statistics --- 11 packets transmitted, 11 received, 0% packet loss, time 10242ms rtt min/avg/max/mdev = 0.110/0.137/0.180/0.022 ms On another cluster i have much better values, with 10G SFP+ and fibre-cables: 64 bytes from large-ipv6-ip: icmp_seq=42 ttl=64 time=0.081 ms 64 bytes from large-ipv6-ip: icmp_seq=43 ttl=64 time=0.078 ms 64 bytes from large-ipv6-ip: icmp_seq=44 ttl=64 time=0.084 ms 64 bytes from large-ipv6-ip: icmp_seq=45 ttl=64 time=0.075 ms 64 bytes from large-ipv6-ip: icmp_seq=46 ttl=64 time=0.071 ms 64 bytes from large-ipv6-ip: icmp_seq=47 ttl=64 time=0.081 ms 64 bytes from large-ipv6-ip: icmp_seq=48 ttl=64 time=0.074 ms 64 bytes from large-ipv6-ip: icmp_seq=49 ttl=64 time=0.085 ms 64 bytes from large-ipv6-ip: icmp_seq=50 ttl=64 time=0.077 ms 64 bytes from large-ipv6-ip: icmp_seq=51 ttl=64 time=0.080 ms 64 bytes from large-ipv6-ip: icmp_seq=52 ttl=64 time=0.084 ms 64 bytes from large-ipv6-ip: icmp_seq=53 ttl=64 time=0.084 ms ^C --- long-ipv6-ip ping statistics --- 53 packets transmitted, 53 received, 0% packet loss, time 53260ms rtt min/avg/max/mdev = 0.071/0.082/0.111/0.006 ms If i want best performance, does the latency difference matter at all? Should i change DAC to SFP-transceivers wwith fibre-cables to improve overall ceph performance or is this nitpicking? Thanks a lot. Stefan

1 0

lost+found is corrupted.
by Malcolm Haak 21 May '24

21 May '24

Hi all, I've almost got my ceph back to normal after a triple drive failure. But it seems my lost+found folder is corrupted. I've followed the process in https://docs.ceph.com/en/latest/cephfs/disaster-recovery-experts/#disaster-… However doing an online scrub, as there is still other damage, fails as it appears my lost+found inode is corrupted. Any advice is appreciated The only thing I can think of is at the beginning of one of the steps it asks if I want to re-create something and, from what I could read from other emails to the list, I thought answering no was the correct answer but I am now wondering if I did want to recreate those entries. (specifically it asks at the cephfs-data-scan init point) Thanks in advance, ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable) Arch linux 3 node cluster 33 OSD's

1 0

CEPH quincy 17.2.5 with Erasure Code
by Andrea Martra 20 May '24

20 May '24

Hi everyone, I'm managing a Ceph Quincy 17.2.5 cluster, waiting to upgrade it to version 17.2.7, composed and configured as follows: - 16 identical nodes 256 GB RAM, 32 CPU Cores (64 threads), 12 x rotary HDD (BLOCK) + 4 x Sata SSD (RocksDB/WAL) - Erasure Code 11+4 (Jerasure) - 10 x S3 RGW on dedicated nodes (5 physical nodes) - 3 x full SSD dedicated nodes for replicated S3 pools - 2 x 10 Gbit Public network (LACP) + 2 x 10 Gbit cluster network (LACP) - On all nodes: Ubuntu 20.04.4 LTS Operating System updated - Ceph deployed on containers on Docker CE (docker-ce 5:20.10.17~3-0~ubuntu-focal). All pools, except the EC data pool, are configured with replication 3 and stored in dedicated SSD devices on 3 dedicated nodes to guarantee the necessary performance. We encountered a constant and random problem relating to the availability of access to bucket data and many slow_ops relating to rotary OSDs (data-pools) not caused by saturation of physical devices nor by the availability of CPU/RAM on all nodes. It sometimes happens that slow_ops caused by some requests to some PCs in a random way are reported. The cluster is currently in the recovery/rebalance phase for the reconstruction of 3 HDDs that we had to recreate from scratch (all 3 HDDs are physically on the same node). By doing some analysis of the events we verified the following from the status of some OSDs impacted by slow_ops: 20/05/2024 10:19 • "description": "osd_op(client.186021790.0:57620 29.258s0 29:1a5928ea:::31497ca8-e7d6-4e53-b150-91f9ac02ac67.246100.6329_storage%2ffirstMemories%2f1010543%2f:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e481205)", "initiated_at": "2024-05-16T15:03:06.015956+0000", "age": 963.83795990199997, "duration": 963.83819621700002, "type_data": { "flag_point": "delayed", "client_info": { "client": "client.186021790", "client_addr": "10.151.11.11:0/3913909849", "tid": 57620 }, "events": [ { "event": "initiated", "time": "2024-05-16T15:03:06.015956+0000", "duration": 0 }, { "event": "throttled", "time": "2024-05-16T15:03:06.015956+0000", "duration": 0 }, { "event": "header_read", "time": "2024-05-16T15:03:06.015954+0000", "duration": 4294967295.9999986 }, { "event": "all_read", "time": "2024-05-16T15:03:06.015961+0000", "duration": 7.2300000000000002e-06 }, { "event": "dispatched", "time": "2024-05-16T15:03:06.015962+0000", "duration": 1.063e-06 }, { "event": "queued_for_pg", "time": "2024-05-16T15:03:06.015966+0000", "duration": 3.332e-06 }, { "event": "reached_pg", "time": "2024-05-16T15:03:06.015992+0000", "duration": 2.6078e-05 }, { "event": "waiting for readable", "time": "2024-05-16T15:03:06.016002+0000", "duration": 1.0348e-05 } ] } } ], "num_ops": 6 } ########### { "event": "reached_pg", "time": "2024-05-16T12:43:11.694220+0000", "duration": 480.97258642200001 }, In essence the operations remain suspended in this condition: "event": "waiting for readable" trying to access some PGs. We intend to update the version as soon as the recovery/rebalance is completed. Does anyone have any idea what checks I could do to analyze the problem more thoroughly? I can't define whether the problem could be the use of the EC or whether the data written in some buckets is in a "non-standard" condition, which causes the access to wait for some reason. Thank you all for your kindness. Greetings Andrea Martra -- -- Andrea Martra +39 393 9048451

1 0

MDS behind on trimming every 4-5 weeks causing issue for ceph filesystem
by Akash Warkhade 20 May '24

20 May '24

Hi, We are using rook-ceph with operator 1.10.8 and ceph 17.2.5. we are using ceph filesystem with 4 mds i.e 2 active & 2 standby MDS every 3-4 weeks filesystem is having issue i.e in ceph status we can see below warnings warnings : 2 MDS reports slow requests 2 MDS Behind on Trimming mds.myfs-a(mds.1) : behind on trimming (6378/128) max_segments:128, num_segments: 6378 mds.myfs-c(mds.1): behind on trimming (6560/128) max_segments:128, num_segments: 6560 to fix it, we have to restart all MDS pods one by one. this is happening every 4-5 weeks. We have seen many ceph issues related to it on ceph tracker and many people are suggesting to increase mds_cache_memory_limit currently for our cluster *mds_cache_memory_limit* is set to default 4GB *mds_log_max_segments* is set to default 128 Should we increase *mds_cache_memory_limit* to 8GB from default 4GB or is there any solution to fix this issue permanently?

2 4

Ceph Squid release / release candidate timeline?
by Peter Sabaini 17 May '24

17 May '24

Hi, is there a ballpark timeline for a Squid release candidate / release? I'm aware of this pad that tracks blockers, is that still accurate or should I be looking at another resource? https://pad.ceph.com/p/squid-upgrade-failures Thanks! peter.

1 0

Reef RGWs stop processing requests
by Iain Stott 17 May '24

17 May '24

Hi, We are running 3 clusters in multisite. All 3 were running Quincy 17.2.6 and using cephadm. We upgraded one of the secondary sites to Reef 18.2.1 a couple of weeks ago and were planning on doing the rest shortly afterwards. We run 3 RGW daemons on separate physical hosts behind an external HAProxy HA pair for each cluster. Since we upgrade to Reef we have had issues with the RGWs stopping processing requests. We can see that they don't crash as they still have entries in the logs about syncing, but as far as request processing goes, they just stop. While debugging this we have 1 of the 3 RGWs running a Quincy image, and this has never had an issue where it stops processing requests. Any Reef containers we deploy have always stopped within 48Hrs of being deployed. We have tried Reef versions 18.2.1, 18.2.2 and 18.1.3 and all exhibit the same issue. We are running podman 4.6.1 on Centos 8 with kernel 4.18.0-513.24.1.el8_9.x86_64. We have enabled debug logs for the RGWs but we have been unable to find anything in them that would shed light on the cause. We are just wondering if anyone had any ideas on what could be causing this or how to debug it further? Thanks Iain Iain Stott OpenStack Engineer Iain.Stott(a)thg.com [THG Ingenuity Logo]<https://www.thg.com> www.thg.com<https://www.thg.com/> [LinkedIn]<https://www.linkedin.com/company/thgplc/?originalSubdomain=uk> [Instagram] <https://www.instagram.com/thg> [X] <https://twitter.com/thgplc?lang=en>

1 0

RadosGW Multisite Zonegroup redirect
by rabe＠telemaxx.de 17 May '24

17 May '24

Hi folks, I am currently in the process of building a Multisite Cluster (v18.2.2) with two Ceph clusters (cluster1 and cluster2). Each Cluster is its own Zone and has its own zonegroup. The idea is to sync metadata but not data. The user should select the zonegroup (region) via the location constraint. I figured out how to set up the two zonegroups but now the following problem appears: When i create a bucket via the cluster2 rgw with the cluster1 zonegroup as a target my minio client shows the following error: mc mb test/test1 --region cluster1 mc: <ERROR> Unable to make bucket `test/test1`. The specified location-constraint is not valid the cluster1 rgw shows the following error: s3:create_bucket location constraint (cluster1) doesn't match zonegroup (cluster2) the cluster2 rgw shows no logs When i do the same with the aws cli i get no error but the rgw seems to ignore the location constraint and just creates the bucket in cluster2: aws s3api --endpoint http://cluster2 --profile test create-bucket --bucket test2 --region cluster1 When I copy a file to a bucket in cluster2 via the cluster1 rgw i get the following error on the cluster1 rgw: op->ERRORHANDLER: err_no=-2024 new_err_no=-2024 The Period is synced on both clusters and the rgws are restarted: { "id": "7189ba3c-c7de-4ea2-8d14-307e4355a9e3", "epoch": 6, "predecessor_uuid": "0253dec1-d804-470f-92b6-90a64ffb5548", "sync_status": [], "period_map": { "id": "7189ba3c-c7de-4ea2-8d14-307e4355a9e3", "zonegroups": [ { "id": "4daa58d4-3eb5-4e59-bffe-e929f64c70b5", "name": "cluster1", "api_name": "cluster1", "is_master": true, "endpoints": [ "http://172.31.4.12:80" ], "hostnames": [ "172.31.4.12", "172.31.4.16" ], "hostnames_s3website": [], "master_zone": "ae3aa0e9-c580-410a-8ed6-2f01657a06f8", "zones": [ { "id": "ae3aa0e9-c580-410a-8ed6-2f01657a06f8", "name": "cluster1-a", "endpoints": [ "http://172.31.4.12:80" ], "log_meta": false, "log_data": false, "bucket_index_max_shards": 11, "read_only": false, "tier_type": "", "sync_from_all": true, "sync_from": [], "redirect_zone": "", "supported_features": [ "compress-encrypted", "resharding" ] } ], "placement_targets": [ { "name": "default-placement", "tags": [], "storage_classes": [ "STANDARD" ] } ], "default_placement": "default-placement", "realm_id": "5e8f4256-cf89-40db-b2a9-bb717f980b4f", "sync_policy": { "groups": [] }, "enabled_features": [ "resharding" ] }, { "id": "8d1d8050-0dea-4c5a-9bc3-d6631ef132f8", "name": "cluster2", "api_name": "cluster2", "is_master": false, "endpoints": [ "http://172.31.4.16:80" ], "hostnames": [ "172.31.4.12", "172.31.4.16" ], "hostnames_s3website": [], "master_zone": "2a882ccc-b540-4de2-abe3-f026fccb9aef", "zones": [ { "id": "2a882ccc-b540-4de2-abe3-f026fccb9aef", "name": "cluster2-a", "endpoints": [ "http://172.31.4.16:80" ], "log_meta": false, "log_data": false, "bucket_index_max_shards": 11, "read_only": false, "tier_type": "", "sync_from_all": true, "sync_from": [], "redirect_zone": "", "supported_features": [ "compress-encrypted", "resharding" ] } ], "placement_targets": [ { "name": "default-placement", "tags": [], "storage_classes": [ "STANDARD" ] } ], "default_placement": "default-placement", "realm_id": "5e8f4256-cf89-40db-b2a9-bb717f980b4f", "sync_policy": { "groups": [] }, "enabled_features": [ "resharding" ] } ], "short_zone_ids": [ { "key": "2a882ccc-b540-4de2-abe3-f026fccb9aef", "val": 4229298211 }, { "key": "ae3aa0e9-c580-410a-8ed6-2f01657a06f8", "val": 1754305595 } ] }, "master_zonegroup": "4daa58d4-3eb5-4e59-bffe-e929f64c70b5", "master_zone": "ae3aa0e9-c580-410a-8ed6-2f01657a06f8", "period_config": { "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "user_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "user_ratelimit": { "max_read_ops": 0, "max_write_ops": 0, "max_read_bytes": 0, "max_write_bytes": 0, "enabled": false }, "bucket_ratelimit": { "max_read_ops": 0, "max_write_ops": 0, "max_read_bytes": 0, "max_write_bytes": 0, "enabled": false }, "anonymous_ratelimit": { "max_read_ops": 0, "max_write_ops": 0, "max_read_bytes": 0, "max_write_bytes": 0, "enabled": false } }, "realm_id": "5e8f4256-cf89-40db-b2a9-bb717f980b4f", "realm_name": "testrealm", "realm_epoch": 2 } According to my understanding this setup should work but I probably missed some config setting. Any idea? Thanks in advance!

1 0

Write issues on CephFS mounted with root_squash
by Nicola Mori 17 May '24

17 May '24

Dear Ceph users, I'm trying to export a CephFS with the root_squash option. This is the client configuration: client.wizardfs_rootsquash key: XXXXXXXXXXXX caps: [mds] allow rw fsname=wizardfs root_squash caps: [mon] allow r fsname=wizardfs caps: [osd] allow rw tag cephfs data=wizardfs I can mount it flawlessly on several machines using the kernel driver, but when a machine writes on it then the content seems fine from the writing machine but it's not actually written on disk since other machines just see an empty file: [12:43 mori@stryke ~]$ echo test > /wizard/ceph/software/el9/test [12:43 mori@stryke ~]$ ll /wizard/ceph/software/el9/test -rw-r--r-- 1 mori wizard 5 mag 15 12:43 /wizard/ceph/software/el9/test [12:43 mori@stryke ~]$ cat /wizard/ceph/software/el9/test test [12:43 mori@stryke ~]$ [mori@fili ~]$ ll /wizard/ceph/software/el9/test -rw-r--r--. 1 mori 1014 0 May 15 06:43 /wizard/ceph/software/el9/test [mori@fili ~]$ cat /wizard/ceph/software/el9/test [mori@fili ~]$ Unmounting and then remounting on "stryke" the file is seen as empty, so I guess that the content shown just after the write is only a cache effect and nothing is effectively written on disk. I checked the posix permissions on the folder and I have rw rights from both the machines. All of the above using Ceph 18.2.2 on the cluster (deployed with cephadm) and both the machines. Machine "fili" has kernel 5.14.0 while "stryke" has 6.8.9. The same issue happens consistently also in the reverse direction (writing from "fili" and reading from "stryke"), and also with other machines. Removing the squash_root option the problem vanishes. I don't know what might

4 7

Low rate Call girls service In Mehrauli Delhi l Just hire me | 8800733197
by Call Girls 16 May '24

16 May '24

1 0

Reminder: User + Dev Monthly Meetup rescheduled to May 23rd
by Laura Flores 15 May '24

15 May '24

Hi all, Those of you subscribed to the Ceph Community Calendar may have already gotten an update that the meeting was rescheduled, but I wanted to send a reminder here as well. The meeting will be held at the usual time a week from now on May 23rd. If you haven't already, please take the new survey we put out! Results will be discussed at the meeting next week: https://www.meetup.com/ceph-user-group/events/300883526/ Thanks, Laura Meeting Details + Survey link here: https://www.meetup.com/ceph-user-group/events/300883526/ -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

1 0

ceph dashboard reef 18.2.2 radosgw
by Christopher Durham 15 May '24

15 May '24

Hello, I am uisng 18.2.2 on Rocky 8 Linux. I am getting http error 500 whe trying to hit the ceph dashboard on reef 18.2.2 when trying to look at any of the radosgw pages. I tracked this down to /usr/share/ceph/mgr/dashboard/controllers/rgw.py It appears to parse the metadata for a given radosgw server improperly. In my varoous rgw ceph.conf entries, I have: rgw frontends = beast ssl_endpoint=0.0.0.0 ssl_certificate=/path/to/pem_with_cert_and_key but, rgw.py pulls the metadata for each server, and it is looking for 'port=' in the metadata for each server. When it doesn't find it based on line 147 in rgw.py, the ceph-mgr logs throwan exception which the manager proper catches and returns a 500. Would changing my frontends definition work? Is this known? I have had the frontends definition for awhile prior to my reef upgrade. Thanks -Chris

3 3

ceph tell mds.0 dirfrag split - syntax of the "frag" argument
by Alexander E. Patrakov 15 May '24

15 May '24

Hello, In the context of https://tracker.ceph.com/issues/64298, I decided to do something manually. In the help output of "ceph tell" for an MDS, I found these possibly useful commands: dirfrag ls <path>: List fragments in directory dirfrag merge <path> <frag>: De-fragment directory by path dirfrag split <path> <frag> <bits:int>: Fragment directory by path They accept the "frag" argument that is underdocumented. In the testsuite, they are used, and it seems like this argument accepts some notation containing a slash, which is also produced as "str" by "dirfrag ls". Can anyone explain the meaning of the parts before and after the slash? What is the relation between the accepted values for "dirfrag split" and the output of "dirfrag ls" - do I just feed the fragment from "dirfrag ls" to "dirfrag split" as-is? Is running "dirfrag split" manually safe on a production cluster? Thanks in advance. -- Alexander Patrakov

1 0

Problem with take-over-existing-cluster.yml playbook
by vladimir franciz blando 15 May '24

15 May '24

I know that only a few are using this script but just trying my luck here if someone has the same issue as mine. But first, who has successfully used this script and what version did you use? Im using this guide on my test environment ( https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/3/html/i… ) Error encountered --- TASK [Generate ceph configuration file] ********************************************************************** *********************************************************************************** fatal: [vladceph-1]: FAILED! => msg: '''_monitor_addresses'' is undefined. ''_monitor_addresses'' is undefined' fatal: [vladceph-3]: FAILED! => msg: '''_monitor_addresses'' is undefined. ''_monitor_addresses'' is undefined' fatal: [vladceph-2]: FAILED! => msg: '''_monitor_addresses'' is undefined. ''_monitor_addresses'' is undefined' --- Regards, Vlad Blando <https://about.me/vblando>

4 9

Upgrading Ceph Cluster OS
by Nima AbolhassanBeigi 14 May '24

14 May '24

Hi, We want to upgrade the OS version of our production ceph cluster by reinstalling the OS on the server. Is there any way to upgrade the OS without having to redeploy the OSDs? We looked at the contents of /var/lib/ceph/osd/ceph-*, the block directory is linked to a lvm device. If we reinstall the OS, will there be any problem with this linked directory? Best Regards, Nima

4 4

Determine client/inode/dnode source of massive explosion in CephFS metadata pool usage (Red Hat Nautilus CephFS)
by Paul Browne 14 May '24

14 May '24

Hello Ceph users, We've recently seen a very massive uptick in the stored capacity of our CephFS metadata pool, 150X the raw stored capacity used in a very short timeframe of only 48 hours or so. The number of stored objects rose by ~1.5 million or so in that timeframe (attached PNG shows the increase) What I'd really like to be able to determine, but haven't yet figured out how, is to map these newly stored objects (over this limited time window) to inodes/dnodes in the filesystem and from there to individual namespaces being used in the filesystem. This should then allow me to track back the increased usage to specific projects using the filesystem for research data storage and give them a mild warning about possibly exhausting the available metadata pool capacity. Would anyone know if there's any capability in CephFs to do something like this, specifically in Nautilus (being run here as Red Hat Ceph Storage 4)? We've scheduled upgrades to later RHCS releases, but I'd like the cluster and CephFS state to be in a better place first if possible. Thanks, Paul Browne [cid:e87ad248-6621-4e9d-948b-da4428f8dbb8] ******************* Paul Browne Research Computing Platforms University Information Services Roger Needham Building JJ Thompson Avenue University of Cambridge Cambridge United Kingdom E-Mail: pfb29(a)cam.ac.uk<mailto:pfb29@cam.ac.uk> Tel: 0044-1223-746548 *******************

4 4

label or pseudo name for cephfs volume path
by Adiga, Anantha 14 May '24

14 May '24

Hi, Under the circumstance that a ceph fs subvolume has to be recreated , the uuid will change and we have to change all sources that reference the volume path. Is there a way to provide a label /tag to the volume path that can be used for pv_root_path so that we do not have to change any references OR any suggestion to address this need ? Spec containing subvolume path: k8s_rook_pv: - name: inventory-pv subvolumegroup: cephfs_data_pool_ec21_subvolumegroup subvolume: cluster_inventory_subvolume data_pool: cephfs_data_pool_ec21 size: 10Gi pvc_name: prometheus-inventory-pvc pv_root_path: /volumes/cephfs_data_pool_ec21_subvolumegroup/cluster_inventory_subvolume/ecfdf977-1fdb-4474-b8dd-c3bedb42620e mode: ReadWriteMany Thank you, Anantha

2 1

Ceph User + Community Meeting and Survey [May 23]
by Noah Lehman 13 May '24

13 May '24

Hi Ceph users and devs, We've announced the next community meeting and survey on Ceph social media and would appreciate help promoting. The next meeting will be Thursday, May 23rd at 10PM EDT. Details can be found here <https://www.meetup.com/ceph-user-group/events/300883526/?utm_campaign=share…> . Ceph twitter <https://twitter.com/Ceph/status/1788243792589427010>/xCeph linkedin <https://www.linkedin.com/feed/update/urn:li:share:7194009505439883264> *Language:* Our next Ceph User + Developer Monthly Meeting is coming May 23! The goal of these meetings is to elicit feedback from the users, companies, and organizations who use #Ceph in their production environments. Take the survey and share your voice: https://t.co/xGAN5FlGDu Thanks! Noah

2 2

Multisite: metadata behind on shards
by Szabo, Istvan (Agoda) 13 May '24

13 May '24

Hi, Wonder what is the mechanism behind the sync mechanism because I need to restart all the gateways every 2 days on the remote sites to keep those it in sync. (Octopus 15.2.7) Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 1

Re: SPDK with cephadm and reef
by xiaowenhao111 13 May '24

13 May '24

1 0

Ceph reef and (slow) backfilling - how to speed it up
by Götz Reinicke 12 May '24

12 May '24

Dear ceph community, I’ve a ceph cluster which got upgraded from nautilus/pacific/…to reef over time. Now I added two new nodes to an existing EC pool as I did with the previous versions of ceph. Now I face the fact, that the previous „backfilling tuning“ I’v used by increasing injectargs --osd-max-backfills=XX --osd-recovery-max-active=YY dose not work anymore. With adjusting thous parameters the backfill was running with up to 2k +- objects/s. As I’m not (yet) familiar with the reef opiont the only speed up so far I found is „ceph config set osd osd_mclock_profile high_recovery_ops“ which currently runs the backfill with up to 600 opbjects/s. My question: What is a best (simple) way to speed that backfill up ? I’v tried to understand the custom profiles (?) but without success - and did not apply anything other yet. Thanks for feedback and suggestions ! Best regards . Götz

7 10