February 2024 - ceph-users

could not find secret_id
by xiaowenhao111 22 Apr '26

22 Apr '26

2 1

OpenSSL security update for Octopus container?
by Robert Sander 02 Oct '25

02 Oct '25

Hi, I am running a Ceph Octopus cluster in version 15.2.10 setup with the cephadm orchestrator and official Docker hub container images. Yesterday an important security fix was released for libssl and is already packaged for all major distributions. I tried to run "ceph orch upgrade check docker.io/ceph/ceph:v15" but it tells me that the containers do not need to be upgraded. How will this security fix of OpenSSL be deployed in a timely manner to users of the Ceph container images? Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 93818 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 1

Problem while upgrade 17.2.6 to 17.2.7
by Jean-Marc FONTANA 21 Feb '25

21 Feb '25

Hello everyone, We operate two clusters that we installed with ceph-deploy in Nautilus version on Debian 10. We use them for external S3 storage (owncloud) and rbd disk images.We had them upgraded to Octopus and Pacific versions on Debian 11 and recently converted them to cephadm and upgraded to Quincy (17.2.6). As we now have the orchestrator, we tried updating to 17.2.7 using the command# ceph orch upgrade start --image quay.io/ceph/ceph:v17.2.7 Everything went well, both clusters work perfectly for our use, except that the Rados gateway configuration is no longer accessible from the dashboard with the following error messageError connecting to Object Gateway: RGW REST API failed request with status code 404. We tried a few solutions found on the internet (reset rgw credentials, restart rgw adnd mgr, reenable dashboard, ...), unsuccessfully. Does somebody have an idea ? Best regards, Jean-Marc Fontana

5 9

Problems with autoscaler (overlapping roots) after changing the pool class
by Massimo Sgaravatto 06 Jan '25

06 Jan '25

Dear all I have just changed the crush rule for all the replicated pools in the following way: ceph osd crush rule create-replicated replicated_hdd default host hdd ceph osd pool set <poolname> crush_rule replicated_hdd See also this [*] thread Before applying this change, these pools were all using the replicated_ruleset rule where the class is not specified. I am noticing now a problem with the autoscaler: "ceph osd pool autoscale-status" doesn't report any output and the mgr log complains about overlapping roots: [pg_autoscaler ERROR root] pool xyz has overlapping roots: {-18, -1} Indeed: # ceph osd crush tree --show-shadow ID CLASS WEIGHT TYPE NAME -18 hdd 1329.26501 root default~hdd -17 hdd 329.14154 rack Rack11-PianoAlto~hdd -15 hdd 54.56085 host ceph-osd-04~hdd 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... ... -1 1329.26501 root default -7 329.14154 rack Rack11-PianoAlto -8 54.56085 host ceph-osd-04 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... I have already read about this behavior but I have no clear ideas how to fix the problem. I read somewhere that the problem happens when there are rules that force some pools to only use one class and there are also pools which does not make any distinction between device classes All the replicated pools are using the replicated_hdd pool but I also have some EC pools which are using a profile where the class is not specified. As far I understand, I can't force these pools to use only the hdd class: according to the doc I can't change this profile specifying the hdd class (or at least the change wouldn't be applied to the existing EC pools) Any suggestions ? The crush map is available at https://cernbox.cern.ch/s/gIyjbQbmoTFHCrr, if you want to have a look Many thanks, Massimo [*] https://www.mail-archive.com/ceph-users@ceph.io/msg18534.html

4 9

multifs and snapshots
by Dmitry Melekhov 12 Nov '24

12 Nov '24

Hello! There is note here https://docs.ceph.com/en/reef/dev/cephfs-snapshots/ about multiply filesystems and snapshots: If each FS gets its own pool things probably work, but this isn’t tested and may not be true. Is it still untested ? Thank you!

3 4

Help needed to configure erasure coding LRC plugin
by Michel Jouvin 16 Oct '24

16 Oct '24

Hi, As discussed in another thread (Crushmap rule for multi-datacenter erasure coding), I'm trying to create an EC pool spanning 3 datacenters (datacenters are present in the crushmap), with the objective to be resilient to 1 DC down, at least keeping the readonly access to the pool and if possible the read-write access, and have a storage efficiency better than 3 replica (let say a storage overhead <= 2). In the discussion, somebody mentioned LRC plugin as a possible jerasure alternative to implement this without tweaking the crushmap rule to implement the 2-step OSD allocation. I looked at the documentation (https://docs.ceph.com/en/latest/rados/operations/erasure-code-lrc/) but I have some questions if someone has experience/expertise with this LRC plugin. I tried to create a rule for using 5 OSDs per datacenter (15 in total), with 3 (9 in total) being data chunks and others being coding chunks. For this, based of my understanding of examples, I used k=9, m=3, l=4. Is it right? Is this configuration equivalent, in terms of redundancy, to a jerasure configuration with k=9, m=6? The resulting rule, which looks correct to me, is: -------- { "rule_id": 6, "rule_name": "test_lrc_2", "ruleset": 6, "type": 3, "min_size": 3, "max_size": 15, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -4, "item_name": "default~hdd" }, { "op": "choose_indep", "num": 3, "type": "datacenter" }, { "op": "chooseleaf_indep", "num": 5, "type": "host" }, { "op": "emit" } ] } ------------ Unfortunately, it doesn't work as expected: a pool created with this rule ends up with its pages active+undersize, which is unexpected for me. Looking at 'ceph health detail` output, I see for each page something like: pg 52.14 is stuck undersized for 27m, current state active+undersized, last acting [90,113,2147483647,103,64,147,164,177,2147483647,133,58,28,8,32,2147483647] For each PG, there is 3 '2147483647' entries and I guess it is the reason of the problem. What are these entries about? Clearly it is not OSD entries... Looks like a negative number, -1, which in terms of crushmap ID is the crushmap root (named "default" in our configuration). Any trivial mistake I would have made? Thanks in advance for any help or for sharing any successful configuration? Best regards, Michel

4 24

rgw connection resets
by Nathan Gleason 10 Oct '24

10 Oct '24

Hello, I’ve noticed what I think may be strange behavior from some rgw transfers. It only seems to be with larger multipart mp4 files. Once a connection comes into rgw it is accepted and processed, data begins to transfer, then I see "write_data failed: Connection reset by peer” and "s3:get_obj iterate_obj() failed with -104”. The client then reconnects and the transfer seems to be ok. This doesn’t appear to happen just downloading a regular file. This is happening to both my production and test cluster. I’m not seeing errors anywhere else, the network looks good and the clusters are healthy. I’m in the process of collecting some tcpdumps but figured I would ask if anyone knows why this is occurring. We run haproxy in front of the rgw. For troubleshooting purposes we have a basic haproxy config to just one rgw. We have tested with multiple browsers. Ceph 17.2.5 - cluster built and managed with cephadm Ubuntu 20.04.6 LTS Log: 2024-01-03T17:40:23.916+0000 7f646033d700 1 ====== starting new request req=0x7f640ca95730 ===== 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s initializing for trans_id = tx000003ec1e4b4dc157efd-0065959c07-104216-default 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s rgw api priority: s3=8 s3website=7 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s host=a2-west.usacld.net 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s final domain/bucket subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 s->info.domain= s->info.request_uri=/n athan:tshoot/testfile.mp4 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s get_handler handler=22RGWHandler_REST_Obj_S3 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s handler=22RGWHandler_REST_Obj_S3 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s getting op 0 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s get_system_obj_state: rctx=0x7f640ca947c0 obj=default.rgw.log:script.prerequest. state=0x55fa31f882e0 s->prefetch_data=0 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s cache get: name=default.rgw.log++script.prerequest. : hit (negative entry) 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj scheduling with throttler client=2 cost=1 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj op=21RGWGetObj_ObjStore_S3 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj verifying requester 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rgw::auth::StrategyRegistry::s3_main_strategy_t: trying rgw::auth::s3::AWSAuthStrategy 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::S3AnonymousEngine 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rgw::auth::s3::S3AnonymousEngine granted access 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rgw::auth::s3::AWSAuthStrategy granted access 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj normalizing buckets and tenants 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s->object=testfile.mp4 s->bucket=nathan/tshoot 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj init permissions 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_system_obj_state: rctx=0x7f640ca94200 obj=default.rgw.meta:root:nathan/tshoot state=0x55fa31f882e0 s->prefetch _data=0 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj cache get: name=default.rgw.meta+root+nathan/tshoot : hit (requested=0x16, cached=0x17) 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_system_obj_state: s->obj_tag was set empty 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj cache get: name=default.rgw.meta+root+nathan/tshoot : hit (requested=0x11, cached=0x17) 2024-01-03T17:40:23.916+0000 7f646033d700 15 req 4522146966294331133 0.000000000s s3:get_obj decode_policy Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner> <ID>nathan$testuser</ID><DisplayName>Nathan API User</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>nathan$testuser </ID><DisplayName>Nathan API User</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy> 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWSI_User_RADOS::read_user_info(): anonymous user 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj recalculating target 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj reading permissions 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_obj_state: rctx=0x7f640ca94ad0 obj=tshoot:testfile.mp4 state=0x55fa31082de8 s->prefetch_data=1 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj WARNING: blocking librados call 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj manifest: total_size = 88811314 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_obj_state: setting s->obj_tag to _0OUNxgsSsqTAs9Z6xh1SOKpJZLV_Vos 2024-01-03T17:40:23.916+0000 7f646033d700 15 req 4522146966294331133 0.000000000s s3:get_obj decode_policy Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner> <ID>nathan$testuser</ID><DisplayName>Nathan API User</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="Group"><URI>http://acs.amazonaws.c om/groups/global/AllUsers</URI></Grantee><Permission>READ</Permission></Grant><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>nathan$testuser</ID><DisplayNa me>Nathan API User</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy> 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj init op 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_system_obj_state: rctx=0x7f640ca94488 obj=default.rgw.meta:users.uid:nathan$testuser state=0x55fa31f882e0 s->p refetch_data=0 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj cache get: name=default.rgw.meta+users.uid+nathan$testuser : hit (requested=0x16, cached=0x17) 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_system_obj_state: s->obj_tag was set empty 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr: user.rgw.idtag 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj cache get: name=default.rgw.meta+users.uid+nathan$testuser : hit (requested=0x13, cached=0x17) 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj verifying op mask 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj required_mask= 1 user.op_mask=7 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj verifying op permissions 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj -- Getting permissions begin with perm_mask=49 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj Searching permissions for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=anonymous, acct_n ame=, subuser=, perm_mask=15, is_admin=0) mask=49 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj Searching permissions for uid=anonymous 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj Permissions for user not found 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj Searching permissions for group=1 mask=49 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj Found permission: 1 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj -- Getting permissions done for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=anonymous, acct_name=, subuser=, perm_mask=15, is_admin=0), owner=nathan$testuser, perm=1 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=anonymous, acct_name=, subuser=, perm_mask =15, is_admin=0) requested perm (type)=1, policy perm=1, user_perm_mask=15, acl perm=1 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj verifying op params 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj pre-executing 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj check rate limiting 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj executing 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_obj_state: rctx=0x7f640ca94ad0 obj=tshoot:testfile.mp4 state=0x55fa31082de8 s->prefetch_data=1 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.acl 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.content_type 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.etag 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.idtag 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.manifest 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.pg_ver 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.source_zone 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.tail_tag 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.x-amz-content-sha256 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.x-amz-date 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.x-amz-meta-s3cmd-attrs 2024-01-03T17:40:23.916+0000 7f646033d700 15 req 4522146966294331133 0.000000000s Encryption mode: 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_obj_state: rctx=0x7f640ca94ad0 obj=tshoot:testfile.mp4 state=0x55fa31082de8 s->prefetch_data=1 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__multipart_testfile.mp4.2~XSiTvBBtaU4 134dQCusv3kMP7YjDBk7.1 obj-ofs=0 read_ofs=0 len=4194304 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=4194304 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): result: ofs=4194304 stripe_ofs=4194304 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134 dQCusv3kMP7YjDBk7.1_1 obj-ofs=4194304 read_ofs=0 len=4194304 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=8388608 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): result: ofs=8388608 stripe_ofs=8388608 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134 dQCusv3kMP7YjDBk7.1_2 obj-ofs=8388608 read_ofs=0 len=4194304 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=12582912 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): result: ofs=12582912 stripe_ofs=12582912 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134 dQCusv3kMP7YjDBk7.1_3 obj-ofs=12582912 read_ofs=0 len=3145728 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=16777216 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): result: ofs=15728640 stripe_ofs=15728640 part_ofs=15728640 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__multipart_testfile.mp4.2~XSiTvBBtaU4 134dQCusv3kMP7YjDBk7.2 obj-ofs=15728640 read_ofs=0 len=4194304 2024-01-03T17:40:23.924+0000 7f6442301700 20 req 4522146966294331133 0.007999974s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.924+0000 7f6442301700 20 req 4522146966294331133 0.007999974s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=19922944 part_ofs=15728640 rule->part_size=15728640 2024-01-03T17:40:23.924+0000 7f6442301700 20 req 4522146966294331133 0.007999974s s3:get_obj RGWObjManifest::operator++(): result: ofs=19922944 stripe_ofs=19922944 part_ofs=15728640 rule->part_size=15728640 2024-01-03T17:40:23.924+0000 7f6442301700 20 req 4522146966294331133 0.007999974s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134 dQCusv3kMP7YjDBk7.2_1 obj-ofs=19922944 read_ofs=0 len=4194304 2024-01-03T17:40:23.932+0000 7f643d2f7700 20 req 4522146966294331133 0.015999949s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.932+0000 7f643d2f7700 20 req 4522146966294331133 0.015999949s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=24117248 part_ofs=15728640 rule->part_size=15728640 2024-01-03T17:40:23.932+0000 7f643d2f7700 20 req 4522146966294331133 0.015999949s s3:get_obj RGWObjManifest::operator++(): result: ofs=24117248 stripe_ofs=24117248 part_ofs=15728640 rule->part_size=15728640 2024-01-03T17:40:23.932+0000 7f643d2f7700 20 req 4522146966294331133 0.015999949s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134 dQCusv3kMP7YjDBk7.2_2 obj-ofs=24117248 read_ofs=0 len=4194304 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_status: e=0, sent=17, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_content_length: e=0, sent=26, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::complete_header: e=0, sent=460, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::set_account: e=1 2024-01-03T17:40:24.068+0000 7f642a2d1700 4 write_data failed: Connection reset by peer 2024-01-03T17:40:24.068+0000 7f642a2d1700 0 req 4522146966294331133 0.151999518s s3:get_obj iterate_obj() failed with -104 2024-01-03T17:40:24.156+0000 7f642eada700 2 req 4522146966294331133 0.239999235s s3:get_obj completing 2024-01-03T17:40:24.156+0000 7f642eada700 20 req 4522146966294331133 0.239999235s get_system_obj_state: rctx=0x7f640ca947c0 obj=default.rgw.log:script.postrequest.nathan state=0x55fa32424760 s->prefetch_data =0 2024-01-03T17:40:24.156+0000 7f642eada700 10 req 4522146966294331133 0.239999235s cache get: name=default.rgw.log++script.postrequest.nathan : hit (negative entry) 2024-01-03T17:40:24.156+0000 7f642eada700 30 AccountingFilter::complete_request: e=1, sent=0, total=0 2024-01-03T17:40:24.156+0000 7f642eada700 2 req 4522146966294331133 0.239999235s s3:get_obj op status=-104 2024-01-03T17:40:24.156+0000 7f642eada700 2 req 4522146966294331133 0.239999235s s3:get_obj http status=200 2024-01-03T17:40:24.156+0000 7f642eada700 1 ====== req done req=0x7f640ca95730 op status=-104 http_status=200 latency=0.239999235s ====== 2024-01-03T17:40:24.156+0000 7f642eada700 1 beast: 0x7f640ca95730: 10.30.0.196 - anonymous [03/Jan/2024:17:40:23.916 +0000] "GET /nathan:tshoot/testfile.mp4 HTTP/1.1" 200 0 - "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" - latency=0.239999235s 2024-01-03T17:40:24.156+0000 7f642eada700 20 failed to read header: end of stream 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_ACCEPT=*/* 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_ACCEPT_ENCODING=identity;q=1, *;q=0 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_ACCEPT_LANGUAGE=en-US,en;q=0.7 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_HOST=a2-west.usacld.net 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_RANGE=bytes=0- 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_REFERER=https://a2-west.usacld.net/nathan:tshoot/testfile.mp4 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_CH_UA="Not_A Brand";v="8", "Chromium";v="120", "Brave";v="120" 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_CH_UA_MOBILE=?0 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_CH_UA_PLATFORM="macOS" 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_FETCH_DEST=video 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_FETCH_MODE=no-cors 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_FETCH_SITE=same-origin 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_GPC=1 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_VERSION=1.1 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_X_FORWARDED_PORT=443 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_X_FORWARDED_PROTO=https 2024-01-03T17:40:24.160+0000 7f6503483700 20 REMOTE_ADDR=10.30.0.196 2024-01-03T17:40:24.160+0000 7f6503483700 20 REQUEST_METHOD=GET 2024-01-03T17:40:24.160+0000 7f6503483700 20 REQUEST_URI=/nathan:tshoot/testfile.mp4 2024-01-03T17:40:24.160+0000 7f6503483700 20 SCRIPT_URI=/nathan:tshoot/testfile.mp4 2024-01-03T17:40:24.160+0000 7f6503483700 20 SERVER_PORT=8000 2024-01-03T17:40:24.160+0000 7f6503483700 1 ====== starting new request req=0x7f640ca95730 ===== Thanks, Nathan

2 1

Not all Bucket Shards being used
by Christian Kugler 01 Oct '24

01 Oct '24

Hi, I have trouble with large OMAP files in a cluster in the RGW index pool. Some background information about the cluster: There is CephFS and RBD usage on the main cluster but for this issue I think only S3 is interesting. There is one realm, one zonegroup with two zones which have a bidirectional sync set up. Since this does not allow for autoresharding we have to do it by hand in this cluster – looking forward to Reef! From the logs: cluster 2023-07-17T22:59:03.018722+0000 osd.75 (osd.75) 623978 : cluster [WRN] Large omap object found. Object: 34:bcec3016:::.dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.5:head PG: 34.680c373d (34.5) Key count: 962091 Size (bytes): 277963182 The offending bucket looks like this: # radosgw-admin bucket stats \ | jq '.[] | select(.marker =="3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9") |"\(.num_shards) \(.usage["rgw.main"].num_objects)"' -r 131 9463833 Last week the number of objects was about 12 million. Which is why I reshareded the offending bucket twice, I think. Once to 129 and the second time to 131 because I wanted some leeway (or lieway? scnr, Sage). Unfortunately, even after a week the objects were still to big (the log line above is quite recent), so I looked into it again. # rados -p raum.rgw.buckets.index ls \ |grep .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9 \ |sort -V .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.0 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.1 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.2 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.3 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.4 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.5 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.6 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.7 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.8 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.9 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.10 # rados -p raum.rgw.buckets.index ls \ |grep .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9 \ |sort -V \ |xargs -IOMAP sh -c \ 'rados -p raum.rgw.buckets.index listomapkeys OMAP | wc -l' 1013854 1011007 1012287 1011232 1013565 998262 1012777 1012713 1012230 1010690 997111 Apparently, only 11 shards are in use. This would explain why the "Key usage" (from the log line) is about ten times higher than I would expect. How can I deal with this issue? One thing I could try to fix this would be to reshard to a lower number, but I am not sure if there are any risks associated with "downsharding". After that I could reshard to something like 97. Or I could directly "downshard" to 97. Also, the second zone has a similar problem, but as the error messsage lets me know, this would be a bad idea. Will it just take more time until the sharding is transferred to the seconds zone? Best, Christian Kugler

2 5

Successfully using dm-cache
by Michael Lipp 12 Sep '24

12 Sep '24

Just in case anybody is interested: Using dm-cache works and boosts performance -- at least for my use case. The "challenge" was to get 100 (identical) Linux-VMs started on a three node hyperconverged cluster. The hardware is nothing special, each node has a Supermicro server board with a single CPU with 24 cores and 4 x 4 TB hard disks. And there's that extra 1 TB NVMe... I know that the general recommendation is to use the NVMe for WAL and metadata, but this didn't seem appropriate for my use case and I'm still not quite sure about failure scenarios with this configuration. So instead I made each drive a logical volume (managed by an OSD) and added 85 GiB NVMe to each LV as read-only cache. Each VM uses as system disk an RBD based on a snapshot from the master image. The idea was that with this configuration, all VMs should share most (actually almost all) of the data on their system disk and this data should be available from the cache. Well, it works. When booting the 100 VMs, almost all read operations are satisfied from the cache. So I get close to NVMe speed but have payed for conventional hard drives only (well, SSDs aren't that much more expensive nowadays, but the hardware is 4 years old). So, nothing sophisticated, but as I couldn't find anything about this kind of setup, it might be of interest nevertheless. - Michael

13 27

cephadm discovery service certificate absent after upgrade.
by Nicolas FOURNIL 09 Jul '24

09 Jul '24

Hello, I've just fresh upgrade from Quincy to Reef and my graphs are now blank... after investigations, it seems that discovery service is not working because of no certificate : # ceph orch sd dump cert Error EINVAL: No certificate found for service discovery Maybe an upgrade issue ? Is there a way to generate or replace the certificate properly ? Regards Nicolas F.

3 8

Fixing BlueFS spillover (pacific 16.2.14)
by Chris Dunlop 09 Jul '24

09 Jul '24

Hi, tl;dr why are my osds still spilling? I've recently upgraded to 16.2.14 from 16.2.9 and started receiving bluefs spillover warnings (due to the "fix spillover alert" per the 16.2.14 release notes). E.g. from 'ceph health detail', the warning on one of these (there are a few): osd.76 spilled over 128 KiB metadata from 'db' device (56 GiB used of 60 GiB) to slow device This is a 15T HDD with only a 60G SSD for the db so it's not surprising it spilled as it's way below the recommendation for rbd usage at db size 1-2% of the storage size. There was some spare space on the db ssd so I increased the size of the db LV up over 400G and did an bluefs-bdev-expand. However, days later, I'm still getting the spillover warning for that osd, including after running a manual compact: # ceph tell osd.76 compact See attached perf-dump-76 for the perf dump output: # cephadm enter --name 'osd.76' ceph daemon 'osd.76' perf dump" | jq -r '.bluefs' In particular, if my understanding is correct, that's telling me the db available size is 487G (i.e. the LV expand worked), of which it's using 59G, and there's 128K spilled to the slow device: "db_total_bytes": 512309059584, # 487G "db_used_bytes": 63470305280, # 59G "slow_used_bytes": 131072, # 128K A "bluefs stats" also says the db is using 128K of slow storage (although perhaps it's getting the info from the same place as the perf dump?): # ceph tell osd.76 bluefs stats 1 : device size 0x7747ffe000 : using 0xea6200000(59 GiB) 2 : device size 0xe8d7fc00000 : using 0x6554d689000(6.3 TiB) RocksDBBlueFSVolumeSelector Usage Matrix: DEV/LEV WAL DB SLOW * * REAL FILES LOG 0 B 10 MiB 0 B 0 B 0 B 8.8 MiB 1 WAL 0 B 2.5 GiB 0 B 0 B 0 B 751 MiB 8 DB 0 B 56 GiB 128 KiB 0 B 0 B 50 GiB 842 SLOW 0 B 0 B 0 B 0 B 0 B 0 B 0 TOTAL 0 B 58 GiB 128 KiB 0 B 0 B 0 B 850 MAXIMUMS: LOG 0 B 22 MiB 0 B 0 B 0 B 18 MiB WAL 0 B 3.9 GiB 0 B 0 B 0 B 1.0 GiB DB 0 B 71 GiB 282 MiB 0 B 0 B 62 GiB SLOW 0 B 0 B 0 B 0 B 0 B 0 B TOTAL 0 B 74 GiB 282 MiB 0 B 0 B 0 B >> SIZE << 0 B 453 GiB 14 TiB I had a look at the "DUMPING STATS" output in the logs bug I don't know how to interpret it. I did try calculating the total of the sizes on the "Sum" lines but that comes to 100G so I don't know what that all means. See attached log-stats-76. I also tried "ceph-kvstore-tool bluestore-kv ... stats": $ { cephadm unit --fsid $clusterid --name osd.76 stop cephadm shell --fsid $clusterid --name osd.76 -- ceph-kvstore-tool bluestore-kv /var/lib/ceph/osd/ceph-76 stats cephadm unit --fsid $clusterid --name osd.76 start } Output attached as bluestore-kv-stats-76. I can't see anything interesting in there, although again I don't really know how to interpret it. So... why is this osd db still spilling onto slow storage, and how do I fix things so it's no longer using the slow storage? And a bonus issue... on another osd that hasn't yet been resized (i.e. again with a grossly undersized 60G db on SSD with a 15T HDD) I'm also getting a spillover warning. The "bluefs stats" seems to be saying the db is NOT currently spilling (i.e. "0 B" the DB/SLOW position in the matrix), but there's "something" currently using 59G on the slow device: $ ceph tell osd.85 bluefs stats 1 : device size 0xeffffe000 : using 0x3a3900000(15 GiB) 2 : device size 0xe8d7fc00000 : using 0x7aea7434000(7.7 TiB) RocksDBBlueFSVolumeSelector Usage Matrix: DEV/LEV WAL DB SLOW * * REAL FILES LOG 0 B 10 MiB 0 B 0 B 0 B 7.4 MiB 1 WAL 0 B 564 MiB 0 B 0 B 0 B 132 MiB 2 DB 0 B 11 GiB 0 B 0 B 0 B 8.1 GiB 177 SLOW 0 B 3.0 GiB 59 GiB 0 B 0 B 56 GiB 898 TOTAL 0 B 13 GiB 59 GiB 0 B 0 B 0 B 1072 MAXIMUMS: LOG 0 B 24 MiB 0 B 0 B 0 B 20 MiB WAL 0 B 2.8 GiB 0 B 0 B 0 B 1.0 GiB DB 0 B 22 GiB 448 KiB 0 B 0 B 18 GiB SLOW 0 B 3.3 GiB 62 GiB 0 B 0 B 62 GiB TOTAL 0 B 27 GiB 62 GiB 0 B 0 B 0 B >> SIZE << 0 B 57 GiB 14 TiB Is there anywhere that describes how to interpret this output, and specifically, what stuff is going into the SLOW row? Seemingly there's 898 "files" there, but not LOG, WAL or DB files - so what are they? Cheers, Chris

4 7

Correct Migration Workflow Replicated -> Erasure Code
by Mac Wynkoop 19 Jun '24

19 Jun '24

Hi Everyone, So, I'm in the process of trying to migrate our rgw.buckets.data pool from a replicated rule pool to an erasure coded pool. I've gotten the EC pool set up, good EC profile and crush ruleset, pool created successfully, but when I go to "rados cppool xxx.rgw.buckets.data xxx.rgw.buckets.data.new", I get this error after it transfers 4GB of data: error copying object: (2) No such file or directory error copying pool xxx.rgw.buckets.data => xxx.rgw.buckets.data.new: (2) No such file or directory Is "rados cppool" still the blessed way to do the migration, or has something better/not deprecated been developed that I can use? Thanks, Mac

4 4

Re: MDS crashes to damaged metadata
by Patrick Donnelly 10 Jun '24

10 Jun '24

On Thu, Dec 15, 2022 at 9:32 AM Stolte, Felix <f.stolte(a)fz-juelich.de> wrote: > > Hi Patrick, > > we used your script to repair the damaged objects on the weekend and it went smoothly. Thanks for your support. > > We adjusted your script to scan for damaged files on a daily basis, runtime is about 6h. Until thursday last week, we had exactly the same 17 Files. On thursday at 13:05 a snapshot was created and our active mds crashed once at this time (snapshot was created): > > 2022-12-08T13:05:48.919+0100 7f440afec700 -1 /build/ceph-16.2.10/src/mds/ScatterLock.h: In function 'void ScatterLock::set_xlock_snap_sync(MDSContext*)' thread 7f440afec700 time 2022-12-08T13:05:48.921223+0100 > /build/ceph-16.2.10/src/mds/ScatterLock.h: 59: FAILED ceph_assert(state LOCK_XLOCK || state LOCK_XLOCKDONE) > > 12 Minutes lates the unlink_local error crashes appeared again. This time with a new file. During debugging we noticed a MTU mismatch between MDS (1500) and client (9000) with cephfs kernel mount. The client is also creating the snapshots via mkdir in the .snap directory. > > We disabled snapshot creation for now, but really need this feature. I uploaded the mds logs of the first crash along with the information above to https://tracker.ceph.com/issues/38452 > > I would greatly appreciate it, if you could answer me the following question: > > Is the Bug related to our MTU Mismatch? We fixed the MTU Issue going back to 1500 on all nodes in the ceph public network on the weekend also. I doubt it. > If you need a debug level 20 log of the ScatterLock for further analysis, i could schedule snapshots at the end of our workdays and increase the debug level 5 Minutes arround snap shot creation. This would be very helpful! -- Patrick Donnelly, Ph.D. He / Him / His Principal Software Engineer Red Hat, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

2 4

Problems adding a new host via orchestration.
by Gary Molenkamp 22 May '24

22 May '24

Happy Friday all. I was hoping someone could point me in the right direction or clarify any limitations that could be impacting an issue I am having. I'm struggling to add a new set of hosts to my ceph cluster using cephadm and orchestration. When trying to add a host: "ceph orch host add <hostname> 172.31.102.41 --labels _admin" returns: "Error EINVAL: Can't communicate with remote host `172.31.102.41`, possibly because python3 is not installed there: [Errno 12] Cannot allocate memory" I've verified that the ceph ssh key works to the remote host, host's name matches that returned from `hostname`, python3 is installed, and "/usr/sbin/cephadm prepare-host" on the new hosts returns "host is ok". In addition, the cluster ssh key works between hosts and the existing hosts are able to ssh in using the ceph key. The existing ceph cluster is Pacific release using docker based containerization on RockyLinux8 base OS. The new hosts are RockyLinux9 based, with the cephadm being installed from Quincy release: ./cephadm add-repo --release quincy ./cephadm install I did try installing cephadm from the Pacific release by changing the repo to el8, but that did not work either. Is there a limitation is mixing RL8 and RL9 container hosts under Pacific? Does this same limitation exist under Quincy? Is there a python version dependency? The reason for RL9 on the new hosts is to stage upgrading the OS's for the cluster. I did this under Octopus for moving from Centos7 to RL8. Thanks and I appreciate any feedback/pointers. Gary I've added the log trace here in case that helps (from `ceph log last cephadm`) 2024-02-02T14:22:32.610048+0000 mgr.storage01.oonvfl (mgr.441023307) 4957871 : cephadm [ERR] Can't communicate with remote host `172.31.102.41`, possibly because python3 is not installed there: [Errno 12] Cannot allocate memory Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/serve.py", line 1524, in _remote_connection conn, connr = self.mgr._get_connection(addr) File "/usr/share/ceph/mgr/cephadm/module.py", line 1370, in _get_connection sudo=True if self.ssh_user != 'root' else False) File "/lib/python3.6/site-packages/remoto/backends/__init__.py", line 35, in __init__ self.gateway = self._make_gateway(hostname) File "/lib/python3.6/site-packages/remoto/backends/__init__.py", line 46, in _make_gateway self._make_connection_string(hostname) File "/lib/python3.6/site-packages/execnet/multi.py", line 133, in makegateway io = gateway_io.create_io(spec, execmodel=self.execmodel) File "/lib/python3.6/site-packages/execnet/gateway_io.py", line 121, in create_io io = Popen2IOMaster(args, execmodel) File "/lib/python3.6/site-packages/execnet/gateway_io.py", line 21, in __init__ self.popen = p = execmodel.PopenPiped(args) File "/lib/python3.6/site-packages/execnet/gateway_base.py", line 184, in PopenPiped return self.subprocess.Popen(args, stdout=PIPE, stdin=PIPE) File "/lib64/python3.6/subprocess.py", line 729, in __init__ restore_signals, start_new_session) File "/lib64/python3.6/subprocess.py", line 1295, in _execute_child restore_signals, start_new_session, preexec_fn) OSError: [Errno 12] Cannot allocate memory During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/serve.py", line 1528, in _remote_connection raise execnet.gateway_bootstrap.HostNotFound(msg) execnet.gateway_bootstrap.HostNotFound: Can't communicate with remote host `172.31.102.41`, possibly because python3 is not installed there: [Errno 12] Cannot allocate memory The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 125, in wrapper return OrchResult(f(*args, **kwargs)) File "/usr/share/ceph/mgr/cephadm/module.py", line 2709, in apply results.append(self._apply(spec)) File "/usr/share/ceph/mgr/cephadm/module.py", line 2574, in _apply return self._add_host(cast(HostSpec, spec)) File "/usr/share/ceph/mgr/cephadm/module.py", line 1517, in _add_host ip_addr = self._check_valid_addr(spec.hostname, spec.addr) File "/usr/share/ceph/mgr/cephadm/module.py", line 1498, in _check_valid_addr error_ok=True, no_fsid=True) File "/usr/share/ceph/mgr/cephadm/serve.py", line 1326, in _run_cephadm with self._remote_connection(host, addr) as tpl: File "/lib64/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/usr/share/ceph/mgr/cephadm/serve.py", line 1558, in _remote_connection raise OrchestratorError(msg) from e orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.31.102.41`, possibly because python3 is not installed there: [Errno 12] Cannot allocate memory -- Gary Molenkamp Science Technology Services Systems Engineer University of Western Ontario molenkam(a)uwo.ca http://sts.sci.uwo.ca (519) 661-2111 x86882 (519) 661-3566

5 9

16.2.14: [progress WARNING root] complete: ev {UUID} does not exist
by Zakhar Kirpichenko 03 May '24

03 May '24

Hi, Mgr of my cluster logs this every few seconds: [progress WARNING root] complete: ev 7de5bb74-790b-4fda-8838-e4af4af18c62 does not exist [progress WARNING root] complete: ev fff93fce-b630-4141-81ee-19e7a3e61483 does not exist [progress WARNING root] complete: ev a02f6966-5b9f-49e8-89c4-b4fb8e6f4423 does not exist [progress WARNING root] complete: ev 8d318560-ff1a-477f-9386-43f6b51080bf does not exist [progress WARNING root] complete: ev ff3740a9-6434-470a-808f-a2762fb542a0 does not exist [progress WARNING root] complete: ev 7d0589f1-545e-4970-867b-8482ce48d7f0 does not exist [progress WARNING root] complete: ev 78d57e43-5be5-43f0-8b1a-cdc60e410892 does not exist I would appreciate an advice on what these warnings mean and how they can be resolved. Best regards, Zakhar

4 7

Re: Recoveries without any misplaced objects?
by Hector Martin 24 Apr '24

24 Apr '24

On 29/05/2023 20.55, Anthony D'Atri wrote: > Check the uptime for the OSDs in question I restarted all my OSDs within the past 10 days or so. Maybe OSD restarts are somehow breaking these stats? > >> On May 29, 2023, at 6:44 AM, Hector Martin <marcan(a)marcan.st> wrote: >> >> Hi, >> >> I'm watching a cluster finish a bunch of backfilling, and I noticed that >> quite often PGs end up with zero misplaced objects, even though they are >> still backfilling. >> >> Right now the cluster is down to 6 backfilling PGs: >> >> data: >> volumes: 1/1 healthy >> pools: 6 pools, 268 pgs >> objects: 18.79M objects, 29 TiB >> usage: 49 TiB used, 25 TiB / 75 TiB avail >> pgs: 262 active+clean >> 6 active+remapped+backfilling >> >> But there are no misplaced objects, and the misplaced column in `ceph pg >> dump` is zero for all PGs. >> >> If I do a `ceph pg dump_json`, I can see `num_objects_recovered` >> increasing for these PGs... but the misplaced count is still 0. >> >> Is there something else that would cause recoveries/backfills other than >> misplaced objects? Or perhaps there is a bug somewhere causing the >> misplaced object count to be misreported as 0 sometimes? >> >> # ceph -v >> ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy >> (stable) >> >> - Hector >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > - Hector

3 3

Status of IPv4 / IPv6 dual stack?
by Robert Sander 23 Apr '24

23 Apr '24

Hi, as the documentation sends mixed signals in https://docs.ceph.com/en/latest/rados/configuration/network-config-ref/#ipv… "Note Binding to IPv4 is enabled by default, so if you just add the option to bind to IPv6 you’ll actually put yourself into dual stack mode." and https://docs.ceph.com/en/latest/rados/configuration/msgr2/#address-formats "Note The ability to bind to multiple ports has paved the way for dual-stack IPv4 and IPv6 support. That said, dual-stack operation is not yet supported as of Quincy v17.2.0." just the quick questions: Is a dual stacked networking with IPv4 and IPv6 now supported or not? From which version on is it considered stable? Are OSDs now able to register themselves with two IP addresses in the cluster map? MONs too? Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

6 6

Ceph 16.2.x mon compactions, disk writes
by Zakhar Kirpichenko 17 Apr '24

17 Apr '24

Hi, Monitors in our 16.2.14 cluster appear to quite often run "manual compaction" tasks: debug 2023-10-09T09:30:53.888+0000 7f48a329a700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1696843853892760, "job": 64225, "event": "flush_started", "num_memtables": 1, "num_entries": 715, "num_deletes": 251, "total_data_size": 3870352, "memory_usage": 3886744, "flush_reason": "Manual Compaction"} debug 2023-10-09T09:30:53.904+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:30:53.908+0000 7f48a3a9b700 4 rocksdb: (Original Log Time 2023/10/09-09:30:53.910204) [db_impl/db_impl_compaction_flush.cc:2516] [default] Manual compaction from level-0 to level-5 from 'paxos .. 'paxos; will stop at (end) debug 2023-10-09T09:30:53.908+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:30:53.908+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:30:53.908+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:30:53.908+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:30:53.908+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:30:53.908+0000 7f48a3a9b700 4 rocksdb: (Original Log Time 2023/10/09-09:30:53.911004) [db_impl/db_impl_compaction_flush.cc:2516] [default] Manual compaction from level-5 to level-6 from 'paxos .. 'paxos; will stop at (end) debug 2023-10-09T09:32:08.956+0000 7f48a329a700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1696843928961390, "job": 64228, "event": "flush_started", "num_memtables": 1, "num_entries": 1580, "num_deletes": 502, "total_data_size": 8404605, "memory_usage": 8465840, "flush_reason": "Manual Compaction"} debug 2023-10-09T09:32:08.972+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:08.976+0000 7f48a3a9b700 4 rocksdb: (Original Log Time 2023/10/09-09:32:08.977739) [db_impl/db_impl_compaction_flush.cc:2516] [default] Manual compaction from level-0 to level-5 from 'logm .. 'logm; will stop at (end) debug 2023-10-09T09:32:08.976+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:08.976+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:08.976+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:08.976+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:08.976+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:08.976+0000 7f48a3a9b700 4 rocksdb: (Original Log Time 2023/10/09-09:32:08.978512) [db_impl/db_impl_compaction_flush.cc:2516] [default] Manual compaction from level-5 to level-6 from 'logm .. 'logm; will stop at (end) debug 2023-10-09T09:32:12.764+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:12.764+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:12.764+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:12.764+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:12.764+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:12.764+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:33:29.028+0000 7f48a329a700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1696844009033151, "job": 64231, "event": "flush_started", "num_memtables": 1, "num_entries": 1430, "num_deletes": 251, "total_data_size": 8975535, "memory_usage": 9035920, "flush_reason": "Manual Compaction"} debug 2023-10-09T09:33:29.044+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:33:29.048+0000 7f48a3a9b700 4 rocksdb: (Original Log Time 2023/10/09-09:33:29.049585) [db_impl/db_impl_compaction_flush.cc:2516] [default] Manual compaction from level-0 to level-5 from 'paxos .. 'paxos; will stop at (end) debug 2023-10-09T09:33:29.048+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:33:29.048+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:33:29.048+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:33:29.048+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:33:29.048+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:33:29.048+0000 7f48a3a9b700 4 rocksdb: (Original Log Time 2023/10/09-09:33:29.050355) [db_impl/db_impl_compaction_flush.cc:2516] [default] Manual compaction from level-5 to level-6 from 'paxos .. 'paxos; will stop at (end) I have removed a lot of interim log messages to save space. During each compaction the monitor process writes approximately 500-600 MB of data to disk over a short period of time. These writes add up to tens of gigabytes per hour and hundreds of gigabytes per day. Monitor rocksdb and compaction options are default: "mon_compact_on_bootstrap": "false", "mon_compact_on_start": "false", "mon_compact_on_trim": "true", "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", Is this expected behavior? Is this something I can adjust in order to extend the system storage life? Best regards, Zakhar

5 32

Setting S3 bucket policies with multi-tenants
by Thomas Bennett 15 Apr '24

15 Apr '24

Hi, I'm running Ceph Quincy (17.2.6) with a rados-gateway. I have muti tenants, for example: - Tenant1$manager - Tenant1$readwrite I would like to set a policy on a bucket (backups for example) owned by *Tenant1$manager* to allow *Tenant1$readwrite* access to that bucket. I can't find any documentation that discusses this scenario. Does anyone know how to specify the Principle and Resource section of a policy.json file? Or any other configuration that I might be missing? I've tried some variations on Principal and Resource including and excluding tenant information, but not no luck yet. For example: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": {"AWS": ["arn:aws:iam:::user/*Tenant1$readwrite*"]}, "Action": ["s3:ListBucket","s3:GetObject", ,"s3:PutObject"], "Resource": [ "arn:aws:s3:::*Tenant1/backups*" ] }] } I'm using s3cmd for testing, so: s3cmd --config s3cfg.manager setpolicy policy.json s3://backups/ Returns: s3://backups/: Policy updated And then testing: s3cmd --config s3cfg.readwrite ls s3://backups/ ERROR: Access to bucket 'backups' was denied ERROR: S3 error: 403 (AccessDenied) Thanks, Tom

4 4

A couple OSDs not starting after host reboot
by Alison Peisker 11 Apr '24

11 Apr '24

Hi all, We rebooted all the nodes in our 17.2.5 cluster after performing kernel updates, but 2 of the OSDs on different nodes are not coming back up. This is a production cluster using cephadm. The error message from the OSD log is ceph-osd[87340]: ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-665: (2) No such file or directory The error message from ceph-volume is 2023-08-23T16:12:43.452-0500 7f0cad968600 2 bluestore(/dev/mapper/ceph--febad5a5--ba44--41aa--a39e--b9897f757752-osd--block--87e548f4--b9b5--4ed8--aca8--de703a341a50) _read_bdev_label unable to decode label at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input We tried restarting the daemons and rebooting the node again, but still see the same error. Has anyone experienced this issue before? How do we fix this? Thanks, Alison

9 13

cephfs creation error
by Ramanathan S 03 Apr '24

03 Apr '24

Hi all, I just had created a ceph cluster to use cephfs. When i create the a ceph fs pool i get the filesystem below error. # ceph osd pool create cephfs_data 128 pool 'cephfs_data' created # ceph osd pool create cephfs_metadata 128 pool 'cephfs_metadata' created # ceph fs new cephfs cephfs_metadata cephfs_data new fs with metadata pool 6 and data pool 5 # ceph -s cluster: id: 1c27def45-f0f9-494d-sfke-eb4323432fd health: HEALTH_ERR 1 filesystem is offline 1 filesystem is online with fewer MDS than max_mds services: mon: 2 daemons, quorum ceph-mon01,ceph-mon02 mgr: ceph-adm01(active) mds: cephfs-0/0/1 up osd: 12 osds: 12 up, 12 in data: pools: 2 pools, 256 pgs objects: 0 objects, 0 B usage: 12 GiB used, 588 GiB / 600 GiB avail pgs: 256 active+clean but when i check the max_mds for the ceph fs it says 1 # ceph fs get cephfs | grep max_mds max_mds 1 Let anyone know what am i missing here? Any inputs is much appreciated. Regards, Ram Ceph-explorer..

3 3

Re: NoSuchKey on key that is visible in s3 list/radosgw bk
by Eric Ivancich 03 Apr '24

03 Apr '24

I have some questions for those who’ve experienced this issue. 1. It seems like those reporting this issue are seeing it strictly after upgrading to Octopus. From what version did each of these sites upgrade to Octopus? From Nautilus? Mimic? Luminous? 2. Does anyone have any lifecycle rules on a bucket experiencing this issue? If so, please describe. 3. Is anyone making copies of the affected objects (to same or to a different bucket) prior to seeing the issue? And if they are making copies, does the destination bucket have lifecycle rules? And if they are making copies, are those copies ever being removed? 4. Is anyone experiencing this issue willing to run their RGWs with 'debug_ms=1'? That would allow us to see a request from an RGW to either remove a tail object or decrement its reference counter (and when its counter reaches 0 it will be deleted). Thanks, Eric > On Nov 12, 2020, at 4:54 PM, huxiaoyu(a)horebdata.cn wrote: > > Looks like this is a very dangerous bug for data safety. Hope the bug would be quickly identified and fixed. > > best regards, > > Samuel > > > > huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn> > > From: Janek Bevendorff > Date: 2020-11-12 18:17 > To: huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn>; EDH - Manuel Rios; Rafael Lopez > CC: Robin H. Johnson; ceph-users > Subject: Re: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > I have never seen this on Luminous. I recently upgraded to Octopus and the issue started occurring only few weeks later. > > On 12/11/2020 16:37, huxiaoyu(a)horebdata.cn wrote: > which Ceph versions are affected by this RGW bug/issues? Luminous, Mimic, Octupos, or the latest? > > any idea? > > samuel > > > > huxiaoyu(a)horebdata.cn > > From: EDH - Manuel Rios > Date: 2020-11-12 14:27 > To: Janek Bevendorff; Rafael Lopez > CC: Robin H. Johnson; ceph-users > Subject: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > This same error caused us to wipe a full cluster of 300TB... will be related to some rados index/database bug not to s3. > > As Janek exposed is a mayor issue, because the error silent happend and you can only detect it with S3, when you're going to delete/purge a S3 bucket. Dropping NoSuchKey. Error is not related to S3 logic .. > > Hope this time dev's can take enought time to find and resolve the issue. Error happens with low ec profiles, even with replica x3 in some cases. > > Regards > > > > -----Mensaje original----- > De: Janek Bevendorff <janek.bevendorff(a)uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de>> > Enviado el: jueves, 12 de noviembre de 2020 14:06 > Para: Rafael Lopez <rafael.lopez(a)monash.edu <mailto:rafael.lopez@monash.edu>> > CC: Robin H. Johnson <robbat2(a)gentoo.org <mailto:robbat2@gentoo.org>>; ceph-users <ceph-users(a)ceph.io <mailto:ceph-users@ceph.io>> > Asunto: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > > Here is a bug report concerning (probably) this exact issue: > https://tracker.ceph.com/issues/47866 <https://tracker.ceph.com/issues/47866> > > I left a comment describing the situation and my (limited) experiences with it. > > > On 11/11/2020 10:04, Janek Bevendorff wrote: >> >> Yeah, that seems to be it. There are 239 objects prefixed >> .8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh in my dump. However, there are none >> of the multiparts from the other file to be found and the head object >> is 0 bytes. >> >> I checked another multipart object with an end pointer of 11. >> Surprisingly, it had way more than 11 parts (39 to be precise) named >> .1, .1_1 .1_2, .1_3, etc. Not sure how Ceph identifies those, but I >> could find them in the dump at least. >> >> I have no idea why the objects disappeared. I ran a Spark job over all >> buckets, read 1 byte of every object and recorded errors. Of the 78 >> buckets, two are missing objects. One bucket is missing one object, >> the other 15. So, luckily, the incidence is still quite low, but the >> problem seems to be expanding slowly. >> >> >> On 10/11/2020 23:46, Rafael Lopez wrote: >>> Hi Janek, >>> >>> What you said sounds right - an S3 single part obj won't have an S3 >>> multipart string as part of the prefix. S3 multipart string looks >>> like "2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme". >>> >>> From memory, single part S3 objects that don't fit in a single rados >>> object are assigned a random prefix that has nothing to do with >>> the object name, and the rados tail/data objects (not the head >>> object) have that prefix. >>> As per your working example, the prefix for that would be >>> '.8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh'. So there would be (239) "shadow" >>> objects with names containing that prefix, and if you add up the >>> sizes it should be the size of your S3 object. >>> >>> You should look at working and non working examples of both single >>> and multipart S3 objects, as they are probably all a bit different >>> when you look in rados. >>> >>> I agree it is a serious issue, because once objects are no longer in >>> rados, they cannot be recovered. If it was a case that there was a >>> link broken or rados objects renamed, then we could work to >>> recover...but as far as I can tell, it looks like stuff is just >>> vanishing from rados. The only explanation I can think of is some >>> (rgw or rados) background process is incorrectly doing something with >>> these objects (eg. renaming/deleting). I had thought perhaps it was a >>> bug with the rgw garbage collector..but that is pure speculation. >>> >>> Once you can articulate the problem, I'd recommend logging a bug >>> tracker upstream. >>> >>> >>> On Wed, 11 Nov 2020 at 06:33, Janek Bevendorff >>> <janek.bevendorff(a)uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de> >>> <mailto:janek.bevendorff@uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de>>> wrote: >>> >>> Here's something else I noticed: when I stat objects that work >>> via radosgw-admin, the stat info contains a "begin_iter" JSON >>> object with RADOS key info like this >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110924034843-crawl420/WIDE-20110924065228-02544.warc.gz", >>> "instance": "", >>> "ns": "" >>> } >>> >>> >>> and then "end_iter" with key info like this: >>> >>> >>> "key": { >>> "name": >>> ".8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh_239", >>> "instance": "", >>> "ns": "shadow" >>> } >>> >>> However, when I check the broken 0-byte object, the "begin_iter" >>> and "end_iter" keys look like this: >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.1", >>> "instance": "", >>> "ns": "multipart" >>> } >>> >>> [...] >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.19", >>> "instance": "", >>> "ns": "multipart" >>> } >>> >>> So, it's the full name plus a suffix and the namespace is >>> multipart, not shadow (or empty). This in itself may just be an >>> artefact of whether the object was uploaded in one go or as a >>> multipart object, but the second difference is that I cannot find >>> any of the multipart objects in my pool's object name dump. I >>> can, however, find the shadow RADOS object of the intact S3 object. >>> >>> >>> >>> >>> -- >>> *Rafael Lopez* >>> Devops Systems Engineer >>> Monash University eResearch Centre >>> >>> T: +61 3 9905 9118 <tel:%2B61%203%209905%209118> >>> E: rafael.lopez(a)monash.edu <mailto:rafael.lopez@monash.edu> >>> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

6 22

Pacific Bug?
by Alex 02 Apr '24

02 Apr '24

Hello Ceph Gurus! I'm running Ceph Pacific version. if I run ceph orch host ls --label osds shows all hosts label osds or ceph orch host ls --host-pattern host1 shows just host1 it works as expected But combining the two the label tag seems to "take over" ceph orch host ls --label osds --host-pattern host1 6 hosts in cluster who had label osds whose hostname matched host1 shows all host with the label osds instead of only host1. So at first the flags seem to act like an OR instead of an AND. ceph orch host ls --label osds --host-pattern foo 6 hosts in cluster who had label osds whose hostname matched foo even though "foo" doesn't even exist ceph orch host ls --label bar --host-pattern host1 0 hosts in cluster who had label bar whose hostname matched host1 if the label and host combo was an OR this should have worked there is no label bar but host1 exists so it just disregards the host-pattern. This started because the osd deployment task had both label and host_pattern. The cluster was attempting to deploy OSDS on all the servers with the given tag instead of the one host we needed, which caused it to go into warning state. If I ran ceph orch ls --export --service_name host1 it also showed both tags and host_pattern. unmanaged: false placement: host_pattern: label: The issue persisted until I removed the label tag. Thanks.

3 5

cephfs inode backtrace information
by Dietmar Rieder 02 Apr '24

02 Apr '24

Hello, I have a question regarding the default pool of a cephfs. According to the docs it is recommended to use a fast ssd replicated pool as default pool for cephfs. I'm asking what are the space requirements for storing the inode backtrace information? Let's say I have a 85 TiB replicated ssd pool (hot data) and as 3 PiB EC data pool (cold data). Does it make sense to create a third pool as default pool which only holds the inode backtrace information (what would be a good size), or is it OK to use the ssd pool as default pool? Thanks Dietmar

6 10

Improving CephFS performance by always putting "default" data pool on SSDs?
by Niklas Hambüchen 29 Mar '24

29 Mar '24

https://docs.ceph.com/en/reef/cephfs/createfs/ says: > The data pool used to create the file system is the “default” data pool and the location for storing all inode backtrace information, which is used for hard link management and disaster recovery. > For this reason, all CephFS inodes have at least one object in the default data pool. If erasure-coded pools are planned for file system data, it is best to configure the default as a replicated pool to improve small-object write and read performance when updating backtraces. This poses the question: Are normal replicated CephFS installations (metadata on SSDs, data on HDDs) set up with suboptimal performance because they don't do this? If having inodes/backtraces on replicated instead of EC improves performance, shouldn't one expect that putting inodes/backtraces on SSD would improve it even more? From the docs I also cannot really conclude when inotes/backtraces become important. Is that all the time, or only sometimes? Thanks!

2 3

Why you might want packages not containers for Ceph deployments
by Matthew Vernon 25 Mar '24

25 Mar '24

Hi, In the discussion after the Ceph Month talks yesterday, there was a bit of chat about cephadm / containers / packages. IIRC, Sage observed that a common reason in the recent user survey for not using cephadm was that it only worked on containerised deployments. I think he then went on to say that he hadn't heard any compelling reasons why not to use containers, and suggested that resistance was essentially a user education question[0]. I'd like to suggest, briefly, that: * containerised deployments are more complex to manage, and this is not simply a matter of familiarity * reducing the complexity of systems makes admins' lives easier * the trade-off of the pros and cons of containers vs packages is not obvious, and will depend on deployment needs * Ceph users will benefit from both approaches being supported into the future We make extensive use of containers at Sanger, particularly for scientific workflows, and also for bundling some web apps (e.g. Grafana). We've also looked at a number of container runtimes (Docker, singularity, charliecloud). They do have advantages - it's easy to distribute a complex userland in a way that will run on (almost) any target distribution; rapid "cloud" deployment; some separation (via namespaces) of network/users/processes. For what I think of as a 'boring' Ceph deploy (i.e. install on a set of dedicated hardware and then run for a long time), I'm not sure any of these benefits are particularly relevant and/or compelling - Ceph upstream produce Ubuntu .debs and Canonical (via their Ubuntu Cloud Archive) provide .debs of a couple of different Ceph releases per Ubuntu LTS - meaning we can easily separate out OS upgrade from Ceph upgrade. And upgrading the Ceph packages _doesn't_ restart the daemons[1], meaning that we maintain control over restart order during an upgrade. And while we might briefly install packages from a PPA or similar to test a bugfix, we roll those (test-)cluster-wide, rather than trying to run a mixed set of versions on a single cluster - and I understand this single-version approach is best practice. Deployment via containers does bring complexity; some examples we've found at Sanger (not all Ceph-related, which we run from packages): * you now have 2 process supervision points - dockerd and systemd * docker updates (via distribution unattended-upgrades) have an unfortunate habit of rudely restarting everything * docker squats on a chunk of RFC 1918 space (and telling it not to can be a bore), which coincides with our internal network... * there is more friction if you need to look inside containers (particularly if you have a lot running on a host and are trying to find out what's going on) * you typically need to be root to build docker containers (unlike packages) * we already have package deployment infrastructure (which we'll need regardless of deployment choice) We also currently use systemd overrides to tweak some of the Ceph units (e.g. to do some network sanity checks before bringing up an OSD), and have some tools to pair OSD / journal / LVM / disk device up; I think these would be more fiddly in a containerised deployment. I'd accept that fixing these might just be a SMOP[2] on our part. Now none of this is show-stopping, and I am most definitely not saying "don't ship containers". But I think there is added complexity to your deployment from going the containers route, and that is not simply a "learn how to use containers" learning curve. I do think it is reasonable for an admin to want to reduce the complexity of what they're dealing with - after all, much of my job is trying to automate or simplify the management of complex systems! I can see from a software maintainer's point of view that just building one container and shipping it everywhere is easier than building packages for a number of different distributions (one of my other hats is a Debian developer, and I have a bunch of machinery for doing this sort of thing). But it would be a bit unfortunate if the general thrust of "let's make Ceph easier to set up and manage" was somewhat derailed with "you must use containers, even if they make your life harder". I'm not going to criticise anyone who decides to use a container-based deployment (and I'm sure there are plenty of setups where it's an obvious win), but if I were advising someone who wanted to set up and use a 'boring' Ceph cluster for the medium term, I'd still advise on using packages. I don't think this makes me a luddite :) Regards, and apologies for the wall of text, Matthew [0] I think that's a fair summary! [1] This hasn't always been true... [2] Simple (sic.) Matter of Programming -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.

26 65

[Urgent] Ceph system Down, Ceph FS volume in recovering
by nguyenvandiep＠baoviet.com.vn 16 Mar '24

16 Mar '24

We have 6 node ( 3 OSD-node and 3 service node), t2/3 OSD nodes was powered off and we got big problem pls check ceph-s result below now we cannot start mds service, ( we tried to start but it stopped after 2 minute) Now my application cannot access to NFS exported Folder What should we do [root@cephgw01 /]# ceph -s cluster: id: 258af72a-cff3-11eb-a261-d4f5ef25154c health: HEALTH_WARN 3 failed cephadm daemon(s) 1 filesystem is degraded insufficient standby MDS daemons available 1 nearfull osd(s) Low space hindering backfill (add storage if this doesn't resolve itself): 21 pgs backfill_toofull 15 pool(s) nearfull 11 daemons have recently crashed services: mon: 6 daemons, quorum cephgw03,cephosd01,cephgw01,cephosd03,cephgw02,cephosd02 (age 30h) mgr: cephgw01.vwoffq(active, since 17h), standbys: cephgw02.nauphz, cephgw03.aipvii mds: 1/1 daemons up osd: 29 osds: 29 up (since 40h), 29 in (since 29h); 402 remapped pgs rgw: 2 daemons active (2 hosts, 1 zones) tcmu-runner: 18 daemons active (2 hosts) data: volumes: 0/1 healthy, 1 recovering pools: 15 pools, 1457 pgs objects: 36.87M objects, 25 TiB usage: 75 TiB used, 41 TiB / 116 TiB avail pgs: 17759672/110607480 objects misplaced (16.056%) 1055 active+clean 363 active+remapped+backfill_wait 18 active+remapped+backfilling 14 active+remapped+backfill_toofull 7 active+remapped+backfill_wait+backfill_toofull io: client: 2.0 MiB/s rd, 395 KiB/s wr, 73 op/s rd, 19 op/s wr recovery: 32 MiB/s, 45 objects/s

6 49

Cephfs error state with one bad file
by Sake Ceph 15 Mar '24

15 Mar '24

Hi again, hopefully for the last time with problems. We had a MDS crash earlier with the MDS staying in failed state and used a command to reset the filesystem (this was wrong, I know now, thanks Patrick Donnelly for pointing this out). I did a full scrub on the filesystem and two files were damaged. One of those got repaired, but the following file keeps giving errors and can't be removed. What can I do now? Below some information. # ceph tell mds.atlassian-prod:0 damage ls [ { "damage_type": "backtrace", "id": 2244444901, "ino": 1099534008829, "path": "/app1/shared/data/repositories/11271/objects/41/8f82507a0737c611720ed224bcc8b7a24fda01" } ] Trying to repair the error (online research shows this should work for a backtrace damage type) ---------- # ceph tell mds.atlassian-prod:0 scrub start /app1/shared/data/repositories/11271 recursive,repair,force { "return_code": 0, "scrub_tag": "d10ead42-5280-4224-971e-4f3022e79278", "mode": "asynchronous" } Cluster logs after this ---------- 1/2/24 9:37:05 AM [INF] scrub summary: idle 1/2/24 9:37:02 AM [INF] scrub summary: idle+waiting paths [/app1/shared/data/repositories/11271] 1/2/24 9:37:01 AM [INF] scrub summary: active paths [/app1/shared/data/repositories/11271] 1/2/24 9:37:01 AM [INF] scrub summary: idle+waiting paths [/app1/shared/data/repositories/11271] 1/2/24 9:37:01 AM [INF] scrub queued for path: /app1/shared/data/repositories/11271 But the error doesn't disappear and still can't remove the file. On the client trying to remove the file (we got a backup) ---------- $ rm -f /mnt/shared_disk-app1/shared/data/repositories/11271/objects/41/8f82507a0737c611720ed224bcc8b7a24fda01 rm: cannot remove '/mnt/shared_disk-app1/shared/data/repositories/11271/objects/41/8f82507a0737c611720ed224bcc8b7a24fda01': Input/output error Best regards, Sake

2 1

MDS subtree pinning
by Sake Ceph 15 Mar '24

15 Mar '24

Hi! As I'm reading through the documentation about subtree pinning, I was wondering if the following is possible. We've got the following directory structure. / /app1 /app2 /app3 /app4 Can I pin /app1 to MDS rank 0 and 1, the directory /app2 to rank 2 and finally /app3 and /app4 to rank 3? I would like to load balance the subfolders of /app1 to 2 (or 3) MDS servers. Best regards, Sake

2 2

Upgrading from Pacific to Quincy fails with "Unexpected error"
by Reza Bakhshayeshi 13 Mar '24

13 Mar '24

Hi all, I have a problem regarding upgrading Ceph cluster from Pacific to Quincy version with cephadm. I have successfully upgraded the cluster to the latest Pacific (16.2.11). But when I run the following command to upgrade the cluster to 17.2.5, After upgrading 3/4 mgrs, the upgrade process stops with "Unexpected error". (everything is on a private network) ceph orch upgrade start my-private-repo/quay-io/ceph/ceph:v17.2.5 I also tried the 17.2.4 version. cephadm fails to check the hosts' status and marks them as offline: cephadm 2023-04-06T10:19:59.998510+0000 mgr.host9.arhpnd (mgr.4516356) 5782 : cephadm [DBG] host host4 (x.x.x.x) failed check cephadm 2023-04-06T10:19:59.998553+0000 mgr.host9.arhpnd (mgr.4516356) 5783 : cephadm [DBG] Host "host4" marked as offline. Skipping daemon refresh cephadm 2023-04-06T10:19:59.998581+0000 mgr.host9.arhpnd (mgr.4516356) 5784 : cephadm [DBG] Host "host4" marked as offline. Skipping gather facts refresh cephadm 2023-04-06T10:19:59.998609+0000 mgr.host9.arhpnd (mgr.4516356) 5785 : cephadm [DBG] Host "host4" marked as offline. Skipping network refresh cephadm 2023-04-06T10:19:59.998633+0000 mgr.host9.arhpnd (mgr.4516356) 5786 : cephadm [DBG] Host "host4" marked as offline. Skipping device refresh cephadm 2023-04-06T10:19:59.998659+0000 mgr.host9.arhpnd (mgr.4516356) 5787 : cephadm [DBG] Host "host4" marked as offline. Skipping osdspec preview refresh cephadm 2023-04-06T10:19:59.998682+0000 mgr.host9.arhpnd (mgr.4516356) 5788 : cephadm [DBG] Host "host4" marked as offline. Skipping autotune cluster 2023-04-06T10:20:00.000151+0000 mon.host8 (mon.0) 158587 : cluster [ERR] Health detail: HEALTH_ERR 9 hosts fail cephadm check; Upgrade: failed due to an unexpected exception cluster 2023-04-06T10:20:00.000191+0000 mon.host8 (mon.0) 158588 : cluster [ERR] [WRN] CEPHADM_HOST_CHECK_FAILED: 9 hosts fail cephadm check cluster 2023-04-06T10:20:00.000202+0000 mon.host8 (mon.0) 158589 : cluster [ERR] host host7 (x.x.x.x) failed check: Unable to reach remote host host7. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000213+0000 mon.host8 (mon.0) 158590 : cluster [ERR] host host2 (x.x.x.x) failed check: Unable to reach remote host host2. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000220+0000 mon.host8 (mon.0) 158591 : cluster [ERR] host host8 (x.x.x.x) failed check: Unable to reach remote host host8. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000228+0000 mon.host8 (mon.0) 158592 : cluster [ERR] host host4 (x.x.x.x) failed check: Unable to reach remote host host4. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000240+0000 mon.host8 (mon.0) 158593 : cluster [ERR] host host3 (x.x.x.x) failed check: Unable to reach remote host host3. Process exited with non-zero exit status 3 and here are some outputs of the commands: [root@host8 ~]# ceph -s cluster: id: xxx health: HEALTH_ERR 9 hosts fail cephadm check Upgrade: failed due to an unexpected exception services: mon: 5 daemons, quorum host8,host1,host7,host2,host9 (age 2w) mgr: host9.arhpnd(active, since 105m), standbys: host8.jowfih, host1.warjsr, host2.qyavjj mds: 1/1 daemons up, 3 standby osd: 37 osds: 37 up (since 8h), 37 in (since 3w) data: io: client: progress: Upgrade to 17.2.5 (0s) [............................] [root@host8 ~]# ceph orch upgrade status { "target_image": "my-private-repo/quay-io/ceph/ceph@sha256 :34c763383e3323c6bb35f3f2229af9f466518d9db926111277f5e27ed543c427", "in_progress": true, "which": "Upgrading all daemon types on all hosts", "services_complete": [], "progress": "3/59 daemons upgraded", "message": "Error: UPGRADE_EXCEPTION: Upgrade: failed due to an unexpected exception", "is_paused": true } [root@host8 ~]# ceph cephadm check-host host7 check-host failed: Host 'host7' not found. Use 'ceph orch host ls' to see all managed hosts. [root@host8 ~]# ceph versions { "mon": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 5 }, "mgr": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 1, "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)": 3 }, "osd": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 37 }, "mds": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 4 }, "overall": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 47, "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)": 3 } } The strange thing is I can rollback the cluster status by failing to not-upgraded mgr like this: ceph mgr fail ceph orch upgrade start my-private-repo/quay-io/ceph/ceph:v16.2.11 Would you happen to have any idea about this? Best regards, Reza

4 9

bluestore_min_alloc_size and bluefs_shared_alloc_size
by Joel Davidow 13 Mar '24

13 Mar '24

Summary ---------- The relationship of the values configured for bluestore_min_alloc_size and bluefs_shared_alloc_size are reported to impact space amplification, partial overwrites in erasure coded pools, and storage capacity as an osd becomes more fragmented and/or more full. Previous discussions including this topic ---------------------------------------- comment #7 in bug 63618 in Dec 2023 - https://tracker.ceph.com/issues/63618#note-7 pad writeup related to bug 62282 likely from late 2023 - https://pad.ceph.com/p/RCA_62282 email sent 13 Sept 2023 in mail list discussion of cannot create new osd - https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/5M4QAXJDCN… comment #9 in bug 58530 likely from early 2023 - https://tracker.ceph.com/issues/58530#note-9 email sent 30 Sept 2021 in mail list discussion of flapping osds - https://www.mail-archive.com/ceph-users@ceph.io/msg13072.html email sent 25 Feb 2020 in mail list discussion of changing allocation size - https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/B3DGKH6THFG… Current situation ----------------- We have three Ceph clusters that were originally built via cephadm on octopus and later upgraded to pacific. All osds are HDD (will be moving to wal+db on SSD) and were resharded after the upgrade to enable rocksdb sharding. The value for bluefs_shared_alloc_size has remained unchanged at 65535. The value for bluestore_min_alloc_size_hdd was 65535 in octopus but is reported as 4096 by ceph daemon osd.<id> config show in pacific. However, the osd label after upgrading to pacific retains the value of 65535 for bfm_bytes_per_block. BitmapFreelistManager.h in Ceph source code (src/os/bluestore/BitmapFreelistManager.h) indicates that bytes_per_block is bdev_block_size. This indicates that the physical layout of the osd has not changed from 65535 despite the return of the ceph dameon command reporting it as 4096. This interpretation is supported by the Minimum Allocation Size part of the Bluestore configuration reference for quincy (https://docs.ceph.com/en/quincy/rados/configuration/bluestore-config-ref/#m…) Questions ---------- What are the pros and cons of the following three cases with two variations per case - when using co-located wal+db on HDD and when using separate wal+db on SSD: 1) bluefs_shared_alloc_size, bluestore_min_alloc_size, and bfm_bytes_per_block all equal 2) bluefs_shared_alloc_size greater than but a multiple of bluestore_min_alloc_size with bfm_bytes_per_block equal to bluestore_min_alloc_size 3) bluefs_shared_alloc_size greater than but a multiple of bluestore_min_alloc_size with bfm_bytes_per_block equal to bluefs_shared_alloc_size

4 6

PG damaged "failed_repair"
by Romain Lebbadi-Breteau 10 Mar '24

10 Mar '24

Hi, We're a student club from Montréal where we host an Openstack cloud with a Ceph backend for storage of virtual machines and volumes using rbd. Two weeks ago we received an email from our ceph cluster saying that some pages were damaged. We ran "sudo ceph pg repair <pg-id>" but then there was an I/O error on the disk during the recovery ("An unrecoverable disk media error occurred on Disk 4 in Backplane 1 of Integrated RAID Controller 1." and "Bad block medium error is detected at block 0x1377e2ad on Virtual Disk 3 on Integrated RAID Controller 1." messages on iDRAC). After that, the PG we tried to repair was in the state "active+recovery_unfound+degraded". After a week, we ran the command "sudo ceph pg 2.1b mark_unfound_lost revert" to try to recover the damaged PG. We tried to boot the virtual machine that had crashed because of this incident, but the volume seemed to have been completely erased, the "mount" command said there was no filesystem on it, so we recreated the VM from a backup. A few days later, the same PG was once again damaged, and since we knew the physical disk on the OSD hosting one part of the PG had problems, we tried to "out" the OSD from the cluster. That resulted in the two other OSDs hosting copies of the problematic PG to go down, which caused timeouts on our virtual machines, so we put the OSD back in. We then tried to repair the PG again, but that failed and the PG is now "active+clean+inconsistent+failed_repair", and whenever it goes down, two other OSDs from two other hosts go down too after a few minutes, so it's impossible to replace the disk right now, even if we have new ones available. We have backups for most of our services, but it would be very disrupting to delete the whole cluster, and we don't know that to do with the broken PG and the OSD that can't be shut down. Any help would be really appreciated, we're not experts with Ceph and Openstack, and it's likely we handled things wrong at some point, but we really want to go back to a healthy Ceph. Here are some information about our cluster : romain:step@alpha-cen ~ $ sudo ceph health detail HEALTH_ERR 1 scrub errors; Possible data damage: 1 pg inconsistent [ERR] OSD_SCRUB_ERRORS: 1 scrub errors [ERR] PG_DAMAGED: Possible data damage: 1 pg inconsistent pg 2.1b is active+clean+inconsistent+failed_repair, acting [3,11,0] romain:step@alpha-cen ~ $ sudo ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 70.94226 root default -7 20.00792 host alpha-cen 3 hdd 1.81879 osd.3 up 1.00000 1.00000 6 hdd 1.81879 osd.6 up 1.00000 1.00000 12 hdd 1.81879 osd.12 up 1.00000 1.00000 13 hdd 1.81879 osd.13 up 1.00000 1.00000 15 hdd 1.81879 osd.15 up 1.00000 1.00000 16 hdd 9.09520 osd.16 up 1.00000 1.00000 17 hdd 1.81879 osd.17 up 1.00000 1.00000 -5 23.64874 host beta-cen 1 hdd 5.45749 osd.1 up 1.00000 1.00000 4 hdd 5.45749 osd.4 up 1.00000 1.00000 8 hdd 5.45749 osd.8 up 1.00000 1.00000 11 hdd 5.45749 osd.11 up 1.00000 1.00000 14 hdd 1.81879 osd.14 up 1.00000 1.00000 -3 27.28560 host gamma-cen 0 hdd 9.09520 osd.0 up 1.00000 1.00000 5 hdd 9.09520 osd.5 up 1.00000 1.00000 9 hdd 9.09520 osd.9 up 1.00000 1.00000 romain:step@alpha-cen ~ $ sudo rados list-inconsistent-obj 2.1b {"epoch":9787,"inconsistents":[]} romain:step@alpha-cen ~ $ sudo ceph pg 2.1b query https://pastebin.com/gsKCPCjr Best regards, Romain Lebbadi-Breteau

3 6

CephFS On Windows 10
by duluxoz 09 Mar '24

09 Mar '24

Hi All, I'm looking for some pointers/help as to why I can't get my Win10 PC to connect to our Ceph Cluster's CephFS Service. Details are as follows: Ceph Cluster: - IP Addresses: 192.168.1.10, 192.168.1.11, 192.168.1.12 - Each node above is a monitor & an MDS - Firewall Ports: open (ie 3300, etc) - CephFS System Name: my_cephfs - Log files: nothing jumps out at me Windows PC: - Keyring file created and findable: ceph.client.me.keyring - dokany installed - ceph-for-windows installed - Can ping all three ceph nodes - Connection command: ceph-dokan -l v -o -id me --debug --client_fs my_cephfs -c C:\ProgramData\Ceph\ceph.conf Ceph.conf contents: ~~~ [global] mon_host = 192.168.1.10, 192.168.1.11, 192.168.1.12 log to stderr = true log to syslog = true run dir = C:/ProgramData/ceph crash dir = C:/logs/ceph debug client = 2 [client] keyring = C:/ProgramData/ceph/ceph.client.me.keyring log file = C:/logs/ceph/$name.$pid.log admin socket = C:/ProgramData/ceph/$name.$pid.asok ~~~ Windows Logfile contents (ieC:/logs/ceph/client.me.NNNN.log): ~~~ 2024-02-28T18:26:45.201+1100 1 0 monclient(hunting): authenticate timed out after 300 2024-02-28T18:31:45.203+1100 1 0 monclient(hunting): authenticate timed out after 300 2024-02-28T18:36:45.205+1100 1 0 monclient(hunting): authenticate timed out after 300 ~~~ Additional info from Windows CLI: ~~~ failed to fetch mon config (--no-mon-config to skip) ~~~ So I've gone through the doco and done some Google-foo and I can't work out *why* I'm getting a failure; why I'm getting the authentication failure. I know it'll be something simple, something staring me in the face, but I'm at the point where I can't see the forest for the trees - please help. Any help greatly appreciated Thanks in advance Cheers Dulux-Oz

3 4

Remove cluster_network without routing
by Torkil Svensgaard 07 Mar '24

07 Mar '24

Hi Cephadm Reef 18.2.0. We would like to remove our cluster_network without stopping the cluster and without having to route between the networks. global advanced cluster_network 192.168.100.0/24 * global advanced public_network 172.21.12.0/22 * The documentation[1] states: " You may specifically assign static IP addresses or override cluster_network settings using the cluster_addr setting for specific OSD daemons. " So for one OSD at a time I could set cluster_addr to override the cluster_network IP and use the public_network IP instead? As the containers are using host networking they have access to both IPs and will just layer 2 the traffic, avoiding routing? When all OSDs are running with a public_network IP set via cluster_addr we can just delete the cluster_network setting and then remove all the ceph_addr settings, as with no cluster_network setting the public_network setting will be used? We tried with one OSD and it seems to work. Anyone see a problem with this approach? Thanks Torkil [1] https://docs.ceph.com/en/latest/rados/configuration/network-config-ref/#id3 -- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil(a)drcmr.dk

2 2

Disable signature url in ceph rgw
by marc＠singer.services 07 Mar '24

07 Mar '24

Hi Ceph users We are using Ceph Pacific (16) in this specific deployment. In our use case we do not want our users to be able to generate signature v4 URLs because they bypass the policies that we set on buckets (e.g IP restrictions). Currently we have a sidecar reverse proxy running that filters requests with signature URL specific request parameters. This is obviously not very efficient and we are looking to replace this somehow in the future. 1. Is there an option in RGW to disable this signed URLs (e.g returning status 403)? 2. If not is this planned or would it make sense to add it as a configuration option? 3. Or is the behaviour of not respecting bucket policies in RGW with signature v4 URLs a bug and they should be actually applied? Thanks you for your help and let me know if you have any questions Marc Singer

4 4

erasure-code-lrc Questions regarding repair
by Ansgar Jazdzewski 07 Mar '24

07 Mar '24

hi folks, I currently test erasure-code-lrc (1) in a multi-room multi-rack setup. The idea is to be able to repair a disk-failures within the rack itself to lower bandwidth-usage ```bash ceph osd erasure-code-profile set lrc_hdd \ plugin=lrc \ crush-root=default \ crush-locality=rack \ crush-failure-domain=host \ crush-device-class=hdd \ mapping=__DDDDD__DDDDD__DDDDD__DDDDD \ layers=' [ [ "_cDDDDD_cDDDDD_cDDDDD_cDDDDD", "" ], [ "cDDDDDD_____________________", "" ], [ "_______cDDDDDD______________", "" ], [ "______________cDDDDDD_______", "" ], [ "_____________________cDDDDDD", "" ], ]' \ crush-steps='[ [ "choose", "room", 4 ], [ "choose", "rack", 1 ], [ "chooseleaf", "host", 7 ], ]' ``` The roule picks 4 out of 5 rooms and keeps the PG in one rack like expected! However it looks like the PG will not move to another Room if the PG is undersized or the entire Room or Rack is down! Questions: * do I miss something to allow LRC (PG's) to move across Racks/Rooms for repair? * Is it even possible to build such a 'Multi-stage' grushmap? Thanks for your help, Ansgar 1) https://docs.ceph.com/en/quincy/rados/operations/erasure-code-jerasure/

2 2

Ceph is constantly scrubbing 1/4 of all PGs and still have pigs not scrubbed in time
by thymus_03fumbler＠icloud.com 07 Mar '24

07 Mar '24

I recently switched from 16.2.x to 18.2.x and migrated to cephadm, since the switch the cluster is constantly scrubbing, 24/7 up to 50 PGs simultaneously and up to 20 deep scrubs simultaneously in a cluster that has only 12 (in use) OSDs. Furthermore it still manages to regularly have a warning with ‘pgs not scrubbed in time’ I have tried various settings, like osd_deep_scrub_interval, osd_max_scrubs, mds_max_scrub_ops_in_progress etc. All those get ignored. Please advice. Here is an output of ceos config dump: WHO MASK LEVEL OPTION VALUE RO global advanced auth_client_required cephx * global advanced auth_cluster_required cephx * global advanced auth_service_required cephx * global advanced auth_supported cephx * global basic container_image quay.io/ceph/ceph@sha256:aca35483144ab3548a7f670db9b79772e6fc51167246421c66c0bd56a6585468 * global basic device_failure_prediction_mode local global advanced mon_allow_pool_delete true global advanced mon_data_avail_warn 20 global advanced mon_max_pg_per_osd 400 global advanced osd_max_pg_per_osd_hard_ratio 10.000000 global advanced osd_pool_default_pg_autoscale_mode on mon advanced auth_allow_insecure_global_id_reclaim false mon advanced mon_crush_min_required_version firefly * mon advanced mon_warn_on_pool_no_redundancy false mon advanced public_network 10.79.0.0/16 * mgr advanced mgr/balancer/active true mgr advanced mgr/balancer/mode upmap mgr advanced mgr/cephadm/manage_etc_ceph_ceph_conf_hosts label:admin * mgr advanced mgr/cephadm/migration_current 6 * mgr advanced mgr/dashboard/GRAFANA_API_PASSWORD admin * mgr advanced mgr/dashboard/GRAFANA_API_SSL_VERIFY false * mgr advanced mgr/dashboard/GRAFANA_API_URL https://10.79.79.12:3000 * mgr advanced mgr/dashboard/PROMETHEUS_API_HOST http://10.79.79.12:9095 * mgr advanced mgr/devicehealth/enable_monitoring true mgr advanced mgr/orchestrator/orchestrator cephadm osd advanced osd_map_cache_size 250 osd advanced osd_map_share_max_epochs 50 osd advanced osd_mclock_profile high_client_ops osd advanced osd_pg_epoch_persisted_max_stale 50 osd.0 basic osd_mclock_max_capacity_iops_hdd 380.869888 osd.1 basic osd_mclock_max_capacity_iops_hdd 441.000000 osd.10 basic osd_mclock_max_capacity_iops_ssd 13677.906485 osd.11 basic osd_mclock_max_capacity_iops_hdd 274.411212 osd.13 basic osd_mclock_max_capacity_iops_hdd 198.492501 osd.2 basic osd_mclock_max_capacity_iops_hdd 251.592009 osd.3 basic osd_mclock_max_capacity_iops_hdd 208.197434 osd.4 basic osd_mclock_max_capacity_iops_hdd 196.544082 osd.5 basic osd_mclock_max_capacity_iops_ssd 12739.225456 osd.6 basic osd_mclock_max_capacity_iops_hdd 211.288660 osd.7 basic osd_mclock_max_capacity_iops_hdd 210.543236 osd.8 basic osd_mclock_max_capacity_iops_hdd 242.241594 osd.9 basic osd_mclock_max_capacity_iops_hdd 559.933780 mds.plexfs basic mds_join_fs plexfs Here is a ceph -s output services: mon: 3 daemons, quorum lxt-prod-ceph-util02,lxt-prod-ceph-util01,lxt-prod-ceph-util03 (age 3w) mgr: lxt-prod-ceph-util02.iyrhxj(active, since 3w), standbys: lxt-prod-ceph-util03.wvstpe mds: 1/1 daemons up osd: 14 osds: 14 up (since 4w), 14 in (since 4w) data: volumes: 1/1 healthy pools: 4 pools, 193 pgs objects: 14.48M objects, 52 TiB usage: 71 TiB used, 39 TiB / 110 TiB avail pgs: 131 active+clean 47 active+clean+scrubbing 15 active+clean+scrubbing+deep

3 2

ceph Quincy to Reef non cephadm upgrade
by sarda.ravi＠gmail.com 07 Mar '24

07 Mar '24

I want to perform non cephadm upgrade from Quincy to Reef. Reason for not using cephadm is do not want to go for ceph in containers. My test deployment is as given below. Total cluster hosts : 5 ceph-mon hosts: 3 ceph-mgr hosts: 3 (ceph-mgr active on one node, and other ceph-mgr each on ceph-mon host) ceph-mds : 1 ceph-osd : 5 (one ceph-osd on each of the host in the cluster.) While I try to follow the steps - https://docs.ceph.com/en/latest/releases/reef/#upgrading-non-cephadm-cluste… - on the step - Upgrade monitors by installing the new packages and restarting the monitor daemons. when I try to upgrade only ceph-mon using "apt upgrade ceph-mon" command it upgrades all packages including ceph-mgr, ceph-mds, ceph-osd etc. as ceph-mon package has dependency on these packages. My question is - does this mean I need to upgrade all ceph packages (ceph, ceph-common) and restart only monitor daemon first? Or there is any way I can upgrade only ceph-mon pacakge first, then ceph-mgr, ceph-osd and so on?

2 1

change ip node and public_network in cluster
by farhad kh 06 Mar '24

06 Mar '24

I have implemented a ceph cluster with cephadm which has three monitors and three OSDs each node have one interface 192.168.0.0/24 network. I want to change the address of the machines to the range 10.4.4.0/24. Is there a solution for this change without data loss and failure? i change the pubic_network in mon and change the ip node but its not worked . how can i sovle this problem? ``` ceph orch host ls HOST ADDR LABELS STATUS ceph-01 192.168.0.130 _admin,rgw ceph-02 192.168.0.131 _admin,rgw ceph-03 192.168.0.132 _admin,rgw 3 hosts in cluster ```` [root@ceph-01 ~]# ceph config get mon public_network 192.168.0.0/24 ```` [root@ceph-01 ~]# ceph orch ls NAME PORTS RUNNING REFRESHED AGE PLACEMENT alertmanager ?:9093,9094 1/1 112s ago 9M count:1 ceph-exporter 3/3 114s ago 8M * crash 3/3 114s ago 9M * grafana ?:3000 1/1 112s ago 8M count:1 mgr 2/2 113s ago 9M count:2 mon 3/3 114s ago 8M count:3 node-exporter ?:9100 3/3 114s ago 9M * osd.dashboard-admin-1685787597651 6 114s ago 8M * prometheus ?:9095 1/1 112s ago 3M count:1 ````

3 3

Slow RGW multisite sync due to "304 Not Modified" responses on primary zone
by Alam Mohammad 05 Mar '24

05 Mar '24

Hi, We have 2 clusters (v18.2.1) primarily used for RGW which has over 2+ billion RGW objects. They are also in multisite configuration totaling to 2 zones and we've got around 2 Gbps of bandwidth dedicated (P2P) for the multisite traffic. We see that using "radosgw-admin sync status" on the zone 2, all the 128 shards are recovering and unfortunately there is very less data transfer from primary zone ie., the link utilization is barely 100 Mbps / 2 Gbps. Our objects are quite small as well like avg. of 1 MB in size. On further inspection, we noticed the rgw access the logs at primary site are mostly yielding "304 Not Modified" for RGWs at site-2. Is this expected? Here are some of the logs (information is redacted) root@host-04:~# tail -f /var/log/haproxy-msync.log Feb 12 05:06:51 host-04 haproxy[971171]: 10.1.85.14:33730 [12/Feb/2024:05:06:51.047] https~ backend/host-04-msync 0/0/0/2/2 304 143 - - ---- 56/55/1/0/0 0/0 "GET /bucket1/object1.jpg?rgwx-zonegroup=71dceb3d-3092-4dc6-897f-a9abf60c9972&rgwx-prepend-metadata=true&rgwx-sync-manifest&rgwx-sync-cloudtiered&rgwx-skip-decrypt&rgwx-if-not-replicated-to=a8204ce2-b69e-4d90-bca1-93edd05a1a29%3Abucket1%3A8b96aea5-c763-40a3-8430-efd67cff0c62.20010.7 HTTP/1.1" Feb 12 05:06:51 host-04 haproxy[971171]: 10.1.85.14:59730 [12/Feb/2024:05:06:51.048] https~ backend/host-04-msync 0/0/0/2/2 304 143 - - ---- 56/55/3/1/0 0/0 "GET /bucket1/object91.jpg?rgwx-zonegroup=71dceb3d-3092-4dc6-897f-a9abf60c9972&rgwx-prepend-metadata=true&rgwx-sync-manifest&rgwx-sync-cloudtiered&rgwx-skip-decrypt&rgwx-if-not-replicated-to=a8204ce2-b69e-4d90-bca1-93edd05a1a29%3Abucket1%3A8b96aea5-c763-40a3-8430-efd67cff0c62.20010.7 HTTP/1.1" We also took a look at our grafana instance and out of 1000 requests / second, 200 are "200 OK" and 800 are "304 Not Modified". Sync threads are run on only 2 rgw daemons per zone and are behind a Load Balancer. "# radosgw-admin sync error list" also contains around 20 errors which are mostly automatically recoverable. As we understand, does it mean that RGW multisite sync logs in the log pool are yet to be generated or some sort? Please provide us some insights and let us know how to resolve this. Thanks, Saif

2 2

ceph-crash NOT reporting crashes due to wrong permissions on /var/lib/ceph/crash/posted (Debian / Ubuntu packages)
by Christian Rohmann 04 Mar '24

04 Mar '24

Hey ceph-users, I just noticed issues with ceph-crash using the Debian /Ubuntu packages (package: ceph-base): While the /var/lib/ceph/crash/posted folder is created by the package install, it's not properly chowned to ceph:ceph by the postinst script. This might also affect RPM based installs somehow, but I did not look into that. I opened a bug report with all the details and two ideas to fix this: https://tracker.ceph.com/issues/64548 The wrong ownership causes ceph-crash to NOT work at all. I myself missed quite a few crash reports. All of them were just sitting around on the machines, but were reported right after I did chown ceph:ceph /var/lib/ceph/crash/posted systemctl restart ceph-crash.service You might want to check if you might be affected as well. Failing to post crashes to the local cluster results in them not being reported back via telemetry. Regards Christian

3 3

Re: Scrub stuck and 'pg has invalid (post-split) stat'
by Eugen Block 01 Mar '24

01 Mar '24

Please don't drop the list from your response. The first question coming to mind is, why do you have a cache-tier if all your pools are on nvme decices anyway? I don't see any benefit here. Did you try the suggested workaround and disable the cache-tier? Zitat von Cedric <yipikai7(a)gmail.com>: > Thanks Eugen, see attached infos. > > Some more details: > > - commands that actually hangs: ceph balancer status ; rbd -p vms ls ; > rados -p vms_cache cache-flush-evict-all > - all scrub running on vms_caches pgs are stall / start in a loop > without actually doing anything > - all io are 0 both from ceph status or iostat on nodes > > On Tue, Feb 20, 2024 at 10:00 AM Eugen Block <eblock(a)nde.ag> wrote: >> >> Hi, >> >> some more details would be helpful, for example what's the pool size >> of the cache pool? Did you issue a PG split before or during the >> upgrade? This thread [1] deals with the same problem, the described >> workaround was to set hit_set_count to 0 and disable the cache layer >> until that is resolved. Afterwards you could enable the cache layer >> again. But keep in mind that the code for cache tier is entirely >> removed in Reef (IIRC). >> >> Regards, >> Eugen >> >> [1] >> https://ceph-users.ceph.narkive.com/zChyOq5D/ceph-strange-issue-after-addin… >> >> Zitat von Cedric <yipikai7(a)gmail.com>: >> >> > Hello, >> > >> > Following an upgrade from Nautilus (14.2.22) to Pacific (16.2.13), we >> > encounter an issue with a cache pool becoming completely stuck, >> > relevant messages below: >> > >> > pg xx.x has invalid (post-split) stats; must scrub before tier agent >> > can activate >> > >> > In OSD logs, scrubs are starting in a loop without succeeding for all >> > pg of this pool. >> > >> > What we already tried without luck so far: >> > >> > - shutdown / restart OSD >> > - rebalance pg between OSD >> > - raise the memory on OSD >> > - repeer PG >> > >> > Any idea what is causing this? any help will be greatly appreciated >> > >> > Thanks >> > >> > Cédric >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users(a)ceph.io >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io

3 5

Renaming an OSD node
by Deep Dish 01 Mar '24

01 Mar '24

Hello. We have a requirement to change the hostname on some of our OSD nodes. All of our nodes are Ubuntu 22.04 based and have been deployed using 17.2.7 Orchestrator. 1. Is there a procedure to rename the existing node, without rebuilding and have it detected by Ceph Orchestrator? If not, 2. To minimize impact on cluster (rebuilding OSDs / balancing, etc) Is it possible to REINTRODUCE existing OSDs into the cluster a the newly rebuilt node? Is there a ceph orch process to scan local node OSDs, detect and create OSD daemons? Thank you.

2 1

Migration from ceph-ansible to Cephadm
by wodel youchi 29 Feb '24

29 Feb '24

Hi, I am in the middle of migration from ceph-ansible to cephadm (version quincy), so far so good ;-). And I have some questions : - I still have the ceph-crash container, what should I do with it? - The new rgw and mds daemons have some random string in their names (like rgw.opsrgw.controllera.*pkajqw*), is this correct ? - How should I proceed with the monitoring stack (grafana, prometheus, alermanager and node-exporter)? should I stop then delete the old ones, then deploy the new ones with ceph orch? Regards.

2 1

Ceph & iSCSI
by Michael Worsham 29 Feb '24

29 Feb '24

I was reading on the Ceph site that iSCSI is no longer under active development since November 2022. Why is that? https://docs.ceph.com/en/latest/rbd/iscsi-overview/ -- Michael This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments.

5 5

Dropping focal for squid
by Reed Dier 28 Feb '24

28 Feb '24

Found this mention in the CLT Minutes posted this morning[1], of a discussion on ceph-dev[2] about dropping ubuntu focal builds for the squid release, and beginning builds of quincy for jammy to facilitate quincy->squid upgrades. > there was a consensus to drop support for ubuntu focal and centos > stream 8 with the squid release, and i'd love to remove those distros > from the shaman build matrix for squid and main branches asap > > however, i see that quincy never supported ubuntu jammy, so our quincy > upgrade tests still have to run against focal. that means we'd still > have to build focal packages for squid > > would it be possible to start building jammy packages for quincy to > allow those upgrade tests to run jammy instead? Just wanting to voice my support for this, as this both seems to match the historical ceph:ubuntu cadence going back roughly a decade, and helps facilitate a narrow upgrade window for ubuntu users to get to jammy. +----------+-----+-----+-----+-----+-----+-----+ | ceph | u14 | u16 | u18 | u20 | u22 | u24 | +----------+-----+-----+-----+-----+-----+-----+ | jewel | + | + | - | - | - | - | | luminous | + | + | - | - | - | - | | mimic | - | + | + | - | - | - | | nautilus | - | + | + | - | - | - | | octopus | - | - | + | + | - | - | | pacific | - | - | + | + | - | - | | quincy | - | - | - | + | M | - | | reef | - | - | - | + | + | - | | squid | - | - | - | - | E | E | | T | - | - | - | - | E | E | +----------+-----+-----+-----+-----+-----+-----+ Hopefully this table translates the mailing list well enough. But going back to both jewel/10 and Ubuntu 14.04/trusty, there has been a consistent 4 ceph releases per ubuntu LTS release, with a dist drop/add every two releases. This gives ample window for users to upgrade ubuntu and ceph at a reasonable pace. However with quincy not being built for jammy (M=missing), this broke the trend and forced anyone looking to get to jammy to have to go all the way to reef, which they may not be ready to do just yet. Running the table out to the T release and ubuntu 24.04/noble, following this trend, it would be expected (E=expected) that squid would be built for jammy (and eventually noble), and the same would be true for the T release. Many words to say that as a user this would be beneficial to me, and likely others. Reed [1] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/SI3KZTU6GLG… <https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/SI3KZTU6GLG…> [2] https://lists.ceph.io/hyperkitty/list/dev@ceph.io/thread/ONAWOAE7MPMT7CP6KH… <https://lists.ceph.io/hyperkitty/list/dev@ceph.io/thread/ONAWOAE7MPMT7CP6KH…>

1 0

Ceph Leadership Team Meeting, 2024-02-28 Minutes
by Patrick Donnelly 28 Feb '24

28 Feb '24

Hi folks, Today we discussed: - [casey] on dropping ubuntu focal support for squid - Discussion thread: https://lists.ceph.io/hyperkitty/list/dev@ceph.io/thread/ONAWOAE7MPMT7CP6KH… - Quincy doesn't build jammy packages, so quincy->squid upgrade tests have to run on focal - proposing to add jammy packages for quincy to enable that upgrade path (from 17.2.8+) - https://github.com/ceph/ceph-build/pull/2206 - Need to indicate that Quincy clusters must upgrade to jammy before upgrading to Squid. - T release name: https://pad.ceph.com/p/t - Tentacle wins! - Patrick to do release kick-off - Cephalocon news? - Planning is in progress; no news as knowledgeable parties not present for this meeting. - Volunteers for compiling the Contributor Credits? - https://tracker.ceph.com/projects/ceph/wiki/Ceph_contributors_list_maintena… - Laura will give it a try. - Plan for tagged vs. named Github milestones? - Continue using priority order for qa testing: exhaust testing on tagged milestone, then go to "release" catch-all milestone - v18.2.2 hotfix release next - Reef HEAD is still cooking with to-be-addressed upgrade issues. - v19.1.0 (first Squid RC) - two rgw features still waiting to go into squid - cephfs quiesce feature to be backported - Nightlies crontab to be updated by Patrick. - V19.1.0 milestone: https://github.com/ceph/ceph/milestone/21 -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

Possible to tune Full Disk warning ??
by Daniel Brown 28 Feb '24

28 Feb '24

Greetings - Is there any way to tune what CEPH will complain about, in terms of “full disks” ?? One of my ceph servers has an NFS mount which is for all intents and purposes “read only” and is sitting at 100% full. Ceph keeps warning me about this, unless I unmount the nfs mount point. Is there any way to tell it to ignore that mount point? I’m using reef 18.2.1, running on Ubuntu, which was setup with cephadm.

2 2

pg repair doesn't fix "got incorrect hash on read" / "candidate had an ec hash mismatch"
by Kai Stian Olstad 27 Feb '24

27 Feb '24

Hi, Short summary PG 404.bc is an EC 4+2 where s0 and s2 report hash mismtach for 698 objects. Ceph pg repair doesn't fix it, because if you run deep-srub on the PG after repair is finished, it still report scrub errors. Why can't ceph pg repair repair this, it has 4 out of 6 should be able to reconstruct the corrupted shards? Is there a way to fix this? Like delete object s0 and s2 so it's forced to recreate them? Long detailed summary A short backstory. * This is aftermath of problems with mclock, post "17.2.7: Backfilling deadlock / stall / stuck / standstill" [1]. - 4 OSDs had a few bad sectors, set all 4 out and cluster stopped. - Solution was to swap from mclock to wpq and restart alle OSD. - When all backfilling was finished all 4 OSD was replaced. - osd.223 and osd.269 was 2 of the 4 OSDs that was replaced. PG / pool 404 is EC 4+2 default.rgw.buckets.data 9 days after the osd.223 og osd.269 was replaced, deep-scub was run and reported errors ceph status ----------- HEALTH_ERR 1396 scrub errors; Possible data damage: 1 pg inconsistent [ERR] OSD_SCRUB_ERRORS: 1396 scrub errors [ERR] PG_DAMAGED: Possible data damage: 1 pg inconsistent pg 404.bc is active+clean+inconsistent, acting [223,297,269,276,136,197] I then run repair ceph pg repair 404.bc And ceph status showed this ceph status ----------- HEALTH_WARN Too many repaired reads on 2 OSDs [WRN] OSD_TOO_MANY_REPAIRS: Too many repaired reads on 2 OSDs osd.223 had 698 reads repaired osd.269 had 698 reads repaired But osd.223 and osd.269 is new disks and the disks has no SMART error or any I/O error in OS logs. So I tried to run deep-scrub again on the PG. ceph pg deep-scrub 404.bc And got this result. ceph status ----------- HEALTH_ERR 1396 scrub errors; Too many repaired reads on 2 OSDs; Possible data damage: 1 pg inconsistent [ERR] OSD_SCRUB_ERRORS: 1396 scrub errors [WRN] OSD_TOO_MANY_REPAIRS: Too many repaired reads on 2 OSDs osd.223 had 698 reads repaired osd.269 had 698 reads repaired [ERR] PG_DAMAGED: Possible data damage: 1 pg inconsistent pg 404.bc is active+clean+scrubbing+deep+inconsistent+repair, acting [223,297,269,276,136,197] 698 + 698 = 1396 so the same amount of errors. Run repair again on 404.bc and ceph status is HEALTH_WARN Too many repaired reads on 2 OSDs [WRN] OSD_TOO_MANY_REPAIRS: Too many repaired reads on 2 OSDs osd.223 had 1396 reads repaired osd.269 had 1396 reads repaired So even when repair finish it doesn't fix the problem since they reappear again after a deep-scrub. The log for osd.223 and osd.269 contain "got incorrect hash on read" and "candidate had an ec hash mismatch" for 698 unique objects. But i only show the logs for 1 of the 698 object, the log is the same for the other 697 objects. osd.223 log (only showing 1 of 698 object named 2021-11-08T19%3a43%3a50,145489260+00%3a00) ----------- Feb 20 10:31:00 ceph-hd-003 ceph-osd[3665432]: osd.223 pg_epoch: 231235 pg[404.bcs0( v 231235'1636919 (231078'1632435,231235'1636919] local-lis/les=226263/226264 n=296580 ec=36041/27862 lis/c=226263/226263 les/c/f=226264/230954/0 sis=226263) [223,297,269,276,136,197]p223(0) r=0 lpr=226263 crt=231235'1636919 lcod 231235'1636918 mlcod 231235'1636918 active+clean+scrubbing+deep+inconsistent+repair [ 404.bcs0: REQ_SCRUB ] MUST_REPAIR MUST_DEEP_SCRUB MUST_SCRUB planned REQ_SCRUB] _scan_list 404:3d001f95:::1f244892-a2e7-406b-aa62-1b13511333a2.625411.3__multipart_2021-11-08T19%3a43%3a50,145489260+00%3a00.2~OoetD5vkh8fyh-2eeR7GF5rZK7d5EVa.1:head got incorrect hash on read 0xc5d1dd1b != expected 0x7c2f86d7 Feb 20 10:31:01 ceph-hd-003 ceph-osd[3665432]: log_channel(cluster) log [ERR] : 404.bc shard 223(0) soid 404:3d001f95:::1f244892-a2e7-406b-aa62-1b13511333a2.625411.3__multipart_2021-11-08T19%3a43%3a50,145489260+00%3a00.2~OoetD5vkh8fyh-2eeR7GF5rZK7d5EVa.1:head : candidate had an ec hash mismatch Feb 20 10:31:01 ceph-hd-003 ceph-osd[3665432]: log_channel(cluster) log [ERR] : 404.bc shard 269(2) soid 404:3d001f95:::1f244892-a2e7-406b-aa62-1b13511333a2.625411.3__multipart_2021-11-08T19%3a43%3a50,145489260+00%3a00.2~OoetD5vkh8fyh-2eeR7GF5rZK7d5EVa.1:head : candidate had an ec hash mismatch Feb 20 10:31:01 ceph-hd-003 ceph-b321e76e-da3a-11eb-b75c-4f948441dcd0-osd-223[3665427]: 2024-02-20T10:31:01.117+0000 7f128a88d700 -1 log_channel(cluster) log [ERR] : 404.bc shard 223(0) soid 404:3d001f95:::1f244892-a2e7-406b-aa62-1b13511333a2.625411.3__multipart_2021-11-08T19%3a43%3a50,145489260+00%3a00.2~OoetD5vkh8fyh-2eeR7GF5rZK7d5EVa.1:head : candidate had an ec hash mismatch Feb 20 10:31:01 ceph-hd-003 ceph-b321e76e-da3a-11eb-b75c-4f948441dcd0-osd-223[3665427]: 2024-02-20T10:31:01.117+0000 7f128a88d700 -1 log_channel(cluster) log [ERR] : 404.bc shard 269(2) soid 404:3d001f95:::1f244892-a2e7-406b-aa62-1b13511333a2.625411.3__multipart_2021-11-08T19%3a43%3a50,145489260+00%3a00.2~OoetD5vkh8fyh-2eeR7GF5rZK7d5EVa.1:head : candidate had an ec hash mismatch osd.269 log (only showing 1 of 698 object named 2021-11-08T19%3a43%3a50,145489260+00%3a00) ----------- Feb 20 10:31:00 ceph-hd-001 ceph-osd[3656897]: osd.269 pg_epoch: 231235 pg[404.bcs2( v 231235'1636919 (231078'1632435,231235'1636919] local-lis/les=226263/226264 n=296580 ec=36041/27862 lis/c=226263/226263 les/c/f=226264/230954/0 sis=226263) [223,297,269,276,136,197]p223(0) r=2 lpr=226263 luod=0'0 crt=231235'1636919 mlcod 231235'1636919 active mbc={}] _scan_list 404:3d001f95:::1f244892-a2e7-406b-aa62-1b13511333a2.625411.3__multipart_2021-11-08T19%3a43%3a50,145489260+00%3a00.2~OoetD5vkh8fyh-2eeR7GF5rZK7d5EVa.1:head got incorrect hash on read 0x7c0871dc != expected 0xcf6f4c58 The log for the other osd in the PG osd.297, osd.276, osd.136 and osd.197 doesn't show any error. If I try to get the object it failes $ s3cmd s3://benchfiles/2021-11-08T19:43:50,145489260+00:00 download: 's3://benchfiles/2021-11-08T19:43:50,145489260+00:00' -> './2021-11-08T19:43:50,145489260+00:00' [1 of 1] ERROR: Download of './2021-11-08T19:43:50,145489260+00:00' failed (Reason: 500 (UnknownError)) ERROR: S3 error: 500 (UnknownError) And the RGW log show this Feb 21 08:27:06 ceph-mon-1 radosgw[1747]: ====== starting new request req=0x7f94b744d660 ===== Feb 21 08:27:06 ceph-mon-1 radosgw[1747]: WARNING: set_req_state_err err_no=5 resorting to 500 Feb 21 08:27:06 ceph-mon-1 radosgw[1747]: ====== starting new request req=0x7f94b6e41660 ===== Feb 21 08:27:06 ceph-mon-1 radosgw[1747]: ====== req done req=0x7f94b744d660 op status=-5 http_status=500 latency=0.020000568s ====== Feb 21 08:27:06 ceph-mon-1 radosgw[1747]: beast: 0x7f94b744d660: 110.2.0.46 - test1 [21/Feb/2024:08:27:06.021 +0000] "GET /benchfiles/2021-11-08T19%3A43%3A50%2C145489260%2B00%3A00 HTTP/1.1" 500 226 - - - latency=0.020000568s [1] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/IPHBE3DLW5A… -- Kai Stian Olstad

3 5

OSD with dm-crypt?
by Michael Worsham 27 Feb '24

27 Feb '24

Is there a how-to document or cheat sheet on how to enable OSD encryption using dm-crypt? -- Michael This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments.

3 4

ceph-mgr client.0 error registering admin socket command: (17) File exists
by Denis Polom 27 Feb '24

27 Feb '24

Hi, running Ceph Quincy 17.2.7 on Ubuntu Focal LTS, ceph-mgr service reports following errors: client.0 error registering admin socket command: (17) File exists I don't use any extra mgr configuration: mgr advanced mgr/balancer/active true mgr advanced mgr/balancer/log_level debug mgr advanced mgr/balancer/log_to_cluster true mgr advanced mgr/balancer/mode upmap mgr advanced mgr/balancer/upmap_max_deviation 1 mgr advanced mgr/balancer/upmap_max_optimizations 20 mgr advanced mgr/prometheus/cache true Do you have some idea, what's the cause and how to fix it? Thank you

2 1

Sata SSD trim latency with (WAL+DB on NVME + Sata OSD)
by Özkan Göksu 26 Feb '24

26 Feb '24

Hello. With the SSD drives without tantalum capacitors Ceph faces trim latency on every write. I wonder if the behavior is the same if we locate WAL+DB on NVME drives with "Tantalum capacitors" ? Do I need to use NVME + SAS SSD to avoid this latency issue? Best regards.

1 0

Some questions about cephadm
by wodel youchi 26 Feb '24

26 Feb '24

Hi, I have some questions about ceph using cephadm. I used to deploy ceph using ceph-ansible, now I have to move to cephadm, I am in my learning journey. - How can I tell my cluster that it's a part of an HCI deployment? With ceph-ansible it was easy using is_hci : yes - The documentation of ceph does not indicate what versions of grafana, prometheus, ...etc should be used with a certain version. - I am trying to deploy Quincy, I did a bootstrap to see what containers were downloaded and their version. - I am asking because I need to use a local registry to deploy those images. - After the bootstrap, the Web interface was accessible : - How can I access the wizard page again? If I don't use it the first time I could not find another way to get it. - I had a problem with telemetry, I did not configure telemetry, then when I clicked the button, the web gui became inaccessible.....????!!! Regards.

6 11

pacific 16.2.15 QE validation status
by Yuri Weinstein 26 Feb '24

26 Feb '24

Details of this release are summarized here: https://tracker.ceph.com/issues/64151#note-1 Seeking approvals/reviews for: rados - Radek, Laura, Travis, Ernesto, Adam King rgw - Casey fs - Venky rbd - Ilya krbd - in progress upgrade/nautilus-x (pacific) - Casey PTL (regweed tests failed) upgrade/octopus-x (pacific) - Casey PTL (regweed tests failed) upgrade/pacific-x (quincy) - in progress upgrade/pacific-p2p - Ilya PTL (maybe rbd related?) ceph-volume - Guillaume TIA YuriW

12 27

Seperate metadata pool in 3x MDS node
by Özkan Göksu 26 Feb '24

26 Feb '24

Hello folks! I'm designing a new Ceph storage from scratch and I want to increase CephFS speed and decrease latency. Usually I always build (WAL+DB on NVME with Sas-Sata SSD's) and I deploy MDS and MON's on the same servers. This time a weird idea came to my mind and I think it has great potential and will perform better on paper with my limited knowledge. I have 5 racks and the 3nd "middle" rack is my storage and management rack. - At RACK-3 I'm gonna locate 8x 1u OSD server (Spec: 2x E5-2690V4, 256GB, 4x 25G, 2x 1.6TB PCI-E NVME "MZ-PLK3T20", 8x 4TB SATA SSD) - My Cephfs kernel clients are 40x GPU nodes located at RACK1,2,4,5 With my current workflow, all the clients; 1- visit the rack data switch 2- jump to main VPC switch via 2x100G, 3- talk with MDS servers, 4- Go back to the client with the answer, 5- To access data follow the same HOP's and visit the OSD's everytime. If I deploy separate metadata pool by using 4x MDS server at top of RACK-1,2,4,5 (Spec: 2x E5-2690V4, 128GB, 2x 10G(Public), 2x 25G (cluster), 2x 960GB U.2 NVME "MZ-PLK3T20") Then all the clients will make the request directly in-rack 1 HOP away MDS servers and if the request is only metadata, then the MDS node doesn't need to redirect the request to OSD nodes. Also by locating MDS servers with seperated metadata pool across all the racks will reduce the high load on main VPC switch at RACK-3 If I'm not missing anything then only Recovery workload will suffer with this topology. What do you think?

3 3

Cephadm and Ceph.conf
by Michael Worsham 26 Feb '24

26 Feb '24

I deployed a Ceph reef cluster using cephadm. When it comes to the ceph.conf file, which file should I be editing for making changes to the cluster - the one running under the docker container or the local one on the Ceph monitors? -- Michael This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments.

2 3

ambigous mds behind on trimming and slowops (ceph 17.2.5 and rook operator 1.10.8)
by a.warkhade98＠gmail.com 26 Feb '24

26 Feb '24

Team, Guys, We were facing cephFs volume mount issue and ceph status it was showing mds slow requests Mds behind on trimming After restarting mds pods it was resolved But wanted to know Root caus of this It was started after 2 hours of one of the active mds was crashed So does that an active mds crash can cause this issue ? Please provide your inputs anyone

2 2

What exactly does the osd pool repair funtion do?
by Aleksander Pähn 26 Feb '24

26 Feb '24

What exactly does the osd pool repair function do? Documentation is not clear. Kind regards, AP This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.

2 1

Is a direct Octopus to Reef Upgrade Possible?
by Alex Hussein-Kershaw (HE/HIM) 26 Feb '24

26 Feb '24

Hi ceph-users, I currently use Ceph Octopus to provide CephFS & S3 Storage for our app servers, deployed in containers by ceph-ansible. I'm planning to take an upgrade to get off Ceph Octopus as it's EOL. I'd love to go straight to reef, but vaguely remember reading a statement that only two major versions can be taken on upgrade. I've failed to find that statement again. Is it possible to go directly from Octopus straight to Reef? I think a sensible approach here is to first migrate our existing deployments to use cephadm, and then use cephadm to upgrade. Any advice on this very welcome. Many thanks, Alex

2 1

Ceph MDS randomly hangs when pg nums reduced
by lokitingyi＠gmail.com 26 Feb '24

26 Feb '24

Hi, I have a CephFS cluster ``` > ceph -s cluster: id: e78987f2-ef1c-11ed-897d-cf8c255417f0 health: HEALTH_WARN 85 pgs not deep-scrubbed in time 85 pgs not scrubbed in time services: mon: 5 daemons, quorum datastone05,datastone06,datastone07,datastone10,datastone09 (age 2w) mgr: datastone05.iitngk(active, since 2w), standbys: datastone06.wjppdy mds: 2/2 daemons up, 1 hot standby osd: 22 osds: 22 up (since 3d), 22 in (since 4w); 8 remapped pgs data: volumes: 1/1 healthy pools: 4 pools, 115 pgs objects: 49.08M objects, 16 TiB usage: 35 TiB used, 2.0 PiB / 2.1 PiB avail pgs: 3807933/98160678 objects misplaced (3.879%) 107 active+clean 8 active+remapped+backfilling io: client: 224 MiB/s rd, 79 MiB/s wr, 844 op/s rd, 33 op/s wr recovery: 8.8 MiB/s, 24 objects/s ``` The pool and pg status ``` > ceph osd pool autoscale-status POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO EFFECTIVE RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE BULK cephfs.myfs.meta 28802M 2.0 2119T 0.0000 4.0 16 on False cephfs.myfs.data 16743G 2.0 2119T 0.0154 1.0 32 on False rbd 19 2.0 2119T 0.0000 1.0 32 on False .mgr 3840k 2.0 2119T 0.0000 1.0 1 on False ``` The pool detail ``` > ceph osd pool ls detail pool 1 'cephfs.myfs.meta' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 16 pgp_num 16 autoscale_mode on last_change 3639 lfor 0/3639/3637 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs pool 2 'cephfs.myfs.data' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 66 pgp_num 58 pg_num_target 32 pgp_num_target 32 autoscale_mode on last_change 5670 lfor 0/5661/5659 flags hashpspool,selfmanaged_snaps stripe_width 0 application cephfs pool 3 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 486 lfor 0/486/478 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd pool 4 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 39 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr ``` When pg numbers reduce, the mds server would have a chance to hang.

2 1

ceph commands on host cannot connect to cluster after cephx disabling
by service.plant＠ya.ru 24 Feb '24

24 Feb '24

Hello everybody, Suddenly faced with a problem with (probably) authorization playing with cephx. So, long story short: 1) Rollout completely new testing cluster by cephadm with only one node 2) According to docs I've set this to /etc/ceph/ceph.conf auth_cluster_required = none auth_service_required = none auth_client_required = none 3) restart ceph.target 4) now even "ceph -s " cannot connect to RADOS saying root@ceph1:/etc/ceph# ceph -s 2024-02-24T18:15:59.219+0000 7f7c10d65700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [1] 2024-02-24T18:15:59.219+0000 7f7c11d67700 0 librados: client.admin authentication error (13) Permission denied [errno 13] RADOS permission denied (error connecting to the cluster) 4) I have ceph.client.admin.keyring in both /etc/ceph and /var/lib/ceph/$fsid/config 5) logs of monitor doesnt show any error. It looks like it keeps normal living and even doesn't know that something goes wrong 6) Tried to set back /etc/ceph/ceph.conf to auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx with no success 7) I have noted that some process (I guess it is one of processes in containers?) always rewrite /etc/ceph/ceph.cong and /var/lib/ceph/$fsid/config/ceph.conf whatever I woud write there. What is the process? Who is it? How to set up settings if I want to keep it in the file?? Ubuntu 20.04, Reef, 18.0.2 Thanks in advance.

1 0

Scrubs Randomly Starting/Stopping
by ashley＠amerrick.co.uk 24 Feb '24

24 Feb '24

Have just upgraded a cluster from 17.2.7 to 18.2.1 Everything is working as expected apart from the amount of scrubs & deep scrubs is bouncing all over the place every second. I have the value set to 1 per OSD but currently the cluster reckons one minute it’s doing 60+ scrubs, and then second this will drop to 40, then back to 70. If I check the ceph live log’s I can see every second it’s reporting multiple PG’s starting either a scrub or deep scrub, it does not look like these are actually running as isn’t having a negative effect on the cluster’s performance. Is this something to be expected off the back of the upgrade and should sort it self out? A sample of the logs: 2024-02-24T00:41:20.055401+0000 osd.54 (osd.54) 3160 : cluster 0 12.9a deep-scrub starts 2024-02-24T00:41:19.658144+0000 osd.41 (osd.41) 4103 : cluster 0 12.cd deep-scrub starts 2024-02-24T00:41:19.823910+0000 osd.33 (osd.33) 5625 : cluster 0 12.ae deep-scrub starts 2024-02-24T00:41:19.846736+0000 osd.65 (osd.65) 3947 : cluster 0 12.53 deep-scrub starts 2024-02-24T00:41:20.007331+0000 osd.20 (osd.20) 7214 : cluster 0 12.142 scrub starts 2024-02-24T00:41:20.114748+0000 osd.10 (osd.10) 6538 : cluster 0 12.2c deep-scrub starts 2024-02-24T00:41:20.247205+0000 osd.36 (osd.36) 4789 : cluster 0 12.16f deep-scrub starts 2024-02-24T00:41:20.908051+0000 osd.68 (osd.68) 3869 : cluster 0 12.d7 deep-scrub starts

2 1

Size return by df
by Albert Shih 24 Feb '24

24 Feb '24

Hi, I got one cephfs with one volume and subvolumes with a erasure coding. If I don't set any quota when I run df on the client I got 0ccbc438-d109-4c5f-b47b-70f8df707c2c/vo 5,8P 78T 5,8P 2% /vo The 78T seem to be the size use by ceph on disk (on the hardware I mean). And I find that very good. But If I set a quota setfattr -n ceph.quota.max_bytes -v 109951162777600 vo then on the same client I got 0ccbc438-d109-4c5f-b47b-70f8df707c2c/vo 100T 51T 50T 51% /vo and that are the size of the data (I using erasure 4/2 so 51*1.5 ~ 77 To) Is they are any way to keep the first answer ? Regards -- Albert SHIH 🦫 🐸 France Heure locale/Local time: jeu. 22 févr. 2024 08:44:17 CET

2 2

PG stuck at recovery
by LeonGao 23 Feb '24

23 Feb '24

Hi community We have a new Ceph cluster deployment with 100 nodes. When we are draining an OSD host from the cluster, we see a small amount of PGs that cannot make any progress to the end. From the logs and metrics, it seems like the recovery progress is stuck (0 recovery ops for several days). Would like to get some ideas on this. Re-peering and OSD restart do resolve to mitigate the issue but we want to get to the root cause of it as draining and recovery happen frequently. I have put some debugging information below. Any help is appreciated, thanks! ceph -s pgs: 4210926/7380034104 objects misplaced (0.057%) 41198 active+clean 71 active+remapped+backfilling 12 active+recovering One of the stuck PG: 6.38f1 active+remapped+backfilling [313,643,727] 313 [313,643,717] 313 PG query result: ceph pg 6.38f1 query { "snap_trimq": "[]", "snap_trimq_len": 0, "state": "active+remapped+backfilling", "epoch": 246856, "up": [ 313, 643, 727 ], "acting": [ 313, 643, 717 ], "backfill_targets": [ "727" ], "acting_recovery_backfill": [ "313", "643", "717", "727" ], "info": { "pgid": "6.38f1", "last_update": "212333'38916", "last_complete": "212333'38916", "log_tail": "80608'37589", "last_user_version": 38833, "last_backfill": "MAX", "purged_snaps": [], "history": { "epoch_created": 3726, "epoch_pool_created": 3279, "last_epoch_started": 243987, "last_interval_started": 243986, "last_epoch_clean": 220174, "last_interval_clean": 220173, "last_epoch_split": 3726, "last_epoch_marked_full": 0, "same_up_since": 238347, "same_interval_since": 243986, "same_primary_since": 3728, "last_scrub": "212333'38916", "last_scrub_stamp": "2024-01-29T13:43:10.654709+0000", "last_deep_scrub": "212333'38916", "last_deep_scrub_stamp": "2024-01-28T07:43:45.920198+0000", "last_clean_scrub_stamp": "2024-01-29T13:43:10.654709+0000", "prior_readable_until_ub": 0 }, "stats": { "version": "212333'38916", "reported_seq": 413425, "reported_epoch": 246856, "state": "active+remapped+backfilling", "last_fresh": "2024-02-05T21:14:40.838785+0000", "last_change": "2024-02-03T22:33:43.052272+0000", "last_active": "2024-02-05T21:14:40.838785+0000", "last_peered": "2024-02-05T21:14:40.838785+0000", "last_clean": "2024-02-03T04:26:35.168232+0000", "last_became_active": "2024-02-03T22:31:16.037823+0000", "last_became_peered": "2024-02-03T22:31:16.037823+0000", "last_unstale": "2024-02-05T21:14:40.838785+0000", "last_undegraded": "2024-02-05T21:14:40.838785+0000", "last_fullsized": "2024-02-05T21:14:40.838785+0000", "mapping_epoch": 243986, "log_start": "80608'37589", "ondisk_log_start": "80608'37589", "created": 3726, "last_epoch_clean": 220174, "parent": "0.0", "parent_split_bits": 14, "last_scrub": "212333'38916", "last_scrub_stamp": "2024-01-29T13:43:10.654709+0000", "last_deep_scrub": "212333'38916", "last_deep_scrub_stamp": "2024-01-28T07:43:45.920198+0000", "last_clean_scrub_stamp": "2024-01-29T13:43:10.654709+0000", "objects_scrubbed": 17743, "log_size": 1327, "log_dups_size": 3000, "ondisk_log_size": 1327, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "last_scrub_duration": 29, "scrub_schedule": "queued for deep scrub", "scrub_duration": 28.699844475999999, "objects_trimmed": 0, "snaptrim_duration": 0, "stat_sum": { "num_bytes": 67167022618, "num_objects": 17743, "num_object_clones": 0, "num_object_copies": 53229, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 12401, "num_objects_unfound": 0, "num_objects_dirty": 17743, "num_whiteouts": 0, "num_read": 16640, "num_read_kb": 30923655, "num_write": 55094, "num_write_kb": 71824558, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 40746, "num_bytes_recovered": 154245233300, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 313, 643, 727 ], "acting": [ 313, 643, 717 ], "avail_no_missing": [ "313", "643", "717" ], "object_location_counts": [ { "shards": "313,643,717", "objects": 17743 } ], "blocked_by": [], "up_primary": 313, "acting_primary": 313, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 243987, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, "peer_info": [ { "peer": "643", "pgid": "6.38f1", "last_update": "212333'38916", "last_complete": "212333'38916", "log_tail": "80608'37589", "last_user_version": 38833, "last_backfill": "MAX", "purged_snaps": [], "history": { "epoch_created": 3726, "epoch_pool_created": 3279, "last_epoch_started": 243987, "last_interval_started": 243986, "last_epoch_clean": 220174, "last_interval_clean": 220173, "last_epoch_split": 3726, "last_epoch_marked_full": 0, "same_up_since": 238347, "same_interval_since": 243986, "same_primary_since": 3728, "last_scrub": "212333'38916", "last_scrub_stamp": "2024-01-29T13:43:10.654709+0000", "last_deep_scrub": "212333'38916", "last_deep_scrub_stamp": "2024-01-28T07:43:45.920198+0000", "last_clean_scrub_stamp": "2024-01-29T13:43:10.654709+0000", "prior_readable_until_ub": 0 }, "stats": { "version": "212331'38915", "reported_seq": 362849, "reported_epoch": 212333, "state": "active+clean+scrubbing+deep", "last_fresh": "2024-01-28T07:43:40.515729+0000", "last_change": "2024-01-28T06:20:14.969313+0000", "last_active": "2024-01-28T07:43:40.515729+0000", "last_peered": "2024-01-28T07:43:40.515729+0000", "last_clean": "2024-01-28T07:43:40.515729+0000", "last_became_active": "2024-01-23T10:01:07.710478+0000", "last_became_peered": "2024-01-23T10:01:07.710478+0000", "last_unstale": "2024-01-28T07:43:40.515729+0000", "last_undegraded": "2024-01-28T07:43:40.515729+0000", "last_fullsized": "2024-01-28T07:43:40.515729+0000", "mapping_epoch": 243986, "log_start": "80608'37589", "ondisk_log_start": "80608'37589", "created": 3726, "last_epoch_clean": 123203, "parent": "0.0", "parent_split_bits": 14, "last_scrub": "80609'38697", "last_scrub_stamp": "2024-01-09T20:27:40.795531+0000", "last_deep_scrub": "80609'38697", "last_deep_scrub_stamp": "2024-01-07T08:20:59.246940+0000", "last_clean_scrub_stamp": "2024-01-09T20:27:40.795531+0000", "objects_scrubbed": 17724, "log_size": 1326, "log_dups_size": 3000, "ondisk_log_size": 1326, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "last_scrub_duration": 29, "scrub_schedule": "deep scrubbing for 5005s", "scrub_duration": 28.446739284, "objects_trimmed": 0, "snaptrim_duration": 0, "stat_sum": { "num_bytes": 67167022618, "num_objects": 17743, "num_object_clones": 0, "num_object_copies": 53229, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 17743, "num_whiteouts": 0, "num_read": 16640, "num_read_kb": 30923655, "num_write": 55094, "num_write_kb": 71824558, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 35403, "num_bytes_recovered": 134021819371, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 313, 643, 727 ], "acting": [ 313, 643, 717 ], "avail_no_missing": [], "object_location_counts": [], "blocked_by": [], "up_primary": 313, "acting_primary": 313, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 243987, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "717", "pgid": "6.38f1", "last_update": "212333'38916", "last_complete": "212333'38916", "log_tail": "80608'37589", "last_user_version": 0, "last_backfill": "MAX", "purged_snaps": [], "history": { "epoch_created": 3726, "epoch_pool_created": 3279, "last_epoch_started": 243987, "last_interval_started": 243986, "last_epoch_clean": 220174, "last_interval_clean": 220173, "last_epoch_split": 3726, "last_epoch_marked_full": 0, "same_up_since": 238347, "same_interval_since": 243986, "same_primary_since": 3728, "last_scrub": "212333'38916", "last_scrub_stamp": "2024-01-29T13:43:10.654709+0000", "last_deep_scrub": "212333'38916", "last_deep_scrub_stamp": "2024-01-28T07:43:45.920198+0000", "last_clean_scrub_stamp": "2024-01-29T13:43:10.654709+0000", "prior_readable_until_ub": 0 }, "stats": { "version": "212331'38915", "reported_seq": 362849, "reported_epoch": 212333, "state": "active+clean+scrubbing+deep", "last_fresh": "2024-01-28T07:43:40.515729+0000", "last_change": "2024-01-28T06:20:14.969313+0000", "last_active": "2024-01-28T07:43:40.515729+0000", "last_peered": "2024-01-28T07:43:40.515729+0000", "last_clean": "2024-01-28T07:43:40.515729+0000", "last_became_active": "2024-01-23T10:01:07.710478+0000", "last_became_peered": "2024-01-23T10:01:07.710478+0000", "last_unstale": "2024-01-28T07:43:40.515729+0000", "last_undegraded": "2024-01-28T07:43:40.515729+0000", "last_fullsized": "2024-01-28T07:43:40.515729+0000", "mapping_epoch": 243986, "log_start": "80608'37589", "ondisk_log_start": "80608'37589", "created": 3726, "last_epoch_clean": 123203, "parent": "0.0", "parent_split_bits": 14, "last_scrub": "80609'38697", "last_scrub_stamp": "2024-01-09T20:27:40.795531+0000", "last_deep_scrub": "80609'38697", "last_deep_scrub_stamp": "2024-01-07T08:20:59.246940+0000", "last_clean_scrub_stamp": "2024-01-09T20:27:40.795531+0000", "objects_scrubbed": 17724, "log_size": 1326, "log_dups_size": 3000, "ondisk_log_size": 1326, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "last_scrub_duration": 29, "scrub_schedule": "deep scrubbing for 5005s", "scrub_duration": 28.446739284, "objects_trimmed": 0, "snaptrim_duration": 0, "stat_sum": { "num_bytes": 67167022618, "num_objects": 17743, "num_object_clones": 0, "num_object_copies": 53229, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 17743, "num_whiteouts": 0, "num_read": 16640, "num_read_kb": 30923655, "num_write": 55094, "num_write_kb": 71824558, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 35403, "num_bytes_recovered": 134021819371, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 313, 643, 727 ], "acting": [ 313, 643, 717 ], "avail_no_missing": [], "object_location_counts": [], "blocked_by": [], "up_primary": 313, "acting_primary": 313, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 243987, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "726", "pgid": "6.38f1", "last_update": "212333'38916", "last_complete": "212333'38916", "log_tail": "80608'37589", "last_user_version": 0, "last_backfill": "MIN", "purged_snaps": [], "history": { "epoch_created": 3726, "epoch_pool_created": 3279, "last_epoch_started": 238348, "last_interval_started": 238347, "last_epoch_clean": 220174, "last_interval_clean": 220173, "last_epoch_split": 3726, "last_epoch_marked_full": 0, "same_up_since": 238347, "same_interval_since": 243986, "same_primary_since": 3728, "last_scrub": "212333'38916", "last_scrub_stamp": "2024-01-29T13:43:10.654709+0000", "last_deep_scrub": "212333'38916", "last_deep_scrub_stamp": "2024-01-28T07:43:45.920198+0000", "last_clean_scrub_stamp": "2024-01-29T13:43:10.654709+0000", "prior_readable_until_ub": 11.834917336 }, "stats": { "version": "0'0", "reported_seq": 0, "reported_epoch": 0, "state": "unknown", "last_fresh": "0.000000", "last_change": "0.000000", "last_active": "0.000000", "last_peered": "0.000000", "last_clean": "0.000000", "last_became_active": "0.000000", "last_became_peered": "0.000000", "last_unstale": "0.000000", "last_undegraded": "0.000000", "last_fullsized": "0.000000", "mapping_epoch": 243986, "log_start": "0'0", "ondisk_log_start": "0'0", "created": 0, "last_epoch_clean": 0, "parent": "0.0", "parent_split_bits": 0, "last_scrub": "0'0", "last_scrub_stamp": "0.000000", "last_deep_scrub": "0'0", "last_deep_scrub_stamp": "0.000000", "last_clean_scrub_stamp": "0.000000", "objects_scrubbed": 0, "log_size": 0, "log_dups_size": 0, "ondisk_log_size": 0, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "last_scrub_duration": 0, "scrub_schedule": "--", "scrub_duration": 0, "objects_trimmed": 0, "snaptrim_duration": 0, "stat_sum": { "num_bytes": 4194304, "num_objects": 0, "num_object_clones": 0, "num_object_copies": 0, "num_objects_missing_on_primary": 0, "num_objects_missing": 17743, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 0, "num_whiteouts": 0, "num_read": 0, "num_read_kb": 0, "num_write": 0, "num_write_kb": 0, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 313, 643, 727 ], "acting": [ 313, 643, 717 ], "avail_no_missing": [], "object_location_counts": [], "blocked_by": [], "up_primary": 313, "acting_primary": 313, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 1, "last_epoch_started": 238320, "hit_set_history": { "current_last_update": "0'0", "history": [] } }, { "peer": "727", "pgid": "6.38f1", "last_update": "212333'38916", "last_complete": "212333'38916", "log_tail": "80608'37589", "last_user_version": 0, "last_backfill": "6:8f1d3293:::d0950b5f-c248-4a0e-8fed-6ebbd0bb2820.970076.1__shadow_.V-JlvDFjbQNgn8GtBLN2LePecv5fLKf_3:head", "purged_snaps": [], "history": { "epoch_created": 3726, "epoch_pool_created": 3279, "last_epoch_started": 243987, "last_interval_started": 243986, "last_epoch_clean": 220174, "last_interval_clean": 220173, "last_epoch_split": 3726, "last_epoch_marked_full": 0, "same_up_since": 238347, "same_interval_since": 243986, "same_primary_since": 3728, "last_scrub": "212333'38916", "last_scrub_stamp": "2024-01-29T13:43:10.654709+0000", "last_deep_scrub": "212333'38916", "last_deep_scrub_stamp": "2024-01-28T07:43:45.920198+0000", "last_clean_scrub_stamp": "2024-01-29T13:43:10.654709+0000", "prior_readable_until_ub": 0 }, "stats": { "version": "0'0", "reported_seq": 0, "reported_epoch": 0, "state": "unknown", "last_fresh": "0.000000", "last_change": "0.000000", "last_active": "0.000000", "last_peered": "0.000000", "last_clean": "0.000000", "last_became_active": "0.000000", "last_became_peered": "0.000000", "last_unstale": "0.000000", "last_undegraded": "0.000000", "last_fullsized": "0.000000", "mapping_epoch": 243986, "log_start": "0'0", "ondisk_log_start": "0'0", "created": 0, "last_epoch_clean": 0, "parent": "0.0", "parent_split_bits": 0, "last_scrub": "0'0", "last_scrub_stamp": "0.000000", "last_deep_scrub": "0'0", "last_deep_scrub_stamp": "0.000000", "last_clean_scrub_stamp": "0.000000", "objects_scrubbed": 0, "log_size": 0, "log_dups_size": 0, "ondisk_log_size": 0, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "last_scrub_duration": 0, "scrub_schedule": "--", "scrub_duration": 0, "objects_trimmed": 0, "snaptrim_duration": 0, "stat_sum": { "num_bytes": 20227608233, "num_objects": 5342, "num_object_clones": 0, "num_object_copies": 0, "num_objects_missing_on_primary": 0, "num_objects_missing": 12401, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 5342, "num_whiteouts": 0, "num_read": 0, "num_read_kb": 0, "num_write": 0, "num_write_kb": 0, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 313, 643, 727 ], "acting": [ 313, 643, 717 ], "avail_no_missing": [], "object_location_counts": [], "blocked_by": [], "up_primary": 313, "acting_primary": 313, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 1, "last_epoch_started": 243987, "hit_set_history": { "current_last_update": "0'0", "history": [] } } ], "recovery_state": [ { "name": "Started/Primary/Active", "enter_time": "2024-02-03T22:31:15.023323+0000", "might_have_unfound": [], "recovery_progress": { "backfill_targets": [ "727" ], "waiting_on_backfill": [], "last_backfill_started": "6:8f1d329f:::d0950b5f-c248-4a0e-8fed-6ebbd0bb2820.970076.1_85-22605:head", "backfill_info": { "begin": "6:8f1d32a2:::d0950b5f-c248-4a0e-8fed-6ebbd0bb2820.1157644.1_121-19698:head", "end": "6:8f1d3549:::d0950b5f-c248-4a0e-8fed-6ebbd0bb2820.970076.1__shadow_.PTxEYjlIXk2vkWpRgKUOu4ygweU-Jqy_5:head", "objects": [ { "object": "6:8f1d32a2:::d0950b5f-c248-4a0e-8fed-6ebbd0bb2820.1157644.1_121-19698:head", "version": "80588'26367" }, { "object": "6:8f1d32b0:::d0950b5f-c248-4a0e-8fed-6ebbd0bb2820.970076.1_18-71435:head", "version": "80588'26368" }, { "object": "6:8f1d32c2:::d0950b5f-c248-4a0e-8fed-6ebbd0bb2820.970076.1__shadow_.o_s0NGR7mrH-4JKpvYS6Ip14hbm4w5o_5:head", "version": "80588'26369" }, { "object": "6:8f1d32c3:::d0950b5f-c248-4a0e-8fed-6ebbd0bb2820.970076.1__shadow_.WV1Ja5V21BKfmI5e1W90a7iQXdcTnSG_6:head", "version": "80588'26370" }, { "object": "6:8f1d32c4:::d0950b5f-c248-4a0e-8fed-6ebbd0bb2820.970076.1__shadow_.96tkpnCzKmtnPTLewpai2cXSr_z_QcR_1:head", "version": "80588'26371" }, { "object": "6:8f1d331d:::d0950b5f-c248-4a0e-8fed-6ebbd0bb2820.1157644.1__shadow_.InACTSlwV7irTe0g9cLnadT8EbQr4Cl_5:head", "version": "80588'26372" }, { "object": "6:8f1d3326:::d0950b5f-c248-4a0e-8fed-6ebbd0bb2820.970076.1_233-155987:head", "version": "80588'26373" }, { "object": "6:8f1d3332:::d0950b5f-c248-4a0e-8fed-6ebbd0bb2820.1157644.1__shadow_.5aipEOj0vV3h4jdSf6tkAUn64yJmvoH_1:head", "version": "80588'26374" }, { "object": "6:8f1d3343:::d0950b5f-c248-4a0e-8fed-6ebbd0bb2820.970076.1__shadow_.QTzZJhDV6JLC5wJ5BDxHvK4sfp3Hr6o_6:head", "version": "80588'26375" }, { "object": "6:8f1d334b:::d0950b5f-c248-4a0e-8fed-6ebbd0bb2820.970076.1__shadow_.QgMCmHWV87HlAjqkfvbO9X5OUZrgIPc_2:head", "version": "80588'26376" }, { "object": "6:8f1d3355:::d0950b5f-c248-4a0e-8fed-6ebbd0bb2820.970076.1__shadow_.H4j6DWgQa0jsrl-xzgugsW22GHCOgGL_7:head", "version": "80588'26377" }, { "object": "6:8f1d3379:::d0950b5f-c248-4a0e-8fed-6ebbd0bb2820.970076.1__shadow_.25g7c0w-VnGqX-z_m53YmnoWUO0R56N_5:head", "version": "80588'26378" }, { "object": "6:8f1d337c:::d0950b5f-c248-4a0e-8fed-6ebbd0bb2820.970076.1__shadow_.WMmV4BoLMm4q6BztoZ1uPc4boiJ-zhW_1:head", "version": "80588'26379" }, { "object": "6:8f1d3385:::d0950b5f-c248-4a0e-8fed-6ebbd0bb2820.970076.1_184-108817:head", "version": "80588'26380" }, { "object": "6:8f1d3389:::d0950b5f-c248-4a0e-8fed-6ebbd0bb2820.970076.1__shadow_.sGNiGV9K-TGyTYRqWSOcSIXkBBOT93X_3:head", "version": "80588'26381" }, { "object": "6:8f1d339c:::d0950b5f-c248-4a0e-8fed-6ebbd0bb2820.970076.1__shadow_.NkMm_dhUxRyxXhBDYqGCuOaSvwNvSuJ_3:head", "version": "80588'26382" }, { "object": "6:8f1d33a3:::d0950b5f-c248-4a0e-8fed-6ebbd0bb2820.970076.1__shadow_.BV__z8jIWqcjMsTf40MvaARU1dVtCv2_4:head", "version": "80588'26383" }, { "object": "6:8f1d33a6:::d0950b5f-c248-4a0e-8fed-6ebbd0bb2820.970076.1__shadow_.XJLSyRSnrDupnOjWzHKc4wykPJjqR5g_1:head", "version": "80588'26384" }, { "object": "6:8f1d33a6:::d0950b5f-c248-4a0e-8fed-6ebbd0bb2820.970076.1__shadow_.oYSnMakOIpFmfLdym2oCx-fLBDCLDmr_4:head", "version": "80588'26385" }, { "object": "6:8f1d33b4:::d0950b5f-c248-4a0e-8fed-6ebbd0bb2820.1157644.1_135-2779:head", "version": "80588'26386" }, { "object": "6:8f1d33b6:::d0950b5f-c248-4a0e-8fed-6ebbd0bb2820.970076.1__shadow_.sEHqthyEGg3FQP4-vdL9zkhSJmQEaIc_2:head", "version": "80588'26387" }, { "object": "6:8f1d33ec:::d0950b5f-c248-4a0e-8fed-6ebbd0bb2820.970076.1__shadow_.wL36lNEfHGE6B4V-YT6fMuAI-n2uV30_1:head", "version": "80588'26388" }, { "object": "6:8f1d33f3:::d0950b5f-c248-4a0e-8fed-6ebbd0bb2820.970076.1__shadow_.j1UloV7DpRwG0-4AKULi3CXOTC0CrdU_4:head", "version": "80588'26389" }, { "object": "6:8f1d33f5:::d0950b5f-c248-4a0e-8fed-6ebbd0bb2820.970076.1__shadow_.q3ExUB7Mp-LggBIsRBBKyIob574rNpG_3:head", "version": "80588'26390" }, { "object": "6:8f1d33f9:::d0950b5f-c248-4a0e-8fed-6ebbd0bb2820.970076.1_72-13403:head", "version": "80588'26391" }, { "object": "6:8f1d33fd:::d0950b5f-c248-4a0e-8fed-6ebbd0bb2820.970076.1_157-172390:head", "version": "80588'26392" }, { "object": "6:8f1d3403:::d0950b5f-c248-4a0e-8fed-6ebbd0bb2820.970076.1__shadow_.Wn4ed97PvQp_oDnk_Y4xlZMke0pQ4z1_3:head", "version": "80588'26393" }, { "object": "6:8f1d3405:::d0950b5f-c248-4a0e-8fed-6ebbd0bb2820.970076.1__shadow_.uFg6eFSD3BCEqzJLCZlw546nDqqg80U_4:head", "version": "80588'26394" }, { "object": "6:8f1d3436:::d0950b5f-c248-4a0e-8fed-6ebbd0bb2820.970076.1__shadow_.xPwN1sJsNFAg2Nr_Vnu4ItFgHFyu_Nx_7:head", "version": "80588'26395" }, { "object": "6:8f1d3438:::d0950b5f-c248-4a0e-8fed-6ebbd0bb2820.970076.1__shadow_.Qgd9EXWAraBWnYcZ3OhFvepsT8r0Ksi_1:head", "version": "80588'26396" }, { "object": "6:8f1d3464:::d0950b5f-c248-4a0e-8fed-6ebbd0bb2820.970076.1__shadow_.1KnQBzx5u-CkXROQVUzUPfJiuTq52zH_3:head", "version": "80588'26397" }, { "object": "6:8f1d3471:::d0950b5f-c248-4a0e-8fed-6ebbd0bb2820.970076.1__shadow_.Aw2QRjM0ZXkf20MtQlhyq31hwVMEWwe_1:head", "version": "80588'26398" }, { "object": "6:8f1d3471:::d0950b5f-c248-4a0e-8fed-6ebbd0bb2820.970076.1__shadow_.xprsKnjtYu1ITOsgDtkPup9Nk6xzGTN_6:head", "version": "80588'26399" }, { "object": "6:8f1d3475:::d0950b5f-c248-4a0e-8fed-6ebbd0bb2820.1157644.1__shadow_.jo4-jEQV1kB9kHewUYdbU1UUNX60EUm_1:head", "version": "80588'26400" }, { "object": "6:8f1d3488:::d0950b5f-c248-4a0e-8fed-6ebbd0bb2820.970076.1__shadow_.f9EE8lnu2FzW1_vGo9J41KY-iFpI9-f_1:head", "version": "80588'26401" }, { "object": "6:8f1d3491:::d0950b5f-c248-4a0e-8fed-6ebbd0bb2820.970076.1_11-138415:head", "version": "80588'26402" }, { "object": "6:8f1d34ac:::d0950b5f-c248-4a0e-8fed-6ebbd0bb2820.970076.1_33-123891:head", "version": "80588'26403" }, { "object": "6:8f1d34ad:::d0950b5f-c248-4a0e-8fed-6ebbd0bb2820.970076.1__shadow_.x3tIcVCAeRnOJmmHhJ2hxCjsUZ3GBsK_3:head", "version": "80588'26404" }, { "object": "6:8f1d34bc:::d0950b5f-c248-4a0e-8fed-6ebbd0bb2820.1157644.1__shadow_.9yQELiJhCNs5Z-foJHH3vEfQ_1TZdBF_1:head", "version": "80588'26405" }, { "object": "6:8f1d34d1:::d0950b5f-c248-4a0e-8fed-6ebbd0bb2820.970076.1__shadow_.Me9Xl2_j9z2_yE1laHWpHdJ2fXvwZDZ_1:head", "version": "80588'26406" }, { "object": "6:8f1d34fa:::d0950b5f-c248-4a0e-8fed-6ebbd0bb2820.970076.1__shadow_.Y5VD6lU4UqjXEh5Yb9GVwusKpsqGzuG_4:head", "version": "80588'26407" }, { "object": "6:8f1d3503:::d0950b5f-c248-4a0e-8fed-6ebbd0bb2820.970076.1_58-178651:head", "version": "80588'26408" }, { "object": "6:8f1d3521:::d0950b5f-c248-4a0e-8fed-6ebbd0bb2820.970076.1__shadow_.FFyoywHasB0i44Y9ZtCi_YgtYhm8cP6_6:head", "version": "80588'26409" }, { "object": "6:8f1d3521:::d0950b5f-c248-4a0e-8fed-6ebbd0bb2820.970076.1__shadow_.lHmY-Z-Q3Ewx7kHwCoLFoFU70Xtrgzu_1:head", "version": "80588'26410" }, { "object": "6:8f1d3522:::d0950b5f-c248-4a0e-8fed-6ebbd0bb2820.970076.1__shadow_.ETPvUyLCwz5Bfmh22s_odAwEIMCC_Vn_2:head", "version": "80588'26411" }, { "object": "6:8f1d3522:::d0950b5f-c248-4a0e-8fed-6ebbd0bb2820.970076.1__shadow_.EdYbzDvA-7HpfrM0DMfHRgalz8ZiLhJ_1:head", "version": "80588'26412" }, { "object": "6:8f1d3529:::d0950b5f-c248-4a0e-8fed-6ebbd0bb2820.1186045.1_164-1500436:head", "version": "80588'26413" } ] }, "peer_backfill_info": [ "727", { "begin": "MAX", "end": "MAX", "objects": [] } ], "backfills_in_flight": [ "6:8f1d329f:::d0950b5f-c248-4a0e-8fed-6ebbd0bb2820.970076.1_85-22605:head" ], "recovering": [ "6:8f1d329f:::d0950b5f-c248-4a0e-8fed-6ebbd0bb2820.970076.1_85-22605:head" ], "pg_backend": { "pull_from_peer": [], "pushing": [ { "pushing": "6:8f1d329f:::d0950b5f-c248-4a0e-8fed-6ebbd0bb2820.970076.1_85-22605:head", "pushing_to": [ { "pushing_to": "727", "push_info": { "recovery_progress": { "first?": 0, "data_complete?": 1, "data_recovered_to": 4194304, "omap_complete?": 1, "omap_recovered_to": "" }, "recovery_info": { "object": "6:8f1d329f:::d0950b5f-c248-4a0e-8fed-6ebbd0bb2820.970076.1_85-22605:head", "at_version": "80588'26366", "size": "4194304", "object_info": { "oid": { "oid": "d0950b5f-c248-4a0e-8fed-6ebbd0bb2820.970076.1_85-22605", "key": "", "snapid": -2, "hash": 4182554865, "max": 0, "pool": 6, "namespace": "" }, "version": "80588'26366", "prior_version": "6896'10987", "last_reqid": "osd.313.0:4380013", "user_version": 10987, "size": 4194304, "mtime": "2023-12-19T07:17:04.766167+0000", "local_mtime": "2023-12-19T07:17:04.799712+0000", "lost": 0, "flags": [ "dirty", "data_digest", "omap_digest" ], "truncate_seq": 0, "truncate_size": 0, "data_digest": "0x9cdfa22f", "omap_digest": "0xffffffff", "expected_object_size": 0, "expected_write_size": 0, "alloc_hint_flags": 0, "manifest": { "type": 0 }, "watchers": {} }, "snapset": { "seq": 0, "clones": [] }, "copy_subset": "[0~4194304]", "clone_subset": "{}", "object_exist": "0" } } } ] } ] } } }, { "name": "Started", "enter_time": "2024-02-03T22:31:13.882074+0000" } ], "scrubber": { "active": false, "must_scrub": false, "must_deep_scrub": false, "must_repair": false, "need_auto": false, "scrub_reg_stamp": "2024-01-30T15:39:19.044093+0000", "schedule": "queued for deep scrub" }, "agent_state": {} }

5 7

list topic shows endpoint url and username e password
by Giada Malatesta 23 Feb '24

23 Feb '24

Hello everyone, we are facing a problem regarding the topic operations to send notification, particularly when using amqp protocol. We are using Ceph version 18.2.1. We have created a topic by giving as attributes all needed information and so the push-endpoint (in our case a rabbit endpoint that is used to collect notification messages). Then we have configured all the buckets in our cluster Ceph so that it is possible to send notification when some changes occur. The problem regards particularly the list_topic operation: we noticed that any authenticated user is able to get a full list of the created topics and with them to get all the information, including endpoint, and so username and password and IP and port, when using the boto3.set_stream_logger(), which is not good for our goal since we do not want the users to know implementation details. There is the possibility to solve this problem? Any help would be useful. Thanks and best regards. GM.

2 1

MDS in ReadOnly and 2 MDS behind on trimming
by Edouard FAZENDA 23 Feb '24

23 Feb '24

Dear Ceph Community, I am having an issue with my Ceph Cluster , there were several osd crashing but now active and recovery finished and now the CephFS filesystem cannot be access by clients in RW (K8S worklod) as the 1 MDS is in Read-Only and 2 are being on trimming The cephfs seems to have volume OK The trimming process seems not going further, maybe stuck ? We are running 3 hosts using ceph Pacific version 16.2.1 Here some logs on the situation : ceph versions { "mon": { "ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable)": 3 }, "mgr": { "ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable)": 3 }, "osd": { "ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable)": 18 }, "mds": { "ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable)": 3 }, "rgw": { "ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable)": 6 }, "overall": { "ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable)": 33 } } ceph orch ps NAME HOST STATUS REFRESHED AGE PORTS VERSION IMAGE ID CONTAINER ID crash.rke-sh1-1 rke-sh1-1 running (21h) 36s ago 21h - 16.2.1 c757e4a3636b e8652edb2b49 crash.rke-sh1-2 rke-sh1-2 running (21h) 3m ago 20M - 16.2.1 c757e4a3636b a1249a605ee0 crash.rke-sh1-3 rke-sh1-3 running (17h) 36s ago 17h - 16.2.1 c757e4a3636b 026667bc1776 mds.cephfs.rke-sh1-1.ojmpnk rke-sh1-1 running (18h) 36s ago 4M - 16.2.1 c757e4a3636b 9b4c2b08b759 mds.cephfs.rke-sh1-2.isqjza rke-sh1-2 running (18h) 3m ago 23M - 16.2.1 c757e4a3636b 71681a5f34d3 mds.cephfs.rke-sh1-3.vdicdn rke-sh1-3 running (17h) 36s ago 3M - 16.2.1 c757e4a3636b e89946ad6b7e mgr.rke-sh1-1.qskoyj rke-sh1-1 running (21h) 36s ago 2y *:8082 *:9283 16.2.1 c757e4a3636b 7ce7cfbb3e55 mgr.rke-sh1-2.lxmguj rke-sh1-2 running (21h) 3m ago 22M *:8082 *:9283 16.2.1 c757e4a3636b 5a0025adfd46 mgr.rke-sh1-3.ckunvo rke-sh1-3 running (17h) 36s ago 6M *:8082 *:9283 16.2.1 c757e4a3636b 2fcaf18f3218 mon.rke-sh1-1 rke-sh1-1 running (20h) 36s ago 20h - 16.2.1 c757e4a3636b c0a90103cabc mon.rke-sh1-2 rke-sh1-2 running (21h) 3m ago 3M - 16.2.1 c757e4a3636b f4b32ba4466b mon.rke-sh1-3 rke-sh1-3 running (17h) 36s ago 17h - 16.2.1 c757e4a3636b d5e44c245998 osd.0 rke-sh1-2 running (20h) 3m ago 2y - 16.2.1 c757e4a3636b 7b0e69942c15 osd.1 rke-sh1-3 running (17h) 36s ago 2y - 16.2.1 c757e4a3636b 4451654d9a2d osd.10 rke-sh1-3 running (17h) 36s ago 2y - 16.2.1 c757e4a3636b 3f9d5f95e284 osd.11 rke-sh1-1 running (21h) 36s ago 2y - 16.2.1 c757e4a3636b db1cc6d2e37f osd.12 rke-sh1-2 running (21h) 3m ago 2y - 16.2.1 c757e4a3636b de416c1ef766 osd.13 rke-sh1-3 running (17h) 36s ago 2y - 16.2.1 c757e4a3636b 25a281cc5a9b osd.14 rke-sh1-1 running (21h) 36s ago 2y - 16.2.1 c757e4a3636b 62f25ba61667 osd.15 rke-sh1-2 running (21h) 3m ago 2y - 16.2.1 c757e4a3636b d3514d823c45 osd.16 rke-sh1-3 running (17h) 36s ago 2y - 16.2.1 c757e4a3636b bba857759bfe osd.17 rke-sh1-1 running (21h) 36s ago 2y - 16.2.1 c757e4a3636b 59281d4bb3d0 osd.2 rke-sh1-1 running (21h) 36s ago 2y - 16.2.1 c757e4a3636b 418041b5e60d osd.3 rke-sh1-2 running (21h) 3m ago 2y - 16.2.1 c757e4a3636b 04a0e29d5623 osd.4 rke-sh1-1 running (20h) 36s ago 2y - 16.2.1 c757e4a3636b 1cc78a5153d3 osd.5 rke-sh1-3 running (17h) 36s ago 2y - 16.2.1 c757e4a3636b 39a4b11e31fb osd.6 rke-sh1-2 running (21h) 3m ago 2y - 16.2.1 c757e4a3636b 2f218ffb566e osd.7 rke-sh1-1 running (20h) 36s ago 2y - 16.2.1 c757e4a3636b cf761fbe4d5f osd.8 rke-sh1-3 running (17h) 36s ago 2y - 16.2.1 c757e4a3636b f9f85480e800 osd.9 rke-sh1-2 running (21h) 3m ago 2y - 16.2.1 c757e4a3636b 664c54ff46d2 rgw.default.rke-sh1-1.dgucwl rke-sh1-1 running (21h) 36s ago 22M *:8000 16.2.1 c757e4a3636b f03212b955a7 rgw.default.rke-sh1-1.vylchc rke-sh1-1 running (21h) 36s ago 22M *:8001 16.2.1 c757e4a3636b da486ce43fe5 rgw.default.rke-sh1-2.dfhhfw rke-sh1-2 running (21h) 3m ago 2y *:8000 16.2.1 c757e4a3636b ef4089d0aef2 rgw.default.rke-sh1-2.efkbum rke-sh1-2 running (21h) 3m ago 2y *:8001 16.2.1 c757e4a3636b 9e053d5a2f7b rgw.default.rke-sh1-3.krfgey rke-sh1-3 running (17h) 36s ago 9M *:8001 16.2.1 c757e4a3636b 45cd3d75edd3 rgw.default.rke-sh1-3.pwdbmp rke-sh1-3 running (17h) 36s ago 9M *:8000 16.2.1 c757e4a3636b e2710265a7f4 ceph health detail HEALTH_WARN 1 MDSs are read only; 2 MDSs behind on trimming [WRN] MDS_READ_ONLY: 1 MDSs are read only mds.cephfs.rke-sh1-2.isqjza(mds.0): MDS in read-only mode [WRN] MDS_TRIM: 2 MDSs behind on trimming mds.cephfs.rke-sh1-2.isqjza(mds.0): Behind on trimming (2149/128) max_segments: 128, num_segments: 2149 mds.cephfs.rke-sh1-1.ojmpnk(mds.0): Behind on trimming (2149/128) max_segments: 128, num_segments: 2149 root@rke-sh1-1:~# ceph fs status cephfs - 27 clients ====== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active cephfs.rke-sh1-2.isqjza Reqs: 8 /s 85.2k 53.2k 1742 101 0-s standby-replay cephfs.rke-sh1-1.ojmpnk Evts: 0 /s 52.2k 20.2k 1737 0 POOL TYPE USED AVAIL cephfs_metadata metadata 1109G 6082G cephfs_data data 8419G 6082G STANDBY MDS cephfs.rke-sh1-3.vdicdn MDS version: ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable) ceph status cluster: id: fcb373ce-7aaa-11eb-984f-e7c6e0038e87 health: HEALTH_WARN 1 MDSs are read only 2 MDSs behind on trimming services: mon: 3 daemons, quorum rke-sh1-2,rke-sh1-1,rke-sh1-3 (age 17h) mgr: rke-sh1-1.qskoyj(active, since 17h), standbys: rke-sh1-2.lxmguj, rke-sh1-3.ckunvo mds: 1/1 daemons up, 1 standby, 1 hot standby osd: 18 osds: 18 up (since 17h), 18 in (since 20h) rgw: 6 daemons active (3 hosts, 1 zones) data: volumes: 1/1 healthy pools: 11 pools, 849 pgs objects: 10.10M objects, 5.3 TiB usage: 11 TiB used, 15 TiB / 26 TiB avail pgs: 849 active+clean io: client: 35 KiB/s rd, 1.0 MiB/s wr, 302 op/s rd, 165 op/s wr # ceph mds stat cephfs:1 {0=cephfs.rke-sh1-2.isqjza=up:active} 1 up:standby-replay 1 up:standby Have you got an idea on what could be my next steps to bring the cluster healthy ? Help will very be appreciated. Thank a lot for your feedback. Best Regards, Edouard FAZENDA Technical Support Chemin du Curé-Desclouds 2, CH-1226 THONEX +41 (0)22 869 04 40 <https://www.csti.ch/> www.csti.ch

2 4

Query Regarding Calculating Ingress/Egress Traffic for Buckets via API
by asad.siddiqui＠rapidcompute.com 23 Feb '24

23 Feb '24

Hi, I am currently working on Ceph object storage and would like to inquire about how we can calculate the ingress and egress traffic for buckets/tenant via API.

2 1

Issue with Setting Public/Private Permissions for Bucket
by asad.siddiqui＠rapidcompute.com 23 Feb '24

23 Feb '24

Hi Team, I'm currently working with Ceph object storage and would like to understand how to set permissions to private or public on buckets/objects in Ceph object storage.

2 1

Re: cephadm purge cluster does not work
by Eugen Block 23 Feb '24

23 Feb '24

It works for me on 17.2.6 as well. Could you be more specific what doesn't work for you? Running that command only removes the cluster configs etc. on that host, it does not orchestrate a removal on all hosts, not sure if you're aware of that. Zitat von Vahideh Alinouri <vahideh.alinouri(a)gmail.com>: > The version that has been installed is 17.2.5. But this method does not > work at all. > > On Fri, Feb 23, 2024, 10:23 AM Eugen Block <eblock(a)nde.ag> wrote: > >> Which ceph version is this? In a small Reef test cluster this works as >> expected: >> >> # cephadm rm-cluster --fsid 2851404a-d09a-11ee-9aaa-fa163e2de51a >> --zap-osds --force >> Using recent ceph image >> >> registry.cloud.hh.nde.ag/ebl/ceph-upstream@sha256:057e08bf8d2d20742173a571bc28b65674b055bebe5f4c6cd488c1a6fd51f685 >> Zapping /dev/sdb... >> Zapping /dev/sdc... >> Zapping /dev/sdd... >> >> and lsblk shows empty drives. >> >> Zitat von Vahideh Alinouri <vahideh.alinouri(a)gmail.com>: >> >> > Hi Guys, >> > >> > I faced an issue. When I wanted to purge, the cluster was not purged >> > using the below command: >> > >> > ceph mgr module disable cephadm >> > cephadm rm-cluster --force --zap-osds --fsid <fsid> >> > >> > The OSDs will remain. There should be some cleanup methods for the >> > whole cluster, not just MON nodes. Is there anything related to this? >> > >> > Regards >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users(a)ceph.io >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>

1 0

cephadm purge cluster does not work
by Vahideh Alinouri 23 Feb '24

23 Feb '24

Hi Guys, I faced an issue. When I wanted to purge, the cluster was not purged using the below command: ceph mgr module disable cephadm cephadm rm-cluster --force --zap-osds --fsid <fsid> The OSDs will remain. There should be some cleanup methods for the whole cluster, not just MON nodes. Is there anything related to this? Regards

2 1

Re-linking subdirectories with root inodes in CephFS
by caskd 23 Feb '24

23 Feb '24

Hello Ceph users, we've had a incident with CephFS recently which resulted in the MDSs crashing for one of the filesystems. First it was a journal corruption which was easily recoverable with no real damage but following that the MDSs fail to start due to a crash related to snapshots. -10> 2024-02-18T20:03:51.656+0000 7f08725bfb38 1 mds.0.234010 handle_mds_map state change up:rejoin --> up:active -9> 2024-02-18T20:03:51.656+0000 7f08725bfb38 1 mds.0.234010 recovery_done -- successful recovery! -8> 2024-02-18T20:03:51.656+0000 7f087271fb38 10 monclient: get_auth_request con 0x7f0871836380 auth_method 0 -7> 2024-02-18T20:03:51.656+0000 7f08726d5b38 10 monclient: get_auth_request con 0x7f08718353c0 auth_method 0 -6> 2024-02-18T20:03:51.656+0000 7f08726fab38 10 monclient: get_auth_request con 0x7f0871895280 auth_method 0 -5> 2024-02-18T20:03:51.656+0000 7f08725bfb38 1 mds.0.234010 active_start -4> 2024-02-18T20:03:51.656+0000 7f08725bfb38 1 mds.0.cache dump_cache to cachedump.234013.mds0 -3> 2024-02-18T20:03:51.736+0000 7f08725bfb38 1 mds.0.234010 cluster recovered. -2> 2024-02-18T20:03:51.736+0000 7f08725bfb38 4 mds.0.234010 set_osd_epoch_barrier: epoch=75012 -1> 2024-02-18T20:03:51.736+0000 7f08722f9b38 -1 /home/buildozer/aports/community/ceph18/src/ceph-18.2.1/src/mds/MDCache.cc: In function 'void MDCache::journal_cow_dentry(MutationImpl*, EMetaBlob*, CDentry*, snapid_t, CInode**, CDentry::linkage_t*)' thread 7f08722f9b38 time 2024-02-18T20:03:51.747600+0000 /home/buildozer/aports/community/ceph18/src/ceph-18.2.1/src/mds/MDCache.cc: 1638: FAILED ceph_assert(follows >= realm->get_newest_seq()) ceph version 18.2.1 (e3fce6809130d78ac0058fc87e537ecd926cd213) reef (stable) 0> 2024-02-18T20:03:51.736+0000 7f08722f9b38 -1 *** Caught signal (Aborted) ** in thread 7f08722f9b38 thread_name:MR_Finisher ceph version 18.2.1 (e3fce6809130d78ac0058fc87e537ecd926cd213) reef (stable) We've sadly attempted to restore the root, which made the entire filesystem tree stray. 1. Is there a way to re-link a directory as the root (or dentry of root) manually? 2. Could snapshots of objects be purged completly manually? Thanks in advance. -- Alex D. RedXen System & Infrastructure Administration https://redxen.eu/

1 3

FS down - mds degraded
by Sake Ceph 23 Feb '24

23 Feb '24

Starting a new thread, forgot subject in the previous. So our FS down. Got the following error, what can I do? # ceph health detail HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged [WRN] FS_DEGRADED: 1 filesystem is degraded fs atlassian/prod is degraded [ERR] MDS_DAMAGE: 1 mds daemon damaged fs atlassian-prod mds.1 is damaged # ceph fs get atlassian-prod Filesystem 'atlassian-prod' (2) fs_name atlassian-prod epoch 43440 flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay created 2023-05-10T08:45:46.911064+0000 modified 2023-12-21T06:47:19.291154+0000 tableserver 0 root 0 session_timeout 60 session_autoclose 300 max_file_size 1099511627776 required_client_features {} last_failure 0 last_failure_osd_epoch 29480 compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} max_mds 3 in 0,1,2 up {0=1073573,2=1073583} failed damaged 1 stopped data_pools [5] metadata_pool 4 inline_data disabled balancer standby_count_wanted 1 [mds.atlassian-prod.pwsoel13142.egsdfl{0:1073573} state up:resolve seq 573 join_fscid=2 addr [v2:10.233.127.22:6800/61692284,v1:10.233.127.22:6801/61692284] compat {c=[1],r=[1],i=[7ff]}] [mds.atlassian-prod.pwsoel13143.qlvypn{2:1073583} state up:resolve seq 571 join_fscid=2 addr [v2:10.233.127.18:6800/3627858294,v1:10.233.127.18:6801/3627858294] compat {c=[1],r=[1],i=[7ff]}] Best regards, Sake

4 7

High IO utilization for bstore_kv_sync
by Work Ceph 22 Feb '24

22 Feb '24

Hello guys, We are running Ceph Octopus on Ubuntu 18.04, and we are noticing spikes of IO utilization for bstore_kv_sync thread during processes such as adding a new pool and increasing/reducing the number of PGs in a pool. It is funny though that the IO utilization (reported with IOTOP) is 99.99%, but the reading for R/W speeds are slow. The devices where we are seeing these issues are all SSDs systems. We are not using high end SSDs devices though. Have you guys seen such behavior? Also, do you guys have any clues on why the IO utilization would be high, when there is such a small amount of data being read and written to the OSD/disks?

3 4

Cannot start ceph after maintenence
by Schweiss, Chip 22 Feb '24

22 Feb '24

I had to temporarily disconnect the network on my entire Ceph cluster, so I prepared the cluster by following what appears to be some incomplete advice. I did the following before disconnecting the network: #ceph osd set noout #ceph osd set norecover #ceph osd set norebalance #ceph osd set nobackfill #ceph osd set nodown #ceph osd set pause Now, all the ceph services are still running, but I cannot undo any flags: root@proxmox01:~# ceph osd unset pause 2024-02-22T13:16:02.220+0000 7f0aab5a26c0 0 monclient(hunting): authenticate timed out after 300 [errno 110] RADOS timed out (error connecting to the cluster) Any advice on how to recover would be greatly appreciated. Thank you, -Chip

2 2

Re: Scrub stuck and 'pg has invalid (post-split) stat'
by Cedric 22 Feb '24

22 Feb '24

Update: we have run fsck and re-shard on all bluestore volume, seems sharding were not applied. Unfortunately scrubs and deep-scrubs are still stuck on PGs of the pool that is suffering the issue, but other PGs scrubs well. The next step will be to remove the cache tier as suggested, but its not available yet as PGs needs to be scrubbed in order for the cache tier can be activated. As we are struggling to make this cluster works again, any help would be greatly appreciated. Cédric > On 20 Feb 2024, at 20:22, Cedric <yipikai7(a)gmail.com> wrote: > > Thanks Eugen, sorry about the missed reply to all. > > The reason we still have the cache tier is because we were not able to flush all dirty entry to remove it (as per the procedure), so the cluster as been migrated from HDD/SSD to NVME a while ago but tiering remains, unfortunately. > > So actually we are trying to understand the root cause > > On Tue, Feb 20, 2024 at 1:43 PM Eugen Block <eblock(a)nde.ag> wrote: >> >> Please don't drop the list from your response. >> >> The first question coming to mind is, why do you have a cache-tier if >> all your pools are on nvme decices anyway? I don't see any benefit here. >> Did you try the suggested workaround and disable the cache-tier? >> >> Zitat von Cedric <yipikai7(a)gmail.com>: >> >>> Thanks Eugen, see attached infos. >>> >>> Some more details: >>> >>> - commands that actually hangs: ceph balancer status ; rbd -p vms ls ; >>> rados -p vms_cache cache-flush-evict-all >>> - all scrub running on vms_caches pgs are stall / start in a loop >>> without actually doing anything >>> - all io are 0 both from ceph status or iostat on nodes >>> >>> On Tue, Feb 20, 2024 at 10:00 AM Eugen Block <eblock(a)nde.ag> wrote: >>>> >>>> Hi, >>>> >>>> some more details would be helpful, for example what's the pool size >>>> of the cache pool? Did you issue a PG split before or during the >>>> upgrade? This thread [1] deals with the same problem, the described >>>> workaround was to set hit_set_count to 0 and disable the cache layer >>>> until that is resolved. Afterwards you could enable the cache layer >>>> again. But keep in mind that the code for cache tier is entirely >>>> removed in Reef (IIRC). >>>> >>>> Regards, >>>> Eugen >>>> >>>> [1] >>>> https://ceph-users.ceph.narkive.com/zChyOq5D/ceph-strange-issue-after-addin… >>>> >>>> Zitat von Cedric <yipikai7(a)gmail.com>: >>>> >>>>> Hello, >>>>> >>>>> Following an upgrade from Nautilus (14.2.22) to Pacific (16.2.13), we >>>>> encounter an issue with a cache pool becoming completely stuck, >>>>> relevant messages below: >>>>> >>>>> pg xx.x has invalid (post-split) stats; must scrub before tier agent >>>>> can activate >>>>> >>>>> In OSD logs, scrubs are starting in a loop without succeeding for all >>>>> pg of this pool. >>>>> >>>>> What we already tried without luck so far: >>>>> >>>>> - shutdown / restart OSD >>>>> - rebalance pg between OSD >>>>> - raise the memory on OSD >>>>> - repeer PG >>>>> >>>>> Any idea what is causing this? any help will be greatly appreciated >>>>> >>>>> Thanks >>>>> >>>>> Cédric >>>>> _______________________________________________ >>>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>> >>>> >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >>

2 7

Sharing our "Containerized Ceph and Radosgw Playground"
by Ansgar Jazdzewski 22 Feb '24

22 Feb '24

Hi Folks, We are excited to announce plans for building a larger Ceph-S3 setup. To ensure its success, extensive testing is needed in advance. Some of these tests don't need a full-blown Ceph cluster on hardware but still require meeting specific logical requirements, such as a multi-site S3 setup. To address this, we're pleased to introduce our ceph-s3-box test environment, which you can access on GitHub: https://github.com/hetznercloud/ceph-s3-box In the spirit of collaboration and knowledge sharing, we've made this testing environment publicly available today. We hope that it proves as beneficial to you as it has been for us. If you have any questions or suggestions, please don't hesitate to reach out. Cheers, Ansgar

1 0

Scrub stuck and 'pg has invalid (post-split) stat'
by Cedric 22 Feb '24

22 Feb '24

Hello, Following an upgrade from Nautilus (14.2.22) to Pacific (16.2.13), we encounter an issue with a cache pool becoming completely stuck, relevant messages below: pg xx.x has invalid (post-split) stats; must scrub before tier agent can activate In OSD logs, scrubs are starting in a loop without succeeding for all pg of this pool. What we already tried without luck so far: - shutdown / restart OSD - rebalance pg between OSD - raise the memory on OSD - repeer PG Any idea what is causing this? any help will be greatly appreciated Thanks Cédric

3 2

help me understand ceph snapshot sizes
by garcetto 22 Feb '24

22 Feb '24

good morning, i am trying to understand ceph snapshot sizing. For example if i have 2.7 GB volume and i create a snap on it, the sizing says: (BEFORE SNAP) rbd du volumes/volume-d954915c-1dc1-41cb-8bf0-0c67e7b6e080 NAME PROVISIONED USED volume-d954915c-1dc1-41cb-8bf0-0c67e7b6e080 10 GiB 2.7 GiB (AFTER SNAP) rbd du volumes/volume-d954915c-1dc1-41cb-8bf0-0c67e7b6e080 NAME PROVISIONED USED volume-d954915c-1dc1-41cb-8bf0-0c67e7b6e080@snap01 10 GiB 2.7 GiB volume-d954915c-1dc1-41cb-8bf0-0c67e7b6e080 10 GiB 0 B <TOTAL> 10 GiB 2.7 GiB why the SNAP is 2.7 GB? is not going to be 0 GB in the beginning and only after the COW start doying its thing (copying original write blocks to snap before overwrite with new ones) it should grow? am i wrong? thank you.

1 0

Reef 18.2.1 unable to join multi-side when rgw_dns_name is configured
by Ansgar Jazdzewski 21 Feb '24

21 Feb '24

Hi folks, i just try to setup a new ceph s3 multisite-setup and it looks to me that dns-style s3 is broken in multi-side as wehn rgw_dns_name is configured the `radosgw-admin period update -commit`from the new mebe will not succeeded! it looks like when ever hostnames is configured it brakes on the new to add cluster https://docs.ceph.com/en/reef/radosgw/multisite/#setting-a-zonegroup Thanks for any advice! Ansgar

1 1

first_virtual_router_id not allowed in ingress manifest
by Ramon Orrù 21 Feb '24

21 Feb '24

Hello, I deployed RGW and NFSGW services over a ceph (version 17.2.6) cluster. Both services are being accessed using 2 (separated) ingresses, actually working as expected when contacted by clients. Besides, I’m experiencing some problem while letting the ingresses work on the same cluster. keepalived logs are full of "(VI_0) received an invalid passwd!” lines, because both ingresses are using the same virtualrouter id, so I’m trying to introduce some additional parameter in service definition manifests to workaround the problem (first_virtual_router_id, default value is 50), below are the manifest content: service_type: ingress service_id: ingress.rgw service_name: ingress.rgw placement: hosts: - c00.domain.org - c01.domain.org - c02.domain.org spec: backend_service: rgw.rgw frontend_port: 8080 monitor_port: 1967 virtual_ips_list: - X.X.X.200/24 first_virtual_router_id: 60 service_type: ingress service_id: nfs.nfsgw service_name: ingress.nfs.nfsgw placement: count: 2 spec: backend_service: nfs.nfsgw frontend_port: 2049 monitor_port: 9049 virtual_ip: X.X.X.222/24 first_virtual_router_id: 70 When I apply the manifests I’m getting the error, for both ingress definitions: Error EINVAL: ServiceSpec: __init__() got an unexpected keyword argument ‘first_virtual_router_id' even the documentation for quincy version describes the option and includes some similar example at: https://docs.ceph.com/en/quincy/cephadm/services/rgw Both manifests are working smoothly if I remove the first_virtual_router_id line. Any ideas on how I can troubleshoot the issue? Thanks in advance Ramon -- Ramon Orrù Servizio di Calcolo Laboratori Nazionali di Frascati Istituto Nazionale di Fisica Nucleare Via E. Fermi, 54 - 00044 Frascati (RM) Italy Tel. +39 06 9403 2345

2 2

Ceph Leadership Team Meeting: 2024-2-21 Minutes
by Casey Bodley 21 Feb '24

21 Feb '24

Estimate on release timeline for 17.2.8? - after pacific 16.2.15 and reef 18.2.2 hotfix (https://tracker.ceph.com/issues/64339, https://tracker.ceph.com/issues/64406) Estimate on release timeline for 19.2.0? - target April, depending on testing and RCs - Testing plan for Squid beyond dev freeze (regression and upgrade tests, performance tests, RCs) Can we fix old.ceph.com? - continued discussion about the need to revive the pg calc tool T release name? - please add and vote for suggestions in https://pad.ceph.com/p/t - need name before we can open "t kickoff" pr

1 0

Re: Debian 12 (bookworm) / Reef 18.2.1 problems
by Chris Palmer 21 Feb '24

21 Feb '24

I have logged this as https://tracker.ceph.com/issues/64213 On 16/01/2024 14:18, DERUMIER, Alexandre wrote: > Hi, > >>> ImportError: PyO3 modules may only be initialized once per >>> interpreter >>> process >>> >>> and ceph -s reports "Module 'dashboard' has failed dependency: PyO3 >>> modules may only be initialized once per interpreter process > We have the same problem on proxmox8 (based on debian12) with ceph > quincy or reef. > > It seem to be related to python version on debian12 > > (we have no fix for this currently) > > >

6 7

User + Dev Meetup February 22 - CephFS Snapshots story!
by Neha Ojha 21 Feb '24

21 Feb '24

Hi everyone, You are invited to join us at the User + Dev meeting this week Thursday, February 22 at 10:00 AM Eastern Time! Focus Topic: CephFS Snapshots Evaluation Presented by: Enrico Bocchi and Abhishek Lekshmanan, Ceph operators from CERN From the presenters: Ceph at CERN provides block, object, and file storage backing the IT infrastructure of the Organization. CephFS, in particular, is largely used through the integration with OpenStack Manila by container-based workloads (Kubernetes, OpenShift), HPC MPI clusters, and as a general-purpose networked file system for enterprise groupware and open infrastructure technologies (code/software repositories, monitoring, analytics, etc.). Our presentation focuses on CephFS snapshots and their implications on performance and stability. Snapshots would be a valuable addition to our existing CephFS service, as they allow for storage rollback and disaster recovery through mirroring. According to our observations, however, they introduce a non-negligible performance penalty and may jeopardize the stability of the file system. In particular, we would like to discuss: 1. Experiences with CephFS snapshots from other operators in the Ceph community. 2. Tools and strategies one can deploy to pre-empty or mitigate issues. 3. How to effectively contribute with upstream developers and interested community users to address the identified limitations. Feel free to add questions or additional topics under the "Open Discussion" section on the agenda: https://pad.ceph.com/p/ceph-user-dev-monthly-minutes If you have an idea for a focus topic you'd like to present at a future meeting, you are welcome to submit it to this Google Form: https://docs.google.com/forms/d/e/1FAIpQLSdboBhxVoBZoaHm8xSmeBoemuXoV_rmh4v… Any Ceph user or developer is eligible to submit! Thanks, Neha

1 0

RoCE?
by Jan Marek 20 Feb '24

20 Feb '24

Hello, we are configure new ceph cluster with Mellanox 2x100Gbps cards. We bond this two ports to MLAG bond0 interface. In the async+posix mode everythink is OK, cluster is in the HELTH_OK state. CEPH version is 18.2.1. Then we tried to configure RoCE for cluster part of network, but without success. Our ceph config dump (only relevant config): global advanced ms_async_rdma_device_name mlx5_bond_0 * global advanced ms_async_rdma_gid_idx 3 global host:ceph1-nvme advanced ms_async_rdma_local_gid 0000:0000:0000:0000:0000:ffff:a0d9:05d8 * global host:ceph2-nvme advanced ms_async_rdma_local_gid 0000:0000:0000:0000:0000:ffff:a0d9:05d7 * global host:ceph3-nvme advanced ms_async_rdma_local_gid 0000:0000:0000:0000:0000:ffff:a0d9:05d6 * global advanced ms_async_rdma_roce_ver 2 global advanced ms_async_rdma_type rdma * global advanced ms_cluster_type async+rdma * global advanced ms_public_type async+posix * On the ceph1-nvme there is this show_gids.sh list: # ./show_gids.sh DEV PORT INDEX GID IPv4 VER DEV --- ---- ----- --- ------------ --- --- mlx5_bond_0 1 0 fe80:0000:0000:0000:0e42:a1ff:fe93:b004 v1 bond0 mlx5_bond_0 1 1 fe80:0000:0000:0000:0e42:a1ff:fe93:b004 v2 bond0 mlx5_bond_0 1 2 0000:0000:0000:0000:0000:ffff:a0d9:05d8 160.217.5.216 v1 bond0 mlx5_bond_0 1 3 0000:0000:0000:0000:0000:ffff:a0d9:05d8 160.217.5.216 v2 bond0 n_gids_found=4 I have set this line in /etc/security/limits.conf: * hard memlock unlimited But when I tried to restart ceph.target, OSD nodes didn't start with this errors, see attachment. Mellanox drivers are from Debian bookworm kernel. Is there somethink missing in config, or some errors? When I change ms_cluster_type to async+posix and restart ceph.target, cluster converged to HEALTH_OK state... Thanks for advices... Sincerely Jan Marek -- Ing. Jan Marek University of South Bohemia Academic Computer Centre Phone: +420389032080 http://www.gnu.org/philosophy/no-word-attachments.cs.html

1 1

What is the proper way to setup Rados Gateway (RGW) under Ceph?
by Michael Worsham 19 Feb '24

19 Feb '24

I have setup a 'reef' Ceph Cluster using Cephadm and Ansible in a VMware ESXi 7 / Ubuntu 22.04 lab environment per the how-to guide provided here: https://computingforgeeks.com/install-ceph-storage-cluster-on-ubuntu-linux-…. The installation steps were fairly easy and I was able to get the environment up and running in about 15 minutes under VMware ESXi 7. I have buckets and pools already setup. However, the ceph.io site is confusing on how to setup the Rados Gateway (radosgw) with Multi-site -- https://docs.ceph.com/en/latest/radosgw/multisite/. Is a copy of HAProxy also needed for handling the front-end load balancing or is it implied that Ceph sets it up? Command-line scripting I was planning on using for setting up the RGW: ``` radosgw-admin realm create --rgw-realm=sandbox --default radosgw-admin zonegroup create --rgw-zonegroup=sandbox --master --default radosgw-admin zone create --rgw-zonegroup=sandbox --rgw-zone=sandbox --master --default radosgw-admin period update --rgw-realm=sandbox --commit ceph orch apply rgw sandbox --realm=sandbox --zone=sandbox --placement="2 ceph-mon1 ceph-mon2" --port=8000 ``` What other steps are needed to get the RGW up and running so that it can be presented to something like Veeam for doing performance and I/O testing concepts? -- Michael This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments.

3 6

cephadm Failed to apply 1 service(s)
by Zakhar Kirpichenko 16 Feb '24

16 Feb '24

Hi, We had a physical drive malfunction in one of our Ceph OSD hosts managed by cephadm (Ceph 16.2.14). I have removed the drive from the system, and the kernel no longer sees it: ceph03 ~]# ls -al /dev/sde ls: cannot access '/dev/sde': No such file or directory I have removed the corresponding OSD from cephadm, crush map, etc. For all intents and purposes that OSD and its block device no longer exist: root@ceph01:/# ceph orch ps | grep osd.26 root@ceph01:/# ceph osd tree| grep 26 root@ceph01:/# ceph orch device ls | grep -E "ceph03.*sde" None of the above commands return anything. Cephadm correctly sees 8 remaining OSDs on the host: root@ceph01:/# ceph orch ls | grep ceph03_c osd.ceph03_combined_osd 8 33s ago 2y ceph03 Unfortunately, cephadm appears to be trying to apply a spec to host ceph03 including the disk that is now missing: RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE= quay.io/ceph/ceph@sha256:843f112990e6489362c625229c3ea3d90b8734bd5e14e0aeaf89942fbb980a8b -e NODE_NAME=ceph03 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=ceph03_combined_osd -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86:/var/run/ceph:z -v /var/log/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86:/var/log/ceph:z -v /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpc7b33pf0:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpq45nkmd6:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.io/ceph/ceph@sha256:843f112990e6489362c625229c3ea3d90b8734bd5e14e0aeaf89942fbb980a8b lvm batch --no-auto /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi --db-devices /dev/nvme0n1 /dev/nvme1n1 --yes --no-systemd Note that `lvm batch` includes the missing drive, /dev/sde. This fails because the drive no longer exists. Other than this cephadm ceph-volume thingy, the cluster is healthy.How can I tell cephadm that it should stop trying to use /dev/sde, which no longer exists, without affecting other OSDs on the host? I would very much appreciate any advice or pointers. Best regards, Zakhar

2 6

concept of ceph and 2 datacenters
by ronny.lippold＠spark5.de 16 Feb '24

16 Feb '24

hi there, i have a design/concept question, to see, whats outside and which kind of redundancy you use. actually, we use 2 ceph cluster with rbd-mirror to have an cold-standby clone. but, rbd mirror is not application consistend. so we cannot be sure, that all vms (kvm/proxy) are running. we also waste a lot of hardware. so now, we think about one big cluster over the two datacenters (two buildings). my queston is, do you care about ceph redundancy or is one ceph with backups enough for you? i know, with ceph, we are aware of hdd or server failure. but, are software failures a real scenario? would be great, to get some ideas from you. also about the bandwidth between the 2 datacenters. we are using 2x 6 proxmox server with 2x6x9 osd (sas ssd). thanks for help, my minds are rotating. kind regards, ronny -- Ronny Lippold System Administrator -- Spark 5 GmbH Rheinstr. 97 64295 Darmstadt Germany -- Fon: +49-6151-8508-050 Fax: +49-6151-8508-111 Mail: ronny.lippold(a)spark5.de Web: https://www.spark5.de -- Geschäftsführer: Henning Munte, Michael Mylius Amtsgericht Darmstadt, HRB 7809 --

4 4

HA service for RGW and dnsmasq
by Jean-Marc FONTANA 16 Feb '24

16 Feb '24

Hello everyone, We operate 2 clusters as S3 external storage for owncloud. Both were installed with ceph-deploy (Nautilus), then converted to cephadm and updated till Reef. Each cluster has one RGW host, managing a dnsmasq pack for DNS delegation. We feel know we should create a HA endpoint for RGW as explained in Reef doc. We have a few questions that we didn't find answers in the doc before proceeding : - is dnsmasq still needed on the existing ans new RGW hosts ? - if answer is yes, how should it be configured ? - if answer is no, how does DNS delegation work ? Any help will be useful. Thanks and best regards JM

1 0

RBD Mirroring
by Michel Niyoyita 15 Feb '24

15 Feb '24

Hello team, I have two clusters in testing environment deployed using ceph-ansible running on ubuntu 20.04 with ceph Pacific version . I am testing mirroring between two clusters , in pool mode . Our production Cluster is for backend storage for openstack. This is how I configured the rbdmirros.yml: ceph_rbd_mirror_configure: true ceph_rbd_mirror_mode: "pool" ceph_rbd_mirror_remote_cluster: "prod" ceph_rbd_mirror_remote_user: "admin" This is my primary cluster "/etc/ceph/ "directory: root@ceph-osd1:/etc/ceph# ls -l total 52 -r-------- 1 root root 151 Feb 13 08:47 bup.client.admin.keyring -rw-r--r-- 1 root root 866 Feb 13 08:45 bup.conf -r-------- 1 ceph ceph 151 Feb 12 13:35 ceph.client.admin.keyring -rw------- 1 ceph ceph 131 Feb 12 13:41 ceph.client.crash.keyring -rw-r--r-- 1 ceph ceph 863 Feb 12 13:35 ceph.conf -rw-rw-r-- 1 ceph ceph 1294 Feb 12 13:40 ceph-dashboard.crt -rw------- 1 ceph ceph 1704 Feb 12 13:40 ceph-dashboard.key -r-------- 1 ceph ceph 140 Feb 12 13:36 ceph.mgr.ceph-osd1.keyring -r-------- 1 ceph ceph 140 Feb 12 13:36 ceph.mgr.ceph-osd2.keyring -r-------- 1 ceph ceph 140 Feb 12 13:36 ceph.mgr.ceph-osd3.keyring -r-------- 1 root root 151 Feb 13 08:38 prod.client.admin.keyring -rw-r--r-- 1 root root 863 Feb 13 08:37 prod.conf -rw-r--r-- 1 root root 92 Aug 29 16:38 rbdmap and this is my secondary "/etc/ceph" directory: root@ceph-osdb1:/etc/ceph# ls -l total 60 -rw------- 1 root root 0 Feb 13 09:26 ansible.1e1q9lzv_ceph-ansible -rw------- 1 root root 0 Feb 13 09:32 ansible.dk1h4kzp_ceph-ansible -r-------- 1 root root 151 Feb 13 09:02 bup.client.admin.keyring -rw-r--r-- 1 root root 866 Feb 13 09:02 bup.conf -r-------- 1 ceph ceph 151 Feb 13 08:23 ceph.client.admin.keyring -rw------- 1 ceph ceph 131 Feb 13 08:29 ceph.client.crash.keyring -rw------- 1 ceph ceph 138 Feb 13 09:19 ceph.client.rbd-mirror.ceph-osdb1.keyring -rw------- 1 ceph ceph 132 Feb 13 09:19 ceph.client.rbd-mirror-peer.keyring -rw-r--r-- 1 ceph ceph 866 Feb 13 08:23 ceph.conf -rw-rw-r-- 1 ceph ceph 1302 Feb 13 08:28 ceph-dashboard.crt -rw------- 1 ceph ceph 1708 Feb 13 08:28 ceph-dashboard.key -r-------- 1 ceph ceph 141 Feb 13 08:24 ceph.mgr.ceph-osdb1.keyring -r-------- 1 ceph ceph 141 Feb 13 08:24 ceph.mgr.ceph-osdb2.keyring -r-------- 1 ceph ceph 141 Feb 13 08:24 ceph.mgr.ceph-osdb3.keyring -r-------- 1 root root 151 Feb 13 09:01 prod.client.admin.keyring -rw-r--r-- 1 root root 863 Feb 13 09:01 prod.conf -rw-r--r-- 1 root root 92 Aug 29 16:38 rbdmap Kindly I need your help if I missing something because while running I am facing the following error: TASK [ceph-rbd-mirror : create a temporary file] ************************************************************************************** Tuesday 13 February 2024 09:58:59 +0000 (0:00:00.349) 0:04:00.698 ****** changed: [ceph-osdb1 -> ceph-osdb1] TASK [ceph-rbd-mirror : write secret to temporary file] ******************************************************************************* Tuesday 13 February 2024 09:58:59 +0000 (0:00:00.533) 0:04:01.232 ****** fatal: [ceph-osdb1 -> {{ groups[mon_group_name][0] }}]: FAILED! => msg: |- The task includes an option with an undefined variable. The error was: 'ceph_rbd_mirror_remote_key' is undefined The error appears to be in '/opt/ceph-ansible/roles/ceph-rbd-mirror/tasks/configure_mirroring.yml': line 150, column 7, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: - name: write secret to temporary file ^ here Kindly help

2 8

CONFIGURE THE CEPH OBJECT GATEWAY
by ashar.khan＠rapidcompute.com 15 Feb '24

15 Feb '24

I have configured Ceph S3 encryption successfully, and the configuration is created successfully. However, when I try to upload a file to the bucket, a failed request occurs. Could you please guide me on how to properly configure it? I follow this link: https://docs.ceph.com/en/quincy/radosgw/vault/#:~:text=aes256%2Dgcm96-,CONF…

1 0

Setting Alerts/Notifications for Full Buckets in Ceph Object Storage
by asad.siddiqui＠rapidcompute.com 15 Feb '24

15 Feb '24

Hi I'm currently working with Ceph object storage (version Reef), and I'd like to know how we can set up alerts/notifications for buckets when they become full.

1 0

Ambigous mds behind on trimming and slowps issue on ceph 17.2.5 with rook 1.10.8 operator
by Akash Warkhade 14 Feb '24

14 Feb '24

Guys, We were facing cephFs volume mount issue and ceph status it was showing mds slow requests Mds behind on trimming After restarting mds pods it was resolved But wanted to know Root caus of this It was started after 2 hours of one of the active mds was crashed So does that an active mds crash can cause this issue ? Please provide your inputs anyone

1 0

Unable to add OSD after removing completely
by salam＠dcl-online.com 14 Feb '24

14 Feb '24

Hello, I have a Ceph cluster created by orchestrator Cephadm. It consists of 3 Dell PowerEdge R730XD servers. This cluster's hard drives used as OSD were configured as RAID 0. The configuration summery is as the following: ceph-node1 (mgr, mon) Public network: 172.16.7.11/22 Cluster network: 10.10.10.11/24, 10.10.10.14/24 ceph-node2 (mgr, mon) Public network: 172.16.7.12/22 Cluster network: 10.10.10.12/24, 10.10.10.15/24 ceph-node3 (mon) Public network: 172.16.7.13/22 Cluster network: 10.10.10.13/24, 10.10.10.16/24 Recently I removed all OSDs from node3 with the following set of commands sudo ceph osd out osd.3 sudo systemctl stop ceph(a)osd.3.service sudo ceph osd rm osd.3 sudo ceph osd crush rm osd.3 sudo ceph auth del osd.3 After this, I configured all OSD hard drives as non-RAIN from the server settings and tried to add the hard drives as OSD again. First I used the following command to add them automatically ceph orch apply osd --all-available-devices --unmanaged=false But this was generating the following error in my Ceph GUI console CEPHADM_APPLY_SPEC_FAIL: Failed to apply 1 service(s): osd.all-available-devices I am also unable to add the hard drives manually with the following command sudo ceph orch daemon add osd ceph-node3:/dev/sdb Can anyone please help me with this issue? I really appreciate any help you can provide.

2 3

RECENT_CRASH: x daemons have recently crashed
by Jaemin Joo 14 Feb '24

14 Feb '24

Hi everyone, I'd like to ask you why this message happened. There are no symptoms with a warning message. After archiving the message, It happened again. It will be very helpful if you give me the hint about what part I should look at. [log]----------------------------------------------- root@ceph-01:~# ceph crash ls-newID ENTITY NEW 2024-02-09T12:14:38.115931Z_a2a9e502-7582-4fb6-8187-f1e8c74291d9 client.rgw.archive.rnd-ceph-01.mbbqyj * 2024-02-10T10:29:08.507254Z_fbfce5ae-34b4-48fe-8a41-73ea489f3b4c client.rgw.archive.rnd-ceph-01.mbbqyj * 2024-02-11T08:43:18.430257Z_eeab3b6b-3241-400a-ab31-00b5fe3cc1a3 client.rgw.archive.rnd-ceph-01.mbbqyj * 2024-02-12T06:57:32.793665Z_39241466-5024-43ec-8e60-859760abaa69 client.rgw.archive.rnd-ceph-01.mbbqyj * 2024-02-13T05:12:17.654693Z_0ec8d096-9ddd-4cb6-a9fa-3f67d07b78ad client.rgw.archive.rnd-ceph-01.mbbqyj * root@ceph-01:~# ceph crash info 2024-02-09T12:14:38.115931Z_a2a9e502-7582-4fb6-8187-f1e8c74291d9 { "backtrace": [ "/lib64/libpthread.so.0(+0x12ce0) [0x7fbe48b1dce0]", "gsignal()", "abort()", "/lib64/libc.so.6(+0x21cd9) [0x7fbe47141cd9]", "/lib64/libc.so.6(+0x473f6) [0x7fbe471673f6]", "(boost::context::basic_protected_fixedsize_stack<boost::context::stack_traits>::allocate()+0x17d) [0x7fbe4b7cac4d]", "/lib64/libradosgw.so.2(+0x64e7e9) [0x7fbe4b79f7e9]", "/lib64/libradosgw.so.2(+0x64e98b) [0x7fbe4b79f98b]", "(boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0u> const, void>::operator()()+0x8d) [0x7fbe4b7cbd9d]","(void boost::asio::io_context::basic_executor_type<std::allocator<void>,0u>::execute<boost::asio::detail::strand_executor_service::allocator_binder<boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0u> const, void>, std::allocator<void> > >(boost::asio::detail::strand_executor_service::allocator_binder<boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0u> const, void>, std::allocator<void> >&&) const+0x12a) [0x7fbe4b7cbf4a]", "/lib64/libradosgw.so.2(+0x653f34) [0x7fbe4b7a4f34]", "/lib64/libradosgw.so.2(+0x658f19) [0x7fbe4b7a9f19]", "(boost::asio::detail::scheduler::run(boost::system::error_code&)+0x538) [0x7fbe4b7baf48]", "/lib64/libradosgw.so.2(+0x64d109) [0x7fbe4b79e109]", "/lib64/libstdc++.so.6(+0xc2ba3) [0x7fbe47b42ba3]", "/lib64/libpthread.so.0(+0x81ca) [0x7fbe48b131ca]", "clone()" ], "ceph_version": "17.2.3", "crash_id": "2024-02-09T12:14:38.115931Z_a2a9e502-7582-4fb6-8187-f1e8c74291d9", "entity_name": "client.rgw.archive.rnd-ceph-01.mbbqyj", "os_id": "centos", "os_name": "CentOS Stream", "os_version": "8", "os_version_id": "8", "process_name": "radosgw", "stack_sig": "5cd72ee2dcf4cbea3fcb0f28778bedd6859eb402e1482d1ef750cc75d05cd47b", "timestamp": "2024-02-09T12:14:38.115931Z", "utsname_hostname": "rnd-ceph-01", "utsname_machine": "x86_64", "utsname_release": "5.4.0-153-generic", "utsname_sysname": "Linux", "utsname_version": "#170-Ubuntu SMP Fri Jun 16 13:43:31 UTC 2023" }

1 0

Help with setting-up Influx MGR module: ERROR - queue is full
by Fulvio Galeazzi 13 Feb '24

13 Feb '24

Hi there! Has anyone any experience with the Influx Ceph mgr module? I am using 17.2.7 on CentOS8-Stream, I configured one of my clusters, I test with "ceph influx send" (whereas official doc https://docs.ceph.com/en/quincy/mgr/influx/ mentions the non-existing "ceph influx self-test") but nothing goes to the influx databases. Here is my config (password not shown): mgr advanced mgr/influx/database cephct * mgr advanced mgr/influx/hostname influxdb-dev.cloud.garr.it * mgr advanced mgr/influx/interval 300 * mgr advanced mgr/influx/password **** * mgr advanced mgr/influx/ssl false * mgr advanced mgr/influx/username cephctusr * mgr advanced mgr/influx/verify_ssl false * After enabling the module, in the MGR/MON logs I see, after a while: 2024-02-13T09:06:41.283+0100 7f5be9fff700 0 [influx ERROR root] Queue is full, failed to add chunk and "ceph health detail" shows: WRN] MGR_INFLUX_QUEUE_FULL: Failed to chunk to InfluxDB Queue Queue is full. InfluxDB might be slow with processing data (edited) (I searched a bit for "failed to chunk" but found nothing) MGR coexist with MON, and I verified (by installing influxdb by hand) that from the MON the command influx -database cephct -username cephctusr -password '****' -host influxdb-dev.cloud.garr.it indeed works. Hmm, actually while making my tests, at some point something arrived to the influxDB server, but only for 5 minutes or so, yesterday morning: it is practically impossible for me now to reconstruct what the configuration was at the time... may be during one server reboot? In any case, only the following measurements ceph_pg_summary_osd ceph_pg_summary_pool were populated, and they do not contain terribly exciting metrics, only status of PGs for each pool and number of PG per OSD. I guess the interesting metrics reported in the documentation (latency, bytes, operations...) should end up into some other measurement. I am not particularly fond of Influx, just seeking for "something"(Influx? Telegraf?) to store metrics and eventually plot to Grafana, to replace the current Zabbix-based solution. I experimented with Prometheus with some satisfaction, some time ago, although it requires a scraper which I'd be happy to avoid, especially given the point below. An additional constraint is that I have at least 3 distinct Ceph production clusters to monitor, so I'd need a way to differentiate them in a simple manner. How are you dealing with these matters, namely storing configuration and metrics "somewhere"? Thanks a lot! (for your patience in reading this, at least) Fulvio -- Fulvio Galeazzi GARR-Net Department tel.: +39-334-6533-250 skype: fgaleazzi70

1 0

Announcing go-ceph v0.26.0
by Sven Anderson 13 Feb '24

13 Feb '24

We are happy to announce another release of the go-ceph API library. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.26.0 <https://github.com/ceph/go-ceph/releases/tag/v0.23.0> The library includes bindings that aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. The library also includes additional APIs that can be used to administer cephfs, rbd, and rgw subsystems. There are already a few consumers of this library in the wild, including the ceph-csi project. Sven

1 0

Accumulation of removed_snaps_queue After Deleting Snapshots in Ceph RBD
by localhost Liam 13 Feb '24

13 Feb '24

Hello, I'm encountering an issue with Ceph when using it as the backend storage for OpenStack Cinder. Specifically, after deleting RBD snapshots through Cinder, I've noticed a significant increase in the removed_snaps_queue entries within the corresponding Ceph pool. It seems to affect the pool's performance and space efficiency. I understand that snapshot deletion in Cinder is an asynchronous operation, and Ceph itself uses a lazy deletion mechanism to handle snapshot removal. However, even after allowing sufficient time, the entries in removed_snaps_queue do not decrease as expected. I have several questions for the community: Are there recommended methods or best practices for managing or reducing entries in removed_snaps_queue? Is there any tool or command that can safely clear these residual snapshot entries without affecting the integrity of active snapshots and data? Is this issue known, and are there any bug reports or plans for fixes related to it? Thank you very much for your assistance!

2 3

Increase number of PGs
by Murilo Morais 12 Feb '24

12 Feb '24

Good morning and happy holidays everyone! Guys, what would be the best strategy to increase the number of PGs in a POOL that is already in production?

2 2

Installing ceph s3.
by Albert Shih 12 Feb '24

12 Feb '24

Hi everyone. I couldn't find a documentation about how to install a S3/Swift API (as I understand it's RadosGW) on quincy. I can find some documentation on octupus (https://docs.ceph.com/en/octopus/install/ceph-deploy/install-ceph-gateway/) Very strangely when I go https://docs.ceph.com/en/quincy/radosgw/ I can see a lot of very detailed documentation about each component, but cannot find a more global documentation. Any new documentation somewhere ? I think it's not a good idea to use the one on octopus... Regards -- Albert SHIH 🦫 🐸 France Heure locale/Local time: lun. 12 févr. 2024 18:08:59 CET

2 2

RGW core dump at start
by Vladimir Sigunov 10 Feb '24

10 Feb '24

Hello Community! I would appreciate any help/suggestions with the massive RGWs outage we are facing. The cluster's overall status is acceptable (HEALTH_WARN because of some pgs not scrubbed in time), and the cluster is operational. However, all RGWs fail to start with a core dump. The only issue I see at the moment is the RGW GC queue (radosgs-admin gc list) that contains 600K records. I believe this could be the root cause of the issue. When I pause OSD iops (ceph osd pause), all RGWs starting with no issues. There are no large OMAPs or any other warnings in ceph -s output. I would appreciate any help or suggestions you can provide. Sincerely, Vladimir.

2 2