March 2024 - ceph-users - lists.ceph.io

could not find secret_id
by xiaowenhao111 22 Apr '26

22 Apr '26

2 1

OpenSSL security update for Octopus container?
by Robert Sander 02 Oct '25

02 Oct '25

Hi, I am running a Ceph Octopus cluster in version 15.2.10 setup with the cephadm orchestrator and official Docker hub container images. Yesterday an important security fix was released for libssl and is already packaged for all major distributions. I tried to run "ceph orch upgrade check docker.io/ceph/ceph:v15" but it tells me that the containers do not need to be upgraded. How will this security fix of OpenSSL be deployed in a timely manner to users of the Ceph container images? Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 93818 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 1

Problem while upgrade 17.2.6 to 17.2.7
by Jean-Marc FONTANA 21 Feb '25

21 Feb '25

Hello everyone, We operate two clusters that we installed with ceph-deploy in Nautilus version on Debian 10. We use them for external S3 storage (owncloud) and rbd disk images.We had them upgraded to Octopus and Pacific versions on Debian 11 and recently converted them to cephadm and upgraded to Quincy (17.2.6). As we now have the orchestrator, we tried updating to 17.2.7 using the command# ceph orch upgrade start --image quay.io/ceph/ceph:v17.2.7 Everything went well, both clusters work perfectly for our use, except that the Rados gateway configuration is no longer accessible from the dashboard with the following error messageError connecting to Object Gateway: RGW REST API failed request with status code 404. We tried a few solutions found on the internet (reset rgw credentials, restart rgw adnd mgr, reenable dashboard, ...), unsuccessfully. Does somebody have an idea ? Best regards, Jean-Marc Fontana

5 9

Problems with autoscaler (overlapping roots) after changing the pool class
by Massimo Sgaravatto 06 Jan '25

06 Jan '25

Dear all I have just changed the crush rule for all the replicated pools in the following way: ceph osd crush rule create-replicated replicated_hdd default host hdd ceph osd pool set <poolname> crush_rule replicated_hdd See also this [*] thread Before applying this change, these pools were all using the replicated_ruleset rule where the class is not specified. I am noticing now a problem with the autoscaler: "ceph osd pool autoscale-status" doesn't report any output and the mgr log complains about overlapping roots: [pg_autoscaler ERROR root] pool xyz has overlapping roots: {-18, -1} Indeed: # ceph osd crush tree --show-shadow ID CLASS WEIGHT TYPE NAME -18 hdd 1329.26501 root default~hdd -17 hdd 329.14154 rack Rack11-PianoAlto~hdd -15 hdd 54.56085 host ceph-osd-04~hdd 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... ... -1 1329.26501 root default -7 329.14154 rack Rack11-PianoAlto -8 54.56085 host ceph-osd-04 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... I have already read about this behavior but I have no clear ideas how to fix the problem. I read somewhere that the problem happens when there are rules that force some pools to only use one class and there are also pools which does not make any distinction between device classes All the replicated pools are using the replicated_hdd pool but I also have some EC pools which are using a profile where the class is not specified. As far I understand, I can't force these pools to use only the hdd class: according to the doc I can't change this profile specifying the hdd class (or at least the change wouldn't be applied to the existing EC pools) Any suggestions ? The crush map is available at https://cernbox.cern.ch/s/gIyjbQbmoTFHCrr, if you want to have a look Many thanks, Massimo [*] https://www.mail-archive.com/ceph-users@ceph.io/msg18534.html

4 9

multifs and snapshots
by Dmitry Melekhov 12 Nov '24

12 Nov '24

Hello! There is note here https://docs.ceph.com/en/reef/dev/cephfs-snapshots/ about multiply filesystems and snapshots: If each FS gets its own pool things probably work, but this isn’t tested and may not be true. Is it still untested ? Thank you!

3 4

Help needed to configure erasure coding LRC plugin
by Michel Jouvin 16 Oct '24

16 Oct '24

Hi, As discussed in another thread (Crushmap rule for multi-datacenter erasure coding), I'm trying to create an EC pool spanning 3 datacenters (datacenters are present in the crushmap), with the objective to be resilient to 1 DC down, at least keeping the readonly access to the pool and if possible the read-write access, and have a storage efficiency better than 3 replica (let say a storage overhead <= 2). In the discussion, somebody mentioned LRC plugin as a possible jerasure alternative to implement this without tweaking the crushmap rule to implement the 2-step OSD allocation. I looked at the documentation (https://docs.ceph.com/en/latest/rados/operations/erasure-code-lrc/) but I have some questions if someone has experience/expertise with this LRC plugin. I tried to create a rule for using 5 OSDs per datacenter (15 in total), with 3 (9 in total) being data chunks and others being coding chunks. For this, based of my understanding of examples, I used k=9, m=3, l=4. Is it right? Is this configuration equivalent, in terms of redundancy, to a jerasure configuration with k=9, m=6? The resulting rule, which looks correct to me, is: -------- { "rule_id": 6, "rule_name": "test_lrc_2", "ruleset": 6, "type": 3, "min_size": 3, "max_size": 15, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -4, "item_name": "default~hdd" }, { "op": "choose_indep", "num": 3, "type": "datacenter" }, { "op": "chooseleaf_indep", "num": 5, "type": "host" }, { "op": "emit" } ] } ------------ Unfortunately, it doesn't work as expected: a pool created with this rule ends up with its pages active+undersize, which is unexpected for me. Looking at 'ceph health detail` output, I see for each page something like: pg 52.14 is stuck undersized for 27m, current state active+undersized, last acting [90,113,2147483647,103,64,147,164,177,2147483647,133,58,28,8,32,2147483647] For each PG, there is 3 '2147483647' entries and I guess it is the reason of the problem. What are these entries about? Clearly it is not OSD entries... Looks like a negative number, -1, which in terms of crushmap ID is the crushmap root (named "default" in our configuration). Any trivial mistake I would have made? Thanks in advance for any help or for sharing any successful configuration? Best regards, Michel

4 24

rgw connection resets
by Nathan Gleason 10 Oct '24

10 Oct '24

Hello, I’ve noticed what I think may be strange behavior from some rgw transfers. It only seems to be with larger multipart mp4 files. Once a connection comes into rgw it is accepted and processed, data begins to transfer, then I see "write_data failed: Connection reset by peer” and "s3:get_obj iterate_obj() failed with -104”. The client then reconnects and the transfer seems to be ok. This doesn’t appear to happen just downloading a regular file. This is happening to both my production and test cluster. I’m not seeing errors anywhere else, the network looks good and the clusters are healthy. I’m in the process of collecting some tcpdumps but figured I would ask if anyone knows why this is occurring. We run haproxy in front of the rgw. For troubleshooting purposes we have a basic haproxy config to just one rgw. We have tested with multiple browsers. Ceph 17.2.5 - cluster built and managed with cephadm Ubuntu 20.04.6 LTS Log: 2024-01-03T17:40:23.916+0000 7f646033d700 1 ====== starting new request req=0x7f640ca95730 ===== 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s initializing for trans_id = tx000003ec1e4b4dc157efd-0065959c07-104216-default 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s rgw api priority: s3=8 s3website=7 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s host=a2-west.usacld.net 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s final domain/bucket subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 s->info.domain= s->info.request_uri=/n athan:tshoot/testfile.mp4 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s get_handler handler=22RGWHandler_REST_Obj_S3 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s handler=22RGWHandler_REST_Obj_S3 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s getting op 0 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s get_system_obj_state: rctx=0x7f640ca947c0 obj=default.rgw.log:script.prerequest. state=0x55fa31f882e0 s->prefetch_data=0 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s cache get: name=default.rgw.log++script.prerequest. : hit (negative entry) 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj scheduling with throttler client=2 cost=1 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj op=21RGWGetObj_ObjStore_S3 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj verifying requester 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rgw::auth::StrategyRegistry::s3_main_strategy_t: trying rgw::auth::s3::AWSAuthStrategy 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::S3AnonymousEngine 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rgw::auth::s3::S3AnonymousEngine granted access 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rgw::auth::s3::AWSAuthStrategy granted access 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj normalizing buckets and tenants 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s->object=testfile.mp4 s->bucket=nathan/tshoot 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj init permissions 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_system_obj_state: rctx=0x7f640ca94200 obj=default.rgw.meta:root:nathan/tshoot state=0x55fa31f882e0 s->prefetch _data=0 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj cache get: name=default.rgw.meta+root+nathan/tshoot : hit (requested=0x16, cached=0x17) 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_system_obj_state: s->obj_tag was set empty 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj cache get: name=default.rgw.meta+root+nathan/tshoot : hit (requested=0x11, cached=0x17) 2024-01-03T17:40:23.916+0000 7f646033d700 15 req 4522146966294331133 0.000000000s s3:get_obj decode_policy Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner> <ID>nathan$testuser</ID><DisplayName>Nathan API User</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>nathan$testuser </ID><DisplayName>Nathan API User</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy> 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWSI_User_RADOS::read_user_info(): anonymous user 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj recalculating target 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj reading permissions 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_obj_state: rctx=0x7f640ca94ad0 obj=tshoot:testfile.mp4 state=0x55fa31082de8 s->prefetch_data=1 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj WARNING: blocking librados call 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj manifest: total_size = 88811314 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_obj_state: setting s->obj_tag to _0OUNxgsSsqTAs9Z6xh1SOKpJZLV_Vos 2024-01-03T17:40:23.916+0000 7f646033d700 15 req 4522146966294331133 0.000000000s s3:get_obj decode_policy Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner> <ID>nathan$testuser</ID><DisplayName>Nathan API User</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="Group"><URI>http://acs.amazonaws.c om/groups/global/AllUsers</URI></Grantee><Permission>READ</Permission></Grant><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>nathan$testuser</ID><DisplayNa me>Nathan API User</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy> 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj init op 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_system_obj_state: rctx=0x7f640ca94488 obj=default.rgw.meta:users.uid:nathan$testuser state=0x55fa31f882e0 s->p refetch_data=0 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj cache get: name=default.rgw.meta+users.uid+nathan$testuser : hit (requested=0x16, cached=0x17) 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_system_obj_state: s->obj_tag was set empty 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr: user.rgw.idtag 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj cache get: name=default.rgw.meta+users.uid+nathan$testuser : hit (requested=0x13, cached=0x17) 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj verifying op mask 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj required_mask= 1 user.op_mask=7 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj verifying op permissions 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj -- Getting permissions begin with perm_mask=49 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj Searching permissions for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=anonymous, acct_n ame=, subuser=, perm_mask=15, is_admin=0) mask=49 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj Searching permissions for uid=anonymous 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj Permissions for user not found 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj Searching permissions for group=1 mask=49 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj Found permission: 1 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj -- Getting permissions done for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=anonymous, acct_name=, subuser=, perm_mask=15, is_admin=0), owner=nathan$testuser, perm=1 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=anonymous, acct_name=, subuser=, perm_mask =15, is_admin=0) requested perm (type)=1, policy perm=1, user_perm_mask=15, acl perm=1 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj verifying op params 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj pre-executing 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj check rate limiting 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj executing 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_obj_state: rctx=0x7f640ca94ad0 obj=tshoot:testfile.mp4 state=0x55fa31082de8 s->prefetch_data=1 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.acl 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.content_type 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.etag 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.idtag 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.manifest 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.pg_ver 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.source_zone 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.tail_tag 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.x-amz-content-sha256 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.x-amz-date 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.x-amz-meta-s3cmd-attrs 2024-01-03T17:40:23.916+0000 7f646033d700 15 req 4522146966294331133 0.000000000s Encryption mode: 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_obj_state: rctx=0x7f640ca94ad0 obj=tshoot:testfile.mp4 state=0x55fa31082de8 s->prefetch_data=1 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__multipart_testfile.mp4.2~XSiTvBBtaU4 134dQCusv3kMP7YjDBk7.1 obj-ofs=0 read_ofs=0 len=4194304 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=4194304 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): result: ofs=4194304 stripe_ofs=4194304 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134 dQCusv3kMP7YjDBk7.1_1 obj-ofs=4194304 read_ofs=0 len=4194304 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=8388608 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): result: ofs=8388608 stripe_ofs=8388608 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134 dQCusv3kMP7YjDBk7.1_2 obj-ofs=8388608 read_ofs=0 len=4194304 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=12582912 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): result: ofs=12582912 stripe_ofs=12582912 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134 dQCusv3kMP7YjDBk7.1_3 obj-ofs=12582912 read_ofs=0 len=3145728 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=16777216 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): result: ofs=15728640 stripe_ofs=15728640 part_ofs=15728640 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__multipart_testfile.mp4.2~XSiTvBBtaU4 134dQCusv3kMP7YjDBk7.2 obj-ofs=15728640 read_ofs=0 len=4194304 2024-01-03T17:40:23.924+0000 7f6442301700 20 req 4522146966294331133 0.007999974s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.924+0000 7f6442301700 20 req 4522146966294331133 0.007999974s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=19922944 part_ofs=15728640 rule->part_size=15728640 2024-01-03T17:40:23.924+0000 7f6442301700 20 req 4522146966294331133 0.007999974s s3:get_obj RGWObjManifest::operator++(): result: ofs=19922944 stripe_ofs=19922944 part_ofs=15728640 rule->part_size=15728640 2024-01-03T17:40:23.924+0000 7f6442301700 20 req 4522146966294331133 0.007999974s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134 dQCusv3kMP7YjDBk7.2_1 obj-ofs=19922944 read_ofs=0 len=4194304 2024-01-03T17:40:23.932+0000 7f643d2f7700 20 req 4522146966294331133 0.015999949s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.932+0000 7f643d2f7700 20 req 4522146966294331133 0.015999949s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=24117248 part_ofs=15728640 rule->part_size=15728640 2024-01-03T17:40:23.932+0000 7f643d2f7700 20 req 4522146966294331133 0.015999949s s3:get_obj RGWObjManifest::operator++(): result: ofs=24117248 stripe_ofs=24117248 part_ofs=15728640 rule->part_size=15728640 2024-01-03T17:40:23.932+0000 7f643d2f7700 20 req 4522146966294331133 0.015999949s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134 dQCusv3kMP7YjDBk7.2_2 obj-ofs=24117248 read_ofs=0 len=4194304 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_status: e=0, sent=17, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_content_length: e=0, sent=26, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::complete_header: e=0, sent=460, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::set_account: e=1 2024-01-03T17:40:24.068+0000 7f642a2d1700 4 write_data failed: Connection reset by peer 2024-01-03T17:40:24.068+0000 7f642a2d1700 0 req 4522146966294331133 0.151999518s s3:get_obj iterate_obj() failed with -104 2024-01-03T17:40:24.156+0000 7f642eada700 2 req 4522146966294331133 0.239999235s s3:get_obj completing 2024-01-03T17:40:24.156+0000 7f642eada700 20 req 4522146966294331133 0.239999235s get_system_obj_state: rctx=0x7f640ca947c0 obj=default.rgw.log:script.postrequest.nathan state=0x55fa32424760 s->prefetch_data =0 2024-01-03T17:40:24.156+0000 7f642eada700 10 req 4522146966294331133 0.239999235s cache get: name=default.rgw.log++script.postrequest.nathan : hit (negative entry) 2024-01-03T17:40:24.156+0000 7f642eada700 30 AccountingFilter::complete_request: e=1, sent=0, total=0 2024-01-03T17:40:24.156+0000 7f642eada700 2 req 4522146966294331133 0.239999235s s3:get_obj op status=-104 2024-01-03T17:40:24.156+0000 7f642eada700 2 req 4522146966294331133 0.239999235s s3:get_obj http status=200 2024-01-03T17:40:24.156+0000 7f642eada700 1 ====== req done req=0x7f640ca95730 op status=-104 http_status=200 latency=0.239999235s ====== 2024-01-03T17:40:24.156+0000 7f642eada700 1 beast: 0x7f640ca95730: 10.30.0.196 - anonymous [03/Jan/2024:17:40:23.916 +0000] "GET /nathan:tshoot/testfile.mp4 HTTP/1.1" 200 0 - "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" - latency=0.239999235s 2024-01-03T17:40:24.156+0000 7f642eada700 20 failed to read header: end of stream 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_ACCEPT=*/* 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_ACCEPT_ENCODING=identity;q=1, *;q=0 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_ACCEPT_LANGUAGE=en-US,en;q=0.7 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_HOST=a2-west.usacld.net 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_RANGE=bytes=0- 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_REFERER=https://a2-west.usacld.net/nathan:tshoot/testfile.mp4 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_CH_UA="Not_A Brand";v="8", "Chromium";v="120", "Brave";v="120" 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_CH_UA_MOBILE=?0 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_CH_UA_PLATFORM="macOS" 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_FETCH_DEST=video 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_FETCH_MODE=no-cors 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_FETCH_SITE=same-origin 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_GPC=1 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_VERSION=1.1 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_X_FORWARDED_PORT=443 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_X_FORWARDED_PROTO=https 2024-01-03T17:40:24.160+0000 7f6503483700 20 REMOTE_ADDR=10.30.0.196 2024-01-03T17:40:24.160+0000 7f6503483700 20 REQUEST_METHOD=GET 2024-01-03T17:40:24.160+0000 7f6503483700 20 REQUEST_URI=/nathan:tshoot/testfile.mp4 2024-01-03T17:40:24.160+0000 7f6503483700 20 SCRIPT_URI=/nathan:tshoot/testfile.mp4 2024-01-03T17:40:24.160+0000 7f6503483700 20 SERVER_PORT=8000 2024-01-03T17:40:24.160+0000 7f6503483700 1 ====== starting new request req=0x7f640ca95730 ===== Thanks, Nathan

2 1

Not all Bucket Shards being used
by Christian Kugler 01 Oct '24

01 Oct '24

Hi, I have trouble with large OMAP files in a cluster in the RGW index pool. Some background information about the cluster: There is CephFS and RBD usage on the main cluster but for this issue I think only S3 is interesting. There is one realm, one zonegroup with two zones which have a bidirectional sync set up. Since this does not allow for autoresharding we have to do it by hand in this cluster – looking forward to Reef! From the logs: cluster 2023-07-17T22:59:03.018722+0000 osd.75 (osd.75) 623978 : cluster [WRN] Large omap object found. Object: 34:bcec3016:::.dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.5:head PG: 34.680c373d (34.5) Key count: 962091 Size (bytes): 277963182 The offending bucket looks like this: # radosgw-admin bucket stats \ | jq '.[] | select(.marker =="3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9") |"\(.num_shards) \(.usage["rgw.main"].num_objects)"' -r 131 9463833 Last week the number of objects was about 12 million. Which is why I reshareded the offending bucket twice, I think. Once to 129 and the second time to 131 because I wanted some leeway (or lieway? scnr, Sage). Unfortunately, even after a week the objects were still to big (the log line above is quite recent), so I looked into it again. # rados -p raum.rgw.buckets.index ls \ |grep .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9 \ |sort -V .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.0 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.1 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.2 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.3 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.4 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.5 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.6 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.7 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.8 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.9 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.10 # rados -p raum.rgw.buckets.index ls \ |grep .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9 \ |sort -V \ |xargs -IOMAP sh -c \ 'rados -p raum.rgw.buckets.index listomapkeys OMAP | wc -l' 1013854 1011007 1012287 1011232 1013565 998262 1012777 1012713 1012230 1010690 997111 Apparently, only 11 shards are in use. This would explain why the "Key usage" (from the log line) is about ten times higher than I would expect. How can I deal with this issue? One thing I could try to fix this would be to reshard to a lower number, but I am not sure if there are any risks associated with "downsharding". After that I could reshard to something like 97. Or I could directly "downshard" to 97. Also, the second zone has a similar problem, but as the error messsage lets me know, this would be a bad idea. Will it just take more time until the sharding is transferred to the seconds zone? Best, Christian Kugler

2 5

Successfully using dm-cache
by Michael Lipp 12 Sep '24

12 Sep '24

Just in case anybody is interested: Using dm-cache works and boosts performance -- at least for my use case. The "challenge" was to get 100 (identical) Linux-VMs started on a three node hyperconverged cluster. The hardware is nothing special, each node has a Supermicro server board with a single CPU with 24 cores and 4 x 4 TB hard disks. And there's that extra 1 TB NVMe... I know that the general recommendation is to use the NVMe for WAL and metadata, but this didn't seem appropriate for my use case and I'm still not quite sure about failure scenarios with this configuration. So instead I made each drive a logical volume (managed by an OSD) and added 85 GiB NVMe to each LV as read-only cache. Each VM uses as system disk an RBD based on a snapshot from the master image. The idea was that with this configuration, all VMs should share most (actually almost all) of the data on their system disk and this data should be available from the cache. Well, it works. When booting the 100 VMs, almost all read operations are satisfied from the cache. So I get close to NVMe speed but have payed for conventional hard drives only (well, SSDs aren't that much more expensive nowadays, but the hardware is 4 years old). So, nothing sophisticated, but as I couldn't find anything about this kind of setup, it might be of interest nevertheless. - Michael

13 27

cephadm discovery service certificate absent after upgrade.
by Nicolas FOURNIL 09 Jul '24

09 Jul '24

Hello, I've just fresh upgrade from Quincy to Reef and my graphs are now blank... after investigations, it seems that discovery service is not working because of no certificate : # ceph orch sd dump cert Error EINVAL: No certificate found for service discovery Maybe an upgrade issue ? Is there a way to generate or replace the certificate properly ? Regards Nicolas F.

3 8

Fixing BlueFS spillover (pacific 16.2.14)
by Chris Dunlop 09 Jul '24

09 Jul '24

Hi, tl;dr why are my osds still spilling? I've recently upgraded to 16.2.14 from 16.2.9 and started receiving bluefs spillover warnings (due to the "fix spillover alert" per the 16.2.14 release notes). E.g. from 'ceph health detail', the warning on one of these (there are a few): osd.76 spilled over 128 KiB metadata from 'db' device (56 GiB used of 60 GiB) to slow device This is a 15T HDD with only a 60G SSD for the db so it's not surprising it spilled as it's way below the recommendation for rbd usage at db size 1-2% of the storage size. There was some spare space on the db ssd so I increased the size of the db LV up over 400G and did an bluefs-bdev-expand. However, days later, I'm still getting the spillover warning for that osd, including after running a manual compact: # ceph tell osd.76 compact See attached perf-dump-76 for the perf dump output: # cephadm enter --name 'osd.76' ceph daemon 'osd.76' perf dump" | jq -r '.bluefs' In particular, if my understanding is correct, that's telling me the db available size is 487G (i.e. the LV expand worked), of which it's using 59G, and there's 128K spilled to the slow device: "db_total_bytes": 512309059584, # 487G "db_used_bytes": 63470305280, # 59G "slow_used_bytes": 131072, # 128K A "bluefs stats" also says the db is using 128K of slow storage (although perhaps it's getting the info from the same place as the perf dump?): # ceph tell osd.76 bluefs stats 1 : device size 0x7747ffe000 : using 0xea6200000(59 GiB) 2 : device size 0xe8d7fc00000 : using 0x6554d689000(6.3 TiB) RocksDBBlueFSVolumeSelector Usage Matrix: DEV/LEV WAL DB SLOW * * REAL FILES LOG 0 B 10 MiB 0 B 0 B 0 B 8.8 MiB 1 WAL 0 B 2.5 GiB 0 B 0 B 0 B 751 MiB 8 DB 0 B 56 GiB 128 KiB 0 B 0 B 50 GiB 842 SLOW 0 B 0 B 0 B 0 B 0 B 0 B 0 TOTAL 0 B 58 GiB 128 KiB 0 B 0 B 0 B 850 MAXIMUMS: LOG 0 B 22 MiB 0 B 0 B 0 B 18 MiB WAL 0 B 3.9 GiB 0 B 0 B 0 B 1.0 GiB DB 0 B 71 GiB 282 MiB 0 B 0 B 62 GiB SLOW 0 B 0 B 0 B 0 B 0 B 0 B TOTAL 0 B 74 GiB 282 MiB 0 B 0 B 0 B >> SIZE << 0 B 453 GiB 14 TiB I had a look at the "DUMPING STATS" output in the logs bug I don't know how to interpret it. I did try calculating the total of the sizes on the "Sum" lines but that comes to 100G so I don't know what that all means. See attached log-stats-76. I also tried "ceph-kvstore-tool bluestore-kv ... stats": $ { cephadm unit --fsid $clusterid --name osd.76 stop cephadm shell --fsid $clusterid --name osd.76 -- ceph-kvstore-tool bluestore-kv /var/lib/ceph/osd/ceph-76 stats cephadm unit --fsid $clusterid --name osd.76 start } Output attached as bluestore-kv-stats-76. I can't see anything interesting in there, although again I don't really know how to interpret it. So... why is this osd db still spilling onto slow storage, and how do I fix things so it's no longer using the slow storage? And a bonus issue... on another osd that hasn't yet been resized (i.e. again with a grossly undersized 60G db on SSD with a 15T HDD) I'm also getting a spillover warning. The "bluefs stats" seems to be saying the db is NOT currently spilling (i.e. "0 B" the DB/SLOW position in the matrix), but there's "something" currently using 59G on the slow device: $ ceph tell osd.85 bluefs stats 1 : device size 0xeffffe000 : using 0x3a3900000(15 GiB) 2 : device size 0xe8d7fc00000 : using 0x7aea7434000(7.7 TiB) RocksDBBlueFSVolumeSelector Usage Matrix: DEV/LEV WAL DB SLOW * * REAL FILES LOG 0 B 10 MiB 0 B 0 B 0 B 7.4 MiB 1 WAL 0 B 564 MiB 0 B 0 B 0 B 132 MiB 2 DB 0 B 11 GiB 0 B 0 B 0 B 8.1 GiB 177 SLOW 0 B 3.0 GiB 59 GiB 0 B 0 B 56 GiB 898 TOTAL 0 B 13 GiB 59 GiB 0 B 0 B 0 B 1072 MAXIMUMS: LOG 0 B 24 MiB 0 B 0 B 0 B 20 MiB WAL 0 B 2.8 GiB 0 B 0 B 0 B 1.0 GiB DB 0 B 22 GiB 448 KiB 0 B 0 B 18 GiB SLOW 0 B 3.3 GiB 62 GiB 0 B 0 B 62 GiB TOTAL 0 B 27 GiB 62 GiB 0 B 0 B 0 B >> SIZE << 0 B 57 GiB 14 TiB Is there anywhere that describes how to interpret this output, and specifically, what stuff is going into the SLOW row? Seemingly there's 898 "files" there, but not LOG, WAL or DB files - so what are they? Cheers, Chris

4 7

Correct Migration Workflow Replicated -> Erasure Code
by Mac Wynkoop 19 Jun '24

19 Jun '24

Hi Everyone, So, I'm in the process of trying to migrate our rgw.buckets.data pool from a replicated rule pool to an erasure coded pool. I've gotten the EC pool set up, good EC profile and crush ruleset, pool created successfully, but when I go to "rados cppool xxx.rgw.buckets.data xxx.rgw.buckets.data.new", I get this error after it transfers 4GB of data: error copying object: (2) No such file or directory error copying pool xxx.rgw.buckets.data => xxx.rgw.buckets.data.new: (2) No such file or directory Is "rados cppool" still the blessed way to do the migration, or has something better/not deprecated been developed that I can use? Thanks, Mac

4 4

Re: MDS crashes to damaged metadata
by Patrick Donnelly 10 Jun '24

10 Jun '24

On Thu, Dec 15, 2022 at 9:32 AM Stolte, Felix <f.stolte(a)fz-juelich.de> wrote: > > Hi Patrick, > > we used your script to repair the damaged objects on the weekend and it went smoothly. Thanks for your support. > > We adjusted your script to scan for damaged files on a daily basis, runtime is about 6h. Until thursday last week, we had exactly the same 17 Files. On thursday at 13:05 a snapshot was created and our active mds crashed once at this time (snapshot was created): > > 2022-12-08T13:05:48.919+0100 7f440afec700 -1 /build/ceph-16.2.10/src/mds/ScatterLock.h: In function 'void ScatterLock::set_xlock_snap_sync(MDSContext*)' thread 7f440afec700 time 2022-12-08T13:05:48.921223+0100 > /build/ceph-16.2.10/src/mds/ScatterLock.h: 59: FAILED ceph_assert(state LOCK_XLOCK || state LOCK_XLOCKDONE) > > 12 Minutes lates the unlink_local error crashes appeared again. This time with a new file. During debugging we noticed a MTU mismatch between MDS (1500) and client (9000) with cephfs kernel mount. The client is also creating the snapshots via mkdir in the .snap directory. > > We disabled snapshot creation for now, but really need this feature. I uploaded the mds logs of the first crash along with the information above to https://tracker.ceph.com/issues/38452 > > I would greatly appreciate it, if you could answer me the following question: > > Is the Bug related to our MTU Mismatch? We fixed the MTU Issue going back to 1500 on all nodes in the ceph public network on the weekend also. I doubt it. > If you need a debug level 20 log of the ScatterLock for further analysis, i could schedule snapshots at the end of our workdays and increase the debug level 5 Minutes arround snap shot creation. This would be very helpful! -- Patrick Donnelly, Ph.D. He / Him / His Principal Software Engineer Red Hat, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

2 4

Are we logging IRC channels?
by Alvaro Soto 08 Jun '24

08 Jun '24

Hi Community!!! Are we logging IRC channels? I ask this because a lot of people only use Slack, and the Slack we use doesn't have a subscription, so messages are lost after 90 days (I believe) I believe it's important to keep track of the technical knowledge we see each day over IRC+Slack Cheers! -- Alvaro Soto *Note: My work hours may not be your work hours. Please do not feel the need to respond during a time that is not convenient for you.* ---------------------------------------------------------- Great people talk about ideas, ordinary people talk about things, small people talk... about other people.

5 9

Help with deep scrub warnings
by Nicola Mori 24 May '24

24 May '24

Dear Ceph users, in order to reduce the deep scrub load on my cluster I set the deep scrub interval to 2 weeks, and tuned other parameters as follows: # ceph config get osd osd_deep_scrub_interval 1209600.000000 # ceph config get osd osd_scrub_sleep 0.100000 # ceph config get osd osd_scrub_load_threshold 0.300000 # ceph config get osd osd_deep_scrub_randomize_ratio 0.100000 # ceph config get osd osd_scrub_min_interval 259200.000000 # ceph config get osd osd_scrub_max_interval 1209600.000000 In my admittedly poor knowledge of Ceph's deep scrub procedures, these settings should spread the deep scrub operations in two weeks instead of the default one week, lowering the scrub frequency and the related load. But I'm currently getting warnings like: [WRN] PG_NOT_DEEP_SCRUBBED: 56 pgs not deep-scrubbed in time pg 3.1e1 not deep-scrubbed since 2024-02-22T00:22:55.296213+0000 pg 3.1d9 not deep-scrubbed since 2024-02-20T03:41:25.461002+0000 pg 3.1d5 not deep-scrubbed since 2024-02-20T09:52:57.334058+0000 pg 3.1cb not deep-scrubbed since 2024-02-20T03:30:40.510979+0000 . . . I don't understand the first one, since the deep scrub interval should be two weeks so I don''t expect warnings for PGs which have been deep-scrubbed less than 14 days ago (at the moment I'm writing it's Tue Mar 5 07:39:07 UTC 2024). Moreover, I don't understand why the deep scrub for so many PGs is lagging behind. Is there something wrong in my settings? Thanks in advance for any help, Nicola

4 4

Problems adding a new host via orchestration.
by Gary Molenkamp 22 May '24

22 May '24

Happy Friday all. I was hoping someone could point me in the right direction or clarify any limitations that could be impacting an issue I am having. I'm struggling to add a new set of hosts to my ceph cluster using cephadm and orchestration. When trying to add a host: "ceph orch host add <hostname> 172.31.102.41 --labels _admin" returns: "Error EINVAL: Can't communicate with remote host `172.31.102.41`, possibly because python3 is not installed there: [Errno 12] Cannot allocate memory" I've verified that the ceph ssh key works to the remote host, host's name matches that returned from `hostname`, python3 is installed, and "/usr/sbin/cephadm prepare-host" on the new hosts returns "host is ok". In addition, the cluster ssh key works between hosts and the existing hosts are able to ssh in using the ceph key. The existing ceph cluster is Pacific release using docker based containerization on RockyLinux8 base OS. The new hosts are RockyLinux9 based, with the cephadm being installed from Quincy release: ./cephadm add-repo --release quincy ./cephadm install I did try installing cephadm from the Pacific release by changing the repo to el8, but that did not work either. Is there a limitation is mixing RL8 and RL9 container hosts under Pacific? Does this same limitation exist under Quincy? Is there a python version dependency? The reason for RL9 on the new hosts is to stage upgrading the OS's for the cluster. I did this under Octopus for moving from Centos7 to RL8. Thanks and I appreciate any feedback/pointers. Gary I've added the log trace here in case that helps (from `ceph log last cephadm`) 2024-02-02T14:22:32.610048+0000 mgr.storage01.oonvfl (mgr.441023307) 4957871 : cephadm [ERR] Can't communicate with remote host `172.31.102.41`, possibly because python3 is not installed there: [Errno 12] Cannot allocate memory Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/serve.py", line 1524, in _remote_connection conn, connr = self.mgr._get_connection(addr) File "/usr/share/ceph/mgr/cephadm/module.py", line 1370, in _get_connection sudo=True if self.ssh_user != 'root' else False) File "/lib/python3.6/site-packages/remoto/backends/__init__.py", line 35, in __init__ self.gateway = self._make_gateway(hostname) File "/lib/python3.6/site-packages/remoto/backends/__init__.py", line 46, in _make_gateway self._make_connection_string(hostname) File "/lib/python3.6/site-packages/execnet/multi.py", line 133, in makegateway io = gateway_io.create_io(spec, execmodel=self.execmodel) File "/lib/python3.6/site-packages/execnet/gateway_io.py", line 121, in create_io io = Popen2IOMaster(args, execmodel) File "/lib/python3.6/site-packages/execnet/gateway_io.py", line 21, in __init__ self.popen = p = execmodel.PopenPiped(args) File "/lib/python3.6/site-packages/execnet/gateway_base.py", line 184, in PopenPiped return self.subprocess.Popen(args, stdout=PIPE, stdin=PIPE) File "/lib64/python3.6/subprocess.py", line 729, in __init__ restore_signals, start_new_session) File "/lib64/python3.6/subprocess.py", line 1295, in _execute_child restore_signals, start_new_session, preexec_fn) OSError: [Errno 12] Cannot allocate memory During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/serve.py", line 1528, in _remote_connection raise execnet.gateway_bootstrap.HostNotFound(msg) execnet.gateway_bootstrap.HostNotFound: Can't communicate with remote host `172.31.102.41`, possibly because python3 is not installed there: [Errno 12] Cannot allocate memory The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 125, in wrapper return OrchResult(f(*args, **kwargs)) File "/usr/share/ceph/mgr/cephadm/module.py", line 2709, in apply results.append(self._apply(spec)) File "/usr/share/ceph/mgr/cephadm/module.py", line 2574, in _apply return self._add_host(cast(HostSpec, spec)) File "/usr/share/ceph/mgr/cephadm/module.py", line 1517, in _add_host ip_addr = self._check_valid_addr(spec.hostname, spec.addr) File "/usr/share/ceph/mgr/cephadm/module.py", line 1498, in _check_valid_addr error_ok=True, no_fsid=True) File "/usr/share/ceph/mgr/cephadm/serve.py", line 1326, in _run_cephadm with self._remote_connection(host, addr) as tpl: File "/lib64/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/usr/share/ceph/mgr/cephadm/serve.py", line 1558, in _remote_connection raise OrchestratorError(msg) from e orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.31.102.41`, possibly because python3 is not installed there: [Errno 12] Cannot allocate memory -- Gary Molenkamp Science Technology Services Systems Engineer University of Western Ontario molenkam(a)uwo.ca http://sts.sci.uwo.ca (519) 661-2111 x86882 (519) 661-3566

5 9

16.2.14: [progress WARNING root] complete: ev {UUID} does not exist
by Zakhar Kirpichenko 03 May '24

03 May '24

Hi, Mgr of my cluster logs this every few seconds: [progress WARNING root] complete: ev 7de5bb74-790b-4fda-8838-e4af4af18c62 does not exist [progress WARNING root] complete: ev fff93fce-b630-4141-81ee-19e7a3e61483 does not exist [progress WARNING root] complete: ev a02f6966-5b9f-49e8-89c4-b4fb8e6f4423 does not exist [progress WARNING root] complete: ev 8d318560-ff1a-477f-9386-43f6b51080bf does not exist [progress WARNING root] complete: ev ff3740a9-6434-470a-808f-a2762fb542a0 does not exist [progress WARNING root] complete: ev 7d0589f1-545e-4970-867b-8482ce48d7f0 does not exist [progress WARNING root] complete: ev 78d57e43-5be5-43f0-8b1a-cdc60e410892 does not exist I would appreciate an advice on what these warnings mean and how they can be resolved. Best regards, Zakhar

4 7

Reset health.
by Albert Shih 03 May '24

03 May '24

Hi, Very basic question : 2 days ago I reboot all the cluster. Everything work fine. But I'm guessing during the shutdown 4 osd was mark as crash [WRN] RECENT_CRASH: 4 daemons have recently crashed osd.381 crashed on host cthulhu5 at 2024-03-20T18:33:12.017102Z osd.379 crashed on host cthulhu4 at 2024-03-20T18:47:13.838839Z osd.376 crashed on host cthulhu3 at 2024-03-20T18:50:00.877536Z osd.373 crashed on host cthulhu1 at 2024-03-20T18:56:46.887394Z is they are any way to «clean» that ? Because otherwise my icinga complain.... I don't like to add a downtime in icinga. Thanks. -- Albert SHIH 🦫 🐸 France Heure locale/Local time: ven. 22 mars 2024 22:24:35 CET

3 2

RGW multisite slowness issue due to the "304 Not Modified" responses on primary zone
by praveenkumargpk17＠gmail.com 01 May '24

01 May '24

Hi, We have 2 clusters (v18.2.1) primarily used for RGW which has over 2+ billion RGW objects. They are also in multisite configuration totaling to 2 zones and we've got around 2 Gbps of bandwidth dedicated (P2P) for the multisite traffic. We see that using "radosgw-admin sync status" on the zone 2, all the 128 shards are recovering and unfortunately there is very less data transfer from primary zone ie., the link utilization is barely 100 Mbps / 2 Gbps. Our objects are quite small as well like avg. of 1 MB in size. On further inspection, we noticed the rgw access the logs at primary site are mostly yielding "304 Not Modified" for RGWs at site-2. Is this expected? Here are some of the logs (information is redacted) root@host-04:~# tail -f /var/log/haproxy-msync.log Feb 12 05:06:51 host-04 haproxy[971171]: 10.1.85.14:33730 [12/Feb/2024:05:06:51.047] https~ backend/host-04-msync 0/0/0/2/2 304 143 - - ---- 56/55/1/0/0 0/0 "GET /bucket1/object1.jpg?rgwx-zonegroup=71dceb3d-3092-4dc6-897f-a9abf60c9972&rgwx-prepend-metadata=true&rgwx-sync-manifest&rgwx-sync-cloudtiered&rgwx-skip-decrypt&rgwx-if-not-replicated-to=a8204ce2-b69e-4d90-bca1-93edd05a1a29%3Abucket1%3A8b96aea5-c763-40a3-8430-efd67cff0c62.20010.7 HTTP/1.1" Feb 12 05:06:51 host-04 haproxy[971171]: 10.1.85.14:59730 [12/Feb/2024:05:06:51.048] https~ backend/host-04-msync 0/0/0/2/2 304 143 - - ---- 56/55/3/1/0 0/0 "GET /bucket1/object91.jpg?rgwx-zonegroup=71dceb3d-3092-4dc6-897f-a9abf60c9972&rgwx-prepend-metadata=true&rgwx-sync-manifest&rgwx-sync-cloudtiered&rgwx-skip-decrypt&rgwx-if-not-replicated-to=a8204ce2-b69e-4d90-bca1-93edd05a1a29%3Abucket1%3A8b96aea5-c763-40a3-8430-efd67cff0c62.20010.7 HTTP/1.1" We also took a look at our Grafana instance and out of 1000 requests / second, 200 requests are getting "200 OK" and 800 requests are getting "304 Not Modified". Sync threads are run on only 2 rgw daemons per zone and are behind a Load Balancer. "# radosgw-admin sync error list" also contains around 20 errors which are mostly automatically recoverable. As we understand, does it mean that RGW multisite sync logs in the log pool are yet to be generated or some sort? Please provide us some insights and let us know how to resolve this. Thanks, Praveen

4 4

Cephadm stacktrace on copying ceph.conf
by Jesper Agerbo Krogh [JSKR] 25 Apr '24

25 Apr '24

Hi. We're currently getting these errors - and I seem to be missing a clear overview over the cause and how to debug. 3/26/24 9:38:09 PM[ERR]executing _write_files((['dkcphhpcadmin01', 'dkcphhpcmgt028', 'dkcphhpcmgt029', 'dkcphhpcmgt031', 'dkcphhpcosd033', 'dkcphhpcosd034', 'dkcphhpcosd035', 'dkcphhpcosd036', 'dkcphhpcosd037', 'dkcphhpcosd038', 'dkcphhpcosd039', 'dkcphhpcosd040', 'dkcphhpcosd041', 'dkcphhpcosd042', 'dkcphhpcosd043', 'dkcphhpcosd044'],)) failed. Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/ssh.py", line 240, in _write_remote_file await asyncssh.scp(f.name, (conn, tmp_path)) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 922, in scp await source.run(srcpath) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 458, in run self.handle_error(exc) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in handle_error raise exc from None File "/lib/python3.6/site-packages/asyncssh/scp.py", line 456, in run await self._send_files(path, b'') File "/lib/python3.6/site-packages/asyncssh/scp.py", line 438, in _send_files self.handle_error(exc) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in handle_error raise exc from None File "/lib/python3.6/site-packages/asyncssh/scp.py", line 434, in _send_files await self._send_file(srcpath, dstpath, attrs) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 365, in _send_file await self._make_cd_request(b'C', attrs, size, srcpath) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 343, in _make_cd_request self._fs.basename(path)) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 224, in make_request raise exc asyncssh.sftp.SFTPFailure: scp: /tmp/var/lib/ceph/5c384430-da91-11ed-af9c-c780a5227aff/config/ceph.conf.new: Permission denied During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/utils.py", line 79, in do_work return f(*arg) File "/usr/share/ceph/mgr/cephadm/serve.py", line 1088, in _write_files self._write_client_files(client_files, host) File "/usr/share/ceph/mgr/cephadm/serve.py", line 1107, in _write_client_files self.mgr.ssh.write_remote_file(host, path, content, mode, uid, gid) File "/usr/share/ceph/mgr/cephadm/ssh.py", line 261, in write_remote_file host, path, content, mode, uid, gid, addr)) File "/usr/share/ceph/mgr/cephadm/module.py", line 615, in wait_async return self.event_loop.get_result(coro) File "/usr/share/ceph/mgr/cephadm/ssh.py", line 56, in get_result return asyncio.run_coroutine_threadsafe(coro, self._loop).result() File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result return self.__get_result() File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result raise self._exception File "/usr/share/ceph/mgr/cephadm/ssh.py", line 249, in _write_remote_file raise OrchestratorError(msg) orchestrator._interface.OrchestratorError: Unable to write dkcphhpcmgt028:/var/lib/ceph/5c384430-da91-11ed-af9c-c780a5227aff/config/ceph.conf: scp: /tmp/var/lib/ceph/5c384430-da91-11ed-af9c-c780a5227aff/config/ceph.conf.new: Permission denied 3/26/24 9:38:09 PM[ERR]Unable to write dkcphhpcmgt028:/var/lib/ceph/5c384430-da91-11ed-af9c-c780a5227aff/config/ceph.conf: scp: /tmp/var/lib/ceph/5c384430-da91-11ed-af9c-c780a5227aff/config/ceph.conf.new: Permission denied Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/ssh.py", line 240, in _write_remote_file await asyncssh.scp(f.name, (conn, tmp_path)) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 922, in scp await source.run(srcpath) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 458, in run self.handle_error(exc) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in handle_error raise exc from None File "/lib/python3.6/site-packages/asyncssh/scp.py", line 456, in run await self._send_files(path, b'') File "/lib/python3.6/site-packages/asyncssh/scp.py", line 438, in _send_files self.handle_error(exc) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in handle_error raise exc from None File "/lib/python3.6/site-packages/asyncssh/scp.py", line 434, in _send_files await self._send_file(srcpath, dstpath, attrs) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 365, in _send_file await self._make_cd_request(b'C', attrs, size, srcpath) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 343, in _make_cd_request self._fs.basename(path)) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 224, in make_request raise exc asyncssh.sftp.SFTPFailure: scp: /tmp/var/lib/ceph/5c384430-da91-11ed-af9c-c780a5227aff/config/ceph.conf.new: Permission denied 3/26/24 9:38:09 PM[INF]Updating dkcphhpcmgt028:/var/lib/ceph/5c384430-da91-11ed-af9c-c780a5227aff/config/ceph.conf It seem to be related to the permissions that the manager writes the files with and the process copying them around. $ sudo ceph -v [sudo] password for adminjskr: ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) Best regards, Jesper Agerbo Krogh Director Digitalization Digitalization Topsoe A/S Haldor Topsøes Allé 1 2800 Kgs. Lyngby Denmark Phone (direct): 27773240     Read more attopsoe.com Topsoe A/S and/or its affiliates. This e-mail message (including attachments, if any) is confidential and may be privileged. It is intended only for the addressee. Any unauthorised distribution or disclosure is prohibited. Disclosure to anyone other than the intended recipient does not constitute waiver of privilege. If you have received this email in error, please notify the sender by email and delete it and any attachments from your computer system and records.

2 1

Re: Recoveries without any misplaced objects?
by Hector Martin 24 Apr '24

24 Apr '24

On 29/05/2023 20.55, Anthony D'Atri wrote: > Check the uptime for the OSDs in question I restarted all my OSDs within the past 10 days or so. Maybe OSD restarts are somehow breaking these stats? > >> On May 29, 2023, at 6:44 AM, Hector Martin <marcan(a)marcan.st> wrote: >> >> Hi, >> >> I'm watching a cluster finish a bunch of backfilling, and I noticed that >> quite often PGs end up with zero misplaced objects, even though they are >> still backfilling. >> >> Right now the cluster is down to 6 backfilling PGs: >> >> data: >> volumes: 1/1 healthy >> pools: 6 pools, 268 pgs >> objects: 18.79M objects, 29 TiB >> usage: 49 TiB used, 25 TiB / 75 TiB avail >> pgs: 262 active+clean >> 6 active+remapped+backfilling >> >> But there are no misplaced objects, and the misplaced column in `ceph pg >> dump` is zero for all PGs. >> >> If I do a `ceph pg dump_json`, I can see `num_objects_recovered` >> increasing for these PGs... but the misplaced count is still 0. >> >> Is there something else that would cause recoveries/backfills other than >> misplaced objects? Or perhaps there is a bug somewhere causing the >> misplaced object count to be misreported as 0 sometimes? >> >> # ceph -v >> ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy >> (stable) >> >> - Hector >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > - Hector

3 3

Status of IPv4 / IPv6 dual stack?
by Robert Sander 23 Apr '24

23 Apr '24

Hi, as the documentation sends mixed signals in https://docs.ceph.com/en/latest/rados/configuration/network-config-ref/#ipv… "Note Binding to IPv4 is enabled by default, so if you just add the option to bind to IPv6 you’ll actually put yourself into dual stack mode." and https://docs.ceph.com/en/latest/rados/configuration/msgr2/#address-formats "Note The ability to bind to multiple ports has paved the way for dual-stack IPv4 and IPv6 support. That said, dual-stack operation is not yet supported as of Quincy v17.2.0." just the quick questions: Is a dual stacked networking with IPv4 and IPv6 now supported or not? From which version on is it considered stable? Are OSDs now able to register themselves with two IP addresses in the cluster map? MONs too? Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

6 6

RGW: Cannot write to bucket anymore
by Malte Stroem 21 Apr '24

21 Apr '24

Hello, there is one bucket for a user in our Ceph cluster who is suddenly not able to write to one of his buckets. Reading works fine. All other buckets work fine. If we copy the bucket to another bucket on the same cluster, the error stays. Writing is not possible in the new bucket, too. Interesting: If we copy the contents of the bucket to a bucket in another Ceph cluster the error is gone. So now we know how to solve this but we do not finde the root cause. I checked the policies, lifecycle and versioning. Nothing. The user has FULL_CONTROL. Same settings for the user's other buckets he can still write to. Wenn setting debugging to higher numbers all I can see is something like this while trying to write to the bucket: s3:put_obj reading permissions s3:put_obj init op s3:put_obj verifying op mask s3:put_obj verifying op permissions op->ERRORHANDLER: err_no=-13 new_err_no=-13 cache get: name=default.rgw.log++script.postrequest. : hit (negative entry) s3:put_obj op status=0 s3:put_obj http status=403 1 ====== req done req=0x7fe8bb60a710 op status=0 http_status=403 latency=0.000000000s ====== I still think there is something with a policy or so. When we copy the bucket to another bucket in the same cluster, at first, while copying you can write to the new bucket but when the copy job progresses at one point writing is not possible anymore. But what is it? Best, Malte

2 4

Ceph 16.2.x mon compactions, disk writes
by Zakhar Kirpichenko 17 Apr '24

17 Apr '24

Hi, Monitors in our 16.2.14 cluster appear to quite often run "manual compaction" tasks: debug 2023-10-09T09:30:53.888+0000 7f48a329a700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1696843853892760, "job": 64225, "event": "flush_started", "num_memtables": 1, "num_entries": 715, "num_deletes": 251, "total_data_size": 3870352, "memory_usage": 3886744, "flush_reason": "Manual Compaction"} debug 2023-10-09T09:30:53.904+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:30:53.908+0000 7f48a3a9b700 4 rocksdb: (Original Log Time 2023/10/09-09:30:53.910204) [db_impl/db_impl_compaction_flush.cc:2516] [default] Manual compaction from level-0 to level-5 from 'paxos .. 'paxos; will stop at (end) debug 2023-10-09T09:30:53.908+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:30:53.908+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:30:53.908+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:30:53.908+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:30:53.908+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:30:53.908+0000 7f48a3a9b700 4 rocksdb: (Original Log Time 2023/10/09-09:30:53.911004) [db_impl/db_impl_compaction_flush.cc:2516] [default] Manual compaction from level-5 to level-6 from 'paxos .. 'paxos; will stop at (end) debug 2023-10-09T09:32:08.956+0000 7f48a329a700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1696843928961390, "job": 64228, "event": "flush_started", "num_memtables": 1, "num_entries": 1580, "num_deletes": 502, "total_data_size": 8404605, "memory_usage": 8465840, "flush_reason": "Manual Compaction"} debug 2023-10-09T09:32:08.972+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:08.976+0000 7f48a3a9b700 4 rocksdb: (Original Log Time 2023/10/09-09:32:08.977739) [db_impl/db_impl_compaction_flush.cc:2516] [default] Manual compaction from level-0 to level-5 from 'logm .. 'logm; will stop at (end) debug 2023-10-09T09:32:08.976+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:08.976+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:08.976+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:08.976+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:08.976+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:08.976+0000 7f48a3a9b700 4 rocksdb: (Original Log Time 2023/10/09-09:32:08.978512) [db_impl/db_impl_compaction_flush.cc:2516] [default] Manual compaction from level-5 to level-6 from 'logm .. 'logm; will stop at (end) debug 2023-10-09T09:32:12.764+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:12.764+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:12.764+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:12.764+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:12.764+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:12.764+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:33:29.028+0000 7f48a329a700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1696844009033151, "job": 64231, "event": "flush_started", "num_memtables": 1, "num_entries": 1430, "num_deletes": 251, "total_data_size": 8975535, "memory_usage": 9035920, "flush_reason": "Manual Compaction"} debug 2023-10-09T09:33:29.044+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:33:29.048+0000 7f48a3a9b700 4 rocksdb: (Original Log Time 2023/10/09-09:33:29.049585) [db_impl/db_impl_compaction_flush.cc:2516] [default] Manual compaction from level-0 to level-5 from 'paxos .. 'paxos; will stop at (end) debug 2023-10-09T09:33:29.048+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:33:29.048+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:33:29.048+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:33:29.048+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:33:29.048+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:33:29.048+0000 7f48a3a9b700 4 rocksdb: (Original Log Time 2023/10/09-09:33:29.050355) [db_impl/db_impl_compaction_flush.cc:2516] [default] Manual compaction from level-5 to level-6 from 'paxos .. 'paxos; will stop at (end) I have removed a lot of interim log messages to save space. During each compaction the monitor process writes approximately 500-600 MB of data to disk over a short period of time. These writes add up to tens of gigabytes per hour and hundreds of gigabytes per day. Monitor rocksdb and compaction options are default: "mon_compact_on_bootstrap": "false", "mon_compact_on_start": "false", "mon_compact_on_trim": "true", "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", Is this expected behavior? Is this something I can adjust in order to extend the system storage life? Best regards, Zakhar

5 32

Ceph Users Feedback Survey
by Neha Ojha 16 Apr '24

16 Apr '24

Hi everyone, On behalf of the Ceph Foundation Board, I would like to announce the creation of, and cordially invite you to, the first of a recurring series of meetings focused solely on gathering feedback from the users of Ceph. The overarching goal of these meetings is to elicit feedback from the users, companies, and organizations who use Ceph in their production environments. You can find more details about the motivation behind this effort in our user survey [1] that we highly encourage all of you to take. This is an extension of the Ceph User Dev Meeting with concerted focus on Performance (led by Vincent Hsu, IBM) and Orchestration/Deployment (led by Matt Leonard, Bloomberg), to start off with. We would like to kick off this series of meetings on March 21, 2024. The survey will be open until March 18, 2024. Looking forward to hearing from you! Thanks, Neha [1] https://docs.google.com/forms/d/15aWxoG4wSQz7ziBaReVNYVv94jA0dSNQsDJGqmHCLM…

1 2

Setting S3 bucket policies with multi-tenants
by Thomas Bennett 15 Apr '24

15 Apr '24

Hi, I'm running Ceph Quincy (17.2.6) with a rados-gateway. I have muti tenants, for example: - Tenant1$manager - Tenant1$readwrite I would like to set a policy on a bucket (backups for example) owned by *Tenant1$manager* to allow *Tenant1$readwrite* access to that bucket. I can't find any documentation that discusses this scenario. Does anyone know how to specify the Principle and Resource section of a policy.json file? Or any other configuration that I might be missing? I've tried some variations on Principal and Resource including and excluding tenant information, but not no luck yet. For example: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": {"AWS": ["arn:aws:iam:::user/*Tenant1$readwrite*"]}, "Action": ["s3:ListBucket","s3:GetObject", ,"s3:PutObject"], "Resource": [ "arn:aws:s3:::*Tenant1/backups*" ] }] } I'm using s3cmd for testing, so: s3cmd --config s3cfg.manager setpolicy policy.json s3://backups/ Returns: s3://backups/: Policy updated And then testing: s3cmd --config s3cfg.readwrite ls s3://backups/ ERROR: Access to bucket 'backups' was denied ERROR: S3 error: 403 (AccessDenied) Thanks, Tom

4 4

MDS Behind on Trimming...
by Erich Weiler 15 Apr '24

15 Apr '24

Hi All, I've been battling this for a while and I'm not sure where to go from here. I have a Ceph health warning as such: # ceph -s cluster: id: 58bde08a-d7ed-11ee-9098-506b4b4da440 health: HEALTH_WARN 1 MDSs report slow requests 1 MDSs behind on trimming services: mon: 5 daemons, quorum pr-md-01,pr-md-02,pr-store-01,pr-store-02,pr-md-03 (age 5d) mgr: pr-md-01.jemmdf(active, since 3w), standbys: pr-md-02.emffhz mds: 1/1 daemons up, 2 standby osd: 46 osds: 46 up (since 9h), 46 in (since 2w) data: volumes: 1/1 healthy pools: 4 pools, 1313 pgs objects: 260.72M objects, 466 TiB usage: 704 TiB used, 424 TiB / 1.1 PiB avail pgs: 1306 active+clean 4 active+clean+scrubbing+deep 3 active+clean+scrubbing io: client: 123 MiB/s rd, 75 MiB/s wr, 109 op/s rd, 1.40k op/s wr And the specifics are: # ceph health detail HEALTH_WARN 1 MDSs report slow requests; 1 MDSs behind on trimming [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests mds.slugfs.pr-md-01.xdtppo(mds.0): 99 slow requests are blocked > 30 secs [WRN] MDS_TRIM: 1 MDSs behind on trimming mds.slugfs.pr-md-01.xdtppo(mds.0): Behind on trimming (13884/250) max_segments: 250, num_segments: 13884 That "num_segments" number slowly keeps increasing. I suspect I just need to tell the MDS servers to trim faster but after hours of googling around I just can't figure out the best way to do it. The best I could come up with was to decrease "mds_cache_trim_decay_rate" from 1.0 to .8 (to start), based on this page: https://www.suse.com/support/kb/doc/?id=000019740 But it doesn't seem to help, maybe I should decrease it further? I am guessing this must be a common issue...? I am running Reef on the MDS servers, but most clients are on Quincy. Thanks for any advice! cheers, erich

5 24

Call for Interest: Managed SMB Protocol Support
by John Mulligan 11 Apr '24

11 Apr '24

Hello Ceph List, I'd like to formally let the wider community know of some work I've been involved with for a while now: adding Managed SMB Protocol Support to Ceph. SMB being the well known network file protocol native to Windows systems and supported by MacOS (and Linux). The other key word "managed" meaning integrating with Ceph management tooling - in this particular case cephadm for orchestration and eventually a new MGR module for managing SMB shares. The effort is still in it's very early stages. We have a PR adding initial support for Samba Containers to cephadm [1] and a prototype for an smb MGR module [2]. We plan on using container images based on the samba-container project [3] - a team I am already part of. What we're aiming for is a feature set similar to the current NFS integration in Ceph, but with a focus on bridging non-Linux/Unix clients to CephFS using a protocol built into those systems. A few major features we have planned include: * Standalone servers (internally defined users/groups) * Active Directory Domain Member Servers * Clustered Samba support * Exporting Samba stats via Prometheus metrics * A `ceph` cli workflow loosely based on the nfs mgr module I wanted to share this information in case there's wider community interest in this effort. I'm happy to take your questions / thoughts / suggestions in this email thread, via Ceph slack (or IRC), or feel free to attend a Ceph Orchestration weekly meeting! I try regularly attend and we sometimes discuss design aspects of the smb effort there. It's on the Ceph Community Calendar. Thanks! [1] - https://github.com/ceph/ceph/pull/55068 [2] - https://github.com/ceph/ceph/pull/56350 [3] - https://github.com/samba-in-kubernetes/samba-container/ Thanks for reading, --John Mulligan

7 16

Cephadm host keeps trying to set osd_memory_target to less than minimum
by mads2a＠gmail.com 11 Apr '24

11 Apr '24

I have a virtual ceph cluster running 17.2.6 with 4 ubuntu 22.04 hosts in it, each with 4 OSD's attached. The first 2 servers hosting mgr's have 32GB of RAM each, and the remaining have 24gb For some reason i am unable to identify, the first host in the cluster appears to constantly be trying to set the osd_memory_target variable to roughly half of what the calculated minimum is for the cluster, i see the following spamming the logs constantly Unable to set osd_memory_target on my-ceph01 to 480485376: error parsing value: Value '480485376' is below minimum 939524096 Default is set to 4294967296. I did double check and osd_memory_base (805306368) + osd_memory_cache_min (134217728) adds up to minimum exactly osd_memory_target_autotune is currently enabled. But i cannot for the life of me figure out how it is arriving at 480485376 as a value for that particular host that even has the most RAM. Neither the cluster or the host is even approaching max utilization on memory, so it's not like there are processes competing for resources.

3 11

A couple OSDs not starting after host reboot
by Alison Peisker 11 Apr '24

11 Apr '24

Hi all, We rebooted all the nodes in our 17.2.5 cluster after performing kernel updates, but 2 of the OSDs on different nodes are not coming back up. This is a production cluster using cephadm. The error message from the OSD log is ceph-osd[87340]: ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-665: (2) No such file or directory The error message from ceph-volume is 2023-08-23T16:12:43.452-0500 7f0cad968600 2 bluestore(/dev/mapper/ceph--febad5a5--ba44--41aa--a39e--b9897f757752-osd--block--87e548f4--b9b5--4ed8--aca8--de703a341a50) _read_bdev_label unable to decode label at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input We tried restarting the daemons and rebooting the node again, but still see the same error. Has anyone experienced this issue before? How do we fix this? Thanks, Alison

9 13

cephadm: daemon osd.x on yyy is in error state
by Zakhar Kirpichenko 06 Apr '24

06 Apr '24

Hi, A disk failed in our cephadm-managed 16.2.15 cluster, the affected OSD is down, out and stopped with cephadm, I also removed the failed drive from the host's service definition. The cluster has finished recovering but the following warning persists: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) daemon osd.11 on ceph02 is in error state Is it possible to remove or suppress this warning without having to completely remove the OSD? I would appreciate any advice or pointers. Best regards, Zakhar

2 3

cephadm auto disk preparation and OSD installation incomplete
by Kuhring, Mathias 03 Apr '24

03 Apr '24

Dear ceph community, We have trouble with new disks not being properly prepared resp. OSDs not being fully installed by cephadm. We just added one new node each with ~40 HDDs each to two of our ceph clusters. In one cluster all but 5 disks got installed automatically. In the other none got installed. We are on ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) on both clusters. (I haven't added new disks since the last upgrade if I recall correctly). This is our OSD service definition: ``` 0|0[root@ceph-3-10 ~]# ceph orch ls osd --export service_type: osd service_id: all-available-devices service_name: osd.all-available-devices placement: host_pattern: '*' spec: data_devices: all: true filter_logic: AND objectstore: bluestore --- service_type: osd service_id: unmanaged service_name: osd.unmanaged unmanaged: true spec: filter_logic: AND objectstore: bluestore ``` Usually, new disks are installed properly (as expected due to all-available-devices). This time, I can see that LVs were created (via `lsblk`, `lvs`, `cephadm ceph-volume lvm list`). And OSDs are entered to the crushmap. However, they are not assigned to a host yet, nor do they have a type or weight, e.g.: ``` 0|0[root@ceph-2-10 ~]# ceph osd tree | grep "0 osd" 518 0 osd.518 down 0 1.00000 519 0 osd.519 down 0 1.00000 520 0 osd.520 down 0 1.00000 521 0 osd.521 down 0 1.00000 522 0 osd.522 down 0 1.00000 ``` And there is also no OSD daemon created (no docker container). So, OSD creation is somehow stuck halfway. I thought of fully cleaning up the OSD/disks. Hopping cephadm might pick them up properly next time. Just zapping was not possible, e.g. `cephadm ceph-volume lvm zap --destroy /dev/sdab` results in these errors: ``` /usr/bin/docker: stderr stderr: wipefs: error: /dev/sdab: probing initialization failed: Device or resource busy /usr/bin/docker: stderr --> failed to wipefs device, will try again to workaround probable race condition ``` So, I cleaned up more manually with purging them from crush and "resetting" disk and LV with dd and dmsetup, resp.: ``` ceph osd purge 480 --force dd if=/dev/zero of=/dev/sdab bs=1M count=1 dmsetup remove ceph--e10e0f08--8705--441a--8caa--4590de22a611-osd--block--d464211c--f513--4513--86c1--c7ad63e6c142 ``` ceph-volume still reported the old volumes, but then zapping actually got rid of them (only cleaned out the left-over entries, I guess). Now, cephadm was able to get one OSD up, when I did this cleanup for only one disk. When I did it in bulk for the rest, they all got stuck again the same way. Looking into ceph-volume logs (here for osd.522 as representative): ``` 0|0[root@ceph-2-11 /var/log/ceph/55633ec3-6c0c-4a02-990c-0f87e0f7a01f]# ll *20240316 -rw-r--r-- 1 ceph ceph 613789 Mar 14 17:10 ceph-osd.522.log-20240316 -rw-r--r-- 1 root root 42473553 Mar 16 03:13 ceph-volume.log-20240316 ``` ceph-volume only reports keyring creation: ``` [2024-03-14 16:10:19,509][ceph_volume.util.prepare][INFO ] Creating keyring file for osd.522 [2024-03-14 16:10:19,510][ceph_volume.process][INFO ] Running command: /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-522/keyring --create-keyring --name osd.522 --add-key AQBfIfNlinc7EBAAHeFicrjmLEjRPGSjuFuLiQ== ``` In the OSD logs I found a couple of these, but don't know if they are related: ``` 2024-03-14T16:10:54.706+0000 7fab26988540 2 rocksdb: [db/column_family.cc:546] Failed to register data paths of column family (id: 11, name: P) ``` Has anyone seen this behaviour before? Or could tell me where I should look next to troubleshoot this (which logs)? Any help is appreciated. Best Wishes, Mathias

2 4

quincy-> reef upgrade non-cephadm
by Christopher Durham 03 Apr '24

03 Apr '24

Hi, I am upgrading my test cluster from 17.2.6 (quincy) to 18.2.2 (reef). As it was an rpm install, i am following the directions here: Reef — Ceph Documentation | | | | Reef — Ceph Documentation | | | The upgrade worked, but I have some observations and questions before I move to my production cluster: 1. I see no systemd units with the fsid in them, as described in the document above. Both before and after the upgrade, my mon and other units are: ceph-mon@<server>.serviceceph-osd(a)[N].service etc Should I be concerned? 2. Does order matter? Based on past upgrades, I do not think so, but I wanted to be sure. For example, can I update: mon/mds/radosgw/mgrs first, then afterwards update the osds? This is what i have done in previous updates and and all was well. 3. Again on order, if a server serves say, a mon and mds, I can't really easily update one without the other, based on shared libraries and such. It appears that that is ok, based on my test cluster, but wanted to be sure. Again if an mds is one of the servers to update, I know I have to updatethe remaining one after max_mds is set to 1 and others are stopped, first. 4. After upgrade of my mgr node I get: "Module [several module names] has missing NOTIFY_TYPES member" in ceph-mgr.<server>.log But the mgr starts up eventually The system is Rocky Linux 8.9 Thanks for any thoughts -Chris

2 1

ceph orchestrator for osds
by Jeffrey Turmelle 03 Apr '24

03 Apr '24

Running on Octopus: While attempting to install a bunch of new OSDs on multiple hosts, I ran some ceph orchestrator commands to install them, such as ceph orch apply osd --all-available-devices ceph orch apply osd -I HDD_drive_group.yaml I assumed these were just helper processes, and they would be short-lived. In fact, they didn’t actually work and I ended up installing each drive by hand like this: ceph orch daemon add osd ceph4.iri.columbia.edu:/dev/sdag However, now I have these services running: # ceph orch ls --service-type=osd NAME RUNNING REFRESHED AGE PLACEMENT IMAGE NAME IMAGE ID osd.HDD_drive_group 2/2 7m ago 6w ceph[456].iri.columbia.edu docker.io/ceph/ceph:v15 2cf504fded39 osd.None 54/0 7m ago - <unmanaged> docker.io/ceph/ceph:v15 2cf504fded39 osd.all-available-devices 1/0 7m ago - <unmanaged> docker.io/ceph/ceph:v15 2cf504fded39 I’m certain none of these actually created any of my running OSD daemons, but I’m not sure if it’s ok to remove them. For example: ceph orch daemon rm osd.all-available-devices ceph orch daemon rm osd.HDD_drive_group ceph orch daemon rm osd.None Does anyone have any insight to this? I can just leave them there, they don’t seem to be doing anything, but on the other hand, I don’t want any new devices to be automatically loaded or any other unintended consequences of these. Thanks for any guidance, Jeff Turmelle International Research Institute for Climate & Society <https://iri.columbia.edu/> The Climate School <https://climate.columbia.edu/> at Columbia University <https://columbia.edu/>

2 1

put bucket notification configuration - access denied
by Giada Malatesta 03 Apr '24

03 Apr '24

Hello everyone, we are facing a problem regarding the s3 operation put bucket notification configuration. We are using Ceph version 17.2.6. We are trying to configure buckets in our cluster so that a notification message is sent via amqps protocol when the content of the bucket change. To do so, we created a local rgw user with "special" capabilities and we wrote ad hoc policies for this user (list of all buckets, read access to all buckets and possibility to add, list and delete bucket configurations). The problems regards the configurations of all buckets except the one he owns, when doing this put bucket notification configuration cross-account operation we get an access denied error. I have the suspect that this problem is related to the version we are using, because when we were doing tests on another cluster we were using version 18.2.1 and we did not face this problem. Can you confirm my hypothesis? Thanks, GM.

2 1

rgw s3 bucket policies limitations (on users)
by garcetto 03 Apr '24

03 Apr '24

good afternoon, i am trying to set bucket policies to allow to different users to access same bucket with different permissions, BUT it seems that is not yet supported, am i wrong? https://docs.ceph.com/en/reef/radosgw/bucketpolicy/#limitations "We do not yet support setting policies on users, groups, or roles." thank you.

2 1

cephfs creation error
by Ramanathan S 03 Apr '24

03 Apr '24

Hi all, I just had created a ceph cluster to use cephfs. When i create the a ceph fs pool i get the filesystem below error. # ceph osd pool create cephfs_data 128 pool 'cephfs_data' created # ceph osd pool create cephfs_metadata 128 pool 'cephfs_metadata' created # ceph fs new cephfs cephfs_metadata cephfs_data new fs with metadata pool 6 and data pool 5 # ceph -s cluster: id: 1c27def45-f0f9-494d-sfke-eb4323432fd health: HEALTH_ERR 1 filesystem is offline 1 filesystem is online with fewer MDS than max_mds services: mon: 2 daemons, quorum ceph-mon01,ceph-mon02 mgr: ceph-adm01(active) mds: cephfs-0/0/1 up osd: 12 osds: 12 up, 12 in data: pools: 2 pools, 256 pgs objects: 0 objects, 0 B usage: 12 GiB used, 588 GiB / 600 GiB avail pgs: 256 active+clean but when i check the max_mds for the ceph fs it says 1 # ceph fs get cephfs | grep max_mds max_mds 1 Let anyone know what am i missing here? Any inputs is much appreciated. Regards, Ram Ceph-explorer..

3 3

Re: NoSuchKey on key that is visible in s3 list/radosgw bk
by Eric Ivancich 03 Apr '24

03 Apr '24

I have some questions for those who’ve experienced this issue. 1. It seems like those reporting this issue are seeing it strictly after upgrading to Octopus. From what version did each of these sites upgrade to Octopus? From Nautilus? Mimic? Luminous? 2. Does anyone have any lifecycle rules on a bucket experiencing this issue? If so, please describe. 3. Is anyone making copies of the affected objects (to same or to a different bucket) prior to seeing the issue? And if they are making copies, does the destination bucket have lifecycle rules? And if they are making copies, are those copies ever being removed? 4. Is anyone experiencing this issue willing to run their RGWs with 'debug_ms=1'? That would allow us to see a request from an RGW to either remove a tail object or decrement its reference counter (and when its counter reaches 0 it will be deleted). Thanks, Eric > On Nov 12, 2020, at 4:54 PM, huxiaoyu(a)horebdata.cn wrote: > > Looks like this is a very dangerous bug for data safety. Hope the bug would be quickly identified and fixed. > > best regards, > > Samuel > > > > huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn> > > From: Janek Bevendorff > Date: 2020-11-12 18:17 > To: huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn>; EDH - Manuel Rios; Rafael Lopez > CC: Robin H. Johnson; ceph-users > Subject: Re: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > I have never seen this on Luminous. I recently upgraded to Octopus and the issue started occurring only few weeks later. > > On 12/11/2020 16:37, huxiaoyu(a)horebdata.cn wrote: > which Ceph versions are affected by this RGW bug/issues? Luminous, Mimic, Octupos, or the latest? > > any idea? > > samuel > > > > huxiaoyu(a)horebdata.cn > > From: EDH - Manuel Rios > Date: 2020-11-12 14:27 > To: Janek Bevendorff; Rafael Lopez > CC: Robin H. Johnson; ceph-users > Subject: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > This same error caused us to wipe a full cluster of 300TB... will be related to some rados index/database bug not to s3. > > As Janek exposed is a mayor issue, because the error silent happend and you can only detect it with S3, when you're going to delete/purge a S3 bucket. Dropping NoSuchKey. Error is not related to S3 logic .. > > Hope this time dev's can take enought time to find and resolve the issue. Error happens with low ec profiles, even with replica x3 in some cases. > > Regards > > > > -----Mensaje original----- > De: Janek Bevendorff <janek.bevendorff(a)uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de>> > Enviado el: jueves, 12 de noviembre de 2020 14:06 > Para: Rafael Lopez <rafael.lopez(a)monash.edu <mailto:rafael.lopez@monash.edu>> > CC: Robin H. Johnson <robbat2(a)gentoo.org <mailto:robbat2@gentoo.org>>; ceph-users <ceph-users(a)ceph.io <mailto:ceph-users@ceph.io>> > Asunto: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > > Here is a bug report concerning (probably) this exact issue: > https://tracker.ceph.com/issues/47866 <https://tracker.ceph.com/issues/47866> > > I left a comment describing the situation and my (limited) experiences with it. > > > On 11/11/2020 10:04, Janek Bevendorff wrote: >> >> Yeah, that seems to be it. There are 239 objects prefixed >> .8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh in my dump. However, there are none >> of the multiparts from the other file to be found and the head object >> is 0 bytes. >> >> I checked another multipart object with an end pointer of 11. >> Surprisingly, it had way more than 11 parts (39 to be precise) named >> .1, .1_1 .1_2, .1_3, etc. Not sure how Ceph identifies those, but I >> could find them in the dump at least. >> >> I have no idea why the objects disappeared. I ran a Spark job over all >> buckets, read 1 byte of every object and recorded errors. Of the 78 >> buckets, two are missing objects. One bucket is missing one object, >> the other 15. So, luckily, the incidence is still quite low, but the >> problem seems to be expanding slowly. >> >> >> On 10/11/2020 23:46, Rafael Lopez wrote: >>> Hi Janek, >>> >>> What you said sounds right - an S3 single part obj won't have an S3 >>> multipart string as part of the prefix. S3 multipart string looks >>> like "2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme". >>> >>> From memory, single part S3 objects that don't fit in a single rados >>> object are assigned a random prefix that has nothing to do with >>> the object name, and the rados tail/data objects (not the head >>> object) have that prefix. >>> As per your working example, the prefix for that would be >>> '.8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh'. So there would be (239) "shadow" >>> objects with names containing that prefix, and if you add up the >>> sizes it should be the size of your S3 object. >>> >>> You should look at working and non working examples of both single >>> and multipart S3 objects, as they are probably all a bit different >>> when you look in rados. >>> >>> I agree it is a serious issue, because once objects are no longer in >>> rados, they cannot be recovered. If it was a case that there was a >>> link broken or rados objects renamed, then we could work to >>> recover...but as far as I can tell, it looks like stuff is just >>> vanishing from rados. The only explanation I can think of is some >>> (rgw or rados) background process is incorrectly doing something with >>> these objects (eg. renaming/deleting). I had thought perhaps it was a >>> bug with the rgw garbage collector..but that is pure speculation. >>> >>> Once you can articulate the problem, I'd recommend logging a bug >>> tracker upstream. >>> >>> >>> On Wed, 11 Nov 2020 at 06:33, Janek Bevendorff >>> <janek.bevendorff(a)uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de> >>> <mailto:janek.bevendorff@uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de>>> wrote: >>> >>> Here's something else I noticed: when I stat objects that work >>> via radosgw-admin, the stat info contains a "begin_iter" JSON >>> object with RADOS key info like this >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110924034843-crawl420/WIDE-20110924065228-02544.warc.gz", >>> "instance": "", >>> "ns": "" >>> } >>> >>> >>> and then "end_iter" with key info like this: >>> >>> >>> "key": { >>> "name": >>> ".8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh_239", >>> "instance": "", >>> "ns": "shadow" >>> } >>> >>> However, when I check the broken 0-byte object, the "begin_iter" >>> and "end_iter" keys look like this: >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.1", >>> "instance": "", >>> "ns": "multipart" >>> } >>> >>> [...] >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.19", >>> "instance": "", >>> "ns": "multipart" >>> } >>> >>> So, it's the full name plus a suffix and the namespace is >>> multipart, not shadow (or empty). This in itself may just be an >>> artefact of whether the object was uploaded in one go or as a >>> multipart object, but the second difference is that I cannot find >>> any of the multipart objects in my pool's object name dump. I >>> can, however, find the shadow RADOS object of the intact S3 object. >>> >>> >>> >>> >>> -- >>> *Rafael Lopez* >>> Devops Systems Engineer >>> Monash University eResearch Centre >>> >>> T: +61 3 9905 9118 <tel:%2B61%203%209905%209118> >>> E: rafael.lopez(a)monash.edu <mailto:rafael.lopez@monash.edu> >>> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

6 22

Failed adding back a node
by Alex 02 Apr '24

02 Apr '24

Hello. We're rebuilding our OSD nodes. Once cluster worked without any issues, this one is being stubborn I attempted to add one back to the cluster and seeing the error below in out logs: cephadm ['--image', 'registry.redhat.io/rhceph/rhceph-5-rhel8:16.2.10-160', 'pull'] 2024-03-27 19:30:53,901 7f49792ed740 DEBUG /bin/podman: 4.6.1 2024-03-27 19:30:53,905 7f49792ed740 INFO Pulling container image registry.redhat.io/rhceph/rhceph-5-rhel8:16.2.10-160... 2024-03-27 19:30:54,045 7f49792ed740 DEBUG /bin/podman: Trying to pull registry.redhat.io/rhceph/rhceph-5-rhel8:16.2.10-160... 2024-03-27 19:30:54,266 7f49792ed740 DEBUG /bin/podman: Error: initializing source docker://registry.redhat.io/rhceph/rhceph-5-rhel8:16.2.10-160: reading manifest 16.2.10-160 in registry.redhat.io/rhceph/rhceph-5-rhel8: manifest unknown 2024-03-27 19:30:54,270 7f49792ed740 INFO Non-zero exit code 125 from /bin/podman pull registry.redhat.io/rhceph/rhceph-5-rhel8:16.2.10-160 2024-03-27 19:30:54,270 7f49792ed740 INFO /bin/podman: stderr Trying to pull registry.redhat.io/rhceph/rhceph-5-rhel8:16.2.10-160... 2024-03-27 19:30:54,270 7f49792ed740 INFO /bin/podman: stderr Error: initializing source docker://registry.redhat.io/rhceph/rhceph-5-rhel8:16.2.10-160: reading manifest 16.2.10-160 in registry.redhat.io/rhceph/rhceph-5-rhel8: manifest unknown 2024-03-27 19:30:54,270 7f49792ed740 ERROR ERROR: Failed command: /bin/podman pull registry.redhat.io/rhceph/rhceph-5-rhel8:16.2.10-160 $ ceph versions { "mon": { "ceph version 16.2.10-208.el8cp (791f73fbb4bbca2ffe53a2ea0f8706dbffadcc0b) pacific (stable)": 1, "ceph version 16.2.10-248.el8cp (0edb63afd9bd3edb333364f2e0031b77e62f4896) pacific (stable)": 2 }, "mgr": { "ceph version 16.2.10-208.el8cp (791f73fbb4bbca2ffe53a2ea0f8706dbffadcc0b) pacific (stable)": 1, "ceph version 16.2.10-248.el8cp (0edb63afd9bd3edb333364f2e0031b77e62f4896) pacific (stable)": 2 }, "osd": { "ceph version 16.2.10-160.el8cp (6977980612de1db28e41e0a90ff779627cde7a8c) pacific (stable)": 160 }, "mds": {}, "rgw": { "ceph version 16.2.10-208.el8cp (791f73fbb4bbca2ffe53a2ea0f8706dbffadcc0b) pacific (stable)": 3 }, "overall": { "ceph version 16.2.10-160.el8cp (6977980612de1db28e41e0a90ff779627cde7a8c) pacific (stable)": 160, "ceph version 16.2.10-208.el8cp (791f73fbb4bbca2ffe53a2ea0f8706dbffadcc0b) pacific (stable)": 5, "ceph version 16.2.10-248.el8cp (0edb63afd9bd3edb333364f2e0031b77e62f4896) pacific (stable)": 4 } } I don't understand why it's trying to pull 16.2.10-160 which doesn't exist. registry.redhat.io/rhceph/rhceph-5-dashboard-rhel8 5 93b3137e7a65 11 months ago 696 MB registry.redhat.io/rhceph/rhceph-5-rhel8 5-416 838cea16e15c 11 months ago 1.02 GB registry.redhat.io/openshift4/ose-prometheus v4.6 ec2d358ca73c 17 months ago 397 MB This happens using cepadm-ansible as well as $ ceph orch ls --export --service_name xxx > xxx.yml $ sudo ceph orch apply -i xxx.yml I tried ceph orch daemon add osd host:/dev/sda which surprisingly created a volume on host:/dev/sda and created an osd i can see in $ ceph osd tree but It did not get added to host I suspect because of the same Podman error and now I'm unable remove it. $ ceph orch osd rm does not work even with the --force flag. I stopped the removal with $ ceph orch osd rm stop after 10+ minutes I'm considering running $ ceph osd purge osd# --force but worried it may only make things worse. ceph -s shows that osd but not up or in. Thanks, and looking forward to any advice!

2 6

Pacific Bug?
by Alex 02 Apr '24

02 Apr '24

Hello Ceph Gurus! I'm running Ceph Pacific version. if I run ceph orch host ls --label osds shows all hosts label osds or ceph orch host ls --host-pattern host1 shows just host1 it works as expected But combining the two the label tag seems to "take over" ceph orch host ls --label osds --host-pattern host1 6 hosts in cluster who had label osds whose hostname matched host1 shows all host with the label osds instead of only host1. So at first the flags seem to act like an OR instead of an AND. ceph orch host ls --label osds --host-pattern foo 6 hosts in cluster who had label osds whose hostname matched foo even though "foo" doesn't even exist ceph orch host ls --label bar --host-pattern host1 0 hosts in cluster who had label bar whose hostname matched host1 if the label and host combo was an OR this should have worked there is no label bar but host1 exists so it just disregards the host-pattern. This started because the osd deployment task had both label and host_pattern. The cluster was attempting to deploy OSDS on all the servers with the given tag instead of the one host we needed, which caused it to go into warning state. If I ran ceph orch ls --export --service_name host1 it also showed both tags and host_pattern. unmanaged: false placement: host_pattern: label: The issue persisted until I removed the label tag. Thanks.

3 5

cephfs inode backtrace information
by Dietmar Rieder 02 Apr '24

02 Apr '24

Hello, I have a question regarding the default pool of a cephfs. According to the docs it is recommended to use a fast ssd replicated pool as default pool for cephfs. I'm asking what are the space requirements for storing the inode backtrace information? Let's say I have a 85 TiB replicated ssd pool (hot data) and as 3 PiB EC data pool (cold data). Does it make sense to create a third pool as default pool which only holds the inode backtrace information (what would be a good size), or is it OK to use the ssd pool as default pool? Thanks Dietmar

6 10

S3 Partial Reads from Erasure Pool
by E3GH75＠gmail.com 02 Apr '24

02 Apr '24

I am dealing with a cluster that is having terrible performance with partial reads from an erasure coded pool. Warp tests and s3bench tests result in acceptable performance but when the application hits the data, performance plummets. Can anyone clear this up for me, When radosgw gets a partial read does it have to assemble all the rados objects that make up the s3 object before returning the range? With a replicated poll i am seeing 6 to 7 GiB/s of read performance and only 1GiB/s of read from the erasure coded pool which leads me to believe that the replicated pool is returning just the rados objects for the partial s3 object and the erasure coded pool is not.

2 1

Can setting mds_session_blocklist_on_timeout to false minize the session eviction?
by Yongseok Oh 01 Apr '24

01 Apr '24

Hi, CephFS is provided as a shared file system service in a private cloud environment of our company, LINE. The number of sessions is approximately more than 5,000, and session evictions occur several times a day. When session eviction occurs, the message 'Cannot send after transport endpoint shutdown' or 'Permission denied' is displayed and file system access is not possible. Our users are very uncomfortable with this issue. In particular, there are no special problems such as network connection or CPU usage. When I access the machine and take a close look, there are no special problems. After this, users feel the inconvenience of having to perform umount/mount tasks and run the application again. In a Kubernetes environment, recovery is a bit more complicated, which causes a lot of frustration. I tested that by setting the mds_session_blocklist_on_timeout and mds_session_blocklist_on_evict options to false and setting client_reconnect_stale to true on the client side, the file system can be accessed even if eviction occurs. It seemed like there was no major problem accessing the file system as the session was still attached. What I'm curious about is if I turn on the above option, will there be any other side effects? For example, should I take some action if the integrity of the file is broken or if there is an issue on the mds side? I am asking a question because there are no details regarding this in the official CephFS documentation. Thank you Yongseok

2 2

recreating a cephfs subvolume with the same absolute path
by Adiga, Anantha 29 Mar '24

29 Mar '24

Hi, ceph fs subvolume getpath cephfs cluster_A_subvolume cephfs_data_pool_ec21_subvolumegroup /volumes/cephfs_data_pool_ec21_subvolumegroup/cluster_A_subvolume/0f90806d-0d70-4fe1-9e2b-f958056ef0c9 If the subvolume got deleted, is it possible to recreate the subvolume with the same absolute path? so that yml specs that use the volume paths need not change Thank you, Anantha

1 0

v17.2.7 Quincy now supports Ubuntu 22.04 (Jammy Jellyfish)
by Casey Bodley 29 Mar '24

29 Mar '24

Ubuntu 22.04 packages are now available for the 17.2.7 Quincy release. The upcoming Squid release will not support Ubuntu 20.04 (Focal Fossa). Ubuntu users planning to upgrade from Quincy to Squid will first need to perform a distro upgrade to 22.04. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-17.2.7.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: b12291d110049b2f35e32e0de30d70e9a4c060d2

1 0

Improving CephFS performance by always putting "default" data pool on SSDs?
by Niklas Hambüchen 29 Mar '24

29 Mar '24

https://docs.ceph.com/en/reef/cephfs/createfs/ says: > The data pool used to create the file system is the “default” data pool and the location for storing all inode backtrace information, which is used for hard link management and disaster recovery. > For this reason, all CephFS inodes have at least one object in the default data pool. If erasure-coded pools are planned for file system data, it is best to configure the default as a replicated pool to improve small-object write and read performance when updating backtraces. This poses the question: Are normal replicated CephFS installations (metadata on SSDs, data on HDDs) set up with suboptimal performance because they don't do this? If having inodes/backtraces on replicated instead of EC improves performance, shouldn't one expect that putting inodes/backtraces on SSD would improve it even more? From the docs I also cannot really conclude when inotes/backtraces become important. Is that all the time, or only sometimes? Thanks!

2 3

Linux Laptop Losing CephFS mounts on Sleep/Hibernate
by matthew＠peregrineit.net 29 Mar '24

29 Mar '24

Hi All, So I've got a Ceph Reef Cluster (latest version) with a CephFS system set up with a number of directories on it. On a Laptop (running Rocky Linux (latest version)) I've used fstab to mount a number of those directories - all good, everything works, happy happy joy joy! :-) However, when the laptop goes into sleep or hibernate mode (ie when I close the lid) and then bring it back out of sleep/hibernate (ie open the lid) the CephFS mounts are "not present". The only way to get them back is to run `mount -a` as either root or as sudo. This, as I'm sure you'll agree, is less than ideal - especially as this is a pilot project for non-admin users (ie they won't have access to the root account or sudo on their own (corporate) laptops). So, my question to the combined wisdom of the Community is what's the best way to resolve this issue? I've looked at autofs, and even tried (half-heartedly - it was late, and I wanted to go home :-) ) to get this running, but I'm note sure if this is the best way to resolve things. All help and advice on this greatly appreciated - thank in advance Cheers Dulux-Oz

5 4

PG's stuck incomplete on EC pool after multiple drive failure
by Malcolm Haak 29 Mar '24

29 Mar '24

Hello all. I have a cluster with ~80TB of spinning disk. Its primary role is cephfs. Recently I had a multiple drive failure (it was not simultaneous) but it's left me with 20 incomplete pg's I know this data is toast, but I need to be able to get what isn't toast out of the cephfs. Well out of that pool and into a new pool. The issue is the PG's that are incomplete block IO and that hinders browsing the filesystem. I'm attempting to use the "new" ceph-objectstore-tool mark-complete operation, but I'm struggling to work out what to mark complete. Being that it's EC and each PG is made up of multiple shards (I think is the right word) and they all have their own status. I did manage to mark one of these shard pg's complete on what appeared to be the primary OSD, however it had no effect on that shard when checking it with ceph pg X query. By that I mean the shard was marked incomplete before and after using ceph-ojbectstore-tool. I'm running Ceph 18.2.2, I have all HDD osd's. I can get any logs that will help, I'm just not 100% sure where to start and don't want to just dump 20 PG's of ceph pg X query on the mailing list Thanks so much Mal

1 0

mclock and massive reads
by Luis Domingues 28 Mar '24

28 Mar '24

Hello, We have a question about mClock scheduling reads on pacific (16.2.14 currently). When we do massive reads, from let's say machines we want to drain containing a lot of data on EC pools, we observe quite frequently slow ops on the source OSDs. Those slow ops affect the client services, talking directly rados. If we kill the OSD that causes slow ops, the recovery stays more or less at the same speed, but no more slow ops. And when we tweak mClock, if we limit on the OSDs that are the source, nothing that we can observe happens. However, if we limit on the target OSDs, the global speed slows down, and the slow ops disappear. So our question, is mClock taking into account the reads as well as the writes? Or are the reads calculate to be less expensive than the writes? Thanks, Luis Domingues Proton AG

2 2

Fwd: Welcome to the "ceph-users" mailing list
by 许晨辉 28 Mar '24

28 Mar '24

---------- Forwarded message --------- 发件人： <ceph-users-request(a)ceph.io> Date: 2024年3月25日周一 20:58 Subject: Welcome to the "ceph-users" mailing list To: <xuchenhuig(a)gmail.com> Welcome to the "ceph-users" mailing list! To post to this list, send your email to: ceph-users(a)ceph.io You can unsubscribe or make adjustments to your options via email by sending a message to: ceph-users-request(a)ceph.io with the word 'help' in the subject or body (don't include the quotes), and you will get back a message with instructions. You will need your password to change your options, but for security purposes, this password is not included here. If you have forgotten your password you will need to reset it via the web UI.

1 0

1x port from bond down causes all osd down in a single machine
by Szabo, Istvan (Agoda) 28 Mar '24

28 Mar '24

Hi, Wonder what we are missing from the netplan configuration on ubuntu which ceph needs to tolerate properly. We are using this bond configuration on ubuntu 20.04 with octopus ceph: bond1: macaddress: x.x.x.x.x.50 dhcp4: no dhcp6: no addresses: - 192.168.199.7/24 interfaces: - ens2f0np0 - ens2f1np1 mtu: 9000 parameters: mii-monitor-interval: 100 mode: 802.3ad lacp-rate: fast transmit-hash-policy: layer3+4 ens2f1np1 failed and caused slow ops, all osd down ... = disaster Any idea what is wrong with this bond config? Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 1

Re: Return value from cephadm host-maintenance?
by John Mulligan 27 Mar '24

27 Mar '24

(adding the list back to the thread) On Wednesday, March 27, 2024 12:54:34 PM EDT Daniel Brown wrote: > John > > > I got curious and was taking another quick look through the python script > for cephadm. > That's always welcome. :-D > This is probably too simple of a question to be asking — or maybe I should > say, I’m not expecting that there’s a simple answer to what might seem like > a simple question - > > Is there anything that notifies the cluster, or the other hosts in a > cluster, when a host is going into maintenance mode that it is going into > maintenance mode, or is cephadm just doing systemctl commands behind the > scenes to stop and later restart the appropriate ceph containers locally on > that host? > > Maybe a better way to say it would be - what is differentiating between > maintenance mode and a host simply crashing or going offline? I'll paraphrase Adam King, tech lead for cephadm here: If one runs the command from cephadm binary directly, it will be disabling/ stopping the systemd target only. The intention is for users to use the `ceph orch host maintenance` ... commands. When you use the orch command (quoting Adam here): ``` when we put something into maintenance mode we 1) disable and stop the systemd target for the daemons on the host 2) set the noout flag for all the OSDs on that host 3) internally to cephadm mark the host as having a status of "maintenance" which has some effects such as us not refreshing metadata on that host or attempting to place/remove daemons from there The main difference from that to a host going offline is the noout flag for the OSDs, and that cephadm will not periodically try to check if the host is alive, as it would do for an offline host. I believe the noout flag stops it from trying to migrate all the data on that OSDs to other OSDs as it shouldn't be necessary if they will be coming back ``` The `cephadm host-maintenance enter` is meant to be a component of the `ceph orch host maintenance` workflow. It still has a bug, the way it always exits with an error is wrong. But you may not want to use it directly. Reference links: https://docs.ceph.com/en/latest/cephadm/host-management/#maintenance-mode https://docs.ceph.com/en/latest/dev/cephadm/host-maintenance/ > > On Mar 22, 2024, at 6:26 AM, Daniel Brown <daniel.h.brown(a)thermify.cloud> > > wrote: > > > > > > Looks like it got OK’ed. I’ll put in something today. > > > > > > -- > > Dan Brown > > > >> On Mar 21, 2024, at 13:44, John Mulligan <phlogistonjohn(a)asynchrono.us> > >> wrote:>> > >> On Thursday, March 21, 2024 11:43:19 AM EDT Daniel Brown wrote: > >>> Assuming I need admin approval to report this on tracker, how long does > >>> it > >>> take to get approved?? Signed up a couple days ago, but still seeing > >>> “Your > >>> account was created and is now pending administrator approval.” > >> > >> That's unfortunate. I pinged about your issue signing up on the ceph > >> slack > >> channel for infrastructure. Hopefully, that'll get somebody's attention. > >> If > >> you don't get access by tomorrow feel free to ping me again directly and > >> then *I'll* file the issue for you instead of having you wait around > >> more.

1 0

nvme hpe
by Albert Shih 27 Mar '24

27 Mar '24

Hi. I notice in the log I got log from each node Mar 27 01:12:59 cthulhu1 sudo: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme mo000800kxprv smart-log-add --json /dev/nvme1n1 Mar 27 01:13:06 cthulhu1 sudo: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme hpe smart-log-add --json /dev/nvme0n1 Mar 27 01:13:01 cthulhu2 sudo: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme mo000800kxprv smart-log-add --json /dev/nvme1n1 Mar 27 01:13:07 cthulhu2 sudo: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme hpe smart-log-add --json /dev/nvme0n1 Mar 27 01:13:02 cthulhu3 sudo: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme mo000800kxprv smart-log-add --json /dev/nvme1n1 Mar 27 01:13:07 cthulhu3 sudo: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme hpe smart-log-add --json /dev/nvme0n1 Mar 27 01:13:03 cthulhu4 sudo: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme mo000800kxnxh smart-log-add --json /dev/nvme2n1 Mar 27 01:13:07 cthulhu4 sudo: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme hpe smart-log-add --json /dev/nvme0n1 Mar 27 01:13:03 cthulhu5 sudo: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme mo000800kxnxh smart-log-add --json /dev/nvme1n1 Mar 27 01:13:06 cthulhu5 sudo: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme hpe smart-log-add --json /dev/nvme0n1 So the problem are : The plugin for mo000800kxnxh is I'm guessing wrong (and does not exist) The plugin hpe doesn't exist either. nvme find (i'm guestion) the model with nvme list, but # nvme list Node SN Model Namespace Usage Format FW Rev ---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- /dev/nvme0n1 PWWVF0DSTHO1E7 HPE NS204i-p Gen10+ Boot Controller 1 480.04 GB / 480.04 GB 512 B + 0 B 12141004 /dev/nvme1n1 231940892591 MO000800KXNXH 1 2.76 GB / 800.17 GB 512 B + 0 B HPS0 /dev/nvme2n1 23194089256A MO000800KXNXH 1 799.47 GB / 800.17 GB 512 B + 0 B HPS0 Still with lshw I find out it's was a micron ssd. So my question : whats the best thing to do ? Which «plugin» should I use and how I tell cephad what to do ? Regards -- Albert SHIH 🦫 🐸 France Heure locale/Local time: mer. 27 mars 2024 15:43:54 CET

1 0

Erasure Code with Autoscaler and Backfill_toofull
by Daniel Williams 27 Mar '24

27 Mar '24

Hey, I'm running ceph version 18.2.1 (reef) but this problem must have existed a long time before reef. The documentation says the autoscaler will target 100 pgs per OSD but I'm only seeing ~10. My erasure encoding is a stripe of 6 data 3 parity. Could that be the reason? PGs numbers for that EC pool are therefore multiplied by k+m by the autoscaler calculations? Is backfill_toofull calculated against the total size of the PG against every OSD it is destined for? For my case I have ~1TiB PGs because the autoscaler is creating only 10 per host, and then backfill too full is considering that one of my OSDs only has 500GiB free, although that doesn't quite add up either because two 1TiB PGs are backfilling two pg's that have OSD 1 in them. My backfill full ratio is set to 97%. Would it be correct for me to change the autoscaler to target ~700 pgs per osd and bias for storagefs and all EC pools to k+m? Should that be the default or the documentation recommended value? How scary is changing PG_NUM while backfilling misplaced PGs? It seems like there's a chance the backfill might succeed so I think I can wait. Any help is greatly appreciated, I've tried to include as much of the relevant debugging output as I can think of. Daniel # ceph osd ls | wc -l 44 # ceph pg ls | wc -l 484 # ceph osd pool autoscale-status POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO EFFECTIVE RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE BULK .rgw.root 216.0k 3.0 480.2T 0.0000 1.0 32 on False default.rgw.control 0 3.0 480.2T 0.0000 1.0 32 on False default.rgw.meta 0 3.0 480.2T 0.0000 1.0 32 on False default.rgw.log 1636k 3.0 480.2T 0.0000 1.0 32 on False storagefs 233.5T 1.5 480.2T 0.7294 1.0 256 on False storagefs-meta 850.2M 4.0 480.2T 0.0000 4.0 32 on False storagefs_wide 355.3G 1.375 480.2T 0.0010 1.0 32 on False .mgr 457.3M 3.0 480.2T 0.0000 1.0 1 on False mgr-backup-2022-08-19 370.6M 3.0 480.2T 0.0000 1.0 32 on False # ceph osd pool ls detail | column -t pool 15 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on pool 16 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on pool 17 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on pool 18 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on pool 36 'storagefs' erasure profile 6.3 size 9 min_size 7 crush_rule 2 object_hash rjenkins pg_num 256 pgp_num 256 autoscale_mode on pool 37 'storagefs-meta' replicated size 4 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on pool 45 'storagefs_wide' erasure profile 8.3 size 11 min_size 9 crush_rule 8 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on pool 46 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on pool 48 'mgr-backup-2022-08-19' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on # ceph osd erasure-code-profile get 6.3 crush-device-class= crush-failure-domain=host crush-root=default jerasure-per-chunk-alignment=false k=6 m=3 plugin=jerasure technique=reed_sol_van w=8 # ceph pg ls | awk 'NR==1 || /backfill_toofull/' | awk '{print $1" "$2" "$4" "$6" "$11" "$15" "$16}' | column -t PG OBJECTS MISPLACED BYTES STATE UP ACTING 36.f 222077 141392 953817797727 active+remapped+backfill_toofull [1,27,41,8,36,17,14,40,32]p1 [33,32,29,23,16,17,28,1,14]p33 36.5c 221761 147015 950692130045 active+remapped+backfill_toofull [26,27,40,29,1,37,39,11,42]p26 [12,24,4,2,31,25,17,33,8]p12 36.60 222710 0 957109050809 active+remapped+backfill_toofull [41,34,22,3,1,35,9,39,29]p41 [2,34,22,3,27,32,28,24,1]p2 36.6b 222202 427168 953843892012 active+remapped+backfill_toofull [20,15,7,21,37,1,38,17,32]p20 [7,2,32,26,5,35,24,17,23]p7 36.74 222681 777546 957679960067 active+remapped+backfill_toofull [42,24,12,34,38,10,27,1,25]p42 [34,33,12,0,19,14,17,30,25]p34 36.7b 222974 1560818 957691042940 active+remapped+backfill_toofull [2,35,27,1,20,18,19,12,8]p2 [31,23,21,24,35,18,19,33,25]p31 36.82 222362 1998670 954507657022 active+remapped+backfill_toofull [37,22,1,38,11,23,27,32,33]p37 [27,33,0,32,5,25,20,13,15]p27 36.b5 221676 1330056 953443725830 active+remapped+backfill_toofull [6,8,38,12,21,1,39,34,27]p6 [33,8,26,12,3,10,22,34,1]p33 36.b6 222669 1335327 956973704883 active+remapped+backfill_toofull [11,13,41,4,12,34,29,6,1]p11 [2,29,34,4,12,9,15,6,28]p2 36.e0 221518 1772144 952581426388 active+remapped+backfill_toofull [1,27,21,31,30,23,37,13,28]p1 [25,21,14,31,1,2,34,17,24]p25 ceph pg ls | awk 'NR==1 || /backfilling/' | grep -e BYTES -e '\[1' -e ',1,' -e '1\]' | awk '{print $1" "$2" "$4" "$6" "$11" "$15" "$16}' | column -t PG OBJECTS MISPLACED BYTES STATE UP ACTING 36.4a 221508 89144 951346455917 active+remapped+backfilling [40,43,33,32,30,38,22,35,9]p40 [27,10,20,7,30,21,1,28,31]p27 36.79 222315 1111575 955797107713 active+remapped+backfilling [1,36,31,33,25,23,14,3,13]p1 [27,6,31,23,25,5,14,29,13]p27 36.8d 222229 1284156 955234423342 active+remapped+backfilling [35,34,27,37,38,36,43,3,16]p35 [35,34,15,26,1,11,27,18,16]p35 36.ba 222039 0 952547107971 active+remapped+backfilling [0,40,33,23,41,4,27,22,28]p0 [0,35,33,27,1,3,30,22,28]p0 36.da 221607 277464 951599928383 active+remapped+backfilling [21,31,8,9,11,25,36,23,28]p21 [0,10,1,22,33,11,35,15,28]p0 36.db 221685 58816 951420054091 active+remapped+backfilling [3,28,12,13,1,38,40,35,43]p3 [27,20,17,21,1,23,28,24,31]p27 # ceph osd df | sort -nk 17 | tail -n 5 21 hdd 9.09598 1.00000 9.1 TiB 7.7 TiB 7.7 TiB 0 B 31 GiB 1.4 TiB 84.62 1.16 68 up 24 hdd 9.09598 1.00000 9.1 TiB 7.7 TiB 7.7 TiB 1 KiB 25 GiB 1.4 TiB 84.98 1.16 69 up 29 hdd 9.09569 1.00000 9.1 TiB 8.0 TiB 8.0 TiB 72 MiB 23 GiB 1.1 TiB 88.42 1.21 73 up 13 hdd 9.09569 1.00000 9.1 TiB 8.1 TiB 8.1 TiB 1 KiB 22 GiB 1023 GiB 89.02 1.22 76 up 1 hdd 7.27698 1.00000 7.3 TiB 6.8 TiB 6.8 TiB 27 MiB 18 GiB 451 GiB 93.94 1.28 64 up # cat /etc/ceph/ceph.conf | grep full mon_osd_full_ratio = .98 mon_osd_nearfull_ratio = .96 mon_osd_backfillfull_ratio = .97 osd_backfill_full_ratio = .97 osd_failsafe_full_ratio = .99

3 3

Ceph user/bucket usage metrics
by Kushagr Gupta 27 Mar '24

27 Mar '24

Hi team, I am new to ceph and I am looking to monitor the user/bucket usage for ceph. As per the following link: https://docs.ceph.com/en/latest/radosgw/metrics/ But when I enabled the same using the command: 'ceph config set client.rgw CONFIG_VARIABLE VALUE' I ould only see the following perf schema: ``` "rgw": { "req": { "type": 10, "metric_type": "counter", "value_type": "integer", "description": "Requests", "nick": "", "priority": 5, "units": "none" }, "failed_req": { "type": 10, "metric_type": "counter", "value_type": "integer", "description": "Aborted requests", "nick": "", "priority": 5, "units": "none" xxxxxxxxxxxxxxxxxxxSNIPxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx ``` But as per the link we should have also gotten the following metrices: ``` "rgw_op": [ { "labels": {}, "counters": { "put_obj_ops": 2, "put_obj_bytes": 5327, "put_obj_lat": { "avgcount": 2, "sum": 2.818064835, "avgtime": 1.409032417 }, "get_obj_ops": 5, "get_obj_bytes": 5325, "get_obj_lat": { "avgcount": 2, "sum": 0.003000069, "avgtime": 0.001500034 }, ... "list_buckets_ops": 1, "list_buckets_lat": { "avgcount": 1, "sum": 0.002300000, "avgtime": 0.002300000 } } }, ] ``` But as per the following links: https://github.com/ceph/ceph/blob/v19.0.0/src/rgw/rgw_perf_counters.cc https://github.com/ceph/ceph/blob/v18.2.2/src/rgw/rgw_perf_counters.cc I don't think this feature is currently supported could anyone please help me with this? Ceph-version being used by us - 18.2.0(reef)/18.2.2 Thanks and Regards, Kushagra Gupta

1 0

Ha proxy and S3
by Albert Shih 27 Mar '24

27 Mar '24

Hi, If I'm correct in a S3 installation it's good practice to have a HA proxy, I also read somewhere the cephadm tool can deploy the HA Proxy. But is it a good practice to use cephadm to deploy the HA Proxy or it's better do deploy it manually on a other server (who does only that). Regards -- Albert SHIH 🦫 🐸 France Heure locale/Local time: mer. 27 mars 2024 09:18:04 CET

3 2

CephFS space usage
by Thorne Lawler 27 Mar '24

27 Mar '24

Hi everyone! My Ceph cluster (17.2.6) has a CephFS volume which is showing 41TB usage for the data pool, but there are only 5.5TB of files in it. There are fewer than 100 files on the filesystem in total, so where is all that space going? How can I analyze my cephfs to understand what is using that space, and if possible, how can I reclaim that space? Thank you. -- Regards, Thorne Lawler - Senior System Administrator *DDNS* | ABN 76 088 607 265 First registrar certified ISO 27001-2013 Data Security Standard ITGOV40172 P +61 499 449 170 _DDNS /_*Please note:* The information contained in this email message and any attached files may be confidential information, and may also be the subject of legal professional privilege. _If you are not the intended recipient any use, disclosure or copying of this email is unauthorised. _If you received this email in error, please notify Discount Domain Name Services Pty Ltd on 03 9815 6868 to report this matter and delete all copies of this transmission together with any attachments. /

9 27

Ceph versus Zabbix: failure: no data sent
by John Jasen 26 Mar '24

26 Mar '24

If the documentation is to be believed, it's just install the zabbix sender, then; ceph mgr module enable zabbix ceph zabbix config-set zabbix_host my-zabbix-server (Optional) Set the identifier to the fsid. And poof. I should now have a discovered entity on my zabbix server to add templates to. However, this has not worked yet on either of my ceph clusters (one RHEL, one proxmox). Reference: https://docs.ceph.com/en/latest/mgr/zabbix/ On Reddit advice, I installed the Ceph templates for Zabbix. https://raw.githubusercontent.com/ceph/ceph/master/src/pybind/mgr/zabbix/za… Still no dice. No traffic at all seems to be generated, that I've seen from packet traces, ... OK. I su'ed to the ceph user on both clusters, and ran zabbix_send: zabbix_sender -v -z 10.0.0.1 -s "$my_fsid" -k ceph.osd_avg_pgs -o 1 Response from "10.0.0.1:10051": "processed: 1; failed: 0; total: 1; seconds spent: 0.000042" sent: 1; skipped: 0; total: 1 As the ceph user, ceph zabbix send/discovery still fail. I am officially stumped. Any ideas as to which tree I should be barking up? Thanks in advance!

2 2

stretch mode item not defined
by ronny.lippold 26 Mar '24

26 Mar '24

hi there, need some help please. we are planning to replace our rbd-mirror setup and go to stretch mode. the goal is, to have the cluster in 2 fire compartment server rooms. start was a default proxmox/ceph setup. now, i followed the howto from: https://docs.ceph.com/en/latest/rados/operations/stretch-mode/ with the crushmap i ended in an error: in rule 'stretch_rule' item 'dc1' not defined (dc1,2,3 is like site1,2,3 in the docu). my osd tree looks like pve-test02-01:~# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 3.80951 root default -15 0.06349 host pve-test01-01 0 ssd 0.06349 osd.0 up 1.00000 1.00000 -19 0.06349 host pve-test01-02 1 ssd 0.06349 osd.1 up 1.00000 1.00000 -17 0.06349 host pve-test01-03 2 ssd 0.06349 osd.2 up 1.00000 1.00000 ... i think there is something missing. should i need adding the buckets manually? ... for disaster failover, any ideas are welcome. man thanks for help and kind regards, ronny

2 2

CephFS filesystem mount tanks on some nodes?
by Erich Weiler 26 Mar '24

26 Mar '24

Hi All, We have a CephFS filesystem where we are running Reef on the servers (OSD/MDS/MGR/MON) and Quincy on the clients. Every once in a while, one of the clients will stop allowing access to my CephFS filesystem, the error being "permission denied" while try to access the filesystem on that node. The fix is to force unmount the filesystem and remount it, then it's fine again. Any idea how I can prevent this? I see this in the client node logs: Mar 25 11:34:46 phoenix-07 kernel: [50508.354036] ? __touch_cap+0x24/0xd0 [ceph] Mar 25 11:34:46 phoenix-07 kernel: [50508.359650] ? __touch_cap+0x24/0xd0 [ceph] Mar 25 11:34:46 phoenix-07 kernel: [50508.367657] ? __touch_cap+0x24/0xd0 [ceph] Mar 25 11:36:46 phoenix-07 kernel: [50629.189000] ? __touch_cap+0x24/0xd0 [ceph] Mar 25 11:36:46 phoenix-07 kernel: [50629.192579] ? __touch_cap+0x24/0xd0 [ceph] Mar 25 11:36:46 phoenix-07 kernel: [50629.196103] ? __touch_cap+0x24/0xd0 [ceph] Mar 25 11:38:47 phoenix-07 kernel: [50750.024268] ? __touch_cap+0x24/0xd0 [ceph] Mar 25 11:38:47 phoenix-07 kernel: [50750.031520] ? __touch_cap+0x24/0xd0 [ceph] Mar 25 11:38:47 phoenix-07 kernel: [50750.038594] ? __touch_cap+0x24/0xd0 [ceph] Mar 25 11:40:48 phoenix-07 kernel: [50870.853281] ? __touch_cap+0x24/0xd0 [ceph] Mar 25 22:55:38 phoenix-07 kernel: [91360.583032] libceph: mds0 (1)10.50.1.75:6801 socket closed (con state OPEN) Mar 25 22:55:38 phoenix-07 kernel: [91360.667914] libceph: mds0 (1)10.50.1.75:6801 session reset Mar 25 22:55:38 phoenix-07 kernel: [91360.667923] ceph: mds0 closed our session Mar 25 22:55:38 phoenix-07 kernel: [91360.667925] ceph: mds0 reconnect start Mar 25 22:55:52 phoenix-07 kernel: [91374.541614] ceph: mds0 reconnect denied Mar 25 22:55:52 phoenix-07 kernel: [91374.541726] ceph: dropping dirty+flushing Fw state for 00000000ea96c18f 1099683115069 Mar 25 22:55:52 phoenix-07 kernel: [91374.541732] ceph: dropping dirty+flushing Fw state for 00000000ce495f00 1099687100635 Mar 25 22:55:52 phoenix-07 kernel: [91374.541737] ceph: dropping dirty+flushing Fw state for 0000000073ebb190 1099687100636 Mar 25 22:55:52 phoenix-07 kernel: [91374.541744] ceph: dropping dirty+flushing Fw state for 0000000091337e6a 1099687100637 Mar 25 22:55:52 phoenix-07 kernel: [91374.541746] ceph: dropping dirty+flushing Fw state for 000000009075ecd8 1099687100634 Mar 25 22:55:52 phoenix-07 kernel: [91374.541751] ceph: dropping dirty+flushing Fw state for 00000000d1d4c51f 1099687100633 Mar 25 22:55:52 phoenix-07 kernel: [91374.541781] ceph: dropping dirty+flushing Fw state for 0000000063dec1e4 1099687100632 Mar 25 22:55:52 phoenix-07 kernel: [91374.541793] ceph: dropping dirty+flushing Fw state for 000000008b3124db 1099687100638 Mar 25 22:55:52 phoenix-07 kernel: [91374.541796] ceph: dropping dirty+flushing Fw state for 00000000d9e76d8b 1099687100471 Mar 25 22:55:52 phoenix-07 kernel: [91374.541798] ceph: dropping dirty+flushing Fw state for 00000000b57da610 1099685041085 Mar 25 22:55:52 phoenix-07 kernel: [91374.542235] libceph: mds0 (1)10.50.1.75:6801 socket closed (con state V1_CONNECT_MSG) Mar 25 22:55:52 phoenix-07 kernel: [91374.791652] ceph: mds0 rejected session Mar 25 23:01:51 phoenix-07 kernel: [91733.308806] ceph: get_quota_realm: ino (1.fffffffffffffffe) null i_snap_realm Mar 25 23:01:56 phoenix-07 kernel: [91738.182127] ceph: check_quota_exceeded: ino (1000a1cb4a8.fffffffffffffffe) null i_snap_realm Mar 25 23:01:56 phoenix-07 kernel: [91738.188225] ceph: check_quota_exceeded: ino (1000a1cb4a8.fffffffffffffffe) null i_snap_realm Mar 25 23:01:56 phoenix-07 kernel: [91738.233658] ceph: check_quota_exceeded: ino (1000a1cb4aa.fffffffffffffffe) null i_snap_realm Mar 25 23:25:52 phoenix-07 kernel: [93174.787630] libceph: mds0 (1)10.50.1.75:6801 socket closed (con state OPEN) Mar 25 23:39:45 phoenix-07 kernel: [94007.751879] ceph: get_quota_realm: ino (1.fffffffffffffffe) null i_snap_realm Mar 26 00:03:28 phoenix-07 kernel: [95430.158646] ceph: get_quota_realm: ino (1.fffffffffffffffe) null i_snap_realm Mar 26 00:39:45 phoenix-07 kernel: [97607.685421] ceph: get_quota_realm: ino (1.fffffffffffffffe) null i_snap_realm Mar 26 00:43:34 phoenix-07 kernel: [97836.681145] ceph: check_quota_exceeded: ino (1000a306503.fffffffffffffffe) null i_snap_realm Mar 26 00:43:34 phoenix-07 kernel: [97836.686797] ceph: check_quota_exceeded: ino (1000a306503.fffffffffffffffe) null i_snap_realm Mar 26 00:43:34 phoenix-07 kernel: [97836.729046] ceph: check_quota_exceeded: ino (1000a306505.fffffffffffffffe) null i_snap_realm Mar 26 00:49:39 phoenix-07 kernel: [98201.302564] ceph: check_quota_exceeded: ino (1000a75677d.fffffffffffffffe) null i_snap_realm Mar 26 00:49:39 phoenix-07 kernel: [98201.305676] ceph: check_quota_exceeded: ino (1000a75677d.fffffffffffffffe) null i_snap_realm Mar 26 00:49:39 phoenix-07 kernel: [98201.347267] ceph: check_quota_exceeded: ino (1000a755fe3.fffffffffffffffe) null i_snap_realm Mar 26 01:04:49 phoenix-07 kernel: [99111.892854] ceph: get_quota_realm: ino (1.fffffffffffffffe) null i_snap_realm Mar 26 01:39:45 phoenix-07 kernel: [101207.645602] ceph: get_quota_realm: ino (1.fffffffffffffffe) null i_snap_realm Mar 26 02:05:35 phoenix-07 kernel: [102757.494073] ceph: get_quota_realm: ino (1.fffffffffffffffe) null i_snap_realm Mar 26 02:39:45 phoenix-07 kernel: [104807.617467] ceph: get_quota_realm: ino (1.fffffffffffffffe) null i_snap_realm Mar 26 03:03:14 phoenix-07 kernel: [106216.519979] ceph: get_quota_realm: ino (1.fffffffffffffffe) null i_snap_realm Mar 26 03:39:45 phoenix-07 kernel: [108407.731139] ceph: get_quota_realm: ino (1.fffffffffffffffe) null i_snap_realm Mar 26 04:04:05 phoenix-07 kernel: [109867.406047] ceph: get_quota_realm: ino (1.fffffffffffffffe) null i_snap_realm Mar 26 04:39:45 phoenix-07 kernel: [112007.672100] ceph: get_quota_realm: ino (1.fffffffffffffffe) null i_snap_realm Mar 26 05:04:39 phoenix-07 kernel: [113501.101302] ceph: get_quota_realm: ino (1.fffffffffffffffe) null i_snap_realm Mar 26 05:39:45 phoenix-07 kernel: [115607.696806] ceph: get_quota_realm: ino (1.fffffffffffffffe) null i_snap_realm Mar 26 06:05:08 phoenix-07 kernel: [117130.484942] ceph: get_quota_realm: ino (1.fffffffffffffffe) null i_snap_realm Mar 26 06:39:45 phoenix-07 kernel: [119207.706740] ceph: get_quota_realm: ino (1.fffffffffffffffe) null i_snap_realm Mar 26 07:03:33 phoenix-07 kernel: [120635.419910] ceph: get_quota_realm: ino (1.fffffffffffffffe) null i_snap_realm Mar 26 07:39:45 phoenix-07 kernel: [122807.700416] ceph: get_quota_realm: ino (1.fffffffffffffffe) null i_snap_realm Mar 26 08:02:44 phoenix-07 kernel: [124186.804150] ceph: get_quota_realm: ino (1.fffffffffffffffe) null i_snap_realm Mar 26 08:39:45 phoenix-07 kernel: [126407.696256] ceph: get_quota_realm: ino (1.fffffffffffffffe) null i_snap_realm Mar 26 09:02:15 phoenix-07 kernel: [127757.573231] ceph: get_quota_realm: ino (1.fffffffffffffffe) null i_snap_realm Mar 26 09:39:45 phoenix-07 kernel: [130007.718852] ceph: get_quota_realm: ino (1.fffffffffffffffe) null i_snap_realm And after I unmount/remount the filesystem I see this: Mar 26 10:02:57 phoenix-07 kernel: [131399.351230] ceph: get_quota_realm: ino (1.fffffffffffffffe) null i_snap_realm Mar 26 10:04:28 phoenix-07 kernel: [131490.722855] ceph: No path or : separator in source Mar 26 10:04:28 phoenix-07 kernel: [131490.727023] libceph: mon0 (1)10.50.1.74:6789 session established Mar 26 10:04:28 phoenix-07 kernel: [131490.730211] libceph: client102330 fsid 58bde08a-d7ed-11ee-9098-506b4b4da440 Is it possibly losing connection to the MDS servers or something? These lines specifically concern me: Mar 25 22:55:38 phoenix-07 kernel: [91360.667923] ceph: mds0 closed our session Mar 25 22:55:38 phoenix-07 kernel: [91360.667925] ceph: mds0 reconnect start Mar 25 22:55:52 phoenix-07 kernel: [91374.541614] ceph: mds0 reconnect denied It looks like it lost its connection, tried to reconnect and was denied? Does that ring a bell for anyone? The filesystem *is* extremely busy... Thanks again! -erich

1 0

cephfs client not released caps when running rsync
by Nikita Borisenkov 26 Mar '24

26 Mar '24

We transfer data (300 million small files) using rsync between cephfs from version 12.2.13 to 18.2.1. After about the same time (about 7 hours in this case), copying stops for a minute ``` health:HEALTH_WARN 1 clients failing to advance oldest client/flush tid 1 MDSs report slow metadata IOs 1 MDSs behind on trimming ``` The only interesting messages in the logs are: ``` ceph-mds[640738]: mds.beacon.cephfs.X missed beacon ack from the monitors ``` I watched debugging on the client (destination) via watch -n1 /sys/kernel/debug/ceph/451eea44-d7a0-11ee-9117-b496914b4c02.client32497 ``` item total ------------------------------------------ opened files / total inodes 1 / 866757 pinned i_caps / total inodes 866757 / 866757 opened inodes / total inodes 1 / 866757 item total avg_lat(us) min_lat(us) max_lat(us) stdev(us) -------------------------------------------------- --------------------------------- read 0 0 0 0 0 write 5129252 194689 9080 54693415 1023 metadata 29045143 670 161 87794124 369 item total miss hit ------------------------------------------------- d_lease 5361 3026 315488442 caps 866757 11 483177478 ``` During the copy, "pinned i_caps / total inodes" are gradually increased until it reaches the value "mds_max_caps_per_client" (default: 1Mi). Then "pinned i_caps / total inodes" begins to decrease to almost 0, at which time HEALTH_WARN appears and transfer stops. "op/s wr" increases from 200 to 1.5k. Then total inodes begin to increase again along with the resumption of copying and the cluster goes into the HEALTHY state. Mount options: /mnt/cephfs-old 10.77.12.90:6789,10.77.12.91:6789,10.77.12.92:6789:/ ceph rw,noatime,nodiratime,name=admin,secret=<hidden>,acl /mnt/cephfs-new 10.77.12.139:6789,10.77.12.140:6789,10.77.12.141:6789:/ ceph rw,noatime,nodiratime,name=admin,secret=<hidden>,acl,caps_max=10000 Client properties on the MDS server (removed unnecessary): ceph daemon mds.cephfs.X client ls 32497 [ { ... "id": 32497, "state": "open", "num_leases": 0, "num_caps": 980679, "request_load_avg": 7913, "requests_in_flight": 466, "num_completed_flushes": 464, "recall_caps": { "value": 0, "halflife": 60 }, "release_caps": { "value": 1732.2552002208533, "halflife": 60 }, "recall_caps_throttle": { "value": 0, "halflife": 1.35000000000000001 }, "recall_caps_throttle2o": { "value": 0, "halflife": 0.5 }, "session_cache_liveness": { "value": 42186.620275326415, "halflife": 300 }, "cap_acquisition": { "value": 0, "halflife": 30 }, ... } ] ceph daemonperf mds.cephfs.X ``` -------------------------------------------mds------ ------------------------------------- --mds_cache--- ------mds_log ------------ -mds_mem- -------mds_server------- mds_ -----objecter------ purg req rlat slr fwd inos caps exi imi hifc crev cgra ctru cfsa cfa hcc hccd hccr prcr|stry recy recd|subm evts segs repl|ino dn |hcr hcs hsr cre cat |sess|actv rd wr rdwr|purg| 114 0 0 0 1.9M 438k 0 0 0 0 253 0 0 0 0 0 0 59 | 0 0 0 |128 123k 129 0 |1.3M 1.9M|114 0 0 0 0 | 3 | 0 0 440 0 | 0 101 0 0 0 1.9M 438k 0 0 0 0 0 0 0 0 0 0 0 53 | 0 0 0 |106 123k 129 0 |1.3M 1.9M|101 0 0 0 0 | 3 | 0 0 0 0 | 0 ... ``` - from this output it is clear that the client does not send cap release at all (column hccr, decoding of the columns "ceph daemon mds.X perf schema") Then I was able to find the right query to google similar problems: https://www.spinics.net/lists/ceph-users/msg50573.html https://ceph-users.ceph.narkive.com/mcyPtEyz/rsync-kernel-client-cepfs-mkst… https://www.spinics.net/lists/ceph-users/msg50158.html https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/B7K6B5VXM3I… It turns out that the problem is in rsync, in the way it works? The only "solution" is to do it on the client according to a schedule (or upon reaching a certain number of open caps) “echo 2 > /proc/sys/vm/drop_caches”. After this command, the cephfs client releases the cached caps. And if there were a lot of them, then MDS becomes slow again. We also tried to mount cephfs with the option "caps_max=10000" so that the client would do a forced release when the specified value is reached, but this did not help. We can limit mds_max_caps_per_client (not tested), but this also affects all clients at once. The command "ceph daemon mds.cephfs.X cache drop" (with or without an additional parameter) does not help Tested on Linux kernels (client side): 5.10 and 6.1 Did I understand everything correctly? is this the expected behavior when running rsync? And one more problem (I don’t know if it’s related or not), when rsync finishes copying, all caps are freed except the last two (pinned i_caps / total inodes 2 / 2) At this moment a warning appears (or remains after releasing a large number of caps): 1 clients failing to advance oldest client/flush tid But then it doesn't disappear. I waited 12 hours. Warning disappears only after executing the "sync" command on the client. and in the client metrics you can see "pinned i_caps / total inodes 1 / 1" Note: running "echo 2 > /proc/sys/vm/drop_caches" does not help in this case.

2 1

Cephadm on mixed architecture hosts
by Iain Stott 26 Mar '24

26 Mar '24

Hi, We are trying to deploy Ceph Reef 18.2.1 using cephadm on mixed architecture hosts using x86_64 for the mons and aarch64 for the OSDs. During deployment we use the following config for the bootstrap process, where $REPOSITORY is our docker repo. [global] container_image = $REPOSITORY/ceph/ceph:v18.2.1 [mgr] mgr/cephadm/container_image_base = $REPOSITORY/ceph/ceph:v18.2.1 mgr/cephadm/container_image_prometheus = $REPOSITORY/ceph/prometheus:v2.33.4 mgr/cephadm/container_image_node_exporter = $REPOSITORY/ceph/node-exporter:v1.3.1 mgr/cephadm/container_image_grafana = $REPOSITORY/ceph/ceph-grafana:8.3.5 mgr/cephadm/container_image_alertmanager = $REPOSITORY/ceph/alertmanager:v0.23.0 [osd] container_image = $REPOSITORY/ceph/ceph:v18.2.1 Once the bootstrap process is complete, if we do a ceph config dump, the container image for global and osd changes from version tag to sha reference, meaning that when deploying the containers on the OSDs they try using the amd64 container image and not the aarch64 image and fail deployment. Is there a config setting we are missing or a workaround for this? Thanks Iain Iain Stott OpenStack Engineer Iain.Stott(a)thg.com [THG Ingenuity Logo]<https://www.thg.com> www.thg.com<https://www.thg.com/> [LinkedIn]<https://www.linkedin.com/company/thgplc/?originalSubdomain=uk> [Instagram] <https://www.instagram.com/thg> [X] <https://twitter.com/thgplc?lang=en>

3 3

Clients failing to advance oldest client?
by Erich Weiler 26 Mar '24

26 Mar '24

Hi Y'all, I'm seeing this warning via 'ceph -s' (this is on Reef): # ceph -s cluster: id: 58bde08a-d7ed-11ee-9098-506b4b4da440 health: HEALTH_WARN 3 clients failing to advance oldest client/flush tid 1 MDSs report slow requests 1 MDSs behind on trimming services: mon: 5 daemons, quorum pr-md-01,pr-md-02,pr-store-01,pr-store-02,pr-md-03 (age 3d) mgr: pr-md-01.jemmdf(active, since 3w), standbys: pr-md-02.emffhz mds: 1/1 daemons up, 1 standby osd: 46 osds: 46 up (since 3d), 46 in (since 2w) data: volumes: 1/1 healthy pools: 4 pools, 1313 pgs objects: 258.13M objects, 454 TiB usage: 688 TiB used, 441 TiB / 1.1 PiB avail pgs: 1303 active+clean 8 active+clean+scrubbing 2 active+clean+scrubbing+deep io: client: 131 MiB/s rd, 111 MiB/s wr, 41 op/s rd, 613 op/s wr I googled around and looked at the docs and it seems like this isn't a critical problem, but I couldn't find a clear path to resolution. Does anyone have any advice on what I can do to resolve the health issues up top? My CephFS filesystem is incredibly busy so I have a feeling that has some impact here, but not 100% sure... Thanks as always for the help! cheers, erich

3 5

How can I set osd fast shutdown = true
by Suyash Dongre 26 Mar '24

26 Mar '24

Hello, I want to set osd fast shutdown = true, how should I achieve this? Regards, Suyash

2 1

Lot log message from one server
by Albert Shih 26 Mar '24

26 Mar '24

Hi, On my active mgr (only one) on my cluster I got repeatedly something like Mar 26 08:50:04 cthulhu2 conmon[2737]: 2024-03-26T07:50:04.778+0000 7f704bce8700 -1 client.0 error registering admin socket command: (17) File exists Mar 26 08:50:04 cthulhu2 conmon[2737]: 2024-03-26T07:50:04.778+0000 7f704bce8700 -1 client.0 error registering admin socket command: (17) File exists Mar 26 08:50:04 cthulhu2 conmon[2737]: 2024-03-26T07:50:04.778+0000 7f704bce8700 -1 client.0 error registering admin socket command: (17) File exists Mar 26 08:50:04 cthulhu2 conmon[2737]: 2024-03-26T07:50:04.778+0000 7f704bce8700 -1 client.0 error registering admin socket command: (17) File exists Mar 26 08:50:04 cthulhu2 conmon[2737]: 2024-03-26T07:50:04.778+0000 7f704bce8700 -1 client.0 error registering admin socket command: (17) File exists Mar 26 08:55:04 cthulhu2 ceph-mgr[2843]: client.0 error registering admin socket command: (17) File exists Mar 26 08:55:04 cthulhu2 ceph-mgr[2843]: client.0 error registering admin socket command: (17) File exists Mar 26 08:55:04 cthulhu2 ceph-mgr[2843]: client.0 error registering admin socket command: (17) File exists Mar 26 08:55:04 cthulhu2 ceph-mgr[2843]: client.0 error registering admin socket command: (17) File exists Mar 26 08:55:04 cthulhu2 ceph-mgr[2843]: client.0 error registering admin socket command: (17) File exists I check the dashboard (first hit with google on that message) and that seem OK, no config about it listen address. Those messages come each ~ 1 hour. I notice those message appear each time just after Mar 26 08:50:04 cthulhu2 ceph-mgr[2843]: [volumes INFO volumes.module] Starting _cmd_fs_subvolume_ls(prefix:fs subvolume ls, target:['mon-mgr', ''], vol_name:cephfs) < "" but I don't known if that's related. Any clue ? Regards -- Albert SHIH 🦫 🐸 France Heure locale/Local time: mar. 26 mars 2024 10:52:53 CET

1 0

Mounting A RBD Via Kernal Modules
by duluxoz 26 Mar '24

26 Mar '24

Hi All, I'm trying to mount a Ceph Reef (v18.2.2 - latest version) RBD Image as a 2nd HDD on a Rocky Linux v9.3 (latest version) host. The EC pool has been created and initialised and the image has been created. The ceph-common package has been installed on the host. The correct keyring has been added to the host (with a chmod of 600) and the host has been configure with an rbdmap file as follows: `my_pool.meta/my_image id=ceph_user,keyring=/etc/ceph/ceph.client.ceph_user.keyring`. When running the rbdmap.service the image appears as both `/dev/rbd0` and `/dev/rbd/my_pool.meta/my_image`, exactly as the Ceph Doco says it should. So everything *appears* AOK up to this point. My question now is: Should I run `mkfs xfs` on `/dev/rbd0` *before* or *after* I try to mount the image (via fstab: `/dev/rbd/my_pool.meta/my_image /mnt/my_image xfs noauto 0 0` - as per the Ceph doco)? The reason I ask is that I've tried this *both* ways and all I get is an error message (sorry, can't remember the exact messages and I'm not currently in front of the host to confirm it :-) - but from memory it was something about not being able to recognise the 1st block - or something like that). So, I'm obviously doing something wrong, but I can't work out what exactly (and the logs don't show any useful info). Do I, for instance, have the process wrong / don't understand the exact process, or is there something else wrong? All comments/suggestions/etc greatly appreciated - thanks in advance Cheers Dulux-Oz

6 16

Best practice in 2024 for simple RGW failover
by E Taka 26 Mar '24

26 Mar '24

Hi, The requirements are actually not high: 1. there should be a generally known address for access. 2. it should be possible to reboot or shut down a server without the RGW connections being down the entire time. A downtime of a few seconds is OK. Constant load balancing would be nice, but is not necessary. I have found various approaches on the Internet - what is currently recommended for a current Ceph installation? Thanks,

2 1

Ceph object gateway metrics
by Kushagr Gupta 26 Mar '24

26 Mar '24

Hi Team, I have been working on ceph to understand the full extent of its capability.*Ceph version*: 18.25 *OS*: Almalinux 8.8 I am working on a project which uses Ceph object gateway(Buckets). I want the metrics for users and buckets to be shown in prometheus. But when I checked the current metrics being sent to prometheus, there were no required metrics that showed the bucket/user usage of the objects. We are getting other metrics from ceph. Kindly refer to the image: [image: image.png] But the bucket and user required info is not present.(Kindly refer the second image) [image: image.png] I got the following links which indicates that a ceph exporter needs to be installed to expose those metrices. 1. https://github.com/SepehrImanian/s3-ceph-exporter 2. https://github.com/blemmenes/radosgw_usage_exporter 3. https://docs.ceph.com/en/latest/radosgw/metrics/ Could anyone please help me with this? Thanks and regards, Kushagra Gupta

2 1

Quincy/Dashboard: Object Gateway not accessible after applying self-signed cert to rgw service
by stephan.budach＠jvm.com 26 Mar '24

26 Mar '24

Hi, I am running a Ceph cluster and configured RGW for S3, initially w/o SSL. The service works nicely and I updated the service usinfg SSL certs, signed by our own CA, just as I already did for the dashboard itself. However, as soon as I applied the new config, the dashboard wasn't able to access and display the service anymore, while the service itself still works, now using the supplied SSL certificate. The error supplied is: Error 500 The server encountered an unexpected condition which prevented it from fulfilling the request. My guess is, that the dashboard for some reason doesn't like the certificate, the rgw service is providing, despite the fact, that itself is using it. Any hints on how to make dashboard display the Object Gatway again?

1 1

Why you might want packages not containers for Ceph deployments
by Matthew Vernon 25 Mar '24

25 Mar '24

Hi, In the discussion after the Ceph Month talks yesterday, there was a bit of chat about cephadm / containers / packages. IIRC, Sage observed that a common reason in the recent user survey for not using cephadm was that it only worked on containerised deployments. I think he then went on to say that he hadn't heard any compelling reasons why not to use containers, and suggested that resistance was essentially a user education question[0]. I'd like to suggest, briefly, that: * containerised deployments are more complex to manage, and this is not simply a matter of familiarity * reducing the complexity of systems makes admins' lives easier * the trade-off of the pros and cons of containers vs packages is not obvious, and will depend on deployment needs * Ceph users will benefit from both approaches being supported into the future We make extensive use of containers at Sanger, particularly for scientific workflows, and also for bundling some web apps (e.g. Grafana). We've also looked at a number of container runtimes (Docker, singularity, charliecloud). They do have advantages - it's easy to distribute a complex userland in a way that will run on (almost) any target distribution; rapid "cloud" deployment; some separation (via namespaces) of network/users/processes. For what I think of as a 'boring' Ceph deploy (i.e. install on a set of dedicated hardware and then run for a long time), I'm not sure any of these benefits are particularly relevant and/or compelling - Ceph upstream produce Ubuntu .debs and Canonical (via their Ubuntu Cloud Archive) provide .debs of a couple of different Ceph releases per Ubuntu LTS - meaning we can easily separate out OS upgrade from Ceph upgrade. And upgrading the Ceph packages _doesn't_ restart the daemons[1], meaning that we maintain control over restart order during an upgrade. And while we might briefly install packages from a PPA or similar to test a bugfix, we roll those (test-)cluster-wide, rather than trying to run a mixed set of versions on a single cluster - and I understand this single-version approach is best practice. Deployment via containers does bring complexity; some examples we've found at Sanger (not all Ceph-related, which we run from packages): * you now have 2 process supervision points - dockerd and systemd * docker updates (via distribution unattended-upgrades) have an unfortunate habit of rudely restarting everything * docker squats on a chunk of RFC 1918 space (and telling it not to can be a bore), which coincides with our internal network... * there is more friction if you need to look inside containers (particularly if you have a lot running on a host and are trying to find out what's going on) * you typically need to be root to build docker containers (unlike packages) * we already have package deployment infrastructure (which we'll need regardless of deployment choice) We also currently use systemd overrides to tweak some of the Ceph units (e.g. to do some network sanity checks before bringing up an OSD), and have some tools to pair OSD / journal / LVM / disk device up; I think these would be more fiddly in a containerised deployment. I'd accept that fixing these might just be a SMOP[2] on our part. Now none of this is show-stopping, and I am most definitely not saying "don't ship containers". But I think there is added complexity to your deployment from going the containers route, and that is not simply a "learn how to use containers" learning curve. I do think it is reasonable for an admin to want to reduce the complexity of what they're dealing with - after all, much of my job is trying to automate or simplify the management of complex systems! I can see from a software maintainer's point of view that just building one container and shipping it everywhere is easier than building packages for a number of different distributions (one of my other hats is a Debian developer, and I have a bunch of machinery for doing this sort of thing). But it would be a bit unfortunate if the general thrust of "let's make Ceph easier to set up and manage" was somewhat derailed with "you must use containers, even if they make your life harder". I'm not going to criticise anyone who decides to use a container-based deployment (and I'm sure there are plenty of setups where it's an obvious win), but if I were advising someone who wanted to set up and use a 'boring' Ceph cluster for the medium term, I'd still advise on using packages. I don't think this makes me a luddite :) Regards, and apologies for the wall of text, Matthew [0] I think that's a fair summary! [1] This hasn't always been true... [2] Simple (sic.) Matter of Programming -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.

26 65

Large number of misplaced PGs but little backfill going on
by Torkil Svensgaard 25 Mar '24

25 Mar '24

Hi We have this after adding some hosts and changing crush failure domain to datacenter: pgs: 1338512379/3162732055 objects misplaced (42.321%) 5970 active+remapped+backfill_wait 4853 active+clean 11 active+remapped+backfilling We have 3 datacenters each with 6 hosts and ~400 HDD OSDs with DB/WAL on NVMe. Using mclock with high_recovery_ops profile. What is the bottleneck here? I would have expected a huge number of simultaneous backfills. Backfill reservation logjam? Mvh. Torkil -- Torkil Svensgaard Systems Administrator Danish Research Centre for Magnetic Resonance DRCMR, Section 714 Copenhagen University Hospital Amager and Hvidovre Kettegaard Allé 30, 2650 Hvidovre, Denmark

5 25

Spam in log file
by Albert Shih 25 Mar '24

25 Mar '24

Hi everyone. On my cluster I got spam by my cluster with message like Mar 25 13:10:13 cthulhu2 ceph-mgr[2843]: mgr finish mon failed to return metadata for mds.cephfs.cthulhu2.dqahyt: (2) No such file or directory Mar 25 13:10:13 cthulhu2 ceph-mgr[2843]: mgr finish mon failed to return metadata for mds.cephfs.cthulhu3.xvboir: (2) No such file or directory Mar 25 13:10:13 cthulhu2 ceph-mgr[2843]: mgr finish mon failed to return metadata for mds.cephfs.cthulhu5.kwmyyg: (2) No such file or directory I got 5 server for the service (cthulhu 1->5) and indeed when from cthulhu1 (or 2) I try : something: root@cthulhu2:/etc/ceph# ceph mds metadata cephfs.cthulhu2.dqahyt {} Error ENOENT: root@cthulhu2: but that works on 1 or 4 root@cthulhu2:/etc/ceph# ceph mds metadata cephfs.cthulhu1.sikvjf { "addr": "[v2:145.238.187.184:6800/1315478297,v1:145.238.187.184:6801/1315478297]", "arch": "x86_64", "ceph_release": "quincy", "ceph_version": "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)", "ceph_version_short": "17.2.7", "container_hostname": "cthulhu1", "container_image": "quay.io/ceph/ceph@sha256:62465e744a80832bde6a57120d3ba076613e8a19884b274f9cc82580e249f6e1", "cpu": "Intel(R) Xeon(R) Silver 4310 CPU @ 2.10GHz", "distro": "centos", "distro_description": "CentOS Stream 8", "distro_version": "8", "hostname": "cthulhu1", "kernel_description": "#1 SMP Debian 5.10.209-2 (2024-01-31)", "kernel_version": "5.10.0-28-amd64", "mem_swap_kb": "16777212", "mem_total_kb": "263803496", "os": "Linux" } root@cthulhu2:/etc/ceph# I check the caps and don't see anything special. I got also (I don't know if it's related) those message : Mar 25 13:18:38 cthulhu2 ceph-mgr[2843]: mgr.server handle_open ignoring open from mds.cephfs.cthulhu2.dqahyt v2:145.238.187.185:6800/2763465960; not ready for session (expect reconnect) Mar 25 13:18:38 cthulhu2 ceph-mgr[2843]: mgr.server handle_open ignoring open from mds.cephfs.cthulhu3.xvboir v2:145.238.187.186:6800/1297104944; not ready for session (expect reconnect) Mar 25 13:18:38 cthulhu2 ceph-mgr[2843]: mgr.server handle_open ignoring open from mds.cephfs.cthulhu5.kwmyyg v2:145.238.187.188:6800/449122091; not ready for session (expect reconnect) Mar 25 13:18:39 cthulhu2 ceph-mgr[2843]: mgr.server handle_open ignoring open from mds.cephfs.cthulhu3.xvboir v2:145.238.187.186:6800/1297104944; not ready for session (expect reconnect) Mar 25 13:18:39 cthulhu2 ceph-mgr[2843]: mgr.server handle_open ignoring open from mds.cephfs.cthulhu2.dqahyt v2:145.238.187.185:6800/2763465960; not ready for session (expect reconnect) Mar 25 13:18:39 cthulhu2 ceph-mgr[2843]: mgr.server handle_open ignoring open from mds.cephfs.cthulhu5.kwmyyg v2:145.238.187.188:6800/449122091; not ready for session (expect reconnect) Regards. -- Albert SHIH 🦫 🐸 France Heure locale/Local time: lun. 25 mars 2024 13:08:33 CET

2 3

March Ceph Science Virtual User Group
by Kevin Hrpcek 25 Mar '24

25 Mar '24

Hey All, We will be having a Ceph science/research/big cluster call on Wednesday March 27th. If anyone wants to discuss something specific they can add it to the pad linked below. If you have questions or comments you can contact me. This is an informal open call of community members mostly from hpc/htc/research/big cluster environments (though anyone is welcome) where we discuss whatever is on our minds regarding ceph. Updates, outages, features, maintenance, etc...there is no set presenter but I do attempt to keep the conversation lively. Pad URL: https://pad.ceph.com/p/Ceph_Science_User_Group_20240327 Virtual event details: March 27, 2024 14:00 UTC 3pm Central European 9am Central US Main pad for discussions: https://pad.ceph.com/p/Ceph_Science_User_Group_Index Meetings will be recorded and posted to the Ceph Youtube channel. To join the meeting on a computer or mobile phone: https://meet.jit.si/ceph-science-wg Kevin -- Kevin Hrpcek NASA VIIRS Atmosphere SIPS/TROPICS Space Science & Engineering Center University of Wisconsin-Madison

1 0

ceph RGW reply "ERROR: S3 error: 404 (NoSuchKey)" but rgw object metadata exist
by xuchenhuig＠gmail.com 25 Mar '24

25 Mar '24

Hi, My ceph cluster has 9 nodes for Ceph Object Store. Recently, I have experienced data loss that reply 404 (NoSuchKey) by s3cmd get xxx command. However, I can get metadata info by s3cmd ls xxx. The RGW object size is above 1GB that have many multipart object. Commanding 'rados -p default.rgw.buckets.data stats object' show that it only have head object, all of multipart and shadow part have gone. The bucket data only support write and read operation, no delete, and has no lifecycle policy. I have found similar problem in https://tracker.ceph.com/issues/47866 that had repaired in v16.0.0. Maybe this is new data loss problem that very serious for us. ceph version: 16.2.5 #command info: s3cmd ls s3://solr-scrapy.commoncrawl-warc/batch_2024031314/Scrapy/main/CC-MAIN-20200118052321-20200118080321-00547.warc.gz 2024-03-13 09:27 1208269953 s3://solr-scrapy.commoncrawl-warc/batch_2024031314/Scrapy/main/CC-MAIN-20200118052321-20200118080321-00547.warc.gz s3cmd get s3://solr-scrapy.commoncrawl-warc/batch_2024031314/Scrapy/main/CC-MAIN-20200118052321-20200118080321-00547.warc.gz download: 's3://solr-scrapy.commoncrawl-warc/batch_2024031314/Scrapy/main/CC-MAIN-20200118052321-20200118080321-00547.warc.gz' -> './CC-MAIN-20200118052321-20200118080321- 00547.warc.gz' [1 of 1] ERROR: Download of './CC-MAIN-20200118052321-20200118080321-00547.warc.gz' failed (Reason: 404 (NoSuchKey)) ERROR: S3 error: 404 (NoSuchKey) # head exist and size is 0, multipart and shadow had lost rados -p default.rgw.buckets.data stat df8c0fe6-01c8-4c07-b310-2d102356c004.76248.1__multipart_batch_2024031314/Scrapy/main/CC-MAIN-20200118052321-20200118080321-00547.warc.gz.2~C2M72EJLHrNe_fnHnifS4N7pw70hVmE.1 error stat-ing eck6m2.rgw.buckets.data/df8c0fe6-01c8-4c07-b310-2d102356c004.76248.1__multipart_batch_2024031314/Scrapy/main/CC-MAIN-20200118052321-20200118080321-00547.warc.gz.2~C2M72EJLHrNe_fnHnifS4N7pw70hVmE.1: (2) No such file or directory thanks.

1 0

ceph cluster extremely unbalanced
by Denis Polom 25 Mar '24

25 Mar '24

Hi guys, recently I took over a care of Ceph cluster that is extremely unbalanced. Cluster is running on Quincy 17.2.7 (upgraded Nautilus -> Octopus -> Quincy) and has 1428 OSDs (HDDs). We are running CephFS on it. Crush failure domain is datacenter (there are 3), data pool is EC 3+3. This cluster had and has balancer disabled for years. And was "balanced" manually by changing OSDs crush weights. So now it is complete mess and I would like to change it to have OSDs crush weight same (3.63898) and to enable balancer with upmap. From `ceph osd df ` sorted from the least used to most used OSDs: ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS MIN/MAX VAR: 0.76/1.16 STDDEV: 5.97 TOTAL 5.1 PiB 3.7 PiB 3.7 PiB 2.9 MiB 8.5 TiB 1.5 PiB 71.50 428 hdd 3.63898 1.00000 3.6 TiB 2.0 TiB 2.0 TiB 1 KiB 5.6 GiB 1.7 TiB 54.55 0.76 96 up 223 hdd 3.63898 1.00000 3.6 TiB 2.0 TiB 2.0 TiB 3 KiB 5.6 GiB 1.7 TiB 54.58 0.76 95 up ... ... ... 591 hdd 3.53999 1.00000 3.6 TiB 3.0 TiB 3.0 TiB 1 KiB 7.0 GiB 680 GiB 81.74 1.14 125 up 832 hdd 3.59999 1.00000 3.6 TiB 3.0 TiB 3.0 TiB 4 KiB 6.9 GiB 680 GiB 81.75 1.14 114 up 248 hdd 3.63898 1.00000 3.6 TiB 3.0 TiB 3.0 TiB 3 KiB 7.2 GiB 646 GiB 82.67 1.16 121 up 559 hdd 3.63799 1.00000 3.6 TiB 3.0 TiB 3.0 TiB 0 B 7.0 GiB 644 GiB 82.70 1.16 123 up TOTAL 5.1 PiB 3.7 PiB 3.6 PiB 2.9 MiB 8.5 TiB 1.5 PiB 71.50 MIN/MAX VAR: 0.76/1.16 STDDEV: 5.97 crush rule: { "rule_id": 10, "rule_name": "ec33hdd_rule", "type": 3, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -2, "item_name": "default~hdd" }, { "op": "choose_indep", "num": 3, "type": "datacenter" }, { "op": "choose_indep", "num": 2, "type": "osd" }, { "op": "emit" } ] } My question is what would be proper and most safer way to make it happen. * should I first enable balancer and let it do its work and after that change the OSDs crush weights to be even? * or should it otherwise - first to make crush weights even and then enable the balancer? * or is there another safe(r) way? What are the ideal balancer settings for that? I'm expecting a large data movement, and this is production cluster. I'm also afraid that during the balancing or changing crush weights some OSDs become full. I've tried that already and had to move some PGs manually to another OSDs in the same failure domain. I would appreciate any suggestion on that. Thank you!

3 4

el7 + nautilus rbd snapshot map + lvs mount crash
by Marc 24 Mar '24

24 Mar '24

Looks like this procedure crashes the ceph node. Tried this now for 2nd time after updating and again crash. el7 + nautilus -> rbd snapshot map -> lvs mount -> crash (lvs are not even duplicate names)

1 0

log_latency slow operation observed for submit_transact, latency = 22.644258499s
by Torkil Svensgaard 24 Mar '24

24 Mar '24

Good morning, Cephadm Reef 18.2.1. We recently added 4 hosts and changed a failure domain from host to datacenter which is the reason for the large misplaced percentage. We were seeing some pretty crazy spikes in "OSD Read Latencies" and "OSD Write Latencies" on the dashboard. Most of the time everything is well but then for periods of time, 1-4 hours, latencies will go to 10+ seconds for one or more OSDs. This also happens outside scrub hours and it is not the same OSDs every time. The OSDs affected are HDD with DB/WAL on NVMe. Log snippet: " ... 2024-03-22T06:48:22.859+0000 7fb184b52700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fb169898700' had timed out after 15.000000954s 2024-03-22T06:48:22.859+0000 7fb185b54700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fb169898700' had timed out after 15.000000954s 2024-03-22T06:48:22.864+0000 7fb169898700 1 heartbeat_map clear_timeout 'OSD::osd_op_tp thread 0x7fb169898700' had timed out after 15.000000954s 2024-03-22T06:48:22.864+0000 7fb169898700 0 bluestore(/var/lib/ceph/osd/ceph-112) log_latency slow operation observed for submit_transact, latency = 17.716707230s 2024-03-22T06:48:22.880+0000 7fb1748ae700 0 bluestore(/var/lib/ceph/osd/ceph-112) log_latency_fn slow operation observed for _txc_committed_kv, latency = 17.732601166s, txc = 0x55a5bcda0f00 2024-03-22T06:48:38.077+0000 7fb184b52700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fb169898700' had timed out after 15.000000954s 2024-03-22T06:48:38.077+0000 7fb184b52700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fb169898700' had timed out after 15.000000954s ... " " [root@dopey ~]# ceph -s cluster: id: 8ee2d228-ed21-4580-8bbf-0649f229e21d health: HEALTH_WARN 1 failed cephadm daemon(s) Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull services: mon: 5 daemons, quorum lazy,jolly,happy,dopey,sleepy (age 3d) mgr: jolly.tpgixt(active, since 10d), standbys: dopey.lxajvk, lazy.xuhetq mds: 1/1 daemons up, 2 standby osd: 540 osds: 539 up (since 6m), 539 in (since 15h); 6250 remapped pgs data: volumes: 1/1 healthy pools: 15 pools, 10849 pgs objects: 546.35M objects, 1.1 PiB usage: 1.9 PiB used, 2.3 PiB / 4.2 PiB avail pgs: 1425479651/3163081036 objects misplaced (45.066%) 6224 active+remapped+backfill_wait 4516 active+clean 67 active+clean+scrubbing 25 active+remapped+backfilling 16 active+clean+scrubbing+deep 1 active+remapped+backfill_wait+backfill_toofull io: client: 117 MiB/s rd, 68 MiB/s wr, 274 op/s rd, 183 op/s wr recovery: 438 MiB/s, 192 objects/s " Anyone know what the issue might be? Given that is happens on and off with large periods of time in between with normal low latencies I think it unlikely that it is just because the cluster is busy. Also, how come there's only a small amount of PGs doing backfill when we have such a large misplaced percentage? Can this be just from backfill reservation logjam? Mvh. Torkil -- Torkil Svensgaard Systems Administrator Danish Research Centre for Magnetic Resonance DRCMR, Section 714 Copenhagen University Hospital Amager and Hvidovre Kettegaard Allé 30, 2650 Hvidovre, Denmark

4 6

Laptop Losing Connectivity To CephFS On Sleep/Hibernation
by duluxoz 23 Mar '24

23 Mar '24

Hi All, I'm looking for some help/advice to solve the issue outlined in the heading. I'm running CepfFS (name: cephfs) on a Ceph Reef (v18.2.2 - latest update) cluster, connecting from a laptop running Rocky Linux v9.3 (latest update) with KDE v5 (latest update). I've set up the laptop to connect to a number of directories on CephFS via the `/etc/fstab' folder, an example of such is: `ceph_user@.cephfs=/my_folder /mnt/my_folder ceph noatime,_netdev 0 0`. Everything is working great; the required Ceph Key is on the laptop (with a chmod of 600), I can access the files on the Ceph Cluster, etc, etc, etc - all good. However, whenever the laptop is in sleep or hibernate mode (ie when I close the laptop's lid) and then bring the laptop out of sleep/hibernation (ie I open the laptop's lid) I've lost the CephFS mountings. The only way to bring them back is to run `mount -a` as root (or sudo). This is, as I'm sure you'll agree, not a long-term viable options - especially as this is a running as a pilot-project and the eventual end-users won't have access to root/sudo. So I'm seeking the collective wisdom of the community in how to solve this issue. I've taken a brief look at autofs, and even half-heartedly had a go at configuring it, but it didn't seem to work - honestly, it was late and I wanted to get home after a long day. :-) Is this the solution to my issue, or is there a better way to construct the fstab entries, or is there another solution I haven't found yet in the doco or via google-foo? All help and advice greatly appreciated - thanks in advance Cheers Dulux-Oz

3 6

Reef (18.2): Some PG not scrubbed/deep scrubbed for 1 month
by Michel Jouvin 23 Mar '24

23 Mar '24

Hi, We have a Reef cluster that started to complain a couple of weeks ago about ~20 PGs (over 10K) not scrubbed/deep-scrubbed in time. Looking at it since a few days, I saw this affect only those PGs that could not be scrubbed since mid-February. Old the other PGs are regularly scrubbed. I decided to look if one OSD was present in all these PGs and found one! I restarted this OSD but it had no effect. Looking at the logs for the suspect OSD, I found nothing related to abnormal behaviour (but the log is very verbose at restart time so easy to miss something...). And there is no error associated with the OSD disk. Any advice about where to look for some useful information would be appreciated! Should I try to destroy the OSD and readd it? I'll be more confortable if I was able to find some diagnostics before... Best regards, Michel

9 24

How you manage log
by Albert Shih 22 Mar '24

22 Mar '24

Hi, With our small cluster (11 nodes) I notice ceph log a lot Beside to keep that somewhere «just in case», is they are anything to check regularly in the log (in prevention of more serious problem) ? Or can we trust «ceph health» and use the log only for debug. Regards -- Albert SHIH 🦫 🐸 France Heure locale/Local time: ven. 22 mars 2024 22:28:42 CET

1 0

High OSD commit_latency after kernel upgrade
by Özkan Göksu 22 Mar '24

22 Mar '24

Hello! After upgrading "5.15.0-84-generic" to "5.15.0-100-generic" (Ubuntu 22.04.2 LTS) , commit latency started acting weird with "CT4000MX500SSD" drives. osd commit_latency(ms) apply_latency(ms) 36 867 867 37 3045 3045 38 15 15 39 18 18 42 1409 1409 43 1224 1224 I downgraded the kernel but the result did not change. I have a similar build and it didn't get upgraded and it is just fine. While I was digging I realised a difference. This is high latency cluster and as you can see the "DISC-GRAN=0B", "DISC-MAX=0B" root@sd-01:~# lsblk -D NAME DISC-ALN DISC-GRAN DISC-MAX DISC-ZERO sdc 0 0B 0B 0 ├─ceph--76b7d255--2a01--4bd4--8d3e--880190181183-osd--block--201d5050--db0c--41b4--85c4--6416ee989d6c │ 0 0B 0B 0 └─ceph--76b7d255--2a01--4bd4--8d3e--880190181183-osd--block--5a376133--47de--4e29--9b75--2314665c2862 root@sd-01:~# find /sys/ -name provisioning_mode -exec grep -H . {} + | sort /sys/devices/pci0000:80/0000:80:03.0/0000:81:00.0/host0/port-0:0/end_device-0:0/target0:0:0/0:0:0:0/scsi_disk/0:0:0:0/provisioning_mode:full ------------------------------------------------------------------------------------------ This is low latency cluster and as you can see the "DISC-GRAN=4K", "DISC-MAX=2G" root@ud-01:~# lsblk -D NAME DISC-ALN DISC-GRAN DISC-MAX DISC-ZERO sdc 0 4K 2G 0 ├─ceph--7496095f--18c7--41fd--90f2--d9b3e382bc8e-osd--block--ec86a029--23f7--4328--9600--a24a290e3003 │ 0 4K 2G 0 └─ceph--7496095f--18c7--41fd--90f2--d9b3e382bc8e-osd--block--5b69b748--d899--4f55--afc3--2ea3c8a05ca1 root@ud-01:~# find /sys/ -name provisioning_mode -exec grep -H . {} + | sort /sys/devices/pci0000:00/0000:00:11.4/ata3/host2/target2:0:0/2:0:0:0/scsi_disk/2:0:0:0/provisioning_mode:writesame_16 I think the problem is related to provisioning_mode but I really did not understand the reason. I boot with a live iso and still the drive was "provisioning_mode:full" so it means this is not related to my OS at all. With the upgrade something changed and I think during boot sequence negotiation between LSI controller, drives and kernel started to assign "provisioning_mode:full" but I'm not sure. What should I do ? Best regards.

2 5

Ceph fs understand usage
by Marcus 22 Mar '24

22 Mar '24

Hi all, I have setup a test cluster with 3 servers, Everything has default values with a replication of 3. I have created one volume called gds-common and the data pool has been configured with compression lz4 and compression_mode aggressive. I have copied 71TB data to this volume but I can not get my head around usage information on the cluster. Most of this data is quite small files that contain plain text, so I expect the compression rate to be quite good. With both the data storage where I copy from and the ceph fs mounted a df -h gives: urd-gds-031:/gds-common 163T 71T 92T 44% /gds-common 10.10.100.0:6789,10.10.100.1:6789,10.10.100.2:6789:/ 92T 68T 25T 74% /ceph-gds-common Looking at this, the compression rate do not seem to be that good, or is the used column showing an uncompressed value? Using ceph and command ceph fs df detail: --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 262 TiB 94 TiB 168 TiB 168 TiB 64.10 TOTAL 262 TiB 94 TiB 168 TiB 168 TiB 64.10 --- POOLS --- POOL ID PGS STORED (DATA) (OMAP) OBJECTS USED (DATA) (OMAP) %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR .mgr 1 1 24 MiB 24 MiB 0 B 8 73 MiB 73 MiB 0 B 0 25 TiB N/A N/A N/A 0 B 0 B gds-common_data 2 1024 67 TiB 67 TiB 0 B 23.31M 167 TiB 167 TiB 0 B 69.43 25 TiB N/A N/A N/A 35 TiB 70 TiB gds-common_metadata 3 32 4.0 GiB 251 MiB 3.8 GiB 680.88k 12 GiB 753 MiB 11 GiB 0.02 25 TiB N/A N/A N/A 0 B 0 B .rgw.root 4 32 1.4 KiB 1.4 KiB 0 B 4 48 KiB 48 KiB 0 B 0 25 TiB N/A N/A N/A 0 B 0 B default.rgw.log 5 32 182 B 182 B 0 B 2 24 KiB 24 KiB 0 B 0 25 TiB N/A N/A N/A 0 B 0 B default.rgw.control 6 32 0 B 0 B 0 B 7 0 B 0 B 0 B 0 25 TiB N/A N/A N/A 0 B 0 B default.rgw.meta 7 32 0 B 0 B 0 B 0 0 B 0 B 0 B 0 25 TiB N/A N/A N/A 0 B 0 B From my understanding the raw storage used contain all the 3 copies so this means 56TB per copy and gives an compression of about 20% if this is a compressed value? Looking at the pool gds-common_data value STORED 67TB is an uncompressed value and a value per copy, right? The used value from gds-common_data is the raw usage of all 3 copies, right? The %RAW USED value make sense (64.10) but the gds-common_data %USED differs (69.43) and I can not figure out what this value relates to? UNDER COMPR is the amount of data that ceph has recognized that it can be used in compression (70TB) so it is about all the data. I did not understand the value USED COMPR (35TB), do this specify how much it has been compressed, so 70TB has been compressed to 35TB? But what values are specified as compressed and what values shows the raw uncompressed values? Are all values uncompressed values and the only place I see compression is "USED COMPR" and "UNDER COMPR"? But when do I run out of storage in my cluster then and what value should I keep my eyes on if %used is calculated on uncompressed data? Does this mean that I have more storage available then shown from %USED? Does df -h on a mount shows the uncompressed used value? Then we have mon_osd_full_ratio does this mean that the first osd that reaches .95 full (default) make the system stop the clients write aso? But does this mon_osd_full_ratio always reaches its limit before %RAW USAGE reaches 100% or pool %USED reaches 100% or what does happen if one of the used values reaches 100% before mon_osd_full_ratio? I am sorry for all the questions but even after reading the documentaion I do not seem to be able to figure this out. All help is appreciated. Many thanks in advance! Best regards Marcus

1 0

Return value from cephadm host-maintenance?
by Daniel Brown 21 Mar '24

21 Mar '24

Possibly a naive question, and possibly seemingly trivial, but is there any good reason to return a “1” on success for cephadm host-maintenance enter and exit: ~$ sudo cephadm host-maintenance enter --fsid XXXX-XXXXXX-XXXX-XXXXX Inferring config /var/lib/ceph/XXXX-XXXXXX-XXXX-XXXXXconfig/ceph.conf Requested to place host into maintenance success - systemd target ceph-XXXX-XXXXXX-XXXX-XXXXX.target disabled ~$ echo $? 1 ~$ sudo cephadm host-maintenance exit --fsidXXXX-XXXXXX-XXXX-XXXXX Inferring config /var/lib/ceph/XXXX-XXXXXX-XXXX-XXXXX /config/ceph.conf Requested to exit maintenance state success - systemd target ceph-XXXX-XXXXXX-XXXX-XXXXX .target enabled and started ~$ echo $? 1

2 3

mon stuck in probing
by faicker mo 21 Mar '24

21 Mar '24

Hello, The problem is a mon stucked in probing state. The env is ceph 18.2.1 on ubuntu22.04 with rdma, 5 mons. One mon memb4 is out of quorum. The debug log is attached. Thanks.

2 4

Upgrading from Reef v18.2.1 to v18.2.2
by Michael Worsham 21 Mar '24

21 Mar '24

I originally used Cephadm to build my sandbox Ceph cluster (Reef v18.2.1) using Cephadm and Ansible. It's stable and works fine. Now that Reef v18.2.2 has come out, is there a set of instructions on how to upgrade to the latest version via using Cephadm? -- Michael This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments.

3 2

OSD does not die when disk has failures
by Daniel Schreiber 21 Mar '24

21 Mar '24

Hi, in our cluster (17.2.6) disks fail from time to time. Block devices are HDD, DB devices are NVME. However, the OSD process does not reliably die. That leads to blocked client IO for all requests for which the OSD with the broken disk is the primary OSD. All pools on these OSDs are EC pools (cephfs data or rbd data). Client IO recovers if I manually stop the OSD. It seems like the error was triggered during deep scrub, because the cluster reported scrub errors afterwards. OSD Log: 2024-03-11T20:12:43+01:00 urzceph1-osd05 bash[9695]: debug 2024-03-11T19:12:43.392+0000 7fe4cad3f700 4 rocksdb: (Original Log Time 2024/03/11-19:12:43.395747) [db/db_impl/db_impl_compaction_flush.cc:2818] Compaction nothing to do 2024-03-11T20:15:58+01:00 urzceph1-osd05 bash[9575]: debug 2024-03-11T19:15:58.285+0000 7f9182765700 -1 bdev(0x55f72b8af800 /var/lib/ceph/osd/ceph-17/block) _aio_thread got r=-5 ((5) Input/output error) 2024-03-11T20:15:58+01:00 urzceph1-osd05 bash[9575]: debug 2024-03-11T19:15:58.289+0000 7f9182765700 -1 bdev(0x55f72b8af800 /var/lib/ceph/osd/ceph-17/block) _aio_thread translating the error to EIO for upper layer 2024-03-11T20:15:58+01:00 urzceph1-osd05 bash[9575]: debug 2024-03-11T19:15:58.289+0000 7f9182765700 -1 bdev(0x55f72b8af800 /var/lib/ceph/osd/ceph-17/block) _aio_thread got r=-5 ((5) Input/output error) 2024-03-11T20:15:58+01:00 urzceph1-osd05 bash[9575]: debug 2024-03-11T19:15:58.289+0000 7f9182765700 -1 bdev(0x55f72b8af800 /var/lib/ceph/osd/ceph-17/block) _aio_thread translating the error to EIO for upper layer 2024-03-11T20:17:02+01:00 urzceph1-osd05 bash[10152]: debug 2024-03-11T19:17:02.357+0000 7fcffadf4700 4 rocksdb: [db/db_impl/db_impl_write.cc:1736] [L] New memtable created with log file: #73918. Immutable memtables: 0. Kernel Log: [Mon Mar 11 20:15:43 2024] ata9.00: exception Emask 0x0 SAct 0xffffffff SErr 0xc0000 action 0x0 [Mon Mar 11 20:15:43 2024] ata9.00: irq_stat 0x40000008 [Mon Mar 11 20:15:43 2024] ata9: SError: { CommWake 10B8B } [Mon Mar 11 20:15:43 2024] ata9.00: failed command: READ FPDMA QUEUED [Mon Mar 11 20:15:43 2024] ata9.00: cmd 60/f8:38:60:b2:8e/00:00:37:00:00/40 tag 7 ncq dma 126976 in res 43/40:f0:68:b2:8e/00:00:37:00:00/40 Emask 0x409 (media error) <F> [Mon Mar 11 20:15:43 2024] ata9.00: status: { DRDY SENSE ERR } [Mon Mar 11 20:15:43 2024] ata9.00: error: { UNC } [Mon Mar 11 20:15:43 2024] ata9: hard resetting link [Mon Mar 11 20:15:43 2024] ata9: SATA link up 6.0 Gbps (SStatus 133 SControl 300) [Mon Mar 11 20:15:43 2024] ata9.00: configured for UDMA/133 [Mon Mar 11 20:15:43 2024] sd 8:0:0:0: [sdj] tag#7 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=3s [Mon Mar 11 20:15:43 2024] sd 8:0:0:0: [sdj] tag#7 Sense Key : Medium Error [current] [Mon Mar 11 20:15:43 2024] sd 8:0:0:0: [sdj] tag#7 Add. Sense: Unrecovered read error - auto reallocate failed [Mon Mar 11 20:15:43 2024] sd 8:0:0:0: [sdj] tag#7 CDB: Read(16) 88 00 00 00 00 00 37 8e b2 60 00 00 00 f8 00 00 [Mon Mar 11 20:15:43 2024] blk_update_request: I/O error, dev sdj, sector 932098664 op 0x0:(READ) flags 0x0 phys_seg 29 prio class 0 [Mon Mar 11 20:15:43 2024] ata9: EH complete Is this expected behavior or a bug? If it is expected how can we keep client IO flowing? Kind regards, Daniel -- Daniel Schreiber Facharbeitsgruppe Systemsoftware Universitaetsrechenzentrum Technische Universität Chemnitz Straße der Nationen 62 (Raum B303) 09111 Chemnitz Germany Tel: +49 371 531 35444 Fax: +49 371 531 835444

4 4

Mounting A RBD Image via Kernal Modules
by matthew＠peregrineit.net 21 Mar '24

21 Mar '24

Hi All, I'm looking for a bit of advice on the subject of this post. I've been "staring at the trees so long I can't see the forest any more". :-) Rocky Linux Client latest version. Ceph Reef latest version. I have read *all* the doco on the Ceph website. I have created a pool (my_pool) and an image (my_image). I had activated the pool for RBD. I can run the `rbdmap map` command on the client and the image shows up as /dev/rbd0 (and also /dev/rbd/my_pool/my_image). But here's where I'm running into issues - and I'm pretty sure it's a 'Level 8' issue, so it'll be something simple that I'm just not "getting": Do I need to run `mkfs` on /dev/rbd0 before I try `mount /dev/rbd/my_pool/my_image /mnt/rbd_image`? The reason I ask is that I've tried to mount the image before I run 'mkfs' and I get back `mount: /mnt/rbd_image: wrong fs type, bad option, bad superblock on /dev/rbdo, missing codepage or helper program, or other error`. I've also tried to mount the image after I run 'mkfs' and I get back `mount: /mnt/rbd_image: can't read superblock on /dev/rbd0`. Basically, as I've said, I'm missing or don't understand *something* about this process - which is why I'm now seeking the collective wisdom of the Community. All help and advice greatly appreciated - thanks in advance Cheers Dulux-Oz

1 0

Need easy way to calculate Ceph cluster space for SolarWinds
by Michael Worsham 21 Mar '24

21 Mar '24

Is there an easy way to poll a Ceph cluster to see how much space is available and how much space is available per bucket? Looking for a way to use SolarWinds to monitor the entire Ceph cluster space utilization and then also be able to break down each RGW bucket to see how much space it was provisioned for and how much is available. -- Michael Get Outlook for Android<https://aka.ms/AAb9ysg> This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments.

3 5

Why a lot of pgs are degraded after host(+osd) restarted?
by Jaemin Joo 21 Mar '24

21 Mar '24

Hi all, While I am testing host failover, there are a lot of degraded pg after host(+osd) is up. In spite that it takes a short time to restart, I don't understand why pg should check all objects related to the failed host(+osd). I'd like to know how to prevent to become degraded pg when osd restart. FYI. degraded pg means "active+undersized+degraded+remapped+backfilling" pg state

3 2

#1359 (update) Ceph filesystem failure | Ceph filesystem probleem
by Postmaster C&CZ (Simon) 20 Mar '24

20 Mar '24

English follows Dutch ## Update 2024-03-19 Positief nieuws; we zijn nu bezig met het kopiëren van data uit CephFS. We konden het filesystem weer mounten met hulp van 42on, onze support club. We kopiëren nu de data en dat lijkt goed te gaan. Op elk moment kunnen we tegen problematische metadata aanlopen, dus afwachten en duimen. Als we de data uit CephFS hebben gekopieerd op tijdelijke storage, kunnen we verdere oplossingen voor de toekomst verzinnen en implementeren. NB: neem contact op met postmaster (mailto:postmaster@science.ru.nl) als je een urgent verzoek hebt voor een *kleine* set van files in een specifieke locatie, dan kunnen we daar prioriteit aan geven. Een Petabyte aan data kopiëren duurt weken/maanden, kleine datasets (< 1TB) kan wel relatief snel. bron: https://cncz.science.ru.nl/nl/cpk/1359 ===================== ## Update 2024-03-19 Somewhat good news; we are now copying the RDR data out of CephFS. We have been able to mount CephFS again with help from 42on (our support party). Copying to temporary storage is going OK, but we can run into issues at any time (or not). We're hopeful that this process will let us recover the data stored in CephFs and then we can look for future solutions. NB: contact postmaster (mailto:postmaster@science.ru.nl) if you have urgent requests for *small* sets of files in a particular location, so we may restore this with priority. A Petabyte of data takes weeks/months to copy, but a small amount (< 1TB) can be retrieved relatively fast. source: https://cncz.science.ru.nl/en/cpk/1359 -- Postmaster: Simon Oosthoek Postmaster Phone: +31 24 365 3535 Personal Phone: +31 24 365 2097

1 0

Leaked clone objects
by Frédéric Nass 19 Mar '24

19 Mar '24

Hello, Over the last few weeks, we have observed a abnormal increase of a pool's data usage (by a factor of 2). It turns out that we are hit by this bug [1]. In short, if you happened to take pool snapshots and removed them by using the following command 'ceph osd pool rmsnap {pool-name} {snap-name}' instead of using this command 'rados -p {pool-name} rmsnap {snap-name}' then you may have leaked clone objects (not trimmed) in your cluster, occupying space that you can't reclaim. You may have such leaked objects if (not exclusively): - 'rados df' reports CLONES for pools with no snapshots - 'rgw-orphan-list' (if RWG pools) reports objects that you can't 'stat' but for which 'listsnaps' shows a cloneid. 'ceph osd pool force-remove-snap {pool-name}' should have the OSDs re-trim these leaked clone objects when [2] makes it to quincy, reef, squid (and hopefully pacific). Hope this helps, Regards, Frédéric. [1] https://tracker.ceph.com/issues/64646 [2] https://github.com/ceph/ceph/pull/53545

1 0

MDS_CLIENT_LATE_RELEASE, MDS_SLOW_METADATA_IO, and MDS_SLOW_REQUEST errors and slow osd_ops despite hardware being fine
by Ivan Clayson 19 Mar '24

19 Mar '24

Hello everyone, We've been experiencing on our quincy CephFS clusters (one 17.2.6 and another 17.2.7) repeated slow ops with our client kernel mounts (Ceph 17.2.7 and version 4 Linux kernels on all clients) that seem to originate from slow ops on osds despite the underlying hardware being fine. Our 2 clusters are similar and are both Alma8 systems where more specifically: * Cluster (1) is Alma8.8 running Ceph version 17.2.7 with 7 NVMe SSD OSDs storing the metadata and 432 spinning SATA disks storing the bulk data in an EC pool (8 data shards and 2 parity blocks) across 40 nodes. The whole cluster is used to support a single file system with 1 active MDS and 2 standby ones. * Cluster (2) is Alma8.7 running Ceph version 17.2.6 with 4 NVMe SSD OSDs storing the metadata and 348 spinning SAS disks storing the bulk data in EC pools (8 data shards and 2 parity blocks). This cluster houses multiple filesystems each with their own dedicated MDS along with 3 communal standby ones. Nearly daily we often find that we're get the following error messages: MDS_CLIENT_LATE_RELEASE, MDS_SLOW_METADATA_IO, and MDS_SLOW_REQUEST. Along with these messages, certain files and directory cannot be stat-ed and any processes involving these files hang indefinitely. We have been fixing this by: 1. First, finding the oldest blocked MDS op and the inode listed there: ~$ ceph tell mds.${my_mds} dump_blocked_ops 2>> /dev/null | grep -c description "description": "client_request(client.251247219:662 getattr AsLsXsFs #0x100922d1102 2024-03-13T12:51:57.988115+0000 caller_uid=26983, caller_gid=26983)", # inode/ object of interest: 100922d1102 2. Second, finding all the current clients that have a cap for this blocked inode from the faulty MDS' session list (i.e. ceph tell mds.${my_mds} session ls --cap-dump) and then examining the client whose had the cap the longest: ~$ ceph tell mds.${my_mds} session ls --cap-dump ... 2024-03-13T13:01:36: client.251247219 2024-03-13T12:50:28: client.245466949 3. Then on the aforementioned oldest client, get the current ops in flight to the OSDs (via the "/sys/kernel/debug/ceph/*/osdc" files) and get the op corresponding to the blocked inode along with the OSD the I/O is going to: root@client245466949 $ grep 100922d1102 /sys/kernel/debug/ceph/*/osdc 48366 osd79 2.249f8a51 2.a51s0 [79,351,232,179,107,195,323,14,128,167]/79 [79,351,232,179,107,195,323,14,128,167]/79 e374191 100922d1102.000000f5 0x400024 1 write # osd causing errors is osd.79 4. Finally, we restart this "hanging" OSD where this results in ls and I/O on the previously "stuck" files no longer "hanging" . Once we get this OSD for which the blocked inode is waiting for, we can see in the system logs that the OSD has slow ops: ~$ systemctl --no-pager --full status ceph-osd@79 ... 2024-03-13T12:49:37 -1 osd.79 374175 get_health_metrics reporting 3 slow ops, oldest is osd_op(client.245466949.0:41350 2.ca4s0 2.ce648ca4 (undecoded) ondisk+write+known_if_redirected e374173) ... Files that these "hanging" inodes correspond to as well as the directories housing these files can't be opened or stat-ed (causing directories to hang) where we've found restarting this OSD with slow ops to be the least disruptive way of resolving this (compared with a forced umount and then re-mount on the client). There are no issues with the underlying hardware for either the osd reporting these slow ops or any other drive within the acting PG and there seems to be no correlation between what processes are involved or what type of files these are. What could be causing these slow ops and certain files and directories to "hang"? There aren't workflows being performed that generate a large number of small files nor are there directories with a large number of files within them. This seems to happen with a wide range of hard-drives and we see this on SATA and SAS type drives where our nodes are interconnected with 25 Gb/s NICs so we can't see how the underlying hardware would be causing any I/O bottlenecks. Has anyone else seen this type of behaviour before and have any ideas? Is there a way to stop these from happening as we are having to solve these nearly daily now and we can't seem to find a way to reduce them. We do use snapshots to backup our cluster where we've been doing this for ~6 months and these issues have only been occurring on and off for a couple of months but much more frequently now. Kindest regards, Ivan Clayson -- Ivan Clayson ----------------- Scientific Computing Officer Room 2N249 Structural Studies MRC Laboratory of Molecular Biology Francis Crick Ave, Cambridge CB2 0QH

4 4

Adding new OSD's - slow_ops and other issues.
by Jesper Agerbo Krogh [JSKR] 19 Mar '24

19 Mar '24

Hi All We have a nicely functional NVMe cluster running – but in the process of expanding it we have encountered slow_ops The drill was: 1. Maintaince mode 2. Ceph orch osd add 3. Change weights to 1 4. Disable maintaince mode. Somethings goes wrong around 3 – where slowops begins to kick in – but restarting the OSD’s underneath can make it go away – tested twice. In one situation the 2) step failed to bring up the OSD – causing the entire process to work 100% correctly. Thus changing weight from 7.68 to 1 In above process on live OSD seem to be the issue. It is actually a flaw that we managed to get the entire cluster created with weight 1 instead of 7.68 in the first place thus: 1. Can we bulk change weights on all existing OSD’s without huge data movement? If so – how? 2. Can we ceph orch add – but with a specific initial weight? Thanks. Best regards, Jesper Agerbo Krogh Director Digitalization Digitalization [cid:image001.png@01DA794D.5131C5B0]<http://www.linkedin.com/company/haldor-topsoe> Topsoe A/S Haldor Topsøes Allé 1 2800 Kgs. Lyngby Denmark Phone (direct): 27773240 [cid:image002.jpg@01DA794D.5131C5B0]<http://www.linkedin.com/company/haldor-topsoe> [cid:image003.jpg@01DA794D.5131C5B0] <https://twitter.com/topsoe_official> [cid:image004.jpg@01DA794D.5131C5B0] <https://www.facebook.com/TopsoeOfficial> [cid:image005.jpg@01DA794D.5131C5B0] <https://www.topsoe.com/our-resources/knowledge/videos/> Read more at topsoe.com<http://www.topsoe.com> Topsoe A/S and/or its affiliates. This e-mail message (including attachments, if any) is confidential and may be privileged. It is intended only for the addressee. Any unauthorised distribution or disclosure is prohibited. Disclosure to anyone other than the intended recipient does not constitute waiver of privilege. If you have received this email in error, please notify the sender by email and delete it and any attachments from your computer system and records.

2 1

Call for interest: VMWare Photon OS support in Cephadm
by Ernesto Puerta 18 Mar '24

18 Mar '24

Hi Cephers, This is to sound out potential interest in supporting VMWare's Photon OS distribution with Ceph/Cephadm. Photon OS is a lightweight distribution that is optimized for deploying containers within virtual machines. It is based on systemd, includes Docker, and uses the tdnf (Tiny DNF) package manager. Cephadm already supports Azure Linux/CBL-Mariner <https://github.com/microsoft/azurelinux>, which is a Microsoft derivative from the Photon OS project for the Azure ecosystem. Thus, no major changes would be required. If you're interested and willing to take part in the deployment beta-test, please drop us a line! Thank you! Kind Regards, Ernesto

2 2

Re: Fwd: Ceph fs snapshot problem
by Marcus 18 Mar '24

18 Mar '24

Hi, There is no such attribute. /mnt: ceph.dir.subvolume: No such attribute I did not have getfattr installed so needed to install attr package. Can it be that this package was not installed when fs was created so ceph.dir.subvolume could not be set at creation? Did not get any warnings at creation though. Thanks for you help!! On lör, mar 16 2024 at 00:53:22 +0530, Neeraj Pratap Singh <neesingh(a)redhat.com> wrote: > Can u pls do getfattr on root directory and tell whats the output? > Run this command: getfattr -n ceph.dir.subvolume /mnt > > On Thu, Mar 14, 2024 at 4:38 PM Marcus <marcus(a)marcux.org > <mailto:marcus@marcux.org>> wrote: >> >> Hi all, >> I have just setup a small ceph cluster with ceph fs. >> The setup is reef 18.2.1 on Debian bookworm. >> The system is up and running the way it should, >> though I have a problem with ceph fs snapshots. >> >> When I read the doc I should be able to make a >> snapshot in any directory in the filesystem. >> I can do a snapshot in the root of the filesystem >> but if I try somewhere else I get: >> Operation not permitted >> This is the same if I do it with mkdir or >> with ceph fs subvolume snapshot create ... >> >> I have created an auth client with rws: >> [client.snap-mount] >> key = **** >> caps mds = "allow rws fsname=gds-common" >> caps mon = "allow r fsname=gds-common" >> caps osd = "allow rw tag cephfs data=gds-common" >> >> Where the filsystem is called gds-common, >> saved in a file on the client: >> /etc/ceph/ceph.client.snap-mount.keyring >> >> I mount ceph fs with: >> mount -t ceph :/ -o name=snap-mount /mnt >> >> If I create a snapshot in root, it works fine, as in: >> mkdir /mnt/.snap/mysnap >> I also notice that in every subdir there is a "snapshot dir" as well >> with the name _mysnap_1, as in: >> /mnt/dir/.snap/_mysnap_1 >> >> My guess that is is a part of the snapshot system, this "snapshot" >> dissapear when the snapshot is removed with: >> rmdir /mnt/.snap/mysnap >> >> If I try to make a snapshot in another directory this does not work: >> mkdir /mnt/dir/.snap/othersnap >> Get the error: >> cannot create directory ‘/mnt/dir/.snap/othersnap’: Operation >> not >> permitted >> >> It is the same thing on the commandline, root works: >> ceph fs subvolume snapshot create gds-common / fromcmd >> >> But not in a subdir: >> ceph fs subvolume snapshot create gds-common /dir dirsnap >> Error EINVAL: invalid value specified for ceph.dir.subvolume >> >> I also notice that when you use any command of type: >> ceph fs subvolume snapshot ... >> You get a new directory (volumes) in the root: >> /mnt/volumes/_legacy/6666cd76f96956469e7be39d750cc7d9.meta >> >> I do not know if I am missing something, some lacking of >> config or so. >> >> Thanks for your help!! >> >> Best regards >> Marcus >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> <mailto:ceph-users@ceph.io> >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> <mailto:ceph-users-leave@ceph.io>

2 2

activating+undersized+degraded+remapped
by Deep Dish 17 Mar '24

17 Mar '24

Hello I found myself in the following situation: [WRN] PG_AVAILABILITY: Reduced data availability: 3 pgs inactive pg 4.3d is stuck inactive for 8d, current state activating+undersized+degraded+remapped, last acting [4,NONE,46,NONE,10,13,NONE,74] pg 4.6e is stuck inactive for 9d, current state activating+undersized+degraded+remapped, last acting [NONE,27,77,79,55,48,50,NONE] pg 4.cb is stuck inactive for 8d, current state activating+undersized+degraded+remapped, last acting [6,NONE,42,8,60,22,35,45] I have one cephfs with two backing pools -- one for replicated data, the other for erasure data. Each pool is mapped to REPLICATED/ vs. ERASURE/ directories on the filesystem. The above pgs. are affecting the ERASURE pool (5+3) backing the FS. How can I get ceph to recover these three PGs? Thank you.

4 4

Num values for 3 DC 4+2 crush rule
by Torkil Svensgaard 16 Mar '24

16 Mar '24

I was just looking at our crush rules as we need to change them from failure domain host to failure domain datacenter. The replicated ones seem trivial but what about this one for EC 4+2? rule rbd_ec_data { id 0 type erasure step set_chooseleaf_tries 5 step set_choose_tries 100 step take default class hdd step chooseleaf indep 0 type host step emit } We already have this crush rule for EC 4+5: " rule cephfs.hdd.data { id 7 type erasure step set_chooseleaf_tries 5 step set_choose_tries 100 step take default class hdd step choose indep 0 type datacenter step chooseleaf indep 3 type host step emit } " I don't understand the "num" argument for the choose step. The documentation[1] says: " If {num} == 0, choose pool-num-replicas buckets (as many buckets as are available). If pool-num-replicas > {num} > 0, choose that many buckets. If {num} < 0, choose pool-num-replicas - {num} buckets. " Num is 0 but it's not replicated so how does this translate to picking 3 of 3 datacenters? I am thinking we should just change 3 to 2 for the chooseleaf line for the 4+2 rule since for 4+5 each DC needs 3 shards and for 4+2 each DC needs 2 shards. Comments? Mvh. Torkil [1] https://docs.ceph.com/en/reef/rados/operations/crush-map-edits/ -- Torkil Svensgaard Systems Administrator Danish Research Centre for Magnetic Resonance DRCMR, Section 714 Copenhagen University Hospital Amager and Hvidovre Kettegaard Allé 30, 2650 Hvidovre, Denmark

2 1

[Urgent] Ceph system Down, Ceph FS volume in recovering
by nguyenvandiep＠baoviet.com.vn 16 Mar '24

16 Mar '24

We have 6 node ( 3 OSD-node and 3 service node), t2/3 OSD nodes was powered off and we got big problem pls check ceph-s result below now we cannot start mds service, ( we tried to start but it stopped after 2 minute) Now my application cannot access to NFS exported Folder What should we do [root@cephgw01 /]# ceph -s cluster: id: 258af72a-cff3-11eb-a261-d4f5ef25154c health: HEALTH_WARN 3 failed cephadm daemon(s) 1 filesystem is degraded insufficient standby MDS daemons available 1 nearfull osd(s) Low space hindering backfill (add storage if this doesn't resolve itself): 21 pgs backfill_toofull 15 pool(s) nearfull 11 daemons have recently crashed services: mon: 6 daemons, quorum cephgw03,cephosd01,cephgw01,cephosd03,cephgw02,cephosd02 (age 30h) mgr: cephgw01.vwoffq(active, since 17h), standbys: cephgw02.nauphz, cephgw03.aipvii mds: 1/1 daemons up osd: 29 osds: 29 up (since 40h), 29 in (since 29h); 402 remapped pgs rgw: 2 daemons active (2 hosts, 1 zones) tcmu-runner: 18 daemons active (2 hosts) data: volumes: 0/1 healthy, 1 recovering pools: 15 pools, 1457 pgs objects: 36.87M objects, 25 TiB usage: 75 TiB used, 41 TiB / 116 TiB avail pgs: 17759672/110607480 objects misplaced (16.056%) 1055 active+clean 363 active+remapped+backfill_wait 18 active+remapped+backfilling 14 active+remapped+backfill_toofull 7 active+remapped+backfill_wait+backfill_toofull io: client: 2.0 MiB/s rd, 395 KiB/s wr, 73 op/s rd, 19 op/s wr recovery: 32 MiB/s, 45 objects/s

6 49

Cephfs error state with one bad file
by Sake Ceph 15 Mar '24

15 Mar '24

Hi again, hopefully for the last time with problems. We had a MDS crash earlier with the MDS staying in failed state and used a command to reset the filesystem (this was wrong, I know now, thanks Patrick Donnelly for pointing this out). I did a full scrub on the filesystem and two files were damaged. One of those got repaired, but the following file keeps giving errors and can't be removed. What can I do now? Below some information. # ceph tell mds.atlassian-prod:0 damage ls [ { "damage_type": "backtrace", "id": 2244444901, "ino": 1099534008829, "path": "/app1/shared/data/repositories/11271/objects/41/8f82507a0737c611720ed224bcc8b7a24fda01" } ] Trying to repair the error (online research shows this should work for a backtrace damage type) ---------- # ceph tell mds.atlassian-prod:0 scrub start /app1/shared/data/repositories/11271 recursive,repair,force { "return_code": 0, "scrub_tag": "d10ead42-5280-4224-971e-4f3022e79278", "mode": "asynchronous" } Cluster logs after this ---------- 1/2/24 9:37:05 AM [INF] scrub summary: idle 1/2/24 9:37:02 AM [INF] scrub summary: idle+waiting paths [/app1/shared/data/repositories/11271] 1/2/24 9:37:01 AM [INF] scrub summary: active paths [/app1/shared/data/repositories/11271] 1/2/24 9:37:01 AM [INF] scrub summary: idle+waiting paths [/app1/shared/data/repositories/11271] 1/2/24 9:37:01 AM [INF] scrub queued for path: /app1/shared/data/repositories/11271 But the error doesn't disappear and still can't remove the file. On the client trying to remove the file (we got a backup) ---------- $ rm -f /mnt/shared_disk-app1/shared/data/repositories/11271/objects/41/8f82507a0737c611720ed224bcc8b7a24fda01 rm: cannot remove '/mnt/shared_disk-app1/shared/data/repositories/11271/objects/41/8f82507a0737c611720ed224bcc8b7a24fda01': Input/output error Best regards, Sake

2 1

MDS subtree pinning
by Sake Ceph 15 Mar '24

15 Mar '24

Hi! As I'm reading through the documentation about subtree pinning, I was wondering if the following is possible. We've got the following directory structure. / /app1 /app2 /app3 /app4 Can I pin /app1 to MDS rank 0 and 1, the directory /app2 to rank 2 and finally /app3 and /app4 to rank 3? I would like to load balance the subfolders of /app1 to 2 (or 3) MDS servers. Best regards, Sake

2 2