January 2024 - ceph-users

could not find secret_id
by xiaowenhao111 22 Apr '26

22 Apr '26

2 1

OpenSSL security update for Octopus container?
by Robert Sander 02 Oct '25

02 Oct '25

Hi, I am running a Ceph Octopus cluster in version 15.2.10 setup with the cephadm orchestrator and official Docker hub container images. Yesterday an important security fix was released for libssl and is already packaged for all major distributions. I tried to run "ceph orch upgrade check docker.io/ceph/ceph:v15" but it tells me that the containers do not need to be upgraded. How will this security fix of OpenSSL be deployed in a timely manner to users of the Ceph container images? Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 93818 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 1

Problem while upgrade 17.2.6 to 17.2.7
by Jean-Marc FONTANA 21 Feb '25

21 Feb '25

Hello everyone, We operate two clusters that we installed with ceph-deploy in Nautilus version on Debian 10. We use them for external S3 storage (owncloud) and rbd disk images.We had them upgraded to Octopus and Pacific versions on Debian 11 and recently converted them to cephadm and upgraded to Quincy (17.2.6). As we now have the orchestrator, we tried updating to 17.2.7 using the command# ceph orch upgrade start --image quay.io/ceph/ceph:v17.2.7 Everything went well, both clusters work perfectly for our use, except that the Rados gateway configuration is no longer accessible from the dashboard with the following error messageError connecting to Object Gateway: RGW REST API failed request with status code 404. We tried a few solutions found on the internet (reset rgw credentials, restart rgw adnd mgr, reenable dashboard, ...), unsuccessfully. Does somebody have an idea ? Best regards, Jean-Marc Fontana

5 9

Problems with autoscaler (overlapping roots) after changing the pool class
by Massimo Sgaravatto 06 Jan '25

06 Jan '25

Dear all I have just changed the crush rule for all the replicated pools in the following way: ceph osd crush rule create-replicated replicated_hdd default host hdd ceph osd pool set <poolname> crush_rule replicated_hdd See also this [*] thread Before applying this change, these pools were all using the replicated_ruleset rule where the class is not specified. I am noticing now a problem with the autoscaler: "ceph osd pool autoscale-status" doesn't report any output and the mgr log complains about overlapping roots: [pg_autoscaler ERROR root] pool xyz has overlapping roots: {-18, -1} Indeed: # ceph osd crush tree --show-shadow ID CLASS WEIGHT TYPE NAME -18 hdd 1329.26501 root default~hdd -17 hdd 329.14154 rack Rack11-PianoAlto~hdd -15 hdd 54.56085 host ceph-osd-04~hdd 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... ... -1 1329.26501 root default -7 329.14154 rack Rack11-PianoAlto -8 54.56085 host ceph-osd-04 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... I have already read about this behavior but I have no clear ideas how to fix the problem. I read somewhere that the problem happens when there are rules that force some pools to only use one class and there are also pools which does not make any distinction between device classes All the replicated pools are using the replicated_hdd pool but I also have some EC pools which are using a profile where the class is not specified. As far I understand, I can't force these pools to use only the hdd class: according to the doc I can't change this profile specifying the hdd class (or at least the change wouldn't be applied to the existing EC pools) Any suggestions ? The crush map is available at https://cernbox.cern.ch/s/gIyjbQbmoTFHCrr, if you want to have a look Many thanks, Massimo [*] https://www.mail-archive.com/ceph-users@ceph.io/msg18534.html

4 9

multifs and snapshots
by Dmitry Melekhov 12 Nov '24

12 Nov '24

Hello! There is note here https://docs.ceph.com/en/reef/dev/cephfs-snapshots/ about multiply filesystems and snapshots: If each FS gets its own pool things probably work, but this isn’t tested and may not be true. Is it still untested ? Thank you!

3 4

Help needed to configure erasure coding LRC plugin
by Michel Jouvin 16 Oct '24

16 Oct '24

Hi, As discussed in another thread (Crushmap rule for multi-datacenter erasure coding), I'm trying to create an EC pool spanning 3 datacenters (datacenters are present in the crushmap), with the objective to be resilient to 1 DC down, at least keeping the readonly access to the pool and if possible the read-write access, and have a storage efficiency better than 3 replica (let say a storage overhead <= 2). In the discussion, somebody mentioned LRC plugin as a possible jerasure alternative to implement this without tweaking the crushmap rule to implement the 2-step OSD allocation. I looked at the documentation (https://docs.ceph.com/en/latest/rados/operations/erasure-code-lrc/) but I have some questions if someone has experience/expertise with this LRC plugin. I tried to create a rule for using 5 OSDs per datacenter (15 in total), with 3 (9 in total) being data chunks and others being coding chunks. For this, based of my understanding of examples, I used k=9, m=3, l=4. Is it right? Is this configuration equivalent, in terms of redundancy, to a jerasure configuration with k=9, m=6? The resulting rule, which looks correct to me, is: -------- { "rule_id": 6, "rule_name": "test_lrc_2", "ruleset": 6, "type": 3, "min_size": 3, "max_size": 15, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -4, "item_name": "default~hdd" }, { "op": "choose_indep", "num": 3, "type": "datacenter" }, { "op": "chooseleaf_indep", "num": 5, "type": "host" }, { "op": "emit" } ] } ------------ Unfortunately, it doesn't work as expected: a pool created with this rule ends up with its pages active+undersize, which is unexpected for me. Looking at 'ceph health detail` output, I see for each page something like: pg 52.14 is stuck undersized for 27m, current state active+undersized, last acting [90,113,2147483647,103,64,147,164,177,2147483647,133,58,28,8,32,2147483647] For each PG, there is 3 '2147483647' entries and I guess it is the reason of the problem. What are these entries about? Clearly it is not OSD entries... Looks like a negative number, -1, which in terms of crushmap ID is the crushmap root (named "default" in our configuration). Any trivial mistake I would have made? Thanks in advance for any help or for sharing any successful configuration? Best regards, Michel

4 24

rgw connection resets
by Nathan Gleason 10 Oct '24

10 Oct '24

Hello, I’ve noticed what I think may be strange behavior from some rgw transfers. It only seems to be with larger multipart mp4 files. Once a connection comes into rgw it is accepted and processed, data begins to transfer, then I see "write_data failed: Connection reset by peer” and "s3:get_obj iterate_obj() failed with -104”. The client then reconnects and the transfer seems to be ok. This doesn’t appear to happen just downloading a regular file. This is happening to both my production and test cluster. I’m not seeing errors anywhere else, the network looks good and the clusters are healthy. I’m in the process of collecting some tcpdumps but figured I would ask if anyone knows why this is occurring. We run haproxy in front of the rgw. For troubleshooting purposes we have a basic haproxy config to just one rgw. We have tested with multiple browsers. Ceph 17.2.5 - cluster built and managed with cephadm Ubuntu 20.04.6 LTS Log: 2024-01-03T17:40:23.916+0000 7f646033d700 1 ====== starting new request req=0x7f640ca95730 ===== 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s initializing for trans_id = tx000003ec1e4b4dc157efd-0065959c07-104216-default 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s rgw api priority: s3=8 s3website=7 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s host=a2-west.usacld.net 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s final domain/bucket subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 s->info.domain= s->info.request_uri=/n athan:tshoot/testfile.mp4 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s get_handler handler=22RGWHandler_REST_Obj_S3 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s handler=22RGWHandler_REST_Obj_S3 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s getting op 0 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s get_system_obj_state: rctx=0x7f640ca947c0 obj=default.rgw.log:script.prerequest. state=0x55fa31f882e0 s->prefetch_data=0 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s cache get: name=default.rgw.log++script.prerequest. : hit (negative entry) 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj scheduling with throttler client=2 cost=1 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj op=21RGWGetObj_ObjStore_S3 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj verifying requester 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rgw::auth::StrategyRegistry::s3_main_strategy_t: trying rgw::auth::s3::AWSAuthStrategy 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::S3AnonymousEngine 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rgw::auth::s3::S3AnonymousEngine granted access 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rgw::auth::s3::AWSAuthStrategy granted access 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj normalizing buckets and tenants 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s->object=testfile.mp4 s->bucket=nathan/tshoot 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj init permissions 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_system_obj_state: rctx=0x7f640ca94200 obj=default.rgw.meta:root:nathan/tshoot state=0x55fa31f882e0 s->prefetch _data=0 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj cache get: name=default.rgw.meta+root+nathan/tshoot : hit (requested=0x16, cached=0x17) 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_system_obj_state: s->obj_tag was set empty 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj cache get: name=default.rgw.meta+root+nathan/tshoot : hit (requested=0x11, cached=0x17) 2024-01-03T17:40:23.916+0000 7f646033d700 15 req 4522146966294331133 0.000000000s s3:get_obj decode_policy Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner> <ID>nathan$testuser</ID><DisplayName>Nathan API User</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>nathan$testuser </ID><DisplayName>Nathan API User</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy> 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWSI_User_RADOS::read_user_info(): anonymous user 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj recalculating target 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj reading permissions 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_obj_state: rctx=0x7f640ca94ad0 obj=tshoot:testfile.mp4 state=0x55fa31082de8 s->prefetch_data=1 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj WARNING: blocking librados call 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj manifest: total_size = 88811314 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_obj_state: setting s->obj_tag to _0OUNxgsSsqTAs9Z6xh1SOKpJZLV_Vos 2024-01-03T17:40:23.916+0000 7f646033d700 15 req 4522146966294331133 0.000000000s s3:get_obj decode_policy Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner> <ID>nathan$testuser</ID><DisplayName>Nathan API User</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="Group"><URI>http://acs.amazonaws.c om/groups/global/AllUsers</URI></Grantee><Permission>READ</Permission></Grant><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>nathan$testuser</ID><DisplayNa me>Nathan API User</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy> 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj init op 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_system_obj_state: rctx=0x7f640ca94488 obj=default.rgw.meta:users.uid:nathan$testuser state=0x55fa31f882e0 s->p refetch_data=0 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj cache get: name=default.rgw.meta+users.uid+nathan$testuser : hit (requested=0x16, cached=0x17) 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_system_obj_state: s->obj_tag was set empty 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr: user.rgw.idtag 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj cache get: name=default.rgw.meta+users.uid+nathan$testuser : hit (requested=0x13, cached=0x17) 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj verifying op mask 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj required_mask= 1 user.op_mask=7 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj verifying op permissions 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj -- Getting permissions begin with perm_mask=49 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj Searching permissions for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=anonymous, acct_n ame=, subuser=, perm_mask=15, is_admin=0) mask=49 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj Searching permissions for uid=anonymous 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj Permissions for user not found 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj Searching permissions for group=1 mask=49 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj Found permission: 1 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj -- Getting permissions done for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=anonymous, acct_name=, subuser=, perm_mask=15, is_admin=0), owner=nathan$testuser, perm=1 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=anonymous, acct_name=, subuser=, perm_mask =15, is_admin=0) requested perm (type)=1, policy perm=1, user_perm_mask=15, acl perm=1 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj verifying op params 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj pre-executing 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj check rate limiting 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj executing 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_obj_state: rctx=0x7f640ca94ad0 obj=tshoot:testfile.mp4 state=0x55fa31082de8 s->prefetch_data=1 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.acl 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.content_type 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.etag 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.idtag 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.manifest 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.pg_ver 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.source_zone 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.tail_tag 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.x-amz-content-sha256 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.x-amz-date 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.x-amz-meta-s3cmd-attrs 2024-01-03T17:40:23.916+0000 7f646033d700 15 req 4522146966294331133 0.000000000s Encryption mode: 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_obj_state: rctx=0x7f640ca94ad0 obj=tshoot:testfile.mp4 state=0x55fa31082de8 s->prefetch_data=1 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__multipart_testfile.mp4.2~XSiTvBBtaU4 134dQCusv3kMP7YjDBk7.1 obj-ofs=0 read_ofs=0 len=4194304 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=4194304 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): result: ofs=4194304 stripe_ofs=4194304 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134 dQCusv3kMP7YjDBk7.1_1 obj-ofs=4194304 read_ofs=0 len=4194304 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=8388608 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): result: ofs=8388608 stripe_ofs=8388608 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134 dQCusv3kMP7YjDBk7.1_2 obj-ofs=8388608 read_ofs=0 len=4194304 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=12582912 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): result: ofs=12582912 stripe_ofs=12582912 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134 dQCusv3kMP7YjDBk7.1_3 obj-ofs=12582912 read_ofs=0 len=3145728 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=16777216 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): result: ofs=15728640 stripe_ofs=15728640 part_ofs=15728640 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__multipart_testfile.mp4.2~XSiTvBBtaU4 134dQCusv3kMP7YjDBk7.2 obj-ofs=15728640 read_ofs=0 len=4194304 2024-01-03T17:40:23.924+0000 7f6442301700 20 req 4522146966294331133 0.007999974s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.924+0000 7f6442301700 20 req 4522146966294331133 0.007999974s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=19922944 part_ofs=15728640 rule->part_size=15728640 2024-01-03T17:40:23.924+0000 7f6442301700 20 req 4522146966294331133 0.007999974s s3:get_obj RGWObjManifest::operator++(): result: ofs=19922944 stripe_ofs=19922944 part_ofs=15728640 rule->part_size=15728640 2024-01-03T17:40:23.924+0000 7f6442301700 20 req 4522146966294331133 0.007999974s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134 dQCusv3kMP7YjDBk7.2_1 obj-ofs=19922944 read_ofs=0 len=4194304 2024-01-03T17:40:23.932+0000 7f643d2f7700 20 req 4522146966294331133 0.015999949s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.932+0000 7f643d2f7700 20 req 4522146966294331133 0.015999949s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=24117248 part_ofs=15728640 rule->part_size=15728640 2024-01-03T17:40:23.932+0000 7f643d2f7700 20 req 4522146966294331133 0.015999949s s3:get_obj RGWObjManifest::operator++(): result: ofs=24117248 stripe_ofs=24117248 part_ofs=15728640 rule->part_size=15728640 2024-01-03T17:40:23.932+0000 7f643d2f7700 20 req 4522146966294331133 0.015999949s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134 dQCusv3kMP7YjDBk7.2_2 obj-ofs=24117248 read_ofs=0 len=4194304 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_status: e=0, sent=17, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_content_length: e=0, sent=26, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::complete_header: e=0, sent=460, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::set_account: e=1 2024-01-03T17:40:24.068+0000 7f642a2d1700 4 write_data failed: Connection reset by peer 2024-01-03T17:40:24.068+0000 7f642a2d1700 0 req 4522146966294331133 0.151999518s s3:get_obj iterate_obj() failed with -104 2024-01-03T17:40:24.156+0000 7f642eada700 2 req 4522146966294331133 0.239999235s s3:get_obj completing 2024-01-03T17:40:24.156+0000 7f642eada700 20 req 4522146966294331133 0.239999235s get_system_obj_state: rctx=0x7f640ca947c0 obj=default.rgw.log:script.postrequest.nathan state=0x55fa32424760 s->prefetch_data =0 2024-01-03T17:40:24.156+0000 7f642eada700 10 req 4522146966294331133 0.239999235s cache get: name=default.rgw.log++script.postrequest.nathan : hit (negative entry) 2024-01-03T17:40:24.156+0000 7f642eada700 30 AccountingFilter::complete_request: e=1, sent=0, total=0 2024-01-03T17:40:24.156+0000 7f642eada700 2 req 4522146966294331133 0.239999235s s3:get_obj op status=-104 2024-01-03T17:40:24.156+0000 7f642eada700 2 req 4522146966294331133 0.239999235s s3:get_obj http status=200 2024-01-03T17:40:24.156+0000 7f642eada700 1 ====== req done req=0x7f640ca95730 op status=-104 http_status=200 latency=0.239999235s ====== 2024-01-03T17:40:24.156+0000 7f642eada700 1 beast: 0x7f640ca95730: 10.30.0.196 - anonymous [03/Jan/2024:17:40:23.916 +0000] "GET /nathan:tshoot/testfile.mp4 HTTP/1.1" 200 0 - "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" - latency=0.239999235s 2024-01-03T17:40:24.156+0000 7f642eada700 20 failed to read header: end of stream 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_ACCEPT=*/* 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_ACCEPT_ENCODING=identity;q=1, *;q=0 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_ACCEPT_LANGUAGE=en-US,en;q=0.7 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_HOST=a2-west.usacld.net 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_RANGE=bytes=0- 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_REFERER=https://a2-west.usacld.net/nathan:tshoot/testfile.mp4 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_CH_UA="Not_A Brand";v="8", "Chromium";v="120", "Brave";v="120" 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_CH_UA_MOBILE=?0 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_CH_UA_PLATFORM="macOS" 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_FETCH_DEST=video 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_FETCH_MODE=no-cors 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_FETCH_SITE=same-origin 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_GPC=1 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_VERSION=1.1 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_X_FORWARDED_PORT=443 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_X_FORWARDED_PROTO=https 2024-01-03T17:40:24.160+0000 7f6503483700 20 REMOTE_ADDR=10.30.0.196 2024-01-03T17:40:24.160+0000 7f6503483700 20 REQUEST_METHOD=GET 2024-01-03T17:40:24.160+0000 7f6503483700 20 REQUEST_URI=/nathan:tshoot/testfile.mp4 2024-01-03T17:40:24.160+0000 7f6503483700 20 SCRIPT_URI=/nathan:tshoot/testfile.mp4 2024-01-03T17:40:24.160+0000 7f6503483700 20 SERVER_PORT=8000 2024-01-03T17:40:24.160+0000 7f6503483700 1 ====== starting new request req=0x7f640ca95730 ===== Thanks, Nathan

2 1

Not all Bucket Shards being used
by Christian Kugler 01 Oct '24

01 Oct '24

Hi, I have trouble with large OMAP files in a cluster in the RGW index pool. Some background information about the cluster: There is CephFS and RBD usage on the main cluster but for this issue I think only S3 is interesting. There is one realm, one zonegroup with two zones which have a bidirectional sync set up. Since this does not allow for autoresharding we have to do it by hand in this cluster – looking forward to Reef! From the logs: cluster 2023-07-17T22:59:03.018722+0000 osd.75 (osd.75) 623978 : cluster [WRN] Large omap object found. Object: 34:bcec3016:::.dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.5:head PG: 34.680c373d (34.5) Key count: 962091 Size (bytes): 277963182 The offending bucket looks like this: # radosgw-admin bucket stats \ | jq '.[] | select(.marker =="3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9") |"\(.num_shards) \(.usage["rgw.main"].num_objects)"' -r 131 9463833 Last week the number of objects was about 12 million. Which is why I reshareded the offending bucket twice, I think. Once to 129 and the second time to 131 because I wanted some leeway (or lieway? scnr, Sage). Unfortunately, even after a week the objects were still to big (the log line above is quite recent), so I looked into it again. # rados -p raum.rgw.buckets.index ls \ |grep .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9 \ |sort -V .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.0 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.1 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.2 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.3 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.4 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.5 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.6 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.7 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.8 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.9 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.10 # rados -p raum.rgw.buckets.index ls \ |grep .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9 \ |sort -V \ |xargs -IOMAP sh -c \ 'rados -p raum.rgw.buckets.index listomapkeys OMAP | wc -l' 1013854 1011007 1012287 1011232 1013565 998262 1012777 1012713 1012230 1010690 997111 Apparently, only 11 shards are in use. This would explain why the "Key usage" (from the log line) is about ten times higher than I would expect. How can I deal with this issue? One thing I could try to fix this would be to reshard to a lower number, but I am not sure if there are any risks associated with "downsharding". After that I could reshard to something like 97. Or I could directly "downshard" to 97. Also, the second zone has a similar problem, but as the error messsage lets me know, this would be a bad idea. Will it just take more time until the sharding is transferred to the seconds zone? Best, Christian Kugler

2 5

Successfully using dm-cache
by Michael Lipp 12 Sep '24

12 Sep '24

Just in case anybody is interested: Using dm-cache works and boosts performance -- at least for my use case. The "challenge" was to get 100 (identical) Linux-VMs started on a three node hyperconverged cluster. The hardware is nothing special, each node has a Supermicro server board with a single CPU with 24 cores and 4 x 4 TB hard disks. And there's that extra 1 TB NVMe... I know that the general recommendation is to use the NVMe for WAL and metadata, but this didn't seem appropriate for my use case and I'm still not quite sure about failure scenarios with this configuration. So instead I made each drive a logical volume (managed by an OSD) and added 85 GiB NVMe to each LV as read-only cache. Each VM uses as system disk an RBD based on a snapshot from the master image. The idea was that with this configuration, all VMs should share most (actually almost all) of the data on their system disk and this data should be available from the cache. Well, it works. When booting the 100 VMs, almost all read operations are satisfied from the cache. So I get close to NVMe speed but have payed for conventional hard drives only (well, SSDs aren't that much more expensive nowadays, but the hardware is 4 years old). So, nothing sophisticated, but as I couldn't find anything about this kind of setup, it might be of interest nevertheless. - Michael

13 27

cephadm discovery service certificate absent after upgrade.
by Nicolas FOURNIL 09 Jul '24

09 Jul '24

Hello, I've just fresh upgrade from Quincy to Reef and my graphs are now blank... after investigations, it seems that discovery service is not working because of no certificate : # ceph orch sd dump cert Error EINVAL: No certificate found for service discovery Maybe an upgrade issue ? Is there a way to generate or replace the certificate properly ? Regards Nicolas F.

3 8

Fixing BlueFS spillover (pacific 16.2.14)
by Chris Dunlop 09 Jul '24

09 Jul '24

Hi, tl;dr why are my osds still spilling? I've recently upgraded to 16.2.14 from 16.2.9 and started receiving bluefs spillover warnings (due to the "fix spillover alert" per the 16.2.14 release notes). E.g. from 'ceph health detail', the warning on one of these (there are a few): osd.76 spilled over 128 KiB metadata from 'db' device (56 GiB used of 60 GiB) to slow device This is a 15T HDD with only a 60G SSD for the db so it's not surprising it spilled as it's way below the recommendation for rbd usage at db size 1-2% of the storage size. There was some spare space on the db ssd so I increased the size of the db LV up over 400G and did an bluefs-bdev-expand. However, days later, I'm still getting the spillover warning for that osd, including after running a manual compact: # ceph tell osd.76 compact See attached perf-dump-76 for the perf dump output: # cephadm enter --name 'osd.76' ceph daemon 'osd.76' perf dump" | jq -r '.bluefs' In particular, if my understanding is correct, that's telling me the db available size is 487G (i.e. the LV expand worked), of which it's using 59G, and there's 128K spilled to the slow device: "db_total_bytes": 512309059584, # 487G "db_used_bytes": 63470305280, # 59G "slow_used_bytes": 131072, # 128K A "bluefs stats" also says the db is using 128K of slow storage (although perhaps it's getting the info from the same place as the perf dump?): # ceph tell osd.76 bluefs stats 1 : device size 0x7747ffe000 : using 0xea6200000(59 GiB) 2 : device size 0xe8d7fc00000 : using 0x6554d689000(6.3 TiB) RocksDBBlueFSVolumeSelector Usage Matrix: DEV/LEV WAL DB SLOW * * REAL FILES LOG 0 B 10 MiB 0 B 0 B 0 B 8.8 MiB 1 WAL 0 B 2.5 GiB 0 B 0 B 0 B 751 MiB 8 DB 0 B 56 GiB 128 KiB 0 B 0 B 50 GiB 842 SLOW 0 B 0 B 0 B 0 B 0 B 0 B 0 TOTAL 0 B 58 GiB 128 KiB 0 B 0 B 0 B 850 MAXIMUMS: LOG 0 B 22 MiB 0 B 0 B 0 B 18 MiB WAL 0 B 3.9 GiB 0 B 0 B 0 B 1.0 GiB DB 0 B 71 GiB 282 MiB 0 B 0 B 62 GiB SLOW 0 B 0 B 0 B 0 B 0 B 0 B TOTAL 0 B 74 GiB 282 MiB 0 B 0 B 0 B >> SIZE << 0 B 453 GiB 14 TiB I had a look at the "DUMPING STATS" output in the logs bug I don't know how to interpret it. I did try calculating the total of the sizes on the "Sum" lines but that comes to 100G so I don't know what that all means. See attached log-stats-76. I also tried "ceph-kvstore-tool bluestore-kv ... stats": $ { cephadm unit --fsid $clusterid --name osd.76 stop cephadm shell --fsid $clusterid --name osd.76 -- ceph-kvstore-tool bluestore-kv /var/lib/ceph/osd/ceph-76 stats cephadm unit --fsid $clusterid --name osd.76 start } Output attached as bluestore-kv-stats-76. I can't see anything interesting in there, although again I don't really know how to interpret it. So... why is this osd db still spilling onto slow storage, and how do I fix things so it's no longer using the slow storage? And a bonus issue... on another osd that hasn't yet been resized (i.e. again with a grossly undersized 60G db on SSD with a 15T HDD) I'm also getting a spillover warning. The "bluefs stats" seems to be saying the db is NOT currently spilling (i.e. "0 B" the DB/SLOW position in the matrix), but there's "something" currently using 59G on the slow device: $ ceph tell osd.85 bluefs stats 1 : device size 0xeffffe000 : using 0x3a3900000(15 GiB) 2 : device size 0xe8d7fc00000 : using 0x7aea7434000(7.7 TiB) RocksDBBlueFSVolumeSelector Usage Matrix: DEV/LEV WAL DB SLOW * * REAL FILES LOG 0 B 10 MiB 0 B 0 B 0 B 7.4 MiB 1 WAL 0 B 564 MiB 0 B 0 B 0 B 132 MiB 2 DB 0 B 11 GiB 0 B 0 B 0 B 8.1 GiB 177 SLOW 0 B 3.0 GiB 59 GiB 0 B 0 B 56 GiB 898 TOTAL 0 B 13 GiB 59 GiB 0 B 0 B 0 B 1072 MAXIMUMS: LOG 0 B 24 MiB 0 B 0 B 0 B 20 MiB WAL 0 B 2.8 GiB 0 B 0 B 0 B 1.0 GiB DB 0 B 22 GiB 448 KiB 0 B 0 B 18 GiB SLOW 0 B 3.3 GiB 62 GiB 0 B 0 B 62 GiB TOTAL 0 B 27 GiB 62 GiB 0 B 0 B 0 B >> SIZE << 0 B 57 GiB 14 TiB Is there anywhere that describes how to interpret this output, and specifically, what stuff is going into the SLOW row? Seemingly there's 898 "files" there, but not LOG, WAL or DB files - so what are they? Cheers, Chris

4 7

Correct Migration Workflow Replicated -> Erasure Code
by Mac Wynkoop 19 Jun '24

19 Jun '24

Hi Everyone, So, I'm in the process of trying to migrate our rgw.buckets.data pool from a replicated rule pool to an erasure coded pool. I've gotten the EC pool set up, good EC profile and crush ruleset, pool created successfully, but when I go to "rados cppool xxx.rgw.buckets.data xxx.rgw.buckets.data.new", I get this error after it transfers 4GB of data: error copying object: (2) No such file or directory error copying pool xxx.rgw.buckets.data => xxx.rgw.buckets.data.new: (2) No such file or directory Is "rados cppool" still the blessed way to do the migration, or has something better/not deprecated been developed that I can use? Thanks, Mac

4 4

Re: MDS crashes to damaged metadata
by Patrick Donnelly 10 Jun '24

10 Jun '24

On Thu, Dec 15, 2022 at 9:32 AM Stolte, Felix <f.stolte(a)fz-juelich.de> wrote: > > Hi Patrick, > > we used your script to repair the damaged objects on the weekend and it went smoothly. Thanks for your support. > > We adjusted your script to scan for damaged files on a daily basis, runtime is about 6h. Until thursday last week, we had exactly the same 17 Files. On thursday at 13:05 a snapshot was created and our active mds crashed once at this time (snapshot was created): > > 2022-12-08T13:05:48.919+0100 7f440afec700 -1 /build/ceph-16.2.10/src/mds/ScatterLock.h: In function 'void ScatterLock::set_xlock_snap_sync(MDSContext*)' thread 7f440afec700 time 2022-12-08T13:05:48.921223+0100 > /build/ceph-16.2.10/src/mds/ScatterLock.h: 59: FAILED ceph_assert(state LOCK_XLOCK || state LOCK_XLOCKDONE) > > 12 Minutes lates the unlink_local error crashes appeared again. This time with a new file. During debugging we noticed a MTU mismatch between MDS (1500) and client (9000) with cephfs kernel mount. The client is also creating the snapshots via mkdir in the .snap directory. > > We disabled snapshot creation for now, but really need this feature. I uploaded the mds logs of the first crash along with the information above to https://tracker.ceph.com/issues/38452 > > I would greatly appreciate it, if you could answer me the following question: > > Is the Bug related to our MTU Mismatch? We fixed the MTU Issue going back to 1500 on all nodes in the ceph public network on the weekend also. I doubt it. > If you need a debug level 20 log of the ScatterLock for further analysis, i could schedule snapshots at the end of our workdays and increase the debug level 5 Minutes arround snap shot creation. This would be very helpful! -- Patrick Donnelly, Ph.D. He / Him / His Principal Software Engineer Red Hat, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

2 4

16.2.14: [progress WARNING root] complete: ev {UUID} does not exist
by Zakhar Kirpichenko 03 May '24

03 May '24

Hi, Mgr of my cluster logs this every few seconds: [progress WARNING root] complete: ev 7de5bb74-790b-4fda-8838-e4af4af18c62 does not exist [progress WARNING root] complete: ev fff93fce-b630-4141-81ee-19e7a3e61483 does not exist [progress WARNING root] complete: ev a02f6966-5b9f-49e8-89c4-b4fb8e6f4423 does not exist [progress WARNING root] complete: ev 8d318560-ff1a-477f-9386-43f6b51080bf does not exist [progress WARNING root] complete: ev ff3740a9-6434-470a-808f-a2762fb542a0 does not exist [progress WARNING root] complete: ev 7d0589f1-545e-4970-867b-8482ce48d7f0 does not exist [progress WARNING root] complete: ev 78d57e43-5be5-43f0-8b1a-cdc60e410892 does not exist I would appreciate an advice on what these warnings mean and how they can be resolved. Best regards, Zakhar

4 7

Re: Recoveries without any misplaced objects?
by Hector Martin 24 Apr '24

24 Apr '24

On 29/05/2023 20.55, Anthony D'Atri wrote: > Check the uptime for the OSDs in question I restarted all my OSDs within the past 10 days or so. Maybe OSD restarts are somehow breaking these stats? > >> On May 29, 2023, at 6:44 AM, Hector Martin <marcan(a)marcan.st> wrote: >> >> Hi, >> >> I'm watching a cluster finish a bunch of backfilling, and I noticed that >> quite often PGs end up with zero misplaced objects, even though they are >> still backfilling. >> >> Right now the cluster is down to 6 backfilling PGs: >> >> data: >> volumes: 1/1 healthy >> pools: 6 pools, 268 pgs >> objects: 18.79M objects, 29 TiB >> usage: 49 TiB used, 25 TiB / 75 TiB avail >> pgs: 262 active+clean >> 6 active+remapped+backfilling >> >> But there are no misplaced objects, and the misplaced column in `ceph pg >> dump` is zero for all PGs. >> >> If I do a `ceph pg dump_json`, I can see `num_objects_recovered` >> increasing for these PGs... but the misplaced count is still 0. >> >> Is there something else that would cause recoveries/backfills other than >> misplaced objects? Or perhaps there is a bug somewhere causing the >> misplaced object count to be misreported as 0 sometimes? >> >> # ceph -v >> ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy >> (stable) >> >> - Hector >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > - Hector

3 3

Status of IPv4 / IPv6 dual stack?
by Robert Sander 23 Apr '24

23 Apr '24

Hi, as the documentation sends mixed signals in https://docs.ceph.com/en/latest/rados/configuration/network-config-ref/#ipv… "Note Binding to IPv4 is enabled by default, so if you just add the option to bind to IPv6 you’ll actually put yourself into dual stack mode." and https://docs.ceph.com/en/latest/rados/configuration/msgr2/#address-formats "Note The ability to bind to multiple ports has paved the way for dual-stack IPv4 and IPv6 support. That said, dual-stack operation is not yet supported as of Quincy v17.2.0." just the quick questions: Is a dual stacked networking with IPv4 and IPv6 now supported or not? From which version on is it considered stable? Are OSDs now able to register themselves with two IP addresses in the cluster map? MONs too? Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

6 6

Ceph 16.2.x mon compactions, disk writes
by Zakhar Kirpichenko 17 Apr '24

17 Apr '24

Hi, Monitors in our 16.2.14 cluster appear to quite often run "manual compaction" tasks: debug 2023-10-09T09:30:53.888+0000 7f48a329a700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1696843853892760, "job": 64225, "event": "flush_started", "num_memtables": 1, "num_entries": 715, "num_deletes": 251, "total_data_size": 3870352, "memory_usage": 3886744, "flush_reason": "Manual Compaction"} debug 2023-10-09T09:30:53.904+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:30:53.908+0000 7f48a3a9b700 4 rocksdb: (Original Log Time 2023/10/09-09:30:53.910204) [db_impl/db_impl_compaction_flush.cc:2516] [default] Manual compaction from level-0 to level-5 from 'paxos .. 'paxos; will stop at (end) debug 2023-10-09T09:30:53.908+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:30:53.908+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:30:53.908+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:30:53.908+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:30:53.908+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:30:53.908+0000 7f48a3a9b700 4 rocksdb: (Original Log Time 2023/10/09-09:30:53.911004) [db_impl/db_impl_compaction_flush.cc:2516] [default] Manual compaction from level-5 to level-6 from 'paxos .. 'paxos; will stop at (end) debug 2023-10-09T09:32:08.956+0000 7f48a329a700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1696843928961390, "job": 64228, "event": "flush_started", "num_memtables": 1, "num_entries": 1580, "num_deletes": 502, "total_data_size": 8404605, "memory_usage": 8465840, "flush_reason": "Manual Compaction"} debug 2023-10-09T09:32:08.972+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:08.976+0000 7f48a3a9b700 4 rocksdb: (Original Log Time 2023/10/09-09:32:08.977739) [db_impl/db_impl_compaction_flush.cc:2516] [default] Manual compaction from level-0 to level-5 from 'logm .. 'logm; will stop at (end) debug 2023-10-09T09:32:08.976+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:08.976+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:08.976+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:08.976+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:08.976+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:08.976+0000 7f48a3a9b700 4 rocksdb: (Original Log Time 2023/10/09-09:32:08.978512) [db_impl/db_impl_compaction_flush.cc:2516] [default] Manual compaction from level-5 to level-6 from 'logm .. 'logm; will stop at (end) debug 2023-10-09T09:32:12.764+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:12.764+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:12.764+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:12.764+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:12.764+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:12.764+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:33:29.028+0000 7f48a329a700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1696844009033151, "job": 64231, "event": "flush_started", "num_memtables": 1, "num_entries": 1430, "num_deletes": 251, "total_data_size": 8975535, "memory_usage": 9035920, "flush_reason": "Manual Compaction"} debug 2023-10-09T09:33:29.044+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:33:29.048+0000 7f48a3a9b700 4 rocksdb: (Original Log Time 2023/10/09-09:33:29.049585) [db_impl/db_impl_compaction_flush.cc:2516] [default] Manual compaction from level-0 to level-5 from 'paxos .. 'paxos; will stop at (end) debug 2023-10-09T09:33:29.048+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:33:29.048+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:33:29.048+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:33:29.048+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:33:29.048+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:33:29.048+0000 7f48a3a9b700 4 rocksdb: (Original Log Time 2023/10/09-09:33:29.050355) [db_impl/db_impl_compaction_flush.cc:2516] [default] Manual compaction from level-5 to level-6 from 'paxos .. 'paxos; will stop at (end) I have removed a lot of interim log messages to save space. During each compaction the monitor process writes approximately 500-600 MB of data to disk over a short period of time. These writes add up to tens of gigabytes per hour and hundreds of gigabytes per day. Monitor rocksdb and compaction options are default: "mon_compact_on_bootstrap": "false", "mon_compact_on_start": "false", "mon_compact_on_trim": "true", "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", Is this expected behavior? Is this something I can adjust in order to extend the system storage life? Best regards, Zakhar

5 32

Setting S3 bucket policies with multi-tenants
by Thomas Bennett 15 Apr '24

15 Apr '24

Hi, I'm running Ceph Quincy (17.2.6) with a rados-gateway. I have muti tenants, for example: - Tenant1$manager - Tenant1$readwrite I would like to set a policy on a bucket (backups for example) owned by *Tenant1$manager* to allow *Tenant1$readwrite* access to that bucket. I can't find any documentation that discusses this scenario. Does anyone know how to specify the Principle and Resource section of a policy.json file? Or any other configuration that I might be missing? I've tried some variations on Principal and Resource including and excluding tenant information, but not no luck yet. For example: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": {"AWS": ["arn:aws:iam:::user/*Tenant1$readwrite*"]}, "Action": ["s3:ListBucket","s3:GetObject", ,"s3:PutObject"], "Resource": [ "arn:aws:s3:::*Tenant1/backups*" ] }] } I'm using s3cmd for testing, so: s3cmd --config s3cfg.manager setpolicy policy.json s3://backups/ Returns: s3://backups/: Policy updated And then testing: s3cmd --config s3cfg.readwrite ls s3://backups/ ERROR: Access to bucket 'backups' was denied ERROR: S3 error: 403 (AccessDenied) Thanks, Tom

4 4

A couple OSDs not starting after host reboot
by Alison Peisker 11 Apr '24

11 Apr '24

Hi all, We rebooted all the nodes in our 17.2.5 cluster after performing kernel updates, but 2 of the OSDs on different nodes are not coming back up. This is a production cluster using cephadm. The error message from the OSD log is ceph-osd[87340]: ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-665: (2) No such file or directory The error message from ceph-volume is 2023-08-23T16:12:43.452-0500 7f0cad968600 2 bluestore(/dev/mapper/ceph--febad5a5--ba44--41aa--a39e--b9897f757752-osd--block--87e548f4--b9b5--4ed8--aca8--de703a341a50) _read_bdev_label unable to decode label at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input We tried restarting the daemons and rebooting the node again, but still see the same error. Has anyone experienced this issue before? How do we fix this? Thanks, Alison

9 13

cephfs creation error
by Ramanathan S 03 Apr '24

03 Apr '24

Hi all, I just had created a ceph cluster to use cephfs. When i create the a ceph fs pool i get the filesystem below error. # ceph osd pool create cephfs_data 128 pool 'cephfs_data' created # ceph osd pool create cephfs_metadata 128 pool 'cephfs_metadata' created # ceph fs new cephfs cephfs_metadata cephfs_data new fs with metadata pool 6 and data pool 5 # ceph -s cluster: id: 1c27def45-f0f9-494d-sfke-eb4323432fd health: HEALTH_ERR 1 filesystem is offline 1 filesystem is online with fewer MDS than max_mds services: mon: 2 daemons, quorum ceph-mon01,ceph-mon02 mgr: ceph-adm01(active) mds: cephfs-0/0/1 up osd: 12 osds: 12 up, 12 in data: pools: 2 pools, 256 pgs objects: 0 objects, 0 B usage: 12 GiB used, 588 GiB / 600 GiB avail pgs: 256 active+clean but when i check the max_mds for the ceph fs it says 1 # ceph fs get cephfs | grep max_mds max_mds 1 Let anyone know what am i missing here? Any inputs is much appreciated. Regards, Ram Ceph-explorer..

3 3

Re: NoSuchKey on key that is visible in s3 list/radosgw bk
by Eric Ivancich 03 Apr '24

03 Apr '24

I have some questions for those who’ve experienced this issue. 1. It seems like those reporting this issue are seeing it strictly after upgrading to Octopus. From what version did each of these sites upgrade to Octopus? From Nautilus? Mimic? Luminous? 2. Does anyone have any lifecycle rules on a bucket experiencing this issue? If so, please describe. 3. Is anyone making copies of the affected objects (to same or to a different bucket) prior to seeing the issue? And if they are making copies, does the destination bucket have lifecycle rules? And if they are making copies, are those copies ever being removed? 4. Is anyone experiencing this issue willing to run their RGWs with 'debug_ms=1'? That would allow us to see a request from an RGW to either remove a tail object or decrement its reference counter (and when its counter reaches 0 it will be deleted). Thanks, Eric > On Nov 12, 2020, at 4:54 PM, huxiaoyu(a)horebdata.cn wrote: > > Looks like this is a very dangerous bug for data safety. Hope the bug would be quickly identified and fixed. > > best regards, > > Samuel > > > > huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn> > > From: Janek Bevendorff > Date: 2020-11-12 18:17 > To: huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn>; EDH - Manuel Rios; Rafael Lopez > CC: Robin H. Johnson; ceph-users > Subject: Re: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > I have never seen this on Luminous. I recently upgraded to Octopus and the issue started occurring only few weeks later. > > On 12/11/2020 16:37, huxiaoyu(a)horebdata.cn wrote: > which Ceph versions are affected by this RGW bug/issues? Luminous, Mimic, Octupos, or the latest? > > any idea? > > samuel > > > > huxiaoyu(a)horebdata.cn > > From: EDH - Manuel Rios > Date: 2020-11-12 14:27 > To: Janek Bevendorff; Rafael Lopez > CC: Robin H. Johnson; ceph-users > Subject: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > This same error caused us to wipe a full cluster of 300TB... will be related to some rados index/database bug not to s3. > > As Janek exposed is a mayor issue, because the error silent happend and you can only detect it with S3, when you're going to delete/purge a S3 bucket. Dropping NoSuchKey. Error is not related to S3 logic .. > > Hope this time dev's can take enought time to find and resolve the issue. Error happens with low ec profiles, even with replica x3 in some cases. > > Regards > > > > -----Mensaje original----- > De: Janek Bevendorff <janek.bevendorff(a)uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de>> > Enviado el: jueves, 12 de noviembre de 2020 14:06 > Para: Rafael Lopez <rafael.lopez(a)monash.edu <mailto:rafael.lopez@monash.edu>> > CC: Robin H. Johnson <robbat2(a)gentoo.org <mailto:robbat2@gentoo.org>>; ceph-users <ceph-users(a)ceph.io <mailto:ceph-users@ceph.io>> > Asunto: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > > Here is a bug report concerning (probably) this exact issue: > https://tracker.ceph.com/issues/47866 <https://tracker.ceph.com/issues/47866> > > I left a comment describing the situation and my (limited) experiences with it. > > > On 11/11/2020 10:04, Janek Bevendorff wrote: >> >> Yeah, that seems to be it. There are 239 objects prefixed >> .8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh in my dump. However, there are none >> of the multiparts from the other file to be found and the head object >> is 0 bytes. >> >> I checked another multipart object with an end pointer of 11. >> Surprisingly, it had way more than 11 parts (39 to be precise) named >> .1, .1_1 .1_2, .1_3, etc. Not sure how Ceph identifies those, but I >> could find them in the dump at least. >> >> I have no idea why the objects disappeared. I ran a Spark job over all >> buckets, read 1 byte of every object and recorded errors. Of the 78 >> buckets, two are missing objects. One bucket is missing one object, >> the other 15. So, luckily, the incidence is still quite low, but the >> problem seems to be expanding slowly. >> >> >> On 10/11/2020 23:46, Rafael Lopez wrote: >>> Hi Janek, >>> >>> What you said sounds right - an S3 single part obj won't have an S3 >>> multipart string as part of the prefix. S3 multipart string looks >>> like "2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme". >>> >>> From memory, single part S3 objects that don't fit in a single rados >>> object are assigned a random prefix that has nothing to do with >>> the object name, and the rados tail/data objects (not the head >>> object) have that prefix. >>> As per your working example, the prefix for that would be >>> '.8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh'. So there would be (239) "shadow" >>> objects with names containing that prefix, and if you add up the >>> sizes it should be the size of your S3 object. >>> >>> You should look at working and non working examples of both single >>> and multipart S3 objects, as they are probably all a bit different >>> when you look in rados. >>> >>> I agree it is a serious issue, because once objects are no longer in >>> rados, they cannot be recovered. If it was a case that there was a >>> link broken or rados objects renamed, then we could work to >>> recover...but as far as I can tell, it looks like stuff is just >>> vanishing from rados. The only explanation I can think of is some >>> (rgw or rados) background process is incorrectly doing something with >>> these objects (eg. renaming/deleting). I had thought perhaps it was a >>> bug with the rgw garbage collector..but that is pure speculation. >>> >>> Once you can articulate the problem, I'd recommend logging a bug >>> tracker upstream. >>> >>> >>> On Wed, 11 Nov 2020 at 06:33, Janek Bevendorff >>> <janek.bevendorff(a)uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de> >>> <mailto:janek.bevendorff@uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de>>> wrote: >>> >>> Here's something else I noticed: when I stat objects that work >>> via radosgw-admin, the stat info contains a "begin_iter" JSON >>> object with RADOS key info like this >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110924034843-crawl420/WIDE-20110924065228-02544.warc.gz", >>> "instance": "", >>> "ns": "" >>> } >>> >>> >>> and then "end_iter" with key info like this: >>> >>> >>> "key": { >>> "name": >>> ".8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh_239", >>> "instance": "", >>> "ns": "shadow" >>> } >>> >>> However, when I check the broken 0-byte object, the "begin_iter" >>> and "end_iter" keys look like this: >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.1", >>> "instance": "", >>> "ns": "multipart" >>> } >>> >>> [...] >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.19", >>> "instance": "", >>> "ns": "multipart" >>> } >>> >>> So, it's the full name plus a suffix and the namespace is >>> multipart, not shadow (or empty). This in itself may just be an >>> artefact of whether the object was uploaded in one go or as a >>> multipart object, but the second difference is that I cannot find >>> any of the multipart objects in my pool's object name dump. I >>> can, however, find the shadow RADOS object of the intact S3 object. >>> >>> >>> >>> >>> -- >>> *Rafael Lopez* >>> Devops Systems Engineer >>> Monash University eResearch Centre >>> >>> T: +61 3 9905 9118 <tel:%2B61%203%209905%209118> >>> E: rafael.lopez(a)monash.edu <mailto:rafael.lopez@monash.edu> >>> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

6 22

cephfs inode backtrace information
by Dietmar Rieder 02 Apr '24

02 Apr '24

Hello, I have a question regarding the default pool of a cephfs. According to the docs it is recommended to use a fast ssd replicated pool as default pool for cephfs. I'm asking what are the space requirements for storing the inode backtrace information? Let's say I have a 85 TiB replicated ssd pool (hot data) and as 3 PiB EC data pool (cold data). Does it make sense to create a third pool as default pool which only holds the inode backtrace information (what would be a good size), or is it OK to use the ssd pool as default pool? Thanks Dietmar

6 10

Why you might want packages not containers for Ceph deployments
by Matthew Vernon 25 Mar '24

25 Mar '24

Hi, In the discussion after the Ceph Month talks yesterday, there was a bit of chat about cephadm / containers / packages. IIRC, Sage observed that a common reason in the recent user survey for not using cephadm was that it only worked on containerised deployments. I think he then went on to say that he hadn't heard any compelling reasons why not to use containers, and suggested that resistance was essentially a user education question[0]. I'd like to suggest, briefly, that: * containerised deployments are more complex to manage, and this is not simply a matter of familiarity * reducing the complexity of systems makes admins' lives easier * the trade-off of the pros and cons of containers vs packages is not obvious, and will depend on deployment needs * Ceph users will benefit from both approaches being supported into the future We make extensive use of containers at Sanger, particularly for scientific workflows, and also for bundling some web apps (e.g. Grafana). We've also looked at a number of container runtimes (Docker, singularity, charliecloud). They do have advantages - it's easy to distribute a complex userland in a way that will run on (almost) any target distribution; rapid "cloud" deployment; some separation (via namespaces) of network/users/processes. For what I think of as a 'boring' Ceph deploy (i.e. install on a set of dedicated hardware and then run for a long time), I'm not sure any of these benefits are particularly relevant and/or compelling - Ceph upstream produce Ubuntu .debs and Canonical (via their Ubuntu Cloud Archive) provide .debs of a couple of different Ceph releases per Ubuntu LTS - meaning we can easily separate out OS upgrade from Ceph upgrade. And upgrading the Ceph packages _doesn't_ restart the daemons[1], meaning that we maintain control over restart order during an upgrade. And while we might briefly install packages from a PPA or similar to test a bugfix, we roll those (test-)cluster-wide, rather than trying to run a mixed set of versions on a single cluster - and I understand this single-version approach is best practice. Deployment via containers does bring complexity; some examples we've found at Sanger (not all Ceph-related, which we run from packages): * you now have 2 process supervision points - dockerd and systemd * docker updates (via distribution unattended-upgrades) have an unfortunate habit of rudely restarting everything * docker squats on a chunk of RFC 1918 space (and telling it not to can be a bore), which coincides with our internal network... * there is more friction if you need to look inside containers (particularly if you have a lot running on a host and are trying to find out what's going on) * you typically need to be root to build docker containers (unlike packages) * we already have package deployment infrastructure (which we'll need regardless of deployment choice) We also currently use systemd overrides to tweak some of the Ceph units (e.g. to do some network sanity checks before bringing up an OSD), and have some tools to pair OSD / journal / LVM / disk device up; I think these would be more fiddly in a containerised deployment. I'd accept that fixing these might just be a SMOP[2] on our part. Now none of this is show-stopping, and I am most definitely not saying "don't ship containers". But I think there is added complexity to your deployment from going the containers route, and that is not simply a "learn how to use containers" learning curve. I do think it is reasonable for an admin to want to reduce the complexity of what they're dealing with - after all, much of my job is trying to automate or simplify the management of complex systems! I can see from a software maintainer's point of view that just building one container and shipping it everywhere is easier than building packages for a number of different distributions (one of my other hats is a Debian developer, and I have a bunch of machinery for doing this sort of thing). But it would be a bit unfortunate if the general thrust of "let's make Ceph easier to set up and manage" was somewhat derailed with "you must use containers, even if they make your life harder". I'm not going to criticise anyone who decides to use a container-based deployment (and I'm sure there are plenty of setups where it's an obvious win), but if I were advising someone who wanted to set up and use a 'boring' Ceph cluster for the medium term, I'd still advise on using packages. I don't think this makes me a luddite :) Regards, and apologies for the wall of text, Matthew [0] I think that's a fair summary! [1] This hasn't always been true... [2] Simple (sic.) Matter of Programming -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.

26 65

Cephfs error state with one bad file
by Sake Ceph 15 Mar '24

15 Mar '24

Hi again, hopefully for the last time with problems. We had a MDS crash earlier with the MDS staying in failed state and used a command to reset the filesystem (this was wrong, I know now, thanks Patrick Donnelly for pointing this out). I did a full scrub on the filesystem and two files were damaged. One of those got repaired, but the following file keeps giving errors and can't be removed. What can I do now? Below some information. # ceph tell mds.atlassian-prod:0 damage ls [ { "damage_type": "backtrace", "id": 2244444901, "ino": 1099534008829, "path": "/app1/shared/data/repositories/11271/objects/41/8f82507a0737c611720ed224bcc8b7a24fda01" } ] Trying to repair the error (online research shows this should work for a backtrace damage type) ---------- # ceph tell mds.atlassian-prod:0 scrub start /app1/shared/data/repositories/11271 recursive,repair,force { "return_code": 0, "scrub_tag": "d10ead42-5280-4224-971e-4f3022e79278", "mode": "asynchronous" } Cluster logs after this ---------- 1/2/24 9:37:05 AM [INF] scrub summary: idle 1/2/24 9:37:02 AM [INF] scrub summary: idle+waiting paths [/app1/shared/data/repositories/11271] 1/2/24 9:37:01 AM [INF] scrub summary: active paths [/app1/shared/data/repositories/11271] 1/2/24 9:37:01 AM [INF] scrub summary: idle+waiting paths [/app1/shared/data/repositories/11271] 1/2/24 9:37:01 AM [INF] scrub queued for path: /app1/shared/data/repositories/11271 But the error doesn't disappear and still can't remove the file. On the client trying to remove the file (we got a backup) ---------- $ rm -f /mnt/shared_disk-app1/shared/data/repositories/11271/objects/41/8f82507a0737c611720ed224bcc8b7a24fda01 rm: cannot remove '/mnt/shared_disk-app1/shared/data/repositories/11271/objects/41/8f82507a0737c611720ed224bcc8b7a24fda01': Input/output error Best regards, Sake

2 1

MDS subtree pinning
by Sake Ceph 15 Mar '24

15 Mar '24

Hi! As I'm reading through the documentation about subtree pinning, I was wondering if the following is possible. We've got the following directory structure. / /app1 /app2 /app3 /app4 Can I pin /app1 to MDS rank 0 and 1, the directory /app2 to rank 2 and finally /app3 and /app4 to rank 3? I would like to load balance the subfolders of /app1 to 2 (or 3) MDS servers. Best regards, Sake

2 2

Upgrading from Pacific to Quincy fails with "Unexpected error"
by Reza Bakhshayeshi 13 Mar '24

13 Mar '24

Hi all, I have a problem regarding upgrading Ceph cluster from Pacific to Quincy version with cephadm. I have successfully upgraded the cluster to the latest Pacific (16.2.11). But when I run the following command to upgrade the cluster to 17.2.5, After upgrading 3/4 mgrs, the upgrade process stops with "Unexpected error". (everything is on a private network) ceph orch upgrade start my-private-repo/quay-io/ceph/ceph:v17.2.5 I also tried the 17.2.4 version. cephadm fails to check the hosts' status and marks them as offline: cephadm 2023-04-06T10:19:59.998510+0000 mgr.host9.arhpnd (mgr.4516356) 5782 : cephadm [DBG] host host4 (x.x.x.x) failed check cephadm 2023-04-06T10:19:59.998553+0000 mgr.host9.arhpnd (mgr.4516356) 5783 : cephadm [DBG] Host "host4" marked as offline. Skipping daemon refresh cephadm 2023-04-06T10:19:59.998581+0000 mgr.host9.arhpnd (mgr.4516356) 5784 : cephadm [DBG] Host "host4" marked as offline. Skipping gather facts refresh cephadm 2023-04-06T10:19:59.998609+0000 mgr.host9.arhpnd (mgr.4516356) 5785 : cephadm [DBG] Host "host4" marked as offline. Skipping network refresh cephadm 2023-04-06T10:19:59.998633+0000 mgr.host9.arhpnd (mgr.4516356) 5786 : cephadm [DBG] Host "host4" marked as offline. Skipping device refresh cephadm 2023-04-06T10:19:59.998659+0000 mgr.host9.arhpnd (mgr.4516356) 5787 : cephadm [DBG] Host "host4" marked as offline. Skipping osdspec preview refresh cephadm 2023-04-06T10:19:59.998682+0000 mgr.host9.arhpnd (mgr.4516356) 5788 : cephadm [DBG] Host "host4" marked as offline. Skipping autotune cluster 2023-04-06T10:20:00.000151+0000 mon.host8 (mon.0) 158587 : cluster [ERR] Health detail: HEALTH_ERR 9 hosts fail cephadm check; Upgrade: failed due to an unexpected exception cluster 2023-04-06T10:20:00.000191+0000 mon.host8 (mon.0) 158588 : cluster [ERR] [WRN] CEPHADM_HOST_CHECK_FAILED: 9 hosts fail cephadm check cluster 2023-04-06T10:20:00.000202+0000 mon.host8 (mon.0) 158589 : cluster [ERR] host host7 (x.x.x.x) failed check: Unable to reach remote host host7. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000213+0000 mon.host8 (mon.0) 158590 : cluster [ERR] host host2 (x.x.x.x) failed check: Unable to reach remote host host2. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000220+0000 mon.host8 (mon.0) 158591 : cluster [ERR] host host8 (x.x.x.x) failed check: Unable to reach remote host host8. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000228+0000 mon.host8 (mon.0) 158592 : cluster [ERR] host host4 (x.x.x.x) failed check: Unable to reach remote host host4. Process exited with non-zero exit status 3 cluster 2023-04-06T10:20:00.000240+0000 mon.host8 (mon.0) 158593 : cluster [ERR] host host3 (x.x.x.x) failed check: Unable to reach remote host host3. Process exited with non-zero exit status 3 and here are some outputs of the commands: [root@host8 ~]# ceph -s cluster: id: xxx health: HEALTH_ERR 9 hosts fail cephadm check Upgrade: failed due to an unexpected exception services: mon: 5 daemons, quorum host8,host1,host7,host2,host9 (age 2w) mgr: host9.arhpnd(active, since 105m), standbys: host8.jowfih, host1.warjsr, host2.qyavjj mds: 1/1 daemons up, 3 standby osd: 37 osds: 37 up (since 8h), 37 in (since 3w) data: io: client: progress: Upgrade to 17.2.5 (0s) [............................] [root@host8 ~]# ceph orch upgrade status { "target_image": "my-private-repo/quay-io/ceph/ceph@sha256 :34c763383e3323c6bb35f3f2229af9f466518d9db926111277f5e27ed543c427", "in_progress": true, "which": "Upgrading all daemon types on all hosts", "services_complete": [], "progress": "3/59 daemons upgraded", "message": "Error: UPGRADE_EXCEPTION: Upgrade: failed due to an unexpected exception", "is_paused": true } [root@host8 ~]# ceph cephadm check-host host7 check-host failed: Host 'host7' not found. Use 'ceph orch host ls' to see all managed hosts. [root@host8 ~]# ceph versions { "mon": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 5 }, "mgr": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 1, "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)": 3 }, "osd": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 37 }, "mds": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 4 }, "overall": { "ceph version 16.2.11 (3cf40e2dca667f68c6ce3ff5cd94f01e711af894) pacific (stable)": 47, "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)": 3 } } The strange thing is I can rollback the cluster status by failing to not-upgraded mgr like this: ceph mgr fail ceph orch upgrade start my-private-repo/quay-io/ceph/ceph:v16.2.11 Would you happen to have any idea about this? Best regards, Reza

4 9

Disable signature url in ceph rgw
by marc＠singer.services 07 Mar '24

07 Mar '24

Hi Ceph users We are using Ceph Pacific (16) in this specific deployment. In our use case we do not want our users to be able to generate signature v4 URLs because they bypass the policies that we set on buckets (e.g IP restrictions). Currently we have a sidecar reverse proxy running that filters requests with signature URL specific request parameters. This is obviously not very efficient and we are looking to replace this somehow in the future. 1. Is there an option in RGW to disable this signed URLs (e.g returning status 403)? 2. If not is this planned or would it make sense to add it as a configuration option? 3. Or is the behaviour of not respecting bucket policies in RGW with signature v4 URLs a bug and they should be actually applied? Thanks you for your help and let me know if you have any questions Marc Singer

4 4

erasure-code-lrc Questions regarding repair
by Ansgar Jazdzewski 07 Mar '24

07 Mar '24

hi folks, I currently test erasure-code-lrc (1) in a multi-room multi-rack setup. The idea is to be able to repair a disk-failures within the rack itself to lower bandwidth-usage ```bash ceph osd erasure-code-profile set lrc_hdd \ plugin=lrc \ crush-root=default \ crush-locality=rack \ crush-failure-domain=host \ crush-device-class=hdd \ mapping=__DDDDD__DDDDD__DDDDD__DDDDD \ layers=' [ [ "_cDDDDD_cDDDDD_cDDDDD_cDDDDD", "" ], [ "cDDDDDD_____________________", "" ], [ "_______cDDDDDD______________", "" ], [ "______________cDDDDDD_______", "" ], [ "_____________________cDDDDDD", "" ], ]' \ crush-steps='[ [ "choose", "room", 4 ], [ "choose", "rack", 1 ], [ "chooseleaf", "host", 7 ], ]' ``` The roule picks 4 out of 5 rooms and keeps the PG in one rack like expected! However it looks like the PG will not move to another Room if the PG is undersized or the entire Room or Rack is down! Questions: * do I miss something to allow LRC (PG's) to move across Racks/Rooms for repair? * Is it even possible to build such a 'Multi-stage' grushmap? Thanks for your help, Ansgar 1) https://docs.ceph.com/en/quincy/rados/operations/erasure-code-jerasure/

2 2

pacific 16.2.15 QE validation status
by Yuri Weinstein 26 Feb '24

26 Feb '24

Details of this release are summarized here: https://tracker.ceph.com/issues/64151#note-1 Seeking approvals/reviews for: rados - Radek, Laura, Travis, Ernesto, Adam King rgw - Casey fs - Venky rbd - Ilya krbd - in progress upgrade/nautilus-x (pacific) - Casey PTL (regweed tests failed) upgrade/octopus-x (pacific) - Casey PTL (regweed tests failed) upgrade/pacific-x (quincy) - in progress upgrade/pacific-p2p - Ilya PTL (maybe rbd related?) ceph-volume - Guillaume TIA YuriW

12 27

FS down - mds degraded
by Sake Ceph 23 Feb '24

23 Feb '24

Starting a new thread, forgot subject in the previous. So our FS down. Got the following error, what can I do? # ceph health detail HEALTH_ERR 1 filesystem is degraded; 1 mds daemon damaged [WRN] FS_DEGRADED: 1 filesystem is degraded fs atlassian/prod is degraded [ERR] MDS_DAMAGE: 1 mds daemon damaged fs atlassian-prod mds.1 is damaged # ceph fs get atlassian-prod Filesystem 'atlassian-prod' (2) fs_name atlassian-prod epoch 43440 flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay created 2023-05-10T08:45:46.911064+0000 modified 2023-12-21T06:47:19.291154+0000 tableserver 0 root 0 session_timeout 60 session_autoclose 300 max_file_size 1099511627776 required_client_features {} last_failure 0 last_failure_osd_epoch 29480 compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} max_mds 3 in 0,1,2 up {0=1073573,2=1073583} failed damaged 1 stopped data_pools [5] metadata_pool 4 inline_data disabled balancer standby_count_wanted 1 [mds.atlassian-prod.pwsoel13142.egsdfl{0:1073573} state up:resolve seq 573 join_fscid=2 addr [v2:10.233.127.22:6800/61692284,v1:10.233.127.22:6801/61692284] compat {c=[1],r=[1],i=[7ff]}] [mds.atlassian-prod.pwsoel13143.qlvypn{2:1073583} state up:resolve seq 571 join_fscid=2 addr [v2:10.233.127.18:6800/3627858294,v1:10.233.127.18:6801/3627858294] compat {c=[1],r=[1],i=[7ff]}] Best regards, Sake

4 7

Re: Debian 12 (bookworm) / Reef 18.2.1 problems
by Chris Palmer 21 Feb '24

21 Feb '24

I have logged this as https://tracker.ceph.com/issues/64213 On 16/01/2024 14:18, DERUMIER, Alexandre wrote: > Hi, > >>> ImportError: PyO3 modules may only be initialized once per >>> interpreter >>> process >>> >>> and ceph -s reports "Module 'dashboard' has failed dependency: PyO3 >>> modules may only be initialized once per interpreter process > We have the same problem on proxmox8 (based on debian12) with ceph > quincy or reef. > > It seem to be related to python version on debian12 > > (we have no fix for this currently) > > >

6 7

Ceph as rootfs?
by Jeremy Hansen 06 Feb '24

06 Feb '24

Is it possible to use Ceph as a root filesystem for a pxe booted host? Thanks

5 4

Throughput metrics missing iwhen updating Ceph Quincy to Reef
by Jose Vicente 05 Feb '24

05 Feb '24

4 8

OSD read latency grows over time
by Roman Pashin 02 Feb '24

02 Feb '24

Hello Ceph users, we see strange issue on last recent Ceph installation v17.6.2. We store data on HDD pool, index pool is on SSD. Each OSD store its wal on NVME partition. Benchmarks didn't expose any issues with cluster, but since we placed production load on it we see constantly growing OSD latency time (osd_read_latency) on SSD disks (where Index pool located). Latency is constantly growing day-by-day, but disks are not utilized even for 50%. Interesting, that when we move Index pool from SSD to NVME disks (disk space allows it for now) - osd latency drops to zero and start increasing from the ground. Also we noticed, that any change of pg_num for index pool (from 256 to 128 for instance) also drops latency to zero. And it starts its growth again (https://postimg.cc/5YHk9bby) From client perspective it looks like one operation takes longer and longer each other day and operation time drops each time when we do some change on index pool. I've enabled debug_optracker 10/0 and it shows, that OSD spend most time in `queued_for_pg` state, but physical disk utilization is about 10-20%. Also per logs I see, that longest operation is Listbucket, but it is strange, that with less than 100'000 items in bucket list even with 'max_keys=1' takes 3-40 seconds. If it matters client is Apache Flink doing checkpoints via S3 protocol. Here is an example of operation with debug_optracking logs: 2023-12-29T16:24:28.873353+0300, event: throttled, op: osd_op(client.1227774 .0:22575820 7.19 7.a84f1a59 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e83833) 2023-12-29T16:24:28.861549+0300, event: header_read, op: osd_op(client. 1227774.0:22575820 7.19 7.a84f1a59 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e83833) 2023-12-29T16:24:28.873358+0300, event: all_read, op: osd_op(client.1227774 .0:22575820 7.19 7.a84f1a59 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e83833) 2023-12-29T16:24:28.873359+0300, event: dispatched, op: osd_op(client. 1227774.0:22575820 7.19 7.a84f1a59 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e83833) 2023-12-29T16:24:28.873389+0300, event: queued_for_pg, op: osd_op(client. 1227774.0:22575820 7.19 7.a84f1a59 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e83833) 2023-12-29T16:24:38.077528+0300, event: reached_pg, op: osd_op(client. 1227774.0:22575820 7.19 7.a84f1a59 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e83833) 2023-12-29T16:24:38.077561+0300, event: started, op: osd_op(client.1227774 .0:22575820 7.19 7:9a58f215:::.dir.68960da3-1c98-45c1-a87a-9e6c39253d27. 9780498.1.83:head [stat,call rgw.guard_bucket_resharding in=36b,call rgw.bucket_prepare_op in=331b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83833) 2023-12-29T16:24:38.077714+0300, event: waiting for subops from 59,494, op: osd_op(client.1227774.0:22575820 7.19 7:9a58f215:::.dir.68960da3-1c98-45c1-a87a-9e6c39253d27.9780498.1.83:head [stat,call rgw.guard_bucket_resharding in=36b,call rgw.bucket_prepare_op in=331b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83833) 2023-12-29T16:24:38.146157+0300, event: sub_op_commit_rec, op: osd_op(client.1227774.0:22575820 7.19 7:9a58f215:::.dir.68960da3-1c98-45c1-a87a-9e6c39253d27.9780498.1.83:head [stat,call rgw.guard_bucket_resharding in=36b,call rgw.bucket_prepare_op in=331b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83833) 2023-12-29T16:24:38.146166+0300, event: op_commit, op: osd_op(client.1227774 .0:22575820 7.19 7:9a58f215:::.dir.68960da3-1c98-45c1-a87a-9e6c39253d27. 9780498.1.83:head [stat,call rgw.guard_bucket_resharding in=36b,call rgw.bucket_prepare_op in=331b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83833) 2023-12-29T16:24:38.146191+0300, event: sub_op_commit_rec, op: osd_op(client.1227774.0:22575820 7.19 7:9a58f215:::.dir.68960da3-1c98-45c1-a87a-9e6c39253d27.9780498.1.83:head [stat,call rgw.guard_bucket_resharding in=36b,call rgw.bucket_prepare_op in=331b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83833) 2023-12-29T16:24:38.146204+0300, event: commit_sent, op: osd_op(client. 1227774.0:22575820 7.19 7:9a58f215:::.dir.68960da3-1c98-45c1-a87a-9e6c39253d27.9780498.1.83:head [stat,call rgw.guard_bucket_resharding in=36b,call rgw.bucket_prepare_op in=331b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83833) 2023-12-29T16:24:38.146216+0300, event: done, op: osd_op(client.1227774.0:22575820 7.19 7:9a58f215:::.dir.68960da3-1c98-45c1-a87a-9e6c39253d27.9780498.1.83:head [stat,call rgw.guard_bucket_resharding in=36b,call rgw.bucket_prepare_op in=331b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83833) Does anybody faced with the same issue? I'd be very grateful for any idea because at this point I've got stuck with what to tune and where to look at. Cluster setup - replication 3x, 15 servers in 3 datacenters with datacenter as failure domain. 7x HDD (data), 2x SSD (index), 1x NVME (wal + OS). ceph config - https://pastebin.com/pCqxXhT3 OSD read latency graph - https://postimg.cc/5YHk9bby -- Thank you, Roman

9 17

Understanding subvolumes
by Matthew Melendy 02 Feb '24

02 Feb '24

In our department we're getting starting with Ceph 'reef', using Ceph FUSE client for our Ubuntu workstations. So far so good, except I can't quite figure out one aspect of subvolumes. When I do the commands: [root@ceph1 ~]# ceph fs subvolumegroup create cephfs csvg [root@ceph1 ~]# ceph fs subvolume create cephfs staff csvg --size 2000000000000 I get these results: - A subvolume group csvg is created on volume cephfs - A subvolume called staff is created in csvg subvolume (like /volumes/csvg/staff ) however there is no size limit set at this folder in the Ceph dashboard view - A folder with an random UUID name is created inside the subvolume staff ( like /volumes/csvg/staff/6a1b3de5-f6ab-4878-aea3-3c3c6f96ffcf ); this folder does have a size set on it of 2TB My questions are: - what's the purpose of this UUID, and is it a requirement? - which directory should be mounted for my clients, staff/ or staff/{UUID}, for the size limit to take effect? - is there any way to hide or disable this UUID for client mounts? (eg in /etc/fstab) [root@ceph1 ~]# ceph fs subvolume getpath cephfs staff csvg /volumes/csvg/staff/6a1b3de5-f6ab-4878-aea3-3c3c6f96ffcf [root@ceph1 ~]# ceph fs subvolume ls cephfs csvg [ { "name": "staff" } ] -- Sincerely, Matthew Melendy IT Services Specialist CS System Services Group FEC 3550, University of New Mexico

4 3

Cannot recreate monitor in upgrade from pacific to quincy (leveldb -> rocksdb)
by Mark Schouten 02 Feb '24

02 Feb '24

Hi, During an upgrade from pacific to quincy, we needed to recreate the mons because the mons were pretty old and still using leveldb. So step one was to destroy one of the mons. After that we recreated the monitor, and although it starts, it remains in state ‘probing’, as you can see below. No matter what I tried, it won’t come up. I’ve seen quite some messages that the MTU might be an issue, but that seems to be ok: root@proxmox03:/var/log/ceph# fping -b 1472 10.10.10.{1..3} -M 10.10.10.1 is alive 10.10.10.2 is alive 10.10.10.3 is alive Does anyone have an idea how to fix this? I’ve tried destroying and recreating the mon a few times now. Could it be that the leveldb mons only support mon.$id notation for the monitors? root@proxmox03:/var/log/ceph# ceph daemon mon.proxmox03 mon_status { "name": “proxmox03”, "rank": 2, "state": “probing”, "election_epoch": 0, "quorum": [], "features": { "required_con": “2449958197560098820”, "required_mon": [ “kraken”, “luminous”, “mimic”, "osdmap-prune”, “nautilus”, “octopus”, “pacific”, "elector-pinging” ], "quorum_con": “0”, "quorum_mon": [] }, "outside_quorum": [ “proxmox03” ], "extra_probe_peers": [], "sync_provider": [], "monmap": { "epoch": 0, "fsid": "39b1e85c-7b47-4262-9f0a-47ae91042bac”, "modified": "2024-01-23T21:02:12.631320Z”, "created": "2017-03-15T14:54:55.743017Z”, "min_mon_release": 16, "min_mon_release_name": “pacific”, "election_strategy": 1, "disallowed_leaders: ": “”, "stretch_mode": false, "tiebreaker_mon": “”, "removed_ranks: ": “2”, "features": { "persistent": [ “kraken”, “luminous”, “mimic”, "osdmap-prune”, “nautilus”, “octopus”, “pacific”, "elector-pinging” ], "optional": [] }, "mons": [ { "rank": 0, "name": “0”, "public_addrs": { "addrvec": [ { "type": “v2”, "addr": "10.10.10.1:3300”, "nonce": 0 }, { "type": “v1”, "addr": "10.10.10.1:6789”, "nonce": 0 } ] }, "addr": "10.10.10.1:6789/0”, "public_addr": "10.10.10.1:6789/0”, "priority": 0, "weight": 0, "crush_location": “{}” }, { "rank": 1, "name": “1”, "public_addrs": { "addrvec": [ { "type": “v2”, "addr": "10.10.10.2:3300”, "nonce": 0 }, { "type": “v1”, "addr": "10.10.10.2:6789”, "nonce": 0 } ] }, "addr": "10.10.10.2:6789/0”, "public_addr": "10.10.10.2:6789/0”, "priority": 0, "weight": 0, "crush_location": “{}” }, { "rank": 2, "name": “proxmox03”, "public_addrs": { "addrvec": [ { "type": “v2”, "addr": "10.10.10.3:3300”, "nonce": 0 }, { "type": “v1”, "addr": "10.10.10.3:6789”, "nonce": 0 } ] }, "addr": "10.10.10.3:6789/0”, "public_addr": "10.10.10.3:6789/0”, "priority": 0, "weight": 0, "crush_location": “{}” } ] }, "feature_map": { "mon": [ { "features": “0x3f01cfbdfffdffff”, "release": “luminous”, "num": 1 } ] }, "stretch_mode": false } — Mark Schouten CTO, Tuxis B.V. +31 318 200208 / mark(a)tuxis.nl

2 6

6 pgs not deep-scrubbed in time
by Michel Niyoyita 01 Feb '24

01 Feb '24

Hello team, I have a cluster in production composed by 3 osds servers with 20 disks each deployed using ceph-ansibleand ubuntu OS , and the version is pacific . These days is in WARN state caused by pgs which are not deep-scrubbed in time . I tried to deep-scrubbed some pg manually but seems that the cluster can be slow, would like your assistance in order that my cluster can be in HEALTH_OK state as before without any interuption of service . The cluster is used as openstack backend storage. Best Regards Michel ceph -s cluster: id: cb0caedc-eb5b-42d1-a34f-96facfda8c27 health: HEALTH_WARN 6 pgs not deep-scrubbed in time services: mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 11M) mgr: ceph-mon2(active, since 11M), standbys: ceph-mon3, ceph-mon1 osd: 48 osds: 48 up (since 11M), 48 in (since 11M) rgw: 6 daemons active (6 hosts, 1 zones) data: pools: 10 pools, 385 pgs objects: 5.97M objects, 23 TiB usage: 151 TiB used, 282 TiB / 433 TiB avail pgs: 381 active+clean 4 active+clean+scrubbing+deep io: client: 59 MiB/s rd, 860 MiB/s wr, 155 op/s rd, 665 op/s wr root@ceph-osd3:~# ceph health detail HEALTH_WARN 6 pgs not deep-scrubbed in time [WRN] PG_NOT_DEEP_SCRUBBED: 6 pgs not deep-scrubbed in time pg 6.78 not deep-scrubbed since 2024-01-11T16:07:54.875746+0200 pg 6.60 not deep-scrubbed since 2024-01-13T19:44:26.922000+0200 pg 6.5c not deep-scrubbed since 2024-01-13T09:07:24.780936+0200 pg 4.12 not deep-scrubbed since 2024-01-13T09:09:22.176240+0200 pg 10.d not deep-scrubbed since 2024-01-12T08:04:02.078062+0200 pg 5.f not deep-scrubbed since 2024-01-12T06:06:00.970665+0200

6 27

how can install latest dev release?
by garcetto 01 Feb '24

01 Feb '24

good morning, how can i install latest dev release using cephadm? thank you.

3 4

Changing A Ceph Cluster's Front- And/Or Back-End Networks IP Address(es)
by duluxoz 01 Feb '24

01 Feb '24

Hi All, Quick Q: How easy/hard is it to change the IP networks of: 1) A Ceph Cluster's "Front-End" Network? 2) A Ceph Cluster's "Back-End" Network? Is it a "simply" matter of: a) Placing the Nodes in maintenance mode b) Changing a config file (I assume it's /etc/ceph/ceph.conf) on each Node c) Rebooting the Nodes d) Taking each Node out of Maintenance Mode Thanks in advance Cheers Dulux-Oz

2 1

S3 object appears in ListObject but 404 when issuing a GET
by Mathias Chapelain 31 Jan '24

31 Jan '24

Hello, Recently we got a problem from an internal customer on our S3. Our setup consist of roughly 10 servers with 140 OSDs. Our 3 RGWs are collocated with monitors on dedicated servers in a HA setup with HAProxy in front. We are running 16.2.14 on Podman with Cephadm. Our S3 is constantly having a traffic of 500 req/s average per RGW instance. The problem is described in this issue: https://tracker.ceph.com/issues/63935. Basically this customer is having a Grafana Mimir instance pushing to our S3 and during a compaction process it does a special pattern like this: ``` 29/Dec/2023:17:13:28.961 rgw-frontend~ rgw-backend/server-mon-01-rgw0 0/0/0/127/127 200 228 - - ---- 132/132/70/67/0 0/0 "PUT /1234/object HTTP/1.1" 29/Dec/2023:17:13:29.101 rgw-frontend~ rgw-backend/server-mon-01-rgw0 0/0/0/1/1 200 381 - - ---- 132/132/76/71/0 0/0 "GET /1234/object HTTP/1.1" 29/Dec/2023:17:13:29.121 rgw-frontend~ rgw-backend/server-mon-01-rgw0 0/0/0/1/1 200 381 - - ---- 132/132/71/59/0 0/0 "GET /1234/object HTTP/1.1" 29/Dec/2023:17:13:29.137 rgw-frontend~ rgw-backend/server-mon-03-rgw0 0/0/0/4/4 204 153 - - ---- 132/132/71/6/0 0/0 "DELETE /1234/object HTTP/1.1" 29/Dec/2023:19:03:21.671 rgw-frontend~ rgw-backend/server-mon-03-rgw0 0/0/0/1/1 404 472 - - ---- 55/55/26/0/0 0/0 "GET /1234/object HTTP/1.1" ``` It is doing PUT, GET and DELETE in the same second. Afterwards the customer can see the deleted object when doing a ListObjects in the bucket but if he tries to access it then RGW returns a 404. After looking in Ceph, it appears the object has a bucket index entry but the associated RADOS object does not exist anymore. The bucket does not have versioning or object locking. Did someone encounter something similar? Thank you! Regards, -- Mathias Chapelain Storage Engineer Proton AG

1 0

Pacific: Drain hosts does not remove mgr daemon
by Mevludin Blazevic 31 Jan '24

31 Jan '24

Hi all, after performing "ceph orch host drain" on one of our host with only the mgr container left, I encounter that another mgr daemon is indeed deployed on another host, but the "old" does not get removed from the drain command. The same happens if I edit the mgr service via UI to define different hosts for the daemon and again the old mgr daemons are not getting removed. Any recommendations? I am using a setup with podman and RHEL. Best, Mevludin

2 1

Help on rgw metrics (was rgw_user_counters_cache)
by garcetto 31 Jan '24

31 Jan '24

good morning, i was struggling trying to understand why i cannot find this setting on my reef version, is it because is only on latest dev ceph version and not before? https://docs.ceph.com/en/*latest* /radosgw/metrics/#user-bucket-counter-caches Reef gives 404.... https://docs.ceph.com/en/reef/radosgw/metrics/ thank you!

2 1

NFS HA - "virtual_ip": null after upgrade to reef
by Torkil Svensgaard 31 Jan '24

31 Jan '24

Hi Last week we created an NFS service like this: " ceph nfs cluster create jumbo "ceph-flash1,ceph-flash2,ceph-flash3" --ingress --virtual_ip 172.21.15.74/22 --ingress-mode keepalive-only " Worked like a charm. Yesterday we upgraded from 17.2.7 to 18.20.0 and the NFS virtual IP seems to have gone missing in the process: " # ceph nfs cluster info jumbo { "jumbo": { "backend": [ { "hostname": "ceph-flash1", "ip": "172.21.15.148", "port": 2049 } ], "virtual_ip": null } } " Service spec: " service_type: nfs service_id: jumbo service_name: nfs.jumbo placement: count: 1 hosts: - ceph-flash1 - ceph-flash2 - ceph-flash3 spec: port: 2049 virtual_ip: 172.21.15.74 " I've tried restarting the nfs.jumbo service which didn't help. Suggestions? Mvh. Torkil -- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil(a)drcmr.dk

2 7

how to avoid pglogs dups bug in Pacific
by ADRIAN NICOLAE 31 Jan '24

31 Jan '24

Hi, I'm running Pacific 16.2.4 and I want to start a manual pg split process on the data pool (from 2048 to 4096). I'm reluctant to upgrade to 16.2.14/15 at this point. Can I avoid the dups bug (https://tracker.ceph.com/issues/53729) if I will increase the pgs slowly with 32 or 64pgs at every increment instead of moving directly to 4096 ? I don't have the autoscaler enabled. Thanks.

1 0

Re: RGW crashes when rgw_enable_ops_log is enabled
by Marc Singer 30 Jan '24

30 Jan '24

Hi Matt Thanks for your answer. Should I open a bug report then? How would I be able to read more from it? Have multiple threads access it and read from it simultaneously? Marc On 1/25/24 20:25, Matt Benjamin wrote: > Hi Marc, > > No, the only thing you need to do with the Unix socket is to keep > reading from it. So it probably is getting backlogged. And while you > could arrange things to make that less likely, you likely can't make > it impossible, so there's a bug here. > > Matt > > On Thu, Jan 25, 2024 at 10:52 AM Marc Singer <marc(a)singer.services> wrote: > > Hi > > I am using a unix socket client to connect with it and read the data > from it. > Do I need to do anything like signal the socket that this data has > been > read? Or am I not reading fast enough and data is backing up? > > What I am also noticing that at some point (probably after something > with the ops socket happens), the log level seems to increase for > some > reason? I did not find anything in the logs yet why this would be > the case. > > *Normal:* > > 2024-01-25T15:47:58.444+0000 7fe98a5c0b00 1 ====== starting new > request > req=0x7fe98712c720 ===== > 2024-01-25T15:47:58.548+0000 7fe98b700b00 1 ====== req done > req=0x7fe98712c720 op status=0 http_status=200 > latency=0.104001537s ====== > 2024-01-25T15:47:58.548+0000 7fe98b700b00 1 beast: 0x7fe98712c720: > redacted - redacted [25/Jan/2024:15:47:58.444 +0000] "PUT > /redacted/redacted/chunks/27/27242/27242514_10_4194304 HTTP/1.1" 200 > 4194304 - "redacted" - latency=0.104001537s > > *Close before crashing: > * > > -509> 2024-01-25T14:54:31.588+0000 7f5186648b00 1 ====== starting > new request req=0x7f517ffca720 ===== > -508> 2024-01-25T14:54:31.588+0000 7f5186648b00 2 req > 2568229052387020224 0.000000000s initializing for trans_id = > tx0000023a42eb7515dcdc0-0065b27627-823feaa-central > -507> 2024-01-25T14:54:31.588+0000 7f5186648b00 2 req > 2568229052387020224 0.000000000s getting op 1 > -506> 2024-01-25T14:54:31.588+0000 7f5186648b00 2 req > 2568229052387020224 0.000000000s s3:put_obj verifying requester > -505> 2024-01-25T14:54:31.588+0000 7f5186648b00 2 req > 2568229052387020224 0.000000000s s3:put_obj normalizing buckets > and tenants > -504> 2024-01-25T14:54:31.588+0000 7f5186648b00 2 req > 2568229052387020224 0.000000000s s3:put_obj init permissions > -503> 2024-01-25T14:54:31.588+0000 7f5186648b00 2 req > 2568229052387020224 0.000000000s s3:put_obj recalculating target > -502> 2024-01-25T14:54:31.588+0000 7f5186648b00 2 req > 2568229052387020224 0.000000000s s3:put_obj reading permissions > -501> 2024-01-25T14:54:31.588+0000 7f5186648b00 2 req > 2568229052387020224 0.000000000s s3:put_obj init op > -500> 2024-01-25T14:54:31.588+0000 7f5186648b00 2 req > 2568229052387020224 0.000000000s s3:put_obj verifying op mask > -499> 2024-01-25T14:54:31.588+0000 7f5186648b00 2 req > 2568229052387020224 0.000000000s s3:put_obj verifying op permissions > -498> 2024-01-25T14:54:31.588+0000 7f5186648b00 5 req > 2568229052387020224 0.000000000s s3:put_obj Searching permissions for > identity=rgw::auth::SysReqApplier -> > rgw::auth::LocalApplier(acct_user=redacted, acct_name=redacted, > subuser=, perm_mask=15, is_admin=0) mask=50 > -497> 2024-01-25T14:54:31.588+0000 7f5186648b00 5 req > 2568229052387020224 0.000000000s s3:put_obj Searching permissions for > uid=redacted > -496> 2024-01-25T14:54:31.588+0000 7f5186648b00 5 req > 2568229052387020224 0.000000000s s3:put_obj Found permission: 15 > -495> 2024-01-25T14:54:31.588+0000 7f5186648b00 5 req > 2568229052387020224 0.000000000s s3:put_obj Searching permissions for > group=1 mask=50 > -494> 2024-01-25T14:54:31.588+0000 7f5186648b00 5 req > 2568229052387020224 0.000000000s s3:put_obj Permissions for group > not found > -493> 2024-01-25T14:54:31.588+0000 7f5186648b00 5 req > 2568229052387020224 0.000000000s s3:put_obj Searching permissions for > group=2 mask=50 > -492> 2024-01-25T14:54:31.588+0000 7f5186648b00 5 req > 2568229052387020224 0.000000000s s3:put_obj Permissions for group > not found > -491> 2024-01-25T14:54:31.588+0000 7f5186648b00 5 req > 2568229052387020224 0.000000000s s3:put_obj -- Getting permissions > done > for identity=rgw::auth::SysReqApplier -> > rgw::auth::LocalApplier(acct_user=redacted, acct_name=redacted, > subuser=, perm_mask=15, is_admin=0), owner=redacted, perm=2 > -490> 2024-01-25T14:54:31.588+0000 7f5186648b00 2 req > 2568229052387020224 0.000000000s s3:put_obj verifying op params > -489> 2024-01-25T14:54:31.588+0000 7f5186648b00 2 req > 2568229052387020224 0.000000000s s3:put_obj pre-executing > -488> 2024-01-25T14:54:31.588+0000 7f5186648b00 2 req > 2568229052387020224 0.000000000s s3:put_obj check rate limiting > -487> 2024-01-25T14:54:31.588+0000 7f5186648b00 2 req > 2568229052387020224 0.000000000s s3:put_obj executing > -486> 2024-01-25T14:54:31.624+0000 7f5183898b00 5 req > 2568229052387020224 0.036000550s s3:put_obj NOTICE: call to > do_aws4_auth_completion > -485> 2024-01-25T14:54:31.624+0000 7f5183898b00 5 req > 2568229052387020224 0.036000550s s3:put_obj NOTICE: call to > do_aws4_auth_completion > -484> 2024-01-25T14:54:31.680+0000 7f5185bc8b00 2 req > 2568229052387020224 0.092001401s s3:put_obj completing > -483> 2024-01-25T14:54:31.680+0000 7f5185bc8b00 2 req > 2568229052387020224 0.092001401s s3:put_obj op status=0 > -482> 2024-01-25T14:54:31.680+0000 7f5185bc8b00 2 req > 2568229052387020224 0.092001401s s3:put_obj http status=200 > > -481> 2024-01-25T14:54:31.680+0000 7f5185bc8b00 1 ====== req done > req=0x7f517ffca720 op status=0 http_status=200 > latency=0.092001401s ====== > > Thanks for your help. > > Marc Singer > > On 1/25/24 16:22, Matt Benjamin wrote: > > Hi Marc, > > > > The ops log code is designed to discard data if the socket is > > flow-controlled, iirc. Maybe we just need to handle the signal. > > > > Of course, you should have something consuming data on the > socket, but it's > > still a problem if radosgw exits unexpectedly. > > > > Matt > > > > On Thu, Jan 25, 2024 at 10:08 AM Marc > Singer<marc(a)singer.services> wrote: > > > >> Hi Ceph Users > >> > >> I am encountering a problem with the RGW Admin Ops Socket. > >> > >> I am setting up the socket as follows: > >> > >> rgw_enable_ops_log = true > >> rgw_ops_log_socket_path = /tmp/ops/rgw-ops.socket > >> rgw_ops_log_data_backlog = 16Mi > >> > >> Seems like the socket fills up over time and it doesn't seem to get > >> flushed, at some point the process runs out of file space. > >> > >> Do I need to configure something or send something for the > socket to flush? > >> > >> See the log here: > >> > >> 0> 2024-01-25T13:10:13.908+0000 7f247b00eb00 -1 *** Caught > signal (File > >> size limit exceeded) ** > >> in thread 7f247b00eb00 thread_name:ops_log_file > >> > >> ceph version 18.2.0 > (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef > >> (stable) > >> NOTE: a copy of the executable, or `objdump -rdS > <executable>` is > >> needed to interpret this. > >> > >> --- logging levels --- > >> 0/ 5 none > >> 0/ 1 lockdep > >> 0/ 1 context > >> 1/ 1 crush > >> 1/ 5 mds > >> 1/ 5 mds_balancer > >> 1/ 5 mds_locker > >> 1/ 5 mds_log > >> 1/ 5 mds_log_expire > >> 1/ 5 mds_migrator > >> 0/ 1 buffer > >> 0/ 1 timer > >> 0/ 1 filer > >> 0/ 1 striper > >> 0/ 1 objecter > >> 0/ 5 rados > >> 0/ 5 rbd > >> 0/ 5 rbd_mirror > >> 0/ 5 rbd_replay > >> 0/ 5 rbd_pwl > >> 0/ 5 journaler > >> 0/ 5 objectcacher > >> 0/ 5 immutable_obj_cache > >> 0/ 5 client > >> 1/ 5 osd > >> 0/ 5 optracker > >> 0/ 5 objclass > >> 1/ 3 filestore > >> 1/ 3 journal > >> 0/ 0 ms > >> 1/ 5 mon > >> 0/10 monc > >> 1/ 5 paxos > >> 0/ 5 tp > >> 1/ 5 auth > >> 1/ 5 crypto > >> 1/ 1 finisher > >> 1/ 1 reserver > >> 1/ 5 heartbeatmap > >> 1/ 5 perfcounter > >> 1/ 5 rgw > >> 1/ 5 rgw_sync > >> 1/ 5 rgw_datacache > >> 1/ 5 rgw_access > >> 1/ 5 rgw_dbstore > >> 1/ 5 rgw_flight > >> 1/ 5 javaclient > >> 1/ 5 asok > >> 1/ 1 throttle > >> 0/ 0 refs > >> 1/ 5 compressor > >> 1/ 5 bluestore > >> 1/ 5 bluefs > >> 1/ 3 bdev > >> 1/ 5 kstore > >> 4/ 5 rocksdb > >> 4/ 5 leveldb > >> 1/ 5 fuse > >> 2/ 5 mgr > >> 1/ 5 mgrc > >> 1/ 5 dpdk > >> 1/ 5 eventtrace > >> 1/ 5 prioritycache > >> 0/ 5 test > >> 0/ 5 cephfs_mirror > >> 0/ 5 cephsqlite > >> 0/ 5 seastore > >> 0/ 5 seastore_onode > >> 0/ 5 seastore_odata > >> 0/ 5 seastore_omap > >> 0/ 5 seastore_tm > >> 0/ 5 seastore_t > >> 0/ 5 seastore_cleaner > >> 0/ 5 seastore_epm > >> 0/ 5 seastore_lba > >> 0/ 5 seastore_fixedkv_tree > >> 0/ 5 seastore_cache > >> 0/ 5 seastore_journal > >> 0/ 5 seastore_device > >> 0/ 5 seastore_backref > >> 0/ 5 alienstore > >> 1/ 5 mclock > >> 0/ 5 cyanstore > >> 1/ 5 ceph_exporter > >> 1/ 5 memstore > >> -2/-2 (syslog threshold) > >> 99/99 (stderr threshold) > >> --- pthread ID / name mapping for recent threads --- > >> 7f2472a89b00 / safe_timer > >> 7f2472cadb00 / radosgw > >> ... > >> log_file > >> > >> > /var/lib/ceph/crash/2024-01-25T13:10:13.909546Z_01ee6e6a-e946-4006-9d32-e17ef2f9df74/log > >> --- end dump of recent events --- > >> reraise_fatal: default handler for signal 25 didn't terminate > the process? > >> > >> Thank you for your help. > >> > >> Marc > >> _______________________________________________ > >> ceph-users mailing list --ceph-users(a)ceph.io > >> To unsubscribe send an email toceph-users-leave(a)ceph.io > >> > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > -- > > Matt Benjamin > Red Hat, Inc. > 315 West Huron Street, Suite 140A > Ann Arbor, Michigan 48103 > > http://www.redhat.com/en/technologies/storage > > tel. 734-821-5101 > fax. 734-769-8938 > cel. 734-216-5309

2 2

Ceph stretch mode connect to local datacenter
by Oleksandr 34 30 Jan '24

30 Jan '24

Hello. I have a ceph cluster which works in stretch mode: *DC1:* node1 (osd, mon, mgr) node2 (osd, mon) node3 (osd, mds) *DC2:* node1 (osd, mon, mgr) node2 (osd, mon) node3 (osd, mds) *DC3:* node1 (mon) Datacenters are distributions between different locations. I use RBD on my clients. How can I set up my clients to connect to the local datacenter? I don't need big traffic between datacenters only for replications, so my client in DC1 should connect to ceph in DC1. But when something happened with DC1 my clients in DC1 should still be working. In configs on my clients I set up all cluster monitors. Is this possible at all?

1 0

Scrubbing?
by Jan Marek 30 Jan '24

30 Jan '24

Hello, last week I've got a HEALTH_OK on our CEPH cluster and I started upgrade firmware in network cards. When I had upgraded the sixth card from nine (one-by-one), this server didn't started correctly and our ProxMox had problem with accessing disk images on CEPH. rbd ls pool was OK, but: rbd ls pool -l didn't work. Our virtual servers had a trouble to work with disks. After I resolve network problem with OSD server, everythink returning to normal state. But I've found, that every OSD nod have very high activity: when I've started 'iotop', there was very high load: around 180MB/s read and 20MB/s write. In this time, cluster was in the HEALTH_OK state. I've found, that there is a massive scrubbing activity... After a few days, I have on our OSD nodes around 90MB/s read and 70MB/s write while 'ceph -s' have client io as 2,5MB/s read and 50MB/s write. I've found in log file of our mon server many lines about starting of scrubbing, but there are many messages about starting of scrubb the same PG? I've grep'ed syslog for some of them and attach it to this e-mail. Is this activity OK? Why CEPH start scrubing this PG once and once again? And another question: Is scrubbing part of mClock scheduler? Many thanks for explanation. Sincerely Jan Marek -- Ing. Jan Marek University of South Bohemia Academic Computer Centre Phone: +420389032080 http://www.gnu.org/philosophy/no-word-attachments.cs.html

4 7

How check local network
by Albert Shih 30 Jan '24

30 Jan '24

Hi When I deploy my cluster I didn't notice on two of my servers the private network was not working (wrong vlan), now it's working, but how can I check the it's indeed working (currently I don't have data). Regards -- Albert SHIH 🦫 🐸 France Heure locale/Local time: lun. 29 janv. 2024 22:36:01 CET

2 2

Unsetting maintenance mode for failed host
by Bryce Nicholls 29 Jan '24

29 Jan '24

Hi We put a host in maintenance and had issues bringing it back. Is there a safe way of exiting maintenance while the host is unreachable / offline? We would like the cluster to rebalance while we are working to get this host back online. Maintenance was set using: ceph orch host maintenance enter osd1 I tried exiting using: ceph orch host maintenance exit osd1 but got the below stacktrace. root@mon1 ~ # ceph orch host maintenance exit osd1 Error EINVAL: Traceback (most recent call last): File "/usr/share/ceph/mgr/mgr_module.py", line 1756, in _handle_command return self.handle_command(inbuf, cmd) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 171, in handle_command return dispatch[cmd['prefix']].call(self, cmd, inbuf) File "/usr/share/ceph/mgr/mgr_module.py", line 462, in call return self.func(mgr, **kwargs) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 107, in <lambda> wrapper_copy = lambda *l_args, **l_kwargs: wrapper(*l_args, **l_kwargs) # noqa: E731 File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 96, in wrapper return func(*args, **kwargs) File "/usr/share/ceph/mgr/orchestrator/module.py", line 455, in _host_maintenance_exit raise_if_exception(completion) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 225, in raise_if_exception e = pickle.loads(c.serialized_exception) TypeError: __init__() missing 2 required positional arguments: 'hostname' and 'addr' Thanks Bryce Bryce Nicholls OpenStack Engineer Bryce.Nicholls92(a)thehutgroup.com [THG Ingenuity Logo]<https://www.thg.com> [https://i.imgur.com/wbpVRW6.png]<https://www.linkedin.com/company/thgplc/?originalSubdomain=uk> [https://i.imgur.com/c3040tr.png] <https://twitter.com/thgplc?lang=en>

2 1

January Ceph Science Virtual User Group
by Kevin Hrpcek 29 Jan '24

29 Jan '24

Hey All, We will be having a Ceph science/research/big cluster call on Wednesday January 31st. If anyone wants to discuss something specific they can add it to the pad linked below. If you have questions or comments you can contact me. This is an informal open call of community members mostly from hpc/htc/research/big cluster environments (though anyone is welcome) where we discuss whatever is on our minds regarding ceph. Updates, outages, features, maintenance, etc...there is no set presenter but I do attempt to keep the conversation lively. Pad URL: https://pad.ceph.com/p/Ceph_Science_User_Group_20240131 Virtual event details: January 31, 2024 15:00 UTC 4pm Central European 9am Central US Description: Main pad for discussions: https://pad.ceph.com/p/Ceph_Science_User_Group_Index Meetings will be recorded and posted to the Ceph Youtube channel. To join the meeting on a computer or mobile phone: https://meet.jit.si/ceph-science-wg Kevin -- Kevin Hrpcek NASA VIIRS Atmosphere SIPS/TROPICS Space Science & Engineering Center University of Wisconsin-Madison

1 0

RadosGW manual deployment
by Jan Kasprzak 29 Jan '24

29 Jan '24

Hi all, how can radosgw be deployed manually? For Ceph cluster deployment, there is still (fortunately!) a documented method which works flawlessly even in Reef: https://docs.ceph.com/en/latest/install/manual-deployment/#monitor-bootstra… But as for radosgw, there is no such description, unless I am missing something. Even going back to the oldest docs still available at docs.ceph.com (mimic), the radosgw installation is described only using ceph-deploy: https://docs.ceph.com/en/mimic/install/install-ceph-gateway/ Is it possible to install a new radosgw instance manually? If so, how can I do it? Thanks! -Yenya -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall

3 9

easy way to find out the number of allocated objects for a RBD image
by Tony Liu 29 Jan '24

29 Jan '24

Hi, Other than get all objects of the pool and filter by image ID, is there any easier way to get the number of allocated objects for a RBD image? What I really want to know is the actual usage of an image. An allocated object could be used partially, but that's fine, no need to be 100% accurate. To get the object count and times object size, that should be sufficient. "rbd export" exports actual used data, but to get the actual usage by exporting the image seems too much. This brings up another question, is there any way to know the export size before running it? Thanks! Tony

4 4

1 clients failing to respond to cache pressure (quincy:17.2.6)
by Özkan Göksu 29 Jan '24

29 Jan '24

Hello. I have 5 node ceph cluster and I'm constantly having "clients failing to respond to cache pressure" warning. I have 84 cephfs kernel clients (servers) and my users are accessing their personal subvolumes located on one pool. My users are software developers and the data is home and user data. (Git, python projects, sample data and generated new data) --------------------------------------------------------------------------------- --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 146 TiB 101 TiB 45 TiB 45 TiB 30.71 TOTAL 146 TiB 101 TiB 45 TiB 45 TiB 30.71 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 356 MiB 90 1.0 GiB 0 30 TiB cephfs.ud-data.meta 9 256 69 GiB 3.09M 137 GiB 0.15 45 TiB cephfs.ud-data.data 10 2048 26 TiB 100.83M 44 TiB 32.97 45 TiB --------------------------------------------------------------------------------- root@ud-01:~# ceph fs status ud-data - 84 clients ======= RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active ud-data.ud-04.seggyv Reqs: 142 /s 2844k 2798k 303k 720k POOL TYPE USED AVAIL cephfs.ud-data.meta metadata 137G 44.9T cephfs.ud-data.data data 44.2T 44.9T STANDBY MDS ud-data.ud-02.xcoojt ud-data.ud-05.rnhcfe ud-data.ud-03.lhwkml ud-data.ud-01.uatjle MDS version: ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) ----------------------------------------------------------------------------------- My MDS settings are below: mds_cache_memory_limit | 8589934592 mds_cache_trim_threshold | 524288 mds_recall_global_max_decay_threshold | 131072 mds_recall_max_caps | 30000 mds_recall_max_decay_rate | 1.500000 mds_recall_max_decay_threshold | 131072 mds_recall_warning_threshold | 262144 I have 2 questions: 1- What should I do to prevent cache pressue warning ? 2- What can I do to increase speed ? - Thanks

4 23

crushmap rules :: host selection
by Adrian Sevcenco 29 Jan '24

29 Jan '24

Hi! I'm new with ceph and i struggle to make a mapping between my current storage knowledge and ceph... So, i will state my understanding of the context and the question so please correct me with anything that i got wrong :) So, files (or pieces of files) are put in PGs that are given sections of OSDs. The crushmap gives a physical OSDs map to be chosen for placement or access Pools are a logical name for a storage space but how can i specify what osds or host are part of a pool? For replication, how can i specify: if a replica is missing (for a given time) start rebuilding on some available OSD? Is there a notion of "spare" so if an osd is missing on action, the rebuild to start on another host and when the old OSD is back (the hdd is replaced, or the machine was repaired) to be automatically cleaned up and used? I'm thinking about a 3 node cluster with the replica=2 failure domain = host, in such a way if one node is down, the data from there to be replicated on the remaining nodes (with some drives kept as spares..) I am almost certain that from the point of view of ceph, what i'm thinking is wrong so i would love to receive some advice :) Thanks a lot! Adrian

3 9

Re: Ceph OSD reported Slow operations
by V A Prabha 28 Jan '24

28 Jan '24

Hi Eugen Please find the details below root@meghdootctr1:/var/log/ceph# ceph -s cluster: id: c59da971-57d1-43bd-b2b7-865d392412a5 health: HEALTH_WARN nodeep-scrub flag(s) set 544 pgs not deep-scrubbed in time services: mon: 3 daemons, quorum meghdootctr1,meghdootctr2,meghdootctr3 (age 5d) mgr: meghdootctr1(active, since 5d), standbys: meghdootctr2, meghdootctr3 mds: 3 up:standby osd: 36 osds: 36 up (since 34h), 36 in (since 34h) flags nodeep-scrub data: pools: 2 pools, 544 pgs objects: 10.14M objects, 39 TiB usage: 116 TiB used, 63 TiB / 179 TiB avail pgs: 544 active+clean io: client: 24 MiB/s rd, 16 MiB/s wr, 2.02k op/s rd, 907 op/s wr Ceph Versions: root@meghdootctr1:/var/log/ceph# ceph --version ceph version 14.2.16 (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus (stable) Ceph df -h https://pastebin.com/1ffucyJg Ceph OSD performance dump https://pastebin.com/1R6YQksE Ceph tell osd.XX bench (Out of 36 osds only 8 OSDs give High IOPS value of 250 +. Out of that 4 OSDs are from HP 3PAR and 4 OSDS from DELL EMC. We are using only 4 OSDs from HP3 par and it is working fine without any latency and iops issues from the beginning but the remaining 32 OSDs are from DELL EMC in which 4 OSDs are much better than the remaining 28 OSDs) https://pastebin.com/CixaQmBi Please help me to identify if the issue is with the DELL EMC Storage, Ceph configuration parameter tuning or the Overload in the cloud setup On November 1, 2023 at 9:48 PM Eugen Block <eblock(a)nde.ag> wrote: > Hi, > > for starters please add more cluster details like 'ceph status', 'ceph > versions', 'ceph osd df tree'. Increasing the to 10G was the right > thing to do, you don't get far with 1G with real cluster load. How are > the OSDs configured (HDD only, SSD only or HDD with rocksdb on SSD)? > How is the disk utilization? > > Regards, > Eugen > > Zitat von prabhav(a)cdac.in: > > > In a production setup of 36 OSDs( SAS disks) totalling 180 TB > > allocated to a single Ceph Cluster with 3 monitors and 3 managers. > > There were 830 volumes and VMs created in Openstack with Ceph as a > > backend. On Sep 21, users reported slowness in accessing the VMs. > > Analysing the logs lead us to problem with SAS , Network congestion > > and Ceph configuration( as all default values were used). We updated > > the Network from 1Gbps to 10Gbps for public and cluster networking. > > There was no change. > > The ceph benchmark performance showed that 28 OSDs out of 36 OSDs > > reported very low IOPS of 30 to 50 while the remaining showed 300+ > > IOPS. > > We gradually started reducing the load on the ceph cluster and now > > the volumes count is 650. Now the slow operations has gradually > > reduced but I am aware that this is not the solution. > > Ceph configuration is updated with increasing the > > osd_journal_size to 10 GB, > > osd_max_backfills = 1 > > osd_recovery_max_active = 1 > > osd_recovery_op_priority = 1 > > bluestore_cache_trim_max_skip_pinned=10000 > > > > After one month, now we faced another issue with Mgr daemon stopped > > in all 3 quorums and 16 OSDs went down. From the > > ceph-mon,ceph-mgr.log could not get the reason. Please guide me as > > its a production setup > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io Thanks & Regards, Ms V A Prabha / श्रीमती प्रभा वी ए Joint Director / संयुक्त निदेशक Centre for Development of Advanced Computing(C-DAC) / प्रगत संगणन विकास केन्द्र(सी-डैक) Tidel Park”, 8th Floor, “D” Block, (North &South) / “टाइडल पार्क”,8वीं मंजिल, “डी” ब्लॉक, (उत्तर और दक्षिण) No.4, Rajiv Gandhi Salai / नं.4, राजीव गांधी सलाई Taramani / तारामणि Chennai / चेन्नई – 600113 Ph.No.:044-22542226/27 Fax No.: 044-22542294 ------------------------------------------------------------------------------------------------------------ [ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ] This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken. ------------------------------------------------------------------------------------------------------------

4 15

17.2.7: Backfilling deadlock / stall / stuck / standstill
by Kai Stian Olstad 28 Jan '24

28 Jan '24

Hi, This is a cluster running 17.2.7 upgraded from 16.2.6 on the 15 January 2024. On Monday 22 January we had 4 HDD all on different server with I/O-error because of some damage sectors, the OSD is hybrid so the DB is on SSD, 5 HDD share 1 SSD. I set the OSD out, ceph osd out 223 269 290 318 and all hell broke loose. I took only minutes before the users complained about Ceph not working. Ceph status reportet slow OPS on the OSDs that was set to out, and “ceph tell osd.<id> dump_ops_in_flight” against the out OSDs it just hang, after 30 minutes I stopped the dump command. Long story short I ended up running “ceph osd set nobackfill” to slow ops was gone and then unset it when the slow ops message disappeared. I needed to run that all the time so the cluster didn’t come to a holt so this oneliner loop was used “while true; do ceph -s | grep -qE "oldest one blocked for [0-9]{2,}" && (date; ceph osd set nobackfill; sleep 15; ceph osd unset nobackfill); sleep 10; done” But now 4 days later the backfilling has stopped progressing completely and the number of misplaced object is increasing. Some PG has 0 misplaced object but sill have backfilling state, and been in this state for over 24 hours now. I have a hunch that it’s because of PG 404.6e7 is in state “active+recovering+degraded+remapped” it’s been in this state for over 48 hours. It’s has possible 2 missing object, but since they are not unfound I can’t delete them with “ceph pg 404.6e7 mark_unfound_lost delete” Could someone please help to solve this? Down below is some output of ceph commands, I’ll also attache them. ceph status (only removed information about no running scrub and deep_scrub) --- cluster: id: b321e76e-da3a-11eb-b75c-4f948441dcd0 health: HEALTH_WARN Degraded data redundancy: 2/6294904971 objects degraded (0.000%), 1 pg degraded services: mon: 3 daemons, quorum ceph-mon-1,ceph-mon-2,ceph-mon-3 (age 11d) mgr: ceph-mon-1.ptrsea(active, since 11d), standbys: ceph-mon-2.mfdanx mds: 1/1 daemons up, 1 standby osd: 355 osds: 355 up (since 22h), 351 in (since 4d); 18 remapped pgs rgw: 7 daemons active (7 hosts, 1 zones) data: volumes: 1/1 healthy pools: 14 pools, 3945 pgs objects: 1.14G objects, 1.1 PiB usage: 1.8 PiB used, 1.2 PiB / 3.0 PiB avail pgs: 2/6294904971 objects degraded (0.000%) 2980455/6294904971 objects misplaced (0.047%) 3901 active+clean 22 active+clean+scrubbing+deep 17 active+remapped+backfilling 4 active+clean+scrubbing 1 active+recovering+degraded+remapped io: client: 167 MiB/s rd, 13 MiB/s wr, 6.02k op/s rd, 2.35k op/s wr ceph health detail (only removed information about no running scrub and deep_scrub) --- HEALTH_WARN Degraded data redundancy: 2/6294902067 objects degraded (0.000%), 1 pg degraded [WRN] PG_DEGRADED: Degraded data redundancy: 2/6294902067 objects degraded (0.000%), 1 pg degraded pg 404.6e7 is active+recovering+degraded+remapped, acting [223,274,243,290,286,283] ceph pg 202.6e7 list_unfound --- { "num_missing": 2, "num_unfound": 0, "objects": [], "state": "Active", "available_might_have_unfound": true, "might_have_unfound": [], "more": false } ceph pg 404.6e7 query | jq .recovery_state --- [ { "name": "Started/Primary/Active", "enter_time": "2024-01-26T09:08:41.918637+0000", "might_have_unfound": [ { "osd": "243(2)", "status": "already probed" }, { "osd": "274(1)", "status": "already probed" }, { "osd": "275(0)", "status": "already probed" }, { "osd": "283(5)", "status": "already probed" }, { "osd": "286(4)", "status": "already probed" }, { "osd": "290(3)", "status": "already probed" }, { "osd": "335(3)", "status": "already probed" } ], "recovery_progress": { "backfill_targets": [ "275(0)", "335(3)" ], "waiting_on_backfill": [], "last_backfill_started": "404:e76011a9:::1f244892-a2e7-406b-aa62-1b13511333a2.625411.18_56463c71-286c-4399-8d5d-0c278b7c97fd:head", "backfill_info": { "begin": "MIN", "end": "MIN", "objects": [] }, "peer_backfill_info": [], "backfills_in_flight": [], "recovering": [], "pg_backend": { "recovery_ops": [], "read_ops": [] } } }, { "name": "Started", "enter_time": "2024-01-26T09:08:40.909151+0000" } ] ceph pg ls recovering backfilling --- PG OBJECTS DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS STATE SINCE VERSION REPORTED UP ACTING 404.bc 287986 0 0 0 512046716673 0 0 10091 0 active+recovering+remapped 2h 217988'1385478 217988:10897565 [193,297,279,276,136,197]p193 [223,297,269,276,136,197]p223 404.c4 288236 0 288236 0 511669837559 0 0 10063 0 active+remapped+backfilling 24h 217988'1378228 217988:11719855 [156,186,178,345,339,177]p156 [223,186,178,345,339,177]p223 404.12a 287544 0 0 0 512246100354 0 0 10009 0 active+remapped+backfilling 24h 217988'1392371 217988:13739524 [248,178,250,145,304,272]p248 [223,178,250,145,304,272]p223 404.1c1 287739 0 286969 0 511800674008 0 0 10047 0 active+remapped+backfilling 2d 217988'1402889 217988:10975174 [332,246,183,169,280,255]p332 [318,246,183,169,280,255]p318 404.258 287737 0 277111 0 510099501390 0 0 10077 0 active+remapped+backfilling 24h 217988'1451778 217988:12780104 [308,199,134,342,188,221]p308 [318,199,134,342,188,221]p318 404.269 287990 0 0 0 512343190608 0 0 10043 0 active+remapped+backfilling 24h 217988'1358446 217988:14020217 [275,205,283,247,211,292]p275 [223,205,283,247,211,292]p223 404.34e 287624 0 277899 0 510447074297 0 0 10002 0 active+remapped+backfilling 24h 217988'1392933 217988:12636557 [322,141,338,168,251,218]p322 [318,141,338,168,251,218]p318 404.39c 287844 0 286692 0 512947685682 0 0 10017 0 active+remapped+backfilling 2d 217988'1414697 217988:11004944 [288,188,131,299,295,181]p288 [318,188,131,299,295,181]p318 404.511 287589 0 0 0 512014863711 0 0 10057 0 active+remapped+backfilling 24h 217988'1368741 217988:11544729 [166,151,327,333,186,150]p166 [223,151,327,333,186,150]p223 404.5f1 288126 0 286621 0 510850256945 0 0 10071 0 active+remapped+backfilling 24h 217988'1365831 217988:10348125 [214,332,289,184,255,160]p214 [223,332,289,184,255,160]p223 404.62a 288035 0 0 0 511318662269 0 0 10014 0 active+remapped+backfilling 3h 217988'1358010 217988:12528704 [322,260,259,319,149,152]p322 [318,260,259,319,149,152]p318 404.63d 287372 0 286559 0 508783837699 0 0 10074 0 active+remapped+backfilling 24h 217988'1402174 217988:11685744 [303,307,186,350,161,267]p303 [318,307,186,350,161,267]p318 404.6e3 288110 0 0 0 509047569016 0 0 10049 0 active+remapped+backfilling 24h 217988'1368547 217988:12202278 [166,317,233,144,337,240]p166 [223,317,233,144,337,240]p223 404.6e7 287856 2 2 0 510383394904 0 0 10047 0 active+recovering+degraded+remapped 3h 217988'1356501 217988:13157749 [275,274,243,335,286,283]p275 [223,274,243,290,286,283]p223 404.7d2 287619 0 286026 0 510708533087 0 0 10093 0 active+remapped+backfilling 3d 217988'1397393 217988:12146656 [185,139,299,222,155,149]p185 [223,139,299,222,155,149]p223 412.119 711468 0 0 0 207473602580 0 0 10099 0 active+remapped+backfilling 24h 217988'21613330 217988:87589096 [352,207,292,314,230,262]p352 [318,207,292,314,230,262]p318 412.12f 711529 0 701279 0 208498170310 0 0 10033 0 active+remapped+backfilling 24h 217988'14873593 217988:86198113 [303,305,183,215,130,244]p303 [318,305,183,215,130,244]p318 412.1fb 713044 0 3166 0 207787641403 0 0 10097 0 active+remapped+backfilling 2d 217988'14893270 217988:105346132 [156,137,228,241,262,353]p156 [223,137,228,241,262,353]p223 ceph osd tree out --- ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 3112.43481 root default -67 192.35847 host ceph-hd-001 269 hdd 12.82390 osd.269 up 0 1.00000 -49 192.35847 host ceph-hd-003 223 hdd 12.82390 osd.223 up 0 1.00000 -73 192.35847 host ceph-hd-011 290 hdd 12.82390 osd.290 up 0 1.00000 -79 192.35847 host ceph-hd-014 318 hdd 12.82390 osd.318 up 0 1.00000

3 4

Quite important: How do I restart a small cluster using cephadm at 18.2.1
by Carl J Taylor 27 Jan '24

27 Jan '24

Hi, Due to idiotic behaviour on my part I made a mistake while replacing some disks in our data centre and our cluster ended up all powered off! I have been using ceph for many years (since firefly) but only recently upgraded to reef and moved to the cephadm / podman setup. I am trying to figure out how to get it all started up again. I am not very familiar with docker at all. I can see the bootstrap option but no "recover" option. It is a small cluster with 3 nodes and two 3TB/4TB disks in each node. I have had a look at https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-mon/#… but wonder if cephadm does this itself automagically? Help please, I don't want to lose my data! Many thanks Carl.

2 1

c-states and OSD performance
by Christopher Durham 27 Jan '24

27 Jan '24

Hi, The following article: https://ceph.io/en/news/blog/2024/ceph-a-journey-to-1tibps/ suggests that dsabling C-states on your CPUs (on the OSD nodes) as one method to improve performance. The article seems to indicate that the scenariobeing addressed in the article was with NVMEs as OSDs. Questions: Will disabling C-states and keeping the processors at max power state help performance for the following: 1. NVME OSDs (yes)2. SSD OSDs3. Spinning disk OSDs -Chris

2 1

Odd auto-scaler warnings about too few/many PGs
by Torkil Svensgaard 26 Jan '24

26 Jan '24

Hi A few years ago we were really strapped for space so we tweaked pg_num for some pools to ensure all pgs were as to close to the same size as possible while stile observing the power of 2 rule, in order to get the most mileage space wise. We set the auto-scaler to off for the tweaked pools to get rid of the warnings. We now have a lot more free space so I flipped the auto-scaler to warn for all pools and set the bulk flag for the pools expected to be data pools, leading to this: " [WRN] POOL_TOO_FEW_PGS: 4 pools have too few placement groups Pool rbd has 512 placement groups, should have 2048 Pool rbd_internal has 1024 placement groups, should have 2048 Pool cephfs.nvme.data has 32 placement groups, should have 4096 Pool cephfs.ssd.data has 32 placement groups, should have 1024 [WRN] POOL_TOO_MANY_PGS: 4 pools have too many placement groups Pool libvirt has 256 placement groups, should have 32 Pool cephfs.cephfs.data has 512 placement groups, should have 32 Pool rbd_ec_data has 4096 placement groups, should have 1024 Pool cephfs.hdd.data has 2048 placement groups, should have 1024 " That's a lot of warnings *ponder* " # ceph osd pool autoscale-status POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO EFFECTIVE RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE BULK libvirt 2567G 3.0 3031T 0.0025 1.0 256 warn False .mgr 807.5M 2.0 6520G 0.0002 1.0 1 warn False rbd_ec 9168k 3.0 6520G 0.0000 1.0 32 warn False nvme 31708G 2.0 209.5T 0.2955 1.0 2048 warn False .nfs 36864 3.0 6520G 0.0000 1.0 32 warn False cephfs.cephfs.meta 24914M 3.0 6520G 0.0112 4.0 32 warn False cephfs.cephfs.data 16384 3.0 6520G 0.0000 1.0 512 warn False rbd.ssd.data 798.1G 2.25 6520G 0.2754 1.0 64 warn False rbd_ec_data 609.2T 1.5 3031T 0.3014 1.0 4096 warn True rbd 68170G 3.0 3031T 0.0659 1.0 512 warn True rbd_internal 69553G 3.0 3031T 0.0672 1.0 1024 warn True cephfs.nvme.data 0 2.0 209.5T 0.0000 1.0 32 warn True cephfs.ssd.data 68609M 2.0 6520G 0.0206 1.0 32 warn True cephfs.hdd.data 111.0T 2.25 3031T 0.0824 1.0 2048 warn True " " # ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 3.0 PiB 1.3 PiB 1.6 PiB 1.6 PiB 54.69 nvme 210 TiB 146 TiB 63 TiB 63 TiB 30.21 ssd 6.4 TiB 4.0 TiB 2.4 TiB 2.4 TiB 37.69 TOTAL 3.2 PiB 1.5 PiB 1.7 PiB 1.7 PiB 53.07 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL rbd 4 512 80 TiB 21.35M 200 TiB 19.31 278 TiB libvirt 5 256 3.0 TiB 810.89k 7.5 TiB 0.89 278 TiB rbd_internal 6 1024 86 TiB 28.22M 204 TiB 19.62 278 TiB .mgr 8 1 4.3 GiB 1.06k 1.6 GiB 0.07 1.0 TiB rbd_ec 10 32 55 MiB 25 27 MiB 0 708 GiB rbd_ec_data 11 4096 683 TiB 180.52M 914 TiB 52.26 556 TiB nvme 23 2048 46 TiB 25.18M 62 TiB 31.62 67 TiB .nfs 25 32 4.6 KiB 10 108 KiB 0 708 GiB cephfs.cephfs.meta 31 32 25 GiB 1.66M 73 GiB 3.32 708 GiB cephfs.cephfs.data 32 679 489 B 40.41M 48 KiB 0 708 GiB cephfs.nvme.data 34 32 0 B 0 0 B 0 67 TiB cephfs.ssd.data 35 32 77 GiB 425.03k 134 GiB 5.94 1.0 TiB cephfs.hdd.data 37 2048 121 TiB 68.42M 250 TiB 23.03 371 TiB rbd.ssd.data 38 64 934 GiB 239.94k 1.8 TiB 45.82 944 GiB " The most weird one: Pool rbd_ec_data stores 683TB in 4096 pgs -> warn should be 1024 Pool rbd_internal stores 86TB in 1024 pgs-> warn should be 2048 That makes no sense to me based on the amount of data stored. Is this a bug or what am I missing? Ceph version is 17.2.7. Mvh. Torkil -- Torkil Svensgaard Systems Administrator Danish Research Centre for Magnetic Resonance DRCMR, Section 714 Copenhagen University Hospital Amager and Hvidovre Kettegaard Allé 30, 2650 Hvidovre, Denmark

3 2

Questions about the CRUSH details
by Henry lol 26 Jan '24

26 Jan '24

Hello, I'm new to ceph and sorry in advance for the naive questions. 1. As far as I know, CRUSH utilizes the cluster map consisting of the PG map and others. I don't understand why CRUSH computation is required on client-side, even though PG-to-OSDs mapping can be acquired from the PG map. 2. how does the client get a valid(old) OSD set when the PG is being remapped to a new ODS set which CRUSH returns? thanks.

5 9

podman / docker issues
by Marc 25 Jan '24

25 Jan '24

More and more I am annoyed with the 'dumb' design decisions of redhat. Just now I have an issue on an 'air gapped' vm that I am unable to start a docker/podman container because it tries to contact the repository to update the image and instead of using the on disk image it just fails. (Not to mention the %$#$%#$ that design containers to download stuff from the internet on startup) I was wondering if this is also an issue with ceph-admin. Is there an issue with starting containers when container image repositories are not available or when there is no internet connection.

3 2

RGW crashes when rgw_enable_ops_log is enabled
by Marc Singer 25 Jan '24

25 Jan '24

Hi Ceph Users I am encountering a problem with the RGW Admin Ops Socket. I am setting up the socket as follows: rgw_enable_ops_log = true rgw_ops_log_socket_path = /tmp/ops/rgw-ops.socket rgw_ops_log_data_backlog = 16Mi Seems like the socket fills up over time and it doesn't seem to get flushed, at some point the process runs out of file space. Do I need to configure something or send something for the socket to flush? See the log here: 0> 2024-01-25T13:10:13.908+0000 7f247b00eb00 -1 *** Caught signal (File size limit exceeded) ** in thread 7f247b00eb00 thread_name:ops_log_file ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable) NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 rbd_pwl 0/ 5 journaler 0/ 5 objectcacher 0/ 5 immutable_obj_cache 0/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 0 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 1 reserver 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 rgw_sync 1/ 5 rgw_datacache 1/ 5 rgw_access 1/ 5 rgw_dbstore 1/ 5 rgw_flight 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 1/ 5 fuse 2/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace 1/ 5 prioritycache 0/ 5 test 0/ 5 cephfs_mirror 0/ 5 cephsqlite 0/ 5 seastore 0/ 5 seastore_onode 0/ 5 seastore_odata 0/ 5 seastore_omap 0/ 5 seastore_tm 0/ 5 seastore_t 0/ 5 seastore_cleaner 0/ 5 seastore_epm 0/ 5 seastore_lba 0/ 5 seastore_fixedkv_tree 0/ 5 seastore_cache 0/ 5 seastore_journal 0/ 5 seastore_device 0/ 5 seastore_backref 0/ 5 alienstore 1/ 5 mclock 0/ 5 cyanstore 1/ 5 ceph_exporter 1/ 5 memstore -2/-2 (syslog threshold) 99/99 (stderr threshold) --- pthread ID / name mapping for recent threads --- 7f2472a89b00 / safe_timer 7f2472cadb00 / radosgw ... log_file /var/lib/ceph/crash/2024-01-25T13:10:13.909546Z_01ee6e6a-e946-4006-9d32-e17ef2f9df74/log --- end dump of recent events --- reraise_fatal: default handler for signal 25 didn't terminate the process? Thank you for your help. Marc

2 2

TLS 1.2 for dashboard
by Sake Ceph 25 Jan '24

25 Jan '24

After upgrading to 17.2.7 our load balancers can't check the status of the manager nodes for the dashboard. After some troubleshooting I noticed only TLS 1.3 is availalbe for the dashboard. Looking at the source (quincy), TLS config got changed from 1.2 to 1.3. Searching in the tracker I found out that we are not the only one with troubles and there will be added an option to the dashboard config. Tracker ID 62940 got backports and the ones for reef and pacific already merged. But the pull request (63068) for Quincy is closed :( What to do? I hope this one can get merged for 17.2.8.

2 5

Stupid question about ceph fs volume
by Albert Shih 25 Jan '24

25 Jan '24

Hi everyone, Stupid question about ceph fs volume create how can I specify the metadata pool and the data pool ? I was able to create a cephfs «manually» with something like ceph fs new vo cephfs_metadata cephfs_data but as I understand the documentation, with this method I need to deploy the mds, and the «new» way to do it is to use ceph fs volume. But with ceph fs volume I didn't find any documentation of how to set the metadata/data pool I also didn't find any way to change after the creation of the volume the pool. Thanks -- Albert SHIH 🦫 🐸 France Heure locale/Local time: mer. 24 janv. 2024 19:24:23 CET

3 8

Ceph 16.2.14: ceph-mgr getting oom-killed
by Zakhar Kirpichenko 25 Jan '24

25 Jan '24

Hi, I'm facing a rather new issue with our Ceph cluster: from time to time ceph-mgr on one of the two mgr nodes gets oom-killed after consuming over 100 GB RAM: [Nov21 15:02] tp_osd_tp invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0 [ +0.000010] oom_kill_process.cold+0xb/0x10 [ +0.000002] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name [ +0.000008] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=504d37b566d9fd442d45904a00584b4f61c93c5d49dc59eb1c948b3d1c096907,mems_allowed=0-1,global_oom,task_memcg=/docker/3826be8f9115479117ddb8b721ca57585b2bdd58a27c7ed7b38e8d83eb795957,task=ceph-mgr,pid=3941610,uid=167 [ +0.000697] Out of memory: Killed process 3941610 (ceph-mgr) total-vm:146986656kB, anon-rss:125340436kB, file-rss:0kB, shmem-rss:0kB, UID:167 pgtables:260356kB oom_score_adj:0 [ +6.509769] oom_reaper: reaped process 3941610 (ceph-mgr), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB The cluster is stable and operating normally, there's nothing unusual going on before, during or after the kill, thus it's unclear what causes the mgr to balloon, use all RAM and get killed. Systemd logs aren't very helpful: they just show normal mgr operations until it fails to allocate memory and gets killed: https://pastebin.com/MLyw9iVi The mgr experienced this issue several times in the last 2 months, and the events don't appear to correlate with any other events in the cluster because basically nothing else happened at around those times. How can I investigate this and figure out what's causing the mgr to consume all memory and get killed? I would very much appreciate any advice! Best regards, Zakhar

6 21

cephfs-top causes 16 mgr modules have recently crashed
by Özkan Göksu 25 Jan '24

25 Jan '24

Hello When I run cephfs-top it causes mgr module crash. Can you please tell me the reason? My environment: My ceph version 17.2.6 Operating System: Ubuntu 22.04.2 LTS Kernel: Linux 5.15.0-84-generic I created the cephfs-top user with the following command: ceph auth get-or-create client.fstop mon 'allow r' mds 'allow r' osd 'allow r' mgr 'allow r' > /etc/ceph/ceph.client.fstop.keyring This is the crash report: root@ud-01:~# ceph crash info 2024-01-22T21:25:59.313305Z_526253e3-e8cc-4d2c-adcb-69a7c9986801 { "backtrace": [ " File \"/usr/share/ceph/mgr/stats/module.py\", line 32, in notify\n self.fs_perf_stats.notify_cmd(notify_id)", " File \"/usr/share/ceph/mgr/stats/fs/perf_stats.py\", line 177, in notify_cmd\n metric_features = int(metadata[CLIENT_METADATA_KEY][\"metric_spec\"][\"metric_flags\"][\"feature_bits\"], 16)", "ValueError: invalid literal for int() with base 16: '0x'" ], "ceph_version": "17.2.6", "crash_id": "2024-01-22T21:25:59.313305Z_526253e3-e8cc-4d2c-adcb-69a7c9986801", "entity_name": "mgr.ud-01.qycnol", "mgr_module": "stats", "mgr_module_caller": "ActivePyModule::notify", "mgr_python_exception": "ValueError", "os_id": "centos", "os_name": "CentOS Stream", "os_version": "8", "os_version_id": "8", "process_name": "ceph-mgr", "stack_sig": "971ae170f1fff7f7bc0b7ae86d164b2b0136a8bd5ca7956166ea5161e51ad42c", "timestamp": "2024-01-22T21:25:59.313305Z", "utsname_hostname": "ud-01", "utsname_machine": "x86_64", "utsname_release": "5.15.0-84-generic", "utsname_sysname": "Linux", "utsname_version": "#93-Ubuntu SMP Tue Sep 5 17:16:10 UTC 2023" } Best regards.

2 4

[DOC] Openstack with RBD DOC update?
by Murilo Morais 25 Jan '24

25 Jan '24

Good afternoon everybody! I have a question regarding the documentation... I was reviewing it and realized that the "vms" pool is not being used anywhere in the configs. The first mention of this pool was in commit 2eab1c1 and, in e9b13fa, the configuration section of nova.conf was removed, but the pool configuration remained there. Would it be correct to ignore all mentions of this pool (I don't see any use for it)? If so, it would be interesting to update the documentation. https://docs.ceph.com/en/latest/rbd/rbd-openstack/#create-a-pool

5 7

CLT meeting notes January 24th 2024
by Adam King 24 Jan '24

24 Jan '24

- Build/package PRs- who to best review these? - Example: https://github.com/ceph/ceph/pull/55218 - Idea: create a GitHub team specifically for these types of PRs https://github.com/orgs/ceph/teams - Laura will try to organize people for the group - Pacific 16.2.15 status - Handful of PRs left in 16.2.15 tag https://github.com/ceph/ceph/pulls?q=is%3Apr+is%3Aopen+milestone%3Av16.2.15 that still need to be tested and merged - Yuri will begin testing RC after that

1 0

Degraded PGs on EC pool when marking an OSD out
by Hector Martin 24 Jan '24

24 Jan '24

I'm having a bit of a weird issue with cluster rebalances with a new EC pool. I have a 3-machine cluster, each machine with 4 HDD OSDs (+1 SSD). Until now I've been using an erasure coded k=5 m=3 pool for most of my data. I've recently started to migrate to a k=5 m=4 pool, so I can configure the CRUSH rule to guarantee that data remains available if a whole host goes down (3 chunks per host, 9 total). I also moved the 5,3 pool to this setup, although by nature I know its PGs will become inactive if a host goes down (need at least k+1 OSDs to be up). I've only just started migrating data to the 5,4 pool, but I've noticed that any time I trigger any kind of backfilling (e.g. take one OSD out), a bunch of PGs in the 5,4 pool become degraded (instead of just misplaced/backfilling). This always seems to happen on that pool only, and the object count is a significant fraction of the total pool object count (it's not just "a few recently written objects while PGs were repeering" or anything like that, I know about that effect). Here are the pools: pool 13 'cephfs2_data_hec5.3' erasure profile ec5.3 size 8 min_size 6 crush_rule 7 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode warn last_change 14133 lfor 0/11307/11305 flags hashpspool,ec_overwrites,bulk stripe_width 20480 application cephfs pool 14 'cephfs2_data_hec5.4' erasure profile ec5.4 size 9 min_size 6 crush_rule 7 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode warn last_change 14509 lfor 0/0/14234 flags hashpspool,ec_overwrites,bulk stripe_width 20480 application cephfs EC profiles: # ceph osd erasure-code-profile get ec5.3 crush-device-class= crush-failure-domain=osd crush-root=default jerasure-per-chunk-alignment=false k=5 m=3 plugin=jerasure technique=reed_sol_van w=8 # ceph osd erasure-code-profile get ec5.4 crush-device-class= crush-failure-domain=osd crush-root=default jerasure-per-chunk-alignment=false k=5 m=4 plugin=jerasure technique=reed_sol_van w=8 They both use the same CRUSH rule, which is designed to select 9 OSDs balanced across the hosts (of which only 8 slots get used for the older 5,3 pool): rule hdd-ec-x3 { id 7 type erasure step set_chooseleaf_tries 5 step set_choose_tries 100 step take default class hdd step choose indep 3 type host step choose indep 3 type osd step emit } If I take out an OSD (14), I get something like this: health: HEALTH_WARN Degraded data redundancy: 37631/120155160 objects degraded (0.031%), 38 pgs degraded All the degraded PGs are in the 5,4 pool, and the total object count is around 50k, so this is *most* of the data in the pool becoming degraded just because I marked an OSD out (without stopping it). If I mark the OSD in again, the degraded state goes away. Example degraded PGs: # ceph pg dump | grep degraded dumped all 14.3c 812 0 838 0 0 11925027758 0 0 1088 0 1088 active+recovery_wait+undersized+degraded+remapped 2024-01-19T18:06:41.786745+0900 15440'1088 15486:10772 [18,17,16,1,3,2,11,13,12] 18 [18,17,16,1,3,2,11,NONE,12] 18 14537'432 2024-01-12T11:25:54.168048+0900 0'0 2024-01-08T15:18:21.654679+0900 0 2 periodic scrub scheduled @ 2024-01-21T08:00:23.572904+0900 241 0 14.3d 772 0 1602 0 0 11303280223 0 0 1283 0 1283 active+recovery_wait+undersized+degraded+remapped 2024-01-19T18:06:41.919971+0900 15470'1283 15486:13384 [18,17,16,3,1,0,13,11,12] 18 [18,17,16,3,1,0,NONE,NONE,12] 18 14990'771 2024-01-15T12:15:59.397469+0900 0'0 2024-01-08T15:18:21.654679+0900 0 3 periodic scrub scheduled @ 2024-01-23T15:56:58.912801+0900 534 0 14.3e 806 0 832 0 0 11843019697 0 0 1035 0 1035 active+recovery_wait+undersized+degraded+remapped 2024-01-19T18:06:42.297251+0900 15465'1035 15486:15423 [18,16,17,12,13,11,1,3,0] 18 [18,16,17,12,13,NONE,1,3,0] 18 14623'500 2024-01-13T08:54:55.709717+0900 0'0 2024-01-08T15:18:21.654679+0900 0 1 periodic scrub scheduled @ 2024-01-22T09:54:51.278368+0900 331 0 14.3f 782 0 813 0 0 11598393034 0 0 1083 0 1083 active+recovery_wait+undersized+degraded+remapped 2024-01-19T18:06:41.845173+0900 15465'1083 15486:18496 [17,18,16,3,0,1,11,12,13] 17 [17,18,16,3,0,1,11,NONE,13] 17 14990'800 2024-01-15T16:42:08.037844+0900 14990'800 2024-01-15T16:42:08.037844+0900 0 40 periodic scrub scheduled @ 2024-01-23T10:44:06.083985+0900 563 0 The first PG when I put the OSD back in: 14.3c 812 0 0 0 0 11925027758 0 0 1088 0 1088 active+clean 2024-01-19T18:07:18.079295+0900 15440'1088 15489:10792 [18,17,16,1,3,2,11,14,12] 18 [18,17,16,1,3,2,11,14,12] 18 14537'432 2024-01-12T11:25:54.168048+0900 0'0 2024-01-08T15:18:21.654679+0900 0 2 periodic scrub scheduled @ 2024-01-21T09:41:43.026836+0900 241 0 As far as I know PGs are not supposed to actually become *degraded* when merely moving data around without any OSDs going down. Am I doing something wrong here? Any idea why this is affecting one pool and not both, even though they are almost identical in setup? It's as if, for this one pool, marking an OSD out has the effect of making its data unavailable entirely, instead of merely backfill to other OSDs (the OSD shows up as NONE in the above dump). OSD tree: ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 89.13765 root default -13 29.76414 host flamingo 11 hdd 7.27739 osd.11 up 1.00000 1.00000 12 hdd 7.27739 osd.12 up 1.00000 1.00000 13 hdd 7.27739 osd.13 up 1.00000 1.00000 14 hdd 7.20000 osd.14 up 1.00000 1.00000 8 ssd 0.73198 osd.8 up 1.00000 1.00000 -10 29.84154 host heart 0 hdd 7.27739 osd.0 up 1.00000 1.00000 1 hdd 7.27739 osd.1 up 1.00000 1.00000 2 hdd 7.27739 osd.2 up 1.00000 1.00000 3 hdd 7.27739 osd.3 up 1.00000 1.00000 9 ssd 0.73198 osd.9 up 1.00000 1.00000 -3 0 host hub -7 29.53197 host soleil 15 hdd 7.20000 osd.15 up 0 1.00000 16 hdd 7.20000 osd.16 up 1.00000 1.00000 17 hdd 7.20000 osd.17 up 1.00000 1.00000 18 hdd 7.20000 osd.18 up 1.00000 1.00000 10 ssd 0.73198 osd.10 up 1.00000 1.00000 (I'm in the middle of doing some reprovisioning so 15 is out, this happens any time I take any OSD out) # ceph --version ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable) - Hector

3 4

cephx client key rotation
by Peter Sabaini 24 Jan '24

24 Jan '24

Hi, this question has come up once in the past[0] afaict, but it was kind of inconclusive so I'm taking the liberty of bringing it up again. I'm looking into implementing a key rotation scheme for Ceph client keys. As it potentially takes some non-zero amount of time to update key material there might be a situation where keys have changed on the MON side but, still one of N clients might not have updated key material and try to auth with an obsolete key which naturally would fail. It would be great if we could have two keys active for an entity at the same time, but aiui that's not really possible, is that right? I'm wondering about ceph auth get-or-create-pending. Per the docs a pending key would become active on first use, so that if one of N clients uses it, this still leaves room for another client to race. What do people do to deal with this situation? [0] https://ceph-users.ceph.narkive.com/ObSMdmxX/rotating-cephx-keys

1 0

Cephadm orchestrator and special label _admin in 17.2.7
by Robert Sander 24 Jan '24

24 Jan '24

Hi, According to the documentation¹ the special host label _admin instructs the cephadm orchestrator to place a valid ceph.conf and the ceph.client.admin.keyring into /etc/ceph of the host. I noticed that (at least) on 17.2.7 only the keyring file is placed in /etc/ceph, but not ceph.conf. Both files are placed into the /var/lib/ceph/<fsid>/config directory. Has something changed? ¹: https://docs.ceph.com/en/quincy/cephadm/host-management/#special-host-labels Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

4 8

List contents of stray buckets with octopus
by Frank Schilder 24 Jan '24

24 Jan '24

Hi all, I need to list the contents of the stray buckets on one of our MDSes. The MDS reports 772674 stray entries. However, if I dump its cache and grep for stray I get only 216 hits. How can I get to the contents of the stray buckets? Please note that Octopus is still hit by https://tracker.ceph.com/issues/57059 so a "dump tree" will not work. In addition, I clearly don't just need the entries in cache, I need a listing of everything. How can I get that? I'm willing to run rados commands and pipe through ceph-encoder if necessary. Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

1 0

How many pool for cephfs
by Albert Shih 24 Jan '24

24 Jan '24

Hi everyone, I like to know how many pool should I create for multiple cephfs ? Knowing I got two class of osd (hdd and ssd), and I have a need of ~ 20/30 cephfs (currently and that number will increase with time). Should I create one cephfs_metadata_replicated one cephfs_data_replicated few cephfs_data_erasure_coding (depending of k/m) and put all my cephfs inside two of them. Or should I create for each cephfs a couple of pool metadata/data ? Il will also need to have ceph S3 storage, same question, should I have a designated pool for S3 storage or can/should I use the same cephfs_data_replicated/erasure pool ? Regards -- Albert SHIH 🦫 🐸 France Heure locale/Local time: mer. 24 janv. 2024 09:33:09 CET

3 6

Unable to locate "bluestore_compressed_allocated" & "bluestore_compressed_original" parameters while executing "ceph daemon osd.X perf dump" command.
by Alam Mohammad 23 Jan '24

23 Jan '24

Hi, We are considering BlueStore compression test in our cluster. For this we have created rbd image on our EC pool. While we are executing "ceph daemon osd.X perf dump | grep -E '(compress_.*_count|bluestore_compressed_)'", we are not locate below parameters, even we tried with ceph tell command. "bluestore_compressed_allocated" "bluestore_compressed_original" As a result, we are unable to determine the extent of compression for specific RBD images. Are there any specific configurations required to locate the given parameters or any alternate method to assess Bluestore compression. Any guidance or insight would be greatly appreciated. Thanks Mohammad Saif Ceph Enthusiast

2 3

RFI: Prometheus, Etc, Services - Optimum Number To Run
by duluxoz 21 Jan '24

21 Jan '24

Hi All, In regards to the monitoring services on a Ceph Cluster (ie Prometheus, Grafana, Alertmanager, Loki, Node-Exported, Promtail, etc) how many instances should/can we run for fault tolerance purposes? I can't seem to recall that advice being in the doco anywhere (but of course, I probably missed it). I'm concerned about HA on those services - will they continue to run if the Ceph Node they're on fails? At the moment we're running only 1 instance of each in the cluster, but several Ceph Nodes are capable of running each - ie/eg 3 nodes configured but only count:1. This is on the latest version of Reef using cephadmin (if it makes a huge difference :-) ). So any advice, etc, would be greatly appreciated, including if we should be running any services not mentioned (not Mgr, Mon, OSD, or iSCSI, obviously :-) ) Cheers Dulux-Oz

2 1

rbd map snapshot, mount lv, node crash
by Marc 19 Jan '24

19 Jan '24

Am I doing something weird when I do on a ceph node (nautilus, el7): rbd snap ls vps-test -p rbd rbd map vps-test(a)vps-test.snap1 -p rbd mount -o ro /dev/mapper/VGnew-LVnew /mnt/disk <--- reset/reboot ceph node

2 1

How does mclock work?
by Frédéric Nass 19 Jan '24

19 Jan '24

Hello, Could someone please explain how mclock works regarding reads and writes? Does mclock intervene on both read and write iops? Or only on reads or only on writes? And what type of underlying hardware performance is calculated and considered by mclock? Seems to be only write performance. The mclock documentation shows HDDs and SSDs specific configuration options (capacity and sequential bandwidth) but nothing regarding hybrid setups and these configuration options do not distinguish reads and writes. But read and write performance are often not in par for a single drive and even less when using hybrid setups. With hybrid setups (RocksDB+WAL on SSDs or NVMes and Data on HDD), if mclock only considers write performance, it may fail to properly schedule read iops (does mclock schedule read iops?) as the calculated iops capacity would be way too high for reads. With HDD only setups (RocksDB+WAL+Data on HDD), if mclock only considers write performance, the OSD may not take advantage of higher read performance. Can someone please shed some light on this? Best regards, Frédéric Nass Sous-direction Infrastructures et Services Direction du Numérique Université de Lorraine Tél : +33 3 72 74 11 35

11 14

Keyring location for ceph-crash?
by Jan Kasprzak 19 Jan '24

19 Jan '24

Hello, Ceph users, what is the correct location of keyring for ceph-crash? I tried to follow this document: https://docs.ceph.com/en/latest/mgr/crash/ # ceph auth get-or-create client.crash mon 'profile crash' mgr 'profile crash' > /etc/ceph/ceph.client.crash.keyring and copy this file to all nodes. When I run ceph-crash.service on the node where client.admin.keyring is available, it works. But on the rest of nodes, it tries to access the admin keyring anyway. Journalctl -u ceph-crash says this: Jan 18 18:03:35 my.node.name systemd[1]: Started Ceph crash dump collector. Jan 18 18:03:35 my.node.name ceph-crash[2973164]: INFO:ceph-crash:pinging cluster to exercise our key Jan 18 18:03:35 my.node.name ceph-crash[2973166]: 2024-01-18T18:03:35.786+0100 7eff34016640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory Jan 18 18:03:35 my.node.name ceph-crash[2973166]: 2024-01-18T18:03:35.786+0100 7eff34016640 -1 AuthRegistry(0x7eff2c063ce8) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx Jan 18 18:03:35 my.node.name ceph-crash[2973166]: 2024-01-18T18:03:35.787+0100 7eff34016640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory Jan 18 18:03:35 my.node.name ceph-crash[2973166]: 2024-01-18T18:03:35.787+0100 7eff34016640 -1 AuthRegistry(0x7eff2c067de0) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx Jan 18 18:03:35 my.node.name ceph-crash[2973166]: 2024-01-18T18:03:35.788+0100 7eff34016640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory Jan 18 18:03:35 my.node.name ceph-crash[2973166]: 2024-01-18T18:03:35.788+0100 7eff34016640 -1 AuthRegistry(0x7eff340150c0) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx Jan 18 18:03:35 my.node.name ceph-crash[2973166]: [errno 2] RADOS object not found (error connecting to the cluster) Jan 18 18:03:35 my.node.name ceph-crash[2973164]: INFO:ceph-crash:monitoring path /var/lib/ceph/crash, delay 600s Is the documentation outdated, or am I doing anything wrong? Thanks for any hint. This is non-containerized Ceph 18.2.1 on AlmaLinux 9. -Yenya -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall

2 2

Indexless bucket constraints of ceph-rgw
by Jaemin Joo 19 Jan '24

19 Jan '24

I am considering Indexless bucket to increase the small file performance and maximum number of files using just HDDs. When I checked the constraints of indexless bucket in ceph-docs, it indicates the possible for bucket list problem, so I just know it makes the problem of versioning and sync. I don't understand what means of it. and I just guessed it's similar with db index feature. but it's not. Could you tell me the indexless bucket constraints?

1 0

Join us for the User + Dev Monthly Meetup - January 18th!
by Laura Flores 18 Jan '24

18 Jan '24

Hi Ceph users and developers, You are invited to join us at the User + Dev meeting this week Thursday, January 18th at 10:00 AM Eastern Time! See below for more meeting details. The focus topic, "Ceph Feature Request from the DKIST Data Center: Add a service backed by tape that is analogous to AWS Glacier", will be presented by Joel Davidow, a Ceph operator from the National Solar Observatory. In his talk, he will propose a feature request to add support for tape as a storage class with lifecycle management for object storage. Feel free to add questions or additional topics under the "Open Discussion" section on the agenda: https://pad.ceph.com/p/ceph-user-dev-monthly-minutes If you have an idea for a focus topic you'd like to present at a future meeting, you are welcome to submit it to this Google Form: https://docs.google.com/forms/d/e/1FAIpQLSdboBhxVoBZoaHm8xSmeBoemuXoV_rmh4v… Any Ceph user or developer is eligible to submit! Thanks, Laura Flores Meeting link: https://meet.jit.si/ceph-user-dev-monthly Time Conversions: UTC: Thursday, January 18, 15:00 UTC Mountain View, CA, US: Thursday, January 18, 7:00 PST Phoenix, AZ, US: Thursday, January 18, 8:00 MST Denver, CO, US: Thursday, January 18, 8:00 MST Huntsville, AL, US: Thursday, January 18, 9:00 CST Raleigh, NC, US: Thursday, January 18, 10:00 EST London, England: Thursday, January 18, 15:00 GMT Paris, France: Thursday, January 18, 16:00 CET Helsinki, Finland: Thursday, January 18, 17:00 EET Tel Aviv, Israel: Thursday, January 18, 17:00 IST Pune, India: Thursday, January 18, 20:30 IST Brisbane, Australia: Friday, January 19, 1:00 AEST Singapore, Asia: Thursday, January 18, 23:00 +08 Auckland, New Zealand: Friday, January 19, 4:00 NZDT -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

1 2

Wide EC pool causes very slow backfill?
by Torkil Svensgaard 18 Jan '24

18 Jan '24

Hi Our 17.2.7 cluster: " -33 886.00842 datacenter 714 -7 209.93135 host ceph-hdd1 -69 69.86389 host ceph-flash1 -6 188.09579 host ceph-hdd2 -3 233.57649 host ceph-hdd3 -12 184.54091 host ceph-hdd4 -34 824.47168 datacenter DCN -73 69.86389 host ceph-flash2 -5 252.27127 host ceph-hdd14 -2 201.78067 host ceph-hdd5 -81 288.26501 host ceph-hdd6 -31 264.56207 host ceph-hdd7 -36 1284.48621 datacenter TBA -77 69.86389 host ceph-flash3 -21 190.83224 host ceph-hdd8 -29 199.08838 host ceph-hdd9 -11 193.85382 host ceph-hdd10 -9 237.28154 host ceph-hdd11 -26 187.19536 host ceph-hdd12 -4 206.37102 host ceph-hdd13 " We recently created an EC 4+5 pool with failure domain datacenter. The DCN datacenter only had 2 hdd hosts so we added one more to make it possible at all, since each DC needs 3 shards, as I understand it. Backfill was really slow though, so we just added another host to the DCN datacenter. Backfill looks like this: " data: volumes: 1/1 healthy pools: 13 pools, 11153 pgs objects: 311.53M objects, 1000 TiB usage: 1.6 PiB used, 1.6 PiB / 3.2 PiB avail pgs: 60/1669775060 objects degraded (0.000%) 373356926/1669775060 objects misplaced (22.360%) 5944 active+clean 5177 active+remapped+backfill_wait 22 active+remapped+backfilling 4 active+recovery_wait+degraded+remapped 3 active+recovery_wait+remapped 2 active+recovery_wait+degraded 1 active+recovering+degraded+remapped io: client: 73 MiB/s rd, 339 MiB/s wr, 1.06k op/s rd, 561 op/s wr recovery: 1.2 GiB/s, 313 objects/s " Given that the first host added had 19 OSDs, with none of them anywhere near the target capacity, and the one we just added has 22 empty OSDs, having just 22 PGs backfilling and 1 recovering seems somewhat underwhelming. Is this to be expected with such a pool? Mclock profile is high_recovery_ops. Mvh. Torkil -- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil(a)drcmr.dk

3 6

Debian 12 (bookworm) / Reef 18.2.1 problems
by Chris Palmer 18 Jan '24

18 Jan '24

I was delighted to see the native Debian 12 (bookworm) packages turn up in Reef 18.2.1. We currently run a number of ceph clusters on Debian11 (bullseye) / Quincy 17.2.7. These are not cephadm-managed. I have attempted to upgrade a test cluster, and it is not going well. Quincy only supports bullseye, and Reef only supports bookworm, we are reinstalling from bare metal. However I don't think either of these two problems are related to that. Problem 1 -------------- A simple "apt install ceph" goes most of the way, then errors with Setting up cephadm (18.2.1-1~bpo12+1) ... usermod: unlocking the user's password would result in a passwordless account. You should set a password with usermod -p to unlock this user's password. mkdir: cannot create directory ‘/home/cephadm/.ssh’: No such file or directory dpkg: error processing package cephadm (--configure): installed cephadm package post-installation script subprocess returned error exit status 1 dpkg: dependency problems prevent configuration of ceph-mgr-cephadm: ceph-mgr-cephadm depends on cephadm; however: Package cephadm is not configured yet. dpkg: error processing package ceph-mgr-cephadm (--configure): dependency problems - leaving unconfigured The two cephadm-related packages are then left in an error state, which apt tries to continue each time it is run. The cephadm user has a login directory of /nonexistent, however the cephadm --configure script is trying to use /home/cephadm (as it was on Quincy/bullseye). So, we aren't using cephadm, and decided to keep going as the other packages were actually installed, and deal with the package state later. Problem 2 --------------- I upgraded 2/3 monitor nodes without any other problems, and (for the moment) removed the other Quincy monitor prior to rebuild. I then shutdown the remaining Quincy manager, and attempted to start the Reef manager. Although the manager is running, "ceph mgr services" shows it is only providing the restful and not the dashboard service. The log file has lots of the following error: ImportError: PyO3 modules may only be initialized once per interpreter process and ceph -s reports "Module 'dashboard' has failed dependency: PyO3 modules may only be initialized once per interpreter process Questions --------------- 1. Have the Reef/bookworm packages ever been tested in a non-cephadm environment? 2. I want to revert this cluster back to a fully functional state. I cannot bring back up the remaining Quincy monitor though ("require release 18 > 17"). Would I have to go through the procedure of starting over, and trying to rescue the monmap from the OSDs? (OSDs and an active MDS are still up and running Quincy). I'm aware that process exists but have never had to delve into it. Thanks, Chris

3 5

Adding OSD's results in slow ops, inactive PG's
by Ruben Vestergaard 18 Jan '24

18 Jan '24

Hi We have a cluster with which currently looks like so: services: mon: 5 daemons, quorum lazy,jolly,happy,dopey,sleepy (age 13d) mgr: jolly.tpgixt(active, since 25h), standbys: dopey.lxajvk, lazy.xuhetq mds: 1/1 daemons up, 2 standby osd: 449 osds: 425 up (since 15m), 425 in (since 5m); 5104 remapped pgs data: volumes: 1/1 healthy pools: 13 pools, 11153 pgs objects: 304.11M objects, 988 TiB usage: 1.6 PiB used, 1.4 PiB / 2.9 PiB avail pgs: 6/1617270006 objects degraded (0.000%) 366696947/1617270006 objects misplaced (22.674%) 6043 active+clean 5041 active+remapped+backfill_wait 66 active+remapped+backfilling 2 active+recovery_wait+degraded+remapped 1 active+recovering+degraded It's currently rebalancing after adding a node, but this rebalance has been rather slow -- right now it's running 66 backfills, but it seems to stabilize around 8 backfills eventually. We figured that perhaps adding another node might speed things up. Immediately upon adding the node, we get slow ops and inactive PG's. Removing the new node gets us back in working order. It turns out that even adding 1 OSD breaks the cluster, and immediately sends it here: [WRN] PG_DEGRADED: Degraded data redundancy: 6/1617265712 objects degraded (0.000%), 3 pgs degraded pg 37.c8 is active+recovery_wait+degraded+remapped, acting [410,163,236,209,7,283,155,143,78] pg 37.1a1 is active+recovering+degraded, acting [234,424,163,74,22,128,177,153,181] pg 37.1da is active+recovery_wait+degraded+remapped, acting [163,408,230,190,93,284,50,78,44] [WRN] SLOW_OPS: 22 slow ops, oldest one blocked for 54 sec, daemons [osd.11,osd.110,osd.112,osd.117,osd.120,osd.123,osd.13,osd.136,osd.144,osd.157]... have slow ops. The OSD added had number 431, so it does not appear to be the immediate cause of the slow ops, however, removing 431 immediately clears the problem. We thought we might be experiencing 'Crush giving up too soon' symptoms [1], as we have seen similar behaviour on another pool, but it does not appear to be the case here. We went through the motions described on the page and everything looked OK. At least one pool which stops working is a 4+2 EC pool, placed on spinning rust, some 200-ish disks distributed across 13 nodes. I'm not sure if other pools break, but that particular 4+2 EC pool is rather important so I'm a little wary of experimenting blindly. Any thoughts on where to look next? Thanks, Ruben Vestergaard [1] https://docs.ceph.com/en/reef/rados/troubleshooting/troubleshooting-pg/#cru…

4 9

minimal permission set for an rbd client
by cek+ceph＠deepunix.net 18 Jan '24

18 Jan '24

I'm following the guide @ https://docs.ceph.com/en/latest/rbd/rados-rbd-cmds/ but I'm not following why would an `mgr` permission be required to have a functioning RBD client? Thanks.

2 1

recommendation for barebones server with 8-12 direct attach NVMe?
by Drew Weaver 17 Jan '24

17 Jan '24

Hello, So we were going to replace a Ceph cluster with some hardware we had laying around using SATA HBAs but I was told that the only right way to build Ceph in 2023 is with direct attach NVMe. Does anyone have any recommendation for a 1U barebones server (we just drop in ram disks and cpus) with 8-10 2.5" NVMe bays that are direct attached to the motherboard without a bridge or HBA for Ceph specifically? Thanks, -Drew

7 19

Re: Stuck in upgrade process to reef
by Igor Fedotov 17 Jan '24

17 Jan '24

Hi Jan, I've just fired an upstream ticket for your case, see https://tracker.ceph.com/issues/64053 for more details. You might want to tune (or preferably just remove) your custom bluestore_cache_.*_ratio settings to fix the issue. This is reproducible and fixable in my lab this way. Hope this helps. Thanks, Igor On 15/01/2024 12:54, Jan Marek wrote: > Hi Igor, > > I've tried to start ceph-sod daemon as you advice me and I'm > sending log osd.1.start.log > > About memory: According to 'top' podman ceph daemon don't reach > 2% of whole server memory (64GB)... > > I have switch on autotune of memory... > > My ceph config dump - see attached dump.txt > > Sincerely > Jan Marek > > Dne Čt, led 11, 2024 at 04:02:02 CET napsal(a) Igor Fedotov: >> Hi Jan, >> >> unfortunately this wasn't very helpful. Moreover the log looks a bit messy - >> looks like a mixture of outputs from multiple running instances or >> something. I'm not an expert in using containerized setups though. >> >> Could you please simplify things by running ceph-osd process manually like >> you did for ceph-objectstore-tool. And enforce log output to a file. Command >> line should look somewhat the following: >> >> ceph-osd -i 0 --log-to-file --log-file <some-file> --debug-bluestore 5/20 >> --debug-prioritycache 10 >> >> Please don't forget to run repair prior to that. >> >> >> Also you haven't answered my questions about custom [memory] settings and >> RAM usage during OSD startup. It would be nice to hear some feedback. >> >> >> Thanks, >> >> Igor >> >> On 11/01/2024 16:47, Jan Marek wrote: >>> Hi Igor, >>> >>> I've tried to start osd.1 with debug_prioritycache and >>> debug_bluestore 5/20, see attached file... >>> >>> Sincerely >>> Jan >>> >>> Dne St, led 10, 2024 at 01:03:07 CET napsal(a) Igor Fedotov: >>>> Hi Jan, >>>> >>>> indeed this looks like some memory allocation problem - may be OSD's RAM >>>> usage threshold reached or something? >>>> >>>> Curious if you have any custom OSD settings or may be any memory caps for >>>> Ceph containers? >>>> >>>> Could you please set debug_bluestore to 5/20 and debug_prioritycache to 10 >>>> and try to start OSD once again. Please monitor process RAM usage along the >>>> process and share the resulting log. >>>> >>>> >>>> Thanks, >>>> >>>> Igor >>>> >>>> On 10/01/2024 11:20, Jan Marek wrote: >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 2

Upgrading nautilus / centos7 to octopus / ubuntu 20.04. - Suggestions and hints?
by Götz Reinicke 17 Jan '24

17 Jan '24

Hi, As I’v read and thought a lot about the migration as this is a bigger project, I was wondering if anyone has done that already and might share some notes or playbooks, because in all readings there where some parts missing or miss understandable to me. I do have some different approaches in mind, so may be you have some suggestions or hints. a) upgrade nautilus on centos 7 with the few missing features like dashboard and prometheus. After that migrate one node after an other to ubuntu 20.04 with octopus and than upgrade ceph to the recent stable version. b) migrate one node after an other to ubuntu 18.04 with nautilus and then upgrade to octupus and after that to ubuntu 20.04. or c) upgrade one node after an other to ubuntu 20.04 with octopus and join it to the cluster until all nodes are upgraded. For test I tried c) with a mon node, but adding that to the cluster fails with some failed state, still probing for the other mons. (I dont have the right log at hand right now.) So my questions are: a) What would be the best (most stable) migration path and b) is it in general possible to add a new octopus mon (not upgraded one) to a nautilus cluster, where the other mons are still on nautilus? I hope my thoughts and questions are understandable :) Thanks for any hint and suggestion. Best . Götz

7 11

ceph pg mark_unfound_lost delete results in confused ceph
by Oliver Dzombic 16 Jan '24

16 Jan '24

Hi, after osd.15 died in the wrong moment there is: #ceph health detail [WRN] PG_AVAILABILITY: Reduced data availability: 1 pg stale pg 10.17 is stuck stale for 3d, current state stale+active+undersized+degraded, last acting [15] [WRN] PG_DEGRADED: Degraded data redundancy: 172/57063399 objects degraded (0.000%), 1 pg degraded, 1 pg undersized pg 10.17 is stuck undersized for 3d, current state stale+active+undersized+degraded, last acting [15] which will never resolv as there is no osd.15 anymore. So a ceph pg 10.17 mark_unfound_lost delete was executed. ceph seems to be a bit confused about pg 10.17 now: While this worked before, its not working anymore # ceph pg 10.17 query Error ENOENT: i don't have pgid 10.17 And while this was pointing to 15 the map changed now to 5 and 6 ( which is correct ): # ceph pg map 10.17 osdmap e14425 pg 10.17 (10.17) -> up [5,6] acting [5,6] According to ceph health, ceph assumes that osd.15 is still somehow in charge. The pg map seems to think that 10.17 is on osd.5 and osd.6 But pg 10.17 seems not to be really existing, as a query will fail. Any idea whats going wrong and howto fix this? Thank you! -- Mit freundlichen Gruessen / Best regards Oliver Dzombic Layer7 Networks mailto:info@layer7.net Anschrift: Layer7 Networks GmbH Zum Sonnenberg 1-3 63571 Gelnhausen HRB 96293 beim Amtsgericht Hanau Geschäftsführung: Oliver Dzombic UST ID: DE259845632

1 1

[quincy 17.2.7] ceph orchestrator not doing anything
by Boris 16 Jan '24

16 Jan '24

Happy new year everybody. I just found out that the orchestrator in one of our clusters is not doing anything. What I tried until now: - disabling / enabling cephadm (no impact) - restarting hosts (no impact) - starting upgrade to same version (no impact) - starting downgrade (no impact) - forcefully removing hosts and adding them again (now I have no daemons anymore) - applying new configurations (no impact) The orchestrator just does nothing. Cluster itself is fine. I also checked the SSH connecability from all hosts to all hosts ( https://docs.ceph.com/en/quincy/cephadm/troubleshooting/#ssh-errors) The logs always show a message like "took the task" but then nothing happens. Cheers Boris

2 2

Email duplicates.
by Roman Pashin 16 Jan '24

16 Jan '24

1 0

[v18.2.1] problem with wrong osd device symlinks after upgrade to 18.2.1
by Reto Gysi 16 Jan '24

16 Jan '24

Hi ceph community I noticed the following problem after upgrading my ceph instance on Debian 12.4 from 17.2.7 to 18.2.1: I had placed bluestore block.db for hdd osd's on raid1/mirrored logical volumes on 2 nvme devices, so that if a single block.db nvme device fails, that not all hdd osds fail. That worked fine under 17.2.7 and had no problems during host/osd restarts. During the upgrade to 18.2.1 the osd's wouldn't with the block.db on mirrored lv wouldn't start anymore because the block.db symlink was updated to pointing to the wrong device mapper device, and the osd startup failed with error message that block.db device is busy. OSD1: 2024-01-05T19:56:43.592+0000 7fdde9f43640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _minimal_open_bluefs add block device(/var/lib/ceph/osd/ceph-1/block.db) returned: (16) Device or resource busy 2024-01-05T19:56:43.592+0000 7fdde9f43640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _open_db failed to prepare db environment: 2024-01-05T19:56:43.592+0000 7fdde9f43640 1 bdev(0x55a2d5014000 /var/lib/ceph/osd/ceph-1/block) close 2024-01-05T19:56:43.892+0000 7fdde9f43640 -1 osd.1 0 OSD:init: unable to mount object store the symlink was updated to point to lrwxrwxrwx 1 ceph ceph 111 Jan 5 20:57 block -> /dev/mapper/ceph--dec5bd7c--d84f--40d9--ba14--6bd8aadf2957-osd--block--cdd02721--6876--4db8--bdb2--12ac6c70127c lrwxrwxrwx 1 ceph ceph 48 Jan 5 20:57 block.db -> /dev/mapper/optane-ceph--db--osd1_rimage_1_iorig the correct symlink would have been: lrwxrwxrwx 1 ceph ceph 111 Jan 5 20:57 block -> /dev/mapper/ceph--dec5bd7c--d84f--40d9--ba14--6bd8aadf2957-osd--block--cdd02721--6876--4db8--bdb2--12ac6c70127c lrwxrwxrwx 1 ceph ceph 48 Jan 5 20:57 block.db -> /dev/mapper/optane-ceph--db--osd1 To continue with the upgrade I converted one by one all the block.db lvm logical volumes back to linear volumes, and fixed the symlinks manually. converting the lv's back to linear was necessary, because even when I fixed the symlink manually, after a osd restart the symlink would be created wrong again if the block.db would point to a raid1 lv. Here's any example how the symlink looked before an osd was touched by the 18.2.1 upgrade: OSD2: lrwxrwxrwx 1 ceph ceph 93 Jan 4 03:38 block -> /dev/ceph-17a894d6-3a64-4e5e-9fa0-8dd3b5f4bf33/osd-block-3cd7a5af-9002-47a7-b4c2-540381d53be7 lrwxrwxrwx 1 ceph ceph 24 Jan 4 03:38 block.db -> /dev/optane/ceph-db-osd2 Here's what the output of lvs -a -o +devices looked like for OSD1 block.db device when it was an raid1 lv: LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert Devices ceph-db-osd1 optane rwi-a-r--- 44.00g 100.00 ceph-db-osd1_rimage_0(0),ceph-db-osd1_rimage_1(0) [ceph-db-osd1_rimage_0] optane gwi-aor--- 44.00g [ceph-db-osd1_rimage_0_iorig] 100.00 ceph-db-osd1_rimage_0_iorig(0) [ceph-db-osd1_rimage_0_imeta] optane ewi-ao---- 428.00m /dev/sdg(55482) [ceph-db-osd1_rimage_0_imeta] optane ewi-ao---- 428.00m /dev/sdg(84566) [ceph-db-osd1_rimage_0_iorig] optane -wi-ao---- 44.00g /dev/sdg(9216) [ceph-db-osd1_rimage_0_iorig] optane -wi-ao---- 44.00g /dev/sdg(82518) [ceph-db-osd1_rimage_1] optane gwi-aor--- 44.00g [ceph-db-osd1_rimage_1_iorig] 100.00 ceph-db-osd1_rimage_1_iorig(0) [ceph-db-osd1_rimage_1_imeta] optane ewi-ao---- 428.00m /dev/sdj(55392) [ceph-db-osd1_rimage_1_imeta] optane ewi-ao---- 428.00m /dev/sdj(75457) [ceph-db-osd1_rimage_1_iorig] optane -wi-ao---- 44.00g /dev/sdj(9218) [ceph-db-osd1_rimage_1_iorig] optane -wi-ao---- 44.00g /dev/sdj(73409) [ceph-db-osd1_rmeta_0] optane ewi-aor--- 4.00m /dev/sdg(55388) [ceph-db-osd1_rmeta_1] optane ewi-aor--- 4.00m /dev/sdj(9217) It would be good if the symlinks were recreated pointing to the correct device even when they point to a raid1 lv. Not sure if this problem has been reported yet. Cheers Reto

2 4

Recomand number of k and m erasure code
by Phong Tran Thanh 16 Jan '24

16 Jan '24

Hi ceph user! I need to determine which erasure code values (k and m) to choose for a Ceph cluster with 10 nodes. I am using the reef version with rbd. Furthermore, when using a larger k, for example, ec6+2 and ec4+2, which erasure coding performance is better, and what are the criteria for choosing the appropriate erasure coding? Please help me Email: tranphong079(a)gmail.com Skype: tranphong079

3 4

RGW - user created bucket with name of already created bucket
by Ondřej Kukla 15 Jan '24

15 Jan '24

Hello, I would like to share a quite worrying experience I’ve just found on one of my production clusters. User successfully created a bucket with name of a bucket that already exists! He is not bucket owner - the original user is, but he is able to see it when he does ListBuckets over s3 api. (Both accounts are able to do it now - only the original owner is able to interact with it) This bucket is also counted to the new users usage stats. Has anyone noticed this before? This cluster is running on Quincy - 17.2.6. Is there a way to detach the bucket from the new owner so he doesn’t have a bucket that doesn’t belong to him? Regards, Ondrej

2 3

3 DC with 4+5 EC not quite working
by Torkil Svensgaard 14 Jan '24

14 Jan '24

We are looking to create a 3 datacenter 4+5 erasure coded pool but can't quite get it to work. Ceph version 17.2.7. These are the hosts (there will eventually be 6 hdd hosts in each datacenter): -33 886.00842 datacenter 714 -7 209.93135 host ceph-hdd1 -69 69.86389 host ceph-flash1 -6 188.09579 host ceph-hdd2 -3 233.57649 host ceph-hdd3 -12 184.54091 host ceph-hdd4 -34 824.47168 datacenter DCN -73 69.86389 host ceph-flash2 -2 201.78067 host ceph-hdd5 -81 288.26501 host ceph-hdd6 -31 264.56207 host ceph-hdd7 -36 1284.48621 datacenter TBA -77 69.86389 host ceph-flash3 -21 190.83224 host ceph-hdd8 -29 199.08838 host ceph-hdd9 -11 193.85382 host ceph-hdd10 -9 237.28154 host ceph-hdd11 -26 187.19536 host ceph-hdd12 -4 206.37102 host ceph-hdd13 We did this: ceph osd erasure-code-profile set DRCMR_k4m5_datacenter_hdd plugin=jerasure k=4 m=5 technique=reed_sol_van crush-root=default crush-failure-domain=datacenter crush-device-class=hdd ceph osd pool create cephfs.hdd.data erasure DRCMR_k4m5_datacenter_hdd ceph osd pool set cephfs.hdd.data allow_ec_overwrites true ceph osd pool set cephfs.hdd.data pg_autoscale_mode warn Didn't quite work: " [WARN] PG_AVAILABILITY: Reduced data availability: 1 pg inactive, 1 pg incomplete pg 33.0 is creating+incomplete, acting [104,219,NONE,NONE,NONE,41,NONE,NONE,NONE] (reducing pool cephfs.hdd.data min_size from 5 may help; search ceph.com/docs for 'incomplete') " I then manually changed the crush rule from this: " rule cephfs.hdd.data { id 7 type erasure step set_chooseleaf_tries 5 step set_choose_tries 100 step take default class hdd step chooseleaf indep 0 type datacenter step emit } " To this: " rule cephfs.hdd.data { id 7 type erasure step set_chooseleaf_tries 5 step set_choose_tries 100 step take default class hdd step choose indep 0 type datacenter step chooseleaf indep 3 type host step emit } " Based on some testing and dialogue I had with Red Hat support last year when we were on RHCS, and it seemed to work. Then: ceph fs add_data_pool cephfs cephfs.hdd.data ceph fs subvolumegroup create hdd --pool_layout cephfs.hdd.data I started copying data to the subvolume and increased pg_num a couple of times: ceph osd pool set cephfs.hdd.data pg_num 256 ceph osd pool set cephfs.hdd.data pg_num 2048 But at some point it failed to activate new PGs eventually leading to this: " [WARN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs mds.cephfs.ceph-flash1.agdajf(mds.0): 64 slow metadata IOs are blocked > 30 secs, oldest blocked for 25455 secs [WARN] MDS_TRIM: 1 MDSs behind on trimming mds.cephfs.ceph-flash1.agdajf(mds.0): Behind on trimming (997/128) max_segments: 128, num_segments: 997 [WARN] PG_AVAILABILITY: Reduced data availability: 5 pgs inactive pg 33.6f6 is stuck inactive for 8h, current state activating+remapped, last acting [50,79,116,299,98,219,164,124,421] pg 33.6fa is stuck inactive for 11h, current state activating+undersized+degraded+remapped, last acting [17,408,NONE,196,223,290,73,39,11] pg 33.705 is stuck inactive for 11h, current state activating+undersized+degraded+remapped, last acting [33,273,71,NONE,411,96,28,7,161] pg 33.721 is stuck inactive for 7h, current state activating+remapped, last acting [283,150,209,423,103,325,118,142,87] pg 33.726 is stuck inactive for 11h, current state activating+undersized+degraded+remapped, last acting [234,NONE,416,121,54,141,277,265,19] [WARN] PG_DEGRADED: Degraded data redundancy: 1818/1282640036 objects degraded (0.000%), 3 pgs degraded, 3 pgs undersized pg 33.6fa is stuck undersized for 7h, current state activating+undersized+degraded+remapped, last acting [17,408,NONE,196,223,290,73,39,11] pg 33.705 is stuck undersized for 7h, current state activating+undersized+degraded+remapped, last acting [33,273,71,NONE,411,96,28,7,161] pg 33.726 is stuck undersized for 7h, current state activating+undersized+degraded+remapped, last acting [234,NONE,416,121,54,141,277,265,19] [WARN] POOL_TOO_FEW_PGS: 1 pools have too few placement groups Pool cephfs.hdd.data has 1024 placement groups, should have 2048 [WARN] SLOW_OPS: 3925 slow ops, oldest one blocked for 26012 sec, daemons [osd.17,osd.234,osd.283,osd.33,osd.50] have slow ops. " We had thought it would only affect that particular data pool but eventually everything ground to a halt, also RBD on unrelated replicated pools. Looking at it now I guess that was because the 5 OSDs were blocked for everything and not just the PGs for that data pool? We tried restarting the 5 blocked OSDs to no avail and eventually resorted to deleting the cephfs.hdd.data data pool to restore service. Any suggestions as to what we did wrong? Something to do with min_size? The crush rule? Thanks. Mvh. Torkil -- Torkil Svensgaard Systems Administrator Danish Research Centre for Magnetic Resonance DRCMR, Section 714 Copenhagen University Hospital Amager and Hvidovre Kettegaard Allé 30, 2650 Hvidovre, Denmark

4 5

Ceph Nautilous 14.2.22 slow OSD memory leak?
by huxiaoyu＠horebdata.cn 13 Jan '24

13 Jan '24

Dear Ceph folks, I am responsible for two Ceph clusters, running Nautilius 14.2.22 version, one with replication 3, and the other with EC 4+2. After around 400 days runing quietly and smoothly, recently the two clusters occured with similar problems: some of OSDs consume ca 18 GB while the memory target is setting at 2GB. What could wrong in the background? Does it mean any slow OSD memory leak issues with 14.2.22 which i do not know yet? I would be highly appreciated if some some provides any clues, ideas, comments ...... best regards, Samuel huxiaoyu(a)horebdata.cn

5 6

About ceph disk slowops effect to cluster
by Phong Tran Thanh 13 Jan '24

13 Jan '24

Hi community I'm currently facing a significant issue with my Ceph cluster. I have a cluster consisting of 10 nodes, and each node is equipped with 6 SSDs of 960GB used for block.db and 18 12TB drives used for data, network bonding 2x10Gbps for public and local networks. I am using a 4+2 erasure code for RBD in my Ceph cluster. When one node becomes unavailable, the cluster initiates the recovery process, and subsequently, slow operations (slowops) logs appear on the disk, impacting the entire cluster. Afterward, additional nodes are marked as failures. Is this phenomenon possibly due to the performance of SSDs and HDDs? When I check the I/O of the disk using the iostat command, the result shows that disk utilization has reached 80-90% Is using a combination of HDDs and SAS SSDs in Ceph a choice leading to poor performance? My Ceph cluster has a bandwidth of 1.9GB/s Thanks and hope someone can help me ---------------------------------------------------------------------------- Email: tranphong079(a)gmail.com Skype: tranphong079

4 7

RGW rate-limiting or anti-hammering for (external) auth requests // Anti-DoS measures
by Christian Rohmann 12 Jan '24

12 Jan '24

Hey Ceph-Users, RGW does have options [1] to rate limit ops or bandwidth per bucket or user. But those only come into play when the request is authenticated. I'd like to also protect the authentication subsystem from malicious or invalid requests. So in case e.g. some EC2 credentials are not valid (anymore) and clients start hammering the RGW with those requests, I'd like to make it cheap to deal with those requests. Especially in case some external authentication like OpenStack Keystone [2] is used, valid access tokens are cached within the RGW. But requests with invalid credentials end up being sent at full rate to the external API [3] as there is no negative caching. And even if there was, that would only limit the external auth requests for the same set of invalid credentials, but it would surely reduce the load in that case: Since the HTTP request is blocking .... > [...] > 2023-12-18T15:25:55.861+0000 7fec91dbb640 20 sending request to > https://keystone.example.com/v3/s3tokens > 2023-12-18T15:25:55.861+0000 7fec91dbb640 20 register_request > mgr=0x561a407ae0c0 req_data->id=778, curl_handle=0x7fedaccb36e0 > 2023-12-18T15:25:55.861+0000 7fec91dbb640 20 WARNING: blocking http > request > 2023-12-18T15:25:55.861+0000 7fede37fe640 20 link_request > req_data=0x561a40a418b0 req_data->id=778, curl_handle=0x7fedaccb36e0 > [...] this does not only stress the external authentication API (keystone in this case), but also blocks RGW threads for the duration of the external call. I am currently looking into using the capabilities of HAProxy to rate limit requests based on their resulting http-response [4]. So in essence to rate-limit or tarpit clients that "produce" a high number of 403 "InvalidAccessKeyId" responses. To have less collateral it might make sense to limit based on the presented credentials themselves. But this would require to extract and track HTTP headers or URL parameters (presigned URLs) [5] and to put them into tables. * What are your thoughts on the matter? * What kind of measures did you put in place? * Does it make sense to extend RGWs capabilities to deal with those cases itself? ** adding negative caching ** rate limits on concurrent external authentication requests (or is there a pool of connections for those requests?) Regards Christian [1] https://docs.ceph.com/en/latest/radosgw/admin/#rate-limit-management [2] https://docs.ceph.com/en/latest/radosgw/keystone/#integrating-with-openstac… [3] https://github.com/ceph/ceph/blob/86bb77eb9633bfd002e73b5e58b863bc2d0df594/… [4] https://www.haproxy.com/documentation/haproxy-configuration-manual/latest/#… [5] https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-reque…

2 3

Is there any way to merge an rbd image's full backup and a diff?
by Satoru Takeuchi 12 Jan '24

12 Jan '24

Hi, I'm developing RBD images' backup system. In my case, a backup data must be stored at least two weeks. To meet this requirement, I'd like to take backups as follows: 1. Take a full backup by rbd export first. 2. Take a differencial backups everyday. 3. Merge the full backup and the oldest (taken two weeks ago) diff. As a result of evaluation, I confirmed there is no problem in step 1 and 2. However, I found that step 3 couldn't be accomplished by `rbd merge-diff <full backup> <diff>` because `rbd merge-diff` only accepts a diff as a first parameter. Is there any way to merge a full backup and a diff? Thanks, Satoru

2 5

Pacific bluestore_volume_selection_policy
by Reed Dier 11 Jan '24

11 Jan '24

After ~3 uneventful weeks after upgrading from 15.2.17 to 16.2.14 I’ve started seeing OSD crashes with "cur >= fnode.size” and "cur >= p.length”, which seems to be resolved in the next point release for pacific later this month, but until then, I’d love to keep the OSDs from flapping. > $ for crash in $(ceph crash ls | grep osd | awk '{print $1}') ; do ceph crash info $crash | egrep "(assert_condition|crash_id)" ; done > "assert_condition": "cur >= fnode.size", > "crash_id": "2024-01-03T09:07:55.698213Z_348af2d3-d4a7-4c27-9f71-70e6dc7c1af7", > "assert_condition": "cur >= p.length", > "crash_id": "2024-01-03T14:21:55.794692Z_4557c416-ffca-4165-aa91-d63698d41454", > "assert_condition": "cur >= fnode.size", > "crash_id": "2024-01-03T22:53:43.010010Z_15dc2b2a-30fb-4355-84b9-2f9560f08ea7", > "assert_condition": "cur >= p.length", > "crash_id": "2024-01-04T02:34:34.408976Z_2954a2c2-25d2-478e-92ad-d79c42d3ba43", > "assert_condition": "cur2 >= p.length", > "crash_id": "2024-01-04T21:57:07.100877Z_12f89c2c-4209-4f5a-b243-f0445ba629d2", > "assert_condition": "cur >= p.length", > "crash_id": "2024-01-05T00:35:08.561753Z_a189d967-ab02-4c61-bf68-1229222fd259", > "assert_condition": "cur >= fnode.size", > "crash_id": "2024-01-05T04:11:48.625086Z_a598cbaf-2c4f-4824-9939-1271eeba13ea", > "assert_condition": "cur >= p.length", > "crash_id": "2024-01-05T13:49:34.911210Z_953e38b9-8ae4-4cfe-8f22-d4b7cdf65cea", > "assert_condition": "cur >= p.length", > "crash_id": "2024-01-05T13:54:25.732770Z_4924b1c0-309c-4471-8c5d-c3aaea49166c", > "assert_condition": "cur >= p.length", > "crash_id": "2024-01-05T16:35:16.485416Z_0bca3d2a-2451-4275-a049-a65c58c1aff1”, As noted in https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/YNJ35HXN4H… <https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/YNJ35HXN4H…> > You can apparently work around the issue by setting > 'bluestore_volume_selection_policy' config parameter to rocksdb_original. However, after trying to set that parameter with `ceph config set osd.$osd bluestore_volume_selection_policy rocksdb_original` it doesn’t appear to set? > $ ceph config show-with-defaults osd.0 | grep bluestore_volume_selection_policy > bluestore_volume_selection_policy use_some_extra > $ ceph config set osd.0 bluestore_volume_selection_policy rocksdb_original > $ ceph config show osd.0 | grep bluestore_volume_selection_policy > bluestore_volume_selection_policy use_some_extra default mom This, I assume, should reflect the new setting, however it still shows the default “use_some_extra” value. But then this seems to imply that the config is set? > $ ceph config dump | grep bluestore_volume_selection_policy > osd.0 dev bluestore_volume_selection_policy rocksdb_original * > [snip] > osd.9 dev bluestore_volume_selection_policy rocksdb_original * Does this need to be set in ceph.conf or is there another setting that also needs to be set? Even after bouncing the OSD daemon, `ceph config show` still reports “use_some_extra" Appreciate any help they can offer to point me towards to bridge the gap between now and the next point release. Thanks, Reed

3 6

Unable to execute radosgw command using cephx users on client side
by Alam Mohammad 11 Jan '24

11 Jan '24

Hello, In our Ceph cluster we encountered issues while attempting to execute "radosgw-admin" command on client side using cephx user having read only permission. Whenever we are executing "radosgw-admin user list" command it is throwing an error. "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)" We have performed below steps in our environment Case-1 : First we created cephx user with below privileges client.rgw.username key: <-------key-------> caps: [mgr] allow r caps: [mon] allow r caps: [osd] allow r tag rgw *=* on client side we copied keyring and ceph.conf file What we noticed on client machine all general command like "ceph -s", "ceph health detail" "ceph df" running fine, even "radosgw-admin zonegroup list --id=rgw.username," command returned the expected output, but when attempting commands like "radosgw-admin user list," "radosgw-admin bucket list," or "radosgw-admin user info," errors were encountered. Below are the outputs that is throwing root@control:~# radosgw-admin user list --id=rgw.username 2024-01-03T17:34:06.498+0000 7f915ece1fc0 0 ERROR: failed reading data (obj=default.rgw.log:bucket.sync-source-hints.), r=-1 2024-01-03T17:34:06.498+0000 7f915ece1fc0 0 ERROR: failed to update sources index for bucket=:[]) r=-1 2024-01-03T17:34:06.498+0000 7f915ece1fc0 0 ERROR: failed to initialize bucket sync policy handler: get_bucket_sync_hints() on bucket=-- returned r=-1 2024-01-03T17:34:06.498+0000 7f915ece1fc0 -1 ERROR: could not initialize zone policy handler for zone=default 2024-01-03T17:34:06.498+0000 7f915ece1fc0 0 ERROR: failed to start notify service ((1) Operation not permitted 2024-01-03T17:34:06.498+0000 7f915ece1fc0 0 ERROR: failed to init services (ret=(1) Operation not permitted) couldn't init storage provider Case- 2 : In this case we granted read permissions to the rgw data pool and index pool for the user, client.rgw.username key: <----key----> caps: [mgr] allow r caps: [mon] allow r caps: [osd] allow r pool=default.rgw.log Despite this, while general commands worked perfectly fine on the client side, but "radosgw-admin" commands still failed to execute. Here is the output root@control:~# radosgw-admin user list --id=rgw.username 2024-01-03T17:43:38.071+0000 7f8b5a8bffc0 0 failed reading realm info: ret -1 (1) Operation not permitted 2024-01-03T17:43:38.071+0000 7f8b5a8bffc0 0 ERROR: failed to start notify service ((1) Operation not permitted 2024-01-03T17:43:38.071+0000 7f8b5a8bffc0 0 ERROR: failed to init services (ret=(1) Operation not permitted) couldn't init storage provider Have I overlooked anything in the process? Any guidance or insight would be greatly appreciated. Thanks, Mohammad Saif Ceph Enthusiast In the first step, we created a CephX user named client.rgw.saif with read permissions for the manager (mgr), monitor (mon), and object storage daemon (osd) components, along with specific RGW capabilities. On the client side, we successfully copied the keyring and ceph.conf, and certain commands, such as radosgw-admin zonegroup list --id=rgw.username,

3 5