April 2024 - ceph-users - lists.ceph.io

could not find secret_id
by xiaowenhao111 22 Apr '26

22 Apr '26

2 1

OpenSSL security update for Octopus container?
by Robert Sander 02 Oct '25

02 Oct '25

Hi, I am running a Ceph Octopus cluster in version 15.2.10 setup with the cephadm orchestrator and official Docker hub container images. Yesterday an important security fix was released for libssl and is already packaged for all major distributions. I tried to run "ceph orch upgrade check docker.io/ceph/ceph:v15" but it tells me that the containers do not need to be upgraded. How will this security fix of OpenSSL be deployed in a timely manner to users of the Ceph container images? Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 93818 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 1

Migrating from S3 to Ceph RGW (Cloud Sync Module)
by James McClune 10 Apr '25

10 Apr '25

Hello Ceph User Community, I currently have a large Amazon S3 environment with terabytes of data spread over dozens of buckets. I'm looking to migrate from Amazon S3 to an on-site Ceph cluster using the RGW. I'm trying to figure out the most efficient way to achieve this. Looking through the documentation, I found articles related to the cloud sync module, released in Mimic ( https://docs.ceph.com/en/latest/radosgw/cloud-sync-module/) I also watched a video on the cloud sync module as well. It *sounds* like this is the functionality I'm looking for. Given I'm moving away from Amazon S3, I'm really just looking for a one-way replication between the buckets (i.e. Provide an Amazon S3 access key/secret which is associated to the buckets and the same for the Ceph environment, so object data can be replicated one-to-one, without creating ad-hoc tooling). Once the data is replicated from S3 to Ceph, I plan on modifying my boto connection objects to use the new Ceph environment. Is what I'm describing feasible with the cloud sync module? Just looking for some affirmation, given I'm not well versed in Ceph's RGW, especially around multi-site configurations. Thanks, Jimmy

7 8

Problem while upgrade 17.2.6 to 17.2.7
by Jean-Marc FONTANA 21 Feb '25

21 Feb '25

Hello everyone, We operate two clusters that we installed with ceph-deploy in Nautilus version on Debian 10. We use them for external S3 storage (owncloud) and rbd disk images.We had them upgraded to Octopus and Pacific versions on Debian 11 and recently converted them to cephadm and upgraded to Quincy (17.2.6). As we now have the orchestrator, we tried updating to 17.2.7 using the command# ceph orch upgrade start --image quay.io/ceph/ceph:v17.2.7 Everything went well, both clusters work perfectly for our use, except that the Rados gateway configuration is no longer accessible from the dashboard with the following error messageError connecting to Object Gateway: RGW REST API failed request with status code 404. We tried a few solutions found on the internet (reset rgw credentials, restart rgw adnd mgr, reenable dashboard, ...), unsuccessfully. Does somebody have an idea ? Best regards, Jean-Marc Fontana

5 9

purging already destroyed OSD leads to degraded and misplaced objects?
by tobias tempel 10 Jan '25

10 Jan '25

Dear Cephers, reorganizing one of our clusters i'm removing some hosts from it, taking "out" all OSDs on these hosts and waiting until all PGs are fine. After stopping and destroying all OSDs on one host i notice, that "purge" of such destroyed OSDs temporarily leads to degraded and misplaced objects (tried "reweight 0" as well, same picture) . Why that? I would - due to my limited experience - expect, that completely removing an already destroyed OSD could not have any effect on object placement at all. This effect yet seems not to be a real problem, as recovery works fine ... but perhaps it indicates some other issue. All of that takes place at Pacific 16.2.14 AlmaLinux 8.9, yes i know, there's some work to do. Perhaps you can give me a hint, where to look, for an explanation? Thank you cheers, toBias -- Tobias Tempel Deutsches Elektronen-Synchrotron - IT Notkestr. 85, 22607 Hamburg, Germany email: tobias.tempel(a)desy.de

4 7

Problems with autoscaler (overlapping roots) after changing the pool class
by Massimo Sgaravatto 06 Jan '25

06 Jan '25

Dear all I have just changed the crush rule for all the replicated pools in the following way: ceph osd crush rule create-replicated replicated_hdd default host hdd ceph osd pool set <poolname> crush_rule replicated_hdd See also this [*] thread Before applying this change, these pools were all using the replicated_ruleset rule where the class is not specified. I am noticing now a problem with the autoscaler: "ceph osd pool autoscale-status" doesn't report any output and the mgr log complains about overlapping roots: [pg_autoscaler ERROR root] pool xyz has overlapping roots: {-18, -1} Indeed: # ceph osd crush tree --show-shadow ID CLASS WEIGHT TYPE NAME -18 hdd 1329.26501 root default~hdd -17 hdd 329.14154 rack Rack11-PianoAlto~hdd -15 hdd 54.56085 host ceph-osd-04~hdd 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... ... -1 1329.26501 root default -7 329.14154 rack Rack11-PianoAlto -8 54.56085 host ceph-osd-04 30 hdd 5.45609 osd.30 31 hdd 5.45609 osd.31 ... I have already read about this behavior but I have no clear ideas how to fix the problem. I read somewhere that the problem happens when there are rules that force some pools to only use one class and there are also pools which does not make any distinction between device classes All the replicated pools are using the replicated_hdd pool but I also have some EC pools which are using a profile where the class is not specified. As far I understand, I can't force these pools to use only the hdd class: according to the doc I can't change this profile specifying the hdd class (or at least the change wouldn't be applied to the existing EC pools) Any suggestions ? The crush map is available at https://cernbox.cern.ch/s/gIyjbQbmoTFHCrr, if you want to have a look Many thanks, Massimo [*] https://www.mail-archive.com/ceph-users@ceph.io/msg18534.html

4 9

multifs and snapshots
by Dmitry Melekhov 12 Nov '24

12 Nov '24

Hello! There is note here https://docs.ceph.com/en/reef/dev/cephfs-snapshots/ about multiply filesystems and snapshots: If each FS gets its own pool things probably work, but this isn’t tested and may not be true. Is it still untested ? Thank you!

3 4

Help needed to configure erasure coding LRC plugin
by Michel Jouvin 16 Oct '24

16 Oct '24

Hi, As discussed in another thread (Crushmap rule for multi-datacenter erasure coding), I'm trying to create an EC pool spanning 3 datacenters (datacenters are present in the crushmap), with the objective to be resilient to 1 DC down, at least keeping the readonly access to the pool and if possible the read-write access, and have a storage efficiency better than 3 replica (let say a storage overhead <= 2). In the discussion, somebody mentioned LRC plugin as a possible jerasure alternative to implement this without tweaking the crushmap rule to implement the 2-step OSD allocation. I looked at the documentation (https://docs.ceph.com/en/latest/rados/operations/erasure-code-lrc/) but I have some questions if someone has experience/expertise with this LRC plugin. I tried to create a rule for using 5 OSDs per datacenter (15 in total), with 3 (9 in total) being data chunks and others being coding chunks. For this, based of my understanding of examples, I used k=9, m=3, l=4. Is it right? Is this configuration equivalent, in terms of redundancy, to a jerasure configuration with k=9, m=6? The resulting rule, which looks correct to me, is: -------- { "rule_id": 6, "rule_name": "test_lrc_2", "ruleset": 6, "type": 3, "min_size": 3, "max_size": 15, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -4, "item_name": "default~hdd" }, { "op": "choose_indep", "num": 3, "type": "datacenter" }, { "op": "chooseleaf_indep", "num": 5, "type": "host" }, { "op": "emit" } ] } ------------ Unfortunately, it doesn't work as expected: a pool created with this rule ends up with its pages active+undersize, which is unexpected for me. Looking at 'ceph health detail` output, I see for each page something like: pg 52.14 is stuck undersized for 27m, current state active+undersized, last acting [90,113,2147483647,103,64,147,164,177,2147483647,133,58,28,8,32,2147483647] For each PG, there is 3 '2147483647' entries and I guess it is the reason of the problem. What are these entries about? Clearly it is not OSD entries... Looks like a negative number, -1, which in terms of crushmap ID is the crushmap root (named "default" in our configuration). Any trivial mistake I would have made? Thanks in advance for any help or for sharing any successful configuration? Best regards, Michel

4 24

rgw connection resets
by Nathan Gleason 10 Oct '24

10 Oct '24

Hello, I’ve noticed what I think may be strange behavior from some rgw transfers. It only seems to be with larger multipart mp4 files. Once a connection comes into rgw it is accepted and processed, data begins to transfer, then I see "write_data failed: Connection reset by peer” and "s3:get_obj iterate_obj() failed with -104”. The client then reconnects and the transfer seems to be ok. This doesn’t appear to happen just downloading a regular file. This is happening to both my production and test cluster. I’m not seeing errors anywhere else, the network looks good and the clusters are healthy. I’m in the process of collecting some tcpdumps but figured I would ask if anyone knows why this is occurring. We run haproxy in front of the rgw. For troubleshooting purposes we have a basic haproxy config to just one rgw. We have tested with multiple browsers. Ceph 17.2.5 - cluster built and managed with cephadm Ubuntu 20.04.6 LTS Log: 2024-01-03T17:40:23.916+0000 7f646033d700 1 ====== starting new request req=0x7f640ca95730 ===== 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s initializing for trans_id = tx000003ec1e4b4dc157efd-0065959c07-104216-default 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s rgw api priority: s3=8 s3website=7 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s host=a2-west.usacld.net 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s final domain/bucket subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 s->info.domain= s->info.request_uri=/n athan:tshoot/testfile.mp4 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s get_handler handler=22RGWHandler_REST_Obj_S3 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s handler=22RGWHandler_REST_Obj_S3 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s getting op 0 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s get_system_obj_state: rctx=0x7f640ca947c0 obj=default.rgw.log:script.prerequest. state=0x55fa31f882e0 s->prefetch_data=0 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s cache get: name=default.rgw.log++script.prerequest. : hit (negative entry) 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj scheduling with throttler client=2 cost=1 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj op=21RGWGetObj_ObjStore_S3 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj verifying requester 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rgw::auth::StrategyRegistry::s3_main_strategy_t: trying rgw::auth::s3::AWSAuthStrategy 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::S3AnonymousEngine 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rgw::auth::s3::S3AnonymousEngine granted access 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rgw::auth::s3::AWSAuthStrategy granted access 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj normalizing buckets and tenants 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s->object=testfile.mp4 s->bucket=nathan/tshoot 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj init permissions 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_system_obj_state: rctx=0x7f640ca94200 obj=default.rgw.meta:root:nathan/tshoot state=0x55fa31f882e0 s->prefetch _data=0 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj cache get: name=default.rgw.meta+root+nathan/tshoot : hit (requested=0x16, cached=0x17) 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_system_obj_state: s->obj_tag was set empty 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj cache get: name=default.rgw.meta+root+nathan/tshoot : hit (requested=0x11, cached=0x17) 2024-01-03T17:40:23.916+0000 7f646033d700 15 req 4522146966294331133 0.000000000s s3:get_obj decode_policy Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner> <ID>nathan$testuser</ID><DisplayName>Nathan API User</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>nathan$testuser </ID><DisplayName>Nathan API User</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy> 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWSI_User_RADOS::read_user_info(): anonymous user 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj recalculating target 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj reading permissions 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_obj_state: rctx=0x7f640ca94ad0 obj=tshoot:testfile.mp4 state=0x55fa31082de8 s->prefetch_data=1 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj WARNING: blocking librados call 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj manifest: total_size = 88811314 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_obj_state: setting s->obj_tag to _0OUNxgsSsqTAs9Z6xh1SOKpJZLV_Vos 2024-01-03T17:40:23.916+0000 7f646033d700 15 req 4522146966294331133 0.000000000s s3:get_obj decode_policy Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner> <ID>nathan$testuser</ID><DisplayName>Nathan API User</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="Group"><URI>http://acs.amazonaws.c om/groups/global/AllUsers</URI></Grantee><Permission>READ</Permission></Grant><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>nathan$testuser</ID><DisplayNa me>Nathan API User</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy> 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj init op 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_system_obj_state: rctx=0x7f640ca94488 obj=default.rgw.meta:users.uid:nathan$testuser state=0x55fa31f882e0 s->p refetch_data=0 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj cache get: name=default.rgw.meta+users.uid+nathan$testuser : hit (requested=0x16, cached=0x17) 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_system_obj_state: s->obj_tag was set empty 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr: user.rgw.idtag 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj cache get: name=default.rgw.meta+users.uid+nathan$testuser : hit (requested=0x13, cached=0x17) 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj verifying op mask 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj required_mask= 1 user.op_mask=7 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj verifying op permissions 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj -- Getting permissions begin with perm_mask=49 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj Searching permissions for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=anonymous, acct_n ame=, subuser=, perm_mask=15, is_admin=0) mask=49 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj Searching permissions for uid=anonymous 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj Permissions for user not found 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj Searching permissions for group=1 mask=49 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj Found permission: 1 2024-01-03T17:40:23.916+0000 7f646033d700 5 req 4522146966294331133 0.000000000s s3:get_obj -- Getting permissions done for identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=anonymous, acct_name=, subuser=, perm_mask=15, is_admin=0), owner=nathan$testuser, perm=1 2024-01-03T17:40:23.916+0000 7f646033d700 10 req 4522146966294331133 0.000000000s s3:get_obj identity=rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=anonymous, acct_name=, subuser=, perm_mask =15, is_admin=0) requested perm (type)=1, policy perm=1, user_perm_mask=15, acl perm=1 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj verifying op params 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj pre-executing 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj check rate limiting 2024-01-03T17:40:23.916+0000 7f646033d700 2 req 4522146966294331133 0.000000000s s3:get_obj executing 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_obj_state: rctx=0x7f640ca94ad0 obj=tshoot:testfile.mp4 state=0x55fa31082de8 s->prefetch_data=1 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.acl 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.content_type 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.etag 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.idtag 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.manifest 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.pg_ver 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.source_zone 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.tail_tag 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.x-amz-content-sha256 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.x-amz-date 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj Read xattr rgw_rados: user.rgw.x-amz-meta-s3cmd-attrs 2024-01-03T17:40:23.916+0000 7f646033d700 15 req 4522146966294331133 0.000000000s Encryption mode: 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj get_obj_state: rctx=0x7f640ca94ad0 obj=tshoot:testfile.mp4 state=0x55fa31082de8 s->prefetch_data=1 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__multipart_testfile.mp4.2~XSiTvBBtaU4 134dQCusv3kMP7YjDBk7.1 obj-ofs=0 read_ofs=0 len=4194304 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=4194304 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): result: ofs=4194304 stripe_ofs=4194304 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134 dQCusv3kMP7YjDBk7.1_1 obj-ofs=4194304 read_ofs=0 len=4194304 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=8388608 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): result: ofs=8388608 stripe_ofs=8388608 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134 dQCusv3kMP7YjDBk7.1_2 obj-ofs=8388608 read_ofs=0 len=4194304 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=12582912 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): result: ofs=12582912 stripe_ofs=12582912 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134 dQCusv3kMP7YjDBk7.1_3 obj-ofs=12582912 read_ofs=0 len=3145728 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=16777216 part_ofs=0 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj RGWObjManifest::operator++(): result: ofs=15728640 stripe_ofs=15728640 part_ofs=15728640 rule->part_size=15728640 2024-01-03T17:40:23.916+0000 7f646033d700 20 req 4522146966294331133 0.000000000s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__multipart_testfile.mp4.2~XSiTvBBtaU4 134dQCusv3kMP7YjDBk7.2 obj-ofs=15728640 read_ofs=0 len=4194304 2024-01-03T17:40:23.924+0000 7f6442301700 20 req 4522146966294331133 0.007999974s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.924+0000 7f6442301700 20 req 4522146966294331133 0.007999974s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=19922944 part_ofs=15728640 rule->part_size=15728640 2024-01-03T17:40:23.924+0000 7f6442301700 20 req 4522146966294331133 0.007999974s s3:get_obj RGWObjManifest::operator++(): result: ofs=19922944 stripe_ofs=19922944 part_ofs=15728640 rule->part_size=15728640 2024-01-03T17:40:23.924+0000 7f6442301700 20 req 4522146966294331133 0.007999974s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134 dQCusv3kMP7YjDBk7.2_1 obj-ofs=19922944 read_ofs=0 len=4194304 2024-01-03T17:40:23.932+0000 7f643d2f7700 20 req 4522146966294331133 0.015999949s s3:get_obj RGWObjManifest::operator++(): rule->part_size=15728640 rules.size()=2 2024-01-03T17:40:23.932+0000 7f643d2f7700 20 req 4522146966294331133 0.015999949s s3:get_obj RGWObjManifest::operator++(): stripe_ofs=24117248 part_ofs=15728640 rule->part_size=15728640 2024-01-03T17:40:23.932+0000 7f643d2f7700 20 req 4522146966294331133 0.015999949s s3:get_obj RGWObjManifest::operator++(): result: ofs=24117248 stripe_ofs=24117248 part_ofs=15728640 rule->part_size=15728640 2024-01-03T17:40:23.932+0000 7f643d2f7700 20 req 4522146966294331133 0.015999949s s3:get_obj rados->get_obj_iterate_cb oid=4e125c16-5355-4d42-a6ba-4d388e8f7bd2.1011497.1__shadow_testfile.mp4.2~XSiTvBBtaU4134 dQCusv3kMP7YjDBk7.2_2 obj-ofs=24117248 read_ofs=0 len=4194304 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_status: e=0, sent=17, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_content_length: e=0, sent=26, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::send_header: e=0, sent=0, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::complete_header: e=0, sent=460, total=0 2024-01-03T17:40:23.932+0000 7f646a351700 30 AccountingFilter::set_account: e=1 2024-01-03T17:40:24.068+0000 7f642a2d1700 4 write_data failed: Connection reset by peer 2024-01-03T17:40:24.068+0000 7f642a2d1700 0 req 4522146966294331133 0.151999518s s3:get_obj iterate_obj() failed with -104 2024-01-03T17:40:24.156+0000 7f642eada700 2 req 4522146966294331133 0.239999235s s3:get_obj completing 2024-01-03T17:40:24.156+0000 7f642eada700 20 req 4522146966294331133 0.239999235s get_system_obj_state: rctx=0x7f640ca947c0 obj=default.rgw.log:script.postrequest.nathan state=0x55fa32424760 s->prefetch_data =0 2024-01-03T17:40:24.156+0000 7f642eada700 10 req 4522146966294331133 0.239999235s cache get: name=default.rgw.log++script.postrequest.nathan : hit (negative entry) 2024-01-03T17:40:24.156+0000 7f642eada700 30 AccountingFilter::complete_request: e=1, sent=0, total=0 2024-01-03T17:40:24.156+0000 7f642eada700 2 req 4522146966294331133 0.239999235s s3:get_obj op status=-104 2024-01-03T17:40:24.156+0000 7f642eada700 2 req 4522146966294331133 0.239999235s s3:get_obj http status=200 2024-01-03T17:40:24.156+0000 7f642eada700 1 ====== req done req=0x7f640ca95730 op status=-104 http_status=200 latency=0.239999235s ====== 2024-01-03T17:40:24.156+0000 7f642eada700 1 beast: 0x7f640ca95730: 10.30.0.196 - anonymous [03/Jan/2024:17:40:23.916 +0000] "GET /nathan:tshoot/testfile.mp4 HTTP/1.1" 200 0 - "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" - latency=0.239999235s 2024-01-03T17:40:24.156+0000 7f642eada700 20 failed to read header: end of stream 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_ACCEPT=*/* 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_ACCEPT_ENCODING=identity;q=1, *;q=0 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_ACCEPT_LANGUAGE=en-US,en;q=0.7 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_HOST=a2-west.usacld.net 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_RANGE=bytes=0- 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_REFERER=https://a2-west.usacld.net/nathan:tshoot/testfile.mp4 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_CH_UA="Not_A Brand";v="8", "Chromium";v="120", "Brave";v="120" 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_CH_UA_MOBILE=?0 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_CH_UA_PLATFORM="macOS" 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_FETCH_DEST=video 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_FETCH_MODE=no-cors 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_FETCH_SITE=same-origin 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_SEC_GPC=1 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_VERSION=1.1 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_X_FORWARDED_PORT=443 2024-01-03T17:40:24.160+0000 7f6503483700 20 HTTP_X_FORWARDED_PROTO=https 2024-01-03T17:40:24.160+0000 7f6503483700 20 REMOTE_ADDR=10.30.0.196 2024-01-03T17:40:24.160+0000 7f6503483700 20 REQUEST_METHOD=GET 2024-01-03T17:40:24.160+0000 7f6503483700 20 REQUEST_URI=/nathan:tshoot/testfile.mp4 2024-01-03T17:40:24.160+0000 7f6503483700 20 SCRIPT_URI=/nathan:tshoot/testfile.mp4 2024-01-03T17:40:24.160+0000 7f6503483700 20 SERVER_PORT=8000 2024-01-03T17:40:24.160+0000 7f6503483700 1 ====== starting new request req=0x7f640ca95730 ===== Thanks, Nathan

2 1

Not all Bucket Shards being used
by Christian Kugler 01 Oct '24

01 Oct '24

Hi, I have trouble with large OMAP files in a cluster in the RGW index pool. Some background information about the cluster: There is CephFS and RBD usage on the main cluster but for this issue I think only S3 is interesting. There is one realm, one zonegroup with two zones which have a bidirectional sync set up. Since this does not allow for autoresharding we have to do it by hand in this cluster – looking forward to Reef! From the logs: cluster 2023-07-17T22:59:03.018722+0000 osd.75 (osd.75) 623978 : cluster [WRN] Large omap object found. Object: 34:bcec3016:::.dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.5:head PG: 34.680c373d (34.5) Key count: 962091 Size (bytes): 277963182 The offending bucket looks like this: # radosgw-admin bucket stats \ | jq '.[] | select(.marker =="3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9") |"\(.num_shards) \(.usage["rgw.main"].num_objects)"' -r 131 9463833 Last week the number of objects was about 12 million. Which is why I reshareded the offending bucket twice, I think. Once to 129 and the second time to 131 because I wanted some leeway (or lieway? scnr, Sage). Unfortunately, even after a week the objects were still to big (the log line above is quite recent), so I looked into it again. # rados -p raum.rgw.buckets.index ls \ |grep .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9 \ |sort -V .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.0 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.1 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.2 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.3 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.4 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.5 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.6 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.7 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.8 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.9 .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9.10 # rados -p raum.rgw.buckets.index ls \ |grep .dir.3caabb9a-4e3b-4b8a-8222-34c33dd63210.10610190.9 \ |sort -V \ |xargs -IOMAP sh -c \ 'rados -p raum.rgw.buckets.index listomapkeys OMAP | wc -l' 1013854 1011007 1012287 1011232 1013565 998262 1012777 1012713 1012230 1010690 997111 Apparently, only 11 shards are in use. This would explain why the "Key usage" (from the log line) is about ten times higher than I would expect. How can I deal with this issue? One thing I could try to fix this would be to reshard to a lower number, but I am not sure if there are any risks associated with "downsharding". After that I could reshard to something like 97. Or I could directly "downshard" to 97. Also, the second zone has a similar problem, but as the error messsage lets me know, this would be a bad idea. Will it just take more time until the sharding is transferred to the seconds zone? Best, Christian Kugler

2 5

Successfully using dm-cache
by Michael Lipp 12 Sep '24

12 Sep '24

Just in case anybody is interested: Using dm-cache works and boosts performance -- at least for my use case. The "challenge" was to get 100 (identical) Linux-VMs started on a three node hyperconverged cluster. The hardware is nothing special, each node has a Supermicro server board with a single CPU with 24 cores and 4 x 4 TB hard disks. And there's that extra 1 TB NVMe... I know that the general recommendation is to use the NVMe for WAL and metadata, but this didn't seem appropriate for my use case and I'm still not quite sure about failure scenarios with this configuration. So instead I made each drive a logical volume (managed by an OSD) and added 85 GiB NVMe to each LV as read-only cache. Each VM uses as system disk an RBD based on a snapshot from the master image. The idea was that with this configuration, all VMs should share most (actually almost all) of the data on their system disk and this data should be available from the cache. Well, it works. When booting the 100 VMs, almost all read operations are satisfied from the cache. So I get close to NVMe speed but have payed for conventional hard drives only (well, SSDs aren't that much more expensive nowadays, but the hardware is 4 years old). So, nothing sophisticated, but as I couldn't find anything about this kind of setup, it might be of interest nevertheless. - Michael

13 27

Public Swift bucket with Openstack Keystone integration - not working in quincy/reef
by Bartosz Bezak 31 Jul '24

31 Jul '24

Hi, Similar case as with previously fixed https://tracker.ceph.com/issues/48382 - https://github.com/ceph/ceph/pull/47308. Confirmed on Cephadm deployed Ceph 18.2.2/17.2.7 with Openstack Antelope/Yoga. I’m getting "404 NoSuchBucket" error with public buckets. Enabled with Swift/Keystone integration - everything else works fine. With rgw_swift_account_in_url = true and proper endpoints: "https://rgw.test/swift/v1/AUTH_%(project_id)s" ticking public access in horizon properly sets ACL on the bucket according to swift client: swift -v stat test-bucket URL: https://rgw.test/swift/v1/AUTH_daksjhdkajdshda/testbucket Auth Token: Account: AUTH_daksjhdkajdshda Container: testbucket Objects: 1 Bytes: 1021036 Read ACL: .r:*,.rlistings Write ACL: Sync To: Sync Key: X-Timestamp: 1710947159.41219 X-Container-Bytes-Used-Actual: 1024000 X-Storage-Policy: default-placement X-Storage-Class: STANDARD Last-Modified: Thu, 21 Mar 2024 10:30:05 GMT X-Trans-Id: tx00000092ac12312312312-1231231231-1701e5-default X-Openstack-Request-Id: tx00000092ac12312312312-1231231231-1701e5-default Accept-Ranges: bytes Content-Type: text/plain; charset=utf-8 however still getting 404 NoSuchBucket error Could someone using the latest version of Ceph with Swift/Keystone integration please test public buckets? Thank you. Best regards, Bartosz Bezak

3 3

reef 18.2.3 QE validation status
by Yuri Weinstein 12 Jul '24

12 Jul '24

Details of this release are summarized here: https://tracker.ceph.com/issues/65393#note-1 Release Notes - TBD LRC upgrade - TBD Seeking approvals/reviews for: smoke - infra issues, still trying, Laura PTL rados - Radek, Laura approved? Travis? Nizamudeen? rgw - Casey approved? fs - Venky approved? orch - Adam King approved? krbd - Ilya approved powercycle - seems fs related, Venky, Brad PTL ceph-volume - will require https://github.com/ceph/ceph/pull/56857/commits/63fe3921638f1fb7fc065907a9e… Guillaume is fixing it. TIA

10 29

cephadm discovery service certificate absent after upgrade.
by Nicolas FOURNIL 09 Jul '24

09 Jul '24

Hello, I've just fresh upgrade from Quincy to Reef and my graphs are now blank... after investigations, it seems that discovery service is not working because of no certificate : # ceph orch sd dump cert Error EINVAL: No certificate found for service discovery Maybe an upgrade issue ? Is there a way to generate or replace the certificate properly ? Regards Nicolas F.

3 8

Fixing BlueFS spillover (pacific 16.2.14)
by Chris Dunlop 09 Jul '24

09 Jul '24

Hi, tl;dr why are my osds still spilling? I've recently upgraded to 16.2.14 from 16.2.9 and started receiving bluefs spillover warnings (due to the "fix spillover alert" per the 16.2.14 release notes). E.g. from 'ceph health detail', the warning on one of these (there are a few): osd.76 spilled over 128 KiB metadata from 'db' device (56 GiB used of 60 GiB) to slow device This is a 15T HDD with only a 60G SSD for the db so it's not surprising it spilled as it's way below the recommendation for rbd usage at db size 1-2% of the storage size. There was some spare space on the db ssd so I increased the size of the db LV up over 400G and did an bluefs-bdev-expand. However, days later, I'm still getting the spillover warning for that osd, including after running a manual compact: # ceph tell osd.76 compact See attached perf-dump-76 for the perf dump output: # cephadm enter --name 'osd.76' ceph daemon 'osd.76' perf dump" | jq -r '.bluefs' In particular, if my understanding is correct, that's telling me the db available size is 487G (i.e. the LV expand worked), of which it's using 59G, and there's 128K spilled to the slow device: "db_total_bytes": 512309059584, # 487G "db_used_bytes": 63470305280, # 59G "slow_used_bytes": 131072, # 128K A "bluefs stats" also says the db is using 128K of slow storage (although perhaps it's getting the info from the same place as the perf dump?): # ceph tell osd.76 bluefs stats 1 : device size 0x7747ffe000 : using 0xea6200000(59 GiB) 2 : device size 0xe8d7fc00000 : using 0x6554d689000(6.3 TiB) RocksDBBlueFSVolumeSelector Usage Matrix: DEV/LEV WAL DB SLOW * * REAL FILES LOG 0 B 10 MiB 0 B 0 B 0 B 8.8 MiB 1 WAL 0 B 2.5 GiB 0 B 0 B 0 B 751 MiB 8 DB 0 B 56 GiB 128 KiB 0 B 0 B 50 GiB 842 SLOW 0 B 0 B 0 B 0 B 0 B 0 B 0 TOTAL 0 B 58 GiB 128 KiB 0 B 0 B 0 B 850 MAXIMUMS: LOG 0 B 22 MiB 0 B 0 B 0 B 18 MiB WAL 0 B 3.9 GiB 0 B 0 B 0 B 1.0 GiB DB 0 B 71 GiB 282 MiB 0 B 0 B 62 GiB SLOW 0 B 0 B 0 B 0 B 0 B 0 B TOTAL 0 B 74 GiB 282 MiB 0 B 0 B 0 B >> SIZE << 0 B 453 GiB 14 TiB I had a look at the "DUMPING STATS" output in the logs bug I don't know how to interpret it. I did try calculating the total of the sizes on the "Sum" lines but that comes to 100G so I don't know what that all means. See attached log-stats-76. I also tried "ceph-kvstore-tool bluestore-kv ... stats": $ { cephadm unit --fsid $clusterid --name osd.76 stop cephadm shell --fsid $clusterid --name osd.76 -- ceph-kvstore-tool bluestore-kv /var/lib/ceph/osd/ceph-76 stats cephadm unit --fsid $clusterid --name osd.76 start } Output attached as bluestore-kv-stats-76. I can't see anything interesting in there, although again I don't really know how to interpret it. So... why is this osd db still spilling onto slow storage, and how do I fix things so it's no longer using the slow storage? And a bonus issue... on another osd that hasn't yet been resized (i.e. again with a grossly undersized 60G db on SSD with a 15T HDD) I'm also getting a spillover warning. The "bluefs stats" seems to be saying the db is NOT currently spilling (i.e. "0 B" the DB/SLOW position in the matrix), but there's "something" currently using 59G on the slow device: $ ceph tell osd.85 bluefs stats 1 : device size 0xeffffe000 : using 0x3a3900000(15 GiB) 2 : device size 0xe8d7fc00000 : using 0x7aea7434000(7.7 TiB) RocksDBBlueFSVolumeSelector Usage Matrix: DEV/LEV WAL DB SLOW * * REAL FILES LOG 0 B 10 MiB 0 B 0 B 0 B 7.4 MiB 1 WAL 0 B 564 MiB 0 B 0 B 0 B 132 MiB 2 DB 0 B 11 GiB 0 B 0 B 0 B 8.1 GiB 177 SLOW 0 B 3.0 GiB 59 GiB 0 B 0 B 56 GiB 898 TOTAL 0 B 13 GiB 59 GiB 0 B 0 B 0 B 1072 MAXIMUMS: LOG 0 B 24 MiB 0 B 0 B 0 B 20 MiB WAL 0 B 2.8 GiB 0 B 0 B 0 B 1.0 GiB DB 0 B 22 GiB 448 KiB 0 B 0 B 18 GiB SLOW 0 B 3.3 GiB 62 GiB 0 B 0 B 62 GiB TOTAL 0 B 27 GiB 62 GiB 0 B 0 B 0 B >> SIZE << 0 B 57 GiB 14 TiB Is there anywhere that describes how to interpret this output, and specifically, what stuff is going into the SLOW row? Seemingly there's 898 "files" there, but not LOG, WAL or DB files - so what are they? Cheers, Chris

4 7

Pacific 16.2.15 `osd noin`
by Zakhar Kirpichenko 08 Jul '24

08 Jul '24

Hi, I'm adding a few OSDs to an existing cluster, the cluster is running with `osd noout,noin`: cluster: id: 3f50555a-ae2a-11eb-a2fc-ffde44714d86 health: HEALTH_WARN noout,noin flag(s) set Specifically `noin` is documented as "prevents booting OSDs from being marked in". But freshly added OSDs were immediately marked `up` and `in`: services: ... osd: 96 osds: 96 up (since 5m), 96 in (since 6m); 338 remapped pgs flags noout,noin # ceph osd tree in | grep -E "osd.11|osd.12|osd.26" 11 hdd 9.38680 osd.11 up 1.00000 1.00000 12 hdd 9.38680 osd.12 up 1.00000 1.00000 26 hdd 9.38680 osd.26 up 1.00000 1.00000 Is this expected behavior? Do I misunderstand the purpose of the `noin` option? Best regards, Zakhar

5 7

Correct Migration Workflow Replicated -> Erasure Code
by Mac Wynkoop 19 Jun '24

19 Jun '24

Hi Everyone, So, I'm in the process of trying to migrate our rgw.buckets.data pool from a replicated rule pool to an erasure coded pool. I've gotten the EC pool set up, good EC profile and crush ruleset, pool created successfully, but when I go to "rados cppool xxx.rgw.buckets.data xxx.rgw.buckets.data.new", I get this error after it transfers 4GB of data: error copying object: (2) No such file or directory error copying pool xxx.rgw.buckets.data => xxx.rgw.buckets.data.new: (2) No such file or directory Is "rados cppool" still the blessed way to do the migration, or has something better/not deprecated been developed that I can use? Thanks, Mac

4 4

Re: MDS crashes to damaged metadata
by Patrick Donnelly 10 Jun '24

10 Jun '24

On Thu, Dec 15, 2022 at 9:32 AM Stolte, Felix <f.stolte(a)fz-juelich.de> wrote: > > Hi Patrick, > > we used your script to repair the damaged objects on the weekend and it went smoothly. Thanks for your support. > > We adjusted your script to scan for damaged files on a daily basis, runtime is about 6h. Until thursday last week, we had exactly the same 17 Files. On thursday at 13:05 a snapshot was created and our active mds crashed once at this time (snapshot was created): > > 2022-12-08T13:05:48.919+0100 7f440afec700 -1 /build/ceph-16.2.10/src/mds/ScatterLock.h: In function 'void ScatterLock::set_xlock_snap_sync(MDSContext*)' thread 7f440afec700 time 2022-12-08T13:05:48.921223+0100 > /build/ceph-16.2.10/src/mds/ScatterLock.h: 59: FAILED ceph_assert(state LOCK_XLOCK || state LOCK_XLOCKDONE) > > 12 Minutes lates the unlink_local error crashes appeared again. This time with a new file. During debugging we noticed a MTU mismatch between MDS (1500) and client (9000) with cephfs kernel mount. The client is also creating the snapshots via mkdir in the .snap directory. > > We disabled snapshot creation for now, but really need this feature. I uploaded the mds logs of the first crash along with the information above to https://tracker.ceph.com/issues/38452 > > I would greatly appreciate it, if you could answer me the following question: > > Is the Bug related to our MTU Mismatch? We fixed the MTU Issue going back to 1500 on all nodes in the ceph public network on the weekend also. I doubt it. > If you need a debug level 20 log of the ScatterLock for further analysis, i could schedule snapshots at the end of our workdays and increase the debug level 5 Minutes arround snap shot creation. This would be very helpful! -- Patrick Donnelly, Ph.D. He / Him / His Principal Software Engineer Red Hat, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

2 4

Are we logging IRC channels?
by Alvaro Soto 08 Jun '24

08 Jun '24

Hi Community!!! Are we logging IRC channels? I ask this because a lot of people only use Slack, and the Slack we use doesn't have a subscription, so messages are lost after 90 days (I believe) I believe it's important to keep track of the technical knowledge we see each day over IRC+Slack Cheers! -- Alvaro Soto *Note: My work hours may not be your work hours. Please do not feel the need to respond during a time that is not convenient for you.* ---------------------------------------------------------- Great people talk about ideas, ordinary people talk about things, small people talk... about other people.

5 9

stretched cluster new pool and second pool with nvme
by ronny.lippold 04 Jun '24

04 Jun '24

hi ... running against the wall, i need your help, again. our test stretched cluster is running fine. now i have 2 questions. whats the right way to add another pool? create pool with 4/2 and use the rule for the stretched mode, finished? the exsisting pools were automaticly set to 4/2 after "ceph mon enable_stretch_mode". the second question, we want to use ssd and nvme together. so, we need to have a second pool for class nvme. i don't know, how to setup a second crush rule for the nvme class. i thought, that i need to filter with 2 rules for the classes. is that correct? thanks for help, ronny

3 7

Help with deep scrub warnings
by Nicola Mori 24 May '24

24 May '24

Dear Ceph users, in order to reduce the deep scrub load on my cluster I set the deep scrub interval to 2 weeks, and tuned other parameters as follows: # ceph config get osd osd_deep_scrub_interval 1209600.000000 # ceph config get osd osd_scrub_sleep 0.100000 # ceph config get osd osd_scrub_load_threshold 0.300000 # ceph config get osd osd_deep_scrub_randomize_ratio 0.100000 # ceph config get osd osd_scrub_min_interval 259200.000000 # ceph config get osd osd_scrub_max_interval 1209600.000000 In my admittedly poor knowledge of Ceph's deep scrub procedures, these settings should spread the deep scrub operations in two weeks instead of the default one week, lowering the scrub frequency and the related load. But I'm currently getting warnings like: [WRN] PG_NOT_DEEP_SCRUBBED: 56 pgs not deep-scrubbed in time pg 3.1e1 not deep-scrubbed since 2024-02-22T00:22:55.296213+0000 pg 3.1d9 not deep-scrubbed since 2024-02-20T03:41:25.461002+0000 pg 3.1d5 not deep-scrubbed since 2024-02-20T09:52:57.334058+0000 pg 3.1cb not deep-scrubbed since 2024-02-20T03:30:40.510979+0000 . . . I don't understand the first one, since the deep scrub interval should be two weeks so I don''t expect warnings for PGs which have been deep-scrubbed less than 14 days ago (at the moment I'm writing it's Tue Mar 5 07:39:07 UTC 2024). Moreover, I don't understand why the deep scrub for so many PGs is lagging behind. Is there something wrong in my settings? Thanks in advance for any help, Nicola

4 4

Problems adding a new host via orchestration.
by Gary Molenkamp 22 May '24

22 May '24

Happy Friday all. I was hoping someone could point me in the right direction or clarify any limitations that could be impacting an issue I am having. I'm struggling to add a new set of hosts to my ceph cluster using cephadm and orchestration. When trying to add a host: "ceph orch host add <hostname> 172.31.102.41 --labels _admin" returns: "Error EINVAL: Can't communicate with remote host `172.31.102.41`, possibly because python3 is not installed there: [Errno 12] Cannot allocate memory" I've verified that the ceph ssh key works to the remote host, host's name matches that returned from `hostname`, python3 is installed, and "/usr/sbin/cephadm prepare-host" on the new hosts returns "host is ok". In addition, the cluster ssh key works between hosts and the existing hosts are able to ssh in using the ceph key. The existing ceph cluster is Pacific release using docker based containerization on RockyLinux8 base OS. The new hosts are RockyLinux9 based, with the cephadm being installed from Quincy release: ./cephadm add-repo --release quincy ./cephadm install I did try installing cephadm from the Pacific release by changing the repo to el8, but that did not work either. Is there a limitation is mixing RL8 and RL9 container hosts under Pacific? Does this same limitation exist under Quincy? Is there a python version dependency? The reason for RL9 on the new hosts is to stage upgrading the OS's for the cluster. I did this under Octopus for moving from Centos7 to RL8. Thanks and I appreciate any feedback/pointers. Gary I've added the log trace here in case that helps (from `ceph log last cephadm`) 2024-02-02T14:22:32.610048+0000 mgr.storage01.oonvfl (mgr.441023307) 4957871 : cephadm [ERR] Can't communicate with remote host `172.31.102.41`, possibly because python3 is not installed there: [Errno 12] Cannot allocate memory Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/serve.py", line 1524, in _remote_connection conn, connr = self.mgr._get_connection(addr) File "/usr/share/ceph/mgr/cephadm/module.py", line 1370, in _get_connection sudo=True if self.ssh_user != 'root' else False) File "/lib/python3.6/site-packages/remoto/backends/__init__.py", line 35, in __init__ self.gateway = self._make_gateway(hostname) File "/lib/python3.6/site-packages/remoto/backends/__init__.py", line 46, in _make_gateway self._make_connection_string(hostname) File "/lib/python3.6/site-packages/execnet/multi.py", line 133, in makegateway io = gateway_io.create_io(spec, execmodel=self.execmodel) File "/lib/python3.6/site-packages/execnet/gateway_io.py", line 121, in create_io io = Popen2IOMaster(args, execmodel) File "/lib/python3.6/site-packages/execnet/gateway_io.py", line 21, in __init__ self.popen = p = execmodel.PopenPiped(args) File "/lib/python3.6/site-packages/execnet/gateway_base.py", line 184, in PopenPiped return self.subprocess.Popen(args, stdout=PIPE, stdin=PIPE) File "/lib64/python3.6/subprocess.py", line 729, in __init__ restore_signals, start_new_session) File "/lib64/python3.6/subprocess.py", line 1295, in _execute_child restore_signals, start_new_session, preexec_fn) OSError: [Errno 12] Cannot allocate memory During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/serve.py", line 1528, in _remote_connection raise execnet.gateway_bootstrap.HostNotFound(msg) execnet.gateway_bootstrap.HostNotFound: Can't communicate with remote host `172.31.102.41`, possibly because python3 is not installed there: [Errno 12] Cannot allocate memory The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 125, in wrapper return OrchResult(f(*args, **kwargs)) File "/usr/share/ceph/mgr/cephadm/module.py", line 2709, in apply results.append(self._apply(spec)) File "/usr/share/ceph/mgr/cephadm/module.py", line 2574, in _apply return self._add_host(cast(HostSpec, spec)) File "/usr/share/ceph/mgr/cephadm/module.py", line 1517, in _add_host ip_addr = self._check_valid_addr(spec.hostname, spec.addr) File "/usr/share/ceph/mgr/cephadm/module.py", line 1498, in _check_valid_addr error_ok=True, no_fsid=True) File "/usr/share/ceph/mgr/cephadm/serve.py", line 1326, in _run_cephadm with self._remote_connection(host, addr) as tpl: File "/lib64/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/usr/share/ceph/mgr/cephadm/serve.py", line 1558, in _remote_connection raise OrchestratorError(msg) from e orchestrator._interface.OrchestratorError: Can't communicate with remote host `172.31.102.41`, possibly because python3 is not installed there: [Errno 12] Cannot allocate memory -- Gary Molenkamp Science Technology Services Systems Engineer University of Western Ontario molenkam(a)uwo.ca http://sts.sci.uwo.ca (519) 661-2111 x86882 (519) 661-3566

5 9

rbd-mirror failed to query services: (13) Permission denied
by Stefan Kooman 21 May '24

21 May '24

Hi, We are testing rbd-mirroring. There seems to be a permission error with the rbd-mirror user. Using this user to query the mirror pool status gives: failed to query services: (13) Permission denied And results in the following output: health: UNKNOWN daemon health: UNKNOWN image health: OK images: 3 total 2 replaying 1 stopped So, this command: rbd --id rbd-mirror mirror pool status rbd So basically the health and daemon health cannot be obtained due to permission errors, but status about images can. When the command is run with admin permissions the health and daemon health are returned without issue. I tested this on Reef 18.2.2. Is this expected behavior? If not, I will create a tracker ticket for it. Gr. Stefan

3 5

Ceph reef and (slow) backfilling - how to speed it up
by Götz Reinicke 12 May '24

12 May '24

Dear ceph community, I’ve a ceph cluster which got upgraded from nautilus/pacific/…to reef over time. Now I added two new nodes to an existing EC pool as I did with the previous versions of ceph. Now I face the fact, that the previous „backfilling tuning“ I’v used by increasing injectargs --osd-max-backfills=XX --osd-recovery-max-active=YY dose not work anymore. With adjusting thous parameters the backfill was running with up to 2k +- objects/s. As I’m not (yet) familiar with the reef opiont the only speed up so far I found is „ceph config set osd osd_mclock_profile high_recovery_ops“ which currently runs the backfill with up to 600 opbjects/s. My question: What is a best (simple) way to speed that backfill up ? I’v tried to understand the custom profiles (?) but without success - and did not apply anything other yet. Thanks for feedback and suggestions ! Best regards . Götz

7 10

cache pressure?
by Erich Weiler 07 May '24

07 May '24

So I'm trying to figure out ways to reduce the number of warnings I'm getting and I'm thinking about the one "client failing to respond to cache pressure". Is there maybe a way to tell a client (or all clients) to reduce the amount of cache it uses or to release caches quickly? Like, all the time? I know the linux kernel (and maybe ceph) likes to cache everything for a while, and rightfully so, but I suspect in my use case it may be more efficient to more quickly purge the cache or to in general just cache way less overall...? We have many thousands of threads all doing different things that are hitting our filesystem, so I suspect the caching isn't really doing me much good anyway due to the churn, and probably is causing more problems than it helping... -erich

3 9

Mysterious Space-Eating Monster
by duluxoz 07 May '24

07 May '24

Hi All, *Something* is chewing up a lot of space on our `\var` partition to the point where we're getting warnings about the Ceph monitor running out of space (ie > 70% full). I've been looking, but I can't find anything significant (ie log files aren't too big, etc) BUT there seem to be a hell of a lot (15) of sub-directories (with GUIDs for names) under the `/var/lib/containers/storage/overlay/` folder, all ending with `merged` - ie `/var/lib/containers/storage/overlay/{{GUID}}/`merged`. Is this normal, or is something going wrong somewhere, or am I looking in the wrong place? Also, if this is the issue, can I delete these folders? Sorry for asking such a noob Q, but the Cephadm/Podman stuff is extremely new to me :-) Thanks in advance Cheers Dulux-Oz

3 3

radosgw sync non-existent bucket ceph reef 18.2.2
by Christopher Durham 06 May '24

06 May '24

Hi, I have a reef cluster 18.2.2 on Rocky 8.9. This cluster has been upgraded from pacific->quincy->reef over the past few years. It is a multi site with one other cluster that works fine with s3/radosgw on both sides, with proper bidirectional data replication. On one of the master cluster's radosgw logs, I noticed a sync request regarding a deleted bucket. I am not sure when this error started, but I know that the bucket in question was deleted a long time before the upgrade to reef. Perhapsthis error existed prior to reef, I do not know. Here is the error in the radosgw log: :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe{s={b=BUCKET_NAME:CLUSTERID ..., z=...., az= ...},d={b=..,az=...}} returned ret=-2 My understanding: s=source, d=destination, each of which is a tuple with the appropriate info necessary.... This happens for BUCKET_NAME every few minutes. Said bucket does not exist on either side of the multisite, but did in the past. Any way I can force radosgw to stop trying to replicate? Thanks -Chris

2 1

Unable to add new OSDs
by ceph＠mikesoffice.com 06 May '24

06 May '24

I'm trying to add a new storage host into a Ceph cluster (quincy 17.2.6). The machine has boot drives, one free SSD and 10 HDDs. The plan is to have each HDD be an OSD with a DB on a equal size lvm of the SDD. This machine is newer but otherwise similar to other machines already in the cluster that are setup and running the same way. But I've been unable to add OSDs and unable to figure out why, or fix it. I have some experience, but I'm not an expert and could be missing something obvious. If anyone has any suggestions, I would appreciate it. I've tried to add OSDs a couple different ways. Via the dashboard, this has worked fine for previous machines. And it appears to succeed and gives no errors that I can find looking in /var/log/ceph and dashboard logs. The OSDs are never created. In fact, the drives still show up as available in Physical Disks and I can do the same creation procedure repeatedly. I've tried creating it in cephadm shell with the following, which has also worked in the past: ceph orch daemon add osd stor04.fqdn:data_devices=/dev/sdb,/dev/sdc,/dev/sdd,/dev/sde,/dev/sdf,/dev/sdg,/dev/sdh,/dev/sdi,/dev/sdj,/dev/sdk,db_devices=/dev/sda,osds_per_device=1 The command just hangs. Again I wasn't able to find any obvious errors. Although this one did seem to cause some slow op errors from the monitors that required restarting a monitor. And it could cause errors with the dashboard locking up and having to restart the manager as well. And I've tried setting 'ceph orch apply osd --all-available-devices --unmanaged=false' to let Ceph automatically add the drives. In the past, this would cause Ceph to automatically add the drives as OSDs but without having associated DBs on the SSD. The SSD would just be another OSD. This time it appears to have no affect and similar to the above, I wasn't able to find any obvious error feedback. -Mike

4 3

Have a problem with haproxy/keepalived/ganesha/docker
by Ruslan Nurabayev 06 May '24

06 May '24

Hello! I've installed my 5-node CEPH cluster next install NFS server by command: ceph nfs cluster create nfshacluster 5 --ingress --virtual_ip 192.168.171.48/26 --ingress-mode haproxy-protocol. I don't understand fully how this must be works but when i stop NFS daemon even on one of this nodes I've see that writing on NFS shares is disappear (testing via vdbench). As i understand it is wrong and IO from stopped daemon must switching to another NFS daemon without any impact on IO. Can someone help me with troubleshoot this issue? Or explain how done full-fledged Active-Active HA NFS Cluster for production use. Thanks! Руслан Нурабаев Старший инженер Сектор ИТ платформы Отдел развития опорной сети Департамент развития сети +77012119272 Ruslan.Nurabayev(a)kcell.kz -----Original Message----- From: ceph-users-request(a)ceph.io <ceph-users-request(a)ceph.io> Sent: Thursday, April 11, 2024 15:07 To: Ruslan Nurabayev <Ruslan.Nurabayev(a)kcell.kz> Subject: Welcome to the "ceph-users" mailing list [You don't often get email from ceph-users-request(a)ceph.io. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] Welcome to the "ceph-users" mailing list! To post to this list, send your email to: ceph-users(a)ceph.io You can unsubscribe or make adjustments to your options via email by sending a message to: ceph-users-request(a)ceph.io with the word 'help' in the subject or body (don't include the quotes), and you will get back a message with instructions. You will need your password to change your options, but for security purposes, this password is not included here. If you have forgotten your password you will need to reset it via the web UI. ________________________________ **************************************************************************************** Осы хабарлама және онымен берілетін кез келген файлдар құпия болып табылады және олар мекенжайда көрсетілген жеке немесе заңды тұлғалардың пайдалануына ғана арналған. Егер сіз болжамды алушы болып табылмайтын болсаңыз, осы арқылы осындай ақпаратты кез келген таратуға, жіберуге, көшіруге немесе пайдалануға қатаң тыйым салынатыны және осы электрондық хабарлама дереу жойылуға тиіс екендігін хабарлаймыз. KCELL осы хабарламадағы кез келген ақпараттың дәлдігіне немесе толықтығына қатысты ешқандай кепілдік бермейді және сол арқылы онда қамтылған ақпарат үшін немесе оны беру, қабылдау, сақтау немесе қандай да бір түрде пайдалану үшін кез келген жауапкершілікті болдырмайды. Осы хабарламада айтылған пікірлер тек жіберушіге ғана тиесілі және KCELL пікірін де білдіруі міндетті емес. Бұл электрондық хабарлама барлық танымал компьютерлік вирустарға тексерілді. **************************************************************************************** Данное сообщение и любые передаваемые с ним файлы являются конфиденциальными и предназначены исключительно для использования физическими или юридическими лицами, которым они адресованы. Если вы не являетесь предполагаемым получателем, настоящим уведомляем о том, что любое распространение, пересылка, копирование или использование такой информации строго запрещено, и данное электронное сообщение должно быть немедленно удалено. KCELL не дает никаких гарантий относительно точности или полноты любой информации, содержащейся в данном сообщении, и тем самым исключает любую ответственность за информацию, содержащуюся в нем, или за ее передачу, прием, хранение или использование каким-либо образом. Мнения, выраженные в данном сообщении, принадлежат только отправителю и не обязательно отражают мнение KCELL. Данное электронное сообщение было проверено на наличие всех известных компьютерных вирусов. **************************************************************************************** This e-mail and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you are not the intended recipient you are hereby notified that any dissemination, forwarding, copying or use of any of the information is strictly prohibited, and the e-mail should immediately be deleted. KCELL makes no warranty as to the accuracy or completeness of any information contained in this message and hereby excludes any liability of any kind for the information contained therein or for the information transmission, reception, storage or use of such in any way whatsoever. The opinions expressed in this message belong to sender alone and may not necessarily reflect the opinions of KCELL. This e-mail has been scanned for all known computer viruses. ****************************************************************************************

6 8

16.2.14: [progress WARNING root] complete: ev {UUID} does not exist
by Zakhar Kirpichenko 03 May '24

03 May '24

Hi, Mgr of my cluster logs this every few seconds: [progress WARNING root] complete: ev 7de5bb74-790b-4fda-8838-e4af4af18c62 does not exist [progress WARNING root] complete: ev fff93fce-b630-4141-81ee-19e7a3e61483 does not exist [progress WARNING root] complete: ev a02f6966-5b9f-49e8-89c4-b4fb8e6f4423 does not exist [progress WARNING root] complete: ev 8d318560-ff1a-477f-9386-43f6b51080bf does not exist [progress WARNING root] complete: ev ff3740a9-6434-470a-808f-a2762fb542a0 does not exist [progress WARNING root] complete: ev 7d0589f1-545e-4970-867b-8482ce48d7f0 does not exist [progress WARNING root] complete: ev 78d57e43-5be5-43f0-8b1a-cdc60e410892 does not exist I would appreciate an advice on what these warnings mean and how they can be resolved. Best regards, Zakhar

4 7

Reset health.
by Albert Shih 03 May '24

03 May '24

Hi, Very basic question : 2 days ago I reboot all the cluster. Everything work fine. But I'm guessing during the shutdown 4 osd was mark as crash [WRN] RECENT_CRASH: 4 daemons have recently crashed osd.381 crashed on host cthulhu5 at 2024-03-20T18:33:12.017102Z osd.379 crashed on host cthulhu4 at 2024-03-20T18:47:13.838839Z osd.376 crashed on host cthulhu3 at 2024-03-20T18:50:00.877536Z osd.373 crashed on host cthulhu1 at 2024-03-20T18:56:46.887394Z is they are any way to «clean» that ? Because otherwise my icinga complain.... I don't like to add a downtime in icinga. Thanks. -- Albert SHIH 🦫 🐸 France Heure locale/Local time: ven. 22 mars 2024 22:24:35 CET

3 2

Reconstructing an OSD server when the boot OS is corrupted
by Peter van Heusden 02 May '24

02 May '24

Dear Ceph Community We have 5 OSD servers running Ceph v15.2.17. The host operating system is Ubuntu 20.04. One of the servers has suffered corruption to its boot operating system. Using a system rescue disk it is possible to mount the root filesystem but it is not possible to boot the operating system at the moment. The OSDs are configured with (spinning disk) data drives, WALs and DBs on partitions of SSDs, but from my examination of the filesystem the configuration in /var/lib/ceph appears to be corrupted. So my question is: what is the best option for repair going forward? Is it possible to do a clean install of the operating system and scan the existing drives in order to reconstruct the OSD configuration? Thank you, Peter P.S. the cause of the original corruption is likely due to an unplanned power outage, an event that hopefully will not recur.

7 9

Ceph Day NYC 2024 Slides
by Matt Vandermeulen 02 May '24

02 May '24

Hi folks! Thanks for a great Ceph Day event in NYC! I wanted to make sure I posted my slides before I forget (and encourage others to do the same). Feel free to reach out in the Ceph Slack https://ceph.io/en/community/connect/ How we Operate Ceph at Scale (DigitalOcean): - https://do-matt-ams3.ams3.digitaloceanspaces.com/2024%20Ceph%20Day%20NYC%20… - https://do-matt-sfo3.sfo3.digitaloceanspaces.com/2024%20Ceph%20Day%20NYC%20… Discards in Ceph (DigitalOcean): - https://do-matt-ams3.ams3.digitaloceanspaces.com/2024%20Ceph%20Day%20NYC%20… - https://do-matt-sfo3.sfo3.digitaloceanspaces.com/2024%20Ceph%20Day%20NYC%20… Matt

2 2

After dockerized ceph cluster to Pacific, the fsid changed in the output of 'ceph -s'
by wjsherry075＠outlook.com 02 May '24

02 May '24

Hello, I had a problem after I finished the 'cephadm adopt' from services to docker containers for mon and mgr. The fsid of `ceph -s` is not the same as the /etc/ceph/ceph.conf. The ceph.conf is correct, but `ceph -s` is incorrect. I followed the https://docs.ceph.com/en/quincy/cephadm/adoption/ ``` 2024-04-25T19:49:17.460652+0000 mgr.cloud-lab-test-mon01 (mgr.65113) 109 : cephadm [ERR] cephadm exited with an error code: 1, stderr:ERROR: fsid does not match ceph.conf Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/serve.py", line 1538, in _remote_connection yield (conn, connr) File "/usr/share/ceph/mgr/cephadm/serve.py", line 1426, in _run_cephadm code, '\n'.join(err))) orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:ERROR: fsid does not match ceph.conf #ceph health detail shows the warning: [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices ``` Does anyone else have any ideas? Thanks, Sherry

2 1

Ceph client cluster compatibility
by Nima AbolhassanBeigi 02 May '24

02 May '24

Hi, We are trying to upgrade our OS version from ubuntu 18.04 to ubuntu 22.04. Our ceph cluster version is 16.2.13 (pacific). The problem is that the ubuntu packages for the ceph pacific release will not be supported for ubuntu 22.04. We were wondering if the ceph client (version 18.2, reef) on ubuntu 22.04 can work with lower version clusters. Thanks in advance Regards Nima

2 1

RGW multisite slowness issue due to the "304 Not Modified" responses on primary zone
by praveenkumargpk17＠gmail.com 01 May '24

01 May '24

Hi, We have 2 clusters (v18.2.1) primarily used for RGW which has over 2+ billion RGW objects. They are also in multisite configuration totaling to 2 zones and we've got around 2 Gbps of bandwidth dedicated (P2P) for the multisite traffic. We see that using "radosgw-admin sync status" on the zone 2, all the 128 shards are recovering and unfortunately there is very less data transfer from primary zone ie., the link utilization is barely 100 Mbps / 2 Gbps. Our objects are quite small as well like avg. of 1 MB in size. On further inspection, we noticed the rgw access the logs at primary site are mostly yielding "304 Not Modified" for RGWs at site-2. Is this expected? Here are some of the logs (information is redacted) root@host-04:~# tail -f /var/log/haproxy-msync.log Feb 12 05:06:51 host-04 haproxy[971171]: 10.1.85.14:33730 [12/Feb/2024:05:06:51.047] https~ backend/host-04-msync 0/0/0/2/2 304 143 - - ---- 56/55/1/0/0 0/0 "GET /bucket1/object1.jpg?rgwx-zonegroup=71dceb3d-3092-4dc6-897f-a9abf60c9972&rgwx-prepend-metadata=true&rgwx-sync-manifest&rgwx-sync-cloudtiered&rgwx-skip-decrypt&rgwx-if-not-replicated-to=a8204ce2-b69e-4d90-bca1-93edd05a1a29%3Abucket1%3A8b96aea5-c763-40a3-8430-efd67cff0c62.20010.7 HTTP/1.1" Feb 12 05:06:51 host-04 haproxy[971171]: 10.1.85.14:59730 [12/Feb/2024:05:06:51.048] https~ backend/host-04-msync 0/0/0/2/2 304 143 - - ---- 56/55/3/1/0 0/0 "GET /bucket1/object91.jpg?rgwx-zonegroup=71dceb3d-3092-4dc6-897f-a9abf60c9972&rgwx-prepend-metadata=true&rgwx-sync-manifest&rgwx-sync-cloudtiered&rgwx-skip-decrypt&rgwx-if-not-replicated-to=a8204ce2-b69e-4d90-bca1-93edd05a1a29%3Abucket1%3A8b96aea5-c763-40a3-8430-efd67cff0c62.20010.7 HTTP/1.1" We also took a look at our Grafana instance and out of 1000 requests / second, 200 requests are getting "200 OK" and 800 requests are getting "304 Not Modified". Sync threads are run on only 2 rgw daemons per zone and are behind a Load Balancer. "# radosgw-admin sync error list" also contains around 20 errors which are mostly automatically recoverable. As we understand, does it mean that RGW multisite sync logs in the log pool are yet to be generated or some sort? Please provide us some insights and let us know how to resolve this. Thanks, Praveen

4 4

Re: Stuck OSD service specification - can't remove
by Wang Jie 30 Apr '24

30 Apr '24

Hello David, did you resolve it? I have the same problem for rgw. I upgraded from N to P. Regards, Jie

1 0

Remove an OSD with hardware issue caused rgw 503
by Mary Zhang 30 Apr '24

30 Apr '24

Hi, We recently removed an osd from our Cepth cluster. Its underlying disk has a hardware issue. We use command: ceph orch osd rm osd_id --zap During the process, sometimes ceph cluster enters warning state with slow ops on this osd. Our rgw also failed to respond to requests and returned 503. We restarted rgw daemon to make it work again. But the same failure occured from time to time. Eventually we noticed that rgw 503 error is a result of osd slow ops. Our cluster has 18 hosts and 210 OSDs. We expect remove an osd with hardware issue won't impact cluster performance & rgw availbility. Is our expectation reasonable? What's the best way to handle osd with hardware failures? Thank you in advance for any comments or suggestions. Best Regards, Mary Zhang

4 11

Ceph Squid released?
by Robert Sander 30 Apr '24

30 Apr '24

Hi, https://www.linuxfoundation.org/press/introducing-ceph-squid-the-future-of-… Does the LF know more than the mailing list? Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

4 6

SPDK with cephadm and reef
by R A 30 Apr '24

30 Apr '24

Hello Community, is there a guide / documentation how to configure spdk with cephadm (running in containers) in reef? BR

1 0

Re: MDS Behind on Trimming...
by Erich Weiler 30 Apr '24

30 Apr '24

Hi Xiubo, Nevermind I was wrong, most the blocked ops were 12 hours old. Ug. I restarted the MDS daemon to clear them. I just reset to having one active MDS instead of two, let's see if that makes a difference. I am beginning to think it may be impossible to catch the logs that matter here. I feel like sometimes the blocked ops are just waiting because of load and sometimes they are waiting because they are stuck. But, it's really hard to tell which, without waiting a while. But, I can't wait while having debug turned on because my root disks (which are 150 GB large) fill up with debug logs in 20 minutes. So it almost seems that unless I could somehow store many TB of debug logs we won't be able to catch this. Let's see how having one MDS helps. Or maybe I actually need like 4 MDSs because the load is too high for only one or two. I don't know. Or maybe it's the lock issue you've been working on. I guess I can test the lock order fix when it's available to test. -erich On 4/19/24 7:26 AM, Erich Weiler wrote: > So I woke up this morning and checked the blocked_ops again, there were > 150 of them. But the age of each ranged from 500 to 4300 seconds. So > it seems as if they are eventually being processed. > > I wonder if we are thinking about this in the wrong way? Maybe I should > be *adding* MDS daemons because my current ones are overloaded? > > Can a single server hold multiple MDS daemons? Right now I have three > physical servers each with one MDS daemon on it. > > I can still try reducing to one. And I'll keep an eye on blocked ops to > see if any get to a very old age (and are thus wedged). > > -erich > > On 4/18/24 8:55 PM, Xiubo Li wrote: >> Okay, please try it to set only one active mds. >> >> >> On 4/19/24 11:54, Erich Weiler wrote: >>> We have 2 active MDS daemons and one standby. >>> >>> On 4/18/24 8:52 PM, Xiubo Li wrote: >>>> BTW, how man active mds you are using ? >>>> >>>> >>>> On 4/19/24 10:55, Erich Weiler wrote: >>>>> OK, I'm sure I caught it in the right order this time, the logs >>>>> should definitely show when the blocked/slow requests start. Check >>>>> out these logs and dumps: >>>>> >>>>> http://hgwdev.gi.ucsc.edu/~weiler/ >>>>> >>>>> It's a 762 MB tarball but it uncompresses to 16 GB. >>>>> >>>>> -erichll >>>>> >>>>> >>>>> On 4/18/24 6:57 PM, Xiubo Li wrote: >>>>>> Okay, could you try this with 18.2.0 ? >>>>>> >>>>>> I just double it was introduce by: >>>>>> >>>>>> commit e610179a6a59c463eb3d85e87152ed3268c808ff >>>>>> Author: Patrick Donnelly <pdonnell(a)redhat.com> >>>>>> Date: Mon Jul 17 16:10:59 2023 -0400 >>>>>> >>>>>> mds: drop locks and retry when lock set changes >>>>>> >>>>>> An optimization was added to avoid an unnecessary gather on >>>>>> the inode >>>>>> filelock when the client can safely get the file size without >>>>>> also >>>>>> getting issued the requested caps. However, if a retry of >>>>>> getattr >>>>>> is necessary, this conditional inclusion of the inode filelock >>>>>> can cause lock-order violations resulting in deadlock. >>>>>> >>>>>> So, if we've already acquired some of the inode's locks then >>>>>> we must >>>>>> drop locks and retry. >>>>>> >>>>>> Fixes: https://tracker.ceph.com/issues/62052 >>>>>> Fixes: c822b3e2573578c288d170d1031672b74e02dced >>>>>> Signed-off-by: Patrick Donnelly <pdonnell(a)redhat.com> >>>>>> (cherry picked from commit >>>>>> b5719ac32fe6431131842d62ffaf7101c03e9bac) >>>>>> >>>>>> >>>>>> On 4/19/24 09:54, Erich Weiler wrote: >>>>>>> I'm on 18.2.1. I think I may have gotten the timing off on the >>>>>>> logs and dumps so I'll try again. Just really hard to capture >>>>>>> because I need to kind of be looking at it in real time to >>>>>>> capture it. Hang on, lemme see if I can get another capture... >>>>>>> >>>>>>> -erich >>>>>>> >>>>>>> On 4/18/24 6:35 PM, Xiubo Li wrote: >>>>>>>> >>>>>>>> BTW, which ceph version you are using ? >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 4/12/24 04:22, Erich Weiler wrote: >>>>>>>>> BTW - it just happened again, I upped the debugging settings as >>>>>>>>> you instructed and got more dumps (then returned the debug >>>>>>>>> settings to normal). >>>>>>>>> >>>>>>>>> Attached are the new dumps. >>>>>>>>> >>>>>>>>> Thanks again, >>>>>>>>> erich >>>>>>>>> >>>>>>>>> On 4/9/24 9:00 PM, Xiubo Li wrote: >>>>>>>>>> >>>>>>>>>> On 4/10/24 11:48, Erich Weiler wrote: >>>>>>>>>>>>> Dos that mean it could be the locker order bug >>>>>>>>>>>>> (https://tracker.ceph.com/issues/62123) as Xiubo suggested? >>>>>>>>>>>> >>>>>>>>>>>> I have raised one PR to fix the lock order issue, if >>>>>>>>>>>> possible please have a try to see could it resolve this issue. >>>>>>>>>>> >>>>>>>>>>> Thank you! Yeah, this issue is happening every couple days >>>>>>>>>>> now. It just happened again today and I got more MDS dumps. >>>>>>>>>>> If it would help, let me know and I can send them! >>>>>>>>>>> >>>>>>>>>> Once this happen if you could enable the mds debug logs will >>>>>>>>>> be better: >>>>>>>>>> >>>>>>>>>> debug mds = 20 >>>>>>>>>> >>>>>>>>>> debug ms = 1 >>>>>>>>>> >>>>>>>>>> And then provide the debug logs together with the MDS dumps. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> I assume if this fix is approved and backported it will then >>>>>>>>>>> appear in like 18.2.3 or something? >>>>>>>>>>> >>>>>>>>>> Yeah, it will be backported after being well tested. >>>>>>>>>> >>>>>>>>>> - Xiubo >>>>>>>>>> >>>>>>>>>>> Thanks again, >>>>>>>>>>> erich >>>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >>

2 3

ceph recipe for nfs exports
by Roberto Maggi ＠ Debian 29 Apr '24

29 Apr '24

Hi you all, I'm almost new to ceph and I'm understanding, day by day, why the official support is so expansive :) I setting up a ceph nfs network cluster whose recipe can be found here below. ####################### --> cluster creation cephadm bootstrap --mon-ip 10.20.20.81 --cluster-network 10.20.20.0/24 --fsid $FSID --initial-dashboard-user adm \ --initial-dashboard-password 'Hi_guys' --dashboard-password-noupdate --allow-fqdn-hostname --ssl-dashboard-port 443 \ --dashboard-crt /etc/ssl/wildcard.it/wildcard.it.crt --dashboard-key /etc/ssl/wildcard.it/wildcard.it.key \ --allow-overwrite --cleanup-on-failure cephadm shell --fsid $FSID -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring cephadm add-repo --release reef && cephadm install ceph-common --> adding hosts and set labels for IP in $(grep ceph /etc/hosts | awk '{print $1}') ; do ssh-copy-id -f -i /etc/ceph/ceph.pub root@$IP ; done ceph orch host add cephstage01 10.20.20.81 --labels _admin,mon,mgr,prometheus,grafana ceph orch host add cephstage02 10.20.20.82 --labels _admin,mon,mgr,prometheus,grafana ceph orch host add cephstage03 10.20.20.83 --labels _admin,mon,mgr,prometheus,grafana ceph orch host add cephstagedatanode01 10.20.20.84 --labels osd,nfs,prometheus ceph orch host add cephstagedatanode02 10.20.20.85 --labels osd,nfs,prometheus ceph orch host add cephstagedatanode03 10.20.20.86 --labels osd,nfs,prometheus --> network setup and daemons deploy ceph config set mon public_network 10.20.20.0/24,192.168.7.0/24 ceph orch apply mon --placement="cephstage01:10.20.20.81,cephstage02:10.20.20.82,cephstage03:10.20.20.83" ceph orch apply mgr --placement="cephstage01:10.20.20.81,cephstage02:10.20.20.82,cephstage03:10.20.20.83" ceph orch apply prometheus --placement="cephstage01:10.20.20.81,cephstage02:10.20.20.82,cephstage03:10.20.20.83,cephstagedatanode01:10.20.20.84,cephstagedatanode02:10.20.20.85,cephstagedatanode03:10.20.20.86" ceph orch apply grafana --placement="cephstage01:10.20.20.81,cephstage02:10.20.20.82,cephstage03:10.20.20.83,cephstagedatanode01:10.20.20.84,cephstagedatanode02:10.20.20.85,cephstagedatanode03:10.20.20.86" ceph orch apply node-exporter ceph orch apply alertmanager ceph config set mgr mgr/cephadm/secure_monitoring_stack true --> disks and osd setup for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; do ssh root@$IP "hostname && wipefs -a -f /dev/sdb&& wipefs -a -f /dev/sdc"; done ceph config set mgr mgr/cephadm/device_enhanced_scan true for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; doceph orch device ls --hostname=$IP --wide --refresh ; done for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; doceph orch device zap $IP /dev/sdb; done for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; doceph orch device zap $IP /dev/sdc ; done for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; doceph orch daemon add osd $IP:/dev/sdb ; done for IP in $(grep cephstagedatanode/etc/hosts | awk '{print $1}') ; doceph orch daemon add osd $IP:/dev/sdc ; done --> ganesha nfs cluster ceph mgr module enable nfs ceph fs volume create vol1 ceph nfs cluster create nfs-cephfs "cephstagedatanode01,cephstagedatanode02,cephstagedatanode03" --ingress --virtual-ip 192.168.7.80 --ingress-mode default ceph nfs export create cephfs --cluster-id nfs-cephfs --pseudo-path /mnt --fsname vol1 --> nfs mount mount -t nfs -o nfsvers=4.1,proto=tcp 192.168.7.80:/mnt /mnt/ceph is my recipe correct? the cluster is set up by 3 mon/mgr nodes and 3 osd/nfs nodes, on the latters I installed one 3tb ssd, for the data, and one 300gb ssd for the journaling but my problems are : - Although I can mount the export I can't write on it - I can't understand how to use the sdc disks for journaling - I can't understand the concept of "pseudo path" here below you can find the json output of the exports --> check ceph nfs export ls nfs-cephfs ceph nfs export info nfs-cephfs /mnt ------------------------------------ json file --------- { "export_id": 1, "path": "/", "cluster_id": "nfs-cephfs", "pseudo": "/mnt", "access_type": "RW", "squash": "none", "security_label": true, "protocols": [ 4 ], "transports": [ "TCP" ], "fsal": { "name": "CEPH", "user_id": "nfs.nfs-cephfs.1", "fs_name": "vol1" }, "clients": [] } ------------------------------------ Thanks in advance Rob

5 7

which grafana version to use with 17.2.x ceph version
by Osama Elswah 29 Apr '24

29 Apr '24

Hi, in quay.io I can find a lot of grafana versions for ceph (https://quay.io/repository/ceph/grafana?tab=tags) how can I find out which version should be used when I upgrade my cluster to 17.2.x ? Can I simply take the latest grafana version? Or is there a specfic grafana version I need to use?

3 2

Impact of large PG splits
by Eugen Block 29 Apr '24

29 Apr '24

Hi, I'm trying to estimate the possible impact when large PGs are splitted. Here's one example of such a PG: PG_STAT OBJECTS BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG UP 86.3ff 277708 414403098409 0 0 3092 3092 [187,166,122,226,171,234,177,163,155,34,81,239,101,13,117,8,57,111] Their main application is RGW on EC (currently 1024 PGs on 240 OSDs), 8TB HDDs backed by SSDs. There are 6 RGWs running behind HAProxies. It took me a while to convince them to do a PG split and now they're trying to assess how big the impact could be. The fullest OSD is already at 85% usage, the least filled one at 59%, so there is definitely room for a better balancing which, will be necessary until the new hardware arrives. The current distribution is around 100 PGs per OSD which usually would be fine, but since the PGs are that large only a few PGs difference have a huge impact on the OSD utilization. I'm targeting 2048 PGs for that pool for now, probably do another split when the new hardware has been integrated. Any comments are appreciated! Eugen

6 16

Re: MDS crash
by Alexey GERASIMOV 28 Apr '24

28 Apr '24

Colleagues, thank you for the advice to check the operability of MGRs. In fact, it is strange also: we checked our nodes for the network issues (ip connectivity, sockets, ACL, DNS) and find nothing wrong - but suddenly just the restart of all MGRs solved the problem with stale PGs and with ceph commands hang! So, we are at the start point again - ceph is working except MDS daemons crash. But now we see some additional errors in MDS logs when try to start the daemon: dir 0x1000dd10fa0 object missing on disk; some files may be lost (/volumes/csi/csi-vol-2eb40f89-f2e1-11ee-b657-3aa98da4c4a6/1080803d-1277-4ad8-ae80-a004bd3a5699/gallery/pc-12083932925583528732) dir 0x1000dd10f9d object missing on disk; some files may be lost (/volumes/csi/csi-vol-2eb40f89-f2e1-11ee-b657-3aa98da4c4a6/1080803d-1277-4ad8-ae80-a004bd3a5699/cadserver-filevault/project-files/661fb14d341d3746ea5c2a8f I promiced to create the bug, so will do it later a bit. But should I try to do something more from my side also? What I did exactly last time: cephfs-journal-tool journal reset cephfs-table-tool all reset session cephfs-data-scan scan_extents cephfs-data-scan scan_inodes cephfs-data-scan scan_links cephfs-data-scan cleanup And one more question: is it possible to access to cephfs content directly, without MDS?

2 1

Re: MDS crash
by Alexey GERASIMOV 27 Apr '24

27 Apr '24

I don't know why, but I miss my topic when I reply to it. Moderators, please delete unnecessary topics and move my answer to the correct topic.

1 0

Re: MDS crash
by Alexey GERASIMOV 27 Apr '24

27 Apr '24

Colleagues, thank you for the advice to check the operability of MGRs. In fact, it is strange also: we checked our nodes for the network issues (ip connectivity, sockets, ACL, DNS) and find nothing wrong - but suddenly just the restart of all MGRs solved the problem with stale PGs and with ceph commands hang! So, we are at the start point again - ceph is working except MDS daemons crash. But now we see some additional errors in MDS logs when try to start the daemon: dir 0x1000dd10fa0 object missing on disk; some files may be lost (/volumes/csi/csi-vol-2eb40f89-f2e1-11ee-b657-3aa98da4c4a6/1080803d-1277-4ad8-ae80-a004bd3a5699/gallery/pc-12083932925583528732) dir 0x1000dd10f9d object missing on disk; some files may be lost (/volumes/csi/csi-vol-2eb40f89-f2e1-11ee-b657-3aa98da4c4a6/1080803d-1277-4ad8-ae80-a004bd3a5699/cadserver-filevault/project-files/661fb14d341d3746ea5c2a8f I promiced to create the bug, so will do it later a bit. But should I try to do something more from my side also? What I did exactly last time: cephfs-journal-tool journal reset cephfs-table-tool all reset session cephfs-data-scan scan_extents cephfs-data-scan scan_inodes cephfs-data-scan scan_links cephfs-data-scan cleanup And one more question: is it possible to access to cephfs content directly, without MDS?

1 0

rbd-mirror get status updates quicker
by Stefan Kooman 27 Apr '24

27 Apr '24

Hi, We're testing with rbd-mirror (mode snapshot) and try to get status updates about snapshots as fast a possible. We want to use rbd-mirror as a migration tool between two clusters and keep downtime during migration as short as possible. Therefore we have tuned the following parameters and set them to 1 second (default 30 seconds): rbd_mirror_pool_replayers_refresh_interval rbd_mirror_image_state_check_interval rbd_mirror_sync_point_update_age However, on the destination cluster, the "last_update:" field is only updated every 30 seconds. Is this tunable? Goal is to determine when the last snapshot that is made on the source has made it to the target and a demote (source) and promote (target) can be initiated. Gr. Stefan

2 1

Re: Add node-exporter using ceph orch
by Robert Sander 26 Apr '24

26 Apr '24

On 4/26/24 15:47, Vahideh Alinouri wrote: > The result of this command shows one of the servers in the cluster, > but I have node-exporter daemons on all servers. The default service specification looks like this: service_type: node-exporter service_name: node-exporter placement: host_pattern: '*' If you apply this YAML code the orchestrator should deploy one node-exporter daemon to each host of the cluster. Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

Add node-exporter using ceph orch
by Vahideh Alinouri 26 Apr '24

26 Apr '24

Hi guys, I have tried to add node-exporter to the new host in ceph cluster by the command mentioned in the document. ceph orch apply node-exporter hostname I think there is a functionality issue because cephadm log print node-exporter was applied successfully, but it didn't work! I tried the below command and it worked! ceph orch daemon add node-exporter hostname Which way is the correct way?

2 1

Setup Ceph over RDMA
by Vahideh Alinouri 26 Apr '24

26 Apr '24

Hi guys, I need setup Ceph over RDMA, but I faced many issues! The info regarding my cluster: Ceph version is Reef Network cards are Broadcom RDMA. RDMA connection between OSD nodes are OK. I just found ms_type = async+rdma config in document and apply it using ceph config set global ms_type async+rdma After this action the cluster crashes. I tried to cluster back, and I did: Put ms_type async+posix in ceph.conf Restart all MON services The cluster is back, but I don't have any active mgr. All OSDs are down too. Is there any order to do for setting up Ceph over RDMA? Thanks

1 1

MDS crash
by alexey.gerasimov＠opencascade.com 26 Apr '24

26 Apr '24

Dear colleagues, hope that anybody can help us. The initial point: Ceph cluster v15.2 (installed and controlled by the Proxmox) with 3 nodes based on physical servers rented from a cloud provider. CephFS is installed also. Yesterday we discovered that some of the applications stopped working. During the investigation we recognized that we have the problem with Ceph, more precisely with СephFS - MDS daemons suddenly crashed. We tried to restart them and found that they crashed again immediately after the start. The crash information: 2024-04-17T17:47:42.841+0000 7f959ced9700 1 mds.0.29134 recovery_done -- successful recovery! 2024-04-17T17:47:42.853+0000 7f959ced9700 1 mds.0.29134 active_start 2024-04-17T17:47:42.881+0000 7f959ced9700 1 mds.0.29134 cluster recovered. 2024-04-17T17:47:43.825+0000 7f959aed5700 -1 ./src/mds/OpenFileTable.cc: In function 'void OpenFileTable::commit(MDSContext*, uint64_t, int)' thread 7f959aed5700 time 2024-04-17T17:47:43.831243+0000 ./src/mds/OpenFileTable.cc: 549: FAILED ceph_assert(count > 0) Next hours we read the tons of articles, studied the documentation, and checked the common state of Ceph cluster by the various diagnostic commands – but didn’t find anything wrong. At evening we decided to upgrade it up to v16, and finally to v17.2.7. Unfortunately, it didn’t solve the problem, MDS continue to crash with the same error. The only difference that we found is “1 MDSs report damaged metadata” in the output of ceph -s – see it below. I supposed that it may be the well-known bug, but couldn’t find the same one on https://tracker.ceph.com - there are several bugs associated with file OpenFileTable.cc but not related to ceph_assert(count > 0) We tried to check the source code of OpenFileTable.cc also, here is a fragment of it, in function OpenFileTable::_journal_finish int omap_idx = anchor.omap_idx; unsigned& count = omap_num_items.at(omap_idx); ceph_assert(count > 0); So, we guess that the object map is empty for some object in Ceph, and it is unexpected behavior. But again, we found nothing wrong in our cluster… Next, we started with https://docs.ceph.com/en/latest/cephfs/disaster-recovery-experts/ article – tried to reset the journal (despite that it was Ok all the time) and wipe the sessions using cephfs-table-tool all reset session command. No result… Now I decided to continue following this article and run cephfs-data-scan scan_extents command, it is working just now. But I have a doubt that it will solve the issue because of no problem with our objects in Ceph. Is it the new bug? or something else? Any idea is welcome! The important outputs: ----- ceph -s cluster: id: 4cd1c477-c8d0-4855-a1f1-cb71d89427ed health: HEALTH_ERR 1 MDSs report damaged metadata insufficient standby MDS daemons available 83 daemons have recently crashed 3 mgr modules have recently crashed services: mon: 3 daemons, quorum asrv-dev-stor-2,asrv-dev-stor-3,asrv-dev-stor-1 (age 22h) mgr: asrv-dev-stor-2(active, since 22h), standbys: asrv-dev-stor-1 mds: 1/1 daemons up osd: 18 osds: 18 up (since 22h), 18 in (since 29h) data: volumes: 1/1 healthy pools: 5 pools, 289 pgs objects: 29.72M objects, 5.6 TiB usage: 21 TiB used, 47 TiB / 68 TiB avail pgs: 287 active+clean 2 active+clean+scrubbing+deep io: client: 2.5 KiB/s rd, 172 KiB/s wr, 261 op/s rd, 195 op/s wr -----ceph fs dump e29480 enable_multiple, ever_enabled_multiple: 0,1 default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} legacy client fscid: 1 Filesystem 'cephfs' (1) fs_name cephfs epoch 29480 flags 12 joinable allow_snaps allow_multimds_snaps created 2022-11-25T15:56:08.507407+0000 modified 2024-04-18T16:52:29.970504+0000 tableserver 0 root 0 session_timeout 60 session_autoclose 300 max_file_size 1099511627776 required_client_features {} last_failure 0 last_failure_osd_epoch 14728 compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} max_mds 1 in 0 up {0=156636152} failed damaged stopped data_pools [5] metadata_pool 6 inline_data disabled balancer standby_count_wanted 1 [mds.asrv-dev-stor-1{0:156636152} state up:active seq 6 laggy since 2024-04-18T16:52:29.970479+0000 addr [v2:172.22.2.91:6800/2487054023,v1:172.22.2.91:6801/2487054023] compat {c=[1],r=[1],i=[7ff]}] -----cephfs-journal-tool --rank=cephfs:0 journal inspect Overall journal integrity: OK -----ceph pg dump summary version 41137 stamp 2024-04-18T21:17:59.133536+0000 last_osdmap_epoch 0 last_pg_scan 0 PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG sum 29717605 0 0 0 0 6112544251872 13374192956 28493480 1806575 1806575 OSD_STAT USED AVAIL USED_RAW TOTAL sum 21 TiB 47 TiB 21 TiB 68 TiB -----ceph pg dump pools POOLID OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG 8 31771 0 0 0 0 131337887503 2482 140 401246 401246 7 839707 0 0 0 0 3519034650971 736 61 399328 399328 6 1319576 0 0 0 0 421044421 13374189738 28493279 206749 206749 5 27526539 0 0 0 0 2461702171417 0 0 792165 792165 2 12 0 0 0 0 48497560 0 0 6991 6991

5 7

Cephadm stacktrace on copying ceph.conf
by Jesper Agerbo Krogh [JSKR] 25 Apr '24

25 Apr '24

Hi. We're currently getting these errors - and I seem to be missing a clear overview over the cause and how to debug. 3/26/24 9:38:09 PM[ERR]executing _write_files((['dkcphhpcadmin01', 'dkcphhpcmgt028', 'dkcphhpcmgt029', 'dkcphhpcmgt031', 'dkcphhpcosd033', 'dkcphhpcosd034', 'dkcphhpcosd035', 'dkcphhpcosd036', 'dkcphhpcosd037', 'dkcphhpcosd038', 'dkcphhpcosd039', 'dkcphhpcosd040', 'dkcphhpcosd041', 'dkcphhpcosd042', 'dkcphhpcosd043', 'dkcphhpcosd044'],)) failed. Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/ssh.py", line 240, in _write_remote_file await asyncssh.scp(f.name, (conn, tmp_path)) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 922, in scp await source.run(srcpath) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 458, in run self.handle_error(exc) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in handle_error raise exc from None File "/lib/python3.6/site-packages/asyncssh/scp.py", line 456, in run await self._send_files(path, b'') File "/lib/python3.6/site-packages/asyncssh/scp.py", line 438, in _send_files self.handle_error(exc) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in handle_error raise exc from None File "/lib/python3.6/site-packages/asyncssh/scp.py", line 434, in _send_files await self._send_file(srcpath, dstpath, attrs) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 365, in _send_file await self._make_cd_request(b'C', attrs, size, srcpath) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 343, in _make_cd_request self._fs.basename(path)) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 224, in make_request raise exc asyncssh.sftp.SFTPFailure: scp: /tmp/var/lib/ceph/5c384430-da91-11ed-af9c-c780a5227aff/config/ceph.conf.new: Permission denied During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/utils.py", line 79, in do_work return f(*arg) File "/usr/share/ceph/mgr/cephadm/serve.py", line 1088, in _write_files self._write_client_files(client_files, host) File "/usr/share/ceph/mgr/cephadm/serve.py", line 1107, in _write_client_files self.mgr.ssh.write_remote_file(host, path, content, mode, uid, gid) File "/usr/share/ceph/mgr/cephadm/ssh.py", line 261, in write_remote_file host, path, content, mode, uid, gid, addr)) File "/usr/share/ceph/mgr/cephadm/module.py", line 615, in wait_async return self.event_loop.get_result(coro) File "/usr/share/ceph/mgr/cephadm/ssh.py", line 56, in get_result return asyncio.run_coroutine_threadsafe(coro, self._loop).result() File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result return self.__get_result() File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result raise self._exception File "/usr/share/ceph/mgr/cephadm/ssh.py", line 249, in _write_remote_file raise OrchestratorError(msg) orchestrator._interface.OrchestratorError: Unable to write dkcphhpcmgt028:/var/lib/ceph/5c384430-da91-11ed-af9c-c780a5227aff/config/ceph.conf: scp: /tmp/var/lib/ceph/5c384430-da91-11ed-af9c-c780a5227aff/config/ceph.conf.new: Permission denied 3/26/24 9:38:09 PM[ERR]Unable to write dkcphhpcmgt028:/var/lib/ceph/5c384430-da91-11ed-af9c-c780a5227aff/config/ceph.conf: scp: /tmp/var/lib/ceph/5c384430-da91-11ed-af9c-c780a5227aff/config/ceph.conf.new: Permission denied Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/ssh.py", line 240, in _write_remote_file await asyncssh.scp(f.name, (conn, tmp_path)) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 922, in scp await source.run(srcpath) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 458, in run self.handle_error(exc) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in handle_error raise exc from None File "/lib/python3.6/site-packages/asyncssh/scp.py", line 456, in run await self._send_files(path, b'') File "/lib/python3.6/site-packages/asyncssh/scp.py", line 438, in _send_files self.handle_error(exc) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 307, in handle_error raise exc from None File "/lib/python3.6/site-packages/asyncssh/scp.py", line 434, in _send_files await self._send_file(srcpath, dstpath, attrs) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 365, in _send_file await self._make_cd_request(b'C', attrs, size, srcpath) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 343, in _make_cd_request self._fs.basename(path)) File "/lib/python3.6/site-packages/asyncssh/scp.py", line 224, in make_request raise exc asyncssh.sftp.SFTPFailure: scp: /tmp/var/lib/ceph/5c384430-da91-11ed-af9c-c780a5227aff/config/ceph.conf.new: Permission denied 3/26/24 9:38:09 PM[INF]Updating dkcphhpcmgt028:/var/lib/ceph/5c384430-da91-11ed-af9c-c780a5227aff/config/ceph.conf It seem to be related to the permissions that the manager writes the files with and the process copying them around. $ sudo ceph -v [sudo] password for adminjskr: ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) Best regards, Jesper Agerbo Krogh Director Digitalization Digitalization Topsoe A/S Haldor Topsøes Allé 1 2800 Kgs. Lyngby Denmark Phone (direct): 27773240     Read more attopsoe.com Topsoe A/S and/or its affiliates. This e-mail message (including attachments, if any) is confidential and may be privileged. It is intended only for the addressee. Any unauthorised distribution or disclosure is prohibited. Disclosure to anyone other than the intended recipient does not constitute waiver of privilege. If you have received this email in error, please notify the sender by email and delete it and any attachments from your computer system and records.

2 1

Re: Recoveries without any misplaced objects?
by Hector Martin 24 Apr '24

24 Apr '24

On 29/05/2023 20.55, Anthony D'Atri wrote: > Check the uptime for the OSDs in question I restarted all my OSDs within the past 10 days or so. Maybe OSD restarts are somehow breaking these stats? > >> On May 29, 2023, at 6:44 AM, Hector Martin <marcan(a)marcan.st> wrote: >> >> Hi, >> >> I'm watching a cluster finish a bunch of backfilling, and I noticed that >> quite often PGs end up with zero misplaced objects, even though they are >> still backfilling. >> >> Right now the cluster is down to 6 backfilling PGs: >> >> data: >> volumes: 1/1 healthy >> pools: 6 pools, 268 pgs >> objects: 18.79M objects, 29 TiB >> usage: 49 TiB used, 25 TiB / 75 TiB avail >> pgs: 262 active+clean >> 6 active+remapped+backfilling >> >> But there are no misplaced objects, and the misplaced column in `ceph pg >> dump` is zero for all PGs. >> >> If I do a `ceph pg dump_json`, I can see `num_objects_recovered` >> increasing for these PGs... but the misplaced count is still 0. >> >> Is there something else that would cause recoveries/backfills other than >> misplaced objects? Or perhaps there is a bug somewhere causing the >> misplaced object count to be misreported as 0 sometimes? >> >> # ceph -v >> ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy >> (stable) >> >> - Hector >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > - Hector

3 3

Slow/blocked reads and writes
by Fábio Sato 24 Apr '24

24 Apr '24

Hello all, I am trying to troubleshoot a ceph cluster version 18.2.2 having users reporting slow and blocked reads and writes. When running "ceph status" I am seeing many warnings about its health state: cluster: id: cc881230-e0dd-11ee-aa9e-37c4e4e5e14b health: HEALTH_WARN 6 clients failing to respond to capability release 2 clients failing to advance oldest client/flush tid 1 MDSs report slow requests 1 MDSs behind on trimming Too many repaired reads on 11 OSDs Degraded data redundancy: 2 pgs degraded 105 pgs not deep-scrubbed in time 109 pgs not scrubbed in time 1 mgr modules have recently crashed 12 slow ops, oldest one blocked for 97678 sec, daemons [osd.11,osd.12,osd.15,osd.16,osd.19,osd.20,osd.28,osd.3,osd.32,osd.34]... have slow ops. services: mon: 3 daemons, quorum file03-xx,file04-xx,file05-xx (age 17h) mgr: file03-xx.xxxxxx(active, since 2w), standbys: file04-xx.xxxxxx mds: 1/1 daemons up, 1 standby osd: 44 osds: 44 up (since 17h), 44 in (since 39h); 492 remapped pgs data: volumes: 1/1 healthy pools: 3 pools, 2065 pgs objects: 66.44M objects, 140 TiB usage: 281 TiB used, 304 TiB / 586 TiB avail pgs: 16511162/134215883 objects misplaced (12.302%) 1508 active+clean 487 active+remapped+backfill_wait 53 active+clean+scrubbing+deep 8 active+clean+scrubbing 5 active+remapped+backfilling 2 active+recovering+degraded+repair 2 active+recovering+repair io: recovery: 47 MiB/s, 37 objects/s When checking the output of `ceph -w` I am flooded with crc error messages like the examples below: 2024-04-24T19:15:40.430334+0000 osd.32 [ERR] 3.566 full-object read crc 0xa5da7fa != expected 0xffffffff on 3:66a8d8f5:::10001c72400.00000007:head 2024-04-24T19:15:40.430507+0000 osd.39 [ERR] 3.270 full-object read crc 0xa1bc3a1e != expected 0xffffffff on 3:0e44aa2f:::1000265a625.00000003:head 2024-04-24T19:15:40.494249+0000 osd.28 [ERR] 3.469 full-object read crc 0x8e757c06 != expected 0xffffffff on 3:962852f4:::10001c72c2d.00000001:head 2024-04-24T19:15:40.529771+0000 osd.32 [ERR] 3.566 full-object read crc 0xa5da7fa != expected 0xffffffff on 3:66a8d8f5:::10001c72400.00000007:head 2024-04-24T19:15:40.582128+0000 osd.19 [ERR] 3.4b full-object read crc 0x9222aec != expected 0xffffffff on 3:d20bddca:::100026fdb01.00000006:head 2024-04-24T19:15:40.583350+0000 osd.19 [ERR] 3.4b full-object read crc 0x9222aec != expected 0xffffffff on 3:d20bddca:::100026fdb01.00000006:head 2024-04-24T19:15:40.662945+0000 osd.28 [ERR] 3.469 full-object read crc 0x8e757c06 != expected 0xffffffff on 3:962852f4:::10001c72c2d.00000001:head 2024-04-24T19:15:40.698197+0000 osd.19 [ERR] 3.4b full-object read crc 0x9222aec != expected 0xffffffff on 3:d20bddca:::100026fdb01.00000006:head 2024-04-24T19:15:40.699389+0000 osd.19 [ERR] 3.4b full-object read crc 0x9222aec != expected 0xffffffff on 3:d20bddca:::100026fdb01.00000006:head 2024-04-24T19:15:40.769191+0000 osd.28 [ERR] 3.469 full-object read crc 0x8e757c06 != expected 0xffffffff on 3:962852f4:::10001c72c2d.00000001:head 2024-04-24T19:15:40.834344+0000 osd.19 [ERR] 3.4b full-object read crc 0x9222aec != expected 0xffffffff on 3:d20bddca:::100026fdb01.00000006:head 2024-04-24T19:15:40.835513+0000 osd.19 [ERR] 3.4b full-object read crc 0x9222aec != expected 0xffffffff on 3:d20bddca:::100026fdb01.00000006:head I suspect this is the main issue affecting the cluster health state and performance so I am trying to address this first. The "expected 0xffffffff" crc seems like a bug to me and I found an open ticket (https://tracker.ceph.com/issues/53240) with similar error messages but I am not sure this is related to my case. Could someone point me to the steps to solve these errors? Cheers, -- Fabio

1 0

Orchestrator not automating services / OSD issue
by Michael Baer 24 Apr '24

24 Apr '24

Hi, This problem started with trying to add a new storage server into a quincy v17.2.6 ceph cluster. Whatever I did, I could not add the drives on the new host as OSDs: via dashboard, via cephadm shell, by setting osd unmanaged to false. But what I started realizing is that orchestrator will also no longer automatically manage services. I.e. if a service is set to manage by labels, removing and adding labels to different hosts for that service has no affect. Same if I set a service to be manage via hostnames. Same if I try to drain a host (the services/podman containers just keep running). Although, I am able to add/rm services via 'cephadm shell ceph orch daemon add/rm'. But Ceph will not manage automatically using labels/hostnames. This apparently includes OSD daemons. I can not create and on the new host either automatically or manually, but I'm hoping the services/OSD issues are related and not two issues. I haven't been able to find any obvious errors in /var/log/ceph, /var/log/syslog, logs <container>, etc. I have been able to get 'slow ops' errors on monitors by trying to add OSDs manually (and having to restart the monitor). I've also gotten cephadm shell to hang. And had to restart managers. I'm not an expert and it could be something obvious, but I haven't been able to figure out a solution. If anyone has any suggestions, I would greatly appreciate them. Thanks, Mike -- Michael Baer ceph(a)mikesoffice.com

2 2

Re: ceph-users Digest, Vol 118, Issue 85
by duluxoz 24 Apr '24

24 Apr '24

Hi Eugen, Thank you for a viable solution to our underlying issue - I'll attempt to implement it shortly. :-) However, with all the respect in world, I believe you are incorrect when you say the doco is correct (but I will be more than happy to be proven wrong). :-) The relevant text (extracted from the document page (the last couple of paragraphs)) says: ~~~ If a client already has a capability for file-system name |a| and path |dir1|, running |fs authorize| again for FS name |a| but path |dir2|, instead of modifying the capabilities client already holds, a new cap for |dir2| will be granted: cephfsauthorizeaclient.x/dir1rw cephauthgetclient.x [client.x] key = AQC1tyVknMt+JxAAp0pVnbZGbSr/nJrmkMNKqA== caps mds = "allow rw fsname=a path=/dir1" caps mon = "allow r fsname=a" caps osd = "allow rw tag cephfs data=a" cephfsauthorizeaclient.x/dir2rw updated caps for client.x cephauthgetclient.x [client.x] key = AQC1tyVknMt+JxAAp0pVnbZGbSr/nJrmkMNKqA== caps mds = "allow rw fsname=a path=dir1, allow rw fsname=a path=dir2" caps mon = "allow r fsname=a" caps osd = "allow rw tag cephfs data=a" ~~~ The above *seems* to me to say (as per the 2nd `cephauthgetclient.x` example) that a 2nd directory (dir2) *will* be added to the `client.x` authorisation. HOWEVER, this does not work in practice - hence my original query. This is what we originally attempted to do (word for word, only substituting our CechFS name for "a") and we got the error in the original post. So if the doco says that something can be done *and* gives a working example, but an end-user (admin) cannot achieve the same results but gets an error instead when following the exact same commands, then either the doco is incorrect *or* there is something else wrong. BUT your statement ("running 'ceph fs authorize' will overwrite the existing caps, it will not add more caps to the client") is in direct contradiction to the documentation ("If a client already has a capability for file-system name |a| and path |dir1|, running |fs authorize| again for FS name |a| but path |dir2|, instead of modifying the capabilities client already holds, a new cap for |dir2| will be granted"). So there's some sort of "disconnect" there. :-) Cheers On 24/04/2024 17:33, ceph-users-request(a)ceph.io wrote: > Send ceph-users mailing list submissions to > ceph-users(a)ceph.io > > To subscribe or unsubscribe via email, send a message with subject or > body 'help' to > ceph-users-request(a)ceph.io > > You can reach the person managing the list at > ceph-users-owner(a)ceph.io > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of ceph-users digest..." > > Today's Topics: > > 1. Re: Latest Doco Out Of Date? (Eugen Block) > 2. Re: stretched cluster new pool and second pool with nvme > (Eugen Block) > 3. Re: Latest Doco Out Of Date? (Frank Schilder) > > _______________________________________________ > ceph-users mailing list --ceph-users(a)ceph.io > To unsubscribe send an email toceph-users-leave(a)ceph.io > %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s

1 0

Latest Doco Out Of Date?
by duluxoz 24 Apr '24

24 Apr '24

Hi All, In reference to this page from the Ceph documentation: https://docs.ceph.com/en/latest/cephfs/client-auth/, down the bottom of that page it says that you can run the following commands: ~~~ ceph fs authorize a client.x /dir1 rw ceph fs authorize a client.x /dir2 rw ~~~ This will allow `client.x` to access both `dir1` and `dir2`. So, having a use case where we need to do this, we are, HOWEVER, getting the following error on running the 2nd command on a Reef 18.2.2 cluster: `Error EINVAL: client.x already has fs capabilities that differ from those supplied. To generate a new auth key for client.x, first remove client.x from configuration files, execute 'ceph auth rm client.x', then execute this command again.` Something we're doing wrong, or is the doco "out of date" (mind you, that's from the "latest" version of the doco, and the "reef" version), or is something else going on? Thanks in advance for the help Cheers Dulux-Oz

4 9

List of bridges irc/slack/discord
by Alvaro Soto 23 Apr '24

23 Apr '24

(Last update) - https://github.com/orgs/opensource-latinamerica/discussions/3 ~~~ Adding a few unofficial/unregistered Ceph IRC channels (cephadm, crimson) IRC -> slack.oss.lat OFTC: starlingx -> slack: starlingx OFTC: openstack-latinamerica -> slack: stack-latinamerica OFTC: openstack-freezer -> slack: stack-freezer OFTC: ceph -> slack: ceph OFTC: sepia -> slack: sepia OFTC: cephfs -> slack: cephfs OFTC: ceph-dashboard -> slack: ceph-dashboard OFTC: ceph-devel -> slack: ceph-devel OFTC:cephadm -> slack:cephadm OFTC:crimson -> slack:crimson IRC -> ceph-storage.slack.com OFTC: ceph -> slack: ceph OFTC: sepia -> slack: sepia OFTC: cephfs -> slack: cephfs OFTC: ceph-dashboard -> slack: ceph-dashboard OFTC: ceph-devel -> slack: ceph-devel OFTC:cephadm -> slack:cephadm OFTC:crimson -> slack:crimson IRC -> discord(a)ceph.io OFTC: ceph -> discord(a)ceph.io: ceph OFTC: sepia -> discord(a)ceph.io: sepia OFTC: cephfs -> discord(a)ceph.io: cephfs OFTC: ceph-dashboard -> discord(a)ceph.io: ceph-dashboard OFTC: ceph-devel -> discord(a)ceph.io: ceph-devel OFTC:cephadm -> discord@ceph.io:cephadm OFTC:crimson -> discord@ceph.io:crimson Ceph invite URL: https://discord.gg/vacj9cZSmm ~~~ Cheers! -- Alvaro Soto *Note: My work hours may not be your work hours. Please do not feel the need to respond during a time that is not convenient for you.* ---------------------------------------------------------- Great people talk about ideas, ordinary people talk about things, small people talk... about other people.

1 0

Best practice and expected benefits of using separate WAL and DB devices with Bluestore
by Niklaus Hofer 23 Apr '24

23 Apr '24

Dear all We have an HDD ceph cluster that could do with some more IOPS. One solution we are considering is installing NVMe SSDs into the storage nodes and using them as WAL- and/or DB devices for the Bluestore OSDs. However, we have some questions about this and are looking for some guidance and advice. The first one is about the expected benefits. Before we undergo the efforts involved in the transition, we are wondering if it is even worth it. How much of a performance boost one can expect when adding NVMe SSDs for WAL-devices to an HDD cluster? Plus, how much faster than that does it get with the DB also being on SSD. Are there rule-of-thumb number of that? Or maybe someone has done benchmarks in the past? The second question is of more practical nature. Are there any best-practices on how to implement this? I was thinking we won't do one SSD per HDD - surely an NVMe SSD is plenty fast to handle the traffic from multiple OSDs. But what is a good ratio? Do I have one NVMe SSD per 4 HDDs? Per 6 or even 8? Also, how should I chop-up the SSD, using partitions or using LVM? Last but not least, if I have one SSD handle WAL and DB for multiple OSDs, losing that SSD means losing multiple OSDs. How do people deal with this risk? Is it generally deemed acceptable or is this something people tend to mitigate and if so how? Do I run multiple SSDs in RAID? I do realize that for some of these, there might not be the one perfect answer that fits all use cases. I am looking for best practices and in general just trying to avoid any obvious mistakes. Any advice is much appreciated. Sincerely Niklaus Hofer -- stepping stone AG Wasserwerkgasse 7 CH-3011 Bern Telefon: +41 31 332 53 63 www.stepping-stone.ch niklaus.hofer(a)stepping-stone.ch

7 6

Status of IPv4 / IPv6 dual stack?
by Robert Sander 23 Apr '24

23 Apr '24

Hi, as the documentation sends mixed signals in https://docs.ceph.com/en/latest/rados/configuration/network-config-ref/#ipv… "Note Binding to IPv4 is enabled by default, so if you just add the option to bind to IPv6 you’ll actually put yourself into dual stack mode." and https://docs.ceph.com/en/latest/rados/configuration/msgr2/#address-formats "Note The ability to bind to multiple ports has paved the way for dual-stack IPv4 and IPv6 support. That said, dual-stack operation is not yet supported as of Quincy v17.2.0." just the quick questions: Is a dual stacked networking with IPv4 and IPv6 now supported or not? From which version on is it considered stable? Are OSDs now able to register themselves with two IP addresses in the cluster map? MONs too? Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

6 6

Why CEPH is better than other storage solutions?
by sebcio_t＠o2.pl 23 Apr '24

23 Apr '24

Hi, I have problem to answer to this question: Why CEPH is better than other storage solutions? I know this high level texts about - scalability, - flexibility, - distributed, - cost-Effectiveness What convince me, but could be received also against, is ceph as a product has everything what I need it mean: block storage (RBD), file storage (CephFS), object storage (S3, Swift) and "plugins" to run NFS, NVMe over Fabric, NFS on object storage. Also many other features which are usually sold as a option (mirroring, geo replication, etc) in paid solutions. I have problem to write it done piece by piece. I want convince my managers we are going in good direction. Why not something from robin.io or purestorage, netapp, dell/EMC. From opensource longhorn or openEBS. If you have ideas please write it. Thanks, S.

8 8

Stuck in replay?
by Erich Weiler 23 Apr '24

23 Apr '24

Hi All, We have a somewhat serious situation where we have a cephfs filesystem (18.2.1), and 2 active MDSs (one standby). ThI tried to restart one of the active daemons to unstick a bunch of blocked requests, and the standby went into 'replay' for a very long time, then RAM on that MDS server filled up, and it just stayed there for a while then eventually appeared to give up and switched to the standby, but the cycle started again. So I restarted that MDS, and now I'm in a situation where I see this: # ceph fs status slugfs - 29 clients ====== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 replay slugfs.pr-md-01.xdtppo 3958k 57.1k 12.2k 0 1 resolve slugfs.pr-md-02.sbblqq 0 3 1 0 POOL TYPE USED AVAIL cephfs_metadata metadata 997G 2948G cephfs_md_and_data data 0 87.6T cephfs_data data 773T 175T STANDBY MDS slugfs.pr-md-03.mclckv MDS version: ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable) It just stays there indefinitely. All my clients are hung. I tried restarting all MDS daemons and they just went back to this state after coming back up. Is there any way I can somehow escape this state of indefinite replay/resolve? Thanks so much! I'm kinda nervous since none of my clients have filesystem access at the moment... cheers, erich

5 8

s3 bucket policy subusers - access denied
by sinan＠turka.nl 23 Apr '24

23 Apr '24

I want to achieve the following: - Create an user - Create 2 subusers - Create 2 buckets - Apply a policy for each bucket - A subuser should only have access to its own bucket Problem: Getting a 403 AccessDenied with subuser credentials when uploading files. I did the following: radosgw-admin user create --uid=foo-user --display_name="Foo Test User" radosgw-admin subuser create --uid=foo-user --gen-access-key --gen-secret --key-type=s3 --subuser=foo-user-subuser radosgw-admin subuser create --uid=foo-user --gen-access-key --gen-secret --key-type=s3 --subuser=foo-user-subuser2 Resulting in: { "user_id": "foo-user", "display_name": "Foo Test User", "email": "", "suspended": 0, "max_buckets": 1000, "subusers": [ { "id": "foo-user:foo-user-subuser", "permissions": "<none>" }, { "id": "foo-user:foo-user-subuser2", "permissions": "<none>" } ], "keys": [ { "user": "foo-user:foo-user-subuser", "access_key": "<key>", "secret_key": "<key>" }, { "user": "foo-user:foo-user-subuser2", "access_key": "<key>", "secret_key": "<key>" }, { "user": "foo-user", "access_key": "<key>", "secret_key": "<key>" } ], "swift_keys": [], "caps": [], "op_mask": "read, write, delete", "default_placement": "", "default_storage_class": "", "placement_tags": [], "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "user_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "temp_url_keys": [], "type": "rgw", "mfa_ids": [] } Using the credentials of the main account (user: foo-user) creating buckets and setting policies: s3cmd mb s3://foo-bucket s3cmd mb s3://foo-bucket2 s3cmd setpolicy foo-test-subuser-policy s3://foo-bucket s3cmd setpolicy foo-test-subuser2-policy s3://foo-bucket2 Resulting in (I am showing just foo-bucket, but the same goes for foo-bucket2): # s3cmd info s3://foo-bucket s3://foo-bucket/ (bucket): Payer: BucketOwner Ownership: none Versioning:none Expiration rule: none Block Public Access: none Policy: { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": [ "arn:aws:iam:::user/foo-user:foo-user-subuser" ] }, "Action": [ "s3:AbortMultipartUpload", "s3:DeleteObject", "s3:GetObject", "s3:ListBucketMultipartUploads", "s3:ListBucket", "s3:ListMultipartUploadParts", "s3:PutObject" ], "Resource": [ "arn:aws:s3:::foo-bucket" ] } ] } CORS: none ACL: Foo Test User: FULL_CONTROL When I try to upload files (using the subuser foo-user-subuser credentials) it doesn't work: # s3cmd ls 2024-04-23 06:59 s3://foo-bucket 2024-04-23 10:05 s3://foo-bucket2 # s3cmd put ~/Documents/file_2.txt s3://foo-bucket upload: '/home/foo/Documents/file_2.txt' -> 's3://foo-bucket/file_2.txt' [1 of 1] 10 of 10 100% in 0s 18.96 B/s done ERROR: S3 error: 403 (AccessDenied) What is wrong with my policy? I thought that I did exactly the same earlier and it worked, but I am in doubt now.... Thanks!

1 0

ceph api rgw/role
by farhad kh 23 Apr '24

23 Apr '24

hi , i used ceph api for create rgw/role but ther is not api for delete or edit rgw/role . how can i delete them or edit ?

2 1

Question about PR merge
by Erich Weiler 23 Apr '24

23 Apr '24

Hello, We are tracking PR #56805: https://github.com/ceph/ceph/pull/56805 And the resolution of this item would potentially fix a pervasive and ongoing issue that needs daily attention in our cephfs cluster. I was wondering if it would be included in 18.2.3 which I *think* should be released soon? Is there any way of knowing if that is true? Thanks again, erich

4 6

RGWs stop processing requests after upgrading to Reef
by Iain Stott 22 Apr '24

22 Apr '24

Hi, We have recently upgraded one of our clusters from Quincy 17.2.6 to Reef 18.2.1, since then we have had 3 instances of our RGWs stop processing requests. We have 3 hosts that run a single instance of RGW on each, and all 3 just seem to stop processing requests at the same time causing our storage to become unavailable. A restart or redeploy of the RGW service brings them back ok. The cluster was deployed using ceph ansible, but since we have adopted it to cephadm which is how the upgrade was performed. We have enabled debug logging as there was nothing out of the ordinary in normal logs and are currently sifting through them from the last crash. We are just wondering if it possible to run Quincy RGWs instead of Reef as we didn't have this issue prior to the upgrade? We have 3 clusters in a multisite setup, we are holding off on upgrading the other 2 clusters due to this issue. Thanks Iain Iain Stott OpenStack Engineer Iain.Stott(a)thg.com [THG Ingenuity Logo]<https://www.thg.com> www.thg.com<https://www.thg.com/> [LinkedIn]<https://www.linkedin.com/company/thgplc/?originalSubdomain=uk> [Instagram] <https://www.instagram.com/thg> [X] <https://twitter.com/thgplc?lang=en>

2 2

Multiple MDS Daemon needed?
by Erich Weiler 22 Apr '24

22 Apr '24

Hi All, We have a slurm cluster with 25 clients, each with 256 cores, each mounting a cephfs filesystem as their main storage target. The workload can be heavy at times. We have two active MDS daemons and one standby. A lot of the time everything is healthy but we sometimes get warnings about MDS daemons being slow on requests, behind on trimming, etc. I realize their may be a bug in play, but also, I was wondering if we simply didn't have enough MDS daemons to handle the load. Is there a way to know if adding a MDS daemon would help? We could add a third active MDS if needed. But I don't want to start adding a bunch of MDS's if that won't help. The OSD servers seem fine. It's mainly the MDS instances that are complaining. We are running reef 18.2.1. For reference, when things look healthy: # ceph fs status slugfs slugfs - 34 clients ====== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active slugfs.pr-md-03.mclckv Reqs: 273 /s 2759k 2636k 362k 1079k 1 active slugfs.pr-md-01.xdtppo Reqs: 194 /s 868k 674k 67.3k 351k POOL TYPE USED AVAIL cephfs_metadata metadata 127G 3281G cephfs_md_and_data data 0 98.3T cephfs_data data 740T 196T STANDBY MDS slugfs.pr-md-02.sbblqq MDS version: ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable) # ceph -s cluster: id: 58bde08a-d7ed-11ee-9098-506b4b4da440 health: HEALTH_OK services: mon: 5 daemons, quorum pr-md-01,pr-md-02,pr-store-01,pr-store-02,pr-md-03 (age 5d) mgr: pr-md-01.jemmdf(active, since 5w), standbys: pr-md-02.emffhz mds: 2/2 daemons up, 1 standby osd: 46 osds: 46 up (since 8d), 46 in (since 4w) data: volumes: 1/1 healthy pools: 4 pools, 1313 pgs objects: 271.17M objects, 493 TiB usage: 744 TiB used, 384 TiB / 1.1 PiB avail pgs: 1307 active+clean 4 active+clean+scrubbing 2 active+clean+scrubbing+deep io: client: 39 MiB/s rd, 108 MiB/s wr, 1.96k op/s rd, 54 op/s wr But when things are in "warning" mode, it looks like this: # ceph -s cluster: id: 58bde08a-d7ed-11ee-9098-506b4b4da440 health: HEALTH_WARN 1 filesystem is degraded 1 clients failing to advance oldest client/flush tid 1 MDSs report slow requests 1 MDSs behind on trimming services: mon: 5 daemons, quorum pr-md-01,pr-md-02,pr-store-01,pr-store-02,pr-md-03 (age 5d) mgr: pr-md-01.jemmdf(active, since 5w), standbys: pr-md-02.emffhz mds: 2/2 daemons up, 1 standby osd: 46 osds: 46 up (since 8d), 46 in (since 4w) data: volumes: 1/1 healthy pools: 4 pools, 1313 pgs objects: 271.28M objects, 494 TiB usage: 746 TiB used, 382 TiB / 1.1 PiB avail pgs: 1307 active+clean 5 active+clean+scrubbing 1 active+clean+scrubbing+deep io: client: 55 MiB/s rd, 2.6 MiB/s wr, 15 op/s rd, 46 op/s wr And this: # ceph health detail HEALTH_WARN 2 clients failing to advance oldest client/flush tid; 2 MDSs report slow requests; 1 MDSs behind on trimming [WRN] MDS_CLIENT_OLDEST_TID: 2 clients failing to advance oldest client/flush tid mds.slugfs.pr-md-01.xdtppo(mds.0): Client phoenix-06.prism failing to advance its oldest client/flush tid. client_id: 125780 mds.slugfs.pr-md-02.sbblqq(mds.1): Client phoenix-00.prism failing to advance its oldest client/flush tid. client_id: 99385 [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests mds.slugfs.pr-md-01.xdtppo(mds.0): 4 slow requests are blocked > 30 secs mds.slugfs.pr-md-02.sbblqq(mds.1): 67 slow requests are blocked > 30 secs [WRN] MDS_TRIM: 1 MDSs behind on trimming mds.slugfs.pr-md-02.sbblqq(mds.1): Behind on trimming (109410/250) max_segments: 250, num_segments: 109410 The "cure" is the restart the active MDS daemons, one at a time. Then everything becomes healthy again, for a time. We also have the following MDS config items in play: mds_cache_memory_limit = 8589934592 mds_cache_trim_decay_rate = .6 mds_log_max_segments = 250 Thanks for any pointers! cheers, erich

2 1

Re: Best practice and expected benefits of using separate WAL and DB devices with Bluestore
by Darren Soothill 21 Apr '24

21 Apr '24

Hi Niklaus, Lots of questions here but let me tray and get through some of them. Personally unless a cluster is for deep archive then I would never suggest configuring or deploying a cluster without Rocks DB and WAL on NVME. There are a number of benefits to this in terms of performance and recovery. Small writes go to the NVME first before being written to the HDD and it makes many recovery operations far more efficient. As to how much faster it makes things that very much depends on the type of workload you have on the system. Lots of small writes will make a significant difference. Very large writes not as much of a difference. Things like compactions of the RocksDB database are a lot faster as they are now running from NVME and not from the HDD. We normally work with a upto 1:12 ratio so 1 NVME for every 12 HDD’s. This is assuming the NVME’s being used are good mixed use enterprise NVME’s with power loss protection. As to failures yes a failure of the NVME would mean a loss of 12 OSD’s but this is no worse than a failure of an entire node. This is something Ceph is designed to handle. I certainly wouldn’t be thinking about putting the NVME’s into raid sets as that will degrade the performance of them when you are trying to get better performance. Darren Soothill Looking for help with your Ceph cluster? Contact us at https://croit.io/ croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io/ | YouTube: https://goo.gl/PGE1Bx

3 3

RGW: Cannot write to bucket anymore
by Malte Stroem 21 Apr '24

21 Apr '24

Hello, there is one bucket for a user in our Ceph cluster who is suddenly not able to write to one of his buckets. Reading works fine. All other buckets work fine. If we copy the bucket to another bucket on the same cluster, the error stays. Writing is not possible in the new bucket, too. Interesting: If we copy the contents of the bucket to a bucket in another Ceph cluster the error is gone. So now we know how to solve this but we do not finde the root cause. I checked the policies, lifecycle and versioning. Nothing. The user has FULL_CONTROL. Same settings for the user's other buckets he can still write to. Wenn setting debugging to higher numbers all I can see is something like this while trying to write to the bucket: s3:put_obj reading permissions s3:put_obj init op s3:put_obj verifying op mask s3:put_obj verifying op permissions op->ERRORHANDLER: err_no=-13 new_err_no=-13 cache get: name=default.rgw.log++script.postrequest. : hit (negative entry) s3:put_obj op status=0 s3:put_obj http status=403 1 ====== req done req=0x7fe8bb60a710 op status=0 http_status=403 latency=0.000000000s ====== I still think there is something with a policy or so. When we copy the bucket to another bucket in the same cluster, at first, while copying you can write to the new bucket but when the copy job progresses at one point writing is not possible anymore. But what is it? Best, Malte

2 4

Upgrading Ceph 15 to 18
by Malte Stroem 21 Apr '24

21 Apr '24

Hello, we'd like to upgrade our cluster from the latest Ceph 15 to Ceph 18. It's running with cephadm. What's the right way to do it? Latest Ceph 15 to latest 16 and then to the latest 17 and then the latest 18? Does that work? Or is it possible to jump from the latest Ceph 16 to the latest Ceph 18? Latest Ceph 15 -> latest Ceph 16 -> latest Ceph 18. Best, Malte

3 4

MDS daemons crash
by Alexey GERASIMOV 21 Apr '24

21 Apr '24

Hello all! Hope that anybody can help us. The initial point: Ceph cluster v15.2 (installed and controlled by the Proxmox) with 3 nodes based on physical servers rented from a cloud provider. The volumes provided by Ceph using CephFS and RBD also. We run 2 MDS daemons but use max_mds=1 so one daemon was in active state, and another in standby. On Thursday some of the applications stopped working. After the investigation it was clear that we have a problem with Ceph, more precisely with СephFS - both MDS daemons suddenly crashed. We tried to restart them and found that they crashed again immediately after the start. The crash information: 2024-04-17T17:47:42.841+0000 7f959ced9700 1 mds.0.29134 recovery_done -- successful recovery! 2024-04-17T17:47:42.853+0000 7f959ced9700 1 mds.0.29134 active_start 2024-04-17T17:47:42.881+0000 7f959ced9700 1 mds.0.29134 cluster recovered. 2024-04-17T17:47:43.825+0000 7f959aed5700 -1 ./src/mds/OpenFileTable.cc: In function 'void OpenFileTable::commit(MDSContext*, uint64_t, int)' thread 7f959aed5700 time 2024-04-17T17:47:43.831243+0000 ./src/mds/OpenFileTable.cc: 549: FAILED ceph_assert(count > 0) Next hours we read tons of articles, studied the documentation, and checked the cluster status in general by the various diagnostic commands - but didn't find anything wrong. At evening we decided to upgrade our Ceph cluster; so, we upgraded it to v16, and finally to v17.2.7. Unfortunately, it didn't solve the problem, MDS continue to crash with the same error. The only difference that we found is the "1 MDSs report damaged metadata" in the output of ceph -s - see it below. I supposed that it may be the well-known bug, but couldn't find the same one on https://tracker.ceph.com - there are several bugs associated with file OpenFileTable.cc but not related to ceph_assert(count > 0) We tried to check the source code of OpenFileTable.cc also, here is a fragment of it, in function OpenFileTable::_journal_finish int omap_idx = anchor.omap_idx; unsigned& count = omap_num_items.at(omap_idx); ceph_assert(count > 0); So, we guess that the object map is empty for some object in Ceph, and it is unexpected behavior. But again, we found nothing wrong in our cluster... Next, we started with https://docs.ceph.com/en/latest/cephfs/disaster-recovery-experts/ article - tried to reset the journal (despite that it was Ok all the time) and wipe the sessions using cephfs-table-tool all reset session command. No result... Now I decided to continue following this article and run cephfs-data-scan scan_extents command, we started it on Friday but it is still working (2 from 3 workers finished, so I'm waiting for the last one; may be I need more workers for the next command cephfs-data-scan scan_inodes that I plan to run ). But I have a doubt that it will solve the issue because, again, we guess that we have no problem with our objects in Ceph but with metadata only... Is it the new bug? or something else? What should we try additionally to run our MDS daemon? Any idea is welcome! The important outputs: ceph -s cluster: id: 4cd1c477-c8d0-4855-a1f1-cb71d89427ed health: HEALTH_ERR 1 MDSs report damaged metadata insufficient standby MDS daemons available 83 daemons have recently crashed 3 mgr modules have recently crashed services: mon: 3 daemons, quorum asrv-dev-stor-2,asrv-dev-stor-3,asrv-dev-stor-1 (age 22h) mgr: asrv-dev-stor-2(active, since 22h), standbys: asrv-dev-stor-1 mds: 1/1 daemons up osd: 18 osds: 18 up (since 22h), 18 in (since 29h) data: volumes: 1/1 healthy pools: 5 pools, 289 pgs objects: 29.72M objects, 5.6 TiB usage: 21 TiB used, 47 TiB / 68 TiB avail pgs: 287 active+clean 2 active+clean+scrubbing+deep io: client: 2.5 KiB/s rd, 172 KiB/s wr, 261 op/s rd, 195 op/s wr ceph fs dump e29480 enable_multiple, ever_enabled_multiple: 0,1 default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} legacy client fscid: 1 Filesystem 'cephfs' (1) fs_name cephfs epoch 29480 flags 12 joinable allow_snaps allow_multimds_snaps created 2022-11-25T15:56:08.507407+0000 modified 2024-04-18T16:52:29.970504+0000 tableserver 0 root 0 session_timeout 60 session_autoclose 300 max_file_size 1099511627776 required_client_features {} last_failure 0 last_failure_osd_epoch 14728 compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} max_mds 1 in 0 up {0=156636152} failed damaged stopped data_pools [5] metadata_pool 6 inline_data disabled balancer standby_count_wanted 1 [mds.asrv-dev-stor-1{0:156636152} state up:active seq 6 laggy since 2024-04-18T16:52:29.970479+0000 addr [v2:172.22.2.91:6800/2487054023,v1:172.22.2.91:6801/2487054023] compat {c=[1],r=[1],i=[7ff]}] cephfs-journal-tool --rank=cephfs:0 journal inspect Overall journal integrity: OK ceph pg dump summary version 41137 stamp 2024-04-18T21:17:59.133536+0000 last_osdmap_epoch 0 last_pg_scan 0 PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG sum 29717605 0 0 0 0 6112544251872 13374192956 28493480 1806575 1806575 OSD_STAT USED AVAIL USED_RAW TOTAL sum 21 TiB 47 TiB 21 TiB 68 TiB ceph pg dump pools POOLID OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG 8 31771 0 0 0 0 131337887503 2482 140 401246 401246 7 839707 0 0 0 0 3519034650971 736 61 399328 399328 6 1319576 0 0 0 0 421044421 13374189738 28493279 206749 206749 5 27526539 0 0 0 0 2461702171417 0 0 792165 792165 2 12 0 0 0 0 48497560 0 0 6991 6991 --- Best regards, Alexey Gerasimov System Manager www.opencascade.com<http://www.opencascade.com/> www.capgemini.com<http://www.capgemini.com> [cid:image001.png@01DA93D0.B001CE80]

1 0

Working ceph cluster reports large amount of pgs in state unknown/undersized and objects degraded
by Tobias Langner 20 Apr '24

20 Apr '24

We operate a tiny ceph cluster (v16.2.7) across three machines, each running two OSDs and one of each mds, mgr, and mon. The cluster serves one main erasure-coded (2+1) storage pool and a few other management-related pools. The cluster has been running smoothly for several months. A few weeks ago we noticed a health warning reporting backfillfull/nearfull osds and pools. Here is the output of `ceph -s` at this point (extraced from logs): -------------------------------------------------------------------------------- cluster: health: HEALTH_WARN 1 backfillfull osd(s) 2 nearfull osd(s) Reduced data availability: 163 pgs inactive, 1 pg peering Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull Degraded data redundancy: 1486709/10911157 objects degraded (13.626%), 68 pgs degraded, 68 pgs undersized 162 pgs not scrubbed in time 6 pool(s) backfillfull services: mon: 3 daemons, quorum mon.101,mon.102,mon.100 (age 5m) mgr: mgr-102(active, since 54m), standbys: mgr-101, mgr-100 mds: 1/1 daemons up, 1 standby, 1 hot standby osd: 6 osds: 6 up (since 4m), 6 in (since 2w); 7 remapped pgs data: volumes: 1/1 healthy pools: 6 pools, 338 pgs objects: 3.64M objects, 14 TiB usage: 13 TiB used, 1.7 TiB / 15 TiB avail pgs: 47.929% pgs unknown 0.296% pgs not active 1486709/10911157 objects degraded (13.626%) 52771/10911157 objects misplaced (0.484%) 162 unknown 106 active+clean 67 active+undersized+degraded 1 active+undersized+degraded+remapped+backfill_toofull 1 remapped+peering 1 active+remapped+backfill_toofull -------------------------------------------------------------------------------- I now see the large amount of pgs in state unknown and the fact that a significant fraction of objects is degraded despite all osds being up, but we didn't notice this back then. Because the cluster continued to act fine from the perspective of the mounted filesystem, we didn't really notice the potential problem and did not intervene. From then one, things have mostly gone downwards. Now, `ceph -s` reports the following: -------------------------------------------------------------------------------- cluster: health: HEALTH_WARN noout flag(s) set Reduced data availability: 117 pgs inactive Degraded data redundancy: 2095625/12121767 objects degraded (17.288%), 114 pgs degraded, 114 pgs undersized 117 pgs not scrubbed in time services: mon: 3 daemons, quorum mon.101,mon.102,mon.100 (age 15h) mgr: mgr-102(active, since 7d), standbys: mgr-100, mgr-101 mds: 1/1 daemons up, 1 standby, 1 hot standby osd: 6 osds: 6 up (since 55m), 6 in (since 3w) flags noout data: volumes: 1/1 healthy pools: 6 pools, 338 pgs objects: 4.04M objects, 15 TiB usage: 12 TiB used, 2.8 TiB / 15 TiB avail pgs: 34.615% pgs unknown 2095625/12121767 objects degraded (17.288%) 117 unknown 114 active+undersized+degraded 107 active+clean -------------------------------------------------------------------------------- Note in particular the still very large number of pgs in state unknown, which hasn't changed in days. Same goes for the degraded pgs. Also, the cluster should have around 37TiB storage available but now it only reports 15 TiB. We did a bit of digging around but couldn't really get to the bottom of the unknown pgs and how we can recover from that. One other data point is that the command `ceph osd df tree` gets stuck on two of the three machines and one the one where it returns something, it looks like this: -------------------------------------------------------------------------------- ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 47.67506 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - root default -13 18.26408 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - datacenter dc.100 -5 18.26408 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - host osd-100 3 hdd 10.91409 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 0 91 up osd.3 5 hdd 7.34999 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 0 48 up osd.5 -9 14.69998 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - datacenter dc.101 -7 14.69998 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - host osd-101 0 hdd 7.34999 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 0 83 up osd.0 1 hdd 7.34999 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 0 86 up osd.1 -11 14.71100 - 15 TiB 12 TiB 12 TiB 77 MiB 21 GiB 2.6 TiB 82.00 1.00 - datacenter dc.102 -17 7.35550 - 7.4 TiB 6.3 TiB 6.2 TiB 16 MiB 11 GiB 1.1 TiB 85.16 1.04 - host osdroid-102-1 4 hdd 7.35550 1.00000 7.4 TiB 6.3 TiB 6.2 TiB 16 MiB 11 GiB 1.1 TiB 85.16 1.04 114 up osd.4 -15 7.35550 - 7.4 TiB 5.8 TiB 5.7 TiB 61 MiB 10 GiB 1.6 TiB 78.83 0.96 - host osdroid-102-2 2 hdd 7.35550 1.00000 7.4 TiB 5.8 TiB 5.7 TiB 61 MiB 10 GiB 1.6 TiB 78.83 0.96 107 up osd.2 TOTAL 15 TiB 12 TiB 12 TiB 77 MiB 21 GiB 2.6 TiB 82.00 MIN/MAX VAR: 0/1.04 STDDEV: 66.97 -------------------------------------------------------------------------------- The odd part here is that for some reason only osd.2 and osd.4 seem to contribute size to the cluster. Interestingly, accessing content from the storage pool works mostly without issues, which shouldn't work if 4 out of 6 OSDs weren't properly up. Even more odd is that while `ceph health detail` reports a lot of pgs in state unknown, undersized, and degraded, inspecting the respective pgs with `ceph pg <pdid> query` results in active+clean for *all* of them... I'm not sure which of the two pieces of information I am supposed to trust... Any ideas what we can do to get our cluster back into a sane state? I'm happy to provide more logs or command output, please let me know. Thanks!

4 5

Ceph image delete error - NetHandler create_socket couldnt create socket
by Pardhiv Karri 19 Apr '24

19 Apr '24

Hi, Trying to delete images in a Ceph pool is causing errors in one of the clusters. I rebooted all the monitor nodes sequentially to see if the error went away, but it still persists. What is the best way to fix this? The Ceph cluster is in an OK state, with no rebalancing or scrubbing happening (I did set the noscrub and deep-noscrub flags) and also no load on the cluster, very few IO. root@ceph-mon01 ~# rbd rm 000dca3d-4f2b-4033-b8f5-95458e0c3444_disk_delete -p compute Removing image: 31% complete...2024-04-18 20:42:52.525135 7f6de0c79700 -1 NetHandler create_socket couldn't create socket (24) Too many open files Removing image: 32% complete...2024-04-18 20:42:52.539882 7f6de9c7b700 -1 NetHandler create_socket couldn't create socket (24) Too many open files 2024-04-18 20:42:52.541508 7f6de947a700 -1 NetHandler create_socket couldn't create socket (24) Too many open files 2024-04-18 20:42:52.546613 7f6de0c79700 -1 NetHandler create_socket couldn't create socket (24) Too many open files 2024-04-18 20:42:52.558133 7f6de9c7b700 -1 NetHandler create_socket couldn't create socket (24) Too many open files 2024-04-18 20:42:52.573819 7f6de947a700 -1 NetHandler create_socket couldn't create socket (24) Too many open files 2024-04-18 20:42:52.589733 7f6de0c79700 -1 NetHandler create_socket couldn't create socket (24) Too many open files Removing image: 33% complete...2024-04-18 20:42:52.643489 7f6de9c7b700 -1 NetHandler create_socket couldn't create socket (24) Too many open files 2024-04-18 20:42:52.727262 7f6de0c79700 -1 NetHandler create_socket couldn't create socket (24) Too many open files 2024-04-18 20:42:52.737135 7f6de9c7b700 -1 NetHandler create_socket couldn't create socket (24) Too many open files 2024-04-18 20:42:52.743292 7f6de947a700 -1 NetHandler create_socket couldn't create socket (24) Too many open files 2024-04-18 20:42:52.746167 7f6de0c79700 -1 NetHandler create_socket couldn't create socket (24) Too many open files 2024-04-18 20:42:52.757404 7f6de9c7b700 -1 NetHandler create_socket couldn't create socket (24) Too many open files Removing image: 34% complete...2024-04-18 20:42:52.773182 7f6de947a700 -1 NetHandler create_socket couldn't create socket (24) Too many open files 2024-04-18 20:42:52.773222 7f6de947a700 -1 NetHandler create_socket couldn't create socket (24) Too many open files 2024-04-18 20:42:52.789847 7f6de0c79700 -1 NetHandler create_socket couldn't create socket (24) Too many open files 2024-04-18 20:42:52.844201 7f6de9c7b700 -1 NetHandler create_socket couldn't create socket (24) Too many open files ^C root@ceph-mon01 ~# Thanks, Pardh

3 4

RGWs stop processing requests after upgrading to Reef
by Iain Stott 18 Apr '24

18 Apr '24

Hi, We have recently upgraded one of our clusters from Quincy 17.2.6 to Reef 18.2.1, since then we have had 3 instances of our RGWs stop processing requests. We have 3 hosts that run a single instance of RGW on each, and all 3 just seem to stop processing requests at the same time causing our storage to become unavailable. A restart or redeploy of the RGW service brings them back ok. The cluster was deployed using ceph ansible, but since we have adopted it to cephadm which is how the upgrade was performed. We have enabled debug logging as there was nothing out of the ordinary in normal logs and are currently sifting through them from the last crash. We are just wondering if it possible to run Quincy RGWs instead of Reef as we didn't have this issue prior to the upgrade? We have 3 clusters in a multisite setup, we are holding off on upgrading the other 2 clusters due to this issue. Thanks Iain Iain Stott OpenStack Engineer Iain.Stott(a)thg.com [THG Ingenuity Logo]<https://www.thg.com> www.thg.com<https://www.thg.com/> [LinkedIn]<https://www.linkedin.com/company/thgplc/?originalSubdomain=uk> [Instagram] <https://www.instagram.com/thg> [X] <https://twitter.com/thgplc?lang=en>

1 0

Prevent users to create buckets
by sinan＠turka.nl 18 Apr '24

18 Apr '24

Hello, I am using Ceph RGW for S3. Is it possible to create (sub)users that cannot create/delete buckets and are limited to specific buckets? At the end, I want to create 3 separate users and for each user I want to create a bucket. The users should only have access to their own bucket and should not be able to create new or delete buckets. One approach could be to limit the max_buckets to 1 so the user cannot create new buckets, but it will still have access to other buckets and will able to delete buckets. Any advice here? Thanks! Sinan

4 3

crushmap history
by Blair Bethwaite 18 Apr '24

18 Apr '24

Hi all, Do the Mons store any crushmap history, and if so how does one get at it please? I ask because we've recently encountered an issue in a medium scale (~5PB raw) EC based RGW focused cluster where "something" happened, which we still don't know, that suddenly caused us to see 94% of objects (5.4 billion of them) misplaced. We've tracked down the first log message of that pgmap state change: Mar 29 10:30:31 mon1 bash\[5804\]: debug 2024-03-29T10:30:31.152+0000 7f3b6e378700 0 log\_channel(cluster) log \[DBG\] : pgmap v44327: 2273 pgs: 225 active+clean, 2038 active+remapped+backfill\_wait, 10 active+remapped+backfilling; 1.6 PiB data, 2.1 PiB used, 2.2 PiB / 4.3 PiB avail; 5426274136/5752755429 objects misplaced (94.325%); 248 MiB/s, 109 objects/s recovering This appears to have been preceded (aside from a single HTTP HEAD request coming into RGW) by a 5 minute gap in logs where either journald couldn't keep up with debug messages or the Mons were stuck. The last log before that occurs seems to be a compaction event kicking off: mon1 bash\[25927\]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Mar 29 10:24:14 mon1 bash\[25927\]: \*\* Compaction Stats \[L\] \*\* Mar 29 10:24:14 mon1 bash\[25927\]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Mar 29 10:24:14 mon1 bash\[25927\]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Mar 29 10:24:14 mon1 bash\[25927\]: Low 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 116.0 11.4 0.02 0.01 7 0.003 490 462 Mar 29 10:24:14 mon1 bash\[25927\]: High 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.9 1.23 1.20 28 0.044 0 0 Mar 29 10:24:14 mon1 bash\[25927\]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 16.4 0.00 0.00 1 0.001 0 0 We're left wondering what the heck has happened to cause such a huge redistribution of data in the cluster when we've not made any corresponding changes, so wanting to see if there's any breadcrumbs we can find. Appreciate any pointers! -- Cheers, ~Blairo

2 2

Client kernel crashes on cephfs access
by Marc Ruhmann 18 Apr '24

18 Apr '24

Hi everyone, I would like to ask for help regarding client kernel crashes that happen on cephfs access. We have been struggling with this for over a month now with over 100 crashes on 7 hosts during that time. Our cluster runs version 18.2.1. Our clients run CentOS Stream. On CentOS Stream 9 the problem started with kernel version 5.14.0-425.el9. Version 5.14.0-419.el9 is the last one without problems. It also occurred on CentOS Stream 8, starting with version 4.18.0-546.el8 (4.18.0-544.el8 being the last good one). The problem presents itself by the client kernel crashing, forcing a reboot of the machine. Apparently it is triggered by a certain level of IO on the cephfs mount. It works perfectly fine when we rollback to the last good kernel version. The exact call trace in vmcore-dmesg.txt differs between occurrences. Here are two typical examples: ``` [ 8641.382499] list_del corruption. next->prev should be ffff88bd0a4d4c80, but was ffff88bcefdfd280 [ 8641.382521] ------------[ cut here ]------------ [ 8641.382521] kernel BUG at lib/list_debug.c:54! [ 8641.382528] invalid opcode: 0000 [#1] PREEMPT SMP NOPTI [ 8641.382591] CPU: 2 PID: 83929 Comm: kworker/2:0 Kdump: loaded Not tainted 5.14.0-432.el9.x86_64 #1 [ 8641.382610] Hardware name: oVirt RHEL/RHEL-AV, BIOS edk2-20230524-4.el9_3 05/24/2023 [ 8641.382624] Workqueue: ceph-cap ceph_cap_unlink_work [ceph] [ 8641.382662] RIP: 0010:__list_del_entry_valid.cold+0x1d/0x47 [ 8641.382681] Code: c7 c7 78 42 d8 b1 e8 f9 87 fe ff 0f 0b 48 89 fe 48 c7 c7 08 43 d8 b1 e8 e8 87 fe ff 0f 0b 48 c7 c7 b8 43 d8 b1 e8 da 87 fe ff <0f> 0b 48 89 f2 48 89 fe 48 c7 c7 78 43 d8 b1 e8 c6 87 fe ff 0f 0b [ 8641.382711] RSP: 0018:ffff95a000d6be60 EFLAGS: 00010246 [ 8641.382722] RAX: 0000000000000054 RBX: ffff88bced76dc00 RCX: 0000000000000000 [ 8641.382734] RDX: 0000000000000000 RSI: ffff88c02eea0840 RDI: ffff88c02eea0840 [ 8641.382746] RBP: ffff88bd0a4d4c80 R08: 80000000ffff8434 R09: 0000000000ffff10 [ 8641.382758] R10: 000000000000000f R11: 000000000000000f R12: ffff88c02eeb2800 [ 8641.382779] R13: ffff88bcc4610258 R14: ffff88bcc46101b8 R15: ffff88bcc46101c8 [ 8641.382793] FS: 0000000000000000(0000) GS:ffff88c02ee80000(0000) knlGS:0000000000000000 [ 8641.382809] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8641.382819] CR2: 00007f35cee8a000 CR3: 0000000105708004 CR4: 00000000007706e0 [ 8641.382832] PKRU: 55555554 [ 8641.382838] Call Trace: [ 8641.382844] <TASK> [ 8641.382850] ? show_trace_log_lvl+0x1c4/0x2df [ 8641.382860] ? show_trace_log_lvl+0x1c4/0x2df [ 8641.382870] ? ceph_cap_unlink_work+0x3f/0x140 [ceph] [ 8641.382893] ? __die_body.cold+0x8/0xd [ 8641.382902] ? die+0x2b/0x50 [ 8641.382911] ? do_trap+0xce/0x120 [ 8641.382919] ? __list_del_entry_valid.cold+0x1d/0x47 [ 8641.382930] ? do_error_trap+0x65/0x80 [ 8641.382938] ? __list_del_entry_valid.cold+0x1d/0x47 [ 8641.382948] ? exc_invalid_op+0x4e/0x70 [ 8641.382958] ? __list_del_entry_valid.cold+0x1d/0x47 [ 8641.382975] ? asm_exc_invalid_op+0x16/0x20 [ 8641.382988] ? __list_del_entry_valid.cold+0x1d/0x47 [ 8641.382998] ceph_cap_unlink_work+0x3f/0x140 [ceph] [ 8641.383021] process_one_work+0x1e2/0x3b0 [ 8641.383032] ? __pfx_worker_thread+0x10/0x10 [ 8641.383043] worker_thread+0x50/0x3a0 [ 8641.383051] ? __pfx_worker_thread+0x10/0x10 [ 8641.383061] kthread+0xdd/0x100 [ 8641.383069] ? __pfx_kthread+0x10/0x10 [ 8641.383078] ret_from_fork+0x29/0x50 [ 8641.383090] </TASK> [ 8641.383095] Modules linked in: tls ceph libceph dns_resolver fscache netfs nft_counter ipt_REJECT xt_owner xt_conntrack nft_compat nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 rfkill ip_set nf_tables libcrc32c nfnetlink vfat fat intel_rapl_msr intel_rapl_common intel_uncore_frequency_common isst_if_common nfit virtio_gpu iTCO_wdt iTCO_vendor_support libnvdimm lpc_ich virtio_dma_buf drm_shmem_helper drm_kms_helper i2c_i801 rapl syscopyarea sysfillrect sysimgblt virtio_balloon fb_sys_fops i2c_smbus pcspkr joydev fuse drm ext4 mbcache jbd2 sr_mod cdrom sd_mod ahci t10_pi sg libahci crct10dif_pclmul crc32_pclmul crc32c_intel libata ghash_clmulni_intel virtio_net virtio_console virtio_scsi net_failover failover serio_raw ``` ``` [ 3538.365469] list_del corruption. next->prev should be ffff8d2b75997c80, but was ffff8d2afcfaae80 [ 3538.365488] ------------[ cut here ]------------ [ 3538.365488] kernel BUG at lib/list_debug.c:54! [ 3538.365493] invalid opcode: 0000 [#1] PREEMPT SMP NOPTI [ 3538.365553] CPU: 0 PID: 910 Comm: php-fpm Kdump: loaded Not tainted 5.14.0-432.el9.x86_64 #1 [ 3538.365569] Hardware name: oVirt RHEL/RHEL-AV, BIOS edk2-20230524-4.el9_3 05/24/2023 [ 3538.365582] RIP: 0010:__list_del_entry_valid.cold+0x1d/0x47 [ 3538.365612] Code: c7 c7 78 42 38 8e e8 f9 87 fe ff 0f 0b 48 89 fe 48 c7 c7 08 43 38 8e e8 e8 87 fe ff 0f 0b 48 c7 c7 b8 43 38 8e e8 da 87 fe ff <0f> 0b 48 89 f2 48 89 fe 48 c7 c7 78 43 38 8e e8 c6 87 fe ff 0f 0b [ 3538.365641] RSP: 0018:ffffae870073fda0 EFLAGS: 00010246 [ 3538.365652] RAX: 0000000000000054 RBX: ffff8d2b75997800 RCX: 0000000000000000 [ 3538.365668] RDX: 0000000000000000 RSI: ffff8d2e2ee20840 RDI: ffff8d2e2ee20840 [ 3538.365681] RBP: ffff8d2b75997ab8 R08: 80000000ffff842f R09: 0000000000ffff10 [ 3538.365693] R10: 000000000000000f R11: 000000000000000f R12: 00000000ffffc032 [ 3538.365705] R13: ffff8d2b75997c80 R14: ffff8d2ac480b800 R15: ffff8d2ac480b9c8 [ 3538.365717] FS: 00007f9be42097c0(0000) GS:ffff8d2e2ee00000(0000) knlGS:0000000000000000 [ 3538.365733] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3538.365744] CR2: 00007fa97dc5398c CR3: 0000000104248004 CR4: 00000000007706f0 [ 3538.365756] PKRU: 55555554 [ 3538.365761] Call Trace: [ 3538.365768] <TASK> [ 3538.365774] ? show_trace_log_lvl+0x1c4/0x2df [ 3538.365785] ? show_trace_log_lvl+0x1c4/0x2df [ 3538.365796] ? ceph_drop_caps_for_unlink+0xb8/0x170 [ceph] [ 3538.365828] ? __die_body.cold+0x8/0xd [ 3538.365836] ? die+0x2b/0x50 [ 3538.365845] ? do_trap+0xce/0x120 [ 3538.365853] ? __list_del_entry_valid.cold+0x1d/0x47 [ 3538.365863] ? do_error_trap+0x65/0x80 [ 3538.365871] ? __list_del_entry_valid.cold+0x1d/0x47 [ 3538.365881] ? exc_invalid_op+0x4e/0x70 [ 3538.365891] ? __list_del_entry_valid.cold+0x1d/0x47 [ 3538.365901] ? asm_exc_invalid_op+0x16/0x20 [ 3538.365912] ? __list_del_entry_valid.cold+0x1d/0x47 [ 3538.365923] ceph_drop_caps_for_unlink+0xb8/0x170 [ceph] [ 3538.365947] ceph_unlink+0xed/0x450 [ceph] [ 3538.365970] vfs_unlink+0x114/0x290 [ 3538.365980] do_unlinkat+0x1af/0x2e0 [ 3538.365990] __x64_sys_unlink+0x3e/0x60 [ 3538.365999] do_syscall_64+0x59/0x90 [ 3538.366008] ? syscall_exit_to_user_mode+0x22/0x40 [ 3538.366018] ? do_syscall_64+0x69/0x90 [ 3538.366027] ? do_syscall_64+0x69/0x90 [ 3538.366035] entry_SYSCALL_64_after_hwframe+0x72/0xdc [ 3538.366046] RIP: 0033:0x7f9be40ff27b [ 3538.366069] Code: f0 ff ff 73 01 c3 48 8b 0d a2 ab 0f 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 75 ab 0f 00 f7 d8 64 89 01 48 [ 3538.367031] RSP: 002b:00007ffd8640de58 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3538.367576] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 00007f9be40ff27b [ 3538.368116] RDX: 0000000000000007 RSI: 0000000000000001 RDI: 00007f9bdd4af698 [ 3538.368646] RBP: 00007f9bdd4af698 R08: 00000000ffffffc9 R09: 0000000000000038 [ 3538.369156] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 3538.369671] R13: 00007f9bdd4af698 R14: 0000000000000001 R15: 00007f9be3c15290 [ 3538.370182] </TASK> [ 3538.370682] Modules linked in: ceph libceph dns_resolver fscache netfs nft_counter ipt_REJECT xt_owner xt_conntrack nft_compat nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 rfkill ip_set nf_tables libcrc32c nfnetlink vfat fat intel_rapl_msr intel_rapl_common intel_uncore_frequency_common virtio_gpu virtio_dma_buf drm_shmem_helper isst_if_common drm_kms_helper nfit syscopyarea sysfillrect sysimgblt fb_sys_fops libnvdimm i2c_i801 iTCO_wdt iTCO_vendor_support lpc_ich i2c_smbus virtio_balloon rapl joydev pcspkr drm fuse ext4 mbcache jbd2 sr_mod cdrom sg ahci libahci crct10dif_pclmul crc32_pclmul crc32c_intel libata ghash_clmulni_intel virtio_net virtio_blk virtio_console net_failover virtio_scsi failover serio_raw ``` I checked the changelogs of the kernel versions and spotted these three commits that were backported: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?… https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?… https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?… The first one adds changes that look related. Does anybody have experienced this as well or know something about this? Thanks and best regards, Marc

6 7

Ceph Community Management Update
by Josh Durgin 18 Apr '24

18 Apr '24

Hi everyone, I’d like to extend a warm thank you to Mike Perez for his years of service as community manager for Ceph. He is changing focuses now to engineering. The Ceph Foundation board decided to use services from the Linux Foundation to fulfill some community management responsibilities, rather than rely on a single member organization employing a community manager. The Linux Foundation will assist with Ceph Foundation membership and governance matters. Please welcome Noah Lehman (cc’d) as our social media and marketing point person - for anything related to this area, including the Ceph YouTube channel, please reach out to him. Ceph days will continue to be organized and funded by organizations around the world, with the help of the Ceph Ambassadors ( https://ceph.io/en/community/ambassadors/) Gaurav Sitlani (cc’d) will help organize the ambassadors going forward. For other matters, please contact council(a)ceph.io and we’ll direct the matter to the appropriate people. Thanks, Neha Ojha, Dan van der Ster, Josh Durgin Ceph Executive Council

2 1

cephadm custom jinja2 service templates
by Stolte, Felix 17 Apr '24

17 Apr '24

Hi folks, I would like to use a custom jina2 template for an ingress service for rendering the keepalived and haproxy config. Can someone tell me how to override the default templates? Best regards Felix --------------------------------------------------------------------------------------------- --------------------------------------------------------------------------------------------- Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Stefan Müller Geschaeftsfuehrung: Prof. Dr. Astrid Lambrecht (Vorsitzende), Karsten Beneke (stellv. Vorsitzender), Dr. Ir. Pieter Jansens, Prof. Dr. Frauke Melchior --------------------------------------------------------------------------------------------- ---------------------------------------------------------------------------------------------

2 1

(deep-)scrubs blocked by backfill
by Frank Schilder 17 Apr '24

17 Apr '24

Hi all, I have a technical question about scrub scheduling. I replaced a disk and it is back-filling slowly. We have set osd_scrub_during_recovery = true and still observe that scrub times continuously increase (number of PGs not scrubbed in time is continuously increasing). Investigating the situation it looks like any OSD that has a PG in states "backfill_wait" or "backfilling" is preventing scrubs to be scheduled on PGs it is a member of. However, it seems it is not quite like that. On the one hand I have never seen a PG in a state like "active+scrubbing+remapped+backfilling", so backfilling PGs at least never seem to scrub. On the other hand, it seems like more PGs are scrubbed than would be eligible if *all* OSDs with a remapped PG on it would refuse scrubs. It looks like something in between "only OSDs with a backfilling PG block requests for scrub reservations" and "all OSDs with a PG in states backfilling or backfill_wait block requests for scrub reservations". Does the position in the backfill reservation queue play a role? If anyone has insight into how scrub reservations are granted and when not in the situation of an OSD backfilling that would be great. My naive interpretation of "osd_scrub_during_recovery = true" was that scrubs proceed as if no backfill was going on. This, however, is clearly not the case. Having an answer to my question above would help me a lot to get an idea when things will go back to normal. Thanks a lot and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

1 0

Performance of volume size, not a block size
by Mitsumasa KONDO 17 Apr '24

17 Apr '24

Hi, In AWS EBS gp3, AWS says that small volume size cannot achieve best performance. I think it's a feature or tendency of general distributed storages including Ceph. Is that right in Ceph block storage? I read many docs on ceph community. I never heard of Ceph storage. https://docs.aws.amazon.com/ebs/latest/userguide/general-purpose.html Regard, -- Mitsumasa KONDO

4 6

feature_map differs across mon_status
by Joel Davidow 17 Apr '24

17 Apr '24

Just curious why the feature_map portions differ in the return of mon_status across a cluster. Below is an example from each of five mons in a healthy 16.2.10 cephadm cluster: root@mon.d:~# ceph tell mon.a mon_status | jq .feature_map { "mon": [ { "features": "0x3f01cfb9fffdffff", "release": "luminous", "num": 1 } ], "osd": [ { "features": "0x3f01cfb9fffdffff", "release": "luminous", "num": 42 } ], "client": [ { "features": "0x3f01cfb9fffdffff", "release": "luminous", "num": 1 } ], "mgr": [ { "features": "0x3f01cfb9fffdffff", "release": "luminous", "num": 1 } ] } root@mon.d:~# ceph tell mon.b mon_status | jq .feature_map { "mon": [ { "features": "0x3f01cfb9fffdffff", "release": "luminous", "num": 1 } ], "osd": [ { "features": "0x3f01cfb9fffdffff", "release": "luminous", "num": 36 } ], "client": [ { "features": "0x3f01cfb9fffdffff", "release": "luminous", "num": 2 } ] } root@mon.d:~# ceph tell mon.c mon_status | jq .feature_map { "mon": [ { "features": "0x3f01cfb9fffdffff", "release": "luminous", "num": 1 } ], "osd": [ { "features": "0x3f01cfb9fffdffff", "release": "luminous", "num": 81 } ], "client": [ { "features": "0x3f01cfb9fffdffff", "release": "luminous", "num": 4 } ], "mgr": [ { "features": "0x3f01cfb9fffdffff", "release": "luminous", "num": 2 } ] } root@mon.d:~# ceph tell mon.d mon_status | jq .feature_map { "mon": [ { "features": "0x3f01cfb9fffdffff", "release": "luminous", "num": 1 } ], "osd": [ { "features": "0x3f01cfb9fffdffff", "release": "luminous", "num": 112 } ], "client": [ { "features": "0x3f01cfb9fffdffff", "release": "luminous", "num": 7 } ] } root@mon.d:~# ceph tell mon.e mon_status | jq .feature_map { "mon": [ { "features": "0x3f01cfb9fffdffff", "release": "luminous", "num": 1 } ], "osd": [ { "features": "0x3f01cfb9fffdffff", "release": "luminous", "num": 88 } ], "client": [ { "features": "0x3f01cfb9fffdffff", "release": "luminous", "num": 4 } ] }

2 1

cephFS on CentOS7
by Dario Graña 17 Apr '24

17 Apr '24

Hello everyone! We deployed a platform with Ceph Quincy and now we need to give access to some old nodes with CentOS7 until 30/07/2024. I found two approaches, the first one, deploying Ganesha NFS and bringing access through the NFS protocol. The second one is to use an older cephfs client, specifically the Octopus client. I would like to know if there is a third option and which the community would recommend. Thanks in advance. Regards! -- Dario Graña PIC (Port d'Informació Científica) Campus UAB, Edificio D E-08193 Bellaterra, Barcelona http://www.pic.es Avis - Aviso - Legal Notice: http://legal.ifae.es

2 2

Working ceph cluster reports large amount of pgs in state unknown/undersized and objects degraded
by Tobias Langner 17 Apr '24

17 Apr '24

We operate a tiny ceph cluster across three machines, each running two OSDs and one of each mds, mgr, and mon. The cluster serves one main erasure-coded (2+1) storage pool and a few other management-related pools. The cluster has been running smoothly for several months. A few weeks ago we noticed a health warning reporting backfillfull/nearfull osds and pools. Here is the output of `ceph -s` at this point (extraced from logs): -------------------------------------------------------------------------------- cluster: health: HEALTH_WARN 1 backfillfull osd(s) 2 nearfull osd(s) Reduced data availability: 163 pgs inactive, 1 pg peering Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull Degraded data redundancy: 1486709/10911157 objects degraded (13.626%), 68 pgs degraded, 68 pgs undersized 162 pgs not scrubbed in time 6 pool(s) backfillfull services: mon: 3 daemons, quorum mon.101,mon.102,mon.100 (age 5m) mgr: mgr-102(active, since 54m), standbys: mgr-101, mgr-100 mds: 1/1 daemons up, 1 standby, 1 hot standby osd: 6 osds: 6 up (since 4m), 6 in (since 2w); 7 remapped pgs data: volumes: 1/1 healthy pools: 6 pools, 338 pgs objects: 3.64M objects, 14 TiB usage: 13 TiB used, 1.7 TiB / 15 TiB avail pgs: 47.929% pgs unknown 0.296% pgs not active 1486709/10911157 objects degraded (13.626%) 52771/10911157 objects misplaced (0.484%) 162 unknown 106 active+clean 67 active+undersized+degraded 1 active+undersized+degraded+remapped+backfill_toofull 1 remapped+peering 1 active+remapped+backfill_toofull -------------------------------------------------------------------------------- I now see the large amount of pgs in state unknown and the fact that a significant fraction of objects is degraded despite all osds being up, but we didn't notice this back then. Because the cluster continued to act fine from an FS access perspective, we didn't really notice the developing problem and did not intervene. From then one, things have mostly gone downwards. Now we `ceph -s` reports the following: -------------------------------------------------------------------------------- cluster: health: HEALTH_WARN noout flag(s) set Reduced data availability: 117 pgs inactive Degraded data redundancy: 2095625/12121767 objects degraded (17.288%), 114 pgs degraded, 114 pgs undersized 117 pgs not scrubbed in time services: mon: 3 daemons, quorum mon.101,mon.102,mon.100 (age 15h) mgr: mgr-102(active, since 7d), standbys: mgr-100, mgr-101 mds: 1/1 daemons up, 1 standby, 1 hot standby osd: 6 osds: 6 up (since 55m), 6 in (since 3w) flags noout data: volumes: 1/1 healthy pools: 6 pools, 338 pgs objects: 4.04M objects, 15 TiB usage: 12 TiB used, 2.8 TiB / 15 TiB avail pgs: 34.615% pgs unknown 2095625/12121767 objects degraded (17.288%) 117 unknown 114 active+undersized+degraded 107 active+clean -------------------------------------------------------------------------------- Note in particular the still very large number of pgs in state unknown, which hasn't changed in days. Same goes for the degraded pgs. Also, the cluster should have around 37TiB storage available but now it only reports 15 TiB. We did a bit of digging around but couldn't really get to the bottom of the unknown pgs and how we can recover from that. One other data point is that the command `ceph osd df tree` gets stuck on two of the three machines and one the one where it returns something, it looks like this: -------------------------------------------------------------------------------- ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 47.67506 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - root default -13 18.26408 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - datacenter dc.100 -5 18.26408 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - host osd-100 3 hdd 10.91409 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 0 91 up osd.3 5 hdd 7.34999 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 0 48 up osd.5 -9 14.69998 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - datacenter dc.101 -7 14.69998 - 0 B 0 B 0 B 0 B 0 B 0 B 0 0 - host osd-101 0 hdd 7.34999 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 0 83 up osd.0 1 hdd 7.34999 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0 0 86 up osd.1 -11 14.71100 - 15 TiB 12 TiB 12 TiB 77 MiB 21 GiB 2.6 TiB 82.00 1.00 - datacenter dc.102 -17 7.35550 - 7.4 TiB 6.3 TiB 6.2 TiB 16 MiB 11 GiB 1.1 TiB 85.16 1.04 - host osdroid-102-1 4 hdd 7.35550 1.00000 7.4 TiB 6.3 TiB 6.2 TiB 16 MiB 11 GiB 1.1 TiB 85.16 1.04 114 up osd.4 -15 7.35550 - 7.4 TiB 5.8 TiB 5.7 TiB 61 MiB 10 GiB 1.6 TiB 78.83 0.96 - host osdroid-102-2 2 hdd 7.35550 1.00000 7.4 TiB 5.8 TiB 5.7 TiB 61 MiB 10 GiB 1.6 TiB 78.83 0.96 107 up osd.2 TOTAL 15 TiB 12 TiB 12 TiB 77 MiB 21 GiB 2.6 TiB 82.00 MIN/MAX VAR: 0/1.04 STDDEV: 66.97 -------------------------------------------------------------------------------- The odd part here is that for some reason only osd.2 and osd.4 seem to contribute size to the cluster. Interestingly, accessing content from the storage pool works mostly without issues, which shouldn't work if 4 out of 6 OSDs weren't properly up. Any ideas what we can do to get our cluster back into a sane state? I'm happy to provide more logs or command output, please let me know. Thanks!

1 0

Ceph 16.2.x mon compactions, disk writes
by Zakhar Kirpichenko 17 Apr '24

17 Apr '24

Hi, Monitors in our 16.2.14 cluster appear to quite often run "manual compaction" tasks: debug 2023-10-09T09:30:53.888+0000 7f48a329a700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1696843853892760, "job": 64225, "event": "flush_started", "num_memtables": 1, "num_entries": 715, "num_deletes": 251, "total_data_size": 3870352, "memory_usage": 3886744, "flush_reason": "Manual Compaction"} debug 2023-10-09T09:30:53.904+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:30:53.908+0000 7f48a3a9b700 4 rocksdb: (Original Log Time 2023/10/09-09:30:53.910204) [db_impl/db_impl_compaction_flush.cc:2516] [default] Manual compaction from level-0 to level-5 from 'paxos .. 'paxos; will stop at (end) debug 2023-10-09T09:30:53.908+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:30:53.908+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:30:53.908+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:30:53.908+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:30:53.908+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:30:53.908+0000 7f48a3a9b700 4 rocksdb: (Original Log Time 2023/10/09-09:30:53.911004) [db_impl/db_impl_compaction_flush.cc:2516] [default] Manual compaction from level-5 to level-6 from 'paxos .. 'paxos; will stop at (end) debug 2023-10-09T09:32:08.956+0000 7f48a329a700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1696843928961390, "job": 64228, "event": "flush_started", "num_memtables": 1, "num_entries": 1580, "num_deletes": 502, "total_data_size": 8404605, "memory_usage": 8465840, "flush_reason": "Manual Compaction"} debug 2023-10-09T09:32:08.972+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:08.976+0000 7f48a3a9b700 4 rocksdb: (Original Log Time 2023/10/09-09:32:08.977739) [db_impl/db_impl_compaction_flush.cc:2516] [default] Manual compaction from level-0 to level-5 from 'logm .. 'logm; will stop at (end) debug 2023-10-09T09:32:08.976+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:08.976+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:08.976+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:08.976+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:08.976+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:08.976+0000 7f48a3a9b700 4 rocksdb: (Original Log Time 2023/10/09-09:32:08.978512) [db_impl/db_impl_compaction_flush.cc:2516] [default] Manual compaction from level-5 to level-6 from 'logm .. 'logm; will stop at (end) debug 2023-10-09T09:32:12.764+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:12.764+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:12.764+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:12.764+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:12.764+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:32:12.764+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:33:29.028+0000 7f48a329a700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1696844009033151, "job": 64231, "event": "flush_started", "num_memtables": 1, "num_entries": 1430, "num_deletes": 251, "total_data_size": 8975535, "memory_usage": 9035920, "flush_reason": "Manual Compaction"} debug 2023-10-09T09:33:29.044+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:33:29.048+0000 7f48a3a9b700 4 rocksdb: (Original Log Time 2023/10/09-09:33:29.049585) [db_impl/db_impl_compaction_flush.cc:2516] [default] Manual compaction from level-0 to level-5 from 'paxos .. 'paxos; will stop at (end) debug 2023-10-09T09:33:29.048+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:33:29.048+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:33:29.048+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:33:29.048+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:33:29.048+0000 7f4899286700 4 rocksdb: [db_impl/db_impl_compaction_flush.cc:1443] [default] Manual compaction starting debug 2023-10-09T09:33:29.048+0000 7f48a3a9b700 4 rocksdb: (Original Log Time 2023/10/09-09:33:29.050355) [db_impl/db_impl_compaction_flush.cc:2516] [default] Manual compaction from level-5 to level-6 from 'paxos .. 'paxos; will stop at (end) I have removed a lot of interim log messages to save space. During each compaction the monitor process writes approximately 500-600 MB of data to disk over a short period of time. These writes add up to tens of gigabytes per hour and hundreds of gigabytes per day. Monitor rocksdb and compaction options are default: "mon_compact_on_bootstrap": "false", "mon_compact_on_start": "false", "mon_compact_on_trim": "true", "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", Is this expected behavior? Is this something I can adjust in order to extend the system storage life? Best regards, Zakhar

5 32

How to make config changes stick for MDS?
by Erich Weiler 17 Apr '24

17 Apr '24

Hi All, I'm having a crazy time getting config items to stick on my MDS daemons. I'm running Reef 18.2.1 on RHEL 9 and the daemons are running in podman, I used cephadm to deploy the daemons. I can adjust the config items in runtime, like so: ceph tell mds.slugfs.pr-md-01.xdtppo config set mds_bal_interval -1 But for the life of me I cannot get that to stick when I restart the MDS daemon. I've tried adding this to /etc/ceph/ceph.conf in the host server: [mds] mds_bal_interval = -1 But that doesn't get picked up on daemon restart. I also added the same config segment to /etc/ceph/ceph.conf *inside* the container, no dice, still doesn't stick. I even tried adding it to /var/lib/ceph/<uuid>/config/ceph.conf and it *still* doesn't stick across daemon restarts. Does anyone know how I can get MDS config items to stick across daemon reboots when the daemon is running in podman under RHEL? Thanks much! -erich

2 1

Ceph Leadership Team Meeting, 2024-04-08
by Laura Flores 16 Apr '24

16 Apr '24

Hi all, Today we discussed: 2024/04/08 - [Zac] CQ#4 is going out this week - https://pad.ceph.com/p/ceph_quarterly_2024_04 - Last chance to review! - [Zac] IcePic Initiative - context-sensitive help - do we regard the docs as a part of the online help? - https://pad.ceph.com/p/2024_04_08_cephadm_context_sensitive_help - docs.ceph.com should be main source of truth; can link to this or reference it generally as "see docs.ceph.com" - Squid RC status - Blockers tracked in: https://pad.ceph.com/p/squid-upgrade-failures - rgw: topic changes merged to main, but introduced some test failures. account changes blocked on topics - Non-blocker for RC0 - centos 9 containerization (status unknown?) - Non-blocker for RC0 - Follow up with Dan / Guillaume - RADOS has one outstanding blocker awaiting QA - Failing to register new account at Ceph tracker - error 404. - Likely related to Redmine upgrade over the weekend - Pacific eol: - Action item: in https://docs.ceph.com/en/latest/releases/, move to "archived" - 18.2.3 - one or two PRs from cephfs left - Milestone: https://github.com/ceph/ceph/milestone/19 Thanks, Laura -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

2 4

Ceph Users Feedback Survey
by Neha Ojha 16 Apr '24

16 Apr '24

Hi everyone, On behalf of the Ceph Foundation Board, I would like to announce the creation of, and cordially invite you to, the first of a recurring series of meetings focused solely on gathering feedback from the users of Ceph. The overarching goal of these meetings is to elicit feedback from the users, companies, and organizations who use Ceph in their production environments. You can find more details about the motivation behind this effort in our user survey [1] that we highly encourage all of you to take. This is an extension of the Ceph User Dev Meeting with concerted focus on Performance (led by Vincent Hsu, IBM) and Orchestration/Deployment (led by Matt Leonard, Bloomberg), to start off with. We would like to kick off this series of meetings on March 21, 2024. The survey will be open until March 18, 2024. Looking forward to hearing from you! Thanks, Neha [1] https://docs.google.com/forms/d/15aWxoG4wSQz7ziBaReVNYVv94jA0dSNQsDJGqmHCLM…

1 2

Announcing go-ceph v0.27.0
by John Mulligan 16 Apr '24

16 Apr '24

We are happy to announce another release of the go-ceph API library. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.27.0 The library includes bindings that aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. The library also includes additional APIs that can be used to administer cephfs, rbd, rgw, and other subsystems. There are already a few consumers of this library in the wild, including the ceph-csi project. -- John Mulligan phlogistonjohn(a)asynchrono.us jmulligan(a)redhat.com

1 0

Setting up Hashicorp Vault for Encryption with Ceph
by Michael Worsham 16 Apr '24

16 Apr '24

Is there a how-to document available on how to setup Hashicorp's Vault for Ceph, preferably in a HA state? Due to some encryption needs, we need to set up LUKS, OSD encryption AND Ceph bucket encryption as well. Yes, we know there will be a performance hit, but the encrypt-everything is a hard requirement for our business needs since we have government and healthcare-related contracts. -- Michael This message and its attachments are from Data Dimensions and are intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately and permanently delete the original email and destroy any copies or printouts of this email as well as any attachments.

2 1

Setting S3 bucket policies with multi-tenants
by Thomas Bennett 15 Apr '24

15 Apr '24

Hi, I'm running Ceph Quincy (17.2.6) with a rados-gateway. I have muti tenants, for example: - Tenant1$manager - Tenant1$readwrite I would like to set a policy on a bucket (backups for example) owned by *Tenant1$manager* to allow *Tenant1$readwrite* access to that bucket. I can't find any documentation that discusses this scenario. Does anyone know how to specify the Principle and Resource section of a policy.json file? Or any other configuration that I might be missing? I've tried some variations on Principal and Resource including and excluding tenant information, but not no luck yet. For example: { "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": {"AWS": ["arn:aws:iam:::user/*Tenant1$readwrite*"]}, "Action": ["s3:ListBucket","s3:GetObject", ,"s3:PutObject"], "Resource": [ "arn:aws:s3:::*Tenant1/backups*" ] }] } I'm using s3cmd for testing, so: s3cmd --config s3cfg.manager setpolicy policy.json s3://backups/ Returns: s3://backups/: Policy updated And then testing: s3cmd --config s3cfg.readwrite ls s3://backups/ ERROR: Access to bucket 'backups' was denied ERROR: S3 error: 403 (AccessDenied) Thanks, Tom

4 4

MDS Behind on Trimming...
by Erich Weiler 15 Apr '24

15 Apr '24

Hi All, I've been battling this for a while and I'm not sure where to go from here. I have a Ceph health warning as such: # ceph -s cluster: id: 58bde08a-d7ed-11ee-9098-506b4b4da440 health: HEALTH_WARN 1 MDSs report slow requests 1 MDSs behind on trimming services: mon: 5 daemons, quorum pr-md-01,pr-md-02,pr-store-01,pr-store-02,pr-md-03 (age 5d) mgr: pr-md-01.jemmdf(active, since 3w), standbys: pr-md-02.emffhz mds: 1/1 daemons up, 2 standby osd: 46 osds: 46 up (since 9h), 46 in (since 2w) data: volumes: 1/1 healthy pools: 4 pools, 1313 pgs objects: 260.72M objects, 466 TiB usage: 704 TiB used, 424 TiB / 1.1 PiB avail pgs: 1306 active+clean 4 active+clean+scrubbing+deep 3 active+clean+scrubbing io: client: 123 MiB/s rd, 75 MiB/s wr, 109 op/s rd, 1.40k op/s wr And the specifics are: # ceph health detail HEALTH_WARN 1 MDSs report slow requests; 1 MDSs behind on trimming [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests mds.slugfs.pr-md-01.xdtppo(mds.0): 99 slow requests are blocked > 30 secs [WRN] MDS_TRIM: 1 MDSs behind on trimming mds.slugfs.pr-md-01.xdtppo(mds.0): Behind on trimming (13884/250) max_segments: 250, num_segments: 13884 That "num_segments" number slowly keeps increasing. I suspect I just need to tell the MDS servers to trim faster but after hours of googling around I just can't figure out the best way to do it. The best I could come up with was to decrease "mds_cache_trim_decay_rate" from 1.0 to .8 (to start), based on this page: https://www.suse.com/support/kb/doc/?id=000019740 But it doesn't seem to help, maybe I should decrease it further? I am guessing this must be a common issue...? I am running Reef on the MDS servers, but most clients are on Quincy. Thanks for any advice! cheers, erich

5 24

PG inconsistent
by Albert Shih 12 Apr '24

12 Apr '24

Hi everyone. I got a warning with root@cthulhu1:/etc/ceph# ceph -s cluster: id: 9c5bb196-c212-11ee-84f3-c3f2beae892d health: HEALTH_ERR 1 scrub errors Possible data damage: 1 pg inconsistent So I find the pg with the issue, and launch a pg repair (still waiting) But I try to find «why» so I check all the OSD related on this pg and didn't find anything, no error from osd daemon, no errors from smartctl, no error from the kernel message. So I just like to know if that's «normal» or should I scratch deeper. JAS -- Albert SHIH 🦫 🐸 France Heure locale/Local time: ven. 12 avril 2024 11:51:37 CEST

4 5

Issue about "ERROR: S3 Temporary Error: Request failed for: /. Please try again later"
by elite_stu＠163.com 11 Apr '24

11 Apr '24

I have mapped port 32505 to 23860, however when connect via s3cmd it fails with "ERROR: S3 Temporary Error: Request failed for: /. Please try again later." . has anyone ecounted same issue? [root@vm-04 ~]# s3cmd ls WARNING: Retrying failed request: / ('') WARNING: Waiting 3 sec... WARNING: Retrying failed request: / ('') WARNING: Waiting 6 sec... WARNING: Retrying failed request: / ('') WARNING: Waiting 9 sec... WARNING: Retrying failed request: / ('') WARNING: Waiting 12 sec... WARNING: Retrying failed request: / ('') WARNING: Waiting 15 sec... ERROR: S3 Temporary Error: Request failed for: /. Please try again later. [root@vm-04 ~]# [root@vm-04 ~]# cat .s3cfg [default] access_key = E9BJAC6QKLTOKVJR4TZC secret_key = KBvm0YqeAFtVM0I1gUhX5wrnwpHZZ5SLhiUa56ss host_base = xxxxxx:23860 host_bucket = xxxxxx:23860/bucket-king use_https = False signature_v2 = False [root@vm-04 ~]# bash-4.4$ ceph -s cluster: id: dd6a082e-868d-44e0-b93b-0f4956e5dd1a health: HEALTH_OK services: mon: 3 daemons, quorum a,b,c (age 4m) mgr: b(active, since 4m), standbys: a mds: 1/1 daemons up, 1 standby osd: 3 osds: 3 up (since 4m), 3 in (since 26m) rgw: 1 daemon active (1 hosts, 1 zones) data: volumes: 1/1 healthy pools: 14 pools, 233 pgs objects: 548 objects, 808 MiB usage: 2.4 GiB used, 148 GiB / 150 GiB avail pgs: 233 active+clean bash-4.4$ [root@vm-01 ~]# kubectl get svc -nrook-ceph NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE rook-ceph-mgr ClusterIP 10.105.165.62 <none> 9283/TCP 26m rook-ceph-mgr-dashboard ClusterIP 10.102.18.178 <none> 8443/TCP 26m rook-ceph-mgr-dashboard-external-https NodePort 10.100.66.254 <none> 8443:31091/TCP 24m rook-ceph-rgw-my-store ClusterIP 10.104.146.147 <none> 80/TCP 16m rook-ceph-rgw-my-store-external NodePort 10.106.39.241 <none> 80:32505/TCP 14m [root@vm-01 ~]# [root@vm-01 ~]# [root@vm-01 ~]# kubectl get pods -nrook-ceph | grep rgw rook-ceph-rgw-my-store-a-67d579d99b-5xd64 2/2 Running 0 5m22s [root@vm-01 ~]# [root@vm-01 ~]# [root@vm-01 ~]# [root@vm-01 ~]# [root@vm-01 ~]# kubectl -nrook-ceph logs rook-ceph-rgw-my-store-a-67d579d99b-5xd64 | tail -20 Defaulted container "rgw" out of: rgw, log-collector, chown-container-data-dir (init) debug 2024-04-11T13:42:33.635+0000 7fee46b43700 1 beast: 0x7fee079c2730: 192.168.100.9 - rook-ceph-internal-s3-user-checker-12e1356a-bbec-4ba6-a333-597842bf6f94 [11/Apr/2024:13:42:33.630 +0000] "DELETE /rook-ceph-bucket-checker-12e1356a-bbec-4ba6-a333-597842bf6f94/rookHealthCheckTestObject HTTP/1.1" 204 0 - "aws-sdk-go/1.44.118 (go1.19.4; linux; amd64)" - latency=0.005000210s debug 2024-04-11T13:42:33.635+0000 7fee4e352700 1 ====== starting new request req=0x7fee079c2730 ===== debug 2024-04-11T13:42:33.647+0000 7fee4333c700 1 ====== req done req=0x7fee079c2730 op status=0 http_status=200 latency=0.012000503s ====== debug 2024-04-11T13:42:33.647+0000 7fee4333c700 1 beast: 0x7fee079c2730: 192.168.100.9 - rook-ceph-internal-s3-user-checker-12e1356a-bbec-4ba6-a333-597842bf6f94 [11/Apr/2024:13:42:33.635 +0000] "PUT /rook-ceph-bucket-checker-12e1356a-bbec-4ba6-a333-597842bf6f94 HTTP/1.1" 200 0 - "aws-sdk-go/1.44.118 (go1.19.4; linux; amd64)" - latency=0.012000503s debug 2024-04-11T13:42:33.648+0000 7fee4e352700 1 ====== starting new request req=0x7fee079c2730 ===== debug 2024-04-11T13:42:33.666+0000 7fee3b32c700 1 ====== req done req=0x7fee079c2730 op status=0 http_status=200 latency=0.018000754s ====== debug 2024-04-11T13:42:33.666+0000 7fee3b32c700 1 beast: 0x7fee079c2730: 192.168.100.9 - rook-ceph-internal-s3-user-checker-12e1356a-bbec-4ba6-a333-597842bf6f94 [11/Apr/2024:13:42:33.648 +0000] "PUT /rook-ceph-bucket-checker-12e1356a-bbec-4ba6-a333-597842bf6f94/rookHealthCheckTestObject HTTP/1.1" 200 21 - "aws-sdk-go/1.44.118 (go1.19.4; linux; amd64)" - latency=0.018000754s debug 2024-04-11T13:42:33.667+0000 7fee4e352700 1 ====== starting new request req=0x7fee079c2730 ===== debug 2024-04-11T13:42:33.669+0000 7fee4e352700 1 ====== req done req=0x7fee079c2730 op status=0 http_status=200 latency=0.002000083s ====== debug 2024-04-11T13:42:33.669+0000 7fee4e352700 1 beast: 0x7fee079c2730: 192.168.100.9 - rook-ceph-internal-s3-user-checker-12e1356a-bbec-4ba6-a333-597842bf6f94 [11/Apr/2024:13:42:33.667 +0000] "GET /rook-ceph-bucket-checker-12e1356a-bbec-4ba6-a333-597842bf6f94/rookHealthCheckTestObject HTTP/1.1" 200 21 - "aws-sdk-go/1.44.118 (go1.19.4; linux; amd64)" - latency=0.002000083s debug 2024-04-11T13:42:33.710+0000 7fee43b3d700 1 ====== starting new request req=0x7fee079c2730 ===== debug 2024-04-11T13:42:33.719+0000 7fee2e312700 1 ====== req done req=0x7fee079c2730 op status=0 http_status=204 latency=0.009000377s ====== debug 2024-04-11T13:42:33.719+0000 7fee2e312700 1 beast: 0x7fee079c2730: 192.168.100.9 - rook-ceph-internal-s3-user-checker-12e1356a-bbec-4ba6-a333-597842bf6f94 [11/Apr/2024:13:42:33.710 +0000] "DELETE /rook-ceph-bucket-checker-12e1356a-bbec-4ba6-a333-597842bf6f94/rookHealthCheckTestObject HTTP/1.1" 204 0 - "aws-sdk-go/1.44.118 (go1.19.4; linux; amd64)" - latency=0.009000377s debug 2024-04-11T13:42:37.863+0000 7fee26b03700 1 ====== starting new request req=0x7fee078c0730 ===== debug 2024-04-11T13:42:37.863+0000 7fee26b03700 1 ====== req done req=0x7fee078c0730 op status=0 http_status=200 latency=0.000000000s ====== debug 2024-04-11T13:42:37.863+0000 7fee26b03700 1 beast: 0x7fee078c0730: 192.168.100.8 - - [11/Apr/2024:13:42:37.863 +0000] "GET /swift/healthcheck HTTP/1.1" 200 0 - "kube-probe/1.28" - latency=0.000000000s debug 2024-04-11T13:42:41.078+0000 7fef3a54b700 0 rgw UsageLogger: WARNING: RGWRados::log_usage(): user name empty (bucket=), skipping debug 2024-04-11T13:42:47.862+0000 7fee24aff700 1 ====== starting new request req=0x7fee07a43730 ===== debug 2024-04-11T13:42:47.862+0000 7fee24aff700 1 ====== req done req=0x7fee07a43730 op status=0 http_status=200 latency=0.000000000s ====== debug 2024-04-11T13:42:47.862+0000 7fee24aff700 1 beast: 0x7fee07a43730: 192.168.100.8 - - [11/Apr/2024:13:42:47.862 +0000] "GET /swift/healthcheck HTTP/1.1" 200 0 - "kube-probe/1.28" - latency=0.000000000s [root@vm-01 ~]# [root@vm-01 ~]#

3 4

Call for Interest: Managed SMB Protocol Support
by John Mulligan 11 Apr '24

11 Apr '24

Hello Ceph List, I'd like to formally let the wider community know of some work I've been involved with for a while now: adding Managed SMB Protocol Support to Ceph. SMB being the well known network file protocol native to Windows systems and supported by MacOS (and Linux). The other key word "managed" meaning integrating with Ceph management tooling - in this particular case cephadm for orchestration and eventually a new MGR module for managing SMB shares. The effort is still in it's very early stages. We have a PR adding initial support for Samba Containers to cephadm [1] and a prototype for an smb MGR module [2]. We plan on using container images based on the samba-container project [3] - a team I am already part of. What we're aiming for is a feature set similar to the current NFS integration in Ceph, but with a focus on bridging non-Linux/Unix clients to CephFS using a protocol built into those systems. A few major features we have planned include: * Standalone servers (internally defined users/groups) * Active Directory Domain Member Servers * Clustered Samba support * Exporting Samba stats via Prometheus metrics * A `ceph` cli workflow loosely based on the nfs mgr module I wanted to share this information in case there's wider community interest in this effort. I'm happy to take your questions / thoughts / suggestions in this email thread, via Ceph slack (or IRC), or feel free to attend a Ceph Orchestration weekly meeting! I try regularly attend and we sometimes discuss design aspects of the smb effort there. It's on the Ceph Community Calendar. Thanks! [1] - https://github.com/ceph/ceph/pull/55068 [2] - https://github.com/ceph/ceph/pull/56350 [3] - https://github.com/samba-in-kubernetes/samba-container/ Thanks for reading, --John Mulligan

7 16

Cephadm host keeps trying to set osd_memory_target to less than minimum
by mads2a＠gmail.com 11 Apr '24

11 Apr '24

I have a virtual ceph cluster running 17.2.6 with 4 ubuntu 22.04 hosts in it, each with 4 OSD's attached. The first 2 servers hosting mgr's have 32GB of RAM each, and the remaining have 24gb For some reason i am unable to identify, the first host in the cluster appears to constantly be trying to set the osd_memory_target variable to roughly half of what the calculated minimum is for the cluster, i see the following spamming the logs constantly Unable to set osd_memory_target on my-ceph01 to 480485376: error parsing value: Value '480485376' is below minimum 939524096 Default is set to 4294967296. I did double check and osd_memory_base (805306368) + osd_memory_cache_min (134217728) adds up to minimum exactly osd_memory_target_autotune is currently enabled. But i cannot for the life of me figure out how it is arriving at 480485376 as a value for that particular host that even has the most RAM. Neither the cluster or the host is even approaching max utilization on memory, so it's not like there are processes competing for resources.

3 11

Have a problem with haproxy/keepalived/ganesha/docker
by ruslan.nurabayev＠kcell.kz 11 Apr '24

11 Apr '24

Hello! I've install my 5-node CEPH cluster next install NFS server by command: ceph nfs cluster create nfshacluster 5 --ingress --virtual_ip 192.168.171.48/26 --ingress-mode haproxy-protocol. I don't understand fully how this must be works but when i stop NFS daemon even on one of this nodes i've see that writing on NFS shares is disappear (testing via vdbench). As i understand it is wrong and IO from stopped daemon must swithcing to another NFS daemon without any impact on IO. Can someone help me with troubleshoot this issue? Or explain how done full-fledged Active-Active HA NFS Cluster for production use.

1 0

RGW/Lua script does not show logs
by soyoon.lee＠samsung.com 11 Apr '24

11 Apr '24

Hello, I wrote a Lua script in order to retrieve RGW logs such as bucket name, bucket owner, etc. However, when I apply a lua script I wrote using the below command, I do not see any logs start with Lua: INFO radosgw-admin script put --infile=/usr/tmp/testPreRequest.lua --context=postrequest <my lua script> function print_bucket_log(bucket) RGWDebugLog(" Name: " .. bucket.Name) end if Request.Bucket then RGWDebugLog("bucket operation logs: ") print_bucket_log(Request.Bucket) end According to the official document regarding Lua Scripting, The RGWDebugLog() function accepts a string and prints it to the debug log with priority 20. Each log message is prefixed Lua INFO:. This function has no return value. even though I set debug_rgw = 20, I do not see any logs. However, if when I apply the below lua script with bucket.Id, I am getting Lua: ERROR like below: Lua ERROR: [string "function print_bucket_log(bucket)..."]:3: attempt to concatenate a nil value (field 'Id') <my lua script> function print_bucket_log(bucket) RGWDebugLog(" Name: " .. bucket.Name) RGWDebugLog(" Id: " .. bucket.Id) end if Request.Bucket then RGWDebugLog("bucket operation logs: ") print_bucket_log(Request.Bucket) end Any help would be very appreciated!

3 2

A couple OSDs not starting after host reboot
by Alison Peisker 11 Apr '24

11 Apr '24

Hi all, We rebooted all the nodes in our 17.2.5 cluster after performing kernel updates, but 2 of the OSDs on different nodes are not coming back up. This is a production cluster using cephadm. The error message from the OSD log is ceph-osd[87340]: ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-665: (2) No such file or directory The error message from ceph-volume is 2023-08-23T16:12:43.452-0500 7f0cad968600 2 bluestore(/dev/mapper/ceph--febad5a5--ba44--41aa--a39e--b9897f757752-osd--block--87e548f4--b9b5--4ed8--aca8--de703a341a50) _read_bdev_label unable to decode label at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input We tried restarting the daemons and rebooting the node again, but still see the same error. Has anyone experienced this issue before? How do we fix this? Thanks, Alison

9 13