- ceph-users - lists.ceph.io

Ceph Reef 18.2.2 - stucked pgs in not scrubbed and deep-scrubbed in time
by Saint Kid 28 Dec '24

28 Dec '24

Hello, I have adjusted the scrubbing schedule and manually kick started the process but it appears the warning remains on these 15 PGs. I am thinking of restarting the OSDs which those PGs are and hoping to clear this condition. Any suggestions? Best Regards, Saint Kid

1 0

OSD_FULL after OSD Node Failures
by Gerard Hand 27 Dec '24

27 Dec '24

Hi, We recently had problems that meant 3 out of 32 OSD hosts went offline for about 10 minutes. The hosts are now back in the cluster as expected and backfilling going on. However we are seeing a couple of problems. We are seeing: 1. Ceph is flagging a handfull of PGs as backfill_toofull when they aren't. https://tracker.ceph.com/issues/61839. 2. Periodically it generates an OSD_FULL error. Are there any plans to look at resolving bug #61839? When an OSD_FULL error has been reported, the OSD in question has been <75% usage. The OSD wasn't used by any of the PGs reporting backfill_toofull. I currently have the full_ratio set to 0.97 and nearfull_ratio set to 0.87 so the OSDs are nowhere near these levels. The %raw usage of the OSDs in the cluster ranges from about 60-80% and the raw usage of the cluster is about 75%. We do not get any "near full" warnings prior to OSD_FULL being set. Having a production system instantly go offline without warning isn't ideal and these things seem to know the least convenient moment. These problems only happen after a host failure. Each time we have added additional OSD hosts into the cluster, the backfilling has finished without problems. We are currently running Reef 18.2.4 but I have experienced these same problems on Pacific 16.2.10 Has anyone else seen this behaviour? Thanks Gerard

2 2

TPM2 in squid 19.2.0
by Ehsan Golpayegani 27 Dec '24

27 Dec '24

Hi ceph users, Has anyone successfully installed Ceph Squid with TPM2 enabled on an OSD? I'm encountering an error during installation: "stderr_lines": ["Error EINVAL: Failed to validate OSD spec "<unnamed>": Feature `tpm2` is not supported"]. I've also attempted to prepare the disks with TPM2 before adding them to the cluster, but I receive the following error: "/usr/bin/podman: stderr ceph-volume lvm prepare: error: unrecognized arguments: --with-tpm". For reference, my cluster is deployed via Cephadm and I'm using Squid 19.2.0. Any guidance would be greatly appreciated. Thanks, Ehsan -- -------------- Kind Regards <http://about.me/ehsan310?promo=email_sig> Ehsan Golpayegani [image: https://]www.ehsan.pw <https://ehsan.pw?promo=email_sig&utm_source=product&utm_medium=email_sig&ut…> SAVE PAPER - THINK BEFORE YOU PRINT!

1 0

RGW sizing in multisite and rgw_run_sync_thread
by Adam Prycki 26 Dec '24

26 Dec '24

Hello, I was recently reading rook ceph multisite documentation and I've found this rgw recommendation: "Scaling the number of gateways that run the synchronization thread to 2 or more can increase the latency of the replication of each S3 object. The recommended way to scale a multisite configuration is to dissociate the gateway dedicated to the synchronization from gateways that serve clients." Are there any official ceph recommendations about how many RGWs with rgw_run_sync_thread=true we should deploy in multisite zone? Have you ever experienced issues related to this in production? My plan for our multisite was to create 3 RGW daemons with rgw_run_sync_thread, put them behind load balancer and put load balancer VIP in zone endpoint. Rook documentation suggests it would be sub optimal configuration. Best regards Adam Prycki

2 1

2024-12-26 Perf Meeting Cancelled
by Matt Vandermeulen 26 Dec '24

26 Dec '24

Hi folks, the perf meeting for today will be cancelled for Boxing Day! Happy holidays! We will (probably) resume next week, on the 2nd. Thanks, Matt

1 0

radosgw stopped working
by Rok Jaklič 24 Dec '24

24 Dec '24

Hi, for some reason radosgw stopped working. Cluster status: [root@ctplmon1 ~]# ceph -v ceph version 17.2.8 (f817ceb7f187defb1d021d6328fa833eb8e943b3) quincy (stable) [root@ctplmon1 ~]# ceph -s cluster: id: 0a6e5422-ac75-4093-af20-528ee00cc847 health: HEALTH_ERR 6 OSD(s) experiencing slow operations in BlueStore 2 backfillfull osd(s) 1 full osd(s) 1 nearfull osd(s) Low space hindering backfill (add storage if this doesn't resolve itself): 32 pgs backfill_toofull Degraded data redundancy: 835306/1285383707 objects degraded (0.065%), 6 pgs degraded, 5 pgs undersized 76 pgs not deep-scrubbed in time 45 pgs not scrubbed in time Full OSDs blocking recovery: 1 pg recovery_toofull 9 pool(s) full 9 daemons have recently crashed services: mon: 3 daemons, quorum ctplmon1,ctplmon3,ctplmon2 (age 36m) mgr: ctplmon1(active, since 65m) mds: 1/1 daemons up osd: 193 osds: 191 up (since 8m), 191 in (since 9m); 267 remapped pgs rgw: 2 daemons active (1 hosts, 1 zones) data: volumes: 1/1 healthy pools: 10 pools, 793 pgs objects: 257.08M objects, 292 TiB usage: 614 TiB used, 386 TiB / 1000 TiB avail pgs: 835306/1285383707 objects degraded (0.065%) 225512620/1285383707 objects misplaced (17.544%) 525 active+clean 230 active+remapped+backfilling 32 active+remapped+backfill_toofull 5 active+undersized+degraded+remapped+backfilling 1 active+recovery_toofull+degraded io: recovery: 978 MiB/s, 825 objects/s --- Do not know if it is related but the cluster has been rebalancing for a few days now, after I've set EC pool only to use hdd. --- If I start rgw with debug I get something like this in logs: [root@ctplmon2 ~]# radosgw -c /etc/ceph/ceph.conf --setuser ceph --setgroup ceph -n client.radosgw.moja.shramba.ctplmon2 -f -m 194.249.4.104:6789 --debug-rgw=99/99 2024-12-21T23:21:59.898+0100 7f659e380640 -1 Initialization timeout, failed to initialize In logs I get: 2024-12-21T23:16:59.898+0100 7f65a19257c0 0 deferred set uid:gid to 167:167 (ceph:ceph) 2024-12-21T23:16:59.898+0100 7f65a19257c0 0 ceph version 17.2.8 (f817ceb7f187defb1d021d6328fa833eb8e943b3) quincy (stable), process radosgw, pid 168935 2024-12-21T23:16:59.898+0100 7f65a19257c0 0 framework: beast 2024-12-21T23:16:59.898+0100 7f65a19257c0 0 framework conf key: port, val: 4444 2024-12-21T23:16:59.898+0100 7f65a19257c0 1 radosgw_Main not setting numa affinity 2024-12-21T23:16:59.901+0100 7f65a19257c0 1 rgw_d3n: rgw_d3n_l1_local_datacache_enabled=0 2024-12-21T23:16:59.901+0100 7f65a19257c0 1 D3N datacache enabled: 0 2024-12-21T23:16:59.901+0100 7f658dffb640 20 reqs_thread_entry: start 2024-12-21T23:16:59.901+0100 7f658d7fa640 10 entry start 2024-12-21T23:16:59.908+0100 7f65a19257c0 20 rgw main: rados->read ofs=0 len=0 2024-12-21T23:16:59.914+0100 7f65a19257c0 20 rgw main: rados_obj.operate() r=-2 bl.length=0 2024-12-21T23:16:59.914+0100 7f65a19257c0 20 rgw main: realm 2024-12-21T23:16:59.914+0100 7f65a19257c0 20 rgw main: rados->read ofs=0 len=0 2024-12-21T23:16:59.915+0100 7f65a19257c0 20 rgw main: rados_obj.operate() r=-2 bl.length=0 2024-12-21T23:16:59.915+0100 7f65a19257c0 4 rgw main: RGWPeriod::init failed to init realm id : (2) No such file or directory 2024-12-21T23:16:59.915+0100 7f65a19257c0 20 rgw main: rados->read ofs=0 len=0 2024-12-21T23:16:59.915+0100 7f65a19257c0 20 rgw main: rados_obj.operate() r=-2 bl.length=0 2024-12-21T23:16:59.915+0100 7f65a19257c0 20 rgw main: rados->read ofs=0 len=0 2024-12-21T23:16:59.917+0100 7f65a19257c0 20 rgw main: rados_obj.operate() r=0 bl.length=46 2024-12-21T23:16:59.917+0100 7f65a19257c0 20 rgw main: rados->read ofs=0 len=0 2024-12-21T23:16:59.945+0100 7f65a19257c0 20 rgw main: rados_obj.operate() r=0 bl.length=873 2024-12-21T23:16:59.945+0100 7f65a19257c0 20 rgw main: searching for the correct realm 2024-12-21T23:17:00.210+0100 7f65a19257c0 20 rgw main: RGWRados::pool_iterate: got zone_info.c2c70444-7a41-4acd-a0d0-9f87d324ec72 2024-12-21T23:17:00.210+0100 7f65a19257c0 20 rgw main: RGWRados::pool_iterate: got zonegroup_info.b1e0d55c-f7cb-4e73-b1cb-6cffa1fd6578 2024-12-21T23:17:00.210+0100 7f65a19257c0 20 rgw main: RGWRados::pool_iterate: got zone_names.default 2024-12-21T23:17:00.210+0100 7f65a19257c0 20 rgw main: RGWRados::pool_iterate: got zonegroups_names.default 2024-12-21T23:17:00.210+0100 7f65a19257c0 20 rgw main: rados->read ofs=0 len=0 2024-12-21T23:17:00.210+0100 7f65a19257c0 20 rgw main: rados_obj.operate() r=-2 bl.length=0 2024-12-21T23:17:00.210+0100 7f65a19257c0 20 rgw main: rados->read ofs=0 len=0 2024-12-21T23:17:00.211+0100 7f65a19257c0 20 rgw main: rados_obj.operate() r=0 bl.length=46 2024-12-21T23:17:00.211+0100 7f65a19257c0 20 rgw main: rados->read ofs=0 len=0 2024-12-21T23:17:00.212+0100 7f65a19257c0 20 rgw main: rados_obj.operate() r=0 bl.length=358 2024-12-21T23:17:00.212+0100 7f65a19257c0 20 rgw main: rados->read ofs=0 len=0 2024-12-21T23:17:00.213+0100 7f65a19257c0 20 rgw main: rados_obj.operate() r=-2 bl.length=0 2024-12-21T23:17:00.213+0100 7f65a19257c0 20 rgw main: rados->read ofs=0 len=0 2024-12-21T23:17:00.214+0100 7f65a19257c0 20 rgw main: rados_obj.operate() r=-2 bl.length=0 2024-12-21T23:17:00.214+0100 7f65a19257c0 20 rgw main: rados->read ofs=0 len=0 2024-12-21T23:17:00.215+0100 7f65a19257c0 20 rgw main: rados_obj.operate() r=0 bl.length=46 2024-12-21T23:17:00.284+0100 7f65a19257c0 20 rgw main: RGWRados::pool_iterate: got zone_info.c2c70444-7a41-4acd-a0d0-9f87d324ec72 2024-12-21T23:17:00.284+0100 7f65a19257c0 20 rgw main: RGWRados::pool_iterate: got zonegroup_info.b1e0d55c-f7cb-4e73-b1cb-6cffa1fd6578 2024-12-21T23:17:00.284+0100 7f65a19257c0 20 rgw main: RGWRados::pool_iterate: got zone_names.default 2024-12-21T23:17:00.284+0100 7f65a19257c0 20 rgw main: RGWRados::pool_iterate: got zonegroups_names.default 2024-12-21T23:17:00.284+0100 7f65a19257c0 20 rgw main: rados->read ofs=0 len=0 2024-12-21T23:17:00.285+0100 7f65a19257c0 20 rgw main: rados_obj.operate() r=0 bl.length=46 2024-12-21T23:17:00.285+0100 7f65a19257c0 20 rgw main: rados->read ofs=0 len=0 2024-12-21T23:17:00.286+0100 7f65a19257c0 20 rgw main: rados_obj.operate() r=0 bl.length=873 2024-12-21T23:17:00.286+0100 7f65a19257c0 20 rgw main: rados->read ofs=0 len=0 2024-12-21T23:17:00.287+0100 7f65a19257c0 20 rgw main: rados_obj.operate() r=0 bl.length=46 2024-12-21T23:17:00.287+0100 7f65a19257c0 20 rgw main: rados->read ofs=0 len=0 2024-12-21T23:17:00.293+0100 7f65a19257c0 20 rgw main: rados_obj.operate() r=0 bl.length=358 2024-12-21T23:17:00.293+0100 7f65a19257c0 20 rgw main: rados->read ofs=0 len=0 2024-12-21T23:17:00.295+0100 7f65a19257c0 20 rgw main: rados_obj.operate() r=-2 bl.length=0 2024-12-21T23:17:00.295+0100 7f65a19257c0 20 zone default found 2024-12-21T23:17:00.295+0100 7f65a19257c0 4 rgw main: Realm: () 2024-12-21T23:17:00.295+0100 7f65a19257c0 4 rgw main: ZoneGroup: default (b1e0d55c-f7cb-4e73-b1cb-6cffa1fd6578) 2024-12-21T23:17:00.295+0100 7f65a19257c0 4 rgw main: Zone: default (c2c70444-7a41-4acd-a0d0-9f87d324ec72) 2024-12-21T23:17:00.295+0100 7f65a19257c0 10 cannot find current period zonegroup using local zonegroup configuration 2024-12-21T23:17:00.295+0100 7f65a19257c0 20 rgw main: zonegroup default 2024-12-21T23:17:00.295+0100 7f65a19257c0 20 rgw main: rados->read ofs=0 len=0 2024-12-21T23:17:00.296+0100 7f65a19257c0 20 rgw main: rados_obj.operate() r=-2 bl.length=0 2024-12-21T23:17:00.296+0100 7f65a19257c0 20 rgw main: rados->read ofs=0 len=0 2024-12-21T23:17:00.299+0100 7f65a19257c0 20 rgw main: rados_obj.operate() r=-2 bl.length=0 2024-12-21T23:17:00.299+0100 7f65a19257c0 20 rgw main: rados->read ofs=0 len=0 2024-12-21T23:17:00.303+0100 7f65a19257c0 20 rgw main: rados_obj.operate() r=-2 bl.length=0 2024-12-21T23:17:00.303+0100 7f65a19257c0 20 rgw main: started sync module instance, tier type = 2024-12-21T23:17:00.303+0100 7f65a19257c0 20 rgw main: started zone id=c2c70444-7a41-4acd-a0d0-9f87d324ec72 (name=default) with tier type = 2024-12-21T23:21:59.898+0100 7f659e380640 -1 Initialization timeout, failed to initialize --- Any ideas what might cause rgw to stop working? Kind regards, Rok

6 21

Issue With Dasboard TLS Certificate (Renewal)
by duluxoz 23 Dec '24

23 Dec '24

Hi All, So we've been using the Ceph (v18.2.4) Dashboard with internally generated TLS Certificates (via our Step-CA CA), one for each of our three Ceph Manager Nodes. Everything was working AOK. The TLS Certificates came up for renewal, which they were successfully renewed. Accordingly, the old Certificates & Keys were overwritten by the new ones and the commands `ceph mgr module disable/enabled dashboard` (respectively) were run. HOWEVER, the Ceph Dashboard stopped working / wouldn't use the renewed Certificates; as per the logs the Dashboard was still using the old Certificates and is now complaining that they have expired, and therefore the Dashboard won't run (unless I disable SSL via the CLI). I've been through the documentation and I can't work out what I've done wrong; according to this page (https://docs.ceph.com/en/latest/mgr/dashboard/#dashboard-ssl-tls-support) - in particular the Blue-Box Note - there's nothing else that needs to be done - EXCEPT that I've done all of that and the logs still tell me the Certificate(s) has expired - ie I can't get the Dashboard to recognise the new Certificates. Any help greatly appreciated. Thanks in advance Dulux-Oz

3 4

TPM2 capabilities
by Ehsan Golpayegani 22 Dec '24

22 Dec '24

Hi ceph users, Has anyone successfully installed Ceph Squid with TPM2 enabled on an OSD? I'm encountering an error during installation: "stderr_lines": ["Error EINVAL: Failed to validate OSD spec "<unnamed>": Feature `tpm2` is not supported"]. I've also attempted to prepare the disks with TPM2 before adding them to the cluster, but I receive the following error: "/usr/bin/podman: stderr ceph-volume lvm prepare: error: unrecognized arguments: --with-tpm". For reference, my cluster is deployed via Cephadm and I'm using Squid 19.2.0. Any guidance would be greatly appreciated. Thanks, Ehsan -------------- Kind Regards <http://about.me/ehsan310?promo=email_sig> Ehsan Golpayegani [image: https://]www.ehsan.pw <https://ehsan.pw?promo=email_sig&utm_source=product&utm_medium=email_sig&ut…> SAVE PAPER - THINK BEFORE YOU PRINT!

1 0

EC pool only for hdd
by Rok Jaklič 21 Dec '24

21 Dec '24

Hi, is it possible to set/change following already used rule to only use hdd? { "rule_id": 1, "rule_name": "ec32", "type": 3, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_indep", "num": 0, "type": "host" }, { "op": "emit" } ] } Kind regards, Rok

4 15

Upgrade stalled after upgrading managers
by Torkil Svensgaard 21 Dec '24

21 Dec '24

Hi Running upgrade from 18.2.4 to 19.2.0 and it managed to upgrade the managers but no further progress. If I fail over the mgr it goes: " [root@ceph-flash1 ~]# ceph orch upgrade status Error ENOTSUP: Module 'orchestrator' is not enabled/loaded (required by command 'orch upgrade status'): use `ceph mgr module enable orchestrator` to enable it " From mgr log: " ... 2024-12-17T10:43:11.729+0000 7f70efafe640 0 log_channel(audit) log [DBG] : from='client.2110010386 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-12-17T10:43:11.733+0000 7f70ebaf6640 0 [cephadm INFO cherrypy.error] [17/Dec/2024:10:43:11] ENGINE Bus STARTING 2024-12-17T10:43:11.733+0000 7f70ebaf6640 0 log_channel(cephadm) log [INF] : [17/Dec/2024:10:43:11] ENGINE Bus STARTING 2024-12-17T10:43:11.811+0000 7f70e7aee640 0 [dashboard INFO dashboard.module] Engine started. 2024-12-17T10:43:11.861+0000 7f70ebaf6640 0 [cephadm INFO cherrypy.error] [17/Dec/2024:10:43:11] ENGINE Serving on https://172.21.15.148:7150 2024-12-17T10:43:11.861+0000 7f70ebaf6640 0 log_channel(cephadm) log [INF] : [17/Dec/2024:10:43:11] ENGINE Serving on https://172.21.15.148:7150 2024-12-17T10:43:11.864+0000 7f70a2d7a640 0 [cephadm ERROR cherrypy.error] [17/Dec/2024:10:43:11] ENGINE Error in HTTPServer.serve Traceback (most recent call last): File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve self._connections.run(self.expiration_interval) File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run self._run(expiration_interval) File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run new_conn = self._from_server_socket(self.server.socket) File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket s, ssl_env = self.server.ssl_adapter.wrap(s) File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap s = self.context.wrap_socket( File "/lib64/python3.9/ssl.py", line 501, in wrap_socket return self.sslsocket_class._create( File "/lib64/python3.9/ssl.py", line 1074, in _create self.do_handshake() File "/lib64/python3.9/ssl.py", line 1343, in do_handshake self._sslobj.do_handshake() ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-12-17T10:43:11.865+0000 7f70a2d7a640 -1 log_channel(cephadm) log [ERR] : [17/Dec/2024:10:43:11] ENGINE Error in HTTPServer.serve Traceback (most recent call last): File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve self._connections.run(self.expiration_interval) File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run self._run(expiration_interval) File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run new_conn = self._from_server_socket(self.server.socket) File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket s, ssl_env = self.server.ssl_adapter.wrap(s) File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap s = self.context.wrap_socket( File "/lib64/python3.9/ssl.py", line 501, in wrap_socket return self.sslsocket_class._create( File "/lib64/python3.9/ssl.py", line 1074, in _create self.do_handshake() File "/lib64/python3.9/ssl.py", line 1343, in do_handshake self._sslobj.do_handshake() ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-12-17T10:43:11.963+0000 7f70ebaf6640 0 [cephadm INFO cherrypy.error] [17/Dec/2024:10:43:11] ENGINE Serving on http://172.21.15.148:8765 2024-12-17T10:43:11.963+0000 7f70ebaf6640 0 log_channel(cephadm) log [INF] : [17/Dec/2024:10:43:11] ENGINE Serving on http://172.21.15.148:8765 2024-12-17T10:43:11.963+0000 7f70ebaf6640 0 [cephadm INFO cherrypy.error] [17/Dec/2024:10:43:11] ENGINE Bus STARTED 2024-12-17T10:43:11.964+0000 7f70ebaf6640 0 log_channel(cephadm) log [INF] : [17/Dec/2024:10:43:11] ENGINE Bus STARTED ... " It will recover after some timeout, maybe 5-10 mins, and then just sit there with no upgrade progress. Nothing in mgr/cephadm/osd_remove_queue. Suggestions? Mvh. Torkil -- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil(a)drcmr.dk

3 5