Hi, the ceph query states: "recovery_state": [ { "name": "Started/Primary/Peering/Down", "enter_time": "2024-09-16T17:48:13.572414+0200", "comment": "not enough up instances of this PG to go active" }, Its missing an OSD (shard 4=2147483647 means "none"): "up": [ 20, 253, 254, 84, 2147483647, 56 ], "acting": [ 20, 253, 254, 84, 2147483647, 56 ], The OSD in this place is down and for some reason ceph cannot find another OSD to take its place. Fastest way forward is probably to get this OSD up again and then look why ceph couldn't assign a replacement. Depending on number of hosts and how your crush rule is defined, you might be in the "ceph gives up too soon" situation or similar. PS: "We wipe the OSD 11, 148 and 280 (one by one and waiting of course the peering to avoid data loss on other PGs)." I hope you mean "waited for recovery" or what does a wipe here mean. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: HARROUIN Loan (PRESTATAIRE CA-GIP) <loan.harrouin-prestataire@ca-gip.fr> Sent: Monday, September 16, 2024 7:33 PM To: ceph-users@ceph.io Cc: CAGIP_DEVOPS_OPENSTACK Subject: [ceph-users] [Ceph incident] PG stuck in peering. Hello dear ceph community, We are facing a strange issue this weekend with a pg (13.6a) that is stuck in peering. Because of that we got lot of ops stuck of course. We are running a ceph in Pacific version 16.2.10, we have only SSD disk and are using erasure coding. cluster: id: f5c69b4a-89e0-4055-95f7-eddc6800d4fe health: HEALTH_WARN Reduced data availability: 1 pg inactive, 1 pg peering 256 slow ops, oldest one blocked for 5274 sec, osd.20 has slow ops services: mon: 3 daemons, quorum cos1-dal-ceph-mon-01,cos1-dal-ceph-mon-02,cos1-dal-ceph-mon-03 (age 17h) mgr: cos1-dal-ceph-mon-02(active, since 17h), standbys: cos1-dal-ceph-mon-03, cos1-dal-ceph-mon-01 osd: 647 osds: 646 up (since 27m), 643 in (since 2h) data: pools: 7 pools, 1921 pgs objects: 432.65M objects, 1.6 PiB usage: 2.4 PiB used, 2.0 PiB / 4.4 PiB avail pgs: 0.052% pgs not active 1916 active+clean 2 active+clean+scrubbing 2 active+clean+scrubbing+deep 1 peering The ‘ceph pg 13.6a query’ hung, so we must restart one of the osd that are part of this PG to temporary unhung the query (because during some seconds the pg isn’t peering yet). In that case, the query only retrieves the information about the shard that was hosted on the OSD that we restart. The result of the query is in attachment (shard 0). First when the issue occurs, we check the logs and restart all the osd linked to this PG. Sadly, it didn’t fix anything. We try to investigate the peering state to understand what was going on the primary OSD. We put the OSD in debug but at first glance anything seems strange (we are not use to deep dive that much into ceph). We find that CERN faced something similar a long time ago: https://indico.cern.ch/event/617118/contributions/2490930/attachments/142279... After reading it, we try to do the empty OSD method that they tried (diapo7). We identify that the shard0 seem in a weird state (and was primary) so it was our candidate. We wipe the OSD 11, 148 and 280 (one by one and waiting of course the peering to avoid data loss on other PGs). After that, the OSD.20 was now elected as the primary but still, the PG stay huge in peering and now all OPS are stuck on OSD.20. We are now in the dark. We plan to maybe deep dive deeper into the log of this new OSD.20, and see if we can plan to upgrade our ceph in order to have the most recent version. Any help or suggestion is welcome 😊