- ceph-users - lists.ceph.io

Re: tentacle 20.2.0 GA QE validation status
by Venky Shankar 22 Oct '25

22 Oct '25

(I sent my approval email from ists.ceph.io/hyperkitty as Yuri's original email landed in spam :/ adding users list back to this chain) On Wed, Oct 22, 2025 at 1:00 PM Venky Shankar <vshankar(a)redhat.com> wrote: > > Hi Yuri, > > fs approved. failures are > > https://tracker.ceph.com/projects/cephfs/wiki/Tentacle#2020 > > Cheers, > Venky > _______________________________________________ > Dev mailing list -- dev(a)ceph.io > To unsubscribe send an email to dev-leave(a)ceph.io > -- Cheers, Venky

1 0

rbd volume map
by Iban Cabrillo 21 Oct '25

21 Oct '25

Good morning, I am having trouble mapping volumes using rbd. It used to work with v1 (I haven't tried it in a couple of years), but now we are using Quincy (v17) and I am getting this error: rbd --mon-host 10.10.3.1:3300,10.10.3.2:3300,10.10.3.3:3300 -p volumes map volume-c33b1ea9-c976-438b-8c3c-21cdf2db8bfd rbd: sysfs write failed In some cases useful info is found in syslog - try “dmesg | tail”. The mons are available and responding on port 3300. The volume I want to map is available, and I have disabled features: rbd --mon-host 10.10.3.1:3300,10.10.3.2:3300,10.10.3.3:3300 -p volumes info volume-c33b1ea9-c976-438b-8c3c-21cdf2db8bfd rbd image ‘volume-c33b1ea9-c976-438b-8c3c-21cdf2db8bfd’: size 200 GiB in 51200 objects order 22 (4 MiB objects) snapshot_count: 0 id: f6b4e9515f007c block_name_prefix: rbd_data.f6b4e9515f007c format: 2 features: layering op_features: flags: create_timestamp: Wed Dec 15 20:11:41 2021 access_timestamp: Thu Oct 2 12:04:14 2025 modify_timestamp: Wed Oct 15 17:36:12 2025 It does NOT seem to be an authentication issue, as I have used the admin user and key and the result is the same. Any ideas are welcome. Regards, I -- Jose Ibán Cabrillo Bartolomé cabrillo(a)ifca.unican.es Tlfno (+34) 942200969 Servicio de Computación Avanzada - Advanced Computing Service advancedcomputing.ifca.es Instituto de Física de Cantabria (IFCA) | CSIC · UC Ed. Juan Jordá, Avda. de los Castros s/n - 39005 Santander (Spain) www.ifca.es --- --- --- Políticas del Servicio de Computación Avanzada disponibles en / Advanced Computing Service policies available at : https://confluence.ifca.es/x/HAAt Este mensaje, y los ficheros adjuntos que pueda incluir, puede contener información confidencial o legalmente protegida y está exclusivamente dirigido a la persona o entidad destinataria. Si usted no consta como destinatario final ni es la persona encargada de recibirlo, no está autorizado a leerlo, retenerlo, modificarlo, distribuirlo o copiarlo, ni a revelar su contenido. Si lo ha recibido por error, informe de ello al remitente y elimine del sistema tanto el mensaje como los ficheros adjuntos que pueda contener. This email message and any attachments it carries may contain confidential or legally protected material and are intended solely for the individual or organization to whom they are addressed. If you are not the intended recipient of this message or the person responsible for processing it, then you are not authorized to read, save, modify, send, copy or disclose any part of it. If you have received the message by mistake, please inform the sender of this and eliminate the message and any attachments it carries from your account.

2 2

How to check if a used disk/SSD is ready to become an OSD?
by filip Mutterer 21 Oct '25

21 Oct '25

Hi! How to check if a used disk/SSD is ready to become an OSD? The thing is I don't know for sure if I prepared the disks already to become an OSD. Greetings filip

2 1

OSDs not balancing
by Ryan Sleeth 21 Oct '25

21 Oct '25

Hello, I am new to Ceph and still learning. I setup a test cluster and copied a lot of 'torture' data to it (20 TB with many files under 4KB). I copied this data to different pools configured for different compression plugins (snappy vs. lz4 vs. zstd), required ratios (default 87 vs. 97), and EC plugins (jerasure vs. isa). All totaled, the pools add up to ~440 TB (out of 1.3 PB) all using EC 4+2. Perhaps the autobalancer is misbehaving because many OSDs were 'stuck' at 1 PG until I manually specified them to be 256--now they are catching up. What is not showing any progress is: 1) I have one HDD OSD at 87% full, the next highest at 69%, and the lowest at 12% and 2) a huge number of PGs not deep-scrubbed in time; probably can ignore this as its a result of the dramatic PG changes The cluster is 9x Dell PowerEdge R730xd each with 8x 20 TB HDDs and 2x 2 TB enterprise NVMes configured for db; setup on Ubuntu 22 with Tentacle 20.1.0 RC0 upgraded from Squid and running with Docker/cephadm. Perhaps I am being impatient waiting for it to rebalance? Does the community have any suggestions about how to approach this? PG autobalancer seems fine. cluster: id: mypool health: HEALTH_WARN 1 nearfull osd(s) 1392 pgs not deep-scrubbed in time 217 pgs not scrubbed in time 21 pool(s) nearfull services: mon: 5 daemons, quorum ceph01,ceph02,ceph03,ceph06,ceph05 (age 2w) [leader: ceph01] mgr: ceph01.qfwdee(active, since 2w), standbys: ceph02.imozkz mds: 2/2 daemons up, 1 standby osd: 90 osds: 90 up (since 2w), 90 in (since 3w); 863 remapped pgs data: volumes: 2/2 healthy pools: 25 pools, 3051 pgs objects: 396.39M objects, 326 TiB usage: 440 TiB used, 896 TiB / 1.3 PiB avail pgs: 428953812/1897506717 objects misplaced (22.606%) 2159 active+clean 861 active+remapped+backfill_wait 16 active+clean+scrubbing+deep 13 active+clean+scrubbing 2 active+remapped+backfilling io: recovery: 29 MiB/s, 21 objects/s -- Ryan Sleeth

2 2

What does logrotate configs?
by lejeczek 17 Oct '25

17 Oct '25

Hi guys. In a containerized env what creates: /etc/logrotate.d/cephadm but also other ceph's logorates? That one above seems to be watched constantly for if removed is recreated almost immediately. It also causes OS' 'lograte' to fail for its content conflict with other ceph-created logrotate files. btw. anybody sees this/similar? many thanks, L.

2 1

DriveGroup Spec question
by Robert Sander 17 Oct '25

17 Oct '25

Hi, we have OSD nodes currently consisting of two 605GB SSDs and six 18TB HDDs. The hosts have room for twelve HDDs. We created a drivegroup spec that looks like this: spec: block_db_size: 100GB data_devices: rotational: true size: '18TB:' db_devices: rotational: false size: '550GB:650GB' db_slots: 6 Initially this creates 6 OSDs with their RocksDB+WAL on the SSDs, 3 each which is nice for load balancing. But when we add another HDD it gets a 17.9TB data volume and a 100GB DB volume, both on the HDD: sdm 8:192 0 18T 0 disk ├─ceph--846e1a59--aff6--4ef8--9b71--de7241531677-osd--block--026e8cef--123d--47d9--9b30--211f94edf96c 252:16 0 17.9T 0 lvm └─ceph--846e1a59--aff6--4ef8--9b71--de7241531677-osd--db--88c47d0b--f5c6--4cec--8909--c5f8036ca459 252:17 0 100G 0 lvm I would have assumed that the remaining 305GB on the SSDs would be used. How do we achieve this? Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 220009 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

2 9

CSC Meeting Notes - 09/29
by Venky Shankar 16 Oct '25

16 Oct '25

- [Ernesto] SCM attack prevention Ceph org repos: request to pin all GH Action deps to SHA-1 commits (instead of version tags). - To decide: deadline (proposed 1 month, end of October) - More context: https://lists.ceph.io/hyperkitty/list/dev@ceph.io/thread/5OJIR3OI2KES3L3QNW… - https://github.com/ceph/ceph/pull/65720 - [Yuri] Tentacle RC1 status: https://tracker.ceph.com/issues/73123 - [Yuri] Approval in process for many - Yuri to follow up individually with tech leads - DONE - LRC upgrades? - [Neha] should plan for an LRC upgrade since this is the last RC - [Yuri] would RC be GA? - [Neha] RC1 would be GA and would _not_ be a full rebase - [Neha] Reassess next RC if we get community feedback for RC1 - [Yuri] - bookworm: issue fixed but not easy to test (build only) - [David] No teuthology runs for Debian, so build only! - noble support? - [Casey] lack of test coverage for qa suites and will not happen for "T" - [Adam King] https://github.com/ceph/ceph/blob/tentacle/src/cephadm/cephadmlib/constants… - https://github.com/ceph/ceph/pull/65719 - [Neha] Start working on Release Notes - Neha and Anthony D to collaborate on slack (ceph-upstream-release channel) - Lab Update - [David] status: probably a month to get a lab running - sepia lab shutdown is likely to be December '25 - target "T" GA before hardware move - [Patrick] Election? - Anthony M is taking a lead on this - Anthony M to share updates in CSC channel (or in CSC meeting) -- Cheers, Venky

3 2

tls certs per manager - does it work?
by lejeczek 16 Oct '25

16 Oct '25

Hi guys. I thought I had a problem with restarting dashboard - following in docs on ssl certs. But ! I might have other issue - restarting dashboard does what is expected of - but, with these: -> $ ceph dashboard set-ssl-certificate-key podster2.mine.priv -i /root/podster2.mine.priv.key -> $ ceph dashboard set-ssl-certificate podster2.mine.priv -i /root/podster2.mine.priv.crt -> $ ceph dashboard set-ssl-certificate-key podster1.mine.priv -i /root/podster1.mine.priv.key -> $ ceph dashboard set-ssl-certificate podster1.mine.priv -i /root/podster1.mine.priv.crt If I go to podster2 I get podster1's cert, perhaps these do not do anything, If I do: -> $ ceph dashboard set-ssl-certificate -i /root/podster2.mine.priv.crt -> $ ceph dashboard set-ssl-certificate-key -i /root/podster2.mine.priv.key then podster1 has podster2's cert - naturally - but (weirdly?) if go to podster2 (by FQDN) then I get redirected to IP (as in URL) of when I do: -> $ ceph dashboard set-ssl-certificate -i /root/podster1.mine.priv.crt -> $ ceph dashboard set-ssl-certificate-key -i /root/podster1.mine.priv.key and I go to podster2 then, no redirection, "only" warning about cert being of podster1's, podster1 has a good cert! If I do (again) - having podster1 have "correct" cert, with no per-node/manager cert - per-node/mgr cert, as shown earlier, then.. again, podster1 has its correct cert, podster2 has podster1's cert (no ! redirection to IP though). How much of an "issue" this might be - given the fact that I deployed anew cluster twice (lab) - and this reproduces each time. Or, I'm missing some obvious, big picture - and it's simply bad luck, deploying cluster twice. ceph version 18.2.7 (6b0e988052ec84cf2d4a54ff9bbbc5e720b621ad) reef (stable) all thoughts are much appreciated. many thanks, L.

2 9

CephFS mount error after successful attach of volume
by Martin Reid 15 Oct '25

15 Oct '25

Hey everyone, I've got a difficult problem with my CephFS that I haven't been able to make any headway with. Maybe you guys can help? The problem I’m trying to provision a volume on a CephFS, using a Ceph cluster installed on Kubernetes (K3s) using Rook, but I’m running into the following error (from the Events in |kubectl describe|: |Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 4m24s default-scheduler Successfully assigned archie/ceph-loader-7989b64fb5-m8ph6 to archie Normal SuccessfulAttachVolume 4m24s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-95b6ca46-cf51-4e58-9bb5-114f00aa4267" Warning FailedMount 3m18s kubelet MountVolume.MountDevice failed for volume "pvc-95b6ca46-cf51-4e58-9bb5-114f00aa4267" : rpc error: code = Internal desc = an error (exit status 32) occurred while running mount args: [-t ceph csi-cephfs-node.1@039a3dba-d55c-476f-90f0-8783a18338aa.main-ceph-fs=/volumes/csi/csi-vol-25d616f5-918f-4e15-bfd6-55b866f9aa9f/4bda56a4-5088-451c-90c8-baa83317d5a5 /var/lib/kubelet/plugins/kubernetes.io/csi/rook-ceph.cephfs.csi.ceph.com/3e10b46e93bcc2c4d3d1b343af01ee628c736ffee7e562e99d478bc397dab10d/globalmount -o mon_addr=10.43.233.111:3300/10.43.237.205:3300/10.43.39.81:3300,secretfile=/tmp/csi/keys/keyfile-2996214224,_netdev] stderr: mount error: no mds (Metadata Server) is up. The cluster might be laggy, or you may not be authorized| I’m kind of new to K8s, and /very/ new to Ceph, so I would love some advice on how to go about debugging this mess. General context *Kubernetes distribution*: K3s *Kubernetes version(s)*: v1.33.4+k3s1 (master), v1.32.7+k3s1 (workers) *Ceph*: installed via Rook *Nodes*: 3 *OS*: Linux (Arch on master, NixOS on workers) What I’ve checked/tried *Note*: Since this is a Rook deployment of Ceph (on Kubernetes), all these checks are performed in the Rook Toolbox <https://rook.io/docs/rook/latest-release/Troubleshooting/ceph-toolbox/> container. MDS status / Ceph cluster health Even I know this is the first go-to when your Ceph cluster is giving you issues. I have the Rook toolbox <https://rook.io/docs/rook/latest-release/Troubleshooting/ceph-toolbox/> running on my K8s cluster, so I went into the toolbox pod and ran: |$ ceph status cluster: id: 039a3dba-d55c-476f-90f0-8783a18338aa health: HEALTH_OK| services: mon: 3 daemons, quorum a,c,b (age 7d) mgr: b(active, since 7d), standbys: a mds: 1/1 daemons up, 1 hot standby osd: 3 osds: 3 up (since 7d), 3 in (since 2w) data: volumes: 1/1 healthy pools: 4 pools, 81 pgs objects: 47 objects, 3.2 MiB usage: 139 MiB used, 502 GiB / 502 GiB avail pgs: 81 active+clean io: client: 1.2 KiB/s rd, 2 op/s rd, 0 op/s wr Since the error we started out with |mount error: no mds (Metadata Server) is up|, I checked the |ceph status| output above for the status of the metadata server. As you can see, all the MDS instances are running. CephFS Status |$ ceph fs status main-ceph-fs - 0 clients ============ RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active main-ceph-fs-a Reqs: 0 /s 143 38 37 0 0-s standby-replay main-ceph-fs-b Evts: 0 /s 159 30 29 0 POOL TYPE USED AVAIL main-ceph-fs-metadata metadata 4176k 158G main-ceph-fs-replicated data 0 158G main-ceph-fs-main-ceph-fs-replicated data 0 158G STANDBY MDS main-ceph-fs-d main-ceph-fs-c MDS version: ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable)| Ceph authorizations for MDS Since the other part of the error indicated that I might not be authorized, I wanted to check what the authorizations were: |$ ceph auth ls mds.main-ceph-fs-a # main MDS for my CephFS instance key: <base64 key> caps: [mds] allow caps: [mon] allow profile mds caps: [osd] allow * mds.main-ceph-fs-b # standby MDS for my CephFS instance key: <different base64 key> caps: [mds] allow caps: [mon] allow profile mds caps: [osd] allow * ... client.csi-cephfs-node.1 # the client mentioned in the error message key: <another base64 key> caps: [mds] allow rw caps: [mgr] allow rw caps: [mon] allow r caps: [osd] allow rwx tag cephfs metadata=*, allow rw tag cephfs data=* ... # more after this| Note: |main-ceph-fs| is the name I gave my CephFS file system. It looks like this should be okay, but I’m not sure. Definitely open to some more insight here. PersistentVolumeClaim binding I checked to make sure the PersistentVolume was provisioned successfully from the PersistentVolumeClaim, and that it bound appropriately: |$ kubectl get pvc -n archie jellyfin-ceph-pvc NAME STATUS VOLUME CAPACITY jellyfin-ceph-pvc Bound pvc-95b6ca46-cf51-4e58-9bb5-114f00aa4267 180Gi | Changing the PVC size to something smaller I tried changing the PVC’s size from 180GB to 1GB, since I thought it might be a free space issue,but the error persisted. Turning off firewalls Turned off all firewall to see if it was that, and still no luck. I’m not quite sure where to go from here. What am I missing? What context should I add? What should I try? What should I check? Thank you so much in advance, Martin

2 1

Critical cluster issues after upgrade from Ceph 15.2.17 to 17.2.7 - OSDs failing with "missing pg_pool_t for deleted pool" error
by kefu chai 15 Oct '25

15 Oct '25

Hello Ceph community, I'm writing on behalf of a friend who is experiencing a critical cluster issue after upgrading and would appreciate any assistance. Environment: - 5 MON nodes, 2 MGR nodes, 40 OSD servers (306 OSDs total) - OS: CentOS 8.2 upgraded to 8.4 - Ceph: 15.2.17 upgraded to 17.2.7 - Upgrade method: yum update in rolling batches Timeline: The upgrade started on October 8th at 1:00 PM. We upgraded MON/MGR servers first, and then upgraded OSD nodes in batches of 5 nodes. The process appeared normal initially, but when approximately 10 OSD servers remained, OSDs began going down. MON Quorum Issue: When the OSDs began failing, the monitors failed to form a quorum. In an attempt to recover, we stopped 4 out of 5 monitors. However, the remaining monitor (mbjson20010) then failed to start due to a missing .ldb file. We eventually recovered this single monitor from OSD using the instructions at https://docs.ceph.com/en/quincy/rados/troubleshooting/troubleshooting-mon/#…, so we now have only 1 MON in the cluster instead of the original 5. However, rebuilding the MON store did not help, and restarting the OSD servers also failed to resolve the issue. The cluster status remains problematic. Current Cluster Status: - Only 1 MON daemon active (quorum: mbjson20010) - down from 5 MONs - OSDs: 91 up / 229 in (out of 306 total) - 88.872% of PGs are not active - 4.779% of PGs are unknown - 3,918 PGs down - 1,311 PGs stale+down - Only 12 PGs active+clean Critical Error: When examining OSD logs, we discovered that some OSDs are failing to start with the following error: osd.43 39677784 init missing pg_pool_t for deleted pool 9 for pg 9.3ds3; please downgrade to luminous and allow pg deletion to complete before upgrading Full error context from one of the failing OSDs: # tail /var/log/ceph/ceph-osd.43.log -7> 2025-10-12T13:40:05.987+0800 7fdd13259540 1 bluestore(/var/lib/ceph/osd/ceph-43) _upgrade_super from 4, latest 4 -6> 2025-10-12T13:40:05.987+0800 7fdd13259540 1 bluestore(/var/lib/ceph/osd/ceph-43) _upgrade_super done -5> 2025-10-12T13:40:05.987+0800 7fdd13259540 2 osd.43 0 journal looks like ssd -4> 2025-10-12T13:40:05.987+0800 7fdd13259540 2 osd.43 0 boot -3> 2025-10-12T13:40:05.987+0800 7fdceb2cc700 5 bluestore.MempoolThread(0x55c7b0c66b40) _resize_shards cache_size: 8589934592 kv_alloc: 1717986918 kv_used: 91136 kv_onode_alloc: 343597383 kv_onode_used: 23328 meta_alloc: 6871947673 meta_used: 2984 data_alloc: 0 data_used: 0 -2> 2025-10-12T13:40:05.989+0800 7fdd13259540 -1 osd.43 39677784 init missing pg_pool_t for deleted pool 9 for pg 9.3ds3; please downgrade to luminous and allow pg deletion to complete before upgrading -1> 2025-10-12T13:40:05.991+0800 7fdd13259540 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.7/rpm/el8/BUILD/ceph-17.2.7/src/osd/OSD.cc: In function 'int OSD::init()' thread 7fdd13259540 time 2025-10-12T13:40:05.990845+0800 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.7/rpm/el8/BUILD/ceph-17.2.7/src/osd/OSD.cc: 3735: ceph_abort_msg("abort() called") # tail /var/log/ceph/ceph-osd.51.log -7> 2025-10-12T13:39:36.739+0800 7f603e5f7540 1 bluestore(/var/lib/ceph/osd/ceph-51) _upgrade_super from 4, latest 4 -6> 2025-10-12T13:39:36.739+0800 7f603e5f7540 1 bluestore(/var/lib/ceph/osd/ceph-51) _upgrade_super done -5> 2025-10-12T13:39:36.739+0800 7f603e5f7540 2 osd.51 0 journal looks like ssd -4> 2025-10-12T13:39:36.739+0800 7f603e5f7540 2 osd.51 0 boot -3> 2025-10-12T13:39:36.739+0800 7f6016669700 5 bluestore.MempoolThread(0x55e839d4cb40) _resize_shards cache_size: 8589934592 kv_alloc: 1717986918 kv_used: 31232 kv_onode_alloc: 343597383 kv_onode_used: 21584 meta_alloc: 6871947673 meta_used: 1168 data_alloc: 0 data_used: 0 -2> 2025-10-12T13:39:36.741+0800 7f603e5f7540 -1 osd.51 39677784 init missing pg_pool_t for deleted pool 6 for pg 6.1f; please downgrade to luminous and allow pg deletion to complete before upgrading -1> 2025-10-12T13:39:36.742+0800 7f603e5f7540 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.7/rpm/el8/BUILD/ceph-17.2.7/src/osd/OSD.cc: In function 'int OSD::init()' thread 7f603e5f7540 time 2025-10-12T13:39:36.742527+0800 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.7/rpm/el8/BUILD/ceph-17.2.7/src/osd/OSD.cc: 3735: ceph_abort_msg("abort() called") Investigation Findings: We examined all OSD instances that failed to start. All of them exhibit the same error pattern in their logs and all contain PG references to non-existent pools. For example, running "ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-51 --op list-pgs" shows PG references to pools that no longer exist (e.g., pool 9, pool 10, pool 4, pool 6, pool 8), while the current pools are numbered 101, 140, 141, 149, 212, 213, 216, 217, 218, 219. Notably, each affected OSD contains only 2-3 PGs referencing these non-existent pools, which is significantly fewer than the hundreds of PGs a regular OSD typically contains. It appears the OSD metadata has been corrupted or overwritten with stale references to deleted pools from previous operations, preventing these OSDs from starting and causing widespread PG state abnormalities across the cluster. 2 PGs referencing non-existent pools were found in osd.51: # ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-51 --op list-pgs 1.0 6.1f # ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-51 --op list Error getting attr on : 1.0_head,#1:00000000::::head#, (61) No data available Error getting attr on : 6.1f_head,#6:f8000000::::head#, (61) No data available ["1.0",{"oid":"","key":"","snapid":-2,"hash":0,"max":0,"pool":1,"namespace":"","max":0}] ["1.0",{"oid":"main.db-journal.0000000000000000","key":"","snapid":-2,"hash":1969844440,"max":0,"pool":1,"namespace":"devicehealth","max":0}] ["1.0",{"oid":"main.db.0000000000000000","key":"","snapid":-2,"hash":1315310604,"max":0,"pool":1,"namespace":"devicehealth","max":0}] ["6.1f",{"oid":"","key":"","snapid":-2,"hash":31,"max":0,"pool":6,"namespace":"","max":0}] We also performed a comprehensive check by listing all PGs from all OSD nodes using "ceph-objectstore-tool --op list-pgs" and comparing the results with the output of "ceph pg dump". This comparison revealed that quite a few PGs are missing from the OSD listings. We suspect that some OSDs that previously held these missing PGs may now be corrupted, which would explain both the missing PGs and the widespread cluster degradation. It appears the OSD metadata has been corrupted or overwritten with stale references to deleted pools from previous operations, preventing these OSDs from starting and causing widespread PG state abnormalities across the cluster. It appears the OSD objectstore's metadata has been corrupted or overwritten with stale references to deleted pools from previous operations, preventing these OSDs from starting and causing widespread PG state abnormalities across the cluster. Questions: 1. How can we safely restore the missing PGs from the OSD without data loss? 2. Has anyone encountered similar issues when upgrading from Octopus (15.2.x) to Quincy (17.2.x)? We understand that skipping major versions may not be officially supported, but we urgently need guidance on the safest recovery path at this point. Any help would be greatly appreciated. Thank you in advance. -- Regards Kefu Chai

3 6