Sometimes PGs inconsistent (although there is no load on them)
Dear list, I'm currently maintaining several Ceph (prod) installations. One of them consists of 3 MON hosts and 6 OSD hosts hosting 40 OSDs in total. And there are 5 separate Proxmox-Hosts - they only host the VMs and use the storage provided by Ceph, but they are not part of Ceph. The worst case happened: due to an outage, all these hosts crashed hardly the same time. Last week, I began to restart (only the Ceph hosts; Proxmox servers are still down). Ceph was very unhappy with the situation as a whole - one OSD host (and its 6 OSDs) is completely gone, some hardware issues (33 OSDs left, networking, PSU, I'm working on it) and 73 out of 129 PGs inconsistent. Meanwhile, the overall status of the cluster is "HEALTHY" again. But nearly every day, one or two PGs get damaged. Never on the same OSDs. And there is no traffic on the storage as the virtualization hosts are not running. I see no further reason in the logs: everything is fine, scrub starts and leaves one or more PGs damaged. Repairing them is successful, but maybe next night, another PG is stuck. Do you have hints to investigate this any further? I would love to understand more before starting the Proxmox cluster again. Using Ceph 18.2.4 (Proxmox packages). Thanks a lot, Marianne
Hi Marianne, Am 3/11/25 um 11:24 schrieb Marianne Spiller:
But nearly every day, one or two PGs get damaged. Never on the same OSDs. And there is no traffic on the storage as the virtualization hosts are not running. I see no further reason in the logs: everything is fine, scrub starts and leaves one or more PGs damaged. Repairing them is successful, but maybe next night, another PG is stuck.
The (deep) scrubbing process will find issues that have not been detected when the hosts were just rebooted. I would recommend to trigger a deep scrub on the whole cluster with ceph osd pool deep-scrub <pool> for each pool. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
This sounds like you have widespread inconsistencies that are surfaced by scrubs, not caused by them. Frequent causes: * Using a RAID HBA with bugs (all of them, in my experience), with broken preserved cache replay, with forced writeback cache without BBU, etc. * Power was out longer than BBUs could last * Volatile cache enabled on HDDs * Client grade SSDs without PLP
On Mar 11, 2025, at 6:57 AM, Martin Konold <martin.konold@konsec.com> wrote:
Hi,
I suspect a hw issue. Please check the networks.
Regards --martin
Am 11.03.2025 11:24 schrieb Marianne Spiller <marianne@spiller.me>: Dear list,
I'm currently maintaining several Ceph (prod) installations. One of them consists of 3 MON hosts and 6 OSD hosts hosting 40 OSDs in total. And there are 5 separate Proxmox-Hosts - they only host the VMs and use the storage provided by Ceph, but they are not part of Ceph.
The worst case happened: due to an outage, all these hosts crashed hardly the same time.
Last week, I began to restart (only the Ceph hosts; Proxmox servers are still down). Ceph was very unhappy with the situation as a whole - one OSD host (and its 6 OSDs) is completely gone, some hardware issues (33 OSDs left, networking, PSU, I'm working on it) and 73 out of 129 PGs inconsistent.
Meanwhile, the overall status of the cluster is "HEALTHY" again. But nearly every day, one or two PGs get damaged. Never on the same OSDs. And there is no traffic on the storage as the virtualization hosts are not running. I see no further reason in the logs: everything is fine, scrub starts and leaves one or more PGs damaged. Repairing them is successful, but maybe next night, another PG is stuck.
Do you have hints to investigate this any further? I would love to understand more before starting the Proxmox cluster again. Using Ceph 18.2.4 (Proxmox packages).
Thanks a lot, Marianne _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
Anthony D'Atri
-
Marianne Spiller
-
Martin Konold
-
Robert Sander