Remove Error - "Possible data damage: 2 pgs recovery_unfound"
Good afternoon! I have a small Ceph-cluster running with Proxmox, and after an update on one of the nodes and a reboot. So far so good. But after a couple of hours, I saw this: root@pve2:~# ceph health detail HEALTH_ERR 16/1101836 objects unfound (0.001%); Possible data damage: 2 pgs recovery_unfound; Degraded data redundancy: 48/3305508 objects degraded (0.001%), 2 pgs degraded, 2 pgs undersized OBJECT_UNFOUND 16/1101836 objects unfound (0.001%) pg 1.37 has 6 unfound objects pg 1.48 has 10 unfound objects PG_DAMAGED Possible data damage: 2 pgs recovery_unfound pg 1.37 is active+recovery_unfound+undersized+degraded+remapped, acting [11,17], 6 unfound pg 1.48 is active+recovery_unfound+undersized+degraded+remapped, acting [5,11], 10 unfound PG_DEGRADED Degraded data redundancy: 48/3305508 objects degraded (0.001%), 2 pgs degraded, 2 pgs undersized pg 1.37 is stuck undersized for 446774.454853, current state active+recovery_unfound+undersized+degraded+remapped, last acting [11,17] pg 1.48 is stuck undersized for 446774.459466, current state active+recovery_unfound+undersized+degraded+remapped, last acting [5,11] root@pve2:~# ceph -s cluster: id: 76e70c34-bce9-4f86-b049-0054f21c3494 health: HEALTH_ERR 16/1101836 objects unfound (0.001%) Possible data damage: 2 pgs recovery_unfound Degraded data redundancy: 48/3305508 objects degraded (0.001%), 2 pgs degraded, 2 pgs undersized services: mon: 3 daemons, quorum pve3,pve1,pve2 (age 2w) mgr: pve3(active, since 2w), standbys: pve1, pve2 mds: cephfs:1 {0=pve1=up:active} 2 up:standby osd: 25 osds: 25 up (since 5d), 25 in (since 8d); 2 remapped pgs data: pools: 4 pools, 672 pgs objects: 1.10M objects, 2.9 TiB usage: 8.6 TiB used, 12 TiB / 21 TiB avail pgs: 48/3305508 objects degraded (0.001%) 16/1101836 objects unfound (0.001%) 669 active+clean 2 active+recovery_unfound+undersized+degraded+remapped 1 active+clean+scrubbing+deep io: client: 680 B/s rd, 2.6 MiB/s wr, 0 op/s rd, 151 op/s wr I am not really concerned over lost data, since I am 99% sure it belonged to a faulty prometheus server anyway. The question is, how can I remove the warnings without affecting the other objects? Thankful for any pointers! -- Jonathan Sélea Website: https://jonathanselea.se PGP Key: 0x8B35B3C894B964DD Fingerprint: 4AF2 10DE 996B 673C 0FD8 AFA0 8B35 B3C8 94B9 64DD
Hi everyone, I just wanted to ask again about your oppinion about this "problem" that I have. Thankful for any answer! On 2020-08-19 13:39, Jonathan Sélea wrote:
Good afternoon!
I have a small Ceph-cluster running with Proxmox, and after an update on one of the nodes and a reboot. So far so good. But after a couple of hours, I saw this:
root@pve2:~# ceph health detail HEALTH_ERR 16/1101836 objects unfound (0.001%); Possible data damage: 2 pgs recovery_unfound; Degraded data redundancy: 48/3305508 objects degraded (0.001%), 2 pgs degraded, 2 pgs undersized OBJECT_UNFOUND 16/1101836 objects unfound (0.001%) pg 1.37 has 6 unfound objects pg 1.48 has 10 unfound objects PG_DAMAGED Possible data damage: 2 pgs recovery_unfound pg 1.37 is active+recovery_unfound+undersized+degraded+remapped, acting [11,17], 6 unfound pg 1.48 is active+recovery_unfound+undersized+degraded+remapped, acting [5,11], 10 unfound PG_DEGRADED Degraded data redundancy: 48/3305508 objects degraded (0.001%), 2 pgs degraded, 2 pgs undersized pg 1.37 is stuck undersized for 446774.454853, current state active+recovery_unfound+undersized+degraded+remapped, last acting [11,17] pg 1.48 is stuck undersized for 446774.459466, current state active+recovery_unfound+undersized+degraded+remapped, last acting [5,11]
root@pve2:~# ceph -s cluster: id: 76e70c34-bce9-4f86-b049-0054f21c3494 health: HEALTH_ERR 16/1101836 objects unfound (0.001%) Possible data damage: 2 pgs recovery_unfound Degraded data redundancy: 48/3305508 objects degraded (0.001%), 2 pgs degraded, 2 pgs undersized
services: mon: 3 daemons, quorum pve3,pve1,pve2 (age 2w) mgr: pve3(active, since 2w), standbys: pve1, pve2 mds: cephfs:1 {0=pve1=up:active} 2 up:standby osd: 25 osds: 25 up (since 5d), 25 in (since 8d); 2 remapped pgs
data: pools: 4 pools, 672 pgs objects: 1.10M objects, 2.9 TiB usage: 8.6 TiB used, 12 TiB / 21 TiB avail pgs: 48/3305508 objects degraded (0.001%) 16/1101836 objects unfound (0.001%) 669 active+clean 2 active+recovery_unfound+undersized+degraded+remapped 1 active+clean+scrubbing+deep
io: client: 680 B/s rd, 2.6 MiB/s wr, 0 op/s rd, 151 op/s wr
I am not really concerned over lost data, since I am 99% sure it belonged to a faulty prometheus server anyway. The question is, how can I remove the warnings without affecting the other objects?
Thankful for any pointers!
Hi, you could try following the troubleshooting pg section https://docs.ceph.com/docs/mimic/rados/troubleshooting/troubleshooting-pg/#u... I had some unfound objects a while ago and managed to restore them to a previous version. If you just want to "get rid" of the error and _really really_ don't care about data loss you can just tell ceph to forget about the lost objects within the PG(s): "# ceph pg $PG mark_unfound_lost delete" (I'd be careful though and exhaust every other possibility before issuing that command). Best regards Philipp On 8/21/20 12:03 PM, Jonathan Sélea wrote:
Hi everyone, I just wanted to ask again about your oppinion about this "problem" that I have. Thankful for any answer!
On 2020-08-19 13:39, Jonathan Sélea wrote:
Good afternoon!
I have a small Ceph-cluster running with Proxmox, and after an update on one of the nodes and a reboot. So far so good. But after a couple of hours, I saw this:
root@pve2:~# ceph health detail HEALTH_ERR 16/1101836 objects unfound (0.001%); Possible data damage: 2 pgs recovery_unfound; Degraded data redundancy: 48/3305508 objects degraded (0.001%), 2 pgs degraded, 2 pgs undersized OBJECT_UNFOUND 16/1101836 objects unfound (0.001%) pg 1.37 has 6 unfound objects pg 1.48 has 10 unfound objects PG_DAMAGED Possible data damage: 2 pgs recovery_unfound pg 1.37 is active+recovery_unfound+undersized+degraded+remapped, acting [11,17], 6 unfound pg 1.48 is active+recovery_unfound+undersized+degraded+remapped, acting [5,11], 10 unfound PG_DEGRADED Degraded data redundancy: 48/3305508 objects degraded (0.001%), 2 pgs degraded, 2 pgs undersized pg 1.37 is stuck undersized for 446774.454853, current state active+recovery_unfound+undersized+degraded+remapped, last acting [11,17] pg 1.48 is stuck undersized for 446774.459466, current state active+recovery_unfound+undersized+degraded+remapped, last acting [5,11]
root@pve2:~# ceph -s cluster: id: 76e70c34-bce9-4f86-b049-0054f21c3494 health: HEALTH_ERR 16/1101836 objects unfound (0.001%) Possible data damage: 2 pgs recovery_unfound Degraded data redundancy: 48/3305508 objects degraded (0.001%), 2 pgs degraded, 2 pgs undersized
services: mon: 3 daemons, quorum pve3,pve1,pve2 (age 2w) mgr: pve3(active, since 2w), standbys: pve1, pve2 mds: cephfs:1 {0=pve1=up:active} 2 up:standby osd: 25 osds: 25 up (since 5d), 25 in (since 8d); 2 remapped pgs
data: pools: 4 pools, 672 pgs objects: 1.10M objects, 2.9 TiB usage: 8.6 TiB used, 12 TiB / 21 TiB avail pgs: 48/3305508 objects degraded (0.001%) 16/1101836 objects unfound (0.001%) 669 active+clean 2 active+recovery_unfound+undersized+degraded+remapped 1 active+clean+scrubbing+deep
io: client: 680 B/s rd, 2.6 MiB/s wr, 0 op/s rd, 151 op/s wr
I am not really concerned over lost data, since I am 99% sure it belonged to a faulty prometheus server anyway. The question is, how can I remove the warnings without affecting the other objects?
Thankful for any pointers!
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (2)
-
Jonathan Sélea
-
Philipp Hocke