Hi, interesting, that’s something we can definitely try! Thanks! Christian
On 5. Sep 2023, at 16:37, Manuel Lausch <manuel.lausch@1und1.de> wrote:
Hi,
in older versions of ceph with the auto-repair feature the PG state of scrubbing PGs had always the repair state as well. With later versions (I don't know exactly at which version) ceph differentiated scrubbing and repair again in the PG state.
I think as long as there are no errors loged all should be fine. If you disable auto repair, the issue should disapear as well. In case of scrub errors you will then see appropriate states.
Regards Manuel
On Tue, 05 Sep 2023 14:14:56 +0000 Eugen Block <eblock@nde.ag> wrote:
Hi,
it sounds like you have auto-repair enabled (osd_scrub_auto_repair). I guess you could disable that to see what's going on with the PGs and their replicas. And/or you could enable debug logs. Are all daemons running the same ceph (minor) version? I remember a customer case where different ceph minor versions (but overall Octopus) caused damaged PGs, a repair fixed them everytime. After they updated all daemons to the same minor version those errors were gone.
Regards, Eugen
Zitat von Christian Theune <ct@flyingcircus.io>:
Hi,
this is a bit older cluster (Nautilus, bluestore only).
We’ve noticed that the cluster is almost continuously repairing PGs. However, they all finish successfully with “0 fixed”. We do not see the trigger why Ceph decides to repair the PGs and it’s happening for a lot of PGs, not any specific individual one.
Deep-scrubs are generally running, but currently a bit late as we had some recoveries in the last week.
Logs look regular aside from the number of repairs. Here’s the last weeks from the perspective of a single PG. There’s one repair, but the same thing seems to happen for all PGs.
2023-08-06 16:08:17.870 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-06 16:08:18.270 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-07 21:52:22.299 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-07 21:52:22.711 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-09 00:33:42.587 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-09 00:33:43.049 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-10 09:36:00.590 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 deep-scrub starts 2023-08-10 09:36:28.811 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 deep-scrub ok 2023-08-11 12:59:14.219 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-11 12:59:14.567 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-12 13:52:44.073 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-12 13:52:44.483 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-14 01:51:04.774 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 deep-scrub starts 2023-08-14 01:51:33.113 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 deep-scrub ok 2023-08-15 05:18:16.093 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-15 05:18:16.520 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-16 09:47:38.520 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-16 09:47:38.930 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-17 19:25:45.352 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-17 19:25:45.775 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-19 05:40:43.663 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-19 05:40:44.073 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-20 12:06:54.343 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-20 12:06:54.809 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-21 19:23:10.801 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 deep-scrub starts 2023-08-21 19:23:39.936 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 deep-scrub ok 2023-08-23 03:43:21.391 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-23 03:43:21.844 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-24 04:21:17.004 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 deep-scrub starts 2023-08-24 04:21:47.972 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 deep-scrub ok 2023-08-25 06:55:13.588 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-25 06:55:14.087 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-26 09:26:01.174 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-26 09:26:01.561 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-27 11:18:10.828 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-27 11:18:11.264 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-28 19:05:42.104 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-28 19:05:42.693 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-30 07:03:10.327 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-08-30 07:03:10.805 7fc49f1e6640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-08-31 14:43:23.849 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 deep-scrub starts 2023-08-31 14:43:50.723 7fc49b1de640 0 log_channel(cluster) log [DBG] : 278.2f3 deep-scrub ok 2023-09-01 20:53:42.749 7f37ca268640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-09-01 20:53:43.389 7f37c6260640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-09-02 22:57:49.542 7f37ca268640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-09-02 22:57:50.065 7f37c6260640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-09-04 03:16:14.754 7f37ca268640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub starts 2023-09-04 03:16:15.295 7f37ca268640 0 log_channel(cluster) log [DBG] : 278.2f3 scrub ok 2023-09-05 14:50:36.064 7f37ca268640 0 log_channel(cluster) log [DBG] : 278.2f3 repair starts 2023-09-05 14:51:04.407 7f37c6260640 0 log_channel(cluster) log [DBG] : 278.2f3 repair ok, 0 fixed
Googling didn’t help, unfortunately and the bug tracker doesn’t appear to have any relevant issue either.
Any ideas?
Liebe Grüße, Christian Theune
-- Christian Theune · ct@flyingcircus.io · +49 345 219401 0 Flying Circus Internet Operations GmbH · https://flyingcircus.io Leipziger Str. 70/71 · 06108 Halle (Saale) · Deutschland HR Stendal HRB 21169 · Geschäftsführer: Christian Theune, Christian Zagrodnick _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Liebe Grüße, Christian Theune -- Christian Theune · ct@flyingcircus.io · +49 345 219401 0 Flying Circus Internet Operations GmbH · https://flyingcircus.io Leipziger Str. 70/71 · 06108 Halle (Saale) · Deutschland HR Stendal HRB 21169 · Geschäftsführer: Christian Theune, Christian Zagrodnick