[ceph-users] no recovery running

17 Oct 2024

      HI,

This is my cluster:

  cluster:
    id:     c300532c-51fa-11ec-9a41-0050569c3b55
    health: HEALTH_WARN
            Degraded data redundancy: 2062374/1331064781 objects degraded
(0.155%), 278 pgs degraded, 40 pgs undersized
            2497 pgs not deep-scrubbed in time
            2497 pgs not scrubbed in time

  services:
    mon: 3 daemons, quorum hbgt-ceph1-mon1,hbgt-ceph1-mon2,hbgt-ceph1-mon3
(age 9d)
    mgr: hbgt-ceph1-mon3.gmfzqm(active, since 10d), standbys:
hbgt-ceph1-mon2.nteihj, hbgt-ceph1-mon1.thrnnu
    osd: 96 osds: 96 up (since 9d), 96 in (since 45h); 1588 remapped pgs
    rgw: 3 daemons active (3 hosts, 2 zones)

  data:
    pools:   16 pools, 2497 pgs
    objects: 266.22M objects, 518 TiB
    usage:   976 TiB used, 808 TiB / 1.7 PiB avail
    pgs:     2062374/1331064781 objects degraded (0.155%)
             349917519/1331064781 objects misplaced (26.289%)
             1312 active+remapped+backfill_wait
             864  active+clean
             199  active+recovery_wait+degraded+remapped
             38   active+recovery_wait+degraded
             33   active+undersized+degraded+remapped+backfill_wait
             33   active+recovery_wait+remapped
             7    active+recovery_wait
             6    active+undersized+degraded+remapped+backfilling
             2    active+recovering+remapped
             1    active+remapped+backfilling
             1    active+recovering+degraded+remapped
             1    active+recovery_wait+undersized+degraded+remapped

  io:
    client:   683 KiB/s rd, 2.2 KiB/s wr, 51 op/s rd, 2 op/s wr

No recovery is running and I don't understand why.
I have free space:

ID   CLASS  WEIGHT      REWEIGHT  SIZE     RAW USE   DATA     OMAP     META
     AVAIL    %USE   VAR   PGS  STATUS  TYPE NAME
 -1         1784.12231         -  1.7 PiB   976 TiB  895 TiB  298 GiB   4.1
TiB  808 TiB  54.72  1.00    -          root default
 -5          208.09680         -  208 TiB   142 TiB  130 TiB   51 GiB   605
GiB   66 TiB  68.14  1.25    -              host hbgt-ceph1-osd01
  1    hdd    17.34140   1.00000   17 TiB    11 TiB   11 TiB   33 KiB    49
GiB  5.9 TiB  66.16  1.21  136      up          osd.1
  3    hdd    17.34140   1.00000   17 TiB    11 TiB   10 TiB   23 GiB    49
GiB  6.3 TiB  63.80  1.17  139      up          osd.3
  5    hdd    17.34140   1.00000   17 TiB    13 TiB   12 TiB  139 MiB    53
GiB  4.8 TiB  72.31  1.32  142      up          osd.5
  7    hdd    17.34140   1.00000   17 TiB    12 TiB   11 TiB   11 GiB    51
GiB  5.6 TiB  67.97  1.24  145      up          osd.7
  9    hdd    17.34140   1.00000   17 TiB    11 TiB   10 TiB  2.2 GiB    49
GiB  6.0 TiB  65.67  1.20  140      up          osd.9
 11    hdd    17.34140   1.00000   17 TiB    12 TiB   11 TiB  329 MiB    50
GiB  5.5 TiB  68.42  1.25  145      up          osd.11
 13    hdd    17.34140   1.00000   17 TiB    12 TiB   11 TiB  1.5 GiB    52
GiB  5.1 TiB  70.45  1.29  153      up          osd.13
 15    hdd    17.34140   1.00000   17 TiB    12 TiB   11 TiB   61 KiB    48
GiB  5.7 TiB  66.85  1.22  144      up          osd.15
 17    hdd    17.34140   1.00000   17 TiB    11 TiB  9.5 TiB  272 MiB    45
GiB  6.8 TiB  60.63  1.11  120      up          osd.17
 19    hdd    17.34140   1.00000   17 TiB    11 TiB   10 TiB   12 GiB    50
GiB  5.9 TiB  65.90  1.20  134      up          osd.19
 21    hdd    17.34140   1.00000   17 TiB    13 TiB   12 TiB  1.6 GiB    57
GiB  4.1 TiB  76.49  1.40  152      up          osd.21
 23    hdd    17.34140   1.00000   17 TiB    13 TiB   12 TiB   31 KiB    54
GiB  4.7 TiB  73.10  1.34  124      up          osd.23
 -3          208.09680         -  208 TiB   146 TiB  134 TiB   64 GiB   629
GiB   62 TiB  70.05  1.28    -              host hbgt-ceph1-osd02
  0    hdd    17.34140   1.00000   17 TiB    11 TiB  9.8 TiB   22 GiB    49
GiB  6.6 TiB  62.07  1.13  124      up          osd.0
  2    hdd    17.34140   1.00000   17 TiB    12 TiB   11 TiB  1.7 GiB    52
GiB  5.2 TiB  70.14  1.28  150      up          osd.2
  4    hdd    17.34140   1.00000   17 TiB    12 TiB   11 TiB  1.8 GiB    48
GiB  5.8 TiB  66.83  1.22  152      up          osd.4
  6    hdd    17.34140   0.85004   17 TiB    13 TiB   12 TiB   11 GiB    58
GiB  4.0 TiB  76.85  1.40  153      up          osd.6
  8    hdd    17.34140   1.00000   17 TiB    12 TiB   11 TiB   11 GiB    54
GiB  4.9 TiB  71.58  1.31  152      up          osd.8
 10    hdd    17.34140   1.00000   17 TiB    11 TiB   10 TiB  6.3 MiB    47
GiB  6.1 TiB  64.91  1.19  133      up          osd.10
 12    hdd    17.34140   1.00000   17 TiB    12 TiB   11 TiB  109 MiB    51
GiB  5.6 TiB  67.72  1.24  137      up          osd.12
 14    hdd    17.34140   1.00000   17 TiB    12 TiB   11 TiB   12 GiB    53
GiB  5.1 TiB  70.37  1.29  148      up          osd.14
 16    hdd    17.34140   1.00000   17 TiB    12 TiB   11 TiB  2.9 GiB    54
GiB  4.9 TiB  71.65  1.31  145      up          osd.16
 18    hdd    17.34140   1.00000   17 TiB    13 TiB   12 TiB   88 MiB    55
GiB  4.8 TiB  72.45  1.32  154      up          osd.18
 20    hdd    17.34140   1.00000   17 TiB    13 TiB   12 TiB  108 MiB    55
GiB  4.6 TiB  73.39  1.34  166      up          osd.20
 22    hdd    17.34140   1.00000   17 TiB    13 TiB   12 TiB  209 MiB    54
GiB  4.7 TiB  72.68  1.33  138      up          osd.22
 -7          208.09680         -  208 TiB   148 TiB  136 TiB   29 GiB   635
GiB   60 TiB  71.03  1.30    -              host hbgt-ceph1-osd03
 24    hdd    17.34140   1.00000   17 TiB    13 TiB   12 TiB  1.4 GiB    56
GiB  4.5 TiB  74.27  1.36  150      up          osd.24
 25    hdd    17.34140   0.95001   17 TiB    13 TiB   12 TiB   32 KiB    57
GiB  4.1 TiB  76.27  1.39  162      up          osd.25
 26    hdd    17.34140   1.00000   17 TiB    11 TiB   10 TiB  1.4 GiB    49
GiB  6.0 TiB  65.30  1.19  134      up          osd.26
 27    hdd    17.34140   1.00000   17 TiB    13 TiB   12 TiB  1.7 GiB    53
GiB  4.7 TiB  73.16  1.34  152      up          osd.27
 28    hdd    17.34140   1.00000   17 TiB    12 TiB   12 TiB  2.3 MiB    52
GiB  4.9 TiB  71.97  1.32  158      up          osd.28
 29    hdd    17.34140   1.00000   17 TiB    12 TiB   11 TiB   10 GiB    52
GiB  5.3 TiB  69.72  1.27  141      up          osd.29
 30    hdd    17.34140   1.00000   17 TiB    13 TiB   12 TiB  266 MiB    54
GiB  4.6 TiB  73.38  1.34  142      up          osd.30
 31    hdd    17.34140   1.00000   17 TiB    12 TiB   11 TiB   93 MiB    50
GiB  5.7 TiB  67.36  1.23  145      up          osd.31
 32    hdd    17.34140   1.00000   17 TiB    11 TiB   10 TiB   13 GiB    50
GiB  6.1 TiB  64.77  1.18  131      up          osd.32
 33    hdd    17.34140   1.00000   17 TiB    12 TiB   11 TiB   12 MiB    53
GiB  5.2 TiB  69.77  1.28  132      up          osd.33
 34    hdd    17.34140   1.00000   17 TiB    12 TiB   11 TiB  242 MiB    50
GiB  5.5 TiB  68.17  1.25  139      up          osd.34
 35    hdd    17.34140   1.00000   17 TiB    14 TiB   13 TiB  135 MiB    58
GiB  3.8 TiB  78.26  1.43  157      up          osd.35
 -9          208.09680         -  208 TiB   146 TiB  134 TiB   38 GiB   627
GiB   62 TiB  70.19  1.28    -              host hbgt-ceph1-osd04
 36    hdd    17.34140   1.00000   17 TiB    11 TiB   10 TiB  210 MiB    47
GiB  6.4 TiB  63.16  1.15  139      up          osd.36
 37    hdd    17.34140   1.00000   17 TiB    12 TiB   11 TiB  222 MiB    53
GiB  5.0 TiB  71.34  1.30  147      up          osd.37
 38    hdd    17.34140   0.95001   17 TiB    13 TiB   12 TiB   13 MiB    55
GiB  4.5 TiB  74.33  1.36  155      up          osd.38
 39    hdd    17.34140   1.00000   17 TiB    12 TiB   11 TiB  135 MiB    52
GiB  5.2 TiB  70.22  1.28  140      up          osd.39
 40    hdd    17.34140   1.00000   17 TiB    12 TiB   11 TiB   25 MiB    53
GiB  4.9 TiB  71.46  1.31  150      up          osd.40
 41    hdd    17.34140   1.00000   17 TiB    12 TiB   11 TiB  1.8 GiB    54
GiB  5.0 TiB  71.36  1.30  150      up          osd.41
 42    hdd    17.34140   1.00000   17 TiB    12 TiB   11 TiB  823 MiB    50
GiB  5.8 TiB  66.49  1.22  156      up          osd.42
 43    hdd    17.34140   1.00000   17 TiB    12 TiB   11 TiB  1.5 GiB    52
GiB  5.2 TiB  69.97  1.28  160      up          osd.43
 44    hdd    17.34140   1.00000   17 TiB    12 TiB   11 TiB   11 GiB    52
GiB  5.3 TiB  69.35  1.27  133      up          osd.44
 45    hdd    17.34140   1.00000   17 TiB    13 TiB   12 TiB   12 GiB    55
GiB  4.8 TiB  72.12  1.32  148      up          osd.45
 46    hdd    17.34140   1.00000   17 TiB    12 TiB   11 TiB   11 GiB    52
GiB  5.1 TiB  70.74  1.29  148      up          osd.46
 47    hdd    17.34140   1.00000   17 TiB    12 TiB   11 TiB  188 MiB    54
GiB  4.9 TiB  71.74  1.31  151      up          osd.47
-11          208.09680         -  208 TiB   147 TiB  135 TiB   27 GiB   627
GiB   62 TiB  70.45  1.29    -              host hbgt-ceph1-osd05
 48    hdd    17.34140   1.00000   17 TiB    12 TiB   11 TiB  150 MiB    49
GiB  5.7 TiB  67.11  1.23  152      up          osd.48
 49    hdd    17.34140   1.00000   17 TiB    12 TiB   11 TiB  150 MiB    52
GiB  5.1 TiB  70.87  1.30  155      up          osd.49
 50    hdd    17.34140   0.90002   17 TiB    12 TiB   11 TiB   11 GiB    54
GiB  5.2 TiB  70.01  1.28  147      up          osd.50
 51    hdd    17.34140   1.00000   17 TiB    13 TiB   12 TiB   84 MiB    54
GiB  4.6 TiB  73.24  1.34  147      up          osd.51
 52    hdd    17.34140   1.00000   17 TiB    13 TiB   12 TiB  337 MiB    53
GiB  4.8 TiB  72.26  1.32  146      up          osd.52
 53    hdd    17.34140   1.00000   17 TiB    12 TiB   11 TiB  1.6 GiB    54
GiB  5.1 TiB  70.56  1.29  132      up          osd.53
 54    hdd    17.34140   1.00000   17 TiB    13 TiB   12 TiB  1.3 GiB    56
GiB  4.5 TiB  73.77  1.35  138      up          osd.54
 55    hdd    17.34140   1.00000   17 TiB    13 TiB   12 TiB   10 GiB    53
GiB  4.8 TiB  72.59  1.33  148      up          osd.55
 56    hdd    17.34140   1.00000   17 TiB    11 TiB  9.6 TiB  1.8 GiB    46
GiB  6.7 TiB  61.16  1.12  127      up          osd.56
 57    hdd    17.34140   1.00000   17 TiB    12 TiB   11 TiB  3.1 MiB    50
GiB  5.6 TiB  67.96  1.24  148      up          osd.57
 58    hdd    17.34140   1.00000   17 TiB    12 TiB   11 TiB  687 KiB    52
GiB  5.0 TiB  70.93  1.30  152      up          osd.58
 59    hdd    17.34140   1.00000   17 TiB    13 TiB   12 TiB   22 KiB    55
GiB  4.4 TiB  74.88  1.37  147      up          osd.59
-13          245.93866         -  246 TiB   112 TiB  106 TiB   28 GiB   498
GiB  134 TiB  45.56  0.83    -              host hbgt-ceph1-osd06
 60    hdd    20.49489   1.00000   20 TiB    11 TiB   11 TiB   10 GiB    51
GiB  9.3 TiB  54.43  0.99  134      up          osd.60
 61    hdd    20.49489   1.00000   20 TiB   9.6 TiB  9.2 TiB  216 MiB    43
GiB   11 TiB  47.06  0.86  120      up          osd.61
 62    hdd    20.49489   1.00000   20 TiB   9.1 TiB  8.6 TiB  330 MiB    41
GiB   11 TiB  44.53  0.81  116      up          osd.62
 63    hdd    20.49489   1.00000   20 TiB   8.9 TiB  8.4 TiB  354 MiB    39
GiB   12 TiB  43.29  0.79  113      up          osd.63
 64    hdd    20.49489   1.00000   20 TiB   8.1 TiB  7.7 TiB  148 MiB    36
GiB   12 TiB  39.72  0.73  117      up          osd.64
 65    hdd    20.49489   1.00000   20 TiB    12 TiB   12 TiB   14 KiB    53
GiB  8.4 TiB  58.90  1.08  159      up          osd.65
 66    hdd    20.49489   1.00000   20 TiB    11 TiB   10 TiB   18 MiB    49
GiB  9.8 TiB  52.33  0.96  129      up          osd.66
 67    hdd    20.49489   1.00000   20 TiB   498 GiB  1.5 GiB    1 KiB   745
MiB   20 TiB   2.37  0.04    6      up          osd.67
 68    hdd    20.49489   1.00000   20 TiB    10 TiB  9.8 TiB   13 GiB    47
GiB   10 TiB  50.12  0.92  138      up          osd.68
 69    hdd    20.49489   1.00000   20 TiB    11 TiB   10 TiB  2.0 GiB    48
GiB  9.7 TiB  52.77  0.96  135      up          osd.69
 70    hdd    20.49489   1.00000   20 TiB    11 TiB   11 TiB  1.6 GiB    50
GiB  9.2 TiB  55.24  1.01  150      up          osd.70
 71    hdd    20.49489   1.00000   20 TiB   9.4 TiB  8.9 TiB   86 MiB    42
GiB   11 TiB  45.96  0.84  130      up          osd.71
-15          245.93866         -  246 TiB   114 TiB  108 TiB   62 GiB   505
GiB  132 TiB  46.18  0.84    -              host hbgt-ceph1-osd07
 72    hdd    20.49489   1.00000   20 TiB    11 TiB   10 TiB   12 GiB    49
GiB  9.6 TiB  53.39  0.98  146      up          osd.72
 73    hdd    20.49489   1.00000   20 TiB   8.5 TiB  8.0 TiB  194 MiB    38
GiB   12 TiB  41.31  0.75  112      up          osd.73
 74    hdd    20.49489   1.00000   20 TiB   9.5 TiB  9.0 TiB   10 GiB    42
GiB   11 TiB  46.12  0.84  124      up          osd.74
 75    hdd    20.49489   1.00000   20 TiB   9.1 TiB  8.6 TiB   12 GiB    41
GiB   11 TiB  44.16  0.81  125      up          osd.75
 76    hdd    20.49489   1.00000   20 TiB   9.4 TiB  8.9 TiB  114 MiB    41
GiB   11 TiB  45.91  0.84  111      up          osd.76
 77    hdd    20.49489   1.00000   20 TiB   9.6 TiB  9.2 TiB  1.5 GiB    42
GiB   11 TiB  47.03  0.86  127      up          osd.77
 78    hdd    20.49489   1.00000   20 TiB   7.7 TiB  7.3 TiB   12 GiB    36
GiB   13 TiB  37.77  0.69  105      up          osd.78
 79    hdd    20.49489   1.00000   20 TiB   8.2 TiB  7.7 TiB    1 KiB    36
GiB   12 TiB  39.92  0.73   91      up          osd.79
 80    hdd    20.49489   1.00000   20 TiB   9.5 TiB  9.0 TiB  371 MiB    41
GiB   11 TiB  46.13  0.84  132      up          osd.80
 81    hdd    20.49489   1.00000   20 TiB   9.8 TiB  9.3 TiB  353 MiB    45
GiB   11 TiB  47.95  0.88  121      up          osd.81
 82    hdd    20.49489   1.00000   20 TiB    11 TiB   11 TiB   12 GiB    51
GiB  9.2 TiB  55.23  1.01  142      up          osd.82
 83    hdd    20.49489   1.00000   20 TiB    10 TiB  9.6 TiB  1.9 GiB    44
GiB   10 TiB  49.28  0.90  125      up          osd.83
-17          251.76105         -  252 TiB    23 TiB   11 TiB   43 MiB    64
GiB  229 TiB   8.99  0.16    -              host hbgt-ceph1-osd08
 84    hdd    20.98009   1.00000   21 TiB   1.0 TiB   65 GiB    1 KiB   1.4
GiB   20 TiB   4.93  0.09    5      up          osd.84
 85    hdd    20.98009   1.00000   21 TiB   995 GiB  1.5 GiB    1 KiB   813
MiB   20 TiB   4.63  0.08    7      up          osd.85
 86    hdd    20.98009   1.00000   21 TiB   995 GiB  1.5 GiB    1 KiB   1.0
GiB   20 TiB   4.63  0.08    2      up          osd.86
 87    hdd    20.98009   1.00000   21 TiB  1008 GiB   14 GiB    1 KiB   1.0
GiB   20 TiB   4.69  0.09    7      up          osd.87
 88    hdd    20.98009   1.00000   21 TiB   995 GiB  1.5 GiB    1 KiB   1.6
GiB   20 TiB   4.63  0.08    6      up          osd.88
 89    hdd    20.98009   1.00000   21 TiB   4.9 TiB  4.0 TiB   17 MiB    20
GiB   16 TiB  23.56  0.43   51      up          osd.89
 90    hdd    20.98009   1.00000   21 TiB   995 GiB  1.5 GiB    1 KiB   1.5
GiB   20 TiB   4.63  0.08    5      up          osd.90
 91    hdd    20.98009   1.00000   21 TiB   3.3 TiB  2.3 TiB    1 KiB    11
GiB   18 TiB  15.70  0.29   40      up          osd.91
 92    hdd    20.98009   1.00000   21 TiB   5.1 TiB  4.1 TiB   26 MiB    20
GiB   16 TiB  24.34  0.44   60      up          osd.92
 93    hdd    20.98009   1.00000   21 TiB   995 GiB  1.5 GiB    1 KiB   1.3
GiB   20 TiB   4.63  0.08    9      up          osd.93
 94    hdd    20.98009   1.00000   21 TiB   1.4 TiB  489 GiB    1 KiB   3.4
GiB   20 TiB   6.90  0.13   14      up          osd.94
 95    hdd    20.98009   1.00000   21 TiB   995 GiB  1.5 GiB    1 KiB  1016
MiB   20 TiB   4.63  0.08    4      up          osd.95
                           TOTAL  1.7 PiB   976 TiB  895 TiB  298 GiB   4.1
TiB  808 TiB  54.72

I tried:
- update backfill and recovery settings
- change osd mclock profile
- change pg/pgp_num

The cluster does not recovery. Clients are working well.

What can I do ?

Thanks for your help

[ceph-users] no recovery running

Joffrey