[ceph-users] Deep-scrub much slower than HDD speed

23 Apr 2023

      I observed that on an otherwise

`iostat` shows only 8-10 MB/s

Changing scrubbing settings

Environment:

* 6 active+clean+scrubbing+deep * Ceph version 16.2.7. * BlueStore * My cluster has many objects

`iostat -x 5` with default settings:

Device            r/s dm-0           198.60   6878.40 dm-1             0.80 dm-10           11.20 dm-11          192.60   6737.60 dm-12          245.40  10194.40 dm-13           30.80   1772.80 dm-14            3.20 dm-15            2.80 dm-16            0.80 dm-17           10.80 dm-18           10.20 dm-19           10.20 dm-2             4.80    435.20 dm-20            0.40 dm-21          191.00   6105.60 dm-3             1.00 dm-4           167.60   4973.60 dm-5            11.20 dm-6            27.20   1644.80 dm-7           217.40   8032.80 dm-8            10.80 dm-9             1.60 md127            1.60    107.20 nvme0n1         42.00    772.00 nvme1n1         37.20    248.00 sda              0.80 sdb            117.80  10195.20 sdc              4.60   1644.80 sdd            109.40   4975.20 sde            115.40   6563.20 sdf              6.40   1772.80 sdg            121.60   8033.60 sdh            122.00   6105.60 sdi            117.00   6856.80 sdj              1.60

With settings

osd_deep_scrub_stride = 4194304 osd_scrub_load_threshold = 20 osd_scrub_chunk_min = 15 osd_scrub_chunk_max = 75 osd_max_scrubs = 3

I get a slight improvement only:

Device            r/s dm-0           400.60  14686.40 dm-1           362.60  10583.20 dm-10            6.40 dm-11           76.60   1939.20 dm-12           93.00   4178.40 dm-13          376.80  12716.00 dm-14           11.80    312.80 dm-15            7.00    430.40 dm-16           11.80    229.60 dm-17            4.00 dm-18            9.00 dm-19            7.40 dm-2             0.00 dm-20           12.40   2375.20 dm-21          321.00   9468.00 dm-3            12.40    125.60 dm-4           374.00  13150.40 dm-5             8.00 dm-6           201.60   6619.20 dm-7           414.40  14476.00 dm-8             0.60 dm-9           478.00  17300.80 md127            5.00 nvme0n1         33.60    822.40 nvme1n1         62.60   3082.40 sda            236.20  10613.60 sdb             35.80   4166.40 sdc            118.20   6612.80 sdd            207.00  13152.00 sde             55.80   1939.20 sdf            221.40  12720.00 sdg            234.60  14490.40 sdh            212.20 sdi            214.00  14684.00 sdj            250.60  17296.80

`osd_max_scrubs` creates more

Overall, this looks like scrub

Looking at `strace -fyp 640105

io_submit(0x7fd5f6668000, io_submit(0x7fd5f6668000, io_submit(0x7fd5f6668000, io_submit(0x7fd5f6668000, io_submit(0x7fd5f6668000, io_submit(0x7fd5f6668000,

This looks like it's reading

Does Ceph scrubbing do a HDD

I had imagined that scrubbing Is that wrong?

Thanks!

idle cluster, scrubbing cannot fully utilise the speed of my HDDs. per disk, instead of the ~100 MB/s most HDDs can easily deliver. does not help (see below). small objects ("402.32M objects, 38 TiB" from "ceph status") due to small files (4 - 32 KiB) on CephFS. rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util 0.00   0.00   12.78    34.63   51.80   2612.80     0.00   0.00   14.82    50.44    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    3.30  91.24 3.20     0.00   0.00   11.50     4.00   52.60   2582.40     0.00   0.00   13.69    49.10    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.73   3.78 71.20     0.00   0.00    0.09     6.36  145.80    583.20     0.00   0.00    0.14     4.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.02   2.62 0.00   0.00   10.74    34.98   34.80   1684.80     0.00   0.00   11.47    48.41    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    2.47  91.40 0.00   0.00    9.43    41.54   21.20    575.20     0.00   0.00    3.94    27.13    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    2.40  87.92 0.00   0.00   11.61    57.56   78.80   4507.20     0.00   0.00   19.78    57.20    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    1.92   9.54 24.80     0.00   0.00    0.12     7.75  125.20    500.80     0.00   0.00    0.12     4.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.01   2.18 19.20     0.00   0.00    0.14     6.86  105.40    421.60     0.00   0.00    0.05     4.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.01   1.76 6.40     0.00   0.00    0.00     8.00  111.00    444.00     0.00   0.00    0.10     4.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.01   1.82 76.80     0.00   0.00    0.09     7.11  151.40    605.60     0.00   0.00    0.08     4.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.01   2.92 67.20     0.00   0.00    0.08     6.59  115.60    462.40     0.00   0.00    0.04     4.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.01   2.16 56.80     0.00   0.00    0.10     5.57  109.00    436.00     0.00   0.00    0.07     4.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.01   2.34 0.00   0.00    0.12    90.67  751.80   6292.80     0.00   0.00    0.07     8.37    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.05  16.14 2.40     0.00   0.00    0.00     6.00  265.00   2459.20     0.00   0.00    0.10     9.28    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.03   5.36 0.00   0.00    6.34    31.97   67.80   3748.00     0.00   0.00   19.56    55.28    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    2.54  42.22 8.80     0.00   0.00    0.00     8.80   91.00    364.00     0.00   0.00    0.04     4.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   1.54 0.00   0.00   10.15    29.68   49.20   2511.20     0.00   0.00   11.39    51.04    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    2.26  89.18 73.60     0.00   0.00    0.12     6.57  124.40    497.60     0.00   0.00    0.07     4.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.01   2.16 0.00   0.00   12.22    60.47   57.20   3316.80     0.00   0.00   15.93    57.99    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    1.24   6.78 0.00   0.00   12.04    36.95   64.40   3654.40     0.00   0.00   23.69    56.75    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    4.14  97.28 70.40     0.00   0.00    0.15     6.52  111.80    447.20     0.00   0.00    0.04     4.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.01   2.32 8.00     0.00   0.00   13.25     5.00   46.60   2563.20     0.00   0.00    9.08    55.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.44   3.70 0.00   0.00    0.00    67.00  142.00   1856.00     0.00   0.00    0.05    13.07    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.01   2.80 0.00   0.00    0.10    18.38 1001.00  10503.50   451.20  31.07    0.03    10.49    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.04  25.20 0.00   0.00    0.09     6.67  609.80   6723.50   369.00  37.70    0.04    11.03    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.03  15.80 3.20     0.00   0.00   11.50     4.00   33.60   2582.40    19.00  36.12    8.43    76.86    0.00      0.00     0.00   0.00    0.00     0.00    3.20    0.25    0.29   3.78 127.80  52.04   11.53    86.55   18.80    575.20     2.40  11.32    3.93    30.60    0.00      0.00     0.00   0.00    0.00     0.00    2.80    3.93    1.44  87.90 22.60  83.09   11.74   357.57   23.60   3316.80    33.60  58.74    8.53   140.54    0.00      0.00     0.00   0.00    0.00     0.00    1.80    4.56    0.26   6.74 58.40  34.80   12.19    45.48   22.00   2511.20    27.20  55.28    5.75   114.15    0.00      0.00     0.00   0.00    0.00     0.00    2.80    6.07    1.48  89.16 77.00  40.02   12.21    56.87   26.20   1684.80     8.60  24.71    7.82    64.31    0.00      0.00     0.00   0.00    0.00     0.00    1.80    6.78    1.63  91.36 24.40  79.22   14.84   277.00   39.20   4507.20    39.60  50.25   13.93   114.98    0.00      0.00     0.00   0.00    0.00     0.00    3.20    2.25    0.65   9.54 95.80  44.07   12.39    66.07   30.60   3654.40    33.80  52.48   12.47   119.42    0.00      0.00     0.00   0.00    0.00     0.00    2.60    8.77    1.91  97.24 69.00  36.13    5.47    50.05   33.00   3748.00    34.80  51.33   16.94   113.58    0.00      0.00     0.00   0.00    0.00     0.00    2.20    5.00    1.24  42.20 81.60  41.09   12.25    58.61   32.00   2612.80    19.80  38.22   10.80    81.65    0.00      0.00     0.00   0.00    0.00     0.00    2.60    8.46    1.80  91.18 8.00     0.00   0.00   13.12     5.00   31.20   2563.20    15.40  33.05    5.06    82.15    0.00      0.00     0.00   0.00    0.00     0.00    2.00    1.70    0.18   3.70 rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util 0.00   0.00   19.93    36.66   25.20   1197.60     0.00   0.00   27.44    47.52    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    8.67  91.32 0.00   0.00   18.58    29.19   30.40   1742.40     0.00   0.00   23.32    57.32    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    7.45  98.14 64.00     0.00   0.00    0.03    10.00   37.60    150.40     0.00   0.00    0.06     4.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   1.02 0.00   0.00   10.61    25.32   35.20   1885.60     0.00   0.00   22.32    53.57    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    1.60  29.06 0.00   0.00   16.04    44.93    4.40     96.00     0.00   0.00    0.68    21.82    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    1.50  19.04 0.00   0.00   15.90    33.75   27.20   1444.80     0.00   0.00   23.76    53.12    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    6.64  97.76 0.00   0.00    0.12    26.51   33.40    133.60     0.00   0.00    0.00     4.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.96 0.00   0.00    0.17    61.49   46.20    184.80     0.00   0.00    0.06     4.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.94 0.00   0.00    0.10    19.46   49.20    196.80     0.00   0.00    0.09     4.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.01   1.20 39.20     0.00   0.00    0.10     9.80   29.60    118.40     0.00   0.00    0.00     4.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.72 94.40     0.00   0.00    0.07    10.49   37.20    148.80     0.00   0.00    0.30     4.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.01   0.92 70.40     0.00   0.00    0.11     9.51   56.80    227.20     0.00   0.00    0.02     4.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   1.04 0.00     0.00   0.00    0.00     0.00   62.00    482.40     0.00   0.00    0.07     7.78    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   1.16 0.00   0.00    0.23   191.55  314.20   1524.80     0.00   0.00    0.03     4.85    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.01   7.90 0.00   0.00    7.86    29.50   31.20   1630.40     0.00   0.00   11.69    52.26    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    2.89  42.84 0.00   0.00    0.08    10.13   37.00    148.00     0.00   0.00    0.02     4.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.84 0.00   0.00   18.31    35.16   23.60   1100.80     0.00   0.00   12.42    46.64    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    7.14  96.70 84.00     0.00   0.00    0.10    10.50   38.00    152.00     0.00   0.00    0.03     4.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   1.00 0.00   0.00   11.56    32.83    3.40    108.80     0.00   0.00    2.29    32.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    2.34  58.24 0.00   0.00   21.99    34.93   10.80    235.20     0.00   0.00    5.19    21.78    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    9.17  98.00 14.40     0.00   0.00    0.00    24.00   40.80    163.20     0.00   0.00    0.04     4.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.70 0.00   0.00   23.06    36.19    4.40     92.00     0.00   0.00    5.00    20.91    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00   11.05  98.00 64.80     0.00   0.00    0.04    12.96  133.80   2014.40     0.00   0.00    0.08    15.06    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.01   2.32 0.00   0.00    0.09    24.48  252.40   3388.10   139.40  35.58    0.04    13.42    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.01   7.24 0.00   0.00    0.10    49.24  427.40   4272.90   177.20  29.31    0.03    10.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.02  13.70 126.20  34.82   15.19    44.93   11.80   1742.40    18.60  61.18   13.83   147.66    0.00      0.00     0.00   0.00    0.00     0.00    1.40   20.29    3.78  98.14 56.80  61.34   10.03   116.38    4.20     96.00     0.20   4.55    0.81    22.86    0.00      0.00     0.00   0.00    0.00     0.00    0.80    0.25    0.36  19.04 83.00  41.25    9.15    55.95    3.00    108.80     0.40  11.76    2.07    36.27    0.00      0.00     0.00   0.00    0.00     0.00    0.60    3.67    1.09  58.20 166.60  44.59   13.92    63.54   15.80   1100.80     7.80  33.05    6.90    69.67    0.00      0.00     0.00   0.00    0.00     0.00    1.80    9.67    3.01  96.66 20.80  27.15    9.06    34.75   17.40   1885.60    17.80  50.57    7.51   108.37    0.00      0.00     0.00   0.00    0.00     0.00    1.80    1.44    0.64  29.04 155.20  41.21   14.34    57.45   14.60   1444.80    12.60  46.32   10.51    98.96    0.00      0.00     0.00   0.00    0.00     0.00    1.40   11.71    3.35  97.70 179.80  43.39   15.66    61.77    9.40    235.20     1.20  11.32    5.36    25.02    0.00      0.00     0.00   0.00    0.00     0.00    2.00   11.10    3.75  97.98 9470.40   109.20  33.98    4.05    44.63   19.40   1630.40    11.80  37.82    6.41    84.04    0.00      0.00     0.00   0.00    0.00     0.00    2.00    4.10    0.99  42.78 186.60  46.58   14.46    68.62   10.00   1197.60    15.20  60.32   11.14   119.76    0.00      0.00     0.00   0.00    0.00     0.00    2.40   13.67    3.24  91.28 227.00  47.53   18.76    69.02    3.60     92.00     0.80  18.18    5.72    25.56    0.00      0.00     0.00   0.00    0.00     0.00    0.80   16.25    4.73  98.00 reads per second (~120 -> ~220), but does not proportionately increase read throughput (as is expected on a spinning disk). operations are seek-bound. -e io_submit` output, for lines that involve my HDD `/dev/dm-12`: 2, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREADV, aio_fildes=30</dev/dm-12>, aio_buf=[{iov_base=0x7fd55baa3000, iov_len=4096}], aio_offset=1654046691328}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREADV, aio_fildes=30</dev/dm-12>, aio_buf=[{iov_base=0x7fd54f378000, iov_len=24576}], aio_offset=1654046666752}]) = 2 2, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREADV, aio_fildes=30</dev/dm-12>, aio_buf=[{iov_base=0x7fd54f262000, iov_len=4096}], aio_offset=1934563819520}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREADV, aio_fildes=30</dev/dm-12>, aio_buf=[{iov_base=0x7fd54f28b000, iov_len=4096}], aio_offset=1934563823616}]) = 2 2, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREADV, aio_fildes=30</dev/dm-12>, aio_buf=[{iov_base=0x7fd54f3c7000, iov_len=28672}], aio_offset=2871307956224}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREADV, aio_fildes=30</dev/dm-12>, aio_buf=[{iov_base=0x7fd54f2d2000, iov_len=4096}], aio_offset=2871307984896}]) = 2 2, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREADV, aio_fildes=30</dev/dm-12>, aio_buf=[{iov_base=0x7fd54f34a000, iov_len=8192}], aio_offset=4056494669824}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREADV, aio_fildes=30</dev/dm-12>, aio_buf=[{iov_base=0x7fd54f35f000, iov_len=4096}], aio_offset=4056494678016}]) = 2 2, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREADV, aio_fildes=30</dev/dm-12>, aio_buf=[{iov_base=0x7fd54f3e3000, iov_len=53248}], aio_offset=1233895051264}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREADV, aio_fildes=30</dev/dm-12>, aio_buf=[{iov_base=0x7fd54f0c6000, iov_len=4096}], aio_offset=1233895104512}]) = 2 2, [{aio_data=0, aio_lio_opcode=IOCB_CMD_PREADV, aio_fildes=30</dev/dm-12>, aio_buf=[{iov_base=0x7fd54f1e2000, iov_len=4096}], aio_offset=2155792314368}, {aio_data=0, aio_lio_opcode=IOCB_CMD_PREADV, aio_fildes=30</dev/dm-12>, aio_buf=[{iov_base=0x7fd54f303000, iov_len=24576}], aio_offset=2155792289792}]) = 2 the HDD all over the place, with small reads (`aio_offset`, `iov_len`). seek for every object? might be able to read linearly through the disk (perhaps skipping larger gaps).

[ceph-users] Deep-scrub much slower than HDD speed

Niklas Hambüchen