Hi Daniel Thank your advide Vào Th 2, 7 thg 10, 2024 vào lúc 17:10 Daniel Baumann < daniel.baumann@bfh.ch> đã viết:

Hi,

On 10/7/24 09:27, Phong Tran Thanh wrote:

...

How about the disable scrub and deep-scrub

neither scrubbing nor deep-scrubbing should be disabled, it is an integral part of ensuring data consistency and data availability.

if you disable it, ceph will not know when/if data on the disks/ssds has silently been corrupted due to e.g. media faults/degradation and therefore will not be able to fix it for you. if you don't notice it in time and manually correct it, you will loose your data.

...

i want to disable it because of its effect on many I/O of my cluster.

on small clusters, this can be indeed an issue. I'd recommend to adjust the settings for how much scrubbing is done in parallel (prefered) as well as how often (last resort).

have a look at the documentation here:

https://docs.ceph.com/en/latest/rados/configuration/osd-config-ref/#scrubbin...

...

If enabled, scrubbing is not complete and takes a long time.

depending on the hardware/disk sizes, scrubbing can indeed take quite some time. but that is normal and not something to worry about.

Regards, Daniel _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

-- Trân trọng, ---------------------------------------------------------------------------- *Tran Thanh Phong* Email: tranphong079@gmail.com Skype: tranphong079

Hi Eugen Can you see and give me some advice, number of PG and PG size.. ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 2 hdd 10.98349 1.00000 11 TiB 4.5 TiB 4.2 TiB 5.0 MiB 12 GiB 6.5 TiB 40.77 1.00 182 up 17 hdd 10.98349 1.00000 11 TiB 4.8 TiB 4.6 TiB 22 MiB 14 GiB 6.1 TiB 44.16 1.08 196 up 32 hdd 10.98349 1.00000 11 TiB 4.3 TiB 4.0 TiB 37 MiB 12 GiB 6.7 TiB 38.80 0.95 173 up 47 hdd 10.98349 1.00000 11 TiB 4.2 TiB 3.9 TiB 655 KiB 11 GiB 6.8 TiB 38.16 0.93 184 up 60 hdd 10.98349 1.00000 11 TiB 4.3 TiB 4.0 TiB 19 MiB 12 GiB 6.6 TiB 39.47 0.96 176 up 74 hdd 10.98349 1.00000 11 TiB 4.2 TiB 3.9 TiB 28 MiB 12 GiB 6.8 TiB 38.10 0.93 187 up 83 hdd 10.98349 1.00000 11 TiB 4.8 TiB 4.5 TiB 1.9 MiB 14 GiB 6.2 TiB 43.47 1.06 180 up 96 hdd 10.98349 1.00000 11 TiB 4.3 TiB 4.0 TiB 38 MiB 12 GiB 6.7 TiB 38.80 0.95 181 up 110 hdd 10.98349 1.00000 11 TiB 4.5 TiB 4.2 TiB 4.3 MiB 13 GiB 6.5 TiB 40.79 1.00 174 up 123 hdd 10.98349 1.00000 11 TiB 4.2 TiB 3.9 TiB 1.9 MiB 13 GiB 6.8 TiB 38.11 0.93 173 up 136 hdd 10.98349 1.00000 11 TiB 4.3 TiB 4.0 TiB 43 MiB 12 GiB 6.6 TiB 39.46 0.96 179 up ..... PG OBJECTS DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS STATE SINCE 52.0 80121 0 0 0 323105721248 0 0 1747 3000 active+clean 94m 52.1 79751 0 0 0 321711556766 0 0 1727 3000 active+clean 21h 52.2 80243 0 0 0 323711878626 0 0 1618 3000 active+clean 30h 52.3 79892 0 0 0 322166010020 0 0 1627 3000 active+clean 9h 52.4 80267 0 0 0 323708219486 0 0 1658 3000 active+clean 5h 52.5 79996 0 0 0 322331504454 0 0 1722 3000 active+clean 18h 52.6 80190 0 0 0 323460394402 0 0 1759 3000 active+clean 15h 52.7 79998 0 0 0 322769143546 0 0 1720 3000 active+clean 26h 52.8 80292 0 0 0 323932173152 0 0 1691 3000 active+clean 21h 52.9 79808 0 0 0 321910742702 0 0 1675 3000 active+clean 7h 52.a 79751 0 0 0 321578061334 0 0 1822 3000 active+clean 26h 52.b 80287 0 0 0 323905164642 0 0 1793 3000 active+clean 6h .... Thanks Eugen Vào Th 2, 7 thg 10, 2024 vào lúc 14:45 Eugen Block <eblock@nde.ag> đã viết:

Hi,

disabling scrubbing in general is bad idea, because you won't notice any data corruption except when it might be too late. But you can fine tune scrubbing, for example increase the interval to allow fewer scrubs to finish in a longer interval. Or if the client load is mainly during business hours, adjust osd_scrub_begin_hour and osd_scrub_end_hour to your needs. And it also depends on the size of your PGs. The larger the PGs are, the longer a deep-scrub would take. So splitting PGs can have a quite positive effect in general. Inspect 'ceph osd df' output as well as 'ceph pg ls' (BYTES column), you can also share it here if you need assistance interpreting those values.

Regards, Eugen

Zitat von Phong Tran Thanh <tranphong079@gmail.com>:

...

Hi ceph users!

How about the disable scrub and deep-scrub, i want to disable it because of its effect on many I/O of my cluster. If I disable scrub how will it affect my cluster? If enabled, scrubbing is not complete and takes a long time.

Thank Skype: tranphong079 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

-- Trân trọng, ---------------------------------------------------------------------------- *Tran Thanh Phong* Email: tranphong079@gmail.com Skype: tranphong079

Hi Eugen Adding PG to the cluster only helps to reduce PG size and reduce scrub time is right? >150 PG and <200 in osd i think it's good number My cluster I/O 2-3GB/s real time in 24h IOPS 2K. If choosing to increase PG is it a good choice? Vào Th 2, 7 thg 10, 2024 vào lúc 15:49 Eugen Block <eblock@nde.ag> đã viết:

So your PGs for pool 52 have a size of around 320 GB, that is quite a lot and not surprising that deep-scrubs take a long time. At the same time, your PGs per OSD are already > 150. We had a similar situation on a customer cluster this year as well, also with 12 TB drives. We decided to increase the pg_num anyway to reduce the pg sizes. They currently have around 380 PGs per large OSD (they have lots of smaller OSDs as well) which still works fine. But they're using it as an archive, so the IO is not very high. If you would decide to split PGs, keep in mind to increase mon_max_pg_per_osd and osd_max_pg_per_osd_hard_ratio as well. I can't explicitly recommend to double your PGs per OSD as I'm not familiar with your cluster, the load etc. It's just something to think about. Doubling the PG count would reduce the PG size to around 160 GB, which is still a lot, but I probably wouldn't go further than that. The OSD utilization is only around 40%, in this case a cluster with more (smaller) OSDs would probably have made more sense.

Zitat von Phong Tran Thanh <tranphong079@gmail.com>:

...

Hi Eugen

Can you see and give me some advice, number of PG and PG size..

ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 2 hdd 10.98349 1.00000 11 TiB 4.5 TiB 4.2 TiB 5.0 MiB 12 GiB 6.5 TiB 40.77 1.00 182 up 17 hdd 10.98349 1.00000 11 TiB 4.8 TiB 4.6 TiB 22 MiB 14 GiB 6.1 TiB 44.16 1.08 196 up 32 hdd 10.98349 1.00000 11 TiB 4.3 TiB 4.0 TiB 37 MiB 12 GiB 6.7 TiB 38.80 0.95 173 up 47 hdd 10.98349 1.00000 11 TiB 4.2 TiB 3.9 TiB 655 KiB 11 GiB 6.8 TiB 38.16 0.93 184 up 60 hdd 10.98349 1.00000 11 TiB 4.3 TiB 4.0 TiB 19 MiB 12 GiB 6.6 TiB 39.47 0.96 176 up 74 hdd 10.98349 1.00000 11 TiB 4.2 TiB 3.9 TiB 28 MiB 12 GiB 6.8 TiB 38.10 0.93 187 up 83 hdd 10.98349 1.00000 11 TiB 4.8 TiB 4.5 TiB 1.9 MiB 14 GiB 6.2 TiB 43.47 1.06 180 up 96 hdd 10.98349 1.00000 11 TiB 4.3 TiB 4.0 TiB 38 MiB 12 GiB 6.7 TiB 38.80 0.95 181 up 110 hdd 10.98349 1.00000 11 TiB 4.5 TiB 4.2 TiB 4.3 MiB 13 GiB 6.5 TiB 40.79 1.00 174 up 123 hdd 10.98349 1.00000 11 TiB 4.2 TiB 3.9 TiB 1.9 MiB 13 GiB 6.8 TiB 38.11 0.93 173 up 136 hdd 10.98349 1.00000 11 TiB 4.3 TiB 4.0 TiB 43 MiB 12 GiB 6.6 TiB 39.46 0.96 179 up .....

PG OBJECTS DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS STATE SINCE 52.0 80121 0 0 0 323105248 0 0 1747 3000 active+clean 94m 52.1 79751 0 0 0 321711556766 0 0 1727 3000 active+clean 21h 52.2 80243 0 0 0 323711878626 0 0 1618 3000 active+clean 30h 52.3 79892 0 0 0 322166010020 0 0 1627 3000 active+clean 9h 52.4 80267 0 0 0 323708219486 0 0 1658 3000 active+clean 5h 52.5 79996 0 0 0 322331504454 0 0 1722 3000 active+clean 18h 52.6 80190 0 0 0 323460394402 0 0 1759 3000 active+clean 15h 52.7 79998 0 0 0 322769143546 0 0 1720 3000 active+clean 26h 52.8 80292 0 0 0 323932173152 0 0 1691 3000 active+clean 21h 52.9 79808 0 0 0 321910742702 0 0 1675 3000 active+clean 7h 52.a 79751 0 0 0 321578061334 0 0 1822 3000 active+clean 26h 52.b 80287 0 0 0 323905164642 0 0 1793 3000 active+clean 6h .... Thanks Eugen

Vào Th 2, 7 thg 10, 2024 vào lúc 14:45 Eugen Block <eblock@nde.ag> đã viết:

...

Hi,

disabling scrubbing in general is bad idea, because you won't notice any data corruption except when it might be too late. But you can fine tune scrubbing, for example increase the interval to allow fewer scrubs to finish in a longer interval. Or if the client load is mainly during business hours, adjust osd_scrub_begin_hour and osd_scrub_end_hour to your needs. And it also depends on the size of your PGs. The larger the PGs are, the longer a deep-scrub would take. So splitting PGs can have a quite positive effect in general. Inspect 'ceph osd df' output as well as 'ceph pg ls' (BYTES column), you can also share it here if you need assistance interpreting those values.

Regards, Eugen

Zitat von Phong Tran Thanh <tranphong079@gmail.com>:

...

Hi ceph users!

How about the disable scrub and deep-scrub, i want to disable it because of its effect on many I/O of my cluster. If I disable scrub how will it affect my cluster? If enabled, scrubbing is not complete and takes a long time.

Thank Skype: tranphong079 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

-- Trân trọng,

----------------------------------------------------------------------------

...

*Tran Thanh Phong*

Email: tranphong079@gmail.com Skype: tranphong079

-- Trân trọng, ---------------------------------------------------------------------------- *Tran Thanh Phong* Email: tranphong079@gmail.com Skype: tranphong079

Yes, I understand. Many thank Eugen Vào Th 2, 7 thg 10, 2024 vào lúc 16:29 Eugen Block <eblock@nde.ag> đã viết:

Well, if you have more PGs, the deep-scrubs per PG finish faster, but you have more PGs to scrub. But that way you can stretch the deep-scrub interval. As I said, I can not recommend to increase them in general. Overloading OSDs with too many PGs can have a negative effect as well. If you can, I'd probably rather add more (smaller) OSDs to spread the PGs.

Zitat von Phong Tran Thanh <tranphong079@gmail.com>:

...

Hi Eugen

Adding PG to the cluster only helps to reduce PG size and reduce scrub time is right? >150 PG and <200 in osd i think it's good number

My cluster I/O 2-3GB/s real time in 24h IOPS 2K. If choosing to increase PG is it a good choice?

Vào Th 2, 7 thg 10, 2024 vào lúc 15:49 Eugen Block <eblock@nde.ag> đã viết:

...

So your PGs for pool 52 have a size of around 320 GB, that is quite a lot and not surprising that deep-scrubs take a long time. At the same time, your PGs per OSD are already > 150. We had a similar situation on a customer cluster this year as well, also with 12 TB drives. We decided to increase the pg_num anyway to reduce the pg sizes. They currently have around 380 PGs per large OSD (they have lots of smaller OSDs as well) which still works fine. But they're using it as an archive, so the IO is not very high. If you would decide to split PGs, keep in mind to increase mon_max_pg_per_osd and osd_max_pg_per_osd_hard_ratio as well. I can't explicitly recommend to double your PGs per OSD as I'm not familiar with your cluster, the load etc. It's just something to think about. Doubling the PG count would reduce the PG size to around 160 GB, which is still a lot, but I probably wouldn't go further than that. The OSD utilization is only around 40%, in this case a cluster with more (smaller) OSDs would probably have made more sense.

Zitat von Phong Tran Thanh <tranphong079@gmail.com>:

...

Hi Eugen

Can you see and give me some advice, number of PG and PG size..

ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 2 hdd 10.98349 1.00000 11 TiB 4.5 TiB 4.2 TiB 5.0 MiB 12 GiB 6.5 TiB 40.77 1.00 182 up 17 hdd 10.98349 1.00000 11 TiB 4.8 TiB 4.6 TiB 22 MiB 14 GiB 6.1 TiB 44.16 1.08 196 up 32 hdd 10.98349 1.00000 11 TiB 4.3 TiB 4.0 TiB 37 MiB 12 GiB 6.7 TiB 38.80 0.95 173 up 47 hdd 10.98349 1.00000 11 TiB 4.2 TiB 3.9 TiB 655 KiB 11 GiB 6.8 TiB 38.16 0.93 184 up 60 hdd 10.98349 1.00000 11 TiB 4.3 TiB 4.0 TiB 19 MiB 12 GiB 6.6 TiB 39.47 0.96 176 up 74 hdd 10.98349 1.00000 11 TiB 4.2 TiB 3.9 TiB 28 MiB 12 GiB 6.8 TiB 38.10 0.93 187 up 83 hdd 10.98349 1.00000 11 TiB 4.8 TiB 4.5 TiB 1.9 MiB 14 GiB 6.2 TiB 43.47 1.06 180 up 96 hdd 10.98349 1.00000 11 TiB 4.3 TiB 4.0 TiB 38 MiB 12 GiB 6.7 TiB 38.80 0.95 181 up 110 hdd 10.98349 1.00000 11 TiB 4.5 TiB 4.2 TiB 4.3 MiB 13 GiB 6.5 TiB 40.79 1.00 174 up 123 hdd 10.98349 1.00000 11 TiB 4.2 TiB 3.9 TiB 1.9 MiB 13 GiB 6.8 TiB 38.11 0.93 173 up 136 hdd 10.98349 1.00000 11 TiB 4.3 TiB 4.0 TiB 43 MiB 12 GiB 6.6 TiB 39.46 0.96 179 up .....

PG OBJECTS DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS STATE SINCE 52.0 80121 0 0 0 323105248 0 0 1747 3000 active+clean 94m 52.1 79751 0 0 0 321711556766 0 0 1727 3000 active+clean 21h 52.2 80243 0 0 0 323711878626 0 0 1618 3000 active+clean 30h 52.3 79892 0 0 0 322166010020 0 0 1627 3000 active+clean 9h 52.4 80267 0 0 0 323708219486 0 0 1658 3000 active+clean 5h 52.5 79996 0 0 0 322331504454 0 0 1722 3000 active+clean 18h 52.6 80190 0 0 0 323460394402 0 0 1759 3000 active+clean 15h 52.7 79998 0 0 0 322769143546 0 0 1720 3000 active+clean 26h 52.8 80292 0 0 0 323932173152 0 0 1691 3000 active+clean 21h 52.9 79808 0 0 0 321910742702 0 0 1675 3000 active+clean 7h 52.a 79751 0 0 0 321578061334 0 0 1822 3000 active+clean 26h 52.b 80287 0 0 0 323905164642 0 0 1793 3000 active+clean 6h .... Thanks Eugen

Vào Th 2, 7 thg 10, 2024 vào lúc 14:45 Eugen Block <eblock@nde.ag> đã viết:

...

Hi,

disabling scrubbing in general is bad idea, because you won't notice any data corruption except when it might be too late. But you can fine tune scrubbing, for example increase the interval to allow fewer scrubs to finish in a longer interval. Or if the client load is mainly during business hours, adjust osd_scrub_begin_hour and osd_scrub_end_hour to your needs. And it also depends on the size of your PGs. The larger the PGs are, the longer a deep-scrub would take. So splitting PGs can have a quite positive effect in general. Inspect 'ceph osd df' output as well as 'ceph pg ls' (BYTES column), you can also share it here if you need assistance interpreting those values.

Regards, Eugen

Zitat von Phong Tran Thanh <tranphong079@gmail.com>:

...

Hi ceph users!

How about the disable scrub and deep-scrub, i want to disable it because of its effect on many I/O of my cluster. If I disable scrub how will it affect my cluster? If enabled, scrubbing is not complete and takes a long time.

Thank Skype: tranphong079 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

-- Trân trọng,

---

...

...

...

*Tran Thanh Phong*

Email: tranphong079@gmail.com Skype: tranphong079

-- Trân trọng,

----------------------------------------------------------------------------

...

*Tran Thanh Phong*

Email: tranphong079@gmail.com Skype: tranphong079

-- Trân trọng, ---------------------------------------------------------------------------- *Tran Thanh Phong* Email: tranphong079@gmail.com Skype: tranphong079