Take hints from this: "544 pgs not deep-scrubbed in time". Your OSDs are
unable to scrub their data in time, likely because they cannot cope with
the client + scrubbing I/O. I.e. there's too much data on too few and too
slow spindles.
You can play with osd_deep_scrub_interval and increase the scrub interval
from the default 604800 seconds (1 week) to 1209600 (2 weeks) or more. It
may be also a good idea to manually force scrubbing of some PGs to spread
scrubbing time more evenly over the selected period.
But in general this is not a balanced setup and little can be done to
alleviate the lack of spindle performance.
/Z
On Wed, 8 Nov 2023 at 17:22, <prabhav(a)cdac.in> wrote:
Hi Eugen
Please find the details below
root@meghdootctr1:/var/log/ceph# ceph -s
cluster:
id: c59da971-57d1-43bd-b2b7-865d392412a5
health: HEALTH_WARN
nodeep-scrub flag(s) set
544 pgs not deep-scrubbed in time
services:
mon: 3 daemons, quorum meghdootctr1,meghdootctr2,meghdootctr3 (age 5d)
mgr: meghdootctr1(active, since 5d), standbys: meghdootctr2, meghdootctr3
mds: 3 up:standby
osd: 36 osds: 36 up (since 34h), 36 in (since 34h)
flags nodeep-scrub
data:
pools: 2 pools, 544 pgs
objects: 10.14M objects, 39 TiB
usage: 116 TiB used, 63 TiB / 179 TiB avail
pgs: 544 active+clean
io:
client: 24 MiB/s rd, 16 MiB/s wr, 2.02k op/s rd, 907 op/s wr
Ceph Versions:
root@meghdootctr1:/var/log/ceph# ceph --version
ceph version 14.2.16 (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus
(stable)
Ceph df -h
https://pastebin.com/1ffucyJg
Ceph OSD performance dump
https://pastebin.com/1R6YQksE
Ceph tell osd.XX bench (Out of 36 osds only 8 OSDs give High IOPS value
of 250 +. Out of that 4 OSDs are from HP 3PAR and 4 OSDS from DELL EMC. We
are using only 4 OSDs from HP3 par and it is working fine without any
latency and iops issues from the beginning but the remaining 32 OSDs are
from DELL EMC in which 4 OSDs are much better than the remaining 28 OSDs)
https://pastebin.com/CixaQmBi
Please help me to identify if the issue is with the DELL EMC Storage, Ceph
configuration parameter tuning or the Overload in the cloud setup
On November 1, 2023 at 9:48 PM Eugen Block <eblock(a)nde.ag> wrote:
Hi,
for starters please add more cluster details like 'ceph status', 'ceph
versions', 'ceph osd df tree'. Increasing the to 10G was the right
thing to do, you don't get far with 1G with real cluster load. How are
the OSDs configured (HDD only, SSD only or HDD with rocksdb on SSD)?
How is the disk utilization?
Regards,
Eugen
Zitat von prabhav(a)cdac.in:
> In a production setup of 36 OSDs( SAS disks) totalling 180 TB
> allocated to a single Ceph Cluster with 3 monitors and 3 managers.
> There were 830 volumes and VMs created in Openstack with Ceph as a
> backend. On Sep 21, users reported slowness in accessing the VMs.
> Analysing the logs lead us to problem with SAS , Network congestion
> and Ceph configuration( as all default values were used). We updated
> the Network from 1Gbps to 10Gbps for public and cluster networking.
> There was no change.
> The ceph benchmark performance showed that 28 OSDs out of 36 OSDs
> reported very low IOPS of 30 to 50 while the remaining showed 300+
> IOPS.
> We gradually started reducing the load on the ceph cluster and now
> the volumes count is 650. Now the slow operations has gradually
> reduced but I am aware that this is not the solution.
> Ceph configuration is updated with increasing the
> osd_journal_size to 10 GB,
> osd_max_backfills = 1
> osd_recovery_max_active = 1
> osd_recovery_op_priority = 1
> bluestore_cache_trim_max_skip_pinned=10000
>
> After one month, now we faced another issue with Mgr daemon stopped
> in all 3 quorums and 16 OSDs went down. From the
> ceph-mon,ceph-mgr.log could not get the reason. Please guide me as
> its a production setup
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io