Re: Ceph OSD reported Slow operations
Hi Eugen Please find the details below root@meghdootctr1:/var/log/ceph# ceph -s cluster: id: c59da971-57d1-43bd-b2b7-865d392412a5 health: HEALTH_WARN nodeep-scrub flag(s) set 544 pgs not deep-scrubbed in time services: mon: 3 daemons, quorum meghdootctr1,meghdootctr2,meghdootctr3 (age 5d) mgr: meghdootctr1(active, since 5d), standbys: meghdootctr2, meghdootctr3 mds: 3 up:standby osd: 36 osds: 36 up (since 34h), 36 in (since 34h) flags nodeep-scrub data: pools: 2 pools, 544 pgs objects: 10.14M objects, 39 TiB usage: 116 TiB used, 63 TiB / 179 TiB avail pgs: 544 active+clean io: client: 24 MiB/s rd, 16 MiB/s wr, 2.02k op/s rd, 907 op/s wr Ceph Versions: root@meghdootctr1:/var/log/ceph# ceph --version ceph version 14.2.16 (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus (stable) Ceph df -h https://pastebin.com/1ffucyJg Ceph OSD performance dump https://pastebin.com/1R6YQksE Ceph tell osd.XX bench (Out of 36 osds only 8 OSDs give High IOPS value of 250 +. Out of that 4 OSDs are from HP 3PAR and 4 OSDS from DELL EMC. We are using only 4 OSDs from HP3 par and it is working fine without any latency and iops issues from the beginning but the remaining 32 OSDs are from DELL EMC in which 4 OSDs are much better than the remaining 28 OSDs) https://pastebin.com/CixaQmBi Please help me to identify if the issue is with the DELL EMC Storage, Ceph configuration parameter tuning or the Overload in the cloud setup On November 1, 2023 at 9:48 PM Eugen Block <eblock@nde.ag> wrote:
Hi,
for starters please add more cluster details like 'ceph status', 'ceph versions', 'ceph osd df tree'. Increasing the to 10G was the right thing to do, you don't get far with 1G with real cluster load. How are the OSDs configured (HDD only, SSD only or HDD with rocksdb on SSD)? How is the disk utilization?
Regards, Eugen
Zitat von prabhav@cdac.in:
In a production setup of 36 OSDs( SAS disks) totalling 180 TB allocated to a single Ceph Cluster with 3 monitors and 3 managers. There were 830 volumes and VMs created in Openstack with Ceph as a backend. On Sep 21, users reported slowness in accessing the VMs. Analysing the logs lead us to problem with SAS , Network congestion and Ceph configuration( as all default values were used). We updated the Network from 1Gbps to 10Gbps for public and cluster networking. There was no change. The ceph benchmark performance showed that 28 OSDs out of 36 OSDs reported very low IOPS of 30 to 50 while the remaining showed 300+ IOPS. We gradually started reducing the load on the ceph cluster and now the volumes count is 650. Now the slow operations has gradually reduced but I am aware that this is not the solution. Ceph configuration is updated with increasing the osd_journal_size to 10 GB, osd_max_backfills = 1 osd_recovery_max_active = 1 osd_recovery_op_priority = 1 bluestore_cache_trim_max_skip_pinned=10000
After one month, now we faced another issue with Mgr daemon stopped in all 3 quorums and 16 OSDs went down. From the ceph-mon,ceph-mgr.log could not get the reason. Please guide me as its a production setup _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io Thanks & Regards, Ms V A Prabha / श्रीमती प्रभा वी ए Joint Director / संयुक्त निदेशक Centre for Development of Advanced Computing(C-DAC) / प्रगत संगणन विकास केन्द्र(सी-डैक) Tidel Park”, 8th Floor, “D” Block, (North &South) / “टाइडल पार्क”,8वीं मंजिल, “डी” ब्लॉक, (उत्तर और दक्षिण) No.4, Rajiv Gandhi Salai / नं.4, राजीव गांधी सलाई Taramani / तारामणि Chennai / चेन्नई – 600113 Ph.No.:044-22542226/27 Fax No.: 044-22542294
[ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ] This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken. ------------------------------------------------------------------------------------------------------------
I'm afraid you're simply hitting the I/O limits of your disks. /Z On Thu, 2 Nov 2023 at 03:40, V A Prabha <prabhav@cdac.in> wrote:
Hi Eugen Please find the details below
root@meghdootctr1:/var/log/ceph# ceph -s cluster: id: c59da971-57d1-43bd-b2b7-865d392412a5 health: HEALTH_WARN nodeep-scrub flag(s) set 544 pgs not deep-scrubbed in time
services: mon: 3 daemons, quorum meghdootctr1,meghdootctr2,meghdootctr3 (age 5d) mgr: meghdootctr1(active, since 5d), standbys: meghdootctr2, meghdootctr3 mds: 3 up:standby osd: 36 osds: 36 up (since 34h), 36 in (since 34h) flags nodeep-scrub
data: pools: 2 pools, 544 pgs objects: 10.14M objects, 39 TiB usage: 116 TiB used, 63 TiB / 179 TiB avail pgs: 544 active+clean
io: client: 24 MiB/s rd, 16 MiB/s wr, 2.02k op/s rd, 907 op/s wr
Ceph Versions:
root@meghdootctr1:/var/log/ceph# ceph --version ceph version 14.2.16 (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus (stable)
Ceph df -h https://pastebin.com/1ffucyJg
Ceph OSD performance dump https://pastebin.com/1R6YQksE
Ceph tell osd.XX bench (Out of 36 osds only 8 OSDs give High IOPS value of 250 +. Out of that 4 OSDs are from HP 3PAR and 4 OSDS from DELL EMC. We are using only 4 OSDs from HP3 par and it is working fine without any latency and iops issues from the beginning but the remaining 32 OSDs are from DELL EMC in which 4 OSDs are much better than the remaining 28 OSDs)
Please help me to identify if the issue is with the DELL EMC Storage, Ceph configuration parameter tuning or the Overload in the cloud setup
On November 1, 2023 at 9:48 PM Eugen Block <eblock@nde.ag> wrote:
Hi,
for starters please add more cluster details like 'ceph status', 'ceph versions', 'ceph osd df tree'. Increasing the to 10G was the right thing to do, you don't get far with 1G with real cluster load. How are the OSDs configured (HDD only, SSD only or HDD with rocksdb on SSD)? How is the disk utilization?
Regards, Eugen
Zitat von prabhav@cdac.in:
In a production setup of 36 OSDs( SAS disks) totalling 180 TB allocated to a single Ceph Cluster with 3 monitors and 3 managers. There were 830 volumes and VMs created in Openstack with Ceph as a backend. On Sep 21, users reported slowness in accessing the VMs. Analysing the logs lead us to problem with SAS , Network congestion and Ceph configuration( as all default values were used). We updated the Network from 1Gbps to 10Gbps for public and cluster networking. There was no change. The ceph benchmark performance showed that 28 OSDs out of 36 OSDs reported very low IOPS of 30 to 50 while the remaining showed 300+ IOPS. We gradually started reducing the load on the ceph cluster and now the volumes count is 650. Now the slow operations has gradually reduced but I am aware that this is not the solution. Ceph configuration is updated with increasing the osd_journal_size to 10 GB, osd_max_backfills = 1 osd_recovery_max_active = 1 osd_recovery_op_priority = 1 bluestore_cache_trim_max_skip_pinned=10000
After one month, now we faced another issue with Mgr daemon stopped in all 3 quorums and 16 OSDs went down. From the ceph-mon,ceph-mgr.log could not get the reason. Please guide me as its a production setup _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io Thanks & Regards, Ms V A Prabha / श्रीमती प्रभा वी ए Joint Director / संयुक्त निदेशक Centre for Development of Advanced Computing(C-DAC) / प्रगत संगणन विकास केन्द्र(सी-डैक) Tidel Park”, 8th Floor, “D” Block, (North &South) / “टाइडल पार्क”,8वीं मंजिल, “डी” ब्लॉक, (उत्तर और दक्षिण) No.4, Rajiv Gandhi Salai / नं.4, राजीव गांधी सलाई Taramani / तारामणि Chennai / चेन्नई – 600113 Ph.No.:044-22542226/27 Fax No.: 044-22542294
------------------------------------------------------------------------------------------------------------ [ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ]
This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken.
------------------------------------------------------------------------------------------------------------
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Can you please elaborate your identifications and the statement . On November 2, 2023 at 9:40 AM Zakhar Kirpichenko <zakhar@gmail.com> wrote:
I'm afraid you're simply hitting the I/O limits of your disks.
/Z
On Thu, 2 Nov 2023 at 03:40, V A Prabha < prabhav@cdac.in <mailto:prabhav@cdac.in> > wrote:
Hi Eugen Please find the details below
root@meghdootctr1:/var/log/ceph# ceph -s cluster: id: c59da971-57d1-43bd-b2b7-865d392412a5 health: HEALTH_WARN nodeep-scrub flag(s) set 544 pgs not deep-scrubbed in time
services: mon: 3 daemons, quorum meghdootctr1,meghdootctr2,meghdootctr3 (age 5d) mgr: meghdootctr1(active, since 5d), standbys: meghdootctr2, meghdootctr3 mds: 3 up:standby osd: 36 osds: 36 up (since 34h), 36 in (since 34h) flags nodeep-scrub
data: pools: 2 pools, 544 pgs objects: 10.14M objects, 39 TiB usage: 116 TiB used, 63 TiB / 179 TiB avail pgs: 544 active+clean
io: client: 24 MiB/s rd, 16 MiB/s wr, 2.02k op/s rd, 907 op/s wr
Ceph Versions:
root@meghdootctr1:/var/log/ceph# ceph --version ceph version 14.2.16 (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus (stable)
Ceph df -h https://pastebin.com/1ffucyJg <https://pastebin.com/1ffucyJg>
Ceph OSD performance dump https://pastebin.com/1R6YQksE <https://pastebin.com/1R6YQksE>
Ceph tell osd.XX bench (Out of 36 osds only 8 OSDs give High IOPS value of 250 +. Out of that 4 OSDs are from HP 3PAR and 4 OSDS from DELL EMC. We are using only 4 OSDs from HP3 par and it is working fine without any latency and iops issues from the beginning but the remaining 32 OSDs are from DELL EMC in which 4 OSDs are much better than the remaining 28 OSDs)
https://pastebin.com/CixaQmBi <https://pastebin.com/CixaQmBi>
Please help me to identify if the issue is with the DELL EMC Storage, Ceph configuration parameter tuning or the Overload in the cloud setup
On November 1, 2023 at 9:48 PM Eugen Block < eblock@nde.ag <mailto:eblock@nde.ag> > wrote:
Hi,
for starters please add more cluster details like 'ceph status', 'ceph versions', 'ceph osd df tree'. Increasing the to 10G was the right thing to do, you don't get far with 1G with real cluster load. How are the OSDs configured (HDD only, SSD only or HDD with rocksdb on SSD)? How is the disk utilization?
Regards, Eugen
Zitat von prabhav@cdac.in <mailto:prabhav@cdac.in> :
In a production setup of 36 OSDs( SAS disks) totalling 180 TB allocated to a single Ceph Cluster with 3 monitors and 3 managers. There were 830 volumes and VMs created in Openstack with Ceph as a backend. On Sep 21, users reported slowness in accessing the VMs. Analysing the logs lead us to problem with SAS , Network congestion and Ceph configuration( as all default values were used). We updated the Network from 1Gbps to 10Gbps for public and cluster networking. There was no change. The ceph benchmark performance showed that 28 OSDs out of 36 OSDs reported very low IOPS of 30 to 50 while the remaining showed 300+ IOPS. We gradually started reducing the load on the ceph cluster and now the volumes count is 650. Now the slow operations has gradually reduced but I am aware that this is not the solution. Ceph configuration is updated with increasing the osd_journal_size to 10 GB, osd_max_backfills = 1 osd_recovery_max_active = 1 osd_recovery_op_priority = 1 bluestore_cache_trim_max_skip_pinned=10000
After one month, now we faced another issue with Mgr daemon stopped in all 3 quorums and 16 OSDs went down. From the ceph-mon,ceph-mgr.log could not get the reason. Please guide me as its a production setup _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io> Thanks & Regards, Ms V A Prabha / श्रीमती प्रभा वी ए Joint Director / संयुक्त निदेशक Centre for Development of Advanced Computing(C-DAC) / प्रगत संगणन विकास केन्द्र(सी-डैक) Tidel Park”, 8th Floor, “D” Block, (North &South) / “टाइडल पार्क”,8वीं मंजिल, “डी” ब्लॉक, (उत्तर और दक्षिण) No.4, Rajiv Gandhi Salai / नं.4, राजीव गांधी सलाई Taramani / तारामणि Chennai / चेन्नई – 600113 Ph.No.:044-22542226/27 Fax No.: 044-22542294
------------------------------------------------------------------------------------------------------------ [ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA <https://www.facebook.com/CDACINDIA> & Twitter: @cdacindia ]
This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken.
------------------------------------------------------------------------------------------------------------
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
Thanks & Regards, Ms V A Prabha / श्रीमती प्रभा वी ए Joint Director / संयुक्त निदेशक Centre for Development of Advanced Computing(C-DAC) / प्रगत संगणन विकास केन्द्र(सी-डैक) Tidel Park”, 8th Floor, “D” Block, (North &South) / “टाइडल पार्क”,8वीं मंजिल, “डी” ब्लॉक, (उत्तर और दक्षिण) No.4, Rajiv Gandhi Salai / नं.4, राजीव गांधी सलाई Taramani / तारामणि Chennai / चेन्नई – 600113 Ph.No.:044-22542226/27 Fax No.: 044-22542294 ------------------------------------------------------------------------------------------------------------ [ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ] This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken. ------------------------------------------------------------------------------------------------------------
Sure, it's 36 OSDs at 200 IOPS each (tops, likely lower), I assume size=3 replication so 1/3 of the total performance, and some 30%-ish OSD overhead. (36 x 200) * 1/3 * 0.7 = 1680. That's how many IOPS you can realistically expect from your cluster. You get more than that, but the cluster is very busy and OSDs aren't coping. Also your nodes are not balanced. /Z On Thu, 2 Nov 2023 at 07:33, V A Prabha <prabhav@cdac.in> wrote:
Can you please elaborate your identifications and the statement .
On November 2, 2023 at 9:40 AM Zakhar Kirpichenko <zakhar@gmail.com> wrote:
I'm afraid you're simply hitting the I/O limits of your disks.
/Z
On Thu, 2 Nov 2023 at 03:40, V A Prabha < prabhav@cdac.in> wrote:
Hi Eugen Please find the details below
root@meghdootctr1:/var/log/ceph# ceph -s cluster: id: c59da971-57d1-43bd-b2b7-865d392412a5 health: HEALTH_WARN nodeep-scrub flag(s) set 544 pgs not deep-scrubbed in time
services: mon: 3 daemons, quorum meghdootctr1,meghdootctr2,meghdootctr3 (age 5d) mgr: meghdootctr1(active, since 5d), standbys: meghdootctr2, meghdootctr3 mds: 3 up:standby osd: 36 osds: 36 up (since 34h), 36 in (since 34h) flags nodeep-scrub
data: pools: 2 pools, 544 pgs objects: 10.14M objects, 39 TiB usage: 116 TiB used, 63 TiB / 179 TiB avail pgs: 544 active+clean
io: client: 24 MiB/s rd, 16 MiB/s wr, 2.02k op/s rd, 907 op/s wr
Ceph Versions:
root@meghdootctr1:/var/log/ceph# ceph --version ceph version 14.2.16 (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus (stable)
Ceph df -h https://pastebin.com/1ffucyJg
Ceph OSD performance dump https://pastebin.com/1R6YQksE
Ceph tell osd.XX bench (Out of 36 osds only 8 OSDs give High IOPS value of 250 +. Out of that 4 OSDs are from HP 3PAR and 4 OSDS from DELL EMC. We are using only 4 OSDs from HP3 par and it is working fine without any latency and iops issues from the beginning but the remaining 32 OSDs are from DELL EMC in which 4 OSDs are much better than the remaining 28 OSDs)
Please help me to identify if the issue is with the DELL EMC Storage, Ceph configuration parameter tuning or the Overload in the cloud setup
On November 1, 2023 at 9:48 PM Eugen Block < eblock@nde.ag> wrote:
Hi,
for starters please add more cluster details like 'ceph status', 'ceph versions', 'ceph osd df tree'. Increasing the to 10G was the right thing to do, you don't get far with 1G with real cluster load. How are the OSDs configured (HDD only, SSD only or HDD with rocksdb on SSD)? How is the disk utilization?
Regards, Eugen
Zitat von prabhav@cdac.in:
In a production setup of 36 OSDs( SAS disks) totalling 180 TB allocated to a single Ceph Cluster with 3 monitors and 3 managers. There were 830 volumes and VMs created in Openstack with Ceph as a backend. On Sep 21, users reported slowness in accessing the VMs. Analysing the logs lead us to problem with SAS , Network congestion and Ceph configuration( as all default values were used). We updated the Network from 1Gbps to 10Gbps for public and cluster networking. There was no change. The ceph benchmark performance showed that 28 OSDs out of 36 OSDs reported very low IOPS of 30 to 50 while the remaining showed 300+ IOPS. We gradually started reducing the load on the ceph cluster and now the volumes count is 650. Now the slow operations has gradually reduced but I am aware that this is not the solution. Ceph configuration is updated with increasing the osd_journal_size to 10 GB, osd_max_backfills = 1 osd_recovery_max_active = 1 osd_recovery_op_priority = 1 bluestore_cache_trim_max_skip_pinned=10000
After one month, now we faced another issue with Mgr daemon stopped in all 3 quorums and 16 OSDs went down. From the ceph-mon,ceph-mgr.log could not get the reason. Please guide me as its a production setup _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io Thanks & Regards, Ms V A Prabha / श्रीमती प्रभा वी ए Joint Director / संयुक्त निदेशक Centre for Development of Advanced Computing(C-DAC) / प्रगत संगणन विकास केन्द्र(सी-डैक) Tidel Park”, 8th Floor, “D” Block, (North &South) / “टाइडल पार्क”,8वीं मंजिल, “डी” ब्लॉक, (उत्तर और दक्षिण) No.4, Rajiv Gandhi Salai / नं.4, राजीव गांधी सलाई Taramani / तारामणि Chennai / चेन्नई – 600113 Ph.No.:044-22542226/27 Fax No.: 044-22542294
[ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ]
This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken. ------------------------------------------------------------------------------------------------------------
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks & Regards, Ms V A Prabha / श्रीमती प्रभा वी ए Joint Director / संयुक्त निदेशक Centre for Development of Advanced Computing(C-DAC) / प्रगत संगणन विकास केन्द्र(सी-डैक) Tidel Park”, 8th Floor, “D” Block, (North &South) / “टाइडल पार्क”,8वीं मंजिल, “डी” ब्लॉक, (उत्तर और दक्षिण) No.4, Rajiv Gandhi Salai / नं.4, राजीव गांधी सलाई Taramani / तारामणि Chennai / चेन्नई – 600113 Ph.No.:044-22542226/27 Fax No.: 044-22542294
------------------------------------------------------------------------------------------------------------
[ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ]
This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken. ------------------------------------------------------------------------------------------------------------
Thanks for your prompt reply .. But the query is 1.The calculated IOPS is for the rw operation right ? 2. Cluster is very busy? Is there any misconfiguration or missing tuning paramater that makes the cluster busy? 3. Nodes are not balanced? you mean to say that the count of OSDs in each server differs. But we have enabled autoscale and optimal distribution so that you can see from the output of ceph osd df tree that is count of pgs(45/OSD) and use% (65 to 67%). Is that not significant? Correct me if my queries are irrelevant On November 2, 2023 at 11:36 AM Zakhar Kirpichenko <zakhar@gmail.com> wrote:
Sure, it's 36 OSDs at 200 IOPS each (tops, likely lower), I assume size=3 replication so 1/3 of the total performance, and some 30%-ish OSD overhead.
(36 x 200) * 1/3 * 0.7 = 1680. That's how many IOPS you can realistically expect from your cluster. You get more than that, but the cluster is very busy and OSDs aren't coping.
Also your nodes are not balanced.
/Z
On Thu, 2 Nov 2023 at 07:33, V A Prabha < prabhav@cdac.in <mailto:prabhav@cdac.in> > wrote:
Can you please elaborate your identifications and the statement .
On November 2, 2023 at 9:40 AM Zakhar Kirpichenko < zakhar@gmail.com <mailto:zakhar@gmail.com> > wrote:
> > > I'm afraid you're simply hitting the I/O limits of your disks.
/Z
On Thu, 2 Nov 2023 at 03:40, V A Prabha < prabhav@cdac.in <mailto:prabhav@cdac.in> > wrote: > > > > Hi Eugen
Please find the details below
root@meghdootctr1:/var/log/ceph# ceph -s cluster: id: c59da971-57d1-43bd-b2b7-865d392412a5 health: HEALTH_WARN nodeep-scrub flag(s) set 544 pgs not deep-scrubbed in time
services: mon: 3 daemons, quorum meghdootctr1,meghdootctr2,meghdootctr3 (age 5d) mgr: meghdootctr1(active, since 5d), standbys: meghdootctr2, meghdootctr3 mds: 3 up:standby osd: 36 osds: 36 up (since 34h), 36 in (since 34h) flags nodeep-scrub
data: pools: 2 pools, 544 pgs objects: 10.14M objects, 39 TiB usage: 116 TiB used, 63 TiB / 179 TiB avail pgs: 544 active+clean
io: client: 24 MiB/s rd, 16 MiB/s wr, 2.02k op/s rd, 907 op/s wr
Ceph Versions:
root@meghdootctr1:/var/log/ceph# ceph --version ceph version 14.2.16 (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus (stable)
Ceph df -h https://pastebin.com/1ffucyJg <https://pastebin.com/1ffucyJg>
Ceph OSD performance dump https://pastebin.com/1R6YQksE <https://pastebin.com/1R6YQksE>
Ceph tell osd.XX bench (Out of 36 osds only 8 OSDs give High IOPS value of 250 +. Out of that 4 OSDs are from HP 3PAR and 4 OSDS from DELL EMC. We are using only 4 OSDs from HP3 par and it is working fine without any latency and iops issues from the beginning but the remaining 32 OSDs are from DELL EMC in which 4 OSDs are much better than the remaining 28 OSDs)
https://pastebin.com/CixaQmBi <https://pastebin.com/CixaQmBi>
Please help me to identify if the issue is with the DELL EMC Storage, Ceph configuration parameter tuning or the Overload in the cloud setup
On November 1, 2023 at 9:48 PM Eugen Block < eblock@nde.ag <mailto:eblock@nde.ag> > wrote: > Hi, > > for starters please add more cluster details like 'ceph status', > 'ceph > versions', 'ceph osd df tree'. Increasing the to 10G was the > right > thing to do, you don't get far with 1G with real cluster load. > How are > the OSDs configured (HDD only, SSD only or HDD with rocksdb on > SSD)? > How is the disk utilization? > > Regards, > Eugen > > Zitat von prabhav@cdac.in <mailto:prabhav@cdac.in> : > > > In a production setup of 36 OSDs( SAS disks) totalling 180 TB > > allocated to a single Ceph Cluster with 3 monitors and 3 > > managers. > > There were 830 volumes and VMs created in Openstack with Ceph > > as a > > backend. On Sep 21, users reported slowness in accessing the > > VMs. > > Analysing the logs lead us to problem with SAS , Network > > congestion > > and Ceph configuration( as all default values were used). We > > updated > > the Network from 1Gbps to 10Gbps for public and cluster > > networking. > > There was no change. > > The ceph benchmark performance showed that 28 OSDs out of 36 > > OSDs > > reported very low IOPS of 30 to 50 while the remaining showed > > 300+ > > IOPS. > > We gradually started reducing the load on the ceph cluster and > > now > > the volumes count is 650. Now the slow operations has > > gradually > > reduced but I am aware that this is not the solution. > > Ceph configuration is updated with increasing the > > osd_journal_size to 10 GB, > > osd_max_backfills = 1 > > osd_recovery_max_active = 1 > > osd_recovery_op_priority = 1 > > bluestore_cache_trim_max_skip_pinned=10000 > > > > After one month, now we faced another issue with Mgr daemon > > stopped > > in all 3 quorums and 16 OSDs went down. From the > > ceph-mon,ceph-mgr.log could not get the reason. Please guide > > me as > > its a production setup > > _______________________________________________ > > ceph-users mailing list -- ceph-users@ceph.io > > <mailto:ceph-users@ceph.io> > > To unsubscribe send an email to ceph-users-leave@ceph.io > > <mailto:ceph-users-leave@ceph.io> > > > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > <mailto:ceph-users@ceph.io> > To unsubscribe send an email to ceph-users-leave@ceph.io > <mailto:ceph-users-leave@ceph.io> Thanks & Regards, Ms V A Prabha / श्रीमती प्रभा वी ए Joint Director / संयुक्त निदेशक Centre for Development of Advanced Computing(C-DAC) / प्रगत संगणन विकास केन्द्र(सी-डैक) Tidel Park”, 8th Floor, “D” Block, (North &South) / “टाइडल पार्क”,8वीं मंजिल, “डी” ब्लॉक, (उत्तर और दक्षिण) No.4, Rajiv Gandhi Salai / नं.4, राजीव गांधी सलाई Taramani / तारामणि Chennai / चेन्नई – 600113 Ph.No.:044-22542226/27 Fax No.: 044-22542294
------------------------------------------------------------------------------------------------------------ [ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA <https://www.facebook.com/CDACINDIA> & Twitter: @cdacindia ]
This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken.
------------------------------------------------------------------------------------------------------------
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io> > > >
Thanks & Regards, Ms V A Prabha / श्रीमती प्रभा वी ए Joint Director / संयुक्त निदेशक Centre for Development of Advanced Computing(C-DAC) / प्रगत संगणन विकास केन्द्र(सी-डैक) Tidel Park”, 8th Floor, “D” Block, (North &South) / “टाइडल पार्क”,8वीं मंजिल, “डी” ब्लॉक, (उत्तर और दक्षिण) No.4, Rajiv Gandhi Salai / नं.4, राजीव गांधी सलाई Taramani / तारामणि Chennai / चेन्नई – 600113 Ph.No.:044-22542226/27 Fax No.: 044-22542294
------------------------------------------------------------------------------------------------------------ [ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA <https://www.facebook.com/CDACINDIA> & Twitter: @cdacindia ]
This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken.
------------------------------------------------------------------------------------------------------------
Thanks & Regards, Ms V A Prabha / श्रीमती प्रभा वी ए Joint Director / संयुक्त निदेशक Centre for Development of Advanced Computing(C-DAC) / प्रगत संगणन विकास केन्द्र(सी-डैक) Tidel Park”, 8th Floor, “D” Block, (North &South) / “टाइडल पार्क”,8वीं मंजिल, “डी” ब्लॉक, (उत्तर और दक्षिण) No.4, Rajiv Gandhi Salai / नं.4, राजीव गांधी सलाई Taramani / तारामणि Chennai / चेन्नई – 600113 Ph.No.:044-22542226/27 Fax No.: 044-22542294 ------------------------------------------------------------------------------------------------------------ [ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ] This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken. ------------------------------------------------------------------------------------------------------------
1. The calculated IOPS is for the rw operation right ?
2. Cluster is very busy? Is there any misconfiguration or missing tuning
3. Nodes are not balanced? you mean to say that the count of OSDs in each server differs. But we have enabled autoscale and optimal distribution so
Total drive IOPS, read or write. Depending on the exact drive models, it may be lower or higher than 200. I took the average for a smaller sized 7.2k rpm SAS drive. Modern drives usually deliver lower read IOPS and higher write IOPS. paramater that makes the cluster busy? You have almost 3k IOPS and your OSDs report slow ops. I'd say the cluster is busy, as in loaded with I/O, perhaps more I/O than it can handle well. that you can see from the output of ceph osd df tree that is count of pgs(45/OSD) and use% (65 to 67%). Is that not significant? Yes, the OSD count differs. This means that the CPU, memory usage, network load and latency differ per node and may cause performance variations, depending on your workload. /Z On Thu, 2 Nov 2023 at 08:18, V A Prabha <prabhav@cdac.in> wrote:
Thanks for your prompt reply .. But the query is 1.The calculated IOPS is for the rw operation right ? 2. Cluster is very busy? Is there any misconfiguration or missing tuning paramater that makes the cluster busy? 3. Nodes are not balanced? you mean to say that the count of OSDs in each server differs. But we have enabled autoscale and optimal distribution so that you can see from the output of ceph osd df tree that is count of pgs(45/OSD) and use% (65 to 67%). Is that not significant? Correct me if my queries are irrelevant
On November 2, 2023 at 11:36 AM Zakhar Kirpichenko <zakhar@gmail.com> wrote:
Sure, it's 36 OSDs at 200 IOPS each (tops, likely lower), I assume size=3 replication so 1/3 of the total performance, and some 30%-ish OSD overhead.
(36 x 200) * 1/3 * 0.7 = 1680. That's how many IOPS you can realistically expect from your cluster. You get more than that, but the cluster is very busy and OSDs aren't coping.
Also your nodes are not balanced.
/Z
On Thu, 2 Nov 2023 at 07:33, V A Prabha < prabhav@cdac.in> wrote:
Can you please elaborate your identifications and the statement .
On November 2, 2023 at 9:40 AM Zakhar Kirpichenko < zakhar@gmail.com> wrote:
I'm afraid you're simply hitting the I/O limits of your disks.
/Z
On Thu, 2 Nov 2023 at 03:40, V A Prabha < prabhav@cdac.in> wrote:
Hi Eugen Please find the details below
root@meghdootctr1:/var/log/ceph# ceph -s cluster: id: c59da971-57d1-43bd-b2b7-865d392412a5 health: HEALTH_WARN nodeep-scrub flag(s) set 544 pgs not deep-scrubbed in time
services: mon: 3 daemons, quorum meghdootctr1,meghdootctr2,meghdootctr3 (age 5d) mgr: meghdootctr1(active, since 5d), standbys: meghdootctr2, meghdootctr3 mds: 3 up:standby osd: 36 osds: 36 up (since 34h), 36 in (since 34h) flags nodeep-scrub
data: pools: 2 pools, 544 pgs objects: 10.14M objects, 39 TiB usage: 116 TiB used, 63 TiB / 179 TiB avail pgs: 544 active+clean
io: client: 24 MiB/s rd, 16 MiB/s wr, 2.02k op/s rd, 907 op/s wr
Ceph Versions:
root@meghdootctr1:/var/log/ceph# ceph --version ceph version 14.2.16 (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus (stable)
Ceph df -h https://pastebin.com/1ffucyJg
Ceph OSD performance dump https://pastebin.com/1R6YQksE
Ceph tell osd.XX bench (Out of 36 osds only 8 OSDs give High IOPS value of 250 +. Out of that 4 OSDs are from HP 3PAR and 4 OSDS from DELL EMC. We are using only 4 OSDs from HP3 par and it is working fine without any latency and iops issues from the beginning but the remaining 32 OSDs are from DELL EMC in which 4 OSDs are much better than the remaining 28 OSDs)
Please help me to identify if the issue is with the DELL EMC Storage, Ceph configuration parameter tuning or the Overload in the cloud setup
On November 1, 2023 at 9:48 PM Eugen Block < eblock@nde.ag> wrote:
Hi,
for starters please add more cluster details like 'ceph status', 'ceph versions', 'ceph osd df tree'. Increasing the to 10G was the right thing to do, you don't get far with 1G with real cluster load. How are the OSDs configured (HDD only, SSD only or HDD with rocksdb on SSD)? How is the disk utilization?
Regards, Eugen
Zitat von prabhav@cdac.in:
In a production setup of 36 OSDs( SAS disks) totalling 180 TB allocated to a single Ceph Cluster with 3 monitors and 3 managers. There were 830 volumes and VMs created in Openstack with Ceph as a backend. On Sep 21, users reported slowness in accessing the VMs. Analysing the logs lead us to problem with SAS , Network congestion and Ceph configuration( as all default values were used). We updated the Network from 1Gbps to 10Gbps for public and cluster networking. There was no change. The ceph benchmark performance showed that 28 OSDs out of 36 OSDs reported very low IOPS of 30 to 50 while the remaining showed 300+ IOPS. We gradually started reducing the load on the ceph cluster and now the volumes count is 650. Now the slow operations has gradually reduced but I am aware that this is not the solution. Ceph configuration is updated with increasing the osd_journal_size to 10 GB, osd_max_backfills = 1 osd_recovery_max_active = 1 osd_recovery_op_priority = 1 bluestore_cache_trim_max_skip_pinned=10000
After one month, now we faced another issue with Mgr daemon stopped in all 3 quorums and 16 OSDs went down. From the ceph-mon,ceph-mgr.log could not get the reason. Please guide me as its a production setup _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io Thanks & Regards, Ms V A Prabha / श्रीमती प्रभा वी ए Joint Director / संयुक्त निदेशक Centre for Development of Advanced Computing(C-DAC) / प्रगत संगणन विकास केन्द्र(सी-डैक) Tidel Park”, 8th Floor, “D” Block, (North &South) / “टाइडल पार्क”,8वीं मंजिल, “डी” ब्लॉक, (उत्तर और दक्षिण) No.4, Rajiv Gandhi Salai / नं.4, राजीव गांधी सलाई Taramani / तारामणि Chennai / चेन्नई – 600113 Ph.No.:044-22542226/27 Fax No.: 044-22542294
[ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ]
This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken. ------------------------------------------------------------------------------------------------------------
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks & Regards, Ms V A Prabha / श्रीमती प्रभा वी ए Joint Director / संयुक्त निदेशक Centre for Development of Advanced Computing(C-DAC) / प्रगत संगणन विकास केन्द्र(सी-डैक) Tidel Park”, 8th Floor, “D” Block, (North &South) / “टाइडल पार्क”,8वीं मंजिल, “डी” ब्लॉक, (उत्तर और दक्षिण) No.4, Rajiv Gandhi Salai / नं.4, राजीव गांधी सलाई Taramani / तारामणि Chennai / चेन्नई – 600113 Ph.No.:044-22542226/27 Fax No.: 044-22542294
------------------------------------------------------------------------------------------------------------
[ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ]
This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken. ------------------------------------------------------------------------------------------------------------
Thanks & Regards, Ms V A Prabha / श्रीमती प्रभा वी ए Joint Director / संयुक्त निदेशक Centre for Development of Advanced Computing(C-DAC) / प्रगत संगणन विकास केन्द्र(सी-डैक) Tidel Park”, 8th Floor, “D” Block, (North &South) / “टाइडल पार्क”,8वीं मंजिल, “डी” ब्लॉक, (उत्तर और दक्षिण) No.4, Rajiv Gandhi Salai / नं.4, राजीव गांधी सलाई Taramani / तारामणि Chennai / चेन्नई – 600113 Ph.No.:044-22542226/27 Fax No.: 044-22542294
------------------------------------------------------------------------------------------------------------
[ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ]
This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken. ------------------------------------------------------------------------------------------------------------
Is it possible to move the OSDs safe (making the OSDs out and move the content to other OSDs and remove it and map it fresh to other nodes which is less loaded) as I have very critical production workloads ( Government applications) ? Please guide what is the safer means to stabilize the environment without data loss? Moreover we have 64 TB of available storage in the ceph cluster..Is it possible to release the storage atleast 50 % or will it make the cluster very busy again? As the client feels that using 3 replicas and holding these much spare storage , we are not using the storage in an optimal way? Please guide us On November 2, 2023 at 12:47 PM Zakhar Kirpichenko <zakhar@gmail.com> wrote: > >1. The calculated IOPS is for the rw operation right ? > > Total drive IOPS, read or write. Depending on the exact drive models, it may > be lower or higher than 200. I took the average for a smaller sized 7.2k rpm > SAS drive. Modern drives usually deliver lower read IOPS and higher write > IOPS. > > >2. Cluster is very busy? Is there any misconfiguration or missing tuning > >paramater that makes the cluster busy? > > You have almost 3k IOPS and your OSDs report slow ops. I'd say the cluster is > busy, as in loaded with I/O, perhaps more I/O than it can handle well. > > >3. Nodes are not balanced? you mean to say that the count of OSDs in each > >server differs. But we have enabled autoscale and optimal distribution so > >that you can see from the output of ceph osd df tree that is count of > >pgs(45/OSD) and use% (65 to 67%). Is that not significant? > > Yes, the OSD count differs. This means that the CPU, memory usage, network > load and latency differ per node and may cause performance variations, > depending on your workload. > > /Z > > On Thu, 2 Nov 2023 at 08:18, V A Prabha < prabhav@cdac.in > <mailto:prabhav@cdac.in> > wrote: > > > Thanks for your prompt reply .. > > But the query is > > 1.The calculated IOPS is for the rw operation right ? > > 2. Cluster is very busy? Is there any misconfiguration or missing tuning > > paramater that makes the cluster busy? > > 3. Nodes are not balanced? you mean to say that the count of OSDs in > > each server differs. But we have enabled autoscale and optimal distribution > > so that you can see from the output of ceph osd df tree that is count of > > pgs(45/OSD) and use% (65 to 67%). Is that not significant? > > Correct me if my queries are irrelevant > > > > > > > > On November 2, 2023 at 11:36 AM Zakhar Kirpichenko < zakhar@gmail.com > > <mailto:zakhar@gmail.com> > wrote: > > > > > > > Sure, it's 36 OSDs at 200 IOPS each (tops, likely lower), I > > > > > assume size=3 replication so 1/3 of the total performance, and > > > > > some 30%-ish OSD overhead. > > > > > > (36 x 200) * 1/3 * 0.7 = 1680. That's how many IOPS you can > > > realistically expect from your cluster. You get more than that, but the > > > cluster is very busy and OSDs aren't coping. > > > > > > Also your nodes are not balanced. > > > > > > /Z > > > > > > On Thu, 2 Nov 2023 at 07:33, V A Prabha < prabhav@cdac.in > > > <mailto:prabhav@cdac.in> > wrote: > > > > > > > Can you please elaborate your identifications and the > > > > > > > statement . > > > > > > > > > > > > On November 2, 2023 at 9:40 AM Zakhar Kirpichenko < > > > > zakhar@gmail.com <mailto:zakhar@gmail.com> > wrote: > > > > > > > > > > > > > I'm afraid you're simply hitting the I/O limits > > > > > > > > > of your disks. > > > > > > > > > > /Z > > > > > > > > > > On Thu, 2 Nov 2023 at 03:40, V A Prabha < prabhav@cdac.in > > > > > <mailto:prabhav@cdac.in> > wrote: > > > > > > > > > > > Hi Eugen > > > > > > Please find the details below > > > > > > > > > > > > > > > > > > root@meghdootctr1:/var/log/ceph# ceph -s > > > > > > cluster: > > > > > > id: c59da971-57d1-43bd-b2b7-865d392412a5 > > > > > > health: HEALTH_WARN > > > > > > nodeep-scrub flag(s) set > > > > > > 544 pgs not deep-scrubbed in time > > > > > > > > > > > > services: > > > > > > mon: 3 daemons, quorum > > > > > > meghdootctr1,meghdootctr2,meghdootctr3 (age 5d) > > > > > > mgr: meghdootctr1(active, since 5d), standbys: > > > > > > meghdootctr2, meghdootctr3 > > > > > > mds: 3 up:standby > > > > > > osd: 36 osds: 36 up (since 34h), 36 in (since 34h) > > > > > > flags nodeep-scrub > > > > > > > > > > > > data: > > > > > > pools: 2 pools, 544 pgs > > > > > > objects: 10.14M objects, 39 TiB > > > > > > usage: 116 TiB used, 63 TiB / 179 TiB avail > > > > > > pgs: 544 active+clean > > > > > > > > > > > > io: > > > > > > client: 24 MiB/s rd, 16 MiB/s wr, 2.02k op/s rd, 907 op/s > > > > > > wr > > > > > > > > > > > > > > > > > > Ceph Versions: > > > > > > > > > > > > root@meghdootctr1:/var/log/ceph# ceph --version > > > > > > ceph version 14.2.16 > > > > > > (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus > > > > > > (stable) > > > > > > > > > > > > Ceph df -h > > > > > > https://pastebin.com/1ffucyJg > > > > > > <https://pastebin.com/1ffucyJg> > > > > > > > > > > > > Ceph OSD performance dump > > > > > > https://pastebin.com/1R6YQksE > > > > > > <https://pastebin.com/1R6YQksE> > > > > > > > > > > > > Ceph tell osd.XX bench (Out of 36 osds only 8 OSDs give > > > > > > High IOPS value of 250 > > > > > > +. Out of that 4 OSDs are from HP 3PAR and 4 OSDS from DELL > > > > > > EMC. We are using > > > > > > only 4 OSDs from HP3 par and it is working fine without any > > > > > > latency and iops > > > > > > issues from the beginning but the remaining 32 OSDs are > > > > > > from DELL EMC in which 4 > > > > > > OSDs are much better than the remaining 28 OSDs) > > > > > > > > > > > > https://pastebin.com/CixaQmBi > > > > > > <https://pastebin.com/CixaQmBi> > > > > > > > > > > > > Please help me to identify if the issue is with the DELL > > > > > > EMC Storage, Ceph > > > > > > configuration parameter tuning or the Overload in the cloud > > > > > > setup > > > > > > > > > > > > > > > > > > > > > > > > On November 1, 2023 at 9:48 PM Eugen Block < eblock@nde.ag > > > > > > <mailto:eblock@nde.ag> > wrote: > > > > > > > Hi, > > > > > > > > > > > > > > for starters please add more cluster details like 'ceph > > > > > > > status', 'ceph > > > > > > > versions', 'ceph osd df tree'. Increasing the to 10G was > > > > > > > the right > > > > > > > thing to do, you don't get far with 1G with real cluster > > > > > > > load. How are > > > > > > > the OSDs configured (HDD only, SSD only or HDD with > > > > > > > rocksdb on SSD)? > > > > > > > How is the disk utilization? > > > > > > > > > > > > > > Regards, > > > > > > > Eugen > > > > > > > > > > > > > > Zitat von prabhav@cdac.in <mailto:prabhav@cdac.in> : > > > > > > > > > > > > > > > In a production setup of 36 OSDs( SAS disks) totalling > > > > > > > > 180 TB > > > > > > > > allocated to a single Ceph Cluster with 3 monitors and > > > > > > > > 3 managers. > > > > > > > > There were 830 volumes and VMs created in Openstack > > > > > > > > with Ceph as a > > > > > > > > backend. On Sep 21, users reported slowness in > > > > > > > > accessing the VMs. > > > > > > > > Analysing the logs lead us to problem with SAS , > > > > > > > > Network congestion > > > > > > > > and Ceph configuration( as all default values were > > > > > > > > used). We updated > > > > > > > > the Network from 1Gbps to 10Gbps for public and cluster > > > > > > > > networking. > > > > > > > > There was no change. > > > > > > > > The ceph benchmark performance showed that 28 OSDs out > > > > > > > > of 36 OSDs > > > > > > > > reported very low IOPS of 30 to 50 while the remaining > > > > > > > > showed 300+ > > > > > > > > IOPS. > > > > > > > > We gradually started reducing the load on the ceph > > > > > > > > cluster and now > > > > > > > > the volumes count is 650. Now the slow operations has > > > > > > > > gradually > > > > > > > > reduced but I am aware that this is not the solution. > > > > > > > > Ceph configuration is updated with increasing the > > > > > > > > osd_journal_size to 10 GB, > > > > > > > > osd_max_backfills = 1 > > > > > > > > osd_recovery_max_active = 1 > > > > > > > > osd_recovery_op_priority = 1 > > > > > > > > bluestore_cache_trim_max_skip_pinned=10000 > > > > > > > > > > > > > > > > After one month, now we faced another issue with Mgr > > > > > > > > daemon stopped > > > > > > > > in all 3 quorums and 16 OSDs went down. From the > > > > > > > > ceph-mon,ceph-mgr.log could not get the reason. Please > > > > > > > > guide me as > > > > > > > > its a production setup > > > > > > > > _______________________________________________ > > > > > > > > ceph-users mailing list -- ceph-users@ceph.io > > > > > > > > <mailto:ceph-users@ceph.io> > > > > > > > > To unsubscribe send an email to > > > > > > > > ceph-users-leave@ceph.io > > > > > > > > <mailto:ceph-users-leave@ceph.io> > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > ceph-users mailing list -- ceph-users@ceph.io > > > > > > > <mailto:ceph-users@ceph.io> > > > > > > > To unsubscribe send an email to ceph-users-leave@ceph.io > > > > > > > <mailto:ceph-users-leave@ceph.io> > > > > > > Thanks & Regards, > > > > > > Ms V A Prabha / श्रीमती प्रभा वी ए > > > > > > Joint Director / संयुक्त निदेशक > > > > > > Centre for Development of Advanced Computing(C-DAC) / प्रगत > > > > > > संगणन विकास > > > > > > केन्द्र(सी-डैक) > > > > > > Tidel Park”, 8th Floor, “D” Block, (North &South) / “टाइडल > > > > > > पार्क”,8वीं मंजिल, > > > > > > “डी” ब्लॉक, (उत्तर और दक्षिण) > > > > > > No.4, Rajiv Gandhi Salai / नं.4, राजीव गांधी सलाई > > > > > > Taramani / तारामणि > > > > > > Chennai / चेन्नई – 600113 > > > > > > Ph.No.:044-22542226/27 > > > > > > Fax No.: 044-22542294 > > > > > > > > > > > > ------------------------------------------------------------------------------------------------------------ > > > > > > [ C-DAC is on Social-Media too. Kindly follow us at: > > > > > > Facebook: https://www.facebook.com/CDACINDIA > > > > > > <https://www.facebook.com/CDACINDIA> & Twitter: @cdacindia ] > > > > > > > > > > > > This e-mail is for the sole use of the intended > > > > > > recipient(s) and may > > > > > > contain confidential and privileged information. If you are > > > > > > not the > > > > > > intended recipient, please contact the sender by reply > > > > > > e-mail and destroy > > > > > > all copies and the original message. Any unauthorized > > > > > > review, use, > > > > > > disclosure, dissemination, forwarding, printing or copying > > > > > > of this email > > > > > > is strictly prohibited and appropriate legal action will be > > > > > > taken. > > > > > > > > > > > > ------------------------------------------------------------------------------------------------------------ > > > > > > > > > > > > _______________________________________________ > > > > > > ceph-users mailing list -- ceph-users@ceph.io > > > > > > <mailto:ceph-users@ceph.io> > > > > > > To unsubscribe send an email to ceph-users-leave@ceph.io > > > > > > <mailto:ceph-users-leave@ceph.io> > > > > > > > > > > > > > > > > > > > > > > > > Thanks & Regards, > > > > Ms V A Prabha / श्रीमती प्रभा वी ए > > > > Joint Director / संयुक्त निदेशक > > > > Centre for Development of Advanced Computing(C-DAC) / प्रगत संगणन > > > > विकास केन्द्र(सी-डैक) > > > > Tidel Park”, 8th Floor, “D” Block, (North &South) / “टाइडल > > > > पार्क”,8वीं मंजिल, “डी” ब्लॉक, (उत्तर और दक्षिण) > > > > No.4, Rajiv Gandhi Salai / नं.4, राजीव गांधी सलाई > > > > Taramani / तारामणि > > > > Chennai / चेन्नई – 600113 > > > > Ph.No.:044-22542226/27 > > > > Fax No.: 044-22542294 > > > > > > > > > > > > ------------------------------------------------------------------------------------------------------------ > > > > [ C-DAC is on Social-Media too. Kindly follow us at: > > > > Facebook: https://www.facebook.com/CDACINDIA > > > > <https://www.facebook.com/CDACINDIA> & Twitter: @cdacindia ] > > > > > > > > This e-mail is for the sole use of the intended recipient(s) and > > > > may > > > > contain confidential and privileged information. If you are not > > > > the > > > > intended recipient, please contact the sender by reply e-mail and > > > > destroy > > > > all copies and the original message. Any unauthorized review, use, > > > > disclosure, dissemination, forwarding, printing or copying of this > > > > email > > > > is strictly prohibited and appropriate legal action will be taken. > > > > > > > > ------------------------------------------------------------------------------------------------------------ > > > > > > > > > > > > > > Thanks & Regards, > > Ms V A Prabha / श्रीमती प्रभा वी ए > > Joint Director / संयुक्त निदेशक > > Centre for Development of Advanced Computing(C-DAC) / प्रगत संगणन विकास > > केन्द्र(सी-डैक) > > Tidel Park”, 8th Floor, “D” Block, (North &South) / “टाइडल पार्क”,8वीं > > मंजिल, “डी” ब्लॉक, (उत्तर और दक्षिण) > > No.4, Rajiv Gandhi Salai / नं.4, राजीव गांधी सलाई > > Taramani / तारामणि > > Chennai / चेन्नई – 600113 > > Ph.No.:044-22542226/27 > > Fax No.: 044-22542294 > > > > > > ------------------------------------------------------------------------------------------------------------ > > [ C-DAC is on Social-Media too. Kindly follow us at: > > Facebook: https://www.facebook.com/CDACINDIA > > <https://www.facebook.com/CDACINDIA> & Twitter: @cdacindia ] > > > > This e-mail is for the sole use of the intended recipient(s) and may > > contain confidential and privileged information. If you are not the > > intended recipient, please contact the sender by reply e-mail and destroy > > all copies and the original message. Any unauthorized review, use, > > disclosure, dissemination, forwarding, printing or copying of this email > > is strictly prohibited and appropriate legal action will be taken. > > > > ------------------------------------------------------------------------------------------------------------ > > > Thanks & Regards, Ms V A Prabha / श्रीमती प्रभा वी ए Joint Director / संयुक्त निदेशक Centre for Development of Advanced Computing(C-DAC) / प्रगत संगणन विकास केन्द्र(सी-डैक) Tidel Park”, 8th Floor, “D” Block, (North &South) / “टाइडल पार्क”,8वीं मंजिल, “डी” ब्लॉक, (उत्तर और दक्षिण) No.4, Rajiv Gandhi Salai / नं.4, राजीव गांधी सलाई Taramani / तारामणि Chennai / चेन्नई – 600113 Ph.No.:044-22542226/27 Fax No.: 044-22542294 ------------------------------------------------------------------------------------------------------------ [ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ] This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken. ------------------------------------------------------------------------------------------------------------
Den tors 2 nov. 2023 kl 23:46 skrev V A Prabha <prabhav@cdac.in>:
Is it possible to move the OSDs safe (making the OSDs out and move the content to other OSDs and remove it and map it fresh to other nodes which is less loaded)
As the client feels that using 3 replicas and holding these much spare storage , we are not using the storage in an optimal way?
These two sentences don't really add up. Ceph has replica=3 as a default in order for drives and hosts to be able to crash, so that you can recover without losing redundancy. As soon as you lose redundancy, there is no "safe" anything, you are immediately in danger of losing data so that you can never get it back from ceph. If the client thinks you are wasting space, then they must not care for the data, because ANY random hiccup, any broken sector somewhere becomes a data-loss event if the other copy is being moved, or that server is having maintenance or whatever. With only two copies of the data, you can never reboot a server, upgrades means the cluster stops serving data. The joke in the 80s (might be older than that of course) was: "Data is binary, either it is important and backed up, or it is not important" Ceph chooses to treat your data as important. You can lower the expectations by reducing replicas, or reduce perf with erasure coding (but I understand that this whole thread is about poor total performance of both client traffic and scrubs and so on), but the defaults are there to protect you from any random bit flip on one of the disks and this will happen. Not "perhaps", with enough drives and/or enough time, this is a certainty. Your design of the storage decides how your will survive such an event. If you have three copies of all data, you can be doing (planned or unplanned) maintenance on one OSD box, notice an error somewhere on another box and still recover from this using the third copy. With only two replicas, you can't. The maintenance OSD host will have missed IO that went on while it was away, and the second copy is known bad. You could hope that you never need to do maintenance, but with few exceptions, hosts will reboot at times, whether you plan it or not. -- May the most significant bit of your life be positive.
Thanks for your prompt reply and that clears my doubt. 4 of the OSDs in 2 different nodes goes down daily by getting errors in multipath failure. All the four paths are going to failure mode that makes the OSD down. My query is that as the ceph cluster is overloaded with IOPS does the multipaths get failed or it is the issue with the SAN device (DELL Unity) itself? On November 3, 2023 at 11:38 AM Janne Johansson <icepic.dz@gmail.com> wrote:
Den tors 2 nov. 2023 kl 23:46 skrev V A Prabha <prabhav@cdac.in>:
Is it possible to move the OSDs safe (making the OSDs out and move the content to other OSDs and remove it and map it fresh to other nodes which is less loaded)
As the client feels that using 3 replicas and holding these much spare storage , we are not using the storage in an optimal way?
These two sentences don't really add up.
Ceph has replica=3 as a default in order for drives and hosts to be able to crash, so that you can recover without losing redundancy. As soon as you lose redundancy, there is no "safe" anything, you are immediately in danger of losing data so that you can never get it back from ceph. If the client thinks you are wasting space, then they must not care for the data, because ANY random hiccup, any broken sector somewhere becomes a data-loss event if the other copy is being moved, or that server is having maintenance or whatever. With only two copies of the data, you can never reboot a server, upgrades means the cluster stops serving data.
The joke in the 80s (might be older than that of course) was: "Data is binary, either it is important and backed up, or it is not important"
Ceph chooses to treat your data as important. You can lower the expectations by reducing replicas, or reduce perf with erasure coding (but I understand that this whole thread is about poor total performance of both client traffic and scrubs and so on), but the defaults are there to protect you from any random bit flip on one of the disks and this will happen. Not "perhaps", with enough drives and/or enough time, this is a certainty. Your design of the storage decides how your will survive such an event.
If you have three copies of all data, you can be doing (planned or unplanned) maintenance on one OSD box, notice an error somewhere on another box and still recover from this using the third copy.
With only two replicas, you can't. The maintenance OSD host will have missed IO that went on while it was away, and the second copy is known bad. You could hope that you never need to do maintenance, but with few exceptions, hosts will reboot at times, whether you plan it or not.
-- May the most significant bit of your life be positive. Thanks & Regards, Ms V A Prabha / श्रीमती प्रभा वी ए Joint Director / संयुक्त निदेशक Centre for Development of Advanced Computing(C-DAC) / प्रगत संगणन विकास केन्द्र(सी-डैक) Tidel Park”, 8th Floor, “D” Block, (North &South) / “टाइडल पार्क”,8वीं मंजिल, “डी” ब्लॉक, (उत्तर और दक्षिण) No.4, Rajiv Gandhi Salai / नं.4, राजीव गांधी सलाई Taramani / तारामणि Chennai / चेन्नई – 600113 Ph.No.:044-22542226/27 Fax No.: 044-22542294
[ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ] This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken. ------------------------------------------------------------------------------------------------------------
Now in this situation how can stabilize my production setup as you have mentioned the cluster is very busy. Is there any configuration parameter tuning will help or the only option is to reduce the applications running on the cluster. Though if I have free available storage of 1.6 TB free in each of my OSD, that will not help in my IOPS issue right? Please guide me On November 2, 2023 at 12:47 PM Zakhar Kirpichenko <zakhar@gmail.com> wrote: > >1. The calculated IOPS is for the rw operation right ? > > Total drive IOPS, read or write. Depending on the exact drive models, it may > be lower or higher than 200. I took the average for a smaller sized 7.2k rpm > SAS drive. Modern drives usually deliver lower read IOPS and higher write > IOPS. > > >2. Cluster is very busy? Is there any misconfiguration or missing tuning > >paramater that makes the cluster busy? > > You have almost 3k IOPS and your OSDs report slow ops. I'd say the cluster is > busy, as in loaded with I/O, perhaps more I/O than it can handle well. > > >3. Nodes are not balanced? you mean to say that the count of OSDs in each > >server differs. But we have enabled autoscale and optimal distribution so > >that you can see from the output of ceph osd df tree that is count of > >pgs(45/OSD) and use% (65 to 67%). Is that not significant? > > Yes, the OSD count differs. This means that the CPU, memory usage, network > load and latency differ per node and may cause performance variations, > depending on your workload. > > /Z > > On Thu, 2 Nov 2023 at 08:18, V A Prabha < prabhav@cdac.in > <mailto:prabhav@cdac.in> > wrote: > > > Thanks for your prompt reply .. > > But the query is > > 1.The calculated IOPS is for the rw operation right ? > > 2. Cluster is very busy? Is there any misconfiguration or missing tuning > > paramater that makes the cluster busy? > > 3. Nodes are not balanced? you mean to say that the count of OSDs in > > each server differs. But we have enabled autoscale and optimal distribution > > so that you can see from the output of ceph osd df tree that is count of > > pgs(45/OSD) and use% (65 to 67%). Is that not significant? > > Correct me if my queries are irrelevant > > > > > > > > On November 2, 2023 at 11:36 AM Zakhar Kirpichenko < zakhar@gmail.com > > <mailto:zakhar@gmail.com> > wrote: > > > > > > > Sure, it's 36 OSDs at 200 IOPS each (tops, likely lower), I > > > > > assume size=3 replication so 1/3 of the total performance, and > > > > > some 30%-ish OSD overhead. > > > > > > (36 x 200) * 1/3 * 0.7 = 1680. That's how many IOPS you can > > > realistically expect from your cluster. You get more than that, but the > > > cluster is very busy and OSDs aren't coping. > > > > > > Also your nodes are not balanced. > > > > > > /Z > > > > > > On Thu, 2 Nov 2023 at 07:33, V A Prabha < prabhav@cdac.in > > > <mailto:prabhav@cdac.in> > wrote: > > > > > > > Can you please elaborate your identifications and the > > > > > > > statement . > > > > > > > > > > > > On November 2, 2023 at 9:40 AM Zakhar Kirpichenko < > > > > zakhar@gmail.com <mailto:zakhar@gmail.com> > wrote: > > > > > > > > > > > > > I'm afraid you're simply hitting the I/O limits > > > > > > > > > of your disks. > > > > > > > > > > /Z > > > > > > > > > > On Thu, 2 Nov 2023 at 03:40, V A Prabha < prabhav@cdac.in > > > > > <mailto:prabhav@cdac.in> > wrote: > > > > > > > > > > > Hi Eugen > > > > > > Please find the details below > > > > > > > > > > > > > > > > > > root@meghdootctr1:/var/log/ceph# ceph -s > > > > > > cluster: > > > > > > id: c59da971-57d1-43bd-b2b7-865d392412a5 > > > > > > health: HEALTH_WARN > > > > > > nodeep-scrub flag(s) set > > > > > > 544 pgs not deep-scrubbed in time > > > > > > > > > > > > services: > > > > > > mon: 3 daemons, quorum > > > > > > meghdootctr1,meghdootctr2,meghdootctr3 (age 5d) > > > > > > mgr: meghdootctr1(active, since 5d), standbys: > > > > > > meghdootctr2, meghdootctr3 > > > > > > mds: 3 up:standby > > > > > > osd: 36 osds: 36 up (since 34h), 36 in (since 34h) > > > > > > flags nodeep-scrub > > > > > > > > > > > > data: > > > > > > pools: 2 pools, 544 pgs > > > > > > objects: 10.14M objects, 39 TiB > > > > > > usage: 116 TiB used, 63 TiB / 179 TiB avail > > > > > > pgs: 544 active+clean > > > > > > > > > > > > io: > > > > > > client: 24 MiB/s rd, 16 MiB/s wr, 2.02k op/s rd, 907 op/s > > > > > > wr > > > > > > > > > > > > > > > > > > Ceph Versions: > > > > > > > > > > > > root@meghdootctr1:/var/log/ceph# ceph --version > > > > > > ceph version 14.2.16 > > > > > > (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus > > > > > > (stable) > > > > > > > > > > > > Ceph df -h > > > > > > https://pastebin.com/1ffucyJg > > > > > > <https://pastebin.com/1ffucyJg> > > > > > > > > > > > > Ceph OSD performance dump > > > > > > https://pastebin.com/1R6YQksE > > > > > > <https://pastebin.com/1R6YQksE> > > > > > > > > > > > > Ceph tell osd.XX bench (Out of 36 osds only 8 OSDs give > > > > > > High IOPS value of 250 > > > > > > +. Out of that 4 OSDs are from HP 3PAR and 4 OSDS from DELL > > > > > > EMC. We are using > > > > > > only 4 OSDs from HP3 par and it is working fine without any > > > > > > latency and iops > > > > > > issues from the beginning but the remaining 32 OSDs are > > > > > > from DELL EMC in which 4 > > > > > > OSDs are much better than the remaining 28 OSDs) > > > > > > > > > > > > https://pastebin.com/CixaQmBi > > > > > > <https://pastebin.com/CixaQmBi> > > > > > > > > > > > > Please help me to identify if the issue is with the DELL > > > > > > EMC Storage, Ceph > > > > > > configuration parameter tuning or the Overload in the cloud > > > > > > setup > > > > > > > > > > > > > > > > > > > > > > > > On November 1, 2023 at 9:48 PM Eugen Block < eblock@nde.ag > > > > > > <mailto:eblock@nde.ag> > wrote: > > > > > > > Hi, > > > > > > > > > > > > > > for starters please add more cluster details like 'ceph > > > > > > > status', 'ceph > > > > > > > versions', 'ceph osd df tree'. Increasing the to 10G was > > > > > > > the right > > > > > > > thing to do, you don't get far with 1G with real cluster > > > > > > > load. How are > > > > > > > the OSDs configured (HDD only, SSD only or HDD with > > > > > > > rocksdb on SSD)? > > > > > > > How is the disk utilization? > > > > > > > > > > > > > > Regards, > > > > > > > Eugen > > > > > > > > > > > > > > Zitat von prabhav@cdac.in <mailto:prabhav@cdac.in> : > > > > > > > > > > > > > > > In a production setup of 36 OSDs( SAS disks) totalling > > > > > > > > 180 TB > > > > > > > > allocated to a single Ceph Cluster with 3 monitors and > > > > > > > > 3 managers. > > > > > > > > There were 830 volumes and VMs created in Openstack > > > > > > > > with Ceph as a > > > > > > > > backend. On Sep 21, users reported slowness in > > > > > > > > accessing the VMs. > > > > > > > > Analysing the logs lead us to problem with SAS , > > > > > > > > Network congestion > > > > > > > > and Ceph configuration( as all default values were > > > > > > > > used). We updated > > > > > > > > the Network from 1Gbps to 10Gbps for public and cluster > > > > > > > > networking. > > > > > > > > There was no change. > > > > > > > > The ceph benchmark performance showed that 28 OSDs out > > > > > > > > of 36 OSDs > > > > > > > > reported very low IOPS of 30 to 50 while the remaining > > > > > > > > showed 300+ > > > > > > > > IOPS. > > > > > > > > We gradually started reducing the load on the ceph > > > > > > > > cluster and now > > > > > > > > the volumes count is 650. Now the slow operations has > > > > > > > > gradually > > > > > > > > reduced but I am aware that this is not the solution. > > > > > > > > Ceph configuration is updated with increasing the > > > > > > > > osd_journal_size to 10 GB, > > > > > > > > osd_max_backfills = 1 > > > > > > > > osd_recovery_max_active = 1 > > > > > > > > osd_recovery_op_priority = 1 > > > > > > > > bluestore_cache_trim_max_skip_pinned=10000 > > > > > > > > > > > > > > > > After one month, now we faced another issue with Mgr > > > > > > > > daemon stopped > > > > > > > > in all 3 quorums and 16 OSDs went down. From the > > > > > > > > ceph-mon,ceph-mgr.log could not get the reason. Please > > > > > > > > guide me as > > > > > > > > its a production setup > > > > > > > > _______________________________________________ > > > > > > > > ceph-users mailing list -- ceph-users@ceph.io > > > > > > > > <mailto:ceph-users@ceph.io> > > > > > > > > To unsubscribe send an email to > > > > > > > > ceph-users-leave@ceph.io > > > > > > > > <mailto:ceph-users-leave@ceph.io> > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > ceph-users mailing list -- ceph-users@ceph.io > > > > > > > <mailto:ceph-users@ceph.io> > > > > > > > To unsubscribe send an email to ceph-users-leave@ceph.io > > > > > > > <mailto:ceph-users-leave@ceph.io> > > > > > > Thanks & Regards, > > > > > > Ms V A Prabha / श्रीमती प्रभा वी ए > > > > > > Joint Director / संयुक्त निदेशक > > > > > > Centre for Development of Advanced Computing(C-DAC) / प्रगत > > > > > > संगणन विकास > > > > > > केन्द्र(सी-डैक) > > > > > > Tidel Park”, 8th Floor, “D” Block, (North &South) / “टाइडल > > > > > > पार्क”,8वीं मंजिल, > > > > > > “डी” ब्लॉक, (उत्तर और दक्षिण) > > > > > > No.4, Rajiv Gandhi Salai / नं.4, राजीव गांधी सलाई > > > > > > Taramani / तारामणि > > > > > > Chennai / चेन्नई – 600113 > > > > > > Ph.No.:044-22542226/27 > > > > > > Fax No.: 044-22542294 > > > > > > > > > > > > ------------------------------------------------------------------------------------------------------------ > > > > > > [ C-DAC is on Social-Media too. Kindly follow us at: > > > > > > Facebook: https://www.facebook.com/CDACINDIA > > > > > > <https://www.facebook.com/CDACINDIA> & Twitter: @cdacindia ] > > > > > > > > > > > > This e-mail is for the sole use of the intended > > > > > > recipient(s) and may > > > > > > contain confidential and privileged information. If you are > > > > > > not the > > > > > > intended recipient, please contact the sender by reply > > > > > > e-mail and destroy > > > > > > all copies and the original message. Any unauthorized > > > > > > review, use, > > > > > > disclosure, dissemination, forwarding, printing or copying > > > > > > of this email > > > > > > is strictly prohibited and appropriate legal action will be > > > > > > taken. > > > > > > > > > > > > ------------------------------------------------------------------------------------------------------------ > > > > > > > > > > > > _______________________________________________ > > > > > > ceph-users mailing list -- ceph-users@ceph.io > > > > > > <mailto:ceph-users@ceph.io> > > > > > > To unsubscribe send an email to ceph-users-leave@ceph.io > > > > > > <mailto:ceph-users-leave@ceph.io> > > > > > > > > > > > > > > > > > > > > > > > > Thanks & Regards, > > > > Ms V A Prabha / श्रीमती प्रभा वी ए > > > > Joint Director / संयुक्त निदेशक > > > > Centre for Development of Advanced Computing(C-DAC) / प्रगत संगणन > > > > विकास केन्द्र(सी-डैक) > > > > Tidel Park”, 8th Floor, “D” Block, (North &South) / “टाइडल > > > > पार्क”,8वीं मंजिल, “डी” ब्लॉक, (उत्तर और दक्षिण) > > > > No.4, Rajiv Gandhi Salai / नं.4, राजीव गांधी सलाई > > > > Taramani / तारामणि > > > > Chennai / चेन्नई – 600113 > > > > Ph.No.:044-22542226/27 > > > > Fax No.: 044-22542294 > > > > > > > > > > > > ------------------------------------------------------------------------------------------------------------ > > > > [ C-DAC is on Social-Media too. Kindly follow us at: > > > > Facebook: https://www.facebook.com/CDACINDIA > > > > <https://www.facebook.com/CDACINDIA> & Twitter: @cdacindia ] > > > > > > > > This e-mail is for the sole use of the intended recipient(s) and > > > > may > > > > contain confidential and privileged information. If you are not > > > > the > > > > intended recipient, please contact the sender by reply e-mail and > > > > destroy > > > > all copies and the original message. Any unauthorized review, use, > > > > disclosure, dissemination, forwarding, printing or copying of this > > > > email > > > > is strictly prohibited and appropriate legal action will be taken. > > > > > > > > ------------------------------------------------------------------------------------------------------------ > > > > > > > > > > > > > > Thanks & Regards, > > Ms V A Prabha / श्रीमती प्रभा वी ए > > Joint Director / संयुक्त निदेशक > > Centre for Development of Advanced Computing(C-DAC) / प्रगत संगणन विकास > > केन्द्र(सी-डैक) > > Tidel Park”, 8th Floor, “D” Block, (North &South) / “टाइडल पार्क”,8वीं > > मंजिल, “डी” ब्लॉक, (उत्तर और दक्षिण) > > No.4, Rajiv Gandhi Salai / नं.4, राजीव गांधी सलाई > > Taramani / तारामणि > > Chennai / चेन्नई – 600113 > > Ph.No.:044-22542226/27 > > Fax No.: 044-22542294 > > > > > > ------------------------------------------------------------------------------------------------------------ > > [ C-DAC is on Social-Media too. Kindly follow us at: > > Facebook: https://www.facebook.com/CDACINDIA > > <https://www.facebook.com/CDACINDIA> & Twitter: @cdacindia ] > > > > This e-mail is for the sole use of the intended recipient(s) and may > > contain confidential and privileged information. If you are not the > > intended recipient, please contact the sender by reply e-mail and destroy > > all copies and the original message. Any unauthorized review, use, > > disclosure, dissemination, forwarding, printing or copying of this email > > is strictly prohibited and appropriate legal action will be taken. > > > > ------------------------------------------------------------------------------------------------------------ > > > Thanks & Regards, Ms V A Prabha / श्रीमती प्रभा वी ए Joint Director / संयुक्त निदेशक Centre for Development of Advanced Computing(C-DAC) / प्रगत संगणन विकास केन्द्र(सी-डैक) Tidel Park”, 8th Floor, “D” Block, (North &South) / “टाइडल पार्क”,8वीं मंजिल, “डी” ब्लॉक, (उत्तर और दक्षिण) No.4, Rajiv Gandhi Salai / नं.4, राजीव गांधी सलाई Taramani / तारामणि Chennai / चेन्नई – 600113 Ph.No.:044-22542226/27 Fax No.: 044-22542294 ------------------------------------------------------------------------------------------------------------ [ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ] This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken. ------------------------------------------------------------------------------------------------------------
You have an IOPS budget, i.e. how much I/O your spinners can deliver. Space utilization doesn't affect it much. You can try disabling write (not read!) cache on your HDDs with sdparm (for example, sdparm -c WCE /dev/bla); in my experience this allows HDDs to deliver 50-100% more write IOPS. If there is lots of free RAM on the OSD nodes, you can play with osd_memory_target and bluestore_cache_size_hdd OSD options; be careful though: depending on you workload, the performance impact may be insignificant, but your OSDs may run out of memory. /z On Sat, 4 Nov 2023 at 12:04, V A Prabha <prabhav@cdac.in> wrote:
Now in this situation how can stabilize my production setup as you have mentioned the cluster is very busy. Is there any configuration parameter tuning will help or the only option is to reduce the applications running on the cluster. Though if I have free available storage of 1.6 TB free in each of my OSD, that will not help in my IOPS issue right? Please guide me
On November 2, 2023 at 12:47 PM Zakhar Kirpichenko <zakhar@gmail.com> wrote:
1. The calculated IOPS is for the rw operation right ?
Total drive IOPS, read or write. Depending on the exact drive models, it may be lower or higher than 200. I took the average for a smaller sized 7.2k rpm SAS drive. Modern drives usually deliver lower read IOPS and higher write IOPS.
2. Cluster is very busy? Is there any misconfiguration or missing tuning paramater that makes the cluster busy?
You have almost 3k IOPS and your OSDs report slow ops. I'd say the cluster is busy, as in loaded with I/O, perhaps more I/O than it can handle well.
3. Nodes are not balanced? you mean to say that the count of OSDs in each server differs. But we have enabled autoscale and optimal distribution so that you can see from the output of ceph osd df tree that is count of pgs(45/OSD) and use% (65 to 67%). Is that not significant?
Yes, the OSD count differs. This means that the CPU, memory usage, network load and latency differ per node and may cause performance variations, depending on your workload.
/Z
On Thu, 2 Nov 2023 at 08:18, V A Prabha < prabhav@cdac.in> wrote:
Thanks for your prompt reply .. But the query is 1.The calculated IOPS is for the rw operation right ? 2. Cluster is very busy? Is there any misconfiguration or missing tuning paramater that makes the cluster busy? 3. Nodes are not balanced? you mean to say that the count of OSDs in each server differs. But we have enabled autoscale and optimal distribution so that you can see from the output of ceph osd df tree that is count of pgs(45/OSD) and use% (65 to 67%). Is that not significant? Correct me if my queries are irrelevant
On November 2, 2023 at 11:36 AM Zakhar Kirpichenko < zakhar@gmail.com> wrote:
Sure, it's 36 OSDs at 200 IOPS each (tops, likely lower), I assume size=3 replication so 1/3 of the total performance, and some 30%-ish OSD overhead.
(36 x 200) * 1/3 * 0.7 = 1680. That's how many IOPS you can realistically expect from your cluster. You get more than that, but the cluster is very busy and OSDs aren't coping.
Also your nodes are not balanced.
/Z
On Thu, 2 Nov 2023 at 07:33, V A Prabha < prabhav@cdac.in> wrote:
Can you please elaborate your identifications and the statement .
On November 2, 2023 at 9:40 AM Zakhar Kirpichenko < zakhar@gmail.com> wrote:
I'm afraid you're simply hitting the I/O limits of your disks.
/Z
On Thu, 2 Nov 2023 at 03:40, V A Prabha < prabhav@cdac.in> wrote:
Hi Eugen Please find the details below
root@meghdootctr1:/var/log/ceph# ceph -s cluster: id: c59da971-57d1-43bd-b2b7-865d392412a5 health: HEALTH_WARN nodeep-scrub flag(s) set 544 pgs not deep-scrubbed in time
services: mon: 3 daemons, quorum meghdootctr1,meghdootctr2,meghdootctr3 (age 5d) mgr: meghdootctr1(active, since 5d), standbys: meghdootctr2, meghdootctr3 mds: 3 up:standby osd: 36 osds: 36 up (since 34h), 36 in (since 34h) flags nodeep-scrub
data: pools: 2 pools, 544 pgs objects: 10.14M objects, 39 TiB usage: 116 TiB used, 63 TiB / 179 TiB avail pgs: 544 active+clean
io: client: 24 MiB/s rd, 16 MiB/s wr, 2.02k op/s rd, 907 op/s wr
Ceph Versions:
root@meghdootctr1:/var/log/ceph# ceph --version ceph version 14.2.16 (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus (stable)
Ceph df -h https://pastebin.com/1ffucyJg
Ceph OSD performance dump https://pastebin.com/1R6YQksE
Ceph tell osd.XX bench (Out of 36 osds only 8 OSDs give High IOPS value of 250 +. Out of that 4 OSDs are from HP 3PAR and 4 OSDS from DELL EMC. We are using only 4 OSDs from HP3 par and it is working fine without any latency and iops issues from the beginning but the remaining 32 OSDs are from DELL EMC in which 4 OSDs are much better than the remaining 28 OSDs)
Please help me to identify if the issue is with the DELL EMC Storage, Ceph configuration parameter tuning or the Overload in the cloud setup
On November 1, 2023 at 9:48 PM Eugen Block < eblock@nde.ag> wrote:
Hi,
for starters please add more cluster details like 'ceph status', 'ceph versions', 'ceph osd df tree'. Increasing the to 10G was the right thing to do, you don't get far with 1G with real cluster load. How are the OSDs configured (HDD only, SSD only or HDD with rocksdb on SSD)? How is the disk utilization?
Regards, Eugen
Zitat von prabhav@cdac.in:
In a production setup of 36 OSDs( SAS disks) totalling 180 TB allocated to a single Ceph Cluster with 3 monitors and 3 managers. There were 830 volumes and VMs created in Openstack with Ceph as a backend. On Sep 21, users reported slowness in accessing the VMs. Analysing the logs lead us to problem with SAS , Network congestion and Ceph configuration( as all default values were used). We updated the Network from 1Gbps to 10Gbps for public and cluster networking. There was no change. The ceph benchmark performance showed that 28 OSDs out of 36 OSDs reported very low IOPS of 30 to 50 while the remaining showed 300+ IOPS. We gradually started reducing the load on the ceph cluster and now the volumes count is 650. Now the slow operations has gradually reduced but I am aware that this is not the solution. Ceph configuration is updated with increasing the osd_journal_size to 10 GB, osd_max_backfills = 1 osd_recovery_max_active = 1 osd_recovery_op_priority = 1 bluestore_cache_trim_max_skip_pinned=10000
After one month, now we faced another issue with Mgr daemon stopped in all 3 quorums and 16 OSDs went down. From the ceph-mon,ceph-mgr.log could not get the reason. Please guide me as its a production setup _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io Thanks & Regards, Ms V A Prabha / श्रीमती प्रभा वी ए Joint Director / संयुक्त निदेशक Centre for Development of Advanced Computing(C-DAC) / प्रगत संगणन विकास केन्द्र(सी-डैक) Tidel Park”, 8th Floor, “D” Block, (North &South) / “टाइडल पार्क”,8वीं मंजिल, “डी” ब्लॉक, (उत्तर और दक्षिण) No.4, Rajiv Gandhi Salai / नं.4, राजीव गांधी सलाई Taramani / तारामणि Chennai / चेन्नई – 600113 Ph.No.:044-22542226/27 Fax No.: 044-22542294
[ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ]
This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken. ------------------------------------------------------------------------------------------------------------
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks & Regards, Ms V A Prabha / श्रीमती प्रभा वी ए Joint Director / संयुक्त निदेशक Centre for Development of Advanced Computing(C-DAC) / प्रगत संगणन विकास केन्द्र(सी-डैक) Tidel Park”, 8th Floor, “D” Block, (North &South) / “टाइडल पार्क”,8वीं मंजिल, “डी” ब्लॉक, (उत्तर और दक्षिण) No.4, Rajiv Gandhi Salai / नं.4, राजीव गांधी सलाई Taramani / तारामणि Chennai / चेन्नई – 600113 Ph.No.:044-22542226/27 Fax No.: 044-22542294
------------------------------------------------------------------------------------------------------------
[ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ]
This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken. ------------------------------------------------------------------------------------------------------------
Thanks & Regards, Ms V A Prabha / श्रीमती प्रभा वी ए Joint Director / संयुक्त निदेशक Centre for Development of Advanced Computing(C-DAC) / प्रगत संगणन विकास केन्द्र(सी-डैक) Tidel Park”, 8th Floor, “D” Block, (North &South) / “टाइडल पार्क”,8वीं मंजिल, “डी” ब्लॉक, (उत्तर और दक्षिण) No.4, Rajiv Gandhi Salai / नं.4, राजीव गांधी सलाई Taramani / तारामणि Chennai / चेन्नई – 600113 Ph.No.:044-22542226/27 Fax No.: 044-22542294
------------------------------------------------------------------------------------------------------------
[ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ]
This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken. ------------------------------------------------------------------------------------------------------------
Thanks & Regards, Ms V A Prabha / श्रीमती प्रभा वी ए Joint Director / संयुक्त निदेशक Centre for Development of Advanced Computing(C-DAC) / प्रगत संगणन विकास केन्द्र(सी-डैक) Tidel Park”, 8th Floor, “D” Block, (North &South) / “टाइडल पार्क”,8वीं मंजिल, “डी” ब्लॉक, (उत्तर और दक्षिण) No.4, Rajiv Gandhi Salai / नं.4, राजीव गांधी सलाई Taramani / तारामणि Chennai / चेन्नई – 600113 Ph.No.:044-22542226/27 Fax No.: 044-22542294
------------------------------------------------------------------------------------------------------------
[ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ]
This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken. ------------------------------------------------------------------------------------------------------------
Please clarify my query. I had 700+ volumes (220 applications) running in 36 OSDs when it reported the slow operations. Due to emergency, we migrated 200+ VMs to another virtualization environment. So we have shutdown all the related VMs in our Openstack production setup running with Ceph. We have not deleted the 200+ volumes from Ceph as waiting for the concurrence from the departments. My query is that even when the applications are down, does the volume at the backend make our cluster busy when there is no active transactions? Is there any parameter in the ceph osd log and ceph mon log that gives me the clue for the cluster business? Is there any zombie or unwanted process that make the ceph cluster busy or the IOPS budget of the disk that makes the cluster busy? On November 4, 2023 at 4:29 PM Zakhar Kirpichenko <zakhar@gmail.com> wrote:
You have an IOPS budget, i.e. how much I/O your spinners can deliver. Space utilization doesn't affect it much.
You can try disabling write (not read!) cache on your HDDs with sdparm (for example, sdparm -c WCE /dev/bla); in my experience this allows HDDs to deliver 50-100% more write IOPS. If there is lots of free RAM on the OSD nodes, you can play with osd_memory_target and bluestore_cache_size_hdd OSD options; be careful though: depending on you workload, the performance impact may be insignificant, but your OSDs may run out of memory.
/z
On Sat, 4 Nov 2023 at 12:04, V A Prabha < prabhav@cdac.in <mailto:prabhav@cdac.in> > wrote:
Now in this situation how can stabilize my production setup as you have mentioned the cluster is very busy. Is there any configuration parameter tuning will help or the only option is to reduce the applications running on the cluster. Though if I have free available storage of 1.6 TB free in each of my OSD, that will not help in my IOPS issue right? Please guide me
On November 2, 2023 at 12:47 PM Zakhar Kirpichenko < zakhar@gmail.com <mailto:zakhar@gmail.com> > wrote:
> > > >1. The calculated IOPS is for the rw operation right ?
Total drive IOPS, read or write. Depending on the exact drive models, it may be lower or higher than 200. I took the average for a smaller sized 7.2k rpm SAS drive. Modern drives usually deliver lower read IOPS and higher write IOPS.
>2. Cluster is very busy? Is there any misconfiguration or missing >tuning paramater that makes the cluster busy?
You have almost 3k IOPS and your OSDs report slow ops. I'd say the cluster is busy, as in loaded with I/O, perhaps more I/O than it can handle well.
>3. Nodes are not balanced? you mean to say that the count of OSDs in >each server differs. But we have enabled autoscale and optimal >distribution so that you can see from the output of ceph osd df tree >that is count of pgs(45/OSD) and use% (65 to 67%). Is that not >significant?
Yes, the OSD count differs. This means that the CPU, memory usage, network load and latency differ per node and may cause performance variations, depending on your workload.
/Z
On Thu, 2 Nov 2023 at 08:18, V A Prabha < prabhav@cdac.in <mailto:prabhav@cdac.in> > wrote: > > > > Thanks for your prompt reply ..
But the query is 1.The calculated IOPS is for the rw operation right ? 2. Cluster is very busy? Is there any misconfiguration or missing tuning paramater that makes the cluster busy? 3. Nodes are not balanced? you mean to say that the count of OSDs in each server differs. But we have enabled autoscale and optimal distribution so that you can see from the output of ceph osd df tree that is count of pgs(45/OSD) and use% (65 to 67%). Is that not significant? Correct me if my queries are irrelevant
On November 2, 2023 at 11:36 AM Zakhar Kirpichenko < zakhar@gmail.com <mailto:zakhar@gmail.com> > wrote:
> > > > > Sure, it's 36 OSDs at 200 IOPS each (tops, > > > > > likely lower), I assume size=3 replication so 1/3 of > > > > > the total performance, and some 30%-ish OSD overhead.
(36 x 200) * 1/3 * 0.7 = 1680. That's how many IOPS you can realistically expect from your cluster. You get more than that, but the cluster is very busy and OSDs aren't coping.
Also your nodes are not balanced.
/Z
On Thu, 2 Nov 2023 at 07:33, V A Prabha < prabhav@cdac.in <mailto:prabhav@cdac.in> > wrote: > > > > > > Can you please elaborate your > > > > > > identifications and the statement .
On November 2, 2023 at 9:40 AM Zakhar Kirpichenko < zakhar@gmail.com <mailto:zakhar@gmail.com> > wrote:
> > > > > > > I'm afraid you're simply hitting > > > > > > > the I/O limits of your disks. > > /Z > > On Thu, 2 Nov 2023 at 03:40, V A Prabha < > prabhav@cdac.in <mailto:prabhav@cdac.in> > wrote: > > > > > > > > > Hi Eugen > > Please find the details below > > > > > > root@meghdootctr1:/var/log/ceph# ceph -s > > cluster: > > id: c59da971-57d1-43bd-b2b7-865d392412a5 > > health: HEALTH_WARN > > nodeep-scrub flag(s) set > > 544 pgs not deep-scrubbed in time > > > > services: > > mon: 3 daemons, quorum > > meghdootctr1,meghdootctr2,meghdootctr3 (age 5d) > > mgr: meghdootctr1(active, since 5d), standbys: > > meghdootctr2, meghdootctr3 > > mds: 3 up:standby > > osd: 36 osds: 36 up (since 34h), 36 in (since 34h) > > flags nodeep-scrub > > > > data: > > pools: 2 pools, 544 pgs > > objects: 10.14M objects, 39 TiB > > usage: 116 TiB used, 63 TiB / 179 TiB avail > > pgs: 544 active+clean > > > > io: > > client: 24 MiB/s rd, 16 MiB/s wr, 2.02k op/s rd, 907 > > op/s wr > > > > > > Ceph Versions: > > > > root@meghdootctr1:/var/log/ceph# ceph --version > > ceph version 14.2.16 > > (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus > > (stable) > > > > Ceph df -h > > https://pastebin.com/1ffucyJg > > <https://pastebin.com/1ffucyJg> > > > > Ceph OSD performance dump > > https://pastebin.com/1R6YQksE > > <https://pastebin.com/1R6YQksE> > > > > Ceph tell osd.XX bench (Out of 36 osds only 8 OSDs > > give High IOPS value of 250 > > +. Out of that 4 OSDs are from HP 3PAR and 4 OSDS > > from DELL EMC. We are using > > only 4 OSDs from HP3 par and it is working fine > > without any latency and iops > > issues from the beginning but the remaining 32 OSDs > > are from DELL EMC in which 4 > > OSDs are much better than the remaining 28 OSDs) > > > > https://pastebin.com/CixaQmBi > > <https://pastebin.com/CixaQmBi> > > > > Please help me to identify if the issue is with the > > DELL EMC Storage, Ceph > > configuration parameter tuning or the Overload in > > the cloud setup > > > > > > > > On November 1, 2023 at 9:48 PM Eugen Block < > > eblock@nde.ag <mailto:eblock@nde.ag> > wrote: > > > Hi, > > > > > > for starters please add more cluster details like > > > 'ceph status', 'ceph > > > versions', 'ceph osd df tree'. Increasing the to > > > 10G was the right > > > thing to do, you don't get far with 1G with real > > > cluster load. How are > > > the OSDs configured (HDD only, SSD only or HDD > > > with rocksdb on SSD)? > > > How is the disk utilization? > > > > > > Regards, > > > Eugen > > > > > > Zitat von prabhav@cdac.in <mailto:prabhav@cdac.in> > > > : > > > > > > > In a production setup of 36 OSDs( SAS disks) > > > > totalling 180 TB > > > > allocated to a single Ceph Cluster with 3 > > > > monitors and 3 managers. > > > > There were 830 volumes and VMs created in > > > > Openstack with Ceph as a > > > > backend. On Sep 21, users reported slowness in > > > > accessing the VMs. > > > > Analysing the logs lead us to problem with SAS , > > > > Network congestion > > > > and Ceph configuration( as all default values > > > > were used). We updated > > > > the Network from 1Gbps to 10Gbps for public and > > > > cluster networking. > > > > There was no change. > > > > The ceph benchmark performance showed that 28 > > > > OSDs out of 36 OSDs > > > > reported very low IOPS of 30 to 50 while the > > > > remaining showed 300+ > > > > IOPS. > > > > We gradually started reducing the load on the > > > > ceph cluster and now > > > > the volumes count is 650. Now the slow > > > > operations has gradually > > > > reduced but I am aware that this is not the > > > > solution. > > > > Ceph configuration is updated with increasing > > > > the > > > > osd_journal_size to 10 GB, > > > > osd_max_backfills = 1 > > > > osd_recovery_max_active = 1 > > > > osd_recovery_op_priority = 1 > > > > bluestore_cache_trim_max_skip_pinned=10000 > > > > > > > > After one month, now we faced another issue with > > > > Mgr daemon stopped > > > > in all 3 quorums and 16 OSDs went down. From the > > > > ceph-mon,ceph-mgr.log could not get the reason. > > > > Please guide me as > > > > its a production setup > > > > _______________________________________________ > > > > ceph-users mailing list -- ceph-users@ceph.io > > > > <mailto:ceph-users@ceph.io> > > > > To unsubscribe send an email to > > > > ceph-users-leave@ceph.io > > > > <mailto:ceph-users-leave@ceph.io> > > > > > > > > > _______________________________________________ > > > ceph-users mailing list -- ceph-users@ceph.io > > > <mailto:ceph-users@ceph.io> > > > To unsubscribe send an email to > > > ceph-users-leave@ceph.io > > > <mailto:ceph-users-leave@ceph.io> > > Thanks & Regards, > > Ms V A Prabha / श्रीमती प्रभा वी ए > > Joint Director / संयुक्त निदेशक > > Centre for Development of Advanced Computing(C-DAC) > > / प्रगत संगणन विकास > > केन्द्र(सी-डैक) > > Tidel Park”, 8th Floor, “D” Block, (North &South) / > > “टाइडल पार्क”,8वीं मंजिल, > > “डी” ब्लॉक, (उत्तर और दक्षिण) > > No.4, Rajiv Gandhi Salai / नं.4, राजीव गांधी सलाई > > Taramani / तारामणि > > Chennai / चेन्नई – 600113 > > Ph.No.:044-22542226/27 > > Fax No.: 044-22542294 > > > > ------------------------------------------------------------------------------------------------------------ > > [ C-DAC is on Social-Media too. Kindly follow us at: > > Facebook: https://www.facebook.com/CDACINDIA > > <https://www.facebook.com/CDACINDIA> & Twitter: @cdacindia ] > > > > This e-mail is for the sole use of the intended > > recipient(s) and may > > contain confidential and privileged information. If > > you are not the > > intended recipient, please contact the sender by > > reply e-mail and destroy > > all copies and the original message. Any > > unauthorized review, use, > > disclosure, dissemination, forwarding, printing or > > copying of this email > > is strictly prohibited and appropriate legal action > > will be taken. > > > > ------------------------------------------------------------------------------------------------------------ > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users@ceph.io > > <mailto:ceph-users@ceph.io> > > To unsubscribe send an email to > > ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io> > > > > > > > > > > > > > > > > Thanks & Regards, Ms V A Prabha / श्रीमती प्रभा वी ए Joint Director / संयुक्त निदेशक Centre for Development of Advanced Computing(C-DAC) / प्रगत संगणन विकास केन्द्र(सी-डैक) Tidel Park”, 8th Floor, “D” Block, (North &South) / “टाइडल पार्क”,8वीं मंजिल, “डी” ब्लॉक, (उत्तर और दक्षिण) No.4, Rajiv Gandhi Salai / नं.4, राजीव गांधी सलाई Taramani / तारामणि Chennai / चेन्नई – 600113 Ph.No.:044-22542226/27 Fax No.: 044-22542294
------------------------------------------------------------------------------------------------------------ [ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA <https://www.facebook.com/CDACINDIA> & Twitter: @cdacindia ]
This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken.
------------------------------------------------------------------------------------------------------------ > > > > >
> > > >
Thanks & Regards, Ms V A Prabha / श्रीमती प्रभा वी ए Joint Director / संयुक्त निदेशक Centre for Development of Advanced Computing(C-DAC) / प्रगत संगणन विकास केन्द्र(सी-डैक) Tidel Park”, 8th Floor, “D” Block, (North &South) / “टाइडल पार्क”,8वीं मंजिल, “डी” ब्लॉक, (उत्तर और दक्षिण) No.4, Rajiv Gandhi Salai / नं.4, राजीव गांधी सलाई Taramani / तारामणि Chennai / चेन्नई – 600113 Ph.No.:044-22542226/27 Fax No.: 044-22542294
------------------------------------------------------------------------------------------------------------ [ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA <https://www.facebook.com/CDACINDIA> & Twitter: @cdacindia ]
This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken.
------------------------------------------------------------------------------------------------------------ > > >
Thanks & Regards, Ms V A Prabha / श्रीमती प्रभा वी ए Joint Director / संयुक्त निदेशक Centre for Development of Advanced Computing(C-DAC) / प्रगत संगणन विकास केन्द्र(सी-डैक) Tidel Park”, 8th Floor, “D” Block, (North &South) / “टाइडल पार्क”,8वीं मंजिल, “डी” ब्लॉक, (उत्तर और दक्षिण) No.4, Rajiv Gandhi Salai / नं.4, राजीव गांधी सलाई Taramani / तारामणि Chennai / चेन्नई – 600113 Ph.No.:044-22542226/27 Fax No.: 044-22542294
------------------------------------------------------------------------------------------------------------ [ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA <https://www.facebook.com/CDACINDIA> & Twitter: @cdacindia ]
This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken.
------------------------------------------------------------------------------------------------------------
Thanks & Regards, Ms V A Prabha / श्रीमती प्रभा वी ए Joint Director / संयुक्त निदेशक Centre for Development of Advanced Computing(C-DAC) / प्रगत संगणन विकास केन्द्र(सी-डैक) Tidel Park”, 8th Floor, “D” Block, (North &South) / “टाइडल पार्क”,8वीं मंजिल, “डी” ब्लॉक, (उत्तर और दक्षिण) No.4, Rajiv Gandhi Salai / नं.4, राजीव गांधी सलाई Taramani / तारामणि Chennai / चेन्नई – 600113 Ph.No.:044-22542226/27 Fax No.: 044-22542294 ------------------------------------------------------------------------------------------------------------ [ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ] This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken. ------------------------------------------------------------------------------------------------------------
Only client I/O, cluster recovery I/O and/or data scrubbing I/O make the cluster "busy". If you have removed client workloads and the cluster is healthy, it should be mostly idle. Simply having data sitting in the cluster but not being accessed or modified doesn't make the cluster do any work, except for scheduled data scrubbing. Note that depending on the data volume and the OSD performance, scrubbing may take considerable time. Monitor logs generally provide a good idea of what's going on with the cluster and whether scrubbing is active. /Z On Mon, 6 Nov 2023 at 12:48, V A Prabha <prabhav@cdac.in> wrote:
Please clarify my query. I had 700+ volumes (220 applications) running in 36 OSDs when it reported the slow operations. Due to emergency, we migrated 200+ VMs to another virtualization environment. So we have shutdown all the related VMs in our Openstack production setup running with Ceph. We have not deleted the 200+ volumes from Ceph as waiting for the concurrence from the departments. My query is that even when the applications are down, does the volume at the backend make our cluster busy when there is no active transactions? Is there any parameter in the ceph osd log and ceph mon log that gives me the clue for the cluster business? Is there any zombie or unwanted process that make the ceph cluster busy or the IOPS budget of the disk that makes the cluster busy?
On November 4, 2023 at 4:29 PM Zakhar Kirpichenko <zakhar@gmail.com> wrote:
You have an IOPS budget, i.e. how much I/O your spinners can deliver. Space utilization doesn't affect it much.
You can try disabling write (not read!) cache on your HDDs with sdparm (for example, sdparm -c WCE /dev/bla); in my experience this allows HDDs to deliver 50-100% more write IOPS. If there is lots of free RAM on the OSD nodes, you can play with osd_memory_target and bluestore_cache_size_hdd OSD options; be careful though: depending on you workload, the performance impact may be insignificant, but your OSDs may run out of memory.
/z
On Sat, 4 Nov 2023 at 12:04, V A Prabha < prabhav@cdac.in> wrote:
Now in this situation how can stabilize my production setup as you have mentioned the cluster is very busy. Is there any configuration parameter tuning will help or the only option is to reduce the applications running on the cluster. Though if I have free available storage of 1.6 TB free in each of my OSD, that will not help in my IOPS issue right? Please guide me
On November 2, 2023 at 12:47 PM Zakhar Kirpichenko < zakhar@gmail.com> wrote:
1. The calculated IOPS is for the rw operation right ?
Total drive IOPS, read or write. Depending on the exact drive models, it may be lower or higher than 200. I took the average for a smaller sized 7.2k rpm SAS drive. Modern drives usually deliver lower read IOPS and higher write IOPS.
2. Cluster is very busy? Is there any misconfiguration or missing tuning paramater that makes the cluster busy?
You have almost 3k IOPS and your OSDs report slow ops. I'd say the cluster is busy, as in loaded with I/O, perhaps more I/O than it can handle well.
3. Nodes are not balanced? you mean to say that the count of OSDs in each server differs. But we have enabled autoscale and optimal distribution so that you can see from the output of ceph osd df tree that is count of pgs(45/OSD) and use% (65 to 67%). Is that not significant?
Yes, the OSD count differs. This means that the CPU, memory usage, network load and latency differ per node and may cause performance variations, depending on your workload.
/Z
On Thu, 2 Nov 2023 at 08:18, V A Prabha < prabhav@cdac.in> wrote:
Thanks for your prompt reply .. But the query is 1.The calculated IOPS is for the rw operation right ? 2. Cluster is very busy? Is there any misconfiguration or missing tuning paramater that makes the cluster busy? 3. Nodes are not balanced? you mean to say that the count of OSDs in each server differs. But we have enabled autoscale and optimal distribution so that you can see from the output of ceph osd df tree that is count of pgs(45/OSD) and use% (65 to 67%). Is that not significant? Correct me if my queries are irrelevant
On November 2, 2023 at 11:36 AM Zakhar Kirpichenko < zakhar@gmail.com> wrote:
Sure, it's 36 OSDs at 200 IOPS each (tops, likely lower), I assume size=3 replication so 1/3 of the total performance, and some 30%-ish OSD overhead.
(36 x 200) * 1/3 * 0.7 = 1680. That's how many IOPS you can realistically expect from your cluster. You get more than that, but the cluster is very busy and OSDs aren't coping.
Also your nodes are not balanced.
/Z
On Thu, 2 Nov 2023 at 07:33, V A Prabha < prabhav@cdac.in> wrote:
Can you please elaborate your identifications and the statement .
On November 2, 2023 at 9:40 AM Zakhar Kirpichenko < zakhar@gmail.com> wrote:
I'm afraid you're simply hitting the I/O limits of your disks.
/Z
On Thu, 2 Nov 2023 at 03:40, V A Prabha < prabhav@cdac.in> wrote:
Hi Eugen Please find the details below
root@meghdootctr1:/var/log/ceph# ceph -s cluster: id: c59da971-57d1-43bd-b2b7-865d392412a5 health: HEALTH_WARN nodeep-scrub flag(s) set 544 pgs not deep-scrubbed in time
services: mon: 3 daemons, quorum meghdootctr1,meghdootctr2,meghdootctr3 (age 5d) mgr: meghdootctr1(active, since 5d), standbys: meghdootctr2, meghdootctr3 mds: 3 up:standby osd: 36 osds: 36 up (since 34h), 36 in (since 34h) flags nodeep-scrub
data: pools: 2 pools, 544 pgs objects: 10.14M objects, 39 TiB usage: 116 TiB used, 63 TiB / 179 TiB avail pgs: 544 active+clean
io: client: 24 MiB/s rd, 16 MiB/s wr, 2.02k op/s rd, 907 op/s wr
Ceph Versions:
root@meghdootctr1:/var/log/ceph# ceph --version ceph version 14.2.16 (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus (stable)
Ceph df -h https://pastebin.com/1ffucyJg
Ceph OSD performance dump https://pastebin.com/1R6YQksE
Ceph tell osd.XX bench (Out of 36 osds only 8 OSDs give High IOPS value of 250 +. Out of that 4 OSDs are from HP 3PAR and 4 OSDS from DELL EMC. We are using only 4 OSDs from HP3 par and it is working fine without any latency and iops issues from the beginning but the remaining 32 OSDs are from DELL EMC in which 4 OSDs are much better than the remaining 28 OSDs)
Please help me to identify if the issue is with the DELL EMC Storage, Ceph configuration parameter tuning or the Overload in the cloud setup
On November 1, 2023 at 9:48 PM Eugen Block < eblock@nde.ag> wrote:
Hi,
for starters please add more cluster details like 'ceph status', 'ceph versions', 'ceph osd df tree'. Increasing the to 10G was the right thing to do, you don't get far with 1G with real cluster load. How are the OSDs configured (HDD only, SSD only or HDD with rocksdb on SSD)? How is the disk utilization?
Regards, Eugen
Zitat von prabhav@cdac.in:
In a production setup of 36 OSDs( SAS disks) totalling 180 TB allocated to a single Ceph Cluster with 3 monitors and 3 managers. There were 830 volumes and VMs created in Openstack with Ceph as a backend. On Sep 21, users reported slowness in accessing the VMs. Analysing the logs lead us to problem with SAS , Network congestion and Ceph configuration( as all default values were used). We updated the Network from 1Gbps to 10Gbps for public and cluster networking. There was no change. The ceph benchmark performance showed that 28 OSDs out of 36 OSDs reported very low IOPS of 30 to 50 while the remaining showed 300+ IOPS. We gradually started reducing the load on the ceph cluster and now the volumes count is 650. Now the slow operations has gradually reduced but I am aware that this is not the solution. Ceph configuration is updated with increasing the osd_journal_size to 10 GB, osd_max_backfills = 1 osd_recovery_max_active = 1 osd_recovery_op_priority = 1 bluestore_cache_trim_max_skip_pinned=10000
After one month, now we faced another issue with Mgr daemon stopped in all 3 quorums and 16 OSDs went down. From the ceph-mon,ceph-mgr.log could not get the reason. Please guide me as its a production setup _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io Thanks & Regards, Ms V A Prabha / श्रीमती प्रभा वी ए Joint Director / संयुक्त निदेशक Centre for Development of Advanced Computing(C-DAC) / प्रगत संगणन विकास केन्द्र(सी-डैक) Tidel Park”, 8th Floor, “D” Block, (North &South) / “टाइडल पार्क”,8वीं मंजिल, “डी” ब्लॉक, (उत्तर और दक्षिण) No.4, Rajiv Gandhi Salai / नं.4, राजीव गांधी सलाई Taramani / तारामणि Chennai / चेन्नई – 600113 Ph.No.:044-22542226/27 Fax No.: 044-22542294
[ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ]
This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken. ------------------------------------------------------------------------------------------------------------
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks & Regards, Ms V A Prabha / श्रीमती प्रभा वी ए Joint Director / संयुक्त निदेशक Centre for Development of Advanced Computing(C-DAC) / प्रगत संगणन विकास केन्द्र(सी-डैक) Tidel Park”, 8th Floor, “D” Block, (North &South) / “टाइडल पार्क”,8वीं मंजिल, “डी” ब्लॉक, (उत्तर और दक्षिण) No.4, Rajiv Gandhi Salai / नं.4, राजीव गांधी सलाई Taramani / तारामणि Chennai / चेन्नई – 600113 Ph.No.:044-22542226/27 Fax No.: 044-22542294
------------------------------------------------------------------------------------------------------------
[ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ]
This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken. ------------------------------------------------------------------------------------------------------------
Thanks & Regards, Ms V A Prabha / श्रीमती प्रभा वी ए Joint Director / संयुक्त निदेशक Centre for Development of Advanced Computing(C-DAC) / प्रगत संगणन विकास केन्द्र(सी-डैक) Tidel Park”, 8th Floor, “D” Block, (North &South) / “टाइडल पार्क”,8वीं मंजिल, “डी” ब्लॉक, (उत्तर और दक्षिण) No.4, Rajiv Gandhi Salai / नं.4, राजीव गांधी सलाई Taramani / तारामणि Chennai / चेन्नई – 600113 Ph.No.:044-22542226/27 Fax No.: 044-22542294
------------------------------------------------------------------------------------------------------------
[ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ]
This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken. ------------------------------------------------------------------------------------------------------------
Thanks & Regards, Ms V A Prabha / श्रीमती प्रभा वी ए Joint Director / संयुक्त निदेशक Centre for Development of Advanced Computing(C-DAC) / प्रगत संगणन विकास केन्द्र(सी-डैक) Tidel Park”, 8th Floor, “D” Block, (North &South) / “टाइडल पार्क”,8वीं मंजिल, “डी” ब्लॉक, (उत्तर और दक्षिण) No.4, Rajiv Gandhi Salai / नं.4, राजीव गांधी सलाई Taramani / तारामणि Chennai / चेन्नई – 600113 Ph.No.:044-22542226/27 Fax No.: 044-22542294
------------------------------------------------------------------------------------------------------------
[ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ]
This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken. ------------------------------------------------------------------------------------------------------------
Thanks & Regards, Ms V A Prabha / श्रीमती प्रभा वी ए Joint Director / संयुक्त निदेशक Centre for Development of Advanced Computing(C-DAC) / प्रगत संगणन विकास केन्द्र(सी-डैक) Tidel Park”, 8th Floor, “D” Block, (North &South) / “टाइडल पार्क”,8वीं मंजिल, “डी” ब्लॉक, (उत्तर और दक्षिण) No.4, Rajiv Gandhi Salai / नं.4, राजीव गांधी सलाई Taramani / तारामणि Chennai / चेन्नई – 600113 Ph.No.:044-22542226/27 Fax No.: 044-22542294
------------------------------------------------------------------------------------------------------------
[ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ]
This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken. ------------------------------------------------------------------------------------------------------------
Hi Just a continuation of this mail, Could you help me out to understand the ceph df output. PFA the screenshot with this mail. 1. Raw storage is 180 TB 2. Stored Value is 37 TB 3. Used Value is 112 TB 4. Available Value is 67 TB 5. Pool Max Available Value is 16 TB Though the Available Value is still 67 TB how can it be utilized for the pools? On November 6, 2023 at 4:18 PM V A Prabha <prabhav@cdac.in> wrote:
Please clarify my query. I had 700+ volumes (220 applications) running in 36 OSDs when it reported the slow operations. Due to emergency, we migrated 200+ VMs to another virtualization environment. So we have shutdown all the related VMs in our Openstack production setup running with Ceph. We have not deleted the 200+ volumes from Ceph as waiting for the concurrence from the departments. My query is that even when the applications are down, does the volume at the backend make our cluster busy when there is no active transactions? Is there any parameter in the ceph osd log and ceph mon log that gives me the clue for the cluster business? Is there any zombie or unwanted process that make the ceph cluster busy or the IOPS budget of the disk that makes the cluster busy?
On November 4, 2023 at 4:29 PM Zakhar Kirpichenko <zakhar@gmail.com> wrote:
You have an IOPS budget, i.e. how much I/O your spinners can deliver. Space utilization doesn't affect it much.
You can try disabling write (not read!) cache on your HDDs with sdparm (for example, sdparm -c WCE /dev/bla); in my experience this allows HDDs to deliver 50-100% more write IOPS. If there is lots of free RAM on the OSD nodes, you can play with osd_memory_target and bluestore_cache_size_hdd OSD options; be careful though: depending on you workload, the performance impact may be insignificant, but your OSDs may run out of memory.
/z
On Sat, 4 Nov 2023 at 12:04, V A Prabha < prabhav@cdac.in <mailto:prabhav@cdac.in> > wrote: > > > Now in this situation how can stabilize my production setup as > > > you have mentioned the cluster is very busy.
Is there any configuration parameter tuning will help or the only option is to reduce the applications running on the cluster. Though if I have free available storage of 1.6 TB free in each of my OSD, that will not help in my IOPS issue right? Please guide me
On November 2, 2023 at 12:47 PM Zakhar Kirpichenko < zakhar@gmail.com <mailto:zakhar@gmail.com> > wrote:
> > > > >1. The calculated IOPS is for the rw operation right ?
Total drive IOPS, read or write. Depending on the exact drive models, it may be lower or higher than 200. I took the average for a smaller sized 7.2k rpm SAS drive. Modern drives usually deliver lower read IOPS and higher write IOPS.
>2. Cluster is very busy? Is there any misconfiguration or missing >tuning paramater that makes the cluster busy?
You have almost 3k IOPS and your OSDs report slow ops. I'd say the cluster is busy, as in loaded with I/O, perhaps more I/O than it can handle well.
>3. Nodes are not balanced? you mean to say that the count of OSDs >in each server differs. But we have enabled autoscale and optimal >distribution so that you can see from the output of ceph osd df >tree that is count of pgs(45/OSD) and use% (65 to 67%). Is that >not significant?
Yes, the OSD count differs. This means that the CPU, memory usage, network load and latency differ per node and may cause performance variations, depending on your workload.
/Z
On Thu, 2 Nov 2023 at 08:18, V A Prabha < prabhav@cdac.in <mailto:prabhav@cdac.in> > wrote: > > > > > Thanks for your prompt reply ..
But the query is 1.The calculated IOPS is for the rw operation right ? 2. Cluster is very busy? Is there any misconfiguration or missing tuning paramater that makes the cluster busy? 3. Nodes are not balanced? you mean to say that the count of OSDs in each server differs. But we have enabled autoscale and optimal distribution so that you can see from the output of ceph osd df tree that is count of pgs(45/OSD) and use% (65 to 67%). Is that not significant? Correct me if my queries are irrelevant
On November 2, 2023 at 11:36 AM Zakhar Kirpichenko < zakhar@gmail.com <mailto:zakhar@gmail.com> > wrote:
> > > > > > Sure, it's 36 OSDs at 200 IOPS each (tops, > > > > > > likely lower), I assume size=3 replication so 1/3 > > > > > > of the total performance, and some 30%-ish OSD > > > > > > overhead.
(36 x 200) * 1/3 * 0.7 = 1680. That's how many IOPS you can realistically expect from your cluster. You get more than that, but the cluster is very busy and OSDs aren't coping.
Also your nodes are not balanced.
/Z
On Thu, 2 Nov 2023 at 07:33, V A Prabha < prabhav@cdac.in <mailto:prabhav@cdac.in> > wrote: > > > > > > > Can you please elaborate your > > > > > > > identifications and the statement . > > > On November 2, 2023 at 9:40 AM Zakhar Kirpichenko < > zakhar@gmail.com <mailto:zakhar@gmail.com> > wrote: > > > > > > > > > > I'm afraid you're simply > > > > > > > > > hitting the I/O limits of your disks. > > > > /Z > > > > On Thu, 2 Nov 2023 at 03:40, V A Prabha < > > prabhav@cdac.in <mailto:prabhav@cdac.in> > wrote: > > > > > > > > > > > Hi Eugen > > > Please find the details below > > > > > > > > > root@meghdootctr1:/var/log/ceph# ceph -s > > > cluster: > > > id: c59da971-57d1-43bd-b2b7-865d392412a5 > > > health: HEALTH_WARN > > > nodeep-scrub flag(s) set > > > 544 pgs not deep-scrubbed in time > > > > > > services: > > > mon: 3 daemons, quorum > > > meghdootctr1,meghdootctr2,meghdootctr3 (age 5d) > > > mgr: meghdootctr1(active, since 5d), standbys: > > > meghdootctr2, meghdootctr3 > > > mds: 3 up:standby > > > osd: 36 osds: 36 up (since 34h), 36 in (since > > > 34h) > > > flags nodeep-scrub > > > > > > data: > > > pools: 2 pools, 544 pgs > > > objects: 10.14M objects, 39 TiB > > > usage: 116 TiB used, 63 TiB / 179 TiB avail > > > pgs: 544 active+clean > > > > > > io: > > > client: 24 MiB/s rd, 16 MiB/s wr, 2.02k op/s rd, > > > 907 op/s wr > > > > > > > > > Ceph Versions: > > > > > > root@meghdootctr1:/var/log/ceph# ceph --version > > > ceph version 14.2.16 > > > (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus > > > (stable) > > > > > > Ceph df -h > > > https://pastebin.com/1ffucyJg > > > <https://pastebin.com/1ffucyJg> > > > > > > Ceph OSD performance dump > > > https://pastebin.com/1R6YQksE > > > <https://pastebin.com/1R6YQksE> > > > > > > Ceph tell osd.XX bench (Out of 36 osds only 8 > > > OSDs give High IOPS value of 250 > > > +. Out of that 4 OSDs are from HP 3PAR and 4 OSDS > > > from DELL EMC. We are using > > > only 4 OSDs from HP3 par and it is working fine > > > without any latency and iops > > > issues from the beginning but the remaining 32 > > > OSDs are from DELL EMC in which 4 > > > OSDs are much better than the remaining 28 OSDs) > > > > > > https://pastebin.com/CixaQmBi > > > <https://pastebin.com/CixaQmBi> > > > > > > Please help me to identify if the issue is with > > > the DELL EMC Storage, Ceph > > > configuration parameter tuning or the Overload in > > > the cloud setup > > > > > > > > > > > > On November 1, 2023 at 9:48 PM Eugen Block < > > > eblock@nde.ag <mailto:eblock@nde.ag> > wrote: > > > > Hi, > > > > > > > > for starters please add more cluster details > > > > like 'ceph status', 'ceph > > > > versions', 'ceph osd df tree'. Increasing the > > > > to 10G was the right > > > > thing to do, you don't get far with 1G with > > > > real cluster load. How are > > > > the OSDs configured (HDD only, SSD only or HDD > > > > with rocksdb on SSD)? > > > > How is the disk utilization? > > > > > > > > Regards, > > > > Eugen > > > > > > > > Zitat von prabhav@cdac.in > > > > <mailto:prabhav@cdac.in> : > > > > > > > > > In a production setup of 36 OSDs( SAS disks) > > > > > totalling 180 TB > > > > > allocated to a single Ceph Cluster with 3 > > > > > monitors and 3 managers. > > > > > There were 830 volumes and VMs created in > > > > > Openstack with Ceph as a > > > > > backend. On Sep 21, users reported slowness > > > > > in accessing the VMs. > > > > > Analysing the logs lead us to problem with > > > > > SAS , Network congestion > > > > > and Ceph configuration( as all default values > > > > > were used). We updated > > > > > the Network from 1Gbps to 10Gbps for public > > > > > and cluster networking. > > > > > There was no change. > > > > > The ceph benchmark performance showed that 28 > > > > > OSDs out of 36 OSDs > > > > > reported very low IOPS of 30 to 50 while the > > > > > remaining showed 300+ > > > > > IOPS. > > > > > We gradually started reducing the load on the > > > > > ceph cluster and now > > > > > the volumes count is 650. Now the slow > > > > > operations has gradually > > > > > reduced but I am aware that this is not the > > > > > solution. > > > > > Ceph configuration is updated with increasing > > > > > the > > > > > osd_journal_size to 10 GB, > > > > > osd_max_backfills = 1 > > > > > osd_recovery_max_active = 1 > > > > > osd_recovery_op_priority = 1 > > > > > bluestore_cache_trim_max_skip_pinned=10000 > > > > > > > > > > After one month, now we faced another issue > > > > > with Mgr daemon stopped > > > > > in all 3 quorums and 16 OSDs went down. From > > > > > the > > > > > ceph-mon,ceph-mgr.log could not get the > > > > > reason. Please guide me as > > > > > its a production setup > > > > > _______________________________________________ > > > > > ceph-users mailing list -- ceph-users@ceph.io > > > > > <mailto:ceph-users@ceph.io> > > > > > To unsubscribe send an email to > > > > > ceph-users-leave@ceph.io > > > > > <mailto:ceph-users-leave@ceph.io> > > > > > > > > > > > > _______________________________________________ > > > > ceph-users mailing list -- ceph-users@ceph.io > > > > <mailto:ceph-users@ceph.io> > > > > To unsubscribe send an email to > > > > ceph-users-leave@ceph.io > > > > <mailto:ceph-users-leave@ceph.io> > > > Thanks & Regards, > > > Ms V A Prabha / श्रीमती प्रभा वी ए > > > Joint Director / संयुक्त निदेशक > > > Centre for Development of Advanced > > > Computing(C-DAC) / प्रगत संगणन विकास > > > केन्द्र(सी-डैक) > > > Tidel Park”, 8th Floor, “D” Block, (North &South) > > > / “टाइडल पार्क”,8वीं मंजिल, > > > “डी” ब्लॉक, (उत्तर और दक्षिण) > > > No.4, Rajiv Gandhi Salai / नं.4, राजीव गांधी सलाई > > > Taramani / तारामणि > > > Chennai / चेन्नई – 600113 > > > Ph.No.:044-22542226/27 > > > Fax No.: 044-22542294 > > > > > > ------------------------------------------------------------------------------------------------------------ > > > [ C-DAC is on Social-Media too. Kindly follow us > > > at: > > > Facebook: https://www.facebook.com/CDACINDIA > > > <https://www.facebook.com/CDACINDIA> & Twitter: @cdacindia ] > > > > > > This e-mail is for the sole use of the intended > > > recipient(s) and may > > > contain confidential and privileged information. > > > If you are not the > > > intended recipient, please contact the sender by > > > reply e-mail and destroy > > > all copies and the original message. Any > > > unauthorized review, use, > > > disclosure, dissemination, forwarding, printing > > > or copying of this email > > > is strictly prohibited and appropriate legal > > > action will be taken. > > > > > > ------------------------------------------------------------------------------------------------------------ > > > > > > _______________________________________________ > > > ceph-users mailing list -- ceph-users@ceph.io > > > <mailto:ceph-users@ceph.io> > > > To unsubscribe send an email to > > > ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io> > > > > > > > > > > > > > > > > > > > > > Thanks & Regards, > Ms V A Prabha / श्रीमती प्रभा वी ए > Joint Director / संयुक्त निदेशक > Centre for Development of Advanced Computing(C-DAC) / > प्रगत संगणन विकास केन्द्र(सी-डैक) > Tidel Park”, 8th Floor, “D” Block, (North &South) / > “टाइडल पार्क”,8वीं मंजिल, “डी” ब्लॉक, (उत्तर और दक्षिण) > No.4, Rajiv Gandhi Salai / नं.4, राजीव गांधी सलाई > Taramani / तारामणि > Chennai / चेन्नई – 600113 > Ph.No.:044-22542226/27 > Fax No.: 044-22542294 > > > ------------------------------------------------------------------------------------------------------------ > [ C-DAC is on Social-Media too. Kindly follow us at: > Facebook: https://www.facebook.com/CDACINDIA > <https://www.facebook.com/CDACINDIA> & Twitter: @cdacindia ] > > This e-mail is for the sole use of the intended > recipient(s) and may > contain confidential and privileged information. If you > are not the > intended recipient, please contact the sender by reply > e-mail and destroy > all copies and the original message. Any unauthorized > review, use, > disclosure, dissemination, forwarding, printing or > copying of this email > is strictly prohibited and appropriate legal action will > be taken. > > ------------------------------------------------------------------------------------------------------------ > > > > > > > > > > > >
Thanks & Regards, Ms V A Prabha / श्रीमती प्रभा वी ए Joint Director / संयुक्त निदेशक Centre for Development of Advanced Computing(C-DAC) / प्रगत संगणन विकास केन्द्र(सी-डैक) Tidel Park”, 8th Floor, “D” Block, (North &South) / “टाइडल पार्क”,8वीं मंजिल, “डी” ब्लॉक, (उत्तर और दक्षिण) No.4, Rajiv Gandhi Salai / नं.4, राजीव गांधी सलाई Taramani / तारामणि Chennai / चेन्नई – 600113 Ph.No.:044-22542226/27 Fax No.: 044-22542294
------------------------------------------------------------------------------------------------------------ [ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA <https://www.facebook.com/CDACINDIA> & Twitter: @cdacindia ]
This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken.
------------------------------------------------------------------------------------------------------------ > > > >
> > >
Thanks & Regards, Ms V A Prabha / श्रीमती प्रभा वी ए Joint Director / संयुक्त निदेशक Centre for Development of Advanced Computing(C-DAC) / प्रगत संगणन विकास केन्द्र(सी-डैक) Tidel Park”, 8th Floor, “D” Block, (North &South) / “टाइडल पार्क”,8वीं मंजिल, “डी” ब्लॉक, (उत्तर और दक्षिण) No.4, Rajiv Gandhi Salai / नं.4, राजीव गांधी सलाई Taramani / तारामणि Chennai / चेन्नई – 600113 Ph.No.:044-22542226/27 Fax No.: 044-22542294
------------------------------------------------------------------------------------------------------------ [ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA <https://www.facebook.com/CDACINDIA> & Twitter: @cdacindia ]
This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken.
------------------------------------------------------------------------------------------------------------
Thanks & Regards, Ms V A Prabha / श्रीमती प्रभा वी ए Joint Director / संयुक्त निदेशक Centre for Development of Advanced Computing(C-DAC) / प्रगत संगणन विकास केन्द्र(सी-डैक) Tidel Park”, 8th Floor, “D” Block, (North &South) / “टाइडल पार्क”,8वीं मंजिल, “डी” ब्लॉक, (उत्तर और दक्षिण) No.4, Rajiv Gandhi Salai / नं.4, राजीव गांधी सलाई Taramani / तारामणि Chennai / चेन्नई – 600113 Ph.No.:044-22542226/27 Fax No.: 044-22542294
Thanks & Regards, Ms V A Prabha / श्रीमती प्रभा वी ए Joint Director / संयुक्त निदेशक Centre for Development of Advanced Computing(C-DAC) / प्रगत संगणन विकास केन्द्र(सी-डैक) Tidel Park”, 8th Floor, “D” Block, (North &South) / “टाइडल पार्क”,8वीं मंजिल, “डी” ब्लॉक, (उत्तर और दक्षिण) No.4, Rajiv Gandhi Salai / नं.4, राजीव गांधी सलाई Taramani / तारामणि Chennai / चेन्नई – 600113 Ph.No.:044-22542226/27 Fax No.: 044-22542294 ------------------------------------------------------------------------------------------------------------ [ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ] This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken. ------------------------------------------------------------------------------------------------------------
Just a continuation of this mail, Could you help me out to understand the ceph df output. PFA the screenshot with this mail.
No idea what PFA means, but attachments usually don’t make it through on mailing lists. Paste text instead.
1. Raw storage is 180 TB
The sum of OSD total capacities.
2. Stored Value is 37 TB
Clients wrote 37 TB of data.
3. Used Value is 112 TB
With default RF=3 replication, Ceph stores 3 copies of data for redundancy. 3x37 ~= 112 TB of raw storage used.
4. Available Value is 67 TB
5. Pool Max Available Value is 16 TB
We need the actual output.
Hi, You have 67 TB of raw space available. With a replication factor of 3, which is what you seem to be using, that is ~22 TB usable space under ideal conditions. MAX AVAIL column shows the available space, taking into account the raw space, the replication factor and the CRUSH map, before the first OSD becomes full. In other words, because of the way the data is distributed across the OSDs in your cluster, you won't be able to utilize the whole 22 TB because one or more OSDs will get full if you write another 16 TB of data to your pools. /Z On Sun, 28 Jan 2024 at 18:36, V A Prabha <prabhav@cdac.in> wrote:
Hi Just a continuation of this mail, Could you help me out to understand the ceph df output. PFA the screenshot with this mail.
1. Raw storage is 180 TB
2. Stored Value is 37 TB
3. Used Value is 112 TB
4. Available Value is 67 TB
5. Pool Max Available Value is 16 TB Though the Available Value is still 67 TB how can it be utilized for the pools? On November 6, 2023 at 4:18 PM V A Prabha <prabhav@cdac.in> wrote:
Please clarify my query. I had 700+ volumes (220 applications) running in 36 OSDs when it reported the slow operations. Due to emergency, we migrated 200+ VMs to another virtualization environment. So we have shutdown all the related VMs in our Openstack production setup running with Ceph. We have not deleted the 200+ volumes from Ceph as waiting for the concurrence from the departments. My query is that even when the applications are down, does the volume at the backend make our cluster busy when there is no active transactions? Is there any parameter in the ceph osd log and ceph mon log that gives me the clue for the cluster business? Is there any zombie or unwanted process that make the ceph cluster busy or the IOPS budget of the disk that makes the cluster busy?
On November 4, 2023 at 4:29 PM Zakhar Kirpichenko <zakhar@gmail.com> wrote:
You have an IOPS budget, i.e. how much I/O your spinners can deliver. Space utilization doesn't affect it much.
You can try disabling write (not read!) cache on your HDDs with sdparm (for example, sdparm -c WCE /dev/bla); in my experience this allows HDDs to deliver 50-100% more write IOPS. If there is lots of free RAM on the OSD nodes, you can play with osd_memory_target and bluestore_cache_size_hdd OSD options; be careful though: depending on you workload, the performance impact may be insignificant, but your OSDs may run out of memory.
/z
On Sat, 4 Nov 2023 at 12:04, V A Prabha < prabhav@cdac.in> wrote:
Now in this situation how can stabilize my production setup as you have mentioned the cluster is very busy. Is there any configuration parameter tuning will help or the only option is to reduce the applications running on the cluster. Though if I have free available storage of 1.6 TB free in each of my OSD, that will not help in my IOPS issue right? Please guide me
On November 2, 2023 at 12:47 PM Zakhar Kirpichenko < zakhar@gmail.com> wrote:
1. The calculated IOPS is for the rw operation right ?
Total drive IOPS, read or write. Depending on the exact drive models, it may be lower or higher than 200. I took the average for a smaller sized 7.2k rpm SAS drive. Modern drives usually deliver lower read IOPS and higher write IOPS.
2. Cluster is very busy? Is there any misconfiguration or missing tuning paramater that makes the cluster busy?
You have almost 3k IOPS and your OSDs report slow ops. I'd say the cluster is busy, as in loaded with I/O, perhaps more I/O than it can handle well.
3. Nodes are not balanced? you mean to say that the count of OSDs in each server differs. But we have enabled autoscale and optimal distribution so that you can see from the output of ceph osd df tree that is count of pgs(45/OSD) and use% (65 to 67%). Is that not significant?
Yes, the OSD count differs. This means that the CPU, memory usage, network load and latency differ per node and may cause performance variations, depending on your workload.
/Z
On Thu, 2 Nov 2023 at 08:18, V A Prabha < prabhav@cdac.in> wrote:
Thanks for your prompt reply .. But the query is 1.The calculated IOPS is for the rw operation right ? 2. Cluster is very busy? Is there any misconfiguration or missing tuning paramater that makes the cluster busy? 3. Nodes are not balanced? you mean to say that the count of OSDs in each server differs. But we have enabled autoscale and optimal distribution so that you can see from the output of ceph osd df tree that is count of pgs(45/OSD) and use% (65 to 67%). Is that not significant? Correct me if my queries are irrelevant
On November 2, 2023 at 11:36 AM Zakhar Kirpichenko < zakhar@gmail.com> wrote:
Sure, it's 36 OSDs at 200 IOPS each (tops, likely lower), I assume size=3 replication so 1/3 of the total performance, and some 30%-ish OSD overhead.
(36 x 200) * 1/3 * 0.7 = 1680. That's how many IOPS you can realistically expect from your cluster. You get more than that, but the cluster is very busy and OSDs aren't coping.
Also your nodes are not balanced.
/Z
On Thu, 2 Nov 2023 at 07:33, V A Prabha < prabhav@cdac.in> wrote:
Can you please elaborate your identifications and the statement .
On November 2, 2023 at 9:40 AM Zakhar Kirpichenko < zakhar@gmail.com> wrote:
I'm afraid you're simply hitting the I/O limits of your disks.
/Z
On Thu, 2 Nov 2023 at 03:40, V A Prabha < prabhav@cdac.in> wrote:
Hi Eugen Please find the details below
root@meghdootctr1:/var/log/ceph# ceph -s cluster: id: c59da971-57d1-43bd-b2b7-865d392412a5 health: HEALTH_WARN nodeep-scrub flag(s) set 544 pgs not deep-scrubbed in time
services: mon: 3 daemons, quorum meghdootctr1,meghdootctr2,meghdootctr3 (age 5d) mgr: meghdootctr1(active, since 5d), standbys: meghdootctr2, meghdootctr3 mds: 3 up:standby osd: 36 osds: 36 up (since 34h), 36 in (since 34h) flags nodeep-scrub
data: pools: 2 pools, 544 pgs objects: 10.14M objects, 39 TiB usage: 116 TiB used, 63 TiB / 179 TiB avail pgs: 544 active+clean
io: client: 24 MiB/s rd, 16 MiB/s wr, 2.02k op/s rd, 907 op/s wr
Ceph Versions:
root@meghdootctr1:/var/log/ceph# ceph --version ceph version 14.2.16 (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus (stable)
Ceph df -h https://pastebin.com/1ffucyJg
Ceph OSD performance dump https://pastebin.com/1R6YQksE
Ceph tell osd.XX bench (Out of 36 osds only 8 OSDs give High IOPS value of 250 +. Out of that 4 OSDs are from HP 3PAR and 4 OSDS from DELL EMC. We are using only 4 OSDs from HP3 par and it is working fine without any latency and iops issues from the beginning but the remaining 32 OSDs are from DELL EMC in which 4 OSDs are much better than the remaining 28 OSDs)
Please help me to identify if the issue is with the DELL EMC Storage, Ceph configuration parameter tuning or the Overload in the cloud setup
On November 1, 2023 at 9:48 PM Eugen Block < eblock@nde.ag> wrote:
Hi,
for starters please add more cluster details like 'ceph status', 'ceph versions', 'ceph osd df tree'. Increasing the to 10G was the right thing to do, you don't get far with 1G with real cluster load. How are the OSDs configured (HDD only, SSD only or HDD with rocksdb on SSD)? How is the disk utilization?
Regards, Eugen
Zitat von prabhav@cdac.in:
In a production setup of 36 OSDs( SAS disks) totalling 180 TB allocated to a single Ceph Cluster with 3 monitors and 3 managers. There were 830 volumes and VMs created in Openstack with Ceph as a backend. On Sep 21, users reported slowness in accessing the VMs. Analysing the logs lead us to problem with SAS , Network congestion and Ceph configuration( as all default values were used). We updated the Network from 1Gbps to 10Gbps for public and cluster networking. There was no change. The ceph benchmark performance showed that 28 OSDs out of 36 OSDs reported very low IOPS of 30 to 50 while the remaining showed 300+ IOPS. We gradually started reducing the load on the ceph cluster and now the volumes count is 650. Now the slow operations has gradually reduced but I am aware that this is not the solution. Ceph configuration is updated with increasing the osd_journal_size to 10 GB, osd_max_backfills = 1 osd_recovery_max_active = 1 osd_recovery_op_priority = 1 bluestore_cache_trim_max_skip_pinned=10000
After one month, now we faced another issue with Mgr daemon stopped in all 3 quorums and 16 OSDs went down. From the ceph-mon,ceph-mgr.log could not get the reason. Please guide me as its a production setup _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io Thanks & Regards, Ms V A Prabha / श्रीमती प्रभा वी ए Joint Director / संयुक्त निदेशक Centre for Development of Advanced Computing(C-DAC) / प्रगत संगणन विकास केन्द्र(सी-डैक) Tidel Park”, 8th Floor, “D” Block, (North &South) / “टाइडल पार्क”,8वीं मंजिल, “डी” ब्लॉक, (उत्तर और दक्षिण) No.4, Rajiv Gandhi Salai / नं.4, राजीव गांधी सलाई Taramani / तारामणि Chennai / चेन्नई – 600113 Ph.No.:044-22542226/27 Fax No.: 044-22542294
[ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ]
This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken. ------------------------------------------------------------------------------------------------------------
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks & Regards, Ms V A Prabha / श्रीमती प्रभा वी ए Joint Director / संयुक्त निदेशक Centre for Development of Advanced Computing(C-DAC) / प्रगत संगणन विकास केन्द्र(सी-डैक) Tidel Park”, 8th Floor, “D” Block, (North &South) / “टाइडल पार्क”,8वीं मंजिल, “डी” ब्लॉक, (उत्तर और दक्षिण) No.4, Rajiv Gandhi Salai / नं.4, राजीव गांधी सलाई Taramani / तारामणि Chennai / चेन्नई – 600113 Ph.No.:044-22542226/27 Fax No.: 044-22542294
------------------------------------------------------------------------------------------------------------
[ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ]
This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken. ------------------------------------------------------------------------------------------------------------
Thanks & Regards, Ms V A Prabha / श्रीमती प्रभा वी ए Joint Director / संयुक्त निदेशक Centre for Development of Advanced Computing(C-DAC) / प्रगत संगणन विकास केन्द्र(सी-डैक) Tidel Park”, 8th Floor, “D” Block, (North &South) / “टाइडल पार्क”,8वीं मंजिल, “डी” ब्लॉक, (उत्तर और दक्षिण) No.4, Rajiv Gandhi Salai / नं.4, राजीव गांधी सलाई Taramani / तारामणि Chennai / चेन्नई – 600113 Ph.No.:044-22542226/27 Fax No.: 044-22542294
------------------------------------------------------------------------------------------------------------
[ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ]
This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken. ------------------------------------------------------------------------------------------------------------
Thanks & Regards, Ms V A Prabha / श्रीमती प्रभा वी ए Joint Director / संयुक्त निदेशक Centre for Development of Advanced Computing(C-DAC) / प्रगत संगणन विकास केन्द्र(सी-डैक) Tidel Park”, 8th Floor, “D” Block, (North &South) / “टाइडल पार्क”,8वीं मंजिल, “डी” ब्लॉक, (उत्तर और दक्षिण) No.4, Rajiv Gandhi Salai / नं.4, राजीव गांधी सलाई Taramani / तारामणि Chennai / चेन्नई – 600113 Ph.No.:044-22542226/27 Fax No.: 044-22542294
------------------------------------------------------------------------------------------------------------
[ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ]
This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken. ------------------------------------------------------------------------------------------------------------
Thanks & Regards, Ms V A Prabha / श्रीमती प्रभा वी ए Joint Director / संयुक्त निदेशक Centre for Development of Advanced Computing(C-DAC) / प्रगत संगणन विकास केन्द्र(सी-डैक) Tidel Park”, 8th Floor, “D” Block, (North &South) / “टाइडल पार्क”,8वीं मंजिल, “डी” ब्लॉक, (उत्तर और दक्षिण) No.4, Rajiv Gandhi Salai / नं.4, राजीव गांधी सलाई Taramani / तारामणि Chennai / चेन्नई – 600113 Ph.No.:044-22542226/27 Fax No.: 044-22542294
Thanks & Regards, Ms V A Prabha / श्रीमती प्रभा वी ए Joint Director / संयुक्त निदेशक Centre for Development of Advanced Computing(C-DAC) / प्रगत संगणन विकास केन्द्र(सी-डैक) Tidel Park”, 8th Floor, “D” Block, (North &South) / “टाइडल पार्क”,8वीं मंजिल, “डी” ब्लॉक, (उत्तर और दक्षिण) No.4, Rajiv Gandhi Salai / नं.4, राजीव गांधी सलाई Taramani / तारामणि Chennai / चेन्नई – 600113 Ph.No.:044-22542226/27 Fax No.: 044-22542294
------------------------------------------------------------------------------------------------------------
[ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ]
This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken. ------------------------------------------------------------------------------------------------------------
participants (4)
-
Anthony D'Atri
-
Janne Johansson
-
V A Prabha
-
Zakhar Kirpichenko