radosgw-admin sync status takes ages to print output
Hello, I have a 3 DC octopus Multisite setup with bucket sync policy applied. I have 2 buckets where I’ve set the shard 24.000 and the other is 9.000 because they want to use 1 bucket but with a huge amount of objects (2.400.000.000 and 900.000.000) and in case of multisite we need to preshard the buckets as it is in the documentation. Do I need to fine tune something on the syncing to make this query faster? This is the output after 5-10 minutes query time not sure is it healthy or good or not to be honest, haven’t really find any good explanation about the output in the ceph documentation. From the master zone I can’r reallt even query because timed out, but in secondary zone can see this: radosgw-admin sync status realm 5fd28798-9195-44ac-b48d-ef3e95caee48 (realm) zonegroup 31a5ea05-c87a-436d-9ca0-ccfcbad481e3 (data) zone 61c9d940-fde4-4bed-9389-edc8d7741817 (sin) metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master data sync source: 9213182a-14ba-48ad-bde9-289a1c0c0de8 (hkg) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 128 shards behind shards: [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127] oldest incremental change not applied: 2021-01-14T12:01:00.131104+0700 [11] source: f20ddd64-924b-4f78-8d2d-dd6c65f98ba9 (ash) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 128 shards behind shards: [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127] oldest incremental change not applied: 2021-01-14T12:05:26.879014+0700 [98] Hope I can find some expert in the multisite area 😊 Thank you in advance. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.
UPDATE: Finally got back the master sync command output: radosgw-admin sync status realm 5fd28798-9195-44ac-b48d-ef3e95caee48 realm) zonegroup 31a5ea05-c87a-436d-9ca0-ccfcbad481e3 (data) zone 9213182a-14ba-48ad-bde9-289a1c0c0de8 (hkg) metadata sync no sync (zone is master) data sync source: 61c9d940-fde4-4bed-9389-edc8d7741817 (sin) syncing full sync: 0/128 shards incremental sync: 128/128 shards source: f20ddd64-924b-4f78-8d2d-dd6c65f98ba9 (ash) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 128 shards behind shards: [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127] oldest incremental change not applied: 2021-01-14T13:03:17.807529+0700 [20] 45 shards are recovering recovering shards: [5,14,23,25,26,34,36,37,38,45,46,47,49,50,51,52,54,55,57,58,60,61,62,67,68,69,71,77,79,80,88,89,90,95,97,100,108,110,111,117,118,120,121,125,126] Sorry for the 2 email. -----Original Message----- From: Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> Sent: Thursday, January 14, 2021 12:57 PM To: ceph-users@ceph.io Subject: [ceph-users] radosgw-admin sync status takes ages to print output Email received from outside the company. If in doubt don't click links nor open attachments! ________________________________ Hello, I have a 3 DC octopus Multisite setup with bucket sync policy applied. I have 2 buckets where I’ve set the shard 24.000 and the other is 9.000 because they want to use 1 bucket but with a huge amount of objects (2.400.000.000 and 900.000.000) and in case of multisite we need to preshard the buckets as it is in the documentation. Do I need to fine tune something on the syncing to make this query faster? This is the output after 5-10 minutes query time not sure is it healthy or good or not to be honest, haven’t really find any good explanation about the output in the ceph documentation. From the master zone I can’r reallt even query because timed out, but in secondary zone can see this: radosgw-admin sync status realm 5fd28798-9195-44ac-b48d-ef3e95caee48 (realm) zonegroup 31a5ea05-c87a-436d-9ca0-ccfcbad481e3 (data) zone 61c9d940-fde4-4bed-9389-edc8d7741817 (sin) metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master data sync source: 9213182a-14ba-48ad-bde9-289a1c0c0de8 (hkg) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 128 shards behind shards: [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127] oldest incremental change not applied: 2021-01-14T12:01:00.131104+0700 [11] source: f20ddd64-924b-4f78-8d2d-dd6c65f98ba9 (ash) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 128 shards behind shards: [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127] oldest incremental change not applied: 2021-01-14T12:05:26.879014+0700 [98] Hope I can find some expert in the multisite area 😊 Thank you in advance. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Istvan; What version of Ceph are you running? Another email chain indicates you're running on CentOS 8, which suggests Octopus (15). We're running multisite replicated radosgw on Nautilus. I don't see the long running time that you are suggesting, though we only have ~35k objects. I generally don't worry about sync unless the "oldest incremental change not applied" is several minutes or more in the past. Our work day has just started, so use isn't very high yet. This afternoon, when anticipated use peaks, I'll set a watch to see how behind the clusters get. According to the command output, you have 64 shards in the metadata, and 128 shards in the data. That seems low, as that's the same number of shards we're running, with our significantly lower object count. Thank you, Dominic L. Hilsbos, MBA Director – Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com -----Original Message----- From: Szabo, Istvan (Agoda) [mailto:Istvan.Szabo@agoda.com] Sent: Wednesday, January 13, 2021 11:18 PM To: ceph-users@ceph.io Subject: [ceph-users] Re: radosgw-admin sync status takes ages to print output UPDATE: Finally got back the master sync command output: radosgw-admin sync status realm 5fd28798-9195-44ac-b48d-ef3e95caee48 realm) zonegroup 31a5ea05-c87a-436d-9ca0-ccfcbad481e3 (data) zone 9213182a-14ba-48ad-bde9-289a1c0c0de8 (hkg) metadata sync no sync (zone is master) data sync source: 61c9d940-fde4-4bed-9389-edc8d7741817 (sin) syncing full sync: 0/128 shards incremental sync: 128/128 shards source: f20ddd64-924b-4f78-8d2d-dd6c65f98ba9 (ash) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 128 shards behind shards: [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127] oldest incremental change not applied: 2021-01-14T13:03:17.807529+0700 [20] 45 shards are recovering recovering shards: [5,14,23,25,26,34,36,37,38,45,46,47,49,50,51,52,54,55,57,58,60,61,62,67,68,69,71,77,79,80,88,89,90,95,97,100,108,110,111,117,118,120,121,125,126] Sorry for the 2 email. -----Original Message----- From: Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> Sent: Thursday, January 14, 2021 12:57 PM To: ceph-users@ceph.io Subject: [ceph-users] radosgw-admin sync status takes ages to print output Email received from outside the company. If in doubt don't click links nor open attachments! ________________________________ Hello, I have a 3 DC octopus Multisite setup with bucket sync policy applied. I have 2 buckets where I’ve set the shard 24.000 and the other is 9.000 because they want to use 1 bucket but with a huge amount of objects (2.400.000.000 and 900.000.000) and in case of multisite we need to preshard the buckets as it is in the documentation. Do I need to fine tune something on the syncing to make this query faster? This is the output after 5-10 minutes query time not sure is it healthy or good or not to be honest, haven’t really find any good explanation about the output in the ceph documentation. From the master zone I can’r reallt even query because timed out, but in secondary zone can see this: radosgw-admin sync status realm 5fd28798-9195-44ac-b48d-ef3e95caee48 (realm) zonegroup 31a5ea05-c87a-436d-9ca0-ccfcbad481e3 (data) zone 61c9d940-fde4-4bed-9389-edc8d7741817 (sin) metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master data sync source: 9213182a-14ba-48ad-bde9-289a1c0c0de8 (hkg) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 128 shards behind shards: [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127] oldest incremental change not applied: 2021-01-14T12:01:00.131104+0700 [11] source: f20ddd64-924b-4f78-8d2d-dd6c65f98ba9 (ash) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 128 shards behind shards: [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127] oldest incremental change not applied: 2021-01-14T12:05:26.879014+0700 [98] Hope I can find some expert in the multisite area 😊 Thank you in advance. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello, I see, do you know how much I should increase and how? Haven’t found much documentation about it :(
On 2021. Jan 14., at 22:36, DHilsbos@performair.com wrote:
Email received from outside the company. If in doubt don't click links nor open attachments! ________________________________
Istvan;
What version of Ceph are you running? Another email chain indicates you're running on CentOS 8, which suggests Octopus (15).
We're running multisite replicated radosgw on Nautilus. I don't see the long running time that you are suggesting, though we only have ~35k objects.
I generally don't worry about sync unless the "oldest incremental change not applied" is several minutes or more in the past. Our work day has just started, so use isn't very high yet. This afternoon, when anticipated use peaks, I'll set a watch to see how behind the clusters get.
According to the command output, you have 64 shards in the metadata, and 128 shards in the data. That seems low, as that's the same number of shards we're running, with our significantly lower object count.
Thank you,
Dominic L. Hilsbos, MBA Director – Information Technology Perform Air International Inc. DHilsbos@PerformAir.com www.PerformAir.com
-----Original Message----- From: Szabo, Istvan (Agoda) [mailto:Istvan.Szabo@agoda.com] Sent: Wednesday, January 13, 2021 11:18 PM To: ceph-users@ceph.io Subject: [ceph-users] Re: radosgw-admin sync status takes ages to print output
UPDATE: Finally got back the master sync command output:
radosgw-admin sync status realm 5fd28798-9195-44ac-b48d-ef3e95caee48 realm) zonegroup 31a5ea05-c87a-436d-9ca0-ccfcbad481e3 (data) zone 9213182a-14ba-48ad-bde9-289a1c0c0de8 (hkg) metadata sync no sync (zone is master) data sync source: 61c9d940-fde4-4bed-9389-edc8d7741817 (sin) syncing full sync: 0/128 shards incremental sync: 128/128 shards source: f20ddd64-924b-4f78-8d2d-dd6c65f98ba9 (ash) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 128 shards behind shards: [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127] oldest incremental change not applied: 2021-01-14T13:03:17.807529+0700 [20] 45 shards are recovering recovering shards: [5,14,23,25,26,34,36,37,38,45,46,47,49,50,51,52,54,55,57,58,60,61,62,67,68,69,71,77,79,80,88,89,90,95,97,100,108,110,111,117,118,120,121,125,126]
Sorry for the 2 email.
-----Original Message----- From: Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> Sent: Thursday, January 14, 2021 12:57 PM To: ceph-users@ceph.io Subject: [ceph-users] radosgw-admin sync status takes ages to print output
Email received from outside the company. If in doubt don't click links nor open attachments! ________________________________
Hello,
I have a 3 DC octopus Multisite setup with bucket sync policy applied.
I have 2 buckets where I’ve set the shard 24.000 and the other is 9.000 because they want to use 1 bucket but with a huge amount of objects (2.400.000.000 and 900.000.000) and in case of multisite we need to preshard the buckets as it is in the documentation.
Do I need to fine tune something on the syncing to make this query faster? This is the output after 5-10 minutes query time not sure is it healthy or good or not to be honest, haven’t really find any good explanation about the output in the ceph documentation.
From the master zone I can’r reallt even query because timed out, but in secondary zone can see this:
radosgw-admin sync status realm 5fd28798-9195-44ac-b48d-ef3e95caee48 (realm) zonegroup 31a5ea05-c87a-436d-9ca0-ccfcbad481e3 (data) zone 61c9d940-fde4-4bed-9389-edc8d7741817 (sin) metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master data sync source: 9213182a-14ba-48ad-bde9-289a1c0c0de8 (hkg) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 128 shards behind shards: [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127] oldest incremental change not applied: 2021-01-14T12:01:00.131104+0700 [11] source: f20ddd64-924b-4f78-8d2d-dd6c65f98ba9 (ash) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is behind on 128 shards behind shards: [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127] oldest incremental change not applied: 2021-01-14T12:05:26.879014+0700 [98]
Hope I can find some expert in the multisite area 😊
Thank you in advance.
________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (2)
-
DHilsbos＠performair.com
-
Szabo, Istvan (Agoda)