radosgw sync using more capacity on secondary than on master zone
Dear Ceph contributors While our (new) rgw secondary zone is doing the initial data sync from our master zone, we noticed that the reported capacity usage was getting higher than on primary zone: Master Zone: ceph version 14.2.5 zone parameters: "log_meta": "false", "log_data": "true", "bucket_index_max_shards": 0, "read_only": "false", "tier_type": "", "sync_from_all": "false", "sync_from": [], "redirect_zone": "" bucket stats: => "size_actual GiB": 269868.9823989868, "size_utilized GiB": 17180102071.868008, <= (cf. other issue below) "num_objects": 100218788, "Compression Rate": 63660.899148714125 pool stats: => "stored": 403834132234240, "objects": 191530416, "kb_used": 692891724288, "bytes_used": 709521125670912, "percent_used": 0.7570806741714478, "max_avail": 136595529269248 erasure-code-profile: crush-device-class=nvme crush-failure-domain=host crush-root=default jerasure-per-chunk-alignment=false k=8 m=2 plugin=jerasure technique=reed_sol_van w=8 osd parameters: bluefs_alloc_size 1048576 default bluefs_shared_alloc_size 65536 default bluestore_extent_map_inline_shard_prealloc_size 256 default bluestore_max_alloc_size 0 default bluestore_min_alloc_size 0 default bluestore_min_alloc_size_hdd 65536 default bluestore_min_alloc_size_ssd 16384 default rgw compression: no Secondary Zone: ceph version 14.2.12 zone parameters: "log_meta": "false", "log_data": "true", "bucket_index_max_shards": 11, "read_only": "true", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" bucket stats: => "size_actual GiB": 65282.37313461304, "size_utilized GiB": 60779.72828538809, "num_objects": 23074921, "Compression Rate": 0.9310281683550253 pool stats: => "stored": 407816305115136, "objects": 118637638, "kb_used": 497822635396, "bytes_used": 509770378645504, "percent_used": 0.7275146245956421, "max_avail": 152744706965504 erasure-code-profile: crush-device-class= crush-failure-domain=host crush-root=default jerasure-per-chunk-alignment=false k=3 m=2 plugin=jerasure technique=reed_sol_van w=8 osd parameters: EC k=8 m=2 bluefs_alloc_size 1048576 default bluefs_shared_alloc_size 65536 default bluestore_extent_map_inline_shard_prealloc_size 256 default bluestore_max_alloc_size 0 default bluestore_min_alloc_size 0 default bluestore_min_alloc_size_hdd 65536 default bluestore_min_alloc_size_ssd 4096 default rgw compression: yes As you see, the secondary zone is using 408 TB vs 404 TB on master zone. Summing size_actual for each buckets gives only 65 TB vs 270 TB on master zone. Any idea about what could cause such a difference? Is it a known issue? The are some known issue with Space Overhead with EC Pools for alloc_size > 4 KiB, cf. : https://www.mail-archive.com/ceph-users@ceph.io/msg06191.html https://lists.ceph.io/hyperkitty/list/dev@ceph.io/thread/OHPO43J54TPBEUISYCK... https://www.spinics.net/lists/ceph-users/msg59587.html But our secondary zone is on nvme osd's with bluestore_min_alloc_size_ssd=4096, so that should be fine. I will also investigate further with "radosgw-admin bucket radoslist" and rgw-orphan-list. Thank you in advance for any help. And Happy New Year very soon ;-) Best Regards Francois PS: The value on master zone "size_utilized GiB": 17180102071.868008 is wrong. This is due to a bucket with wrong stats: { "bucket": "XXX", "tenant": "", "zonegroup": "d29ea82c-4f77-40af-952b-2ab0705ad268", "placement_rule": "default-placement", "explicit_placement": { "data_pool": "", "data_extra_pool": "", "index_pool": "" }, "id": "cb1594b3-a782-49d0-a19f-68cd48870a63.3119460.1", "marker": "cb1594b3-a782-49d0-a19f-68cd48870a63.63841.1481", "index_type": "Normal", "owner": "e13d054f6e9c4eea881f687923d7d380", "ver": "0#373075,1#286613,2#290913,3#341069,4#360862,5#341416,6#279526,7#352172,8#255944,9#314797,10#317650,11#305698,12#289557,13#345344,14#345273,15#294708,16#241001,17#298577,18#274866,19#293952,20#307635,21#334606,22#265355,23#302567,24#277505,25#307278,26#266963,27#297452,28#332274,29#319133,30#361027,31#314294,32#282887,33#324849,34#278560,35#307506,36#287269,37#344789,38#345389,39#323814,40#386483,41#280319,42#358072,43#336651,44#339176,45#248079,46#356784,47#381496,48#295152,49#251661,50#318661,51#330530,52#263564,53#332005,54#332937,55#320163,56#300485,57#296138,58#343271,59#359351,60#295711,61#275751,62#332264,63#351532", "master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0,11#0,12#0,13#0,14#0,15#0,16#0,17#0,18#0,19#0,20#0,21#0,22#0,23#0,24#0,25#0,26#0,27#0,28#0,29#0,30#0,31#0,32#0,33#0,34#0,35#0,36#0,37#0,38#0,39#0,40#0,41#0,42#0,43#0,44#0,45#0,46#0,47#0,48#0,49#0,50#0,51#0,52#0,53#0,54#0,55#0,56#0,57#0,58#0,59#0,60#0,61#0,62#0,63#0", "mtime": "2020-06-03 12:13:01.207610Z", "max_marker": "0#,1#,2#,3#00000341068.137793799.5,4#00000360861.156871075.5,5#00000341415.97799619.5,6#00000279525.155275260.5,7#,8#,9#00000314796.95564320.5,10#,11#,12#00000289556.137784091.5,13#,14#,15#,16#00000241000.126242121.5,17#,18#00000274865.124884405.5,19#,20#00000307634.137793798.5,21#00000334605.93836734.5,22#,23#00000302566.125226103.5,24#,25#,26#,27#00000297451.125229375.5,28#,29#00000319132.155275278.5,30#00000361026.98341455.5,31#,32#,33#00000324848.126242117.5,34#00000278559.124884417.5,35#,36#,37#00000344788.125945123.5,38#00000345388.97796715.5,39#,40#00000386482.98341457.5,41#,42#00000358071.124884415.5,43#,44#00000339175.135084366.5,45#00000248078.155263912.5,46#00000356783.98341461.5,47#00000381495.94538350.5,48#00000295151.138701826.5,49#00000251660.137793803.5,50#00000318660.93848186.5,51#,52#,53#00000332004.126242119.5,54#00000332936.138701824.5,55#,56#00000300484.156871073.5,57#,58#00000343270.98341459.5,59#00000359350.94257302.5,60#,61#,62#,63#00000351531.135084368.5", "usage": { "rgw.none": { "size": 0, "size_actual": 0, "size_utilized": 0, "size_kb": 0, "size_kb_actual": 0, "size_kb_utilized": 0, "num_objects": 18446744073709551604 }, "rgw.main": { "size": 22615692324233, "size_actual": 22617004142592, "size_utilized": 18446736471689615156, "size_kb": 22085637036, "size_kb_actual": 22086918108, "size_kb_utilized": 18014391085634390, "num_objects": 521927 }, "rgw.multimeta": { "size": 0, "size_actual": 0, "size_utilized": 0, "size_kb": 0, "size_kb_actual": 0, "size_kb_utilized": 0, "num_objects": 977 } }, "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 } } This is another issue, probably not related to our sync issue. -- EveryWare AG François Scheurer Senior Systems Engineer Zurlindenstrasse 52a CH-8003 Zürich tel: +41 44 466 60 00 fax: +41 44 466 60 10 mail: francois.scheurer@everyware.ch web: http://www.everyware.ch
Hello , It could be related to the « erasure-code-profile » which is defined with different k+m on master and secondary . For the size of buckets I guess its probalbly due to the parameter : compression enabled on secondary radosgw « rgw compression : yes » . Regards De : Scheurer François <francois.scheurer@everyware.ch> Envoyé : jeudi 31 décembre 2020 15:45 À : ceph-users@ceph.io Objet : [ceph-users] radosgw sync using more capacity on secondary than on master zone Dear Ceph contributors While our (new) rgw secondary zone is doing the initial data sync from our master zone, we noticed that the reported capacity usage was getting higher than on primary zone: Master Zone: ceph version 14.2.5 zone parameters: "log_meta": "false", "log_data": "true", "bucket_index_max_shards": 0, "read_only": "false", "tier_type": "", "sync_from_all": "false", "sync_from": [], "redirect_zone": "" bucket stats: => "size_actual GiB": 269868.9823989868, "size_utilized GiB": 17180102071.868008, <= (cf. other issue below) "num_objects": 100218788, "Compression Rate": 63660.899148714125 pool stats: => "stored": 403834132234240, "objects": 191530416, "kb_used": 692891724288, "bytes_used": 709521125670912, "percent_used": 0.7570806741714478, "max_avail": 136595529269248 erasure-code-profile: crush-device-class=nvme crush-failure-domain=host crush-root=default jerasure-per-chunk-alignment=false k=8 m=2 plugin=jerasure technique=reed_sol_van w=8 osd parameters: bluefs_alloc_size 1048576 default bluefs_shared_alloc_size 65536 default bluestore_extent_map_inline_shard_prealloc_size 256 default bluestore_max_alloc_size 0 default bluestore_min_alloc_size 0 default bluestore_min_alloc_size_hdd 65536 default bluestore_min_alloc_size_ssd 16384 default rgw compression: no Secondary Zone: ceph version 14.2.12 zone parameters: "log_meta": "false", "log_data": "true", "bucket_index_max_shards": 11, "read_only": "true", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" bucket stats: => "size_actual GiB": 65282.37313461304, "size_utilized GiB": 60779.72828538809, "num_objects": 23074921, "Compression Rate": 0.9310281683550253 pool stats: => "stored": 407816305115136, "objects": 118637638, "kb_used": 497822635396, "bytes_used": 509770378645504, "percent_used": 0.7275146245956421, "max_avail": 152744706965504 erasure-code-profile: crush-device-class= crush-failure-domain=host crush-root=default jerasure-per-chunk-alignment=false k=3 m=2 plugin=jerasure technique=reed_sol_van w=8 osd parameters: EC k=8 m=2 bluefs_alloc_size 1048576 default bluefs_shared_alloc_size 65536 default bluestore_extent_map_inline_shard_prealloc_size 256 default bluestore_max_alloc_size 0 default bluestore_min_alloc_size 0 default bluestore_min_alloc_size_hdd 65536 default bluestore_min_alloc_size_ssd 4096 default rgw compression: yes As you see, the secondary zone is using 408 TB vs 404 TB on master zone. Summing size_actual for each buckets gives only 65 TB vs 270 TB on master zone. Any idea about what could cause such a difference? Is it a known issue? The are some known issue with Space Overhead with EC Pools for alloc_size > 4 KiB, cf. : https://www.mail-archive.com/ceph-users@ceph.io/msg06191.html https://lists.ceph.io/hyperkitty/list/dev@ceph.io/thread/OHPO43J54TPBEUISYCK... https://www.spinics.net/lists/ceph-users/msg59587.html But our secondary zone is on nvme osd's with bluestore_min_alloc_size_ssd=4096, so that should be fine. I will also investigate further with "radosgw-admin bucket radoslist" and rgw-orphan-list. Thank you in advance for any help. And Happy New Year very soon ;-) Best Regards Francois PS: The value on master zone "size_utilized GiB": 17180102071.868008 is wrong. This is due to a bucket with wrong stats: { "bucket": "XXX", "tenant": "", "zonegroup": "d29ea82c-4f77-40af-952b-2ab0705ad268", "placement_rule": "default-placement", "explicit_placement": { "data_pool": "", "data_extra_pool": "", "index_pool": "" }, "id": "cb1594b3-a782-49d0-a19f-68cd48870a63.3119460.1", "marker": "cb1594b3-a782-49d0-a19f-68cd48870a63.63841.1481", "index_type": "Normal", "owner": "e13d054f6e9c4eea881f687923d7d380", "ver": "0#373075,1#286613,2#290913,3#341069,4#360862,5#341416,6#279526,7#352172,8#255944,9#314797,10#317650,11#305698,12#289557,13#345344,14#345273,15#294708,16#241001,17#298577,18#274866,19#293952,20#307635,21#334606,22#265355,23#302567,24#277505,25#307278,26#266963,27#297452,28#332274,29#319133,30#361027,31#314294,32#282887,33#324849,34#278560,35#307506,36#287269,37#344789,38#345389,39#323814,40#386483,41#280319,42#358072,43#336651,44#339176,45#248079,46#356784,47#381496,48#295152,49#251661,50#318661,51#330530,52#263564,53#332005,54#332937,55#320163,56#300485,57#296138,58#343271,59#359351,60#295711,61#275751,62#332264,63#351532", "master_ver": "0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0,11#0,12#0,13#0,14#0,15#0,16#0,17#0,18#0,19#0,20#0,21#0,22#0,23#0,24#0,25#0,26#0,27#0,28#0,29#0,30#0,31#0,32#0,33#0,34#0,35#0,36#0,37#0,38#0,39#0,40#0,41#0,42#0,43#0,44#0,45#0,46#0,47#0,48#0,49#0,50#0,51#0,52#0,53#0,54#0,55#0,56#0,57#0,58#0,59#0,60#0,61#0,62#0,63#0", "mtime": "2020-06-03 12:13:01.207610Z", "max_marker": "0#,1#,2#,3#00000341068.137793799.5,4#00000360861.156871075.5,5#00000341415.97799619.5,6#00000279525.155275260.5,7#,8#,9#00000314796.95564320.5,10#,11#,12#00000289556.137784091.5,13#,14#,15#,16#00000241000.126242121.5,17#,18#00000274865.124884405.5,19#,20#00000307634.137793798.5,21#00000334605.93836734.5,22#,23#00000302566.125226103.5,24#,25#,26#,27#00000297451.125229375.5,28#,29#00000319132.155275278.5,30#00000361026.98341455.5,31#,32#,33#00000324848.126242117.5,34#00000278559.124884417.5,35#,36#,37#00000344788.125945123.5,38#00000345388.97796715.5,39#,40#00000386482.98341457.5,41#,42#00000358071.124884415.5,43#,44#00000339175.135084366.5,45#00000248078.155263912.5,46#00000356783.98341461.5,47#00000381495.94538350.5,48#00000295151.138701826.5,49#00000251660.137793803.5,50#00000318660.93848186.5,51#,52#,53#00000332004.126242119.5,54#00000332936.138701824.5,55#,56#00000300484.156871073.5,57#,58#00000343270.98341459.5,59#00000359350.94257302.5,60#,61#,62#,63#00000351531.135084368.5", "usage": { "rgw.none": { "size": 0, "size_actual": 0, "size_utilized": 0, "size_kb": 0, "size_kb_actual": 0, "size_kb_utilized": 0, "num_objects": 18446744073709551604 }, "rgw.main": { "size": 22615692324233, "size_actual": 22617004142592, "size_utilized": 18446736471689615156, "size_kb": 22085637036, "size_kb_actual": 22086918108, "size_kb_utilized": 18014391085634390, "num_objects": 521927 }, "rgw.multimeta": { "size": 0, "size_actual": 0, "size_utilized": 0, "size_kb": 0, "size_kb_actual": 0, "size_kb_utilized": 0, "num_objects": 977 } }, "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 } } This is another issue, probably not related to our sync issue. -- EveryWare AG François Scheurer Senior Systems Engineer Zurlindenstrasse 52a CH-8003 Zürich tel: +41 44 466 60 00 fax: +41 44 466 60 10 mail: francois.scheurer@everyware.ch<mailto:francois.scheurer@everyware.ch> web: http://www.everyware.ch
Hi Wissem Thank you for your reply. As the erasure-code-profile is a pool setting, it is on a lower layer and rgw should be unaware and independent of it, but it could play role regarding this know issue with space allocation and EC pools. Anyway this does so seem to be the cause here; after running rgw-orphan-list it appears that the secondary had about 80 M orphan objects. I do not know the reason yet but I will restart the sync from scratch and investigate further with the logs... (The other Issue with the wrong size_utilized was there long before we created the secondary zone) Cheers Francois --
participants (2)
-
Scheurer François
-
Wissem MIMOUNA