31 Dec
2020
31 Dec
'20
11:44 p.m.
Dear Ceph contributors
While our (new) rgw secondary zone is doing the initial data sync from our master zone,
we noticed that the reported capacity usage was getting higher than on primary zone:
Master Zone:
ceph version 14.2.5
zone parameters:
"log_meta": "false",
"log_data": "true",
"bucket_index_max_shards": 0,
"read_only": "false",
"tier_type": "",
"sync_from_all": "false",
"sync_from": [],
"redirect_zone": ""
bucket stats:
=> "size_actual GiB": 269868.9823989868,
"size_utilized GiB": 17180102071.868008, <= (cf. other issue
below)
"num_objects": 100218788,
"Compression Rate": 63660.899148714125
pool stats:
=> "stored": 403834132234240,
"objects": 191530416,
"kb_used": 692891724288,
"bytes_used": 709521125670912,
"percent_used": 0.7570806741714478,
"max_avail": 136595529269248
erasure-code-profile:
crush-device-class=nvme
crush-failure-domain=host
crush-root=default
jerasure-per-chunk-alignment=false
k=8
m=2
plugin=jerasure
technique=reed_sol_van
w=8
osd parameters:
bluefs_alloc_size 1048576 default
bluefs_shared_alloc_size 65536 default
bluestore_extent_map_inline_shard_prealloc_size 256 default
bluestore_max_alloc_size 0 default
bluestore_min_alloc_size 0 default
bluestore_min_alloc_size_hdd 65536 default
bluestore_min_alloc_size_ssd 16384 default
rgw compression: no
Secondary Zone:
ceph version 14.2.12
zone parameters:
"log_meta": "false",
"log_data": "true",
"bucket_index_max_shards": 11,
"read_only": "true",
"tier_type": "",
"sync_from_all": "true",
"sync_from": [],
"redirect_zone": ""
bucket stats:
=> "size_actual GiB": 65282.37313461304,
"size_utilized GiB": 60779.72828538809,
"num_objects": 23074921,
"Compression Rate": 0.9310281683550253
pool stats:
=> "stored": 407816305115136,
"objects": 118637638,
"kb_used": 497822635396,
"bytes_used": 509770378645504,
"percent_used": 0.7275146245956421,
"max_avail": 152744706965504
erasure-code-profile:
crush-device-class=
crush-failure-domain=host
crush-root=default
jerasure-per-chunk-alignment=false
k=3
m=2
plugin=jerasure
technique=reed_sol_van
w=8
osd parameters:
EC k=8 m=2
bluefs_alloc_size 1048576 default
bluefs_shared_alloc_size 65536 default
bluestore_extent_map_inline_shard_prealloc_size 256 default
bluestore_max_alloc_size 0 default
bluestore_min_alloc_size 0 default
bluestore_min_alloc_size_hdd 65536 default
bluestore_min_alloc_size_ssd 4096 default
rgw compression: yes
As you see, the secondary zone is using 408 TB vs 404 TB on master zone.
Summing size_actual for each buckets gives only 65 TB vs 270 TB on master zone.
Any idea about what could cause such a difference?
Is it a known issue?
The are some known issue with Space Overhead with EC Pools for alloc_size > 4 KiB, cf.
:
https://www.mail-archive.com/ceph-users@ceph.io/msg06191.html
https://lists.ceph.io/hyperkitty/list/dev@ceph.io/thread/OHPO43J54TPBEUISYC…
https://www.spinics.net/lists/ceph-users/msg59587.html
But our secondary zone is on nvme osd's with bluestore_min_alloc_size_ssd=4096, so
that should be fine.
I will also investigate further with "radosgw-admin bucket radoslist" and
rgw-orphan-list.
Thank you in advance for any help.
And Happy New Year very soon ;-)
Best Regards
Francois
PS:
The value on master zone "size_utilized GiB": 17180102071.868008 is wrong.
This is due to a bucket with wrong stats:
{
"bucket": "XXX",
"tenant": "",
"zonegroup": "d29ea82c-4f77-40af-952b-2ab0705ad268",
"placement_rule": "default-placement",
"explicit_placement": {
"data_pool": "",
"data_extra_pool": "",
"index_pool": ""
},
"id": "cb1594b3-a782-49d0-a19f-68cd48870a63.3119460.1",
"marker": "cb1594b3-a782-49d0-a19f-68cd48870a63.63841.1481",
"index_type": "Normal",
"owner": "e13d054f6e9c4eea881f687923d7d380",
"ver":
"0#373075,1#286613,2#290913,3#341069,4#360862,5#341416,6#279526,7#352172,8#255944,9#314797,10#317650,11#305698,12#289557,13#345344,14#345273,15#294708,16#241001,17#298577,18#274866,19#293952,20#307635,21#334606,22#265355,23#302567,24#277505,25#307278,26#266963,27#297452,28#332274,29#319133,30#361027,31#314294,32#282887,33#324849,34#278560,35#307506,36#287269,37#344789,38#345389,39#323814,40#386483,41#280319,42#358072,43#336651,44#339176,45#248079,46#356784,47#381496,48#295152,49#251661,50#318661,51#330530,52#263564,53#332005,54#332937,55#320163,56#300485,57#296138,58#343271,59#359351,60#295711,61#275751,62#332264,63#351532",
"master_ver":
"0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0,11#0,12#0,13#0,14#0,15#0,16#0,17#0,18#0,19#0,20#0,21#0,22#0,23#0,24#0,25#0,26#0,27#0,28#0,29#0,30#0,31#0,32#0,33#0,34#0,35#0,36#0,37#0,38#0,39#0,40#0,41#0,42#0,43#0,44#0,45#0,46#0,47#0,48#0,49#0,50#0,51#0,52#0,53#0,54#0,55#0,56#0,57#0,58#0,59#0,60#0,61#0,62#0,63#0",
"mtime": "2020-06-03 12:13:01.207610Z",
"max_marker":
"0#,1#,2#,3#00000341068.137793799.5,4#00000360861.156871075.5,5#00000341415.97799619.5,6#00000279525.155275260.5,7#,8#,9#00000314796.95564320.5,10#,11#,12#00000289556.137784091.5,13#,14#,15#,16#00000241000.126242121.5,17#,18#00000274865.124884405.5,19#,20#00000307634.137793798.5,21#00000334605.93836734.5,22#,23#00000302566.125226103.5,24#,25#,26#,27#00000297451.125229375.5,28#,29#00000319132.155275278.5,30#00000361026.98341455.5,31#,32#,33#00000324848.126242117.5,34#00000278559.124884417.5,35#,36#,37#00000344788.125945123.5,38#00000345388.97796715.5,39#,40#00000386482.98341457.5,41#,42#00000358071.124884415.5,43#,44#00000339175.135084366.5,45#00000248078.155263912.5,46#00000356783.98341461.5,47#00000381495.94538350.5,48#00000295151.138701826.5,49#00000251660.137793803.5,50#00000318660.93848186.5,51#,52#,53#00000332004.126242119.5,54#00000332936.138701824.5,55#,56#00000300484.156871073.5,57#,58#00000343270.98341459.5,59#00000359350.94257302.5,60#,61#,62#,63#00000351531.135084368.5",
"usage": {
"rgw.none": {
"size": 0,
"size_actual": 0,
"size_utilized": 0,
"size_kb": 0,
"size_kb_actual": 0,
"size_kb_utilized": 0,
"num_objects": 18446744073709551604
},
"rgw.main": {
"size": 22615692324233,
"size_actual": 22617004142592,
"size_utilized": 18446736471689615156,
"size_kb": 22085637036,
"size_kb_actual": 22086918108,
"size_kb_utilized": 18014391085634390,
"num_objects": 521927
},
"rgw.multimeta": {
"size": 0,
"size_actual": 0,
"size_utilized": 0,
"size_kb": 0,
"size_kb_actual": 0,
"size_kb_utilized": 0,
"num_objects": 977
}
},
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
}
}
This is another issue, probably not related to our sync issue.
--
EveryWare AG
François Scheurer
Senior Systems Engineer
Zurlindenstrasse 52a
CH-8003 Zürich
tel: +41 44 466 60 00
fax: +41 44 466 60 10
mail: francois.scheurer(a)everyware.ch
web: http://www.everyware.ch
1 Jan
1 Jan
12:47 a.m.
Hello ,
It could be related to the « erasure-code-profile » which is defined with different k+m on
master and secondary .
For the size of buckets I guess its probalbly due to the parameter : compression enabled
on secondary radosgw « rgw compression : yes » .
Regards
De : Scheurer François <francois.scheurer(a)everyware.ch>
Envoyé : jeudi 31 décembre 2020 15:45
À : ceph-users(a)ceph.io
Objet : [ceph-users] radosgw sync using more capacity on secondary than on master zone
Dear Ceph contributors
While our (new) rgw secondary zone is doing the initial data sync from our master zone,
we noticed that the reported capacity usage was getting higher than on primary zone:
Master Zone:
ceph version 14.2.5
zone parameters:
"log_meta": "false",
"log_data": "true",
"bucket_index_max_shards": 0,
"read_only": "false",
"tier_type": "",
"sync_from_all": "false",
"sync_from": [],
"redirect_zone": ""
bucket stats:
=> "size_actual GiB": 269868.9823989868,
"size_utilized GiB": 17180102071.868008, <= (cf. other issue
below)
"num_objects": 100218788,
"Compression Rate": 63660.899148714125
pool stats:
=> "stored": 403834132234240,
"objects": 191530416,
"kb_used": 692891724288,
"bytes_used": 709521125670912,
"percent_used": 0.7570806741714478,
"max_avail": 136595529269248
erasure-code-profile:
crush-device-class=nvme
crush-failure-domain=host
crush-root=default
jerasure-per-chunk-alignment=false
k=8
m=2
plugin=jerasure
technique=reed_sol_van
w=8
osd parameters:
bluefs_alloc_size 1048576 default
bluefs_shared_alloc_size 65536 default
bluestore_extent_map_inline_shard_prealloc_size 256 default
bluestore_max_alloc_size 0 default
bluestore_min_alloc_size 0 default
bluestore_min_alloc_size_hdd 65536 default
bluestore_min_alloc_size_ssd 16384 default
rgw compression: no
Secondary Zone:
ceph version 14.2.12
zone parameters:
"log_meta": "false",
"log_data": "true",
"bucket_index_max_shards": 11,
"read_only": "true",
"tier_type": "",
"sync_from_all": "true",
"sync_from": [],
"redirect_zone": ""
bucket stats:
=> "size_actual GiB": 65282.37313461304,
"size_utilized GiB": 60779.72828538809,
"num_objects": 23074921,
"Compression Rate": 0.9310281683550253
pool stats:
=> "stored": 407816305115136,
"objects": 118637638,
"kb_used": 497822635396,
"bytes_used": 509770378645504,
"percent_used": 0.7275146245956421,
"max_avail": 152744706965504
erasure-code-profile:
crush-device-class=
crush-failure-domain=host
crush-root=default
jerasure-per-chunk-alignment=false
k=3
m=2
plugin=jerasure
technique=reed_sol_van
w=8
osd parameters:
EC k=8 m=2
bluefs_alloc_size 1048576 default
bluefs_shared_alloc_size 65536 default
bluestore_extent_map_inline_shard_prealloc_size 256 default
bluestore_max_alloc_size 0 default
bluestore_min_alloc_size 0 default
bluestore_min_alloc_size_hdd 65536 default
bluestore_min_alloc_size_ssd 4096 default
rgw compression: yes
As you see, the secondary zone is using 408 TB vs 404 TB on master zone.
Summing size_actual for each buckets gives only 65 TB vs 270 TB on master zone.
Any idea about what could cause such a difference?
Is it a known issue?
The are some known issue with Space Overhead with EC Pools for alloc_size > 4 KiB, cf.
:
https://www.mail-archive.com/ceph-users@ceph.io/msg06191.html
https://lists.ceph.io/hyperkitty/list/dev@ceph.io/thread/OHPO43J54TPBEUISYC…
https://www.spinics.net/lists/ceph-users/msg59587.html
But our secondary zone is on nvme osd's with bluestore_min_alloc_size_ssd=4096, so
that should be fine.
I will also investigate further with "radosgw-admin bucket radoslist" and
rgw-orphan-list.
Thank you in advance for any help.
And Happy New Year very soon ;-)
Best Regards
Francois
PS:
The value on master zone "size_utilized GiB": 17180102071.868008 is wrong.
This is due to a bucket with wrong stats:
{
"bucket": "XXX",
"tenant": "",
"zonegroup": "d29ea82c-4f77-40af-952b-2ab0705ad268",
"placement_rule": "default-placement",
"explicit_placement": {
"data_pool": "",
"data_extra_pool": "",
"index_pool": ""
},
"id": "cb1594b3-a782-49d0-a19f-68cd48870a63.3119460.1",
"marker": "cb1594b3-a782-49d0-a19f-68cd48870a63.63841.1481",
"index_type": "Normal",
"owner": "e13d054f6e9c4eea881f687923d7d380",
"ver":
"0#373075,1#286613,2#290913,3#341069,4#360862,5#341416,6#279526,7#352172,8#255944,9#314797,10#317650,11#305698,12#289557,13#345344,14#345273,15#294708,16#241001,17#298577,18#274866,19#293952,20#307635,21#334606,22#265355,23#302567,24#277505,25#307278,26#266963,27#297452,28#332274,29#319133,30#361027,31#314294,32#282887,33#324849,34#278560,35#307506,36#287269,37#344789,38#345389,39#323814,40#386483,41#280319,42#358072,43#336651,44#339176,45#248079,46#356784,47#381496,48#295152,49#251661,50#318661,51#330530,52#263564,53#332005,54#332937,55#320163,56#300485,57#296138,58#343271,59#359351,60#295711,61#275751,62#332264,63#351532",
"master_ver":
"0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0,11#0,12#0,13#0,14#0,15#0,16#0,17#0,18#0,19#0,20#0,21#0,22#0,23#0,24#0,25#0,26#0,27#0,28#0,29#0,30#0,31#0,32#0,33#0,34#0,35#0,36#0,37#0,38#0,39#0,40#0,41#0,42#0,43#0,44#0,45#0,46#0,47#0,48#0,49#0,50#0,51#0,52#0,53#0,54#0,55#0,56#0,57#0,58#0,59#0,60#0,61#0,62#0,63#0",
"mtime": "2020-06-03 12:13:01.207610Z",
"max_marker":
"0#,1#,2#,3#00000341068.137793799.5,4#00000360861.156871075.5,5#00000341415.97799619.5,6#00000279525.155275260.5,7#,8#,9#00000314796.95564320.5,10#,11#,12#00000289556.137784091.5,13#,14#,15#,16#00000241000.126242121.5,17#,18#00000274865.124884405.5,19#,20#00000307634.137793798.5,21#00000334605.93836734.5,22#,23#00000302566.125226103.5,24#,25#,26#,27#00000297451.125229375.5,28#,29#00000319132.155275278.5,30#00000361026.98341455.5,31#,32#,33#00000324848.126242117.5,34#00000278559.124884417.5,35#,36#,37#00000344788.125945123.5,38#00000345388.97796715.5,39#,40#00000386482.98341457.5,41#,42#00000358071.124884415.5,43#,44#00000339175.135084366.5,45#00000248078.155263912.5,46#00000356783.98341461.5,47#00000381495.94538350.5,48#00000295151.138701826.5,49#00000251660.137793803.5,50#00000318660.93848186.5,51#,52#,53#00000332004.126242119.5,54#00000332936.138701824.5,55#,56#00000300484.156871073.5,57#,58#00000343270.98341459.5,59#00000359350.94257302.5,60#,61#,62#,63#00000351531.135084368.5",
"usage": {
"rgw.none": {
"size": 0,
"size_actual": 0,
"size_utilized": 0,
"size_kb": 0,
"size_kb_actual": 0,
"size_kb_utilized": 0,
"num_objects": 18446744073709551604
},
"rgw.main": {
"size": 22615692324233,
"size_actual": 22617004142592,
"size_utilized": 18446736471689615156,
"size_kb": 22085637036,
"size_kb_actual": 22086918108,
"size_kb_utilized": 18014391085634390,
"num_objects": 521927
},
"rgw.multimeta": {
"size": 0,
"size_actual": 0,
"size_utilized": 0,
"size_kb": 0,
"size_kb_actual": 0,
"size_kb_utilized": 0,
"num_objects": 977
}
},
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
}
}
This is another issue, probably not related to our sync issue.
--
EveryWare AG
François Scheurer
Senior Systems Engineer
Zurlindenstrasse 52a
CH-8003 Zürich
tel: +41 44 466 60 00
fax: +41 44 466 60 10
mail: francois.scheurer@everyware.ch<mailto:francois.scheurer@everyware.ch>
web: http://www.everyware.ch
7 Jan
7 Jan
2:53 a.m.
Hi Wissem
Thank you for your reply.
As the erasure-code-profile is a pool setting, it is on a lower layer and rgw should be
unaware and independent of it, but it could play role regarding this know issue with space
allocation and EC pools.
Anyway this does so seem to be the cause here; after running rgw-orphan-list it appears
that the secondary had about 80 M orphan objects.
I do not know the reason yet but I will restart the sync from scratch and investigate
further with the logs...
(The other Issue with the wrong size_utilized was there long before we created the
secondary zone)
Cheers
Francois
--
1199
days inactive
1205
days old
2 comments
2 participants
participants (2)
-
Scheurer François -
Wissem MIMOUNA