High usage (DATA column) on dedicated for OMAP only OSDs
Hi! We have S3/CephFS cluster with dedicated SSDs for bucket indexes, CephFS metadata and for a few small rgw metadata pools. We have special crush rules for these pools. All of our pools (pool naming is old because it's very old cluster) that are placed on SSDs: POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .rgw.root 1 32 1.3 KiB 5 108 KiB 0 690 GiB .rgw.control 2 32 0 B 8 0 B 0 690 GiB .rgw 3 32 2.0 MiB 8.02k 328 MiB 0.02 690 GiB .rgw.gc 4 64 1.6 GiB 64 4.9 GiB 0.24 690 GiB .users.uid 5 32 1.8 MiB 2.63k 67 MiB 0 690 GiB .users 6 32 275 KiB 7.02k 379 MiB 0.02 690 GiB .usage 7 32 30 MiB 64 30 MiB 0 690 GiB .intent-log 8 64 0 B 0 0 B 0 690 GiB .log 9 64 6.6 GiB 34.29k 21 GiB 1.01 690 GiB .rgw.buckets.index 12 4096 1.9 TiB 248.96k 1.9 TiB 48.02 690 GiB .users.email 13 32 64 KiB 1.61k 72 MiB 0 690 GiB fs1_meta 14 64 467 MiB 35.39k 829 MiB 0.04 690 GiB But on all our SSDs we can see high DATA usage, for example: ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME 57 ssd 0.00005 1.00000 447 GiB 406 GiB 309 GiB 96 GiB 1.4 GiB 40 GiB 90.96 1.37 138 up osd.57 19 ssd 0.00005 1.00000 447 GiB 402 GiB 309 GiB 92 GiB 1.4 GiB 44 GiB 90.09 1.35 145 up osd.19 10 ssd 0.00005 1.00000 447 GiB 406 GiB 309 GiB 97 GiB 1.1 GiB 40 GiB 91.01 1.37 137 up osd.10 12 ssd 0.00005 1.00000 447 GiB 401 GiB 309 GiB 91 GiB 1.3 GiB 46 GiB 89.77 1.35 134 up osd.12 All SSDs are bluestore. Ceph 16.3.11 and many SSDs were deployed on 16.x. As I understand, the majority of these pools contain only "OMAP" which is stored in RocksDB and tracked in stats as *OMAP*. And I don't understand why we have such high *DATA* and we can see that DATA usage grows by 1-2 GB/day for the last 90 days (we don't have more monitoring data)! I have checked crush rules and all SSDs for PGs from other pools and can't see any other PGs from "pure" data pools. Rules are correct. How can I see what is stored in bluestore DATA on some OSD? I can export full RocksDB, but not DATA stored in bluestore.. What kind of DATA can be stored in these OSDs with these pools? We did a lot of offline compactions, 2-5 times for some SSDs in the last 90 days. But i'm not sure if it's related
Hi Alexander, I recall a couple of cases when permanent osdmap epoch growth has been filling OSD with relevant osd map info. Which could be tricky to catch. Please run 'ceph tell osd.N status" for a couple of affected OSDs twice within e.g. 10 min interval. Then check the delta between oldest_map and newest_map fields - neither the delta should be very large (hundreds of thousands) nor it should grow rapidly within the observed interval. If so - please share these reports, 'ceph health detailed' output and exact Ceph release version you're using. Thanks, Igor On 9/18/2024 2:32 PM, Александр Руденко wrote:
erstand, the majority of these pools contain only
-- Igor Fedotov Ceph Lead Developer Looking for help with your Ceph cluster? Contact us athttps://croit.io croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web:https://croit.io | YouTube:https://goo.gl/PGE1Bx
Hi, Igor. Thank you for your reply! ceph tell osd.10 status| grep map; echo ---; sleep 600; ceph tell osd.10 status| grep map "oldest_map": 2408326, "newest_map": 2635631, --- "oldest_map": 2408326, "newest_map": 2635647, ceph version is 16.2.13 ceph health: HEALTH_WARN mons a,b,c,d,e are using a lot of disk space; 8 backfillfull osd(s); 143 nearfull osd(s); Low space hindering backfill (add storage if this doesn't resolve itself): 25 pgs backfill_toofull; (muted: BLUESTORE_NO_PER_POOL_OMAP PG_NOT_DEEP_SCRUBBED POOL_BACKFILLFULL POOL_NEARFULL) (MUTED, STICKY) [WRN] BLUESTORE_NO_PER_POOL_OMAP: 1756 OSD(s) reporting legacy (not per-pool) BlueStore omap usage stats osd.1 legacy (not per-pool) omap detected, suggest to run store repair to benefit from per-pool omap usage statistics ... osd.2772 legacy (not per-pool) omap detected, suggest to run store repair to benefit from per-pool omap usage statistics [WRN] MON_DISK_BIG: mons a,b,c,d,e are using a lot of disk space mon.a is 56 GiB >= mon_data_size_warn (15 GiB) ... [WRN] OSD_BACKFILLFULL: 8 backfillfull osd(s) osd.12 is backfill full ... [WRN] OSD_NEARFULL: 143 nearfull osd(s) osd.1 is near full ... [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 25 pgs backfill_toofull pg 10.6ea is active+remapped+backfill_toofull, acting [1507,941,2649] ... (MUTED, STICKY) [WRN] PG_NOT_DEEP_SCRUBBED: 2302 pgs not deep-scrubbed in time pg 10.7ffe not deep-scrubbed since 2024-08-23T19:28:22.749150+0300 ... (MUTED, STICKY) [WRN] POOL_BACKFILLFULL: 19 pool(s) backfillfull pool '.rgw.root' is backfillfull pool '.rgw.control' is backfillfull pool '.rgw' is backfillfull pool '.rgw.gc' is backfillfull pool '.users.uid' is backfillfull pool '.users' is backfillfull pool '.usage' is backfillfull pool '.intent-log' is backfillfull pool '.log' is backfillfull pool '.rgw.buckets' is backfillfull pool '.rgw.buckets.extra' is backfillfull pool '.rgw.buckets.index' is backfillfull pool '.users.email' is backfillfull pool 'fs1_meta' is backfillfull pool 'fs1_data' is backfillfull pool 'fs1_tmp' is backfillfull pool 'device_health_metrics' is backfillfull pool 'default.rgw.meta' is backfillfull ср, 18 сент. 2024 г. в 18:23, Igor Fedotov <igor.fedotov@croit.io>:
Hi Alexander,
I recall a couple of cases when permanent osdmap epoch growth has been filling OSD with relevant osd map info. Which could be tricky to catch.
Please run 'ceph tell osd.N status" for a couple of affected OSDs twice within e.g. 10 min interval.
Then check the delta between oldest_map and newest_map fields - neither the delta should be very large (hundreds of thousands) nor it should grow rapidly within the observed interval. If so - please share these reports, 'ceph health detailed' output and exact Ceph release version you're using.
Thanks,
Igor
On 9/18/2024 2:32 PM, Александр Руденко wrote:
erstand, the majority of these pools contain only
-- Igor Fedotov Ceph Lead Developer
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx
Hi Alexander, so newwest_map looks slowly growing. And (which is worse) oldest_map is constant. Which means no old map pruning is happening and more and more maps are coming. What are the numbers today? You can assess the number of objects in "meta" pool (that's where osdmaps are kept) for an OSD by using ceph-objectstore-tool's meta-list command. This needs OSD to be offline. Then multiply that amount by 4K (presuming min alloc unit for these OSDs is 4K) and then multiply by amount of OSDs to learn the minimal space taken by this data. In reality osdmap can be much(?) larger than 4K but I don't know an easy way to assess that in Pacific. Quincy has got this patch for ceph-objectstore-tool to retrieve such object though: https://github.com/ceph/ceph/pull/39082 And please note that the above doesn't mean there are no other sources for utilization growth. But apparently it's worth some attention. Hope this helps, Igor On 9/18/2024 9:20 PM, Александр Руденко wrote:
Hi, Igor.
Thank you for your reply!
ceph tell osd.10 status| grep map; echo ---; sleep 600; ceph tell osd.10 status| grep map "oldest_map": 2408326, "newest_map": 2635631, --- "oldest_map": 2408326, "newest_map": 2635647,
ceph version is 16.2.13
ceph health:
HEALTH_WARN mons a,b,c,d,e are using a lot of disk space; 8 backfillfull osd(s); 143 nearfull osd(s); Low space hindering backfill (add storage if this doesn't resolve itself): 25 pgs backfill_toofull; (muted: BLUESTORE_NO_PER_POOL_OMAP PG_NOT_DEEP_SCRUBBED POOL_BACKFILLFULL POOL_NEARFULL) (MUTED, STICKY) [WRN] BLUESTORE_NO_PER_POOL_OMAP: 1756 OSD(s) reporting legacy (not per-pool) BlueStore omap usage stats osd.1 legacy (not per-pool) omap detected, suggest to run store repair to benefit from per-pool omap usage statistics ... osd.2772 legacy (not per-pool) omap detected, suggest to run store repair to benefit from per-pool omap usage statistics [WRN] MON_DISK_BIG: mons a,b,c,d,e are using a lot of disk space mon.a is 56 GiB >= mon_data_size_warn (15 GiB) ... [WRN] OSD_BACKFILLFULL: 8 backfillfull osd(s) osd.12 is backfill full ... [WRN] OSD_NEARFULL: 143 nearfull osd(s) osd.1 is near full ... [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 25 pgs backfill_toofull pg 10.6ea is active+remapped+backfill_toofull, acting [1507,941,2649] ... (MUTED, STICKY) [WRN] PG_NOT_DEEP_SCRUBBED: 2302 pgs not deep-scrubbed in time pg 10.7ffe not deep-scrubbed since 2024-08-23T19:28:22.749150+0300 ... (MUTED, STICKY) [WRN] POOL_BACKFILLFULL: 19 pool(s) backfillfull pool '.rgw.root' is backfillfull pool '.rgw.control' is backfillfull pool '.rgw' is backfillfull pool '.rgw.gc' is backfillfull pool '.users.uid' is backfillfull pool '.users' is backfillfull pool '.usage' is backfillfull pool '.intent-log' is backfillfull pool '.log' is backfillfull pool '.rgw.buckets' is backfillfull pool '.rgw.buckets.extra' is backfillfull pool '.rgw.buckets.index' is backfillfull pool '.users.email' is backfillfull pool 'fs1_meta' is backfillfull pool 'fs1_data' is backfillfull pool 'fs1_tmp' is backfillfull pool 'device_health_metrics' is backfillfull pool 'default.rgw.meta' is backfillfull
ср, 18 сент. 2024 г. в 18:23, Igor Fedotov <igor.fedotov@croit.io>:
Hi Alexander,
I recall a couple of cases when permanent osdmap epoch growth has been filling OSD with relevant osd map info. Which could be tricky to catch.
Please run 'ceph tell osd.N status" for a couple of affected OSDs twice within e.g. 10 min interval.
Then check the delta between oldest_map and newest_map fields - neither the delta should be very large (hundreds of thousands) nor it should grow rapidly within the observed interval.
If so - please share these reports, 'ceph health detailed' output and exact Ceph release version you're using.
Thanks,
Igor
On 9/18/2024 2:32 PM, Александр Руденко wrote:
erstand, the majority of these pools contain only
-- Igor Fedotov Ceph Lead Developer
Looking for help with your Ceph cluster? Contact us athttps://croit.io
croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web:https://croit.io | YouTube:https://goo.gl/PGE1Bx
-- Igor Fedotov Ceph Lead Developer Looking for help with your Ceph cluster? Contact us athttps://croit.io croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web:https://croit.io | YouTube:https://goo.gl/PGE1Bx
Hi Igor,
On 18 Sep 2024, at 18:22, Igor Fedotov <igor.fedotov@croit.io> wrote:
I recall a couple of cases when permanent osdmap epoch growth has been filling OSD with relevant osd map info. Which could be tricky to catch.
Please run 'ceph tell osd.N status" for a couple of affected OSDs twice within e.g. 10 min interval.
Then check the delta between oldest_map and newest_map fields - neither the delta should be very large (hundreds of thousands) nor it should grow rapidly within the observed interval.
Side question by topic. What is option controls how much maps to prune? Currently I need to trim 1M osdmaps, but when new map issued, only 30 old maps are removed. What option controls value=30? Thanks, k
Hi Konstantin, osd_target_transaction_size should control that. I've heard of it being raized to 150 with no obvious issues. Going beyond is at your own risk. So I'd suggest to apply incremental increase if needed. Thanks, Igor On 9/19/2024 10:44 AM, Konstantin Shalygin wrote:
Hi Igor,
On 18 Sep 2024, at 18:22, Igor Fedotov <igor.fedotov@croit.io> wrote:
I recall a couple of cases when permanent osdmap epoch growth has been filling OSD with relevant osd map info. Which could be tricky to catch.
Please run 'ceph tell osd.N status" for a couple of affected OSDs twice within e.g. 10 min interval.
Then check the delta between oldest_map and newest_map fields - neither the delta should be very large (hundreds of thousands) nor it should grow rapidly within the observed interval.
Side question by topic. What is option controls how much maps to prune? Currently I need to trim 1M osdmaps, but when new map issued, only 30 old maps are removed. What option controls value=30?
Thanks, k
-- Igor Fedotov Ceph Lead Developer Looking for help with your Ceph cluster? Contact us athttps://croit.io croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web:https://croit.io | YouTube:https://goo.gl/PGE1Bx
Hi,
On 19 Sep 2024, at 12:33, Igor Fedotov <igor.fedotov@croit.io> wrote:
osd_target_transaction_size should control that.
I've heard of it being raized to 150 with no obvious issues. Going beyond is at your own risk. So I'd suggest to apply incremental increase if needed.
Thanks! Now much better k
Igor, thanks!
What are the numbers today?
and then multiply by amount of OSDs to learn the minimal space taken by
Today we have the same "oldest_map": 2408326 and "newest_map": 2637838, *+2191*. ceph-objectstore-tool --op meta-list --data-path /var/lib/ceph/osd/ceph-70 | grep osdmap | wc -l 458994 Can you clarify this, please: this data 458994 * 4k * OSDs count = "*size of osdmaps on ONE OSD*" or "*total size of osdmaps on ALL OSDs*" ? Because we have about 3k OSDS and 458994 * 4k * 3000 = ~5TB and it can be placed on ONE OSD. But if it is TOTAL osdmap size, I think it is a very small size per OSD. But we have a lot of osds with min_alloc_size=64k which was default in previous ceph's versions for rotational drives (all our SSDs behind old RAID controllers). ceph daemon osd.10 bluestore allocator dump block | head -10 { "capacity": 479557844992, "alloc_unit": 65536, But even with min_alloc=64k it will not be a big amount of data 458994 * 64k = *~23GB*. I think we have about *150GB+* extra per SSD OSDs. For example, SSD with min_alloc=4k: ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 126 ssd 0.00005 1.00000 447 GiB 374 GiB 300 GiB 72 GiB 1.4 GiB 73 GiB 83.64 1.00 137 up with min_alloc=64k: ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 10 ssd 0.00005 0.75000 447 GiB 405 GiB 320 GiB 83 GiB 1.4 GiB 42 GiB 90.59 1.00 114 up Diff is not as big as 4k vs 64k.. чт, 19 сент. 2024 г. в 12:33, Igor Fedotov <igor.fedotov@croit.io>:
Hi Konstantin,
osd_target_transaction_size should control that.
I've heard of it being raized to 150 with no obvious issues. Going beyond is at your own risk. So I'd suggest to apply incremental increase if needed.
Thanks,
Igor On 9/19/2024 10:44 AM, Konstantin Shalygin wrote:
Hi Igor,
On 18 Sep 2024, at 18:22, Igor Fedotov <igor.fedotov@croit.io> <igor.fedotov@croit.io> wrote:
I recall a couple of cases when permanent osdmap epoch growth has been filling OSD with relevant osd map info. Which could be tricky to catch.
Please run 'ceph tell osd.N status" for a couple of affected OSDs twice within e.g. 10 min interval.
Then check the delta between oldest_map and newest_map fields - neither the delta should be very large (hundreds of thousands) nor it should grow rapidly within the observed interval.
Side question by topic. What is option controls how much maps to prune? Currently I need to trim 1M osdmaps, but when new map issued, only 30 old maps are removed. What option controls value=30?
Thanks, k
-- Igor Fedotov Ceph Lead Developer
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx
please see my comments inline. On 9/19/2024 1:53 PM, Александр Руденко wrote:
Igor, thanks!
What are the numbers today?
Today we have the same "oldest_map": 2408326 and "newest_map": 2637838, *+2191*.
ceph-objectstore-tool --op meta-list --data-path /var/lib/ceph/osd/ceph-70 | grep osdmap | wc -l 458994
Can you clarify this, please:
and then multiply by amount of OSDs to learn the minimal space taken by this data
458994 * 4k * OSDs count = "_size of osdmaps on *ONE* OSD_" or "_total size of osdmaps on *ALL* OSDs_" ?
Yes, this is a lower bound estimation for osdmap size on all OSDs.
Because we have about 3k OSDS and 458994 * 4k * 3000 = ~5TB and it can be placed on ONE OSD. But if it is TOTAL osdmap size, I think it is a very small size per OSD.
Highly likely that osdmap for 3K OSDs takes much more than 4K on disk. So again that was just lower bound estimation. In fact one can use 'ceph osd getmap >out.dat' and get better estimation of osdmap size. So please substitute 4K in the formala above to get better estimation for the overall space taken. It's a bit simplified though since just half of the entries in 'meta' pool are full osdmaps. Hence you might want to use 458994/2 * sizeof(osdmap) + 458994/2 * 4K in the above formula. Which is again a sort of low bound estimation but with a better accuracy.
But we have a lot of osds with min_alloc_size=64k which was default in previous ceph's versions for rotational drives (all our SSDs behind old RAID controllers).
ceph daemon osd.10 bluestore allocator dump block | head -10 { "capacity": 479557844992, "alloc_unit": 65536,
But even with min_alloc=64k it will not be a big amount of data 458994 * 64k = *~23GB*. I think we have about *150GB+* extra per SSD OSDs.
Yeah, you should use 64K instead of 4K for the above formula if you have the majority of OSDs using 64K alloc unit. Or take this into account somehow else (e.g. take half 4K and half 64K). But I'm leaving this as a "home excercise" for yourself. The main point here is that a single object would take at least alloc_unit size. And hence I was trying to make the assessment without knowing actual osdmap size but using alloc unit one. Just to check ifwe get numbers of the same order of magnitude. And 23GB and 150GB aren't THAT differ - having e.g. 1M osdmap might easily do the trick. I.e. the osdmap leak indeed could be a real factor here. And hence it's worth additional investigation. Anyway - please use the obtained osdmap size. It could adjust the resulting estimation value dramatically.
For example, SSD with min_alloc=4k: ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 126 ssd 0.00005 1.00000 447 GiB 374 GiB 300 GiB 72 GiB 1.4 GiB 73 GiB 83.64 1.00 137 up
with min_alloc=64k: ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 10 ssd 0.00005 0.75000 447 GiB 405 GiB 320 GiB 83 GiB 1.4 GiB 42 GiB 90.59 1.00 114 up
Diff is not as big as 4k vs 64k..
Right. Don't know the reason atm. May be leaking osdmaps is not the only iissue. Please do the corrected math as per above though..
чт, 19 сент. 2024 г. в 12:33, Igor Fedotov <igor.fedotov@croit.io>:
Hi Konstantin,
osd_target_transaction_size should control that.
I've heard of it being raized to 150 with no obvious issues. Going beyond is at your own risk. So I'd suggest to apply incremental increase if needed.
Thanks,
Igor
On 9/19/2024 10:44 AM, Konstantin Shalygin wrote:
Hi Igor,
On 18 Sep 2024, at 18:22, Igor Fedotov <igor.fedotov@croit.io> <mailto:igor.fedotov@croit.io> wrote:
I recall a couple of cases when permanent osdmap epoch growth has been filling OSD with relevant osd map info. Which could be tricky to catch.
Please run 'ceph tell osd.N status" for a couple of affected OSDs twice within e.g. 10 min interval.
Then check the delta between oldest_map and newest_map fields - neither the delta should be very large (hundreds of thousands) nor it should grow rapidly within the observed interval.
Side question by topic. What is option controls how much maps to prune? Currently I need to trim 1M osdmaps, but when new map issued, only 30 old maps are removed. What option controls value=30?
Thanks, k
-- Igor Fedotov Ceph Lead Developer
Looking for help with your Ceph cluster? Contact us athttps://croit.io
croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web:https://croit.io | YouTube:https://goo.gl/PGE1Bx
-- Igor Fedotov Ceph Lead Developer Looking for help with your Ceph cluster? Contact us athttps://croit.io croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web:https://croit.io | YouTube:https://goo.gl/PGE1Bx
Igor, thanks, very helpful. Our current osdmap weighs 1.4MB. And it changes all calculations.. Looks like it can be our case. I think we have this situation due to long backfilling which takes place now and going for the last 3 weeks. Can we drop some amount of osdmaps before rebalance completes? чт, 19 сент. 2024 г. в 15:38, Igor Fedotov <igor.fedotov@croit.io>:
please see my comments inline.
On 9/19/2024 1:53 PM, Александр Руденко wrote:
Igor, thanks!
What are the numbers today?
Today we have the same "oldest_map": 2408326 and "newest_map": 2637838, *+2191*.
ceph-objectstore-tool --op meta-list --data-path /var/lib/ceph/osd/ceph-70 | grep osdmap | wc -l 458994
Can you clarify this, please:
and then multiply by amount of OSDs to learn the minimal space taken by this data
458994 * 4k * OSDs count = "*size of osdmaps on ONE OSD*" or "*total size of osdmaps on ALL OSDs*" ?
Yes, this is a lower bound estimation for osdmap size on all OSDs.
Because we have about 3k OSDS and 458994 * 4k * 3000 = ~5TB and it can be placed on ONE OSD. But if it is TOTAL osdmap size, I think it is a very small size per OSD.
Highly likely that osdmap for 3K OSDs takes much more than 4K on disk. So again that was just lower bound estimation.
In fact one can use 'ceph osd getmap >out.dat' and get better estimation of osdmap size. So please substitute 4K in the formala above to get better estimation for the overall space taken.
It's a bit simplified though since just half of the entries in 'meta' pool are full osdmaps. Hence you might want to use 458994/2 * sizeof(osdmap) + 458994/2 * 4K in the above formula.
Which is again a sort of low bound estimation but with a better accuracy.
But we have a lot of osds with min_alloc_size=64k which was default in previous ceph's versions for rotational drives (all our SSDs behind old RAID controllers).
ceph daemon osd.10 bluestore allocator dump block | head -10 { "capacity": 479557844992, "alloc_unit": 65536,
But even with min_alloc=64k it will not be a big amount of data 458994 * 64k = *~23GB*. I think we have about *150GB+* extra per SSD OSDs.
Yeah, you should use 64K instead of 4K for the above formula if you have the majority of OSDs using 64K alloc unit. Or take this into account somehow else (e.g. take half 4K and half 64K). But I'm leaving this as a "home excercise" for yourself. The main point here is that a single object would take at least alloc_unit size. And hence I was trying to make the assessment without knowing actual osdmap size but using alloc unit one. Just to check ifwe get numbers of the same order of magnitude. And 23GB and 150GB aren't THAT differ - having e.g. 1M osdmap might easily do the trick. I.e. the osdmap leak indeed could be a real factor here. And hence it's worth additional investigation.
Anyway - please use the obtained osdmap size. It could adjust the resulting estimation value dramatically.
For example, SSD with min_alloc=4k: ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 126 ssd 0.00005 1.00000 447 GiB 374 GiB 300 GiB 72 GiB 1.4 GiB 73 GiB 83.64 1.00 137 up
with min_alloc=64k: ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 10 ssd 0.00005 0.75000 447 GiB 405 GiB 320 GiB 83 GiB 1.4 GiB 42 GiB 90.59 1.00 114 up
Diff is not as big as 4k vs 64k..
Right. Don't know the reason atm. May be leaking osdmaps is not the only iissue. Please do the corrected math as per above though..
чт, 19 сент. 2024 г. в 12:33, Igor Fedotov <igor.fedotov@croit.io>:
Hi Konstantin,
osd_target_transaction_size should control that.
I've heard of it being raized to 150 with no obvious issues. Going beyond is at your own risk. So I'd suggest to apply incremental increase if needed.
Thanks,
Igor On 9/19/2024 10:44 AM, Konstantin Shalygin wrote:
Hi Igor,
On 18 Sep 2024, at 18:22, Igor Fedotov <igor.fedotov@croit.io> <igor.fedotov@croit.io> wrote:
I recall a couple of cases when permanent osdmap epoch growth has been filling OSD with relevant osd map info. Which could be tricky to catch.
Please run 'ceph tell osd.N status" for a couple of affected OSDs twice within e.g. 10 min interval.
Then check the delta between oldest_map and newest_map fields - neither the delta should be very large (hundreds of thousands) nor it should grow rapidly within the observed interval.
Side question by topic. What is option controls how much maps to prune? Currently I need to trim 1M osdmaps, but when new map issued, only 30 old maps are removed. What option controls value=30?
Thanks, k
-- Igor Fedotov Ceph Lead Developer
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx
-- Igor Fedotov Ceph Lead Developer
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web: https://croit.io | YouTube: https://goo.gl/PGE1Bx
Here it goes beyond of my expertise. I saw unbounded osdmap epoch growth for two completely different cases. And unable to say what's causing it this time. But IMO you shouldn't do any osdmap trimming yourself - that could likely result in an unpredictable behavior. So I'd encourage you to find a way for the cluster to do that gracefully by itself. Thanks, Igor On 9/19/2024 5:16 PM, Александр Руденко wrote:
Igor, thanks, very helpful.
Our current osdmap weighs 1.4MB. And it changes all calculations..
Looks like it can be our case.
I think we have this situation due to long backfilling which takes place now and going for the last 3 weeks. Can we drop some amount of osdmaps before rebalance completes?
чт, 19 сент. 2024 г. в 15:38, Igor Fedotov <igor.fedotov@croit.io>:
please see my comments inline.
On 9/19/2024 1:53 PM, Александр Руденко wrote:
Igor, thanks!
> What are the numbers today?
Today we have the same "oldest_map": 2408326 and "newest_map": 2637838, *+2191*.
ceph-objectstore-tool --op meta-list --data-path /var/lib/ceph/osd/ceph-70 | grep osdmap | wc -l 458994
Can you clarify this, please:
> and then multiply by amount of OSDs to learn the minimal space taken by this data
458994 * 4k * OSDs count = "_size of osdmaps on *ONE* OSD_" or "_total size of osdmaps on *ALL* OSDs_" ?
Yes, this is a lower bound estimation for osdmap size on all OSDs.
Because we have about 3k OSDS and 458994 * 4k * 3000 = ~5TB and it can be placed on ONE OSD. But if it is TOTAL osdmap size, I think it is a very small size per OSD.
Highly likely that osdmap for 3K OSDs takes much more than 4K on disk. So again that was just lower bound estimation.
In fact one can use 'ceph osd getmap >out.dat' and get better estimation of osdmap size. So please substitute 4K in the formala above to get better estimation for the overall space taken.
It's a bit simplified though since just half of the entries in 'meta' pool are full osdmaps. Hence you might want to use 458994/2 * sizeof(osdmap) + 458994/2 * 4K in the above formula.
Which is again a sort of low bound estimation but with a better accuracy.
But we have a lot of osds with min_alloc_size=64k which was default in previous ceph's versions for rotational drives (all our SSDs behind old RAID controllers).
ceph daemon osd.10 bluestore allocator dump block | head -10 { "capacity": 479557844992, "alloc_unit": 65536,
But even with min_alloc=64k it will not be a big amount of data 458994 * 64k = *~23GB*. I think we have about *150GB+* extra per SSD OSDs.
Yeah, you should use 64K instead of 4K for the above formula if you have the majority of OSDs using 64K alloc unit. Or take this into account somehow else (e.g. take half 4K and half 64K). But I'm leaving this as a "home excercise" for yourself. The main point here is that a single object would take at least alloc_unit size. And hence I was trying to make the assessment without knowing actual osdmap size but using alloc unit one. Just to check ifwe get numbers of the same order of magnitude. And 23GB and 150GB aren't THAT differ - having e.g. 1M osdmap might easily do the trick. I.e. the osdmap leak indeed could be a real factor here. And hence it's worth additional investigation.
Anyway - please use the obtained osdmap size. It could adjust the resulting estimation value dramatically.
For example, SSD with min_alloc=4k: ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 126 ssd 0.00005 1.00000 447 GiB 374 GiB 300 GiB 72 GiB 1.4 GiB 73 GiB 83.64 1.00 137 up
with min_alloc=64k: ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 10 ssd 0.00005 0.75000 447 GiB 405 GiB 320 GiB 83 GiB 1.4 GiB 42 GiB 90.59 1.00 114 up
Diff is not as big as 4k vs 64k..
Right. Don't know the reason atm. May be leaking osdmaps is not the only iissue. Please do the corrected math as per above though..
чт, 19 сент. 2024 г. в 12:33, Igor Fedotov <igor.fedotov@croit.io>:
Hi Konstantin,
osd_target_transaction_size should control that.
I've heard of it being raized to 150 with no obvious issues. Going beyond is at your own risk. So I'd suggest to apply incremental increase if needed.
Thanks,
Igor
On 9/19/2024 10:44 AM, Konstantin Shalygin wrote:
Hi Igor,
On 18 Sep 2024, at 18:22, Igor Fedotov <igor.fedotov@croit.io> <mailto:igor.fedotov@croit.io> wrote:
I recall a couple of cases when permanent osdmap epoch growth has been filling OSD with relevant osd map info. Which could be tricky to catch.
Please run 'ceph tell osd.N status" for a couple of affected OSDs twice within e.g. 10 min interval.
Then check the delta between oldest_map and newest_map fields - neither the delta should be very large (hundreds of thousands) nor it should grow rapidly within the observed interval.
Side question by topic. What is option controls how much maps to prune? Currently I need to trim 1M osdmaps, but when new map issued, only 30 old maps are removed. What option controls value=30?
Thanks, k
-- Igor Fedotov Ceph Lead Developer
Looking for help with your Ceph cluster? Contact us athttps://croit.io
croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web:https://croit.io | YouTube:https://goo.gl/PGE1Bx
-- Igor Fedotov Ceph Lead Developer
Looking for help with your Ceph cluster? Contact us athttps://croit.io
croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web:https://croit.io | YouTube:https://goo.gl/PGE1Bx
-- Igor Fedotov Ceph Lead Developer Looking for help with your Ceph cluster? Contact us athttps://croit.io croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web:https://croit.io | YouTube:https://goo.gl/PGE1Bx
Whenever we've seen osdmaps not being trimmed, we've made sure that any down OSDs are out+destroyed, and then have rolled a restart through the mons. As of recent Pacific at least this seems to have reliably gotten us out of this situation. Josh On Thu, Sep 19, 2024 at 9:14 AM Igor Fedotov <igor.fedotov@croit.io> wrote:
Here it goes beyond of my expertise.
I saw unbounded osdmap epoch growth for two completely different cases. And unable to say what's causing it this time.
But IMO you shouldn't do any osdmap trimming yourself - that could likely result in an unpredictable behavior. So I'd encourage you to find a way for the cluster to do that gracefully by itself.
Thanks,
Igor
On 9/19/2024 5:16 PM, Александр Руденко wrote:
Igor, thanks, very helpful.
Our current osdmap weighs 1.4MB. And it changes all calculations..
Looks like it can be our case.
I think we have this situation due to long backfilling which takes place now and going for the last 3 weeks. Can we drop some amount of osdmaps before rebalance completes?
чт, 19 сент. 2024 г. в 15:38, Igor Fedotov <igor.fedotov@croit.io>:
please see my comments inline.
On 9/19/2024 1:53 PM, Александр Руденко wrote:
Igor, thanks!
> What are the numbers today?
Today we have the same "oldest_map": 2408326 and "newest_map": 2637838, *+2191*.
ceph-objectstore-tool --op meta-list --data-path /var/lib/ceph/osd/ceph-70 | grep osdmap | wc -l 458994
Can you clarify this, please:
> and then multiply by amount of OSDs to learn the minimal space taken by this data
458994 * 4k * OSDs count = "_size of osdmaps on *ONE* OSD_" or "_total size of osdmaps on *ALL* OSDs_" ?
Yes, this is a lower bound estimation for osdmap size on all OSDs.
Because we have about 3k OSDS and 458994 * 4k * 3000 = ~5TB and it can be placed on ONE OSD. But if it is TOTAL osdmap size, I think it is a very small size per OSD.
Highly likely that osdmap for 3K OSDs takes much more than 4K on disk. So again that was just lower bound estimation.
In fact one can use 'ceph osd getmap >out.dat' and get better estimation of osdmap size. So please substitute 4K in the formala above to get better estimation for the overall space taken.
It's a bit simplified though since just half of the entries in 'meta' pool are full osdmaps. Hence you might want to use 458994/2 * sizeof(osdmap) + 458994/2 * 4K in the above formula.
Which is again a sort of low bound estimation but with a better accuracy.
But we have a lot of osds with min_alloc_size=64k which was default in previous ceph's versions for rotational drives (all our SSDs behind old RAID controllers).
ceph daemon osd.10 bluestore allocator dump block | head -10 { "capacity": 479557844992, "alloc_unit": 65536,
But even with min_alloc=64k it will not be a big amount of data 458994 * 64k = *~23GB*. I think we have about *150GB+* extra per SSD OSDs.
Yeah, you should use 64K instead of 4K for the above formula if you have the majority of OSDs using 64K alloc unit. Or take this into account somehow else (e.g. take half 4K and half 64K). But I'm leaving this as a "home excercise" for yourself. The main point here is that a single object would take at least alloc_unit size. And hence I was trying to make the assessment without knowing actual osdmap size but using alloc unit one. Just to check ifwe get numbers of the same order of magnitude. And 23GB and 150GB aren't THAT differ - having e.g. 1M osdmap might easily do the trick. I.e. the osdmap leak indeed could be a real factor here. And hence it's worth additional investigation.
Anyway - please use the obtained osdmap size. It could adjust the resulting estimation value dramatically.
For example, SSD with min_alloc=4k: ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 126 ssd 0.00005 1.00000 447 GiB 374 GiB 300 GiB 72 GiB 1.4 GiB 73 GiB 83.64 1.00 137 up
with min_alloc=64k: ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 10 ssd 0.00005 0.75000 447 GiB 405 GiB 320 GiB 83 GiB 1.4 GiB 42 GiB 90.59 1.00 114 up
Diff is not as big as 4k vs 64k..
Right. Don't know the reason atm. May be leaking osdmaps is not the only iissue. Please do the corrected math as per above though..
чт, 19 сент. 2024 г. в 12:33, Igor Fedotov <igor.fedotov@croit.io>:
Hi Konstantin,
osd_target_transaction_size should control that.
I've heard of it being raized to 150 with no obvious issues. Going beyond is at your own risk. So I'd suggest to apply incremental increase if needed.
Thanks,
Igor
On 9/19/2024 10:44 AM, Konstantin Shalygin wrote:
Hi Igor,
On 18 Sep 2024, at 18:22, Igor Fedotov <igor.fedotov@croit.io> <mailto:igor.fedotov@croit.io> wrote:
I recall a couple of cases when permanent osdmap epoch growth has been filling OSD with relevant osd map info. Which could be tricky to catch.
Please run 'ceph tell osd.N status" for a couple of affected OSDs twice within e.g. 10 min interval.
Then check the delta between oldest_map and newest_map fields - neither the delta should be very large (hundreds of thousands) nor it should grow rapidly within the observed interval.
Side question by topic. What is option controls how much maps to prune? Currently I need to trim 1M osdmaps, but when new map issued, only 30 old maps are removed. What option controls value=30?
Thanks, k
-- Igor Fedotov Ceph Lead Developer
Looking for help with your Ceph cluster? Contact us athttps://croit.io
croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web:https://croit.io | YouTube:https://goo.gl/PGE1Bx
-- Igor Fedotov Ceph Lead Developer
Looking for help with your Ceph cluster? Contact us athttps://croit.io
croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web:https://croit.io | YouTube:https://goo.gl/PGE1Bx
-- Igor Fedotov Ceph Lead Developer
Looking for help with your Ceph cluster? Contact us athttps://croit.io
croit GmbH, Freseniusstr. 31h, 81247 Munich CEO: Martin Verges - VAT-ID: DE310638492 Com. register: Amtsgericht Munich HRB 231263 Web:https://croit.io | YouTube:https://goo.gl/PGE1Bx _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi,
On 19 Sep 2024, at 18:26, Joshua Baergen <jbaergen@digitalocean.com> wrote:
Whenever we've seen osdmaps not being trimmed, we've made sure that any down OSDs are out+destroyed, and then have rolled a restart through the mons. As of recent Pacific at least this seems to have reliably gotten us out of this situation.
This helps for a years. For the mons. But sometimes not for the OSD Alex, I suggest to you - use pgremapper for cancel-backfill. Then see for OSD status - maps is prune or not In case, when everything is clean, but maps are not prune - I issued new osdmaps via setting and then unsetting osd flags (noout, for example). After each new map, by default OSD should prune 30 old osdmaps P.S.: Am I right in understanding that the OSD deletes the old osdmaps (prune) only if a new one is issued (as trigger)? k
Ah, yes, that's a good point - if there's backfill going on then buildup like this can happen. On Thu, Sep 19, 2024 at 10:08 AM Konstantin Shalygin <k0ste@k0ste.ru> wrote:
Hi,
On 19 Sep 2024, at 18:26, Joshua Baergen <jbaergen@digitalocean.com> wrote:
Whenever we've seen osdmaps not being trimmed, we've made sure that any down OSDs are out+destroyed, and then have rolled a restart through the mons. As of recent Pacific at least this seems to have reliably gotten us out of this situation.
This helps for a years. For the mons. But sometimes not for the OSD Alex, I suggest to you - use pgremapper for cancel-backfill. Then see for OSD status - maps is prune or not In case, when everything is clean, but maps are not prune - I issued new osdmaps via setting and then unsetting osd flags (noout, for example). After each new map, by default OSD should prune 30 old osdmaps
P.S.: Am I right in understanding that the OSD deletes the old osdmaps (prune) only if a new one is issued (as trigger)?
k
Hi, guys, thanks to all! We have completed our long rebalance and restarted all MONs and each OSD freed up approximately 180Gb. And after a few days oldest_map almost equals newest_map. чт, 19 сент. 2024 г. в 19:26, Joshua Baergen <jbaergen@digitalocean.com>:
Ah, yes, that's a good point - if there's backfill going on then buildup like this can happen.
On Thu, Sep 19, 2024 at 10:08 AM Konstantin Shalygin <k0ste@k0ste.ru> wrote:
Hi,
On 19 Sep 2024, at 18:26, Joshua Baergen <jbaergen@digitalocean.com>
wrote:
Whenever we've seen osdmaps not being trimmed, we've made sure that any down OSDs are out+destroyed, and then have rolled a restart through the mons. As of recent Pacific at least this seems to have reliably gotten us out of this situation.
This helps for a years. For the mons. But sometimes not for the OSD Alex, I suggest to you - use pgremapper for cancel-backfill. Then see
for OSD status - maps is prune or not
In case, when everything is clean, but maps are not prune - I issued new osdmaps via setting and then unsetting osd flags (noout, for example). After each new map, by default OSD should prune 30 old osdmaps
P.S.: Am I right in understanding that the OSD deletes the old osdmaps (prune) only if a new one is issued (as trigger)?
k
participants (4)
-
Igor Fedotov
-
Joshua Baergen
-
Konstantin Shalygin
-
Александр Руденко