31 May
2021
31 May
'21
8:25 a.m.
Hi,
Any way to clean up large-omap in the index pool?
PG deep_scrub didn't help.
I know how to clean in the log pool, but no idea in the index pool :/
It's an octopus deployment 15.2.10.
Thank you
________________________________
This message is confidential and is for the sole use of the intended recipient(s). It may
also be privileged or otherwise protected by copyright or other legal rules. If you have
received it by mistake please let us know by reply email and delete it from your system.
It is prohibited to copy this message or disclose its content to anyone. Any
confidentiality or privilege is not waived or lost by any mistaken delivery or
unauthorized disclosure of the message. All messages sent to and from Agoda may be
monitored to ensure compliance with company policies, to protect the company's
interests and to remove potential malware. Electronic messages may be intercepted,
amended, lost or deleted, or contain viruses.
31 May
31 May
4:42 p.m.
All the index data will be in OMAP, which you can see a listing of with
`ceph osd df tree`
Do you have large buckets (many, many objects in a single bucket) with
few shards? You may have to reshard one (or some) of your buckets.
It'll take some reading if you're using multisite, in order to
coordinate it (though I'm unfamiliar with how it works with multisite in
Octopus).
On 2021-05-31 02:25, Szabo, Istvan (Agoda) wrote:
> Hi,
>
> Any way to clean up large-omap in the index pool?
> PG deep_scrub didn't help.
> I know how to clean in the log pool, but no idea in the index pool :/
> It's an octopus deployment 15.2.10.
>
> Thank you
>
> ________________________________
> This message is confidential and is for the sole use of the intended
> recipient(s). It may also be privileged or otherwise protected by
> copyright or other legal rules. If you have received it by mistake
> please let us know by reply email and delete it from your system. It
> is prohibited to copy this message or disclose its content to anyone.
> Any confidentiality or privilege is not waived or lost by any mistaken
> delivery or unauthorized disclosure of the message. All messages sent
> to and from Agoda may be monitored to ensure compliance with company
> policies, to protect the company's interests and to remove potential
> malware. Electronic messages may be intercepted, amended, lost or
> deleted, or contain viruses.
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
5:52 p.m.
New subject: [Suspicious newsletter] Re: The always welcomed large omap
Yeah, I found a bucket at the moment in progress the deleting, will preshard it.
Istvan Szabo
Senior Infrastructure Engineer
---------------------------------------------------
Agoda Services Co., Ltd.
e: istvan.szabo(a)agoda.com
---------------------------------------------------
-----Original Message-----
From: Matt Vandermeulen <storage(a)reenigne.net>
Sent: Monday, May 31, 2021 9:43 PM
To: Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com>
Cc: ceph-users(a)ceph.io
Subject: [Suspicious newsletter] [ceph-users] Re: The always welcomed large omap
All the index data will be in OMAP, which you can see a listing of with
`ceph osd df tree`
Do you have large buckets (many, many objects in a single bucket) with few shards? You
may have to reshard one (or some) of your buckets.
It'll take some reading if you're using multisite, in order to coordinate it
(though I'm unfamiliar with how it works with multisite in Octopus).
On 2021-05-31 02:25, Szabo, Istvan (Agoda) wrote:
Hi,
Any way to clean up large-omap in the index pool?
PG deep_scrub didn't help.
I know how to clean in the log pool, but no idea in the index pool :/
It's an octopus deployment 15.2.10.
Thank you
________________________________
This message is confidential and is for the sole use of the intended
recipient(s). It may also be privileged or otherwise protected by
copyright or other legal rules. If you have received it by mistake
please let us know by reply email and delete it from your system. It
is prohibited to copy this message or disclose its content to anyone.
Any confidentiality or privilege is not waived or lost by any mistaken
delivery or unauthorized disclosure of the message. All messages sent
to and from Agoda may be monitored to ensure compliance with company
policies, to protect the company's interests and to remove potential
malware. Electronic messages may be intercepted, amended, lost or
deleted, or contain viruses.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an
email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to
ceph-users-leave(a)ceph.io
________________________________
This message is confidential and is for the sole use of the intended recipient(s). It may
also be privileged or otherwise protected by copyright or other legal rules. If you have
received it by mistake please let us know by reply email and delete it from your system.
It is prohibited to copy this message or disclose its content to anyone. Any
confidentiality or privilege is not waived or lost by any mistaken delivery or
unauthorized disclosure of the message. All messages sent to and from Agoda may be
monitored to ensure compliance with company policies, to protect the company's
interests and to remove potential malware. Electronic messages may be intercepted,
amended, lost or deleted, or contain viruses.
1 Jun
1 Jun
4:27 a.m.
New subject: [Suspicious newsletter] Re: The always welcomed large omap
Here is the command output:
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL
%USE VAR PGS STATUS TYPE NAME
-1 530.89032 - 531 TiB 29 TiB 11 TiB 31 GiB 338 GiB 502 TiB
5.43 1.00 - root default
-5 85.57072 - 86 TiB 1.9 TiB 1.9 TiB 5.5 GiB 58 GiB 84 TiB
2.26 0.42 - host server-6s01
36 nvme 1.74660 1.00000 1.7 TiB 6.0 GiB 17 MiB 5.4 GiB 564 MiB 1.7 TiB
0.33 0.06 31 up osd.36
0 ssd 13.97069 1.00000 14 TiB 248 GiB 241 GiB 23 MiB 7.6 GiB 14 TiB
1.74 0.32 19 up osd.0
8 ssd 13.97069 1.00000 14 TiB 616 GiB 600 GiB 25 MiB 16 GiB 13 TiB
4.31 0.79 23 up osd.8
15 ssd 13.97069 1.00000 14 TiB 246 GiB 239 GiB 2.4 MiB 7.3 GiB 14 TiB
1.72 0.32 12 up osd.15
18 ssd 13.97069 1.00000 14 TiB 247 GiB 240 GiB 8.4 MiB 6.9 GiB 14 TiB
1.73 0.32 15 up osd.18
24 ssd 13.97069 1.00000 14 TiB 309 GiB 299 GiB 1014 KiB 9.8 GiB 14 TiB
2.16 0.40 10 up osd.24
30 ssd 13.97069 1.00000 14 TiB 309 GiB 299 GiB 15 MiB 9.8 GiB 14 TiB
2.16 0.40 16 up osd.30
-3 89.06392 - 89 TiB 5.4 TiB 1.9 TiB 6.7 GiB 56 GiB 84 TiB
6.04 1.11 - host server-6s02
37 nvme 1.74660 1.00000 1.7 TiB 7.1 GiB 18 MiB 6.6 GiB 544 MiB 1.7 TiB
0.40 0.07 39 up osd.37
1 ssd 14.55289 1.00000 15 TiB 1017 GiB 420 GiB 36 MiB 13 GiB 14 TiB
6.82 1.26 18 up osd.1
11 ssd 14.55289 1.00000 15 TiB 1.1 TiB 480 GiB 63 MiB 13 GiB 14 TiB
7.23 1.33 18 up osd.11
17 ssd 14.55289 1.00000 15 TiB 897 GiB 300 GiB 26 MiB 8.5 GiB 14 TiB
6.02 1.11 17 up osd.17
23 ssd 14.55289 1.00000 15 TiB 957 GiB 360 GiB 13 MiB 10 GiB 14 TiB
6.42 1.18 13 up osd.23
28 ssd 14.55289 1.00000 15 TiB 896 GiB 299 GiB 27 MiB 8.5 GiB 14 TiB
6.01 1.11 23 up osd.28
35 ssd 14.55289 1.00000 15 TiB 657 GiB 60 GiB 14 MiB 2.1 GiB 14 TiB
4.41 0.81 16 up osd.35
-11 89.06392 - 89 TiB 5.4 TiB 1.9 TiB 5.4 GiB 56 GiB 84 TiB
6.04 1.11 - host server-6s03
41 nvme 1.74660 1.00000 1.7 TiB 5.9 GiB 18 MiB 5.2 GiB 675 MiB 1.7 TiB
0.33 0.06 33 up osd.41
2 ssd 14.55289 1.00000 15 TiB 957 GiB 360 GiB 53 MiB 10 GiB 14 TiB
6.42 1.18 22 up osd.2
6 ssd 14.55289 1.00000 15 TiB 716 GiB 119 GiB 34 MiB 3.8 GiB 14 TiB
4.81 0.89 15 up osd.6
13 ssd 14.55289 1.00000 15 TiB 896 GiB 299 GiB 7.0 MiB 8.5 GiB 14 TiB
6.01 1.11 11 up osd.13
19 ssd 14.55289 1.00000 15 TiB 777 GiB 180 GiB 38 MiB 5.4 GiB 14 TiB
5.21 0.96 13 up osd.19
26 ssd 14.55289 1.00000 15 TiB 837 GiB 240 GiB 11 MiB 7.5 GiB 14 TiB
5.62 1.03 13 up osd.26
32 ssd 14.55289 1.00000 15 TiB 1.3 TiB 720 GiB 23 MiB 20 GiB 13 TiB
8.84 1.63 23 up osd.32
-13 89.06392 - 89 TiB 5.4 TiB 1.9 TiB 4.8 GiB 56 GiB 84 TiB
6.04 1.11 - host server-6s04
38 nvme 1.74660 1.00000 1.7 TiB 5.2 GiB 17 MiB 4.7 GiB 568 MiB 1.7 TiB
0.29 0.05 26 up osd.38
5 ssd 14.55289 1.00000 15 TiB 776 GiB 179 GiB 21 MiB 5.9 GiB 14 TiB
5.21 0.96 14 up osd.5
7 ssd 14.55289 1.00000 15 TiB 897 GiB 300 GiB 53 MiB 8.5 GiB 14 TiB
6.02 1.11 19 up osd.7
14 ssd 14.55289 1.00000 15 TiB 897 GiB 299 GiB 3.1 MiB 8.6 GiB 14 TiB
6.02 1.11 11 up osd.14
20 ssd 14.55289 1.00000 15 TiB 1017 GiB 420 GiB 3.5 MiB 12 GiB 14 TiB
6.83 1.26 13 up osd.20
25 ssd 14.55289 1.00000 15 TiB 896 GiB 299 GiB 16 MiB 9.0 GiB 14 TiB
6.01 1.11 18 up osd.25
31 ssd 14.55289 1.00000 15 TiB 1018 GiB 421 GiB 42 MiB 12 GiB 14 TiB
6.83 1.26 17 up osd.31
-9 89.06392 - 89 TiB 5.4 TiB 1.9 TiB 2.9 GiB 56 GiB 84 TiB
6.04 1.11 - host server-6s05
40 nvme 1.74660 1.00000 1.7 TiB 3.5 GiB 18 MiB 2.8 GiB 745 MiB 1.7 TiB
0.20 0.04 29 up osd.40
3 ssd 14.55289 1.00000 15 TiB 837 GiB 240 GiB 37 MiB 7.0 GiB 14 TiB
5.62 1.03 10 up osd.3
10 ssd 14.55289 1.00000 15 TiB 957 GiB 360 GiB 22 MiB 10 GiB 14 TiB
6.42 1.18 23 up osd.10
12 ssd 14.55289 1.00000 15 TiB 896 GiB 299 GiB 9.9 MiB 8.6 GiB 14 TiB
6.01 1.11 12 up osd.12
21 ssd 14.55289 1.00000 15 TiB 1.2 TiB 659 GiB 19 MiB 18 GiB 13 TiB
8.43 1.55 23 up osd.21
29 ssd 14.55289 1.00000 15 TiB 776 GiB 179 GiB 25 MiB 6.2 GiB 14 TiB
5.21 0.96 16 up osd.29
33 ssd 14.55289 1.00000 15 TiB 778 GiB 181 GiB 16 MiB 5.4 GiB 14 TiB
5.22 0.96 10 up osd.33
-7 89.06392 - 89 TiB 5.4 TiB 1.9 TiB 5.5 GiB 56 GiB 84 TiB
6.04 1.11 - host server-6s06
39 nvme 1.74660 1.00000 1.7 TiB 5.9 GiB 18 MiB 5.4 GiB 497 MiB 1.7 TiB
0.33 0.06 34 up osd.39
4 ssd 14.55289 1.00000 15 TiB 717 GiB 120 GiB 10 MiB 3.8 GiB 14 TiB
4.81 0.89 19 up osd.4
9 ssd 14.55289 1.00000 15 TiB 897 GiB 299 GiB 21 MiB 8.5 GiB 14 TiB
6.02 1.11 12 up osd.9
16 ssd 14.55289 1.00000 15 TiB 1017 GiB 420 GiB 17 MiB 12 GiB 14 TiB
6.83 1.26 13 up osd.16
22 ssd 14.55289 1.00000 15 TiB 1.1 TiB 538 GiB 13 MiB 16 GiB 13 TiB
7.62 1.40 24 up osd.22
27 ssd 14.55289 1.00000 15 TiB 778 GiB 180 GiB 18 MiB 5.4 GiB 14 TiB
5.22 0.96 11 up osd.27
34 ssd 14.55289 1.00000 15 TiB 957 GiB 360 GiB 16 MiB 10 GiB 14 TiB
6.42 1.18 17 up osd.34
TOTAL 531 TiB 29 TiB 11 TiB 31 GiB 338 GiB 502 TiB
5.43
MIN/MAX VAR: 0.04/1.63 STDDEV: 2.52
Istvan Szabo
Senior Infrastructure Engineer
---------------------------------------------------
Agoda Services Co., Ltd.
e: istvan.szabo(a)agoda.com
---------------------------------------------------
-----Original Message-----
From: Matt Vandermeulen <storage(a)reenigne.net>
Sent: Monday, May 31, 2021 9:43 PM
To: Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com>
Cc: ceph-users(a)ceph.io
Subject: [Suspicious newsletter] [ceph-users] Re: The always welcomed large omap
All the index data will be in OMAP, which you can see a listing of with
`ceph osd df tree`
Do you have large buckets (many, many objects in a single bucket) with few shards? You
may have to reshard one (or some) of your buckets.
It'll take some reading if you're using multisite, in order to coordinate it
(though I'm unfamiliar with how it works with multisite in Octopus).
On 2021-05-31 02:25, Szabo, Istvan (Agoda) wrote:
Hi,
Any way to clean up large-omap in the index pool?
PG deep_scrub didn't help.
I know how to clean in the log pool, but no idea in the index pool :/
It's an octopus deployment 15.2.10.
Thank you
________________________________
This message is confidential and is for the sole use of the intended
recipient(s). It may also be privileged or otherwise protected by
copyright or other legal rules. If you have received it by mistake
please let us know by reply email and delete it from your system. It
is prohibited to copy this message or disclose its content to anyone.
Any confidentiality or privilege is not waived or lost by any mistaken
delivery or unauthorized disclosure of the message. All messages sent
to and from Agoda may be monitored to ensure compliance with company
policies, to protect the company's interests and to remove potential
malware. Electronic messages may be intercepted, amended, lost or
deleted, or contain viruses.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an
email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to
ceph-users-leave(a)ceph.io
________________________________
This message is confidential and is for the sole use of the intended recipient(s). It may
also be privileged or otherwise protected by copyright or other legal rules. If you have
received it by mistake please let us know by reply email and delete it from your system.
It is prohibited to copy this message or disclose its content to anyone. Any
confidentiality or privilege is not waived or lost by any mistaken delivery or
unauthorized disclosure of the message. All messages sent to and from Agoda may be
monitored to ensure compliance with company policies, to protect the company's
interests and to remove potential malware. Electronic messages may be intercepted,
amended, lost or deleted, or contain viruses.
1054
days inactive
1055
days old
3 comments
2 participants
participants (2)
-
Matt Vandermeulen -
Szabo, Istvan (Agoda)