14 May
2021
14 May
'21
5:20 a.m.
Hi,
I had issue with the snaptrim after a hug amount of deleted data, it slows down the team
operations due to the snaptrim and snaptrim_wait pgs.
I've changed couple of things:
debug_ms = 0/0 #default 0/5
osd_snap_trim_priority = 1 # default 5
osd_pg_max_concurrent_snap_trims = 1 # default 2
But didn't help.
I've found this thread about buffered io and seems like it helped to them:
https://forum.proxmox.com/threads/ceph-storage-all-pgs-snaptrim-every-night…
I don't use swap on the OSD nodes, so I gave a try on 1 osd node and it caused
basically the complete node's pg-s are degraded. Is it normal? I hope it will not
rebalance the complete node because I don't have space for that. I changed it back but
still slowly decreasing, so not sure this settings is correct or not or this behavior is
good or not?
2021-05-14 12:18:11.447628 mon.2004 [WRN] Health check update: 3353/91976715 objects
misplaced (0.004%) (OBJECT_MISPLACED)
2021-05-14 12:18:11.447640 mon.2004 [WRN] Health check update: Degraded data redundancy:
33078466/91976715 objects degraded (35.964%), 254 pgs degraded, 253 pgs undersized
(PG_DEGRADED)
Istvan Szabo
Senior Infrastructure Engineer
---------------------------------------------------
Agoda Services Co., Ltd.
e: istvan.szabo@agoda.com<mailto:istvan.szabo@agoda.com>
---------------------------------------------------
________________________________
This message is confidential and is for the sole use of the intended recipient(s). It may
also be privileged or otherwise protected by copyright or other legal rules. If you have
received it by mistake please let us know by reply email and delete it from your system.
It is prohibited to copy this message or disclose its content to anyone. Any
confidentiality or privilege is not waived or lost by any mistaken delivery or
unauthorized disclosure of the message. All messages sent to and from Agoda may be
monitored to ensure compliance with company policies, to protect the company's
interests and to remove potential malware. Electronic messages may be intercepted,
amended, lost or deleted, or contain viruses.
14 May
14 May
6:11 a.m.
Hi,
This is not a normal, It's something different I think, like a crush changes on
restart. This option will be enabled by default again in Nautilus next, so you can use it
now with 14.2.19-20
k
Sent from my iPhone
On 14 May 2021, at 08:21, Szabo, Istvan (Agoda)
<Istvan.Szabo(a)agoda.com> wrote:
Hi,
I had issue with the snaptrim after a hug amount of deleted data, it slows down the team
operations due to the snaptrim and snaptrim_wait pgs.
I've changed couple of things:
debug_ms = 0/0 #default 0/5
osd_snap_trim_priority = 1 # default 5
osd_pg_max_concurrent_snap_trims = 1 # default 2
But didn't help.
I've found this thread about buffered io and seems like it helped to them:
https://forum.proxmox.com/threads/ceph-storage-all-pgs-snaptrim-every-night…
I don't use swap on the OSD nodes, so I gave a try on 1 osd node and it caused
basically the complete node's pg-s are degraded. Is it normal? I hope it will not
rebalance the complete node because I don't have space for that. I changed it back but
still slowly decreasing, so not sure this settings is correct or not or this behavior is
good or not?
2021-05-14 12:18:11.447628 mon.2004 [WRN] Health check update: 3353/91976715 objects
misplaced (0.004%) (OBJECT_MISPLACED)
2021-05-14 12:18:11.447640 mon.2004 [WRN] Health check update: Degraded data redundancy:
33078466/91976715 objects degraded (35.964%), 254 pgs degraded, 253 pgs undersized
(PG_DEGRADED)
Istvan Szabo
Senior Infrastructure Engineer
---------------------------------------------------
Agoda Services Co., Ltd.
e: istvan.szabo@agoda.com<mailto:istvan.szabo@agoda.com>
---------------------------------------------------
________________________________
This message is confidential and is for the sole use of the intended recipient(s). It may
also be privileged or otherwise protected by copyright or other legal rules. If you have
received it by mistake please let us know by reply email and delete it from your system.
It is prohibited to copy this message or disclose its content to anyone. Any
confidentiality or privilege is not waived or lost by any mistaken delivery or
unauthorized disclosure of the message. All messages sent to and from Agoda may be
monitored to ensure compliance with company policies, to protect the company's
interests and to remove potential malware. Electronic messages may be intercepted,
amended, lost or deleted, or contain viruses.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
6:22 a.m.
New subject: [Suspicious newsletter] Re: bluefs_buffered_io turn to true
Hi,
It is quite an older cluster, luminous 12.2.8.
Istvan Szabo
Senior Infrastructure Engineer
---------------------------------------------------
Agoda Services Co., Ltd.
e: istvan.szabo(a)agoda.com
---------------------------------------------------
-----Original Message-----
From: Konstantin Shalygin <k0ste(a)k0ste.ru>
Sent: Friday, May 14, 2021 1:12 PM
To: Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com>
Cc: ceph-users(a)ceph.io
Subject: [Suspicious newsletter] [ceph-users] Re: bluefs_buffered_io turn to true
Hi,
This is not a normal, It's something different I think, like a crush changes on
restart. This option will be enabled by default again in Nautilus next, so you can use it
now with 14.2.19-20
k
Sent from my iPhone
On 14 May 2021, at 08:21, Szabo, Istvan (Agoda)
<Istvan.Szabo(a)agoda.com> wrote:
Hi,
I had issue with the snaptrim after a hug amount of deleted data, it slows down the team
operations due to the snaptrim and snaptrim_wait pgs.
I've changed couple of things:
debug_ms = 0/0 #default 0/5
osd_snap_trim_priority = 1 # default 5
osd_pg_max_concurrent_snap_trims = 1 # default 2
But didn't help.
I've found this thread about buffered io and seems like it helped to them:
https://forum.proxmox.com/threads/ceph-storage-all-pgs-snaptrim-every-
night-slowing-down-vms.71573/
I don't use swap on the OSD nodes, so I gave a try on 1 osd node and it caused
basically the complete node's pg-s are degraded. Is it normal? I hope it will not
rebalance the complete node because I don't have space for that. I changed it back but
still slowly decreasing, so not sure this settings is correct or not or this behavior is
good or not?
2021-05-14 12:18:11.447628 mon.2004 [WRN] Health check update:
3353/91976715 objects misplaced (0.004%) (OBJECT_MISPLACED)
2021-05-14 12:18:11.447640 mon.2004 [WRN] Health check update:
Degraded data redundancy: 33078466/91976715 objects degraded
(35.964%), 254 pgs degraded, 253 pgs undersized (PG_DEGRADED)
Istvan Szabo
Senior Infrastructure Engineer
---------------------------------------------------
Agoda Services Co., Ltd.
e: istvan.szabo@agoda.com<mailto:istvan.szabo@agoda.com>
---------------------------------------------------
________________________________
This message is confidential and is for the sole use of the intended recipient(s). It may
also be privileged or otherwise protected by copyright or other legal rules. If you have
received it by mistake please let us know by reply email and delete it from your system.
It is prohibited to copy this message or disclose its content to anyone. Any
confidentiality or privilege is not waived or lost by any mistaken delivery or
unauthorized disclosure of the message. All messages sent to and from Agoda may be
monitored to ensure compliance with company policies, to protect the company's
interests and to remove potential malware. Electronic messages may be intercepted,
amended, lost or deleted, or contain viruses.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an
email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to
ceph-users-leave(a)ceph.io
7:12 a.m.
New subject: [Suspicious newsletter] Re: bluefs_buffered_io turn to true
I recommend to upgrade at least to 12.2.13, for luminous even .12 and .13 is significant
difference in code.
k
On 14 May 2021, at 09:22, Szabo, Istvan (Agoda)
<Istvan.Szabo(a)agoda.com> wrote:
It is quite an older cluster, luminous 12.2.8.
7:50 a.m.
New subject: [Suspicious newsletter] Re: bluefs_buffered_io turn to true
Is it also normal if this buffered_ioturned on, it eats all the memory on the system?
Hmmm.
Istvan Szabo
Senior Infrastructure Engineer
---------------------------------------------------
Agoda Services Co., Ltd.
e: istvan.szabo@agoda.com<mailto:istvan.szabo@agoda.com>
---------------------------------------------------
From: Konstantin Shalygin <k0ste(a)k0ste.ru>
Sent: Friday, May 14, 2021 2:12 PM
To: Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com>
Cc: ceph-users(a)ceph.io
Subject: Re: [Suspicious newsletter] [ceph-users] Re: bluefs_buffered_io turn to true
I recommend to upgrade at least to 12.2.13, for luminous even .12 and .13 is significant
difference in code.
k
On 14 May 2021, at 09:22, Szabo, Istvan (Agoda)
<Istvan.Szabo@agoda.com<mailto:Istvan.Szabo@agoda.com>> wrote:
It is quite an older cluster, luminous 12.2.8.
________________________________
This message is confidential and is for the sole use of the intended recipient(s). It may
also be privileged or otherwise protected by copyright or other legal rules. If you have
received it by mistake please let us know by reply email and delete it from your system.
It is prohibited to copy this message or disclose its content to anyone. Any
confidentiality or privilege is not waived or lost by any mistaken delivery or
unauthorized disclosure of the message. All messages sent to and from Agoda may be
monitored to ensure compliance with company policies, to protect the company's
interests and to remove potential malware. Electronic messages may be intercepted,
amended, lost or deleted, or contain viruses.
7:59 a.m.
New subject: [Suspicious newsletter] Re: bluefs_buffered_io turn to true
On 14 May 2021, at 10:50, Szabo, Istvan (Agoda)
<Istvan.Szabo(a)agoda.com> wrote:
Is it also normal if this buffered_ioturned on, it eats all the memory on the system?
Hmmm.
This is what actually do this option - eat all free memory as cached for bluefs
speedups
k
8:20 a.m.
New subject: [Suspicious newsletter] Re: bluefs_buffered_io turn to true
When this stop 😃 ? When died … :D
Istvan Szabo
Senior Infrastructure Engineer
---------------------------------------------------
Agoda Services Co., Ltd.
e: istvan.szabo@agoda.com<mailto:istvan.szabo@agoda.com>
---------------------------------------------------
From: Konstantin Shalygin <k0ste(a)k0ste.ru>
Sent: Friday, May 14, 2021 3:00 PM
To: Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com>
Cc: ceph-users(a)ceph.io
Subject: Re: [Suspicious newsletter] [ceph-users] Re: bluefs_buffered_io turn to true
On 14 May 2021, at 10:50, Szabo, Istvan (Agoda)
<Istvan.Szabo@agoda.com<mailto:Istvan.Szabo@agoda.com>> wrote:
Is it also normal if this buffered_ioturned on, it eats all the memory on the system?
Hmmm.
This is what actually do this option - eat all free memory as cached for bluefs speedups
k
________________________________
This message is confidential and is for the sole use of the intended recipient(s). It may
also be privileged or otherwise protected by copyright or other legal rules. If you have
received it by mistake please let us know by reply email and delete it from your system.
It is prohibited to copy this message or disclose its content to anyone. Any
confidentiality or privilege is not waived or lost by any mistaken delivery or
unauthorized disclosure of the message. All messages sent to and from Agoda may be
monitored to ensure compliance with company policies, to protect the company's
interests and to remove potential malware. Electronic messages may be intercepted,
amended, lost or deleted, or contain viruses.
8:26 a.m.
New subject: [Suspicious newsletter] Re: bluefs_buffered_io turn to true
Nope, kernel reserves enough memory to free on pressure, for example 36OSD 0.5TiB RAM
host:
total used free shared buff/cache available
Mem: 502G 168G 2.9G 18M 331G 472G
Swap: 952M 248M 704M
k
On 14 May 2021, at 11:20, Szabo, Istvan (Agoda)
<Istvan.Szabo(a)agoda.com> wrote:
When this stop 😃 ? When died … :D
9:51 a.m.
New subject: [Suspicious newsletter] Re: bluefs_buffered_io turn to true
Ok, seems like it doesn’t go below 600MB out of the 256GB, let’s wait until the
pg_degradation healed.
Did I do something wrong?
I set in the global config the bluefs option, and restarted ceph.target on the osd node :/
?
Doe this need some special thing to apply?
Istvan Szabo
Senior Infrastructure Engineer
---------------------------------------------------
Agoda Services Co., Ltd.
e: istvan.szabo@agoda.com<mailto:istvan.szabo@agoda.com>
---------------------------------------------------
From: Konstantin Shalygin <k0ste(a)k0ste.ru>
Sent: Friday, May 14, 2021 3:26 PM
To: Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com>
Cc: ceph-users(a)ceph.io
Subject: Re: [ceph-users] [Suspicious newsletter] Re: bluefs_buffered_io turn to true
Nope, kernel reserves enough memory to free on pressure, for example 36OSD 0.5TiB RAM
host:
total used free shared buff/cache available
Mem: 502G 168G 2.9G 18M 331G 472G
Swap: 952M 248M 704M
k
On 14 May 2021, at 11:20, Szabo, Istvan (Agoda)
<Istvan.Szabo@agoda.com<mailto:Istvan.Szabo@agoda.com>> wrote:
When this stop 😃 ? When died … :D
________________________________
This message is confidential and is for the sole use of the intended recipient(s). It may
also be privileged or otherwise protected by copyright or other legal rules. If you have
received it by mistake please let us know by reply email and delete it from your system.
It is prohibited to copy this message or disclose its content to anyone. Any
confidentiality or privilege is not waived or lost by any mistaken delivery or
unauthorized disclosure of the message. All messages sent to and from Agoda may be
monitored to ensure compliance with company policies, to protect the company's
interests and to remove potential malware. Electronic messages may be intercepted,
amended, lost or deleted, or contain viruses.
10:32 a.m.
New subject: [Suspicious newsletter] Re: bluefs_buffered_io turn to true
It's enough, should be true now...
k
On 14 May 2021, at 12:51, Szabo, Istvan (Agoda)
<Istvan.Szabo(a)agoda.com> wrote:
Did I do something wrong?
I set in the global config the bluefs option, and restarted ceph.target on the osd node
:/ ?
Doe this need some special thing to apply?
10:45 a.m.
New subject: [Suspicious newsletter] Re: bluefs_buffered_io turn to true
Is there anything that should be set just to be sure oom kill not happen? Or nothing?
Istvan Szabo
Senior Infrastructure Engineer
---------------------------------------------------
Agoda Services Co., Ltd.
e: istvan.szabo@agoda.com<mailto:istvan.szabo@agoda.com>
---------------------------------------------------
From: Konstantin Shalygin <k0ste(a)k0ste.ru>
Sent: Friday, May 14, 2021 5:32 PM
To: Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com>
Cc: ceph-users(a)ceph.io
Subject: Re: [ceph-users] [Suspicious newsletter] Re: bluefs_buffered_io turn to true
It's enough, should be true now...
k
On 14 May 2021, at 12:51, Szabo, Istvan (Agoda)
<Istvan.Szabo@agoda.com<mailto:Istvan.Szabo@agoda.com>> wrote:
Did I do something wrong?
I set in the global config the bluefs option, and restarted ceph.target on the osd node :/
?
Doe this need some special thing to apply?
________________________________
This message is confidential and is for the sole use of the intended recipient(s). It may
also be privileged or otherwise protected by copyright or other legal rules. If you have
received it by mistake please let us know by reply email and delete it from your system.
It is prohibited to copy this message or disclose its content to anyone. Any
confidentiality or privilege is not waived or lost by any mistaken delivery or
unauthorized disclosure of the message. All messages sent to and from Agoda may be
monitored to ensure compliance with company policies, to protect the company's
interests and to remove potential malware. Electronic messages may be intercepted,
amended, lost or deleted, or contain viruses.
11:07 a.m.
New subject: [Suspicious newsletter] Re: bluefs_buffered_io turn to true
I suggest to look into vm.min_free_kbytes kernel option, and increase it twice
k
On 14 May 2021, at 13:45, Szabo, Istvan (Agoda)
<Istvan.Szabo(a)agoda.com> wrote:
Is there anything that should be set just to be sure oom kill not happen? Or nothing?
11:20 a.m.
New subject: [Suspicious newsletter] Re: bluefs_buffered_io turn to true
Howmuch is yours? Mine is vm.min_free_kbytes = 90112.
Istvan Szabo
Senior Infrastructure Engineer
---------------------------------------------------
Agoda Services Co., Ltd.
e: istvan.szabo@agoda.com<mailto:istvan.szabo@agoda.com>
---------------------------------------------------
From: Konstantin Shalygin <k0ste(a)k0ste.ru>
Sent: Friday, May 14, 2021 6:07 PM
To: Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com>
Cc: ceph-users(a)ceph.io
Subject: Re: [ceph-users] [Suspicious newsletter] Re: bluefs_buffered_io turn to true
I suggest to look into vm.min_free_kbytes kernel option, and increase it twice
k
On 14 May 2021, at 13:45, Szabo, Istvan (Agoda)
<Istvan.Szabo@agoda.com<mailto:Istvan.Szabo@agoda.com>> wrote:
Is there anything that should be set just to be sure oom kill not happen? Or nothing?
________________________________
This message is confidential and is for the sole use of the intended recipient(s). It may
also be privileged or otherwise protected by copyright or other legal rules. If you have
received it by mistake please let us know by reply email and delete it from your system.
It is prohibited to copy this message or disclose its content to anyone. Any
confidentiality or privilege is not waived or lost by any mistaken delivery or
unauthorized disclosure of the message. All messages sent to and from Agoda may be
monitored to ensure compliance with company policies, to protect the company's
interests and to remove potential malware. Electronic messages may be intercepted,
amended, lost or deleted, or contain viruses.
11:27 a.m.
New subject: [Suspicious newsletter] Re: bluefs_buffered_io turn to true
Hi.
https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/2/html/c…
пт, 14 мая 2021 г. в 14:21, Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com>om>:
Howmuch is yours? Mine is vm.min_free_kbytes = 90112.
Istvan Szabo
Senior Infrastructure Engineer
---------------------------------------------------
Agoda Services Co., Ltd.
e: istvan.szabo@agoda.com<mailto:istvan.szabo@agoda.com>
---------------------------------------------------
From: Konstantin Shalygin <k0ste(a)k0ste.ru>
Sent: Friday, May 14, 2021 6:07 PM
To: Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com>
Cc: ceph-users(a)ceph.io
Subject: Re: [ceph-users] [Suspicious newsletter] Re: bluefs_buffered_io
turn to true
I suggest to look into vm.min_free_kbytes kernel option, and increase it
twice
k
On 14 May 2021, at 13:45, Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com
<mailto:Istvan.Szabo@agoda.com>> wrote:
Is there anything that should be set just to be sure oom kill not happen?
Or nothing?
________________________________
This message is confidential and is for the sole use of the intended
recipient(s). It may also be privileged or otherwise protected by copyright
or other legal rules. If you have received it by mistake please let us know
by reply email and delete it from your system. It is prohibited to copy
this message or disclose its content to anyone. Any confidentiality or
privilege is not waived or lost by any mistaken delivery or unauthorized
disclosure of the message. All messages sent to and from Agoda may be
monitored to ensure compliance with company policies, to protect the
company's interests and to remove potential malware. Electronic messages
may be intercepted, amended, lost or deleted, or contain viruses.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
11:31 a.m.
New subject: [Suspicious newsletter] Re: bluefs_buffered_io turn to true
Yup, I just saw, should have 3GB :/ I will wait until the system goes back to normal and
will increase.
Istvan Szabo
Senior Infrastructure Engineer
---------------------------------------------------
Agoda Services Co., Ltd.
e: istvan.szabo@agoda.com<mailto:istvan.szabo@agoda.com>
---------------------------------------------------
From: Irek Fasikhov <malmyzh(a)gmail.com>
Sent: Friday, May 14, 2021 6:28 PM
To: Szabo, Istvan (Agoda) <Istvan.Szabo(a)agoda.com>
Cc: Konstantin Shalygin <k0ste(a)k0ste.ru>ru>; ceph-users(a)ceph.io
Subject: Re: [ceph-users] Re: [Suspicious newsletter] Re: bluefs_buffered_io turn to
true
Hi.
https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/2/html/c…
пт, 14 мая 2021 г. в 14:21, Szabo, Istvan (Agoda)
<Istvan.Szabo@agoda.com<mailto:Istvan.Szabo@agoda.com>>:
Howmuch is yours? Mine is vm.min_free_kbytes = 90112.
Istvan Szabo
Senior Infrastructure Engineer
---------------------------------------------------
Agoda Services Co., Ltd.
e:
istvan.szabo@agoda.com<mailto:istvan.szabo@agoda.com><mailto:istvan.szabo@agoda.com<mailto:istvan.szabo@agoda.com>>
---------------------------------------------------
From: Konstantin Shalygin <k0ste@k0ste.ru<mailto:k0ste@k0ste.ru>>
Sent: Friday, May 14, 2021 6:07 PM
To: Szabo, Istvan (Agoda)
<Istvan.Szabo@agoda.com<mailto:Istvan.Szabo@agoda.com>>
Cc: ceph-users@ceph.io<mailto:ceph-users@ceph.io>
Subject: Re: [ceph-users] [Suspicious newsletter] Re: bluefs_buffered_io turn to true
I suggest to look into vm.min_free_kbytes kernel option, and increase it twice
k
On 14 May 2021, at 13:45, Szabo, Istvan (Agoda)
<Istvan.Szabo@agoda.com<mailto:Istvan.Szabo@agoda.com><mailto:Istvan.Szabo@agoda.com<mailto:Istvan.Szabo@agoda.com>>>
wrote:
Is there anything that should be set just to be sure oom kill not happen? Or nothing?
________________________________
This message is confidential and is for the sole use of the intended recipient(s). It may
also be privileged or otherwise protected by copyright or other legal rules. If you have
received it by mistake please let us know by reply email and delete it from your system.
It is prohibited to copy this message or disclose its content to anyone. Any
confidentiality or privilege is not waived or lost by any mistaken delivery or
unauthorized disclosure of the message. All messages sent to and from Agoda may be
monitored to ensure compliance with company policies, to protect the company's
interests and to remove potential malware. Electronic messages may be intercepted,
amended, lost or deleted, or contain viruses.
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io>
To unsubscribe send an email to
ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
11:39 a.m.
New subject: [Suspicious newsletter] Re: bluefs_buffered_io turn to true
On 14 May 2021, at 14:20, Szabo, Istvan (Agoda)
<Istvan.Szabo(a)agoda.com> wrote:
Howmuch is yours? Mine is vm.min_free_kbytes = 90112.
I use 135168
On 14 May 2021, at 14:31, Szabo, Istvan (Agoda)
<Istvan.Szabo(a)agoda.com> wrote:
Yup, I just saw, should have 3GB :/ I will wait until the system goes back to normal and
will increase.
GBytes is okay too. You can write value in kernel in any time and
kernel will reclaim RAM to this target
k
17 May
17 May
6:15 a.m.
New subject: [Suspicious newsletter] Re: bluefs_buffered_io turn to true
What happens if we are using buffered_io and the machine restared due to some power
failure? Everything that was in the cache will be lost or how ceph handle this?
Istvan Szabo
Senior Infrastructure Engineer
---------------------------------------------------
Agoda Services Co., Ltd.
e: istvan.szabo@agoda.com<mailto:istvan.szabo@agoda.com>
---------------------------------------------------
From: Szabo, Istvan (Agoda)
Sent: Friday, May 14, 2021 3:21 PM
To: 'Konstantin Shalygin' <k0ste(a)k0ste.ru>
Cc: ceph-users(a)ceph.io
Subject: RE: [Suspicious newsletter] [ceph-users] Re: bluefs_buffered_io turn to true
When this stop 😃 ? When died … :D
Istvan Szabo
Senior Infrastructure Engineer
---------------------------------------------------
Agoda Services Co., Ltd.
e: istvan.szabo@agoda.com<mailto:istvan.szabo@agoda.com>
---------------------------------------------------
From: Konstantin Shalygin <k0ste@k0ste.ru<mailto:k0ste@k0ste.ru>>
Sent: Friday, May 14, 2021 3:00 PM
To: Szabo, Istvan (Agoda)
<Istvan.Szabo@agoda.com<mailto:Istvan.Szabo@agoda.com>>
Cc: ceph-users@ceph.io<mailto:ceph-users@ceph.io>
Subject: Re: [Suspicious newsletter] [ceph-users] Re: bluefs_buffered_io turn to true
On 14 May 2021, at 10:50, Szabo, Istvan (Agoda)
<Istvan.Szabo@agoda.com<mailto:Istvan.Szabo@agoda.com>> wrote:
Is it also normal if this buffered_ioturned on, it eats all the memory on the system?
Hmmm.
This is what actually do this option - eat all free memory as cached for bluefs speedups
k
________________________________
This message is confidential and is for the sole use of the intended recipient(s). It may
also be privileged or otherwise protected by copyright or other legal rules. If you have
received it by mistake please let us know by reply email and delete it from your system.
It is prohibited to copy this message or disclose its content to anyone. Any
confidentiality or privilege is not waived or lost by any mistaken delivery or
unauthorized disclosure of the message. All messages sent to and from Agoda may be
monitored to ensure compliance with company policies, to protect the company's
interests and to remove potential malware. Electronic messages may be intercepted,
amended, lost or deleted, or contain viruses.
6:53 a.m.
New subject: [Suspicious newsletter] Re: bluefs_buffered_io turn to true
Den mån 17 maj 2021 kl 08:15 skrev Szabo, Istvan (Agoda) <
Istvan.Szabo(a)agoda.com>gt;:
What happens if we are using buffered_io and the
machine restared due to
some power failure? Everything that was in the cache will be lost or how
ceph handle this?
Not to be picky, but between any client writing data to storage and the
magnetic layers of a drive there are some 8-9-10 layers of caches and
buffers, and a varying degree of lying to the layer above on when data
actually hits the spinning rust. Just because we found yet another one here
which may or may not impact writes, it doesn't change the situation in the
larger scheme.
If power goes, any IO in flight will be disrupted in some way, either it
will be finalized if possible or reverted if not, where reverted might be
as late as "on the subsequent fsck/scrub over that part of the disk". The
WAL/DB/Journals help a bit in figuring out if it can be finalized and what
was in flight before the crash, but all in all, this is not totally
unexpected from a storage point of view.
Most often I guess the IO will be counted as not-done and when the OSD
comes back, it will get the correct data replayed from another PG replica
during recovery/backfill and that's more or less it.
--
May the most significant bit of your life be positive.
1074
days inactive
1077
days old
17 comments
4 participants
participants (4)
-
Irek Fasikhov -
Janne Johansson -
Konstantin Shalygin -
Szabo, Istvan (Agoda)