bluefs_buffered_io turn to true
Hi, I had issue with the snaptrim after a hug amount of deleted data, it slows down the team operations due to the snaptrim and snaptrim_wait pgs. I've changed couple of things: debug_ms = 0/0 #default 0/5 osd_snap_trim_priority = 1 # default 5 osd_pg_max_concurrent_snap_trims = 1 # default 2 But didn't help. I've found this thread about buffered io and seems like it helped to them: https://forum.proxmox.com/threads/ceph-storage-all-pgs-snaptrim-every-night-... I don't use swap on the OSD nodes, so I gave a try on 1 osd node and it caused basically the complete node's pg-s are degraded. Is it normal? I hope it will not rebalance the complete node because I don't have space for that. I changed it back but still slowly decreasing, so not sure this settings is correct or not or this behavior is good or not? 2021-05-14 12:18:11.447628 mon.2004 [WRN] Health check update: 3353/91976715 objects misplaced (0.004%) (OBJECT_MISPLACED) 2021-05-14 12:18:11.447640 mon.2004 [WRN] Health check update: Degraded data redundancy: 33078466/91976715 objects degraded (35.964%), 254 pgs degraded, 253 pgs undersized (PG_DEGRADED) Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo@agoda.com<mailto:istvan.szabo@agoda.com> --------------------------------------------------- ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.
Hi, This is not a normal, It's something different I think, like a crush changes on restart. This option will be enabled by default again in Nautilus next, so you can use it now with 14.2.19-20 k Sent from my iPhone
On 14 May 2021, at 08:21, Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> wrote:
Hi,
I had issue with the snaptrim after a hug amount of deleted data, it slows down the team operations due to the snaptrim and snaptrim_wait pgs.
I've changed couple of things:
debug_ms = 0/0 #default 0/5 osd_snap_trim_priority = 1 # default 5 osd_pg_max_concurrent_snap_trims = 1 # default 2
But didn't help.
I've found this thread about buffered io and seems like it helped to them: https://forum.proxmox.com/threads/ceph-storage-all-pgs-snaptrim-every-night-...
I don't use swap on the OSD nodes, so I gave a try on 1 osd node and it caused basically the complete node's pg-s are degraded. Is it normal? I hope it will not rebalance the complete node because I don't have space for that. I changed it back but still slowly decreasing, so not sure this settings is correct or not or this behavior is good or not?
2021-05-14 12:18:11.447628 mon.2004 [WRN] Health check update: 3353/91976715 objects misplaced (0.004%) (OBJECT_MISPLACED) 2021-05-14 12:18:11.447640 mon.2004 [WRN] Health check update: Degraded data redundancy: 33078466/91976715 objects degraded (35.964%), 254 pgs degraded, 253 pgs undersized (PG_DEGRADED)
Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo@agoda.com<mailto:istvan.szabo@agoda.com> ---------------------------------------------------
________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, It is quite an older cluster, luminous 12.2.8. Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo@agoda.com --------------------------------------------------- -----Original Message----- From: Konstantin Shalygin <k0ste@k0ste.ru> Sent: Friday, May 14, 2021 1:12 PM To: Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> Cc: ceph-users@ceph.io Subject: [Suspicious newsletter] [ceph-users] Re: bluefs_buffered_io turn to true Hi, This is not a normal, It's something different I think, like a crush changes on restart. This option will be enabled by default again in Nautilus next, so you can use it now with 14.2.19-20 k Sent from my iPhone
On 14 May 2021, at 08:21, Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> wrote:
Hi,
I had issue with the snaptrim after a hug amount of deleted data, it slows down the team operations due to the snaptrim and snaptrim_wait pgs.
I've changed couple of things:
debug_ms = 0/0 #default 0/5 osd_snap_trim_priority = 1 # default 5 osd_pg_max_concurrent_snap_trims = 1 # default 2
But didn't help.
I've found this thread about buffered io and seems like it helped to them: https://forum.proxmox.com/threads/ceph-storage-all-pgs-snaptrim-every- night-slowing-down-vms.71573/
I don't use swap on the OSD nodes, so I gave a try on 1 osd node and it caused basically the complete node's pg-s are degraded. Is it normal? I hope it will not rebalance the complete node because I don't have space for that. I changed it back but still slowly decreasing, so not sure this settings is correct or not or this behavior is good or not?
2021-05-14 12:18:11.447628 mon.2004 [WRN] Health check update: 3353/91976715 objects misplaced (0.004%) (OBJECT_MISPLACED) 2021-05-14 12:18:11.447640 mon.2004 [WRN] Health check update: Degraded data redundancy: 33078466/91976715 objects degraded (35.964%), 254 pgs degraded, 253 pgs undersized (PG_DEGRADED)
Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo@agoda.com<mailto:istvan.szabo@agoda.com> ---------------------------------------------------
________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I recommend to upgrade at least to 12.2.13, for luminous even .12 and .13 is significant difference in code. k
On 14 May 2021, at 09:22, Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> wrote:
It is quite an older cluster, luminous 12.2.8.
Is it also normal if this buffered_ioturned on, it eats all the memory on the system? Hmmm. Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo@agoda.com<mailto:istvan.szabo@agoda.com> --------------------------------------------------- From: Konstantin Shalygin <k0ste@k0ste.ru> Sent: Friday, May 14, 2021 2:12 PM To: Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> Cc: ceph-users@ceph.io Subject: Re: [Suspicious newsletter] [ceph-users] Re: bluefs_buffered_io turn to true I recommend to upgrade at least to 12.2.13, for luminous even .12 and .13 is significant difference in code. k On 14 May 2021, at 09:22, Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com<mailto:Istvan.Szabo@agoda.com>> wrote: It is quite an older cluster, luminous 12.2.8. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.
On 14 May 2021, at 10:50, Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> wrote:
Is it also normal if this buffered_ioturned on, it eats all the memory on the system? Hmmm.
This is what actually do this option - eat all free memory as cached for bluefs speedups k
When this stop 😃 ? When died … :D Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo@agoda.com<mailto:istvan.szabo@agoda.com> --------------------------------------------------- From: Konstantin Shalygin <k0ste@k0ste.ru> Sent: Friday, May 14, 2021 3:00 PM To: Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> Cc: ceph-users@ceph.io Subject: Re: [Suspicious newsletter] [ceph-users] Re: bluefs_buffered_io turn to true On 14 May 2021, at 10:50, Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com<mailto:Istvan.Szabo@agoda.com>> wrote: Is it also normal if this buffered_ioturned on, it eats all the memory on the system? Hmmm. This is what actually do this option - eat all free memory as cached for bluefs speedups k ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.
Nope, kernel reserves enough memory to free on pressure, for example 36OSD 0.5TiB RAM host: total used free shared buff/cache available Mem: 502G 168G 2.9G 18M 331G 472G Swap: 952M 248M 704M k
On 14 May 2021, at 11:20, Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> wrote:
When this stop 😃 ? When died … :D
Ok, seems like it doesn’t go below 600MB out of the 256GB, let’s wait until the pg_degradation healed. Did I do something wrong? I set in the global config the bluefs option, and restarted ceph.target on the osd node :/ ? Doe this need some special thing to apply? Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo@agoda.com<mailto:istvan.szabo@agoda.com> --------------------------------------------------- From: Konstantin Shalygin <k0ste@k0ste.ru> Sent: Friday, May 14, 2021 3:26 PM To: Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> Cc: ceph-users@ceph.io Subject: Re: [ceph-users] [Suspicious newsletter] Re: bluefs_buffered_io turn to true Nope, kernel reserves enough memory to free on pressure, for example 36OSD 0.5TiB RAM host: total used free shared buff/cache available Mem: 502G 168G 2.9G 18M 331G 472G Swap: 952M 248M 704M k On 14 May 2021, at 11:20, Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com<mailto:Istvan.Szabo@agoda.com>> wrote: When this stop 😃 ? When died … :D ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.
It's enough, should be true now... k
On 14 May 2021, at 12:51, Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> wrote:
Did I do something wrong? I set in the global config the bluefs option, and restarted ceph.target on the osd node :/ ?
Doe this need some special thing to apply?
Is there anything that should be set just to be sure oom kill not happen? Or nothing? Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo@agoda.com<mailto:istvan.szabo@agoda.com> --------------------------------------------------- From: Konstantin Shalygin <k0ste@k0ste.ru> Sent: Friday, May 14, 2021 5:32 PM To: Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> Cc: ceph-users@ceph.io Subject: Re: [ceph-users] [Suspicious newsletter] Re: bluefs_buffered_io turn to true It's enough, should be true now... k On 14 May 2021, at 12:51, Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com<mailto:Istvan.Szabo@agoda.com>> wrote: Did I do something wrong? I set in the global config the bluefs option, and restarted ceph.target on the osd node :/ ? Doe this need some special thing to apply? ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.
I suggest to look into vm.min_free_kbytes kernel option, and increase it twice k
On 14 May 2021, at 13:45, Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> wrote:
Is there anything that should be set just to be sure oom kill not happen? Or nothing?
Howmuch is yours? Mine is vm.min_free_kbytes = 90112. Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo@agoda.com<mailto:istvan.szabo@agoda.com> --------------------------------------------------- From: Konstantin Shalygin <k0ste@k0ste.ru> Sent: Friday, May 14, 2021 6:07 PM To: Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> Cc: ceph-users@ceph.io Subject: Re: [ceph-users] [Suspicious newsletter] Re: bluefs_buffered_io turn to true I suggest to look into vm.min_free_kbytes kernel option, and increase it twice k On 14 May 2021, at 13:45, Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com<mailto:Istvan.Szabo@agoda.com>> wrote: Is there anything that should be set just to be sure oom kill not happen? Or nothing? ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.
Hi. https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/2/html/ce... пт, 14 мая 2021 г. в 14:21, Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com>:
Howmuch is yours? Mine is vm.min_free_kbytes = 90112.
Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo@agoda.com<mailto:istvan.szabo@agoda.com> ---------------------------------------------------
From: Konstantin Shalygin <k0ste@k0ste.ru> Sent: Friday, May 14, 2021 6:07 PM To: Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> Cc: ceph-users@ceph.io Subject: Re: [ceph-users] [Suspicious newsletter] Re: bluefs_buffered_io turn to true
I suggest to look into vm.min_free_kbytes kernel option, and increase it twice
k
On 14 May 2021, at 13:45, Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com <mailto:Istvan.Szabo@agoda.com>> wrote:
Is there anything that should be set just to be sure oom kill not happen? Or nothing?
________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Yup, I just saw, should have 3GB :/ I will wait until the system goes back to normal and will increase. Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo@agoda.com<mailto:istvan.szabo@agoda.com> --------------------------------------------------- From: Irek Fasikhov <malmyzh@gmail.com> Sent: Friday, May 14, 2021 6:28 PM To: Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> Cc: Konstantin Shalygin <k0ste@k0ste.ru>; ceph-users@ceph.io Subject: Re: [ceph-users] Re: [Suspicious newsletter] Re: bluefs_buffered_io turn to true Hi. https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/2/html/ce... пт, 14 мая 2021 г. в 14:21, Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com<mailto:Istvan.Szabo@agoda.com>>: Howmuch is yours? Mine is vm.min_free_kbytes = 90112. Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo@agoda.com<mailto:istvan.szabo@agoda.com><mailto:istvan.szabo@agoda.com<mailto:istvan.szabo@agoda.com>> --------------------------------------------------- From: Konstantin Shalygin <k0ste@k0ste.ru<mailto:k0ste@k0ste.ru>> Sent: Friday, May 14, 2021 6:07 PM To: Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com<mailto:Istvan.Szabo@agoda.com>> Cc: ceph-users@ceph.io<mailto:ceph-users@ceph.io> Subject: Re: [ceph-users] [Suspicious newsletter] Re: bluefs_buffered_io turn to true I suggest to look into vm.min_free_kbytes kernel option, and increase it twice k On 14 May 2021, at 13:45, Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com<mailto:Istvan.Szabo@agoda.com><mailto:Istvan.Szabo@agoda.com<mailto:Istvan.Szabo@agoda.com>>> wrote: Is there anything that should be set just to be sure oom kill not happen? Or nothing? ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
On 14 May 2021, at 14:20, Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> wrote:
Howmuch is yours? Mine is vm.min_free_kbytes = 90112.
I use 135168
On 14 May 2021, at 14:31, Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com> wrote:
Yup, I just saw, should have 3GB :/ I will wait until the system goes back to normal and will increase. GBytes is okay too. You can write value in kernel in any time and kernel will reclaim RAM to this target
k
What happens if we are using buffered_io and the machine restared due to some power failure? Everything that was in the cache will be lost or how ceph handle this? Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo@agoda.com<mailto:istvan.szabo@agoda.com> --------------------------------------------------- From: Szabo, Istvan (Agoda) Sent: Friday, May 14, 2021 3:21 PM To: 'Konstantin Shalygin' <k0ste@k0ste.ru> Cc: ceph-users@ceph.io Subject: RE: [Suspicious newsletter] [ceph-users] Re: bluefs_buffered_io turn to true When this stop 😃 ? When died … :D Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo@agoda.com<mailto:istvan.szabo@agoda.com> --------------------------------------------------- From: Konstantin Shalygin <k0ste@k0ste.ru<mailto:k0ste@k0ste.ru>> Sent: Friday, May 14, 2021 3:00 PM To: Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com<mailto:Istvan.Szabo@agoda.com>> Cc: ceph-users@ceph.io<mailto:ceph-users@ceph.io> Subject: Re: [Suspicious newsletter] [ceph-users] Re: bluefs_buffered_io turn to true On 14 May 2021, at 10:50, Szabo, Istvan (Agoda) <Istvan.Szabo@agoda.com<mailto:Istvan.Szabo@agoda.com>> wrote: Is it also normal if this buffered_ioturned on, it eats all the memory on the system? Hmmm. This is what actually do this option - eat all free memory as cached for bluefs speedups k ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.
Den mån 17 maj 2021 kl 08:15 skrev Szabo, Istvan (Agoda) < Istvan.Szabo@agoda.com>:
What happens if we are using buffered_io and the machine restared due to some power failure? Everything that was in the cache will be lost or how ceph handle this?
Not to be picky, but between any client writing data to storage and the magnetic layers of a drive there are some 8-9-10 layers of caches and buffers, and a varying degree of lying to the layer above on when data actually hits the spinning rust. Just because we found yet another one here which may or may not impact writes, it doesn't change the situation in the larger scheme. If power goes, any IO in flight will be disrupted in some way, either it will be finalized if possible or reverted if not, where reverted might be as late as "on the subsequent fsck/scrub over that part of the disk". The WAL/DB/Journals help a bit in figuring out if it can be finalized and what was in flight before the crash, but all in all, this is not totally unexpected from a storage point of view. Most often I guess the IO will be counted as not-done and when the OSD comes back, it will get the correct data replayed from another PG replica during recovery/backfill and that's more or less it. -- May the most significant bit of your life be positive.
participants (4)
-
Irek Fasikhov
-
Janne Johansson
-
Konstantin Shalygin
-
Szabo, Istvan (Agoda)