Huge RAM Ussage on OSD recovery
Hi, today mi Infra provider has a blackout, then the Ceph was try to recover but are in an inconsistent state because many OSD can recover itself because the kernel kill it by OOM. Even now one OSD that was OK, go down by OOM killed. Even in a server with 32GB RAM the OSD use ALL that and never recover, i think that can be a memory leak, ceph version octopus 15.2.3 In: https://pastebin.pl/view/59089adc You can see that buffer_anon get 32GB, but why?? all my cluster is down because that.
On 2020-10-20 23:57, Ing. Luis Felipe Domínguez Vega wrote:
Hi, today mi Infra provider has a blackout, then the Ceph was try to recover but are in an inconsistent state because many OSD can recover itself because the kernel kill it by OOM. Even now one OSD that was OK, go down by OOM killed.
Even in a server with 32GB RAM the OSD use ALL that and never recover, i think that can be a memory leak, ceph version octopus 15.2.3
Especially during recovery Ceph can use a lot of memory. How many OSDDs do you have in that server? How many PGs per OSD? The default osd_memory_target is 4 GB, and besides that extra memory is used. You can try to limit the mem target and restart the OSD (for example: ceph config set osd osd_memory_target 2G). Gr. Stefan
Hi, This might be the pglog issue which has been coming up a few times on the list. If the OSD cannot boot without going OOM, you might have success by trimming the pglog, e.g. search this list for "ceph-objectstore-tool --op trim-pg-log" for some recipes. The thread "OSDs taking too much memory, for pglog" in particular might help. Cheers, Dan On Tue, Oct 20, 2020 at 11:57 PM Ing. Luis Felipe Domínguez Vega <luis.dominguez@desoft.cu> wrote:
Hi, today mi Infra provider has a blackout, then the Ceph was try to recover but are in an inconsistent state because many OSD can recover itself because the kernel kill it by OOM. Even now one OSD that was OK, go down by OOM killed.
Even in a server with 32GB RAM the OSD use ALL that and never recover, i think that can be a memory leak, ceph version octopus 15.2.3
In: https://pastebin.pl/view/59089adc You can see that buffer_anon get 32GB, but why?? all my cluster is down because that. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Theoretically we shouldn't be spiking memory as much these days during recovery, but the code is complicated and it's tough to reproduce these kinds of issues in-house. If you happen to catch it in the act, do you see the pglog mempool stats also spiking up? Mark On 10/21/20 2:34 AM, Dan van der Ster wrote:
Hi,
This might be the pglog issue which has been coming up a few times on the list. If the OSD cannot boot without going OOM, you might have success by trimming the pglog, e.g. search this list for "ceph-objectstore-tool --op trim-pg-log" for some recipes. The thread "OSDs taking too much memory, for pglog" in particular might help.
Cheers, Dan
On Tue, Oct 20, 2020 at 11:57 PM Ing. Luis Felipe Domínguez Vega <luis.dominguez@desoft.cu> wrote:
Hi, today mi Infra provider has a blackout, then the Ceph was try to recover but are in an inconsistent state because many OSD can recover itself because the kernel kill it by OOM. Even now one OSD that was OK, go down by OOM killed.
Even in a server with 32GB RAM the OSD use ALL that and never recover, i think that can be a memory leak, ceph version octopus 15.2.3
In: https://pastebin.pl/view/59089adc You can see that buffer_anon get 32GB, but why?? all my cluster is down because that. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
El 2020-10-20 17:57, Ing. Luis Felipe Domínguez Vega escribió:
Hi, today mi Infra provider has a blackout, then the Ceph was try to recover but are in an inconsistent state because many OSD can recover itself because the kernel kill it by OOM. Even now one OSD that was OK, go down by OOM killed.
Even in a server with 32GB RAM the OSD use ALL that and never recover, i think that can be a memory leak, ceph version octopus 15.2.3
In: https://pastebin.pl/view/59089adc You can see that buffer_anon get 32GB, but why?? all my cluster is down because that. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io This is a log when starts consuming resources: https://pastebin.ubuntu.com/p/dwbXtX7wTP/
El 2020-10-20 17:57, Ing. Luis Felipe Domínguez Vega escribió:
Hi, today mi Infra provider has a blackout, then the Ceph was try to recover but are in an inconsistent state because many OSD can recover itself because the kernel kill it by OOM. Even now one OSD that was OK, go down by OOM killed.
Even in a server with 32GB RAM the OSD use ALL that and never recover, i think that can be a memory leak, ceph version octopus 15.2.3
In: https://pastebin.pl/view/59089adc You can see that buffer_anon get 32GB, but why?? all my cluster is down because that. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io Used the --op export-remove and then --op import of ceph-objectstore-tool for the failing PG and now the OSD is running great.
On 10/21/20 10:54 PM, Ing. Luis Felipe Domínguez Vega wrote:
El 2020-10-20 17:57, Ing. Luis Felipe Domínguez Vega escribió:
Hi, today mi Infra provider has a blackout, then the Ceph was try to recover but are in an inconsistent state because many OSD can recover itself because the kernel kill it by OOM. Even now one OSD that was OK, go down by OOM killed.
Even in a server with 32GB RAM the OSD use ALL that and never recover, i think that can be a memory leak, ceph version octopus 15.2.3
In: https://pastebin.pl/view/59089adc You can see that buffer_anon get 32GB, but why?? all my cluster is down because that. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io Used the --op export-remove and then --op import of ceph-objectstore-tool for the failing PG and now the OSD is running great.
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
That's great news! ...but hopefully we'll figure out what's going on so we can avoid the problem in the first place. :) Mark
El 2020-10-22 09:07, Mark Nelson escribió:
On 10/21/20 10:54 PM, Ing. Luis Felipe Domínguez Vega wrote:
El 2020-10-20 17:57, Ing. Luis Felipe Domínguez Vega escribió:
Hi, today mi Infra provider has a blackout, then the Ceph was try to recover but are in an inconsistent state because many OSD can recover itself because the kernel kill it by OOM. Even now one OSD that was OK, go down by OOM killed.
Even in a server with 32GB RAM the OSD use ALL that and never recover, i think that can be a memory leak, ceph version octopus 15.2.3
In: https://pastebin.pl/view/59089adc You can see that buffer_anon get 32GB, but why?? all my cluster is down because that. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io Used the --op export-remove and then --op import of ceph-objectstore-tool for the failing PG and now the OSD is running great.
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
That's great news! ...but hopefully we'll figure out what's going on so we can avoid the problem in the first place. :)
Mark _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
umm not at all, the OSD is not killed but is using huge ammount of RAM and many log meesages like this: osd.46 osd.46 41072109 : slow request osd_op(client.72068484.0:1851999 5.d 5.1aef4f8d (undecoded) ondisk+write+known_if_redirected e155365) initiated 2020-10-22T11:21:56.949886+0000 currently queued for pg
On 10/22/20 9:02 AM, Ing. Luis Felipe Domínguez Vega wrote:
El 2020-10-22 09:07, Mark Nelson escribió:
On 10/21/20 10:54 PM, Ing. Luis Felipe Domínguez Vega wrote:
El 2020-10-20 17:57, Ing. Luis Felipe Domínguez Vega escribió:
Hi, today mi Infra provider has a blackout, then the Ceph was try to recover but are in an inconsistent state because many OSD can recover itself because the kernel kill it by OOM. Even now one OSD that was OK, go down by OOM killed.
Even in a server with 32GB RAM the OSD use ALL that and never recover, i think that can be a memory leak, ceph version octopus 15.2.3
In: https://pastebin.pl/view/59089adc You can see that buffer_anon get 32GB, but why?? all my cluster is down because that. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io Used the --op export-remove and then --op import of ceph-objectstore-tool for the failing PG and now the OSD is running great.
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
That's great news! ...but hopefully we'll figure out what's going on so we can avoid the problem in the first place. :)
Mark _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
umm not at all, the OSD is not killed but is using huge ammount of RAM and many log meesages like this:
osd.46 osd.46 41072109 : slow request osd_op(client.72068484.0:1851999 5.d 5.1aef4f8d (undecoded) ondisk+write+known_if_redirected e155365) initiated 2020-10-22T11:21:56.949886+0000 currently queued for pg
Do you mean that the --op-export-remove and --op import step didn't end up fixing it in the end? I had interpreted "running great" to mean the OSD was no longer using tons of memory (but it's not a real fix, just a workaround). Mark
El 2020-10-22 10:48, Mark Nelson escribió:
On 10/22/20 9:02 AM, Ing. Luis Felipe Domínguez Vega wrote:
El 2020-10-22 09:07, Mark Nelson escribió:
On 10/21/20 10:54 PM, Ing. Luis Felipe Domínguez Vega wrote:
El 2020-10-20 17:57, Ing. Luis Felipe Domínguez Vega escribió:
Hi, today mi Infra provider has a blackout, then the Ceph was try to recover but are in an inconsistent state because many OSD can recover itself because the kernel kill it by OOM. Even now one OSD that was OK, go down by OOM killed.
Even in a server with 32GB RAM the OSD use ALL that and never recover, i think that can be a memory leak, ceph version octopus 15.2.3
In: https://pastebin.pl/view/59089adc You can see that buffer_anon get 32GB, but why?? all my cluster is down because that. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io Used the --op export-remove and then --op import of ceph-objectstore-tool for the failing PG and now the OSD is running great.
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
That's great news! ...but hopefully we'll figure out what's going on so we can avoid the problem in the first place. :)
Mark _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
umm not at all, the OSD is not killed but is using huge ammount of RAM and many log meesages like this:
osd.46 osd.46 41072109 : slow request osd_op(client.72068484.0:1851999 5.d 5.1aef4f8d (undecoded) ondisk+write+known_if_redirected e155365) initiated 2020-10-22T11:21:56.949886+0000 currently queued for pg
Do you mean that the --op-export-remove and --op import step didn't end up fixing it in the end? I had interpreted "running great" to mean the OSD was no longer using tons of memory (but it's not a real fix, just a workaround).
Mark Yes, yesterday was running great, but today is consuming huge RAM, but not OOM killed, is working now with high ammount of RAM, almost 96% of my server
participants (4)
-
Dan van der Ster
-
Ing. Luis Felipe Domínguez Vega
-
Mark Nelson
-
Stefan Kooman