another osd_pglog memory usage incident
Hi all, This morning some osds in our S3 cluster started going OOM, after restarting them I noticed that the osd_pglog is using >1.5GB per osd. (This is on an osd with osd_memory_target = 2GB, hosting 112PGs, all PGs are active+clean). After reading through this list and trying a few things, I'd like to share the following observations for your feedback: 1. The pg log contains 3000 entries by default (on nautilus). These 3000 entries can legitimately consume gigabytes of ram for some use-cases. (I haven't determined exactly which ops triggered this today). 2. The pg log length is decided by the primary osd -- setting osd_max_pg_log_entries/osd_min_pg_log_entries on one single OSD does not have a big effect (because most of the PGs are primaried somewhere else). You need to set it on all the osds for it to be applied to all PGs. 3. We eventually set osd_max_pg_log_entries = 500 everywhere. This decreased the osd_pglog mempool from more than 1.5GB on our largest osds to less that 500MB. 4. The osd_pglog mempool is not accounted for in the osd_memory_target (in nautilus). 5. I have opened a feature request to limit the pg_log length by memory size (https://tracker.ceph.com/issues/47775). This way we could allocate a fraction of memory to the pg log and it would shorten the pglog length (budget) accordingly. 6. Would it be feasible to add an osd option to 'trim pg log at boot' ? This way we could avoid the cumbersome ceph-objectstore-tool trim-pg-log in cases of disaster (osds going oom at boot). For those that had pglog memory usage incidents -- does this match your experience? Thanks! Dan
On 07/10/2020 14:08, Dan van der Ster wrote:
Hi all,
This morning some osds in our S3 cluster started going OOM, after restarting them I noticed that the osd_pglog is using >1.5GB per osd. (This is on an osd with osd_memory_target = 2GB, hosting 112PGs, all PGs are active+clean).
After reading through this list and trying a few things, I'd like to share the following observations for your feedback:
1. The pg log contains 3000 entries by default (on nautilus). These 3000 entries can legitimately consume gigabytes of ram for some use-cases. (I haven't determined exactly which ops triggered this today). 2. The pg log length is decided by the primary osd -- setting osd_max_pg_log_entries/osd_min_pg_log_entries on one single OSD does not have a big effect (because most of the PGs are primaried somewhere else). You need to set it on all the osds for it to be applied to all PGs. 3. We eventually set osd_max_pg_log_entries = 500 everywhere. This decreased the osd_pglog mempool from more than 1.5GB on our largest osds to less that 500MB. 4. The osd_pglog mempool is not accounted for in the osd_memory_target (in nautilus). 5. I have opened a feature request to limit the pg_log length by memory size (https://tracker.ceph.com/issues/47775). This way we could allocate a fraction of memory to the pg log and it would shorten the pglog length (budget) accordingly. 6. Would it be feasible to add an osd option to 'trim pg log at boot' ? This way we could avoid the cumbersome ceph-objectstore-tool trim-pg-log in cases of disaster (osds going oom at boot).
For those that had pglog memory usage incidents -- does this match your experience?
Not really. I have an active case where reducing pglog lenght works for a short period after which memory consumption grows again. These OSDs however show data being used in buffer anon which is probably something different. Regarding the trim on boot, that sounds feasible. I already added a 'compact on boot' setting, but trimming all PGs on boot should be doable. It loads all the PGs and at that point they can be trimmed. Wido
Thanks!
Dan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Wed, Oct 7, 2020 at 3:29 PM Wido den Hollander <wido@42on.com> wrote:
On 07/10/2020 14:08, Dan van der Ster wrote:
Hi all,
This morning some osds in our S3 cluster started going OOM, after restarting them I noticed that the osd_pglog is using >1.5GB per osd. (This is on an osd with osd_memory_target = 2GB, hosting 112PGs, all PGs are active+clean).
After reading through this list and trying a few things, I'd like to share the following observations for your feedback:
1. The pg log contains 3000 entries by default (on nautilus). These 3000 entries can legitimately consume gigabytes of ram for some use-cases. (I haven't determined exactly which ops triggered this today). 2. The pg log length is decided by the primary osd -- setting osd_max_pg_log_entries/osd_min_pg_log_entries on one single OSD does not have a big effect (because most of the PGs are primaried somewhere else). You need to set it on all the osds for it to be applied to all PGs. 3. We eventually set osd_max_pg_log_entries = 500 everywhere. This decreased the osd_pglog mempool from more than 1.5GB on our largest osds to less that 500MB. 4. The osd_pglog mempool is not accounted for in the osd_memory_target (in nautilus). 5. I have opened a feature request to limit the pg_log length by memory size (https://tracker.ceph.com/issues/47775). This way we could allocate a fraction of memory to the pg log and it would shorten the pglog length (budget) accordingly. 6. Would it be feasible to add an osd option to 'trim pg log at boot' ? This way we could avoid the cumbersome ceph-objectstore-tool trim-pg-log in cases of disaster (osds going oom at boot).
For those that had pglog memory usage incidents -- does this match your experience?
Not really. I have an active case where reducing pglog lenght works for a short period after which memory consumption grows again.
These OSDs however show data being used in buffer anon which is probably something different.
Well in fact at the very beginning of this incident we had excessive buffer_anon -- and I only rebooted the osds a couple hours ago and buffer_anon might indeed be growing still: # ceph daemon osd.245 dump_mempools | jq .mempool.by_pool.buffer_anon { "items": 36762, "bytes": 436869187 } Did you have any clues yet what is triggering that? How do you work around? Is there a tracker for this? -- dan
Regarding the trim on boot, that sounds feasible. I already added a 'compact on boot' setting, but trimming all PGs on boot should be doable. It loads all the PGs and at that point they can be trimmed.
Wido
Thanks!
Dan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 07/10/2020 16:00, Dan van der Ster wrote:
On Wed, Oct 7, 2020 at 3:29 PM Wido den Hollander <wido@42on.com> wrote:
On 07/10/2020 14:08, Dan van der Ster wrote:
Hi all,
This morning some osds in our S3 cluster started going OOM, after restarting them I noticed that the osd_pglog is using >1.5GB per osd. (This is on an osd with osd_memory_target = 2GB, hosting 112PGs, all PGs are active+clean).
After reading through this list and trying a few things, I'd like to share the following observations for your feedback:
1. The pg log contains 3000 entries by default (on nautilus). These 3000 entries can legitimately consume gigabytes of ram for some use-cases. (I haven't determined exactly which ops triggered this today). 2. The pg log length is decided by the primary osd -- setting osd_max_pg_log_entries/osd_min_pg_log_entries on one single OSD does not have a big effect (because most of the PGs are primaried somewhere else). You need to set it on all the osds for it to be applied to all PGs. 3. We eventually set osd_max_pg_log_entries = 500 everywhere. This decreased the osd_pglog mempool from more than 1.5GB on our largest osds to less that 500MB. 4. The osd_pglog mempool is not accounted for in the osd_memory_target (in nautilus). 5. I have opened a feature request to limit the pg_log length by memory size (https://tracker.ceph.com/issues/47775). This way we could allocate a fraction of memory to the pg log and it would shorten the pglog length (budget) accordingly. 6. Would it be feasible to add an osd option to 'trim pg log at boot' ? This way we could avoid the cumbersome ceph-objectstore-tool trim-pg-log in cases of disaster (osds going oom at boot).
For those that had pglog memory usage incidents -- does this match your experience?
Not really. I have an active case where reducing pglog lenght works for a short period after which memory consumption grows again.
These OSDs however show data being used in buffer anon which is probably something different.
Well in fact at the very beginning of this incident we had excessive buffer_anon -- and I only rebooted the osds a couple hours ago and buffer_anon might indeed be growing still:
# ceph daemon osd.245 dump_mempools | jq .mempool.by_pool.buffer_anon { "items": 36762, "bytes": 436869187 }
Did you have any clues yet what is triggering that? How do you work around?
In this case writing to the RGW seems to keep it workable. If we stop writing to RADOS the OSD's their memory explodes and they OOM. We do not have a clue or solution yet. In this case we also see a lot of BlueFS spillovers and RocksDB growing almost unbounded, a lot of compactions are required to keep it working.
Is there a tracker for this?
No, not yet. We do have a couple of messages on the ML about this. Wido
-- dan
Regarding the trim on boot, that sounds feasible. I already added a 'compact on boot' setting, but trimming all PGs on boot should be doable. It loads all the PGs and at that point they can be trimmed.
Wido
Thanks!
Dan _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 07.10.20 21:00, Wido den Hollander wrote:
On 07/10/2020 16:00, Dan van der Ster wrote:
On Wed, Oct 7, 2020 at 3:29 PM Wido den Hollander <wido@42on.com> wrote:
On 07/10/2020 14:08, Dan van der Ster wrote:
Hi all,
This morning some osds in our S3 cluster started going OOM, after restarting them I noticed that the osd_pglog is using >1.5GB per osd. (This is on an osd with osd_memory_target = 2GB, hosting 112PGs, all PGs are active+clean).
[...] Hi all, As Wido said, our case may be a bit different. This is still on 14.2.8. Trouble started with lots of small objects. There were 2 Veeam buckets with more than 400M objects each, on a pool with EC 8+3. This means that there were about 10 billion object shards. DB space on SSD was tiny (originally built for filestore, there was space for 25GB, i.e. only 3GB really usable as we know now). Then OSD memory started to grow, mostly buffer_anon. Decreasing osd_max_pg_log_entries helped (with buffer_anon!). We added RAM, only to have more OOMs a few days later. And we realized that DB slow bytes had started to grow, without bounds. We could delete the objects (took several weeks), and there were no OOMs during this time. But afterwards again growing buffer_anon. Once I observed free memory improving when a customer was writing heavily. So we started to write constantly (small objects to dummy buckets). This helps with buffer_anon and also with the DB growth. It seems that at least 14.2.8 does not trim buffer_anon periodically, but only when writing: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/EVPELHOL4KL... One possible explanation (just an idea) for the large amount of buffer_anon: DB slow bytes got spread over lots and lots of small allocations on the HDD. We rebuilt all OSDs with bigger DBs (31GB). And we limit the amount of slow bytes, with manual compactions. With the big amount of small objects gone, the cluster was still unhealthy. Then we realized that the RGW Garbage Collector did not keep up with the load. A possible reason for the GC backlog: customers using features like versioning more heavily than before. There are high refcounts in GC, and there were times with lots of HEAD requests from some customers. GC load is mostly read load. Combined with only low write activity, this may be problematic. We tuned GC up and the backlog is going down now, slowly (again this takes weeks). Cheers Harry
On Fri, Oct 9, 2020 at 1:42 PM Harald Staub <harald.staub@switch.ch> wrote:
On 07.10.20 21:00, Wido den Hollander wrote:
On 07/10/2020 16:00, Dan van der Ster wrote:
On Wed, Oct 7, 2020 at 3:29 PM Wido den Hollander <wido@42on.com> wrote:
On 07/10/2020 14:08, Dan van der Ster wrote:
Hi all,
This morning some osds in our S3 cluster started going OOM, after restarting them I noticed that the osd_pglog is using >1.5GB per osd. (This is on an osd with osd_memory_target = 2GB, hosting 112PGs, all PGs are active+clean).
[...]
Hi all,
As Wido said, our case may be a bit different.
This is still on 14.2.8. Trouble started with lots of small objects. There were 2 Veeam buckets with more than 400M objects each, on a pool with EC 8+3. This means that there were about 10 billion object shards. DB space on SSD was tiny (originally built for filestore, there was space for 25GB, i.e. only 3GB really usable as we know now).
Then OSD memory started to grow, mostly buffer_anon. Decreasing osd_max_pg_log_entries helped (with buffer_anon!). We added RAM, only to have more OOMs a few days later. And we realized that DB slow bytes had started to grow, without bounds.
We could delete the objects (took several weeks), and there were no OOMs during this time. But afterwards again growing buffer_anon.
Once I observed free memory improving when a customer was writing heavily. So we started to write constantly (small objects to dummy buckets). This helps with buffer_anon and also with the DB growth.
It seems that at least 14.2.8 does not trim buffer_anon periodically, but only when writing: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/EVPELHOL4KL...
One possible explanation (just an idea) for the large amount of buffer_anon: DB slow bytes got spread over lots and lots of small allocations on the HDD.
We rebuilt all OSDs with bigger DBs (31GB). And we limit the amount of slow bytes, with manual compactions.
With the big amount of small objects gone, the cluster was still unhealthy. Then we realized that the RGW Garbage Collector did not keep up with the load. A possible reason for the GC backlog: customers using features like versioning more heavily than before.
There are high refcounts in GC, and there were times with lots of HEAD requests from some customers.
GC load is mostly read load. Combined with only low write activity, this may be problematic.
We tuned GC up and the backlog is going down now, slowly (again this takes weeks).
Thanks Wido and Harald for the info. On our side the issue was not so severe -- it started with huge buffer_anon across the OSDs, but then after restarting an affected osd a similar amount of memory would be accounted in the osd_pglog mempool. We have mitigated by keeping only 500 pglog entries for now, and at the moment things are looking quite stable, without any mempools leaking. I also noticed a possible relationship with scrubbing -- One week ago we increased to osd_max_scrubs=5 to clear out a scrubbing backlog; I wonder if the increased read/write ratio somehow led to an exploding buffer_anon. Do things stabilize on your side if you temporarily disable scrubbing? Otherwise, we've just instrumented the OSDs on this cluster so we can track all the mempools in grafana. If we learn anything we'll share that here. Cheers, Dan
On 09.10.20 13:55, Dan van der Ster wrote: [...]
I also noticed a possible relationship with scrubbing -- One week ago we increased to osd_max_scrubs=5 to clear out a scrubbing backlog; I wonder if the increased read/write ratio somehow led to an exploding buffer_anon. Do things stabilize on your side if you temporarily disable scrubbing?
During the worst periods, we had disabled scrubbing. When we re-enabled, we had our write-job to mitigate the problems. And currently, scrub load is low. So I cannot tell, but it is very plausible. Cheers Harry
1. The pg log contains 3000 entries by default (on nautilus). These 3000 entries can legitimately consume gigabytes of ram for some use-cases. (I haven't determined exactly which ops triggered this today).
How can I check how much ram my pg_logs are using? -----Original Message----- Cc: ceph-users Subject: [ceph-users] Re: another osd_pglog memory usage incident On 09.10.20 13:55, Dan van der Ster wrote: [...]
I also noticed a possible relationship with scrubbing -- One week ago we increased to osd_max_scrubs=5 to clear out a scrubbing backlog; I wonder if the increased read/write ratio somehow led to an exploding buffer_anon. Do things stabilize on your side if you temporarily disable scrubbing?
During the worst periods, we had disabled scrubbing. When we re-enabled, we had our write-job to mitigate the problems. And currently, scrub load is low. So I cannot tell, but it is very plausible. Cheers Harry _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
Dan van der Ster
-
Harald Staub
-
Marc Roos
-
Wido den Hollander