7 Oct
2020
7 Oct
'20
9:08 p.m.
Hi all,
This morning some osds in our S3 cluster started going OOM, after
restarting them I noticed that the osd_pglog is using >1.5GB per osd.
(This is on an osd with osd_memory_target = 2GB, hosting 112PGs, all
PGs are active+clean).
After reading through this list and trying a few things, I'd like to
share the following observations for your feedback:
1. The pg log contains 3000 entries by default (on nautilus). These
3000 entries can legitimately consume gigabytes of ram for some
use-cases. (I haven't determined exactly which ops triggered this
today).
2. The pg log length is decided by the primary osd -- setting
osd_max_pg_log_entries/osd_min_pg_log_entries on one single OSD does
not have a big effect (because most of the PGs are primaried somewhere
else). You need to set it on all the osds for it to be applied to all
PGs.
3. We eventually set osd_max_pg_log_entries = 500 everywhere. This
decreased the osd_pglog mempool from more than 1.5GB on our largest
osds to less that 500MB.
4. The osd_pglog mempool is not accounted for in the osd_memory_target
(in nautilus).
5. I have opened a feature request to limit the pg_log length by
memory size (https://tracker.ceph.com/issues/47775). This way we could
allocate a fraction of memory to the pg log and it would shorten the
pglog length (budget) accordingly.
6. Would it be feasible to add an osd option to 'trim pg log at boot'
? This way we could avoid the cumbersome ceph-objectstore-tool
trim-pg-log in cases of disaster (osds going oom at boot).
For those that had pglog memory usage incidents -- does this match
your experience?
Thanks!
Dan
7 Oct
7 Oct
10:29 p.m.
On 07/10/2020 14:08, Dan van der Ster wrote:
Hi all,
This morning some osds in our S3 cluster started going OOM, after
restarting them I noticed that the osd_pglog is using >1.5GB per osd.
(This is on an osd with osd_memory_target = 2GB, hosting 112PGs, all
PGs are active+clean).
After reading through this list and trying a few things, I'd like to
share the following observations for your feedback:
1. The pg log contains 3000 entries by default (on nautilus). These
3000 entries can legitimately consume gigabytes of ram for some
use-cases. (I haven't determined exactly which ops triggered this
today).
2. The pg log length is decided by the primary osd -- setting
osd_max_pg_log_entries/osd_min_pg_log_entries on one single OSD does
not have a big effect (because most of the PGs are primaried somewhere
else). You need to set it on all the osds for it to be applied to all
PGs.
3. We eventually set osd_max_pg_log_entries = 500 everywhere. This
decreased the osd_pglog mempool from more than 1.5GB on our largest
osds to less that 500MB.
4. The osd_pglog mempool is not accounted for in the osd_memory_target
(in nautilus).
5. I have opened a feature request to limit the pg_log length by
memory size (https://tracker.ceph.com/issues/47775). This way we could
allocate a fraction of memory to the pg log and it would shorten the
pglog length (budget) accordingly.
6. Would it be feasible to add an osd option to 'trim pg log at boot'
? This way we could avoid the cumbersome ceph-objectstore-tool
trim-pg-log in cases of disaster (osds going oom at boot).
For those that had pglog memory usage incidents -- does this match
your experience?
Not really. I have an active case where reducing pglog lenght works for
a short period after which memory consumption grows again.
These OSDs however show data being used in buffer anon which is probably
something different.
Regarding the trim on boot, that sounds feasible. I already added a
'compact on boot' setting, but trimming all PGs on boot should be
doable. It loads all the PGs and at that point they can be trimmed.
Wido
>
> Thanks!
>
> Dan
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
11 p.m.
On Wed, Oct 7, 2020 at 3:29 PM Wido den Hollander <wido(a)42on.com> wrote:
On 07/10/2020 14:08, Dan van der Ster wrote:
Well in fact at the very beginning of this incident we had excessive
buffer_anon -- and I only rebooted the osds a couple hours ago and
buffer_anon might indeed be growing still:
# ceph daemon osd.245 dump_mempools | jq .mempool.by_pool.buffer_anon
{
"items": 36762,
"bytes": 436869187
}
Did you have any clues yet what is triggering that? How do you work around?
Is there a tracker for this?
-- dan
>
> Regarding the trim on boot, that sounds feasible. I already added a
> 'compact on boot' setting, but trimming all PGs on boot should be
> doable. It loads all the PGs and at that point they can be trimmed.
>
> Wido
>
> >
> > Thanks!
> >
> > Dan
> > _______________________________________________
> > ceph-users mailing list -- ceph-users(a)ceph.io
> > To unsubscribe send an email to ceph-users-leave(a)ceph.io
> >
Hi all,
This morning some osds in our S3 cluster started going OOM, after
restarting them I noticed that the osd_pglog is using >1.5GB per osd.
(This is on an osd with osd_memory_target = 2GB, hosting 112PGs, all
PGs are active+clean).
After reading through this list and trying a few things, I'd like to
share the following observations for your feedback:
1. The pg log contains 3000 entries by default (on nautilus). These
3000 entries can legitimately consume gigabytes of ram for some
use-cases. (I haven't determined exactly which ops triggered this
today).
2. The pg log length is decided by the primary osd -- setting
osd_max_pg_log_entries/osd_min_pg_log_entries on one single OSD does
not have a big effect (because most of the PGs are primaried somewhere
else). You need to set it on all the osds for it to be applied to all
PGs.
3. We eventually set osd_max_pg_log_entries = 500 everywhere. This
decreased the osd_pglog mempool from more than 1.5GB on our largest
osds to less that 500MB.
4. The osd_pglog mempool is not accounted for in the osd_memory_target
(in nautilus).
5. I have opened a feature request to limit the pg_log length by
memory size (https://tracker.ceph.com/issues/47775). This way we could
allocate a fraction of memory to the pg log and it would shorten the
pglog length (budget) accordingly.
6. Would it be feasible to add an osd option to 'trim pg log at boot'
? This way we could avoid the cumbersome ceph-objectstore-tool
trim-pg-log in cases of disaster (osds going oom at boot).
For those that had pglog memory usage incidents -- does this match
your experience?
Not really. I have an active case where reducing pglog lenght works for
a short period after which memory consumption grows again.
These OSDs however show data being used in buffer anon which is probably
something different.
8 Oct
8 Oct
4 a.m.
On 07/10/2020 16:00, Dan van der Ster wrote:
On Wed, Oct 7, 2020 at 3:29 PM Wido den Hollander
<wido(a)42on.com> wrote:
In this case writing to the RGW seems to keep it workable. If we stop
writing to RADOS the OSD's their memory explodes and they OOM.
We do not have a clue or solution yet.
In this case we also see a lot of BlueFS spillovers and RocksDB growing
almost unbounded, a lot of compactions are required to keep it working.
On 07/10/2020 14:08, Dan van der Ster wrote:
Well in fact at the very beginning of this incident we had excessive
buffer_anon -- and I only rebooted the osds a couple hours ago and
buffer_anon might indeed be growing still:
# ceph daemon osd.245 dump_mempools | jq .mempool.by_pool.buffer_anon
{
"items": 36762,
"bytes": 436869187
}
Did you have any clues yet what is triggering that? How do you work around? Hi all,
This morning some osds in our S3 cluster started going OOM, after
restarting them I noticed that the osd_pglog is using >1.5GB per osd.
(This is on an osd with osd_memory_target = 2GB, hosting 112PGs, all
PGs are active+clean).
After reading through this list and trying a few things, I'd like to
share the following observations for your feedback:
1. The pg log contains 3000 entries by default (on nautilus). These
3000 entries can legitimately consume gigabytes of ram for some
use-cases. (I haven't determined exactly which ops triggered this
today).
2. The pg log length is decided by the primary osd -- setting
osd_max_pg_log_entries/osd_min_pg_log_entries on one single OSD does
not have a big effect (because most of the PGs are primaried somewhere
else). You need to set it on all the osds for it to be applied to all
PGs.
3. We eventually set osd_max_pg_log_entries = 500 everywhere. This
decreased the osd_pglog mempool from more than 1.5GB on our largest
osds to less that 500MB.
4. The osd_pglog mempool is not accounted for in the osd_memory_target
(in nautilus).
5. I have opened a feature request to limit the pg_log length by
memory size (https://tracker.ceph.com/issues/47775). This way we could
allocate a fraction of memory to the pg log and it would shorten the
pglog length (budget) accordingly.
6. Would it be feasible to add an osd option to 'trim pg log at boot'
? This way we could avoid the cumbersome ceph-objectstore-tool
trim-pg-log in cases of disaster (osds going oom at boot).
For those that had pglog memory usage incidents -- does this match
your experience?
Not really. I have an active case where reducing pglog lenght works for
a short period after which memory consumption grows again.
These OSDs however show data being used in buffer anon which is probably
something different. Is there a tracker for this?
No, not yet. We do have a couple of messages on the ML about this.
Wido
>
> -- dan
>
>
>>
>> Regarding the trim on boot, that sounds feasible. I already added a
>> 'compact on boot' setting, but trimming all PGs on boot should be
>> doable. It loads all the PGs and at that point they can be trimmed.
>>
>> Wido
>>
>>>
>>> Thanks!
>>>
>>> Dan
>>> _______________________________________________
>>> ceph-users mailing list -- ceph-users(a)ceph.io
>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>>>
9 Oct
9 Oct
8:41 p.m.
On 07.10.20 21:00, Wido den Hollander wrote:
On 07/10/2020 16:00, Dan van der Ster wrote:
> On Wed, Oct 7, 2020 at 3:29 PM Wido den Hollander <wido(a)42on.com> wrote:
>>
>>
>>
>> On 07/10/2020 14:08, Dan van der Ster wrote:
>>> Hi all,
>>>
>>> This morning some osds in our S3 cluster started going OOM, after
>>> restarting them I noticed that the osd_pglog is using >1.5GB per osd.
>>> (This is on an osd with osd_memory_target = 2GB, hosting 112PGs, all
>>> PGs are active+clean).
[...]
Hi all,
As Wido said, our case may be a bit different.
This is still on 14.2.8. Trouble started with lots of small objects.
There were 2 Veeam buckets with more than 400M objects each,
on a pool with EC 8+3. This means that there were about 10 billion
object shards. DB space on SSD was tiny (originally built for filestore,
there was space for 25GB, i.e. only 3GB really usable as we know now).
Then OSD memory started to grow, mostly buffer_anon. Decreasing
osd_max_pg_log_entries helped (with buffer_anon!). We added RAM, only to
have more OOMs a few days later. And we realized that DB slow bytes had
started to grow, without bounds.
We could delete the objects (took several weeks), and there were no OOMs
during this time. But afterwards again growing buffer_anon.
Once I observed free memory improving when a customer was writing
heavily. So we started to write constantly (small objects to dummy
buckets). This helps with buffer_anon and also with the DB growth.
It seems that at least 14.2.8 does not trim buffer_anon periodically,
but only when writing:
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/EVPELHOL4K…
One possible explanation (just an idea) for the large amount of
buffer_anon: DB slow bytes got spread over lots and lots of small
allocations on the HDD.
We rebuilt all OSDs with bigger DBs (31GB). And we limit the amount of
slow bytes, with manual compactions.
With the big amount of small objects gone, the cluster was still
unhealthy. Then we realized that the RGW Garbage Collector did not keep
up with the load. A possible reason for the GC backlog: customers using
features like versioning more heavily than before.
There are high refcounts in GC, and there were times with lots of HEAD
requests from some customers.
GC load is mostly read load. Combined with only low write activity, this
may be problematic.
We tuned GC up and the backlog is going down now, slowly (again this
takes weeks).
Cheers
Harry
8:55 p.m.
On Fri, Oct 9, 2020 at 1:42 PM Harald Staub <harald.staub(a)switch.ch> wrote:
On 07.10.20 21:00, Wido den Hollander wrote:
Thanks Wido and Harald for the info. On our side the issue was not so
severe -- it started with huge buffer_anon across the OSDs, but then
after restarting an affected osd a similar amount of memory would be
accounted in the osd_pglog mempool.
We have mitigated by keeping only 500 pglog entries for now, and at
the moment things are looking quite stable, without any mempools
leaking.
I also noticed a possible relationship with scrubbing -- One week ago
we increased to osd_max_scrubs=5 to clear out a scrubbing backlog; I
wonder if the increased read/write ratio somehow led to an exploding
buffer_anon. Do things stabilize on your side if you temporarily
disable scrubbing?
Otherwise, we've just instrumented the OSDs on this cluster so we can
track all the mempools in grafana. If we learn anything we'll share
that here.
Cheers, Dan
On 07/10/2020 16:00, Dan van der Ster wrote:
> On Wed, Oct 7, 2020 at 3:29 PM Wido den Hollander <wido(a)42on.com> wrote:
>>
>>
>>
>> On 07/10/2020 14:08, Dan van der Ster wrote:
>>> Hi all,
>>>
>>> This morning some osds in our S3 cluster started going OOM, after
>>> restarting them I noticed that the osd_pglog is using >1.5GB per osd.
>>> (This is on an osd with osd_memory_target = 2GB, hosting 112PGs, all
>>> PGs are active+clean).
[...]
Hi all,
As Wido said, our case may be a bit different.
This is still on 14.2.8. Trouble started with lots of small objects.
There were 2 Veeam buckets with more than 400M objects each,
on a pool with EC 8+3. This means that there were about 10 billion
object shards. DB space on SSD was tiny (originally built for filestore,
there was space for 25GB, i.e. only 3GB really usable as we know now).
Then OSD memory started to grow, mostly buffer_anon. Decreasing
osd_max_pg_log_entries helped (with buffer_anon!). We added RAM, only to
have more OOMs a few days later. And we realized that DB slow bytes had
started to grow, without bounds.
We could delete the objects (took several weeks), and there were no OOMs
during this time. But afterwards again growing buffer_anon.
Once I observed free memory improving when a customer was writing
heavily. So we started to write constantly (small objects to dummy
buckets). This helps with buffer_anon and also with the DB growth.
It seems that at least 14.2.8 does not trim buffer_anon periodically,
but only when writing:
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/EVPELHOL4K…
One possible explanation (just an idea) for the large amount of
buffer_anon: DB slow bytes got spread over lots and lots of small
allocations on the HDD.
We rebuilt all OSDs with bigger DBs (31GB). And we limit the amount of
slow bytes, with manual compactions.
With the big amount of small objects gone, the cluster was still
unhealthy. Then we realized that the RGW Garbage Collector did not keep
up with the load. A possible reason for the GC backlog: customers using
features like versioning more heavily than before.
There are high refcounts in GC, and there were times with lots of HEAD
requests from some customers.
GC load is mostly read load. Combined with only low write activity, this
may be problematic.
We tuned GC up and the backlog is going down now, slowly (again this
takes weeks).
9:44 p.m.
On 09.10.20 13:55, Dan van der Ster wrote:
[...]
I also noticed a possible relationship with scrubbing
-- One week ago
we increased to osd_max_scrubs=5 to clear out a scrubbing backlog; I
wonder if the increased read/write ratio somehow led to an exploding
buffer_anon. Do things stabilize on your side if you temporarily
disable scrubbing?
During the worst periods, we had disabled scrubbing. When we re-enabled,
we had our write-job to mitigate the problems. And currently, scrub load
is low. So I cannot tell, but it is very plausible.
Cheers
Harry
10:11 p.m.
1. The pg log contains 3000 entries by default (on
nautilus). These
3000 entries can legitimately consume gigabytes of ram for some
use-cases. (I haven't determined exactly which ops triggered this
today).
How can I check how much ram my pg_logs are using?
-----Original Message-----
Cc: ceph-users
Subject: [ceph-users] Re: another osd_pglog memory usage incident
On 09.10.20 13:55, Dan van der Ster wrote:
[...]
I also noticed a possible relationship with scrubbing
-- One week ago
we increased to osd_max_scrubs=5 to clear out a scrubbing backlog; I
wonder if the increased read/write ratio somehow led to an exploding
buffer_anon. Do things stabilize on your side if you temporarily
disable scrubbing?
During the worst periods, we had disabled scrubbing. When we re-enabled,
we had our write-job to mitigate the problems. And currently, scrub load
is low. So I cannot tell, but it is very plausible.
Cheers
Harry
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an
email to ceph-users-leave(a)ceph.io
1288
days inactive
1290
days old
7 comments
4 participants
participants (4)
-
Dan van der Ster -
Harald Staub -
Marc Roos -
Wido den Hollander