osd_memory_target ignored
I recently upgraded from 13.2.2 to 13.2.8 and observe two changes that I struggle with: - from release notes: The bluestore_cache_* options are no longer needed. They are replaced by osd_memory_target, defaulting to 4GB. - the default for bluestore_allocator has changed from stupid to bitmap, which seem to conflict each other, or at least I seem unable to achieve what I want. I have a number of OSDs for which I would like to increase the cache size. In the past I used bluestore_cache_size=8G and it worked like a charm. I now changed that to osd_memory_target=8G without any effect. The usage stays at 4G and the virtual size is about 5G. I would expect both to be close to 8G. The read cache for these OSDs usually fills up within a few hours. The cluster is now running a few days with the new configs to no avail. The documentation of osd_memory_target refers to tcmalloc a lot. Is this in conflict with allocator=bitmap? If so, what is the way to tune cache sizes (say if tcmalloc is not used/how to check?)? Are bluestore_cache_* indeed obsolete as the above release notes suggest, or is this not true? Many thanks for your help. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
Hi, Quoting Frank Schilder (frans@dtu.dk):
I recently upgraded from 13.2.2 to 13.2.8 and observe two changes that I struggle with:
- from release notes: The bluestore_cache_* options are no longer needed. They are replaced by osd_memory_target, defaulting to 4GB. - the default for bluestore_allocator has changed from stupid to bitmap,
which seem to conflict each other, or at least I seem unable to achieve what I want.
I have a number of OSDs for which I would like to increase the cache size. In the past I used bluestore_cache_size=8G and it worked like a charm. I now changed that to osd_memory_target=8G without any effect. The usage stays at 4G and the virtual size is about 5G. I would expect both to be close to 8G. The read cache for these OSDs usually fills up within a few hours. The cluster is now running a few days with the new configs to no avail.
How do you check the memory usage? We have a osd_memory_target=11G and the OSDs consume this exact amount of RAM (ps aux |grep osd). We are running 13.2.8. ceph daemon osd.$id dump_mempools would give ~ 4 GiB of RAM. So there is more RAM usage than only specified by "mempool" obviously.
The documentation of osd_memory_target refers to tcmalloc a lot. Is this in conflict with allocator=bitmap? If so, what is the way to tune cache sizes (say if tcmalloc is not used/how to check?)? Are bluestore_cache_* indeed obsolete as the above release notes suggest, or is this not true?
AFAIK these are not related. We use "bluefs_allocator": "bitmap" and "bluestore_allocator": "bitmap". Gr. Stefan -- | BIT BV https://www.bit.nl/ Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / info@bit.nl
Dear Stefan, I check with top the total allocation. ps -aux gives: USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND ceph 784155 15.8 3.1 6014276 4215008 ? Sl Jan31 932:13 /usr/bin/ceph-osd --cluster ceph -f -i 243 ... ceph 784732 16.6 3.0 6058736 4082504 ? Sl Jan31 976:59 /usr/bin/ceph-osd --cluster ceph -f -i 247 ... ceph 785812 17.1 3.0 5989576 3959996 ? Sl Jan31 1008:46 /usr/bin/ceph-osd --cluster ceph -f -i 254 ... ceph 786352 14.9 3.1 5955520 4132840 ? Sl Jan31 874:37 /usr/bin/ceph-osd --cluster ceph -f -i 256 ... These should have 8GB resident by now, but stay at or just below 4G. The other options are set as [root@ceph-04 ~]# ceph config get osd.243 bluefs_allocator bitmap [root@ceph-04 ~]# ceph config get osd.243 bluestore_allocator bitmap [root@ceph-04 ~]# ceph config get osd.243 osd_memory_target 8589934592 Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Stefan Kooman <stefan@bit.nl> Sent: 04 February 2020 16:34:34 To: Frank Schilder Cc: ceph-users Subject: Re: [ceph-users] osd_memory_target ignored Hi, Quoting Frank Schilder (frans@dtu.dk):
I recently upgraded from 13.2.2 to 13.2.8 and observe two changes that I struggle with:
- from release notes: The bluestore_cache_* options are no longer needed. They are replaced by osd_memory_target, defaulting to 4GB. - the default for bluestore_allocator has changed from stupid to bitmap,
which seem to conflict each other, or at least I seem unable to achieve what I want.
I have a number of OSDs for which I would like to increase the cache size. In the past I used bluestore_cache_size=8G and it worked like a charm. I now changed that to osd_memory_target=8G without any effect. The usage stays at 4G and the virtual size is about 5G. I would expect both to be close to 8G. The read cache for these OSDs usually fills up within a few hours. The cluster is now running a few days with the new configs to no avail.
How do you check the memory usage? We have a osd_memory_target=11G and the OSDs consume this exact amount of RAM (ps aux |grep osd). We are running 13.2.8. ceph daemon osd.$id dump_mempools would give ~ 4 GiB of RAM. So there is more RAM usage than only specified by "mempool" obviously.
The documentation of osd_memory_target refers to tcmalloc a lot. Is this in conflict with allocator=bitmap? If so, what is the way to tune cache sizes (say if tcmalloc is not used/how to check?)? Are bluestore_cache_* indeed obsolete as the above release notes suggest, or is this not true?
AFAIK these are not related. We use "bluefs_allocator": "bitmap" and "bluestore_allocator": "bitmap". Gr. Stefan -- | BIT BV https://www.bit.nl/ Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / info@bit.nl
Quoting Frank Schilder (frans@dtu.dk):
Dear Stefan,
I check with top the total allocation. ps -aux gives:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND ceph 784155 15.8 3.1 6014276 4215008 ? Sl Jan31 932:13 /usr/bin/ceph-osd --cluster ceph -f -i 243 ... ceph 784732 16.6 3.0 6058736 4082504 ? Sl Jan31 976:59 /usr/bin/ceph-osd --cluster ceph -f -i 247 ... ceph 785812 17.1 3.0 5989576 3959996 ? Sl Jan31 1008:46 /usr/bin/ceph-osd --cluster ceph -f -i 254 ... ceph 786352 14.9 3.1 5955520 4132840 ? Sl Jan31 874:37 /usr/bin/ceph-osd --cluster ceph -f -i 256 ...
These should have 8GB resident by now, but stay at or just below 4G. The other options are set as
[root@ceph-04 ~]# ceph config get osd.243 bluefs_allocator bitmap [root@ceph-04 ~]# ceph config get osd.243 bluestore_allocator bitmap [root@ceph-04 ~]# ceph config get osd.243 osd_memory_target 8589934592
What does "bluestore_cache_size" read? Our OSDs report "0". Gr. Stefan -- | BIT BV https://www.bit.nl/ Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / info@bit.nl
Hi Stefan, its all at the defaults it seems: [root@gnosis ~]# ceph config get osd.243 bluestore_cache_size 0 [root@gnosis ~]# ceph config get osd.243 bluestore_cache_size_ssd 3221225472 I explicitly removed the old settings with commands like ceph config rm osd.243 bluestore_cache_size Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Stefan Kooman <stefan@bit.nl> Sent: 04 February 2020 21:14:28 To: Frank Schilder Cc: ceph-users Subject: Re: [ceph-users] osd_memory_target ignored Quoting Frank Schilder (frans@dtu.dk):
Dear Stefan,
I check with top the total allocation. ps -aux gives:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND ceph 784155 15.8 3.1 6014276 4215008 ? Sl Jan31 932:13 /usr/bin/ceph-osd --cluster ceph -f -i 243 ... ceph 784732 16.6 3.0 6058736 4082504 ? Sl Jan31 976:59 /usr/bin/ceph-osd --cluster ceph -f -i 247 ... ceph 785812 17.1 3.0 5989576 3959996 ? Sl Jan31 1008:46 /usr/bin/ceph-osd --cluster ceph -f -i 254 ... ceph 786352 14.9 3.1 5955520 4132840 ? Sl Jan31 874:37 /usr/bin/ceph-osd --cluster ceph -f -i 256 ...
These should have 8GB resident by now, but stay at or just below 4G. The other options are set as
[root@ceph-04 ~]# ceph config get osd.243 bluefs_allocator bitmap [root@ceph-04 ~]# ceph config get osd.243 bluestore_allocator bitmap [root@ceph-04 ~]# ceph config get osd.243 osd_memory_target 8589934592
What does "bluestore_cache_size" read? Our OSDs report "0". Gr. Stefan -- | BIT BV https://www.bit.nl/ Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / info@bit.nl
Dear Stefan, is it possible that there is a mistake in the documentation or a bug? Out of curiosity, I restarted one of these OSDs and the memory usage starts going up: ceph 881203 15.4 4.0 6201580 5344764 ? Sl 09:18 6:38 /usr/bin/ceph-osd --cluster ceph -f -i 243 --setuser ceph --setgroup disk The documentation of ods_memory_target says "Can update at runtime: true", but it seems that a restart is required to activate the setting, so it can *not* be updated at runtime (meaning it takes effect without restart). In addition to that, I would like to have different default memory targets set for different device classes. Unfortunately, there seem not to be different memory_target_[devide class] default options. Is there a good way to set different while avoiding to bloat "ceph config dump" unnecessarily? Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: 05 February 2020 09:09:22 To: Stefan Kooman Cc: ceph-users Subject: [ceph-users] Re: osd_memory_target ignored Hi Stefan, its all at the defaults it seems: [root@gnosis ~]# ceph config get osd.243 bluestore_cache_size 0 [root@gnosis ~]# ceph config get osd.243 bluestore_cache_size_ssd 3221225472 I explicitly removed the old settings with commands like ceph config rm osd.243 bluestore_cache_size Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Stefan Kooman <stefan@bit.nl> Sent: 04 February 2020 21:14:28 To: Frank Schilder Cc: ceph-users Subject: Re: [ceph-users] osd_memory_target ignored Quoting Frank Schilder (frans@dtu.dk):
Dear Stefan,
I check with top the total allocation. ps -aux gives:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND ceph 784155 15.8 3.1 6014276 4215008 ? Sl Jan31 932:13 /usr/bin/ceph-osd --cluster ceph -f -i 243 ... ceph 784732 16.6 3.0 6058736 4082504 ? Sl Jan31 976:59 /usr/bin/ceph-osd --cluster ceph -f -i 247 ... ceph 785812 17.1 3.0 5989576 3959996 ? Sl Jan31 1008:46 /usr/bin/ceph-osd --cluster ceph -f -i 254 ... ceph 786352 14.9 3.1 5955520 4132840 ? Sl Jan31 874:37 /usr/bin/ceph-osd --cluster ceph -f -i 256 ...
These should have 8GB resident by now, but stay at or just below 4G. The other options are set as
[root@ceph-04 ~]# ceph config get osd.243 bluefs_allocator bitmap [root@ceph-04 ~]# ceph config get osd.243 bluestore_allocator bitmap [root@ceph-04 ~]# ceph config get osd.243 osd_memory_target 8589934592
What does "bluestore_cache_size" read? Our OSDs report "0". Gr. Stefan -- | BIT BV https://www.bit.nl/ Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / info@bit.nl _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Quoting Frank Schilder (frans@dtu.dk):
Dear Stefan,
is it possible that there is a mistake in the documentation or a bug? Out of curiosity, I restarted one of these OSDs and the memory usage starts going up:
ceph 881203 15.4 4.0 6201580 5344764 ? Sl 09:18 6:38 /usr/bin/ceph-osd --cluster ceph -f -i 243 --setuser ceph --setgroup disk
The documentation of ods_memory_target says "Can update at runtime: true", but it seems that a restart is required to activate the setting, so it can *not* be updated at runtime (meaning it takes effect without restart).
Ah, that might be. If the documentation states it can be updated at runtime it's a bug (in eiter the code or the documentation).
In addition to that, I would like to have different default memory targets set for different device classes. Unfortunately, there seem not to be different memory_target_[devide class] default options. Is there a good way to set different while avoiding to bloat "ceph config dump" unnecessarily?
I'm afraid not. You might want to file a tracker issue with an enhancement request. Gr. Stefan -- | BIT BV https://www.bit.nl/ Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / info@bit.nl
Dear Stefan, thanks for your help. I opened these: https://tracker.ceph.com/issues/44010 https://tracker.ceph.com/issues/44011 Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Stefan Kooman <stefan@bit.nl> Sent: 05 February 2020 10:29 To: Frank Schilder Cc: ceph-users Subject: Re: [ceph-users] Re: osd_memory_target ignored Quoting Frank Schilder (frans@dtu.dk):
Dear Stefan,
is it possible that there is a mistake in the documentation or a bug? Out of curiosity, I restarted one of these OSDs and the memory usage starts going up:
ceph 881203 15.4 4.0 6201580 5344764 ? Sl 09:18 6:38 /usr/bin/ceph-osd --cluster ceph -f -i 243 --setuser ceph --setgroup disk
The documentation of ods_memory_target says "Can update at runtime: true", but it seems that a restart is required to activate the setting, so it can *not* be updated at runtime (meaning it takes effect without restart).
Ah, that might be. If the documentation states it can be updated at runtime it's a bug (in eiter the code or the documentation).
In addition to that, I would like to have different default memory targets set for different device classes. Unfortunately, there seem not to be different memory_target_[devide class] default options. Is there a good way to set different while avoiding to bloat "ceph config dump" unnecessarily?
I'm afraid not. You might want to file a tracker issue with an enhancement request. Gr. Stefan -- | BIT BV https://www.bit.nl/ Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / info@bit.nl
For everybody reading this: It is already possible to set configuration options based on device class using masks, for example, ceph config set osd/class:hdd osd_memory_target 2147483648 will set a memory target for HDDs only. ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: 06 February 2020 10:40 To: Stefan Kooman Cc: ceph-users Subject: [ceph-users] Re: osd_memory_target ignored Dear Stefan, thanks for your help. I opened these: https://tracker.ceph.com/issues/44010 https://tracker.ceph.com/issues/44011 Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
participants (2)
-
Frank Schilder
-
Stefan Kooman