Ceph 16.2.14: how to set mon_rocksdb_options to enable RocksDB compression?
Hi, I'm still trying to fight large Ceph monitor writes. One option I considered is enabling RocksDB compression, as our nodes have more than sufficient RAM and CPU. Unfortunately, monitors seem to completely ignore the compression setting: I tried: - setting ceph config set mon.ceph05 mon_rocksdb_options "write_buffer_size=33554432,compression=kLZ4Compression,level_compaction_dynamic_level_bytes=true", restarting the test monitor. The monitor started with no RocksDB compression: debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: Compression algorithms supported: debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: kZSTDNotFinalCompression supported: 0 debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: kXpressCompression supported: 0 debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: kLZ4HCCompression supported: 1 debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: kLZ4Compression supported: 1 debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: kBZip2Compression supported: 0 debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: kZlibCompression supported: 1 debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: kSnappyCompression supported: 1 ... debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: Options.compression: NoCompression debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: Options.bottommost_compression: Disabled - setting ceph config set mon mon_rocksdb_options "write_buffer_size=33554432,compression=kLZ4Compression,level_compaction_dynamic_level_bytes=true", restarting the test monitor. The monitor started with no RocksDB compression, the same way as above. In each case config options were correctly set and readable with config get. I also found a suggestion in ceph-users ( https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/KJM232IHN7F...) to set compression in a similar manner. Unfortunately, these options appear to be ignored. How can I enable RocksDB compression in Ceph monitors? I would very much appreciate your advices and comments. Best regards, Zakhar
I also tried setting RocksDB compression options and deploying a new monitor. The monitor started with no RocksDB compression again. Ceph monitors seem to ignore mon_rocksdb_options set at runtime, at mon start and at mon deploy. How can I enable RocksDB compression in Ceph monitors? Any input from anyone, please? /Z On Fri, 13 Oct 2023 at 23:01, Zakhar Kirpichenko <zakhar@gmail.com> wrote:
Hi,
I'm still trying to fight large Ceph monitor writes. One option I considered is enabling RocksDB compression, as our nodes have more than sufficient RAM and CPU. Unfortunately, monitors seem to completely ignore the compression setting:
I tried:
- setting ceph config set mon.ceph05 mon_rocksdb_options "write_buffer_size=33554432,compression=kLZ4Compression,level_compaction_dynamic_level_bytes=true", restarting the test monitor. The monitor started with no RocksDB compression:
debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: Compression algorithms supported: debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: kZSTDNotFinalCompression supported: 0 debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: kXpressCompression supported: 0 debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: kLZ4HCCompression supported: 1 debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: kLZ4Compression supported: 1 debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: kBZip2Compression supported: 0 debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: kZlibCompression supported: 1 debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: kSnappyCompression supported: 1 ... debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: Options.compression: NoCompression debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: Options.bottommost_compression: Disabled
- setting ceph config set mon mon_rocksdb_options "write_buffer_size=33554432,compression=kLZ4Compression,level_compaction_dynamic_level_bytes=true", restarting the test monitor. The monitor started with no RocksDB compression, the same way as above.
In each case config options were correctly set and readable with config get. I also found a suggestion in ceph-users ( https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/KJM232IHN7F...) to set compression in a similar manner. Unfortunately, these options appear to be ignored.
How can I enable RocksDB compression in Ceph monitors?
I would very much appreciate your advices and comments.
Best regards, Zakhar
Apologies if you tried this already and I missed it - have you tried configuring that setting in /etc/ceph/ceph.conf (or wherever your conf file is) instead of via 'ceph config'? I wonder if mon settings like this one won't actually apply the way you want because they're needed before the mon has the ability to obtain configuration from, effectively, itself. Josh On Sat, Oct 14, 2023 at 1:32 AM Zakhar Kirpichenko <zakhar@gmail.com> wrote:
I also tried setting RocksDB compression options and deploying a new monitor. The monitor started with no RocksDB compression again.
Ceph monitors seem to ignore mon_rocksdb_options set at runtime, at mon start and at mon deploy. How can I enable RocksDB compression in Ceph monitors?
Any input from anyone, please?
/Z
On Fri, 13 Oct 2023 at 23:01, Zakhar Kirpichenko <zakhar@gmail.com> wrote:
Hi,
I'm still trying to fight large Ceph monitor writes. One option I considered is enabling RocksDB compression, as our nodes have more than sufficient RAM and CPU. Unfortunately, monitors seem to completely ignore the compression setting:
I tried:
- setting ceph config set mon.ceph05 mon_rocksdb_options "write_buffer_size=33554432,compression=kLZ4Compression,level_compaction_dynamic_level_bytes=true", restarting the test monitor. The monitor started with no RocksDB compression:
debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: Compression algorithms supported: debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: kZSTDNotFinalCompression supported: 0 debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: kXpressCompression supported: 0 debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: kLZ4HCCompression supported: 1 debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: kLZ4Compression supported: 1 debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: kBZip2Compression supported: 0 debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: kZlibCompression supported: 1 debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: kSnappyCompression supported: 1 ... debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: Options.compression: NoCompression debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: Options.bottommost_compression: Disabled
- setting ceph config set mon mon_rocksdb_options "write_buffer_size=33554432,compression=kLZ4Compression,level_compaction_dynamic_level_bytes=true", restarting the test monitor. The monitor started with no RocksDB compression, the same way as above.
In each case config options were correctly set and readable with config get. I also found a suggestion in ceph-users ( https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/KJM232IHN7F...) to set compression in a similar manner. Unfortunately, these options appear to be ignored.
How can I enable RocksDB compression in Ceph monitors?
I would very much appreciate your advices and comments.
Best regards, Zakhar
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks for your response, Josh. Our ceph.conf doesn't have anything but the mon addresses, modern Ceph versions store their configuration in the monitor configuration database. This works rather well for various Ceph components, including the monitors. RocksDB options are also applied to monitors correctly, but for some reason are being ignored. /Z On Sat, 14 Oct 2023, 17:40 Josh Baergen, <jbaergen@digitalocean.com> wrote:
Apologies if you tried this already and I missed it - have you tried configuring that setting in /etc/ceph/ceph.conf (or wherever your conf file is) instead of via 'ceph config'? I wonder if mon settings like this one won't actually apply the way you want because they're needed before the mon has the ability to obtain configuration from, effectively, itself.
Josh
On Sat, Oct 14, 2023 at 1:32 AM Zakhar Kirpichenko <zakhar@gmail.com> wrote:
I also tried setting RocksDB compression options and deploying a new monitor. The monitor started with no RocksDB compression again.
Ceph monitors seem to ignore mon_rocksdb_options set at runtime, at mon start and at mon deploy. How can I enable RocksDB compression in Ceph monitors?
Any input from anyone, please?
/Z
On Fri, 13 Oct 2023 at 23:01, Zakhar Kirpichenko <zakhar@gmail.com>
wrote:
Hi,
I'm still trying to fight large Ceph monitor writes. One option I considered is enabling RocksDB compression, as our nodes have more than sufficient RAM and CPU. Unfortunately, monitors seem to completely
ignore
the compression setting:
I tried:
- setting ceph config set mon.ceph05 mon_rocksdb_options
"write_buffer_size=33554432,compression=kLZ4Compression,level_compaction_dynamic_level_bytes=true",
restarting the test monitor. The monitor started with no RocksDB compression:
debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: Compression algorithms supported: debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: kZSTDNotFinalCompression supported: 0 debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: kXpressCompression supported: 0 debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: kLZ4HCCompression supported: 1 debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: kLZ4Compression supported: 1 debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: kBZip2Compression supported: 0 debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: kZlibCompression supported: 1 debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: kSnappyCompression supported: 1 ... debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: Options.compression: NoCompression debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: Options.bottommost_compression: Disabled
- setting ceph config set mon mon_rocksdb_options
"write_buffer_size=33554432,compression=kLZ4Compression,level_compaction_dynamic_level_bytes=true",
restarting the test monitor. The monitor started with no RocksDB compression, the same way as above.
In each case config options were correctly set and readable with config get. I also found a suggestion in ceph-users (
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/KJM232IHN7F... )
to set compression in a similar manner. Unfortunately, these options appear to be ignored.
How can I enable RocksDB compression in Ceph monitors?
I would very much appreciate your advices and comments.
Best regards, Zakhar
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Out of curiosity, I tried setting mon_rocksdb_options via ceph.conf. This didn't work either: ceph.conf gets overridden at monitor start, the resulting ceph.conf inside the monitor container doesn't have mon_rocksdb_options, the monitor starts with no RocksDB compression. I would appreciate it if someone from the Ceph team could please chip in and suggest a working way to enable RocksDB compression in Ceph monitors. /Z On Sat, 14 Oct 2023 at 19:16, Zakhar Kirpichenko <zakhar@gmail.com> wrote:
Thanks for your response, Josh. Our ceph.conf doesn't have anything but the mon addresses, modern Ceph versions store their configuration in the monitor configuration database.
This works rather well for various Ceph components, including the monitors. RocksDB options are also applied to monitors correctly, but for some reason are being ignored.
/Z
On Sat, 14 Oct 2023, 17:40 Josh Baergen, <jbaergen@digitalocean.com> wrote:
Apologies if you tried this already and I missed it - have you tried configuring that setting in /etc/ceph/ceph.conf (or wherever your conf file is) instead of via 'ceph config'? I wonder if mon settings like this one won't actually apply the way you want because they're needed before the mon has the ability to obtain configuration from, effectively, itself.
Josh
On Sat, Oct 14, 2023 at 1:32 AM Zakhar Kirpichenko <zakhar@gmail.com> wrote:
I also tried setting RocksDB compression options and deploying a new monitor. The monitor started with no RocksDB compression again.
Ceph monitors seem to ignore mon_rocksdb_options set at runtime, at mon start and at mon deploy. How can I enable RocksDB compression in Ceph monitors?
Any input from anyone, please?
/Z
On Fri, 13 Oct 2023 at 23:01, Zakhar Kirpichenko <zakhar@gmail.com>
wrote:
Hi,
I'm still trying to fight large Ceph monitor writes. One option I considered is enabling RocksDB compression, as our nodes have more
than
sufficient RAM and CPU. Unfortunately, monitors seem to completely ignore the compression setting:
I tried:
- setting ceph config set mon.ceph05 mon_rocksdb_options
"write_buffer_size=33554432,compression=kLZ4Compression,level_compaction_dynamic_level_bytes=true",
restarting the test monitor. The monitor started with no RocksDB compression:
debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: Compression algorithms supported: debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: kZSTDNotFinalCompression supported: 0 debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: kXpressCompression supported: 0 debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: kLZ4HCCompression supported: 1 debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: kLZ4Compression supported: 1 debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: kBZip2Compression supported: 0 debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: kZlibCompression supported: 1 debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: kSnappyCompression supported: 1 ... debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: Options.compression: NoCompression debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: Options.bottommost_compression: Disabled
- setting ceph config set mon mon_rocksdb_options
"write_buffer_size=33554432,compression=kLZ4Compression,level_compaction_dynamic_level_bytes=true",
restarting the test monitor. The monitor started with no RocksDB compression, the same way as above.
In each case config options were correctly set and readable with config get. I also found a suggestion in ceph-users (
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/KJM232IHN7F... )
to set compression in a similar manner. Unfortunately, these options appear to be ignored.
How can I enable RocksDB compression in Ceph monitors?
I would very much appreciate your advices and comments.
Best regards, Zakhar
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
the resulting ceph.conf inside the monitor container doesn't have mon_rocksdb_options
I don't know where this particular ceph.conf copy comes from, but I still suspect that this is where this particular option needs to be set. The reason I think this is that rocksdb mount options are needed _before_ the mon is able to access any of the centralized conf data, which I believe is itself stored in rocksdb. Josh On Sun, Oct 15, 2023 at 10:29 PM Zakhar Kirpichenko <zakhar@gmail.com> wrote:
Out of curiosity, I tried setting mon_rocksdb_options via ceph.conf. This didn't work either: ceph.conf gets overridden at monitor start, the resulting ceph.conf inside the monitor container doesn't have mon_rocksdb_options, the monitor starts with no RocksDB compression.
I would appreciate it if someone from the Ceph team could please chip in and suggest a working way to enable RocksDB compression in Ceph monitors.
/Z
On Sat, 14 Oct 2023 at 19:16, Zakhar Kirpichenko <zakhar@gmail.com> wrote:
Thanks for your response, Josh. Our ceph.conf doesn't have anything but the mon addresses, modern Ceph versions store their configuration in the monitor configuration database.
This works rather well for various Ceph components, including the monitors. RocksDB options are also applied to monitors correctly, but for some reason are being ignored.
/Z
On Sat, 14 Oct 2023, 17:40 Josh Baergen, <jbaergen@digitalocean.com> wrote:
Apologies if you tried this already and I missed it - have you tried configuring that setting in /etc/ceph/ceph.conf (or wherever your conf file is) instead of via 'ceph config'? I wonder if mon settings like this one won't actually apply the way you want because they're needed before the mon has the ability to obtain configuration from, effectively, itself.
Josh
On Sat, Oct 14, 2023 at 1:32 AM Zakhar Kirpichenko <zakhar@gmail.com> wrote:
I also tried setting RocksDB compression options and deploying a new monitor. The monitor started with no RocksDB compression again.
Ceph monitors seem to ignore mon_rocksdb_options set at runtime, at mon start and at mon deploy. How can I enable RocksDB compression in Ceph monitors?
Any input from anyone, please?
/Z
On Fri, 13 Oct 2023 at 23:01, Zakhar Kirpichenko <zakhar@gmail.com> wrote:
Hi,
I'm still trying to fight large Ceph monitor writes. One option I considered is enabling RocksDB compression, as our nodes have more than sufficient RAM and CPU. Unfortunately, monitors seem to completely ignore the compression setting:
I tried:
- setting ceph config set mon.ceph05 mon_rocksdb_options "write_buffer_size=33554432,compression=kLZ4Compression,level_compaction_dynamic_level_bytes=true", restarting the test monitor. The monitor started with no RocksDB compression:
debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: Compression algorithms supported: debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: kZSTDNotFinalCompression supported: 0 debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: kXpressCompression supported: 0 debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: kLZ4HCCompression supported: 1 debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: kLZ4Compression supported: 1 debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: kBZip2Compression supported: 0 debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: kZlibCompression supported: 1 debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: kSnappyCompression supported: 1 ... debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: Options.compression: NoCompression debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: Options.bottommost_compression: Disabled
- setting ceph config set mon mon_rocksdb_options "write_buffer_size=33554432,compression=kLZ4Compression,level_compaction_dynamic_level_bytes=true", restarting the test monitor. The monitor started with no RocksDB compression, the same way as above.
In each case config options were correctly set and readable with config get. I also found a suggestion in ceph-users ( https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/KJM232IHN7F...) to set compression in a similar manner. Unfortunately, these options appear to be ignored.
How can I enable RocksDB compression in Ceph monitors?
I would very much appreciate your advices and comments.
Best regards, Zakhar
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks for the suggestion, Josh! That part is relatively simple: the container gets ceph.conf from the host's filesystem, for example: "HostConfig": { "Binds": [ "/dev:/dev", "/run/udev:/run/udev", "/var/run/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86:/var/run/ceph:z", "/var/log/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86:/var/log/ceph:z", "/var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/crash:/var/lib/ceph/crash:z", "/var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/mon.ceph05:/var/lib/ceph/mon/ceph-ceph05:z", "/var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/mon.ceph05/config:/etc/ceph/ceph.conf:z" ], When I stop the monitor, edit the file directly and restart the monitor, mon_rocksdb_options seem to be applied correctly! Unfortunately, I specify global mon_rocksdb_options and redeploy the monitor, the new ceph.conf doesn't have mon_rocksdb_options at all. I am not sure that this is a reliable way to enable compression, but it works - so it's better than other ways which don't work :-) /Z On Mon, 16 Oct 2023 at 16:16, Josh Baergen <jbaergen@digitalocean.com> wrote:
the resulting ceph.conf inside the monitor container doesn't have mon_rocksdb_options
I don't know where this particular ceph.conf copy comes from, but I still suspect that this is where this particular option needs to be set. The reason I think this is that rocksdb mount options are needed _before_ the mon is able to access any of the centralized conf data, which I believe is itself stored in rocksdb.
Josh
On Sun, Oct 15, 2023 at 10:29 PM Zakhar Kirpichenko <zakhar@gmail.com> wrote:
Out of curiosity, I tried setting mon_rocksdb_options via ceph.conf.
This didn't work either: ceph.conf gets overridden at monitor start, the resulting ceph.conf inside the monitor container doesn't have mon_rocksdb_options, the monitor starts with no RocksDB compression.
I would appreciate it if someone from the Ceph team could please chip in
and suggest a working way to enable RocksDB compression in Ceph monitors.
/Z
On Sat, 14 Oct 2023 at 19:16, Zakhar Kirpichenko <zakhar@gmail.com>
Thanks for your response, Josh. Our ceph.conf doesn't have anything but
This works rather well for various Ceph components, including the
monitors. RocksDB options are also applied to monitors correctly, but for some reason are being ignored.
/Z
On Sat, 14 Oct 2023, 17:40 Josh Baergen, <jbaergen@digitalocean.com>
wrote:
Apologies if you tried this already and I missed it - have you tried configuring that setting in /etc/ceph/ceph.conf (or wherever your conf file is) instead of via 'ceph config'? I wonder if mon settings like this one won't actually apply the way you want because they're needed before the mon has the ability to obtain configuration from, effectively, itself.
Josh
On Sat, Oct 14, 2023 at 1:32 AM Zakhar Kirpichenko <zakhar@gmail.com>
wrote:
I also tried setting RocksDB compression options and deploying a new monitor. The monitor started with no RocksDB compression again.
Ceph monitors seem to ignore mon_rocksdb_options set at runtime, at
mon
start and at mon deploy. How can I enable RocksDB compression in Ceph monitors?
Any input from anyone, please?
/Z
On Fri, 13 Oct 2023 at 23:01, Zakhar Kirpichenko <zakhar@gmail.com> wrote:
Hi,
I'm still trying to fight large Ceph monitor writes. One option I considered is enabling RocksDB compression, as our nodes have more
sufficient RAM and CPU. Unfortunately, monitors seem to completely ignore the compression setting:
I tried:
- setting ceph config set mon.ceph05 mon_rocksdb_options
"write_buffer_size=33554432,compression=kLZ4Compression,level_compaction_dynamic_level_bytes=true",
restarting the test monitor. The monitor started with no RocksDB compression:
debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: Compression algorithms supported: debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: kZSTDNotFinalCompression supported: 0 debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: kXpressCompression supported: 0 debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: kLZ4HCCompression supported: 1 debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: kLZ4Compression supported: 1 debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: kBZip2Compression supported: 0 debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: kZlibCompression supported: 1 debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: kSnappyCompression supported: 1 ... debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: Options.compression: NoCompression debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: Options.bottommost_compression: Disabled
- setting ceph config set mon mon_rocksdb_options
"write_buffer_size=33554432,compression=kLZ4Compression,level_compaction_dynamic_level_bytes=true",
restarting the test monitor. The monitor started with no RocksDB compression, the same way as above.
In each case config options were correctly set and readable with config get. I also found a suggestion in ceph-users (
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/KJM232IHN7F... )
to set compression in a similar manner. Unfortunately, these
wrote: the mon addresses, modern Ceph versions store their configuration in the monitor configuration database. than options appear
to be ignored.
How can I enable RocksDB compression in Ceph monitors?
I would very much appreciate your advices and comments.
Best regards, Zakhar
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, I managed to get the compression setting into the MONs by using the extra-entrypoint-arguments [1]: ceph01:~ # cat mon-specs.yaml service_type: mon placement: hosts: - ceph01 - ceph02 - ceph03 extra_entrypoint_args: - '--mon-rocksdb-options=write_buffer_size=33554432,compression=kLZ4Compression,level_compaction_dynamic_level_bytes=true,bottommost_compression=kLZ4HCCompression,max_background_jobs=4,max_subcompactions=2' Just note that if you make a mistake and run 'ceph orch apply -i mon-specs.yaml' with incorrect options your MON containers will all fail. So test that in a non-critical environment first. In case the daemons fail to start you can remove those options from the unit.run file and get them up again. But for me that worked and the daemons have the compression setting enabled now. What remains unclear is which config options can be changed as usual with the config database and which require this extra-entrypoint-argument. Thanks again, Mykola! Eugen [1] https://docs.ceph.com/en/quincy/cephadm/services/#extra-entrypoint-arguments Zitat von Zakhar Kirpichenko <zakhar@gmail.com>:
Thanks for the suggestion, Josh!
That part is relatively simple: the container gets ceph.conf from the host's filesystem, for example:
"HostConfig": { "Binds": [ "/dev:/dev", "/run/udev:/run/udev",
"/var/run/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86:/var/run/ceph:z",
"/var/log/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86:/var/log/ceph:z",
"/var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/crash:/var/lib/ceph/crash:z",
"/var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/mon.ceph05:/var/lib/ceph/mon/ceph-ceph05:z",
"/var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/mon.ceph05/config:/etc/ceph/ceph.conf:z" ],
When I stop the monitor, edit the file directly and restart the monitor, mon_rocksdb_options seem to be applied correctly!
Unfortunately, I specify global mon_rocksdb_options and redeploy the monitor, the new ceph.conf doesn't have mon_rocksdb_options at all. I am not sure that this is a reliable way to enable compression, but it works - so it's better than other ways which don't work :-)
/Z
On Mon, 16 Oct 2023 at 16:16, Josh Baergen <jbaergen@digitalocean.com> wrote:
the resulting ceph.conf inside the monitor container doesn't have mon_rocksdb_options
I don't know where this particular ceph.conf copy comes from, but I still suspect that this is where this particular option needs to be set. The reason I think this is that rocksdb mount options are needed _before_ the mon is able to access any of the centralized conf data, which I believe is itself stored in rocksdb.
Josh
On Sun, Oct 15, 2023 at 10:29 PM Zakhar Kirpichenko <zakhar@gmail.com> wrote:
Out of curiosity, I tried setting mon_rocksdb_options via ceph.conf.
This didn't work either: ceph.conf gets overridden at monitor start, the resulting ceph.conf inside the monitor container doesn't have mon_rocksdb_options, the monitor starts with no RocksDB compression.
I would appreciate it if someone from the Ceph team could please chip in
and suggest a working way to enable RocksDB compression in Ceph monitors.
/Z
On Sat, 14 Oct 2023 at 19:16, Zakhar Kirpichenko <zakhar@gmail.com>
Thanks for your response, Josh. Our ceph.conf doesn't have anything but
This works rather well for various Ceph components, including the
monitors. RocksDB options are also applied to monitors correctly, but for some reason are being ignored.
/Z
On Sat, 14 Oct 2023, 17:40 Josh Baergen, <jbaergen@digitalocean.com>
wrote:
Apologies if you tried this already and I missed it - have you tried configuring that setting in /etc/ceph/ceph.conf (or wherever your conf file is) instead of via 'ceph config'? I wonder if mon settings like this one won't actually apply the way you want because they're needed before the mon has the ability to obtain configuration from, effectively, itself.
Josh
On Sat, Oct 14, 2023 at 1:32 AM Zakhar Kirpichenko <zakhar@gmail.com>
wrote:
I also tried setting RocksDB compression options and deploying a new monitor. The monitor started with no RocksDB compression again.
Ceph monitors seem to ignore mon_rocksdb_options set at runtime, at
mon
start and at mon deploy. How can I enable RocksDB compression in Ceph monitors?
Any input from anyone, please?
/Z
On Fri, 13 Oct 2023 at 23:01, Zakhar Kirpichenko <zakhar@gmail.com> wrote:
> Hi, > > I'm still trying to fight large Ceph monitor writes. One option I > considered is enabling RocksDB compression, as our nodes have more
> sufficient RAM and CPU. Unfortunately, monitors seem to completely ignore > the compression setting: > > I tried: > > - setting ceph config set mon.ceph05 mon_rocksdb_options > "write_buffer_size=33554432,compression=kLZ4Compression,level_compaction_dynamic_level_bytes=true", > restarting the test monitor. The monitor started with no RocksDB > compression: > > debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: Compression > algorithms supported: > debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: > kZSTDNotFinalCompression supported: 0 > debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: > kXpressCompression supported: 0 > debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: > kLZ4HCCompression supported: 1 > debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: > kLZ4Compression supported: 1 > debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: > kBZip2Compression supported: 0 > debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: > kZlibCompression supported: 1 > debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: > kSnappyCompression supported: 1 > ... > debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: > Options.compression: NoCompression > debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: > Options.bottommost_compression: Disabled > > - setting ceph config set mon mon_rocksdb_options > "write_buffer_size=33554432,compression=kLZ4Compression,level_compaction_dynamic_level_bytes=true", > restarting the test monitor. The monitor started with no RocksDB > compression, the same way as above. > > In each case config options were correctly set and readable with config > get. I also found a suggestion in ceph-users ( > https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/KJM232IHN7F... ) > to set compression in a similar manner. Unfortunately, these
wrote: the mon addresses, modern Ceph versions store their configuration in the monitor configuration database. than options appear
> to be ignored. > > How can I enable RocksDB compression in Ceph monitors? > > I would very much appreciate your advices and comments. > > Best regards, > Zakhar > > > _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks for this, Eugen. I think I'll stick to adding the option to the config file, it seems like a safer way to do it. /Z On Tue, 17 Oct 2023, 15:21 Eugen Block, <eblock@nde.ag> wrote:
Hi,
I managed to get the compression setting into the MONs by using the extra-entrypoint-arguments [1]:
ceph01:~ # cat mon-specs.yaml service_type: mon placement: hosts: - ceph01 - ceph02 - ceph03 extra_entrypoint_args: -
'--mon-rocksdb-options=write_buffer_size=33554432,compression=kLZ4Compression,level_compaction_dynamic_level_bytes=true,bottommost_compression=kLZ4HCCompression,max_background_jobs=4,max_subcompactions=2'
Just note that if you make a mistake and run 'ceph orch apply -i mon-specs.yaml' with incorrect options your MON containers will all fail. So test that in a non-critical environment first. In case the daemons fail to start you can remove those options from the unit.run file and get them up again. But for me that worked and the daemons have the compression setting enabled now. What remains unclear is which config options can be changed as usual with the config database and which require this extra-entrypoint-argument.
Thanks again, Mykola! Eugen
[1]
https://docs.ceph.com/en/quincy/cephadm/services/#extra-entrypoint-arguments
Zitat von Zakhar Kirpichenko <zakhar@gmail.com>:
Thanks for the suggestion, Josh!
That part is relatively simple: the container gets ceph.conf from the host's filesystem, for example:
"HostConfig": { "Binds": [ "/dev:/dev", "/run/udev:/run/udev",
"/var/run/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86:/var/run/ceph:z",
"/var/log/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86:/var/log/ceph:z",
"/var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/crash:/var/lib/ceph/crash:z",
"/var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/mon.ceph05:/var/lib/ceph/mon/ceph-ceph05:z",
"/var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/mon.ceph05/config:/etc/ceph/ceph.conf:z"
],
When I stop the monitor, edit the file directly and restart the monitor, mon_rocksdb_options seem to be applied correctly!
Unfortunately, I specify global mon_rocksdb_options and redeploy the monitor, the new ceph.conf doesn't have mon_rocksdb_options at all. I am not sure that this is a reliable way to enable compression, but it works
-
so it's better than other ways which don't work :-)
/Z
On Mon, 16 Oct 2023 at 16:16, Josh Baergen <jbaergen@digitalocean.com> wrote:
the resulting ceph.conf inside the monitor container doesn't have mon_rocksdb_options
I don't know where this particular ceph.conf copy comes from, but I still suspect that this is where this particular option needs to be set. The reason I think this is that rocksdb mount options are needed _before_ the mon is able to access any of the centralized conf data, which I believe is itself stored in rocksdb.
Josh
On Sun, Oct 15, 2023 at 10:29 PM Zakhar Kirpichenko <zakhar@gmail.com> wrote:
Out of curiosity, I tried setting mon_rocksdb_options via ceph.conf.
This didn't work either: ceph.conf gets overridden at monitor start, the resulting ceph.conf inside the monitor container doesn't have mon_rocksdb_options, the monitor starts with no RocksDB compression.
I would appreciate it if someone from the Ceph team could please chip
in and suggest a working way to enable RocksDB compression in Ceph monitors.
/Z
On Sat, 14 Oct 2023 at 19:16, Zakhar Kirpichenko <zakhar@gmail.com>
Thanks for your response, Josh. Our ceph.conf doesn't have anything
but
This works rather well for various Ceph components, including the
monitors. RocksDB options are also applied to monitors correctly, but for some reason are being ignored.
/Z
On Sat, 14 Oct 2023, 17:40 Josh Baergen, <jbaergen@digitalocean.com>
wrote:
Apologies if you tried this already and I missed it - have you tried configuring that setting in /etc/ceph/ceph.conf (or wherever your
conf
file is) instead of via 'ceph config'? I wonder if mon settings like this one won't actually apply the way you want because they're needed before the mon has the ability to obtain configuration from, effectively, itself.
Josh
On Sat, Oct 14, 2023 at 1:32 AM Zakhar Kirpichenko < zakhar@gmail.com> wrote: > > I also tried setting RocksDB compression options and deploying a new > monitor. The monitor started with no RocksDB compression again. > > Ceph monitors seem to ignore mon_rocksdb_options set at runtime, at mon > start and at mon deploy. How can I enable RocksDB compression in Ceph > monitors? > > Any input from anyone, please? > > /Z > > On Fri, 13 Oct 2023 at 23:01, Zakhar Kirpichenko < zakhar@gmail.com> wrote: > > > Hi, > > > > I'm still trying to fight large Ceph monitor writes. One option I > > considered is enabling RocksDB compression, as our nodes have more
wrote: the mon addresses, modern Ceph versions store their configuration in the monitor configuration database. than
> > sufficient RAM and CPU. Unfortunately, monitors seem to completely ignore > > the compression setting: > > > > I tried: > > > > - setting ceph config set mon.ceph05 mon_rocksdb_options > >
"write_buffer_size=33554432,compression=kLZ4Compression,level_compaction_dynamic_level_bytes=true",
> > restarting the test monitor. The monitor started with no RocksDB > > compression: > > > > debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: Compression > > algorithms supported: > > debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: > > kZSTDNotFinalCompression supported: 0 > > debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: > > kXpressCompression supported: 0 > > debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: > > kLZ4HCCompression supported: 1 > > debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: > > kLZ4Compression supported: 1 > > debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: > > kBZip2Compression supported: 0 > > debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: > > kZlibCompression supported: 1 > > debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: > > kSnappyCompression supported: 1 > > ... > > debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: > > Options.compression: NoCompression > > debug 2023-10-13T19:47:00.403+0000 7f1cd967a880 4 rocksdb: > > Options.bottommost_compression: Disabled > > > > - setting ceph config set mon mon_rocksdb_options > >
"write_buffer_size=33554432,compression=kLZ4Compression,level_compaction_dynamic_level_bytes=true",
> > restarting the test monitor. The monitor started with no RocksDB > > compression, the same way as above. > > > > In each case config options were correctly set and readable with config > > get. I also found a suggestion in ceph-users ( > >
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/KJM232IHN7F...
> > to set compression in a similar manner. Unfortunately, these
) options appear
> > to be ignored. > > > > How can I enable RocksDB compression in Ceph monitors? > > > > I would very much appreciate your advices and comments. > > > > Best regards, > > Zakhar > > > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
Eugen Block
-
Josh Baergen
-
Zakhar Kirpichenko