Hi, I have upgraded my test and production cephadm-managed clusters from 16.2.14 to 16.2.15. The upgrade was smooth and completed without issues. There were a few things which I noticed after each upgrade: 1. RocksDB options, which I provided to each mon via their configuration files, got overwritten during mon redeployment and I had to re-add mon_rocksdb_options back. 2. Monitor debug_rocksdb option got silently reset back to the default 4/5, I had to set it back to 1/5. 3. For roughly 2 hours after the upgrade, despite the clusters being healthy and operating normally, all monitors would run manual compactions very often and write to disks at very high rates. For example, production monitors had their rocksdb:low0 thread write to store.db: monitors without RocksDB compression: ~8 GB/5 min, or ~96 GB/hour; monitors with RocksDB compression: ~1.5 GB/5 min, or ~18 GB/hour. After roughly 2 hours with no changes to the cluster the write rates dropped to ~0.4-0.6 GB/5 min and ~120 MB/5 min respectively. The reason for frequent manual compactions and high write rates wasn't immediately apparent. 4. Crash deployment broke ownership of /var/lib/ceph/FSID/crash /var/lib/ceph/FSID/crash/posted, despite I already fixed it manually after the upgrade to 16.2.14 which had broken it as well. 5. Mgr RAM usage appears to be increasing at a slower rate than it did with 16.2.14, although it's too early to tell whether the issue with mgrs randomly consuming all RAM and getting OOM-killed has been fixed - with 16.2.14 this would normally take several days. Overall, things look good. Thanks to the Ceph team for this release! Zakhar
Hi,
1. RocksDB options, which I provided to each mon via their configuration files, got overwritten during mon redeployment and I had to re-add mon_rocksdb_options back.
IIRC, you didn't use the extra_entrypoint_args for that option but added it directly to the container unit.run file. So it's expected that it's removed after an update. If you want it to persist a container update you should consider using the extra_entrypoint_args: cat mon.yaml service_type: mon service_name: mon placement: hosts: - host1 - host2 - host3 extra_entrypoint_args: - '--mon-rocksdb-options=write_buffer_size=33554432,compression=kLZ4Compression,level_compaction_dynamic_level_bytes=true,bottommost_compression=kLZ4HCCompression,max_background_jobs=4,max_subcompactions=2' Regards, Eugen Zitat von Zakhar Kirpichenko <zakhar@gmail.com>:
Hi,
I have upgraded my test and production cephadm-managed clusters from 16.2.14 to 16.2.15. The upgrade was smooth and completed without issues. There were a few things which I noticed after each upgrade:
1. RocksDB options, which I provided to each mon via their configuration files, got overwritten during mon redeployment and I had to re-add mon_rocksdb_options back.
2. Monitor debug_rocksdb option got silently reset back to the default 4/5, I had to set it back to 1/5.
3. For roughly 2 hours after the upgrade, despite the clusters being healthy and operating normally, all monitors would run manual compactions very often and write to disks at very high rates. For example, production monitors had their rocksdb:low0 thread write to store.db:
monitors without RocksDB compression: ~8 GB/5 min, or ~96 GB/hour; monitors with RocksDB compression: ~1.5 GB/5 min, or ~18 GB/hour.
After roughly 2 hours with no changes to the cluster the write rates dropped to ~0.4-0.6 GB/5 min and ~120 MB/5 min respectively. The reason for frequent manual compactions and high write rates wasn't immediately apparent.
4. Crash deployment broke ownership of /var/lib/ceph/FSID/crash /var/lib/ceph/FSID/crash/posted, despite I already fixed it manually after the upgrade to 16.2.14 which had broken it as well.
5. Mgr RAM usage appears to be increasing at a slower rate than it did with 16.2.14, although it's too early to tell whether the issue with mgrs randomly consuming all RAM and getting OOM-killed has been fixed - with 16.2.14 this would normally take several days.
Overall, things look good. Thanks to the Ceph team for this release!
Zakhar _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Eugen, It is correct that I manually added the configuration, but not to the unit.run but rather to each mon's config (i.e. /var/lib/ceph/FSID/mon.*/config). I also added it to the cluster config with "ceph config set mon mon_rocksdb_options", but it seems that this option doesn't have any effect at all. /Z On Tue, 5 Mar 2024 at 09:58, Eugen Block <eblock@nde.ag> wrote:
Hi,
1. RocksDB options, which I provided to each mon via their configuration files, got overwritten during mon redeployment and I had to re-add mon_rocksdb_options back.
IIRC, you didn't use the extra_entrypoint_args for that option but added it directly to the container unit.run file. So it's expected that it's removed after an update. If you want it to persist a container update you should consider using the extra_entrypoint_args:
cat mon.yaml service_type: mon service_name: mon placement: hosts: - host1 - host2 - host3 extra_entrypoint_args: -
'--mon-rocksdb-options=write_buffer_size=33554432,compression=kLZ4Compression,level_compaction_dynamic_level_bytes=true,bottommost_compression=kLZ4HCCompression,max_background_jobs=4,max_subcompactions=2'
Regards, Eugen
Zitat von Zakhar Kirpichenko <zakhar@gmail.com>:
Hi,
I have upgraded my test and production cephadm-managed clusters from 16.2.14 to 16.2.15. The upgrade was smooth and completed without issues. There were a few things which I noticed after each upgrade:
1. RocksDB options, which I provided to each mon via their configuration files, got overwritten during mon redeployment and I had to re-add mon_rocksdb_options back.
2. Monitor debug_rocksdb option got silently reset back to the default 4/5, I had to set it back to 1/5.
3. For roughly 2 hours after the upgrade, despite the clusters being healthy and operating normally, all monitors would run manual compactions very often and write to disks at very high rates. For example, production monitors had their rocksdb:low0 thread write to store.db:
monitors without RocksDB compression: ~8 GB/5 min, or ~96 GB/hour; monitors with RocksDB compression: ~1.5 GB/5 min, or ~18 GB/hour.
After roughly 2 hours with no changes to the cluster the write rates dropped to ~0.4-0.6 GB/5 min and ~120 MB/5 min respectively. The reason for frequent manual compactions and high write rates wasn't immediately apparent.
4. Crash deployment broke ownership of /var/lib/ceph/FSID/crash /var/lib/ceph/FSID/crash/posted, despite I already fixed it manually after the upgrade to 16.2.14 which had broken it as well.
5. Mgr RAM usage appears to be increasing at a slower rate than it did with 16.2.14, although it's too early to tell whether the issue with mgrs randomly consuming all RAM and getting OOM-killed has been fixed - with 16.2.14 this would normally take several days.
Overall, things look good. Thanks to the Ceph team for this release!
Zakhar _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi,
I also added it to the cluster config with "ceph config set mon mon_rocksdb_options", but it seems that this option doesn't have any effect at all.
that's because it's an option that has to be present *during* mon startup, not *after* the startup when it can read the config store. Zitat von Zakhar Kirpichenko <zakhar@gmail.com>:
Hi Eugen,
It is correct that I manually added the configuration, but not to the unit.run but rather to each mon's config (i.e. /var/lib/ceph/FSID/mon.*/config). I also added it to the cluster config with "ceph config set mon mon_rocksdb_options", but it seems that this option doesn't have any effect at all.
/Z
On Tue, 5 Mar 2024 at 09:58, Eugen Block <eblock@nde.ag> wrote:
Hi,
1. RocksDB options, which I provided to each mon via their configuration files, got overwritten during mon redeployment and I had to re-add mon_rocksdb_options back.
IIRC, you didn't use the extra_entrypoint_args for that option but added it directly to the container unit.run file. So it's expected that it's removed after an update. If you want it to persist a container update you should consider using the extra_entrypoint_args:
cat mon.yaml service_type: mon service_name: mon placement: hosts: - host1 - host2 - host3 extra_entrypoint_args: -
'--mon-rocksdb-options=write_buffer_size=33554432,compression=kLZ4Compression,level_compaction_dynamic_level_bytes=true,bottommost_compression=kLZ4HCCompression,max_background_jobs=4,max_subcompactions=2'
Regards, Eugen
Zitat von Zakhar Kirpichenko <zakhar@gmail.com>:
Hi,
I have upgraded my test and production cephadm-managed clusters from 16.2.14 to 16.2.15. The upgrade was smooth and completed without issues. There were a few things which I noticed after each upgrade:
1. RocksDB options, which I provided to each mon via their configuration files, got overwritten during mon redeployment and I had to re-add mon_rocksdb_options back.
2. Monitor debug_rocksdb option got silently reset back to the default 4/5, I had to set it back to 1/5.
3. For roughly 2 hours after the upgrade, despite the clusters being healthy and operating normally, all monitors would run manual compactions very often and write to disks at very high rates. For example, production monitors had their rocksdb:low0 thread write to store.db:
monitors without RocksDB compression: ~8 GB/5 min, or ~96 GB/hour; monitors with RocksDB compression: ~1.5 GB/5 min, or ~18 GB/hour.
After roughly 2 hours with no changes to the cluster the write rates dropped to ~0.4-0.6 GB/5 min and ~120 MB/5 min respectively. The reason for frequent manual compactions and high write rates wasn't immediately apparent.
4. Crash deployment broke ownership of /var/lib/ceph/FSID/crash /var/lib/ceph/FSID/crash/posted, despite I already fixed it manually after the upgrade to 16.2.14 which had broken it as well.
5. Mgr RAM usage appears to be increasing at a slower rate than it did with 16.2.14, although it's too early to tell whether the issue with mgrs randomly consuming all RAM and getting OOM-killed has been fixed - with 16.2.14 this would normally take several days.
Overall, things look good. Thanks to the Ceph team for this release!
Zakhar _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Well, that option could be included in new mon configs generated during mon upgrades. But it isn't being used, a minimal config is written instead. I.e. it seems that the configuration option is useless for all intents and purposes, as it doesn't seem to be taken into account at any stage of a mon's lifecycle. /Z On Tue, 5 Mar 2024 at 10:09, Eugen Block <eblock@nde.ag> wrote:
Hi,
I also added it to the cluster config with "ceph config set mon mon_rocksdb_options", but it seems that this option doesn't have any effect at all.
that's because it's an option that has to be present *during* mon startup, not *after* the startup when it can read the config store.
Zitat von Zakhar Kirpichenko <zakhar@gmail.com>:
Hi Eugen,
It is correct that I manually added the configuration, but not to the unit.run but rather to each mon's config (i.e. /var/lib/ceph/FSID/mon.*/config). I also added it to the cluster config with "ceph config set mon mon_rocksdb_options", but it seems that this option doesn't have any effect at all.
/Z
On Tue, 5 Mar 2024 at 09:58, Eugen Block <eblock@nde.ag> wrote:
Hi,
1. RocksDB options, which I provided to each mon via their configuration files, got overwritten during mon redeployment and I had to re-add mon_rocksdb_options back.
IIRC, you didn't use the extra_entrypoint_args for that option but added it directly to the container unit.run file. So it's expected that it's removed after an update. If you want it to persist a container update you should consider using the extra_entrypoint_args:
cat mon.yaml service_type: mon service_name: mon placement: hosts: - host1 - host2 - host3 extra_entrypoint_args: -
'--mon-rocksdb-options=write_buffer_size=33554432,compression=kLZ4Compression,level_compaction_dynamic_level_bytes=true,bottommost_compression=kLZ4HCCompression,max_background_jobs=4,max_subcompactions=2'
Regards, Eugen
Zitat von Zakhar Kirpichenko <zakhar@gmail.com>:
Hi,
I have upgraded my test and production cephadm-managed clusters from 16.2.14 to 16.2.15. The upgrade was smooth and completed without
There were a few things which I noticed after each upgrade:
1. RocksDB options, which I provided to each mon via their configuration files, got overwritten during mon redeployment and I had to re-add mon_rocksdb_options back.
2. Monitor debug_rocksdb option got silently reset back to the default 4/5, I had to set it back to 1/5.
3. For roughly 2 hours after the upgrade, despite the clusters being healthy and operating normally, all monitors would run manual compactions very often and write to disks at very high rates. For example,
issues. production
monitors had their rocksdb:low0 thread write to store.db:
monitors without RocksDB compression: ~8 GB/5 min, or ~96 GB/hour; monitors with RocksDB compression: ~1.5 GB/5 min, or ~18 GB/hour.
After roughly 2 hours with no changes to the cluster the write rates dropped to ~0.4-0.6 GB/5 min and ~120 MB/5 min respectively. The reason for frequent manual compactions and high write rates wasn't immediately apparent.
4. Crash deployment broke ownership of /var/lib/ceph/FSID/crash /var/lib/ceph/FSID/crash/posted, despite I already fixed it manually after the upgrade to 16.2.14 which had broken it as well.
5. Mgr RAM usage appears to be increasing at a slower rate than it did with 16.2.14, although it's too early to tell whether the issue with mgrs randomly consuming all RAM and getting OOM-killed has been fixed - with 16.2.14 this would normally take several days.
Overall, things look good. Thanks to the Ceph team for this release!
Zakhar _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, On 3/5/24 08:57, Eugen Block wrote:
extra_entrypoint_args: - '--mon-rocksdb-options=write_buffer_size=33554432,compression=kLZ4Compression,level_compaction_dynamic_level_bytes=true,bottommost_compression=kLZ4HCCompression,max_background_jobs=4,max_subcompactions=2'
When I try this on my test cluster with Reef 18.2.1 the orchestrator tells me: # ceph orch apply -i mon.yml Error EINVAL: ServiceSpec: __init__() got an unexpected keyword argument 'extra_entrypoint_args' It's a documented feature: https://docs.ceph.com/en/reef/cephadm/services/#cephadm-extra-entrypoint-arg... Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
Oh, you're right. I just checked on Quincy as well at it failed with the same error message. For pacific it still works. I'll check for existing tracker issues. Zitat von Robert Sander <r.sander@heinlein-support.de>:
Hi,
On 3/5/24 08:57, Eugen Block wrote:
extra_entrypoint_args: - '--mon-rocksdb-options=write_buffer_size=33554432,compression=kLZ4Compression,level_compaction_dynamic_level_bytes=true,bottommost_compression=kLZ4HCCompression,max_background_jobs=4,max_subcompactions=2'
When I try this on my test cluster with Reef 18.2.1 the orchestrator tells me:
# ceph orch apply -i mon.yml Error EINVAL: ServiceSpec: __init__() got an unexpected keyword argument 'extra_entrypoint_args'
It's a documented feature:
https://docs.ceph.com/en/reef/cephadm/services/#cephadm-extra-entrypoint-arg...
Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: 030 / 405051-43 Fax: 030 / 405051-19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
It seems to be an issue with the service type (in this case "mon"), it's not entirely "broken", with the node-exporter it works: quincy-1:~ # cat node-exporter.yaml service_type: node-exporter service_name: node-exporter placement: host_pattern: '*' extra_entrypoint_args: - "--collector.textfile.directory=/var/lib/node_exporter/textfile_collector2" quincy-1:~ # ceph orch apply -i node-exporter.yaml Scheduled node-exporter update... I'll keep looking... unless one of the devs is reading this thread and finds it quicker. Zitat von Eugen Block <eblock@nde.ag>:
Oh, you're right. I just checked on Quincy as well at it failed with the same error message. For pacific it still works. I'll check for existing tracker issues.
Zitat von Robert Sander <r.sander@heinlein-support.de>:
Hi,
On 3/5/24 08:57, Eugen Block wrote:
extra_entrypoint_args: - '--mon-rocksdb-options=write_buffer_size=33554432,compression=kLZ4Compression,level_compaction_dynamic_level_bytes=true,bottommost_compression=kLZ4HCCompression,max_background_jobs=4,max_subcompactions=2'
When I try this on my test cluster with Reef 18.2.1 the orchestrator tells me:
# ceph orch apply -i mon.yml Error EINVAL: ServiceSpec: __init__() got an unexpected keyword argument 'extra_entrypoint_args'
It's a documented feature:
https://docs.ceph.com/en/reef/cephadm/services/#cephadm-extra-entrypoint-arg...
Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: 030 / 405051-43 Fax: 030 / 405051-19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
There was a bug with this that was fixed by https://github.com/ceph/ceph/pull/52122 (which also specifically added an integration test for this case). It looks like it's missing a reef and quincy backport though unfortunately. I'll try to open one for both. On Tue, Mar 5, 2024 at 8:26 AM Eugen Block <eblock@nde.ag> wrote:
It seems to be an issue with the service type (in this case "mon"), it's not entirely "broken", with the node-exporter it works:
quincy-1:~ # cat node-exporter.yaml service_type: node-exporter service_name: node-exporter placement: host_pattern: '*' extra_entrypoint_args: - "--collector.textfile.directory=/var/lib/node_exporter/textfile_collector2"
quincy-1:~ # ceph orch apply -i node-exporter.yaml Scheduled node-exporter update...
I'll keep looking... unless one of the devs is reading this thread and finds it quicker.
Zitat von Eugen Block <eblock@nde.ag>:
Oh, you're right. I just checked on Quincy as well at it failed with the same error message. For pacific it still works. I'll check for existing tracker issues.
Zitat von Robert Sander <r.sander@heinlein-support.de>:
Hi,
On 3/5/24 08:57, Eugen Block wrote:
extra_entrypoint_args: -
'--mon-rocksdb-options=write_buffer_size=33554432,compression=kLZ4Compression,level_compaction_dynamic_level_bytes=true,bottommost_compression=kLZ4HCCompression,max_background_jobs=4,max_subcompactions=2'
When I try this on my test cluster with Reef 18.2.1 the orchestrator tells me:
# ceph orch apply -i mon.yml Error EINVAL: ServiceSpec: __init__() got an unexpected keyword argument 'extra_entrypoint_args'
It's a documented feature:
https://docs.ceph.com/en/reef/cephadm/services/#cephadm-extra-entrypoint-arg...
Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: 030 / 405051-43 Fax: 030 / 405051-19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks for chiming in, Adam. Zitat von Adam King <adking@redhat.com>:
There was a bug with this that was fixed by https://github.com/ceph/ceph/pull/52122 (which also specifically added an integration test for this case). It looks like it's missing a reef and quincy backport though unfortunately. I'll try to open one for both.
On Tue, Mar 5, 2024 at 8:26 AM Eugen Block <eblock@nde.ag> wrote:
It seems to be an issue with the service type (in this case "mon"), it's not entirely "broken", with the node-exporter it works:
quincy-1:~ # cat node-exporter.yaml service_type: node-exporter service_name: node-exporter placement: host_pattern: '*' extra_entrypoint_args: - "--collector.textfile.directory=/var/lib/node_exporter/textfile_collector2"
quincy-1:~ # ceph orch apply -i node-exporter.yaml Scheduled node-exporter update...
I'll keep looking... unless one of the devs is reading this thread and finds it quicker.
Zitat von Eugen Block <eblock@nde.ag>:
Oh, you're right. I just checked on Quincy as well at it failed with the same error message. For pacific it still works. I'll check for existing tracker issues.
Zitat von Robert Sander <r.sander@heinlein-support.de>:
Hi,
On 3/5/24 08:57, Eugen Block wrote:
extra_entrypoint_args: -
'--mon-rocksdb-options=write_buffer_size=33554432,compression=kLZ4Compression,level_compaction_dynamic_level_bytes=true,bottommost_compression=kLZ4HCCompression,max_background_jobs=4,max_subcompactions=2'
When I try this on my test cluster with Reef 18.2.1 the orchestrator tells me:
# ceph orch apply -i mon.yml Error EINVAL: ServiceSpec: __init__() got an unexpected keyword argument 'extra_entrypoint_args'
It's a documented feature:
https://docs.ceph.com/en/reef/cephadm/services/#cephadm-extra-entrypoint-arg...
Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: 030 / 405051-43 Fax: 030 / 405051-19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
Adam King
-
Eugen Block
-
Robert Sander
-
Zakhar Kirpichenko