v14.2.5 Nautilus released
This is the fifth release of the Ceph Nautilus release series. Among the many notable changes, this release fixes a critical BlueStore bug that was introduced in 14.2.3. All Nautilus users are advised to upgrade to this release. For the complete changelog entry, please visit the release blog at https://ceph.io/releases/v14-2-5-nautilus-released/ Notable Changes --------------- Critical fix: * This release fixes a `critical BlueStore bug <https://tracker.ceph.com/issues/42223>`_ introduced in 14.2.3 (and also present in 14.2.4) that can lead to data corruption when a separate "WAL" device is used. New health warnings: * Ceph will now issue health warnings if daemons have recently crashed. Ceph has been collecting crash reports since the initial Nautilus release, but the health alerts are new. To view new crashes (or all crashes, if you've just upgraded):: ceph crash ls-new To acknowledge a particular crash (or all crashes) and silence the health warning:: ceph crash archive <crash-id> ceph crash archive-all * Ceph will now issue a health warning if a RADOS pool has a ``pg_num`` value that is not a power of two. This can be fixed by adjusting the pool to a nearby power of two:: ceph osd pool set <pool-name> pg_num <new-pg-num> Alternatively, the warning can be silenced with:: ceph config set global mon_warn_on_pool_pg_num_not_power_of_two false * Ceph will issue a health warning if a RADOS pool's ``size`` is set to 1 or, in other words, if the pool is configured with no redundancy. Ceph will stop issuing the warning if the pool size is set to the minimum recommended value:: ceph osd pool set <pool-name> size <num-replicas> The warning can be silenced with:: ceph config set global mon_warn_on_pool_no_redundancy false * A health warning is now generated if the average osd heartbeat ping time exceeds a configurable threshold for any of the intervals computed. The OSD computes 1 minute, 5 minute and 15 minute intervals with average, minimum and maximum values. New configuration option `mon_warn_on_slow_ping_ratio` specifies a percentage of `osd_heartbeat_grace` to determine the threshold. A value of zero disables the warning. New configuration option `mon_warn_on_slow_ping_time` specified in milliseconds over-rides the computed value, causes a warning when OSD heartbeat pings take longer than the specified amount. A new admin command, `ceph daemon mgr.# dump_osd_network [threshold]`, will list all connections with a ping time longer than the specified threshold or value determined by the config options, for the average for any of the 3 intervals. Another new admin command, `ceph daemon osd.# dump_osd_network [threshold]`, will do the same but only including heartbeats initiated by the specified OSD. Changes in the telemetry module: * The telemetry module now has a 'device' channel, enabled by default, that will report anonymized hard disk and SSD health metrics to telemetry.ceph.com in order to build and improve device failure prediction algorithms. Because the content of telemetry reports has changed, you will need to re-opt-in with:: ceph telemetry on You can view exactly what information will be reported first with:: ceph telemetry show ceph telemetry show device # specifically show the device channel If you are not comfortable sharing device metrics, you can disable that channel first before re-opting-in: ceph config set mgr mgr/telemetry/channel_crash false ceph telemetry on * The telemetry module now reports more information about CephFS file systems, including: - how many MDS daemons (in total and per file system) - which features are (or have been) enabled - how many data pools - approximate file system age (year + month of creation) - how many files, bytes, and snapshots - how much metadata is being cached We have also added: - which Ceph release the monitors are running - whether msgr v1 or v2 addresses are used for the monitors - whether IPv4 or IPv6 addresses are used for the monitors - whether RADOS cache tiering is enabled (and which mode) - whether pools are replicated or erasure coded, and which erasure code profile plugin and parameters are in use - how many hosts are in the cluster, and how many hosts have each type of daemon - whether a separate OSD cluster network is being used - how many RBD pools and images are in the cluster, and how many pools have RBD mirroring enabled - how many RGW daemons, zones, and zonegroups are present; which RGW frontends are in use - aggregate stats about the CRUSH map, like which algorithms are used, how big buckets are, how many rules are defined, and what tunables are in use If you had telemetry enabled, you will need to re-opt-in with:: ceph telemetry on You can view exactly what information will be reported first with:: ceph telemetry show # see everything ceph telemetry show basic # basic cluster info (including all of the new info) OSD: * A new OSD daemon command, 'dump_recovery_reservations', reveals the recovery locks held (in_progress) and waiting in priority queues. * Another new OSD daemon command, 'dump_scrub_reservations', reveals the scrub reservations that are held for local (primary) and remote (replica) PGs. RGW: * RGW now supports S3 Object Lock set of APIs allowing for a WORM model for storing objects. 6 new APIs have been added put/get bucket object lock, put/get object retention, put/get object legal hold. * RGW now supports List Objects V2 Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-14.2.5.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: ad5bd132e1492173c85fda2cc863152730b16a92 -- Abhishek Lekshmanan SUSE Software Solutions Germany GmbH
Thanks all! On 10/12/2019 09:45, Abhishek Lekshmanan wrote:
This is the fifth release of the Ceph Nautilus release series. Among the many notable changes, this release fixes a critical BlueStore bug that was introduced in 14.2.3. All Nautilus users are advised to upgrade to this release.
For the complete changelog entry, please visit the release blog at https://ceph.io/releases/v14-2-5-nautilus-released/
After updating/restarting the manager to v14.2.5 we are no longer able to send data to our zabbix servers. Ceph reports a non-zero exit status from zabbix_sender, but I have not been able to identify the cause of the non-zero exit. # ceph health detail HEALTH_WARN Failed to send data to Zabbix MGR_ZABBIX_SEND_FAILED Failed to send data to Zabbix /usr/bin/zabbix_sender exited non-zero: Setting "debug mgr = 20" yields no additional information that I could see wrt to above issue. zabbix configuration in ceph has not changed since the v14.2.5 update, and was working under v14.2.4: # ceph zabbix config-show {"zabbix_port": 10051, "zabbix_host": "controller03.mgmt.cloud", "identifier": "controller02.mgmt.cloud", "zabbix_sender": "/usr/bin/zabbix_sender", "interval": 60} And I can force a send without error: # /usr/bin/zabbix_sender -z controller03.mgmt.cloud -s controller02.mgmt.cloud -p 10051 -k ceph.total_used_bytes -o 0 Response from "controller03.mgmt.cloud:10051": "processed: 1; failed: 0; total: 1; seconds spent: 0.000062" sent: 1; skipped: 0; total: 1 # echo $? 0 Any pointers/assistance would be appreciated. Thanks Gary -- Gary Molenkamp Computer Science/Science Technology Services Systems Administrator University of Western Ontario molenkam@uwo.ca http://www.csd.uwo.ca (519) 661-2111 x86882 (519) 661-3566
Piggybacking on this thread to say that I too am having the same behavior. Ubuntu 18.04 zabbix-sender 4.4.3-1+bionic ceph-mgr 14.2.5-1bionic I am still getting all metrics in my Zabbix host, it is just the error being thrown by the ceph-mgr. Reed
On Dec 11, 2019, at 9:54 AM, Gary Molenkamp <molenkam@uwo.ca> wrote:
After updating/restarting the manager to v14.2.5 we are no longer able to send data to our zabbix servers.
Ceph reports a non-zero exit status from zabbix_sender, but I have not been able to identify the cause of the non-zero exit.
# ceph health detail HEALTH_WARN Failed to send data to Zabbix MGR_ZABBIX_SEND_FAILED Failed to send data to Zabbix /usr/bin/zabbix_sender exited non-zero:
Setting "debug mgr = 20" yields no additional information that I could see wrt to above issue.
zabbix configuration in ceph has not changed since the v14.2.5 update, and was working under v14.2.4:
# ceph zabbix config-show {"zabbix_port": 10051, "zabbix_host": "controller03.mgmt.cloud", "identifier": "controller02.mgmt.cloud", "zabbix_sender": "/usr/bin/zabbix_sender", "interval": 60}
And I can force a send without error: # /usr/bin/zabbix_sender -z controller03.mgmt.cloud -s controller02.mgmt.cloud -p 10051 -k ceph.total_used_bytes -o 0 Response from "controller03.mgmt.cloud:10051": "processed: 1; failed: 0; total: 1; seconds spent: 0.000062" sent: 1; skipped: 0; total: 1 # echo $? 0
Any pointers/assistance would be appreciated. Thanks Gary
--
Gary Molenkamp Computer Science/Science Technology Services Systems Administrator University of Western Ontario molenkam@uwo.ca http://www.csd.uwo.ca (519) 661-2111 x86882 (519) 661-3566
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I dislike replying to my own post, but I found the issue: Looking at the changelog for 14.2.5, the zabbix key ceph.num_pg_wait_backfill has been renamed to ceph.num_pg_backfill_wait. This needs to be updated in the zabbix_template.yml Before the change: # /usr/bin/zabbix_sender -z controller03.mgmt.cloud -s controller02.mgmt.cloud -p 10051 -k ceph.num_pg_backfill_wait -o 0 Response from "controller03.mgmt.cloud:10051": "processed: 0; failed: 1; total: 1; seconds spent: 0.000033" sent: 1; skipped: 0; total: 1 # /usr/bin/zabbix_sender -z controller03.mgmt.cloud -s controller02.mgmt.cloud -p 10051 -k ceph.num_pg_wait_backfill -o 0 Response from "controller03.mgmt.cloud:10051": "processed: 1; failed: 0; total: 1; seconds spent: 0.000059" sent: 1; skipped: 0; total: 1 After the key update: # /usr/bin/zabbix_sender -z controller03.mgmt.cloud -s controller02.mgmt.cloud -p 10051 -k ceph.num_pg_backfill_wait -o 0 Response from "controller03.mgmt.cloud:10051": "processed: 1; failed: 0; total: 1; seconds spent: 0.000053" sent: 1; skipped: 0; total: 1 # /usr/bin/zabbix_sender -z controller03.mgmt.cloud -s controller02.mgmt.cloud -p 10051 -k ceph.num_pg_wait_backfill -o 0 Response from "controller03.mgmt.cloud:10051": "processed: 0; failed: 1; total: 1; seconds spent: 0.000032" sent: 1; skipped: 0; total: 1 Gary. On 2019-12-11 10:54 a.m., Gary Molenkamp wrote:
After updating/restarting the manager to v14.2.5 we are no longer able to send data to our zabbix servers.
Ceph reports a non-zero exit status from zabbix_sender, but I have not been able to identify the cause of the non-zero exit.
# ceph health detail HEALTH_WARN Failed to send data to Zabbix MGR_ZABBIX_SEND_FAILED Failed to send data to Zabbix /usr/bin/zabbix_sender exited non-zero:
Setting "debug mgr = 20" yields no additional information that I could see wrt to above issue.
zabbix configuration in ceph has not changed since the v14.2.5 update, and was working under v14.2.4:
# ceph zabbix config-show {"zabbix_port": 10051, "zabbix_host": "controller03.mgmt.cloud", "identifier": "controller02.mgmt.cloud", "zabbix_sender": "/usr/bin/zabbix_sender", "interval": 60}
And I can force a send without error: # /usr/bin/zabbix_sender -z controller03.mgmt.cloud -s controller02.mgmt.cloud -p 10051 -k ceph.total_used_bytes -o 0 Response from "controller03.mgmt.cloud:10051": "processed: 1; failed: 0; total: 1; seconds spent: 0.000062" sent: 1; skipped: 0; total: 1 # echo $? 0
Any pointers/assistance would be appreciated. Thanks Gary
-- Gary Molenkamp Computer Science/Science Technology Services Systems Administrator University of Western Ontario molenkam@uwo.ca http://www.csd.uwo.ca (519) 661-2111 x86882 (519) 661-3566
Excellent sleuthing. I was able to change the key on my Zabbix server instance and all is happy in Ceph again. Thanks, Reed
On Dec 11, 2019, at 10:24 AM, Gary Molenkamp <molenkam@uwo.ca> wrote:
I dislike replying to my own post, but I found the issue:
Looking at the changelog for 14.2.5, the zabbix key ceph.num_pg_wait_backfill has been renamed to ceph.num_pg_backfill_wait.
This needs to be updated in the zabbix_template.yml
Before the change:
# /usr/bin/zabbix_sender -z controller03.mgmt.cloud -s controller02.mgmt.cloud -p 10051 -k ceph.num_pg_backfill_wait -o 0 Response from "controller03.mgmt.cloud:10051": "processed: 0; failed: 1; total: 1; seconds spent: 0.000033" sent: 1; skipped: 0; total: 1 # /usr/bin/zabbix_sender -z controller03.mgmt.cloud -s controller02.mgmt.cloud -p 10051 -k ceph.num_pg_wait_backfill -o 0 Response from "controller03.mgmt.cloud:10051": "processed: 1; failed: 0; total: 1; seconds spent: 0.000059" sent: 1; skipped: 0; total: 1
After the key update:
# /usr/bin/zabbix_sender -z controller03.mgmt.cloud -s controller02.mgmt.cloud -p 10051 -k ceph.num_pg_backfill_wait -o 0 Response from "controller03.mgmt.cloud:10051": "processed: 1; failed: 0; total: 1; seconds spent: 0.000053" sent: 1; skipped: 0; total: 1 # /usr/bin/zabbix_sender -z controller03.mgmt.cloud -s controller02.mgmt.cloud -p 10051 -k ceph.num_pg_wait_backfill -o 0 Response from "controller03.mgmt.cloud:10051": "processed: 0; failed: 1; total: 1; seconds spent: 0.000032" sent: 1; skipped: 0; total: 1
Gary.
On 2019-12-11 10:54 a.m., Gary Molenkamp wrote:
After updating/restarting the manager to v14.2.5 we are no longer able to send data to our zabbix servers.
Ceph reports a non-zero exit status from zabbix_sender, but I have not been able to identify the cause of the non-zero exit.
# ceph health detail HEALTH_WARN Failed to send data to Zabbix MGR_ZABBIX_SEND_FAILED Failed to send data to Zabbix /usr/bin/zabbix_sender exited non-zero:
Setting "debug mgr = 20" yields no additional information that I could see wrt to above issue.
zabbix configuration in ceph has not changed since the v14.2.5 update, and was working under v14.2.4:
# ceph zabbix config-show {"zabbix_port": 10051, "zabbix_host": "controller03.mgmt.cloud", "identifier": "controller02.mgmt.cloud", "zabbix_sender": "/usr/bin/zabbix_sender", "interval": 60}
And I can force a send without error: # /usr/bin/zabbix_sender -z controller03.mgmt.cloud -s controller02.mgmt.cloud -p 10051 -k ceph.total_used_bytes -o 0 Response from "controller03.mgmt.cloud:10051": "processed: 1; failed: 0; total: 1; seconds spent: 0.000062" sent: 1; skipped: 0; total: 1 # echo $? 0
Any pointers/assistance would be appreciated. Thanks Gary
-- Gary Molenkamp Computer Science/Science Technology Services Systems Administrator University of Western Ontario molenkam@uwo.ca http://www.csd.uwo.ca (519) 661-2111 x86882 (519) 661-3566
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Tue, Dec 10, 2019 at 10:45 AM Abhishek Lekshmanan <abhishek@suse.com> wrote:
This is the fifth release of the Ceph Nautilus release series. Among the many notable changes, this release fixes a critical BlueStore bug that was introduced in 14.2.3. All Nautilus users are advised to upgrade to this release.
For the complete changelog entry, please visit the release blog at https://ceph.io/releases/v14-2-5-nautilus-released/
Notable Changes ---------------
Critical fix:
* This release fixes a `critical BlueStore bug <https://tracker.ceph.com/issues/42223>`_ introduced in 14.2.3 (and also present in 14.2.4) that can lead to data corruption when a separate "WAL" device is used.
New health warnings:
* Ceph will now issue health warnings if daemons have recently crashed. Ceph has been collecting crash reports since the initial Nautilus release, but the health alerts are new. To view new crashes (or all crashes, if you've just upgraded)::
ceph crash ls-new
To acknowledge a particular crash (or all crashes) and silence the health warning::
ceph crash archive <crash-id> ceph crash archive-all
* Ceph will now issue a health warning if a RADOS pool has a ``pg_num`` value that is not a power of two. This can be fixed by adjusting the pool to a nearby power of two::
ceph osd pool set <pool-name> pg_num <new-pg-num>
Alternatively, the warning can be silenced with::
ceph config set global mon_warn_on_pool_pg_num_not_power_of_two false
* Ceph will issue a health warning if a RADOS pool's ``size`` is set to 1 or, in other words, if the pool is configured with no redundancy. Ceph will stop issuing the warning if the pool size is set to the minimum recommended value::
ceph osd pool set <pool-name> size <num-replicas>
The warning can be silenced with::
ceph config set global mon_warn_on_pool_no_redundancy false
* A health warning is now generated if the average osd heartbeat ping time exceeds a configurable threshold for any of the intervals computed. The OSD computes 1 minute, 5 minute and 15 minute intervals with average, minimum and maximum values. New configuration option `mon_warn_on_slow_ping_ratio` specifies a percentage of `osd_heartbeat_grace` to determine the threshold. A value of zero disables the warning. New configuration option `mon_warn_on_slow_ping_time` specified in milliseconds over-rides the computed value, causes a warning when OSD heartbeat pings take longer than the specified amount. A new admin command, `ceph daemon mgr.# dump_osd_network [threshold]`, will list all connections with a ping time longer than the specified threshold or value determined by the config options, for the average for any of the 3 intervals. Another new admin command, `ceph daemon osd.# dump_osd_network [threshold]`, will do the same but only including heartbeats initiated by the specified OSD.
Changes in the telemetry module:
* The telemetry module now has a 'device' channel, enabled by default, that will report anonymized hard disk and SSD health metrics to telemetry.ceph.com in order to build and improve device failure prediction algorithms. Because the content of telemetry reports has changed, you will need to re-opt-in with::
ceph telemetry on
You can view exactly what information will be reported first with::
ceph telemetry show ceph telemetry show device # specifically show the device channel
If you are not comfortable sharing device metrics, you can disable that channel first before re-opting-in:
ceph config set mgr mgr/telemetry/channel_crash false
This should be channel_device, right? Thanks, Ilya
If you are not comfortable sharing device metrics, you can disable that channel first before re-opting-in:
ceph config set mgr mgr/telemetry/channel_crash false
This should be channel_device, right?
Yep! https://github.com/ceph/ceph/pull/32148 Thanks, sage
Thanks for this! We're having trouble installing the ceph-debuginfo rpm, which shows a negative file size (apparently because it now exceeds 2GB): https://pastebin.com/5bnNCGHh Does this change need to be applied to the release builds too? https://tracker.ceph.com/issues/39387 -- Dan On Tue, Dec 10, 2019 at 10:45 AM Abhishek Lekshmanan <abhishek@suse.com> wrote:
This is the fifth release of the Ceph Nautilus release series. Among the many notable changes, this release fixes a critical BlueStore bug that was introduced in 14.2.3. All Nautilus users are advised to upgrade to this release.
For the complete changelog entry, please visit the release blog at https://ceph.io/releases/v14-2-5-nautilus-released/
Notable Changes ---------------
Critical fix:
* This release fixes a `critical BlueStore bug <https://tracker.ceph.com/issues/42223>`_ introduced in 14.2.3 (and also present in 14.2.4) that can lead to data corruption when a separate "WAL" device is used.
New health warnings:
* Ceph will now issue health warnings if daemons have recently crashed. Ceph has been collecting crash reports since the initial Nautilus release, but the health alerts are new. To view new crashes (or all crashes, if you've just upgraded)::
ceph crash ls-new
To acknowledge a particular crash (or all crashes) and silence the health warning::
ceph crash archive <crash-id> ceph crash archive-all
* Ceph will now issue a health warning if a RADOS pool has a ``pg_num`` value that is not a power of two. This can be fixed by adjusting the pool to a nearby power of two::
ceph osd pool set <pool-name> pg_num <new-pg-num>
Alternatively, the warning can be silenced with::
ceph config set global mon_warn_on_pool_pg_num_not_power_of_two false
* Ceph will issue a health warning if a RADOS pool's ``size`` is set to 1 or, in other words, if the pool is configured with no redundancy. Ceph will stop issuing the warning if the pool size is set to the minimum recommended value::
ceph osd pool set <pool-name> size <num-replicas>
The warning can be silenced with::
ceph config set global mon_warn_on_pool_no_redundancy false
* A health warning is now generated if the average osd heartbeat ping time exceeds a configurable threshold for any of the intervals computed. The OSD computes 1 minute, 5 minute and 15 minute intervals with average, minimum and maximum values. New configuration option `mon_warn_on_slow_ping_ratio` specifies a percentage of `osd_heartbeat_grace` to determine the threshold. A value of zero disables the warning. New configuration option `mon_warn_on_slow_ping_time` specified in milliseconds over-rides the computed value, causes a warning when OSD heartbeat pings take longer than the specified amount. A new admin command, `ceph daemon mgr.# dump_osd_network [threshold]`, will list all connections with a ping time longer than the specified threshold or value determined by the config options, for the average for any of the 3 intervals. Another new admin command, `ceph daemon osd.# dump_osd_network [threshold]`, will do the same but only including heartbeats initiated by the specified OSD.
Changes in the telemetry module:
* The telemetry module now has a 'device' channel, enabled by default, that will report anonymized hard disk and SSD health metrics to telemetry.ceph.com in order to build and improve device failure prediction algorithms. Because the content of telemetry reports has changed, you will need to re-opt-in with::
ceph telemetry on
You can view exactly what information will be reported first with::
ceph telemetry show ceph telemetry show device # specifically show the device channel
If you are not comfortable sharing device metrics, you can disable that channel first before re-opting-in:
ceph config set mgr mgr/telemetry/channel_crash false ceph telemetry on
* The telemetry module now reports more information about CephFS file systems, including:
- how many MDS daemons (in total and per file system) - which features are (or have been) enabled - how many data pools - approximate file system age (year + month of creation) - how many files, bytes, and snapshots - how much metadata is being cached
We have also added:
- which Ceph release the monitors are running - whether msgr v1 or v2 addresses are used for the monitors - whether IPv4 or IPv6 addresses are used for the monitors - whether RADOS cache tiering is enabled (and which mode) - whether pools are replicated or erasure coded, and which erasure code profile plugin and parameters are in use - how many hosts are in the cluster, and how many hosts have each type of daemon - whether a separate OSD cluster network is being used - how many RBD pools and images are in the cluster, and how many pools have RBD mirroring enabled - how many RGW daemons, zones, and zonegroups are present; which RGW frontends are in use - aggregate stats about the CRUSH map, like which algorithms are used, how big buckets are, how many rules are defined, and what tunables are in use
If you had telemetry enabled, you will need to re-opt-in with::
ceph telemetry on
You can view exactly what information will be reported first with::
ceph telemetry show # see everything ceph telemetry show basic # basic cluster info (including all of the new info)
OSD:
* A new OSD daemon command, 'dump_recovery_reservations', reveals the recovery locks held (in_progress) and waiting in priority queues.
* Another new OSD daemon command, 'dump_scrub_reservations', reveals the scrub reservations that are held for local (primary) and remote (replica) PGs.
RGW:
* RGW now supports S3 Object Lock set of APIs allowing for a WORM model for storing objects. 6 new APIs have been added put/get bucket object lock, put/get object retention, put/get object legal hold.
* RGW now supports List Objects V2
Getting Ceph ------------
* Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-14.2.5.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: ad5bd132e1492173c85fda2cc863152730b16a92
-- Abhishek Lekshmanan SUSE Software Solutions Germany GmbH
On Thu, Dec 12, 2019 at 9:06 AM Dan van der Ster <dan@vanderster.com> wrote:
Thanks for this!
We're having trouble installing the ceph-debuginfo rpm, which shows a negative file size (apparently because it now exceeds 2GB): https://pastebin.com/5bnNCGHh
Does this change need to be applied to the release builds too? https://tracker.ceph.com/issues/39387
Good catch! I just checked and yes, we need to use the --no-database flag
-- Dan
On Tue, Dec 10, 2019 at 10:45 AM Abhishek Lekshmanan <abhishek@suse.com> wrote:
This is the fifth release of the Ceph Nautilus release series. Among the many notable changes, this release fixes a critical BlueStore bug that was introduced in 14.2.3. All Nautilus users are advised to upgrade to this release.
For the complete changelog entry, please visit the release blog at https://ceph.io/releases/v14-2-5-nautilus-released/
Notable Changes ---------------
Critical fix:
* This release fixes a `critical BlueStore bug <https://tracker.ceph.com/issues/42223>`_ introduced in 14.2.3 (and also present in 14.2.4) that can lead to data corruption when a separate "WAL" device is used.
New health warnings:
* Ceph will now issue health warnings if daemons have recently crashed. Ceph has been collecting crash reports since the initial Nautilus release, but the health alerts are new. To view new crashes (or all crashes, if you've just upgraded)::
ceph crash ls-new
To acknowledge a particular crash (or all crashes) and silence the health warning::
ceph crash archive <crash-id> ceph crash archive-all
* Ceph will now issue a health warning if a RADOS pool has a ``pg_num`` value that is not a power of two. This can be fixed by adjusting the pool to a nearby power of two::
ceph osd pool set <pool-name> pg_num <new-pg-num>
Alternatively, the warning can be silenced with::
ceph config set global mon_warn_on_pool_pg_num_not_power_of_two false
* Ceph will issue a health warning if a RADOS pool's ``size`` is set to 1 or, in other words, if the pool is configured with no redundancy. Ceph will stop issuing the warning if the pool size is set to the minimum recommended value::
ceph osd pool set <pool-name> size <num-replicas>
The warning can be silenced with::
ceph config set global mon_warn_on_pool_no_redundancy false
* A health warning is now generated if the average osd heartbeat ping time exceeds a configurable threshold for any of the intervals computed. The OSD computes 1 minute, 5 minute and 15 minute intervals with average, minimum and maximum values. New configuration option `mon_warn_on_slow_ping_ratio` specifies a percentage of `osd_heartbeat_grace` to determine the threshold. A value of zero disables the warning. New configuration option `mon_warn_on_slow_ping_time` specified in milliseconds over-rides the computed value, causes a warning when OSD heartbeat pings take longer than the specified amount. A new admin command, `ceph daemon mgr.# dump_osd_network [threshold]`, will list all connections with a ping time longer than the specified threshold or value determined by the config options, for the average for any of the 3 intervals. Another new admin command, `ceph daemon osd.# dump_osd_network [threshold]`, will do the same but only including heartbeats initiated by the specified OSD.
Changes in the telemetry module:
* The telemetry module now has a 'device' channel, enabled by default, that will report anonymized hard disk and SSD health metrics to telemetry.ceph.com in order to build and improve device failure prediction algorithms. Because the content of telemetry reports has changed, you will need to re-opt-in with::
ceph telemetry on
You can view exactly what information will be reported first with::
ceph telemetry show ceph telemetry show device # specifically show the device channel
If you are not comfortable sharing device metrics, you can disable that channel first before re-opting-in:
ceph config set mgr mgr/telemetry/channel_crash false ceph telemetry on
* The telemetry module now reports more information about CephFS file systems, including:
- how many MDS daemons (in total and per file system) - which features are (or have been) enabled - how many data pools - approximate file system age (year + month of creation) - how many files, bytes, and snapshots - how much metadata is being cached
We have also added:
- which Ceph release the monitors are running - whether msgr v1 or v2 addresses are used for the monitors - whether IPv4 or IPv6 addresses are used for the monitors - whether RADOS cache tiering is enabled (and which mode) - whether pools are replicated or erasure coded, and which erasure code profile plugin and parameters are in use - how many hosts are in the cluster, and how many hosts have each type of daemon - whether a separate OSD cluster network is being used - how many RBD pools and images are in the cluster, and how many pools have RBD mirroring enabled - how many RGW daemons, zones, and zonegroups are present; which RGW frontends are in use - aggregate stats about the CRUSH map, like which algorithms are used, how big buckets are, how many rules are defined, and what tunables are in use
If you had telemetry enabled, you will need to re-opt-in with::
ceph telemetry on
You can view exactly what information will be reported first with::
ceph telemetry show # see everything ceph telemetry show basic # basic cluster info (including all of the new info)
OSD:
* A new OSD daemon command, 'dump_recovery_reservations', reveals the recovery locks held (in_progress) and waiting in priority queues.
* Another new OSD daemon command, 'dump_scrub_reservations', reveals the scrub reservations that are held for local (primary) and remote (replica) PGs.
RGW:
* RGW now supports S3 Object Lock set of APIs allowing for a WORM model for storing objects. 6 new APIs have been added put/get bucket object lock, put/get object retention, put/get object legal hold.
* RGW now supports List Objects V2
Getting Ceph ------------
* Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-14.2.5.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: ad5bd132e1492173c85fda2cc863152730b16a92
-- Abhishek Lekshmanan SUSE Software Solutions Germany GmbH
I just re-signed and pushed the 14.2.5 packages after adding the --no-database fix. Can you confirm ceph-debuginfo installs as expected now please? On 12/12/19 9:05 AM, Dan van der Ster wrote:
Thanks for this!
We're having trouble installing the ceph-debuginfo rpm, which shows a negative file size (apparently because it now exceeds 2GB): https://pastebin.com/5bnNCGHh
Does this change need to be applied to the release builds too? https://tracker.ceph.com/issues/39387
-- Dan
On Tue, Dec 10, 2019 at 10:45 AM Abhishek Lekshmanan <abhishek@suse.com> wrote:
This is the fifth release of the Ceph Nautilus release series. Among the many notable changes, this release fixes a critical BlueStore bug that was introduced in 14.2.3. All Nautilus users are advised to upgrade to this release.
For the complete changelog entry, please visit the release blog at https://ceph.io/releases/v14-2-5-nautilus-released/
Notable Changes ---------------
Critical fix:
* This release fixes a `critical BlueStore bug <https://tracker.ceph.com/issues/42223>`_ introduced in 14.2.3 (and also present in 14.2.4) that can lead to data corruption when a separate "WAL" device is used.
New health warnings:
* Ceph will now issue health warnings if daemons have recently crashed. Ceph has been collecting crash reports since the initial Nautilus release, but the health alerts are new. To view new crashes (or all crashes, if you've just upgraded)::
ceph crash ls-new
To acknowledge a particular crash (or all crashes) and silence the health warning::
ceph crash archive <crash-id> ceph crash archive-all
* Ceph will now issue a health warning if a RADOS pool has a ``pg_num`` value that is not a power of two. This can be fixed by adjusting the pool to a nearby power of two::
ceph osd pool set <pool-name> pg_num <new-pg-num>
Alternatively, the warning can be silenced with::
ceph config set global mon_warn_on_pool_pg_num_not_power_of_two false
* Ceph will issue a health warning if a RADOS pool's ``size`` is set to 1 or, in other words, if the pool is configured with no redundancy. Ceph will stop issuing the warning if the pool size is set to the minimum recommended value::
ceph osd pool set <pool-name> size <num-replicas>
The warning can be silenced with::
ceph config set global mon_warn_on_pool_no_redundancy false
* A health warning is now generated if the average osd heartbeat ping time exceeds a configurable threshold for any of the intervals computed. The OSD computes 1 minute, 5 minute and 15 minute intervals with average, minimum and maximum values. New configuration option `mon_warn_on_slow_ping_ratio` specifies a percentage of `osd_heartbeat_grace` to determine the threshold. A value of zero disables the warning. New configuration option `mon_warn_on_slow_ping_time` specified in milliseconds over-rides the computed value, causes a warning when OSD heartbeat pings take longer than the specified amount. A new admin command, `ceph daemon mgr.# dump_osd_network [threshold]`, will list all connections with a ping time longer than the specified threshold or value determined by the config options, for the average for any of the 3 intervals. Another new admin command, `ceph daemon osd.# dump_osd_network [threshold]`, will do the same but only including heartbeats initiated by the specified OSD.
Changes in the telemetry module:
* The telemetry module now has a 'device' channel, enabled by default, that will report anonymized hard disk and SSD health metrics to telemetry.ceph.com in order to build and improve device failure prediction algorithms. Because the content of telemetry reports has changed, you will need to re-opt-in with::
ceph telemetry on
You can view exactly what information will be reported first with::
ceph telemetry show ceph telemetry show device # specifically show the device channel
If you are not comfortable sharing device metrics, you can disable that channel first before re-opting-in:
ceph config set mgr mgr/telemetry/channel_crash false ceph telemetry on
* The telemetry module now reports more information about CephFS file systems, including:
- how many MDS daemons (in total and per file system) - which features are (or have been) enabled - how many data pools - approximate file system age (year + month of creation) - how many files, bytes, and snapshots - how much metadata is being cached
We have also added:
- which Ceph release the monitors are running - whether msgr v1 or v2 addresses are used for the monitors - whether IPv4 or IPv6 addresses are used for the monitors - whether RADOS cache tiering is enabled (and which mode) - whether pools are replicated or erasure coded, and which erasure code profile plugin and parameters are in use - how many hosts are in the cluster, and how many hosts have each type of daemon - whether a separate OSD cluster network is being used - how many RBD pools and images are in the cluster, and how many pools have RBD mirroring enabled - how many RGW daemons, zones, and zonegroups are present; which RGW frontends are in use - aggregate stats about the CRUSH map, like which algorithms are used, how big buckets are, how many rules are defined, and what tunables are in use
If you had telemetry enabled, you will need to re-opt-in with::
ceph telemetry on
You can view exactly what information will be reported first with::
ceph telemetry show # see everything ceph telemetry show basic # basic cluster info (including all of the new info)
OSD:
* A new OSD daemon command, 'dump_recovery_reservations', reveals the recovery locks held (in_progress) and waiting in priority queues.
* Another new OSD daemon command, 'dump_scrub_reservations', reveals the scrub reservations that are held for local (primary) and remote (replica) PGs.
RGW:
* RGW now supports S3 Object Lock set of APIs allowing for a WORM model for storing objects. 6 new APIs have been added put/get bucket object lock, put/get object retention, put/get object legal hold.
* RGW now supports List Objects V2
Getting Ceph ------------
* Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-14.2.5.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: ad5bd132e1492173c85fda2cc863152730b16a92
-- Abhishek Lekshmanan SUSE Software Solutions Germany GmbH
Sorry I forgot to reply to this -- yes it's all good after you re-signed and pushed. Thanks! dan On Thu, Dec 12, 2019 at 9:57 PM David Galloway <dgallowa@redhat.com> wrote:
I just re-signed and pushed the 14.2.5 packages after adding the --no-database fix. Can you confirm ceph-debuginfo installs as expected now please?
Thanks for this!
We're having trouble installing the ceph-debuginfo rpm, which shows a negative file size (apparently because it now exceeds 2GB): https://pastebin.com/5bnNCGHh
Does this change need to be applied to the release builds too? https://tracker.ceph.com/issues/39387
-- Dan
On Tue, Dec 10, 2019 at 10:45 AM Abhishek Lekshmanan <abhishek@suse.com> wrote:
This is the fifth release of the Ceph Nautilus release series. Among
notable changes, this release fixes a critical BlueStore bug that was introduced in 14.2.3. All Nautilus users are advised to upgrade to this release.
For the complete changelog entry, please visit the release blog at https://ceph.io/releases/v14-2-5-nautilus-released/
Notable Changes ---------------
Critical fix:
* This release fixes a `critical BlueStore bug < https://tracker.ceph.com/issues/42223>`_ introduced in 14.2.3 (and also present in 14.2.4) that can lead to data corruption when a separate "WAL" device is used.
New health warnings:
* Ceph will now issue health warnings if daemons have recently crashed. Ceph has been collecting crash reports since the initial Nautilus release, but the health alerts are new. To view new crashes (or all crashes, if you've just upgraded)::
ceph crash ls-new
To acknowledge a particular crash (or all crashes) and silence the health warning::
ceph crash archive <crash-id> ceph crash archive-all
* Ceph will now issue a health warning if a RADOS pool has a ``pg_num`` value that is not a power of two. This can be fixed by adjusting the pool to a nearby power of two::
ceph osd pool set <pool-name> pg_num <new-pg-num>
Alternatively, the warning can be silenced with::
ceph config set global mon_warn_on_pool_pg_num_not_power_of_two false
* Ceph will issue a health warning if a RADOS pool's ``size`` is set to 1 or, in other words, if the pool is configured with no redundancy. Ceph will stop issuing the warning if the pool size is set to the minimum recommended value::
ceph osd pool set <pool-name> size <num-replicas>
The warning can be silenced with::
ceph config set global mon_warn_on_pool_no_redundancy false
* A health warning is now generated if the average osd heartbeat ping time exceeds a configurable threshold for any of the intervals computed. The OSD computes 1 minute, 5 minute and 15 minute intervals with average, minimum and maximum values. New configuration option `mon_warn_on_slow_ping_ratio` specifies a percentage of `osd_heartbeat_grace` to determine the threshold. A value of zero disables the warning. New configuration option `mon_warn_on_slow_ping_time` specified in milliseconds over-rides the computed value, causes a warning when OSD heartbeat pings take longer than the specified amount. A new admin command, `ceph daemon mgr.# dump_osd_network [threshold]`, will list all connections with a ping time longer than the specified
value determined by the config options, for the average for any of
Another new admin command, `ceph daemon osd.# dump_osd_network [threshold]`, will do the same but only including heartbeats initiated by the specified OSD.
Changes in the telemetry module:
* The telemetry module now has a 'device' channel, enabled by default,
will report anonymized hard disk and SSD health metrics to telemetry.ceph.com in order to build and improve device failure prediction algorithms. Because the content of telemetry reports has changed, you will need to re-opt-in with::
ceph telemetry on
You can view exactly what information will be reported first with::
ceph telemetry show ceph telemetry show device # specifically show the device channel
If you are not comfortable sharing device metrics, you can disable
channel first before re-opting-in:
ceph config set mgr mgr/telemetry/channel_crash false ceph telemetry on
* The telemetry module now reports more information about CephFS file systems, including:
- how many MDS daemons (in total and per file system) - which features are (or have been) enabled - how many data pools - approximate file system age (year + month of creation) - how many files, bytes, and snapshots - how much metadata is being cached
We have also added:
- which Ceph release the monitors are running - whether msgr v1 or v2 addresses are used for the monitors - whether IPv4 or IPv6 addresses are used for the monitors - whether RADOS cache tiering is enabled (and which mode) - whether pools are replicated or erasure coded, and which erasure code profile plugin and parameters are in use - how many hosts are in the cluster, and how many hosts have each type of daemon - whether a separate OSD cluster network is being used - how many RBD pools and images are in the cluster, and how many
- how many RGW daemons, zones, and zonegroups are present; which
RGW frontends are in use
- aggregate stats about the CRUSH map, like which algorithms are
used, how
big buckets are, how many rules are defined, and what tunables
are in
use
If you had telemetry enabled, you will need to re-opt-in with::
ceph telemetry on
You can view exactly what information will be reported first with::
ceph telemetry show # see everything ceph telemetry show basic # basic cluster info (including all of
On 12/12/19 9:05 AM, Dan van der Ster wrote: the many threshold or the 3 intervals. that that pools have RBD mirroring enabled the new info)
OSD:
* A new OSD daemon command, 'dump_recovery_reservations', reveals the recovery locks held (in_progress) and waiting in priority queues.
* Another new OSD daemon command, 'dump_scrub_reservations', reveals the scrub reservations that are held for local (primary) and remote
(replica) PGs.
RGW:
* RGW now supports S3 Object Lock set of APIs allowing for a WORM model
for
storing objects. 6 new APIs have been added put/get bucket object lock, put/get object retention, put/get object legal hold.
* RGW now supports List Objects V2
Getting Ceph ------------
* Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-14.2.5.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: ad5bd132e1492173c85fda2cc863152730b16a92
-- Abhishek Lekshmanan SUSE Software Solutions Germany GmbH
participants (9)
-
Abhishek Lekshmanan
-
Alfredo Deza
-
Dan van der Ster
-
David Galloway
-
Gary Molenkamp
-
Ilya Dryomov
-
Reed Dier
-
Sage Weil
-
Simon Ironside