We're happy to announce 13th bug fix release of the Luminous v12.2.x
long term stable release series. We recommend that all users upgrade to
this release. Many thanks to all the contributors, in particular Yuri &
Nathan, in getting this release out of the door. This shall be the last
release of the Luminous series.
For a detailed release notes, please check out the official blog entry
at https://ceph.io/releases/v12-2-13-luminous-released/
Notable Changes
---------------
* Ceph now packages python bindings for python3.6 instead of python3.4,
because EPEL7 recently switched from python3.4 to python3.6 as the
native python3. see the announcement[1] for more details on the
background of this change.
* We now have telemetry support via a ceph-mgr module. The telemetry module is
absolutely on an opt-in basis, and is meant to collect generic cluster
information and push it to a central endpoint. By default, we're pushing it
to a project endpoint at https://telemetry.ceph.com/report, but this is
customizable using by setting the 'url' config option with::
ceph telemetry config-set url '<your url>'
You will have to opt-in on sharing your information with::
ceph telemetry on
You can view exactly what information will be reported first with::
ceph telemetry show
Should you opt-in, your information will be licensed under the
Community Data License Agreement - Sharing - Version 1.0, which you can
read at https://cdla.io/sharing-1-0/
The telemetry module reports information about CephFS file systems,
including:
- how many MDS daemons (in total and per file system)
- which features are (or have been) enabled
- how many data pools
- approximate file system age (year + month of creation)
- how much metadata is being cached per file system
As well as:
- whether IPv4 or IPv6 addresses are used for the monitors
- whether RADOS cache tiering is enabled (and which mode)
- whether pools are replicated or erasure coded, and
which erasure code profile plugin and parameters are in use
- how many RGW daemons, zones, and zonegroups are present; which RGW frontends are in use
- aggregate stats about the CRUSH map, like which algorithms are used, how
big buckets are, how many rules are defined, and what tunables are in use
* A health warning is now generated if the average osd heartbeat ping
time exceeds a configurable threshold for any of the intervals
computed. The OSD computes 1 minute, 5 minute and 15 minute intervals
with average, minimum and maximum values. New configuration option
`mon_warn_on_slow_ping_ratio` specifies a percentage of
`osd_heartbeat_grace` to determine the threshold. A value of zero
disables the warning. New configuration option
`mon_warn_on_slow_ping_time` specified in milliseconds over-rides the
computed value, causes a warning when OSD heartbeat pings take longer
than the specified amount. New admin command `ceph daemon mgr.#
dump_osd_network [threshold]` command will list all connections with a
ping time longer than the specified threshold or value determined by
the config options, for the average for any of the 3 intervals. New
admin command `ceph daemon osd.# dump_osd_network [threshold]` will do
the same but only including heartbeats initiated by the specified OSD.
* The configuration value `osd_calc_pg_upmaps_max_stddev` used for upmap
balancing has been removed. Instead use the mgr balancer config
`upmap_max_deviation` which now is an integer number of PGs of
deviation from the target PGs per OSD. This can be set with a command
like `ceph config set mgr mgr/balancer/upmap_max_deviation 2`. The
default `upmap_max_deviation` is 1. There are situations where crush
rules would not allow a pool to ever have completely balanced PGs. For
example, if crush requires 1 replica on each of 3 racks, but there are
fewer OSDs in 1 of the racks. In those cases, the configuration value
can be increased.
Getting Ceph
------------
* Git at git://github.com/ceph/ceph.git
* Tarball at http://download.ceph.com/tarballs/ceph-12.2.13.tar.gz
* For packages, see http://docs.ceph.com/docs/master/install/get-packages/
* Release git sha1: 584a20eb0237c657dc0567da126be145106aa47e
[1]: https://lists.fedoraproject.org/archives/list/epel-announce@lists.fedorapro…
--
Abhishek Lekshmanan
SUSE Software Solutions Germany GmbH
GF: Felix Imendörffer HRB 21284 (AG Nürnberg)
Hi Cephers,
We have just posted some upcoming Ceph Days. We are looking for sponsors
and content:
* Ceph Day Istanbul: March 17
* Ceph Day Oslo: May 13
* Ceph Day Vancouver: May 13
https://ceph.com/cephdays/
Also don't forget about our big event Cephalocon Seoul March 3-5.
Registration, schedule. sponsorship and hotel information is available:
https://ceph.io/cephalocon/seoul-2020/
If you have any questions about these events, please contact us at
events(a)ceph.io.
Any help with promoting these events is greatly appreciated. Thanks!
--
Mike Perez
he/him
Ceph Community Manager
M: +1-951-572-2633
494C 5D25 2968 D361 65FB 3829 94BC D781 ADA8 8AEA
@Thingee <https://twitter.com/thingee> Thingee
<https://www.linkedin.com/thingee> <https://www.facebook.com/RedHatInc>
<https://www.redhat.com>
Hello Cephers,
We're excited to announce the schedule for Cephalocon 2020 in Seoul!
https://ceph.io/cephalocon/seoul-2020/
We have March 3rd set for the developer summit to figure out technical
challenges together in person. On March 4-5, we will have some outstanding
keynotes and fantastic breakout sessions by the Ceph community.
https://ceph2020.sched.com/
Early bird registration is available now until January 19th.
http://www.cvent.com/d/chq5w0/4W
The Ceph Foundation's diversity scholarship application is now available
until January 31st. This program provides support to those from
traditionally underrepresented and/or marginalized groups in the technology
and/or open-source communities (including, but not limited to: persons
identifying as LGBTIQ, women, persons of color, and/or persons with
disabilities) who may not otherwise have the opportunity to attend Linux
Foundation events for financial reasons.
https://zfrmz.com/2csvRBgv6M8yduKIu1oJ
Sponsorship is available with limited spots about to fill up, so we
recommend Foundation members and other companies to consider soon.
https://ceph.io/wp-content/uploads/2019/12/sponsor-Cephalocon20-112719.pdf
If you have any questions, please contact events(a)ceph.io or me. We look
forward to you joining us!
--
Mike Perez
he/him
Ceph Community Manager
M: +1-951-572-2633
494C 5D25 2968 D361 65FB 3829 94BC D781 ADA8 8AEA
@Thingee <https://twitter.com/thingee> Thingee
<https://www.linkedin.com/thingee> <https://www.facebook.com/RedHatInc>
<https://www.redhat.com>
This is the eighth backport release in the Ceph Mimic stable release
series. Its sole purpose is to fix a regression that found its way into
the previous release.
Notable Changes
---------------
* Due to a missed backport, clusters in the process of being upgraded from
13.2.6 to 13.2.7 might suffer an OSD crash in build_incremental_map_msg.
This regression was reported in https://tracker.ceph.com/issues/43106
and is fixed in 13.2.8 (this release). Users of 13.2.6 can upgrade to 13.2.8
directly - i.e., skip 13.2.7 - to avoid this.
Changelog
---------
* osd: fix sending incremental map messages (issue#43106 pr#32000, Sage Weil)
* tests: added missing point release versions (pr#32087, Yuri Weinstein)
* tests: rgw: add missing force-branch: ceph-mimic for swift tasks (pr#32033, Casey Bodley)
For a blog with links to PRs and issues please check out
https://ceph.io/releases/v13-2-8-mimic-released/
Getting Ceph
------------
* Git at git://github.com/ceph/ceph.git
* Tarball at http://download.ceph.com/tarballs/ceph-13.2.8.tar.gz
* For packages, see http://docs.ceph.com/docs/master/install/get-packages/
* Release git sha1: 5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0
--
Abhishek Lekshmanan
SUSE Software Solutions Germany GmbH
On Thu, Dec 12, 2019 at 9:06 AM Dan van der Ster <dan(a)vanderster.com> wrote:
>
> Thanks for this!
>
> We're having trouble installing the ceph-debuginfo rpm, which shows a
> negative file size (apparently because it now exceeds 2GB):
> https://pastebin.com/5bnNCGHh
>
> Does this change need to be applied to the release builds too?
> https://tracker.ceph.com/issues/39387
Good catch! I just checked and yes, we need to use the --no-database flag
>
> -- Dan
>
> On Tue, Dec 10, 2019 at 10:45 AM Abhishek Lekshmanan <abhishek(a)suse.com> wrote:
> >
> > This is the fifth release of the Ceph Nautilus release series. Among the many
> > notable changes, this release fixes a critical BlueStore bug that was introduced
> > in 14.2.3. All Nautilus users are advised to upgrade to this release.
> >
> > For the complete changelog entry, please visit the release blog at
> > https://ceph.io/releases/v14-2-5-nautilus-released/
> >
> > Notable Changes
> > ---------------
> >
> > Critical fix:
> >
> > * This release fixes a `critical BlueStore bug <https://tracker.ceph.com/issues/42223>`_
> > introduced in 14.2.3 (and also present in 14.2.4) that can lead to data
> > corruption when a separate "WAL" device is used.
> >
> > New health warnings:
> >
> > * Ceph will now issue health warnings if daemons have recently crashed. Ceph
> > has been collecting crash reports since the initial Nautilus release, but the
> > health alerts are new. To view new crashes (or all crashes, if you've just
> > upgraded)::
> >
> > ceph crash ls-new
> >
> > To acknowledge a particular crash (or all crashes) and silence the health warning::
> >
> > ceph crash archive <crash-id>
> > ceph crash archive-all
> >
> > * Ceph will now issue a health warning if a RADOS pool has a ``pg_num``
> > value that is not a power of two. This can be fixed by adjusting
> > the pool to a nearby power of two::
> >
> > ceph osd pool set <pool-name> pg_num <new-pg-num>
> >
> > Alternatively, the warning can be silenced with::
> >
> > ceph config set global mon_warn_on_pool_pg_num_not_power_of_two false
> >
> > * Ceph will issue a health warning if a RADOS pool's ``size`` is set to 1
> > or, in other words, if the pool is configured with no redundancy. Ceph will
> > stop issuing the warning if the pool size is set to the minimum
> > recommended value::
> >
> > ceph osd pool set <pool-name> size <num-replicas>
> >
> > The warning can be silenced with::
> >
> > ceph config set global mon_warn_on_pool_no_redundancy false
> >
> > * A health warning is now generated if the average osd heartbeat ping
> > time exceeds a configurable threshold for any of the intervals
> > computed. The OSD computes 1 minute, 5 minute and 15 minute
> > intervals with average, minimum and maximum values. New configuration
> > option `mon_warn_on_slow_ping_ratio` specifies a percentage of
> > `osd_heartbeat_grace` to determine the threshold. A value of zero
> > disables the warning. New configuration option `mon_warn_on_slow_ping_time`
> > specified in milliseconds over-rides the computed value, causes a warning
> > when OSD heartbeat pings take longer than the specified amount.
> > A new admin command, `ceph daemon mgr.# dump_osd_network [threshold]`, will
> > list all connections with a ping time longer than the specified threshold or
> > value determined by the config options, for the average for any of the 3 intervals.
> > Another new admin command, `ceph daemon osd.# dump_osd_network [threshold]`,
> > will do the same but only including heartbeats initiated by the specified OSD.
> >
> > Changes in the telemetry module:
> >
> > * The telemetry module now has a 'device' channel, enabled by default, that
> > will report anonymized hard disk and SSD health metrics to telemetry.ceph.com
> > in order to build and improve device failure prediction algorithms. Because
> > the content of telemetry reports has changed, you will need to re-opt-in
> > with::
> >
> > ceph telemetry on
> >
> > You can view exactly what information will be reported first with::
> >
> > ceph telemetry show
> > ceph telemetry show device # specifically show the device channel
> >
> > If you are not comfortable sharing device metrics, you can disable that
> > channel first before re-opting-in:
> >
> > ceph config set mgr mgr/telemetry/channel_crash false
> > ceph telemetry on
> >
> > * The telemetry module now reports more information about CephFS file systems,
> > including:
> >
> > - how many MDS daemons (in total and per file system)
> > - which features are (or have been) enabled
> > - how many data pools
> > - approximate file system age (year + month of creation)
> > - how many files, bytes, and snapshots
> > - how much metadata is being cached
> >
> > We have also added:
> >
> > - which Ceph release the monitors are running
> > - whether msgr v1 or v2 addresses are used for the monitors
> > - whether IPv4 or IPv6 addresses are used for the monitors
> > - whether RADOS cache tiering is enabled (and which mode)
> > - whether pools are replicated or erasure coded, and
> > which erasure code profile plugin and parameters are in use
> > - how many hosts are in the cluster, and how many hosts have each type of daemon
> > - whether a separate OSD cluster network is being used
> > - how many RBD pools and images are in the cluster, and how many pools have RBD mirroring enabled
> > - how many RGW daemons, zones, and zonegroups are present; which RGW frontends are in use
> > - aggregate stats about the CRUSH map, like which algorithms are used, how
> > big buckets are, how many rules are defined, and what tunables are in
> > use
> >
> > If you had telemetry enabled, you will need to re-opt-in with::
> >
> > ceph telemetry on
> >
> > You can view exactly what information will be reported first with::
> >
> > ceph telemetry show # see everything
> > ceph telemetry show basic # basic cluster info (including all of the new info)
> >
> > OSD:
> >
> > * A new OSD daemon command, 'dump_recovery_reservations', reveals the
> > recovery locks held (in_progress) and waiting in priority queues.
> >
> > * Another new OSD daemon command, 'dump_scrub_reservations', reveals the
> > scrub reservations that are held for local (primary) and remote (replica) PGs.
> >
> > RGW:
> >
> > * RGW now supports S3 Object Lock set of APIs allowing for a WORM model for
> > storing objects. 6 new APIs have been added put/get bucket object lock,
> > put/get object retention, put/get object legal hold.
> >
> > * RGW now supports List Objects V2
> >
> > Getting Ceph
> > ------------
> >
> > * Git at git://github.com/ceph/ceph.git
> > * Tarball at http://download.ceph.com/tarballs/ceph-14.2.5.tar.gz
> > * For packages, see http://docs.ceph.com/docs/master/install/get-packages/
> > * Release git sha1: ad5bd132e1492173c85fda2cc863152730b16a92
> >
> > --
> > Abhishek Lekshmanan
> > SUSE Software Solutions Germany GmbH
>
Hi everyone,
The next Cephalocon is coming up on March 3-5 in Seoul! The CFP is open
until Friday (get your talks in!). We expect to have the program
ready for the first week of January. Registration (early bird) will be
available soon.
We're also looking for sponsors for the conference. The prospectus is
available here:
https://ceph.io/wp-content/uploads/2019/12/sponsor-Cephalocon20-112719.pdf
Thanks!
sage
This is the fifth release of the Ceph Nautilus release series. Among the many
notable changes, this release fixes a critical BlueStore bug that was introduced
in 14.2.3. All Nautilus users are advised to upgrade to this release.
For the complete changelog entry, please visit the release blog at
https://ceph.io/releases/v14-2-5-nautilus-released/
Notable Changes
---------------
Critical fix:
* This release fixes a `critical BlueStore bug <https://tracker.ceph.com/issues/42223>`_
introduced in 14.2.3 (and also present in 14.2.4) that can lead to data
corruption when a separate "WAL" device is used.
New health warnings:
* Ceph will now issue health warnings if daemons have recently crashed. Ceph
has been collecting crash reports since the initial Nautilus release, but the
health alerts are new. To view new crashes (or all crashes, if you've just
upgraded)::
ceph crash ls-new
To acknowledge a particular crash (or all crashes) and silence the health warning::
ceph crash archive <crash-id>
ceph crash archive-all
* Ceph will now issue a health warning if a RADOS pool has a ``pg_num``
value that is not a power of two. This can be fixed by adjusting
the pool to a nearby power of two::
ceph osd pool set <pool-name> pg_num <new-pg-num>
Alternatively, the warning can be silenced with::
ceph config set global mon_warn_on_pool_pg_num_not_power_of_two false
* Ceph will issue a health warning if a RADOS pool's ``size`` is set to 1
or, in other words, if the pool is configured with no redundancy. Ceph will
stop issuing the warning if the pool size is set to the minimum
recommended value::
ceph osd pool set <pool-name> size <num-replicas>
The warning can be silenced with::
ceph config set global mon_warn_on_pool_no_redundancy false
* A health warning is now generated if the average osd heartbeat ping
time exceeds a configurable threshold for any of the intervals
computed. The OSD computes 1 minute, 5 minute and 15 minute
intervals with average, minimum and maximum values. New configuration
option `mon_warn_on_slow_ping_ratio` specifies a percentage of
`osd_heartbeat_grace` to determine the threshold. A value of zero
disables the warning. New configuration option `mon_warn_on_slow_ping_time`
specified in milliseconds over-rides the computed value, causes a warning
when OSD heartbeat pings take longer than the specified amount.
A new admin command, `ceph daemon mgr.# dump_osd_network [threshold]`, will
list all connections with a ping time longer than the specified threshold or
value determined by the config options, for the average for any of the 3 intervals.
Another new admin command, `ceph daemon osd.# dump_osd_network [threshold]`,
will do the same but only including heartbeats initiated by the specified OSD.
Changes in the telemetry module:
* The telemetry module now has a 'device' channel, enabled by default, that
will report anonymized hard disk and SSD health metrics to telemetry.ceph.com
in order to build and improve device failure prediction algorithms. Because
the content of telemetry reports has changed, you will need to re-opt-in
with::
ceph telemetry on
You can view exactly what information will be reported first with::
ceph telemetry show
ceph telemetry show device # specifically show the device channel
If you are not comfortable sharing device metrics, you can disable that
channel first before re-opting-in:
ceph config set mgr mgr/telemetry/channel_crash false
ceph telemetry on
* The telemetry module now reports more information about CephFS file systems,
including:
- how many MDS daemons (in total and per file system)
- which features are (or have been) enabled
- how many data pools
- approximate file system age (year + month of creation)
- how many files, bytes, and snapshots
- how much metadata is being cached
We have also added:
- which Ceph release the monitors are running
- whether msgr v1 or v2 addresses are used for the monitors
- whether IPv4 or IPv6 addresses are used for the monitors
- whether RADOS cache tiering is enabled (and which mode)
- whether pools are replicated or erasure coded, and
which erasure code profile plugin and parameters are in use
- how many hosts are in the cluster, and how many hosts have each type of daemon
- whether a separate OSD cluster network is being used
- how many RBD pools and images are in the cluster, and how many pools have RBD mirroring enabled
- how many RGW daemons, zones, and zonegroups are present; which RGW frontends are in use
- aggregate stats about the CRUSH map, like which algorithms are used, how
big buckets are, how many rules are defined, and what tunables are in
use
If you had telemetry enabled, you will need to re-opt-in with::
ceph telemetry on
You can view exactly what information will be reported first with::
ceph telemetry show # see everything
ceph telemetry show basic # basic cluster info (including all of the new info)
OSD:
* A new OSD daemon command, 'dump_recovery_reservations', reveals the
recovery locks held (in_progress) and waiting in priority queues.
* Another new OSD daemon command, 'dump_scrub_reservations', reveals the
scrub reservations that are held for local (primary) and remote (replica) PGs.
RGW:
* RGW now supports S3 Object Lock set of APIs allowing for a WORM model for
storing objects. 6 new APIs have been added put/get bucket object lock,
put/get object retention, put/get object legal hold.
* RGW now supports List Objects V2
Getting Ceph
------------
* Git at git://github.com/ceph/ceph.git
* Tarball at http://download.ceph.com/tarballs/ceph-14.2.5.tar.gz
* For packages, see http://docs.ceph.com/docs/master/install/get-packages/
* Release git sha1: ad5bd132e1492173c85fda2cc863152730b16a92
--
Abhishek Lekshmanan
SUSE Software Solutions Germany GmbH
Hi Cephers,
To better understand how our current users utilize Ceph, we conducted a
public community survey. This information is a guide to the community of
how we spend our contribution efforts for future development. The survey
results will remain anonymous and aggregated in future Ceph Foundation
publications to the community.
I'm pleased to announce after much discussion on the Ceph dev mailing list
[0] that the community has formed the Ceph Survey for 2019.
The deadline for this survey due to it being out later than we'd like will
be January 31st, 2020 at 11:59 PT.
https://ceph.io/user-survey/
We have discussed in the future to use the Ceph telemetry module to collect
the data to save time for our users. Please let me know of any mistakes
that need to be corrected on the survey. Thanks!
[0] -
https://lists.ceph.io/hyperkitty/list/dev@ceph.io/thread/WU374ZJP5N3NKY22X2…
--
Mike Perez
he/him
Ceph Community Manager
M: +1-951-572-2633
494C 5D25 2968 D361 65FB 3829 94BC D781 ADA8 8AEA
@Thingee <https://twitter.com/thingee> Thingee
<https://www.linkedin.com/thingee> <https://www.facebook.com/RedHatInc>
<https://www.redhat.com>
This is the seventh bugfix release of the Mimic v13.2.x long term stable
release series. We recommend all Mimic users upgrade.
For the full release notes, see
https://ceph.io/releases/v13-2-7-mimic-released/
Notable Changes
MDS:
- Cache trimming is now throttled. Dropping the MDS cache via the “ceph
tell mds.<foo> cache drop” command or large reductions in the cache size
will no longer cause service unavailability.
- Behavior with recalling caps has been significantly improved to not
attempt recalling too many caps at once, leading to instability. MDS with
a large cache (64GB+) should be more stable.
- MDS now provides a config option “mds_max_caps_per_client” (default:
1M) to limit the number of caps a client session may hold. Long running
client sessions with a large number of caps have been a source of
instability in the MDS when all of these caps need to be processed during
certain session events. It is recommended to not unnecessarily increase
this value.
- The “mds_recall_state_timeout” config parameter has been removed. Late
client recall warnings are now generated based on the number of caps the
MDS has recalled which have not been released. The new config parameters
“mds_recall_warning_threshold” (default: 32K) and
“mds_recall_warning_decay_rate” (default: 60s) set the threshold for this
warning.
- The “cache drop” admin socket command has been removed. The “ceph tell
mds.X cache drop” remains.
OSD:
- A health warning is now generated if the average osd heartbeat ping
time exceeds a configurable threshold for any of the intervals computed.
The OSD computes 1 minute, 5 minute and 15 minute intervals with average,
minimum and maximum values. New configuration option
“mon_warn_on_slow_ping_ratio” specifies a percentage of
“osd_heartbeat_grace” to determine the threshold. A value of zero disables
the warning. A new configuration option “mon_warn_on_slow_ping_time”,
specified in milliseconds, overrides the computed value, causing a warning
when OSD heartbeat pings take longer than the specified amount. A new
admin command “ceph daemon mgr.# dump_osd_network [threshold]” lists all
connections with a ping time longer than the specified threshold or value
determined by the config options, for the average for any of the 3
intervals. A new admin command ceph daemon osd.# dump_osd_network
[threshold]” does the same but only including heartbeats initiated by the
specified OSD.
- The default value of the
“osd_deep_scrub_large_omap_object_key_threshold” parameter has been
lowered to detect an object with large number of omap keys more easily.
RGW:
- radosgw-admin introduces two subcommands that allow the managing of
expire-stale objects that might be left behind after a bucket reshard in
earlier versions of RGW. One subcommand lists such objects and the other
deletes them. Read the troubleshooting section of the dynamic resharding
docs for details.
