This is the seventh bugfix release of Luminous v12.2.x long term
stable release series. This release contains several fixes for
regressions in the v12.2.6 and v12.2.5 releases. We recommend that
all users upgrade.
*NOTE* The v12.2.6 release has serious known regressions, while 12.2.6
wasn't formally announced in the mailing lists or blog, the packages
were built and available on download.ceph.com since last week. If you
installed this release, please see the upgrade procedure below.
*NOTE* The v12.2.5 release has a potential data corruption issue with
erasure coded pools. If you ran v12.2.5 with erasure coding, please see
below.
The full blog post alongwith the complete changelog is published at the
official ceph blog at https://ceph.com/releases/12-2-7-luminous-released/
Upgrading from v12.2.6
----------------------
v12.2.6 included an incomplete backport of an optimization for
BlueStore OSDs that avoids maintaining both the per-object checksum
and the internal BlueStore checksum. Due to the accidental omission
of a critical follow-on patch, v12.2.6 corrupts (fails to update) the
stored per-object checksum value for some objects. This can result in
an EIO error when trying to read those objects.
#. If your cluster uses FileStore only, no special action is required.
This problem only affects clusters with BlueStore.
#. If your cluster has only BlueStore OSDs (no FileStore), then you
should enable the following OSD option::
osd skip data digest = true
This will avoid setting and start ignoring the full-object digests
whenever the primary for a PG is BlueStore.
#. If you have a mix of BlueStore and FileStore OSDs, then you should
enable the following OSD option::
osd distrust data digest = true
This will avoid setting and start ignoring the full-object digests
in all cases. This weakens the data integrity checks for
FileStore (although those checks were always only opportunistic).
If your cluster includes BlueStore OSDs and was affected, deep scrubs
will generate errors about mismatched CRCs for affected objects.
Currently the repair operation does not know how to correct them
(since all replicas do not match the expected checksum it does not
know how to proceed). These warnings are harmless in the sense that
IO is not affected and the replicas are all still in sync. The number
of affected objects is likely to drop (possibly to zero) on their own
over time as those objects are modified. We expect to include a scrub
improvement in v12.2.8 to clean up any remaining objects.
Additionally, see the notes below, which apply to both v12.2.5 and v12.2.6.
Upgrading from v12.2.5 or v12.2.6
---------------------------------
If you used v12.2.5 or v12.2.6 in combination with erasure coded
pools, there is a small risk of corruption under certain workloads.
Specifically, when:
* An erasure coded pool is in use
* The pool is busy with successful writes
* The pool is also busy with updates that result in an error result to
the librados user. RGW garbage collection is the most common
example of this (it sends delete operations on objects that don't
always exist.)
* Some OSDs are reasonably busy. One known example of such load is
FileStore splitting, although in principle any load on the cluster
could also trigger the behavior.
* One or more OSDs restarts.
This combination can trigger an OSD crash and possibly leave PGs in a state
where they fail to peer.
Notably, upgrading a cluster involves OSD restarts and as such may
increase the risk of encountering this bug. For this reason, for
clusters with erasure coded pools, we recommend the following upgrade
procedure to minimize risk:
1. Install the v12.2.7 packages.
2. Temporarily quiesce IO to cluster::
ceph osd pause
3. Restart all OSDs and wait for all PGs to become active.
4. Resume IO::
ceph osd unpause
This will cause an availability outage for the duration of the OSD
restarts. If this in unacceptable, an *more risky* alternative is to
disable RGW garbage collection (the primary known cause of these rados
operations) for the duration of the upgrade::
1. Set ``rgw_enable_gc_threads = false`` in ceph.conf
2. Restart all radosgw daemons
3. Upgrade and restart all OSDs
4. Remove ``rgw_enable_gc_threads = false`` from ceph.conf
5. Restart all radosgw daemons
Upgrading from other versions
-----------------------------
If your cluster did not run v12.2.5 or v12.2.6 then none of the above
issues apply to you and you should upgrade normally.
v12.2.7 Changelog
-----------------
* mon/AuthMonitor: improve error message (issue#21765, pr#22963, Douglas Fuller)
* osd/PG: do not blindly roll forward to log.head (issue#24597, pr#22976, Sage Weil)
* osd/PrimaryLogPG: rebuild attrs from clients (issue#24768 , pr#22962, Sage Weil)
* osd: work around data digest problems in 12.2.6 (version 2) (issue#24922, pr#23055, Sage Weil)
* rgw: objects in cache never refresh after rgw_cache_expiry_interval (issue#24346, pr#22369, Casey Bodley, Matt Benjamin)
Notable changes in v12.2.6 Luminous
===================================
:note: This is a broken release with serious known regressions. Do not
install it. The release notes below are to help track the changes that
went in 12.2.6 and hence a part of 12.2.7
- *Auth*:
* In 12.2.4 and earlier releases, keyring caps were not checked for validity,
so the caps string could be anything. As of 12.2.6, caps strings are
validated and providing a keyring with an invalid caps string to, e.g.,
"ceph auth add" will result in an error.
* CVE 2018-1128: auth: cephx authorizer subject to replay attack (issue#24836, Sage Weil)
* CVE 2018-1129: auth: cephx signature check is weak (issue#24837, Sage Weil)
* CVE 2018-10861: mon: auth checks not correct for pool ops (issue#24838, Jason Dillaman)
- The config-key interface can store arbitrary binary blobs but JSON
can only express printable strings. If binary blobs are present,
the 'ceph config-key dump' command will show them as something like
``<<< binary blob of length N >>>``.
The full changelog for 12.2.6 is published in the release blog.
Getting ceph:
* Git at git://github.com/ceph/ceph.git
* Tarball at http://download.ceph.com/tarballs/ceph-12.2.7.tar.gz
* For packages, see http://docs.ceph.com/docs/master/install/get-packages/
* Release git sha1: 3ec878d1e53e1aeb47a9f619c49d9e7c0aa384d5
--
Abhishek Lekshmanan
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton,
HRB 21284 (AG Nürnberg)
Hi everyone,
tl;dr: Please avoid the 12.2.6 packages that are currently present on
download.ceph.com. We will have a 12.2.7 published ASAP (probably
Monday).
If you do not use bluestore or erasure-coded pools, none of the issues
affect you.
Details:
We built 12.2.6 and pushed it to the repos Wednesday, but as that was
happening realized there was a potentially dangerous regression in
12.2.5[1] that an upgrade might exacerbate. While we sorted that issue
out, several people noticed the updated version in the repo and
upgraded. That turned up two other regressions[2][3]. We have fixes for
those, but are working on an additional fix to make the damage from [3]
be transparently repaired.
More details:
-- [1] http://tracker.ceph.com/issues/24597 --
This is actually a regression in 12.2.5 that affects erasure-coded pools.
If there are (1) normal erasure code writes, and simultanously (2) erasure
code writes that result in rados returning an error (for example, a delete
of a non-existent object, which commonly happens when rgw is doing garbage
collection), and (3) OSDs that are somewhat heavily loaded and then
restart, then the bug might incorrectly roll-forward the in-progress EC
operations. When the PG repeers this results in an OSD crash like
src/os/filestore/FileStore.cc: 5524: FAILED assert(0 == "ERROR: source must exist")
It seems to affect filestore and busy clusters with this specific
workload. The OSDs recover once restarted. However, it is also unclear
whether it damages the objects in question. For this reason, please avoid
unnecessary OSD restarts if you are running 12.2.5 or 12.2.6. When we
release 12.2.7, we will have an upgrade procedure in the release notes
that quiesces RADOS IO to minimize the probability that this bug will
affect you.
If you do not have erasure-coded pools, this bug does not affect you.
-- [2] https://tracker.ceph.com/issues/24903 --
ceph-volume has had a bug for a while that leaves the
/var/lib/ceph/osd/*/block.db or block.wal symlinks for bluestore OSDs
owned by root:root. This didn't matter because bluestore was ignoring
these symlinks and using an internally stored value instead.
Both of these were fixed/changed in 12.2.6. However, after upgrading and
restarting, the symlink is still present in the /var/lib/ceph/osd/*/
tmpfs and the OSD won't restart. Rerunning ceph-volume will fix it, as
will manually chown -h ceph:ceph /var/lib/ceph/osd/*/block*, or a reboot.
12.2.7 has a packaging fix to fixed this up on upgrade so there is no
disruption.
If you do not run bluestore, this bug does not affect you.
-- [3] https://tracker.ceph.com/issues/23871 --
We modified the OSD recently to avoid storing full-object CRCs when
bluestore is in use because those CRCs are redundant. There was a bug in
this code that was later fixed in master. This code was backported to
luminous, but the follow-on fix was missed. The result is that a sequence
of
- running 12.2.5
- deep-scrub (updates stored whole-object crc)
- upgrade to 12.2.6
- writefull to existing (on 12.2.6) fails to clear the whole-object crc
- read of full object -> crc mismatch
which leads to an (incorrect) EIO error. We have fixed the original
problem by backporting the missing fix. However, users who mistakenly
installed 12.2.6 may have many objects with a mismatched whole-object crc.
We are currently working on a fix to ignore the whole-object CRC if the
same conditions are met that make us skip them entirely (i.e., running
bluestore), and to clear/repair them on scrub. Once this is done, we'll
push out 12.2.7.
If you do not run bluestore, this bug does no affect you.
We don't have an easy workaround for this one at the moment,
unfortunately.
Exciting week! Thanks everyone,
sage
We're glad to announce v10.2.11 release of the Jewel stable release
series. This point releases brings a number of important bugfixes and
has a few important security fixes. This is most likely going to be the
final Jewel release (shine on you crazy diamond). We thank everyone in
the community for contributing towards this release and particularly
want to thank Nathan and Yuri for their relentless efforts in
backporting and testing this release.
We recommend that all Jewel 10.2.x users upgrade.
Notable Changes
---------------
* CVE 2018-1128: auth: cephx authorizer subject to replay attack (issue#24836 http://tracker.ceph.com/issues/24836, Sage Weil)
* CVE 2018-1129: auth: cephx signature check is weak (issue#24837 http://tracker.ceph.com/issues/24837, Sage Weil)
* CVE 2018-10861: mon: auth checks not correct for pool ops (issue#24838 http://tracker.ceph.com/issues/24838, Jason Dillaman)
* The RBD C API's rbd_discard method and the C++ API's Image::discard method
now enforce a maximum length of 2GB. This restriction prevents overflow of
the result code.
* New OSDs will now use rocksdb for omap data by default, rather than
leveldb. omap is used by RGW bucket indexes and CephFS directories,
and when a single leveldb grows to 10s of GB with a high write or
delete workload, it can lead to high latency when leveldb's
single-threaded compaction cannot keep up. rocksdb supports multiple
threads for compaction, which avoids this problem.
* The CephFS client now catches failures to clear dentries during startup
and refuses to start as consistency and untrimmable cache issues may
develop. The new option client_die_on_failed_dentry_invalidate (default:
true) may be turned off to allow the client to proceed (dangerous!).
* In 10.2.10 and earlier releases, keyring caps were not checked for validity,
so the caps string could be anything. As of 10.2.11, caps strings are
validated and providing a keyring with an invalid caps string to, e.g.,
"ceph auth add" will result in an error.
The changelog and the full release notes are at the release blog entry
at https://ceph.com/releases/v10-2-11-jewel-released/
Getting Ceph
------------
* Git at git://github.com/ceph/ceph.git
* Tarball at http://download.ceph.com/tarballs/ceph-10.2.11.tar.gz
* For packages, see http://docs.ceph.com/docs/master/install/get-packages/
* Release git sha1: e4b061b47f07f583c92a050d9e84b1813a35671e
Best,
Abhishek
--
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton,
HRB 21284 (AG Nürnberg)
