[Ceph-announce] v12.2.7 Luminous released

This is the seventh bugfix release of Luminous v12.2.x long term stable release series. This release contains several fixes for regressions in the v12.2.6 and v12.2.5 releases. We recommend that all users upgrade. *NOTE* The v12.2.6 release has serious known regressions, while 12.2.6 wasn't formally announced in the mailing lists or blog, the packages were built and available on download.ceph.com since last week. If you installed this release, please see the upgrade procedure below. *NOTE* The v12.2.5 release has a potential data corruption issue with erasure coded pools. If you ran v12.2.5 with erasure coding, please see below. The full blog post alongwith the complete changelog is published at the official ceph blog at https://ceph.com/releases/12-2-7-luminous-released/ Upgrading from v12.2.6 ---------------------- v12.2.6 included an incomplete backport of an optimization for BlueStore OSDs that avoids maintaining both the per-object checksum and the internal BlueStore checksum. Due to the accidental omission of a critical follow-on patch, v12.2.6 corrupts (fails to update) the stored per-object checksum value for some objects. This can result in an EIO error when trying to read those objects. #. If your cluster uses FileStore only, no special action is required. This problem only affects clusters with BlueStore. #. If your cluster has only BlueStore OSDs (no FileStore), then you should enable the following OSD option:: osd skip data digest = true This will avoid setting and start ignoring the full-object digests whenever the primary for a PG is BlueStore. #. If you have a mix of BlueStore and FileStore OSDs, then you should enable the following OSD option:: osd distrust data digest = true This will avoid setting and start ignoring the full-object digests in all cases. This weakens the data integrity checks for FileStore (although those checks were always only opportunistic). If your cluster includes BlueStore OSDs and was affected, deep scrubs will generate errors about mismatched CRCs for affected objects. Currently the repair operation does not know how to correct them (since all replicas do not match the expected checksum it does not know how to proceed). These warnings are harmless in the sense that IO is not affected and the replicas are all still in sync. The number of affected objects is likely to drop (possibly to zero) on their own over time as those objects are modified. We expect to include a scrub improvement in v12.2.8 to clean up any remaining objects. Additionally, see the notes below, which apply to both v12.2.5 and v12.2.6. Upgrading from v12.2.5 or v12.2.6 --------------------------------- If you used v12.2.5 or v12.2.6 in combination with erasure coded pools, there is a small risk of corruption under certain workloads. Specifically, when: * An erasure coded pool is in use * The pool is busy with successful writes * The pool is also busy with updates that result in an error result to the librados user. RGW garbage collection is the most common example of this (it sends delete operations on objects that don't always exist.) * Some OSDs are reasonably busy. One known example of such load is FileStore splitting, although in principle any load on the cluster could also trigger the behavior. * One or more OSDs restarts. This combination can trigger an OSD crash and possibly leave PGs in a state where they fail to peer. Notably, upgrading a cluster involves OSD restarts and as such may increase the risk of encountering this bug. For this reason, for clusters with erasure coded pools, we recommend the following upgrade procedure to minimize risk: 1. Install the v12.2.7 packages. 2. Temporarily quiesce IO to cluster:: ceph osd pause 3. Restart all OSDs and wait for all PGs to become active. 4. Resume IO:: ceph osd unpause This will cause an availability outage for the duration of the OSD restarts. If this in unacceptable, an *more risky* alternative is to disable RGW garbage collection (the primary known cause of these rados operations) for the duration of the upgrade:: 1. Set ``rgw_enable_gc_threads = false`` in ceph.conf 2. Restart all radosgw daemons 3. Upgrade and restart all OSDs 4. Remove ``rgw_enable_gc_threads = false`` from ceph.conf 5. Restart all radosgw daemons Upgrading from other versions ----------------------------- If your cluster did not run v12.2.5 or v12.2.6 then none of the above issues apply to you and you should upgrade normally. v12.2.7 Changelog ----------------- * mon/AuthMonitor: improve error message (issue#21765, pr#22963, Douglas Fuller) * osd/PG: do not blindly roll forward to log.head (issue#24597, pr#22976, Sage Weil) * osd/PrimaryLogPG: rebuild attrs from clients (issue#24768 , pr#22962, Sage Weil) * osd: work around data digest problems in 12.2.6 (version 2) (issue#24922, pr#23055, Sage Weil) * rgw: objects in cache never refresh after rgw_cache_expiry_interval (issue#24346, pr#22369, Casey Bodley, Matt Benjamin) Notable changes in v12.2.6 Luminous =================================== :note: This is a broken release with serious known regressions. Do not install it. The release notes below are to help track the changes that went in 12.2.6 and hence a part of 12.2.7 - *Auth*: * In 12.2.4 and earlier releases, keyring caps were not checked for validity, so the caps string could be anything. As of 12.2.6, caps strings are validated and providing a keyring with an invalid caps string to, e.g., "ceph auth add" will result in an error. * CVE 2018-1128: auth: cephx authorizer subject to replay attack (issue#24836, Sage Weil) * CVE 2018-1129: auth: cephx signature check is weak (issue#24837, Sage Weil) * CVE 2018-10861: mon: auth checks not correct for pool ops (issue#24838, Jason Dillaman) - The config-key interface can store arbitrary binary blobs but JSON can only express printable strings. If binary blobs are present, the 'ceph config-key dump' command will show them as something like ``<<< binary blob of length N >>>``. The full changelog for 12.2.6 is published in the release blog. Getting ceph: * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-12.2.7.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: 3ec878d1e53e1aeb47a9f619c49d9e7c0aa384d5 -- Abhishek Lekshmanan SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)