On Fri, Apr 2, 2021 at 12:08 PM Alexander Sporleder
<asporleder(a)age.mpg.de> wrote:
>
> Hello Sage, thank you for your response!
>
> I had some problems updating 15.2.8 -> 15.2.9 but after updating Podman
> to 3.0.1 and Ceph to 15.2.10 everything was fine again.
>
> Then I started the update 15.2.10 -> 16.2.0 and in the beging
> everything worked well. But at some point the update got stucked and
> something broke the dashboard (port is in use). I stopped the update
> but it was not possible to start the update process again without the
> loop.
>
> Now my mons, the mgr, a few OSDs and cephadm are V16.2.0.
>
> Unfortunately the mgr is not logging to a file after I converted the
> cluster to cephadm.
ceph config set global log_to_file true
You might also need to chown -R 167.167 /var/log/ceph/*/. if you're on
debian/ubuntu (the packages installed on the host may have fiddled
with the ownership; the uid for fedora/rhel/centos is different than
the one for debian/ubuntu).
s
Hello,
I tried to upgrade one node in my cluster to pacific in Ubuntu 20.04.2 and
kernel 5.8.0-49. After upgrade OSDs wont start giving below error message
This non-cephadm cluster is running bluestore OSDs which many were
created in Luminous.
2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1
bluestore(/var/lib/ceph/osd/ceph-29/block) _read_bdev_label failed to open
/var/lib/ceph/osd/ceph-29/block: (1) Operation not permitted
2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 ESC[0;31m ** ERROR: unable to
open OSD superblock on /var/lib/ceph/osd/ceph-29: (2) No such file or
directoryESC[0m
the directories are existing and have right permission. I have downgraded
back to octopus and everything is running ok
df -alh
tmpfs 16G 28K 16G 1%
/var/lib/ceph/osd/ceph-32
tmpfs 16G 28K 16G 1%
/var/lib/ceph/osd/ceph-29
tmpfs 16G 28K 16G 1%
/var/lib/ceph/osd/ceph-33
tmpfs 16G 28K 16G 1%
/var/lib/ceph/osd/ceph-31
tmpfs 16G 28K 16G 1%
/var/lib/ceph/osd/ceph-28
tmpfs 16G 52K 16G 1%
/var/lib/ceph/osd/ceph-30
Hello Ceph user list!
I tried to update Ceph 15.2.10 to 16.2.0 via ceph orch. In the
beginning everything seems to work fine and the new MGR and MONs where
deployed. But now I enden up in a pulling loop and I am unable to fix
the issue by my self.
#ceph -W cephadm --watch-debu
2021-04-02T10:36:20.704960+0200 mgr.mon-a-02.tvcrfq [INF] Upgrade: Need
to upgrade myself (mgr.mon-a-02.tvcrfq)
2021-04-02T10:36:21.837596+0200 mgr.mon-a-02.tvcrfq [INF] Upgrade:
Pulling
ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e0
0951949000 on mon-a-01
2021-04-02T10:36:24.591487+0200 mgr.mon-a-02.tvcrfq [INF] Upgrade:
image
ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e0
0951949000 pull on mon-a-01 got new digests
['docker.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6
c43120b5e5e00951949000',
'docker.io/ceph/ceph@sha256:9b04c0f15704c49591640a37c7adfd40ffad0a4b42f
ecb950c3407687cb4f29a'] (not
['ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5
e00951949000']), restarting
5704c49591640a37c7adfd40ffad0a4b42fecb950c3407687cb4f29a']), restarting
2021-04-02T10:36:37.054786+0200 mgr.mon-a-02.tvcrfq [INF] Upgrade: Need
to upgrade myself (mgr.mon-a-02.tvcrfq)
2021-04-02T10:36:38.419014+0200 mgr.mon-a-02.tvcrfq [INF] Upgrade:
Pulling
ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e0
0951949000 on mon-a-01
2021-04-02T10:36:41.172835+0200 mgr.mon-a-02.tvcrfq [INF] Upgrade:
image
ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e0
0951949000 pull on mon-a-01 got new digests
['docker.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6
c43120b5e5e00951949000',
'docker.io/ceph/ceph@sha256:9b04c0f15704c49591640a37c7adfd40ffad0a4b42f
ecb950c3407687cb4f29a'] (not
['ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5
e00951949000']), restarting
After I stoppend the update I got the following health error:
Module 'cephadm' has failed: 'NoneType' object has no attribute
'target_digests'
Thanks in advance!
Best,
Alex
Dear ceph users,
On one of our clusters I have some difficulties with the upmap
balancer. We started with a reasonably well balanced cluster (using the
balancer in upmap mode). After a node failure, we crush reweighted all
the OSDs of the node to take it out of the cluster - and waited for the
cluster to rebalance. Obviously, this significantly changes the crush
map - hence the nice balance created by the balancer was gone. The
recovery mostly completed - but some of the OSDs became too full - so we
neded up with a few PGs that were backfill_toofull. The cluster has
plenty of space (overall perhaps 65% full), only a few OSDs are >90% (we
have backfillfull_ratio at 92%). The balancer refuses to change
anything since the cluster is not clean. Yet - the cluster can't become
clean without a few upmaps to help the top 3 or 4 most full OSDs.
I would think this is a fairly common situation - trying to recover
after some failure. Are there any recommendations on how to proceed?
Obviously I can manually find and insert upmaps - but for a large
cluster with tens of thousands of PGs, that isn't too practical. Is
there a way to tell the balancer to still do something even though some
PGs are undersized (with a quick look at the python module - I didn't
see any)?
The cluster is on Nautilus 14.2.15.
Thanks,
Andras
Hi,
just installed pacific on our test-cluster. This really is a minimal, but
fully functional cluster.
Everything works as expected, except for the new (and by me anticipated)
cephfs-top.
When I run that tool, it says: "cluster ceph does not exist"
If I point it to the correct config file:
# cephfs-top --conffile /etc/ceph/ceph.conf
I still get the same error.
Doesn't matter if I run this as the ceph user or as root.
I added the "client.fstop"-user as required by the documentation.
module "stats" is enabled and functioning (tested with "ceph fs perf
stats").
Anyone any suggestions what might be wrong?
Regards,
Erwin
We're glad to announce the first release of the Pacific v16.2.0 stable
series. There have been a lot of changes across components from the
previous Ceph releases, and we advise everyone to go through the release
and upgrade notes carefully.
Major Changes from Octopus
--------------------------
General
~~~~~~~
* Cephadm can automatically upgrade an Octopus cluster to Pacific with a single
command to start the process.
* Cephadm has improved significantly over the past year, with improved
support for RGW (standalone and multisite), and new support for NFS
and iSCSI. Most of these changes have already been backported to
recent Octopus point releases, but with the Pacific release we will
switch to backporting bug fixes only.
* Packages are built for the following distributions:
- CentOS 8
- Ubuntu 20.04 (Focal)
- Ubuntu 18.04 (Bionic)
- Debian Buster
- Container image (based on CentOS 8)
With the exception of Debian Buster, packages and containers are
built for both x86_64 and aarch64 (arm64) architectures.
Note that cephadm clusters may work on many other distributions,
provided Python 3 and a recent version of Docker or Podman is
available to manage containers. For more information, see
`cephadm-host-requirements`.
Dashboard
~~~~~~~~~
The `mgr-dashboard` brings improvements in the following management areas:
* Orchestrator/Cephadm:
- Host management: maintenance mode, labels.
- Services: display placement specification.
- OSD: disk replacement, display status of ongoing deletion, and improved
health/SMART diagnostics reporting.
* Official `mgr ceph api`:
- OpenAPI v3 compliant.
- Stability commitment starting from Pacific release.
- Versioned via HTTP `Accept` header (starting with v1.0).
- Thoroughly tested (>90% coverage and per Pull Request validation).
- Fully documented.
* RGW:
- Multi-site synchronization monitoring.
- Management of multiple RGW daemons and their resources (buckets and users).
- Bucket and user quota usage visualization.
- Improved configuration of S3 tenanted users.
* Security (multiple enhancements and fixes resulting from a pen testing conducted by IBM):
- Account lock-out after a configurable number of failed log-in attempts.
- Improved cookie policies to mitigate XSS/CSRF attacks.
- Reviewed and improved security in HTTP headers.
- Sensitive information reviewed and removed from logs and error messages.
- TLS 1.0 and 1.1 support disabled.
- Debug mode when enabled triggers HEALTH_WARN.
* Pools:
- Improved visualization of replication and erasure coding modes.
- CLAY erasure code plugin supported.
* Alerts and notifications:
- Alert triggered on MTU mismatches in the cluster network.
- Favicon changes according cluster status.
* Other:
- Landing page: improved charts and visualization.
- Telemetry configuration wizard.
- OSDs: management of individual OSD flags.
- RBD: per-RBD image Grafana dashboards.
- CephFS: Dirs and Caps displayed.
- NFS: v4 support only (v3 backward compatibility planned).
- Front-end: Angular 10 update.
RADOS
~~~~~
* Pacific introduces RocksDB sharding, which reduces disk space requirements.
* Ceph now provides QoS between client I/O and background operations via the
mclock scheduler.
* The balancer is now on by default in upmap mode to improve distribution of
PGs across OSDs.
* The output of `ceph -s` has been improved to show recovery progress in
one progress bar. More detailed progress bars are visible via the
`ceph progress` command.
RBD block storage
~~~~~~~~~~~~~~~~~
* Image live-migration feature has been extended to support external data
sources. Images can now be instantly imported from local files, remote
files served over HTTP(S) or remote S3 buckets in `raw` (`rbd export v1`)
or basic `qcow` and `qcow2` formats. Support for `rbd export v2`
format, advanced QCOW features and `rbd export-diff` snapshot differentials
is expected in future releases.
* Initial support for client-side encryption has been added. This is based
on LUKS and in future releases will allow using per-image encryption keys
while maintaining snapshot and clone functionality -- so that parent image
and potentially multiple clone images can be encrypted with different keys.
* A new persistent write-back cache is available. The cache operates in
a log-structured manner, providing full point-in-time consistency for the
backing image. It should be particularly suitable for PMEM devices.
* A Windows client is now available in the form of `librbd.dll` and
`rbd-wnbd` (Windows Network Block Device) daemon. It allows mapping,
unmapping and manipulating images similar to `rbd-nbd`.
* librbd API now offers quiesce/unquiesce hooks, allowing for coordinated
snapshot creation.
RGW object storage
~~~~~~~~~~~~~~~~~~
* Initial support for S3 Select. See `s3-select-feature-table` for supported queries.
* Bucket notification topics can be configured as `persistent`, where events
are recorded in rados for reliable delivery.
* Bucket notifications can be delivered to SSL-enabled AMQP endpoints.
* Lua scripts can be run during requests and access their metadata.
* SSE-KMS now supports KMIP as a key management service.
* Multisite data logs can now be deployed on `cls_fifo` to avoid large omap
cluster warnings and make their trimming cheaper. See `rgw_data_log_backing`.
CephFS distributed file system
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
* The CephFS MDS modifies on-RADOS metadata such that the new format is no
longer backwards compatible. It is not possible to downgrade a file system from
Pacific (or later) to an older release.
* Multiple file systems in a single Ceph cluster is now stable. New Ceph
clusters enable support for multiple file systems by default. Existing clusters
must still set the "enable_multiple" flag on the FS. See also
`cephfs-multifs`.
* A new `mds_autoscaler` `ceph-mgr` plugin is available for automatically
deploying MDS daemons in response to changes to the `max_mds` configuration.
Expect further enhancements in the future to simplify and automate MDS scaling.
* `cephfs-top` is a new utility for looking at performance metrics from CephFS
clients. It is development preview quality and will have bugs. For more
information, see `cephfs-top`.
* A new `snap_schedule` `ceph-mgr` plugin provides a command toolset for
scheduling snapshots on a CephFS file system. For more information, see
`snap-schedule`.
* First class NFS gateway support in Ceph is here! It's now possible to create
scale-out ("active-active") NFS gateway clusters that export CephFS using
a few commands. The gateways are deployed via cephadm (or Rook, in the future).
For more information, see `cephfs-nfs`.
* Multiple active MDS file system scrub is now stable. It is no longer necessary
to set `max_mds` to 1 and wait for non-zero ranks to stop. Scrub commands
can only be sent to rank 0: `ceph tell mds.<fs_name>:0 scrub start /path ...`.
For more information, see `mds-scrub`.
* Ephemeral pinning -- policy based subtree pinning -- is considered stable.
`mds_export_ephemeral_random` and `mds_export_ephemeral_distributed` now
default to true. For more information, see `cephfs-ephemeral-pinning`.
* A new `cephfs-mirror` daemon is available to mirror CephFS file systems to
a remote Ceph cluster.
* A Windows client is now available for connecting to CephFS. This is offered
through a new `ceph-dokan` utility which operates via the Dokan userspace
API, similar to FUSE. For more information, see `ceph-dokan`.
Upgrading from Octopus or Nautilus
----------------------------------
Before starting, make sure your cluster is stable and healthy (no down or
recovering OSDs). (This is optional, but recommended.)
Upgrading cephadm clusters
~~~~~~~~~~~~~~~~~~~~~~~~~~
If your cluster is deployed with cephadm (first introduced in Octopus), then
the upgrade process is entirely automated. To initiate the upgrade,
ceph orch upgrade start --ceph-version 16.2.0
The same process is used to upgrade to future minor releases.
Upgrade progress can be monitored with `ceph -s` (which provides a simple
progress bar) or more verbosely with
ceph -W cephadm
The upgrade can be paused or resumed with
ceph orch upgrade pause # to pause
ceph orch upgrade resume # to resume
or canceled with
ceph orch upgrade stop
Note that canceling the upgrade simply stops the process; there is no ability to
downgrade back to Octopus.
Upgrading non-cephadm clusters
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If you cluster is running Octopus (15.2.x), you might choose
to first convert it to use cephadm so that the upgrade to Pacific
is automated (see above). For more information, see
`cephadm-adoption`.
#. Set the `noout` flag for the duration of the upgrade. (Optional,
but recommended.):
# ceph osd set noout
#. Upgrade monitors by installing the new packages and restarting the
monitor daemons. For example, on each monitor host,:
# systemctl restart ceph-mon.target
Once all monitors are up, verify that the monitor upgrade is
complete by looking for the `octopus` string in the mon
map. The command:
# ceph mon dump | grep min_mon_release
should report:
min_mon_release 16 (pacific)
If it doesn't, that implies that one or more monitors hasn't been
upgraded and restarted and/or the quorum does not include all monitors.
#. Upgrade `ceph-mgr` daemons by installing the new packages and
restarting all manager daemons. For example, on each manager host,:
# systemctl restart ceph-mgr.target
Verify the `ceph-mgr` daemons are running by checking `ceph
-s`
# ceph -s
...
services:
mon: 3 daemons, quorum foo,bar,baz
mgr: foo(active), standbys: bar, baz
...
#. Upgrade all OSDs by installing the new packages and restarting the
ceph-osd daemons on all OSD hosts:
# systemctl restart ceph-osd.target
Note that if you are upgrading from Nautilus, the first time each
OSD starts, it will do a format conversion to improve the
accounting for "omap" data. This may take a few minutes to as much
as a few hours (for an HDD with lots of omap data). You can
disable this automatic conversion with:
# ceph config set osd bluestore_fsck_quick_fix_on_mount false
You can monitor the progress of the OSD upgrades with the
`ceph versions` or `ceph osd versions` commands:
# ceph osd versions
{
"ceph version 14.2.5 (...) nautilus (stable)": 12,
"ceph version 16.2.0 (...) pacific (stable)": 22,
}
#. Upgrade all CephFS MDS daemons. For each CephFS file system,
#. Disable standby_replay:
# ceph fs set <fs_name> allow_standby_replay false
#. Reduce the number of ranks to 1. (Make note of the original
number of MDS daemons first if you plan to restore it later.):
# ceph status
# ceph fs set <fs_name> max_mds 1
#. Wait for the cluster to deactivate any non-zero ranks by
periodically checking the status:
# ceph status
#. Take all standby MDS daemons offline on the appropriate hosts with:
# systemctl stop ceph-mds@<daemon_name>
#. Confirm that only one MDS is online and is rank 0 for your FS:
# ceph status
#. Upgrade the last remaining MDS daemon by installing the new
packages and restarting the daemon:
# systemctl restart ceph-mds.target
#. Restart all standby MDS daemons that were taken offline:
# systemctl start ceph-mds.target
#. Restore the original value of `max_mds` for the volume:
# ceph fs set <fs_name> max_mds <original_max_mds>
#. Upgrade all radosgw daemons by upgrading packages and restarting
daemons on all hosts:
# systemctl restart ceph-radosgw.target
#. Complete the upgrade by disallowing pre-Pacific OSDs and enabling
all new Pacific-only functionality:
# ceph osd require-osd-release pacific
#. If you set `noout` at the beginning, be sure to clear it with:
# ceph osd unset noout
#. Consider transitioning your cluster to use the cephadm deployment
and orchestration framework to simplify cluster management and
future upgrades. For more information on converting an existing
cluster to cephadm, see `cephadm-adoption`.
Post-upgrade
~~~~~~~~~~~~
#. Verify the cluster is healthy with `ceph health`.
If your CRUSH tunables are older than Hammer, Ceph will now issue a
health warning. If you see a health alert to that effect, you can
revert this change with:
ceph config set mon mon_crush_min_required_version firefly
If Ceph does not complain, however, then we recommend you also
switch any existing CRUSH buckets to straw2, which was added back
in the Hammer release. If you have any 'straw' buckets, this will
result in a modest amount of data movement, but generally nothing
too severe.:
ceph osd getcrushmap -o backup-crushmap
ceph osd crush set-all-straw-buckets-to-straw2
If there are problems, you can easily revert with:
ceph osd setcrushmap -i backup-crushmap
Moving to 'straw2' buckets will unlock a few recent features, like
the `crush-compat` `balancer <balancer>` mode added back in Luminous.
#. If you did not already do so when upgrading from Mimic, we
recommened you enable the new `v2 network protocol <msgr2>`,
issue the following command:
ceph mon enable-msgr2
This will instruct all monitors that bind to the old default port
6789 for the legacy v1 protocol to also bind to the new 3300 v2
protocol port. To see if all monitors have been updated,:
ceph mon dump
and verify that each monitor has both a `v2:` and `v1:` address
listed.
#. Consider enabling the `telemetry module <telemetry>` to send
anonymized usage statistics and crash information to the Ceph
upstream developers. To see what would be reported (without actually
sending any information to anyone),:
ceph mgr module enable telemetry
ceph telemetry show
If you are comfortable with the data that is reported, you can opt-in to
automatically report the high-level cluster metadata with:
ceph telemetry on
The public dashboard that aggregates Ceph telemetry can be found at
https://telemetry-public.ceph.com.
For more information about the telemetry module, see `the
telemetry documentation.
Upgrade from pre-Nautilus releases (like Mimic or Luminous)
-----------------------------------------------------------
You must first upgrade to Nautilus (14.2.z) or Octopus (15.2.z) before
upgrading to Pacific.
Notable Changes
---------------
* A new library is available, libcephsqlite. It provides a SQLite Virtual File
System (VFS) on top of RADOS. The database and journals are striped over
RADOS across multiple objects for virtually unlimited scaling and throughput
only limited by the SQLite client. Applications using SQLite may change to
the Ceph VFS with minimal changes, usually just by specifying the alternate
VFS. We expect the library to be most impactful and useful for applications
that were storing state in RADOS omap, especially without striping which
limits scalability.
* New `bluestore_rocksdb_options_annex` config parameter. Complements
`bluestore_rocksdb_options` and allows setting rocksdb options without
repeating the existing defaults.
* $pid expansion in config paths like `admin_socket` will now properly expand
to the daemon pid for commands like `ceph-mds` or `ceph-osd`. Previously
only `ceph-fuse`/`rbd-nbd` expanded `$pid` with the actual daemon pid.
* The allowable options for some `radosgw-admin` commands have been changed.
* `mdlog-list`, `datalog-list`, `sync-error-list` no longer accepts
start and end dates, but does accept a single optional start marker.
* `mdlog-trim`, `datalog-trim`, `sync-error-trim` only accept a
single marker giving the end of the trimmed range.
* Similarly the date ranges and marker ranges have been removed on
the RESTful DATALog and MDLog list and trim operations.
* ceph-volume: The `lvm batch` subcommand received a major rewrite. This
closed a number of bugs and improves usability in terms of size specification
and calculation, as well as idempotency behaviour and disk replacement
process.
Please refer to https://docs.ceph.com/en/latest/ceph-volume/lvm/batch/ for
more detailed information.
* Configuration variables for permitted scrub times have changed. The legal
values for `osd_scrub_begin_hour` and `osd_scrub_end_hour` are 0 - 23.
The use of 24 is now illegal. Specifying `0` for both values causes every
hour to be allowed. The legal values for `osd_scrub_begin_week_day` and
`osd_scrub_end_week_day` are 0 - 6. The use of 7 is now illegal.
Specifying `0` for both values causes every day of the week to be allowed.
* volume/nfs: Recently "ganesha-" prefix from cluster id and nfs-ganesha common
config object was removed, to ensure consistent namespace across different
orchestrator backends. Please delete any existing nfs-ganesha clusters prior
to upgrading and redeploy new clusters after upgrading to Pacific.
* A new health check, DAEMON_OLD_VERSION, will warn if different versions of Ceph are running
on daemons. It will generate a health error if multiple versions are detected.
This condition must exist for over mon_warn_older_version_delay (set to 1 week by default) in order for the
health condition to be triggered. This allows most upgrades to proceed
without falsely seeing the warning. If upgrade is paused for an extended
time period, health mute can be used like this
"ceph health mute DAEMON_OLD_VERSION --sticky". In this case after
upgrade has finished use "ceph health unmute DAEMON_OLD_VERSION".
* MGR: progress module can now be turned on/off, using the commands:
`ceph progress on` and `ceph progress off`.
* An AWS-compliant API: "GetTopicAttributes" was added to replace the existing "GetTopic" API. The new API
should be used to fetch information about topics used for bucket notifications.
* librbd: The shared, read-only parent cache's config option `immutable_object_cache_watermark` now has been updated
to property reflect the upper cache utilization before space is reclaimed. The default `immutable_object_cache_watermark`
now is `0.9`. If the capacity reaches 90% the daemon will delete cold cache.
* OSD: the option `osd_fast_shutdown_notify_mon` has been introduced to allow
the OSD to notify the monitor it is shutting down even if `osd_fast_shutdown`
is enabled. This helps with the monitor logs on larger clusters, that may get
many 'osd.X reported immediately failed by osd.Y' messages, and confuse tools.
* The mclock scheduler has been refined. A set of built-in profiles are now available that
provide QoS between the internal and external clients of Ceph. To enable the mclock
scheduler, set the config option "osd_op_queue" to "mclock_scheduler". The
"high_client_ops" profile is enabled by default, and allocates more OSD bandwidth to
external client operations than to internal client operations (such as background recovery
and scrubs). Other built-in profiles include "high_recovery_ops" and "balanced". These
built-in profiles optimize the QoS provided to clients of mclock scheduler.
* The balancer is now on by default in upmap mode. Since upmap mode requires
`require_min_compat_client` luminous, new clusters will only support luminous
and newer clients by default. Existing clusters can enable upmap support by running
`ceph osd set-require-min-compat-client luminous`. It is still possible to turn
the balancer off using the `ceph balancer off` command. In earlier versions,
the balancer was included in the `always_on_modules` list, but needed to be
turned on explicitly using the `ceph balancer on` command.
* Version 2 of the cephx authentication protocol (`CEPHX_V2` feature bit) is
now required by default. It was introduced in 2018, adding replay attack
protection for authorizers and making msgr v1 message signatures stronger
(CVE-2018-1128 and CVE-2018-1129). Support is present in Jewel 10.2.11,
Luminous 12.2.6, Mimic 13.2.1, Nautilus 14.2.0 and later; upstream kernels
4.9.150, 4.14.86, 4.19 and later; various distribution kernels, in particular
CentOS 7.6 and later. To enable older clients, set `cephx_require_version`
and `cephx_service_require_version` config options to 1.
* `blacklist` has been replaced with `blocklist` throughout. The following commands have changed:
- `ceph osd blacklist ...` are now `ceph osd blocklist ...`
- `ceph <tell|daemon> osd.<NNN> dump_blacklist` is now `ceph <tell|daemon> osd.<NNN> dump_blocklist`
* The following config options have changed:
- `mon osd blacklist default expire` is now `mon osd blocklist default expire`
- `mon mds blacklist interval` is now `mon mds blocklist interval`
- `mon mgr blacklist interval` is now ''mon mgr blocklist interval`
- `rbd blacklist on break lock` is now `rbd blocklist on break lock`
- `rbd blacklist expire seconds` is now `rbd blocklist expire seconds`
- `mds session blacklist on timeout` is now `mds session blocklist on timeout`
- `mds session blacklist on evict` is now `mds session blocklist on evict`
* The following librados API calls have changed:
- `rados_blacklist_add` is now `rados_blocklist_add`; the former will issue a deprecation warning and be removed in a future release.
- `rados.blacklist_add` is now `rados.blocklist_add` in the C++ API.
* The JSON output for the following commands now shows `blocklist` instead of `blacklist`:
- `ceph osd dump`
- `ceph <tell|daemon> osd.<N> dump_blocklist`
* Monitors now have config option `mon_allow_pool_size_one`, which is disabled
by default. However, if enabled, user now have to pass the
`--yes-i-really-mean-it` flag to `osd pool set size 1`, if they are really
sure of configuring pool size 1.
* `ceph pg #.# list_unfound` output has been enhanced to provide
might_have_unfound information which indicates which OSDs may
contain the unfound objects.
* OSD: A new configuration option `osd_compact_on_start` has been added which triggers
an OSD compaction on start. Setting this option to `true` and restarting an OSD
will result in an offline compaction of the OSD prior to booting.
* OSD: the option named `bdev_nvme_retry_count` has been removed. Because
in SPDK v20.07, there is no easy access to bdev_nvme options, and this
option is hardly used, so it was removed.
* Alpine build related script, documentation and test have been removed since
the most updated APKBUILD script of Ceph is already included by Alpine Linux's
aports repository.
Getting Ceph
------------
* Git at git://github.com/ceph/ceph.git
* Tarball at http://download.ceph.com/tarballs/ceph-16.2.0.tar.gz
* For packages, see http://docs.ceph.com/docs/master/install/get-packages/
* Release git sha1: 0c2054e95bcd9b30fdd908a79ac1d8bbc3394442
