Hello,
thanks for a very interesting new Ceph Release.
Are there any plans to build for Debian bullseye as well? It's in
"Hard Freeze" since 2021-03-12 and at the moment it comes with a
Nautilus release that will be EOL when Debian bullseye will be
official stable. That will be a pain for Debian users and if it's
still possible we should try to avoid that. Is there something we
could help to make it happen?
--
Martin Verges
Managing director
Mobile: +49 174 9335695
E-Mail: martin.verges(a)croit.io
Chat:
https://t.me/MartinVerges
croit GmbH, Freseniusstr. 31h, 81247 Munich
CEO: Martin Verges - VAT-ID: DE310638492
Com. register: Amtsgericht Munich HRB 231263
Web:
https://croit.io
YouTube:
https://goo.gl/PGE1Bx
On Thu, 1 Apr 2021 at 16:27, David Galloway <dgallowa(a)redhat.com> wrote:
>
> We're glad to announce the first release of the Pacific v16.2.0 stable
> series. There have been a lot of changes across components from the
> previous Ceph releases, and we advise everyone to go through the release
> and upgrade notes carefully.
>
> Major Changes from Octopus
> --------------------------
>
> General
> ~~~~~~~
>
> * Cephadm can automatically upgrade an Octopus cluster to Pacific with a single
> command to start the process.
>
> * Cephadm has improved significantly over the past year, with improved
> support for RGW (standalone and multisite), and new support for NFS
> and iSCSI. Most of these changes have already been backported to
> recent Octopus point releases, but with the Pacific release we will
> switch to backporting bug fixes only.
>
> * Packages are built for the following distributions:
>
> - CentOS 8
> - Ubuntu 20.04 (Focal)
> - Ubuntu 18.04 (Bionic)
> - Debian Buster
> - Container image (based on CentOS 8)
>
> With the exception of Debian Buster, packages and containers are
> built for both x86_64 and aarch64 (arm64) architectures.
>
> Note that cephadm clusters may work on many other distributions,
> provided Python 3 and a recent version of Docker or Podman is
> available to manage containers. For more information, see
> `cephadm-host-requirements`.
>
>
> Dashboard
> ~~~~~~~~~
>
> The `mgr-dashboard` brings improvements in the following management areas:
>
> * Orchestrator/Cephadm:
>
> - Host management: maintenance mode, labels.
> - Services: display placement specification.
> - OSD: disk replacement, display status of ongoing deletion, and improved
> health/SMART diagnostics reporting.
>
> * Official `mgr ceph api`:
>
> - OpenAPI v3 compliant.
> - Stability commitment starting from Pacific release.
> - Versioned via HTTP `Accept` header (starting with v1.0).
> - Thoroughly tested (>90% coverage and per Pull Request validation).
> - Fully documented.
>
> * RGW:
>
> - Multi-site synchronization monitoring.
> - Management of multiple RGW daemons and their resources (buckets and users).
> - Bucket and user quota usage visualization.
> - Improved configuration of S3 tenanted users.
>
> * Security (multiple enhancements and fixes resulting from a pen testing conducted by
IBM):
>
> - Account lock-out after a configurable number of failed log-in attempts.
> - Improved cookie policies to mitigate XSS/CSRF attacks.
> - Reviewed and improved security in HTTP headers.
> - Sensitive information reviewed and removed from logs and error messages.
> - TLS 1.0 and 1.1 support disabled.
> - Debug mode when enabled triggers HEALTH_WARN.
>
> * Pools:
>
> - Improved visualization of replication and erasure coding modes.
> - CLAY erasure code plugin supported.
>
> * Alerts and notifications:
>
> - Alert triggered on MTU mismatches in the cluster network.
> - Favicon changes according cluster status.
>
> * Other:
>
> - Landing page: improved charts and visualization.
> - Telemetry configuration wizard.
> - OSDs: management of individual OSD flags.
> - RBD: per-RBD image Grafana dashboards.
> - CephFS: Dirs and Caps displayed.
> - NFS: v4 support only (v3 backward compatibility planned).
> - Front-end: Angular 10 update.
>
>
> RADOS
> ~~~~~
>
> * Pacific introduces RocksDB sharding, which reduces disk space requirements.
>
> * Ceph now provides QoS between client I/O and background operations via the
> mclock scheduler.
>
> * The balancer is now on by default in upmap mode to improve distribution of
> PGs across OSDs.
>
> * The output of `ceph -s` has been improved to show recovery progress in
> one progress bar. More detailed progress bars are visible via the
> `ceph progress` command.
>
>
> RBD block storage
> ~~~~~~~~~~~~~~~~~
>
> * Image live-migration feature has been extended to support external data
> sources. Images can now be instantly imported from local files, remote
> files served over HTTP(S) or remote S3 buckets in `raw` (`rbd export v1`)
> or basic `qcow` and `qcow2` formats. Support for `rbd export v2`
> format, advanced QCOW features and `rbd export-diff` snapshot differentials
> is expected in future releases.
>
> * Initial support for client-side encryption has been added. This is based
> on LUKS and in future releases will allow using per-image encryption keys
> while maintaining snapshot and clone functionality -- so that parent image
> and potentially multiple clone images can be encrypted with different keys.
>
> * A new persistent write-back cache is available. The cache operates in
> a log-structured manner, providing full point-in-time consistency for the
> backing image. It should be particularly suitable for PMEM devices.
>
> * A Windows client is now available in the form of `librbd.dll` and
> `rbd-wnbd` (Windows Network Block Device) daemon. It allows mapping,
> unmapping and manipulating images similar to `rbd-nbd`.
>
> * librbd API now offers quiesce/unquiesce hooks, allowing for coordinated
> snapshot creation.
>
>
> RGW object storage
> ~~~~~~~~~~~~~~~~~~
>
> * Initial support for S3 Select. See `s3-select-feature-table` for supported
queries.
>
> * Bucket notification topics can be configured as `persistent`, where events
> are recorded in rados for reliable delivery.
>
> * Bucket notifications can be delivered to SSL-enabled AMQP endpoints.
>
> * Lua scripts can be run during requests and access their metadata.
>
> * SSE-KMS now supports KMIP as a key management service.
>
> * Multisite data logs can now be deployed on `cls_fifo` to avoid large omap
> cluster warnings and make their trimming cheaper. See `rgw_data_log_backing`.
>
>
> CephFS distributed file system
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> * The CephFS MDS modifies on-RADOS metadata such that the new format is no
> longer backwards compatible. It is not possible to downgrade a file system from
> Pacific (or later) to an older release.
>
> * Multiple file systems in a single Ceph cluster is now stable. New Ceph
> clusters enable support for multiple file systems by default. Existing clusters
> must still set the "enable_multiple" flag on the FS. See also
> `cephfs-multifs`.
>
> * A new `mds_autoscaler` `ceph-mgr` plugin is available for automatically
> deploying MDS daemons in response to changes to the `max_mds` configuration.
> Expect further enhancements in the future to simplify and automate MDS scaling.
>
> * `cephfs-top` is a new utility for looking at performance metrics from CephFS
> clients. It is development preview quality and will have bugs. For more
> information, see `cephfs-top`.
>
> * A new `snap_schedule` `ceph-mgr` plugin provides a command toolset for
> scheduling snapshots on a CephFS file system. For more information, see
> `snap-schedule`.
>
> * First class NFS gateway support in Ceph is here! It's now possible to create
> scale-out ("active-active") NFS gateway clusters that export CephFS
using
> a few commands. The gateways are deployed via cephadm (or Rook, in the future).
> For more information, see `cephfs-nfs`.
>
> * Multiple active MDS file system scrub is now stable. It is no longer necessary
> to set `max_mds` to 1 and wait for non-zero ranks to stop. Scrub commands
> can only be sent to rank 0: `ceph tell mds.<fs_name>:0 scrub start /path
...`.
> For more information, see `mds-scrub`.
>
> * Ephemeral pinning -- policy based subtree pinning -- is considered stable.
> `mds_export_ephemeral_random` and `mds_export_ephemeral_distributed` now
> default to true. For more information, see `cephfs-ephemeral-pinning`.
>
> * A new `cephfs-mirror` daemon is available to mirror CephFS file systems to
> a remote Ceph cluster.
>
> * A Windows client is now available for connecting to CephFS. This is offered
> through a new `ceph-dokan` utility which operates via the Dokan userspace
> API, similar to FUSE. For more information, see `ceph-dokan`.
>
>
> Upgrading from Octopus or Nautilus
> ----------------------------------
>
> Before starting, make sure your cluster is stable and healthy (no down or
> recovering OSDs). (This is optional, but recommended.)
>
> Upgrading cephadm clusters
> ~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> If your cluster is deployed with cephadm (first introduced in Octopus), then
> the upgrade process is entirely automated. To initiate the upgrade,
>
> ceph orch upgrade start --ceph-version 16.2.0
>
> The same process is used to upgrade to future minor releases.
>
> Upgrade progress can be monitored with `ceph -s` (which provides a simple
> progress bar) or more verbosely with
>
> ceph -W cephadm
>
> The upgrade can be paused or resumed with
>
> ceph orch upgrade pause # to pause
> ceph orch upgrade resume # to resume
>
> or canceled with
>
> ceph orch upgrade stop
>
> Note that canceling the upgrade simply stops the process; there is no ability to
> downgrade back to Octopus.
>
>
> Upgrading non-cephadm clusters
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> If you cluster is running Octopus (15.2.x), you might choose
> to first convert it to use cephadm so that the upgrade to Pacific
> is automated (see above). For more information, see
> `cephadm-adoption`.
>
> #. Set the `noout` flag for the duration of the upgrade. (Optional,
> but recommended.):
>
> # ceph osd set noout
>
> #. Upgrade monitors by installing the new packages and restarting the
> monitor daemons. For example, on each monitor host,:
>
> # systemctl restart ceph-mon.target
>
> Once all monitors are up, verify that the monitor upgrade is
> complete by looking for the `octopus` string in the mon
> map. The command:
>
> # ceph mon dump | grep min_mon_release
>
> should report:
>
> min_mon_release 16 (pacific)
>
> If it doesn't, that implies that one or more monitors hasn't been
> upgraded and restarted and/or the quorum does not include all monitors.
>
> #. Upgrade `ceph-mgr` daemons by installing the new packages and
> restarting all manager daemons. For example, on each manager host,:
>
> # systemctl restart ceph-mgr.target
>
> Verify the `ceph-mgr` daemons are running by checking `ceph
> -s`
>
> # ceph -s
>
> ...
> services:
> mon: 3 daemons, quorum foo,bar,baz
> mgr: foo(active), standbys: bar, baz
> ...
>
> #. Upgrade all OSDs by installing the new packages and restarting the
> ceph-osd daemons on all OSD hosts:
>
> # systemctl restart ceph-osd.target
>
> Note that if you are upgrading from Nautilus, the first time each
> OSD starts, it will do a format conversion to improve the
> accounting for "omap" data. This may take a few minutes to as much
> as a few hours (for an HDD with lots of omap data). You can
> disable this automatic conversion with:
>
> # ceph config set osd bluestore_fsck_quick_fix_on_mount false
>
> You can monitor the progress of the OSD upgrades with the
> `ceph versions` or `ceph osd versions` commands:
>
> # ceph osd versions
> {
> "ceph version 14.2.5 (...) nautilus (stable)": 12,
> "ceph version 16.2.0 (...) pacific (stable)": 22,
> }
>
> #. Upgrade all CephFS MDS daemons. For each CephFS file system,
>
> #. Disable standby_replay:
>
> # ceph fs set <fs_name> allow_standby_replay false
>
> #. Reduce the number of ranks to 1. (Make note of the original
> number of MDS daemons first if you plan to restore it later.):
>
> # ceph status
> # ceph fs set <fs_name> max_mds 1
>
> #. Wait for the cluster to deactivate any non-zero ranks by
> periodically checking the status:
>
> # ceph status
>
> #. Take all standby MDS daemons offline on the appropriate hosts with:
>
> # systemctl stop ceph-mds@<daemon_name>
>
> #. Confirm that only one MDS is online and is rank 0 for your FS:
>
> # ceph status
>
> #. Upgrade the last remaining MDS daemon by installing the new
> packages and restarting the daemon:
>
> # systemctl restart ceph-mds.target
>
> #. Restart all standby MDS daemons that were taken offline:
>
> # systemctl start ceph-mds.target
>
> #. Restore the original value of `max_mds` for the volume:
>
> # ceph fs set <fs_name> max_mds <original_max_mds>
>
> #. Upgrade all radosgw daemons by upgrading packages and restarting
> daemons on all hosts:
>
> # systemctl restart ceph-radosgw.target
>
> #. Complete the upgrade by disallowing pre-Pacific OSDs and enabling
> all new Pacific-only functionality:
>
> # ceph osd require-osd-release pacific
>
> #. If you set `noout` at the beginning, be sure to clear it with:
>
> # ceph osd unset noout
>
> #. Consider transitioning your cluster to use the cephadm deployment
> and orchestration framework to simplify cluster management and
> future upgrades. For more information on converting an existing
> cluster to cephadm, see `cephadm-adoption`.
>
>
> Post-upgrade
> ~~~~~~~~~~~~
>
> #. Verify the cluster is healthy with `ceph health`.
>
> If your CRUSH tunables are older than Hammer, Ceph will now issue a
> health warning. If you see a health alert to that effect, you can
> revert this change with:
>
> ceph config set mon mon_crush_min_required_version firefly
>
> If Ceph does not complain, however, then we recommend you also
> switch any existing CRUSH buckets to straw2, which was added back
> in the Hammer release. If you have any 'straw' buckets, this will
> result in a modest amount of data movement, but generally nothing
> too severe.:
>
> ceph osd getcrushmap -o backup-crushmap
> ceph osd crush set-all-straw-buckets-to-straw2
>
> If there are problems, you can easily revert with:
>
> ceph osd setcrushmap -i backup-crushmap
>
> Moving to 'straw2' buckets will unlock a few recent features, like
> the `crush-compat` `balancer <balancer>` mode added back in Luminous.
>
> #. If you did not already do so when upgrading from Mimic, we
> recommened you enable the new `v2 network protocol <msgr2>`,
> issue the following command:
>
> ceph mon enable-msgr2
>
> This will instruct all monitors that bind to the old default port
> 6789 for the legacy v1 protocol to also bind to the new 3300 v2
> protocol port. To see if all monitors have been updated,:
>
> ceph mon dump
>
> and verify that each monitor has both a `v2:` and `v1:` address
> listed.
>
> #. Consider enabling the `telemetry module <telemetry>` to send
> anonymized usage statistics and crash information to the Ceph
> upstream developers. To see what would be reported (without actually
> sending any information to anyone),:
>
> ceph mgr module enable telemetry
> ceph telemetry show
>
> If you are comfortable with the data that is reported, you can opt-in to
> automatically report the high-level cluster metadata with:
>
> ceph telemetry on
>
> The public dashboard that aggregates Ceph telemetry can be found at
>
https://telemetry-public.ceph.com.
>
> For more information about the telemetry module, see `the
> telemetry documentation.
>
>
> Upgrade from pre-Nautilus releases (like Mimic or Luminous)
> -----------------------------------------------------------
>
> You must first upgrade to Nautilus (14.2.z) or Octopus (15.2.z) before
> upgrading to Pacific.
>
>
> Notable Changes
> ---------------
>
> * A new library is available, libcephsqlite. It provides a SQLite Virtual File
> System (VFS) on top of RADOS. The database and journals are striped over
> RADOS across multiple objects for virtually unlimited scaling and throughput
> only limited by the SQLite client. Applications using SQLite may change to
> the Ceph VFS with minimal changes, usually just by specifying the alternate
> VFS. We expect the library to be most impactful and useful for applications
> that were storing state in RADOS omap, especially without striping which
> limits scalability.
>
> * New `bluestore_rocksdb_options_annex` config parameter. Complements
> `bluestore_rocksdb_options` and allows setting rocksdb options without
> repeating the existing defaults.
>
> * $pid expansion in config paths like `admin_socket` will now properly expand
> to the daemon pid for commands like `ceph-mds` or `ceph-osd`. Previously
> only `ceph-fuse`/`rbd-nbd` expanded `$pid` with the actual daemon pid.
>
> * The allowable options for some `radosgw-admin` commands have been changed.
>
> * `mdlog-list`, `datalog-list`, `sync-error-list` no longer accepts
> start and end dates, but does accept a single optional start marker.
> * `mdlog-trim`, `datalog-trim`, `sync-error-trim` only accept a
> single marker giving the end of the trimmed range.
> * Similarly the date ranges and marker ranges have been removed on
> the RESTful DATALog and MDLog list and trim operations.
>
> * ceph-volume: The `lvm batch` subcommand received a major rewrite. This
> closed a number of bugs and improves usability in terms of size specification
> and calculation, as well as idempotency behaviour and disk replacement
> process.
> Please refer to
https://docs.ceph.com/en/latest/ceph-volume/lvm/batch/ for
> more detailed information.
>
> * Configuration variables for permitted scrub times have changed. The legal
> values for `osd_scrub_begin_hour` and `osd_scrub_end_hour` are 0 - 23.
> The use of 24 is now illegal. Specifying `0` for both values causes every
> hour to be allowed. The legal values for `osd_scrub_begin_week_day` and
> `osd_scrub_end_week_day` are 0 - 6. The use of 7 is now illegal.
> Specifying `0` for both values causes every day of the week to be allowed.
>
> * volume/nfs: Recently "ganesha-" prefix from cluster id and nfs-ganesha
common
> config object was removed, to ensure consistent namespace across different
> orchestrator backends. Please delete any existing nfs-ganesha clusters prior
> to upgrading and redeploy new clusters after upgrading to Pacific.
>
> * A new health check, DAEMON_OLD_VERSION, will warn if different versions of Ceph are
running
> on daemons. It will generate a health error if multiple versions are detected.
> This condition must exist for over mon_warn_older_version_delay (set to 1 week by
default) in order for the
> health condition to be triggered. This allows most upgrades to proceed
> without falsely seeing the warning. If upgrade is paused for an extended
> time period, health mute can be used like this
> "ceph health mute DAEMON_OLD_VERSION --sticky". In this case after
> upgrade has finished use "ceph health unmute DAEMON_OLD_VERSION".
>
> * MGR: progress module can now be turned on/off, using the commands:
> `ceph progress on` and `ceph progress off`.
>
> * An AWS-compliant API: "GetTopicAttributes" was added to replace the
existing "GetTopic" API. The new API
> should be used to fetch information about topics used for bucket notifications.
>
> * librbd: The shared, read-only parent cache's config option
`immutable_object_cache_watermark` now has been updated
> to property reflect the upper cache utilization before space is reclaimed. The
default `immutable_object_cache_watermark`
> now is `0.9`. If the capacity reaches 90% the daemon will delete cold cache.
>
> * OSD: the option `osd_fast_shutdown_notify_mon` has been introduced to allow
> the OSD to notify the monitor it is shutting down even if `osd_fast_shutdown`
> is enabled. This helps with the monitor logs on larger clusters, that may get
> many 'osd.X reported immediately failed by osd.Y' messages, and confuse
tools.
>
> * The mclock scheduler has been refined. A set of built-in profiles are now available
that
> provide QoS between the internal and external clients of Ceph. To enable the
mclock
> scheduler, set the config option "osd_op_queue" to
"mclock_scheduler". The
> "high_client_ops" profile is enabled by default, and allocates more OSD
bandwidth to
> external client operations than to internal client operations (such as background
recovery
> and scrubs). Other built-in profiles include "high_recovery_ops" and
"balanced". These
> built-in profiles optimize the QoS provided to clients of mclock scheduler.
>
> * The balancer is now on by default in upmap mode. Since upmap mode requires
> `require_min_compat_client` luminous, new clusters will only support luminous
> and newer clients by default. Existing clusters can enable upmap support by
running
> `ceph osd set-require-min-compat-client luminous`. It is still possible to turn
> the balancer off using the `ceph balancer off` command. In earlier versions,
> the balancer was included in the `always_on_modules` list, but needed to be
> turned on explicitly using the `ceph balancer on` command.
>
> * Version 2 of the cephx authentication protocol (`CEPHX_V2` feature bit) is
> now required by default. It was introduced in 2018, adding replay attack
> protection for authorizers and making msgr v1 message signatures stronger
> (CVE-2018-1128 and CVE-2018-1129). Support is present in Jewel 10.2.11,
> Luminous 12.2.6, Mimic 13.2.1, Nautilus 14.2.0 and later; upstream kernels
> 4.9.150, 4.14.86, 4.19 and later; various distribution kernels, in particular
> CentOS 7.6 and later. To enable older clients, set `cephx_require_version`
> and `cephx_service_require_version` config options to 1.
>
> * `blacklist` has been replaced with `blocklist` throughout. The following commands
have changed:
>
> - `ceph osd blacklist ...` are now `ceph osd blocklist ...`
> - `ceph <tell|daemon> osd.<NNN> dump_blacklist` is now `ceph
<tell|daemon> osd.<NNN> dump_blocklist`
>
> * The following config options have changed:
>
> - `mon osd blacklist default expire` is now `mon osd blocklist default expire`
> - `mon mds blacklist interval` is now `mon mds blocklist interval`
> - `mon mgr blacklist interval` is now ''mon mgr blocklist interval`
> - `rbd blacklist on break lock` is now `rbd blocklist on break lock`
> - `rbd blacklist expire seconds` is now `rbd blocklist expire seconds`
> - `mds session blacklist on timeout` is now `mds session blocklist on timeout`
> - `mds session blacklist on evict` is now `mds session blocklist on evict`
>
> * The following librados API calls have changed:
>
> - `rados_blacklist_add` is now `rados_blocklist_add`; the former will issue a
deprecation warning and be removed in a future release.
> - `rados.blacklist_add` is now `rados.blocklist_add` in the C++ API.
>
> * The JSON output for the following commands now shows `blocklist` instead of
`blacklist`:
>
> - `ceph osd dump`
> - `ceph <tell|daemon> osd.<N> dump_blocklist`
>
> * Monitors now have config option `mon_allow_pool_size_one`, which is disabled
> by default. However, if enabled, user now have to pass the
> `--yes-i-really-mean-it` flag to `osd pool set size 1`, if they are really
> sure of configuring pool size 1.
>
> * `ceph pg #.# list_unfound` output has been enhanced to provide
> might_have_unfound information which indicates which OSDs may
> contain the unfound objects.
>
> * OSD: A new configuration option `osd_compact_on_start` has been added which
triggers
> an OSD compaction on start. Setting this option to `true` and restarting an OSD
> will result in an offline compaction of the OSD prior to booting.
>
> * OSD: the option named `bdev_nvme_retry_count` has been removed. Because
> in SPDK v20.07, there is no easy access to bdev_nvme options, and this
> option is hardly used, so it was removed.
>
> * Alpine build related script, documentation and test have been removed since
> the most updated APKBUILD script of Ceph is already included by Alpine Linux's
> aports repository.
>
>
>
> Getting Ceph
> ------------
> * Git at
git://github.com/ceph/ceph.git
> * Tarball at
http://download.ceph.com/tarballs/ceph-16.2.0.tar.gz
> * For packages, see
http://docs.ceph.com/docs/master/install/get-packages/
> * Release git sha1: 0c2054e95bcd9b30fdd908a79ac1d8bbc3394442
> _______________________________________________
> Ceph-maintainers mailing list -- ceph-maintainers(a)ceph.io
> To unsubscribe send an email to ceph-maintainers-leave(a)ceph.io