v20.1.0 Tentacle RC0 released
This is the first release candidate for Tentacle. Ubuntu 22.04, 24.04 and CentOS 9 were built for this RC0 Feature highlights: * RADOS: New features include long expected performance optimizations (FastEC) for EC pools, including partial reads and partial writes. Users can also expect to see BlueStore improvements such as compression and a new, faster WAL (write-ahead-log). Additional improvements include switching all components to the faster OMAP iteration interface, bypassable ceph_assert()s, fixed mclock bugs and configuration defaults, and testing improvements for dencoding verification. * MGR: Highlights include the ability to force-disable always-on modules and the removal of the restful and zabbix modules (both deprecated since 2020). Note that the dashboard module's richer and better-maintained RESTful API can be used as an alternative to the restful module, and the prometheus module can be used as an alternative monitoring solution for zabbix. * RGW: Multiple fixes: Lua scripts will not run against health checks, properly quoted ETag values returned by S3 CopyPart, PostObject and CompleteMultipartUpload responses. * RGW: IAM policy evaluation now supports conditions ArnEquals and ArnLike, along with their Not and IfExists variants. * RBD: New live migration features: RBD images can now be instantly imported from another Ceph cluster (native format) or from a wide variety of external sources/formats with the help of the new NBD stream and an appropriately capable NBD server such as `qemu-nbd`. Also added support for RBD namespace remapping while mirroring between Ceph clusters, new `rbd group info` and `rbd group snap info` commands and enhanced `rbd group snap ls` command. `rbd device map` command now defaults to msgr2. * CephFS: Directories may now be configured with case-insensitive or normalized directory entry names. This is an inheritable configuration making it apply to an entire directory tree. For more information, see https://docs.ceph.com/en/latest/cephfs/charmap/ * CephFS: Modifying the FS setting variable "max_mds" when a cluster is unhealthy now requires users to pass the confirmation flag (--yes-i-really-mean-it). This has been added as a precaution to tell the users that modifying "max_mds" may not help with troubleshooting or recovery effort. Instead, it might further destabilize the cluster. * CephFS: EOPNOTSUPP (Operation not supported ) is now returned by the CephFS fuse client for `fallocate` for the default case (i.e. mode == 0) since CephFS does not support disk space reservation. The only flags supported are `FALLOC_FL_KEEP_SIZE` and `FALLOC_FL_PUNCH_HOLE`. * Dashboard: Added support for NVMe/TCP (gateway groups, multiple namespaces), multi-cluster management, oAuth2 integration, and enhanced RGW/SMB features including multi-site automation, tiering, policies, lifecycles, notifications, and granular replication. * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-20.1.0.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: 010a3ad647c9962d47812a66ad6feda26ab28aa4
Hi, thanks for the info, I'm excited to test the new release canidate! And I already got my first issue, trying to upgrade a tiny single-node cluster from 19.2.3 to 20.1.0 fails (health detail at the end). The host is a VM running openSUSE Leap 15.6. The first MGR seems to have been upgraded successfully: soc9-ceph:~ # ceph versions -f json | jq -r '.mgr' { "ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable)": 1, "ceph version 20.1.0 (010a3ad647c9962d47812a66ad6feda26ab28aa4) tentacle (rc - RelWithDebInfo)": 1 } Is this already a known issue? Thanks! Eugen # ceph orch upgrade status { "in_progress": true, "target_image": "quay.io/ceph/ceph@sha256:06b298a25e7cee11677f06a54ad90bb69f9b295e0d5482663f26b439d14d4045", "services_complete": [], "which": "Upgrading all daemon types on all hosts", "progress": "1/10 daemons upgraded", "message": "Error: UPGRADE_FAILED_PULL: Upgrade: failed to pull target image", "is_paused": true } soc9-ceph:~ # ceph health detail HEALTH_WARN failed to probe daemons or devices; Upgrade: failed to pull target image [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices host soc9-ceph `cephadm ls` failed: cephadm exited with an error code: 1, stderr: Traceback (most recent call last): File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/var/lib/ceph/95f0ae1e-7d97-11f0-918f-fa163e2ad8c5/cephadm.07ba6d1a818cf6ded98e09fc882a9f4f1772aeb82d4664369096243131efe94f/__main__.py", line 190, in <module> File "/var/lib/ceph/95f0ae1e-7d97-11f0-918f-fa163e2ad8c5/cephadm.07ba6d1a818cf6ded98e09fc882a9f4f1772aeb82d4664369096243131efe94f/cephadmlib/listing.py", line 53, in <module> ImportError: cannot import name 'TypedDict' host soc9-ceph `cephadm gather-facts` failed: cephadm exited with an error code: 1, stderr: Traceback (most recent call last): File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/var/lib/ceph/95f0ae1e-7d97-11f0-918f-fa163e2ad8c5/cephadm.07ba6d1a818cf6ded98e09fc882a9f4f1772aeb82d4664369096243131efe94f/__main__.py", line 190, in <module> File "/var/lib/ceph/95f0ae1e-7d97-11f0-918f-fa163e2ad8c5/cephadm.07ba6d1a818cf6ded98e09fc882a9f4f1772aeb82d4664369096243131efe94f/cephadmlib/listing.py", line 53, in <module> ImportError: cannot import name 'TypedDict' host soc9-ceph `cephadm list-networks` failed: cephadm exited with an error code: 1, stderr: Traceback (most recent call last): File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/var/lib/ceph/95f0ae1e-7d97-11f0-918f-fa163e2ad8c5/cephadm.07ba6d1a818cf6ded98e09fc882a9f4f1772aeb82d4664369096243131efe94f/__main__.py", line 190, in <module> File "/var/lib/ceph/95f0ae1e-7d97-11f0-918f-fa163e2ad8c5/cephadm.07ba6d1a818cf6ded98e09fc882a9f4f1772aeb82d4664369096243131efe94f/cephadmlib/listing.py", line 53, in <module> ImportError: cannot import name 'TypedDict' [WRN] UPGRADE_FAILED_PULL: Upgrade: failed to pull target image failed to pull quay.io/ceph/ceph@sha256:06b298a25e7cee11677f06a54ad90bb69f9b295e0d5482663f26b439d14d4045 on host soc9-ceph Zitat von Yuri Weinstein <yweinste@redhat.com>:
This is the first release candidate for Tentacle. Ubuntu 22.04, 24.04 and CentOS 9 were built for this RC0
Feature highlights:
* RADOS: New features include long expected performance optimizations (FastEC) for EC pools, including partial reads and partial writes. Users can also expect to see BlueStore improvements such as compression and a new, faster WAL (write-ahead-log). Additional improvements include switching all components to the faster OMAP iteration interface, bypassable ceph_assert()s, fixed mclock bugs and configuration defaults, and testing improvements for dencoding verification. * MGR: Highlights include the ability to force-disable always-on modules and the removal of the restful and zabbix modules (both deprecated since 2020). Note that the dashboard module's richer and better-maintained RESTful API can be used as an alternative to the restful module, and the prometheus module can be used as an alternative monitoring solution for zabbix. * RGW: Multiple fixes: Lua scripts will not run against health checks, properly quoted ETag values returned by S3 CopyPart, PostObject and CompleteMultipartUpload responses. * RGW: IAM policy evaluation now supports conditions ArnEquals and ArnLike, along with their Not and IfExists variants. * RBD: New live migration features: RBD images can now be instantly imported from another Ceph cluster (native format) or from a wide variety of external sources/formats with the help of the new NBD stream and an appropriately capable NBD server such as `qemu-nbd`. Also added support for RBD namespace remapping while mirroring between Ceph clusters, new `rbd group info` and `rbd group snap info` commands and enhanced `rbd group snap ls` command. `rbd device map` command now defaults to msgr2. * CephFS: Directories may now be configured with case-insensitive or normalized directory entry names. This is an inheritable configuration making it apply to an entire directory tree. For more information, see https://docs.ceph.com/en/latest/cephfs/charmap/ * CephFS: Modifying the FS setting variable "max_mds" when a cluster is unhealthy now requires users to pass the confirmation flag (--yes-i-really-mean-it). This has been added as a precaution to tell the users that modifying "max_mds" may not help with troubleshooting or recovery effort. Instead, it might further destabilize the cluster. * CephFS: EOPNOTSUPP (Operation not supported ) is now returned by the CephFS fuse client for `fallocate` for the default case (i.e. mode == 0) since CephFS does not support disk space reservation. The only flags supported are `FALLOC_FL_KEEP_SIZE` and `FALLOC_FL_PUNCH_HOLE`. * Dashboard: Added support for NVMe/TCP (gateway groups, multiple namespaces), multi-cluster management, oAuth2 integration, and enhanced RGW/SMB features including multi-site automation, tiering, policies, lifecycles, notifications, and granular replication.
* Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-20.1.0.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: 010a3ad647c9962d47812a66ad6feda26ab28aa4 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Bootstrapping a new cluster fails as well with the same trace: /usr/bin/ceph: stderr ImportError: cannot import name 'TypedDict' ... Adding key to root@localhost authorized_keys... Adding host soc9-ceph... Non-zero exit code 22 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v20.1.0 -e NODE_NAME=soc9-ceph -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/dba035cc-8db1-11f0-9bf6-fa163e2ad8c5:/var/log/ceph:z -v /tmp/ceph-tmp9fytdo0h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp5n_50w5k:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v20.1.0 orch host add soc9-ceph 192.168.124.186 /usr/bin/ceph: stderr Error EINVAL: check-host failed: /usr/bin/ceph: stderr Traceback (most recent call last): /usr/bin/ceph: stderr File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main /usr/bin/ceph: stderr "__main__", mod_spec) /usr/bin/ceph: stderr File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code /usr/bin/ceph: stderr exec(code, run_globals) /usr/bin/ceph: stderr File "/var/lib/ceph/dba035cc-8db1-11f0-9bf6-fa163e2ad8c5/cephadm.07ba6d1a818cf6ded98e09fc882a9f4f1772aeb82d4664369096243131efe94f/__main__.py", line 190, in <module> /usr/bin/ceph: stderr File "/var/lib/ceph/dba035cc-8db1-11f0-9bf6-fa163e2ad8c5/cephadm.07ba6d1a818cf6ded98e09fc882a9f4f1772aeb82d4664369096243131efe94f/cephadmlib/listing.py", line 53, in <module> /usr/bin/ceph: stderr ImportError: cannot import name 'TypedDict' ERROR: Failed to add host <soc9-ceph>: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --authfile=/etc/ceph/podman-auth.json --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.io/ceph/ceph:v20.1.0 -e NODE_NAME=soc9-ceph -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/dba035cc-8db1-11f0-9bf6-fa163e2ad8c5:/var/log/ceph:z -v /tmp/ceph-tmp9fytdo0h:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmp5n_50w5k:/etc/ceph/ceph.conf:z quay.io/ceph/ceph:v20.1.0 orch host add soc9-ceph 192.168.124.186 Zitat von Eugen Block <eblock@nde.ag>:
Hi,
thanks for the info, I'm excited to test the new release canidate!
And I already got my first issue, trying to upgrade a tiny single-node cluster from 19.2.3 to 20.1.0 fails (health detail at the end). The host is a VM running openSUSE Leap 15.6. The first MGR seems to have been upgraded successfully:
soc9-ceph:~ # ceph versions -f json | jq -r '.mgr' { "ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable)": 1, "ceph version 20.1.0 (010a3ad647c9962d47812a66ad6feda26ab28aa4) tentacle (rc - RelWithDebInfo)": 1 }
Is this already a known issue?
Thanks! Eugen
# ceph orch upgrade status { "in_progress": true, "target_image": "quay.io/ceph/ceph@sha256:06b298a25e7cee11677f06a54ad90bb69f9b295e0d5482663f26b439d14d4045", "services_complete": [], "which": "Upgrading all daemon types on all hosts", "progress": "1/10 daemons upgraded", "message": "Error: UPGRADE_FAILED_PULL: Upgrade: failed to pull target image", "is_paused": true }
soc9-ceph:~ # ceph health detail HEALTH_WARN failed to probe daemons or devices; Upgrade: failed to pull target image [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices host soc9-ceph `cephadm ls` failed: cephadm exited with an error code: 1, stderr: Traceback (most recent call last): File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/var/lib/ceph/95f0ae1e-7d97-11f0-918f-fa163e2ad8c5/cephadm.07ba6d1a818cf6ded98e09fc882a9f4f1772aeb82d4664369096243131efe94f/__main__.py", line 190, in <module> File "/var/lib/ceph/95f0ae1e-7d97-11f0-918f-fa163e2ad8c5/cephadm.07ba6d1a818cf6ded98e09fc882a9f4f1772aeb82d4664369096243131efe94f/cephadmlib/listing.py", line 53, in <module> ImportError: cannot import name 'TypedDict' host soc9-ceph `cephadm gather-facts` failed: cephadm exited with an error code: 1, stderr: Traceback (most recent call last): File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/var/lib/ceph/95f0ae1e-7d97-11f0-918f-fa163e2ad8c5/cephadm.07ba6d1a818cf6ded98e09fc882a9f4f1772aeb82d4664369096243131efe94f/__main__.py", line 190, in <module> File "/var/lib/ceph/95f0ae1e-7d97-11f0-918f-fa163e2ad8c5/cephadm.07ba6d1a818cf6ded98e09fc882a9f4f1772aeb82d4664369096243131efe94f/cephadmlib/listing.py", line 53, in <module> ImportError: cannot import name 'TypedDict' host soc9-ceph `cephadm list-networks` failed: cephadm exited with an error code: 1, stderr: Traceback (most recent call last): File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/var/lib/ceph/95f0ae1e-7d97-11f0-918f-fa163e2ad8c5/cephadm.07ba6d1a818cf6ded98e09fc882a9f4f1772aeb82d4664369096243131efe94f/__main__.py", line 190, in <module> File "/var/lib/ceph/95f0ae1e-7d97-11f0-918f-fa163e2ad8c5/cephadm.07ba6d1a818cf6ded98e09fc882a9f4f1772aeb82d4664369096243131efe94f/cephadmlib/listing.py", line 53, in <module> ImportError: cannot import name 'TypedDict' [WRN] UPGRADE_FAILED_PULL: Upgrade: failed to pull target image failed to pull quay.io/ceph/ceph@sha256:06b298a25e7cee11677f06a54ad90bb69f9b295e0d5482663f26b439d14d4045 on host soc9-ceph
Zitat von Yuri Weinstein <yweinste@redhat.com>:
This is the first release candidate for Tentacle. Ubuntu 22.04, 24.04 and CentOS 9 were built for this RC0
Feature highlights:
* RADOS: New features include long expected performance optimizations (FastEC) for EC pools, including partial reads and partial writes. Users can also expect to see BlueStore improvements such as compression and a new, faster WAL (write-ahead-log). Additional improvements include switching all components to the faster OMAP iteration interface, bypassable ceph_assert()s, fixed mclock bugs and configuration defaults, and testing improvements for dencoding verification. * MGR: Highlights include the ability to force-disable always-on modules and the removal of the restful and zabbix modules (both deprecated since 2020). Note that the dashboard module's richer and better-maintained RESTful API can be used as an alternative to the restful module, and the prometheus module can be used as an alternative monitoring solution for zabbix. * RGW: Multiple fixes: Lua scripts will not run against health checks, properly quoted ETag values returned by S3 CopyPart, PostObject and CompleteMultipartUpload responses. * RGW: IAM policy evaluation now supports conditions ArnEquals and ArnLike, along with their Not and IfExists variants. * RBD: New live migration features: RBD images can now be instantly imported from another Ceph cluster (native format) or from a wide variety of external sources/formats with the help of the new NBD stream and an appropriately capable NBD server such as `qemu-nbd`. Also added support for RBD namespace remapping while mirroring between Ceph clusters, new `rbd group info` and `rbd group snap info` commands and enhanced `rbd group snap ls` command. `rbd device map` command now defaults to msgr2. * CephFS: Directories may now be configured with case-insensitive or normalized directory entry names. This is an inheritable configuration making it apply to an entire directory tree. For more information, see https://docs.ceph.com/en/latest/cephfs/charmap/ * CephFS: Modifying the FS setting variable "max_mds" when a cluster is unhealthy now requires users to pass the confirmation flag (--yes-i-really-mean-it). This has been added as a precaution to tell the users that modifying "max_mds" may not help with troubleshooting or recovery effort. Instead, it might further destabilize the cluster. * CephFS: EOPNOTSUPP (Operation not supported ) is now returned by the CephFS fuse client for `fallocate` for the default case (i.e. mode == 0) since CephFS does not support disk space reservation. The only flags supported are `FALLOC_FL_KEEP_SIZE` and `FALLOC_FL_PUNCH_HOLE`. * Dashboard: Added support for NVMe/TCP (gateway groups, multiple namespaces), multi-cluster management, oAuth2 integration, and enhanced RGW/SMB features including multi-site automation, tiering, policies, lifecycles, notifications, and granular replication.
* Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-20.1.0.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: 010a3ad647c9962d47812a66ad6feda26ab28aa4 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Tuesday, September 9, 2025 3:16:18 PM Eastern Daylight Time Eugen Block wrote:
Hi,
thanks for the info, I'm excited to test the new release canidate!
And I already got my first issue, trying to upgrade a tiny single-node cluster from 19.2.3 to 20.1.0 fails (health detail at the end). The host is a VM running openSUSE Leap 15.6. The first MGR seems to have been upgraded successfully:
soc9-ceph:~ # ceph versions -f json | jq -r '.mgr' { "ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable)": 1, "ceph version 20.1.0 (010a3ad647c9962d47812a66ad6feda26ab28aa4) tentacle (rc - RelWithDebInfo)": 1 }
Is this already a known issue?
Thanks! Eugen
# ceph orch upgrade status { "in_progress": true, "target_image": "quay.io/ceph/ceph@sha256:06b298a25e7cee11677f06a54ad90bb69f9b295e0d5482663f 26b439d14d4045", "services_complete": [], "which": "Upgrading all daemon types on all hosts", "progress": "1/10 daemons upgraded", "message": "Error: UPGRADE_FAILED_PULL: Upgrade: failed to pull target image", "is_paused": true }
soc9-ceph:~ # ceph health detail HEALTH_WARN failed to probe daemons or devices; Upgrade: failed to pull target image [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices host soc9-ceph `cephadm ls` failed: cephadm exited with an error code: 1, stderr: Traceback (most recent call last): File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/var/lib/ceph/95f0ae1e-7d97-11f0-918f-fa163e2ad8c5/cephadm.07ba6d1a818cf6de d98e09fc882a9f4f1772aeb82d4664369096243131efe94f/__main__.py", line 190, in <module> File "/var/lib/ceph/95f0ae1e-7d97-11f0-918f-fa163e2ad8c5/cephadm.07ba6d1a818cf6de d98e09fc882a9f4f1772aeb82d4664369096243131efe94f/cephadmlib/listing.py", line 53, in <module> ImportError: cannot import name 'TypedDict' host soc9-ceph `cephadm gather-facts` failed: cephadm exited with an error code: 1, stderr: Traceback (most recent call last): File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/var/lib/ceph/95f0ae1e-7d97-11f0-918f-fa163e2ad8c5/cephadm.07ba6d1a818cf6de d98e09fc882a9f4f1772aeb82d4664369096243131efe94f/__main__.py", line 190, in <module> File "/var/lib/ceph/95f0ae1e-7d97-11f0-918f-fa163e2ad8c5/cephadm.07ba6d1a818cf6de d98e09fc882a9f4f1772aeb82d4664369096243131efe94f/cephadmlib/listing.py", line 53, in <module> ImportError: cannot import name 'TypedDict' host soc9-ceph `cephadm list-networks` failed: cephadm exited with an error code: 1, stderr: Traceback (most recent call last): File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/var/lib/ceph/95f0ae1e-7d97-11f0-918f-fa163e2ad8c5/cephadm.07ba6d1a818cf6de d98e09fc882a9f4f1772aeb82d4664369096243131efe94f/__main__.py", line 190, in <module> File "/var/lib/ceph/95f0ae1e-7d97-11f0-918f-fa163e2ad8c5/cephadm.07ba6d1a818cf6de d98e09fc882a9f4f1772aeb82d4664369096243131efe94f/cephadmlib/listing.py", line 53, in <module> ImportError: cannot import name 'TypedDict' [WRN] UPGRADE_FAILED_PULL: Upgrade: failed to pull target image failed to pull quay.io/ceph/ceph@sha256:06b298a25e7cee11677f06a54ad90bb69f9b295e0d5482663f2 6b439d14d4045 on host soc9-ceph
Zitat von Yuri Weinstein <yweinste@redhat.com>:
This is the first release candidate for Tentacle. Ubuntu 22.04, 24.04 and CentOS 9 were built for this RC0
Feature highlights:
* RADOS: New features include long expected performance optimizations (FastEC)
for EC pools, including partial reads and partial writes. Users can also expect to see BlueStore improvements such as compression
and a new,
faster WAL (write-ahead-log). Additional improvements include switching
all components
to the faster OMAP iteration interface, bypassable ceph_assert()s, fixed mclock bugs and configuration defaults, and testing improvements
for
dencoding verification.
* MGR: Highlights include the ability to force-disable always-on modules and
the removal of the restful and zabbix modules (both deprecated since
2020).
Note that the dashboard module's richer and better-maintained RESTful API
can be used
as an alternative to the restful module, and the prometheus module can be
used
as an alternative monitoring solution for zabbix.
* RGW: Multiple fixes: Lua scripts will not run against health checks,
properly quoted ETag values returned by S3 CopyPart, PostObject and CompleteMultipartUpload responses.
* RGW: IAM policy evaluation now supports conditions ArnEquals and ArnLike,
along with their Not and IfExists variants.
* RBD: New live migration features: RBD images can now be instantly
imported from another Ceph cluster (native format) or from a wide variety of external sources/formats with the help of the new NBD stream and an appropriately capable NBD server such as `qemu-nbd`. Also added support for RBD namespace remapping while mirroring between Ceph clusters, new `rbd group info` and `rbd group snap info` commands and enhanced `rbd group snap ls` command. `rbd device map` command now defaults to msgr2.
* CephFS: Directories may now be configured with case-insensitive or
normalized directory entry names. This is an inheritable configuration
making
it apply to an entire directory tree. For more information, see https://docs.ceph.com/en/latest/cephfs/charmap/
* CephFS: Modifying the FS setting variable "max_mds" when a cluster is
unhealthy now requires users to pass the confirmation flag (--yes-i-really-mean-it). This has been added as a precaution to tell the users that modifying "max_mds" may not help with troubleshooting or
recovery
effort. Instead, it might further destabilize the cluster.
* CephFS: EOPNOTSUPP (Operation not supported ) is now returned by the CephFS
fuse client for `fallocate` for the default case (i.e. mode == 0) since CephFS does not support disk space reservation. The only flags supported
are
`FALLOC_FL_KEEP_SIZE` and `FALLOC_FL_PUNCH_HOLE`.
* Dashboard: Added support for NVMe/TCP (gateway groups, multiple namespaces),
multi-cluster management, oAuth2 integration, and enhanced RGW/SMB
features
including multi-site automation, tiering, policies, lifecycles, notifications, and granular replication.
* Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-20.1.0.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: 010a3ad647c9962d47812a66ad6feda26ab28aa4 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
The minimum version of python for cephadm is now 3.9 in tentacle. Does your distro have a python version older than that? We probably need to update the matrix at https://docs.ceph.com/en/tentacle/ start/os-recommendations/ to include tentacle itself, too.
Thanks for the info, in the meantime a colleague pointed me to the python version being too old. On a Ubuntu 24 VM I was able to bootstrap a 20.1.0 cluster successfully (well, at least a mon and a mgr). I agree that an info somewhere in the release notes and in the docs would be helpful. Zitat von John Mulligan <phlogistonjohn@asynchrono.us>:
On Tuesday, September 9, 2025 3:16:18 PM Eastern Daylight Time Eugen Block wrote:
Hi,
thanks for the info, I'm excited to test the new release canidate!
And I already got my first issue, trying to upgrade a tiny single-node cluster from 19.2.3 to 20.1.0 fails (health detail at the end). The host is a VM running openSUSE Leap 15.6. The first MGR seems to have been upgraded successfully:
soc9-ceph:~ # ceph versions -f json | jq -r '.mgr' { "ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable)": 1, "ceph version 20.1.0 (010a3ad647c9962d47812a66ad6feda26ab28aa4) tentacle (rc - RelWithDebInfo)": 1 }
Is this already a known issue?
Thanks! Eugen
# ceph orch upgrade status { "in_progress": true, "target_image": "quay.io/ceph/ceph@sha256:06b298a25e7cee11677f06a54ad90bb69f9b295e0d5482663f 26b439d14d4045", "services_complete": [], "which": "Upgrading all daemon types on all hosts", "progress": "1/10 daemons upgraded", "message": "Error: UPGRADE_FAILED_PULL: Upgrade: failed to pull target image", "is_paused": true }
soc9-ceph:~ # ceph health detail HEALTH_WARN failed to probe daemons or devices; Upgrade: failed to pull target image [WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices host soc9-ceph `cephadm ls` failed: cephadm exited with an error code: 1, stderr: Traceback (most recent call last): File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/var/lib/ceph/95f0ae1e-7d97-11f0-918f-fa163e2ad8c5/cephadm.07ba6d1a818cf6de d98e09fc882a9f4f1772aeb82d4664369096243131efe94f/__main__.py", line 190, in <module> File "/var/lib/ceph/95f0ae1e-7d97-11f0-918f-fa163e2ad8c5/cephadm.07ba6d1a818cf6de d98e09fc882a9f4f1772aeb82d4664369096243131efe94f/cephadmlib/listing.py", line 53, in <module> ImportError: cannot import name 'TypedDict' host soc9-ceph `cephadm gather-facts` failed: cephadm exited with an error code: 1, stderr: Traceback (most recent call last): File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/var/lib/ceph/95f0ae1e-7d97-11f0-918f-fa163e2ad8c5/cephadm.07ba6d1a818cf6de d98e09fc882a9f4f1772aeb82d4664369096243131efe94f/__main__.py", line 190, in <module> File "/var/lib/ceph/95f0ae1e-7d97-11f0-918f-fa163e2ad8c5/cephadm.07ba6d1a818cf6de d98e09fc882a9f4f1772aeb82d4664369096243131efe94f/cephadmlib/listing.py", line 53, in <module> ImportError: cannot import name 'TypedDict' host soc9-ceph `cephadm list-networks` failed: cephadm exited with an error code: 1, stderr: Traceback (most recent call last): File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/var/lib/ceph/95f0ae1e-7d97-11f0-918f-fa163e2ad8c5/cephadm.07ba6d1a818cf6de d98e09fc882a9f4f1772aeb82d4664369096243131efe94f/__main__.py", line 190, in <module> File "/var/lib/ceph/95f0ae1e-7d97-11f0-918f-fa163e2ad8c5/cephadm.07ba6d1a818cf6de d98e09fc882a9f4f1772aeb82d4664369096243131efe94f/cephadmlib/listing.py", line 53, in <module> ImportError: cannot import name 'TypedDict' [WRN] UPGRADE_FAILED_PULL: Upgrade: failed to pull target image failed to pull quay.io/ceph/ceph@sha256:06b298a25e7cee11677f06a54ad90bb69f9b295e0d5482663f2 6b439d14d4045 on host soc9-ceph
Zitat von Yuri Weinstein <yweinste@redhat.com>:
This is the first release candidate for Tentacle. Ubuntu 22.04, 24.04 and CentOS 9 were built for this RC0
Feature highlights:
* RADOS: New features include long expected performance optimizations (FastEC)
for EC pools, including partial reads and partial writes. Users can also expect to see BlueStore improvements such as compression
and a new,
faster WAL (write-ahead-log). Additional improvements include switching
all components
to the faster OMAP iteration interface, bypassable ceph_assert()s, fixed mclock bugs and configuration defaults, and testing improvements
for
dencoding verification.
* MGR: Highlights include the ability to force-disable always-on modules and
the removal of the restful and zabbix modules (both deprecated since
2020).
Note that the dashboard module's richer and better-maintained RESTful API
can be used
as an alternative to the restful module, and the prometheus module can be
used
as an alternative monitoring solution for zabbix.
* RGW: Multiple fixes: Lua scripts will not run against health checks,
properly quoted ETag values returned by S3 CopyPart, PostObject and CompleteMultipartUpload responses.
* RGW: IAM policy evaluation now supports conditions ArnEquals and ArnLike,
along with their Not and IfExists variants.
* RBD: New live migration features: RBD images can now be instantly
imported from another Ceph cluster (native format) or from a wide variety of external sources/formats with the help of the new NBD stream and an appropriately capable NBD server such as `qemu-nbd`. Also added support for RBD namespace remapping while mirroring between Ceph clusters, new `rbd group info` and `rbd group snap info` commands and enhanced `rbd group snap ls` command. `rbd device map` command now defaults to msgr2.
* CephFS: Directories may now be configured with case-insensitive or
normalized directory entry names. This is an inheritable configuration
making
it apply to an entire directory tree. For more information, see https://docs.ceph.com/en/latest/cephfs/charmap/
* CephFS: Modifying the FS setting variable "max_mds" when a cluster is
unhealthy now requires users to pass the confirmation flag (--yes-i-really-mean-it). This has been added as a precaution to tell the users that modifying "max_mds" may not help with troubleshooting or
recovery
effort. Instead, it might further destabilize the cluster.
* CephFS: EOPNOTSUPP (Operation not supported ) is now returned by the CephFS
fuse client for `fallocate` for the default case (i.e. mode == 0) since CephFS does not support disk space reservation. The only flags supported
are
`FALLOC_FL_KEEP_SIZE` and `FALLOC_FL_PUNCH_HOLE`.
* Dashboard: Added support for NVMe/TCP (gateway groups, multiple namespaces),
multi-cluster management, oAuth2 integration, and enhanced RGW/SMB
features
including multi-site automation, tiering, policies, lifecycles, notifications, and granular replication.
* Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-20.1.0.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: 010a3ad647c9962d47812a66ad6feda26ab28aa4 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
The minimum version of python for cephadm is now 3.9 in tentacle. Does your distro have a python version older than that?
We probably need to update the matrix at https://docs.ceph.com/en/tentacle/ start/os-recommendations/ to include tentacle itself, too.
Hi, Am 9/9/25 um 17:34 schrieb Yuri Weinstein:
* Dashboard: Added support for NVMe/TCP (gateway groups, multiple namespaces),
I have just upgraded a test cluster and NVMEoF does not work any more. The previous version running was quay.ceph.io/ceph-ci/ceph:squid-nvmeof and NVMEoF was working there. The gateways seem to work and subsystem and namespaces are still configured but an initiator does not see namespaces. # ceph orch ps --daemon_type nvmeof NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID nvmeof.hosa.b236asbrce101.ccaujq b236asbrce101 *:5500,4420,8009,10008 running (13h) 36s ago 13h 128M - 1.5.7 2c3e7ef350b8 74672e87a945 nvmeof.hosa.b236asbrce102.gnbwsc b236asbrce102 *:5500,4420,8009,10008 running (13h) 36s ago 13h 120M - 1.5.7 2c3e7ef350b8 1929979ab7e1 nvmeof.hosa.b236asbrce103.bthewi b236asbrce103 *:5500,4420,8009,10008 running (13h) 36s ago 13h 122M - 1.5.7 2c3e7ef350b8 14885171d1a5 nvmeof.hosa.b236asbrce104.vcgzvg b236asbrce104 *:5500,4420,8009,10008 running (13h) 36s ago 13h 122M - 1.5.7 2c3e7ef350b8 7b00b0f3356a # ceph nvmeof subsystem list +---------------------------------+------------------+--------------------+---------------+-------+--------------+--------------+--------------+-------------------+ |Nqn |Serial Number |Model Number |Namespace Count|Subtype|Max Namespaces|Has Dhchap Key|Allow Any Host|Created Without Key| +---------------------------------+------------------+--------------------+---------------+-------+--------------+--------------+--------------+-------------------+ |nqn.2025-08.com.example:HOSA.hosa|Ceph30844860597194|Ceph bdev Controller|2 |NVMe |128 |False |True |False | +---------------------------------+------------------+--------------------+---------------+-------+--------------+--------------+--------------+-------------------+ # ceph nvmeof namespace list nqn.2025-08.com.example:HOSA.hosa +-----------------------------------------+---------+--------+--------+--------------+----------+-----------+-----------+------------+-------------+------------+-----+----+------------------------------------+---------------------------------+-----------+-------------------+---------+ |Bdev Name |Rbd Image|Rbd Pool|Lb Group|Rbd Image Size|Block Size|R/W Ios/Sec|R/W Mbs/Sec|Read Mbs/Sec|Write Mbs/Sec|Auto Visible|Hosts|Nsid|Uuid |Ns Subsystem Nqn |Trash Image|Disable Auto Resize|Read Only| +-----------------------------------------+---------+--------+--------+--------------+----------+-----------+-----------+------------+-------------+------------+-----+----+------------------------------------+---------------------------------+-----------+-------------------+---------+ |bdev_57f2c81a-0b07-406b-84aa-c62d66b5f10d|ns01 |nvmeof |5 |100GB |512B |0 |0 |0 |0 |True |[] |1 |57f2c81a-0b07-406b-84aa-c62d66b5f10d|nqn.2025-08.com.example:HOSA.hosa|False |False |False | |bdev_b4bf328a-8c67-4879-bf13-a799884d6dbe|ns02 |nvmeof |5 |200GB |512B |0 |0 |0 |0 |True |[] |2 |b4bf328a-8c67-4879-bf13-a799884d6dbe|nqn.2025-08.com.example:HOSA.hosa|False |False |False | +-----------------------------------------+---------+--------+--------+--------------+----------+-----------+-----------+------------+-------------+------------+-----+----+------------------------------------+---------------------------------+-----------+-------------------+---------+ # ceph nvmeof listener list nqn.2025-08.com.example:HOSA.hosa +-------------+------+--------------+------+------+-------+ |Host Name |Trtype|Traddr |Secure|Adrfam|Trsvcid| +-------------+------+--------------+------+------+-------+ |b236asbrce101|TCP |10.105.141.195|False |ipv4 |4420 | |b236asbrce102|TCP |10.105.141.196|False |ipv4 |4420 | |b236asbrce103|TCP |10.105.141.197|False |ipv4 |4420 | |b236asbrce104|TCP |10.105.141.198|False |ipv4 |4420 | +-------------+------+--------------+------+------+-------+ On the initiator host: # nvme discover -t tcp -a 10.105.141.195 -s 8009 -v warning: use hostid which does not match uuid in hostnqn nvme0: nqn.2014-08.org.nvmexpress.discovery connected Discovery Log Number of Records 4, Generation counter 6 =====Discovery Log Entry 0====== trtype: tcp adrfam: ipv4 subtype: nvme subsystem treq: not required portid: 0 trsvcid: 4420 subnqn: nqn.2025-08.com.example:HOSA.hosa traddr: 10.105.141.195 eflags: none sectype: none =====Discovery Log Entry 1====== trtype: tcp adrfam: ipv4 subtype: nvme subsystem treq: not required portid: 1 trsvcid: 4420 subnqn: nqn.2025-08.com.example:HOSA.hosa traddr: 10.105.141.196 eflags: none sectype: none =====Discovery Log Entry 2====== trtype: tcp adrfam: ipv4 subtype: nvme subsystem treq: not required portid: 2 trsvcid: 4420 subnqn: nqn.2025-08.com.example:HOSA.hosa traddr: 10.105.141.197 eflags: none sectype: none =====Discovery Log Entry 3====== trtype: tcp adrfam: ipv4 subtype: nvme subsystem treq: not required portid: 3 trsvcid: 4420 subnqn: nqn.2025-08.com.example:HOSA.hosa traddr: 10.105.141.198 eflags: none sectype: none nvme0: nqn.2014-08.org.nvmexpress.discovery disconnected # nvme connect-all -t tcp -a 10.105.141.195 -s 8009 -v warning: use hostid which does not match uuid in hostnqn nvme0: nqn.2014-08.org.nvmexpress.discovery connected nvme1: nqn.2025-08.com.example:HOSA.hosa connected nvme2: nqn.2025-08.com.example:HOSA.hosa connected nvme3: nqn.2025-08.com.example:HOSA.hosa connected nvme4: nqn.2025-08.com.example:HOSA.hosa connected nvme0: nqn.2014-08.org.nvmexpress.discovery disconnected Looking good so far, but: # nvme list Node Generic SN Model Namespace Usage Format FW Rev --------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- -------- # nvme list -v Subsystem Subsystem-NQN Controllers ---------------- -------------------------------------------------------------------------------------------- ---------------- nvme-subsys1 nqn.2025-08.com.example:HOSA.hosa nvme1, nvme2, nvme3, nvme4 Device SN MN FR TxPort Address Slot Subsystem Namespaces -------- -------------------- ---------------------------------------- -------- ------ -------------- ------ ------------ ---------------- nvme1 Ceph30844860597194 Ceph bdev Controller 25.05 tcp traddr=10.105.141.195,trsvcid=4420,src_addr=10.105.141.212 nvme-subsys1 nvme2 Ceph30844860597194 Ceph bdev Controller 25.05 tcp traddr=10.105.141.196,trsvcid=4420,src_addr=10.105.141.212 nvme-subsys1 nvme3 Ceph30844860597194 Ceph bdev Controller 25.05 tcp traddr=10.105.141.197,trsvcid=4420,src_addr=10.105.141.212 nvme-subsys1 nvme4 Ceph30844860597194 Ceph bdev Controller 25.05 tcp traddr=10.105.141.198,trsvcid=4420,src_addr=10.105.141.212 nvme-subsys1 Device Generic NSID Usage Format Controllers ------------ ------------ ---------- -------------------------- ---------------- ---------------- # nvme list-ns /dev/nvme1 -v -a opcode : 06 flags : 00 rsvd1 : 0000 nsid : 00000000 cdw2 : 00000000 cdw3 : 00000000 data_len : 00001000 metadata_len : 00000000 addr : 616b6af40000 metadata : 0 cdw10 : 00000010 cdw11 : 00000000 cdw12 : 00000000 cdw13 : 00000000 cdw14 : 00000000 cdw15 : 00000000 timeout_ms : 00000000 result : 00000000 err : 2 latency : 569 us NVMe status: Invalid Field in Command: A reserved coded value or an unsupported value in a defined field(0x2) How do I make it work again? Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
Hi, after creating a new subsystem this subsystem and its namespaces works: # ceph nvmeof subsystem list +---------------------------------+------------------+--------------------+---------------+-------+--------------+--------------+--------------+-------------------+ |Nqn |Serial Number |Model Number |Namespace Count|Subtype|Max Namespaces|Has Dhchap Key|Allow Any Host|Created Without Key| +---------------------------------+------------------+--------------------+---------------+-------+--------------+--------------+--------------+-------------------+ |nqn.2025-08.com.example:HOSA.hosa|Ceph30844860597194|Ceph bdev Controller|2 |NVMe |128 |False |True |False | |nqn.2025-08.com.example:TEST |Ceph44670645393042|Ceph bdev Controller|1 |NVMe |32 |False |True |False | +---------------------------------+------------------+--------------------+---------------+-------+--------------+--------------+--------------+-------------------+ # ceph nvmeof listener list nqn.2025-08.com.example:TEST +-------------+------+--------------+------+------+-------+ |Host Name |Trtype|Traddr |Secure|Adrfam|Trsvcid| +-------------+------+--------------+------+------+-------+ |b236asbrce101|TCP |10.105.141.195|False |ipv4 |4420 | |b236asbrce102|TCP |10.105.141.196|False |ipv4 |4420 | |b236asbrce103|TCP |10.105.141.197|False |ipv4 |4420 | |b236asbrce104|TCP |10.105.141.198|False |ipv4 |4420 | +-------------+------+--------------+------+------+-------+ # ceph nvmeof namespace list nqn.2025-08.com.example:TEST +-----------------------------------------+---------------------------+--------+--------+--------------+----------+-----------+-----------+------------+-------------+------------+-----+----+------------------------------------+----------------------------+-----------+-------------------+---------+ |Bdev Name |Rbd Image |Rbd Pool|Lb Group|Rbd Image Size|Block Size|R/W Ios/Sec|R/W Mbs/Sec|Read Mbs/Sec|Write Mbs/Sec|Auto Visible|Hosts|Nsid|Uuid |Ns Subsystem Nqn |Trash Image|Disable Auto Resize|Read Only| +-----------------------------------------+---------------------------+--------+--------+--------------+----------+-----------+-----------+------------+-------------+------------+-----+----+------------------------------------+----------------------------+-----------+-------------------+---------+ |bdev_b9b0259a-3702-415b-b223-0946cd91eaa3|nvme_nvmeof_hosa_w2h0v7yb8q|nvmeof |8 |150GB |512B |0 |0 |0 |0 |True |[] |1 |b9b0259a-3702-415b-b223-0946cd91eaa3|nqn.2025-08.com.example:TEST|False |False |False | +-----------------------------------------+---------------------------+--------+--------+--------------+----------+-----------+-----------+------------+-------------+------------+-----+----+------------------------------------+----------------------------+-----------+-------------------+---------+ On the initiator: # nvme connect-all -t tcp -a 10.105.141.195 -s 8009 -v warning: use hostid which does not match uuid in hostnqn nvme0: nqn.2014-08.org.nvmexpress.discovery connected nvme1: nqn.2025-08.com.example:HOSA.hosa connected nvme2: nqn.2025-08.com.example:HOSA.hosa connected nvme3: nqn.2025-08.com.example:HOSA.hosa connected nvme4: nqn.2025-08.com.example:HOSA.hosa connected nvme5: nqn.2025-08.com.example:TEST connected nvme6: nqn.2025-08.com.example:TEST connected nvme7: nqn.2025-08.com.example:TEST connected nvme8: nqn.2025-08.com.example:TEST connected nvme0: nqn.2014-08.org.nvmexpress.discovery disconnected # nvme list -v Subsystem Subsystem-NQN Controllers ---------------- -------------------------------------------------------------------------------------------- ---------------- nvme-subsys1 nqn.2025-08.com.example:HOSA.hosa nvme1, nvme2, nvme3, nvme4 nvme-subsys5 nqn.2025-08.com.example:TEST nvme5, nvme6, nvme7, nvme8 Device SN MN FR TxPort Address Slot Subsystem Namespaces -------- -------------------- ---------------------------------------- -------- ------ -------------- ------ ------------ ---------------- nvme1 Ceph30844860597194 Ceph bdev Controller 25.05 tcp traddr=10.105.141.195,trsvcid=4420,src_addr=10.105.141.212 nvme-subsys1 nvme2 Ceph30844860597194 Ceph bdev Controller 25.05 tcp traddr=10.105.141.196,trsvcid=4420,src_addr=10.105.141.212 nvme-subsys1 nvme3 Ceph30844860597194 Ceph bdev Controller 25.05 tcp traddr=10.105.141.197,trsvcid=4420,src_addr=10.105.141.212 nvme-subsys1 nvme4 Ceph30844860597194 Ceph bdev Controller 25.05 tcp traddr=10.105.141.198,trsvcid=4420,src_addr=10.105.141.212 nvme-subsys1 nvme5 Ceph44670645393042 Ceph bdev Controller 25.05 tcp traddr=10.105.141.195,trsvcid=4420,src_addr=10.105.141.212 nvme-subsys5 nvme5n1 nvme6 Ceph44670645393042 Ceph bdev Controller 25.05 tcp traddr=10.105.141.196,trsvcid=4420,src_addr=10.105.141.212 nvme-subsys5 nvme5n1 nvme7 Ceph44670645393042 Ceph bdev Controller 25.05 tcp traddr=10.105.141.197,trsvcid=4420,src_addr=10.105.141.212 nvme-subsys5 nvme5n1 nvme8 Ceph44670645393042 Ceph bdev Controller 25.05 tcp traddr=10.105.141.198,trsvcid=4420,src_addr=10.105.141.212 nvme-subsys5 nvme5n1 Device Generic NSID Usage Format Controllers ------------ ------------ ---------- -------------------------- ---------------- ---------------- /dev/nvme5n1 /dev/ng5n1 0x1 161.06 GB / 161.06 GB 512 B + 0 B nvme5, nvme6, nvme7, nvme8 # lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS sda 8:0 0 60G 0 disk ├─sda1 8:1 0 1G 0 part /boot/efi ├─sda2 8:2 0 2G 0 part /boot └─sda3 8:3 0 56.9G 0 part ├─system-swap 252:0 0 100M 0 lvm [SWAP] └─system-root 252:1 0 56.8G 0 lvm /var/lib/containers/storage/overlay / sr0 11:0 1 4M 0 rom nvme5n1 259:9 0 150G 0 disk So there must be a slight difference in the config of the subsystems / namespaces between v19 and v20. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
Am 9/10/25 um 10:27 schrieb Robert Sander:
So there must be a slight difference in the config of the subsystems / namespaces between v19 and v20.
After removing both namespaces (RBD ns01 and ns02) from the old subsystem and adding them again the initiator sees them: # nvme list Node Generic SN Model Namespace Usage Format FW Rev --------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- -------- /dev/nvme1n1 /dev/ng1n1 Ceph30844860597194 Ceph bdev Controller 0x1 107.37 GB / 107.37 GB 512 B + 0 B 25.05 /dev/nvme1n2 /dev/ng1n2 Ceph30844860597194 Ceph bdev Controller 0x2 214.75 GB / 214.75 GB 512 B + 0 B 25.05 /dev/nvme5n1 /dev/ng5n1 Ceph44670645393042 Ceph bdev Controller 0x1 161.06 GB / 161.06 GB 512 B + 0 B 25.05 # lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS sda 8:0 0 60G 0 disk ├─sda1 8:1 0 1G 0 part /boot/efi ├─sda2 8:2 0 2G 0 part /boot └─sda3 8:3 0 56.9G 0 part ├─system-swap 252:0 0 100M 0 lvm [SWAP] └─system-root 252:1 0 56.8G 0 lvm /var/lib/containers/storage/overlay / sr0 11:0 1 4M 0 rom nvme1n1 259:4 0 100G 0 disk └─nvme1n1p1 259:5 0 100G 0 part nvme5n1 259:9 0 150G 0 disk nvme1n2 259:14 0 200G 0 disk └─nvme1n2p1 259:16 0 200G 0 part Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
On Wed, Sep 10, 2025 at 10:34 AM Robert Sander <r.sander@heinlein-support.de> wrote:
Am 9/10/25 um 10:27 schrieb Robert Sander:
So there must be a slight difference in the config of the subsystems / namespaces between v19 and v20.
After removing both namespaces (RBD ns01 and ns02) from the old subsystem and adding them again the initiator sees them:
# nvme list Node Generic SN Model Namespace Usage Format FW Rev --------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- -------- /dev/nvme1n1 /dev/ng1n1 Ceph30844860597194 Ceph bdev Controller 0x1 107.37 GB / 107.37 GB 512 B + 0 B 25.05 /dev/nvme1n2 /dev/ng1n2 Ceph30844860597194 Ceph bdev Controller 0x2 214.75 GB / 214.75 GB 512 B + 0 B 25.05 /dev/nvme5n1 /dev/ng5n1 Ceph44670645393042 Ceph bdev Controller 0x1 161.06 GB / 161.06 GB 512 B + 0 B 25.05
# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS sda 8:0 0 60G 0 disk ├─sda1 8:1 0 1G 0 part /boot/efi ├─sda2 8:2 0 2G 0 part /boot └─sda3 8:3 0 56.9G 0 part ├─system-swap 252:0 0 100M 0 lvm [SWAP] └─system-root 252:1 0 56.8G 0 lvm /var/lib/containers/storage/overlay / sr0 11:0 1 4M 0 rom nvme1n1 259:4 0 100G 0 disk └─nvme1n1p1 259:5 0 100G 0 part nvme5n1 259:9 0 150G 0 disk nvme1n2 259:14 0 200G 0 disk └─nvme1n2p1 259:16 0 200G 0 part
Hi Robert, I'm forwarding some questions on behalf of Aviv as the reproduction attempt wasn't fruitful: - Were you using quay.ceph.io/ceph-ci/ceph:squid-nvmeof container image before upgrading from squid to tentacle? - If so, what GW version was deployed? Was it 1.4.21? - Can you provide the output of "nvme-gw show" command? - Can you provide ceph-mon log from the leader monitor/node? - Can you provide some GW logs? Thanks, Ilya
On Wed, Sep 10, 2025 at 8:49 AM Robert Sander <r.sander@heinlein-support.de> wrote:
Hi,
Am 9/9/25 um 17:34 schrieb Yuri Weinstein:
* Dashboard: Added support for NVMe/TCP (gateway groups, multiple namespaces),
I have just upgraded a test cluster and NVMEoF does not work any more.
The previous version running was quay.ceph.io/ceph-ci/ceph:squid-nvmeof and NVMEoF was working there.
The gateways seem to work and subsystem and namespaces are still configured but an initiator does not see namespaces.
# ceph orch ps --daemon_type nvmeof NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID nvmeof.hosa.b236asbrce101.ccaujq b236asbrce101 *:5500,4420,8009,10008 running (13h) 36s ago 13h 128M - 1.5.7 2c3e7ef350b8 74672e87a945 nvmeof.hosa.b236asbrce102.gnbwsc b236asbrce102 *:5500,4420,8009,10008 running (13h) 36s ago 13h 120M - 1.5.7 2c3e7ef350b8 1929979ab7e1 nvmeof.hosa.b236asbrce103.bthewi b236asbrce103 *:5500,4420,8009,10008 running (13h) 36s ago 13h 122M - 1.5.7 2c3e7ef350b8 14885171d1a5 nvmeof.hosa.b236asbrce104.vcgzvg b236asbrce104 *:5500,4420,8009,10008 running (13h) 36s ago 13h 122M - 1.5.7 2c3e7ef350b8 7b00b0f3356a # ceph nvmeof subsystem list +---------------------------------+------------------+--------------------+---------------+-------+--------------+--------------+--------------+-------------------+ |Nqn |Serial Number |Model Number |Namespace Count|Subtype|Max Namespaces|Has Dhchap Key|Allow Any Host|Created Without Key| +---------------------------------+------------------+--------------------+---------------+-------+--------------+--------------+--------------+-------------------+ |nqn.2025-08.com.example:HOSA.hosa|Ceph30844860597194|Ceph bdev Controller|2 |NVMe |128 |False |True |False | +---------------------------------+------------------+--------------------+---------------+-------+--------------+--------------+--------------+-------------------+ # ceph nvmeof namespace list nqn.2025-08.com.example:HOSA.hosa +-----------------------------------------+---------+--------+--------+--------------+----------+-----------+-----------+------------+-------------+------------+-----+----+------------------------------------+---------------------------------+-----------+-------------------+---------+ |Bdev Name |Rbd Image|Rbd Pool|Lb Group|Rbd Image Size|Block Size|R/W Ios/Sec|R/W Mbs/Sec|Read Mbs/Sec|Write Mbs/Sec|Auto Visible|Hosts|Nsid|Uuid |Ns Subsystem Nqn |Trash Image|Disable Auto Resize|Read Only| +-----------------------------------------+---------+--------+--------+--------------+----------+-----------+-----------+------------+-------------+------------+-----+----+------------------------------------+---------------------------------+-----------+-------------------+---------+ |bdev_57f2c81a-0b07-406b-84aa-c62d66b5f10d|ns01 |nvmeof |5 |100GB |512B |0 |0 |0 |0 |True |[] |1 |57f2c81a-0b07-406b-84aa-c62d66b5f10d|nqn.2025-08.com.example:HOSA.hosa|False |False |False | |bdev_b4bf328a-8c67-4879-bf13-a799884d6dbe|ns02 |nvmeof |5 |200GB |512B |0 |0 |0 |0 |True |[] |2 |b4bf328a-8c67-4879-bf13-a799884d6dbe|nqn.2025-08.com.example:HOSA.hosa|False |False |False | +-----------------------------------------+---------+--------+--------+--------------+----------+-----------+-----------+------------+-------------+------------+-----+----+------------------------------------+---------------------------------+-----------+-------------------+---------+ # ceph nvmeof listener list nqn.2025-08.com.example:HOSA.hosa +-------------+------+--------------+------+------+-------+ |Host Name |Trtype|Traddr |Secure|Adrfam|Trsvcid| +-------------+------+--------------+------+------+-------+ |b236asbrce101|TCP |10.105.141.195|False |ipv4 |4420 | |b236asbrce102|TCP |10.105.141.196|False |ipv4 |4420 | |b236asbrce103|TCP |10.105.141.197|False |ipv4 |4420 | |b236asbrce104|TCP |10.105.141.198|False |ipv4 |4420 | +-------------+------+--------------+------+------+-------+
On the initiator host:
# nvme discover -t tcp -a 10.105.141.195 -s 8009 -v warning: use hostid which does not match uuid in hostnqn nvme0: nqn.2014-08.org.nvmexpress.discovery connected
Discovery Log Number of Records 4, Generation counter 6 =====Discovery Log Entry 0====== trtype: tcp adrfam: ipv4 subtype: nvme subsystem treq: not required portid: 0 trsvcid: 4420 subnqn: nqn.2025-08.com.example:HOSA.hosa traddr: 10.105.141.195 eflags: none sectype: none =====Discovery Log Entry 1====== trtype: tcp adrfam: ipv4 subtype: nvme subsystem treq: not required portid: 1 trsvcid: 4420 subnqn: nqn.2025-08.com.example:HOSA.hosa traddr: 10.105.141.196 eflags: none sectype: none =====Discovery Log Entry 2====== trtype: tcp adrfam: ipv4 subtype: nvme subsystem treq: not required portid: 2 trsvcid: 4420 subnqn: nqn.2025-08.com.example:HOSA.hosa traddr: 10.105.141.197 eflags: none sectype: none =====Discovery Log Entry 3====== trtype: tcp adrfam: ipv4 subtype: nvme subsystem treq: not required portid: 3 trsvcid: 4420 subnqn: nqn.2025-08.com.example:HOSA.hosa traddr: 10.105.141.198 eflags: none sectype: none nvme0: nqn.2014-08.org.nvmexpress.discovery disconnected
# nvme connect-all -t tcp -a 10.105.141.195 -s 8009 -v warning: use hostid which does not match uuid in hostnqn nvme0: nqn.2014-08.org.nvmexpress.discovery connected nvme1: nqn.2025-08.com.example:HOSA.hosa connected nvme2: nqn.2025-08.com.example:HOSA.hosa connected nvme3: nqn.2025-08.com.example:HOSA.hosa connected nvme4: nqn.2025-08.com.example:HOSA.hosa connected nvme0: nqn.2014-08.org.nvmexpress.discovery disconnected
Looking good so far, but:
# nvme list Node Generic SN Model Namespace Usage Format FW Rev --------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- -------- # nvme list -v Subsystem Subsystem-NQN Controllers ---------------- -------------------------------------------------------------------------------------------- ---------------- nvme-subsys1 nqn.2025-08.com.example:HOSA.hosa nvme1, nvme2, nvme3, nvme4
Device SN MN FR TxPort Address Slot Subsystem Namespaces -------- -------------------- ---------------------------------------- -------- ------ -------------- ------ ------------ ---------------- nvme1 Ceph30844860597194 Ceph bdev Controller 25.05 tcp traddr=10.105.141.195,trsvcid=4420,src_addr=10.105.141.212 nvme-subsys1 nvme2 Ceph30844860597194 Ceph bdev Controller 25.05 tcp traddr=10.105.141.196,trsvcid=4420,src_addr=10.105.141.212 nvme-subsys1 nvme3 Ceph30844860597194 Ceph bdev Controller 25.05 tcp traddr=10.105.141.197,trsvcid=4420,src_addr=10.105.141.212 nvme-subsys1 nvme4 Ceph30844860597194 Ceph bdev Controller 25.05 tcp traddr=10.105.141.198,trsvcid=4420,src_addr=10.105.141.212 nvme-subsys1
Device Generic NSID Usage Format Controllers ------------ ------------ ---------- -------------------------- ---------------- ----------------
# nvme list-ns /dev/nvme1 -v -a opcode : 06 flags : 00 rsvd1 : 0000 nsid : 00000000 cdw2 : 00000000 cdw3 : 00000000 data_len : 00001000 metadata_len : 00000000 addr : 616b6af40000 metadata : 0 cdw10 : 00000010 cdw11 : 00000000 cdw12 : 00000000 cdw13 : 00000000 cdw14 : 00000000 cdw15 : 00000000 timeout_ms : 00000000 result : 00000000 err : 2 latency : 569 us NVMe status: Invalid Field in Command: A reserved coded value or an unsupported value in a defined field(0x2)
How do I make it work again?
Hi Robert, Aviv is going to take a look at this. Thanks, Ilya
Hi, I also tested zapping OSDs during cluster removal with 'cephadm rm-cluster ... --zap-osds', and it still doesn't work. I had created a tracker for Squid 19.2.3 a month ago and updated it: https://tracker.ceph.com/issues/72513 Thanks, Eugen Zitat von Yuri Weinstein <yweinste@redhat.com>:
This is the first release candidate for Tentacle. Ubuntu 22.04, 24.04 and CentOS 9 were built for this RC0
Feature highlights:
* RADOS: New features include long expected performance optimizations (FastEC) for EC pools, including partial reads and partial writes. Users can also expect to see BlueStore improvements such as compression and a new, faster WAL (write-ahead-log). Additional improvements include switching all components to the faster OMAP iteration interface, bypassable ceph_assert()s, fixed mclock bugs and configuration defaults, and testing improvements for dencoding verification. * MGR: Highlights include the ability to force-disable always-on modules and the removal of the restful and zabbix modules (both deprecated since 2020). Note that the dashboard module's richer and better-maintained RESTful API can be used as an alternative to the restful module, and the prometheus module can be used as an alternative monitoring solution for zabbix. * RGW: Multiple fixes: Lua scripts will not run against health checks, properly quoted ETag values returned by S3 CopyPart, PostObject and CompleteMultipartUpload responses. * RGW: IAM policy evaluation now supports conditions ArnEquals and ArnLike, along with their Not and IfExists variants. * RBD: New live migration features: RBD images can now be instantly imported from another Ceph cluster (native format) or from a wide variety of external sources/formats with the help of the new NBD stream and an appropriately capable NBD server such as `qemu-nbd`. Also added support for RBD namespace remapping while mirroring between Ceph clusters, new `rbd group info` and `rbd group snap info` commands and enhanced `rbd group snap ls` command. `rbd device map` command now defaults to msgr2. * CephFS: Directories may now be configured with case-insensitive or normalized directory entry names. This is an inheritable configuration making it apply to an entire directory tree. For more information, see https://docs.ceph.com/en/latest/cephfs/charmap/ * CephFS: Modifying the FS setting variable "max_mds" when a cluster is unhealthy now requires users to pass the confirmation flag (--yes-i-really-mean-it). This has been added as a precaution to tell the users that modifying "max_mds" may not help with troubleshooting or recovery effort. Instead, it might further destabilize the cluster. * CephFS: EOPNOTSUPP (Operation not supported ) is now returned by the CephFS fuse client for `fallocate` for the default case (i.e. mode == 0) since CephFS does not support disk space reservation. The only flags supported are `FALLOC_FL_KEEP_SIZE` and `FALLOC_FL_PUNCH_HOLE`. * Dashboard: Added support for NVMe/TCP (gateway groups, multiple namespaces), multi-cluster management, oAuth2 integration, and enhanced RGW/SMB features including multi-site automation, tiering, policies, lifecycles, notifications, and granular replication.
* Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-20.1.0.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: 010a3ad647c9962d47812a66ad6feda26ab28aa4 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (5)
-
Eugen Block
-
Ilya Dryomov
-
John Mulligan
-
Robert Sander
-
Yuri Weinstein