Re: [Ceph-maintainers] Re: [ceph-users] Re: v20.2.3 Tentacle released
On Fri, Aug 7, 2026 at 6:44 PM Salatiel Filho via Ceph-maintainers < ceph-maintainers@ceph.io> wrote:
What exactly do you mean by CephFS cloning ? This cluster is used exclusively to provide RWX volumes to a k8s using cephcsi. Velero does backup though and creates volume snapshots , but I am not sure if that's what you mean. Any commands I can run to confirm if cloning is being used ?
[Afreen] Yes that confirms then . So this bugs is being fixed and it only happens with cephfs subvolums die to a wrong format internally used https://tracker.ceph.com/issues/75619
"ceph progress clear" did the trick to load the pools page in the dashboard, but you said: "Its just clears the display events and does not stop operations." What operations are still running ? post-upgrade operations? why? Will them eventually finish ?
[Afreen] What I meant was that running that command was not harmful. And good to know it worked for you. Thanks
Thanks!
Atenciosamente/Kind regards, Salatiel
On Thu, Aug 6, 2026 at 8:31 PM Afreen <afreen23.git@gmail.com> wrote:
Hey,
That seems like some issue in parsing the format and so far what we
And pools page is failing because its polling /summary API - fetching
found out is that it could happen when CephFS cloning is used , are you using that ? progress internally.
You can try clearing progress and then refresh - "ceph progress clear".
Its just clears the display events and does not stop operations.
- Afreen
On Thu, Aug 6, 2026 at 6:42 PM Salatiel Filho via ceph-users <
After upgrading from 20.2.2 to 20.2.3, the dashboard is partially broken. List pools don't work. Error 500. Also get error 500 popup as soon as I login to the dashboard.
Logs show this: 2026-08-06 10:04:23 2026-08-06T13:04:23.386+0000 7f5afcd91640 0 [dashboard ERROR dashboard.services.exception] Internal Server Error 2026-08-06 10:04:23 Traceback (most recent call last): 2026-08-06 10:04:23 File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 47, in dashboard_exception_handler 2026-08-06 10:04:23 return handler(*args, **kwargs) 2026-08-06 10:04:23 File "/lib/python3.9/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__ 2026-08-06 10:04:23 return self.callable(*self.args,
**self.kwargs)
2026-08-06 10:04:23 File "/usr/share/ceph/mgr/dashboard/controllers/_base_controller.py", line 264, in inner 2026-08-06 10:04:23 ret = func(*args, **kwargs) 2026-08-06 10:04:23 File "/usr/share/ceph/mgr/dashboard/controllers/summary.py", line 105, in __call__ 2026-08-06 10:04:23 e, f = progress.get_progress_tasks() 2026-08-06 10:04:23 File "/usr/share/ceph/mgr/dashboard/services/progress.py", line 89, in get_progress_tasks 2026-08-06 10:04:23 finished_t.append(_progress_event_to_dashboard_task(ev, True)) 2026-08-06 10:04:23 File "/usr/share/ceph/mgr/dashboard/services/progress.py", line 60, in _progress_event_to_dashboard_task 2026-08-06 10:04:23 _progress_event_to_dashboard_task_common(event, task) 2026-08-06 10:04:23 File "/usr/share/ceph/mgr/dashboard/services/progress.py", line 52, in _progress_event_to_dashboard_task_common 2026-08-06 10:04:23 'metadata': dict(event.get('refs', {})), 2026-08-06 10:04:23 ValueError: dictionary update sequence element #0 has length 3; 2 is required 2026-08-06 10:04:23 2026-08-06T13:04:23.390+0000 7f5afcd91640 0 [dashboard ERROR dashboard.tools] [::ffff:10.175.143.9:53998] [GET] [500] [0.009s] [admin] [513.0B] /api/summary 2026-08-06 10:04:23 2026-08-06T13:04:23.390+0000 7f5afcd91640 0 [dashboard ERROR dashboard.tools] [b'{"status": "500 Internal Server Error", "detail": "The server encountered an unexpected condition which prevented it from fulfilling the request.", "request_id": "c7349113-44d8-4e48-88d6-3a469786397f"}
'] 2026-08-06 10:04:23 2026-08-06T13:04:23.494+0000 7f5afed95640 0 [dashboard ERROR dashboard.controllers.frontend_logging] (https://ceph.mydomain/#/pool): Http failure response for https://ceph.mydomain/api/summary: 500 Internal Server Error
Atenciosamente/Kind regards, Salatiel
On Wed, Aug 5, 2026 at 11:06 AM Yuri Weinstein via ceph-users <ceph-users@ceph.io> wrote:
This is the third minor release in the Tentacle series. We recommend that all users update to this release.
Notable Changes ---------------
MDS (Metadata Server) ---------------------
* Fixed a crash and shutdown hang that could occur when ephemeral
ceph-users@ceph.io> wrote: pins were
active and max_mds was set to 0. * Session reclaim could miss blocklisting an old session; this is now handled correctly. * Fixed a case where scrub was unable to identify damage from an irreparable hard link.
OSD (Object Storage Daemon) ----------------------------
* BlueStore: Fixed a BlueFS WAL envelope-mode bug that caused write buffer misalignment. * Erasure Coding: Fixed truncate+write planning for EC shard sizes. * Fixed a condition that prevented rolling forward of PG log entries. * Prevented OSDMap::check_health() from asserting when new OSDs are found in a subtree. * Scrub: "repairing" scrubs are now allowed at all times.
RGW (RADOS Gateway) --------------------
* Added the ssl_ciphersuites option for the Beast front-end to support TLS 1.3 cipher customization. * Fixed PutObject's canned_acl comparison bug affecting BlockPublicAcls. * Fixed crashes occurring on realm reload. * Implemented CopyObject support for encrypted objects. * Added mTLS client-certificate authentication support for Kafka notifications. * Fixed removal of delete markers in lifecycle processing.
RADOS / librados / neorados ----------------------------
* neorados: Fixed an infinite trim loop on empty data log shards. * Fixed cancellation-slot cleanup in librados/asio's associated executor.
Dashboard -----------------
* Added a hardware-monitoring dashboard backed by node-proxy metrics. * Fixed a bind-address regression caused by CherryPy isolation. * Added support for adding hosts via CSV upload. * NVMeoF: multiple CLI and UI backports (EC pool support, IO statistics, role management). * Object: The global RGW Roles tab has been removed. Role management is now scoped under RGW Accounts, where roles can be listed, created, edited, and deleted for a selected account.
ceph-volume -----------
* Reworked OSD mapper lifecycle handling (LVM + raw) for activate. * Improved detection of rotational media under dm-crypt to bypass the workqueue correctly. * Fixed OSD re-deployment issues with disk-selection filters and DB devices. * Added retry handling for lvs when it returns an empty result or a "devices file is missing" error.
NVMe-oF -------
* Added configurable delayed failback (default 0, no delay) to accommodate initiators that recover paths with a delay. * Gateways in the DELETING state now ignore beacons and send empty maps instead of processing them. * rbd_with_crc32c is now enabled by default via cephadm. * Introduced a new NVMeoF mgr module. The module automatically creates the ".nvmeof" metadata pool if it does not already exist. Gateway-group state files are stored in this pool unless the user specifies another pool. The module is enabled automatically on fresh installs; on upgrades from an older cluster, it must be enabled manually.
RBD ---
* Fixed a use-after-free bug releasing object map locks during deep copy. * Fixed memory leaks in PWL discard operations. * Obsolete primary mirror snapshots are now pruned after relocation.
We recommend users to update to this release. For detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/en/news/blog/2026/v20-2-3-tentacle-released/
Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-20.2.3.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: 06c2f9c35b67055a8a6fb99d1be236b3c4832ace _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Ceph-maintainers mailing list -- ceph-maintainers@ceph.io To unsubscribe send an email to ceph-maintainers-leave@ceph.io
On Fri, Aug 7, 2026 at 10:51 PM Afreen <afreen23.git@gmail.com> wrote:
On Fri, Aug 7, 2026 at 6:44 PM Salatiel Filho via Ceph-maintainers < ceph-maintainers@ceph.io> wrote:
What exactly do you mean by CephFS cloning ? This cluster is used exclusively to provide RWX volumes to a k8s using cephcsi. Velero does backup though and creates volume snapshots , but I am not sure if that's what you mean. Any commands I can run to confirm if cloning is being used ?
[Afreen] Yes that confirms then . So this bugs is being fixed and it only happens with cephfs subvolums die to a wrong format internally used https://tracker.ceph.com/issues/75619
[Afreen] Yes that confirms then . So this bugs is being fixed and it only happens with cephfs subvolums due to a wrong format internally used https://tracker.ceph.com/issues/75619 *typo my bad!
"ceph progress clear" did the trick to load the pools page in the
dashboard, but you said: "Its just clears the display events and does not stop operations." What operations are still running ? post-upgrade operations? why? Will them eventually finish ?
[Afreen] What I meant was that running that command was not harmful. And good to know it worked for you.
Thanks
Thanks!
Atenciosamente/Kind regards, Salatiel
On Thu, Aug 6, 2026 at 8:31 PM Afreen <afreen23.git@gmail.com> wrote:
Hey,
That seems like some issue in parsing the format and so far what we
And pools page is failing because its polling /summary API - fetching
found out is that it could happen when CephFS cloning is used , are you using that ? progress internally.
You can try clearing progress and then refresh - "ceph progress clear".
Its just clears the display events and does not stop operations.
- Afreen
On Thu, Aug 6, 2026 at 6:42 PM Salatiel Filho via ceph-users <
After upgrading from 20.2.2 to 20.2.3, the dashboard is partially broken. List pools don't work. Error 500. Also get error 500 popup as soon as I login to the dashboard.
Logs show this: 2026-08-06 10:04:23 2026-08-06T13:04:23.386+0000 7f5afcd91640 0 [dashboard ERROR dashboard.services.exception] Internal Server Error 2026-08-06 10:04:23 Traceback (most recent call last): 2026-08-06 10:04:23 File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 47, in dashboard_exception_handler 2026-08-06 10:04:23 return handler(*args, **kwargs) 2026-08-06 10:04:23 File "/lib/python3.9/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__ 2026-08-06 10:04:23 return self.callable(*self.args,
**self.kwargs)
2026-08-06 10:04:23 File "/usr/share/ceph/mgr/dashboard/controllers/_base_controller.py", line 264, in inner 2026-08-06 10:04:23 ret = func(*args, **kwargs) 2026-08-06 10:04:23 File "/usr/share/ceph/mgr/dashboard/controllers/summary.py", line 105, in __call__ 2026-08-06 10:04:23 e, f = progress.get_progress_tasks() 2026-08-06 10:04:23 File "/usr/share/ceph/mgr/dashboard/services/progress.py", line 89, in get_progress_tasks 2026-08-06 10:04:23 finished_t.append(_progress_event_to_dashboard_task(ev, True)) 2026-08-06 10:04:23 File "/usr/share/ceph/mgr/dashboard/services/progress.py", line 60, in _progress_event_to_dashboard_task 2026-08-06 10:04:23 _progress_event_to_dashboard_task_common(event, task) 2026-08-06 10:04:23 File "/usr/share/ceph/mgr/dashboard/services/progress.py", line 52, in _progress_event_to_dashboard_task_common 2026-08-06 10:04:23 'metadata': dict(event.get('refs', {})), 2026-08-06 10:04:23 ValueError: dictionary update sequence element #0 has length 3; 2 is required 2026-08-06 10:04:23 2026-08-06T13:04:23.390+0000 7f5afcd91640 0 [dashboard ERROR dashboard.tools] [::ffff:10.175.143.9:53998] [GET] [500] [0.009s] [admin] [513.0B] /api/summary 2026-08-06 10:04:23 2026-08-06T13:04:23.390+0000 7f5afcd91640 0 [dashboard ERROR dashboard.tools] [b'{"status": "500 Internal Server Error", "detail": "The server encountered an unexpected condition which prevented it from fulfilling the request.", "request_id": "c7349113-44d8-4e48-88d6-3a469786397f"}
'] 2026-08-06 10:04:23 2026-08-06T13:04:23.494+0000 7f5afed95640 0 [dashboard ERROR dashboard.controllers.frontend_logging] (https://ceph.mydomain/#/pool): Http failure response for https://ceph.mydomain/api/summary: 500 Internal Server Error
Atenciosamente/Kind regards, Salatiel
On Wed, Aug 5, 2026 at 11:06 AM Yuri Weinstein via ceph-users <ceph-users@ceph.io> wrote:
This is the third minor release in the Tentacle series. We recommend that all users update to this release.
Notable Changes ---------------
MDS (Metadata Server) ---------------------
* Fixed a crash and shutdown hang that could occur when ephemeral
active and max_mds was set to 0. * Session reclaim could miss blocklisting an old session; this is now handled correctly. * Fixed a case where scrub was unable to identify damage from an irreparable hard link.
OSD (Object Storage Daemon) ----------------------------
* BlueStore: Fixed a BlueFS WAL envelope-mode bug that caused write buffer misalignment. * Erasure Coding: Fixed truncate+write planning for EC shard sizes. * Fixed a condition that prevented rolling forward of PG log entries. * Prevented OSDMap::check_health() from asserting when new OSDs are found in a subtree. * Scrub: "repairing" scrubs are now allowed at all times.
RGW (RADOS Gateway) --------------------
* Added the ssl_ciphersuites option for the Beast front-end to support TLS 1.3 cipher customization. * Fixed PutObject's canned_acl comparison bug affecting BlockPublicAcls. * Fixed crashes occurring on realm reload. * Implemented CopyObject support for encrypted objects. * Added mTLS client-certificate authentication support for Kafka notifications. * Fixed removal of delete markers in lifecycle processing.
RADOS / librados / neorados ----------------------------
* neorados: Fixed an infinite trim loop on empty data log shards. * Fixed cancellation-slot cleanup in librados/asio's associated executor.
Dashboard -----------------
* Added a hardware-monitoring dashboard backed by node-proxy metrics. * Fixed a bind-address regression caused by CherryPy isolation. * Added support for adding hosts via CSV upload. * NVMeoF: multiple CLI and UI backports (EC pool support, IO statistics, role management). * Object: The global RGW Roles tab has been removed. Role management is now scoped under RGW Accounts, where roles can be listed, created, edited, and deleted for a selected account.
ceph-volume -----------
* Reworked OSD mapper lifecycle handling (LVM + raw) for activate. * Improved detection of rotational media under dm-crypt to bypass the workqueue correctly. * Fixed OSD re-deployment issues with disk-selection filters and DB devices. * Added retry handling for lvs when it returns an empty result or a "devices file is missing" error.
NVMe-oF -------
* Added configurable delayed failback (default 0, no delay) to accommodate initiators that recover paths with a delay. * Gateways in the DELETING state now ignore beacons and send empty maps instead of processing them. * rbd_with_crc32c is now enabled by default via cephadm. * Introduced a new NVMeoF mgr module. The module automatically creates the ".nvmeof" metadata pool if it does not already exist. Gateway-group state files are stored in this pool unless the user specifies another
ceph-users@ceph.io> wrote: pins were pool. The
module is enabled automatically on fresh installs; on upgrades from an older cluster, it must be enabled manually.
RBD ---
* Fixed a use-after-free bug releasing object map locks during deep copy. * Fixed memory leaks in PWL discard operations. * Obsolete primary mirror snapshots are now pruned after relocation.
We recommend users to update to this release. For detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/en/news/blog/2026/v20-2-3-tentacle-released/
Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-20.2.3.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: 06c2f9c35b67055a8a6fb99d1be236b3c4832ace _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Ceph-maintainers mailing list -- ceph-maintainers@ceph.io To unsubscribe send an email to ceph-maintainers-leave@ceph.io
participants (1)
-
Afreen