cephfs-top causes 16 mgr modules have recently crashed
Hello When I run cephfs-top it causes mgr module crash. Can you please tell me the reason? My environment: My ceph version 17.2.6 Operating System: Ubuntu 22.04.2 LTS Kernel: Linux 5.15.0-84-generic I created the cephfs-top user with the following command: ceph auth get-or-create client.fstop mon 'allow r' mds 'allow r' osd 'allow r' mgr 'allow r' > /etc/ceph/ceph.client.fstop.keyring This is the crash report: root@ud-01:~# ceph crash info 2024-01-22T21:25:59.313305Z_526253e3-e8cc-4d2c-adcb-69a7c9986801 { "backtrace": [ " File \"/usr/share/ceph/mgr/stats/module.py\", line 32, in notify\n self.fs_perf_stats.notify_cmd(notify_id)", " File \"/usr/share/ceph/mgr/stats/fs/perf_stats.py\", line 177, in notify_cmd\n metric_features = int(metadata[CLIENT_METADATA_KEY][\"metric_spec\"][\"metric_flags\"][\"feature_bits\"], 16)", "ValueError: invalid literal for int() with base 16: '0x'" ], "ceph_version": "17.2.6", "crash_id": "2024-01-22T21:25:59.313305Z_526253e3-e8cc-4d2c-adcb-69a7c9986801", "entity_name": "mgr.ud-01.qycnol", "mgr_module": "stats", "mgr_module_caller": "ActivePyModule::notify", "mgr_python_exception": "ValueError", "os_id": "centos", "os_name": "CentOS Stream", "os_version": "8", "os_version_id": "8", "process_name": "ceph-mgr", "stack_sig": "971ae170f1fff7f7bc0b7ae86d164b2b0136a8bd5ca7956166ea5161e51ad42c", "timestamp": "2024-01-22T21:25:59.313305Z", "utsname_hostname": "ud-01", "utsname_machine": "x86_64", "utsname_release": "5.15.0-84-generic", "utsname_sysname": "Linux", "utsname_version": "#93-Ubuntu SMP Tue Sep 5 17:16:10 UTC 2023" } Best regards.
Please have this fix: https://tracker.ceph.com/issues/59551. It's backported to quincy. On 23/01/24 03:11, Özkan Göksu wrote:
Hello
When I run cephfs-top it causes mgr module crash. Can you please tell me the reason?
My environment: My ceph version 17.2.6 Operating System: Ubuntu 22.04.2 LTS Kernel: Linux 5.15.0-84-generic
I created the cephfs-top user with the following command: ceph auth get-or-create client.fstop mon 'allow r' mds 'allow r' osd 'allow r' mgr 'allow r' > /etc/ceph/ceph.client.fstop.keyring
This is the crash report:
root@ud-01:~# ceph crash info 2024-01-22T21:25:59.313305Z_526253e3-e8cc-4d2c-adcb-69a7c9986801 { "backtrace": [ " File \"/usr/share/ceph/mgr/stats/module.py\", line 32, in notify\n self.fs_perf_stats.notify_cmd(notify_id)", " File \"/usr/share/ceph/mgr/stats/fs/perf_stats.py\", line 177, in notify_cmd\n metric_features = int(metadata[CLIENT_METADATA_KEY][\"metric_spec\"][\"metric_flags\"][\"feature_bits\"], 16)", "ValueError: invalid literal for int() with base 16: '0x'" ], "ceph_version": "17.2.6", "crash_id": "2024-01-22T21:25:59.313305Z_526253e3-e8cc-4d2c-adcb-69a7c9986801", "entity_name": "mgr.ud-01.qycnol", "mgr_module": "stats", "mgr_module_caller": "ActivePyModule::notify", "mgr_python_exception": "ValueError", "os_id": "centos", "os_name": "CentOS Stream", "os_version": "8", "os_version_id": "8", "process_name": "ceph-mgr", "stack_sig": "971ae170f1fff7f7bc0b7ae86d164b2b0136a8bd5ca7956166ea5161e51ad42c", "timestamp": "2024-01-22T21:25:59.313305Z", "utsname_hostname": "ud-01", "utsname_machine": "x86_64", "utsname_release": "5.15.0-84-generic", "utsname_sysname": "Linux", "utsname_version": "#93-Ubuntu SMP Tue Sep 5 17:16:10 UTC 2023" }
Best regards. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello Jos. Thank you for the reply. I can upgrade to 17.2.7 but I wonder can I only upgrade MON+MGR for this issue or do I need to upgrade all the parts? Otherwise I need to wait few weeks. I don't want to request maintenance during delivery time. root@ud-01:~# ceph orch upgrade ls { "image": "quay.io/ceph/ceph", "registry": "quay.io", "bare_image": "ceph/ceph", "versions": [ "18.2.1", "18.2.0", "18.1.3", "18.1.2", "18.1.1", "18.1.0", "17.2.7", "17.2.6", "17.2.5", "17.2.4", "17.2.3", "17.2.2", "17.2.1", "17.2.0" ] } Best regards Jos Collin <jcollin@redhat.com>, 23 Oca 2024 Sal, 07:42 tarihinde şunu yazdı:
Please have this fix: https://tracker.ceph.com/issues/59551. It's backported to quincy.
On 23/01/24 03:11, Özkan Göksu wrote:
Hello
When I run cephfs-top it causes mgr module crash. Can you please tell me the reason?
My environment: My ceph version 17.2.6 Operating System: Ubuntu 22.04.2 LTS Kernel: Linux 5.15.0-84-generic
I created the cephfs-top user with the following command: ceph auth get-or-create client.fstop mon 'allow r' mds 'allow r' osd 'allow r' mgr 'allow r' > /etc/ceph/ceph.client.fstop.keyring
This is the crash report:
root@ud-01:~# ceph crash info 2024-01-22T21:25:59.313305Z_526253e3-e8cc-4d2c-adcb-69a7c9986801 { "backtrace": [ " File \"/usr/share/ceph/mgr/stats/module.py\", line 32, in notify\n self.fs_perf_stats.notify_cmd(notify_id)", " File \"/usr/share/ceph/mgr/stats/fs/perf_stats.py\", line 177, in notify_cmd\n metric_features =
int(metadata[CLIENT_METADATA_KEY][\"metric_spec\"][\"metric_flags\"][\"feature_bits\"],
16)", "ValueError: invalid literal for int() with base 16: '0x'" ], "ceph_version": "17.2.6", "crash_id": "2024-01-22T21:25:59.313305Z_526253e3-e8cc-4d2c-adcb-69a7c9986801", "entity_name": "mgr.ud-01.qycnol", "mgr_module": "stats", "mgr_module_caller": "ActivePyModule::notify", "mgr_python_exception": "ValueError", "os_id": "centos", "os_name": "CentOS Stream", "os_version": "8", "os_version_id": "8", "process_name": "ceph-mgr", "stack_sig": "971ae170f1fff7f7bc0b7ae86d164b2b0136a8bd5ca7956166ea5161e51ad42c", "timestamp": "2024-01-22T21:25:59.313305Z", "utsname_hostname": "ud-01", "utsname_machine": "x86_64", "utsname_release": "5.15.0-84-generic", "utsname_sysname": "Linux", "utsname_version": "#93-Ubuntu SMP Tue Sep 5 17:16:10 UTC 2023" }
Best regards. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
This fix is in the mds. I think you need to read https://docs.ceph.com/en/quincy/cephadm/upgrade/#staggered-upgrade. On 23/01/24 12:19, Özkan Göksu wrote:
Hello Jos. Thank you for the reply.
I can upgrade to 17.2.7 but I wonder can I only upgrade MON+MGR for this issue or do I need to upgrade all the parts? Otherwise I need to wait few weeks. I don't want to request maintenance during delivery time.
root@ud-01:~# ceph orch upgrade ls { "image": "quay.io/ceph/ceph <http://quay.io/ceph/ceph>", "registry": "quay.io <http://quay.io>", "bare_image": "ceph/ceph", "versions": [ "18.2.1", "18.2.0", "18.1.3", "18.1.2", "18.1.1", "18.1.0", "17.2.7", "17.2.6", "17.2.5", "17.2.4", "17.2.3", "17.2.2", "17.2.1", "17.2.0" ] }
Best regards
Jos Collin <jcollin@redhat.com>, 23 Oca 2024 Sal, 07:42 tarihinde şunu yazdı:
Please have this fix: https://tracker.ceph.com/issues/59551. It's backported to quincy.
On 23/01/24 03:11, Özkan Göksu wrote: > Hello > > When I run cephfs-top it causes mgr module crash. Can you please tell me > the reason? > > My environment: > My ceph version 17.2.6 > Operating System: Ubuntu 22.04.2 LTS > Kernel: Linux 5.15.0-84-generic > > I created the cephfs-top user with the following command: > ceph auth get-or-create client.fstop mon 'allow r' mds 'allow r' osd 'allow > r' mgr 'allow r' > /etc/ceph/ceph.client.fstop.keyring > > This is the crash report: > > root@ud-01:~# ceph crash info > 2024-01-22T21:25:59.313305Z_526253e3-e8cc-4d2c-adcb-69a7c9986801 > { > "backtrace": [ > " File \"/usr/share/ceph/mgr/stats/module.py\", line 32, in > notify\n self.fs_perf_stats.notify_cmd(notify_id)", > " File \"/usr/share/ceph/mgr/stats/fs/perf_stats.py\", line 177, > in notify_cmd\n metric_features = > int(metadata[CLIENT_METADATA_KEY][\"metric_spec\"][\"metric_flags\"][\"feature_bits\"], > 16)", > "ValueError: invalid literal for int() with base 16: '0x'" > ], > "ceph_version": "17.2.6", > "crash_id": > "2024-01-22T21:25:59.313305Z_526253e3-e8cc-4d2c-adcb-69a7c9986801", > "entity_name": "mgr.ud-01.qycnol", > "mgr_module": "stats", > "mgr_module_caller": "ActivePyModule::notify", > "mgr_python_exception": "ValueError", > "os_id": "centos", > "os_name": "CentOS Stream", > "os_version": "8", > "os_version_id": "8", > "process_name": "ceph-mgr", > "stack_sig": > "971ae170f1fff7f7bc0b7ae86d164b2b0136a8bd5ca7956166ea5161e51ad42c", > "timestamp": "2024-01-22T21:25:59.313305Z", > "utsname_hostname": "ud-01", > "utsname_machine": "x86_64", > "utsname_release": "5.15.0-84-generic", > "utsname_sysname": "Linux", > "utsname_version": "#93-Ubuntu SMP Tue Sep 5 17:16:10 UTC 2023" > } > > > Best regards. > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io > _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello Jos. I check the diff and notice the difference: https://github.com/ceph/ceph/pull/52127/files Thank you for the guide link and for the fix. Have a great day. Regards. 23 Oca 2024 Sal 11:07 tarihinde Jos Collin <jcollin@redhat.com> şunu yazdı:
This fix is in the mds. I think you need to read https://docs.ceph.com/en/quincy/cephadm/upgrade/#staggered-upgrade.
On 23/01/24 12:19, Özkan Göksu wrote:
Hello Jos. Thank you for the reply.
I can upgrade to 17.2.7 but I wonder can I only upgrade MON+MGR for this issue or do I need to upgrade all the parts? Otherwise I need to wait few weeks. I don't want to request maintenance during delivery time.
root@ud-01:~# ceph orch upgrade ls { "image": "quay.io/ceph/ceph", "registry": "quay.io", "bare_image": "ceph/ceph", "versions": [ "18.2.1", "18.2.0", "18.1.3", "18.1.2", "18.1.1", "18.1.0", "17.2.7", "17.2.6", "17.2.5", "17.2.4", "17.2.3", "17.2.2", "17.2.1", "17.2.0" ] }
Best regards
Jos Collin <jcollin@redhat.com>, 23 Oca 2024 Sal, 07:42 tarihinde şunu yazdı:
Please have this fix: https://tracker.ceph.com/issues/59551. It's backported to quincy.
On 23/01/24 03:11, Özkan Göksu wrote:
Hello
When I run cephfs-top it causes mgr module crash. Can you please tell me the reason?
My environment: My ceph version 17.2.6 Operating System: Ubuntu 22.04.2 LTS Kernel: Linux 5.15.0-84-generic
I created the cephfs-top user with the following command: ceph auth get-or-create client.fstop mon 'allow r' mds 'allow r' osd 'allow r' mgr 'allow r' > /etc/ceph/ceph.client.fstop.keyring
This is the crash report:
root@ud-01:~# ceph crash info 2024-01-22T21:25:59.313305Z_526253e3-e8cc-4d2c-adcb-69a7c9986801 { "backtrace": [ " File \"/usr/share/ceph/mgr/stats/module.py\", line 32, in notify\n self.fs_perf_stats.notify_cmd(notify_id)", " File \"/usr/share/ceph/mgr/stats/fs/perf_stats.py\", line 177, in notify_cmd\n metric_features =
int(metadata[CLIENT_METADATA_KEY][\"metric_spec\"][\"metric_flags\"][\"feature_bits\"],
16)", "ValueError: invalid literal for int() with base 16: '0x'" ], "ceph_version": "17.2.6", "crash_id": "2024-01-22T21:25:59.313305Z_526253e3-e8cc-4d2c-adcb-69a7c9986801", "entity_name": "mgr.ud-01.qycnol", "mgr_module": "stats", "mgr_module_caller": "ActivePyModule::notify", "mgr_python_exception": "ValueError", "os_id": "centos", "os_name": "CentOS Stream", "os_version": "8", "os_version_id": "8", "process_name": "ceph-mgr", "stack_sig": "971ae170f1fff7f7bc0b7ae86d164b2b0136a8bd5ca7956166ea5161e51ad42c", "timestamp": "2024-01-22T21:25:59.313305Z", "utsname_hostname": "ud-01", "utsname_machine": "x86_64", "utsname_release": "5.15.0-84-generic", "utsname_sysname": "Linux", "utsname_version": "#93-Ubuntu SMP Tue Sep 5 17:16:10 UTC 2023" }
Best regards. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (2)
-
Jos Collin
-
Özkan Göksu