Client kernel crashes on cephfs access
Hi everyone, I would like to ask for help regarding client kernel crashes that happen on cephfs access. We have been struggling with this for over a month now with over 100 crashes on 7 hosts during that time. Our cluster runs version 18.2.1. Our clients run CentOS Stream. On CentOS Stream 9 the problem started with kernel version 5.14.0-425.el9. Version 5.14.0-419.el9 is the last one without problems. It also occurred on CentOS Stream 8, starting with version 4.18.0-546.el8 (4.18.0-544.el8 being the last good one). The problem presents itself by the client kernel crashing, forcing a reboot of the machine. Apparently it is triggered by a certain level of IO on the cephfs mount. It works perfectly fine when we rollback to the last good kernel version. The exact call trace in vmcore-dmesg.txt differs between occurrences. Here are two typical examples: ``` [ 8641.382499] list_del corruption. next->prev should be ffff88bd0a4d4c80, but was ffff88bcefdfd280 [ 8641.382521] ------------[ cut here ]------------ [ 8641.382521] kernel BUG at lib/list_debug.c:54! [ 8641.382528] invalid opcode: 0000 [#1] PREEMPT SMP NOPTI [ 8641.382591] CPU: 2 PID: 83929 Comm: kworker/2:0 Kdump: loaded Not tainted 5.14.0-432.el9.x86_64 #1 [ 8641.382610] Hardware name: oVirt RHEL/RHEL-AV, BIOS edk2-20230524-4.el9_3 05/24/2023 [ 8641.382624] Workqueue: ceph-cap ceph_cap_unlink_work [ceph] [ 8641.382662] RIP: 0010:__list_del_entry_valid.cold+0x1d/0x47 [ 8641.382681] Code: c7 c7 78 42 d8 b1 e8 f9 87 fe ff 0f 0b 48 89 fe 48 c7 c7 08 43 d8 b1 e8 e8 87 fe ff 0f 0b 48 c7 c7 b8 43 d8 b1 e8 da 87 fe ff <0f> 0b 48 89 f2 48 89 fe 48 c7 c7 78 43 d8 b1 e8 c6 87 fe ff 0f 0b [ 8641.382711] RSP: 0018:ffff95a000d6be60 EFLAGS: 00010246 [ 8641.382722] RAX: 0000000000000054 RBX: ffff88bced76dc00 RCX: 0000000000000000 [ 8641.382734] RDX: 0000000000000000 RSI: ffff88c02eea0840 RDI: ffff88c02eea0840 [ 8641.382746] RBP: ffff88bd0a4d4c80 R08: 80000000ffff8434 R09: 0000000000ffff10 [ 8641.382758] R10: 000000000000000f R11: 000000000000000f R12: ffff88c02eeb2800 [ 8641.382779] R13: ffff88bcc4610258 R14: ffff88bcc46101b8 R15: ffff88bcc46101c8 [ 8641.382793] FS: 0000000000000000(0000) GS:ffff88c02ee80000(0000) knlGS:0000000000000000 [ 8641.382809] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8641.382819] CR2: 00007f35cee8a000 CR3: 0000000105708004 CR4: 00000000007706e0 [ 8641.382832] PKRU: 55555554 [ 8641.382838] Call Trace: [ 8641.382844] <TASK> [ 8641.382850] ? show_trace_log_lvl+0x1c4/0x2df [ 8641.382860] ? show_trace_log_lvl+0x1c4/0x2df [ 8641.382870] ? ceph_cap_unlink_work+0x3f/0x140 [ceph] [ 8641.382893] ? __die_body.cold+0x8/0xd [ 8641.382902] ? die+0x2b/0x50 [ 8641.382911] ? do_trap+0xce/0x120 [ 8641.382919] ? __list_del_entry_valid.cold+0x1d/0x47 [ 8641.382930] ? do_error_trap+0x65/0x80 [ 8641.382938] ? __list_del_entry_valid.cold+0x1d/0x47 [ 8641.382948] ? exc_invalid_op+0x4e/0x70 [ 8641.382958] ? __list_del_entry_valid.cold+0x1d/0x47 [ 8641.382975] ? asm_exc_invalid_op+0x16/0x20 [ 8641.382988] ? __list_del_entry_valid.cold+0x1d/0x47 [ 8641.382998] ceph_cap_unlink_work+0x3f/0x140 [ceph] [ 8641.383021] process_one_work+0x1e2/0x3b0 [ 8641.383032] ? __pfx_worker_thread+0x10/0x10 [ 8641.383043] worker_thread+0x50/0x3a0 [ 8641.383051] ? __pfx_worker_thread+0x10/0x10 [ 8641.383061] kthread+0xdd/0x100 [ 8641.383069] ? __pfx_kthread+0x10/0x10 [ 8641.383078] ret_from_fork+0x29/0x50 [ 8641.383090] </TASK> [ 8641.383095] Modules linked in: tls ceph libceph dns_resolver fscache netfs nft_counter ipt_REJECT xt_owner xt_conntrack nft_compat nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 rfkill ip_set nf_tables libcrc32c nfnetlink vfat fat intel_rapl_msr intel_rapl_common intel_uncore_frequency_common isst_if_common nfit virtio_gpu iTCO_wdt iTCO_vendor_support libnvdimm lpc_ich virtio_dma_buf drm_shmem_helper drm_kms_helper i2c_i801 rapl syscopyarea sysfillrect sysimgblt virtio_balloon fb_sys_fops i2c_smbus pcspkr joydev fuse drm ext4 mbcache jbd2 sr_mod cdrom sd_mod ahci t10_pi sg libahci crct10dif_pclmul crc32_pclmul crc32c_intel libata ghash_clmulni_intel virtio_net virtio_console virtio_scsi net_failover failover serio_raw ``` ``` [ 3538.365469] list_del corruption. next->prev should be ffff8d2b75997c80, but was ffff8d2afcfaae80 [ 3538.365488] ------------[ cut here ]------------ [ 3538.365488] kernel BUG at lib/list_debug.c:54! [ 3538.365493] invalid opcode: 0000 [#1] PREEMPT SMP NOPTI [ 3538.365553] CPU: 0 PID: 910 Comm: php-fpm Kdump: loaded Not tainted 5.14.0-432.el9.x86_64 #1 [ 3538.365569] Hardware name: oVirt RHEL/RHEL-AV, BIOS edk2-20230524-4.el9_3 05/24/2023 [ 3538.365582] RIP: 0010:__list_del_entry_valid.cold+0x1d/0x47 [ 3538.365612] Code: c7 c7 78 42 38 8e e8 f9 87 fe ff 0f 0b 48 89 fe 48 c7 c7 08 43 38 8e e8 e8 87 fe ff 0f 0b 48 c7 c7 b8 43 38 8e e8 da 87 fe ff <0f> 0b 48 89 f2 48 89 fe 48 c7 c7 78 43 38 8e e8 c6 87 fe ff 0f 0b [ 3538.365641] RSP: 0018:ffffae870073fda0 EFLAGS: 00010246 [ 3538.365652] RAX: 0000000000000054 RBX: ffff8d2b75997800 RCX: 0000000000000000 [ 3538.365668] RDX: 0000000000000000 RSI: ffff8d2e2ee20840 RDI: ffff8d2e2ee20840 [ 3538.365681] RBP: ffff8d2b75997ab8 R08: 80000000ffff842f R09: 0000000000ffff10 [ 3538.365693] R10: 000000000000000f R11: 000000000000000f R12: 00000000ffffc032 [ 3538.365705] R13: ffff8d2b75997c80 R14: ffff8d2ac480b800 R15: ffff8d2ac480b9c8 [ 3538.365717] FS: 00007f9be42097c0(0000) GS:ffff8d2e2ee00000(0000) knlGS:0000000000000000 [ 3538.365733] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3538.365744] CR2: 00007fa97dc5398c CR3: 0000000104248004 CR4: 00000000007706f0 [ 3538.365756] PKRU: 55555554 [ 3538.365761] Call Trace: [ 3538.365768] <TASK> [ 3538.365774] ? show_trace_log_lvl+0x1c4/0x2df [ 3538.365785] ? show_trace_log_lvl+0x1c4/0x2df [ 3538.365796] ? ceph_drop_caps_for_unlink+0xb8/0x170 [ceph] [ 3538.365828] ? __die_body.cold+0x8/0xd [ 3538.365836] ? die+0x2b/0x50 [ 3538.365845] ? do_trap+0xce/0x120 [ 3538.365853] ? __list_del_entry_valid.cold+0x1d/0x47 [ 3538.365863] ? do_error_trap+0x65/0x80 [ 3538.365871] ? __list_del_entry_valid.cold+0x1d/0x47 [ 3538.365881] ? exc_invalid_op+0x4e/0x70 [ 3538.365891] ? __list_del_entry_valid.cold+0x1d/0x47 [ 3538.365901] ? asm_exc_invalid_op+0x16/0x20 [ 3538.365912] ? __list_del_entry_valid.cold+0x1d/0x47 [ 3538.365923] ceph_drop_caps_for_unlink+0xb8/0x170 [ceph] [ 3538.365947] ceph_unlink+0xed/0x450 [ceph] [ 3538.365970] vfs_unlink+0x114/0x290 [ 3538.365980] do_unlinkat+0x1af/0x2e0 [ 3538.365990] __x64_sys_unlink+0x3e/0x60 [ 3538.365999] do_syscall_64+0x59/0x90 [ 3538.366008] ? syscall_exit_to_user_mode+0x22/0x40 [ 3538.366018] ? do_syscall_64+0x69/0x90 [ 3538.366027] ? do_syscall_64+0x69/0x90 [ 3538.366035] entry_SYSCALL_64_after_hwframe+0x72/0xdc [ 3538.366046] RIP: 0033:0x7f9be40ff27b [ 3538.366069] Code: f0 ff ff 73 01 c3 48 8b 0d a2 ab 0f 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 75 ab 0f 00 f7 d8 64 89 01 48 [ 3538.367031] RSP: 002b:00007ffd8640de58 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3538.367576] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 00007f9be40ff27b [ 3538.368116] RDX: 0000000000000007 RSI: 0000000000000001 RDI: 00007f9bdd4af698 [ 3538.368646] RBP: 00007f9bdd4af698 R08: 00000000ffffffc9 R09: 0000000000000038 [ 3538.369156] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 3538.369671] R13: 00007f9bdd4af698 R14: 0000000000000001 R15: 00007f9be3c15290 [ 3538.370182] </TASK> [ 3538.370682] Modules linked in: ceph libceph dns_resolver fscache netfs nft_counter ipt_REJECT xt_owner xt_conntrack nft_compat nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 rfkill ip_set nf_tables libcrc32c nfnetlink vfat fat intel_rapl_msr intel_rapl_common intel_uncore_frequency_common virtio_gpu virtio_dma_buf drm_shmem_helper isst_if_common drm_kms_helper nfit syscopyarea sysfillrect sysimgblt fb_sys_fops libnvdimm i2c_i801 iTCO_wdt iTCO_vendor_support lpc_ich i2c_smbus virtio_balloon rapl joydev pcspkr drm fuse ext4 mbcache jbd2 sr_mod cdrom sg ahci libahci crct10dif_pclmul crc32_pclmul crc32c_intel libata ghash_clmulni_intel virtio_net virtio_blk virtio_console net_failover virtio_scsi failover serio_raw ``` I checked the changelogs of the kernel versions and spotted these three commits that were backported: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?i... https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?i... https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?i... The first one adds changes that look related. Does anybody have experienced this as well or know something about this? Thanks and best regards, Marc
I would like to ask for help regarding client kernel crashes that happen on cephfs access. We have been struggling with this for over a month now with over 100 crashes on 7 hosts during that time.
Our cluster runs version 18.2.1. Our clients run CentOS Stream.
On CentOS Stream 9 the problem started with kernel version 5.14.0-425.el9. Version 5.14.0-419.el9 is the last one without problems. It also occurred on CentOS Stream 8, starting with version 4.18.0-546.el8 (4.18.0-544.el8 being the last good one).
The problem presents itself by the client kernel crashing, forcing a reboot of the machine. Apparently it is triggered by a certain level of IO on the cephfs mount. It works perfectly fine when we rollback to the last good kernel version.
The exact call trace in vmcore-dmesg.txt differs between occurrences. Here are two typical examples:
``` [ 8641.382499] list_del corruption. next->prev should be ffff88bd0a4d4c80, but was ffff88bcefdfd280 [ 8641.382521] ------------[ cut here ]------------ [ 8641.382521] kernel BUG at lib/list_debug.c:54! [ 8641.382528] invalid opcode: 0000 [#1] PREEMPT SMP NOPTI [ 8641.382591] CPU: 2 PID: 83929 Comm: kworker/2:0 Kdump: loaded Not tainted 5.14.0-432.el9.x86_64 #1 [ 8641.382610] Hardware name: oVirt RHEL/RHEL-AV, BIOS edk2-20230524- 4.el9_3 05/24/2023 [ 8641.382624] Workqueue: ceph-cap ceph_cap_unlink_work [ceph] [ 8641.382662] RIP: 0010:__list_del_entry_valid.cold+0x1d/0x47 [ 8641.382681] Code: c7 c7 78 42 d8 b1 e8 f9 87 fe ff 0f 0b 48 89 fe 48 c7 c7 08 43 d8 b1 e8 e8 87 fe ff 0f 0b 48 c7 c7 b8 43 d8 b1 e8 da 87 fe ff <0f> 0b 48 89 f2 48 89 fe 48 c7 c7 78 43 d8 b1 e8 c6 87 fe ff 0f 0b [ 8641.382711] RSP: 0018:ffff95a000d6be60 EFLAGS: 00010246 [ 8641.382722] RAX: 0000000000000054 RBX: ffff88bced76dc00 RCX: 0000000000000000 [ 8641.382734] RDX: 0000000000000000 RSI: ffff88c02eea0840 RDI: ffff88c02eea0840 [ 8641.382746] RBP: ffff88bd0a4d4c80 R08: 80000000ffff8434 R09: 0000000000ffff10 [ 8641.382758] R10: 000000000000000f R11: 000000000000000f R12: ffff88c02eeb2800 [ 8641.382779] R13: ffff88bcc4610258 R14: ffff88bcc46101b8 R15: ffff88bcc46101c8 [ 8641.382793] FS: 0000000000000000(0000) GS:ffff88c02ee80000(0000) knlGS:0000000000000000 [ 8641.382809] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8641.382819] CR2: 00007f35cee8a000 CR3: 0000000105708004 CR4: 00000000007706e0 [ 8641.382832] PKRU: 55555554 [ 8641.382838] Call Trace: [ 8641.382844] <TASK> [ 8641.382850] ? show_trace_log_lvl+0x1c4/0x2df [ 8641.382860] ? show_trace_log_lvl+0x1c4/0x2df [ 8641.382870] ? ceph_cap_unlink_work+0x3f/0x140 [ceph] [ 8641.382893] ? __die_body.cold+0x8/0xd [ 8641.382902] ? die+0x2b/0x50 [ 8641.382911] ? do_trap+0xce/0x120 [ 8641.382919] ? __list_del_entry_valid.cold+0x1d/0x47 [ 8641.382930] ? do_error_trap+0x65/0x80 [ 8641.382938] ? __list_del_entry_valid.cold+0x1d/0x47 [ 8641.382948] ? exc_invalid_op+0x4e/0x70 [ 8641.382958] ? __list_del_entry_valid.cold+0x1d/0x47 [ 8641.382975] ? asm_exc_invalid_op+0x16/0x20 [ 8641.382988] ? __list_del_entry_valid.cold+0x1d/0x47 [ 8641.382998] ceph_cap_unlink_work+0x3f/0x140 [ceph] [ 8641.383021] process_one_work+0x1e2/0x3b0 [ 8641.383032] ? __pfx_worker_thread+0x10/0x10 [ 8641.383043] worker_thread+0x50/0x3a0 [ 8641.383051] ? __pfx_worker_thread+0x10/0x10 [ 8641.383061] kthread+0xdd/0x100 [ 8641.383069] ? __pfx_kthread+0x10/0x10 [ 8641.383078] ret_from_fork+0x29/0x50 [ 8641.383090] </TASK> [ 8641.383095] Modules linked in: tls ceph libceph dns_resolver fscache netfs nft_counter ipt_REJECT xt_owner xt_conntrack nft_compat nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 rfkill ip_set nf_tables libcrc32c nfnetlink vfat fat intel_rapl_msr intel_rapl_common intel_uncore_frequency_common isst_if_common nfit virtio_gpu iTCO_wdt iTCO_vendor_support libnvdimm lpc_ich virtio_dma_buf drm_shmem_helper drm_kms_helper i2c_i801 rapl syscopyarea sysfillrect sysimgblt virtio_balloon fb_sys_fops i2c_smbus pcspkr joydev fuse drm ext4 mbcache jbd2 sr_mod cdrom sd_mod ahci t10_pi sg libahci crct10dif_pclmul crc32_pclmul crc32c_intel libata ghash_clmulni_intel virtio_net virtio_console virtio_scsi net_failover failover serio_raw ```
``` [ 3538.365469] list_del corruption. next->prev should be ffff8d2b75997c80, but was ffff8d2afcfaae80 [ 3538.365488] ------------[ cut here ]------------ [ 3538.365488] kernel BUG at lib/list_debug.c:54! [ 3538.365493] invalid opcode: 0000 [#1] PREEMPT SMP NOPTI [ 3538.365553] CPU: 0 PID: 910 Comm: php-fpm Kdump: loaded Not tainted 5.14.0-432.el9.x86_64 #1 [ 3538.365569] Hardware name: oVirt RHEL/RHEL-AV, BIOS edk2-20230524- 4.el9_3 05/24/2023 [ 3538.365582] RIP: 0010:__list_del_entry_valid.cold+0x1d/0x47 [ 3538.365612] Code: c7 c7 78 42 38 8e e8 f9 87 fe ff 0f 0b 48 89 fe 48 c7 c7 08 43 38 8e e8 e8 87 fe ff 0f 0b 48 c7 c7 b8 43 38 8e e8 da 87 fe ff <0f> 0b 48 89 f2 48 89 fe 48 c7 c7 78 43 38 8e e8 c6 87 fe ff 0f 0b [ 3538.365641] RSP: 0018:ffffae870073fda0 EFLAGS: 00010246 [ 3538.365652] RAX: 0000000000000054 RBX: ffff8d2b75997800 RCX: 0000000000000000 [ 3538.365668] RDX: 0000000000000000 RSI: ffff8d2e2ee20840 RDI: ffff8d2e2ee20840 [ 3538.365681] RBP: ffff8d2b75997ab8 R08: 80000000ffff842f R09: 0000000000ffff10 [ 3538.365693] R10: 000000000000000f R11: 000000000000000f R12: 00000000ffffc032 [ 3538.365705] R13: ffff8d2b75997c80 R14: ffff8d2ac480b800 R15: ffff8d2ac480b9c8 [ 3538.365717] FS: 00007f9be42097c0(0000) GS:ffff8d2e2ee00000(0000) knlGS:0000000000000000 [ 3538.365733] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3538.365744] CR2: 00007fa97dc5398c CR3: 0000000104248004 CR4: 00000000007706f0 [ 3538.365756] PKRU: 55555554 [ 3538.365761] Call Trace: [ 3538.365768] <TASK> [ 3538.365774] ? show_trace_log_lvl+0x1c4/0x2df [ 3538.365785] ? show_trace_log_lvl+0x1c4/0x2df [ 3538.365796] ? ceph_drop_caps_for_unlink+0xb8/0x170 [ceph] [ 3538.365828] ? __die_body.cold+0x8/0xd [ 3538.365836] ? die+0x2b/0x50 [ 3538.365845] ? do_trap+0xce/0x120 [ 3538.365853] ? __list_del_entry_valid.cold+0x1d/0x47 [ 3538.365863] ? do_error_trap+0x65/0x80 [ 3538.365871] ? __list_del_entry_valid.cold+0x1d/0x47 [ 3538.365881] ? exc_invalid_op+0x4e/0x70 [ 3538.365891] ? __list_del_entry_valid.cold+0x1d/0x47 [ 3538.365901] ? asm_exc_invalid_op+0x16/0x20 [ 3538.365912] ? __list_del_entry_valid.cold+0x1d/0x47 [ 3538.365923] ceph_drop_caps_for_unlink+0xb8/0x170 [ceph] [ 3538.365947] ceph_unlink+0xed/0x450 [ceph] [ 3538.365970] vfs_unlink+0x114/0x290 [ 3538.365980] do_unlinkat+0x1af/0x2e0 [ 3538.365990] __x64_sys_unlink+0x3e/0x60 [ 3538.365999] do_syscall_64+0x59/0x90 [ 3538.366008] ? syscall_exit_to_user_mode+0x22/0x40 [ 3538.366018] ? do_syscall_64+0x69/0x90 [ 3538.366027] ? do_syscall_64+0x69/0x90 [ 3538.366035] entry_SYSCALL_64_after_hwframe+0x72/0xdc [ 3538.366046] RIP: 0033:0x7f9be40ff27b [ 3538.366069] Code: f0 ff ff 73 01 c3 48 8b 0d a2 ab 0f 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 75 ab 0f 00 f7 d8 64 89 01 48 [ 3538.367031] RSP: 002b:00007ffd8640de58 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3538.367576] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 00007f9be40ff27b [ 3538.368116] RDX: 0000000000000007 RSI: 0000000000000001 RDI: 00007f9bdd4af698 [ 3538.368646] RBP: 00007f9bdd4af698 R08: 00000000ffffffc9 R09: 0000000000000038 [ 3538.369156] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 3538.369671] R13: 00007f9bdd4af698 R14: 0000000000000001 R15: 00007f9be3c15290 [ 3538.370182] </TASK> [ 3538.370682] Modules linked in: ceph libceph dns_resolver fscache netfs nft_counter ipt_REJECT xt_owner xt_conntrack nft_compat nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 rfkill ip_set nf_tables libcrc32c nfnetlink vfat fat intel_rapl_msr intel_rapl_common intel_uncore_frequency_common virtio_gpu virtio_dma_buf drm_shmem_helper isst_if_common drm_kms_helper nfit syscopyarea sysfillrect sysimgblt fb_sys_fops libnvdimm i2c_i801 iTCO_wdt iTCO_vendor_support lpc_ich i2c_smbus virtio_balloon rapl joydev pcspkr drm fuse ext4 mbcache jbd2 sr_mod cdrom sg ahci libahci crct10dif_pclmul crc32_pclmul crc32c_intel libata ghash_clmulni_intel virtio_net virtio_blk virtio_console net_failover virtio_scsi failover serio_raw ```
I checked the changelogs of the kernel versions and spotted these three commits that were backported:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit /?id=dbc347ef7f0c53aa4a5383238a804d7ebbb0b5ca https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit /?id=902d6d013f75b68f31d208c6f3ff9cdca82648a7 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit /?id=07045648c07c5632e0dfd5ce084d3cd0cec0258a
The first one adds changes that look related.
Does anybody have experienced this as well or know something about this?
I have a guaranteed crash + reboot with el7 - nautilus accessing a snapshot. rbd snap ls vps-xxx -p rbd rbd map vps-xxx@vps-xxx.bak1 -p rbd some lvm stuff like this (pvscan --cache; pvs; lvchange -a y VGxxx/LVyyy) mount -o ro /dev/mapper/VGxxx-LVyyy /mnt/disk <---- C04 CRASH!!!! I bypass this now with creating a clone of the snapshot.
On Mon, Apr 8, 2024 at 10:22 AM Marc <Marc@f1-outsourcing.eu> wrote:
I have a guaranteed crash + reboot with el7 - nautilus accessing a snapshot.
rbd snap ls vps-xxx -p rbd rbd map vps-xxx@vps-xxx.bak1 -p rbd
some lvm stuff like this (pvscan --cache; pvs; lvchange -a y VGxxx/LVyyy)
mount -o ro /dev/mapper/VGxxx-LVyyy /mnt/disk <---- C04 CRASH!!!!
I bypass this now with creating a clone of the snapshot.
Hi Marc, I saw you report this in at least one other thread on this list but you never provided any details. With only RBD involved, it's definitely not related to the CephFS issue raised by Marc R here. I suspect that you are hitting https://tracker.ceph.com/issues/39010, suffering from the fact that for historical reasons the block layer has to be somewhat lax with read-only devices and that the related RBD fixup hasn't made it into RHEL 7. Thanks, Ilya
Hi Marc, Thanks for reporting this, I generated one patch to fix it. Will send it out after testing is done. - Xiubo On 4/8/24 16:01, Marc Ruhmann wrote:
Hi everyone,
I would like to ask for help regarding client kernel crashes that happen on cephfs access. We have been struggling with this for over a month now with over 100 crashes on 7 hosts during that time.
Our cluster runs version 18.2.1. Our clients run CentOS Stream.
On CentOS Stream 9 the problem started with kernel version 5.14.0-425.el9. Version 5.14.0-419.el9 is the last one without problems. It also occurred on CentOS Stream 8, starting with version 4.18.0-546.el8 (4.18.0-544.el8 being the last good one).
The problem presents itself by the client kernel crashing, forcing a reboot of the machine. Apparently it is triggered by a certain level of IO on the cephfs mount. It works perfectly fine when we rollback to the last good kernel version.
The exact call trace in vmcore-dmesg.txt differs between occurrences. Here are two typical examples:
``` [ 8641.382499] list_del corruption. next->prev should be ffff88bd0a4d4c80, but was ffff88bcefdfd280 [ 8641.382521] ------------[ cut here ]------------ [ 8641.382521] kernel BUG at lib/list_debug.c:54! [ 8641.382528] invalid opcode: 0000 [#1] PREEMPT SMP NOPTI [ 8641.382591] CPU: 2 PID: 83929 Comm: kworker/2:0 Kdump: loaded Not tainted 5.14.0-432.el9.x86_64 #1 [ 8641.382610] Hardware name: oVirt RHEL/RHEL-AV, BIOS edk2-20230524-4.el9_3 05/24/2023 [ 8641.382624] Workqueue: ceph-cap ceph_cap_unlink_work [ceph] [ 8641.382662] RIP: 0010:__list_del_entry_valid.cold+0x1d/0x47 [ 8641.382681] Code: c7 c7 78 42 d8 b1 e8 f9 87 fe ff 0f 0b 48 89 fe 48 c7 c7 08 43 d8 b1 e8 e8 87 fe ff 0f 0b 48 c7 c7 b8 43 d8 b1 e8 da 87 fe ff <0f> 0b 48 89 f2 48 89 fe 48 c7 c7 78 43 d8 b1 e8 c6 87 fe ff 0f 0b [ 8641.382711] RSP: 0018:ffff95a000d6be60 EFLAGS: 00010246 [ 8641.382722] RAX: 0000000000000054 RBX: ffff88bced76dc00 RCX: 0000000000000000 [ 8641.382734] RDX: 0000000000000000 RSI: ffff88c02eea0840 RDI: ffff88c02eea0840 [ 8641.382746] RBP: ffff88bd0a4d4c80 R08: 80000000ffff8434 R09: 0000000000ffff10 [ 8641.382758] R10: 000000000000000f R11: 000000000000000f R12: ffff88c02eeb2800 [ 8641.382779] R13: ffff88bcc4610258 R14: ffff88bcc46101b8 R15: ffff88bcc46101c8 [ 8641.382793] FS: 0000000000000000(0000) GS:ffff88c02ee80000(0000) knlGS:0000000000000000 [ 8641.382809] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 8641.382819] CR2: 00007f35cee8a000 CR3: 0000000105708004 CR4: 00000000007706e0 [ 8641.382832] PKRU: 55555554 [ 8641.382838] Call Trace: [ 8641.382844] <TASK> [ 8641.382850] ? show_trace_log_lvl+0x1c4/0x2df [ 8641.382860] ? show_trace_log_lvl+0x1c4/0x2df [ 8641.382870] ? ceph_cap_unlink_work+0x3f/0x140 [ceph] [ 8641.382893] ? __die_body.cold+0x8/0xd [ 8641.382902] ? die+0x2b/0x50 [ 8641.382911] ? do_trap+0xce/0x120 [ 8641.382919] ? __list_del_entry_valid.cold+0x1d/0x47 [ 8641.382930] ? do_error_trap+0x65/0x80 [ 8641.382938] ? __list_del_entry_valid.cold+0x1d/0x47 [ 8641.382948] ? exc_invalid_op+0x4e/0x70 [ 8641.382958] ? __list_del_entry_valid.cold+0x1d/0x47 [ 8641.382975] ? asm_exc_invalid_op+0x16/0x20 [ 8641.382988] ? __list_del_entry_valid.cold+0x1d/0x47 [ 8641.382998] ceph_cap_unlink_work+0x3f/0x140 [ceph] [ 8641.383021] process_one_work+0x1e2/0x3b0 [ 8641.383032] ? __pfx_worker_thread+0x10/0x10 [ 8641.383043] worker_thread+0x50/0x3a0 [ 8641.383051] ? __pfx_worker_thread+0x10/0x10 [ 8641.383061] kthread+0xdd/0x100 [ 8641.383069] ? __pfx_kthread+0x10/0x10 [ 8641.383078] ret_from_fork+0x29/0x50 [ 8641.383090] </TASK> [ 8641.383095] Modules linked in: tls ceph libceph dns_resolver fscache netfs nft_counter ipt_REJECT xt_owner xt_conntrack nft_compat nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 rfkill ip_set nf_tables libcrc32c nfnetlink vfat fat intel_rapl_msr intel_rapl_common intel_uncore_frequency_common isst_if_common nfit virtio_gpu iTCO_wdt iTCO_vendor_support libnvdimm lpc_ich virtio_dma_buf drm_shmem_helper drm_kms_helper i2c_i801 rapl syscopyarea sysfillrect sysimgblt virtio_balloon fb_sys_fops i2c_smbus pcspkr joydev fuse drm ext4 mbcache jbd2 sr_mod cdrom sd_mod ahci t10_pi sg libahci crct10dif_pclmul crc32_pclmul crc32c_intel libata ghash_clmulni_intel virtio_net virtio_console virtio_scsi net_failover failover serio_raw ```
``` [ 3538.365469] list_del corruption. next->prev should be ffff8d2b75997c80, but was ffff8d2afcfaae80 [ 3538.365488] ------------[ cut here ]------------ [ 3538.365488] kernel BUG at lib/list_debug.c:54! [ 3538.365493] invalid opcode: 0000 [#1] PREEMPT SMP NOPTI [ 3538.365553] CPU: 0 PID: 910 Comm: php-fpm Kdump: loaded Not tainted 5.14.0-432.el9.x86_64 #1 [ 3538.365569] Hardware name: oVirt RHEL/RHEL-AV, BIOS edk2-20230524-4.el9_3 05/24/2023 [ 3538.365582] RIP: 0010:__list_del_entry_valid.cold+0x1d/0x47 [ 3538.365612] Code: c7 c7 78 42 38 8e e8 f9 87 fe ff 0f 0b 48 89 fe 48 c7 c7 08 43 38 8e e8 e8 87 fe ff 0f 0b 48 c7 c7 b8 43 38 8e e8 da 87 fe ff <0f> 0b 48 89 f2 48 89 fe 48 c7 c7 78 43 38 8e e8 c6 87 fe ff 0f 0b [ 3538.365641] RSP: 0018:ffffae870073fda0 EFLAGS: 00010246 [ 3538.365652] RAX: 0000000000000054 RBX: ffff8d2b75997800 RCX: 0000000000000000 [ 3538.365668] RDX: 0000000000000000 RSI: ffff8d2e2ee20840 RDI: ffff8d2e2ee20840 [ 3538.365681] RBP: ffff8d2b75997ab8 R08: 80000000ffff842f R09: 0000000000ffff10 [ 3538.365693] R10: 000000000000000f R11: 000000000000000f R12: 00000000ffffc032 [ 3538.365705] R13: ffff8d2b75997c80 R14: ffff8d2ac480b800 R15: ffff8d2ac480b9c8 [ 3538.365717] FS: 00007f9be42097c0(0000) GS:ffff8d2e2ee00000(0000) knlGS:0000000000000000 [ 3538.365733] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3538.365744] CR2: 00007fa97dc5398c CR3: 0000000104248004 CR4: 00000000007706f0 [ 3538.365756] PKRU: 55555554 [ 3538.365761] Call Trace: [ 3538.365768] <TASK> [ 3538.365774] ? show_trace_log_lvl+0x1c4/0x2df [ 3538.365785] ? show_trace_log_lvl+0x1c4/0x2df [ 3538.365796] ? ceph_drop_caps_for_unlink+0xb8/0x170 [ceph] [ 3538.365828] ? __die_body.cold+0x8/0xd [ 3538.365836] ? die+0x2b/0x50 [ 3538.365845] ? do_trap+0xce/0x120 [ 3538.365853] ? __list_del_entry_valid.cold+0x1d/0x47 [ 3538.365863] ? do_error_trap+0x65/0x80 [ 3538.365871] ? __list_del_entry_valid.cold+0x1d/0x47 [ 3538.365881] ? exc_invalid_op+0x4e/0x70 [ 3538.365891] ? __list_del_entry_valid.cold+0x1d/0x47 [ 3538.365901] ? asm_exc_invalid_op+0x16/0x20 [ 3538.365912] ? __list_del_entry_valid.cold+0x1d/0x47 [ 3538.365923] ceph_drop_caps_for_unlink+0xb8/0x170 [ceph] [ 3538.365947] ceph_unlink+0xed/0x450 [ceph] [ 3538.365970] vfs_unlink+0x114/0x290 [ 3538.365980] do_unlinkat+0x1af/0x2e0 [ 3538.365990] __x64_sys_unlink+0x3e/0x60 [ 3538.365999] do_syscall_64+0x59/0x90 [ 3538.366008] ? syscall_exit_to_user_mode+0x22/0x40 [ 3538.366018] ? do_syscall_64+0x69/0x90 [ 3538.366027] ? do_syscall_64+0x69/0x90 [ 3538.366035] entry_SYSCALL_64_after_hwframe+0x72/0xdc [ 3538.366046] RIP: 0033:0x7f9be40ff27b [ 3538.366069] Code: f0 ff ff 73 01 c3 48 8b 0d a2 ab 0f 00 f7 d8 64 89 01 48 83 c8 ff c3 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 57 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 75 ab 0f 00 f7 d8 64 89 01 48 [ 3538.367031] RSP: 002b:00007ffd8640de58 EFLAGS: 00000246 ORIG_RAX: 0000000000000057 [ 3538.367576] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 00007f9be40ff27b [ 3538.368116] RDX: 0000000000000007 RSI: 0000000000000001 RDI: 00007f9bdd4af698 [ 3538.368646] RBP: 00007f9bdd4af698 R08: 00000000ffffffc9 R09: 0000000000000038 [ 3538.369156] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 3538.369671] R13: 00007f9bdd4af698 R14: 0000000000000001 R15: 00007f9be3c15290 [ 3538.370182] </TASK> [ 3538.370682] Modules linked in: ceph libceph dns_resolver fscache netfs nft_counter ipt_REJECT xt_owner xt_conntrack nft_compat nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 rfkill ip_set nf_tables libcrc32c nfnetlink vfat fat intel_rapl_msr intel_rapl_common intel_uncore_frequency_common virtio_gpu virtio_dma_buf drm_shmem_helper isst_if_common drm_kms_helper nfit syscopyarea sysfillrect sysimgblt fb_sys_fops libnvdimm i2c_i801 iTCO_wdt iTCO_vendor_support lpc_ich i2c_smbus virtio_balloon rapl joydev pcspkr drm fuse ext4 mbcache jbd2 sr_mod cdrom sg ahci libahci crct10dif_pclmul crc32_pclmul crc32c_intel libata ghash_clmulni_intel virtio_net virtio_blk virtio_console net_failover virtio_scsi failover serio_raw ```
I checked the changelogs of the kernel versions and spotted these three commits that were backported:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?i...
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?i...
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?i...
The first one adds changes that look related.
Does anybody have experienced this as well or know something about this?
Thanks and best regards,
Marc
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi,
On 9 Apr 2024, at 04:07, Xiubo Li <xiubli@redhat.com> wrote:
Thanks for reporting this, I generated one patch to fix it. Will send it out after testing is done.
Trace from our users, but from mainline kernel. Look like as trace above kernel: ------------[ cut here ]------------ kernel: list_add corruption. next->prev should be prev (ffff8fb9c22139b8), but was ffff8fb9c18910c0. (next=ffff8fb9c18910c0). kernel: WARNING: CPU: 2 PID: 861985 at lib/list_debug.c:29 __list_add_valid_or_report+0x6a/0xa0 kernel: Modules linked in: cbc ceph libceph libcrc32c dns_resolver netfs intel_rapl_msr intel_rapl_common crct10dif_pclmul crc32_pclmul polyval_clmulni polyval_generic gf128mul ghash_clmulni_intel sha512_ssse3 sha256_ssse3 sha1_ssse3 cfg80211 aesni_intel crypto_simd cryptd rapl rfkill joydev mousedev psmouse i2c_piix4 pcspkr mac_hid dm_mod fuse loop nfnetlink qemu_fw_cfg ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 virtio_net net_failover virtio_rng failover virtio_scsi virtio_balloon virtio_console hid_generic usbhid sr_mod cdrom serio_raw ata_generic atkbd pata_acpi virtio_pci dr> kernel: CPU: 2 PID: 861985 Comm: kworker/u36:4 Tainted: G D W 6.8.4-arch1-1 #1 7ea0d8fced45b5f098eb034690645970f116c34c kernel: Hardware name: oVirt RHEL, BIOS 1.14.0-1.el8 04/01/2014 kernel: Workqueue: writeback wb_workfn (flush-ceph-1) kernel: RIP: 0010:__list_add_valid_or_report+0x6a/0xa0 kernel: Code: a4 ff 0f 0b 31 c0 c3 cc cc cc cc 48 c7 c7 40 95 a6 8e e8 d9 31 a4 ff 0f 0b eb e9 48 89 c1 48 c7 c7 68 95 a6 8e e8 c6 31 a4 ff <0f> 0b eb d6 48 89 d1 48 89 c6 4c 89 c2 48 c7 c7 b8 95 a6 8e e8 ad kernel: RSP: 0018:ffffafb5c4e03b70 EFLAGS: 00010286 kernel: RAX: 0000000000000000 RBX: ffff8fb9f1f7e820 RCX: 0000000000000027 kernel: RDX: ffff8fbaf7aa19c8 RSI: 0000000000000001 RDI: ffff8fbaf7aa19c0 kernel: RBP: ffff8fb9f1f7ead8 R08: 0000000000000000 R09: ffffafb5c4e03a00 kernel: R10: ffffafb5c4e039f8 R11: 0000000000000003 R12: ffff8fb9f1f7eca0 kernel: R13: ffff8fb9c2213800 R14: ffff8fb9c22139c8 R15: 0000000000000001 kernel: FS: 0000000000000000(0000) GS:ffff8fbaf7a80000(0000) knlGS:0000000000000000 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 kernel: CR2: 00007dcc9182fd08 CR3: 000000010bc78005 CR4: 00000000003706f0 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 kernel: Call Trace: kernel: <TASK> kernel: ? __list_add_valid_or_report+0x6a/0xa0 kernel: ? __warn+0x81/0x130 kernel: ? __list_add_valid_or_report+0x6a/0xa0 kernel: ? report_bug+0x171/0x1a0 kernel: ? console_unlock+0x78/0x120 kernel: ? handle_bug+0x3c/0x80 kernel: ? exc_invalid_op+0x17/0x70 kernel: ? asm_exc_invalid_op+0x1a/0x20 kernel: ? __list_add_valid_or_report+0x6a/0xa0 kernel: ceph_write_inode+0x35b/0x390 [ceph e22e5e8fbb2e717a1d900c81c3b686cc119b9d81] kernel: ? do_writepages+0x1bb/0x1e0 kernel: __writeback_single_inode+0x2a9/0x360 kernel: writeback_sb_inodes+0x1ed/0x4b0 kernel: __writeback_inodes_wb+0x4c/0xf0 kernel: wb_writeback+0x298/0x310 kernel: wb_workfn+0x368/0x510 kernel: process_one_work+0x178/0x350 kernel: worker_thread+0x30f/0x450 kernel: ? __pfx_worker_thread+0x10/0x10 kernel: kthread+0xe5/0x120 kernel: ? __pfx_kthread+0x10/0x10 kernel: ret_from_fork+0x31/0x50 kernel: ? __pfx_kthread+0x10/0x10 kernel: ret_from_fork_asm+0x1b/0x30 kernel: </TASK> kernel: ---[ end trace 0000000000000000 ]--- k
Hello, is there any estimation when seastore and crimson will become production ready? BR Reza
Hi Konstantin, We have fixed it, please see https://patchwork.kernel.org/project/ceph-devel/list/?series=842682&archive=both. - Xiubo On 4/18/24 00:05, Konstantin Shalygin wrote:
Hi,
On 9 Apr 2024, at 04:07, Xiubo Li <xiubli@redhat.com> wrote:
Thanks for reporting this, I generated one patch to fix it. Will send it out after testing is done. Trace from our users, but from mainline kernel. Look like as trace above
kernel: ------------[ cut here ]------------ kernel: list_add corruption. next->prev should be prev (ffff8fb9c22139b8), but was ffff8fb9c18910c0. (next=ffff8fb9c18910c0). kernel: WARNING: CPU: 2 PID: 861985 at lib/list_debug.c:29 __list_add_valid_or_report+0x6a/0xa0 kernel: Modules linked in: cbc ceph libceph libcrc32c dns_resolver netfs intel_rapl_msr intel_rapl_common crct10dif_pclmul crc32_pclmul polyval_clmulni polyval_generic gf128mul ghash_clmulni_intel sha512_ssse3 sha256_ssse3 sha1_ssse3 cfg80211 aesni_intel crypto_simd cryptd rapl rfkill joydev mousedev psmouse i2c_piix4 pcspkr mac_hid dm_mod fuse loop nfnetlink qemu_fw_cfg ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 virtio_net net_failover virtio_rng failover virtio_scsi virtio_balloon virtio_console hid_generic usbhid sr_mod cdrom serio_raw ata_generic atkbd pata_acpi virtio_pci dr> kernel: CPU: 2 PID: 861985 Comm: kworker/u36:4 Tainted: G D W 6.8.4-arch1-1 #1 7ea0d8fced45b5f098eb034690645970f116c34c kernel: Hardware name: oVirt RHEL, BIOS 1.14.0-1.el8 04/01/2014 kernel: Workqueue: writeback wb_workfn (flush-ceph-1) kernel: RIP: 0010:__list_add_valid_or_report+0x6a/0xa0 kernel: Code: a4 ff 0f 0b 31 c0 c3 cc cc cc cc 48 c7 c7 40 95 a6 8e e8 d9 31 a4 ff 0f 0b eb e9 48 89 c1 48 c7 c7 68 95 a6 8e e8 c6 31 a4 ff <0f> 0b eb d6 48 89 d1 48 89 c6 4c 89 c2 48 c7 c7 b8 95 a6 8e e8 ad kernel: RSP: 0018:ffffafb5c4e03b70 EFLAGS: 00010286 kernel: RAX: 0000000000000000 RBX: ffff8fb9f1f7e820 RCX: 0000000000000027 kernel: RDX: ffff8fbaf7aa19c8 RSI: 0000000000000001 RDI: ffff8fbaf7aa19c0 kernel: RBP: ffff8fb9f1f7ead8 R08: 0000000000000000 R09: ffffafb5c4e03a00 kernel: R10: ffffafb5c4e039f8 R11: 0000000000000003 R12: ffff8fb9f1f7eca0 kernel: R13: ffff8fb9c2213800 R14: ffff8fb9c22139c8 R15: 0000000000000001 kernel: FS: 0000000000000000(0000) GS:ffff8fbaf7a80000(0000) knlGS:0000000000000000 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 kernel: CR2: 00007dcc9182fd08 CR3: 000000010bc78005 CR4: 00000000003706f0 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 kernel: Call Trace: kernel: <TASK> kernel: ? __list_add_valid_or_report+0x6a/0xa0 kernel: ? __warn+0x81/0x130 kernel: ? __list_add_valid_or_report+0x6a/0xa0 kernel: ? report_bug+0x171/0x1a0 kernel: ? console_unlock+0x78/0x120 kernel: ? handle_bug+0x3c/0x80 kernel: ? exc_invalid_op+0x17/0x70 kernel: ? asm_exc_invalid_op+0x1a/0x20 kernel: ? __list_add_valid_or_report+0x6a/0xa0 kernel: ceph_write_inode+0x35b/0x390 [ceph e22e5e8fbb2e717a1d900c81c3b686cc119b9d81] kernel: ? do_writepages+0x1bb/0x1e0 kernel: __writeback_single_inode+0x2a9/0x360 kernel: writeback_sb_inodes+0x1ed/0x4b0 kernel: __writeback_inodes_wb+0x4c/0xf0 kernel: wb_writeback+0x298/0x310 kernel: wb_workfn+0x368/0x510 kernel: process_one_work+0x178/0x350 kernel: worker_thread+0x30f/0x450 kernel: ? __pfx_worker_thread+0x10/0x10 kernel: kthread+0xe5/0x120 kernel: ? __pfx_kthread+0x10/0x10 kernel: ret_from_fork+0x31/0x50 kernel: ? __pfx_kthread+0x10/0x10 kernel: ret_from_fork_asm+0x1b/0x30 kernel: </TASK> kernel: ---[ end trace 0000000000000000 ]---
k _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Xiubo, Seems patch already landed to kernel 6.8.7, thanks! k Sent from my iPhone
On 18 Apr 2024, at 05:31, Xiubo Li <xiubli@redhat.com> wrote:
Hi Konstantin,
We have fixed it, please see https://patchwork.kernel.org/project/ceph-devel/list/?series=842682&archive=both.
- Xiubo
participants (6)
-
Ilya Dryomov
-
Konstantin Shalygin
-
Marc
-
Marc Ruhmann
-
R A
-
Xiubo Li