- ceph-users - lists.ceph.io

Re: tentacle and libvirt
by Alfonso Escribano Merino 12 Feb '26

12 Feb '26

1 0

Re: Data is still actively corrupted!
by Dominik Baack 12 Feb '26

12 Feb '26

Hi, I noticed now that files are still actively corrupted / replaced by empty files when open saved. Access was done via Ceph 19.2.1 from Ubuntu 24 with kernel mount. Ceph servers are still running 20.2 Tentacle (deployed via cephadm) with an Ubuntu 24.04 host. Options currently set are to noout, norebalance, noscrub and nodeep-scrub I am currently setting up a read only mount to copy all existing data for a backup. I have currently no clue whats going on, I currently was able to observe this behavior only on the nodes. As those were upgraded as well (MLNX driver, nvidia-fs,...) can it be network? Any Idea how to recover from this? Cheers Dominik Am 11.02.2026 um 18:44 schrieb dominik.baack via ceph-users: > Hi, > > after an controlled shutdown of the whole cluster do to external > circumstances we decided to update from 19.2 to 20.2 after the > restart. The system was health before and after the update. > The nodes mounting the filesystem were not equally lucky and were > partially shutdown hard. Storage was kept running additional ~30min > after node shutdown, all inflight operations should have finished. > > Now we discover the some of the user files seem to be replaced with > zeros. For example: > > stat .gitignore > File: .gitignore > Size: 4429 Blocks: 9 IO Block: 4194304 regular file > Device: 0,48 Inode: 1100241384598 Links: 1 > > > hexdump -C .gitignore > 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 > |................| > * > 00001140 00 00 00 00 00 00 00 00 00 00 00 00 00 |.............| > 0000114d > > > Scanning for files containing only zeros show several issues of files > that were likely accessed before or during the shutdown of the nodes. > > How should I progress from here?

1 0

Excessive OSD memory bloat during Sequential Read (Tentacle v20.2.0, EC 6+2)
by Chanu Yu 11 Feb '26

11 Feb '26

Hi all, I am experiencing a critical memory issue with OSD daemons during sequential read tests. I would appreciate your advice on whether this is a known issue or if there is a flaw in my methodology. Environment Ceph Version: Tentacle v20.2.0Deployment: cephadmOSD Nodes: 8 nodes (128GB RAM each)OSD Configuration: 10 OSDs per node (Total 80 OSDs)Pool Type: Erasure Coding 6+2 (allow_ec_optimizations: true, allow_ec_overwrites: true)Workload: fio Sequential Read (12 clients, 40 jobs/client, BS=4M, QD=128, direct=1)Key Observations Random OSD Spikes: In every test run, a small number of OSD daemons exhibit a massive memory spike. However, these few daemons consume enough memory to exhaust the entire host’s RAM and swap space.Lack of Memory Reclamation: The memory is never reclaimed automatically after the workload stops. It remains occupied until I manually intervene (e.g., by restarting daemons or clearing heap free list of tcmalloc).Production Concern: My primary concern is that if this behavior occurs in a live production environment, it will inevitably lead to severe system-wide thrashing and service instability.Configuration Efforts: I have set osd_memory_target to 10G (with autotune off) and bluestore_cache_size to 7G, but the OSDs continue to ignore these targets during the read workload.Attempted Workarounds TCMALLOC_RELEASE_RATE=1000: No effect.TCMALLOC_HEAP_LIMIT_MB / Container Memory Limits: Problematic OSDs are repeatedly killed by OOM.TCMALLOC_AGGRESSIVE_DECOMMIT=true: Keeps memory stable but causes a 20% performance drop.Bluestore Cache Autotune: Controls memory but results in a 40% performance drop.Questions Is this behavior normal, or is there a flaw in my experimental methodology? Given that the memory usage seems abnormal, I am wondering if my fio configuration or OSD setup is fundamentally misconfigured for this version.Why does this spike occur on random OSDs during reads? Could this be related to how specific OSDs handle EC primary shards or is it a known TCMalloc fragmentation issue in v20.2.0?Is there a way to strictly enforce memory limits without such significant performance degradation?Any insights or suggestions for further debugging would be greatly appreciated. Best regards, Chanu Yu

1 0

Re: reef 18.2.8 QE validation status
by Laura Flores 11 Feb '26

11 Feb '26

The rados suite is approved: https://tracker.ceph.com/projects/rados/wiki/REEF#reef-v1828-httpstrackerce… On Tue, Feb 10, 2026 at 6:33 PM Brad Hubbard via Dev <dev(a)ceph.io> wrote: > Powercycle approved. > > On Tue, Feb 10, 2026 at 10:35 AM Brad Hubbard via Dev <dev(a)ceph.io> wrote: > >> >> >> On Fri, Feb 6, 2026 at 12:36 PM Brad Hubbard <bhubbard(a)redhat.com> wrote: >> >>> >>> >>> On Fri, Feb 6, 2026 at 8:33 AM Brad Hubbard via ceph-users < >>> ceph-users(a)ceph.io> wrote: >>> >>>> On Fri, Feb 6, 2026 at 7:59 AM Yuri Weinstein via ceph-users < >>>> ceph-users(a)ceph.io> wrote: >>>> >>>> > We are continuing on this release. >>>> > >>>> > >>>> > powercycle - FAILED Brad, approved? >>>> > >>>> >>>> Unfortunately, I don't see any logs anywhere where I have access to >>>> them so >>>> I have no way of checking this? >>>> >>> >>> OK, I have the latest logs. The three failing jobs appear to have all >>> timed out after running for about six hours. At this time I don't know why >>> but I continue to investigate. >>> >> >> One point of commonality of the three tests that failed is that they are >> all running tests using ceph fuse. I'll reach out to the FS guys to see if >> they can help me get to the bottom of these failures. >> >> >>> >>> >>>> >>>> >>>> > ceph-volume - Approved Guillaume? >>>> > >>>> > Patrick was reviewing some tests, please see his comments in the same >>>> > tracker. >>>> > TIA >>>> > >>>> > >>>> > On Sat, Dec 13, 2025 at 1:19 AM Ilya Dryomov via ceph-users < >>>> > ceph-users(a)ceph.io> wrote: >>>> > >>>> > > On Mon, Dec 8, 2025 at 11:31 PM Yuri Weinstein via Dev <dev(a)ceph.io >>>> > >>>> > > wrote: >>>> > > > >>>> > > > This test branch was built loner then 2.5 weeks ago >>>> > > > I have to rebuild it first and then will schedule krbd >>>> > > >>>> > > krbd approved based on >>>> > > >>>> > > >>>> > > >>>> > >>>> https://pulpito.ceph.com/yuriw-2025-12-10_21:26:56-krbd-reef-release-testin… >>>> > > >>>> > > >>>> > >>>> https://pulpito.ceph.com/yuriw-2025-12-12_18:20:32-krbd-reef-release-testin… >>>> > > >>>> > > Thanks, >>>> > > >>>> > > Ilya >>>> > > >>>> > > > >>>> > > > On Mon, Dec 8, 2025 at 1:39 PM Ilya Dryomov via ceph-users < >>>> > > ceph-users(a)ceph.io> wrote: >>>> > > >> >>>> > > >> On Mon, Dec 8, 2025 at 8:50 PM Ilya Dryomov <idryomov(a)gmail.com> >>>> > wrote: >>>> > > >> > >>>> > > >> > On Mon, Dec 8, 2025 at 5:00 PM Venky Shankar via ceph-users >>>> > > >> > <ceph-users(a)ceph.io> wrote: >>>> > > >> > > >>>> > > >> > > Hi Yuri, >>>> > > >> > > >>>> > > >> > > On Fri, Nov 21, 2025 at 4:34 AM Yuri Weinstein < >>>> > yweinste(a)redhat.com> >>>> > > wrote: >>>> > > >> > > > >>>> > > >> > > > Note: this is an attempt to test and publish post EOL reef >>>> > point >>>> > > release >>>> > > >> > > > >>>> > > >> > > > Details of this release are summarized here: >>>> > > >> > > > >>>> > > >> > > > https://tracker.ceph.com/issues/73906#note-1 >>>> > > >> > > > Release Notes - TBD >>>> > > >> > > > LRC upgrade - N/A >>>> > > >> > > > >>>> > > >> > > > distro => jammy focal centos9 windows bookworm >>>> > > >> > > > >>>> > > >> > > > Seeking approvals/reviews for: >>>> > > >> > > > >>>> > > >> > > > smoke - Laura approved? >>>> > > >> > > > >>>> > > >> > > > rados - Radek, Laura approved? Travis? Nizamudeen? Adam >>>> King >>>> > > approved? >>>> > > >> > > > (rados is also running on build 2 == build 1 + >>>> > > >> > > > https://github.com/ceph/ceph/pull/66056 per Igor's >>>> request) >>>> > > >> > > > >>>> > > >> > > > rgw - Adam E approved? >>>> > > >> > > > >>>> > > >> > > > fs - Venky approved? >>>> > > >> > > >>>> > > >> > > There are lot many failures in the fs suite resembling >>>> > > >> > > >>>> > > >> > > Error mounting /var/lib/ceph: mount: /var/lib/ceph: >>>> wrong >>>> > fs >>>> > > >> > > type, bad option ... >>>> > > >> > > >>>> > > >> > > which is an issue with the testing kernel. Adding Alex who >>>> was >>>> > > looking >>>> > > >> > > into this issue. >>>> > > >> > > >>>> > > >> > > As far as the run validation is concerned, it's risky to >>>> ignore >>>> > the >>>> > > >> > > large number of failures. We have two options: >>>> > > >> > > >>>> > > >> > > - fix the testing kernel issue and rerun >>>> > > >> > > - rerun with `for-linus` branch (with `-k for-linus` in the >>>> > > >> > > teuthology-suite schedule command) >>>> > > >> > > >>>> > > >> > > The latter would be using an older kernel branch, but I >>>> think it >>>> > > >> > > should suffice for this case. >>>> > > >> > >>>> > > >> > Hi Venky, >>>> > > >> > >>>> > > >> > With your report of the fs suite being affected as well, I >>>> carved >>>> > out >>>> > > >> > some time to take a look. >>>> > > >> > >>>> > > >> > I think it's happening because XFS maintainers disabled >>>> support for >>>> > V4 >>>> > > >> > filesystems that was deprecated 5 years ago -- as of 6.18 >>>> kernel >>>> > it's >>>> > > >> > no longer compiled in by default. The proper fix would be to >>>> create >>>> > > an >>>> > > >> > image with/move to V5 on-disk format; I have opened [1] to try >>>> to >>>> > > >> > achieve that. We can also re-enable V4 on-disk format in our >>>> kernel >>>> > > >> > builds as a workaround. >>>> > > >> > >>>> > > >> > [1] https://github.com/ceph/ceph-cm-ansible/pull/799 >>>> > > >> >>>> > > >> This should be fixed now. Yuri, please schedule a rerun for >>>> krbd. >>>> > > >> >>>> > > >> Thanks, >>>> > > >> >>>> > > >> Ilya >>>> > > >> _______________________________________________ >>>> > > >> ceph-users mailing list -- ceph-users(a)ceph.io >>>> > > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>> > > > >>>> > > > _______________________________________________ >>>> > > > Dev mailing list -- dev(a)ceph.io >>>> > > > To unsubscribe send an email to dev-leave(a)ceph.io >>>> > > _______________________________________________ >>>> > > ceph-users mailing list -- ceph-users(a)ceph.io >>>> > > To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>> > > >>>> > _______________________________________________ >>>> > ceph-users mailing list -- ceph-users(a)ceph.io >>>> > To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>> > >>>> >>>> >>>> -- >>>> Cheers, >>>> Brad >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>>> >>> >>> >>> -- >>> Cheers, >>> Brad >>> >> >> >> -- >> Cheers, >> Brad >> _______________________________________________ >> Dev mailing list -- dev(a)ceph.io >> To unsubscribe send an email to dev-leave(a)ceph.io >> > > > -- > Cheers, > Brad > _______________________________________________ > Dev mailing list -- dev(a)ceph.io > To unsubscribe send an email to dev-leave(a)ceph.io > -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

1 0

Re: Kernel NULL dereference in ceph_mds_auth_match (Linux 6.17, CephFS kernel client)
by Ilya Dryomov 11 Feb '26

11 Feb '26

On Wed, Feb 11, 2026 at 5:29 PM Fabien Sirjean via ceph-users <ceph-users(a)ceph.io> wrote: > > Hello, > > We are encountering a reproducible issue with the CephFS kernel client > on Ubuntu 24.04 and would appreciate your feedback. > > Context: > We are preparing an updated Ubuntu 24.04 image for our scientific users. > With kernel |6.17.0-14-generic|, the CephFS mount succeeds and the > filesystem is visible via |df -h|, but any attempt to access the > directory contents (e.g. |ls|) immediately triggers a kernel crash. > > From userland, the process is simply killed: > > |# ls /net/serdon/illdata Killed| > > The mount appears healthy and the filesystem is correctly reported: > > |# mount | grep ceph 192.168.170.61,192.168.170.69,192.168.170.70:/ on > /net/serdon/illdata type ceph (rw,relatime,name=visa,acl) # df -h | grep > serdon 192.168.170.61,192.168.170.69,192.168.170.70:/ 4.1P 2.0P 2.1P 49% > /net/serdon/illdata| > > However, any access to the mount point triggers a kernel NULL pointer > dereference in the Ceph module: > > |BUG: kernel NULL pointer dereference, address: 0000000000000000 RIP: > strcmp+0x10/0x40 Call Trace: ceph_mds_auth_match ceph_mds_check_access > ceph_open| > > Environment: > > |Linux 6.17.0-14-generic #14~24.04.1-Ubuntu CephFS mounted via kernel > client (mount -t ceph)| > > The issue is fully reproducible: any access to the mount point causes > the same kernel crash. > > Full stack trace: > https://pastebin.com/ZgQHpN0q <https://pastebin.com/ZgQHpN0q> > > Questions: > - Is this a known issue with the CephFS kernel client on Linux 6.17? Hi Fabien, Yes, it has just been fixed: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/f… > - Does this look like an upstream CephFS bug, or something specific to > the Ubuntu 6.17 kernel? It's an upstream bug; the fix should be trivial to backport to any Ubuntu kernel that is affected. Thanks, Ilya

1 0

Zeroed files after shutdown + update
by dominik.baack 11 Feb '26

11 Feb '26

Hi, after an controlled shutdown of the whole cluster do to external circumstances we decided to update from 19.2 to 20.2 after the restart. The system was health before and after the update. The nodes mounting the filesystem were not equally lucky and were partially shutdown hard. Storage was kept running additional ~30min after node shutdown, all inflight operations should have finished. Now we discover the some of the user files seem to be replaced with zeros. For example: stat .gitignore File: .gitignore Size: 4429 Blocks: 9 IO Block: 4194304 regular file Device: 0,48 Inode: 1100241384598 Links: 1 hexdump -C .gitignore 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 00001140 00 00 00 00 00 00 00 00 00 00 00 00 00 |.............| 0000114d Scanning for files containing only zeros show several issues of files that were likely accessed before or during the shutdown of the nodes. How should I progress from here? Cheers Dominik

1 0

Kernel NULL dereference in ceph_mds_auth_match (Linux 6.17, CephFS kernel client)
by Fabien Sirjean 11 Feb '26

11 Feb '26

Hello, We are encountering a reproducible issue with the CephFS kernel client on Ubuntu 24.04 and would appreciate your feedback. Context: We are preparing an updated Ubuntu 24.04 image for our scientific users. With kernel |6.17.0-14-generic|, the CephFS mount succeeds and the filesystem is visible via |df -h|, but any attempt to access the directory contents (e.g. |ls|) immediately triggers a kernel crash. From userland, the process is simply killed: |# ls /net/serdon/illdata Killed| The mount appears healthy and the filesystem is correctly reported: |# mount | grep ceph 192.168.170.61,192.168.170.69,192.168.170.70:/ on /net/serdon/illdata type ceph (rw,relatime,name=visa,acl) # df -h | grep serdon 192.168.170.61,192.168.170.69,192.168.170.70:/ 4.1P 2.0P 2.1P 49% /net/serdon/illdata| However, any access to the mount point triggers a kernel NULL pointer dereference in the Ceph module: |BUG: kernel NULL pointer dereference, address: 0000000000000000 RIP: strcmp+0x10/0x40 Call Trace: ceph_mds_auth_match ceph_mds_check_access ceph_open| Environment: |Linux 6.17.0-14-generic #14~24.04.1-Ubuntu CephFS mounted via kernel client (mount -t ceph)| The issue is fully reproducible: any access to the mount point causes the same kernel crash. Full stack trace: https://pastebin.com/ZgQHpN0q <https://pastebin.com/ZgQHpN0q> Questions: - Is this a known issue with the CephFS kernel client on Linux 6.17? - Does this look like an upstream CephFS bug, or something specific to the Ubuntu 6.17 kernel? We can provide additional details (Ceph cluster version, MDS version, etc.) if needed. Thank you for your help. Best regards, Fabien

1 0

Re: tentacle and libvirt
by Joachim Kraftmayer 11 Feb '26

11 Feb '26

Hi, my first question is, what kind of ceph client version is used by libvirt? Regards, Joachim www.clyso.com Hohenzollernstr. 27, 80801 Munich Utting a. A. | HR: Augsburg | HRB: 25866 | USt. ID-Nr.: DE2754306 Eugen Block via ceph-users <ceph-users(a)ceph.io> schrieb am Mi., 11. Feb. 2026, 12:24: > Hi, > > your question is a bit "thin", I'd say. A lot has changed in Tentacle [0], > but nothing of the mentioned changes looks critical wrt libvirt from my > perspective. Do you have any logs or more detailed descriptions what > exactly fails? We haven't upgraded to v20 yet, at least not our prod > cluster, we also use RBD heavily (OpenStack on Ubuntu), so this is quite > relevant for us. > > Thanks, > Eugen > > [0] https://ceph.io/en/news/blog/2025/v20-2-0-tentacle-released/#rbd > > Am Di., 10. Feb. 2026 um 00:06 Uhr schrieb Piotr Pisz via ceph-users < > ceph-users(a)ceph.io>: > > > Hello, > > > > > > > > We upgraded Ceph to Tentacle, and everything went fine, but all hosts > > running Ubuntu 22 and 24 lost connectivity to RBD at the libvirt level. > > > > What has changed in the tentacle? > > > > > > > > Best regards, > > > > Piotr > > > > > > > > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: tentacle and libvirt
by Eugen Block 11 Feb '26

11 Feb '26

Hi, your question is a bit "thin", I'd say. A lot has changed in Tentacle [0], but nothing of the mentioned changes looks critical wrt libvirt from my perspective. Do you have any logs or more detailed descriptions what exactly fails? We haven't upgraded to v20 yet, at least not our prod cluster, we also use RBD heavily (OpenStack on Ubuntu), so this is quite relevant for us. Thanks, Eugen [0] https://ceph.io/en/news/blog/2025/v20-2-0-tentacle-released/#rbd Am Di., 10. Feb. 2026 um 00:06 Uhr schrieb Piotr Pisz via ceph-users < ceph-users(a)ceph.io>: > Hello, > > > > We upgraded Ceph to Tentacle, and everything went fine, but all hosts > running Ubuntu 22 and 24 lost connectivity to RBD at the libvirt level. > > What has changed in the tentacle? > > > > Best regards, > > Piotr > > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: reef 18.2.8 QE validation status
by Brad Hubbard 10 Feb '26

10 Feb '26

Powercycle approved. On Tue, Feb 10, 2026 at 10:35 AM Brad Hubbard via Dev <dev(a)ceph.io> wrote: > > > On Fri, Feb 6, 2026 at 12:36 PM Brad Hubbard <bhubbard(a)redhat.com> wrote: > >> >> >> On Fri, Feb 6, 2026 at 8:33 AM Brad Hubbard via ceph-users < >> ceph-users(a)ceph.io> wrote: >> >>> On Fri, Feb 6, 2026 at 7:59 AM Yuri Weinstein via ceph-users < >>> ceph-users(a)ceph.io> wrote: >>> >>> > We are continuing on this release. >>> > >>> > >>> > powercycle - FAILED Brad, approved? >>> > >>> >>> Unfortunately, I don't see any logs anywhere where I have access to them >>> so >>> I have no way of checking this? >>> >> >> OK, I have the latest logs. The three failing jobs appear to have all >> timed out after running for about six hours. At this time I don't know why >> but I continue to investigate. >> > > One point of commonality of the three tests that failed is that they are > all running tests using ceph fuse. I'll reach out to the FS guys to see if > they can help me get to the bottom of these failures. > > >> >> >>> >>> >>> > ceph-volume - Approved Guillaume? >>> > >>> > Patrick was reviewing some tests, please see his comments in the same >>> > tracker. >>> > TIA >>> > >>> > >>> > On Sat, Dec 13, 2025 at 1:19 AM Ilya Dryomov via ceph-users < >>> > ceph-users(a)ceph.io> wrote: >>> > >>> > > On Mon, Dec 8, 2025 at 11:31 PM Yuri Weinstein via Dev <dev(a)ceph.io> >>> > > wrote: >>> > > > >>> > > > This test branch was built loner then 2.5 weeks ago >>> > > > I have to rebuild it first and then will schedule krbd >>> > > >>> > > krbd approved based on >>> > > >>> > > >>> > > >>> > >>> https://pulpito.ceph.com/yuriw-2025-12-10_21:26:56-krbd-reef-release-testin… >>> > > >>> > > >>> > >>> https://pulpito.ceph.com/yuriw-2025-12-12_18:20:32-krbd-reef-release-testin… >>> > > >>> > > Thanks, >>> > > >>> > > Ilya >>> > > >>> > > > >>> > > > On Mon, Dec 8, 2025 at 1:39 PM Ilya Dryomov via ceph-users < >>> > > ceph-users(a)ceph.io> wrote: >>> > > >> >>> > > >> On Mon, Dec 8, 2025 at 8:50 PM Ilya Dryomov <idryomov(a)gmail.com> >>> > wrote: >>> > > >> > >>> > > >> > On Mon, Dec 8, 2025 at 5:00 PM Venky Shankar via ceph-users >>> > > >> > <ceph-users(a)ceph.io> wrote: >>> > > >> > > >>> > > >> > > Hi Yuri, >>> > > >> > > >>> > > >> > > On Fri, Nov 21, 2025 at 4:34 AM Yuri Weinstein < >>> > yweinste(a)redhat.com> >>> > > wrote: >>> > > >> > > > >>> > > >> > > > Note: this is an attempt to test and publish post EOL reef >>> > point >>> > > release >>> > > >> > > > >>> > > >> > > > Details of this release are summarized here: >>> > > >> > > > >>> > > >> > > > https://tracker.ceph.com/issues/73906#note-1 >>> > > >> > > > Release Notes - TBD >>> > > >> > > > LRC upgrade - N/A >>> > > >> > > > >>> > > >> > > > distro => jammy focal centos9 windows bookworm >>> > > >> > > > >>> > > >> > > > Seeking approvals/reviews for: >>> > > >> > > > >>> > > >> > > > smoke - Laura approved? >>> > > >> > > > >>> > > >> > > > rados - Radek, Laura approved? Travis? Nizamudeen? Adam >>> King >>> > > approved? >>> > > >> > > > (rados is also running on build 2 == build 1 + >>> > > >> > > > https://github.com/ceph/ceph/pull/66056 per Igor's request) >>> > > >> > > > >>> > > >> > > > rgw - Adam E approved? >>> > > >> > > > >>> > > >> > > > fs - Venky approved? >>> > > >> > > >>> > > >> > > There are lot many failures in the fs suite resembling >>> > > >> > > >>> > > >> > > Error mounting /var/lib/ceph: mount: /var/lib/ceph: >>> wrong >>> > fs >>> > > >> > > type, bad option ... >>> > > >> > > >>> > > >> > > which is an issue with the testing kernel. Adding Alex who was >>> > > looking >>> > > >> > > into this issue. >>> > > >> > > >>> > > >> > > As far as the run validation is concerned, it's risky to >>> ignore >>> > the >>> > > >> > > large number of failures. We have two options: >>> > > >> > > >>> > > >> > > - fix the testing kernel issue and rerun >>> > > >> > > - rerun with `for-linus` branch (with `-k for-linus` in the >>> > > >> > > teuthology-suite schedule command) >>> > > >> > > >>> > > >> > > The latter would be using an older kernel branch, but I think >>> it >>> > > >> > > should suffice for this case. >>> > > >> > >>> > > >> > Hi Venky, >>> > > >> > >>> > > >> > With your report of the fs suite being affected as well, I >>> carved >>> > out >>> > > >> > some time to take a look. >>> > > >> > >>> > > >> > I think it's happening because XFS maintainers disabled support >>> for >>> > V4 >>> > > >> > filesystems that was deprecated 5 years ago -- as of 6.18 kernel >>> > it's >>> > > >> > no longer compiled in by default. The proper fix would be to >>> create >>> > > an >>> > > >> > image with/move to V5 on-disk format; I have opened [1] to try >>> to >>> > > >> > achieve that. We can also re-enable V4 on-disk format in our >>> kernel >>> > > >> > builds as a workaround. >>> > > >> > >>> > > >> > [1] https://github.com/ceph/ceph-cm-ansible/pull/799 >>> > > >> >>> > > >> This should be fixed now. Yuri, please schedule a rerun for krbd. >>> > > >> >>> > > >> Thanks, >>> > > >> >>> > > >> Ilya >>> > > >> _______________________________________________ >>> > > >> ceph-users mailing list -- ceph-users(a)ceph.io >>> > > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> > > > >>> > > > _______________________________________________ >>> > > > Dev mailing list -- dev(a)ceph.io >>> > > > To unsubscribe send an email to dev-leave(a)ceph.io >>> > > _______________________________________________ >>> > > ceph-users mailing list -- ceph-users(a)ceph.io >>> > > To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> > > >>> > _______________________________________________ >>> > ceph-users mailing list -- ceph-users(a)ceph.io >>> > To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> > >>> >>> >>> -- >>> Cheers, >>> Brad >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> >> >> >> -- >> Cheers, >> Brad >> > > > -- > Cheers, > Brad > _______________________________________________ > Dev mailing list -- dev(a)ceph.io > To unsubscribe send an email to dev-leave(a)ceph.io > -- Cheers, Brad

1 0