I jumped the gun too quickly, dirlisting is still hanging with no entries in `ceph osd blacklist ls`. But when i restart the active MDS and the standby goes active dirlisting finishes and i get 2 entries in blacklist with the IP address on the previously active MDS. On Tue, Oct 29, 2019 at 1:03 PM Kári Bertilsson <karibertils@gmail.com> wrote:
I am noticing i have many entries in `ceph osd blacklist ls` and dirlisting works again after i removed all entries. What can cause this and is there any way to disable blacklisting ?
On Tue, Oct 29, 2019 at 11:56 AM Kári Bertilsson <karibertils@gmail.com> wrote:
The file system was created on luminous and the problems started after upgrading from luminous to nautilus. All CephFS configuration should be pretty much default except i enabled snapshots which was disabled by default on luminous.
On Tue, Oct 29, 2019 at 11:48 AM Kári Bertilsson <karibertils@gmail.com> wrote:
All clients are using the kernel client on proxmox kernel version 5.0.21-3-pve.
The mds logs are not showing anything interesting and have very little in them except for the restarts, maybe i need to increase debug level ?
On Tue, Oct 29, 2019 at 6:33 AM Lars Täuber <taeuber@bbaw.de> wrote:
Hi!
What kind of client (kernel vs. FUSE) do you use? I experience a lot of the following problems with the most recent ubuntu 18.04.3 kernel 4.15.0-66-generic : kernel: [260144.644232] cache_from_obj: Wrong slab cache. inode_cache but object is from ceph_inode_info
Other clients with older kernels (e.g. 4.15.0-47-generic) work without interruption on the same CephFS.
Lars
Any ideas or tips on how to debug further ?
On Mon, Oct 28, 2019 at 7:17 PM Kári Bertilsson < karibertils@gmail.com> wrote:
Hello Patrick,
Here is output from those commands https://pastebin.com/yUmuQuYj
5 clients have the file system mounted, but only 2 of them have most of the activity.
On Mon, Oct 28, 2019 at 6:54 PM Patrick Donnelly <
Mon, 28 Oct 2019 22:10:25 +0000 Kári Bertilsson <karibertils@gmail.com> ==> Patrick Donnelly < pdonnell@redhat.com> : pdonnell@redhat.com>
wrote:
> Hello Kári, > > On Mon, Oct 28, 2019 at 11:14 AM Kári Bertilsson < karibertils@gmail.com> > wrote: > > This seems to happen mostly when listing folders containing 10k+ > folders. > > > > The dirlisting hangs indefinitely or until i restart the active MDS and > then the hanging "ls" command will finish running. > > > > Every time restarting the active MDS fixes the problem for a while. > > Please share details about your cluster. `fs dump`, `ceph status`, and > `ceph versions`. How many clients are using the file system? > > -- > Patrick Donnelly, Ph.D. > He / Him / His > Senior Software Engineer > Red Hat Sunnyvale, CA > GPG: 19F28A586F808C2402351B93C3301A3E258DD79D > >
-- Informationstechnologie Berlin-Brandenburgische Akademie der Wissenschaften Jägerstraße 22-23 10117 Berlin Tel.: +49 30 20370-352 http://www.bbaw.de _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io