28 Oct
2019
28 Oct
'19
3:13 p.m.
This seems to happen mostly when listing folders containing 10k+ folders.
The dirlisting hangs indefinitely or until i restart the active MDS and
then the hanging "ls" command will finish running.
Every time restarting the active MDS fixes the problem for a while.
28 Oct
28 Oct
3:53 p.m.
Hello Kári,
On Mon, Oct 28, 2019 at 11:14 AM Kári Bertilsson <karibertils(a)gmail.com> wrote:
This seems to happen mostly when listing folders
containing 10k+ folders.
The dirlisting hangs indefinitely or until i restart the active MDS and then the hanging
"ls" command will finish running.
Every time restarting the active MDS fixes the problem for a while.
Please share details about your cluster. `fs dump`, `ceph status`, and
`ceph versions`. How many clients are using the file system?
--
Patrick Donnelly, Ph.D.
He / Him / His
Senior Software Engineer
Red Hat Sunnyvale, CA
GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
4:17 p.m.
Hello Patrick,
Here is output from those commands
https://pastebin.com/yUmuQuYj
5 clients have the file system mounted, but only 2 of them have most of the
activity.
On Mon, Oct 28, 2019 at 6:54 PM Patrick Donnelly <pdonnell(a)redhat.com>
wrote:
Hello Kári,
On Mon, Oct 28, 2019 at 11:14 AM Kári Bertilsson <karibertils(a)gmail.com>
wrote:
This seems to happen mostly when listing folders
containing 10k+ folders.
The dirlisting hangs indefinitely or until i restart the active MDS and
then the
hanging "ls" command will finish running.
Every time restarting the active MDS fixes the problem for a while.
Please share details about your cluster. `fs dump`, `ceph status`, and
`ceph versions`. How many clients are using the file system?
--
Patrick Donnelly, Ph.D.
He / Him / His
Senior Software Engineer
Red Hat Sunnyvale, CA
GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
7:10 p.m.
Any ideas or tips on how to debug further ?
On Mon, Oct 28, 2019 at 7:17 PM Kári Bertilsson <karibertils(a)gmail.com>
wrote:
Hello Patrick,
Here is output from those commands
https://pastebin.com/yUmuQuYj
5 clients have the file system mounted, but only 2 of them have most of
the activity.
On Mon, Oct 28, 2019 at 6:54 PM Patrick Donnelly <pdonnell(a)redhat.com>
wrote:
> Hello Kári,
>
> On Mon, Oct 28, 2019 at 11:14 AM Kári Bertilsson <karibertils(a)gmail.com>
> wrote:
> > This seems to happen mostly when listing folders containing 10k+
> folders.
> >
> > The dirlisting hangs indefinitely or until i restart the active MDS and
> then the hanging "ls" command will finish running.
> >
> > Every time restarting the active MDS fixes the problem for a while.
>
> Please share details about your cluster. `fs dump`, `ceph status`, and
> `ceph versions`. How many clients are using the file system?
>
> --
> Patrick Donnelly, Ph.D.
> He / Him / His
> Senior Software Engineer
> Red Hat Sunnyvale, CA
> GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
>
>
29 Oct
29 Oct
3:32 a.m.
Hi!
What kind of client (kernel vs. FUSE) do you use?
I experience a lot of the following problems with the most recent ubuntu 18.04.3 kernel
4.15.0-66-generic :
kernel: [260144.644232] cache_from_obj: Wrong slab cache. inode_cache but object is from
ceph_inode_info
Other clients with older kernels (e.g. 4.15.0-47-generic) work without interruption on the
same CephFS.
Lars
Mon, 28 Oct 2019 22:10:25 +0000
Kári Bertilsson <karibertils(a)gmail.com> ==> Patrick Donnelly
<pdonnell(a)redhat.com> :
Any ideas or tips on how to debug further ?
On Mon, Oct 28, 2019 at 7:17 PM Kári Bertilsson <karibertils(a)gmail.com>
wrote:
> Hello Patrick,
>
> Here is output from those commands
> https://pastebin.com/yUmuQuYj
>
> 5 clients have the file system mounted, but only 2 of them have most of
> the activity.
>
>
>
> On Mon, Oct 28, 2019 at 6:54 PM Patrick Donnelly <pdonnell(a)redhat.com>
> wrote:
>
>> Hello Kári,
>>
>> On Mon, Oct 28, 2019 at 11:14 AM Kári Bertilsson <karibertils(a)gmail.com>
>> wrote:
>> > This seems to happen mostly when listing folders containing 10k+
>> folders.
>> >
>> > The dirlisting hangs indefinitely or until i restart the active MDS and
>> then the hanging "ls" command will finish running.
>> >
>> > Every time restarting the active MDS fixes the problem for a while.
>>
>> Please share details about your cluster. `fs dump`, `ceph status`, and
>> `ceph versions`. How many clients are using the file system?
>>
>> --
>> Patrick Donnelly, Ph.D.
>> He / Him / His
>> Senior Software Engineer
>> Red Hat Sunnyvale, CA
>> GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
>>
>>
--
Informationstechnologie
Berlin-Brandenburgische Akademie der Wissenschaften
Jägerstraße 22-23 10117 Berlin
Tel.: +49 30 20370-352 http://www.bbaw.de
8:48 a.m.
All clients are using the kernel client on proxmox kernel
version 5.0.21-3-pve.
The mds logs are not showing anything interesting and have very little in
them except for the restarts, maybe i need to increase debug level ?
On Tue, Oct 29, 2019 at 6:33 AM Lars Täuber <taeuber(a)bbaw.de> wrote:
Hi!
What kind of client (kernel vs. FUSE) do you use?
I experience a lot of the following problems with the most recent ubuntu
18.04.3 kernel 4.15.0-66-generic :
kernel: [260144.644232] cache_from_obj: Wrong slab cache. inode_cache but
object is from ceph_inode_info
Other clients with older kernels (e.g. 4.15.0-47-generic) work without
interruption on the same CephFS.
Lars
Mon, 28 Oct 2019 22:10:25 +0000
Kári Bertilsson <karibertils(a)gmail.com> ==> Patrick Donnelly <
pdonnell(a)redhat.com> :
Any ideas or tips on how to debug further ?
On Mon, Oct 28, 2019 at 7:17 PM Kári Bertilsson <karibertils(a)gmail.com>
wrote:
> Hello Patrick,
>
> Here is output from those commands
> https://pastebin.com/yUmuQuYj
>
> 5 clients have the file system mounted, but only 2 of them have most of
> the activity.
>
>
>
> On Mon, Oct 28, 2019 at 6:54 PM Patrick Donnelly <pdonnell(a)redhat.com>
> wrote:
>
>> Hello Kári,
>>
>> On Mon, Oct 28, 2019 at 11:14 AM Kári Bertilsson <
karibertils(a)gmail.com>
>> wrote:
>> > This seems to happen mostly when listing folders containing 10k+
>> folders.
>> >
>> > The dirlisting hangs indefinitely or until i restart the active MDS
and
>> then the hanging "ls" command
will finish running.
>> >
>> > Every time restarting the active MDS fixes the problem for a
while.
>>
>> Please share details about your cluster. `fs dump`, `ceph status`, and
>> `ceph versions`. How many clients are using the file system?
>>
>> --
>> Patrick Donnelly, Ph.D.
>> He / Him / His
>> Senior Software Engineer
>> Red Hat Sunnyvale, CA
>> GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
>>
>>
--
Informationstechnologie
Berlin-Brandenburgische Akademie der Wissenschaften
Jägerstraße 22-23 10117 Berlin
Tel.: +49 30 20370-352 http://www.bbaw.de
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
8:56 a.m.
The file system was created on luminous and the problems started after
upgrading from luminous to nautilus.
All CephFS configuration should be pretty much default except i enabled
snapshots which was disabled by default on luminous.
On Tue, Oct 29, 2019 at 11:48 AM Kári Bertilsson <karibertils(a)gmail.com>
wrote:
All clients are using the kernel client on proxmox
kernel
version 5.0.21-3-pve.
The mds logs are not showing anything interesting and have very little in
them except for the restarts, maybe i need to increase debug level ?
On Tue, Oct 29, 2019 at 6:33 AM Lars Täuber <taeuber(a)bbaw.de> wrote:
Hi!
What kind of client (kernel vs. FUSE) do you use?
I experience a lot of the following problems with the most recent ubuntu
18.04.3 kernel 4.15.0-66-generic :
kernel: [260144.644232] cache_from_obj: Wrong slab cache. inode_cache but
object is from ceph_inode_info
Other clients with older kernels (e.g. 4.15.0-47-generic) work without
interruption on the same CephFS.
Lars
Mon, 28 Oct 2019 22:10:25 +0000
Kári Bertilsson <karibertils(a)gmail.com> ==> Patrick Donnelly <
pdonnell(a)redhat.com> :
karibertils(a)gmail.com>
Any ideas or tips on how to debug further ?
On Mon, Oct 28, 2019 at 7:17 PM Kári Bertilsson <karibertils(a)gmail.com>
wrote:
> Hello Patrick,
>
> Here is output from those commands
> https://pastebin.com/yUmuQuYj
>
> 5 clients have the file system mounted, but only 2 of them have most
of
the
activity.
On Mon, Oct 28, 2019 at 6:54 PM Patrick Donnelly <pdonnell(a)redhat.com
> wrote:
>
>> Hello Kári,
>>
>> On Mon, Oct 28, 2019 at 11:14 AM Kári Bertilsson < >> wrote:
>> > This seems to happen mostly when listing folders containing 10k+
>> folders.
>> >
>> > The dirlisting hangs indefinitely or until i restart the active
MDS
and
>> then the hanging "ls" command
will finish running.
>> >
>> > Every time restarting the active MDS fixes the problem for a
while.
>>
>> Please share details about your cluster. `fs dump`, `ceph status`,
and
>> `ceph versions`. How many clients are
using the file system?
>>
>> --
>> Patrick Donnelly, Ph.D.
>> He / Him / His
>> Senior Software Engineer
>> Red Hat Sunnyvale, CA
>> GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
>>
>>
--
Informationstechnologie
Berlin-Brandenburgische Akademie der Wissenschaften
Jägerstraße 22-23 10117 Berlin
Tel.: +49 30 20370-352 http://www.bbaw.de
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
10:03 a.m.
I am noticing i have many entries in `ceph osd blacklist ls` and dirlisting
works again after i removed all entries.
What can cause this and is there any way to disable blacklisting ?
On Tue, Oct 29, 2019 at 11:56 AM Kári Bertilsson <karibertils(a)gmail.com>
wrote:
The file system was created on luminous and the
problems started after
upgrading from luminous to nautilus.
All CephFS configuration should be pretty much default except i enabled
snapshots which was disabled by default on luminous.
On Tue, Oct 29, 2019 at 11:48 AM Kári Bertilsson <karibertils(a)gmail.com>
wrote:
> All clients are using the kernel client on proxmox kernel
> version 5.0.21-3-pve.
>
> The mds logs are not showing anything interesting and have very little in
> them except for the restarts, maybe i need to increase debug level ?
>
> On Tue, Oct 29, 2019 at 6:33 AM Lars Täuber <taeuber(a)bbaw.de> wrote:
>
>> Hi!
>>
>> What kind of client (kernel vs. FUSE) do you use?
>> I experience a lot of the following problems with the most recent ubuntu
>> 18.04.3 kernel 4.15.0-66-generic :
>> kernel: [260144.644232] cache_from_obj: Wrong slab cache. inode_cache
>> but object is from ceph_inode_info
>>
>> Other clients with older kernels (e.g. 4.15.0-47-generic) work without
>> interruption on the same CephFS.
>>
>>
>> Lars
>>
>>
>> Mon, 28 Oct 2019 22:10:25 +0000
>> Kári Bertilsson <karibertils(a)gmail.com> ==> Patrick Donnelly <
>> pdonnell(a)redhat.com> :
>> > Any ideas or tips on how to debug further ?
>> >
>> > On Mon, Oct 28, 2019 at 7:17 PM Kári Bertilsson <karibertils(a)gmail.com
>> >
>> > wrote:
>> >
>> > > Hello Patrick,
>> > >
>> > > Here is output from those commands
>> > > https://pastebin.com/yUmuQuYj
>> > >
>> > > 5 clients have the file system mounted, but only 2 of them have most
>> of
>> > > the activity.
>> > >
>> > >
>> > >
>> > > On Mon, Oct 28, 2019 at 6:54 PM Patrick Donnelly <
>> pdonnell(a)redhat.com>
>> > > wrote:
>> > >
>> > >> Hello Kári,
>> > >>
>> > >> On Mon, Oct 28, 2019 at 11:14 AM Kári Bertilsson <
>> karibertils(a)gmail.com>
>> > >> wrote:
>> > >> > This seems to happen mostly when listing folders containing
10k+
>> > >> folders.
>> > >> >
>> > >> > The dirlisting hangs indefinitely or until i restart the
active
>> MDS and
>> > >> then the hanging "ls" command will finish running.
>> > >> >
>> > >> > Every time restarting the active MDS fixes the problem for a
>> while.
>> > >>
>> > >> Please share details about your cluster. `fs dump`, `ceph status`,
>> and
>> > >> `ceph versions`. How many clients are using the file system?
>> > >>
>> > >> --
>> > >> Patrick Donnelly, Ph.D.
>> > >> He / Him / His
>> > >> Senior Software Engineer
>> > >> Red Hat Sunnyvale, CA
>> > >> GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
>> > >>
>> > >>
>>
>>
>> --
>> Informationstechnologie
>> Berlin-Brandenburgische Akademie der Wissenschaften
>> Jägerstraße 22-23 10117 Berlin
>> Tel.: +49 30 20370-352 http://www.bbaw.de
>> _______________________________________________
>> ceph-users mailing list -- ceph-users(a)ceph.io
>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>>
>
11:23 a.m.
I jumped the gun too quickly, dirlisting is still hanging with no entries
in `ceph osd blacklist ls`.
But when i restart the active MDS and the standby goes active dirlisting
finishes and i get 2 entries in blacklist with the IP address on the
previously active MDS.
On Tue, Oct 29, 2019 at 1:03 PM Kári Bertilsson <karibertils(a)gmail.com>
wrote:
I am noticing i have many entries in `ceph osd
blacklist ls` and
dirlisting works again after i removed all entries.
What can cause this and is there any way to disable blacklisting ?
On Tue, Oct 29, 2019 at 11:56 AM Kári Bertilsson <karibertils(a)gmail.com>
wrote:
> The file system was created on luminous and the problems started after
> upgrading from luminous to nautilus.
> All CephFS configuration should be pretty much default except i enabled
> snapshots which was disabled by default on luminous.
>
> On Tue, Oct 29, 2019 at 11:48 AM Kári Bertilsson <karibertils(a)gmail.com>
> wrote:
>
>> All clients are using the kernel client on proxmox kernel
>> version 5.0.21-3-pve.
>>
>> The mds logs are not showing anything interesting and have very little
>> in them except for the restarts, maybe i need to increase debug level ?
>>
>> On Tue, Oct 29, 2019 at 6:33 AM Lars Täuber <taeuber(a)bbaw.de> wrote:
>>
>>> Hi!
>>>
>>> What kind of client (kernel vs. FUSE) do you use?
>>> I experience a lot of the following problems with the most recent
>>> ubuntu 18.04.3 kernel 4.15.0-66-generic :
>>> kernel: [260144.644232] cache_from_obj: Wrong slab cache. inode_cache
>>> but object is from ceph_inode_info
>>>
>>> Other clients with older kernels (e.g. 4.15.0-47-generic) work without
>>> interruption on the same CephFS.
>>>
>>>
>>> Lars
>>>
>>>
>>> Mon, 28 Oct 2019 22:10:25 +0000
>>> Kári Bertilsson <karibertils(a)gmail.com> ==> Patrick Donnelly <
>>> pdonnell(a)redhat.com> :
>>> > Any ideas or tips on how to debug further ?
>>> >
>>> > On Mon, Oct 28, 2019 at 7:17 PM Kári Bertilsson <
>>> karibertils(a)gmail.com>
>>> > wrote:
>>> >
>>> > > Hello Patrick,
>>> > >
>>> > > Here is output from those commands
>>> > > https://pastebin.com/yUmuQuYj
>>> > >
>>> > > 5 clients have the file system mounted, but only 2 of them have
>>> most of
>>> > > the activity.
>>> > >
>>> > >
>>> > >
>>> > > On Mon, Oct 28, 2019 at 6:54 PM Patrick Donnelly <
>>> pdonnell(a)redhat.com>
>>> > > wrote:
>>> > >
>>> > >> Hello Kári,
>>> > >>
>>> > >> On Mon, Oct 28, 2019 at 11:14 AM Kári Bertilsson <
>>> karibertils(a)gmail.com>
>>> > >> wrote:
>>> > >> > This seems to happen mostly when listing folders
containing
>>> 10k+
>>> > >> folders.
>>> > >> >
>>> > >> > The dirlisting hangs indefinitely or until i restart the
active
>>> MDS and
>>> > >> then the hanging "ls" command will finish running.
>>> > >> >
>>> > >> > Every time restarting the active MDS fixes the problem for
a
>>> while.
>>> > >>
>>> > >> Please share details about your cluster. `fs dump`, `ceph
status`,
>>> and
>>> > >> `ceph versions`. How many clients are using the file system?
>>> > >>
>>> > >> --
>>> > >> Patrick Donnelly, Ph.D.
>>> > >> He / Him / His
>>> > >> Senior Software Engineer
>>> > >> Red Hat Sunnyvale, CA
>>> > >> GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
>>> > >>
>>> > >>
>>>
>>>
>>> --
>>> Informationstechnologie
>>> Berlin-Brandenburgische Akademie der Wissenschaften
>>> Jägerstraße 22-23 10117 Berlin
>>> Tel.: +49 30 20370-352 http://www.bbaw.de
>>> _______________________________________________
>>> ceph-users mailing list -- ceph-users(a)ceph.io
>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>>>
>>
5:53 p.m.
On Mon, Oct 28, 2019 at 11:33 PM Lars Täuber <taeuber(a)bbaw.de> wrote:
Hi!
What kind of client (kernel vs. FUSE) do you use?
I experience a lot of the following problems with the most recent ubuntu 18.04.3 kernel
4.15.0-66-generic :
kernel: [260144.644232] cache_from_obj: Wrong slab cache. inode_cache but object is from
ceph_inode_info
Other clients with older kernels (e.g. 4.15.0-47-generic) work without interruption on
the same CephFS.
Sounds like another issue reported on ceph-users: "[ceph-users] CephFS
kernel module lockups in Ubuntu linux-image-5.0.0-32-generic?"
Suggest you contact Canonical to get them to fix their kernel or
downgrade your kernel.
--
Patrick Donnelly, Ph.D.
He / Him / His
Senior Software Engineer
Red Hat Sunnyvale, CA
GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
6:03 p.m.
Ubuntu's 4.15.0-66 has this bug, yes. -65 is safe and -67 will have the fix.
On Tue, Oct 29, 2019 at 4:54 PM Patrick Donnelly <pdonnell(a)redhat.com> wrote:
>
> On Mon, Oct 28, 2019 at 11:33 PM Lars Täuber <taeuber(a)bbaw.de> wrote:
> >
> > Hi!
> >
> > What kind of client (kernel vs. FUSE) do you use?
> > I experience a lot of the following problems with the most recent ubuntu 18.04.3
kernel 4.15.0-66-generic :
> > kernel: [260144.644232] cache_from_obj: Wrong slab cache. inode_cache but object
is from ceph_inode_info
> >
> > Other clients with older kernels (e.g. 4.15.0-47-generic) work without
interruption on the same CephFS.
>
> Sounds like another issue reported on ceph-users: "[ceph-users] CephFS
> kernel module lockups in Ubuntu linux-image-5.0.0-32-generic?"
>
> Suggest you contact Canonical to get them to fix their kernel or
> downgrade your kernel.
>
> --
> Patrick Donnelly, Ph.D.
> He / Him / His
> Senior Software Engineer
> Red Hat Sunnyvale, CA
> GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
7:43 p.m.
I am encountering the dirlist hanging issue on multiple clients and none of
them are Ubuntu.
Debian buster running kernel 4.19.0-2-amd64. This one was working fine
until after ceph was upgraded to nautilus
Proxmox running kernels 5.0.21-1-pve and 5.0.18-1-pve
On Tue, Oct 29, 2019 at 9:04 PM Nathan Fish <lordcirth(a)gmail.com> wrote:
Ubuntu's 4.15.0-66 has this bug, yes. -65 is safe
and -67 will have the
fix.
On Tue, Oct 29, 2019 at 4:54 PM Patrick Donnelly <pdonnell(a)redhat.com>
wrote:
On Mon, Oct 28, 2019 at 11:33 PM Lars Täuber <taeuber(a)bbaw.de> wrote:
>
> Hi!
>
> What kind of client (kernel vs. FUSE) do you use?
> I experience a lot of the following problems with the most recent
ubuntu
18.04.3 kernel 4.15.0-66-generic :
> kernel: [260144.644232] cache_from_obj:
Wrong slab cache. inode_cache
but object is from ceph_inode_info
>
> Other clients with older kernels (e.g. 4.15.0-47-generic) work without
interruption on the same CephFS.
Sounds like another issue reported on ceph-users: "[ceph-users] CephFS
kernel module lockups in Ubuntu linux-image-5.0.0-32-generic?"
Suggest you contact Canonical to get them to fix their kernel or
downgrade your kernel.
--
Patrick Donnelly, Ph.D.
He / Him / His
Senior Software Engineer
Red Hat Sunnyvale, CA
GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
30 Oct
30 Oct
3:52 a.m.
Hi Kári,
what about this:
health: HEALTH_WARN
854 pgs not deep-scrubbed in time
maybe you should
$ ceph –cluster first pg scrub XX.YY
or
$ ceph –cluster first pg deep-scrub XX.YY
all the PGs.
Tue, 29 Oct 2019 22:43:28 +0000
Kári Bertilsson <karibertils(a)gmail.com> ==> Nathan Fish
<lordcirth(a)gmail.com> :
I am encountering the dirlist hanging issue on
multiple clients and none of
them are Ubuntu.
Debian buster running kernel 4.19.0-2-amd64. This one was working fine
until after ceph was upgraded to nautilus
Proxmox running kernels 5.0.21-1-pve and 5.0.18-1-pve
On Tue, Oct 29, 2019 at 9:04 PM Nathan Fish <lordcirth(a)gmail.com> wrote:
> Ubuntu's 4.15.0-66 has this bug, yes. -65 is safe and -67 will have the
> fix.
>
> On Tue, Oct 29, 2019 at 4:54 PM Patrick Donnelly <pdonnell(a)redhat.com>
> wrote:
> >
> > On Mon, Oct 28, 2019 at 11:33 PM Lars Täuber <taeuber(a)bbaw.de> wrote:
> > >
> > > Hi!
> > >
> > > What kind of client (kernel vs. FUSE) do you use?
> > > I experience a lot of the following problems with the most recent
> ubuntu 18.04.3 kernel 4.15.0-66-generic :
> > > kernel: [260144.644232] cache_from_obj: Wrong slab cache. inode_cache
> but object is from ceph_inode_info
> > >
> > > Other clients with older kernels (e.g. 4.15.0-47-generic) work without
> interruption on the same CephFS.
> >
> > Sounds like another issue reported on ceph-users: "[ceph-users] CephFS
> > kernel module lockups in Ubuntu linux-image-5.0.0-32-generic?"
> >
> > Suggest you contact Canonical to get them to fix their kernel or
> > downgrade your kernel.
> >
> > --
> > Patrick Donnelly, Ph.D.
> > He / Him / His
> > Senior Software Engineer
> > Red Hat Sunnyvale, CA
> > GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
> > _______________________________________________
> > ceph-users mailing list -- ceph-users(a)ceph.io
> > To unsubscribe send an email to ceph-users-leave(a)ceph.io
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
--
Informationstechnologie
Berlin-Brandenburgische Akademie der Wissenschaften
Jägerstraße 22-23 10117 Berlin
Tel.: +49 30 20370-352 http://www.bbaw.de
1:36 p.m.
Not sure if this is related to the dirlisting issue since the deep-scrubs
have always been way behind schedule.
But lets see if it has any effect to clear this warning. But it seems i can
only deep-scrub 5 pgs at a time. How can i increase this ?
On Wed, Oct 30, 2019 at 6:53 AM Lars Täuber <taeuber(a)bbaw.de> wrote:
Hi Kári,
what about this:
health: HEALTH_WARN
854 pgs not deep-scrubbed in time
maybe you should
$ ceph –cluster first pg scrub XX.YY
or
$ ceph –cluster first pg deep-scrub XX.YY
all the PGs.
Tue, 29 Oct 2019 22:43:28 +0000
Kári Bertilsson <karibertils(a)gmail.com> ==> Nathan Fish <
lordcirth(a)gmail.com> :
I am encountering the dirlist hanging issue on
multiple clients and none
of
them are Ubuntu.
Debian buster running kernel 4.19.0-2-amd64. This one was working fine
until after ceph was upgraded to nautilus
Proxmox running kernels 5.0.21-1-pve and 5.0.18-1-pve
On Tue, Oct 29, 2019 at 9:04 PM Nathan Fish <lordcirth(a)gmail.com> wrote:
> Ubuntu's 4.15.0-66 has this bug, yes. -65 is safe and -67 will have the
> fix.
>
> On Tue, Oct 29, 2019 at 4:54 PM Patrick Donnelly <pdonnell(a)redhat.com>
> wrote:
> >
> > On Mon, Oct 28, 2019 at 11:33 PM Lars Täuber <taeuber(a)bbaw.de>
wrote:
> > >
> > > Hi!
> > >
> > > What kind of client (kernel vs. FUSE) do you use?
> > > I experience a lot of the following problems with the most recent
> ubuntu 18.04.3 kernel 4.15.0-66-generic :
> > > kernel: [260144.644232] cache_from_obj: Wrong slab cache.
inode_cache
> but object is from ceph_inode_info
> > >
> > > Other clients with older kernels (e.g. 4.15.0-47-generic) work
without
> interruption on the same CephFS.
> >
> > Sounds like another issue reported on ceph-users: "[ceph-users]
CephFS
> > kernel module lockups in Ubuntu
linux-image-5.0.0-32-generic?"
> >
> > Suggest you contact Canonical to get them to fix their kernel or
> > downgrade your kernel.
> >
> > --
> > Patrick Donnelly, Ph.D.
> > He / Him / His
> > Senior Software Engineer
> > Red Hat Sunnyvale, CA
> > GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
> > _______________________________________________
> > ceph-users mailing list -- ceph-users(a)ceph.io
> > To unsubscribe send an email to ceph-users-leave(a)ceph.io
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
--
Informationstechnologie
Berlin-Brandenburgische Akademie der Wissenschaften
Jägerstraße 22-23 10117 Berlin
Tel.: +49 30 20370-352 http://www.bbaw.de
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
28 Oct
28 Oct
7:27 p.m.
On Mon, Oct 28, 2019 at 12:17 PM Kári Bertilsson <karibertils(a)gmail.com> wrote:
Hello Patrick,
Here is output from those commands
https://pastebin.com/yUmuQuYj
5 clients have the file system mounted, but only 2 of them have most of the activity.
Have you modified any CephFS configurations?
A copy of the MDS log during one of these events may shed light on the cause.
--
Patrick Donnelly, Ph.D.
He / Him / His
Senior Software Engineer
Red Hat Sunnyvale, CA
GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
1651
days inactive
1653
days old
14 comments
4 participants
participants (4)
-
Kári Bertilsson -
Lars Täuber -
Nathan Fish -
Patrick Donnelly