16 Sep
2019
16 Sep
'19
9:59 a.m.
Hi,
I tried to run this command with failure:
root@ld3955:/mnt/rbd# ceph osd set-require-min-compat-client luminous
Error EPERM: cannot set require_min_compat_client to luminous: 6
connected client(s) look like jewel (missing 0xa00000000200000); 19
connected client(s) look like jewel (missing 0x800000000000000); 23
connected client(s) look like jewel (missing 0x800000000000000); add
--yes-i-really-mean-it to do it anyway
root@ld3955:/mnt/rbd# ceph features
{
"mon": [
{
"features": "0x3ffddff8ffacffff",
"release": "luminous",
"num": 3
}
],
"mds": [
{
"features": "0x3ffddff8ffacffff",
"release": "luminous",
"num": 2
}
],
"osd": [
{
"features": "0x3ffddff8ffacffff",
"release": "luminous",
"num": 368
}
],
"client": [
{
"features": "0x40106b84a842a42",
"release": "jewel",
"num": 6
},
{
"features": "0x27018eb84aa42a52",
"release": "jewel",
"num": 19
},
{
"features": "0x27018fb86aa42ada",
"release": "jewel",
"num": 23
},
{
"features": "0x3ffddff8ffacffff",
"release": "luminous",
"num": 35
}
],
"mgr": [
{
"features": "0x3ffddff8ffacffff",
"release": "luminous",
"num": 3
}
]
}
But I think the number of clients older than Luminous is incorrect.
I'm running 98% of the clients with SLES 12SPx, and therefore the
question is:
Can you confirm in which SLES 12 release the function upmap is supported?
And is it safe to execute ceph osd set-require-min-compat-client
luminous --yes-i-really-mean-it? What happens to the clients then?
Regards
Thomas
16 Sep
16 Sep
10:06 a.m.
On 9/16/19 3:59 PM, Thomas wrote:
I tried to run this command with failure:
root@ld3955:/mnt/rbd# ceph osd set-require-min-compat-client luminous
Error EPERM: cannot set require_min_compat_client to luminous: 6
connected client(s) look like jewel (missing 0xa00000000200000); 19
connected client(s) look like jewel (missing 0x800000000000000); 23
connected client(s) look like jewel (missing 0x800000000000000); add
--yes-i-really-mean-it to do it anyway
root@ld3955:/mnt/rbd# ceph features
{
"mon": [
{
"features": "0x3ffddff8ffacffff",
"release": "luminous",
"num": 3
}
],
"mds": [
{
"features": "0x3ffddff8ffacffff",
"release": "luminous",
"num": 2
}
],
"osd": [
{
"features": "0x3ffddff8ffacffff",
"release": "luminous",
"num": 368
}
],
"client": [
{
"features": "0x40106b84a842a42",
"release": "jewel",
"num": 6
},
{
"features": "0x27018eb84aa42a52",
"release": "jewel",
"num": 19
},
{
"features": "0x27018fb86aa42ada",
"release": "jewel",
"num": 23
},
{
"features": "0x3ffddff8ffacffff",
"release": "luminous",
"num": 35
}
],
"mgr": [
{
"features": "0x3ffddff8ffacffff",
"release": "luminous",
"num": 3
}
]
}
But I think the number of clients older than Luminous is incorrect.
I'm running 98% of the clients with SLES 12SPx, and therefore the
question is:
Can you confirm in which SLES 12 release the function upmap is supported?
And is it safe to execute ceph osd set-require-min-compat-client
luminous --yes-i-really-mean-it? What happens to the clients then?
Your kernel should be 4.13+ to be upmap compat. Then you should tell
`--yes-i-really-mean-it` to apply changes.
k
10:11 a.m.
Am 16.09.19 um 11:06 schrieb Konstantin Shalygin:
On 9/16/19 3:59 PM, Thomas wrote:
SLES 12 is still on 3.12 I think. However, the question is probably rather:
Does SUSE backport things such as pg-upmap to older Kernels, such as RedHat does?
Cheers,
Oliver
I tried to run this command with failure:
root@ld3955:/mnt/rbd# ceph osd set-require-min-compat-client luminous
Error EPERM: cannot set require_min_compat_client to luminous: 6
connected client(s) look like jewel (missing 0xa00000000200000); 19
connected client(s) look like jewel (missing 0x800000000000000); 23
connected client(s) look like jewel (missing 0x800000000000000); add
--yes-i-really-mean-it to do it anyway
root@ld3955:/mnt/rbd# ceph features
{
"mon": [
{
"features": "0x3ffddff8ffacffff",
"release": "luminous",
"num": 3
}
],
"mds": [
{
"features": "0x3ffddff8ffacffff",
"release": "luminous",
"num": 2
}
],
"osd": [
{
"features": "0x3ffddff8ffacffff",
"release": "luminous",
"num": 368
}
],
"client": [
{
"features": "0x40106b84a842a42",
"release": "jewel",
"num": 6
},
{
"features": "0x27018eb84aa42a52",
"release": "jewel",
"num": 19
},
{
"features": "0x27018fb86aa42ada",
"release": "jewel",
"num": 23
},
{
"features": "0x3ffddff8ffacffff",
"release": "luminous",
"num": 35
}
],
"mgr": [
{
"features": "0x3ffddff8ffacffff",
"release": "luminous",
"num": 3
}
]
}
But I think the number of clients older than Luminous is incorrect.
I'm running 98% of the clients with SLES 12SPx, and therefore the
question is:
Can you confirm in which SLES 12 release the function upmap is supported?
And is it safe to execute ceph osd set-require-min-compat-client
luminous --yes-i-really-mean-it? What happens to the clients then?
Your kernel should be 4.13+ to be upmap compat. Then you should tell
`--yes-i-really-mean-it` to apply changes.
k
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
1:18 p.m.
Hello,
the current kernel with SLES 12SP3 is:
ld3195:~ # uname -r
4.4.176-94.88-default
Assuming that this kernel is not supporting upmap, do you recommend to
use balance mode crush-compat then?
Regards
Thomas
Am 16.09.2019 um 11:11 schrieb Oliver Freyermuth:
Am 16.09.19 um 11:06 schrieb Konstantin Shalygin:
On 9/16/19 3:59 PM, Thomas wrote:
SLES 12 is still on 3.12 I think. However, the question is probably
rather:
Does SUSE backport things such as pg-upmap to older Kernels, such as
RedHat does?
Cheers,
Oliver
I tried to run this command with failure:
root@ld3955:/mnt/rbd# ceph osd set-require-min-compat-client luminous
Error EPERM: cannot set require_min_compat_client to luminous: 6
connected client(s) look like jewel (missing 0xa00000000200000); 19
connected client(s) look like jewel (missing 0x800000000000000); 23
connected client(s) look like jewel (missing 0x800000000000000); add
--yes-i-really-mean-it to do it anyway
root@ld3955:/mnt/rbd# ceph features
{
"mon": [
{
"features": "0x3ffddff8ffacffff",
"release": "luminous",
"num": 3
}
],
"mds": [
{
"features": "0x3ffddff8ffacffff",
"release": "luminous",
"num": 2
}
],
"osd": [
{
"features": "0x3ffddff8ffacffff",
"release": "luminous",
"num": 368
}
],
"client": [
{
"features": "0x40106b84a842a42",
"release": "jewel",
"num": 6
},
{
"features": "0x27018eb84aa42a52",
"release": "jewel",
"num": 19
},
{
"features": "0x27018fb86aa42ada",
"release": "jewel",
"num": 23
},
{
"features": "0x3ffddff8ffacffff",
"release": "luminous",
"num": 35
}
],
"mgr": [
{
"features": "0x3ffddff8ffacffff",
"release": "luminous",
"num": 3
}
]
}
But I think the number of clients older than Luminous is incorrect.
I'm running 98% of the clients with SLES 12SPx, and therefore the
question is:
Can you confirm in which SLES 12 release the function upmap is
supported?
And is it safe to execute ceph osd set-require-min-compat-client
luminous --yes-i-really-mean-it? What happens to the clients then?
Your kernel should be 4.13+ to be upmap compat. Then you should tell
`--yes-i-really-mean-it` to apply changes.
k
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
2:23 p.m.
On Mon, Sep 16, 2019 at 2:20 PM Thomas Schneider <74cmonty(a)gmail.com> wrote:
Hello,
the current kernel with SLES 12SP3 is:
ld3195:~ # uname -r
4.4.176-94.88-default
Assuming that this kernel is not supporting upmap, do you recommend to
use balance mode crush-compat then?
Hi Thomas,
Among the clients listed in your output, only 6 don't support upmap
(those with features 0x40106b84a842a42). The rest do.
Regards
Thomas
Am 16.09.2019 um 11:11 schrieb Oliver Freyermuth:
> Am 16.09.19 um 11:06 schrieb Konstantin Shalygin:
>> On 9/16/19 3:59 PM, Thomas wrote:
>>> I tried to run this command with failure:
>>> root@ld3955:/mnt/rbd# ceph osd set-require-min-compat-client luminous
>>> Error EPERM: cannot set require_min_compat_client to luminous: 6
>>> connected client(s) look like jewel (missing 0xa00000000200000); 19
>>> connected client(s) look like jewel (missing 0x800000000000000); 23
>>> connected client(s) look like jewel (missing 0x800000000000000); add
>>> --yes-i-really-mean-it to do it anyway
>>>
>>> root@ld3955:/mnt/rbd# ceph features
>>> {
>>> "mon": [
>>> {
>>> "features": "0x3ffddff8ffacffff",
>>> "release": "luminous",
>>> "num": 3
>>> }
>>> ],
>>> "mds": [
>>> {
>>> "features": "0x3ffddff8ffacffff",
>>> "release": "luminous",
>>> "num": 2
>>> }
>>> ],
>>> "osd": [
>>> {
>>> "features": "0x3ffddff8ffacffff",
>>> "release": "luminous",
>>> "num": 368
>>> }
>>> ],
>>> "client": [
>>> {
>>> "features": "0x40106b84a842a42",
>>> "release": "jewel",
>>> "num": 6
>>> },
>>> {
>>> "features": "0x27018eb84aa42a52",
>>> "release": "jewel",
>>> "num": 19
>>> },
>>> {
>>> "features": "0x27018fb86aa42ada",
>>> "release": "jewel",
>>> "num": 23
>>> },
>>> {
>>> "features": "0x3ffddff8ffacffff",
>>> "release": "luminous",
>>> "num": 35
>>> }
>>> ],
>>> "mgr": [
>>> {
>>> "features": "0x3ffddff8ffacffff",
>>> "release": "luminous",
>>> "num": 3
>>> }
>>> ]
>>> }
Thanks,
Ilya
2:56 p.m.
Bit 21 in the features bitfield is upmap support
Paul
--
Paul Emmerich
Looking for help with your Ceph cluster? Contact us at https://croit.io
croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90
On Mon, Sep 16, 2019 at 3:21 PM Ilya Dryomov <idryomov(a)gmail.com> wrote:
>
> On Mon, Sep 16, 2019 at 2:20 PM Thomas Schneider <74cmonty(a)gmail.com> wrote:
> >
> > Hello,
> >
> > the current kernel with SLES 12SP3 is:
> > ld3195:~ # uname -r
> > 4.4.176-94.88-default
> >
> >
> > Assuming that this kernel is not supporting upmap, do you recommend to
> > use balance mode crush-compat then?
>
> Hi Thomas,
>
> Among the clients listed in your output, only 6 don't support upmap
> (those with features 0x40106b84a842a42). The rest do.
>
> >
> > Regards
> > Thomas
> >
> >
> > Am 16.09.2019 um 11:11 schrieb Oliver Freyermuth:
> > > Am 16.09.19 um 11:06 schrieb Konstantin Shalygin:
> > >> On 9/16/19 3:59 PM, Thomas wrote:
> > >>> I tried to run this command with failure:
> > >>> root@ld3955:/mnt/rbd# ceph osd set-require-min-compat-client
luminous
> > >>> Error EPERM: cannot set require_min_compat_client to luminous: 6
> > >>> connected client(s) look like jewel (missing 0xa00000000200000);
19
> > >>> connected client(s) look like jewel (missing 0x800000000000000);
23
> > >>> connected client(s) look like jewel (missing 0x800000000000000);
add
> > >>> --yes-i-really-mean-it to do it anyway
> > >>>
> > >>> root@ld3955:/mnt/rbd# ceph features
> > >>> {
> > >>> "mon": [
> > >>> {
> > >>> "features": "0x3ffddff8ffacffff",
> > >>> "release": "luminous",
> > >>> "num": 3
> > >>> }
> > >>> ],
> > >>> "mds": [
> > >>> {
> > >>> "features": "0x3ffddff8ffacffff",
> > >>> "release": "luminous",
> > >>> "num": 2
> > >>> }
> > >>> ],
> > >>> "osd": [
> > >>> {
> > >>> "features": "0x3ffddff8ffacffff",
> > >>> "release": "luminous",
> > >>> "num": 368
> > >>> }
> > >>> ],
> > >>> "client": [
> > >>> {
> > >>> "features": "0x40106b84a842a42",
> > >>> "release": "jewel",
> > >>> "num": 6
> > >>> },
> > >>> {
> > >>> "features": "0x27018eb84aa42a52",
> > >>> "release": "jewel",
> > >>> "num": 19
> > >>> },
> > >>> {
> > >>> "features": "0x27018fb86aa42ada",
> > >>> "release": "jewel",
> > >>> "num": 23
> > >>> },
> > >>> {
> > >>> "features": "0x3ffddff8ffacffff",
> > >>> "release": "luminous",
> > >>> "num": 35
> > >>> }
> > >>> ],
> > >>> "mgr": [
> > >>> {
> > >>> "features": "0x3ffddff8ffacffff",
> > >>> "release": "luminous",
> > >>> "num": 3
> > >>> }
> > >>> ]
> > >>> }
>
> Thanks,
>
> Ilya
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
3:40 p.m.
Hi,
thanks for your valuable input.
Question:
Can I get more information of the 6 clients (those with features
0x40106b84a842a42), e.g. IP, that allows me to identify it easily?
Regards
Thomas
Am 16.09.2019 um 15:56 schrieb Paul Emmerich:
Bit 21 in the features bitfield is upmap support
Paul
Am 16.09.2019 um 15:23 schrieb Ilya Dryomov:
On Mon, Sep 16, 2019 at 2:20 PM Thomas Schneider
<74cmonty(a)gmail.com> wrote:
Hello,
the current kernel with SLES 12SP3 is:
ld3195:~ # uname -r
4.4.176-94.88-default
Assuming that this kernel is not supporting upmap, do you recommend to
use balance mode crush-compat then?
Hi Thomas,
Among the clients listed in your output, only 6 don't support upmap
(those with features 0x40106b84a842a42). The rest do.
Regards
Thomas
Am 16.09.2019 um 11:11 schrieb Oliver Freyermuth:
> Am 16.09.19 um 11:06 schrieb Konstantin Shalygin:
>> On 9/16/19 3:59 PM, Thomas wrote:
>>> I tried to run this command with failure:
>>> root@ld3955:/mnt/rbd# ceph osd set-require-min-compat-client luminous
>>> Error EPERM: cannot set require_min_compat_client to luminous: 6
>>> connected client(s) look like jewel (missing 0xa00000000200000); 19
>>> connected client(s) look like jewel (missing 0x800000000000000); 23
>>> connected client(s) look like jewel (missing 0x800000000000000); add
>>> --yes-i-really-mean-it to do it anyway
>>>
>>> root@ld3955:/mnt/rbd# ceph features
>>> {
>>> "mon": [
>>> {
>>> "features": "0x3ffddff8ffacffff",
>>> "release": "luminous",
>>> "num": 3
>>> }
>>> ],
>>> "mds": [
>>> {
>>> "features": "0x3ffddff8ffacffff",
>>> "release": "luminous",
>>> "num": 2
>>> }
>>> ],
>>> "osd": [
>>> {
>>> "features": "0x3ffddff8ffacffff",
>>> "release": "luminous",
>>> "num": 368
>>> }
>>> ],
>>> "client": [
>>> {
>>> "features": "0x40106b84a842a42",
>>> "release": "jewel",
>>> "num": 6
>>> },
>>> {
>>> "features": "0x27018eb84aa42a52",
>>> "release": "jewel",
>>> "num": 19
>>> },
>>> {
>>> "features": "0x27018fb86aa42ada",
>>> "release": "jewel",
>>> "num": 23
>>> },
>>> {
>>> "features": "0x3ffddff8ffacffff",
>>> "release": "luminous",
>>> "num": 35
>>> }
>>> ],
>>> "mgr": [
>>> {
>>> "features": "0x3ffddff8ffacffff",
>>> "release": "luminous",
>>> "num": 3
>>> }
>>> ]
>>> }
Thanks,
Ilya
3:56 p.m.
On Mon, Sep 16, 2019 at 4:40 PM Thomas Schneider <74cmonty(a)gmail.com> wrote:
Hi,
thanks for your valuable input.
Question:
Can I get more information of the 6 clients (those with features
0x40106b84a842a42), e.g. IP, that allows me to identify it easily?
Yes, although it's not integrated into "ceph features". Log into
a monitor node and run "ceph daemon mon.a sessions" (mon.a is the name
of the monitor, substitute accordingly).
Thanks,
Ilya
4:10 p.m.
Wonderbra.
I found some relevant sessions on 2 of 3 monitor nodes.
And I found some others:
root@ld5505:~# ceph daemon mon.ld5505 sessions | grep 0x40106b84a842a42
root@ld5505:~# ceph daemon mon.ld5505 sessions | grep -v luminous
[
"MonSession(client.32679861 v1:10.97.206.92:0/1183647891 is open
allow *, features 0x27018fb86aa42ada (jewel))",
"MonSession(client.32692978 v1:10.97.206.91:0/3689092992 is open
allow *, features 0x27018fb86aa42ada (jewel))",
"MonSession(client.11935413 v1:10.96.6.116:0/3187655474 is open
allow r, features 0x27018eb84aa42a52 (jewel))",
"MonSession(client.3941901 v1:10.76.179.23:0/2967896845 is open
allow r, features 0x27018fb86aa42ada (jewel))",
"MonSession(client.28313343 v1:10.76.177.108:0/1303617860 is open
allow r, features 0x27018fb86aa42ada (jewel))",
"MonSession(client.29311725 v1:10.97.206.94:0/224438037 is open
allow *, features 0x27018fb86aa42ada (jewel))",
"MonSession(client.4535833 v1:10.76.177.133:0/1269608815 is open
allow r, features 0x27018fb86aa42ada (jewel))",
"MonSession(client.3919902 v1:10.96.4.243:0/293623521 is open allow
r, features 0x27018eb84aa42a52 (jewel))",
"MonSession(client.35678944 v1:10.76.179.211:0/4218086982 is open
allow r, features 0x27018eb84aa42a52 (jewel))",
"MonSession(client.35751316 v1:10.76.179.30:0/1348696702 is open
allow r, features 0x27018eb84aa42a52 (jewel))",
"MonSession(client.28246527 v1:10.96.4.228:0/1495661381 is open
allow r, features 0x27018fb86aa42ada (jewel))",
"MonSession(client.3917843 v1:10.76.179.22:0/489863209 is open allow
r, features 0x27018fb86aa42ada (jewel))",
"MonSession(unknown.0 - is open allow r, features 0x27018eb84aa42a52
(jewel))",
]
Would it make sense to shutdown these clients, too?
What confuses me is that the list includes clients that belong to the
Ceph cluster, namely 10.97.206.0/24.
All nodes of the Ceph cluster are identical in terms of OS, kernel, Ceph.
Regards
Thomas
Am 16.09.2019 um 16:56 schrieb Ilya Dryomov:
On Mon, Sep 16, 2019 at 4:40 PM Thomas Schneider
<74cmonty(a)gmail.com> wrote:
Hi,
thanks for your valuable input.
Question:
Can I get more information of the 6 clients (those with features
0x40106b84a842a42), e.g. IP, that allows me to identify it easily?
Yes, although
it's not integrated into "ceph features". Log into
a monitor node and run "ceph daemon mon.a sessions" (mon.a is the name
of the monitor, substitute accordingly).
Thanks,
Ilya
4:36 p.m.
On Mon, Sep 16, 2019 at 5:10 PM Thomas Schneider <74cmonty(a)gmail.com> wrote:
Wonderbra.
I found some relevant sessions on 2 of 3 monitor nodes.
And I found some others:
root@ld5505:~# ceph daemon mon.ld5505 sessions | grep 0x40106b84a842a42
root@ld5505:~# ceph daemon mon.ld5505 sessions | grep -v luminous
[
"MonSession(client.32679861 v1:10.97.206.92:0/1183647891 is open
allow *, features 0x27018fb86aa42ada (jewel))",
"MonSession(client.32692978 v1:10.97.206.91:0/3689092992 is open
allow *, features 0x27018fb86aa42ada (jewel))",
"MonSession(client.11935413 v1:10.96.6.116:0/3187655474 is open
allow r, features 0x27018eb84aa42a52 (jewel))",
"MonSession(client.3941901 v1:10.76.179.23:0/2967896845 is open
allow r, features 0x27018fb86aa42ada (jewel))",
"MonSession(client.28313343 v1:10.76.177.108:0/1303617860 is open
allow r, features 0x27018fb86aa42ada (jewel))",
"MonSession(client.29311725 v1:10.97.206.94:0/224438037 is open
allow *, features 0x27018fb86aa42ada (jewel))",
"MonSession(client.4535833 v1:10.76.177.133:0/1269608815 is open
allow r, features 0x27018fb86aa42ada (jewel))",
"MonSession(client.3919902 v1:10.96.4.243:0/293623521 is open allow
r, features 0x27018eb84aa42a52 (jewel))",
"MonSession(client.35678944 v1:10.76.179.211:0/4218086982 is open
allow r, features 0x27018eb84aa42a52 (jewel))",
"MonSession(client.35751316 v1:10.76.179.30:0/1348696702 is open
allow r, features 0x27018eb84aa42a52 (jewel))",
"MonSession(client.28246527 v1:10.96.4.228:0/1495661381 is open
allow r, features 0x27018fb86aa42ada (jewel))",
"MonSession(client.3917843 v1:10.76.179.22:0/489863209 is open allow
r, features 0x27018fb86aa42ada (jewel))",
"MonSession(unknown.0 - is open allow r, features 0x27018eb84aa42a52
(jewel))",
]
Would it make sense to shutdown these clients, too?
What confuses me is that the list includes clients that belong to the
Ceph cluster, namely 10.97.206.0/24.
All nodes of the Ceph cluster are identical in terms of OS, kernel, Ceph.
The above output seems consistent with your "ceph features" output: it
lists clients with features 0x27018eb84aa42a52 and 0x27018fb86aa42ada.
Like I said in my previous email, both of these support upmap.
If you temporarily shut them down, set-require-min-compat-client will
work without --yes-i-really-mean-it.
Thanks,
Ilya
17 Sep
17 Sep
1:24 p.m.
Hi,
I was checking the connected sessions on all of my 3 MON nodes with this
command:
ceph daemon mon.<id> sessions | grep -v luminous
This returns the following 2 clients featuring
0x27018fb86aa42ada
and
0x27018eb84aa42a52
This is mapping to OS / Kernel:
0x27018fb86aa42ada
Debian 10.1
Kernel 5.0.21-1-pve
and
SLES 12SP4
Kernel 4.12.14-95.13-default
0x27018eb84aa42a52
SLES 12SP3
4.4.176-94.88-default
and
SLES 12SP3
4.4.180-94.97-default
and
SLES 12SP4
4.4.156-94.64-default
Based on your previous information 0x27018fb86aa42ada and
0x27018eb84aa42a52 is ready for upmap.
But then I wonder why the output of ceph daemon mon.<id> sessions marks
these clients as "jewel"?
I have checked the installed ceph version on each client and can confirm
that it is:
ceph version 12.2 luminous
This would drive the conclusion that the ouput of ceph daemon mon.<id>
sessions is pointing incorrectly to "jewel".
Regards
Thomas
Am 16.09.2019 um 17:36 schrieb Ilya Dryomov:
On Mon, Sep 16, 2019 at 5:10 PM Thomas Schneider
<74cmonty(a)gmail.com> wrote:
Wonderbra.
I found some relevant sessions on 2 of 3 monitor nodes.
And I found some others:
root@ld5505:~# ceph daemon mon.ld5505 sessions | grep 0x40106b84a842a42
root@ld5505:~# ceph daemon mon.ld5505 sessions | grep -v luminous
[
"MonSession(client.32679861 v1:10.97.206.92:0/1183647891 is open
allow *, features 0x27018fb86aa42ada (jewel))",
"MonSession(client.32692978 v1:10.97.206.91:0/3689092992 is open
allow *, features 0x27018fb86aa42ada (jewel))",
"MonSession(client.11935413 v1:10.96.6.116:0/3187655474 is open
allow r, features 0x27018eb84aa42a52 (jewel))",
"MonSession(client.3941901 v1:10.76.179.23:0/2967896845 is open
allow r, features 0x27018fb86aa42ada (jewel))",
"MonSession(client.28313343 v1:10.76.177.108:0/1303617860 is open
allow r, features 0x27018fb86aa42ada (jewel))",
"MonSession(client.29311725 v1:10.97.206.94:0/224438037 is open
allow *, features 0x27018fb86aa42ada (jewel))",
"MonSession(client.4535833 v1:10.76.177.133:0/1269608815 is open
allow r, features 0x27018fb86aa42ada (jewel))",
"MonSession(client.3919902 v1:10.96.4.243:0/293623521 is open allow
r, features 0x27018eb84aa42a52 (jewel))",
"MonSession(client.35678944 v1:10.76.179.211:0/4218086982 is open
allow r, features 0x27018eb84aa42a52 (jewel))",
"MonSession(client.35751316 v1:10.76.179.30:0/1348696702 is open
allow r, features 0x27018eb84aa42a52 (jewel))",
"MonSession(client.28246527 v1:10.96.4.228:0/1495661381 is open
allow r, features 0x27018fb86aa42ada (jewel))",
"MonSession(client.3917843 v1:10.76.179.22:0/489863209 is open allow
r, features 0x27018fb86aa42ada (jewel))",
"MonSession(unknown.0 - is open allow r, features 0x27018eb84aa42a52
(jewel))",
]
Would it make sense to shutdown these clients, too?
What confuses me is that the list includes clients that belong to the
Ceph cluster, namely 10.97.206.0/24.
All nodes of the Ceph cluster are identical in terms of OS, kernel, Ceph.
The
above output seems consistent with your "ceph features" output: it
lists clients with features 0x27018eb84aa42a52 and 0x27018fb86aa42ada.
Like I said in my previous email, both of these support upmap.
If you temporarily shut them down, set-require-min-compat-client will
work without --yes-i-really-mean-it.
Thanks,
Ilya 
1:53 p.m.
Hi,
I have checked the installed ceph version on each
client and can confirm
that it is:
ceph version 12.2 luminous
This would drive the conclusion that the ouput of ceph daemon mon.<id>
sessions is pointing incorrectly to "jewel".
I have the same in an upgraded Nautilus cluster. There are a couple of
"unknown" clients and some Jewel clients, but checking those clients
they all have at least Luminous. I rebooted some of those clients,
too, but this didn't change anything. The min-compat-client already
has been forced to luminous (knowing that in fact there are no jewel
clients):
host1:~ # ceph osd get-require-min-compat-client
luminous
I'd be curious how to get rid of the jewel client sessions.
Regards,
Eugen
Zitat von Thomas Schneider <74cmonty(a)gmail.com>om>:
> Hi,
>
> I was checking the connected sessions on all of my 3 MON nodes with this
> command:
> ceph daemon mon.<id> sessions | grep -v luminous
>
> This returns the following 2 clients featuring
> 0x27018fb86aa42ada
> and
> 0x27018eb84aa42a52
>
> This is mapping to OS / Kernel:
> 0x27018fb86aa42ada
> Debian 10.1
> Kernel 5.0.21-1-pve
> and
> SLES 12SP4
> Kernel 4.12.14-95.13-default
>
> 0x27018eb84aa42a52
> SLES 12SP3
> 4.4.176-94.88-default
> and
> SLES 12SP3
> 4.4.180-94.97-default
> and
> SLES 12SP4
> 4.4.156-94.64-default
>
> Based on your previous information 0x27018fb86aa42ada and
> 0x27018eb84aa42a52 is ready for upmap.
> But then I wonder why the output of ceph daemon mon.<id> sessions marks
> these clients as "jewel"?
>
I have checked the installed ceph version on each
client and can confirm
that it is:
ceph version 12.2 luminous
This would drive the conclusion that the ouput of ceph daemon mon.<id>
sessions is pointing incorrectly to "jewel".
>
> Regards
> Thomas
>
> Am 16.09.2019 um 17:36 schrieb Ilya Dryomov:
>> On Mon, Sep 16, 2019 at 5:10 PM Thomas Schneider <74cmonty(a)gmail.com>
wrote:
>>> Wonderbra.
>>>
>>> I found some relevant sessions on 2 of 3 monitor nodes.
>>> And I found some others:
>>> root@ld5505:~# ceph daemon mon.ld5505 sessions | grep 0x40106b84a842a42
>>> root@ld5505:~# ceph daemon mon.ld5505 sessions | grep -v luminous
>>> [
>>> "MonSession(client.32679861 v1:10.97.206.92:0/1183647891 is open
>>> allow *, features 0x27018fb86aa42ada (jewel))",
>>> "MonSession(client.32692978 v1:10.97.206.91:0/3689092992 is open
>>> allow *, features 0x27018fb86aa42ada (jewel))",
>>> "MonSession(client.11935413 v1:10.96.6.116:0/3187655474 is open
>>> allow r, features 0x27018eb84aa42a52 (jewel))",
>>> "MonSession(client.3941901 v1:10.76.179.23:0/2967896845 is open
>>> allow r, features 0x27018fb86aa42ada (jewel))",
>>> "MonSession(client.28313343 v1:10.76.177.108:0/1303617860 is open
>>> allow r, features 0x27018fb86aa42ada (jewel))",
>>> "MonSession(client.29311725 v1:10.97.206.94:0/224438037 is open
>>> allow *, features 0x27018fb86aa42ada (jewel))",
>>> "MonSession(client.4535833 v1:10.76.177.133:0/1269608815 is open
>>> allow r, features 0x27018fb86aa42ada (jewel))",
>>> "MonSession(client.3919902 v1:10.96.4.243:0/293623521 is open allow
>>> r, features 0x27018eb84aa42a52 (jewel))",
>>> "MonSession(client.35678944 v1:10.76.179.211:0/4218086982 is open
>>> allow r, features 0x27018eb84aa42a52 (jewel))",
>>> "MonSession(client.35751316 v1:10.76.179.30:0/1348696702 is open
>>> allow r, features 0x27018eb84aa42a52 (jewel))",
>>> "MonSession(client.28246527 v1:10.96.4.228:0/1495661381 is open
>>> allow r, features 0x27018fb86aa42ada (jewel))",
>>> "MonSession(client.3917843 v1:10.76.179.22:0/489863209 is open
allow
>>> r, features 0x27018fb86aa42ada (jewel))",
>>> "MonSession(unknown.0 - is open allow r, features
0x27018eb84aa42a52
>>> (jewel))",
>>> ]
>>>
>>> Would it make sense to shutdown these clients, too?
>>>
>>> What confuses me is that the list includes clients that belong to the
>>> Ceph cluster, namely 10.97.206.0/24.
>>> All nodes of the Ceph cluster are identical in terms of OS, kernel, Ceph.
>> The above output seems consistent with your "ceph features" output: it
>> lists clients with features 0x27018eb84aa42a52 and 0x27018fb86aa42ada.
>> Like I said in my previous email, both of these support upmap.
>>
>> If you temporarily shut them down, set-require-min-compat-client will
>> work without --yes-i-really-mean-it.
>>
>> Thanks,
>>
>> Ilya
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
2:25 p.m.
On Tue, Sep 17, 2019 at 2:54 PM Eugen Block <eblock(a)nde.ag> wrote:
Hi,
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/JP44CGR4FO…
Thanks,
Ilya
I have checked the installed ceph version on each
client and can confirm
that it is:
ceph version 12.2 luminous
This would drive the conclusion that the ouput of ceph daemon mon.<id>
sessions is pointing incorrectly to "jewel".
I have the same in an upgraded Nautilus cluster. There are a couple of
"unknown" clients and some Jewel clients, but checking those clients
they all have at least Luminous. I rebooted some of those clients,
too, but this didn't change anything. The min-compat-client already
has been forced to luminous (knowing that in fact there are no jewel
clients):
host1:~ # ceph osd get-require-min-compat-client
luminous
I'd be curious how to get rid of the jewel client sessions.
3:08 p.m.
Thank you very much, Ilya!
Zitat von Ilya Dryomov <idryomov(a)gmail.com>om>:
On Tue, Sep 17, 2019 at 2:54 PM Eugen Block
<eblock(a)nde.ag> wrote:
Hi,
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/JP44CGR4FO…
Thanks,
Ilya I have checked the installed ceph version on each
client and can confirm
that it is:
ceph version 12.2 luminous
This would drive the conclusion that the ouput of ceph daemon mon.<id>
sessions is pointing incorrectly to "jewel".
I have the same in an upgraded Nautilus cluster. There are a couple of
"unknown" clients and some Jewel clients, but checking those clients
they all have at least Luminous. I rebooted some of those clients,
too, but this didn't change anything. The min-compat-client already
has been forced to luminous (knowing that in fact there are no jewel
clients):
host1:~ # ceph osd get-require-min-compat-client
luminous
I'd be curious how to get rid of the jewel client sessions. 
8:48 a.m.
Thomas <74cmonty(a)gmail.com> writes:
But I think the number of clients older than Luminous
is incorrect.
I'm running 98% of the clients with SLES 12SPx, and therefore the
question is:
Can you confirm in which SLES 12 release the function upmap is supported?
I can confirm that SLE12-SP3 *does* support upmap; if you're running
older clients (e.g., SP2), upmap is not supported. As Ilya already
mentioned, client 6 doesn't seem to support this feature, so I'm
assuming you're using SP2 or older there.
Cheers,
--
Luis
> And is it safe to execute ceph osd set-require-min-compat-client
> luminous --yes-i-really-mean-it? What happens to the clients then?
>
> Regards
> Thomas
>
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
1706
days inactive
1707
days old
14 comments
8 participants
participants (8)
-
Eugen Block -
Ilya Dryomov -
Konstantin Shalygin -
Luis Henriques -
Oliver Freyermuth -
Paul Emmerich -
Thomas -
Thomas Schneider