upmap supported in SLES 12SPx
Hi, I tried to run this command with failure: root@ld3955:/mnt/rbd# ceph osd set-require-min-compat-client luminous Error EPERM: cannot set require_min_compat_client to luminous: 6 connected client(s) look like jewel (missing 0xa00000000200000); 19 connected client(s) look like jewel (missing 0x800000000000000); 23 connected client(s) look like jewel (missing 0x800000000000000); add --yes-i-really-mean-it to do it anyway root@ld3955:/mnt/rbd# ceph features { "mon": [ { "features": "0x3ffddff8ffacffff", "release": "luminous", "num": 3 } ], "mds": [ { "features": "0x3ffddff8ffacffff", "release": "luminous", "num": 2 } ], "osd": [ { "features": "0x3ffddff8ffacffff", "release": "luminous", "num": 368 } ], "client": [ { "features": "0x40106b84a842a42", "release": "jewel", "num": 6 }, { "features": "0x27018eb84aa42a52", "release": "jewel", "num": 19 }, { "features": "0x27018fb86aa42ada", "release": "jewel", "num": 23 }, { "features": "0x3ffddff8ffacffff", "release": "luminous", "num": 35 } ], "mgr": [ { "features": "0x3ffddff8ffacffff", "release": "luminous", "num": 3 } ] } But I think the number of clients older than Luminous is incorrect. I'm running 98% of the clients with SLES 12SPx, and therefore the question is: Can you confirm in which SLES 12 release the function upmap is supported? And is it safe to execute ceph osd set-require-min-compat-client luminous --yes-i-really-mean-it? What happens to the clients then? Regards Thomas
On 9/16/19 3:59 PM, Thomas wrote:
I tried to run this command with failure: root@ld3955:/mnt/rbd# ceph osd set-require-min-compat-client luminous Error EPERM: cannot set require_min_compat_client to luminous: 6 connected client(s) look like jewel (missing 0xa00000000200000); 19 connected client(s) look like jewel (missing 0x800000000000000); 23 connected client(s) look like jewel (missing 0x800000000000000); add --yes-i-really-mean-it to do it anyway
root@ld3955:/mnt/rbd# ceph features { "mon": [ { "features": "0x3ffddff8ffacffff", "release": "luminous", "num": 3 } ], "mds": [ { "features": "0x3ffddff8ffacffff", "release": "luminous", "num": 2 } ], "osd": [ { "features": "0x3ffddff8ffacffff", "release": "luminous", "num": 368 } ], "client": [ { "features": "0x40106b84a842a42", "release": "jewel", "num": 6 }, { "features": "0x27018eb84aa42a52", "release": "jewel", "num": 19 }, { "features": "0x27018fb86aa42ada", "release": "jewel", "num": 23 }, { "features": "0x3ffddff8ffacffff", "release": "luminous", "num": 35 } ], "mgr": [ { "features": "0x3ffddff8ffacffff", "release": "luminous", "num": 3 } ] }
But I think the number of clients older than Luminous is incorrect.
I'm running 98% of the clients with SLES 12SPx, and therefore the question is: Can you confirm in which SLES 12 release the function upmap is supported? And is it safe to execute ceph osd set-require-min-compat-client luminous --yes-i-really-mean-it? What happens to the clients then?
Your kernel should be 4.13+ to be upmap compat. Then you should tell `--yes-i-really-mean-it` to apply changes. k
Am 16.09.19 um 11:06 schrieb Konstantin Shalygin:
On 9/16/19 3:59 PM, Thomas wrote:
I tried to run this command with failure: root@ld3955:/mnt/rbd# ceph osd set-require-min-compat-client luminous Error EPERM: cannot set require_min_compat_client to luminous: 6 connected client(s) look like jewel (missing 0xa00000000200000); 19 connected client(s) look like jewel (missing 0x800000000000000); 23 connected client(s) look like jewel (missing 0x800000000000000); add --yes-i-really-mean-it to do it anyway
root@ld3955:/mnt/rbd# ceph features { "mon": [ { "features": "0x3ffddff8ffacffff", "release": "luminous", "num": 3 } ], "mds": [ { "features": "0x3ffddff8ffacffff", "release": "luminous", "num": 2 } ], "osd": [ { "features": "0x3ffddff8ffacffff", "release": "luminous", "num": 368 } ], "client": [ { "features": "0x40106b84a842a42", "release": "jewel", "num": 6 }, { "features": "0x27018eb84aa42a52", "release": "jewel", "num": 19 }, { "features": "0x27018fb86aa42ada", "release": "jewel", "num": 23 }, { "features": "0x3ffddff8ffacffff", "release": "luminous", "num": 35 } ], "mgr": [ { "features": "0x3ffddff8ffacffff", "release": "luminous", "num": 3 } ] }
But I think the number of clients older than Luminous is incorrect.
I'm running 98% of the clients with SLES 12SPx, and therefore the question is: Can you confirm in which SLES 12 release the function upmap is supported? And is it safe to execute ceph osd set-require-min-compat-client luminous --yes-i-really-mean-it? What happens to the clients then?
Your kernel should be 4.13+ to be upmap compat. Then you should tell `--yes-i-really-mean-it` to apply changes.
SLES 12 is still on 3.12 I think. However, the question is probably rather: Does SUSE backport things such as pg-upmap to older Kernels, such as RedHat does? Cheers, Oliver
k _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello, the current kernel with SLES 12SP3 is: ld3195:~ # uname -r 4.4.176-94.88-default Assuming that this kernel is not supporting upmap, do you recommend to use balance mode crush-compat then? Regards Thomas Am 16.09.2019 um 11:11 schrieb Oliver Freyermuth:
Am 16.09.19 um 11:06 schrieb Konstantin Shalygin:
On 9/16/19 3:59 PM, Thomas wrote:
I tried to run this command with failure: root@ld3955:/mnt/rbd# ceph osd set-require-min-compat-client luminous Error EPERM: cannot set require_min_compat_client to luminous: 6 connected client(s) look like jewel (missing 0xa00000000200000); 19 connected client(s) look like jewel (missing 0x800000000000000); 23 connected client(s) look like jewel (missing 0x800000000000000); add --yes-i-really-mean-it to do it anyway
root@ld3955:/mnt/rbd# ceph features { "mon": [ { "features": "0x3ffddff8ffacffff", "release": "luminous", "num": 3 } ], "mds": [ { "features": "0x3ffddff8ffacffff", "release": "luminous", "num": 2 } ], "osd": [ { "features": "0x3ffddff8ffacffff", "release": "luminous", "num": 368 } ], "client": [ { "features": "0x40106b84a842a42", "release": "jewel", "num": 6 }, { "features": "0x27018eb84aa42a52", "release": "jewel", "num": 19 }, { "features": "0x27018fb86aa42ada", "release": "jewel", "num": 23 }, { "features": "0x3ffddff8ffacffff", "release": "luminous", "num": 35 } ], "mgr": [ { "features": "0x3ffddff8ffacffff", "release": "luminous", "num": 3 } ] }
But I think the number of clients older than Luminous is incorrect.
I'm running 98% of the clients with SLES 12SPx, and therefore the question is: Can you confirm in which SLES 12 release the function upmap is supported? And is it safe to execute ceph osd set-require-min-compat-client luminous --yes-i-really-mean-it? What happens to the clients then?
Your kernel should be 4.13+ to be upmap compat. Then you should tell `--yes-i-really-mean-it` to apply changes.
SLES 12 is still on 3.12 I think. However, the question is probably rather: Does SUSE backport things such as pg-upmap to older Kernels, such as RedHat does?
Cheers, Oliver
k _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Mon, Sep 16, 2019 at 2:20 PM Thomas Schneider <74cmonty@gmail.com> wrote:
Hello,
the current kernel with SLES 12SP3 is: ld3195:~ # uname -r 4.4.176-94.88-default
Assuming that this kernel is not supporting upmap, do you recommend to use balance mode crush-compat then?
Hi Thomas, Among the clients listed in your output, only 6 don't support upmap (those with features 0x40106b84a842a42). The rest do.
Regards Thomas
Am 16.09.2019 um 11:11 schrieb Oliver Freyermuth:
Am 16.09.19 um 11:06 schrieb Konstantin Shalygin:
On 9/16/19 3:59 PM, Thomas wrote:
I tried to run this command with failure: root@ld3955:/mnt/rbd# ceph osd set-require-min-compat-client luminous Error EPERM: cannot set require_min_compat_client to luminous: 6 connected client(s) look like jewel (missing 0xa00000000200000); 19 connected client(s) look like jewel (missing 0x800000000000000); 23 connected client(s) look like jewel (missing 0x800000000000000); add --yes-i-really-mean-it to do it anyway
root@ld3955:/mnt/rbd# ceph features { "mon": [ { "features": "0x3ffddff8ffacffff", "release": "luminous", "num": 3 } ], "mds": [ { "features": "0x3ffddff8ffacffff", "release": "luminous", "num": 2 } ], "osd": [ { "features": "0x3ffddff8ffacffff", "release": "luminous", "num": 368 } ], "client": [ { "features": "0x40106b84a842a42", "release": "jewel", "num": 6 }, { "features": "0x27018eb84aa42a52", "release": "jewel", "num": 19 }, { "features": "0x27018fb86aa42ada", "release": "jewel", "num": 23 }, { "features": "0x3ffddff8ffacffff", "release": "luminous", "num": 35 } ], "mgr": [ { "features": "0x3ffddff8ffacffff", "release": "luminous", "num": 3 } ] }
Thanks, Ilya
Bit 21 in the features bitfield is upmap support Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Mon, Sep 16, 2019 at 3:21 PM Ilya Dryomov <idryomov@gmail.com> wrote:
On Mon, Sep 16, 2019 at 2:20 PM Thomas Schneider <74cmonty@gmail.com> wrote:
Hello,
the current kernel with SLES 12SP3 is: ld3195:~ # uname -r 4.4.176-94.88-default
Assuming that this kernel is not supporting upmap, do you recommend to use balance mode crush-compat then?
Hi Thomas,
Among the clients listed in your output, only 6 don't support upmap (those with features 0x40106b84a842a42). The rest do.
Regards Thomas
Am 16.09.2019 um 11:11 schrieb Oliver Freyermuth:
Am 16.09.19 um 11:06 schrieb Konstantin Shalygin:
On 9/16/19 3:59 PM, Thomas wrote:
I tried to run this command with failure: root@ld3955:/mnt/rbd# ceph osd set-require-min-compat-client luminous Error EPERM: cannot set require_min_compat_client to luminous: 6 connected client(s) look like jewel (missing 0xa00000000200000); 19 connected client(s) look like jewel (missing 0x800000000000000); 23 connected client(s) look like jewel (missing 0x800000000000000); add --yes-i-really-mean-it to do it anyway
root@ld3955:/mnt/rbd# ceph features { "mon": [ { "features": "0x3ffddff8ffacffff", "release": "luminous", "num": 3 } ], "mds": [ { "features": "0x3ffddff8ffacffff", "release": "luminous", "num": 2 } ], "osd": [ { "features": "0x3ffddff8ffacffff", "release": "luminous", "num": 368 } ], "client": [ { "features": "0x40106b84a842a42", "release": "jewel", "num": 6 }, { "features": "0x27018eb84aa42a52", "release": "jewel", "num": 19 }, { "features": "0x27018fb86aa42ada", "release": "jewel", "num": 23 }, { "features": "0x3ffddff8ffacffff", "release": "luminous", "num": 35 } ], "mgr": [ { "features": "0x3ffddff8ffacffff", "release": "luminous", "num": 3 } ] }
Thanks,
Ilya _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, thanks for your valuable input. Question: Can I get more information of the 6 clients (those with features 0x40106b84a842a42), e.g. IP, that allows me to identify it easily? Regards Thomas Am 16.09.2019 um 15:56 schrieb Paul Emmerich:
Bit 21 in the features bitfield is upmap support
Paul
Am 16.09.2019 um 15:23 schrieb Ilya Dryomov:
On Mon, Sep 16, 2019 at 2:20 PM Thomas Schneider <74cmonty@gmail.com> wrote:
Hello,
the current kernel with SLES 12SP3 is: ld3195:~ # uname -r 4.4.176-94.88-default
Assuming that this kernel is not supporting upmap, do you recommend to use balance mode crush-compat then? Hi Thomas,
Among the clients listed in your output, only 6 don't support upmap (those with features 0x40106b84a842a42). The rest do.
Regards Thomas
Am 16.09.2019 um 11:11 schrieb Oliver Freyermuth:
Am 16.09.19 um 11:06 schrieb Konstantin Shalygin:
On 9/16/19 3:59 PM, Thomas wrote:
I tried to run this command with failure: root@ld3955:/mnt/rbd# ceph osd set-require-min-compat-client luminous Error EPERM: cannot set require_min_compat_client to luminous: 6 connected client(s) look like jewel (missing 0xa00000000200000); 19 connected client(s) look like jewel (missing 0x800000000000000); 23 connected client(s) look like jewel (missing 0x800000000000000); add --yes-i-really-mean-it to do it anyway
root@ld3955:/mnt/rbd# ceph features { "mon": [ { "features": "0x3ffddff8ffacffff", "release": "luminous", "num": 3 } ], "mds": [ { "features": "0x3ffddff8ffacffff", "release": "luminous", "num": 2 } ], "osd": [ { "features": "0x3ffddff8ffacffff", "release": "luminous", "num": 368 } ], "client": [ { "features": "0x40106b84a842a42", "release": "jewel", "num": 6 }, { "features": "0x27018eb84aa42a52", "release": "jewel", "num": 19 }, { "features": "0x27018fb86aa42ada", "release": "jewel", "num": 23 }, { "features": "0x3ffddff8ffacffff", "release": "luminous", "num": 35 } ], "mgr": [ { "features": "0x3ffddff8ffacffff", "release": "luminous", "num": 3 } ] } Thanks,
Ilya
On Mon, Sep 16, 2019 at 4:40 PM Thomas Schneider <74cmonty@gmail.com> wrote:
Hi,
thanks for your valuable input.
Question: Can I get more information of the 6 clients (those with features 0x40106b84a842a42), e.g. IP, that allows me to identify it easily?
Yes, although it's not integrated into "ceph features". Log into a monitor node and run "ceph daemon mon.a sessions" (mon.a is the name of the monitor, substitute accordingly). Thanks, Ilya
Wonderbra. I found some relevant sessions on 2 of 3 monitor nodes. And I found some others: root@ld5505:~# ceph daemon mon.ld5505 sessions | grep 0x40106b84a842a42 root@ld5505:~# ceph daemon mon.ld5505 sessions | grep -v luminous [ "MonSession(client.32679861 v1:10.97.206.92:0/1183647891 is open allow *, features 0x27018fb86aa42ada (jewel))", "MonSession(client.32692978 v1:10.97.206.91:0/3689092992 is open allow *, features 0x27018fb86aa42ada (jewel))", "MonSession(client.11935413 v1:10.96.6.116:0/3187655474 is open allow r, features 0x27018eb84aa42a52 (jewel))", "MonSession(client.3941901 v1:10.76.179.23:0/2967896845 is open allow r, features 0x27018fb86aa42ada (jewel))", "MonSession(client.28313343 v1:10.76.177.108:0/1303617860 is open allow r, features 0x27018fb86aa42ada (jewel))", "MonSession(client.29311725 v1:10.97.206.94:0/224438037 is open allow *, features 0x27018fb86aa42ada (jewel))", "MonSession(client.4535833 v1:10.76.177.133:0/1269608815 is open allow r, features 0x27018fb86aa42ada (jewel))", "MonSession(client.3919902 v1:10.96.4.243:0/293623521 is open allow r, features 0x27018eb84aa42a52 (jewel))", "MonSession(client.35678944 v1:10.76.179.211:0/4218086982 is open allow r, features 0x27018eb84aa42a52 (jewel))", "MonSession(client.35751316 v1:10.76.179.30:0/1348696702 is open allow r, features 0x27018eb84aa42a52 (jewel))", "MonSession(client.28246527 v1:10.96.4.228:0/1495661381 is open allow r, features 0x27018fb86aa42ada (jewel))", "MonSession(client.3917843 v1:10.76.179.22:0/489863209 is open allow r, features 0x27018fb86aa42ada (jewel))", "MonSession(unknown.0 - is open allow r, features 0x27018eb84aa42a52 (jewel))", ] Would it make sense to shutdown these clients, too? What confuses me is that the list includes clients that belong to the Ceph cluster, namely 10.97.206.0/24. All nodes of the Ceph cluster are identical in terms of OS, kernel, Ceph. Regards Thomas Am 16.09.2019 um 16:56 schrieb Ilya Dryomov:
On Mon, Sep 16, 2019 at 4:40 PM Thomas Schneider <74cmonty@gmail.com> wrote:
Hi,
thanks for your valuable input.
Question: Can I get more information of the 6 clients (those with features 0x40106b84a842a42), e.g. IP, that allows me to identify it easily? Yes, although it's not integrated into "ceph features". Log into a monitor node and run "ceph daemon mon.a sessions" (mon.a is the name of the monitor, substitute accordingly).
Thanks,
Ilya
On Mon, Sep 16, 2019 at 5:10 PM Thomas Schneider <74cmonty@gmail.com> wrote:
Wonderbra.
I found some relevant sessions on 2 of 3 monitor nodes. And I found some others: root@ld5505:~# ceph daemon mon.ld5505 sessions | grep 0x40106b84a842a42 root@ld5505:~# ceph daemon mon.ld5505 sessions | grep -v luminous [ "MonSession(client.32679861 v1:10.97.206.92:0/1183647891 is open allow *, features 0x27018fb86aa42ada (jewel))", "MonSession(client.32692978 v1:10.97.206.91:0/3689092992 is open allow *, features 0x27018fb86aa42ada (jewel))", "MonSession(client.11935413 v1:10.96.6.116:0/3187655474 is open allow r, features 0x27018eb84aa42a52 (jewel))", "MonSession(client.3941901 v1:10.76.179.23:0/2967896845 is open allow r, features 0x27018fb86aa42ada (jewel))", "MonSession(client.28313343 v1:10.76.177.108:0/1303617860 is open allow r, features 0x27018fb86aa42ada (jewel))", "MonSession(client.29311725 v1:10.97.206.94:0/224438037 is open allow *, features 0x27018fb86aa42ada (jewel))", "MonSession(client.4535833 v1:10.76.177.133:0/1269608815 is open allow r, features 0x27018fb86aa42ada (jewel))", "MonSession(client.3919902 v1:10.96.4.243:0/293623521 is open allow r, features 0x27018eb84aa42a52 (jewel))", "MonSession(client.35678944 v1:10.76.179.211:0/4218086982 is open allow r, features 0x27018eb84aa42a52 (jewel))", "MonSession(client.35751316 v1:10.76.179.30:0/1348696702 is open allow r, features 0x27018eb84aa42a52 (jewel))", "MonSession(client.28246527 v1:10.96.4.228:0/1495661381 is open allow r, features 0x27018fb86aa42ada (jewel))", "MonSession(client.3917843 v1:10.76.179.22:0/489863209 is open allow r, features 0x27018fb86aa42ada (jewel))", "MonSession(unknown.0 - is open allow r, features 0x27018eb84aa42a52 (jewel))", ]
Would it make sense to shutdown these clients, too?
What confuses me is that the list includes clients that belong to the Ceph cluster, namely 10.97.206.0/24. All nodes of the Ceph cluster are identical in terms of OS, kernel, Ceph.
The above output seems consistent with your "ceph features" output: it lists clients with features 0x27018eb84aa42a52 and 0x27018fb86aa42ada. Like I said in my previous email, both of these support upmap. If you temporarily shut them down, set-require-min-compat-client will work without --yes-i-really-mean-it. Thanks, Ilya
Hi, I was checking the connected sessions on all of my 3 MON nodes with this command: ceph daemon mon.<id> sessions | grep -v luminous This returns the following 2 clients featuring 0x27018fb86aa42ada and 0x27018eb84aa42a52 This is mapping to OS / Kernel: 0x27018fb86aa42ada Debian 10.1 Kernel 5.0.21-1-pve and SLES 12SP4 Kernel 4.12.14-95.13-default 0x27018eb84aa42a52 SLES 12SP3 4.4.176-94.88-default and SLES 12SP3 4.4.180-94.97-default and SLES 12SP4 4.4.156-94.64-default Based on your previous information 0x27018fb86aa42ada and 0x27018eb84aa42a52 is ready for upmap. But then I wonder why the output of ceph daemon mon.<id> sessions marks these clients as "jewel"? I have checked the installed ceph version on each client and can confirm that it is: ceph version 12.2 luminous This would drive the conclusion that the ouput of ceph daemon mon.<id> sessions is pointing incorrectly to "jewel". Regards Thomas Am 16.09.2019 um 17:36 schrieb Ilya Dryomov:
Wonderbra.
I found some relevant sessions on 2 of 3 monitor nodes. And I found some others: root@ld5505:~# ceph daemon mon.ld5505 sessions | grep 0x40106b84a842a42 root@ld5505:~# ceph daemon mon.ld5505 sessions | grep -v luminous [ "MonSession(client.32679861 v1:10.97.206.92:0/1183647891 is open allow *, features 0x27018fb86aa42ada (jewel))", "MonSession(client.32692978 v1:10.97.206.91:0/3689092992 is open allow *, features 0x27018fb86aa42ada (jewel))", "MonSession(client.11935413 v1:10.96.6.116:0/3187655474 is open allow r, features 0x27018eb84aa42a52 (jewel))", "MonSession(client.3941901 v1:10.76.179.23:0/2967896845 is open allow r, features 0x27018fb86aa42ada (jewel))", "MonSession(client.28313343 v1:10.76.177.108:0/1303617860 is open allow r, features 0x27018fb86aa42ada (jewel))", "MonSession(client.29311725 v1:10.97.206.94:0/224438037 is open allow *, features 0x27018fb86aa42ada (jewel))", "MonSession(client.4535833 v1:10.76.177.133:0/1269608815 is open allow r, features 0x27018fb86aa42ada (jewel))", "MonSession(client.3919902 v1:10.96.4.243:0/293623521 is open allow r, features 0x27018eb84aa42a52 (jewel))", "MonSession(client.35678944 v1:10.76.179.211:0/4218086982 is open allow r, features 0x27018eb84aa42a52 (jewel))", "MonSession(client.35751316 v1:10.76.179.30:0/1348696702 is open allow r, features 0x27018eb84aa42a52 (jewel))", "MonSession(client.28246527 v1:10.96.4.228:0/1495661381 is open allow r, features 0x27018fb86aa42ada (jewel))", "MonSession(client.3917843 v1:10.76.179.22:0/489863209 is open allow r, features 0x27018fb86aa42ada (jewel))", "MonSession(unknown.0 - is open allow r, features 0x27018eb84aa42a52 (jewel))", ]
Would it make sense to shutdown these clients, too?
What confuses me is that the list includes clients that belong to the Ceph cluster, namely 10.97.206.0/24. All nodes of the Ceph cluster are identical in terms of OS, kernel, Ceph. The above output seems consistent with your "ceph features" output: it
On Mon, Sep 16, 2019 at 5:10 PM Thomas Schneider <74cmonty@gmail.com> wrote: lists clients with features 0x27018eb84aa42a52 and 0x27018fb86aa42ada. Like I said in my previous email, both of these support upmap.
If you temporarily shut them down, set-require-min-compat-client will work without --yes-i-really-mean-it.
Thanks,
Ilya
Hi,
I have checked the installed ceph version on each client and can confirm that it is: ceph version 12.2 luminous
This would drive the conclusion that the ouput of ceph daemon mon.<id> sessions is pointing incorrectly to "jewel".
I have the same in an upgraded Nautilus cluster. There are a couple of "unknown" clients and some Jewel clients, but checking those clients they all have at least Luminous. I rebooted some of those clients, too, but this didn't change anything. The min-compat-client already has been forced to luminous (knowing that in fact there are no jewel clients): host1:~ # ceph osd get-require-min-compat-client luminous I'd be curious how to get rid of the jewel client sessions. Regards, Eugen Zitat von Thomas Schneider <74cmonty@gmail.com>:
Hi,
I was checking the connected sessions on all of my 3 MON nodes with this command: ceph daemon mon.<id> sessions | grep -v luminous
This returns the following 2 clients featuring 0x27018fb86aa42ada and 0x27018eb84aa42a52
This is mapping to OS / Kernel: 0x27018fb86aa42ada Debian 10.1 Kernel 5.0.21-1-pve and SLES 12SP4 Kernel 4.12.14-95.13-default
0x27018eb84aa42a52 SLES 12SP3 4.4.176-94.88-default and SLES 12SP3 4.4.180-94.97-default and SLES 12SP4 4.4.156-94.64-default
Based on your previous information 0x27018fb86aa42ada and 0x27018eb84aa42a52 is ready for upmap. But then I wonder why the output of ceph daemon mon.<id> sessions marks these clients as "jewel"?
I have checked the installed ceph version on each client and can confirm that it is: ceph version 12.2 luminous
This would drive the conclusion that the ouput of ceph daemon mon.<id> sessions is pointing incorrectly to "jewel".
Regards Thomas
Am 16.09.2019 um 17:36 schrieb Ilya Dryomov:
Wonderbra.
I found some relevant sessions on 2 of 3 monitor nodes. And I found some others: root@ld5505:~# ceph daemon mon.ld5505 sessions | grep 0x40106b84a842a42 root@ld5505:~# ceph daemon mon.ld5505 sessions | grep -v luminous [ "MonSession(client.32679861 v1:10.97.206.92:0/1183647891 is open allow *, features 0x27018fb86aa42ada (jewel))", "MonSession(client.32692978 v1:10.97.206.91:0/3689092992 is open allow *, features 0x27018fb86aa42ada (jewel))", "MonSession(client.11935413 v1:10.96.6.116:0/3187655474 is open allow r, features 0x27018eb84aa42a52 (jewel))", "MonSession(client.3941901 v1:10.76.179.23:0/2967896845 is open allow r, features 0x27018fb86aa42ada (jewel))", "MonSession(client.28313343 v1:10.76.177.108:0/1303617860 is open allow r, features 0x27018fb86aa42ada (jewel))", "MonSession(client.29311725 v1:10.97.206.94:0/224438037 is open allow *, features 0x27018fb86aa42ada (jewel))", "MonSession(client.4535833 v1:10.76.177.133:0/1269608815 is open allow r, features 0x27018fb86aa42ada (jewel))", "MonSession(client.3919902 v1:10.96.4.243:0/293623521 is open allow r, features 0x27018eb84aa42a52 (jewel))", "MonSession(client.35678944 v1:10.76.179.211:0/4218086982 is open allow r, features 0x27018eb84aa42a52 (jewel))", "MonSession(client.35751316 v1:10.76.179.30:0/1348696702 is open allow r, features 0x27018eb84aa42a52 (jewel))", "MonSession(client.28246527 v1:10.96.4.228:0/1495661381 is open allow r, features 0x27018fb86aa42ada (jewel))", "MonSession(client.3917843 v1:10.76.179.22:0/489863209 is open allow r, features 0x27018fb86aa42ada (jewel))", "MonSession(unknown.0 - is open allow r, features 0x27018eb84aa42a52 (jewel))", ]
Would it make sense to shutdown these clients, too?
What confuses me is that the list includes clients that belong to the Ceph cluster, namely 10.97.206.0/24. All nodes of the Ceph cluster are identical in terms of OS, kernel, Ceph. The above output seems consistent with your "ceph features" output: it
On Mon, Sep 16, 2019 at 5:10 PM Thomas Schneider <74cmonty@gmail.com> wrote: lists clients with features 0x27018eb84aa42a52 and 0x27018fb86aa42ada. Like I said in my previous email, both of these support upmap.
If you temporarily shut them down, set-require-min-compat-client will work without --yes-i-really-mean-it.
Thanks,
Ilya
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Tue, Sep 17, 2019 at 2:54 PM Eugen Block <eblock@nde.ag> wrote:
Hi,
I have checked the installed ceph version on each client and can confirm that it is: ceph version 12.2 luminous
This would drive the conclusion that the ouput of ceph daemon mon.<id> sessions is pointing incorrectly to "jewel".
I have the same in an upgraded Nautilus cluster. There are a couple of "unknown" clients and some Jewel clients, but checking those clients they all have at least Luminous. I rebooted some of those clients, too, but this didn't change anything. The min-compat-client already has been forced to luminous (knowing that in fact there are no jewel clients):
host1:~ # ceph osd get-require-min-compat-client luminous
I'd be curious how to get rid of the jewel client sessions.
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/JP44CGR4FO5... Thanks, Ilya
Thank you very much, Ilya! Zitat von Ilya Dryomov <idryomov@gmail.com>:
On Tue, Sep 17, 2019 at 2:54 PM Eugen Block <eblock@nde.ag> wrote:
Hi,
I have checked the installed ceph version on each client and can confirm that it is: ceph version 12.2 luminous
This would drive the conclusion that the ouput of ceph daemon mon.<id> sessions is pointing incorrectly to "jewel".
I have the same in an upgraded Nautilus cluster. There are a couple of "unknown" clients and some Jewel clients, but checking those clients they all have at least Luminous. I rebooted some of those clients, too, but this didn't change anything. The min-compat-client already has been forced to luminous (knowing that in fact there are no jewel clients):
host1:~ # ceph osd get-require-min-compat-client luminous
I'd be curious how to get rid of the jewel client sessions.
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/JP44CGR4FO5...
Thanks,
Ilya
Thomas <74cmonty@gmail.com> writes:
But I think the number of clients older than Luminous is incorrect.
I'm running 98% of the clients with SLES 12SPx, and therefore the question is: Can you confirm in which SLES 12 release the function upmap is supported?
I can confirm that SLE12-SP3 *does* support upmap; if you're running older clients (e.g., SP2), upmap is not supported. As Ilya already mentioned, client 6 doesn't seem to support this feature, so I'm assuming you're using SP2 or older there. Cheers, -- Luis
And is it safe to execute ceph osd set-require-min-compat-client luminous --yes-i-really-mean-it? What happens to the clients then?
Regards Thomas
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (8)
-
Eugen Block
-
Ilya Dryomov
-
Konstantin Shalygin
-
Luis Henriques
-
Oliver Freyermuth
-
Paul Emmerich
-
Thomas
-
Thomas Schneider