Hi, On Mon, Apr 13, 2020 at 3:08 PM Frank Schilder <frans@dtu.dk> wrote:

Hi Paul,

thanks for the fast reply. When you say "bit 21", do you mean "(feature_map & 2^21) == true" (i.e., counting from 0 starting at the right-hand end)?

yes

Assuming upmap is supported by all clients. If I understand correctly, to use the upmap mode with balancer, I need to set

ceph osd set-require-min-compat-client luminous

Which I would guess will not allow the jewel clients to reconnect. I would be grateful if you could clarify these points to me:

yes

1) Can I use up-map mode without setting this?

no

2) If so, what happens if a jewel client without this feature bit set tries to connect?

it'll error out with a message about feature mismatch; it checks actually relevant feature flags not the reverse mapping to a release which is usually wrong for kernel clients

3) I guess that in case that as soon as an up-map table is created, only clients with this bit set can connect. In case we run into problems, is there a way to roll back?

yes, you can remove the upmap items manually and change the client requirement; I don't know the command to do this off the top of my head Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90

Many thanks and best regards,

================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

________________________________________ From: Paul Emmerich <paul.emmerich@croit.io> Sent: 13 April 2020 13:32:40 To: Frank Schilder Cc: ceph-users Subject: Re: [ceph-users] Check if upmap is supported by client?

bit 21 in the features bitmap is upmap support

Paul

-- Paul Emmerich

Looking for help with your Ceph cluster? Contact us at https://croit.io

croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90

On Mon, Apr 13, 2020 at 11:53 AM Frank Schilder <frans@dtu.dk> wrote:

...

Dear all,

I would like to enable the balancer on a mimic 13.2.8 cluster in upmap mode. Unfortunately, I have a lot of ceph fs kernel clients that report their version as jewel, but might already support upmap. The ceph client kernel module received already a lot of back-ports and supports features of later ceph versions, for example, quotas. I guess they report back jewel, because not all luminous/mimic features are back-ported yet. Is there a way to check if a client supports upmap?

Here some info:

[root@gnosis ~]# ceph features { [...] "client": [ { "features": "0x27018fb86aa42ada", "release": "jewel", "num": 1676 }, { "features": "0x2f018fb86aa42ada", "release": "luminous", "num": 1 }, { "features": "0x3ffddff8ffacfffb", "release": "luminous", "num": 167 } ],

The fs clients are the top two entries, the third entry is rbd clients. Note that the feature key for the fs clients is almost identical. Here a snippet from mds session ls for one such jewel client:

{ "id": 25641514, "num_leases": 0, "num_caps": 1, "state": "open", "request_load_avg": 0, "uptime": 588563.550276, "replay_requests": 0, "completed_requests": 0, "reconnecting": false, "inst": "client.25641514 192.168.57.124:0/3398308464", "client_metadata": { "features": "00000000000000ff", "entity_id": "con-fs2-hpc", "hostname": "sn253.hpc.ait.dtu.dk", "kernel_version": "3.10.0-957.12.2.el7.x86_64", "root": "/hpc/groups" } }

Since I would like to use upmap right from the beginning, my alternative is to re-weight a few of the really bad outliers manually to simplify changing back.

What would you suggest?

Thanks and best regards,

================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io