How to (permanently) disable msgr v1 on Ceph?
Hi, For new clusters one of the first things I do is to disable messenger v1: ceph config set mon ms_bind_msgr1 false However, that is not enough, as a restart of the monitors will leave v1 enabled in the monmap. Why is that? That looks like a bug to me. Therefore I explicitly set the addrs and port of the monitors, i.e.: ceph mon set-addrs stefan-ceph-test01 $ip:3300 Be careful not to make a typo, that will ruin your day, see [1]. This immediately disables v1 and a new monmap is created. All good, however ... When a monitor is removed, say mon1, and added again, the new mon is added with both a v1 and a v2 address. I don't want that, I want to disable legacy v1 permanently. How do I do that? Adding: [mon] ms_bind_msgr1 = false to /etc/ceph/ceph.conf does not help either. When I performed a test this morning on a fresh 18.2.4 cluster (remove a mon, add it back again) to reproduce this issue the v1 was enabled again on two monitors (a third was left with only v2). Repeating the "ceph mon set-addrs" command removes the v1 binding again. But I would like to avoid messing around with this, triggering monmaps / elections, especially when making an error with this brings your cluster to a halt [1]. In what Ceph release will v1 be disabled by default on new clusters? Gr. Stefan [1] https://tracker.ceph.com/issues/54744
Hi Stefan, If ceph-mon respected ms_bind_msgr1 = false, then one could add --ms-bind-msgr1=false as extra_entrypoint_args in the mon service_type [1], so as to have any ceph-mon daemons deployed or redeployed using msgr v2 exclusively. Unfortunately, ceph-mon doesn't respect this setting as suspected by Ilya here [2] and confirmed on a test cluster on my side. I think we can make this a tracker. As for your other question as to when v1 will be disabled by default on new Ceph clusters, I have no idea at all. Cheers, Frédéric. [1] https://docs.ceph.com/en/quincy/cephadm/services/#extra-entrypoint-arguments [2] https://www.spinics.net/lists/ceph-users/msg73320.html ----- Le 13 Mar 25, à 14:30, Stefan Kooman stefan@bit.nl a écrit :
Hi,
For new clusters one of the first things I do is to disable messenger v1:
ceph config set mon ms_bind_msgr1 false
However, that is not enough, as a restart of the monitors will leave v1 enabled in the monmap. Why is that? That looks like a bug to me.
Therefore I explicitly set the addrs and port of the monitors, i.e.:
ceph mon set-addrs stefan-ceph-test01 $ip:3300 Be careful not to make a typo, that will ruin your day, see [1].
This immediately disables v1 and a new monmap is created.
All good, however ...
When a monitor is removed, say mon1, and added again, the new mon is added with both a v1 and a v2 address. I don't want that, I want to disable legacy v1 permanently. How do I do that?
Adding:
[mon] ms_bind_msgr1 = false
to /etc/ceph/ceph.conf does not help either.
When I performed a test this morning on a fresh 18.2.4 cluster (remove a mon, add it back again) to reproduce this issue the v1 was enabled again on two monitors (a third was left with only v2).
Repeating the "ceph mon set-addrs" command removes the v1 binding again. But I would like to avoid messing around with this, triggering monmaps / elections, especially when making an error with this brings your cluster to a halt [1].
In what Ceph release will v1 be disabled by default on new clusters?
Gr. Stefan
[1] https://tracker.ceph.com/issues/54744 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 13-03-2025 16:08, Frédéric Nass wrote:
Hi Stefan,
If ceph-mon respected ms_bind_msgr1 = false, then one could add --ms-bind-msgr1=false as extra_entrypoint_args in the mon service_type [1], so as to have any ceph-mon daemons deployed or redeployed using msgr v2 exclusively. Unfortunately, ceph-mon doesn't respect this setting as suspected by Ilya here [2] and confirmed on a test cluster on my side. I think we can make this a tracker.
Thanks for double checking, I've created [1].
As for your other question as to when v1 will be disabled by default on new Ceph clusters, I have no idea at all.
I'll leave it to the devs to discuss this one. Gr. Stefan [1]: https://tracker.ceph.com/issues/70457
On 13-03-2025 16:08, Frédéric Nass wrote:
If ceph-mon respected ms_bind_msgr1 = false, then one could add --ms-bind-msgr1=false as extra_entrypoint_args in the mon service_type [1], so as to have any ceph-mon daemons deployed or redeployed using msgr v2 exclusively. Unfortunately, ceph-mon doesn't respect this setting as suspected by Ilya here [2] and confirmed on a test cluster on my side. I think we can make this a tracker.
Thanks for double checking, I've created [1].
As for your other question as to when v1 will be disabled by default on new Ceph clusters, I have no idea at all.
I'll leave it to the devs to discuss this one.
It would be nice if the defaults for newly created clusters also came with the global reclaim id thing disabled, so we didn't have to manually enable msgrv2 (and disable v1 possibly as per this thread) and also disable the reclaim thing every time. If you upgrade an old cluster into Reef/Squid it might also hold old clients, but newly built ones probably won't see a lot of old clients that must have the reclaim thing enabled, right?
On 14-03-2025 09:53, Janne Johansson wrote:
On 13-03-2025 16:08, Frédéric Nass wrote:
If ceph-mon respected ms_bind_msgr1 = false, then one could add --ms-bind-msgr1=false as extra_entrypoint_args in the mon service_type [1], so as to have any ceph-mon daemons deployed or redeployed using msgr v2 exclusively. Unfortunately, ceph-mon doesn't respect this setting as suspected by Ilya here [2] and confirmed on a test cluster on my side. I think we can make this a tracker.
Thanks for double checking, I've created [1].
As for your other question as to when v1 will be disabled by default on new Ceph clusters, I have no idea at all.
I'll leave it to the devs to discuss this one.
It would be nice if the defaults for newly created clusters also came with the global reclaim id thing disabled, so we didn't have to manually enable msgrv2 (and disable v1 possibly as per this thread) and also disable the reclaim thing every time. If you upgrade an old cluster into Reef/Squid it might also hold old clients, but newly built ones probably won't see a lot of old clients that must have the reclaim thing enabled, right?
For the Reef clusters that I have created the "auth_allow_insecure_global_id_reclaim" is false. So this already seems to be the default (as it should be, shipping known weaknesses is bad practice to say the least). Gr. Stefan
I'll leave it to the devs to discuss this one.
It would be nice if the defaults for newly created clusters also came with the global reclaim id thing disabled, so we didn't have to manually enable msgrv2 (and disable v1 possibly as per this thread) and also disable the reclaim thing every time. If you upgrade an old cluster into Reef/Squid it might also hold old clients, but newly built ones probably won't see a lot of old clients that must have the reclaim thing enabled, right?
For the Reef clusters that I have created the "auth_allow_insecure_global_id_reclaim" is false. So this already seems to be the default (as it should be, shipping known weaknesses is bad practice to say the least).
I literally made an 18.2.4 cluster yesterday and had to enable msgrv2 and disable global id reclaim. Installed from .debs on ubuntu hosts. https://github.com/ceph/ceph/blob/d4ce7b60b2f2472c86f8f41f89907aedec8b1c54/s... seems to list it as true still. -- May the most significant bit of your life be positive.
On 14-03-2025 10:44, Janne Johansson wrote:
I'll leave it to the devs to discuss this one.
It would be nice if the defaults for newly created clusters also came with the global reclaim id thing disabled, so we didn't have to manually enable msgrv2 (and disable v1 possibly as per this thread) and also disable the reclaim thing every time. If you upgrade an old cluster into Reef/Squid it might also hold old clients, but newly built ones probably won't see a lot of old clients that must have the reclaim thing enabled, right?
For the Reef clusters that I have created the "auth_allow_insecure_global_id_reclaim" is false. So this already seems to be the default (as it should be, shipping known weaknesses is bad practice to say the least).
I literally made an 18.2.4 cluster yesterday and had to enable msgrv2 and disable global id reclaim. Installed from .debs on ubuntu hosts.
https://github.com/ceph/ceph/blob/d4ce7b60b2f2472c86f8f41f89907aedec8b1c54/s... seems to list it as true still.
Ah, interesting. I have it installed via Cephadm ... so apparently it's fixed there, but not in packages ... :/. For the record, v1 and v2 are enabled by default (as the docs state [1]) ... on a Cephadm based cluster that is ... Gr. Stefan [1]: https://docs.ceph.com/en/reef/rados/configuration/msgr2/#bind-configuration-...
Thank you the bug report! Yeah, there was no so much concern about disabling v1 AFAIK.
In what Ceph release will v1 be disabled by default on new clusters?
This a really good question that has many common points with the old discussions on our guarantees toward the compatibility with clients. So far it was mostly about dencoding, feature bits etc, but it applies to the v1 as well. Perhaps we should reiterate on CDM (or CSC at least). CCing Ilya. Regards, Radek
Just a supplement. I'm aware there are different stances on the client compatibility ranging from perpetual to N-3. Yet, being "compatible by default" is somewhat different from having "optional compatibility" (by opt-in). On Fri, Mar 14, 2025 at 11:34 AM Radoslaw Zarzynski <rzarzyns@redhat.com> wrote:
Thank you the bug report! Yeah, there was no so much concern about disabling v1 AFAIK.
In what Ceph release will v1 be disabled by default on new clusters?
This a really good question that has many common points with the old discussions on our guarantees toward the compatibility with clients. So far it was mostly about dencoding, feature bits etc, but it applies to the v1 as well.
Perhaps we should reiterate on CDM (or CSC at least).
CCing Ilya.
Regards, Radek
participants (4)
-
Frédéric Nass
-
Janne Johansson
-
Radoslaw Zarzynski
-
Stefan Kooman