Help in upgrading CEPH
Dear list, we are upgrading our ceph infrastructure from mimic to octopus (please be kind, we known that we are working with "old" tools, but these ceph releases are tied to our openstack installation needs) and _*all*_ the ceph actors (mon/mgr/osd/rgw - no mds as we do not serve filesystem) were upgraded fine as follow:
# ceph -v ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable) We're using a ubuntu/juju orchestrator for managing ceph (and openstack too).
It seems all ok, but if I ask a mon to dump the sessions, we get that_all of them_ are on luminous:
"con_features": 4540138292840890367, "con_features_hex": "3f01cfb8ffedffff", "con_features_release": "luminous", We found this oddity when we tried to unset the "insecure global_id reclaim" flag and all things got broken, so we had to reactivate the flag.
All ceph network layers are "closed", so we're not so urged to remove the flag, but we would like to understand if this is a known problem or an error done by us. Any hints ? thanks in advance. -- ing. Sergio Rabellino Università degli Studi di Torino Dipartimento di Informatica Tecnico di Ricerca Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 C.so Svizzera , 185 - 10149 - Torino <http://www.di.unito.it>
What is the output of: ceph osd dump | grep require_osd_release Have you upgraded OpenStack (librados) as well? Op do 15 mei 2025 om 10:50 schreef Sergio Rabellino <rabellino@di.unito.it>:
Dear list,
we are upgrading our ceph infrastructure from mimic to octopus (please be kind, we known that we are working with "old" tools, but these ceph releases are tied to our openstack installation needs) and _*all*_ the ceph actors (mon/mgr/osd/rgw - no mds as we do not serve filesystem) were upgraded fine as follow:
# ceph -v ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable) We're using a ubuntu/juju orchestrator for managing ceph (and openstack too).
It seems all ok, but if I ask a mon to dump the sessions, we get that_all of them_ are on luminous:
"con_features": 4540138292840890367, "con_features_hex": "3f01cfb8ffedffff", "con_features_release": "luminous",
We found this oddity when we tried to unset the "insecure global_id reclaim" flag and all things got broken, so we had to reactivate the flag.
All ceph network layers are "closed", so we're not so urged to remove the flag, but we would like to understand if this is a known problem or an error done by us.
Any hints ?
thanks in advance.
-- ing. Sergio Rabellino
Università degli Studi di Torino Dipartimento di Informatica Tecnico di Ricerca Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 C.so Svizzera , 185 - 10149 - Torino
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
# ceph osd dump |grep require_osd_release require_osd_release octopus We are aware of this setting and change it on release change (we did in
Thanks for your prompt reply, dump follows: the last days mimic->nautilus->octopus). The very strange thing it's that we upgraded luminous->mimic more than two years ago. About librados, I get:
# rados -v ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable) so it seems fine to me.
Il 15/05/2025 11:01, Sinan Polat ha scritto:
What is the output of: ceph osd dump | grep require_osd_release
Have you upgraded OpenStack (librados) as well?
Op do 15 mei 2025 om 10:50 schreef Sergio Rabellino <rabellino@di.unito.it <mailto:rabellino@di.unito.it>>:
Dear list,
we are upgrading our ceph infrastructure from mimic to octopus (please be kind, we known that we are working with "old" tools, but these ceph releases are tied to our openstack installation needs) and _*all*_ the ceph actors (mon/mgr/osd/rgw - no mds as we do not serve filesystem) were upgraded fine as follow:
> # ceph -v > ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) > octopus (stable) We're using a ubuntu/juju orchestrator for managing ceph (and openstack too).
It seems all ok, but if I ask a mon to dump the sessions, we get that_all of them_ are on luminous:
> "con_features": 4540138292840890367, > "con_features_hex": "3f01cfb8ffedffff", > "con_features_release": "luminous", We found this oddity when we tried to unset the "insecure global_id reclaim" flag and all things got broken, so we had to reactivate the flag.
All ceph network layers are "closed", so we're not so urged to remove the flag, but we would like to understand if this is a known problem or an error done by us.
Any hints ?
thanks in advance.
-- ing. Sergio Rabellino
Università degli Studi di Torino Dipartimento di Informatica Tecnico di Ricerca Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 C.so Svizzera , 185 - 10149 - Torino
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
-- ing. Sergio Rabellino Università degli Studi di Torino Dipartimento di Informatica Tecnico di Ricerca Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 C.so Svizzera , 185 - 10149 - Torino <http://www.di.unito.it>
The output of 'ceph features' (or the mon session output) does not reflect that it's an older client. It just represents a collection of supported features. This has been discussed multiple times on this list. This is not unusual. Regarding the insecure global_id, does your cluster warn about it? Can you share these configs? ceph config get mon mon_warn_on_insecure_global_id_reclaim_allowed ceph config get mon mon_warn_on_insecure_global_id_reclaim Zitat von Sergio Rabellino <rabellino@di.unito.it>:
Thanks for your prompt reply, dump follows:
# ceph osd dump |grep require_osd_release require_osd_release octopus We are aware of this setting and change it on release change (we did in the last days mimic->nautilus->octopus). The very strange thing it's that we upgraded luminous->mimic more than two years ago.
About librados, I get:
# rados -v ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable) so it seems fine to me.
Il 15/05/2025 11:01, Sinan Polat ha scritto:
What is the output of: ceph osd dump | grep require_osd_release
Have you upgraded OpenStack (librados) as well?
Op do 15 mei 2025 om 10:50 schreef Sergio Rabellino <rabellino@di.unito.it <mailto:rabellino@di.unito.it>>:
Dear list,
we are upgrading our ceph infrastructure from mimic to octopus (please be kind, we known that we are working with "old" tools, but these ceph releases are tied to our openstack installation needs) and _*all*_ the ceph actors (mon/mgr/osd/rgw - no mds as we do not serve filesystem) were upgraded fine as follow:
# ceph -v ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable) We're using a ubuntu/juju orchestrator for managing ceph (and openstack too).
It seems all ok, but if I ask a mon to dump the sessions, we get that_all of them_ are on luminous:
"con_features": 4540138292840890367, "con_features_hex": "3f01cfb8ffedffff", "con_features_release": "luminous", We found this oddity when we tried to unset the "insecure global_id reclaim" flag and all things got broken, so we had to reactivate the flag.
All ceph network layers are "closed", so we're not so urged to remove the flag, but we would like to understand if this is a known problem or an error done by us.
Any hints ?
thanks in advance.
-- ing. Sergio Rabellino
Università degli Studi di Torino Dipartimento di Informatica Tecnico di Ricerca Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 C.so Svizzera , 185 - 10149 - Torino
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
-- ing. Sergio Rabellino
Università degli Studi di Torino Dipartimento di Informatica Tecnico di Ricerca Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 C.so Svizzera , 185 - 10149 - Torino
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi. Have you verified that the new libraries on the openstack compute nodes are in use? If you have upgraded librados on the compute nodes you also need to stop and start all the vms to start using the new libraries. You can do that by stop/start or live-migration for example. We usually do it through live-migration during the openstack upgrade of all the compute nodes (we empty each node before upgrading openstack, libvirt, qemu and ceph components, then live-migrate back). /Jan On 2025-05-15 11:39, Eugen Block wrote:
The output of 'ceph features' (or the mon session output) does not reflect that it's an older client. It just represents a collection of supported features. This has been discussed multiple times on this list. This is not unusual.
Regarding the insecure global_id, does your cluster warn about it? Can you share these configs?
ceph config get mon mon_warn_on_insecure_global_id_reclaim_allowed ceph config get mon mon_warn_on_insecure_global_id_reclaim
Zitat von Sergio Rabellino <rabellino@di.unito.it>:
Thanks for your prompt reply, dump follows:
# ceph osd dump |grep require_osd_release require_osd_release octopus We are aware of this setting and change it on release change (we did in the last days mimic->nautilus->octopus). The very strange thing it's that we upgraded luminous->mimic more than two years ago.
About librados, I get:
# rados -v ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable) so it seems fine to me.
Il 15/05/2025 11:01, Sinan Polat ha scritto:
What is the output of: ceph osd dump | grep require_osd_release
Have you upgraded OpenStack (librados) as well?
Op do 15 mei 2025 om 10:50 schreef Sergio Rabellino <rabellino@di.unito.it <mailto:rabellino@di.unito.it>>:
Dear list,
we are upgrading our ceph infrastructure from mimic to octopus (please be kind, we known that we are working with "old" tools, but these ceph releases are tied to our openstack installation needs) and _*all*_ the ceph actors (mon/mgr/osd/rgw - no mds as we do not serve filesystem) were upgraded fine as follow:
> # ceph -v > ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) > octopus (stable) We're using a ubuntu/juju orchestrator for managing ceph (and openstack too).
It seems all ok, but if I ask a mon to dump the sessions, we get that_all of them_ are on luminous:
> "con_features": 4540138292840890367, > "con_features_hex": "3f01cfb8ffedffff", > "con_features_release": "luminous", We found this oddity when we tried to unset the "insecure global_id reclaim" flag and all things got broken, so we had to reactivate the flag.
All ceph network layers are "closed", so we're not so urged to remove the flag, but we would like to understand if this is a known problem or an error done by us.
Any hints ?
thanks in advance.
-- ing. Sergio Rabellino
Università degli Studi di Torino Dipartimento di Informatica Tecnico di Ricerca Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 C.so Svizzera , 185 - 10149 - Torino
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
-- ing. Sergio Rabellino
Università degli Studi di Torino Dipartimento di Informatica Tecnico di Ricerca Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 C.so Svizzera , 185 - 10149 - Torino
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Yes, they are up-to-date at the same level. We can't use live-migration because most of the VMs are equipped with GPUs and do not move between nodes. Anyway we restarted some VMs without changes. Thanks anyway. Il 15/05/2025 12:04, Jan Johansson ha scritto:
Hi.
Have you verified that the new libraries on the openstack compute nodes are in use? If you have upgraded librados on the compute nodes you also need to stop and start all the vms to start using the new libraries. You can do that by stop/start or live-migration for example.
We usually do it through live-migration during the openstack upgrade of all the compute nodes (we empty each node before upgrading openstack, libvirt, qemu and ceph components, then live-migrate back).
/Jan
On 2025-05-15 11:39, Eugen Block wrote:
The output of 'ceph features' (or the mon session output) does not reflect that it's an older client. It just represents a collection of supported features. This has been discussed multiple times on this list. This is not unusual.
Regarding the insecure global_id, does your cluster warn about it? Can you share these configs?
ceph config get mon mon_warn_on_insecure_global_id_reclaim_allowed ceph config get mon mon_warn_on_insecure_global_id_reclaim
Zitat von Sergio Rabellino <rabellino@di.unito.it>:
Thanks for your prompt reply, dump follows:
# ceph osd dump |grep require_osd_release require_osd_release octopus We are aware of this setting and change it on release change (we did in the last days mimic->nautilus->octopus). The very strange thing it's that we upgraded luminous->mimic more than two years ago.
About librados, I get:
# rados -v ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable) so it seems fine to me.
Il 15/05/2025 11:01, Sinan Polat ha scritto:
What is the output of: ceph osd dump | grep require_osd_release
Have you upgraded OpenStack (librados) as well?
Op do 15 mei 2025 om 10:50 schreef Sergio Rabellino <rabellino@di.unito.it <mailto:rabellino@di.unito.it>>:
Dear list,
we are upgrading our ceph infrastructure from mimic to octopus (please be kind, we known that we are working with "old" tools, but these ceph releases are tied to our openstack installation needs) and _*all*_ the ceph actors (mon/mgr/osd/rgw - no mds as we do not serve filesystem) were upgraded fine as follow:
> # ceph -v > ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) > octopus (stable) We're using a ubuntu/juju orchestrator for managing ceph (and openstack too).
It seems all ok, but if I ask a mon to dump the sessions, we get that_all of them_ are on luminous:
> "con_features": 4540138292840890367, > "con_features_hex": "3f01cfb8ffedffff", > "con_features_release": "luminous", We found this oddity when we tried to unset the "insecure global_id reclaim" flag and all things got broken, so we had to reactivate the flag.
All ceph network layers are "closed", so we're not so urged to remove the flag, but we would like to understand if this is a known problem or an error done by us.
Any hints ?
thanks in advance.
-- ing. Sergio Rabellino
Università degli Studi di Torino Dipartimento di Informatica Tecnico di Ricerca Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 C.so Svizzera , 185 - 10149 - Torino
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
-- ing. Sergio Rabellino
Università degli Studi di Torino Dipartimento di Informatica Tecnico di Ricerca Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 C.so Svizzera , 185 - 10149 - Torino
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- ing. Sergio Rabellino Università degli Studi di Torino Dipartimento di Informatica Tecnico di Ricerca Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 C.so Svizzera , 185 - 10149 - Torino <http://www.di.unito.it>
To be clear, did you do an API / nova restart, or a reboot of the OS within the VM? The latter won’t effect the package update.
On May 15, 2025, at 8:45 AM, Sergio Rabellino <rabellino@di.unito.it> wrote:
Yes, they are up-to-date at the same level. We can't use live-migration because most of the VMs are equipped with GPUs and do not move between nodes.
Anyway we restarted some VMs without changes.
Thanks anyway.
Il 15/05/2025 12:04, Jan Johansson ha scritto:
Hi.
Have you verified that the new libraries on the openstack compute nodes are in use? If you have upgraded librados on the compute nodes you also need to stop and start all the vms to start using the new libraries. You can do that by stop/start or live-migration for example.
We usually do it through live-migration during the openstack upgrade of all the compute nodes (we empty each node before upgrading openstack, libvirt, qemu and ceph components, then live-migrate back).
/Jan
On 2025-05-15 11:39, Eugen Block wrote:
The output of 'ceph features' (or the mon session output) does not reflect that it's an older client. It just represents a collection of supported features. This has been discussed multiple times on this list. This is not unusual.
Regarding the insecure global_id, does your cluster warn about it? Can you share these configs?
ceph config get mon mon_warn_on_insecure_global_id_reclaim_allowed ceph config get mon mon_warn_on_insecure_global_id_reclaim
Zitat von Sergio Rabellino <rabellino@di.unito.it>:
Thanks for your prompt reply, dump follows:
# ceph osd dump |grep require_osd_release require_osd_release octopus We are aware of this setting and change it on release change (we did in the last days mimic->nautilus->octopus). The very strange thing it's that we upgraded luminous->mimic more than two years ago.
About librados, I get:
# rados -v ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable) so it seems fine to me.
Il 15/05/2025 11:01, Sinan Polat ha scritto:
What is the output of: ceph osd dump | grep require_osd_release
Have you upgraded OpenStack (librados) as well?
Op do 15 mei 2025 om 10:50 schreef Sergio Rabellino <rabellino@di.unito.it <mailto:rabellino@di.unito.it>>:
Dear list,
we are upgrading our ceph infrastructure from mimic to octopus (please be kind, we known that we are working with "old" tools, but these ceph releases are tied to our openstack installation needs) and _*all*_ the ceph actors (mon/mgr/osd/rgw - no mds as we do not serve filesystem) were upgraded fine as follow:
# ceph -v ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable) We're using a ubuntu/juju orchestrator for managing ceph (and openstack too).
It seems all ok, but if I ask a mon to dump the sessions, we get that_all of them_ are on luminous:
"con_features": 4540138292840890367, "con_features_hex": "3f01cfb8ffedffff", "con_features_release": "luminous",
We found this oddity when we tried to unset the "insecure global_id reclaim" flag and all things got broken, so we had to reactivate the flag.
All ceph network layers are "closed", so we're not so urged to remove the flag, but we would like to understand if this is a known problem or an error done by us.
Any hints ?
thanks in advance.
-- ing. Sergio Rabellino
Università degli Studi di Torino Dipartimento di Informatica Tecnico di Ricerca Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 C.so Svizzera , 185 - 10149 - Torino
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
-- ing. Sergio Rabellino
Università degli Studi di Torino Dipartimento di Informatica Tecnico di Ricerca Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 C.so Svizzera , 185 - 10149 - Torino
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- ing. Sergio Rabellino
Università degli Studi di Torino Dipartimento di Informatica Tecnico di Ricerca Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 C.so Svizzera , 185 - 10149 - Torino
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
We did a complete reboot of a bare-metal node and of all nova, so infrastructure. Obviously restarting the os inside the vm does not reload the underlying connection to ceph volumes. Il 15/05/2025 17:55, Anthony D'Atri ha scritto:
To be clear, did you do an API / nova restart, or a reboot of the OS within the VM? The latter won’t effect the package update.
On May 15, 2025, at 8:45 AM, Sergio Rabellino <rabellino@di.unito.it> wrote:
Yes, they are up-to-date at the same level. We can't use live-migration because most of the VMs are equipped with GPUs and do not move between nodes.
Anyway we restarted some VMs without changes.
Thanks anyway.
Il 15/05/2025 12:04, Jan Johansson ha scritto:
Hi.
Have you verified that the new libraries on the openstack compute nodes are in use? If you have upgraded librados on the compute nodes you also need to stop and start all the vms to start using the new libraries. You can do that by stop/start or live-migration for example.
We usually do it through live-migration during the openstack upgrade of all the compute nodes (we empty each node before upgrading openstack, libvirt, qemu and ceph components, then live-migrate back).
/Jan
On 2025-05-15 11:39, Eugen Block wrote:
The output of 'ceph features' (or the mon session output) does not reflect that it's an older client. It just represents a collection of supported features. This has been discussed multiple times on this list. This is not unusual.
Regarding the insecure global_id, does your cluster warn about it? Can you share these configs?
ceph config get mon mon_warn_on_insecure_global_id_reclaim_allowed ceph config get mon mon_warn_on_insecure_global_id_reclaim
Zitat von Sergio Rabellino <rabellino@di.unito.it>:
Thanks for your prompt reply, dump follows:
# ceph osd dump |grep require_osd_release require_osd_release octopus We are aware of this setting and change it on release change (we did in the last days mimic->nautilus->octopus). The very strange thing it's that we upgraded luminous->mimic more than two years ago.
About librados, I get:
# rados -v ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable) so it seems fine to me.
Il 15/05/2025 11:01, Sinan Polat ha scritto:
What is the output of: ceph osd dump | grep require_osd_release
Have you upgraded OpenStack (librados) as well?
Op do 15 mei 2025 om 10:50 schreef Sergio Rabellino <rabellino@di.unito.it <mailto:rabellino@di.unito.it>>:
Dear list,
we are upgrading our ceph infrastructure from mimic to octopus (please be kind, we known that we are working with "old" tools, but these ceph releases are tied to our openstack installation needs) and _*all*_ the ceph actors (mon/mgr/osd/rgw - no mds as we do not serve filesystem) were upgraded fine as follow:
> # ceph -v > ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) > octopus (stable) We're using a ubuntu/juju orchestrator for managing ceph (and openstack too).
It seems all ok, but if I ask a mon to dump the sessions, we get that_all of them_ are on luminous:
> "con_features": 4540138292840890367, > "con_features_hex": "3f01cfb8ffedffff", > "con_features_release": "luminous", We found this oddity when we tried to unset the "insecure global_id reclaim" flag and all things got broken, so we had to reactivate the flag.
All ceph network layers are "closed", so we're not so urged to remove the flag, but we would like to understand if this is a known problem or an error done by us.
Any hints ?
thanks in advance.
-- ing. Sergio Rabellino
Università degli Studi di Torino Dipartimento di Informatica Tecnico di Ricerca Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 C.so Svizzera , 185 - 10149 - Torino
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
-- ing. Sergio Rabellino
Università degli Studi di Torino Dipartimento di Informatica Tecnico di Ricerca Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 C.so Svizzera , 185 - 10149 - Torino
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- ing. Sergio Rabellino
Università degli Studi di Torino Dipartimento di Informatica Tecnico di Ricerca Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 C.so Svizzera , 185 - 10149 - Torino
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- ing. Sergio Rabellino Università degli Studi di Torino Dipartimento di Informatica Tecnico di Ricerca Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 C.so Svizzera , 185 - 10149 - Torino <http://www.di.unito.it>
Sorry to bother the list with old questions again, I'm a little new into this list. But if 'ceph daemon mon.name sessions' does not reports the actual protocol used, how I can find this info ? Both the configs you asked for are "true": are you saying that I'm concerned on null problem ? If so (and it may be), why if we remove the allow of insecure_global_id on mon(s) all the VMs stop talking with the underlying volumes ? Bests. Il 15/05/2025 11:39, Eugen Block ha scritto:
The output of 'ceph features' (or the mon session output) does not reflect that it's an older client. It just represents a collection of supported features. This has been discussed multiple times on this list. This is not unusual.
Regarding the insecure global_id, does your cluster warn about it? Can you share these configs?
ceph config get mon mon_warn_on_insecure_global_id_reclaim_allowed ceph config get mon mon_warn_on_insecure_global_id_reclaim
Zitat von Sergio Rabellino <rabellino@di.unito.it>:
Thanks for your prompt reply, dump follows:
# ceph osd dump |grep require_osd_release require_osd_release octopus We are aware of this setting and change it on release change (we did in the last days mimic->nautilus->octopus). The very strange thing it's that we upgraded luminous->mimic more than two years ago.
About librados, I get:
# rados -v ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable) so it seems fine to me.
Il 15/05/2025 11:01, Sinan Polat ha scritto:
What is the output of: ceph osd dump | grep require_osd_release
Have you upgraded OpenStack (librados) as well?
Op do 15 mei 2025 om 10:50 schreef Sergio Rabellino <rabellino@di.unito.it <mailto:rabellino@di.unito.it>>:
Dear list,
we are upgrading our ceph infrastructure from mimic to octopus (please be kind, we known that we are working with "old" tools, but these ceph releases are tied to our openstack installation needs) and _*all*_ the ceph actors (mon/mgr/osd/rgw - no mds as we do not serve filesystem) were upgraded fine as follow:
> # ceph -v > ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) > octopus (stable) We're using a ubuntu/juju orchestrator for managing ceph (and openstack too).
It seems all ok, but if I ask a mon to dump the sessions, we get that_all of them_ are on luminous:
> "con_features": 4540138292840890367, > "con_features_hex": "3f01cfb8ffedffff", > "con_features_release": "luminous", We found this oddity when we tried to unset the "insecure global_id reclaim" flag and all things got broken, so we had to reactivate the flag.
All ceph network layers are "closed", so we're not so urged to remove the flag, but we would like to understand if this is a known problem or an error done by us.
Any hints ?
thanks in advance.
-- ing. Sergio Rabellino
Università degli Studi di Torino Dipartimento di Informatica Tecnico di Ricerca Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 C.so Svizzera , 185 - 10149 - Torino
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
-- ing. Sergio Rabellino
Università degli Studi di Torino Dipartimento di Informatica Tecnico di Ricerca Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 C.so Svizzera , 185 - 10149 - Torino
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- ing. Sergio Rabellino Università degli Studi di Torino Dipartimento di Informatica Tecnico di Ricerca Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 C.so Svizzera , 185 - 10149 - Torino <http://www.di.unito.it>
It's been a while since I dealt with all that global_id stuff: https://docs.ceph.com/en/octopus/rados/operations/health-checks/#auth-insecu... Can you share this output? for i in $(ceph config ls | grep global_id); do echo "Config: $i"; ceph config get mon $i; done Config: mon_warn_on_insecure_global_id_reclaim true Config: mon_warn_on_insecure_global_id_reclaim_allowed false Config: auth_allow_insecure_global_id_reclaim true Config: auth_expose_insecure_global_id_reclaim true We still allow older clients, but we're aware and don't want the warning. Zitat von Sergio Rabellino <rabellino@di.unito.it>:
Sorry to bother the list with old questions again, I'm a little new into this list.
But if 'ceph daemon mon.name sessions' does not reports the actual protocol used, how I can find this info ?
Both the configs you asked for are "true": are you saying that I'm concerned on null problem ?
If so (and it may be), why if we remove the allow of insecure_global_id on mon(s) all the VMs stop talking with the underlying volumes ?
Bests.
Il 15/05/2025 11:39, Eugen Block ha scritto:
The output of 'ceph features' (or the mon session output) does not reflect that it's an older client. It just represents a collection of supported features. This has been discussed multiple times on this list. This is not unusual.
Regarding the insecure global_id, does your cluster warn about it? Can you share these configs?
ceph config get mon mon_warn_on_insecure_global_id_reclaim_allowed ceph config get mon mon_warn_on_insecure_global_id_reclaim
Zitat von Sergio Rabellino <rabellino@di.unito.it>:
Thanks for your prompt reply, dump follows:
# ceph osd dump |grep require_osd_release require_osd_release octopus We are aware of this setting and change it on release change (we did in the last days mimic->nautilus->octopus). The very strange thing it's that we upgraded luminous->mimic more than two years ago.
About librados, I get:
# rados -v ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable) so it seems fine to me.
Il 15/05/2025 11:01, Sinan Polat ha scritto:
What is the output of: ceph osd dump | grep require_osd_release
Have you upgraded OpenStack (librados) as well?
Op do 15 mei 2025 om 10:50 schreef Sergio Rabellino <rabellino@di.unito.it <mailto:rabellino@di.unito.it>>:
Dear list,
we are upgrading our ceph infrastructure from mimic to octopus (please be kind, we known that we are working with "old" tools, but these ceph releases are tied to our openstack installation needs) and _*all*_ the ceph actors (mon/mgr/osd/rgw - no mds as we do not serve filesystem) were upgraded fine as follow:
> # ceph -v > ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) > octopus (stable) We're using a ubuntu/juju orchestrator for managing ceph (and openstack too).
It seems all ok, but if I ask a mon to dump the sessions, we get that_all of them_ are on luminous:
> "con_features": 4540138292840890367, > "con_features_hex": "3f01cfb8ffedffff", > "con_features_release": "luminous", We found this oddity when we tried to unset the "insecure global_id reclaim" flag and all things got broken, so we had to reactivate the flag.
All ceph network layers are "closed", so we're not so urged to remove the flag, but we would like to understand if this is a known problem or an error done by us.
Any hints ?
thanks in advance.
-- ing. Sergio Rabellino
Università degli Studi di Torino Dipartimento di Informatica Tecnico di Ricerca Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 C.so Svizzera , 185 - 10149 - Torino
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
-- ing. Sergio Rabellino
Università degli Studi di Torino Dipartimento di Informatica Tecnico di Ricerca Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 C.so Svizzera , 185 - 10149 - Torino
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- ing. Sergio Rabellino
Università degli Studi di Torino Dipartimento di Informatica Tecnico di Ricerca Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 C.so Svizzera , 185 - 10149 - Torino
this is the output by now that allows normal operation. Changing the third breaks client access (and maybe other bad behaviour)
root@juju-XXX7ca-YY62-lxd-0:~# for i in $(ceph config ls | grep global_id); do echo "Config: $i"; ceph config get mon $i; done Config: mon_warn_on_insecure_global_id_reclaim true Config: mon_warn_on_insecure_global_id_reclaim_allowed true Config: auth_allow_insecure_global_id_reclaim true Config: auth_expose_insecure_global_id_reclaim true We could disable the warnings, but is this affordable ?
Il 16/05/2025 13:45, Eugen Block ha scritto:
It's been a while since I dealt with all that global_id stuff:
https://docs.ceph.com/en/octopus/rados/operations/health-checks/#auth-insecu...
Can you share this output?
for i in $(ceph config ls | grep global_id); do echo "Config: $i"; ceph config get mon $i; done Config: mon_warn_on_insecure_global_id_reclaim true Config: mon_warn_on_insecure_global_id_reclaim_allowed false Config: auth_allow_insecure_global_id_reclaim true Config: auth_expose_insecure_global_id_reclaim true
We still allow older clients, but we're aware and don't want the warning.
Zitat von Sergio Rabellino <rabellino@di.unito.it>:
Sorry to bother the list with old questions again, I'm a little new into this list.
But if 'ceph daemon mon.name sessions' does not reports the actual protocol used, how I can find this info ?
Both the configs you asked for are "true": are you saying that I'm concerned on null problem ?
If so (and it may be), why if we remove the allow of insecure_global_id on mon(s) all the VMs stop talking with the underlying volumes ?
Bests.
Il 15/05/2025 11:39, Eugen Block ha scritto:
The output of 'ceph features' (or the mon session output) does not reflect that it's an older client. It just represents a collection of supported features. This has been discussed multiple times on this list. This is not unusual.
Regarding the insecure global_id, does your cluster warn about it? Can you share these configs?
ceph config get mon mon_warn_on_insecure_global_id_reclaim_allowed ceph config get mon mon_warn_on_insecure_global_id_reclaim
Zitat von Sergio Rabellino <rabellino@di.unito.it>:
Thanks for your prompt reply, dump follows:
# ceph osd dump |grep require_osd_release require_osd_release octopus We are aware of this setting and change it on release change (we did in the last days mimic->nautilus->octopus). The very strange thing it's that we upgraded luminous->mimic more than two years ago.
About librados, I get:
# rados -v ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable) so it seems fine to me.
Il 15/05/2025 11:01, Sinan Polat ha scritto:
What is the output of: ceph osd dump | grep require_osd_release
Have you upgraded OpenStack (librados) as well?
Op do 15 mei 2025 om 10:50 schreef Sergio Rabellino <rabellino@di.unito.it <mailto:rabellino@di.unito.it>>:
Dear list,
we are upgrading our ceph infrastructure from mimic to octopus (please be kind, we known that we are working with "old" tools, but these ceph releases are tied to our openstack installation needs) and _*all*_ the ceph actors (mon/mgr/osd/rgw - no mds as we do not serve filesystem) were upgraded fine as follow:
> # ceph -v > ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) > octopus (stable) We're using a ubuntu/juju orchestrator for managing ceph (and openstack too).
It seems all ok, but if I ask a mon to dump the sessions, we get that_all of them_ are on luminous:
> "con_features": 4540138292840890367, > "con_features_hex": "3f01cfb8ffedffff", > "con_features_release": "luminous", We found this oddity when we tried to unset the "insecure global_id reclaim" flag and all things got broken, so we had to reactivate the flag.
All ceph network layers are "closed", so we're not so urged to remove the flag, but we would like to understand if this is a known problem or an error done by us.
Any hints ?
thanks in advance.
-- ing. Sergio Rabellino
Università degli Studi di Torino Dipartimento di Informatica Tecnico di Ricerca Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 C.so Svizzera , 185 - 10149 - Torino
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
-- ing. Sergio Rabellino
Università degli Studi di Torino Dipartimento di Informatica Tecnico di Ricerca Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 C.so Svizzera , 185 - 10149 - Torino
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- ing. Sergio Rabellino
Università degli Studi di Torino Dipartimento di Informatica Tecnico di Ricerca Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 C.so Svizzera , 185 - 10149 - Torino
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- ing. Sergio Rabellino Università degli Studi di Torino Dipartimento di Informatica Tecnico di Ricerca Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 C.so Svizzera , 185 - 10149 - Torino <http://www.di.unito.it>
If auth_expose_insecure_global_id_reclaim is true, do you see the client in 'ceph health detail'? Zitat von Sergio Rabellino <rabellino@di.unito.it>:
this is the output by now that allows normal operation. Changing the third breaks client access (and maybe other bad behaviour)
root@juju-XXX7ca-YY62-lxd-0:~# for i in $(ceph config ls | grep global_id); do echo "Config: $i"; ceph config get mon $i; done Config: mon_warn_on_insecure_global_id_reclaim true Config: mon_warn_on_insecure_global_id_reclaim_allowed true Config: auth_allow_insecure_global_id_reclaim true Config: auth_expose_insecure_global_id_reclaim true We could disable the warnings, but is this affordable ?
Il 16/05/2025 13:45, Eugen Block ha scritto:
It's been a while since I dealt with all that global_id stuff:
https://docs.ceph.com/en/octopus/rados/operations/health-checks/#auth-insecu... Can you share this output?
for i in $(ceph config ls | grep global_id); do echo "Config: $i"; ceph config get mon $i; done Config: mon_warn_on_insecure_global_id_reclaim true Config: mon_warn_on_insecure_global_id_reclaim_allowed false Config: auth_allow_insecure_global_id_reclaim true Config: auth_expose_insecure_global_id_reclaim true
We still allow older clients, but we're aware and don't want the warning.
Zitat von Sergio Rabellino <rabellino@di.unito.it>:
Sorry to bother the list with old questions again, I'm a little new into this list.
But if 'ceph daemon mon.name sessions' does not reports the actual protocol used, how I can find this info ?
Both the configs you asked for are "true": are you saying that I'm concerned on null problem ?
If so (and it may be), why if we remove the allow of insecure_global_id on mon(s) all the VMs stop talking with the underlying volumes ?
Bests.
Il 15/05/2025 11:39, Eugen Block ha scritto:
The output of 'ceph features' (or the mon session output) does not reflect that it's an older client. It just represents a collection of supported features. This has been discussed multiple times on this list. This is not unusual.
Regarding the insecure global_id, does your cluster warn about it? Can you share these configs?
ceph config get mon mon_warn_on_insecure_global_id_reclaim_allowed ceph config get mon mon_warn_on_insecure_global_id_reclaim
Zitat von Sergio Rabellino <rabellino@di.unito.it>:
Thanks for your prompt reply, dump follows:
# ceph osd dump |grep require_osd_release require_osd_release octopus We are aware of this setting and change it on release change (we did in the last days mimic->nautilus->octopus). The very strange thing it's that we upgraded luminous->mimic more than two years ago.
About librados, I get:
# rados -v ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable) so it seems fine to me.
Il 15/05/2025 11:01, Sinan Polat ha scritto:
What is the output of: ceph osd dump | grep require_osd_release
Have you upgraded OpenStack (librados) as well?
Op do 15 mei 2025 om 10:50 schreef Sergio Rabellino <rabellino@di.unito.it <mailto:rabellino@di.unito.it>>:
Dear list,
we are upgrading our ceph infrastructure from mimic to octopus (please be kind, we known that we are working with "old" tools, but these ceph releases are tied to our openstack installation needs) and _*all*_ the ceph actors (mon/mgr/osd/rgw - no mds as we do not serve filesystem) were upgraded fine as follow:
> # ceph -v > ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) > octopus (stable) We're using a ubuntu/juju orchestrator for managing ceph (and openstack too).
It seems all ok, but if I ask a mon to dump the sessions, we get that_all of them_ are on luminous:
> "con_features": 4540138292840890367, > "con_features_hex": "3f01cfb8ffedffff", > "con_features_release": "luminous", We found this oddity when we tried to unset the "insecure global_id reclaim" flag and all things got broken, so we had to reactivate the flag.
All ceph network layers are "closed", so we're not so urged to remove the flag, but we would like to understand if this is a known problem or an error done by us.
Any hints ?
thanks in advance.
-- ing. Sergio Rabellino
Università degli Studi di Torino Dipartimento di Informatica Tecnico di Ricerca Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 C.so Svizzera , 185 - 10149 - Torino
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io <mailto:ceph-users-leave@ceph.io>
-- ing. Sergio Rabellino
Università degli Studi di Torino Dipartimento di Informatica Tecnico di Ricerca Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 C.so Svizzera , 185 - 10149 - Torino
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- ing. Sergio Rabellino
Università degli Studi di Torino Dipartimento di Informatica Tecnico di Ricerca Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 C.so Svizzera , 185 - 10149 - Torino
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- ing. Sergio Rabellino
Università degli Studi di Torino Dipartimento di Informatica Tecnico di Ricerca Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 C.so Svizzera , 185 - 10149 - Torino
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Probably yes, but the list it's too long, so in the middle there's a "..." : that's why I searched for a ceph command able to return all the results for session properties without ellipsing. For this purpose, in my search I found the command "ceph daemon mon-name sessions" were I saw the "luminous" word that in my mind was "wrong" from my top post of this thread. Il 16/05/2025 14:56, Eugen Block ha scritto:
If auth_expose_insecure_global_id_reclaim is true, do you see the client in 'ceph health detail'?
Zitat von Sergio Rabellino <rabellino@di.unito.it>:
this is the output by now that allows normal operation. Changing the third breaks client access (and maybe other bad behaviour)
root@juju-XXX7ca-YY62-lxd-0:~# for i in $(ceph config ls | grep global_id); do echo "Config: $i"; ceph config get mon $i; done Config: mon_warn_on_insecure_global_id_reclaim true Config: mon_warn_on_insecure_global_id_reclaim_allowed true Config: auth_allow_insecure_global_id_reclaim true Config: auth_expose_insecure_global_id_reclaim true We could disable the warnings, but is this affordable ?
Il 16/05/2025 13:45, Eugen Block ha scritto:
It's been a while since I dealt with all that global_id stuff:
https://docs.ceph.com/en/octopus/rados/operations/health-checks/#auth-insecu... Can you share this output?
for i in $(ceph config ls | grep global_id); do echo "Config: $i"; ceph config get mon $i; done Config: mon_warn_on_insecure_global_id_reclaim true Config: mon_warn_on_insecure_global_id_reclaim_allowed false Config: auth_allow_insecure_global_id_reclaim true Config: auth_expose_insecure_global_id_reclaim true
We still allow older clients, but we're aware and don't want the warning.
Zitat von Sergio Rabellino <rabellino@di.unito.it>:
Sorry to bother the list with old questions again, I'm a little new into this list.
But if 'ceph daemon mon.name sessions' does not reports the actual protocol used, how I can find this info ?
Both the configs you asked for are "true": are you saying that I'm concerned on null problem ?
If so (and it may be), why if we remove the allow of insecure_global_id on mon(s) all the VMs stop talking with the underlying volumes ?
Bests.
Il 15/05/2025 11:39, Eugen Block ha scritto:
The output of 'ceph features' (or the mon session output) does not reflect that it's an older client. It just represents a collection of supported features. This has been discussed multiple times on this list. This is not unusual.
Regarding the insecure global_id, does your cluster warn about it? Can you share these configs?
ceph config get mon mon_warn_on_insecure_global_id_reclaim_allowed ceph config get mon mon_warn_on_insecure_global_id_reclaim
Zitat von Sergio Rabellino <rabellino@di.unito.it>:
Thanks for your prompt reply, dump follows:
> # ceph osd dump |grep require_osd_release > require_osd_release octopus We are aware of this setting and change it on release change (we did in the last days mimic->nautilus->octopus). The very strange thing it's that we upgraded luminous->mimic more than two years ago.
About librados, I get: > # rados -v > ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) > octopus (stable) so it seems fine to me.
Il 15/05/2025 11:01, Sinan Polat ha scritto: > What is the output of: > ceph osd dump | grep require_osd_release > > Have you upgraded OpenStack (librados) as well? > > Op do 15 mei 2025 om 10:50 schreef Sergio Rabellino > <rabellino@di.unito.it <mailto:rabellino@di.unito.it>>: > > Dear list, > > we are upgrading our ceph infrastructure from mimic to octopus > (please > be kind, we known that we are working with "old" tools, but > these > ceph > releases are tied to our openstack installation needs) and > _*all*_ > the > ceph actors (mon/mgr/osd/rgw - no mds as we do not serve > filesystem) > were upgraded fine as follow: > > > # ceph -v > > ceph version 15.2.17 > (8a82819d84cf884bd39c17e3236e0632ac146dc4) > > octopus (stable) > We're using a ubuntu/juju orchestrator for managing ceph (and > openstack > too). > > It seems all ok, but if I ask a mon to dump the sessions, we get > that_all of them_ are on luminous: > > > "con_features": 4540138292840890367, > > "con_features_hex": "3f01cfb8ffedffff", > > "con_features_release": "luminous", > We found this oddity when we tried to unset the "insecure > global_id > reclaim" flag and all things got broken, so we had to reactivate > the flag. > > All ceph network layers are "closed", so we're not so urged > to remove > the flag, but we would like to understand if this is a known > problem or > an error done by us. > > Any hints ? > > thanks in advance. > > -- ing. Sergio Rabellino > > Università degli Studi di Torino > Dipartimento di Informatica > Tecnico di Ricerca > Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 > C.so Svizzera , 185 - 10149 - Torino > > <http://www.di.unito.it> > > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > <mailto:ceph-users@ceph.io> > To unsubscribe send an email to ceph-users-leave@ceph.io > <mailto:ceph-users-leave@ceph.io> >
-- ing. Sergio Rabellino
Università degli Studi di Torino Dipartimento di Informatica Tecnico di Ricerca Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 C.so Svizzera , 185 - 10149 - Torino
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- ing. Sergio Rabellino
Università degli Studi di Torino Dipartimento di Informatica Tecnico di Ricerca Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 C.so Svizzera , 185 - 10149 - Torino
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- ing. Sergio Rabellino
Università degli Studi di Torino Dipartimento di Informatica Tecnico di Ricerca Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 C.so Svizzera , 185 - 10149 - Torino
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- ing. Sergio Rabellino Università degli Studi di Torino Dipartimento di Informatica Tecnico di Ricerca Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 C.so Svizzera , 185 - 10149 - Torino <http://www.di.unito.it>
The mon sessions dump also shows the global_id_status, this could help identify the clients to focus on. I would recommend to inspect this output: ceph tell mon.<MON> sessions | jq -r '.[] | .name,.socket_addr.addr,.global_id_status' and check each client. While looking at our own mon sessions, it seems like I could probably disable auth_allow_insecure_global_id_reclaim, looks like all older clients are either new_ok or reclaim_ok. And we still have plenty of "con_features_release" entries with "luminous", just FYI. I don't really recall doing anything special with our clients (also OpenStack servers), tbh. We upgraded them along with the Ceph versions, and now it looks okay. Maybe a reboot after an upgrade suffices, I don't know. Here's the list of possible global_id_status entries with a short comment: https://github.com/ceph/ceph/blob/df7b75547aab3bb055fff16588cd697e593dc19a/s... Zitat von Sergio Rabellino <rabellino@di.unito.it>:
Probably yes, but the list it's too long, so in the middle there's a "..." : that's why I searched for a ceph command able to return all the results for session properties without ellipsing.
For this purpose, in my search I found the command "ceph daemon mon-name sessions" were I saw the "luminous" word that in my mind was "wrong" from my top post of this thread.
Il 16/05/2025 14:56, Eugen Block ha scritto:
If auth_expose_insecure_global_id_reclaim is true, do you see the client in 'ceph health detail'?
Zitat von Sergio Rabellino <rabellino@di.unito.it>:
this is the output by now that allows normal operation. Changing the third breaks client access (and maybe other bad behaviour)
root@juju-XXX7ca-YY62-lxd-0:~# for i in $(ceph config ls | grep global_id); do echo "Config: $i"; ceph config get mon $i; done Config: mon_warn_on_insecure_global_id_reclaim true Config: mon_warn_on_insecure_global_id_reclaim_allowed true Config: auth_allow_insecure_global_id_reclaim true Config: auth_expose_insecure_global_id_reclaim true We could disable the warnings, but is this affordable ?
Il 16/05/2025 13:45, Eugen Block ha scritto:
It's been a while since I dealt with all that global_id stuff:
https://docs.ceph.com/en/octopus/rados/operations/health-checks/#auth-insecu... Can you share this output?
for i in $(ceph config ls | grep global_id); do echo "Config: $i"; ceph config get mon $i; done Config: mon_warn_on_insecure_global_id_reclaim true Config: mon_warn_on_insecure_global_id_reclaim_allowed false Config: auth_allow_insecure_global_id_reclaim true Config: auth_expose_insecure_global_id_reclaim true
We still allow older clients, but we're aware and don't want the warning.
Zitat von Sergio Rabellino <rabellino@di.unito.it>:
Sorry to bother the list with old questions again, I'm a little new into this list.
But if 'ceph daemon mon.name sessions' does not reports the actual protocol used, how I can find this info ?
Both the configs you asked for are "true": are you saying that I'm concerned on null problem ?
If so (and it may be), why if we remove the allow of insecure_global_id on mon(s) all the VMs stop talking with the underlying volumes ?
Bests.
Il 15/05/2025 11:39, Eugen Block ha scritto:
The output of 'ceph features' (or the mon session output) does not reflect that it's an older client. It just represents a collection of supported features. This has been discussed multiple times on this list. This is not unusual.
Regarding the insecure global_id, does your cluster warn about it? Can you share these configs?
ceph config get mon mon_warn_on_insecure_global_id_reclaim_allowed ceph config get mon mon_warn_on_insecure_global_id_reclaim
Zitat von Sergio Rabellino <rabellino@di.unito.it>:
> Thanks for your prompt reply, dump follows: > >> # ceph osd dump |grep require_osd_release >> require_osd_release octopus > We are aware of this setting and change it on release change > (we did in the last days mimic->nautilus->octopus). > The very strange thing it's that we upgraded luminous->mimic > more than two years ago. > > About librados, I get: >> # rados -v >> ceph version 15.2.17 >> (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable) > so it seems fine to me. > > Il 15/05/2025 11:01, Sinan Polat ha scritto: >> What is the output of: >> ceph osd dump | grep require_osd_release >> >> Have you upgraded OpenStack (librados) as well? >> >> Op do 15 mei 2025 om 10:50 schreef Sergio Rabellino >> <rabellino@di.unito.it <mailto:rabellino@di.unito.it>>: >> >> Dear list, >> >> we are upgrading our ceph infrastructure from mimic to octopus >> (please >> be kind, we known that we are working with "old" tools, but these >> ceph >> releases are tied to our openstack installation needs) and _*all*_ >> the >> ceph actors (mon/mgr/osd/rgw - no mds as we do not serve >> filesystem) >> were upgraded fine as follow: >> >> > # ceph -v >> > ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) >> > octopus (stable) >> We're using a ubuntu/juju orchestrator for managing ceph (and >> openstack >> too). >> >> It seems all ok, but if I ask a mon to dump the sessions, we get >> that_all of them_ are on luminous: >> >> > "con_features": 4540138292840890367, >> > "con_features_hex": "3f01cfb8ffedffff", >> > "con_features_release": "luminous", >> We found this oddity when we tried to unset the "insecure global_id >> reclaim" flag and all things got broken, so we had to reactivate >> the flag. >> >> All ceph network layers are "closed", so we're not so >> urged to remove >> the flag, but we would like to understand if this is a known >> problem or >> an error done by us. >> >> Any hints ? >> >> thanks in advance. >> >> -- ing. Sergio Rabellino >> >> Università degli Studi di Torino >> Dipartimento di Informatica >> Tecnico di Ricerca >> Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 >> C.so Svizzera , 185 - 10149 - Torino >> >> <http://www.di.unito.it> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> <mailto:ceph-users@ceph.io> >> To unsubscribe send an email to ceph-users-leave@ceph.io >> <mailto:ceph-users-leave@ceph.io> >> > > -- > ing. Sergio Rabellino > > Università degli Studi di Torino > Dipartimento di Informatica > Tecnico di Ricerca > Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 > C.so Svizzera , 185 - 10149 - Torino > > <http://www.di.unito.it> > > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- ing. Sergio Rabellino
Università degli Studi di Torino Dipartimento di Informatica Tecnico di Ricerca Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 C.so Svizzera , 185 - 10149 - Torino
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- ing. Sergio Rabellino
Università degli Studi di Torino Dipartimento di Informatica Tecnico di Ricerca Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 C.so Svizzera , 185 - 10149 - Torino
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- ing. Sergio Rabellino
Università degli Studi di Torino Dipartimento di Informatica Tecnico di Ricerca Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 C.so Svizzera , 185 - 10149 - Torino
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Just a quick update: I set auth_allow_insecure_global_id_reclaim to false because all the client sessions we had showed either new_ok or reclaim_ok in global_id_status. No complaints so far. :-) Zitat von Eugen Block <eblock@nde.ag>:
The mon sessions dump also shows the global_id_status, this could help identify the clients to focus on. I would recommend to inspect this output:
ceph tell mon.<MON> sessions | jq -r '.[] | .name,.socket_addr.addr,.global_id_status'
and check each client.
While looking at our own mon sessions, it seems like I could probably disable auth_allow_insecure_global_id_reclaim, looks like all older clients are either new_ok or reclaim_ok. And we still have plenty of "con_features_release" entries with "luminous", just FYI. I don't really recall doing anything special with our clients (also OpenStack servers), tbh. We upgraded them along with the Ceph versions, and now it looks okay. Maybe a reboot after an upgrade suffices, I don't know.
Here's the list of possible global_id_status entries with a short comment:
https://github.com/ceph/ceph/blob/df7b75547aab3bb055fff16588cd697e593dc19a/s...
Zitat von Sergio Rabellino <rabellino@di.unito.it>:
Probably yes, but the list it's too long, so in the middle there's a "..." : that's why I searched for a ceph command able to return all the results for session properties without ellipsing.
For this purpose, in my search I found the command "ceph daemon mon-name sessions" were I saw the "luminous" word that in my mind was "wrong" from my top post of this thread.
Il 16/05/2025 14:56, Eugen Block ha scritto:
If auth_expose_insecure_global_id_reclaim is true, do you see the client in 'ceph health detail'?
Zitat von Sergio Rabellino <rabellino@di.unito.it>:
this is the output by now that allows normal operation. Changing the third breaks client access (and maybe other bad behaviour)
root@juju-XXX7ca-YY62-lxd-0:~# for i in $(ceph config ls | grep global_id); do echo "Config: $i"; ceph config get mon $i; done Config: mon_warn_on_insecure_global_id_reclaim true Config: mon_warn_on_insecure_global_id_reclaim_allowed true Config: auth_allow_insecure_global_id_reclaim true Config: auth_expose_insecure_global_id_reclaim true We could disable the warnings, but is this affordable ?
Il 16/05/2025 13:45, Eugen Block ha scritto:
It's been a while since I dealt with all that global_id stuff:
https://docs.ceph.com/en/octopus/rados/operations/health-checks/#auth-insecu... Can you share this output?
for i in $(ceph config ls | grep global_id); do echo "Config: $i"; ceph config get mon $i; done Config: mon_warn_on_insecure_global_id_reclaim true Config: mon_warn_on_insecure_global_id_reclaim_allowed false Config: auth_allow_insecure_global_id_reclaim true Config: auth_expose_insecure_global_id_reclaim true
We still allow older clients, but we're aware and don't want the warning.
Zitat von Sergio Rabellino <rabellino@di.unito.it>:
Sorry to bother the list with old questions again, I'm a little new into this list.
But if 'ceph daemon mon.name sessions' does not reports the actual protocol used, how I can find this info ?
Both the configs you asked for are "true": are you saying that I'm concerned on null problem ?
If so (and it may be), why if we remove the allow of insecure_global_id on mon(s) all the VMs stop talking with the underlying volumes ?
Bests.
Il 15/05/2025 11:39, Eugen Block ha scritto: > The output of 'ceph features' (or the mon session output) does > not reflect that it's an older client. It just represents a > collection of supported features. This has been discussed > multiple times on this list. This is not unusual. > > Regarding the insecure global_id, does your cluster warn about > it? Can you share these configs? > > ceph config get mon mon_warn_on_insecure_global_id_reclaim_allowed > ceph config get mon mon_warn_on_insecure_global_id_reclaim > > > Zitat von Sergio Rabellino <rabellino@di.unito.it>: > >> Thanks for your prompt reply, dump follows: >> >>> # ceph osd dump |grep require_osd_release >>> require_osd_release octopus >> We are aware of this setting and change it on release change >> (we did in the last days mimic->nautilus->octopus). >> The very strange thing it's that we upgraded luminous->mimic >> more than two years ago. >> >> About librados, I get: >>> # rados -v >>> ceph version 15.2.17 >>> (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable) >> so it seems fine to me. >> >> Il 15/05/2025 11:01, Sinan Polat ha scritto: >>> What is the output of: >>> ceph osd dump | grep require_osd_release >>> >>> Have you upgraded OpenStack (librados) as well? >>> >>> Op do 15 mei 2025 om 10:50 schreef Sergio Rabellino >>> <rabellino@di.unito.it <mailto:rabellino@di.unito.it>>: >>> >>> Dear list, >>> >>> we are upgrading our ceph infrastructure from mimic to octopus >>> (please >>> be kind, we known that we are working with "old" tools, but these >>> ceph >>> releases are tied to our openstack installation needs) and _*all*_ >>> the >>> ceph actors (mon/mgr/osd/rgw - no mds as we do not serve >>> filesystem) >>> were upgraded fine as follow: >>> >>> > # ceph -v >>> > ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) >>> > octopus (stable) >>> We're using a ubuntu/juju orchestrator for managing ceph (and >>> openstack >>> too). >>> >>> It seems all ok, but if I ask a mon to dump the sessions, we get >>> that_all of them_ are on luminous: >>> >>> > "con_features": 4540138292840890367, >>> > "con_features_hex": "3f01cfb8ffedffff", >>> > "con_features_release": "luminous", >>> We found this oddity when we tried to unset the "insecure >>> global_id >>> reclaim" flag and all things got broken, so we had to reactivate >>> the flag. >>> >>> All ceph network layers are "closed", so we're not so >>> urged to remove >>> the flag, but we would like to understand if this is a known >>> problem or >>> an error done by us. >>> >>> Any hints ? >>> >>> thanks in advance. >>> >>> -- ing. Sergio Rabellino >>> >>> Università degli Studi di Torino >>> Dipartimento di Informatica >>> Tecnico di Ricerca >>> Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 >>> C.so Svizzera , 185 - 10149 - Torino >>> >>> <http://www.di.unito.it> >>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users@ceph.io >>> <mailto:ceph-users@ceph.io> >>> To unsubscribe send an email to ceph-users-leave@ceph.io >>> <mailto:ceph-users-leave@ceph.io> >>> >> >> -- >> ing. Sergio Rabellino >> >> Università degli Studi di Torino >> Dipartimento di Informatica >> Tecnico di Ricerca >> Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 >> C.so Svizzera , 185 - 10149 - Torino >> >> <http://www.di.unito.it> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io >
-- ing. Sergio Rabellino
Università degli Studi di Torino Dipartimento di Informatica Tecnico di Ricerca Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 C.so Svizzera , 185 - 10149 - Torino
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- ing. Sergio Rabellino
Università degli Studi di Torino Dipartimento di Informatica Tecnico di Ricerca Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 C.so Svizzera , 185 - 10149 - Torino
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- ing. Sergio Rabellino
Università degli Studi di Torino Dipartimento di Informatica Tecnico di Ricerca Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 C.so Svizzera , 185 - 10149 - Torino
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks for your hints, we'll check asap our status and I'll update this thread. Il 19/05/2025 21:14, Eugen Block ha scritto:
Just a quick update: I set auth_allow_insecure_global_id_reclaim to false because all the client sessions we had showed either new_ok or reclaim_ok in global_id_status. No complaints so far. :-)
Zitat von Eugen Block <eblock@nde.ag>:
The mon sessions dump also shows the global_id_status, this could help identify the clients to focus on. I would recommend to inspect this output:
ceph tell mon.<MON> sessions | jq -r '.[] | .name,.socket_addr.addr,.global_id_status'
and check each client.
While looking at our own mon sessions, it seems like I could probably disable auth_allow_insecure_global_id_reclaim, looks like all older clients are either new_ok or reclaim_ok. And we still have plenty of "con_features_release" entries with "luminous", just FYI. I don't really recall doing anything special with our clients (also OpenStack servers), tbh. We upgraded them along with the Ceph versions, and now it looks okay. Maybe a reboot after an upgrade suffices, I don't know.
Here's the list of possible global_id_status entries with a short comment:
https://github.com/ceph/ceph/blob/df7b75547aab3bb055fff16588cd697e593dc19a/s...
Zitat von Sergio Rabellino <rabellino@di.unito.it>:
Probably yes, but the list it's too long, so in the middle there's a "..." : that's why I searched for a ceph command able to return all the results for session properties without ellipsing.
For this purpose, in my search I found the command "ceph daemon mon-name sessions" were I saw the "luminous" word that in my mind was "wrong" from my top post of this thread.
Il 16/05/2025 14:56, Eugen Block ha scritto:
If auth_expose_insecure_global_id_reclaim is true, do you see the client in 'ceph health detail'?
Zitat von Sergio Rabellino <rabellino@di.unito.it>:
this is the output by now that allows normal operation. Changing the third breaks client access (and maybe other bad behaviour)
root@juju-XXX7ca-YY62-lxd-0:~# for i in $(ceph config ls | grep global_id); do echo "Config: $i"; ceph config get mon $i; done Config: mon_warn_on_insecure_global_id_reclaim true Config: mon_warn_on_insecure_global_id_reclaim_allowed true Config: auth_allow_insecure_global_id_reclaim true Config: auth_expose_insecure_global_id_reclaim true We could disable the warnings, but is this affordable ?
Il 16/05/2025 13:45, Eugen Block ha scritto:
It's been a while since I dealt with all that global_id stuff:
https://docs.ceph.com/en/octopus/rados/operations/health-checks/#auth-insecu... Can you share this output?
for i in $(ceph config ls | grep global_id); do echo "Config: $i"; ceph config get mon $i; done Config: mon_warn_on_insecure_global_id_reclaim true Config: mon_warn_on_insecure_global_id_reclaim_allowed false Config: auth_allow_insecure_global_id_reclaim true Config: auth_expose_insecure_global_id_reclaim true
We still allow older clients, but we're aware and don't want the warning.
Zitat von Sergio Rabellino <rabellino@di.unito.it>:
> Sorry to bother the list with old questions again, I'm a little > new into this list. > > But if 'ceph daemon mon.name sessions' does not reports the > actual protocol used, how I can find this info ? > > Both the configs you asked for are "true": are you saying that > I'm concerned on null problem ? > > If so (and it may be), why if we remove the allow of > insecure_global_id on mon(s) all the VMs stop talking with the > underlying volumes ? > > Bests. > > > Il 15/05/2025 11:39, Eugen Block ha scritto: >> The output of 'ceph features' (or the mon session output) does >> not reflect that it's an older client. It just represents a >> collection of supported features. This has been discussed >> multiple times on this list. This is not unusual. >> >> Regarding the insecure global_id, does your cluster warn about >> it? Can you share these configs? >> >> ceph config get mon mon_warn_on_insecure_global_id_reclaim_allowed >> ceph config get mon mon_warn_on_insecure_global_id_reclaim >> >> >> Zitat von Sergio Rabellino <rabellino@di.unito.it>: >> >>> Thanks for your prompt reply, dump follows: >>> >>>> # ceph osd dump |grep require_osd_release >>>> require_osd_release octopus >>> We are aware of this setting and change it on release change >>> (we did in the last days mimic->nautilus->octopus). >>> The very strange thing it's that we upgraded luminous->mimic >>> more than two years ago. >>> >>> About librados, I get: >>>> # rados -v >>>> ceph version 15.2.17 >>>> (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable) >>> so it seems fine to me. >>> >>> Il 15/05/2025 11:01, Sinan Polat ha scritto: >>>> What is the output of: >>>> ceph osd dump | grep require_osd_release >>>> >>>> Have you upgraded OpenStack (librados) as well? >>>> >>>> Op do 15 mei 2025 om 10:50 schreef Sergio Rabellino >>>> <rabellino@di.unito.it <mailto:rabellino@di.unito.it>>: >>>> >>>> Dear list, >>>> >>>> we are upgrading our ceph infrastructure from mimic to >>>> octopus >>>> (please >>>> be kind, we known that we are working with "old" tools, >>>> but these >>>> ceph >>>> releases are tied to our openstack installation needs) and >>>> _*all*_ >>>> the >>>> ceph actors (mon/mgr/osd/rgw - no mds as we do not serve >>>> filesystem) >>>> were upgraded fine as follow: >>>> >>>> > # ceph -v >>>> > ceph version 15.2.17 >>>> (8a82819d84cf884bd39c17e3236e0632ac146dc4) >>>> > octopus (stable) >>>> We're using a ubuntu/juju orchestrator for managing ceph (and >>>> openstack >>>> too). >>>> >>>> It seems all ok, but if I ask a mon to dump the sessions, >>>> we get >>>> that_all of them_ are on luminous: >>>> >>>> > "con_features": 4540138292840890367, >>>> > "con_features_hex": "3f01cfb8ffedffff", >>>> > "con_features_release": "luminous", >>>> We found this oddity when we tried to unset the "insecure >>>> global_id >>>> reclaim" flag and all things got broken, so we had to >>>> reactivate >>>> the flag. >>>> >>>> All ceph network layers are "closed", so we're not so >>>> urged to remove >>>> the flag, but we would like to understand if this is a known >>>> problem or >>>> an error done by us. >>>> >>>> Any hints ? >>>> >>>> thanks in advance. >>>> >>>> -- ing. Sergio Rabellino >>>> >>>> Università degli Studi di Torino >>>> Dipartimento di Informatica >>>> Tecnico di Ricerca >>>> Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 >>>> C.so Svizzera , 185 - 10149 - Torino >>>> >>>> <http://www.di.unito.it> >>>> >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users@ceph.io >>>> <mailto:ceph-users@ceph.io> >>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>> <mailto:ceph-users-leave@ceph.io> >>>> >>> >>> -- >>> ing. Sergio Rabellino >>> >>> Università degli Studi di Torino >>> Dipartimento di Informatica >>> Tecnico di Ricerca >>> Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 >>> C.so Svizzera , 185 - 10149 - Torino >>> >>> <http://www.di.unito.it> >>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users@ceph.io >>> To unsubscribe send an email to ceph-users-leave@ceph.io >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io >> > > -- > ing. Sergio Rabellino > > Università degli Studi di Torino > Dipartimento di Informatica > Tecnico di Ricerca > Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 > C.so Svizzera , 185 - 10149 - Torino > > <http://www.di.unito.it>
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- ing. Sergio Rabellino
Università degli Studi di Torino Dipartimento di Informatica Tecnico di Ricerca Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 C.so Svizzera , 185 - 10149 - Torino
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- ing. Sergio Rabellino
Università degli Studi di Torino Dipartimento di Informatica Tecnico di Ricerca Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 C.so Svizzera , 185 - 10149 - Torino
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- ing. Sergio Rabellino Università degli Studi di Torino Dipartimento di Informatica Tecnico di Ricerca Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 C.so Svizzera , 185 - 10149 - Torino <http://www.di.unito.it>
Dear all, as promised, here I am following up on this thread. After a some days since the last message on this thread, most of the insecure connections from clients "magically" disappeared and the latest ones were from not-yet upgraded openstack nova-compute nodes. Now that all nova nodes are completely up-to-date and rebooted at the same ceph distribution level, we were able to remove the allow_insecure from MONs and all things seems fine. Thank you again for the insightful suggestions. Il 19/05/2025 21:14, Eugen Block ha scritto:
Just a quick update: I set auth_allow_insecure_global_id_reclaim to false because all the client sessions we had showed either new_ok or reclaim_ok in global_id_status. No complaints so far. :-)
Zitat von Eugen Block <eblock@nde.ag>:
The mon sessions dump also shows the global_id_status, this could help identify the clients to focus on. I would recommend to inspect this output:
ceph tell mon.<MON> sessions | jq -r '.[] | .name,.socket_addr.addr,.global_id_status'
and check each client.
While looking at our own mon sessions, it seems like I could probably disable auth_allow_insecure_global_id_reclaim, looks like all older clients are either new_ok or reclaim_ok. And we still have plenty of "con_features_release" entries with "luminous", just FYI. I don't really recall doing anything special with our clients (also OpenStack servers), tbh. We upgraded them along with the Ceph versions, and now it looks okay. Maybe a reboot after an upgrade suffices, I don't know.
Here's the list of possible global_id_status entries with a short comment:
https://github.com/ceph/ceph/blob/df7b75547aab3bb055fff16588cd697e593dc19a/s...
Zitat von Sergio Rabellino <rabellino@di.unito.it>:
Probably yes, but the list it's too long, so in the middle there's a "..." : that's why I searched for a ceph command able to return all the results for session properties without ellipsing.
For this purpose, in my search I found the command "ceph daemon mon-name sessions" were I saw the "luminous" word that in my mind was "wrong" from my top post of this thread.
Il 16/05/2025 14:56, Eugen Block ha scritto:
If auth_expose_insecure_global_id_reclaim is true, do you see the client in 'ceph health detail'?
Zitat von Sergio Rabellino <rabellino@di.unito.it>:
this is the output by now that allows normal operation. Changing the third breaks client access (and maybe other bad behaviour)
root@juju-XXX7ca-YY62-lxd-0:~# for i in $(ceph config ls | grep global_id); do echo "Config: $i"; ceph config get mon $i; done Config: mon_warn_on_insecure_global_id_reclaim true Config: mon_warn_on_insecure_global_id_reclaim_allowed true Config: auth_allow_insecure_global_id_reclaim true Config: auth_expose_insecure_global_id_reclaim true We could disable the warnings, but is this affordable ?
Il 16/05/2025 13:45, Eugen Block ha scritto:
It's been a while since I dealt with all that global_id stuff:
https://docs.ceph.com/en/octopus/rados/operations/health-checks/#auth-insecu... Can you share this output?
for i in $(ceph config ls | grep global_id); do echo "Config: $i"; ceph config get mon $i; done Config: mon_warn_on_insecure_global_id_reclaim true Config: mon_warn_on_insecure_global_id_reclaim_allowed false Config: auth_allow_insecure_global_id_reclaim true Config: auth_expose_insecure_global_id_reclaim true
We still allow older clients, but we're aware and don't want the warning.
Zitat von Sergio Rabellino <rabellino@di.unito.it>:
> Sorry to bother the list with old questions again, I'm a little > new into this list. > > But if 'ceph daemon mon.name sessions' does not reports the > actual protocol used, how I can find this info ? > > Both the configs you asked for are "true": are you saying that > I'm concerned on null problem ? > > If so (and it may be), why if we remove the allow of > insecure_global_id on mon(s) all the VMs stop talking with the > underlying volumes ? > > Bests. > > > Il 15/05/2025 11:39, Eugen Block ha scritto: >> The output of 'ceph features' (or the mon session output) does >> not reflect that it's an older client. It just represents a >> collection of supported features. This has been discussed >> multiple times on this list. This is not unusual. >> >> Regarding the insecure global_id, does your cluster warn about >> it? Can you share these configs? >> >> ceph config get mon mon_warn_on_insecure_global_id_reclaim_allowed >> ceph config get mon mon_warn_on_insecure_global_id_reclaim >> >> >> Zitat von Sergio Rabellino <rabellino@di.unito.it>: >> >>> Thanks for your prompt reply, dump follows: >>> >>>> # ceph osd dump |grep require_osd_release >>>> require_osd_release octopus >>> We are aware of this setting and change it on release change >>> (we did in the last days mimic->nautilus->octopus). >>> The very strange thing it's that we upgraded luminous->mimic >>> more than two years ago. >>> >>> About librados, I get: >>>> # rados -v >>>> ceph version 15.2.17 >>>> (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable) >>> so it seems fine to me. >>> >>> Il 15/05/2025 11:01, Sinan Polat ha scritto: >>>> What is the output of: >>>> ceph osd dump | grep require_osd_release >>>> >>>> Have you upgraded OpenStack (librados) as well? >>>> >>>> Op do 15 mei 2025 om 10:50 schreef Sergio Rabellino >>>> <rabellino@di.unito.it <mailto:rabellino@di.unito.it>>: >>>> >>>> Dear list, >>>> >>>> we are upgrading our ceph infrastructure from mimic to >>>> octopus >>>> (please >>>> be kind, we known that we are working with "old" tools, >>>> but these >>>> ceph >>>> releases are tied to our openstack installation needs) and >>>> _*all*_ >>>> the >>>> ceph actors (mon/mgr/osd/rgw - no mds as we do not serve >>>> filesystem) >>>> were upgraded fine as follow: >>>> >>>> > # ceph -v >>>> > ceph version 15.2.17 >>>> (8a82819d84cf884bd39c17e3236e0632ac146dc4) >>>> > octopus (stable) >>>> We're using a ubuntu/juju orchestrator for managing ceph (and >>>> openstack >>>> too). >>>> >>>> It seems all ok, but if I ask a mon to dump the sessions, >>>> we get >>>> that_all of them_ are on luminous: >>>> >>>> > "con_features": 4540138292840890367, >>>> > "con_features_hex": "3f01cfb8ffedffff", >>>> > "con_features_release": "luminous", >>>> We found this oddity when we tried to unset the "insecure >>>> global_id >>>> reclaim" flag and all things got broken, so we had to >>>> reactivate >>>> the flag. >>>> >>>> All ceph network layers are "closed", so we're not so >>>> urged to remove >>>> the flag, but we would like to understand if this is a known >>>> problem or >>>> an error done by us. >>>> >>>> Any hints ? >>>> >>>> thanks in advance. >>>> >>>> -- ing. Sergio Rabellino >>>> >>>> Università degli Studi di Torino >>>> Dipartimento di Informatica >>>> Tecnico di Ricerca >>>> Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 >>>> C.so Svizzera , 185 - 10149 - Torino >>>> >>>> <http://www.di.unito.it> >>>> >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users@ceph.io >>>> <mailto:ceph-users@ceph.io> >>>> To unsubscribe send an email to ceph-users-leave@ceph.io >>>> <mailto:ceph-users-leave@ceph.io> >>>> >>> >>> -- >>> ing. Sergio Rabellino >>> >>> Università degli Studi di Torino >>> Dipartimento di Informatica >>> Tecnico di Ricerca >>> Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 >>> C.so Svizzera , 185 - 10149 - Torino >>> >>> <http://www.di.unito.it> >>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users@ceph.io >>> To unsubscribe send an email to ceph-users-leave@ceph.io >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io >> > > -- > ing. Sergio Rabellino > > Università degli Studi di Torino > Dipartimento di Informatica > Tecnico di Ricerca > Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 > C.so Svizzera , 185 - 10149 - Torino > > <http://www.di.unito.it>
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- ing. Sergio Rabellino
Università degli Studi di Torino Dipartimento di Informatica Tecnico di Ricerca Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 C.so Svizzera , 185 - 10149 - Torino
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- ing. Sergio Rabellino
Università degli Studi di Torino Dipartimento di Informatica Tecnico di Ricerca Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 C.so Svizzera , 185 - 10149 - Torino
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- ing. Sergio Rabellino Università degli Studi di Torino Dipartimento di Informatica Tecnico di Ricerca Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 C.so Svizzera , 185 - 10149 - Torino <http://www.di.unito.it>
participants (5)
-
Anthony D'Atri
-
Eugen Block
-
Jan Johansson
-
Sergio Rabellino
-
Sinan Polat