Cephadm multiple public networks
Hi all, I've installed latest Pacific version 16.2.1 using Cephadm. I try using multiple public networks with this setting: ceph config set mon public_network "100.90.1.0/24,100.90.2.0/24" The networks seem to be successfully passed to /etc/ceph/ceph.conf on the daemons, however I constantly see the following messages in the log: 4/26/21 11:40:05 PM[INF]Filtered out host mon-1: could not verify host allowed virtual ips 4/26/21 11:40:05 PM[INF]Filtered out host mon-2: could not verify host allowed virtual ips 4/26/21 11:40:05 PM[INF]Filtered out host mon-3: could not verify host allowed virtual ips As the result, ceph orch doesn't deploy any monitor. Eventually I was able to deploy monitors by first setting public_network to 100.90.1.0/24, then deploy monitors in this subnet, then setting public_network to 100.90.2.0/24, deploy the rest of monitors. Is this some kind of bug or I miss something? Thanks for the replies in advance.
Hello! I'm working with Openstack Wallaby (1 controller, 2 compute nodes) connected to Ceph Pacific cluster in a devel environment. With Openstack Victoria and Ceph Pacific (before last friday update) everything was running like a charm. Then, I upgraded Openstack to Wallaby and Ceph to version 16.2.1. (Because of auth_allow_insecure_global_id_reclaim I had to upgrade many clients... but that's another story...) After upgrade, when I try to create a volume from image, openstack volume create --image f1df058d-be99-4401-82d9-4af9410744bc debian10_volume1 --size 5 with "show_image_direct_url = True", I get "No valid backend" in /var/log/cinder/cinder-scheduler.log 2021-04-26 20:35:24.957 41348 ERROR cinder.scheduler.flows.create_volume [req-651937e5-148f-409c-8296-33f200892e48 c048e887df994f9cb978554008556546 f02ae99c34cf44fd8ab3b1fd1b3be964 - - -] Failed to run task cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create: No valid backend was found. Exceeded max scheduling attempts 3 for resource 56fbb645-2c34-477d-9a59-beec78f4fd3f: cinder.exception.NoValidBackend: No valid backend was found. Exceeded max scheduling attempts 3 for resource 56fbb645-2c34-477d-9a59-beec78f4fd3f and 2021-04-26 20:35:24.968 41347 ERROR oslo_messaging.rpc.server [req-651937e5-148f-409c-8296-33f200892e48 c048e887df994f9cb978554008556546 f02ae99c34cf44fd8ab3b1fd 1b3be964 - - -] Exception during message handling: rbd.InvalidArgument: [errno 22] RBD invalid argument (error creating clone) in /var/log/cinder/cinder-volume.log If I disable "show_image_direct_url = False", volume creation from image works fine. I have spent the last four days googling and reading lots of docs, old and new ones, unlucly... Does anybody have a clue, (please)? Thanks in advance! Javier.-
Hi, the glance option "show_image_direct_url" has been marked as deprecated for quite some time because it's a security issue, but without it the interaction between glance and ceph didn't work very well, I can't quite remember what the side effects were. It seems that they now actually tried to get rid of it, and it seems to work for you if you set it to false, right? Do you see any other side effects when you set it to false? Regards, Eugen Zitat von "Tecnología CHARNE.NET" <tecno@charne.net>:
Hello!
I'm working with Openstack Wallaby (1 controller, 2 compute nodes) connected to Ceph Pacific cluster in a devel environment.
With Openstack Victoria and Ceph Pacific (before last friday update) everything was running like a charm.
Then, I upgraded Openstack to Wallaby and Ceph to version 16.2.1. (Because of auth_allow_insecure_global_id_reclaim I had to upgrade many clients... but that's another story...)
After upgrade, when I try to create a volume from image,
openstack volume create --image f1df058d-be99-4401-82d9-4af9410744bc debian10_volume1 --size 5
with "show_image_direct_url = True", I get "No valid backend" in /var/log/cinder/cinder-scheduler.log
2021-04-26 20:35:24.957 41348 ERROR cinder.scheduler.flows.create_volume [req-651937e5-148f-409c-8296-33f200892e48 c048e887df994f9cb978554008556546 f02ae99c34cf44fd8ab3b1fd1b3be964 - - -] Failed to run task cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create: No valid backend was found. Exceeded max scheduling attempts 3 for resource 56fbb645-2c34-477d-9a59-beec78f4fd3f: cinder.exception.NoValidBackend: No valid backend was found. Exceeded max scheduling attempts 3 for resource 56fbb645-2c34-477d-9a59-beec78f4fd3f
and
2021-04-26 20:35:24.968 41347 ERROR oslo_messaging.rpc.server [req-651937e5-148f-409c-8296-33f200892e48 c048e887df994f9cb978554008556546 f02ae99c34cf44fd8ab3b1fd 1b3be964 - - -] Exception during message handling: rbd.InvalidArgument: [errno 22] RBD invalid argument (error creating clone)
in /var/log/cinder/cinder-volume.log
If I disable "show_image_direct_url = False", volume creation from image works fine.
I have spent the last four days googling and reading lots of docs, old and new ones, unlucly...
Does anybody have a clue, (please)?
Thanks in advance!
Javier.- _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks, Eugen, for your quick answer! Yes, If I set "show_image_direct_url" to false, creation of volumes from images works fine. But creation takes much more time, because of data movements out-and-in ceph cluster, instead snap and copy-on-write approach. All documentation recommends "show_image_direct_url" setted to true with Ceph storage, even if it exposes image locations. Thanks again! El 28/4/21 a las 03:41, Eugen Block escribió:
Hi,
the glance option "show_image_direct_url" has been marked as deprecated for quite some time because it's a security issue, but without it the interaction between glance and ceph didn't work very well, I can't quite remember what the side effects were. It seems that they now actually tried to get rid of it, and it seems to work for you if you set it to false, right? Do you see any other side effects when you set it to false?
Regards, Eugen
Zitat von "Tecnología CHARNE.NET" <tecno@charne.net>:
Hello!
I'm working with Openstack Wallaby (1 controller, 2 compute nodes) connected to Ceph Pacific cluster in a devel environment.
With Openstack Victoria and Ceph Pacific (before last friday update) everything was running like a charm.
Then, I upgraded Openstack to Wallaby and Ceph to version 16.2.1. (Because of auth_allow_insecure_global_id_reclaim I had to upgrade many clients... but that's another story...)
After upgrade, when I try to create a volume from image,
openstack volume create --image f1df058d-be99-4401-82d9-4af9410744bc debian10_volume1 --size 5
with "show_image_direct_url = True", I get "No valid backend" in /var/log/cinder/cinder-scheduler.log
2021-04-26 20:35:24.957 41348 ERROR cinder.scheduler.flows.create_volume [req-651937e5-148f-409c-8296-33f200892e48 c048e887df994f9cb978554008556546 f02ae99c34cf44fd8ab3b1fd1b3be964 - - -] Failed to run task cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create: No valid backend was found. Exceeded max scheduling attempts 3 for resource 56fbb645-2c34-477d-9a59-beec78f4fd3f: cinder.exception.NoValidBackend: No valid backend was found. Exceeded max scheduling attempts 3 for resource 56fbb645-2c34-477d-9a59-beec78f4fd3f
and
2021-04-26 20:35:24.968 41347 ERROR oslo_messaging.rpc.server [req-651937e5-148f-409c-8296-33f200892e48 c048e887df994f9cb978554008556546 f02ae99c34cf44fd8ab3b1fd 1b3be964 - - -] Exception during message handling: rbd.InvalidArgument: [errno 22] RBD invalid argument (error creating clone)
in /var/log/cinder/cinder-volume.log
If I disable "show_image_direct_url = False", volume creation from image works fine.
I have spent the last four days googling and reading lots of docs, old and new ones, unlucly...
Does anybody have a clue, (please)?
Thanks in advance!
Javier.- _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi,
Yes, If I set "show_image_direct_url" to false, creation of volumes from images works fine. But creation takes much more time, because of data movements out-and-in ceph cluster, instead snap and copy-on-write approach.
All documentation recommends "show_image_direct_url" setted to true with Ceph storage, even if it exposes image locations.
hm, that's not good, I think this should be addressed again in the openstack community. It sounds like it's still not properly resolved. Zitat von "Tecnologia Charne.Net" <tecno@charne.net>:
Thanks, Eugen, for your quick answer!
Yes, If I set "show_image_direct_url" to false, creation of volumes from images works fine. But creation takes much more time, because of data movements out-and-in ceph cluster, instead snap and copy-on-write approach.
All documentation recommends "show_image_direct_url" setted to true with Ceph storage, even if it exposes image locations.
Thanks again!
El 28/4/21 a las 03:41, Eugen Block escribió:
Hi,
the glance option "show_image_direct_url" has been marked as deprecated for quite some time because it's a security issue, but without it the interaction between glance and ceph didn't work very well, I can't quite remember what the side effects were. It seems that they now actually tried to get rid of it, and it seems to work for you if you set it to false, right? Do you see any other side effects when you set it to false?
Regards, Eugen
Zitat von "Tecnología CHARNE.NET" <tecno@charne.net>:
Hello!
I'm working with Openstack Wallaby (1 controller, 2 compute nodes) connected to Ceph Pacific cluster in a devel environment.
With Openstack Victoria and Ceph Pacific (before last friday update) everything was running like a charm.
Then, I upgraded Openstack to Wallaby and Ceph to version 16.2.1. (Because of auth_allow_insecure_global_id_reclaim I had to upgrade many clients... but that's another story...)
After upgrade, when I try to create a volume from image,
openstack volume create --image f1df058d-be99-4401-82d9-4af9410744bc debian10_volume1 --size 5
with "show_image_direct_url = True", I get "No valid backend" in /var/log/cinder/cinder-scheduler.log
2021-04-26 20:35:24.957 41348 ERROR cinder.scheduler.flows.create_volume [req-651937e5-148f-409c-8296-33f200892e48 c048e887df994f9cb978554008556546 f02ae99c34cf44fd8ab3b1fd1b3be964 - - -] Failed to run task cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create: No valid backend was found. Exceeded max scheduling attempts 3 for resource 56fbb645-2c34-477d-9a59-beec78f4fd3f: cinder.exception.NoValidBackend: No valid backend was found. Exceeded max scheduling attempts 3 for resource 56fbb645-2c34-477d-9a59-beec78f4fd3f
and
2021-04-26 20:35:24.968 41347 ERROR oslo_messaging.rpc.server [req-651937e5-148f-409c-8296-33f200892e48 c048e887df994f9cb978554008556546 f02ae99c34cf44fd8ab3b1fd 1b3be964 - - -] Exception during message handling: rbd.InvalidArgument: [errno 22] RBD invalid argument (error creating clone)
in /var/log/cinder/cinder-volume.log
If I disable "show_image_direct_url = False", volume creation from image works fine.
I have spent the last four days googling and reading lots of docs, old and new ones, unlucly...
Does anybody have a clue, (please)?
Thanks in advance!
Javier.- _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Update on this issue: If multiple public networks are indeed allowed (and I saw some docs mentioning they are), then it seems to be a bug in src/pybind/mgr/cephadm/serve.py _apply_service() method. It just takes whatever is stored in "public_network" config variable and tries to match it against discovered network of the monitor host. It obviously fails as the "public_network" is comma-separated list. Waiting for my bugtracker account to be approved to create a bugreport+PR, seems to be an easy fix. On 26/04/2021 23:45, Stanislav Datskevych wrote:
Hi all,
I've installed latest Pacific version 16.2.1 using Cephadm. I try using multiple public networks with this setting:
ceph config set mon public_network "100.90.1.0/24,100.90.2.0/24"
The networks seem to be successfully passed to /etc/ceph/ceph.conf on the daemons, however I constantly see the following messages in the log:
4/26/21 11:40:05 PM[INF]Filtered out host mon-1: could not verify host allowed virtual ips 4/26/21 11:40:05 PM[INF]Filtered out host mon-2: could not verify host allowed virtual ips 4/26/21 11:40:05 PM[INF]Filtered out host mon-3: could not verify host allowed virtual ips
As the result, ceph orch doesn't deploy any monitor. Eventually I was able to deploy monitors by first setting public_network to 100.90.1.0/24, then deploy monitors in this subnet, then setting public_network to 100.90.2.0/24, deploy the rest of monitors.
Is this some kind of bug or I miss something?
Thanks for the replies in advance.
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (4)
-
Eugen Block
-
Stanislav Datskevych
-
Tecnologia Charne.Net
-
Tecnología CHARNE.NET