Some questions about cephadm
Hi, I have some questions about ceph using cephadm. I used to deploy ceph using ceph-ansible, now I have to move to cephadm, I am in my learning journey. - How can I tell my cluster that it's a part of an HCI deployment? With ceph-ansible it was easy using is_hci : yes - The documentation of ceph does not indicate what versions of grafana, prometheus, ...etc should be used with a certain version. - I am trying to deploy Quincy, I did a bootstrap to see what containers were downloaded and their version. - I am asking because I need to use a local registry to deploy those images. - After the bootstrap, the Web interface was accessible : - How can I access the wizard page again? If I don't use it the first time I could not find another way to get it. - I had a problem with telemetry, I did not configure telemetry, then when I clicked the button, the web gui became inaccessible.....????!!! Regards.
Cephadm does not have some variable that explicitly says it's an HCI deployment. However, the HCI variable in ceph ansible I believe only controlled the osd_memory_target attribute, which would automatically set it to 20% or 70% respectively of the memory on the node divided by the number of OSDs on the node depending on whether it's HCI or not. Cephadm doesn't have that exactly, but has a similar feature of osd memory autotuning which has some docs here https://docs.ceph.com/en/latest/cephadm/services/osd/#automatically-tuning-o.... The warning indicates that it isn't ideal for HCI to use this, but I think if you set the mgr/cephadm/autotune_memory_target_ratio to a value closer to 0.2 rather than the default 0.7, it might end up working out close to how ceph-ansible worked with the is_hci option set to true. Otherwise, you can set the option yourself after doing a similar calculation to what ceph-ansible did with something like `ceph config set osd/host:<hostname> osd_memory_target <amount>` where that amount is the per OSD memory target you want for OSDs on that host. I don't think we have documentation of what version to use for the monitoring stack daemons, but the assumption is the default version defined in cephadm should be okay to use unless you have some specific use case that requires a different one. There are docs on how to change it to use a different image if you'd like to do so https://docs.ceph.com/en/latest/cephadm/services/monitoring/#using-custom-im... Can't speak much to the setup wizard in the dashboard and whether it's possible to get it going again after closing out of it, or the telemetry related dashboard issue. On Wed, Feb 21, 2024 at 11:08 AM wodel youchi <wodel.youchi@gmail.com> wrote:
Hi,
I have some questions about ceph using cephadm.
I used to deploy ceph using ceph-ansible, now I have to move to cephadm, I am in my learning journey.
- How can I tell my cluster that it's a part of an HCI deployment? With ceph-ansible it was easy using is_hci : yes - The documentation of ceph does not indicate what versions of grafana, prometheus, ...etc should be used with a certain version. - I am trying to deploy Quincy, I did a bootstrap to see what containers were downloaded and their version. - I am asking because I need to use a local registry to deploy those images. - After the bootstrap, the Web interface was accessible : - How can I access the wizard page again? If I don't use it the first time I could not find another way to get it. - I had a problem with telemetry, I did not configure telemetry, then when I clicked the button, the web gui became inaccessible.....????!!!
Regards. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 21.02.2024 17:07, wodel youchi wrote:
- The documentation of ceph does not indicate what versions of grafana, prometheus, ...etc should be used with a certain version. - I am trying to deploy Quincy, I did a bootstrap to see what containers were downloaded and their version. - I am asking because I need to use a local registry to deploy those images.
You need to check the cephadm source for the version you would like to use https://github.com/ceph/ceph/blob/v17.2.7/src/cephadm/cephadm#L46 -- Kai Stian Olstad
Hi, just responding to the last questions:
- After the bootstrap, the Web interface was accessible : - How can I access the wizard page again? If I don't use it the first time I could not find another way to get it.
I don't know how to recall the wizard, but you should be able to create a new dashboard user with your desired role (e. g. administrator) from the CLI: ceph dashboard ac-user-create <username> [<rolename>] -i <file_with_password>
- I had a problem with telemetry, I did not configure telemetry, then when I clicked the button, the web gui became inaccessible.....????!!!
You can see what happened in the active MGR log. Zitat von wodel youchi <wodel.youchi@gmail.com>:
Hi,
I have some questions about ceph using cephadm.
I used to deploy ceph using ceph-ansible, now I have to move to cephadm, I am in my learning journey.
- How can I tell my cluster that it's a part of an HCI deployment? With ceph-ansible it was easy using is_hci : yes - The documentation of ceph does not indicate what versions of grafana, prometheus, ...etc should be used with a certain version. - I am trying to deploy Quincy, I did a bootstrap to see what containers were downloaded and their version. - I am asking because I need to use a local registry to deploy those images. - After the bootstrap, the Web interface was accessible : - How can I access the wizard page again? If I don't use it the first time I could not find another way to get it. - I had a problem with telemetry, I did not configure telemetry, then when I clicked the button, the web gui became inaccessible.....????!!!
Regards. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thank you all for your help. @Adam From the reading you gave me I have understood the following : 1 - Set osd_memory_target_autotune to true then set autotune_memory_target_ratio to 0.2 2 - Or do the math. For my setup I have 384Go per node, each node has 4 nvme disks of 7.6To, 0.2 of memory is 19.5G. So each OSD will have 19G of memory. Question : Should I take into account the size of the disk when calculating the required memory for an OSD? I have another problem, the local registry. I deployed a local registry with the required images, then I used cephadm-ansible to prepare my hosts and inject the local registry url into /etc/container/registry.conf file Then I tried to deploy using this command on the admin node: cephadm --image 192.168.2.36:4000/ceph/ceph:v17 bootstrap --mon-ip 10.1.0.23 --cluster-network 10.2.0.0/16 After the boot strap I found that it still downloads the images from the internet, even the ceph image itself, I see two images one from my registry the second from quay. There is a section that talks about using a local registry here https://docs.ceph.com/en/reef/cephadm/install/#deployment-in-an-isolated-env..., but it's not clear especially about the other images. It talks about preparing a temporary file named initial-ceph.conf, then it does not use it???!!! Could you help? Regards. Le jeu. 22 févr. 2024 à 11:10, Eugen Block <eblock@nde.ag> a écrit :
Hi,
just responding to the last questions:
- After the bootstrap, the Web interface was accessible : - How can I access the wizard page again? If I don't use it the first time I could not find another way to get it.
I don't know how to recall the wizard, but you should be able to create a new dashboard user with your desired role (e. g. administrator) from the CLI:
ceph dashboard ac-user-create <username> [<rolename>] -i <file_with_password>
- I had a problem with telemetry, I did not configure telemetry,
then
when I clicked the button, the web gui became
inaccessible.....????!!!
You can see what happened in the active MGR log.
Zitat von wodel youchi <wodel.youchi@gmail.com>:
Hi,
I have some questions about ceph using cephadm.
I used to deploy ceph using ceph-ansible, now I have to move to cephadm, I am in my learning journey.
- How can I tell my cluster that it's a part of an HCI deployment? With ceph-ansible it was easy using is_hci : yes - The documentation of ceph does not indicate what versions of grafana, prometheus, ...etc should be used with a certain version. - I am trying to deploy Quincy, I did a bootstrap to see what containers were downloaded and their version. - I am asking because I need to use a local registry to deploy those images. - After the bootstrap, the Web interface was accessible : - How can I access the wizard page again? If I don't use it the first time I could not find another way to get it. - I had a problem with telemetry, I did not configure telemetry, then when I clicked the button, the web gui became inaccessible.....????!!!
Regards. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, On 26.02.24 11:08, wodel youchi wrote:
Then I tried to deploy using this command on the admin node: cephadm --image 192.168.2.36:4000/ceph/ceph:v17 bootstrap --mon-ip 10.1.0.23 --cluster-network 10.2.0.0/16
After the boot strap I found that it still downloads the images from the internet, even the ceph image itself, I see two images one from my registry the second from quay.
To quote the docs: you can run cephadm bootstrap -h to see all of cephadm’s available options. These options are available: --registry-url REGISTRY_URL url for custom registry --registry-username REGISTRY_USERNAME username for custom registry --registry-password REGISTRY_PASSWORD password for custom registry --registry-json REGISTRY_JSON json file with custom registry login info (URL, Username, Password) Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 220009 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin
Hi, No didn't work, the bootstrap is still downloading the images from quay. PS : My local registry does not require any login/pass authentication, I used fake ones since it's mandatory to give them. cephadm --image 192.168.2.36:4000/ceph/ceph:v17 bootstrap --registry-url 192.168.2.36:4000 --registry-username admin --registry-password admin --mon-ip 10.1.0.23 --cluster-network 10.2.0.0/16 [root@controllera ~]# podman images REPOSITORY TAG IMAGE ID CREATED SIZE 192.168.2.36:4000/ceph/ceph v17 56993389bc29 11 days ago 1.29 GB quay.io/ceph/ceph-grafana 9.4.7 954c08fa6188 2 months ago 647 MB quay.io/prometheus/prometheus v2.43.0 a07b618ecd1d 11 months ago 235 MB quay.io/prometheus/alertmanager v0.25.0 c8568f914cd2 14 months ago 66.5 MB quay.io/prometheus/node-exporter v1.5.0 0da6a335fe13 15 months ago 23.9 MB Regards. Le lun. 26 févr. 2024 à 11:42, Robert Sander <r.sander@heinlein-support.de> a écrit :
Hi,
On 26.02.24 11:08, wodel youchi wrote:
Then I tried to deploy using this command on the admin node: cephadm --image 192.168.2.36:4000/ceph/ceph:v17 bootstrap --mon-ip 10.1.0.23 --cluster-network 10.2.0.0/16
After the boot strap I found that it still downloads the images from the internet, even the ceph image itself, I see two images one from my registry the second from quay.
To quote the docs: you can run cephadm bootstrap -h to see all of cephadm’s available options.
These options are available:
--registry-url REGISTRY_URL url for custom registry --registry-username REGISTRY_USERNAME username for custom registry --registry-password REGISTRY_PASSWORD password for custom registry --registry-json REGISTRY_JSON json file with custom registry login info (URL, Username, Password)
Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
http://www.heinlein-support.de
Tel: 030 / 405051-43 Fax: 030 / 405051-19
Zwangsangaben lt. §35a GmbHG: HRB 220009 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, On 2/26/24 13:22, wodel youchi wrote:
No didn't work, the bootstrap is still downloading the images from quay.
For the image locations of the monitoring stack you have to create an initical ceph.conf like it is mentioned in the chapter you referred earlier: https://docs.ceph.com/en/reef/cephadm/install/#deployment-in-an-isolated-env... Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
I've read that, but I didn't find how to use it? should I use the : --config *CONFIG_FILE *options? Le lun. 26 févr. 2024 à 13:59, Robert Sander <r.sander@heinlein-support.de> a écrit :
Hi,
On 2/26/24 13:22, wodel youchi wrote:
No didn't work, the bootstrap is still downloading the images from quay.
For the image locations of the monitoring stack you have to create an initical ceph.conf like it is mentioned in the chapter you referred earlier:
https://docs.ceph.com/en/reef/cephadm/install/#deployment-in-an-isolated-env...
Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: 030 / 405051-43 Fax: 030 / 405051-19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi; So it was that, create the initial-ceph.conf and use the --config Now All images are from the local registry. Thank you all for your help. Regards. Le lun. 26 févr. 2024 à 14:09, wodel youchi <wodel.youchi@gmail.com> a écrit :
I've read that, but I didn't find how to use it? should I use the : --config *CONFIG_FILE *options?
Le lun. 26 févr. 2024 à 13:59, Robert Sander <r.sander@heinlein-support.de> a écrit :
Hi,
On 2/26/24 13:22, wodel youchi wrote:
No didn't work, the bootstrap is still downloading the images from quay.
For the image locations of the monitoring stack you have to create an initical ceph.conf like it is mentioned in the chapter you referred earlier:
https://docs.ceph.com/en/reef/cephadm/install/#deployment-in-an-isolated-env...
Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: 030 / 405051-43 Fax: 030 / 405051-19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I have another problem, the local registry. I deployed a local registry with the required images, then I used cephadm-ansible to prepare my hosts and inject the local registry url into /etc/container/registry.conf file
Then I tried to deploy using this command on the admin node: cephadm --image 192.168.2.36:4000/ceph/ceph:v17 bootstrap --mon-ip 10.1.0.23 --cluster-network 10.2.0.0/16
After the boot strap I found that it still downloads the images from the internet, even the ceph image itself, I see two images one from my registry the second from quay.
There is a section that talks about using a local registry here https://docs.ceph.com/en/reef/cephadm/install/#deployment-in-an-isolated-env ironment,
but it's not clear especially about the other images. It talks
about preparing a temporary file named initial-ceph.conf, then it does not use it???!!!
Could you help?
That's right, the docs are not clear here. I know I had a previous conversation about this but I can't find it nor recall if the conversation happened on the mailing list, slack, or a tracker issue. Regardless, the option you need to pass the initial conf to bootstrap is `-- config/-c`. As per the section you need to customize the image names and then they will be downloaded from the local registry.
In regards to
From the reading you gave me I have understood the following : 1 - Set osd_memory_target_autotune to true then set autotune_memory_target_ratio to 0.2 2 - Or do the math. For my setup I have 384Go per node, each node has 4 nvme disks of 7.6To, 0.2 of memory is 19.5G. So each OSD will have 19G of memory.
Question : Should I take into account the size of the disk when calculating the required memory for an OSD?
The memory in question is RAM, not disk space. To see the exact value cephadm will see for the amount of memory (in kb, we multiply by 1024 when actually using it) when doing this autotuning, you can run [root@vm-00 ~]# cephadm gather-facts | grep memory_total "memory_total_kb": 40802184, on your machine. Then it multiplies that by the ratio and subtracts out an amount for every non-OSD daemon on the node. Specifically (taking this from the code) min_size_by_type = { 'mds': 4096 * 1048576, 'mgr': 4096 * 1048576, 'mon': 1024 * 1048576, 'crash': 128 * 1048576, 'keepalived': 128 * 1048576, 'haproxy': 128 * 1048576, } default_size = 1024 * 1048576 so 1 GB for most daemons, with mgr and mds requiring extra (although for mds it also uses the `mds_cache_memory_limit` config option if it's set) and some others requiring less. What's left after all that is done is then divided by the number of OSDs deployed on the host. If that number ends up too small, however, there is some floor that it won't set below, but I can't remember off the top of my head what that is. Maybe 4 GB. On Mon, Feb 26, 2024 at 5:10 AM wodel youchi <wodel.youchi@gmail.com> wrote:
Thank you all for your help.
@Adam From the reading you gave me I have understood the following : 1 - Set osd_memory_target_autotune to true then set autotune_memory_target_ratio to 0.2 2 - Or do the math. For my setup I have 384Go per node, each node has 4 nvme disks of 7.6To, 0.2 of memory is 19.5G. So each OSD will have 19G of memory.
Question : Should I take into account the size of the disk when calculating the required memory for an OSD?
I have another problem, the local registry. I deployed a local registry with the required images, then I used cephadm-ansible to prepare my hosts and inject the local registry url into /etc/container/registry.conf file
Then I tried to deploy using this command on the admin node: cephadm --image 192.168.2.36:4000/ceph/ceph:v17 bootstrap --mon-ip 10.1.0.23 --cluster-network 10.2.0.0/16
After the boot strap I found that it still downloads the images from the internet, even the ceph image itself, I see two images one from my registry the second from quay.
There is a section that talks about using a local registry here
https://docs.ceph.com/en/reef/cephadm/install/#deployment-in-an-isolated-env... , but it's not clear especially about the other images. It talks about preparing a temporary file named initial-ceph.conf, then it does not use it???!!!
Could you help?
Regards.
Le jeu. 22 févr. 2024 à 11:10, Eugen Block <eblock@nde.ag> a écrit :
Hi,
just responding to the last questions:
- After the bootstrap, the Web interface was accessible : - How can I access the wizard page again? If I don't use it the first time I could not find another way to get it.
I don't know how to recall the wizard, but you should be able to create a new dashboard user with your desired role (e. g. administrator) from the CLI:
ceph dashboard ac-user-create <username> [<rolename>] -i <file_with_password>
- I had a problem with telemetry, I did not configure telemetry,
then
when I clicked the button, the web gui became
inaccessible.....????!!!
You can see what happened in the active MGR log.
Zitat von wodel youchi <wodel.youchi@gmail.com>:
Hi,
I have some questions about ceph using cephadm.
I used to deploy ceph using ceph-ansible, now I have to move to cephadm, I am in my learning journey.
- How can I tell my cluster that it's a part of an HCI deployment? With ceph-ansible it was easy using is_hci : yes - The documentation of ceph does not indicate what versions of grafana, prometheus, ...etc should be used with a certain version. - I am trying to deploy Quincy, I did a bootstrap to see what containers were downloaded and their version. - I am asking because I need to use a local registry to deploy those images. - After the bootstrap, the Web interface was accessible : - How can I access the wizard page again? If I don't use it the first time I could not find another way to get it. - I had a problem with telemetry, I did not configure telemetry, then when I clicked the button, the web gui became inaccessible.....????!!!
Regards. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (6)
-
Adam King
-
Eugen Block
-
John Mulligan
-
Kai Stian Olstad
-
Robert Sander
-
wodel youchi