Cephadm orchestrator and special label _admin in 17.2.7
Hi, According to the documentation¹ the special host label _admin instructs the cephadm orchestrator to place a valid ceph.conf and the ceph.client.admin.keyring into /etc/ceph of the host. I noticed that (at least) on 17.2.7 only the keyring file is placed in /etc/ceph, but not ceph.conf. Both files are placed into the /var/lib/ceph/<fsid>/config directory. Has something changed? ¹: https://docs.ceph.com/en/quincy/cephadm/host-management/#special-host-labels Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
Hi, I just tired that in my test cluster, removed the ceph.conf and admin keyring from /etc/ceph and then added the _admin label to the host via 'ceph orch' and both were created immediately. # no label for quincy-2 quincy-1:~ # ceph orch host ls HOST ADDR LABELS STATUS quincy-1 172.17.2.252 _admin quincy-2 172.17.2.13 2 hosts in cluster # moved existing files quincy-2:~ # mv /etc/ceph/ceph.client.admin.keyring /etc/ceph/xxxadminxxx.keyring quincy-2:~ # mv /etc/ceph/ceph.conf /etc/ceph/conf # added label quincy-1:~ # ceph orch host label add quincy-2 _admin Added label _admin to host quincy-2 # mgr deploys files Jan 18 14:05:52 quincy-2 ceph-mgr[3880499]: log_channel(cephadm) log [INF] : Added label _admin to host quincy-2 Jan 18 14:05:52 quincy-2 ceph-mgr[3880499]: log_channel(cephadm) log [INF] : Updating quincy-2:/etc/ceph/ceph.conf Jan 18 14:05:53 quincy-2 ceph-mgr[3880499]: log_channel(cephadm) log [INF] : Updating quincy-2:/var/lib/ceph/1e6e5cb6-73e8-11ee-b195-fa163ee43e22/config/ceph.conf Jan 18 14:05:54 quincy-2 ceph-mgr[3880499]: log_channel(cephadm) log [INF] : Updating quincy-2:/etc/ceph/ceph.client.admin.keyring Jan 18 14:05:54 quincy-2 ceph-mgr[3880499]: log_channel(cephadm) log [INF] : Updating quincy-2:/var/lib/ceph/1e6e5cb6-73e8-11ee-b195-fa163ee43e22/config/ceph.client.admin.keyring quincy-2:~ # ll /etc/ceph/ insgesamt 24 -rw------- 1 root root 151 18. Jan 14:05 ceph.client.admin.keyring -rw-r--r-- 1 root root 175 18. Jan 14:05 ceph.conf -rw-r--r-- 1 root root 595 26. Okt 13:17 ceph.pub -rw-r--r-- 1 root root 175 26. Okt 13:17 conf -rw-r--r-- 1 root root 92 7. Aug 14:32 rbdmap -rw------- 1 root root 151 26. Okt 13:17 xxxadminxxx.keyring This is on 17.2.7: quincy-2:~ # ceph versions { ... "overall": { "ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable)": 10 } } Zitat von Robert Sander <r.sander@heinlein-support.de>:
Hi,
According to the documentation¹ the special host label _admin instructs the cephadm orchestrator to place a valid ceph.conf and the ceph.client.admin.keyring into /etc/ceph of the host.
I noticed that (at least) on 17.2.7 only the keyring file is placed in /etc/ceph, but not ceph.conf.
Both files are placed into the /var/lib/ceph/<fsid>/config directory.
Has something changed?
¹: https://docs.ceph.com/en/quincy/cephadm/host-management/#special-host-labels
Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: 030 / 405051-43 Fax: 030 / 405051-19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, On 1/18/24 14:07, Eugen Block wrote:
I just tired that in my test cluster, removed the ceph.conf and admin keyring from /etc/ceph and then added the _admin label to the host via 'ceph orch' and both were created immediately.
This is strange, I only get this: 2024-01-18T11:47:07.870079+0100 mgr.cephtest32.ybltym [INF] Added label _admin to host cephtest23 2024-01-18T11:47:07.878786+0100 mgr.cephtest32.ybltym [INF] Updating cephtest23:/var/lib/ceph/ba37db20-2b13-11eb-b8a9-871ba11409f6/config/ceph.conf 2024-01-18T11:47:08.045347+0100 mgr.cephtest32.ybltym [INF] Updating cephtest23:/etc/ceph/ceph.client.admin.keyring 2024-01-18T11:47:08.212303+0100 mgr.cephtest32.ybltym [INF] Updating cephtest23:/var/lib/ceph/ba37db20-2b13-11eb-b8a9-871ba11409f6/config/ceph.client.admin.keyring Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
Is your admin keyring under management? The docs say [1]:
When a client keyring is placed under management, cephadm will: build a list of target hosts based on the specified placement spec (see Daemon Placement) store a copy of the /etc/ceph/ceph.conf file on the specified host(s)
In my case the keyring is under management: quincy-2:~ # ceph orch client-keyring ls ENTITY PLACEMENT MODE OWNER PATH client.admin label:_admin rw------- 0:0 /etc/ceph/ceph.client.admin.keyring [1] https://docs.ceph.com/en/quincy/cephadm/operations/#etc-ceph-ceph-conf Zitat von Robert Sander <r.sander@heinlein-support.de>:
Hi,
On 1/18/24 14:07, Eugen Block wrote:
I just tired that in my test cluster, removed the ceph.conf and admin keyring from /etc/ceph and then added the _admin label to the host via 'ceph orch' and both were created immediately.
This is strange, I only get this:
2024-01-18T11:47:07.870079+0100 mgr.cephtest32.ybltym [INF] Added label _admin to host cephtest23 2024-01-18T11:47:07.878786+0100 mgr.cephtest32.ybltym [INF] Updating cephtest23:/var/lib/ceph/ba37db20-2b13-11eb-b8a9-871ba11409f6/config/ceph.conf 2024-01-18T11:47:08.045347+0100 mgr.cephtest32.ybltym [INF] Updating cephtest23:/etc/ceph/ceph.client.admin.keyring 2024-01-18T11:47:08.212303+0100 mgr.cephtest32.ybltym [INF] Updating cephtest23:/var/lib/ceph/ba37db20-2b13-11eb-b8a9-871ba11409f6/config/ceph.client.admin.keyring
Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: 030 / 405051-43 Fax: 030 / 405051-19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 1/18/24 14:28, Eugen Block wrote:
Is your admin keyring under management?
There is no issue with the admin keyring but with ceph.conf. The config setting "mgr mgr/cephadm/manage_etc_ceph_ceph_conf" is set to true and "mgr mgr/cephadm/manage_etc_ceph_ceph_conf_hosts" was at "*", so all hosts. I have set that to "label:_admin". It still does not put ceph.conf into /etc/ceph when adding the label _admin. Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
Hi, more strang behaviour: When I isssue "ceph mgr fail" a backup MGR takes over and updates all config files on all hosts including /etc/ceph/ceph.conf. At first I thought that this was the solution but when I now remove the _admin label and add it again the new MGR also does not update /etc/ceph/ceph.conf. Only when I again do "ceph mgr fail" the new MGR will update /etc/ceph/ceph.conf on the hosts labeled with _admin. Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
Oh that does sound strange indeed. I don't have a good idea right now, hopefully someone from the dev team can shed some light on this. Zitat von Robert Sander <r.sander@heinlein-support.de>:
Hi,
more strang behaviour:
When I isssue "ceph mgr fail" a backup MGR takes over and updates all config files on all hosts including /etc/ceph/ceph.conf.
At first I thought that this was the solution but when I now remove the _admin label and add it again the new MGR also does not update /etc/ceph/ceph.conf.
Only when I again do "ceph mgr fail" the new MGR will update /etc/ceph/ceph.conf on the hosts labeled with _admin.
Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin
https://www.heinlein-support.de
Tel: 030 / 405051-43 Fax: 030 / 405051-19
Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Le 18/01/2024 à 11:58:47+0100, Robert Sander a écrit Hi,
According to the documentation¹ the special host label _admin instructs the cephadm orchestrator to place a valid ceph.conf and the ceph.client.admin.keyring into /etc/ceph of the host.
I noticed that (at least) on 17.2.7 only the keyring file is placed in /etc/ceph, but not ceph.conf.
Both files are placed into the /var/lib/ceph/<fsid>/config directory.
Has something changed?
Just like to known if it's a very bad idea to do a rsync of /etc/ceph from the «_admin» server to the other ceph cluster server. I in fact add something like for host in `cat /usr/local/etc/ceph_list_noeuds.txt` do /usr/bin/rsync -av /etc/ceph/ceph* $host:/etc/ceph/ done in a cronjob Regards. -- Albert SHIH 🦫 🐸 France Heure locale/Local time: mar. 23 janv. 2024 18:18:39 CET
On 23.01.2024 18:19, Albert Shih wrote:
Just like to known if it's a very bad idea to do a rsync of /etc/ceph from the «_admin» server to the other ceph cluster server.
I in fact add something like
for host in `cat /usr/local/etc/ceph_list_noeuds.txt` do /usr/bin/rsync -av /etc/ceph/ceph* $host:/etc/ceph/ done
in a cronjob
Why not just add the _admin label to the host and let Ceph do the job? You can also run this to get the ceph.conf copied to all host ceph config set mgr/cephadm/manage_etc_ceph_ceph_conf true Anyway, I don't se any problem with rsync it, it's just ceph.conf and the admin key. -- Kai Stian Olstad
participants (4)
-
Albert Shih
-
Eugen Block
-
Kai Stian Olstad
-
Robert Sander