ceph.conf on every node
Hi, Just like to know if it's a bad idea (like something going to bite me in few year) to have a copy of the /etc/ceph/ceph.conf from the master copy on every node of the cluster (install with cephadm). I «need» the ceph.conf to able to get the fsid on every node. Regards. -- Albert SHIH 🦫 🐸 France Heure locale/Local time: mar. 18 nov. 2025 10:40:25 CET
# ceph fsid 44928f74-thx-1138-8862-d96497f06d07 Or you could define a custom facter fact... Just sayin... ... that said, before Mimic brought us the central config db, maintaining <clustername>.conf across clusters was a pain. When testing or troubleshooting one would need to persist changes across all nodes, and in the heat of an escalation it was all too easy to forget to persist changes. Even with the file templated in automation, consider what happens when an important change happens while one or more nodes is down ... then it comes back up. -aad
On Nov 18, 2025, at 4:41 AM, Albert Shih <Albert.Shih@obspm.fr> wrote:
Hi,
Just like to know if it's a bad idea (like something going to bite me in few year) to have a copy of the /etc/ceph/ceph.conf from the master copy on every node of the cluster (install with cephadm).
I «need» the ceph.conf to able to get the fsid on every node.
Regards.
-- Albert SHIH 🦫 🐸 France Heure locale/Local time: mar. 18 nov. 2025 10:40:25 CET _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Le 18/11/2025 à 07:53:48-0500, Anthony D'Atri a écrit Hi,
# ceph fsid 44928f74-thx-1138-8862-d96497f06d07
Well, if I don't have the file the commande don't work root@ceph:~# ceph fsid Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)') root@ceph:~#
Or you could define a custom facter fact...
It's indeed what I'm doing with my puppet, but I need to known how to get that info. Currently I have the facter running to get the fsid so I can use it in my puppet module, but the facter doesn't return anything when I don't have the file. That's why I'm thinking to create a cron with rsync /etc/ceph/ceph.conf other_nodes:/etc/ceph on the master node.
Just sayin...
... that said, before Mimic brought us the central config db, maintaining <clustername>.conf across clusters was a pain. When testing or troubleshooting one would need to persist changes across all nodes, and in the heat of an escalation it was all too easy to forget to persist changes. Even with the file templated in automation, consider what happens when an important change happens while one or more nodes is down ... then it comes back up.
No doubt but my question is what happens if I put the ceph.conf, can be any unexpected side effect. Regards.
On Nov 18, 2025, at 4:41 AM, Albert Shih <Albert.Shih@obspm.fr> wrote:
Hi,
Just like to know if it's a bad idea (like something going to bite me in few year) to have a copy of the /etc/ceph/ceph.conf from the master copy on every node of the cluster (install with cephadm).
I «need» the ceph.conf to able to get the fsid on every node.
-- Albert SHIH 🦫 🐸 France Heure locale/Local time: mar. 18 nov. 2025 14:02:38 CET
On Nov 18, 2025, at 8:12 AM, Albert Shih <Albert.Shih@obspm.fr> wrote:
Le 18/11/2025 à 07:53:48-0500, Anthony D'Atri a écrit Hi,
# ceph fsid 44928f74-thx-1138-8862-d96497f06d07
Well, if I don't have the file the commande don't work
root@ceph:~# ceph fsid Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)') root@ceph:~#
You do need the conf file to point to the mons.
Or you could define a custom facter fact...
It's indeed what I'm doing with my puppet, but I need to known how to get that info.
Currently I have the facter running to get the fsid so I can use it in my puppet module, but the facter doesn't return anything when I don't have the file. That's why I'm thinking to create a cron with
rsync /etc/ceph/ceph.conf other_nodes:/etc/ceph
on the master node.
Just sayin...
... that said, before Mimic brought us the central config db, maintaining <clustername>.conf across clusters was a pain. When testing or troubleshooting one would need to persist changes across all nodes, and in the heat of an escalation it was all too easy to forget to persist changes. Even with the file templated in automation, consider what happens when an important change happens while one or more nodes is down ... then it comes back up.
No doubt but my question is what happens if I put the ceph.conf, can be any unexpected side effect.
Regards.
On Nov 18, 2025, at 4:41 AM, Albert Shih <Albert.Shih@obspm.fr> wrote:
Hi,
Just like to know if it's a bad idea (like something going to bite me in few year) to have a copy of the /etc/ceph/ceph.conf from the master copy on every node of the cluster (install with cephadm).
I «need» the ceph.conf to able to get the fsid on every node.
-- Albert SHIH 🦫 🐸 France Heure locale/Local time: mar. 18 nov. 2025 14:02:38 CET _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Tue, 18 Nov 2025 at 07:25, Anthony D'Atri <anthony.datri@gmail.com> wrote:
On Nov 18, 2025, at 8:12 AM, Albert Shih <Albert.Shih@obspm.fr> wrote: Well, if I don't have the file the commande don't work
root@ceph:~# ceph fsid Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)') root@ceph:~#
You do need the conf file to point to the mons.
Isn't DNS SRV meant to remove this dependency? or if it doesn't could SRV be made to do it? thanks.
... that said, before Mimic brought us the central config db, maintaining <clustername>.conf across clusters was a pain. When testing or troubleshooting one would need to persist changes across all nodes, and in the heat of an escalation it was all too easy to forget to persist changes. Even with the file templated in automation, consider what happens when an important change happens while one or more nodes is down ... then it comes back up.
No doubt but my question is what happens if I put the ceph.conf, can be any unexpected side effect.
ceph.conf keeps data in sections, so all the global stuff gets read by all daemons and clients, then you can have OSD specifics and down to instance-specific data for osd.123 in a section of its own if need be, so it is certainly made so one file could work for all of the parts of a ceph cluster. But as said, now with the ceph config db, you only need a super short conf with the ips of the mons and possibly the fsid and it will pick up the rest from the config db itself. -- May the most significant bit of your life be positive.
Not sure why you need the ceph.conf in a cephadm installation (as most of the config is done through the config db). Anyway, If you are using a (modern) cephadm installation then you can just put the label "*_admin*" on the hosts where you would like to have the config and the ceph.conf and keyring will be copied to */var/lib/ceph/<fsid>/config/* When you start a cephadm shell normally it first checks if there's any running monitor, if found then it will use its config file, otherwise it will check for the above directory to get the fsid. Regards, Redo. On Tue, Nov 18, 2025 at 2:28 PM Janne Johansson <icepic.dz@gmail.com> wrote:
... that said, before Mimic brought us the central config db, maintaining <clustername>.conf across clusters was a pain. When testing or troubleshooting one would need to persist changes across all nodes, and in the heat of an escalation it was all too easy to forget to persist changes. Even with the file templated in automation, consider what happens when an important change happens while one or more nodes is down ... then it comes back up.
No doubt but my question is what happens if I put the ceph.conf, can be any unexpected side effect.
ceph.conf keeps data in sections, so all the global stuff gets read by all daemons and clients, then you can have OSD specifics and down to instance-specific data for osd.123 in a section of its own if need be, so it is certainly made so one file could work for all of the parts of a ceph cluster. But as said, now with the ceph config db, you only need a super short conf with the ips of the mons and possibly the fsid and it will pick up the rest from the config db itself.
-- May the most significant bit of your life be positive. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Le 18/11/2025 à 14:35:54+0100, Redouane Kachach a écrit Hi,
Not sure why you need the ceph.conf in a cephadm installation (as most of the config is done through the config db). Anyway, If you are using a (modern) cephadm installation then you can just put the label "*_admin*" on the hosts where you would like to have the config and the ceph.conf and keyring will be copied to */var/lib/ceph/<fsid>/config/*
When you start a cephadm shell normally it first checks if there's any running monitor, if found then it will use its config file, otherwise it will check for the above directory to get the fsid.
I don't need the ceph.conf for ceph himself. I «need» the ceph.conf so I can launch ceph fsid on any node. As why I need to launch ceph fsid on any node, it's because I'm using puppet to manage everything, and currently I need to add in the cron table of every node something like chown -R ceph:ceph /var/lib/ceph/FSID/crash/posted so with the hability to launch ceph fsid on any node I can get that info through a «facter» in puppet so I can add this line in the crontab. Of course I can manage the fsid by putting it in some variable, but that's less portable and less «elegant». Regards. -- Albert SHIH 🦫 🐸 France Heure locale/Local time: mer. 19 nov. 2025 13:40:18 CET
On Nov 19, 2025, at 7:45 AM, Albert Shih <Albert.Shih@obspm.fr> wrote:
Le 18/11/2025 à 14:35:54+0100, Redouane Kachach a écrit Hi,
Not sure why you need the ceph.conf in a cephadm installation (as most of the config is done through the config db). Anyway, If you are using a (modern) cephadm installation then you can just put the label "*_admin*" on the hosts where you would like to have the config and the ceph.conf and keyring will be copied to */var/lib/ceph/<fsid>/config/*
When you start a cephadm shell normally it first checks if there's any running monitor, if found then it will use its config file, otherwise it will check for the above directory to get the fsid.
I don't need the ceph.conf for ceph himself. I «need» the ceph.conf so I can launch
ceph fsid
on any node.
I suspect that the requirement for this to be in ceph.conf may be an artifact of the days when vanity cluster names were intended for multiple logical clusters to run on the same node. One might assert today that a client should be able to get this from the mons, I would suggest entering a tracker ticket to that effect.
As why I need to launch ceph fsid on any node, it's because I'm using puppet to manage everything, and currently I need to add in the cron table of every node something like
chown -R ceph:ceph /var/lib/ceph/FSID/crash/posted
so with the hability to launch ceph fsid on any node I can get that info through a «facter» in puppet so I can add this line in the crontab.
I have to ask: why not use a wildcard?
Of course I can manage the fsid by putting it in some variable, but that's less portable and less «elegant».
Regards.
-- Albert SHIH 🦫 🐸 France Heure locale/Local time: mer. 19 nov. 2025 13:40:18 CET _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Le 19/11/2025 à 09:39:00-0500, Anthony D'Atri a écrit
Hi,
I don't need the ceph.conf for ceph himself. I «need» the ceph.conf so I can launch
ceph fsid
on any node.
I suspect that the requirement for this to be in ceph.conf may be an artifact of the days when vanity cluster names were intended for multiple logical clusters to run on the same node. One might assert today that a client should be able to get this from the mons, I would suggest entering a tracker ticket to that effect.
Ok, but just to be clear I got only [global] fsid = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX mon_host = [v2:a.b.c.10:3300/0,v1:a.b.c.10:6789/0] [v2:a.b.c.11:3300/0,v1:a.b.c.11:6789/0] [v2:a.b.c.12:3300/0,v1:a.b.c.12:6789/0] [v2:a.b.c. 13:3300/0,v1:a.b.c.13:6789/0] [v2:a.b.c.14:3300/0,v1:a.b.c.14:6789/0] [v2:a.b.c.15:3300/0,v1:a.b.c.15:6789/0] [v2:a.b.c.16:3300/0,v1:a.b.c.16:6789/0] [v2:a.b.c.17:3300/0,v1:a.b.c.17:6789/0] [v2:a.b.c.18:3300/0,v1:a.b.c.18:6789/0]
chown -R ceph:ceph /var/lib/ceph/FSID/crash/posted
so with the hability to launch ceph fsid on any node I can get that info through a «facter» in puppet so I can add this line in the crontab.
I have to ask: why not use a wildcard?
Because this is just one example of the using of fsid in puppet. But you'r right I can use a wildcard. Regards -- Albert SHIH 🦫 🐸 France Heure locale/Local time: mer. 19 nov. 2025 15:54:44 CET
Indeed. mon_host tells a client how to find the mons, with the central config DB that's all that really should be needed.
On Nov 19, 2025, at 10:00 AM, Albert Shih <Albert.Shih@obspm.fr> wrote:
Le 19/11/2025 à 09:39:00-0500, Anthony D'Atri a écrit
Hi,
I don't need the ceph.conf for ceph himself. I «need» the ceph.conf so I can launch
ceph fsid
on any node.
I suspect that the requirement for this to be in ceph.conf may be an artifact of the days when vanity cluster names were intended for multiple logical clusters to run on the same node. One might assert today that a client should be able to get this from the mons, I would suggest entering a tracker ticket to that effect.
Ok, but just to be clear I got only
[global] fsid = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX mon_host = [v2:a.b.c.10:3300/0,v1:a.b.c.10:6789/0] [v2:a.b.c.11:3300/0,v1:a.b.c.11:6789/0] [v2:a.b.c.12:3300/0,v1:a.b.c.12:6789/0] [v2:a.b.c. 13:3300/0,v1:a.b.c.13:6789/0] [v2:a.b.c.14:3300/0,v1:a.b.c.14:6789/0] [v2:a.b.c.15:3300/0,v1:a.b.c.15:6789/0] [v2:a.b.c.16:3300/0,v1:a.b.c.16:6789/0] [v2:a.b.c.17:3300/0,v1:a.b.c.17:6789/0] [v2:a.b.c.18:3300/0,v1:a.b.c.18:6789/0]
chown -R ceph:ceph /var/lib/ceph/FSID/crash/posted
so with the hability to launch ceph fsid on any node I can get that info through a «facter» in puppet so I can add this line in the crontab.
I have to ask: why not use a wildcard?
Because this is just one example of the using of fsid in puppet. But you'r right I can use a wildcard.
Regards
-- Albert SHIH 🦫 🐸 France Heure locale/Local time: mer. 19 nov. 2025 15:54:44 CET _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (6)
-
Albert Shih
-
Anthony D'Atri
-
Anthony D'Atri
-
Janne Johansson
-
Nigel Williams
-
Redouane Kachach