It's not necessarily a ceph mount that shows older features, but the kernel version might still be relevant. Just a quick example, a fresh 19.2.1 install on a virtual machine with kernel: soc9-ceph:~ # uname -r 6.4.0-150600.23.33-default soc9-ceph:~ # ceph versions { "mon": { "ceph version 19.2.1 (58a7fab8be0a062d730ad7da874972fd3fba59fb) squid (stable)": 1 }, "mgr": { "ceph version 19.2.1 (58a7fab8be0a062d730ad7da874972fd3fba59fb) squid (stable)": 2 }, "overall": { "ceph version 19.2.1 (58a7fab8be0a062d730ad7da874972fd3fba59fb) squid (stable)": 3 } } Within the cephadm shell, I have only squid features: [ceph: root@soc9-ceph /]# ceph features { "mon": [ { "features": "0x3f03cffffffdffff", "release": "squid", "num": 1 } ], "client": [ { "features": "0x3f03cffffffdffff", "release": "squid", "num": 4 } ], "mgr": [ { "features": "0x3f03cffffffdffff", "release": "squid", "num": 2 } ] } But if I run the command outside of cephadm shell, I have a Reef feature: soc9-ceph:~ # ceph features { "mon": [ { "features": "0x3f03cffffffdffff", "release": "squid", "num": 1 } ], "client": [ { "features": "0x3f01cfbdfffdffff", "release": "reef", "num": 1 }, ... The locally installed rpm version is Pacific, though: soc9-ceph:~ # ceph --version ceph version 16.2.13-66-g54799ee0666 (54799ee06669271880ee5fc715f99202002aa371) pacific (stable) If you enforced min-compat-client to reef (ceph osd set-require-min-compat-client reef --yes-i-really-mean-it), you would not be able to use Ceph commands from the regular shell of your system(s), but you would be required to always use the cephadm shell. Zitat von quaglio@bol.com.br:
I added 38 disks.
In short, even though I didn't specify the number of disks, what I want to say is that it's a clean installation.
The OSD containers are also from the squid version (I didn't configure anything other than what's in the documentation). I don't understand why you're asking this kind of question.
Could you tell me all the information you need to help me understand the problem with the luminous clients?
-------------------------
DE: "Anthony D'Atri" <anthony.datri@gmail.com> ENVIADA: 2025/04/08 10:44:23 PARA: quaglio@bol.com.br CC: eblock@nde.ag, ceph-users@ceph.io ASSUNTO: Re: [ceph-users] Ceph squid fresh install So, OSD nodes that do not have any OSDs on them yet?
On Apr 8, 2025, at 9:41 AM, quaglio@bol.com.br wrote: More complete description:
1-) I formatted and installed the operating system
2-) This is "ceph installed":
curl --silent --remote-name --location https://download.ceph.com/rpm-19.2.1/el9/noarch/cephadm chmod +x cephadm
./cephadm add-repo --release squid ./cephadm install
cephadm -v bootstrap --mon-ip 172.27.254.6 --cluster-network 172.28.254.0/24 --log-to-file
cephadm install ceph-common -------------------------
DE: "Anthony D'Atri" <anthony.datri@gmail.com> ENVIADA: 2025/04/08 10:35:22 PARA: quaglio@bol.com.br CC: eblock@nde.ag, ceph-users@ceph.io ASSUNTO: Re: [ceph-users] Ceph squid fresh install What does “ceph installed” mean? I suspect that this description is not complete.
On Apr 8, 2025, at 9:21 AM, quaglio@bol.com.br wrote: What is a “storage server”? These are machines that only have the operating system and ceph installed. -------------------------
DE: "Anthony D'Atri" <anthony.datri@gmail.com> ENVIADA: 2025/04/08 10:19:08 PARA: quaglio@bol.com.br CC: eblock@nde.ag, ceph-users@ceph.io ASSUNTO: Re: [ceph-users] Ceph squid fresh install
On Apr 8, 2025, at 9:13 AM, quaglio@bol.com.br wrote:
These 2 IPs are from the storage servers.
What is a “storage server”?
There are no user processes running on them. It only has the operating system and ceph installed.
Nobody said anything about user processes.
Rafael.
De: "Eugen Block" <eblock@nde.ag> Enviada: 2025/04/08 09:35:35 Para: quaglio@bol.com.br Cc: ceph-users@ceph.io Assunto: Re: [ceph-users] Ceph squid fresh install
These are your two Luminous clients:
---snip--- { "name": "unknown.0", "entity_name": "client.admin", "addrs": { "addrvec": [ { "type": "none", "addr": "172.27.254.7:0", "nonce": 443842330 } ] }, "socket_addr": { "type": "none", "addr": "172.27.254.7:0", "nonce": 443842330 }, "con_type": "client", "con_features": 3387146417253690110, "con_features_hex": "2f018fb87aa4aafe", "con_features_release": "luminous", ...
{ "name": "client.104098", "entity_name": "client.admin", "addrs": { "addrvec": [ { "type": "v1", "addr": "172.27.254.6:0", "nonce": 2027668300 } ] }, "socket_addr": { "type": "v1", "addr": "172.27.254.6:0", "nonce": 2027668300 }, "con_type": "client", "con_features": 3387146417253690110, "con_features_hex": "2f018fb87aa4aafe", "con_features_release": "luminous", ---snip---
Zitat von quaglio@bol.com.br:
Hi Eugen! Thanks a lot! I was able to find luminous connections, but I still can't identify which client process. Here is the output: Rafael.
──────────────────────────────────────────────────────────────────
De: "Eugen Block" Enviada: 2025/04/08 04:37:47 Para: ceph-users@ceph.io Assunto: [ceph-users] Re: Ceph squid fresh install Hi, you can query the MON sessions to identify your older clients with: ceph tell mon. sessions It will show you the IP address, con_features_release (Luminous) and a couple of other things. Zitat von Laura Flores : > Hi Rafael,
I would not force the min_compat_client to be reef when there are still > luminous clients connected, as it is important for all clients to be =Reef to understand/encode the pg_upmap_primary feature in the osdmap. >> As for checking which processes are still luminous, I am copying @Radoslaw > Zarzynski who may be able to help more with that. >> Thanks, > Laura Flores >> On Mon, Apr 7, 2025 at 11:30 AM quaglio@bol.com.br > wrote: >> Hi, >> I just did a new Ceph installation and would like to enable the "read >> balancer". >> However, the documentation requires that the minimum client version >> be reef. I checked this information through "ceph features" and came across >> the situation of having 2 luminous clients. >> # ceph features >> { >> "mon": [ >> { >> "features": "0x3f03cffffffdffff", > "release": "squid", >> "num": 2 >> } >> ], >> "mds": [ >> { >> "features": "0x3f03cffffffdffff", >> "release": "squid", >> "num": 2 } > ], >> "osd": [ >> { >> "features": "0x3f03cffffffdffff", >> "release": "squid", >> "num": 38 >> } >> ], >> "client": [ >> { >> "features": "0x2f018fb87aa4aafe", >> "release": "luminous", >> "num": 2 >> }, >> { >> "features": "0x3f03cffffffdffff", >> "release": "squid", >> "num": 5 } > ], >> "mgr": [ >> { >> "features": "0x3f03cffffffdffff", >> "release": "squid", >> "num": 2 >> } >> ] >> } >>>> I tryed to configure the minimum version to reef and received the >> following alert: >> # ceph osd set-require-min-compat-client reef >> Error EPERM: cannot set require_min_compat_client to reef: 2 connected >> client(s) look like luminous (missing 0x80000000); add >> --yes-i-really-mean-it to do it anyway >>>> Is it ok do confirm anyway? >> Which processes are still as luminous? >>>> Rafael. >>
> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>>> -- >> Laura Flores >> She/Her/Hers > Software Engineer, Ceph Storage >>> Chicago, IL >> lflores@ibm.com | lflores@redhat.com > M: +17087388804 > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io