Update of MDS (non-cephadm cluster)
I have done the update of the ceph software several times, but this is the first time I have to do an update (from quincy to reef) considering also ceph-fs I am not using cephadm I have 3 MDSs (2 active and one in standby) I'm looking at point 5 of: https://ceph.io/en/news/blog/2023/v18-2-0-reef-released/ If I have got it right It says to temporarily set max_mds to 1 and when you have only one active: 1 - turn off the MDS service on the 2 inactive MDS nodes 2- update and restart MDS on the active instance 3- restart (after updating the software: this is not written in the doc but it seems obvious to me) MDS on the other 2 instances 4 put max_mds back to 2 Did I understand correctly? But doesn't this mean to have a short period of time when the cluster is in error state (in 2, during the restart) and therefore with possible problems on the cephfs clients ? Thanks, Massimo
Am 4/7/25 um 10:16 schrieb Massimo Sgaravatto:
Did I understand correctly? But doesn't this mean to have a short period of time when the cluster is in error state (in 2, during the restart) and therefore with possible problems on the cephfs clients ?
Your understanding of the procedure and its consequences is correct. The CephFS clients will block IO for the time the remaining MDS is restarted. This is similar to a restarting NFS server. An update to the MDS may affect the metadata pool, therefor it has to be done this way. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin
Hi, I just did a new Ceph installation and would like to enable the "read balancer". However, the documentation requires that the minimum client version be reef. I checked this information through "ceph features" and came across the situation of having 2 luminous clients. # ceph features { "mon": [ { "features": "0x3f03cffffffdffff", "release": "squid", "num": 2 } ], "mds": [ { "features": "0x3f03cffffffdffff", "release": "squid", "num": 2 } ], "osd": [ { "features": "0x3f03cffffffdffff", "release": "squid", "num": 38 } ], "client": [ { "features": "0x2f018fb87aa4aafe", "release": "luminous", "num": 2 }, { "features": "0x3f03cffffffdffff", "release": "squid", "num": 5 } ], "mgr": [ { "features": "0x3f03cffffffdffff", "release": "squid", "num": 2 } ] } I tryed to configure the minimum version to reef and received the following alert: # ceph osd set-require-min-compat-client reef Error EPERM: cannot set require_min_compat_client to reef: 2 connected client(s) look like luminous (missing 0x80000000); add --yes-i-really-mean-it to do it anyway Is it ok do confirm anyway? Which processes are still as luminous? Rafael.
Hi Rafael, I would not force the min_compat_client to be reef when there are still luminous clients connected, as it is important for all clients to be >=Reef to understand/encode the pg_upmap_primary feature in the osdmap. As for checking which processes are still luminous, I am copying @Radoslaw Zarzynski <rzarzyns@redhat.com> who may be able to help more with that. Thanks, Laura Flores On Mon, Apr 7, 2025 at 11:30 AM quaglio@bol.com.br <quaglio@bol.com.br> wrote:
Hi, I just did a new Ceph installation and would like to enable the "read balancer". However, the documentation requires that the minimum client version be reef. I checked this information through "ceph features" and came across the situation of having 2 luminous clients. # ceph features { "mon": [ { "features": "0x3f03cffffffdffff", "release": "squid", "num": 2 } ], "mds": [ { "features": "0x3f03cffffffdffff", "release": "squid", "num": 2 } ], "osd": [ { "features": "0x3f03cffffffdffff", "release": "squid", "num": 38 } ], "client": [ { "features": "0x2f018fb87aa4aafe", "release": "luminous", "num": 2 }, { "features": "0x3f03cffffffdffff", "release": "squid", "num": 5 } ], "mgr": [ { "features": "0x3f03cffffffdffff", "release": "squid", "num": 2 } ] }
I tryed to configure the minimum version to reef and received the following alert: # ceph osd set-require-min-compat-client reef Error EPERM: cannot set require_min_compat_client to reef: 2 connected client(s) look like luminous (missing 0x80000000); add --yes-i-really-mean-it to do it anyway
Is it ok do confirm anyway? Which processes are still as luminous?
Rafael. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores@ibm.com | lflores@redhat.com <lflores@redhat.com> M: +17087388804
Hi, you can query the MON sessions to identify your older clients with: ceph tell mon.<MON> sessions It will show you the IP address, con_features_release (Luminous) and a couple of other things. Zitat von Laura Flores <lflores@redhat.com>:
Hi Rafael,
I would not force the min_compat_client to be reef when there are still luminous clients connected, as it is important for all clients to be >=Reef to understand/encode the pg_upmap_primary feature in the osdmap.
As for checking which processes are still luminous, I am copying @Radoslaw Zarzynski <rzarzyns@redhat.com> who may be able to help more with that.
Thanks, Laura Flores
On Mon, Apr 7, 2025 at 11:30 AM quaglio@bol.com.br <quaglio@bol.com.br> wrote:
Hi, I just did a new Ceph installation and would like to enable the "read balancer". However, the documentation requires that the minimum client version be reef. I checked this information through "ceph features" and came across the situation of having 2 luminous clients. # ceph features { "mon": [ { "features": "0x3f03cffffffdffff", "release": "squid", "num": 2 } ], "mds": [ { "features": "0x3f03cffffffdffff", "release": "squid", "num": 2 } ], "osd": [ { "features": "0x3f03cffffffdffff", "release": "squid", "num": 38 } ], "client": [ { "features": "0x2f018fb87aa4aafe", "release": "luminous", "num": 2 }, { "features": "0x3f03cffffffdffff", "release": "squid", "num": 5 } ], "mgr": [ { "features": "0x3f03cffffffdffff", "release": "squid", "num": 2 } ] }
I tryed to configure the minimum version to reef and received the following alert: # ceph osd set-require-min-compat-client reef Error EPERM: cannot set require_min_compat_client to reef: 2 connected client(s) look like luminous (missing 0x80000000); add --yes-i-really-mean-it to do it anyway
Is it ok do confirm anyway? Which processes are still as luminous?
Rafael. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
--
Laura Flores
She/Her/Hers
Software Engineer, Ceph Storage <https://ceph.io>
Chicago, IL
lflores@ibm.com | lflores@redhat.com <lflores@redhat.com> M: +17087388804 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
These are your two Luminous clients: ---snip--- { "name": "unknown.0", "entity_name": "client.admin", "addrs": { "addrvec": [ { "type": "none", "addr": "172.27.254.7:0", "nonce": 443842330 } ] }, "socket_addr": { "type": "none", "addr": "172.27.254.7:0", "nonce": 443842330 }, "con_type": "client", "con_features": 3387146417253690110, "con_features_hex": "2f018fb87aa4aafe", "con_features_release": "luminous", ... { "name": "client.104098", "entity_name": "client.admin", "addrs": { "addrvec": [ { "type": "v1", "addr": "172.27.254.6:0", "nonce": 2027668300 } ] }, "socket_addr": { "type": "v1", "addr": "172.27.254.6:0", "nonce": 2027668300 }, "con_type": "client", "con_features": 3387146417253690110, "con_features_hex": "2f018fb87aa4aafe", "con_features_release": "luminous", ---snip--- Zitat von quaglio@bol.com.br:
Hi Eugen! Thanks a lot! I was able to find luminous connections, but I still can't identify which client process. Here is the output: Rafael. ────────────────────────────────────────────────────────────────── De: "Eugen Block" Enviada: 2025/04/08 04:37:47 Para: ceph-users@ceph.io Assunto: [ceph-users] Re: Ceph squid fresh install Hi, you can query the MON sessions to identify your older clients with: ceph tell mon. sessions It will show you the IP address, con_features_release (Luminous) and a couple of other things. Zitat von Laura Flores : > Hi Rafael, >> I would not force the min_compat_client to be reef when there are still > luminous clients connected, as it is important for all clients to be >=Reef
to understand/encode the pg_upmap_primary feature in the osdmap. >> As for checking which processes are still luminous, I am copying @Radoslaw > Zarzynski who may be able to help more with that. >> Thanks, > Laura Flores >> On Mon, Apr 7, 2025 at 11:30 AM quaglio@bol.com.br > wrote:
Hi, >> I just did a new Ceph installation and would like to enable the "read >> balancer". >> However, the documentation requires that the minimum client version >> be reef. I checked this information through "ceph features" and came across >> the situation of having 2 luminous clients. >> # ceph features >> { >> "mon": [ >> { >> "features": "0x3f03cffffffdffff", "release": "squid", >> "num": 2 >> } >> ], >> "mds": [ >> { >> "features": "0x3f03cffffffdffff", >> "release": "squid", >> "num": 2 >> } ], >> "osd": [ >> { >> "features": "0x3f03cffffffdffff", >> "release": "squid", >> "num": 38 >> } >> ], >> "client": [ >> { >> "features": "0x2f018fb87aa4aafe", >> "release": "luminous", >> "num": 2 >> }, >> { >> "features": "0x3f03cffffffdffff", >> "release": "squid", >> "num": 5 >> } ], >> "mgr": [ >> { >> "features": "0x3f03cffffffdffff", >> "release": "squid", >> "num": 2 >> } >> ] >> } >>>> I tryed to configure the minimum version to reef and received the >> following alert: >> # ceph osd set-require-min-compat-client reef >> Error EPERM: cannot set require_min_compat_client to reef: 2 connected >> client(s) look like luminous (missing 0x80000000); add >> --yes-i-really-mean-it to do it anyway >>>> Is it ok do confirm anyway? >> Which processes are still as luminous? >>>> Rafael. >> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>>> -- >> Laura Flores >> She/Her/Hers Software Engineer, Ceph Storage >>> Chicago, IL >> lflores@ibm.com | lflores@redhat.com > M: +17087388804 > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
What is the locally installed ceph version? Run 'ceph --version' to get the local rpm/deb package version. Zitat von quaglio@bol.com.br:
These 2 IPs are from the storage servers. There are no user processes running on them. It only has the operating system and ceph installed.
Rafael.
-------------------------
DE: "Eugen Block" <eblock@nde.ag> ENVIADA: 2025/04/08 09:35:35 PARA: quaglio@bol.com.br CC: ceph-users@ceph.io ASSUNTO: Re: [ceph-users] Ceph squid fresh install These are your two Luminous clients:
---snip--- { "name": "unknown.0", "entity_name": "client.admin", "addrs": { "addrvec": [ { "type": "none", "addr": "172.27.254.7:0", "nonce": 443842330 } ] }, "socket_addr": { "type": "none", "addr": "172.27.254.7:0", "nonce": 443842330 }, "con_type": "client", "con_features": 3387146417253690110, "con_features_hex": "2f018fb87aa4aafe", "con_features_release": "luminous", ...
{ "name": "client.104098", "entity_name": "client.admin", "addrs": { "addrvec": [ { "type": "v1", "addr": "172.27.254.6:0", "nonce": 2027668300 } ] }, "socket_addr": { "type": "v1", "addr": "172.27.254.6:0", "nonce": 2027668300 }, "con_type": "client", "con_features": 3387146417253690110, "con_features_hex": "2f018fb87aa4aafe", "con_features_release": "luminous", ---snip---
Zitat von quaglio@bol.com.br:
Hi Eugen! Thanks a lot! I was able to find luminous connections, but I still can't identify which client process. Here is the output: Rafael.
──────────────────────────────────────────────────────────────────
De: "Eugen Block" Enviada: 2025/04/08 04:37:47 Para: ceph-users@ceph.io Assunto: [ceph-users] Re: Ceph squid fresh install Hi, you can query the MON sessions to identify your older clients with: ceph tell mon. sessions It will show you the IP address, con_features_release (Luminous) and a couple of other things. Zitat von Laura Flores : > Hi Rafael, >> I would not force the min_compat_client to be reef when there are still > luminous clients connected, as it is important for all clients to be =Reef
to understand/encode the pg_upmap_primary feature in the osdmap. >> As for checking which processes are still luminous, I am copying @Radoslaw > Zarzynski who may be able to help more with that. >> Thanks, > Laura Flores >> On Mon, Apr 7, 2025 at 11:30 AM quaglio@bol.com.br > wrote:
Hi, >> I just did a new Ceph installation and would like to enable the "read >> balancer". >> However, the documentation requires that the minimum client version >> be reef. I checked this information through "ceph features" and came across >> the situation of having 2 luminous clients.
# ceph features >> { >> "mon": [ >> { >> "features": "0x3f03cffffffdffff",
"release": "squid", >> "num": 2 >> } >> ], >> "mds": [ >> { >> "features": "0x3f03cffffffdffff", >> "release": "squid", >> "num": 2 >> } ], >> "osd": [ >> { >> "features": "0x3f03cffffffdffff", >> "release": "squid", >> "num": 38 >> } >> ], >> "client": [ >> { >> "features": "0x2f018fb87aa4aafe", >> "release": "luminous", >> "num": 2 >> }, >> { >> "features": "0x3f03cffffffdffff", >> "release": "squid", >> "num": 5 >> } ], >> "mgr": [ >> { >> "features": "0x3f03cffffffdffff", >> "release": "squid", >> "num": 2 >> } >> ] >> } >>>> I tryed to configure the minimum version to reef and received the >> following alert: >> # ceph osd set-require-min-compat-client reef >> Error EPERM: cannot set require_min_compat_client to reef: 2 connected >> client(s) look like luminous (missing 0x80000000); add >> --yes-i-really-mean-it to do it anyway >>>> Is it ok do confirm anyway? >> Which processes are still as luminous? >>>> Rafael. >> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>>> -- >> Laura Flores >> She/Her/Hers Software Engineer, Ceph Storage >>> Chicago, IL >> lflores@ibm.com | lflores@redhat.com > M: +17087388804 > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On Apr 8, 2025, at 9:13 AM, quaglio@bol.com.br wrote:
These 2 IPs are from the storage servers.
What is a “storage server”?
There are no user processes running on them. It only has the operating system and ceph installed.
Nobody said anything about user processes.
Rafael.
De: "Eugen Block" <eblock@nde.ag> Enviada: 2025/04/08 09:35:35 Para: quaglio@bol.com.br Cc: ceph-users@ceph.io Assunto: Re: [ceph-users] Ceph squid fresh install
These are your two Luminous clients:
---snip--- { "name": "unknown.0", "entity_name": "client.admin", "addrs": { "addrvec": [ { "type": "none", "addr": "172.27.254.7:0", "nonce": 443842330 } ] }, "socket_addr": { "type": "none", "addr": "172.27.254.7:0", "nonce": 443842330 }, "con_type": "client", "con_features": 3387146417253690110, "con_features_hex": "2f018fb87aa4aafe", "con_features_release": "luminous", ...
{ "name": "client.104098", "entity_name": "client.admin", "addrs": { "addrvec": [ { "type": "v1", "addr": "172.27.254.6:0", "nonce": 2027668300 } ] }, "socket_addr": { "type": "v1", "addr": "172.27.254.6:0", "nonce": 2027668300 }, "con_type": "client", "con_features": 3387146417253690110, "con_features_hex": "2f018fb87aa4aafe", "con_features_release": "luminous", ---snip---
Zitat von quaglio@bol.com.br:
Hi Eugen! Thanks a lot! I was able to find luminous connections, but I still can't identify which client process. Here is the output: Rafael. ────────────────────────────────────────────────────────────────── De: "Eugen Block" Enviada: 2025/04/08 04:37:47 Para: ceph-users@ceph.io Assunto: [ceph-users] Re: Ceph squid fresh install Hi, you can query the MON sessions to identify your older clients with: ceph tell mon. sessions It will show you the IP address, con_features_release (Luminous) and a couple of other things. Zitat von Laura Flores : > Hi Rafael, >> I would not force the min_compat_client to be reef when there are still > luminous clients connected, as it is important for all clients to be >=Reef
to understand/encode the pg_upmap_primary feature in the osdmap. >> As for checking which processes are still luminous, I am copying @Radoslaw > Zarzynski who may be able to help more with that. >> Thanks, > Laura Flores >> On Mon, Apr 7, 2025 at 11:30 AM quaglio@bol.com.br > wrote:
Hi, >> I just did a new Ceph installation and would like to enable the "read >> balancer". >> However, the documentation requires that the minimum client version >> be reef. I checked this information through "ceph features" and came across >> the situation of having 2 luminous clients. >> # ceph features >> { >> "mon": [ >> { >> "features": "0x3f03cffffffdffff", "release": "squid", >> "num": 2 >> } >> ], >> "mds": [ >> { >> "features": "0x3f03cffffffdffff", >> "release": "squid", >> "num": 2 >> } ], >> "osd": [ >> { >> "features": "0x3f03cffffffdffff", >> "release": "squid", >> "num": 38 >> } >> ], >> "client": [ >> { >> "features": "0x2f018fb87aa4aafe", >> "release": "luminous", >> "num": 2 >> }, >> { >> "features": "0x3f03cffffffdffff", >> "release": "squid", >> "num": 5 >> } ], >> "mgr": [ >> { >> "features": "0x3f03cffffffdffff", >> "release": "squid", >> "num": 2 >> } >> ] >> } >>>> I tryed to configure the minimum version to reef and received the >> following alert: >> # ceph osd set-require-min-compat-client reef >> Error EPERM: cannot set require_min_compat_client to reef: 2 connected >> client(s) look like luminous (missing 0x80000000); add >> --yes-i-really-mean-it to do it anyway >>>> Is it ok do confirm anyway? >> Which processes are still as luminous? >>>> Rafael. >> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>>> -- >> Laura Flores >> She/Her/Hers Software Engineer, Ceph Storage >>> Chicago, IL >> lflores@ibm.com | lflores@redhat.com > M: +17087388804 > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
What does “ceph installed” mean? I suspect that this description is not complete.
On Apr 8, 2025, at 9:21 AM, quaglio@bol.com.br wrote:
What is a “storage server”? These are machines that only have the operating system and ceph installed.
De: "Anthony D'Atri" <anthony.datri@gmail.com> Enviada: 2025/04/08 10:19:08 Para: quaglio@bol.com.br Cc: eblock@nde.ag, ceph-users@ceph.io Assunto: Re: [ceph-users] Ceph squid fresh install
On Apr 8, 2025, at 9:13 AM, quaglio@bol.com.br wrote:
These 2 IPs are from the storage servers.
What is a “storage server”?
There are no user processes running on them. It only has the operating system and ceph installed.
Nobody said anything about user processes.
Rafael.
De: "Eugen Block" <eblock@nde.ag> Enviada: 2025/04/08 09:35:35 Para: quaglio@bol.com.br Cc: ceph-users@ceph.io Assunto: Re: [ceph-users] Ceph squid fresh install
These are your two Luminous clients:
---snip--- { "name": "unknown.0", "entity_name": "client.admin", "addrs": { "addrvec": [ { "type": "none", "addr": "172.27.254.7:0", "nonce": 443842330 } ] }, "socket_addr": { "type": "none", "addr": "172.27.254.7:0", "nonce": 443842330 }, "con_type": "client", "con_features": 3387146417253690110, "con_features_hex": "2f018fb87aa4aafe", "con_features_release": "luminous", ...
{ "name": "client.104098", "entity_name": "client.admin", "addrs": { "addrvec": [ { "type": "v1", "addr": "172.27.254.6:0", "nonce": 2027668300 } ] }, "socket_addr": { "type": "v1", "addr": "172.27.254.6:0", "nonce": 2027668300 }, "con_type": "client", "con_features": 3387146417253690110, "con_features_hex": "2f018fb87aa4aafe", "con_features_release": "luminous", ---snip---
Zitat von quaglio@bol.com.br:
Hi Eugen! Thanks a lot! I was able to find luminous connections, but I still can't identify which client process. Here is the output: Rafael. ────────────────────────────────────────────────────────────────── De: "Eugen Block" Enviada: 2025/04/08 04:37:47 Para: ceph-users@ceph.io Assunto: [ceph-users] Re: Ceph squid fresh install Hi, you can query the MON sessions to identify your older clients with: ceph tell mon. sessions It will show you the IP address, con_features_release (Luminous) and a couple of other things. Zitat von Laura Flores : > Hi Rafael, >> I would not force the min_compat_client to be reef when there are still > luminous clients connected, as it is important for all clients to be >=Reef
to understand/encode the pg_upmap_primary feature in the osdmap. >> As for checking which processes are still luminous, I am copying @Radoslaw > Zarzynski who may be able to help more with that. >> Thanks, > Laura Flores >> On Mon, Apr 7, 2025 at 11:30 AM quaglio@bol.com.br > wrote:
Hi, >> I just did a new Ceph installation and would like to enable the "read >> balancer". >> However, the documentation requires that the minimum client version >> be reef. I checked this information through "ceph features" and came across >> the situation of having 2 luminous clients. >> # ceph features >> { >> "mon": [ >> { >> "features": "0x3f03cffffffdffff", "release": "squid", >> "num": 2 >> } >> ], >> "mds": [ >> { >> "features": "0x3f03cffffffdffff", >> "release": "squid", >> "num": 2 >> } ], >> "osd": [ >> { >> "features": "0x3f03cffffffdffff", >> "release": "squid", >> "num": 38 >> } >> ], >> "client": [ >> { >> "features": "0x2f018fb87aa4aafe", >> "release": "luminous", >> "num": 2 >> }, >> { >> "features": "0x3f03cffffffdffff", >> "release": "squid", >> "num": 5 >> } ], >> "mgr": [ >> { >> "features": "0x3f03cffffffdffff", >> "release": "squid", >> "num": 2 >> } >> ] >> } >>>> I tryed to configure the minimum version to reef and received the >> following alert: >> # ceph osd set-require-min-compat-client reef >> Error EPERM: cannot set require_min_compat_client to reef: 2 connected >> client(s) look like luminous (missing 0x80000000); add >> --yes-i-really-mean-it to do it anyway >>>> Is it ok do confirm anyway? >> Which processes are still as luminous? >>>> Rafael. >> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>>> -- >> Laura Flores >> She/Her/Hers Software Engineer, Ceph Storage >>> Chicago, IL >> lflores@ibm.com | lflores@redhat.com > M: +17087388804 > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
So, OSD nodes that do not have any OSDs on them yet?
On Apr 8, 2025, at 9:41 AM, quaglio@bol.com.br wrote:
More complete description:
1-) I formatted and installed the operating system
2-) This is "ceph installed":
curl --silent --remote-name --location https://download.ceph.com/rpm-19.2.1/el9/noarch/cephadm chmod +x cephadm
./cephadm add-repo --release squid ./cephadm install
cephadm -v bootstrap --mon-ip 172.27.254.6 --cluster-network 172.28.254.0/24 --log-to-file
cephadm install ceph-common
De: "Anthony D'Atri" <anthony.datri@gmail.com> Enviada: 2025/04/08 10:35:22 Para: quaglio@bol.com.br Cc: eblock@nde.ag, ceph-users@ceph.io Assunto: Re: [ceph-users] Ceph squid fresh install
What does “ceph installed” mean? I suspect that this description is not complete.
On Apr 8, 2025, at 9:21 AM, quaglio@bol.com.br wrote:
What is a “storage server”? These are machines that only have the operating system and ceph installed.
De: "Anthony D'Atri" <anthony.datri@gmail.com> Enviada: 2025/04/08 10:19:08 Para: quaglio@bol.com.br Cc: eblock@nde.ag, ceph-users@ceph.io Assunto: Re: [ceph-users] Ceph squid fresh install
On Apr 8, 2025, at 9:13 AM, quaglio@bol.com.br wrote:
These 2 IPs are from the storage servers.
What is a “storage server”?
There are no user processes running on them. It only has the operating system and ceph installed.
Nobody said anything about user processes.
Rafael.
De: "Eugen Block" <eblock@nde.ag> Enviada: 2025/04/08 09:35:35 Para: quaglio@bol.com.br Cc: ceph-users@ceph.io Assunto: Re: [ceph-users] Ceph squid fresh install
These are your two Luminous clients:
---snip--- { "name": "unknown.0", "entity_name": "client.admin", "addrs": { "addrvec": [ { "type": "none", "addr": "172.27.254.7:0", "nonce": 443842330 } ] }, "socket_addr": { "type": "none", "addr": "172.27.254.7:0", "nonce": 443842330 }, "con_type": "client", "con_features": 3387146417253690110, "con_features_hex": "2f018fb87aa4aafe", "con_features_release": "luminous", ...
{ "name": "client.104098", "entity_name": "client.admin", "addrs": { "addrvec": [ { "type": "v1", "addr": "172.27.254.6:0", "nonce": 2027668300 } ] }, "socket_addr": { "type": "v1", "addr": "172.27.254.6:0", "nonce": 2027668300 }, "con_type": "client", "con_features": 3387146417253690110, "con_features_hex": "2f018fb87aa4aafe", "con_features_release": "luminous", ---snip---
Zitat von quaglio@bol.com.br:
Hi Eugen! Thanks a lot! I was able to find luminous connections, but I still can't identify which client process. Here is the output: Rafael. ────────────────────────────────────────────────────────────────── De: "Eugen Block" Enviada: 2025/04/08 04:37:47 Para: ceph-users@ceph.io Assunto: [ceph-users] Re: Ceph squid fresh install Hi, you can query the MON sessions to identify your older clients with: ceph tell mon. sessions It will show you the IP address, con_features_release (Luminous) and a couple of other things. Zitat von Laura Flores : > Hi Rafael, >> I would not force the min_compat_client to be reef when there are still > luminous clients connected, as it is important for all clients to be >=Reef
to understand/encode the pg_upmap_primary feature in the osdmap. >> As for checking which processes are still luminous, I am copying @Radoslaw > Zarzynski who may be able to help more with that. >> Thanks, > Laura Flores >> On Mon, Apr 7, 2025 at 11:30 AM quaglio@bol.com.br > wrote:
> Hi, >> I just did a new Ceph installation and would like to enable the "read >> balancer". >> However, the documentation requires that the minimum client version >> be reef. I checked this information through "ceph features" and came across >> the situation of having 2 luminous clients. >> # ceph features >> { >> "mon": [ >> { >> "features": "0x3f03cffffffdffff", "release": "squid", >> "num": 2 >> } >> ], >> "mds": [ >> { >> "features": "0x3f03cffffffdffff", >> "release": "squid", >> "num": 2 >> } ], >> "osd": [ >> { >> "features": "0x3f03cffffffdffff", >> "release": "squid", >> "num": 38 >> } >> ], >> "client": [ >> { >> "features": "0x2f018fb87aa4aafe", >> "release": "luminous", >> "num": 2 >> }, >> { >> "features": "0x3f03cffffffdffff", >> "release": "squid", >> "num": 5 >> } ], >> "mgr": [ >> { >> "features": "0x3f03cffffffdffff", >> "release": "squid", >> "num": 2 >> } >> ] >> } >>>> I tryed to configure the minimum version to reef and received the >> following alert: >> # ceph osd set-require-min-compat-client reef >> Error EPERM: cannot set require_min_compat_client to reef: 2 connected >> client(s) look like luminous (missing 0x80000000); add >> --yes-i-really-mean-it to do it anyway >>>> Is it ok do confirm anyway? >> Which processes are still as luminous? >>>> Rafael. >> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>>> -- >> Laura Flores >> She/Her/Hers Software Engineer, Ceph Storage >>> Chicago, IL >> lflores@ibm.com | lflores@redhat.com > M: +17087388804 > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi all, This most likely boils down to the kernel version of connected clients. We faced the same problem some time ago and just kicked away the client mds connections with a follow up kernel update on the servers doing the mount. I can check the version details later for more clarity. Best, Laimis J. On Tue, Apr 8, 2025, 16:48 Anthony D'Atri <anthony.datri@gmail.com> wrote:
So, OSD nodes that do not have any OSDs on them yet?
On Apr 8, 2025, at 9:41 AM, quaglio@bol.com.br wrote:
More complete description:
1-) I formatted and installed the operating system
2-) This is "ceph installed":
curl --silent --remote-name --location https://download.ceph.com/rpm-19.2.1/el9/noarch/cephadm chmod +x cephadm
./cephadm add-repo --release squid ./cephadm install
cephadm -v bootstrap --mon-ip 172.27.254.6 --cluster-network 172.28.254.0/24 --log-to-file
cephadm install ceph-common
De: "Anthony D'Atri" <anthony.datri@gmail.com> Enviada: 2025/04/08 10:35:22 Para: quaglio@bol.com.br Cc: eblock@nde.ag, ceph-users@ceph.io Assunto: Re: [ceph-users] Ceph squid fresh install
What does “ceph installed” mean? I suspect that this description is not complete.
On Apr 8, 2025, at 9:21 AM, quaglio@bol.com.br wrote:
What is a “storage server”? These are machines that only have the operating system and
ceph installed.
De: "Anthony D'Atri" <anthony.datri@gmail.com> Enviada: 2025/04/08 10:19:08 Para: quaglio@bol.com.br Cc: eblock@nde.ag, ceph-users@ceph.io Assunto: Re: [ceph-users] Ceph squid fresh install
On Apr 8, 2025, at 9:13 AM, quaglio@bol.com.br wrote:
These 2 IPs are from the storage servers.
What is a “storage server”?
There are no user processes running on them. It only has the
operating system and ceph installed.
Nobody said anything about user processes.
Rafael.
De: "Eugen Block" <eblock@nde.ag> Enviada: 2025/04/08 09:35:35 Para: quaglio@bol.com.br Cc: ceph-users@ceph.io Assunto: Re: [ceph-users] Ceph squid fresh install
These are your two Luminous clients:
---snip--- { "name": "unknown.0", "entity_name": "client.admin", "addrs": { "addrvec": [ { "type": "none", "addr": "172.27.254.7:0", "nonce": 443842330 } ] }, "socket_addr": { "type": "none", "addr": "172.27.254.7:0", "nonce": 443842330 }, "con_type": "client", "con_features": 3387146417253690110, "con_features_hex": "2f018fb87aa4aafe", "con_features_release": "luminous", ...
{ "name": "client.104098", "entity_name": "client.admin", "addrs": { "addrvec": [ { "type": "v1", "addr": "172.27.254.6:0", "nonce": 2027668300 } ] }, "socket_addr": { "type": "v1", "addr": "172.27.254.6:0", "nonce": 2027668300 }, "con_type": "client", "con_features": 3387146417253690110, "con_features_hex": "2f018fb87aa4aafe", "con_features_release": "luminous", ---snip---
Zitat von quaglio@bol.com.br:
Hi Eugen! Thanks a lot! I was able to find luminous connections, but I still can't identify which client process. Here is the output: Rafael. ────────────────────────────────────────────────────────────────── De: "Eugen Block" Enviada: 2025/04/08 04:37:47 Para:
Assunto: [ceph-users] Re: Ceph squid fresh install Hi, you can query the MON sessions to identify your older clients with: ceph tell mon. sessions It will show you the IP address, con_features_release (Luminous) and a couple of other things. Zitat von Laura Flores : > Hi Rafael, I would not force the min_compat_client to be reef when there are still > luminous clients connected, as it is important for all clients to be >=Reef
to understand/encode the pg_upmap_primary feature in the osdmap. As for checking which processes are still luminous, I am copying @Radoslaw > Zarzynski who may be able to help more with that. >> Thanks, > Laura Flores >> On Mon, Apr 7, 2025 at 11:30 AM quaglio@bol.com.br > wrote: >> Hi, >> I just did a new Ceph installation and would like to enable the "read >> balancer". >> However, the documentation requires that the minimum client version >> be reef. I checked this information through "ceph features" and came across >> the situation of having 2 luminous clients. >> # ceph features >> { >> "mon": [ >> { >> "features": "0x3f03cffffffdffff", > "release": "squid", >> "num": 2 >> } >> ], >> "mds": [ >> { >> "features": "0x3f03cffffffdffff", >> "release": "squid", >> "num": 2 >> } > ], >> "osd": [ >> { >> "features": "0x3f03cffffffdffff", >> "release": "squid", >> "num": 38 >> } >> ], >> "client": [ >> { >> "features": "0x2f018fb87aa4aafe", >> "release": "luminous", >> "num": 2 >> }, { >> "features": "0x3f03cffffffdffff", >> "release": "squid", >> "num": 5 >> } > ], >> "mgr": [ >> { >> "features": "0x3f03cffffffdffff", >> "release": "squid", >> "num": 2 >> } >> ] >> } >>>> I tryed to configure the minimum version to reef and received the >> following alert: >> # ceph osd set-require-min-compat-client reef >> Error EPERM: cannot set require_min_compat_client to reef: 2 connected >> client(s) look
ceph-users@ceph.io like
luminous (missing 0x80000000); add >> --yes-i-really-mean-it to do it anyway >>>> Is it ok do confirm anyway? >> Which processes are still as luminous? >>>> Rafael. >>
> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>>> -- >> Laura Flores >> She/Her/Hers > Software Engineer, Ceph Storage >>> Chicago, IL >> lflores@ibm.com | lflores@redhat.com > M: +17087388804 > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
It's not necessarily a ceph mount that shows older features, but the kernel version might still be relevant. Just a quick example, a fresh 19.2.1 install on a virtual machine with kernel: soc9-ceph:~ # uname -r 6.4.0-150600.23.33-default soc9-ceph:~ # ceph versions { "mon": { "ceph version 19.2.1 (58a7fab8be0a062d730ad7da874972fd3fba59fb) squid (stable)": 1 }, "mgr": { "ceph version 19.2.1 (58a7fab8be0a062d730ad7da874972fd3fba59fb) squid (stable)": 2 }, "overall": { "ceph version 19.2.1 (58a7fab8be0a062d730ad7da874972fd3fba59fb) squid (stable)": 3 } } Within the cephadm shell, I have only squid features: [ceph: root@soc9-ceph /]# ceph features { "mon": [ { "features": "0x3f03cffffffdffff", "release": "squid", "num": 1 } ], "client": [ { "features": "0x3f03cffffffdffff", "release": "squid", "num": 4 } ], "mgr": [ { "features": "0x3f03cffffffdffff", "release": "squid", "num": 2 } ] } But if I run the command outside of cephadm shell, I have a Reef feature: soc9-ceph:~ # ceph features { "mon": [ { "features": "0x3f03cffffffdffff", "release": "squid", "num": 1 } ], "client": [ { "features": "0x3f01cfbdfffdffff", "release": "reef", "num": 1 }, ... The locally installed rpm version is Pacific, though: soc9-ceph:~ # ceph --version ceph version 16.2.13-66-g54799ee0666 (54799ee06669271880ee5fc715f99202002aa371) pacific (stable) If you enforced min-compat-client to reef (ceph osd set-require-min-compat-client reef --yes-i-really-mean-it), you would not be able to use Ceph commands from the regular shell of your system(s), but you would be required to always use the cephadm shell. Zitat von quaglio@bol.com.br:
I added 38 disks.
In short, even though I didn't specify the number of disks, what I want to say is that it's a clean installation.
The OSD containers are also from the squid version (I didn't configure anything other than what's in the documentation). I don't understand why you're asking this kind of question.
Could you tell me all the information you need to help me understand the problem with the luminous clients?
-------------------------
DE: "Anthony D'Atri" <anthony.datri@gmail.com> ENVIADA: 2025/04/08 10:44:23 PARA: quaglio@bol.com.br CC: eblock@nde.ag, ceph-users@ceph.io ASSUNTO: Re: [ceph-users] Ceph squid fresh install So, OSD nodes that do not have any OSDs on them yet?
On Apr 8, 2025, at 9:41 AM, quaglio@bol.com.br wrote: More complete description:
1-) I formatted and installed the operating system
2-) This is "ceph installed":
curl --silent --remote-name --location https://download.ceph.com/rpm-19.2.1/el9/noarch/cephadm chmod +x cephadm
./cephadm add-repo --release squid ./cephadm install
cephadm -v bootstrap --mon-ip 172.27.254.6 --cluster-network 172.28.254.0/24 --log-to-file
cephadm install ceph-common -------------------------
DE: "Anthony D'Atri" <anthony.datri@gmail.com> ENVIADA: 2025/04/08 10:35:22 PARA: quaglio@bol.com.br CC: eblock@nde.ag, ceph-users@ceph.io ASSUNTO: Re: [ceph-users] Ceph squid fresh install What does “ceph installed” mean? I suspect that this description is not complete.
On Apr 8, 2025, at 9:21 AM, quaglio@bol.com.br wrote: What is a “storage server”? These are machines that only have the operating system and ceph installed. -------------------------
DE: "Anthony D'Atri" <anthony.datri@gmail.com> ENVIADA: 2025/04/08 10:19:08 PARA: quaglio@bol.com.br CC: eblock@nde.ag, ceph-users@ceph.io ASSUNTO: Re: [ceph-users] Ceph squid fresh install
On Apr 8, 2025, at 9:13 AM, quaglio@bol.com.br wrote:
These 2 IPs are from the storage servers.
What is a “storage server”?
There are no user processes running on them. It only has the operating system and ceph installed.
Nobody said anything about user processes.
Rafael.
De: "Eugen Block" <eblock@nde.ag> Enviada: 2025/04/08 09:35:35 Para: quaglio@bol.com.br Cc: ceph-users@ceph.io Assunto: Re: [ceph-users] Ceph squid fresh install
These are your two Luminous clients:
---snip--- { "name": "unknown.0", "entity_name": "client.admin", "addrs": { "addrvec": [ { "type": "none", "addr": "172.27.254.7:0", "nonce": 443842330 } ] }, "socket_addr": { "type": "none", "addr": "172.27.254.7:0", "nonce": 443842330 }, "con_type": "client", "con_features": 3387146417253690110, "con_features_hex": "2f018fb87aa4aafe", "con_features_release": "luminous", ...
{ "name": "client.104098", "entity_name": "client.admin", "addrs": { "addrvec": [ { "type": "v1", "addr": "172.27.254.6:0", "nonce": 2027668300 } ] }, "socket_addr": { "type": "v1", "addr": "172.27.254.6:0", "nonce": 2027668300 }, "con_type": "client", "con_features": 3387146417253690110, "con_features_hex": "2f018fb87aa4aafe", "con_features_release": "luminous", ---snip---
Zitat von quaglio@bol.com.br:
Hi Eugen! Thanks a lot! I was able to find luminous connections, but I still can't identify which client process. Here is the output: Rafael.
──────────────────────────────────────────────────────────────────
De: "Eugen Block" Enviada: 2025/04/08 04:37:47 Para: ceph-users@ceph.io Assunto: [ceph-users] Re: Ceph squid fresh install Hi, you can query the MON sessions to identify your older clients with: ceph tell mon. sessions It will show you the IP address, con_features_release (Luminous) and a couple of other things. Zitat von Laura Flores : > Hi Rafael,
I would not force the min_compat_client to be reef when there are still > luminous clients connected, as it is important for all clients to be =Reef to understand/encode the pg_upmap_primary feature in the osdmap. >> As for checking which processes are still luminous, I am copying @Radoslaw > Zarzynski who may be able to help more with that. >> Thanks, > Laura Flores >> On Mon, Apr 7, 2025 at 11:30 AM quaglio@bol.com.br > wrote: >> Hi, >> I just did a new Ceph installation and would like to enable the "read >> balancer". >> However, the documentation requires that the minimum client version >> be reef. I checked this information through "ceph features" and came across >> the situation of having 2 luminous clients. >> # ceph features >> { >> "mon": [ >> { >> "features": "0x3f03cffffffdffff", > "release": "squid", >> "num": 2 >> } >> ], >> "mds": [ >> { >> "features": "0x3f03cffffffdffff", >> "release": "squid", >> "num": 2 } > ], >> "osd": [ >> { >> "features": "0x3f03cffffffdffff", >> "release": "squid", >> "num": 38 >> } >> ], >> "client": [ >> { >> "features": "0x2f018fb87aa4aafe", >> "release": "luminous", >> "num": 2 >> }, >> { >> "features": "0x3f03cffffffdffff", >> "release": "squid", >> "num": 5 } > ], >> "mgr": [ >> { >> "features": "0x3f03cffffffdffff", >> "release": "squid", >> "num": 2 >> } >> ] >> } >>>> I tryed to configure the minimum version to reef and received the >> following alert: >> # ceph osd set-require-min-compat-client reef >> Error EPERM: cannot set require_min_compat_client to reef: 2 connected >> client(s) look like luminous (missing 0x80000000); add >> --yes-i-really-mean-it to do it anyway >>>> Is it ok do confirm anyway? >> Which processes are still as luminous? >>>> Rafael. >>
> ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>>> -- >> Laura Flores >> She/Her/Hers > Software Engineer, Ceph Storage >>> Chicago, IL >> lflores@ibm.com | lflores@redhat.com > M: +17087388804 > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hello everyone, I am facing a situation that I have not yet been able to find a solution for. There is a client machine that is running a parallel job that generates an output file over time. The problem is that a file with the same name is being generated. There are two files with the same name and different dates. See: # ls -la drwxr-xr-x 5 userxxx groupyyy 12 May 6 23:11 . drwxr-xr-x 19 userxxx groupyyy 74 May 6 14:13 .. -rw-r--r-- 1 userxxx groupyyy 53274 May 3 16:48 lixo -rw-r--r-- 1 userxxx groupyyy 1317 May 3 16:33 mos2_bands.in -rw-r--r-- 1 userxxx groupyyy 54370 May 3 16:49 mos2_nscf.in -rw-r--r-- 1 userxxx groupyyy 2242 May 6 17:16 mos2_nscf.out -rw-r--r-- 1 userxxx groupyyy 1865 May 3 17:28 mos2_nscf.out -rw-r--r-- 1 userxxx groupyyy 1132 May 3 16:43 mos2_scf.in -rw-r--r-- 1 userxxx groupyyy 30626 May 6 17:16 mos2_scf.out -rw-r--r-- 1 userxxx groupyyy 30287 May 3 17:27 mos2_scf.out In POSIX semantics, inside a directory structure, two files/directories with the same name should be not allowed. But when I "change" the ls command, this situation no longer appears. See: # ls -la *out -rw-r--r-- 1 userxxx groupyyy 2242 May 6 17:16 mos2_nscf.out -rw-r--r-- 1 userxxx groupyyy 30626 May 6 17:16 mos2_scf.out This way of running ls, files with the same name do not appear. I currently have a new server in the cluster that is still in the backfill process. This situation could cause this problem. Could you help me identify the problem? Thanks Rafael.
You have some whitespace character at the end of the filename, so it looks like the same name but it is not.
erxxx groupyyy 54370 May 3 16:49 mos2_nscf.in -rw-r--r-- 1 userxxx groupyyy 2242 May 6 17:16 "mos2_nscf.out" -rw-r--r-- 1 userxxx groupyyy 1865 May 3 17:28 "mos2_nscf.out " -rw-r--r-- 1 userxxx groupyyy 1132 May 3 16:43 "mos2_scf.in" -rw-r--r-- 1 userxxx groupyyy 30626 May 6 17:16 "mos2_scf.out" -rw-r--r-- 1 userxxx groupyyy 30287 May 3 17:27 "mos2_scf.out "
Could be that the "out files" are created programmatically and there are any invisible chars after the "t" in the second one e.g. a space? Il 07/05/2025 14:57, quaglio@bol.com.br ha scritto:
Hello everyone, I am facing a situation that I have not yet been able to find a solution for. There is a client machine that is running a parallel job that generates an output file over time. The problem is that a file with the same name is being generated. There are two files with the same name and different dates.
See: # ls -la drwxr-xr-x 5 userxxx groupyyy 12 May 6 23:11 . drwxr-xr-x 19 userxxx groupyyy 74 May 6 14:13 .. -rw-r--r-- 1 userxxx groupyyy 53274 May 3 16:48 lixo -rw-r--r-- 1 userxxx groupyyy 1317 May 3 16:33 mos2_bands.in -rw-r--r-- 1 userxxx groupyyy 54370 May 3 16:49 mos2_nscf.in -rw-r--r-- 1 userxxx groupyyy 2242 May 6 17:16 mos2_nscf.out -rw-r--r-- 1 userxxx groupyyy 1865 May 3 17:28 mos2_nscf.out -rw-r--r-- 1 userxxx groupyyy 1132 May 3 16:43 mos2_scf.in -rw-r--r-- 1 userxxx groupyyy 30626 May 6 17:16 mos2_scf.out -rw-r--r-- 1 userxxx groupyyy 30287 May 3 17:27 mos2_scf.out
In POSIX semantics, inside a directory structure, two files/directories with the same name should be not allowed. But when I "change" the ls command, this situation no longer appears. See:
# ls -la *out -rw-r--r-- 1 userxxx groupyyy 2242 May 6 17:16 mos2_nscf.out -rw-r--r-- 1 userxxx groupyyy 30626 May 6 17:16 mos2_scf.out
This way of running ls, files with the same name do not appear.
I currently have a new server in the cluster that is still in the backfill process. This situation could cause this problem. Could you help me identify the problem?
Thanks Rafael. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- ing. Sergio Rabellino Università degli Studi di Torino Dipartimento di Informatica Tecnico di Ricerca Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 C.so Svizzera , 185 - 10149 - Torino <http://www.di.unito.it>
That is my first thought too. Try ls | od -c See if there’s anything tricky before the newlines.
On May 7, 2025, at 10:00 AM, Sergio Rabellino <rabellino@di.unito.it> wrote:
Could be that the "out files" are created programmatically and there are any invisible chars after the "t" in the second one e.g. a space?
Il 07/05/2025 14:57, quaglio@bol.com.br ha scritto:
Hello everyone, I am facing a situation that I have not yet been able to find a solution for. There is a client machine that is running a parallel job that generates an output file over time. The problem is that a file with the same name is being generated. There are two files with the same name and different dates.
See: # ls -la drwxr-xr-x 5 userxxx groupyyy 12 May 6 23:11 . drwxr-xr-x 19 userxxx groupyyy 74 May 6 14:13 .. -rw-r--r-- 1 userxxx groupyyy 53274 May 3 16:48 lixo -rw-r--r-- 1 userxxx groupyyy 1317 May 3 16:33 mos2_bands.in -rw-r--r-- 1 userxxx groupyyy 54370 May 3 16:49 mos2_nscf.in -rw-r--r-- 1 userxxx groupyyy 2242 May 6 17:16 mos2_nscf.out -rw-r--r-- 1 userxxx groupyyy 1865 May 3 17:28 mos2_nscf.out -rw-r--r-- 1 userxxx groupyyy 1132 May 3 16:43 mos2_scf.in -rw-r--r-- 1 userxxx groupyyy 30626 May 6 17:16 mos2_scf.out -rw-r--r-- 1 userxxx groupyyy 30287 May 3 17:27 mos2_scf.out
In POSIX semantics, inside a directory structure, two files/directories with the same name should be not allowed. But when I "change" the ls command, this situation no longer appears. See:
# ls -la *out -rw-r--r-- 1 userxxx groupyyy 2242 May 6 17:16 mos2_nscf.out -rw-r--r-- 1 userxxx groupyyy 30626 May 6 17:16 mos2_scf.out
This way of running ls, files with the same name do not appear.
I currently have a new server in the cluster that is still in the backfill process. This situation could cause this problem. Could you help me identify the problem?
Thanks Rafael. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
-- ing. Sergio Rabellino
Università degli Studi di Torino Dipartimento di Informatica Tecnico di Ricerca Cel +39-342-529-5409 Tel +39-011-670-6701 Fax +39-011-751603 C.so Svizzera , 185 - 10149 - Torino
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hey Rafael, So these client mounts probably use the kernel mounts with an older kernel version that gets identified as the luminous client. You can try upgrading the kernel version and remounting. Laimis J.
On 8 Apr 2025, at 16:13, quaglio@bol.com.br wrote:
These 2 IPs are from the storage servers. There are no user processes running on them. It only has the operating system and ceph installed.
Rafael.
De: "Eugen Block" <eblock@nde.ag> Enviada: 2025/04/08 09:35:35 Para: quaglio@bol.com.br Cc: ceph-users@ceph.io Assunto: Re: [ceph-users] Ceph squid fresh install
These are your two Luminous clients:
---snip--- { "name": "unknown.0", "entity_name": "client.admin", "addrs": { "addrvec": [ { "type": "none", "addr": "172.27.254.7:0", "nonce": 443842330 } ] }, "socket_addr": { "type": "none", "addr": "172.27.254.7:0", "nonce": 443842330 }, "con_type": "client", "con_features": 3387146417253690110, "con_features_hex": "2f018fb87aa4aafe", "con_features_release": "luminous", ...
{ "name": "client.104098", "entity_name": "client.admin", "addrs": { "addrvec": [ { "type": "v1", "addr": "172.27.254.6:0", "nonce": 2027668300 } ] }, "socket_addr": { "type": "v1", "addr": "172.27.254.6:0", "nonce": 2027668300 }, "con_type": "client", "con_features": 3387146417253690110, "con_features_hex": "2f018fb87aa4aafe", "con_features_release": "luminous", ---snip---
Zitat von quaglio@bol.com.br:
Hi Eugen! Thanks a lot! I was able to find luminous connections, but I still can't identify which client process. Here is the output: Rafael. ────────────────────────────────────────────────────────────────── De: "Eugen Block" Enviada: 2025/04/08 04:37:47 Para: ceph-users@ceph.io Assunto: [ceph-users] Re: Ceph squid fresh install Hi, you can query the MON sessions to identify your older clients with: ceph tell mon. sessions It will show you the IP address, con_features_release (Luminous) and a couple of other things. Zitat von Laura Flores : > Hi Rafael, >> I would not force the min_compat_client to be reef when there are still > luminous clients connected, as it is important for all clients to be >=Reef
to understand/encode the pg_upmap_primary feature in the osdmap. >> As for checking which processes are still luminous, I am copying @Radoslaw > Zarzynski who may be able to help more with that. >> Thanks, > Laura Flores >> On Mon, Apr 7, 2025 at 11:30 AM quaglio@bol.com.br > wrote:
Hi, >> I just did a new Ceph installation and would like to enable the "read >> balancer". >> However, the documentation requires that the minimum client version >> be reef. I checked this information through "ceph features" and came across >> the situation of having 2 luminous clients. >> # ceph features >> { >> "mon": [ >> { >> "features": "0x3f03cffffffdffff", "release": "squid", >> "num": 2 >> } >> ], >> "mds": [ >> { >> "features": "0x3f03cffffffdffff", >> "release": "squid", >> "num": 2 >> } ], >> "osd": [ >> { >> "features": "0x3f03cffffffdffff", >> "release": "squid", >> "num": 38 >> } >> ], >> "client": [ >> { >> "features": "0x2f018fb87aa4aafe", >> "release": "luminous", >> "num": 2 >> }, >> { >> "features": "0x3f03cffffffdffff", >> "release": "squid", >> "num": 5 >> } ], >> "mgr": [ >> { >> "features": "0x3f03cffffffdffff", >> "release": "squid", >> "num": 2 >> } >> ] >> } >>>> I tryed to configure the minimum version to reef and received the >> following alert: >> # ceph osd set-require-min-compat-client reef >> Error EPERM: cannot set require_min_compat_client to reef: 2 connected >> client(s) look like luminous (missing 0x80000000); add >> --yes-i-really-mean-it to do it anyway >>>> Is it ok do confirm anyway? >> Which processes are still as luminous? >>>> Rafael. >> _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io >> To unsubscribe send an email to ceph-users-leave@ceph.io >>>>>> -- >> Laura Flores >> She/Her/Hers Software Engineer, Ceph Storage >>> Chicago, IL >> lflores@ibm.com | lflores@redhat.com > M: +17087388804 > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (9)
-
Anthony D'Atri
-
Eugen Block
-
Laimis Juzeliūnas
-
Laura Flores
-
Massimo Sgaravatto
-
Peter Linder
-
quaglio＠bol.com.br
-
Robert Sander
-
Sergio Rabellino