- ceph-users - lists.ceph.io

Re: Object replication issue between zones for RGW Account owned buckets
by Ashok Kumar M 08 Jan '26

08 Jan '26

Hi, I’m currently using Ceph version 19.2.3 and have a setup with two zones (zone-a and zone-b) under a single zonegroup (zonegroup-a) and realm (realm-a). When I replicate objects from bucket-a (in zone-a) to bucket-b (in zone-b) for buckets owned by a regular user (ex: my-user), the replication works successfully. However, replication does not work for buckets owned by the RGW Account (ex: RGW84754594545284760). In this case: There are no errors shown in the sync status on either zone. The replication status of the objects in the source bucket remains in a “PENDING” state indefinitely. Has anyone else encountered a similar issue, or is there any known workaround for this? I would really appreciate any kind of advice or guidance regarding this. Thanks, Ashok M Disclaimer : The content of this email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to which they are addressed. If you have received this email in error, please notify the sender and remove the messages from your system. If you are not the named addressee, it is strictly forbidden for you to share, circulate, distribute or copy any part of this e-mail to any third party without the written consent of the sender. E-mail transmission cannot be guaranteed to be secured or error free as information could be intercepted, corrupted, lost, destroyed, arrive late, incomplete, or may contain viruses. Therefore, we do not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. The recipient should check this e-mail and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email."

1 1

Has anyone been able to use NFS in tentacle or it is really broken?
by Salatiel Filho 07 Jan '26

07 Jan '26

Hi everyone, has anyone been able to use NFS in tentacle? The few people I know using it in squid said it has broken after tentacle upgrade. I have tried to create a NFS from scratch in a newly deployed instance of tentacle and it just does not work: The ganesha nfs server will keep dying. I've been able to workaround that by fully disabling the protocol 3 in the ganesha template directly in the tentacle image and deploying this new image, but I was expecting more people talking about this if it is a general problem. Maybe NFS is not being used anymore ? I can see a couple tickets in the issue tracker, but with no reply at all. Atenciosamente/Kind regards, Salatiel

1 0

Untrimmed purged_snaps impacts osd startup, possibly oom killer
by Eugen Block 07 Jan '26

07 Jan '26

Hi, this is kind of a follow-up on a two years old thread [0] and I wanted to raise some awareness for the corresponding tracker [1]. Back then we managed to limit the impact on the mon store performance with some paxos configs, but now the OSDs are impacted as well when there are new ones created: Each newly created OSD process grows in RAM usage to around 140 GB within a few minutes, easily causing oom killers on hosts if multiple OSDs are created at once. The residual RAM usage drops to the memory target after it successfully booted. The reason are the purged_snaps that are loaded during OSD boot (snap_mapper.record_purged_snaps purged_snaps), two years ago the customer had more than 42 million purged_snap entries. I don't know how many there are today, I don't have access myself, but I'll try to get a current number. Anyway, the only way to safely create OSDs is one by one, maybe two at once depending on the host's RAM capacity. An automated (unattended) OSD deployment is currently not possible. Unfortunately, the new tool [2] doesn't seem to work as expected, at least in my test cluster it didn't have any impact on the number of purged_snaps in the mon store. That's why we haven't tried it on the customer cluster(s) yet. How do other operators/admins/users deal with this kind of scenario? Having many snapshots can't be a corner case, but I can't remember having read anything like this on the list(s). I'd appreciate any comments, although I'm aware that everybody is probably busy travelling to Cephalocon. ;-) Thanks! Eugen [0] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/ZEMGKBLMERE… [1] https://tracker.ceph.com/issues/64519 [2] https://github.com/ceph/ceph/pull/57548

1 1

Re: Network Card
by Anthony D'Atri 06 Jan '26

06 Jan '26

Others may have more specific experience, but FWIW having 100GE of any make puts you ahead of a lot of deployments. That said, I suspect that the Mellanox CX6 may be overkill unless you can leverage substantial offload. OCP lets you conserve scarce standard PCIe slots for other, future purposes, so I like to provision such mezzanine cards when possible. > On Jan 6, 2026, at 10:59 AM, ceph_elk via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi, > > is there any specific proposal/measurement about good/bad 100G network cards ? > > We plan Dell Servers with Broadcom 57508 Dual Port 100GbE QSFP56, OCP NIC 3.0 card but Nviadia is available too > NVIDIA ConnectX-6-Adapter, 2 Anschlüsse, 100 GbE QSFP56, OCP 3.0, mit Kabel > > Which one should be the ceph prefered one? > > Thanks in advance > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Network Card
by ceph_elk 06 Jan '26

06 Jan '26

Hi, is there any specific proposal/measurement about good/bad 100G network cards ? We plan Dell Servers with Broadcom 57508 Dual Port 100GbE QSFP56, OCP NIC 3.0 card but Nviadia is available too NVIDIA ConnectX-6-Adapter, 2 Anschlüsse, 100 GbE QSFP56, OCP 3.0, mit Kabel Which one should be the ceph prefered one? Thanks in advance

1 0

Re: standby MDSes ?
by Gregory Farnum 05 Jan '26

05 Jan '26

On Sun, Jan 4, 2026 at 1:48 AM lejeczek via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi guys. > > Is this ok & expected that standby MDS for a filesystem > looks as such: > > -> $ ceph fs status APKI > APKI - 4 clients > ==== > RANK STATE MDS ACTIVITY DNS > INOS DIRS CAPS > 0 active APKI.podster2.gsniot Reqs: 0 /s 8505 > 5516 143 2757 > POOL TYPE USED AVAIL > cephfs.APKI.meta metadata 671M 74.0G > cephfs.APKI.data data 16.1G 74.0G > STANDBY MDS > APKI.podster3.utwwpp > MONERO.podster3.gwtegj > MDS version: ceph version 20.1.1 > (dd9c546413d50a90668289255a256022ea21f0c0) tentacle (rc - > RelWithDebInfo) > > What puzzles me from that above is: MONERO.podster3.gwtegj > I have only two files systems - this is a small lab with > ceph - and both were created with "default" ceph cmd, > meaning that pools where created automatically by cmd. > Is there no special and exclusive relationship between file > system & MDS - MONERO.podster3 was created when file > system/volume MONERO was created, yet it is available to a > different file system? Yes, this is fine. The monitors can generally assign a standby MDS to any filesystem, and if you look at the overall "ceph fs status" view you will see the standby daemons are listed out separately from the active FSes for that reason. > (what also boggles my mind is "size" the two file systems > report, like they share something, but not relevant yet in > case I have something seriously dis-configured relating what > stat "status" showed) The size statistics you are seeing here are drawn from RADOS reporting, so unless you have FSes configured to use different OSDs (via different CRUSH rules for the relevant pools) or have set pool quotas, they are going to report the same available capacity. -Greg > > many thanks, L. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Ceohfs+S3
by ceph_elk 05 Jan '26

05 Jan '26

Hi folks, Please be so kind and tell me how I can provide a bunch of files via cephfs AND S3? I'll have a hunderts of folders with a couple of files which needs to be shared via S3 to different users one per folder Thanks in advance Elkman

1 0

Re: multi site divergence?
by Casey Bodley 05 Jan '26

05 Jan '26

hi Chris, if you create a new placement target for this fast storage, you'd have to provide pool mappings for it on both zones even if you only want one zone to use it. and you're right that sync policy can disable replication, but that just means both zones will have separate views of the bucket. nothing would prevent clients from uploading objects to the 'wrong' zone, or deleting the empty bucket on one zone even though it's populated on the other you might prefer to create a separate zonegroup/zone for these non-replicated buckets. buckets created in that zonegroup would still be visible to the original two zones, but http requests would redirect to the correct zonegroup/zone so there'd be no confusion about what zones contain its object data. the downside is that it means deploying additional radosgws for this new zonegroup/zone at one of the sites and teaching clients/applications how to create buckets there in https://github.com/ceph/ceph/pull/62772 i've been exploring a "zone-local bucket" feature that uses zone redirects without requiring a separate zonegroup/zone, but the boto3 sdk doesn't properly handle these redirects - and that apparently applies to existing cross-zonegroup redirects as well On Sun, Dec 21, 2025 at 9:11 PM Christopher Durham via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi, > I have a multi site configuration with two sites that works just find for S3 bucket replication in both directions for the sites. - Rocky 9 and Ceph 19.2.2. > The customer now wishes to add some additional higher speed storage on one side that does not get replicated for the buckets on that storage. > My thought is that this should be a separate pool. and use bucket sync policy to prevent buckets created on that pool from replicating. But as I understand it, the bucket itself must exist on both sides. What do I do to properly create that bucket on the side that will not have the data? Do I do something clever with placement groups to have the bucket created on existing storage on the one side, but have it created on the new storage pool on the side where the new storage will be? > Thanks for any information you all may have > -Chris > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: Problems with orchestrator
by Carsten Götze 05 Jan '26

05 Jan '26

Hello! Here is the snippet from cephadm.log: cephadm ['--image', 'quay.io/ceph/ceph@sha256:1228c3d05e45fbc068a8c33614e4409b6dac688bcc77369b06009b5830fa8d86', '--timeout', '895', '_orch', 'deploy', '--fsid', '9a7485f3-9530-11f0-8bb9-6c92cff2f5d0'] 2025-12-10 09:45:26,706 7e80250e7080 DEBUG Loaded deploy configuration: {'fsid': '9a7485f3-9530-11f0-8bb9-6c92cff2f5d0', 'name': 'nfs.rootfs.0.0.atlas.luxjmr', 'image': '', 'deploy_arguments': [], 'params': {}, 'meta': {'service_n ame': 'nfs.rootfs', 'ports': [], 'ip': None, 'deployed_by': ['quay.io/ceph/ceph@sha256:1228c3d05e45fbc068a8c33614e4409b6dac688bcc77369b06009b5830fa8d86'], 'rank': 0, 'rank_generation': 0, 'extra_container_args': None, 'extra_entry point_args': None}, 'config_blobs': {'pool': '.nfs', 'namespace': 'rootfs', 'userid': 'nfs.rootfs.0.0.atlas.luxjmr', 'extra_args': ['-N', 'NIV_EVENT'], 'files': {'ganesha.conf': '# This file is generated by cephadm.\nNFS_CORE_PARAM {\n Enable_NLM = false;\n Enable_RQUOTA = false;\n Protocols = 3, 4;\n mount_path_pseudo = true;\n Enable_UDP = false;\n NFS_Port = 2049;\n allow_set_io_flusher_fail = true;\n Monitoring_Port = 9587;\n}\n\nNFSv4 {\n Delegations = false;\n RecoveryBackend = "rados_cluster";\n Minor_Versions = 1, 2;\n Server_Scope = "9a7485f3-9530-11f0-8bb9-6c92cff2f5d0-rootfs";\n IdmapConf = "/etc/ganesha/idmap.conf";\n}\n\nRADOS_KV {\n UserId = "nfs.rootfs.0.0.atlas.luxjmr";\n nodeid = 0;\n pool = ".nfs";\n namespace = "rootfs";\n}\n\nRADOS_URLS {\n UserId = "nfs.rootfs.0.0.atlas.luxjmr";\n watch_url = "rados://.nfs/rootfs/conf-nfs.rootfs";\n}\n\nRGW {\n cluster = "ceph";\n name = "client.nfs.rootfs.0.0.atlas.luxjmr-rgw";\n}\n\n%url rados://.nfs/rootfs/conf-nfs.rootfs', 'idmap.conf': ''}, 'config': '# minimal ceph.conf for 9a7485f3-9530-11f0-8bb9-6c92cff2f5d0\n[global]\n\tfsid = 9a7485f3-9530-11f0-8bb9-6c92cff2f5d0\n\tmon_host = [v2:134.169.37.21:3300/0,v1:134.169.37.21:6789/0] [v2:134.169.37.17:3300/0,v1:134.169.37.17:6789/0] [v2:134.169.37.12:3300/0,v1:134.169.37.12:6789/0] [v2:134.169.37.15:3300/0,v1:134.169.37.15:6789/0] [v2:134.169.37.14:3300/0,v1:134.169.37.14:6789/0]\n', 'keyring': '[client.nfs.rootfs.0.0.atlas.luxjmr]\nkey = AQAlMzlpAZ3EMBAAXYArqFVcLYC0ND5DjamvWw==\n', 'rgw': {'cluster': 'ceph', 'user': 'nfs.rootfs.0.0.atlas.luxjmr-rgw', 'keyring': '[client.nfs.rootfs.0.0.atlas.luxjmr-rgw]\nkey = AQAmMzlpyOP1BxAARjK7MCMV/u1CHCuCnCeqXg==\n'}}} 2025-12-10 09:45:26,706 7e80250e7080 DEBUG Determined image: 'quay.io/ceph/ceph@sha256:1228c3d05e45fbc068a8c33614e4409b6dac688bcc77369b06009b5830fa8d86' 2025-12-10 09:45:26,735 7e80250e7080 INFO Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-9a7485f3-9530-11f0-8bb9-6c92cff2f5d0-nfs-rootfs-0-0-atlas-luxjmr 2025-12-10 09:45:26,735 7e80250e7080 INFO /usr/bin/docker: stdout 2025-12-10 09:45:26,735 7e80250e7080 INFO /usr/bin/docker: stderr Error response from daemon: No such container: ceph-9a7485f3-9530-11f0-8bb9-6c92cff2f5d0-nfs-rootfs-0-0-atlas-luxjmr 2025-12-10 09:45:26,748 7e80250e7080 INFO Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-9a7485f3-9530-11f0-8bb9-6c92cff2f5d0-nfs.rootfs.0.0.atlas.luxjmr 2025-12-10 09:45:26,748 7e80250e7080 INFO /usr/bin/docker: stdout 2025-12-10 09:45:26,748 7e80250e7080 INFO /usr/bin/docker: stderr Error response from daemon: No such container: ceph-9a7485f3-9530-11f0-8bb9-6c92cff2f5d0-nfs.rootfs.0.0.atlas.luxjmr 2025-12-10 09:45:26,748 7e80250e7080 INFO Deploy daemon nfs.rootfs.0.0.atlas.luxjmr ... 2025-12-10 09:45:26,925 7e80250e7080 INFO Verifying port 0.0.0.0:2049 ... 2025-12-10 09:45:26,927 7e80250e7080 INFO Creating ganesha config... 2025-12-10 09:45:26,927 7e80250e7080 INFO Write file: /var/lib/ceph/9a7485f3-9530-11f0-8bb9-6c92cff2f5d0/nfs.rootfs.0.0.atlas.luxjmr/etc/ganesha/ganesha.conf 2025-12-10 09:45:26,927 7e80250e7080 INFO Write file: /var/lib/ceph/9a7485f3-9530-11f0-8bb9-6c92cff2f5d0/nfs.rootfs.0.0.atlas.luxjmr/etc/ganesha/idmap.conf 2025-12-10 09:45:27,242 7e80250e7080 DEBUG Non-zero exit code 1 from systemctl reset-failed ceph-9a7485f3-9530-11f0-8bb9-6c92cff2f5d0(a)nfs.rootfs.0.0.atlas.luxjmr 2025-12-10 09:45:27,242 7e80250e7080 DEBUG systemctl: stderr Failed to reset failed state of unit ceph-9a7485f3-9530-11f0-8bb9-6c92cff2f5d0(a)nfs.rootfs.0.0.atlas.luxjmr.service: Unit ceph-9a7485f3-9530-11f0-8bb9-6c92cff2f5d0(a)nfs.rootfs.0.0.atlas.luxjmr.service not loaded. 2025-12-10 09:45:27,541 7e80250e7080 DEBUG systemctl: stderr Created symlink /etc/systemd/system/ceph-9a7485f3-9530-11f0-8bb9-6c92cff2f5d0.target.wants/ceph-9a7485f3-9530-11f0-8bb9-6c92cff2f5d0(a)nfs.rootfs.0.0.atlas.luxjmr.service → /etc/systemd/system/ceph-9a7485f3-9530-11f0-8bb9-6c92cff2f5d0@.service. 2025-12-10 09:45:27,572 7e80250e7080 DEBUG firewalld does not appear to be present 2025-12-10 09:45:27,573 7e80250e7080 DEBUG Not possible to enable service <nfs>. firewalld.service is not available 2025-12-10 09:45:27,573 7e80250e7080 DEBUG firewalld does not appear to be present 2025-12-10 09:45:27,573 7e80250e7080 DEBUG Not possible to open ports <[2049]>. firewalld.service is not available With best regards Carsten Götze ------------------------------------------------------------------ Carsten Goetze Computer Graphics tel: +49 531 391-2109 TU Braunschweig fax: +49 531 391-2103 Muehlenpfordtstr. 23 eMail: c.goetze(a)cg.cs.tu-bs.de D-38106 Braunschweig http://www.cg.cs.tu-bs.de/people/goetze > Am 03.01.2026 um 18:52 schrieb Salatiel Filho <salatiel.filho(a)gmail.com>: > > Does nfs deploy in ceph 20.2.0 requires firewalld? > Would you mind sharing some logs that complain about that ? > > Atenciosamente/Kind regards, > Salatiel > > On Tue, Dec 16, 2025 at 9:54 AM Carsten Götze via ceph-users > <ceph-users(a)ceph.io> wrote: >> >> Hi! >> >> I'm running a ceph cluster using cephadm and recently upgraded from squid to tentacle 20.2.0. >> Until recently everything worked fine until I started the nfs module. The nfs daemons were reported to be running but after some 10 minutes or so all of them were reported to be dead except one. nfs service on port 2049 was never provided on any of the nodes even while the daemons were supposed to be running. >> As i found out later, the nfs daemons were never started at all, because the setup process required a systemd-firewalld to be installed on the system which of course wasn't. >> After some headaches with the newly installed firewalld I decided to roll back, delete the firewalld and postpone the nfs deployment. >> I then tried to stop the nfs daemons with 'ceph orch daemon stop', which did nothing, even after waiting some 10 minutes. I had to reissue the command several times to make the reportedly dead nfs daemons vanish from the 'ceph orch ps' list. The one daemon that was reported to be still running however would only die after 'ceph orch daemon stop --force' and was in an 'error' state thereafter and could not be removed from the 'ceph orch ps' list by no means. So I decided to delete the managing nfs service from the 'ceph orch ls' list, in hope that it would also tear down the remaining nfs daemon. >> This obviously was a bad idea since, the service is now in the state of deleting. However it cannot be deleted, because there is still the one daemon in error state, which cannot be deleted because it was never running at all. >> As a last measure I forcefully removed the docker container on the node with the cephadm command, but even though there are no traces left of that nfs daemon, it is still listed when running 'ceph orch ps'. >> I also noticed that the 'ceph orch device ls' is out of sync with reality and 'ceph orch ps' is still listing osds that I've already shutdown and deleted. I therefore suspect, that the orchestrator has stopped collecting state information from the nodes. >> Is there a way to force the orchestrator to sync its state information with the nodes? >> Where do I find meaningful logs for the orchestrator? >> >> With best regards, >> Carsten Götze >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: standby MDSes ?
by Eugen Block 05 Jan '26

05 Jan '26

Hi, if you don't specifically assign an MDS to a CephFS, the standby daemons will be standby in general, so for all Filesystems. You can configure a FS affinity though [0]: > You may set mds_join_fs on the standby to enforce your preference: > $ ceph config set mds.b mds_join_fs cephfs So in your case, you set it like this: $ ceph config set mds.MONERO.podster3.gwtegj mds_join_fs MONERO Regarding the size you might need to provide more details, not sure what exactly you're referring to from your output. Regards, Eugen [0] https://docs.ceph.com/en/latest/cephfs/standby/#configuring-mds-file-system… Zitat von lejeczek via ceph-users <ceph-users(a)ceph.io>: > Hi guys. > > Is this ok & expected that standby MDS for a filesystem looks as such: > > -> $ ceph fs status APKI > APKI - 4 clients > ==== > RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS > 0 active APKI.podster2.gsniot Reqs: 0 /s 8505 5516 143 2757 > POOL TYPE USED AVAIL > cephfs.APKI.meta metadata 671M 74.0G > cephfs.APKI.data data 16.1G 74.0G > STANDBY MDS > APKI.podster3.utwwpp > MONERO.podster3.gwtegj > MDS version: ceph version 20.1.1 > (dd9c546413d50a90668289255a256022ea21f0c0) tentacle (rc - > RelWithDebInfo) > > What puzzles me from that above is: MONERO.podster3.gwtegj > I have only two files systems - this is a small lab with ceph - and > both were created with "default" ceph cmd, meaning that pools where > created automatically by cmd. > Is there no special and exclusive relationship between file system & > MDS - MONERO.podster3 was created when file system/volume MONERO was > created, yet it is available to a different file system? > > (what also boggles my mind is "size" the two file systems report, > like they share something, but not relevant yet in case I have > something seriously dis-configured relating what stat "status" showed) > > many thanks, L. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0