- ceph-users - lists.ceph.io

Untrimmed purged_snaps impacts osd startup, possibly oom killer
by Eugen Block 07 Jan '26

07 Jan '26

Hi, this is kind of a follow-up on a two years old thread [0] and I wanted to raise some awareness for the corresponding tracker [1]. Back then we managed to limit the impact on the mon store performance with some paxos configs, but now the OSDs are impacted as well when there are new ones created: Each newly created OSD process grows in RAM usage to around 140 GB within a few minutes, easily causing oom killers on hosts if multiple OSDs are created at once. The residual RAM usage drops to the memory target after it successfully booted. The reason are the purged_snaps that are loaded during OSD boot (snap_mapper.record_purged_snaps purged_snaps), two years ago the customer had more than 42 million purged_snap entries. I don't know how many there are today, I don't have access myself, but I'll try to get a current number. Anyway, the only way to safely create OSDs is one by one, maybe two at once depending on the host's RAM capacity. An automated (unattended) OSD deployment is currently not possible. Unfortunately, the new tool [2] doesn't seem to work as expected, at least in my test cluster it didn't have any impact on the number of purged_snaps in the mon store. That's why we haven't tried it on the customer cluster(s) yet. How do other operators/admins/users deal with this kind of scenario? Having many snapshots can't be a corner case, but I can't remember having read anything like this on the list(s). I'd appreciate any comments, although I'm aware that everybody is probably busy travelling to Cephalocon. ;-) Thanks! Eugen [0] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/ZEMGKBLMERE… [1] https://tracker.ceph.com/issues/64519 [2] https://github.com/ceph/ceph/pull/57548

1 1

Re: Network Card
by Anthony D'Atri 06 Jan '26

06 Jan '26

Others may have more specific experience, but FWIW having 100GE of any make puts you ahead of a lot of deployments. That said, I suspect that the Mellanox CX6 may be overkill unless you can leverage substantial offload. OCP lets you conserve scarce standard PCIe slots for other, future purposes, so I like to provision such mezzanine cards when possible. > On Jan 6, 2026, at 10:59 AM, ceph_elk via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi, > > is there any specific proposal/measurement about good/bad 100G network cards ? > > We plan Dell Servers with Broadcom 57508 Dual Port 100GbE QSFP56, OCP NIC 3.0 card but Nviadia is available too > NVIDIA ConnectX-6-Adapter, 2 Anschlüsse, 100 GbE QSFP56, OCP 3.0, mit Kabel > > Which one should be the ceph prefered one? > > Thanks in advance > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Network Card
by ceph_elk 06 Jan '26

06 Jan '26

Hi, is there any specific proposal/measurement about good/bad 100G network cards ? We plan Dell Servers with Broadcom 57508 Dual Port 100GbE QSFP56, OCP NIC 3.0 card but Nviadia is available too NVIDIA ConnectX-6-Adapter, 2 Anschlüsse, 100 GbE QSFP56, OCP 3.0, mit Kabel Which one should be the ceph prefered one? Thanks in advance

1 0

Re: standby MDSes ?
by Gregory Farnum 05 Jan '26

05 Jan '26

On Sun, Jan 4, 2026 at 1:48 AM lejeczek via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi guys. > > Is this ok & expected that standby MDS for a filesystem > looks as such: > > -> $ ceph fs status APKI > APKI - 4 clients > ==== > RANK STATE MDS ACTIVITY DNS > INOS DIRS CAPS > 0 active APKI.podster2.gsniot Reqs: 0 /s 8505 > 5516 143 2757 > POOL TYPE USED AVAIL > cephfs.APKI.meta metadata 671M 74.0G > cephfs.APKI.data data 16.1G 74.0G > STANDBY MDS > APKI.podster3.utwwpp > MONERO.podster3.gwtegj > MDS version: ceph version 20.1.1 > (dd9c546413d50a90668289255a256022ea21f0c0) tentacle (rc - > RelWithDebInfo) > > What puzzles me from that above is: MONERO.podster3.gwtegj > I have only two files systems - this is a small lab with > ceph - and both were created with "default" ceph cmd, > meaning that pools where created automatically by cmd. > Is there no special and exclusive relationship between file > system & MDS - MONERO.podster3 was created when file > system/volume MONERO was created, yet it is available to a > different file system? Yes, this is fine. The monitors can generally assign a standby MDS to any filesystem, and if you look at the overall "ceph fs status" view you will see the standby daemons are listed out separately from the active FSes for that reason. > (what also boggles my mind is "size" the two file systems > report, like they share something, but not relevant yet in > case I have something seriously dis-configured relating what > stat "status" showed) The size statistics you are seeing here are drawn from RADOS reporting, so unless you have FSes configured to use different OSDs (via different CRUSH rules for the relevant pools) or have set pool quotas, they are going to report the same available capacity. -Greg > > many thanks, L. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Ceohfs+S3
by ceph_elk 05 Jan '26

05 Jan '26

Hi folks, Please be so kind and tell me how I can provide a bunch of files via cephfs AND S3? I'll have a hunderts of folders with a couple of files which needs to be shared via S3 to different users one per folder Thanks in advance Elkman

1 0

Re: multi site divergence?
by Casey Bodley 05 Jan '26

05 Jan '26

hi Chris, if you create a new placement target for this fast storage, you'd have to provide pool mappings for it on both zones even if you only want one zone to use it. and you're right that sync policy can disable replication, but that just means both zones will have separate views of the bucket. nothing would prevent clients from uploading objects to the 'wrong' zone, or deleting the empty bucket on one zone even though it's populated on the other you might prefer to create a separate zonegroup/zone for these non-replicated buckets. buckets created in that zonegroup would still be visible to the original two zones, but http requests would redirect to the correct zonegroup/zone so there'd be no confusion about what zones contain its object data. the downside is that it means deploying additional radosgws for this new zonegroup/zone at one of the sites and teaching clients/applications how to create buckets there in https://github.com/ceph/ceph/pull/62772 i've been exploring a "zone-local bucket" feature that uses zone redirects without requiring a separate zonegroup/zone, but the boto3 sdk doesn't properly handle these redirects - and that apparently applies to existing cross-zonegroup redirects as well On Sun, Dec 21, 2025 at 9:11 PM Christopher Durham via ceph-users <ceph-users(a)ceph.io> wrote: > > Hi, > I have a multi site configuration with two sites that works just find for S3 bucket replication in both directions for the sites. - Rocky 9 and Ceph 19.2.2. > The customer now wishes to add some additional higher speed storage on one side that does not get replicated for the buckets on that storage. > My thought is that this should be a separate pool. and use bucket sync policy to prevent buckets created on that pool from replicating. But as I understand it, the bucket itself must exist on both sides. What do I do to properly create that bucket on the side that will not have the data? Do I do something clever with placement groups to have the bucket created on existing storage on the one side, but have it created on the new storage pool on the side where the new storage will be? > Thanks for any information you all may have > -Chris > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: Problems with orchestrator
by Carsten Götze 05 Jan '26

05 Jan '26

Hello! Here is the snippet from cephadm.log: cephadm ['--image', 'quay.io/ceph/ceph@sha256:1228c3d05e45fbc068a8c33614e4409b6dac688bcc77369b06009b5830fa8d86', '--timeout', '895', '_orch', 'deploy', '--fsid', '9a7485f3-9530-11f0-8bb9-6c92cff2f5d0'] 2025-12-10 09:45:26,706 7e80250e7080 DEBUG Loaded deploy configuration: {'fsid': '9a7485f3-9530-11f0-8bb9-6c92cff2f5d0', 'name': 'nfs.rootfs.0.0.atlas.luxjmr', 'image': '', 'deploy_arguments': [], 'params': {}, 'meta': {'service_n ame': 'nfs.rootfs', 'ports': [], 'ip': None, 'deployed_by': ['quay.io/ceph/ceph@sha256:1228c3d05e45fbc068a8c33614e4409b6dac688bcc77369b06009b5830fa8d86'], 'rank': 0, 'rank_generation': 0, 'extra_container_args': None, 'extra_entry point_args': None}, 'config_blobs': {'pool': '.nfs', 'namespace': 'rootfs', 'userid': 'nfs.rootfs.0.0.atlas.luxjmr', 'extra_args': ['-N', 'NIV_EVENT'], 'files': {'ganesha.conf': '# This file is generated by cephadm.\nNFS_CORE_PARAM {\n Enable_NLM = false;\n Enable_RQUOTA = false;\n Protocols = 3, 4;\n mount_path_pseudo = true;\n Enable_UDP = false;\n NFS_Port = 2049;\n allow_set_io_flusher_fail = true;\n Monitoring_Port = 9587;\n}\n\nNFSv4 {\n Delegations = false;\n RecoveryBackend = "rados_cluster";\n Minor_Versions = 1, 2;\n Server_Scope = "9a7485f3-9530-11f0-8bb9-6c92cff2f5d0-rootfs";\n IdmapConf = "/etc/ganesha/idmap.conf";\n}\n\nRADOS_KV {\n UserId = "nfs.rootfs.0.0.atlas.luxjmr";\n nodeid = 0;\n pool = ".nfs";\n namespace = "rootfs";\n}\n\nRADOS_URLS {\n UserId = "nfs.rootfs.0.0.atlas.luxjmr";\n watch_url = "rados://.nfs/rootfs/conf-nfs.rootfs";\n}\n\nRGW {\n cluster = "ceph";\n name = "client.nfs.rootfs.0.0.atlas.luxjmr-rgw";\n}\n\n%url rados://.nfs/rootfs/conf-nfs.rootfs', 'idmap.conf': ''}, 'config': '# minimal ceph.conf for 9a7485f3-9530-11f0-8bb9-6c92cff2f5d0\n[global]\n\tfsid = 9a7485f3-9530-11f0-8bb9-6c92cff2f5d0\n\tmon_host = [v2:134.169.37.21:3300/0,v1:134.169.37.21:6789/0] [v2:134.169.37.17:3300/0,v1:134.169.37.17:6789/0] [v2:134.169.37.12:3300/0,v1:134.169.37.12:6789/0] [v2:134.169.37.15:3300/0,v1:134.169.37.15:6789/0] [v2:134.169.37.14:3300/0,v1:134.169.37.14:6789/0]\n', 'keyring': '[client.nfs.rootfs.0.0.atlas.luxjmr]\nkey = AQAlMzlpAZ3EMBAAXYArqFVcLYC0ND5DjamvWw==\n', 'rgw': {'cluster': 'ceph', 'user': 'nfs.rootfs.0.0.atlas.luxjmr-rgw', 'keyring': '[client.nfs.rootfs.0.0.atlas.luxjmr-rgw]\nkey = AQAmMzlpyOP1BxAARjK7MCMV/u1CHCuCnCeqXg==\n'}}} 2025-12-10 09:45:26,706 7e80250e7080 DEBUG Determined image: 'quay.io/ceph/ceph@sha256:1228c3d05e45fbc068a8c33614e4409b6dac688bcc77369b06009b5830fa8d86' 2025-12-10 09:45:26,735 7e80250e7080 INFO Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-9a7485f3-9530-11f0-8bb9-6c92cff2f5d0-nfs-rootfs-0-0-atlas-luxjmr 2025-12-10 09:45:26,735 7e80250e7080 INFO /usr/bin/docker: stdout 2025-12-10 09:45:26,735 7e80250e7080 INFO /usr/bin/docker: stderr Error response from daemon: No such container: ceph-9a7485f3-9530-11f0-8bb9-6c92cff2f5d0-nfs-rootfs-0-0-atlas-luxjmr 2025-12-10 09:45:26,748 7e80250e7080 INFO Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-9a7485f3-9530-11f0-8bb9-6c92cff2f5d0-nfs.rootfs.0.0.atlas.luxjmr 2025-12-10 09:45:26,748 7e80250e7080 INFO /usr/bin/docker: stdout 2025-12-10 09:45:26,748 7e80250e7080 INFO /usr/bin/docker: stderr Error response from daemon: No such container: ceph-9a7485f3-9530-11f0-8bb9-6c92cff2f5d0-nfs.rootfs.0.0.atlas.luxjmr 2025-12-10 09:45:26,748 7e80250e7080 INFO Deploy daemon nfs.rootfs.0.0.atlas.luxjmr ... 2025-12-10 09:45:26,925 7e80250e7080 INFO Verifying port 0.0.0.0:2049 ... 2025-12-10 09:45:26,927 7e80250e7080 INFO Creating ganesha config... 2025-12-10 09:45:26,927 7e80250e7080 INFO Write file: /var/lib/ceph/9a7485f3-9530-11f0-8bb9-6c92cff2f5d0/nfs.rootfs.0.0.atlas.luxjmr/etc/ganesha/ganesha.conf 2025-12-10 09:45:26,927 7e80250e7080 INFO Write file: /var/lib/ceph/9a7485f3-9530-11f0-8bb9-6c92cff2f5d0/nfs.rootfs.0.0.atlas.luxjmr/etc/ganesha/idmap.conf 2025-12-10 09:45:27,242 7e80250e7080 DEBUG Non-zero exit code 1 from systemctl reset-failed ceph-9a7485f3-9530-11f0-8bb9-6c92cff2f5d0(a)nfs.rootfs.0.0.atlas.luxjmr 2025-12-10 09:45:27,242 7e80250e7080 DEBUG systemctl: stderr Failed to reset failed state of unit ceph-9a7485f3-9530-11f0-8bb9-6c92cff2f5d0(a)nfs.rootfs.0.0.atlas.luxjmr.service: Unit ceph-9a7485f3-9530-11f0-8bb9-6c92cff2f5d0(a)nfs.rootfs.0.0.atlas.luxjmr.service not loaded. 2025-12-10 09:45:27,541 7e80250e7080 DEBUG systemctl: stderr Created symlink /etc/systemd/system/ceph-9a7485f3-9530-11f0-8bb9-6c92cff2f5d0.target.wants/ceph-9a7485f3-9530-11f0-8bb9-6c92cff2f5d0(a)nfs.rootfs.0.0.atlas.luxjmr.service → /etc/systemd/system/ceph-9a7485f3-9530-11f0-8bb9-6c92cff2f5d0@.service. 2025-12-10 09:45:27,572 7e80250e7080 DEBUG firewalld does not appear to be present 2025-12-10 09:45:27,573 7e80250e7080 DEBUG Not possible to enable service <nfs>. firewalld.service is not available 2025-12-10 09:45:27,573 7e80250e7080 DEBUG firewalld does not appear to be present 2025-12-10 09:45:27,573 7e80250e7080 DEBUG Not possible to open ports <[2049]>. firewalld.service is not available With best regards Carsten Götze ------------------------------------------------------------------ Carsten Goetze Computer Graphics tel: +49 531 391-2109 TU Braunschweig fax: +49 531 391-2103 Muehlenpfordtstr. 23 eMail: c.goetze(a)cg.cs.tu-bs.de D-38106 Braunschweig http://www.cg.cs.tu-bs.de/people/goetze > Am 03.01.2026 um 18:52 schrieb Salatiel Filho <salatiel.filho(a)gmail.com>: > > Does nfs deploy in ceph 20.2.0 requires firewalld? > Would you mind sharing some logs that complain about that ? > > Atenciosamente/Kind regards, > Salatiel > > On Tue, Dec 16, 2025 at 9:54 AM Carsten Götze via ceph-users > <ceph-users(a)ceph.io> wrote: >> >> Hi! >> >> I'm running a ceph cluster using cephadm and recently upgraded from squid to tentacle 20.2.0. >> Until recently everything worked fine until I started the nfs module. The nfs daemons were reported to be running but after some 10 minutes or so all of them were reported to be dead except one. nfs service on port 2049 was never provided on any of the nodes even while the daemons were supposed to be running. >> As i found out later, the nfs daemons were never started at all, because the setup process required a systemd-firewalld to be installed on the system which of course wasn't. >> After some headaches with the newly installed firewalld I decided to roll back, delete the firewalld and postpone the nfs deployment. >> I then tried to stop the nfs daemons with 'ceph orch daemon stop', which did nothing, even after waiting some 10 minutes. I had to reissue the command several times to make the reportedly dead nfs daemons vanish from the 'ceph orch ps' list. The one daemon that was reported to be still running however would only die after 'ceph orch daemon stop --force' and was in an 'error' state thereafter and could not be removed from the 'ceph orch ps' list by no means. So I decided to delete the managing nfs service from the 'ceph orch ls' list, in hope that it would also tear down the remaining nfs daemon. >> This obviously was a bad idea since, the service is now in the state of deleting. However it cannot be deleted, because there is still the one daemon in error state, which cannot be deleted because it was never running at all. >> As a last measure I forcefully removed the docker container on the node with the cephadm command, but even though there are no traces left of that nfs daemon, it is still listed when running 'ceph orch ps'. >> I also noticed that the 'ceph orch device ls' is out of sync with reality and 'ceph orch ps' is still listing osds that I've already shutdown and deleted. I therefore suspect, that the orchestrator has stopped collecting state information from the nodes. >> Is there a way to force the orchestrator to sync its state information with the nodes? >> Where do I find meaningful logs for the orchestrator? >> >> With best regards, >> Carsten Götze >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: standby MDSes ?
by Eugen Block 05 Jan '26

05 Jan '26

Hi, if you don't specifically assign an MDS to a CephFS, the standby daemons will be standby in general, so for all Filesystems. You can configure a FS affinity though [0]: > You may set mds_join_fs on the standby to enforce your preference: > $ ceph config set mds.b mds_join_fs cephfs So in your case, you set it like this: $ ceph config set mds.MONERO.podster3.gwtegj mds_join_fs MONERO Regarding the size you might need to provide more details, not sure what exactly you're referring to from your output. Regards, Eugen [0] https://docs.ceph.com/en/latest/cephfs/standby/#configuring-mds-file-system… Zitat von lejeczek via ceph-users <ceph-users(a)ceph.io>: > Hi guys. > > Is this ok & expected that standby MDS for a filesystem looks as such: > > -> $ ceph fs status APKI > APKI - 4 clients > ==== > RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS > 0 active APKI.podster2.gsniot Reqs: 0 /s 8505 5516 143 2757 > POOL TYPE USED AVAIL > cephfs.APKI.meta metadata 671M 74.0G > cephfs.APKI.data data 16.1G 74.0G > STANDBY MDS > APKI.podster3.utwwpp > MONERO.podster3.gwtegj > MDS version: ceph version 20.1.1 > (dd9c546413d50a90668289255a256022ea21f0c0) tentacle (rc - > RelWithDebInfo) > > What puzzles me from that above is: MONERO.podster3.gwtegj > I have only two files systems - this is a small lab with ceph - and > both were created with "default" ceph cmd, meaning that pools where > created automatically by cmd. > Is there no special and exclusive relationship between file system & > MDS - MONERO.podster3 was created when file system/volume MONERO was > created, yet it is available to a different file system? > > (what also boggles my mind is "size" the two file systems report, > like they share something, but not relevant yet in case I have > something seriously dis-configured relating what stat "status" showed) > > many thanks, L. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Override nfs parameter in NFS_CORE_PARAM
by Salatiel Filho 04 Jan '26

04 Jan '26

Hello , everyone. Apparently there is something wrong in the default config for ganesha nfs in tentacle. The nfs server will never start. ( https://tracker.ceph.com/issues/74191 / https://tracker.ceph.com/issues/74307 ) I've been able to successfully make the ganesha-server start by editing the ganesha.conf.j2 inside the ceph base container and replacing the line Protocols = 3,4 with Protocols = 4 inside the NFS_CORE_PARAM block. Unfortunately this temporary workaround is hard to maintain, so i have decided to look for a way to override the value in the ceph way of doing overrides. I used ceph nfs cluster config set main -i nfs-override.yml to override that parameter. NFS_CORE_PARAM { Protocols = 4; } I can see that the config is set by # ceph nfs cluster config get main NFS_CORE_PARAM { Protocols = 4; } But even after redeploy, the defaults of Protocols = 3,4 is never overridden. Any suggestions on how to override this? Atenciosamente/Kind regards, Salatiel

1 0

standby MDSes ?
by lejeczek 04 Jan '26

04 Jan '26

Hi guys. Is this ok & expected that standby MDS for a filesystem looks as such: -> $ ceph fs status APKI APKI - 4 clients ==== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active APKI.podster2.gsniot Reqs: 0 /s 8505 5516 143 2757 POOL TYPE USED AVAIL cephfs.APKI.meta metadata 671M 74.0G cephfs.APKI.data data 16.1G 74.0G STANDBY MDS APKI.podster3.utwwpp MONERO.podster3.gwtegj MDS version: ceph version 20.1.1 (dd9c546413d50a90668289255a256022ea21f0c0) tentacle (rc - RelWithDebInfo) What puzzles me from that above is: MONERO.podster3.gwtegj I have only two files systems - this is a small lab with ceph - and both were created with "default" ceph cmd, meaning that pools where created automatically by cmd. Is there no special and exclusive relationship between file system & MDS - MONERO.podster3 was created when file system/volume MONERO was created, yet it is available to a different file system? (what also boggles my mind is "size" the two file systems report, like they share something, but not relevant yet in case I have something seriously dis-configured relating what stat "status" showed) many thanks, L.

1 0