December 2025 - ceph-users

could not find secret_id
by xiaowenhao111 22 Apr '26

22 Apr '26

2 1

cephadm without containers
by Adrian Sevcenco 10 Apr '26

10 Apr '26

Hi! Is there any possibility to use cephadm without the container based tooling? (so to use bare-metal, with the packages directly installed on the machine) If not, is there something technical that prevents it to happen or just that there is a policy that container based deployment is the only best way to do it therefore cephadm will only use it this way? I ask this in the context of finding out that ceph-ansible is deprecated and that it should not be used, but i still need to use ansible for management and i think (so without factual experience) that bare metal services would be easier to manage after doing by hand deployment with cephadm Thank you! Adrian

11 20

CephFS: no MDS does join the filesystem
by Robert Sander 24 Mar '26

24 Mar '26

Hi, we are having an issue at a customer site where a 3PB CephFS is in failed state. The cluster itself is unhealthy and awaits replacements disks: # ceph -s cluster: id: 28ca2bfa-d87e-11ed-83a3-1070fddda30f health: HEALTH_ERR 4 failed cephadm daemon(s) There are daemons running an older version of ceph 1 filesystem is degraded 1 filesystem is offline 1 mds daemon damaged 8 nearfull osd(s) Low space hindering backfill (add storage if this doesn't resolve itself): 46 pgs backfill_toofull Possible data damage: 4 pgs inconsistent Degraded data redundancy: 6427646/15858772167 objects degraded (0.041%), 8 pgs degraded, 8 pgs undersized 6 pool(s) nearfull (muted: OSDMAP_FLAGS OSD_SCRUB_ERRORS(2d) PG_NOT_DEEP_SCRUBBED PG_NOT_SCRUBBED) services: mon: 3 daemons, quorum sn01,sn03,sn02 (age 3w) mgr: sn03.crlpzh(active, since 33h), standbys: sn01.tegfya, sn02.mzvgcr mds: 18/19 daemons up, 1 standby osd: 181 osds: 174 up (since 4d), 172 in (since 4d); 206 remapped pgs flags nodeep-scrub data: volumes: 2/3 healthy, 1 recovering; 1 damaged pools: 12 pools, 3585 pgs objects: 1.93G objects, 1.3 PiB usage: 2.5 PiB used, 501 TiB / 3.0 PiB avail pgs: 6427646/15858772167 objects degraded (0.041%) 293845758/15858772167 objects misplaced (1.853%) 2844 active+clean 532 active+clean+scrubbing 147 active+remapped+backfill_wait 28 active+remapped+backfill_toofull 11 active+remapped+backfill_wait+backfill_toofull 10 active+remapped+backfilling 6 active+undersized+degraded+remapped+backfill_toofull 2 active+clean+inconsistent 1 active+clean+scrubbing+deep+inconsistent+repair 1 active+undersized+remapped+backfilling 1 active+undersized+degraded+remapped+backfilling 1 active+recovering+degraded+remapped 1 active+remapped+inconsistent+backfill_toofull io: recovery: 183 MiB/s, 312 objects/s The CephFS metadata pool is not affected by the inconsistent PGs. The MDSs have this line in their logfile: "Monitors have assigned me to become a standby." The filesystem is joinable: # ceph fs lsflags storage_cluster joinable allow_snaps allow_multimds_snaps refuse_client_session But no MDS joins: # ceph fs status storage_cluster - 0 clients =============== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 failed POOL TYPE USED AVAIL cephfs_metadata metadata 490G 12.9T cephfs_data data 970T 54.1T shared_data data 1351T 22.5T STANDBY MDS storage_cluster.sn04.cbvzzu MDS version: ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable) Why? Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

6 13

problem with mgr prometheus module
by Dario Graña 17 Mar '26

17 Mar '26

Hi all! I'm running ceph quincy 17.2.7 in a cluster. On monday I updated the OS to AlmaLinux 9.3 to 9.4, since then grafana shows "No Data" message in all ceph related fields but, for example, the nodes information is still fine (Host Detail Dashboard). I have redeployed the mgr service with cephadm, disabled and re-enabled mgr prometheus module , but nothing changed. Digging into the problem, I accessed the prometheus interface. When I access prometheus, and found this error[image: Screen Shot 2024-06-04 at 15.22.37.png] When I access the node shown as down, it reports 503 Service Unavailable No cached data available yet Traceback (most recent call last): File "/lib/python3.6/site-packages/cherrypy/_cprequest.py", line 638, in respond self._do_respond(path_info) File "/lib/python3.6/site-packages/cherrypy/_cprequest.py", line 697, in _do_respond response.body = self.handler() File "/lib/python3.6/site-packages/cherrypy/lib/encoding.py", line 219, in __call__ self.body = self.oldhandler(*args, **kwargs) File "/lib/python3.6/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__ return self.callable(*self.args, **self.kwargs) File "/usr/share/ceph/mgr/prometheus/module.py", line 1751, in metrics return self._metrics(_global_instance) File "/usr/share/ceph/mgr/prometheus/module.py", line 1762, in _metrics raise cherrypy.HTTPError(503, 'No cached data available yet') cherrypy._cperror.HTTPError: (503, 'No cached data available yet') I checked the mgr prometheus address and port [ceph: root@ceph-admin01 /]# ceph config get mgr mgr/prometheus/server_addr :: [ceph: root@ceph-admin01 /]# ceph config get mgr mgr/prometheus/server_port 9283 It seems to be ok. When I check the master manager node for the port, I found [root@ceph-hn01 ~]# netstat -natup | grep 9283 tcp6 0 0 :::9283 :::* LISTEN 2453/ceph-mgr tcp6 0 0 192.168.97.51:9283 192.168.97.60:36130 ESTABLISHED 2453/ceph-mgr I don't understand why it is showing as IPv6, the node doesn't have a dual stack. I also tried to use a newer version of the prometheus container image, the 1.6.0, but it keeps reporting the same, so I rolled it back to the original one. Has anyone experienced an issue like this? Where can I look for more information about it? Thanks in advance. Regards. -- Dario Graña PIC (Port d'Informació Científica) Campus UAB, Edificio D E-08193 Bellaterra, Barcelona http://www.pic.es Avis - Aviso - Legal Notice: http://legal.ifae.es

2 2

reef 18.2.8 QE validation status
by Yuri Weinstein 13 Feb '26

13 Feb '26

Note: this is an attempt to test and publish post EOL reef point release Details of this release are summarized here: https://tracker.ceph.com/issues/73906#note-1 Release Notes - TBD LRC upgrade - N/A distro => jammy focal centos9 windows bookworm Seeking approvals/reviews for: smoke - Laura approved? rados - Radek, Laura approved? Travis? Nizamudeen? Adam King approved? (rados is also running on build 2 == build 1 + https://github.com/ceph/ceph/pull/66056 per Igor's request) rgw - Adam E approved? fs - Venky approved? orch - Adam King approved? rbd - Ilya approved? krbd - Ilya approved? upgrade/pacific-x (reef) - can this be deprecated? Josh? Neha? (same as in 18.2.5) upgrade/quincy-x (reef) - Laura, Prashant please take a look. (same as in 18.2.5) powercycle - Brad approved? ceph-volume - see orch, Guillaume approved? Notably upgrades suites show same results as for 18.2.5 (tested in March 2025) To compare, see 18.2.5 tests https://tracker.ceph.com/issues/70563#note-1 TIA

4 5

Debian Packages for Trixie
by Andrew 09 Feb '26

09 Feb '26

Hello Team, I can see that there are official packages for Debian Old-Stable (Bookworm) available on the official site: https://download.ceph.com/debian-19.2.3/dists/ Does anyone know if there is an approximate (or likely) ETA for the Debian Packages to be released for current stable version (Trixie)? Many thanks, Andrew

6 6

Re: Object replication issue between zones for RGW Account owned buckets
by Ashok Kumar M 08 Jan '26

08 Jan '26

Hi, I’m currently using Ceph version 19.2.3 and have a setup with two zones (zone-a and zone-b) under a single zonegroup (zonegroup-a) and realm (realm-a). When I replicate objects from bucket-a (in zone-a) to bucket-b (in zone-b) for buckets owned by a regular user (ex: my-user), the replication works successfully. However, replication does not work for buckets owned by the RGW Account (ex: RGW84754594545284760). In this case: There are no errors shown in the sync status on either zone. The replication status of the objects in the source bucket remains in a “PENDING” state indefinitely. Has anyone else encountered a similar issue, or is there any known workaround for this? I would really appreciate any kind of advice or guidance regarding this. Thanks, Ashok M Disclaimer : The content of this email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to which they are addressed. If you have received this email in error, please notify the sender and remove the messages from your system. If you are not the named addressee, it is strictly forbidden for you to share, circulate, distribute or copy any part of this e-mail to any third party without the written consent of the sender. E-mail transmission cannot be guaranteed to be secured or error free as information could be intercepted, corrupted, lost, destroyed, arrive late, incomplete, or may contain viruses. Therefore, we do not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. The recipient should check this e-mail and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email."

1 1

Untrimmed purged_snaps impacts osd startup, possibly oom killer
by Eugen Block 07 Jan '26

07 Jan '26

Hi, this is kind of a follow-up on a two years old thread [0] and I wanted to raise some awareness for the corresponding tracker [1]. Back then we managed to limit the impact on the mon store performance with some paxos configs, but now the OSDs are impacted as well when there are new ones created: Each newly created OSD process grows in RAM usage to around 140 GB within a few minutes, easily causing oom killers on hosts if multiple OSDs are created at once. The residual RAM usage drops to the memory target after it successfully booted. The reason are the purged_snaps that are loaded during OSD boot (snap_mapper.record_purged_snaps purged_snaps), two years ago the customer had more than 42 million purged_snap entries. I don't know how many there are today, I don't have access myself, but I'll try to get a current number. Anyway, the only way to safely create OSDs is one by one, maybe two at once depending on the host's RAM capacity. An automated (unattended) OSD deployment is currently not possible. Unfortunately, the new tool [2] doesn't seem to work as expected, at least in my test cluster it didn't have any impact on the number of purged_snaps in the mon store. That's why we haven't tried it on the customer cluster(s) yet. How do other operators/admins/users deal with this kind of scenario? Having many snapshots can't be a corner case, but I can't remember having read anything like this on the list(s). I'd appreciate any comments, although I'm aware that everybody is probably busy travelling to Cephalocon. ;-) Thanks! Eugen [0] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/ZEMGKBLMERE… [1] https://tracker.ceph.com/issues/64519 [2] https://github.com/ceph/ceph/pull/57548

1 1

Re: CephFS performance issue with a certain application
by Mark Nelson 31 Dec '25

31 Dec '25

Hi Martin, How many active MDS processes are you running? Given you have so many sub-directories and they appear to be static, you may want to set up a new parallel test directory where you have all 65535 sub-directories round-robin pinned across multiple active MDSes and see how that compares to the existing configuration. You can achieve real gains this way, though multi-active MDSes may hit some bugs that aren't present in single-active MDS setups. Since you are not seeing obvious CPU load on any of the existing MDS processes, I would also be curious what a wallclock profile of the authoritative MDS for the parent directory would show. You can try mine if you'd like, it's available here: https://github.com/markhpc/uwpmp You'll need debug symbols installed. The libunwind backend is the most reliable so I would stick with that even though it is slower than libdw. Thanks, Mark On 12/31/25 2:50 AM, Martin Gerhard Loschwitz via ceph-users wrote: > Folks, > > i’m getting a bit desperate here debugging a performance issue on one of our CephFS clusters. Ceph 19.2.3, normal replication with three replicas, Ceph cluster is performance-optimized according to best pratices, CephFS is running with a standby-replay MDS. All storage devices are 15.6TB NVMes in recent Dell servers. Content structure of CephFS is a single directory with 65535 directories and 2000-4000 files in each one of these. We’re seeing massive performance issues with a certain application that does roughly the following: > > * Read a file from the CephFS > * Modify it > * Write it back to the MDS in a new path with changed content > > Average file size is somewhere in the ballpark of 200-300kb. > > The effect we are now seeing is this: When we have the above mentioned application write to CephFS using four threads, there already is little performance impact that is visible to the outside. When we increase that thread count to 40, all read and write operations associated with said filesystem become notably slower. Node interconnect is 10Gbit/s, we have done extensive tests using iperf and can guarantee that network bandwidth is not a concern on the physical layer. We have also tried to reproduce the problem using fio and with numerous different combinations of queue depths, numbers of jobs and block sizes, but to no avail. We can see fio writing reliably to the cluster with bandwidth rates of 1200-1400Mb/s and seemingly arbitrary numbers of parallelization, even when running from the same systems where the problematic application is running. > > We have also conducted extensive tests with debug logging in the MDS enabled, and we see that there is a notable delay for reads and particularly writes between the „set_trace_dist added snap head“ and „link_primary_inode“ steps, with the delay varying between being almost unnoticeable and 2-3 seconds. Even when running with four threads only. And even then we see numerous processes in the system trying to access CephFS being in D state, according to their stack in /proc waiting for an MDS operation to finish. > > We’ve examined the app in question to rule out that it does something strange with regards to stat() or some such, but that is not the case. Application is Java based, it uses nio to open the files when reading and writing. > > We’ve ruled out all the usual suspects, mds cache limit on the MDS is 64 Gigabytes, CPUs are AMD EPYCs and we do not see any notable load on the servers hosting the MDSes while encountering the delays. We also do not see the particular MDS process running at 100% CPU load on any core. And we do not see network congestion while the problem appears. > > I’m running out of ideas here on what to debug and what to look for. Any hints or tips would be greatly appreciated. Thank you very much in advance, and a happy new year to everyone! > > Best regards > Martin > -- Best Regards, Mark Nelson Head of R&D (USA) Clyso GmbH p: +49 89 21552391 12 a: Loristraße 8 | 80335 München | Germany w: https://clyso.com | e: mark.nelson(a)clyso.com We are hiring: https://www.clyso.com/jobs/

1 0

Re: Best upgrade path?
by Robert Sander 31 Dec '25

31 Dec '25

Hi Jan, On 12/29/25 09:28, Jan Marek via ceph-users wrote: > I have a ceph cluster in version 16.1.10 on Ubuntu not managed by > cephadm. > > I would like to upgrade it to 19.1.3 and to cephadm mode. > > Please, can you advise me the best upgrade path? IMHO the path would be to adapt the cluster to cephadm first and then let the orchestrator do the upgrades. https://docs.ceph.com/en/pacific/cephadm/adoption/ https://docs.ceph.com/en/pacific/cephadm/upgrade/ Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 220009 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

1 0

CephFS performance issue with a certain application
by Martin Gerhard Loschwitz 31 Dec '25

31 Dec '25

Folks, i’m getting a bit desperate here debugging a performance issue on one of our CephFS clusters. Ceph 19.2.3, normal replication with three replicas, Ceph cluster is performance-optimized according to best pratices, CephFS is running with a standby-replay MDS. All storage devices are 15.6TB NVMes in recent Dell servers. Content structure of CephFS is a single directory with 65535 directories and 2000-4000 files in each one of these. We’re seeing massive performance issues with a certain application that does roughly the following: * Read a file from the CephFS * Modify it * Write it back to the MDS in a new path with changed content Average file size is somewhere in the ballpark of 200-300kb. The effect we are now seeing is this: When we have the above mentioned application write to CephFS using four threads, there already is little performance impact that is visible to the outside. When we increase that thread count to 40, all read and write operations associated with said filesystem become notably slower. Node interconnect is 10Gbit/s, we have done extensive tests using iperf and can guarantee that network bandwidth is not a concern on the physical layer. We have also tried to reproduce the problem using fio and with numerous different combinations of queue depths, numbers of jobs and block sizes, but to no avail. We can see fio writing reliably to the cluster with bandwidth rates of 1200-1400Mb/s and seemingly arbitrary numbers of parallelization, even when running from the same systems where the problematic application is running. We have also conducted extensive tests with debug logging in the MDS enabled, and we see that there is a notable delay for reads and particularly writes between the „set_trace_dist added snap head“ and „link_primary_inode“ steps, with the delay varying between being almost unnoticeable and 2-3 seconds. Even when running with four threads only. And even then we see numerous processes in the system trying to access CephFS being in D state, according to their stack in /proc waiting for an MDS operation to finish. We’ve examined the app in question to rule out that it does something strange with regards to stat() or some such, but that is not the case. Application is Java based, it uses nio to open the files when reading and writing. We’ve ruled out all the usual suspects, mds cache limit on the MDS is 64 Gigabytes, CPUs are AMD EPYCs and we do not see any notable load on the servers hosting the MDSes while encountering the delays. We also do not see the particular MDS process running at 100% CPU load on any core. And we do not see network congestion while the problem appears. I’m running out of ideas here on what to debug and what to look for. Any hints or tips would be greatly appreciated. Thank you very much in advance, and a happy new year to everyone! Best regards Martin -- Martin Gerhard Loschwitz Geschäftsführer / CEO, True West IT Services GmbH Phone: +49 2433 5253130 Mobile: +49 176 61832178 Address: Schmiedegasse 24a, 41836 Hückelhoven, Germany Legal: HRB 21985, Amtsgericht Mönchengladbach VAT: DE363893844 True West IT Services GmbH is compliant with the GDPR regulation on data protection and privacy in the European Union and the European Economic Area. You can request the information on how we collect and process your private data according to the law by contacting the email sender.

1 0

Thank You, Ceph Community and Looking Ahead to 2026
by Anthony Middleton 30 Dec '25

30 Dec '25

Hello, Ceph friends! I hope the holiday season is treating you well and that you' re finding time to relax and recharge after a great year. I want to sincerely thank you for the warm welcome as I step into the role of Ceph Community Manager. It has been a privilege to engage with such a collaborative, thoughtful, and technically driven community, and I am grateful for the openness and support you have shown from day one. The Ceph Community celebrated many big wins this year, including the release of V 20. 2. 0 "Tentacle." Congratulations to everyone involved, from CSC members to other Ceph contributors. The related blog post has already reached over 3, 3,000 views on ceph. io, reflecting both the quality of the work and the community' s interest in sharing it. The Ceph Foundation appreciates all Ceph Community members who continue to invest time to improving documentation, publishing blogs, fixing bugs, and answering questions on Slack, mailing lists, and GitHub. These efforts lower barriers for new users and contributors and enhance usability for operators day to day. Thanks also to everyone who helped make Cephalocon 2025 in Vancouver a success, along with five outstanding Ceph Days events around the world in 2025: Ceph Days India, Silicon Valley, Seattle, London, and Berlin. These events do not happen without significant volunteer efforts, and your commitment continues to showcase the strength of the Ceph community worldwide. Looking ahead to 2026, I encourage everyone to stay involved and help us accomplish even more. Let' s work to expand Ceph Days into new regions, boost engagement on LinkedIn, Slack, our mailing lists, and Twitter, and find new ways to connect users and contributors. If you' re interested in becoming a Ceph Ambassador or want more information about contributing to Ceph code, I invite you to fill out the Ceph Contributors Intake Form: https://t. ly/contributors <https://t.ly/contributors>. Thank you again for everything you do to support Ceph and one another. I am excited about what we can achieve together in the year ahead. Best, -- Anthony Middleton Community Manager, Ceph Foundation community-manager(a)ceph.io LinkedIn <https://www.linkedin.com/company/ceph>| BlueSky <https://bsky.app/profile/ceph.io> | Twitter <https://x.com/Ceph>

1 0

Re: Achieve no loss-of-write with 2 node failure?
by Anthony D'Atri 30 Dec '25

30 Dec '25

> * The filesystem (BlueStore or whatever) or DBMS (RocksDB or > whatever) is required to issue "write buffer commit" commands > to the device whenever necessary to ensure metadata or data is > not lost by cache power loss and remains consistent. .. which is why it’s advisable to disable HDD volatile write cache. Yet, I’ve seen data lost when power is lost with my own eyes. When an SSD is too cheap to include a supercap, I’m sure not going to trust that it implements the above rigorously. PLP != DRAMless. > * _Some_ loss is inevitable: there can be data still in the > middle of being written if the loss happen during a "write to > device" operation and efficient filesystems can do pretty > large ones. I did a PDU test and 2/3 of the OSDs were corrupted. YMMV, not going to debate you, Jerry. >> Just don’t waste money on 3 DWPD SKUs or RAID HBAs. > > Usually but I have seen some cases where the DB/WAL SSDs really > should have been 3 DWPD, especially if the SSD is shared among > data-only OSDs, which is almost inevitable in case of servers > with many HDDs. 3DWPD and 1DWPD SSDs are usually the exact same hardware, with different overprovisioning. You can turn one into the other, or pick something in between, with software. That said, see above where you wrote about operational false economies, hassling with hybrid OSDs is right up there. >In those cases better to be safe than sorry. Offload SSDs require paying extra for universal drive bays or rear cages. The former are prone to fussy tri-mode HBAs, and the latter to too large an HDD to SSD ratio. Unless you go down the dead-end of SATA SSDs. IMHO many deployments are better off with monolithic HDD OSDs, using more available slots for more spindles, with separate index/metadata pools. ymmv.

1 0

Re: Achieve no loss-of-write with 2 node failure?
by pg＠ceph.list.sabi.co.UK 30 Dec '25

30 Dec '25

>> For Ceph they are really necessary for log/database drives >> and ideally also for Ceph and CephFS metadata but non-PLP >> SSDs can be used if they are many for data if the workload >> does not require lots of small data writes. > It’s not about IOPS, it's about not losing data in flight when > the host/rack/DC loses power. * The filesystem (BlueStore or whatever) or DBMS (RocksDB or whatever) is required to issue "write buffer commit" commands to the device whenever necessary to ensure metadata or data is not lost by cache power loss and remains consistent. * The main effect of PLP is to allow the device to ignore "write buffer commit" commands and do commits less often, and that is what improves IOPS (and write amplification in the case of SSDs). * _Some_ loss is inevitable: there can be data still in the middle of being written if the loss happen during a "write to device" operation and efficient filesystems can do pretty large ones. Note: as to the latter point some non-PLP SSDs still have minimal capacitor backup to be able to complete writes from the RAM write buffer to the flash chip. Those that do not are less desirable :-). > [...] 4 hosts is the prudent minimum for R3 pools. For EC, > k+m+1 at a minimum. [...] But the prudent minimum number of buckets is usually not a great idea because of the big percent of object replicas damaged by a bucket failure. I would prefer as a minimum twice the width of replication/stripes for buckets, especially for EC pools where recovery is particularly expensive. > I usually recommend k + m < 12. Above, say, 6+2 the space > savings by going wider diminish, and the write performance and > other factors often aren’t worth the increasingly thin delta. > [...] I’ve seen a cluster with dense toploaders full of > spinners require that weighting up a single 8TB SSD had to be > done over 4 weeks so as to not tank the S3 API. Good anecdote (I have seen several similar cases). So I think that 4+2 (only in desperate cases 6+2) is the widest that is cost-effective; wider may have cost that is lower up-front but the lifetime cost in terms of huge recovery times and the impact on client workload can be quite bad. > At least one commercial vendor considers a cluster that > requires > 8 hours to recover from the loss of a single OSD to > be unsupported. That is a good argument from that vendor. > If DC space is free, sure. Until it isn’t. [...] There are false economies and a major one is to accept a very high cost of "maintenance" operations like recovery and scrubbing as in the anecdote above. That said fiber infrastructure makes distance rather less important than in the past so data centres can be located in cheaper areas (also I am a fan of "colocation" data centres). Still High Frequency Trading people like to put their servers as near as possible to the physical location of exchange data centres to really minimize latency and space there is usually tight and expensive so high density servers and racks make more sense. > Just don’t waste money on 3 DWPD SKUs or RAID HBAs. Usually but I have seen some cases where the DB/WAL SSDs really should have been 3 DWPD, especially if the SSD is shared among data-only OSDs, which is almost inevitable in case of servers with many HDDs. In those cases better to be safe than sorry.

1 0

Re: Achieve no loss-of-write with 2 node failure?
by Anthony D'Atri 29 Dec '25

29 Dec '25

> For Ceph they are really necessary for log/database drives and > ideally also for Ceph and CephFS metadata but non-PLP SSDs can > be used if they are many for data if the workload does not > require lots of small data writes. After all one can use for > that also HDDs (as long as they are small so IOPS-per-TB are not > too low). It’s not about IOPS, it's about not losing data in flight when the host/rack/DC loses power. >> The tentative current shopping list is 24x 7.68TB Samsung >> PM893 or Kingston DC600M drives. > > I have got several dozen DC600M (in some Lustre setups) and they > seem very reliable with good speed, and the PM893 have a good > reputation even if I have never tried them. But it will be hard > to source them now. I have recently tried 7.68TB and 30.72TB > SSDs from a relatively new chinese brand "Memblaze" I’m wary of firmware availability. ymmv. > >> Could we just increase the number of PGs to avoid this? > > Yes, rather more than "usual" guidelines but that has downsides > too but I think less severe than having too large PGs. In 2025, I would agree. More memory use and more peering. > However too large PGs are less of a problem on SSDs as transfer rates > are much higher and they have lots of IOPS. Very large PGs are still a problem for the balancer. > The usual Ceph guidelines say something like 100-150 PGs per OSD To be clear, PG *replicas*, not PGs proper. So inn an R3 pool that means 3 * pg_num. For an EC 4+2 pool, it means 6 * pg_num. When you have a mix of multiple pools, the manual calculations become tricky, though one can use the venerable pgcalc to help with the math. > but I have unfortunately seen 18TB HDDs with around that much > and it was pretty bad. My guess is that the guideline was > applicable to HDDs in the 1TB-2TB range. The guidance was ~200 at one point, then it was retconned to 100 to help keep people from OOMing themselves. In 2025 and with BlueStore, not Filestore OSDs, my sense is that a target of 100 is too low, especially when interpreted by the PG autoscaler, with which that default is actually a max, and you’ll generally see an even lower number. For modern clusters (BlueStore, > 2TB OSDs) I suggest global advanced mon_max_pg_per_osd 600 global advanced mon_target_pg_per_osd 300 mgr advanced mgr/balancer/upmap_max_deviation 1 mgr basic target_max_misplaced_ratio 0.020000 with the PG autoscaler enabled for all pools. RGW index, RBD/CephFS metadata pools should have a BIAS of 4, and no OSDS should have a legacy reweight value other than 1.000000 With these non-default values, the PG autoscaler usually does okay. I’ve seen a cluster with defaults with like a dozen RBD pools. Each pool usually has a minimum pg_num of 32, so in a modestly-sized cluster, a dozen of those takes a significant bite out of the autoscaler’s budget, and its figurative hands are tied. A cluster with only a single, RBD pool (and yes the trivial .mgr pool) is easier for the autoscaler to deal with, though you’ll still usually end up with an inadequate PGS value (right column of `ceph osd df` aka the PG ratio is what we’re targeting). I’ve seen more than one cluster as low as ~20 PG replicas per OSD due to this, and one (/me puckers and shakes) with single digits. > The basic issue that as to aggregating Ceph can only do one level so one has to balance > too few PGs resulting in huge numbers of objects per PG and > amplified recovery traffic This is not wrong, you do effectively more in some cases data that doesn’t strictly need to be moved, it comes along for the ride. Very large PGs also complicate scrubs. cf. https://www.evernote.com/shard/s189/sh/3a5bde91-e94b-4dbb-adbe-9d8a1379f3d5… With a wildly inadequate PG count, each PG represents a significant fraction of the storage on each PG, and the balancer can’t do a very good job. Another, rough analogy is consider how smooth a wood board gets with 40 grit sandpaper vs 400 grit. Rough analogy, see what I did there? ;) As an aside, for posterity I must note that PGs do not have a fixed size, they represent a *fraction* of the total data stored in a pool. > while more PGs means larger per-pool > and per-OSD PG tables. Several thousand PGs per pool is rather > common nowadays and I suspect it can be more. There’s a default max of 64K as a safeguard. Who among us has never fat-fingered an extra zero or two? With very large clusters, this can AFAIK be raised. I’ve worked on clusters with ~3100 OSDs, and there are definite ones larger in the wild. The orchestrator has issues larger than ~3500 OSDs currently but that’s being worked on. >>> That is indeed a good suggestion: the fewer the drives per >>> server the better. Ideally just one drive per server :-). > > To explain a bit more: in an ideal world Ceph was designed to > run on something like a lot of NUCs each fronting a small HDD or > an SSD. Well, I wouldn’t necessarily say that, https://ceph.io/en/news/blog/2016/500-osd-ceph-cluster/ notwithstanding. For sure there are certain aspects and defaults that are holdovers from the days of 1TB spinners, 1GE networking, and less-evolved CRUSH code and tunables. Like the optional replication network, but I digress... I often recommend a minimum of 7 chassis so that 4+2 EC can be done without gymnastics, even if not fully populated with drives. Plan for enough CPU cores, though. RAM can be upgrade, so long as you plan for empty slots. If you fill all the slots with 16 GiB modules and need to expand, you’re stuck. Back in the day the Sun 4/110 could be ordered with a minimum of 8 MiB. They shipped 32x 256KB SIMMs, filling all the slots, and requiring tossing those useless modules in order to add more. So for SSDs, a 1U server with 10-15 capacity drives (depending on U.2, E1.S, E3.S) is a starting point, with M.2 or rear boot. For spinners, I personally suggest a max of 24 SATA slots. Sure you can get a 100 slot toploader, but beware the pitfalls of a cluster comprising only 3 of those, the weight your rack/floor can support, how far that monster sticks out into the aisle, potentially preventing doors from closing, if you can afford fancy-pants doors. If you can, that’s money better spent on getting SSDs instead ;) Modest drive counts also help avoid the need for a replication network or higher link speeds (100 GE+). > In theory one could put the OSD daemon code in firmware and have > OSD-protocol instead of SCSI-protocol HDDs or SSDs with an > Ethernet socket instead of a SAS/SATA/USB/NVME socket. See the above experiment. > The reason why many setups have many-OSD servers is because the > capacity/price curve has a (shallow) sweet spot at around a > couple dozen OSDs per server but often "penny wise is pound > foolish". Yep. Consider what happens when 100 OSDs are offline at once during a failure, and that’s fully 1/3 of your cluster. >> Would I actually see a major advantage going from 6 nodes to >> 8, from 8 to 12, or from 12 to 24? (Given 24 disks in each >> case.) Diminishing returns, but 8 gives you the ability to run EC with k+m <= 7, 12 gives you the ability to run 8+3. > I guess it is not too bad to have up to half a dozen OSDs per > server (depending on network bandwidth, RAM, CPU usage, size, > ...) but in your case 6 per server would mean just 4 servers > which seems fragile to me. 4 hosts is the prudent minimum for R3 pools. For EC, k+m+1 at a minimum. Now if you’re converging with compute, the tradeoffs are different. > The issue is particularly severe with EC "pools" (especially of > the EC profile is wide which is a bad idea anyhow in most cases) I usually recommend k + m < 12. Above, say, 6+2 the space savings by going wider diminish, and the write performance and other factors often aren’t worth the increasingly thin delta. https://docs.ceph.com/en/reef/rados/operations/erasure-code/#id2 > and less bad with 3-way replication as then having just 6 > buckets means that on average the loss of a bucket will trigger > recovery on "only" half of the objects/PGs and so on. Recovery > eats *a lot* of write IOPS (and secondarily network banwidth) so > it is less costly with SSDs (those with PLP in particular). I’ve seen a cluster with dense toploaders full of spinners require that weighting up a single 8TB SSD had to be done over 4 weeks so as to not tank the S3 API. Consider how long it takes to recover from the loss of a 30+ TB spinner, or a node with a hundred of them. During which time you have reduced redundancy and thus increased risk. For large spinners strongly consider m = 3 for this reason. At least one commercial vendor considers a cluster that requires > 8 hours to recover from the loss of a single OSD to be unsupported. https://www.youtube.com/watch?v=Wy3EetoIqng >> I hate to see new spend on SATA/SAS SSDs, since those >> interfaces are progressively disappearing from product >> roadmaps. I would take 4-6 NVMe nodes any day. > > Unfortunately many of us still have old SATA-only or SATA+SAS > servers and it is often better to have more low-density old > servers than fewer higher-density new servers (even if one can > afford the latter). If DC space is free, sure. Until it isn’t. You can also mix and match with Ceph, within reason. You can get a used R640 with a bunch of cores really inexpensively these days. The chassis might not outlive the SSDs, but the SSDs are the bulk of the cost. Just don’t waste money on 3 DWPD SKUs or RAID HBAs. > For the future the ideal for smaller organizations would be U.3 > servers so one can buy new U.3 NVME devices and recycle existing > SATA or SAS devices along them but given the current constraint > on RAM availability I guess getting more servers at all will be > rather hard in 2026. Universal slots, sure - but note that a U.2 / U.3 chassis slot doesn’t necessarily accept SATA, and SAS usually means paying for an annoying HBA. Since HDDs these days are mostly LFF, a universal slot also means suboptimal RU density. When you skip the RAID HBA expense, the cost of an enhanced BMC license you don’t really need, procure wisely, an all-NVMe server including drives can actually cost less. I’ve seen it with my own eyes.

1 0

Re: Achieve no loss-of-write with 2 node failure?
by pg＠ceph.list.sabi.co.UK 29 Dec '25

29 Dec '25

>>> On Tue, 16 Dec 2025 21:46:36 +0000, Anthony D'Atri via >>> ceph-users <ceph-users(a)ceph.io> said: > I hate to see new spend on SATA/SAS SSDs, since those > interfaces are progressively disappearing from product > roadmaps. I would take 4-6 NVMe nodes any day. Unfortunately many of us still have old SATA-only or SATA+SAS servers and it is often better to have more low-density old servers than fewer higher-density new servers (even if one can afford the latter). For the future the ideal for smaller organizations would be U.3 servers so one can buy new U.3 NVME devices and recycle existing SATA or SAS devices along them but given the current constraint on RAM availability I guess getting more servers at all will be rather hard in 2026. Besides there used to be the strange situation where (I guess because of the huge installed base of U.2 servers) U.2 models were somewhat more available (and cheaper) than U.3 ones or E3.S or E1.S ones but all this is moot again because of the RAM (etc.) availability issues.

1 0

Re: Achieve no loss-of-write with 2 node failure?
by pg＠ceph.list.sabi.co.UK 29 Dec '25

29 Dec '25

>> Note: larger HDDs have really low IOPS-per-TB; SSDs avoid that >> issue but cheap SSDs do not have PLP so write IOPS are much >> lower than read IOPS. > That is something I've seen mentioned a lot, so we've only got > PLP drives on the shopping list. For Ceph they are really necessary for log/database drives and ideally also for Ceph and CephFS metadata but non-PLP SSDs can be used if they are many for data if the workload does not require lots of small data writes. After all one can use for that also HDDs (as long as they are small so IOPS-per-TB are not too low). > The tentative current shopping list is 24x 7.68TB Samsung > PM893 or Kingston DC600M drives. I have got several dozen DC600M (in some Lustre setups) and they seem very reliable with good speed, and the PM893 have a good reputation even if I have never tried them. But it will be hard to source them now. I have recently tried 7.68TB and 30.72TB SSDs from a relatively new chinese brand "Memblaze" and they were pretty good and there seems to be still some availability of those outside the USA, they have just setup an european subsidiary. >> Whether the drive is SSD or HDD larger ones also usually mean >> large PGs which is not so good. With SSDs at least it is >> possible (and in some cases advisable) to split them into >> multiple OSDs though. > Could we just increase the number of PGs to avoid this? Yes, rather more than "usual" guidelines but that has downsides too but I think less severe than having too large PGs. However too large PGs are less of a problem on SSDs as transfer rates are much higher and they have lots of IOPS. The usual Ceph guidelines say something like 100-150 PGs per OSD but I have unfortunately seen 18TB HDDs with around that much and it was pretty bad. My guess is that the guideline was applicable to HDDs in the 1TB-2TB range. The basic issue that as to aggregating Ceph can only do one level so one has to balance too few PGs resulting in huge numbers of objects per PG and amplified recovery traffic while more PGs means larger per-pool and per-OSD PG tables. Several thousand PGs per pool is rather common nowadays and I suspect it can be more. >> That is indeed a good suggestion: the fewer the drives per >> server the better. Ideally just one drive per server :-). To explain a bit more: in an ideal world Ceph was designed to run on something like a lot of NUCs each fronting a small HDD or an SSD. The basic idea of Ceph was/should be to turn each HDD/SSD from a block storage device into an object storage device by fronting it with an OSD daemon and accessing it over ethernet. In theory one could put the OSD daemon code in firmware and have OSD-protocol instead of SCSI-protocol HDDs or SSDs with an Ethernet socket instead of a SAS/SATA/USB/NVME socket. The reason why many setups have many-OSD servers is because the capacity/price curve has a (shallow) sweet spot at around a couple dozen OSDs per server but often "penny wise is pound foolish". > Would I actually see a major advantage going from 6 nodes to > 8, from 8 to 12, or from 12 to 24? (Given 24 disks in each > case.) I guess it is not too bad to have up to half a dozen OSDs per server (depending on network bandwidth, RAM, CPU usage, size, ...) but in your case 6 per server would mean just 4 servers which seems fragile to me. More smaller OSDs and more servers increase parallelism and available banwidth which may or may not be required, but even if the parallelism and banwidth requirements are not high the main issue with too many OSDs per server (apart from resource usage) is that then there are too few "buckets" (if server == bucket) and almost any bucket loss triggers recovery/rebalance of almost every PG/object. The issue is particularly severe with EC "pools" (especially of the EC profile is wide which is a bad idea anyhow in most cases) and less bad with 3-way replication as then having just 6 buckets means that on average the loss of a bucket will trigger recovery on "only" half of the objects/PGs and so on. Recovery eats *a lot* of write IOPS (and secondarily network banwidth) so it is less costly with SSDs (those with PLP in particular).

1 0

Re: Cephadm role an its relation with Ceph relases
by John Mulligan 29 Dec '25

29 Dec '25

On Monday, December 29, 2025 10:22:13 AM Eastern Standard Time Iztok Gregori via ceph-users wrote: > On 29/12/25 14:57, Anthony D'Atri wrote: > > > > > > >> > >> > >> In my case it will be used to "adopt" the services, but after that what > >> is its intended usage? The "orchestrator" should be used to add/remove > >> new hosts and to upgrade the cluster. > > > > > Cephadm *is* the orchestrator. The nomenclature may seem misleading. > > There’s part that runs on each node as well as a Manager module, and CLI > > commands. > > Understand. My question was around the "cephadm" command, the one > documentation says that has to be installed on all the nodes to start > the "adoption" process. I'm not sure if this command is needed only for > that process or is an integral part of the cephadm/orchestrator. The cephadm system is split into two parts - a manager module and the cephadm command. The command is often called "the binary" or a little more verbosely "the cephadm binary". A system may often have two or more copies of the cephadm binary on a node. There's the copy that is installed by an administrator (typically) that is run for bootstrapping or adoption and for some administrative commands. Then there's a copy that is pushed to nodes from the mgr module. This copy of the binary is coupled to the mgr module (you can find cached copies under /var/lib/ ceph). I hope this short architectural description is helpful. :-) > > > >> I'm asking this question because my upgrade path consists in multiple > >> consecutive upgrades and I see that for the latest version of Quincy > >> (17.2.9) and for newer Ceph releases there is no cephadm "package" for > >> el8 > > > > > Check the compatibility matrix in the docs. > > > Mhmm... I suspect that I'm entering in some edge case here (end exiting > the subject of this thread). I have a cluster running Ceph Octopus > 15.2.17 on EL8 (almost all the nodes, some nodes are EL7 but can be > reinstalled/removed). My goal is to upgrade it AND to embrace cephadm as > orchestrator. My plan is/was: > > 1. Adopt cephadm (with octopus containers) > 2. Upgrade to Quincy. > 3. Upgrade to Squid. > > But now looking to the various compatibility matrices I'm start to worry > that this is not possible: > > - Podman on EL8 is version 4, but from the table > https://docs.ceph.com/en/squid/cephadm/compatibility/#cephadm-compatibility-> with-podman podman will not run with Octopus. > - There is no mention of EL8 in > https://docs.ceph.com/en/squid/start/os-recommendations/ > > I could skip the adoption with Octopus containers and go directly with > adoption+upgrade to Quincy. But I'll end up with in any case with most > hosts running EL8 which I cannot find in any matrices. > > So the best solution will be to upgrade all the machines to EL9 and > concurrently upgrade Ceph to Quincy. For the MON and MGR shouldn't be > too much of a problem (it should be possible to add a new monitor with > newer ceph release to an existing cluster), but how to reinstall the OSD > servers without data movement? > > In theory it should suffice to put the cluster in 'noout', reinstall the > host without touching the OSD disks and then recreate the OSDs. Is this > possible? > > Thank you for your time! > Iztok > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Cephadm role an its relation with Ceph relases
by Anthony D'Atri 29 Dec '25

29 Dec '25

>>> >>> In my case it will be used to "adopt" the services, but after that what is its intended usage? The "orchestrator" should be used to add/remove new hosts and to upgrade the cluster. >> Cephadm *is* the orchestrator. The nomenclature may seem misleading. There’s part that runs on each node as well as a Manager module, and CLI commands. > > Understand. My question was around the "cephadm" command, the one documentation says that has to be installed on all the nodes to start the "adoption" process. I'm not sure if this command is needed only for that process or is an integral part of the cephadm/orchestrator. Both. >>> I'm asking this question because my upgrade path consists in multiple consecutive upgrades and I see that for the latest version of Quincy (17.2.9) and for newer Ceph releases there is no cephadm "package" for el8 >> Check the compatibility matrix in the docs. > > Mhmm... I suspect that I'm entering in some edge case here (end exiting the subject of this thread). Edge cases keep this list spicy. > I have a cluster running Ceph Octopus 15.2.17 on EL8 (almost all the nodes, some nodes are EL7 but can be reinstalled/removed). Yep, you’ll want to bump those to EL8 as a first step. Depending on your setup you may be able to do that without having to recreate the OSDs. Any OSDs that are still Filestore or with bluestore_min_alloc_size > 4KiB will want to be redeployed at some point though, which given what I’ve been reading lately should probably be done on Pacific, Reef, or Quincy. Unless you have coarse-IU QLC. > My goal is to upgrade it AND to embrace cephadm as orchestrator. As well it should — it will take multiple steps to get there. > My plan is/was: > > 1. Adopt cephadm (with octopus containers) > 2. Upgrade to Quincy. > 3. Upgrade to Squid. > > But now looking to the various compatibility matrices I'm start to worry that this is not possible: > > - Podman on EL8 is version 4, but from the table https://docs.ceph.com/en/squid/cephadm/compatibility/#cephadm-compatibility… podman will not run with Octopus. I think updating to Pacific or Quincy before messing with containerization is the way to go. Pacific and Quincy are expected, vis-a-vis that table, to work with Podman 3+. Note this section in the Quincy release notes: Users should expect to see the el8 rpm subdirectory empty and the “dnf” commands are expected to fail with 17.2.9. They can choose to use 17.2.9 RPM packages for centos 8/el8 provided by CERN as a community member or stay at 17.2.7 following instructions from https://docs.ceph.com/en/latest/install/get-packages/#rhel, the ceph.repo file should point to https://download.ceph.com/rpm-17.2.7/el8 instead of https://download.ceph.com/rpm-quincy/el8 > - There is no mention of EL8 in https://docs.ceph.com/en/squid/start/os-recommendations/ You’ll want to be on EL9 before going to Squid. > I could skip the adoption with Octopus containers and go directly with adoption+upgrade to Quincy. I do recommend minimizing the changes you make in one step. Maybe something like: * Update EL7 hosts to EL8 * Update Ceph packages to Quincy * Update hosts to EL9 * Adopt OSDs * Update Ceph to Squid > But I'll end up with in any case with most hosts running EL8 which I cannot find in any matrices. EL8 is an ex-parrot. > So the best solution will be to upgrade all the machines to EL9 and concurrently upgrade Ceph to Quincy. See above re one thing at a time. Trust me on this. > For the MON and MGR shouldn't be too much of a problem (it should be possible to add a new monitor with newer ceph release to an existing cluster), but how to reinstall the OSD servers without data movement? This varies depending on your deployment, so I hesitate to make an authoritative statement. Depending on whether one has RHEL, CentOS*, and/or Rocky, the options for in-situ OS upgrades vary. Are you using LUKS-encrypted OSDs aka dmcrypt? If you can update to 8 then 9 in-situ, then the OSDs should just work. Worst case you manually activate them after the OS update. One node at a time. If you can’t update in-situ, or want to change your partitioning (get rid of swap, stop balkanizing the boot volume, etc), then you can have your OS provisioning system protect the OSD devices. You don’t have boot and OSDs on the same devices, I hope? Kickstart: ignoredisk, clearpart —drives, etc. Don’t assume that the drive enumeration and usage is the same on every system. Set the `noout` flag, reimage the OS, install the Ceph packages, reboot. The OSDs may come right up. If they don’t, try activating them. Lather, rinse, repeat. Unset noout. Worst case, wipe one system entirely and deploy new OSDs, unset noout, once recovery completes, move to the next. > In theory it should suffice to put the cluster in 'noout', reinstall the host without touching the OSD disks and then recreate the OSDs. Is this possible? I’ve done it, when circumstances required moving from Ubuntu Precise to RHEL 7 in-situ. It’s easier if you’re using ceph-volume than with ceph-disk, but possible either way. > > Thank you for your time! > Iztok > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Cephadm role an its relation with Ceph relases
by Anthony D'Atri 29 Dec '25

29 Dec '25

> > In my case it will be used to "adopt" the services, but after that what is its intended usage? The "orchestrator" should be used to add/remove new hosts and to upgrade the cluster. Cephadm *is* the orchestrator. The nomenclature may seem misleading. There’s part that runs on each node as well as a Manager module, and CLI commands. > > I'm asking this question because my upgrade path consists in multiple consecutive upgrades and I see that for the latest version of Quincy (17.2.9) and for newer Ceph releases there is no cephadm "package" for el8 Check the compatibility matrix in the docs. > (my cluster is 95% Rocky Linux 8). EL8 in general has been deprecated for a couple of years. Toolchain and infrastructure limitations limit the ability to support older OSes indefinitely. That package and container strategies are still both supported complicates this. > I think that for the adoption and the (concurrent?) update to Quincy the latest provided cephadm package for el8 (cephadm-17.2.7.el8) should be adequate (right?), I’ll have to defer to others on that. I’ve BCCd the question to the upgrade lead. > but what happens when I'll upgrade the cluster to squid/tentacle? There are no cephadm packages for that releases for EL8... All roads lead to EL9. A typical strategy is to land on a Ceph release supported by both the current and next major OS releases, update the OS, then proceed to the desired Ceph release. Ceph upgrades themselves mostly become trivial once you’re containerized.

2 1

Re: Best upgrade path?
by Eugen Block 29 Dec '25

29 Dec '25

Hi, the easiest way would be to convert the cluster to cephadm [0] and then upgrade it to Reef first, then to Squid or Tentacle. I hope you mean 19.2.3, the x.2.z release is the stable version. If you haven't done a conversion yet, I recommend to try it on a test cluster first, just to get familiar with the tools. Although I can't recall having any serious issues during conversion, I'd still advise to test it first. And make sure to read the upgrade notes for each version you're going to upgrade to. Regards, Eugen [0] https://docs.ceph.com/en/latest/cephadm/adoption/ Zitat von Jan Marek via ceph-users <ceph-users(a)ceph.io>: > Hello, > > I have a ceph cluster in version 16.1.10 on Ubuntu not managed by > cephadm. > > I would like to upgrade it to 19.1.3 and to cephadm mode. > > Please, can you advise me the best upgrade path? > > Thanks a lot. > > Sincerely > Jan Marek > -- > Ing. Jan Marek > správce IT > > Jihočeská univerzita v Českých Budějovicích > Centum informačních technologií > Branišovská 1645/31a, 370 05 České Budějovice > T/ +420 38 903 2080 > M/ +420 604 660 121 > E/ jmarek(a)jcu.cz > www.jcu.cz

1 0

Cephadm role an its relation with Ceph relases
by Iztok Gregori 29 Dec '25

29 Dec '25

Hi to all! I'm in the process to upgrading my Ceph cluster (from Octopus to Quincy, then to squid/tentacle) and, at the same time, to adopt the cluster with cephadm. I'm not sure to understand the role of cephadm and, more importantly, the relation between cephadm and Ceph versions. I understand that cephadm is used to: - bootstrap a new cluster - "prepare" a host (cheking and installing packages, as docker/podman, lvm2, ntp...) - adopt a "legacy" cluster In my case it will be used to "adopt" the services, but after that what is its intended usage? The "orchestrator" should be used to add/remove new hosts and to upgrade the cluster. I'm asking this question because my upgrade path consists in multiple consecutive upgrades and I see that for the latest version of Quincy (17.2.9) and for newer Ceph releases there is no cephadm "package" for el8 (my cluster is 95% Rocky Linux 8). I think that for the adoption and the (concurrent?) update to Quincy the latest provided cephadm package for el8 (cephadm-17.2.7.el8) should be adequate (right?), but what happens when I'll upgrade the cluster to squid/tentacle? There are no cephadm packages for that releases for EL8... My gut feeling is that it doesn't matter, I'll use cephadm to "adopt" the cluster then "orchestrator" to manage it. The new nodes will be (at least) EL9 and the older nodes will be reinstalled/phased out when needed. But I'm not sure about it, so I ask for your wisdom. Thank you for your attention! Iztok -- Iztok Gregori ICT Systems and Services Elettra - Sincrotrone Trieste S.C.p.A. Telephone: +39 040 3758948 http://www.elettra.eu

1 0

Best upgrade path?
by Jan Marek 29 Dec '25

29 Dec '25

Hello, I have a ceph cluster in version 16.1.10 on Ubuntu not managed by cephadm. I would like to upgrade it to 19.1.3 and to cephadm mode. Please, can you advise me the best upgrade path? Thanks a lot. Sincerely Jan Marek -- Ing. Jan Marek správce IT Jihočeská univerzita v Českých Budějovicích Centum informačních technologií Branišovská 1645/31a, 370 05 České Budějovice T/ +420 38 903 2080 M/ +420 604 660 121 E/ jmarek(a)jcu.cz www.jcu.cz

1 0

Re: Expanding cluster
by Anthony D'Atri 28 Dec '25

28 Dec '25

>> >> >>>> I strongly recommend running 5 mons. Given how you phrase the above it >>>> seems as though you might mean 3 dedicated mon nodes? If so I’d still >> run >>>> 5 mons, add the mon cephadm host label to two of the OSD nodes. >>>> >>> >>> Right now. only one mon is "active" and used all of the time. You would >>> still run 5 mons anyway? >> >> Are you conflating mons and mgrs? Mons form a quorum, you need more than >> half of them able to reach each other. >> > > No. So you’re running a total of 1 mon daemon? > >>> I checked bluestore_min_alloc_size and indeed there is a 4K value set for >>> all of them. Even on new ones. >> >> Groovy. You checked with “ceph osd metadata”? >> > > Yes. Should it be set to different size/value? No. The default used to be 64 KiB; in the Octopus / Pacific era code improvements were made that allowed the defaults to change to 4 KiB. This is baked into each OSD, which is why I asked how I did: you can inject or set in central config or ceph.conf a value different from how a given OSD was built, but it doesn’t actually take effect. This is a subtlety often missed, in fact just last week I worked with a vendor doc team to correct their docs.

2 1

Re: Expanding cluster
by Rok Jaklič 28 Dec '25

28 Dec '25

On Sun, Dec 28, 2025 at 5:01 PM Anthony D'Atri via ceph-users < ceph-users(a)ceph.io> wrote: > > > >> I strongly recommend running 5 mons. Given how you phrase the above it > >> seems as though you might mean 3 dedicated mon nodes? If so I’d still > run > >> 5 mons, add the mon cephadm host label to two of the OSD nodes. > >> > > > > Right now. only one mon is "active" and used all of the time. You would > > still run 5 mons anyway? > > Are you conflating mons and mgrs? Mons form a quorum, you need more than > half of them able to reach each other. > No. > > > >> Do you have your index pool on SSDs? How old are those existing OSDs? > If > >> they were deployed in the 6 TB spinner era, they may have the old > >> bluestore_min_alloc_size value baked in. Check `ceph osd metadata`, > see if > >> any are not 4KiB. If you have or will have a significant population of > >> small RGW objects, serially redeploying the old OSDs — especially if > >> they’re Filestore — would have advantages. > >> > > > > Index pool is on HDD. HDDs (mostly Seagate) are actually quite old > > At 6 TB that was kinda assumed ;) > > > > from 2016 and lately we are seeing a bigger failure rate, > > That’s very much to be expected. > However they were deployed around 2020, so they have little less working hours. > > > I checked bluestore_min_alloc_size and indeed there is a 4K value set for > > all of them. Even on new ones. > > Groovy. You checked with “ceph osd metadata”? > Yes. Should it be set to different size/value?

1 0

Re: Expanding cluster
by Anthony D'Atri 28 Dec '25

28 Dec '25

>> I strongly recommend running 5 mons. Given how you phrase the above it >> seems as though you might mean 3 dedicated mon nodes? If so I’d still run >> 5 mons, add the mon cephadm host label to two of the OSD nodes. >> > > Right now. only one mon is "active" and used all of the time. You would > still run 5 mons anyway? Are you conflating mons and mgrs? Mons form a quorum, you need more than half of them able to reach each other. Yes I would run 5 mons, to survive a double failure. At at least 3 mgrs. > > NVME is used only for OS. I was considering offloading WAL+DB to NVME, but > that would mean 3-5NVME per OSD/JBOD node and we do not have configuration > like that right now. Nothing stops you from adding those, though for many applications the benefit isn’t worth it. > >> Do you have your index pool on SSDs? How old are those existing OSDs? If >> they were deployed in the 6 TB spinner era, they may have the old >> bluestore_min_alloc_size value baked in. Check `ceph osd metadata`, see if >> any are not 4KiB. If you have or will have a significant population of >> small RGW objects, serially redeploying the old OSDs — especially if >> they’re Filestore — would have advantages. >> > > Index pool is on HDD. HDDs (mostly Seagate) are actually quite old At 6 TB that was kinda assumed ;) > from 2016 and lately we are seeing a bigger failure rate, That’s very much to be expected. > I checked bluestore_min_alloc_size and indeed there is a 4K value set for > all of them. Even on new ones. Groovy. You checked with “ceph osd metadata”? > > We are replacing failed disks on "same OSD id" with something like: > > # destroy failed osd > systemctl stop ceph-osd@{id} > ceph osd down osd.{id} > ceph osd destroy {id} --yes-i-really-mean-it > > # prepare new one on same osd id > ceph-volume lvm prepare --osd-id {id} --data /dev/sdX > ceph-volume lvm list > ceph-volume lvm activate {id} {osd fsid} > > Is this "a problem"? Not at all.

1 0

Re: Expanding cluster
by Anthony D'Atri 28 Dec '25

28 Dec '25

> we are planning to expand our ceph cluster (reef) from 8 OSD nodes and 3 > monitors to 12 OSD nodes and 3 monitors. I strongly recommend running 5 mons. Given how you phrase the above it seems as though you might mean 3 dedicated mon nodes? If so I’d still run 5 mons, add the mon cephadm host label to two of the OSD nodes. > Currently each OSD node has a JBOD > with 28 6TB HDD disks and we are using EC 3+2, since at first there were > just 5 OSD nodes. Situations like that are why I often suggest starting with at least 7 nodes, even if they’re half populated, that lets you do 4+2 EC without CRUSH gymnastics. > No NVMEs are used since we do not have enough of them on > one particular node. ? Do you mean as OSDs, or to offload WAL+DB? Either way isn’t an all or nothing proposition. > Bare metal, no containers I suggest considering migrating to cephadm. The orchestrator works pretty well these days. > we build binaries for Alma > Linux ourselves. We are using RGW S3 frontend and have on average around > 50MB of upload/write during the day with spikes to up to 700MB. > > It works well. Do you have your index pool on SSDs? How old are those existing OSDs? If they were deployed in the 6 TB spinner era, they may have the old bluestore_min_alloc_size value baked in. Check `ceph osd metadata`, see if any are not 4KiB. If you have or will have a significant population of small RGW objects, serially redeploying the old OSDs — especially if they’re Filestore — would have advantages. > New osd nodes will have JBOD with 30 12TB disks. Also no NVMEs for > OSDs, just for OS. > > We will add each new node to the cluster then adding new OSD one by one > (until one rebalances), then draining old OSDs one by one (until one > rebalances). Will be a slow process, but we do not want to overload > anything. That will work, but will be inefficient. I suggest instead the process described here: https://ceph.io/assets/pdfs/events/2024/ceph-days-nyc/Mastering%20Ceph%20Op… Basically, disable rebalancing, add all the OSDs. No data moves. upmap PGs to where they are currently, and let the balancer incrementally do the work. You can modulate that process in the usual fashion. > I am thinking of creating a new EC pool with 8+3 (leaving one node for > "backup") and migrating user per user to the new EC pool. Glad to see that you’re planning for >= k+m+1 failure domains, if you only had 11 hosts total, you would effectively strand a lot of the raw capacity on the new systems. Through Tentacle you can add a second RGW storage class on the new data pool and use LC policies to migrate the data transparently, and set the default storageclass for given users to the new one. In Tentacle, turn on EC optimizations on the EC pools before moving data, so that new writes will save you the prior zero-padding. With Umbrella, due out … sometime next year, it is planned to be able to do such migrations transparently at the RADOS level. AIUI the old pool will need to remain permanently, but it’ll be worth it.

2 1

Expanding cluster
by Rok Jaklič 27 Dec '25

27 Dec '25

Hi, we are planning to expand our ceph cluster (reef) from 8 OSD nodes and 3 monitors to 12 OSD nodes and 3 monitors. Currently each OSD node has a JBOD with 28 6TB HDD disks and we are using EC 3+2, since at first there were just 5 OSD nodes. No NVMEs are used since we do not have enough of them on one particular node. Bare metal, no containers, we build binaries for Alma Linux ourselves. We are using RGW S3 frontend and have on average around 50MB of upload/write during the day with spikes to up to 700MB. It works well. New osd nodes will have JBOD with 30 12TB disks. Also no NVMEs for OSDs, just for OS. We will add each new node to the cluster then adding new OSD one by one (until one rebalances), then draining old OSDs one by one (until one rebalances). Will be a slow process, but we do not want to overload anything. I am thinking of creating a new EC pool with 8+3 (leaving one node for "backup") and migrating user per user to the new EC pool. Has anyone had a similar experience? Any thoughts and advice appreciated. Kind regards, Rok

1 0

Re: Ceph system disk on non-raid drive
by Anthony D'Atri 24 Dec '25

24 Dec '25

> Just my 2 cents, please wait for a second opinion. I wouldn't want to give incorrect/incomplete advice. You must be new to the interwebs ;) > It's best practice to present Ceph the raw block devices on which you're going to deploy your OSDs. I think the OP was asking about the boot / OS volume, not OSDs > No (active) RAID controller in between, just the raw device. You can use a RAID controller See my history of posts about those things. Nothing but trouble and expense. Sometimes a pure-NVMe system without any HBA costs *less* than one with a RAID HBA. Almost nobody monitors RAID HBAs properly, or keeps the firmware updated. > but configure it so that the OS sees the RAW device, not a block device the raid controller itself presents to the OS. I think i't called IT-mode or hbamode. Integrated Target HBAs are all passthrough all the time. IR Integrated RAID are capable of virtual drives. Sometimes one sees people flashing IT firmware on IR HBAs for better performance, though this practice is fraught. These days IR HBAs can usually pass-through non-VD drives so the OS sees them directly, which makes for improved observability and the ability to do wacky things like update firmware on them. In the past one would have to wrap every drive in degenerate VD just for the OS to see it. > Please someone correct me if I'm wrong but AFAIK, it doesn't really matter on which block devices you're running your OS, including where the Ceph binaries are located and/or /var/lib/ceph. Just make sure it's reliable storage and sufficiently fast like you would spec your regular servers. Sharing boot with data is in general not a great strategy. Boot drives are not designed for an OSD duty cycle, and I’ve seen them cause Ceph daemon impact when too much other crap is presented to them. Separate boot drives also make it a lot easier to reimage the OS without losing OSDs. > Also make sure to choose suitable block devices for OSDs! HDDs are definitely possible and used in Ceph clusters but result in a relatively slow cluster unless you've got a massive amount of them. We lose money with every sale but make up for it in volume! > Eg, if you want to use it for archival purposes, and performance doesn't really matter, it might work. At scale, including things like DC racks, coarse-IU QLC can actually cost less than HDDs, and less risky because it doesn’t take a month or more to recover from a failure. > For SSDs, definitely go enterprise class with PLP! Don't bargain on PLP and don't go for consumer class SSDs. Its a big big no-no you definitely want to avoid! Absolutely. I know of a cluster with the CephFS metadata pool on Sabrents. Ticking bomb. > > And in case you skipped the paragraph above: Don't compromise on PLP and go for Enterprise class SSDs, pretty please 😉🙂. > > And while thinking about SSDs, our Ceph support partner once told me there are certain SSDs that are known to cause problems. I don't know the specifics of it, but once you're speccing your cluster, I think it's a good idea to post your proposal here or in the Slack channel for review. Who knows, someone might possibly chime in with an actual experience with your hardware. We have to be careful when discussing specific companies, but most SSD failures are firmware and can be addressed in-situ with an upgrade. I’ve been through this myself. > > Tentacle has just been released, while it's a stable release you might not want to go Tentacle just yet until there's at least one point release. > > There's Squid, but you might want to be careful and disable bluestore_elastic_shared_blobs before you start deploying OSDs especially if you're looking to use Erasure Coded pools rather than replicated pools. … or deploy on the latest Reef then upgrade. > > I discussed about some practices for ceph node hardware configuration and > specifically about system disk on non-RAID drive (e.g. internal m2). It’s a strategy. > > Failure domain is host or rack, nvme cluster. > > Is it OK or is it one of possible and "adopted" way to deploy ceph nodes > with system disk without RAID1 (hw or sw)? Do you have some experience with > such nodes? Sure, if you like. I would shy away from doing that on smaller clusters, where the loss of a whole failure domain means, say, 5% or more of cluster capacity. If you have 250 nodes in your cluster, then losing one isn’t nearly so big a deal. HBA RAID is IMHO mostly an artifact of the days of a graphical desktop OS mistaken for a server OS. I would much rather use MD; ymmv.

1 0

Re: Ceph system disk on non-raid drive
by Wannes Smet 24 Dec '25

24 Dec '25

Hi, Just my 2 cents, please wait for a second opinion. I wouldn't want to give incorrect/incomplete advice. It's best practice to present Ceph the raw block devices on which you're going to deploy your OSDs. No (active) RAID controller in between, just the raw device. You can use a RAID controller but configure it so that the OS sees the RAW device, not a block device the raid controller itself presents to the OS. I think i't called IT-mode or hbamode. Please someone correct me if I'm wrong but AFAIK, it doesn't really matter on which block devices you're running your OS, including where the Ceph binaries are located and/or /var/lib/ceph. Just make sure it's reliable storage and sufficiently fast like you would spec your regular servers. Also make sure to choose suitable block devices for OSDs! HDDs are definitely possible and used in Ceph clusters but result in a relatively slow cluster unless you've got a massive amount of them. Eg, if you want to use it for archival purposes, and performance doesn't really matter, it might work. For SSDs, definitely go enterprise class with PLP! Don't bargain on PLP and don't go for consumer class SSDs. Its a big big no-no you definitely want to avoid! And in case you skipped the paragraph above: Don't compromise on PLP and go for Enterprise class SSDs, pretty please 😉🙂. And while thinking about SSDs, our Ceph support partner once told me there are certain SSDs that are known to cause problems. I don't know the specifics of it, but once you're speccing your cluster, I think it's a good idea to post your proposal here or in the Slack channel for review. Who knows, someone might possibly chime in with an actual experience with your hardware. Tentacle has just been released, while it's a stable release you might not want to go Tentacle just yet until there's at least one point release. There's Squid, but you might want to be careful and disable bluestore_elastic_shared_blobs before you start deploying OSDs especially if you're looking to use Erasure Coded pools rather than replicated pools. I wasn't aware while deploying my cluster of the bluestore_elastic_shared_blobs bug. I did redeploy all my OSDs just to be on the safe side which took around a week to complete. And last but not least: Here is the official documentation which provides a good overview of what you might want for hardware: https://docs.ceph.com/en/tentacle/start/hardware-recommendations/ You'll probably pick up on some more dos and don'ts. Cheers and happy holidays! Wannes ________________________________ From: Nekopilot via ceph-users <ceph-users(a)ceph.io> Sent: Wednesday, December 24, 2025 11:12 To: ceph-users(a)ceph.io <ceph-users(a)ceph.io> Subject: [ceph-users] Ceph system disk on non-raid drive Dear ceph users I discussed about some practices for ceph node hardware configuration and specifically about system disk on non-RAID drive (e. g. internal m2). Failure domain is host or rack, nvme cluster. Is it OK or is it one of possible and ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Dear ceph users I discussed about some practices for ceph node hardware configuration and specifically about system disk on non-RAID drive (e.g. internal m2). Failure domain is host or rack, nvme cluster. Is it OK or is it one of possible and "adopted" way to deploy ceph nodes with system disk without RAID1 (hw or sw)? Do you have some experience with such nodes? _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Ceph system disk on non-raid drive
by Eugen Block 24 Dec '25

24 Dec '25

Hi, we switched to such a setup to have one more slot for OSDs per server when we needed to expand our cluster. We have autoinstallation profiles for all kinds of servers, if a node fails (os-wise) we’re able to deploy it within half an hour or so. So for us it works. Regards, Eugen Zitat von Nekopilot via ceph-users <ceph-users(a)ceph.io>: > Dear ceph users > > I discussed about some practices for ceph node hardware configuration and > specifically about system disk on non-RAID drive (e.g. internal m2). > > Failure domain is host or rack, nvme cluster. > > Is it OK or is it one of possible and "adopted" way to deploy ceph nodes > with system disk without RAID1 (hw or sw)? Do you have some experience with > such nodes? > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Ceph system disk on non-raid drive
by Nekopilot 24 Dec '25

24 Dec '25

Dear ceph users I discussed about some practices for ceph node hardware configuration and specifically about system disk on non-RAID drive (e.g. internal m2). Failure domain is host or rack, nvme cluster. Is it OK or is it one of possible and "adopted" way to deploy ceph nodes with system disk without RAID1 (hw or sw)? Do you have some experience with such nodes?

1 0

Re: Happy holidays to all the Cephalopods out there!
by Alex Gorbachev 23 Dec '25

23 Dec '25

Wishing you a Tentacool New Year! May all your problems be Luminous in hindsight. -- Alex Gorbachev ISS On Tue, Dec 23, 2025 at 4:44 AM Eugen Block via ceph-users < ceph-users(a)ceph.io> wrote: > And a happy new year, too! > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: Happy holidays to all the Cephalopods out there!
by Anthony D'Atri 23 Dec '25

23 Dec '25

May the tentacles be with you > On Dec 23, 2025, at 4:43 AM, Eugen Block via ceph-users <ceph-users(a)ceph.io> wrote: > > And a happy new year, too! > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Happy holidays to all the Cephalopods out there!
by Wannes Smet 23 Dec '25

23 Dec '25

Thank you 🙂 To all of you too!! Kr, Wannes ________________________________ From: Eugen Block via ceph-users <ceph-users(a)ceph.io> Sent: Tuesday, December 23, 2025 10:44 To: ceph-users(a)ceph.io <ceph-users(a)ceph.io> Subject: [ceph-users] Happy holidays to all the Cephalopods out there! And a happy new year, too! _______________________________________________ ceph-users mailing list -- ceph-users@ ceph. io To unsubscribe send an email to ceph-users-leave@ ceph. io ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd And a happy new year, too! _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Happy holidays to all the Cephalopods out there!
by Eugen Block 23 Dec '25

23 Dec '25

And a happy new year, too!

1 0

Re: Low number of inflight compare to op_wip
by Anthony D'Atri 22 Dec '25

22 Dec '25

> On Dec 22, 2025, at 6:47 AM, George Shuklin via ceph-users <ceph-users(a)ceph.io> wrote: > > On 12/16/25 3:37 PM, Anthony D'Atri via ceph-users wrote: >>> I'm tryting to debug low performance on nvme-based cluster. >> Which specific drives? > > INTEL SSDPF2KE032T1 P5520, excellent drive. We see people on the list using random client-class drives, so the drives in use are a first question to ask, because so many people don't understand the differences. I work with a community member who has his CephFS metadata pool on *Sabrent* client drives. Poster child for "flirting with disaster". > > Ceph accepts about 20 requests, proccess them slowly, and then write to devices (super fast), so device sits idle most of the time. OSD daemons are not CPU/memory constrained (~40 cores idle, plenty of memory), os it's just disparity between Ceph OSD speed and backend speed. For sure there are serializations in the PG and OSD code, which is one reason why a healthy PG ratio matters. Remember that NVMe SSDs can process lots of operations in parallel, too. Have you discussed this with Mark Nelson? > I remember, few years ago (on lesser hardware) I done benchmark for ceph using brd (block ram disk) and ceph was able to churn up to 10k IOPS per daemon. Nowdays, it can up to 20k, I think. Mark would know for sure. When doing such tests, be sure that your pool is size=1 so that you aren't constrained by replication ops. In practice, network latency and replication are often significant factors in speed. Also, is the subject system free from swap? Do you have a static osd_memory_target, or do you have the autotuner enabled? > Actually, it's a good question: what is the maxium iops a single OSD daemon can deliver with perfectly fast underlaying storage and negligible network? Mark territory. > >>> but I've noticed, that in-flight for drives is, basically, around 1. That means, that at a given time only one request is processed. This is match match for OSD count/3 /latency formula, and with one in-flight nvme is showing about 10% of it's specs (Intel, DC grade). >> Have you updated to the latest firmware with SST? > > Yep. Certified and the most up to date firmware. Good, with the latest SST? Or if you got the drives from Dell, DSU? I worked for the Solidigm product team for a year, some of the firmware updates are very important. > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Low number of inflight compare to op_wip
by Janne Johansson 22 Dec '25

22 Dec '25

> Actually, it's a good question: what is the maxium iops a single OSD > daemon can deliver with perfectly fast underlaying storage and > negligible network? If it's possible to run an OSD over some ramdisk device, this should be easy to test. Don't know if rocksdb+lvm+bluestore makes this harder, but on filestore it was certainly possible. -- May the most significant bit of your life be positive.

2 1

Re: Low number of inflight compare to op_wip
by George Shuklin 22 Dec '25

22 Dec '25

On 12/16/25 3:37 PM, Anthony D'Atri via ceph-users wrote: >> I'm tryting to debug low performance on nvme-based cluster. > Which specific drives? INTEL SSDPF2KE032T1 Per specs it should give about 300k IOPS. >> I have 24 NVME in 4 servers, plenty of CPU, cluster perfectly balanced, no scrubbing or replication atm, 1024 pg for 24 OSD/17TB. >> >> I expect to see some performance (~250k IOPS total, 50% r/w, few hundred volumes with capped by IOPS, pre-warmed). I see about half of it. >> >> I looked at drive utilization, it's about 70% (per atop) > That's meaningless for SSDs. Yes, I looked deeper, and I found that most of the time there is 0 or 1, with very rarely 2 inflight operations, and all of them are done (from nvme point of view) within 50-70µs. At the same time (under constant load from fio) I can see about 10-20 op_wip, so my current theory goes like this: Ceph accepts about 20 requests, proccess them slowly, and then write to devices (super fast), so device sits idle most of the time. OSD daemons are not CPU/memory constrained (~40 cores idle, plenty of memory), os it's just disparity between Ceph OSD speed and backend speed. I remember, few years ago (on lesser hardware) I done benchmark for ceph using brd (block ram disk) and ceph was able to churn up to 10k IOPS per daemon. Nowdays, it can up to 20k, I think. Actually, it's a good question: what is the maxium iops a single OSD daemon can deliver with perfectly fast underlaying storage and negligible network? >> but I've noticed, that in-flight for drives is, basically, around 1. That means, that at a given time only one request is processed. This is match match for OSD count/3 /latency formula, and with one in-flight nvme is showing about 10% of it's specs (Intel, DC grade). > Have you updated to the latest firmware with SST? Yep. Certified and the most up to date firmware.

1 0

Re: Upgrade Advice: Ceph Reef (18.2.7) to Squid (19.2.3) on Proxmox VMs
by Văn Trần Quốc 22 Dec '25

22 Dec '25

Hi everyone, I just want to say a big thank you to Eugen, Wannes, and Frédéric for sharing your experiences and logs. I really appreciate the quick and detailed help from the community. Best regards, Van Tran

1 0

multi site divergence?
by Christopher Durham 22 Dec '25

22 Dec '25

Hi, I have a multi site configuration with two sites that works just find for S3 bucket replication in both directions for the sites. - Rocky 9 and Ceph 19.2.2. The customer now wishes to add some additional higher speed storage on one side that does not get replicated for the buckets on that storage. My thought is that this should be a separate pool. and use bucket sync policy to prevent buckets created on that pool from replicating. But as I understand it, the bucket itself must exist on both sides. What do I do to properly create that bucket on the side that will not have the data? Do I do something clever with placement groups to have the bucket created on existing storage on the one side, but have it created on the new storage pool on the side where the new storage will be? Thanks for any information you all may have -Chris

1 0

S3 API support for cross-tenant in Tentacle
by Pierre-Olivier Quirion 19 Dec '25

19 Dec '25

Hello all, The Tentacle release document states that "S3 API support for cross-tenant names such as Bucket='tenant:bucketname' [will be removed]". Does it mean that there will be no way to share data directly with users if they are not in the tenant or the account, or does it mean that there will be another way to do it but with another syntax, and what about cross tenancy sharing with the swift api? Cheers, P_O

1 0

Re: Is there an efficient way to list all namespaces in a pool?
by Casey Bodley 19 Dec '25

19 Dec '25

On Fri, Dec 19, 2025 at 6:32 AM Florian Haas via ceph-users <ceph-users(a)ceph.io> wrote: > > Hello everyone, > > I've been wondering if there is a way of listing all namespaces in a > RADOS pool in a way that is both human- and machine-readable, and is > more efficient than this: > > rados -p <pool> ls --all -f json \ > | jq -r "[.[] | .namespace] | unique" > > There are of course several variations of this approach, but they all > seem to boil down to enumerating *all objects* and then picking out > their namespaces — which is, shall we say, impractical if a pool > contains millions of objects. > > Does a fundamentally different option exist? the osds don't maintain an index of namespace names, so i'm afraid you're stuck scanning all objects > > Cheers, > Florian > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Is there an efficient way to list all namespaces in a pool?
by Florian Haas 19 Dec '25

19 Dec '25

Hello everyone, I've been wondering if there is a way of listing all namespaces in a RADOS pool in a way that is both human- and machine-readable, and is more efficient than this: rados -p <pool> ls --all -f json \ | jq -r "[.[] | .namespace] | unique" There are of course several variations of this approach, but they all seem to boil down to enumerating *all objects* and then picking out their namespaces — which is, shall we say, impractical if a pool contains millions of objects. Does a fundamentally different option exist? Cheers, Florian

1 0

Announcing go-ceph v0.37.0
by Anoop C S 19 Dec '25

19 Dec '25

Hi, I'm happy to announce another release of the go-ceph API library. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.37.0 More details are available at the link above. The library includes bindings that aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. The library also includes additional APIs that can be used to administer cephfs, rbd, and rgw subsystems. There are already a few consumers of this library in the wild, including the ceph-csi project. Anoop C S.

1 0

Re: Upgrade Advice: Ceph Reef (18.2.7) to Squid (19.2.3) on Proxmox VMs
by Frédéric Nass 19 Dec '25

19 Dec '25

Hi Văn Trần, For msg30399.html <https://www.mail-archive.com/ceph-users@ceph.io/msg30399.html>, the root cause is unknown and likely specific to the user's environment. For msg31238.html <https://www.mail-archive.com/ceph-users@ceph.io/msg31238.html>, the impact is noticeable on clusters with a high density of OSDs per node. With 7 OSDs per node, you likely won't notice it. It's being investigated here [1]. For tracker #70390 <https://tracker.ceph.com/issues/70390>, it only impacts OSDs created in Squid, so you're safe. Just make sure to disable bluestore_elastic_shared_blobs with 'ceph config set osd bluestore_elastic_shared_blobs 0' right after the upgrade; this ensures that newly created or recreated OSDs will be safe as well. Regarding Squid critical bugs, I recommend reading this list, which we will update periodically [2]. Best regards, Frédéric. [1] https://tracker.ceph.com/issues/73107 [2] https://docs.clyso.com/docs/kb/known-bugs/squid/ -- Frédéric Nass Ceph Ambassador France | Senior Ceph Engineer @ CLYSO Check our online Config Diff Tool, it's great! https://analyzer.clyso.com/#/analyzer/config-diff https://clyso.com | frederic.nass(a)clyso.com Le jeu. 18 déc. 2025 à 08:08, Văn Trần Quốc via ceph-users < ceph-users(a)ceph.io> a écrit : > Hi Ceph Users, > > I am planning a major upgrade for our production cluster from Reef (18.2.7) > to Squid (19.2.3) and would like to seek advice regarding stability and > potential risks. > > Infrastructure Overview: > > + Deployment: Cephadm. > + Cluster Size: 7 Nodes total. > + Hardware/Virtualization: Each node is a Virtual Machine hosted on > Proxmox. > + OSD Layout: 7 OSDs total (7 OSD per node). > + Other Daemons: 5 MONs, 7 MDSs, 3 MGRs. Services include RGW, CephFS, and > Block Devices. > + Pool Type: Replicated. > > The Concern: We are currently running stable on Reef 18.2.7. However, I > have been following recent discussions on the mailing list and tracker > regarding critical failures when upgrading to Squid, specifically > concerning OSD crashes and data corruption. > I am particularly worried about the issues reported in these threads, where > users experienced failures during or after the upgrade: > + https://www.mail-archive.com/ceph-users@ceph.io/msg30399.html > + https://www.mail-archive.com/ceph-users@ceph.io/msg31238.html > + https://tracker.ceph.com/issues/70390 > > Given our deployment topology and the jump to a new major version (Squid), > I have a few questions: > + Stability & Success Stories: Is Ceph Squid 19.2.3 considered safe > regarding the OSD crash/corruption bugs mentioned in the links above? Has > anyone in the community successfully completed the upgrade from 18.2.7 to > 19.2.3 without issues? Confirmation of a clean upgrade path would be very > reassuring. > + Upgrade Path: Are there any known regressions or critical "gotchas" when > moving from 18.2.7 directly to 19.2.3 in a virtualized environment? > > Any experiences or warnings from those running Squid in similar > environments would be greatly appreciated. > > Thank you, > Van Tran > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: Upgrade Advice: Ceph Reef (18.2.7) to Squid (19.2.3) on Proxmox VMs
by Eugen Block 18 Dec '25

18 Dec '25

Just adding the list again. Văn Trần <vantran15122004(a)gmail.com> schrieb am Do. 18. Dez. 2025 um 12:41: > Hi Eugen, > > Thank you very much for the clarification. > > It is very reassuring to know that you are successfully running a mix of > Replicated and EC pools on Squid, and that disabling > bluestore_elastic_shared_blobs was the only necessary precaution. > > I will follow your lead and apply this flag as a safeguard before > proceeding with my upgrade and future expansion. > > Appreciate your time and help! > > Best regards, > Van Tran >

1 0

Re: Problems with orchestrator
by Redouane Kachach 18 Dec '25

18 Dec '25

Hi Cartsen, Plz find the answers inline: Best, Redo. On Wed, Dec 17, 2025 at 3:12 PM Carsten Götze via ceph-users < ceph-users(a)ceph.io> wrote: > Hi! > > Thank you for the pointers. > > Unfortunately restarting the active mgr daemon didn't help. > > However looking at cephadm.log gave me at least a clue what was going on. > It seems that installing and deinstalling systemd-firewalld has completely > messed up the firewall rules. And restarting one of the nodes has made > things even worse. The osd's on that node died after a few minutes. I had > to restart the osd's systemd services, only to see them tumbling down a few > minutes later. Only after I finally flushed all the firewall rules from > every single node things went back to "normal". However, orchestrator is > still out of sync with reality. > > So a few more questions: > Where does orchestrator store its state information? > [Redo] Orchestrator (cephadm in this case) use the mon-store to store the services/daemon status information. In order to get a "fresh" view of the services state you can run: > ceph orch ps --refresh > Is firewalld required for tentacle? I didn't need it for squid, but It > sure is now required for the nfs module.Do the other modules also use it to > control the firewall? > [Redo] Firewalld isn’t required for cephadm-managed services to run, but it’s an important operational security layer. Whether you enable it depends on your org/company security model and requirements. In many environments, host firewall rules (often combined with network segmentation) help ensure Ceph ports and services are only reachable from the intended networks and hosts, reducing the risk of accidental exposure and limiting blast radius if something is misconfigured or bcz some service is exposing exploited ports. With firewalld installed, would restarting the nodes help to restore the > firewall rules? > [Redo] Firewalld is an external service. It stores its configuration on the local host. Restarting the node will just get you back to the same status. If you want to reset/restore any configuration you have to use firewalld commands. > What ports have to be open in the firewall? So far I have: 3000, 3300, > 5666, 6789, 6800:7568, 8443, 9093, 9094, 9095, 9100. Anything else? > [Redo] When firewalld is enabled cephadm will automatically open the corresponding port for the service, you don't have to worry about opening the port manually. > Would it be safe to downgrade to squid again? > [Redo] No, unfortunately you can't downgrade the ceph version (bcz of several technical reasons). > > With best regards > Carsten Götze > > > Am 16.12.2025 um 14:10 schrieb Robert Sander via ceph-users < > ceph-users(a)ceph.io>: > > > > Hi, > > > > Am 16.12.25 um 2:03 PM schrieb Carsten Götze via ceph-users: > > > >> Is there a way to force the orchestrator to sync its state information > with the nodes? > > > > Try to restart the active MGR daemon with systemctl on its node. > > > >> Where do I find meaningful logs for the orchestrator? > > > > The journal of the active MGR contains cephadm output > > > > Regards > > -- > > Robert Sander > > Linux Consultant > > > > Heinlein Consulting GmbH > > Schwedter Str. 8/9b, 10119 Berlin > > > > https://www.heinlein-support.de > > > > Tel: +49 30 405051 - 0 > > Fax: +49 30 405051 - 19 > > > > Amtsgericht Berlin-Charlottenburg - HRB 220009 B > > Geschäftsführer: Peer Heinlein - Sitz: Berlin > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: Upgrade Advice: Ceph Reef (18.2.7) to Squid (19.2.3) on Proxmox VMs
by Eugen Block 18 Dec '25

18 Dec '25

Hi, we upgraded two months ago from 18.2.7 to 19.2.3 without any issues. We use RBD, RGW and CephFS. The only thing I made sure of was to disable bluestore_elastic_shared_blobs as Wannes already mentioned because we needed to deploy additional OSDs. Regards, Eugen Am Do., 18. Dez. 2025 um 09:07 Uhr schrieb Wannes Smet via ceph-users < ceph-users(a)ceph.io>: > I can't directly answer your question, but I'm currently redeploying all > my OSDs with `bluestore_elastic_shared_blobs` disabled. I don't know for > your situation, but you might want to investigate if that's a good idea > before upgrading from 18.2.7 to 19.2.3. > > AFAIK the issue has also only be reported on EC pools. Not on replicated > pools that I'm aware of. > > Wannes > ________________________________ > From: Văn Trần Quốc via ceph-users <ceph-users(a)ceph.io> > Sent: Thursday, December 18, 2025 08:09 > To: ceph-users(a)ceph.io <ceph-users(a)ceph.io> > Subject: [ceph-users] Upgrade Advice: Ceph Reef (18.2.7) to Squid (19.2.3) > on Proxmox VMs > > Hi Ceph Users, I am planning a major upgrade for our production cluster > from Reef (18. 2. 7) to Squid (19. 2. 3) and would like to seek advice > regarding stability and potential risks. Infrastructure Overview: + > Deployment: Cephadm. + Cluster Size: > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > > ZjQcmQRYFpfptBannerEnd > > Hi Ceph Users, > > I am planning a major upgrade for our production cluster from Reef (18.2.7) > to Squid (19.2.3) and would like to seek advice regarding stability and > potential risks. > > Infrastructure Overview: > > + Deployment: Cephadm. > + Cluster Size: 7 Nodes total. > + Hardware/Virtualization: Each node is a Virtual Machine hosted on > Proxmox. > + OSD Layout: 7 OSDs total (7 OSD per node). > + Other Daemons: 5 MONs, 7 MDSs, 3 MGRs. Services include RGW, CephFS, and > Block Devices. > + Pool Type: Replicated. > > The Concern: We are currently running stable on Reef 18.2.7. However, I > have been following recent discussions on the mailing list and tracker > regarding critical failures when upgrading to Squid, specifically > concerning OSD crashes and data corruption. > I am particularly worried about the issues reported in these threads, where > users experienced failures during or after the upgrade: > + > https://urldefense.com/v3/__https://www.mail-archive.com/ceph-users@ceph.io… > + > https://urldefense.com/v3/__https://www.mail-archive.com/ceph-users@ceph.io… > + > https://urldefense.com/v3/__https://tracker.ceph.com/issues/70390__;!!Ftrht… > > Given our deployment topology and the jump to a new major version (Squid), > I have a few questions: > + Stability & Success Stories: Is Ceph Squid 19.2.3 considered safe > regarding the OSD crash/corruption bugs mentioned in the links above? Has > anyone in the community successfully completed the upgrade from 18.2.7 to > 19.2.3 without issues? Confirmation of a clean upgrade path would be very > reassuring. > + Upgrade Path: Are there any known regressions or critical "gotchas" when > moving from 18.2.7 directly to 19.2.3 in a virtualized environment? > > Any experiences or warnings from those running Squid in similar > environments would be greatly appreciated. > > Thank you, > Van Tran > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: Upgrade Advice: Ceph Reef (18.2.7) to Squid (19.2.3) on Proxmox VMs
by Wannes Smet 18 Dec '25

18 Dec '25

I can't directly answer your question, but I'm currently redeploying all my OSDs with `bluestore_elastic_shared_blobs` disabled. I don't know for your situation, but you might want to investigate if that's a good idea before upgrading from 18.2.7 to 19.2.3. AFAIK the issue has also only be reported on EC pools. Not on replicated pools that I'm aware of. Wannes ________________________________ From: Văn Trần Quốc via ceph-users <ceph-users(a)ceph.io> Sent: Thursday, December 18, 2025 08:09 To: ceph-users(a)ceph.io <ceph-users(a)ceph.io> Subject: [ceph-users] Upgrade Advice: Ceph Reef (18.2.7) to Squid (19.2.3) on Proxmox VMs Hi Ceph Users, I am planning a major upgrade for our production cluster from Reef (18. 2. 7) to Squid (19. 2. 3) and would like to seek advice regarding stability and potential risks. Infrastructure Overview: + Deployment: Cephadm. + Cluster Size: ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi Ceph Users, I am planning a major upgrade for our production cluster from Reef (18.2.7) to Squid (19.2.3) and would like to seek advice regarding stability and potential risks. Infrastructure Overview: + Deployment: Cephadm. + Cluster Size: 7 Nodes total. + Hardware/Virtualization: Each node is a Virtual Machine hosted on Proxmox. + OSD Layout: 7 OSDs total (7 OSD per node). + Other Daemons: 5 MONs, 7 MDSs, 3 MGRs. Services include RGW, CephFS, and Block Devices. + Pool Type: Replicated. The Concern: We are currently running stable on Reef 18.2.7. However, I have been following recent discussions on the mailing list and tracker regarding critical failures when upgrading to Squid, specifically concerning OSD crashes and data corruption. I am particularly worried about the issues reported in these threads, where users experienced failures during or after the upgrade: + https://urldefense.com/v3/__https://www.mail-archive.com/ceph-users@ceph.io… + https://urldefense.com/v3/__https://www.mail-archive.com/ceph-users@ceph.io… + https://urldefense.com/v3/__https://tracker.ceph.com/issues/70390__;!!Ftrht… Given our deployment topology and the jump to a new major version (Squid), I have a few questions: + Stability & Success Stories: Is Ceph Squid 19.2.3 considered safe regarding the OSD crash/corruption bugs mentioned in the links above? Has anyone in the community successfully completed the upgrade from 18.2.7 to 19.2.3 without issues? Confirmation of a clean upgrade path would be very reassuring. + Upgrade Path: Are there any known regressions or critical "gotchas" when moving from 18.2.7 directly to 19.2.3 in a virtualized environment? Any experiences or warnings from those running Squid in similar environments would be greatly appreciated. Thank you, Van Tran _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Upgrade Advice: Ceph Reef (18.2.7) to Squid (19.2.3) on Proxmox VMs
by Văn Trần Quốc 18 Dec '25

18 Dec '25

Hi Ceph Users, I am planning a major upgrade for our production cluster from Reef (18.2.7) to Squid (19.2.3) and would like to seek advice regarding stability and potential risks. Infrastructure Overview: + Deployment: Cephadm. + Cluster Size: 7 Nodes total. + Hardware/Virtualization: Each node is a Virtual Machine hosted on Proxmox. + OSD Layout: 7 OSDs total (7 OSD per node). + Other Daemons: 5 MONs, 7 MDSs, 3 MGRs. Services include RGW, CephFS, and Block Devices. + Pool Type: Replicated. The Concern: We are currently running stable on Reef 18.2.7. However, I have been following recent discussions on the mailing list and tracker regarding critical failures when upgrading to Squid, specifically concerning OSD crashes and data corruption. I am particularly worried about the issues reported in these threads, where users experienced failures during or after the upgrade: + https://www.mail-archive.com/ceph-users@ceph.io/msg30399.html + https://www.mail-archive.com/ceph-users@ceph.io/msg31238.html + https://tracker.ceph.com/issues/70390 Given our deployment topology and the jump to a new major version (Squid), I have a few questions: + Stability & Success Stories: Is Ceph Squid 19.2.3 considered safe regarding the OSD crash/corruption bugs mentioned in the links above? Has anyone in the community successfully completed the upgrade from 18.2.7 to 19.2.3 without issues? Confirmation of a clean upgrade path would be very reassuring. + Upgrade Path: Are there any known regressions or critical "gotchas" when moving from 18.2.7 directly to 19.2.3 in a virtualized environment? Any experiences or warnings from those running Squid in similar environments would be greatly appreciated. Thank you, Van Tran

1 0

Re: Proposal to end support for some erasure code plugins
by Jeremy Austin 17 Dec '25

17 Dec '25

Jamie, I run a single cluster using CLAY, and have quietly and successfully for 6+ years. Initially, I was concerned about network bandwidth during recovery. I have slowly upgraded network speeds and now that is not a primary consideration. I've read the relevant blog post and conclusions, and am willing to switch to a recommend Fast EC-supported plugin and technique. That's one datum, I guess. /Jeremy On Tue, Dec 16, 2025 at 11:42 AM Jamie Pryde via ceph-users < ceph-users(a)ceph.io> wrote: > Hi all, > > I want to make you aware of the Fast EC team's proposal to end support for > some erasure code plugins and techniques. > > We want to do this for a few reasons: > 1. Continuing to support a large list of plugins and techniques requires > development and test effort. We don't think this effort is valuable because > 2. Telemetry data shows that most Ceph clusters that are using EC are > using Jerasure reed_sol_van or ISA-L reed_sol_van, and > 3. Benchmarking shows that reed_sol_van has very good performance on > modern CPUs. > > We are proposing that the following list of plugins and techniques should > be deprecated in the Umbrella release, and support be removed in the V > release: > > * > Jerasure (reed_sol_r6_op, cauchy_orig, cauchy_good, liberation, > blaum_roth, liber8tion) > * > SHEC > * > CLAY > > The following list of plugins and techniques would continue to be > supported in the V release: > > * > Jerasure (reed_sol_van) > * > ISA-L (reed_sol_van, cauchy) > * > LRC (Although Fast EC doesn’t currently support LRC and we wouldn't > recommend using LRC yet, we think we will be able to use it in future to > improve support for erasure coded pools in stretched clusters.) > > I have written a post for the ceph.io blog which contains more details > and shows some of the data behind the proposal. It also talks about the > pool migration feature we are planning to add in Umbrella. This can be used > to migrate data from a pool to a new EC pool that uses a supported > plugin/technique. You can find the blog PR here: > https://github.com/ceph/ceph.io/pull/988 > > We are interested to hear if anyone has any objections to this plan. > Please reply to this mail or leave a comment on the blog PR if you do have > any objections or concerns. We plan to publish the blog post sometime > mid-January. > > > Thanks, > > > > -- > > > > Jamie Pryde > > Software Engineer > > IBM UK > > Unless otherwise stated above: > > IBM United Kingdom Limited > Registered in England and Wales with number 741598 > Registered office: Building C, IBM Hursley Office, Hursley Park Road, > Winchester, Hampshire SO21 2JN > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- Jeremy Austin jhaustin(a)gmail.com

1 0

Test email
by Eugen Block 17 Dec '25

17 Dec '25

Test

1 1

Re: Problems with orchestrator
by Carsten Götze 17 Dec '25

17 Dec '25

Hi! Thank you for the pointers. Unfortunately restarting the active mgr daemon didn't help. However looking at cephadm.log gave me at least a clue what was going on. It seems that installing and deinstalling systemd-firewalld has completely messed up the firewall rules. And restarting one of the nodes has made things even worse. The osd's on that node died after a few minutes. I had to restart the osd's systemd services, only to see them tumbling down a few minutes later. Only after I finally flushed all the firewall rules from every single node things went back to "normal". However, orchestrator is still out of sync with reality. So a few more questions: Where does orchestrator store its state information? Is firewalld required for tentacle? I didn't need it for squid, but It sure is now required for the nfs module.Do the other modules also use it to control the firewall? With firewalld installed, would restarting the nodes help to restore the firewall rules? What ports have to be open in the firewall? So far I have: 3000, 3300, 5666, 6789, 6800:7568, 8443, 9093, 9094, 9095, 9100. Anything else? Would it be safe to downgrade to squid again? With best regards Carsten Götze > Am 16.12.2025 um 14:10 schrieb Robert Sander via ceph-users <ceph-users(a)ceph.io>: > > Hi, > > Am 16.12.25 um 2:03 PM schrieb Carsten Götze via ceph-users: > >> Is there a way to force the orchestrator to sync its state information with the nodes? > > Try to restart the active MGR daemon with systemctl on its node. > >> Where do I find meaningful logs for the orchestrator? > > The journal of the active MGR contains cephadm output > > Regards > -- > Robert Sander > Linux Consultant > > Heinlein Consulting GmbH > Schwedter Str. 8/9b, 10119 Berlin > > https://www.heinlein-support.de > > Tel: +49 30 405051 - 0 > Fax: +49 30 405051 - 19 > > Amtsgericht Berlin-Charlottenburg - HRB 220009 B > Geschäftsführer: Peer Heinlein - Sitz: Berlin > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Temporay credentials in STS
by Rok Jaklič 17 Dec '25

17 Dec '25

To answer to myself and for those interested. "You" just need one user in this case TESTER and policy_document has to be: policy_document = '''{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam::TENANT:user/USER"]},"Action":["sts:AssumeRole"]}]}''' If TENANT is TENANT1 and USER is USER1, then policy_document should be: policy_document = '''{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":["arn:aws:iam::TENANT1:user/USER1"]},"Action":["sts:AssumeRole"]}]}''' Kind regards, Rok On Tue, Dec 16, 2025 at 3:55 PM Rok Jaklič via ceph-users < ceph-users(a)ceph.io> wrote: > I get an error like, when trying to assume role: > response = sts_client.assume_role( > File > "/home/user/um-test-venv/lib64/python3.9/site-packages/botocore/client.py", > line 602, in _api_call > return self._make_api_call(operation_name, kwargs) > File > > "/home/user/um-test-venv/lib64/python3.9/site-packages/botocore/context.py", > line 123, in wrapper > return func(*args, **kwargs) > File > "/home/user/um-test-venv/lib64/python3.9/site-packages/botocore/client.py", > line 1078, in _make_api_call > raise error_class(parsed_response, operation_name) > botocore.exceptions.ClientError: An error occurred (Unknown) when calling > the AssumeRole operation: Unknown > (um-test-venv) [user@server ~]$ client_loop: send disconnect: Broken pipe > > Documentation is a little bit ambiguous. There is user TESTER and user > TESTER1. Do I need to create both users? > > --- > > What is the right way to set policy and role for it if I have just USER1 > and TENANT1 and ACCESS_KEY1 and SECRET_KEY1 and want to have temporary > credentials? > > On Tue, Dec 16, 2025 at 6:31 AM Pritha Srivastava <prsrivas(a)redhat.com> > wrote: > > > Hi, > > > > The temporary credentials are present in the assume role call response. > > > > Thanks, > > Pritha > > > > > > On Tue, Dec 16, 2025 at 1:09 AM Rok Jaklič via ceph-users < > > ceph-users(a)ceph.io> wrote: > > > >> Hi, > >> > >> I try to follow https://docs.ceph.com/en/reef/radosgw/STS/ > >> > >> If I print role_response, there are no credentials in the response > object? > >> Where can I get credentials for TESTER1 after creating a role and > setting > >> policy for it? > >> > >> Kind regards, > >> Rok > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users(a)ceph.io > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >> > >> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: Achieve no loss-of-write with 2 node failure?
by Anthony D'Atri 16 Dec '25

16 Dec '25

> >> Note: larger HDDs have really low IOPS-per-TB; SSDs avoid that >> issue but cheap SSDs do not have PLP so write IOPS are much >> lower than read IOPS. > > That is something I've seen mentioned a lot, so we've only got PLP drives on the shopping list. The tentative current shopping list is 24x 7.68TB Samsung PM893 or Kingston DC600M drives. PLP is crucial, but there are other factors. If you have a choice, NVMe SSDs give a lot more bang for the buck than SATA. The Kingstons don't have IIRC stellar performance, but maybe you don't need it to be. > >> Whether the drive is SSD or HDD larger >> ones also usually mean large PGs which is not so good. With SSDs >> at least it is possible (and in some cases advisable) to split >> them into multiple OSDs though. > > Could we just increase the number of PGs to avoid this? With Quincy and later releases, the former advice to split became a wash. Splitting HDDs was never widely practiced or advised. With modern SSDs > 30 TB splitting might make sense. There are serializations both in OSD and PG code. > >> That is indeed a good suggestion: the fewer the drives per >> server the better. Ideally just one drive per server :-). > > This might just be possible, since we've got a couple of racks of render nodes that I can probably make the case to retire from render duties. Would I actually see a major advantage going from 6 nodes to 8, from 8 to 12, or from 12 to 24? (Given 24 disks in each case.) Those chassis can even only have their server trays partly populated too. Were you to have 8 chassis, you could make `chassis` your CRUSH failure domain, with however many drives per chassis/host. > > Andrew > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 2

Re: Achieve no loss-of-write with 2 node failure?
by Andrew Klaassen 16 Dec '25

16 Dec '25

On Tuesday, December 16, 2025 at 02:33:33 p.m. EST, Anthony D'Atri via ceph-users <ceph-users(a)ceph.io> wrote: > Indeed. I suggest, if Proxmox lets you, encoding the chassis into your > CRUSH topology, even if you don't use it today. Maybe in the future > you can add chassis, which would expand your CRUSH options. > > root default > chassis chassis1 > host host1 > host host2 > host host3 > host host4 > chassis chassis2 > host host5 > ... > > With `host` as the CRUSH failure domain, you do run the risk of a PG > being placed on 3 or even 4 nodes within a single chassis, which > would not be good. A custom CRUSH rule might let you constrain > so that PG placements always span chassis; those are beyond my ken. > Someone else on the list could likely help with that rarified task. > > Alternately there are the shiny new MSR CRUSH rules: > https://docs.ceph.com/en/latest/dev/crush-msr/ > https://www.youtube.com/watch?v=JpkLPkizUt4 Thanks, I will look into that. Andrew

1 0

Re: Achieve no loss-of-write with 2 node failure?
by Andrew Klaassen 16 Dec '25

16 Dec '25

On Tuesday, December 16, 2025 at 03:28:01 p.m. EST, Peter Grandi via ceph-users <ceph-users(a)ceph.io> wrote: > * Ceph is designed around the idea of many servers and many > small drives with a few small drives per server as it needs to > have lots of IOPS-per-TB and low impact per system failure. > * Minimizing up-front costs means fewer larger drives and higher > density servers. > * The common result is extreme congestion during balancing and > recovery times and high latencies during parallel accesses. > This mailing list is full of "cost-optimized" horror stories. Good to know, and thanks for the further explanations why. > Note: larger HDDs have really low IOPS-per-TB; SSDs avoid that > issue but cheap SSDs do not have PLP so write IOPS are much > lower than read IOPS. That is something I've seen mentioned a lot, so we've only got PLP drives on the shopping list. The tentative current shopping list is 24x 7.68TB Samsung PM893 or Kingston DC600M drives. > Whether the drive is SSD or HDD larger > ones also usually mean large PGs which is not so good. With SSDs > at least it is possible (and in some cases advisable) to split > them into multiple OSDs though. Could we just increase the number of PGs to avoid this? > That is indeed a good suggestion: the fewer the drives per > server the better. Ideally just one drive per server :-). This might just be possible, since we've got a couple of racks of render nodes that I can probably make the case to retire from render duties. Would I actually see a major advantage going from 6 nodes to 8, from 8 to 12, or from 12 to 24? (Given 24 disks in each case.) Andrew

1 0

Proposal to end support for some erasure code plugins
by Jamie Pryde 16 Dec '25

16 Dec '25

Hi all, I want to make you aware of the Fast EC team's proposal to end support for some erasure code plugins and techniques. We want to do this for a few reasons: 1. Continuing to support a large list of plugins and techniques requires development and test effort. We don't think this effort is valuable because 2. Telemetry data shows that most Ceph clusters that are using EC are using Jerasure reed_sol_van or ISA-L reed_sol_van, and 3. Benchmarking shows that reed_sol_van has very good performance on modern CPUs. We are proposing that the following list of plugins and techniques should be deprecated in the Umbrella release, and support be removed in the V release: * Jerasure (reed_sol_r6_op, cauchy_orig, cauchy_good, liberation, blaum_roth, liber8tion) * SHEC * CLAY The following list of plugins and techniques would continue to be supported in the V release: * Jerasure (reed_sol_van) * ISA-L (reed_sol_van, cauchy) * LRC (Although Fast EC doesn’t currently support LRC and we wouldn't recommend using LRC yet, we think we will be able to use it in future to improve support for erasure coded pools in stretched clusters.) I have written a post for the ceph.io blog which contains more details and shows some of the data behind the proposal. It also talks about the pool migration feature we are planning to add in Umbrella. This can be used to migrate data from a pool to a new EC pool that uses a supported plugin/technique. You can find the blog PR here: https://github.com/ceph/ceph.io/pull/988 We are interested to hear if anyone has any objections to this plan. Please reply to this mail or leave a comment on the blog PR if you do have any objections or concerns. We plan to publish the blog post sometime mid-January. Thanks, -- Jamie Pryde Software Engineer IBM UK Unless otherwise stated above: IBM United Kingdom Limited Registered in England and Wales with number 741598 Registered office: Building C, IBM Hursley Office, Hursley Park Road, Winchester, Hampshire SO21 2JN

1 0

Re: Achieve no loss-of-write with 2 node failure?
by pg＠ceph.list.sabi.co.UK 16 Dec '25

16 Dec '25

>>> b) a minimum of k+m+2 nodes? > [...] If up-front CapEx is your concern The perhaps Ceph is not such a great idea: * Ceph is designed around the idea of many servers and many small drives with a few small drives per server as it needs to have lots of IOPS-per-TB and low impact per system failure. * Minimizing up-front costs means fewer larger drives and higher density servers. * The common result is extreme congestion during balancing and recovery times and high latencies during parallel accesses. This mailing list is full of "cost-optimized" horror stories. Note: larger HDDs have really low IOPS-per-TB; SSDs avoid that issue but cheap SSDs do not have PLP so write IOPS are much lower than read IOPS. Whether the drive is SSD or HDD larger ones also usually mean large PGs which is not so good. With SSDs at least it is possible (and in some cases advisable) to split them into multiple OSDs though. > remember that you don't have to fully populate nodes with > drives, at least not initially. That is indeed a good suggestion: the fewer the drives per server the better. Ideally just one drive per server :-). > I sometimes recommend a minimum of 7 nodes so that 4+2 or 3+3 > EC can be done safely. For me 7 buckets and 6-wide EC stripes is less desirable as that means that *any* bucket failure will cause *nearly all* objects to be degraded, and if one uses CephFS which chunks files across objects, too bad. I have seen long, long, long recovery storms in configurations like that. > Seven nodes half-full is better in multiple ways than 4 nodes > fully populated. Indeed. But with 6-7 buckets I would only use 3-ways replication.

1 0

Re: Achieve no loss-of-write with 2 node failure?
by Anthony D'Atri 16 Dec '25

16 Dec '25

> We're in a situation where we've got nodes that we can use, so it's mostly about the drive cost and configuration. Ack. > The nodes are in chassis of 4 nodes each. Would there be any problem with putting a 4+2 or 3+3 config into 8 nodes? With `host` as the CRUSH failure domain, that works. > > And any problem with having 7 monitors running on Proxmox nodes that are part of the Ceph cluster but aren't running any OSDs? Don't deploy 7. 5 is plenty. No reason to couple mons with OSDs. It's a good idea, though, to ensure that you spread mons across failure domains, so that losing one host or chassis doesn't lead to a loss of quorum. > > (Obviously we wouldn't be able to survive the loss of a chassis in this config.) Indeed. I suggest, if Proxmox lets you, encoding the chassis into your CRUSH topology, even if you don't use it today. Maybe in the future you can add chassis, which would expand your CRUSH options. root default chassis chassis1 host host1 host host2 host host3 host host4 chassis chassis2 host host5 ... With `host` as the CRUSH failure domain, you do run the risk of a PG being placed on 3 or even 4 nodes within a single chassis, which would not be good. A custom CRUSH rule might let you constrain so that PG placements always span chassis; those are beyond my ken. Someone else on the list could likely help with that rarified task. Alternately there are the shiny new MSR CRUSH rules: https://docs.ceph.com/en/latest/dev/crush-msr/ https://www.youtube.com/watch?v=JpkLPkizUt4 > > Andrew > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Achieve no loss-of-write with 2 node failure?
by Andrew Klaassen 16 Dec '25

16 Dec '25

On Tuesday, December 16, 2025 at 07:16:28 a.m. EST, Robert Sander via ceph-users <ceph-users(a)ceph.io> wrote: > This is recommended because with this number of nodes the cluster still > has "spare" nodes to recover to in case of a node failure. Thanks. Andrew

1 0

Re: Achieve no loss-of-write with 2 node failure?
by Andrew Klaassen 16 Dec '25

16 Dec '25

On Tuesday, December 16, 2025 at 08:21:43 a.m. EST, Anthony D'Atri via ceph-users <ceph-users(a)ceph.io> wrote: > If up-front CapEx is your concern, remember that you don't have to fully > populate nodes with drives, at least not initially. I sometimes recommend > a minimum of 7 nodes so that 4+2 or 3+3 EC can be done safely. Seven nodes > half-full is better in multiple ways than 4 nodes fully populated. > > As for nodes, used Dell R640 can be had with 8 or 10 NVMe bays and lots > of cores very inexpensively these days. Will they give another (let's be honest) > 10 years of service? Hard to say. But today's SSDs can be transplanted into > tomorrow's chassis, and when the latter is cheap, one can easily afford to lay > in a couple of spares. It's like with full-frame digital photography: a body > from 10 years ago is meh by today's standards, but that $10,000 lens for > shooting sports ball is still golden. We're in a situation where we've got nodes that we can use, so it's mostly about the drive cost and configuration. The nodes are in chassis of 4 nodes each. Would there be any problem with putting a 4+2 or 3+3 config into 8 nodes? And any problem with having 7 monitors running on Proxmox nodes that are part of the Ceph cluster but aren't running any OSDs? (Obviously we wouldn't be able to survive the loss of a chassis in this config.) Andrew

1 0

Re: Temporay credentials in STS
by Pritha Srivastava 16 Dec '25

16 Dec '25

Hi, The temporary credentials are present in the assume role call response. Thanks, Pritha On Tue, Dec 16, 2025 at 1:09 AM Rok Jaklič via ceph-users < ceph-users(a)ceph.io> wrote: > Hi, > > I try to follow https://docs.ceph.com/en/reef/radosgw/STS/ > > If I print role_response, there are no credentials in the response object? > Where can I get credentials for TESTER1 after creating a role and setting > policy for it? > > Kind regards, > Rok > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

2 1

Re: Low number of inflight compare to op_wip
by Anthony D'Atri 16 Dec '25

16 Dec '25

> I'm tryting to debug low performance on nvme-based cluster. Which specific drives? > I have 24 NVME in 4 servers, plenty of CPU, cluster perfectly balanced, no scrubbing or replication atm, 1024 pg for 24 OSD/17TB. > > I expect to see some performance (~250k IOPS total, 50% r/w, few hundred volumes with capped by IOPS, pre-warmed). I see about half of it. > > I looked at drive utilization, it's about 70% (per atop) That's meaningless for SSDs. > but I've noticed, that in-flight for drives is, basically, around 1. That means, that at a given time only one request is processed. This is match match for OSD count/3 /latency formula, and with one in-flight nvme is showing about 10% of it's specs (Intel, DC grade). Have you updated to the latest firmware with SST? > > I looked at osd (they loaded uniformely, so any of them shows the same results). > > I see "op_wip": 10-27, but in-flight value is about 0-2, mostly around 1. > > I can't get away from the feeling that somehow osd is doing operations (almost) sequential. I already played with osd_op_num_threads_per_shard (4), osd_op_num_shards (8), set mclock profile to high_client_ops, ms_async_op_threads 24, and it can't get more inflight ios. > > I feel I miss something. How to make Ceph to send more requests to underlaying NVME? > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Achieve no loss-of-write with 2 node failure?
by Anthony D'Atri 16 Dec '25

16 Dec '25

> >> If I set min_size=k+1, does that mean I'd need a minimum of m=3 to avoid loss-of-write with a 2-node failure? > > Yes. Having at least k+m+1 failure domains (nodes in your case) also, subtly, means that your failure domains can be of different aggregate CRUSH weights without the capacity delta being unusable. > >> And to avoid loss-of-write with a 2-node failure, would I need: >> a) a minimum of k+m nodes, or > > This is the bare minimum for EC without any recovery location. > >> b) a minimum of k+m+2 nodes? > > This is recommended because with this number of nodes the cluster still has "spare" nodes to recover to in case of a node failure. Indeed, Ceph is all about strong consistency. It would rather interdict writes than have you make a risky write. If up-front CapEx is your concern, remember that you don't have to fully populate nodes with drives, at least not initially. I sometimes recommend a minimum of 7 nodes so that 4+2 or 3+3 EC can be done safely. Seven nodes half-full is better in multiple ways than 4 nodes fully populated. As for nodes, used Dell R640 can be had with 8 or 10 NVMe bays and lots of cores very inexpensively these days. Will they give another (let's be honest) 10 years of service? Hard to say. But today's SSDs can be transplanted into tomorrow's chassis, and when the latter is cheap, one can easily afford to lay in a couple of spares. It's like with full-frame digital photography: a body from 10 years ago is meh by today's standards, but that $10,000 lens for shooting sports ball is still golden.

1 0

Re: Problems with orchestrator
by Robert Sander 16 Dec '25

16 Dec '25

Hi, Am 16.12.25 um 2:03 PM schrieb Carsten Götze via ceph-users: > Is there a way to force the orchestrator to sync its state information with the nodes? Try to restart the active MGR daemon with systemctl on its node. > Where do I find meaningful logs for the orchestrator? The journal of the active MGR contains cephadm output Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

Problems with orchestrator
by Carsten Götze 16 Dec '25

16 Dec '25

Hi! I'm running a ceph cluster using cephadm and recently upgraded from squid to tentacle 20.2.0. Until recently everything worked fine until I started the nfs module. The nfs daemons were reported to be running but after some 10 minutes or so all of them were reported to be dead except one. nfs service on port 2049 was never provided on any of the nodes even while the daemons were supposed to be running. As i found out later, the nfs daemons were never started at all, because the setup process required a systemd-firewalld to be installed on the system which of course wasn't. After some headaches with the newly installed firewalld I decided to roll back, delete the firewalld and postpone the nfs deployment. I then tried to stop the nfs daemons with 'ceph orch daemon stop', which did nothing, even after waiting some 10 minutes. I had to reissue the command several times to make the reportedly dead nfs daemons vanish from the 'ceph orch ps' list. The one daemon that was reported to be still running however would only die after 'ceph orch daemon stop --force' and was in an 'error' state thereafter and could not be removed from the 'ceph orch ps' list by no means. So I decided to delete the managing nfs service from the 'ceph orch ls' list, in hope that it would also tear down the remaining nfs daemon. This obviously was a bad idea since, the service is now in the state of deleting. However it cannot be deleted, because there is still the one daemon in error state, which cannot be deleted because it was never running at all. As a last measure I forcefully removed the docker container on the node with the cephadm command, but even though there are no traces left of that nfs daemon, it is still listed when running 'ceph orch ps'. I also noticed that the 'ceph orch device ls' is out of sync with reality and 'ceph orch ps' is still listing osds that I've already shutdown and deleted. I therefore suspect, that the orchestrator has stopped collecting state information from the nodes. Is there a way to force the orchestrator to sync its state information with the nodes? Where do I find meaningful logs for the orchestrator? With best regards, Carsten Götze

1 0

Re: Achieve no loss-of-write with 2 node failure?
by Robert Sander 16 Dec '25

16 Dec '25

Hi, Am 16.12.25 um 1:05 PM schrieb Andrew Klaassen via ceph-users: > If I set min_size=k+1, does that mean I'd need a minimum of m=3 to avoid loss-of-write with a 2-node failure? Yes. > And to avoid loss-of-write with a 2-node failure, would I need: > > a) a minimum of k+m nodes, or This is the bare minimum for EC without any recovery location. > b) a minimum of k+m+2 nodes? This is recommended because with this number of nodes the cluster still has "spare" nodes to recover to in case of a node failure. Regards -- Robert Sander Linux Consultant Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: +49 30 405051 - 0 Fax: +49 30 405051 - 19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

Achieve no loss-of-write with 2 node failure?
by Andrew Klaassen 16 Dec '25

16 Dec '25

I've been trying to wrap my head around avoiding loss-of-write with a 2-node failure with erasure coding. If I set min_size=k+1, does that mean I'd need a minimum of m=3 to avoid loss-of-write with a 2-node failure? And to avoid loss-of-write with a 2-node failure, would I need: a) a minimum of k+m nodes, or b) a minimum of k+m+2 nodes? If the answer is k+m nodes, would I want to go with k+m+2 nodes anyway for other reasons? Thanks. Andrew

1 0

Low number of inflight compare to op_wip
by George Shuklin 16 Dec '25

16 Dec '25

I'm tryting to debug low performance on nvme-based cluster. I have 24 NVME in 4 servers, plenty of CPU, cluster perfectly balanced, no scrubbing or replication atm, 1024 pg for 24 OSD/17TB. I expect to see some performance (~250k IOPS total, 50% r/w, few hundred volumes with capped by IOPS, pre-warmed). I see about half of it. I looked at drive utilization, it's about 70% (per atop), but I've noticed, that in-flight for drives is, basically, around 1. That means, that at a given time only one request is processed. This is match match for OSD count/3 /latency formula, and with one in-flight nvme is showing about 10% of it's specs (Intel, DC grade). I looked at osd (they loaded uniformely, so any of them shows the same results). I see "op_wip": 10-27, but in-flight value is about 0-2, mostly around 1. I can't get away from the feeling that somehow osd is doing operations (almost) sequential. I already played with osd_op_num_threads_per_shard (4), osd_op_num_shards (8), set mclock profile to high_client_ops, ms_async_op_threads 24, and it can't get more inflight ios. I feel I miss something. How to make Ceph to send more requests to underlaying NVME?

1 0

Temporay credentials in STS
by Rok Jaklič 15 Dec '25

15 Dec '25

Hi, I try to follow https://docs.ceph.com/en/reef/radosgw/STS/ If I print role_response, there are no credentials in the response object? Where can I get credentials for TESTER1 after creating a role and setting policy for it? Kind regards, Rok

1 0

Re: Fwd: On your post: "[Tentacle 20.2.0]: Inconsistent pg's after enabling ec optimisation flag"
by David Walter 15 Dec '25

15 Dec '25

Hi Jeff, What migration did you do that you say the mismatches went away? Did the crashing of OSDs also stop? Migration to a new pool on the same or different OSDs? With or without EC optimization enabled? Or re-writing the data in the same pool by purging one OSD at a time? Best, David On 12/5/25 11:02, Jeff Bailey via ceph-users wrote: > I've seen OSD's crash when running an fstrim on an RBD with an EC data > pool with the EC optimizations enabled. Always two OSD's. It's a 8+2 > pool but I don't know if the "2"s are a coincidence or not. Other than > that, I haven't seen any OSD crashes with EC optimizations. We had > the same size mismatches that went away after migrating the data. > > > On 12/5/2025 10:34 AM, Reto Gysi via ceph-users wrote: >> Hi Bill >> >> Am Fr., 5. Dez. 2025 um 10:02 Uhr schrieb Bill Scales < >> bill_scales(a)uk.ibm.com>: >> >>> Hi Reto, >>> >>> Sorry to hear about your problems with turning on ec optimizations. >>> I've >>> led the team that developed this feature, so we are keen to help >>> understand >>> what happened to your system. >>> >>> Your configuration looks fine as far as being supported with ec >>> optimizations. The daemons (mons, osds and mgr) need to be running >>> tentacle >>> code to use this feature, there is no requirement to update any of your >>> clients. >>> >>> Do you have any more examples of the inconsistent objects logs that you >>> can share with me? >>> >>> There was a bug that we fixed late in the development cycle which was >>> scrubbing incorrectly reporting a size mismatch for objects written >>> prior >>> to turning on optimizations. This is because prior to turning on >>> optimizations objects are padded to a multiple of the stripe_width, >>> afterwards objects are no longer padded. The scrub code was >>> occasionally >>> getting confused and incorrectly reporting an inconsistency. In this >>> case >>> the scrub was reporting false positive errors - there was nothing wrong >>> with the data and no problems accessing the data. I notice the log you >>> shared in the email was for a size mismatch, I'm interested whether >>> all the >>> logs were for mismatched sizes. We will do some further work to confirm >>> that the fix is in the 20.2.0 tentacle release and that there are no >>> problems with the back port. >>> >> As far I could see the errors where all about the mismatched sizes. >> >> >>> You also mention that you saw some OSD crashes. Do you have any further >>> information about these? >>> >> I just could cause another crash with starting up a windows VM, and >> doing a >> WIndows file history backup to a drive that is on an RBD Image on pool >> rbd_ecpool where I had the allow_ec_optimization flag enabled. >> >> <disk type="network" device="disk"> >> <driver name="qemu" type="raw" cache="writethrough" io="threads" >> discard="unmap" detect_zeroes="unmap"/> >> <auth username="admin"> >> <secret type="ceph" uuid="878b0bc5-c471-4ec6-a92a-f65282ffbdf6"/> >> </auth> >> <source protocol="rbd" name="rbd/game_windows_backup_drive" >> index="3"> >> <host name="zephir" port="6789"/> >> </source> >> <target dev="vdd" bus="virtio"/> >> <alias name="virtio-disk3"/> >> <address type="pci" domain="0x0000" bus="0x00" slot="0x02" >> function="0x0"/> >> </disk> >> >> root@zephir:~# ceph -s >> cluster: >> id: 27923302-87a5-11ec-ac5b-976d21a49941 >> health: HEALTH_WARN >> 2 osds down >> Reduced data availability: 49 pgs inactive >> Degraded data redundancy: 15722192/79848340 objects degraded >> (19.690%), 246 pgs degraded, 247 pgs undersized >> >> services: >> mon: 3 daemons, quorum zephir,debian,raspi (age 20h) >> [leader: >> zephir] >> mgr: zephir.enywvy(active, since 21h), standbys: >> debian.nusuye >> mds: 3/3 daemons up, 3 standby >> osd: 18 osds: 16 up (since 5m), 18 in (since 3d); 8 >> remapped >> pgs >> cephfs-mirror: 2 daemons active (2 hosts) >> rbd-mirror: 2 daemons active (2 hosts) >> rgw: 1 daemon active (1 hosts, 1 zones) >> tcmu-runner: 5 portals active (2 hosts) >> >> data: >> volumes: 3/3 healthy >> pools: 25 pools, 450 pgs >> objects: 17.52M objects, 62 TiB >> usage: 116 TiB used, 63 TiB / 179 TiB avail >> pgs: 10.889% pgs not active >> 15722192/79848340 objects degraded (19.690%) >> 197 active+undersized+degraded >> 183 active+clean >> 49 undersized+degraded+peered >> 15 active+clean+scrubbing >> 5 active+clean+scrubbing+deep >> 1 active+undersized >> >> io: >> client: 1.7 KiB/s rd, 1023 B/s wr, 1 op/s rd, 0 op/s wr >> >> root@zephir:~# ceph osd status >> ID HOST USED AVAIL WR OPS WR DATA RD OPS RD DATA STATE >> 0 debian 11.2T 5289G 0 0 0 0 exists,up >> 1 zephir 11.6T 4846G 0 0 0 0 exists >> 2 zephir 10.7T 5827G 0 0 2 4 exists,up >> 3 zephir 11.0T 5497G 0 0 0 0 exists >> 4 zephir 159G 1376G 0 0 2 0 exists,up >> 5 zephir 120G 1415G 0 0 0 0 exists,up >> 6 debian 11.3T 5200G 0 0 1 0 exists,up >> 7 zephir 305G 1230G 0 0 0 0 exists,up >> 8 zephir 10.9T 5594G 0 0 0 0 exists,up >> 9 zephir 11.9T 4616G 0 0 0 0 exists,up >> 10 zephir 235G 1300G 0 0 0 0 exists,up >> 11 zephir 223G 1312G 0 0 0 0 exists,up >> 12 zephir 11.0T 5479G 0 0 0 0 exists,up >> 13 debian 118G 631G 0 0 3 8 exists,up >> 14 debian 345G 1190G 0 0 0 0 exists,up >> 15 debian 11.4T 5079G 0 0 0 0 exists,up >> 16 debian 554G 3029G 0 0 3 8 exists,up >> 17 debian 12.5T 5874G 0 0 1 0 exists,up >> root@zephir:~# >> >> I attached the log of osd.1 to >> https://filebin.net/jwvs6kuqrc7hx8id >> >> Best Regards, >> >> Reto >> >> >> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Re: scrape-health-metrics fails with sudo PAM account management error
by Flemming Frandsen 15 Dec '25

15 Dec '25

I think I tracked down the problem to lacking permissions on the /etc/shadow file in the squid container image: docker exec -it ceph-eac6da30-d72a-11f0-88be-7cc255639332-osd-9 ls -l /etc/shadow ---------- 1 root root 583 Nov 4 16:31 /etc/shadow I have verified that the same situation exists in the tentacle images. if I make the file readable by root: docker exec -it ceph-eac6da30-d72a-11f0-88be-7cc255639332-osd-9 chmod u+r /etc/shadow docker exec -it ceph-eac6da30-d72a-11f0-88be-7cc255639332-osd-9 ls -l /etc/shadow -r-------- 1 root root 583 Nov 4 16:31 /etc/shadow Then the sudo command needed by osd works perfectly. Looking at the hosts dmesg I finally noticed that the problem is with the default apparmor profile called unix_chkpwd: [Mon Dec 15 09:52:25 2025] audit: type=1400 audit(1765792348.851:248): apparmor="DENIED" operation="open" class="file" profile="unix-chkpwd" name="/dev/console" pid=1028560 comm="unix_chkpwd" requested_mask="w" denied_mask="w" fsuid= 0 ouid=0 I disabled the profile by running these commands on the host: mkdir /etc/apparmor.d/disable/ ln -s /etc/apparmor.d/unix-chkpwd /etc/apparmor.d/disable/ apparmor_parser -R /etc/apparmor.d/unix-chkpwd After getting rid of the app armor profile all the osd containers started working perfectly. It's quite possible that there is a more elegant solution, but I can't be bothered to find it at the moment. On Fri, 12 Dec 2025 at 11:58, Flemming Frandsen via ceph-users < ceph-users(a)ceph.io> wrote: > I've been building a new cluster with cephadm, the OS is Ubuntu 24.04 and > I'm using the ubuntu provided host packages, docker is 29.1.2 and > containerd is 2.2.0 and the ceph release is squid 19.2.3. > > Everything seems to work just perfectly, except for scrape-health-metrics, > which records this result (for all of the osds in the 4 hosts in the > cluster): > "20251212-074916": { > "dev": "/dev/nvme5n1", > "error": "smartctl failed", > "nvme_smart_health_information_add_log_error": "nvme returned an > error: sudo: exit status: 1", > "nvme_smart_health_information_add_log_error_code": -22, > "nvme_vendor": "samsung", > "smartctl_error_code": -22, > "smartctl_output": "smartctl returned an error (1): stderr:\nsudo: > exit status: 1\nstdout:\n" > }, > > Digging though the source I found that it's the OSD container that runs the > command: sudo /usr/sbin/smartctl -x --json=o /dev/nvme5n1 > > I can run the command in the container using: > docker exec -it -u ceph ceph-eac6da30-d72a-11f0-88be-7cc255639332-osd-3 > sudo /usr/sbin/smartctl -x --json=o /dev/nvme5n1 > > The result of the manual command is: > sudo: PAM account management error: Authentication service cannot retrieve > authentication info > sudo: a password is required > > I have enabled debugging for sudo, in the container and verified that the > command that the osd is running, is same one and that the error is the same > as well, running ceph device scrape-health-metrics causes this sudo debug > output: > > Dec 12 10:30:35 sudo[845] user command "/usr/sbin/smartctl -x --json=o > /dev/nvme5n1" matches sudoers command "/usr/sbin/smartctl -x --json=o > /dev/*": true @ command_matches() ./match_command.c:667 > Dec 12 10:30:35 sudo[845] userspec matched @ > /etc/sudoers.d/ceph-smartctl:4:57: allowed @ sudoers_lookup_check() > ./parse.c:167 > Dec 12 10:30:35 sudo[845] sudo_putenv: SUDO_COMMAND=/usr/sbin/smartctl -x > --json=o /dev/nvme5n1 > Dec 12 10:30:35 sudo[845] <- new_logline @ ./eventlog.c:218 := PAM account > management error: Authentication service cannot retrieve authentication > info ; TTY=pts/8 ; PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x > --json=o /dev/nvme5n1 > > > My only guess is that I have chosen a too-new Ubuntu version for Ceph and I > should just bite the bullet and re-install with Ubuntu 22.04, but if anyone > has a better idea, please let me know. > > -- > Flemming Frandsen - YAPH - http://osaa.dk - http://dren.dk/ > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- Flemming Frandsen - YAPH - http://osaa.dk - http://dren.dk/

1 0

Re: reef 18.2.8 QE validation status
by Ilya Dryomov 13 Dec '25

13 Dec '25

On Mon, Dec 8, 2025 at 11:31 PM Yuri Weinstein via Dev <dev(a)ceph.io> wrote: > > This test branch was built loner then 2.5 weeks ago > I have to rebuild it first and then will schedule krbd krbd approved based on https://pulpito.ceph.com/yuriw-2025-12-10_21:26:56-krbd-reef-release-testin… https://pulpito.ceph.com/yuriw-2025-12-12_18:20:32-krbd-reef-release-testin… Thanks, Ilya > > On Mon, Dec 8, 2025 at 1:39 PM Ilya Dryomov via ceph-users <ceph-users(a)ceph.io> wrote: >> >> On Mon, Dec 8, 2025 at 8:50 PM Ilya Dryomov <idryomov(a)gmail.com> wrote: >> > >> > On Mon, Dec 8, 2025 at 5:00 PM Venky Shankar via ceph-users >> > <ceph-users(a)ceph.io> wrote: >> > > >> > > Hi Yuri, >> > > >> > > On Fri, Nov 21, 2025 at 4:34 AM Yuri Weinstein <yweinste(a)redhat.com> wrote: >> > > > >> > > > Note: this is an attempt to test and publish post EOL reef point release >> > > > >> > > > Details of this release are summarized here: >> > > > >> > > > https://tracker.ceph.com/issues/73906#note-1 >> > > > Release Notes - TBD >> > > > LRC upgrade - N/A >> > > > >> > > > distro => jammy focal centos9 windows bookworm >> > > > >> > > > Seeking approvals/reviews for: >> > > > >> > > > smoke - Laura approved? >> > > > >> > > > rados - Radek, Laura approved? Travis? Nizamudeen? Adam King approved? >> > > > (rados is also running on build 2 == build 1 + >> > > > https://github.com/ceph/ceph/pull/66056 per Igor's request) >> > > > >> > > > rgw - Adam E approved? >> > > > >> > > > fs - Venky approved? >> > > >> > > There are lot many failures in the fs suite resembling >> > > >> > > Error mounting /var/lib/ceph: mount: /var/lib/ceph: wrong fs >> > > type, bad option ... >> > > >> > > which is an issue with the testing kernel. Adding Alex who was looking >> > > into this issue. >> > > >> > > As far as the run validation is concerned, it's risky to ignore the >> > > large number of failures. We have two options: >> > > >> > > - fix the testing kernel issue and rerun >> > > - rerun with `for-linus` branch (with `-k for-linus` in the >> > > teuthology-suite schedule command) >> > > >> > > The latter would be using an older kernel branch, but I think it >> > > should suffice for this case. >> > >> > Hi Venky, >> > >> > With your report of the fs suite being affected as well, I carved out >> > some time to take a look. >> > >> > I think it's happening because XFS maintainers disabled support for V4 >> > filesystems that was deprecated 5 years ago -- as of 6.18 kernel it's >> > no longer compiled in by default. The proper fix would be to create an >> > image with/move to V5 on-disk format; I have opened [1] to try to >> > achieve that. We can also re-enable V4 on-disk format in our kernel >> > builds as a workaround. >> > >> > [1] https://github.com/ceph/ceph-cm-ansible/pull/799 >> >> This should be fixed now. Yuri, please schedule a rerun for krbd. >> >> Thanks, >> >> Ilya >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > _______________________________________________ > Dev mailing list -- dev(a)ceph.io > To unsubscribe send an email to dev-leave(a)ceph.io

1 0

Re: v20.2.0 Tentacle intentional released but no build for el10?
by Gregory Farnum 12 Dec '25

12 Dec '25

el10 upgrades the python release and ceph-mgr DOES NOT WORK past I believe Python 3.12 due to ABI changes. (ceph-mgr makes extensive use of submodules, which has always been a bit iffy, and cpython upstream did a big rework of them which should be more supportable long-term but changed the implementation surface in the meantime.) Work is in progress to resolve and we will release el10 packages once that happens. -Greg On Fri, Dec 12, 2025 at 3:19 AM Moritz Baumann via ceph-users < ceph-users(a)ceph.io> wrote: > Hi > > is there a reason that there is no el10 under: > > https://download.ceph.com/rpm-20.2.0/ > https://download.ceph.com/rpm-tentacle/ > > > building with mock for Alma10.1/Rocky10.1 just works fine: > > curl -O -L > https://download.ceph.com/rpm-tentacle/el9/SRPMS/ceph-20.2.0-0.el9.src.rpm > mkdir rpm > mock -r /etc/mock/alma+epel-10-x86_64.cfg rebuild > ./ceph-20.2.0-0.el9.src.rpm --enable-network --resultdir rpm > # or > mock -r /etc/mock/rocky+epel-10-x86_64.cfg rebuild > ./ceph-20.2.0-0.el9.src.rpm --enable-network --resultdir rpm > > both work fine. > > Best > Moritz > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

v20.2.0 Tentacle intentional released but no build for el10?
by Moritz Baumann 12 Dec '25

12 Dec '25

Hi is there a reason that there is no el10 under: https://download.ceph.com/rpm-20.2.0/ https://download.ceph.com/rpm-tentacle/ building with mock for Alma10.1/Rocky10.1 just works fine: curl -O -L https://download.ceph.com/rpm-tentacle/el9/SRPMS/ceph-20.2.0-0.el9.src.rpm mkdir rpm mock -r /etc/mock/alma+epel-10-x86_64.cfg rebuild ./ceph-20.2.0-0.el9.src.rpm --enable-network --resultdir rpm # or mock -r /etc/mock/rocky+epel-10-x86_64.cfg rebuild ./ceph-20.2.0-0.el9.src.rpm --enable-network --resultdir rpm both work fine. Best Moritz

1 0

scrape-health-metrics fails with sudo PAM account management error
by Flemming Frandsen 12 Dec '25

12 Dec '25

I've been building a new cluster with cephadm, the OS is Ubuntu 24.04 and I'm using the ubuntu provided host packages, docker is 29.1.2 and containerd is 2.2.0 and the ceph release is squid 19.2.3. Everything seems to work just perfectly, except for scrape-health-metrics, which records this result (for all of the osds in the 4 hosts in the cluster): "20251212-074916": { "dev": "/dev/nvme5n1", "error": "smartctl failed", "nvme_smart_health_information_add_log_error": "nvme returned an error: sudo: exit status: 1", "nvme_smart_health_information_add_log_error_code": -22, "nvme_vendor": "samsung", "smartctl_error_code": -22, "smartctl_output": "smartctl returned an error (1): stderr:\nsudo: exit status: 1\nstdout:\n" }, Digging though the source I found that it's the OSD container that runs the command: sudo /usr/sbin/smartctl -x --json=o /dev/nvme5n1 I can run the command in the container using: docker exec -it -u ceph ceph-eac6da30-d72a-11f0-88be-7cc255639332-osd-3 sudo /usr/sbin/smartctl -x --json=o /dev/nvme5n1 The result of the manual command is: sudo: PAM account management error: Authentication service cannot retrieve authentication info sudo: a password is required I have enabled debugging for sudo, in the container and verified that the command that the osd is running, is same one and that the error is the same as well, running ceph device scrape-health-metrics causes this sudo debug output: Dec 12 10:30:35 sudo[845] user command "/usr/sbin/smartctl -x --json=o /dev/nvme5n1" matches sudoers command "/usr/sbin/smartctl -x --json=o /dev/*": true @ command_matches() ./match_command.c:667 Dec 12 10:30:35 sudo[845] userspec matched @ /etc/sudoers.d/ceph-smartctl:4:57: allowed @ sudoers_lookup_check() ./parse.c:167 Dec 12 10:30:35 sudo[845] sudo_putenv: SUDO_COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme5n1 Dec 12 10:30:35 sudo[845] <- new_logline @ ./eventlog.c:218 := PAM account management error: Authentication service cannot retrieve authentication info ; TTY=pts/8 ; PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme5n1 My only guess is that I have chosen a too-new Ubuntu version for Ceph and I should just bite the bullet and re-install with Ubuntu 22.04, but if anyone has a better idea, please let me know. -- Flemming Frandsen - YAPH - http://osaa.dk - http://dren.dk/

1 0

Mechanism behind the fix for EC Data Corruption (min_size=k) in Tentacle v20.2.0
by 유찬우 12 Dec '25

12 Dec '25

Hi Ceph Community, I am writing to inquire about a specific behavior change regarding data consistency in Erasure Coded pools between Ceph Tentacle v20.1.0 and v20.2.0. I previously encountered a deterministic data corruption issue during recovery in v20.1.0. After upgrading to v20.2.0 and repeating the same test, the issue appears to be resolved. I am interested in understanding the technical logic behind this fix. Here are the details of my test scenario: Cluster Configuration Version: Tested on Tentacle v20.1.0 (Issue observed) vs. v20.2.0 (Issue resolved) Layout: 8 OSD hosts EC Profile: k=6, m=2 (8 total shards) min_size: Manually lowered from 7 to 6 (k) Test Scenario (Reproduction Steps) 1. Client Setup: Created RBD images, formatted with ext4, and mounted them on 3 separate clients (1 TB image per client). 2. I/O Start: Started writing data patterns to the mount points until the mounted filesystems were filled to capacity. 3. Failure Injection: While writing, I forcibly brought down 2 OSD hosts. 4. Write Completion: Since min_size=6, writes continued and succeeded in a degraded state (Acting set size = 6). 5. Pre-Recovery Verification: Before bringing the nodes back up, I verified the data (MD5 checksum). - Result: Data was intact/readable (likely reading the 6 available systematic chunks). 6. Recovery: I brought the 2 down OSD hosts back up. The cluster performed recovery/peering and reached Active+Clean. 7. Post-Recovery Verification: After the cluster became Clean, I performed the integrity check again   The Observation On v20.1.0 MD5 Mismatch. The data became corrupted after recovery. This happened with 100/100 files tested. On v20.2.0 No data loss or corruption. MD5 checksums matched perfectly after recovery. It is great to see that v20.2.0 has resolved this critical issue. I am curious to learn what caused the corruption in the earlier version and how the recovery logic has been improved to handle this scenario correctly. Could anyone share insights into the root cause or the nature of the fix? If there are specific Pull Requests or Issue tracker links available for reference, I would greatly appreciate it. Thank you for your time and assistance. Best regards, Chanu Yu

1 0

Re: rgw index auto resharding not acting
by Anthony D'Atri 11 Dec '25

11 Dec '25

I don't recall where it is in the pipeline, but AIUI this has been recognized and there has been recent or incipient code work for resharding to account for versioned objects (and to reshard without blocking the bucket). > On Dec 11, 2025, at 5:12 AM, Johan Thomsen via ceph-users <ceph-users(a)ceph.io> wrote: > > I looks like "limit check" counts _all_ objects, but the resharder only > uses stats from rgw.main. Is that correct? > > If so, why? Shouldn't the auto-resharding act on the total number of > objects in the bucket? > > I have around 5 million objects in rgw.none and 3 mil in rgw.main... limit > check reports 8-9 million objects. > > (ceph version 19.2.3)

2 2

Re: [mgmt-gateway] monitoring problems, origin not allowed in grafana
by Redouane Kachach 11 Dec '25

11 Dec '25

Hi Gilles, I was able to reproduce a similar problem locally (the error related to "origin not allowed). As of my local testing (and observations) Grafana was failing with “origin not allowed” only when user login was enabled (user/password), but everything worked fine when using anonymous access. I think this difference is due to how Grafana’s CSRF protection works: With anonymous access: - You are effectively a non-authenticated user. - Most requests are simple GETs and there is no session cookie to protect. - Grafana’s CSRF checks are much less strict, so it doesn’t really care about the Origin header. With user login enabled: - After logging in, the browser sends requests with a session cookie and an Origin header like: Origin: https://<your-fqdn-domain> - Because this is now an authenticated session, Grafana enforces strict CSRF checks. - If the Origin does not exactly match what Grafana considers its own URL or trusted origins (Grafana is configured with an internal hostname, while you access it via the mgmt-gateway FQDN), Grafana rejects the request with: “origin not allowed”. I think that explains why the problem only appears once user authentication is enabled. The problem is that Grafana is deployed in this case behind mgmt-gateway (which acts as reverse-proxy). So Grafana has no idea what's the fqdn in use as it comes from the mgmt-gateway. When oauth2-proxy is also used so authentication goes through SSO this issue doesn't happen. To solve the problem, grafana provides a config field csrf_trusted_origins which can be used to indicate manually your trusted origins. In this case, you would need to add something like: [security] csrf_trusted_origins = <your-fqdn-domain> For example: [security] csrf_trusted_origins = my-cloud.ceph-node-0.com In your grafana.ini (located at /var/lib/ceph/<fsid>/grafana.<your-grafana-node>/etc/grafana/grafana.ini To test this quickly you can edit .ini manually and then run "ceph orch restart grafana" from the cephadm shell. This change is unfortunately not permanent, if you redeploy your grafana or move it to another node, you will lose the change. To make it permanent you have to adjust the grafana Jinja template, used by cephadm to generate the ini file. For that you have to modify the template to add the above entry and store it for example in a grafana.ini.j2 file then run the following commands: ceph config-key set mgr/cephadm/services/grafana/grafana.ini -i grafana.ini.j2 ceph orch reconfig grafana To get the template you can either find it in github, for tentacle (plz adjust the path to your exact release): https://github.com/ceph/ceph/blob/tentacle/src/pybind/mgr/cephadm/templates… or directly from the ceph-mgr container: /usr/share/ceph/mgr/cephadm/templates/services/grafana/grafana.ini.j2 Best, Redouane. On Thu, Dec 11, 2025 at 9:27 AM Redouane Kachach via ceph-users < ceph-users(a)ceph.io> wrote: > Thank you Gilles for updating the ticket with the info, > > One last thing that could shed some light is to see the cert used by > Prometheus (then let's continue the discussion on the tracker), > > openssl x509 -in > /var/lib/ceph/<fsid>/prometheus.<host>/etc/prometheus/prometheus.crt -text > > (Plz omit any sensitive information for your lab from the output) > > BTW: there's a Slack channel dedicated for cephadm that can be used for a > more interactive dicussion > https://ceph-storage.slack.com/archives/C04SNUBD2M6 > > Best, > Redouane. > > > > On Wed, Dec 10, 2025 at 5:06 PM Gilles Mocellin < > gilles.mocellin(a)nuagelibre.org> wrote: > > > Thank you Redouane, > > > > I have updated the ticket with all what you made me do and what you > > asked. > > Except that "--include-cephadm-signed" does not exist on my version for > > `ceph orch certmgr cert ls`. > > > > https://tracker.ceph.com/issues/74164 > > > > Le 2025-12-10 15:36, Redouane Kachach a écrit : > > > Thanks for the feedback Gilles. > > > > > > Using your own cert for the mgmt-gateway is Okay and shouldn't be an > > > issue. > > > Errors arise when you use custom certs for the Dashboard itself as > > > bcz when the mgmt-gateway is enabled mTLS is enforced for internal > > > communications (Dashboard <-> Prometheus and Alertmanager <-> Dashbaord > > > i.e) > > > > > > From what you are commenting it seems like Grafana is not able to get > > > metrics from Prometheus. > > > > > > Can you plz update the ticket with the output of the command (obviously > > > omitting any private/sensitive data): > > > > > >> ceph orch certmgr cert ls --include-cephadm-signed > > > > > > This would show all the certificates in use for the different services. > > > > > > It may also help see the config of Grafana: > > > > > > cat /var/lib/ceph/<fsid>/grafana.<host>/etc/grafana/grafana.ini > > > cat > > > > > > /var/lib/ceph/<fsid>/grafana.<host>/etc/grafana/provisioning/datasources/ceph-dashboard.yml > > > > > > > > > Thanks, > > > Redouane. > > > > > > > > > > > > On Wed, Dec 10, 2025 at 2:33 PM Gilles Mocellin < > > > gilles.mocellin(a)nuagelibre.org> wrote: > > > > > >> Better but not completely. > > >> > > >> I don't have errors anymore concerning alertmanager, and I can see > > >> alerts, thereis on concerning mgr/prometheus module not responding. > > >> Even if I disable/enable it. > > >> > > >> In the MGR logs, I see certificate problem : > > >> > > >> Dec 10 13:27:40 fidcl-lyo1-sto-sds-lab-01 > > >> > > > ceph-8ec7575a-7de5-11f0-a78a-246e96bd90a4-mgr-fidcl-lyo1-sto-sds-lab-01-gtcifn[7494]: > > >> > > >> [10/Dec/2025:13:27:40] ENGINE Client ('::ffff:100.70.100.51', 55206, > > >> 0, > > >> 0) lost — peer dropped the TLS connection suddenly, during handshake: > > >> (1, '[SSL: SSLV3_ALERT_BAD_CERTIFICATE] ssl/tls alert bad certificate > > >> (_ssl.c:1147)') > > >> Dec 10 13:27:40 fidcl-lyo1-sto-sds-lab-01 ceph-mgr[7789]: [prometheus > > >> INFO cherrypy.error] [10/Dec/2025:13:27:40] ENGINE Client > > >> ('::ffff:100.70.100.51', 55206, 0, 0) lost — peer dropped the TLS > > >> connection suddenly, during handshake: (1, '[SSL: > > >> SSLV3_ALERT_BAD_CERTIFICATE] ssl/tls alert bad certificate > > >> (_ssl.c:1147)') > > >> > > >> Grafana Dashboard still don't work. > > >> In embeded mode, I still have Unauthorized messages. > > >> If I open directly Grafana, with the DNS name, I can list the > > >> Dashboard, > > >> but in them, there is still "origin not allowed" messages. > > >> I cannot log in, with also the message "origin not allowed". > > >> > > >> > > >> Le 2025-12-10 14:15, Gilles Mocellin a écrit : > > >> > Hi Redouane, > > >> > > > >> > Yes, I'm using my own certificates, signed by our internal PKI. > > >> > > > >> > I will try, but I still want a valid certificate for my DNS entry, > > >> > certificate guven to the mgmt-gateway service. > > >> > > > >> > Le 10 décembre 2025 13:28:45 GMT+01:00, Redouane Kachach > > >> > <rkachach(a)redhat.com> a écrit : > > >> >> Hi Gilles, > > >> >> > > >> >> What certificate are you using on your Dashboard? > > >> >> > > >> >> Just to rule out any certificate issues: when using mgmt-gateway, > the > > >> >> dashboard must be configured to use cephadm-signed certificates. > > >> >> This is normally done automatically, but if you were using custom > > >> >> certificates for your dashboard, those will not be overwritten by > > >> >> cephadm. > > >> >> If this is the case then plz just follow this procedure to fix > them: > > >> >> > > >> >> 1) generate cephadm-signed certs for the dashboard (from cephadm > > >> >> shell): > > >> >> > > >> >> ceph orch certmgr generate-certificates dashboard \ > > >> >> | tee \ > > >> >> >(jq -r '.cert' > dashboard.cert.pem) \ > > >> >> >(jq -r '.key' > dashboard.key.pem) \ > > >> >> > /dev/null > > >> >> > > >> >> 2) The above will generate two files: dashboard.cert.pem and > > >> >> dashboard.key.pem, now we have to use them to set the dashboard > > >> >> cert/key: > > >> >> > > >> >> ceph dashboard set-ssl-certificate -i dashboard.cert.pem > > >> >> ceph dashboard set-ssl-certificate-key -i dashboard.key.pem > > >> >> > > >> >> 3) Reload the dashboard, so it starts using the new populated > > >> >> certificates: > > >> >> > > >> >> ceph mgr module disable dashboard; ceph mgr module enable > dashboard > > >> >> --force > > >> >> > > >> >> Best, > > >> >> Redouane. > > >> >> > > >> >> > > >> >> On Wed, Dec 10, 2025 at 11:31 AM Gilles Mocellin via ceph-users < > > >> >> ceph-users(a)ceph.io> wrote: > > >> >> > > >> >>> Hello ! > > >> >>> > > >> >>> Does someone manage to use mgmt-gateway, with a FQDN, and the > > >> >>> monitoring > > >> >>> stack working ? > > >> >>> > > >> >>> In my LAB, after upgrading from 10.2.3 to 20.2.0, > > >> >>> My dashboard was working fine, with grafana dashboards. > > >> >>> > > >> >>> I decided to test mgmt-gateway. > > >> >>> I use a VIP with a specific DNS name pointing at that VIP. > > >> >>> I have a certificate generated with our internal PKI, valid for > the > > >> >>> DNS > > >> >>> name and the VIP. > > >> >>> > > >> >>> Dashboard works, but no monitoring anymore. > > >> >>> - Popups concerning alermanager : > > >> >>> > > >> >>> Failed to execute prometheus > > >> >>> Error parsing Prometheus Alertmanager response: Expecting value > > >> >>> 9/12/25 04:07 PM > > >> >>> > > >> >>> - No graphs in dashboard, with errors "origin not allowed" in each > > >> >>> panel > > >> >>> > > >> >>> If I open Grafana directly, I see : > > >> >>> - that the VIP is used, not my DNS name, but none works. > > >> >>> - Grafana works but I can't see dashboards, and if I put a URL > that > > >> >>> should work (https://$VIP/grafana/d/edtb0oxdq/ceph-cluster), I > > have a > > >> >>> message : > > >> >>> > > >> >>> Failed to load dashboard > > >> >>> Unauthorized > > >> >>> > > >> >>> Some more infos : > > >> >>> > > >> >>> https://$DNS/grafana/dashboards => show the list of Dashboards > > >> >>> Open a dashboard show errors 'origin not allowed" on panels > > >> >>> > > >> >>> https://$VIP/grafana/dashboards => cannot list the Dashboards, > > error > > >> >>> Unauthorized > > >> >>> > > >> >>> Concerning the Altermanger errors, I see thoses XHR calls in my > > >> >>> browser, > > >> >>> ending in HTTP error 400 Bad Request : > > >> >>> https://$DNS/api/prometheus?cluster_filter=true > > >> >>> > > >> >>> I opened an issue here : https://tracker.ceph.com/issues/74164 > > >> >>> > > >> >>> -- > > >> >>> Gilles > > >> >>> _______________________________________________ > > >> >>> ceph-users mailing list -- ceph-users(a)ceph.io > > >> >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > >> >>> > > >> >>> > > >> > > >> > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

Re: Version compatibility with QEMU/KVM
by Anthony D'Atri 11 Dec '25

11 Dec '25

The clients being post-Luminous is the biggest factor here, so they can handle pg-umap. > On Dec 10, 2025, at 6:39 AM, Kees Meijs | Nefos via ceph-users <ceph-users(a)ceph.io> wrote: > >> And as far as ceph stuff goes, RBD is (when used with mostly defaults) >> is quite simple and the protocol or stored data isn't changing much >> between releases. It does develop new features like being able to be >> stored on EC pools, fast diffs and things like that, but the basic >> read and write operations are quite the same.

2 1

Re: rgw index auto resharding not acting
by Janne Johansson 11 Dec '25

11 Dec '25

> I looks like "limit check" counts _all_ objects, but the resharder only > uses stats from rgw.main. Is that correct? > > If so, why? Shouldn't the auto-resharding act on the total number of > objects in the bucket? > > I have around 5 million objects in rgw.none and 3 mil in rgw.main... limit > check reports 8-9 million objects. In the older days, ceph sometimes got confused an thought there were 18 billion million objects belonging to the category rgw.none, and based its reshard decisions on that, which also wasn't so good. -- May the most significant bit of your life be positive.

2 1

Re: Changing crush map and editing crush rules to match
by Torkil Svensgaard 11 Dec '25

11 Dec '25

On 11/12/2025 12:27, Torkil Svensgaard via ceph-users wrote: > Hi > > Ceph squid. We currently have all our ceph servers at one physical > location but in different rooms. We are looking at creating a new pool > with some servers placed at a different physical location. > > Current crush map is this: > > root default > datacenter 714 > hosts > datacenter HX1 > hosts > datacenter UX1 > hosts > > Sample rule: > > { > "rule_id": 0, > "rule_name": "rbd_ec_data", > "type": 3, > "steps": [ > { > "op": "set_chooseleaf_tries", > "num": 5 > }, > { > "op": "set_choose_tries", > "num": 100 > }, > { > "op": "take", > "item": -24, > "item_name": "default~hdd" > }, > { > "op": "choose_indep", > "num": 0, > "type": "datacenter" > }, > { > "op": "chooseleaf_indep", > "num": 2, > "type": "host" > }, > { > "op": "emit" > } > ] > }, > > New crush map: > > root default > datacenter HVH > room 714 > hosts > room HX1 > hosts > room UX1 > hosts > datacenter TBA > hosts > > Edited rule, changed root from default to HVH and failure domain from > datacenter to room: Not failure domain, just changed datacenter to room for the relevant step. Mvh. Torkil > > { > "rule_id": 0, > "rule_name": "rbd_ec_data", > "type": 3, > "steps": [ > { > "op": "set_chooseleaf_tries", > "num": 5 > }, > { > "op": "set_choose_tries", > "num": 100 > }, > { > "op": "take", > "item": -24, > "item_name": "HVH~hdd" > }, > { > "op": "choose_indep", > "num": 0, > "type": "room" > }, > { > "op": "chooseleaf_indep", > "num": 2, > "type": "host" > }, > { > "op": "emit" > } > ] > }, > > Does that look right? Can we just set norebalance, move hosts around in > the crush map, edit our crush rules, run pgremapper and unset norebalance? > > Mvh. > > Torkil > -- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil(a)drcmr.dk

1 0

Changing crush map and editing crush rules to match
by Torkil Svensgaard 11 Dec '25

11 Dec '25

Hi Ceph squid. We currently have all our ceph servers at one physical location but in different rooms. We are looking at creating a new pool with some servers placed at a different physical location. Current crush map is this: root default datacenter 714 hosts datacenter HX1 hosts datacenter UX1 hosts Sample rule: { "rule_id": 0, "rule_name": "rbd_ec_data", "type": 3, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -24, "item_name": "default~hdd" }, { "op": "choose_indep", "num": 0, "type": "datacenter" }, { "op": "chooseleaf_indep", "num": 2, "type": "host" }, { "op": "emit" } ] }, New crush map: root default datacenter HVH room 714 hosts room HX1 hosts room UX1 hosts datacenter TBA hosts Edited rule, changed root from default to HVH and failure domain from datacenter to room: { "rule_id": 0, "rule_name": "rbd_ec_data", "type": 3, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -24, "item_name": "HVH~hdd" }, { "op": "choose_indep", "num": 0, "type": "room" }, { "op": "chooseleaf_indep", "num": 2, "type": "host" }, { "op": "emit" } ] }, Does that look right? Can we just set norebalance, move hosts around in the crush map, edit our crush rules, run pgremapper and unset norebalance? Mvh. Torkil -- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil(a)drcmr.dk

1 0

Re: Tentacle 20.2.0 NFS ganesh and mgmt-gateway - The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8)
by Salatiel Filho 11 Dec '25

11 Dec '25

Does anyone know where I can get the Dockerfile ceph uses to create those containers for amd64 so I can try to build them for arm ? Atenciosamente/Kind regards, Salatiel On Fri, Dec 5, 2025 at 8:07 PM Salatiel Filho via ceph-users <ceph-users(a)ceph.io> wrote: > > Are there any plans to make a linux/arm64/v8 container image for > quay.io/ceph/keepalived:2.2.4, quay.io/ceph/haproxy:2.3 and > quay.io/ceph/nginx:sclorg-nginx-126 ? > Currently it is not possible to start a HA NFS Ganesh in ceph or even > the management gateway on ARM devices. > They all complain about: > WARNING: The requested image's platform (linux/amd64) does not match > the detected host platform (linux/arm64/v8) and no specific platform > was requested > > Atenciosamente/Kind regards, > Salatiel > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: rgw index auto resharding not acting
by Johan Thomsen 11 Dec '25

11 Dec '25

I looks like "limit check" counts _all_ objects, but the resharder only uses stats from rgw.main. Is that correct? If so, why? Shouldn't the auto-resharding act on the total number of objects in the bucket? I have around 5 million objects in rgw.none and 3 mil in rgw.main... limit check reports 8-9 million objects. (ceph version 19.2.3) Den man. 8. dec. 2025 kl. 10.10 skrev Johan Thomsen via ceph-users < ceph-users(a)ceph.io>: > Hi > > For some reason, the auto-resharding is not doing anything, even though it > reports fill status over 100%. See below: > > $ radosgw-admin bucket limit check --rgw-zonegroup=k1-objectstore > --rgw-zone=k1-objectstore > > > 2025-12-08T08:45:57.143+0000 7f8921528900 0 ERROR: current period > 43e92ddb-d0d2-4fbc-b18b-dc87126a7664 does not contain zone id > 1da0f602-9397-4997-8e76-3e3425468746 > > 2025-12-08T08:45:57.166+0000 7f8921528900 0 period > (43e92ddb-d0d2-4fbc-b18b-dc87126a7664 does not have zone > 1da0f602-9397-4997-8e76-3e3425468746 configured > > [ > > { > > "user_id": "k1-ceph-user", > > "buckets": [ > > { > > "bucket": "k1-snapshots", > > "tenant": "", > > "num_objects": 5766564, > > "num_shards": 97, > > "objects_per_shard": 59449, > > "fill_status": "OVER 114%" > > } > > ] > > } > > > Here's a dump of related config: > > > rgw_dynamic_resharding true > > rgw_max_dynamic_shards 1999 > > rgw_max_objs_per_shard 50000 > > rgw_md_log_max_shards 64 > > rgw_objexp_hints_num_shards 127 > > rgw_override_bucket_index_max_shards 1031 > > rgw_reshard_batch_size 64 > > rgw_reshard_bucket_lock_duration 360 > > rgw_reshard_max_aio 128 > > rgw_reshard_num_logs 16 > > rgw_reshard_thread_interval 600 > > rgw_safe_max_objects_per_shard 52000 > > rgw_shard_warning_threshold 90.000000 > > rgw_usage_max_shards 32 > > > Does anyone have an idea why it is not auto-resharding to a value below at > least 52k objs per shard? > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Re: [mgmt-gateway] monitoring problems, origin not allowed in grafana
by Redouane Kachach 11 Dec '25

11 Dec '25

Thank you Gilles for updating the ticket with the info, One last thing that could shed some light is to see the cert used by Prometheus (then let's continue the discussion on the tracker), openssl x509 -in /var/lib/ceph/<fsid>/prometheus.<host>/etc/prometheus/prometheus.crt -text (Plz omit any sensitive information for your lab from the output) BTW: there's a Slack channel dedicated for cephadm that can be used for a more interactive dicussion https://ceph-storage.slack.com/archives/C04SNUBD2M6 Best, Redouane. On Wed, Dec 10, 2025 at 5:06 PM Gilles Mocellin < gilles.mocellin(a)nuagelibre.org> wrote: > Thank you Redouane, > > I have updated the ticket with all what you made me do and what you > asked. > Except that "--include-cephadm-signed" does not exist on my version for > `ceph orch certmgr cert ls`. > > https://tracker.ceph.com/issues/74164 > > Le 2025-12-10 15:36, Redouane Kachach a écrit : > > Thanks for the feedback Gilles. > > > > Using your own cert for the mgmt-gateway is Okay and shouldn't be an > > issue. > > Errors arise when you use custom certs for the Dashboard itself as > > bcz when the mgmt-gateway is enabled mTLS is enforced for internal > > communications (Dashboard <-> Prometheus and Alertmanager <-> Dashbaord > > i.e) > > > > From what you are commenting it seems like Grafana is not able to get > > metrics from Prometheus. > > > > Can you plz update the ticket with the output of the command (obviously > > omitting any private/sensitive data): > > > >> ceph orch certmgr cert ls --include-cephadm-signed > > > > This would show all the certificates in use for the different services. > > > > It may also help see the config of Grafana: > > > > cat /var/lib/ceph/<fsid>/grafana.<host>/etc/grafana/grafana.ini > > cat > > > /var/lib/ceph/<fsid>/grafana.<host>/etc/grafana/provisioning/datasources/ceph-dashboard.yml > > > > > > Thanks, > > Redouane. > > > > > > > > On Wed, Dec 10, 2025 at 2:33 PM Gilles Mocellin < > > gilles.mocellin(a)nuagelibre.org> wrote: > > > >> Better but not completely. > >> > >> I don't have errors anymore concerning alertmanager, and I can see > >> alerts, thereis on concerning mgr/prometheus module not responding. > >> Even if I disable/enable it. > >> > >> In the MGR logs, I see certificate problem : > >> > >> Dec 10 13:27:40 fidcl-lyo1-sto-sds-lab-01 > >> > ceph-8ec7575a-7de5-11f0-a78a-246e96bd90a4-mgr-fidcl-lyo1-sto-sds-lab-01-gtcifn[7494]: > >> > >> [10/Dec/2025:13:27:40] ENGINE Client ('::ffff:100.70.100.51', 55206, > >> 0, > >> 0) lost — peer dropped the TLS connection suddenly, during handshake: > >> (1, '[SSL: SSLV3_ALERT_BAD_CERTIFICATE] ssl/tls alert bad certificate > >> (_ssl.c:1147)') > >> Dec 10 13:27:40 fidcl-lyo1-sto-sds-lab-01 ceph-mgr[7789]: [prometheus > >> INFO cherrypy.error] [10/Dec/2025:13:27:40] ENGINE Client > >> ('::ffff:100.70.100.51', 55206, 0, 0) lost — peer dropped the TLS > >> connection suddenly, during handshake: (1, '[SSL: > >> SSLV3_ALERT_BAD_CERTIFICATE] ssl/tls alert bad certificate > >> (_ssl.c:1147)') > >> > >> Grafana Dashboard still don't work. > >> In embeded mode, I still have Unauthorized messages. > >> If I open directly Grafana, with the DNS name, I can list the > >> Dashboard, > >> but in them, there is still "origin not allowed" messages. > >> I cannot log in, with also the message "origin not allowed". > >> > >> > >> Le 2025-12-10 14:15, Gilles Mocellin a écrit : > >> > Hi Redouane, > >> > > >> > Yes, I'm using my own certificates, signed by our internal PKI. > >> > > >> > I will try, but I still want a valid certificate for my DNS entry, > >> > certificate guven to the mgmt-gateway service. > >> > > >> > Le 10 décembre 2025 13:28:45 GMT+01:00, Redouane Kachach > >> > <rkachach(a)redhat.com> a écrit : > >> >> Hi Gilles, > >> >> > >> >> What certificate are you using on your Dashboard? > >> >> > >> >> Just to rule out any certificate issues: when using mgmt-gateway, the > >> >> dashboard must be configured to use cephadm-signed certificates. > >> >> This is normally done automatically, but if you were using custom > >> >> certificates for your dashboard, those will not be overwritten by > >> >> cephadm. > >> >> If this is the case then plz just follow this procedure to fix them: > >> >> > >> >> 1) generate cephadm-signed certs for the dashboard (from cephadm > >> >> shell): > >> >> > >> >> ceph orch certmgr generate-certificates dashboard \ > >> >> | tee \ > >> >> >(jq -r '.cert' > dashboard.cert.pem) \ > >> >> >(jq -r '.key' > dashboard.key.pem) \ > >> >> > /dev/null > >> >> > >> >> 2) The above will generate two files: dashboard.cert.pem and > >> >> dashboard.key.pem, now we have to use them to set the dashboard > >> >> cert/key: > >> >> > >> >> ceph dashboard set-ssl-certificate -i dashboard.cert.pem > >> >> ceph dashboard set-ssl-certificate-key -i dashboard.key.pem > >> >> > >> >> 3) Reload the dashboard, so it starts using the new populated > >> >> certificates: > >> >> > >> >> ceph mgr module disable dashboard; ceph mgr module enable dashboard > >> >> --force > >> >> > >> >> Best, > >> >> Redouane. > >> >> > >> >> > >> >> On Wed, Dec 10, 2025 at 11:31 AM Gilles Mocellin via ceph-users < > >> >> ceph-users(a)ceph.io> wrote: > >> >> > >> >>> Hello ! > >> >>> > >> >>> Does someone manage to use mgmt-gateway, with a FQDN, and the > >> >>> monitoring > >> >>> stack working ? > >> >>> > >> >>> In my LAB, after upgrading from 10.2.3 to 20.2.0, > >> >>> My dashboard was working fine, with grafana dashboards. > >> >>> > >> >>> I decided to test mgmt-gateway. > >> >>> I use a VIP with a specific DNS name pointing at that VIP. > >> >>> I have a certificate generated with our internal PKI, valid for the > >> >>> DNS > >> >>> name and the VIP. > >> >>> > >> >>> Dashboard works, but no monitoring anymore. > >> >>> - Popups concerning alermanager : > >> >>> > >> >>> Failed to execute prometheus > >> >>> Error parsing Prometheus Alertmanager response: Expecting value > >> >>> 9/12/25 04:07 PM > >> >>> > >> >>> - No graphs in dashboard, with errors "origin not allowed" in each > >> >>> panel > >> >>> > >> >>> If I open Grafana directly, I see : > >> >>> - that the VIP is used, not my DNS name, but none works. > >> >>> - Grafana works but I can't see dashboards, and if I put a URL that > >> >>> should work (https://$VIP/grafana/d/edtb0oxdq/ceph-cluster), I > have a > >> >>> message : > >> >>> > >> >>> Failed to load dashboard > >> >>> Unauthorized > >> >>> > >> >>> Some more infos : > >> >>> > >> >>> https://$DNS/grafana/dashboards => show the list of Dashboards > >> >>> Open a dashboard show errors 'origin not allowed" on panels > >> >>> > >> >>> https://$VIP/grafana/dashboards => cannot list the Dashboards, > error > >> >>> Unauthorized > >> >>> > >> >>> Concerning the Altermanger errors, I see thoses XHR calls in my > >> >>> browser, > >> >>> ending in HTTP error 400 Bad Request : > >> >>> https://$DNS/api/prometheus?cluster_filter=true > >> >>> > >> >>> I opened an issue here : https://tracker.ceph.com/issues/74164 > >> >>> > >> >>> -- > >> >>> Gilles > >> >>> _______________________________________________ > >> >>> ceph-users mailing list -- ceph-users(a)ceph.io > >> >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >> >>> > >> >>> > >> > >> > >

1 0

Re: Tentacle 20.2.0 NFS ganesh and mgmt-gateway - The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8)
by Salatiel Filho 10 Dec '25

10 Dec '25

Not sure if anyone is interested, but I created this issue https://tracker.ceph.com/issues/74180 Atenciosamente/Kind regards, Salatiel On Fri, Dec 5, 2025 at 8:07 PM Salatiel Filho via ceph-users <ceph-users(a)ceph.io> wrote: > > Are there any plans to make a linux/arm64/v8 container image for > quay.io/ceph/keepalived:2.2.4, quay.io/ceph/haproxy:2.3 and > quay.io/ceph/nginx:sclorg-nginx-126 ? > Currently it is not possible to start a HA NFS Ganesh in ceph or even > the management gateway on ARM devices. > They all complain about: > WARNING: The requested image's platform (linux/amd64) does not match > the detected host platform (linux/arm64/v8) and no specific platform > was requested > > Atenciosamente/Kind regards, > Salatiel > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Tentacle and Ubuntu 24
by Mark Lehrer 10 Dec '25

10 Dec '25

>> Ubuntu 24 might become fully supported and tested by Tentacle (or Squid)? The current version in the Ubuntu 24.04 repos is Squid (19.2). On Wed, Dec 10, 2025 at 12:15 PM Ryan Sleeth via ceph-users <ceph-users(a)ceph.io> wrote: > > Does the community know when Ubuntu 24 might become fully supported and > tested by Tentacle (or Squid)? I am referring to the web documentation > here: https://docs.ceph.com/en/tentacle/start/os-recommendations/ > > -- > Ryan Sleeth > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Tentacle and Ubuntu 24
by Ryan Sleeth 10 Dec '25

10 Dec '25

Does the community know when Ubuntu 24 might become fully supported and tested by Tentacle (or Squid)? I am referring to the web documentation here: https://docs.ceph.com/en/tentacle/start/os-recommendations/ -- Ryan Sleeth

1 0

Re: SMB on ARM
by John Mulligan 10 Dec '25

10 Dec '25

On Wednesday, December 10, 2025 3:07:34 AM Eastern Standard Time Robert Sander via ceph-users wrote: > Am 09.12.25 um 8:05 PM schrieb John Mulligan: > ´ > > > I have filed a PR [1]. I have tried to mark all options that can be > > supplied more than once as such in each items description list. > > > Thank you. How is the interface determined where a public address has to > be bound to when the cli argument looks like this? > > --public-address=192.168.76.110/24 > This simple form is designed to work well when the cluster is configured for a "common subnet" scenario. Let's imagine our ceph cluster is 3-nodes large and we'll be deploying an smb instance for each one. For example: node A - 192.0.2.100/24 (eth0) node B - 192.0.2.101/24 (eth0) node C - 192.0.2.102/24 (eth0) We'll use public address 192.0.2.110/24. Let's assume that the address is to be assigned to node A. The system will detect that 192.0.2.110/24 and 192.0.2.100/24 are both on the 192.0.2.0/24 network and use the eth0 device because 192.0.2.100/24 is assigned to that device. If this scenario is not possible - you don't have the same network for the public addressees and node adresses - one can use the `destination` to specify a different network to match. In the imperative cluster create command that's something like `--public-address= 198.51.100.50/24%192.0.2.0/24` to tell it to use the device matching the 192.0.2.100/24 network (eth0 in our example above)? When I have some time I may try to use these examples in future doc updates. > BTW: there are IP ranges reserved for documentation: > https://datatracker.ietf.org/doc/html/rfc5737 > > "The blocks 192.0.2.0/24 (TEST-NET-1), 198.51.100.0/24 (TEST-NET-2), > and 203.0.113.0/24 (TEST-NET-3) are provided for use in > documentation." > Thanks for the reminder. I'm usually pretty good about properly using reserved dns names for documentation, but I forget when it comes to IPs and tend to just copy-paste from my actual test clusters.

1 0

Re: [mgmt-gateway] monitoring problems, origin not allowed in grafana
by Redouane Kachach 10 Dec '25

10 Dec '25

Thanks for the feedback Gilles. Using your own cert for the mgmt-gateway is Okay and shouldn't be an issue. Errors arise when you use custom certs for the Dashboard itself as bcz when the mgmt-gateway is enabled mTLS is enforced for internal communications (Dashboard <-> Prometheus and Alertmanager <-> Dashbaord i.e) From what you are commenting it seems like Grafana is not able to get metrics from Prometheus. Can you plz update the ticket with the output of the command (obviously omitting any private/sensitive data): > ceph orch certmgr cert ls --include-cephadm-signed This would show all the certificates in use for the different services. It may also help see the config of Grafana: cat /var/lib/ceph/<fsid>/grafana.<host>/etc/grafana/grafana.ini cat /var/lib/ceph/<fsid>/grafana.<host>/etc/grafana/provisioning/datasources/ceph-dashboard.yml Thanks, Redouane. On Wed, Dec 10, 2025 at 2:33 PM Gilles Mocellin < gilles.mocellin(a)nuagelibre.org> wrote: > Better but not completely. > > I don't have errors anymore concerning alertmanager, and I can see > alerts, thereis on concerning mgr/prometheus module not responding. > Even if I disable/enable it. > > In the MGR logs, I see certificate problem : > > Dec 10 13:27:40 fidcl-lyo1-sto-sds-lab-01 > ceph-8ec7575a-7de5-11f0-a78a-246e96bd90a4-mgr-fidcl-lyo1-sto-sds-lab-01-gtcifn[7494]: > > [10/Dec/2025:13:27:40] ENGINE Client ('::ffff:100.70.100.51', 55206, 0, > 0) lost — peer dropped the TLS connection suddenly, during handshake: > (1, '[SSL: SSLV3_ALERT_BAD_CERTIFICATE] ssl/tls alert bad certificate > (_ssl.c:1147)') > Dec 10 13:27:40 fidcl-lyo1-sto-sds-lab-01 ceph-mgr[7789]: [prometheus > INFO cherrypy.error] [10/Dec/2025:13:27:40] ENGINE Client > ('::ffff:100.70.100.51', 55206, 0, 0) lost — peer dropped the TLS > connection suddenly, during handshake: (1, '[SSL: > SSLV3_ALERT_BAD_CERTIFICATE] ssl/tls alert bad certificate > (_ssl.c:1147)') > > Grafana Dashboard still don't work. > In embeded mode, I still have Unauthorized messages. > If I open directly Grafana, with the DNS name, I can list the Dashboard, > but in them, there is still "origin not allowed" messages. > I cannot log in, with also the message "origin not allowed". > > > Le 2025-12-10 14:15, Gilles Mocellin a écrit : > > Hi Redouane, > > > > Yes, I'm using my own certificates, signed by our internal PKI. > > > > I will try, but I still want a valid certificate for my DNS entry, > > certificate guven to the mgmt-gateway service. > > > > Le 10 décembre 2025 13:28:45 GMT+01:00, Redouane Kachach > > <rkachach(a)redhat.com> a écrit : > >> Hi Gilles, > >> > >> What certificate are you using on your Dashboard? > >> > >> Just to rule out any certificate issues: when using mgmt-gateway, the > >> dashboard must be configured to use cephadm-signed certificates. > >> This is normally done automatically, but if you were using custom > >> certificates for your dashboard, those will not be overwritten by > >> cephadm. > >> If this is the case then plz just follow this procedure to fix them: > >> > >> 1) generate cephadm-signed certs for the dashboard (from cephadm > >> shell): > >> > >> ceph orch certmgr generate-certificates dashboard \ > >> | tee \ > >> >(jq -r '.cert' > dashboard.cert.pem) \ > >> >(jq -r '.key' > dashboard.key.pem) \ > >> > /dev/null > >> > >> 2) The above will generate two files: dashboard.cert.pem and > >> dashboard.key.pem, now we have to use them to set the dashboard > >> cert/key: > >> > >> ceph dashboard set-ssl-certificate -i dashboard.cert.pem > >> ceph dashboard set-ssl-certificate-key -i dashboard.key.pem > >> > >> 3) Reload the dashboard, so it starts using the new populated > >> certificates: > >> > >> ceph mgr module disable dashboard; ceph mgr module enable dashboard > >> --force > >> > >> Best, > >> Redouane. > >> > >> > >> On Wed, Dec 10, 2025 at 11:31 AM Gilles Mocellin via ceph-users < > >> ceph-users(a)ceph.io> wrote: > >> > >>> Hello ! > >>> > >>> Does someone manage to use mgmt-gateway, with a FQDN, and the > >>> monitoring > >>> stack working ? > >>> > >>> In my LAB, after upgrading from 10.2.3 to 20.2.0, > >>> My dashboard was working fine, with grafana dashboards. > >>> > >>> I decided to test mgmt-gateway. > >>> I use a VIP with a specific DNS name pointing at that VIP. > >>> I have a certificate generated with our internal PKI, valid for the > >>> DNS > >>> name and the VIP. > >>> > >>> Dashboard works, but no monitoring anymore. > >>> - Popups concerning alermanager : > >>> > >>> Failed to execute prometheus > >>> Error parsing Prometheus Alertmanager response: Expecting value > >>> 9/12/25 04:07 PM > >>> > >>> - No graphs in dashboard, with errors "origin not allowed" in each > >>> panel > >>> > >>> If I open Grafana directly, I see : > >>> - that the VIP is used, not my DNS name, but none works. > >>> - Grafana works but I can't see dashboards, and if I put a URL that > >>> should work (https://$VIP/grafana/d/edtb0oxdq/ceph-cluster), I have a > >>> message : > >>> > >>> Failed to load dashboard > >>> Unauthorized > >>> > >>> Some more infos : > >>> > >>> https://$DNS/grafana/dashboards => show the list of Dashboards > >>> Open a dashboard show errors 'origin not allowed" on panels > >>> > >>> https://$VIP/grafana/dashboards => cannot list the Dashboards, error > >>> Unauthorized > >>> > >>> Concerning the Altermanger errors, I see thoses XHR calls in my > >>> browser, > >>> ending in HTTP error 400 Bad Request : > >>> https://$DNS/api/prometheus?cluster_filter=true > >>> > >>> I opened an issue here : https://tracker.ceph.com/issues/74164 > >>> > >>> -- > >>> Gilles > >>> _______________________________________________ > >>> ceph-users mailing list -- ceph-users(a)ceph.io > >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >>> > >>> > >

1 0

Re: Version compatibility with QEMU/KVM
by Janne Johansson 10 Dec '25

10 Dec '25

> We've got some QEMU/KVM work load linked to librbd 14.2.21 (with > backported patches), being complicated to "just upgrade" for the time being. > > With regard to compatibility, I'm curious about cluster versions. Are > there specific features or other caveats to look out for when connecting > such work load to an Octopus, a Pacific, a Quincy or even a Reef cluster? My guess is that it probably never would be a problem to run a newer client for an RBD image made on an older cluster, since they do set feature flags on the images, and an old release would only set flags it know about, and the later versions would of course handle them, but the opposite might not be equal, so it a Reef client creates an RBD image it might ask for flags that a later Oct client doesn't understand what to do with. And as far as ceph stuff goes, RBD is (when used with mostly defaults) is quite simple and the protocol or stored data isn't changing much between releases. It does develop new features like being able to be stored on EC pools, fast diffs and things like that, but the basic read and write operations are quite the same. -- May the most significant bit of your life be positive.

2 1

[mgmt-gateway] monitoring problems, origin not allowed in grafana
by Gilles Mocellin 10 Dec '25

10 Dec '25

Hello ! Does someone manage to use mgmt-gateway, with a FQDN, and the monitoring stack working ? In my LAB, after upgrading from 10.2.3 to 20.2.0, My dashboard was working fine, with grafana dashboards. I decided to test mgmt-gateway. I use a VIP with a specific DNS name pointing at that VIP. I have a certificate generated with our internal PKI, valid for the DNS name and the VIP. Dashboard works, but no monitoring anymore. - Popups concerning alermanager : Failed to execute prometheus Error parsing Prometheus Alertmanager response: Expecting value 9/12/25 04:07 PM - No graphs in dashboard, with errors "origin not allowed" in each panel If I open Grafana directly, I see : - that the VIP is used, not my DNS name, but none works. - Grafana works but I can't see dashboards, and if I put a URL that should work (https://$VIP/grafana/d/edtb0oxdq/ceph-cluster), I have a message : Failed to load dashboard Unauthorized Some more infos : https://$DNS/grafana/dashboards => show the list of Dashboards Open a dashboard show errors 'origin not allowed" on panels https://$VIP/grafana/dashboards => cannot list the Dashboards, error Unauthorized Concerning the Altermanger errors, I see thoses XHR calls in my browser, ending in HTTP error 400 Bad Request : https://$DNS/api/prometheus?cluster_filter=true I opened an issue here : https://tracker.ceph.com/issues/74164 -- Gilles

1 0

Re: SMB on ARM
by John Mulligan 10 Dec '25

10 Dec '25

On Monday, December 8, 2025 11:31:48 AM Eastern Standard Time Robert Sander via ceph-users wrote: > Hi, > > Am 08.12.25 um 4:02 PM schrieb Daniel Brown via ceph-users: > > > > > I did try to setup SMB > > The SMB service and the SMB MGR module are far from being production > ready. I only get errors when trying to use them on AMD64. > > Regards > -- Do you have a moment to elaborate on what happened? What kind of errors did you get? Was this on a tentacle branch build or something else?

2 5

Re: Cannot list or access SMB shares as any specified user, anonymous can list shares
by Michael Sudnick 10 Dec '25

10 Dec '25

Thanks for following up and trying to reproduce the error. I've tried the commands you posted: root@ceph0:~# cephadm enter --name smb.smb.4.0.ceph0.xcbqff smbclient -d 10 -U 'media%media' //localhost/media -c ls <trimmed output> gensec_update_send: ntlmssp[0x5579a2320a20]: subreq: 0x5579a233aec0 gensec_update_send: spnego[0x5579a2324c40]: subreq: 0x5579a233b2e0 gensec_update_done: ntlmssp[0x5579a2320a20]: NT_STATUS_OK tevent_req[0x5579a233aec0/../../auth/ntlmssp/ntlmssp.c:189]: state[2] error[0 (0x0)] state[struct gensec_ntlmssp_update_state (0x5579a233b0a0)] timer[(nil)] finish[../../auth/ntlmssp/ntlmssp.c:231] gensec_update_done: spnego[0x5579a2324c40]: NT_STATUS_MORE_PROCESSING_REQUIRED tevent_req[0x5579a233b2e0/../../auth/gensec/spnego.c:1614]: state[2] error[0 (0x0)] state[struct gensec_spnego_update_state (0x5579a233b4c0)] timer[(nil)] finish[../../auth/gensec/spnego.c:2109] SPNEGO login failed: The attempted logon is invalid. This is either due to a bad username or authentication information. Freeing parametrics: session setup failed: NT_STATUS_LOGON_FAILURE root@ceph0:~# cephadm enter --name smb.smb.4.0.ceph0.xcbqff getent passwd media Inferring fsid aca834ef-5617-47fd-be18-283faba1f0b1 media:x:1000:1000::/invalid:/bin/false root@ceph0:~# cephadm enter --name smb.smb.4.0.ceph0.xcbqff pdbedit -L Inferring fsid aca834ef-5617-47fd-be18-283faba1f0b1 (no output after pbdedit, just the next prompt line) I sadly can't for the life of me figure out where to find the logs for the samba server. -Michael Sudnick Sent from Proton Mail for Android. -------- Original Message -------- On Tuesday, 12/09/25 at 14:45 John Mulligan via ceph-users <ceph-users(a)ceph.io> wrote: On Tuesday, December 9, 2025 11:45:43 AM Eastern Standard Time Michael Sudnick via ceph-users wrote: > Sorry for the wall of text. I'm getting an error when trying to access smb > shares (or even list shares as an authenticated user. An anonymous user can > list shares) from a Linux client. Tests are performed with: podman run --rm > -it quay.io/samba.org/samba-client:latest > > # smbclient -N -L 10.0.150.77 > Anonymous login successful > > Sharename Type Comment > --------- ---- ------- > IPC$ IPC IPC Service (Samba 4.23.2) > media Disk > SMB1 disabled -- no workgroup available > > However when I attempt to specify a user: > I get the following error from the client with the following command: > # smbclient -d 10 -U media%media -L 10.0.150.77 > > gensec_update_send: ntlmssp[0x55a593b774c0]: subreq: 0x55a593b72b80 > gensec_update_send: spnego[0x55a593b75800]: subreq: 0x55a593b8b1e0 > gensec_update_done: ntlmssp[0x55a593b774c0]: NT_STATUS_OK > tevent_req[0x55a593b72b80/../../auth/ntlmssp/ntlmssp.c:189]: state[2] > error[0 (0x0)] state[struct gensec_ntlmssp_up date_state (0x55a593b72d60)] > timer[(nil)] finish[../../auth/ntlmssp/ntlmssp.c:231] gensec_update_done: > spnego[0x55a593b75800]: NT_STATUS_MORE_PROCESSING_REQUIRED > tevent_req[0x55a593b8b1e0/../../auth/gensec/spnego.c:1614]: state[2] > error[0 (0x0)] state[stru ct gensec_spnego_update_state (0x55a593b8b3c0)] > timer[(nil)] finish[../../auth/gensec/spnego.c:2109] SPNEGO login failed: > The attempted logon is invalid. This is either due to a bad username or > authentication information. session setup failed: NT_STATUS_LOGON_FAILURE > Freeing parametrics: > > My cluster and share definitions are as follows. It looks like there are a > few remnants of various attempts are getting it working: # ceph smb show > { > "resources": [ > { > "resource_type": "ceph.smb.cluster", > "cluster_id": "smb", > "auth_mode": "user", > "intent": "present", > "user_group_settings": [ > { > "source_type": "resource", > "ref": "smbeskkuhxm" > } > ], > "placement": { > "count": 5 > }, > "clustering": "always", > "public_addrs": [ > { > "address": "10.0.150.77/16" > } > ] > }, > { > "resource_type": "ceph.smb.share", > "cluster_id": "smb", > "share_id": "media", > "intent": "present", > "name": "media", > "readonly": false, > "browseable": true, > "cephfs": { > "volume": "cephfs", > "path": "/", > "subvolumegroup": "smb", > "subvolume": "media", > "provider": "samba-vfs" > }, > "login_control": [ > { > "name": "media", > "category": "user", > "access": "admin" > } > ] > }, > { > "resource_type": "ceph.smb.usersgroups", > "users_groups_id": "smbeskkuhxm", > "intent": "present", > "values": { > "users": [ > { > "name": "media", > "password": "media" > } > ], > "groups": [] > }, > "linked_to_cluster": "smb" > } > ] > } > > -Michael Sudnick Thank you for trying the SMB support out! Also, thanks for providing the JSON so I know what your configuration generally looks like. I adapted the JSON to one of my own clusters (changed the IPs and placement) and deployed it. I was able to connect to the share on my cluster: smbclient -U 'media%media' //192.168.76.202/media Try "help" to get a list of possible commands. smb: \> So I'm not entirely sure what has happened on your cluster. If I had reproduced the error I would have first tried again to see if using the smbclient inside the smb container image produced the same result, and then I would have looked to see if the users got created in the container image correctly. Some example commands: (on a ceph cluster node) # cephadm enter -i smb smbclient -U 'media%media' //localhost/media -c ls # cephadm enter -i smb getent passwd media # cephadm enter -i smb pdbedit -L You could also try to redploy with smbd logging cranked way up and see if anything interesting appears in the logs. For now you can add: {"custom_smb_global_options": { "log level": "10", "_allow_customization: "i- take-responsibility-for-all-samba-configuration-errors"}} to the cluster resource json (We're going to make enabling debug logging easier in future versions FWIW) _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Cannot list or access SMB shares as any specified user, anonymous can list shares
by John Mulligan 09 Dec '25

09 Dec '25

On Tuesday, December 9, 2025 11:45:43 AM Eastern Standard Time Michael Sudnick via ceph-users wrote: > Sorry for the wall of text. I'm getting an error when trying to access smb > shares (or even list shares as an authenticated user. An anonymous user can > list shares) from a Linux client. Tests are performed with: podman run --rm > -it quay.io/samba.org/samba-client:latest > > # smbclient -N -L 10.0.150.77 > Anonymous login successful > > Sharename Type Comment > --------- ---- ------- > IPC$ IPC IPC Service (Samba 4.23.2) > media Disk > SMB1 disabled -- no workgroup available > > However when I attempt to specify a user: > I get the following error from the client with the following command: > # smbclient -d 10 -U media%media -L 10.0.150.77 > > gensec_update_send: ntlmssp[0x55a593b774c0]: subreq: 0x55a593b72b80 > gensec_update_send: spnego[0x55a593b75800]: subreq: 0x55a593b8b1e0 > gensec_update_done: ntlmssp[0x55a593b774c0]: NT_STATUS_OK > tevent_req[0x55a593b72b80/../../auth/ntlmssp/ntlmssp.c:189]: state[2] > error[0 (0x0)] state[struct gensec_ntlmssp_up date_state (0x55a593b72d60)] > timer[(nil)] finish[../../auth/ntlmssp/ntlmssp.c:231] gensec_update_done: > spnego[0x55a593b75800]: NT_STATUS_MORE_PROCESSING_REQUIRED > tevent_req[0x55a593b8b1e0/../../auth/gensec/spnego.c:1614]: state[2] > error[0 (0x0)] state[stru ct gensec_spnego_update_state (0x55a593b8b3c0)] > timer[(nil)] finish[../../auth/gensec/spnego.c:2109] SPNEGO login failed: > The attempted logon is invalid. This is either due to a bad username or > authentication information. session setup failed: NT_STATUS_LOGON_FAILURE > Freeing parametrics: > > My cluster and share definitions are as follows. It looks like there are a > few remnants of various attempts are getting it working: # ceph smb show > { > "resources": [ > { > "resource_type": "ceph.smb.cluster", > "cluster_id": "smb", > "auth_mode": "user", > "intent": "present", > "user_group_settings": [ > { > "source_type": "resource", > "ref": "smbeskkuhxm" > } > ], > "placement": { > "count": 5 > }, > "clustering": "always", > "public_addrs": [ > { > "address": "10.0.150.77/16" > } > ] > }, > { > "resource_type": "ceph.smb.share", > "cluster_id": "smb", > "share_id": "media", > "intent": "present", > "name": "media", > "readonly": false, > "browseable": true, > "cephfs": { > "volume": "cephfs", > "path": "/", > "subvolumegroup": "smb", > "subvolume": "media", > "provider": "samba-vfs" > }, > "login_control": [ > { > "name": "media", > "category": "user", > "access": "admin" > } > ] > }, > { > "resource_type": "ceph.smb.usersgroups", > "users_groups_id": "smbeskkuhxm", > "intent": "present", > "values": { > "users": [ > { > "name": "media", > "password": "media" > } > ], > "groups": [] > }, > "linked_to_cluster": "smb" > } > ] > } > > -Michael Sudnick Thank you for trying the SMB support out! Also, thanks for providing the JSON so I know what your configuration generally looks like. I adapted the JSON to one of my own clusters (changed the IPs and placement) and deployed it. I was able to connect to the share on my cluster: smbclient -U 'media%media' //192.168.76.202/media Try "help" to get a list of possible commands. smb: \> So I'm not entirely sure what has happened on your cluster. If I had reproduced the error I would have first tried again to see if using the smbclient inside the smb container image produced the same result, and then I would have looked to see if the users got created in the container image correctly. Some example commands: (on a ceph cluster node) # cephadm enter -i smb smbclient -U 'media%media' //localhost/media -c ls # cephadm enter -i smb getent passwd media # cephadm enter -i smb pdbedit -L You could also try to redploy with smbd logging cranked way up and see if anything interesting appears in the logs. For now you can add: {"custom_smb_global_options": { "log level": "10", "_allow_customization: "i- take-responsibility-for-all-samba-configuration-errors"}} to the cluster resource json (We're going to make enabling debug logging easier in future versions FWIW)

1 0

Cannot list or access SMB shares as any specified user, anonymous can list shares
by Michael Sudnick 09 Dec '25

09 Dec '25

Sorry for the wall of text. I'm getting an error when trying to access smb shares (or even list shares as an authenticated user. An anonymous user can list shares) from a Linux client. Tests are performed with: podman run --rm -it quay.io/samba.org/samba-client:latest # smbclient -N -L 10.0.150.77 Anonymous login successful Sharename Type Comment --------- ---- ------- IPC$ IPC IPC Service (Samba 4.23.2) media Disk SMB1 disabled -- no workgroup available However when I attempt to specify a user: I get the following error from the client with the following command: # smbclient -d 10 -U media%media -L 10.0.150.77 gensec_update_send: ntlmssp[0x55a593b774c0]: subreq: 0x55a593b72b80 gensec_update_send: spnego[0x55a593b75800]: subreq: 0x55a593b8b1e0 gensec_update_done: ntlmssp[0x55a593b774c0]: NT_STATUS_OK tevent_req[0x55a593b72b80/../../auth/ntlmssp/ntlmssp.c:189]: state[2] error[0 (0x0)] state[struct gensec_ntlmssp_up date_state (0x55a593b72d60)] timer[(nil)] finish[../../auth/ntlmssp/ntlmssp.c:231] gensec_update_done: spnego[0x55a593b75800]: NT_STATUS_MORE_PROCESSING_REQUIRED tevent_req[0x55a593b8b1e0/../../auth/gensec/spnego.c:1614]: state[2] error[0 (0x0)] state[stru ct gensec_spnego_update_state (0x55a593b8b3c0)] timer[(nil)] finish[../../auth/gensec/spnego.c:2109] SPNEGO login failed: The attempted logon is invalid. This is either due to a bad username or authentication information. session setup failed: NT_STATUS_LOGON_FAILURE Freeing parametrics: My cluster and share definitions are as follows. It looks like there are a few remnants of various attempts are getting it working: # ceph smb show { "resources": [ { "resource_type": "ceph.smb.cluster", "cluster_id": "smb", "auth_mode": "user", "intent": "present", "user_group_settings": [ { "source_type": "resource", "ref": "smbeskkuhxm" } ], "placement": { "count": 5 }, "clustering": "always", "public_addrs": [ { "address": "10.0.150.77/16" } ] }, { "resource_type": "ceph.smb.share", "cluster_id": "smb", "share_id": "media", "intent": "present", "name": "media", "readonly": false, "browseable": true, "cephfs": { "volume": "cephfs", "path": "/", "subvolumegroup": "smb", "subvolume": "media", "provider": "samba-vfs" }, "login_control": [ { "name": "media", "category": "user", "access": "admin" } ] }, { "resource_type": "ceph.smb.usersgroups", "users_groups_id": "smbeskkuhxm", "intent": "present", "values": { "users": [ { "name": "media", "password": "media" } ], "groups": [] }, "linked_to_cluster": "smb" } ] } -Michael Sudnick

1 0

Version compatibility with QEMU/KVM
by Kees Meijs | Nefos 09 Dec '25

09 Dec '25

Hi list, We've got some QEMU/KVM work load linked to librbd 14.2.21 (with backported patches), being complicated to "just upgrade" for the time being. With regard to compatibility, I'm curious about cluster versions. Are there specific features or other caveats to look out for when connecting such work load to an Octopus, a Pacific, a Quincy or even a Reef cluster? Sharing real world experience would be really, really appreciated. Cheers, Kees

1 0