- ceph-users - lists.ceph.io

Ceph 14.2.22 MONs keep crashing in MDSMonitor::maybe_resize_cluster (out of range)
by Miles Goodhew 20 Jun '25

20 Jun '25

Hi, I've been called-in by a client with an ancient SUSE-based Ceph Nautilus (14.2.22) who's MONs keep dieing oddly. Apparently the issue started with MDS daemons not working and eventuallt a MON restart killed the cluster. OS: SLES 15-SP1 (out of support) Ceph: 14.2.22 "Nautilus" (Deployed with Salt... I think) 3 MONs; 5 MDSs; 3 MGRs; 4 RGWs; 336 OSDs on 21 nodes. Client services: "One of everything at least", but RBD/Openstack, S3/RGW and CephFS are big ones. After sorting out some of the logs here are some things I know: Disk space, RAM availability, inodes and network connectivity seem OK to me. After shutting-down all the MONs, MGRs and MDSes, one MON can usually be started, but it sits there spamming-out log messages like "[SERVICE_ID](probing) e6 handle_auth_request failed to assign global_id" (maybe 50 - 100 times per second). All the while the syslog shows 'e6 get_health_metrics reporting [INCREASING_NUMBER] slow ops` fairly often. This is probably due to OSDs and clients being active. If I restart one of the other MONs, the running one will die with a stack trace at (Limiting to C++/library internal calls): ``` 8: (std::__throw_out_of_range(char const*)+0x41) [0x7f2a5983fa07] 9: (MDSMonitor::maybe_resize_cluster(FSMap&, int)+0xcf0) [0x55b441e37490] 10: (MDSMonitor::tick()+0xc9) [0x55b441e38ce9] 11: (MDSMonitor::on_active()+0x28) [0x55b441e22fa8] 12: (PaxosService::_active()+0xdd) [0x55b441d7188d] 13: (Context::complete(int)+0x9) [0x55b441c888a9] 14: (void finish_contexts<std::__cxx11::list<Context*, std::allocator<Context*> > >(CephContext*, std::__cxx11::list<Context*, std::allocator<Context*> >&, int)+0xa8) [0x55b441cb2408] 15: (Paxos::finish_round()+0x76) [0x55b441d681b6] 16: (Paxos::handle_last(boost::intrusive_ptr<MonOpRequest>)+0xc1f) [0x55b441d693df] 17: (Paxos::dispatch(boost::intrusive_ptr<MonOpRequest>)+0x233) [0x55b441d69e23] 18: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x1668) [0x55b441c820b8] 19: (Monitor::_ms_dispatch(Message*)+0xa3a) [0x55b441c82b5a] 20: (Monitor::ms_dispatch(Message*)+0x26) [0x55b441cb3646] 21: (Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x26) [0x55b441cb00b6] 22: (DispatchQueue::entry()+0x1279) [0x7f2a5b188379] 23: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f2a5b238a5d] 24: (()+0x8539) [0x7f2a59db7539] 25: (clone()+0x3f) [0x7f2a58f87ecf] ``` Anyone got any clues about how to diagnose or better-yet repair this? Sorry, I know this is a bit half-baked, but I'm trying to dump this help request at COB to see if I can hook anyone's interest overnight. Thanks for at least reading this far, M0les.

2 14

Securing Public RGW /admin access
by Deep Dish 19 Jun '25

19 Jun '25

Hello, I'm using Ceph Squid, currently 19.2.1. I'd like to use either the built-in ingress service or an approach with the RGW service to restrict /admin access from public endpoints. My setup: Firewall (NGINX reverse proxy - TCP 443 stream passthrough only - no SSL) | Ingress Service (SSL Certificates applied) | RGW Gateways (SSL Certificates applied) All administrative functions interact with RGW Gateways directly. Only public flows pass through the Ingress Service. As I understand /admin API is required for gateway and RGW service management. I want to manage my SSL certs ideally via ceph spec files (we have automation for that). The frontend firewall is designed to act as a simple L3 / L4 ACL. Is there a way to modify the Ingress service spec file to restrict /admin access. According to GROK adding the following to the ingress spec YAML should work, but the custom_configs section throws an error as follows. I can't seem to find any official documentation around this. Feels like it should be something to easily configure. How are others securing this? Thanks. --- service_type: ingress service_id: ingress01 placement: hosts: - ingress01 spec: backend_service: rgw.public virtual_ip: 10.1.1.1/24 frontend_port: 443 monitor_port: 1967 custom_configs: - | frontend frontend acl block_admin path_beg /admin http-request deny if block_admin ssl_cert: | -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- -----BEGIN PRIVATE KEY----- ... -----END PRIVATE KEY----- Error EINVAL: Traceback (most recent call last): File "/usr/share/ceph/mgr/mgr_module.py", line 1862, in _handle_command return self.handle_command(inbuf, cmd) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 184, in handle_command return dispatch[cmd['prefix']].call(self, cmd, inbuf) File "/usr/share/ceph/mgr/mgr_module.py", line 499, in call return self.func(mgr, **kwargs) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 120, in <lambda> wrapper_copy = lambda *l_args, **l_kwargs: wrapper(*l_args, **l_kwargs) # noqa: E731 File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 109, in wrapper return func(*args, **kwargs) File "/usr/share/ceph/mgr/orchestrator/module.py", line 1697, in apply_misc return self._apply_misc(specs, dry_run, format, no_overwrite) File "/usr/share/ceph/mgr/orchestrator/module.py", line 1700, in _apply_misc completion = self.apply(specs, no_overwrite) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 1737, in inner completion = self._oremote(method_name, args, kwargs) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 1806, in _oremote mgr.log.debug("_oremote {} -> {}.{}(*{}, **{})".format(mgr.module_name, o, meth, args, kwargs)) File "/lib/python3.9/site-packages/ceph/deployment/service_spec.py", line 1109, in __repr__ y = yaml.dump(cast(dict, self), default_flow_style=False) File "/lib64/python3.9/site-packages/yaml/__init__.py", line 290, in dump return dump_all([data], stream, Dumper=Dumper, **kwds) File "/lib64/python3.9/site-packages/yaml/__init__.py", line 278, in dump_all dumper.represent(data) File "/lib64/python3.9/site-packages/yaml/representer.py", line 27, in represent node = self.represent_data(data) File "/lib64/python3.9/site-packages/yaml/representer.py", line 48, in represent_data node = self.yaml_representers[data_types[0]](self, data) File "/lib/python3.9/site-packages/ceph/deployment/service_spec.py", line 1122, in yaml_representer return dumper.represent_dict(cast(Mapping, data.to_json().items())) File "/lib/python3.9/site-packages/ceph/deployment/service_spec.py", line 1062, in to_json ret['custom_configs'] = [c.to_json() for c in self.custom_configs] File "/lib/python3.9/site-packages/ceph/deployment/service_spec.py", line 1062, in <listcomp> ret['custom_configs'] = [c.to_json() for c in self.custom_configs] AttributeError: 'str' object has no attribute 'to_json' ---

1 0

ceph rbd mirror snapshot-based does not work if mirror-mode is pool and not image
by Julien Laurenceau 19 Jun '25

19 Jun '25

Hi, I have 2 running ceph squid clusters (19.2.2). On each cluster there is an rbd pool named k8s-1 that I want to mirror using the snapshot based mode. if I configure both clusters in two way mirroring using the mode=image everything works fine. ``` rbd mirror pool enable ${mypool} image rbd mirror image enable ${mypool}/${myimage} snapshot ``` When I take a snapshot of myimage it gets mirrored. if I configure both clusters in two way mirroring using the mode=pool nothings happen. The snapshot are never mirrored in the second cluster. Reading the documentation https://docs.ceph.com/en/squid/rbd/rbd-mirroring/, I think that the problem is that the default mode for this command is "journal": rbd mirror image enable {pool-name}/{image-name} {mode} But, when enabling mirror on the pool there is no way to define mode=journal/snapshot: rbd mirror pool enable [--site-name {local-site-name}] {pool-name} {mode} with mode=pool (default) or image. Then I think when I do : rbd mirror pool enable ${mypool} image I think it implicitly do : rbd mirror pool enable ${mypool} image journal # THIS COMMAND DOES NOT WORK. But I would need to have: rbd mirror pool enable ${mypool} image snapshot # THIS COMMAND DOES NOT WORK. In addition, when I am configured on mirror mode = pool, if I try to disable mirroring on an image I get the following error : rbd mirror image disable ${mypool}/${myimage} 2025-06-18T15:37:42.177+0000 7f24ec53ce00 -1 librbd::api::Mirror: image_disable: cannot disable mirroring in the current pool mirroring mode Also error when I am configured on mirror mode = pool, if I try to enable snapshot mirroring on an image : rbd mirror image enable ${mypool}/${myimage} snapshot 2025-06-18T15:39:44.507+0000 7f533df1fe00 -1 librbd::api::Mirror: image_enable: cannot enable mirroring: pool is not in image mirror mode Could someone please confirm, that this is not supported ? What is the best workaround ? Cron cephadm shell script to enable mirror on all images ? PS : I do not want to use the journal based mode since the ceph-csi mounter nbd is not prod-ready.

2 3

RadosGW: Even more large omap objects after resharding
by Niklaus Hofer 19 Jun '25

19 Jun '25

Dear all We are running Ceph Pacific (16.2.15) with RadosGW and have been getting "large omap objects" health warnings on the RadosGW index pool. Indeed we had one bucket in particular that was positively huge with 8'127'198 objects that had just a single shard. But we have been seeing the message on some other buckets, too. Eventually, we activated automatic resharding (rgw_dynamic_resharding = true) and indeed this bucket was resharded to now 167 shards. However, I am now getting even more large omap object warnings. On that same bucket, too. The other buckets have not been resharded at all. They are not in the queue, either: | radosgw-admin reshard list > [] | grep 'Large omap object found' /var/log/ceph/ceph* | grep 'PG: ' | cut -d: -f 10 | cut -d. -f 4-5 | sort | uniq -c 2 7936686773.215 12 7937604172.149 10 7955243979.1209 9 7955243979.2480 13 7955243979.2481 12 7968198782.110 13 7968913553.67 11 7968913553.68 10 7968913553.69 11 7981210604.1 74 7981624399.1 217 7988881492.1 | radosgw-admin metadata list --metadata-key bucket.instance | grep -i 7988881492 > "<bucket1_name>:<pool_name>.7988881492.1", | radosgw-admin bucket stats --bucket <bucket1_name> > { > "bucket": "<bucket1_name>", > "num_shards": 167, > [...] > "usage": { > "rgw.main": { > "size": 9669928611955, > "size_actual": 9692804734976, > "size_utilized": 9669928611955, > "size_kb": 9443289661, > "size_kb_actual": 9465629624, > "size_kb_utilized": 9443289661, > "num_objects": 8134437 > } > }, Let's check another one, too: | radosgw-admin metadata list --metadata-key bucket.instance | grep -i 7968198782.110 > "<bucket2_name>:<pool_name>.7968198782.110", | radosgw-admin bucket stats --bucket <bucket2_name> > [...] > "num_objects": 38690 > [...] According to the documentation in [0], buckets are resharded at a threshold of 100'000 objects per shard. For both of these, that applies nicely, so it makes sense that they are not getting resharded any further. But why then am I getting these warnings? Reading the documentation in [1], I can see that warnings are printed at 200'000 entries per omap object. Can I assume that one object in an RGW bucket means 1 entry in an omap object? Or is that a missconception? Now, here is my working theory. Please let me know if that has any merit or if I'm completely off: The affected buckets have versioning activated. Plus object locking too. They get used by a backup software (Kopia) that uses these features to provide ransomware protection. So my thinking is that maybe with versioning active, each object in a bucket could result in multiple omap entries, maybe one per version or something? If that is the case, then maybe I should reduce `rgw_max_objs_per_shard` from 100'000 to something like 10'000 to have the buckets resharded more aggressively? But then again, that assumes a lot. For example, that assumes that the num_objects counter in the bucket stats does not count up on versioned objects. So my assumption could be completely whack. What do you think? What can I do to get rid of the large omap objects? Is more resharding going to help? What else could I check? Sincerely Niklaus Hofer Links: [0] https://docs.ceph.com/en/latest/radosgw/dynamicresharding/#confval-rgw_max_… [1] https://docs.ceph.com/en/latest/rados/operations/health-checks/#large-omap-… -- stepping stone AG Wasserwerkgasse 7 CH-3011 Bern Telefon: +41 31 332 53 63 www.stepping-stone.ch niklaus.hofer(a)stepping-stone.ch

4 12

Autoscale warnings depite autoscaler being off
by Niklaus Hofer 19 Jun '25

19 Jun '25

Dear all After upgrading to Pacific, we are now getting health warnings from the auto scaler: > 10 pools have too few placement groups > 8 pools have too many placement groups The thing is, I do not particularly care about the recommendations from the auto scaler. So I checked and saw that it was set to warning on all pools: | ceph osd pool autoscale-status > POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO EFFECTIVE RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE BULK > openstack-glance 902.7G 3.0 618.2T 0.0043 1.0 128 warn False > openstack-cinder 99.48T 3.0 618.2T 0.4827 1.0 2048 warn False > openstack-nova 169.6G 3.0 618.2T 0.0008 1.0 128 warn False > [...] No worries, I just turn it off: | ceph osd pool set noautoscale Check it's off: | ceph osd pool autoscale-status > POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO EFFECTIVE RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE BULK > openstack-glance 902.7G 3.0 618.2T 0.0043 1.0 128 off False > openstack-cinder 99.48T 3.0 618.2T 0.4827 1.0 2048 off False > openstack-nova 169.6G 3.0 618.2T 0.0008 1.0 128 off False > [...] That looks as expected. But despite that, I'm still getting the health warnings: | ceph --status > [...] > 10 pools have too few placement groups > 8 pools have too many placement groups So how can I get rid of the warnings (without changing the number of PGs on my pools!)? Sincerely Niklaus Hofer -- stepping stone AG Wasserwerkgasse 7 CH-3011 Bern Telefon: +41 31 332 53 63 www.stepping-stone.ch niklaus.hofer(a)stepping-stone.ch

3 3

4k IOPS: miserable performance in All-SSD cluster
by Martin Gerhard Loschwitz 18 Jun '25

18 Jun '25

Folks, I am getting somewhat desperate debugging multiple setups here within the same environment. Three clusters, two SSD-only, one HDD-only, and what they all have in common is abysmal 4k IOPS performance when measuring with „rados bench“. Abysmal means: In an All-SSD cluster I will get roughly 400 IOPS over more than 250 devices. I’ve know SAS-SSDs are not ideal, but 250 looks a bit on the low side of things to me. In the second cluster, also All-SSD based, I get roughly 120 4k IOPS. And the HDD-only cluster delivers 60 4k IOPS. The latter both with substantially fewer devices, granted. But even with 20 HDDs, 68 4k IOPS seems like a very bad value to me. I’ve tried to rule out everything I know of: BIOS misconfigurations, HBA problems, networking trouble (I am seeing comparably bad values with a size=1 pool) and so further and so on. But to no avail. Has anybody dealt with something similar on Dell hardware or in general? What could cause such extremely bad benchmark results? I measure with rados bench and qd=1 at 4k block size. „ceph tell osd bench“ with 4k blocks yields 30k+ IOPS for every single device in the big cluster, and all that leads to is 400 IOPS in total when writing to it? Even with no replication in place? That looks a bit off, doesn't it? Any help will be greatly appreciated, thank you very much in advance. Even a pointer to the right direction would be held in high esteem right now. Thank you very much in advance! Best regards Martin

8 13

ceph rbd mirror snapshot-based does not work if mirror-mode is pool and not image
by Julien Laurenceau 18 Jun '25

18 Jun '25

Hi, I have 2 running ceph squid clusters (19.2.2). On each cluster there is an rbd pool named k8s-1 that I want to mirror using the snapshot based mode. if I configure both clusters in two way mirroring using the mode=image everything works fine. ``` rbd mirror pool enable ${mypool} image rbd mirror image enable ${mypool}/${myimage} snapshot ``` When I take a snapshot of myimage it gets mirrored. if I configure both clusters in two way mirroring using the mode=pool nothings happen. The snapshot are never mirrored in the second cluster. Reading the documentation https://docs.ceph.com/en/squid/rbd/rbd-mirroring/, I think that the problem is that the default mode for this command is "journal": rbd mirror image enable {pool-name}/{image-name} {mode} But, when enabling mirror on the pool there is no way to define mode=journal/snapshot: rbd mirror pool enable [--site-name {local-site-name}] {pool-name} {mode} with mode=pool (default) or image. Then I think when I do : rbd mirror pool enable ${mypool} image I think it implicitly do : rbd mirror pool enable ${mypool} image journal # THIS COMMAND DOES NOT WORK. But I would need to have: rbd mirror pool enable ${mypool} image snapshot # THIS COMMAND DOES NOT WORK. In addition, when I am configured on mirror mode = pool, if I try to disable mirroring on an image I get the following error : rbd mirror image disable ${mypool}/${myimage} 2025-06-18T15:37:42.177+0000 7f24ec53ce00 -1 librbd::api::Mirror: image_disable: cannot disable mirroring in the current pool mirroring mode Also error when I am configured on mirror mode = pool, if I try to enable snapshot mirroring on an image : rbd mirror image enable ${mypool}/${myimage} snapshot 2025-06-18T15:39:44.507+0000 7f533df1fe00 -1 librbd::api::Mirror: image_enable: cannot enable mirroring: pool is not in image mirror mode Could someone please confirm, that this is not supported ? What is the best workaround ? Cron cephadm shell script to enable mirror on all images ? PS : I do not want to use the journal based mode since the ceph-csi mounter nbd is not prod-ready.

1 0

How to clear the "slow operations" warning?
by Jan Kasprzak 18 Jun '25

18 Jun '25

Hello, Ceph users, TL;DR: how to clear/acknowledge the following warning from ceph -s? health: HEALTH_WARN 1 OSD(s) experiencing slow operations in BlueStore Details: This is caused by a bad sector on the physical HDD - the time of this warning is the same as the most recent error in the SMART log of that HDD. # ceph health detail HEALTH_WARN 1 OSD(s) experiencing slow operations in BlueStore [WRN] BLUESTORE_SLOW_OP_ALERT: 1 OSD(s) experiencing slow operations in BlueStore osd.33 observed slow operation indications in BlueStore When I restart osd.33 using "systemctl restart ceph-osd(a)33.service" on the OSD host, the warning disappears. But is there a "softer" way how to acknowledge this warning? Thanks! -Yenya -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | https://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall

11 15

CephFS scrub resulting in MDS_CACHE_OVERSIZED
by Kasper Rasmussen 18 Jun '25

18 Jun '25

After starting a recursive scrub on a cephfs with alot of files the MDS cache went oversized. Scrub command: ceph... scrub start / recursive,repair,force I kept an eye on the MDS memory usage - since I was warned that it might go crazy.. and after 2-3 hours, I started getting the warning [WRN] MDS_CACHE_OVERSIZED: 1 MDSs report oversized cache mds.generic-mds.<host>.asddje(mds.0): MDS cache is too large (63GB/36GB); 1250394 inodes in use by clients, 28888 stray files I then paused the scrub, resulting in scrub status { "status": "PAUSED (22837086 inodes in the stack)", "scrubs": { "27f0e32a-bc8c-443d-b1f0-534474798ddf": { "path": "/", "tag": "27f0e32a-bc8c-443d-b1f0-534474798ddf", "options": "recursive,repair,force" } } } and expected the cache size to go down again - but it didn't. After +12 hours with no change, I opted to abort the scrub - again expecting that the inodes in the stack would be offloaded from memory. The status after abort command: { "status": "PAUSED (0 inodes in the stack)", "scrubs": {} } But still no changes to the cache size. Since the status after the abort command had "PAUSED" in it, I resumed the scrub, resulting in status: { "status": "no active scrubs running", "scrubs": {} } Still no changes to the cache size. The log from the MDS in standard log level was: debug 2025-06-03T06:48:24.122+0000 7f319065d640 1 mds.generic-mds.<host>.asddje asok_command: scrub start {path=/,prefix=scrub start,scrubops=[recursive,repair,force]} (starting...) debug 2025-06-03T06:48:24.122+0000 7f318864d640 0 log_channel(cluster) log [INF] : scrub queued for path: / debug 2025-06-03T06:48:24.122+0000 7f318864d640 0 log_channel(cluster) log [INF] : scrub summary: idle+waiting paths [/] debug 2025-06-03T06:48:24.122+0000 7f318864d640 0 log_channel(cluster) log [INF] : scrub summary: active paths [/] debug 2025-06-03T06:48:24.126+0000 7f3189e50640 1 mds.0.cache.dir(0x10041e16a55) mismatch between head items and fnode.fragstat! printing dentries debug 2025-06-03T06:48:24.126+0000 7f3189e50640 1 mds.0.cache.dir(0x10041e16a55) get_num_head_items() = 38; fnode.fragstat.nfiles=28 fnode.fragstat.nsubdirs=11 debug 2025-06-03T06:48:24.126+0000 7f3189e50640 1 mds.0.cache.dir(0x10041e16a55) mismatch between child accounted_rstats and my rstats! debug 2025-06-03T06:48:24.126+0000 7f3189e50640 1 mds.0.cache.dir(0x10041e16a55) total of child dentries: n(v0 rc2025-06-03T06:48:11.042059+0000 b1661845634 127=95+32) debug 2025-06-03T06:48:24.126+0000 7f3189e50640 1 mds.0.cache.dir(0x10041e16a55) my rstats: n(v544237 rc2025-06-03T06:48:11.042059+0000 b1661845650 128=96+32) debug 2025-06-03T06:49:38.689+0000 7f319065d640 1 mds.generic-mds.<host>.asddje asok_command: scrub status {prefix=scrub status} (starting...) debug 2025-06-03T06:51:49.782+0000 7f319065d640 1 mds.generic-mds.<host>.asddje asok_command: scrub status {prefix=scrub status} (starting...) debug 2025-06-03T06:55:39.654+0000 7f319065d640 1 mds.generic-mds.<host>.asddje asok_command: scrub status {prefix=scrub status} (starting...) debug 2025-06-03T07:00:56.205+0000 7f319065d640 1 mds.generic-mds.<host>.asddje asok_command: scrub status .. .. From here it's either - asok_command: scrub status {prefix=scrub status} (starting...) - Updating MDS map to version xxxxxx from mon.3 Until I pause the scrub. Extracts from the perf dump from the MDS: "mds": { .. .. .. "inodes": 23121955, "inodes_top": 3684, "inodes_bottom": 1728, "inodes_pin_tail": 23116543, "inodes_pinned": 23116691, "inodes_expired": 39049803601, "inodes_with_caps": 84593, .. .. } .. .. "mds_mem": { "ino": 23114378, "ino+": 38966647328, "ino-": 38943532950, "dir": 513065, "dir+": 130921896, "dir-": 130408831, "dn": 23121954, "dn+": 39349549680, "dn-": 39326427726, "cap": 87280, "cap+": 6964477825, "cap-": 6964390545, "rss": 79730620, "heap": 223508 }, I've have been reluctant to just fail the MDS, to clear the memory, but when I finally came around to do so I got the error "Error EPERM: MDS has one of two health warnings which could extend recovery: MDS_TRIM or MDS_CACHE_OVERSIZED. MDS failover is not recommended since it might cause unexpected file system unavailability. If you wish to proceed, pass --yes-i-really-mean-it" At this moment the number strays reported in the MDS_CACHE_OVERSIZED warning, are now up with a factor 10 (approx. 280000) Which made me pause. This seems like a bug.. But To be honest I don't know quite what to expect, if I just execute with "--yes-i-really-mean-it".. Will the MDS eat huge amount of RAM during replay? (I've seen this before during failover - where MDS ate almost 200GB ram, even though the cache was not oversized.) Any advice on how to proceed? BR. Kasper

2 3

db_slots in service spec. is ignored
by Kasper Rasmussen 17 Jun '25

17 Jun '25

Hi I have a service spec for osds with the db_slots parameter set. However it's being ignored. After doing some code review, it seems to me that the reason is, that it's not implemented in translate.py - e.g. https://github.com/ceph/ceph/blob/v18.2.7/src/python-common/ceph/deployment… (or other versions) The translate.py simply does not have translation to "--block-db-slots" that is used by the ceph-volume command as specified here https://docs.ceph.com/en/reef/ceph-volume/lvm/batch/#implicit-sizing Something along the lines of: if db_slots: cmd += " --block-db-slots {}".format(self.spec.db_slots) Should probably be added after line 119 of v.18.2.7 at least - placement will defer depending on version - Any devs in here that can confirm this? Seems like a quick fix. * found this 2y old track for it as well - https://tracker.ceph.com/issues/57939

1 0