- ceph-users - lists.ceph.io

Re: Many empty objects in CephFS data pool
by Loïc Tortay 30 Jul '26

30 Jul '26

On 30/07/2026 11:42, Robert Sander via ceph-users wrote: > Hi, > > we have a CephFS with a replicated data pool and an additional EC data > pool. All files store their data in the EC pool. > > But "ceph df" shows that the replicated data pool contains over 200 > million objects with a total size of 700 bytes. So the vast majority are > empty objects. > > How can this be? > Hello, The replicated pool is the "default pool" and contains files metadata ? (unlike the "metadata" pool) Like in this filesystem with ~1.4 billion files and 85 million directories: POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL cephfs_EC_data 3 8192 6.6 PiB 3.00G 8.3 PiB 79.90 1.7 PiB cephfs_metadata 4 256 778 GiB 84.70M 2.3 TiB 3.38 22 TiB cephfs_default 5 128 0 B 1.37G 0 B 0 22 TiB Loïc. -- | Loic Tortay <tortay(a)cc.in2p3.fr> - IN2P3 Computing Centre |

1 0

Re: Many empty objects in CephFS data pool
by Janne Johansson 30 Jul '26

30 Jul '26

> > > we have a CephFS with a replicated data pool and an additional EC data > pool. All files store their data in the EC pool. > > But "ceph df" shows that the replicated data pool contains over 200 > million objects with a total size of 700 bytes. So the vast majority are > empty objects. > > How can this be? > I don't do cephfs, but at least radosgw likes to stash information into zero-sized objects, so the sum of the sizes of those objects was small, but the data usage would of course be larger than zero on those rgw pools. Perhaps cephfs does the same? "ceph df detail" on one of our rgw clusters show this: POOL ID PGS STORED (DATA) (OMAP) OBJECTS USED (DATA) (OMAP) %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR default.rgw.buckets.index 7 512 51 GiB 0 B 51 GiB 9.57k 154 GiB 0 B 154 GiB 0.02 225 TiB N/A N/A N/A 0 B 0 B the "DATA" column is zero, even though there are 9.57k objects in there, but OMAP is lots more, probably meaning it stashed information into the metadata of said zero-sized objs. If we stat one of the .dir.xxxx objects, it says: rados stat -p default.rgw.buckets.index ".dir.b9865dc1-e648-4479-abcd-ed812345678.144664284.33737.1.20" default.rgw.buckets.index/.dir.b9865dc1-e648-4479-abcd-ed87654321.144664284.33737.1.20 mtime 2026-07-30T09:48:05.000000+0000, size 0 -- May the most significant bit of your life be positive.

1 0

some details on "split brain" and 'min_size' vs. 'size' and 'k,m'
by pg＠ceph.list.sabi.co.UK 30 Jul '26

30 Jul '26

I have been slightly perplexed about some discussions around 'min_size' and had about that a discussion with a gooda LLM about some related technical details: https://chat.deepseek.com/share/u1jxte2mbbeetwr0sl tl;dr: why worry about details, be happy! :-)

1 0

Many empty objects in CephFS data pool
by Robert Sander 30 Jul '26

30 Jul '26

Hi, we have a CephFS with a replicated data pool and an additional EC data pool. All files store their data in the EC pool. But "ceph df" shows that the replicated data pool contains over 200 million objects with a total size of 700 bytes. So the vast majority are empty objects. How can this be? Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 220009 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

1 0

Re: PGs stuck remapped
by Maximilian Hill 30 Jul '26

30 Jul '26

Hi, Well, at least the PGs are empty (0 objects according to ceph pg dump). Declaring them lost might be a thing. It looks like it might not be that urgent, so I check on the managers tomorrow morning. On 2026-07-29 20:48, Eugen Block via ceph-users wrote: > Oh that’s not good. Can you fail the mgr and then see if anything changes? > > Zitat von Maximilian Hill <max(a)maxhill.eu>: > >> Hi, >> >> There are plenty hosts for backfilling. >> The cluster consists of 13 hosts configured like a stretch cluster but >> without enabling stretch mode. (6 osd nodes per "room" + 1 mon in a >> third "room", pools have size 4 (2 each "room")). >> >> Since I first wrote there has been a few changes: >> - 2 PGs show state "unknown" >> - output of ceph health detail: >> [WRN] PG_AVAILABILITY: Reduced data availability: 2 pgs inactive >> pg 20.25 is stuck inactive for 12h, current state unknown, last >> acting [] >> pg 28.ca is stuck inactive for 12h, current state unknown, last >> acting [] >> >> Maybe that's part of the problem. It doesn't make much sense to me, >> that it appeared a few hours ago but ceph health detail shows 12h. >> >> Ceph pq 20.25 query, as well as 28.ca show both the following error: >> >> # ceph pg 20.25 query >> Couldn't parse JSON : Expecting value: line 1 column 1 (char 0) >> Traceback (most recent call last): >> File "/usr/bin/ceph", line 1327, in <module> >> retval = main() >> File "/usr/bin/ceph", line 1247, in main >> sigdict = parse_json_funcsigs(outbuf.decode('utf-8'), 'cli') >> File "/usr/lib/python3.9/site-packages/ceph_argparse.py", line 1006, >> in parse_json_funcsigs >> raise e >> File "/usr/lib/python3.9/site-packages/ceph_argparse.py", line 1003, >> in parse_json_funcsigs >> overall = json.loads(s) >> File "/usr/lib64/python3.9/json/__init__.py", line 346, in loads >> return _default_decoder.decode(s) >> File "/usr/lib64/python3.9/json/decoder.py", line 337, in decode >> obj, end = self.raw_decode(s, idx=_w(s, 0).end()) >> File "/usr/lib64/python3.9/json/decoder.py", line 355, in raw_decode >> raise JSONDecodeError("Expecting value", s, err.value) from None >> json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) >> >> >> >> On 2026-07-29 19:00, Eugen Block via ceph-users wrote: >>> Hi, >>> >>> without more information I assume the affected PGs are from pools >>> that rely on the host you’re trying to drain. >>> How many hosts do you have, what’s the failure domain of the crush >>> rules in use? Can you share the crush rule dump and ‚ceph osd pool ls >>> detail‘? >>> Lets assume that you have 3 hosts with replica 3 with failure domain >>> host. Draining a host doesn’t allow any backfill because there aren’t >>> any hosts available to backfill to without breaking the crush rule. >>> Another reason could be mclock, but it usually showed very little >>> backfill IO and not none in previous reports. So I’m guessing it’s >>> the crush rule(s) and the number of hosts not suitable for recovery/ >>> backfill. >>> >>> Zitat von Maximilian Hill via ceph-users <ceph-users(a)ceph.io>: >>> >>>> Hello, >>>> >>>> I'm running a cluster on Ceph squid 19.2.3. >>>> Currently a drain (ceph orch host drain) was started for a node. >>>> >>>> Now I have misplaced objects and 605 PGs stuck in active+clean+remapped >>>> but no backfilling is active. >>>> >>>> Can someone give me some advice? >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> >>> >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 5

Re: PGs stuck remapped
by Maximilian Hill 29 Jul '26

29 Jul '26

Hi, There are plenty hosts for backfilling. The cluster consists of 13 hosts configured like a stretch cluster but without enabling stretch mode. (6 osd nodes per "room" + 1 mon in a third "room", pools have size 4 (2 each "room")). Since I first wrote there has been a few changes: - 2 PGs show state "unknown" - output of ceph health detail: [WRN] PG_AVAILABILITY: Reduced data availability: 2 pgs inactive pg 20.25 is stuck inactive for 12h, current state unknown, last acting [] pg 28.ca is stuck inactive for 12h, current state unknown, last acting [] Maybe that's part of the problem. It doesn't make much sense to me, that it appeared a few hours ago but ceph health detail shows 12h. Ceph pq 20.25 query, as well as 28.ca show both the following error: # ceph pg 20.25 query Couldn't parse JSON : Expecting value: line 1 column 1 (char 0) Traceback (most recent call last): File "/usr/bin/ceph", line 1327, in <module> retval = main() File "/usr/bin/ceph", line 1247, in main sigdict = parse_json_funcsigs(outbuf.decode('utf-8'), 'cli') File "/usr/lib/python3.9/site-packages/ceph_argparse.py", line 1006, in parse_json_funcsigs raise e File "/usr/lib/python3.9/site-packages/ceph_argparse.py", line 1003, in parse_json_funcsigs overall = json.loads(s) File "/usr/lib64/python3.9/json/__init__.py", line 346, in loads return _default_decoder.decode(s) File "/usr/lib64/python3.9/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib64/python3.9/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) On 2026-07-29 19:00, Eugen Block via ceph-users wrote: > Hi, > > without more information I assume the affected PGs are from pools that > rely on the host you’re trying to drain. > How many hosts do you have, what’s the failure domain of the crush rules > in use? Can you share the crush rule dump and ‚ceph osd pool ls detail‘? > Lets assume that you have 3 hosts with replica 3 with failure domain > host. Draining a host doesn’t allow any backfill because there aren’t > any hosts available to backfill to without breaking the crush rule. > Another reason could be mclock, but it usually showed very little > backfill IO and not none in previous reports. So I’m guessing it’s the > crush rule(s) and the number of hosts not suitable for recovery/backfill. > > Zitat von Maximilian Hill via ceph-users <ceph-users(a)ceph.io>: > >> Hello, >> >> I'm running a cluster on Ceph squid 19.2.3. >> Currently a drain (ceph orch host drain) was started for a node. >> >> Now I have misplaced objects and 605 PGs stuck in active+clean+remapped >> but no backfilling is active. >> >> Can someone give me some advice? >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Re: PGs stuck remapped
by Eugen Block 29 Jul '26

29 Jul '26

Hi, without more information I assume the affected PGs are from pools that rely on the host you’re trying to drain. How many hosts do you have, what’s the failure domain of the crush rules in use? Can you share the crush rule dump and ‚ceph osd pool ls detail‘? Lets assume that you have 3 hosts with replica 3 with failure domain host. Draining a host doesn’t allow any backfill because there aren’t any hosts available to backfill to without breaking the crush rule. Another reason could be mclock, but it usually showed very little backfill IO and not none in previous reports. So I’m guessing it’s the crush rule(s) and the number of hosts not suitable for recovery/backfill. Zitat von Maximilian Hill via ceph-users <ceph-users(a)ceph.io>: > Hello, > > I'm running a cluster on Ceph squid 19.2.3. > Currently a drain (ceph orch host drain) was started for a node. > > Now I have misplaced objects and 605 PGs stuck in active+clean+remapped > but no backfilling is active. > > Can someone give me some advice? > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Mounting CephFS on Ceph nodes themselves
by Anthony D'Atri 29 Jul '26

29 Jul '26

Might bumping vm.min_free_kbytes mitigate this dynamic? The default is kind of parsimonious; I routinely set to 4GB. > On Jul 29, 2026, at 3:14 AM, Dan van der Ster via ceph-users <ceph-users(a)ceph.io> wrote: > > On Wed, Jul 29, 2026 at 4:01 AM Gregory Farnum via ceph-users > <ceph-users(a)ceph.io> wrote: >>> On Tue, Jul 28, 2026 at 3:05 AM Janne Johansson via ceph-users >>> <ceph-users(a)ceph.io> wrote: >>> I think the idea was that if you mount cephfs on a box running OSDs or a >>> mon daemon, and it gets into some kind of trouble like running out of >>> memory or so, then both the client side and the "server" side might be >>> competing for the same resources that aren't there, making it worse. >> This is a classic problem in network filesystems: >> 1) The kernel runs out of memory and needs to flush out buffered >> writes to free up space >> 2) The client sends out a network message, which comes back in to the server >> 3) The server needs to allocate memory to handle the incoming network message >> 4) ...stuck >> RBD and CephFS are susceptible to this, if you mount the kernels on >> the same hosts as the OSDs. But the odds of hitting it are a lot lower >> on modern systems with their RAM capacities than in the past. > > It's been a couple of years, but I was still hitting this deadlock on > el8 kernels with CephFS / ceph-osd co-located alongside some HPC-type > MPI jobs. > It's like Greg wrote -- a large client read/write that comes back to > an OSD on the same host deadlocks the box. > > Cheers, Dan > > > >> -Greg >>> If you have separate OSD hosts and they are having issues, then cephfs >>> clients will be slow, but not much more than that, and if clients are >>> having resource dips, a separate cluster doesn't mind at all. I think that >>> is where the non-recommendation comes from, in case you do combine them, it >>> will be hard to get out of this situation, any cpu/ram given to the client >>> side would cause more IO to be requested, from the server side that doesn't >>> have it. In any case it might make it lots harder to diagnose which end >>> actually caused the issue if both suffer at the same time. >>> Den tis 28 juli 2026 kl 14:45 skrev Marek Szuba via ceph-users < >>> ceph-users(a)ceph.io>: >>>> Dear fellow Cephies, >>>> A few weeks ago I came across in Tentacle documentation on docs.ceph.org >>>> (and OF COURSE now that I'm trying to quote it, I cannot find it any >>>> more) a mention that CephFS should not be mounted on systems serving >>>> Ceph because of there being a risk of harmful interference in the >>>> kernel. The context suggested this pertained to one-node Ceph clusters, >>>> that said I do not believe that was explicitly stated. Therefore, I very >>>> much hope someone will be able to enlighten me whether: >>>> * this is indeed still the case with Tentacle, >>>> * it applies to one-node clusters only or all Ceph clusters, >>>> * it only applies to CephFS mounts using the kernel driver, or ceph-fuse >>>> as well. >>>> Thank you in advance! >>>> -- >>>> MS >>>> _______________________________________________ >>>> ceph-users mailing list -- ceph-users(a)ceph.io >>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> -- >>> May the most significant bit of your life be positive. >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

PGs stuck remapped
by Maximilian Hill 29 Jul '26

29 Jul '26

Hello, I'm running a cluster on Ceph squid 19.2.3. Currently a drain (ceph orch host drain) was started for a node. Now I have misplaced objects and 605 PGs stuck in active+clean+remapped but no backfilling is active. Can someone give me some advice?

1 0

Re: problem uploading dashboard certificates
by Simon Oosthoek 29 Jul '26

29 Jul '26

Hi Eugen, that worked! ceph config-key get mgr/dashboard/cephmon1/crt ceph config-key get mgr/dashboard/cephmon1/key ceph config-key rm mgr/dashboard/crt ceph config-key rm mgr/dashboard/key ceph dashboard set-ssl-certificate -i cephmon.science.ru.nl.pem >failed! ceph mgr module disable dashboard ceph dashboard set-ssl-certificate -i cephmon.science.ru.nl.pem >failed! ceph mgr module enable dashboard ceph dashboard set-ssl-certificate -i cephmon.science.ru.nl.pem ceph dashboard set-ssl-certificate-key -i cephmon.science.ru.nl.key Then it worked again :-) Cheers, /Simon On Mon, 27 Jul 2026 at 13:11, Eugen Block via ceph-users <ceph-users(a)ceph.io> wrote: > Hi, > > the dashboard cert and key files are stored in: > > ceph config-key get mgr/dashboard/crt > ceph config-key get mgr/dashboard/key > > Or something similar, maybe per host entries like (the "random" string > is cephadm specific and won't apply in your case, of course): > > ceph config-key get mgr/dashboard/host1.random/crt > ceph config-key get mgr/dashboard/host1.random/key > > You should be able to remove those faulty entries with > > ceph config-key rm mgr/dashboard/host1.random/crt > ceph config-key rm mgr/dashboard/host1.random/key > > Then fail the mgr and inject your correct cert/key files again. I > think that should work, if not, let us know. > > Regards, > Eugen > > Zitat von Simon Oosthoek via ceph-users <ceph-users(a)ceph.io>: > > > Sorry, forgot to add info about the installation; this is ceph 17.2.x on > > ubuntu 22.04.x, no cephadm and the cluster has about 360 OSDs of 12TB > HDD. > > > > On Mon, 27 Jul 2026 at 11:50, Simon Oosthoek via ceph-users < > > ceph-users(a)ceph.io> wrote: > > > >> Hi all, > >> we've had some issues when Harica revoked a bunch of our certificates, I > >> then accidentally uploaded mismatching key/pem files. When I wanted to > fix > >> that with the correct ones I got > >> > >> # ceph dashboard set-ssl-certificate -i cephmon.science.ru.nl.pem > >> Error EIO: Module 'dashboard' has experienced an error and cannot handle > >> commands: [('x509 certificate routines', '', 'key values mismatch')] > >> > >> Is there another way to fix this? > >> > >> Cheers, > >> > >> /Simon > >> -- > >> I'm using my gmail.com address, because the gmail.com dmarc policy is > >> "none", some mail servers will reject this (microsoft?) others will > instead > >> allow this when I send mail to a mailling list which has not yet been > >> configured to send mail "on behalf of" the sender, but rather do a kind > of > >> "forward". The latter situation causes dkim/dmarc failures and the dmarc > >> policy will be applied. see https://wiki.list.org/DEV/DMARC for more > >> details > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users(a)ceph.io > >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >> > > > > > > -- > > I'm using my gmail.com address, because the gmail.com dmarc policy is > > "none", some mail servers will reject this (microsoft?) others will > instead > > allow this when I send mail to a mailling list which has not yet been > > configured to send mail "on behalf of" the sender, but rather do a kind > of > > "forward". The latter situation causes dkim/dmarc failures and the dmarc > > policy will be applied. see https://wiki.list.org/DEV/DMARC for more > details > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- I'm using my gmail.com address, because the gmail.com dmarc policy is "none", some mail servers will reject this (microsoft?) others will instead allow this when I send mail to a mailling list which has not yet been configured to send mail "on behalf of" the sender, but rather do a kind of "forward". The latter situation causes dkim/dmarc failures and the dmarc policy will be applied. see https://wiki.list.org/DEV/DMARC for more details

2 1