Hi, Well, at least the PGs are empty (0 objects according to ceph pg dump). Declaring them lost might be a thing. It looks like it might not be that urgent, so I check on the managers tomorrow morning. On 2026-07-29 20:48, Eugen Block via ceph-users wrote:
Oh that’s not good. Can you fail the mgr and then see if anything changes?
Zitat von Maximilian Hill <max@maxhill.eu>:
Hi,
There are plenty hosts for backfilling. The cluster consists of 13 hosts configured like a stretch cluster but without enabling stretch mode. (6 osd nodes per "room" + 1 mon in a third "room", pools have size 4 (2 each "room")).
Since I first wrote there has been a few changes: - 2 PGs show state "unknown" - output of ceph health detail: [WRN] PG_AVAILABILITY: Reduced data availability: 2 pgs inactive pg 20.25 is stuck inactive for 12h, current state unknown, last acting [] pg 28.ca is stuck inactive for 12h, current state unknown, last acting []
Maybe that's part of the problem. It doesn't make much sense to me, that it appeared a few hours ago but ceph health detail shows 12h.
Ceph pq 20.25 query, as well as 28.ca show both the following error:
# ceph pg 20.25 query Couldn't parse JSON : Expecting value: line 1 column 1 (char 0) Traceback (most recent call last): File "/usr/bin/ceph", line 1327, in <module> retval = main() File "/usr/bin/ceph", line 1247, in main sigdict = parse_json_funcsigs(outbuf.decode('utf-8'), 'cli') File "/usr/lib/python3.9/site-packages/ceph_argparse.py", line 1006, in parse_json_funcsigs raise e File "/usr/lib/python3.9/site-packages/ceph_argparse.py", line 1003, in parse_json_funcsigs overall = json.loads(s) File "/usr/lib64/python3.9/json/__init__.py", line 346, in loads return _default_decoder.decode(s) File "/usr/lib64/python3.9/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib64/python3.9/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
On 2026-07-29 19:00, Eugen Block via ceph-users wrote:
Hi,
without more information I assume the affected PGs are from pools that rely on the host you’re trying to drain. How many hosts do you have, what’s the failure domain of the crush rules in use? Can you share the crush rule dump and ‚ceph osd pool ls detail‘? Lets assume that you have 3 hosts with replica 3 with failure domain host. Draining a host doesn’t allow any backfill because there aren’t any hosts available to backfill to without breaking the crush rule. Another reason could be mclock, but it usually showed very little backfill IO and not none in previous reports. So I’m guessing it’s the crush rule(s) and the number of hosts not suitable for recovery/ backfill.
Zitat von Maximilian Hill via ceph-users <ceph-users@ceph.io>:
Hello,
I'm running a cluster on Ceph squid 19.2.3. Currently a drain (ceph orch host drain) was started for a node.
Now I have misplaced objects and 605 PGs stuck in active+clean+remapped but no backfilling is active.
Can someone give me some advice? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I wouldn’t trust the output unless you failed the mgr. Zitat von Maximilian Hill <max@maxhill.eu>:
Hi,
Well, at least the PGs are empty (0 objects according to ceph pg dump). Declaring them lost might be a thing. It looks like it might not be that urgent, so I check on the managers tomorrow morning.
On 2026-07-29 20:48, Eugen Block via ceph-users wrote:
Oh that’s not good. Can you fail the mgr and then see if anything changes?
Zitat von Maximilian Hill <max@maxhill.eu>:
Hi,
There are plenty hosts for backfilling. The cluster consists of 13 hosts configured like a stretch cluster but without enabling stretch mode. (6 osd nodes per "room" + 1 mon in a third "room", pools have size 4 (2 each "room")).
Since I first wrote there has been a few changes: - 2 PGs show state "unknown" - output of ceph health detail: [WRN] PG_AVAILABILITY: Reduced data availability: 2 pgs inactive pg 20.25 is stuck inactive for 12h, current state unknown, last acting [] pg 28.ca is stuck inactive for 12h, current state unknown, last acting []
Maybe that's part of the problem. It doesn't make much sense to me, that it appeared a few hours ago but ceph health detail shows 12h.
Ceph pq 20.25 query, as well as 28.ca show both the following error:
# ceph pg 20.25 query Couldn't parse JSON : Expecting value: line 1 column 1 (char 0) Traceback (most recent call last): File "/usr/bin/ceph", line 1327, in <module> retval = main() File "/usr/bin/ceph", line 1247, in main sigdict = parse_json_funcsigs(outbuf.decode('utf-8'), 'cli') File "/usr/lib/python3.9/site-packages/ceph_argparse.py", line 1006, in parse_json_funcsigs raise e File "/usr/lib/python3.9/site-packages/ceph_argparse.py", line 1003, in parse_json_funcsigs overall = json.loads(s) File "/usr/lib64/python3.9/json/__init__.py", line 346, in loads return _default_decoder.decode(s) File "/usr/lib64/python3.9/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib64/python3.9/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
On 2026-07-29 19:00, Eugen Block via ceph-users wrote:
Hi,
without more information I assume the affected PGs are from pools that rely on the host you’re trying to drain. How many hosts do you have, what’s the failure domain of the crush rules in use? Can you share the crush rule dump and ‚ceph osd pool ls detail‘? Lets assume that you have 3 hosts with replica 3 with failure domain host. Draining a host doesn’t allow any backfill because there aren’t any hosts available to backfill to without breaking the crush rule. Another reason could be mclock, but it usually showed very little backfill IO and not none in previous reports. So I’m guessing it’s the crush rule(s) and the number of hosts not suitable for recovery/ backfill.
Zitat von Maximilian Hill via ceph-users <ceph-users@ceph.io>:
Hello,
I'm running a cluster on Ceph squid 19.2.3. Currently a drain (ceph orch host drain) was started for a node.
Now I have misplaced objects and 605 PGs stuck in active+clean+remapped but no backfilling is active.
Can someone give me some advice? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I did fail the mgr in the end and the output stayed the same. Tried all three of them that we have in the cluster.
On 29. Jul 2026, at 23:00, Eugen Block <eblock@nde.ag> wrote:
I wouldn’t trust the output unless you failed the mgr.
Zitat von Maximilian Hill <max@maxhill.eu>:
Hi,
Well, at least the PGs are empty (0 objects according to ceph pg dump). Declaring them lost might be a thing. It looks like it might not be that urgent, so I check on the managers tomorrow morning.
On 2026-07-29 20:48, Eugen Block via ceph-users wrote: Oh that’s not good. Can you fail the mgr and then see if anything changes?
Zitat von Maximilian Hill <max@maxhill.eu>:
Hi,
There are plenty hosts for backfilling. The cluster consists of 13 hosts configured like a stretch cluster but without enabling stretch mode. (6 osd nodes per "room" + 1 mon in a third "room", pools have size 4 (2 each "room")).
Since I first wrote there has been a few changes: - 2 PGs show state "unknown" - output of ceph health detail: [WRN] PG_AVAILABILITY: Reduced data availability: 2 pgs inactive pg 20.25 is stuck inactive for 12h, current state unknown, last acting [] pg 28.ca is stuck inactive for 12h, current state unknown, last acting []
Maybe that's part of the problem. It doesn't make much sense to me, that it appeared a few hours ago but ceph health detail shows 12h.
Ceph pq 20.25 query, as well as 28.ca show both the following error:
# ceph pg 20.25 query Couldn't parse JSON : Expecting value: line 1 column 1 (char 0) Traceback (most recent call last): File "/usr/bin/ceph", line 1327, in <module> retval = main() File "/usr/bin/ceph", line 1247, in main sigdict = parse_json_funcsigs(outbuf.decode('utf-8'), 'cli') File "/usr/lib/python3.9/site-packages/ceph_argparse.py", line 1006, in parse_json_funcsigs raise e File "/usr/lib/python3.9/site-packages/ceph_argparse.py", line 1003, in parse_json_funcsigs overall = json.loads(s) File "/usr/lib64/python3.9/json/__init__.py", line 346, in loads return _default_decoder.decode(s) File "/usr/lib64/python3.9/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib64/python3.9/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
On 2026-07-29 19:00, Eugen Block via ceph-users wrote:
Hi,
without more information I assume the affected PGs are from pools that rely on the host you’re trying to drain. How many hosts do you have, what’s the failure domain of the crush rules in use? Can you share the crush rule dump and ‚ceph osd pool ls detail‘? Lets assume that you have 3 hosts with replica 3 with failure domain host. Draining a host doesn’t allow any backfill because there aren’t any hosts available to backfill to without breaking the crush rule. Another reason could be mclock, but it usually showed very little backfill IO and not none in previous reports. So I’m guessing it’s the crush rule(s) and the number of hosts not suitable for recovery/ backfill.
Zitat von Maximilian Hill via ceph-users <ceph-users@ceph.io>:
Hello,
I'm running a cluster on Ceph squid 19.2.3. Currently a drain (ceph orch host drain) was started for a node.
Now I have misplaced objects and 605 PGs stuck in active+clean+remapped but no backfilling is active.
Can someone give me some advice? _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
That's not a good sign. Something else must have happenend after/while draining the node. Did other OSDs or hosts fail in the meantime so that there are not enough replicas to recover? Are all pools using the same crush rule? How exactly did you drain the node? Some more context and history are required to be able to find out what's going on. Zitat von Maximilian Hill <max@maxhill.eu>:
I did fail the mgr in the end and the output stayed the same. Tried all three of them that we have in the cluster.
On 29. Jul 2026, at 23:00, Eugen Block <eblock@nde.ag> wrote:
I wouldn’t trust the output unless you failed the mgr.
Zitat von Maximilian Hill <max@maxhill.eu>:
Hi,
Well, at least the PGs are empty (0 objects according to ceph pg dump). Declaring them lost might be a thing. It looks like it might not be that urgent, so I check on the managers tomorrow morning.
On 2026-07-29 20:48, Eugen Block via ceph-users wrote: Oh that’s not good. Can you fail the mgr and then see if anything changes?
Zitat von Maximilian Hill <max@maxhill.eu>:
Hi,
There are plenty hosts for backfilling. The cluster consists of 13 hosts configured like a stretch cluster but without enabling stretch mode. (6 osd nodes per "room" + 1 mon in a third "room", pools have size 4 (2 each "room")).
Since I first wrote there has been a few changes: - 2 PGs show state "unknown" - output of ceph health detail: [WRN] PG_AVAILABILITY: Reduced data availability: 2 pgs inactive pg 20.25 is stuck inactive for 12h, current state unknown, last acting [] pg 28.ca is stuck inactive for 12h, current state unknown, last acting []
Maybe that's part of the problem. It doesn't make much sense to me, that it appeared a few hours ago but ceph health detail shows 12h.
Ceph pq 20.25 query, as well as 28.ca show both the following error:
# ceph pg 20.25 query Couldn't parse JSON : Expecting value: line 1 column 1 (char 0) Traceback (most recent call last): File "/usr/bin/ceph", line 1327, in <module> retval = main() File "/usr/bin/ceph", line 1247, in main sigdict = parse_json_funcsigs(outbuf.decode('utf-8'), 'cli') File "/usr/lib/python3.9/site-packages/ceph_argparse.py", line 1006, in parse_json_funcsigs raise e File "/usr/lib/python3.9/site-packages/ceph_argparse.py", line 1003, in parse_json_funcsigs overall = json.loads(s) File "/usr/lib64/python3.9/json/__init__.py", line 346, in loads return _default_decoder.decode(s) File "/usr/lib64/python3.9/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib64/python3.9/json/decoder.py", line 355, in raw_decode raise JSONDecodeError("Expecting value", s, err.value) from None json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
On 2026-07-29 19:00, Eugen Block via ceph-users wrote:
Hi,
without more information I assume the affected PGs are from pools that rely on the host you’re trying to drain. How many hosts do you have, what’s the failure domain of the crush rules in use? Can you share the crush rule dump and ‚ceph osd pool ls detail‘? Lets assume that you have 3 hosts with replica 3 with failure domain host. Draining a host doesn’t allow any backfill because there aren’t any hosts available to backfill to without breaking the crush rule. Another reason could be mclock, but it usually showed very little backfill IO and not none in previous reports. So I’m guessing it’s the crush rule(s) and the number of hosts not suitable for recovery/ backfill.
Zitat von Maximilian Hill via ceph-users <ceph-users@ceph.io>:
> Hello, > > I'm running a cluster on Ceph squid 19.2.3. > Currently a drain (ceph orch host drain) was started for a node. > > Now I have misplaced objects and 605 PGs stuck in active+clean+remapped > but no backfilling is active. > > Can someone give me some advice? > _______________________________________________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I just solved the problem. One of the hosts bucket was incorrectly assigned to "root" directly instead of the correct "rack". After moving the "host" bucket it started working again. On 2026-07-29 19:00, Eugen Block via ceph-users wrote:
Hi,
Draining a host doesn’t allow any backfill because there aren’t any hosts available to backfill to without breaking the crush rule.
I guess you were right about this. Thanks for your help On 2026-07-30 09:43, Eugen Block wrote:
That's not a good sign. Something else must have happenend after/while draining the node. Did other OSDs or hosts fail in the meantime so that there are not enough replicas to recover? Are all pools using the same crush rule? How exactly did you drain the node? Some more context and history are required to be able to find out what's going on.
Ah, that was an easy fix. Great that you figured it out. Zitat von Maximilian Hill <max@maxhill.eu>:
I just solved the problem. One of the hosts bucket was incorrectly assigned to "root" directly instead of the correct "rack". After moving the "host" bucket it started working again.
On 2026-07-29 19:00, Eugen Block via ceph-users wrote:
Hi,
Draining a host doesn’t allow any backfill because there aren’t any hosts available to backfill to without breaking the crush rule.
I guess you were right about this.
Thanks for your help
On 2026-07-30 09:43, Eugen Block wrote:
That's not a good sign. Something else must have happenend after/while draining the node. Did other OSDs or hosts fail in the meantime so that there are not enough replicas to recover? Are all pools using the same crush rule? How exactly did you drain the node? Some more context and history are required to be able to find out what's going on.
participants (2)
-
Eugen Block
-
Maximilian Hill