Thanks, Konstantin. It's been a while since I was last bitten by the choose_tries being too low... Unfortunately, I won't be able to verify that... But I'll definitely keep that in mind, or least I'll try to. :-D Thanks! Zitat von Konstantin Shalygin <k0ste@k0ste.ru>:

Hi Eugen

...

On 21 May 2024, at 15:26, Eugen Block <eblock@nde.ag> wrote:

step set_choose_tries 100

I think you should try to increase set_choose_tries to 200 Last year we had an Pacific EC 8+2 deployment of 10 racks. And even with 50 hosts, the value of 100 not worked for us

k

Hi Eugen, I'm afraid the description of your observation breaks a bit with causality and this might be the reason for the few replies. To produce a bit more structure for when exactly what happened, let's look at what I did and didn't get: Before adding the hosts you have situation 1) default DCA host A1 ... AN DCB host B1 ... BM Now you add K+L hosts, they go into the default root and we have situation 2) default host C1 ... CK, D1 ... DL DCA host A1 ... AN DCB host B1 ... BM As a last step, you move the hosts to their final locations and we arrive at situation 3) default DCA host A1 ... AN, C1 ... CK DCB host B1 ... BM, D1 ... DL Please correct if this is wrong. Assuming its correct, I conclude the following. Now, from your description it is not clear to me on which of the transitions 1->2 or 2->3 you observe - peering and/or - unknown PGs. We use a somewhat similar procedure except that we have a second root (separate disjoint tree) for new hosts/OSDs. However, in terms of peering it is the same and if everything is configured correctly I would expect this to happen (this is what happens when we add OSDs/hosts): transition 1->2: hosts get added: no peering, no remapped objects, nothing, just new OSDs doing nothing transition 2->3: hosts get moved: peering starts and remapped objects appear, all PGs active+clean Unknown PGs should not occur (maybe only temporarily when the primary changes or the PG is slow to respond/report status??). The crush bug with too few set_choose_tries is observed if one has *just enough hosts* for the EC profile and should not be observed if all PGs are active+clean and one *adds hosts*. Persistent unknown PGs can (to my understanding, does unknown mean "has no primary"?) only occur if the number of PGs changes (autoscaler messing around??) because all PGs were active+clean before. The crush bug leads to incomplete PGs, so PGs can go incomplete but they should always have an acting primary. This is assuming no OSDs went down/out during the process. Can you please check if my interpretation is correct and describe at which step exactly things start diverging from my expectations. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Eugen Block <eblock@nde.ag> Sent: Thursday, May 23, 2024 12:05 PM To: ceph-users@ceph.io Subject: [ceph-users] Re: unknown PGs after adding hosts in different subtree Hi again, I'm still wondering if I misunderstand some of the ceph concepts. Let's assume the choose_tries value is too low and ceph can't find enough OSDs for the remapping. I would expect that there are some PG chunks in remapping state or unknown or whatever, but why would it affect the otherwise healthy cluster in such a way? Even if ceph doesn't know where to put some of the chunks, I wouldn't expect inactive PGs and have a service interruption. What am I missing here? Thanks, Eugen Zitat von Eugen Block <eblock@nde.ag>:

Thanks, Konstantin. It's been a while since I was last bitten by the choose_tries being too low... Unfortunately, I won't be able to verify that... But I'll definitely keep that in mind, or least I'll try to. :-D

Thanks!

Zitat von Konstantin Shalygin <k0ste@k0ste.ru>:

...

Hi Eugen

...

On 21 May 2024, at 15:26, Eugen Block <eblock@nde.ag> wrote:

step set_choose_tries 100

I think you should try to increase set_choose_tries to 200 Last year we had an Pacific EC 8+2 deployment of 10 racks. And even with 50 hosts, the value of 100 not worked for us

k

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Hi Eugen, thanks for this clarification. Yes, with the observations you describe for transition 1->2, something is very wrong. Nothing should happen. Unfortunately, I'm going to be on holidays and, generally, don't have too much time. If they can afford to share the osdmap (ceph osd getmap -o file), I could also take a look at some point. I don't think it has to do with set_choose_tries, there is likely something else screwed up badly. There should simply not be any remapping going on at this stage. Just for fun, you should be able to produce a clean crushmap from scratch with a similar or the same tree and check if you see the same problems. Using the full osdmap with osdmaptool allows to reproduce the exact mappings as used in the cluster and it encodes other important information as well. That's why I'm asking for this instead of just the crush map. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Eugen Block <eblock@nde.ag> Sent: Thursday, May 23, 2024 1:26 PM To: Frank Schilder Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Re: unknown PGs after adding hosts in different subtree Hi Frank, thanks for chiming in here.

Please correct if this is wrong. Assuming its correct, I conclude the following.

You assume correctly.

Now, from your description it is not clear to me on which of the transitions 1->2 or 2->3 you observe - peering and/or - unknown PGs.

The unknown PGs were observed during/after 1 -> 2. All or almost all PGs were reported as "remapped", I don't remember the exact number, but it was more than 4k, and the largest pool has 4096 PGs. We didn't see down OSDs at all. Only after moving the hosts into their designated location (the DCs) the unknown PGs cleared and the application resumed its operation. I don't want to overload this thread but I asked for a copy of their crushmap to play around a bit. I moved the new hosts out of the DCs into the default root via 'crushtool --move ...', then running the crushtool --test command # crushtool -i crushmap --test --rule 1 --num-rep 18 --show-choose-tries [--show-bad-mappings] --show-utilization results in a couple of issues: - there are lots of bad mappings no matter how high the number for set_choose_tries is set - the show-utilization output shows 240 OSDs in usage (there were 240 OSDs before the expansion), but plenty of them have only 9 chunks assigned: rule 1 (rule-ec-k7m11), x = 0..1023, numrep = 18..18 rule 1 (rule-ec-k7m11) num_rep 18 result size == 0: 55/1024 rule 1 (rule-ec-k7m11) num_rep 18 result size == 9: 488/1024 rule 1 (rule-ec-k7m11) num_rep 18 result size == 18: 481/1024 And this reminds me of the inactive PGs we saw before I failed the mgr, those inactive PGs showed only 9 chunks in the acting set. With k=7 (and min_size=8) that should still be enough, we have successfully tested disaster recovery with one entire DC down multiple times. - with --show-mappings some lines contain an empty set like this: CRUSH rule 1 x 22 [] And one more observation: with the currently active crushmap there are no bad mappings at all when the hosts are in their designated location. So there's definitely something wrong here, I just can't tell what it is yet. I'll play a bit more with that crushmap... Thanks! Eugen Zitat von Frank Schilder <frans@dtu.dk>:

Hi Eugen,

I'm afraid the description of your observation breaks a bit with causality and this might be the reason for the few replies. To produce a bit more structure for when exactly what happened, let's look at what I did and didn't get:

Before adding the hosts you have situation

1) default DCA host A1 ... AN DCB host B1 ... BM

Now you add K+L hosts, they go into the default root and we have situation

2) default host C1 ... CK, D1 ... DL DCA host A1 ... AN DCB host B1 ... BM

As a last step, you move the hosts to their final locations and we arrive at situation

3) default DCA host A1 ... AN, C1 ... CK DCB host B1 ... BM, D1 ... DL

Please correct if this is wrong. Assuming its correct, I conclude the following.

Now, from your description it is not clear to me on which of the transitions 1->2 or 2->3 you observe - peering and/or - unknown PGs.

We use a somewhat similar procedure except that we have a second root (separate disjoint tree) for new hosts/OSDs. However, in terms of peering it is the same and if everything is configured correctly I would expect this to happen (this is what happens when we add OSDs/hosts):

transition 1->2: hosts get added: no peering, no remapped objects, nothing, just new OSDs doing nothing transition 2->3: hosts get moved: peering starts and remapped objects appear, all PGs active+clean

Unknown PGs should not occur (maybe only temporarily when the primary changes or the PG is slow to respond/report status??). The crush bug with too few set_choose_tries is observed if one has *just enough hosts* for the EC profile and should not be observed if all PGs are active+clean and one *adds hosts*. Persistent unknown PGs can (to my understanding, does unknown mean "has no primary"?) only occur if the number of PGs changes (autoscaler messing around??) because all PGs were active+clean before. The crush bug leads to incomplete PGs, so PGs can go incomplete but they should always have an acting primary.

This is assuming no OSDs went down/out during the process.

Can you please check if my interpretation is correct and describe at which step exactly things start diverging from my expectations.

Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

________________________________________ From: Eugen Block <eblock@nde.ag> Sent: Thursday, May 23, 2024 12:05 PM To: ceph-users@ceph.io Subject: [ceph-users] Re: unknown PGs after adding hosts in different subtree

Hi again,

I'm still wondering if I misunderstand some of the ceph concepts. Let's assume the choose_tries value is too low and ceph can't find enough OSDs for the remapping. I would expect that there are some PG chunks in remapping state or unknown or whatever, but why would it affect the otherwise healthy cluster in such a way? Even if ceph doesn't know where to put some of the chunks, I wouldn't expect inactive PGs and have a service interruption. What am I missing here?

Thanks, Eugen

Zitat von Eugen Block <eblock@nde.ag>:

...

Thanks, Konstantin. It's been a while since I was last bitten by the choose_tries being too low... Unfortunately, I won't be able to verify that... But I'll definitely keep that in mind, or least I'll try to. :-D

Thanks!

Zitat von Konstantin Shalygin <k0ste@k0ste.ru>:

...

Hi Eugen

...

On 21 May 2024, at 15:26, Eugen Block <eblock@nde.ag> wrote:

step set_choose_tries 100

I think you should try to increase set_choose_tries to 200 Last year we had an Pacific EC 8+2 deployment of 10 racks. And even with 50 hosts, the value of 100 not worked for us

k

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

In my small lab cluster I can at least reproduce that a bunch of PGs are remapped after adding hosts to the default root, but they are not in their designated location yet. I have 3 „rooms“ underneath the default root. Although I can’t reproduce the unknown PGs, maybe this is enough to investigate? I’m on my mobile right now, I’ll add my own osdmap to the thread soon. Zitat von Eugen Block <eblock@nde.ag>:

Thanks, Frank, I appreciate your help. I already asked for the osdmap, but I’ll also try to find a reproducer.

Zitat von Frank Schilder <frans@dtu.dk>:

...

Hi Eugen,

thanks for this clarification. Yes, with the observations you describe for transition 1->2, something is very wrong. Nothing should happen. Unfortunately, I'm going to be on holidays and, generally, don't have too much time. If they can afford to share the osdmap (ceph osd getmap -o file), I could also take a look at some point.

I don't think it has to do with set_choose_tries, there is likely something else screwed up badly. There should simply not be any remapping going on at this stage. Just for fun, you should be able to produce a clean crushmap from scratch with a similar or the same tree and check if you see the same problems.

Using the full osdmap with osdmaptool allows to reproduce the exact mappings as used in the cluster and it encodes other important information as well. That's why I'm asking for this instead of just the crush map.

Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

________________________________________ From: Eugen Block <eblock@nde.ag> Sent: Thursday, May 23, 2024 1:26 PM To: Frank Schilder Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Re: unknown PGs after adding hosts in different subtree

Hi Frank,

thanks for chiming in here.

...

Please correct if this is wrong. Assuming its correct, I conclude the following.

You assume correctly.

...

Now, from your description it is not clear to me on which of the transitions 1->2 or 2->3 you observe - peering and/or - unknown PGs.

The unknown PGs were observed during/after 1 -> 2. All or almost all PGs were reported as "remapped", I don't remember the exact number, but it was more than 4k, and the largest pool has 4096 PGs. We didn't see down OSDs at all. Only after moving the hosts into their designated location (the DCs) the unknown PGs cleared and the application resumed its operation.

I don't want to overload this thread but I asked for a copy of their crushmap to play around a bit. I moved the new hosts out of the DCs into the default root via 'crushtool --move ...', then running the crushtool --test command

# crushtool -i crushmap --test --rule 1 --num-rep 18 --show-choose-tries [--show-bad-mappings] --show-utilization

results in a couple of issues:

- there are lots of bad mappings no matter how high the number for set_choose_tries is set - the show-utilization output shows 240 OSDs in usage (there were 240 OSDs before the expansion), but plenty of them have only 9 chunks assigned:

rule 1 (rule-ec-k7m11), x = 0..1023, numrep = 18..18 rule 1 (rule-ec-k7m11) num_rep 18 result size == 0: 55/1024 rule 1 (rule-ec-k7m11) num_rep 18 result size == 9: 488/1024 rule 1 (rule-ec-k7m11) num_rep 18 result size == 18: 481/1024

And this reminds me of the inactive PGs we saw before I failed the mgr, those inactive PGs showed only 9 chunks in the acting set. With k=7 (and min_size=8) that should still be enough, we have successfully tested disaster recovery with one entire DC down multiple times.

- with --show-mappings some lines contain an empty set like this:

CRUSH rule 1 x 22 []

And one more observation: with the currently active crushmap there are no bad mappings at all when the hosts are in their designated location. So there's definitely something wrong here, I just can't tell what it is yet. I'll play a bit more with that crushmap...

Thanks! Eugen

Zitat von Frank Schilder <frans@dtu.dk>:

...

Hi Eugen,

I'm afraid the description of your observation breaks a bit with causality and this might be the reason for the few replies. To produce a bit more structure for when exactly what happened, let's look at what I did and didn't get:

Before adding the hosts you have situation

1) default DCA host A1 ... AN DCB host B1 ... BM

Now you add K+L hosts, they go into the default root and we have situation

2) default host C1 ... CK, D1 ... DL DCA host A1 ... AN DCB host B1 ... BM

As a last step, you move the hosts to their final locations and we arrive at situation

3) default DCA host A1 ... AN, C1 ... CK DCB host B1 ... BM, D1 ... DL

Please correct if this is wrong. Assuming its correct, I conclude the following.

Now, from your description it is not clear to me on which of the transitions 1->2 or 2->3 you observe - peering and/or - unknown PGs.

We use a somewhat similar procedure except that we have a second root (separate disjoint tree) for new hosts/OSDs. However, in terms of peering it is the same and if everything is configured correctly I would expect this to happen (this is what happens when we add OSDs/hosts):

transition 1->2: hosts get added: no peering, no remapped objects, nothing, just new OSDs doing nothing transition 2->3: hosts get moved: peering starts and remapped objects appear, all PGs active+clean

Unknown PGs should not occur (maybe only temporarily when the primary changes or the PG is slow to respond/report status??). The crush bug with too few set_choose_tries is observed if one has *just enough hosts* for the EC profile and should not be observed if all PGs are active+clean and one *adds hosts*. Persistent unknown PGs can (to my understanding, does unknown mean "has no primary"?) only occur if the number of PGs changes (autoscaler messing around??) because all PGs were active+clean before. The crush bug leads to incomplete PGs, so PGs can go incomplete but they should always have an acting primary.

This is assuming no OSDs went down/out during the process.

Can you please check if my interpretation is correct and describe at which step exactly things start diverging from my expectations.

Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

________________________________________ From: Eugen Block <eblock@nde.ag> Sent: Thursday, May 23, 2024 12:05 PM To: ceph-users@ceph.io Subject: [ceph-users] Re: unknown PGs after adding hosts in different subtree

Hi again,

I'm still wondering if I misunderstand some of the ceph concepts. Let's assume the choose_tries value is too low and ceph can't find enough OSDs for the remapping. I would expect that there are some PG chunks in remapping state or unknown or whatever, but why would it affect the otherwise healthy cluster in such a way? Even if ceph doesn't know where to put some of the chunks, I wouldn't expect inactive PGs and have a service interruption. What am I missing here?

Thanks, Eugen

Zitat von Eugen Block <eblock@nde.ag>:

...

Thanks, Konstantin. It's been a while since I was last bitten by the choose_tries being too low... Unfortunately, I won't be able to verify that... But I'll definitely keep that in mind, or least I'll try to. :-D

Thanks!

Zitat von Konstantin Shalygin <k0ste@k0ste.ru>:

...

Hi Eugen

...

On 21 May 2024, at 15:26, Eugen Block <eblock@nde.ag> wrote:

step set_choose_tries 100

I think you should try to increase set_choose_tries to 200 Last year we had an Pacific EC 8+2 deployment of 10 racks. And even with 50 hosts, the value of 100 not worked for us

k

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Hi Eugen, I'm at home now. Could you please check all the remapped PGs that they have no shards on the new OSDs, i.e. its just shuffling around mappings within the same set of OSDs under rooms? If this is the case, it is possible that this is partly intentional and partly buggy. The remapping is then probably intentional and the method I use with a disjoint tree for new hosts prevents such remappings initially (the crush code sees the new OSDs in the root, doesn't use them but their presence does change choice orders resulting in remapped PGs). However, the unknown PGs should clearly not occur. I'm afraid that the peering code has quite a few bugs, I reported something at least similarly weird a long time ago: https://tracker.ceph.com/issues/56995 and https://tracker.ceph.com/issues/46847. Might even be related. It looks like peering can loose track of PG members in certain situations (specifically after adding OSDs until rebalancing completed). In my cases, I get degraded objects even though everything is obviously still around. Flipping between the crush-maps before/after the change re-discovers everything again. Issue 46847 is long-standing and still unresolved. In case you need to file a tracker, please consider to refer to the two above as well as "might be related" if you deem that they might be related. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

Hi Frank, thanks for looking up those trackers. I haven't looked into them yet, I'll read your response in detail later, but I wanted to add some new observation: I added another root bucket (custom) to the osd tree: # ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -12 0 root custom -1 0.27698 root default -8 0.09399 room room1 -3 0.04700 host host1 7 hdd 0.02299 osd.7 up 1.00000 1.00000 10 hdd 0.02299 osd.10 up 1.00000 1.00000 ... Then I tried this approach to add a new host directly to the non-default root: # cat host5.yaml service_type: host hostname: host5 addr: 192.168.168.54 location: root: custom labels: - osd # ceph orch apply -i host5.yaml # ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -12 0.04678 root custom -23 0.04678 host host5 1 hdd 0.02339 osd.1 up 1.00000 1.00000 13 hdd 0.02339 osd.13 up 1.00000 1.00000 -1 0.27698 root default -8 0.09399 room room1 -3 0.04700 host host1 7 hdd 0.02299 osd.7 up 1.00000 1.00000 10 hdd 0.02299 osd.10 up 1.00000 1.00000 ... host5 is placed directly underneath the new custom root correctly, but not a single PG is marked "remapped"! So this is actually what I (or we) expected. I'm not sure yet what to make of it, but I'm leaning towards using this approach in the future and add hosts underneath a different root first, and then move it to its designated location. Just to validate again, I added host6 without a location spec, so it's placed underneath the default root again: # ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -12 0.04678 root custom -23 0.04678 host host5 1 hdd 0.02339 osd.1 up 1.00000 1.00000 13 hdd 0.02339 osd.13 up 1.00000 1.00000 -1 0.32376 root default -25 0.04678 host host6 14 hdd 0.02339 osd.14 up 1.00000 1.00000 15 hdd 0.02339 osd.15 up 1.00000 1.00000 -8 0.09399 room room1 -3 0.04700 host host1 ... And this leads to remapped PGs again. I assume this must be related to the default root. I'm gonna investigate further. Thanks! Eugen Zitat von Frank Schilder <frans@dtu.dk>:

Hi Eugen,

just to add another strangeness observation from long ago: https://www.spinics.net/lists/ceph-users/msg74655.html. I didn't see any reweights in your trees, so its something else. However, there seem to be multiple issues with EC pools and peering.

I also want to clarify:

...

If this is the case, it is possible that this is partly intentional and partly buggy.

"Partly intentional" here means the code behaviour changes when you add OSDs to the root outside the rooms and this change is not considered a bug. It is clearly *not* expected as it means you cannot do maintenance on a pool living on a tree A without affecting pools on the same device class living on an unmodified subtree of A.

From a ceph user's point of view everything you observe looks buggy. I would really like to see a good explanation why the mappings in the subtree *should* change when adding OSDs above that subtree as in your case when the expectation for good reasons is that they don't. This would help devising clean procedures for adding hosts when you (and I) want to add OSDs first without any peering and then move OSDs into place to have it happen separate from adding and not a total mess with everything in parallel.

Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: Thursday, May 23, 2024 6:32 PM To: Eugen Block Cc: ceph-users@ceph.io Subject: [ceph-users] Re: unknown PGs after adding hosts in different subtree

Hi Eugen,

I'm at home now. Could you please check all the remapped PGs that they have no shards on the new OSDs, i.e. its just shuffling around mappings within the same set of OSDs under rooms?

If this is the case, it is possible that this is partly intentional and partly buggy. The remapping is then probably intentional and the method I use with a disjoint tree for new hosts prevents such remappings initially (the crush code sees the new OSDs in the root, doesn't use them but their presence does change choice orders resulting in remapped PGs). However, the unknown PGs should clearly not occur.

I'm afraid that the peering code has quite a few bugs, I reported something at least similarly weird a long time ago: https://tracker.ceph.com/issues/56995 and https://tracker.ceph.com/issues/46847. Might even be related. It looks like peering can loose track of PG members in certain situations (specifically after adding OSDs until rebalancing completed). In my cases, I get degraded objects even though everything is obviously still around. Flipping between the crush-maps before/after the change re-discovers everything again.

Issue 46847 is long-standing and still unresolved. In case you need to file a tracker, please consider to refer to the two above as well as "might be related" if you deem that they might be related.

Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Hi Eugen, so it is partly "unexpectedly expected" and partly buggy. I really wish the crush implementation was honouring a few obvious invariants. It is extremely counter-intuitive that mappings taken from a sub-set change even if both, the sub-set and the mapping instructions themselves don't.

- Use different root names

That's what we are doing and it works like a charm, also for draining OSDs.

more specific crush rules.

I guess you mean use something like "step take DCA class hdd" instead of "step take default class hdd" as in: rule rule-ec-k7m11 { id 1 type erasure min_size 3 max_size 18 step set_chooseleaf_tries 5 step set_choose_tries 100 step take DCA class hdd step chooseleaf indep 9 type host step take DCB class hdd step chooseleaf indep 9 type host step emit } According to the documentation, this should actually work and be almost equivalent to your crush rule. The difference here is that it will make sure that the first 9 shards are from DCA and the second 9 shards from DCB (its an ordering). Side effect is that all primary OSDs will be in DCA if both DCs are up. I remember people asking for that as a feature in multi-DC set-ups to pick the one with lowest latency to have the primary OSDs by default. Can you give this crush rule a try and report back whether or not the behaviour when adding hosts changes? In case you have time, it would be great if you could collect information on (reproducing) the fatal peering problem. While remappings might be "unexpectedly expected" it is clearly a serious bug that incomplete and unknown PGs show up in the process of adding hosts at the root. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Eugen Block <eblock@nde.ag> Sent: Friday, May 24, 2024 2:51 PM To: ceph-users@ceph.io Subject: [ceph-users] Re: unknown PGs after adding hosts in different subtree I start to think that the root cause of the remapping is just the fact that the crush rule(s) contain(s) the "step take default" line: step take default class hdd My interpretation is that crush simply tries to honor the rule: consider everything underneath the "default" root, so PGs get remapped if new hosts are added there (but not in their designated subtree buckets). The effect (unknown PGs) is bad, but there are a couple of options to avoid that: - Use different root names and/or more specific crush rules. - Use host spec file(s) to place new hosts directly where they belong. - Set osd_crush_initial_weight = 0 to avoid remapping until everything is where it's supposed to be, then reweight the OSDs. Zitat von Eugen Block <eblock@nde.ag>:

Hi Frank,

thanks for looking up those trackers. I haven't looked into them yet, I'll read your response in detail later, but I wanted to add some new observation:

I added another root bucket (custom) to the osd tree:

# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -12 0 root custom -1 0.27698 root default -8 0.09399 room room1 -3 0.04700 host host1 7 hdd 0.02299 osd.7 up 1.00000 1.00000 10 hdd 0.02299 osd.10 up 1.00000 1.00000 ...

Then I tried this approach to add a new host directly to the non-default root:

# cat host5.yaml service_type: host hostname: host5 addr: 192.168.168.54 location: root: custom labels: - osd

# ceph orch apply -i host5.yaml

# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -12 0.04678 root custom -23 0.04678 host host5 1 hdd 0.02339 osd.1 up 1.00000 1.00000 13 hdd 0.02339 osd.13 up 1.00000 1.00000 -1 0.27698 root default -8 0.09399 room room1 -3 0.04700 host host1 7 hdd 0.02299 osd.7 up 1.00000 1.00000 10 hdd 0.02299 osd.10 up 1.00000 1.00000 ...

host5 is placed directly underneath the new custom root correctly, but not a single PG is marked "remapped"! So this is actually what I (or we) expected. I'm not sure yet what to make of it, but I'm leaning towards using this approach in the future and add hosts underneath a different root first, and then move it to its designated location.

Just to validate again, I added host6 without a location spec, so it's placed underneath the default root again:

# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -12 0.04678 root custom -23 0.04678 host host5 1 hdd 0.02339 osd.1 up 1.00000 1.00000 13 hdd 0.02339 osd.13 up 1.00000 1.00000 -1 0.32376 root default -25 0.04678 host host6 14 hdd 0.02339 osd.14 up 1.00000 1.00000 15 hdd 0.02339 osd.15 up 1.00000 1.00000 -8 0.09399 room room1 -3 0.04700 host host1 ...

And this leads to remapped PGs again. I assume this must be related to the default root. I'm gonna investigate further.

Thanks! Eugen

Zitat von Frank Schilder <frans@dtu.dk>:

...

Hi Eugen,

just to add another strangeness observation from long ago: https://www.spinics.net/lists/ceph-users/msg74655.html. I didn't see any reweights in your trees, so its something else. However, there seem to be multiple issues with EC pools and peering.

I also want to clarify:

...

If this is the case, it is possible that this is partly intentional and partly buggy.

"Partly intentional" here means the code behaviour changes when you add OSDs to the root outside the rooms and this change is not considered a bug. It is clearly *not* expected as it means you cannot do maintenance on a pool living on a tree A without affecting pools on the same device class living on an unmodified subtree of A.

From a ceph user's point of view everything you observe looks buggy. I would really like to see a good explanation why the mappings in the subtree *should* change when adding OSDs above that subtree as in your case when the expectation for good reasons is that they don't. This would help devising clean procedures for adding hosts when you (and I) want to add OSDs first without any peering and then move OSDs into place to have it happen separate from adding and not a total mess with everything in parallel.

Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: Thursday, May 23, 2024 6:32 PM To: Eugen Block Cc: ceph-users@ceph.io Subject: [ceph-users] Re: unknown PGs after adding hosts in different subtree

Hi Eugen,

I'm at home now. Could you please check all the remapped PGs that they have no shards on the new OSDs, i.e. its just shuffling around mappings within the same set of OSDs under rooms?

If this is the case, it is possible that this is partly intentional and partly buggy. The remapping is then probably intentional and the method I use with a disjoint tree for new hosts prevents such remappings initially (the crush code sees the new OSDs in the root, doesn't use them but their presence does change choice orders resulting in remapped PGs). However, the unknown PGs should clearly not occur.

I'm afraid that the peering code has quite a few bugs, I reported something at least similarly weird a long time ago: https://tracker.ceph.com/issues/56995 and https://tracker.ceph.com/issues/46847. Might even be related. It looks like peering can loose track of PG members in certain situations (specifically after adding OSDs until rebalancing completed). In my cases, I get degraded objects even though everything is obviously still around. Flipping between the crush-maps before/after the change re-discovers everything again.

Issue 46847 is long-standing and still unresolved. In case you need to file a tracker, please consider to refer to the two above as well as "might be related" if you deem that they might be related.

Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

Hi, after more testing and investigating I created a tracker issue: https://tracker.ceph.com/issues/66310 My current theory for the inactive/unkown PGs is that the MGR gets overloaded with mon messages. The default for mgr_mon_messages is only 128, the mgr service has taken over more roles over the years, but the defaults haven't been adjusted. Although I couldn't reproduce actual unknown PGs in my lab, I still see quite high get_or_fail_fail messages, even bursts when I add nodes to the lab cluster. This would also explain why we have to fail the mgr so often. It feels like it has become the first suggestion to almost every mgr related issue reported in this list. I don't know what values would make sense, or if mgr_mon_bytes should be increased as well. Regards, Eugen Zitat von Eugen Block <eblock@nde.ag>:

Hi,

...

I guess you mean use something like "step take DCA class hdd" instead of "step take default class hdd" as in:

rule rule-ec-k7m11 { id 1 type erasure min_size 3 max_size 18 step set_chooseleaf_tries 5 step set_choose_tries 100 step take DCA class hdd step chooseleaf indep 9 type host step take DCB class hdd step chooseleaf indep 9 type host step emit }

Almost, yes. There needs to be an "emit" step after the first chooseleaf, so something like this:

...

step take DCA class hdd step chooseleaf indep 9 type host step emit step take DCB class hdd step chooseleaf indep 9 type host step emit

Otherwise the placement according to crushtool would be incomplete and only 9 chunks get a mapping. With this rule (omit "default") there are not bad mappings reported, so that would most likely work as well. But having all primaries in one DC is not optimal, although for this specific customer it probably wouldn't make a difference. But in general I agree, not ideal.

...

In case you have time, it would be great if you could collect information on (reproducing) the fatal peering problem. While remappings might be "unexpectedly expected" it is clearly a serious bug that incomplete and unknown PGs show up in the process of adding hosts at the root.

Time wouldn't be an issue, but there's no way for me to do that on the customer's cluster. In my lab it doesn't behave as observed which isn't surprising without much data and no client load. I'm not sure yet how to achieve that.

Thanks, Eugen

Zitat von Frank Schilder <frans@dtu.dk>:

...

Hi Eugen,

so it is partly "unexpectedly expected" and partly buggy. I really wish the crush implementation was honouring a few obvious invariants. It is extremely counter-intuitive that mappings taken from a sub-set change even if both, the sub-set and the mapping instructions themselves don't.

...

- Use different root names

That's what we are doing and it works like a charm, also for draining OSDs.

...

more specific crush rules.

I guess you mean use something like "step take DCA class hdd" instead of "step take default class hdd" as in:

rule rule-ec-k7m11 { id 1 type erasure min_size 3 max_size 18 step set_chooseleaf_tries 5 step set_choose_tries 100 step take DCA class hdd step chooseleaf indep 9 type host step take DCB class hdd step chooseleaf indep 9 type host step emit }

According to the documentation, this should actually work and be almost equivalent to your crush rule. The difference here is that it will make sure that the first 9 shards are from DCA and the second 9 shards from DCB (its an ordering). Side effect is that all primary OSDs will be in DCA if both DCs are up. I remember people asking for that as a feature in multi-DC set-ups to pick the one with lowest latency to have the primary OSDs by default.

Can you give this crush rule a try and report back whether or not the behaviour when adding hosts changes?

In case you have time, it would be great if you could collect information on (reproducing) the fatal peering problem. While remappings might be "unexpectedly expected" it is clearly a serious bug that incomplete and unknown PGs show up in the process of adding hosts at the root.

Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

________________________________________ From: Eugen Block <eblock@nde.ag> Sent: Friday, May 24, 2024 2:51 PM To: ceph-users@ceph.io Subject: [ceph-users] Re: unknown PGs after adding hosts in different subtree

I start to think that the root cause of the remapping is just the fact that the crush rule(s) contain(s) the "step take default" line:

step take default class hdd

My interpretation is that crush simply tries to honor the rule: consider everything underneath the "default" root, so PGs get remapped if new hosts are added there (but not in their designated subtree buckets). The effect (unknown PGs) is bad, but there are a couple of options to avoid that:

- Use different root names and/or more specific crush rules. - Use host spec file(s) to place new hosts directly where they belong. - Set osd_crush_initial_weight = 0 to avoid remapping until everything is where it's supposed to be, then reweight the OSDs.

Zitat von Eugen Block <eblock@nde.ag>:

...

Hi Frank,

thanks for looking up those trackers. I haven't looked into them yet, I'll read your response in detail later, but I wanted to add some new observation:

I added another root bucket (custom) to the osd tree:

# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -12 0 root custom -1 0.27698 root default -8 0.09399 room room1 -3 0.04700 host host1 7 hdd 0.02299 osd.7 up 1.00000 1.00000 10 hdd 0.02299 osd.10 up 1.00000 1.00000 ...

Then I tried this approach to add a new host directly to the non-default root:

# cat host5.yaml service_type: host hostname: host5 addr: 192.168.168.54 location: root: custom labels: - osd

# ceph orch apply -i host5.yaml

# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -12 0.04678 root custom -23 0.04678 host host5 1 hdd 0.02339 osd.1 up 1.00000 1.00000 13 hdd 0.02339 osd.13 up 1.00000 1.00000 -1 0.27698 root default -8 0.09399 room room1 -3 0.04700 host host1 7 hdd 0.02299 osd.7 up 1.00000 1.00000 10 hdd 0.02299 osd.10 up 1.00000 1.00000 ...

host5 is placed directly underneath the new custom root correctly, but not a single PG is marked "remapped"! So this is actually what I (or we) expected. I'm not sure yet what to make of it, but I'm leaning towards using this approach in the future and add hosts underneath a different root first, and then move it to its designated location.

Just to validate again, I added host6 without a location spec, so it's placed underneath the default root again:

# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -12 0.04678 root custom -23 0.04678 host host5 1 hdd 0.02339 osd.1 up 1.00000 1.00000 13 hdd 0.02339 osd.13 up 1.00000 1.00000 -1 0.32376 root default -25 0.04678 host host6 14 hdd 0.02339 osd.14 up 1.00000 1.00000 15 hdd 0.02339 osd.15 up 1.00000 1.00000 -8 0.09399 room room1 -3 0.04700 host host1 ...

And this leads to remapped PGs again. I assume this must be related to the default root. I'm gonna investigate further.

Thanks! Eugen

Zitat von Frank Schilder <frans@dtu.dk>:

...

Hi Eugen,

just to add another strangeness observation from long ago: https://www.spinics.net/lists/ceph-users/msg74655.html. I didn't see any reweights in your trees, so its something else. However, there seem to be multiple issues with EC pools and peering.

I also want to clarify:

...

If this is the case, it is possible that this is partly intentional and partly buggy.

"Partly intentional" here means the code behaviour changes when you add OSDs to the root outside the rooms and this change is not considered a bug. It is clearly *not* expected as it means you cannot do maintenance on a pool living on a tree A without affecting pools on the same device class living on an unmodified subtree of A.

From a ceph user's point of view everything you observe looks buggy. I would really like to see a good explanation why the mappings in the subtree *should* change when adding OSDs above that subtree as in your case when the expectation for good reasons is that they don't. This would help devising clean procedures for adding hosts when you (and I) want to add OSDs first without any peering and then move OSDs into place to have it happen separate from adding and not a total mess with everything in parallel.

Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

________________________________________ From: Frank Schilder <frans@dtu.dk> Sent: Thursday, May 23, 2024 6:32 PM To: Eugen Block Cc: ceph-users@ceph.io Subject: [ceph-users] Re: unknown PGs after adding hosts in different subtree

Hi Eugen,

I'm at home now. Could you please check all the remapped PGs that they have no shards on the new OSDs, i.e. its just shuffling around mappings within the same set of OSDs under rooms?

If this is the case, it is possible that this is partly intentional and partly buggy. The remapping is then probably intentional and the method I use with a disjoint tree for new hosts prevents such remappings initially (the crush code sees the new OSDs in the root, doesn't use them but their presence does change choice orders resulting in remapped PGs). However, the unknown PGs should clearly not occur.

I'm afraid that the peering code has quite a few bugs, I reported something at least similarly weird a long time ago: https://tracker.ceph.com/issues/56995 and https://tracker.ceph.com/issues/46847. Might even be related. It looks like peering can loose track of PG members in certain situations (specifically after adding OSDs until rebalancing completed). In my cases, I get degraded objects even though everything is obviously still around. Flipping between the crush-maps before/after the change re-discovers everything again.

Issue 46847 is long-standing and still unresolved. In case you need to file a tracker, please consider to refer to the two above as well as "might be related" if you deem that they might be related.

Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io

_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io