Best way to change bucket hierarchy
Helo, I have a live ceph cluster, and I’m in the need of modifying the bucket hierarchy. I am currently using the default crush rule (ie. keep each replica on a different host). My need is to add a “chassis” level, and keep replicas on a per-chassis level. From what I read in the documentation, I would have to edit the crush file manually, however this sounds kinda scary for a live cluster. Are there any “best known methods” to achieve that goal without messing things up? In my current scenario, I have one host per chassis, and planning on later adding nodes where there would be >1 hosts per chassis. It looks like “in theory” there wouldn’t be a need for any data movement after the crush map changes. Will reality match theory? Anything else I need to watch out for? Thank you! George
You can use the command-line without editing the crush map. Look at the documentation of commands like ceph osd crush add-bucket ... ceph osd crush move ... Before starting this, set "ceph osd set norebalance" and unset after you are happy with the crush tree. Let everything peer. You should see misplaced objects and remapped PGs, but no degraded objects or PGs. Do this only when cluster is helth_ok, otherwise things can get really complicated. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Kyriazis, George <george.kyriazis@intel.com> Sent: 03 June 2020 22:45:11 To: ceph-users Subject: [ceph-users] Best way to change bucket hierarchy Helo, I have a live ceph cluster, and I’m in the need of modifying the bucket hierarchy. I am currently using the default crush rule (ie. keep each replica on a different host). My need is to add a “chassis” level, and keep replicas on a per-chassis level. From what I read in the documentation, I would have to edit the crush file manually, however this sounds kinda scary for a live cluster. Are there any “best known methods” to achieve that goal without messing things up? In my current scenario, I have one host per chassis, and planning on later adding nodes where there would be >1 hosts per chassis. It looks like “in theory” there wouldn’t be a need for any data movement after the crush map changes. Will reality match theory? Anything else I need to watch out for? Thank you! George _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks Frank, I don’t have too much experience editing crush rules, but I assume the chooseleaf step would also have to change to: step chooseleaf firstn 0 type chassis Correct? Is that the only other change that is needed? It looks like the rule change can happen both inside and outside the “norebalance” setting (again with CLI commands), but is it safer to do it inside (ie. while not rebalancing)? If I keep a backup of the crush rule map (with “ceph osd getcrushmap”), I assume I can restore the old map if something goes bad? Thanks again! George
On Jun 3, 2020, at 5:24 PM, Frank Schilder <frans@dtu.dk> wrote:
You can use the command-line without editing the crush map. Look at the documentation of commands like
ceph osd crush add-bucket ... ceph osd crush move ...
Before starting this, set "ceph osd set norebalance" and unset after you are happy with the crush tree. Let everything peer. You should see misplaced objects and remapped PGs, but no degraded objects or PGs.
Do this only when cluster is helth_ok, otherwise things can get really complicated.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Kyriazis, George <george.kyriazis@intel.com> Sent: 03 June 2020 22:45:11 To: ceph-users Subject: [ceph-users] Best way to change bucket hierarchy
Helo,
I have a live ceph cluster, and I’m in the need of modifying the bucket hierarchy. I am currently using the default crush rule (ie. keep each replica on a different host). My need is to add a “chassis” level, and keep replicas on a per-chassis level.
From what I read in the documentation, I would have to edit the crush file manually, however this sounds kinda scary for a live cluster.
Are there any “best known methods” to achieve that goal without messing things up?
In my current scenario, I have one host per chassis, and planning on later adding nodes where there would be >1 hosts per chassis. It looks like “in theory” there wouldn’t be a need for any data movement after the crush map changes. Will reality match theory? Anything else I need to watch out for?
Thank you!
George
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
I think 'chassis' is OK. If you change host to chassis, you should have chassis declaration in the crushmap, as osds and hosts do. Using command for example, "ceph osd crush add-bucket chassis-1 chassis" and "ceph osd crush move host-1 chassis=chassis-1" should be executed.
Yes, that’s what I had in mind. Thank you! George
On Jun 3, 2020, at 8:41 PM, Zhenshi Zhou <deaderzzs@gmail.com> wrote:
I think 'chassis' is OK. If you change host to chassis, you should have chassis declaration in the crushmap, as osds and hosts do. Using command for example, "ceph osd crush add-bucket chassis-1 chassis" and "ceph osd crush move host-1 chassis=chassis-1" should be executed.
Hi George, for replicated rules you can simply create a new crush rule with the new failure domain set to chassis and change any pool's crush rule to this new one. If you have EC pools, then the chooseleaf needs to be edited by hand. I did this before as well. (A really unfortunate side effect is, that the EC profile attached to the pool goes out of sync with the crush map and there is nothing one can do about that. This is annoying yet harmless.) The intend of doing these changes while norebalance is set is - to avoid unnecessary data movement due to successive changes happening step by step and - to make sure peering is successful before starting to move data. I believe OSDs peer a bit faster with norebalance set and there is then a shorter interrupt to ongoing I/O (no I/O happens to a PG during peering). Yes, if you safe the old crush map, you can undo everything. It is a good idea to have a backup also just for reference and to compare before and after. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Kyriazis, George <george.kyriazis@intel.com> Sent: 04 June 2020 00:58:20 To: Frank Schilder Cc: ceph-users Subject: Re: Best way to change bucket hierarchy Thanks Frank, I don’t have too much experience editing crush rules, but I assume the chooseleaf step would also have to change to: step chooseleaf firstn 0 type chassis Correct? Is that the only other change that is needed? It looks like the rule change can happen both inside and outside the “norebalance” setting (again with CLI commands), but is it safer to do it inside (ie. while not rebalancing)? If I keep a backup of the crush rule map (with “ceph osd getcrushmap”), I assume I can restore the old map if something goes bad? Thanks again! George
On Jun 3, 2020, at 5:24 PM, Frank Schilder <frans@dtu.dk> wrote:
You can use the command-line without editing the crush map. Look at the documentation of commands like
ceph osd crush add-bucket ... ceph osd crush move ...
Before starting this, set "ceph osd set norebalance" and unset after you are happy with the crush tree. Let everything peer. You should see misplaced objects and remapped PGs, but no degraded objects or PGs.
Do this only when cluster is helth_ok, otherwise things can get really complicated.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Kyriazis, George <george.kyriazis@intel.com> Sent: 03 June 2020 22:45:11 To: ceph-users Subject: [ceph-users] Best way to change bucket hierarchy
Helo,
I have a live ceph cluster, and I’m in the need of modifying the bucket hierarchy. I am currently using the default crush rule (ie. keep each replica on a different host). My need is to add a “chassis” level, and keep replicas on a per-chassis level.
From what I read in the documentation, I would have to edit the crush file manually, however this sounds kinda scary for a live cluster.
Are there any “best known methods” to achieve that goal without messing things up?
In my current scenario, I have one host per chassis, and planning on later adding nodes where there would be >1 hosts per chassis. It looks like “in theory” there wouldn’t be a need for any data movement after the crush map changes. Will reality match theory? Anything else I need to watch out for?
Thank you!
George
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Thanks Frank, Interesting info about the EC profile. I do have an EC pool, but I noticed the following when I dumped the profile: # ceph osd erasure-code-profile get ec22 crush-device-class=hdd crush-failure-domain=host crush-root=default jerasure-per-chunk-alignment=false k=2 m=2 plugin=jerasure technique=reed_sol_van w=8 # Which says that the failure domain of the EC profile is also set to host. Looks like I need to change the EC profile, too, but since it associated with the pool, maybe I can’t do that after pool creation? Or…. Since it the property is named “crush-failure-domain”, it’s automatically inherited from the crush profile, so I don’t have to do anything? Thanks, George On Jun 4, 2020, at 1:51 AM, Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk>> wrote: Hi George, for replicated rules you can simply create a new crush rule with the new failure domain set to chassis and change any pool's crush rule to this new one. If you have EC pools, then the chooseleaf needs to be edited by hand. I did this before as well. (A really unfortunate side effect is, that the EC profile attached to the pool goes out of sync with the crush map and there is nothing one can do about that. This is annoying yet harmless.) The intend of doing these changes while norebalance is set is - to avoid unnecessary data movement due to successive changes happening step by step and - to make sure peering is successful before starting to move data. I believe OSDs peer a bit faster with norebalance set and there is then a shorter interrupt to ongoing I/O (no I/O happens to a PG during peering). Yes, if you safe the old crush map, you can undo everything. It is a good idea to have a backup also just for reference and to compare before and after. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Kyriazis, George <george.kyriazis@intel.com<mailto:george.kyriazis@intel.com>> Sent: 04 June 2020 00:58:20 To: Frank Schilder Cc: ceph-users Subject: Re: Best way to change bucket hierarchy Thanks Frank, I don’t have too much experience editing crush rules, but I assume the chooseleaf step would also have to change to: step chooseleaf firstn 0 type chassis Correct? Is that the only other change that is needed? It looks like the rule change can happen both inside and outside the “norebalance” setting (again with CLI commands), but is it safer to do it inside (ie. while not rebalancing)? If I keep a backup of the crush rule map (with “ceph osd getcrushmap”), I assume I can restore the old map if something goes bad? Thanks again! George On Jun 3, 2020, at 5:24 PM, Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk>> wrote: You can use the command-line without editing the crush map. Look at the documentation of commands like ceph osd crush add-bucket ... ceph osd crush move ... Before starting this, set "ceph osd set norebalance" and unset after you are happy with the crush tree. Let everything peer. You should see misplaced objects and remapped PGs, but no degraded objects or PGs. Do this only when cluster is helth_ok, otherwise things can get really complicated. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Kyriazis, George <george.kyriazis@intel.com<mailto:george.kyriazis@intel.com>> Sent: 03 June 2020 22:45:11 To: ceph-users Subject: [ceph-users] Best way to change bucket hierarchy Helo, I have a live ceph cluster, and I’m in the need of modifying the bucket hierarchy. I am currently using the default crush rule (ie. keep each replica on a different host). My need is to add a “chassis” level, and keep replicas on a per-chassis level. From what I read in the documentation, I would have to edit the crush file manually, however this sounds kinda scary for a live cluster. Are there any “best known methods” to achieve that goal without messing things up? In my current scenario, I have one host per chassis, and planning on later adding nodes where there would be >1 hosts per chassis. It looks like “in theory” there wouldn’t be a need for any data movement after the crush map changes. Will reality match theory? Anything else I need to watch out for? Thank you! George _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
Hi George, you don't need to worry about that too much. The EC profile contains two types of information, one part about the actual EC encoding and another part about crush parameters. Unfortunately, actually. Part of this information is mutable after pool creation while the rest is not. Mutable here means outside of the profile. You can change the failure domain in the crush map without issues, but the profile won't reflect that change. That's an inconsistency we currently have to live with and it would have been better to separate mutable data (like failure domain) from immutable data (like k and m) or provide a meaningful interface to maintain consistency of mutable information. In short, don't believe everything the EC profile tells you. Some information might be out of date, like the failure domain or the device class (basically everything starting with crush-). If you remember that, you are out of trouble. Always dump the crush rule of an EC pool explicitly to see the true parameters in action. Having said that, to change the failure domain for an EC pool, change the crush rule for the EC profile - I did this too and it works just fine. The crush rule has by default the same name as the pool. I'm afraid, here you will have to do a manual edit of the crush rule as Wido explained. There is no other way - at least currently not. You can ask in this list for confirmation that your change is doing what you want. Do not try to touch an EC profile, they are read-only any ways. The crush parameters are only used at pool creation and never looked at again. You can override these by editing the crush rule as explained above. Best regards and good luck, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Kyriazis, George <george.kyriazis@intel.com> Sent: 04 June 2020 20:56:38 To: Frank Schilder Cc: ceph-users Subject: Re: Best way to change bucket hierarchy Thanks Frank, Interesting info about the EC profile. I do have an EC pool, but I noticed the following when I dumped the profile: # ceph osd erasure-code-profile get ec22 crush-device-class=hdd crush-failure-domain=host crush-root=default jerasure-per-chunk-alignment=false k=2 m=2 plugin=jerasure technique=reed_sol_van w=8 # Which says that the failure domain of the EC profile is also set to host. Looks like I need to change the EC profile, too, but since it associated with the pool, maybe I can’t do that after pool creation? Or…. Since it the property is named “crush-failure-domain”, it’s automatically inherited from the crush profile, so I don’t have to do anything? Thanks, George On Jun 4, 2020, at 1:51 AM, Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk>> wrote: Hi George, for replicated rules you can simply create a new crush rule with the new failure domain set to chassis and change any pool's crush rule to this new one. If you have EC pools, then the chooseleaf needs to be edited by hand. I did this before as well. (A really unfortunate side effect is, that the EC profile attached to the pool goes out of sync with the crush map and there is nothing one can do about that. This is annoying yet harmless.) The intend of doing these changes while norebalance is set is - to avoid unnecessary data movement due to successive changes happening step by step and - to make sure peering is successful before starting to move data. I believe OSDs peer a bit faster with norebalance set and there is then a shorter interrupt to ongoing I/O (no I/O happens to a PG during peering). Yes, if you safe the old crush map, you can undo everything. It is a good idea to have a backup also just for reference and to compare before and after. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Kyriazis, George <george.kyriazis@intel.com<mailto:george.kyriazis@intel.com>> Sent: 04 June 2020 00:58:20 To: Frank Schilder Cc: ceph-users Subject: Re: Best way to change bucket hierarchy Thanks Frank, I don’t have too much experience editing crush rules, but I assume the chooseleaf step would also have to change to: step chooseleaf firstn 0 type chassis Correct? Is that the only other change that is needed? It looks like the rule change can happen both inside and outside the “norebalance” setting (again with CLI commands), but is it safer to do it inside (ie. while not rebalancing)? If I keep a backup of the crush rule map (with “ceph osd getcrushmap”), I assume I can restore the old map if something goes bad? Thanks again! George On Jun 3, 2020, at 5:24 PM, Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk>> wrote: You can use the command-line without editing the crush map. Look at the documentation of commands like ceph osd crush add-bucket ... ceph osd crush move ... Before starting this, set "ceph osd set norebalance" and unset after you are happy with the crush tree. Let everything peer. You should see misplaced objects and remapped PGs, but no degraded objects or PGs. Do this only when cluster is helth_ok, otherwise things can get really complicated. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Kyriazis, George <george.kyriazis@intel.com<mailto:george.kyriazis@intel.com>> Sent: 03 June 2020 22:45:11 To: ceph-users Subject: [ceph-users] Best way to change bucket hierarchy Helo, I have a live ceph cluster, and I’m in the need of modifying the bucket hierarchy. I am currently using the default crush rule (ie. keep each replica on a different host). My need is to add a “chassis” level, and keep replicas on a per-chassis level. From what I read in the documentation, I would have to edit the crush file manually, however this sounds kinda scary for a live cluster. Are there any “best known methods” to achieve that goal without messing things up? In my current scenario, I have one host per chassis, and planning on later adding nodes where there would be >1 hosts per chassis. It looks like “in theory” there wouldn’t be a need for any data movement after the crush map changes. Will reality match theory? Anything else I need to watch out for? Thank you! George _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io>
Hmm, So I tried all that, and I got almost all of my PGs being remapped. Crush map looks correct. Is that normal? Thanks, George On Jun 4, 2020, at 2:33 PM, Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk>> wrote: Hi George, you don't need to worry about that too much. The EC profile contains two types of information, one part about the actual EC encoding and another part about crush parameters. Unfortunately, actually. Part of this information is mutable after pool creation while the rest is not. Mutable here means outside of the profile. You can change the failure domain in the crush map without issues, but the profile won't reflect that change. That's an inconsistency we currently have to live with and it would have been better to separate mutable data (like failure domain) from immutable data (like k and m) or provide a meaningful interface to maintain consistency of mutable information. In short, don't believe everything the EC profile tells you. Some information might be out of date, like the failure domain or the device class (basically everything starting with crush-). If you remember that, you are out of trouble. Always dump the crush rule of an EC pool explicitly to see the true parameters in action. Having said that, to change the failure domain for an EC pool, change the crush rule for the EC profile - I did this too and it works just fine. The crush rule has by default the same name as the pool. I'm afraid, here you will have to do a manual edit of the crush rule as Wido explained. There is no other way - at least currently not. You can ask in this list for confirmation that your change is doing what you want. Do not try to touch an EC profile, they are read-only any ways. The crush parameters are only used at pool creation and never looked at again. You can override these by editing the crush rule as explained above. Best regards and good luck, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Kyriazis, George <george.kyriazis@intel.com<mailto:george.kyriazis@intel.com>> Sent: 04 June 2020 20:56:38 To: Frank Schilder Cc: ceph-users Subject: Re: Best way to change bucket hierarchy Thanks Frank, Interesting info about the EC profile. I do have an EC pool, but I noticed the following when I dumped the profile: # ceph osd erasure-code-profile get ec22 crush-device-class=hdd crush-failure-domain=host crush-root=default jerasure-per-chunk-alignment=false k=2 m=2 plugin=jerasure technique=reed_sol_van w=8 # Which says that the failure domain of the EC profile is also set to host. Looks like I need to change the EC profile, too, but since it associated with the pool, maybe I can’t do that after pool creation? Or…. Since it the property is named “crush-failure-domain”, it’s automatically inherited from the crush profile, so I don’t have to do anything? Thanks, George On Jun 4, 2020, at 1:51 AM, Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk><mailto:frans@dtu.dk>> wrote: Hi George, for replicated rules you can simply create a new crush rule with the new failure domain set to chassis and change any pool's crush rule to this new one. If you have EC pools, then the chooseleaf needs to be edited by hand. I did this before as well. (A really unfortunate side effect is, that the EC profile attached to the pool goes out of sync with the crush map and there is nothing one can do about that. This is annoying yet harmless.) The intend of doing these changes while norebalance is set is - to avoid unnecessary data movement due to successive changes happening step by step and - to make sure peering is successful before starting to move data. I believe OSDs peer a bit faster with norebalance set and there is then a shorter interrupt to ongoing I/O (no I/O happens to a PG during peering). Yes, if you safe the old crush map, you can undo everything. It is a good idea to have a backup also just for reference and to compare before and after. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Kyriazis, George <george.kyriazis@intel.com<mailto:george.kyriazis@intel.com><mailto:george.kyriazis@intel.com>> Sent: 04 June 2020 00:58:20 To: Frank Schilder Cc: ceph-users Subject: Re: Best way to change bucket hierarchy Thanks Frank, I don’t have too much experience editing crush rules, but I assume the chooseleaf step would also have to change to: step chooseleaf firstn 0 type chassis Correct? Is that the only other change that is needed? It looks like the rule change can happen both inside and outside the “norebalance” setting (again with CLI commands), but is it safer to do it inside (ie. while not rebalancing)? If I keep a backup of the crush rule map (with “ceph osd getcrushmap”), I assume I can restore the old map if something goes bad? Thanks again! George On Jun 3, 2020, at 5:24 PM, Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk><mailto:frans@dtu.dk>> wrote: You can use the command-line without editing the crush map. Look at the documentation of commands like ceph osd crush add-bucket ... ceph osd crush move ... Before starting this, set "ceph osd set norebalance" and unset after you are happy with the crush tree. Let everything peer. You should see misplaced objects and remapped PGs, but no degraded objects or PGs. Do this only when cluster is helth_ok, otherwise things can get really complicated. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Kyriazis, George <george.kyriazis@intel.com<mailto:george.kyriazis@intel.com><mailto:george.kyriazis@intel.com>> Sent: 03 June 2020 22:45:11 To: ceph-users Subject: [ceph-users] Best way to change bucket hierarchy Helo, I have a live ceph cluster, and I’m in the need of modifying the bucket hierarchy. I am currently using the default crush rule (ie. keep each replica on a different host). My need is to add a “chassis” level, and keep replicas on a per-chassis level. From what I read in the documentation, I would have to edit the crush file manually, however this sounds kinda scary for a live cluster. Are there any “best known methods” to achieve that goal without messing things up? In my current scenario, I have one host per chassis, and planning on later adding nodes where there would be >1 hosts per chassis. It looks like “in theory” there wouldn’t be a need for any data movement after the crush map changes. Will reality match theory? Anything else I need to watch out for? Thank you! George _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io><mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io><mailto:ceph-users-leave@ceph.io>
Its hard to tell without knowing what the diff is, but from your description I take it that you changed the failure domain for every(?) pool from host to chassis. I don't know what a chassis is in your architecture, but if each chassis contains several host buckets, then yes, I would expect almost every PG to be affected. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Kyriazis, George <george.kyriazis@intel.com> Sent: 05 June 2020 00:28:43 To: Frank Schilder Cc: ceph-users Subject: Re: Best way to change bucket hierarchy Hmm, So I tried all that, and I got almost all of my PGs being remapped. Crush map looks correct. Is that normal? Thanks, George On Jun 4, 2020, at 2:33 PM, Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk>> wrote: Hi George, you don't need to worry about that too much. The EC profile contains two types of information, one part about the actual EC encoding and another part about crush parameters. Unfortunately, actually. Part of this information is mutable after pool creation while the rest is not. Mutable here means outside of the profile. You can change the failure domain in the crush map without issues, but the profile won't reflect that change. That's an inconsistency we currently have to live with and it would have been better to separate mutable data (like failure domain) from immutable data (like k and m) or provide a meaningful interface to maintain consistency of mutable information. In short, don't believe everything the EC profile tells you. Some information might be out of date, like the failure domain or the device class (basically everything starting with crush-). If you remember that, you are out of trouble. Always dump the crush rule of an EC pool explicitly to see the true parameters in action. Having said that, to change the failure domain for an EC pool, change the crush rule for the EC profile - I did this too and it works just fine. The crush rule has by default the same name as the pool. I'm afraid, here you will have to do a manual edit of the crush rule as Wido explained. There is no other way - at least currently not. You can ask in this list for confirmation that your change is doing what you want. Do not try to touch an EC profile, they are read-only any ways. The crush parameters are only used at pool creation and never looked at again. You can override these by editing the crush rule as explained above. Best regards and good luck, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Kyriazis, George <george.kyriazis@intel.com<mailto:george.kyriazis@intel.com>> Sent: 04 June 2020 20:56:38 To: Frank Schilder Cc: ceph-users Subject: Re: Best way to change bucket hierarchy Thanks Frank, Interesting info about the EC profile. I do have an EC pool, but I noticed the following when I dumped the profile: # ceph osd erasure-code-profile get ec22 crush-device-class=hdd crush-failure-domain=host crush-root=default jerasure-per-chunk-alignment=false k=2 m=2 plugin=jerasure technique=reed_sol_van w=8 # Which says that the failure domain of the EC profile is also set to host. Looks like I need to change the EC profile, too, but since it associated with the pool, maybe I can’t do that after pool creation? Or…. Since it the property is named “crush-failure-domain”, it’s automatically inherited from the crush profile, so I don’t have to do anything? Thanks, George On Jun 4, 2020, at 1:51 AM, Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk><mailto:frans@dtu.dk>> wrote: Hi George, for replicated rules you can simply create a new crush rule with the new failure domain set to chassis and change any pool's crush rule to this new one. If you have EC pools, then the chooseleaf needs to be edited by hand. I did this before as well. (A really unfortunate side effect is, that the EC profile attached to the pool goes out of sync with the crush map and there is nothing one can do about that. This is annoying yet harmless.) The intend of doing these changes while norebalance is set is - to avoid unnecessary data movement due to successive changes happening step by step and - to make sure peering is successful before starting to move data. I believe OSDs peer a bit faster with norebalance set and there is then a shorter interrupt to ongoing I/O (no I/O happens to a PG during peering). Yes, if you safe the old crush map, you can undo everything. It is a good idea to have a backup also just for reference and to compare before and after. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Kyriazis, George <george.kyriazis@intel.com<mailto:george.kyriazis@intel.com><mailto:george.kyriazis@intel.com>> Sent: 04 June 2020 00:58:20 To: Frank Schilder Cc: ceph-users Subject: Re: Best way to change bucket hierarchy Thanks Frank, I don’t have too much experience editing crush rules, but I assume the chooseleaf step would also have to change to: step chooseleaf firstn 0 type chassis Correct? Is that the only other change that is needed? It looks like the rule change can happen both inside and outside the “norebalance” setting (again with CLI commands), but is it safer to do it inside (ie. while not rebalancing)? If I keep a backup of the crush rule map (with “ceph osd getcrushmap”), I assume I can restore the old map if something goes bad? Thanks again! George On Jun 3, 2020, at 5:24 PM, Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk><mailto:frans@dtu.dk>> wrote: You can use the command-line without editing the crush map. Look at the documentation of commands like ceph osd crush add-bucket ... ceph osd crush move ... Before starting this, set "ceph osd set norebalance" and unset after you are happy with the crush tree. Let everything peer. You should see misplaced objects and remapped PGs, but no degraded objects or PGs. Do this only when cluster is helth_ok, otherwise things can get really complicated. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Kyriazis, George <george.kyriazis@intel.com<mailto:george.kyriazis@intel.com><mailto:george.kyriazis@intel.com>> Sent: 03 June 2020 22:45:11 To: ceph-users Subject: [ceph-users] Best way to change bucket hierarchy Helo, I have a live ceph cluster, and I’m in the need of modifying the bucket hierarchy. I am currently using the default crush rule (ie. keep each replica on a different host). My need is to add a “chassis” level, and keep replicas on a per-chassis level. From what I read in the documentation, I would have to edit the crush file manually, however this sounds kinda scary for a live cluster. Are there any “best known methods” to achieve that goal without messing things up? In my current scenario, I have one host per chassis, and planning on later adding nodes where there would be >1 hosts per chassis. It looks like “in theory” there wouldn’t be a need for any data movement after the crush map changes. Will reality match theory? Anything else I need to watch out for? Thank you! George _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io><mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io><mailto:ceph-users-leave@ceph.io>
Understand that it’s difficult to debug remotely. :-) In my current scenario I have 5 machines (1 host per chassis), but planning on adding some additional chassis with 4 hosts per chassis in the near future. Currently I am going through the first stage of adding “stub” chassis for the 5 hosts/chassis that I have, basically reparenting each host to its own chassis, as shown below: ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 203.72598 root default -5 40.01700 chassis chassis-hsw1 -9 40.01700 host vis-hsw-01 3 hdd 10.91299 osd.3 up 1.00000 1.00000 6 hdd 14.55199 osd.6 up 1.00000 1.00000 10 hdd 14.55199 osd.10 up 1.00000 1.00000 -6 40.01700 chassis chassis-hsw2 -13 40.01700 host vis-hsw-02 0 hdd 10.91299 osd.0 up 1.00000 1.00000 7 hdd 14.55199 osd.7 up 1.00000 1.00000 11 hdd 14.55199 osd.11 up 1.00000 1.00000 -7 40.01700 chassis chassis-hsw3 -11 40.01700 host vis-hsw-03 4 hdd 10.91299 osd.4 up 1.00000 1.00000 8 hdd 14.55199 osd.8 up 1.00000 1.00000 12 hdd 14.55199 osd.12 up 1.00000 1.00000 -8 40.01700 chassis chassis-hsw4 -3 40.01700 host vis-hsw-04 5 hdd 10.91299 osd.5 up 1.00000 1.00000 9 hdd 14.55199 osd.9 up 1.00000 1.00000 13 hdd 14.55199 osd.13 up 1.00000 1.00000 -17 43.65799 chassis chassis-hsw5 -15 43.65799 host vis-hsw-05 1 hdd 14.55299 osd.1 up 1.00000 1.00000 2 hdd 14.55299 osd.2 up 1.00000 1.00000 14 hdd 14.55299 osd.14 up 1.00000 1.00000 There is no additional constraint that is being added, so ideally there would be no data movement. However, I can imagine that the CRUSH algorithm could hash the PGs into different OSDs now because there is a new thing to consider (namely the chassis). Does it do that? Thanks, George On Jun 4, 2020, at 6:22 PM, Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk>> wrote: Its hard to tell without knowing what the diff is, but from your description I take it that you changed the failure domain for every(?) pool from host to chassis. I don't know what a chassis is in your architecture, but if each chassis contains several host buckets, then yes, I would expect almost every PG to be affected. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Kyriazis, George <george.kyriazis@intel.com<mailto:george.kyriazis@intel.com>> Sent: 05 June 2020 00:28:43 To: Frank Schilder Cc: ceph-users Subject: Re: Best way to change bucket hierarchy Hmm, So I tried all that, and I got almost all of my PGs being remapped. Crush map looks correct. Is that normal? Thanks, George On Jun 4, 2020, at 2:33 PM, Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk><mailto:frans@dtu.dk>> wrote: Hi George, you don't need to worry about that too much. The EC profile contains two types of information, one part about the actual EC encoding and another part about crush parameters. Unfortunately, actually. Part of this information is mutable after pool creation while the rest is not. Mutable here means outside of the profile. You can change the failure domain in the crush map without issues, but the profile won't reflect that change. That's an inconsistency we currently have to live with and it would have been better to separate mutable data (like failure domain) from immutable data (like k and m) or provide a meaningful interface to maintain consistency of mutable information. In short, don't believe everything the EC profile tells you. Some information might be out of date, like the failure domain or the device class (basically everything starting with crush-). If you remember that, you are out of trouble. Always dump the crush rule of an EC pool explicitly to see the true parameters in action. Having said that, to change the failure domain for an EC pool, change the crush rule for the EC profile - I did this too and it works just fine. The crush rule has by default the same name as the pool. I'm afraid, here you will have to do a manual edit of the crush rule as Wido explained. There is no other way - at least currently not. You can ask in this list for confirmation that your change is doing what you want. Do not try to touch an EC profile, they are read-only any ways. The crush parameters are only used at pool creation and never looked at again. You can override these by editing the crush rule as explained above. Best regards and good luck, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Kyriazis, George <george.kyriazis@intel.com<mailto:george.kyriazis@intel.com><mailto:george.kyriazis@intel.com>> Sent: 04 June 2020 20:56:38 To: Frank Schilder Cc: ceph-users Subject: Re: Best way to change bucket hierarchy Thanks Frank, Interesting info about the EC profile. I do have an EC pool, but I noticed the following when I dumped the profile: # ceph osd erasure-code-profile get ec22 crush-device-class=hdd crush-failure-domain=host crush-root=default jerasure-per-chunk-alignment=false k=2 m=2 plugin=jerasure technique=reed_sol_van w=8 # Which says that the failure domain of the EC profile is also set to host. Looks like I need to change the EC profile, too, but since it associated with the pool, maybe I can’t do that after pool creation? Or…. Since it the property is named “crush-failure-domain”, it’s automatically inherited from the crush profile, so I don’t have to do anything? Thanks, George On Jun 4, 2020, at 1:51 AM, Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk><mailto:frans@dtu.dk><mailto:frans@dtu.dk>> wrote: Hi George, for replicated rules you can simply create a new crush rule with the new failure domain set to chassis and change any pool's crush rule to this new one. If you have EC pools, then the chooseleaf needs to be edited by hand. I did this before as well. (A really unfortunate side effect is, that the EC profile attached to the pool goes out of sync with the crush map and there is nothing one can do about that. This is annoying yet harmless.) The intend of doing these changes while norebalance is set is - to avoid unnecessary data movement due to successive changes happening step by step and - to make sure peering is successful before starting to move data. I believe OSDs peer a bit faster with norebalance set and there is then a shorter interrupt to ongoing I/O (no I/O happens to a PG during peering). Yes, if you safe the old crush map, you can undo everything. It is a good idea to have a backup also just for reference and to compare before and after. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Kyriazis, George <george.kyriazis@intel.com<mailto:george.kyriazis@intel.com><mailto:george.kyriazis@intel.com><mailto:george.kyriazis@intel.com>> Sent: 04 June 2020 00:58:20 To: Frank Schilder Cc: ceph-users Subject: Re: Best way to change bucket hierarchy Thanks Frank, I don’t have too much experience editing crush rules, but I assume the chooseleaf step would also have to change to: step chooseleaf firstn 0 type chassis Correct? Is that the only other change that is needed? It looks like the rule change can happen both inside and outside the “norebalance” setting (again with CLI commands), but is it safer to do it inside (ie. while not rebalancing)? If I keep a backup of the crush rule map (with “ceph osd getcrushmap”), I assume I can restore the old map if something goes bad? Thanks again! George On Jun 3, 2020, at 5:24 PM, Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk><mailto:frans@dtu.dk><mailto:frans@dtu.dk>> wrote: You can use the command-line without editing the crush map. Look at the documentation of commands like ceph osd crush add-bucket ... ceph osd crush move ... Before starting this, set "ceph osd set norebalance" and unset after you are happy with the crush tree. Let everything peer. You should see misplaced objects and remapped PGs, but no degraded objects or PGs. Do this only when cluster is helth_ok, otherwise things can get really complicated. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Kyriazis, George <george.kyriazis@intel.com<mailto:george.kyriazis@intel.com><mailto:george.kyriazis@intel.com><mailto:george.kyriazis@intel.com>> Sent: 03 June 2020 22:45:11 To: ceph-users Subject: [ceph-users] Best way to change bucket hierarchy Helo, I have a live ceph cluster, and I’m in the need of modifying the bucket hierarchy. I am currently using the default crush rule (ie. keep each replica on a different host). My need is to add a “chassis” level, and keep replicas on a per-chassis level. From what I read in the documentation, I would have to edit the crush file manually, however this sounds kinda scary for a live cluster. Are there any “best known methods” to achieve that goal without messing things up? In my current scenario, I have one host per chassis, and planning on later adding nodes where there would be >1 hosts per chassis. It looks like “in theory” there wouldn’t be a need for any data movement after the crush map changes. Will reality match theory? Anything else I need to watch out for? Thank you! George _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io><mailto:ceph-users@ceph.io><mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io><mailto:ceph-users-leave@ceph.io><mailto:ceph-users-leave@ceph.io>
Hi George, yes, I believe your interpretation is correct. because the chassis buckets have new bucket IDs, the distribution hashing will change. I also believe that the trick to avoid data movement in your situation is, to export the new crush map, swap the IDs between corresponding host and bucket in *all* (!!!) occurrences and import. This is possible because currently you have the special case of a one-to-one correspondence between hosts and chassis. This would be the procedure Wido explained and there is no other choice for this edit. If you want to do that depends on how far you are into the data movement. If its almost done, I wouldn't bother. If its another month, it might be worth trying. As far as I can see, your crush map is going to be a short text file, so it should be feasible to edit. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Kyriazis, George <george.kyriazis@intel.com> Sent: 05 June 2020 01:36 To: Frank Schilder Cc: ceph-users Subject: Re: Best way to change bucket hierarchy Understand that it’s difficult to debug remotely. :-) In my current scenario I have 5 machines (1 host per chassis), but planning on adding some additional chassis with 4 hosts per chassis in the near future. Currently I am going through the first stage of adding “stub” chassis for the 5 hosts/chassis that I have, basically reparenting each host to its own chassis, as shown below: ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 203.72598 root default -5 40.01700 chassis chassis-hsw1 -9 40.01700 host vis-hsw-01 3 hdd 10.91299 osd.3 up 1.00000 1.00000 6 hdd 14.55199 osd.6 up 1.00000 1.00000 10 hdd 14.55199 osd.10 up 1.00000 1.00000 -6 40.01700 chassis chassis-hsw2 -13 40.01700 host vis-hsw-02 0 hdd 10.91299 osd.0 up 1.00000 1.00000 7 hdd 14.55199 osd.7 up 1.00000 1.00000 11 hdd 14.55199 osd.11 up 1.00000 1.00000 -7 40.01700 chassis chassis-hsw3 -11 40.01700 host vis-hsw-03 4 hdd 10.91299 osd.4 up 1.00000 1.00000 8 hdd 14.55199 osd.8 up 1.00000 1.00000 12 hdd 14.55199 osd.12 up 1.00000 1.00000 -8 40.01700 chassis chassis-hsw4 -3 40.01700 host vis-hsw-04 5 hdd 10.91299 osd.5 up 1.00000 1.00000 9 hdd 14.55199 osd.9 up 1.00000 1.00000 13 hdd 14.55199 osd.13 up 1.00000 1.00000 -17 43.65799 chassis chassis-hsw5 -15 43.65799 host vis-hsw-05 1 hdd 14.55299 osd.1 up 1.00000 1.00000 2 hdd 14.55299 osd.2 up 1.00000 1.00000 14 hdd 14.55299 osd.14 up 1.00000 1.00000 There is no additional constraint that is being added, so ideally there would be no data movement. However, I can imagine that the CRUSH algorithm could hash the PGs into different OSDs now because there is a new thing to consider (namely the chassis). Does it do that? Thanks, George On Jun 4, 2020, at 6:22 PM, Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk>> wrote: Its hard to tell without knowing what the diff is, but from your description I take it that you changed the failure domain for every(?) pool from host to chassis. I don't know what a chassis is in your architecture, but if each chassis contains several host buckets, then yes, I would expect almost every PG to be affected. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Kyriazis, George <george.kyriazis@intel.com<mailto:george.kyriazis@intel.com>> Sent: 05 June 2020 00:28:43 To: Frank Schilder Cc: ceph-users Subject: Re: Best way to change bucket hierarchy Hmm, So I tried all that, and I got almost all of my PGs being remapped. Crush map looks correct. Is that normal? Thanks, George On Jun 4, 2020, at 2:33 PM, Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk><mailto:frans@dtu.dk>> wrote: Hi George, you don't need to worry about that too much. The EC profile contains two types of information, one part about the actual EC encoding and another part about crush parameters. Unfortunately, actually. Part of this information is mutable after pool creation while the rest is not. Mutable here means outside of the profile. You can change the failure domain in the crush map without issues, but the profile won't reflect that change. That's an inconsistency we currently have to live with and it would have been better to separate mutable data (like failure domain) from immutable data (like k and m) or provide a meaningful interface to maintain consistency of mutable information. In short, don't believe everything the EC profile tells you. Some information might be out of date, like the failure domain or the device class (basically everything starting with crush-). If you remember that, you are out of trouble. Always dump the crush rule of an EC pool explicitly to see the true parameters in action. Having said that, to change the failure domain for an EC pool, change the crush rule for the EC profile - I did this too and it works just fine. The crush rule has by default the same name as the pool. I'm afraid, here you will have to do a manual edit of the crush rule as Wido explained. There is no other way - at least currently not. You can ask in this list for confirmation that your change is doing what you want. Do not try to touch an EC profile, they are read-only any ways. The crush parameters are only used at pool creation and never looked at again. You can override these by editing the crush rule as explained above. Best regards and good luck, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Kyriazis, George <george.kyriazis@intel.com<mailto:george.kyriazis@intel.com><mailto:george.kyriazis@intel.com>> Sent: 04 June 2020 20:56:38 To: Frank Schilder Cc: ceph-users Subject: Re: Best way to change bucket hierarchy Thanks Frank, Interesting info about the EC profile. I do have an EC pool, but I noticed the following when I dumped the profile: # ceph osd erasure-code-profile get ec22 crush-device-class=hdd crush-failure-domain=host crush-root=default jerasure-per-chunk-alignment=false k=2 m=2 plugin=jerasure technique=reed_sol_van w=8 # Which says that the failure domain of the EC profile is also set to host. Looks like I need to change the EC profile, too, but since it associated with the pool, maybe I can’t do that after pool creation? Or…. Since it the property is named “crush-failure-domain”, it’s automatically inherited from the crush profile, so I don’t have to do anything? Thanks, George On Jun 4, 2020, at 1:51 AM, Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk><mailto:frans@dtu.dk><mailto:frans@dtu.dk>> wrote: Hi George, for replicated rules you can simply create a new crush rule with the new failure domain set to chassis and change any pool's crush rule to this new one. If you have EC pools, then the chooseleaf needs to be edited by hand. I did this before as well. (A really unfortunate side effect is, that the EC profile attached to the pool goes out of sync with the crush map and there is nothing one can do about that. This is annoying yet harmless.) The intend of doing these changes while norebalance is set is - to avoid unnecessary data movement due to successive changes happening step by step and - to make sure peering is successful before starting to move data. I believe OSDs peer a bit faster with norebalance set and there is then a shorter interrupt to ongoing I/O (no I/O happens to a PG during peering). Yes, if you safe the old crush map, you can undo everything. It is a good idea to have a backup also just for reference and to compare before and after. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Kyriazis, George <george.kyriazis@intel.com<mailto:george.kyriazis@intel.com><mailto:george.kyriazis@intel.com><mailto:george.kyriazis@intel.com>> Sent: 04 June 2020 00:58:20 To: Frank Schilder Cc: ceph-users Subject: Re: Best way to change bucket hierarchy Thanks Frank, I don’t have too much experience editing crush rules, but I assume the chooseleaf step would also have to change to: step chooseleaf firstn 0 type chassis Correct? Is that the only other change that is needed? It looks like the rule change can happen both inside and outside the “norebalance” setting (again with CLI commands), but is it safer to do it inside (ie. while not rebalancing)? If I keep a backup of the crush rule map (with “ceph osd getcrushmap”), I assume I can restore the old map if something goes bad? Thanks again! George On Jun 3, 2020, at 5:24 PM, Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk><mailto:frans@dtu.dk><mailto:frans@dtu.dk>> wrote: You can use the command-line without editing the crush map. Look at the documentation of commands like ceph osd crush add-bucket ... ceph osd crush move ... Before starting this, set "ceph osd set norebalance" and unset after you are happy with the crush tree. Let everything peer. You should see misplaced objects and remapped PGs, but no degraded objects or PGs. Do this only when cluster is helth_ok, otherwise things can get really complicated. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Kyriazis, George <george.kyriazis@intel.com<mailto:george.kyriazis@intel.com><mailto:george.kyriazis@intel.com><mailto:george.kyriazis@intel.com>> Sent: 03 June 2020 22:45:11 To: ceph-users Subject: [ceph-users] Best way to change bucket hierarchy Helo, I have a live ceph cluster, and I’m in the need of modifying the bucket hierarchy. I am currently using the default crush rule (ie. keep each replica on a different host). My need is to add a “chassis” level, and keep replicas on a per-chassis level. From what I read in the documentation, I would have to edit the crush file manually, however this sounds kinda scary for a live cluster. Are there any “best known methods” to achieve that goal without messing things up? In my current scenario, I have one host per chassis, and planning on later adding nodes where there would be >1 hosts per chassis. It looks like “in theory” there wouldn’t be a need for any data movement after the crush map changes. Will reality match theory? Anything else I need to watch out for? Thank you! George _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io><mailto:ceph-users@ceph.io><mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io><mailto:ceph-users-leave@ceph.io><mailto:ceph-users-leave@ceph.io>
Hmm, Sounds quite dangerous. On the other hand, and from prior experience, it could take weeks/months for the cluster to rebalance, so I give it a try. From the looks of it, there is no other reference to IDs, is that correct? Just swap IDs between chassis and host and I should be OK? (Sorry, I’m not following the list closely, so I am not aware of Wido’s procedure). Thanks, George
On Jun 5, 2020, at 1:29 AM, Frank Schilder <frans@dtu.dk> wrote:
Hi George,
yes, I believe your interpretation is correct. because the chassis buckets have new bucket IDs, the distribution hashing will change. I also believe that the trick to avoid data movement in your situation is, to export the new crush map, swap the IDs between corresponding host and bucket in *all* (!!!) occurrences and import. This is possible because currently you have the special case of a one-to-one correspondence between hosts and chassis.
This would be the procedure Wido explained and there is no other choice for this edit.
If you want to do that depends on how far you are into the data movement. If its almost done, I wouldn't bother. If its another month, it might be worth trying. As far as I can see, your crush map is going to be a short text file, so it should be feasible to edit.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Kyriazis, George <george.kyriazis@intel.com> Sent: 05 June 2020 01:36 To: Frank Schilder Cc: ceph-users Subject: Re: Best way to change bucket hierarchy
Understand that it’s difficult to debug remotely. :-)
In my current scenario I have 5 machines (1 host per chassis), but planning on adding some additional chassis with 4 hosts per chassis in the near future. Currently I am going through the first stage of adding “stub” chassis for the 5 hosts/chassis that I have, basically reparenting each host to its own chassis, as shown below:
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 203.72598 root default -5 40.01700 chassis chassis-hsw1 -9 40.01700 host vis-hsw-01 3 hdd 10.91299 osd.3 up 1.00000 1.00000 6 hdd 14.55199 osd.6 up 1.00000 1.00000 10 hdd 14.55199 osd.10 up 1.00000 1.00000 -6 40.01700 chassis chassis-hsw2 -13 40.01700 host vis-hsw-02 0 hdd 10.91299 osd.0 up 1.00000 1.00000 7 hdd 14.55199 osd.7 up 1.00000 1.00000 11 hdd 14.55199 osd.11 up 1.00000 1.00000 -7 40.01700 chassis chassis-hsw3 -11 40.01700 host vis-hsw-03 4 hdd 10.91299 osd.4 up 1.00000 1.00000 8 hdd 14.55199 osd.8 up 1.00000 1.00000 12 hdd 14.55199 osd.12 up 1.00000 1.00000 -8 40.01700 chassis chassis-hsw4 -3 40.01700 host vis-hsw-04 5 hdd 10.91299 osd.5 up 1.00000 1.00000 9 hdd 14.55199 osd.9 up 1.00000 1.00000 13 hdd 14.55199 osd.13 up 1.00000 1.00000 -17 43.65799 chassis chassis-hsw5 -15 43.65799 host vis-hsw-05 1 hdd 14.55299 osd.1 up 1.00000 1.00000 2 hdd 14.55299 osd.2 up 1.00000 1.00000 14 hdd 14.55299 osd.14 up 1.00000 1.00000
There is no additional constraint that is being added, so ideally there would be no data movement. However, I can imagine that the CRUSH algorithm could hash the PGs into different OSDs now because there is a new thing to consider (namely the chassis). Does it do that?
Thanks,
George
On Jun 4, 2020, at 6:22 PM, Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk>> wrote:
Its hard to tell without knowing what the diff is, but from your description I take it that you changed the failure domain for every(?) pool from host to chassis. I don't know what a chassis is in your architecture, but if each chassis contains several host buckets, then yes, I would expect almost every PG to be affected.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Kyriazis, George <george.kyriazis@intel.com<mailto:george.kyriazis@intel.com>> Sent: 05 June 2020 00:28:43 To: Frank Schilder Cc: ceph-users Subject: Re: Best way to change bucket hierarchy
Hmm,
So I tried all that, and I got almost all of my PGs being remapped. Crush map looks correct. Is that normal?
Thanks,
George
On Jun 4, 2020, at 2:33 PM, Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk><mailto:frans@dtu.dk>> wrote:
Hi George,
you don't need to worry about that too much. The EC profile contains two types of information, one part about the actual EC encoding and another part about crush parameters. Unfortunately, actually. Part of this information is mutable after pool creation while the rest is not. Mutable here means outside of the profile. You can change the failure domain in the crush map without issues, but the profile won't reflect that change. That's an inconsistency we currently have to live with and it would have been better to separate mutable data (like failure domain) from immutable data (like k and m) or provide a meaningful interface to maintain consistency of mutable information.
In short, don't believe everything the EC profile tells you. Some information might be out of date, like the failure domain or the device class (basically everything starting with crush-). If you remember that, you are out of trouble. Always dump the crush rule of an EC pool explicitly to see the true parameters in action.
Having said that, to change the failure domain for an EC pool, change the crush rule for the EC profile - I did this too and it works just fine. The crush rule has by default the same name as the pool. I'm afraid, here you will have to do a manual edit of the crush rule as Wido explained. There is no other way - at least currently not.
You can ask in this list for confirmation that your change is doing what you want.
Do not try to touch an EC profile, they are read-only any ways. The crush parameters are only used at pool creation and never looked at again. You can override these by editing the crush rule as explained above.
Best regards and good luck, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Kyriazis, George <george.kyriazis@intel.com<mailto:george.kyriazis@intel.com><mailto:george.kyriazis@intel.com>> Sent: 04 June 2020 20:56:38 To: Frank Schilder Cc: ceph-users Subject: Re: Best way to change bucket hierarchy
Thanks Frank,
Interesting info about the EC profile. I do have an EC pool, but I noticed the following when I dumped the profile:
# ceph osd erasure-code-profile get ec22 crush-device-class=hdd crush-failure-domain=host crush-root=default jerasure-per-chunk-alignment=false k=2 m=2 plugin=jerasure technique=reed_sol_van w=8 #
Which says that the failure domain of the EC profile is also set to host. Looks like I need to change the EC profile, too, but since it associated with the pool, maybe I can’t do that after pool creation? Or…. Since it the property is named “crush-failure-domain”, it’s automatically inherited from the crush profile, so I don’t have to do anything?
Thanks,
George
On Jun 4, 2020, at 1:51 AM, Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk><mailto:frans@dtu.dk><mailto:frans@dtu.dk>> wrote:
Hi George,
for replicated rules you can simply create a new crush rule with the new failure domain set to chassis and change any pool's crush rule to this new one. If you have EC pools, then the chooseleaf needs to be edited by hand. I did this before as well. (A really unfortunate side effect is, that the EC profile attached to the pool goes out of sync with the crush map and there is nothing one can do about that. This is annoying yet harmless.)
The intend of doing these changes while norebalance is set is
- to avoid unnecessary data movement due to successive changes happening step by step and - to make sure peering is successful before starting to move data.
I believe OSDs peer a bit faster with norebalance set and there is then a shorter interrupt to ongoing I/O (no I/O happens to a PG during peering).
Yes, if you safe the old crush map, you can undo everything. It is a good idea to have a backup also just for reference and to compare before and after.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Kyriazis, George <george.kyriazis@intel.com<mailto:george.kyriazis@intel.com><mailto:george.kyriazis@intel.com><mailto:george.kyriazis@intel.com>> Sent: 04 June 2020 00:58:20 To: Frank Schilder Cc: ceph-users Subject: Re: Best way to change bucket hierarchy
Thanks Frank,
I don’t have too much experience editing crush rules, but I assume the chooseleaf step would also have to change to:
step chooseleaf firstn 0 type chassis
Correct? Is that the only other change that is needed? It looks like the rule change can happen both inside and outside the “norebalance” setting (again with CLI commands), but is it safer to do it inside (ie. while not rebalancing)?
If I keep a backup of the crush rule map (with “ceph osd getcrushmap”), I assume I can restore the old map if something goes bad?
Thanks again!
George
On Jun 3, 2020, at 5:24 PM, Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk><mailto:frans@dtu.dk><mailto:frans@dtu.dk>> wrote:
You can use the command-line without editing the crush map. Look at the documentation of commands like
ceph osd crush add-bucket ... ceph osd crush move ...
Before starting this, set "ceph osd set norebalance" and unset after you are happy with the crush tree. Let everything peer. You should see misplaced objects and remapped PGs, but no degraded objects or PGs.
Do this only when cluster is helth_ok, otherwise things can get really complicated.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Kyriazis, George <george.kyriazis@intel.com<mailto:george.kyriazis@intel.com><mailto:george.kyriazis@intel.com><mailto:george.kyriazis@intel.com>> Sent: 03 June 2020 22:45:11 To: ceph-users Subject: [ceph-users] Best way to change bucket hierarchy
Helo,
I have a live ceph cluster, and I’m in the need of modifying the bucket hierarchy. I am currently using the default crush rule (ie. keep each replica on a different host). My need is to add a “chassis” level, and keep replicas on a per-chassis level.
From what I read in the documentation, I would have to edit the crush file manually, however this sounds kinda scary for a live cluster.
Are there any “best known methods” to achieve that goal without messing things up?
In my current scenario, I have one host per chassis, and planning on later adding nodes where there would be >1 hosts per chassis. It looks like “in theory” there wouldn’t be a need for any data movement after the crush map changes. Will reality match theory? Anything else I need to watch out for?
Thank you!
George
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io><mailto:ceph-users@ceph.io><mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io><mailto:ceph-users-leave@ceph.io><mailto:ceph-users-leave@ceph.io>
Wido replied to you, check this thread. You really need to understand the file you get exactly. The IDs are used to refer to items from within other items. You need to make sure that any such cross-reference is updated as well. It is not just changing the ID tag in a bucket item, you also need to update all places that refer to a bucket by ID. The crush map defines a tree structure and a wrong reference can get you into serious trouble. Before attempting anything like this, make sure you have a backup of the original crush map (in several places). Generally speaking, your tweaking of the crush map is maybe a bi premature. You wrote you want to add quite a number of servers. Why don't you do the crush map change together with that? All the data will be reshuffled then any ways. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Kyriazis, George <george.kyriazis@intel.com> Sent: 05 June 2020 17:21 To: Frank Schilder Cc: ceph-users; Wido den Hollander Subject: Re: Best way to change bucket hierarchy Hmm, Sounds quite dangerous. On the other hand, and from prior experience, it could take weeks/months for the cluster to rebalance, so I give it a try. From the looks of it, there is no other reference to IDs, is that correct? Just swap IDs between chassis and host and I should be OK? (Sorry, I’m not following the list closely, so I am not aware of Wido’s procedure). Thanks, George
On Jun 5, 2020, at 1:29 AM, Frank Schilder <frans@dtu.dk> wrote:
Hi George,
yes, I believe your interpretation is correct. because the chassis buckets have new bucket IDs, the distribution hashing will change. I also believe that the trick to avoid data movement in your situation is, to export the new crush map, swap the IDs between corresponding host and bucket in *all* (!!!) occurrences and import. This is possible because currently you have the special case of a one-to-one correspondence between hosts and chassis.
This would be the procedure Wido explained and there is no other choice for this edit.
If you want to do that depends on how far you are into the data movement. If its almost done, I wouldn't bother. If its another month, it might be worth trying. As far as I can see, your crush map is going to be a short text file, so it should be feasible to edit.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Kyriazis, George <george.kyriazis@intel.com> Sent: 05 June 2020 01:36 To: Frank Schilder Cc: ceph-users Subject: Re: Best way to change bucket hierarchy
Understand that it’s difficult to debug remotely. :-)
In my current scenario I have 5 machines (1 host per chassis), but planning on adding some additional chassis with 4 hosts per chassis in the near future. Currently I am going through the first stage of adding “stub” chassis for the 5 hosts/chassis that I have, basically reparenting each host to its own chassis, as shown below:
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 203.72598 root default -5 40.01700 chassis chassis-hsw1 -9 40.01700 host vis-hsw-01 3 hdd 10.91299 osd.3 up 1.00000 1.00000 6 hdd 14.55199 osd.6 up 1.00000 1.00000 10 hdd 14.55199 osd.10 up 1.00000 1.00000 -6 40.01700 chassis chassis-hsw2 -13 40.01700 host vis-hsw-02 0 hdd 10.91299 osd.0 up 1.00000 1.00000 7 hdd 14.55199 osd.7 up 1.00000 1.00000 11 hdd 14.55199 osd.11 up 1.00000 1.00000 -7 40.01700 chassis chassis-hsw3 -11 40.01700 host vis-hsw-03 4 hdd 10.91299 osd.4 up 1.00000 1.00000 8 hdd 14.55199 osd.8 up 1.00000 1.00000 12 hdd 14.55199 osd.12 up 1.00000 1.00000 -8 40.01700 chassis chassis-hsw4 -3 40.01700 host vis-hsw-04 5 hdd 10.91299 osd.5 up 1.00000 1.00000 9 hdd 14.55199 osd.9 up 1.00000 1.00000 13 hdd 14.55199 osd.13 up 1.00000 1.00000 -17 43.65799 chassis chassis-hsw5 -15 43.65799 host vis-hsw-05 1 hdd 14.55299 osd.1 up 1.00000 1.00000 2 hdd 14.55299 osd.2 up 1.00000 1.00000 14 hdd 14.55299 osd.14 up 1.00000 1.00000
There is no additional constraint that is being added, so ideally there would be no data movement. However, I can imagine that the CRUSH algorithm could hash the PGs into different OSDs now because there is a new thing to consider (namely the chassis). Does it do that?
Thanks,
George
On Jun 4, 2020, at 6:22 PM, Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk>> wrote:
Its hard to tell without knowing what the diff is, but from your description I take it that you changed the failure domain for every(?) pool from host to chassis. I don't know what a chassis is in your architecture, but if each chassis contains several host buckets, then yes, I would expect almost every PG to be affected.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Kyriazis, George <george.kyriazis@intel.com<mailto:george.kyriazis@intel.com>> Sent: 05 June 2020 00:28:43 To: Frank Schilder Cc: ceph-users Subject: Re: Best way to change bucket hierarchy
Hmm,
So I tried all that, and I got almost all of my PGs being remapped. Crush map looks correct. Is that normal?
Thanks,
George
On Jun 4, 2020, at 2:33 PM, Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk><mailto:frans@dtu.dk>> wrote:
Hi George,
you don't need to worry about that too much. The EC profile contains two types of information, one part about the actual EC encoding and another part about crush parameters. Unfortunately, actually. Part of this information is mutable after pool creation while the rest is not. Mutable here means outside of the profile. You can change the failure domain in the crush map without issues, but the profile won't reflect that change. That's an inconsistency we currently have to live with and it would have been better to separate mutable data (like failure domain) from immutable data (like k and m) or provide a meaningful interface to maintain consistency of mutable information.
In short, don't believe everything the EC profile tells you. Some information might be out of date, like the failure domain or the device class (basically everything starting with crush-). If you remember that, you are out of trouble. Always dump the crush rule of an EC pool explicitly to see the true parameters in action.
Having said that, to change the failure domain for an EC pool, change the crush rule for the EC profile - I did this too and it works just fine. The crush rule has by default the same name as the pool. I'm afraid, here you will have to do a manual edit of the crush rule as Wido explained. There is no other way - at least currently not.
You can ask in this list for confirmation that your change is doing what you want.
Do not try to touch an EC profile, they are read-only any ways. The crush parameters are only used at pool creation and never looked at again. You can override these by editing the crush rule as explained above.
Best regards and good luck, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Kyriazis, George <george.kyriazis@intel.com<mailto:george.kyriazis@intel.com><mailto:george.kyriazis@intel.com>> Sent: 04 June 2020 20:56:38 To: Frank Schilder Cc: ceph-users Subject: Re: Best way to change bucket hierarchy
Thanks Frank,
Interesting info about the EC profile. I do have an EC pool, but I noticed the following when I dumped the profile:
# ceph osd erasure-code-profile get ec22 crush-device-class=hdd crush-failure-domain=host crush-root=default jerasure-per-chunk-alignment=false k=2 m=2 plugin=jerasure technique=reed_sol_van w=8 #
Which says that the failure domain of the EC profile is also set to host. Looks like I need to change the EC profile, too, but since it associated with the pool, maybe I can’t do that after pool creation? Or…. Since it the property is named “crush-failure-domain”, it’s automatically inherited from the crush profile, so I don’t have to do anything?
Thanks,
George
On Jun 4, 2020, at 1:51 AM, Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk><mailto:frans@dtu.dk><mailto:frans@dtu.dk>> wrote:
Hi George,
for replicated rules you can simply create a new crush rule with the new failure domain set to chassis and change any pool's crush rule to this new one. If you have EC pools, then the chooseleaf needs to be edited by hand. I did this before as well. (A really unfortunate side effect is, that the EC profile attached to the pool goes out of sync with the crush map and there is nothing one can do about that. This is annoying yet harmless.)
The intend of doing these changes while norebalance is set is
- to avoid unnecessary data movement due to successive changes happening step by step and - to make sure peering is successful before starting to move data.
I believe OSDs peer a bit faster with norebalance set and there is then a shorter interrupt to ongoing I/O (no I/O happens to a PG during peering).
Yes, if you safe the old crush map, you can undo everything. It is a good idea to have a backup also just for reference and to compare before and after.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Kyriazis, George <george.kyriazis@intel.com<mailto:george.kyriazis@intel.com><mailto:george.kyriazis@intel.com><mailto:george.kyriazis@intel.com>> Sent: 04 June 2020 00:58:20 To: Frank Schilder Cc: ceph-users Subject: Re: Best way to change bucket hierarchy
Thanks Frank,
I don’t have too much experience editing crush rules, but I assume the chooseleaf step would also have to change to:
step chooseleaf firstn 0 type chassis
Correct? Is that the only other change that is needed? It looks like the rule change can happen both inside and outside the “norebalance” setting (again with CLI commands), but is it safer to do it inside (ie. while not rebalancing)?
If I keep a backup of the crush rule map (with “ceph osd getcrushmap”), I assume I can restore the old map if something goes bad?
Thanks again!
George
On Jun 3, 2020, at 5:24 PM, Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk><mailto:frans@dtu.dk><mailto:frans@dtu.dk>> wrote:
You can use the command-line without editing the crush map. Look at the documentation of commands like
ceph osd crush add-bucket ... ceph osd crush move ...
Before starting this, set "ceph osd set norebalance" and unset after you are happy with the crush tree. Let everything peer. You should see misplaced objects and remapped PGs, but no degraded objects or PGs.
Do this only when cluster is helth_ok, otherwise things can get really complicated.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Kyriazis, George <george.kyriazis@intel.com<mailto:george.kyriazis@intel.com><mailto:george.kyriazis@intel.com><mailto:george.kyriazis@intel.com>> Sent: 03 June 2020 22:45:11 To: ceph-users Subject: [ceph-users] Best way to change bucket hierarchy
Helo,
I have a live ceph cluster, and I’m in the need of modifying the bucket hierarchy. I am currently using the default crush rule (ie. keep each replica on a different host). My need is to add a “chassis” level, and keep replicas on a per-chassis level.
From what I read in the documentation, I would have to edit the crush file manually, however this sounds kinda scary for a live cluster.
Are there any “best known methods” to achieve that goal without messing things up?
In my current scenario, I have one host per chassis, and planning on later adding nodes where there would be >1 hosts per chassis. It looks like “in theory” there wouldn’t be a need for any data movement after the crush map changes. Will reality match theory? Anything else I need to watch out for?
Thank you!
George
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io><mailto:ceph-users@ceph.io><mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io><mailto:ceph-users-leave@ceph.io><mailto:ceph-users-leave@ceph.io>
Hmm, From what I see in the crush map, “nodes” refers to other “nodes” by name, not by ID. In fact, I don’t see anything in the crush map referred to by ID. As we said before, though, the crush algorithm figures out the hashes based on the IDs. I am not sure what else refers to them, though (outside the crush map) to make sure the references are correct. Thanks, George
On Jun 5, 2020, at 10:32 AM, Frank Schilder <frans@dtu.dk> wrote:
Wido replied to you, check this thread.
You really need to understand the file you get exactly. The IDs are used to refer to items from within other items. You need to make sure that any such cross-reference is updated as well. It is not just changing the ID tag in a bucket item, you also need to update all places that refer to a bucket by ID. The crush map defines a tree structure and a wrong reference can get you into serious trouble.
Before attempting anything like this, make sure you have a backup of the original crush map (in several places).
Generally speaking, your tweaking of the crush map is maybe a bi premature. You wrote you want to add quite a number of servers. Why don't you do the crush map change together with that? All the data will be reshuffled then any ways.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Kyriazis, George <george.kyriazis@intel.com> Sent: 05 June 2020 17:21 To: Frank Schilder Cc: ceph-users; Wido den Hollander Subject: Re: Best way to change bucket hierarchy
Hmm,
Sounds quite dangerous. On the other hand, and from prior experience, it could take weeks/months for the cluster to rebalance, so I give it a try.
From the looks of it, there is no other reference to IDs, is that correct? Just swap IDs between chassis and host and I should be OK? (Sorry, I’m not following the list closely, so I am not aware of Wido’s procedure).
Thanks,
George
On Jun 5, 2020, at 1:29 AM, Frank Schilder <frans@dtu.dk> wrote:
Hi George,
yes, I believe your interpretation is correct. because the chassis buckets have new bucket IDs, the distribution hashing will change. I also believe that the trick to avoid data movement in your situation is, to export the new crush map, swap the IDs between corresponding host and bucket in *all* (!!!) occurrences and import. This is possible because currently you have the special case of a one-to-one correspondence between hosts and chassis.
This would be the procedure Wido explained and there is no other choice for this edit.
If you want to do that depends on how far you are into the data movement. If its almost done, I wouldn't bother. If its another month, it might be worth trying. As far as I can see, your crush map is going to be a short text file, so it should be feasible to edit.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Kyriazis, George <george.kyriazis@intel.com> Sent: 05 June 2020 01:36 To: Frank Schilder Cc: ceph-users Subject: Re: Best way to change bucket hierarchy
Understand that it’s difficult to debug remotely. :-)
In my current scenario I have 5 machines (1 host per chassis), but planning on adding some additional chassis with 4 hosts per chassis in the near future. Currently I am going through the first stage of adding “stub” chassis for the 5 hosts/chassis that I have, basically reparenting each host to its own chassis, as shown below:
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 203.72598 root default -5 40.01700 chassis chassis-hsw1 -9 40.01700 host vis-hsw-01 3 hdd 10.91299 osd.3 up 1.00000 1.00000 6 hdd 14.55199 osd.6 up 1.00000 1.00000 10 hdd 14.55199 osd.10 up 1.00000 1.00000 -6 40.01700 chassis chassis-hsw2 -13 40.01700 host vis-hsw-02 0 hdd 10.91299 osd.0 up 1.00000 1.00000 7 hdd 14.55199 osd.7 up 1.00000 1.00000 11 hdd 14.55199 osd.11 up 1.00000 1.00000 -7 40.01700 chassis chassis-hsw3 -11 40.01700 host vis-hsw-03 4 hdd 10.91299 osd.4 up 1.00000 1.00000 8 hdd 14.55199 osd.8 up 1.00000 1.00000 12 hdd 14.55199 osd.12 up 1.00000 1.00000 -8 40.01700 chassis chassis-hsw4 -3 40.01700 host vis-hsw-04 5 hdd 10.91299 osd.5 up 1.00000 1.00000 9 hdd 14.55199 osd.9 up 1.00000 1.00000 13 hdd 14.55199 osd.13 up 1.00000 1.00000 -17 43.65799 chassis chassis-hsw5 -15 43.65799 host vis-hsw-05 1 hdd 14.55299 osd.1 up 1.00000 1.00000 2 hdd 14.55299 osd.2 up 1.00000 1.00000 14 hdd 14.55299 osd.14 up 1.00000 1.00000
There is no additional constraint that is being added, so ideally there would be no data movement. However, I can imagine that the CRUSH algorithm could hash the PGs into different OSDs now because there is a new thing to consider (namely the chassis). Does it do that?
Thanks,
George
On Jun 4, 2020, at 6:22 PM, Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk>> wrote:
Its hard to tell without knowing what the diff is, but from your description I take it that you changed the failure domain for every(?) pool from host to chassis. I don't know what a chassis is in your architecture, but if each chassis contains several host buckets, then yes, I would expect almost every PG to be affected.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Kyriazis, George <george.kyriazis@intel.com<mailto:george.kyriazis@intel.com>> Sent: 05 June 2020 00:28:43 To: Frank Schilder Cc: ceph-users Subject: Re: Best way to change bucket hierarchy
Hmm,
So I tried all that, and I got almost all of my PGs being remapped. Crush map looks correct. Is that normal?
Thanks,
George
On Jun 4, 2020, at 2:33 PM, Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk><mailto:frans@dtu.dk>> wrote:
Hi George,
you don't need to worry about that too much. The EC profile contains two types of information, one part about the actual EC encoding and another part about crush parameters. Unfortunately, actually. Part of this information is mutable after pool creation while the rest is not. Mutable here means outside of the profile. You can change the failure domain in the crush map without issues, but the profile won't reflect that change. That's an inconsistency we currently have to live with and it would have been better to separate mutable data (like failure domain) from immutable data (like k and m) or provide a meaningful interface to maintain consistency of mutable information.
In short, don't believe everything the EC profile tells you. Some information might be out of date, like the failure domain or the device class (basically everything starting with crush-). If you remember that, you are out of trouble. Always dump the crush rule of an EC pool explicitly to see the true parameters in action.
Having said that, to change the failure domain for an EC pool, change the crush rule for the EC profile - I did this too and it works just fine. The crush rule has by default the same name as the pool. I'm afraid, here you will have to do a manual edit of the crush rule as Wido explained. There is no other way - at least currently not.
You can ask in this list for confirmation that your change is doing what you want.
Do not try to touch an EC profile, they are read-only any ways. The crush parameters are only used at pool creation and never looked at again. You can override these by editing the crush rule as explained above.
Best regards and good luck, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Kyriazis, George <george.kyriazis@intel.com<mailto:george.kyriazis@intel.com><mailto:george.kyriazis@intel.com>> Sent: 04 June 2020 20:56:38 To: Frank Schilder Cc: ceph-users Subject: Re: Best way to change bucket hierarchy
Thanks Frank,
Interesting info about the EC profile. I do have an EC pool, but I noticed the following when I dumped the profile:
# ceph osd erasure-code-profile get ec22 crush-device-class=hdd crush-failure-domain=host crush-root=default jerasure-per-chunk-alignment=false k=2 m=2 plugin=jerasure technique=reed_sol_van w=8 #
Which says that the failure domain of the EC profile is also set to host. Looks like I need to change the EC profile, too, but since it associated with the pool, maybe I can’t do that after pool creation? Or…. Since it the property is named “crush-failure-domain”, it’s automatically inherited from the crush profile, so I don’t have to do anything?
Thanks,
George
On Jun 4, 2020, at 1:51 AM, Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk><mailto:frans@dtu.dk><mailto:frans@dtu.dk>> wrote:
Hi George,
for replicated rules you can simply create a new crush rule with the new failure domain set to chassis and change any pool's crush rule to this new one. If you have EC pools, then the chooseleaf needs to be edited by hand. I did this before as well. (A really unfortunate side effect is, that the EC profile attached to the pool goes out of sync with the crush map and there is nothing one can do about that. This is annoying yet harmless.)
The intend of doing these changes while norebalance is set is
- to avoid unnecessary data movement due to successive changes happening step by step and - to make sure peering is successful before starting to move data.
I believe OSDs peer a bit faster with norebalance set and there is then a shorter interrupt to ongoing I/O (no I/O happens to a PG during peering).
Yes, if you safe the old crush map, you can undo everything. It is a good idea to have a backup also just for reference and to compare before and after.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Kyriazis, George <george.kyriazis@intel.com<mailto:george.kyriazis@intel.com><mailto:george.kyriazis@intel.com><mailto:george.kyriazis@intel.com>> Sent: 04 June 2020 00:58:20 To: Frank Schilder Cc: ceph-users Subject: Re: Best way to change bucket hierarchy
Thanks Frank,
I don’t have too much experience editing crush rules, but I assume the chooseleaf step would also have to change to:
step chooseleaf firstn 0 type chassis
Correct? Is that the only other change that is needed? It looks like the rule change can happen both inside and outside the “norebalance” setting (again with CLI commands), but is it safer to do it inside (ie. while not rebalancing)?
If I keep a backup of the crush rule map (with “ceph osd getcrushmap”), I assume I can restore the old map if something goes bad?
Thanks again!
George
On Jun 3, 2020, at 5:24 PM, Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk><mailto:frans@dtu.dk><mailto:frans@dtu.dk>> wrote:
You can use the command-line without editing the crush map. Look at the documentation of commands like
ceph osd crush add-bucket ... ceph osd crush move ...
Before starting this, set "ceph osd set norebalance" and unset after you are happy with the crush tree. Let everything peer. You should see misplaced objects and remapped PGs, but no degraded objects or PGs.
Do this only when cluster is helth_ok, otherwise things can get really complicated.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Kyriazis, George <george.kyriazis@intel.com<mailto:george.kyriazis@intel.com><mailto:george.kyriazis@intel.com><mailto:george.kyriazis@intel.com>> Sent: 03 June 2020 22:45:11 To: ceph-users Subject: [ceph-users] Best way to change bucket hierarchy
Helo,
I have a live ceph cluster, and I’m in the need of modifying the bucket hierarchy. I am currently using the default crush rule (ie. keep each replica on a different host). My need is to add a “chassis” level, and keep replicas on a per-chassis level.
From what I read in the documentation, I would have to edit the crush file manually, however this sounds kinda scary for a live cluster.
Are there any “best known methods” to achieve that goal without messing things up?
In my current scenario, I have one host per chassis, and planning on later adding nodes where there would be >1 hosts per chassis. It looks like “in theory” there wouldn’t be a need for any data movement after the crush map changes. Will reality match theory? Anything else I need to watch out for?
Thank you!
George
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io><mailto:ceph-users@ceph.io><mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io><mailto:ceph-users-leave@ceph.io><mailto:ceph-users-leave@ceph.io>
I never changed IDs before, I'm just extra cautious. If they do not show up explicitly anywhere else than inside the bucket definitions, then it is probably an easy edit and just swapping them. If you try this, could you please report back to the list if it works as expected, maybe with example crush maps/items included to illustrate the edits for documentation purposes? Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Kyriazis, George <george.kyriazis@intel.com> Sent: 05 June 2020 17:46 To: Frank Schilder Cc: ceph-users; Wido den Hollander Subject: Re: Best way to change bucket hierarchy Hmm, From what I see in the crush map, “nodes” refers to other “nodes” by name, not by ID. In fact, I don’t see anything in the crush map referred to by ID. As we said before, though, the crush algorithm figures out the hashes based on the IDs. I am not sure what else refers to them, though (outside the crush map) to make sure the references are correct. Thanks, George
On Jun 5, 2020, at 10:32 AM, Frank Schilder <frans@dtu.dk> wrote:
Wido replied to you, check this thread.
You really need to understand the file you get exactly. The IDs are used to refer to items from within other items. You need to make sure that any such cross-reference is updated as well. It is not just changing the ID tag in a bucket item, you also need to update all places that refer to a bucket by ID. The crush map defines a tree structure and a wrong reference can get you into serious trouble.
Before attempting anything like this, make sure you have a backup of the original crush map (in several places).
Generally speaking, your tweaking of the crush map is maybe a bi premature. You wrote you want to add quite a number of servers. Why don't you do the crush map change together with that? All the data will be reshuffled then any ways.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Kyriazis, George <george.kyriazis@intel.com> Sent: 05 June 2020 17:21 To: Frank Schilder Cc: ceph-users; Wido den Hollander Subject: Re: Best way to change bucket hierarchy
Hmm,
Sounds quite dangerous. On the other hand, and from prior experience, it could take weeks/months for the cluster to rebalance, so I give it a try.
From the looks of it, there is no other reference to IDs, is that correct? Just swap IDs between chassis and host and I should be OK? (Sorry, I’m not following the list closely, so I am not aware of Wido’s procedure).
Thanks,
George
On Jun 5, 2020, at 1:29 AM, Frank Schilder <frans@dtu.dk> wrote:
Hi George,
yes, I believe your interpretation is correct. because the chassis buckets have new bucket IDs, the distribution hashing will change. I also believe that the trick to avoid data movement in your situation is, to export the new crush map, swap the IDs between corresponding host and bucket in *all* (!!!) occurrences and import. This is possible because currently you have the special case of a one-to-one correspondence between hosts and chassis.
This would be the procedure Wido explained and there is no other choice for this edit.
If you want to do that depends on how far you are into the data movement. If its almost done, I wouldn't bother. If its another month, it might be worth trying. As far as I can see, your crush map is going to be a short text file, so it should be feasible to edit.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Kyriazis, George <george.kyriazis@intel.com> Sent: 05 June 2020 01:36 To: Frank Schilder Cc: ceph-users Subject: Re: Best way to change bucket hierarchy
Understand that it’s difficult to debug remotely. :-)
In my current scenario I have 5 machines (1 host per chassis), but planning on adding some additional chassis with 4 hosts per chassis in the near future. Currently I am going through the first stage of adding “stub” chassis for the 5 hosts/chassis that I have, basically reparenting each host to its own chassis, as shown below:
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 203.72598 root default -5 40.01700 chassis chassis-hsw1 -9 40.01700 host vis-hsw-01 3 hdd 10.91299 osd.3 up 1.00000 1.00000 6 hdd 14.55199 osd.6 up 1.00000 1.00000 10 hdd 14.55199 osd.10 up 1.00000 1.00000 -6 40.01700 chassis chassis-hsw2 -13 40.01700 host vis-hsw-02 0 hdd 10.91299 osd.0 up 1.00000 1.00000 7 hdd 14.55199 osd.7 up 1.00000 1.00000 11 hdd 14.55199 osd.11 up 1.00000 1.00000 -7 40.01700 chassis chassis-hsw3 -11 40.01700 host vis-hsw-03 4 hdd 10.91299 osd.4 up 1.00000 1.00000 8 hdd 14.55199 osd.8 up 1.00000 1.00000 12 hdd 14.55199 osd.12 up 1.00000 1.00000 -8 40.01700 chassis chassis-hsw4 -3 40.01700 host vis-hsw-04 5 hdd 10.91299 osd.5 up 1.00000 1.00000 9 hdd 14.55199 osd.9 up 1.00000 1.00000 13 hdd 14.55199 osd.13 up 1.00000 1.00000 -17 43.65799 chassis chassis-hsw5 -15 43.65799 host vis-hsw-05 1 hdd 14.55299 osd.1 up 1.00000 1.00000 2 hdd 14.55299 osd.2 up 1.00000 1.00000 14 hdd 14.55299 osd.14 up 1.00000 1.00000
There is no additional constraint that is being added, so ideally there would be no data movement. However, I can imagine that the CRUSH algorithm could hash the PGs into different OSDs now because there is a new thing to consider (namely the chassis). Does it do that?
Thanks,
George
On Jun 4, 2020, at 6:22 PM, Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk>> wrote:
Its hard to tell without knowing what the diff is, but from your description I take it that you changed the failure domain for every(?) pool from host to chassis. I don't know what a chassis is in your architecture, but if each chassis contains several host buckets, then yes, I would expect almost every PG to be affected.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Kyriazis, George <george.kyriazis@intel.com<mailto:george.kyriazis@intel.com>> Sent: 05 June 2020 00:28:43 To: Frank Schilder Cc: ceph-users Subject: Re: Best way to change bucket hierarchy
Hmm,
So I tried all that, and I got almost all of my PGs being remapped. Crush map looks correct. Is that normal?
Thanks,
George
On Jun 4, 2020, at 2:33 PM, Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk><mailto:frans@dtu.dk>> wrote:
Hi George,
you don't need to worry about that too much. The EC profile contains two types of information, one part about the actual EC encoding and another part about crush parameters. Unfortunately, actually. Part of this information is mutable after pool creation while the rest is not. Mutable here means outside of the profile. You can change the failure domain in the crush map without issues, but the profile won't reflect that change. That's an inconsistency we currently have to live with and it would have been better to separate mutable data (like failure domain) from immutable data (like k and m) or provide a meaningful interface to maintain consistency of mutable information.
In short, don't believe everything the EC profile tells you. Some information might be out of date, like the failure domain or the device class (basically everything starting with crush-). If you remember that, you are out of trouble. Always dump the crush rule of an EC pool explicitly to see the true parameters in action.
Having said that, to change the failure domain for an EC pool, change the crush rule for the EC profile - I did this too and it works just fine. The crush rule has by default the same name as the pool. I'm afraid, here you will have to do a manual edit of the crush rule as Wido explained. There is no other way - at least currently not.
You can ask in this list for confirmation that your change is doing what you want.
Do not try to touch an EC profile, they are read-only any ways. The crush parameters are only used at pool creation and never looked at again. You can override these by editing the crush rule as explained above.
Best regards and good luck, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Kyriazis, George <george.kyriazis@intel.com<mailto:george.kyriazis@intel.com><mailto:george.kyriazis@intel.com>> Sent: 04 June 2020 20:56:38 To: Frank Schilder Cc: ceph-users Subject: Re: Best way to change bucket hierarchy
Thanks Frank,
Interesting info about the EC profile. I do have an EC pool, but I noticed the following when I dumped the profile:
# ceph osd erasure-code-profile get ec22 crush-device-class=hdd crush-failure-domain=host crush-root=default jerasure-per-chunk-alignment=false k=2 m=2 plugin=jerasure technique=reed_sol_van w=8 #
Which says that the failure domain of the EC profile is also set to host. Looks like I need to change the EC profile, too, but since it associated with the pool, maybe I can’t do that after pool creation? Or…. Since it the property is named “crush-failure-domain”, it’s automatically inherited from the crush profile, so I don’t have to do anything?
Thanks,
George
On Jun 4, 2020, at 1:51 AM, Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk><mailto:frans@dtu.dk><mailto:frans@dtu.dk>> wrote:
Hi George,
for replicated rules you can simply create a new crush rule with the new failure domain set to chassis and change any pool's crush rule to this new one. If you have EC pools, then the chooseleaf needs to be edited by hand. I did this before as well. (A really unfortunate side effect is, that the EC profile attached to the pool goes out of sync with the crush map and there is nothing one can do about that. This is annoying yet harmless.)
The intend of doing these changes while norebalance is set is
- to avoid unnecessary data movement due to successive changes happening step by step and - to make sure peering is successful before starting to move data.
I believe OSDs peer a bit faster with norebalance set and there is then a shorter interrupt to ongoing I/O (no I/O happens to a PG during peering).
Yes, if you safe the old crush map, you can undo everything. It is a good idea to have a backup also just for reference and to compare before and after.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Kyriazis, George <george.kyriazis@intel.com<mailto:george.kyriazis@intel.com><mailto:george.kyriazis@intel.com><mailto:george.kyriazis@intel.com>> Sent: 04 June 2020 00:58:20 To: Frank Schilder Cc: ceph-users Subject: Re: Best way to change bucket hierarchy
Thanks Frank,
I don’t have too much experience editing crush rules, but I assume the chooseleaf step would also have to change to:
step chooseleaf firstn 0 type chassis
Correct? Is that the only other change that is needed? It looks like the rule change can happen both inside and outside the “norebalance” setting (again with CLI commands), but is it safer to do it inside (ie. while not rebalancing)?
If I keep a backup of the crush rule map (with “ceph osd getcrushmap”), I assume I can restore the old map if something goes bad?
Thanks again!
George
On Jun 3, 2020, at 5:24 PM, Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk><mailto:frans@dtu.dk><mailto:frans@dtu.dk>> wrote:
You can use the command-line without editing the crush map. Look at the documentation of commands like
ceph osd crush add-bucket ... ceph osd crush move ...
Before starting this, set "ceph osd set norebalance" and unset after you are happy with the crush tree. Let everything peer. You should see misplaced objects and remapped PGs, but no degraded objects or PGs.
Do this only when cluster is helth_ok, otherwise things can get really complicated.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Kyriazis, George <george.kyriazis@intel.com<mailto:george.kyriazis@intel.com><mailto:george.kyriazis@intel.com><mailto:george.kyriazis@intel.com>> Sent: 03 June 2020 22:45:11 To: ceph-users Subject: [ceph-users] Best way to change bucket hierarchy
Helo,
I have a live ceph cluster, and I’m in the need of modifying the bucket hierarchy. I am currently using the default crush rule (ie. keep each replica on a different host). My need is to add a “chassis” level, and keep replicas on a per-chassis level.
From what I read in the documentation, I would have to edit the crush file manually, however this sounds kinda scary for a live cluster.
Are there any “best known methods” to achieve that goal without messing things up?
In my current scenario, I have one host per chassis, and planning on later adding nodes where there would be >1 hosts per chassis. It looks like “in theory” there wouldn’t be a need for any data movement after the crush map changes. Will reality match theory? Anything else I need to watch out for?
Thank you!
George
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io><mailto:ceph-users@ceph.io><mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io><mailto:ceph-users-leave@ceph.io><mailto:ceph-users-leave@ceph.io>
I’m hesitant to do this, too. I think I’ll pass and just wait for the remapping. :-) George
On Jun 5, 2020, at 12:58 PM, Frank Schilder <frans@dtu.dk> wrote:
I never changed IDs before, I'm just extra cautious. If they do not show up explicitly anywhere else than inside the bucket definitions, then it is probably an easy edit and just swapping them.
If you try this, could you please report back to the list if it works as expected, maybe with example crush maps/items included to illustrate the edits for documentation purposes?
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Kyriazis, George <george.kyriazis@intel.com> Sent: 05 June 2020 17:46 To: Frank Schilder Cc: ceph-users; Wido den Hollander Subject: Re: Best way to change bucket hierarchy
Hmm,
From what I see in the crush map, “nodes” refers to other “nodes” by name, not by ID. In fact, I don’t see anything in the crush map referred to by ID. As we said before, though, the crush algorithm figures out the hashes based on the IDs. I am not sure what else refers to them, though (outside the crush map) to make sure the references are correct.
Thanks,
George
On Jun 5, 2020, at 10:32 AM, Frank Schilder <frans@dtu.dk> wrote:
Wido replied to you, check this thread.
You really need to understand the file you get exactly. The IDs are used to refer to items from within other items. You need to make sure that any such cross-reference is updated as well. It is not just changing the ID tag in a bucket item, you also need to update all places that refer to a bucket by ID. The crush map defines a tree structure and a wrong reference can get you into serious trouble.
Before attempting anything like this, make sure you have a backup of the original crush map (in several places).
Generally speaking, your tweaking of the crush map is maybe a bi premature. You wrote you want to add quite a number of servers. Why don't you do the crush map change together with that? All the data will be reshuffled then any ways.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Kyriazis, George <george.kyriazis@intel.com> Sent: 05 June 2020 17:21 To: Frank Schilder Cc: ceph-users; Wido den Hollander Subject: Re: Best way to change bucket hierarchy
Hmm,
Sounds quite dangerous. On the other hand, and from prior experience, it could take weeks/months for the cluster to rebalance, so I give it a try.
From the looks of it, there is no other reference to IDs, is that correct? Just swap IDs between chassis and host and I should be OK? (Sorry, I’m not following the list closely, so I am not aware of Wido’s procedure).
Thanks,
George
On Jun 5, 2020, at 1:29 AM, Frank Schilder <frans@dtu.dk> wrote:
Hi George,
yes, I believe your interpretation is correct. because the chassis buckets have new bucket IDs, the distribution hashing will change. I also believe that the trick to avoid data movement in your situation is, to export the new crush map, swap the IDs between corresponding host and bucket in *all* (!!!) occurrences and import. This is possible because currently you have the special case of a one-to-one correspondence between hosts and chassis.
This would be the procedure Wido explained and there is no other choice for this edit.
If you want to do that depends on how far you are into the data movement. If its almost done, I wouldn't bother. If its another month, it might be worth trying. As far as I can see, your crush map is going to be a short text file, so it should be feasible to edit.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Kyriazis, George <george.kyriazis@intel.com> Sent: 05 June 2020 01:36 To: Frank Schilder Cc: ceph-users Subject: Re: Best way to change bucket hierarchy
Understand that it’s difficult to debug remotely. :-)
In my current scenario I have 5 machines (1 host per chassis), but planning on adding some additional chassis with 4 hosts per chassis in the near future. Currently I am going through the first stage of adding “stub” chassis for the 5 hosts/chassis that I have, basically reparenting each host to its own chassis, as shown below:
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 203.72598 root default -5 40.01700 chassis chassis-hsw1 -9 40.01700 host vis-hsw-01 3 hdd 10.91299 osd.3 up 1.00000 1.00000 6 hdd 14.55199 osd.6 up 1.00000 1.00000 10 hdd 14.55199 osd.10 up 1.00000 1.00000 -6 40.01700 chassis chassis-hsw2 -13 40.01700 host vis-hsw-02 0 hdd 10.91299 osd.0 up 1.00000 1.00000 7 hdd 14.55199 osd.7 up 1.00000 1.00000 11 hdd 14.55199 osd.11 up 1.00000 1.00000 -7 40.01700 chassis chassis-hsw3 -11 40.01700 host vis-hsw-03 4 hdd 10.91299 osd.4 up 1.00000 1.00000 8 hdd 14.55199 osd.8 up 1.00000 1.00000 12 hdd 14.55199 osd.12 up 1.00000 1.00000 -8 40.01700 chassis chassis-hsw4 -3 40.01700 host vis-hsw-04 5 hdd 10.91299 osd.5 up 1.00000 1.00000 9 hdd 14.55199 osd.9 up 1.00000 1.00000 13 hdd 14.55199 osd.13 up 1.00000 1.00000 -17 43.65799 chassis chassis-hsw5 -15 43.65799 host vis-hsw-05 1 hdd 14.55299 osd.1 up 1.00000 1.00000 2 hdd 14.55299 osd.2 up 1.00000 1.00000 14 hdd 14.55299 osd.14 up 1.00000 1.00000
There is no additional constraint that is being added, so ideally there would be no data movement. However, I can imagine that the CRUSH algorithm could hash the PGs into different OSDs now because there is a new thing to consider (namely the chassis). Does it do that?
Thanks,
George
On Jun 4, 2020, at 6:22 PM, Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk>> wrote:
Its hard to tell without knowing what the diff is, but from your description I take it that you changed the failure domain for every(?) pool from host to chassis. I don't know what a chassis is in your architecture, but if each chassis contains several host buckets, then yes, I would expect almost every PG to be affected.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Kyriazis, George <george.kyriazis@intel.com<mailto:george.kyriazis@intel.com>> Sent: 05 June 2020 00:28:43 To: Frank Schilder Cc: ceph-users Subject: Re: Best way to change bucket hierarchy
Hmm,
So I tried all that, and I got almost all of my PGs being remapped. Crush map looks correct. Is that normal?
Thanks,
George
On Jun 4, 2020, at 2:33 PM, Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk><mailto:frans@dtu.dk>> wrote:
Hi George,
you don't need to worry about that too much. The EC profile contains two types of information, one part about the actual EC encoding and another part about crush parameters. Unfortunately, actually. Part of this information is mutable after pool creation while the rest is not. Mutable here means outside of the profile. You can change the failure domain in the crush map without issues, but the profile won't reflect that change. That's an inconsistency we currently have to live with and it would have been better to separate mutable data (like failure domain) from immutable data (like k and m) or provide a meaningful interface to maintain consistency of mutable information.
In short, don't believe everything the EC profile tells you. Some information might be out of date, like the failure domain or the device class (basically everything starting with crush-). If you remember that, you are out of trouble. Always dump the crush rule of an EC pool explicitly to see the true parameters in action.
Having said that, to change the failure domain for an EC pool, change the crush rule for the EC profile - I did this too and it works just fine. The crush rule has by default the same name as the pool. I'm afraid, here you will have to do a manual edit of the crush rule as Wido explained. There is no other way - at least currently not.
You can ask in this list for confirmation that your change is doing what you want.
Do not try to touch an EC profile, they are read-only any ways. The crush parameters are only used at pool creation and never looked at again. You can override these by editing the crush rule as explained above.
Best regards and good luck, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Kyriazis, George <george.kyriazis@intel.com<mailto:george.kyriazis@intel.com><mailto:george.kyriazis@intel.com>> Sent: 04 June 2020 20:56:38 To: Frank Schilder Cc: ceph-users Subject: Re: Best way to change bucket hierarchy
Thanks Frank,
Interesting info about the EC profile. I do have an EC pool, but I noticed the following when I dumped the profile:
# ceph osd erasure-code-profile get ec22 crush-device-class=hdd crush-failure-domain=host crush-root=default jerasure-per-chunk-alignment=false k=2 m=2 plugin=jerasure technique=reed_sol_van w=8 #
Which says that the failure domain of the EC profile is also set to host. Looks like I need to change the EC profile, too, but since it associated with the pool, maybe I can’t do that after pool creation? Or…. Since it the property is named “crush-failure-domain”, it’s automatically inherited from the crush profile, so I don’t have to do anything?
Thanks,
George
On Jun 4, 2020, at 1:51 AM, Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk><mailto:frans@dtu.dk><mailto:frans@dtu.dk>> wrote:
Hi George,
for replicated rules you can simply create a new crush rule with the new failure domain set to chassis and change any pool's crush rule to this new one. If you have EC pools, then the chooseleaf needs to be edited by hand. I did this before as well. (A really unfortunate side effect is, that the EC profile attached to the pool goes out of sync with the crush map and there is nothing one can do about that. This is annoying yet harmless.)
The intend of doing these changes while norebalance is set is
- to avoid unnecessary data movement due to successive changes happening step by step and - to make sure peering is successful before starting to move data.
I believe OSDs peer a bit faster with norebalance set and there is then a shorter interrupt to ongoing I/O (no I/O happens to a PG during peering).
Yes, if you safe the old crush map, you can undo everything. It is a good idea to have a backup also just for reference and to compare before and after.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Kyriazis, George <george.kyriazis@intel.com<mailto:george.kyriazis@intel.com><mailto:george.kyriazis@intel.com><mailto:george.kyriazis@intel.com>> Sent: 04 June 2020 00:58:20 To: Frank Schilder Cc: ceph-users Subject: Re: Best way to change bucket hierarchy
Thanks Frank,
I don’t have too much experience editing crush rules, but I assume the chooseleaf step would also have to change to:
step chooseleaf firstn 0 type chassis
Correct? Is that the only other change that is needed? It looks like the rule change can happen both inside and outside the “norebalance” setting (again with CLI commands), but is it safer to do it inside (ie. while not rebalancing)?
If I keep a backup of the crush rule map (with “ceph osd getcrushmap”), I assume I can restore the old map if something goes bad?
Thanks again!
George
On Jun 3, 2020, at 5:24 PM, Frank Schilder <frans@dtu.dk<mailto:frans@dtu.dk><mailto:frans@dtu.dk><mailto:frans@dtu.dk>> wrote:
You can use the command-line without editing the crush map. Look at the documentation of commands like
ceph osd crush add-bucket ... ceph osd crush move ...
Before starting this, set "ceph osd set norebalance" and unset after you are happy with the crush tree. Let everything peer. You should see misplaced objects and remapped PGs, but no degraded objects or PGs.
Do this only when cluster is helth_ok, otherwise things can get really complicated.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Kyriazis, George <george.kyriazis@intel.com<mailto:george.kyriazis@intel.com><mailto:george.kyriazis@intel.com><mailto:george.kyriazis@intel.com>> Sent: 03 June 2020 22:45:11 To: ceph-users Subject: [ceph-users] Best way to change bucket hierarchy
Helo,
I have a live ceph cluster, and I’m in the need of modifying the bucket hierarchy. I am currently using the default crush rule (ie. keep each replica on a different host). My need is to add a “chassis” level, and keep replicas on a per-chassis level.
From what I read in the documentation, I would have to edit the crush file manually, however this sounds kinda scary for a live cluster.
Are there any “best known methods” to achieve that goal without messing things up?
In my current scenario, I have one host per chassis, and planning on later adding nodes where there would be >1 hosts per chassis. It looks like “in theory” there wouldn’t be a need for any data movement after the crush map changes. Will reality match theory? Anything else I need to watch out for?
Thank you!
George
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io<mailto:ceph-users@ceph.io><mailto:ceph-users@ceph.io><mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave@ceph.io<mailto:ceph-users-leave@ceph.io><mailto:ceph-users-leave@ceph.io><mailto:ceph-users-leave@ceph.io>
Why don't you do the crush map change together with that? All the data will be reshuffled then any w
Couldn’t the upmap trick be used to manage this? Compute a full set to pin the before mappings, then incrementally remove them so that data moves in a controlled fashion? I ssupect that one would heed a bit of headroom on the OSDs to avoid fillage.
On 6/4/20 12:24 AM, Frank Schilder wrote:
You can use the command-line without editing the crush map. Look at the documentation of commands like
ceph osd crush add-bucket ... ceph osd crush move ...
Before starting this, set "ceph osd set norebalance" and unset after you are happy with the crush tree. Let everything peer. You should see misplaced objects and remapped PGs, but no degraded objects or PGs.
Do this only when cluster is helth_ok, otherwise things can get really complicated.
Yes and No. This will cause many CRUSHMap updates where a manual update is only a single change. I would do: $ ceph osd getcrushmap -o crushmap $ cp crushmap crushmap.backup $ crushtool -d crushmap -o crushmap.txt $ vi crushmap.txt (now make your changes) $ crushtool -c crushmap.txt -o crushmap.new $ crushtool -i crushmap.new --tree (check if all OK) $ crushtool -i crushmap.new --test --rule 0 --num-rep 3 --show-mappings If all is good: $ ceph osd setcrushmap -i crushmap.new If all goes bad, simply revert to your old crushmap. Wido
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Kyriazis, George <george.kyriazis@intel.com> Sent: 03 June 2020 22:45:11 To: ceph-users Subject: [ceph-users] Best way to change bucket hierarchy
Helo,
I have a live ceph cluster, and I’m in the need of modifying the bucket hierarchy. I am currently using the default crush rule (ie. keep each replica on a different host). My need is to add a “chassis” level, and keep replicas on a per-chassis level.
From what I read in the documentation, I would have to edit the crush file manually, however this sounds kinda scary for a live cluster.
Are there any “best known methods” to achieve that goal without messing things up?
In my current scenario, I have one host per chassis, and planning on later adding nodes where there would be >1 hosts per chassis. It looks like “in theory” there wouldn’t be a need for any data movement after the crush map changes. Will reality match theory? Anything else I need to watch out for?
Thank you!
George
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Yes and No. This will cause many CRUSHMap updates where a manual update is only a single change.
I would do:
$ ceph osd getcrushmap -o crushmap
Well, that's a yes and a no as well. If you are experienced and edit crush maps on a regular basis, you can go that way. I would still enclose the change in a norebalance setting. If you are not experienced, you are likely to shoot your cluster. In particular, adding and moving buckets is not fun this way. You need to be careful what IDs you assign, and there are many options to choose from with documentation targeted at experienced cephers. CLI commands will prevent a lot of stupid typos, errors and forgotten mandatory lines. I learned that the hard way and decided to use a direct edit only when absolutely necessary. A couple of extra peerings is a low-cost operation compared with trying to find a stupid typo that just killed all pools when angry users stand next to you. My recommendation would be to save the original crush map, apply commands and look at changes these commands do. That's a great way to learn how to do it right. And in general, better be safe than sorry. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Wido den Hollander <wido@42on.com> Sent: 04 June 2020 08:50:16 To: Frank Schilder; Kyriazis, George; ceph-users Subject: Re: [ceph-users] Re: Best way to change bucket hierarchy On 6/4/20 12:24 AM, Frank Schilder wrote:
You can use the command-line without editing the crush map. Look at the documentation of commands like
ceph osd crush add-bucket ... ceph osd crush move ...
Before starting this, set "ceph osd set norebalance" and unset after you are happy with the crush tree. Let everything peer. You should see misplaced objects and remapped PGs, but no degraded objects or PGs.
Do this only when cluster is helth_ok, otherwise things can get really complicated.
Yes and No. This will cause many CRUSHMap updates where a manual update is only a single change. I would do: $ ceph osd getcrushmap -o crushmap $ cp crushmap crushmap.backup $ crushtool -d crushmap -o crushmap.txt $ vi crushmap.txt (now make your changes) $ crushtool -c crushmap.txt -o crushmap.new $ crushtool -i crushmap.new --tree (check if all OK) $ crushtool -i crushmap.new --test --rule 0 --num-rep 3 --show-mappings If all is good: $ ceph osd setcrushmap -i crushmap.new If all goes bad, simply revert to your old crushmap. Wido
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Kyriazis, George <george.kyriazis@intel.com> Sent: 03 June 2020 22:45:11 To: ceph-users Subject: [ceph-users] Best way to change bucket hierarchy
Helo,
I have a live ceph cluster, and I’m in the need of modifying the bucket hierarchy. I am currently using the default crush rule (ie. keep each replica on a different host). My need is to add a “chassis” level, and keep replicas on a per-chassis level.
From what I read in the documentation, I would have to edit the crush file manually, however this sounds kinda scary for a live cluster.
Are there any “best known methods” to achieve that goal without messing things up?
In my current scenario, I have one host per chassis, and planning on later adding nodes where there would be >1 hosts per chassis. It looks like “in theory” there wouldn’t be a need for any data movement after the crush map changes. Will reality match theory? Anything else I need to watch out for?
Thank you!
George
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 6/4/20 9:17 AM, Frank Schilder wrote:
Yes and No. This will cause many CRUSHMap updates where a manual update is only a single change.
I would do:
$ ceph osd getcrushmap -o crushmap
Well, that's a yes and a no as well.
If you are experienced and edit crush maps on a regular basis, you can go that way. I would still enclose the change in a norebalance setting. If you are not experienced, you are likely to shoot your cluster. In particular, adding and moving buckets is not fun this way. You need to be careful what IDs you assign, and there are many options to choose from with documentation targeted at experienced cephers.
CLI commands will prevent a lot of stupid typos, errors and forgotten mandatory lines. I learned that the hard way and decided to use a direct edit only when absolutely necessary. A couple of extra peerings is a low-cost operation compared with trying to find a stupid typo that just killed all pools when angry users stand next to you.
My recommendation would be to save the original crush map, apply commands and look at changes these commands do. That's a great way to learn how to do it right. And in general, better be safe than sorry.
I think we understand each other :-) Main thing: Backup your crushmap! Then you can always roll back if things go wrong. Wido
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Wido den Hollander <wido@42on.com> Sent: 04 June 2020 08:50:16 To: Frank Schilder; Kyriazis, George; ceph-users Subject: Re: [ceph-users] Re: Best way to change bucket hierarchy
On 6/4/20 12:24 AM, Frank Schilder wrote:
You can use the command-line without editing the crush map. Look at the documentation of commands like
ceph osd crush add-bucket ... ceph osd crush move ...
Before starting this, set "ceph osd set norebalance" and unset after you are happy with the crush tree. Let everything peer. You should see misplaced objects and remapped PGs, but no degraded objects or PGs.
Do this only when cluster is helth_ok, otherwise things can get really complicated.
Yes and No. This will cause many CRUSHMap updates where a manual update is only a single change.
I would do:
$ ceph osd getcrushmap -o crushmap $ cp crushmap crushmap.backup $ crushtool -d crushmap -o crushmap.txt $ vi crushmap.txt (now make your changes) $ crushtool -c crushmap.txt -o crushmap.new $ crushtool -i crushmap.new --tree (check if all OK) $ crushtool -i crushmap.new --test --rule 0 --num-rep 3 --show-mappings
If all is good:
$ ceph osd setcrushmap -i crushmap.new
If all goes bad, simply revert to your old crushmap.
Wido
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Kyriazis, George <george.kyriazis@intel.com> Sent: 03 June 2020 22:45:11 To: ceph-users Subject: [ceph-users] Best way to change bucket hierarchy
Helo,
I have a live ceph cluster, and I’m in the need of modifying the bucket hierarchy. I am currently using the default crush rule (ie. keep each replica on a different host). My need is to add a “chassis” level, and keep replicas on a per-chassis level.
From what I read in the documentation, I would have to edit the crush file manually, however this sounds kinda scary for a live cluster.
Are there any “best known methods” to achieve that goal without messing things up?
In my current scenario, I have one host per chassis, and planning on later adding nodes where there would be >1 hosts per chassis. It looks like “in theory” there wouldn’t be a need for any data movement after the crush map changes. Will reality match theory? Anything else I need to watch out for?
Thank you!
George
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Yes, that makes total sense. Thanks, George
On Jun 4, 2020, at 2:17 AM, Frank Schilder <frans@dtu.dk> wrote:
Yes and No. This will cause many CRUSHMap updates where a manual update is only a single change.
I would do:
$ ceph osd getcrushmap -o crushmap
Well, that's a yes and a no as well.
If you are experienced and edit crush maps on a regular basis, you can go that way. I would still enclose the change in a norebalance setting. If you are not experienced, you are likely to shoot your cluster. In particular, adding and moving buckets is not fun this way. You need to be careful what IDs you assign, and there are many options to choose from with documentation targeted at experienced cephers.
CLI commands will prevent a lot of stupid typos, errors and forgotten mandatory lines. I learned that the hard way and decided to use a direct edit only when absolutely necessary. A couple of extra peerings is a low-cost operation compared with trying to find a stupid typo that just killed all pools when angry users stand next to you.
My recommendation would be to save the original crush map, apply commands and look at changes these commands do. That's a great way to learn how to do it right. And in general, better be safe than sorry.
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Wido den Hollander <wido@42on.com> Sent: 04 June 2020 08:50:16 To: Frank Schilder; Kyriazis, George; ceph-users Subject: Re: [ceph-users] Re: Best way to change bucket hierarchy
On 6/4/20 12:24 AM, Frank Schilder wrote:
You can use the command-line without editing the crush map. Look at the documentation of commands like
ceph osd crush add-bucket ... ceph osd crush move ...
Before starting this, set "ceph osd set norebalance" and unset after you are happy with the crush tree. Let everything peer. You should see misplaced objects and remapped PGs, but no degraded objects or PGs.
Do this only when cluster is helth_ok, otherwise things can get really complicated.
Yes and No. This will cause many CRUSHMap updates where a manual update is only a single change.
I would do:
$ ceph osd getcrushmap -o crushmap $ cp crushmap crushmap.backup $ crushtool -d crushmap -o crushmap.txt $ vi crushmap.txt (now make your changes) $ crushtool -c crushmap.txt -o crushmap.new $ crushtool -i crushmap.new --tree (check if all OK) $ crushtool -i crushmap.new --test --rule 0 --num-rep 3 --show-mappings
If all is good:
$ ceph osd setcrushmap -i crushmap.new
If all goes bad, simply revert to your old crushmap.
Wido
Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14
________________________________________ From: Kyriazis, George <george.kyriazis@intel.com> Sent: 03 June 2020 22:45:11 To: ceph-users Subject: [ceph-users] Best way to change bucket hierarchy
Helo,
I have a live ceph cluster, and I’m in the need of modifying the bucket hierarchy. I am currently using the default crush rule (ie. keep each replica on a different host). My need is to add a “chassis” level, and keep replicas on a per-chassis level.
From what I read in the documentation, I would have to edit the crush file manually, however this sounds kinda scary for a live cluster.
Are there any “best known methods” to achieve that goal without messing things up?
In my current scenario, I have one host per chassis, and planning on later adding nodes where there would be >1 hosts per chassis. It looks like “in theory” there wouldn’t be a need for any data movement after the crush map changes. Will reality match theory? Anything else I need to watch out for?
Thank you!
George
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (5)
-
Anthony D'Atri
-
Frank Schilder
-
Kyriazis, George
-
Wido den Hollander
-
Zhenshi Zhou