stretched cluster new pool and second pool with nvme
hi ... running against the wall, i need your help, again. our test stretched cluster is running fine. now i have 2 questions. whats the right way to add another pool? create pool with 4/2 and use the rule for the stretched mode, finished? the exsisting pools were automaticly set to 4/2 after "ceph mon enable_stretch_mode". the second question, we want to use ssd and nvme together. so, we need to have a second pool for class nvme. i don't know, how to setup a second crush rule for the nvme class. i thought, that i need to filter with 2 rules for the classes. is that correct? thanks for help, ronny
Hi,
whats the right way to add another pool? create pool with 4/2 and use the rule for the stretched mode, finished? the exsisting pools were automaticly set to 4/2 after "ceph mon enable_stretch_mode".
if that is what you require, then yes, it's as easy as that. Although I haven't played too much with the strech mode yet, you'll probably want to check the PG distribution after you created the pool. Maybe start with only a few PGs (4 or 8) and then inspect the distribution: ceph pg ls-by-pool <new_pool> Then verify if the PGs are properly distributed across both DCs. Of course, there's also the crushtool [1] to test such changes before actually applying them: - Get current crushmap (ceph osd getcrushmap -o crushmap.bin) - Decompile it (crushtool -d crushmap.bin -o crushmap.txt) - Add new crush rule - Compile it (crushtool -c crushmap.txt -o crushmap.new) - Test it (crushtool -i crushmap.new --test --rule <NEW_RULE> --num-rep 4 --show-mappings) If you're satisfied with the mappings (also check if there are bad mappings with --show-bad-mappings) you can apply the new crushmap (ceph osd setcrushmap -i crushmap.new). Be careful when injecting a new crushmap!!!
i don't know, how to setup a second crush rule for the nvme class. i thought, that i need to filter with 2 rules for the classes. is that correct?
If your device class is called "nvme", you can create a rule like this: host01:~ # ceph osd crush rule create-replicated nvme-rule default host nvme The second parameter "default" is the "root" of your crush tree. The third parameter is your failure domain. You can also create rules in the dashboard, of course. Regards, Eugen [1] https://docs.ceph.com/en/latest/man/8/crushtool/ Zitat von "ronny.lippold" <ceph@spark5.de>:
hi ... running against the wall, i need your help, again.
our test stretched cluster is running fine. now i have 2 questions.
whats the right way to add another pool? create pool with 4/2 and use the rule for the stretched mode, finished? the exsisting pools were automaticly set to 4/2 after "ceph mon enable_stretch_mode".
the second question, we want to use ssd and nvme together. so, we need to have a second pool for class nvme.
i don't know, how to setup a second crush rule for the nvme class. i thought, that i need to filter with 2 rules for the classes. is that correct?
thanks for help, ronny _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
On 23-04-2024 14:40, Eugen Block wrote:
Hi,
whats the right way to add another pool? create pool with 4/2 and use the rule for the stretched mode, finished? the exsisting pools were automaticly set to 4/2 after "ceph mon enable_stretch_mode".
It should be that simple. However, it does not seem to work. I tried to do just that, use two separate pools, hdd and ssd in that case, but it would not work, see this tracker: https://tracker.ceph.com/issues/64817 If your experience is different please update the tracker ticket. If it indeed does not work, please also update the tracker ticket with a "+1". Thanks, Gr. Stefan
Oh, I see. Unfortunately, I don't have a cluster in stretch mode so I can't really test that. Thanks for pointing to the tracker. Zitat von Stefan Kooman <stefan@bit.nl>:
On 23-04-2024 14:40, Eugen Block wrote:
Hi,
whats the right way to add another pool? create pool with 4/2 and use the rule for the stretched mode, finished? the exsisting pools were automaticly set to 4/2 after "ceph mon enable_stretch_mode".
It should be that simple. However, it does not seem to work. I tried to do just that, use two separate pools, hdd and ssd in that case, but it would not work, see this tracker: https://tracker.ceph.com/issues/64817
If your experience is different please update the tracker ticket. If it indeed does not work, please also update the tracker ticket with a "+1".
Thanks,
Gr. Stefan
hi stefan ... you are the hero of the month ;) i don't know, why i did not found your bug report. i have the exact same problem and resolved the HEALTH only with "ceph osd force_healthy_stretch_mode --yes-i-really-mean-it" will comment the report soon. actually, we think about 4/2 size without stretch mode enable. what was your solution? many thanks ... ronny Am 2024-04-23 15:03, schrieb Stefan Kooman:
On 23-04-2024 14:40, Eugen Block wrote:
Hi,
whats the right way to add another pool? create pool with 4/2 and use the rule for the stretched mode, finished? the exsisting pools were automaticly set to 4/2 after "ceph mon enable_stretch_mode".
It should be that simple. However, it does not seem to work. I tried to do just that, use two separate pools, hdd and ssd in that case, but it would not work, see this tracker: https://tracker.ceph.com/issues/64817
If your experience is different please update the tracker ticket. If it indeed does not work, please also update the tracker ticket with a "+1".
Thanks,
Gr. Stefan
On 30-04-2024 11:22, ronny.lippold wrote:
hi stefan ... you are the hero of the month ;)
:p.
i don't know, why i did not found your bug report.
i have the exact same problem and resolved the HEALTH only with "ceph osd force_healthy_stretch_mode --yes-i-really-mean-it" will comment the report soon.
actually, we think about 4/2 size without stretch mode enable.
what was your solution?
This specific setup (on which I did the testing) is going to be full flash (SSD). So the HDDs are going to be phased out. And only the default non-device-class crush rule will be used. While that will work for this (small) cluster, it is not a solution. This issue should be fixed, as I figure there are quite a few cluster that want to use device-classes and use stretch mode at the same time. Gr. Stefan
hi stefan ... i did the next step and need your help. my idea was to stretch the cluster without stretch mode. so we decided to reserve a size of 4 on each side. the setup is the same as stretched mode, also crush rule, location, election_strategy and tie breaker. only "ceph mon enable_stretch_mode e stretch_rule datacenter" wasn't made. now in my test, i made a split brain and expect, that on the remaining side, the cluster will rebuild the 4 replica. but that did not happen. actually, the cluster, is doing the same stuff, as stretch mode enabled. writeable with 2 replica. can you explain me why? i'm spinning around. this is the status during split brain: ###################################### pve-test02-01:~# ceph -s cluster: id: 376fcdef-bba0-4e58-b63e-c9754dc948fa health: HEALTH_WARN 6/13 mons down, quorum pve-test01-01,pve-test01-03,pve-test01-05,pve-test02-01,pve-test02-03,pve-test02-05,tie-breaker 1 datacenter (8 osds) down 8 osds down 6 hosts (8 osds) down Degraded data redundancy: 2116/4232 objects degraded (50.000%), 95 pgs degraded, 113 pgs undersized services: mon: 13 daemons, quorum pve-test01-01,pve-test01-03,pve-test01-05,pve-test02-01,pve-test02-03,pve-test02-05,tie-breaker (age 54m), out of quorum: pve-test01-02, pve-test01-04, pve-test01-06, pve-test02-02, pve-test02-04, pve-test02-06 mgr: pve-test02-05(active, since 53m), standbys: pve-test01-05, pve-test01-01, pve-test01-03, pve-test02-01, pve-test02-03 mds: 1/1 daemons up, 1 standby osd: 16 osds: 8 up (since 54m), 16 in (since 77m) data: volumes: 1/1 healthy pools: 5 pools, 113 pgs objects: 1.06k objects, 3.9 GiB usage: 9.7 GiB used, 580 GiB / 590 GiB avail pgs: 2116/4232 objects degraded (50.000%) 95 active+undersized+degraded 18 active+undersized io: client: 17 KiB/s wr, 0 op/s rd, 10 op/s wr ###################################### thanks a lot, ronny Am 2024-04-30 11:42, schrieb Stefan Kooman:
On 30-04-2024 11:22, ronny.lippold wrote:
hi stefan ... you are the hero of the month ;)
:p.
i don't know, why i did not found your bug report.
i have the exact same problem and resolved the HEALTH only with "ceph osd force_healthy_stretch_mode --yes-i-really-mean-it" will comment the report soon.
actually, we think about 4/2 size without stretch mode enable.
what was your solution?
This specific setup (on which I did the testing) is going to be full flash (SSD). So the HDDs are going to be phased out. And only the default non-device-class crush rule will be used. While that will work for this (small) cluster, it is not a solution. This issue should be fixed, as I figure there are quite a few cluster that want to use device-classes and use stretch mode at the same time.
Gr. Stefan
How exactly does your crush rule look right now? I assume it's supposed to distribute data across two sites, and since one site is missing, the PGs stay in degraded state until the site comes back up. You would need to either change the crush rule or assign a different one to that pool which would allow to recover one the remaining site. Zitat von "ronny.lippold" <ceph@spark5.de>:
hi stefan ... i did the next step and need your help.
my idea was to stretch the cluster without stretch mode. so we decided to reserve a size of 4 on each side.
the setup is the same as stretched mode, also crush rule, location, election_strategy and tie breaker. only "ceph mon enable_stretch_mode e stretch_rule datacenter" wasn't made.
now in my test, i made a split brain and expect, that on the remaining side, the cluster will rebuild the 4 replica. but that did not happen. actually, the cluster, is doing the same stuff, as stretch mode enabled. writeable with 2 replica.
can you explain me why? i'm spinning around.
this is the status during split brain:
###################################### pve-test02-01:~# ceph -s cluster: id: 376fcdef-bba0-4e58-b63e-c9754dc948fa health: HEALTH_WARN 6/13 mons down, quorum pve-test01-01,pve-test01-03,pve-test01-05,pve-test02-01,pve-test02-03,pve-test02-05,tie-breaker 1 datacenter (8 osds) down 8 osds down 6 hosts (8 osds) down Degraded data redundancy: 2116/4232 objects degraded (50.000%), 95 pgs degraded, 113 pgs undersized
services: mon: 13 daemons, quorum pve-test01-01,pve-test01-03,pve-test01-05,pve-test02-01,pve-test02-03,pve-test02-05,tie-breaker (age 54m), out of quorum: pve-test01-02, pve-test01-04, pve-test01-06, pve-test02-02, pve-test02-04, pve-test02-06 mgr: pve-test02-05(active, since 53m), standbys: pve-test01-05, pve-test01-01, pve-test01-03, pve-test02-01, pve-test02-03 mds: 1/1 daemons up, 1 standby osd: 16 osds: 8 up (since 54m), 16 in (since 77m)
data: volumes: 1/1 healthy pools: 5 pools, 113 pgs objects: 1.06k objects, 3.9 GiB usage: 9.7 GiB used, 580 GiB / 590 GiB avail pgs: 2116/4232 objects degraded (50.000%) 95 active+undersized+degraded 18 active+undersized
io: client: 17 KiB/s wr, 0 op/s rd, 10 op/s wr ######################################
thanks a lot, ronny
Am 2024-04-30 11:42, schrieb Stefan Kooman:
On 30-04-2024 11:22, ronny.lippold wrote:
hi stefan ... you are the hero of the month ;)
:p.
i don't know, why i did not found your bug report.
i have the exact same problem and resolved the HEALTH only with "ceph osd force_healthy_stretch_mode --yes-i-really-mean-it" will comment the report soon.
actually, we think about 4/2 size without stretch mode enable.
what was your solution?
This specific setup (on which I did the testing) is going to be full flash (SSD). So the HDDs are going to be phased out. And only the default non-device-class crush rule will be used. While that will work for this (small) cluster, it is not a solution. This issue should be fixed, as I figure there are quite a few cluster that want to use device-classes and use stretch mode at the same time.
Gr. Stefan
ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (3)
-
Eugen Block
-
ronny.lippold
-
Stefan Kooman