Nautilus pg autoscale, data lost?
Hi. I am new to ceph but have set it up on my homelab and started using it. It seemed very good intil I desided to try pg autoscale. After enabling autoscale to 3 of my pools, autoscale tried(?) to reduce the number of PGs and the pools are now unaccessible. I have tried to turn it off again, but no luck! Please help. ceph status: https://pastebin.com/88qNivJi (do not know why it lists 4 pools, I have 3. Maybe one of the pools I created after and deleted are in limbo?) ceph osd pool ls detail: https://pastebin.com/HZLz6yHL ceph health detail: https://pastebin.com/Kqd2YMtm
On 10/1/19 12:16 PM, Raymond Berg Hansen wrote:
Hi. I am new to ceph but have set it up on my homelab and started using it. It seemed very good intil I desided to try pg autoscale. After enabling autoscale to 3 of my pools, autoscale tried(?) to reduce the number of PGs and the pools are now unaccessible. I have tried to turn it off again, but no luck! Please help.
Are you sure the data is not available? The 'unknown' status can sometimes happen if the Mgr isn't receiving the data. Have you tried to restart the active Manager? Wido
ceph status: https://pastebin.com/88qNivJi (do not know why it lists 4 pools, I have 3. Maybe one of the pools I created after and deleted are in limbo?)
ceph osd pool ls detail: https://pastebin.com/HZLz6yHL
ceph health detail: https://pastebin.com/Kqd2YMtm _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi, we had a problem with the autoscaler just recently, we had to turn it off because the MONs suddenly became laggy [1]. Did you check the MON processes? Try disabling autoscaler, wait until that change is applied and then restart MONs one by one. Regards, Eugen [1] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/TZZIRVRGQK4... Zitat von Raymond Berg Hansen <raymondbh@gmail.com>:
Yes I am sure, tried to restart the whole cluster. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
It seems that the metadata pool is not affected, those PGs are all active+clean. Are different rulesets applied to 'cephfs_data' and 'cephfs_metadata'? Have you changed anything regarding crush rules or device classes or something related to the ceph osd tree? Zitat von Raymond Berg Hansen <raymondbh@gmail.com>:
Thanks for the tip, but I already tried that to. First thing I tried actually. _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
You are absolutly right, I had made a crush rule for device class hdd. Did not put this in connection with this problem. When I put the pools back in the default crush rule things are starting to fix itself it seems. Have I done something wrong with this crush rule? # rules rule replicated_rule { id 0 type replicated min_size 1 max_size 10 step take default step chooseleaf firstn 0 type host step emit } rule replicated-hdd { id 1 type replicated min_size 1 max_size 10 step take default class hdd step chooseleaf firstn 0 type datacenter step emit }
Some time ago on Luminous I also had to change the crush rules on a all hdd cluster to hdd (to prepare for adding ssd's and ssd pools). And pg's started migrating while everything already was on hdd's, looks like this is still not fixed? -----Original Message----- From: Raymond Berg Hansen [mailto:raymondbh@gmail.com] Sent: dinsdag 1 oktober 2019 14:32 To: ceph-users@ceph.io Subject: [ceph-users] Re: Nautilus pg autoscale, data lost? You are absolutly right, I had made a crush rule for device class hdd. Did not put this in connection with this problem. When I put the pools back in the default crush rule things are starting to fix itself it seems. Have I done something wrong with this crush rule? # rules rule replicated_rule { id 0 type replicated min_size 1 max_size 10 step take default step chooseleaf firstn 0 type host step emit } rule replicated-hdd { id 1 type replicated min_size 1 max_size 10 step take default class hdd step chooseleaf firstn 0 type datacenter step emit } _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Some time ago on Luminous I also had to change the crush rules on a all hdd cluster to hdd (to prepare for adding ssd's and ssd pools). And pg's started migrating while everything already was on hdd's, looks like this is still not fixed?
Sage responded to a thread yesterday, how to change crush device classes without rebalancing (crushtool reclassify): https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/675QZ2JXXX4R... Zitat von Marc Roos <M.Roos@f1-outsourcing.eu>:
Some time ago on Luminous I also had to change the crush rules on a all hdd cluster to hdd (to prepare for adding ssd's and ssd pools). And pg's started migrating while everything already was on hdd's, looks like this is still not fixed?
-----Original Message----- From: Raymond Berg Hansen [mailto:raymondbh@gmail.com] Sent: dinsdag 1 oktober 2019 14:32 To: ceph-users@ceph.io Subject: [ceph-users] Re: Nautilus pg autoscale, data lost?
You are absolutly right, I had made a crush rule for device class hdd. Did not put this in connection with this problem. When I put the pools back in the default crush rule things are starting to fix itself it seems. Have I done something wrong with this crush rule?
# rules rule replicated_rule { id 0 type replicated min_size 1 max_size 10 step take default step chooseleaf firstn 0 type host step emit } rule replicated-hdd { id 1 type replicated min_size 1 max_size 10 step take default class hdd step chooseleaf firstn 0 type datacenter step emit } _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
Hi Raymond, I believe the "type datacenter" bit in your replicated-hdd rule should read "type host" instead, as in the original replicated rule. Best Mattia On 10/1/19 2:31 PM, Raymond Berg Hansen wrote:
You are absolutly right, I had made a crush rule for device class hdd. Did not put this in connection with this problem. When I put the pools back in the default crush rule things are starting to fix itself it seems. Have I done something wrong with this crush rule?
# rules rule replicated_rule { id 0 type replicated min_size 1 max_size 10 step take default step chooseleaf firstn 0 type host step emit } rule replicated-hdd { id 1 type replicated min_size 1 max_size 10 step take default class hdd step chooseleaf firstn 0 type datacenter step emit } _______________________________________________ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send an email to ceph-users-leave@ceph.io
participants (5)
-
Eugen Block
-
Marc Roos
-
Mattia Belluco
-
Raymond Berg Hansen
-
Wido den Hollander