1 Oct
2019
1 Oct
'19
3:46 p.m.
Hi. I am new to ceph but have set it up on my homelab and started using it. It seemed very
good intil I desided to try pg autoscale.
After enabling autoscale to 3 of my pools, autoscale tried(?) to reduce the number of PGs
and the pools are now unaccessible.
I have tried to turn it off again, but no luck! Please help.
ceph status:
https://pastebin.com/88qNivJi (do not know why it lists 4 pools, I have 3. Maybe one of
the pools I created after and deleted are in limbo?)
ceph osd pool ls detail:
https://pastebin.com/HZLz6yHL
ceph health detail:
https://pastebin.com/Kqd2YMtm
1 Oct
1 Oct
3:50 p.m.
On 10/1/19 12:16 PM, Raymond Berg Hansen wrote:
Hi. I am new to ceph but have set it up on my homelab
and started using it. It seemed very good intil I desided to try pg autoscale.
After enabling autoscale to 3 of my pools, autoscale tried(?) to reduce the number of PGs
and the pools are now unaccessible.
I have tried to turn it off again, but no luck! Please help.
Are you sure the data is not available? The 'unknown' status can
sometimes happen if the Mgr isn't receiving the data.
Have you tried to restart the active Manager?
Wido
> ceph status:
> https://pastebin.com/88qNivJi (do not know why it lists 4 pools, I have 3. Maybe one
of the pools I created after and deleted are in limbo?)
>
> ceph osd pool ls detail:
> https://pastebin.com/HZLz6yHL
>
> ceph health detail:
> https://pastebin.com/Kqd2YMtm
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
4:14 p.m.
Hi,
we had a problem with the autoscaler just recently, we had to turn it
off because the MONs suddenly became laggy [1]. Did you check the MON
processes? Try disabling autoscaler, wait until that change is applied
and then restart MONs one by one.
Regards,
Eugen
[1]
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/TZZIRVRGQK…
Zitat von Raymond Berg Hansen <raymondbh(a)gmail.com>om>:
Yes I am sure, tried to restart the whole cluster.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
5:38 p.m.
It seems that the metadata pool is not affected, those PGs are all
active+clean. Are different rulesets applied to 'cephfs_data' and
'cephfs_metadata'?
Have you changed anything regarding crush rules or device classes or
something related to the ceph osd tree?
Zitat von Raymond Berg Hansen <raymondbh(a)gmail.com>om>:
Thanks for the tip, but I already tried that to. First
thing I tried
actually.
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
6:01 p.m.
You are absolutly right, I had made a crush rule for device class hdd. Did not put this in
connection with this problem. When I put the pools back in the default crush rule things
are starting to fix itself it seems.
Have I done something wrong with this crush rule?
# rules
rule replicated_rule {
id 0
type replicated
min_size 1
max_size 10
step take default
step chooseleaf firstn 0 type host
step emit
}
rule replicated-hdd {
id 1
type replicated
min_size 1
max_size 10
step take default class hdd
step chooseleaf firstn 0 type datacenter
step emit
}
6:08 p.m.
Some time ago on Luminous I also had to change the crush rules on a all
hdd cluster to hdd (to prepare for adding ssd's and ssd pools). And pg's
started migrating while everything already was on hdd's, looks like this
is still not fixed?
-----Original Message-----
From: Raymond Berg Hansen [mailto:raymondbh@gmail.com]
Sent: dinsdag 1 oktober 2019 14:32
To: ceph-users(a)ceph.io
Subject: [ceph-users] Re: Nautilus pg autoscale, data lost?
You are absolutly right, I had made a crush rule for device class hdd.
Did not put this in connection with this problem. When I put the pools
back in the default crush rule things are starting to fix itself it
seems.
Have I done something wrong with this crush rule?
# rules
rule replicated_rule {
id 0
type replicated
min_size 1
max_size 10
step take default
step chooseleaf firstn 0 type host
step emit
}
rule replicated-hdd {
id 1
type replicated
min_size 1
max_size 10
step take default class hdd
step chooseleaf firstn 0 type datacenter
step emit
}
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an
email to ceph-users-leave(a)ceph.io
6:16 p.m.
Some time ago on Luminous I also had to change the
crush rules on a all
hdd cluster to hdd (to prepare for adding ssd's and ssd pools). And pg's
started migrating while everything already was on hdd's, looks like this
is still not fixed?
Sage responded to a thread yesterday, how to change crush device
classes without rebalancing (crushtool reclassify):
https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/675QZ2JXXX4…
Zitat von Marc Roos <M.Roos(a)f1-outsourcing.eu>eu>:
Some time ago on Luminous I also had to change the
crush rules on a all
hdd cluster to hdd (to prepare for adding ssd's and ssd pools). And pg's
started migrating while everything already was on hdd's, looks like this
is still not fixed?
>
>
>
>
>
> -----Original Message-----
> From: Raymond Berg Hansen [mailto:raymondbh@gmail.com]
> Sent: dinsdag 1 oktober 2019 14:32
> To: ceph-users(a)ceph.io
> Subject: [ceph-users] Re: Nautilus pg autoscale, data lost?
>
> You are absolutly right, I had made a crush rule for device class hdd.
> Did not put this in connection with this problem. When I put the pools
> back in the default crush rule things are starting to fix itself it
> seems.
> Have I done something wrong with this crush rule?
>
> # rules
> rule replicated_rule {
> id 0
> type replicated
> min_size 1
> max_size 10
> step take default
> step chooseleaf firstn 0 type host
> step emit
> }
> rule replicated-hdd {
> id 1
> type replicated
> min_size 1
> max_size 10
> step take default class hdd
> step chooseleaf firstn 0 type datacenter
> step emit
> }
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an
> email to ceph-users-leave(a)ceph.io
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
8:39 p.m.
Hi Raymond,
I believe the "type datacenter" bit in your replicated-hdd rule should
read "type host" instead, as in the original replicated rule.
Best
Mattia
On 10/1/19 2:31 PM, Raymond Berg Hansen wrote:
You are absolutly right, I had made a crush rule for
device class hdd. Did not put this in connection with this problem. When I put the pools
back in the default crush rule things are starting to fix itself it seems.
Have I done something wrong with this crush rule?
# rules
rule replicated_rule {
id 0
type replicated
min_size 1
max_size 10
step take default
step chooseleaf firstn 0 type host
step emit
}
rule replicated-hdd {
id 1
type replicated
min_size 1
max_size 10
step take default class hdd
step chooseleaf firstn 0 type datacenter
step emit
}
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
1669
days inactive
1669
days old
9 comments
5 participants
participants (5)
-
Eugen Block -
Marc Roos -
Mattia Belluco -
Raymond Berg Hansen -
Wido den Hollander