- ceph-users - lists.ceph.io

Re: mds crash loop
by Karsten Nielsen 06 Nov '19

06 Nov '19

-----Original message----- From: Yan, Zheng <ukernel(a)gmail.com> Sent: Wed 06-11-2019 08:15 Subject: Re: [ceph-users] mds crash loop To: Karsten Nielsen <karsten(a)foo-bar.dk>; CC: ceph-users(a)ceph.io; > On Tue, Nov 5, 2019 at 5:29 PM Karsten Nielsen <karsten(a)foo-bar.dk> wrote: > > > > Hi, > > > > Last week I upgraded my ceph cluster from luminus to mimic 13.2.6 > > It was running fine for a while but yesterday my mds went into a crash loop. > > > > I have 1 active and 1 standby mds for my cephfs both of which is running the > same crash loop. > > I am running ceph based on https://hub.docker.com/r/ceph/daemon version > v3.2.7-stable-3.2-minic-centos-7-x86_64 with a etcd kv store. > > > > Log details are: https://paste.debian.net/1113943/ > > > > please try again with debug_mds=20. Thanks > > Yan, Zheng Yes I have set that and had to move to pastebin.com as debian apperently only supports 150k https://pastebin.com/Gv7c5h54 - Karsten > > > Thanks for any hints > > - Karsten > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

2 1

mds crash loop
by Karsten Nielsen 06 Nov '19

06 Nov '19

Hi, Last week I upgraded my ceph cluster from luminus to mimic 13.2.6 It was running fine for a while but yesterday my mds went into a crash loop. I have 1 active and 1 standby mds for my cephfs both of which is running the same crash loop. I am running ceph based on https://hub.docker.com/r/ceph/daemon version v3.2.7-stable-3.2-minic-centos-7-x86_64 with a etcd kv store. Log details are: https://paste.debian.net/1113943/ Thanks for any hints - Karsten

2 1

Balancer configuration fails with Error EINVAL: unrecognized config option 'mgr/balancer/max_misplaced'
by Thomas Schneider 06 Nov '19

06 Nov '19

Hi, I want to adjust balancer throttling and executed this command that returns an error: root@ld3955:~# ceph config set mgr mgr/balancer/max_misplaced .01 Error EINVAL: unrecognized config option 'mgr/balancer/max_misplaced' root@ld3955:~# ceph balancer status { "active": true, "plans": [], "mode": "upmap" } Ceph is healthy, though: root@ld3955:~# ceph -s cluster: id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae health: HEALTH_WARN mon ld5505 is low on available space services: mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 19h) mgr: ld5507(active, since 23h), standbys: ld5506, ld5505, ld5508 mds: cephfs:1 {0=ld4464=up:active} 1 up:standby osd: 442 osds: 442 up, 442 in data: pools: 6 pools, 8312 pgs objects: 62.07M objects, 237 TiB usage: 710 TiB used, 821 TiB / 1.5 PiB avail pgs: 8312 active+clean io: client: 1.3 KiB/s rd, 913 KiB/s wr, 1 op/s rd, 34 op/s wr Can you please advise howto adjust balancer throttling? THX

2 1

stretch repository only has ceph-deploy
by Rodrigo Severo - Fábrica 05 Nov '19

05 Nov '19

Hi, I'm trying ceph for the first time. I'm trying to use the repository below: deb https://download.ceph.com/debian-nautilus/ stretch main But it seems that this repository only has the ceph-deploy package, not the rest of ceph. Why is that? How can I get all updated nautilus packages? Regards, Rodrigo Severo

2 1

multiple pgs down with all disks online
by Kári Bertilsson 05 Nov '19

05 Nov '19

pgs: 14.377% pgs not active 3749681/537818808 objects misplaced (0.697%) 810 active+clean 156 down 124 active+remapped+backfilling 1 active+remapped+backfill_toofull 1 down+inconsistent when looking at the down pg's all disks are online 41.3db 53775 0 0 0 401643186092 0 0 3044 down 6m 161222'303144 162913:4630171 [32,96,128,115,86,129,113,124,57,109]p32 [32,96,128,115,86,129,113,124,57,109]p32 2019-11-03 Any way to see why the pg is down ?

2 2

Re: mds crash loop
by Karsten Nielsen 05 Nov '19

05 Nov '19

from ceph -w [root@k8s-node-01 /]# ceph -w cluster: id: 571d4bfe-2c5d-45ca-8da1-91dcaf69942c health: HEALTH_WARN 1 filesystem is degraded services: mon: 3 daemons, quorum k8s-node-00,k8s-node-01,k8s-node-02 mgr: k8s-node-01(active) mds: cephfs-1/1/1 up recovery-fs-1/1/1 up {[cephfs:0]=k8s-node-00=up:active,[recovery-fs:0]=k8s-node-02=up:rejoin}, 1 up:standby osd: 6 osds: 6 up, 6 in rgw: 2 daemons active data: pools: 13 pools, 408 pgs objects: 745.2 k objects, 1.8 TiB usage: 3.6 TiB used, 8.7 TiB / 12 TiB avail pgs: 408 active+clean io: client: 3.3 KiB/s rd, 40 KiB/s wr, 1 op/s rd, 0 op/s wr 2019-11-05 09:43:20.168699 mon.k8s-node-00 [INF] daemon mds.k8s-node-02 restarted 2019-11-05 09:43:21.248009 mon.k8s-node-00 [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2019-11-05 09:43:21.348459 mon.k8s-node-00 [INF] Standby daemon mds.k8s-node-02 assigned to filesystem recovery-fs as rank 0 2019-11-05 09:43:21.349477 mon.k8s-node-00 [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2019-11-05 09:43:32.129516 mon.k8s-node-00 [INF] daemon mds.k8s-node-02 restarted 2019-11-05 09:43:32.171052 mon.k8s-node-00 [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2019-11-05 09:43:32.237675 mon.k8s-node-00 [INF] Standby daemon mds.k8s-node-02 assigned to filesystem recovery-fs as rank 0 2019-11-05 09:43:32.238331 mon.k8s-node-00 [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2019-11-05 09:43:46.825739 mon.k8s-node-00 [INF] daemon mds.k8s-node-02 restarted 2019-11-05 09:43:47.780044 mon.k8s-node-00 [ERR] Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2019-11-05 09:43:47.821611 mon.k8s-node-00 [INF] Standby daemon mds.k8s-node-02 assigned to filesystem recovery-fs as rank 0 2019-11-05 09:43:47.821976 mon.k8s-node-00 [INF] Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) ... ... -----Original message----- From: Karsten Nielsen <karsten(a)foo-bar.dk> Sent: Tue 05-11-2019 10:29 Subject: [ceph-users] mds crash loop To: ceph-users(a)ceph.io; > Hi, > > Last week I upgraded my ceph cluster from luminus to mimic 13.2.6 > It was running fine for a while but yesterday my mds went into a crash loop. > > I have 1 active and 1 standby mds for my cephfs both of which is running the > same crash loop. > I am running ceph based on https://hub.docker.com/r/ceph/daemon version > v3.2.7-stable-3.2-minic-centos-7-x86_64 with a etcd kv store. > > Log details are: https://paste.debian.net/1113943/ > > Thanks for any hints > - Karsten > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

[ceph-user] Upload objects failed on FIPS enable ceph cluster
by Amit Ghadge 04 Nov '19

04 Nov '19

Hi All, I'm using ceph-14.2.4 and testing in FIPS enable cluster. Downloading objects are works but ceph raised segmentation exception while uploading. Please help me here. And please provide debugging stage, So I could take in development environment. Thanks, Amit G

1 0

Ceph + Rook Day San Diego - November 18
by Mike Perez 04 Nov '19

04 Nov '19

Hi Cephers, I'm happy to announce the availability of the schedule for Ceph + Rook Day San Diego. Registration for this event will be free until Tuesday 11:59 UTC, so register now: https://ceph.io/cephdays/ceph-rook-day-san-diego-2019/ We still have some open spots in the schedule, but the lineup is already looking great. We are looking for more Rook related topics. If you're interested, you can submit them through our CFP form for the selection committee to review: https://zfrmz.com/hkg8EF9NYb6IvWnvoRUi We currently have SoftIron, Red Hat and SUSE set as sponsors for this event, we'd like to invite more to contact us at events(a)ceph.io: https://ceph.io/wp-content/uploads/2019/11/Ceph-Day-Partner-Sponsorship.pdf -- Mike Perez he/him Ceph Community Manager M: +1-951-572-2633 494C 5D25 2968 D361 65FB 3829 94BC D781 ADA8 8AEA @Thingee <https://twitter.com/thingee> Thingee <https://www.linkedin.com/thingee> <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

1 0

Run optimizer to create a new plan on specific pool fails
by Thomas Schneider 04 Nov '19

04 Nov '19

Hi, I want to create an optimizer plan on each pool. My cluster has multiple crush roots, and multiple pools each representing a specific drive (HDD, SSD, NVME). Some pools are balanced, some are not. Therefore I want to run optimizer to create a new plan on specific pool. However this fails for any pool with this error message: root@ld3955:~# ceph balancer optimize hdd-plan hdd Error EALREADY: Unable to find further optimization, or pool(s)' pg_num is decreasing, or distribution is already perfect root@ld3955:~# ceph balancer optimize ssd-plan ssd Error EALREADY: Unable to find further optimization, or pool(s)' pg_num is decreasing, or distribution is already perfect root@ld3955:~# ceph balancer optimize hdb_backup-plan hdb_backup Error EALREADY: Unable to find further optimization, or pool(s)' pg_num is decreasing, or distribution is already perfect root@ld3955:~# ceph osd pool ls hdb_backup hdd ssd nvme cephfs_data cephfs_metadata What is causing this error? THX

1 0

RocksDB device selection (performance requirements)
by Huseyin Cotuk 04 Nov '19

04 Nov '19

Hi all, The only recommendation I can find about db device selection is about the capacity (4% of the data disk) on the documents. Is there any suggestions about technical specs like throughput, IOPS and db device per data disk? While designing a specific infrastructure with filestore, we were looking its specs to meet requirements of all the disks behind the journal device. But in bluestore, data is directly written into data device via bluefs adapter while metadata is written into db (RocksDB) device. I know that it depends on the workload, but is there any best practice or recommendation about selection of db device? IMO, using NVME disks that we used for filestore journals as db devices is not meaningful. Because NVME disks have minimal latency and extraordinary throughput and IOPS performance. So I am not sure that DB device needs that kind of performance. So I want to use those NVME disks for a full flash pool, and choose another disks for db device. Any suggestion or recommendation would be appreciated. Best regards, Huseyin Cotuk hcotuk(a)gmail.com

1 0