- ceph-users - lists.ceph.io

MDS Issues
by DHilsbos＠performair.com 07 Mar '20

07 Mar '20

All; We are in the middle of upgrading our primary cluster from 14.2.5 to 14.2.8. Our cluster utilizes 6 MDSs for 3 CephFS file systems. 3 MDSs are collocated with MON/MGR, and 3 MDSs are collocated with OSDs. At this point we have upgraded all 3 of the MON/MDS/MGR servers. The MDS on 2 of the 3 is currently not working, and we are seeing the below log messages. 2020-03-06 11:12:56.184 <> -1 mds.<daemon> unable to obtain rotating service keys; retrying 2020-03-06 11:13:26.184 <> 0 monclient: wait_auth_rotating timed out after 30 2020-03-06 11:13:26.184 <> -1 mds.<daemon> ERROR: failed to refresh rotating keys, maximum retry time reached. 2020-03-06 11:13:26.184 <> 1 mds.<daemon> suicide! Wanted state up:boot Any ideas? Thank you, Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. DHilsbos(a)PerformAir.com www.PerformAir.com

2 2

Ceph Performance of Micron 5210 SATA?
by Hermann Himmelbauer 06 Mar '20

06 Mar '20

Hi, Does someone know if the following harddisk has a decent performance in a ceph cluster: Micron 5210 ION 1.92TB, SATA (MTFDDAK1T9QDE-2AV1ZABYY) The spec state, that the disk has power loss protection, however, I'd nevertheless like to make sure that all goes well with this disk. Best Regards, Hermann -- hermann(a)qwer.tk PGP/GPG: 299893C7 (on keyservers)

5 7

How to get the size of cephfs snapshot?
by Marc Roos 06 Mar '20

06 Mar '20

If I do getfattr --only-values --absolute-names -d -m ceph.dir.rbytes .snap/snap-6 then I am getting the same size as the unaltered source. This needs to be 0 not? And when the source is changing, this should increase? Or do I not understand correctly?

2 1

pg_num as power of two adjustment: only downwards?
by Rodrigo Severo - Fábrica 06 Mar '20

06 Mar '20

Hi, After upgrading to ceph 14.2.8 I started getting the "1 pool(s) have non-power-of-two pg_num" warning. The offending pool had 100 pgs. I tried to adjust it to 128 but it didn't work: # ceph osd pool set default.rgw.offending pgp_num 128 Error EINVAL: specified pgp_num 128 > pg_num 100 It only worked downwards: # ceph osd pool set default.rgw.offending pgp_num 64 set pool 7 pgp_num to 64 Is that expected? And having reduced my pg_num to 64 (isn't it to small?) should I be prepared for problems in the future? Regards, Rodrigo Severo

2 2

Aborted multipart uploads still visible
by Maks Kowalik 06 Mar '20

06 Mar '20

Hello, I have a problem with one particular bucket that aborting incomplete multipart uploads does not remove them from the list i.e. the storage space is freed up, but 's3cmd multipart' is still showing them. When i retry aborting, I'm getting 404 which is correct. The issue here is that I can only display first 1000 of such uploads, thus can't remove next ones. Does anyone know how can I make the aborted ones disappear from the listing (this was checked on a different bucket and they disappear as expected)? Ceph version is 14.2.4. Kind regards, Maks Kowalik

2 1

Re: How can I fix "object unfound" error?
by Chad William Seys 06 Mar '20

06 Mar '20

> Maybe.... I've marked the object as "lost" and removed the failed > OSD. > > The cluster now is healthy, but I'd like to understand if it's likely > to bother me again in the future. Yeah, I don't know. Within the last month there are 4 separate instances of people mentioning "unfound" object in their cluster. I'm deferring as long as possible any OSD drive upgrades. I ran into the problem when "draining" an OSD. "draining" means remove OSD from crush map, wait for all PG to be stored elsewhere, then replace drive with larger one. Under those circumstances there should be no PG unfound. BTW, are you using cache tiering ? The bug report mentions this, but some people did not have this enabled. Chad.

3 4

Can't add a ceph-mon to existing large cluster
by Dan van der Ster 05 Mar '20

05 Mar '20

Hi all, There's something broken in our env when we try to add new mons to existing clusters, confirmed on two clusters running mimic and nautilus. It's basically this issue https://tracker.ceph.com/issues/42830 In case something is wrong with our puppet manifests, I'm trying to doing it manually. First we --mkfs the mon and start it, but as soon as the new mon starts synchronizing, the existing leader becomes unresponsive and an election is triggered. Here's exactly what I'm doing: # cd /var/lib/ceph/tmp/ # scp cephmon1:/var/lib/ceph/tmp/keyring.mon.cephmon1 keyring.mon.cephmon4 # ceph mon getmap -o monmap # ceph-mon --mkfs -i cephmon4 --monmap monmap --keyrin keyring.mon.cephmon4 --setuser ceph --setgroup ceph # vi /etc/ceph/ceph.conf <add the new mon to ceph.conf like this> [mon.cephmon4] host = cephmon4 mon addr = a.b.c.d:6790 # systemctl start ceph-mon@cephmon4 The log file on the new mon shows it start synchronizing, then immediately the CPU usage on the leader goes to 100% and elections start happening, and ceph health shows mon slow ops. perf top of the ceph-mon with 100% CPU is shown below [1]. On a small nautilus cluster, the new mon gets added withing a minute or so (but not cleanly -- the leader is unresponsive for quite awhile until the new mon joins). debug_mon=20 on the leader doesn't show anything very interesting. On our large mimic cluster we tried waiting more than 10 minutes -- suffering through several mon elections and 100% usage bouncing around between leaders -- until we gave up. I'm pulling my hair out a bit on this -- it's really weird! Did anyone add a new mon to an existing large cluster recently, and it went smoothly? Cheers, Dan [1] 15.12% ceph-mon [.] MonitorDBStore::Transaction::encode 8.95% libceph-common.so.0 [.] ceph::buffer::v14_2_0::ptr::append 8.68% libceph-common.so.0 [.] ceph::buffer::v14_2_0::list::append 7.69% libceph-common.so.0 [.] ceph::buffer::v14_2_0::ptr::release 5.86% libceph-common.so.0 [.] ceph::buffer::v14_2_0::ptr::ptr

4 12

rbd-mirror - which direction?
by Ml Ml 05 Mar '20

05 Mar '20

Hello, i am running luminous and i would like to back up my cluster from Site-A to Site-B (one way) So i decided to mirror it to an off-site ceph cluster. I read: https://docs.ceph.com/docs/luminous/rbd/rbd-mirroring/ But i liked I https://github.com/MiracleMa/Blog/issues/2 a little better. But which parameter/part show the direction it will mirror? Thanks, Mario

1 0

Unexpected recovering after nautilus 14.2.7 -> 14.2.8
by Rainer Krienke 05 Mar '20

05 Mar '20

Hello, at the moment my ceph is still working but in a degraded state after I upgraded one (in 9) hosts from 14.2.7 to 14.2.8 and rebooting this host (node2, one monitor in 3) after the upgrade. Usually before rebooting I set ceph osd set noout ceph osd set nobackfill ceph osd set norecover before rebooting, but I fogot this time. After having realized my error I thought, ok I forgot to set the flags but I configured mon_osd_down_out_interval to 900sec: # ceph config get mon.mon_osd_down_out_interval WHO MASK LEVEL OPTION VALUE RO mon advanced mon_osd_down_out_interval 900 The reboot took 5min so I expected nothing to happen. But it did and now I do not understand why and if there are more timeout values I could/should set to avoid this happening again if I ever should again forget to set the noout , nobackfill, norecover flags prior to a reboot? Thanks if anyone can explain to me what might have happened.... Rainer The current ceph state is: # ceph -s cluster: id: xyz health: HEALTH_WARN Degraded data redundancy: 191629/76527549 objects degraded (0.250%), 18 pgs degraded, 18 pgs undersized services: mon: 3 daemons, quorum node2,node5,node8 (age 51m) mgr: node5(active, since 53m), standbys: node8, node-admin, node2 mds: mycephfs:1 {0=node3=up:active} 2 up:standby osd: 144 osds: 144 up (since 51m), 144 in (since 3M); 48 remapped pgs data: pools: 13 pools, 3460 pgs objects: 12.76M objects, 48 TiB usage: 95 TiB used, 429 TiB / 524 TiB avail pgs: 191629/76527549 objects degraded (0.250%) 3098164/76527549 objects misplaced (4.048%) 3412 active+clean 30 active+remapped+backfill_wait 13 active+undersized+degraded+remapped+backfill_wait 5 active+undersized+degraded+remapped+backfilling io: client: 33 MiB/s rd, 7.2 MiB/s wr, 91 op/s rd, 186 op/s wr recovery: 83 MiB/s, 20 objects/s -- Rainer Krienke, Uni Koblenz, Rechenzentrum, A22, Universitaetsstrasse 1 56070 Koblenz, Tel: +49261287 1312 Fax +49261287 100 1312 Web: http://userpages.uni-koblenz.de/~krienke PGP: http://userpages.uni-koblenz.de/~krienke/mypgp.html

3 9

PGs unknown after pool creation (Nautilus 14.2.4/6)
by dg＠terralink.de 05 Mar '20

05 Mar '20

Hello, I have a small ceph cluster running with 3 MON/MGR and 3 OSD hosts. There are also 3 virtual hosts in the crushmap to have a seperate SSD pool. Currently two pools are running, one of that exclusive to the SSD device class. My problem now is, that any new pool I try to create won't become functional as all new pgs are in unknown state. I've tried varying pg number, crush ruleset, size and so on, but nothing helped. my OSDs regularly show an error message like "var/log/ceph/ceph-osd.42.log:2020-03-04 17:09:04.641 7f76240c9700 0 --1- [v2:192.168.44.110:6834/23888,v1:192.168.44.110:6835/23888] >> v1:192.168.44.111:6826/484449 conn(0x55901c805800 0x55901d5bf000 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=398 cs=186 l=0).handle_connect_reply_2 connect got BADAUTHORIZER" ,but I don't find any reasons for that (clocks are synchronized). Also one of my mons is two minor versions newer then the other nodes, but would not really like to update the whole cluster right now as I've had somewhat bad experience with the last update :) Does anyone have any idea what I could try ?

1 0