- ceph-users - lists.ceph.io

reset pgs not deep-scrubbed in time
by Stefan Priebe - Profihost AG 11 Mar '20

11 Mar '20

Hello, is there any way to reset deep-scrubbed time for pgs? The cluster was accidently in state nodeep-scrub and is now unable to deep scrub fast enough. Is there any way to force mark all pgs as deep scrubbed to start from 0 again? Greets, Stefan

2 1

rbd-mirror replay is very slow - but initial bootstrap is fast
by Ml Ml 11 Mar '20

11 Mar '20

Hello List, when i initially enable journal/mirror on an image it gets bootstrapped to my site-b pretty quickly with 250MB/sec which is about the IO Write limit. Once its up2date, the replay is very slow. About 15KB/sec and the entries_behind_maste is just running away: root@ceph01:~# rbd --cluster backup mirror pool status rbd-cluster6 --verbose health: OK images: 3 total 3 replaying ... vm-112-disk-0: global_id: 60a795c3-9f5d-4be3-b9bd-3df971e531fa state: up+replaying description: replaying, master_position=[object_number=623, tag_tid=3, entry_tid=345567], mirror_position=[object_number=35, tag_tid=3, entry_tid=18371], entries_behind_master=327196 last_update: 2020-03-10 11:36:44 ... Write traffic on the source is about 20/25MB/sec. On the Source i run 14.2.6 and on the destination 12.2.13. Any idea why the replaying is sooo slow? Thanks, Michael

3 8

FW: Warning: could not send message for past 4 hours
by Marc Roos 11 Mar '20

11 Mar '20

-----Original Message----- From: Mail Delivery Subsystem [mailto:MAILER-DAEMON] Sent: 10 March 2020 19:01 Subject: Warning: could not send message for past 4 hours ********************************************** ** THIS IS A WARNING MESSAGE ONLY ** ** YOU DO NOT NEED TO RESEND YOUR MESSAGE ** ********************************************** The original message was received at Tue, 10 Mar 2020 15:00:07 +0100 from localhost.localdomain [127.0.0.1] ----- Transcript of session follows ----- 451 croit.io: Name server timeout Warning: message still undelivered after 4 hours Will keep trying until message is 5 days old

1 0

Re: cephfs snap mkdir strange timestamp
by Paul Emmerich 10 Mar '20

10 Mar '20

There's an xattr for this: ceph.snap.btime IIRC Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Tue, Mar 10, 2020 at 11:42 AM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > > > If I make a directory in linux the directory has the date of now, why is > this not with creating a snap dir? Is this not a bug? One expects this > to be the same as in linux not???? > > [ @ test]$ mkdir temp > > [ @os0 test]$ ls -arltn > total 28 > drwxrwxrwt. 27 0 0 20480 Mar 10 11:38 .. > drwxrwxr-x 2 801 801 4096 Mar 10 11:38 temp > drwxrwxr-x 3 801 801 4096 Mar 10 11:38 . > > > [ @ .snap]# mkdir test > [ @ .snap]# ls -lartn > total 0 > drwxr-xr-x 861886554 0 0 8390344070786420358 Jan 1 1970 . > drwxr-xr-x 4 0 0 2 Mar 6 14:43 test > drwxr-xr-x 4 0 0 2 Mar 6 14:43 snap-9 > drwxr-xr-x 4 0 0 2 Mar 6 14:43 snap-8 > drwxr-xr-x 4 0 0 2 Mar 6 14:43 snap-7 > drwxr-xr-x 4 0 0 2 Mar 6 14:43 snap-6 > drwxr-xr-x 4 0 0 2 Mar 6 14:43 snap-5 > drwxr-xr-x 4 0 0 2 Mar 6 14:43 snap-10 > drwxr-xr-x 4 0 0 2 Mar 6 14:43 .. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

ceph: Can't lookup inode 1 (err: -13)
by Marc Roos 10 Mar '20

10 Mar '20

For testing purposes I changed the kernel 3.10 for a 5.5, now I am getting these messages. I assume the 3.10 was just never displaying these. Could this be a problem with my caps of the fs id user? [Mon Mar 9 23:10:52 2020] ceph: Can't lookup inode 1 (err: -13) [Mon Mar 9 23:12:03 2020] ceph: Can't lookup inode 1 (err: -13) [Mon Mar 9 23:13:12 2020] ceph: Can't lookup inode 1 (err: -13) [Mon Mar 9 23:14:19 2020] ceph: Can't lookup inode 1 (err: -13)

2 4

Nautilus cephfs usage
by Yoann Moulin 10 Mar '20

10 Mar '20

Hello, I have a Nautilus cluster with a cephfs volume, on grafana, it shows that cephfs_data pool is almost full[1] but if I give a look to the pool usage, it looks like I have plenty of space. Which metrics are used by grafana? 1. https://framapic.org/5r7J86s55x6k/jGSIsjEUPYMU.png pool usage: > artemis@icitsrv5:~$ ceph df detail > RAW STORAGE: > CLASS SIZE AVAIL USED RAW USED %RAW USED > hdd 662 TiB 296 TiB 366 TiB 366 TiB 55.32 > TOTAL 662 TiB 296 TiB 366 TiB 366 TiB 55.32 > > POOLS: > POOL ID STORED OBJECTS USED %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR > .rgw.root 3 8.1 KiB 15 2.8 MiB 0 63 TiB N/A N/A 15 0 B 0 B > default.rgw.control 4 0 B 8 0 B 0 63 TiB N/A N/A 8 0 B 0 B > default.rgw.meta 5 26 KiB 85 16 MiB 0 63 TiB N/A N/A 85 0 B 0 B > default.rgw.log 6 0 B 207 0 B 0 63 TiB N/A N/A 207 0 B 0 B > cephfs_data 7 113 TiB 139.34M 186 TiB 49.47 138 TiB N/A N/A 139.34M 0 B 0 B > cephfs_metadata 8 54 GiB 10.21M 57 GiB 0.03 63 TiB N/A N/A 10.21M 0 B 0 B > default.rgw.buckets.data 9 122 TiB 54.57M 173 TiB 47.70 138 TiB N/A N/A 54.57M 0 B 0 B > default.rgw.buckets.index 10 2.6 GiB 19.97k 2.6 GiB 0 63 TiB N/A N/A 19.97k 0 B 0 B > default.rgw.buckets.non-ec 11 67 MiB 186 102 MiB 0 63 TiB N/A N/A 186 0 B 0 B > device_health_metrics 12 1.2 MiB 145 1.2 MiB 0 63 TiB N/A N/A 145 0 B 0 B Best, -- Yoann Moulin EPFL IC-IT

1 0

Radosgw dynamic sharding jewel -> luminous
by Robert LeBlanc 10 Mar '20

10 Mar '20

I just upgraded a cluster that I inherited from Jewel to Luminous and trying to work through the new warnings/errors. I got the message about 3 OMAP objects being too big, all of them in the default.rgw.buckets.index pool. I expected that dynamic sharding should kick in, but no luck after several days. I looked at $ radosgw-admin reshard list [2020-03-02 04:27:22.303601 7f8bb58b8e40 -1 ERROR: failed to list reshard log entries, oid=reshard.0000000000 2020-03-02 04:27:22.305403 7f8bb58b8e40 -1 ERROR: failed to list reshard log entries, oid=reshard.0000000001 2020-03-02 04:27:22.307038 7f8bb58b8e40 -1 ERROR: failed to list reshard log entries, oid=reshard.0000000002 2020-03-02 04:27:22.317932 7f8bb58b8e40 -1 ERROR: failed to list reshard log entries, oid=reshard.0000000003 2020-03-02 04:27:22.348383 7f8bb58b8e40 -1 ERROR: failed to list reshard log entries, oid=reshard.0000000004 2020-03-02 04:27:22.349212 7f8bb58b8e40 -1 ERROR: failed to list reshard log entries, oid=reshard.0000000005 2020-03-02 04:27:22.349853 7f8bb58b8e40 -1 ERROR: failed to list reshard log entries, oid=reshard.0000000006 2020-03-02 04:27:22.350490 7f8bb58b8e40 -1 ERROR: failed to list reshard log entries, oid=reshard.0000000007 2020-03-02 04:27:22.351256 7f8bb58b8e40 -1 ERROR: failed to list reshard log entries, oid=reshard.0000000008 2020-03-02 04:27:22.351843 7f8bb58b8e40 -1 ERROR: failed to list reshard log entries, oid=reshard.0000000009 2020-03-02 04:27:22.353225 7f8bb58b8e40 -1 ERROR: failed to list reshard log entries, oid=reshard.0000000010 2020-03-02 04:27:22.353910 7f8bb58b8e40 -1 ERROR: failed to list reshard log entries, oid=reshard.0000000011 2020-03-02 04:27:22.367161 7f8bb58b8e40 -1 ERROR: failed to list reshard log entries, oid=reshard.0000000012 2020-03-02 04:27:22.367741 7f8bb58b8e40 -1 ERROR: failed to list reshard log entries, oid=reshard.0000000013 2020-03-02 04:27:22.368498 7f8bb58b8e40 -1 ERROR: failed to list reshard log entries, oid=reshard.0000000014 ] And searching on the Internet indicates tha Luminous added a new "reshard" namespace that the radosgw user needs access to. I'm not sure which pool this namespace was added to (being from Jewel there are a slew of rgw pools) and I'm not sure which rados-gw user it's talking about. I can't find a keyring for rados-admin, but it works so I assume that it is using the admin key ring. The permissions are open. I even appended the namespace option to the admin caps like follows: [client.admin] key = SECRET caps mds = "allow *" caps mon = "allow *" caps: [osd] allow * namespace="*" But I get a new error: $ radosgw-admin reshard list 2020-03-02 05:31:40.875642 7fdb983f7e40 0 failed reading realm info: ret -1 (1) Operation not permitted Any nudge in the right direction would be helpful. I manually sharded the indexes, but I'd really like to have it done automatically from now on so I don't have to worry about it. Thank you, Robert LeBlanc ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1

2 4

Monitors' election failed on VMs : e4 handle_auth_request failed to assign global_id
by Yoann Moulin 10 Mar '20

10 Mar '20

Hello, On a Nautilus cluster, I'd like to move monitors from bare metal servers to VMs to prepare a migration. I have added 3 new monitors on 3 VMs and I'd like to stop the 3 old monitors daemon. But I soon as I stop the 3rd old monitor, the cluster stuck because the election of a new monitor fails. The 3 old monitors are in 14.2.4-1xenial The 3 new monitors are in 14.2.7-1bionic > 2020-03-09 16:06:00.167 7fc4a3138700 1 mon.icvm0017(a)3(peon).paxos(paxos active c 20918592..20919120) lease_timeout -- calling new election > 2020-03-09 16:06:02.143 7fc49f931700 1 mon.icvm0017@3(probing) e4 handle_auth_request failed to assign global_id Did I miss something? In attachment : some logs and ceph.conf Thanks for your help. Best, -- Yoann Moulin EPFL IC-IT

3 3

Clear health warning
by Peter Eisch 10 Mar '20

10 Mar '20

Hi, (nautilus, 14.2.8, whole cluster) I doodled with adding a second cephfs and the project got canceled. I removed the unused cephfs with "ceph fs rm dream --yes-i-really-mean-it" and that worked as expected. I have a lingering health warning though which won't clear. The original cephfs1 volume exists and is healthy: [root@cephmon-03]# ceph fs ls name: cephfs1, metadata pool: stp.cephfs_metadata, data pools: [stp.cephfs_data ] [root@cephmon-03]# ceph mds stat cephfs1:3 {0=cephmon-03=up:active,1=cephmon-02=up:active,2=cephmon-01=up:active} [root@cephmon-03]# ceph health detail HEALTH_WARN insufficient standby MDS daemons available MDS_INSUFFICIENT_STANDBY insufficient standby MDS daemons available have 0; want 1 more [root@cephmon-03]# I have not yet deleted the pools for 'dream', the second cephfs definition. (There is nothing in it.) Before deleting the pools is there a command to clear this warning? Thanks! peter Peter Eisch Senior Site Reliability Engineer T1.612.445.5135 virginpulse.com |virginpulse.com/global-challenge Australia | Bosnia and Herzegovina | Brazil | Canada | Singapore | Switzerland | United Kingdom | USA Confidentiality Notice: The information contained in this e-mail, including any attachment(s), is intended solely for use by the designated recipient(s). Unauthorized use, dissemination, distribution, or reproduction of this message by anyone other than the intended recipient(s), or a person designated as responsible for delivering such messages to the intended recipient, is strictly prohibited and may be unlawful. This e-mail may contain proprietary, confidential or privileged information. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Virgin Pulse, Inc. If you have received this message in error, or are not the named recipient(s), please immediately notify the sender and delete this e-mail message. v2.64

2 4

Link to Nautilus upgrade
by Peter Eisch 09 Mar '20

09 Mar '20

Hi, When upgrading a cluster from Luminous to Nautilus I followed a page on ceph.com. I need to do another cluster and while I tagged the link, the page no longer exists. https://docs.ceph.com/master/releases/nautilus/#nautilus-old-upgrade Might anyone have either an updated link or point me to how to find the contents of best steps? Thanks, peter Peter Eisch Senior Site Reliability Engineer T1.612.445.5135 virginpulse.com |virginpulse.com/global-challenge Australia | Bosnia and Herzegovina | Brazil | Canada | Singapore | Switzerland | United Kingdom | USA Confidentiality Notice: The information contained in this e-mail, including any attachment(s), is intended solely for use by the designated recipient(s). Unauthorized use, dissemination, distribution, or reproduction of this message by anyone other than the intended recipient(s), or a person designated as responsible for delivering such messages to the intended recipient, is strictly prohibited and may be unlawful. This e-mail may contain proprietary, confidential or privileged information. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Virgin Pulse, Inc. If you have received this message in error, or are not the named recipient(s), please immediately notify the sender and delete this e-mail message. v2.64

2 2