- ceph-users - lists.ceph.io

ERROR: osd init failed: (13) Permission denied
by Marc Roos 02 Dec '19

02 Dec '19

Node does not start osds! Why do I have this error? Previous boot was just fine (upgraded recently to nautilus)

2 3

Not able to create and remove snapshots
by Marc Roos 30 Nov '19

30 Nov '19

I am not sure since when, but I am not able to create nor delete snapshots anymore. I am getting a permission denied. I upgraded recently from Luminous to Nautilus and set this allow_new_snaps as mentioned on[1] [@ .snap]# ls snap-1 snap-2 snap-3 snap-4 snap-5 snap-6 snap-7 [@ .snap]# rmdir snap-7 rmdir: failed to remove ‘snap-7’: Permission denied [@ .snap]# mkdir snap-8 mkdir: cannot create directory ‘snap-8’: Permission denied [1] https://docs.ceph.com/docs/nautilus/dev/cephfs-snapshots/

1 1

ceph auth
by Marc Roos 30 Nov '19

30 Nov '19

Is there a ceph auth command that just list all clients? Without dumping keys to the console. I would recommend making 'ceph auth ls' just display client names/ids. If you want the key, there are already other commands.

1 2

Questions about the EC pool
by majia xiao 29 Nov '19

29 Nov '19

Hello, We have a Ceph cluster (version 12.2.4) with 10 hosts, and there are 21 OSDs on each host. An EC pool is created with the following commands: ceph osd erasure-code-profile set profile_jerasure_4_3_reed_sol_van \ plugin=jerasure \ k=4 \ m=3 \ technique=reed_sol_van \ packetsize=2048 \ crush-device-class=hdd \ crush-failure-domain=host ceph osd pool create pool_jerasure_4_3_reed_sol_van 2048 2048 erasure profile_jerasure_4_3_reed_sol_van Here are my questions: 1. The EC pool is created using k=4, m=3, and crush-device-class=hdd, so we just disable the network interfaces of some hosts (using "ifdown" command) to verify the functionality of the EC pool while performing ‘rados bench’ command. However, the IO rate drops immediately to 0 when a single host goes offline, and it takes a long time (~100 seconds) for the IO rate becoming normal. As far as I know, the default value of min_size is k+1 or 5, which means that the EC pool can be still working even if there are two hosts offline. Is there something wrong with my understanding? 2. According to our observations, it seems that the IO rate becomes normal when Ceph detects all OSDs corresponding to the failed host. Is there any way to reduce the time needed for Ceph to detect all failed OSDs? Thanks for any help. Best regards, Majia Xiao

2 1

Re: mimic 13.2.6 too much broken connexions
by Vincent Godin 29 Nov '19

29 Nov '19

If it was a network issue, the counters should explose (as i said, with a log level of 5 on the messenger, we observed more then 80 000 lossy channels per minute) but nothing abnormal is relevant on the counters (on switchs and servers) On the switchs no drop, no crc error, no packet loss, only some output discards but not enough to be significant. On the NICs on the servers via ethtool -S, nothing is relevant. And as i said, an other mimic cluster with different hardware has the same behavior Ceph uses connexions pools from host to host but how does it check the availability of these connexions over the time ? And as the network doesn't seem to be guilty, what can explain these broken channels ? Le mer. 27 nov. 2019 à 19:05, Anthony D'Atri <aad(a)dreamsnake.net> a écrit : > > Are you bonding NIC ports? If so do you have the correct hash policy defined? Have you looked at the *switch* side for packet loss, CRC errors, etc? What you report could be consistent with this. Since the host interface for a given connection will vary by the bond hash, some OSD connections will use one port and some the other. So if one port has switch side errors, or is blackholed on the switch, you could see some heart beating impacted but not others. > > Also make sure you have the optimal reporters value. > > > On Nov 27, 2019, at 7:31 AM, Vincent Godin <vince.mlist(a)gmail.com> wrote: > > > > Till i submit the mail below few days ago, we found some clues > > We observed a lot of lossy connexion like : > > ceph-osd.9.log:2019-11-27 11:03:49.369 7f6bb77d0700 0 -- > > 192.168.4.181:6818/2281415 >> 192.168.4.41:0/1962809518 > > conn(0x563979a9f600 :6818 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH > > pgs=0 cs=0 l=1).handle_connect_msg accept replacing existing (lossy) > > channel (new one lossy=1) > > We raised the log of the messenger to 5/5 and observed for the whole > > cluster more than 80 000 lossy connexion per minute !!! > > We adjusted the "ms_tcp_read_timeout" from 900 to 60 sec then no more > > lossy connexion in logs nor health check failed > > It's just a workaround but there is a real problem with these broken > > sessions and it leads to two > > assertions : > > - Ceph take too much time to detect broken session and should recycle quicker ! > > - The reasons for these broken sessions ? > > > > We have a other mimic cluster on different hardware and observed the > > same behavior : lot of lossy sessions, slow ops and co. > > Symptoms are the same : > > - some OSDs on one host have no response from an other osd on a different hosts > > - after some time, slow ops are detected > > - sometime it leads to ioblocked > > - after about 15mn the problem vanish > > > > ----------- > > > > Help on diag needed : heartbeat_failed > > > > We encounter a strange behavior on our Mimic 13.2.6 cluster. A any > > time, and without any load, some OSDs become unreachable from only > > some hosts. It last 10 mn and then the problem vanish. > > It 's not always the same OSDs and the same hosts. There is no network > > failure on any of the host (because only some OSDs become unreachable) > > nor disk freeze as we can see in our grafana dashboard. Logs message > > are : > > first msg : > > 2019-11-24 09:19:43.292 7fa9980fc700 -1 osd.596 146481 > > heartbeat_check: no reply from 192.168.6.112:6817 osd.394 since back > > 2019-11-24 09:19:22.761142 front 2019-11-24 09:19:39.769138 (cutoff > > 2019-11-24 09:19:23.293436) > > last msg: > > 2019-11-24 09:30:33.735 7f632354f700 -1 osd.591 146481 > > heartbeat_check: no reply from 192.168.6.123:6828 osd.600 since back > > 2019-11-24 09:27:05.269330 front 2019-11-24 09:30:33.214874 (cutoff > > 2019-11-24 09:30:13.736517) > > During this time, 3 hosts were involved : host-18, host-20 and host-30 : > > host-30 is the only one who can't see osds 346,356,and 352 on host-18 > > host-30 is the only one who can't see osds 387 and 394 on host-20 > > host-18 is the only one who can't see osds 583, 585, 591 and 597 on host-30 > > We can't see any strange behavior on hosts 18, 20 and 30 in our node > > exporter data during this time > > Any ideas or advices ?

2 2

How to set size for CephFs
by Alokkumar Mahajan 28 Nov '19

28 Nov '19

Hello, I am new to Ceph and currently i am working on setting up CephFs and RBD environment. I have successfully setup Ceph Cluster with 4 OSD's (2 OSD's with size 50GB and 2 OSD's with size 300GB). But while setting up CephFs the size which i see allocated for CephFs Data and metadata pools is 55GB. But i want to have 300GB assigned for CephFs. I tried using "target_size_bytes" flag while creating pool but it is not working (it saus invalid command). Same result when i use target_size_bytes with (ceph osd pool set) after creating pool. I am not sure if i am doing something silly here. Can someone please guide me on this? Thanks in adv.!

3 5

Re: cephfs worm feature
by j j 28 Nov '19

28 Nov '19

Thanks for the information, I'll take a look at this pr and think it over. Jeff Layton <jlayton(a)redhat.com> 于2019年11月27日周三下午6:50写道： > On Wed, 2019-11-27 at 15:14 +0800, j j wrote: > > Hi all, > > > > Recently I encountered a situation requires reliable file storage > with cephfs, and the point is those data is not allowed to get modified or > deleted. > > After some learning I found that the WORM(write once read many) feature > is exactly what I need.Unfortunately, as far as I know, there is no worm > > feature in cephfs. > > So I was wondering is there any plan or design about this feature? > > > > Thanks. > > There's a pull request for this that has been stalled since spring: > > https://github.com/ceph/ceph/pull/26691 > > Personally, I don't see how we can get away with making file data 100% > immutable. We'll need to allow _some_ entity to un-WORM the thing, and > it was never clear to me how that would work. > -- > Jeff Layton <jlayton(a)redhat.com> > >

1 0

mimic 13.2.6 too much broken connexions
by Vincent Godin 27 Nov '19

27 Nov '19

Till i submit the mail below few days ago, we found some clues We observed a lot of lossy connexion like : ceph-osd.9.log:2019-11-27 11:03:49.369 7f6bb77d0700 0 -- 192.168.4.181:6818/2281415 >> 192.168.4.41:0/1962809518 conn(0x563979a9f600 :6818 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg accept replacing existing (lossy) channel (new one lossy=1) We raised the log of the messenger to 5/5 and observed for the whole cluster more than 80 000 lossy connexion per minute !!! We adjusted the "ms_tcp_read_timeout" from 900 to 60 sec then no more lossy connexion in logs nor health check failed It's just a workaround but there is a real problem with these broken sessions and it leads to two assertions : - Ceph take too much time to detect broken session and should recycle quicker ! - The reasons for these broken sessions ? We have a other mimic cluster on different hardware and observed the same behavior : lot of lossy sessions, slow ops and co. Symptoms are the same : - some OSDs on one host have no response from an other osd on a different hosts - after some time, slow ops are detected - sometime it leads to ioblocked - after about 15mn the problem vanish ----------- Help on diag needed : heartbeat_failed We encounter a strange behavior on our Mimic 13.2.6 cluster. A any time, and without any load, some OSDs become unreachable from only some hosts. It last 10 mn and then the problem vanish. It 's not always the same OSDs and the same hosts. There is no network failure on any of the host (because only some OSDs become unreachable) nor disk freeze as we can see in our grafana dashboard. Logs message are : first msg : 2019-11-24 09:19:43.292 7fa9980fc700 -1 osd.596 146481 heartbeat_check: no reply from 192.168.6.112:6817 osd.394 since back 2019-11-24 09:19:22.761142 front 2019-11-24 09:19:39.769138 (cutoff 2019-11-24 09:19:23.293436) last msg: 2019-11-24 09:30:33.735 7f632354f700 -1 osd.591 146481 heartbeat_check: no reply from 192.168.6.123:6828 osd.600 since back 2019-11-24 09:27:05.269330 front 2019-11-24 09:30:33.214874 (cutoff 2019-11-24 09:30:13.736517) During this time, 3 hosts were involved : host-18, host-20 and host-30 : host-30 is the only one who can't see osds 346,356,and 352 on host-18 host-30 is the only one who can't see osds 387 and 394 on host-20 host-18 is the only one who can't see osds 583, 585, 591 and 597 on host-30 We can't see any strange behavior on hosts 18, 20 and 30 in our node exporter data during this time Any ideas or advices ?

1 0

EC PGs stuck activating, 2^31-1 as OSD ID, automatic recovery not kicking in
by Zoltan Arnold Nagy 27 Nov '19

27 Nov '19

Hi, We have a cluster where we mix HDDs and NVMe drives using device classes with a specific crush role for each class. One of our NVMe drives physically died which caused some of our PGs to go into this state: pg 26.ac is stuck undersized for 60830.991784, current state activating+undersized+degraded+remapped, last acting [353,373,368,377,2147483647,350] pg 26.d1 is stuck undersized for 60830.587711, current state activating+undersized+degraded+remapped, last acting [343,2147483647,347,358,366,355] pg 26.e1 is stuck undersized for 60830.980585, current state activating+undersized+degraded+remapped, last acting [340,349,370,2147483647,360,376] ... and so on. Recovery never happened and we had to manually restart all affected OSDs for all PGs stuck in such a state. The 2^31-1 in there seems to indicate an overflow somewhere - the way we were able to figure out where exactly is to query the PG and compare the "up" and "acting" sets - only _one_ of them had the 2^31-1 number in place of the correct OSD number. We restarted that and the PG started doing its job and recovered. The issue seems to be going back to 2015: http://lists.ceph.com/pipermail/ceph-users-ceph.com/2015-May/001661.html however no solution... I'm more concerned about the cluster not being able to recover (it's a 4+2 EC pool across 12 hosts - plenty of room to heal) than about the weird print-out. The VMs who wanted to access data in any of the affected PGs of course died. Are we missing some settings to let the cluster self-heal even for EC pools? First EC pool in production :) Cheers, Zoltan

4 4

EC pool used space high
by Erdem Agaoglu 27 Nov '19

27 Nov '19

Hi, I have a rook-provisioned cluster to be used for RBDs only. I have 2 pools named replicated-metadata-pool and ec-data-pool. EC parameters are 6+3. I've been writing some data to this cluster for some time and noticed that the reported usage is not what I was expecting. # ceph df RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 5.4 PiB 4.3 PiB 1.2 PiB 1.2 PiB 21.77 TOTAL 5.4 PiB 4.3 PiB 1.2 PiB 1.2 PiB 21.77 POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL replicated-metadata-pool 1 90 KiB 408 38 MiB 0 1.2 PiB ec-data-pool 2 722 TiB 191.64M 1.2 PiB 25.04 2.4 PiB Since these numbers are rounded a bit too much, I generally use prometheus metrics on mgr, which are as follows: ceph_pool_stored : 793,746 G for ec-data-pool and 92323 for replicated-metadata-pool ceph_pool_stored_raw: 1,190,865 G for ec-data-pool and 99213 for replicated-metadata-pool ceph_cluster_total_used_bytes: 1,329,374 G ceph_cluster_total_used_raw_bytes: 1,333,013 G sum(ceph_bluefs_db_used_bytes) : 3,638 G So ceph_pool_stored for the EC pool is a bit higher than the total used space of the formatted RBDs. I think that's because of the sparse nature and deleted blocks not being fstrimmed yet. That's OK. ceph_pool_stored_raw is almost exactly 1.5x ceph_pool_stored which is what I'd expect considering EC parameters of 6+3. What I can't find is the 138,509 G difference between the ceph_cluster_total_used_bytes and ceph_pool_stored_raw. This is not static BTW, checking the same data historically shows we have about 1.12x of what we expect. This seems to make our 1.5x EC overhead a 1.68x overhead in reality. Anyone have any ideas for why this is the case? We also have a ceph_cluster_total_used_raw_bytes metric, I believe to be close to data+metadata. Which is why I tried to show with sum(ceph_bluefs_db_used_bytes). Is that correct? Best, -- erdem agaoglu

3 5