October 2019 - ceph-users

by fleg＠lpnhe.in2p3.fr

Hi, We have a ceph+cephfs cluster runing nautilus version 14.2.4 We have debian buster/ubuntu bionic clients mounting cephfs in kernel mode without problems. We now want to mount cephfs from our new centos 8 clients. Unfortunately, ceph-common is needed but there are no packages available for el8 (only el7). And no way to install the el7 packages on centos 8 (missing deps). Thus, despite the fact that centos 8 have a 4.18 kernel (required to use quota, snapshots etc...), it seems impossible to mount in kernel mode (good perfs) and we still have to use the so slow fuse mode. Is it possible to workaround this problem ? Or when is it planned to provides (even as beta) the ceph packages for centos 8 ? Thanks.

4 years, 4 months

4
5
0 0

Past_interval start interval mismatch (last_clean_epoch reported)

by Huseyin Cotuk

Hi all, I also hit the bug #24866 in my test environment. According to the logs, the last_clean_epoch in the specified OSD/PG is 17703, but the interval starts with 17895. So the OSD fails to start. There are some other OSDs in the same status. 2019-10-14 18:22:51.908 7f0a275f1700 -1 osd.21 pg_epoch: 18432 pg[18.51( v 18388'4 lc 18386'3 (0'0,18388'4] local-lis/les=18430/18431 n=1 ec=295/295 lis/c 18430/17702 les/c/f 18431/17703/0 18428/18430/18421) [11,21]/[11,21,20] r=1 lpr=18431 pi=[17895,18430)/3 crt=18388'4 lcod 0'0 unknown m=1 mbc={}] 18.51 past_intervals [17895,18430) start interval does not contain the required bound [17703,18430) start The cause is pg 18.51 went clean in 17703 but 17895 is reported to the monitor. I am using the last stable version of Mimic (13.2.6). Any idea how to fix it? Is there any way to bypass this check or fix the reported epoch #? Thanks in advance. Best regards, Huseyin Cotuk hcotuk(a)gmail.com

4 years, 5 months

3
2
0 0

Re: Wrong %USED and MAX AVAIL stats for pool

by Yordan Yordanov (Innologica)

The documentation states: https://docs.ceph.com/docs/mimic/rados/operations/monitoring/ The POOLS section of the output provides a list of pools and the notional usage of each pool. The output from this section DOES NOT reflect replicas, clones or snapshots. For example, if you store an object with 1MB of data, the notional usage will be 1MB, but the actual usage may be 2MB or more depending on the number of replicas, clones and snapshots. However in our case we are clearly seeing the USAGE field multiplying the total object sizes to the number of replicas. [root@blackmirror ~]# ceph df RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 80 TiB 34 TiB 46 TiB 46 TiB 58.10 TOTAL 80 TiB 34 TiB 46 TiB 46 TiB 58.10 POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL one 2 15 TiB 4.05M 46 TiB 68.32 7.2 TiB bench 5 250 MiB 67 250 MiB 0 22 TiB [root@blackmirror ~]# rbd du -p one NAME PROVISIONED USED ... <TOTAL> 20 TiB 15 TiB This is causing several apps (including ceph dashboard) to display inaccurate percentages, because they calculate the total pool capacity as USED + MAX AVAIL, which in this case yields 53.2TB, which is way off. 7.2TB is about 13% of that, so we receive alarms and this is bugging us for quite some time now.

4 years, 5 months

4
4
0 0

iSCSI write performance

by Ryan

I'm in the process of testing the iscsi target feature of ceph. The cluster is running ceph 14.2.4 and ceph-iscsi 3.3. It consists of 5 hosts with 12 SSD OSDs per host. Some basic testing moving VMs to a ceph backed datastore is only showing 60MB/s transfers. However moving these back off the datastore is fast at 200-300MB/s. What should I be looking at to track down the write performance issue? In comparison with the Nimble Storage arrays I can see 200-300MB/s in both directions. Thanks, Ryan

4 years, 5 months

8
20
0 0

Ceph Health error right after starting balancer

by Thomas Schneider

Hi, after enabling ceph balancer (with command ceph balancer on) the health status changed to error. This is the current output of ceph health detail: root@ld3955:~# ceph health detail HEALTH_ERR 1438 slow requests are blocked > 32 sec; 861 stuck requests are blocked > 4096 sec; mon ld5505 is low on available space REQUEST_SLOW 1438 slow requests are blocked > 32 sec 683 ops are blocked > 2097.15 sec 436 ops are blocked > 1048.58 sec 191 ops are blocked > 524.288 sec 78 ops are blocked > 262.144 sec 35 ops are blocked > 131.072 sec 11 ops are blocked > 65.536 sec 4 ops are blocked > 32.768 sec osd.62 has blocked requests > 65.536 sec osds 39,72 have blocked requests > 262.144 sec osds 6,19,67,173,174,187,188,269,434 have blocked requests > 524.288 sec osds 8,16,35,36,37,61,63,64,68,73,75,178,186,271,369,420,429,431,433,436 have blocked requests > 1048.58 sec osds 3,5,7,24,34,38,40,41,59,66,69,74,180,270,370,421,432,435 have blocked requests > 2097.15 sec REQUEST_STUCK 861 stuck requests are blocked > 4096 sec 25 ops are blocked > 8388.61 sec 836 ops are blocked > 4194.3 sec osds 2,28,29,32,60,65,181,185,268,368,423,424,426 have stuck requests > 4194.3 sec osds 0,30,70,71,184 have stuck requests > 8388.61 sec I understand that when balancer starts shifting PGs to other OSDs that this caused IO load on the cluster. However I don't understand why this is affecting OSD so heavily. And I don't understand why OSD of specific type (SSD, NVME) suffer although there's no balancing occuring on them. Regards Thomas

4 years, 5 months

3
4
0 0

subtrees have overcommitted (target_size_bytes / target_size_ratio)

by Lars Täuber

Hello everybody! What does this mean? health: HEALTH_WARN 1 subtrees have overcommitted pool target_size_bytes 1 subtrees have overcommitted pool target_size_ratio and what does it have to do with the autoscaler? When I deactivate the autoscaler the warning goes away. $ ceph osd pool autoscale-status POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE cephfs_metadata 15106M 3.0 2454G 0.0180 0.3000 4.0 256 on cephfs_data 113.6T 1.5 165.4T 1.0306 0.9000 1.0 512 on $ ceph health detail HEALTH_WARN 1 subtrees have overcommitted pool target_size_bytes; 1 subtrees have overcommitted pool target_size_ratio POOL_TARGET_SIZE_BYTES_OVERCOMMITTED 1 subtrees have overcommitted pool target_size_bytes Pools ['cephfs_data'] overcommit available storage by 1.031x due to target_size_bytes 0 on pools [] POOL_TARGET_SIZE_RATIO_OVERCOMMITTED 1 subtrees have overcommitted pool target_size_ratio Pools ['cephfs_data'] overcommit available storage by 1.031x due to target_size_ratio 0.900 on pools ['cephfs_data'] Thanks Lars

4 years, 5 months

3
10
0 0

V/v Multiple pool for data in Ceph object

by tuan dung

hi ceph-users, i have a cluster run ceph object using version 14.2.1. I want to creat 2 pool for bucket data for purposes for security: + one bucket-data pool for public client access from internet (name *zone1.rgw.buckets.data-pub) * + one bucket-data pool for private client access from local network (name *zone1.rgw.buckets.data-pub)* each pool bucket-data has one individual access key: access key public (access pool public) and access key private (access pool private). Can you give me a recomment for this or bestpractice that you've done? what needs to be done? Or give me your best solution for securiy a cluster ceph object with public client access and private client access? Thank you very much Br, ---------------------------------------------- Dương Tuấn Dũng Email: dungdt.aicgroup(a)gmail.com ĐT: 0986153686

4 years, 5 months

2
2
0 0

ceph pg dump hangs on mons w/o mgr

by Frank R

I have a 12.2.12 cluster with 3 mons where mgr will be active on 1. I have noticed that the command "ceph pg dump" hangs on all mons except the one where the mgr is running. "ceph pg dump" also runs fine on osd nodes. Is this expected behavior? thx Frank

4 years, 5 months

1
0
0 0

Error in MGR Log: auth: could not find secret_id=<number>

by Thomas Schneider

Hello, I noticed several error messages in MGR (active) log: 2019-10-31 10:40:16.341 7ff9edd64700 0 auth: could not find secret_id=3714 2019-10-31 10:40:16.341 7ff9edd64700 0 cephx: verify_authorizer could not get service secret for service mgr secret_id=3714 2019-10-31 10:40:16.541 7ff9ecd62700 0 auth: could not find secret_id=3714 2019-10-31 10:40:16.541 7ff9ecd62700 0 cephx: verify_authorizer could not get service secret for service mgr secret_id=3714 2019-10-31 10:40:16.941 7ff9ecd62700 0 auth: could not find secret_id=3714 2019-10-31 10:40:16.941 7ff9ecd62700 0 cephx: verify_authorizer could not get service secret for service mgr secret_id=3714 2019-10-31 10:40:17.741 7ff9ecd62700 0 auth: could not find secret_id=3714 2019-10-31 10:40:17.741 7ff9ecd62700 0 cephx: verify_authorizer could not get service secret for service mgr secret_id=3714 These error message are logged every 5 sec. Can you please advise how to troubleshoot this issue? THX

4 years, 5 months

1
0
0 0

Splitting PGs not happening on Nautilus 14.2.2

by Bryan Stillwell

This morning I noticed that on a new cluster the number of PGs for the default.rgw.buckets.data pool was way too small (just 8 PGs), but when I try to split the PGs the cluster doesn't do anything: # ceph osd pool set default.rgw.buckets.data pg_num 16 set pool 13 pg_num to 16 It seems to set the pg_num_target/pgp_num_target to 16, but pg_num/pgp_num never increase: # ceph osd dump | grep default.rgw.buckets.data pool 13 'default.rgw.buckets.data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 pg_num_target 16 pgp_num_target 16 autoscale_mode warn last_change 43217 flags hashpspool,creating stripe_width 0 compression_mode aggressive application rgw The clusters has 295 OSDs, so I need to do a bit of splitting today. Any ideas why the splits aren't starting? Thanks, Bryan

4 years, 5 months

1
1
0 0

2024

2023

2022

2021

2020

2019

ceph-users October 2019