- ceph-users - lists.ceph.io

v14.2.4 Nautilus released
by Abhishek Lekshmanan 17 Sep '19

17 Sep '19

This is the fourth release in the Ceph Nautilus stable release series. Its sole purpose is to fix a regression that found its way into the previous release. Notable Changes --------------- The ceph-volume in Nautilus v14.2.3 was found to contain a serious regression, described in https://tracker.ceph.com/issues/41660, which prevented deployment tools like ceph-ansible, DeepSea, Rook, etc. from deploying/removing OSDs. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-14.2.3.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: 75f4de193b3ea58512f204623e6c5a16e6c1e1ba -- Abhishek Lekshmanan SUSE Software Solutions Germany GmbH

1 1

osds xxx have blocked requests > 1048.58 sec / osd.yyy has stuck requests > 67108.9 sec
by Thomas 17 Sep '19

17 Sep '19

Hi, I have defined pool hdd which is exclusively used by virtual disks of multiple KVMs / LXCs. Yesterday I run these commands osdmaptool om --upmap out.txt --upmap-pool hdd source out.txt and Ceph started rebalancing this pool. However since then no KVM / LXC is reacting anymore. If I try to start a new KVM it hangs in boot process. This is the output of ceph health detail: root@ld3955:/mnt/rbd# ceph health detail HEALTH_ERR 28 nearfull osd(s); 1 pool(s) nearfull; Reduced data availability: 1 pg inactive, 1 pg peering; Degraded data redundancy (low space): 8 pgs backfill_toofull; 1 subtrees have overcommitted pool target_size_bytes; 1 subtrees have overcommitted pool target_size_ratio; 2 pools have too many placement groups; 672 slow requests are blocked > 32 sec; 4752 stuck requests are blocked > 4096 sec OSD_NEARFULL 28 nearfull osd(s) osd.42 is near full osd.44 is near full osd.45 is near full osd.77 is near full osd.84 is near full osd.94 is near full osd.101 is near full osd.103 is near full osd.106 is near full osd.109 is near full osd.113 is near full osd.118 is near full osd.120 is near full osd.136 is near full osd.138 is near full osd.142 is near full osd.147 is near full osd.156 is near full osd.159 is near full osd.161 is near full osd.168 is near full osd.192 is near full osd.202 is near full osd.206 is near full osd.208 is near full osd.226 is near full osd.234 is near full osd.247 is near full POOL_NEARFULL 1 pool(s) nearfull pool 'hdb_backup' is nearfull PG_AVAILABILITY Reduced data availability: 1 pg inactive, 1 pg peering pg 30.1b9 is stuck peering for 4722.750977, current state peering, last acting [183,27,63] PG_DEGRADED_FULL Degraded data redundancy (low space): 8 pgs backfill_toofull pg 11.465 is active+remapped+backfill_wait+backfill_toofull, acting [308,351,58] pg 11.5c4 is active+remapped+backfill_wait+backfill_toofull, acting [318,336,54] pg 11.afd is active+remapped+backfill_wait+backfill_toofull, acting [347,220,315] pg 11.b82 is active+remapped+backfill_toofull, acting [314,320,254] pg 11.1803 is active+remapped+backfill_wait+backfill_toofull, acting [88,363,302] pg 11.1aac is active+remapped+backfill_wait+backfill_toofull, acting [328,275,95] pg 11.1c09 is active+remapped+backfill_wait+backfill_toofull, acting [55,124,278] pg 11.1e36 is active+remapped+backfill_wait+backfill_toofull, acting [351,92,315] POOL_TARGET_SIZE_BYTES_OVERCOMMITTED 1 subtrees have overcommitted pool target_size_bytes Pools ['hdb_backup'] overcommit available storage by 1.708x due to target_size_bytes 0 on pools [] POOL_TARGET_SIZE_RATIO_OVERCOMMITTED 1 subtrees have overcommitted pool target_size_ratio Pools ['hdb_backup'] overcommit available storage by 1.708x due to target_size_ratio 0.000 on pools [] POOL_TOO_MANY_PGS 2 pools have too many placement groups Pool hdd has 512 placement groups, should have 128 Pool pve_cephfs_metadata has 32 placement groups, should have 4 REQUEST_SLOW 672 slow requests are blocked > 32 sec 249 ops are blocked > 2097.15 sec 284 ops are blocked > 1048.58 sec 108 ops are blocked > 524.288 sec 9 ops are blocked > 262.144 sec 22 ops are blocked > 131.072 sec osd.9 has blocked requests > 524.288 sec osds 0,2,6,68 have blocked requests > 1048.58 sec osd.3 has blocked requests > 2097.15 sec REQUEST_STUCK 4752 stuck requests are blocked > 4096 sec 1431 ops are blocked > 67108.9 sec 513 ops are blocked > 33554.4 sec 909 ops are blocked > 16777.2 sec 1809 ops are blocked > 8388.61 sec 90 ops are blocked > 4194.3 sec osd.63 has stuck requests > 67108.9 sec My interpretation is that Ceph a) is busy with remapping PGs of pool hdb_backup b) has identified several OSDs with either blocked or stuck requests. Any of these OSDs belongs to pool hdd, though. osd.9 belongs to node A, osd.63 and osd.68 belongs to node C (there are 4 nodes serving OSD in the cluster). I have tried to fix this issue, but it failed with - ceph osd set noout - restart of relevant OSD by systemctl restart ceph-osd@<id> and finally server reboot. I also tried to migrate the virtual disks to another pool, but this fails, too. There are no changes on server side, like network or disks or whatsoever. How can I resolve this issue? THX Thomas

2 1

upmap supported in SLES 12SPx
by Thomas 17 Sep '19

17 Sep '19

Hi, I tried to run this command with failure: root@ld3955:/mnt/rbd# ceph osd set-require-min-compat-client luminous Error EPERM: cannot set require_min_compat_client to luminous: 6 connected client(s) look like jewel (missing 0xa00000000200000); 19 connected client(s) look like jewel (missing 0x800000000000000); 23 connected client(s) look like jewel (missing 0x800000000000000); add --yes-i-really-mean-it to do it anyway root@ld3955:/mnt/rbd# ceph features { "mon": [ { "features": "0x3ffddff8ffacffff", "release": "luminous", "num": 3 } ], "mds": [ { "features": "0x3ffddff8ffacffff", "release": "luminous", "num": 2 } ], "osd": [ { "features": "0x3ffddff8ffacffff", "release": "luminous", "num": 368 } ], "client": [ { "features": "0x40106b84a842a42", "release": "jewel", "num": 6 }, { "features": "0x27018eb84aa42a52", "release": "jewel", "num": 19 }, { "features": "0x27018fb86aa42ada", "release": "jewel", "num": 23 }, { "features": "0x3ffddff8ffacffff", "release": "luminous", "num": 35 } ], "mgr": [ { "features": "0x3ffddff8ffacffff", "release": "luminous", "num": 3 } ] } But I think the number of clients older than Luminous is incorrect. I'm running 98% of the clients with SLES 12SPx, and therefore the question is: Can you confirm in which SLES 12 release the function upmap is supported? And is it safe to execute ceph osd set-require-min-compat-client luminous --yes-i-really-mean-it? What happens to the clients then? Regards Thomas

8 14

Re: download.ceph.com repository changes
by vitalif＠yourcmc.ru 17 Sep '19

17 Sep '19

The worst part about the official repository is that it lacks Debian packages Also of course it would be very convenient to be able to install any version from the repos, not just the latest one. It's certainly possible with debian repos...

1 0

Unsubscribe
by Rimma Iontel 17 Sep '19

17 Sep '19

2 1

14.2.4 Packages Avaliable
by Ashley Merrick 17 Sep '19

17 Sep '19

Have just noticed their is packages available for 14.2.4.. I know with the whole 14.2.3 release and the notes not going out to a good day or so later.. but this is not long after the 14.2.3 release..? Was this release even meant to have come out? Makes it difficult for people installing a new node if they can't reply on the current "stable" packages that apt/yum will give them.

10 12

New lines "choose_args" in crush map
by Thomas 17 Sep '19

17 Sep '19

Hi! I recently noticed new lines in my crush map. Here's an extract of it: # choose_args choose_args 18446744073709551615 { { bucket_id -1 weight_set [ [ 33.839 34.159 34.041 34.768 1.810 1.832 ] ] } { bucket_id -2 weight_set [ [ 28.019 28.339 28.221 28.948 1.810 1.832 ] ] } { bucket_id -3 weight_set [ [ 2.910 2.910 1.663 1.605 1.646 1.646 1.663 1.694 1.671 1.663 1.605 1.678 1.617 1.656 1.678 1.588 1.653 1.635 1.660 ] ] } { bucket_id -4 weight_set [ [ 1.663 1.605 1.646 1.646 1.663 1.694 1.671 1.663 1.605 1.678 1.617 1.656 1.678 1.588 1.653 1.635 1.660 ] ] } { bucket_id -5 weight_set [ [ 2.910 2.910 ] ] } { bucket_id -6 weight_set [ [ 5.820 5.820 5.820 5.820 0.000 0.000 ] ] } [...] Question: What is writing these lines into crush map? What is the function behind these lines? THX Thomas

1 0

KRBD use Luminous upmap feature.Which version of the kernel should i ues?
by 潘东元 17 Sep '19

17 Sep '19

hi, my ceph cluster version is Luminous run the kernel version Linux 3.10 [root@node-1 ~]# ceph features { "mon": { "group": { "features": "0x3ffddff8eeacfffb", "release": "luminous", "num": 3 } }, "osd": { "group": { "features": "0x3ffddff8eeacfffb", "release": "luminous", "num": 15 } }, "client": { "group": { "features": "0x40106b84a842a52", "release": "jewel", "num": 3 }, "group": { "features": "0x3ffddff8eeacfffb", "release": "luminous", "num": 170 } } } Which version of the kernel should i ues? Thanks!

3 5

Nautilus: pg_autoscaler causes mon slow ops
by Eugen Block 17 Sep '19

17 Sep '19

Hi everyone, we upgraded our production cluster just recently to version: ceph version 14.2.3-349-g7b1552ea82 (7b1552ea827cf5167b6edbba96dd1c4a9dc16937) nautilus (stable) We then activated pg_autoscaler for two pools that had a bad pg_num and the result is satisfying. However, after the rebalance finished the cluster became laggy. We noticed that two out of three MONs had a much higher CPU usage than usual, according to `top` the MON processes consumed more than 100%. Restarting the MON services and disabling pg_autoscaler resolved the issue. I've read that the balancer module can cause a higher load on the MGR daemon, is this somehow related? Another thing to mention is the confusing calculation of the autoscaler. After the pg numbers had been corrected we got the warning about overcommitted pools: > 1 subtrees have overcommitted pool target_size_bytes > 1 subtrees have overcommitted pool target_size_ratio The images pool was responsible for that. The confusing part was that sometimes autoscale-status displayed the size of that pool with more than 14 TB: ceph osd pool autoscale-status POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE images 14399G 3.0 33713G 1.2813 1.0 128 on And a couple of minutes later the pool suddenly only had around 4 TB of data: ceph osd pool autoscale-status POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE images 4112G 3.0 33713G 0.3659 1.0 128 on There seems to be some kind of inconsistency here. The actual used storage of this pool according to `ceph df` is: POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL images 1 4.1 TiB 1.01M 12 TiB 49.73 4.1 TiB Has anyone experienced something similar? Are these known issues? Regards, Eugen

1 0

Intergrate Metadata with ElasticSeach
by tuan dung 17 Sep '19

17 Sep '19

Dear all, Can you show me steps how to intergrate Metadata of Ceph object with ElasticSeach to improve searching medata performance? thank you very much. ----------------------------- Br, Dương Tuấn Dũng 0986153686

1 0