January 2020 - ceph-users

Benchmark results for Seagate Exos2X14 Dual Actuator HDDs
by Paul Emmerich 16 Jan '20

16 Jan '20

Hi, we ran some benchmarks with a few samples of Seagate's new HDDs that some of you might find interesting: Blog post: https://croit.io/2020/01/06/2020-01-06-benchmark-mach2 GitHub repo with scripts and raw data: https://github.com/croit/benchmarks/tree/master/mach2-disks Tl;dr: way faster for writes, somewhat faster for reads in some scenarios Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io Looking for Ceph training? We have some free spots available https://croit.io/training/4-days-ceph-in-depth-training croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90

5 6

High CPU usage by ceph-mgr in 14.2.5
by Bryan Stillwell 16 Jan '20

16 Jan '20

After upgrading one of our clusters from Nautilus 14.2.2 to Nautilus 14.2.5 I'm seeing 100% CPU usage by a single ceph-mgr thread (found using 'top -H'). Attaching to the thread with strace shows a lot of mmap and munmap calls. Here's the distribution after watching it for a few minutes: 48.73% - mmap 49.48% - munmap 1.75% - futex 0.05% - madvise I've upgraded 3 other clusters so far (120 OSDs, 30 OSDs, 200 OSDs), but this is the only one which has seen the problem (355 OSDs). Perhaps it has something to do with its size? I was suspecting it might have to do with one of the modules misbehaving, so I disabled all of them: # ceph mgr module ls | jq -r '.enabled_modules' [] But that didn't help (I restarted the mgrs after disabling the modules too). I also tried setting debug_mgr and debug_mgrc to 20, but nothing popped out at me as being the cause of the problem. It only seems to affect the active mgr. If I stop the active mgr the problem moves to one of the other mgrs. Any guesses or tips on what next steps I should take to figure out what's going on? Thanks, Bryan

16 24

Uneven Node utilization
by Sasha Litvak 16 Jan '20

16 Jan '20

Hello, Cephers, I have a small 6 node cluster with 36 OSDs. When running the benchmark/torture tests I noticed that some nodes, usually storage2n6-la and also sometimes others are utilized much more. I see some osds are used 100% and load average goes up to 21 while on the others the load average is 5 - 6 and osds are within 40 - 50 - 60% of utilization. I cannot use upmap mode for balancer because I still have some client machines using hammer. I wonder if my issue is caused by compat balancing mode as compat weight shows nodes with the same number and disk size but different compat weights. If so what can I do to improve the load/disk usage distribution in the cluster? Also, my legacy client machines only need to access cephfs on the new cluster, so I wonder if keeping hammer as the oldest client version makes sense and I should change it to jewel and set crush tunables to optimal. Help is greatly appreciated, ceph df RAW STORAGE:. CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 94 TiB 88 TiB 5.9 TiB 5.9 TiB 6.29 TOTAL 94 TiB 88 TiB 5.9 TiB 5.9 TiB 6.29 POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL cephfs_data 1 1.6 TiB 3.77M 4.9 TiB 5.57 28 TiB cephfs_metadata 2 3.9 GiB 367.34k 4.3 GiB 0 28 TiB one 5 344 GiB 90.94k 1.0 TiB 1.20 28 TiB ceph -s cluster: id: 9b4468b7-5bf2-4964-8aec-4b2f4bee87ad health: HEALTH_OK services: mon: 3 daemons, quorum storage2n1-la,storage2n2-la,storage2n3-la (age 39h) mgr: storage2n1-la(active, since 39h), standbys: storage2n2-la, storage2n3-la mds: cephfs:1 {0=storage2n4-la=up:active} 1 up:standby-replay 1 up:standby osd: 36 osds: 36 up (since 37h), 36 in (since 10w) data: pools: 3 pools, 1664 pgs objects: 4.23M objects, 1.9 TiB usage: 5.9 TiB used, 88 TiB / 94 TiB avail pgs: 1664 active+clean io: client: 1.2 KiB/s rd, 46 KiB/s wr, 5 op/s rd, 2 op/s wr Ceph osd df looks like this ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 6 ssd 1.74609 1.00000 1.7 TiB 115 GiB 114 GiB 186 MiB 838 MiB 1.6 TiB 6.45 1.02 92 up 12 ssd 1.74609 1.00000 1.7 TiB 122 GiB 121 GiB 90 MiB 934 MiB 1.6 TiB 6.81 1.08 92 up 18 ssd 1.74609 1.00000 1.7 TiB 112 GiB 111 GiB 107 MiB 917 MiB 1.6 TiB 6.24 0.99 91 up 24 ssd 3.49219 1.00000 3.5 TiB 233 GiB 232 GiB 206 MiB 818 MiB 3.3 TiB 6.53 1.04 185 up 30 ssd 3.49219 1.00000 3.5 TiB 224 GiB 223 GiB 246 MiB 778 MiB 3.3 TiB 6.25 0.99 187 up 35 ssd 3.49219 1.00000 3.5 TiB 216 GiB 215 GiB 252 MiB 772 MiB 3.3 TiB 6.04 0.96 184 up 5 ssd 1.74609 1.00000 1.7 TiB 112 GiB 111 GiB 88 MiB 936 MiB 1.6 TiB 6.28 1.00 92 up 11 ssd 1.74609 1.00000 1.7 TiB 112 GiB 111 GiB 112 MiB 912 MiB 1.6 TiB 6.26 0.99 92 up 17 ssd 1.74609 1.00000 1.7 TiB 112 GiB 111 GiB 274 MiB 750 MiB 1.6 TiB 6.25 0.99 94 up 23 ssd 3.49219 1.00000 3.5 TiB 234 GiB 233 GiB 192 MiB 832 MiB 3.3 TiB 6.54 1.04 183 up 29 ssd 3.49219 1.00000 3.5 TiB 216 GiB 215 GiB 356 MiB 668 MiB 3.3 TiB 6.03 0.96 184 up 34 ssd 3.49219 1.00000 3.5 TiB 227 GiB 226 GiB 267 MiB 757 MiB 3.3 TiB 6.34 1.01 184 up 4 ssd 1.74609 1.00000 1.7 TiB 125 GiB 124 GiB 16 MiB 1008 MiB 1.6 TiB 7.00 1.11 94 up 10 ssd 1.74609 1.00000 1.7 TiB 108 GiB 107 GiB 163 MiB 861 MiB 1.6 TiB 6.01 0.96 93 up 16 ssd 1.74609 1.00000 1.7 TiB 107 GiB 106 GiB 163 MiB 861 MiB 1.6 TiB 6.00 0.95 94 up 22 ssd 3.49219 1.00000 3.5 TiB 221 GiB 220 GiB 385 MiB 700 MiB 3.3 TiB 6.18 0.98 187 up 28 ssd 3.49219 1.00000 3.5 TiB 223 GiB 222 GiB 257 MiB 767 MiB 3.3 TiB 6.23 0.99 186 up 33 ssd 3.49219 1.00000 3.5 TiB 241 GiB 240 GiB 233 MiB 791 MiB 3.3 TiB 6.74 1.07 185 up 1 ssd 1.74609 1.00000 1.7 TiB 103 GiB 102 GiB 240 MiB 784 MiB 1.6 TiB 5.76 0.92 93 up 7 ssd 1.74609 1.00000 1.7 TiB 117 GiB 116 GiB 70 MiB 954 MiB 1.6 TiB 6.56 1.04 91 up 13 ssd 1.74609 1.00000 1.7 TiB 126 GiB 125 GiB 76 MiB 948 MiB 1.6 TiB 7.03 1.12 95 up 19 ssd 3.49219 1.00000 3.5 TiB 230 GiB 229 GiB 307 MiB 717 MiB 3.3 TiB 6.44 1.02 186 up 25 ssd 3.49219 1.00000 3.5 TiB 220 GiB 219 GiB 309 MiB 715 MiB 3.3 TiB 6.15 0.98 185 up 31 ssd 3.49219 1.00000 3.5 TiB 223 GiB 222 GiB 205 MiB 819 MiB 3.3 TiB 6.23 0.99 186 up 0 ssd 1.74609 1.00000 1.7 TiB 116 GiB 115 GiB 151 MiB 873 MiB 1.6 TiB 6.49 1.03 93 up 3 ssd 1.74609 1.00000 1.7 TiB 121 GiB 120 GiB 89 MiB 935 MiB 1.6 TiB 6.77 1.08 91 up 9 ssd 1.74609 1.00000 1.7 TiB 104 GiB 103 GiB 183 MiB 841 MiB 1.6 TiB 5.81 0.92 93 up 15 ssd 3.49219 1.00000 3.5 TiB 222 GiB 221 GiB 205 MiB 819 MiB 3.3 TiB 6.20 0.98 185 up 21 ssd 3.49219 1.00000 3.5 TiB 213 GiB 212 GiB 312 MiB 712 MiB 3.3 TiB 5.95 0.95 182 up 27 ssd 3.49219 1.00000 3.5 TiB 221 GiB 220 GiB 219 MiB 805 MiB 3.3 TiB 6.17 0.98 185 up 2 ssd 1.74609 1.00000 1.7 TiB 104 GiB 103 GiB 116 MiB 908 MiB 1.6 TiB 5.80 0.92 92 up 8 ssd 1.74609 1.00000 1.7 TiB 111 GiB 110 GiB 118 MiB 906 MiB 1.6 TiB 6.21 0.99 91 up 14 ssd 1.74609 1.00000 1.7 TiB 106 GiB 105 GiB 192 MiB 832 MiB 1.6 TiB 5.95 0.94 92 up 20 ssd 3.49219 1.00000 3.5 TiB 226 GiB 225 GiB 196 MiB 828 MiB 3.3 TiB 6.31 1.00 185 up 26 ssd 3.49219 1.00000 3.5 TiB 232 GiB 231 GiB 231 MiB 793 MiB 3.3 TiB 6.47 1.03 184 up 32 ssd 3.49219 1.00000 3.5 TiB 226 GiB 225 GiB 229 MiB 795 MiB 3.3 TiB 6.31 1.00 184 up TOTAL 94 TiB 5.9 TiB 5.9 TiB 6.9 GiB 29 GiB 88 TiB 6.29 MIN/MAX VAR: 0.92/1.12 STDDEV: 0.31 ceph osd crush tree looks like this: ID CLASS WEIGHT (compat) TYPE NAME -1 94.28906 root default -9 15.71484 16.19038 host storage2n1-la 6 ssd 1.74609 1.77313 osd.6 12 ssd 1.74609 1.82532 osd.12 18 ssd 1.74609 2.10315 osd.18 24 ssd 3.49219 3.50087 osd.24 30 ssd 3.49219 3.01933 osd.30 35 ssd 3.49219 3.96858 osd.35 -11 15.71484 15.75711 host storage2n2-la 5 ssd 1.74609 1.84412 osd.5 11 ssd 1.74609 1.71651 osd.11 17 ssd 1.74609 1.76128 osd.17 23 ssd 3.49219 3.73497 osd.23 29 ssd 3.49219 3.27397 osd.29 34 ssd 3.49219 3.42627 osd.34 -7 15.71484 14.19093 host storage2n3-la 4 ssd 1.74609 1.66724 osd.4 10 ssd 1.74609 1.60271 osd.10 16 ssd 1.74609 1.39088 osd.16 22 ssd 3.49219 3.11852 osd.22 28 ssd 3.49219 3.04280 osd.28 33 ssd 3.49219 3.36879 osd.33 -5 15.71484 15.87343 host storage2n4-la 1 ssd 1.74609 1.92644 osd.1 7 ssd 1.74609 2.12386 osd.7 13 ssd 1.74609 1.42424 osd.13 19 ssd 3.49219 3.52307 osd.19 25 ssd 3.49219 3.55241 osd.25 31 ssd 3.49219 3.32341 osd.31 -3 15.71484 16.08948 host storage2n5-la 0 ssd 1.74609 1.97093 osd.0 3 ssd 1.74609 1.87062 osd.3 9 ssd 1.74609 1.57335 osd.9 15 ssd 3.49219 3.82397 osd.15 21 ssd 3.49219 3.59575 osd.21 27 ssd 3.49219 3.25485 osd.27 -13 15.71484 16.18745 host storage2n6-la 2 ssd 1.74609 2.18393 osd.2 8 ssd 1.74609 1.69547 osd.8 14 ssd 1.74609 1.95445 osd.14 20 ssd 3.49219 3.49811 osd.20 26 ssd 3.49219 3.42702 osd.26 32 ssd 3.49219 3.42848 osd.32 ceph balancer status { "last_optimize_duration": "0:00:00.903346", "plans": [], "mode": "crush-compat", "active": true, "optimize_result": "Unable to find further optimization, change balancer mode and retry might help", "last_optimize_started": "Thu Jan 16 05:52:53 2020" }

1 0

Re: Objects not removed (completely) when removing a rbd image
by 徐蕴 16 Jan '20

16 Jan '20

Hi, I just looked through the rbd driver of OpenStack cinder. It seems there is no additional clear_volume step implemented for rbd driver. In my case, objects of this rbd image were deleted partially, so I doubt it’s related Ceph instead of Cinder driver. br, Xu Yun > 2020年1月15日下午7:36，EDH - Manuel Rios <mriosfer(a)easydatahost.com> 写道： > > Hi > > For huge volumes in Openstack and Ceph, setup in your cinder this param: > > volume_clear_size = 50 > > That will wipe only the first 50MB of the file and then ask to ceph to fully delete instead wipe all disk with zeros that sometimes in huge volumes cause timeout. > > In our deploy that was the solution, Openstack Queens here > > > -----Mensaje original----- > De: Eugen Block <eblock(a)nde.ag> > Enviado el: miércoles, 15 de enero de 2020 8:51 > Para: ceph-users(a)ceph.io > Asunto: [ceph-users] Re: Objects not removed (completely) when removing a rbd image > > Hi, > > this might happen if you try to delete images/instances/volumes in openstack that are somehow linked, e.g. if there are snapshots etc. I have experienced this in Ocata, too. Deleting a base image worked but there were existing clones so basically just the openstack database was updated, but the base image still existed within ceph. > > Try to figure out if that is also the case. If it's something else, check the logs in your openstack environment, maybe they reveal something. Also check the ceph logs. > > Regards, > Eugen > > > Zitat von 徐蕴 <yunxu(a)me.com>: > >> Hello, >> >> My setup is Ceph pike working with OpenStack. When I deleted an image, >> I found that the space was not reclaimed. I checked with rbd ls and >> confirmed that this image was disappeared. But when I check the >> objects with rados ls, most objects named rbd_data.xxx are still >> existed in my cluster. rbd_object_map and rbd_header were already >> deleted. I waited for several hours and there is no further deletion >> happed. Is it a known issue, or something wrong with my configuration? >> >> br, >> Xu Yun >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Objects not removed (completely) when removing a rbd image
by 徐蕴 15 Jan '20

15 Jan '20

Hello, My setup is Ceph pike working with OpenStack. When I deleted an image, I found that the space was not reclaimed. I checked with rbd ls and confirmed that this image was disappeared. But when I check the objects with rados ls, most objects named rbd_data.xxx are still existed in my cluster. rbd_object_map and rbd_header were already deleted. I waited for several hours and there is no further deletion happed. Is it a known issue, or something wrong with my configuration? br, Xu Yun

4 6

dpdk used issue in master
by zhengyin＠cmss.chinamobile.com 15 Jan '20

15 Jan '20

Hello everyone If ms_type = async + dpdk, Ceph can work in the master version? I have a problem using dpdk in the master version. Mon cannot be started. Please hep me, thank you very much! 1、My network card configuration information： [root@ebs12 dpdk]# python usertools/dpdk-devbind.py --status Network devices using DPDK-compatible driver ============================================ 0000:05:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=igb_uio unused=ixgbe 0000:05:00.1 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=igb_uio unused=ixgbe Network devices using kernel driver =================================== 0000:08:00.0 'I350 Gigabit Network Connection 1521' if=enp8s0f0 drv=igb unused=igb_uio 0000:08:00.1 'I350 Gigabit Network Connection 1521' if=enp8s0f1 drv=igb unused=igb_uio No 'Crypto' devices detected ============================ No 'Eventdev' devices detected ============================== No 'Mempool' devices detected ============================= No 'Compress' devices detected 2、Dpdk can receive and contract normally testpmd> start io packet forwarding - ports=2 - cores=1 - streams=2 - NUMA support enabled, MP allocation mode: native Logical Core 1 (socket 0) forwards packets on 2 streams: RX P=0/Q=0 (socket 0) -> TX P=1/Q=0 (socket 0) peer=02:00:00:00:00:01 RX P=1/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=02:00:00:00:00:00 io packet forwarding packets/burst=32 nb forwarding cores=1 - nb forwarding ports=2 port 0: RX queue number: 1 Tx queue number: 1 Rx offloads=0x0 Tx offloads=0x0 RX queue: 0 RX desc=256 - RX free threshold=32 RX threshold registers: pthresh=8 hthresh=8 wthresh=0 RX Offloads=0x0 TX queue: 0 TX desc=256 - TX free threshold=32 TX threshold registers: pthresh=32 hthresh=0 wthresh=0 TX offloads=0x0 - TX RS bit threshold=32 port 1: RX queue number: 1 Tx queue number: 1 Rx offloads=0x0 Tx offloads=0x0 RX queue: 0 RX desc=256 - RX free threshold=32 RX threshold registers: pthresh=8 hthresh=8 wthresh=0 RX Offloads=0x0 TX queue: 0 TX desc=256 - TX free threshold=32 TX threshold registers: pthresh=32 hthresh=0 wthresh=0 TX offloads=0x0 - TX RS bit threshold=32 testpmd> stop Telling cores to stop... Waiting for lcores to finish... ---------------------- Forward statistics for port 0 ---------------------- RX-packets: 10 RX-dropped: 0 RX-total: 10 TX-packets: 9 TX-dropped: 0 TX-total: 9 RX-bursts : 10 [100% of 1 pkts] TX-bursts : 9 [100% of 1 pkts] ---------------------------------------------------------------------------- ---------------------- Forward statistics for port 1 ---------------------- RX-packets: 9 RX-dropped: 0 RX-total: 9 TX-packets: 10 TX-dropped: 0 TX-total: 10 RX-bursts : 9 [100% of 1 pkts] TX-bursts : 10 [100% of 1 pkts] ---------------------------------------------------------------------------- +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++ RX-packets: 19 RX-dropped: 0 RX-total: 19 TX-packets: 19 TX-dropped: 0 TX-total: 19 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ CPU cycles/packet=2538 (total cycles=48235 / total RX packets=19) Done. 3、When I start Mon, an error will be reported. I set mon addr = 0000:05:00.0 in ceph.conf [root@ebs12 build]# ./bin/ceph-mon -i a -c ceph.conf 2020-01-15T10:54:20.795+0800 7fce87f5c540 -1 WARNING: all dangerous and experimental features are enabled. 2020-01-15T10:54:20.795+0800 7fce87f5c540 -1 WARNING: all dangerous and experimental features are enabled. 2020-01-15T10:54:20.806+0800 7fce87f5c540 -1 WARNING: all dangerous and experimental features are enabled. 2020-01-15T10:54:20.807+0800 7fce87f5c540 -1 asok(0x7fce8bafa000) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/tmp/ceph-asok.EUSssX/mon.a.asok': (2) No such file or directory 2020-01-15T10:54:20.972+0800 7fce87f5c540 -1 WARNING: invalid 'mon addr' config option continuing with monmap configuration EAL: Detected 32 lcore(s) EAL: Detected 2 NUMA nodes EAL: Multi-process socket /var/run/dpdk/rte_mon.a/mp_socket EAL: No available hugepages reported in hugepages-1048576kB EAL: Probing VFIO support... EAL: Error - exiting with code: 1 Cause: No Ethernet ports - bye /home/zy/data/shequ/bak/ceph/src/common/mutex_debug.h: In function 'ceph::mutex_debug_detail::mutex_debug_impl<<anonymous> >::~mutex_debug_impl() [with bool Recursive = false]' thread 7fce2c44b700 time 2020-01-15T10:54:21.019921+0800 /home/zy/data/shequ/bak/ceph/src/common/mutex_debug.h: 111: FAILED ceph_assert(r == 0) ceph version 11.0.2-46853-g0e52da0 (0e52da069bc02704f6a7d5f6a1ce5240bfafc8f0) octopus (dev) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1aa) [0x7fce7e8b9030] 2: (()+0x15162b2) [0x7fce7e8b92b2] 3: (ceph::mutex_debug_detail::mutex_debug_impl<false>::~mutex_debug_impl()+0x34) [0x7fce7e763ab2] 4: (()+0x39c99) [0x7fce7a002c99] 5: (()+0x39ce7) [0x7fce7a002ce7] 6: (()+0x1e245a4) [0x7fce7f1c75a4] 7: (create_dpdk_net_device(CephContext*, unsigned int, unsigned char, bool, bool)+0x6b) [0x7fce7f17b809] 8: (DPDKWorker::initialize()+0x210) [0x7fce7f11770a] 9: (()+0x1893ff1) [0x7fce7ec36ff1] 10: (()+0x1895641) [0x7fce7ec38641] 11: (std::function<void ()>::operator()() const+0x32) [0x7fce7ec35a1a] 12: (()+0x1d7444f) [0x7fce7f11744f] 13: (()+0x13ab30e) [0x7fce7e74e30e] 14: (()+0x7e65) [0x7fce7b40de65] 15: (clone()+0x6d) [0x7fce7a0c788d] *** Caught signal (Aborted) ** in thread 7fce2c44b700 thread_name:msgr-worker-0 2020-01-15T10:54:21.043+0800 7fce2c44b700 -1 /home/zy/data/shequ/bak/ceph/src/common/mutex_debug.h: In function 'ceph::mutex_debug_detail::mutex_debug_impl<<anonymous> >::~mutex_debug_impl() [with bool Recursive = false]' thread 7fce2c44b700 time 2020-01-15T10:54:21.019921+0800 /home/zy/data/shequ/bak/ceph/src/common/mutex_debug.h: 111: FAILED ceph_assert(r == 0) ceph version 11.0.2-46853-g0e52da0 (0e52da069bc02704f6a7d5f6a1ce5240bfafc8f0) octopus (dev) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1aa) [0x7fce7e8b9030] 2: (()+0x15162b2) [0x7fce7e8b92b2] 3: (ceph::mutex_debug_detail::mutex_debug_impl<false>::~mutex_debug_impl()+0x34) [0x7fce7e763ab2] 4: (()+0x39c99) [0x7fce7a002c99] 5: (()+0x39ce7) [0x7fce7a002ce7] 6: (()+0x1e245a4) [0x7fce7f1c75a4] 7: (create_dpdk_net_device(CephContext*, unsigned int, unsigned char, bool, bool)+0x6b) [0x7fce7f17b809] 8: (DPDKWorker::initialize()+0x210) [0x7fce7f11770a] 9: (()+0x1893ff1) [0x7fce7ec36ff1] 10: (()+0x1895641) [0x7fce7ec38641] 11: (std::function<void ()>::operator()() const+0x32) [0x7fce7ec35a1a] 12: (()+0x1d7444f) [0x7fce7f11744f] 13: (()+0x13ab30e) [0x7fce7e74e30e] 14: (()+0x7e65) [0x7fce7b40de65] 15: (clone()+0x6d) [0x7fce7a0c788d] ceph version 11.0.2-46853-g0e52da0 (0e52da069bc02704f6a7d5f6a1ce5240bfafc8f0) octopus (dev) 1: (()+0x1489a10) [0x7fce89412a10] 2: (()+0xf5f0) [0x7fce7b4155f0] 3: (gsignal()+0x37) [0x7fce79fff337] 4: (abort()+0x148) [0x7fce7a000a28] 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x379) [0x7fce7e8b91ff] 6: (()+0x15162b2) [0x7fce7e8b92b2] 7: (ceph::mutex_debug_detail::mutex_debug_impl<false>::~mutex_debug_impl()+0x34) [0x7fce7e763ab2] 8: (()+0x39c99) [0x7fce7a002c99] 9: (()+0x39ce7) [0x7fce7a002ce7] 10: (()+0x1e245a4) [0x7fce7f1c75a4] 11: (create_dpdk_net_device(CephContext*, unsigned int, unsigned char, bool, bool)+0x6b) [0x7fce7f17b809] 12: (DPDKWorker::initialize()+0x210) [0x7fce7f11770a] 13: (()+0x1893ff1) [0x7fce7ec36ff1] 14: (()+0x1895641) [0x7fce7ec38641] 15: (std::function<void ()>::operator()() const+0x32) [0x7fce7ec35a1a] 16: (()+0x1d7444f) [0x7fce7f11744f] 17: (()+0x13ab30e) [0x7fce7e74e30e] 18: (()+0x7e65) [0x7fce7b40de65] 19: (clone()+0x6d) [0x7fce7a0c788d] 2020-01-15T10:54:21.067+0800 7fce2c44b700 -1 *** Caught signal (Aborted) ** in thread 7fce2c44b700 thread_name:msgr-worker-0 ceph version 11.0.2-46853-g0e52da0 (0e52da069bc02704f6a7d5f6a1ce5240bfafc8f0) octopus (dev) 1: (()+0x1489a10) [0x7fce89412a10] 2: (()+0xf5f0) [0x7fce7b4155f0] 3: (gsignal()+0x37) [0x7fce79fff337] 4: (abort()+0x148) [0x7fce7a000a28] 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x379) [0x7fce7e8b91ff] 6: (()+0x15162b2) [0x7fce7e8b92b2] 7: (ceph::mutex_debug_detail::mutex_debug_impl<false>::~mutex_debug_impl()+0x34) [0x7fce7e763ab2] 8: (()+0x39c99) [0x7fce7a002c99] 9: (()+0x39ce7) [0x7fce7a002ce7] 10: (()+0x1e245a4) [0x7fce7f1c75a4] 11: (create_dpdk_net_device(CephContext*, unsigned int, unsigned char, bool, bool)+0x6b) [0x7fce7f17b809] 12: (DPDKWorker::initialize()+0x210) [0x7fce7f11770a] 13: (()+0x1893ff1) [0x7fce7ec36ff1] 14: (()+0x1895641) [0x7fce7ec38641] 15: (std::function<void ()>::operator()() const+0x32) [0x7fce7ec35a1a] 16: (()+0x1d7444f) [0x7fce7f11744f] 17: (()+0x13ab30e) [0x7fce7e74e30e] 18: (()+0x7e65) [0x7fce7b40de65] 19: (clone()+0x6d) [0x7fce7a0c788d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. -250> 2020-01-15T10:54:20.795+0800 7fce87f5c540 -1 WARNING: all dangerous and experimental features are enabled. -247> 2020-01-15T10:54:20.795+0800 7fce87f5c540 -1 WARNING: all dangerous and experimental features are enabled. -232> 2020-01-15T10:54:20.806+0800 7fce87f5c540 -1 WARNING: all dangerous and experimental features are enabled. -229> 2020-01-15T10:54:20.807+0800 7fce87f5c540 -1 asok(0x7fce8bafa000) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/tmp/ceph-asok.EUSssX/mon.a.asok': (2) No such file or directory -40> 2020-01-15T10:54:20.972+0800 7fce87f5c540 -1 WARNING: invalid 'mon addr' config option continuing with monmap configuration -1> 2020-01-15T10:54:21.043+0800 7fce2c44b700 -1 /home/zy/data/shequ/bak/ceph/src/common/mutex_debug.h: In function 'ceph::mutex_debug_detail::mutex_debug_impl<<anonymous> >::~mutex_debug_impl() [with bool Recursive = false]' thread 7fce2c44b700 time 2020-01-15T10:54:21.019921+0800 /home/zy/data/shequ/bak/ceph/src/common/mutex_debug.h: 111: FAILED ceph_assert(r == 0) ceph version 11.0.2-46853-g0e52da0 (0e52da069bc02704f6a7d5f6a1ce5240bfafc8f0) octopus (dev) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1aa) [0x7fce7e8b9030] 2: (()+0x15162b2) [0x7fce7e8b92b2] 3: (ceph::mutex_debug_detail::mutex_debug_impl<false>::~mutex_debug_impl()+0x34) [0x7fce7e763ab2] 4: (()+0x39c99) [0x7fce7a002c99] 5: (()+0x39ce7) [0x7fce7a002ce7] 6: (()+0x1e245a4) [0x7fce7f1c75a4] 7: (create_dpdk_net_device(CephContext*, unsigned int, unsigned char, bool, bool)+0x6b) [0x7fce7f17b809] 8: (DPDKWorker::initialize()+0x210) [0x7fce7f11770a] 9: (()+0x1893ff1) [0x7fce7ec36ff1] 10: (()+0x1895641) [0x7fce7ec38641] 11: (std::function<void ()>::operator()() const+0x32) [0x7fce7ec35a1a] 12: (()+0x1d7444f) [0x7fce7f11744f] 13: (()+0x13ab30e) [0x7fce7e74e30e] 14: (()+0x7e65) [0x7fce7b40de65] 15: (clone()+0x6d) [0x7fce7a0c788d] 0> 2020-01-15T10:54:21.067+0800 7fce2c44b700 -1 *** Caught signal (Aborted) ** in thread 7fce2c44b700 thread_name:msgr-worker-0 ceph version 11.0.2-46853-g0e52da0 (0e52da069bc02704f6a7d5f6a1ce5240bfafc8f0) octopus (dev) 1: (()+0x1489a10) [0x7fce89412a10] 2: (()+0xf5f0) [0x7fce7b4155f0] 3: (gsignal()+0x37) [0x7fce79fff337] 4: (abort()+0x148) [0x7fce7a000a28] 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x379) [0x7fce7e8b91ff] 6: (()+0x15162b2) [0x7fce7e8b92b2] 7: (ceph::mutex_debug_detail::mutex_debug_impl<false>::~mutex_debug_impl()+0x34) [0x7fce7e763ab2] 8: (()+0x39c99) [0x7fce7a002c99] 9: (()+0x39ce7) [0x7fce7a002ce7] 10: (()+0x1e245a4) [0x7fce7f1c75a4] 11: (create_dpdk_net_device(CephContext*, unsigned int, unsigned char, bool, bool)+0x6b) [0x7fce7f17b809] 12: (DPDKWorker::initialize()+0x210) [0x7fce7f11770a] 13: (()+0x1893ff1) [0x7fce7ec36ff1] 14: (()+0x1895641) [0x7fce7ec38641] 15: (std::function<void ()>::operator()() const+0x32) [0x7fce7ec35a1a] 16: (()+0x1d7444f) [0x7fce7f11744f] 17: (()+0x13ab30e) [0x7fce7e74e30e] 18: (()+0x7e65) [0x7fce7b40de65] 19: (clone()+0x6d) [0x7fce7a0c788d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. -250> 2020-01-15T10:54:20.795+0800 7fce87f5c540 -1 WARNING: all dangerous and experimental features are enabled. -247> 2020-01-15T10:54:20.795+0800 7fce87f5c540 -1 WARNING: all dangerous and experimental features are enabled. -232> 2020-01-15T10:54:20.806+0800 7fce87f5c540 -1 WARNING: all dangerous and experimental features are enabled. -229> 2020-01-15T10:54:20.807+0800 7fce87f5c540 -1 asok(0x7fce8bafa000) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/tmp/ceph-asok.EUSssX/mon.a.asok': (2) No such file or directory -40> 2020-01-15T10:54:20.972+0800 7fce87f5c540 -1 WARNING: invalid 'mon addr' config option continuing with monmap configuration -1> 2020-01-15T10:54:21.043+0800 7fce2c44b700 -1 /home/zy/data/shequ/bak/ceph/src/common/mutex_debug.h: In function 'ceph::mutex_debug_detail::mutex_debug_impl<<anonymous> >::~mutex_debug_impl() [with bool Recursive = false]' thread 7fce2c44b700 time 2020-01-15T10:54:21.019921+0800 /home/zy/data/shequ/bak/ceph/src/common/mutex_debug.h: 111: FAILED ceph_assert(r == 0) ceph version 11.0.2-46853-g0e52da0 (0e52da069bc02704f6a7d5f6a1ce5240bfafc8f0) octopus (dev) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1aa) [0x7fce7e8b9030] 2: (()+0x15162b2) [0x7fce7e8b92b2] 3: (ceph::mutex_debug_detail::mutex_debug_impl<false>::~mutex_debug_impl()+0x34) [0x7fce7e763ab2] 4: (()+0x39c99) [0x7fce7a002c99] 5: (()+0x39ce7) [0x7fce7a002ce7] 6: (()+0x1e245a4) [0x7fce7f1c75a4] 7: (create_dpdk_net_device(CephContext*, unsigned int, unsigned char, bool, bool)+0x6b) [0x7fce7f17b809] 8: (DPDKWorker::initialize()+0x210) [0x7fce7f11770a] 9: (()+0x1893ff1) [0x7fce7ec36ff1] 10: (()+0x1895641) [0x7fce7ec38641] 11: (std::function<void ()>::operator()() const+0x32) [0x7fce7ec35a1a] 12: (()+0x1d7444f) [0x7fce7f11744f] 13: (()+0x13ab30e) [0x7fce7e74e30e] 14: (()+0x7e65) [0x7fce7b40de65] 15: (clone()+0x6d) [0x7fce7a0c788d] 0> 2020-01-15T10:54:21.067+0800 7fce2c44b700 -1 *** Caught signal (Aborted) ** in thread 7fce2c44b700 thread_name:msgr-worker-0 ceph version 11.0.2-46853-g0e52da0 (0e52da069bc02704f6a7d5f6a1ce5240bfafc8f0) octopus (dev) 1: (()+0x1489a10) [0x7fce89412a10] 2: (()+0xf5f0) [0x7fce7b4155f0] 3: (gsignal()+0x37) [0x7fce79fff337] 4: (abort()+0x148) [0x7fce7a000a28] 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x379) [0x7fce7e8b91ff] 6: (()+0x15162b2) [0x7fce7e8b92b2] 7: (ceph::mutex_debug_detail::mutex_debug_impl<false>::~mutex_debug_impl()+0x34) [0x7fce7e763ab2] 8: (()+0x39c99) [0x7fce7a002c99] 9: (()+0x39ce7) [0x7fce7a002ce7] 10: (()+0x1e245a4) [0x7fce7f1c75a4] 11: (create_dpdk_net_device(CephContext*, unsigned int, unsigned char, bool, bool)+0x6b) [0x7fce7f17b809] 12: (DPDKWorker::initialize()+0x210) [0x7fce7f11770a] 13: (()+0x1893ff1) [0x7fce7ec36ff1] 14: (()+0x1895641) [0x7fce7ec38641] 15: (std::function<void ()>::operator()() const+0x32) [0x7fce7ec35a1a] 16: (()+0x1d7444f) [0x7fce7f11744f] 17: (()+0x13ab30e) [0x7fce7e74e30e] 18: (()+0x7e65) [0x7fce7b40de65] 19: (clone()+0x6d) [0x7fce7a0c788d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. 云计算产品部郑印中移（苏州）软件技术有限公司中国移动云能力中心 18896726650 zhengyin(a)cmss.chinamobile.com 江苏省苏州市高新区科技城昆仑山路58号中移软件园 215153

1 0

bluestore_default_buffered_write = true
by Adam Koczarski 14 Jan '20

14 Jan '20

Has anyone ever tried using this feature? I've added it to the [global] section of the ceph.conf on my POC cluster but I'm not sure how to tell if it's actually working. I did find a reference to this feature via Google and they had it in their [OSD] section?? I've tried that too.. TIA Adam

1 0

CephFS ghost usage/inodes
by Florian Pritz 14 Jan '20

14 Jan '20

Hi, When we tried putting some load on our test cephfs setup by restoring a backup in artifactory, we eventually ran out of space (around 95% used in `df` = 3.5TB) which caused artifactory to abort the restore and clean up. However, while a simple `find` no longer shows the files, `df` still claims that we have around 2.1TB of data on the cephfs. `df -i` also shows 2.4M used inodes. When using `du -sh` on a top-level mountpoint, I get 31G used, which is data that is still really here and which is expected to be here. Consequently, we also get the following warning: > MANY_OBJECTS_PER_PG 1 pools have many more objects per pg than average > pool cephfs_data objects per pg (38711) is more than 231.802 times cluster average (167) We are running ceph 14.2.5. We have snapshots enabled on cephfs, but there are currently no active snapshots listed by `ceph daemon mds.$hostname dump snaps --server` (see below). I can't say for sure if we created snapshots during the backup restore. > { > "last_snap": 39, > "last_created": 38, > "last_destroyed": 39, > "pending_noop": [], > "snaps": [], > "need_to_purge": {}, > "pending_update": [], > "pending_destroy": [] > } We only have a single CephFS. We use the pool_namespace xattr for our various directory trees on the cephfs. `ceph df` shows: > POOL ID STORED OBJECTS USED %USED MAX AVAIL > cephfs_data 6 2.1 TiB 2.48M 2.1 TiB 24.97 3.1 TiB `ceph daemon mds.$hostname perf dump | grep stray` shows: > "num_strays": 0, > "num_strays_delayed": 0, > "num_strays_enqueuing": 0, > "strays_created": 5097138, > "strays_enqueued": 5097138, > "strays_reintegrated": 0, > "strays_migrated": 0, `rados -p cephfs_data df` shows: > POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR > cephfs_data 2.1 TiB 2477540 0 4955080 0 0 0 10699626 6.9 TiB 86911076 35 TiB 0 B 0 B > > total_objects 29718 > total_used 329 GiB > total_avail 7.5 TiB > total_space 7.8 TiB When I combine the usage and the free space shown by `df` we would exceed our cluster size. Our test cluster currently has 7.8TB total space with a replication size of 2 for all pools. With 2.1TB "used" on the cephfs according to `df` + 3.1TB being shows as "free" I get 5.2TB total size. This would mean >10TB of data when accounted for replication. Clearly this can't fit on a cluster with only 7.8TB of capacity. Do you have any ideas why we see so many objects and so much reported usage? Is there any way to fix this without recreating the cephfs? Florian -- Florian Pritz Research Industrial Systems Engineering (RISE) Forschungs-, Entwicklungs- und Großprojektberatung GmbH Concorde Business Park F 2320 Schwechat Austria E-Mail: florian.pritz(a)rise-world.com Web: www.rise-world.com Firmenbuch: FN 280353i Landesgericht Korneuburg UID: ATU62886416

1 0

centralized config map error
by Frank R 14 Jan '20

14 Jan '20

Hi all, When upgrading from Luminous to Nautilus the global configmap options for cluster_network and public_network were inadvertently set to an incorrect value (10.192.80.0/24): ----- [root@ceph-osd134 ceph]# ceph config dump | grep network <snip> global advanced cluster_network 10.192.80.0/24 * global advanced public_network 10.192.80.0/24 ----- Ceph.conf on all nodes is correctly set to 10.0.0.0/8. Even after restarting the mons I see the following errors with every ceph command: ----- 2020-01-10 20:06:30.815 7f6deffff700 -1 set_mon_vals failed to set cluster_network = 10.192.80.0/24: Configuration option 'cluster_network' may not be modified at runtime 2020-01-10 20:06:30.815 7f6deffff700 -1 set_mon_vals failed to set public_network = 10.192.80.0/24: Configuration option 'public_network' may not be modified at runtime ----- How do I safely change/remove the centralized config network settings? Thanks, Frank

2 1

Kworker 100% with ceph-msgr (after upgrade to 14.2.6?)
by Marc Roos 14 Jan '20

14 Jan '20

I think this is new since I upgraded to 14.2.6. kworker/7:3 100% [@~]# echo l > /proc/sysrq-trigger [Tue Jan 14 10:05:08 2020] CPU: 7 PID: 2909400 Comm: kworker/7:0 Not tainted 3.10.0-1062.4.3.el7.x86_64 #1 [Tue Jan 14 10:05:08 2020] Workqueue: ceph-msgr ceph_con_workfn [libceph] [Tue Jan 14 10:05:08 2020] task: ffffa0d2cb9db150 ti: ffffa0d3040f0000 task.ti: ffffa0d3040f0000 [Tue Jan 14 10:05:08 2020] RIP: 0010:[<ffffffffb0192e7e>] [<ffffffffb0192e7e>] generic_swap+0x1e/0x30 [Tue Jan 14 10:05:08 2020] RSP: 0018:ffffa0d3040f3a20 EFLAGS: 00000206 [Tue Jan 14 10:05:08 2020] RAX: 0000000000000072 RBX: 0000000000000060 RCX: 0000000000000072 [Tue Jan 14 10:05:08 2020] RDX: 0000000000000006 RSI: ffffa0c788914a7a RDI: ffffa0c788914a42 [Tue Jan 14 10:05:08 2020] RBP: ffffa0d3040f3a20 R08: 0000000000000028 R09: 0000000000000016 [Tue Jan 14 10:05:08 2020] R10: 0000000000000036 R11: ffffe94984224500 R12: ffffa0c788914a40 [Tue Jan 14 10:05:08 2020] R13: ffffffffc08d7da0 R14: ffffa0c788914a18 R15: ffffa0c788914a78 [Tue Jan 14 10:05:08 2020] FS: 0000000000000000(0000) GS:ffffa0d2cfbc0000(0000) knlGS:0000000000000000 [Tue Jan 14 10:05:08 2020] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Tue Jan 14 10:05:08 2020] CR2: 000055f0404d5e40 CR3: 0000001813010000 CR4: 00000000000627e0 [Tue Jan 14 10:05:08 2020] Call Trace: [Tue Jan 14 10:05:08 2020] [<ffffffffb0193055>] sort+0x1c5/0x260 [Tue Jan 14 10:05:08 2020] [<ffffffffb0192e60>] ? u32_swap+0x10/0x10 [Tue Jan 14 10:05:08 2020] [<ffffffffc08d807b>] build_snap_context+0x12b/0x290 [ceph] [Tue Jan 14 10:05:08 2020] [<ffffffffc08d820c>] rebuild_snap_realms+0x2c/0x90 [ceph] [Tue Jan 14 10:05:08 2020] [<ffffffffc08d822b>] rebuild_snap_realms+0x4b/0x90 [ceph] [Tue Jan 14 10:05:08 2020] [<ffffffffc08d91fc>] ceph_update_snap_trace+0x3ec/0x530 [ceph] [Tue Jan 14 10:05:08 2020] [<ffffffffc08e2239>] handle_reply+0x359/0xc60 [ceph] [Tue Jan 14 10:05:08 2020] [<ffffffffc08e48ba>] dispatch+0x11a/0xb00 [ceph] [Tue Jan 14 10:05:08 2020] [<ffffffffb042e56a>] ? kernel_recvmsg+0x3a/0x50 [Tue Jan 14 10:05:08 2020] [<ffffffffc05fcff4>] try_read+0x544/0x1300 [libceph] [Tue Jan 14 10:05:08 2020] [<ffffffffafee13ce>] ? account_entity_dequeue+0xae/0xd0 [Tue Jan 14 10:05:08 2020] [<ffffffffafee4d5c>] ? dequeue_entity+0x11c/0x5e0 [Tue Jan 14 10:05:08 2020] [<ffffffffb042e417>] ? kernel_sendmsg+0x37/0x50 [Tue Jan 14 10:05:08 2020] [<ffffffffc05fdfb4>] ceph_con_workfn+0xe4/0x1530 [libceph] [Tue Jan 14 10:05:08 2020] [<ffffffffb057f568>] ? __schedule+0x448/0x9c0 [Tue Jan 14 10:05:08 2020] [<ffffffffafebe21f>] process_one_work+0x17f/0x440 [Tue Jan 14 10:05:08 2020] [<ffffffffafebf336>] worker_thread+0x126/0x3c0 [Tue Jan 14 10:05:08 2020] [<ffffffffafebf210>] ? manage_workers.isra.26+0x2a0/0x2a0 [Tue Jan 14 10:05:08 2020] [<ffffffffafec61f1>] kthread+0xd1/0xe0 [Tue Jan 14 10:05:08 2020] [<ffffffffafec6120>] ? insert_kthread_work+0x40/0x40 [Tue Jan 14 10:05:08 2020] [<ffffffffb058cd37>] ret_from_fork_nospec_begin+0x21/0x21 [Tue Jan 14 10:05:08 2020] [<ffffffffafec6120>] ? insert_kthread_work+0x40/0x40 [Tue Jan 14 10:05:08 2020] CPU: 7 PID: 2909400 Comm: kworker/7:0 Not tainted 3.10.0-1062.4.3.el7.x86_64 #1 [Tue Jan 14 10:05:08 2020] Workqueue: ceph-msgr ceph_con_workfn [libceph] [Tue Jan 14 10:05:08 2020] task: ffffa0d2cb9db150 ti: ffffa0d3040f0000 task.ti: ffffa0d3040f0000 [Tue Jan 14 10:05:08 2020] RIP: 0010:[<ffffffffb0192200>] [<ffffffffb0192200>] __x86_indirect_thunk_rax+0x0/0x20 [Tue Jan 14 10:05:08 2020] RSP: 0018:ffffa0d3040f3a28 EFLAGS: 00000286 [Tue Jan 14 10:05:08 2020] RAX: ffffffffb0192e60 RBX: 0000000000000010 RCX: 0000000000000010 [Tue Jan 14 10:05:08 2020] RDX: 0000000000000008 RSI: ffffa0d0e108b828 RDI: ffffa0d0e108b818 [Tue Jan 14 10:05:08 2020] RBP: ffffa0d3040f3a98 R08: 0000000000000000 R09: 0000000000000016 [Tue Jan 14 10:05:08 2020] R10: 0000000000000036 R11: ffffe949a9842280 R12: ffffa0d0e108b818 [Tue Jan 14 10:05:08 2020] R13: ffffffffc08d7da0 R14: ffffa0d0e108b818 R15: ffffa0d0e108b828 [Tue Jan 14 10:05:08 2020] FS: 0000000000000000(0000) GS:ffffa0d2cfbc0000(0000) knlGS:0000000000000000 [Tue Jan 14 10:05:08 2020] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [Tue Jan 14 10:05:08 2020] CR2: 000055f0404d5e40 CR3: 0000001813010000 CR4: 00000000000627e0 [Tue Jan 14 10:05:08 2020] Call Trace: [Tue Jan 14 10:05:08 2020] [<ffffffffb0193055>] ? sort+0x1c5/0x260 [Tue Jan 14 10:05:08 2020] [<ffffffffb0192e60>] ? u32_swap+0x10/0x10 [Tue Jan 14 10:05:08 2020] [<ffffffffc08d807b>] build_snap_context+0x12b/0x290 [ceph] [Tue Jan 14 10:05:08 2020] [<ffffffffc08d820c>] rebuild_snap_realms+0x2c/0x90 [ceph] [Tue Jan 14 10:05:08 2020] [<ffffffffc08d822b>] rebuild_snap_realms+0x4b/0x90 [ceph] [Tue Jan 14 10:05:08 2020] [<ffffffffc08d91fc>] ceph_update_snap_trace+0x3ec/0x530 [ceph] [Tue Jan 14 10:05:08 2020] [<ffffffffc08e2239>] handle_reply+0x359/0xc60 [ceph] [Tue Jan 14 10:05:08 2020] [<ffffffffc08e48ba>] dispatch+0x11a/0xb00 [ceph] [Tue Jan 14 10:05:08 2020] [<ffffffffb042e56a>] ? kernel_recvmsg+0x3a/0x50 [Tue Jan 14 10:05:08 2020] [<ffffffffc05fcff4>] try_read+0x544/0x1300 [libceph] [Tue Jan 14 10:05:08 2020] [<ffffffffafee13ce>] ? account_entity_dequeue+0xae/0xd0 [Tue Jan 14 10:05:08 2020] [<ffffffffafee4d5c>] ? dequeue_entity+0x11c/0x5e0 [Tue Jan 14 10:05:08 2020] [<ffffffffb042e417>] ? kernel_sendmsg+0x37/0x50 [Tue Jan 14 10:05:08 2020] [<ffffffffc05fdfb4>] ceph_con_workfn+0xe4/0x1530 [libceph] [Tue Jan 14 10:05:08 2020] [<ffffffffb057f568>] ? __schedule+0x448/0x9c0 [Tue Jan 14 10:05:08 2020] [<ffffffffafebe21f>] process_one_work+0x17f/0x440 [Tue Jan 14 10:05:08 2020] [<ffffffffafebf336>] worker_thread+0x126/0x3c0 [Tue Jan 14 10:05:08 2020] [<ffffffffafebf210>] ? manage_workers.isra.26+0x2a0/0x2a0 [Tue Jan 14 10:05:08 2020] [<ffffffffafec61f1>] kthread+0xd1/0xe0 [Tue Jan 14 10:05:08 2020] [<ffffffffafec6120>] ? insert_kthread_work+0x40/0x40 [Tue Jan 14 10:05:08 2020] [<ffffffffb058cd37>] ret_from_fork_nospec_begin+0x21/0x21 [Tue Jan 14 10:05:08 2020] [<ffffffffafec6120>] ? insert_kthread_work+0x40/0x40

1 0

unset centralized config read only global setting
by Frank R 13 Jan '20

13 Jan '20

Hi All, Sorry for the repost. How do you unset a global config setting from the centralized config with mimic+ (specifically public_network and cluster_network) "ceph config rm global public_network" doesn't seem to do the trick. These were set inadvertently during an upgrade with: "ceph config assimilate-conf" https://ceph.io/community/new-mimic-centralized-configuration-management/ The settings I wish to unset: ----- [root@ceph-mon001 ceph]# ceph config dump | grep network global advanced cluster_network 10.192.80.0/24 * global advanced public_network 10.192.80.0/24 * ----- thx Frank

1 0

best practices for cephfs on hard drives mimic
by Chad W Seys 13 Jan '20

13 Jan '20

Hi all, In the era of Mimic, what are best practices for setting up cephfs on a hard drive only cluster? Our old cluster which began life in Emperor and has been upgraded until now running Mimic. 21 hard drives ranging from 1 to 4 TB. (Yeah, ceph doesn't like that.) It has a triply replicated cache tier in front of a k2m2 erasure coded pool. Our main usage these days is cephfs. I vaguely remember a few ideas, which might be out of date or never true: Bluestore is slower than filestore on hard drives. ? Before release X replicated caches were needed to interface between writers and the erasure coded pool, but after release X one could write directly to the erasure coded pool. ? Thanks for your input! Chad.

2 2

Slow Operations observed on one OSD (dedicated for RGW indexes), caused by problematic Placement Group
by P. O. 13 Jan '20

13 Jan '20

Hi All, I am observing strange behaviour on one OSD, which is dedicated for RGW index (pool: default.rgw.buckets.index). My ceph is: ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable) In log files are many messages like below: 2020-01-08 10:37:53.054 7f1fa22dd700 0 bluestore(/var/lib/ceph/osd/ceph-62) log_latency slow operation observed for next, latency = 22.4641s ... 2020-01-08 14:57:06.987 7f1fa22dd700 0 bluestore(/var/lib/ceph/osd/ceph-62) log_latency_fn slow operation observed for upper_bound, latency = 28.9336s, after = �0_ omap_iterator(cid = 230.2c_head, oid = #230:36f53ff9:::.dir.12307ed8-2342-4293-9f14-12234220a234.13221775.224244.31:head#) 2020-01-08 15:17:39.972 7f1fa22dd700 0 bluestore(/var/lib/ceph/osd/ceph-62) log_latency_fn slow operation observed for upper_bound, latency = 23.9168s, after = �0_ omap_iterator(cid = 230.2c_head, oid = #230:36f53ff9:::.dir.12307ed8-2342-4293-9f14-12234220a234.13221775.224244.31:head#) 2020-01-08 15:37:44.221 7f1fa62e5700 0 bluestore(/var/lib/ceph/osd/ceph-62) log_latency_fn slow operation observed for upper_bound, latency = 25.8316s, after = �0_ omap_iterator(cid = 230.2c_head, oid = #230:36f53ff9:::.dir.12307ed8-2342-4293-9f14-12234220a234.13221775.224244.31:head#) 2020-01-08 15:58:12.279 7f1fa62e5700 0 bluestore(/var/lib/ceph/osd/ceph-62) log_latency_fn slow operation observed for upper_bound, latency = 27.8207s, after = �0_ omap_iterator(cid = 230.2c_head, oid = #230:36f53ff9:::.dir.12307ed8-2342-4293-9f14-12234220a234.13221775.224244.31:head#) I discovered that these times are associated with GC bilogs (it runs every 30 minutes). # /etc/ceph/ceph.conf rgw sync log trim interval = 1200 ID of bucket bucket_1: 12307ed8-2342-4293-9f14-12234220a234.13221775.224244 Shard nr. 31 of bucket (bucket_1) index: .dir.12307ed8-2342-4293-9f14-12234220a234.13221775.224244.31 [root@mon-1 ~]# ceph osd map default.rgw.buckets.index .dir.12307ed8-2342-4293-9f14-12234220a234.13221775.224244.31 osdmap e23417 pool 'default.rgw.buckets.index' (230) object '.dir.12307ed8-2342-4293-9f14-12234220a234.13221775.224244.31' -> pg 230.9ffcaf6c (230.2c) -> up ([62,56,60], p62) acting ([62,56,60], p62) OSD: 62,56,60 Problematic OSD: 62 Problematic PG: 230.2c digging deeper.. I checked that when I run the command: [root@mon-1 ~]# radosgw-admin bilog list --bucket bucket_1 [] Even when there are no bilogs, there are the same symptoms: 1) In logs: 2020-01-08 16:31:24.728 7f1fa22dd700 0 bluestore(/var/lib/ceph/osd/ceph-62) log_latency_fn slow operation observed for upper_bound, latency = 28.653s, after = �0_ omap_iterator(cid = 230.2c_head, oid = #230:36f53ff9:::.dir.12307ed8-2342-4293-9f14-12234220a234.13221775.224244.31:head#) 2) Any other radosgw-admin commands are very slow while other process is iterating through bilogs (of this bucket). [dc-2 root@mon-1 ~]# time radosgw-admin bucket stats --bucket=bucket234 ... real 0m25.641s user 0m0.148s sys 0m0.075s When i try to run deep scrub on this pg 230.2c, in logs appears: [root@mon-1 ~]# ceph pg deep-scrub 230.2c 2020-01-08 16:38:32.551 7f1fc50f0700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f1fa22dd700' had timed out after 15 ... 2020-01-08 16:38:46.735 7f1fc60f2700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f1fa62e5700' had timed out after 15 2020-01-08 16:38:46.735 7f1fc60f2700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f1fa22dd700' had timed out after 15 2020-01-08 16:38:46.735 7f1fc60f2700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f1fa62e5700' had timed out after 15 2020-01-08 16:38:46.911 7f1fa22dd700 0 bluestore(/var/lib/ceph/osd/ceph-62) log_latency_fn slow operation observed for upper_bound, latency = 29.8655s, after = <80>0_ omap_iterator(cid = 230.2c_head, oid = #230:36f53ff9:::.dir.12307ed8-2342-4293-9f14-12234220a234.13221775.224244.31:head#) 2020-01-08 16:38:46.911 7f1fa22dd700 1 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7f1fa22dd700' had timed out after 15 2020-01-08 16:38:46.911 7f1fa62e5700 1 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7f1fa62e5700' had timed out after 15 2020-01-08 10:37:53.054 7f1fa22dd700 0 bluestore(/var/lib/ceph/osd/ceph-62) log_latency slow operation observed for next, latency = 22.4641s When i try to iterate over listomapkeys for: .dir.12307ed8-2342-4293-9f14-12234220a234.13221775.224244.31 [root@mon-1 ~]# rados -p default.rgw.buckets.index listomapkeys .dir.12307ed8-2342-4293-9f14-12234220a234.13221775.224244.31 In logs i observe: 2020-01-09 11:08:58.847 7f1fa62e5700 0 bluestore(/var/lib/ceph/osd/ceph-62) log_latency slow operation observed for next, latency = 24.1949s The same for listomapvals [root@mon-1 ~]# rados -p default.rgw.buckets.index listomapvals .dir.12307ed8-2342-4293-9f14-12234220a234.13221775.224244.31 2020-01-09 11:11:36.508 7f1fa62e5700 0 bluestore(/var/lib/ceph/osd/ceph-62) log_latency slow operation observed for next, latency = 28.2137s I tried to purge and recreate this OSD, also I trimmed bilogs for this bucket (radosgw-admin bilog trim --bucket bucket_1), but this didn't help. I suspect that any operations on PG: 230.2c on OSD: 62 can causing slow operations on my cluster. I have no idea how to debug/resolve this problem. Any help/hint is appreciated! Thanks, Piotr

3 3

Ceph BoF at SCALE 18x
by Mike Perez 13 Jan '20

13 Jan '20

Hi everyone, SCALE 18x is taking place at the same time as of Cephalocon 2020 in Seoul. I wanted to see if there is interest in me registering us a BoF slot and if anybody in the area will be attending to run it? https://www.socallinuxexpo.org/scale/18x/bird-feather-sessions -- Mike Perez he/him Ceph Community Manager M: +1-951-572-2633 494C 5D25 2968 D361 65FB 3829 94BC D781 ADA8 8AEA @Thingee <https://twitter.com/thingee> Thingee <https://www.linkedin.com/thingee> <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

1 0

where does 100% RBD utilization come from?
by Philip Brown 10 Jan '20

10 Jan '20

oops. I posted this to the "Old" list, but supposedly this is the new list and the better place to ask questions? A google search didnt seem to find the answer on this, so thought I'd ask here: what determines if an rdb is "100% busy"? I have some backend OSDs, and an iSCSI gateway, serving out some RBDs. iostat on the gateway says rbd is 100% utilized iostat on individual OSds only goes as high as about 60% on a per-device basis. CPU is idle. Doesnt seem like network interface is capped either. So.. how do I improve RBD throughput?

1 0

heads up about the pg autoscaler
by Dan van der Ster 10 Jan '20

10 Jan '20

Hi all, Just a quick heads up that if you're using the pg-autoscaler with EC pools, it looks to me that it creates too many PGs. See https://tracker.ceph.com/issues/43546 for the details. As a workaround you can decrease mon_target_pg_per_osd accordingly. Cheers, Dan

1 0

Trying to install nautilus, keep getting mimic
by Jorge Garcia 10 Jan '20

10 Jan '20

I have been trying to install Nautilus using ceph-deploy, but somehow keep getting mimic installed. I'm not sure where mimic is even coming from... Here's what I'm doing (basically following the docs): * Fresh install of Centos 7.7 sudo yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm cat << EOM > /etc/yum.repos.d/ceph.repo [ceph-noarch] name=Ceph noarch packages baseurl=https://download.ceph.com/rpm-nautilus/el7/noarch enabled=1 gpgcheck=1 type=rpm-md gpgkey=https://download.ceph.com/keys/release.asc EOM sudo yum update sudo yum install ceph-deploy * I also install ntp and openssh * Create a ceph-deploy user, enable sudo and distribute keys mkdir my-cluster cd my-cluster ceph-deploy new node1 ceph-deploy install node1 [ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephpub/.cephdeploy.conf [ceph_deploy.cli][INFO ] Invoked (2.0.1): /bin/ceph-deploy install node1 [ceph_deploy.cli][INFO ] ceph-deploy options: [ceph_deploy.cli][INFO ] verbose : False [ceph_deploy.cli][INFO ] testing : None [ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f26ef6e4518> [ceph_deploy.cli][INFO ] cluster : ceph [ceph_deploy.cli][INFO ] dev_commit : None [ceph_deploy.cli][INFO ] install_mds : False [ceph_deploy.cli][INFO ] stable : None [ceph_deploy.cli][INFO ] default_release : False [ceph_deploy.cli][INFO ] username : None [ceph_deploy.cli][INFO ] adjust_repos : True [ceph_deploy.cli][INFO ] func : <function install at 0x7f26f07c3a28> [ceph_deploy.cli][INFO ] install_mgr : False [ceph_deploy.cli][INFO ] install_all : False [ceph_deploy.cli][INFO ] repo : False [ceph_deploy.cli][INFO ] host : ['node1'] [ceph_deploy.cli][INFO ] install_rgw : False [ceph_deploy.cli][INFO ] install_tests : False [ceph_deploy.cli][INFO ] repo_url : None [ceph_deploy.cli][INFO ] ceph_conf : None [ceph_deploy.cli][INFO ] install_osd : False [ceph_deploy.cli][INFO ] version_kind : stable [ceph_deploy.cli][INFO ] install_common : False [ceph_deploy.cli][INFO ] overwrite_conf : False [ceph_deploy.cli][INFO ] quiet : False [ceph_deploy.cli][INFO ] dev : master [ceph_deploy.cli][INFO ] nogpgcheck : False [ceph_deploy.cli][INFO ] local_mirror : None [ceph_deploy.cli][INFO ] release : None [ceph_deploy.cli][INFO ] install_mon : False [ceph_deploy.cli][INFO ] gpg_url : None [ceph_deploy.install][DEBUG ] Installing stable version *mimic* on cluster ceph hosts node1 Where did it get that I wanted to install *mimic*? It all goes downhill from there, with my repo getting replaced by the mimic repo and mimic getting installed. What am I missing?

2 1

Multi-site clusters
by eduard.rushanyan＠centro.net 09 Jan '20

09 Jan '20

CEPH Community! Cheers to 2020 We have a single cluster running in two modes: CephFS as mountable filesystem and Rados Object Gateway (S3 like for storing objects) - all good there no questions, CEPH is awesome! Were planning to get 2nd cluster in a different Datacenter and have both clusters connected / replicated / mirrored (e.g. multi-site). Is there a way to achieve that for CephFS and Rados modes, any ideas / recommendations / best practices / articles someone can share? Any insight would be much, much appreciated! P.S. The RBD mirroring which seemed like is the solution - went as far as discovering that RBD mirroring works for block devices only, can’t be used with CephFS. There are some references as to Rados supporting multi-site with potentially significant impact on write latency. Thanks, Eduard

1 1

v14.2.6 Nautilus released
by Abhishek Lekshmanan 09 Jan '20

09 Jan '20

We're happy to announce the availability of the sixth update to the Ceph Nautilus release series. This is a hotfix release primarily fixing a regression introduced in v14.2.5, all nautilus users are advised to upgrade to this release. Notable Changes --------------- * This release fixes a `ceph-mgr` bug that caused mgr becoming unresponsive on larger clusters issue#43364 https://tracker.ceph.com/issues/43364, pr#32466 (https://github.com/ceph/ceph/pull/32466 David Zafman, Neha Ojha) The release notes can be also found at https://ceph.io/releases/v14-2-6-nautilus-released/ Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-14.2.6.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9 -- Abhishek Lekshmanan SUSE Software Solutions Germany GmbH

1 0

S3 Object Lock feature in 14.2.5
by Robert Sander 09 Jan '20

09 Jan '20

Hi, Ceph Nautilus 14.2.5 got the S3 Object Lock feature backported. It should now be possible to create such a bucket with: s3cmd --add-header=x-amz-bucket-object-lock-enabled:true mb s3://lockedbucket But an S3 client does not recognize the bucket having the Object Lock feature. Am I missing something here? Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 93818 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

why osd's heartbeat partner comes from another root tree?
by opengers 08 Jan '20

08 Jan '20

Hello: According to my understanding, osd's heartbeat partners only come from those osds who assume the same pg See below(# ceph osd tree), osd.10 and osd.0-6 cannot assume the same pg, because osd.10 and osd.0-6 are from different root tree, and pg in my cluster doesn't map across root trees(# ceph osd crush rule dump). so, osd.0-6 cannot become the heartbeat partner of osd.10 But, below is the log on osd.10, It can be seen that the osd.10's heartbeat partner include osd.0/1/2/5, why? thanks for any help > # osd.10 log > 2019-11-20 09:21:50.431799 7fbb369fb700 -1 osd.10 7344 heartbeat_check: no > reply from 10.13.6.162:6806 osd.2 since back 2019-11-20 09:21:19.979712 > front 2019-11-20 09:21:19.979712 (cutoff 2019-11-20 09:21:30.431768) > 2019-11-20 13:15:59.175060 7fbb369fb700 -1 osd.10 7357 heartbeat_check: no > reply from 10.13.6.162:6806 <http://10.13.6.152:6806> osd.2 since back > 2019-11-20 13:15:38.710424 front 2019-11-20 13:15:38.710424 (cutoff > 2019-11-20 13:15:39.175058) > 2019-11-20 13:15:59.175110 7fbb369fb700 -1 osd.10 7357 heartbeat_check: no > reply from 10.13.6.160:6803 osd.0 since back 2019-11-20 13:15:38.710424 > front 2019-11-20 13:15:38.710424 (cutoff 2019-11-20 13:15:39.175058) > 2019-11-20 13:15:59.175118 7fbb369fb700 -1 osd.10 7357 heartbeat_check: no > reply from 10.13.6.161:6803 osd.1 since back 2019-11-20 13:15:38.710424 > front 2019-11-20 13:15:38.710424 (cutoff 2019-11-20 13:15:39.175058) > 2019-11-21 02:52:24.656783 7fbb369fb700 -1 osd.10 7374 heartbeat_check: no > reply from 10.13.6.158:6810 osd.5 since back 2019-11-21 02:52:04.557548 > front 2019-11-21 02:52:04.557548 (cutoff 2019-11-21 02:52:04.656781) > # ceph osd tree > -17 3.29095 root ssd-storage > > -25 1.09698 rack rack-ssd-A > > -18 1.09698 host ssd-osd01 > 10 hdd 1.09698 osd.10 up 1.00000 > 1.00000 > -26 1.09698 rack rack-ssd-B > > -19 1.09698 host ssd-osd02 > 11 hdd 1.09698 osd.11 up 1.00000 > 1.00000 > -27 1.09698 rack rack-ssd-C > > -20 1.09698 host ssd-osd03 > 12 hdd 1.09698 osd.12 up 1.00000 > 1.00000 > -1 3.22256 root default > > -3 0.29300 host test-osd01 > > 0 hdd 0.29300 osd.0 up 1.00000 > 1.00000 > -5 0.29300 host test-osd02 > > 1 hdd 0.29300 osd.1 up 0.89999 > 1.00000 > -7 0.29300 host test-osd03 > > 2 hdd 0.29300 osd.2 up 0.79999 > 1.00000 > -9 0.29300 host test-osd04 > > 3 hdd 0.29300 osd.3 up 1.00000 > 1.00000 > -11 0.29300 host test-osd05 > > 4 hdd 0.29300 osd.4 up 1.00000 > 1.00000 > -13 0.29300 host test-osd06 > > 5 hdd 0.29300 osd.5 up 1.00000 > 1.00000 > -15 0.29300 host test-osd07 > > 6 hdd 0.29300 osd.6 up 1.00000 > 1.00000 # ceph osd crush rule dump > > [ > { > "rule_id": 0, > "rule_name": "replicated_rule", > "ruleset": 0, > "type": 1, > "min_size": 1, > "max_size": 10, > "steps": [ > { > "op": "take", > "item": -1, > "item_name": "default" > }, > { > "op": "chooseleaf_firstn", > "num": 0, > "type": "host" > }, > { > "op": "emit" > } > ] > }, > { > "rule_id": 1, > "rule_name": "replicated_rule_ssd", > "ruleset": 1, > "type": 1, > "min_size": 1, > "max_size": 10, > "steps": [ > { > "op": "take", > "item": -17, > "item_name": "ssd-storage" > }, > { > "op": "chooseleaf_firstn", > "num": 0, > "type": "rack" > }, > { > "op": "emit" > } > ] > } > ] > # some parameters > "mon_osd_down_out_interval": "600", > "mon_osd_down_out_subtree_limit": "rack", > "mds_debug_subtrees": "false", > "mon_osd_down_out_subtree_limit": "rack", > "mon_osd_reporter_subtree_level": "host",

3 3

Poor performance after (incomplete?) upgrade to Nautilus
by Georg F 08 Jan '20

08 Jan '20

Hi all, since a few weeks our Nautilus cluster was struggling with severe performance issues. When an OSD would go down, the rebalancing was really slow. Long periods with no data transfer at all (client and rebalancing!) and times with rebalancing traffic only. However, client traffic was almost stalled for the whole period until all objects were in place again (VMs were frozen). PGs were stuck in peering or inactive for long times. Sometimes we had to restart the ceph-mon in order to get the whole process running again. The issues started all of a sudden, we don't remember doing any changes to the configuration. The whole cluster has been updated from Mimic to Nautilus (14.2.3) in September while the issue occurred just a few weeks ago. Updating it to 14.2.5 did not resolve the issue back then. Looking through mailing lists I found the following message: http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-July/028035.html So I ran "ceph osd require-osd-release nautilus" and all of a sudden the problems where gone! I do not recall executing that command right after the upgrade because the documentation states "Complete the upgrade by disallowing pre-Nautilus OSDs and enabling all new Nautilus-only functionality.". As by that point in time all OSDs, MONs and MGRs were successfully updated there was no reason to believe this command would be necessary. Therefore I got two questions: 1. What exactly does the command do besides preventing old OSDs from joining? 2. What could have been the issue with the cluster and how did this command fix it? If it is really that important to run the command, the docs should state this more clearly. I appreciate any insight on this topic. Thanks, Georg

1 0

ceph balancer <argument> runs for minutes or hangs
by Thomas Schneider 08 Jan '20

08 Jan '20

Hi, happy new year to you! I'm running a multinode cluster with 3 MGR nodes. The issue I'm facing now is that ceph balancer <argument> runs for minutes or, in worst case, hangs. I have documented the runtime of the following executions: root@ld3955:~# date && time ceph balancer status Mon Dec 23 10:06:12 CET 2019 { "active": true, "plans": [], "mode": "upmap" } real 1m45,045s user 0m0,315s sys 0m0,026s root@ld3955:~# date && time ceph balancer status Tue Jan 7 08:11:24 CET 2020 ^CInterrupted Traceback (most recent call last): File "/usr/bin/ceph", line 1263, in <module> retval = main() File "/usr/bin/ceph", line 1194, in main verbose) File "/usr/bin/ceph", line 619, in new_style_command ret, outbuf, outs = do_command(parsed_args, target, cmdargs, sigdict, inbuf, verbose) File "/usr/bin/ceph", line 593, in do_command return ret, '', '' UnboundLocalError: local variable 'ret' referenced before assignment real 102m44,084s user 0m2,404s sys 0m1,065s root@ld3955:~# date && time ceph balancer off Tue Jan 7 09:57:36 CET 2020 real 1m45,371s user 0m0,358s sys 0m0,013s root@ld3955:~# date && time ceph balancer on Tue Jan 7 14:57:03 CET 2020 real 0m0,452s user 0m0,284s sys 0m0,020s root@ld3955:~# date && time ceph balancer status Tue Jan 7 14:57:11 CET 2020 { "active": true, "plans": [], "mode": "upmap" } real 1m52,902s user 0m0,301s sys 0m0,042s root@ld3955:~# date && time ceph balancer off Wed Jan 8 08:49:26 CET 2020 ^CInterrupted Traceback (most recent call last): File "/usr/bin/ceph", line 1263, in <module> retval = main() File "/usr/bin/ceph", line 1194, in main verbose) File "/usr/bin/ceph", line 619, in new_style_command ret, outbuf, outs = do_command(parsed_args, target, cmdargs, sigdict, inbuf, verbose) File "/usr/bin/ceph", line 593, in do_command return ret, '', '' UnboundLocalError: local variable 'ret' referenced before assignment real 14m29,097s user 0m0,579s sys 0m0,157s In correlation with this finding I have identified that active MGR node is using +100% CPU, to be pricise 108-120%. To workaround this issue I must stop the active MRG node service and wait until another node becomes active. What's the issue with MGR service here? Should I open a bug report? Regards

1 0

RBD Mirroring down+unknown
by miguel.castillo＠centro.net 07 Jan '20

07 Jan '20

Happy New Year Ceph Community! I'm in the process of figuring out RBD mirroring with Ceph and having a really tough time with it. I'm trying to set up just one way mirroring right now on some test systems (baremetal servers, all Debian 9). The first cluster is 3 nodes, and the 2nd cluster is 2 nodes (not worried about a properly performing setup, just the functionality of RBD mirroring right now). The purpose is to have a passive failover ceph cluster in a separate DC. Mirroring seems like the best solution, but if we can't get it working, we'll end up resorting to a scheduled rsync which is less than ideal. I've followed several guides, read through a lot of documentation, and nothing has worked for me thus far. If anyone can offer some troubleshooting help or insight into what I might have missed in this setup, I'd greatly appreciate it! I also don't fully understand the relationship between images and pools and how you're supposed to configure statically sized images for a pool that has a variable amount of data, but that's a question for afterwards, I think :) Once RBD mirroring is set up, the mirror test image status shows as down+unknown: On ceph1-dc2: rbd --cluster dc1ceph mirror pool status fs_data --verbose health: WARNING images: 1 total 1 unknown mirror_test: global_id: c335017c-9b8f-49ee-9bc1-888789537c47 state: down+unknown description: status not found last_update: Here are the commands I run using ceph-deploy on both clusters to get everything up and running (run from a deploy directory on the first node of each cluster). The clusters are created at the same time, and rbd setup commands are only run after the clusters are up and healthy, and the fs_data pool is created. ----------------------------------------------------------- Cluster 1 (dc1ceph): ceph-deploy new ceph1-dc1 ceph2-dc1 ceph3-dc1 sed -i '$ s,.*,public_network = *.*.*.0/24\n,g' ceph.conf ceph-deploy install ceph1-dc1 ceph2-dc1 ceph3-dc1 --release luminous ceph-deploy mon create-initial ceph-deploy admin ceph1-dc1 ceph2-dc1 ceph3-dc1 ceph-deploy mgr create ceph1-dc1 ceph2-dc1 ceph3-dc1 for x in b c d e f g h i j k; do ceph-deploy osd create --data /dev/sd${x}1 ceph1-dc1 ; done for x in b c d e f g h i j k; do ceph-deploy osd create --data /dev/sd${x}1 ceph2-dc1 ; done for x in b c d e f g h i j k; do ceph-deploy osd create --data /dev/sd${x}1 ceph3-dc1 ; done ceph-deploy mds create ceph1-dc1 ceph2-dc1 ceph3-dc1 ceph-deploy rgw create ceph1-dc1 ceph2-dc1 ceph3-dc1 for f in 1 2 ; do scp ceph.client.admin.keyring ceph$f-dc2:/etc/ceph/dc1ceph.client.admin.keyring ; done for f in 1 2 ; do scp ceph.conf ceph$f-dc2:/etc/ceph/dc1ceph.conf ; done for f in 1 2 ; do ssh ceph$f-dc2 "chown ceph.ceph /etc/ceph/dc1ceph*" ; done ceph osd pool create fs_data 512 512 replicated rbd --cluster ceph mirror pool enable fs_data image rbd --cluster dc2ceph mirror pool enable fs_data image rbd --cluster ceph mirror pool peer add fs_data client.admin@dc2ceph (generated id: b5e347b3-0515-4142-bc49-921a07636865) rbd create fs_data/mirror_test --size=1G rbd feature enable fs_data/mirror_test journaling rbd mirror image enable fs_data/mirror_test chown ceph.ceph ceph.client.admin.keyring Cluster 2 (dc2ceph): ceph-deploy new ceph1-dc2 ceph2-dc2 sed -i '$ s,.*,public_network = *.*.*.0/24\n,g' ceph.conf ceph-deploy install ceph1-dc2 ceph2-dc2 --release luminous ceph-deploy mon create-initial ceph-deploy admin ceph1-dc2 ceph2-dc2 ceph-deploy mgr create ceph1-dc2 ceph2-dc2 for x in b c d e f g h i j k; do ceph-deploy osd create --data /dev/sd${x}1 ceph1-dc2 ; done for x in b c d e f g h i j k; do ceph-deploy osd create --data /dev/sd${x}1 ceph2-dc2 ; done ceph-deploy mds create ceph1-dc2 ceph2-dc2 ceph-deploy rgw create ceph1-dc2 ceph2-dc2 apt install rbd-mirror for f in 1 2 3 ; do scp ceph.conf ceph$f-dc1:/etc/ceph/dc2ceph.conf ; done for f in 1 2 3 ; do scp ceph.client.admin.keyring ceph$f-dc1:/etc/ceph/dc2ceph.client.admin.keyring ; done for f in 1 2 3 ; do ssh ceph$f-dc1 "chown ceph.ceph /etc/ceph/dc2ceph*" ; done ceph osd pool create fs_data 512 512 replicated rbd --cluster ceph mirror pool peer add fs_data client.admin@dc1ceph (generated id: e486c401-e24d-49bc-9800-759760822282) systemctl enable ceph-rbd-mirror@admin systemctl start ceph-rbd-mirror@admin rbd --cluster dc1ceph mirror pool status fs_data --verbose Cluster 1: ls /etc/ceph: ceph.client.admin.keyring ceph.conf dc2ceph.client.admin.keyring dc2ceph.conf rbdmap tmpG36OYs cat /etc/ceph/ceph.conf: [global] fsid = 8fede407-50e1-4487-8356-3dc98b30c500 mon_initial_members = ceph1-dc1, ceph2-dc1, ceph3-dc1 mon_host = *.*.*.1,*.*.*.27,*.*.*.41 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx public_network = *.*.*.0/24 cat /etc/ceph/dc2ceph.conf [global] fsid = 813ff410-02dc-47bd-b678-38add38495bb mon_initial_members = ceph1-dc2, ceph2-dc2 mon_host = *.*.*.56,*.*.*.0 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx public_network = *.*.*.0/24 Cluster 2: ls /etc/ceph: ceph.client.admin.keyring ceph.conf dc1ceph.client.admin.keyring dc1ceph.conf rbdmap tmp_yxkPs cat /etc/ceph/ceph.conf [global] fsid = 813ff410-02dc-47bd-b678-38add38495bb mon_initial_members = ceph1-dc2, ceph2-dc2 mon_host = *.*.*.56,*.*.*.70 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx public_network = *.*.*.0/24 cat /etc/ceph/dc1ceph.conf [global] fsid = 8fede407-50e1-4487-8356-3dc98b30c500 mon_initial_members = ceph1-dc1, ceph2-dc1, ceph3-dc1 mon_host = *.*.*.1,*.*.*.27,*.*.*.41 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx public_network = *.*.*.0/24 RBD Mirror daemon status: ceph-rbd-mirror(a)admin.service - Ceph rbd mirror daemon Loaded: loaded (/lib/systemd/system/ceph-rbd-mirror@.service; enabled; vendor preset: enabled) Active: inactive (dead) since Mon 2020-01-06 16:21:44 EST; 3s ago Process: 910178 ExecStart=/usr/bin/rbd-mirror -f --cluster ${CLUSTER} --id admin --setuser ceph --setgroup ceph (code=exited, status=0/SUCCESS) Main PID: 910178 (code=exited, status=0/SUCCESS) Jan 06 16:21:44 ceph1-dc2 systemd[1]: Started Ceph rbd mirror daemon. Jan 06 16:21:44 ceph1-dc2 rbd-mirror[910178]: 2020-01-06 16:21:44.462916 7f76ecf88780 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory Jan 06 16:21:44 ceph1-dc2 rbd-mirror[910178]: 2020-01-06 16:21:44.462949 7f76ecf88780 -1 monclient: ERROR: missing keyring, cannot use cephx for authentication Jan 06 16:21:44 ceph1-dc2 rbd-mirror[910178]: failed to initialize: (2) No such file or directory2020-01-06 16:21:44.463874 7f76ecf88780 -1 rbd::mirror::Mirror: 0x558d3ce6ce20 init: error connecting to local cluster ------------------------------------------- I also tried running the ExecStart command manually, substituting in different values for the parameters, and just never got it to work. If more info is needed, please don’t hesitate to ask. Thanks in advance! -Miguel

2 4

MDS failing under load with large cache sizes
by Janek Bevendorff 07 Jan '20

07 Jan '20

Hi, I am trying to copy the contents of our storage server into a CephFS, but am experiencing stability issues with my MDSs. The CephFS sits on top of an erasure-coded pool with 5 MONs, 5 MDSs and a max_mds setting of two. My Ceph cluster version is Nautilus, the client is Mimic and uses the kernel module to mount the FS. The index of filenames to copy is about 23GB and I am using 16 parallel rsync processes over a 10G link to copy the files over to Ceph. This works perfectly for a while, but then the MDSs start reporting oversized caches (between 20 and 50GB, sometimes more) and an inode count between 1 and 4 million. Particularly the Inode count seems quite high to me. Each rsync job has 25k files to work with, so if all 16 processes open all their files at the same time, I should not exceed 400k. Even if I double this number to account for the client's page cache, I should get nowhere near that number of inodes (a sync flush takes about 1 second). Then after a few hours, my MDSs start failing with messages like this: -21> 2019-07-22 14:00:05.877 7f67eacec700 1 heartbeat_map is_healthy 'MDSRank' had timed out after 15 -20> 2019-07-22 14:00:05.877 7f67eacec700 0 mds.beacon.XXX Skipping beacon heartbeat to monitors (last acked 24.0042s ago); MDS internal heartbeat is not healthy! The standby nodes try to take over, but take forever to become active and will fail as well eventually. During my research, I found this related topic: http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-January/015959.html, but I tried everything in there from increasing to lowering my cache size, the number of segments etc. I also played around with the number of active MDSs and two appears to work the best, whereas one cannot keep up with the load and three seems to be the worst of all choices. Do you have any ideas how I can improve the stability of my MDS damons to handle the load properly? single 10G link is a toy and we could query the cluster with a lot more requests per second, but it's already yielding to 16 rsync processes. Thanks

4 33

Disk fail, some question...
by Marco Gaiarin 07 Jan '20

07 Jan '20

Happy new year to all! In these holidays i've suffered a disk failure, but hitted also an 'inconsistent pg' error, and i want to understand. Ceph 12.2.12, filestore. Starting from 27/12 i get classical disk error: Dec 27 20:52:21 capitanmarvel kernel: [345907.286795] ata1.00: exception Emask 0x0 SAct 0xfe00000 SErr 0x0 action 0x0 Dec 27 20:52:21 capitanmarvel kernel: [345907.286849] ata1.00: irq_stat 0x40000008 Dec 27 20:52:21 capitanmarvel kernel: [345907.286880] ata1.00: failed command: READ FPDMA QUEUED Dec 27 20:52:21 capitanmarvel kernel: [345907.286920] ata1.00: cmd 60/00:a8:20:87:3b/04:00:00:00:00/40 tag 21 ncq dma 524288 in Dec 27 20:52:21 capitanmarvel kernel: [345907.286920] res 41/40:00:46:8a:3b/00:00:00:00:00/40 Emask 0x409 (media error) <F> Dec 27 20:52:21 capitanmarvel kernel: [345907.287018] ata1.00: status: { DRDY ERR } Dec 27 20:52:21 capitanmarvel kernel: [345907.287046] ata1.00: error: { UNC } Dec 27 20:52:21 capitanmarvel kernel: [345907.288676] ata1.00: configured for UDMA/133 Dec 27 20:52:21 capitanmarvel kernel: [345907.288698] sd 1:0:0:0: [sdc] tag#21 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Dec 27 20:52:21 capitanmarvel kernel: [345907.288702] sd 1:0:0:0: [sdc] tag#21 Sense Key : Medium Error [current] Dec 27 20:52:21 capitanmarvel kernel: [345907.288705] sd 1:0:0:0: [sdc] tag#21 Add. Sense: Unrecovered read error - auto reallocate failed Dec 27 20:52:21 capitanmarvel kernel: [345907.288708] sd 1:0:0:0: [sdc] tag#21 CDB: Read(10) 28 00 00 3b 87 20 00 04 00 00 Dec 27 20:52:21 capitanmarvel kernel: [345907.288711] print_req_error: I/O error, dev sdc, sector 3902022 but also: Dec 27 20:52:24 capitanmarvel ceph-osd[3852]: 2019-12-27 20:52:24.714716 7f821fbfd700 -1 log_channel(cluster) log [ERR] : 4.9b missing primary copy of 4:d97871c4:::rbd_data.142b816b8b4567.0000000000012ae1:head, will try copies on 8,14 OSD 'flip-flop' a bit for some days. At first scrub, i got: cluster: id: 8794c124-c2ec-4e81-8631-742992159bd6 health: HEALTH_ERR 1 scrub errors Possible data damage: 1 pg inconsistent services: mon: 5 daemons, quorum blackpanther,capitanmarvel,4,2,3 mgr: hulk(active), standbys: blackpanther, deadpool, thor, capitanmarvel osd: 12 osds: 12 up, 12 in data: pools: 3 pools, 768 pgs objects: 671.04k objects, 2.54TiB usage: 7.62TiB used, 9.66TiB / 17.3TiB avail pgs: 766 active+clean 1 active+clean+inconsistent 1 active+clean+scrubbing+deep finally, OSD die, and so i got (after automatic remapping): cluster: id: 8794c124-c2ec-4e81-8631-742992159bd6 health: HEALTH_ERR 1 scrub errors Possible data damage: 1 pg inconsistent services: mon: 5 daemons, quorum blackpanther,capitanmarvel,4,2,3 mgr: hulk(active), standbys: blackpanther, deadpool, thor, capitanmarvel osd: 12 osds: 11 up, 11 in data: pools: 3 pools, 768 pgs objects: 674.26k objects, 2.55TiB usage: 7.65TiB used, 8.71TiB / 16.4TiB avail pgs: 767 active+clean 1 active+clean+inconsistent To fix the issue i've tried to read the docs (looking for 'OSD_SCRUB_ERRORS'), finding: https://docs.ceph.com/docs/doc-12.2.0-major-changes/rados/operations/health… but the link within is empty: https://docs.ceph.com/docs/doc-12.2.0-major-changes/rados/operations/pg-rep… and after fiddling a bit with google, i've found: https://ceph.io/geen-categorie/ceph-manually-repair-object/ that permit me to fix the issue easily with 'ceph pg repair'. Two question: 1) the missing page on 'pg-repair' is a bug of documentation? There's something i can do? 2) what happens? - While, if the OSD was not able to write data to the OSD, they are not automatically relocated to other OSD? This violate the crushmap? - while, when the failing OSD get out, the inconsistent PG get not automatically fixed? I've count=3, the other 2 copies are not coherent? But, if so, how ceph was able to fix them? Sorry... and thanks. ;) -- dott. Marco Gaiarin GNUPG Key ID: 240A3D66 Associazione ``La Nostra Famiglia'' http://www.lanostrafamiglia.it/ Polo FVG - Via della Bontà, 7 - 33078 - San Vito al Tagliamento (PN) marco.gaiarin(at)lanostrafamiglia.it t +39-0434-842711 f +39-0434-842797 Dona il 5 PER MILLE a LA NOSTRA FAMIGLIA! http://www.lanostrafamiglia.it/index.php/it/sostienici/5x1000 (cf 00307430132, categoria ONLUS oppure RICERCA SANITARIA)

1 0

RESEND: Re: PG Balancer Upmap mode not working
by David Zafman 07 Jan '20

07 Jan '20

Please file a tracker with the symptom and examples. Please attach your OSDMap (ceph osd getmap > osdmap.bin). Note that https://github.com/ceph/ceph/pull/31956 has the Nautilus version of improved upmap code. It also changes osdmaptool to match the mgr behavior, so that one can observe the behavior of the upmap balancer offline. Thanks David On 12/8/19 11:04 AM, Philippe D'Anjou wrote: > It's only getting worse after raising PGs now. > > Anything between: > 96 hdd 9.09470 1.00000 9.1 TiB 4.9 TiB 4.9 TiB 97 KiB 13 GiB 4.2 > TiB 53.62 0.76 54 up > > and > > 89 hdd 9.09470 1.00000 9.1 TiB 8.1 TiB 8.1 TiB 88 KiB 21 GiB 1001 > GiB 89.25 1.27 87 up > > How is that possible? I dont know how much more proof I need to > present that there's a bug. > > > > _______________________________________________ > ceph-users mailing list > ceph-users(a)lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

5 20

RBD Mirroring down+unknown
by Miguel Castillo 06 Jan '20

06 Jan '20

Happy New Year Ceph Community! I'm in the process of figuring out RBD mirroring with Ceph and having a really tough time with it. I'm trying to set up just one way mirroring right now on some test systems (baremetal servers, all Debian 9). The first cluster is 3 nodes, and the 2nd cluster is 2 nodes (not worried about a properly performing setup, just the functionality of RBD mirroring right now). The purpose is to have a passive failover ceph cluster in a separate DC. Mirroring seems like the best solution, but if we can't get it working, we'll end up resorting to a scheduled rsync which is less than ideal. I've followed several guides, read through a lot of documentation, and nothing has worked for me thus far. If anyone can offer some troubleshooting help or insight into what I might have missed in this setup, I'd greatly appreciate it! I also don't fully understand the relationship between images and pools and how you're supposed to configure statically sized images for a pool that has a variable amount of data, but that's a question for afterwards, I think :) Once RBD mirroring is set up, the mirror test image status shows as down+unknown: On ceph1-dc2: rbd --cluster dc1ceph mirror pool status fs_data --verbose health: WARNING images: 1 total 1 unknown mirror_test: global_id: c335017c-9b8f-49ee-9bc1-888789537c47 state: down+unknown description: status not found last_update: Here are the commands I run using ceph-deploy on both clusters to get everything up and running (run from a deploy directory on the first node of each cluster). The clusters are created at the same time, and rbd setup commands are only run after the clusters are up and healthy, and the fs_data pool is created. ----------------------------------------------------------- Cluster 1 (dc1ceph): ceph-deploy new ceph1-dc1 ceph2-dc1 ceph3-dc1 sed -i '$ s,.*,public_network = *.*.*.0/24\n,g' ceph.conf ceph-deploy install ceph1-dc1 ceph2-dc1 ceph3-dc1 --release luminous ceph-deploy mon create-initial ceph-deploy admin ceph1-dc1 ceph2-dc1 ceph3-dc1 ceph-deploy mgr create ceph1-dc1 ceph2-dc1 ceph3-dc1 for x in b c d e f g h i j k; do ceph-deploy osd create --data /dev/sd${x}1 ceph1-dc1 ; done for x in b c d e f g h i j k; do ceph-deploy osd create --data /dev/sd${x}1 ceph2-dc1 ; done for x in b c d e f g h i j k; do ceph-deploy osd create --data /dev/sd${x}1 ceph3-dc1 ; done ceph-deploy mds create ceph1-dc1 ceph2-dc1 ceph3-dc1 ceph-deploy rgw create ceph1-dc1 ceph2-dc1 ceph3-dc1 for f in 1 2 ; do scp ceph.client.admin.keyring ceph$f-dc2:/etc/ceph/dc1ceph.client.admin.keyring ; done for f in 1 2 ; do scp ceph.conf ceph$f-dc2:/etc/ceph/dc1ceph.conf ; done for f in 1 2 ; do ssh ceph$f-dc2 "chown ceph.ceph /etc/ceph/dc1ceph*" ; done ceph osd pool create fs_data 512 512 replicated rbd --cluster ceph mirror pool enable fs_data image rbd --cluster dc2ceph mirror pool enable fs_data image rbd --cluster ceph mirror pool peer add fs_data client.admin@dc2ceph (generated id: b5e347b3-0515-4142-bc49-921a07636865) rbd create fs_data/mirror_test --size=1G rbd feature enable fs_data/mirror_test journaling rbd mirror image enable fs_data/mirror_test chown ceph.ceph ceph.client.admin.keyring Cluster 2 (dc2ceph): ceph-deploy new ceph1-dc2 ceph2-dc2 sed -i '$ s,.*,public_network = *.*.*.0/24\n,g' ceph.conf ceph-deploy install ceph1-dc2 ceph2-dc2 --release luminous ceph-deploy mon create-initial ceph-deploy admin ceph1-dc2 ceph2-dc2 ceph-deploy mgr create ceph1-dc2 ceph2-dc2 for x in b c d e f g h i j k; do ceph-deploy osd create --data /dev/sd${x}1 ceph1-dc2 ; done for x in b c d e f g h i j k; do ceph-deploy osd create --data /dev/sd${x}1 ceph2-dc2 ; done ceph-deploy mds create ceph1-dc2 ceph2-dc2 ceph-deploy rgw create ceph1-dc2 ceph2-dc2 apt install rbd-mirror for f in 1 2 3 ; do scp ceph.conf ceph$f-dc1:/etc/ceph/dc2ceph.conf ; done for f in 1 2 3 ; do scp ceph.client.admin.keyring ceph$f-dc1:/etc/ceph/dc2ceph.client.admin.keyring ; done for f in 1 2 3 ; do ssh ceph$f-dc1 "chown ceph.ceph /etc/ceph/dc2ceph*" ; done ceph osd pool create fs_data 512 512 replicated rbd --cluster ceph mirror pool peer add fs_data client.admin@dc1ceph (generated id: e486c401-e24d-49bc-9800-759760822282) systemctl enable ceph-rbd-mirror@admin systemctl start ceph-rbd-mirror@admin rbd --cluster dc1ceph mirror pool status fs_data --verbose Cluster 1: ls /etc/ceph: ceph.client.admin.keyring ceph.conf dc2ceph.client.admin.keyring dc2ceph.conf rbdmap tmpG36OYs cat /etc/ceph/ceph.conf: [global] fsid = 8fede407-50e1-4487-8356-3dc98b30c500 mon_initial_members = ceph1-dc1, ceph2-dc1, ceph3-dc1 mon_host = *.*.*.1,*.*.*.27,*.*.*.41 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx public_network = *.*.*.0/24 cat /etc/ceph/dc2ceph.conf [global] fsid = 813ff410-02dc-47bd-b678-38add38495bb mon_initial_members = ceph1-dc2, ceph2-dc2 mon_host = *.*.*.56,*.*.*.0 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx public_network = *.*.*.0/24 Cluster 2: ls /etc/ceph: ceph.client.admin.keyring ceph.conf dc1ceph.client.admin.keyring dc1ceph.conf rbdmap tmp_yxkPs cat /etc/ceph/ceph.conf [global] fsid = 813ff410-02dc-47bd-b678-38add38495bb mon_initial_members = ceph1-dc2, ceph2-dc2 mon_host = *.*.*.56,*.*.*.70 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx public_network = *.*.*.0/24 cat /etc/ceph/dc1ceph.conf [global] fsid = 8fede407-50e1-4487-8356-3dc98b30c500 mon_initial_members = ceph1-dc1, ceph2-dc1, ceph3-dc1 mon_host = *.*.*.1,*.*.*.27,*.*.*.41 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx public_network = *.*.*.0/24 RBD Mirror daemon status: ceph-rbd-mirror(a)admin.service - Ceph rbd mirror daemon Loaded: loaded (/lib/systemd/system/ceph-rbd-mirror@.service; enabled; vendor preset: enabled) Active: inactive (dead) since Mon 2020-01-06 16:21:44 EST; 3s ago Process: 910178 ExecStart=/usr/bin/rbd-mirror -f --cluster ${CLUSTER} --id admin --setuser ceph --setgroup ceph (code=exited, status=0/SUCCESS) Main PID: 910178 (code=exited, status=0/SUCCESS) Jan 06 16:21:44 ceph1-dc2 systemd[1]: Started Ceph rbd mirror daemon. Jan 06 16:21:44 ceph1-dc2 rbd-mirror[910178]: 2020-01-06 16:21:44.462916 7f76ecf88780 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory Jan 06 16:21:44 ceph1-dc2 rbd-mirror[910178]: 2020-01-06 16:21:44.462949 7f76ecf88780 -1 monclient: ERROR: missing keyring, cannot use cephx for authentication Jan 06 16:21:44 ceph1-dc2 rbd-mirror[910178]: failed to initialize: (2) No such file or directory2020-01-06 16:21:44.463874 7f76ecf88780 -1 rbd::mirror::Mirror: 0x558d3ce6ce20 init: error connecting to local cluster ------------------------------------------- I also tried running the ExecStart command manually, substituting in different values for the parameters, and just never got it to work. If more info is needed, please don't hesitate to ask. Thanks in advance! -Miguel

1 0

Balancing PGs across OSDs
by Thomas Schneider 06 Jan '20

06 Jan '20

Hi, in this <https://ceph.io/community/the-first-telemetry-results-are-in/> blog post I find this statement: "So, in our ideal world so far (assuming equal size OSDs), every OSD now has the same number of PGs assigned." My issue is that accross all pools the number of PGs per OSD is not equal. And I conclude that this is causing very unbalanced data placement. As a matter of fact the data stored on my 1.6TB HDD in specific pool "hdb_backup" is in a range starting with osd.228 size: 1.6 usage: 52.61 reweight: 1.00000 and ending with osd.145 size: 1.6 usage: 81.11 reweight: 1.00000 This impacts the amount of data that can be stored in the cluster heavily. Ceph balancer is enabled, but this is not solving this issue. root@ld3955:~# ceph balancer status { "active": true, "plans": [], "mode": "upmap" } Therefore I would ask you for suggestions how to work on this unbalanced data distribution. I have attached pastebin for - ceph osd df sorted by usage <https://pastebin.com/QLQHjA9g> - ceph osd df tree <https://pastebin.com/SvhP2hp5> My cluster has multiple crush roots respresenting different disks. In addition I have defined multiple pools, one pool for each disk type: hdd, ssd, nvme. THX

6 23

Re: report librbd bug export-diff
by Jason Dillaman 03 Jan '20

03 Jan '20

On Sun, Dec 29, 2019 at 9:21 PM zhengyin(a)cmss.chinamobile.com <zhengyin(a)cmss.chinamobile.com> wrote: > > Hello dillaman: > > Problem: create a clone from a parent image, create a snapshot on the clone, when I use command "rbd export-diff <pool>/clone@snap <file> --whole-object", it can't diff parent image data. But I use this command without "--whole-object", it is ok. > > steps: > 1、rbd create volumes/test1 -s 1G > > 2、write data to volumes/test1 (offset=0, len=8388608) > > 3、rbd snap create volumes/test1@snap > > 4、rbd snap protect volumes/test1@snap > > 5、rbd clone volumes/test1@snap volumes/clone1 > > 6、write data to volumes/clone1 (offset=16777216, len=4194304) > > 7、rbd snap create volumes/clone1@snap1 > > 8、rbd export-diff volumes/clone1@snap1 /root/diff1 --whole-object > It only diffs data [16777216L, 20971520L] > > 9、rbd export-diff volumes/clone1@snap1 /root/diff2 > It is ok and can diffs data [0L, 8388608L], [16777216L, 20971520L] > > If you confirm that this is a bug or fix it, please let me know, thank you very much. Is this the same as this ticket [1] that was recently fixed in master and is pending backport? > > ________________________________ > zhengyin(a)cmss.chinamobile.com [1] https://tracker.ceph.com/issues/42248 -- Jason

2 2

acting_primary is an osd with primary-affinity of 0, which seems wrong
by Wesley Dillingham 03 Jan '20

03 Jan '20

In an exploration of trying to speedup the long tail of backfills resulting from marking a failing OSD out I began looking at my PGs to see if i could tune some settings and noticed the following: Scenario: on a 12.2.12 Cluster, I am alerted of an inconsistent PG and am alerted of SMART failures on that OSD. I inspect that PG and notice it is a read_error from the SMART-failing osd. Steps I take: Set the primary affinity of the failing OSD to 0 (thought process being, I dont want a failing drive to be responsible for backfilling data), wait for peering to complete, then mark the OSD out. At this point backfill begins. 90% of the PGs complete backfill very quickly. Towards the tail end of the backfill I have 20 PGs or so in backfill_wait and 1 backfilling (presuming because of osd_max_backfills = 1). I do a `ceph pg ls backfill_wait` and notice that 100% of the tail end PGs are such that all OSDs in the up_set are different than those of acting_set and that the acting_primary is the OSD that was set with primary affinity 0 and marked out. My questions are the following: - Upon learning a disk has failed smart and has an inconsistent PG I want to prevent its potentially-corrupt data from being replicated out to other OSDs, even for PGs which may not have been discovered to be inconsistent yet so I set primary affinity to 0. At this step shouldn't the acting_primary be another OSD from the acting_set and backfill be copied out of a different OSD? - Should I be additionally marking the OSD as down, which would cause the PGs to go degraded until backfill finishes but would presumably finish faster as more OSDs would become the acting_primary and I wouldnt be throttled by osd_max_backfills. My thought here is its best to avoid degraded PGs as I do not want to drop below min_size. I recognize some of these things may be different in Nautilus but I am waiting on the 14.2.6 release as i am aware of some bugs I do not want to contend with. Thanks. Respectfully, *Wes Dillingham* wes(a)wesdillingham.com LinkedIn <http://www.linkedin.com/in/wesleydillingham>

1 0

rgw multisite rebuild
by Frank R 03 Jan '20

03 Jan '20

Hi all, It looks like I have an RGW multisite setup that I need to rebuild to get metadata syncing again (I did some stupid things to break it). If it possible to remove the slave zone from the zonegroup and then re-add it without destroying the rgw data pool (bucket data)? thx Frank

1 0

radosgw - Etags suffixed with #x0e
by Ingo Reimann 03 Jan '20

03 Jan '20

Hi, We had a strange problem with some buckets. After a s3cmd sync, some objects got ETAGs with the suffix "#x0e". This rendered the XML output of "GET /" e.g. (s3cmd du) invalid. Unfortunately, this behaviour was not reproducable but could be fixed by "GET /{object}" + "PUT /{object}" (s3cmd get + s3cmd put). I am not sure, how this appeared and how to avoid that. Just now, we have nautilus mons and osds with jewel radosgws. At the time of first appearence, also a nautilus gateway had been online, but the requests had been handled by both types. Any ideas? best regards, Ingo -- Ingo Reimann Teamleiter Technik [ https://www.dunkel.de/ ] Dunkel GmbH Philipp-Reis-Straße 2 65795 Hattersheim Fon: +49 6190 889-100 Fax: +49 6190 889-399 eMail: support(a)dunkel.de http://www.Dunkel.de/ Amtsgericht Frankfurt/Main HRB: 37971 Geschäftsführer: Axel Dunkel Ust-ID: DE 811622001

2 4

Experience with messenger v2 in Nautilus
by Stefan Kooman 03 Jan '20

03 Jan '20

Hi, I'm wondering how many of are using messenger v2 in Nautilus after upgrading from a previous release (Luminous / Mimic). Does it work for you? Or why did you not enable it (yet)? Thanks, Gr. Stefan -- | BIT BV https://www.bit.nl/ Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / info(a)bit.nl

3 2

Mimic 13.2.8 deep scrub error: "size 333447168 > 134217728 is too large"
by Robert Sander 03 Jan '20

03 Jan '20

Hi, when Ceph is running a deep scrub it sometimes complains about an object being too large: Dez 17 07:11:44 al23 ceph-osd[3339]: 2019-12-17 07:11:44.066 7f8d38064700 0 log_channel(cluster) log [DBG] : 8.2d scrub starts Dez 17 07:11:44 al23 ceph-osd[3339]: 2019-12-17 07:11:44.090 7f8d38064700 -1 log_channel(cluster) log [ERR] : 8.2d soid 8:b431ae8d:::gparted-live-0.31.0-1-amd64.iso:head : size 333447168 > 134217728 is too large Dez 17 07:11:44 al23 ceph-osd[3339]: 2019-12-17 07:11:44.102 7f8d38064700 -1 log_channel(cluster) log [ERR] : 8.2d scrub 0 missing, 1 inconsistent objects All OSDs are BlueStore. What is happening here? Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 93818 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

2 2

Access ceph cluster health from REST API
by Vikram Giriraj 03 Jan '20

03 Jan '20

Hi, I am trying to get ceph cluster health and cluster status via REST api. My cluster is running the latest nautilus release(v14). I found that the "ceph-rest-api" is deprecated ( https://docs.ceph.com/docs/mimic/releases/luminous/ ) and now there is ceph RESTful module. But cluster health information is not available from the ceph Restful module. Could someone help me with this. Regards, -- Vikram Giriraj Software Engineer,

1 0

rgw multisite debugging
by Frank R 02 Jan '20

02 Jan '20

Hi all, This is a self inflicted issue but I am wondering if there is a way to recover. Setup: - RGW Multisite with 1 realm, 1 zonegroup and 2 zones. - Metadata and data replication enable (data bi-directional). - Only the master side is currently used by clients. - Ceph 12.2.12 My mistake was to delete all the logs in the rgw.log pool on the master end and some on the secondary side. Data replication is surprisingly working fine but metadata replication is not. When I run "metadata sync init" from the secondary side I see: ----- # radosgw-admin metadata sync init ERROR: sync.init_sync_status() returned ret=-5 2020-01-02 16:10:30.215774 7f26e9544dc0 0 meta sync: ERROR: failed to fetch mdlog info 2020-01-02 16:10:30.215797 7f26e9544dc0 -1 meta sync: ERROR: fail to fetch master log info (r=-5) ----- With rgw-debug=20 and debug-ms=1 I see: ----- received header:HTTP/1.1 500 Internal Server Error ----- On the master side I can run the following commands successfully: # radosgw-admin mdlog status # radosgw-admin mdlog list From the debug session on the master side I see: ----- cache get: name=default.rgw.log++meta.history : hit (requested=0x1, cached=0x17) failed to decode the mdlog history: buffer::end_of_buffer failed to read mdlog history: (5) Input/output error ----- I can view the meta.history log though: # radosgw-admin log show --object=meta.history { "bucket_id": "", "bucket_owner": "", "bucket": "", "log_entries": [ { "bucket": "", "time": "0.000000", "time_local": "0.000000", "remote_addr": "", "user": "", "operation": "", "uri": "", "http_status": "", "error_code": "", "bytes_sent": 7738151095376705906, "bytes_received": 4856413621961454149, "object_size": 7595168404649370735, "total_time": 0, "user_agent": "", "referrer": "" } ], "log_sum": { "bytes_sent": 7738151095376705906, "bytes_received": 4856413621961454149, "total_time": 0, "total_entries": 1 } } ----- Any ideas on how to further troubleshoot the issue? thx Frank

1 0