November 2020 - ceph-users

Ceph RBD - High IOWait during the Writes
by athreyavc＠gmail.com 17 Nov '20

17 Nov '20

Hi, We have recently deployed a Ceph cluster with 12 OSD nodes(16 Core + 200GB RAM + 30 disks each of 14TB) Running CentOS 8 3 Monitoring Nodes (8 Core + 16GB RAM) Running CentOS 8 We are using Ceph Octopus and we are using RBD block devices. We have three Ceph client nodes(16core + 30GB RAM, Running CentOS 8) across which RBDs are mapped and mounted, 25 RBDs each on each client node. Each RBD size is 10TB. Each RBD is formatted as EXT4 file system. From network side, we have 10Gbps Active/Passive Bond on all the Ceph cluster nodes, including the clients. Jumbo frames enabled and MTU is 9000 This is a new cluster and cluster health reports Ok. But we see high IO wait during the writes. From one of the clients, 15:14:30 CPU %user %nice %system %iowait %steal %idle 15:14:31 all 0.06 0.00 1.00 45.03 0.00 53.91 15:14:32 all 0.06 0.00 0.94 41.28 0.00 57.72 15:14:33 all 0.06 0.00 1.25 45.78 0.00 52.91 15:14:34 all 0.00 0.00 1.06 40.07 0.00 58.86 15:14:35 all 0.19 0.00 1.38 41.04 0.00 57.39 Average: all 0.08 0.00 1.13 42.64 0.00 56.16 and the system load shows very high top - 15:19:15 up 34 days, 41 min, 2 users, load average: 13.49, 13.62, 13.83 From 'atop' one of the CPUs shows this CPU | sys 7% | user 1% | irq 2% | idle 1394% | wait 195% | steal 0% | guest 0% | ipc initial | cycl initial | curf 806MHz | curscal ?% On the OSD nodes, don't see much %utilization of the disks. RBD caching values are default. Are we overlooking some configuration item ? Thanks and Regards, At

4 5

Reclassify crush map
by Seena Fallah 17 Nov '20

17 Nov '20

Hi all, I want to reclassify my crushmap. I have two roots, one hiops and one default. In hiops root I have one datacenter and in that I have three rack and in each rack I have 3 osds. When I run the command below it says "item -55 in bucket -54 is not also a reclassified bucket". I see the new crushmap before reclassify-bucket command and item -55 was for my datacenter! What should I do? If I reclassify my datacenter I will lose my datacenter! crushtool -i crush-map --reclassify \ --reclassify-root default hdd \ --set-subtree-class default hdd \ --reclassify-root hiops ssd \ --reclassify-bucket hiops ssd default \ -o new-crush-map Thanks

1 1

Bucket notification is working strange
by Krasaev 17 Nov '20

17 Nov '20

Hi everyone, I asked the same question in stackoverflow, but will repeat here. I configured bucket notification using a bucket owner creds and when the owner does actions I can see new events in a configured endpoint(kafka actually). However, when I try to do actions in the bucket, but with another user creds I do not see events in the configured notification topic. Is it expected behavior and each user has to configure own topic(is it possible if a user is not system at all)? Or I have missed something? Thank you. https://stackoverflow.com/questions/64384060/enable-bucket-notifications-fo…

2 1

CephFS: Recovering from broken Mount
by Julian Fölsch 17 Nov '20

17 Nov '20

Hello, We are running a Octopus cluster however we still have some older Ubuntu 16.04 clients connecting using libcephfs2 version 14.2.13-1xenial. From time to time it happened that the network was having issues so the clients lost the connection to the cluster. But the system still thinks the mount is running and it has to be restarted manually. In some cases we even had to restart the complete machine because it would refuse to unmount. I also attached a dmesg log so you can see the systems behaivior. How do you deal with such issues? Kind regards, Julian Fölsch -- Julian Fölsch Arbeitsgemeinschaft Dresdner Studentennetz (AG DSN) Telefon: +49 351 271816 69 Mobil: +49 152 22915871 Fax: +49 351 46469685 Email: julian.foelsch(a)agdsn.de Studierendenrat der TU Dresden Helmholtzstr. 10 01069 Dresden

1 0

How to configure restful cert/key under nautilus
by Gary Molenkamp 16 Nov '20

16 Nov '20

I am attempting to configure the restful api for integration with zabbix agent2, but I am unable to configure the ssl cert and key based on the documentation here: https://docs.ceph.com/en/nautilus/mgr/restful/ #> ceph config-key set mgr/restful/controller02/crt -i ./mycert.crt WARNING: it looks like you might be trying to set a ceph-mgr module configuration key. Since Ceph 13.0.0 (Mimic), mgr module configuration is done with `config set`, and new values set using `config-key set` will be ignored. I found this issue tracker https://tracker.ceph.com/issues/40013 that also includes the alternate methods that I tried. While the tracker has be closed as resolved, its not clear to me what steps are needed to properly configure ssl on the restful api. Any assistance/clarification would be appreciated. Thanks. NOTE: ceph version 14.2.13

1 0

Re: Octopus OSDs dropping out of cluster: _check_auth_rotating possible clock skew, rotating keys expired way too early
by Lazuardi Nasution 16 Nov '20

16 Nov '20

Hi, I have the same situation with some OSD on Octopus 15.2.5 (Ubuntu 20,04). But, I have no problem with MGR. Any clue about this? Best regards, Date: Tue, 9 Jun 2020 23:47:24 +0200 > From: Wido den Hollander <wido(a)42on.com> > Subject: [ceph-users] Octopus OSDs dropping out of cluster: > _check_auth_rotating possible clock skew, rotating keys expired way > too early > To: "ceph-users(a)ceph.io" <ceph-users(a)ceph.io> > Message-ID: <be7aadc4-2142-ea31-caa8-28ca6db03d15(a)42on.com> > Content-Type: text/plain; charset=utf-8 > > Hi, > > On a recently deployed Octopus (15.2.2) cluster (240 OSDs) we are seeing > OSDs randomly drop out of the cluster. > > Usually it's 2 to 4 OSDs spread out over different nodes. Each node has > 16 OSDs and not all the failing OSDs are on the same node. > > The OSDs are marked as down and all they keep print in their logs: > > monclient: _check_auth_rotating possible clock skew, rotating keys > expired way too early (before 2020-06-04T07:57:17.706529-0400) > > Looking at their status through the admin socket: > > { > "cluster_fsid": "68653193-9b84-478d-bc39-1a811dd50836", > "osd_fsid": "87231b5d-ae5f-4901-93c5-18034381e5ec", > "whoami": 206, > "state": "active", > "oldest_map": 73697, > "newest_map": 75795, > "num_pgs": 19 > } > > The message brought me to my own ticket I created 2 years ago: > https://tracker.ceph.com/issues/23460 > > The first thing I've checked is NTP/time. Double, triple check this. All > the times are in sync on the cluster. Nothing wrong there. > > Again, it's not all the OSDs on a node failing. Just 1 or 2 dropping out. > > Restarting them brings them back right away and then within 24h some > other OSDs will drop out. > > Has anybody seen this behavior with Octopus as well? > > Wido >

1 0

OSD memory leak?
by Frank Schilder 16 Nov '20

16 Nov '20

Hi all, on a mimic 13.2.8 cluster I observe a gradual increase of memory usage by OSD daemons, in particular, under heavy load. For our spinners I use osd_memory_target=2G. The daemons overrun the 2G in virt size rather quickly and grow to something like 4G virtual. The real memory consumption stays more or less around the 2G of the target. There are some overshoots, but these go down again during periods with less load. What I observe now is that the actual memory consumption slowly grows and OSDs start using more than 2G virtual memory. I see this as slowly growing swap usage despite having more RAM available (swappiness=10). This indicates allocated but unused memory or memory not accessed for a long time, usually a leak. Here some heap stats: Before restart: osd.101 tcmalloc heap stats:------------------------------------------------ MALLOC: 3438940768 ( 3279.6 MiB) Bytes in use by application MALLOC: + 5611520 ( 5.4 MiB) Bytes in page heap freelist MALLOC: + 257307352 ( 245.4 MiB) Bytes in central cache freelist MALLOC: + 357376 ( 0.3 MiB) Bytes in transfer cache freelist MALLOC: + 6727368 ( 6.4 MiB) Bytes in thread cache freelists MALLOC: + 25559040 ( 24.4 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 3734503424 ( 3561.5 MiB) Actual memory used (physical + swap) MALLOC: + 575946752 ( 549.3 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 4310450176 ( 4110.8 MiB) Virtual address space used MALLOC: MALLOC: 382884 Spans in use MALLOC: 35 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------ # ceph daemon osd.101 dump_mempools { "mempool": { "by_pool": { "bloom_filter": { "items": 0, "bytes": 0 }, "bluestore_alloc": { "items": 4691828, "bytes": 37534624 }, "bluestore_cache_data": { "items": 0, "bytes": 0 }, "bluestore_cache_onode": { "items": 51, "bytes": 28968 }, "bluestore_cache_other": { "items": 5761276, "bytes": 46292425 }, "bluestore_fsck": { "items": 0, "bytes": 0 }, "bluestore_txc": { "items": 67, "bytes": 46096 }, "bluestore_writing_deferred": { "items": 208, "bytes": 26037057 }, "bluestore_writing": { "items": 52, "bytes": 6789398 }, "bluefs": { "items": 9478, "bytes": 183720 }, "buffer_anon": { "items": 291450, "bytes": 28093473 }, "buffer_meta": { "items": 546, "bytes": 34944 }, "osd": { "items": 98, "bytes": 1139152 }, "osd_mapbl": { "items": 78, "bytes": 8204276 }, "osd_pglog": { "items": 341944, "bytes": 120607952 }, "osdmap": { "items": 10687217, "bytes": 186830528 }, "osdmap_mapping": { "items": 0, "bytes": 0 }, "pgmap": { "items": 0, "bytes": 0 }, "mds_co": { "items": 0, "bytes": 0 }, "unittest_1": { "items": 0, "bytes": 0 }, "unittest_2": { "items": 0, "bytes": 0 } }, "total": { "items": 21784293, "bytes": 461822613 } } } Right after restart + health_ok: osd.101 tcmalloc heap stats:------------------------------------------------ MALLOC: 1173996280 ( 1119.6 MiB) Bytes in use by application MALLOC: + 3727360 ( 3.6 MiB) Bytes in page heap freelist MALLOC: + 25493688 ( 24.3 MiB) Bytes in central cache freelist MALLOC: + 17101824 ( 16.3 MiB) Bytes in transfer cache freelist MALLOC: + 20301904 ( 19.4 MiB) Bytes in thread cache freelists MALLOC: + 5242880 ( 5.0 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 1245863936 ( 1188.1 MiB) Actual memory used (physical + swap) MALLOC: + 20488192 ( 19.5 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 1266352128 ( 1207.7 MiB) Virtual address space used MALLOC: MALLOC: 54160 Spans in use MALLOC: 33 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------ Am I looking at a memory leak here or are these heap stats expected? I don't mind the swap usage, it doesn't have impact. I'm just wondering if I need to restart OSDs regularly. The "leakage" above occurred within only 2 months. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

4 25

Re: set rbd metadata 'conf_rbd_qos_bps_limit', make 'mkfs.xfs /dev/nbdX ' blocked
by Jason Dillaman 16 Nov '20

16 Nov '20

(CCing ceph-users since that is the appropriate forum). On Mon, Nov 16, 2020 at 4:44 AM liugangbiao(a)cmss.chinamobile.com < liugangbiao(a)cmss.chinamobile.com> wrote: > Hi dillaman， > > After I set rbd metadata 'conf_rbd_qos_bps_limit' as blow (600MB/s), the > image size is 1TB > > Then I use 'mkfs.xfs' to format this image, It's stuck for a long time > and never return. > > After unset 'conf_rbd_qos_bps_limit', 'mkfs.xfs' can return > successfully, > The *Ceph version is 14.2.10*, Any suggestion to reslove this porblem? > > Have you tried using the "-K" option to "mkfs.xfs" to disable discard at mkfs time? Perhaps it's issuing a discard region bigger than 600MiB? > Beset Regards， > LiuGangbiao > -- Jason

1 0

Mimic updated to Nautilus - pg's 'update_creating_pgs' in log, but they exist and cluster is healthy.
by m.sliwinski＠lh.pl 16 Nov '20

16 Nov '20

Hi With the help of Dan van der Steri i managed to confirm my suspicions that the problem with osdmap not trimming correctly was caused by PGs that somehow wasn't marked as created by MONs. For example, from log: 2020-11-16 12:57:00.514 7f131496f700 10 mon.monb01(a)0(probing).osd e72792 update_creating_pgs will instruct osd.265 to create 28.3ff@67698 2020-11-16 12:57:25.982 7f1315971700 10 mon.monb01(a)0(leader).osd e72792 update_creating_pgs will instruct osd.265 to create 28.3ff@72792 But: root@monb01:/var/log/ceph# ceph pg dump |grep 28.3ff dumped all 28.3ff 3841 0 0 0 0 15970230272 0 0 3028 3028 active+clean 2020-11-16 05:38:27.338826 72792'87928 72792:335764 [265,277,282] 265 [265,277,282] 265 72792'85741 2020-11-16 05:38:27.338783 72588'79082 2020-11-10 18:42:43.182436 0 root@monb01:/var/log/ceph# root@monb01:/var/log/ceph# ceph health detail HEALTH_OK root@monb01:/var/log/ceph# Does anyone know how to fix this? I found that it can be cleared by deleting and recreating that pool, but there's a lot of data there so is there way to fix this some other way? -- Best regards Marcin

1 0

build nautilus 14.2.13 packages and container
by Engelmann Florian 16 Nov '20

16 Nov '20

Hi, I was not able to find any complete guide on how to build ceph (14.2.x) from source, create packages and build containers based on those packages. Ubuntu or centos, does not matter. I tried so far: ############### docker pull centos:7 docker run -ti centos:7 /bin/bash yum install -y git rpm-build rpmdevtools wget epel-release yum install -y python-virtualenv python-pip jq cmake3 make gcc-c++ rpm-build which sudo createrepo git clone https://github.com/ceph/ceph cd ceph git checkout v14.2.13 ./make-srpm.sh ./install-deps.sh ################# but install-deps.sh fails with: Error: No Package found for python-scipy The following error message appeared before: ---------------- http://vault.centos.org/centos/7/sclo/Source/rh/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found Trying other mirror. To address this issue please refer to the below wiki article https://wiki.centos.org/yum-errors If above article doesn't help to resolve this issue please use https://bugs.centos.org/. http://vault.centos.org/centos/7/sclo/Source/sclo/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found -------------------- centos:8 fails as well. dependency error: Error: Problem: package python36-rpm-macros-3.6.8-2.module_el8.1.0+245+c39af44f.noarch conflicts with python-modular-rpm-macros > 3.6 provided by python38-rpm-macros-3.8.0-6.module_el8.2.0+317+61fa6e7d.noarch - conflicting requests Any helpful links? All the best, Florian EveryWare AG Florian Engelmann Cloud Platform Architect Zurlindenstrasse 52a CH-8003 Zürich T +41 44 466 60 00 F +41 44 466 60 10 florian.engelmann(a)everyware.ch www.everyware.ch

1 1

Re: which of cpu frequency and number of threads servers osd better?
by Frank Schilder 16 Nov '20

16 Nov '20

We are starting to use 18TB spindles, have loads of cold data and only a thin layer of hot data. One 4/8TB NVMe drive as a cache in front of 6x18TB will provide close to or even matching SSD performance for the hot data at a reasonable extra cost per TB storage. My plan is to wait for 1-2 more years for prices for PCI-NVMe to drop and then start using this method. The second advantage is, that one can continue to deploy collocated HDD OSDs as WAL/DB will certainly land and stay in cache. The cache can be added to existing OSDs without redeployment. In addition, dm-cache uses a hit count method for computing promotion to cache, which works very different from promotion to ceph cache pools. Dm-cache can afford that due to its local nature. In particular, it doesn't promote on just 1 access, which means that a weekly or monthly backup will not flush the entire cache every time. All SSD pools for this data (ceph-fs in EC pool on HDD) will be unaffordable to us for a long time. Not to mention that these large SSDs are almost certainly QLC, which have much less sustained throughput compared with the 18TB He-drives (they have higher IOP/s though, which is not so relevant for our FS use workloads). The cache method will provide at least the additional IOP/s that WAB/DB devices would, but due to its size also data caching. We need to go NVMe, because the servers we plan to use (R740xd2) provide the largest capacity configuration with 24xHDD+4xPCI NVMe. You can either choose 2 extra drives or 4 PCI NVMe, but not both. So, NVMe cannot be exchanged by fast SSDs as they would eat drive slots. There were a few threads over the past 1-2 years where people dropped in some of these observations and I just took note of it. It is used in production already and from what I got people are happy with it. Much easier than WAL/DB partitions plus all the sizing problems for L0/L1/... are sorted trivially. With the size of NVMe growing rapidly beyond what WAL/DB devices can utilize and since LVM is the new OSD device, using LVM dm-cache seems to be the way forward for me. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Anthony D'Atri <anthony.datri(a)gmail.com> Sent: 16 November 2020 03:00:38 To: Frank Schilder Subject: Re: [ceph-users] Re: which of cpu frequency and number of threads servers osd better? Thanks. I’m curious how the economics for that compare with just using all SSDs: * HDDs are cheaper * But colo SSDs are operationally simpler * And depending on configuration you can provision a cheaper HBA > On Nov 14, 2020, at 2:04 AM, Frank Schilder <frans(a)dtu.dk> wrote: > > My plan is to use at least 500GB NVMe per HDD OSD. I have not started that yet, but there are threads of other people sharing their experience. If you go beyond 300GB per OSD, apparently the WAL/DB options cannot really use the extra capacity. With dm-cache or the like you would additionally start holding hot data in cache. > > Ideally, I can split a 4TB or even a 8TB NVMe over 6 OSDs. > > Best regards, > ================= > Frank Schilder > AIT Risø Campus > Bygning 109, rum S14 > > ________________________________________ > From: Anthony D'Atri <anthony.datri(a)gmail.com> > Sent: 14 November 2020 10:57:57 > To: Frank Schilder > Subject: Re: [ceph-users] Re: which of cpu frequency and number of threads servers osd better? > > Guten Tag. > >> My plan for the future is to use dm-cache for LVM OSDs instead of WAL/DB device. > > Do you have any insights into the benefits of that approach instead of WAL/DB, and of dm-cache vs bcache vs dm-writecache vs … ? And any for sizing the cache device and handling failures? Presumably the DB will be active enough that it will persist in the cache, so sizing should be at a minimum that to hold 2 copies of the DB to accomodate compaction? > > I have an existing RGW cluster on HDDs that utilizes a cache tier; the high water mark is set fairly low so that it doesn’t fill up, something that apparently happened last Christmas. I’ve been wanting to get a feel for OSD cache as an alternative to deprecated and fussy cache tiering, as well as something like a Varnish cache on RGW load balancers to short-circult small requests. > > — Anthony > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Beginner's installation questions about network
by E Taka 15 Nov '20

15 Nov '20

Hello, I want to install Ceph Octopus on Ubuntu 20.04. The nodes for have 2 network interfaces: 192.168.1.0/24 for the cluster network, and a 10.10.0.0/16 is the public network. When I bootstrap with cephadm, which Network do I use? That means, do i use cephadm bootstrap --mon-ip 192.168.1.1 or do I have to use the other network? When adding the other host with ceph orch, which network I have to use: ceph orch host add ceph03i 192.168.20.2 … (or the 10.10 network) Thanks, Erich

5 5

Problem in MGR deamon
by Hamidreza Hosseini 14 Nov '20

14 Nov '20

1. Hi, 2. I've installed ceph mon and mgr as cluster on 3 nodes, 3. I have this errors on status and log of standby mgr: ubuntu 20.04 ceph version: 15.2.5 1. ● ceph-mgr(a)dev13.service - Ceph cluster manager daemon 2. Loaded: loaded (/lib/systemd/system/ceph-mgr@.service; enabled; vendor preset: enabled) 3. Active: active (running) since Sat 2020-11-14 14:00:57 +0330; 40min ago 4. Main PID: 13485 (ceph-mgr) 5. Tasks: 20 (limit: 9451) 6. Memory: 233.2M 7. CGroup: /system.slice/system-ceph\x2dmgr.slice/ceph-mgr(a)dev13.service 8. └─13485 /usr/bin/ceph-mgr -f --cluster ceph --id dev13 --setuser ceph --setgroup ceph 9. 10. Nov 14 14:01:03 dev13 ceph-mgr[13485]: File "/lib/python3/dist-packages/numpy/core/__init__.py", line 17, in <module> 11. Nov 14 14:01:03 dev13 ceph-mgr[13485]: from . import multiarray 12. Nov 14 14:01:03 dev13 ceph-mgr[13485]: File "/lib/python3/dist-packages/numpy/core/multiarray.py", line 14, in <module> 13. Nov 14 14:01:03 dev13 ceph-mgr[13485]: from . import overrides 14. Nov 14 14:01:03 dev13 ceph-mgr[13485]: File "/lib/python3/dist-packages/numpy/core/overrides.py", line 16, in <module> 15. Nov 14 14:01:03 dev13 ceph-mgr[13485]: add_docstring( 16. Nov 14 14:01:03 dev13 ceph-mgr[13485]: RuntimeError: implement_array_function method already has a docstring 17. Nov 14 14:01:03 dev13 ceph-mgr[13485]: 2020-11-14T14:01:03.034+0330 7eff2173b040 -1 mgr[py] Class not found in module 'rook' 18. Nov 14 14:01:03 dev13 ceph-mgr[13485]: 2020-11-14T14:01:03.034+0330 7eff2173b040 -1 mgr[py] Error loading module 'rook': (2) No such file or directory 19. Nov 14 14:01:04 dev13 ceph-mgr[13485]: 2020-11-14T14:01:04.478+0330 7eff2173b040 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: rook 20. 21. test@dev12:/etc/ceph$ sudo journalctl -xe 22. Nov 14 13:27:06 dev12 ceph-mgr[15041]: File "/usr/share/ceph/mgr/rook/module.py", line 17, in <module> 23. Nov 14 13:27:06 dev12 ceph-mgr[15041]: from kubernetes import client, config 24. Nov 14 13:27:06 dev12 ceph-mgr[15041]: File "/lib/python3/dist-packages/kubernetes/__init__.py", line 22, in <module> 25. Nov 14 13:27:06 dev12 ceph-mgr[15041]: import kubernetes.stream 26. Nov 14 13:27:06 dev12 ceph-mgr[15041]: File "/lib/python3/dist-packages/kubernetes/stream/__init__.py", line 15, in <module> 27. Nov 14 13:27:06 dev12 ceph-mgr[15041]: from .stream import stream 28. Nov 14 13:27:06 dev12 ceph-mgr[15041]: File "/lib/python3/dist-packages/kubernetes/stream/stream.py", line 13, in <module> 29. Nov 14 13:27:06 dev12 ceph-mgr[15041]: from . import ws_client 30. Nov 14 13:27:06 dev12 ceph-mgr[15041]: File "/lib/python3/dist-packages/kubernetes/stream/ws_client.py", line 19, in <module> 31. Nov 14 13:27:06 dev12 ceph-mgr[15041]: from websocket import WebSocket, ABNF, enableTrace 32. Nov 14 13:27:06 dev12 ceph-mgr[15041]: File "/lib/python3/dist-packages/websocket/__init__.py", line 22, in <module> 33. Nov 14 13:27:06 dev12 ceph-mgr[15041]: from ._abnf import * 34. Nov 14 13:27:06 dev12 ceph-mgr[15041]: File "/lib/python3/dist-packages/websocket/_abnf.py", line 34, in <module> 35. Nov 14 13:27:06 dev12 ceph-mgr[15041]: import numpy 36. Nov 14 13:27:06 dev12 ceph-mgr[15041]: File "/lib/python3/dist-packages/numpy/__init__.py", line 142, in <module> 37. Nov 14 13:27:06 dev12 ceph-mgr[15041]: from . import core 38. Nov 14 13:27:06 dev12 ceph-mgr[15041]: File "/lib/python3/dist-packages/numpy/core/__init__.py", line 17, in <module> 39. Nov 14 13:27:06 dev12 ceph-mgr[15041]: from . import multiarray 40. Nov 14 13:27:06 dev12 ceph-mgr[15041]: File "/lib/python3/dist-packages/numpy/core/multiarray.py", line 14, in <module> 41. Nov 14 13:27:06 dev12 ceph-mgr[15041]: from . import overrides 42. Nov 14 13:27:06 dev12 ceph-mgr[15041]: File "/lib/python3/dist-packages/numpy/core/overrides.py", line 16, in <module> 43. Nov 14 13:27:06 dev12 ceph-mgr[15041]: add_docstring( 44. Nov 14 13:27:06 dev12 ceph-mgr[15041]: RuntimeError: implement_array_function method already has a docstring 45. Nov 14 13:27:06 dev12 ceph-mgr[15041]: 2020-11-14T13:27:06.159+0330 7fdeafd88040 -1 mgr[py] Class not found in module 'rook' 46. Nov 14 13:27:06 dev12 ceph-mgr[15041]: 2020-11-14T13:27:06.159+0330 7fdeafd88040 -1 mgr[py] Error loading module 'rook': (2) No such file or directory 47. Nov 14 13:27:07 dev12 ceph-mgr[15041]: 2020-11-14T13:27:07.547+0330 7fdeafd88040 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: rook 48. 2020-11-14T13:27:06.431+0330 7fdeafd88040 1 mgr[py] Loading python module 'telegraf' 49. 2020-11-14T13:27:06.667+0330 7fdeafd88040 1 mgr[py] Loading python module 'telemetry' 50. 2020-11-14T13:27:06.987+0330 7fdeafd88040 1 mgr[py] Loading python module 'test_orchestrator' 51. 2020-11-14T13:27:07.187+0330 7fdeafd88040 1 mgr[py] Loading python module 'volumes' 52. 2020-11-14T13:27:07.459+0330 7fdeafd88040 1 mgr[py] Loading python module 'zabbix' 53. 2020-11-14T13:27:07.547+0330 7fdeafd88040 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: rook 54. 2020-11-14T13:27:07.551+0330 7fdeaad22700 0 ms_deliver_dispatch: unhandled message 0x55dcf177e2c0 mon_map magic: 0 v1 from mon.2 v2:192.168.200.23:3300/0 55. 2020-11-14T13:27:07.555+0330 7fdeaad22700 0 ms_deliver_dispatch: unhandled message 0x55dcf177fa20 mon_map magic: 0 v1 from mon.2 v2:192.168.200.23:3300/0 56. 2020-11-14T13:42:07.558+0330 7fdeaad22700 0 client.0 ms_handle_reset on v2:192.168.200.21:6800/6434 57. 2020-11-14T13:57:07.569+0330 7fdeaad22700 0 client.0 ms_handle_reset on v2:192.168.200.21:6800/6434 58. 59.

1 0

Re: which of cpu frequency and number of threads servers osd better?
by Frank Schilder 14 Nov '20

14 Nov '20

My plan is to use at least 500GB NVMe per HDD OSD. I have not started that yet, but there are threads of other people sharing their experience. If you go beyond 300GB per OSD, apparently the WAL/DB options cannot really use the extra capacity. With dm-cache or the like you would additionally start holding hot data in cache. Ideally, I can split a 4TB or even a 8TB NVMe over 6 OSDs. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Anthony D'Atri <anthony.datri(a)gmail.com> Sent: 14 November 2020 10:57:57 To: Frank Schilder Subject: Re: [ceph-users] Re: which of cpu frequency and number of threads servers osd better? Guten Tag. > My plan for the future is to use dm-cache for LVM OSDs instead of WAL/DB device. Do you have any insights into the benefits of that approach instead of WAL/DB, and of dm-cache vs bcache vs dm-writecache vs … ? And any for sizing the cache device and handling failures? Presumably the DB will be active enough that it will persist in the cache, so sizing should be at a minimum that to hold 2 copies of the DB to accomodate compaction? I have an existing RGW cluster on HDDs that utilizes a cache tier; the high water mark is set fairly low so that it doesn’t fill up, something that apparently happened last Christmas. I’ve been wanting to get a feel for OSD cache as an alternative to deprecated and fussy cache tiering, as well as something like a Varnish cache on RGW load balancers to short-circult small requests. — Anthony

1 0

which of cpu frequency and number of threads servers osd better?
by Tony Liu 14 Nov '20

14 Nov '20

Hi, For example, 16 threads with 3.2GHz and 32 threads with 3.0GHz, which makes 11 OSDs (10x12TB HDD and 1x960GB SSD) with better performance? Thanks! Tony

4 10

monitor sst files continue growing
by Zhenshi Zhou 14 Nov '20

14 Nov '20

Hi all, My cluster is in wrong state. SST files in /var/lib/ceph/mon/xxx/store.db continue growing. It claims mon are using a lot of disk space. I set "mon compact on start = true" and restart one of the monitors. But it started and campacting for a long time, seems it has no end. [image: image.png]

4 14

How to Improve RGW Bucket Stats Performance
by Denis Krienbühl 13 Nov '20

13 Nov '20

Hi! To bill our customers we regularly call radosgw-admin bucket stats --uid <uid>. Since upgrading from Mimic to Octopus (with a short stop at Nautilus), we’ve been seeing much slower response times for this command. It went from less than a minute for our largest customers, to 5 minutes (with some variance depending on load). Assuming this is not a bug, is there any way to get these stats quicker? Ceph seems to do in a single call here, which seems to me like something you could spread out over time (keep a counter somewhere and just return the latest value on request). One thing we did notice, is that we get a lot of these when the stats are synced: 2020-11-13T14:56:17.288+0100 7f15347e0700 0 check_bucket_shards: resharding needed: stats.num_objects=5776982 shard max_objects=3200000 Could that hint at a problem in our configuration? Anything else we could maybe tune to get this time down? Appreciate any hints and I hope everyone is about to have a great weekend. Denis

1 0

OverlayFS with Cephfs to mount a snapshot read/write
by Frédéric Nass 13 Nov '20

13 Nov '20

Hello, I would like to use a cephfs snapshot as a read/write volume without having to clone it first as the cloning operation is - if I'm not mistaken - still inefficient as of now. This is for a data restore use case with Moodle application needing a writable data directory to start. The idea that came to mind was to use overlayFS with cephfs set up as a read-only lower layer and a writable local directory set up as an upper layer. With this set up, any modifications to the read-only .snap/testsnap directory would normally go to the upper directory making the snapshot directory somehow writable to the Moodle application. While this works fine when a local read-only filesystem is set up as the lower layer, it fails when cephfs is set up as the lower layer. Any modifications to the .snap/testsnap tree in the /cephfs-snap directory fails with an "Operation not supported". $ mkdir /cephfs /upperdir /workdir /cephfs-snap $ mount -t ceph 100.74.191.129:/volumes/group1/subvolume1/ /cephfs -o name=admin,secretfile=/etc/ceph/admin.secret $ mount -t overlay overlay -o redirect_dir=on,lowerdir=/cephfs/.snap/testsnap,upperdir=/upperdir,workdir=/workdir /cephfs-snap $ ls /cephfs-snap usr $ touch /cephfs-snap/foo.txt <---- writing outside the lowerdir succeeds $ ls /cephfs-snap foo.txt usr $ ls /usr/etc $ touch /cephfs-snap/usr/etc/foo <---- writing inside the lowerdir fails touch: impossible de faire un touch « /cephfs-snap/usr/etc/foo »: Opération non supportée I tried to mount the whole cephfs tree read-only (-o ro), tried to disable ACLs (-o noacl) as seen here [1] but of no help. Mounting with ceph-fuse didn't help either. There's been a recent discussion about this here [2] between Greg and Robert but with no real solution. Did someone manage to do this? Regards, Frédéric. [1] https://blog.fai-project.org/posts/overlayfs/ [2] https://tracker.ceph.com/issues/44821

5 10

question about rgw delete speed
by Adrian Nicolae 13 Nov '20

13 Nov '20

Hey guys, I'm in charge of a local cloud-storage service. Our primary object storage is a vendor-based one and I want to replace it in the near future with Ceph with the following setup : - 6 OSD servers with 36 SATA 16TB drives each and 3 big NVME per server (1 big NVME for every 12 drives so I can reserve 300GB NVME storage for every SATA drive), 3 MON, 2 RGW with Epyc 7402p and 128GB RAM. So in the end we'll have ~ 3PB of raw data and 216 SATA drives. Currently we have ~ 100 millions of files on the primary storage with the following distribution : - ~10% = very small files ( less than 1MB - thumbnails, text&office files and so on) - ~60%= small files (between 1MB and 10MB) - 20% = medium files ( between 10MB and 1GB) - 10% = big files (over 1GB). My main concern is the speed of delete operations. We have around 500k-600k delete ops every 24 hours so quite a lot. Our current storage is not deleting all the files fast enough (it's always 1 week-10 days behind) , I guess is not only a software issue and probably the delete speed will get better if we add more drives (we now have 108). What do you think about Ceph delete speed ? I read on other threads that it's not very fast . I wonder if this hw setup can handle our current delete load better than our current storage. On RGW servers I want to use Swift , not S3. And another question : can I start deploying in production directly the latest Ceph version (Octopus) or is it safer to start with Nautilus until Octopus will be more stable ? Any input would be greatly appreciated ! Thanks, Adrian.

4 4

Not able to read file from ceph kernel mount
by Amudhan P 13 Nov '20

13 Nov '20

Hi, I am trying to read file from my ceph kernel mount and file read stays in bytes for very long and I am getting below error msg in dmesg. [ 167.591095] ceph: loaded (mds proto 32) [ 167.600010] libceph: mon0 10.0.103.1:6789 session established [ 167.601167] libceph: client144519 fsid f8bc7682-0d11-11eb-a332- 0cc47a5ec98a [ 272.132787] libceph: osd1 10.0.104.1:6891 socket closed (con state CONNECTING) Ceph cluster status is healthy no error It was working fine until before my entire cluster was down. Using Ceph octopus in debian. Regards Amudhan P

1 3

Cephfs Kernel client not working properly without ceph cluster IP
by Amudhan P 13 Nov '20

13 Nov '20

Hi, I have mounted my cephfs (ceph octopus) thru kernel client in Debian. I get following error in "dmesg" when I try to read any file from my mount. "[ 236.429897] libceph: osd1 10.100.4.1:6891 socket closed (con state CONNECTING)" I use public IP (10.100.3.1) and cluster IP (10.100.4.1) in my ceph cluster. I think public IP is enough to mount the share and work on it but in my case, it needs me to assign public IP also to the client to work properly. Does anyone have experience in this? I have earlier also mailed the ceph-user group but I didn't get any response. So sending again not sure my mail went through. regards Amudhan

4 10

Autoscale - enable or not on main pool?
by Brent Kennedy 12 Nov '20

12 Nov '20

I recently setup a new octopus cluster and was testing the autoscale feature. Used ceph-ansible so its enabled by default. Anyhow, I have three other clusters that are on nautilus, so I wanted to see if it made sense to enable it there on the main pool. Here is a print out of the autoscale status: POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO EFFECTIVE RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE default.rgw.buckets.non-ec 0 2.0 55859G 0.0000 1.0 32 on default.rgw.meta 9298 3.0 55859G 0.0000 1.0 32 on default.rgw.buckets.index 18058M 3.0 55859G 0.0009 1.0 32 on default.rgw.control 0 3.0 55859G 0.0000 1.0 32 on default.rgw.buckets.data 9126G 2.0 55859G 0.3268 1.0 4096 1024 off .rgw.root 3155 3.0 55859G 0.0000 1.0 32 on rbd 155.5G 2.0 55859G 0.0056 1.0 32 on default.rgw.log 374.4k 3.0 55859G 0.0000 1.0 64 on For this entry: default.rgw.buckets.data 9126G 2.0 55859G 0.3268 1.0 4096 1024 off I have it disabled because it showed a warn message, but its recommending a 1024 PG setting. When I use the online ceph calculator at ceph.io, its saying the 4096 setting is correct. So why is autoscaler saying 1024? There are 6 osd servers with 10 OSDs each ( all SSD ). 60 TB total. Pool LS output: pool 1 '.rgw.root' replicated size 3 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 8800 lfor 0/0/344 flags hashpspool stripe_width 0 application rgw pool 2 'default.rgw.control' replicated size 3 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 8799 lfor 0/0/346 flags hashpspool stripe_width 0 application rgw pool 3 'default.rgw.meta' replicated size 3 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 8798 lfor 0/0/350 flags hashpspool stripe_width 0 application rgw pool 4 'default.rgw.log' replicated size 3 min_size 1 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode on last_change 8802 lfor 0/0/298 flags hashpspool stripe_width 0 application rgw pool 5 'default.rgw.buckets.index' replicated size 3 min_size 1 crush_rule 0 object_hash rjenkins pg_num 638 pgp_num 608 pg_num_target 32 pgp_num_target 32 autoscale_mode on last_change 10320 lfor 0/10320/10318 owner 18446744073709551615 flags hashpspool stripe_width 0 application rgw pool 7 'default.rgw.buckets.data' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 4096 pgp_num 4096 last_change 9467 lfor 0/0/552 owner 18446744073709551615 flags hashpspool stripe_width 0 application rgw pool 8 'default.rgw.buckets.non-ec' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 8797 lfor 0/0/348 owner 18446744073709551615 flags hashpspool stripe_width 0 application rgw pool 9 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 8801 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd Regards, -Brent Existing Clusters: Test: Ocotpus 15.2.5 ( all virtual on nvme ) US Production(HDD): Nautilus 14.2.11 with 11 osd servers, 3 mons, 4 gateways, 2 iscsi gateways UK Production(HDD): Nautilus 14.2.11 with 18 osd servers, 3 mons, 4 gateways, 2 iscsi gateways US Production(SSD): Nautilus 14.2.11 with 6 osd servers, 3 mons, 4 gateways, 2 iscsi gateways UK Production(SSD): Octopus 15.2.5 with 5 osd servers, 3 mons, 4 gateways

1 0

Re: NoSuchKey on key that is visible in s3 list/radosgw bk
by Janek Bevendorff 12 Nov '20

12 Nov '20

We are having the exact same problem (also Octopus). The object is listed by s3cmd, but trying to download it results in a 404 error. radosgw-admin object stat shows that the object still exists. Any further ideas how I can restore access to this object? (Sorry if this is a duplicate, but it seems like the mailing list hasn't accepted my original mail). > Mariusz Gronczewski wrote: > > >> Dnia 2020-07-27, o godz. 21:31:33 >> "Robin H. Johnson" <robbat2(a)gentoo.org >> <mailto:robbat2@gentoo.org>> napisał(a): >> >> >>> On Mon, Jul 27, 2020 at 08:02:23PM +0200, Mariusz Gronczewski wrote: >>> >>>> Hi, >>>> I've got a problem on Octopus (15.2.3, debian packages) install, >>>> bucket S3 index shows a file: >>>> s3cmd ls s3://upvid/255/38355 --recursive >>>> 2020-07-27 17:48 50584342 >>>> s3://upvid/255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4 >>>> radosgw-admin bi list also shows it >>>> { >>>> "type": "plain", >>>> "idx": >>>> "255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4", >>>> "entry": { "name": >>>> "255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4", >>>> "instance": "", "ver": { >>>> "pool": 11, >>>> "epoch": 853842 >>>> }, >>>> "locator": "", >>>> "exists": "true", >>>> "meta": { >>>> "category": 1, >>>> "size": 50584342, >>>> "mtime": "2020-07-27T17:48:27.203008Z", >>>> "etag": "2b31cc8ce8b1fb92a5f65034f2d12581-7", >>>> "storage_class": "", >>>> "owner": "filmweb-app", >>>> "owner_display_name": "filmweb app user", >>>> "content_type": "", >>>> "accounted_size": 50584342, >>>> "user_data": "", >>>> "appendable": "false" >>>> }, >>>> "tag": "_3ubjaztglHXfZr05wZCFCPzebQf-ZFP", >>>> "flags": 0, >>>> "pending_map": [], >>>> "versioned_epoch": 0 >>>> } >>>> }, >>>> but trying to download it via curl (I've set permissions to public0 >>>> >>> only gets me >>> Does the RADOS object for this still exist? >>> >>> try: >>> radosgw-admin object stat --bucket ... --object >>> '255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4' >>> >>> If that doesn't return, then the backing object is gone, and you have >>> a stale index entry that can be cleaned up in most cases with check >>> bucket. >>> For cases where that doesn't fix it, my recommended way to fix it is >>> write a new 0-byte object to the same name, then delete it. >>> >> >> >> >> it does exist: >> >> { >> "name": >> "255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4", >> "size": 50584342, "policy": { >> "acl": {...}, >> "owner": {...} >> }, >> "etag": "2b31cc8ce8b1fb92a5f65034f2d12581-7", >> "tag": "_3ubjaztglHXfZr05wZCFCPzebQf-ZFP", >> "manifest": { >> "objs": [], >> "obj_size": 50584342, >> "explicit_objs": "false", >> "head_size": 0, >> "max_head_size": 0, >> "prefix": >> "255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4.2~NTy88SkDkXR9ifSrrRcw5WPDxqN3PO2", >> "rules": [ { >> "key": 0, >> "val": { >> "start_part_num": 1, >> "start_ofs": 0, >> "part_size": 8388608, >> "stripe_max_size": 4194304, >> "override_prefix": "" >> } >> }, >> { >> "key": 50331648, >> "val": { >> "start_part_num": 7, >> "start_ofs": 50331648, >> "part_size": 252694, >> "stripe_max_size": 4194304, >> "override_prefix": "" >> } >> } >> ], >> "tail_instance": "", >> "tail_placement": { >> "bucket": { >> "name": "upvid", >> "marker": >> "88d4f221-0da5-444d-81a8-517771278350.665933.2", "bucket_id": >> "88d4f221-0da5-444d-81a8-517771278350.665933.2", "tenant": "", >> "explicit_placement": { >> "data_pool": "", >> "data_extra_pool": "", >> "index_pool": "" >> } >> }, >> "placement_rule": "default-placement" >> }, >> "begin_iter": { >> "part_ofs": 0, >> "stripe_ofs": 0, >> "ofs": 0, >> "stripe_size": 4194304, >> "cur_part_id": 1, >> "cur_stripe": 0, >> "cur_override_prefix": "", >> "location": { >> "placement_rule": "default-placement", >> "obj": { >> "bucket": { >> "name": "upvid", >> "marker": >> "88d4f221-0da5-444d-81a8-517771278350.665933.2", "bucket_id": >> "88d4f221-0da5-444d-81a8-517771278350.665933.2", "tenant": "", >> "explicit_placement": { >> "data_pool": "", >> "data_extra_pool": "", >> "index_pool": "" >> } >> }, >> "key": { >> "name": >> "255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4.2~NTy88SkDkXR9ifSrrRcw5WPDxqN3PO2.1", >> "instance": "", "ns": "multipart" >> } >> }, >> "raw_obj": { >> "pool": "", >> "oid": "", >> "loc": "" >> }, >> "is_raw": false >> } >> }, >> "end_iter": { >> "part_ofs": 50584342, >> "stripe_ofs": 50584342, >> "ofs": 50584342, >> "stripe_size": 252694, >> "cur_part_id": 8, >> "cur_stripe": 0, >> "cur_override_prefix": "", >> "location": { >> "placement_rule": "default-placement", >> "obj": { >> "bucket": { >> "name": "upvid", >> "marker": >> "88d4f221-0da5-444d-81a8-517771278350.665933.2", "bucket_id": >> "88d4f221-0da5-444d-81a8-517771278350.665933.2", "tenant": "", >> "explicit_placement": { >> "data_pool": "", >> "data_extra_pool": "", >> "index_pool": "" >> } >> }, >> "key": { >> "name": >> "255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4.2~NTy88SkDkXR9ifSrrRcw5WPDxqN3PO2.8", >> "instance": "", "ns": "multipart" >> } >> }, >> "raw_obj": { >> "pool": "", >> "oid": "", >> "loc": "" >> }, >> "is_raw": false >> } >> } >> }, >> "attrs": { >> "user.rgw.pg_ver": "/u0005u", >> "user.rgw.source_zone": "�I}�", >> "user.rgw.tail_tag": >> "88d4f221-0da5-444d-81a8-517771278350.658638.5538809", >> "user.rgw.x-amz-acl": "private", "user.rgw.x-amz-content-sha256": >> "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", >> "user.rgw.x-amz-date": "20200619T194517Z" } >> } >> >> >> >> >> >> -- >> Mariusz Gronczewski, Administrator >> >> Efigence S. A. >> ul. Wołoska 9a, 02-583 Warszawa >> T: [+48] 22 380 13 13 >> NOC: [+48] 22 380 10 20 >> E: admin(a)efigence.com <mailto:admin@efigence.com> >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io <mailto:ceph-users@ceph.io> >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> <mailto:ceph-users-leave@ceph.io>

4 11

Re: Is there a way to make Cephfs kernel client to write data to ceph osd smoothly with buffer io
by Frank Schilder 12 Nov '20

12 Nov '20

Yes, that's right. It would be nice if there was a mount option to have such parameters adjusted on a per-file system basis. I should mention that I observed a significant performance improvement for HDD throughput of the local disk as well when adjusting these parameters for ceph. This is largely due to the "too much memory problem" on big servers. The kernel defaults are suitable for machines with 4-8G of RAM. Any enterprise server will beat that with the consequence of insanely large amounts of dirty buffers, leading to buffer flush panic overloading in particular, network file systems (there is a nice article by SUSE https://www.suse.com/support/kb/doc/?id=000017857) Adjusting these parameters to play nice with ceph might actually improve overall performance as a side effect. I would give it a go. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Sage Meng <lkkey80(a)gmail.com> Sent: 12 November 2020 16:00:08 To: Frank Schilder Cc: ceph-users(a)ceph.io Subject: Re: [ceph-users] Is there a way to make Cephfs kernel client to write data to ceph osd smoothly with buffer io vm.dirty_bytes and vm.dirty_background_bytes are all system-wide control parameters, it will influence all the system jobs by adjusting them. Better to have a Ceph Special way to make the transfer more smoothly. Frank Schilder <frans(a)dtu.dk<mailto:frans@dtu.dk>> 于2020年11月11日周三下午3:28写道： These kernel parameters influence the flushing of data, and also performance: vm.dirty_bytes vm.dirty_background_bytes Smaller vm.dirty_background_bytes will make the transfer more smooth and the ceph cluster will like that. However, it reduces the chances of merge operations in cache and the ceph cluster will not like that. The tuning is heavily workload dependent. Test with realistic workloads and a reasonably large spectrum of values. I got good results by tuning down vm.dirty_background_bytes just to the point when it reduced client performance of copying large files. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Sage Meng <lkkey80(a)gmail.com<mailto:lkkey80@gmail.com>> Sent: 06 November 2020 13:45:53 To: ceph-users(a)ceph.io<mailto:ceph-users@ceph.io> Subject: [ceph-users] Is there a way to make Cephfs kernel client to write data to ceph osd smoothly with buffer io Hi All, Cephfs kernel client is influenced by kernel page cache when we write data to it, outgoing data will be huge when os starts flush page cache. So Is there a way to make Cephfs kernel client to write data to ceph osd smoothly when buffer io is used ? _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave(a)ceph.io<mailto:ceph-users-leave@ceph.io>

1 0

Rados Crashing
by Brent Kennedy 12 Nov '20

12 Nov '20

We are performing file maintenance( deletes essentially ) and when the process gets to a certain point, all four rados gateways crash with the following: Log output: -5> 2020-10-20 06:09:53.996 7f15f1543700 2 req 7 0.000s s3:delete_obj verifying op params -4> 2020-10-20 06:09:53.996 7f15f1543700 2 req 7 0.000s s3:delete_obj pre-executing -3> 2020-10-20 06:09:53.996 7f15f1543700 2 req 7 0.000s s3:delete_obj executing -2> 2020-10-20 06:09:53.997 7f161758f700 10 monclient: get_auth_request con 0x55d2c02ff800 auth_method 0 -1> 2020-10-20 06:09:54.009 7f1609d74700 5 process_single_shard(): failed to acquire lock on obj_delete_at_hint.0000000079 0> 2020-10-20 06:09:54.035 7f15f1543700 -1 *** Caught signal (Segmentation fault) ** in thread 7f15f1543700 thread_name:civetweb-worker ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable) 1: (()+0xf5d0) [0x7f161d3405d0] 2: (()+0x2bec80) [0x55d2bcd1fc80] 3: (std::string::assign(std::string const&)+0x2e) [0x55d2bcd2870e] 4: (rgw_bucket::operator=(rgw_bucket const&)+0x11) [0x55d2bce3e551] 5: (RGWObjManifest::obj_iterator::update_location()+0x184) [0x55d2bced7114] 6: (RGWObjManifest::obj_iterator::operator++()+0x263) [0x55d2bd092793] 7: (RGWRados::update_gc_chain(rgw_obj&, RGWObjManifest&, cls_rgw_obj_chain*)+0x51a) [0x55d2bd0939ea] 8: (RGWRados::Object::complete_atomic_modification()+0x83) [0x55d2bd093c63] 9: (RGWRados::Object::Delete::delete_obj()+0x74d) [0x55d2bd0a87ad] 10: (RGWDeleteObj::execute()+0x915) [0x55d2bd04b6d5] 11: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, bool)+0x915) [0x55d2bcdfbb35] 12: (process_request(RGWRados*, RGWREST*, RGWRequest*, std::string const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, optional_yield, rgw::dmclock::Scheduler*, int*)+0x1cd8) [0x55d2bcdfdea8] 13: (RGWCivetWebFrontend::process(mg_connection*)+0x38e) [0x55d2bcd41a1e] 14: (()+0x36bace) [0x55d2bcdccace] 15: (()+0x36d76f) [0x55d2bcdce76f] 16: (()+0x36dc18) [0x55d2bcdcec18] 17: (()+0x7dd5) [0x7f161d338dd5] 18: (clone()+0x6d) [0x7f161c84302d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. My guess is that we need to add more resources to the gateways? They have 2 CPUs and 12GB of memory running as virtual machines on centOS 7.6 . Any thoughts? -Brent

2 2

Re: Nautilus - osdmap not trimming
by m.sliwinski＠lh.pl 12 Nov '20

12 Nov '20

Hi Thanks for the reply. Yeah, i restarted all of the mon servers, in sequence, and yesterday just leader alone without any success. Reports: root@monb01:~# ceph report | grep committed report 4002437698 "monmap_first_committed": 1, "monmap_last_committed": 6, "osdmap_first_committed": 67114, "osdmap_last_committed": 72592, "mdsmap_first_committed": 1, "mdsmap_last_committed": 1, "first_committed": 609225, "last_committed": 609251, "first_committed": 180754137, "last_committed": 180754777, root@monb01:~# root@monb01:~# ceph report | jq .osdmap_clean_epochs report 395175214 { "min_last_epoch_clean": 72592, "last_epoch_clean": { "per_pool": [ { "poolid": 0, "floor": 72592 }, { "poolid": 1, "floor": 72592 }, { "poolid": 2, "floor": 72592 }, { "poolid": 3, "floor": 72592 }, { "poolid": 4, "floor": 72592 }, { "poolid": 5, "floor": 72592 }, { "poolid": 26, "floor": 72592 }, { "poolid": 27, "floor": 72592 }, { "poolid": 28, "floor": 72592 } ] }, "osd_epochs": [ { "id": 0, "epoch": 72592 }, { "id": 1, "epoch": 72592 }, { "id": 2, "epoch": 72592 }, { "id": 3, "epoch": 72592 }, { "id": 4, "epoch": 72592 }, { "id": 5, "epoch": 72592 }, { "id": 6, "epoch": 72592 }, { "id": 7, "epoch": 72592 }, { "id": 8, "epoch": 72592 }, { "id": 9, "epoch": 72592 }, { "id": 10, "epoch": 72592 }, { "id": 11, "epoch": 72592 }, { "id": 12, "epoch": 72592 }, { "id": 13, "epoch": 72592 }, { "id": 14, "epoch": 72592 }, { "id": 15, "epoch": 72592 }, { "id": 16, "epoch": 72592 }, { "id": 17, "epoch": 72592 }, { "id": 18, "epoch": 72592 }, { "id": 19, "epoch": 72592 }, { "id": 20, "epoch": 72592 }, { "id": 21, "epoch": 72592 }, { "id": 22, "epoch": 72592 }, { "id": 23, "epoch": 72592 }, { "id": 24, "epoch": 72592 }, { "id": 25, "epoch": 72592 }, { "id": 26, "epoch": 72592 }, { "id": 27, "epoch": 72592 }, { "id": 28, "epoch": 72592 }, { "id": 29, "epoch": 72592 }, { "id": 30, "epoch": 72592 }, { "id": 31, "epoch": 72592 }, { "id": 32, "epoch": 72592 }, { "id": 33, "epoch": 72592 }, { "id": 34, "epoch": 72592 }, { "id": 35, "epoch": 72592 }, { "id": 36, "epoch": 72592 }, { "id": 37, "epoch": 72592 }, { "id": 38, "epoch": 72592 }, { "id": 39, "epoch": 72592 }, { "id": 40, "epoch": 72592 }, { "id": 41, "epoch": 72592 }, { "id": 42, "epoch": 72592 }, { "id": 43, "epoch": 72592 }, { "id": 44, "epoch": 72592 }, { "id": 45, "epoch": 72592 }, { "id": 46, "epoch": 72592 }, { "id": 47, "epoch": 72592 }, { "id": 48, "epoch": 72592 }, { "id": 49, "epoch": 72592 }, { "id": 50, "epoch": 72592 }, { "id": 51, "epoch": 72592 }, { "id": 52, "epoch": 72592 }, { "id": 53, "epoch": 72592 }, { "id": 54, "epoch": 72592 }, { "id": 55, "epoch": 72592 }, { "id": 56, "epoch": 72592 }, { "id": 57, "epoch": 72592 }, { "id": 58, "epoch": 72592 }, { "id": 59, "epoch": 72592 }, { "id": 60, "epoch": 72592 }, { "id": 61, "epoch": 72592 }, { "id": 62, "epoch": 72592 }, { "id": 63, "epoch": 72592 }, { "id": 64, "epoch": 72592 }, { "id": 65, "epoch": 72592 }, { "id": 66, "epoch": 72592 }, { "id": 67, "epoch": 72592 }, { "id": 68, "epoch": 72592 }, { "id": 69, "epoch": 72592 }, { "id": 70, "epoch": 72592 }, { "id": 71, "epoch": 72592 }, { "id": 72, "epoch": 72592 }, { "id": 73, "epoch": 72592 }, { "id": 74, "epoch": 72592 }, { "id": 75, "epoch": 72592 }, { "id": 76, "epoch": 72592 }, { "id": 77, "epoch": 72592 }, { "id": 78, "epoch": 72592 }, { "id": 79, "epoch": 72592 }, { "id": 80, "epoch": 72592 }, { "id": 81, "epoch": 72592 }, { "id": 83, "epoch": 72592 }, { "id": 84, "epoch": 72592 }, { "id": 85, "epoch": 72592 }, { "id": 86, "epoch": 72592 }, { "id": 87, "epoch": 72592 }, { "id": 88, "epoch": 72592 }, { "id": 89, "epoch": 72592 }, { "id": 90, "epoch": 72592 }, { "id": 91, "epoch": 72592 }, { "id": 92, "epoch": 72592 }, { "id": 93, "epoch": 72592 }, { "id": 94, "epoch": 72592 }, { "id": 95, "epoch": 72592 }, { "id": 96, "epoch": 72592 }, { "id": 97, "epoch": 72592 }, { "id": 98, "epoch": 72592 }, { "id": 99, "epoch": 72592 }, { "id": 100, "epoch": 72592 }, { "id": 101, "epoch": 72592 }, { "id": 102, "epoch": 72592 }, { "id": 103, "epoch": 72592 }, { "id": 104, "epoch": 72592 }, { "id": 105, "epoch": 72592 }, { "id": 106, "epoch": 72592 }, { "id": 107, "epoch": 72592 }, { "id": 108, "epoch": 72592 }, { "id": 109, "epoch": 72592 }, { "id": 110, "epoch": 72592 }, { "id": 111, "epoch": 72592 }, { "id": 112, "epoch": 72592 }, { "id": 113, "epoch": 72592 }, { "id": 114, "epoch": 72592 }, { "id": 115, "epoch": 72592 }, { "id": 116, "epoch": 72592 }, { "id": 117, "epoch": 72592 }, { "id": 118, "epoch": 72592 }, { "id": 119, "epoch": 72592 }, { "id": 120, "epoch": 72592 }, { "id": 121, "epoch": 72592 }, { "id": 122, "epoch": 72592 }, { "id": 123, "epoch": 72592 }, { "id": 124, "epoch": 72592 }, { "id": 125, "epoch": 72592 }, { "id": 126, "epoch": 72592 }, { "id": 127, "epoch": 72592 }, { "id": 128, "epoch": 72592 }, { "id": 129, "epoch": 72592 }, { "id": 130, "epoch": 72592 }, { "id": 131, "epoch": 72592 }, { "id": 132, "epoch": 72592 }, { "id": 133, "epoch": 72592 }, { "id": 134, "epoch": 72592 }, { "id": 135, "epoch": 72592 }, { "id": 136, "epoch": 72592 }, { "id": 137, "epoch": 72592 }, { "id": 138, "epoch": 72592 }, { "id": 139, "epoch": 72592 }, { "id": 140, "epoch": 72592 }, { "id": 141, "epoch": 72592 }, { "id": 142, "epoch": 72592 }, { "id": 143, "epoch": 72592 }, { "id": 144, "epoch": 72592 }, { "id": 145, "epoch": 72592 }, { "id": 146, "epoch": 72592 }, { "id": 147, "epoch": 72592 }, { "id": 148, "epoch": 72592 }, { "id": 149, "epoch": 72592 }, { "id": 150, "epoch": 72592 }, { "id": 151, "epoch": 72592 }, { "id": 152, "epoch": 72592 }, { "id": 153, "epoch": 72592 }, { "id": 154, "epoch": 72592 }, { "id": 155, "epoch": 72592 }, { "id": 156, "epoch": 72592 }, { "id": 157, "epoch": 72592 }, { "id": 158, "epoch": 72592 }, { "id": 159, "epoch": 72592 }, { "id": 160, "epoch": 72592 }, { "id": 161, "epoch": 72592 }, { "id": 162, "epoch": 72592 }, { "id": 163, "epoch": 72592 }, { "id": 164, "epoch": 72592 }, { "id": 165, "epoch": 72592 }, { "id": 166, "epoch": 72592 }, { "id": 167, "epoch": 72592 }, { "id": 168, "epoch": 72592 }, { "id": 169, "epoch": 72592 }, { "id": 170, "epoch": 72592 }, { "id": 171, "epoch": 72592 }, { "id": 172, "epoch": 72592 }, { "id": 173, "epoch": 72592 }, { "id": 174, "epoch": 72592 }, { "id": 175, "epoch": 72592 }, { "id": 176, "epoch": 72592 }, { "id": 177, "epoch": 72592 }, { "id": 178, "epoch": 72592 }, { "id": 179, "epoch": 72592 }, { "id": 180, "epoch": 72592 }, { "id": 181, "epoch": 72592 }, { "id": 182, "epoch": 72592 }, { "id": 183, "epoch": 72592 }, { "id": 184, "epoch": 72592 }, { "id": 185, "epoch": 72592 }, { "id": 186, "epoch": 72592 }, { "id": 187, "epoch": 72592 }, { "id": 188, "epoch": 72592 }, { "id": 189, "epoch": 72592 }, { "id": 190, "epoch": 72592 }, { "id": 191, "epoch": 72592 }, { "id": 192, "epoch": 72592 }, { "id": 193, "epoch": 72592 }, { "id": 194, "epoch": 72592 }, { "id": 195, "epoch": 72592 }, { "id": 196, "epoch": 72592 }, { "id": 197, "epoch": 72592 }, { "id": 198, "epoch": 72592 }, { "id": 199, "epoch": 72592 }, { "id": 200, "epoch": 72592 }, { "id": 201, "epoch": 72592 }, { "id": 202, "epoch": 72592 }, { "id": 203, "epoch": 72592 }, { "id": 204, "epoch": 72592 }, { "id": 205, "epoch": 72592 }, { "id": 206, "epoch": 72592 }, { "id": 207, "epoch": 72592 }, { "id": 208, "epoch": 72592 }, { "id": 209, "epoch": 72592 }, { "id": 210, "epoch": 72592 }, { "id": 211, "epoch": 72592 }, { "id": 212, "epoch": 72592 }, { "id": 213, "epoch": 72592 }, { "id": 214, "epoch": 72592 }, { "id": 215, "epoch": 72592 }, { "id": 216, "epoch": 72592 }, { "id": 217, "epoch": 72592 }, { "id": 218, "epoch": 72592 }, { "id": 219, "epoch": 72592 }, { "id": 220, "epoch": 72592 }, { "id": 221, "epoch": 72592 }, { "id": 222, "epoch": 72592 }, { "id": 223, "epoch": 72592 }, { "id": 224, "epoch": 72592 }, { "id": 225, "epoch": 72592 }, { "id": 226, "epoch": 72592 }, { "id": 227, "epoch": 72592 }, { "id": 228, "epoch": 72592 }, { "id": 229, "epoch": 72592 }, { "id": 230, "epoch": 72592 }, { "id": 231, "epoch": 72592 }, { "id": 232, "epoch": 72592 }, { "id": 233, "epoch": 72592 }, { "id": 234, "epoch": 72592 }, { "id": 235, "epoch": 72592 }, { "id": 236, "epoch": 72592 }, { "id": 237, "epoch": 72592 }, { "id": 238, "epoch": 72592 }, { "id": 239, "epoch": 72592 }, { "id": 240, "epoch": 72592 }, { "id": 241, "epoch": 72592 }, { "id": 242, "epoch": 72592 }, { "id": 243, "epoch": 72592 }, { "id": 244, "epoch": 72592 }, { "id": 245, "epoch": 72592 }, { "id": 246, "epoch": 72592 }, { "id": 247, "epoch": 72592 }, { "id": 248, "epoch": 72592 }, { "id": 249, "epoch": 72592 }, { "id": 250, "epoch": 72592 }, { "id": 251, "epoch": 72592 }, { "id": 252, "epoch": 72592 }, { "id": 253, "epoch": 72592 }, { "id": 254, "epoch": 72592 }, { "id": 255, "epoch": 72592 }, { "id": 256, "epoch": 72592 }, { "id": 257, "epoch": 72592 }, { "id": 258, "epoch": 72592 }, { "id": 259, "epoch": 72592 }, { "id": 260, "epoch": 72592 }, { "id": 261, "epoch": 72592 }, { "id": 262, "epoch": 72592 }, { "id": 263, "epoch": 72592 }, { "id": 264, "epoch": 72592 }, { "id": 265, "epoch": 72592 }, { "id": 266, "epoch": 72592 }, { "id": 267, "epoch": 72592 }, { "id": 268, "epoch": 72592 }, { "id": 269, "epoch": 72592 }, { "id": 270, "epoch": 72592 }, { "id": 271, "epoch": 72592 }, { "id": 272, "epoch": 72592 }, { "id": 273, "epoch": 72592 }, { "id": 274, "epoch": 72592 }, { "id": 275, "epoch": 72592 }, { "id": 276, "epoch": 72592 }, { "id": 277, "epoch": 72592 }, { "id": 278, "epoch": 72592 }, { "id": 279, "epoch": 72592 }, { "id": 280, "epoch": 72592 }, { "id": 281, "epoch": 72592 }, { "id": 282, "epoch": 72592 }, { "id": 283, "epoch": 72592 }, { "id": 284, "epoch": 72592 } ] } root@monb01:~# W dniu 2020-11-12 11:58, Dan van der Ster napisał(a): > I found another possible trimming bug this morning, but I don't expect > it applies to you because you said you restarted the mon leader: > https://tracker.ceph.com/issues/48212 > > Otherwise, couple you please share the output of > > ceph report | grep committed > ceph report | jq .osdmap_clean_epochs > > Thanks, > > Dan > > On Thu, Nov 12, 2020 at 10:56 AM <m.sliwinski(a)lh.pl> wrote: >> >> Hi >> >> Thanks for the response. Our cluster is currently mostly on 14.2.13, >> especially all MONs and MGRs are. >> Some OSDs are still on 14.2.9, but it shouldn't block osdmap trimming >> i >> think, because atm we don't have any down OSDs, i checked for that >> when >> we first noticed the issue. >> I'm working of course on bringing all OSDs to 14.2.13, but it will >> take >> some time as i have to create and test Debian packages for that. >> Could there be any other reason? I found posts about PGs for new pool >> not bein marked as created in MON db while cluster still reporting >> everything as active+clean, but i dont know how to debug that. >> >> -- >> Best regards >> Marcin >> >> >> W dniu 2020-11-11 16:50, Dan van der Ster napisał(a): >> > Hi, >> > >> > v14.2.13 has an important fix in this area: >> > https://tracker.ceph.com/issues/47290 >> > Without this fix, your cluster will not trim if there are any *down* >> > osds in the cluster. >> > >> > On our clusters we are running v14.2.11 patched with commit >> > "mon/OSDMonitor: only take in osd into consideration when trimming >> > osdmaps" -- this trims maps perfectly afaict. >> > >> > I can't vouch for the rest of 14.2.13, so better test that adequately >> > before upgrading. >> > >> > Cheers, Dan >> > >> > >> > On Tue, Nov 10, 2020 at 6:57 PM <m.sliwinski(a)lh.pl> wrote: >> >> >> >> Hi >> >> >> >> We have ceph cluster running on Nautilus, recently upgraded from >> >> Mimic. >> >> When in Mimic we noticed issue with osdmap not trimming, which caused >> >> part of our cluster to crash due to osdmap cache misses. We solved it >> >> by >> >> adding "osd_map_cache_size = 5000" to our ceph.conf >> >> Because we had at that time mixed OSD versions from both Mimic and >> >> Nautilus we decided to finish upgrade, but it didn't solve our >> >> problem. >> >> We have at the moment: "oldest_map": 67114, "newest_map": 72588,and >> >> the >> >> difference is not shrinking even thought cluster is in active+clean >> >> state. Restarting all mon's didn't help. It seems bug is similar to >> >> https://tracker.ceph.com/issues/44184 but there's no solution there. >> >> What else can i check or do? >> >> I don't want do to cangerous things like mon_osd_force_trim_to or >> >> something similar without finding the cause. >> >> >> >> I noticed in MON debug log: >> >> >> >> 2020-11-10 17:11:14.612 7f9592d5b700 10 mon.monb01(a)0(leader).osd >> >> e72571 >> >> should_prune could only prune 4957 epochs (67114..72071), which is >> >> less >> >> than the required minimum (10000) >> >> 2020-11-10 17:11:19.612 7f9592d5b700 10 mon.monb01(a)0(leader).osd >> >> e72571 >> >> should_prune could only prune 4957 epochs (67114..72071), which is >> >> less >> >> than the required minimum (10000) >> >> >> >> So i added config options to reduce those values: >> >> >> >> mon dev mon_debug_block_osdmap_trim false >> >> mon advanced mon_min_osdmap_epochs 100 >> >> mon advanced mon_osdmap_full_prune_min 500 >> >> mon advanced paxos_service_trim_min 10 >> >> >> >> But it didn't help: >> >> >> >> 2020-11-10 18:28:26.165 7f1b700ab700 20 mon.monb01(a)0(leader).osd >> >> e72588 >> >> load_osdmap_manifest osdmap manifest detected in store; reload. >> >> 2020-11-10 18:28:26.169 7f1b700ab700 10 mon.monb01(a)0(leader).osd >> >> e72588 >> >> load_osdmap_manifest store osdmap manifest pinned (67114 .. 72484) >> >> 2020-11-10 18:28:26.169 7f1b700ab700 10 mon.monb01(a)0(leader).osd >> >> e72588 >> >> should_prune not enough epochs to form an interval (last pinned: >> >> 72484, >> >> last to pin: 72488, interval: 10) >> >> >> >> Command "ceph report | jq '.osdmap_manifest' |jq '.pinned_maps[]'" >> >> shows >> >> 67114 on the top, but i'm unable to determine why. >> >> >> >> Same with 'ceph report | jq .osdmap_first_committed': >> >> >> >> root@monb01:/var/log/ceph# ceph report | jq .osdmap_first_committed >> >> report 4073203295 >> >> 67114 >> >> root@monb01:/var/log/ceph# >> >> >> >> When i try to derermine if a certain PG or OSD is keeping it so low i >> >> don't get anything. >> >> >> >> And in MON debug log i get: >> >> >> >> 2020-11-10 18:42:41.767 7f1b74721700 10 mon.monb01@0(leader) e6 >> >> refresh_from_paxos >> >> 2020-11-10 18:42:41.767 7f1b74721700 10 >> >> mon.monb01(a)0(leader).paxosservice(mdsmap 1..1) refresh >> >> 2020-11-10 18:42:41.767 7f1b74721700 10 >> >> mon.monb01(a)0(leader).paxosservice(osdmap 67114..72588) refresh >> >> 2020-11-10 18:42:41.767 7f1b74721700 20 mon.monb01(a)0(leader).osd >> >> e72588 >> >> load_osdmap_manifest osdmap manifest detected in store; reload. >> >> 2020-11-10 18:42:41.767 7f1b74721700 10 mon.monb01(a)0(leader).osd >> >> e72588 >> >> load_osdmap_manifest store osdmap manifest pinned (67114 .. 72484) >> >> >> >> I also get: >> >> >> >> root@monb01:/var/log/ceph# ceph report |grep "min_last_epoch_clean" >> >> report 2716976759 >> >> "min_last_epoch_clean": 0, >> >> root@monb01:/var/log/ceph# >> >> >> >> >> >> Additional info: >> >> root@monb01:/var/log/ceph# ceph versions >> >> { >> >> "mon": { >> >> "ceph version 14.2.13 >> >> (1778d63e55dbff6cedb071ab7d367f8f52a8699f) >> >> nautilus (stable)": 3 >> >> }, >> >> "mgr": { >> >> "ceph version 14.2.13 >> >> (1778d63e55dbff6cedb071ab7d367f8f52a8699f) >> >> nautilus (stable)": 3 >> >> }, >> >> "osd": { >> >> "ceph version 14.2.13 >> >> (1778d63e55dbff6cedb071ab7d367f8f52a8699f) >> >> nautilus (stable)": 120, >> >> "ceph version 14.2.9 >> >> (581f22da52345dba46ee232b73b990f06029a2a0) >> >> nautilus (stable)": 164 >> >> }, >> >> "mds": {}, >> >> "overall": { >> >> "ceph version 14.2.13 >> >> (1778d63e55dbff6cedb071ab7d367f8f52a8699f) >> >> nautilus (stable)": 126, >> >> "ceph version 14.2.9 >> >> (581f22da52345dba46ee232b73b990f06029a2a0) >> >> nautilus (stable)": 164 >> >> } >> >> } >> >> >> >> >> >> root@monb01:/var/log/ceph# ceph mon feature ls >> >> >> >> all features >> >> supported: [kraken,luminous,mimic,osdmap-prune,nautilus] >> >> persistent: [kraken,luminous,mimic,osdmap-prune,nautilus] >> >> on current monmap (epoch 6) >> >> persistent: [kraken,luminous,mimic,osdmap-prune,nautilus] >> >> required: [kraken,luminous,mimic,osdmap-prune,nautilus] >> >> >> >> >> >> root@monb01:/var/log/ceph# ceph osd dump | grep require >> >> require_min_compat_client luminous >> >> require_osd_release nautilus >> >> >> >> >> >> root@monb01:/var/log/ceph# ceph report | jq >> >> '.osdmap_manifest.pinned_maps | length' >> >> report 1777129876 >> >> 538 >> >> >> >> root@monb01:/var/log/ceph# ceph pg dump -f json | jq .osd_epochs >> >> dumped all >> >> null >> >> >> >> -- >> >> Best regards >> >> Marcin >> >> _______________________________________________ >> >> ceph-users mailing list -- ceph-users(a)ceph.io >> >> To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 2

How to run ceph_osd_dump
by Denis Krienbühl 12 Nov '20

12 Nov '20

Hi We’ve recently encountered the following errors: [WRN] OSD_SLOW_PING_TIME_BACK: Slow OSD heartbeats on back (longest 2752.832ms) Slow OSD heartbeats on back from osd.2 [nvme-a] to osd.290 [nvme-c] 2752.832 msec ... Truncated long network list. Use ceph daemon mgr.# dump_osd_network for more information To get more information we wanted to run the dump_osd_network command, but it doesn’t seem to be a valid command: ceph daemon /var/run/ceph/ceph-mgr.$(hostname).asok dump_osd_network 0 no valid command found; 10 closest matches: 0 1 2 abort assert config diff config diff get <var> config get <var> config help [<var>] config set <var> <val>... admin_socket: invalid command Other commands, like ceph daemon dump_cache work, so it seems to hit the right socket. What am I doing wrong? Cheers, Denis

2 2

How to use ceph-volume to create multiple OSDs per NVMe disk, and with fixed WAL/DB partition on another device?
by victorhooi＠yahoo.com 11 Nov '20

11 Nov '20

I'm building a new 4-node Proxmox/Ceph cluster, to hold disk images for our VMs. (Ceph version is 15.2.5). Each node has 6 x NVMe SSDs (4TB), and 1 x Optane drive (960GB). CPU is AMD Rome 7442, so there should be plenty of CPU capacity to spare. My aim is to create 4 x OSDs per NVMe SSD (to make more effective use of the NVMe performance) and use the Optane drive to store the WAL/DB partition for each OSD. (I.e. total of 24 x 35GB WAL/DB partitions). However, I am struggling to get the right ceph-volume command to achieve this. Thanks to a very kind Redditor, I was able to get close: /dev/nvme0n1 is an Optane device (900GB). /dev/nvme2n1 is an Intel NVMe SSD (4TB). ``` # ceph-volume lvm batch --osds-per-device 4 /dev/nvme2n1 --db-devices /dev/nvme0n1 Total OSDs: 4 Solid State VG: Targets: block.db Total size: 893.00 GB Total LVs: 16 Size per LV: 223.25 GB Devices: /dev/nvme0n1 Type Path LV Size % of device ---------------------------------------------------------------------------------------------------- [data] /dev/nvme2n1 931.25 GB 25.0% [block.db] vg: vg/lv 223.25 GB 25% ---------------------------------------------------------------------------------------------------- [data] /dev/nvme2n1 931.25 GB 25.0% [block.db] vg: vg/lv 223.25 GB 25% ---------------------------------------------------------------------------------------------------- [data] /dev/nvme2n1 931.25 GB 25.0% [block.db] vg: vg/lv 223.25 GB 25% ---------------------------------------------------------------------------------------------------- [data] /dev/nvme2n1 931.25 GB 25.0% [block.db] vg: vg/lv 223.25 GB 25% --> The above OSDs would be created if the operation continues --> do you want to proceed? (yes/no) ``` This does split up the NVMe disk into 4 OSDs, and creates WAL/DB partition on the Optane drive - however, it creates 4 x 223 GB partitions on the Optane (whereas I want 35GB partitions). Is there any way to specify the WAL/DB partition size in the above? And can it be done, such that you can run successive ceph-volume commands, to add the OSDs and WAL/DB partitions for each NVMe disk? (Or if there's an easier way to achieve the above layout, please let me know). That being said - I also just saw this ceph-users thread: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/3Y6DEJCF7ZM… It talks there about "osd op num shards" and "osd op num threads per shard" - is there some way to set those, to achieve similar performance to say, 4 x OSDs per NVMe drive, but with only 1 x NVMe? Has anybody done any testing/benchmarking on this they can share?

5 4

newbie question: direct objects of different sizes to different pools?
by andersnb＠ucar.edu 11 Nov '20

11 Nov '20

Hi All, I'm exploring deploying Ceph at my organization for use as an object storage system (using the S3 RGW interface). My users have range of file sizes and I'd like to direct small files to a pool that uses replication and large files to a pool that uses erasure encoding. Is that possible? Thanks! Bill

3 4

Nautilus - osdmap not trimming
by m.sliwinski＠lh.pl 11 Nov '20

11 Nov '20

Hi We have ceph cluster running on Nautilus, recently upgraded from Mimic. When in Mimic we noticed issue with osdmap not trimming, which caused part of our cluster to crash due to osdmap cache misses. We solved it by adding "osd_map_cache_size = 5000" to our ceph.conf Because we had at that time mixed OSD versions from both Mimic and Nautilus we decided to finish upgrade, but it didn't solve our problem. We have at the moment: "oldest_map": 67114, "newest_map": 72588,and the difference is not shrinking even thought cluster is in active+clean state. Restarting all mon's didn't help. It seems bug is similar to https://tracker.ceph.com/issues/44184 but there's no solution there. What else can i check or do? I don't want do to cangerous things like mon_osd_force_trim_to or something similar without finding the cause. I noticed in MON debug log: 2020-11-10 17:11:14.612 7f9592d5b700 10 mon.monb01(a)0(leader).osd e72571 should_prune could only prune 4957 epochs (67114..72071), which is less than the required minimum (10000) 2020-11-10 17:11:19.612 7f9592d5b700 10 mon.monb01(a)0(leader).osd e72571 should_prune could only prune 4957 epochs (67114..72071), which is less than the required minimum (10000) So i added config options to reduce those values: mon dev mon_debug_block_osdmap_trim false mon advanced mon_min_osdmap_epochs 100 mon advanced mon_osdmap_full_prune_min 500 mon advanced paxos_service_trim_min 10 But it didn't help: 2020-11-10 18:28:26.165 7f1b700ab700 20 mon.monb01(a)0(leader).osd e72588 load_osdmap_manifest osdmap manifest detected in store; reload. 2020-11-10 18:28:26.169 7f1b700ab700 10 mon.monb01(a)0(leader).osd e72588 load_osdmap_manifest store osdmap manifest pinned (67114 .. 72484) 2020-11-10 18:28:26.169 7f1b700ab700 10 mon.monb01(a)0(leader).osd e72588 should_prune not enough epochs to form an interval (last pinned: 72484, last to pin: 72488, interval: 10) Command "ceph report | jq '.osdmap_manifest' |jq '.pinned_maps[]'" shows 67114 on the top, but i'm unable to determine why. Same with 'ceph report | jq .osdmap_first_committed': root@monb01:/var/log/ceph# ceph report | jq .osdmap_first_committed report 4073203295 67114 root@monb01:/var/log/ceph# When i try to derermine if a certain PG or OSD is keeping it so low i don't get anything. And in MON debug log i get: 2020-11-10 18:42:41.767 7f1b74721700 10 mon.monb01@0(leader) e6 refresh_from_paxos 2020-11-10 18:42:41.767 7f1b74721700 10 mon.monb01(a)0(leader).paxosservice(mdsmap 1..1) refresh 2020-11-10 18:42:41.767 7f1b74721700 10 mon.monb01(a)0(leader).paxosservice(osdmap 67114..72588) refresh 2020-11-10 18:42:41.767 7f1b74721700 20 mon.monb01(a)0(leader).osd e72588 load_osdmap_manifest osdmap manifest detected in store; reload. 2020-11-10 18:42:41.767 7f1b74721700 10 mon.monb01(a)0(leader).osd e72588 load_osdmap_manifest store osdmap manifest pinned (67114 .. 72484) I also get: root@monb01:/var/log/ceph# ceph report |grep "min_last_epoch_clean" report 2716976759 "min_last_epoch_clean": 0, root@monb01:/var/log/ceph# Additional info: root@monb01:/var/log/ceph# ceph versions { "mon": { "ceph version 14.2.13 (1778d63e55dbff6cedb071ab7d367f8f52a8699f) nautilus (stable)": 3 }, "mgr": { "ceph version 14.2.13 (1778d63e55dbff6cedb071ab7d367f8f52a8699f) nautilus (stable)": 3 }, "osd": { "ceph version 14.2.13 (1778d63e55dbff6cedb071ab7d367f8f52a8699f) nautilus (stable)": 120, "ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)": 164 }, "mds": {}, "overall": { "ceph version 14.2.13 (1778d63e55dbff6cedb071ab7d367f8f52a8699f) nautilus (stable)": 126, "ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)": 164 } } root@monb01:/var/log/ceph# ceph mon feature ls all features supported: [kraken,luminous,mimic,osdmap-prune,nautilus] persistent: [kraken,luminous,mimic,osdmap-prune,nautilus] on current monmap (epoch 6) persistent: [kraken,luminous,mimic,osdmap-prune,nautilus] required: [kraken,luminous,mimic,osdmap-prune,nautilus] root@monb01:/var/log/ceph# ceph osd dump | grep require require_min_compat_client luminous require_osd_release nautilus root@monb01:/var/log/ceph# ceph report | jq '.osdmap_manifest.pinned_maps | length' report 1777129876 538 root@monb01:/var/log/ceph# ceph pg dump -f json | jq .osd_epochs dumped all null -- Best regards Marcin

2 1

_get_class not permitted to load rgw_gc
by Dan van der Ster 11 Nov '20

11 Nov '20

Hi, We have this "not permitted to load rgw_gc" error on some of our osds. Anyone knows what this is and how to fix it? Nautilus 14.2.11 and CentOS 7 / 8: 2020-11-11 09:48:15.914 7f665c1ea700 0 _get_class not permitted to load rgw_gc 2020-11-11 09:48:15.914 7f665c1ea700 -1 osd.874 163331 class rgw_gc open got (1) Operation not permitted Cheers, Dan

1 3

disable / remove multisite sync RGW (Ceph Nautilus)
by Markus Gans 11 Nov '20

11 Nov '20

Hello everybody, we are running a multisite (active/active) gateway on 2 ceph clusters. One production and one backup cluster. Now we make a backup with rclone from the master and don't need anymore the second Gateway. What is the best way to shutdown the second Gateway and remove the multisite sync from the master without lost of the data on the master site. greetings Markus

2 1

Is there a way to make Cephfs kernel client to write data to ceph osd smoothly with buffer io
by Sage Meng 11 Nov '20

11 Nov '20

Hi All, Cephfs kernel client is influenced by kernel page cache when we write data to it, outgoing data will be huge when os starts flush page cache. So Is there a way to make Cephfs kernel client to write data to ceph osd smoothly when buffer io is used ?

2 1

safest way to re-crush a pool
by Michael Thomas 11 Nov '20

11 Nov '20

I'm setting up a radosgw for my ceph Octopus cluster. As soon as I started the radosgw service, I notice that it created a handful of new pools. These pools were assigned the 'replicated_data' crush rule automatically. I have a mixed hdd/ssd/nvme cluster, and this 'replicated_data' crush rule spans all device types. I would like radosgw to use a replicated SSD pool and avoid the HDDs. What is the recommended way to change the crush device class for these pools without risking the loss of any data in the pools? I will note that I have not yet written any user data to the pools. Everything in them was added by the radosgw process automatically. --Mike

2 2

Ceph RBD - High IOWait during the Writes
by athreyavc 10 Nov '20

10 Nov '20

Hi All, We have recently deployed a new CEPH cluster Octopus 15.2.4 which consists of 12 OSD Nodes(16 Core + 200GB RAM, 30x14TB disks, CentOS 8) 3 Mon Nodes (8 Cores + 15GB, CentOS 8) We use Erasure Coded Pool and RBD block devices. 3 Ceph clients use the RBD devices, each has 25 RBDs and Each RBD size is 10TB. Each RBD is partitioned with the EXT4 file system. Cluster Health Is OK and Hardware is New and good. All the machines have 10Gbps (Active/Passive) bond Interface configured on it. Read operation of the cluster is OK, however, writes are very slow. One one of the RBDs we did the perf test. fio -ioengine=libaio -direct=1 -invalidate=1 -name=test -bs=4k -iodepth=128 -rw=randread -runtime=60 -filename=/dev/rbd40 Run status group 0 (all jobs): READ: bw=401MiB/s (420MB/s), 401MiB/s-401MiB/s (420MB/s-420MB/s), io=23.5GiB (25.2GB), run=60054-60054msec fio -ioengine=libaio -direct=1 -invalidate=1 -name=test -bs=4k -iodepth=128 -rw=randwrite -runtime=60 -filename=/dev/rbd40 Run status group 0 (all jobs): WRITE: bw=217KiB/s (222kB/s), 217KiB/s-217KiB/s (222kB/s-222kB/s), io=13.2MiB (13.9MB), run=62430-62430msec I see a High IO wait from the client. Any suggestions/pointers address this issue is really appreciated. Thanks and Regards, Athreya

3 3

RGW multisite sync and latencies problem
by Miroslav Bohac 10 Nov '20

10 Nov '20

Hi, I have problem with RGW in multisite configuration with Nautilus 14.2.11. Both zones with SSD and 10Gbps network. Master zone consist from 5x DELL R740XD servers (every 256GB RAM, 8x800GB SSD for CEPH, 24xCPU). Secondary zone (temporary for testing) consist from 3x HPE DL360 Gen10 servers (every 256GB RAM, 6x800GB SSD, 48CPU). We have 17 test buckets with manual sharding (101 shards). Every bucket with 10M of small objects (10kB - 15kB). Zonegroup configuration is attached bellow. Replication of 150M objects from master to secondary zone takes almost 28 hours and the replication completed with success. After deleting objects from one bucket in master zone is not possible to sync zones properly. I tried to restart both secondary RGWs, but without success. Sync status on secondary zone is behind master. The number of objects in buckets on master zone is different than on secondary zone. Ceph HEALTH status is WARNING on both zones. On master zone I have 146 large objects found in pool 'prg2a-1.rgw.buckets.index' 16 large objects found in pool 'prg2a-1.rgw.log'. On secondary zone 88 large objects found in pool 'prg2a-2.rgw.log' 1584 large objects found in pool 'prg2a-2.rgw.buckets.index'. AVG OSD latencies on secondary zone during sync was "read 0,158ms, write 1,897ms, overwrite 1,634ms". After unsuccesfull sync (after 12h of sync fall down RGW requests, IOPS and throughput) jumps up AVG OSD latencies to "read 125ms, write 30ms, overwrite 272ms". After stopping of both RGWs on secondary zone are AVG OSD latencies almost 0ms, but when I start RGWs on secondary zone again, OSD latencies will rise again to "read 125ms, write 30ms, overwrite 272ms" with spikes up to 3 seconds. We have seen the same behaviour of ceph multisite with large number of object in one bucket (150M+ objects), so we tried different strategy with smaller buckets, but results are same. I will appreciate any help or advice, how tune or diagnose multisite problems. Does anyone else have any ideas? Is there anyone else with a similar use-case? I do not know what is wrong. Thank you and best regards, Miroslav radosgw-admin zonegroup get { "id": "ac0005da-2e9f-4f38-835f-72b289c240d0", "name": "prg2a", "api_name": "prg2a", "is_master": "true", "endpoints": [ "http://s3.prg1a.sys.cz:80", "http://s3.prg2a.sys.cz:80" ], "hostnames": [], "hostnames_s3website": [], "master_zone": "d9ebbd1f-3312-4083-b4c2-843e1fb899ad", "zones": [ { "id": "d9ebbd1f-3312-4083-b4c2-843e1fb899ad", "name": "prg2a-1", "endpoints": [ "http://10.104.200.101:7480", "http://10.104.200.102:7480" ], "log_meta": "false", "log_data": "true", "bucket_index_max_shards": 0, "read_only": "false", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" }, { "id": "fdd76c02-c679-4ec7-8e7d-c14d2ac74fb4", "name": "prg2a-2", "endpoints": [ "http://10.104.200.221:7480", "http://10.104.200.222:7480" ], "log_meta": "false", "log_data": "true", "bucket_index_max_shards": 0, "read_only": "false", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" } ], "placement_targets": [ { "name": "default-placement", "tags": [], "storage_classes": [ "STANDARD" ] } ], "default_placement": "default-placement", "realm_id": "cb831094-e219-44b8-89f3-fe25fc288c00" ii radosgw 14.2.11-pve1 amd64 REST gateway for RADOS distributed object store ii ceph 14.2.11-pve1 amd64 distributed storage and file system ii ceph-base 14.2.11-pve1 amd64 common ceph daemon libraries and management tools ii ceph-common 14.2.11-pve1 amd64 common utilities to mount and interact with a ceph storage cluster ii ceph-fuse 14.2.11-pve1 amd64 FUSE-based client for the Ceph distributed file system ii ceph-mds 14.2.11-pve1 amd64 metadata server for the ceph distributed file system ii ceph-mgr 14.2.11-pve1 amd64 manager for the ceph distributed storage system ii ceph-mon 14.2.11-pve1 amd64 monitor server for the ceph storage system ii ceph-osd 14.2.11-pve1 amd64 OSD server for the ceph storage system ii libcephfs2 14.2.11-pve1 amd64 Ceph distributed file system client library

1 0

Slow ops and "stuck peering"
by shehzaad.chakowree＠teamto.com 10 Nov '20

10 Nov '20

Hello all, We're trying to debug a "slow ops" situation on our cluster running Nautilus (latest version). Things were running smoothly for a while, but we had a few issues that made things fall apart (possible clock skew, faulty disk...) - We've checked the ntp, everything seems fine, the whole cluster shows no clock skew. Network config seems fine too (we're using jumbo frames throughout the cluster). - We have multiple PGs that are in a "stuck peering" or "stuck inactive" state. ceph health detail HEALTH_WARN Reduced data availability: 1020 pgs inactive, 1008 pgs peering; Degraded data redundancy: 208352/95157861 objects degraded (0.219%), 9 pgs degraded, 9 pgs undersized; 2 pgs not deep-scrubbed in time; 2 pgs not scrubbed in time; 3 daemons have recently crashed; 1184 slow ops, oldest one blocked for 1792 sec, daemons [osd.100,osd.101,osd.102,osd.103,osd.104,osd.105,osd.106,osd.107,osd.108,osd.109]... have slow ops. PG_AVAILABILITY Reduced data availability: 1020 pgs inactive, 1008 pgs peering pg 12.3cd is stuck inactive for 8939.938831, current state peering, last acting [111,75,53] pg 12.3ce is stuck peering for 350761.931800, current state peering, last acting [48,103,76] pg 12.3cf is stuck peering for 345518.349253, current state peering, last acting [80,46,116] pg 12.3d0 is stuck peering for 396432.771388, current state peering, last acting [114,95,42] pg 12.3d1 is stuck peering for 389771.820478, current state peering, last acting [33,99,122] pg 12.3d2 is stuck peering for 16385.796714, current state peering, last acting [48,75,105] pg 12.3d3 is stuck peering for 375090.876123, current state peering, last acting [53,118,90] pg 12.3d4 is stuck peering for 350665.788611, current state peering, last acting [59,81,40] pg 12.3d5 is stuck peering for 344195.934260, current state peering, last acting [104,73,87] pg 12.3d6 is stuck peering for 388515.338772, current state peering, last acting [57,79,60] pg 12.3d7 is stuck peering for 27320.368320, current state peering, last acting [35,56,109] pg 12.3d8 is stuck peering for 345470.520103, current state peering, last acting [91,41,74] pg 12.3d9 is stuck peering for 347582.613090, current state peering, last acting [85,66,103] pg 12.3da is stuck peering for 346518.712024, current state peering, last acting [87,63,56] pg 12.3db is stuck peering for 348804.986864, current state peering, last acting [100,122,46] pg 12.3dc is stuck peering for 343796.439591, current state peering, last acting [55,90,125] pg 12.3dd is stuck peering for 345621.663979, current state peering, last acting [83,38,125] pg 12.3de is stuck peering for 348026.449482, current state peering, last acting [38,113,82] pg 12.3df is stuck peering for 350263.925579, current state peering, last acting [41,104,87] pg 12.3e0 is stuck peering for 8738.645205, current state peering, last acting [57,86,108] pg 12.3e1 is stuck peering for 397082.568164, current state peering, last acting [124,46] pg 12.3e2 is stuck peering for 345232.402459, current state peering, last acting [80,114,65] pg 12.3e3 is stuck peering for 347014.276511, current state peering, last acting [63,102,83] pg 12.3e4 is stuck peering for 345470.524144, current state peering, last acting [91,38,71] pg 12.3e5 is stuck peering for 346636.837554, current state peering, last acting [64,85,118] pg 12.3e6 is stuck peering for 398952.293609, current state peering, last acting [92,36,75] pg 12.3e7 is stuck peering for 346973.264600, current state peering, last acting [31,94,53] pg 12.3e8 is stuck peering for 370098.248268, current state peering, last acting [119,90,72] pg 12.3e9 is stuck peering for 345134.069457, current state peering, last acting [96,105,36] pg 12.3ea is stuck peering for 346305.043394, current state peering, last acting [94,103,51] pg 12.3eb is stuck peering for 388515.112735, current state peering, last acting [57,116,59] pg 12.3ec is stuck peering for 348097.249845, current state peering, last acting [56,111,84] pg 12.3ed is stuck peering for 346636.835287, current state peering, last acting [64,106,101] pg 12.3ee is stuck peering for 398197.856231, current state peering, last acting [53,105,80] pg 12.3ef is stuck peering for 347061.858678, current state peering, last acting [47,64,80] pg 12.3f0 is stuck peering for 371495.723196, current state peering, last acting [77,115,81] pg 12.3f1 is stuck peering for 27539.717691, current state peering, last acting [123,69,48] pg 12.3f2 is stuck peering for 346973.596729, current state peering, last acting [31,80,45] pg 12.3f3 is stuck peering for 345419.834162, current state peering, last acting [108,89,40] pg 12.3f4 is stuck peering for 347400.170304, current state peering, last acting [82,67,104] pg 12.3f5 is stuck peering for 346793.349638, current state peering, last acting [116,51,68] pg 12.3f6 is stuck peering for 372361.763947, current state peering, last acting [114,46,93] pg 12.3f7 is stuck inactive for 346840.292765, current state activating, last acting [125,77,47] pg 12.3f8 is stuck peering for 347004.967439, current state peering, last acting [42,31,116] pg 12.3f9 is stuck peering for 346894.489185, current state peering, last acting [40,94,67] pg 12.3fa is stuck peering for 395041.494033, current state peering, last acting [58,97,112] pg 12.3fb is stuck peering for 346337.742759, current state peering, last acting [79,55,61] pg 12.3fc is stuck peering for 347634.039502, current state peering, last acting [66,54,101] pg 12.3fd is stuck peering for 345340.666831, current state peering, last acting [112,32,87] pg 12.3fe is stuck peering for 345777.554974, current state peering, last acting [98,30,44] pg 12.3ff is stuck peering for 18040.716533, current state peering, last acting [86,51,59] PG_DEGRADED Degraded data redundancy: 208352/95157861 objects degraded (0.219%), 9 pgs degraded, 9 pgs undersized pg 7.3b is stuck undersized for 1119305.639931, current state active+undersized+degraded, last acting [29,19] pg 7.5a is stuck undersized for 351332.251298, current state active+undersized+degraded, last acting [29,11] pg 8.b9 is stuck undersized for 351332.246585, current state active+undersized+degraded, last acting [22,17] pg 8.c1 is stuck undersized for 351332.257178, current state active+undersized+degraded, last acting [24,14] pg 8.db is stuck undersized for 350987.698147, current state active+undersized+degraded, last acting [24,7] pg 8.1c2 is stuck undersized for 350999.603413, current state active+undersized+degraded, last acting [20,2] pg 9.32 is stuck undersized for 351332.258240, current state active+undersized+degraded, last acting [21,11] pg 9.a5 is stuck undersized for 351332.266130, current state active+undersized+degraded, last acting [25,14] pg 9.df is stuck undersized for 351333.298597, current state active+undersized+degraded, last acting [25,19] PG_NOT_DEEP_SCRUBBED 2 pgs not deep-scrubbed in time pg 8.db not deep-scrubbed since 2020-10-24 07:09:12.599242 pg 7.3b not deep-scrubbed since 2020-10-24 14:10:59.877193 PG_NOT_SCRUBBED 2 pgs not scrubbed in time pg 8.db not scrubbed since 2020-10-24 07:09:12.599242 pg 7.3b not scrubbed since 2020-10-24 14:10:59.877193 RECENT_CRASH 3 daemons have recently crashed osd.70 crashed on host starfish-osd-05 at 2020-10-30 09:16:06.981832Z osd.57 crashed on host starfish-osd-04 at 2020-11-06 10:07:47.868835Z mds.starfish-mon-01 crashed on host starfish-mon-01 at 2020-11-02 18:36:25.266426Z SLOW_OPS 1184 slow ops, oldest one blocked for 1792 sec, daemons [osd.100,osd.101,osd.102,osd.103,osd.104,osd.105,osd.106,osd.107,osd.108,osd.109]... have slow ops. - The 9 degraded / undersized pgs are on a different pool that has osds that need to be reweighed. OSDs 1-29 are on another root on the crush map. - When querying one of the PGs that are in a "stuck peering" state,, there are a lot of ".handle_connect_reply_2 connect got BADAUTHORIZER" replys. - The OSDs logs show the following message (they dissapear for a while if the osd is restarted) : 2020-11-10 11:58:58.671 7f90430da700 0 auth: could not find secret_id=14160 2020-11-10 11:58:58.671 7f90430da700 0 cephx: verify_authorizer could not get service secret for service osd secret_id=14160 2020-11-10 11:58:58.671 7f90430da700 0 --1- [v2:10.100.0.7:6851/483865,v1:10.100.0.7:6854/483865] >> v1:10.100.0.3:6815/8007973 conn(0x5632b1a64000 0x5632a64fb000 :6854 s=ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=0).handle_connect_message_2: got bad authorizer, auth_reply_len=0 Cheers ! Regards, Shehzaad

1 0

Re: Dovecot and fnctl locks
by Dan van der Ster 10 Nov '20

10 Nov '20

Hi, Yeah the negative pid is interesting. AFAICT we use a negative pid to indicate that the lock was taken on another host: https://github.com/torvalds/linux/blob/master/fs/ceph/locks.c#L119 https://github.com/torvalds/linux/commit/9d5b86ac13c573795525ecac6ed2db39ab… "Finally, we convert remote filesystems to present remote pids using negative numbers. Have lustre, 9p, ceph, cifs, and dlm negate the remote pid returned for F_GETLK lock requests." The good news is that my colleagues managed to clear this filelock by restarting dovecot on a couple nodes. But I'm still curious if others have a nice way to debug such things. Cheers, Dan On Mon, Nov 9, 2020 at 8:11 PM Anthony D'Atri <anthony.datri(a)gmail.com> wrote: > > Looks like a - in front of the 9605 — signed/unsigned int flern? > > > On Nov 9, 2020, at 4:59 AM, Dan van der Ster <dan(a)vanderster.com> wrote: > > > > Hi all, > > > > MDS version v14.2.11 > > Client kernel 3.10.0-1127.19.1.el7.x86_64 > > > > We are seeing a strange issue with a dovecot use-case on cephfs. > > Occasionally we have dovecot reporting a file locked, such as: > > > > Nov 09 13:55:00 dovecot-backend-00.cern.ch dovecot[27710]: > > imap(reguero)<23945><fRA6B6yznq68uE28>: Error: Mailbox Deleted Items: > > Timeout (180s) while waiting for lock for transaction log file > > /mail/users/r/reguero//mdbox/mailboxes/Deleted > > Items/dbox-Mails/dovecot.index.log (WRITE lock held by pid -9605) > > > > We checked all hosts that have mounted the cephfs -- there is no pid 9605. > > > > Is there any way to see who exactly created the lock? ceph_filelock > > has a client id, but I didn't find a way to inspect the > > cephfs_metadata to see the ceph_filelock directly. > > > > Otherwise, are other Dovecot/CephFS users seeing this? Did you switch > > to flock or lockfile instead of fnctlk locks? > > > > Thanks! > > > > Dan > > > > P.S. here is the output from print locks tool from the kernel client: > > > > Read lock: > > Type: 1 (0: Read, 1: Write, 2: Unlocked) > > Whence: 0 (0: start, 1: current, 2: end) > > Offset: 0 > > Len: 1 > > Pid: -9605 > > Write lock: > > Type: 1 (0: Read, 1: Write, 2: Unlocked) > > Whence: 0 (0: start, 1: current, 2: end) > > Offset: 0 > > Len: 1 > > Pid: -9605 > > > > and same file from a 15.2.5 fuse client : > > > > Read lock: > > Type: 1 (0: Read, 1: Write, 2: Unlocked) > > Whence: 0 (0: start, 1: current, 2: end) > > Offset: 0 > > Len: 0 > > Pid: 0 > > Write lock: > > Type: 1 (0: Read, 1: Write, 2: Unlocked) > > Whence: 0 (0: start, 1: current, 2: end) > > Offset: 0 > > Len: 0 > > Pid: 0 > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

2 1

cephfs - blacklisted client coming back?
by Andras Pataki 10 Nov '20

10 Nov '20

We had some network problems (high packet drops) to some cephfs client nodes that run ceph-fuse (14.2.13) against a Nautilus cluster (on version 14.2.8). As a result a couple of clients got evicted (as one would expect). What was really odd is that the clients were trying to flush data they had in cache and kept getting rejected by OSD's for almost an hour, and then magically the data flush worked. When asked afterwards, the client reported that it was no longer backlisted. How would that happen? I certainly didn't run any commands to un-blacklist a client and the docs say that otherwise the client will stay blacklisted until the file system gets remounted. Here is the status of the client when it was backlisted: [root@worker2033 ceph]# ceph daemon /var/run/ceph/ceph-client.cephfs2.7698.93825141588944.asok status { "metadata": { "ceph_sha1": "1778d63e55dbff6cedb071ab7d367f8f52a8699f", "ceph_version": "ceph version 14.2.13 (1778d63e55dbff6cedb071ab7d367f8f52a8699f) nautilus (stable)", "entity_id": "cephfs2", "hostname": "worker2033", "mount_point": "/mnt/ceph", "pid": "7698", "root": "/" }, "dentry_count": 252, "dentry_pinned_count": 9, "id": 111995680, "inst": { "name": { "type": "client", "num": 111995680 }, "addr": { "type": "v1", "addr": "10.254.65.33:0", "nonce": 410851087 } }, "addr": { "type": "v1", "addr": "10.254.65.33:0", "nonce": 410851087 }, "inst_str": "client.111995680 10.254.65.33:0/410851087", "addr_str": "10.254.65.33:0/410851087", "inode_count": 251, "mds_epoch": 3376260, "osd_epoch": 1717896, "osd_epoch_barrier": 1717893, "blacklisted": true } This corresponds to server side log messages: 2020-11-09 15:56:31.578 7fffe59a4700 1 mds.0.3376160 Evicting (and blacklisting) client session 111995680 (10.254.65.33:0/410851087) 2020-11-09 15:56:31.578 7fffe59a4700 0 log_channel(cluster) log [INF] : Evicting (and blacklisting) client session 111995680 (10.254.65.33:0/410851087) 2020-11-09 15:56:31.706 7fffe59a4700 1 mds.0.3376160 Evicting (and blacklisting) client session 111995680 (10.254.65.33:0/410851087) 2020-11-09 15:56:31.706 7fffe59a4700 0 log_channel(cluster) log [INF] : Evicting (and blacklisting) client session 111995680 (10.254.65.33:0/410851087) and them some time later (perhaps half an hour or so) I got this from the client: [root@worker2033 ceph]# ceph daemon /var/run/ceph/ceph-client.cephfs2.7698.93825141588944.asok status { "metadata": { "ceph_sha1": "1778d63e55dbff6cedb071ab7d367f8f52a8699f", "ceph_version": "ceph version 14.2.13 (1778d63e55dbff6cedb071ab7d367f8f52a8699f) nautilus (stable)", "entity_id": "cephfs2", "hostname": "worker2033", "mount_point": "/mnt/ceph", "pid": "7698", "root": "/" }, "dentry_count": 252, "dentry_pinned_count": 9, "id": 111995680, "inst": { "name": { "type": "client", "num": 111995680 }, "addr": { "type": "v1", "addr": "10.254.65.33:0", "nonce": 410851087 } }, "addr": { "type": "v1", "addr": "10.254.65.33:0", "nonce": 410851087 }, "inst_str": "client.111995680 10.254.65.33:0/410851087", "addr_str": "10.254.65.33:0/410851087", "inode_count": 251, "mds_epoch": 3376260, "osd_epoch": 1717897, "osd_epoch_barrier": 1717893, "blacklisted": false } The cluster was otherwise healthy - nothing wrong with MDS's, or any placement groups, etc. I also don't see any further log messages regarding eviction/backlisting in the MDS logs. I didn't run any ceph commands that would change the state of the cluster - I was just looking around, increasing log levels. Any ideas how could that have happened? A separate problem (perhaps needs a ticket filed) that while the ceph-fuse client was in a blacklisted state, it was retrying in an infinite loop to flush data to the OSD's and got rejected every time. I have some logs for the details of this too. Andras

3 6

disable / remove multisite sync RGW (Ceph Nautilus)
by gans＠teamnet.de 10 Nov '20

10 Nov '20

Hello everybody, we are running a multisite (active/active) gateway on 2 ceph clusters. One production and one backup cluster. Now we make a backup with rclone from the master and don't need anymore the second Gateway. What is the best way to shutdown the second Gateway and remove the multisite sync from the master without lost of the data on the master site. greetings Markus

1 0

The feasibility of mixed SSD and HDD replicated pool
by huww98＠outlook.com 10 Nov '20

10 Nov '20

Hi all, We are planning for a new pool to store our dataset using CephFS. These data are almost read-only (but not guaranteed) and consist of a lot of small files. Each node in our cluster has 1 * 1T SSD and 2 * 6T HDD, and we will deploy about 10 such nodes. We aim at getting the highest read throughput. If we just use a replicated pool of size 3 on SSD, we should get the best performance, however, that only leave us 1/3 of usable SSD space. And EC pools are not friendly to such small object read workload, I think. Now I’m evaluating a mixed SSD and HDD replication strategy. Ideally, I want 3 data replications, each on a different host (fail domain). 1 of them on SSD, the other 2 on HDD. And normally every read request is directed to SSD. So, if every SSD OSD is up, I’d expect the same read throughout as the all SSD deployment. I’ve read the documents and did some tests. Here is the crush rule I’m testing with: rule mixed_replicated_rule { id 3 type replicated min_size 1 max_size 10 step take default class ssd step chooseleaf firstn 1 type host step emit step take default class hdd step chooseleaf firstn -1 type host step emit } Now I have the following conclusions, but I’m not very sure: * The first OSD produced by crush will be the primary OSD (at least if I don’t change the “primary affinity”). So, the above rule is guaranteed to map SSD OSD as primary in pg. And every read request will read from SSD if it is up. * It is currently not possible to enforce SSD and HDD OSD to be chosen from different hosts. So, if I want to ensure data availability even if 2 hosts fail, I need to choose 1 SSD and 3 HDD OSD. That means setting the replication size to 4, instead of the ideal value 3, on the pool using the above crush rule. Am I correct about the above statements? How would this work from your experience? Thanks.

9 18

move rgw bucket to different pool
by Frank Ritchie 09 Nov '20

09 Nov '20

Hi All, Are there any methods for relocating an entire RGW bucket to a different storage pool other than copying the contents? Thanks, Frank

1 0

Mon went down and won't come back
by Paul Mezzanini 09 Nov '20

09 Nov '20

Hi everyone, I figure it's time to pull in more brain power on this one. We had an NVMe mostly die in one of our monitors and it caused the write latency for the machine to spike. Ceph did the RightThing(tm) and when it lost quorum on that machine it was ignored. I pulled the bad drive out of the array and tried to bring the mon and mgr back in (our monitors double-duty as managers). The manager came up 0 problems but the monitor got stuck probing. I removed the bad host from the monmap and stood up a new one on an OSD node to get back to 3 active. That new node added perfectly using the same methods I've tried on the old one. Network appears to be clean between all hosts. Packet captures show them chatting just fine. Since we are getting ready to upgrade from RHEL7 to RHEL8 I took this as an opportunity to reinstall the monitor as an 8 box to get that process rolling. Box is now on RHEL8 with no changes to how ceph-mon is acting. I install machines with a kickstart and use our own ansible roles to get it 95% into service. I then follow the manual install instructions (https://docs.ceph.com/en/latest/rados/operations/add-or-rm-mons/#adding-mon…) Time is in sync, /var/lib/ceph/mon/* is owned by the right UID, keys are in sync, configs are in sync. I pulled the old mon out of "mon initial members" and "mon host". `nc` can talk to all the ports in question and we've tried it with firewalld off as well (ditto with selinux). Cleaned up some stale DNS and even tried a different IP (same DNS name). I started all of this with 14.2.12 but .13 was released while debugging so I've got that on the broken monitor at the moment. I manually start the daemon in debug mode (/usr/bin/ceph-mon -d --cluster ceph --id ceph-mon-02 --setuser ceph --setgroup ceph) until it's joined in then use the systemd scripts to start it once it's clean. The current state is: (Lightly sanitized output) :snip: 2020-11-04 11:38:57.049 7f4232fb3540 0 mon.ceph-mon-02 does not exist in monmap, will attempt to join an existing cluster 2020-11-04 11:38:57.049 7f4232fb3540 0 using public_addr v2:Num.64:0/0 -> [v2:Num.64:3300/0,v1:Num.64:6789/0] 2020-11-04 11:38:57.050 7f4232fb3540 0 starting mon.ceph-mon-02 rank -1 at public addrs [v2:Num.64:3300/0,v1:Num.64:6789/0] at bind addrs [v2:Num.64:3300/0,v1:Num.64:6789/0] mon_data /var/lib/ceph/mon/ceph-ceph-mon-02 fsid 8514c8d5-4cd3-4dee-b460-27633e3adb1a 2020-11-04 11:38:57.051 7f4232fb3540 1 mon.ceph-mon-02@-1(???) e25 preinit fsid 8514c8d5-4cd3-4dee-b460-27633e3adb1a 2020-11-04 11:38:57.051 7f4232fb3540 1 mon.ceph-mon-02@-1(???) e25 initial_members ceph-mon-01,ceph-mon-03, filtering seed monmap 2020-11-04 11:38:57.051 7f4232fb3540 0 mon.ceph-mon-02(a)-1(???).mds e430081 new map 2020-11-04 11:38:57.051 7f4232fb3540 0 mon.ceph-mon-02(a)-1(???).mds e430081 print_map :snip: 2020-11-04 11:38:57.053 7f4232fb3540 0 mon.ceph-mon-02(a)-1(???).osd e1198618 crush map has features 288514119978713088, adjusting msgr requires 2020-11-04 11:38:57.053 7f4232fb3540 0 mon.ceph-mon-02(a)-1(???).osd e1198618 crush map has features 288514119978713088, adjusting msgr requires 2020-11-04 11:38:57.053 7f4232fb3540 0 mon.ceph-mon-02(a)-1(???).osd e1198618 crush map has features 3314933069571702784, adjusting msgr requires 2020-11-04 11:38:57.053 7f4232fb3540 0 mon.ceph-mon-02(a)-1(???).osd e1198618 crush map has features 288514119978713088, adjusting msgr requires 2020-11-04 11:38:57.054 7f4232fb3540 1 mon.ceph-mon-02(a)-1(???).paxosservice(auth 54141..54219) refresh upgraded, format 0 -> 3 2020-11-04 11:38:57.069 7f421d891700 1 mon.ceph-mon-02@-1(probing) e25 handle_auth_request failed to assign global_id ^^^ last line repeated every few seconds until process killed I've exhausted everything I can think of so I've just been doing the scientific shotgun (one slug at a time) approach to see what changes. Does anyone else have any ideas? -- Paul Mezzanini Sr Systems Administrator / Engineer, Research Computing Information & Technology Services Finance & Administration Rochester Institute of Technology o:(585) 475-3245 | pfmeec(a)rit.edu CONFIDENTIALITY NOTE: The information transmitted, including attachments, is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and destroy any copies of this information. ------------------------

2 6

Re: high latency after maintenance
by Marcel Kuiper 09 Nov '20

09 Nov '20

Yes the OSDs are all bluestore. So does this mean that we can assign most of the memory to the osd processes by setting the osd_memory_target? > > If your OSDs are all BlueStore, page cache isnât nearly as important as > with Filestore. > > >>

1 0

Multisite sync not working - permission denied
by Michael Breen 09 Nov '20

09 Nov '20

Hi, radosgw-admin -v ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable) Multisite sync was something I had working with a previous cluster and an earlier Ceph version, but it doesn't now, and I can't understand why. If anyone with an idea of a possible cause could give me a clue I would be grateful. I have clusters set up using Rook, but as far as I can tell, that's not a factor. On the primary cluster, I have this: radosgw-admin zonegroup get --rgw-zonegroup zonegroup-a { "id": "b115d74a-2d5f-4127-b621-0223f1e96c71", "name": "zonegroup-a", "api_name": "zonegroup-a", "is_master": "true", "endpoints": [ "http://192.168.30.8:80" ], "hostnames": [], "hostnames_s3website": [], "master_zone": "024687e0-1461-4f45-9149-9e571791c2b3", "zones": [ { "id": "024687e0-1461-4f45-9149-9e571791c2b3", "name": "zone-a", "endpoints": [ "http://192.168.30.8:80" ], "log_meta": "false", "log_data": "true", "bucket_index_max_shards": 11, "read_only": "false", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" }, { "id": "6ba0ee26-0155-48f9-b057-2803336f0d66", "name": "zone-b", "endpoints": [ "http://192.168.30.108:80" ], "log_meta": "false", "log_data": "true", "bucket_index_max_shards": 11, "read_only": "false", "tier_type": "", "sync_from_all": "true", "sync_from": [], "redirect_zone": "" } ], "placement_targets": [ { "name": "default-placement", "tags": [], "storage_classes": [ "STANDARD" ] } ], "default_placement": "default-placement", "realm_id": "8c38fa05-c19d-4e30-bc98-e2bc84eccb68", "sync_policy": { "groups": [] } } It's identical on the secondary (that's after a realm pull, an update of the zone-b endpoints, and a period commit), which I double-checked by piping the output to md5sum on both sides. The system user created on the primary is radosgw-admin user info --uid realm-a-system-user { ... "keys": [ { "user": "realm-a-system-user", "access_key": "IUs+USI5IjA8WkZPRjU=", "secret_key": "PGRDSzRERD4lbF9AYThuLzkvW1QvL148Q147PA==" } ... } The zones on both sides have these keys radosgw-admin zone get --rgw-zone zone-a { ... "system_key": { "access_key": "IUs+USI5IjA8WkZPRjU=", "secret_key": "PGRDSzRERD4lbF9AYThuLzkvW1QvL148Q147PA==" }, ... } radosgw-admin zone get --rgw-zonegroup zonegroup-a --rgw-zone zone-b { ... "system_key": { "access_key": "IUs+USI5IjA8WkZPRjU=", "secret_key": "PGRDSzRERD4lbF9AYThuLzkvW1QvL148Q147PA==" }, ... } Yet, on the secondary radosgw-admin sync status realm 8c38fa05-c19d-4e30-bc98-e2bc84eccb68 (realm-a) zonegroup b115d74a-2d5f-4127-b621-0223f1e96c71 (zonegroup-a) zone 6ba0ee26-0155-48f9-b057-2803336f0d66 (zone-b) metadata sync preparing for full sync full sync: 64/64 shards full sync: 0 entries to sync incremental sync: 0/64 shards metadata is behind on 64 shards behind shards: [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63] data sync source: 024687e0-1461-4f45-9149-9e571791c2b3 (zone-a) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is caught up with source and on the primary radosgw-admin sync status realm 8c38fa05-c19d-4e30-bc98-e2bc84eccb68 (realm-a) zonegroup b115d74a-2d5f-4127-b621-0223f1e96c71 (zonegroup-a) zone 024687e0-1461-4f45-9149-9e571791c2b3 (zone-a) metadata sync no sync (zone is master) 2020-11-06T10:58:46.345+0000 7fa805c201c0 0 data sync zone:6ba0ee26 ERROR: failed to fetch datalog info data sync source: 6ba0ee26-0155-48f9-b057-2803336f0d66 (zone-b) failed to retrieve sync info: (13) Permission denied Given that all the keys above match, that "permission denied" is a mystery to me, but it does accord with: export AWS_ACCESS_KEY_ID="IUs+USI5IjA8WkZPRjU=" export AWS_SECRET_ACCESS_KEY="PGRDSzRERD4lbF9AYThuLzkvW1QvL148Q147PA==" s3cmd ls --no-ssl --host-bucket= --host=192.168.30.8 # OK, but: s3cmd ls --no-ssl --host-bucket= --host=192.168.30.108 # ERROR: S3 error: 403 (InvalidAccessKeyId) # Although curl -L http://192.168.30.108 # works: <?xml version="1.0" encoding="UTF-8 ... 192.168.30.108 is the external IP, but just to be certain I was hitting zone-b, I tried this also within the cluster using its internal IP s3cmd ls --no-ssl --host-bucket= --host=10.41.157.115 # ERROR: S3 error: 403 (InvalidAccessKeyId) This seems to be the reason it's not syncing, but why? The user with those keys existed on the primary before the realm pull, in agreement with every procedure I have seen for setting up multisite. Any suggestions? Regards, Michael -- CONFIDENTIALITY This e-mail message and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail message, you are hereby notified that any dissemination, distribution or copying of this e-mail message, and any attachments thereto, is strictly prohibited. If you have received this e-mail message in error, please immediately notify the sender and permanently delete the original and any copies of this email and any prints thereof. ABSENT AN EXPRESS STATEMENT TO THE CONTRARY HEREINABOVE, THIS E-MAIL IS NOT INTENDED AS A SUBSTITUTE FOR A WRITING. Notwithstanding the Uniform Electronic Transactions Act or the applicability of any other law of similar substance and effect, absent an express statement to the contrary hereinabove, this e-mail message its contents, and any attachments hereto are not intended to represent an offer or acceptance to enter into a contract and are not otherwise intended to bind the sender, Sanmina Corporation (or any of its subsidiaries), or any other person or entity.

2 4

pg xyz is stuck undersized for long time
by Frank Schilder 09 Nov '20

09 Nov '20

Hi all, I moved the crush location of 8 OSDs and rebalancing went on happily (misplaced objects only). Today, osd.1 crashed, restarted and rejoined the cluster. However, it seems not to re-join some PGs it was a member of. I have now undersized PGs for no real reason I would believe: PG_DEGRADED Degraded data redundancy: 52173/2268789087 objects degraded (0.002%), 2 pgs degraded, 7 pgs undersized pg 11.52 is stuck undersized for 663.929664, current state active+undersized+remapped+backfilling, last acting [237,60,2147483647,74,233,232,292,86] The up and acting sets are: "up": [ 237, 2, 74, 289, 233, 232, 292, 86 ], "acting": [ 237, 60, 2147483647, 74, 233, 232, 292, 86 ], How can I get the PG to complete peering and osd.1 to join? I have an unreasonable number of degraded objects where the missing part is on this OSD. For completeness, here the cluster status: # ceph status cluster: id: ... health: HEALTH_ERR noout,norebalance flag(s) set 1 large omap objects 35815902/2268938858 objects misplaced (1.579%) Degraded data redundancy: 46122/2268938858 objects degraded (0.002%), 2 pgs degraded, 7 pgs undersized Degraded data redundancy (low space): 28 pgs backfill_toofull services: mon: 3 daemons, quorum ceph-01,ceph-02,ceph-03 mgr: ceph-01(active), standbys: ceph-03, ceph-02 mds: con-fs2-1/1/1 up {0=ceph-08=up:active}, 1 up:standby-replay osd: 299 osds: 275 up, 275 in; 301 remapped pgs flags noout,norebalance data: pools: 11 pools, 3215 pgs objects: 268.8 M objects, 675 TiB usage: 854 TiB used, 1.1 PiB / 1.9 PiB avail pgs: 46122/2268938858 objects degraded (0.002%) 35815902/2268938858 objects misplaced (1.579%) 2907 active+clean 219 active+remapped+backfill_wait 47 active+remapped+backfilling 28 active+remapped+backfill_wait+backfill_toofull 6 active+clean+scrubbing+deep 5 active+undersized+remapped+backfilling 2 active+undersized+degraded+remapped+backfilling 1 active+clean+scrubbing io: client: 13 MiB/s rd, 196 MiB/s wr, 2.82 kop/s rd, 1.81 kop/s wr recovery: 57 MiB/s, 14 objects/s Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 2

Dovecot and fnctl locks
by Dan van der Ster 09 Nov '20

09 Nov '20

Hi all, MDS version v14.2.11 Client kernel 3.10.0-1127.19.1.el7.x86_64 We are seeing a strange issue with a dovecot use-case on cephfs. Occasionally we have dovecot reporting a file locked, such as: Nov 09 13:55:00 dovecot-backend-00.cern.ch dovecot[27710]: imap(reguero)<23945><fRA6B6yznq68uE28>: Error: Mailbox Deleted Items: Timeout (180s) while waiting for lock for transaction log file /mail/users/r/reguero//mdbox/mailboxes/Deleted Items/dbox-Mails/dovecot.index.log (WRITE lock held by pid -9605) We checked all hosts that have mounted the cephfs -- there is no pid 9605. Is there any way to see who exactly created the lock? ceph_filelock has a client id, but I didn't find a way to inspect the cephfs_metadata to see the ceph_filelock directly. Otherwise, are other Dovecot/CephFS users seeing this? Did you switch to flock or lockfile instead of fnctlk locks? Thanks! Dan P.S. here is the output from print locks tool from the kernel client: Read lock: Type: 1 (0: Read, 1: Write, 2: Unlocked) Whence: 0 (0: start, 1: current, 2: end) Offset: 0 Len: 1 Pid: -9605 Write lock: Type: 1 (0: Read, 1: Write, 2: Unlocked) Whence: 0 (0: start, 1: current, 2: end) Offset: 0 Len: 1 Pid: -9605 and same file from a 15.2.5 fuse client : Read lock: Type: 1 (0: Read, 1: Write, 2: Unlocked) Whence: 0 (0: start, 1: current, 2: end) Offset: 0 Len: 0 Pid: 0 Write lock: Type: 1 (0: Read, 1: Write, 2: Unlocked) Whence: 0 (0: start, 1: current, 2: end) Offset: 0 Len: 0 Pid: 0

1 0

Re: high latency after maintenance
by Marcel Kuiper 09 Nov '20

09 Nov '20

Hi Anthony > > Did you add a bunch of data since then, or change the Ceph release? Do > you have bluefs_buffered_io set to false? > > We did not change ceph release in the meantime. It is very welll possible that the delays were just not noticed during out previous maintenances. bluefs_buffered_io is set to false (default setting in 14.2.11). I posted a question about this setting some time ago without any respons. Perhaps you are able to answer this: - If bluefs_buffered_io is set to false does that mean that all ceph buffering is done in the osd processes? Or is the linux buffer still used somewhere?? If the linux buffer is still used then what would be your advices in setting the osd_memory_target vs leaving space for linux buffers > > PGs block while peering, so it pays to spread out the peering load. > > Scrubs vary as a function of a number of things. Remember that shallow > scrubs are cheap and frequent, so if you have downtime theyâll need to > catch up when they come back. Especially if you also limit the times of > day when scrubs can run (which is usually a bad idea). Scrubs are not > themselves part of peering. Thank you we will keep that in mind for the next maintenance > > Are you using EC? What networking technology? > > We are not using EC. The network for OSD's consist of 2 x 10Gbps link in a bond. There is no separate cluster network. So far it look slike the network is nowhere used near its maximum Kind Regards Marcel

1 0

ceph command on cephadm install stuck
by Oliver Weinmann 09 Nov '20

09 Nov '20

Hi, on my fresh deployed cephadm bootstrap node I can no longer run the ceph command. It is just stuck: [root@gedasvl02 ~]# ceph orch dev ls INFO:cephadm:Inferring fsid c7879f24-1f90-11eb-8ba2-005056b703af INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15 [root@gedasvl02 ~]# ceph -s INFO:cephadm:Inferring fsid c7879f24-1f90-11eb-8ba2-005056b703af INFO:cephadm:Using recent ceph image docker.io/ceph/ceph:v15 I rebooted the node, but this didn't solve the issue. :( systemctl looks fine no stopped service. Any ideas? It's just a test cluster I can rebuild at any time, but I'm curious what has happened. Best Regards, Oliver

1 0

Multisite mechanism deeper understanding
by Szabo, Istvan (Agoda) 09 Nov '20

09 Nov '20

Hi, Couple of questions came up which is not really documented anywhere, hopefully someone knows the answers: 1. Is there a way to see the replication queue? I want to create metrics like is there any delay in the replication etc ... 2. Is the replication FIFO? 3. Actually how a replication works on a lower level? Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0

150mb per sec on NVMe pool
by Alex L 08 Nov '20

08 Nov '20

Hi, I have invested in SAMSUNG PM983 (MZ1LB960HAJQ-00007) x3 to run a fast pool on. However I am only getting 150mb/sec from these. vfio results directly on the NVMe's: https://docs.google.com/spreadsheets/d/1LXupjEUnNdf011QNr24pkAiDBphzpz5_MwM… Config and Results of ceph bench: https://pastebin.com/cScBv7Fv Appreciate any help you can give me. A

1 0

Ceph as a distributed filesystem and kerberos integration
by Alessandro Piazza 07 Nov '20

07 Nov '20

Dear all, I am experimenting with Ceph as a replacement for the AndrewFileSystem (https://en.wikipedia.org/wiki/Andrew_File_System) In my current setup, I am using AFS as a distributed filesystem for approximately 1000 users to store personal data and let them access their home directories and other shared data from multiple locations across different buildings. The authentication is managed by Kerberos (+ LDAP server). My goal is to replace AFS with CephFS but keep the current Kerberos database. Right now I've managed to set up a testing Ceph cluster with 6 nodes and 11 osds and I can mount CephFS using the kernel driver + CephX. However, from the Ceph docs, I can't understand if this might be a correct use-case for Ceph since the default authentication method CephX doesn't have a standard username/password authentication protocol. As far as I understand it requires the creation of a keyring with a random password generated on-the-fly which can then be used to mount the filesystem using the CephFS kernel module (https://docs.ceph.com/en/latest/cephfs/mount-using-kernel-driver/#mounting-…) As for the Kerberos integration, I found in the docs this page https://docs.ceph.com/en/latest/dev/ceph_krb_auth/ which is still a draft even if the last update was almost 2 years ago. From this page, I don't understand if the current version of Ceph supports full integration with GSSAPI/kerberos/LDAP. Since the docs only refer to keytab files, I was wondering if Kerberos can only be used as an authentication protocol between Ceph monitors/osds/metadata-servers and not for mounting the filesystem. Therefore I am asking - if anyone has tried Ceph for a similar use-case - what is the current status of Kerberos integration - if there are alternatives to CephX for mounting CephFS using kernel drivers which uses a username/password protocol Thank you and best regards, Alessandro Piazza

4 4

Debugging slow ops
by Void Star Nill 07 Nov '20

07 Nov '20

Hello, I am trying to debug slow operations in our cluster running Nautilus 14.2.13. I am analysing the output of "ceph daemon osd.N dump_historic_ops" command. I am noticing that the I am noticing that most of the time is spent between "header_read" and "throttled" events. For example, below is an operation that took ~160 seconds to complete and almost all of that time was spent between these 2 events. Going by the descriptions at https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd/#… - header_read: When the messenger first started reading the message off the wire. - throttled: When the messenger tried to acquire memory throttle space to read the message into memory. - all_read: When the messenger finished reading the message off the wire. Does this mean that the slowness I am observing is because OSD's messaging layer is not able to acquire the memory required for the message fast enough? The system has lots of available memory (over 300G), so how do I tune OSD to perform better at this? Appreciate any feedback on this. { "description": "osd_op(client.405792.0:98299 3.313 3:c8c63189:::rbd_data.51b046b8b4567.0000000000000180:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304] snapc 0=[] ondisk+write+known_if_redirected e1073)", "initiated_at": "2020-11-06 16:16:40.924448", "age": 164.32155802899999, "duration": 159.57800813, "type_data": { "flag_point": "commit sent; apply or cleanup", "client_info": { "client": "client.405792", "client_addr": "v1:x.y.156.101:0/3840080733", "tid": 98299 }, "events": [ { "time": "2020-11-06 16:16:40.924448", "event": "initiated" }, { "time": "2020-11-06 16:16:40.924448", "event": "header_read" }, { "time": "2020-11-06 16:19:20.481593", "event": "throttled" }, { "time": "2020-11-06 16:19:20.487331", "event": "all_read" }, { "time": "2020-11-06 16:19:20.487333", "event": "dispatched" }, { "time": "2020-11-06 16:19:20.487340", "event": "queued_for_pg" }, { "time": "2020-11-06 16:19:20.487372", "event": "reached_pg" }, { "time": "2020-11-06 16:19:20.487507", "event": "started" }, { "time": "2020-11-06 16:19:20.487586", "event": "waiting for subops from 1,94" }, { "time": "2020-11-06 16:19:20.491873", "event": "op_commit" }, { "time": "2020-11-06 16:19:20.501164", "event": "sub_op_commit_rec" }, { "time": "2020-11-06 16:19:20.502423", "event": "sub_op_commit_rec" }, { "time": "2020-11-06 16:19:20.502438", "event": "commit_sent" }, { "time": "2020-11-06 16:19:20.502456", "event": "done" } ] } }

1 0

using msgr-v1 for OSDs on nautilus
by Void Star Nill 06 Nov '20

06 Nov '20

Hello, I am running nautilus cluster. Is there a way to force the cluster to use msgr-v1 instead of msgr-v2? I am debugging an issue and it seems like it could be related to the msgr layer, so want to test it by using msgr-v1. Thanks, Shridhar

2 2

Re: high latency after maintenance]
by Marcel Kuiper 06 Nov '20

06 Nov '20

Hi Anthony Thank you for your respons I am looking at the"OSDs highest latency of write operations" panel of the grafana dashboard found in the ceph source in ./monitoring/grafana/dashboards/osds-overview.json. It is a topk graph that uses ceph_osd_op_w_latency_sum / ceph_osd_op_w_latency_count. During normal operations we see sometime latency spikes of 4 seconds max but during the bringing back of the rack we saw a consistent increase in latency for a lot of osds into the 20 seconds range The cluster has 1139 osds total of which we had 5 x 9 - 45 in maintenance We did not throttle the backfilling proces because we succesfully did the same maintenance before on a few occasions for other racks without problems. I will throttle backfills next time we have the same sort of maintenance in the next rack Can you elaborate a bit more what happens exactly during the peering process? I understand that the osds need to catch up. I also see that the nr of scrubs increases a lot when osds are brought back online. Is that part of the peering proces? Thx, Marcel > HDDs and concern for latency donât mix. That said, you donât specify > what you mean by âlatencyâ. Does that mean average client write > latency? median? P99? Something else? > > If you have a 15 node cluster and you took a third of it down for two > hours then yeah youâll have a lot to catch up on when you come back. > Bringing the nodes back one at a time can help, to spread out the peering. > Did you throttle backfill/recovery tunables all the way down to 1? In a > way that the restarted OSDs would use the throttled values as they boot? > > > > >> On Nov 5, 2020, at 6:47 AM, Marcel Kuiper <ceph(a)mknet.nl> wrote: >> >> Hi >> >> We had a rack down for 2hours for maintenance. 5 storage nodes were >> involved. We had noout en norebalance flags set before the start of the >> maintenance >> >> When the systems were brought back online we noticed a lot of osds with >> high latency (in 20 seconds range) . Mostly osds that are not on the >> storage nodes that were down. It took about 20 minutes for things to >> settle down. >> >> We're running nautilus 14.2.11. The storage nodes run bluestore and have >> 9 >> x 8T HDD's and 3 x SSD for rocksdb. Each with 3 x 123G LV >> >> - Can anyone give a reason for these high latencies? >> - Is there a way to avoid or lower these latencies when bringing systems >> back into operation? >> >> Best Regards >> >> Marcel >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

2 1

Hadoop to Ceph
by Szabo, Istvan (Agoda) 06 Nov '20

06 Nov '20

Hi, Is there anybody tried to migrate data from Hadoop to Ceph? If yes what is the right way? Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 3

Low Memory Nodes
by Ml Ml 06 Nov '20

06 Nov '20

Hello List, i think 3 of 6 Nodes have to less memory. This triggers the effect, that the nodes will swap a lot and almost kill themselfes. That triggers OSDs to go down, which triggers a rebalance which does not really help :D I already ordered more ram. Can i turn temporary down the RAM usage of the OSDs to not get into that vicious cycle and just suffer small but stable performance? This is ceph version 15.2.5 with bluestore. Thanks, Michael

3 2

NoSuchKey on key that is visible in s3 list/radosgw bk
by Mariusz Gronczewski 06 Nov '20

06 Nov '20

Hi, I've got a problem on Octopus (15.2.3, debian packages) install, bucket S3 index shows a file: s3cmd ls s3://upvid/255/38355 --recursive 2020-07-27 17:48 50584342 s3://upvid/255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4 radosgw-admin bi list also shows it { "type": "plain", "idx": "255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4", "entry": { "name": "255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4", "instance": "", "ver": { "pool": 11, "epoch": 853842 }, "locator": "", "exists": "true", "meta": { "category": 1, "size": 50584342, "mtime": "2020-07-27T17:48:27.203008Z", "etag": "2b31cc8ce8b1fb92a5f65034f2d12581-7", "storage_class": "", "owner": "filmweb-app", "owner_display_name": "filmweb app user", "content_type": "", "accounted_size": 50584342, "user_data": "", "appendable": "false" }, "tag": "_3ubjaztglHXfZr05wZCFCPzebQf-ZFP", "flags": 0, "pending_map": [], "versioned_epoch": 0 } }, but trying to download it via curl (I've set permissions to public0 only gets me <?xml version="1.0" encoding="UTF-8"?><Error><Code>NoSuchKey</Code><BucketName>upvid</BucketName><RequestId>tx0000000000000000e716d-005f1f14cb-e478a-pl-war1</RequestId><HostId>e478a-pl-war1-pl</HostId></Error> (the actually nonexisting files shows access denied in same context) same with other tools: $ s3cmd get s3://upvid/255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4 /tmp download: 's3://upvid/255/38355/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4' -> '/tmp/juz_nie_zyjesz_sezon_2___oficjalny_zwiastun___netflix_mp4' [1 of 1] ERROR: S3 error: 404 (NoSuchKey) cluster health is OK Any ideas what is happening here ? -- Mariusz Gronczewski, Administrator Efigence S. A. ul. Wołoska 9a, 02-583 Warszawa T: [+48] 22 380 13 13 NOC: [+48] 22 380 10 20 E: admin(a)efigence.com

3 3

msgr-v2 log flooding on OSD proceses
by Void Star Nill 06 Nov '20

06 Nov '20

Hello, I am running 14.2.13-1xenial version and I am seeing lot of logs from msgv2 layer on the OSDs. Attached are some of the logs. It looks like these logs are not controlled by the standard log level configuration, so I couldn't find a way to disable these logs. I am concerned that these logs may be hurting the system performance. Any inputs on how I can disable the level of these logs? Regards, Shridhar

1 0

Problem with checking mon for new map after upgrade
by Ingo Ebel 05 Nov '20

05 Nov '20

Hi, we upgraded our ceph cluster from 14.2.9 to 15.2.5 but osds with 15.2.5 are not joining the cluster after restart. They hang with "1234 tick checking mon for new map" The Systems are Centos 7.8 I tried everything i could think of. But nothing helped. The mons and mgrs are 15.2.5. Anyone an Idea want could cause this? 2020-11-05T15:18:13.142+0100 7f02fce81700 1 osd.114 pg_epoch: 1234 pg[5.c9s0( v 839'39171 (839'37583,839'39171] local-lis/les=1129/1130 n=495 ec=813/813 lis/c=1129/1081 les/c/f=1130/1082/0 sis=1210) [NONE,NONE,NONE,NONE,124,NONE,157,NONE,NONE,149,NONE]p124(4) r=-1 lpr=1212 pi=[1081,1210)/1 crt=839'39171 lcod 0'0 mlcod 0'0 unknown mbc={} ps=[1~3]] state<Start>: transitioning to Stray 2020-11-05T15:18:13.143+0100 7f02fce81700 1 osd.114 pg_epoch: 1234 pg[1.736( v 1105'9685 (815'8100,1105'9685] local-lis/les=1131/1132 n=6 ec=719/719 lis/c=1131/1019 les/c/f=1132/1020/0 sis=1210 pruub=11.135073235s) [] r=-1 lpr=1212 pi=[1019,1210)/1 crt=1105'9685 lcod 0'0 mlcod 0'0 unknown mbc={}] state<Start>: transitioning to Stray 2020-11-05T15:18:13.143+0100 7f02fce81700 1 osd.114 pg_epoch: 1234 pg[4.3fd( v 1109'105689 (1109'102600,1109'105689] local-lis/les=1131/1132 n=32 ec=740/740 lis/c=1131/1089 les/c/f=1132/1090/0 sis=1210 pruub=11.134586488s) [] r=-1 lpr=1212 pi=[1089,1210)/1 crt=1109'105689 lcod 0'0 mlcod 0'0 unknown mbc={}] state<Start>: transitioning to Stray 2020-11-05T15:18:13.144+0100 7f02fce81700 1 osd.114 pg_epoch: 1234 pg[5.123s10( v 839'38976 (839'37410,839'38976] local-lis/les=1119/1120 n=498 ec=813/813 lis/c=1119/1064 les/c/f=1120/1065/0 sis=1210) [NONE,NONE,NONE,NONE,NONE,NONE,NONE,NONE,NONE,NONE,NONE]p? r=-1 lpr=1212 pi=[1064,1210)/1 crt=839'38976 lcod 0'0 mlcod 0'0 unknown mbc={} ps=[1~3]] state<Start>: transitioning to Stray 2020-11-05T15:18:13.145+0100 7f03140b5700 1 osd.114 1234 set_numa_affinity public network bond0 numa node 2 2020-11-05T15:18:13.145+0100 7f03140b5700 1 osd.114 1234 set_numa_affinity cluster network bond1 numa node 0 2020-11-05T15:18:13.145+0100 7f03140b5700 1 osd.114 1234 set_numa_affinity public and cluster network numa nodes do not match 2020-11-05T15:18:13.145+0100 7f03140b5700 1 osd.114 1234 set_numa_affinity not setting numa affinity 2020-11-05T15:18:14.072+0100 7f031713a700 1 osd.114 1234 tick checking mon for new map 2020-11-05T15:18:44.397+0100 7f031713a700 1 osd.114 1234 tick checking mon for new map 2020-11-05T15:19:15.370+0100 7f031713a700 1 osd.114 1234 tick checking mon for new map 2020-11-05T15:19:46.206+0100 7f031713a700 1 osd.114 1234 tick checking mon for new map 2020-11-05T15:20:16.400+0100 7f031713a700 1 osd.114 1234 tick checking mon for new map 2020-11-05T15:20:46.483+0100 7f031713a700 1 osd.114 1234 tick checking mon for new map Regards -- Ingo Ebel Human knowledge belongs to the world. RadioTux.de - Internet-Radio rund um Linux und Open Source ## https://twitter.com/ingoebel ## https://keybase.io/savar ## Jabber: ingo.ebel(a)ingoebel.de

2 2

high latency after maintenance
by Marcel Kuiper 05 Nov '20

05 Nov '20

Hi We had a rack down for 2hours for maintenance. 5 storage nodes were involved. We had noout en norebalance flags set before the start of the maintenance When the systems were brought back online we noticed a lot of osds with high latency (in 20 seconds range) . Mostly osds that are not on the storage nodes that were down. It took about 20 minutes for things to settle down. We're running nautilus 14.2.11. The storage nodes run bluestore and have 9 x 8T HDD's and 3 x SSD for rocksdb. Each with 3 x 123G LV - Can anyone give a reason for these high latencies? - Is there a way to avoid or lower these latencies when bringing systems back into operation? Best Regards Marcel

1 0

Re: cephadm POC deployment with two networks, can't mount cephfs
by Juan Miguel Olmo Martinez 05 Nov '20

05 Nov '20

Hi Oliver Review this "step by step" guide to see if you forgot something: BR NFS: 1. chmod +x cephadm 2. ./cephadm bootstrap - Record dashboard user & password printed out at the end 3. ADD OTHER HOSTS (assuming 3+ total after adding) 4. ./cephadm shell 5. ceph orch apply osd --all-available-devices 6. ceph fs volume create test 1 7. ceph orch apply mds test 3 8. ceph nfs cluster create cephfs testnfs 9. ceph nfs cluster info testnfs - (verify hostname, ip and port are listed) - Record ip and port for later 10. ceph nfs export create cephfs test testnfs /cephfs 11. ceph auth ls - (check “client.testnfs1” keyring is present) 12. ceph nfs export get testnfs /cephfs - (should have output) 13. rados -p nfs-ganesha -N testnfs get export-1 - testnfs/cephnfs - (check that export was successfully created) 14. ceph nfs export ls testnfs - (should show pseudo path “/cephfs”) 15. Verify nfs export exists on dashboard - Login to dashboard with credentials from bootstrap - URL will be https://{host-ip}:8443/ - Navigate to NFS page - Table should contain the export you just created 16. Exit shell - Command should just be “exit” 17. systemctl status nfs-server - If service is listed as inactive, run “systemctl start nfs-server” - Run “systemctl status nfs-server”. Should now be active 18. sudo mount -t nfs -o port={nfs-port} {nfs-ip}:/cephfs /mnt - Port and ip should be from “ceph nfs cluster info testnfs” command ran earlier Ex: mount -t nfs -o port=2049 10.8.128.94:/cephfs /mnt/cephfs/ Then give mount command -> to check if its mounted #mount Output: 10.8.128.94:/cephfs on /mnt/cephfs type nfs4 (rw,relatime,seclabel,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.8.128.94,local_lock=none,addr=10.8.128.94) -- Juan Miguel Olmo Martínez Senior Software Engineer Red Hat <https://www.redhat.com/> jolmomar(a)redhat.com <https://www.redhat.com/>

1 0

RGW pubsub deprecation
by Yuval Lifshitz 05 Nov '20

05 Nov '20

Dear Community, Since Nautilus, we have 2 mechanisms for notifying 3rd parties on changes in buckets and objects: "bucket notifications" [1] and "pubsub" [2]. In "bucket notifications" (="push mode") the events are sent from the RGW to an external entity (kafka, rabbitmq etc.), while in "pubsub" (="pull mode") the events are synched with a special zone, where they are stored and could be later fetched by an external app. From communications that I've seen so far, users preferred to use "bucket notifications" over "pubsub". Since supporting both modes has maintenance overhead, I was considering deprecating "pubsub". However, before doing that I would like to see what the community has to say! So, if you are currently using pubsub, or plan to use it, as "pull mode" fits your usecase better than "push mode" please chime in. Yuval [1] https://docs.ceph.com/en/latest/radosgw/notifications/ [2] https://docs.ceph.com/en/latest/radosgw/pubsub-module/

1 0

cephadm POC deployment with two networks, can't mount cephfs
by Oliver Weinmann 05 Nov '20

05 Nov '20

Hi, I setup a small 3 node cluster as a POC. I bootstrapped the cluster with separate networks for frontend (public network 192.168.30.0/24) and backend (cluster network 192.168.41.0/24). 1st small question: After the bootstrap, I recognized that I had mixed up cluster and public network. :( Is there a way to fix this on a running cluster? Last resort I would rebuild the cluster. Never the less I can't mount cephfs on a Linux client using any of the two networks. My linux client is CentOS 7 (latest updates) and has 3 nics, two of them in one of the public and cluster networks. I bootstrapped the cluster using the following conf file to have two networks: /root/ceph.conf: [global] public network = 192.168.41.0/24 cluster network = 192.168.30.0/24 cephadm bootstrap -c /root/ceph.conf --mon-ip 192.168.30.11 I have 2 mons, one running on the bootstrap host (192.168.30.11 / 192.168.41.11) and one (gedaopl01 192.168.30.12/ 192.168.41.12) running on one of the 3 osds: [root@gedasvl02 ~]# ceph -s cluster: id: dad3c9fa-1ec7-11eb-94d6-005056b703af health: HEALTH_OK services: mon: 2 daemons, quorum gedasvl02,gedaopl01 (age 5h) mgr: gedasvl02.cspuee(active, since 12h), standbys: gedaopl01.llogef mds: cephfs:1 {0=cephfs.gedaopl03.prrkll=up:active} 1 up:standby osd: 3 osds: 3 up (since 11h), 3 in (since 11h) task status: scrub status: mds.cephfs.gedaopl03.prrkll: idle data: pools: 3 pools, 81 pgs objects: 29 objects, 2.2 KiB usage: 450 GiB used, 407 GiB / 857 GiB avail pgs: 81 active+clean [root@gedasvl02 ~]# ceph osd metadata 2 | grep addr "back_addr": "[v2:192.168.30.12:6800/3112350288,v1:192.168.30.12:6801/3112350288]", "front_addr": "[v2:192.168.41.12:6800/3112350288,v1:192.168.41.12:6801/3112350288]", "hb_back_addr": "[v2:192.168.30.12:6802/3112350288,v1:192.168.30.12:6803/3112350288]", "hb_front_addr": "[v2:192.168.41.12:6802/3112350288,v1:192.168.41.12:6803/3112350288]", Now when I try to mount cephfs from the linux client, the mount command is just stuck and runs into a timeout. I can ping the mon from the client on both IPs, public (192.168.41.12) and cluster (192.168.30.12) and I can also see packets coming in on the mon using tcpdump. What could be wrong here? I'm using fuse-cephfs. One more question regarding rebuilding the cluster using cephadm, is there a simple tear-down command? My bootstrap host is a VM so I can use snapshots, but the nodes I have to clean manually, by removing all pods and ceph directories. Best Regards, Oliver

1 0

File read are not completing and IO shows in bytes able to not reading from cephfs
by Amudhan P 05 Nov '20

05 Nov '20

Hi, In my test ceph octopus cluster I was trying to simulate a failure case of when client mounted cephfs thru kernel client and doing read and write process, shutting down entire cluster with OSD flags like no down, no out, no backfiling and no recovery. Cluster is 4 node composed of 3 mons, 2 mgr, 2 mds, 48 OSD's. Public IP range : 10.0.103.0 and Cluster IP range : 10.0.104.0 Write and Read got stalled after some time cluster was brought live and healthy. But when reading file thru kernel mount read start at above 100MB/s and suddenly drops to byte and continues for long. only error msg I could see in the client machine. [ 167.591095] ceph: loaded (mds proto 32) [ 167.600010] libceph: mon0 10.0.103.1:6789 session established [ 167.601167] libceph: client144519 fsid f8bc7682-0d11-11eb-a332-0cc47a5ec98a [ 272.132787] libceph: osd1 10.0.104.1:6891 socket closed (con state CONNECTING) What went wrong why is this issue.? regards Amudhan P

1 1

Ceph flash deployment
by Seena Fallah 04 Nov '20

04 Nov '20

Hi all, Does this guid is still valid for a bluestore deployment with nautilus or octopus? https://tracker.ceph.com/projects/ceph/wiki/Tuning_for_All_Flash_Deployments Thanks.

3 5

RBD image stuck and no erros on logs
by Salsa 04 Nov '20

04 Nov '20

Hi, This same error keeps happening to me: after writing some amount of data to an RBD image it gets stuck and no read or write operation on it works. Every operation hangs. I cannot resize, alter features, read or write data. I can mount it, but using parted or fdisk hangs indefinitely. In the end all I can do is remove the image. Again, I see no errors on the logs and Ceph's status is OK. I tried to alter some log levels, but still no helpful info. Is there anything I should check? Rados? -- Salsa Sent with ProtonMail Secure Email.

1 0

Re: Cephadm: module not found
by Nadiia Kotelnikova 04 Nov '20

04 Nov '20

Hi, i am experience the same problem. Could you please advise something how to resolve this issue? The fix should be shipped with 15.2.6 version of "ceph-common" or ceph version? I have my cluster in docker containers and systemd services. How can I upgrade cluster to 15.2.6 if the command for upgrading fails? sudo ceph orch upgrade start --ceph-version 15.2.5 Error ENOENT: Module not found

2 5

bluefs_buffered_io
by Marcel Kuiper 04 Nov '20

04 Nov '20

Hi list, I see a few changes in the (minor) version changelogs in the default for bluefs_buffered_io setting. Sometimes it is set to true in our version (14.2.11) it is set to false Can someone shed a light on this setting? I fail to find any documentation on it. ceph config help is not entirely clear to me as well - What does it do exactly when true - If false does that mean that the linux buffer cache is always skipped? And caching happens in the osd proces only? - if enabled should we lower the osd_memory_target to leave more space for the linux buffer cache? What would be the percentage of memory that we then assign to osd_memory_targets Marcel

1 0

Ceph 14.2 - some PGs stuck peering.
by m.sliwinski＠lh.pl 04 Nov '20

04 Nov '20

Hi We have a weird issue iwth our ceph cluster - almost all PGs assigned to one specific pool became stuck, locking out all operations without reporting any errors. Story: We have 3 different pools, hdd-backed, ssd-backed and nvme-backed. Pool ssh worked fine for few months. Today one of the hosts assigned to nvme pool restarted triggering recovery in that pool. It wnet fast and cluster went to OK state. During these events or shortly after them ssd pool became unresponsive. It was impossible to either read or write from/to it. We decided to slowly restart fist OSDs assigned to it, thenm as it didn't help - all the mons, wihout breaking quorum of course. At this moment both nvme and hdd polls are working fine, ssd one is stuck in recovery. All OSDs in that ssd pool use large amount of CPU and are exchanging approx 1Mpps per OSD server between each other. PGs seem to be slowly migrating from peering to activating but it's going very slowly - approx 10PGs during last hour. We were using 14.2.2 OSDs when issues happened, upgrade to 14.2.13 didn't help. We increased heartbeat grace, but it didn't change anything. It doesn't seem that there's a network problem as OSDs don't report problems with connecting to MONs or each other. Other OSDs - nvme, connected to that same set of switches work without issues. Can you help? Point me to what should i check or do? I looked on-line and on the group for causes of peering issues and checked most of them, nothing helped. I can't use 'ceph pg 28.1cc query' as it hangs, even for PGs that are marked as active+clean in the results of 'ceph pg dump' I checked status of the one of stuck PGs via ceph-objectstore-tool --data-path [...] --op info --pgid 28.29d for all three copies and got: { "pgid": "28.29d", "last_update": "68160'205094", "last_complete": "68160'205094", "log_tail": "68062'202000", "last_user_version": 205094, "last_backfill": "MAX", "last_backfill_bitwise": 0, "purged_snaps": [ { "start": "1", "length": "3" } ], "history": { "epoch_created": 67698, "epoch_pool_created": 67698, "last_epoch_started": 68871, "last_interval_started": 68851, "last_epoch_clean": 67746, "last_interval_clean": 67745, "last_epoch_split": 0, "last_epoch_marked_full": 0, "same_up_since": 69447, "same_interval_since": 69447, "same_primary_since": 69411, "last_scrub": "68062'199623", "last_scrub_stamp": "2020-11-03 03:32:46.895988", "last_deep_scrub": "68062'177321", "last_deep_scrub_stamp": "2020-11-02 01:07:15.963916", "last_clean_scrub_stamp": "2020-11-03 03:32:46.895988" }, "stats": { "version": "68160'205094", "reported_seq": "378496", "reported_epoch": "69447", "state": "peering", "last_fresh": "2020-11-03 20:55:39.247348", "last_change": "2020-11-03 20:55:39.247348", "last_active": "2020-11-03 15:26:24.270088", "last_peered": "2020-11-03 19:04:43.152655", "last_clean": "2020-11-03 14:45:02.988293", "last_became_active": "2020-09-01 13:52:40.091759", "last_became_peered": "2020-11-03 19:04:42.939991", "last_unstale": "2020-11-03 20:55:39.247348", "last_undegraded": "2020-11-03 20:55:39.247348", "last_fullsized": "2020-11-03 20:55:39.247348", "mapping_epoch": 69447, "log_start": "68062'202000", "ondisk_log_start": "68062'202000", "created": 67698, "last_epoch_clean": 67746, "parent": "0.0", "parent_split_bits": 0, "last_scrub": "68062'199623", "last_scrub_stamp": "2020-11-03 03:32:46.895988", "last_deep_scrub": "68062'177321", "last_deep_scrub_stamp": "2020-11-02 01:07:15.963916", "last_clean_scrub_stamp": "2020-11-03 03:32:46.895988", "log_size": 3094, "ondisk_log_size": 3094, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 15173849600, "num_objects": 3647, "num_object_clones": 0, "num_object_copies": 10941, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 3647, "num_whiteouts": 0, "num_read": 172836, "num_read_kb": 6824184, "num_write": 196190, "num_write_kb": 21380176, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 261, 284, 271 ], "acting": [ 261, 284, 271 ], "avail_no_missing": [], "object_location_counts": [], "blocked_by": [ 271, 284 ], "up_primary": 261, "acting_primary": 261, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 69422, "hit_set_history": { "current_last_update": "0'0", "history": [] } } { "pgid": "28.29d", "last_update": "68160'205094", "last_complete": "68160'205094", "log_tail": "68062'202000", "last_user_version": 205094, "last_backfill": "MAX", "last_backfill_bitwise": 0, "purged_snaps": [ { "start": "1", "length": "3" } ], "history": { "epoch_created": 67698, "epoch_pool_created": 67698, "last_epoch_started": 68871, "last_interval_started": 68851, "last_epoch_clean": 67746, "last_interval_clean": 67745, "last_epoch_split": 0, "last_epoch_marked_full": 0, "same_up_since": 69630, "same_interval_since": 69630, "same_primary_since": 69628, "last_scrub": "68062'199623", "last_scrub_stamp": "2020-11-03 03:32:46.895988", "last_deep_scrub": "68062'177321", "last_deep_scrub_stamp": "2020-11-02 01:07:15.963916", "last_clean_scrub_stamp": "2020-11-03 03:32:46.895988" }, "stats": { "version": "68160'205094", "reported_seq": "378445", "reported_epoch": "69627", "state": "peering", "last_fresh": "2020-11-03 21:15:08.819278", "last_change": "2020-11-03 21:14:18.360957", "last_active": "2020-11-03 15:26:24.270088", "last_peered": "2020-11-03 19:04:43.152655", "last_clean": "2020-11-03 14:45:02.988293", "last_became_active": "2020-09-01 13:52:40.091759", "last_became_peered": "2020-11-03 19:04:42.939991", "last_unstale": "2020-11-03 21:15:08.819278", "last_undegraded": "2020-11-03 21:15:08.819278", "last_fullsized": "2020-11-03 21:15:08.819278", "mapping_epoch": 69630, "log_start": "68062'202000", "ondisk_log_start": "68062'202000", "created": 67698, "last_epoch_clean": 67746, "parent": "0.0", "parent_split_bits": 0, "last_scrub": "68062'199623", "last_scrub_stamp": "2020-11-03 03:32:46.895988", "last_deep_scrub": "68062'177321", "last_deep_scrub_stamp": "2020-11-02 01:07:15.963916", "last_clean_scrub_stamp": "2020-11-03 03:32:46.895988", "log_size": 3094, "ondisk_log_size": 3094, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 15173849600, "num_objects": 3647, "num_object_clones": 0, "num_object_copies": 10941, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 3647, "num_whiteouts": 0, "num_read": 172836, "num_read_kb": 6824184, "num_write": 196190, "num_write_kb": 21380176, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 261, 284 ], "acting": [ 261, 284 ], "avail_no_missing": [], "object_location_counts": [], "blocked_by": [ 271 ], "up_primary": 261, "acting_primary": 261, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 69392, "hit_set_history": { "current_last_update": "0'0", "history": [] } } { "pgid": "28.29d", "last_update": "68160'205094", "last_complete": "68160'205094", "log_tail": "68062'202000", "last_user_version": 205094, "last_backfill": "MAX", "last_backfill_bitwise": 0, "purged_snaps": [ { "start": "1", "length": "3" } ], "history": { "epoch_created": 67698, "epoch_pool_created": 67698, "last_epoch_started": 68871, "last_interval_started": 68851, "last_epoch_clean": 67746, "last_interval_clean": 67745, "last_epoch_split": 0, "last_epoch_marked_full": 0, "same_up_since": 69411, "same_interval_since": 69411, "same_primary_since": 69411, "last_scrub": "68062'199623", "last_scrub_stamp": "2020-11-03 03:32:46.895988", "last_deep_scrub": "68062'177321", "last_deep_scrub_stamp": "2020-11-02 01:07:15.963916", "last_clean_scrub_stamp": "2020-11-03 03:32:46.895988" }, "stats": { "version": "68070'205093", "reported_seq": "378344", "reported_epoch": "68160", "state": "active+clean", "last_fresh": "2020-11-03 14:45:02.988293", "last_change": "2020-11-03 03:32:46.896044", "last_active": "2020-11-03 14:45:02.988293", "last_peered": "2020-11-03 14:45:02.988293", "last_clean": "2020-11-03 14:45:02.988293", "last_became_active": "2020-09-01 13:52:40.091759", "last_became_peered": "2020-09-01 13:52:40.091759", "last_unstale": "2020-11-03 14:45:02.988293", "last_undegraded": "2020-11-03 14:45:02.988293", "last_fullsized": "2020-11-03 14:45:02.988293", "mapping_epoch": 69411, "log_start": "68062'202000", "ondisk_log_start": "68062'202000", "created": 67698, "last_epoch_clean": 67746, "parent": "0.0", "parent_split_bits": 0, "last_scrub": "68062'199623", "last_scrub_stamp": "2020-11-03 03:32:46.895988", "last_deep_scrub": "68062'177321", "last_deep_scrub_stamp": "2020-11-02 01:07:15.963916", "last_clean_scrub_stamp": "2020-11-03 03:32:46.895988", "log_size": 3093, "ondisk_log_size": 3093, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 15173849600, "num_objects": 3647, "num_object_clones": 0, "num_object_copies": 10941, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 3647, "num_whiteouts": 0, "num_read": 172836, "num_read_kb": 6824184, "num_write": 196190, "num_write_kb": 21380176, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 261, 284, 271 ], "acting": [ 261, 284, 271 ], "avail_no_missing": [], "object_location_counts": [], "blocked_by": [], "up_primary": 261, "acting_primary": 261, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 67746, "hit_set_history": { "current_last_update": "0'0", "history": [] } } Current status of the cluster: Reduced data availability: 1021 pgs inactive, 999 pgs peering Degraded data redundancy: 18357/94939584 objects degraded (0.019%), 3 pgs degraded, 5 pgs undersized services: mon: 3 daemons, quorum monb01,monb02,monb03 mgr: monb03(active), standbys: monb01, monb02 osd: 285 osds: 284 up, 284 in data: pools: 9 pools, 9546 pgs objects: 31.65 M objects, 120 TiB usage: 363 TiB used, 127 TiB / 490 TiB avail pgs: 10.696% pgs not active 18357/94939584 objects degraded (0.019%) 8520 active+clean 999 peering 18 activating 3 active+clean+scrubbing+deep 2 activating+undersized+degraded 2 activating+undersized 1 active+clean+scrubbing 1 active+undersized+degraded io: client: 367 MiB/s rd, 195 MiB/s wr, 24.51 kop/s rd, 5.95 kop/s wr cache: 24 MiB/s flush, 90 MiB/s evict, 23 op/s promote

2 2

Re: Seriously degraded performance after update to Octopus
by Martin Rasmus Lundquist Hansen 04 Nov '20

04 Nov '20

Thank you for the suggestion. It does indeed seem to explain why the OSD nodes are no longer using the Buffers for caching. Unfortunately, changing the value bluefs_buffered_io does not seem to make any difference in performance. I will keep looking for clues.

1 0

How to reset Log Levels
by Ml Ml 04 Nov '20

04 Nov '20

Hello, i played around with some log level i can´t remember and my logs are now getting bigger than my DVD-Movie collection. E.g.: journalctl -b -u ceph-5436dd5d-83d4-4dc8-a93b-60ab5db145df(a)mon.ceph03.service > out.file is 1,1GB big. I did already try: ceph tell mon.ceph03 config set debug_mon 0/10 ceph tell mon.ceph03 config set debug_osd 0/10 ceph tell mon.ceph03 config set debug_mgr 0/10 ceph tell mon.ceph03 config set "mon_health_to_clog" false ceph tell mon.ceph03 config set "mon_health_log_update_period" 30 ceph tell mon.ceph03 config set "debug_mgr" "0/0" which made it better, but i really cant remember it all and would like to have the default values. Is there a way to reset those Log Values? Cheers, Michael

2 2

Ceph 14.2 - stuck peering.
by m.sliwinski＠lh.pl 04 Nov '20

04 Nov '20

Hi We have weird issue iwth our ceph cluster - almost all PGs assigned to one specific pool became stuck, locking out all operations without reporting any errors. Story: We have 3 different pools, hdd-backed, ssd-backed and nvme-backed. Pool ssh worked fine for few months. Today one of the hosts assigned to nvme pool restarted triggering recovery in that pool. It wnet fast and cluster went to OK state. During these events or shortly after them ssd pool became unresponsive. It was impossible to either read or write from/to it. We decided to slowly restart fist OSDs assigned to it, thenm as it didn't help - all the mons, wihout breaking quorum of course. At this moment both nvme and hdd polls are working fine, ssd one is stuck in recovery. All OSDs in that ssd pool use large amount of CPU and are exchanging approx 1Mpps per OSD server between each other. PGs seem to be slowly migrating from peering to activating but it's going very slowly - approx 10PGs during last hour. We were using 14.2.2 OSDs when issues happened, upgrade to 14.2.13 didn't help. We increased heartbeat grace, but it didn't change anything. It doesn't seem that there's a network problem as OSDs don't report problems with connecting to MONs or each other. Other OSDs - nvme, connected to that same set of switches work without issues. Can you help? Point me to what should i check or do? I looked on-line and on the group for causes of peering issues and checked most of them, nothing helped. I can't use 'ceph pg 28.1cc query' as it hangs, even for PGs that are marked as active+clean in the results of 'ceph pg dump' I checked status of the one of stuck PGs via ceph-objectstore-tool --data-path [...] --op info --pgid 28.29d for all three copies and got: { "pgid": "28.29d", "last_update": "68160'205094", "last_complete": "68160'205094", "log_tail": "68062'202000", "last_user_version": 205094, "last_backfill": "MAX", "last_backfill_bitwise": 0, "purged_snaps": [ { "start": "1", "length": "3" } ], "history": { "epoch_created": 67698, "epoch_pool_created": 67698, "last_epoch_started": 68871, "last_interval_started": 68851, "last_epoch_clean": 67746, "last_interval_clean": 67745, "last_epoch_split": 0, "last_epoch_marked_full": 0, "same_up_since": 69447, "same_interval_since": 69447, "same_primary_since": 69411, "last_scrub": "68062'199623", "last_scrub_stamp": "2020-11-03 03:32:46.895988", "last_deep_scrub": "68062'177321", "last_deep_scrub_stamp": "2020-11-02 01:07:15.963916", "last_clean_scrub_stamp": "2020-11-03 03:32:46.895988" }, "stats": { "version": "68160'205094", "reported_seq": "378496", "reported_epoch": "69447", "state": "peering", "last_fresh": "2020-11-03 20:55:39.247348", "last_change": "2020-11-03 20:55:39.247348", "last_active": "2020-11-03 15:26:24.270088", "last_peered": "2020-11-03 19:04:43.152655", "last_clean": "2020-11-03 14:45:02.988293", "last_became_active": "2020-09-01 13:52:40.091759", "last_became_peered": "2020-11-03 19:04:42.939991", "last_unstale": "2020-11-03 20:55:39.247348", "last_undegraded": "2020-11-03 20:55:39.247348", "last_fullsized": "2020-11-03 20:55:39.247348", "mapping_epoch": 69447, "log_start": "68062'202000", "ondisk_log_start": "68062'202000", "created": 67698, "last_epoch_clean": 67746, "parent": "0.0", "parent_split_bits": 0, "last_scrub": "68062'199623", "last_scrub_stamp": "2020-11-03 03:32:46.895988", "last_deep_scrub": "68062'177321", "last_deep_scrub_stamp": "2020-11-02 01:07:15.963916", "last_clean_scrub_stamp": "2020-11-03 03:32:46.895988", "log_size": 3094, "ondisk_log_size": 3094, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 15173849600, "num_objects": 3647, "num_object_clones": 0, "num_object_copies": 10941, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 3647, "num_whiteouts": 0, "num_read": 172836, "num_read_kb": 6824184, "num_write": 196190, "num_write_kb": 21380176, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 261, 284, 271 ], "acting": [ 261, 284, 271 ], "avail_no_missing": [], "object_location_counts": [], "blocked_by": [ 271, 284 ], "up_primary": 261, "acting_primary": 261, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 69422, "hit_set_history": { "current_last_update": "0'0", "history": [] } } { "pgid": "28.29d", "last_update": "68160'205094", "last_complete": "68160'205094", "log_tail": "68062'202000", "last_user_version": 205094, "last_backfill": "MAX", "last_backfill_bitwise": 0, "purged_snaps": [ { "start": "1", "length": "3" } ], "history": { "epoch_created": 67698, "epoch_pool_created": 67698, "last_epoch_started": 68871, "last_interval_started": 68851, "last_epoch_clean": 67746, "last_interval_clean": 67745, "last_epoch_split": 0, "last_epoch_marked_full": 0, "same_up_since": 69630, "same_interval_since": 69630, "same_primary_since": 69628, "last_scrub": "68062'199623", "last_scrub_stamp": "2020-11-03 03:32:46.895988", "last_deep_scrub": "68062'177321", "last_deep_scrub_stamp": "2020-11-02 01:07:15.963916", "last_clean_scrub_stamp": "2020-11-03 03:32:46.895988" }, "stats": { "version": "68160'205094", "reported_seq": "378445", "reported_epoch": "69627", "state": "peering", "last_fresh": "2020-11-03 21:15:08.819278", "last_change": "2020-11-03 21:14:18.360957", "last_active": "2020-11-03 15:26:24.270088", "last_peered": "2020-11-03 19:04:43.152655", "last_clean": "2020-11-03 14:45:02.988293", "last_became_active": "2020-09-01 13:52:40.091759", "last_became_peered": "2020-11-03 19:04:42.939991", "last_unstale": "2020-11-03 21:15:08.819278", "last_undegraded": "2020-11-03 21:15:08.819278", "last_fullsized": "2020-11-03 21:15:08.819278", "mapping_epoch": 69630, "log_start": "68062'202000", "ondisk_log_start": "68062'202000", "created": 67698, "last_epoch_clean": 67746, "parent": "0.0", "parent_split_bits": 0, "last_scrub": "68062'199623", "last_scrub_stamp": "2020-11-03 03:32:46.895988", "last_deep_scrub": "68062'177321", "last_deep_scrub_stamp": "2020-11-02 01:07:15.963916", "last_clean_scrub_stamp": "2020-11-03 03:32:46.895988", "log_size": 3094, "ondisk_log_size": 3094, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 15173849600, "num_objects": 3647, "num_object_clones": 0, "num_object_copies": 10941, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 3647, "num_whiteouts": 0, "num_read": 172836, "num_read_kb": 6824184, "num_write": 196190, "num_write_kb": 21380176, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 261, 284 ], "acting": [ 261, 284 ], "avail_no_missing": [], "object_location_counts": [], "blocked_by": [ 271 ], "up_primary": 261, "acting_primary": 261, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 69392, "hit_set_history": { "current_last_update": "0'0", "history": [] } } { "pgid": "28.29d", "last_update": "68160'205094", "last_complete": "68160'205094", "log_tail": "68062'202000", "last_user_version": 205094, "last_backfill": "MAX", "last_backfill_bitwise": 0, "purged_snaps": [ { "start": "1", "length": "3" } ], "history": { "epoch_created": 67698, "epoch_pool_created": 67698, "last_epoch_started": 68871, "last_interval_started": 68851, "last_epoch_clean": 67746, "last_interval_clean": 67745, "last_epoch_split": 0, "last_epoch_marked_full": 0, "same_up_since": 69411, "same_interval_since": 69411, "same_primary_since": 69411, "last_scrub": "68062'199623", "last_scrub_stamp": "2020-11-03 03:32:46.895988", "last_deep_scrub": "68062'177321", "last_deep_scrub_stamp": "2020-11-02 01:07:15.963916", "last_clean_scrub_stamp": "2020-11-03 03:32:46.895988" }, "stats": { "version": "68070'205093", "reported_seq": "378344", "reported_epoch": "68160", "state": "active+clean", "last_fresh": "2020-11-03 14:45:02.988293", "last_change": "2020-11-03 03:32:46.896044", "last_active": "2020-11-03 14:45:02.988293", "last_peered": "2020-11-03 14:45:02.988293", "last_clean": "2020-11-03 14:45:02.988293", "last_became_active": "2020-09-01 13:52:40.091759", "last_became_peered": "2020-09-01 13:52:40.091759", "last_unstale": "2020-11-03 14:45:02.988293", "last_undegraded": "2020-11-03 14:45:02.988293", "last_fullsized": "2020-11-03 14:45:02.988293", "mapping_epoch": 69411, "log_start": "68062'202000", "ondisk_log_start": "68062'202000", "created": 67698, "last_epoch_clean": 67746, "parent": "0.0", "parent_split_bits": 0, "last_scrub": "68062'199623", "last_scrub_stamp": "2020-11-03 03:32:46.895988", "last_deep_scrub": "68062'177321", "last_deep_scrub_stamp": "2020-11-02 01:07:15.963916", "last_clean_scrub_stamp": "2020-11-03 03:32:46.895988", "log_size": 3093, "ondisk_log_size": 3093, "stats_invalid": false, "dirty_stats_invalid": false, "omap_stats_invalid": false, "hitset_stats_invalid": false, "hitset_bytes_stats_invalid": false, "pin_stats_invalid": false, "manifest_stats_invalid": false, "snaptrimq_len": 0, "stat_sum": { "num_bytes": 15173849600, "num_objects": 3647, "num_object_clones": 0, "num_object_copies": 10941, "num_objects_missing_on_primary": 0, "num_objects_missing": 0, "num_objects_degraded": 0, "num_objects_misplaced": 0, "num_objects_unfound": 0, "num_objects_dirty": 3647, "num_whiteouts": 0, "num_read": 172836, "num_read_kb": 6824184, "num_write": 196190, "num_write_kb": 21380176, "num_scrub_errors": 0, "num_shallow_scrub_errors": 0, "num_deep_scrub_errors": 0, "num_objects_recovered": 0, "num_bytes_recovered": 0, "num_keys_recovered": 0, "num_objects_omap": 0, "num_objects_hit_set_archive": 0, "num_bytes_hit_set_archive": 0, "num_flush": 0, "num_flush_kb": 0, "num_evict": 0, "num_evict_kb": 0, "num_promote": 0, "num_flush_mode_high": 0, "num_flush_mode_low": 0, "num_evict_mode_some": 0, "num_evict_mode_full": 0, "num_objects_pinned": 0, "num_legacy_snapsets": 0, "num_large_omap_objects": 0, "num_objects_manifest": 0, "num_omap_bytes": 0, "num_omap_keys": 0, "num_objects_repaired": 0 }, "up": [ 261, 284, 271 ], "acting": [ 261, 284, 271 ], "avail_no_missing": [], "object_location_counts": [], "blocked_by": [], "up_primary": 261, "acting_primary": 261, "purged_snaps": [] }, "empty": 0, "dne": 0, "incomplete": 0, "last_epoch_started": 67746, "hit_set_history": { "current_last_update": "0'0", "history": [] } } Current status of the cluster: Reduced data availability: 1021 pgs inactive, 999 pgs peering Degraded data redundancy: 18357/94939584 objects degraded (0.019%), 3 pgs degraded, 5 pgs undersized services: mon: 3 daemons, quorum monb01,monb02,monb03 mgr: monb03(active), standbys: monb01, monb02 osd: 285 osds: 284 up, 284 in data: pools: 9 pools, 9546 pgs objects: 31.65 M objects, 120 TiB usage: 363 TiB used, 127 TiB / 490 TiB avail pgs: 10.696% pgs not active 18357/94939584 objects degraded (0.019%) 8520 active+clean 999 peering 18 activating 3 active+clean+scrubbing+deep 2 activating+undersized+degraded 2 activating+undersized 1 active+clean+scrubbing 1 active+undersized+degraded io: client: 367 MiB/s rd, 195 MiB/s wr, 24.51 kop/s rd, 5.95 kop/s wr cache: 24 MiB/s flush, 90 MiB/s evict, 23 op/s promote

1 0

Inconsistent Space Usage reporting
by Vikas Rana 03 Nov '20

03 Nov '20

Hi Friends, We have some inconsistent storage space usage reporting. We used only 46TB with single copy but the space used on the pool is close to 128TB. Any idea where's the extra space is utilized and how to reclaim it? Ceph Version : 12.2.11 with XFS OSDs. We are planning to upgrade soon. # ceph df detail GLOBAL: SIZE AVAIL RAW USED %RAW USED OBJECTS 363TiB 131TiB 231TiB 63.83 43.80M POOLS: NAME ID QUOTA OBJECTS QUOTA BYTES USED %USED MAX AVAIL OBJECTS DIRTY READ WRITE RAW USED fcp 15 N/A N/A 23.6TiB 42.69 31.7TiB 3053801 3.05M 6.10GiB 12.6GiB 47.3TiB nfs 16 N/A N/A 128TiB 66.91 63.4TiB 33916181 33.92M 3.93GiB 4.73GiB 128TiB # df -h Filesystem Size Used Avail Use% Mounted on /dev/nbd0 200T 46T 155T 23% /vol/dir_research #ceph osd pool get nfs all size: 1 min_size: 1 crash_replay_interval: 0 pg_num: 128 pgp_num: 128 crush_rule: replicated_ruleset hashpspool: true nodelete: false nopgchange: false nosizechange: false write_fadvise_dontneed: false noscrub: false nodeep-scrub: false use_gmt_hitset: 1 auid: 0 fast_read: 0 Appreciate your help. Thanks, -Vikas

2 3

Restart Error: osd.47 already exists in network host
by Ml Ml 03 Nov '20

03 Nov '20

Hello List, sometimes some OSD get taken our for some reason ( i am still looking for the reason, and i guess its due to some overload), however, when i try to restart them i get: Nov 02 08:05:26 ceph05 bash[9811]: Error: No such container: ceph-5436dd5d-83d4-4dc8-a93b-60ab5db145df-osd.47 Nov 02 08:05:29 ceph05 bash[9811]: /usr/bin/docker: Error response from daemon: endpoint with name ceph-5436dd5d-83d4-4dc8-a93b-60ab5db145df-osd.47 already exists in network host. Nov 02 08:05:29 ceph05 systemd[1]: ceph-5436dd5d-83d4-4dc8-a93b-60ab5db145df(a)osd.47.service: Main process exited, code=exited, status=125/n/a Nov 02 08:05:34 ceph05 systemd[1]: ceph-5436dd5d-83d4-4dc8-a93b-60ab5db145df(a)osd.47.service: Failed with result 'exit-code'. Nov 02 08:05:44 ceph05 systemd[1]: ceph-5436dd5d-83d4-4dc8-a93b-60ab5db145df(a)osd.47.service: Service RestartSec=10s expired, scheduling restart. Nov 02 08:05:44 ceph05 systemd[1]: ceph-5436dd5d-83d4-4dc8-a93b-60ab5db145df(a)osd.47.service: Scheduled restart job, restart counter is at 5. Nov 02 08:05:44 ceph05 systemd[1]: Stopped Ceph osd.47 for 5436dd5d-83d4-4dc8-a93b-60ab5db145df. Nov 02 08:05:44 ceph05 systemd[1]: ceph-5436dd5d-83d4-4dc8-a93b-60ab5db145df(a)osd.47.service: Start request repeated too quickly. Nov 02 08:05:44 ceph05 systemd[1]: ceph-5436dd5d-83d4-4dc8-a93b-60ab5db145df(a)osd.47.service: Failed with result 'exit-code'. Nov 02 08:05:44 ceph05 systemd[1]: Failed to start Ceph osd.47 for 5436dd5d-83d4-4dc8-a93b-60ab5db145df. I need to reboot the full host to get the OSD back in again. As far i can see this is some docker problem? root@ceph05:~# docker ps | grep osd.47 => no hit root@ceph05:~# docker network prune => does not solve the problem Any hint on that? Thanks, Michael

2 3

Re: How to recover from active+clean+inconsistent+failed_repair? [SOLVED]
by Sagara Wijetunga 03 Nov '20

03 Nov '20

Hi Frank Found the issue and fixed. It was a one copy of 0 byte object. Removed it. Deep scrub the PG fixed the issue. # find /var/lib/ceph/osd/ -type f -name "1000023675e*" /var/lib/ceph/osd/ceph-2/current/3.b_head/DIR_B/DIR_A/DIR_E/1000023675e.00000000__head_AE97EEAB__3 # ls -l /var/lib/ceph/osd/ceph-2/current/3.b_head/DIR_B/DIR_A/DIR_E/1000023675e.00000000__head_AE97EEAB__3-rw-r--r-- 1 ceph ceph 0 Oct 31 19:18 /var/lib/ceph/osd/ceph-2/current/3.b_head/DIR_B/DIR_A/DIR_E/1000023675e.00000000__head_AE97EEAB__3 Once again, many thanks for your help. Best regards Sagara

2 1

Updating client caps online
by Dan van der Ster 03 Nov '20

03 Nov '20

Hi all, We still have legacy caps on our nautilus rbd cluster. I just wanted to check if this is totally safe (and to post here ftr because I don't think this has ever been documented) Here are the current caps: [client.images] key = xxx caps mgr = "allow r" caps mon = "allow r, allow command \"osd blacklist\"" caps osd = "allow class-read object_prefix rbd_children, allow rwx pool=images" [client.volumes] key = xxx caps mgr = "allow r" caps mon = "allow r, allow command \"osd blacklist\"" caps osd = "allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rx pool=images, allow rwx pool=cinder-critical" Now that we upgraded to nautilus we would do: # ceph auth caps client.images mon 'profile rbd' osd 'profile rbd pool=images' mgr 'profile rbd pool=images' # ceph auth caps client.volumes mon 'profile rbd' osd 'profile rbd pool=volumes, profile rbd-read-only pool=images, profile rbd pool=cinder-critical' mgr 'profile rbd pool=volumes, profile rbd pool=cinder-critical' Does that look correct? Does this apply without impacting any client IOs ? Thanks! Dan

2 2

Does it make sense to have separate HDD based DB/WAL partition
by huxiaoyu＠horebdata.cn 03 Nov '20

03 Nov '20

Dear Ceph folks, I am faced with a special use Ceph scenario, where no SSD is available. Three node Ceph cluster, each with 14 HDDs, no SSD. In this case, should i create a separate DB/WAL partition for each OSD on a HDD? and if yes, how big the partition? Which is the best in terms of performance, with a separate DB/WAL partition, or without? Thanks a lot, samuel huxiaoyu(a)horebdata.cn

2 1

How to recover from active+clean+inconsistent+failed_repair?
by Sagara Wijetunga 03 Nov '20

03 Nov '20

Hi all I have a Ceph cluster (Nautilus 14.2.11) with 3 Ceph nodes. A crash happened and all 3 Ceph nodes went down. One (1) PG turned "active+clean+inconsistent", I tried to repair it. After the repair, now shows "active+clean+inconsistent+failed_repair" for the PG in the question and cannot bring the cluster to "active+clean". How do I rescue the cluster? Is this a false positive? Here are the detail: All three Ceph nodes run ceph-mon, ceph-mgr, ceph-osd and ceph-mds. 1. ceph -s health: HEALTH_ERR 3 scrub errors Possible data damage: 1 pg inconsistent pgs: 191 active+clean 1 active+clean+inconsistent 2. ceph health detailHEALTH_ERR 3 scrub errors; Possible data damage: 1 pg inconsistentOSD_SCRUB_ERRORS 3 scrub errorsPG_DAMAGED Possible data damage: 1 pg inconsistent pg 3.b is active+clean+inconsistent, acting [0,1,2] 3. rados list-inconsistent-pg rbd[] 4. ceph pg deep-scrub 3.b 5. ceph pg repair 3.b 6. ceph health detailHEALTH_ERR 3 scrub errors; Possible data damage: 1 pg inconsistentOSD_SCRUB_ERRORS 3 scrub errorsPG_DAMAGED Possible data damage: 1 pg inconsistent pg 3.b is active+clean+inconsistent+failed_repair, acting [0,1,2] 7. rados list-inconsistent-obj 3.b --format=json-pretty{ "epoch": 4769, "inconsistents": []} 8. ceph pg 3.b list_unfound { "num_missing": 0, "num_unfound": 0, "objects": [], "more": false} Appreciate your help. ThanksSagara

2 17

Monitor persistently out-of-quorum
by Ki Wong 02 Nov '20

02 Nov '20

Hello, I am at my wit's end. So I made a mistake in the configuration of my router and one of the monitors (out of 3) dropped out of the quorum and nothing I’ve done allow it to rejoin. That includes reinstalling the monitor with ceph-ansible. The connectivity issue is fixed. I’ve tested it using “nc” and the host can connect to both port 3300 and 6789 of the other monitors. But the wayward monitor continue to stay out of quorum. What is wrong? I see a bunch of “EBUSY” errors in the log, with the message: e1 handle_auth_request haven't formed initial quorum, EBUSY How do I fix this? Any help would be greatly appreciated. Many thanks, -kc With debug_mon at 1/10, I got these log snippets: 2020-10-28 15:40:05.961 7fb79253a700 4 mon.mgmt03@2(probing) e1 probe_timeout 0x564050353ec0 2020-10-28 15:40:05.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 bootstrap 2020-10-28 15:40:05.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 sync_reset_requester 2020-10-28 15:40:05.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 unregister_cluster_logger - not registered 2020-10-28 15:40:05.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 cancel_probe_timeout (none scheduled) 2020-10-28 15:40:05.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 monmap e1: 3 mons at {mgmt01=[v2:10.0.1.1:3300/0,v1:10.0.1.1:6789/0],mgmt02=[v2:10.1.1.1:3300/0,v1:10.1.1.1:6789/0],mgmt03=[v2:10.2.1.1:3300/0,v1:10.2.1.1:6789/0]} 2020-10-28 15:40:05.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 _reset 2020-10-28 15:40:05.961 7fb79253a700 10 mon.mgmt03(a)2(probing).auth v0 _set_mon_num_rank num 0 rank 0 2020-10-28 15:40:05.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 cancel_probe_timeout (none scheduled) 2020-10-28 15:40:05.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 timecheck_finish 2020-10-28 15:40:05.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 scrub_event_cancel 2020-10-28 15:40:05.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 scrub_reset 2020-10-28 15:40:05.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 cancel_probe_timeout (none scheduled) 2020-10-28 15:40:05.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 reset_probe_timeout 0x564050347ce0 after 2 seconds 2020-10-28 15:40:05.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 probing other monitors 2020-10-28 15:40:07.961 7fb79253a700 4 mon.mgmt03@2(probing) e1 probe_timeout 0x564050347ce0 2020-10-28 15:40:07.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 bootstrap 2020-10-28 15:40:07.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 sync_reset_requester 2020-10-28 15:40:07.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 unregister_cluster_logger - not registered 2020-10-28 15:40:07.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 cancel_probe_timeout (none scheduled) 2020-10-28 15:40:07.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 monmap e1: 3 mons at {mgmt01=[v2:10.0.1.1:3300/0,v1:10.0.1.1:6789/0],mgmt02=[v2:10.1.1.1:3300/0,v1:10.1.1.1:6789/0],mgmt03=[v2:10.2.1.1:3300/0,v1:10.2.1.1:6789/0]} 2020-10-28 15:40:07.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 _reset 2020-10-28 15:40:07.961 7fb79253a700 10 mon.mgmt03(a)2(probing).auth v0 _set_mon_num_rank num 0 rank 0 2020-10-28 15:40:07.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 cancel_probe_timeout (none scheduled) 2020-10-28 15:40:07.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 timecheck_finish 2020-10-28 15:40:07.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 scrub_event_cancel 2020-10-28 15:40:07.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 scrub_reset 2020-10-28 15:40:07.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 cancel_probe_timeout (none scheduled) 2020-10-28 15:40:07.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 reset_probe_timeout 0x564050360660 after 2 seconds 2020-10-28 15:40:07.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 probing other monitors 2020-10-28 15:40:09.107 7fb79253a700 -1 mon.mgmt03@2(probing) e1 get_health_metrics reporting 7 slow ops, oldest is log(1 entries from seq 1 at 2020-10-27 23:03:41.586915) 2020-10-28 15:40:09.961 7fb79253a700 4 mon.mgmt03@2(probing) e1 probe_timeout 0x564050360660 2020-10-28 15:40:09.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 bootstrap 2020-10-28 15:40:09.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 sync_reset_requester 2020-10-28 15:40:09.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 unregister_cluster_logger - not registered 2020-10-28 15:40:09.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 cancel_probe_timeout (none scheduled) 2020-10-28 15:40:09.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 monmap e1: 3 mons at {mgmt01=[v2:10.0.1.1:3300/0,v1:10.0.1.1:6789/0],mgmt02=[v2:10.1.1.1:3300/0,v1:10.1.1.1:6789/0],mgmt03=[v2:10.2.1.1:3300/0,v1:10.2.1.1:6789/0]} 2020-10-28 15:40:09.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 _reset 2020-10-28 15:40:09.961 7fb79253a700 10 mon.mgmt03(a)2(probing).auth v0 _set_mon_num_rank num 0 rank 0 2020-10-28 15:40:09.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 cancel_probe_timeout (none scheduled) 2020-10-28 15:40:09.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 timecheck_finish 2020-10-28 15:40:09.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 scrub_event_cancel 2020-10-28 15:40:09.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 scrub_reset 2020-10-28 15:40:09.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 cancel_probe_timeout (none scheduled) 2020-10-28 15:40:09.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 reset_probe_timeout 0x5640503606c0 after 2 seconds 2020-10-28 15:40:09.961 7fb79253a700 10 mon.mgmt03@2(probing) e1 probing other monitors 2020-10-28 15:40:11.962 7fb79253a700 4 mon.mgmt03@2(probing) e1 probe_timeout 0x5640503606c0 2020-10-28 15:40:11.962 7fb79253a700 10 mon.mgmt03@2(probing) e1 bootstrap 2020-10-28 15:40:11.962 7fb79253a700 10 mon.mgmt03@2(probing) e1 sync_reset_requester 2020-10-28 15:40:11.962 7fb79253a700 10 mon.mgmt03@2(probing) e1 unregister_cluster_logger - not registered 2020-10-28 15:40:11.962 7fb79253a700 10 mon.mgmt03@2(probing) e1 cancel_probe_timeout (none scheduled) 2020-10-28 15:40:11.962 7fb79253a700 10 mon.mgmt03@2(probing) e1 monmap e1: 3 mons at {mgmt01=[v2:10.0.1.1:3300/0,v1:10.0.1.1:6789/0],mgmt02=[v2:10.1.1.1:3300/0,v1:10.1.1.1:6789/0],mgmt03=[v2:10.2.1.1:3300/0,v1:10.2.1.1:6789/0]} 2020-10-28 15:40:11.962 7fb79253a700 10 mon.mgmt03@2(probing) e1 _reset 2020-10-28 15:40:11.962 7fb79253a700 10 mon.mgmt03(a)2(probing).auth v0 _set_mon_num_rank num 0 rank 0 2020-10-28 15:40:11.962 7fb79253a700 10 mon.mgmt03@2(probing) e1 cancel_probe_timeout (none scheduled) 2020-10-28 15:40:11.962 7fb79253a700 10 mon.mgmt03@2(probing) e1 timecheck_finish 2020-10-28 15:40:11.962 7fb79253a700 10 mon.mgmt03@2(probing) e1 scrub_event_cancel 2020-10-28 15:40:11.962 7fb79253a700 10 mon.mgmt03@2(probing) e1 scrub_reset 2020-10-28 15:40:11.962 7fb79253a700 10 mon.mgmt03@2(probing) e1 cancel_probe_timeout (none scheduled) 2020-10-28 15:40:11.962 7fb79253a700 10 mon.mgmt03@2(probing) e1 reset_probe_timeout 0x564050360900 after 2 seconds 2020-10-28 15:40:11.962 7fb79253a700 10 mon.mgmt03@2(probing) e1 probing other monitors 2020-10-28 15:40:12.354 7fb79453e700 10 mon.mgmt03@2(probing) e1 handle_auth_request con 0x56404cf25400 (start) method 2 payload 32 2020-10-28 15:40:12.354 7fb79453e700 10 mon.mgmt03@2(probing) e1 handle_auth_request haven't formed initial quorum, EBUSY 2020-10-28 15:40:12.354 7fb78fd35700 10 mon.mgmt03@2(probing) e1 ms_handle_reset 0x56404cf25400 - … 2020-10-28 15:40:59.968 7fb79253a700 10 mon.mgmt03@2(probing) e1 probing other monitors 2020-10-28 15:41:00.110 7fb79453e700 10 mon.mgmt03@2(probing) e1 handle_auth_request con 0x56404cae1000 (start) method 2 payload 22 2020-10-28 15:41:00.110 7fb79453e700 10 mon.mgmt03@2(probing) e1 handle_auth_request haven't formed initial quorum, EBUSY 2020-10-28 15:41:00.110 7fb78fd35700 10 mon.mgmt03@2(probing) e1 ms_handle_reset 0x56404cae1000 - 2020-10-28 15:41:00.110 7fb793d3d700 10 mon.mgmt03@2(probing) e1 handle_auth_request con 0x56404c8d4c00 (start) method 2 payload 22 2020-10-28 15:41:00.110 7fb793d3d700 10 mon.mgmt03@2(probing) e1 handle_auth_request haven't formed initial quorum, EBUSY 2020-10-28 15:41:00.110 7fb78fd35700 10 mon.mgmt03@2(probing) e1 ms_handle_reset 0x56404c8d4c00 - 2020-10-28 15:41:00.117 7fb79453e700 10 mon.mgmt03@2(probing) e1 handle_auth_request con 0x56404c630800 (start) method 2 payload 22 2020-10-28 15:41:00.117 7fb79453e700 10 mon.mgmt03@2(probing) e1 handle_auth_request haven't formed initial quorum, EBUSY 2020-10-28 15:41:00.117 7fb78fd35700 10 mon.mgmt03@2(probing) e1 ms_handle_reset 0x56404c630800 - … 2020-10-28 15:42:42.379 7fb78d530700 4 rocksdb: [db/db_impl.cc:777] ------- DUMPING STATS ------- 2020-10-28 15:42:42.379 7fb78d530700 4 rocksdb: [db/db_impl.cc:778] ** DB Stats ** Uptime(secs): 60000.0 total, 600.0 interval Cumulative writes: 6 writes, 7 keys, 6 commit groups, 0.9 writes per commit group, ingest: 0.00 GB, 0.00 MB/s Cumulative WAL: 6 writes, 6 syncs, 0.86 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s Interval stall: 00:00:0.000 H:M:S, 0.0 percent ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 2/0 3.02 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.7 0.00 0.00 1 0.001 0 0 Sum 2/0 3.02 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.7 0.00 0.00 1 0.001 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.7 0.00 0.00 1 0.001 0 0 Uptime(secs): 60000.0 total, 600.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count … 2020-10-28 17:17:11.781 7f5f694821c0 0 using public_addr v2:10.2.1.1:0/0 -> [v2:10.2.1.1:3300/0,v1:10.2.1.1:6789/0] 2020-10-28 17:17:11.781 7f5f694821c0 0 starting mon.mgmt03 rank -1 at public addrs [v2:10.2.1.1:3300/0,v1:10.2.1.1:6789/0] at bind addrs [v2:10.2.1.1:3300/0,v1:10.2.1.1:6789/0] mon_data /var/lib/ceph/mon/ceph-mgmt03 fsid 374aed9e-5fc1-47e1-8d29-4416f7425e76 2020-10-28 17:17:11.783 7f5f694821c0 1 mon.mgmt03@-1(???) e2 preinit fsid 374aed9e-5fc1-47e1-8d29-4416f7425e76 2020-10-28 17:17:11.783 7f5f694821c0 1 mon.mgmt03@-1(???) e2 initial_members mgmt01,mgmt02,mgmt03, filtering seed monmap 2020-10-28 17:17:11.783 7f5f694821c0 1 mon.mgmt03@-1(???) e2 preinit clean up potentially inconsistent store state 2020-10-28 17:17:11.785 7f5f694821c0 0 -- [v2:10.2.1.1:3300/0,v1:10.2.1.1:6789/0] send_to message mon_probe(probe 374aed9e-5fc1-47e1-8d29-4416f7425e76 name mgmt03 new mon_release 14) v7 with empty dest 2020-10-28 17:17:13.191 7f5f5170d700 0 mon.mgmt03@-1(probing) e3 monmap addrs for rank 2 changed, i am [v2:10.2.1.1:3300/0,v1:10.2.1.1:6789/0], monmap is v2:10.2.1.1:3300/0, respawning 2020-10-28 17:17:13.191 7f5f5170d700 -1 mon.mgmt03@-1(probing) e3 stashing newest monmap 3 for next startup 2020-10-28 17:17:13.191 7f5f5170d700 0 mon.mgmt03@-1(probing) e3 respawn 2020-10-28 17:17:13.191 7f5f5170d700 1 mon.mgmt03@-1(probing) e3 e: '/usr/bin/ceph-mon' 2020-10-28 17:17:13.191 7f5f5170d700 1 mon.mgmt03@-1(probing) e3 0: '/usr/bin/ceph-mon' 2020-10-28 17:17:13.191 7f5f5170d700 1 mon.mgmt03@-1(probing) e3 1: '-f' 2020-10-28 17:17:13.191 7f5f5170d700 1 mon.mgmt03@-1(probing) e3 2: '--cluster' 2020-10-28 17:17:13.191 7f5f5170d700 1 mon.mgmt03@-1(probing) e3 3: 'ceph' 2020-10-28 17:17:13.191 7f5f5170d700 1 mon.mgmt03@-1(probing) e3 4: '--id' 2020-10-28 17:17:13.191 7f5f5170d700 1 mon.mgmt03@-1(probing) e3 5: 'mgmt03' 2020-10-28 17:17:13.191 7f5f5170d700 1 mon.mgmt03@-1(probing) e3 6: '--setuser' 2020-10-28 17:17:13.191 7f5f5170d700 1 mon.mgmt03@-1(probing) e3 7: 'ceph' 2020-10-28 17:17:13.192 7f5f5170d700 1 mon.mgmt03@-1(probing) e3 8: '--setgroup' 2020-10-28 17:17:13.192 7f5f5170d700 1 mon.mgmt03@-1(probing) e3 9: 'ceph' 2020-10-28 17:17:13.192 7f5f5170d700 1 mon.mgmt03@-1(probing) e3 respawning with exe /usr/bin/ceph-mon 2020-10-28 17:17:13.192 7f5f5170d700 1 mon.mgmt03@-1(probing) e3 exe_path /proc/self/exe 2020-10-28 17:17:13.217 7eff1f7cd1c0 0 ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable), process ceph-mon, pid 24265 2020-10-28 17:17:13.217 7eff1f7cd1c0 0 pidfile_write: ignore empty --pid-file 2020-10-28 17:17:13.246 7eff1f7cd1c0 0 load: jerasure load: lrc load: isa 2020-10-28 17:17:13.246 7eff1f7cd1c0 0 set rocksdb option compression = kNoCompression 2020-10-28 17:17:13.246 7eff1f7cd1c0 0 set rocksdb option level_compaction_dynamic_level_bytes = true 2020-10-28 17:17:13.246 7eff1f7cd1c0 0 set rocksdb option write_buffer_size = 33554432 2020-10-28 17:17:13.246 7eff1f7cd1c0 0 set rocksdb option compression = kNoCompression 2020-10-28 17:17:13.246 7eff1f7cd1c0 0 set rocksdb option level_compaction_dynamic_level_bytes = true 2020-10-28 17:17:13.246 7eff1f7cd1c0 0 set rocksdb option write_buffer_size = 33554432 2020-10-28 17:17:13.246 7eff1f7cd1c0 1 rocksdb: do_open column families: [default] 2020-10-28 17:17:13.246 7eff1f7cd1c0 4 rocksdb: RocksDB version: 6.1.2 2020-10-28 17:17:13.246 7eff1f7cd1c0 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2020-10-28 17:17:13.246 7eff1f7cd1c0 4 rocksdb: Compile date Aug 13 2020 2020-10-28 17:17:13.246 7eff1f7cd1c0 4 rocksdb: DB SUMMARY 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: CURRENT file: CURRENT 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: IDENTITY file: IDENTITY 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: MANIFEST file: MANIFEST-009877 size: 194 Bytes 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-mgmt03/store.db dir, Total Num: 2, files: 009874.sst 009876.sst 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-mgmt03/store.db: 009878.log size: 451 ; 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.error_if_exists: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.create_if_missing: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.paranoid_checks: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.env: 0x55beb27a5780 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.info_log: 0x55beb3d8e300 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_file_opening_threads: 16 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.statistics: (nil) 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.use_fsync: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_log_file_size: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_manifest_file_size: 1073741824 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.log_file_time_to_roll: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.keep_log_file_num: 1000 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.recycle_log_file_num: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.allow_fallocate: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.allow_mmap_reads: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.allow_mmap_writes: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.use_direct_reads: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.create_missing_column_families: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.db_log_dir: 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-mgmt03/store.db 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.table_cache_numshardbits: 6 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_subcompactions: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_background_flushes: -1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.WAL_ttl_seconds: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.WAL_size_limit_MB: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.manifest_preallocation_size: 4194304 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.is_fd_close_on_exec: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.advise_random_on_open: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.db_write_buffer_size: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.write_buffer_manager: 0x55beb3d98b40 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.access_hint_on_compaction_start: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.use_adaptive_mutex: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.rate_limiter: (nil) 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.wal_recovery_mode: 2 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.enable_thread_tracking: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.enable_pipelined_write: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.write_thread_max_yield_usec: 100 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.row_cache: None 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.wal_filter: None 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.avoid_flush_during_recovery: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.allow_ingest_behind: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.preserve_deletes: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.two_write_queues: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.manual_wal_flush: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.atomic_flush: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_background_jobs: 2 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_background_compactions: -1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.delayed_write_rate : 16777216 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_total_wal_size: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.stats_dump_period_sec: 600 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.stats_persist_period_sec: 600 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.stats_history_buffer_size: 1048576 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_open_files: -1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.bytes_per_sync: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.wal_bytes_per_sync: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compaction_readahead_size: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Compression algorithms supported: 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: kZSTDNotFinalCompression supported: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: kZSTD supported: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: kXpressCompression supported: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: kLZ4HCCompression supported: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: kLZ4Compression supported: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: kBZip2Compression supported: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: kZlibCompression supported: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: kSnappyCompression supported: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Fast CRC32 supported: Supported on x86 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: [db/version_set.cc:3543] Recovering from manifest file: /var/lib/ceph/mon/ceph-mgmt03/store.db/MANIFEST-009877 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: [db/column_family.cc:477] --------------- Options for column family [default]: 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.merge_operator: 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compaction_filter: None 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compaction_filter_factory: None 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.memtable_factory: SkipListFactory 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.table_factory: BlockBasedTable 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55beb30daac0) cache_index_and_filter_blocks: 1 cache_index_and_filter_blocks_with_high_priority: 1 pin_l0_filter_and_index_blocks_in_cache: 0 pin_top_level_index_and_filter: 1 index_type: 0 data_block_index_type: 0 data_block_hash_table_util_ratio: 0.750000 hash_index_allow_collision: 1 checksum: 1 no_block_cache: 0 block_cache: 0x55beb30f7010 block_cache_name: BinnedLRUCache block_cache_options: capacity : 536870912 num_shard_bits : 4 strict_capacity_limit : 0 high_pri_pool_ratio: 0.000 block_cache_compressed: (nil) persistent_cache: (nil) block_size: 4096 block_size_deviation: 10 block_restart_interval: 16 index_block_restart_interval: 1 metadata_block_size: 4096 partition_filters: 0 use_delta_encoding: 1 filter_policy: rocksdb.BuiltinBloomFilter whole_key_filtering: 1 verify_compression: 0 read_amp_bytes_per_bit: 0 format_version: 2 enable_index_compression: 1 block_align: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.write_buffer_size: 33554432 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_write_buffer_number: 2 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compression: NoCompression 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.bottommost_compression: Disabled 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.prefix_extractor: nullptr 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.num_levels: 7 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compression_opts.window_bits: -14 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compression_opts.level: 32767 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compression_opts.strategy: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compression_opts.enabled: false 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.level0_stop_writes_trigger: 36 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.target_file_size_base: 67108864 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.target_file_size_multiplier: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_compaction_bytes: 1677721600 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.arena_block_size: 4194304 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.disable_auto_compactions: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.table_properties_collectors: 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.inplace_update_support: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.inplace_update_num_locks: 10000 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.memtable_whole_key_filtering: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.memtable_huge_page_size: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.bloom_locality: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.max_successive_merges: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.optimize_filters_for_hits: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.paranoid_file_checks: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.force_consistency_checks: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.report_bg_io_stats: 0 2020-10-28 17:17:13.247 7eff1f7cd1c0 4 rocksdb: Options.ttl: 0 2020-10-28 17:17:13.248 7eff1f7cd1c0 3 rocksdb: [db/version_set.cc:2581] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2020-10-28 17:17:13.248 7eff1f7cd1c0 4 rocksdb: [db/version_set.cc:3757] Recovered from manifest file:/var/lib/ceph/mon/ceph-mgmt03/store.db/MANIFEST-009877 succeeded,manifest_file_number is 9877, next_file_number is 9879, last_sequence is 2479, log_number is 9872,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2020-10-28 17:17:13.248 7eff1f7cd1c0 4 rocksdb: [db/version_set.cc:3766] Column family [default] (ID 0), log number is 9872 2020-10-28 17:17:13.248 7eff1f7cd1c0 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1603930633249084, "job": 1, "event": "recovery_started", "log_files": [9878]} 2020-10-28 17:17:13.248 7eff1f7cd1c0 4 rocksdb: [db/db_impl_open.cc:583] Recovering log #9878 mode 2 2020-10-28 17:17:13.249 7eff1f7cd1c0 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1603930633250074, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 9879, "file_size": 1336, "table_properties": {"data_size": 434, "index_size": 28, "filter_size": 69, "raw_key_size": 34, "raw_average_key_size": 34, "raw_value_size": 383, "raw_average_value_size": 383, "num_data_blocks": 1, "num_entries": 1, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}} 2020-10-28 17:17:13.249 7eff1f7cd1c0 4 rocksdb: [db/version_set.cc:3036] Creating manifest 9880 2020-10-28 17:17:13.249 7eff1f7cd1c0 3 rocksdb: [db/version_set.cc:2581] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2020-10-28 17:17:13.250 7eff1f7cd1c0 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1603930633251509, "job": 1, "event": "recovery_finished"} 2020-10-28 17:17:13.253 7eff1f7cd1c0 4 rocksdb: DB pointer 0x55beb3d26400 2020-10-28 17:17:13.253 7eff05253700 4 rocksdb: [db/db_impl.cc:777] ------- DUMPING STATS ------- 2020-10-28 17:17:13.253 7eff05253700 4 rocksdb: [db/db_impl.cc:778] ** DB Stats ** Uptime(secs): 0.0 total, 0.0 interval Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s Interval stall: 00:00:0.000 H:M:S, 0.0 percent ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 2/0 2.61 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.4 0.00 0.00 1 0.001 0 0 L6 1/0 2.39 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Sum 3/0 5.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.4 0.00 0.00 1 0.001 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.4 0.00 0.00 1 0.001 0 0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.4 0.00 0.00 1 0.001 0 0 Uptime(secs): 0.0 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 0.22 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 0.22 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [default] ** ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 2/0 2.61 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.4 0.00 0.00 1 0.001 0 0 L6 1/0 2.39 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Sum 3/0 5.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.4 0.00 0.00 1 0.001 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.4 0.00 0.00 1 0.001 0 0 Uptime(secs): 0.0 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 0.21 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [default] ** 2020-10-28 17:17:13.253 7eff1f7cd1c0 0 mon.mgmt03 does not exist in monmap, will attempt to join an existing cluster 2020-10-28 17:17:13.254 7eff1f7cd1c0 0 using public_addr v2:10.2.1.1:0/0 -> [v2:10.2.1.1:3300/0,v1:10.2.1.1:6789/0] 2020-10-28 17:17:13.254 7eff1f7cd1c0 0 starting mon.mgmt03 rank -1 at public addrs [v2:10.2.1.1:3300/0,v1:10.2.1.1:6789/0] at bind addrs [v2:10.2.1.1:3300/0,v1:10.2.1.1:6789/0] mon_data /var/lib/ceph/mon/ceph-mgmt03 fsid 374aed9e-5fc1-47e1-8d29-4416f7425e76 2020-10-28 17:17:13.256 7eff1f7cd1c0 1 mon.mgmt03@-1(???) e2 preinit fsid 374aed9e-5fc1-47e1-8d29-4416f7425e76 2020-10-28 17:17:13.256 7eff1f7cd1c0 1 mon.mgmt03@-1(???) e2 initial_members mgmt01,mgmt02,mgmt03, filtering seed monmap 2020-10-28 17:17:13.256 7eff1f7cd1c0 1 mon.mgmt03@-1(???) e2 preinit clean up potentially inconsistent store state 2020-10-28 17:17:13.258 7eff1f7cd1c0 0 -- [v2:10.2.1.1:3300/0,v1:10.2.1.1:6789/0] send_to message mon_probe(probe 374aed9e-5fc1-47e1-8d29-4416f7425e76 name mgmt03 new mon_release 14) v7 with empty dest

3 4

read latency
by Tony Liu 02 Nov '20

02 Nov '20

Hi, AWIK, the read latency primarily depends on HW latency, not much can be tuned in SW. Is that right? I ran a fio random read with iodepth 1 within a VM backed by Ceph with HDD OSD and here is what I got. ================= read: IOPS=282, BW=1130KiB/s (1157kB/s)(33.1MiB/30001msec) slat (usec): min=4, max=181, avg=14.04, stdev=10.16 clat (usec): min=178, max=393831, avg=3521.86, stdev=5771.35 lat (usec): min=188, max=393858, avg=3536.38, stdev=5771.51 ================= I checked HDD average latency is 2.9 ms. Looks like the test result makes perfect sense, isn't it? If I want to get shorter latency (more IOPS), I will have to go for better disk, eg. SSD. Right? Thanks! Tony

2 4

v14.2.13 Nautilus released
by Abhishek Lekshmanan 02 Nov '20

02 Nov '20

This is the 13th backport release in the Nautilus series. This release fixes a regression introduced in v14.2.12, and a few ceph-volume & RGW fixes. We recommend users to update to this release. Notable Changes --------------- * Fixed a regression that caused breakage in clusters that referred to ceph-mon hosts using dns names instead of ip addresses in the `mon_host` param in `ceph.conf` (issue#47951) * ceph-volume: the ``lvm batch`` subcommand received a major rewrite Changelog --------- * ceph-volume: major batch refactor (pr#37522, Jan Fajerski) * mgr/dashboard: Proper format iSCSI target portals (pr#37060, Volker Theile) * rpm: move python-enum34 into rhel 7 conditional (pr#37747, Nathan Cutler) * mon/MonMap: fix unconditional failure for init_with_hosts (pr#37816, Nathan Cutler, Patrick Donnelly) * rgw: allow rgw-orphan-list to note when rados objects are in namespace (pr#37799, J. Eric Ivancich) * rgw: fix setting of namespace in ordered and unordered bucket listing (pr#37798, J. Eric Ivancich) -- Abhishek

1 0

Re: Seriously degraded performance after update to Octopus
by Vladimir Prokofev 02 Nov '20

02 Nov '20

Just shooting in the dark here, but you may be affected by similar issue I had a while back, it was discussed here: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/ZOPBOY6XQOY… In short - they've changed setting bluefs_buffered_io to false in the recent Nautilus release. I guess the same was applied to newer releases. That lead to severe performance issues and similar symptoms, i.e. lower memory usage on OSD nodes. Worth checking out. Of course, it may be something completely different. You should look into monitoring all your OSDs separately, checking their utilization, await, and other parameters, at the same time comparing them to pre-upgrade values, to find the root cause. пн, 2 нояб. 2020 г. в 11:55, Marc Roos <M.Roos(a)f1-outsourcing.eu>: > > I am advocating already a long time for publishing testing data of some > basic test cluster against different ceph releases. Just a basic ceph > cluster that covers most configs and run the same tests, so you can > compare just ceph performance. That would mean a lot for smaller > companies that do not have access to a good test environment. I have > asked also about this at some ceph seminar. > > > > -----Original Message----- > From: Martin Rasmus Lundquist Hansen [mailto:hansen@imada.sdu.dk] > Sent: Monday, November 02, 2020 7:53 AM > To: ceph-users(a)ceph.io > Subject: [ceph-users] Seriously degraded performance after update to > Octopus > > Two weeks ago we updated our Ceph cluster from Nautilus (14.2.0) to > Octopus (15.2.5), an update that was long overdue. We used the Ansible > playbooks to perform a rolling update and except from a few minor > problems with the Ansible code, the update went well. The Ansible > playbooks were also used for setting up the cluster in the first place. > Before updating the Ceph software we also performed a full update of > CentOS and the Linux kernel (this part of the update had already been > tested on one of the OSD nodes the week before and we didn't notice any > problems). > > However, after the update we are seeing a serious decrease in > performance, more than a factor of 10x in some cases. I spend a week > trying to come up with an explantion or solution, but I am completely > blank. Independently of Ceph I tested the network performance and the > performance of the OSD disks, and I am not really seeing any problems > here. > > The specifications of the cluster is: > - 3x Monitor nodes running mgr+mon+mds (Intel(R) Xeon(R) Silver 4108 CPU > @ 1.80GHz, 16 cores, 196 GB RAM) > - 14x OSD nodes, each with 18 HDDs and 1 NVME (Intel(R) Xeon(R) Gold > 6126 CPU @ 2.60GHz, 24 cores, 384 GB RAM) > - CentOS 7.8 and Kernel 5.4.51 > - 100 Gbps Infiniband > > We are collecting various metrics using Prometheus, and on the OSD nodes > we are seeing some clear differences when it comes to CPU and Memory > usage. I collected some graphs here: http://mitsted.dk/ceph . After the > update the system load is highly reduced, there is almost no longer any > iowait for the CPU, and the free memory is no longer used for Buffers (I > can confirm that the changes in these metrics are not due to the update > of CentOS or the Linux kernel). All in all, now the OSD nodes are almost > completely idle all the time (and so are the monitors). On the linked > page I also attached two RADOS benchmarks. The first benchmark was > performed when the cluster was initially configured, and the second is > the same benchmark after the update to Octopus. When comparing these > two, it is clear that the performance has changed dramatically. For > example, in the write test the bandwidth is reduced from 320 MB/s to 21 > MB/s and the number of IOPS has also dropped significantly. > > I temporarily tried to disable the firewall and SELinux on all nodes to > see if it made any difference, but it didnt look like it (I did not > restart any services during this test, I am not sure if that could be > necessary). > > Any suggestions for finding the root cause of this performance decrease > would be greatly appreciated. > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

cephfs cannot write
by Patrick 02 Nov '20

02 Nov '20

Hi all, My ceph cluster is HEALTH_OK, but I cannot write on cephfs. OS: Ubuntu 20.04, ceph version 15.2.5, deploy with cephadm. root@RK01-OSD-A001:~# ceph -s   cluster:     id:     9091b472-1bdb-11eb-b217-abff3468259e     health: HEALTH_OK     services:     mon: 3 daemons, quorum RK01-OSD-A001,RK02-OSD-A002,RK03-OSD-A003 (age 18s)     mgr: RK01-OSD-A001.jwrjgj(active, since 51m), standbys: RK03-OSD-A003.tulrii     mds: cephfs:1 {0=cephfs.RK02-OSD-A002.lwpgaw=up:active} 1 up:standby     osd: 6 osds: 6 up (since 44m), 6 in (since 44m)     task status:     scrub status:         mds.cephfs.RK02-OSD-A002.lwpgaw: idle     data:     pools:   3 pools, 65 pgs     objects: 24 objects, 67 KiB     usage:   6.0 GiB used, 44 TiB / 44 TiB avail     pgs:     65 active+clean root@RK01-OSD-A001:~# ceph fs status cephfs - 1 clients ====== RANK  STATE               MDS                 ACTIVITY     DNS    INOS    0    active  cephfs.RK02-OSD-A002.lwpgaw  Reqs:    0 /s    13     15           POOL           TYPE     USED  AVAIL   cephfs.cephfs.meta  metadata  1152k  20.7T   cephfs.cephfs.data    data       0   20.7T           STANDBY MDS           cephfs.RK03-OSD-A003.xchwqj   MDS version: ceph version 15.2.5 (2c93eff00150f0cc5f106a559557a58d3d7b6f1f) octopus (stable) root@RK05-FRP-A001:~# df -h|grep "ceph-test" 172.16.65.1,172.16.65.2,172.16.65.3:6789:/   21T     0   21T   0% /ceph-test root@RK05-FRP-A001:~# echo 123 > /ceph-test/1.txt -bash: echo: write error: Operation not permitted root@RK05-FRP-A001:~# ls -l /ceph-test/1.txt -rw-r--r-- 1 root root 0 Nov  1 09:40 /ceph-test/1.txt root@RK05-FRP-A001:~# ls -ld /ceph-test/ drwxr-xr-x 2 root root 1 Nov  1 09:40 /ceph-test/ root@RK01-OSD-A001:~# cd /var/log/ceph/`ceph fsid` root@RK01-OSD-A001:/var/log/ceph/9091b472-1bdb-11eb-b217-abff3468259e# cat ceph-volume.log | grep err | grep sdx [2020-11-01 08:53:51,384][ceph_volume.process][INFO  ] stderr Failed to find physical volume "/dev/sdx". [2020-11-01 08:53:51,417][ceph_volume.process][INFO  ] stderr unable to read label for /dev/sdx: (2) No such file or directory [2020-11-01 08:53:51,445][ceph_volume.process][INFO  ] stderr unable to read label for /dev/sdx: (2) No such file or directory root@RK01-OSD-A001:~# pvs|grep sdx   /dev/sdx   ceph-41b09a52-e44b-43c5-ad86-0eada11b48b6 lvm2 a--  <7.28t    0  root@RK01-OSD-A001:~# lsblk|grep sdx sdx                                                                                                   65:112  0   7.3T  0 disk  root@RK01-OSD-A001:~# parted -s /dev/sdx print Error: /dev/sdx: unrecognised disk label Model: LSI MR9261-8i (scsi) Disk /dev/sdx: 8001GB Sector size (logical/physical): 512B/4096B Partition Table: unknown Disk Flags:  root@RK01-OSD-A001:~# 

2 1

RGW seems to not clean up after some requests
by Denis Krienbühl 02 Nov '20

02 Nov '20

Hi everyone We have faced some RGW outages recently, with the RGW returning HTTP 503. First for a few, then for most, then all requests - in the course of 1-2 hours. This seems to have started since we have updated from 15.2.4 to 15.2.5. The line that accompanies these outages in the log is the following: s3:list_bucket Scheduling request failed with -2218 It first pops up a few times here and there, until it eventually applies to all requests. It seems to indicate that the throttler has reached the limit of open connections. As we run a pair of HAProxy instances in front of RGW, which limit the number of connections to the two RGW instances to 400, this limit should never be reached. We do use RGW metadata sync between the instances, which could account for some extra connections, but if I look at open TCP connections between the instances I can count no more than 20 at any given time. I also noticed that some connections in the RGW log seem to never complete. That is, I can find a ‘starting new request’ line, but no associated ‘req done’ or ‘beast’ line. I don’t think there are any hung connections around, as they are killed by HAProxy after a short timeout. Looking at the code, it seems as if the throttler in use (SimpleThrottler), eventually reaches the maximum count of 1024 connections (outstanding_requests), and never recovers. I believe that the request_complete function is not called in all cases, but I am not familiar with the Ceph codebase, so I am not sure. See https://github.com/ceph/ceph/blob/cc17681b478594aa39dd80437256a54e388432f0/… <https://github.com/ceph/ceph/blob/cc17681b478594aa39dd80437256a54e388432f0/…> Does anyone see the same phenomenon? Could this be a bug in the request handling of RGW, or am I wrong in my assumptions? For now we’re just restarting our RGWs regularly, which seems to keep the problem at bay. Thanks for any hints. Denis

2 2

pgs stuck backfill_toofull
by Mark Johnson 02 Nov '20

02 Nov '20

I've been struggling with this one for a few days now. We had an OSD report as near full a few days ago. Had this happen a couple of times before and a reweight-by-utilization has sorted it out in the past. Tried the same again but this time we ended up with a couple of pgs in a state of backfill_toofull and a handful of misplaced objects as a result. Tried doing the reweight a few more times and it's been moving data around. We did have another osd trigger the near full alert but running the reweight a couple more times seems to have moved some of that data around a bit better. However, the original near_full osd doesn't seem to have changed much and the backfill_toofull pgs are still there. I'd keep doing the reweight-by-utilization but I'm not sure if I'm heading down the right path and if it will eventually sort it out. We have 14 pools, but the vast majority of data resides in just one of those pools (pool 20). The pgs in the backfill state are in pool 2 (as far as I can tell). That particular pool is used for some cephfs stuff and has a handful of large files in there (not sure if this is significant to the problem). All up, our utilization is showing as 55.13% but some of our OSDs are showing as 76% in use with this one problem sitting at 85.02%. Right now, I'm just not sure what the proper corrective action is. The last couple of reweights I've run have been a bit more targetted in that I've set it to only function on two OSDs at a time. If I run a test-reweight targetting only one osd, it does say it will reweight OSD 9 (the one at 85.02%). I gather this will move data away from this OSD and potentially get it below the threshold. However, at one point in the past couple of days, it's shown as no OSDs in a near full state, yet the two pgs in backfill_toofull didn't change. So, that's why I'm not sure continually reweighting is going to solve this issue. I'm a long way from knowledgable on Ceph so I'm not really sure what information is useful here. Here's a bit of info on what I'm seeing. Can provide anything else that might help. Basically, we have a three node cluster but only two have OSDs. The third is there simply to enable a quorum to be established. The OSDs are evenly spread across these two needs and the configuration of each is identical. We are running Jewel and are not in a position to upgrade at this stage. # ceph --version ceph version 10.2.11 (e4b061b47f07f583c92a050d9e84b1813a35671e) # ceph health detail HEALTH_WARN 2 pgs backfill_toofull; 2 pgs stuck unclean; recovery 33/62099566 objects misplaced (0.000%); 1 near full osd(s) pg 2.52 is stuck unclean for 201822.031280, current state active+remapped+backfill_toofull, last acting [17,3] pg 2.18 is stuck unclean for 202114.617682, current state active+remapped+backfill_toofull, last acting [18,2] pg 2.18 is active+remapped+backfill_toofull, acting [18,2] pg 2.52 is active+remapped+backfill_toofull, acting [17,3] recovery 33/62099566 objects misplaced (0.000%) osd.9 is near full at 85% # ceph osd df ID WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS 2 1.37790 1.00000 1410G 842G 496G 59.75 1.08 33 3 1.37790 0.45013 1410G 1079G 259G 76.49 1.39 21 4 1.37790 0.95001 1410G 1086G 253G 76.98 1.40 44 5 1.37790 1.00000 1410G 617G 722G 43.74 0.79 43 6 1.37790 0.65009 1410G 616G 722G 43.69 0.79 39 7 1.37790 0.95001 1410G 495G 844G 35.10 0.64 40 8 1.37790 1.00000 1410G 732G 606G 51.93 0.94 52 9 1.37790 0.70007 1410G 1199G 139G 85.02 1.54 37 10 1.37790 1.00000 1410G 611G 727G 43.35 0.79 41 11 1.37790 0.75006 1410G 495G 843G 35.11 0.64 32 0 1.37790 1.00000 1410G 731G 608G 51.82 0.94 43 12 1.37790 1.00000 1410G 851G 487G 60.36 1.09 44 13 1.37790 1.00000 1410G 378G 960G 26.82 0.49 38 14 1.37790 1.00000 1410G 969G 370G 68.68 1.25 37 15 1.37790 1.00000 1410G 724G 614G 51.35 0.93 35 16 1.37790 1.00000 1410G 491G 847G 34.84 0.63 43 17 1.37790 1.00000 1410G 862G 476G 61.16 1.11 50 18 1.37790 0.80005 1410G 1083G 255G 76.78 1.39 26 19 1.37790 0.65009 1410G 963G 375G 68.29 1.24 23 20 1.37790 1.00000 1410G 724G 614G 51.38 0.93 42 TOTAL 28219G 15557G 11227G 55.13 MIN/MAX VAR: 0.49/1.54 STDDEV: 15.57 # ceph pg ls backfill_toofull pg_stat objects mip degr misp unf bytes log disklog state state_stamp v reported up up_primary acting acting_primary last_scrub scrub_stamp last_deep_scrub deep_scrub_stamp 2.18 9 0 0 18 0 0 3653 3653 active+remapped+backfill_toofull 2020-10-29 05:31:20.429912 610'549153 656:390372 [9,12] 9 [18,2] 18 594'547482 2020-10-25 20:28:39.680744 594'543841 2020-10-21 21:21:33.092868 2.52 15 0 0 15 0 0 4883 4883 active+remapped+backfill_toofull 2020-10-29 05:31:28.277898 652'502085 656:367288 [17,9] 17 [17,3] 17 594'499108 2020-10-26 11:06:48.417825 594'499108 2020-10-26 11:06:48.417825 pool : 17 18 19 11 20 21 12 13 0 14 1 15 2 16 | SUM -------------------------------------------------------------------------------------------------------------------------------- osd.4 3 0 0 0 9 2 0 0 12 1 9 0 7 1 | 44 osd.17 1 0 0 0 7 3 1 0 8 1 17 1 11 0 | 50 osd.18 0 0 0 0 9 0 0 0 4 0 7 0 5 0 | 25 osd.5 0 0 0 2 5 1 1 0 5 0 16 0 11 2 | 43 osd.6 0 1 0 1 5 2 0 0 9 0 13 1 7 0 | 39 osd.19 0 0 1 0 8 2 0 1 2 0 6 0 3 0 | 23 osd.7 0 0 0 0 4 1 1 0 3 0 12 0 19 0 | 40 osd.8 0 1 0 0 6 3 0 2 10 1 13 1 15 0 | 52 osd.9 1 0 2 0 10 2 0 0 4 1 6 1 10 0 | 37 osd.10 0 0 1 1 5 2 0 1 7 0 12 0 11 1 | 41 osd.20 1 0 0 0 6 1 0 1 7 0 8 1 17 0 | 42 osd.11 0 0 0 0 4 1 1 1 5 0 11 0 9 0 | 32 osd.12 0 0 1 1 7 1 0 0 5 1 12 1 14 1 | 44 osd.13 0 2 0 0 3 1 0 0 10 1 11 0 10 0 | 38 osd.0 0 1 0 1 6 3 0 1 7 0 11 0 13 0 | 43 osd.14 1 0 0 0 8 1 1 0 4 1 12 0 9 0 | 37 osd.15 1 0 2 1 6 1 1 0 8 0 7 0 6 2 | 35 osd.2 0 2 1 0 7 2 1 0 7 1 4 1 6 0 | 32 osd.3 0 0 0 0 9 0 0 0 2 0 4 0 5 0 | 20 osd.16 0 1 0 1 4 3 1 1 9 0 9 1 12 1 | 43 -------------------------------------------------------------------------------------------------------------------------------- SUM : 8 8 8 8 128 32 8 8 128 8 200 8 200 8 |

4 8

Fix PGs states
by Ing. Luis Felipe Domínguez Vega 02 Nov '20

02 Nov '20

Hi: I have this ceph status: ----------------------------------------------------------------------------- cluster: id: 039bf268-b5a6-11e9-bbb7-d06726ca4a78 health: HEALTH_WARN noout flag(s) set 1 osds down Reduced data availability: 191 pgs inactive, 2 pgs down, 35 pgs incomplete, 290 pgs stale 5 pgs not deep-scrubbed in time 7 pgs not scrubbed in time 327 slow ops, oldest one blocked for 233398 sec, daemons [osd.12,osd.36,osd.5] have slow ops. services: mon: 1 daemons, quorum fond-beagle (age 23h) mgr: fond-beagle(active, since 7h) osd: 48 osds: 45 up (since 95s), 46 in (since 8h); 4 remapped pgs flags noout data: pools: 7 pools, 2305 pgs objects: 350.37k objects, 1.5 TiB usage: 3.0 TiB used, 38 TiB / 41 TiB avail pgs: 6.681% pgs unknown 1.605% pgs not active 1835 active+clean 279 stale+active+clean 154 unknown 22 incomplete 10 stale+incomplete 2 down 2 remapped+incomplete 1 stale+remapped+incomplete -------------------------------------------------------------------------------------------- How can i fix all of unknown, incomplete, remmaped+incomplete, etc... i dont care if i need remove PGs

5 9

Intel SSD firmware guys contacts, if any
by vitalif＠yourcmc.ru 02 Nov '20

02 Nov '20

Hi! I have an interesting question regarding SSDs and I'll try to ask about it here. During my testing of Ceph & Vitastor & Linstor on servers equipped with Intel D3-4510 SSDs I discovered a very funny problem with these SSDs: They don't like overwrites of the same sector. That is, if you overwrite the same sector over and over again you get very low iops: $ fio -direct=1 -rw=write -bs=4k -size=4k -loops=100000 -iodepth=1 write: IOPS=3142, BW=12.3MiB/s (12.9MB/s)(97.9MiB/7977msec) And if you overwrite at least ~128k of other sectors between overwriting the same sector you get normal results: $ fio -direct=1 -rw=write -bs=4k -size=128k -loops=100000 -iodepth=1 write: IOPS=20.8k, BW=81.4MiB/s (85.3MB/s)(543MiB/6675msec) This slowdown almost doesn't hurt Ceph, slightly hurts Vitastor (the impact was greater before I added a fix), and MASSIVELY hurts Linstor/DRBD9 because of its "bitmap". By now I've only seen it on this particular model of SSD. For example, Intel P4500, Micron 9300 Pro, Samsung PM983 don't have this issue. Do you have any contacts of Intel SSD firmware guys to ask them about this bug-o-feature? :-) -- With best regards, Vitaliy Filippov

1 0

how to rbd export image from group snap?
by Timo Weingärtner 02 Nov '20

02 Nov '20

Hi, we're using rbd for VM disk images and want to make consistent backups of groups of them. I know I can create a group and make consistent snapshots of all of them: # rbd --version ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable) # rbd create test_foo --size 1M # rbd create test_bar --size 1M # rbd group create test # rbd group image add test test_foo # rbd group image add test test_bar # rbd group snap create test@1 But how can I export the individual image snapshots? I tried different ways of addressing them, but nothing worked: # rbd export test_foo@1 - error setting snapshot context: (2) No such file or directory # rbd export test_foo@test/1 - rbd: error opening pool 'test_foo@test': (2) No such file or directory # rbd export rbd/test_foo@test/1 - error setting snapshot context: (2) No such file or directory # rbd export test@1/test_foo - rbd: error opening pool 'test@1': (2) No such file or directory # rbd export rbd/test@1/test_foo - rbd: error opening image test: (2) No such file or directory Am I missing something? Mit freundlichen Grüßen, Timo Weingärtner Systemadministrator -- ITscope GmbH Ludwig-Erhard-Allee 20 D-76131 Karlsruhe Tel: +49 721 62737637 Fax: +49 721 66499175 https://www.itscope.com Handelsregister: AG Mannheim, HRB 232782 Sitz der Gesellschaft: Karlsruhe Geschäftsführer: Alexander Münkel, Benjamin Mund, Stefan Reger

1 0

14.2.12 breaks mon_host pointing to Round Robin DNS entry
by Wido den Hollander 02 Nov '20

02 Nov '20

Hi, I already submitted a ticket: https://tracker.ceph.com/issues/47951 Maybe other people noticed this as well. Situation: - Cluster is running IPv6 - mon_host is set to a DNS entry - DNS entry is a Round Robin with three AAAA-records root@wido-standard-benchmark:~# ceph -s unable to parse addrs in 'mon.objects.xx.xxx.net' [errno 22] error connecting to the cluster root@wido-standard-benchmark:~# The relevant part of the ceph.conf: [global] auth_client_required = cephx auth_cluster_required = cephx auth_service_required = cephx mon_host = mon.objects.xxx.xxx.xxx ms_bind_ipv6 = true This works fine with 14.2.11 and breaks under 14.2.12 Anybody else seeing this as well? Wido

5 5

Seriously degraded performance after update to Octopus
by Martin Rasmus Lundquist Hansen 02 Nov '20

02 Nov '20

Two weeks ago we updated our Ceph cluster from Nautilus (14.2.0) to Octopus (15.2.5), an update that was long overdue. We used the Ansible playbooks to perform a rolling update and except from a few minor problems with the Ansible code, the update went well. The Ansible playbooks were also used for setting up the cluster in the first place. Before updating the Ceph software we also performed a full update of CentOS and the Linux kernel (this part of the update had already been tested on one of the OSD nodes the week before and we didn't notice any problems). However, after the update we are seeing a serious decrease in performance, more than a factor of 10x in some cases. I spend a week trying to come up with an explantion or solution, but I am completely blank. Independently of Ceph I tested the network performance and the performance of the OSD disks, and I am not really seeing any problems here. The specifications of the cluster is: - 3x Monitor nodes running mgr+mon+mds (Intel(R) Xeon(R) Silver 4108 CPU @ 1.80GHz, 16 cores, 196 GB RAM) - 14x OSD nodes, each with 18 HDDs and 1 NVME (Intel(R) Xeon(R) Gold 6126 CPU @ 2.60GHz, 24 cores, 384 GB RAM) - CentOS 7.8 and Kernel 5.4.51 - 100 Gbps Infiniband We are collecting various metrics using Prometheus, and on the OSD nodes we are seeing some clear differences when it comes to CPU and Memory usage. I collected some graphs here: http://mitsted.dk/ceph . After the update the system load is highly reduced, there is almost no longer any iowait for the CPU, and the free memory is no longer used for Buffers (I can confirm that the changes in these metrics are not due to the update of CentOS or the Linux kernel). All in all, now the OSD nodes are almost completely idle all the time (and so are the monitors). On the linked page I also attached two RADOS benchmarks. The first benchmark was performed when the cluster was initially configured, and the second is the same benchmark after the update to Octopus. When comparing these two, it is clear that the performance has changed dramatically. For example, in the write test the bandwidth is reduced from 320 MB/s to 21 MB/s and the number of IOPS has also dropped significantly. I temporarily tried to disable the firewall and SELinux on all nodes to see if it made any difference, but it didn’t look like it (I did not restart any services during this test, I am not sure if that could be necessary). Any suggestions for finding the root cause of this performance decrease would be greatly appreciated.

2 1