- ceph-users - lists.ceph.io

Mimic updated to Nautilus - pg's 'update_creating_pgs' in log, but they exist and cluster is healthy.
by m.sliwinski＠lh.pl 16 Nov '20

16 Nov '20

Hi With the help of Dan van der Steri i managed to confirm my suspicions that the problem with osdmap not trimming correctly was caused by PGs that somehow wasn't marked as created by MONs. For example, from log: 2020-11-16 12:57:00.514 7f131496f700 10 mon.monb01(a)0(probing).osd e72792 update_creating_pgs will instruct osd.265 to create 28.3ff@67698 2020-11-16 12:57:25.982 7f1315971700 10 mon.monb01(a)0(leader).osd e72792 update_creating_pgs will instruct osd.265 to create 28.3ff@72792 But: root@monb01:/var/log/ceph# ceph pg dump |grep 28.3ff dumped all 28.3ff 3841 0 0 0 0 15970230272 0 0 3028 3028 active+clean 2020-11-16 05:38:27.338826 72792'87928 72792:335764 [265,277,282] 265 [265,277,282] 265 72792'85741 2020-11-16 05:38:27.338783 72588'79082 2020-11-10 18:42:43.182436 0 root@monb01:/var/log/ceph# root@monb01:/var/log/ceph# ceph health detail HEALTH_OK root@monb01:/var/log/ceph# Does anyone know how to fix this? I found that it can be cleared by deleting and recreating that pool, but there's a lot of data there so is there way to fix this some other way? -- Best regards Marcin

1 0

build nautilus 14.2.13 packages and container
by Engelmann Florian 16 Nov '20

16 Nov '20

Hi, I was not able to find any complete guide on how to build ceph (14.2.x) from source, create packages and build containers based on those packages. Ubuntu or centos, does not matter. I tried so far: ############### docker pull centos:7 docker run -ti centos:7 /bin/bash yum install -y git rpm-build rpmdevtools wget epel-release yum install -y python-virtualenv python-pip jq cmake3 make gcc-c++ rpm-build which sudo createrepo git clone https://github.com/ceph/ceph cd ceph git checkout v14.2.13 ./make-srpm.sh ./install-deps.sh ################# but install-deps.sh fails with: Error: No Package found for python-scipy The following error message appeared before: ---------------- http://vault.centos.org/centos/7/sclo/Source/rh/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found Trying other mirror. To address this issue please refer to the below wiki article https://wiki.centos.org/yum-errors If above article doesn't help to resolve this issue please use https://bugs.centos.org/. http://vault.centos.org/centos/7/sclo/Source/sclo/repodata/repomd.xml: [Errno 14] HTTP Error 404 - Not Found -------------------- centos:8 fails as well. dependency error: Error: Problem: package python36-rpm-macros-3.6.8-2.module_el8.1.0+245+c39af44f.noarch conflicts with python-modular-rpm-macros > 3.6 provided by python38-rpm-macros-3.8.0-6.module_el8.2.0+317+61fa6e7d.noarch - conflicting requests Any helpful links? All the best, Florian EveryWare AG Florian Engelmann Cloud Platform Architect Zurlindenstrasse 52a CH-8003 Zürich T +41 44 466 60 00 F +41 44 466 60 10 florian.engelmann(a)everyware.ch www.everyware.ch

1 1

Re: which of cpu frequency and number of threads servers osd better?
by Frank Schilder 16 Nov '20

16 Nov '20

We are starting to use 18TB spindles, have loads of cold data and only a thin layer of hot data. One 4/8TB NVMe drive as a cache in front of 6x18TB will provide close to or even matching SSD performance for the hot data at a reasonable extra cost per TB storage. My plan is to wait for 1-2 more years for prices for PCI-NVMe to drop and then start using this method. The second advantage is, that one can continue to deploy collocated HDD OSDs as WAL/DB will certainly land and stay in cache. The cache can be added to existing OSDs without redeployment. In addition, dm-cache uses a hit count method for computing promotion to cache, which works very different from promotion to ceph cache pools. Dm-cache can afford that due to its local nature. In particular, it doesn't promote on just 1 access, which means that a weekly or monthly backup will not flush the entire cache every time. All SSD pools for this data (ceph-fs in EC pool on HDD) will be unaffordable to us for a long time. Not to mention that these large SSDs are almost certainly QLC, which have much less sustained throughput compared with the 18TB He-drives (they have higher IOP/s though, which is not so relevant for our FS use workloads). The cache method will provide at least the additional IOP/s that WAB/DB devices would, but due to its size also data caching. We need to go NVMe, because the servers we plan to use (R740xd2) provide the largest capacity configuration with 24xHDD+4xPCI NVMe. You can either choose 2 extra drives or 4 PCI NVMe, but not both. So, NVMe cannot be exchanged by fast SSDs as they would eat drive slots. There were a few threads over the past 1-2 years where people dropped in some of these observations and I just took note of it. It is used in production already and from what I got people are happy with it. Much easier than WAL/DB partitions plus all the sizing problems for L0/L1/... are sorted trivially. With the size of NVMe growing rapidly beyond what WAL/DB devices can utilize and since LVM is the new OSD device, using LVM dm-cache seems to be the way forward for me. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Anthony D'Atri <anthony.datri(a)gmail.com> Sent: 16 November 2020 03:00:38 To: Frank Schilder Subject: Re: [ceph-users] Re: which of cpu frequency and number of threads servers osd better? Thanks. I’m curious how the economics for that compare with just using all SSDs: * HDDs are cheaper * But colo SSDs are operationally simpler * And depending on configuration you can provision a cheaper HBA > On Nov 14, 2020, at 2:04 AM, Frank Schilder <frans(a)dtu.dk> wrote: > > My plan is to use at least 500GB NVMe per HDD OSD. I have not started that yet, but there are threads of other people sharing their experience. If you go beyond 300GB per OSD, apparently the WAL/DB options cannot really use the extra capacity. With dm-cache or the like you would additionally start holding hot data in cache. > > Ideally, I can split a 4TB or even a 8TB NVMe over 6 OSDs. > > Best regards, > ================= > Frank Schilder > AIT Risø Campus > Bygning 109, rum S14 > > ________________________________________ > From: Anthony D'Atri <anthony.datri(a)gmail.com> > Sent: 14 November 2020 10:57:57 > To: Frank Schilder > Subject: Re: [ceph-users] Re: which of cpu frequency and number of threads servers osd better? > > Guten Tag. > >> My plan for the future is to use dm-cache for LVM OSDs instead of WAL/DB device. > > Do you have any insights into the benefits of that approach instead of WAL/DB, and of dm-cache vs bcache vs dm-writecache vs … ? And any for sizing the cache device and handling failures? Presumably the DB will be active enough that it will persist in the cache, so sizing should be at a minimum that to hold 2 copies of the DB to accomodate compaction? > > I have an existing RGW cluster on HDDs that utilizes a cache tier; the high water mark is set fairly low so that it doesn’t fill up, something that apparently happened last Christmas. I’ve been wanting to get a feel for OSD cache as an alternative to deprecated and fussy cache tiering, as well as something like a Varnish cache on RGW load balancers to short-circult small requests. > > — Anthony > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Beginner's installation questions about network
by E Taka 15 Nov '20

15 Nov '20

Hello, I want to install Ceph Octopus on Ubuntu 20.04. The nodes for have 2 network interfaces: 192.168.1.0/24 for the cluster network, and a 10.10.0.0/16 is the public network. When I bootstrap with cephadm, which Network do I use? That means, do i use cephadm bootstrap --mon-ip 192.168.1.1 or do I have to use the other network? When adding the other host with ceph orch, which network I have to use: ceph orch host add ceph03i 192.168.20.2 … (or the 10.10 network) Thanks, Erich

5 5

Problem in MGR deamon
by Hamidreza Hosseini 14 Nov '20

14 Nov '20

1. Hi, 2. I've installed ceph mon and mgr as cluster on 3 nodes, 3. I have this errors on status and log of standby mgr: ubuntu 20.04 ceph version: 15.2.5 1. ● ceph-mgr(a)dev13.service - Ceph cluster manager daemon 2. Loaded: loaded (/lib/systemd/system/ceph-mgr@.service; enabled; vendor preset: enabled) 3. Active: active (running) since Sat 2020-11-14 14:00:57 +0330; 40min ago 4. Main PID: 13485 (ceph-mgr) 5. Tasks: 20 (limit: 9451) 6. Memory: 233.2M 7. CGroup: /system.slice/system-ceph\x2dmgr.slice/ceph-mgr(a)dev13.service 8. └─13485 /usr/bin/ceph-mgr -f --cluster ceph --id dev13 --setuser ceph --setgroup ceph 9. 10. Nov 14 14:01:03 dev13 ceph-mgr[13485]: File "/lib/python3/dist-packages/numpy/core/__init__.py", line 17, in <module> 11. Nov 14 14:01:03 dev13 ceph-mgr[13485]: from . import multiarray 12. Nov 14 14:01:03 dev13 ceph-mgr[13485]: File "/lib/python3/dist-packages/numpy/core/multiarray.py", line 14, in <module> 13. Nov 14 14:01:03 dev13 ceph-mgr[13485]: from . import overrides 14. Nov 14 14:01:03 dev13 ceph-mgr[13485]: File "/lib/python3/dist-packages/numpy/core/overrides.py", line 16, in <module> 15. Nov 14 14:01:03 dev13 ceph-mgr[13485]: add_docstring( 16. Nov 14 14:01:03 dev13 ceph-mgr[13485]: RuntimeError: implement_array_function method already has a docstring 17. Nov 14 14:01:03 dev13 ceph-mgr[13485]: 2020-11-14T14:01:03.034+0330 7eff2173b040 -1 mgr[py] Class not found in module 'rook' 18. Nov 14 14:01:03 dev13 ceph-mgr[13485]: 2020-11-14T14:01:03.034+0330 7eff2173b040 -1 mgr[py] Error loading module 'rook': (2) No such file or directory 19. Nov 14 14:01:04 dev13 ceph-mgr[13485]: 2020-11-14T14:01:04.478+0330 7eff2173b040 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: rook 20. 21. test@dev12:/etc/ceph$ sudo journalctl -xe 22. Nov 14 13:27:06 dev12 ceph-mgr[15041]: File "/usr/share/ceph/mgr/rook/module.py", line 17, in <module> 23. Nov 14 13:27:06 dev12 ceph-mgr[15041]: from kubernetes import client, config 24. Nov 14 13:27:06 dev12 ceph-mgr[15041]: File "/lib/python3/dist-packages/kubernetes/__init__.py", line 22, in <module> 25. Nov 14 13:27:06 dev12 ceph-mgr[15041]: import kubernetes.stream 26. Nov 14 13:27:06 dev12 ceph-mgr[15041]: File "/lib/python3/dist-packages/kubernetes/stream/__init__.py", line 15, in <module> 27. Nov 14 13:27:06 dev12 ceph-mgr[15041]: from .stream import stream 28. Nov 14 13:27:06 dev12 ceph-mgr[15041]: File "/lib/python3/dist-packages/kubernetes/stream/stream.py", line 13, in <module> 29. Nov 14 13:27:06 dev12 ceph-mgr[15041]: from . import ws_client 30. Nov 14 13:27:06 dev12 ceph-mgr[15041]: File "/lib/python3/dist-packages/kubernetes/stream/ws_client.py", line 19, in <module> 31. Nov 14 13:27:06 dev12 ceph-mgr[15041]: from websocket import WebSocket, ABNF, enableTrace 32. Nov 14 13:27:06 dev12 ceph-mgr[15041]: File "/lib/python3/dist-packages/websocket/__init__.py", line 22, in <module> 33. Nov 14 13:27:06 dev12 ceph-mgr[15041]: from ._abnf import * 34. Nov 14 13:27:06 dev12 ceph-mgr[15041]: File "/lib/python3/dist-packages/websocket/_abnf.py", line 34, in <module> 35. Nov 14 13:27:06 dev12 ceph-mgr[15041]: import numpy 36. Nov 14 13:27:06 dev12 ceph-mgr[15041]: File "/lib/python3/dist-packages/numpy/__init__.py", line 142, in <module> 37. Nov 14 13:27:06 dev12 ceph-mgr[15041]: from . import core 38. Nov 14 13:27:06 dev12 ceph-mgr[15041]: File "/lib/python3/dist-packages/numpy/core/__init__.py", line 17, in <module> 39. Nov 14 13:27:06 dev12 ceph-mgr[15041]: from . import multiarray 40. Nov 14 13:27:06 dev12 ceph-mgr[15041]: File "/lib/python3/dist-packages/numpy/core/multiarray.py", line 14, in <module> 41. Nov 14 13:27:06 dev12 ceph-mgr[15041]: from . import overrides 42. Nov 14 13:27:06 dev12 ceph-mgr[15041]: File "/lib/python3/dist-packages/numpy/core/overrides.py", line 16, in <module> 43. Nov 14 13:27:06 dev12 ceph-mgr[15041]: add_docstring( 44. Nov 14 13:27:06 dev12 ceph-mgr[15041]: RuntimeError: implement_array_function method already has a docstring 45. Nov 14 13:27:06 dev12 ceph-mgr[15041]: 2020-11-14T13:27:06.159+0330 7fdeafd88040 -1 mgr[py] Class not found in module 'rook' 46. Nov 14 13:27:06 dev12 ceph-mgr[15041]: 2020-11-14T13:27:06.159+0330 7fdeafd88040 -1 mgr[py] Error loading module 'rook': (2) No such file or directory 47. Nov 14 13:27:07 dev12 ceph-mgr[15041]: 2020-11-14T13:27:07.547+0330 7fdeafd88040 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: rook 48. 2020-11-14T13:27:06.431+0330 7fdeafd88040 1 mgr[py] Loading python module 'telegraf' 49. 2020-11-14T13:27:06.667+0330 7fdeafd88040 1 mgr[py] Loading python module 'telemetry' 50. 2020-11-14T13:27:06.987+0330 7fdeafd88040 1 mgr[py] Loading python module 'test_orchestrator' 51. 2020-11-14T13:27:07.187+0330 7fdeafd88040 1 mgr[py] Loading python module 'volumes' 52. 2020-11-14T13:27:07.459+0330 7fdeafd88040 1 mgr[py] Loading python module 'zabbix' 53. 2020-11-14T13:27:07.547+0330 7fdeafd88040 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: rook 54. 2020-11-14T13:27:07.551+0330 7fdeaad22700 0 ms_deliver_dispatch: unhandled message 0x55dcf177e2c0 mon_map magic: 0 v1 from mon.2 v2:192.168.200.23:3300/0 55. 2020-11-14T13:27:07.555+0330 7fdeaad22700 0 ms_deliver_dispatch: unhandled message 0x55dcf177fa20 mon_map magic: 0 v1 from mon.2 v2:192.168.200.23:3300/0 56. 2020-11-14T13:42:07.558+0330 7fdeaad22700 0 client.0 ms_handle_reset on v2:192.168.200.21:6800/6434 57. 2020-11-14T13:57:07.569+0330 7fdeaad22700 0 client.0 ms_handle_reset on v2:192.168.200.21:6800/6434 58. 59.

1 0

Re: which of cpu frequency and number of threads servers osd better?
by Frank Schilder 14 Nov '20

14 Nov '20

My plan is to use at least 500GB NVMe per HDD OSD. I have not started that yet, but there are threads of other people sharing their experience. If you go beyond 300GB per OSD, apparently the WAL/DB options cannot really use the extra capacity. With dm-cache or the like you would additionally start holding hot data in cache. Ideally, I can split a 4TB or even a 8TB NVMe over 6 OSDs. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________________________________________ From: Anthony D'Atri <anthony.datri(a)gmail.com> Sent: 14 November 2020 10:57:57 To: Frank Schilder Subject: Re: [ceph-users] Re: which of cpu frequency and number of threads servers osd better? Guten Tag. > My plan for the future is to use dm-cache for LVM OSDs instead of WAL/DB device. Do you have any insights into the benefits of that approach instead of WAL/DB, and of dm-cache vs bcache vs dm-writecache vs … ? And any for sizing the cache device and handling failures? Presumably the DB will be active enough that it will persist in the cache, so sizing should be at a minimum that to hold 2 copies of the DB to accomodate compaction? I have an existing RGW cluster on HDDs that utilizes a cache tier; the high water mark is set fairly low so that it doesn’t fill up, something that apparently happened last Christmas. I’ve been wanting to get a feel for OSD cache as an alternative to deprecated and fussy cache tiering, as well as something like a Varnish cache on RGW load balancers to short-circult small requests. — Anthony

1 0

which of cpu frequency and number of threads servers osd better?
by Tony Liu 14 Nov '20

14 Nov '20

Hi, For example, 16 threads with 3.2GHz and 32 threads with 3.0GHz, which makes 11 OSDs (10x12TB HDD and 1x960GB SSD) with better performance? Thanks! Tony

4 10

monitor sst files continue growing
by Zhenshi Zhou 14 Nov '20

14 Nov '20

Hi all, My cluster is in wrong state. SST files in /var/lib/ceph/mon/xxx/store.db continue growing. It claims mon are using a lot of disk space. I set "mon compact on start = true" and restart one of the monitors. But it started and campacting for a long time, seems it has no end. [image: image.png]

4 14

How to Improve RGW Bucket Stats Performance
by Denis Krienbühl 13 Nov '20

13 Nov '20

Hi! To bill our customers we regularly call radosgw-admin bucket stats --uid <uid>. Since upgrading from Mimic to Octopus (with a short stop at Nautilus), we’ve been seeing much slower response times for this command. It went from less than a minute for our largest customers, to 5 minutes (with some variance depending on load). Assuming this is not a bug, is there any way to get these stats quicker? Ceph seems to do in a single call here, which seems to me like something you could spread out over time (keep a counter somewhere and just return the latest value on request). One thing we did notice, is that we get a lot of these when the stats are synced: 2020-11-13T14:56:17.288+0100 7f15347e0700 0 check_bucket_shards: resharding needed: stats.num_objects=5776982 shard max_objects=3200000 Could that hint at a problem in our configuration? Anything else we could maybe tune to get this time down? Appreciate any hints and I hope everyone is about to have a great weekend. Denis

1 0

OverlayFS with Cephfs to mount a snapshot read/write
by Frédéric Nass 13 Nov '20

13 Nov '20

Hello, I would like to use a cephfs snapshot as a read/write volume without having to clone it first as the cloning operation is - if I'm not mistaken - still inefficient as of now. This is for a data restore use case with Moodle application needing a writable data directory to start. The idea that came to mind was to use overlayFS with cephfs set up as a read-only lower layer and a writable local directory set up as an upper layer. With this set up, any modifications to the read-only .snap/testsnap directory would normally go to the upper directory making the snapshot directory somehow writable to the Moodle application. While this works fine when a local read-only filesystem is set up as the lower layer, it fails when cephfs is set up as the lower layer. Any modifications to the .snap/testsnap tree in the /cephfs-snap directory fails with an "Operation not supported". $ mkdir /cephfs /upperdir /workdir /cephfs-snap $ mount -t ceph 100.74.191.129:/volumes/group1/subvolume1/ /cephfs -o name=admin,secretfile=/etc/ceph/admin.secret $ mount -t overlay overlay -o redirect_dir=on,lowerdir=/cephfs/.snap/testsnap,upperdir=/upperdir,workdir=/workdir /cephfs-snap $ ls /cephfs-snap usr $ touch /cephfs-snap/foo.txt <---- writing outside the lowerdir succeeds $ ls /cephfs-snap foo.txt usr $ ls /usr/etc $ touch /cephfs-snap/usr/etc/foo <---- writing inside the lowerdir fails touch: impossible de faire un touch « /cephfs-snap/usr/etc/foo »: Opération non supportée I tried to mount the whole cephfs tree read-only (-o ro), tried to disable ACLs (-o noacl) as seen here [1] but of no help. Mounting with ceph-fuse didn't help either. There's been a recent discussion about this here [2] between Greg and Robert but with no real solution. Did someone manage to do this? Regards, Frédéric. [1] https://blog.fai-project.org/posts/overlayfs/ [2] https://tracker.ceph.com/issues/44821

5 10