- ceph-users - lists.ceph.io

Re: Understanding what ceph-volume does, with bootstrap-osd/ceph.keyring, tmpfs
by Marc Roos 22 Sep '20

22 Sep '20

At least ceph thought you the essence of doing first proper testing ;) Because if you test your use case you either get a positive or negative result and not a problem. However I do have to admit that ceph could be more transparent with publishing testing and performance results. I have already discussed this with them on such a ceph day. It does not make sense to have to do everything yourself eg the luks overhead and putting the db/wal on ssd, rbd performance on hdds etc. Those can quickly show if ceph can be a candidate or not. -----Original Message----- From: Kevin Myers [mailto:response@ifastnet.com] Cc: Janne Johansson; Marc Roos; ceph-devel; ceph-users Subject: Re: [ceph-users] Re: Understanding what ceph-volume does, with bootstrap-osd/ceph.keyring, tmpfs Tbh ceph caused us more problems than it tried to fix ymmv good luck > On 22 Sep 2020, at 13:04, tri(a)postix.net wrote: > > The key is stored in the ceph cluster config db. It can be retrieved > by > > KEY=`/usr/bin/ceph --cluster ceph --name > client.osd-lockbox.${OSD_FSID} --keyring $OSD_PATH/lockbox.keyring > config-key get dm-crypt/osd/$OSD_FSID/luks` > > September 22, 2020 2:25 AM, "Janne Johansson" <icepic.dz(a)gmail.com> wrote: > >> Den mån 21 sep. 2020 kl 16:15 skrev Marc Roos <M.Roos(a)f1-outsourcing.eu>: >> >>> When I create a new encrypted osd with ceph volume[1] >>> >>> Q4: Where is this luks passphrase stored? >> >> I think the OSD asks the mon for it after auth:ing, so "in the mon DBs" >> somewhere. >> >> -- >> May the most significant bit of your life be positive. >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an >> email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io

1 0

Ceph MDS stays in "up:replay" for hours. MDS failover takes 10-15 hours.
by heilig.oleg＠gmail.com 22 Sep '20

22 Sep '20

Hi there, We have 9 nodes Ceph cluster. Ceph version is 15.2.5. The cluster has 175 OSD (HDD) + 3 NVMe for cache tier for "cephfs_data" pool. CephFS pools info: POOL ID STORED OBJECTS USED %USED MAX AVAIL cephfs_data 1 350 TiB 179.53M 350 TiB 66.93 87 TiB cephfs_metadata 3 3.1 TiB 17.69M 3.1 TiB 1.77 87 TiB We use multiple active MDS instances: 3 "active" and 3 "standby". Each MDS server has 128GB RAM, "mds cache memory limit" = 64GB. Failover to a standby MDS instance takes 10-15 hours! CephFS is unreachable for the clients all this time. The MDS instance just stays in "up:replay" state for all this time. It looks like MDS demon checking all of the folders: 2020-09-22T02:43:44.406-0700 7f22ae99e700 10 mds.0.journal EOpen.replay 2020-09-22T02:43:44.406-0700 7f22ae99e700 10 mds.0.journal EMetaBlob.replay 3 dirlumps by unknown.0 2020-09-22T02:43:44.406-0700 7f22ae99e700 10 mds.0.journal EMetaBlob.replay dir 0x300000041c5 2020-09-22T02:43:44.406-0700 7f22ae99e700 10 mds.0.journal EMetaBlob.replay updated dir [dir 0x300000041c5 /repository/files/14/ [2,head] auth v=2070324 cv=0/0 state=1610612737|complete f(v0 m2020-09-10T13:05:29.297254-0700 515=0+515) n(v46584 rc2020-09-21T20:38:49.071043-0700 b3937793650802 1056114=601470+454644) hs=515+0,ss=0+0 dirty=75 | child=1 subtree=0 dirty=1 0x55d4c9359b80] 2020-09-22T02:43:44.406-0700 7f22ae99e700 10 mds.0.journal EMetaBlob.replay for [2,head] had [dentry #0x1/repository/files/14/14119 [2,head] auth (dversion lock) v=2049516 ino=0x30000812e2f state=1073741824 | inodepin=1 0x55db2463a1c0] 2020-09-22T02:43:44.406-0700 7f22ae99e700 10 mds.0.journal EMetaBlob.replay for [2,head] had [inode 0x30000812e2f [...2,head] /repository/files/14/14119/ auth fragtree_t(*^3) v2049516 f(v0 m2020-09-18T10:17:53.379121-0700 13498=0+13498) n(v6535 rc2020-09-19T05:52:25.035403-0700 b272027384385 112669=81992+30677) (iversion lock) | dirfrag=8 0x55db24643000] 2020-09-22T02:43:44.406-0700 7f22ae99e700 10 mds.0.journal EMetaBlob.replay dir 0x30000812e2f.000* 2020-09-22T02:43:44.406-0700 7f22ae99e700 10 mds.0.journal EMetaBlob.replay updated dir [dir 0x30000812e2f.000* /repository/files/14/14119/ [2,head] auth v=77082 cv=0/0 state=1073741824 f(v0 m2020-09-18T10:17:53.371122-0700 1636=0+1636) n(v6535 rc2020-09-19T05:51:18.063949-0700 b33321023818 13707=9986+3721) hs=885+0,ss=0+0 | child=1 0x55db845bf080] 2020-09-22T02:43:44.406-0700 7f22ae99e700 10 mds.0.journal EMetaBlob.replay added (full) [dentry #0x1/repository/files/14/14119/39823 [2,head] auth NULL (dversion lock) v=0 ino=(nil) state=1073741888|bottomlru 0x55d82061a900] We tried standby-replay and it helps but doesn't eliminate the root cause. We have millions folders with millions of small files. When the folders/subfolders scan is done, CephFS is active again. I believe 10 hours downtime is unexpected behaviour. Is there any way to force MDS to change status to active and run all of the required directory checks in the background? How can I localise the root cause?

2 1

one-liner getting block device from mounted osd
by Marc Roos 22 Sep '20

22 Sep '20

I have a optimize script that I run after the reboot of a ceph node. It sets among other things /sys/block/sdg/queue/read_ahead_kb and /sys/block/sdg/queue/nr_requests of block devices being used for osd's. Normally I am using the mount command to discover these but with the tmpfs and ceph-volume this does not work. Is anyone able to share a simple onliner getting from osd's the used block devices? Eg from both types at the same time. Or do I really need to traverse this block -> /dev/mapper/... ?

1 1

Re: rgw.none vs quota
by Jean-Sebastien Landry 22 Sep '20

22 Sep '20

Manuel & Konstantin, thank you to confirm this. I should upgrade to Nautilus in the next few weeks. I just live with it for now. Thanks!

1 0

Slow cluster and incorrect peers
by Nico Schottelius 22 Sep '20

22 Sep '20

Hello again, following up on the previous mail, one cluster gets rather slow at the moment and we have spotted something "funny": When checking ceph pg dump we see some osds have HB peers with osds that they should not have any pg in common with. When restarting one of the effected osds, we get the following message: mon_cmd_maybe_osd_create fail: 'osd.12 has already bound to class 'xxx-ssd', can not reset class to 'hdd'; use 'ceph osd crush rm-device-class <id>' to remove old class first': (16) Device or resource busy When checking the output of ceph osd tree, it seems to be in the correct class: 12 xxx-ssd 0.21767 osd.12 up 1.00000 1.00000 Is it possible that the osd has "multiple" classes / that the cluster remebers a class that was set to osd.12 when it used to be an HDD? The output of ceph pg dump includes at the bottom this OSD_STAT USED AVAIL USED_RAW TOTAL HB_PEERS PG_SUM PRIMARY_PG_SUM 12 150 GiB 72 GiB 151 GiB 223 GiB [3,11,13,25,36,43,54,64,71,82] 128 35 which is wrong, because osd.12 should only peer with osd.3 and osd.25, which are the only ones in the same pool that has the replicated rule set to match on xxx-ssd. And the obvious question: how do we fix this? At the moment we see around 75 pgs in peering and 39 activating, most of them which are in a pool with slower SSDs, but it seems that these peerings affect another pool that should have faster SSDs. Best regards, Nico -- Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch

1 0

Mount CEPH-FS on multiple hosts with concurrent access to the same data objects?
by René Bartsch 22 Sep '20

22 Sep '20

I'm new on the list, so a "Hello" to all! :-) We're planning a Proxmox-Cluster. The data-center operator advised to use a virtual machine with NFS on top of a single CEPH-FS instance to mount the shared CEPH-FS storage on multiple hosts/VMs. As this NFS/CEPH-FS-VM could be a bottle-neck I was wondering if CEPH- FS is capable to manage concurrent access and locking itself. Is it possible to mount CEPH-FS instances on multiple hosts (e.g. /srv) all accessing the same data objects without data-loss or dead-locks by concurrent access? Will this perform better than a single NFS/CEPH-FS instance (VM)? Thanx for any hint Renne

5 7

RBD-Mirror: snapshots automatically created?
by Eugen Block 22 Sep '20

22 Sep '20

Hi *, I encountered a rather strange (or at least unexpected) behaviour of the rbd-mirror. Maybe I don't fully understand the feature so please correct me if my assumptions are wrong. My two (virtual one-node) clusters are still on ceph version 15.2.4-864-g0f510cb110 and the following happens if I'm trying to configure snapshot-based rbd-mirroring: - I create a one-way replication from site A to site B - Mirror mode is image (for snapshot-based mirroring) - Import image on site A (no journal-feature enabled to make sure) - Image is replicated as soon as I enable the mirror mode on that image with: > rbd mirror image enable pool/image3 snapshot - rbd info shows "snapshot_count: 1" although I haven't created a snapshot of this image yet, and there's also no schedule: > siteA:~ # rbd mirror snapshot schedule ls > siteA:~ # This is the third image in this test, I only created one snapshot for image1, not for the other images. Is this expected? From the docs I assumed I have to either create snapshots manually or configure the rbd mirror snapshot schedule. Could anyone please clarify? Thank you! Eugen

2 3

Re: Mount CEPH-FS on multiple hosts with concurrent access to the same data objects?
by Stefan Kooman 22 Sep '20

22 Sep '20

On 2020-09-21 21:12, Wout van Heeswijk wrote: > Hi Rene, > > Yes, cephfs is a good filesystem for concurrent writing. When using CephFS with ganesha you can even scale out. > > It will perform better but why don't you mount CephFS inside the VM? ^^ This. But it depends on the VMs you are going to use as clients. Do you trust those clients enough that they are allowed to be part of your cluster. Clients are really part of the cluster, at least that is how I see it. If possible, you want to use modern (5.7, 5.8) linux kernels for cephfs (rm operations: is slower on 4.15/5.3/5.4 for files created with 5.3/5.4 kernel). We have issues (sometimes) with older kernel clients (Ubuntu xenial, 4.15 kernel) and "MDS messages client failed to rdlock") but we don't have 100% proof yet it is because of this kernel version. They generally fix themselves though, so not a big issue. Gr. Stefan

2 1

ceph docs redirect not good
by Marc Roos 22 Sep '20

22 Sep '20

https://docs.ceph.com/docs/mimic/man/8/ceph-volume-systemd/

1 1

Setting up a small experimental CEPH network
by Philip Rhoades 21 Sep '20

21 Sep '20

People, I am interested in experimenting with CEPH on say 4 or 8 small form factor computers (SBCs?) - any suggestions about how to get started? I haven't bought anything yet - I have some working Fedora Workstations and Servers and a laptop but I don't want to experiment on them . . Thanks, Phil. -- Philip Rhoades PO Box 896 Cowra NSW 2794 Australia E-mail: phil(a)pricom.com.au

7 9