- ceph-users - lists.ceph.io

cephfs auditing
by Michael Thomas 28 May '21

28 May '21

Is there a way to log or track which cephfs files are being accessed? This would help us in planning where to place certain datasets based on popularity, eg on a EC HDD pool or a replicated SSD pool. I know I can run inotify on the ceph clients, but I was hoping that the MDS would have a way to log this information centrally. --Mike

3 2

XFS on RBD on EC painfully slow
by Reed Dier 28 May '21

28 May '21

Hoping someone may be able to help point out where my bottleneck(s) may be. I have an 80TB kRBD image on an EC8:2 pool, with an XFS filesystem on top of that. This was not an ideal scenario, rather it was a rescue mission to dump a large, aging raid array before it was too late, so I'm working with the hand I was dealt. To further conflate the issues, the main directory structure consists of lots and lots of small file sizes, and deep directories. My goal is to try and rsync (or otherwise) data from the RBD to cephfs, but its just unbearably slow and will take ~150 days to transfer ~35TB, which is far from ideal. > 15.41G 79% 4.36MB/s 0:56:09 (xfr#23165, ir-chk=4061/27259) > avg-cpu: %user %nice %system %iowait %steal %idle > 0.17 0.00 1.34 13.23 0.00 85.26 > > Device r/s rMB/s rrqm/s %rrqm r_await rareq-sz w/s wMB/s wrqm/s %wrqm w_await wareq-sz d/s dMB/s drqm/s %drqm d_await dareq-sz aqu-sz %util > rbd0 124.00 0.66 0.00 0.00 17.30 5.48 50.00 0.17 0.00 0.00 31.70 3.49 0.00 0.00 0.00 0.00 0.00 0.00 3.39 96.40 Rsync progress and iostat (during the rsync) from the rbd to a local ssd, to remove any bottlenecks doubling back to cephfs. About 16G in 1h, not exactly blazing, this being 5 of the 7000 directories I'm looking to offload to cephfs. Currently running 15.2.11, and the host is Ubuntu 20.04 (5.4.0-72-generic) with a single E5-2620, 64GB of memory, and 4x10GbT bond talking to ceph, iperf proves it out. EC8:2, across about 16 hosts, 240 OSDs, with 24 of those being 8TB 7.2k SAS, and the other 216 being 2TB 7.2K SATA. So there are quite a few spindles in play here. Only 128 PGs, in this pool, but its the only RBD image in this pool. Autoscaler recommends going to 512, but was hoping to avoid the performance overhead of the PG splits if possible, given perf is bad enough as is. Examining the main directory structure it looks like there are 7000 files per directory, about 60% of which are <1MiB, and in all totaling nearly 5GiB per directory. My fstab for this is: > xfs _netdev,noatime 0 0 I tried to increase the read_ahead_kb to 4M from 128K at /sys/block/rbd0/queue/read_ahead_kb to match the object/stripe size of the EC pool, but that doesn't appear to have had much of an impact. The only thing I can think of that I could possibly try as a change would be to increase the queue depth in the rbdmap up from 128, so thats my next bullet to fire. Attaching xfs_info in case there are any useful nuggets: > meta-data=/dev/rbd0 isize=256 agcount=81, agsize=268435455 blks > = sectsz=512 attr=2, projid32bit=0 > = crc=0 finobt=0, sparse=0, rmapbt=0 > = reflink=0 > data = bsize=4096 blocks=21483470848, imaxpct=5 > = sunit=0 swidth=0 blks > naming =version 2 bsize=4096 ascii-ci=0, ftype=0 > log =internal log bsize=4096 blocks=32768, version=2 > = sectsz=512 sunit=0 blks, lazy-count=0 > realtime =none extsz=4096 blocks=0, rtextents=0 And rbd-info: > rbd image 'rbd-image-name: > size 85 TiB in 22282240 objects > order 22 (4 MiB objects) > snapshot_count: 0 > id: a09cac2b772af5 > data_pool: rbd-ec82-pool > block_name_prefix: rbd_data.29.a09cac2b772af5 > format: 2 > features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, data-pool > op_features: > flags: > create_timestamp: Mon Apr 12 18:44:38 2021 > access_timestamp: Mon Apr 12 18:44:38 2021 > modify_timestamp: Mon Apr 12 18:44:38 2021 Any other ideas or hints are greatly appreciated. Thanks, Reed

4 4

Re: XFS on RBD on EC painfully slow
by Reed Dier 28 May '21

28 May '21

I guess I should probably have been more clear, this is one pool of many, so the other OSDs aren't idle. So I don't necessarily think that the PG bump would be the worst thing to try, but its definitely not as bad as I may have made it sound. Thanks, Reed > On May 27, 2021, at 11:37 PM, Anthony D'Atri <anthony.datri(a)gmail.com> wrote: > > That gives you a PG ratio of …. 5.3 ??? > > Run `ceph osd df` ; I wouldn’t be surprised if some of your drives have 0 PGs on them, for sure I would suspect that they aren’t even at all. > > There are bottlenecks in the PG code, and in the OSD code — one reason why with NVMe clusters it’s common to split each drive into at least 2 OSDs. With spinners you don’t want to do that, but you get the idea. > > The pg autoscaler is usually out of its Vulcan mind. 512 would give you a ratio of just 21. > > Prior to 12.2.1 conventional wisdom was a PG ratio of 100-200 on spinners. > > 2048 PGs would give you a ratio of 85, which current (retconned) guidance would call good. I’d probably go to 4096 but 2048 would be way better than 128. > > I strongly suspect that PG splitting would still get you done faster than the way it is, esp. if you’re running BlueStore OSDs. > > Try bumping pg_num up to say 262 and see how bad it is, and if when pgp_num catches up if your ingest rate isn’t a bit higher than it was before. > >> EC8:2, across about 16 hosts, 240 OSDs, with 24 of those being 8TB 7.2k SAS, and the other 216 being 2TB 7.2K SATA. So there are quite a few spindles in play here. >> Only 128 PGs, in this pool, but its the only RBD image in this pool. Autoscaler recommends going to 512, but was hoping to avoid the performance overhead of the PG splits if possible, given perf is bad enough as is. > >

1 0

Messed up placement of MDS
by mabi 28 May '21

28 May '21

Hello, I am trying to place the two MDS daemons for CephFS on dedicated nodes. For that purpose I tried out a few different "cephadm orch apply ..." commands with a label but at the end it looks like I messed up with the placement as I now have two mds service_types as you can see below: # ceph orch ls --service-type mds --export service_type: mds service_id: ceph1fs service_name: mds.ceph1fs placement: count: 2 hosts: - ceph1g - ceph1a --- service_type: mds service_id: label:mds service_name: mds.label:mds placement: count: 2 This second entry at the bottom seems totally wrong and I would like to remove it but I haven't found how to remove it totally. Any ideas? Ideally I just want to place two MDS daemons on node ceph1a and ceph1g. Regards, Mabi

1 1

Ceph osd will not start.
by Peter Childs 27 May '21

27 May '21

I'm attempting to get get ceph up and running, and currently feel like I'm going around in circles. I'm attempting to use cephadm and Pacific, currently on debian buster, mostly because centos7 ain't supported any more and cenotos8 ain't support by some of my hardware. Anyway I have a few nodes with 59x 7.2TB disks but for some reason the osd daemons don't start, the disks get formatted and the osd are created but the daemons never come up. They are probably the wrong spec for ceph (48gb of memory and only 4 cores) but I was expecting them to start and be either dirt slow or crash later, anyway I've got upto 30 of them, so I was hoping on getting at least get 6PB of raw storage out of them. As yet I've not spotted any helpful error messages. This is for a archive / slow ceph cluster so I'm not expecting speed. Thanks in advance. Peter.

4 8

rebalancing after node more
by Rok Jaklič 27 May '21

27 May '21

Hi, I have removed one node, but now ceph seems to stuck in: Degraded data redundancy: 67/2393 objects degraded (2.800%), 12 pgs degraded, 12 pgs undersized How to "force" rebalancing? Or should I just wait a little bit more? Kind regards, rok

2 5

cephadm: How to replace failed HDD where DB is on SSD
by Kai Stian Olstad 27 May '21

27 May '21

Hi The server run 15.2.9 and has 15 HDD and 3 SSD. The OSDs was created with this YAML file hdd.yml -------- service_type: osd service_id: hdd placement: host_pattern: 'pech-hd-*' data_devices: rotational: 1 db_devices: rotational: 0 The result was that the 3 SSD is added to 1 VG with 15 LV on it. # vgs | egrep "VG|dbs" VG #PV #LV #SN Attr VSize VFree ceph-block-dbs-563432b7-f52d-4cfe-b952-11542594843b 3 15 0 wz--n- <5.24t 48.00m One of the osd failed and I run rm with replace # ceph orch osd rm 178 --replace and the result is # ceph osd tree | grep "ID|destroyed" ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 178 hdd 12.82390 osd.178 destroyed 0 1.00000 But I'm not able to replace the disk with the same YAML file as shown above. # ceph orch apply osd -i hdd.yml --dry-run ################ OSDSPEC PREVIEWS ################ +---------+------+------+------+----+-----+ |SERVICE |NAME |HOST |DATA |DB |WAL | +---------+------+------+------+----+-----+ +---------+------+------+------+----+-----+ I guess this is the wrong way to do it, but I can't find the answer in the documentation. So how can I replace this failed disk in Cephadm? -- Kai Stian Olstad

3 10

How to add back stray OSD daemon after node re-installation
by mabi 27 May '21

27 May '21

Hello, I have by mistake re-installed the OS of an OSD node of my Octopus cluster (managed by cephadm). Luckily the OSD data is on a separate disk and did not get affected by the re-install. Now I have the following state: health: HEALTH_WARN 1 stray daemon(s) not managed by cephadm 1 osds down 1 host (1 osds) down To fix that I tried to run: # ceph orch daemon add osd ceph1f:/dev/sda Created no osd(s) on host ceph1f; already created? That did not work, so I tried: # ceph cephadm osd activate ceph1f no valid command found; 10 closest matches: ... Error EINVAL: invalid command Did not work either. So I wanted to ask how can I "adopt" back an OSD disk to my cluster? Thanks for your help. Regards, Mabi

2 10

MDS stuck in up:stopping state
by Martin Rasmus Lundquist Hansen 27 May '21

27 May '21

After scaling the number of MDS daemons down, we now have a daemon stuck in the "up:stopping" state. The documentation says it can take several minutes to stop the daemon, but it has been stuck in this state for almost a full day. According to the "ceph fs status" output attached below, it still holds information about 2 inodes, which we assume is the reason why it cannot stop completely. Does anyone know what we can do to finally stop it? cephfs - 71 clients ====== RANK STATE MDS ACTIVITY DNS INOS 0 active ceph-mon-01 Reqs: 0 /s 15.7M 15.4M 1 active ceph-mon-02 Reqs: 48 /s 19.7M 17.1M 2 stopping ceph-mon-03 0 2 POOL TYPE USED AVAIL cephfs_metadata metadata 652G 185T cephfs_data data 1637T 539T STANDBY MDS ceph-mon-03-mds-2 MDS version: ceph version 15.2.11 (e3523634d9c2227df9af89a4eac33d16738c49cb) octopus (stable)

3 7

cephfs:: store files on different pools?
by Adrian Sevcenco 27 May '21

27 May '21

Hi! is is (technically) possible to instruct cephfs to store files < 1Mib on a (replicate) pool and the others files on another (ec) pool? And even more, is it possible to take the same kind of decision on the path of the file? (let's say that critical files with names like r"/critical_path/critical_.*" i want them in a 6x replication ssd pool) Thank you! Adrian

2 1