- ceph-users - lists.ceph.io

increasing PG count - limiting disruption
by Frank R 14 Nov '19

14 Nov '19

Hi all, When increasing the number of placement groups for a pool by a large amount (say 2048 to 4096) is it better to go in small steps or all at once? This is a filestore cluster. Thanks, Frank

2 1

mds crash loop - cephfs disaster recovery
by Karsten Nielsen 14 Nov '19

14 Nov '19

I am a problem with my mds that is in a crash loop, with the help of Yan, Zheng I have run a few attempts to save it but it seems that it is not going the way it should. I am reading through this documentation. https://docs.ceph.com/docs/mimic/cephfs/disaster-recovery/ If I use the last step to get it back by creating an alternative metadata pool for recovery. If that works how do I remove the broken metadata pool and rename the new one to match the broken one ? Thanks, - Karsten

1 0

Adding new non-containerised hosts to current contanerised environment and moving away from containers forward
by Jeremi Avenant 14 Nov '19

14 Nov '19

Good day We currently have 12 nodes in 4 Racks (3x4) and getting another 3 nodes to complete the 5th rack on Version 12.2.12, using ceph-ansible & docker containers. With the 3 new nodes (1 rack bucket) we would like to make use of a non-containerised setup since our long term plan is to completely move away from OSD containers. How would one go forward running a hybrid environment in the interim until we've rebuilt all our existing osd nodes? I can assume that i would require a separate /ceph-ansible directory with site.yml to my existing site-docker.yml Would I require separate inventory files, one for the existing 12 nodes & one for the 3 new nodes? How would I connect the new nodes to the current controller (mds,mgr, mon) container nodes? Regards -- *Jeremi-Ernst Avenant, Mr.*Cloud Infrastructure Specialist Inter-University Institute for Data Intensive Astronomy 5th Floor, Department of Physics and Astronomy, University of Cape Town Tel: 021 959 4137 <0219592327> Web: www.idia.ac.za <http://www.uwc.ac.za/> E-mail (IDIA): jeremi(a)idia.ac.za <mfundo(a)idia.ac.za> Rondebosch, Cape Town, 7600, South Africa

2 1

Allowing cephfs clients to reconnect
by Mikael Öhman 14 Nov '19

14 Nov '19

Hi, I'm trying to make our system a bit more fault tolerant, and I struggle a bit with letting clients reconnect if they have lost contact for a while. When there is a temporary network problem, I would like clients to block I/O, wait for a connection, and resume. Do I have any options other than just increasing mds_session_autoclose ? Is there a downside for using very large value here (like, a full day?)? I expect all clients to be connected at all times anyway when things are running normally. What I see right now (if the disconnect is sufficiently long) is that the ceph client releases the I/O block, and you get permission denied on all I/O operations on the existing mount point. Re-mounting it works, but, this also requires killing off all active session blocking unmounting. Basically, just overall bad is this happens, and I would prefer almost any other option. I can see that the client tries a reconnect when this happens: Nov 12 11:53:24 hebbe01-3 kernel: libceph: mds0 10.43.20.3:6800 connection reset Nov 12 11:53:24 hebbe01-3 kernel: libceph: reset on mds0 Nov 12 11:53:24 hebbe01-3 kernel: ceph: mds0 closed our session Nov 12 11:53:24 hebbe01-3 kernel: ceph: mds0 reconnect start Nov 12 11:53:24 hebbe01-3 kernel: ceph: mds0 reconnect denied Nov 12 11:56:55 hebbe01-3 kernel: libceph: mds0 10.43.20.3:6800 socket closed (con state NEGOTIATING) Nov 12 11:56:55 hebbe01-3 kernel: ceph: mds0 rejected session but the logs on the MDS server disallows it as it's not in a "reconnect state"- So, if I understand this correctly, reconnecting is just available in the case that the MDS server was rebooted? Best regards, Mikael

2 1

Re: custom x-amz-request-id
by Robin H. Johnson 13 Nov '19

13 Nov '19

On Wed, Nov 13, 2019 at 04:32:45PM +0000, Arash Shams wrote: > Hi everybody > Im using Nginx in front of radosgw and I generate the request id header on nginx can I pass the same value to radosgw and tell it use this header instead of generating a new one ? > > nginx sample : > > more_set_input_headers "x-amz-request-id: $txid" You'll have to patch RGW's source to do this: 1. Copy the header from request to response. 2. Do NOT store the header as metadata in PutObject. -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Treasurer E-Mail : robbat2(a)gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136

1 0

Counting OSD maps
by Bryan Stillwell 13 Nov '19

13 Nov '19

With FileStore you can get the number of OSD maps for an OSD by using a simple find command: # rpm -q ceph ceph-12.2.12-0.el7.x86_64 # find /var/lib/ceph/osd/ceph-420/current/meta/ -name 'osdmap*' | wc -l 42486 Does anyone know of an equivalent command that can be used with BlueStore? Thanks, Bryan

2 1

custom x-amz-request-id
by Arash Shams 13 Nov '19

13 Nov '19

Hi everybody Im using Nginx in front of radosgw and I generate the request id header on nginx can I pass the same value to radosgw and tell it use this header instead of generating a new one ? nginx sample : more_set_input_headers "x-amz-request-id: $txid" Thanks

1 0

Ceph I/O issues on all SSD cluster
by Dennis Højgaard | Powerhosting Support 13 Nov '19

13 Nov '19

OK, where to start. I have been debugging intensively the last two days, but can't seem to wrap my head around the performance issues we see in one of our two hyperconverged (ceph) proxmox clusters. Let me introduce our two clusters and some of the debugging results. *1. Cluster for internal purposes (performs as expected)* 3 x Supermicro servers with identical specs: CPU: 1 x 7-7700K CPU @ 4.20GHz (1 Socket) 4 cores / 8 threads RAM: 64 GB RAM OSDs: 4 per node. 1 per SSD (Intel S4610) (12 OSDs in all) 1 x 10GbE RJ45 nic. MTU 9000 No bonding _A total of 3 servers with a total of 12 OSDs_ Network: 1 x Unifi Switch 16 XG *2. Cluster for VPS's for customers (performs much worse than internal)* 3 x Dell R630 with the following specs: CPU: 2 x E5-2697 v3 @ 2.60GHz (2 Sockets) 28 cores / 56 threads RAM 256GB OSDs: 10 per node. 1 per SSD (Intel S4610) 1 x 10GbE SFP+ nic with 2 ports bonded via LACP (bond-xmit-hash-policy layer3+4). MTU 9000 2 x Supermicro X11SRM-VF with the following specs: CPU 1 x 1 W-2145 CPU @ 3.70GHz (1 Socket) 8 cores / 16 threads RAM: 256 GB OSDs 8 per node. 1 per SSD (Intel S4610) 1 x 10GbE SFP+ nic with 2 ports bonded via LACP (bond-xmit-hash-policy layer3+4). MTU 9000 1 x Dell R630 with the following specs: CPU 2 x CPU E5-2696 v4 @ 2.20GHz (2 Sockets) 44 cores / 88 threads RAM: 256 GB OSDs 8 per node. 1 per SSD (Intel S4610) 1 x 10GbE SFP+ nic with 2 ports bonded via LACP (bond-xmit-hash-policy layer3+4). MTU 9000 _A total of 6 servers with a total of 54 OSDs_ Network: 2 x Dell N4032F 10GbE SFP+ Switch connected with MLAG. Each node is connected to each switch. To get a fair comparison i made the following fio tests one one host in each cluster on a rbd block device that i created: *1. cluster:* |fio --randrepeat=1 --ioengine=libaio --sync=1 --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randwrite test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64 fio-3.12 Starting 1 process test: Laying out IO file (1 file / 4096MiB) Jobs: 1 (f=1): [w(1)][100.0%][w=25.0MiB/s][w=6409 IOPS][eta 00m:00s] test: (groupid=0, jobs=1): err= 0: pid=869177: Wed Nov 13 11:15:59 2019 write: IOPS=4158, BW=16.2MiB/s (17.0MB/s)(4096MiB/252126msec); 0 zone resets bw ( KiB/s): min= 2075, max=32968, per=99.96%, avg=16627.60, stdev=9635.42, samples=504 iops : min= 518, max= 8242, avg=4156.88, stdev=2408.86, samples=504 cpu : usr=0.53%, sys=3.81%, ctx=109599, majf=0, minf=7 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=0,1048576,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): WRITE: bw=16.2MiB/s (17.0MB/s), 16.2MiB/s-16.2MiB/s (17.0MB/s-17.0MB/s), io=4096MiB (4295MB), run=252126-252126msec Disk stats (read/write): rbd0: ios=46/1221898, merge=0/1870438, ticks=25/4654920, in_queue=1980016, util=84.70%| *2. cluster* |fio --randrepeat=1 --ioengine=libaio --sync=1 --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randwrite test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64 fio-3.12 Starting 1 process Jobs: 1 (f=1): [w(1)][99.9%][w=7024KiB/s][w=1756 IOPS][eta 00m:01s] test: (groupid=0, jobs=1): err= 0: pid=794096: Wed Nov 13 11:25:56 2019 write: IOPS=1353, BW=5415KiB/s (5545kB/s)(4096MiB/774601msec); 0 zone resets bw ( KiB/s): min= 40, max=30600, per=100.00%, avg=5420.24, stdev=3710.17, samples=1547 iops : min= 10, max= 7650, avg=1355.06, stdev=927.54, samples=1547 cpu : usr=0.16%, sys=1.19%, ctx=100028, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=0,1048576,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): WRITE: bw=5415KiB/s (5545kB/s), 5415KiB/s-5415KiB/s (5545kB/s-5545kB/s), io=4096MiB (4295MB), run=774601-774601msec Disk stats (read/write): rbd0: ios=0/1222639, merge=0/1784089, ticks=0/12124812, in_queue=9514280, util=45.14%| And identical rados bench tests: 1. cluster:https://i.imgur.com/AdARCA6.png 2. clusterhttps://i.imgur.com/Di7mYQh.png I have fio tested all disks. I have tested the network. I can't seem to find the reason why performance on my 2. cluster is relatively poor compared to 1. cluster. -- Dennis Højgaard Powerhosting Support *t:* +45 7222 4457 |*e:* dh(a)powerhosting.dk |*w:* https://powerhosting.dk

1 0

xattrs on snapshots
by Toby Darling 13 Nov '19

13 Nov '19

Hi Folks I'm writing a backup/sync process for our ceph cluster. The process takes a snapshot of the system that's being backed up and rsync's from that to another ceph cluster I was hoping to use the snapshot xattrs to verify a successful backup by comparing ceph.dir.rbytes and ceph.dir.rentries between the snapshot and backup target. However, while files added to the source directory after the snapshot is taken don't appear in the snapshot, ceph.dir.rbytes and ceph.dir.rctime are updated in the xattrs of both. $ fallocate -l 1K test1 $ getfattr -n ceph.dir.rbytes . # file: . ceph.dir.rbytes="1024" $ mkdir .snap/testsnap $ fallocate -l 2K test2 $ getfattr -n ceph.dir.rbytes . .snap/testsnap # file: . ceph.dir.rbytes="3072" # file: .snap/testsnap ceph.dir.rbytes="3072" $ getfattr -n ceph.dir.rctime . .snap/testsnap # file: . ceph.dir.rctime="1573558448.09736131125" # file: .snap/testsnap ceph.dir.rctime="1573558448.09736131125" $ tree -a --si . .snap . |-- [1.0k] test1 `-- [2.0k] test2 .snap `-- [3.1k] testsnap `-- [1.0k] test1 1 directory, 3 files Have I misunderstood? Is this expected behaviour? Cheers Toby -- Toby Darling, Scientific Computing (2N249) MRC Laboratory of Molecular Biology Francis Crick Avenue Cambridge Biomedical Campus Cambridge CB2 0QH Phone 01223 267070

3 3

ceph clients and cluster map
by Frank R 12 Nov '19

12 Nov '19

Hi all, When the cluster map changes say due to a failed OSD how are the clients with established sessions notified? thx Frank

1 0