- ceph-users - lists.ceph.io

how to handle incomplete PGs
by huxiaoyu＠horebdata.cn 15 Aug '20

15 Aug '20

Dear Ceph folks, Recently i encountered incomplete PGs when replacing an OSD node with new handware. I noticed multiple OSD ups and downs, and eventually a few PGs got stucked at PG incomplete status. Questions 1: is there a reliable way to avoid the occurence of incomplete PGs? 2: is there a good tool or scriptes to handle incomplete PGs without lossing data best regards, samuel huxiaoyu(a)horebdata.cn

1 0

CephFS clients waiting for lock when one of them goes slow
by Petr Belyaev 14 Aug '20

14 Aug '20

Hi all, I have a very small Ceph setup - 3 OSDs, 3 MDS, 3 MON, CephFS with ~20 ceph-fuse clients connected. All run version 14.2.9 (both servers and clients). Some clients work with a relatively large CephFS directory, few tens of thousands of files, with lots of files being added and deleted all the time. Almost every day I see exactly the same pattern when system goes under load: - One of the clients starts running slow (e.g. high CPU or out-of-memory, sometimes with OOM killer intervention) - A number of requests from other clients gets blocked for ~15 seconds waiting for the (r/w/x)lock on other files in the same directory - Objecter requests empty - No requests visibly running in the dump_ops_in_flight output, only those waiting for the lock MDS cache size is 1.5 gb, total capacity - around 1TB, usual load is ~300 [rw]ops, 3-4 mb/s, most of the config is the default one. Have somebody seen similar issues before? Best regards, Petr Belyaev

4 3

Nautilus slow using "ceph tell osd.* bench"
by Jim Forde 14 Aug '20

14 Aug '20

I have 2 clusters. Cluster 1 started at Hammer and has upgraded through the versions all the way to Nautilus 14.2.10 (Luminous to Nautilus in July 2020) . Cluster 2 started as Luminous and is now Nautilus 14.2.2 (Upgraded in September 2019) The clusters are basically identical 5 OSD Nodes with 6 osd's per node. They are both using disk drives. No SSD's. Prior to upgrading Cluster 1 running "ceph tell osd.0 bench -f plain" produced similar results across both clusters. ceph tell osd.0 bench -f plain bench: wrote 1 GiB in blocks of 4 MiB in 0.954819 sec at 1.0 GiB/sec 268 IOPS Now cluster 1 results are terrible, about 25% from before the upgrade. ceph tell osd.0 bench -f plain bench: wrote 1 GiB in blocks of 4 MiB in 4.03434 sec at 254 MiB/sec 63 IOPS Ceph -s shows HEALTH_OK. Dashboard looks good. 2 pools MON Dump min_mon_release 14 (nautilus) OSD Dump require_min_compat_client luminous min_compat_client jewel require_osd_release nautilus Not sure what is causing the slow performance. Ideas?

3 6

Resolving a pg inconsistent Issue
by Steven Pine 14 Aug '20

14 Aug '20

Hi Ceph users, We are running a suse SES 5.5 cluster that's largely based on luminous with some mimic backports. We've been doing some large reshuffling from adding in additional OSDs and during this process we have an inconsistent pg group, investigation suggests there was a read error. We would like to target it with a deep-scrub and possibly repair attempt before marking the individual osd down and replacing the drive, but because of the weeks of reshuffling, our cluster is behind on deep-scrubs and it seemingly is refusing to scrub the marked pg, or at least it is scheduling other pg for deep-scrubs first. Is there any method we can use to grant deep-scrub priority? How about adjusting deep-scrub timeframes to be 6 months temporarily, would that allow us to force the desired pg deep-scrub to occur prior? Or could set some of the no scrub flags and proceed with a pg repair attempt? Thank you for any suggestions or advice, -- Steven Pine webair.com *P* 516.938.4100 x *E * steven.pine(a)webair.com <https://www.facebook.com/WebairInc/> <https://www.linkedin.com/company/webair>

2 1

Radosgw Multiside Sync
by Ansgar Jazdzewski 14 Aug '20

14 Aug '20

Hi Folks, i'am trying to move from our own custom bucket synchronization to the rados-gateway build in one. Multisite setup is working https://docs.ceph.com/docs/master/radosgw/multisite/ All buckes and users are visible in both clusters Next i tried to setup the multi-side-sync https://docs.ceph.com/docs/master/radosgw/multisite-sync-policy/ for the first test i liked to have a full symmetric setup so i configured it as followed radosgw-admin sync group create \ --group-id=group1 \ --status=allowed radosgw-admin sync group flow create \ --group-id=group1 \ --flow-id=flow-mirror \ --flow-type=symmetrical \ --zones=* radosgw-admin sync group pipe create \ --group-id=group1 \ --pipe-id=pipe1 \ --source-zones='*' \ --source-bucket='*' \ --dest-zones='*' \ --dest-bucket='*' radosgw-admin sync group modify \ --group-id=group1 \ --status=enabled radosgw-admin period update \ --commit but the objects are not copied around do i need to start the copy-process? can i find some debug information somewhere? Thanks, Ansgar

2 4

Re: SED drives ,*how to fio test all disks, poor performance
by Ed Kalk 14 Aug '20

14 Aug '20

screenshot attached showing the IOPs and LAtency from iostat -xtc 2 On 8/14/2020 9:09 AM, Ed Kalk wrote: > ubuntu@ubuntu:/mnt$ sudo fio > --filename=/mnt/sda1/file1.fio:/mnt/sdb1/file2.fio:/mnt/sdc1/file3.fio:/mnt/sdd1/file4.fio:/mnt/sde1/file5.fio:/mnt/sdf1/file6.fio:/mnt/sdg1/file7.fio:/mnt/sdh1/file8.fio:/mnt/sdi1/file9.fio:/mnt/sdj1/file10.fio:/mnt/sdk1/file11.fio:/mnt/sdl1/file12.fio > --size=2GB --direct=1 --rw=randrw --bs=4k --ioengine=libaio > --iodepth=256 --runtime=500 --numjobs=12 --time_based > --group_reporting --name=iops-test-job > > ^generate fio to all disks > > ----- > # iostat -xtc 2 > > ^monitor IOPs to all disks > ----- > > ^how to test “fio” to multiple disks and monitor via “iostat -xtc 2“. > I booted via live CD, made test partitions, file systems and mounted. > disks took a while to initialize and settle in after filesystem > create. monitored via iostat till all disks were quiet. then ran fio > to test. > > we observed the fist 6 disks (seagate constellation v1 SED) have a > problem, but next 6 don’t. (seagate constellation v2 non-SED) > > The “w_await” shows latency is 100x on v1’s, this was a reproduction > of how they performed in ceph. > > ***suspecting disk firmware, will try updating them. > > -Edward Kalk > > On Aug 9, 2020, at 10:44 AM, Edward kalk <ekalk(a)socket.net> wrote: > > Thank you for the reply Anthony. > Yes the R510 is of a previous gen. The H700 isn’t my top suspect > though. It’s been clocked to pull 15,000+ iops (with SSD). I’m > familiar with bottlenecking an hba with 24 disk chassis. I only have 6 > ceph OSDs on the H700. (this system should be overkill for that. > (ejected all other disks, only 8 inserted) > I’ll try investigating for FW updates on disks and hba. > will also take another look at disk health via smart. they are > repurposed disks and have seen action to say the least. thanks. > -Ed > >>>>> > >>>>>> On Aug 8, 2020, at 6:56 PM, Anthony D'Atri > <anthony.datri(a)gmail.com> wrote: > >>>>>> > >>>>>> Ugh, that looks like a pretty old HBA (and system). > >>>>>> > >>>>>> On an R730xd I found an H330 mini to be something of a > bottleneck with 24x SATA SSD. > >>>>>> > >>>>>> o Check your drives for available firmware updates > >>>>>> > >>>>>> o Check your HBA for a firmware update > >>>>>> > >>>>>> o Check that your HBA’s firmware is supported with your > kernel’s driver version — there should be a matrix at Dell somewhere > >>>>>> > >>>>>> o If you have a BBU/supercap, check its status (perccli /c0 … ) > >>>>>> > >>>>>> o Check dmesg/logs for HBA errors > >>>>>> > >>>>>> o Check smartctl -a output on the drives, look for wear, > reallocated sectors, errors > >>>>>> > >>>>>>>> On Aug 8, 2020, at 2:55 PM, Edward kalk <ekalk(a)socket.net> wrote: > >>>>>>> R0, the controller config. (faux-JBOD) > >>>>>>> Single disk group. NoN-Raid. > >>>>>>> > >>>>>>> -Ed > >>>>>>> > >>>>>>>>> On Aug 8, 2020, at 9:20 AM, Anthony D'Atri > <anthony.datri(a)gmail.com> wrote: > >>>>>>>> By R0 you mean hdparm ? > >>>>>>>> > >>>>>>>>> On Aug 8, 2020, at 3:49 AM, Edward kalk <ekalk(a)socket.net> > wrote: > >>>>>>>>> > >>>>>>>>> Im getting poor performance with 5 of my OSDs, Seagate > Constellation ES SED (1) 10k SAS 2TB 3.5 drives. > >>>>>>>>> disk write latency keeps drifting high , > >>>>>>>>> 100ms-230ms on writes. the other 30 OSDs are performing > well. avg latency 10-20ms > >>>>>>>>> > >>>>>>>>> We observe stats via “iostat -xtc 2” on CEPH server. w_await > showing writes to disk latency. > >>>>>>>>> > >>>>>>>>> **anyone else have poor latency on SEDs? > >>>>>>>>> > >>>>>>>>> The disks were included on a server we acquired. Didn’t know > they would be SEDs. > >>>>>>>>> I’ve read that seagate SED vs. non SED shouldn’t perform > differently, but looks like they are awful. All the SEDs are in the > newly added CEPH node. A Dell R510 with H700 raid card. all disks in > entire cluster are R0, direct write to disk, no cache. > >>>>>>>>> The new server and disk control spec out at about 10x the > ability and IOPs of other servers, so I suspect the SEDs. > >>>>>>>>> > >>>>>>>>> -Ed > >>>>>>>>> _______________________________________________ > >>>>>>>>> ceph-users mailing list -- ceph-users(a)ceph.io > >>>>>>>>> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >>> -- > >>> Thank you for your time, > >>> > >>> > >>> > >>> Edward H. Kalk IV > >>> Information Technology Dept. > >>> Server Specialist > >>> Datacenter Virtualization and Storage Systems > >>> Socket Telecom, LLC. > >>> 2703 Clark Lane > >>> Columbia, MO 65202 > >>> 573-817-0000 or 800-socket3 X218 > >> -- > > -- Thank you for your time, Edward H. Kalk IV Information Technology Dept. Server Specialist Datacenter Virtualization and Storage Systems Socket Telecom, LLC. 2703 Clark Lane Columbia, MO 65202 573-817-0000 or 800-socket3 X218

1 0

How to separate WAL DB and DATA using cephadm or other method?
by Popoi Zen 14 Aug '20

14 Aug '20

Hi, I just want to know if it possible to separate my NVME disk and HDD disk roles on cephadm. Example I have an NVME paritition: /dev/nvme0n0 and HDD: /dev/sdb. How to pair that 2 disk, so I can use /dev/sdb as OSD and the whole /dev/nvme0n0 as WAL DB?

1 0

RGW Lifecycle Processing and Promote Master Process
by Alex Hussein-Kershaw 14 Aug '20

14 Aug '20

Hi, I've previously discussed some issues I've had with the RGW lifecycle processing. I've discovered that the root cause of my problem is that: * I'm running a multisite configuration * Life cycle processing is done on the master site each night. `radosgw-admin lc list` correctly returns all buckets with lc config. * I simulate the master site being destroyed from my VM host. * I promote the secondary site to master following the instructions here: https://docs.ceph.com/docs/master/radosgw/multisite/ * The new master site isn't doing any lifecycle processing. `radosgw-admin lc list` returns empty. * I recreate a cluster and pair it with the new master site to get back to having multisite redundancy. * Neither site is doing any lifecycle processing. `radosgw-admin lc list` returns empty. So in the process of failover/recovery I have gone from having two paired clusters performing lifecycle processing, to two paired clusters NOT performing lifecycle processing. Is this behaviour expected? I've found `radosgw-admin lc reshard fix` will "remind" the cluster that I run it on that it needs to do lifecycle processing. Although I found no mention of having to use this in the docs, for that command the docs state it's only relevant on earlier Ceph versions. I'm running Nautilus 14.2.9. In addition, if I have two healthy clusters paired in a multisite system, and swap the master cluster by promoting the non-master, the demoted cluster seems to still continue doing lifecycle processing, while the promote does not. If I run `radosgw-admin lc reshard fix` on the promoted cluster, then both clusters seem to claim they are doing the processing. Is this a happy state to be in? Does anyone have any experience with this? Thanks, Alex

1 0

Single node all-in-one install for testing
by Richard W.M. Jones 14 Aug '20

14 Aug '20

I have one spare machine with a single 1TB on it, and I'd like to test a local Ceph install. This is just for testing, I don't care that it won't have redundancy, failover, etc. Is there any canonical documentation for this case? - - - Longer story is this morning I found this documentation through many Google searches: https://medium.com/@balderscape/setting-up-a-virtual-single-node-ceph-stora… It kind of works up to a point. I was able to get Ceph installed, get both the command line and web interfaces up and running, and got as far as: # ceph status cluster: id: 723e09aa-dbd3-11ea-8587-94c691189836 health: HEALTH_WARN Reduced data availability: 1 pg inactive OSD count 0 < osd_pool_default_size 3 services: mon: 1 daemons, quorum dev1 (age 77m) mgr: dev1.ziibhq(active, since 76m) osd: 0 osds: 0 up, 0 in data: pools: 1 pools, 1 pgs objects: 0 objects, 0 B usage: 0 B used, 0 B / 0 B avail pgs: 100.000% pgs unknown 1 unknown # ceph orch device ls HOST PATH TYPE SIZE DEVICE AVAIL REJECT REASONS dev1 /dev/sdb hdd 14.9G Cruzer_Blade_4C530210050318117591 True dev1 /dev/sda hdd 931G WDC_WD10JFCX-68N_WD-WXD1AB73AUX5 False LVM detected, locked (The USB stick is left over from the install and isn't suitable as an OSD). However I've been completely unable to add any OSDs using the local disk. The disk (/dev/sda3) is a PV and there is plenty of space. I even created some devices that I was hoping to use as OSDs: # lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert osd1 rhel_dev1 -wi-a----- 100.00g osd2 rhel_dev1 -wi-a----- 100.00g osd3 rhel_dev1 -wi-a----- 100.00g root rhel_dev1 -wi-ao---- 50.00g swap rhel_dev1 -wi-ao---- <7.84g but it simply doesn't do anything when I try to add them: # ceph orch daemon add osd localhost:rhel_dev1/osd1 (no error, exit code 0) So I guess the fact that the device is marked as "AVAIL = False" and "locked" is bad somehow. Can I add the OSDs anyway? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html

2 1

Ceph Tech Talk: A Different Scale – Running small ceph clusters in multiple data centers by Yuval Freund
by Mike Perez 13 Aug '20

13 Aug '20

Hi everyone, Get ready for July 23rd at 17:00 UTC another Ceph Tech Talk but a different scale: running small Ceph clusters in multiple data centers. Thanks to Yuval Freund for giving time to provide us content this month. You can find the calendar invite and archive here: https://ceph.io/ceph-tech-talks/ -- Mike Perez He/Him Ceph Community Manager Red Hat Los Angeles <https://www.redhat.com> thingee(a)redhat.com <mailto:thingee@redhat.com> M: 1-951-572-2633 <tel:1-951-572-2633> IM: IRC Freenode/OFTC: thingee 494C 5D25 2968 D361 65FB 3829 94BC D781 ADA8 8AEA @Thingee <https://twitter.com/thingee> <https://www.redhat.com> <https://redhat.com/summit>

1 1