- ceph-users - lists.ceph.io

Troubleshooting stuck unclean PGs?
by Matt Larson 22 Sep '20

22 Sep '20

Hi, Our Ceph cluster is reporting several PGs that have not been scrubbed or deep scrubbed in time. It is over a week for these PGs to have been scrubbed. When I checked the `ceph health detail`, there are 29 pgs not deep-scrubbed in time and 22 pgs not scrubbed in time. I tried to manually start a scrub on the PGs, but it appears that they are actually in an unclean state that needs to be resolved first. This is a cluster running: ceph version 15.2.1 (9fd2f65f91d9246fae2c841a6222d34d121680ee) octopus (stable) Following the information at [Troubleshooting PGs](https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-…, I checked for PGs that are stuck stale | inactive | unclean. There were no PGs that are stale or inactive, but there are several that are stuck unclean: ``` PG_STAT STATE UP UP_PRIMARY ACTING ACTING_PRIMARY 8.3c active+remapped+backfill_wait [124,41,108,8,87,16,79,157,49] 124 [139,57,16,125,154,65,109,86,45] 139 8.3e active+remapped+backfill_wait [108,2,58,146,130,29,37,66,118] 108 [127,92,24,50,33,6,130,66,149] 127 8.3f active+remapped+backfill_wait [19,34,86,132,59,78,153,99,6] 19 [90,45,147,4,105,61,30,66,125] 90 8.40 active+remapped+backfill_wait [19,131,80,76,42,101,61,3,144] 19 [28,106,132,3,151,36,65,60,83] 28 8.3a active+remapped+backfilling [32,72,151,30,103,131,62,84,120] 32 [91,60,7,133,101,117,78,20,158] 91 8.7e active+remapped+backfill_wait [108,2,58,146,130,29,37,66,118] 108 [127,92,24,50,33,6,130,66,149] 127 8.3b active+remapped+backfill_wait [34,113,148,63,18,95,70,129,13] 34 [66,17,132,90,14,52,101,47,115] 66 8.7f active+remapped+backfill_wait [19,34,86,132,59,78,153,99,6] 19 [90,45,147,4,105,61,30,66,125] 90 8.78 active+remapped+backfill_wait [96,113,159,63,29,133,73,8,89] 96 [138,121,15,103,55,41,146,69,18] 138 8.7d active+remapped+backfilling [0,90,60,124,159,19,71,101,135] 0 [150,72,124,129,63,10,94,29,41] 150 8.7c active+remapped+backfill_wait [124,41,108,8,87,16,79,157,49] 124 [139,57,16,125,154,65,109,86,45] 139 8.79 active+remapped+backfill_wait [59,15,41,82,131,20,73,156,113] 59 [13,51,120,102,29,149,42,79,132] 13 ``` If I query one of the PGs that is backfilling, 8.3a, it shows it's state as : "recovery_state": [ { "name": "Started/Primary/Active", "enter_time": "2020-09-19T20:45:44.027759+0000", "might_have_unfound": [], "recovery_progress": { "backfill_targets": [ "30(3)", "32(0)", "62(6)", "72(1)", "84(7)", "103(4)", "120(8)", "131(5)", "151(2)" ], Q1: Is there anything that I should check/fix to enable the PGs to resolve from the `unclean` state? Q2: I have also seen that the podman containers on one of our OSD servers are taking large amounts of disk space. Is there a way to limit the growth of disk space for podman containers, when administering a Ceph cluster using `cephadm` tools? At last check, a server running 16 OSDs and 1 MON is using 39G of disk space for its running containers. Can restarting containers help to start with a fresh slate or reduce the disk use? Thanks, Matt ------------------------ Matt Larson Associate Scientist Computer Scientist/System Administrator UW-Madison Cryo-EM Research Center 433 Babcock Drive, Madison, WI 53706

2 1

What is the advice, one disk per OSD, or multiple disks
by Kees Bakker 21 Sep '20

21 Sep '20

Hello, Being new to CEPH, I need some advice how to setup a cluster. Given a node that has multiple disks, should I create one OSD for all disks, or is it better to have one OSD per disk. -- Kees Bakker

3 2

Understanding what ceph-volume does, with bootstrap-osd/ceph.keyring, tmpfs
by Marc Roos 21 Sep '20

21 Sep '20

When I create a new encrypted osd with ceph volume[1] I assume something like this is being done, please correct what is wrong. - it creates the pv on the block device - it creates the ceph vg on the block device - it creates the osd lv in the vg - it uses cryptsetup to encrypt this lv (or is there some internal support for luks in lvm?) - it sets all the tags on the vg (shown by: lvs -o lv_tags vg) - it creates and enables ceph-volume@lvm-osdid-osdfsid - it creates and enables ceph-osd@osdid When a node is restarted, these lvm osds are started with - running ceph-volume@lvm-osdid-osdfsid (creating this tmpfs mount?) - running ceph-osd@osdid Q1: I had to create bootstrap-osd/ceph.keyring (ownership root.root). For what is that being used? Does it need to exist upon node restart? Q2: I had some issues with a node starting, solving this with adding a nofail to the fstab. How is this done with ceph-volume? Q3: Why these strange permissions on the mounted folder? drwxrwxrwt 2 ceph ceph 340 Sep 19 15:24 ceph-40 Q4: Where is this luks passphrase stored? Q5: Where does this tmpfs+content come from? How can I mount this myself from the command line? Q6: My lvm tags show ceph.crush_device_class=None, while ceph osd tree shows the correct class. Is this correct? Q7: I saw in my ceph-volume output sometimes 'disabling cephx', what does this mean? How can I verify this and fix it? Links to manuals are also welcome, these ceph-volume[2] are not to clear about this. [1] ceph-volume lvm create --data /dev/sdk --dmcrypt [2] https://docs.ceph.com/en/latest/ceph-volume/lvm/activate/

1 0

ceph 14.2.8 tracing ceph with blking compile error
by 陈晓波 21 Sep '20

21 Sep '20

I want to trace a request in ceph processing in 14.2.8 version. I open the blkin with do_cmake.sh -DWITH_BLKIN=ON, when compile there is a error in below: ../lib/libblkin.a(tp.c.o): undefined reference to symbol 'lttng_probe_register' //lib64/liblttng-ust.so.0: error adding symbols: DSO missing from command line collect2: error: ld returned 1 exit status make[2]: *** [src/CMakeFiles/ceph-osd.dir/build.make:130: bin/ceph-osd] Error 1 make[1]: *** Waiting for unfinished jobs.......

1 0

Record recuperation mistake prompting Unlock Yahoo Account? Contact help group.
by mary smith 21 Sep '20

21 Sep '20

The record recuperation mistake can make a disappointment Unlock Yahoo Account. This issue can be settled by utilizing the assist that with canning be found in the tech consultancies. What's more, you can generally visit Youtube and take a gander at some tech vids that will help you in managing the issue. https://www.customercare-email.com/blog/unlock-my-yahoo-account/

1 0

Issue in the Epson interface? Find uphold from Epson Printer Support.
by mary smith 21 Sep '20

21 Sep '20

In the event that you face an interface issue in your gadget, by then you should reboot it. Regardless, on the off chance that the issue doesn't get unraveled, by then you can take help from the tech consultancies and utilize their investigating procedures. You can also contact the Epson Printer Support for finding support. https://www.epsonprintersupportpro.net/

1 0

Is ceph-mon disk write i/o normal at more than 1/2TB a day on an empty cluster?
by tri＠postix.net 20 Sep '20

20 Sep '20

Hi all, I'm in the process of testing out ceph on a small cluster. The cluster is virtually empty with no clients, just a few OSDs. I noticed extensive disk write I/O on some nodes and tracked them down to the ceph-mon daemon. A cursory log over 1 hour shows that each monitor generates about 22GB worth of disk write I/O or more than 1/2TB a day. The log was generated using iotop. It seems the 3 monitors in the cluster are doing the same thing. Is this normal or something I should be taking a closer look at? Cheers, --Tri Hoang

1 0

Process for adding a separate block.db to an osd
by tri＠postix.net 20 Sep '20

20 Sep '20

Hey all, I'm trying to figure out the appropriate process for adding a separate SSD block.db to an existing OSD. From what I gather the two steps are: 1. Use ceph-bluestore-tool bluefs-bdev-new-db to add the new db device 2. Migrate the data ceph-bluestore-tool bluefs-bdev-migrate I followed this and got both executed fine without any error. Yet when the OSD got started up, it keeps on using the integrated block.db instead of the new db. The block.db link to the new db device was deleted. Again, no error, just not using the new db Any suggestion? Thanks. --Tri Hoang root@elmo:/#CEPH_ARGS="--bluestore_block_db_size=26843545600 --bluestore_block_db_create=true" ceph-bluestore-tool --path /mnt/ceph/c258000c-f3e4-11ea-9ebe-c3c75e8e9028/osd.2 bluefs-bdev-new-db --dev-target /dev/vg/sdc.db inferring bluefs devices from bluestore path DB device added /dev/dm-8 root@elmo:/# ceph-bluestore-tool --path /mnt/ceph/c258000c-f3e4-11ea-9ebe-c3c75e8e9028/osd.2 --devs-source /mnt/ceph/c258000c-f3e4-11ea-9ebe-c3c75e8e9028/osd.2/block --dev-target /mnt/ceph/c258000c-f3e4-11ea-9ebe-c3c75e8e9028/osd.2/block.db bluefs-bdev-migrate inferring bluefs devices from bluestore path

2 2

Re: Benchmark WAL/DB on SSD and HDD for RGW RBD CephFS
by Maged Mokhtar 20 Sep '20

20 Sep '20

https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/dri… On 19/09/2020 10:31, huxiaoyu(a)horebdata.cn wrote: > Dear Maged, > > Thanks a lot for detailed explanantion on dm-writecache with Ceph. > > You mentioned REQ_FUA support patch for dm-writecache, does such a > patch not included into recent dm-writecache source code? I am using > 4.4 and 4.15/4.19 kernels, where do i get the mentioned patch? > > best regards, > > Samuel > > > > ------------------------------------------------------------------------ > huxiaoyu(a)horebdata.cn > > *From:* Maged Mokhtar <mailto:mmokhtar@petasan.org> > *Date:* 2020-09-18 18:20 > *To:* vitalif <mailto:vitalif@yourcmc.ru>; huxiaoyu > <mailto:huxiaoyu@horebdata.cn>; ceph-users <mailto:ceph-users@ceph.io> > *Subject:* Re: [ceph-users] Re: Benchmark WAL/DB on SSD and HDD > for RGW RBD CephFS > dm-writecache works using a high and low watermarks, set at 45 and > 50%. > All writes land in cache, once cache fills to the high watermark > backfilling to the slow device starts and stops when reaching the low > watermark. Backfilling uses b-tree with LRU blocks and tries merge > blocks to reduce hdd seeks, this is further helped by the io > scheduler > (cfq/deadline) ordering. > Each sync write op to the device requires 2 sync write ops, one > for data > and one for metadata, metadata is always in ram so there is no > additional metada read op (at the expense of using 2.5% of your cache > partition size in ram). So for pure sync writes (those with > REQ_FUA or > REQ_FLUSH which is used by Ceph) get half the SSD iops performance at > the device level. > Now the questions, what sustained performance would you get during > backfilling: it totally depends on whether your workload is > sequential > or random. For pure sequential workloads, all blocks are merged so > there > will not be a drop in input iops and backfilling occurs in small step > like intervals, but for such workloads you could get good performance > even without a cache. For purely random writes theoretically you > should > drop to the hdd random iops speed ( ie 80-150 iops ), but in our > testing > with fio pure random we would get 400-450 sustained iops, this is > probably related to the non-random-ness of fio rather than any magic. > For real life workloads that have a mix of both, this is where the > real > benefit of the cache will be felt, however it is not easy to simulate > such workloads, fio does offer a zipf/theta random distribution > control > but it was difficult for us to simulate real life workloads with > it, we > did some manual workloads such as installing and copying multiple vms > and we found the cache helped by 3-4 times the time to complete. > dm-writecache does serve reads if in cache, however the OSD cache > does > help for reads as well as any client read-ahead and in general writes > are the performance issue with hdd in Ceph. > For bcache, the only configuration we did was to enable write back > mode, > we did not set the block size to 4k. > If you want to try dm-writecache, use a recent 5.4+ kernel or a > kernel > with REQ_FUA support patch we did. You would need a recent lvm tools > package to support dm-writecache. We also limit the number of > backfill > blocks inflight to 100k blocks ie 400 MB. > /Maged > On 18/09/2020 13:38, vitalif(a)yourcmc.ru wrote: > >> we did test dm-cache, bcache and dm-writecache, we found the > later to be > >> much better. > > Did you set bcache block size to 4096 during your tests? Without > this setting it's slow because 99.9% SSDs don't handle 512 byte > overwrites well. Otherwise I don't think bcache should be worse > than dm-writecache. Also dm-writecache only caches writes, and > bcache also caches reads. And lvmcache is trash because it only > writes to SSD when the block is already on the SSD. > > > > Please post some details about the comparison if you have them :) >

1 0

Cephadm adoption not properly working
by Julian Fölsch 20 Sep '20

20 Sep '20

Good day everybody, We recently upgraded from Nautilus to Octopus and tried to adopt our cluster to cephadm following the guide in the documentation[0]. We only experienced some at first glance minor hickups where the cluster failed to refresh because it could not pull the container image[1]. This was fixed by setting `container_image` to the location where it could find the image in our network. But when we tried to run `ceph orch ps` it returned only a empty list. The one OSD we adopted after fixing the container_image variable was appearing fine in `ceph orch ps` then. However all Manager and Monitor daemons are now in a floating state where `cephadm ls` reports them as style "cephadm:v1"[2] but the orchestrator considers them stray[3]. Functionally however, those daemons just work fine and work together in the cluster. Can you help out how to get those daemons under the orchestrators control? Kind regards, Julian Fölsch [0] https://docs.ceph.com/en/octopus/cephadm/adoption/ [1] https://agdsn.me/~paktosan/ceph/adoption/image_failure.txt [2] https://agdsn.me/~paktosan/ceph/adoption/cephadm_ls.txt [3] https://agdsn.me/~paktosan/ceph/adoption/stray_daemons.txt -- Julian Fölsch Arbeitsgemeinschaft Dresdner Studentennetz (AG DSN) Telefon: +49 351 271816 69 Mobil: +49 152 22915871 Fax: +49 351 46469685 Email: julian.foelsch(a)agdsn.de Studierendenrat der TU Dresden Helmholtzstr. 10 01069 Dresden

1 0