- ceph-users - lists.ceph.io

Ceph RBD latency with synchronous writes?
by René Bartsch 23 Sep '20

23 Sep '20

Hi, we're considering running KVM virtual machine images on Ceph RBD block devices. How does Ceph RBD perform with the synchronous writes of databases (MariaDB)? Best regards, Renne

2 1

Vitastor, a fast Ceph-like block storage for VMs
by vitalif＠yourcmc.ru 23 Sep '20

23 Sep '20

Hi! After almost a year of development in my spare time I present my own software-defined block storage system: Vitastor - https://vitastor.io I designed it similar to Ceph in many ways, it also has Pools, PGs, OSDs, different coding schemes, rebalancing and so on. However it's much simpler and much faster. In a test cluster with SATA SSDs it achieved Q1T1 latency of 0.14ms which is especially great compared to Ceph RBD's 1ms for writes and 0.57ms for reads. In an "iops saturation" parallel load benchmark it reached 895k read / 162k write iops, compared to Ceph's 480k / 100k on the same hardware, but the most interesting part was CPU usage: Ceph OSDs were using 40 CPU cores out of 64 on each node and Vitastor was only using 4. Of course it's an early pre-release which means that, for example, it lacks snapshot support and other useful features. However the base is finished - it works and runs QEMU VMs. I like the design and I plan to develop it further. There are more details in the README file which currently opens from the domain https://vitastor.io Sorry if it was a bit off-topic, I just thought it could be interesting for you :) -- With best regards, Vitaliy Filippov

6 9

Update erasure code profile
by Thomas Svedberg 23 Sep '20

23 Sep '20

Hi, We are running a Nautilus cluster and have some old and new erasure code profiles. For example: # ceph osd erasure-code-profile get m_erasure crush-device-class=hdd crush-failure-domain=host crush-root=default jerasure-per-chunk-alignment=false k=6 m=2 plugin=jerasure technique=reed_sol_van w=8 # ceph osd erasure-code-profile get c_erasure crush-device-class= crush-failure-domain=host crush-root=default jerasure-per-chunk-alignment=false k=4 m=2 plugin=jerasure technique=reed_sol_van w=8 Here we want to add crush-device-class information to the second profile. Is the following correct? Safe? What will actually happen if we run this command? # ceph osd erasure-code-profile set c_erasure crush-device-class=hdd k=4 m=2 --force Thanks for any input! Regards, Thomas Svedberg

2 1

Documentation broken
by Frank Schilder 23 Sep '20

23 Sep '20

Hi all, during the migration of documentation, would it be possible to make the old documentation available somehow? A lot of pages are broken and I can't access the documentation for mimic at all any more. Is there an archive or something similar? Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 2

switching to ceph-volume requires changing the default lvm.conf?
by Marc Roos 23 Sep '20

23 Sep '20

I was wondering if switching to ceph-volume requires me to change the default centos lvm.conf? Eg. The default has issue_discards = 0 Also I wonder if trimming is the default on lvm's on ssds? I read somewhere that the dmcrypt passthrough of trimming was still secure in combination with a btrfs filesystem.

1 0

Re: Troubleshooting stuck unclean PGs?
by Matt Larson 23 Sep '20

23 Sep '20

Hi Wout, None of the OSDs are greater than 20% full. However, only 1 PG is backfilling at a time, while the others are backfill_wait. I had recently added a large amount of data to the Ceph cluster, and this may have caused the # of PGs to increase causing the need to rebalance or move objects. It appears that I could increase the # of backfill operations that happen simultaneously by increasing `osd_max_backfills` and/or `osd_recovery_max_active`. It looks like I should maybe consider increasing the number of max backfills happening at a time because the overall io during the backfill is pretty small. Does this seem reasonable? If so, with Ceph Octopus/cephadm, how can adjust the parameters? Thanks, Matt On Mon, Sep 21, 2020 at 2:21 PM Wout van Heeswijk <wout(a)42on.com> wrote: > > Hi Matt, > > The mon data can grow during when PGs are stuck unclean. Don't restart the mons. > > You need to find out why your placement groups are "backfill_wait". Likely some of your OSDs are (near)full. > > If you have space elsewhere you can use the ceph balancer module or reweighting of OSDs to rebalance data. > > Scrubbing will continue once the PGs are "active+clean" > > Kind regards, > > Wout > 42on > > ________________________________________ > From: Matt Larson <larsonmattr(a)gmail.com> > Sent: Monday, September 21, 2020 6:22 PM > To: ceph-users(a)ceph.io > Subject: [ceph-users] Troubleshooting stuck unclean PGs? > > Hi, > > Our Ceph cluster is reporting several PGs that have not been scrubbed > or deep scrubbed in time. It is over a week for these PGs to have been > scrubbed. When I checked the `ceph health detail`, there are 29 pgs > not deep-scrubbed in time and 22 pgs not scrubbed in time. I tried to > manually start a scrub on the PGs, but it appears that they are > actually in an unclean state that needs to be resolved first. > > This is a cluster running: > ceph version 15.2.1 (9fd2f65f91d9246fae2c841a6222d34d121680ee) octopus (stable) > > Following the information at [Troubleshooting > PGs](https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-…, > I checked for PGs that are stuck stale | inactive | unclean. There > were no PGs that are stale or inactive, but there are several that are > stuck unclean: > > ``` > PG_STAT STATE UP > UP_PRIMARY ACTING ACTING_PRIMARY > 8.3c active+remapped+backfill_wait > [124,41,108,8,87,16,79,157,49] 124 > [139,57,16,125,154,65,109,86,45] 139 > 8.3e active+remapped+backfill_wait > [108,2,58,146,130,29,37,66,118] 108 > [127,92,24,50,33,6,130,66,149] 127 > 8.3f active+remapped+backfill_wait > [19,34,86,132,59,78,153,99,6] 19 > [90,45,147,4,105,61,30,66,125] 90 > 8.40 active+remapped+backfill_wait > [19,131,80,76,42,101,61,3,144] 19 > [28,106,132,3,151,36,65,60,83] 28 > 8.3a active+remapped+backfilling > [32,72,151,30,103,131,62,84,120] 32 > [91,60,7,133,101,117,78,20,158] 91 > 8.7e active+remapped+backfill_wait > [108,2,58,146,130,29,37,66,118] 108 > [127,92,24,50,33,6,130,66,149] 127 > 8.3b active+remapped+backfill_wait > [34,113,148,63,18,95,70,129,13] 34 > [66,17,132,90,14,52,101,47,115] 66 > 8.7f active+remapped+backfill_wait > [19,34,86,132,59,78,153,99,6] 19 > [90,45,147,4,105,61,30,66,125] 90 > 8.78 active+remapped+backfill_wait > [96,113,159,63,29,133,73,8,89] 96 > [138,121,15,103,55,41,146,69,18] 138 > 8.7d active+remapped+backfilling > [0,90,60,124,159,19,71,101,135] 0 > [150,72,124,129,63,10,94,29,41] 150 > 8.7c active+remapped+backfill_wait > [124,41,108,8,87,16,79,157,49] 124 > [139,57,16,125,154,65,109,86,45] 139 > 8.79 active+remapped+backfill_wait > [59,15,41,82,131,20,73,156,113] 59 > [13,51,120,102,29,149,42,79,132] 13 > ``` > > If I query one of the PGs that is backfilling, 8.3a, it shows it's state as : > "recovery_state": [ > { > "name": "Started/Primary/Active", > "enter_time": "2020-09-19T20:45:44.027759+0000", > "might_have_unfound": [], > "recovery_progress": { > "backfill_targets": [ > "30(3)", > "32(0)", > "62(6)", > "72(1)", > "84(7)", > "103(4)", > "120(8)", > "131(5)", > "151(2)" > ], > > Q1: Is there anything that I should check/fix to enable the PGs to > resolve from the `unclean` state? > Q2: I have also seen that the podman containers on one of our OSD > servers are taking large amounts of disk space. Is there a way to > limit the growth of disk space for podman containers, when > administering a Ceph cluster using `cephadm` tools? At last check, a > server running 16 OSDs and 1 MON is using 39G of disk space for its > running containers. Can restarting containers help to start with a > fresh slate or reduce the disk use? > > Thanks, > Matt > > ------------------------ > > Matt Larson > Associate Scientist > Computer Scientist/System Administrator > UW-Madison Cryo-EM Research Center > 433 Babcock Drive, Madison, WI 53706 > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io -- Matt Larson, PhD Madison, WI 53705 U.S.A.

4 7

How HP Printer Support help for troubleshooting?
by mary smith 23 Sep '20

23 Sep '20

To troubleshoot with the hp printer issue, you are required to download the HP Print and Scan Doctor from its official website. After that, you have to run HPPdr.exe from the download location on your system. And then, hit on the Start link and select your printer accordingly. Now, click fix printing. If there is an issue, use HP Printer Support. https://www.amiytech.com/hp-printer-support/

1 0

ceph-volume lvm cannot zap???
by Marc Roos 23 Sep '20

23 Sep '20

[@~]# ceph-volume lvm zap /dev/sdi --> Zapping: /dev/sdi --> --destroy was not specified, but zapping a whole device will remove the partition table stderr: wipefs: error: /dev/sdi: probing initialization failed: Device or resource busy --> failed to wipefs device, will try again to workaround probable race condition stderr: wipefs: error: /dev/sdi: probing initialization failed: Device or resource busy --> failed to wipefs device, will try again to workaround probable race condition I can see where it is busy, at least not in lsof

3 3

[nautilus] ceph tell hanging
by Nico Schottelius 22 Sep '20

22 Sep '20

Hello, recently we wanted to re-adjust rebalancing speed in one cluster with ceph tell osd.* injectargs '--osd-max-backfills 4' ceph tell osd.* injectargs '--osd-recovery-max-active 4' The first osds responded and after about 6-7 osds ceph tell stopped progressing, just after it encountered a dead osd (osd.10). We have since then removed osd.10 and all osds in the cluster are up. However as soon as we issue either of the above tell commands, it just hangs. Furthermore when ceph tell hangs, pg are also becoming stuck in "Activating" and "Peering" states. It seems to be related, as soon as we stop ceph tell (ctrl-c it), a few minutes later the pgs are peered/active. We can reproduce this problem also with very busy osds, which have been moved to another host - they also do not react to the ceph tell commands. We are mostly on 14.2.9, besides the rgw: [16:44:47] black2.place6:~# ceph versions { "mon": { "ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)": 3 }, "mgr": { "ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)": 3 }, "osd": { "ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)": 85 }, "mds": {}, "rgw": { "ceph version 20200428-923-g4004f081ec (4004f081ec047d60e84d76c2dad6f31e2ac44484) nautilus (stable)": 1 }, "overall": { "ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)": 91, "ceph version 20200428-923-g4004f081ec (4004f081ec047d60e84d76c2dad6f31e2ac44484) nautilus (stable)": 1 } } Did anyone see this before and/or do you have a hint on how to debug ceph tell as it is not a daemon on its own? Best regards, Nico -- Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch

2 6

Unknown PGs after osd move
by Nico Schottelius 22 Sep '20

22 Sep '20

Hello, after having moved 4 ssds to another host (+ the ceph tell hanging issue - see previous mail), we ran into 241 unknown pgs: cluster: id: 1ccd84f6-e362-4c50-9ffe-59436745e445 health: HEALTH_WARN noscrub flag(s) set 2 nearfull osd(s) 1 pool(s) nearfull Reduced data availability: 241 pgs inactive 1532 slow requests are blocked > 32 sec 789 slow ops, oldest one blocked for 1949 sec, daemons [osd.12,osd.14,osd.2,osd.20,osd.23,osd.25,osd.3,osd.33,osd.35,osd.50]... have slow ops. services: mon: 3 daemons, quorum black1,black2,black3 (age 97m) mgr: black2(active, since 96m), standbys: black1, black3 osd: 85 osds: 85 up, 82 in; 118 remapped pgs flags noscrub rgw: 1 daemon active (admin) data: pools: 12 pools, 3000 pgs objects: 33.96M objects, 129 TiB usage: 388 TiB used, 159 TiB / 548 TiB avail pgs: 8.033% pgs unknown 409151/101874117 objects misplaced (0.402%) 2634 active+clean 241 unknown 107 active+remapped+backfill_wait 11 active+remapped+backfilling 7 active+clean+scrubbing+deep io: client: 91 MiB/s rd, 28 MiB/s wr, 1.76k op/s rd, 686 op/s wr recovery: 67 MiB/s, 17 objects/s This used to be around 700+ unknown, however these 241 are stuck in this state for more than 1h. Below is a sample of pgs from "ceph pg dump all | grep unknown" 2.7f7 0 0 0 0 0 0 0 0 0 0 unknown 2020-09-22 19:03:00.694873 0'0 0:0 [] -1 [] -1 0'0 2020-09-22 19:03:00.694873 0'0 2020-09-22 19:03:00.694873 0 2.7c7 0 0 0 0 0 0 0 0 0 0 unknown 2020-09-22 19:03:00.694873 0'0 0:0 [] -1 [] -1 0'0 2020-09-22 19:03:00.694873 0'0 2020-09-22 19:03:00.694873 0 2.7c2 0 0 0 0 0 0 0 0 0 0 unknown 2020-09-22 19:03:00.694873 0'0 0:0 [] -1 [] -1 0'0 2020-09-22 19:03:00.694873 0'0 2020-09-22 19:03:00.694873 0 2.7ab 0 0 0 0 0 0 0 0 0 0 unknown 2020-09-22 19:03:00.694873 0'0 0:0 [] -1 [] -1 0'0 2020-09-22 19:03:00.694873 0'0 2020-09-22 19:03:00.694873 0 2.78b 0 0 0 0 0 0 0 0 0 0 unknown 2020-09-22 19:03:00.694873 0'0 0:0 [] -1 [] -1 0'0 2020-09-22 19:03:00.694873 0'0 2020-09-22 19:03:00.694873 0 2.788 0 0 0 0 0 0 0 0 0 0 unknown 2020-09-22 19:03:00.694873 0'0 0:0 [] -1 [] -1 0'0 2020-09-22 19:03:00.694873 0'0 2020-09-22 19:03:00.694873 0 2.76e 0 Using ceph pg 2.7f7 query hangs. We checked and one server did have an incorrect MTU setting (9204 instead of the correct 9000), but that was fixed some hours ago. Does anyone have a hint on how to find those unknown osds? Version wise this is 14.2.9: [20:42:20] black2.place6:~# ceph versions { "mon": { "ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)": 3 }, "mgr": { "ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)": 3 }, "osd": { "ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)": 85 }, "mds": {}, "rgw": { "ceph version 20200428-923-g4004f081ec (4004f081ec047d60e84d76c2dad6f31e2ac44484) nautilus (stable)": 1 }, "overall": { "ceph version 14.2.9 (581f22da52345dba46ee232b73b990f06029a2a0) nautilus (stable)": 91, "ceph version 20200428-923-g4004f081ec (4004f081ec047d60e84d76c2dad6f31e2ac44484) nautilus (stable)": 1 } } From ceph health detail: [20:42:58] black2.place6:~# ceph health detail HEALTH_WARN noscrub flag(s) set; 2 nearfull osd(s); 1 pool(s) nearfull; Reduced data availability: 241 pgs inactive; 1575 slow requests are blocked > 32 sec; 751 slow ops, oldest one blocked for 1986 sec, daemons [osd.12,osd.14,osd.2,osd.20,osd.23,osd.25,osd.3,osd.31,osd.33,osd.35]... have slow ops. OSDMAP_FLAGS noscrub flag(s) set OSD_NEARFULL 2 nearfull osd(s) osd.36 is near full osd.54 is near full POOL_NEARFULL 1 pool(s) nearfull pool 'ssd' is nearfull PG_AVAILABILITY Reduced data availability: 241 pgs inactive pg 2.82 is stuck inactive for 6027.042489, current state unknown, last acting [] pg 2.88 is stuck inactive for 6027.042489, current state unknown, last acting [] ... pg 19.6e is stuck inactive for 6027.042489, current state unknown, last acting [] pg 20.69 is stuck inactive for 6027.042489, current state unknown, last acting [] As can be seen, multiple pools are affected even though most missing pgs are from pool 2. Best regards, Nico -- Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch

3 10