- ceph-users - lists.ceph.io

OSD crash on Onode::put
by Dongdong Tao 08 Jan '23

08 Jan '23

-- Resending this mail, it seems ceph-users(a)ceph.io was down for the last few days. I see many users recently reporting that they have been struggling with this Onode::put race condition issue[1] on both the latest Octopus and pacific. Igor opened a PR [2] to address this issue, I've been reviewing it for a while and looks good to me. I'm hoping this could get some priority from the community. For those who had been hitting this issue, I would like to share a workaround that could very likely unblock you: During the investigation of this issue, I found this race condition always happens after the bluestore onode cache size becomes 0. Setting debug_bluestore = 1/30 will allow you to see the cache size after the crash: --- 2022-10-25T00:47:26.562+0000 7f424f78e700 30 bluestore.MempoolThread(0x564a9dae2a68) _resize_shards max_shard_onodes: 0 max_shard_buffer: 8388608 --- This is apparently wrong as this means the bluestore metadata cache is basically disabled, but it makes much sense to explain why we are hitting the race condition so easily -- An onode will be trimmed right away after it's unpinned. Keep going with the investigation, it turns out the culprit for the 0-sized cache is the leak that happened in bluestore_cache_other mempool Please refer to the bug tracker [3] which has the detail of the leak issue, it was already fixed by [4], and the next Pacific point release will have it. But it was never backported to Octopus, so if you are hitting the same: For Octopus, you can manually backport this patch to fix the leak and prevent the race condition from happening. For Pacific, you can wait for 16.2.11 (or manually backport the fix as well if can't wait). By the way, I'm backporting the fix to ubuntu Octopus and Pacific through this SRU [5], so it will be landed in ubuntu's package soon. [1] https://tracker.ceph.com/issues/56382 [2] https://github.com/ceph/ceph/pull/47702 [3] https://tracker.ceph.com/issues/56424 [4] https://github.com/ceph/ceph/pull/46911 [5] https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1996010 Cheers, Dongdong

1 0

ceph-users list archive missing almost all mail
by Matthias Ferdinand 08 Jan '23

08 Jan '23

Hi, I found some mailing list archive links from my notes to throw "Page not found" errors, e.g. https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/J4U24YRJEJ… Looking around in the archive web interface, it appears only some of the most recent threads are found, everything else says "no email threads could be found for this month". Could somebody please look into this? Regards Matthias Ferdinand

1 0

Serious cluster issue - data inaccessible
by Deep Dish 08 Jan '23

08 Jan '23

Hello. I really screwed up my ceph cluster. Hoping to get data off it so I can rebuild it. In summary, too many changes too quickly caused the cluster to develop incomplete pgs. Some PGS were reporting that OSDs were to be probes. I've created those OSD IDs (empty), however this wouldn't clear incompletes. Incompletes are part of EC pools. Running 17.2.5. This is the overall state: cluster: id: 49057622-69fc-11ed-b46e-d5acdedaae33 health: HEALTH_WARN Failed to apply 1 service(s): osd.dashboard-admin-1669078094056 1 hosts fail cephadm check cephadm background work is paused Reduced data availability: 28 pgs inactive, 28 pgs incomplete Degraded data redundancy: 55 pgs undersized 2 slow ops, oldest one blocked for 4449 sec, daemons [osd.25,osd.50,osd.51] have slow ops. These are PGs that are incomplete that HAVE DATA (Objects > 0) [ via ceph pg ls incomplete ]: 2.35 23199 0 0 0 95980273664 0 0 2477 incomplete 10s 2104'46277 28260:686871 [44,4,37,3,40,32]p44 [44,4,37,3,40,32]p44 2023-01-03T03:54:47.821280+0000 2022-12-29T18:53:09.287203+0000 14 queued for deep scrub 2.53 22821 0 0 0 94401175552 0 0 2745 remapped+incomplete 10s 2104'45845 28260:565267 [60,48,52,65,67,7]p60 [60]p60 2023-01-03T10:18:13.388383+0000 2023-01-03T10:18:13.388383+0000 408 queued for scrub 2.9f 22858 0 0 0 94555983872 0 0 2736 remapped+incomplete 10s 2104'45636 28260:759872 [56,59,3,57,5,32]p56 [56]p56 2023-01-03T10:55:49.848693+0000 2023-01-03T10:55:49.848693+0000 376 queued for scrub 2.be 22870 0 0 0 94429110272 0 0 2661 remapped+incomplete 10s 2104'45561 28260:813759 [41,31,37,9,7,69]p41 [41]p41 2023-01-03T14:02:15.790077+0000 2023-01-03T14:02:15.790077+0000 360 queued for scrub 2.e4 22953 0 0 0 94912278528 0 0 2648 remapped+incomplete 20m 2104'46048 28259:732896 [37,46,33,4,48,49]p37 [37]p37 2023-01-02T18:38:46.268723+0000 2022-12-29T18:05:47.431468+0000 18 queued for deep scrub 17.78 20169 0 0 0 84517834400 0 0 2198 remapped+incomplete 10s 3735'53405 28260:1243673 [4,37,2,36,66,0]p4 [41]p41 2023-01-03T14:21:41.563424+0000 2023-01-03T14:21:41.563424+0000 348 queued for scrub 17.d8 20328 0 0 0 85196053130 0 0 1852 remapped+incomplete 10s 3735'54458 28260:1309564 [38,65,61,37,58,39]p38 [53]p53 2023-01-02T18:32:35.371071+0000 2022-12-28T19:08:29.492244+0000 21 queued for deep scrub At present I'm unable to reliably access my data due to incomplete pages above. I'll post whatever outputs requested (won't post now as it can be rather verbose). Is there hope?

1 0

Setting Prometheus retention_time
by Robert Sander 08 Jan '23

08 Jan '23

Hi, The Quincy documentation shows that we could set the Prometheus retention_time within a service specification: https://docs.ceph.com/en/quincy/cephadm/services/monitoring/#setting-up-pro… When trying this "ceph orch apply" only shows: Error EINVAL: ServiceSpec: __init__() got an unexpected keyword argument 'retention_time' It looks like release 17.2.5 does not contain this code yet. Why is the content of the documentation already online when https://github.com/ceph/ceph/pull/47943 has not been released yet? Regards -- Robert Sander Heinlein Support GmbH Linux: Akademie - Support - Hosting http://www.heinlein-support.de Tel: 030-405051-43 Fax: 030-405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

2 1

test message
by Joe Comeau 08 Jan '23

08 Jan '23

Hi Just testing as I have not received a message from the list in a couple days Thanks Joe

1 0

Re: Missing SSDs disk on ceph deployment
by Michel Niyoyita 08 Jan '23

08 Jan '23

Dear team, Kindly help on this , I am completely blocked. Best Regards Michel On Thu, Jan 5, 2023 at 2:45 PM Michel Niyoyita <micou12(a)gmail.com> wrote: > Dear team, > > I have deployed the ceph cluster in production using ceph-ansible on > ubuntu OS 20.04 it consists of 3 monitors and 3 osds (each osd has 20 disks > , 16 hdd and 4 ssd) after deployment the ceph cluster healthy is OK , but > instead of getting 60 disk it appears 48 disks which are hdd only > (according to the output of ceph osd df tree) below are output of ceph osd > df tree and lsblk command. > > hdd: 7.71349 T > ssd: 7 T > > ceph osd df tree > root@ceph-mon3:~# ceph osd df tree > ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META > AVAIL %USE VAR PGS STATUS TYPE NAME > -1 370.24731 - 370 TiB 21 TiB 249 MiB 0 B 1.2 GiB > 349 TiB 5.66 1.00 - root default > -5 123.41577 - 123 TiB 7.0 TiB 83 MiB 0 B 403 MiB > 116 TiB 5.66 1.00 - host ceph-osd1 > 0 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 30 MiB > 7.3 TiB 5.66 1.00 6 up osd.0 > 3 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB > 7.3 TiB 5.66 1.00 8 up osd.3 > 6 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 35 MiB > 7.3 TiB 5.66 1.00 12 up osd.6 > 9 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 21 MiB > 7.3 TiB 5.66 1.00 4 up osd.9 > 12 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB > 7.3 TiB 5.66 1.00 6 up osd.12 > 16 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 21 MiB > 7.3 TiB 5.66 1.00 3 up osd.16 > 18 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB > 7.3 TiB 5.66 1.00 5 up osd.18 > 21 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 21 MiB > 7.3 TiB 5.66 1.00 6 up osd.21 > 24 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 35 MiB > 7.3 TiB 5.66 1.00 7 up osd.24 > 27 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 20 MiB > 7.3 TiB 5.66 1.00 7 up osd.27 > 30 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 20 MiB > 7.3 TiB 5.66 1.00 7 up osd.30 > 33 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB > 7.3 TiB 5.66 1.00 7 up osd.33 > 36 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB > 7.3 TiB 5.66 1.00 5 up osd.36 > 39 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB > 7.3 TiB 5.66 1.00 9 up osd.39 > 42 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 30 MiB > 7.3 TiB 5.66 1.00 6 up osd.42 > 45 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 20 MiB > 7.3 TiB 5.66 1.00 7 up osd.45 > -3 123.41577 - 123 TiB 7.0 TiB 83 MiB 0 B 397 MiB > 116 TiB 5.66 1.00 - host ceph-osd2 > 1 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 26 MiB > 7.3 TiB 5.66 1.00 7 up osd.1 > 5 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 26 MiB > 7.3 TiB 5.66 1.00 7 up osd.5 > 8 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 26 MiB > 7.3 TiB 5.66 1.00 6 up osd.8 > 11 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 21 MiB > 7.3 TiB 5.66 1.00 5 up osd.11 > 14 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB > 7.3 TiB 5.66 1.00 7 up osd.14 > 15 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 26 MiB > 7.3 TiB 5.66 1.00 12 up osd.15 > 19 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 20 MiB > 7.3 TiB 5.66 1.00 5 up osd.19 > 22 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 26 MiB > 7.3 TiB 5.66 1.00 4 up osd.22 > 25 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 20 MiB > 7.3 TiB 5.66 1.00 2 up osd.25 > 28 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 21 MiB > 7.3 TiB 5.66 1.00 3 up osd.28 > 31 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 20 MiB > 7.3 TiB 5.66 1.00 6 up osd.31 > 34 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 30 MiB > 7.3 TiB 5.66 1.00 7 up osd.34 > 37 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 30 MiB > 7.3 TiB 5.66 1.00 8 up osd.37 > 40 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 30 MiB > 7.3 TiB 5.66 1.00 8 up osd.40 > 43 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 20 MiB > 7.3 TiB 5.66 1.00 6 up osd.43 > 46 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 30 MiB > 7.3 TiB 5.66 1.00 12 up osd.46 > -7 123.41577 - 123 TiB 7.0 TiB 83 MiB 0 B 409 MiB > 116 TiB 5.66 1.00 - host ceph-osd3 > 2 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 21 MiB > 7.3 TiB 5.66 1.00 4 up osd.2 > 4 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 21 MiB > 7.3 TiB 5.66 1.00 7 up osd.4 > 7 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB > 7.3 TiB 5.66 1.00 5 up osd.7 > 10 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 21 MiB > 7.3 TiB 5.66 1.00 4 up osd.10 > 13 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 30 MiB > 7.3 TiB 5.66 1.00 12 up osd.13 > 17 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 26 MiB > 7.3 TiB 5.66 1.00 6 up osd.17 > 20 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB > 7.3 TiB 5.66 1.00 7 up osd.20 > 23 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB > 7.3 TiB 5.66 1.00 8 up osd.23 > 26 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 30 MiB > 7.3 TiB 5.66 1.00 8 up osd.26 > 29 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB > 7.3 TiB 5.66 1.00 6 up osd.29 > 32 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB > 7.3 TiB 5.66 1.00 6 up osd.32 > 35 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 30 MiB > 7.3 TiB 5.66 1.00 8 up osd.35 > 38 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 30 MiB > 7.3 TiB 5.66 1.00 9 up osd.38 > 41 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 20 MiB > 7.3 TiB 5.66 1.00 3 up osd.41 > 44 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 25 MiB > 7.3 TiB 5.66 1.00 4 up osd.44 > 47 hdd 7.71349 1.00000 7.7 TiB 447 GiB 5.2 MiB 0 B 30 MiB > 7.3 TiB 5.66 1.00 8 up osd.47 > TOTAL 370 TiB 21 TiB 249 MiB 0 B 1.2 GiB > 349 TiB 5.66 > > lsblk out put > ========== > root@ceph-osd1:~# lsblk > NAME > MAJ:MIN RM SIZE RO TYPE MOUNTPOINT > loop0 > 7:0 0 55.4M 1 loop /snap/core18/1944 > loop1 > 7:1 0 69.9M 1 loop /snap/lxd/19188 > loop2 > 7:2 0 31.1M 1 loop /snap/snapd/10707 > sda > 8:0 0 7.3T 0 disk > └─ceph--f011a10f--3f22--4785--8fe8--1eb4e64d380f-osd--block--1c3eaee6--60a7--4b2d--aff1--dff0a909d48e > 253:1 0 7.3T 0 lvm > sdb > 8:16 0 7.3T 0 disk > └─ceph--ba3fef60--84a2--4cd7--9aeb--4af05a17733d-osd--block--a4926772--4913--4ad1--8c01--348e8cb30159 > 253:3 0 7.3T 0 lvm > sdc > 8:32 0 7.3T 0 disk > └─ceph--14676d53--0a82--47b2--acc2--10ebae5e2cb1-osd--block--b1032109--80e0--4065--923a--80541669c316 > 253:5 0 7.3T 0 lvm > sdd > 8:48 0 7.3T 0 disk > └─ceph--2e7875da--a136--4b39--b354--21cfa8643951-osd--block--5a1ad840--225c--4d79--ab23--8589f13b9f3d > 253:7 0 7.3T 0 lvm > sde > 8:64 0 7T 0 disk > ├─ceph--61d22533--5bda--40d5--83b3--b546f4335acf-osd--db--be9d5f0f--ac60--42e1--850d--bf2baf913b3b > 253:26 0 447.1G 0 lvm > ├─ceph--61d22533--5bda--40d5--83b3--b546f4335acf-osd--db--5e7310c4--2b18--4938--ab6e--dc37e860d9d5 > 253:28 0 447.1G 0 lvm > ├─ceph--61d22533--5bda--40d5--83b3--b546f4335acf-osd--db--d2d78859--2e60--4ccc--b417--d33ff55837b6 > 253:30 0 447.1G 0 lvm > └─ceph--61d22533--5bda--40d5--83b3--b546f4335acf-osd--db--31ff57f6--e9d3--4bf1--b08d--95939083b4da > 253:32 0 447.1G 0 lvm > sdf > 8:80 0 7T 0 disk > ├─ceph--f3a65d83--2676--475e--ae6e--1b36138185b2-osd--db--7a8d393e--3af3--4c89--a99f--d3efcbb78066 > 253:18 0 447.1G 0 lvm > ├─ceph--f3a65d83--2676--475e--ae6e--1b36138185b2-osd--db--5abd44ca--aed8--48af--8f12--a64a6ae7938c > 253:20 0 447.1G 0 lvm > ├─ceph--f3a65d83--2676--475e--ae6e--1b36138185b2-osd--db--cd5768aa--b9ce--4c3b--9bc8--38c01b5bdc49 > 253:22 0 447.1G 0 lvm > └─ceph--f3a65d83--2676--475e--ae6e--1b36138185b2-osd--db--364e42ff--6e66--4a76--9df2--b2bb04080cd0 > 253:24 0 447.1G 0 lvm > sdg > 8:96 0 7T 0 disk > ├─ceph--04a9eb75--f85b--40ac--999d--9727c6ed5ea9-osd--db--3c469d07--0f1d--4544--861a--dd84283332fe > 253:10 0 447.1G 0 lvm > ├─ceph--04a9eb75--f85b--40ac--999d--9727c6ed5ea9-osd--db--61d08fe1--8d5a--4ef8--ba2d--ba1fc84636db > 253:12 0 447.1G 0 lvm > ├─ceph--04a9eb75--f85b--40ac--999d--9727c6ed5ea9-osd--db--2b2a8af7--33f5--4ca5--b09b--2f24ebf583b3 > 253:14 0 447.1G 0 lvm > └─ceph--04a9eb75--f85b--40ac--999d--9727c6ed5ea9-osd--db--fd2e105a--4fdf--4f15--a48b--1303717b8961 > 253:16 0 447.1G 0 lvm > sdh > 8:112 0 7T 0 disk > ├─ceph--31db9bd9--f703--46b7--bccb--d7cb9ab6e7b3-osd--db--6e289081--5404--4294--a75a--f4a1a8fabe52 > 253:2 0 447.1G 0 lvm > ├─ceph--31db9bd9--f703--46b7--bccb--d7cb9ab6e7b3-osd--db--e89285a3--4efa--4239--bbb7--a95064ced7f2 > 253:4 0 447.1G 0 lvm > ├─ceph--31db9bd9--f703--46b7--bccb--d7cb9ab6e7b3-osd--db--0b7f1fd0--bc32--47b7--b8e4--c6f0f9f21047 > 253:6 0 447.1G 0 lvm > └─ceph--31db9bd9--f703--46b7--bccb--d7cb9ab6e7b3-osd--db--614e085d--ce04--4f5d--a485--ddca51bae65b > 253:8 0 447.1G 0 lvm > sdi > 8:128 0 7.3T 0 disk > └─ceph--8343ded8--765b--4038--9855--5e6684b55906-osd--block--054a0d83--4aea--4d4f--b623--77db554e207d > 253:9 0 7.3T 0 lvm > sdj > 8:144 0 7.3T 0 disk > └─ceph--88f64697--ab62--4fed--a1b4--88ef3563eaea-osd--block--97189221--a7f6--4d9d--b98a--a474fec95fc7 > 253:11 0 7.3T 0 lvm > sdk > 8:160 0 7.3T 0 disk > └─ceph--592f15db--face--49d8--b028--15fee31d77cd-osd--block--776b773d--347f--4a5c--8de7--66d4a9d56feb > 253:13 0 7.3T 0 lvm > sdl > 8:176 0 7.3T 0 disk > └─ceph--3ef3da28--0aa0--4bfe--80fa--7871a0ebbb2a-osd--block--9b91fccb--3562--4543--916a--6fd0baa00efd > 253:15 0 7.3T 0 lvm > sdm > 8:192 0 7.3T 0 disk > └─ceph--604f22a2--bf89--40a8--bd37--bb5c317f0525-osd--block--ff387140--c528--40e2--ac75--3add53fd5bbb > 253:17 0 7.3T 0 lvm > sdn > 8:208 0 7.3T 0 disk > └─ceph--b702f897--06ad--43fe--b04b--3e03477ffbc3-osd--block--07cc426f--b4c1--4459--a6bc--81ee7a034060 > 253:19 0 7.3T 0 lvm > sdo > 8:224 0 7.3T 0 disk > └─ceph--5e1b4c03--f01b--45d7--971f--a02fe109ef13-osd--block--653b2370--dcb0--46bc--a71b--f62af6777feb > 253:21 0 7.3T 0 lvm > sdp > 8:240 0 7.3T 0 disk > └─ceph--e3ed1f19--15f6--4bd2--81f0--93f9720d7fad-osd--block--4121fcc9--1c00--4da5--b01e--a041280d330a > 253:23 0 7.3T 0 lvm > sdq > 65:0 0 7.3T 0 disk > └─ceph--417cb977--615a--4b86--a4a9--d6af7587b670-osd--block--e77b7eab--7dd8--4300--bd98--27d010c2e17e > 253:25 0 7.3T 0 lvm > sdr > 65:16 0 7.3T 0 disk > └─ceph--ad66d2d6--581d--4985--b945--154586b39903-osd--block--6b1b3de2--8e13--4361--bcd0--16ad609177b0 > 253:27 0 7.3T 0 lvm > sds > 65:32 0 7.3T 0 disk > └─ceph--66aebaac--de6b--42ff--94f1--a489a755e51a-osd--block--a581f3e1--7d86--4f8f--9d0a--de87ef713436 > 253:29 0 7.3T 0 lvm > sdt > 65:48 0 7.3T 0 disk > └─ceph--1558f2d7--741a--4feb--9e68--a900b823eac7-osd--block--52d32f21--9f3a--491e--a26c--41304d63a0ee > 253:31 0 7.3T 0 lvm > sdu > 65:64 0 223.5G 0 disk > ├─sdu1 > 65:65 0 512M 0 part /boot/efi > ├─sdu2 > 65:66 0 1G 0 part /boot > └─sdu3 > 65:67 0 222G 0 part > └─ubuntu--vg-ubuntu--lv > 253:0 0 111G 0 lvm / > > Kindly help me to configure in order to get all disk up and work. > > Best Regards > > Michel > > >

1 0

Re: docs.ceph.com -- Do you use the header navigation bar? (RESPONSES REQUESTED)
by John Mulligan 08 Jan '23

08 Jan '23

On Wednesday, January 4, 2023 10:35:56 AM EST John Zachary Dover wrote: > Do you use the header navigation bar on docs.ceph.com? See the attached > file (sticky_header.png) if you are unsure of what "header navigation bar" > means. In the attached file, the header navigation bar is indicated by > means of two large, ugly, red-and-green arrows. > > *Cards on the Table* > The navigation bar is the kind of thing that is sometimes referred to as a > "sticky header", and it can get in the way of linked-to sections. I would > like to remove this header bar. If there is community support for the > header bar, though, I won't remove it. > > *What is Zac Complaining About?* > Follow this procedure to see the behavior that has provoked my complaint: > > 1. Go to https://docs.ceph.com/en/quincy/glossary/ > 2. Scroll down to the "Ceph Cluster Map" entry. > 3. Click the "Cluster Map" link in the line that reads "See Cluster Map". > 4. Notice that the header navigation bar obscures the headword "Cluster > Map". > > If you have any opinion at all on this matter, voice it. Please. > FWIW I am not able to reproduce the problem you are describing. In all cases the thin blue-green bar appeared above the term with the selected anchor link. I tried Firefox (108, Linux), Chromium (107, Linux) and for giggles Firefox on Android. In all cases things looked fine to me and the selected term was not hidden by that nav bar. I share because I was surprised by the result given that others on the list seem to see the problem. But I also don't see what I would describe as "two large, ugly, red-and-green arrows." Perhaps the page is rendering differently for some people and we don't hit the issue in that case? PS. I also didn't see the png file in question. Perhaps this list strips attachments?

1 0

v16.2.5 Pacific released
by David Galloway 08 Jul '21

08 Jul '21

We're happy to announce the 5th backport release in the Pacific series. We recommend users to update to this release. For a detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/en/news/blog/2021/v16-2-5-pacific-released Notable Changes --------------- * `ceph-mgr-modules-core` debian package does not recommend `ceph-mgr-rook` anymore. As the latter depends on `python3-numpy` which cannot be imported in different Python sub-interpreters multi-times if the version of `python3-numpy` is older than 1.19. Since `apt-get` installs the `Recommends` packages by default, `ceph-mgr-rook` was always installed along with `ceph-mgr` debian package as an indirect dependency. If your workflow depends on this behavior, you might want to install `ceph-mgr-rook` separately. * mgr/nfs: `nfs` module is moved out of volumes plugin. Prior using the `ceph nfs` commands, `nfs` mgr module must be enabled. * volumes/nfs: The `cephfs` cluster type has been removed from the `nfs cluster create` subcommand. Clusters deployed by cephadm can support an NFS export of both `rgw` and `cephfs` from a single NFS cluster instance. * The `nfs cluster update` command has been removed. You can modify the placement of an existing NFS service (and/or its associated ingress service) using `orch ls --export` and `orch apply -i ...`. * The `orch apply nfs` command no longer requires a pool or namespace argument. We strongly encourage users to use the defaults so that the `nfs cluster ls` and related commands will work properly. * The `nfs cluster delete` and `nfs export delete` commands are deprecated and will be removed in a future release. Please use `nfs cluster rm` and `nfs export rm` instead. * A long-standing bug that prevented 32-bit and 64-bit client/server interoperability under msgr v2 has been fixed. In particular, mixing armv7l (armhf) and x86_64 or aarch64 servers in the same cluster now works. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-16.2.5.tar.gz * For packages, see https://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: 0883bdea7337b95e4b611c768c0279868462204a

3 2

Cephfs slow, not busy, but doing high traffic in the metadata pool
by Flemming Frandsen 08 Jul '21

08 Jul '21

We have a nautilus cluster where any metadata write operation is very slow. We're seeing very light load from clients, as reported by dumping ops in flight, often it's zero. We're also seeing about 100 MB/s writes to the metadata pool, constantly, for weeks on end, which seems excessive, as only 22GB is utilized. Should the writes to the metadata pool not quiet down when there's nothing going on? Is there any way i can get information about why the MDSes are thrashing so badly?

2 3

RocksDB resharding does not work
by Robert Sander 08 Jul '21

08 Jul '21

Hi, I am trying to apply the resharding to a containerized OSD (16.2.4) as described here: https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#r… # ceph osd set noout # ceph orch daemon stop osd.13 # cephadm shell --name osd.13 # ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-13 --sharding="m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L P" reshard 2021-07-08T15:12:03.392+0000 7f2e2173b3c0 -1 rocksdb: prepare_for_reshard failure parsing column options: block_cache={type=binned_lru} error resharding: (22) Invalid argument # exit # ceph orch daemon start osd.13 # ceph osd unset noout The OSD cannot start any more and has these errors in its log: Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.293+0000 7efc32db4080 0 osd.13:7.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196) Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.293+0000 7efc32db4080 1 bluestore(/var/lib/ceph/osd/ceph-13) _mount path /var/lib/ceph/osd/ceph-13 Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.293+0000 7efc32db4080 0 bluestore(/var/lib/ceph/osd/ceph-13) _open_db_and_around read-only:0 repair:0 Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.293+0000 7efc32db4080 1 bdev(0x564271210800 /var/lib/ceph/osd/ceph-13/block) open path /var/lib/ceph/osd/ceph-13/block Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.293+0000 7efc32db4080 1 bdev(0x564271210800 /var/lib/ceph/osd/ceph-13/block) open size 107369988096 (0x18ffc00000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.293+0000 7efc32db4080 1 bluestore(/var/lib/ceph/osd/ceph-13) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.293+0000 7efc32db4080 1 bdev(0x564271210c00 /var/lib/ceph/osd/ceph-13/block) open path /var/lib/ceph/osd/ceph-13/block Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.293+0000 7efc32db4080 1 bdev(0x564271210c00 /var/lib/ceph/osd/ceph-13/block) open size 107369988096 (0x18ffc00000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.293+0000 7efc32db4080 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-13/block size 100 GiB Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.293+0000 7efc32db4080 1 bluefs mount Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.293+0000 7efc32db4080 1 bluefs _init_alloc shared, id 1, capacity 0x18ffc00000, block size 0x10000 Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.325+0000 7efc32db4080 1 bluefs mount shared_bdev_used = 0 Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.325+0000 7efc32db4080 -1 rocksdb: verify_sharding extra columns in rocksdb. rocksdb columns = [default,m-0,m-1,m-2,p-0,p-1,p-2] target columns = [reshardingXcommencingXlocked] Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.325+0000 7efc32db4080 -1 bluestore(/var/lib/ceph/osd/ceph-13) _open_db erroring opening db: Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.325+0000 7efc32db4080 1 bluefs umount Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.325+0000 7efc32db4080 1 bdev(0x564271210c00 /var/lib/ceph/osd/ceph-13/block) close Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.577+0000 7efc32db4080 1 bdev(0x564271210800 /var/lib/ceph/osd/ceph-13/block) close Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.825+0000 7efc32db4080 -1 osd.13 0 OSD:init: unable to mount object store Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.825+0000 7efc32db4080 -1 ** ERROR: osd init failed: (5) Input/output error How do I correct the issue? Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

1 0