- ceph-users - lists.ceph.io

Re: How big an OSD disk could be?
by Janne Johansson 15 Mar '21

15 Mar '21

Den fre 12 mars 2021 kl 18:10 skrev huxiaoyu(a)horebdata.cn <huxiaoyu(a)horebdata.cn>: > Dear cephers, > Just wonder how big an OSD disk could be? Currently the biggest HDD has a capacity of 18TB or 20TB. It is suitable for an OSD still? > Is there a limitation of the capacity of a single OSD? Can it be 30TB , 50TB or 100TB a single OSD? What could be the potential limit? I think your patience will be the limit. If you stick 8x or 10x 20TB drives in a box, it will take a very long time to fill them to say, 50%. That would be some 80-100TB to move over the network and a lot of IO spread over only 8-10 drives before it settles, which would be a long time on spinning drives. Unless the objects are huge, 100TB of files/objects would mean many many millions of writes just to fill the OSD and its list of contents. For spinners, it would mean many many millions of head movements just to write it all out, hopefully never encountering a random bad block. After you have filled that up, if such a host crashes or needs maintenance, another 80-100TB will need recreating from the other huge drives. I don't think there are specific limitations on the size itself, but as the single drive becomes larger and larger, just adding a new host or a drive will mean the cluster is rebalancing for days or weeks if not more. At some point you would end up having the cluster almost never in HEALTH_OK state because of normal replacements, expansions and other surprises, which in turn could cause secondary problems with mon DBs and things like that. -- May the most significant bit of your life be positive.

3 2

How big an OSD disk could be?
by huxiaoyu＠horebdata.cn 14 Mar '21

14 Mar '21

Dear cephers, Just wonder how big an OSD disk could be? Currently the biggest HDD has a capacity of 18TB or 20TB. It is suitable for an OSD still? Is there a limitation of the capacity of a single OSD? Can it be 30TB , 50TB or 100TB a single OSD? What could be the potential limit? best regards, samuel huxiaoyu(a)horebdata.cn

7 13

Re: Some confusion around PG, OSD and balancing issue
by Darrin Hodges 14 Mar '21

14 Mar '21

HI Matthew, the results of the commands are: ceph df detail --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 190 TiB 61 TiB 129 TiB 129 TiB 67.70 TOTAL 190 TiB 61 TiB 129 TiB 129 TiB 67.70 --- POOLS --- POOL ID PGS STORED (DATA) (OMAP) OBJECTS USED (DATA) (OMAP) %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR device_health_metrics 1 1 31 MiB 0 B 31 MiB 73 92 MiB 0 B 92 MiB 0 9.8 TiB N/A N/A 73 0 B 0 B libvirt-pool 3 512 43 TiB 43 TiB 9.1 MiB 11.18M 128 TiB 128 TiB 27 MiB 81.21 9.8 TiB N/A N/A 11.18M 0 B 0 B ceph osd pool ls detail pool 1 'device_health_metrics' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 32288 flags hashpspool stripe_width 0 pg_num_min 1 application mgr_devicehealth pool 3 'libvirt-pool' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 512 pgp_num 512 autoscale_mode on last_change 31905 lfor 0/28367/30002 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd ceph balancer status { "active": true, "last_optimize_duration": "0:00:00.003773", "last_optimize_started": "Sun Mar 14 06:28:22 2021", "mode": "upmap", "optimize_result": "Unable to find further optimization, or pool(s) pg_num is decreasing, or distribution is already perfect", "plans": [] } We are using a vm with rbd disk, in this case 6 x 10T disk, given that the pool is replicated 3, it going to run out of space. We are in the process of upgrading the 3T disks to 8T but still won't be enough space if I understand things correctly, may need to considering changing to replication2, is that possible? many thanks Darrin On 13/3/21 3:07 pm, Matthew H wrote: > Hey Darrin, > > Can you provide the output of the following commands? > > ceph df detail > ceph osd pool ls detail > ceph balancer status > > Thanks so much, > > ------------------------------------------------------------------------ > *From:* Darrin Hodges <darrin(a)catalyst-au.net> > *Sent:* Wednesday, March 10, 2021 8:41 PM > *To:* ceph-users(a)ceph.io <ceph-users(a)ceph.io> > *Subject:* [ceph-users] Some confusion around PG, OSD and balancing issue > > HI all, > > Just looking for clarification around the relationship between PGs,OSDs > and balancing on a ceph (octopus) cluster. We have pg autobalance on > and balancing is set to upmap. There are 2 pools, one is the default > metric pool with 1 pg, the other is the pool we are using for > everything, it has 512 PG. There are 60 OSD's split across 4 hosts. The > USD usage ranges between 39% and 68%: > > > * current cluster score 0.048908 > > ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP > META AVAIL %USE VAR PGS STATUS > 0 hdd 2.74599 1.00000 2.7 TiB 1.3 TiB 1.3 TiB 2.3 MiB 2.3 > GiB 1.5 TiB 47.08 0.86 19 up > > 1 hdd 2.74550 1.00000 2.7 TiB 1.4 TiB 1.3 TiB 3.2 MiB 2.3 > GiB 1.4 TiB 49.46 0.90 20 up > > 2 hdd 2.74550 1.00000 2.7 TiB 1.4 TiB 1.4 TiB 2.8 MiB 2.4 > GiB 1.3 TiB 51.81 0.95 21 up > > 3 hdd 2.74550 1.00000 2.7 TiB 1.4 TiB 1.4 TiB 589 KiB 2.4 > GiB 1.3 TiB 51.73 0.95 21 up > > 4 hdd 2.74550 1.00000 2.7 TiB 1.6 TiB 1.6 TiB 2.2 MiB 2.7 > GiB 1.1 TiB 59.05 1.08 24 up > > 5 hdd 9.00000 1.00000 9.1 TiB 5.2 TiB 5.2 TiB 6.9 MiB 8.2 > GiB 3.9 TiB 56.75 1.04 77 up > > 6 hdd 2.74550 1.00000 2.7 TiB 1.5 TiB 1.5 TiB 1.9 MiB 2.8 > GiB 1.3 TiB 54.21 0.99 22 up > > 7 hdd 2.74550 1.00000 2.7 TiB 1.4 TiB 1.3 TiB 1.1 MiB 2.7 > GiB 1.4 TiB 49.32 0.90 20 up > > 8 hdd 2.74550 1.00000 2.7 TiB 1.4 TiB 1.3 TiB 1.1 MiB 2.3 > GiB 1.4 TiB 49.36 0.90 20 up > > 9 hdd 2.74550 1.00000 2.7 TiB 1.6 TiB 1.6 TiB 3.3 MiB 3.0 > GiB 1.1 TiB 59.21 1.08 24 up > > 10 hdd 2.74550 1.00000 2.7 TiB 1.8 TiB 1.8 TiB 2.2 MiB 3.4 > GiB 941 GiB 66.53 1.22 27 up > > 11 hdd 2.74550 1.00000 2.7 TiB 1.1 TiB 1.1 TiB 3.9 MiB 2.0 > GiB 1.7 TiB 39.69 0.73 16 up > > 12 hdd 2.74550 1.00000 2.7 TiB 1.2 TiB 1.2 TiB 1.2 MiB 2.1 > GiB 1.5 TiB 44.69 0.82 18 up > > 13 hdd 2.74550 1.00000 2.7 TiB 1.1 TiB 1.1 TiB 2.4 MiB 1.9 > GiB 1.7 TiB 39.59 0.72 16 up > > 14 hdd 2.74550 1.00000 2.7 TiB 1.8 TiB 1.8 TiB 2.1 MiB 3.1 > GiB 945 GiB 66.37 1.21 27 up > > 15 hdd 2.74599 1.00000 2.7 TiB 1.6 TiB 1.5 TiB 1.7 MiB 2.6 > GiB 1.2 TiB 56.68 1.04 23 up > > 16 hdd 2.74599 1.00000 2.7 TiB 1.3 TiB 1.3 TiB 3.3 MiB 2.3 > GiB 1.5 TiB 46.76 0.86 19 up > > 17 hdd 2.74599 0.95001 2.7 TiB 1.8 TiB 1.8 TiB 2.8 MiB 3.1 > GiB 953 GiB 66.12 1.21 26 up > > 18 hdd 2.74599 0.95001 2.7 TiB 1.8 TiB 1.7 TiB 1.5 MiB 3.0 > GiB 1010 GiB 64.08 1.17 26 up > > 19 hdd 2.74599 1.00000 2.7 TiB 1.8 TiB 1.7 TiB 3.0 MiB 2.9 > GiB 1016 GiB 63.88 1.17 26 up > > 20 hdd 9.00000 1.00000 9.1 TiB 4.7 TiB 4.7 TiB 28 MiB 7.5 > GiB 4.4 TiB 51.47 0.94 70 up > > 21 hdd 2.74599 1.00000 2.7 TiB 1.2 TiB 1.2 TiB 2.8 MiB 2.2 > GiB 1.5 TiB 44.43 0.81 18 up > > 22 hdd 2.74599 1.00000 2.7 TiB 1.4 TiB 1.4 TiB 653 KiB 2.4 > GiB 1.3 TiB 51.89 0.95 21 up > > 23 hdd 2.74599 1.00000 2.7 TiB 1.6 TiB 1.6 TiB 2.3 MiB 2.7 > GiB 1.1 TiB 59.26 1.08 24 up > > 24 hdd 2.74599 1.00000 2.7 TiB 1.6 TiB 1.5 TiB 3.1 MiB 3.0 > GiB 1.2 TiB 56.67 1.04 23 up > > 25 hdd 2.74599 0.90002 2.7 TiB 1.7 TiB 1.7 TiB 5.3 MiB 4.0 > GiB 1.1 TiB 61.47 1.12 25 up > > 26 hdd 2.74599 1.00000 2.7 TiB 1.4 TiB 1.4 TiB 1.5 MiB 2.4 > GiB 1.3 TiB 51.82 0.95 21 up > > 27 hdd 2.74599 1.00000 2.7 TiB 1.3 TiB 1.3 TiB 1.1 MiB 2.2 > GiB 1.5 TiB 46.93 0.86 19 up > > 28 hdd 2.74599 1.00000 2.7 TiB 1.3 TiB 1.3 TiB 2.6 MiB 2.2 > GiB 1.5 TiB 46.88 0.86 19 up > > 29 hdd 2.74599 1.00000 2.7 TiB 1.4 TiB 1.3 TiB 557 KiB 2.6 > GiB 1.4 TiB 49.37 0.90 20 up > > 45 hdd 2.74599 1.00000 2.7 TiB 1.6 TiB 1.6 TiB 2.3 MiB 2.7 > GiB 1.1 TiB 59.18 1.08 24 up > > 46 hdd 2.74599 1.00000 2.7 TiB 1.4 TiB 1.4 TiB 28 MiB 2.5 > GiB 1.3 TiB 51.76 0.95 22 up > > 47 hdd 2.74599 1.00000 2.7 TiB 1.6 TiB 1.6 TiB 977 KiB 3.1 > GiB 1.1 TiB 59.07 1.08 24 up > > 48 hdd 2.74599 1.00000 2.7 TiB 1.3 TiB 1.3 TiB 625 KiB 2.6 > GiB 1.5 TiB 46.86 0.86 19 up > > 49 hdd 2.74599 1.00000 2.7 TiB 1.7 TiB 1.7 TiB 1.9 MiB 3.2 > GiB 1.1 TiB 61.68 1.13 25 up > > 50 hdd 2.74599 1.00000 2.7 TiB 1.7 TiB 1.7 TiB 1.0 MiB 2.7 > GiB 1.1 TiB 61.53 1.13 25 up > > 51 hdd 2.74599 1.00000 2.7 TiB 1.2 TiB 1.1 TiB 2.5 MiB 2.3 > GiB 1.6 TiB 41.88 0.77 17 up > > 52 hdd 2.74599 0.95001 2.7 TiB 1.8 TiB 1.8 TiB 1.8 MiB 3.8 > GiB 942 GiB 66.49 1.22 27 up > > 53 hdd 9.00000 1.00000 9.1 TiB 4.3 TiB 4.3 TiB 6.2 MiB 6.8 > GiB 4.8 TiB 47.46 0.87 64 up > > 54 hdd 2.74599 0.95001 2.7 TiB 1.8 TiB 1.7 TiB 1.3 MiB 2.9 > GiB 1008 GiB 64.13 1.17 26 up > > 55 hdd 2.74599 1.00000 2.7 TiB 1.8 TiB 1.7 TiB 1.5 MiB 3.0 > GiB 1014 GiB 63.95 1.17 26 up > > 56 hdd 2.74599 1.00000 2.7 TiB 1.8 TiB 1.7 TiB 1.3 MiB 2.8 > GiB 1013 GiB 63.99 1.17 26 up > > 57 hdd 2.74599 1.00000 2.7 TiB 1.6 TiB 1.5 TiB 2.9 MiB 2.6 > GiB 1.2 TiB 56.67 1.04 23 up > > 58 hdd 2.74599 1.00000 2.7 TiB 1.5 TiB 1.5 TiB 1.2 MiB 2.5 > GiB 1.3 TiB 54.41 1.00 22 up > > 59 hdd 2.74599 0.95001 2.7 TiB 1.8 TiB 1.7 TiB 1.3 MiB 2.9 > GiB 1015 GiB 63.92 1.17 26 up > > 30 hdd 2.74599 1.00000 2.7 TiB 1.9 TiB 1.9 TiB 1.6 MiB 3.1 > GiB 878 GiB 68.79 1.26 28 up > > 31 hdd 2.74599 1.00000 2.7 TiB 1.3 TiB 1.3 TiB 2.3 MiB 2.2 > GiB 1.5 TiB 47.03 0.86 19 up > > 32 hdd 2.74599 1.00000 2.7 TiB 1.8 TiB 1.8 TiB 3.2 MiB 3.3 > GiB 944 GiB 66.45 1.22 27 up > > 33 hdd 2.74599 1.00000 2.7 TiB 1.1 TiB 1.1 TiB 3.1 MiB 1.9 > GiB 1.7 TiB 39.88 0.73 16 up > > 34 hdd 2.74599 1.00000 2.7 TiB 1.9 TiB 1.9 TiB 28 MiB 3.0 > GiB 874 GiB 68.93 1.26 29 up > > 35 hdd 2.74599 1.00000 2.7 TiB 1.3 TiB 1.3 TiB 1.3 MiB 2.2 > GiB 1.5 TiB 47.01 0.86 19 up > > 36 hdd 2.74599 1.00000 2.7 TiB 1.8 TiB 1.8 TiB 1.6 MiB 3.0 > GiB 947 GiB 66.31 1.21 27 up > > 37 hdd 2.74599 1.00000 2.7 TiB 1.6 TiB 1.6 TiB 1.8 MiB 2.9 > GiB 1.1 TiB 58.96 1.08 24 up > > 38 hdd 2.74599 1.00000 2.7 TiB 1.9 TiB 1.9 TiB 2.8 MiB 3.4 > GiB 877 GiB 68.82 1.26 28 up > > 39 hdd 2.74599 0.85004 2.7 TiB 1.6 TiB 1.6 TiB 2.4 MiB 3.5 > GiB 1.1 TiB 59.20 1.08 24 up > > 40 hdd 2.74599 1.00000 2.7 TiB 1.4 TiB 1.3 TiB 1.5 MiB 2.3 > GiB 1.4 TiB 49.35 0.90 20 up > > 41 hdd 2.74599 1.00000 2.7 TiB 1.4 TiB 1.4 TiB 2.3 MiB 2.4 > GiB 1.3 TiB 51.73 0.95 21 up > > 42 hdd 9.00000 1.00000 9.1 TiB 4.5 TiB 4.5 TiB 2.1 MiB 7.2 > GiB 4.6 TiB 49.76 0.91 67 up > > 43 hdd 2.74599 0.90002 2.7 TiB 1.6 TiB 1.6 TiB 2.9 MiB 3.2 > GiB 1.1 TiB 59.17 1.08 24 up > > 44 hdd 2.74599 1.00000 2.7 TiB 1.2 TiB 1.2 TiB 4.0 MiB 2.2 > GiB 1.5 TiB 44.34 0.81 18 up > > > > 54 of the OSD are 3TiB and there are four 8TiB - can this cause > imbalance? Or do I need to increase the PG from 512? or is this as good > as it gets? > > > > many thanks > > Darrin > > -- > CONFIDENTIALITY NOTICE: This email is intended for the named > recipients only. It may contain privileged, confidential or copyright > information. If you are not the named recipients, any use, reliance > upon, disclosure or copying of this email or any attachments is > unauthorised. If you have received this email in error, please reply > via email or telephone +61 2 8004 5928. > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io -- CONFIDENTIALITY NOTICE: This email is intended for the named recipients only. It may contain privileged, confidential or copyright information. If you are not the named recipients, any use, reliance upon, disclosure or copying of this email or any attachments is unauthorised. If you have received this email in error, please reply via email or telephone +61 2 8004 5928.

1 0

Removing secondary data pool from mds
by Michael Thomas 13 Mar '21

13 Mar '21

I have a cephfs secondary (non-root) data pool with unfound and degraded objects that I have not been able to recover[1]. I created an additional data pool and used "setfattr -n ceph.dir.layout.pool' and a very long rsync to move the files off of the degraded pool and onto the new pool. This has completed, and using find + 'getfattr -n ceph.file.layout.pool', I verified that no files are using the old pool anymore. No ceph.dir.layout.pool attributes point to the old pool either. However, the old pool still reports that there are objects in the old pool, likely the same ones that were unfound/degraded from before: https://pastebin.com/qzVA7eZr Based on a old message from the mailing list[2], I checked the MDS for stray objects (ceph daemon mds.ceph4 dump cache file.txt ; grep -i stray file.txt) and found 36 stray entries in the cache: https://pastebin.com/MHkpw3DV. However, I'm not certain how to map these stray cache objects to clients that may be accessing them. 'rados -p fs.data.archive.frames ls' shows 145 objects. Looking at the parent of each object shows 2 strays: for obj in $(cat rados.ls.txt) ; do echo $obj ; rados -p fs.data.archive.frames getxattr $obj parent | strings ; done [...] 10000020fa1.00000000 10000020fa1 stray6 10000020fbc.00000000 10000020fbc stray6 [...] ...before getting stuck on one object for over 5 minutes (then I gave up): 1000005b1af.00000083 What can I do to make sure this pool is ready to be safely deleted from cephfs (ceph fs rm_data_pool archive fs.data.archive.frames)? --Mike [1]https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/QHFOGEKXK7VDNNSKR74BA6IIMGGIXBXA/#7YQ6SSTESM5LTFVLQK3FSYFW5FDXJ5CF [2]http://lists.ceph.com/pipermail/ceph-users-ceph.com/2015-October/005233.h…

2 5

Location of Crush Map and CEPH metadata
by Ed Kalk 13 Mar '21

13 Mar '21

Hello, I have been googling for the answer to this and not found it. Does anyone know this? Where does CEPH store the crush map and the critical cluster metadata? What prevents a loss of this metadata when a node is lost? -- Thank you for your time, Edward H. Kalk IV Information Technology Dept. Server Specialist Datacenter Virtualization and Storage Systems Socket Telecom, LLC. 2703 Clark Lane Columbia, MO 65202 573-817-0000 or 800-socket3 X218

3 2

ceph repo cert expired
by Philip Brown 12 Mar '21

12 Mar '21

To whom it may concern: failure: repodata/repomd.xml from Ceph: [Errno 256] No more mirrors to try. https://download.ceph.com/rpm-octopus/el7/x86_64/repodata/repomd.xml: [Errno 14] curl#60 - "Peer's Certificate has expired."

1 0

Ceph 14.2.17 ceph-mgr module issue
by Stefan Kooman 12 Mar '21

12 Mar '21

Hi, After upgrading a Ceph cluster to 14.2.17 with ceph-ansible (docker containers) the manager hits an issue: Module 'volumes' has failed dependency: No module named typing, python trace: 2021-03-12 17:04:22.358 7f299ac75e40 1 mgr[py] Loading python module 'volumes' 2021-03-12 17:04:22.458 7f299ac75e40 -1 mgr[py] Module not found: 'volumes' 2021-03-12 17:04:22.458 7f299ac75e40 -1 mgr[py] Traceback (most recent call last): File "/usr/share/ceph/mgr/volumes/__init__.py", line 2, in <module> from .module import Module File "/usr/share/ceph/mgr/volumes/module.py", line 10, in <module> from .fs.volume import VolumeClient File "/usr/share/ceph/mgr/volumes/fs/volume.py", line 13, in <module> from .operations.subvolume import open_subvol, create_subvol, remove_subvol, \ File "/usr/share/ceph/mgr/volumes/fs/operations/subvolume.py", line 8, in <module> from .versions import loaded_subvolumes File "/usr/share/ceph/mgr/volumes/fs/operations/versions/__init__.py", line 9, in <module> from .subvolume_v1 import SubvolumeV1 File "/usr/share/ceph/mgr/volumes/fs/operations/versions/subvolume_v1.py", line 9, in <module> from typing import List, Dict ImportError: No module named typing I created tracker [1] for this issue with this information. Not sure if non containerized deployments hit this issue as well. I will find thta out somewhere next week. FYI, Gr. Stefan [1]: https://tracker.ceph.com/issues/49770

4 7

Question about delayed write IOs, octopus, mixed storage
by Philip Brown 12 Mar '21

12 Mar '21

I'm running some tests with mixed storage units, and octopus. 8 nodes, each with 2 SSDs, and 8 HDDs . the SSDsare relatively small: around 100GB each. Im mapping 8 rbds, striping them together, and running fio on them for testing. # fio --filename=/...../fio.testfile --size=120GB --rw=randrw --bs=8k --direct=1 --ioengine=libaio --iodepth=64 --numjobs=4 --time_based --group_reporting --name=readwritelatency-test-job --runtime=120 --eta-newline=1 Trouble is, I'm seeing sporadic delays of IOs. When I test ZFS, for example, it has this neat wait clumping status check: zpool iostat -w 20 and it shows me that some write io's are taking over 4 secomds to complete. Many are taking 1s or 2s This kind of thing has sort of happened before(but previously, I think I was using SSDs exclusively). When I emailed the list, people suggested turning off RBD cache, which worked great in that situation. This time, I have already done that (I believe), but still see this behavior. Would folks have any further suggestions to smooth performance out? The odd thing is, I read that bluestore is supposed to smooth things out and provide consistent response time, but that doesnt seem to be the case. Sample output from the zpool iostat below: twelve total_wait disk_wait syncq_wait asyncq_wait latency read write read write read write read write scrub trim ---------- ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- .. snip ... 1ms 1.01K 0 1.00K 0 0 0 0 0 0 0 2ms 29 0 29 18 0 0 0 1 0 0 4ms 23 3 23 14 0 0 0 3 0 0 8ms 64 6 64 9 0 0 0 7 0 0 16ms 74 10 74 59 0 0 0 11 0 0 33ms 24 17 24 154 0 0 0 19 0 0 67ms 7 25 7 100 0 0 0 26 0 0 134ms 3 40 3 36 0 0 0 36 0 0 268ms 1 59 1 18 0 0 0 59 0 0 536ms 0 116 0 3 0 0 0 113 0 0 1s 0 109 0 0 0 0 0 98 0 0 2s 0 24 0 0 0 0 0 20 0 0 4s 0 2 0 0 0 0 0 1 0 0 8s 0 0 0 0 0 0 0 0 0 0 17s 0 0 0 0 0 0 0 0 0 0 -- Philip Brown| Sr. Linux System Administrator | Medata, Inc. 5 Peters Canyon Rd Suite 250 Irvine CA 92606 Office 714.918.1310| Fax 714.918.1325 pbrown(a)medata.com| www.medata.com

2 6

Container deployment - Ceph-volume activation
by Cloud Guy 12 Mar '21

12 Mar '21

Hello, TL;DR Looking for guidance on ceph-volume lvm activate --all as it would apply to a containerized ceph deployment (Nautilus or Octopus). Detail: I’m planning to upgrade my Nautilus non-container cluster to Octopus (eventually containerized). There’s an expanded procedure that was tested and working in our lab, however won’t go into the whole process. My question is around existing OSD hosts. I have to re-platform the host OS, and one of the ways in the OSDs were reactivated previously when this was done (non-containerized) was to install ceph packages, deploy keys, config, etc. then run ceph-volume lvm activate --all to magically bring up all OSDs. Looking for a similar approach except if the OSDs are containerized, and I re-platform the host OS (Centos -> Ubuntu), how could I reactivate all OSDs as containers and avoid rebuilding data on the OSDs? Thank you.

4 4

OSDs crashing after server reboot.
by Cassiano Pilipavicius 12 Mar '21

12 Mar '21

Hi, please if someone know how to help, I have an HDD pool in mycluster and after rebooting one server, my osds has started to crash. This pool is a backup pool and have OSD as failure domain with an size of 2. After rebooting one server, My osds started to crash, and the thing is only getting worse. I have then tried to run ceph-bluestore-tool repair and I receive what I think is the same error that shows on the osd logs: [root@cwvh13 ~]# ceph-bluestore-tool repair --path /var/lib/ceph/osd/ceph-81 --log-level 10 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.16/rpm/el7/BUILD/ceph-14.2.16/src/os/bluestore/Allocator.cc: In function 'virtual Allocator::SocketHook::~SocketHook()' thread 7f6467ffcec0 time 2021-03-11 12:13:12.121766 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.16/rpm/el7/BUILD/ceph-14.2.16/src/os/bluestore/Allocator.cc: 53: FAILED ceph_assert(r == 0) ceph version 14.2.16 (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7f645e1a7b27] 2: (()+0x25ccef) [0x7f645e1a7cef] 3: (()+0x3cd57f) [0x5642e85c457f] 4: (HybridAllocator::~HybridAllocator()+0x17) [0x5642e85f3f37] 5: (BlueStore::_close_alloc()+0x42) [0x5642e84379d2] 6: (BlueStore::_close_db_and_around(bool)+0x2f8) [0x5642e84bbac8] 7: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x293) [0x5642e84bbf13] 8: (main()+0x13cc) [0x5642e83caaec] 9: (__libc_start_main()+0xf5) [0x7f645ae24555] 10: (()+0x1fae9f) [0x5642e83f1e9f]

2 3