- ceph-users - lists.ceph.io

MDS is reporting damaged metadata damage- followup
by ricardo.re.azevedo＠gmail.com 02 Mar '21

02 Mar '21

Hi all, Following up on a previous issue. My cephfs MDS is reporting damaged metadata following the addition (and remapping) of 12 new OSDs. `ceph tell mds.database-0 damage ls` reports ~85 files damaged. All of type "backtrace". ` ceph tell mds.database-0 scrub start / recursive repair` seems to have no effect on the damage. ` ceph tell mds.database-0 scrub start / recursive repair force` also has no effect. I understand this seems to be an issue with mapping the file to a filesystem path. Is there anything I can do to recover these files? Any manual methods? > ceph status reports: cluster: id: 692905c0-f271-4cd8-9e43-1c32ef8abd13 health: HEALTH_ERR 1 MDSs report damaged metadata 300 pgs not deep-scrubbed in time 300 pgs not scrubbed in time services: mon: 3 daemons, quorum database-0,file-server,webhost (age 37m) mgr: webhost(active, since 3d), standbys: file-server, database-0 mds: cephfs:1 {0=database-0=up:active} 2 up:standby osd: 48 osds: 48 up (since 56m), 48 in (since 13d); 10 remapped pgs task status: scrub status: mds.database-0: idle data: pools: 7 pools, 633 pgs objects: 60.82M objects, 231 TiB usage: 336 TiB used, 246 TiB / 582 TiB avail pgs: 623 active+clean 6 active+remapped+backfilling 4 active+remapped+backfill_wait Thanks for the help. Best, Ricardo

1 0

Ceph Object Gateway setup/tutorial
by Rok Jaklič 02 Mar '21

02 Mar '21

Hi, installation of cluster/osds went "by the book" https://docs.ceph.com/, but now I want to setup Ceph Object Gateway, but documentation on https://docs.ceph.com/en/latest/radosgw/ seems to lack information about what and where to restart for example when setting [client.rgw.gateway-node1] in /etc/ceph/ceph.conf. Also where should we set this? In cephadm shell or on the host ...? Is there some tutorial howto setup gateway from the beginning? Kind regards, Rok

1 0

reboot breaks OSDs converted from ceph-disk to ceph-volume simple
by Frank Schilder 02 Mar '21

02 Mar '21

Dear all, ceph version: mimic 13.2.10 I'm facing a serious bug with devices converted from "ceph-disk" to "ceph-volume simple". I "converted" all ceph-disk devices using "ceph-volume simple scan ..." And everything worked fine at the beginning. Today I needed to reboot an OSD host and since then most ceph-disk OSDs are screwed up. Apparently, "ceph-volume simple scan ..." creates symlinks to the block partition /dev/sd?2 using the "/dev/sd?2" name for the link target. These names are not stable and are expected to change after every reboot. Now I have a bunch of OSDs with new /dev/sd?2" names that won't boot any more, because this link points to the wrong block partition. Doing another "ceph-volume simple scan ..." doesn't help, it just "rediscovers" the wrong location. Here is what a broken OSD looks like (fresh "ceph-volume simple scan --stdout ..." output): { "active": "ok", "block": { "path": "/dev/sda2", "uuid": "b5ac1462-510a-4483-8f42-604e6adc5c9d" }, "block_uuid": "1d9d89a2-18c7-4610-9dcd-167d44ce1879", "bluefs": 1, "ceph_fsid": "e4ece518-f2cb-4708-b00f-b6bf511e91d9", "cluster_name": "ceph", "data": { "path": "/dev/sdb1", "uuid": "c35a7efb-8c1c-42a1-8027-cf422d7e7ecb" }, "fsid": "c35a7efb-8c1c-42a1-8027-cf422d7e7ecb", "keyring": "AQAZJ6ddedALDxAAJI7NLJ2CRFoQWK5STRpHuw==", "kv_backend": "rocksdb", "magic": "ceph osd volume v026", "mkfs_done": "yes", "none": "", "ready": "ready", "require_osd_release": "", "type": "bluestore", "whoami": 241 } OSD 241's data partition looks like this (after mount /dev/sdb1 /var/lib/ceph/osd/ceph-241): [root@ceph-adm:ceph-18 ceph-241]# ls -l /var/lib/ceph/osd/ceph-241 total 56 -rw-r--r--. 1 root root 411 Oct 16 2019 activate.monmap -rw-r--r--. 1 ceph ceph 3 Oct 16 2019 active lrwxrwxrwx. 1 root root 9 Mar 2 14:19 block -> /dev/sda2 -rw-r--r--. 1 ceph ceph 37 Oct 16 2019 block_uuid -rw-r--r--. 1 ceph disk 2 Oct 16 2019 bluefs -rw-r--r--. 1 ceph ceph 37 Oct 16 2019 ceph_fsid -rw-r--r--. 1 ceph ceph 37 Oct 16 2019 fsid -rw-------. 1 ceph ceph 58 Oct 16 2019 keyring -rw-r--r--. 1 ceph disk 8 Oct 16 2019 kv_backend -rw-r--r--. 1 ceph ceph 21 Oct 16 2019 magic -rw-r--r--. 1 ceph disk 4 Oct 16 2019 mkfs_done -rw-r--r--. 1 ceph ceph 0 Nov 23 14:58 none -rw-r--r--. 1 ceph disk 6 Oct 16 2019 ready -rw-r--r--. 1 ceph disk 2 Jan 31 2020 require_osd_release -rw-r--r--. 1 ceph ceph 10 Oct 16 2019 type -rw-r--r--. 1 ceph ceph 4 Oct 16 2019 whoami The symlink "block -> /dev/sda2" goes to the wrong disk. How can I fix that in a stable way? Also, why are not stable "/dev/disk/by-uuid/..." link targets created instead? Can I change that myself? Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

1 1

RBD clone on Bluestore
by Pawel S 01 Mar '21

01 Mar '21

hello! I'm trying to understand how Bluestore cooperates with RBD image clones, so my test is simple 1. create an image (2G) and fill with data 2. create a snapshot 3. protect it 4. create a clone of the image 5. write a small portion of data (4K) to clone 6. check how it changed and if just 4K are used to prove CoW allocated new extent instead of copying out snapped data. Unfortunately it occurs that at least rbd du reports that 4M was changed and the clone consumes 4M of data instead of expected 4K... ''' rbd du rbd/clone1 NAME PROVISIONED USED clone1 2 GiB 4 MiB ''' How can I trace/prove Bluestore CoW really works in this case, and prevent copying the rest of the 4M stripe like Filestore did ? p.s tested on Luminous/Octopus, ssd devices, min_alloc_size: 16k, block_size: 4k best regards! -- Pawel S.

3 4

Ceph iscsi help
by Várkonyi János 01 Mar '21

01 Mar '21

Hi All, I2d like to install a Ceph Nautilus on Ubuntu 18.04 LTS and give the storage to 2 windows server via ISCSI. I choose the Nautilus because of the deploy function I don't want to another VM to cephadm. So I can isntall the ceph and it is working properly but can't setup the icsi gateway. The services running like tcmu-runner, ebd-target-gw and rbd-target-api. I can going into gwcli but I can't create the first gw I get this msessage: /iscsi-target...-igw/gateways> create cf01 192.168.203.51 skipchecks=true OS version/package checks have been bypassed Get gateway hostname failed : 403 Forbidden Please check api_host setting and make sure host cf01 IP is listening on port 5000 In the syslog at the same time: Mar 1 15:43:02 cf01 there is no tcmu-runner data avaliable Mar 1 15:43:06 cf01 ::ffff:127.0.0.1 - - [01/Mar/2021 15:43:06] "GET /api/config HTTP/1.1" 200 - I can see the python listening on port 5000 (mybe this is my problem) netstat -tulpn | grep 5000 tcp6 0 0 :::5000 :::* LISTEN 1976/python I cannot find anything about this error and I can't figure out what is solution. Ubuntu 18.04.5 LTS 4.15.0-136-generic I also tried with 4.20.0-042000-generic but the erorr was the same. jansz0

3 2

15.2.8 mgr keep crashing every few days
by levin ng 01 Mar '21

01 Mar '21

Hi all, I’d recently deployed ceph 15.2.8 with 3(mon,mgr,rgw,mds) and 4 (osd) total 7 host, however I encountered mgr crash a few times a week, the crashing mgr can be any one of 3. I couldn’t identify the problem behind and here is the crash info, appreciate anyone if you have suggestions that I could narrow it down. Thank you very much. { "assert_condition": "ret == 0", "assert_file": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.8/rpm/el8/BUILD/ceph-15.2.8/src/common/Thread.cc", "assert_func": "void Thread::create(const char*, size_t)", "assert_line": 157, "assert_msg": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.8/rpm/el8/BUILD/ceph-15.2.8/src/common/Thread.cc: In function 'void Thread::create(const char*, size_t)' thread 7f833addc700 time 2021-02-10T20:00:32.980508+0000\n/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/15.2.8/rpm/el8/BUILD/ceph-15.2.8/src/common/Thread.cc: 157: FAILED ceph_assert(ret == 0)\n", "assert_thread_name": "mgr-fin", "backtrace": [ "(()+0x12b20) [0x7f835a51cb20]", "(gsignal()+0x10f) [0x7f8358f6d7ff]", "(abort()+0x127) [0x7f8358f57c35]", "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f835c07b735]", "(()+0x27a8fe) [0x7f835c07b8fe]", "(()+0x34cef6) [0x7f835c14def6]", "(DispatchQueue::start()+0x3a) [0x7f835c29697a]", "(AsyncMessenger::ready()+0xcd) [0x7f835c3340cd]", "(Messenger::add_dispatcher_head(Dispatcher*)+0x68) [0x7f835c3f8478]", "(MonClient::get_monmap_and_config()+0xbb) [0x7f835c3f66ab]", "(ceph_mount_info::init()+0x4d) [0x7f834298435d]", "(()+0x3680f) [0x7f8342cd280f]", "(()+0x19d421) [0x7f835ba5c421]", "(_PyEval_EvalFrameDefault()+0x498) [0x7f835ba5ce08]", "(()+0x179c78) [0x7f835ba38c78]", "(()+0x19d1c7) [0x7f835ba5c1c7]", "(_PyEval_EvalFrameDefault()+0x498) [0x7f835ba5ce08]", "(()+0x179c78) [0x7f835ba38c78]", "(()+0x19d1c7) [0x7f835ba5c1c7]", "(_PyEval_EvalFrameDefault()+0x498) [0x7f835ba5ce08]", "(()+0x1221d4) [0x7f835b9e11d4]", "(()+0x122c55) [0x7f835b9e1c55]", "(()+0x19cf27) [0x7f835ba5bf27]", "(_PyEval_EvalFrameDefault()+0x498) [0x7f835ba5ce08]", "(_PyFunction_FastCallDict()+0x122) [0x7f835b9b9ec2]", "(_PyObject_FastCallDict()+0x70e) [0x7f835b9bac9e]", "(()+0x10dc70) [0x7f835b9ccc70]", "(_PyObject_FastCallDict()+0x6ec) [0x7f835b9bac7c]", "(PyObject_CallFunctionObjArgs()+0xe8) [0x7f835b9dbd48]", "(_PyEval_EvalFrameDefault()+0x2588) [0x7f835ba5eef8]", "(()+0xf99b4) [0x7f835b9b89b4]", "(()+0x179e60) [0x7f835ba38e60]", "(()+0x19d1c7) [0x7f835ba5c1c7]", "(_PyEval_EvalFrameDefault()+0x10d5) [0x7f835ba5da45]", "(()+0x179c78) [0x7f835ba38c78]", "(()+0x19d1c7) [0x7f835ba5c1c7]", "(_PyEval_EvalFrameDefault()+0x498) [0x7f835ba5ce08]", "(()+0xfa326) [0x7f835b9b9326]", "(()+0x179e60) [0x7f835ba38e60]", "(()+0x19d1c7) [0x7f835ba5c1c7]", "(_PyEval_EvalFrameDefault()+0x498) [0x7f835ba5ce08]", "(()+0x179c78) [0x7f835ba38c78]", "(()+0x19d1c7) [0x7f835ba5c1c7]", "(_PyEval_EvalFrameDefault()+0x498) [0x7f835ba5ce08]", "(_PyFunction_FastCallDict()+0x122) [0x7f835b9b9ec2]", "(_PyObject_FastCallDict()+0x70e) [0x7f835b9bac9e]", "(()+0x10dc70) [0x7f835b9ccc70]", "(PyObject_Call()+0x4b) [0x7f835b9c1acb]", "(PyObject_CallMethod()+0x10b) [0x7f835ba5ac6b]", "(ActivePyModule::handle_command(ModuleCommand const&, MgrSession const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, boost::variant<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, double, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::vector<long, std::allocator<long> >, std::vector<double, std::allocator<double> > >, std::less<void>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, boost::variant<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, double, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::vector<long, std::allocator<long> >, std::vector<double, std::allocator<double> > > > > > const&, ceph::buffer::v15_2_0::list const&, std::__cxx11::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >*)+0x222) [0x55bc0b8a0cb2]", "(()+0x1b0fdd) [0x55bc0b8f5fdd]", "(Context::complete(int)+0xd) [0x55bc0b8b0bdd]", "(Finisher::finisher_thread_entry()+0x1a5) [0x7f835c10b465]", "(()+0x814a) [0x7f835a51214a]", "(clone()+0x43) [0x7f8359032f23]" ], "ceph_version": "15.2.8", "crash_id": "2021-02-10T20:00:32.989661Z_201fd5fb-6e0a-4b50-8a95-fdf9ed9aeb81", "entity_name": "mgr.sds01-cp.cwcxek", "os_id": "centos", "os_name": "CentOS Linux", "os_version": "8", "os_version_id": "8", "process_name": "ceph-mgr", "stack_sig": "e1c15d685283e7598b128a37a328ba86ec433dfef97597ac9453b5d52608feda", "timestamp": "2021-02-10T20:00:32.989661Z", "utsname_hostname": "sds01-cp", "utsname_machine": "x86_64", "utsname_release": "4.18.0-240.10.1.el8_3.x86_64", "utsname_sysname": "Linux", "utsname_version": "#1 SMP Wed Dec 16 03:30:52 EST 2020" }

4 5

'ceph df' %USED explanation
by Mark Johnson 28 Feb '21

28 Feb '21

I'm in the middle of increasing PG count for one of our pools by making small increments, waiting for the process to complete, rinse and repeat. I'm doing it this way so I can control when all this activity is happening and keeping it away from the busier production traffic times. I'm expecting some inbalance as PGs get created on already unbalanced OSDs, however our monitoring picked up something today that I'm not really understanding. Our total utilization is just over 50% and about 96% of our total data is in this one pool. Due to there not being enough PGs, the amount of data in each is quite large and since they aren't evenly spread across the OSDs, there's a bit of inbalance. That's all cool and to be expected, which is the reason for increasing the PG count in the first place. However, as some PGs are splitting, the new PGs are sometimes being created on OSDs that already have a disproportionate amount of data. Again, not totally unexpected. Our monitoring detected the usage of this pool to be >85% today as I neared the end of another increase in PG count. What I'm not understanding is how this value is determined. I've read other posts and the calculations suggested don't give a result that equals what shows in my %USED column. I'm suspecting that it's somehow related to the MAX AVAIL value (which I believe is somewhat indirectly related to the amount available based on the individual OSD utilization), but none of the posts I read mention this in their calculations and I've been unable to create a formula with any of the values I have to end up with the &USED value I have. For the record, my current total utilization based on a 'ceph osd df' looks like this: TOTAL 39507G(SIZE) 19931G(USE) 17568G(AVAIL) 50.45(%USE) My most utilised OSD (currently in the process of moving some data off this OSD) is 81.58% used with 188G available and a variance of 1.62. A cut-down output of 'ceph df' looks like this: GLOBAL: SIZE AVAIL RAW USED %RAW USED 39507G 17569G 19930G 50.45 POOLS: NAME ID USED %USED MAX AVAIL OBJECTS default.rgw.buckets.data 30 9552G 86.05 1548G 36285066 I suspect that as I get the utilization of my over-utilized OSDs down, this %USED value will drop. But, I'd just love to fully understand how this value is calculated. Thanks, Mark J

1 0

Getting started with cephadm
by Peter Childs 28 Feb '21

28 Feb '21

I'm new to ceph, and I've been trying to set up a new cluster with 16 computers with 30 disks each and 6 SSD (plus boot disks), 256G of memory, IB Networking. (ok its currently 15 but never mind) When I take them over about 10 OSD's each they start having problems starting the OSD up and I can normally fix this by rebooting them and it will continue again for a while, and it is possible to get them up to the full complement with a bit of poking around. (Once its working it fne unless you start adding services or moving the OSD's around Is there anything I can change to make it a bit more stable. I've already set fs.aio-max-nr = 1048576 kernel.pid_max = 4194303 fs.file-max = 500000 which made it a bit better, but I feel it could be even better. I'm currently trying to upgrade to 15.2.9 from the default cephadm version of octopus. The upgrade is going very very slowly. I'm currently using podman if that helps, I'm not sure if docker would be better? (I've mainly used singularity when I've handled containers before) Thanks in advance Peter Childs

2 4

Slow cluster / misplaced objects - Ceph 15.2.9
by David Orman 27 Feb '21

27 Feb '21

Hi, We've got an interesting issue we're running into on Ceph 15.2.9. We're experiencing VERY slow performance from the cluster, and extremely slow misplaced object correction, with very little cpu/disk/network utilization (almost idle) across all nodes in the cluster. We have 7 servers in this cluster, 24 rotational OSDs, and two NVMEs with 12 OSD's worth of DB/WAL files on them. The OSDs are all equal weighted, so the tree is pretty straightforward: root@ceph01:~# ceph osd tree Inferring fsid 41bb9256-c3bf-11ea-85b9-9e07b0435492 Inferring config /var/lib/ceph/41bb9256-c3bf-11ea-85b9-9e07b0435492/mon.ceph01/config Using recent ceph image docker.io/ceph/ceph@sha256:4e710662986cf366c282323bfb4c4ca507d7e117c5ccf691a8273732073297e5 ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 2149.39062 root default -2 2149.39062 rack rack1 -5 307.05579 host ceph01 0 hdd 12.79399 osd.0 up 1.00000 1.00000 1 hdd 12.79399 osd.1 up 1.00000 1.00000 2 hdd 12.79399 osd.2 up 1.00000 1.00000 3 hdd 12.79399 osd.3 up 1.00000 1.00000 4 hdd 12.79399 osd.4 up 1.00000 1.00000 5 hdd 12.79399 osd.5 up 1.00000 1.00000 6 hdd 12.79399 osd.6 up 1.00000 1.00000 7 hdd 12.79399 osd.7 up 1.00000 1.00000 8 hdd 12.79399 osd.8 up 1.00000 1.00000 9 hdd 12.79399 osd.9 up 1.00000 1.00000 10 hdd 12.79399 osd.10 up 1.00000 1.00000 11 hdd 12.79399 osd.11 up 1.00000 1.00000 12 hdd 12.79399 osd.12 up 1.00000 1.00000 13 hdd 12.79399 osd.13 up 1.00000 1.00000 14 hdd 12.79399 osd.14 up 1.00000 1.00000 15 hdd 12.79399 osd.15 up 1.00000 1.00000 16 hdd 12.79399 osd.16 up 1.00000 1.00000 17 hdd 12.79399 osd.17 up 1.00000 1.00000 18 hdd 12.79399 osd.18 up 1.00000 1.00000 19 hdd 12.79399 osd.19 up 1.00000 1.00000 20 hdd 12.79399 osd.20 up 1.00000 1.00000 21 hdd 12.79399 osd.21 up 1.00000 1.00000 22 hdd 12.79399 osd.22 up 1.00000 1.00000 23 hdd 12.79399 osd.23 up 1.00000 1.00000 -7 307.05579 host ceph02 24 hdd 12.79399 osd.24 up 1.00000 1.00000 25 hdd 12.79399 osd.25 up 1.00000 1.00000 26 hdd 12.79399 osd.26 up 1.00000 1.00000 27 hdd 12.79399 osd.27 up 1.00000 1.00000 28 hdd 12.79399 osd.28 up 1.00000 1.00000 29 hdd 12.79399 osd.29 up 1.00000 1.00000 30 hdd 12.79399 osd.30 up 1.00000 1.00000 31 hdd 12.79399 osd.31 up 1.00000 1.00000 32 hdd 12.79399 osd.32 up 1.00000 1.00000 33 hdd 12.79399 osd.33 up 1.00000 1.00000 34 hdd 12.79399 osd.34 up 1.00000 1.00000 35 hdd 12.79399 osd.35 up 1.00000 1.00000 36 hdd 12.79399 osd.36 up 1.00000 1.00000 37 hdd 12.79399 osd.37 up 1.00000 1.00000 38 hdd 12.79399 osd.38 up 1.00000 1.00000 39 hdd 12.79399 osd.39 up 1.00000 1.00000 40 hdd 12.79399 osd.40 up 1.00000 1.00000 41 hdd 12.79399 osd.41 up 1.00000 1.00000 42 hdd 12.79399 osd.42 up 1.00000 1.00000 43 hdd 12.79399 osd.43 up 1.00000 1.00000 44 hdd 12.79399 osd.44 up 1.00000 1.00000 45 hdd 12.79399 osd.45 up 1.00000 1.00000 46 hdd 12.79399 osd.46 up 1.00000 1.00000 47 hdd 12.79399 osd.47 up 1.00000 1.00000 -9 307.05579 host ceph03 48 hdd 12.79399 osd.48 up 1.00000 1.00000 49 hdd 12.79399 osd.49 up 1.00000 1.00000 50 hdd 12.79399 osd.50 up 1.00000 1.00000 51 hdd 12.79399 osd.51 up 1.00000 1.00000 52 hdd 12.79399 osd.52 up 1.00000 1.00000 53 hdd 12.79399 osd.53 up 1.00000 1.00000 54 hdd 12.79399 osd.54 up 1.00000 1.00000 55 hdd 12.79399 osd.55 up 1.00000 1.00000 56 hdd 12.79399 osd.56 up 1.00000 1.00000 57 hdd 12.79399 osd.57 up 1.00000 1.00000 58 hdd 12.79399 osd.58 up 1.00000 1.00000 59 hdd 12.79399 osd.59 up 1.00000 1.00000 60 hdd 12.79399 osd.60 up 1.00000 1.00000 61 hdd 12.79399 osd.61 up 1.00000 1.00000 62 hdd 12.79399 osd.62 up 1.00000 1.00000 63 hdd 12.79399 osd.63 up 1.00000 1.00000 64 hdd 12.79399 osd.64 up 1.00000 1.00000 65 hdd 12.79399 osd.65 up 1.00000 1.00000 66 hdd 12.79399 osd.66 up 1.00000 1.00000 67 hdd 12.79399 osd.67 up 1.00000 1.00000 68 hdd 12.79399 osd.68 up 1.00000 1.00000 69 hdd 12.79399 osd.69 up 1.00000 1.00000 70 hdd 12.79399 osd.70 up 1.00000 1.00000 71 hdd 12.79399 osd.71 up 1.00000 1.00000 -11 307.05579 host ceph04 72 hdd 12.79399 osd.72 up 1.00000 1.00000 73 hdd 12.79399 osd.73 up 1.00000 1.00000 74 hdd 12.79399 osd.74 up 1.00000 1.00000 75 hdd 12.79399 osd.75 up 1.00000 1.00000 76 hdd 12.79399 osd.76 up 1.00000 1.00000 77 hdd 12.79399 osd.77 up 1.00000 1.00000 78 hdd 12.79399 osd.78 up 1.00000 1.00000 79 hdd 12.79399 osd.79 up 1.00000 1.00000 80 hdd 12.79399 osd.80 up 1.00000 1.00000 81 hdd 12.79399 osd.81 up 1.00000 1.00000 82 hdd 12.79399 osd.82 up 1.00000 1.00000 83 hdd 12.79399 osd.83 up 1.00000 1.00000 84 hdd 12.79399 osd.84 up 1.00000 1.00000 85 hdd 12.79399 osd.85 up 1.00000 1.00000 86 hdd 12.79399 osd.86 up 1.00000 1.00000 87 hdd 12.79399 osd.87 up 1.00000 1.00000 88 hdd 12.79399 osd.88 up 1.00000 1.00000 89 hdd 12.79399 osd.89 up 1.00000 1.00000 90 hdd 12.79399 osd.90 up 1.00000 1.00000 91 hdd 12.79399 osd.91 up 1.00000 1.00000 92 hdd 12.79399 osd.92 up 1.00000 1.00000 93 hdd 12.79399 osd.93 up 1.00000 1.00000 94 hdd 12.79399 osd.94 up 1.00000 1.00000 95 hdd 12.79399 osd.95 up 1.00000 1.00000 -13 307.05579 host ceph05 96 hdd 12.79399 osd.96 up 1.00000 1.00000 97 hdd 12.79399 osd.97 up 1.00000 1.00000 98 hdd 12.79399 osd.98 up 1.00000 1.00000 99 hdd 12.79399 osd.99 up 1.00000 1.00000 100 hdd 12.79399 osd.100 up 1.00000 1.00000 101 hdd 12.79399 osd.101 up 1.00000 1.00000 102 hdd 12.79399 osd.102 up 1.00000 1.00000 103 hdd 12.79399 osd.103 up 1.00000 1.00000 104 hdd 12.79399 osd.104 up 1.00000 1.00000 105 hdd 12.79399 osd.105 up 1.00000 1.00000 106 hdd 12.79399 osd.106 up 1.00000 1.00000 107 hdd 12.79399 osd.107 up 1.00000 1.00000 108 hdd 12.79399 osd.108 up 1.00000 1.00000 109 hdd 12.79399 osd.109 up 1.00000 1.00000 110 hdd 12.79399 osd.110 up 1.00000 1.00000 111 hdd 12.79399 osd.111 up 1.00000 1.00000 112 hdd 12.79399 osd.112 up 1.00000 1.00000 113 hdd 12.79399 osd.113 up 1.00000 1.00000 114 hdd 12.79399 osd.114 up 1.00000 1.00000 115 hdd 12.79399 osd.115 up 1.00000 1.00000 116 hdd 12.79399 osd.116 up 1.00000 1.00000 117 hdd 12.79399 osd.117 up 1.00000 1.00000 118 hdd 12.79399 osd.118 up 1.00000 1.00000 119 hdd 12.79399 osd.119 up 1.00000 1.00000 -15 307.05579 host ceph06 120 hdd 12.79399 osd.120 up 1.00000 1.00000 121 hdd 12.79399 osd.121 up 1.00000 1.00000 122 hdd 12.79399 osd.122 up 1.00000 1.00000 123 hdd 12.79399 osd.123 up 1.00000 1.00000 124 hdd 12.79399 osd.124 up 1.00000 1.00000 125 hdd 12.79399 osd.125 up 1.00000 1.00000 126 hdd 12.79399 osd.126 up 1.00000 1.00000 127 hdd 12.79399 osd.127 up 1.00000 1.00000 128 hdd 12.79399 osd.128 up 1.00000 1.00000 129 hdd 12.79399 osd.129 up 1.00000 1.00000 130 hdd 12.79399 osd.130 up 1.00000 1.00000 131 hdd 12.79399 osd.131 up 1.00000 1.00000 132 hdd 12.79399 osd.132 up 1.00000 1.00000 133 hdd 12.79399 osd.133 up 1.00000 1.00000 134 hdd 12.79399 osd.134 up 1.00000 1.00000 135 hdd 12.79399 osd.135 up 1.00000 1.00000 136 hdd 12.79399 osd.136 up 1.00000 1.00000 137 hdd 12.79399 osd.137 up 1.00000 1.00000 138 hdd 12.79399 osd.138 up 1.00000 1.00000 139 hdd 12.79399 osd.139 up 1.00000 1.00000 140 hdd 12.79399 osd.140 up 1.00000 1.00000 141 hdd 12.79399 osd.141 up 1.00000 1.00000 142 hdd 12.79399 osd.142 up 1.00000 1.00000 143 hdd 12.79399 osd.143 up 1.00000 1.00000 -17 307.05579 host ceph07 144 hdd 12.79399 osd.144 up 1.00000 1.00000 145 hdd 12.79399 osd.145 up 1.00000 1.00000 146 hdd 12.79399 osd.146 up 1.00000 1.00000 147 hdd 12.79399 osd.147 up 1.00000 1.00000 148 hdd 12.79399 osd.148 up 1.00000 1.00000 149 hdd 12.79399 osd.149 up 1.00000 1.00000 150 hdd 12.79399 osd.150 up 1.00000 1.00000 151 hdd 12.79399 osd.151 up 1.00000 1.00000 152 hdd 12.79399 osd.152 up 1.00000 1.00000 153 hdd 12.79399 osd.153 up 1.00000 1.00000 154 hdd 12.79399 osd.154 up 1.00000 1.00000 155 hdd 12.79399 osd.155 up 1.00000 1.00000 156 hdd 12.79399 osd.156 up 1.00000 1.00000 157 hdd 12.79399 osd.157 up 1.00000 1.00000 158 hdd 12.79399 osd.158 up 1.00000 1.00000 159 hdd 12.79399 osd.159 up 1.00000 1.00000 160 hdd 12.79399 osd.160 up 1.00000 1.00000 161 hdd 12.79399 osd.161 up 1.00000 1.00000 162 hdd 12.79399 osd.162 up 1.00000 1.00000 163 hdd 12.79399 osd.163 up 1.00000 1.00000 164 hdd 12.79399 osd.164 up 1.00000 1.00000 165 hdd 12.79399 osd.165 up 1.00000 1.00000 166 hdd 12.79399 osd.166 up 1.00000 1.00000 167 hdd 12.79399 osd.167 up 1.00000 1.00000 root@ceph01:~# The data distribution looks relatively even (we use upmap as the balancer): root@ceph01:~# ceph osd df Inferring fsid 41bb9256-c3bf-11ea-85b9-9e07b0435492 Inferring config /var/lib/ceph/41bb9256-c3bf-11ea-85b9-9e07b0435492/mon.ceph01/config Using recent ceph image docker.io/ceph/ceph@sha256:4e710662986cf366c282323bfb4c4ca507d7e117c5ccf691a8273732073297e5 ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 0 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 5.0 MiB 14 GiB 7.1 TiB 44.73 0.93 35 up 1 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 1.8 MiB 13 GiB 7.2 TiB 43.46 0.90 33 up 2 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 4.4 MiB 16 GiB 6.2 TiB 51.27 1.07 39 up 3 hdd 12.79399 1.00000 13 TiB 6.1 TiB 6.0 TiB 4.9 MiB 15 GiB 6.7 TiB 47.38 0.99 36 up 4 hdd 12.79399 1.00000 13 TiB 5.5 TiB 5.5 TiB 812 KiB 13 GiB 7.2 TiB 43.37 0.90 35 up 5 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 31 KiB 16 GiB 6.2 TiB 51.27 1.07 40 up 6 hdd 12.79399 1.00000 13 TiB 5.5 TiB 5.5 TiB 506 KiB 13 GiB 7.3 TiB 43.32 0.90 34 up 7 hdd 12.79399 1.00000 13 TiB 6.4 TiB 6.3 TiB 904 KiB 15 GiB 6.4 TiB 49.93 1.04 38 up 8 hdd 12.79399 1.00000 13 TiB 6.1 TiB 6.0 TiB 3.6 MiB 14 GiB 6.7 TiB 47.35 0.98 37 up 9 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 5.5 MiB 16 GiB 6.1 TiB 52.58 1.09 40 up 10 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.6 TiB 75 MiB 16 GiB 6.1 TiB 52.42 1.09 41 up 11 hdd 12.79399 1.00000 13 TiB 5.5 TiB 5.5 TiB 1.4 MiB 13 GiB 7.2 TiB 43.36 0.90 34 up 12 hdd 12.79399 1.00000 13 TiB 5.5 TiB 5.5 TiB 586 KiB 13 GiB 7.3 TiB 43.31 0.90 33 up 13 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 222 KiB 14 GiB 7.1 TiB 44.71 0.93 35 up 14 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 1.0 MiB 16 GiB 6.1 TiB 52.58 1.09 41 up 15 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 4.2 MiB 14 GiB 7.1 TiB 44.73 0.93 34 up 16 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 1.0 MiB 16 GiB 6.1 TiB 52.53 1.09 40 up 17 hdd 12.79399 1.00000 13 TiB 6.4 TiB 6.3 TiB 3.3 MiB 15 GiB 6.4 TiB 49.92 1.04 38 up 18 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 1.0 MiB 16 GiB 6.1 TiB 52.61 1.09 45 up 19 hdd 12.79399 1.00000 13 TiB 5.4 TiB 5.3 TiB 4.0 MiB 13 GiB 7.4 TiB 42.15 0.88 34 up 20 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 705 KiB 16 GiB 6.1 TiB 52.53 1.09 40 up 21 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 2.5 MiB 16 GiB 6.1 TiB 52.58 1.09 40 up 22 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 4.7 MiB 16 GiB 6.1 TiB 52.59 1.09 40 up 23 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 4.9 MiB 16 GiB 6.1 TiB 52.58 1.09 41 up 24 hdd 12.79399 1.00000 13 TiB 6.1 TiB 6.0 TiB 3.0 MiB 14 GiB 6.7 TiB 47.35 0.98 36 up 25 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 4.1 MiB 16 GiB 6.1 TiB 52.56 1.09 41 up 26 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 592 KiB 14 GiB 7.1 TiB 44.75 0.93 34 up 27 hdd 12.79399 1.00000 13 TiB 5.9 TiB 5.8 TiB 2.8 MiB 14 GiB 6.9 TiB 46.07 0.96 36 up 28 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 5.4 MiB 16 GiB 6.1 TiB 52.56 1.09 40 up 29 hdd 12.79399 1.00000 13 TiB 6.4 TiB 6.3 TiB 340 KiB 15 GiB 6.4 TiB 49.92 1.04 39 up 30 hdd 12.79399 1.00000 13 TiB 5.9 TiB 5.8 TiB 93 KiB 14 GiB 6.9 TiB 46.03 0.96 35 up 31 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 2.1 MiB 13 GiB 7.2 TiB 43.57 0.91 35 up 32 hdd 12.79399 1.00000 13 TiB 6.2 TiB 6.2 TiB 5.0 MiB 15 GiB 6.6 TiB 48.68 1.01 38 up 33 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 175 KiB 13 GiB 7.2 TiB 43.44 0.90 33 up 34 hdd 12.79399 1.00000 13 TiB 6.9 TiB 6.8 TiB 77 MiB 16 GiB 5.9 TiB 53.86 1.12 43 up 35 hdd 12.79399 1.00000 13 TiB 6.2 TiB 6.2 TiB 2.3 MiB 15 GiB 6.6 TiB 48.68 1.01 37 up 36 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 1.2 MiB 16 GiB 6.2 TiB 51.25 1.07 41 up 37 hdd 12.79399 1.00000 13 TiB 6.1 TiB 6.0 TiB 5.0 MiB 14 GiB 6.7 TiB 47.37 0.99 36 up 38 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 3.7 MiB 13 GiB 7.2 TiB 43.44 0.90 35 up 39 hdd 12.79399 1.00000 13 TiB 6.1 TiB 6.0 TiB 341 KiB 15 GiB 6.7 TiB 47.38 0.99 37 up 40 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 4.1 MiB 14 GiB 7.1 TiB 44.73 0.93 34 up 41 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 22 KiB 16 GiB 6.1 TiB 52.59 1.09 40 up 42 hdd 12.79399 1.00000 13 TiB 6.1 TiB 6.0 TiB 1.9 MiB 14 GiB 6.7 TiB 47.33 0.98 36 up 43 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 485 KiB 16 GiB 6.1 TiB 52.59 1.09 40 up 44 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 2.1 MiB 16 GiB 6.1 TiB 52.56 1.09 40 up 45 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 2.3 MiB 16 GiB 6.1 TiB 52.53 1.09 40 up 46 hdd 12.79399 1.00000 13 TiB 5.9 TiB 5.8 TiB 124 KiB 14 GiB 6.9 TiB 46.08 0.96 37 up 47 hdd 12.79399 1.00000 13 TiB 5.5 TiB 5.5 TiB 1.2 MiB 13 GiB 7.2 TiB 43.37 0.90 33 up 48 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 141 KiB 16 GiB 6.1 TiB 52.58 1.09 40 up 49 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 1.9 MiB 13 GiB 7.2 TiB 43.64 0.91 33 up 50 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 745 KiB 16 GiB 6.1 TiB 52.52 1.09 41 up 51 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 186 KiB 16 GiB 6.1 TiB 52.55 1.09 41 up 52 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 4.0 MiB 16 GiB 6.2 TiB 51.24 1.07 41 up 53 hdd 12.79399 1.00000 13 TiB 6.1 TiB 6.0 TiB 737 KiB 14 GiB 6.7 TiB 47.33 0.98 38 up 54 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 1.3 MiB 14 GiB 7.1 TiB 44.66 0.93 34 up 55 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 97 KiB 14 GiB 7.1 TiB 44.73 0.93 34 up 56 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 1.2 MiB 14 GiB 7.1 TiB 44.76 0.93 35 up 57 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 3.5 MiB 14 GiB 7.1 TiB 44.78 0.93 35 up 58 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 80 MiB 16 GiB 6.1 TiB 52.56 1.09 41 up 59 hdd 12.79399 1.00000 13 TiB 5.9 TiB 5.8 TiB 4.5 MiB 14 GiB 6.9 TiB 46.08 0.96 36 up 60 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 903 KiB 14 GiB 7.1 TiB 44.73 0.93 34 up 61 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 1 KiB 14 GiB 7.1 TiB 44.65 0.93 34 up 62 hdd 12.79399 1.00000 13 TiB 6.9 TiB 6.8 TiB 425 KiB 16 GiB 5.9 TiB 53.87 1.12 41 up 63 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 4.1 MiB 14 GiB 7.2 TiB 43.56 0.91 34 up 64 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 578 KiB 16 GiB 6.2 TiB 51.27 1.07 39 up 65 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 3.6 MiB 13 GiB 7.2 TiB 43.44 0.90 33 up 66 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 288 KiB 13 GiB 7.2 TiB 43.40 0.90 34 up 67 hdd 12.79399 1.00000 13 TiB 6.1 TiB 6.0 TiB 413 KiB 14 GiB 6.7 TiB 47.34 0.98 37 up 68 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 4.3 MiB 16 GiB 6.1 TiB 52.56 1.09 41 up 69 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 1.0 MiB 13 GiB 7.2 TiB 43.44 0.90 33 up 70 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 3.2 MiB 16 GiB 6.1 TiB 52.56 1.09 40 up 71 hdd 12.79399 1.00000 13 TiB 5.5 TiB 5.5 TiB 1.3 MiB 14 GiB 7.2 TiB 43.36 0.90 35 up 72 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 4.7 MiB 16 GiB 6.2 TiB 51.29 1.07 39 up 73 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 1.8 MiB 14 GiB 7.1 TiB 44.73 0.93 34 up 74 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.6 TiB 1.2 MiB 14 GiB 7.1 TiB 44.64 0.93 34 up 75 hdd 12.79399 1.00000 13 TiB 6.4 TiB 6.3 TiB 1.5 MiB 15 GiB 6.4 TiB 49.95 1.04 38 up 76 hdd 12.79399 1.00000 13 TiB 6.2 TiB 6.2 TiB 3.8 MiB 15 GiB 6.6 TiB 48.68 1.01 37 up 77 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 5.0 MiB 14 GiB 7.1 TiB 44.65 0.93 34 up 78 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 1.8 MiB 14 GiB 7.1 TiB 44.68 0.93 35 up 79 hdd 12.79399 1.00000 13 TiB 6.1 TiB 6.0 TiB 984 KiB 14 GiB 6.7 TiB 47.36 0.98 36 up 80 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 2.6 MiB 13 GiB 7.2 TiB 43.45 0.90 34 up 81 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 1.2 MiB 16 GiB 6.2 TiB 51.26 1.07 41 up 82 hdd 12.79399 1.00000 13 TiB 6.2 TiB 6.2 TiB 4.9 MiB 15 GiB 6.6 TiB 48.61 1.01 37 up 83 hdd 12.79399 1.00000 13 TiB 6.1 TiB 6.0 TiB 827 KiB 14 GiB 6.7 TiB 47.33 0.98 36 up 84 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 2.7 MiB 16 GiB 6.2 TiB 51.22 1.07 39 up 85 hdd 12.79399 1.00000 13 TiB 6.4 TiB 6.3 TiB 868 KiB 15 GiB 6.4 TiB 49.94 1.04 40 up 86 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 4.1 MiB 16 GiB 6.1 TiB 52.57 1.09 42 up 87 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 702 KiB 16 GiB 6.1 TiB 52.54 1.09 41 up 88 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 12 KiB 14 GiB 7.1 TiB 44.76 0.93 35 up 89 hdd 12.79399 1.00000 13 TiB 5.9 TiB 5.8 TiB 2.9 MiB 14 GiB 6.9 TiB 46.07 0.96 35 up 90 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 1.8 MiB 13 GiB 7.2 TiB 43.48 0.90 34 up 91 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 836 KiB 16 GiB 6.1 TiB 52.55 1.09 40 up 92 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 4.0 MiB 13 GiB 7.2 TiB 43.43 0.90 33 up 93 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 1.5 MiB 16 GiB 6.1 TiB 52.57 1.09 41 up 94 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 459 KiB 13 GiB 7.2 TiB 43.44 0.90 34 up 95 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 4.1 MiB 16 GiB 6.2 TiB 51.28 1.07 39 up 96 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 4.1 MiB 14 GiB 7.1 TiB 44.75 0.93 35 up 97 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 662 KiB 14 GiB 7.1 TiB 44.75 0.93 34 up 98 hdd 12.79399 1.00000 13 TiB 5.9 TiB 5.8 TiB 2.9 MiB 14 GiB 6.9 TiB 45.97 0.96 35 up 99 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 3.8 MiB 16 GiB 6.1 TiB 52.56 1.09 40 up 100 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 1.8 MiB 14 GiB 7.1 TiB 44.76 0.93 36 up 101 hdd 12.79399 1.00000 13 TiB 6.4 TiB 6.3 TiB 3.6 MiB 15 GiB 6.4 TiB 49.95 1.04 40 up 102 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 4.9 MiB 14 GiB 7.1 TiB 44.65 0.93 34 up 103 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 643 KiB 16 GiB 6.1 TiB 52.57 1.09 41 up 104 hdd 12.79399 1.00000 13 TiB 6.4 TiB 6.3 TiB 5.5 MiB 15 GiB 6.4 TiB 49.98 1.04 38 up 105 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 830 KiB 13 GiB 7.2 TiB 43.46 0.90 33 up 106 hdd 12.79399 1.00000 13 TiB 6.1 TiB 6.0 TiB 1.0 MiB 14 GiB 6.7 TiB 47.34 0.98 36 up 107 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 286 KiB 16 GiB 6.2 TiB 51.23 1.07 39 up 108 hdd 12.79399 1.00000 13 TiB 6.2 TiB 6.2 TiB 961 KiB 15 GiB 6.6 TiB 48.68 1.01 37 up 109 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 291 KiB 16 GiB 6.1 TiB 52.56 1.09 42 up 110 hdd 12.79399 1.00000 13 TiB 5.9 TiB 5.8 TiB 1.0 MiB 14 GiB 6.9 TiB 46.02 0.96 36 up 111 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 2.7 MiB 16 GiB 6.2 TiB 51.22 1.07 39 up 112 hdd 12.79399 1.00000 13 TiB 6.2 TiB 6.2 TiB 631 KiB 15 GiB 6.6 TiB 48.67 1.01 37 up 113 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 4.5 MiB 16 GiB 6.1 TiB 52.56 1.09 41 up 114 hdd 12.79399 1.00000 13 TiB 6.4 TiB 6.3 TiB 4.0 MiB 15 GiB 6.4 TiB 49.94 1.04 39 up 115 hdd 12.79399 1.00000 13 TiB 5.9 TiB 5.8 TiB 1.5 MiB 14 GiB 6.9 TiB 46.05 0.96 35 up 116 hdd 12.79399 1.00000 13 TiB 6.4 TiB 6.3 TiB 4.1 MiB 15 GiB 6.4 TiB 49.93 1.04 38 up 117 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 194 KiB 13 GiB 7.2 TiB 43.44 0.90 35 up 118 hdd 12.79399 1.00000 13 TiB 6.4 TiB 6.3 TiB 3.1 MiB 15 GiB 6.4 TiB 49.96 1.04 38 up 119 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 3.9 MiB 14 GiB 7.2 TiB 43.46 0.90 33 up 120 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 1.8 MiB 16 GiB 6.1 TiB 52.55 1.09 40 up 121 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 3.7 MiB 16 GiB 6.1 TiB 52.54 1.09 40 up 122 hdd 12.79399 1.00000 13 TiB 6.4 TiB 6.3 TiB 2.4 MiB 15 GiB 6.4 TiB 49.97 1.04 38 up 123 hdd 12.79399 1.00000 13 TiB 5.9 TiB 5.8 TiB 1003 KiB 14 GiB 6.9 TiB 46.02 0.96 35 up 124 hdd 12.79399 1.00000 13 TiB 6.1 TiB 6.0 TiB 526 KiB 14 GiB 6.7 TiB 47.35 0.98 36 up 125 hdd 12.79399 1.00000 13 TiB 6.1 TiB 6.0 TiB 92 KiB 15 GiB 6.7 TiB 47.37 0.98 37 up 126 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 4.9 MiB 14 GiB 7.1 TiB 44.74 0.93 34 up 127 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 5.5 MiB 16 GiB 6.2 TiB 51.26 1.07 39 up 128 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 5.4 MiB 16 GiB 6.1 TiB 52.56 1.09 40 up 129 hdd 12.79399 1.00000 13 TiB 6.2 TiB 6.2 TiB 3.6 MiB 15 GiB 6.6 TiB 48.66 1.01 38 up 130 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 2.8 MiB 16 GiB 6.1 TiB 52.56 1.09 41 up 131 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 5.1 MiB 16 GiB 6.2 TiB 51.28 1.07 40 up 132 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 1.7 MiB 16 GiB 6.2 TiB 51.25 1.07 39 up 133 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 230 KiB 16 GiB 6.2 TiB 51.25 1.07 39 up 134 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 2.0 MiB 14 GiB 7.2 TiB 43.45 0.90 34 up 135 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 4.4 MiB 13 GiB 7.2 TiB 43.46 0.90 33 up 136 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 5.6 MiB 16 GiB 6.2 TiB 51.29 1.07 39 up 137 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 3.6 MiB 14 GiB 7.1 TiB 44.75 0.93 36 up 138 hdd 12.79399 1.00000 13 TiB 6.4 TiB 6.3 TiB 4.9 MiB 15 GiB 6.4 TiB 49.98 1.04 40 up 139 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 21 KiB 16 GiB 6.2 TiB 51.23 1.07 39 up 140 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 5.4 MiB 14 GiB 7.1 TiB 44.74 0.93 36 up 141 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 359 KiB 13 GiB 7.2 TiB 43.42 0.90 34 up 142 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 27 KiB 13 GiB 7.2 TiB 43.46 0.90 33 up 143 hdd 12.79399 1.00000 13 TiB 5.9 TiB 5.8 TiB 395 KiB 14 GiB 6.9 TiB 46.07 0.96 35 up 144 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 1.2 MiB 14 GiB 7.1 TiB 44.75 0.93 34 up 145 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 3.3 MiB 14 GiB 7.1 TiB 44.66 0.93 35 up 146 hdd 12.79399 1.00000 13 TiB 5.9 TiB 5.8 TiB 4.0 MiB 14 GiB 6.9 TiB 46.06 0.96 35 up 147 hdd 12.79399 1.00000 13 TiB 6.7 TiB 6.7 TiB 46 KiB 16 GiB 6.1 TiB 52.55 1.09 40 up 148 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 4.8 MiB 16 GiB 6.2 TiB 51.23 1.07 39 up 149 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 4.4 MiB 14 GiB 7.1 TiB 44.70 0.93 35 up 150 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 4.4 MiB 14 GiB 7.1 TiB 44.65 0.93 35 up 151 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 3.7 MiB 14 GiB 7.1 TiB 44.65 0.93 35 up 152 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 5.3 MiB 15 GiB 6.2 TiB 51.26 1.07 39 up 153 hdd 12.79399 1.00000 13 TiB 6.4 TiB 6.3 TiB 2.3 MiB 15 GiB 6.4 TiB 49.94 1.04 39 up 154 hdd 12.79399 1.00000 13 TiB 6.1 TiB 6.0 TiB 3.0 MiB 14 GiB 6.7 TiB 47.37 0.99 36 up 155 hdd 12.79399 1.00000 13 TiB 6.1 TiB 6.0 TiB 3.0 MiB 14 GiB 6.7 TiB 47.35 0.98 38 up 156 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 4.0 MiB 16 GiB 6.2 TiB 51.26 1.07 41 up 157 hdd 12.79399 1.00000 13 TiB 5.9 TiB 5.8 TiB 2.8 MiB 14 GiB 6.9 TiB 46.07 0.96 36 up 158 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 3.0 MiB 16 GiB 6.2 TiB 51.25 1.07 40 up 159 hdd 12.79399 1.00000 13 TiB 6.4 TiB 6.3 TiB 3.9 MiB 15 GiB 6.4 TiB 49.93 1.04 38 up 160 hdd 12.79399 1.00000 13 TiB 5.9 TiB 5.8 TiB 2.4 MiB 14 GiB 6.9 TiB 46.03 0.96 35 up 161 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 1.1 MiB 16 GiB 6.2 TiB 51.25 1.07 39 up 162 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.6 TiB 4.1 MiB 13 GiB 7.1 TiB 44.64 0.93 34 up 163 hdd 12.79399 1.00000 13 TiB 5.7 TiB 5.7 TiB 445 KiB 14 GiB 7.1 TiB 44.69 0.93 34 up 164 hdd 12.79399 1.00000 13 TiB 6.4 TiB 6.3 TiB 4.9 MiB 15 GiB 6.4 TiB 49.97 1.04 38 up 165 hdd 12.79399 1.00000 13 TiB 6.6 TiB 6.5 TiB 5.5 MiB 16 GiB 6.2 TiB 51.25 1.07 40 up 166 hdd 12.79399 1.00000 13 TiB 5.6 TiB 5.5 TiB 2.5 MiB 13 GiB 7.2 TiB 43.45 0.90 33 up 167 hdd 12.79399 1.00000 13 TiB 6.4 TiB 6.3 TiB 4.0 MiB 15 GiB 6.4 TiB 49.98 1.04 38 up TOTAL 2.1 PiB 1.0 PiB 1023 TiB 630 MiB 2.4 TiB 1.1 PiB 48.09 MIN/MAX VAR: 0.88/1.12 STDDEV: 3.50 root@ceph01:~# The cluster status, however, shows we have many misplaced objects and a few remapped PGs. We had been performing maintenance (upgrade to Ceph 15.2.9), and initially the misplaced object count was dropping rapidly at a rate of about 400-500MB/s and 100-200 objects/s. However, once it hit a certain point, it dropped to 0-4MB/s and 0 objects/s (it's not 0, it is progressing, but it's below 1 object/s): root@ceph01:~# ceph -s Inferring fsid 41bb9256-c3bf-11ea-85b9-9e07b0435492 Inferring config /var/lib/ceph/41bb9256-c3bf-11ea-85b9-9e07b0435492/mon.ceph01/config Using recent ceph image docker.io/ceph/ceph@sha256:4e710662986cf366c282323bfb4c4ca507d7e117c5ccf691a8273732073297e5 cluster: id: 41bb9256-c3bf-11ea-85b9-9e07b0435492 health: HEALTH_WARN 5 slow ops, oldest one blocked for 84 sec, daemons [osd.141,osd.46] have slow ops. services: mon: 5 daemons, quorum ceph01,ceph04,ceph02,ceph03,ceph05 (age 25h) mgr: ceph05.yropto(active, since 11h), standbys: ceph01.aqkgbl, ceph03.ytkuyr, ceph04.smbdew, ceph02.ndynmo osd: 168 osds: 168 up (since 9h), 168 in (since 6M); 10 remapped pgs task status: data: pools: 3 pools, 1057 pgs objects: 166.90M objects, 632 TiB usage: 1.0 PiB used, 1.1 PiB / 2.1 PiB avail pgs: 583177/1001370417 objects misplaced (0.058%) 1039 active+clean 11 active+clean+scrubbing 5 active+remapped+backfilling 1 active+clean+scrubbing+deep 1 active+clean+laggy io: client: 4.5 MiB/s rd, 31 MiB/s wr, 75 op/s rd, 59 op/s wr recovery: 2.7 MiB/s, 0 objects/s root@ceph01:~# root@ceph01:~# ceph pg dump |grep remapped Inferring fsid 41bb9256-c3bf-11ea-85b9-9e07b0435492 Inferring config /var/lib/ceph/41bb9256-c3bf-11ea-85b9-9e07b0435492/mon.ceph01/config Using recent ceph image docker.io/ceph/ceph@sha256:4e710662986cf366c282323bfb4c4ca507d7e117c5ccf691a8273732073297e5 2.3d1 163805 0 0 0 0 682273001472 0 0 7462 7462 active+*remapped*+backfilling 2021-02-26T01:28:06.212904+0000 53782'2773494 53782:5593409 [48,122,99,92,158,31] 48 [48,122,99,92,158,34] 48 50907'2762333 2021-02-23T20:13:50.501757+0000 50907'2762333 2021-02-23T20:13:50.501757+0000 0 2.341 163180 0 0 136844 0 679698268160 0 0 8569 8569 active+*remapped*+backfilling 2021-02-26T01:27:30.693520+0000 53782'2694466 53782:5497907 [65,125,79,147,45,100] 65 [65,18,79,147,45,100] 65 51073'2688628 2021-02-24T09:38:41.588350+0000 51073'2688628 2021-02-24T09:38:41.588350+0000 0 2.1e1 162402 0 0 150624 0 676505321472 0 0 7893 7893 active+*remapped*+backfilling 2021-02-25T15:36:29.121352+0000 53782'3105605 53782:5885140 [112,63,126,0,147,25] 112 [112,50,126,0,147,25] 112 50907'3086340 2021-02-22T23:14:49.510535+0000 47263'2883486 2021-02-01T15:43:32.089129+0000 0 2.268 163459 0 0 136459 0 680743612416 0 0 7337 7337 active+*remapped*+backfilling 2021-02-25T15:27:58.957497+0000 53782'2713107 53782:5536140 [93,98,150,49,26,135] 93 [93,98,150,62,26,135] 93 50907'2698319 2021-02-23T14:16:11.171370+0000 50907'2698319 2021-02-23T14:16:11.171370+0000 0 2.285 163380 0 0 158866 0 680507670528 0 0 8148 8148 active+*remapped*+backfilling 2021-02-26T01:26:31.442018+0000 53782'3191803 53782:5986426 [67,166,37,90,114,138] 67 [67,147,37,90,114,138] 67 51073'3177251 2021-02-23T23:51:51.751625+0000 47597'2917541 2021-02-03T20:14:38.762384+0000 0 dumped all root@ceph01:~# Even if we stop the scrubs/deep scrubs, things don't improve re: recovery rate or cluster performance. We've monitored the HDDs/NVME backing these OSDs and with the scrubs stopped, they are nearly idle. CPU is almost idle. There's nothing to indicate any problem at all with the underlying hardware that we can find, at least from a load/failure perspective. Also, interestingly, rados bench on a newly created **replicated** pool is also extremely slow, as in < 50MB/s. Crush rule for our EC (4+2) pool, but we see the performance issues on even a newly created triple-replicated pool: root@ceph01:~# ceph osd crush rule dump ecpool Inferring fsid 41bb9256-c3bf-11ea-85b9-9e07b0435492 Inferring config /var/lib/ceph/41bb9256-c3bf-11ea-85b9-9e07b0435492/mon.ceph01/config Using recent ceph image docker.io/ceph/ceph@sha256:4e710662986cf366c282323bfb4c4ca507d7e117c5ccf691a8273732073297e5 { "rule_id": 1, "rule_name": "ecpool", "ruleset": 1, "type": 3, "min_size": 3, "max_size": 6, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_indep", "num": 0, "type": "host" }, { "op": "emit" } ] } Cluster/pool space: root@ceph01:~# ceph df Inferring fsid 41bb9256-c3bf-11ea-85b9-9e07b0435492 Inferring config /var/lib/ceph/41bb9256-c3bf-11ea-85b9-9e07b0435492/mon.ceph01/config Using recent ceph image docker.io/ceph/ceph@sha256:4e710662986cf366c282323bfb4c4ca507d7e117c5ccf691a8273732073297e5 --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 2.1 PiB 1.1 PiB 1.0 PiB 1.0 PiB 48.09 TOTAL 2.1 PiB 1.1 PiB 1.0 PiB 1.0 PiB 48.09 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL device_health_metrics 1 1 78 MiB 182 233 MiB 0 295 TiB ecpool 2 1024 632 TiB 166.90M 1023 TiB 53.65 589 TiB rbd 3 32 59 MiB 5 178 MiB 0 295 TiB root@ceph01:~# We're at a loss for why this cluster has suddenly started behaving this way. With the disks (all of them, on all nodes) effectively being idle with all the scrubs disabled, plenty of CPU available, and no network issues (iperf on all paths is full line rate - ~10GB/s - we have a dedicated public and cluster network), it's befuddling. We appreciate any assistance you can provide, and are happy to provide any information that might be useful.

4 6

MDSs report damaged metadata
by ricardo.re.azevedo＠gmail.com 26 Feb '21

26 Feb '21

Hi all, My cephfs MDS is reporting damaged metadata following the addition (and remapping) of 12 new OSDs. `ceph tell mds.database-0 damage ls` reports ~85 files damaged. All of type "backtrace" which is very concerning. ` ceph tell mds.database-0 scrub start / recursive repair` seems to have no effect on the damage. What does this sort of damage mean? Is there anything I can do to recover these files? > ceph status reports: cluster: id: 692905c0-f271-4cd8-9e43-1c32ef8abd13 health: HEALTH_ERR 1 MDSs report damaged metadata 630 pgs not deep-scrubbed in time 630 pgs not scrubbed in time services: mon: 3 daemons, quorum database-0,file-server,webhost (age 37m) mgr: webhost(active, since 3d), standbys: file-server, database-0 mds: cephfs:1 {0=database-0=up:active} 2 up:standby osd: 48 osds: 48 up (since 56m), 48 in (since 13d); 10 remapped pgs task status: scrub status: mds.database-0: idle data: pools: 7 pools, 633 pgs objects: 60.82M objects, 231 TiB usage: 336 TiB used, 246 TiB / 582 TiB avail pgs: 623 active+clean 6 active+remapped+backfilling 4 active+remapped+backfill_wait Thanks for the help. Best, Ricardo

2 2