February 2020 - ceph-users

Benefits of high RAM on a metadata server?
by Matt Larson 14 Feb '20

14 Feb '20

Hi, we are planning out a Ceph storage cluster and were choosing between 64GB, 128GB, or even 256GB on metadata servers. We are considering having 2 metadata servers overall. Does going to high levels of RAM possibly yield any performance benefits? Is there a size beyond which there are just diminishing returns vs cost? The expected use case would be for a cluster where there might be 10-20 concurrent users working on individual datasets of 5TB in size. I expect there would be lots of reads of the 5TB datasets matched with the creation of hundreds to thousands of smaller files during processing of the images. Thanks! -Matt -- Matt Larson, PhD Madison, WI 53705 U.S.A.

6 7

Strange speed issues with XFS and very small writes
by Arvydas Opulskis 14 Feb '20

14 Feb '20

Hi, Cephers. I would like to hear your ideas about strange situation we have in one of our clusters. It's Luminous 12.2.12 cluster. Recently we added 3 nodes with 10x SSD OSDs to it and dedicated them to SSD pool for our OpenStack volumes. Initial tests went well, IOPS were great, throughput was perfect - all good. Until we got first real usage there. Very limited IOPS (~450), high disk utilization (near 100%) and throughput (less than 1 MB/s) put us into the tears. After some investigation we found, that this situation only occurs when all conditions are met: 1. Disk is RBD (test went fine from same server with local disks) 2. File system is XFS (no problems with ext4) 3. Block size is bigger than write 4. Only one FIO thread (numjobs) is used When at least one of these conditions are not met - we get ~40k IOPS, great throughput, etc. We did tests with fio, testing different values, but it's quite clear: if write size is 4kb (same as block size) iops go up to 40k. If write size is 3kb, then it limits to ~450 iops. From this point, it doesn't matter how small the write is - it's always ~450 iops. After changing block size to 2kb, situation is same - great speed until write is less than 2kb in size. If we rise fio paramether "numjobs" to 10 we get maximum possible iops: ~40k. Which is more than simple 10x increase. Any ideas what is going on and why smaller writes take such a big impact on performance in XFS, but no problems in EXT4? Thank you for all the ideas! Arvydas

1 0

Ceph MDS ASSERT In function 'MDRequestRef'
by Stefan Kooman 13 Feb '20

13 Feb '20

Hi, We hit the following assert: -10001> 2020-02-13 17:42:35.543 7f11b5669700 -1 /build/ceph-13.2.8/src/mds/MDCache.cc: In function 'MDRequestRef MDCa che::request_get(metareqid_t)' thread 7f11b5669700 time 2020-02-13 17:42:35.545815 /build/ceph-13.2.8/src/mds/MDCache.cc: 9523: FAILED assert(p != active_requests.end()) ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14e) [0x7f11bd8e69de] 2: (()+0x287b67) [0x7f11bd8e6b67] 3: (MDCache::request_get(metareqid_t)+0x94) [0x560cde8bb214] 4: (Server::journal_close_session(Session*, int, Context*)+0x9dd) [0x560cde829d1d] 5: (Server::handle_client_session(MClientSession*)+0x1071) [0x560cde82b0f1] 6: (Server::dispatch(Message*)+0x30b) [0x560cde86f87b] 7: (MDSRank::handle_deferrable_message(Message*)+0x434) [0x560cde7e1664] 8: (MDSRank::_dispatch(Message*, bool)+0x89b) [0x560cde7f8c7b] 9: (MDSRankDispatcher::ms_dispatch(Message*)+0xa3) [0x560cde7f92e3] 10: (MDSDaemon::ms_dispatch(Message*)+0xd3) [0x560cde7d92b3] 11: (DispatchQueue::entry()+0xb92) [0x7f11bd9a9e52] 12: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f11bda46e2d] 13: (()+0x76db) [0x7f11bd1d76db] 14: (clone()+0x3f) [0x7f11bc3bd88f] Before we hit this assert there were a few (kernel clients, 5.3.0-26/28) that were not playing nicely: 16:32 < bitrot> mds.mds1 [WRN] client.61994841 isn't responding to mclientcaps(revoke), ino 0x1003846ddc5 pending pAsLsXsFscr issued pAsLsXsFscr, sent 62.342791 seconds ago 16:32 < bitrot> mon.mon1 [WRN] Health check failed: 1 clients failing to respond to capability release (MDS_CLIENT_LATE_RELEASE) We rebooted both clients. After that one of them again had some slow requests. We umounted the file system, slowly after that the MDS hit the assert. Failover went fine this time. This looks like issue: https://tracker.ceph.com/issues/23059 ... but that should already have been resolved. Is this the same issue, and or a regression? We run 13.2.8. Thanks, Stefan -- | BIT BV https://www.bit.nl/ Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / info(a)bit.nl

2 2

Identify slow ops
by Thomas Schneider 13 Feb '20

13 Feb '20

Hi, the current output of ceph -s reports a warning: 2 slow ops, oldest one blocked for 347335 sec, mon.ld5505 has slow ops This time is increasing. root@ld3955:~# ceph -s cluster: id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae health: HEALTH_WARN 9 daemons have recently crashed 2 slow ops, oldest one blocked for 347335 sec, mon.ld5505 has slow ops services: mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 3d) mgr: ld5507(active, since 8m), standbys: ld5506, ld5505 mds: cephfs:2 {0=ld5507=up:active,1=ld5505=up:active} 2 up:standby-replay 3 up:standby osd: 442 osds: 442 up (since 8d), 442 in (since 9d) data: pools: 7 pools, 19628 pgs objects: 65.78M objects, 251 TiB usage: 753 TiB used, 779 TiB / 1.5 PiB avail pgs: 19628 active+clean io: client: 427 KiB/s rd, 22 MiB/s wr, 851 op/s rd, 647 op/s wr The details are as follows: root@ld3955:~# ceph health detail HEALTH_WARN 9 daemons have recently crashed; 2 slow ops, oldest one blocked for 347755 sec, mon.ld5505 has slow ops RECENT_CRASH 9 daemons have recently crashed mds.ld4464 crashed on host ld4464 at 2020-02-09 07:33:59.131171Z mds.ld5506 crashed on host ld5506 at 2020-02-09 07:42:52.036592Z mds.ld4257 crashed on host ld4257 at 2020-02-09 07:47:44.369505Z mds.ld4464 crashed on host ld4464 at 2020-02-09 06:10:24.515912Z mds.ld5507 crashed on host ld5507 at 2020-02-09 07:13:22.400268Z mds.ld4257 crashed on host ld4257 at 2020-02-09 06:48:34.742475Z mds.ld5506 crashed on host ld5506 at 2020-02-09 06:10:24.680648Z mds.ld4465 crashed on host ld4465 at 2020-02-09 06:52:33.204855Z mds.ld5506 crashed on host ld5506 at 2020-02-06 07:59:37.089007Z SLOW_OPS 2 slow ops, oldest one blocked for 347755 sec, mon.ld5505 has slow ops There's no error on services (mgr, mon, osd). Can you please advise how to identify the root cause of this slow ops? THX

2 1

Changing the failure-domain of an erasure coded pool
by Neukum, Max (ETP) 13 Feb '20

13 Feb '20

Hi ceph enthusiasts, We have a ceph cluster with cephfs and two pools: one replicated for metadata on ssd and one with ec (4+2) on hdd. Recently, we upgraded from 4 to 7 nodes and now want to change the failure domain of the erasure coded pool from 'OSD' to 'HOST'. What we did was to create a new crush-rule and changed the rule of our ec pool. It still uses the old profile. Details can be found below. Now there are a couple of questions: 1) Is this equivalent to changing the profile? Below you can see in the profile 'crush-failure-domain=osd' and in the crush-rule '"op": "chooseleaf_indep", "type": "host"'. 2) If we need to change the failure-domain in the profile, can this be done without creating a new pool, which seems troublesome? 3) Finally, if we really need to create a new pool to do this... what is the best way? For the record: our cluster is now (after the upgrade) ~40% full (400TB/1Pb) with 173 OSDs. Cheers, Max some more details: [root@ceph-node-a ~]# ceph osd lspools 1 ec42 2 cephfs_metadata [root@ceph-node-a ~]# ceph osd pool get ec42 erasure_code_profile erasure_code_profile: ec42 [root@ceph-node-a ~]# ceph osd pool get ec42 crush_rule crush_rule: ec42_host_hdd [root@ceph-node-a ~]# ceph osd erasure-code-profile get ec42 crush-device-class= crush-failure-domain=osd crush-root=default jerasure-per-chunk-alignment=false k=4 m=2 plugin=jerasure technique=reed_sol_van w=8 [root@ceph-node-a ~]# ceph osd crush rule dump ec42_host_hdd { "rule_id": 6, "rule_name": "ec42_host_hdd", "ruleset": 6, "type": 3, "min_size": 3, "max_size": 6, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -2, "item_name": "default~hdd" }, { "op": "chooseleaf_indep", "num": 0, "type": "host" }, { "op": "emit" } ] }

2 2

extract disk usage stats from running ceph cluster
by lists 13 Feb '20

13 Feb '20

Hi, We would like to replace the current seagate ST4000NM0034 HDDs in our ceph cluster with SSDs, and before doing that, we would like to checkout the typical usage of our current drives, over the last years, so we can select the best (price/performance/endurance) SSD to replace them with. I am trying to extract this info from the fields "Blocks received from initiator" / "blocks sent to initiator", as these are the fields smartctl gets from the seagate disks. But the numbers seem strange, and I would like to request feedback here. Three nodes, all equal, 8 OSDs per node, all 4TB ST4000NM0034 (filestore) HDDs with SSD-based journals: > root@node1:~# ceph osd crush tree > ID CLASS WEIGHT TYPE NAME > -1 87.35376 root default > -2 29.11688 host node1 > 0 hdd 3.64000 osd.0 > 1 hdd 3.64000 osd.1 > 2 hdd 3.63689 osd.2 > 3 hdd 3.64000 osd.3 > 12 hdd 3.64000 osd.12 > 13 hdd 3.64000 osd.13 > 14 hdd 3.64000 osd.14 > 15 hdd 3.64000 osd.15 > -3 29.12000 host node2 > 4 hdd 3.64000 osd.4 > 5 hdd 3.64000 osd.5 > 6 hdd 3.64000 osd.6 > 7 hdd 3.64000 osd.7 > 16 hdd 3.64000 osd.16 > 17 hdd 3.64000 osd.17 > 18 hdd 3.64000 osd.18 > 19 hdd 3.64000 osd.19 > -4 29.11688 host node3 > 8 hdd 3.64000 osd.8 > 9 hdd 3.64000 osd.9 > 10 hdd 3.64000 osd.10 > 11 hdd 3.64000 osd.11 > 20 hdd 3.64000 osd.20 > 21 hdd 3.64000 osd.21 > 22 hdd 3.64000 osd.22 > 23 hdd 3.63689 osd.23 We are looking at the numbers from smartctl, and basing our calculations on this output for each individual various OSD: > Vendor (Seagate) cache information > Blocks sent to initiator = 3783529066 > Blocks received from initiator = 3121186120 > Blocks read from cache and sent to initiator = 545427169 > Number of read and write commands whose size <= segment size = 93877358 > Number of read and write commands whose size > segment size = 2290879 I created the following spreadsheet: > blocks sent blocks received total blocks > to initiator from initiator calculated read% write% aka > node1 > osd0 905060564 1900663448 2805724012 32,26% 67,74% sda > osd1 2270442418 3756215880 6026658298 37,67% 62,33% sdb > osd2 3531938448 3940249192 7472187640 47,27% 52,73% sdc > osd3 2824808123 3130655416 5955463539 47,43% 52,57% sdd > osd12 1956722491 1294854032 3251576523 60,18% 39,82% sdg > osd13 3410188306 1265443936 4675632242 72,94% 27,06% sdh > osd14 3765454090 3115079112 6880533202 54,73% 45,27% sdi > osd15 2272246730 2218847264 4491093994 50,59% 49,41% sdj > > node2 > osd4 3974937107 740853712 4715790819 84,29% 15,71% sda > osd5 1181377668 2109150744 3290528412 35,90% 64,10% sdb > osd5 1903438106 608869008 2512307114 75,76% 24,24% sdc > osd7 3511170043 724345936 4235515979 82,90% 17,10% sdd > osd16 2642731906 3981984640 6624716546 39,89% 60,11% sdg > osd17 3994977805 3703856288 7698834093 51,89% 48,11% sdh > osd18 3992157229 2096991672 6089148901 65,56% 34,44% sdi > osd19 279766405 1053039640 1332806045 20,99% 79,01% sdj > > node3 > osd8 3711322586 234696960 3946019546 94,05% 5,95% sda > osd9 1203912715 3132990000 4336902715 27,76% 72,24% sdb > osd10 912356010 1681434416 2593790426 35,17% 64,83% sdc > osd11 810488345 2626589896 3437078241 23,58% 76,42% sdd > osd20 1506879946 2421596680 3928476626 38,36% 61,64% sdg > osd21 2991526593 7525120 2999051713 99,75% 0,25% sdh > osd22 29560337 3226114552 3255674889 0,91% 99,09% sdi > osd23 2019195656 2563506320 4582701976 44,06% 55,94% sdj But as can be seen above, this results in some very strange numbers, for example node3/osd21 and node2/osd19, node3/osd8, the numbers are unlikely. So, probably we're doing something wrong in our logic here. Can someone explain what we're doing wrong, and is it possible to obtain stats like these also from ceph directly? Does ceph keep historical stats like above..? MJ

5 7

Cleanup old messages in ceph health
by Thomas Schneider 13 Feb '20

13 Feb '20

Hi, the current outpu of ceph -s reports a warning: 9 daemons have recently crashed root@ld3955:~# ceph -s cluster: id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae health: HEALTH_WARN 9 daemons have recently crashed 2 slow ops, oldest one blocked for 347335 sec, mon.ld5505 has slow ops services: mon: 3 daemons, quorum ld5505,ld5506,ld5507 (age 3d) mgr: ld5507(active, since 8m), standbys: ld5506, ld5505 mds: cephfs:2 {0=ld5507=up:active,1=ld5505=up:active} 2 up:standby-replay 3 up:standby osd: 442 osds: 442 up (since 8d), 442 in (since 9d) data: pools: 7 pools, 19628 pgs objects: 65.78M objects, 251 TiB usage: 753 TiB used, 779 TiB / 1.5 PiB avail pgs: 19628 active+clean io: client: 427 KiB/s rd, 22 MiB/s wr, 851 op/s rd, 647 op/s wr The details are as follows: root@ld3955:~# ceph health detail HEALTH_WARN 9 daemons have recently crashed; 2 slow ops, oldest one blocked for 347755 sec, mon.ld5505 has slow ops RECENT_CRASH 9 daemons have recently crashed mds.ld4464 crashed on host ld4464 at 2020-02-09 07:33:59.131171Z mds.ld5506 crashed on host ld5506 at 2020-02-09 07:42:52.036592Z mds.ld4257 crashed on host ld4257 at 2020-02-09 07:47:44.369505Z mds.ld4464 crashed on host ld4464 at 2020-02-09 06:10:24.515912Z mds.ld5507 crashed on host ld5507 at 2020-02-09 07:13:22.400268Z mds.ld4257 crashed on host ld4257 at 2020-02-09 06:48:34.742475Z mds.ld5506 crashed on host ld5506 at 2020-02-09 06:10:24.680648Z mds.ld4465 crashed on host ld4465 at 2020-02-09 06:52:33.204855Z mds.ld5506 crashed on host ld5506 at 2020-02-06 07:59:37.089007Z SLOW_OPS 2 slow ops, oldest one blocked for 347755 sec, mon.ld5505 has slow ops However, any crashed host is up and running. Therefore I would prefer to remove these error messages Can you please advise how to cleanup the error messages? THX

3 2

[ceph-user] SSD disk utilization high on ceph-12.2.12
by Amit Ghadge 13 Feb '20

13 Feb '20

Hello All, We seen one of the Ceph data nodes, all osd's are 90-100% disk utilized , those all are SSD drive and traffic is normal compare to other data nodes. How can we debug it?

1 0

CephFS hangs with access denied
by Dietmar Rieder 13 Feb '20

13 Feb '20

Hi, we sometimes loose access to our cephfs mount and get permission denied if we try to cd into it. This happens apparently only on some of our HPC cephfs-client nodes (fs mounted via kernel client) when they are busy with calculation and I/O. When we then manually force unmount the fs and remount it, everything is working again. This is the dmesg output of the affected client node: <https://pastebin.com/z5wxUgYS> All HPC clients and ceph servers are running CentOS 7.7 with the same kernel: $ uname -a Linux apollo-08.local 3.10.0-1062.12.1.el7.x86_64 #1 SMP Tue Feb 4 23:02:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux and all are running ceph version 14.2.7 $ ceph -v ceph version 14.2.7 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8) nautilus (stable) Maybe someone has an idea what goes wrong, and how we can fix/avoid this. Thanks Dietmar -- _________________________________________ D i e t m a r R i e d e r, Mag.Dr. Innsbruck Medical University Biocenter - Institute of Bioinformatics Email: dietmar.rieder(a)i-med.ac.at Web: http://www.icbi.at

3 7

Re: cephfs slow, howto investigate and tune mds configuration?
by Yan, Zheng 12 Feb '20

12 Feb '20

On Wed, Feb 12, 2020 at 6:08 PM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > > > > > >> > >> Say I think my cephfs is slow when I rsync to it, slower than it > used > >> to be. First of all, I do not get why it reads so much data. I > assume > >> the file attributes need to come from the mds server, so the rsync > >> backup should mostly cause writes not? > >> > > > >Are you running one or multiple MDS? I've seen cases where the > >synchronization between the different MDS slow down rsync. > > One > > >The problem is that rsync creates and renames files a lot. When doing > >this with small files it can be very heavy for the MDS. > > > > Strange thing is that I did not have performance problems with luminous, > after upgrading to nautilus and enabling snapshots on a different tree > of the cephfs. Rsync is taking 10 hours more. > There is also another option, degrading performance on the source. > However it is impossible for me to verify this. > I have increased the mds_cache_memory_limit from 8GB to 16GB, see > what that brings. > how many snapshot are there ? > > > > >> I think it started being slow, after enabling snapshots on the file > >> system. > >> > >> - how can I determine if mds_cache_memory_limit = 8000000000 is > still > >> correct? > >> > >> - how can I test the mds performance from the command line, so I can > > >> experiment with cpu power configurations, and see if this brings a > >> significant change? > >> _______________________________________________ > >> ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > > >> email to ceph-users-leave(a)ceph.io > >> > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

PR #26095 experience (backported/cherry-picked to Nauilus)
by Simon Leinen 12 Feb '20

12 Feb '20

We have been using RadosGW with Keystone integration for a couple of years, to allow users of our OpenStack-based IaaS to create their own credentials for our object store. This has caused us a fair amount of performance headaches. Last year, Jjames Weaver (BBC) has contributed a patch (PR #26095) that changes the handling of S3 authentication when Keystone is used as a backend for credentials. It was merged to master in March 2019. We run Nautilus on our production clusters, which doesn't include the patch. A few weeks ago, we decided to cherry-pick PR #26095 on top of Nautilus (12.4.5/6/7) and deploy that in production. So far we haven't noticed any issues. Load on our Keystone system has decreased significantly, response times for small requests are now consistently low, and we don't have to re-provision S3 credentials locally anymore to fix performance emergencies. Thanks a lot! Blog post with a few performance graphs: https://cloudblog.switch.ch/2020/02/10/radosgw-keystone-integration-perform… -- Simon.

1 0

Re: Ceph Erasure Coding - Stored vs used
by Kristof Coucke 12 Feb '20

12 Feb '20

Hi Simon and Janne, Thanks for the reply. It seems indeed related to the bluestore_min_alloc_size. In an old thread I've also found the following: *S3 object saving pipeline:* *- S3 object is divided into multipart shards by client.* *- Rgw shards each multipart shard into rados objects of size* *rgw_obj_stripe_size.* *- Primary osd stripes rados object into ec stripes of width ==* *ec.k*profile.stripe_unit, ec code them and send units into secondary* *osds and write into object store (bluestore).* *- Each subobject of rados object has size == (rados object size)/k.* *- Then while writing into disk bluestore can divide rados subobject into* *extents of minimal size == bluestore_min_alloc_size_hdd.* *Next rules can save some space and iops:* *- rgw_multipart_min_part_size SHOULD be multiple of rgw_obj_stripe_size* *(client can use different value greater than)* *- MUST rgw_obj_stripe_size == rgw_max_chunk_size* *- ec stripe == osd_pool_erasure_code_stripe_unit or profile.stripe_unit* *- rgw_obj_stripe_size SHOULD be multiple of profile.stripe_unit*ec.k* *- bluestore_min_alloc_size_hdd MAY be equal to bluefs_alloc_size (to* *avoid fragmentation)* *- rgw_obj_stripe_size/ec.k SHOULD be multiple of* *bluestore_min_alloc_size_hdd* *- bluestore_min_alloc_size_hdd MAY be multiple of profile.stripe_unit* Doing this calculation would result in the fact that smaller files of 135KB end up in chunks of +/- 22KB. Writing this in 64KB gives me quite some wasted space. As far as I found, the allocation setting is kept on OSD level and set during creation. Adapting the setting requires each OSD to be recreated. As we have around 150 OSDs, I know what to script :-) We will perform some testing in our test environment and I'll try to post our feedback as long as I don't forget it... To be sure, we just want to check the size on disk of the object. Afaik, we'll need to export the rocks db and launch some queries on that, unless someone else can help me on this one? Before Bluestore this was quite easy to do... Regards,

1 0

Ceph Erasure Coding - Stored vs used
by Kristof Coucke 12 Feb '20

12 Feb '20

Hi all, I have an issue on my Ceph cluster. For one of my pools I have 107TiB STORED and 298TiB USED. This is strange, since I've configured erasure coding (6 data chunks, 3 coding chunks). So, in an ideal world this should result in approx. 160.5TiB USED. The question now is why this is the case... There are 473+M objects stored. Lot's of these files are pretty small. (Read 150kb files). Not all of them though. I am running Nautilus version 14.2.4. I suspect that the stripe size is related with this issue. This is still the default (4MB), but I am not sure. Before BlueFS it was easy to check the size of the chunks on the disk... With BlueFS this is another story. I have the following questions: 1. How can I check this to be sure that this is the case? I actually want to drill down starting from an object I've sent to the Ceph cluster thru the RGW. I would like to see where the chunks are stored and which size is allocated for these on the disks. 2. If it is related to the stripe size, can I safely adapt this parameter or is this going to work forward only, or will it also work reversely? Many thanks, Kristof

3 3

Ceph Erasure Coding - Stored vs used
by Kristof Coucke 12 Feb '20

12 Feb '20

Hi all, I have an issue on my Ceph cluster. For one of my pools I have 107TiB STORED and 298TiB USED. This is strange, since I've configured erasure coding (6 data chunks, 3 coding chunks). So, in an ideal world this should result in approx. 160.5TiB USED. The question now is why this is the case... There are 473+M objects stored. Lot's of these files are pretty small. (Read 150kb files). Not all of them though. I am running Nautilus version 14.2.4. I suspect that the stripe size is related with this issue. This is still the default (4MB), but I am not sure. Before BlueFS it was easy to check the size of the chunks on the disk... With BlueFS this is another story. I have the following questions: 1. How can I check this to be sure that this is the case? I actually want to drill down starting from an object I've sent to the Ceph cluster thru the RGW. I would like to see where the chunks are stored and which size is allocated for these on the disks. 2. If it is related to the stripe size, can I safely adapt this parameter or is this going to work forward only, or will it also work reversely? Many thanks, Kristof

1 0

Fwd: Finding erasure-code-profile of crush rule
by David Seith 12 Feb '20

12 Feb '20

---------- Forwarded message --------- From: David Seith <david.seith(a)kit.edu> Date: Wed, 12 Feb 2020 at 11:24 Subject: Finding erasure-code-profile of crush rule To: <ceph-users(a)lists.ceph.com> Dear all, On our ceph cluster we have created multiple erasure coding profiles and then created a number of crush rules for this profiles using: ceph osd crush rule create-erasure {name} {profile-name} Is it now possible to find out which erasure coding profile is used by a certain crush rule? Doing: ceph osd crush rule dump does not show the name of the erasure coding profile associated to the rules. Best, David

1 0

cephfs slow, howto investigate and tune mds configuration?
by Marc Roos 12 Feb '20

12 Feb '20

Say I think my cephfs is slow when I rsync to it, slower than it used to be. First of all, I do not get why it reads so much data. I assume the file attributes need to come from the mds server, so the rsync backup should mostly cause writes not? I think it started being slow, after enabling snapshots on the file system. - how can I determine if mds_cache_memory_limit = 8000000000 is still correct? - how can I test the mds performance from the command line, so I can experiment with cpu power configurations, and see if this brings a significant change?

4 5

"mds daemon damaged" after restarting MDS - Filesystem DOWN
by Luca Cervigni 12 Feb '20

12 Feb '20

Not sure if the previous message went through since I was not subscribed. If yes sorry for the spam. Dear all Running nautilus 14.2.7. The data in the FS are important and cannot be lost. Today I increased the PGS of the volume pool from 8k to 16k. The active mds started reporting slow ops. (the filesystem is not in the volume pool). After few hours the FS was very slow, I reduced the backfill to 1 and since the situation was not improving, I restarted the MDS (no other standby MDSs. it was a single mds). After that the crash. The mds does not goes back up with this error: 020-02-07 07:03:32.477 7fbf69647700 -1 NetHandler create_socket couldn't create socket (97) Address family not supported by protocol 2020-02-07 07:03:32.541 7fbf65e6a700 1 mds.ceph-mon-01 Updating MDS map to version 48461 from mon.2 2020-02-07 07:03:37.613 7fbf65e6a700 1 mds.ceph-mon-01 Updating MDS map to version 48462 from mon.2 2020-02-07 07:03:37.613 7fbf65e6a700 1 mds.ceph-mon-01 Map has assigned me to become a standby 2020-02-07 07:14:11.789 7fbf66e42700 -1 received signal: Terminated from /sbin/init (PID: 1) UID: 0 2020-02-07 07:14:11.789 7fbf66e42700 -1 mds.ceph-mon-01 *** got signal Terminated *** 2020-02-07 07:14:11.789 7fbf66e42700 1 mds.ceph-mon-01 suicide! Wanted state up:standby 2020-02-07 07:14:12.565 7fbf65e6a700 0 ms_deliver_dispatch: unhandled message 0x563fcb438d00 mdsmap(e 48465) v1 from mon.2 v1:10.3.78.32:6789/0 2020-02-07 07:25:16.782 7f26c39de2c0 0 set uid:gid to 64045:64045 (ceph:ceph) 2020-02-07 07:25:16.782 7f26c39de2c0 0 ceph version 14.2.7 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8) nautilus (stable), process ceph-mds, pid 3724 2020-02-07 07:25:16.782 7f26c39de2c0 0 pidfile_write: ignore empty --pid-file 2020-02-07 07:25:16.786 7f26b5326700 -1 NetHandler create_socket couldn't create socket (97) Address family not supported by protocol 2020-02-07 07:25:16.790 7f26b1b49700 1 mds.ceph-mon-01 Updating MDS map to version 48472 from mon.0 2020-02-07 07:25:17.691 7f26b1b49700 1 mds.ceph-mon-01 Updating MDS map to version 48473 from mon.0 2020-02-07 07:25:17.691 7f26b1b49700 1 mds.ceph-mon-01 Map has assigned me to become a standby 2020-02-07 07:29:50.306 7f26b2b21700 -1 received signal: Terminated from /sbin/init (PID: 1) UID: 0 2020-02-07 07:29:50.306 7f26b2b21700 -1 mds.ceph-mon-01 *** got signal Terminated *** 2020-02-07 07:29:50.306 7f26b2b21700 1 mds.ceph-mon-01 suicide! Wanted state up:standby 2020-02-07 07:29:50.526 7f26b5b27700 1 mds.beacon.ceph-mon-01 discarding unexpected beacon reply down:dne seq 70 dne 2020-02-07 07:29:52.802 7f26b1b49700 0 ms_deliver_dispatch: unhandled message 0x55ef110ab200 mdsmap(e 48474) v1 from mon.0 v1:10.3.78.22:6789/0 Rebooting did not help I asked #CEPH OFTC and they suggested to bring up another "fresh" mds. I did that, and they do not start, going to standby. LOGS: 2020-02-07 07:12:46.696 7fe4b388b2c0 0 set uid:gid to 64045:64045 (ceph:ceph) 2020-02-07 07:12:46.696 7fe4b388b2c0 0 ceph version 14.2.7 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8) nautilus (stable), process ceph-mds, pid 74742 2020-02-07 07:12:46.696 7fe4b388b2c0 0 pidfile_write: ignore empty --pid-file 2020-02-07 07:12:46.704 7fe4a19f6700 1 mds.ceph-mon-02 Updating MDS map to version 48462 from mon.0 2020-02-07 07:12:47.456 7fe4a19f6700 1 mds.ceph-mon-02 Updating MDS map to version 48463 from mon.0 2020-02-07 07:12:47.456 7fe4a19f6700 1 mds.ceph-mon-02 Map has assigned me to become a standby 2020-02-07 07:14:16.615 7fe4a29ce700 -1 received signal: Terminated from /sbin/init (PID: 1) UID: 0 2020-02-07 07:14:16.615 7fe4a29ce700 -1 mds.ceph-mon-02 *** got signal Terminated *** 2020-02-07 07:14:16.615 7fe4a29ce700 1 mds.ceph-mon-02 suicide! Wanted state up:standby 2020-02-07 07:14:16.947 7fe4a51d3700 1 mds.beacon.ceph-mon-02 discarding unexpected beacon reply down:dne seq 24 dne 2020-02-07 07:14:18.715 7fe4a19f6700 0 ms_deliver_dispatch: unhandled message 0x5602fbc6df80 mdsmap(e 48466) v1 from mon.0 v2:10.3.78.22:3300/0 2020-02-07 07:25:02.093 7f3c2f92a2c0 0 set uid:gid to 64045:64045 (ceph:ceph) 2020-02-07 07:25:02.093 7f3c2f92a2c0 0 ceph version 14.2.7 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8) nautilus (stable), process ceph-mds, pid 75471 2020-02-07 07:25:02.093 7f3c2f92a2c0 0 pidfile_write: ignore empty --pid-file 2020-02-07 07:25:02.097 7f3c1da95700 1 mds.ceph-mon-02 Updating MDS map to version 48471 from mon.2 2020-02-07 07:25:06.413 7f3c1da95700 1 mds.ceph-mon-02 Updating MDS map to version 48472 from mon.2 2020-02-07 07:25:06.413 7f3c1da95700 1 mds.ceph-mon-02 Map has assigned me to become a standby 2020-02-07 07:29:56.869 7f3c1ea6d700 -1 received signal: Terminated from /sbin/init (PID: 1) UID: 0 2020-02-07 07:29:56.869 7f3c1ea6d700 -1 mds.ceph-mon-02 *** got signal Terminated *** 2020-02-07 07:29:56.869 7f3c1ea6d700 1 mds.ceph-mon-02 suicide! Wanted state up:standby 2020-02-07 07:29:58.113 7f3c1da95700 0 ms_deliver_dispatch: unhandled message 0x563c5df33f80 mdsmap(e 48475) v1 from mon.2 v2:10.3.78.32:3300/0 Here ceph status cluster: id: a8dde71d-ca7b-4cf5-bd38-8989c6a27011 health: HEALTH_ERR 1 filesystem is degraded 1 filesystem is offline 1 mds daemon damaged 2 daemons have recently crashed services: mon: 3 daemons, quorum ceph-mon-01,ceph-mon-02,ceph-mon-03 (age 41m) mgr: ceph-mon-02(active, since 41m), standbys: ceph-mon-03, ceph-mon-01 mds: pawsey-sync-fs:0/1, 1 damaged osd: 925 osds: 715 up (since 2h), 715 in (since 23h) rgw: 3 daemons active (radosgw-01, radosgw-02, radosgw-03) data: pools: 24 pools, 26569 pgs objects: 52.64M objects, 199 TiB usage: 685 TiB used, 6.7 PiB / 7.3 PiB avail pgs: 26513 active+clean 54 active+clean+scrubbing+deep 2 active+clean+scrubbing Ceph osd ls detail: https://pastebin.com/raw/bxi4HSa5 the metadata pool is on NVMe Can anyone give me some help? Any command run like journal repairs do not work as they expect the MDs to be up. Thanks Cheers -- Luca Cervigni Infrastructure Architect Tel. +61864368802 Pawsey Supercomputing Centre 1 Bryce Ave, Kensington WA 6151 Australia

2 3

luminous -> nautilus upgrade path
by Wolfgang Lendl 12 Feb '20

12 Feb '20

hello, we plan to upgrade from luminous to nautilus. does it make sense to do the mimic step instead of going directly for nautilus? br wolfgang

5 4

MDS: obscene buffer_anon memory use when scanning lots of files (continued)
by John Madden 11 Feb '20

11 Feb '20

Following the list migration I need to re-open this thread: http://lists.ceph.com/pipermail/ceph-users-ceph.com/2020-January/038014.html ... Upgraded to 14.2.7, doesn't appear to have affected the behavior. As requested: ~$ ceph tell mds.mds1 heap stats 2020-02-10 16:52:44.313 7fbda2cae700 0 client.59208005 ms_handle_reset on v2:x.x.x.x:6800/3372494505 2020-02-10 16:52:44.337 7fbda3cb0700 0 client.59249562 ms_handle_reset on v2:x.x.x.x:6800/3372494505 mds.mds1 tcmalloc heap stats:------------------------------------------------ MALLOC: 50000388656 (47684.1 MiB) Bytes in use by application MALLOC: + 0 ( 0.0 MiB) Bytes in page heap freelist MALLOC: + 174879528 ( 166.8 MiB) Bytes in central cache freelist MALLOC: + 14511680 ( 13.8 MiB) Bytes in transfer cache freelist MALLOC: + 14089320 ( 13.4 MiB) Bytes in thread cache freelists MALLOC: + 90534048 ( 86.3 MiB) Bytes in malloc metadata MALLOC: ------------ MALLOC: = 50294403232 (47964.5 MiB) Actual memory used (physical + swap) MALLOC: + 50987008 ( 48.6 MiB) Bytes released to OS (aka unmapped) MALLOC: ------------ MALLOC: = 50345390240 (48013.1 MiB) Virtual address space used MALLOC: MALLOC: 260018 Spans in use MALLOC: 20 Thread heaps in use MALLOC: 8192 Tcmalloc page size ------------------------------------------------ Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()). Bytes released to the OS take up virtual address space but no physical memory. ~$ ceph tell mds.mds1 heap release 2020-02-10 16:52:47.205 7f037eff5700 0 client.59249625 ms_handle_reset on v2:x.x.x.x:6800/3372494505 2020-02-10 16:52:47.237 7f037fff7700 0 client.59249634 ms_handle_reset on v2:x.x.x.x:6800/3372494505 mds.mds1 releasing free RAM back to system. The pools over 15 minutes or so: ~$ ceph daemon mds.mds1 dump_mempools | jq .mempool.by_pool.buffer_anon { "items": 2045, "bytes": 3069493686 } ~$ ceph daemon mds.mds1 dump_mempools | jq .mempool.by_pool.buffer_anon { "items": 2445, "bytes": 3111162538 } ~$ ceph daemon mds.mds1 dump_mempools | jq .mempool.by_pool.buffer_anon { "items": 7850, "bytes": 7658678767 } ~$ ceph daemon mds.mds1 dump_mempools | jq .mempool.by_pool.buffer_anon { "items": 12274, "bytes": 11436728978 } ~$ ceph daemon mds.mds1 dump_mempools | jq .mempool.by_pool.buffer_anon { "items": 13747, "bytes": 11539478519 } ~$ ceph daemon mds.mds1 dump_mempools | jq .mempool.by_pool.buffer_anon { "items": 14615, "bytes": 13859676992 } ~$ ceph daemon mds.mds1 dump_mempools | jq .mempool.by_pool.buffer_anon { "items": 23267, "bytes": 22290063830 } ~$ ceph daemon mds.mds1 dump_mempools | jq .mempool.by_pool.buffer_anon { "items": 44944, "bytes": 40726959425 } And one about a minute after the heap release showing continued growth: ~$ ceph daemon mds.mds1 dump_mempools | jq .mempool.by_pool.buffer_anon { "items": 50694, "bytes": 47343942094 } This is on a single active MDS with 2 standbys, scan for about a million files with about 20 parallel threads on two clients, open and read each if it exists.

1 0

Bluestore cache parameter precedence
by Boris Epstein 11 Feb '20

11 Feb '20

Hello list, As stated in this document: https://docs.ceph.com/docs/master/rados/configuration/bluestore-config-ref/ there are multiple parameters defining cache limits for BlueStore. You have bluestore_cache_size (presumably controlling the cache size), bluestore_cache_size_hdd (presumably doing the same for HDD storage only) and bluestore_cache_size_ssd (presumably being the equivalent for SSD). My question is, does bluestore_cache_size override the disk-specific parameters, or do I need to set the disk-specific (or, rather, storage type specific ones separately if I want to keep them to a certain value. Thanks in advance. Boris.

3 4

Fwd: PrimaryLogPG.cc: 11550: FAILED ceph_assert(head_obc)
by Jake Grimmett 11 Feb '20

11 Feb '20

Dear All, Following a clunky* cluster restart, we had 23 "objects unfound" 14 pg recovery_unfound We could see no way to recover the unfound objects, we decided to mark the objects in one pg unfound... [root@ceph1 bad_oid]# ceph pg 5.f2f mark_unfound_lost delete pg has 2 objects unfound and apparently lost marking Unfortunately, this immediately crashed the primary OSD for this PG: OSD log showing the osd crashing 3 times here: <http://p.ip.fi/gV8r> the assert was :> 2020-02-10 13:38:45.003 7fa713ef3700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.6/rpm/el7/BUILD/ceph-14.2.6/src/osd/PrimaryLogPG.cc: In function 'int PrimaryLogPG::recover_missing(const hobject_t&, eversion_t, int, PGBackend::RecoveryHandle*)' thread 7fa713ef3700 time 2020-02-10 13:38:45.000875 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.6/rpm/el7/BUILD/ceph-14.2.6/src/osd/PrimaryLogPG.cc: 11550: FAILED ceph_assert(head_obc) Questions.. 1) Is it possible to recover the flapping OSD? or should we fail out the flapping OSD and hope the cluster recovers? 2) We have 13 other pg with unfound objects. Do we need to mark_unfound these one at a time, and then fail out their primary OSD? (allowing the cluster to recover before mark_unfound the next pg & failing it's primary OSD) * thread describing the bad restart :> <https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/IRKCDRRAH7Y…> many thanks! Jake -- Dr Jake Grimmett MRC Laboratory of Molecular Biology Francis Crick Avenue, Cambridge CB2 0QH, UK. -- Dr Jake Grimmett Head Of Scientific Computing MRC Laboratory of Molecular Biology Francis Crick Avenue, Cambridge CB2 0QH, UK. Phone 01223 267019 Mobile 0776 9886539

1 1

ERROR: osd init failed: (1) Operation not permitted
by Ml Ml 11 Feb '20

11 Feb '20

Hello List, first of all: Yes - i made mistakes. Now i am trying to recover :-/ I had a healthy 3 node cluster which i wanted to convert to a single one. My goal was to reinstall a fresh 3 Node cluster and start with 2 nodes. I was able to healthy turn it from a 3 Node Cluster to a 2 Node cluster. Then the problems began. I started to change size=1 and min_size=1. (i know, i know, i will never ever to that again!) Health was okay until here. Then over sudden both nodes got fenced...one node refused to boot, mons where missing, etc...to make long story short, here is where i am right now: root@node03:~ # ceph -s cluster b3be313f-d0ef-42d5-80c8-6b41380a47e3 health HEALTH_WARN 53 pgs stale 53 pgs stuck stale monmap e4: 2 mons at {0=10.15.15.3:6789/0,1=10.15.15.2:6789/0} election epoch 298, quorum 0,1 1,0 osdmap e6097: 14 osds: 9 up, 9 in pgmap v93644673: 512 pgs, 1 pools, 1193 GB data, 304 kobjects 1088 GB used, 32277 GB / 33366 GB avail 459 active+clean 53 stale+active+clean root@node03:~ # ceph osd tree ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY -1 32.56990 root default -2 25.35992 host node03 0 3.57999 osd.0 up 1.00000 1.00000 5 3.62999 osd.5 up 1.00000 1.00000 6 3.62999 osd.6 up 1.00000 1.00000 7 3.62999 osd.7 up 1.00000 1.00000 8 3.62999 osd.8 up 1.00000 1.00000 19 3.62999 osd.19 up 1.00000 1.00000 20 3.62999 osd.20 up 1.00000 1.00000 -3 7.20998 host node02 3 3.62999 osd.3 up 1.00000 1.00000 4 3.57999 osd.4 up 1.00000 1.00000 1 0 osd.1 down 0 1.00000 9 0 osd.9 down 0 1.00000 10 0 osd.10 down 0 1.00000 17 0 osd.17 down 0 1.00000 18 0 osd.18 down 0 1.00000 my main mistakes seemd to be: -------------------------------- ceph osd out osd.1 ceph auth del osd.1 systemctl stop ceph-osd@1 ceph osd rm 1 umount /var/lib/ceph/osd/ceph-1 ceph osd crush remove osd.1 As far as i can tell, ceph waits and needs data from that OSD.1 (which i removed) root@node03:~ # ceph health detail HEALTH_WARN 53 pgs stale; 53 pgs stuck stale pg 0.1a6 is stuck stale for 5086.552795, current state stale+active+clean, last acting [1] pg 0.142 is stuck stale for 5086.552784, current state stale+active+clean, last acting [1] pg 0.1e is stuck stale for 5086.552820, current state stale+active+clean, last acting [1] pg 0.e0 is stuck stale for 5086.552855, current state stale+active+clean, last acting [1] pg 0.1d is stuck stale for 5086.552822, current state stale+active+clean, last acting [1] pg 0.13c is stuck stale for 5086.552791, current state stale+active+clean, last acting [1] [...] SNIP [...] pg 0.e9 is stuck stale for 5086.552955, current state stale+active+clean, last acting [1] pg 0.87 is stuck stale for 5086.552939, current state stale+active+clean, last acting [1] When i try to start ODS.1 manually, i get: -------------------------------------------- 2020-02-10 18:48:26.107444 7f9ce31dd880 0 ceph version 0.94.10 (b1e0532418e4631af01acbc0cedd426f1905f4af), process ceph-osd, pid 10210 2020-02-10 18:48:26.134417 7f9ce31dd880 0 filestore(/var/lib/ceph/osd/ceph-1) backend xfs (magic 0x58465342) 2020-02-10 18:48:26.184202 7f9ce31dd880 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-1) detect_features: FIEMAP ioctl is supported and appears to work 2020-02-10 18:48:26.184209 7f9ce31dd880 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-1) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2020-02-10 18:48:26.184526 7f9ce31dd880 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-1) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2020-02-10 18:48:26.184585 7f9ce31dd880 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-1) detect_feature: extsize is disabled by conf 2020-02-10 18:48:26.309755 7f9ce31dd880 0 filestore(/var/lib/ceph/osd/ceph-1) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled 2020-02-10 18:48:26.633926 7f9ce31dd880 1 journal _open /var/lib/ceph/osd/ceph-1/journal fd 20: 5367660544 bytes, block size 4096 bytes, directio = 1, aio = 1 2020-02-10 18:48:26.642185 7f9ce31dd880 1 journal _open /var/lib/ceph/osd/ceph-1/journal fd 20: 5367660544 bytes, block size 4096 bytes, directio = 1, aio = 1 2020-02-10 18:48:26.664273 7f9ce31dd880 0 <cls> cls/hello/cls_hello.cc:271: loading cls_hello 2020-02-10 18:48:26.732154 7f9ce31dd880 0 osd.1 6002 crush map has features 1107558400, adjusting msgr requires for clients 2020-02-10 18:48:26.732163 7f9ce31dd880 0 osd.1 6002 crush map has features 1107558400 was 8705, adjusting msgr requires for mons 2020-02-10 18:48:26.732167 7f9ce31dd880 0 osd.1 6002 crush map has features 1107558400, adjusting msgr requires for osds 2020-02-10 18:48:26.732179 7f9ce31dd880 0 osd.1 6002 load_pgs 2020-02-10 18:48:31.939810 7f9ce31dd880 0 osd.1 6002 load_pgs opened 53 pgs 2020-02-10 18:48:31.940546 7f9ce31dd880 -1 osd.1 6002 log_to_monitors {default=true} 2020-02-10 18:48:31.942471 7f9ce31dd880 1 journal close /var/lib/ceph/osd/ceph-1/journal 2020-02-10 18:48:31.969205 7f9ce31dd880 -1 ESC[0;31m ** ERROR: osd init failed: (1) Operation not permittedESC[0m Its mounted: /dev/sdg1 3.7T 127G 3.6T 4% /var/lib/ceph/osd/ceph-1 Is there any way i can get the OSD.1 back in? Thanks a lot, mario

2 1

Running cephadm as a nonroot user
by Jason Borden 11 Feb '20

11 Feb '20

We have been using ceph-deploy in our existing cluster running as a non root user with sudo permissions. I've been working on getting an octopus cluster working using cephadm. During bootstrap I ran into a "execnet.gateway_bootstrap.HostNotFound" issue. It turns out that the problem was caused by an sshd setting we use: "PermitRootLogin no". Since we do not allow root ssh login directly, is there a way to make cephadm use ssh as a nonroot user with sudo permissions like we did with ceph-deploy?

2 4

How to monitor Ceph MDS operation latencies when slow cephfs performance
by jalagam.ceph＠gmail.com 10 Feb '20

10 Feb '20

Hello , Cephfs operations are slow in our cluster , I see low number of operations or throughput in the pools and all other resources as well. I think it is MDS operations that are causing the issue. I increased mds_cache_memory_limit to 3 GB from 1 GB but not seeing any improvements in the user access times. How do I monitor the MDS operations like metadata operations latencies including inode access times update time and directory operations latencies ? we am using 14.2.3 ceph version. I have increased mds_cache_memory_limit but not sure how to check what is been used and how effectively we are using it. # ceph config get mds.0 mds_cache_memory_limit 3221225472 I also see this , we are maninging PG using autoscale , however I see BIAS as 4.0 where as all pools have 1.0 not sure what is this number exactly and how it effect cluster . # ceph osd pool autoscale-status | egrep "cephfs|POOL" POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE cephfs01-metadata 1775M 3.0 167.6T 0.0000 4.0 8 on cephfs01-data0 739.5G 3.0 167.6T 0.0129 1.0 32 on There is one large OMAP. [root@knode25 /]# ceph health detail HEALTH_WARN 1 large omap objects LARGE_OMAP_OBJECTS 1 large omap objects 1 large objects found in pool 'cephfs01-metadata' Search the cluster log for 'Large omap object found' for more details. I recently had similar one and I was able to remove that by running deep scrub , not sure why they are keep forming and how to solve this for good ? Thanks, Uday.

1 0

Write i/o in CephFS metadata pool
by Samy Ascha 10 Feb '20

10 Feb '20

Hi! I've been running CephFS for a while now and ever since setting it up, I've seen unexpectedly large write i/o on the CephFS metadata pool. The filesystem is otherwise stable and I'm seeing no usage issues. I'm in a read-intensive environment, from the clients' perspective and throughput for the metadata pool is consistently larger than that of the data pool. For example: # ceph osd pool stats pool cephfs_data id 1 client io 7.6 MiB/s rd, 19 KiB/s wr, 404 op/s rd, 1 op/s wr pool cephfs_metadata id 2 client io 338 KiB/s rd, 43 MiB/s wr, 84 op/s rd, 26 op/s wr I realise, of course, that this is a momentary display of statistics, but I see this unbalanced r/w activity consistently when monitoring it live. I would like some insight into what may be causing this large imbalance in r/w, especially since I'm in a read-intensive (web hosting) environment. Some of it may be expected in when considering details of my environment and CephFS implementation specifics, so please ask away if more details are needed. With my experience using NFS, I would start by looking at client io stats, like `nfsstat` and tuning e.g. mount options, but I haven't been able to find such statistics for CephFS clients. Is there anything of the sort for CephFS? Are similar stats obtainable in some other way? This might be a somewhat broad question and shallow description, so yeah, let me know if there's anything you would like more details on. Thanks a lot, Samy

5 8

Is there a performance impact of enabling the iostat module?
by Marc Roos 09 Feb '20

09 Feb '20

Is there an performance impact of 'ceph mgr module enable iostat'?

1 0

'ceph mgr module ls' does not show rbd_support
by Marc Roos 09 Feb '20

09 Feb '20

ceph mgr module ls does not show the rbd_support, should this not be listed?

1 0

about rbd-nbd auto mount at boot time
by 6442642＠163.com 09 Feb '20

09 Feb '20

want to auto mount the ceph blockdriver at the boot time .becauseI use RBD-mirror,I just only can use nbd type to mount the blockdriver.I try to use /etc/ceph/rbdmap and /etc/fstab with _netdev.But I find it cann't mount at boot time.It just map for ndb type as /dev/nbd0.I need to mount manually use command "mount /dev/nbd0 /mnt".Has anyone solved this problem?

1 0

As mon should be deployed in odd numbers, and I have a fourth node, can I deploy a fourth mds only? - 14.2.7
by marcopizzolo＠gmail.com 07 Feb '20

07 Feb '20

Presently I have about 1.2B objects (400M w/3 Replicas) and I'm finding the PG scrubbing and deep scrubbing are not completing. There is only 1 client accessing the data, a Samba server. I found large disparities in PG Distribution and Drive utilization. I enabled pg_autoscaler and found that it was reducing the number of PGs per OSD from 116 to 104.5 at this time, but it wasn't helping with space consumption equalization. I found loadbalancer and enabled that and it is in the process of evening out. As we were also having mds crashes even after increasing mds memory, i tried enabling multimds with rank:2 I currently have only 1 spare and would like to potentially enable the mds component on the fourth node (no mon present) but am having some difficulty. Is mon a requirement? I tried ceph-deploy mds create node4 but am having errors. I tried manually creating the /var/lib/ceph/mds/node4 directory and the command to create the keyring but still no joy. What am i missing? Thanks,

3 3

MDS daemons seem to not be getting assigned a rank and crash. Nautilus 14.2.7
by Michael Sudnick 07 Feb '20

07 Feb '20

I made a bug report here: https://tracker.ceph.com/issues/44023 I updated from 14.2.6 yesterday and after the update my MDS daemons would not start. I looked at the logs and seemed to initially have an auth error. Setting the keyring location manually in ceph.conf fixed that, but I now get an error where my MDS daemons try to reconnect replay and rejoin, but then crash. Any suggestions on what I can do to trouble shoot? The big tracker post has some logs attached. Rolling back did not fix things. Thank you. -Michael

1 0

getting rid of incomplete pg errors
by Hartwig Hauschild 07 Feb '20

07 Feb '20

Hi. before I descend into what happened and why it happened: I'm talking about a test-cluster so I don't really care about the data in this case. We've recently started upgrading from luminous to nautilus, and for us that means we're retiring ceph-disk in favour of ceph-volume with lvm and dmcrypt. Our setup is in containers and we've got DBs separated from Data. When testing our upgrade-path we discovered that running the host on ubuntu-xenial and the containers on centos-7.7 leads to lvm inside the containers not using lvmetad because it's too old. That in turn means that not running `vgscan --cache` on the host before adding a LV to a VG essentially zeros the metadata for all LVs in that VG. That happened on two out of three hosts for a bunch of OSDs and those OSDs are gone. I have no way of getting them back, they've been overwritten multiple times trying to figure out what went wrong. So now I have a cluster that's got 16 pgs in 'incomplete', 14 of them with 0 objects, 2 with about 150 objects each. I have found a couple of howtos that tell me to use ceph-objectstore-tool to find the pgs on the active osds and I've given that a try, but ceph-objectstore-tool always tells me it can't find the pg I am looking for. Can I tell ceph to re-init the pgs? Do I have to delete the pools and recreate them? There's no data I can't get back in there, I just don't feel like scrapping and redeploying the whole cluster. -- Cheers, Hardy

3 4

osd_memory_target ignored
by Frank Schilder 07 Feb '20

07 Feb '20

I recently upgraded from 13.2.2 to 13.2.8 and observe two changes that I struggle with: - from release notes: The bluestore_cache_* options are no longer needed. They are replaced by osd_memory_target, defaulting to 4GB. - the default for bluestore_allocator has changed from stupid to bitmap, which seem to conflict each other, or at least I seem unable to achieve what I want. I have a number of OSDs for which I would like to increase the cache size. In the past I used bluestore_cache_size=8G and it worked like a charm. I now changed that to osd_memory_target=8G without any effect. The usage stays at 4G and the virtual size is about 5G. I would expect both to be close to 8G. The read cache for these OSDs usually fills up within a few hours. The cluster is now running a few days with the new configs to no avail. The documentation of osd_memory_target refers to tcmalloc a lot. Is this in conflict with allocator=bitmap? If so, what is the way to tune cache sizes (say if tcmalloc is not used/how to check?)? Are bluestore_cache_* indeed obsolete as the above release notes suggest, or is this not true? Many thanks for your help. Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

2 8

Warning about non-existing (?) large omap object
by Alexandre Berthaud 07 Feb '20

07 Feb '20

Hello everyone, For the second time now, we have a warning on a Ceph cluster about a large omap object. This object is supposed to be in default.rgw.log, except after running listomapkeys on every object of that pool (`for i in `rados --cluster=ceph-par -p default.rgw.log ls`; do echo -n "$i:"; rados --cluster=ceph-par -p default.rgw.log listomapkeys $i |wc -l; done`), we get 0 for every object. As I understand it, objects in that pool are short-lived so the object which triggered the warning does not even exist anymore. Am I mistaken? The first time, we tried quite a lot of things, like triggering a deep scrub on the relevant pg, even looking into other pools (just in case) if we could find object with large omap objects (and we did not). The warning ended up going away on its own after a bit less than a week iirc. Is this a bug? Is there a way to clear this warning (if it is indeed about a non-existing object)? Thanks,

1 0

Reorganize crush map and replicated rules
by 5 db S 07 Feb '20

07 Feb '20

Hi, I'd like to fix the crush tree and crush rule and would like to know the correct steps and worst case scenario what can happen during the maintenance. Steps should be like: 1. Create the rack structured crush tree under root default 2. create the replicated crush rules 3. Move the nodes under the crush tree mixed with ssd and hdd 4. Apply the new replicated rules ... ?? Here is the tree: https://pastebin.com/raw/CuuzuBsz Thank you

1 1

"mds daemon damaged" after restarting MDS - Filesystem DOWN
by Luca Cervigni 07 Feb '20

07 Feb '20

Dear all Running nautilus 14.2.7. The data in the FS are important and cannot be lost. Today I increased the PGS of the volume pool from 8k to 16k. The active mds started reporting slow ops. (the filesystem is not in the volume pool). After few hours the FS was very slow, I reduced the backfill to 1 and since the situation was not improving, I restarted the MDS (no other standby MDSs. it was a single mds). After that the crash. The mds does not goes back up with this error: 020-02-07 07:03:32.477 7fbf69647700 -1 NetHandler create_socket couldn't create socket (97) Address family not supported by protocol 2020-02-07 07:03:32.541 7fbf65e6a700 1 mds.ceph-mon-01 Updating MDS map to version 48461 from mon.2 2020-02-07 07:03:37.613 7fbf65e6a700 1 mds.ceph-mon-01 Updating MDS map to version 48462 from mon.2 2020-02-07 07:03:37.613 7fbf65e6a700 1 mds.ceph-mon-01 Map has assigned me to become a standby 2020-02-07 07:14:11.789 7fbf66e42700 -1 received signal: Terminated from /sbin/init (PID: 1) UID: 0 2020-02-07 07:14:11.789 7fbf66e42700 -1 mds.ceph-mon-01 *** got signal Terminated *** 2020-02-07 07:14:11.789 7fbf66e42700 1 mds.ceph-mon-01 suicide! Wanted state up:standby 2020-02-07 07:14:12.565 7fbf65e6a700 0 ms_deliver_dispatch: unhandled message 0x563fcb438d00 mdsmap(e 48465) v1 from mon.2 v1:10.3.78.32:6789/0 2020-02-07 07:25:16.782 7f26c39de2c0 0 set uid:gid to 64045:64045 (ceph:ceph) 2020-02-07 07:25:16.782 7f26c39de2c0 0 ceph version 14.2.7 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8) nautilus (stable), process ceph-mds, pid 3724 2020-02-07 07:25:16.782 7f26c39de2c0 0 pidfile_write: ignore empty --pid-file 2020-02-07 07:25:16.786 7f26b5326700 -1 NetHandler create_socket couldn't create socket (97) Address family not supported by protocol 2020-02-07 07:25:16.790 7f26b1b49700 1 mds.ceph-mon-01 Updating MDS map to version 48472 from mon.0 2020-02-07 07:25:17.691 7f26b1b49700 1 mds.ceph-mon-01 Updating MDS map to version 48473 from mon.0 2020-02-07 07:25:17.691 7f26b1b49700 1 mds.ceph-mon-01 Map has assigned me to become a standby 2020-02-07 07:29:50.306 7f26b2b21700 -1 received signal: Terminated from /sbin/init (PID: 1) UID: 0 2020-02-07 07:29:50.306 7f26b2b21700 -1 mds.ceph-mon-01 *** got signal Terminated *** 2020-02-07 07:29:50.306 7f26b2b21700 1 mds.ceph-mon-01 suicide! Wanted state up:standby 2020-02-07 07:29:50.526 7f26b5b27700 1 mds.beacon.ceph-mon-01 discarding unexpected beacon reply down:dne seq 70 dne 2020-02-07 07:29:52.802 7f26b1b49700 0 ms_deliver_dispatch: unhandled message 0x55ef110ab200 mdsmap(e 48474) v1 from mon.0 v1:10.3.78.22:6789/0 Rebooting did not help I asked #CEPH OFTC and they suggested to bring up another "fresh" mds. I did that, and they do not start, going to standby. LOGS: 2020-02-07 07:12:46.696 7fe4b388b2c0 0 set uid:gid to 64045:64045 (ceph:ceph) 2020-02-07 07:12:46.696 7fe4b388b2c0 0 ceph version 14.2.7 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8) nautilus (stable), process ceph-mds, pid 74742 2020-02-07 07:12:46.696 7fe4b388b2c0 0 pidfile_write: ignore empty --pid-file 2020-02-07 07:12:46.704 7fe4a19f6700 1 mds.ceph-mon-02 Updating MDS map to version 48462 from mon.0 2020-02-07 07:12:47.456 7fe4a19f6700 1 mds.ceph-mon-02 Updating MDS map to version 48463 from mon.0 2020-02-07 07:12:47.456 7fe4a19f6700 1 mds.ceph-mon-02 Map has assigned me to become a standby 2020-02-07 07:14:16.615 7fe4a29ce700 -1 received signal: Terminated from /sbin/init (PID: 1) UID: 0 2020-02-07 07:14:16.615 7fe4a29ce700 -1 mds.ceph-mon-02 *** got signal Terminated *** 2020-02-07 07:14:16.615 7fe4a29ce700 1 mds.ceph-mon-02 suicide! Wanted state up:standby 2020-02-07 07:14:16.947 7fe4a51d3700 1 mds.beacon.ceph-mon-02 discarding unexpected beacon reply down:dne seq 24 dne 2020-02-07 07:14:18.715 7fe4a19f6700 0 ms_deliver_dispatch: unhandled message 0x5602fbc6df80 mdsmap(e 48466) v1 from mon.0 v2:10.3.78.22:3300/0 2020-02-07 07:25:02.093 7f3c2f92a2c0 0 set uid:gid to 64045:64045 (ceph:ceph) 2020-02-07 07:25:02.093 7f3c2f92a2c0 0 ceph version 14.2.7 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8) nautilus (stable), process ceph-mds, pid 75471 2020-02-07 07:25:02.093 7f3c2f92a2c0 0 pidfile_write: ignore empty --pid-file 2020-02-07 07:25:02.097 7f3c1da95700 1 mds.ceph-mon-02 Updating MDS map to version 48471 from mon.2 2020-02-07 07:25:06.413 7f3c1da95700 1 mds.ceph-mon-02 Updating MDS map to version 48472 from mon.2 2020-02-07 07:25:06.413 7f3c1da95700 1 mds.ceph-mon-02 Map has assigned me to become a standby 2020-02-07 07:29:56.869 7f3c1ea6d700 -1 received signal: Terminated from /sbin/init (PID: 1) UID: 0 2020-02-07 07:29:56.869 7f3c1ea6d700 -1 mds.ceph-mon-02 *** got signal Terminated *** 2020-02-07 07:29:56.869 7f3c1ea6d700 1 mds.ceph-mon-02 suicide! Wanted state up:standby 2020-02-07 07:29:58.113 7f3c1da95700 0 ms_deliver_dispatch: unhandled message 0x563c5df33f80 mdsmap(e 48475) v1 from mon.2 v2:10.3.78.32:3300/0 Here ceph status cluster: id: a8dde71d-ca7b-4cf5-bd38-8989c6a27011 health: HEALTH_ERR 1 filesystem is degraded 1 filesystem is offline 1 mds daemon damaged 2 daemons have recently crashed services: mon: 3 daemons, quorum ceph-mon-01,ceph-mon-02,ceph-mon-03 (age 41m) mgr: ceph-mon-02(active, since 41m), standbys: ceph-mon-03, ceph-mon-01 mds: pawsey-sync-fs:0/1, 1 damaged osd: 925 osds: 715 up (since 2h), 715 in (since 23h) rgw: 3 daemons active (radosgw-01, radosgw-02, radosgw-03) data: pools: 24 pools, 26569 pgs objects: 52.64M objects, 199 TiB usage: 685 TiB used, 6.7 PiB / 7.3 PiB avail pgs: 26513 active+clean 54 active+clean+scrubbing+deep 2 active+clean+scrubbing Ceph osd ls detail: https://pastebin.com/raw/bxi4HSa5 the metadata pool is on NVMe Can anyone give me some help? Any command run like journal repairs do not work as they expect the MDs to be up. Thanks Cheers -- Luca Cervigni Infrastructure Architect Tel. +61864368802 Pawsey Supercomputing Centre 1 Bryce Ave, Kensington WA 6151 Australia

1 0

Ubuntu 18.04.4 Ceph 12.2.12
by Atherion 06 Feb '20

06 Feb '20

Hi Ceph Community. We currently have a luminous cluster running and some machines still on Ubuntu 14.04 We are looking to upgrade these machines to 18.04 but the only upgrade path for luminous with the ceph repo is through 16.04. It is doable to get to Mimic but then we have to upgrade all those machines to 16.04 but then we have to upgrade again to 18.04 when we get to Mimic, it is becoming a huge time sink. I did notice in the Ubuntu repos they have added 12.2.12 in 18.04.4 release. Is this a reliable build we can use? https://ubuntu.pkgs.org/18.04/ubuntu-proposed-main-amd64/ceph_12.2.12-0ubun… If so then we can go straight to 18.04.4 and not waste so much time. Best

3 3

Re: Different memory usage on OSD nodes after update to Nautilus
by Massimo Sgaravatto 06 Feb '20

06 Feb '20

Thanks for your feedback The Ganglia graphs are available here: https://cernbox.cern.ch/index.php/s/0xBDVwNkRqcoGdF Replying to the other questions: - Free Memory in ganglia is derived from "MemFree" in /proc/meminfo - Memory Buffers in ganglia is derived from "Buffers" in /proc/meminfo - On this host, the OSDs are 6TB. On other hosts we have 10TB OSDs - "osd memory target" is set to ~ 4.5 GB (actually, while debugging this issue, I have just lowered the value to 3.2 GB) - "ceph tell osd.x heap stats" basically always reports 0 (or a very low value) for "Bytes in page heap freelist" and a heap release doesn't change the memory usage - I can agree that swap is antiquated. But so far it was simply not used and didn't cause any problems. At any rate I am now going to remove the swap (or setting the swappiness to 0). Thanks again ! Cheers, Massimo On Thu, Feb 6, 2020 at 6:28 PM Anthony D'Atri <aad(a)dreamsnake.net> wrote: > Attachments are usually filtered by mailing lists. Yours did not come > through. A URL to Skitch or some other hosting works better. > > Your kernel version sounds like RHEL / CentOS? I can say that memory > accounting definitely did change between upstream 3.19 and 4.9 > > > osd04-cephstorage1-gsc:~ # head /proc/meminfo > MemTotal: 197524684 kB > MemFree: 80388504 kB > MemAvailable: 86055708 kB > Buffers: 633768 kB > Cached: 4705408 kB > SwapCached: 0 kB > > Specifically, node_memory_Active as reported by node_exporter changes > dramatically, and MemAvailable is the more meaningful metric. What is your > “FreeMem” metric actually derived from? > > 64GB for 10 OSDs might be on the light side, how large are those OSDs? > > For sure swap is antiquated. If your systems have any swap provisioned at > all, you’re doing it wrong. I’ve had good results setting it to 1. > > Do `ceph daemon osd.xx heap stats`, see if your OSD processes have much > unused memory that has not been released to the OS. If they do, “heap > release” can be useful. > > > > > On Feb 6, 2020, at 9:08 AM, Massimo Sgaravatto < > massimo.sgaravatto(a)gmail.com> wrote: > > > > Dear all > > > > In the mid of January I updated my ceph cluster from Luminous to > Nautilus. > > > > Attached you can see the memory metrics collected on one OSD node (I see > > the very same behavior on all OSD hosts) graphed via Ganglia > > This is Centos 7 node, with 64 GB of RAM, hosting 10 OSDs. > > > > So before the update there were about 20 GB of FreeMem. > > Now FreeMem is basically 0, but I see 20 GB of Buffers, > > > > I guess this triggered some swapping, probably because I forgot to > > set vm.swappiness to 0 (it was set to 60, the default value). > > > > I was wondering if this the expected behavior > > > > PS: Actually besides updating ceph, I also updated all the other packages > > (yum update), so I am not sure that this different memory usage is > because > > of the ceph update > > For the record in this update the kernel was updated from 3.10.0-1062.1.2 > > to 3.10.0-1062.9.1 > > > > Thanks, Massimo > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

Different memory usage on OSD nodes after update to Nautilus
by Massimo Sgaravatto 06 Feb '20

06 Feb '20

Dear all In the mid of January I updated my ceph cluster from Luminous to Nautilus. Attached you can see the memory metrics collected on one OSD node (I see the very same behavior on all OSD hosts) graphed via Ganglia This is Centos 7 node, with 64 GB of RAM, hosting 10 OSDs. So before the update there were about 20 GB of FreeMem. Now FreeMem is basically 0, but I see 20 GB of Buffers, I guess this triggered some swapping, probably because I forgot to set vm.swappiness to 0 (it was set to 60, the default value). I was wondering if this the expected behavior PS: Actually besides updating ceph, I also updated all the other packages (yum update), so I am not sure that this different memory usage is because of the ceph update For the record in this update the kernel was updated from 3.10.0-1062.1.2 to 3.10.0-1062.9.1 Thanks, Massimo

1 0

RBD cephx read-only key
by Andras Pataki 06 Feb '20

06 Feb '20

I'm trying to set up a cephx key to mount RBD images read-only. I have the following two keys: [client.rbd] key = xxx caps mgr = "profile rbd" caps mon = "profile rbd" caps osd = "profile rbd pool=rbd_vm" [client.rbd-ro] key = xxx caps mgr = "profile rbd-read-only" caps mon = "profile rbd" caps osd = "profile rbd-read-only pool=rbd_vm" The following works: # rbd map --pool rbd_vm andras_test --name client.rbd /dev/rbd0 and so does this: # rbd map --pool rbd_vm andras_test --name client.rbd --read-only /dev/rbd0 but the using the rbd-ro key doesn't work: # rbd map --pool rbd_vm andras_test --name client.rbd-ro --read-only rbd: sysfs write failed In some cases useful info is found in syslog - try "dmesg | tail". rbd: map failed: (1) Operation not permitted the logs only have the following: [1281776.788709] libceph: mon4 10.128.150.14:6789 session established [1281776.801747] libceph: client88900164 fsid d7b33135-0940-4e48-8aa6-1d2026597c2f The back end of mimic 13.2.8, the kernel is the CentOS kernel 3.10.0-957.27.2.el7.x86_64 Any ideas what I'm doing wrong here? Andras

2 2

Need info about ceph bluestore autorepair
by Mario Giammarco 06 Feb '20

06 Feb '20

Hello, if I have a pool with replica 3 what happens when one replica is corrupted? I suppose ceph detects bad replica using checksums and replace it with good one If I have a pool with replica 2 what happens? Thanks, Mario

2 1

Stuck with an unavailable iscsi gateway
by jcharles＠provectio.fr 06 Feb '20

06 Feb '20

Hello I can't find, a way to resolve my problem. I lost a iscsi gateway in a pool of 4 gateway, there is 3 lefts. I can't delete the lost gateway from host and I can't change the Owner of the resource owned by the lost gateway. Finally, I have ressources which are inaccessible from clients and I can't reconfigure them because of the lost gateway. Please, tell me there is a way to remove a lost gateway and that I won't be stuck for ever. If I do delete compute04.adm.local it answers Failed : Gateway deletion failed, gateway(s) unavailable:compute04.adm.local(UNKNOWN state) I saw a reference of my problem in the thread "Error in add new ISCSI gateway" but unfortunatly, no answer seems to be avalaible. Thanks for any help

2 1

Understanding Bluestore performance characteristics
by Bradley Kite 06 Feb '20

06 Feb '20

Hi, We have a production cluster of 27 OSD's across 5 servers (all SSD's running bluestore), and have started to notice a possible performance issue. In order to isolate the problem, we built a single server with a single OSD, and ran a few FIO tests. The results are puzzling, not that we were expecting good performance on a single OSD. In short, with a sequential write test, we are seeing huge numbers of reads hitting the actual SSD Key FIO parameters are: [global] pool=benchmarks rbdname=disk-1 direct=1 numjobs=2 iodepth=1 blocksize=4k group_reporting=1 [writer] readwrite=write iostat results are: Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util nvme0n1 0.00 105.00 4896.00 294.00 312080.00 1696.00 120.92 17.25 3.35 3.55 0.02 0.02 12.60 There are nearly ~5000 reads/second (~300 MB/sec), compared with only ~300 writes (~1.5MB/sec), when we are doing a sequential write test? The system is otherwise idle, with no other workload. Running the same fio test with only 1 thread (numjobs=1) still shows a high number of reads (110). Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util nvme0n1 0.00 1281.00 110.00 1463.00 440.00 12624.00 16.61 0.03 0.02 0.05 0.02 0.02 3.40 Can anyone kindly offer any comments on why we are seeing this behaviour? I can understand if there's the occasional read here and there if RocksDB/WAL entries need to be read from disk during the sequential write test, but this seems significantly high and unusual. FIO results (numjobs=2) writer: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=rbd, iodepth=1 ... fio-3.7 Starting 2 processes Jobs: 1 (f=1): [W(1),_(1)][52.4%][r=0KiB/s,w=208KiB/s][r=0,w=52 IOPS][eta 01m:00s] writer: (groupid=0, jobs=2): err= 0: pid=19553: Mon Feb 3 22:46:16 2020 write: IOPS=34, BW=137KiB/s (140kB/s)(8228KiB/60038msec) slat (nsec): min=5402, max=77083, avg=27305.33, stdev=7786.83 clat (msec): min=2, max=210, avg=58.32, stdev=70.54 lat (msec): min=2, max=210, avg=58.35, stdev=70.54 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 3], | 30.00th=[ 3], 40.00th=[ 3], 50.00th=[ 54], 60.00th=[ 62], | 70.00th=[ 65], 80.00th=[ 174], 90.00th=[ 188], 95.00th=[ 194], | 99.00th=[ 201], 99.50th=[ 203], 99.90th=[ 209], 99.95th=[ 209], | 99.99th=[ 211] bw ( KiB/s): min= 24, max= 144, per=49.69%, avg=68.08, stdev=38.22, samples=239 iops : min= 6, max= 36, avg=16.97, stdev= 9.55, samples=239 lat (msec) : 4=49.83%, 10=0.10%, 100=29.90%, 250=20.18% cpu : usr=0.08%, sys=0.08%, ctx=2100, majf=0, minf=118 IO depths : 1=105.3%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2057,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=137KiB/s (140kB/s), 137KiB/s-137KiB/s (140kB/s-140kB/s), io=8228KiB (8425kB), run=60038-60038msec

5 11

Strange performance drop and low oss performance
by quexian da 06 Feb '20

06 Feb '20

Hello, I'm a beginner on ceph. I set up three ceph clusters on google cloud. Cluster1 has three nodes and each node has three disks. Cluster2 has three nodes and each node has two disks. Cluster3 has five nodes and each node has five disks. All disks are HDD. Disk speed shown by `dd if=/dev/zero of=here bs=1G count=1 oflag=direct` is 117MB/s. The network is 10Gbps. Ceph version is 12.2.12. I found something strange: 1. When running `rados bench`, the write performance of all clusters drops dramatically after a few minutes. I created a pool named "scbench" with replicated size 1 (I know it is not safe but I want the highest write speed). The write performance (shown by rados bench -p scbench 1000 write) before and after the drop are: cluster1: 297MB/s 94.5MB/s cluster2: 304MB/s 67.4MB/s cluster3: 494MB/s 267.6MB/s It looks like the performance before the drop is nodes_num * 100MB/s, and the performance after the drop is about osds_num * 10MB/s. I have no idea why there is such a drop and why the performances before the drop are linear with nodes_num. 2. The write performance of object storage (shown by swift-bench -c 64 -s 4096000 -n 100000 -g 0 swift.conf) is much lower than that of storage cluster(shown by rados bench -p scbench 1000 write). I have set the replicated size of "default.rgw.buckets.data" and "default.rgw.buckets.index" to 1 The speed of cluster1 oss is 117MB/s (before the drop) and 26MB/s (after the drop), and the speed of cluster3 oss is 118MB/s (the drop does not happen). Is it normal that the oss write performance is worse than rados write performance? If not, how can I solve the problem? Thanks!

3 6

Fwd: BlueFS spillover yet again
by Vladimir Prokofev 06 Feb '20

06 Feb '20

Cluster upgraded from 12.2.12 to 14.2.5. All went smooth, except BlueFS spillover warning. We create OSDs with ceph-deploy, command goes like this: ceph-deploy osd create --bluestore --data /dev/sdf --block-db /dev/sdb5 --block-wal /dev/sdb6 ceph-osd3 where block-db and block-wal are SSD partitions. Default ceph-deploy settings created partitions ~1GB which is, of course, too small. So we redeployed OSDs using manually partitioned SSD for block-db/block-wal with sizes of 20G/5G respectively. But now we still get BlueFS spillover warning for redeployed OSDs: osd.10 spilled over 2.4 GiB metadata from 'db' device (2.8 GiB used of 19 GiB) to slow device osd.19 spilled over 3.7 GiB metadata from 'db' device (2.7 GiB used of 19 GiB) to slow device osd.20 spilled over 4.2 GiB metadata from 'db' device (2.6 GiB used of 19 GiB) to slow device osd size is 1.8 TiB. These OSDs are used primarily for RBD as a backup drives, so a lot of snapshots held there. They also have RGW pool assigned to them, but it has no data. I know of sizing recommendations[1] for block-db/block-wal, but I assumed since it's primarily RBD 1%(~20G) should be enough. Also, compaction stats doesn't make sense to me[2]. It states that sum of DB is only 5.08GB, that should be placed on block-db without a problem? Am I understanding all this wrong? Should block-db size be greater in my case? [1] https://docs.ceph.com/docs/master/rados/configuration/bluestore-config-ref/… [2] osd.10 logs as an example https://pastebin.com/hC6w6jSn

5 5

osd is immidietly down and uses CPU full.
by 西宮牧人 06 Feb '20

06 Feb '20

Servers: 6 (include 7osds) total 42osdsl OS: Centos7 Ceph: 10.2.5 Hi, everyone The cluster is used for VM image storage and object storage. And I have a bucket which has more than 20 million objects. Now, I have a problem that cluster blocks operation. Suddenly cluster blocked operations, then VMs can't read disk. After a few hours, osd.1 was down. There is no disk fail messages in dmesg. And no error is in smartctl -a /dev/sde. I tried to wake up osd.1, but osd.1 is down soon. Just after re-waking up osd.1, VM can access to the disk. But osd.1 always uses 100% CPU, then cluster marked osd.1 down and the osd was dead by suicide timeout. I found that the osdmap epoch of osd.1 is different from other one. So I think osd.1 was dead. Question. (1) Why does the epoch of osd.1 differ from other osds ones ? I checked all osds oldest_map and newest_map by ~ceph daemon osd.X status~ All osd's ecpoch are same number except osd.1 (2) Why does osd.1 use CPU full? After the cluster marked osd.1 down, osd.1 keeps up busy. When I execute "ceph tell osd.1 injectargs --debug-ms 5/1", osd.1 doesn't answer. Thank you. -- Makito

8 8

data loss on full file system?
by Håkan T Johansson 05 Feb '20

05 Feb '20

Hi, for test purposes, I have set up two 100 GB OSDs, one taking a data pool and the other metadata pool for cephfs. Am running 14.2.6-1-gffd69200ad-1 with packages from https://mirror.croit.io/debian-nautilus Am then running a program that creates a lot of 1 MiB files by calling fopen() fwrite() fclose() for each of them. Error codes are checked. This works successfully for ~100 GB of data, and then strangely also succeeds for many more 100 GB of data... ?? All written files have size 1 MiB with 'ls', and thus should contain the data written. However, on inspection, the files written after the first ~100 GiB, are full of just 0s. (hexdump -C) To further test this, I use the standard tool 'cp' to copy a few random-content files into the full cephfs filessystem. cp reports no complaints, and after the copy operations, content is seen with hexdump -C. However, after forcing the data out of cache on the client by reading other earlier created files, hexdump -C show all-0 content for the files copied with 'cp'. Data that was there is suddenly gone...? I am new to ceph. Is there an option I have missed to avoid this behaviour? (I could not find one in https://docs.ceph.com/docs/master/man/8/mount.ceph/ ) Is this behaviour related to https://docs.ceph.com/docs/mimic/cephfs/full/ ? (That page states 'sometime after a write call has already returned 0'. But if write returns 0, then no data has been written, so the user program would not assume any kind of success.) Best regards, Håkan

2 4

Problem with OSD - stuck in CPU loop after rbd snapshot mount
by Jan Pekař - Imatic 05 Feb '20

05 Feb '20

Hi all, I have small cluster and yesterday I tried to mount older RBD snapshot torecover data. (I have approx. 230 daily snapshots of one RBD image on my small ceph). After I did mount and ls operation, cluster was stuck and I notice that 2of my OSD's eaten CPU and raise in memory usage (more than 4x more memory than usual). Cluster was unusable after that and it didn't help stopping rbd mount operation and restarting OSD's etc. Strange is, that it affects only 2 of my OSD's - one is on NVME and has cache PG's mapped, other one is rotational HDD. IOPS are normal, only CPU and memory usage are unusual. I noticed hundred thousands of slow requests (on idle cluster) so I startto debug OSD operations. You can find OSD 8 log below. OSD 9 contains nvme cached PG's and there is nothing unusual in standard log. In the middle of log I raised debug level to 10. It look's to me, that it is trying to do one operation with one snapshotted RBD object in PG 2.b (rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3) in a loop. Is there any way how to stop that? What is causing the problem? Can I somehow flush OSD's "journals" or "queues" to stop current operation and make cluster usable again? Is there any way how I can access old snapshot data? Problem started on Luminous 12.2.12 and upgrade to 12.2.13 didn't help. Thank you Jan Pekar 2020-02-03 15:09:22.470277 7f51f6af7700 0 log_channel(cluster) log [WRN] : 30966 slow requests, 5 included below; oldest blocked for > 37.505482 secs 2020-02-03 15:09:22.470336 7f51f6af7700 0 log_channel(cluster) log [WRN] : slow request 37.503404 seconds old, received at 2020-02-03 15:08:44.956422: osd_op(osd.9.59923:2670991 2.b 2.162d8f8b (undecoded) ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected e59939) currently queued_for_pg 2020-02-03 15:09:22.470360 7f51f6af7700 0 log_channel(cluster) log [WRN] : slow request 37.500695 seconds old, received at 2020-02-03 15:08:44.959132: osd_op(osd.9.59923:2671023 2.b 2.162d8f8b (undecoded) ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected e59939) currently queued_for_pg 2020-02-03 15:09:22.470366 7f51f6af7700 0 log_channel(cluster) log [WRN] : slow request 37.498013 seconds old, received at 2020-02-03 15:08:44.961814: osd_op(osd.9.59923:2671055 2.b 2.162d8f8b (undecoded) ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected e59939) currently queued_for_pg 2020-02-03 15:09:22.470371 7f51f6af7700 0 log_channel(cluster) log [WRN] : slow request 37.494836 seconds old, received at 2020-02-03 15:08:44.964990: osd_op(osd.9.59923:2671087 2.b 2.162d8f8b (undecoded) ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected e59939) currently queued_for_pg 2020-02-03 15:09:22.470376 7f51f6af7700 0 log_channel(cluster) log [WRN] : slow request 37.491950 seconds old, received at 2020-02-03 15:08:44.967877: osd_op(osd.9.59923:2671119 2.b 2.162d8f8b (undecoded) ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected e59939) currently queued_for_pg ... 2020-02-03 15:09:35.537259 7f51ddac5700 0 osd.8 59939 do_command r=0 2020-02-03 15:09:35.588753 7f51ddac5700 0 osd.8 59939 do_command r=0 Set debug_osd to 10/10 2020-02-03 15:09:35.588814 7f51ddac5700 0 log_channel(cluster) log [INF] : Set debug_osd to 10/10 2020-02-03 15:09:35.597667 7f51f0945700 2 osd.8 59939 ms_handle_reset con 0x5647736ac800 session 0x564785519e00 2020-02-03 15:09:35.599056 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] dropping ondisk_read_lock 2020-02-03 15:09:35.599209 7f51e0acb700 10 osd.8 59939 dequeue_op 0x5647873d5800 finish 2020-02-03 15:09:35.599250 7f51e0acb700 10 osd.8 59939 dequeue_op 0x5647873d59c0 prio 63 cost 29 latency 22.224501 osd_op(osd.9.59923:2779946 2.b 2.162d8f8b (undecoded) ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected e59939) v8 pg pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] 2020-02-03 15:09:35.599293 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] _handle_message: 0x5647873d59c0 2020-02-03 15:09:35.599347 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] do_op osd_op(osd.9.59923:2779946 2.b 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected e59939) v8 may_read -> write-ordered flags ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected 2020-02-03 15:09:35.599398 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] find_object_context 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 @7e3 snapset c07=[c07,c04,c01,bfe,bf9,bf6,bf3,bf0,bed,bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab,ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d,957,951,94b,945,93f,939,933,92d,927,921,91b,915,90f,909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f,849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9,7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff,9]:{11=[9],7e3=[7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff],849=[849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9],909=[909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f],915=[915,90f],957=[957,951,94b,945,93f,939,933,92d,927,921,91b],ba5=[ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d],bea=[bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab],bf9=[bf9,bf6,bf3,bf0,bed],c07=[c07,c04,c01,bfe]} map_snapid_to_clone=true 2020-02-03 15:09:35.599506 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] find_object_context 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 @7e3 snapset c07=[c07,c04,c01,bfe,bf9,bf6,bf3,bf0,bed,bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab,ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d,957,951,94b,945,93f,939,933,92d,927,921,91b,915,90f,909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f,849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9,7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff,9]:{11=[9],7e3=[7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff],849=[849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9],909=[909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f],915=[915,90f],957=[957,951,94b,945,93f,939,933,92d,927,921,91b],ba5=[ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d],bea=[bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab],bf9=[bf9,bf6,bf3,bf0,bed],c07=[c07,c04,c01,bfe]} maps to 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 2020-02-03 15:09:35.599606 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] get_object_context: found obc in cache: 0x564785fb3880 2020-02-03 15:09:35.613724 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] get_object_context: 0x564785fb3880 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 rwstate(none n=0 w=0) oi: 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3(58682'4215046 osd.8.0:393237 dirty|data_digest|omap_digest s 4194304 uv 7633387 dd 41a45f29 od ffffffff alloc_hint [0 0 0]) exists: 1 ssc: 0x564783df4160 snapset: c07=[c07,c04,c01,bfe,bf9,bf6,bf3,bf0,bed,bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab,ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d,957,951,94b,945,93f,939,933,92d,927,921,91b,915,90f,909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f,849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9,7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff,9]:{11=[9],7e3=[7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff],849=[849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9],909=[909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f],915=[915,90f],957=[957,951,94b,945,93f,939,933,92d,927,921,91b],ba5=[ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d],bea=[bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab],bf9=[bf9,bf6,bf3,bf0,bed],c07=[c07,c04,c01,bfe]} 2020-02-03 15:09:35.613881 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] find_object_context 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 @7e3 snapset c07=[c07,c04,c01,bfe,bf9,bf6,bf3,bf0,bed,bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab,ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d,957,951,94b,945,93f,939,933,92d,927,921,91b,915,90f,909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f,849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9,7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff,9]:{11=[9],7e3=[7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff],849=[849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9],909=[909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f],915=[915,90f],957=[957,951,94b,945,93f,939,933,92d,927,921,91b],ba5=[ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d],bea=[bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab],bf9=[bf9,bf6,bf3,bf0,bed],c07=[c07,c04,c01,bfe]} 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 HIT 2020-02-03 15:09:35.614024 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] execute_ctx 0x5647843a3000 2020-02-03 15:09:35.614053 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] execute_ctx 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 [copy-get max 8388608] ov 58682'4215046 2020-02-03 15:09:35.614077 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] taking ondisk_read_lock 2020-02-03 15:09:35.614095 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] do_osd_op 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 [copy-get max 8388608] 2020-02-03 15:09:35.614112 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] do_osd_op copy-get max 8388608 2020-02-03 15:09:35.637123 7f51f72f8700 10 osd.8 59939 tick 2020-02-03 15:09:35.637149 7f51f72f8700 10 osd.8 59939 do_waiters -- start 2020-02-03 15:09:35.637152 7f51f72f8700 10 osd.8 59939 do_waiters -- finish 2020-02-03 15:09:35.645761 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] dropping ondisk_read_lock 2020-02-03 15:09:35.645887 7f51e0acb700 10 osd.8 59939 dequeue_op 0x5647873d59c0 finish 2020-02-03 15:09:35.645924 7f51e0acb700 10 osd.8 59939 dequeue_op 0x5647873d5b80 prio 63 cost 29 latency 22.271117 osd_op(osd.9.59923:2779947 2.b 2.162d8f8b (undecoded) ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected e59939) v8 pg pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] 2020-02-03 15:09:35.645960 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] _handle_message: 0x5647873d5b80 2020-02-03 15:09:35.646007 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] do_op osd_op(osd.9.59923:2779947 2.b 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected e59939) v8 may_read -> write-ordered flags ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected 2020-02-03 15:09:35.646047 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] find_object_context 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 @7e3 snapset c07=[c07,c04,c01,bfe,bf9,bf6,bf3,bf0,bed,bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab,ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d,957,951,94b,945,93f,939,933,92d,927,921,91b,915,90f,909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f,849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9,7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff,9]:{11=[9],7e3=[7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff],849=[849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9],909=[909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f],915=[915,90f],957=[957,951,94b,945,93f,939,933,92d,927,921,91b],ba5=[ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d],bea=[bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab],bf9=[bf9,bf6,bf3,bf0,bed],c07=[c07,c04,c01,bfe]} map_snapid_to_clone=true 2020-02-03 15:09:35.646145 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] find_object_context 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 @7e3 snapset c07=[c07,c04,c01,bfe,bf9,bf6,bf3,bf0,bed,bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab,ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d,957,951,94b,945,93f,939,933,92d,927,921,91b,915,90f,909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f,849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9,7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff,9]:{11=[9],7e3=[7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff],849=[849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9],909=[909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f],915=[915,90f],957=[957,951,94b,945,93f,939,933,92d,927,921,91b],ba5=[ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d],bea=[bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab],bf9=[bf9,bf6,bf3,bf0,bed],c07=[c07,c04,c01,bfe]} maps to 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 2020-02-03 15:09:35.646239 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] get_object_context: found obc in cache: 0x564785fb3880 2020-02-03 15:09:35.646255 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] get_object_context: 0x564785fb3880 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 rwstate(none n=0 w=0) oi: 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3(58682'4215046 osd.8.0:393237 dirty|data_digest|omap_digest s 4194304 uv 7633387 dd 41a45f29 od ffffffff alloc_hint [0 0 0]) exists: 1 ssc: 0x564783df4160 snapset: c07=[c07,c04,c01,bfe,bf9,bf6,bf3,bf0,bed,bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab,ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d,957,951,94b,945,93f,939,933,92d,927,921,91b,915,90f,909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f,849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9,7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff,9]:{11=[9],7e3=[7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff],849=[849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9],909=[909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f],915=[915,90f],957=[957,951,94b,945,93f,939,933,92d,927,921,91b],ba5=[ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d],bea=[bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab],bf9=[bf9,bf6,bf3,bf0,bed],c07=[c07,c04,c01,bfe]} 2020-02-03 15:09:35.646359 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] find_object_context 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 @7e3 snapset c07=[c07,c04,c01,bfe,bf9,bf6,bf3,bf0,bed,bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab,ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d,957,951,94b,945,93f,939,933,92d,927,921,91b,915,90f,909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f,849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9,7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff,9]:{11=[9],7e3=[7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff],849=[849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9],909=[909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f],915=[915,90f],957=[957,951,94b,945,93f,939,933,92d,927,921,91b],ba5=[ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d],bea=[bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab],bf9=[bf9,bf6,bf3,bf0,bed],c07=[c07,c04,c01,bfe]} 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 HIT 2020-02-03 15:09:35.646469 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] execute_ctx 0x5647843a3000 2020-02-03 15:09:35.646494 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] execute_ctx 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 [copy-get max 8388608] ov 58682'4215046 2020-02-03 15:09:35.646513 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] taking ondisk_read_lock 2020-02-03 15:09:35.646530 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] do_osd_op 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 [copy-get max 8388608] 2020-02-03 15:09:35.646547 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] do_osd_op copy-get max 8388608 2020-02-03 15:09:35.681653 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] dropping ondisk_read_lock 2020-02-03 15:09:35.681780 7f51e0acb700 10 osd.8 59939 dequeue_op 0x5647873d5b80 finish 2020-02-03 15:09:35.681817 7f51e0acb700 10 osd.8 59939 dequeue_op 0x5647873d5d40 prio 63 cost 29 latency 22.306952 osd_op(osd.9.59923:2779948 2.b 2.162d8f8b (undecoded) ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected e59939) v8 pg pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] 2020-02-03 15:09:35.681851 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] _handle_message: 0x5647873d5d40 2020-02-03 15:09:35.681898 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] do_op osd_op(osd.9.59923:2779948 2.b 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 [copy-get max 8388608] snapc 0=[] ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected e59939) v8 may_read -> write-ordered flags ondisk+read+rwordered+ignore_cache+ignore_overlay+map_snap_clone+known_if_redirected 2020-02-03 15:09:35.681937 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] find_object_context 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 @7e3 snapset c07=[c07,c04,c01,bfe,bf9,bf6,bf3,bf0,bed,bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab,ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d,957,951,94b,945,93f,939,933,92d,927,921,91b,915,90f,909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f,849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9,7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff,9]:{11=[9],7e3=[7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff],849=[849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9],909=[909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f],915=[915,90f],957=[957,951,94b,945,93f,939,933,92d,927,921,91b],ba5=[ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d],bea=[bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab],bf9=[bf9,bf6,bf3,bf0,bed],c07=[c07,c04,c01,bfe]} map_snapid_to_clone=true 2020-02-03 15:09:35.682035 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] find_object_context 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 @7e3 snapset c07=[c07,c04,c01,bfe,bf9,bf6,bf3,bf0,bed,bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab,ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d,957,951,94b,945,93f,939,933,92d,927,921,91b,915,90f,909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f,849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9,7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff,9]:{11=[9],7e3=[7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff],849=[849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9],909=[909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f],915=[915,90f],957=[957,951,94b,945,93f,939,933,92d,927,921,91b],ba5=[ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d],bea=[bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab],bf9=[bf9,bf6,bf3,bf0,bed],c07=[c07,c04,c01,bfe]} maps to 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 2020-02-03 15:09:35.682128 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] get_object_context: found obc in cache: 0x564785fb3880 2020-02-03 15:09:35.682144 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] get_object_context: 0x564785fb3880 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 rwstate(none n=0 w=0) oi: 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3(58682'4215046 osd.8.0:393237 dirty|data_digest|omap_digest s 4194304 uv 7633387 dd 41a45f29 od ffffffff alloc_hint [0 0 0]) exists: 1 ssc: 0x564783df4160 snapset: c07=[c07,c04,c01,bfe,bf9,bf6,bf3,bf0,bed,bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab,ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d,957,951,94b,945,93f,939,933,92d,927,921,91b,915,90f,909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f,849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9,7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff,9]:{11=[9],7e3=[7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff],849=[849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9],909=[909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f],915=[915,90f],957=[957,951,94b,945,93f,939,933,92d,927,921,91b],ba5=[ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d],bea=[bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab],bf9=[bf9,bf6,bf3,bf0,bed],c07=[c07,c04,c01,bfe]} 2020-02-03 15:09:35.682245 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] find_object_context 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 @7e3 snapset c07=[c07,c04,c01,bfe,bf9,bf6,bf3,bf0,bed,bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab,ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d,957,951,94b,945,93f,939,933,92d,927,921,91b,915,90f,909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f,849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9,7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff,9]:{11=[9],7e3=[7e3,7dd,7d7,7d1,7cb,7c5,7bf,7b9,7b3,7ad,7a7,7a1,79b,795,78f,789,783,77d,777,771,76b,765,75f,759,753,74d,747,741,73b,735,72f,729,723,71d,717,711,70b,705,6ff],849=[849,843,83d,837,831,82b,825,81f,819,813,80d,807,801,7fb,7f5,7ef,7e9],909=[909,903,8fd,8f7,8f1,8eb,8e5,8df,8d9,8d3,8cd,8c7,8c1,8bb,8b5,8af,8a9,8a3,89d,897,891,88b,885,87f,879,873,86d,867,861,85b,855,84f],915=[915,90f],957=[957,951,94b,945,93f,939,933,92d,927,921,91b],ba5=[ba5,b9f,b99,b93,b8d,b87,b81,b7b,b75,b6f,b69,b63,b5d,b57,b51,b4b,b45,b3f,b39,b33,b2d,b27,b21,b1b,b15,b0d,b07,b01,afb,af5,aef,ae9,ae3,add,ad7,ad1,acb,ac5,abf,ab9,ab3,aad,aa7,aa1,a9b,a95,a8f,a89,a83,a7d,a77,a71,a6b,a65,a5f,a59,a53,a4d,a47,a41,a3b,a35,a2f,a29,a23,a1d,a17,a11,a0b,a05,9ff,9f9,9f3,9ed,9e7,9e1,9db,9d5,9cf,9c9,9c3,9bd,9b7,9b1,9ab,9a5,99f,999,993,98d,987,981,97b,975,96f,969,963,95d],bea=[bea,be7,be4,be1,bde,bdb,bd6,bd3,bd0,bcd,bca,bc7,bc4,bc1,bbe,bbb,bb8,bb5,bab],bf9=[bf9,bf6,bf3,bf0,bed],c07=[c07,c04,c01,bfe]} 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 HIT 2020-02-03 15:09:35.682356 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] execute_ctx 0x5647843a3000 2020-02-03 15:09:35.682385 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] execute_ctx 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 [copy-get max 8388608] ov 58682'4215046 2020-02-03 15:09:35.682404 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] taking ondisk_read_lock 2020-02-03 15:09:35.682420 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] do_osd_op 2:d1f1b468:::rbd_data.9e87fa74b0dc51.000000000001d1a9:7e3 [copy-get max 8388608] 2020-02-03 15:09:35.682437 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] do_osd_op copy-get max 8388608 2020-02-03 15:09:35.714076 7f51e0acb700 10 osd.8 pg_epoch: 59939 pg[2.b( v 59908'4218227 (58731'4216723,59908'4218227] local-lis/les=59932/59934 n=4693 ec=1/1 lis/c 59932/59932 les/c/f 59934/59934/38237 59932/59932/59932) [8,0] r=0 lpr=59932 crt=59908'4218227 lcod 0'0 mlcod 0'0 active+clean] dropping ondisk_read_lock 2020-02-03 15:09:35.714196 7f51e0acb700 10 osd.8 59939 dequeue_op 0x5647873d5d40 finish

1 1

Re: recovery_unfound
by Chad William Seys 05 Feb '20

05 Feb '20

Hi Jake and all, We're having what looks to be the exact same problem. In our case it happened when I was "draining" an OSD for removal. (ceph crush remove...) Adding the OSD back doesn't help workaround the bug. Everything is either triply replicated or EC k3m2, either of which should stand loss of two hosts (much less one OSD). We're running 13.2.6 . I tried various OSD restarts, deep-scrubs, with no change. I'm leaving things alone hoping that croit.io will update their package to 13.2.8 soonish. Maybe that will help kick it in the pants. Chad.

2 2

Mixed FileStore and BlueStore OSDs in Nautilus and beyond
by Thomas Byrne - UKRI STFC 05 Feb '20

05 Feb '20

Hi, Our large Luminous cluster still has around 2k FileStore OSDs (35% of OSDs). We haven't had any particular need to move these over to BlueStore yet, as the performance is fine for our use case. Obviously, it would be easiest if we could let the FileStore OSDs stay in the cluster until the hardware generation is removed from the cluster in a few years. While reading around for our upgrade to Nautilus, and I stumbled across a thread [1] where Sage mentioned they had some issues getting FileStore OSDs and PG merging to play nicely. While I'm not planning on trying to merge PGs on this cluster any time soon, it raised a few more general questions about the OSD back ends in general. - How much extra risk are we exposed to by running both types of OSDs in the same pools. So far we have not run into any issues due to this, but I get the feeling it is not a well-tested configuration compared to BlueStore only clusters. - Are FileStore OSDs going to continue to be supported indefinitely? I know they have their uses, but if there is an aspiration to reduce support or stop supporting them entirely in the coming releases then we may need to plan around that. Any thoughts, opinions, or pointers to previous discussions about this would be great. Thanks, Tom [1] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/GRQOYECPFTO…

1 0

Re: [Ceph-community] HEALTH_WARN - daemons have recently crashed
by Sage Weil 05 Feb '20

05 Feb '20

[Moving this to ceph-users(a)ceph.io] This looks like https://tracker.ceph.com/issues/43365, which *looks* like it is an issue with the standard libraries in ubuntu 18.04. One user said: "After upgrading our monitor Ubuntu 18.04 packages (apt-get upgrade) with the 5.3.0-26-generic kernel, it seems that the crashes have been fixed (they run stable now for 8 days)." Can you give that a try? Also, On Wed, 5 Feb 2020, Micha Ballmann wrote: > Hi, > > i have a Proxmox Ceph Cluster VE 6.1-5. > > # ceph -v > ceph version 14.2.6 (ba51347bdbe28c7c0e2e9172fa2983111137bb60) nautilus > (stable) > > My problem since version 14.2.6 im receiving nearly everyday the following > messages: > > # cephs -s > > ... > > health: HEALTH_WARN > 2 daemons have recently crashed > > ... > > I archive the messages: > > # ceph crash archive-all > > But one or two days laters the same problem occurs. Trying to found out what > is the problem: > > For example: > > # ceph crash info <ID> note that this ID is a hash of the stack trace and is meant to be a unique signature for this crash/bug but contains no identifying information. Yours is probably one of 6a617f9d477ab8df2d068af0768ff741c68adabcc5c1ecb5dd3e9872d613c943 dacbff55030f3d0837e58d8f4961441b6902d5750b0e1579682df5650c33d44d Please consider turning on telemetry so we get this crash information automatically: https://docs.ceph.com/docs/master/mgr/telemetry/ Thanks! sage > > Node4 > { > "os_version_id": "10", > "utsname_machine": "x86_64", > "entity_name": "mon.promo4", > "backtrace": [ > "(()+0x12730) [0x7f30ca142730]", > "(gsignal()+0x10b) [0x7f30c9c257bb]", > "(abort()+0x121) [0x7f30c9c10535]", > "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a3) > [0x7f30cb27be79]", > "(()+0x282000) [0x7f30cb27c000]", > "(Paxos::store_state(MMonPaxos*)+0xaa8) [0x5602540626f8]", > "(Paxos::handle_commit(boost::intrusive_ptr<MonOpRequest>)+0x2ea) > [0x560254062a5a]", > "(Paxos::dispatch(boost::intrusive_ptr<MonOpRequest>)+0x223) > [0x560254068213]", > "(Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x131c) > [0x560253f9db1c]", > "(Monitor::_ms_dispatch(Message*)+0x4aa) [0x560253f9e10a]", > "(Monitor::ms_dispatch(Message*)+0x26) [0x560253fcda36]", > "(Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x26) > [0x560253fc9f66]", > "(DispatchQueue::entry()+0x1a49) [0x7f30cb4b4e69]", > "(DispatchQueue::DispatchThread::entry()+0xd) [0x7f30cb5629ed]", > "(()+0x7fa3) [0x7f30ca137fa3]", > "(clone()+0x3f) [0x7f30c9ce74cf]" > ], > "process_name": "ceph-mon", > "assert_line": 485, > "archived": "2020-01-21 07:02:49.036123", > "assert_file": > "/mnt/npool/tlamprecht/pve-ceph/ceph-14.2.6/src/common/ceph_time.h", > "utsname_sysname": "Linux", > "os_version": "10 (buster)", > "os_id": "10", > "assert_msg": > "/mnt/npool/tlamprecht/pve-ceph/ceph-14.2.6/src/common/ceph_time.h: In > function 'ceph::time_detail::timespan > ceph::to_timespan(ceph::time_detail::signedspan)' thread 7f30c11fe700 time > 2020-01-21 > 03:43:48.848411\n/mnt/npool/tlamprecht/pve-ceph/ceph-14.2.6/src/common/ceph_time.h: > 485: FAILED ceph_assert(z >= signedspan::zero())\n", > "assert_func": "ceph::time_detail::timespan > ceph::to_timespan(ceph::time_detail::signedspan)", > "ceph_version": "14.2.6", > "os_name": "Debian GNU/Linux 10 (buster)", > "timestamp": "2020-01-21 02:43:48.891122Z", > "assert_thread_name": "ms_dispatch", > "utsname_release": "5.3.13-1-pve", > "utsname_hostname": "promo4", > "crash_id": > "2020-01-21_02:43:48.891122Z_0aade13c-463f-43fe-9b05-76ca71f6bc1b", > "assert_condition": "z >= signedspan::zero()", > "utsname_version": "#1 SMP PVE 5.3.13-1 (Thu, 05 Dec 2019 07:18:14 +0100)" > } > > Node2 > { > "os_version_id": "10", > "utsname_machine": "x86_64", > "entity_name": "mon.promo2", > "backtrace": [ > "(()+0x12730) [0x7f74f6c3f730]", > "(gsignal()+0x10b) [0x7f74f67227bb]", > "(abort()+0x121) [0x7f74f670d535]", > "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a3) > [0x7f74f7d78e79]", > "(()+0x282000) [0x7f74f7d79000]", > "(Paxos::store_state(MMonPaxos*)+0xaa8) [0x55b9540ae6f8]", > "(Paxos::handle_commit(boost::intrusive_ptr<MonOpRequest>)+0x2ea) > [0x55b9540aea5a]", > "(Paxos::dispatch(boost::intrusive_ptr<MonOpRequest>)+0x223) > [0x55b9540b4213]", > "(Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x131c) > [0x55b953fe9b1c]", > "(Monitor::_ms_dispatch(Message*)+0x4aa) [0x55b953fea10a]", > "(Monitor::ms_dispatch(Message*)+0x26) [0x55b954019a36]", > "(Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x26) > [0x55b954015f66]", > "(DispatchQueue::entry()+0x1a49) [0x7f74f7fb1e69]", > "(DispatchQueue::DispatchThread::entry()+0xd) [0x7f74f805f9ed]", > "(()+0x7fa3) [0x7f74f6c34fa3]", > "(clone()+0x3f) [0x7f74f67e44cf]" > ], > "process_name": "ceph-mon", > "assert_line": 485, > "archived": "2020-01-21 07:02:49.041386", > "assert_file": > "/mnt/npool/tlamprecht/pve-ceph/ceph-14.2.6/src/common/ceph_time.h", > "utsname_sysname": "Linux", > "os_version": "10 (buster)", > "os_id": "10", > "assert_msg": > "/mnt/npool/tlamprecht/pve-ceph/ceph-14.2.6/src/common/ceph_time.h: In > function 'ceph::time_detail::timespan > ceph::to_timespan(ceph::time_detail::signedspan)' thread 7f74edcfb700 time > 2020-01-20 > 22:32:56.933800\n/mnt/npool/tlamprecht/pve-ceph/ceph-14.2.6/src/common/ceph_time.h: > 485: FAILED ceph_assert(z >= signedspan::zero())\n", > "assert_func": "ceph::time_detail::timespan > ceph::to_timespan(ceph::time_detail::signedspan)", > "ceph_version": "14.2.6", > "os_name": "Debian GNU/Linux 10 (buster)", > "timestamp": "2020-01-20 21:32:56.947402Z", > "assert_thread_name": "ms_dispatch", > "utsname_release": "5.3.13-1-pve", > "utsname_hostname": "promo2", > "crash_id": > "2020-01-20_21:32:56.947402Z_3ae7220c-23c9-478a-a22d-626c2fa34414", > "assert_condition": "z >= signedspan::zero()", > "utsname_version": "#1 SMP PVE 5.3.13-1 (Thu, 05 Dec 2019 07:18:14 +0100)" > } > > Is there a problem with my NTP? Im syncing my time with CHRONY to my local NTP > Server. > > It would be nice if you can help. > > I have to say my ceph cluster is clean and works without any issue. All OSDs > are up and after ceph crash archive-all; cephs -s says; HEALTH_OK > > Regards > > Micha > >

1 0

Re: OSDs crashing
by Raymond Clotfelter 05 Feb '20

05 Feb '20

I have found that if I set norecovery then I can get almost all OSDs to come up and stay up, but the moment I unset norecovery so that the cluster can heal itself, tons of OSDs go offline again. The OSD host servers have plenty of available RAM, they are not maxing on CPU or I/O near as I can tell. The OSDs just keep crashing with error logs similar what I posted before. Any help would be appreciated. -- ray On Tue, Feb 4, 2020 at 6:22 AM Raymond Clotfelter <ray(a)ksu.edu<mailto:ray@ksu.edu>> wrote: I have 30 or so OSDs on a cluster with 240 that just keep crashing. Below is the last part of one of the log files showing the crash, can anyone please help me read this to figure out what is going on and how to correct it? When I start the OSDs they generally seem to work for 5-30 minutes, and then one by one they will start dropping out with logs similar to this. Thanks. -29> 2020-02-04 06:00:23.447 7fe300d41700 5 osd.168 pg_epoch: 459335 pg[6.1217s2( v 443432'5 (0'0,443432'5] local-lis/les=459328/459329 n=1 ec=260267/6574 lis/c 459331/428950 les/c/f 459332/440468/290442 459333/459334/459294) [2147483647,107,168,2147483647,102]/[81,107,168,89,102]p81(0) r=2 lpr=459334 pi=[428950,459334)/82 crt=443432'5 lcod 0'0 remapped NOTIFY mbc={}] exit Started/Stray 1.017145 6 0.000323 -28> 2020-02-04 06:00:23.447 7fe300d41700 5 osd.168 pg_epoch: 459335 pg[6.1217s2( v 443432'5 (0'0,443432'5] local-lis/les=459328/459329 n=1 ec=260267/6574 lis/c 459331/428950 les/c/f 459332/440468/290442 459333/459334/459294) [2147483647,107,168,2147483647,102]/[81,107,168,89,102]p81(0) r=2 lpr=459334 pi=[428950,459334)/82 crt=443432'5 lcod 0'0 remapped NOTIFY mbc={}] enter Started/ReplicaActive -27> 2020-02-04 06:00:23.447 7fe300d41700 5 osd.168 pg_epoch: 459335 pg[6.1217s2( v 443432'5 (0'0,443432'5] local-lis/les=459328/459329 n=1 ec=260267/6574 lis/c 459331/428950 les/c/f 459332/440468/290442 459333/459334/459294) [2147483647,107,168,2147483647,102]/[81,107,168,89,102]p81(0) r=2 lpr=459334 pi=[428950,459334)/82 crt=443432'5 lcod 0'0 remapped NOTIFY mbc={}] enter Started/ReplicaActive/RepNotRecovering -26> 2020-02-04 06:00:23.455 7fe309d53700 3 osd.168 459335 handle_osd_map epochs [459335,459335], i have 459335, src has [403399,459335] -25> 2020-02-04 06:00:23.455 7fe309d53700 3 osd.168 459335 handle_osd_map epochs [459335,459335], i have 459335, src has [403399,459335] -24> 2020-02-04 06:00:23.455 7fe309d53700 3 osd.168 459335 handle_osd_map epochs [459335,459335], i have 459335, src has [403399,459335] -23> 2020-02-04 06:00:23.459 7fe309d53700 3 osd.168 459335 handle_osd_map epochs [459335,459335], i have 459335, src has [403399,459335] -22> 2020-02-04 06:00:23.459 7fe309d53700 3 osd.168 459335 handle_osd_map epochs [459335,459335], i have 459335, src has [403399,459335] -21> 2020-02-04 06:00:23.459 7fe309d53700 3 osd.168 459335 handle_osd_map epochs [459335,459335], i have 459335, src has [403399,459335] -20> 2020-02-04 06:00:23.459 7fe309d53700 3 osd.168 459335 handle_osd_map epochs [459335,459335], i have 459335, src has [403399,459335] -19> 2020-02-04 06:00:23.463 7fe309d53700 3 osd.168 459335 handle_osd_map epochs [459335,459335], i have 459335, src has [403399,459335] -18> 2020-02-04 06:00:23.463 7fe309d53700 3 osd.168 459335 handle_osd_map epochs [459335,459335], i have 459335, src has [403399,459335] -17> 2020-02-04 06:00:23.471 7fe309d53700 3 osd.168 459335 handle_osd_map epochs [459335,459335], i have 459335, src has [403399,459335] -16> 2020-02-04 06:00:23.471 7fe309d53700 3 osd.168 459335 handle_osd_map epochs [459335,459335], i have 459335, src has [403399,459335] -15> 2020-02-04 06:00:23.471 7fe300d41700 5 osd.168 pg_epoch: 459335 pg[6.1217s2( v 443432'5 (0'0,443432'5] local-lis/les=459334/459335 n=1 ec=260267/6574 lis/c 459334/428950 les/c/f 459335/440468/290442 459333/459334/459294) [2147483647,107,168,2147483647,102]/[81,107,168,89,102]p81(0) r=2 lpr=459334 pi=[428950,459334)/82 luod=0'0 crt=443432'5 lcod 0'0 active+remapped mbc={}] exit Started/ReplicaActive/RepNotRecovering 0.021923 2 0.000098 -14> 2020-02-04 06:00:23.471 7fe300d41700 5 osd.168 pg_epoch: 459335 pg[6.1217s2( v 443432'5 (0'0,443432'5] local-lis/les=459334/459335 n=1 ec=260267/6574 lis/c 459334/428950 les/c/f 459335/440468/290442 459333/459334/459294) [2147483647,107,168,2147483647,102]/[81,107,168,89,102]p81(0) r=2 lpr=459334 pi=[428950,459334)/82 luod=0'0 crt=443432'5 lcod 0'0 active+remapped mbc={}] enter Started/ReplicaActive/RepWaitRecoveryReserved -13> 2020-02-04 06:00:23.471 7fe300d41700 5 osd.168 pg_epoch: 459335 pg[6.1217s2( v 443432'5 (0'0,443432'5] local-lis/les=459334/459335 n=1 ec=260267/6574 lis/c 459334/428950 les/c/f 459335/440468/290442 459333/459334/459294) [2147483647,107,168,2147483647,102]/[81,107,168,89,102]p81(0) r=2 lpr=459334 pi=[428950,459334)/82 luod=0'0 crt=443432'5 lcod 0'0 active+remapped mbc={}] exit Started/ReplicaActive/RepWaitRecoveryReserved 0.000137 1 0.000080 -12> 2020-02-04 06:00:23.471 7fe300d41700 5 osd.168 pg_epoch: 459335 pg[6.1217s2( v 443432'5 (0'0,443432'5] local-lis/les=459334/459335 n=1 ec=260267/6574 lis/c 459334/428950 les/c/f 459335/440468/290442 459333/459334/459294) [2147483647,107,168,2147483647,102]/[81,107,168,89,102]p81(0) r=2 lpr=459334 pi=[428950,459334)/82 luod=0'0 crt=443432'5 lcod 0'0 active+remapped mbc={}] enter Started/ReplicaActive/RepRecovering -11> 2020-02-04 06:00:23.471 7fe309d53700 3 osd.168 459335 handle_osd_map epochs [459335,459335], i have 459335, src has [403399,459335] -10> 2020-02-04 06:00:23.475 7fe309d53700 3 osd.168 459335 handle_osd_map epochs [459335,459335], i have 459335, src has [403399,459335] -9> 2020-02-04 06:00:23.483 7fe300540700 5 osd.168 pg_epoch: 459335 pg[6.1961s0( v 436281'1 (0'0,436281'1] local-lis/les=459334/459335 n=0 ec=260267/6574 lis/c 459334/406589 les/c/f 459335/436403/290442 459333/459334/459334) [168,2147483647,2147483647,196,151]/[168,116,60,196,151]p168(0) r=0 lpr=459334 pi=[436277,459334)/33 crt=436281'1 lcod 0'0 mlcod 0'0 active+degraded+remapped m=30 mbc={0={(0+0)=30},1={(0+2)=30},2={(0+3)=30},3={(1+1)=30},4={(0+0)=30}}] exit Started/Primary/Active/Activating 0.034400 28 0.002950 -8> 2020-02-04 06:00:23.483 7fe300540700 5 osd.168 pg_epoch: 459335 pg[6.1961s0( v 436281'1 (0'0,436281'1] local-lis/les=459334/459335 n=0 ec=260267/6574 lis/c 459334/406589 les/c/f 459335/436403/290442 459333/459334/459334) [168,2147483647,2147483647,196,151]/[168,116,60,196,151]p168(0) r=0 lpr=459334 pi=[436277,459334)/33 crt=436281'1 lcod 0'0 mlcod 0'0 active+degraded+remapped m=30 mbc={0={(0+0)=30},1={(0+2)=30},2={(0+3)=30},3={(1+1)=30},4={(0+0)=30}}] enter Started/Primary/Active/WaitLocalRecoveryReserved -7> 2020-02-04 06:00:23.483 7fe300540700 5 osd.168 pg_epoch: 459335 pg[6.1961s0( v 436281'1 (0'0,436281'1] local-lis/les=459334/459335 n=0 ec=260267/6574 lis/c 459334/406589 les/c/f 459335/436403/290442 459333/459334/459334) [168,2147483647,2147483647,196,151]/[168,116,60,196,151]p168(0) r=0 lpr=459334 pi=[436277,459334)/33 crt=436281'1 lcod 0'0 mlcod 0'0 active+recovery_wait+degraded+remapped m=30 mbc={0={(0+0)=30},1={(0+2)=30},2={(0+3)=30},3={(1+1)=30},4={(0+0)=30}}] exit Started/Primary/Active/WaitLocalRecoveryReserved 0.000213 1 0.000221 -6> 2020-02-04 06:00:23.483 7fe300540700 5 osd.168 pg_epoch: 459335 pg[6.1961s0( v 436281'1 (0'0,436281'1] local-lis/les=459334/459335 n=0 ec=260267/6574 lis/c 459334/406589 les/c/f 459335/436403/290442 459333/459334/459334) [168,2147483647,2147483647,196,151]/[168,116,60,196,151]p168(0) r=0 lpr=459334 pi=[436277,459334)/33 crt=436281'1 lcod 0'0 mlcod 0'0 active+recovery_wait+degraded+remapped m=30 mbc={0={(0+0)=30},1={(0+2)=30},2={(0+3)=30},3={(1+1)=30},4={(0+0)=30}}] enter Started/Primary/Active/WaitRemoteRecoveryReserved -5> 2020-02-04 06:00:23.483 7fe300540700 5 osd.168 pg_epoch: 459335 pg[6.1961s0( v 436281'1 (0'0,436281'1] local-lis/les=459334/459335 n=0 ec=260267/6574 lis/c 459334/406589 les/c/f 459335/436403/290442 459333/459334/459334) [168,2147483647,2147483647,196,151]/[168,116,60,196,151]p168(0) r=0 lpr=459334 pi=[436277,459334)/33 crt=436281'1 lcod 0'0 mlcod 0'0 active+recovery_wait+degraded+remapped m=30 mbc={0={(0+0)=30},1={(0+2)=30},2={(0+3)=30},3={(1+1)=30},4={(0+0)=30}}] exit Started/Primary/Active/WaitRemoteRecoveryReserved 0.002796 5 0.000231 -4> 2020-02-04 06:00:23.483 7fe300540700 5 osd.168 pg_epoch: 459335 pg[6.1961s0( v 436281'1 (0'0,436281'1] local-lis/les=459334/459335 n=0 ec=260267/6574 lis/c 459334/406589 les/c/f 459335/436403/290442 459333/459334/459334) [168,2147483647,2147483647,196,151]/[168,116,60,196,151]p168(0) r=0 lpr=459334 pi=[436277,459334)/33 crt=436281'1 lcod 0'0 mlcod 0'0 active+recovery_wait+degraded+remapped m=30 mbc={0={(0+0)=30},1={(0+2)=30},2={(0+3)=30},3={(1+1)=30},4={(0+0)=30}}] enter Started/Primary/Active/Recovering -3> 2020-02-04 06:00:23.491 7fe309d53700 3 osd.168 459335 handle_osd_map epochs [459335,459335], i have 459335, src has [403399,459335] -2> 2020-02-04 06:00:23.519 7fe309d53700 3 osd.168 459335 handle_osd_map epochs [459335,459335], i have 459335, src has [403399,459335] -1> 2020-02-04 06:00:23.779 7fe300540700 -1 /build/ceph-14.2.7/src/osd/ECBackend.cc: In function 'void ECBackend::do_read_op(ECBackend::ReadOp&)' thread 7fe300540700 time 2020-02-04 06:00:23.774430 /build/ceph-14.2.7/src/osd/ECBackend.cc: 1742: FAILED ceph_assert(!need_attrs) ceph version 14.2.7 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8) nautilus (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x5579b9ecce4c] 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x5579b9ecd027] 3: (ECBackend::do_read_op(ECBackend::ReadOp&)+0xf76) [0x5579ba295106] 4: (ECBackend::send_all_remaining_reads(hobject_t const&, ECBackend::ReadOp&)+0x4d1) [0x5579ba2a6cf1] 5: (ECBackend::handle_sub_read_reply(pg_shard_t, ECSubReadReply&, RecoveryMessages*, ZTracer::Trace const&)+0xcf6) [0x5579ba2a7f26] 6: (ECBackend::_handle_message(boost::intrusive_ptr<OpRequest>)+0xbc) [0x5579ba2a8a8c] 7: (PGBackend::handle_message(boost::intrusive_ptr<OpRequest>)+0x97) [0x5579ba17f757] 8: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x705) [0x5579ba12dee5] 9: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x1bf) [0x5579b9f4ff2f] 10: (PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x62) [0x5579ba206e82] 11: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xbf5) [0x5579b9f6ea05] 12: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x4ac) [0x5579ba5714cc] 13: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x5579ba574690] 14: (()+0x7fa3) [0x7fe31ff5cfa3] 15: (clone()+0x3f) [0x7fe31fb0c4cf] 0> 2020-02-04 06:00:23.787 7fe300540700 -1 *** Caught signal (Aborted) ** in thread 7fe300540700 thread_name:tp_osd_tp ceph version 14.2.7 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8) nautilus (stable) 1: (()+0x12730) [0x7fe31ff67730] 2: (gsignal()+0x10b) [0x7fe31fa4a7bb] 3: (abort()+0x121) [0x7fe31fa35535] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a3) [0x5579b9ecce9d] 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x5579b9ecd027] 6: (ECBackend::do_read_op(ECBackend::ReadOp&)+0xf76) [0x5579ba295106] 7: (ECBackend::send_all_remaining_reads(hobject_t const&, ECBackend::ReadOp&)+0x4d1) [0x5579ba2a6cf1] 8: (ECBackend::handle_sub_read_reply(pg_shard_t, ECSubReadReply&, RecoveryMessages*, ZTracer::Trace const&)+0xcf6) [0x5579ba2a7f26] 9: (ECBackend::_handle_message(boost::intrusive_ptr<OpRequest>)+0xbc) [0x5579ba2a8a8c] 10: (PGBackend::handle_message(boost::intrusive_ptr<OpRequest>)+0x97) [0x5579ba17f757] 11: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x705) [0x5579ba12dee5] 12: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x1bf) [0x5579b9f4ff2f] 13: (PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x62) [0x5579ba206e82] 14: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xbf5) [0x5579b9f6ea05] 15: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x4ac) [0x5579ba5714cc] 16: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x5579ba574690] 17: (()+0x7fa3) [0x7fe31ff5cfa3] 18: (clone()+0x3f) [0x7fe31fb0c4cf] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 0 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 1 reserver 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 rgw_sync 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 kinetic 1/ 5 fuse 1/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace 1/ 5 prioritycache -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.168.log --- end dump of recent events --- -- ray _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave(a)ceph.io<mailto:ceph-users-leave@ceph.io>

1 0

Re: Cephalocon Seoul is canceled
by Romit Misra 05 Feb '20

05 Feb '20

Hi Sage, Just read the news on Cancellation of Cephlacon 2020, although the site is still status quo/ Double checking that we can proceed with the cancellation of logistics for South Korea Thanks Romit On Tue, Feb 4, 2020 at 11:02 PM <ceph-users-request(a)ceph.io> wrote: > Send ceph-users mailing list submissions to > ceph-users(a)ceph.io > > To subscribe or unsubscribe via email, send a message with subject or > body 'help' to > ceph-users-request(a)ceph.io > > You can reach the person managing the list at > ceph-users-owner(a)ceph.io > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of ceph-users digest..." > > Today's Topics: > > 1. Re: More OMAP Issues (Paul Emmerich) > 2. Re: More OMAP Issues (DHilsbos(a)performair.com) > 3. Re: Bluestore cache parameter precedence (Igor Fedotov) > 4. Re: Understanding Bluestore performance characteristics > (vitalif(a)yourcmc.ru) > 5. Cephalocon Seoul is canceled (Sage Weil) > 6. Re: Bluestore cache parameter precedence (Boris Epstein) > 7. Bucket rename with (EDH - Manuel Rios) > > > ---------------------------------------------------------------------- > > Date: Tue, 4 Feb 2020 17:51:40 +0100 > From: Paul Emmerich <paul.emmerich(a)croit.io> > Subject: [ceph-users] Re: More OMAP Issues > To: DHilsbos(a)performair.com > Cc: ceph-users <ceph-users(a)ceph.io> > Message-ID: > < > CAD9yTbEp1BrAagzWzkaAQ-aCq-4ghyEwVJSDdyJzL-So52whBA(a)mail.gmail.com> > Content-Type: text/plain; charset="UTF-8" > > Are you running a multi-site setup? > In this case it's best to set the default shard size to large enough > number *before* enabling multi-site. > > If you didn't do this: well... I think the only way is still to > completely re-sync the second site... > > > Paul > > -- > Paul Emmerich > > Looking for help with your Ceph cluster? Contact us at https://croit.io > > croit GmbH > Freseniusstr. 31h > 81247 München > www.croit.io > Tel: +49 89 1896585 90 > > On Tue, Feb 4, 2020 at 5:23 PM <DHilsbos(a)performair.com> wrote: > > > > All; > > > > We're backing to having large OMAP object warnings regarding our RGW > index pool. > > > > This cluster is now in production, so I can simply dump the buckets / > pools and hope everything works out. > > > > I did some additional research on this issue, and it looks like I need > to (re)shard the bucket (index?). I found information that suggests that, > for older versions of Ceph, buckets couldn't be sharded after creation[1]. > Other information suggests the Nautilus (which we are running), can > re-shard dynamically, but not when multi-site replication is configured[2]. > > > > This suggests that a "manual" resharding of a Nautilus cluster should be > possible, but I can't find the commands to do it. Has anyone done this? > Does anyone have the commands to do it? I can schedule down time for the > cluster, and take the RADOSGW instance(s), and dependent user services > offline. > > > > [1]: https://ceph.io/geen-categorie/radosgw-big-index/ > > [2]: https://docs.ceph.com/docs/master/radosgw/dynamicresharding/ > > > > Thank you, > > > > Dominic L. Hilsbos, MBA > > Director - Information Technology > > Perform Air International Inc. > > DHilsbos(a)PerformAir.com > > www.PerformAir.com > > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > ------------------------------ > > Date: Tue, 4 Feb 2020 17:04:24 +0000 > From: <DHilsbos(a)performair.com> > Subject: [ceph-users] Re: More OMAP Issues > To: <ceph-users(a)ceph.io> > Cc: <paul.emmerich(a)croit.io> > Message-ID: > <0670B960225633449A24709C291A525243605D57(a)COM01.performair.local> > Content-Type: text/plain; charset="utf-8" > > Paul; > > Yes, we are running a multi-site setup. > > Re-sync would be acceptable at this point, as we only have 4 TiB in use > right now. > > Tearing down and reconfiguring the second site would also be acceptable, > except that I've never been able to cleanly remove a zone from a zone > group. The only way I've found to remove a zone completely is to tear down > the entire RADOSGW configuration (delete .rgw.root pool from both clusters). > > Thank you, > > Dominic L. Hilsbos, MBA > Director – Information Technology > Perform Air International Inc. > DHilsbos(a)PerformAir.com > www.PerformAir.com > > > > -----Original Message----- > From: Paul Emmerich [mailto:paul.emmerich@croit.io] > Sent: Tuesday, February 04, 2020 9:52 AM > To: Dominic Hilsbos > Cc: ceph-users > Subject: Re: [ceph-users] More OMAP Issues > > Are you running a multi-site setup? > In this case it's best to set the default shard size to large enough > number *before* enabling multi-site. > > If you didn't do this: well... I think the only way is still to > completely re-sync the second site... > > > Paul > > -- > Paul Emmerich > > Looking for help with your Ceph cluster? Contact us at https://croit.io > > croit GmbH > Freseniusstr. 31h > 81247 München > www.croit.io > Tel: +49 89 1896585 90 > > On Tue, Feb 4, 2020 at 5:23 PM <DHilsbos(a)performair.com> wrote: > > > > All; > > > > We're backing to having large OMAP object warnings regarding our RGW > index pool. > > > > This cluster is now in production, so I can simply dump the buckets / > pools and hope everything works out. > > > > I did some additional research on this issue, and it looks like I need > to (re)shard the bucket (index?). I found information that suggests that, > for older versions of Ceph, buckets couldn't be sharded after creation[1]. > Other information suggests the Nautilus (which we are running), can > re-shard dynamically, but not when multi-site replication is configured[2]. > > > > This suggests that a "manual" resharding of a Nautilus cluster should be > possible, but I can't find the commands to do it. Has anyone done this? > Does anyone have the commands to do it? I can schedule down time for the > cluster, and take the RADOSGW instance(s), and dependent user services > offline. > > > > [1]: https://ceph.io/geen-categorie/radosgw-big-index/ > > [2]: https://docs.ceph.com/docs/master/radosgw/dynamicresharding/ > > > > Thank you, > > > > Dominic L. Hilsbos, MBA > > Director - Information Technology > > Perform Air International Inc. > > DHilsbos(a)PerformAir.com > > www.PerformAir.com > > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > ------------------------------ > > Date: Tue, 4 Feb 2020 20:10:20 +0300 > From: Igor Fedotov <ifedotov(a)suse.de> > Subject: [ceph-users] Re: Bluestore cache parameter precedence > To: Boris Epstein <borepstein(a)gmail.com>, ceph-users(a)ceph.io > Message-ID: <0cb36a39-7dba-01b5-5383-dc1116f459a4(a)suse.de> > Content-Type: text/plain; charset=utf-8; format=flowed > > Hi Boris, > > general settings (unless they are set to zero) override disk-specific > settings . > > I.e. bluestore_cache_size overrides both bluestore_cache_size_hdd and > bluestore_cache_size_ssd. > > Here is the code snippet in case you know C++ > > if (cct->_conf->bluestore_cache_size) { > cache_size = cct->_conf->bluestore_cache_size; > } else { > // choose global cache size based on backend type > if (_use_rotational_settings()) { > cache_size = cct->_conf->bluestore_cache_size_hdd; > } else { > cache_size = cct->_conf->bluestore_cache_size_ssd; > } > } > > Thanks, > > Igor > > On 2/4/2020 2:14 PM, Boris Epstein wrote: > > Hello list, > > > > As stated in this document: > > > > > https://docs.ceph.com/docs/master/rados/configuration/bluestore-config-ref/ > > > > there are multiple parameters defining cache limits for BlueStore. You > have > > bluestore_cache_size (presumably controlling the cache size), > > bluestore_cache_size_hdd (presumably doing the same for HDD storage only) > > and bluestore_cache_size_ssd (presumably being the equivalent for SSD). > My > > question is, does bluestore_cache_size override the disk-specific > > parameters, or do I need to set the disk-specific (or, rather, storage > type > > specific ones separately if I want to keep them to a certain value. > > > > Thanks in advance. > > > > Boris. > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > ------------------------------ > > Date: Tue, 04 Feb 2020 20:22:30 +0300 > From: vitalif(a)yourcmc.ru > Subject: [ceph-users] Re: Understanding Bluestore performance > characteristics > To: Bradley Kite <bradley.kite(a)gmail.com> > Cc: ceph-users(a)ceph.io > Message-ID: <c381a59989a4f4f6760d061e745a281a(a)yourcmc.ru> > Content-Type: text/plain; charset=US-ASCII; format=flowed > > SSD (block.db) partition contains object metadata in RocksDB so it > probably loads the metadata before modifying objects (if it's not in > cache yet). Also it sometimes performs compaction which also results in > disk reads and writes. There are other things going on that I'm not > completely aware of. There's the RBD object map... Maybe there are some > locks that come into action when you parallel writes... > > There's a config option to enable RocksDB performance counters. You can > have a look into it. > > However if you're just trying to understand why RBD isn't super fast > then I don't think these reads are the cause... > > > Hi Vitaliy > > > > Yes - I tried this and I can still see a number of reads (~110 iops, > > 440KB/sec) on the SSD, so it is significantly better, but the result > > is still puzzling - I'm trying to understand what is causing the > > reads. The problem is amplified with numjobs >= 2 but it looks like it > > is still there with just 1. > > > > Like some caching parameter is not correct, and the same blocks are > > being read over and over when doing a write? > > > > Could anyone advise on the best way for me to investigate further? > > > > I've tried strace (with -k) and 'perf record' but neither produce any > > useful stack traces to help understand what's going on. > > > > Regards > > -- > > Brad > > ------------------------------ > > Date: Tue, 4 Feb 2020 17:24:37 +0000 (UTC) > From: Sage Weil <sage(a)newdream.net> > Subject: [ceph-users] Cephalocon Seoul is canceled > To: ceph-announce(a)ceph.io, ceph-users(a)ceph.io, dev(a)ceph.io, > ceph-devel(a)vger.kernel.org > Message-ID: <alpine.DEB.2.21.2002041649050.21136(a)piezo.novalocal> > Content-Type: text/plain; charset=US-ASCII > > Hi everyone, > > We are sorry to announce that, due to the recent coronavirus outbreak, we > are canceling Cephalocon for March 3-5 in Seoul. > > More details will follow about how to best handle cancellation of hotel > reservations and so forth. Registrations will of course be > refunded--expect an email with details in the next day or two. > > We are still looking into whether it makes sense to reschedule the event > for later in the year. > > Thank you to everyone who has helped to plan this event, submitted talks, > and agreed to sponsor. It makes us sad to cancel, but the safety of > our community is of the utmost importance, and it was looking increasing > unlikely that we could make this event a success. > > Stay tuned... > > ------------------------------ > > Date: Tue, 4 Feb 2020 12:29:13 -0500 > From: Boris Epstein <borepstein(a)gmail.com> > Subject: [ceph-users] Re: Bluestore cache parameter precedence > To: Igor Fedotov <ifedotov(a)suse.de> > Cc: ceph-users(a)ceph.io > Message-ID: > <CADeF1XHrPzTq1+8S_WG=ZH=SVNAbdLaY= > FR7UaLGHn3O_yWLnw(a)mail.gmail.com> > Content-Type: text/plain; charset="UTF-8" > > Hi Igor, > > Thanks! > > I think the code needs to be corrected - the choice criteria for which > setting to use when > > cct->_conf->bluestore_cache_size == 0 > > should be as follows: > > 1) See what kind of storage you have. > > 2) Select type-appropriate storage. > > Is this code public-editable? I'll be happy to correct that. > > Regards, > > Boris. > > On Tue, Feb 4, 2020 at 12:10 PM Igor Fedotov <ifedotov(a)suse.de> wrote: > > > Hi Boris, > > > > general settings (unless they are set to zero) override disk-specific > > settings . > > > > I.e. bluestore_cache_size overrides both bluestore_cache_size_hdd and > > bluestore_cache_size_ssd. > > > > Here is the code snippet in case you know C++ > > > > if (cct->_conf->bluestore_cache_size) { > > cache_size = cct->_conf->bluestore_cache_size; > > } else { > > // choose global cache size based on backend type > > if (_use_rotational_settings()) { > > cache_size = cct->_conf->bluestore_cache_size_hdd; > > } else { > > cache_size = cct->_conf->bluestore_cache_size_ssd; > > } > > } > > > > Thanks, > > > > Igor > > > > On 2/4/2020 2:14 PM, Boris Epstein wrote: > > > Hello list, > > > > > > As stated in this document: > > > > > > > > > https://docs.ceph.com/docs/master/rados/configuration/bluestore-config-ref/ > > > > > > there are multiple parameters defining cache limits for BlueStore. You > > have > > > bluestore_cache_size (presumably controlling the cache size), > > > bluestore_cache_size_hdd (presumably doing the same for HDD storage > only) > > > and bluestore_cache_size_ssd (presumably being the equivalent for SSD). > > My > > > question is, does bluestore_cache_size override the disk-specific > > > parameters, or do I need to set the disk-specific (or, rather, storage > > type > > > specific ones separately if I want to keep them to a certain value. > > > > > > Thanks in advance. > > > > > > Boris. > > > _______________________________________________ > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > ------------------------------ > > Date: Tue, 4 Feb 2020 17:29:55 +0000 > From: EDH - Manuel Rios <mriosfer(a)easydatahost.com> > Subject: [ceph-users] Bucket rename with > To: "ceph-users(a)ceph.io" <ceph-users(a)ceph.io> > Message-ID: <HE1P195MB02521946493264331A2CE2A3B0030(a)HE1P195MB0252.EUR > P195.PROD.OUTLOOK.COM> > Content-Type: text/plain; charset="us-ascii" > > Hi > > Some Customer asked us for a normal easy problem, they want rename a > bucket. > > Checking the Nautilus documentation looks by now its not possible, but I > checked master documentation and a CLI should be accomplish this apparently. > > $ radosgw-admin bucket link --bucket=foo --bucket-new-name=bar --uid=johnny > > Will be backported to Nautilus? Or its still just for developer/master > users? > > https://docs.ceph.com/docs/master/man/8/radosgw-admin/ > > Regards > Manuel > > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s > > > ------------------------------ > > End of ceph-users Digest, Vol 85, Issue 17 > ****************************************** > -- *-----------------------------------------------------------------------------------------* *This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.***** **** *Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the organization. Any information on shares, debentures or similar instruments, recommended product pricing, valuations and the like are for information purposes only. It is not meant to be an instruction or recommendation, as the case may be, to buy or to sell securities, products, services nor an offer to buy or sell securities, products or services unless specifically stated to be so on behalf of the Flipkart group. Employees of the Flipkart group of companies are expressly required not to make defamatory statements and not to infringe or authorise any infringement of copyright or any other legal right by email communications. Any such communication is contrary to organizational policy and outside the scope of the employment of the individual concerned. The organization will not accept any liability in respect of such communication, and the employee responsible will be personally liable for any damages or other liability arising.***** **** *Our organization accepts no liability for the content of this email, or for the consequences of any actions taken on the basis of the information *provided,* unless that information is subsequently confirmed in writing. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.* _-----------------------------------------------------------------------------------------_

1 0

Performance drops and low oss performance
by quexian da 05 Feb '20

05 Feb '20

Hello, I'm a beginner on ceph. I set up some ceph clusters in google cloud. Cluster1 has three nodes and each node has three disks. Cluster2 has three nodes and each node has two disks. Cluster3 has five nodes and each node has five disks. Disk speed shown by `dd if=/dev/zero of=here bs=1G count=1 oflag=direct` is 117MB/s. The network is 10Gbps. However, I found something strange: 1. The write performance of all clusters drops dramatically after a few minutes. I created a pool named "scbench" with replicated size 1 (I know it is not safe but I want the highest write speed). The write performance (shown by rados bench -p scbench 1000 write) before and after the drop are: cluster1: 297MB/s 94.5MB/s cluster2: 304MB/s 67.4MB/s cluster3: 494MB/s 267.6MB/s It looks like the performance before the drop is nodes_num * 100MB/s, and the performance after the drop is about osds_num * 10MB/s. I have no idea why there is such a drop and why the performances before the drop are linear with nodes_num. 2. The write performance of object storage (shown by swift-bench -c 64 -s 4096000 -n 100000 -g 0 swift.conf) is much lower than that of storage cluster(shown by rados bench -p scbench 1000 write). I have set the replicated size of "default.rgw.buckets.data" and "default.rgw.buckets.index" to 1 The speed of cluster1 oss is 117MB/s (before the drop) and 26MB/s (after the drop), and the speed of cluster3 oss is 118MB/s (the drop does not happen). Is it normal that the oss write performance is worse than rados write performance? If not, how can I solve the problem? Thanks!

1 0

Migrate journal to Nvme from old SSD journal drive?
by Alex L 04 Feb '20

04 Feb '20

Hi, I finally got my Samsung PM983 [1] to use as journal for about 6 drives plus drive cache replacing a consumer SSD - Kingston SV300. But I can't for the life of me figure out how to move an existing journal to this NVME on my Nautilus cluster. # Created a new big partition on the NVME sgdisk --new=1:2048:+33GiB --change-name="1:ceph block.db" --typecode="1:30cd0809-c2b2-499c-8879-2d6b78529876" --mbrtogpt /dev/nvme0n1 partprobe sgdisk -p /dev/nvme0n1 # The below assumes there is already a partition+ fs on the nvme? ceph-bluestore-tool bluefs-bdev-migrate –dev-target /dev/nvme0n1p1 -devs-source /var/lib/ceph/osd/ceph-1/block.db - too many positional options have been specified on the command line ceph-bluestore-tool bluefs-bdev-migrate -–path /var/lib/ceph/osd/ceph-1/block.db –-dev-target /dev/nvme0n1p1 - too many positional options have been specified on the command line # Or should I create a new block device? if yes, will WAL come along ? And how do I remove the SSD journal partition (the old) ceph-bluestore-tool bluefs-bdev-new-db -–path /var/lib/ceph/osd/ceph-1/block.db –-dev-target /dev/nvme0n1p1 The documentation is not very clear on what migration does, nor has the same concept of a DEVICE (/dev/sda is a device for me) it seems. Thanks in advance, Alex ---- [1] - Performance stats: https://docs.google.com/spreadsheets/d/1LXupjEUnNdf011QNr24pkAiDBphzpz5_MwM…

1 0

All pgs peering indefinetely
by Rodrigo Severo - Fábrica 04 Feb '20

04 Feb '20

Hi, I have a rather small cephfs cluster with 3 machines right now: all of them sharing MDS, MON, MGS and OSD roles. I had to move all machines to a new physical location and, unfortunately, I had to move all of them at the same time. They are already on again but ceph won't be accessible as all pgs are in peering state and OSD keep going down and up again. Here is some info about my cluster: ------------------------------------------- # ceph -s cluster: id: e348b63c-d239-4a15-a2ce-32f29a00431c health: HEALTH_WARN 1 filesystem is degraded 1 MDSs report slow metadata IOs 2 osds down 1 host (2 osds) down Reduced data availability: 324 pgs inactive, 324 pgs peering 7 daemons have recently crashed 10 slow ops, oldest one blocked for 206 sec, mon.a2-df has slow ops services: mon: 3 daemons, quorum a2-df,a3-df,a1-df (age 47m) mgr: a2-df(active, since 82m), standbys: a3-df, a1-df mds: cephfs:1/1 {0=a2-df=up:replay} 2 up:standby osd: 6 osds: 4 up (since 5s), 6 in (since 47m) rgw: 1 daemon active (a2-df) data: pools: 7 pools, 324 pgs objects: 850.25k objects, 744 GiB usage: 2.3 TiB used, 14 TiB / 16 TiB avail pgs: 100.000% pgs not active 324 peering ------------------------------------------- ------------------------------------------- # ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 16.37366 - 16 TiB 2.3 TiB 2.3 TiB 1.1 GiB 8.1 GiB 14 TiB 13.83 1.00 - root default -10 16.37366 - 16 TiB 2.3 TiB 2.3 TiB 1.1 GiB 8.1 GiB 14 TiB 13.83 1.00 - datacenter df -3 5.45799 - 5.5 TiB 773 GiB 770 GiB 382 MiB 2.7 GiB 4.7 TiB 13.83 1.00 - host a1-df 3 hdd-slow 3.63899 1.00000 3.6 TiB 1.1 GiB 90 MiB 0 B 1 GiB 3.6 TiB 0.03 0.00 0 down osd.3 0 hdd 1.81898 1.00000 1.8 TiB 772 GiB 770 GiB 382 MiB 1.7 GiB 1.1 TiB 41.43 3.00 0 down osd.0 -5 5.45799 - 5.5 TiB 773 GiB 770 GiB 370 MiB 2.7 GiB 4.7 TiB 13.83 1.00 - host a2-df 4 hdd-slow 3.63899 1.00000 3.6 TiB 1.1 GiB 90 MiB 0 B 1 GiB 3.6 TiB 0.03 0.00 100 up osd.4 1 hdd 1.81898 1.00000 1.8 TiB 772 GiB 770 GiB 370 MiB 1.7 GiB 1.1 TiB 41.42 3.00 224 up osd.1 -7 5.45767 - 5.5 TiB 773 GiB 770 GiB 387 MiB 2.7 GiB 4.7 TiB 13.83 1.00 - host a3-df 5 hdd-slow 3.63869 1.00000 3.6 TiB 1.1 GiB 90 MiB 0 B 1 GiB 3.6 TiB 0.03 0.00 100 up osd.5 2 hdd 1.81898 1.00000 1.8 TiB 772 GiB 770 GiB 387 MiB 1.7 GiB 1.1 TiB 41.43 3.00 224 up osd.2 TOTAL 16 TiB 2.3 TiB 2.3 TiB 1.1 GiB 8.1 GiB 14 TiB 13.83 MIN/MAX VAR: 0.00/3.00 STDDEV: 21.82 ------------------------------------------- At this exact moment both OSDs from server a1-df were down but that's changing. Sometimes I have only one OSD down, but most of the times I have 2. And exactly which ones are actually down keeps changing. What should I do to get my cluster back up? Just wait? Regards, Rodrigo Severo

3 7

Bucket rename with
by EDH - Manuel Rios 04 Feb '20

04 Feb '20

Hi Some Customer asked us for a normal easy problem, they want rename a bucket. Checking the Nautilus documentation looks by now its not possible, but I checked master documentation and a CLI should be accomplish this apparently. $ radosgw-admin bucket link --bucket=foo --bucket-new-name=bar --uid=johnny Will be backported to Nautilus? Or its still just for developer/master users? https://docs.ceph.com/docs/master/man/8/radosgw-admin/ Regards Manuel

1 0

Cephalocon Seoul is canceled
by Sage Weil 04 Feb '20

04 Feb '20

Hi everyone, We are sorry to announce that, due to the recent coronavirus outbreak, we are canceling Cephalocon for March 3-5 in Seoul. More details will follow about how to best handle cancellation of hotel reservations and so forth. Registrations will of course be refunded--expect an email with details in the next day or two. We are still looking into whether it makes sense to reschedule the event for later in the year. Thank you to everyone who has helped to plan this event, submitted talks, and agreed to sponsor. It makes us sad to cancel, but the safety of our community is of the utmost importance, and it was looking increasing unlikely that we could make this event a success. Stay tuned...

1 0

More OMAP Issues
by DHilsbos＠performair.com 04 Feb '20

04 Feb '20

All; We're backing to having large OMAP object warnings regarding our RGW index pool. This cluster is now in production, so I can simply dump the buckets / pools and hope everything works out. I did some additional research on this issue, and it looks like I need to (re)shard the bucket (index?). I found information that suggests that, for older versions of Ceph, buckets couldn't be sharded after creation[1]. Other information suggests the Nautilus (which we are running), can re-shard dynamically, but not when multi-site replication is configured[2]. This suggests that a "manual" resharding of a Nautilus cluster should be possible, but I can't find the commands to do it. Has anyone done this? Does anyone have the commands to do it? I can schedule down time for the cluster, and take the RADOSGW instance(s), and dependent user services offline. [1]: https://ceph.io/geen-categorie/radosgw-big-index/ [2]: https://docs.ceph.com/docs/master/radosgw/dynamicresharding/ Thank you, Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. DHilsbos(a)PerformAir.com www.PerformAir.com

2 2

Doubt about AVAIL space on df
by German Anders 04 Feb '20

04 Feb '20

Hello Everyone, I would like to understand if this output is right: *# ceph df* GLOBAL: SIZE AVAIL RAW USED %RAW USED 85.1TiB 43.7TiB 41.4TiB 48.68 POOLS: NAME ID USED %USED MAX AVAIL OBJECTS volumes 13 13.8TiB 64.21 7.68TiB 3620495 I only have (1) pool called 'volumes' which is using 13.8TiB (we have a replica of 3) so it's actually using 41,4TiB and that would be the RAW USED, at this point is fine, but, then it said on the GLOBAL section that the AVAIL space is 43.7TiB and the %RAW USED is only 48.68%. So if I use the 7.68TiB of MAX AVAIL and the pool goes up to 100% of usage, that would not lead to the total space of the cluster, right? I mean were are those 43.7TiB of AVAIL space? I'm using Luminous 12.2.12 release. Sorry if it's a silly question or if it has been answered before. Thanks in advance, Best regards,

3 2

Re: Doubt about AVAIL space on df
by German Anders 04 Feb '20

04 Feb '20

Manuel, find the output of ceph osd df tree command: # ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS TYPE NAME -7 84.00099 - 85.1TiB 41.6TiB 43.6TiB 48.82 1.00 - root root -5 12.00000 - 13.1TiB 5.81TiB 7.29TiB 44.38 0.91 - rack rack1 -1 12.00000 - 13.1TiB 5.81TiB 7.29TiB 44.38 0.91 - node cpn01 0 nvme 1.00000 1.00000 1.09TiB 496GiB 621GiB 44.40 0.91 143 osd.0 1 nvme 1.00000 1.00000 1.09TiB 489GiB 629GiB 43.72 0.90 141 osd.1 2 nvme 1.00000 1.00000 1.09TiB 537GiB 581GiB 48.03 0.98 155 osd.2 3 nvme 1.00000 1.00000 1.09TiB 474GiB 644GiB 42.40 0.87 137 osd.3 4 nvme 1.00000 1.00000 1.09TiB 532GiB 586GiB 47.57 0.97 153 osd.4 5 nvme 1.00000 1.00000 1.09TiB 476GiB 642GiB 42.60 0.87 137 osd.5 6 nvme 1.00000 1.00000 1.09TiB 467GiB 650GiB 41.82 0.86 135 osd.6 7 nvme 1.00000 1.00000 1.09TiB 544GiB 574GiB 48.65 1.00 157 osd.7 8 nvme 1.00000 1.00000 1.09TiB 482GiB 636GiB 43.12 0.88 139 osd.8 9 nvme 1.00000 1.00000 1.09TiB 458GiB 660GiB 40.96 0.84 133 osd.9 10 nvme 1.00000 1.00000 1.09TiB 514GiB 604GiB 45.97 0.94 148 osd.10 11 nvme 1.00000 1.00000 1.09TiB 484GiB 633GiB 43.34 0.89 140 osd.11 -6 12.00099 - 13.1TiB 6.02TiB 7.08TiB 45.98 0.94 - rack rack2 -2 12.00099 - 13.1TiB 6.02TiB 7.08TiB 45.98 0.94 - node cpn02 12 nvme 1.00000 1.00000 1.09TiB 499GiB 619GiB 44.61 0.91 144 osd.12 13 nvme 1.00000 1.00000 1.09TiB 561GiB 557GiB 50.19 1.03 162 osd.13 14 nvme 1.00000 1.00000 1.09TiB 577GiB 541GiB 51.60 1.06 167 osd.14 15 nvme 1.00000 1.00000 1.09TiB 546GiB 572GiB 48.84 1.00 158 osd.15 16 nvme 1.00000 1.00000 1.09TiB 537GiB 580GiB 48.07 0.98 155 osd.16 17 nvme 1.00000 1.00000 1.09TiB 508GiB 610GiB 45.41 0.93 147 osd.17 18 nvme 1.00000 1.00000 1.09TiB 490GiB 628GiB 43.86 0.90 142 osd.18 19 nvme 1.00000 1.00000 1.09TiB 534GiB 584GiB 47.76 0.98 155 osd.19 20 nvme 1.00000 1.00000 1.09TiB 467GiB 651GiB 41.80 0.86 134 osd.20 21 nvme 1.00000 1.00000 1.09TiB 447GiB 671GiB 40.01 0.82 129 osd.21 22 nvme 1.00099 1.00000 1.09TiB 561GiB 556GiB 50.21 1.03 162 osd.22 23 nvme 1.00000 1.00000 1.09TiB 441GiB 677GiB 39.45 0.81 127 osd.23 -15 12.00000 - 13.1TiB 5.92TiB 7.18TiB 45.20 0.93 - rack rack3 -3 12.00000 - 13.1TiB 5.92TiB 7.18TiB 45.20 0.93 - node cpn03 24 nvme 1.00000 1.00000 1.09TiB 500GiB 617GiB 44.77 0.92 145 osd.24 25 nvme 1.00000 1.00000 1.09TiB 462GiB 655GiB 41.37 0.85 133 osd.25 26 nvme 1.00000 1.00000 1.09TiB 446GiB 672GiB 39.88 0.82 129 osd.26 27 nvme 1.00000 1.00000 1.09TiB 565GiB 553GiB 50.54 1.04 162 osd.27 28 nvme 1.00000 1.00000 1.09TiB 513GiB 605GiB 45.89 0.94 148 osd.28 29 nvme 1.00000 1.00000 1.09TiB 554GiB 564GiB 49.55 1.01 160 osd.29 30 nvme 1.00000 1.00000 1.09TiB 527GiB 591GiB 47.12 0.97 152 osd.30 31 nvme 1.00000 1.00000 1.09TiB 484GiB 634GiB 43.31 0.89 140 osd.31 32 nvme 1.00000 1.00000 1.09TiB 505GiB 612GiB 45.21 0.93 146 osd.32 33 nvme 1.00000 1.00000 1.09TiB 551GiB 567GiB 49.28 1.01 159 osd.33 34 nvme 1.00000 1.00000 1.09TiB 498GiB 620GiB 44.52 0.91 143 osd.34 35 nvme 1.00000 1.00000 1.09TiB 457GiB 660GiB 40.93 0.84 132 osd.35 -16 12.00000 - 13.1TiB 6.00TiB 7.10TiB 45.77 0.94 - rack rack4 -4 12.00000 - 13.1TiB 6.00TiB 7.10TiB 45.77 0.94 - node cpn04 36 nvme 1.00000 1.00000 1.09TiB 540GiB 578GiB 48.29 0.99 156 osd.36 37 nvme 1.00000 1.00000 1.09TiB 517GiB 601GiB 46.25 0.95 149 osd.37 38 nvme 1.00000 1.00000 1.09TiB 519GiB 599GiB 46.42 0.95 149 osd.38 39 nvme 1.00000 1.00000 1.09TiB 457GiB 661GiB 40.85 0.84 132 osd.39 40 nvme 1.00000 1.00000 1.09TiB 527GiB 590GiB 47.17 0.97 152 osd.40 41 nvme 1.00000 1.00000 1.09TiB 537GiB 581GiB 48.01 0.98 155 osd.41 42 nvme 1.00000 1.00000 1.09TiB 522GiB 596GiB 46.68 0.96 151 osd.42 43 nvme 1.00000 1.00000 1.09TiB 459GiB 658GiB 41.09 0.84 132 osd.43 44 nvme 1.00000 1.00000 1.09TiB 550GiB 568GiB 49.17 1.01 158 osd.44 45 nvme 1.00000 1.00000 1.09TiB 570GiB 548GiB 51.00 1.04 164 osd.45 46 nvme 1.00000 1.00000 1.09TiB 451GiB 667GiB 40.32 0.83 130 osd.46 47 nvme 1.00000 1.00000 1.09TiB 492GiB 626GiB 44.03 0.90 142 osd.47 -20 12.00000 - 10.9TiB 5.77TiB 5.15TiB 52.84 1.08 - rack rack5 -19 12.00000 - 10.9TiB 5.77TiB 5.15TiB 52.84 1.08 - node cpn05 48 nvme 1.00000 1.00000 931GiB 551GiB 380GiB 59.19 1.21 159 osd.48 49 nvme 1.00000 1.00000 931GiB 469GiB 462GiB 50.39 1.03 136 osd.49 50 nvme 1.00000 1.00000 931GiB 548GiB 384GiB 58.83 1.20 158 osd.50 51 nvme 1.00000 1.00000 931GiB 380GiB 551GiB 40.83 0.84 109 osd.51 52 nvme 1.00000 1.00000 931GiB 487GiB 445GiB 52.24 1.07 141 osd.52 53 nvme 1.00000 1.00000 931GiB 503GiB 429GiB 53.98 1.11 146 osd.53 54 nvme 1.00000 1.00000 931GiB 479GiB 452GiB 51.47 1.05 139 osd.54 55 nvme 1.00000 1.00000 931GiB 522GiB 410GiB 55.99 1.15 150 osd.55 56 nvme 1.00000 1.00000 931GiB 571GiB 360GiB 61.31 1.26 165 osd.56 57 nvme 1.00000 1.00000 931GiB 405GiB 527GiB 43.46 0.89 117 osd.57 58 nvme 1.00000 1.00000 931GiB 456GiB 475GiB 48.97 1.00 132 osd.58 59 nvme 1.00000 1.00000 931GiB 535GiB 396GiB 57.45 1.18 154 osd.59 -23 12.00000 - 10.9TiB 5.98TiB 4.93TiB 54.79 1.12 - rack rack6 -24 12.00000 - 10.9TiB 5.98TiB 4.93TiB 54.79 1.12 - node cpn06 60 nvme 1.00000 1.00000 931GiB 499GiB 432GiB 53.61 1.10 144 osd.60 61 nvme 1.00000 1.00000 931GiB 447GiB 485GiB 47.94 0.98 129 osd.61 62 nvme 1.00000 1.00000 931GiB 505GiB 426GiB 54.24 1.11 146 osd.62 63 nvme 1.00000 1.00000 931GiB 563GiB 368GiB 60.47 1.24 162 osd.63 64 nvme 1.00000 1.00000 931GiB 605GiB 326GiB 65.01 1.33 175 osd.64 65 nvme 1.00000 1.00000 931GiB 476GiB 455GiB 51.15 1.05 138 osd.65 66 nvme 1.00000 1.00000 931GiB 461GiB 471GiB 49.44 1.01 133 osd.66 67 nvme 1.00000 1.00000 931GiB 483GiB 448GiB 51.86 1.06 140 osd.67 68 nvme 1.00000 1.00000 931GiB 521GiB 411GiB 55.92 1.15 151 osd.68 69 nvme 1.00000 1.00000 931GiB 481GiB 450GiB 51.69 1.06 139 osd.69 70 nvme 1.00000 1.00000 931GiB 506GiB 426GiB 54.29 1.11 146 osd.70 71 nvme 1.00000 1.00000 931GiB 576GiB 355GiB 61.87 1.27 166 osd.71 -27 12.00000 - 10.9TiB 6.06TiB 4.85TiB 55.56 1.14 - rack rack7 -28 12.00000 - 10.9TiB 6.06TiB 4.85TiB 55.56 1.14 - node cpn07 72 nvme 1.00000 1.00000 931GiB 554GiB 378GiB 59.43 1.22 160 osd.72 73 nvme 1.00000 1.00000 931GiB 443GiB 489GiB 47.52 0.97 128 osd.73 74 nvme 1.00000 1.00000 931GiB 450GiB 481GiB 48.33 0.99 130 osd.74 75 nvme 1.00000 1.00000 931GiB 529GiB 403GiB 56.77 1.16 153 osd.75 76 nvme 1.00000 1.00000 931GiB 489GiB 443GiB 52.48 1.08 141 osd.76 77 nvme 1.00000 1.00000 931GiB 571GiB 360GiB 61.32 1.26 165 osd.77 78 nvme 1.00000 1.00000 931GiB 474GiB 458GiB 50.87 1.04 137 osd.78 79 nvme 1.00000 1.00000 931GiB 536GiB 395GiB 57.58 1.18 155 osd.79 80 nvme 1.00000 1.00000 931GiB 492GiB 440GiB 52.79 1.08 142 osd.80 81 nvme 1.00000 1.00000 931GiB 511GiB 421GiB 54.84 1.12 148 osd.81 82 nvme 1.00000 1.00000 931GiB 563GiB 368GiB 60.48 1.24 162 osd.82 83 nvme 1.00000 1.00000 931GiB 599GiB 332GiB 64.32 1.32 173 osd.83 TOTAL 85.1TiB 41.6TiB 43.6TiB 48.82 MIN/MAX VAR: 0.81/1.33 STDDEV: 6.30 Is there any documentation or scripts regarding manually distribution for rebalancing disk with minimal I/O impact on clients io? On Tue, Feb 4, 2020 at 10:25 AM EDH - Manuel Rios <mriosfer(a)easydatahost.com> wrote: > With “ceph osd df tree” will be clear but right now I can see that some > %USE osd between 44% and 65%. > > > > Ceph osd df tree give also the balance at host level. > > > > Do you have balancer enabled ?No “perfect” distribution cause that you > cant use the full space. > > > > In our case we gain space manually rebalancing disk, that cause some > objects moves to other osd but you can so fast space available. > > > > Regards > > > > > > *De:* German Anders <yodasbunker(a)gmail.com> > *Enviado el:* martes, 4 de febrero de 2020 14:20 > *Para:* EDH - Manuel Rios <mriosfer(a)easydatahost.com> > *CC:* ceph-users(a)ceph.com > *Asunto:* Re: [ceph-users] Doubt about AVAIL space on df > > > > Hi Manuel, > > > > Sure thing: > > > > # ceph osd df > ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS > 0 nvme 1.00000 1.00000 1.09TiB 496GiB 622GiB 44.35 0.91 143 > 1 nvme 1.00000 1.00000 1.09TiB 488GiB 630GiB 43.63 0.89 141 > 2 nvme 1.00000 1.00000 1.09TiB 537GiB 581GiB 48.05 0.99 155 > 3 nvme 1.00000 1.00000 1.09TiB 473GiB 644GiB 42.36 0.87 137 > 4 nvme 1.00000 1.00000 1.09TiB 531GiB 587GiB 47.52 0.97 153 > 5 nvme 1.00000 1.00000 1.09TiB 476GiB 642GiB 42.55 0.87 137 > 6 nvme 1.00000 1.00000 1.09TiB 467GiB 651GiB 41.77 0.86 135 > 7 nvme 1.00000 1.00000 1.09TiB 543GiB 574GiB 48.61 1.00 157 > 8 nvme 1.00000 1.00000 1.09TiB 481GiB 636GiB 43.08 0.88 139 > 9 nvme 1.00000 1.00000 1.09TiB 457GiB 660GiB 40.92 0.84 133 > 10 nvme 1.00000 1.00000 1.09TiB 513GiB 604GiB 45.92 0.94 148 > 11 nvme 1.00000 1.00000 1.09TiB 484GiB 634GiB 43.29 0.89 140 > 12 nvme 1.00000 1.00000 1.09TiB 498GiB 620GiB 44.57 0.91 144 > 13 nvme 1.00000 1.00000 1.09TiB 560GiB 557GiB 50.13 1.03 162 > 14 nvme 1.00000 1.00000 1.09TiB 576GiB 542GiB 51.55 1.06 167 > 15 nvme 1.00000 1.00000 1.09TiB 545GiB 572GiB 48.78 1.00 158 > 16 nvme 1.00000 1.00000 1.09TiB 537GiB 581GiB 48.02 0.98 155 > 17 nvme 1.00000 1.00000 1.09TiB 507GiB 611GiB 45.36 0.93 147 > 18 nvme 1.00000 1.00000 1.09TiB 490GiB 628GiB 43.86 0.90 142 > 19 nvme 1.00000 1.00000 1.09TiB 533GiB 584GiB 47.72 0.98 155 > 20 nvme 1.00000 1.00000 1.09TiB 467GiB 651GiB 41.75 0.86 134 > 21 nvme 1.00000 1.00000 1.09TiB 447GiB 671GiB 39.97 0.82 129 > 22 nvme 1.00099 1.00000 1.09TiB 561GiB 557GiB 50.16 1.03 162 > 23 nvme 1.00000 1.00000 1.09TiB 441GiB 677GiB 39.46 0.81 127 > 24 nvme 1.00000 1.00000 1.09TiB 500GiB 618GiB 44.72 0.92 145 > 25 nvme 1.00000 1.00000 1.09TiB 462GiB 656GiB 41.30 0.85 133 > 26 nvme 1.00000 1.00000 1.09TiB 445GiB 672GiB 39.85 0.82 129 > 27 nvme 1.00000 1.00000 1.09TiB 564GiB 554GiB 50.45 1.03 162 > 28 nvme 1.00000 1.00000 1.09TiB 512GiB 605GiB 45.84 0.94 148 > 29 nvme 1.00000 1.00000 1.09TiB 553GiB 565GiB 49.49 1.01 160 > 30 nvme 1.00000 1.00000 1.09TiB 526GiB 592GiB 47.07 0.97 152 > 31 nvme 1.00000 1.00000 1.09TiB 484GiB 633GiB 43.34 0.89 140 > 32 nvme 1.00000 1.00000 1.09TiB 504GiB 613GiB 45.13 0.93 146 > 33 nvme 1.00000 1.00000 1.09TiB 550GiB 567GiB 49.23 1.01 159 > 34 nvme 1.00000 1.00000 1.09TiB 497GiB 620GiB 44.51 0.91 143 > 35 nvme 1.00000 1.00000 1.09TiB 457GiB 661GiB 40.88 0.84 132 > 36 nvme 1.00000 1.00000 1.09TiB 539GiB 578GiB 48.25 0.99 156 > 37 nvme 1.00000 1.00000 1.09TiB 516GiB 601GiB 46.19 0.95 149 > 38 nvme 1.00000 1.00000 1.09TiB 518GiB 600GiB 46.35 0.95 149 > 39 nvme 1.00000 1.00000 1.09TiB 456GiB 662GiB 40.81 0.84 132 > 40 nvme 1.00000 1.00000 1.09TiB 527GiB 591GiB 47.13 0.97 152 > 41 nvme 1.00000 1.00000 1.09TiB 536GiB 581GiB 47.98 0.98 155 > 42 nvme 1.00000 1.00000 1.09TiB 521GiB 597GiB 46.62 0.96 151 > 43 nvme 1.00000 1.00000 1.09TiB 459GiB 659GiB 41.05 0.84 132 > 44 nvme 1.00000 1.00000 1.09TiB 549GiB 569GiB 49.12 1.01 158 > 45 nvme 1.00000 1.00000 1.09TiB 569GiB 548GiB 50.95 1.04 164 > 46 nvme 1.00000 1.00000 1.09TiB 450GiB 668GiB 40.28 0.83 130 > 47 nvme 1.00000 1.00000 1.09TiB 491GiB 626GiB 43.97 0.90 142 > 48 nvme 1.00000 1.00000 931GiB 551GiB 381GiB 59.13 1.21 159 > 49 nvme 1.00000 1.00000 931GiB 469GiB 463GiB 50.34 1.03 136 > 50 nvme 1.00000 1.00000 931GiB 548GiB 384GiB 58.78 1.21 158 > 51 nvme 1.00000 1.00000 931GiB 380GiB 552GiB 40.79 0.84 109 > 52 nvme 1.00000 1.00000 931GiB 486GiB 445GiB 52.20 1.07 141 > 53 nvme 1.00000 1.00000 931GiB 502GiB 429GiB 53.93 1.11 146 > 54 nvme 1.00000 1.00000 931GiB 479GiB 452GiB 51.42 1.05 139 > 55 nvme 1.00000 1.00000 931GiB 521GiB 410GiB 55.93 1.15 150 > 56 nvme 1.00000 1.00000 931GiB 570GiB 361GiB 61.25 1.26 165 > 57 nvme 1.00000 1.00000 931GiB 404GiB 527GiB 43.43 0.89 117 > 58 nvme 1.00000 1.00000 931GiB 455GiB 476GiB 48.89 1.00 132 > 59 nvme 1.00000 1.00000 931GiB 535GiB 397GiB 57.39 1.18 154 > 60 nvme 1.00000 1.00000 931GiB 499GiB 433GiB 53.56 1.10 144 > 61 nvme 1.00000 1.00000 931GiB 446GiB 485GiB 47.92 0.98 129 > 62 nvme 1.00000 1.00000 931GiB 505GiB 427GiB 54.18 1.11 146 > 63 nvme 1.00000 1.00000 931GiB 563GiB 369GiB 60.39 1.24 162 > 64 nvme 1.00000 1.00000 931GiB 605GiB 326GiB 64.99 1.33 175 > 65 nvme 1.00000 1.00000 931GiB 476GiB 455GiB 51.10 1.05 138 > 66 nvme 1.00000 1.00000 931GiB 460GiB 471GiB 49.38 1.01 133 > 67 nvme 1.00000 1.00000 931GiB 483GiB 449GiB 51.82 1.06 140 > 68 nvme 1.00000 1.00000 931GiB 520GiB 411GiB 55.86 1.15 151 > 69 nvme 1.00000 1.00000 931GiB 481GiB 450GiB 51.64 1.06 139 > 70 nvme 1.00000 1.00000 931GiB 505GiB 426GiB 54.24 1.11 146 > 71 nvme 1.00000 1.00000 931GiB 576GiB 356GiB 61.81 1.27 166 > 72 nvme 1.00000 1.00000 931GiB 552GiB 379GiB 59.30 1.22 160 > 73 nvme 1.00000 1.00000 931GiB 442GiB 489GiB 47.47 0.97 128 > 74 nvme 1.00000 1.00000 931GiB 450GiB 482GiB 48.28 0.99 130 > 75 nvme 1.00000 1.00000 931GiB 529GiB 403GiB 56.77 1.16 153 > 76 nvme 1.00000 1.00000 931GiB 488GiB 443GiB 52.44 1.08 141 > 77 nvme 1.00000 1.00000 931GiB 570GiB 361GiB 61.25 1.26 165 > 78 nvme 1.00000 1.00000 931GiB 473GiB 458GiB 50.79 1.04 137 > 79 nvme 1.00000 1.00000 931GiB 536GiB 396GiB 57.54 1.18 155 > 80 nvme 1.00000 1.00000 931GiB 491GiB 440GiB 52.74 1.08 142 > 81 nvme 1.00000 1.00000 931GiB 510GiB 421GiB 54.78 1.12 148 > 82 nvme 1.00000 1.00000 931GiB 563GiB 369GiB 60.42 1.24 162 > 83 nvme 1.00000 1.00000 931GiB 599GiB 333GiB 64.28 1.32 173 > TOTAL 85.1TiB 41.5TiB 43.6TiB 48.77 > MIN/MAX VAR: 0.81/1.33 STDDEV: 6.30 > > > > > > Thanks in advance, > > > > Best regards, > > > > > > > > On Tue, Feb 4, 2020 at 10:15 AM EDH - Manuel Rios < > mriosfer(a)easydatahost.com> wrote: > > Hi German, > > Can you post , ceph osd df tree ? > > Looks like your usage distribution is not perfect and that's why you got > less space than real. > Regards > > > -----Mensaje original----- > De: German Anders <yodasbunker(a)gmail.com> > Enviado el: martes, 4 de febrero de 2020 14:00 > Para: ceph-users(a)ceph.com > Asunto: [ceph-users] Doubt about AVAIL space on df > > Hello Everyone, > > I would like to understand if this output is right: > > *# ceph df* > GLOBAL: > SIZE AVAIL RAW USED %RAW USED > 85.1TiB 43.7TiB 41.4TiB 48.68 > POOLS: > NAME ID USED %USED MAX AVAIL OBJECTS > volumes 13 13.8TiB 64.21 7.68TiB 3620495 > > I only have (1) pool called 'volumes' which is using 13.8TiB (we have a > replica of 3) so it's actually using 41,4TiB and that would be the RAW > USED, at this point is fine, but, then it said on the GLOBAL section that > the AVAIL space is 43.7TiB and the %RAW USED is only 48.68%. > > So if I use the 7.68TiB of MAX AVAIL and the pool goes up to 100% of > usage, that would not lead to the total space of the cluster, right? I mean > were are those 43.7TiB of AVAIL space? > > I'm using Luminous 12.2.12 release. > > Sorry if it's a silly question or if it has been answered before. > > Thanks in advance, > > Best regards, > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > >

1 0

Re: Doubt about AVAIL space on df
by German Anders 04 Feb '20

04 Feb '20

Hi Manuel, Sure thing: # ceph osd df ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS 0 nvme 1.00000 1.00000 1.09TiB 496GiB 622GiB 44.35 0.91 143 1 nvme 1.00000 1.00000 1.09TiB 488GiB 630GiB 43.63 0.89 141 2 nvme 1.00000 1.00000 1.09TiB 537GiB 581GiB 48.05 0.99 155 3 nvme 1.00000 1.00000 1.09TiB 473GiB 644GiB 42.36 0.87 137 4 nvme 1.00000 1.00000 1.09TiB 531GiB 587GiB 47.52 0.97 153 5 nvme 1.00000 1.00000 1.09TiB 476GiB 642GiB 42.55 0.87 137 6 nvme 1.00000 1.00000 1.09TiB 467GiB 651GiB 41.77 0.86 135 7 nvme 1.00000 1.00000 1.09TiB 543GiB 574GiB 48.61 1.00 157 8 nvme 1.00000 1.00000 1.09TiB 481GiB 636GiB 43.08 0.88 139 9 nvme 1.00000 1.00000 1.09TiB 457GiB 660GiB 40.92 0.84 133 10 nvme 1.00000 1.00000 1.09TiB 513GiB 604GiB 45.92 0.94 148 11 nvme 1.00000 1.00000 1.09TiB 484GiB 634GiB 43.29 0.89 140 12 nvme 1.00000 1.00000 1.09TiB 498GiB 620GiB 44.57 0.91 144 13 nvme 1.00000 1.00000 1.09TiB 560GiB 557GiB 50.13 1.03 162 14 nvme 1.00000 1.00000 1.09TiB 576GiB 542GiB 51.55 1.06 167 15 nvme 1.00000 1.00000 1.09TiB 545GiB 572GiB 48.78 1.00 158 16 nvme 1.00000 1.00000 1.09TiB 537GiB 581GiB 48.02 0.98 155 17 nvme 1.00000 1.00000 1.09TiB 507GiB 611GiB 45.36 0.93 147 18 nvme 1.00000 1.00000 1.09TiB 490GiB 628GiB 43.86 0.90 142 19 nvme 1.00000 1.00000 1.09TiB 533GiB 584GiB 47.72 0.98 155 20 nvme 1.00000 1.00000 1.09TiB 467GiB 651GiB 41.75 0.86 134 21 nvme 1.00000 1.00000 1.09TiB 447GiB 671GiB 39.97 0.82 129 22 nvme 1.00099 1.00000 1.09TiB 561GiB 557GiB 50.16 1.03 162 23 nvme 1.00000 1.00000 1.09TiB 441GiB 677GiB 39.46 0.81 127 24 nvme 1.00000 1.00000 1.09TiB 500GiB 618GiB 44.72 0.92 145 25 nvme 1.00000 1.00000 1.09TiB 462GiB 656GiB 41.30 0.85 133 26 nvme 1.00000 1.00000 1.09TiB 445GiB 672GiB 39.85 0.82 129 27 nvme 1.00000 1.00000 1.09TiB 564GiB 554GiB 50.45 1.03 162 28 nvme 1.00000 1.00000 1.09TiB 512GiB 605GiB 45.84 0.94 148 29 nvme 1.00000 1.00000 1.09TiB 553GiB 565GiB 49.49 1.01 160 30 nvme 1.00000 1.00000 1.09TiB 526GiB 592GiB 47.07 0.97 152 31 nvme 1.00000 1.00000 1.09TiB 484GiB 633GiB 43.34 0.89 140 32 nvme 1.00000 1.00000 1.09TiB 504GiB 613GiB 45.13 0.93 146 33 nvme 1.00000 1.00000 1.09TiB 550GiB 567GiB 49.23 1.01 159 34 nvme 1.00000 1.00000 1.09TiB 497GiB 620GiB 44.51 0.91 143 35 nvme 1.00000 1.00000 1.09TiB 457GiB 661GiB 40.88 0.84 132 36 nvme 1.00000 1.00000 1.09TiB 539GiB 578GiB 48.25 0.99 156 37 nvme 1.00000 1.00000 1.09TiB 516GiB 601GiB 46.19 0.95 149 38 nvme 1.00000 1.00000 1.09TiB 518GiB 600GiB 46.35 0.95 149 39 nvme 1.00000 1.00000 1.09TiB 456GiB 662GiB 40.81 0.84 132 40 nvme 1.00000 1.00000 1.09TiB 527GiB 591GiB 47.13 0.97 152 41 nvme 1.00000 1.00000 1.09TiB 536GiB 581GiB 47.98 0.98 155 42 nvme 1.00000 1.00000 1.09TiB 521GiB 597GiB 46.62 0.96 151 43 nvme 1.00000 1.00000 1.09TiB 459GiB 659GiB 41.05 0.84 132 44 nvme 1.00000 1.00000 1.09TiB 549GiB 569GiB 49.12 1.01 158 45 nvme 1.00000 1.00000 1.09TiB 569GiB 548GiB 50.95 1.04 164 46 nvme 1.00000 1.00000 1.09TiB 450GiB 668GiB 40.28 0.83 130 47 nvme 1.00000 1.00000 1.09TiB 491GiB 626GiB 43.97 0.90 142 48 nvme 1.00000 1.00000 931GiB 551GiB 381GiB 59.13 1.21 159 49 nvme 1.00000 1.00000 931GiB 469GiB 463GiB 50.34 1.03 136 50 nvme 1.00000 1.00000 931GiB 548GiB 384GiB 58.78 1.21 158 51 nvme 1.00000 1.00000 931GiB 380GiB 552GiB 40.79 0.84 109 52 nvme 1.00000 1.00000 931GiB 486GiB 445GiB 52.20 1.07 141 53 nvme 1.00000 1.00000 931GiB 502GiB 429GiB 53.93 1.11 146 54 nvme 1.00000 1.00000 931GiB 479GiB 452GiB 51.42 1.05 139 55 nvme 1.00000 1.00000 931GiB 521GiB 410GiB 55.93 1.15 150 56 nvme 1.00000 1.00000 931GiB 570GiB 361GiB 61.25 1.26 165 57 nvme 1.00000 1.00000 931GiB 404GiB 527GiB 43.43 0.89 117 58 nvme 1.00000 1.00000 931GiB 455GiB 476GiB 48.89 1.00 132 59 nvme 1.00000 1.00000 931GiB 535GiB 397GiB 57.39 1.18 154 60 nvme 1.00000 1.00000 931GiB 499GiB 433GiB 53.56 1.10 144 61 nvme 1.00000 1.00000 931GiB 446GiB 485GiB 47.92 0.98 129 62 nvme 1.00000 1.00000 931GiB 505GiB 427GiB 54.18 1.11 146 63 nvme 1.00000 1.00000 931GiB 563GiB 369GiB 60.39 1.24 162 64 nvme 1.00000 1.00000 931GiB 605GiB 326GiB 64.99 1.33 175 65 nvme 1.00000 1.00000 931GiB 476GiB 455GiB 51.10 1.05 138 66 nvme 1.00000 1.00000 931GiB 460GiB 471GiB 49.38 1.01 133 67 nvme 1.00000 1.00000 931GiB 483GiB 449GiB 51.82 1.06 140 68 nvme 1.00000 1.00000 931GiB 520GiB 411GiB 55.86 1.15 151 69 nvme 1.00000 1.00000 931GiB 481GiB 450GiB 51.64 1.06 139 70 nvme 1.00000 1.00000 931GiB 505GiB 426GiB 54.24 1.11 146 71 nvme 1.00000 1.00000 931GiB 576GiB 356GiB 61.81 1.27 166 72 nvme 1.00000 1.00000 931GiB 552GiB 379GiB 59.30 1.22 160 73 nvme 1.00000 1.00000 931GiB 442GiB 489GiB 47.47 0.97 128 74 nvme 1.00000 1.00000 931GiB 450GiB 482GiB 48.28 0.99 130 75 nvme 1.00000 1.00000 931GiB 529GiB 403GiB 56.77 1.16 153 76 nvme 1.00000 1.00000 931GiB 488GiB 443GiB 52.44 1.08 141 77 nvme 1.00000 1.00000 931GiB 570GiB 361GiB 61.25 1.26 165 78 nvme 1.00000 1.00000 931GiB 473GiB 458GiB 50.79 1.04 137 79 nvme 1.00000 1.00000 931GiB 536GiB 396GiB 57.54 1.18 155 80 nvme 1.00000 1.00000 931GiB 491GiB 440GiB 52.74 1.08 142 81 nvme 1.00000 1.00000 931GiB 510GiB 421GiB 54.78 1.12 148 82 nvme 1.00000 1.00000 931GiB 563GiB 369GiB 60.42 1.24 162 83 nvme 1.00000 1.00000 931GiB 599GiB 333GiB 64.28 1.32 173 TOTAL 85.1TiB 41.5TiB 43.6TiB 48.77 MIN/MAX VAR: 0.81/1.33 STDDEV: 6.30 Thanks in advance, Best regards, On Tue, Feb 4, 2020 at 10:15 AM EDH - Manuel Rios <mriosfer(a)easydatahost.com> wrote: > Hi German, > > Can you post , ceph osd df tree ? > > Looks like your usage distribution is not perfect and that's why you got > less space than real. > Regards > > > -----Mensaje original----- > De: German Anders <yodasbunker(a)gmail.com> > Enviado el: martes, 4 de febrero de 2020 14:00 > Para: ceph-users(a)ceph.com > Asunto: [ceph-users] Doubt about AVAIL space on df > > Hello Everyone, > > I would like to understand if this output is right: > > *# ceph df* > GLOBAL: > SIZE AVAIL RAW USED %RAW USED > 85.1TiB 43.7TiB 41.4TiB 48.68 > POOLS: > NAME ID USED %USED MAX AVAIL OBJECTS > volumes 13 13.8TiB 64.21 7.68TiB 3620495 > > I only have (1) pool called 'volumes' which is using 13.8TiB (we have a > replica of 3) so it's actually using 41,4TiB and that would be the RAW > USED, at this point is fine, but, then it said on the GLOBAL section that > the AVAIL space is 43.7TiB and the %RAW USED is only 48.68%. > > So if I use the 7.68TiB of MAX AVAIL and the pool goes up to 100% of > usage, that would not lead to the total space of the cluster, right? I mean > were are those 43.7TiB of AVAIL space? > > I'm using Luminous 12.2.12 release. > > Sorry if it's a silly question or if it has been answered before. > > Thanks in advance, > > Best regards, > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io >

2 1

OSDs crashing
by Raymond Clotfelter 04 Feb '20

04 Feb '20

I have 30 or so OSDs on a cluster with 240 that just keep crashing. Below is the last part of one of the log files showing the crash, can anyone please help me read this to figure out what is going on and how to correct it? When I start the OSDs they generally seem to work for 5-30 minutes, and then one by one they will start dropping out with logs similar to this. Thanks. -29> 2020-02-04 06:00:23.447 7fe300d41700 5 osd.168 pg_epoch: 459335 pg[6.1217s2( v 443432'5 (0'0,443432'5] local-lis/les=459328/459329 n=1 ec=260267/6574 lis/c 459331/428950 les/c/f 459332/440468/290442 459333/459334/459294) [2147483647,107,168,2147483647,102]/[81,107,168,89,102]p81(0) r=2 lpr=459334 pi=[428950,459334)/82 crt=443432'5 lcod 0'0 remapped NOTIFY mbc={}] exit Started/Stray 1.017145 6 0.000323 -28> 2020-02-04 06:00:23.447 7fe300d41700 5 osd.168 pg_epoch: 459335 pg[6.1217s2( v 443432'5 (0'0,443432'5] local-lis/les=459328/459329 n=1 ec=260267/6574 lis/c 459331/428950 les/c/f 459332/440468/290442 459333/459334/459294) [2147483647,107,168,2147483647,102]/[81,107,168,89,102]p81(0) r=2 lpr=459334 pi=[428950,459334)/82 crt=443432'5 lcod 0'0 remapped NOTIFY mbc={}] enter Started/ReplicaActive -27> 2020-02-04 06:00:23.447 7fe300d41700 5 osd.168 pg_epoch: 459335 pg[6.1217s2( v 443432'5 (0'0,443432'5] local-lis/les=459328/459329 n=1 ec=260267/6574 lis/c 459331/428950 les/c/f 459332/440468/290442 459333/459334/459294) [2147483647,107,168,2147483647,102]/[81,107,168,89,102]p81(0) r=2 lpr=459334 pi=[428950,459334)/82 crt=443432'5 lcod 0'0 remapped NOTIFY mbc={}] enter Started/ReplicaActive/RepNotRecovering -26> 2020-02-04 06:00:23.455 7fe309d53700 3 osd.168 459335 handle_osd_map epochs [459335,459335], i have 459335, src has [403399,459335] -25> 2020-02-04 06:00:23.455 7fe309d53700 3 osd.168 459335 handle_osd_map epochs [459335,459335], i have 459335, src has [403399,459335] -24> 2020-02-04 06:00:23.455 7fe309d53700 3 osd.168 459335 handle_osd_map epochs [459335,459335], i have 459335, src has [403399,459335] -23> 2020-02-04 06:00:23.459 7fe309d53700 3 osd.168 459335 handle_osd_map epochs [459335,459335], i have 459335, src has [403399,459335] -22> 2020-02-04 06:00:23.459 7fe309d53700 3 osd.168 459335 handle_osd_map epochs [459335,459335], i have 459335, src has [403399,459335] -21> 2020-02-04 06:00:23.459 7fe309d53700 3 osd.168 459335 handle_osd_map epochs [459335,459335], i have 459335, src has [403399,459335] -20> 2020-02-04 06:00:23.459 7fe309d53700 3 osd.168 459335 handle_osd_map epochs [459335,459335], i have 459335, src has [403399,459335] -19> 2020-02-04 06:00:23.463 7fe309d53700 3 osd.168 459335 handle_osd_map epochs [459335,459335], i have 459335, src has [403399,459335] -18> 2020-02-04 06:00:23.463 7fe309d53700 3 osd.168 459335 handle_osd_map epochs [459335,459335], i have 459335, src has [403399,459335] -17> 2020-02-04 06:00:23.471 7fe309d53700 3 osd.168 459335 handle_osd_map epochs [459335,459335], i have 459335, src has [403399,459335] -16> 2020-02-04 06:00:23.471 7fe309d53700 3 osd.168 459335 handle_osd_map epochs [459335,459335], i have 459335, src has [403399,459335] -15> 2020-02-04 06:00:23.471 7fe300d41700 5 osd.168 pg_epoch: 459335 pg[6.1217s2( v 443432'5 (0'0,443432'5] local-lis/les=459334/459335 n=1 ec=260267/6574 lis/c 459334/428950 les/c/f 459335/440468/290442 459333/459334/459294) [2147483647,107,168,2147483647,102]/[81,107,168,89,102]p81(0) r=2 lpr=459334 pi=[428950,459334)/82 luod=0'0 crt=443432'5 lcod 0'0 active+remapped mbc={}] exit Started/ReplicaActive/RepNotRecovering 0.021923 2 0.000098 -14> 2020-02-04 06:00:23.471 7fe300d41700 5 osd.168 pg_epoch: 459335 pg[6.1217s2( v 443432'5 (0'0,443432'5] local-lis/les=459334/459335 n=1 ec=260267/6574 lis/c 459334/428950 les/c/f 459335/440468/290442 459333/459334/459294) [2147483647,107,168,2147483647,102]/[81,107,168,89,102]p81(0) r=2 lpr=459334 pi=[428950,459334)/82 luod=0'0 crt=443432'5 lcod 0'0 active+remapped mbc={}] enter Started/ReplicaActive/RepWaitRecoveryReserved -13> 2020-02-04 06:00:23.471 7fe300d41700 5 osd.168 pg_epoch: 459335 pg[6.1217s2( v 443432'5 (0'0,443432'5] local-lis/les=459334/459335 n=1 ec=260267/6574 lis/c 459334/428950 les/c/f 459335/440468/290442 459333/459334/459294) [2147483647,107,168,2147483647,102]/[81,107,168,89,102]p81(0) r=2 lpr=459334 pi=[428950,459334)/82 luod=0'0 crt=443432'5 lcod 0'0 active+remapped mbc={}] exit Started/ReplicaActive/RepWaitRecoveryReserved 0.000137 1 0.000080 -12> 2020-02-04 06:00:23.471 7fe300d41700 5 osd.168 pg_epoch: 459335 pg[6.1217s2( v 443432'5 (0'0,443432'5] local-lis/les=459334/459335 n=1 ec=260267/6574 lis/c 459334/428950 les/c/f 459335/440468/290442 459333/459334/459294) [2147483647,107,168,2147483647,102]/[81,107,168,89,102]p81(0) r=2 lpr=459334 pi=[428950,459334)/82 luod=0'0 crt=443432'5 lcod 0'0 active+remapped mbc={}] enter Started/ReplicaActive/RepRecovering -11> 2020-02-04 06:00:23.471 7fe309d53700 3 osd.168 459335 handle_osd_map epochs [459335,459335], i have 459335, src has [403399,459335] -10> 2020-02-04 06:00:23.475 7fe309d53700 3 osd.168 459335 handle_osd_map epochs [459335,459335], i have 459335, src has [403399,459335] -9> 2020-02-04 06:00:23.483 7fe300540700 5 osd.168 pg_epoch: 459335 pg[6.1961s0( v 436281'1 (0'0,436281'1] local-lis/les=459334/459335 n=0 ec=260267/6574 lis/c 459334/406589 les/c/f 459335/436403/290442 459333/459334/459334) [168,2147483647,2147483647,196,151]/[168,116,60,196,151]p168(0) r=0 lpr=459334 pi=[436277,459334)/33 crt=436281'1 lcod 0'0 mlcod 0'0 active+degraded+remapped m=30 mbc={0={(0+0)=30},1={(0+2)=30},2={(0+3)=30},3={(1+1)=30},4={(0+0)=30}}] exit Started/Primary/Active/Activating 0.034400 28 0.002950 -8> 2020-02-04 06:00:23.483 7fe300540700 5 osd.168 pg_epoch: 459335 pg[6.1961s0( v 436281'1 (0'0,436281'1] local-lis/les=459334/459335 n=0 ec=260267/6574 lis/c 459334/406589 les/c/f 459335/436403/290442 459333/459334/459334) [168,2147483647,2147483647,196,151]/[168,116,60,196,151]p168(0) r=0 lpr=459334 pi=[436277,459334)/33 crt=436281'1 lcod 0'0 mlcod 0'0 active+degraded+remapped m=30 mbc={0={(0+0)=30},1={(0+2)=30},2={(0+3)=30},3={(1+1)=30},4={(0+0)=30}}] enter Started/Primary/Active/WaitLocalRecoveryReserved -7> 2020-02-04 06:00:23.483 7fe300540700 5 osd.168 pg_epoch: 459335 pg[6.1961s0( v 436281'1 (0'0,436281'1] local-lis/les=459334/459335 n=0 ec=260267/6574 lis/c 459334/406589 les/c/f 459335/436403/290442 459333/459334/459334) [168,2147483647,2147483647,196,151]/[168,116,60,196,151]p168(0) r=0 lpr=459334 pi=[436277,459334)/33 crt=436281'1 lcod 0'0 mlcod 0'0 active+recovery_wait+degraded+remapped m=30 mbc={0={(0+0)=30},1={(0+2)=30},2={(0+3)=30},3={(1+1)=30},4={(0+0)=30}}] exit Started/Primary/Active/WaitLocalRecoveryReserved 0.000213 1 0.000221 -6> 2020-02-04 06:00:23.483 7fe300540700 5 osd.168 pg_epoch: 459335 pg[6.1961s0( v 436281'1 (0'0,436281'1] local-lis/les=459334/459335 n=0 ec=260267/6574 lis/c 459334/406589 les/c/f 459335/436403/290442 459333/459334/459334) [168,2147483647,2147483647,196,151]/[168,116,60,196,151]p168(0) r=0 lpr=459334 pi=[436277,459334)/33 crt=436281'1 lcod 0'0 mlcod 0'0 active+recovery_wait+degraded+remapped m=30 mbc={0={(0+0)=30},1={(0+2)=30},2={(0+3)=30},3={(1+1)=30},4={(0+0)=30}}] enter Started/Primary/Active/WaitRemoteRecoveryReserved -5> 2020-02-04 06:00:23.483 7fe300540700 5 osd.168 pg_epoch: 459335 pg[6.1961s0( v 436281'1 (0'0,436281'1] local-lis/les=459334/459335 n=0 ec=260267/6574 lis/c 459334/406589 les/c/f 459335/436403/290442 459333/459334/459334) [168,2147483647,2147483647,196,151]/[168,116,60,196,151]p168(0) r=0 lpr=459334 pi=[436277,459334)/33 crt=436281'1 lcod 0'0 mlcod 0'0 active+recovery_wait+degraded+remapped m=30 mbc={0={(0+0)=30},1={(0+2)=30},2={(0+3)=30},3={(1+1)=30},4={(0+0)=30}}] exit Started/Primary/Active/WaitRemoteRecoveryReserved 0.002796 5 0.000231 -4> 2020-02-04 06:00:23.483 7fe300540700 5 osd.168 pg_epoch: 459335 pg[6.1961s0( v 436281'1 (0'0,436281'1] local-lis/les=459334/459335 n=0 ec=260267/6574 lis/c 459334/406589 les/c/f 459335/436403/290442 459333/459334/459334) [168,2147483647,2147483647,196,151]/[168,116,60,196,151]p168(0) r=0 lpr=459334 pi=[436277,459334)/33 crt=436281'1 lcod 0'0 mlcod 0'0 active+recovery_wait+degraded+remapped m=30 mbc={0={(0+0)=30},1={(0+2)=30},2={(0+3)=30},3={(1+1)=30},4={(0+0)=30}}] enter Started/Primary/Active/Recovering -3> 2020-02-04 06:00:23.491 7fe309d53700 3 osd.168 459335 handle_osd_map epochs [459335,459335], i have 459335, src has [403399,459335] -2> 2020-02-04 06:00:23.519 7fe309d53700 3 osd.168 459335 handle_osd_map epochs [459335,459335], i have 459335, src has [403399,459335] -1> 2020-02-04 06:00:23.779 7fe300540700 -1 /build/ceph-14.2.7/src/osd/ECBackend.cc: In function 'void ECBackend::do_read_op(ECBackend::ReadOp&)' thread 7fe300540700 time 2020-02-04 06:00:23.774430 /build/ceph-14.2.7/src/osd/ECBackend.cc: 1742: FAILED ceph_assert(!need_attrs) ceph version 14.2.7 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8) nautilus (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x5579b9ecce4c] 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x5579b9ecd027] 3: (ECBackend::do_read_op(ECBackend::ReadOp&)+0xf76) [0x5579ba295106] 4: (ECBackend::send_all_remaining_reads(hobject_t const&, ECBackend::ReadOp&)+0x4d1) [0x5579ba2a6cf1] 5: (ECBackend::handle_sub_read_reply(pg_shard_t, ECSubReadReply&, RecoveryMessages*, ZTracer::Trace const&)+0xcf6) [0x5579ba2a7f26] 6: (ECBackend::_handle_message(boost::intrusive_ptr<OpRequest>)+0xbc) [0x5579ba2a8a8c] 7: (PGBackend::handle_message(boost::intrusive_ptr<OpRequest>)+0x97) [0x5579ba17f757] 8: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x705) [0x5579ba12dee5] 9: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x1bf) [0x5579b9f4ff2f] 10: (PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x62) [0x5579ba206e82] 11: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xbf5) [0x5579b9f6ea05] 12: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x4ac) [0x5579ba5714cc] 13: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x5579ba574690] 14: (()+0x7fa3) [0x7fe31ff5cfa3] 15: (clone()+0x3f) [0x7fe31fb0c4cf] 0> 2020-02-04 06:00:23.787 7fe300540700 -1 *** Caught signal (Aborted) ** in thread 7fe300540700 thread_name:tp_osd_tp ceph version 14.2.7 (3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8) nautilus (stable) 1: (()+0x12730) [0x7fe31ff67730] 2: (gsignal()+0x10b) [0x7fe31fa4a7bb] 3: (abort()+0x121) [0x7fe31fa35535] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a3) [0x5579b9ecce9d] 5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x5579b9ecd027] 6: (ECBackend::do_read_op(ECBackend::ReadOp&)+0xf76) [0x5579ba295106] 7: (ECBackend::send_all_remaining_reads(hobject_t const&, ECBackend::ReadOp&)+0x4d1) [0x5579ba2a6cf1] 8: (ECBackend::handle_sub_read_reply(pg_shard_t, ECSubReadReply&, RecoveryMessages*, ZTracer::Trace const&)+0xcf6) [0x5579ba2a7f26] 9: (ECBackend::_handle_message(boost::intrusive_ptr<OpRequest>)+0xbc) [0x5579ba2a8a8c] 10: (PGBackend::handle_message(boost::intrusive_ptr<OpRequest>)+0x97) [0x5579ba17f757] 11: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x705) [0x5579ba12dee5] 12: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x1bf) [0x5579b9f4ff2f] 13: (PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x62) [0x5579ba206e82] 14: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xbf5) [0x5579b9f6ea05] 15: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x4ac) [0x5579ba5714cc] 16: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x5579ba574690] 17: (()+0x7fa3) [0x7fe31ff5cfa3] 18: (clone()+0x3f) [0x7fe31fb0c4cf] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 0 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 1 reserver 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 rgw_sync 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 kinetic 1/ 5 fuse 1/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace 1/ 5 prioritycache -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-osd.168.log --- end dump of recent events --- -- ray

1 0

recovery_unfound
by Jake Grimmett 04 Feb '20

04 Feb '20

Dear All, Due to a mistake in my "rolling restart" script, one of our ceph clusters now has a number of unfound objects: There is an 8+2 erasure encoded data pool, 3x replicated metadata pool, all data is stored as cephfs. root@ceph7 ceph-archive]# ceph health HEALTH_ERR 24/420880027 objects unfound (0.000%); Possible data damage: 14 pgs recovery_unfound; Degraded data redundancy: 64/4204261148 objects degraded (0.000%), 14 pgs degraded "ceph health detail" gives me a handle on which pgs are affected. e.g: pg 5.f2f has 2 unfound objects pg 5.5c9 has 2 unfound objects pg 5.4c1 has 1 unfound objects and so on... plus more entries of this type: pg 5.6d is active+recovery_unfound+degraded, acting [295,104,57,442,240,338,219,33,150,382], 1 unfound pg 5.3fa is active+recovery_unfound+degraded, acting [343,147,21,131,315,63,214,365,264,437], 2 unfound pg 5.41d is active+recovery_unfound+degraded, acting [20,104,190,377,52,141,418,358,240,289], 1 unfound Digging deeper into one of the bad pg, we see the oid for the two unfound objects: root@ceph7 ceph-archive]# ceph pg 5.f2f list_unfound { "num_missing": 4, "num_unfound": 2, "objects": [ { "oid": { "oid": "1000ba25e49.00000207", "key": "", "snapid": -2, "hash": 854007599, "max": 0, "pool": 5, "namespace": "" }, "need": "22541'3088478", "have": "0'0", "flags": "none", "locations": [ "189(8)", "263(9)" ] }, { "oid": { "oid": "1000bb25a5b.00000091", "key": "", "snapid": -2, "hash": 3637976879, "max": 0, "pool": 5, "namespace": "" }, "need": "22541'3088476", "have": "0'0", "flags": "none", "locations": [ "189(8)", "263(9)" ] } ], "more": false } While it would be nice to recover the data, this cluster is only used for storing backups. As all OSD are up and running, presumably the data blocks are permanently lost? If it's hard / impossible to recover the data, presumably we should now consider using "ceph pg 5.f2f mark_unfound_lost delete" on each affected pg? Finally, can we use the oid to identify the affected files? best regards, Jake -- Jake Grimmett MRC Laboratory of Molecular Biology Francis Crick Avenue, Cambridge CB2 0QH, UK.

2 2

ceph positions
by Frank R 04 Feb '20

04 Feb '20

Hi all, I really hope this isn't seen as spam. I am looking to find a position where I can focus on Linux storage/Ceph. If anyone is currently looking please let me know. Linkedin profile frankritchie. Thanks, Frank

2 1

cephf_metadata: Large omap object found
by Yoann Moulin 03 Feb '20

03 Feb '20

Hello, I have this message on my new ceph cluster in Nautilus. I have a cephfs with a copy of ~100TB in progress. > /var/log/ceph/artemis.log:2020-02-03 16:22:49.970437 osd.66 (osd.66) 1137 : cluster [WRN] Large omap object found. Object: 8:579bf162:::mds3_openfiles.0:head PG: 8.468fd9ea (8.2a) Key count: 206548 Size (bytes): 6691941 > /var/log/ceph/artemis-osd.66.log:2020-02-03 16:22:49.966 7fe77af62700 0 log_channel(cluster) log [WRN] : Large omap object found. Object: 8:579bf162:::mds3_openfiles.0:head PG: 8.468fd9ea (8.2a) Key count: 206548 Size (bytes): 6691941 I found this thread about a similar issue in the archives of the list https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/JUFYDCQ2AHF… But I'm not sure what I can do in my situation, can I increase osd_deep_scrub_large_omap_object_key_threshold or it's a bad idea? Thanks for your help. Here some useful (I guess) information: > Filesystem Size Used Avail Use% Mounted on > 10.90.37.4,10.90.37.6,10.90.37.8:/ 329T 32T 297T 10% /artemis > artemis@icitsrv5:~$ ceph -s > cluster: > id: 815ea021-7839-4a63-9dc1-14f8c5feecc6 > health: HEALTH_WARN > 1 large omap objects > > services: > mon: 3 daemons, quorum iccluster003,iccluster005,iccluster007 (age 2w) > mgr: iccluster021(active, since 7h), standbys: iccluster009, iccluster023 > mds: cephfs:5 5 up:active > osd: 120 osds: 120 up (since 5d), 120 in (since 5d) > rgw: 8 daemons active (iccluster003.rgw0, iccluster005.rgw0, iccluster007.rgw0, iccluster013.rgw0, iccluster015.rgw0, iccluster019.rgw0, iccluster021.rgw0, iccluster023.rgw0) > > data: > pools: 10 pools, 2161 pgs > objects: 72.02M objects, 125 TiB > usage: 188 TiB used, 475 TiB / 662 TiB avail > pgs: 2157 active+clean > 4 active+clean+scrubbing+deep > > io: > client: 31 KiB/s rd, 803 KiB/s wr, 31 op/s rd, 184 op/s wr > artemis@icitsrv5:~$ ceph health detail > HEALTH_WARN 1 large omap objects > LARGE_OMAP_OBJECTS 1 large omap objects > 1 large objects found in pool 'cephfs_metadata' > Search the cluster log for 'Large omap object found' for more details. > artemis@icitsrv5:~$ ceph fs status > cephfs - 3 clients > ====== > +------+--------+--------------+---------------+-------+-------+ > | Rank | State | MDS | Activity | dns | inos | > +------+--------+--------------+---------------+-------+-------+ > | 0 | active | iccluster015 | Reqs: 0 /s | 251k | 251k | > | 1 | active | iccluster001 | Reqs: 3 /s | 20.2k | 19.1k | > | 2 | active | iccluster017 | Reqs: 1 /s | 116k | 112k | > | 3 | active | iccluster019 | Reqs: 0 /s | 263k | 263k | > | 4 | active | iccluster013 | Reqs: 123 /s | 16.3k | 16.3k | > +------+--------+--------------+---------------+-------+-------+ > +-----------------+----------+-------+-------+ > | Pool | type | used | avail | > +-----------------+----------+-------+-------+ > | cephfs_metadata | metadata | 13.9G | 135T | > | cephfs_data | data | 51.3T | 296T | > +-----------------+----------+-------+-------+ > +-------------+ > | Standby MDS | > +-------------+ > +-------------+ > MDS version: ceph version 14.2.6 (f0aa067ac7a02ee46ea48aa26c6e298b5ea272e9) nautilus (stable) > root@iccluster019:~# ceph --cluster artemis daemon osd.13 config show | grep large_omap > "osd_deep_scrub_large_omap_object_key_threshold": "200000", > "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", > artemis@icitsrv5:~$ rados -p cephfs_metadata listxattr mds3_openfiles.0 > artemis@icitsrv5:~$ rados -p cephfs_metadata getomapheader mds3_openfiles.0 > header (42 bytes) : > 00000000 13 00 00 00 63 65 70 68 20 66 73 20 76 6f 6c 75 |....ceph fs volu| > 00000010 6d 65 20 76 30 31 31 01 01 0d 00 00 00 14 63 00 |me v011.......c.| > 00000020 00 00 00 00 00 01 00 00 00 00 |..........| > 0000002a Best regards, -- Yoann Moulin EPFL IC-IT

2 1

v14.2.7 Nautilus released
by Abhishek Lekshmanan 03 Feb '20

03 Feb '20

This is the seventh update to the Ceph Nautilus release series. This is a hotfix release primarily fixing a couple of security issues. We recommend that all users upgrade to this release. Notable Changes --------------- * CVE-2020-1699: Fixed a path traversal flaw in Ceph dashboard that could allow for potential information disclosure (Ernesto Puerta) * CVE-2020-1700: Fixed a flaw in RGW beast frontend that could lead to denial of service from an unauthenticated client (Or Friedmann) Blog Link: https://ceph.io/releases/v14-2-7-nautilus-released/ Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-14.2.7.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: 3d58626ebeec02d8385a4cefb92c6cbc3a45bfe8 -- Abhishek Lekshmanan SUSE Software Solutions Germany GmbH

1 0

v12.2.13 Luminous released
by Abhishek Lekshmanan 03 Feb '20

03 Feb '20

We're happy to announce 13th bug fix release of the Luminous v12.2.x long term stable release series. We recommend that all users upgrade to this release. Many thanks to all the contributors, in particular Yuri & Nathan, in getting this release out of the door. This shall be the last release of the Luminous series. For a detailed release notes, please check out the official blog entry at https://ceph.io/releases/v12-2-13-luminous-released/ Notable Changes --------------- * Ceph now packages python bindings for python3.6 instead of python3.4, because EPEL7 recently switched from python3.4 to python3.6 as the native python3. see the announcement[1] for more details on the background of this change. * We now have telemetry support via a ceph-mgr module. The telemetry module is absolutely on an opt-in basis, and is meant to collect generic cluster information and push it to a central endpoint. By default, we're pushing it to a project endpoint at https://telemetry.ceph.com/report, but this is customizable using by setting the 'url' config option with:: ceph telemetry config-set url '<your url>' You will have to opt-in on sharing your information with:: ceph telemetry on You can view exactly what information will be reported first with:: ceph telemetry show Should you opt-in, your information will be licensed under the Community Data License Agreement - Sharing - Version 1.0, which you can read at https://cdla.io/sharing-1-0/ The telemetry module reports information about CephFS file systems, including: - how many MDS daemons (in total and per file system) - which features are (or have been) enabled - how many data pools - approximate file system age (year + month of creation) - how much metadata is being cached per file system As well as: - whether IPv4 or IPv6 addresses are used for the monitors - whether RADOS cache tiering is enabled (and which mode) - whether pools are replicated or erasure coded, and which erasure code profile plugin and parameters are in use - how many RGW daemons, zones, and zonegroups are present; which RGW frontends are in use - aggregate stats about the CRUSH map, like which algorithms are used, how big buckets are, how many rules are defined, and what tunables are in use * A health warning is now generated if the average osd heartbeat ping time exceeds a configurable threshold for any of the intervals computed. The OSD computes 1 minute, 5 minute and 15 minute intervals with average, minimum and maximum values. New configuration option `mon_warn_on_slow_ping_ratio` specifies a percentage of `osd_heartbeat_grace` to determine the threshold. A value of zero disables the warning. New configuration option `mon_warn_on_slow_ping_time` specified in milliseconds over-rides the computed value, causes a warning when OSD heartbeat pings take longer than the specified amount. New admin command `ceph daemon mgr.# dump_osd_network [threshold]` command will list all connections with a ping time longer than the specified threshold or value determined by the config options, for the average for any of the 3 intervals. New admin command `ceph daemon osd.# dump_osd_network [threshold]` will do the same but only including heartbeats initiated by the specified OSD. * The configuration value `osd_calc_pg_upmaps_max_stddev` used for upmap balancing has been removed. Instead use the mgr balancer config `upmap_max_deviation` which now is an integer number of PGs of deviation from the target PGs per OSD. This can be set with a command like `ceph config set mgr mgr/balancer/upmap_max_deviation 2`. The default `upmap_max_deviation` is 1. There are situations where crush rules would not allow a pool to ever have completely balanced PGs. For example, if crush requires 1 replica on each of 3 racks, but there are fewer OSDs in 1 of the racks. In those cases, the configuration value can be increased. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-12.2.13.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: 584a20eb0237c657dc0567da126be145106aa47e [1]: https://lists.fedoraproject.org/archives/list/epel-announce@lists.fedorapro… -- Abhishek Lekshmanan SUSE Software Solutions Germany GmbH GF: Felix Imendörffer HRB 21284 (AG Nürnberg)

1 0

cpu and memory for OSD server
by Wyatt Chun 03 Feb '20

03 Feb '20

We have 18 Sata disks (each 2TB) on a physical server, each disk with an OSD deployed. I am not sure how much CPU and memory resources should be prepared for this server. Does each OSD require a physical CPU? and how to calculate memory usage? Thanks.

1 0

ceph fs dir-layouts and sub-directory mounts
by Frank Schilder 03 Feb '20

03 Feb '20

I would like to (in this order) - set the data pool for the root "/" of a ceph-fs to a custom value, say "P" (not the initial data pool used in fs new) - create a sub-directory of "/", for example "/a" - mount the sub-directory "/a" with a client key with access restricted to "/a" The client will not be able to see the dir layout attribute set at "/", its not mounted. Will the data of this client still go to the pool "P", that is, does "/a" inherit the dir layout transparently to the client when following the steps above? Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

3 6

TR: Understand ceph df details
by CUZA Frédéric 03 Feb '20

03 Feb '20

Turns out it is probably orphans. We are running ceph luminous : 12.2.12 And the orphans find is stuck in the stage : "iterate_bucket_index" on shard "0" for 2 days now. Anyone is facing this issue ? Regards, De : ceph-users <ceph-users-bounces(a)lists.ceph.com<mailto:ceph-users-bounces@lists.ceph.com>> Envoyé : 21 January 2020 10:10 À : ceph-users(a)lists.ceph.com<mailto:ceph-users@lists.ceph.com> Objet : [ceph-users] Understand ceph df details Hi everyone, I'm trying to understand where is the difference between the command : ceph df details And the result I'm getting when I run this script : total_bytes=0 while read user; do echo $user bytes=$(radosgw-admin user stats --uid=${user} | grep total_bytes_rounded | tr -dc "0-9") if [ ! -z ${bytes} ]; then total_bytes=$((total_bytes + bytes)) pretty_bytes=$(echo "scale=2; $bytes / 1000^4" | bc) echo " ($bytes B) $pretty_bytes TiB" fi pretty_total_bytes=$(echo "scale=2; $total_bytes / 1000^4" | bc) done <<< "$(radosgw-admin user list | jq -r .[])" echo "" echo "Total : ($total_bytes B) $pretty_total_bytes TiB" When I run df I get this : default.rgw.buckets.data 70 N/A N/A 226TiB 89.23 27.2TiB 61676992 61.68M 2.05GiB 726MiB 677TiB And when I use my script I don't have the same result : Total : (207579728699392 B) 207.57 TiB It means that I have 20 TiB somewhere but I can't find and must of all understand where this 20 TiB. Does anyone have an explanation ? Fi : [root@ceph_monitor01 ~]# radosgw-admin gc list -include-all | grep oid | wc -l 23

2 1

Questions on Erasure Coding
by Dave Hall 03 Feb '20

03 Feb '20

Hello. Thanks to advice from bauen1 I now have OSDs on Debian/Nautilus and have been able to move on to MDS and CephFS. Also, looking around in the Dashboard I noticed the options for Crush Failure Domain and further that it's possible to select 'OSD'. As I mentioned earlier our cluster is fairly small at this point (3 hosts, 24 OSDs) , but we want to get as much usable storage as possible until we can get more nodes. SInce the nodes are brand new we are probably more concerned about disk failures than about node failures for the next few months. If I interpret Crush Failure Domain = OSD, this means it's possible to create pools that behave somewhat similar to RAID 6 - something like 8 + 2 except dispersed across multiple nodes. With the pool spread around like this loosing any one disk shouldn't put the cluster into read-only mode - if a disk did fail, would the cluster re-balance and reconstruct the lost data until the failed OSD was replaced. Does this make sense? Or is it just wishful thinking. Thanks. -Dave -- Dave Hall Binghamton University

2 1

small cluster HW upgrade
by Philipp Schwaha 02 Feb '20

02 Feb '20

We have a small ceph cluster running built from components that were phased out from compute applications. the current cluster consists of i7-860s. 6 disks (5TB, 7200RPM) per node and 8 nodes totaling 48 OSDs. A compute cluster will be discontinued, which will make Ryzen 5-1600 hardware available (8 nodes with 16GB RAM each) with which to replace the CPUs of the current setup. How could we best distribute the OSDs (keeping existing disks for storage) to the Ryzen systems to get a good performance improvement? Unfortunately the interconnect is still only 1Gb/s so expected to be a limiting factor. Would it make sense to create fewer bigger nodes, e.g., to use 6 nodes with 8 disks each or even more condensed? We would like to move the Luminous cluster to Nautilus/Bluestore and can get SSDs for each of the nodes, as it appears to be essential to get performance. Can actually benefit from improvements in the OSDs if the network is so limited? Would bonding of network interfaces be a workaround until we can get a network update or are we overestimating the power of the upgraded OSD nodes? What strategy would you suggest with these resources? Any comments and suggestions would be highly welcome :) Thanks in advance Philipp

4 4

CephFS - objects in default data pool
by CASS Philip 02 Feb '20

02 Feb '20

I have a query about https://docs.ceph.com/docs/master/cephfs/createfs/: "The data pool used to create the file system is the "default" data pool and the location for storing all inode backtrace information, used for hard link management and disaster recovery. For this reason, all inodes created in CephFS have at least one object in the default data pool." This does not match my experience (nautilus servers, nautlius FUSE client or Centos 7 kernel client). I have a cephfs with a replicated top-level pool and a directory set to use erasure coding with setfattr, though I also did the same test using the subvolume commands with the same result. "Ceph df detail" shows no objects used in the top level pool, as shown in https://gist.github.com/pcass-epcc/af24081cf014a66809e801f33bcb535b (also displayed in-line below) It would be useful if indeed clients didn't have to write to the top-level pool, since that would mean we could give different clients permission only to pool-associated subdirectories without giving everyone write access to a pool with data structures shared between all users of the filesystem. [root@hdr-admon01 ec]# ceph df detail; ceph fs ls; ceph fs status RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 3.3 PiB 3.3 PiB 32 TiB 32 TiB 0.95 nvme 2.9 TiB 2.9 TiB 504 MiB 2.5 GiB 0.08 TOTAL 3.3 PiB 3.3 PiB 32 TiB 32 TiB 0.95 POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL QUOTA OBJECTS QUOTA BYTES DIRTY USED COMPR UNDER COMPR cephfs.fs1.metadata 5 162 MiB 63 324 MiB 0.01 1.4 TiB N/A N/A 63 0 B 0 B cephfs.fs1-replicated.data 6 0 B 0 0 B 0 1.0 PiB N/A N/A 0 0 B 0 B cephfs.fs1-ec.data 7 8.0 GiB 2.05k 11 GiB 0 2.4 PiB N/A N/A 2.05k 0 B 0 B name: fs1, metadata pool: cephfs.fs1.metadata, data pools: [cephfs.fs1-replicated.data cephfs.fs1-ec.data ] fs1 - 4 clients === +------+--------+------------+---------------+-------+-------+ | Rank | State | MDS | Activity | dns | inos | +------+--------+------------+---------------+-------+-------+ | 0 | active | hdr-meta02 | Reqs: 0 /s | 29 | 16 | +------+--------+------------+---------------+-------+-------+ +----------------------------+----------+-------+-------+ | Pool | type | used | avail | +----------------------------+----------+-------+-------+ | cephfs.fs1.metadata | metadata | 324M | 1414G | | cephfs.fs1-replicated.data | data | 0 | 1063T | | cephfs.fs1-ec.data | data | 11.4G | 2505T | +----------------------------+----------+-------+-------+ +-------------+ | Standby MDS | +-------------+ | hdr-meta01 | +-------------+ MDS version: ceph version 14.2.5 (ad5bd132e1492173c85fda2cc863152730b16a92) nautilus (stable) [root@hdr-admon01 ec]# ll /test-fs/ec/ total 12582912 -rw-r--r--. 1 root root 4294967296 Jan 27 22:26 new-file -rw-r--r--. 2 root root 4294967296 Jan 28 14:06 new-file2 -rw-r--r--. 2 root root 4294967296 Jan 28 14:06 new-file-same-inode-as-newfile2 Regards, Phil _________________________________________ Philip Cass HPC Systems Specialist - Senior Systems Administrator EPCC [cid:image002.png@01D5D5EF.2E463230] Advanced Computing Facility Bush Estate Penicuik Tel: +44 (0)131 4457815 Email: p.cass(a)epcc.ed.ac.uk<mailto:p.cass@epcc.ed.ac.uk> _________________________________________ The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. The information contained in this e-mail (including any attachments) is confidential and is intended for the use of the addressee only. If you have received this message in error, please delete it and notify the originator immediately. Please consider the environment before printing this email.

4 6

Changing failure domain
by Francois Legrand 01 Feb '20

01 Feb '20

Hi, I have a cephfs in production based on 2 pools (data+metadata). Data is in erasure coding with the profile : crush-failure-domain=host crush-root=default jerasure-per-chunk-alignment=false k=3 m=2 plugin=jerasure technique=reed_sol_van w=8 Metadata is in replicated mode with k=3 The crush rules are as follow : [ { "rule_id": 0, "rule_name": "replicated_rule", "ruleset": 0, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 1, "rule_name": "ec_data", "ruleset": 1, "type": 3, "min_size": 3, "max_size": 5, "steps": [ { "op": "set_chooseleaf_tries", "num": 5 }, { "op": "set_choose_tries", "num": 100 }, { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_indep", "num": 0, "type": "host" }, { "op": "emit" } ] } ] When we installed it, everything was in the same room, but know we splitted our cluster (6 servers but soon 8) in 2 rooms. Thus we updated the crushmap by adding a room layer (with ceph osd crush add-bucket room1 room etc) and move all our servers in the tree to the correct place (ceph osd crush move server1 room=room1 etc...). Now, we would like to change the rules to set a failure domain to room instead of host (to be sure that in case of disaster in one of the rooms we will still have a copy in the other). What is the best strategy to do this ? F.

4 10

General question CephFS or RBD
by Willi Schiegel 01 Feb '20

01 Feb '20

Hello All, I have a HW RAID based 240 TB data pool with about 200 million files for users in a scientific institution. Data sizes range from tiny parameter files for scientific calculations and experiments to huge images of brain scans. There are group directories, home directories, Windows roaming profile directories organized in ZFS pools on Solaris operating systems, exported via NFS and Samba to Linux, macOS, and Windows clients. I would like to switch to CephFS because of the flexibility and expandability but I cannot find any recommendations for which storage backend would be suitable for all the functionality we have. Since I like the features of ZFS like immediate snapshots of very large data pools, quotas for each file system within hierarchical data trees and dynamic expandability by simply adding new disks or disk images without manual resizing would it be a good idea to create RBD images, map them onto the file servers and create zpools on the mapped images? I know that ZFS best works with raw disks but maybe a RBD image is close enough to a raw disk? Or would CephFS be the way to go? Can there be multiple CephFS pools for the group data folders and for the user's home directory folders for example or do I have to have everything in one single file space? Maybe someone can share his or her field experience? Thank you very much. Best regards Willi

3 2

Getting rid of trim_object Snap .... not in clones
by Andreas John 01 Feb '20

01 Feb '20

Hello, in my cluster one after the other OSD dies until I recognized that it was simply an "abort" in the daemon caused probably by 2020-01-31 15:54:42.535930 7faf8f716700 -1 log_channel(cluster) log [ERR] : trim_object Snap 29c44 not in clones Close to this msg I get a stracktrace: ceph version 0.94.10 (b1e0532418e4631af01acbc0cedd426f1905f4af) 1: /usr/bin/ceph-osd() [0xb35f7d] 2: (()+0x11390) [0x7f0fec74b390] 3: (gsignal()+0x38) [0x7f0feab43428] 4: (abort()+0x16a) [0x7f0feab4502a] 5: (__gnu_cxx::__verbose_terminate_handler()+0x16d) [0x7f0feb48684d] 6: (()+0x8d6b6) [0x7f0feb4846b6] 7: (()+0x8d701) [0x7f0feb484701] 8: (()+0x8d919) [0x7f0feb484919] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x27e) [0xc3776e] 10: (ReplicatedPG::eval_repop(ReplicatedPG::RepGather*)+0x10dd) [0x868cfd] 11: (ReplicatedPG::repop_all_committed(ReplicatedPG::RepGather*)+0x80) [0x8690e0] 12: (Context::complete(int)+0x9) [0x6c8799] 13: (void ReplicatedBackend::sub_op_modify_reply<MOSDRepOpReply, 113>(std::tr1::shared_ptr<OpRequest>)+0x21b) [0xa5ae0b] 14: (ReplicatedBackend::handle_message(std::tr1::shared_ptr<OpRequest>)+0x15b) [0xa53edb] 15: (ReplicatedPG::do_request(std::tr1::shared_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x1cb) [0x84c78b] 16: (OSD::dequeue_op(boost::intrusive_ptr<PG>, std::tr1::shared_ptr<OpRequest>, ThreadPool::TPHandle&)+0x3ef) [0x6966ff] 17: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x4e4) [0x696e14] 18: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x71e) [0xc264fe] 19: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xc29950] 20: (()+0x76ba) [0x7f0fec7416ba] 21: (clone()+0x6d) [0x7f0feac1541d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Yes, I know it's still hammer, I want to upgrade soon, but I want to resolve that issue first. If I lose that PG, I don't worry. So: What it the best approach? Can I use something like ceph-objectstore-tool ... <object> remove-clone-metadata <cloneid> ? I assume 29c44 is my Object, but what's the clone od? Best regards, derjohn

1 3

Near Perfect PG distrubtion apart from two OSD
by Ashley Merrick 01 Feb '20

01 Feb '20

So I recently updated CEPH and rebooted the OSD node's the two OSD's are now even more unbalanced and CEPH is actually currently moving more PG's too the OSD's in question (OSD 7,8), any ideas? ceph balancer status { "last_optimize_duration": "0:00:00.000289", "plans": [], "mode": "upmap", "active": true, "optimize_result": "Too many objects (0.008051 > 0.002000) are misplaced; try again later", "last_optimize_started": "Sat Feb 1 12:00:01 2020" } data: pools: 3 pools, 613 pgs objects: 35.12M objects, 130 TiB usage: 183 TiB used, 90 TiB / 273 TiB avail pgs: 2826030/351190811 objects misplaced (0.805%) 580 active+clean 29 active+remapped+backfill_wait 4 active+remapped+backfilling ceph osd df ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 23 hdd 0.00999 1.00000 10 GiB 1.4 GiB 460 MiB 1.5 MiB 1023 MiB 8.5 GiB 14.50 0.22 33 up 24 hdd 0.00999 1.00000 10 GiB 1.5 GiB 467 MiB 24 KiB 1024 MiB 8.5 GiB 14.57 0.22 34 up 25 hdd 0.00999 1.00000 10 GiB 1.5 GiB 462 MiB 28 KiB 1024 MiB 8.5 GiB 14.52 0.22 34 up 26 hdd 0.00999 1.00000 10 GiB 1.5 GiB 464 MiB 1.4 MiB 1023 MiB 8.5 GiB 14.53 0.22 34 up 27 hdd 0.00999 1.00000 10 GiB 1.5 GiB 464 MiB 40 KiB 1024 MiB 8.5 GiB 14.53 0.22 33 up 28 hdd 0.00999 1.00000 10 GiB 1.5 GiB 462 MiB 12 KiB 1024 MiB 8.5 GiB 14.52 0.22 34 up 3 hdd 9.09599 1.00000 9.1 TiB 6.1 TiB 6.1 TiB 76 KiB 19 GiB 3.0 TiB 67.06 1.00 170 up 4 hdd 9.09599 1.00000 9.1 TiB 5.9 TiB 5.9 TiB 32 KiB 18 GiB 3.2 TiB 64.57 0.96 164 up 5 hdd 9.09599 1.00000 9.1 TiB 6.4 TiB 6.4 TiB 44 KiB 19 GiB 2.7 TiB 70.77 1.06 180 up 6 hdd 9.09599 1.00000 9.1 TiB 6.2 TiB 6.1 TiB 20 KiB 19 GiB 2.9 TiB 67.65 1.01 171 up 7 hdd 9.09599 1.00000 9.1 TiB 7.0 TiB 7.0 TiB 8 KiB 21 GiB 2.1 TiB 77.19 1.15 196 up 8 hdd 9.09599 1.00000 9.1 TiB 6.6 TiB 6.5 TiB 56 KiB 20 GiB 2.5 TiB 72.04 1.07 183 up 9 hdd 9.09599 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 72 KiB 19 GiB 3.1 TiB 66.21 0.99 168 up 10 hdd 9.09599 1.00000 9.1 TiB 6.1 TiB 6.0 TiB 8 KiB 18 GiB 3.0 TiB 66.63 0.99 168 up 11 hdd 9.09599 1.00000 9.1 TiB 6.1 TiB 6.1 TiB 92 KiB 19 GiB 3.0 TiB 67.42 1.01 171 up 12 hdd 9.09599 1.00000 9.1 TiB 6.1 TiB 6.1 TiB 4 KiB 19 GiB 3.0 TiB 66.92 1.00 169 up 13 hdd 9.09599 1.00000 9.1 TiB 6.1 TiB 6.1 TiB 80 KiB 19 GiB 3.0 TiB 66.80 1.00 169 up 14 hdd 9.09599 1.00000 9.1 TiB 6.1 TiB 6.0 TiB 12 KiB 19 GiB 3.0 TiB 66.62 0.99 169 up 15 hdd 9.09599 1.00000 9.1 TiB 6.0 TiB 5.9 TiB 64 KiB 19 GiB 3.1 TiB 65.60 0.98 165 up 16 hdd 9.09599 1.00000 9.1 TiB 6.1 TiB 6.1 TiB 96 KiB 20 GiB 3.0 TiB 67.02 1.00 170 up 17 hdd 9.09599 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 44 KiB 19 GiB 3.1 TiB 66.13 0.99 168 up 18 hdd 9.09599 1.00000 9.1 TiB 6.3 TiB 6.3 TiB 12 KiB 20 GiB 2.8 TiB 69.36 1.03 176 up 19 hdd 9.09599 1.00000 9.1 TiB 6.2 TiB 6.2 TiB 60 KiB 19 GiB 2.9 TiB 67.87 1.01 173 up 20 hdd 9.09599 1.00000 9.1 TiB 6.2 TiB 6.1 TiB 48 KiB 19 GiB 2.9 TiB 67.77 1.01 171 up 21 hdd 9.09599 1.00000 9.1 TiB 6.3 TiB 6.3 TiB 52 KiB 20 GiB 2.8 TiB 68.96 1.03 175 up 22 hdd 9.09599 1.00000 9.1 TiB 6.1 TiB 6.1 TiB 16 KiB 19 GiB 3.0 TiB 67.15 1.00 170 up 29 hdd 9.09599 1.00000 9.1 TiB 5.8 TiB 5.8 TiB 80 KiB 18 GiB 3.3 TiB 63.48 0.95 163 up 30 hdd 9.09599 1.00000 9.1 TiB 5.9 TiB 5.9 TiB 40 KiB 18 GiB 3.2 TiB 64.80 0.97 167 up 31 hdd 9.09599 1.00000 9.1 TiB 6.2 TiB 6.2 TiB 152 KiB 19 GiB 2.9 TiB 68.29 1.02 175 up 32 hdd 9.09599 1.00000 9.1 TiB 6.1 TiB 6.0 TiB 128 KiB 18 GiB 3.0 TiB 66.55 0.99 171 up 33 hdd 9.09599 1.00000 9.1 TiB 6.1 TiB 6.1 TiB 48 KiB 19 GiB 3.0 TiB 67.19 1.00 173 up 34 hdd 9.09599 1.00000 9.1 TiB 6.1 TiB 6.1 TiB 60 KiB 18 GiB 3.0 TiB 66.90 1.00 172 up 35 hdd 9.09599 1.00000 9.1 TiB 5.8 TiB 5.8 TiB 52 KiB 18 GiB 3.3 TiB 64.22 0.96 165 up 36 hdd 9.09599 1.00000 9.1 TiB 5.4 TiB 5.4 TiB 128 KiB 17 GiB 3.7 TiB 59.20 0.88 152 up 37 hdd 9.09599 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 80 KiB 18 GiB 3.1 TiB 66.10 0.99 170 up 38 hdd 9.09599 1.00000 9.1 TiB 5.9 TiB 5.9 TiB 84 KiB 19 GiB 3.2 TiB 64.65 0.96 166 up 0 hdd 0.00999 1.00000 10 GiB 1.5 GiB 464 MiB 3 KiB 1024 MiB 8.5 GiB 14.54 0.22 34 up 1 hdd 0.00999 1.00000 10 GiB 1.4 GiB 460 MiB 1.4 MiB 1023 MiB 8.5 GiB 14.50 0.22 34 up 2 hdd 0.00999 1.00000 10 GiB 1.5 GiB 466 MiB 24 KiB 1024 MiB 8.5 GiB 14.55 0.22 33 up 23 hdd 0.00999 1.00000 10 GiB 1.4 GiB 460 MiB 1.5 MiB 1023 MiB 8.5 GiB 14.50 0.22 33 up 24 hdd 0.00999 1.00000 10 GiB 1.5 GiB 467 MiB 24 KiB 1024 MiB 8.5 GiB 14.57 0.22 34 up 25 hdd 0.00999 1.00000 10 GiB 1.5 GiB 462 MiB 28 KiB 1024 MiB 8.5 GiB 14.52 0.22 34 up 26 hdd 0.00999 1.00000 10 GiB 1.5 GiB 464 MiB 1.4 MiB 1023 MiB 8.5 GiB 14.53 0.22 34 up 27 hdd 0.00999 1.00000 10 GiB 1.5 GiB 464 MiB 40 KiB 1024 MiB 8.5 GiB 14.53 0.22 33 up 28 hdd 0.00999 1.00000 10 GiB 1.5 GiB 462 MiB 12 KiB 1024 MiB 8.5 GiB 14.52 0.22 34 up TOTAL 273 TiB 183 TiB 182 TiB 6.1 MiB 574 GiB 90 TiB 67.02 MIN/MAX VAR: 0.22/1.15 STDDEV: 30.40 ---- On Fri, 10 Jan 2020 14:57:05 +0800 Ashley Merrick <mailto:singapore@amerrick.co.uk> wrote ---- Hey, I have a cluster of 30 OSD's that is near perfect distribution minus two OSD's. I am running ceph version 14.2.6 however has been the same for the previous versions, I have the balance module enabled in upmap and it says no improvements, I have also tried in crush mode. ceph balancer status { "last_optimize_duration": "0:00:01.123659", "plans": [], "mode": "upmap", "active": true, "optimize_result": "Unable to find further optimization, or pool(s)' pg_num is decreasing, or distribution is already perfect", "last_optimize_started": "Fri Jan 10 06:11:08 2020" } I have read a few email threads on the ML recently about similar cases but not sure if I am hitting the same "bug" as its only two that are off the rest are almost perfect. ceph osd df ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 23 hdd 0.00999 1.00000 10 GiB 1.4 GiB 434 MiB 1.4 MiB 1023 MiB 8.6 GiB 14.24 0.21 33 up 24 hdd 0.00999 1.00000 10 GiB 1.4 GiB 441 MiB 48 KiB 1024 MiB 8.6 GiB 14.31 0.21 34 up 25 hdd 0.00999 1.00000 10 GiB 1.4 GiB 435 MiB 24 KiB 1024 MiB 8.6 GiB 14.26 0.21 34 up 26 hdd 0.00999 1.00000 10 GiB 1.4 GiB 436 MiB 1.4 MiB 1023 MiB 8.6 GiB 14.27 0.21 34 up 27 hdd 0.00999 1.00000 10 GiB 1.4 GiB 437 MiB 16 KiB 1024 MiB 8.6 GiB 14.27 0.21 33 up 28 hdd 0.00999 1.00000 10 GiB 1.4 GiB 436 MiB 36 KiB 1024 MiB 8.6 GiB 14.26 0.21 34 up 3 hdd 9.09599 1.00000 9.1 TiB 6.1 TiB 6.1 TiB 76 KiB 19 GiB 3.0 TiB 67.26 1.00 170 up 4 hdd 9.09599 1.00000 9.1 TiB 6.2 TiB 6.1 TiB 44 KiB 19 GiB 2.9 TiB 67.77 1.01 172 up 5 hdd 9.09599 1.00000 9.1 TiB 6.3 TiB 6.3 TiB 112 KiB 20 GiB 2.8 TiB 69.50 1.03 176 up 6 hdd 9.09599 1.00000 9.1 TiB 6.1 TiB 6.1 TiB 17 KiB 19 GiB 2.9 TiB 67.58 1.01 171 up 7 hdd 9.09599 1.00000 9.1 TiB 6.7 TiB 6.7 TiB 88 KiB 21 GiB 2.4 TiB 73.98 1.10 187 up 8 hdd 9.09599 1.00000 9.1 TiB 6.5 TiB 6.5 TiB 76 KiB 20 GiB 2.6 TiB 71.84 1.07 182 up 9 hdd 9.09599 1.00000 9.1 TiB 6.1 TiB 6.1 TiB 120 KiB 19 GiB 3.0 TiB 67.24 1.00 170 up 10 hdd 9.09599 1.00000 9.1 TiB 6.1 TiB 6.1 TiB 72 KiB 19 GiB 3.0 TiB 67.19 1.00 170 up 11 hdd 9.09599 1.00000 9.1 TiB 6.2 TiB 6.2 TiB 40 KiB 19 GiB 2.9 TiB 68.06 1.01 172 up 12 hdd 9.09599 1.00000 9.1 TiB 6.1 TiB 6.1 TiB 28 KiB 19 GiB 3.0 TiB 67.48 1.00 170 up 13 hdd 9.09599 1.00000 9.1 TiB 6.1 TiB 6.1 TiB 36 KiB 19 GiB 3.0 TiB 67.04 1.00 170 up 14 hdd 9.09599 1.00000 9.1 TiB 6.1 TiB 6.1 TiB 108 KiB 19 GiB 3.0 TiB 67.30 1.00 170 up 15 hdd 9.09599 1.00000 9.1 TiB 6.1 TiB 6.1 TiB 68 KiB 19 GiB 3.0 TiB 67.41 1.00 170 up 16 hdd 9.09599 1.00000 9.1 TiB 6.1 TiB 6.1 TiB 152 KiB 19 GiB 2.9 TiB 67.61 1.01 171 up 17 hdd 9.09599 1.00000 9.1 TiB 6.1 TiB 6.1 TiB 36 KiB 19 GiB 3.0 TiB 67.16 1.00 170 up 18 hdd 9.09599 1.00000 9.1 TiB 6.1 TiB 6.1 TiB 41 KiB 19 GiB 3.0 TiB 67.19 1.00 170 up 19 hdd 9.09599 1.00000 9.1 TiB 6.1 TiB 6.1 TiB 64 KiB 19 GiB 3.0 TiB 67.49 1.00 171 up 20 hdd 9.09599 1.00000 9.1 TiB 6.1 TiB 6.1 TiB 12 KiB 19 GiB 3.0 TiB 67.55 1.01 171 up 21 hdd 9.09599 1.00000 9.1 TiB 6.2 TiB 6.1 TiB 76 KiB 19 GiB 2.9 TiB 67.76 1.01 171 up 22 hdd 9.09599 1.00000 9.1 TiB 6.2 TiB 6.2 TiB 12 KiB 19 GiB 2.9 TiB 68.05 1.01 172 up 29 hdd 9.09599 1.00000 9.1 TiB 5.8 TiB 5.8 TiB 108 KiB 17 GiB 3.3 TiB 63.59 0.95 163 up 30 hdd 9.09599 1.00000 9.1 TiB 5.9 TiB 5.9 TiB 24 KiB 18 GiB 3.2 TiB 65.18 0.97 167 up 31 hdd 9.09599 1.00000 9.1 TiB 6.1 TiB 6.1 TiB 44 KiB 18 GiB 3.0 TiB 66.74 0.99 171 up 32 hdd 9.09599 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 220 KiB 18 GiB 3.1 TiB 66.31 0.99 170 up 33 hdd 9.09599 1.00000 9.1 TiB 6.0 TiB 5.9 TiB 36 KiB 18 GiB 3.1 TiB 65.54 0.98 168 up 34 hdd 9.09599 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 44 KiB 18 GiB 3.1 TiB 66.33 0.99 170 up 35 hdd 9.09599 1.00000 9.1 TiB 5.9 TiB 5.9 TiB 68 KiB 18 GiB 3.2 TiB 64.77 0.96 166 up 36 hdd 9.09599 1.00000 9.1 TiB 5.8 TiB 5.8 TiB 168 KiB 17 GiB 3.3 TiB 63.60 0.95 163 up 37 hdd 9.09599 1.00000 9.1 TiB 6.0 TiB 6.0 TiB 60 KiB 18 GiB 3.1 TiB 65.91 0.98 169 up 38 hdd 9.09599 1.00000 9.1 TiB 5.9 TiB 5.9 TiB 68 KiB 18 GiB 3.2 TiB 65.15 0.97 167 up 0 hdd 0.00999 1.00000 10 GiB 1.4 GiB 437 MiB 28 KiB 1024 MiB 8.6 GiB 14.27 0.21 34 up 1 hdd 0.00999 1.00000 10 GiB 1.4 GiB 434 MiB 1.4 MiB 1023 MiB 8.6 GiB 14.24 0.21 34 up 2 hdd 0.00999 1.00000 10 GiB 1.4 GiB 439 MiB 36 KiB 1024 MiB 8.6 GiB 14.29 0.21 33 up 23 hdd 0.00999 1.00000 10 GiB 1.4 GiB 434 MiB 1.4 MiB 1023 MiB 8.6 GiB 14.24 0.21 33 up 24 hdd 0.00999 1.00000 10 GiB 1.4 GiB 441 MiB 48 KiB 1024 MiB 8.6 GiB 14.31 0.21 34 up 25 hdd 0.00999 1.00000 10 GiB 1.4 GiB 435 MiB 24 KiB 1024 MiB 8.6 GiB 14.26 0.21 34 up 26 hdd 0.00999 1.00000 10 GiB 1.4 GiB 436 MiB 1.4 MiB 1023 MiB 8.6 GiB 14.27 0.21 34 up 27 hdd 0.00999 1.00000 10 GiB 1.4 GiB 437 MiB 16 KiB 1024 MiB 8.6 GiB 14.27 0.21 33 up 28 hdd 0.00999 1.00000 10 GiB 1.4 GiB 436 MiB 36 KiB 1024 MiB 8.6 GiB 14.26 0.21 34 up TOTAL 273 TiB 183 TiB 183 TiB 6.4 MiB 567 GiB 90 TiB 67.17 ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -12 0.05798 root default -11 0.02899 host sn-m02 23 hdd 0.00999 osd.23 up 1.00000 1.00000 24 hdd 0.00999 osd.24 up 1.00000 1.00000 25 hdd 0.00999 osd.25 up 1.00000 1.00000 -15 0.02899 host sn-m03 26 hdd 0.00999 osd.26 up 1.00000 1.00000 27 hdd 0.00999 osd.27 up 1.00000 1.00000 28 hdd 0.00999 osd.28 up 1.00000 1.00000 -6 272.87100 root ec -5 90.95700 host sn-s01 3 hdd 9.09599 osd.3 up 1.00000 1.00000 4 hdd 9.09599 osd.4 up 1.00000 1.00000 5 hdd 9.09599 osd.5 up 1.00000 1.00000 6 hdd 9.09599 osd.6 up 1.00000 1.00000 7 hdd 9.09599 osd.7 up 1.00000 1.00000 8 hdd 9.09599 osd.8 up 1.00000 1.00000 9 hdd 9.09599 osd.9 up 1.00000 1.00000 10 hdd 9.09599 osd.10 up 1.00000 1.00000 11 hdd 9.09599 osd.11 up 1.00000 1.00000 12 hdd 9.09599 osd.12 up 1.00000 1.00000 -9 90.95700 host sn-s02 13 hdd 9.09599 osd.13 up 1.00000 1.00000 14 hdd 9.09599 osd.14 up 1.00000 1.00000 15 hdd 9.09599 osd.15 up 1.00000 1.00000 16 hdd 9.09599 osd.16 up 1.00000 1.00000 17 hdd 9.09599 osd.17 up 1.00000 1.00000 18 hdd 9.09599 osd.18 up 1.00000 1.00000 19 hdd 9.09599 osd.19 up 1.00000 1.00000 20 hdd 9.09599 osd.20 up 1.00000 1.00000 21 hdd 9.09599 osd.21 up 1.00000 1.00000 22 hdd 9.09599 osd.22 up 1.00000 1.00000 -17 90.95700 host sn-s03 29 hdd 9.09599 osd.29 up 1.00000 1.00000 30 hdd 9.09599 osd.30 up 1.00000 1.00000 31 hdd 9.09599 osd.31 up 1.00000 1.00000 32 hdd 9.09599 osd.32 up 1.00000 1.00000 33 hdd 9.09599 osd.33 up 1.00000 1.00000 34 hdd 9.09599 osd.34 up 1.00000 1.00000 35 hdd 9.09599 osd.35 up 1.00000 1.00000 36 hdd 9.09599 osd.36 up 1.00000 1.00000 37 hdd 9.09599 osd.37 up 1.00000 1.00000 38 hdd 9.09599 osd.38 up 1.00000 1.00000 -1 0.08698 root meta -3 0.02899 host sn-m01 0 hdd 0.00999 osd.0 up 1.00000 1.00000 1 hdd 0.00999 osd.1 up 1.00000 1.00000 2 hdd 0.00999 osd.2 up 1.00000 1.00000 -11 0.02899 host sn-m02 23 hdd 0.00999 osd.23 up 1.00000 1.00000 24 hdd 0.00999 osd.24 up 1.00000 1.00000 25 hdd 0.00999 osd.25 up 1.00000 1.00000 -15 0.02899 host sn-m03 26 hdd 0.00999 osd.26 up 1.00000 1.00000 27 hdd 0.00999 osd.27 up 1.00000 1.00000 28 hdd 0.00999 osd.28 up 1.00000 1.00000 OSD 7,8 are the issue OSD's sitting at 182,187 PG's where the others are all sitting at 170,171. Am I hitting the same issue? Or is there something I can do to re balance these extra PG's across the rest of the OSD better? Thanks

1 0