- ceph-users - lists.ceph.io

How to reset an OSD
by Pfannes, Fabian 14 Jan '21

14 Jan '21

We running a small Ceph cluster with two nodes. Our failureDomain is set to host to have the data replicated between the two hosts. The other night one host crashed hard and three OSDs won't recovert with either debug 2021-01-13T08:13:17.855+0000 7f9bfbd6ef40 -1 osd.23 0 OSD::init() : unable to read osd superblock debug 2021-01-13T08:13:17.855+0000 7f9bfbd6ef40 1 bluestore(/var/lib/ceph/osd/ceph-23) umount debug 2021-01-13T08:13:17.855+0000 7f9bea85a700 0 bluestore(/var/lib/ceph/osd/ceph-23) allocation stats probe 0: cnt: 0 frags: 0 size: 0 debug 2021-01-13T08:13:17.855+0000 7f9bea85a700 0 bluestore(/var/lib/ceph/osd/ceph-23) probe -1: 0, 0, 0 debug 2021-01-13T08:13:17.855+0000 7f9bea85a700 0 bluestore(/var/lib/ceph/osd/ceph-23) probe -2: 0, 0, 0 debug 2021-01-13T08:13:17.855+0000 7f9bea85a700 0 bluestore(/var/lib/ceph/osd/ceph-23) probe -4: 0, 0, 0 debug 2021-01-13T08:13:17.855+0000 7f9bea85a700 0 bluestore(/var/lib/ceph/osd/ceph-23) probe -8: 0, 0, 0 debug 2021-01-13T08:13:17.855+0000 7f9bea85a700 0 bluestore(/var/lib/ceph/osd/ceph-23) probe -16: 0, 0, 0 debug 2021-01-13T08:13:17.855+0000 7f9bea85a700 0 bluestore(/var/lib/ceph/osd/ceph-23) ------------ debug 2021-01-13T08:13:17.855+0000 7f9bfbd6ef40 4 rocksdb: [db/db_impl.cc:390] Shutdown: canceling all background work debug 2021-01-13T08:13:17.855+0000 7f9bfbd6ef40 4 rocksdb: [db/db_impl.cc:563] Shutdown complete debug 2021-01-13T08:13:17.855+0000 7f9bfbd6ef40 1 bluefs umount debug 2021-01-13T08:13:17.855+0000 7f9bfbd6ef40 1 bdev(0x557150e20700 /var/lib/ceph/osd/ceph-23/block) close debug 2021-01-13T08:13:18.167+0000 7f9bfbd6ef40 1 freelist shutdown debug 2021-01-13T08:13:18.167+0000 7f9bfbd6ef40 1 bdev(0x557150e20000 /var/lib/ceph/osd/ceph-23/block) close debug 2021-01-13T08:13:18.411+0000 7f9bfbd6ef40 -1 ** ERROR: osd init failed: (22) Invalid argument or debug -2> 2021-01-13T08:13:29.991+0000 7f402c5f9700 -1 rocksdb: submit_common error: Corruption: block checksum mismatch: expected 2795871023, got 2381104739 in db/000060.sst offset 748408 size 3819 code = 2 Rocksdb transaction: How can I delete the OSDs to get them back fully operational? Any help appreciated! /Fabian

2 2

Decoding pgmap
by George Shuklin 14 Jan '21

14 Jan '21

There is a command `ceph pg getmap`. It produces a binary file. Are there any utility to decode it?

1 0

Latency spike investigations on all SSD hardware cluster
by Martin Hronek 14 Jan '21

14 Jan '21

Hello fellow CEPH-users, we are currently investigating latency spikes in our CEPH(14.2.11) prod cluster, usually occurring when under heavy load. TLDR: Do you have an idea where to investigate some kv commit latency spikes on a CEPH cluster with a LSI 9300-8i HBA and all SSD(Intel, Micron) OSDs? The cluster consists of 3MDS nodes(2 active + 1 standby-replay), 3MON nodes(each running MGR+MON daemon) and 4OSD nodes(each having 8 SSD bluestore OSD disks). All nodes are running ubuntu latest 18.04 with kernel version 5.4(2 OSD Server still have 4.15 - but spike are seen on all of the OSD servers). As the spikes seem to be randomly distributed across time(under load) and OSD, we followed the spikes to find following messages on the OSD nodes: ``` bluestore(/var/lib/ceph/osd/ceph-31) log_latency slow operation observed for kv_sync, latency = 5.22298s bluestore(/var/lib/ceph/osd/ceph-31) log_latency_fn slow operation observed for _txc_committed_kv, latency = 5.5732s, txc = 0x55b1a98d9e00 ... bluestore(/var/lib/ceph/osd/ceph-31) log_latency_fn slow operation observed for _txc_committed_kv, latency = 5.50842s, txc = 0x55b1aa197800 bluestore(/var/lib/ceph/osd/ceph-31) log_latency_fn slow operation observed for _txc_committed_kv, latency = 5.5058s, txc = 0x55b1b7e75c00 ``` We found timely correlated kernel messages, which suggest that it might has something to do with the underlying SSDs ``` kernel: [3613612.312027] sd 4:0:10:0: attempting task abort!scmd(0x00000000dac86408), outstanding for 31384 ms & timeout 30000 ms kernel: [3613612.312034] sd 4:0:10:0: [sdg] tag#744 CDB: Write(10) 2a 00 be 11 b8 80 00 00 08 00 kernel: [3613612.312036] scsi target4:0:10: handle(0x0013), sas_address(0x4433221104000000), phy(4) kernel: [3613612.312038] scsi target4:0:10: enclosure logical id(0x500605b00e70a7b0), slot(7) kernel: [3613612.312039] scsi target4:0:10: enclosure level(0x0000), connector name( ) kernel: [3613612.312040] sd 4:0:10:0: No reference found at driver, assuming scmd(0x00000000dac86408) might have completed kernel: [3613612.312042] sd 4:0:10:0: task abort: SUCCESS scmd(0x00000000dac86408) ``` There are lots of the above blocks until the Kernel apparently has enough of them and just resets the device/interface: ``` kernel: [3613612.312267] sd 4:0:10:0: attempting task abort!scmd(0x00000000d7aaff5a), outstanding for 31388 ms & timeout 30000 ms kernel: [3613612.312269] sd 4:0:10:0: [sdg] tag#520 CDB: Write(10) 2a 00 be 11 b4 e0 00 00 08 00 kernel: [3613612.312269] scsi target4:0:10: handle(0x0013), sas_address(0x4433221104000000), phy(4) kernel: [3613612.312270] scsi target4:0:10: enclosure logical id(0x500605b00e70a7b0), slot(7) kernel: [3613612.312271] scsi target4:0:10: enclosure level(0x0000), connector name( ) kernel: [3613612.312272] sd 4:0:10:0: No reference found at driver, assuming scmd(0x00000000d7aaff5a) might have completed kernel: [3613612.312273] sd 4:0:10:0: task abort: SUCCESS scmd(0x00000000d7aaff5a) kernel: [3613612.653004] sd 4:0:10:0: Power-on or device reset occurred kernel: [3613613.254064] sd 4:0:10:0: Power-on or device reset occurred ``` OSD nodes are equipped with an "LSI 9300-8i SAS HBA" also we use two types of SSDs "Intel SSD D3-S4510 Series 1,92 TB", "Micron 5210 ION 1.92TB SSD". Since these resets happen on both SSDs we figured the least common denominator is the HBA so we did an Upgrade to the latest FW/BIOS on one OSD node. Sadly this did not solve the issue. * The question is now has someone a similar hardware configuration and issues with it ? * Do you have an idea what could be the cause of this behaviour ? * Or which part to investigate further ? Thanks for your hints and time reading :) M

1 0

Ceph RadosGW & OpenStack swift problem
by Mika Saari 14 Jan '21

14 Jan '21

Hi, Using Ceph 15.2.8 installed with cephadm. Trying to get RadosGW to work. I have managed to get the RadosGW working. I can manage it through a dashboard and use aws s3 client to create new buckets etc. When trying to use swift I get errors. Not sure how to continue to track the problem here. Any tips are welcome. Thank you very much, -Mika ------- What I have done and what are the results. Some data changed manually ------- What I have done: At OpenStack Side: 1) openstack user create --domain default --password-prompt swift 2) openstack role add --project service --user swift admin 3) openstack endpoint create --region RegionOne object-store public http://ceph1/swift/v1/AUTH_%\(project_id\)s 4) openstack endpoint create --region RegionOne object-store internal http://ceph1/swift/v1/AUTH_%\(project_id\)s 5) openstack endpoint create --region RegionOne object-store admin http://ceph1/swift/v1 At Ceph side: 1) ceph config set mgr rgw_keystone_api_version 3 2) ceph config set mgr rgw_keystone_url http://controller:5000 3) ceph config set mgr rgw_keystone_accepted_admin_roles admin 4) ceph config set mgr rgw_keystone_admin_user swift 5) ceph config set mgr rgw_keystone_admin_password swift_test 6) ceph config set mgr rgw_keystone_admin_domain default 7) ceph config set mgr rgw_keystone_admin_project service for project I have tested different projects e.g. service and admin Now when testing the API using swift client I get next: 1) swift post test3 --debug DEBUG:keystoneclient.auth.identity.v3.base:Making authentication request to http://controller:5000/v3/auth/tokens DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): controller:5000 DEBUG:urllib3.connectionpool:http://controller:5000 "POST /v3/auth/tokens HTTP/1.1" 201 7032 . some openstack data here . DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): ceph1:80 DEBUG:urllib3.connectionpool:http://ceph1:80 "POST /swift/v1/AUTH_adsfasdfasdfasdfasdfasdf/test3 HTTP/1.1" 401 12 INFO:swiftclient:REQ: curl -i http://ceph1/swift/v1/AUTH_adsfasdfasdfasdfasdfasdf/test3 -X POST -H "X-Auth-Token: <Token would be here>" -H "Content-Length: 0" INFO:swiftclient:RESP STATUS: 401 Unauthorized and finally I get Container POST failed: http://ceph1/swift/v1/AUTH_adsfasdfasdfasdfasdfasdf/test3 401 Unauthorized b'AccessDenied'

4 12

OSDs in pool full : can't restart to clean
by Paul Mezzanini 13 Jan '21

13 Jan '21

Hey all We landed in a bad place (tm) with our nvme metadata tier. I'll root cause how we got here after it's all back up. I suspect it was a pool got misconfigured and just filled it all up. Short version, the OSDs are all full (or full enough) that I can't get them to spin back up. They crash with enospc. Average fragmentation for block is in the .8 range and bluefs-db is slightly better (using ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-412 free-score). I've tried all sorts of things. I was able to get a few to spin up but once they came up and rejoined they tried to pull MORE data in and crashed out again. I changed the crush_rule for the pool I care about to a much larger (and slower) set of disks. That way if I get anything else to come up I'm not just making it worse. I increased the size of the backing LV for one of the OSDs to see if I could get ceph-bluestore-tool to expand it, but that too crashes out enospc. In theory, there are a few pools I don't care about as much on there and I could delete them to make space, but I can't get them up enough -or- get the offline tools to do it. Some logs from the attempted expansion that fails: [root@ceph-b-07 ceph-412]# ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-412 bluefs-bdev-expand inferring bluefs devices from bluestore path 1 : device size 0x44aa000000 : own 0x[520000~20000,23e0000~620000,2ae0000~4d20000,78d0000~f30000,8900000~1600000,9fc0000~30000,a000000~5d00000,fe00000~3b00000,139e0000~5420000,19000000~100000, ::snip:: 4f0000~20000,25c17c0000~10000,25c2ea0000~20000,25c9f20000~10000,25d0860000~10000,25d50e0000~20000,25d5170000~10000,25ded20000~20000,25f4fc0000~20000] = 0x59c5b0000 : using 0x58f220000(22 GiB) : bluestore has 0x10260000(258 MiB) available Expanding DB/WAL... Expanding Main... 2021-01-13 16:40:46.481 7f33d1998ec0 -1 bluestore(/var/lib/ceph/osd/ceph-412) allocate_bluefs_freespace failed to allocate on 0x32c70000 min_size 0xf700000 > allocated total 0x1e80000 bluefs_shared_alloc_size 0x10000 allocated 0x1e80000 available 0x 90210000 2021-01-13 16:40:46.482 7f33d1998ec0 -1 bluefs _allocate failed to expand slow device to fit +0xf6f0def 2021-01-13 16:40:46.482 7f33d1998ec0 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: 0xf6f0def /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.15/rpm/el7/BUILD/ceph-14.2.15/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7f33d1998ec0 time 2021-01-13 16:40:46.482978 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.15/rpm/el7/BUILD/ceph-14.2.15/src/os/bluestore/BlueFS.cc: 2351: ceph_abort_msg("bluefs enospc") ceph version 14.2.15 (afdd217ae5fb1ed3f60e16bd62357ca58cc650e5) nautilus (stable) The orignal LV under that is 172g and the new LV size is double that. I'm going to keep poking at this, but I'm really hoping for some new info. Either to increase the size of the OSDs to get it back up enough so I can then rebuild them with a different layout, delete some data I don't care about, pull the data off and put it back to defrag... I don't care which so long as I get it back up. Thanks -paul

1 1

Which version of Ceph fully supports CephFS Snapshot?
by fantastic2085 13 Jan '21

13 Jan '21

I would like to use the Cephfs Snapshot feature, which version of Ceph supports it

3 2

Question about osdmap
by Andrea Bolzonella 12 Jan '21

12 Jan '21

Hi all. I'm running version 14.2.16. My cluster is remapping and backfilling after "reweight-by-utilization". This morning I started to have some issues: - all my mgr active and standby crashed - MDS stopped reporting statistics - MON is using lot of memory - Cluster was very slow I restarted everything but osd I managed to get the statistics back, restarting the MDS a couple of times. All above issues are gone but I have this "mon.node1 (mon.0) 1529395 : cluster [DBG] osdmap e??????:" about 1 every 2 second with a different epoc. Question: Is it normal or there is something wrong? Should be the osdmap more "stable" ? What can change the osdmap? Thanks for the help Andrea

1 0

Re: Global AVAIL vs Pool MAX AVAIL
by Mark Johnson 12 Jan '21

12 Jan '21

Thanks Anthony, Shortly after I made that post, I found a Server Fault post where someone had asked the exact same question. The reply was this - "The 'MAX AVAIL' column represents the amount of data that can be used before the first OSD becomes full. It takes into account the projected distribution of data across disks from the CRUSH map and uses the 'first OSD to fill up' as the target." To answer your question, yes we have a rather unbalanced cluster which is something I'm working on. When I saw these figures, I got scared that I had less time to work on it than I thought. There are about 10 pools in the cluster, but we primarily use one for almost all of our storage and it only has 64 pgs & 1 replica across 20 OSDs. So, as data has grown, it works out that each PG in this cluster accounts for about 148GB, and the OSDs are about 1.4TB each, so it's easy to see how it's found itself way out of balance. Anyway, once I've added the OSDs and data has rebalanced, I'm going to start the process of incrementally increasing the PG count for this pool in a staged process to reduce the amount of data per PG and (hopefully) balance out the data distribution better than it is. This is one big learning process - I just wish I wasn't learning in production so much. On Mon, 2021-01-11 at 15:58 -0800, Anthony D'Atri wrote: Either you have multiple CRUSH roots or device classes, or you have unbalanced OSD utilization. What version of Ceph? Do you have any balancing enabled? Do ceph osd df | sort -nk8 | head ceph osd df | sort -nk8 | tail and I’ll bet you have OSDs way more full than others. The STDDEV value that ceph df reports I suspect is accordingly high On Jan 11, 2021, at 2:07 PM, Mark Johnson < <mailto:markj@iovox.com> markj(a)iovox.com > wrote: Can someone please explain to me the difference between the Global "AVAIL" and the "MAX AVAIL" in the pools table when I do a "ceph df detail"? The reason being that we have a total of 14 pools, however almost all of our data exists in one pool. A "ceph df detail" shows the following: GLOBAL: SIZE AVAIL RAW USED %RAW USED OBJECTS 28219G 6840G 19945G 70.68 36112k But the POOLS table from the same output shows the MAX AVAIL for each pool as 498G and the pool with all the data shows 9472G used with a %USED of 95.00. If it matters, the pool size is set to 2 so my guess is the global available figure is raw, meaning I should still have approx. 3.4TB available, but that 95% used has me concerned. I'm going to be adding some OSDs soon but still would like to understand the difference and how much trouble I'm in at this point. _______________________________________________ ceph-users mailing list -- <mailto:ceph-users@ceph.io> ceph-users(a)ceph.io To unsubscribe send an email to <mailto:ceph-users-leave@ceph.io> ceph-users-leave(a)ceph.io

2 1

Unable to cancel buckets from resharding queue
by Wout van Heeswijk 12 Jan '21

12 Jan '21

We are experiencing some issues with the bucket resharding queue in Ceph Mimic at one of our customers. I suspect that some of the issues are related to the upgrades of earlier versions of the cluster/radosgw. 1) When we cancel the resharding of a bucket, the bucket resharding entry is removed from the queue and almost immediately re-added. We confirm the removal by listing the omapkeys of all the reshard.0000## objects. The relevant omap key is temporarily removed. After a short time it is re-added. I haven't yet determined which process adds it back, but I can only think it is one of the two rgws. 2) We see a lot of objects,1265, in the reshard_pool: "default.rgw.log:reshard". Most look like they might be old to buckets markers or something like that. Most of these objects have not been touched in a long time (mtime 2018). Some numbers: Total buckets: 569 Objects in index pool: 649 Objects in default.rgw.log:reshard namespace: 1265 (of which 16 are the ‘rgw_reshard_num_logs’ objects) these are size '0' Buckets in reshard queue: 41 The objects in "default.rgw.log:reshard" have a similar naming schema to the index objects or , but I cannot relate them directly. Obviously there are loads more than the bucket index objects. The pools used for RGW are for an older generation of Radosgw: .rgw.root default.rgw.control default.rgw.data.root default.rgw.gc default.rgw.log default.rgw.users.uid default.rgw.users.keys default.rgw.buckets.index default.rgw.buckets.data default.rgw.users.email default.rgw.buckets.non-ec default.rgw.users.swift default.rgw.usage My two main questions are: 1) What process, other than dynamic resharding, could cause the re-adding of these buckets to the resharding queue? 2) Do more people see lots of objects in the reshard pool/namespace and can somebody help me understand what these object are? If somebody can point me in the direction of some more documentation or a good talk regarding the resharding mechanism that would also be great. Thanks and with kind regards, Wout 42on

1 0

Global AVAIL vs Pool MAX AVAIL
by Mark Johnson 11 Jan '21

11 Jan '21

Can someone please explain to me the difference between the Global "AVAIL" and the "MAX AVAIL" in the pools table when I do a "ceph df detail"? The reason being that we have a total of 14 pools, however almost all of our data exists in one pool. A "ceph df detail" shows the following: GLOBAL: SIZE AVAIL RAW USED %RAW USED OBJECTS 28219G 6840G 19945G 70.68 36112k But the POOLS table from the same output shows the MAX AVAIL for each pool as 498G and the pool with all the data shows 9472G used with a %USED of 95.00. If it matters, the pool size is set to 2 so my guess is the global available figure is raw, meaning I should still have approx. 3.4TB available, but that 95% used has me concerned. I'm going to be adding some OSDs soon but still would like to understand the difference and how much trouble I'm in at this point.

1 0