- ceph-users - lists.ceph.io

Ceph dashboard reports CephNodeNetworkPacketErrors
by Dominique Ramaekers 07 Nov '23

07 Nov '23

Hi, I'm using Ceph on a 4-host cluster for a year now. I recently discovered the Ceph Dashboard :-) No I see that the Dashboard reports CephNodeNetworkPacketErrors >0.01% or >10 packets/s... Although all systems work great, I'm worried. 'ip -s link show eno5' results: 2: eno5: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP mode DEFAULT group default qlen 1000 link/ether 7a:3b:79:9c:f6:d1 brd ff:ff:ff:ff:ff:ff permaddr 5c:ba:2c:08:b3:90 RX: bytes packets errors dropped missed mcast 734153938129 645770129 20160 0 0 342301 TX: bytes packets errors dropped carrier collsns 1085134190597 923843839 0 0 0 0 altname enp178s0f0 So in average 0,0003% of RX packet errors! All the four hosts uses the same 10Gb HP switch. The hosts themselves are HP Proliant G10 servers. I would expect 0% packet loss... Anyway. Should I be worried about data consistency? Or can Ceph handle this amount of packet errors? Greetings, Dominique.

2 2

Many pgs inactive after node failure
by Matthew Booth 06 Nov '23

06 Nov '23

I have a 3 node ceph cluster in my home lab. One of the pools spans 3 hdds, one on each node, and has size 2, min size 1. One of my nodes is currently down, and I have 160 pgs in 'unknown' state. The other 2 hosts are up and the cluster has quorum. Example `ceph health detail` output: pg 9.0 is stuck inactive for 25h, current state unknown, last acting [] I have 3 questions: Why would the pgs be in an unknown state? I would like to recover the cluster without recovering the failed node, primarily so that I know I can. Is that possible? The boot nvme of the host has failed, so I will most likely rebuild it. I'm running rook, and I will most likely delete the old node and create a new one with the same name. AFAIK, the OSDs are fine. When rook rediscovers the OSDs, will it add them back with data intact? If not, is there any way I can make it so it will? Thanks! -- Matthew Booth

3 3

Nautilus: Decommission an OSD Node
by Dave Hall 05 Nov '23

05 Nov '23

Hello., I've recently made the decision to gradually decommission my Nautilus cluster and migrate the hardware to a new Pacific or Quincy cluster. By gradually, I mean that as I expand the new cluster I will move (copy/erase) content from the old cluster to the new, making room to decommission more nodes and move them over. In order to do this I will, of course, need to remove OSD nodes by first emptying the OSDs on each node. I noticed that pgremapper (a version prior to October 2021) has a 'drain' subcommand that allows one to control which target OSDs would receive the PGs from the source OSD being drained. This seemed like a good idea: If one simply marks an OSD 'out', it's contents would be rebalanced to other OSDs on the same node that are still active, which seems like it would make a lot of unnecessary data movement and also make removing the next OSD take longer. So I went through the trouble of creating a 'really long' pgremapper drain command excluding the OSDs of two nodes as targets: # bin/pgremapper drain 16 --target-osds 00,01,02,03,04,05,06,07,24,25,16,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71 --allow-movement-across host --max-source-backfills 75 --concurrency 20 --verbose --yes However, when this is complete OSD 16 actually contains more PGs than before I started. It appears that the mapping generated by pgremapper also back-filled the OSD as it was draining it. So did I miss something here? What is the best way to proceed? I understand that it would be mayhem to mark 8 of 72 OSDs out and then turn backfill/rebalance/recover back on. But it seems like there should be a better way. Suggestions? Thanks. -Dave -- Dave Hall Binghamton University kdhall(a)binghamton.edu

2 1

OSD not starting
by Amudhan P 05 Nov '23

05 Nov '23

Hi, One of the server in Ceph cluster accidentally shutdown abruptly due to power failure. After restarting OSD's not coming up and in Ceph health check it shows osd down. When checking OSD status "osd.26 18865 unable to obtain rotating service keys; retrying" For every 30 seconds it's just putting a message and it's all the same in all OSD in the system. Nov 04 20:03:05 strg-node-03 bash[34287]: debug 2023-11-04T14:33:05.089+0000 7f1f5693c080 -1 osd.26 18865 unable to obtain rotating service keys; retrying Nov 04 20:03:35 strg-node-03 bash[34287]: debug 2023-11-04T14:33:35.090+0000 7f1f5693c080 -1 osd.26 18865 unable to obtain rotating service keys; retrying ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable) on Debian 11 bullseye and cephadm based installation. Tried to search for errors and msg couldn't find anything useful. How do I fix this issue ? regards, Amudhan

2 2

RGW: Quincy 17.2.7 and rgw_crypt_default_encryption_key
by Jayanth Reddy 04 Nov '23

04 Nov '23

Hello Users, It is great to hear a note about RGW "S3 multipart uploads using Server-Side Encryption now replicate correctly in multi-site" in Quincy v17.2.7 release. But I see that users who are using [1] still have a dependency on the item tracked at [2]. I tested with Reef 18.2.0 as well and the PR [3] seems to be merged into reef but the configuration is not being effective. We're currently stuck at version 17.2.3 and in a situation where we can not upgrade to later because "rgw_crypt_default_encryption_key" is still WIP and also MPU with SSE (default key) is only fixed in 17.2.7 :( [1] https://docs.ceph.com/en/quincy/radosgw/encryption/#automatic-encryption-fo… [2] https://tracker.ceph.com/issues/61473 [3] https://github.com/ceph/ceph/pull/52796 Appreciate any help. Regards, Jayanth

1 0

resharding RocksDB after upgrade to Pacific breaks OSDs
by Denis Polom 03 Nov '23

03 Nov '23

Hi we upgraded our Ceph cluster from latest Octopus to Pacific 16.2.14 and then we followed the docs (https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#r… <https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#r…>) to reshard RocksDB on our OSDs. Despite resharding reports operation as successful, OSD fails to start. # ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-5/ --sharding="m(3) p(3,0-12) o(3,0-13)=block_cache={type=binned_lru} l p" reshard reshard success Oct 30 12:44:17 octopus2 ceph-osd[4521]: /build/ceph-16.2.14/src/kv/RocksDBStore.cc: 1223: FAILED ceph_assert(recreate_mode) Oct 30 12:44:17 octopus2 ceph-osd[4521]: ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable) Oct 30 12:44:17 octopus2 ceph-osd[4521]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14b) [0x564047cb92b2] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 2: /usr/bin/ceph-osd(+0xaa948a) [0x564047cb948a] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 3: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1609) [0x564048794829] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 4: (BlueStore::_open_db(bool, bool, bool)+0x601) [0x564048240421] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 5: (BlueStore::_open_db_and_around(bool, bool)+0x26b) [0x5640482a5f8b] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 6: (BlueStore::_mount()+0x9c) [0x5640482a896c] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 7: (OSD::init()+0x38a) [0x564047daacea] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 8: main() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 9: __libc_start_main() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 10: _start() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 0> 2023-10-30T12:44:17.088+0000 7f4971ed2100 -1 *** Caught signal (Aborted) ** Oct 30 12:44:17 octopus2 ceph-osd[4521]: in thread 7f4971ed2100 thread_name:ceph-osd Oct 30 12:44:17 octopus2 ceph-osd[4521]: ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable) Oct 30 12:44:17 octopus2 ceph-osd[4521]: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12730) [0x7f4972921730] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 2: gsignal() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 3: abort() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x564047cb9303] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 5: /usr/bin/ceph-osd(+0xaa948a) [0x564047cb948a] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 6: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1609) [0x564048794829] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 7: (BlueStore::_open_db(bool, bool, bool)+0x601) [0x564048240421] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 8: (BlueStore::_open_db_and_around(bool, bool)+0x26b) [0x5640482a5f8b] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 9: (BlueStore::_mount()+0x9c) [0x5640482a896c] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 10: (OSD::init()+0x38a) [0x564047daacea] Oct 30 12:44:17 octopus2 ceph-osd[4521]: 11: main() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 12: __libc_start_main() Oct 30 12:44:17 octopus2 ceph-osd[4521]: 13: _start() Oct 30 12:44:17 octopus2 ceph-osd[4521]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Oct 30 12:44:17 octopus2 ceph-osd[4521]: -1> 2023-10-30T12:44:17.084+0000 7f4971ed2100 -1 /build/ceph-16.2.14/src/kv/RocksDBStore.cc: In function 'int RocksDBStore::do_open(std::ostream&, bool, bool, const string&)' thread 7f4971ed2100 time 2023-10-30T12:44:17.087172+0000 I've submitted bug report here https://tracker.ceph.com/issues/63353 but may be community here have some ideas how to fix it unless it's really a bug. Thanks

7 8

ceph orch problem
by Dario Graña 03 Nov '23

03 Nov '23

Hi everyone! I've a ceph cluster running AlmaLinux9, podman and Ceph Quincy (17.6.0). Since yesterday I was having some problems but the last one is that the ceph orch command is hanging up. I have seen the logs but I didn't find anything relevant that could help to fix the problem. Podman shows the daemons running and if I stop one daemon it appears again after a few seconds. I also tried *ceph mgr fail* command, daemon order changes, but ceph orch still not working. ceph orch pause/resume also are not working. Disabling and enabling *cephadm* module didn't help. Any help to understand what's going on would be welcome. Thanks in advance. -- Dario Graña PIC (Port d'Informació Científica) Campus UAB, Edificio D E-08193 Bellaterra, Barcelona http://www.pic.es Avis - Aviso - Legal Notice: http://legal.ifae.es

2 3

data corruption after rbd migration
by Nikola Ciprich 03 Nov '23

03 Nov '23

Dear ceph users and developers, we're struggling with strange issue which I think might be a bug causing snapshot data corruption while migrating RBD image we've tracked it to minimal set of steps to reproduce using VM with one 32G drive: rbd create --size 32768 sata/D2 virsh create xml_orig.xml rbd snap create ssd/D1@snap1 rbd export-diff ssd/D1@snap1 - | rbd import-diff - sata/D2 rbd export --export-format 1 --no-progress ssd/D1@snap1 - | xxh64sum 505dde3c49775773 rbd export --export-format 1 --no-progress sata/D2@snap1 - | xxh64sum 505dde3c49775773 # <- checksums match - OK virsh shutdown VM rbd migration prepare ssd/D1 sata/D1Z virsh create xml_new.xml rbd snap create sata/D1Z@snap2 rbd export-diff --from-snap snap1 sata/D1Z@snap2 - | rbd import-diff - sata/D2 rbd migration execute sata/D1Z rbd migration commit sata/D1Z rbd export --export-format 1 --no-progress sata/D1Z@snap2 - | xxh64sum 19892545c742c1de rbd export --export-format 1 --no-progress sata/D2@snap2 - | xxh64sum cc045975baf74ba8 # <- snapshosts differ OS is alma 9 based, kernel 5.15.105, CEPH 17.2.6, qemu-8.0.3 we tried disabling VM disk caches as well as discard, to no avail. my first question is, is it correct to assume creating snapshots while live migrating data is safe? if so, any ideas on where the problem could be? If I could provide more info, please let me know with regards nikola ciprich -- ------------------------------------- Ing. Nikola CIPRICH LinuxBox.cz, s.r.o. 28.rijna 168, 709 00 Ostrava tel.: +420 591 166 214 fax: +420 596 621 273 mobil: +420 777 093 799 www.linuxbox.cz mobil servis: +420 737 238 656 email servis: servis(a)linuxbox.cz -------------------------------------

2 2

17.2.7 quincy
by Matthew Darwin 03 Nov '23

03 Nov '23

Hi all, I see17.2.7 quincy is published as debian-bullseye packages. So I tried it on a test cluster. I must say I was not expecting the big dashboard change in a patch release. Also all the "cluster utilization" numbers are all blank now (any way to fix it?), so the dashboard is much less usable now. Thoughts?

2 7

upgrade 17.2.6 to 17.2.7 , any issues?
by Dmitry Melekhov 02 Nov '23

02 Nov '23

Hello! I want to do upgrade from 17.2.6 to 17.2.7 , all 3 servers run alma 9. Could you share your experience? Any issues? Thank you!

2 2