- ceph-users - lists.ceph.io

Mysterious Space-Eating Monster
by duluxoz 07 May '24

07 May '24

Hi All, *Something* is chewing up a lot of space on our `\var` partition to the point where we're getting warnings about the Ceph monitor running out of space (ie > 70% full). I've been looking, but I can't find anything significant (ie log files aren't too big, etc) BUT there seem to be a hell of a lot (15) of sub-directories (with GUIDs for names) under the `/var/lib/containers/storage/overlay/` folder, all ending with `merged` - ie `/var/lib/containers/storage/overlay/{{GUID}}/`merged`. Is this normal, or is something going wrong somewhere, or am I looking in the wrong place? Also, if this is the issue, can I delete these folders? Sorry for asking such a noob Q, but the Cephadm/Podman stuff is extremely new to me :-) Thanks in advance Cheers Dulux-Oz

3 3

MDS 17.2.7 crashes at rejoin
by Robert Sander 07 May '24

07 May '24

Hi, a 17.2.7 cluster with two filesystems has suddenly non-working MDSs: # ceph -s cluster: id: f54eea86-265a-11eb-a5d0-457857ba5742 health: HEALTH_ERR 22 failed cephadm daemon(s) 2 filesystems are degraded 1 mds daemon damaged insufficient standby MDS daemons available services: mon: 5 daemons, quorum ceph00,ceph03,ceph04,ceph01,ceph02 (age 4h) mgr: ceph03.odfupq(active, since 4h), standbys: ppc721.vsincn, ceph00.lvbddp, ceph02.zhyxjg, ceph06.eifppc mds: 4/5 daemons up osd: 145 osds: 145 up (since 20h), 145 in (since 2d) rgw: 12 daemons active (4 hosts, 1 zones) data: volumes: 0/2 healthy, 2 recovering; 1 damaged pools: 15 pools, 4897 pgs objects: 195.64M objects, 195 TiB usage: 617 TiB used, 527 TiB / 1.1 PiB avail pgs: 4892 active+clean 5 active+clean+scrubbing+deep io: client: 2.5 MiB/s rd, 20 MiB/s wr, 665 op/s rd, 938 op/s wr # ceph fs status ABC - 4 clients =========== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 failed 1 resolve ABC.ceph04.lzlkdu 0 3 1 0 2 resolve ABC.ppc721.rzfmyi 0 3 1 0 3 resolve ABC.ceph04.jiepaw 249 252 13 0 POOL TYPE USED AVAIL cephfs.ABC.meta metadata 33.0G 104T cephfs.ABC.data data 390T 104T DEF - 154 clients =========== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 rejoin(laggy) DEF.ceph06.etthum 30.9k 30.8k 5084 0 POOL TYPE USED AVAIL cephfs.DEF.meta metadata 190G 104T cephfs.DEF.data data 118T 104T MDS version: ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) The first filesystem will not get an MDS in rank 0, we already tried to set max_msd to 1 but to no avail. The second filesystem's MDS shows "replay" for a while and then it crashes in the rejoin phase with: -92> 2024-05-06T16:07:15.514+0000 7f1927e9d700 1 mds.0.501522 handle_mds_map i am now mds.0.501522 -91> 2024-05-06T16:07:15.514+0000 7f1927e9d700 1 mds.0.501522 handle_mds_map state change up:reconnect --> up:rejoin -90> 2024-05-06T16:07:15.514+0000 7f1927e9d700 1 mds.0.501522 rejoin_start -89> 2024-05-06T16:07:15.514+0000 7f1927e9d700 1 mds.0.501522 rejoin_joint_start -88> 2024-05-06T16:07:15.514+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x300005bfece err -22/0 -87> 2024-05-06T16:07:15.514+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x30000671eb5 err -22/0 -86> 2024-05-06T16:07:15.514+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x300005bfed3 err -22/0 -85> 2024-05-06T16:07:15.514+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc94c err -22/0 -84> 2024-05-06T16:07:15.514+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x300005b0274 err -22/0 -83> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x30000671eb5 err -22/0 -82> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc94c err -22/0 -81> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x30000671ebd err -22/-22 -80> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x30000671ecd err -22/-22 -79> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc9ea err -22/-22 -78> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x300005bfed3 err -22/0 -77> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc9c3 err -22/-22 -76> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc978 err -22/-22 -75> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc99d err -22/-22 -74> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc95b err -22/-22 -73> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc980 err -22/-22 -72> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x300005b0274 err -22/0 -71> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x20001dc7a7e err -22/-22 -70> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012be364 err -22/-22 -69> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x300005b2e32 err -22/-22 -68> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x30000671eb5 err -22/0 -67> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc94c err -22/0 -66> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x30000671ebd err -22/-22 -65> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x30000671ecd err -22/-22 -64> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc9ea err -22/-22 -63> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc978 err -22/-22 -62> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x3000069373a err -22/-22 -61> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012dc5d8 err -22/-22 -60> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x400000a32e8 err -22/-22 -59> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x30000696952 err -22/-22 -58> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc94c err -22/0 -57> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x300005bfed3 err -22/0 -56> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc99d err -22/-22 -55> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc980 err -22/-22 -54> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x20001dc7a7e err -22/-22 -53> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x400000a58cf err -22/-22 -52> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012c3a0e err -22/-22 -51> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc94c err -22/0 -50> 2024-05-06T16:07:15.518+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x300005b0274 err -22/0 -49> 2024-05-06T16:07:15.522+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x20001dc7a7f err -22/-22 -48> 2024-05-06T16:07:15.522+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc95b err -22/-22 -47> 2024-05-06T16:07:15.522+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x300005b2e32 err -22/-22 -46> 2024-05-06T16:07:15.522+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012be364 err -22/-22 -45> 2024-05-06T16:07:15.522+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x300005b388d err -22/-22 -44> 2024-05-06T16:07:15.522+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x10007185ac2 err -22/-22 -43> 2024-05-06T16:07:15.522+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc94c err -22/0 -42> 2024-05-06T16:07:15.522+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc980 err -22/-22 -41> 2024-05-06T16:07:15.522+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc99d err -22/-22 -40> 2024-05-06T16:07:15.522+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x400000a58cf err -22/-22 -39> 2024-05-06T16:07:15.522+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012c3a0e err -22/-22 -38> 2024-05-06T16:07:15.522+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x400000a58db err -22/-22 -37> 2024-05-06T16:07:15.522+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x40000d63bff err -22/-22 -36> 2024-05-06T16:07:15.522+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc94c err -22/0 -35> 2024-05-06T16:07:15.526+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc95b err -22/-22 -34> 2024-05-06T16:07:15.526+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012be364 err -22/-22 -33> 2024-05-06T16:07:15.526+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x10007185ac2 err -22/-22 -32> 2024-05-06T16:07:15.526+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc94c err -22/0 -31> 2024-05-06T16:07:15.526+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x10007185ac4 err -22/-22 -30> 2024-05-06T16:07:15.526+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc980 err -22/-22 -29> 2024-05-06T16:07:15.526+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x400000a58cf err -22/-22 -28> 2024-05-06T16:07:15.526+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x400000a58db err -22/-22 -27> 2024-05-06T16:07:15.526+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x400000a58f4 err -22/-22 -26> 2024-05-06T16:07:15.530+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc94c err -22/0 -25> 2024-05-06T16:07:15.534+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc980 err -22/-22 -24> 2024-05-06T16:07:15.534+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x400000a58cf err -22/-22 -23> 2024-05-06T16:07:15.534+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x400000a58db err -22/-22 -22> 2024-05-06T16:07:15.534+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x400000a58f4 err -22/-22 -21> 2024-05-06T16:07:15.534+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x400000a5fc4 err -22/-22 -20> 2024-05-06T16:07:15.534+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x400000a634d err -22/-22 -19> 2024-05-06T16:07:15.534+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x400000a63bf err -22/-22 -18> 2024-05-06T16:07:15.542+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc94c err -22/0 -17> 2024-05-06T16:07:15.542+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bc980 err -22/-22 -16> 2024-05-06T16:07:15.546+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x400000a58cf err -22/-22 -15> 2024-05-06T16:07:15.550+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x400000a58db err -22/-22 -14> 2024-05-06T16:07:15.550+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x400000a58f4 err -22/-22 -13> 2024-05-06T16:07:15.550+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x400000a5fc4 err -22/-22 -12> 2024-05-06T16:07:15.550+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x400000a634d err -22/-22 -11> 2024-05-06T16:07:15.550+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x400000a63bf err -22/-22 -10> 2024-05-06T16:07:15.550+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bfd9c err -22/-22 -9> 2024-05-06T16:07:15.550+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x200012bfb78 err -22/-22 -8> 2024-05-06T16:07:15.554+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x300005b0274 err -22/0 -7> 2024-05-06T16:07:15.554+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x300005b2e32 err -22/-22 -6> 2024-05-06T16:07:15.554+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x300005b388d err -22/-22 -5> 2024-05-06T16:07:15.562+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x300005b0274 err -22/0 -4> 2024-05-06T16:07:15.562+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x300005b2e32 err -22/-22 -3> 2024-05-06T16:07:15.562+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x300005b388d err -22/-22 -2> 2024-05-06T16:07:15.562+0000 7f1921e91700 0 mds.0.cache failed to open ino 0x40000d5a226 err -22/-22 -1> 2024-05-06T16:07:15.634+0000 7f1921e91700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.7/rpm/el8/BUILD/ceph-17.2.7/src/mds/MDCache.cc: In function 'void MDCache::rejoin_send_rejoins()' thread 7f1921e91700 time 2024-05-06T16:07:15.635683+0000 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.7/rpm/el8/BUILD/ceph-17.2.7/src/mds/MDCache.cc: 4086: FAILED ceph_assert(auth >= 0) ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4c060d2) quincy (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f1930ad94a3] 2: /usr/lib64/ceph/libceph-common.so.2(+0x269669) [0x7f1930ad9669] 3: (MDCache::rejoin_send_rejoins()+0x216b) [0x5614ac8747eb] 4: (MDCache::process_imported_caps()+0x1993) [0x5614ac872353] 5: (Context::complete(int)+0xd) [0x5614ac6e182d] 6: (MDSContext::complete(int)+0x5f) [0x5614aca41f4f] 7: (void finish_contexts<std::vector<MDSContext*, std::allocator<MDSContext*> > >(ceph::common::CephContext*, std::vector<MDSContext*, std::allocator<MDSContext*> >&, int)+0x8d) [0x5614ac6e6f5d] 8: (OpenFileTable::_open_ino_finish(inodeno_t, int)+0x156) [0x5614aca765a6] 9: (MDSContext::complete(int)+0x5f) [0x5614aca41f4f] 10: (void finish_contexts<std::vector<MDSContext*, std::allocator<MDSContext*> > >(ceph::common::CephContext*, std::vector<MDSContext*, std::allocator<MDSContext*> >&, int)+0x8d) [0x5614ac6e6f5d] 11: (MDCache::open_ino_finish(inodeno_t, MDCache::open_ino_info_t&, int)+0x138) [0x5614ac867168] 12: (MDCache::_open_ino_backtrace_fetched(inodeno_t, ceph::buffer::v15_2_0::list&, int)+0x290) [0x5614ac87ff90] 13: (MDSContext::complete(int)+0x5f) [0x5614aca41f4f] 14: (MDSIOContextBase::complete(int)+0x534) [0x5614aca426e4] 15: (Finisher::finisher_thread_entry()+0x18d) [0x7f1930b7884d] 16: /lib64/libpthread.so.0(+0x81ca) [0x7f192fac81ca] 17: clone() How do we solve this issue? Regards -- Robert Sander Heinlein Support GmbH Linux: Akademie - Support - Hosting http://www.heinlein-support.de Tel: 030-405051-43 Fax: 030-405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

2 3

MDS crashes shortly after starting
by E Taka 06 May '24

06 May '24

Hi all, we have a serious problem with CephFS. A few days ago, the CephFS file systems became inaccessible, with the message MDS_DAMAGE: 1 mds daemon damaged The cephfs-journal-tool tells us: "Overall journal integrity: OK" The usual attempts with redeploy were unfortunately not successful. After many attempts to achieve something with the orchestrator, we set the MDS to “failed” and provoked the creation of new MDS with “ceph fs reset”. But this MDS crashes: ceph-17.2.7/src/mds/MDCache.cc: In function 'void MDCache::rejoin_send_rejoins()' ceph-17.2.7/src/mds/MDCache.cc: 4086: FAILED ceph_assert(auth >= 0) (The full trace is attached). What can we do now? We are grateful for any help!

2 1

Luminous OSDs failing with FAILED assert(clone_size.count(clone))
by sergio.rabellino＠unito.it 06 May '24

06 May '24

Dear Ceph users, I'm pretty new on this list, but I've been using Ceph with satisfaction since 2020. I faced some problems through these years consulting the list archive, but now we're down with a problem that seems without an answer. After a power failure, we have a bunch of OSDs that during rebalance/refilling goes down with this error: /build/ceph-OM2K9O/ceph-13.2.9/src/osd/osd_types.cc: In function 'uint64_t SnapSet::get_clone_bytes(snapid_t) const' thread 7fdcb2523700 time 2024-05-02 17:18:40.680350 /build/ceph-OM2K9O/ceph-13.2.9/src/osd/osd_types.cc: 5084: FAILED assert(clone_size.count(clone)) ceph version 13.2.9 (58a2a9b31fd08d8bb3089fce0e312331502ff945) mimic (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14e) [0x7fdcd38f63ee] 2: (()+0x287577) [0x7fdcd38f6577] 3: (SnapSet::get_clone_bytes(snapid_t) const+0x125) [0x555e697c2725] 4: (PrimaryLogPG::add_object_context_to_pg_stat(std::shared_ptr<ObjectContext>, pg_stat_t*)+0x2c8) [0x555e696d8208] 5: (PrimaryLogPG::recover_backfill(unsigned long, ThreadPool::TPHandle&, bool*)+0x1169) [0x555e6973f749] 6: (PrimaryLogPG::start_recovery_ops(unsigned long, ThreadPool::TPHandle&, unsigned long*)+0x1018) [0x555e69743b98] 7: (OSD::do_recovery(PG*, unsigned int, unsigned long, ThreadPool::TPHandle&)+0x36a) [0x555e695b07da] 8: (PGRecovery::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x19) [0x555e69813c99] 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x52d) [0x555e695b220d] 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x476) [0x7fdcd38fc516] 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x7fdcd38fd6d0] 12: (()+0x76db) [0x7fdcd23e46db] 13: (clone()+0x3f) [0x7fdcd13ad61f] -6171> 2024-05-02 17:18:40.680 7fdcb2523700 -1 *** Caught signal (Aborted) ** in thread 7fdcb2523700 thread_name:tp_osd_tp And we're unable to understand what's happening. Yes, actually we're in Luminous but we planned to upgrade to Pacific in June, but before upgrading I believe it's important to have a positive health check. The pools in error are EC pools. Some hints ?

2 1

CLT meeting notes May 6th 2024
by Adam King 06 May '24

06 May '24

- DigitalOcean credits - things to ask - what would promotional material require - how much are credits worth - Neha to ask - 19.1.0 centos9 container status - close to being ready - will be building centos 8 and 9 containers simultaneously - should test on orch and upgrade suites before publishing RC - should first RC be tested on LRC? - skip first RC on LRC - performance differences to 18.2.2 with cephfs being investigated - 18.2.3 - fix for https://tracker.ceph.com/issues/65733 almost ready - will upgrade LRC to version with fix - will happen before 19.1.0 most likely

1 0

Off-Site monitor node over VPN
by Stefan Pinter 06 May '24

06 May '24

Hi! i hope someone can help us out here :) We need to move from 3 datacenters to 2 datacenters (+ 1 small serverroom reachable via layer 3 VPN) NOW we have a ceph-mon in each datacenter, which is fine. But we have to move and will only have 2 datacenters in the future (that are connected, so devices that are separated geographically can still communicate over their VLANs that they are used to have) in order to still have 3 - to continue being quorate - we have the idea to only move 1 ceph-mon into a small separate datacenter. we only are able to connect it via VPN though and we'd need a separate IP network and the traffic would be over a Firewall and over VPN. having a separate IP network should work, as mentioned here: 2.1.1.2<https://2.1.1.2/> Monitoring nodes on different subnets https://documentation.suse.com/ses/7/html/ses-all/storage-bp-hwreq.html what could possibly go wrong here? can the added latency be a problem? which devices would need to be allowed to connect over VPN? All OSDs < - vpn - > ceph-mon thank you for any input! Kind regards Stefan ________________________________ BearingPoint GmbH Sitz: Wien Firmenbuchgericht: Handelsgericht Wien Firmenbuchnummer: FN 175524z The information in this email is confidential and may be legally privileged. If you are not the intended recipient of this message, any review, disclosure, copying, distribution, retention, or any action taken or omitted to be taken in reliance on it is prohibited and may be unlawful. If you are not the intended recipient, please reply to or forward a copy of this message to the sender and delete the message, any attachments, and any copies thereof from your system.

1 0

radosgw sync non-existent bucket ceph reef 18.2.2
by Christopher Durham 06 May '24

06 May '24

Hi, I have a reef cluster 18.2.2 on Rocky 8.9. This cluster has been upgraded from pacific->quincy->reef over the past few years. It is a multi site with one other cluster that works fine with s3/radosgw on both sides, with proper bidirectional data replication. On one of the master cluster's radosgw logs, I noticed a sync request regarding a deleted bucket. I am not sure when this error started, but I know that the bucket in question was deleted a long time before the upgrade to reef. Perhapsthis error existed prior to reef, I do not know. Here is the error in the radosgw log: :get_bucket_index_log_status ERROR: rgw_read_bucket_full_sync_status() on pipe{s={b=BUCKET_NAME:CLUSTERID ..., z=...., az= ...},d={b=..,az=...}} returned ret=-2 My understanding: s=source, d=destination, each of which is a tuple with the appropriate info necessary.... This happens for BUCKET_NAME every few minutes. Said bucket does not exist on either side of the multisite, but did in the past. Any way I can force radosgw to stop trying to replicate? Thanks -Chris

2 1

Unable to add new OSDs
by ceph＠mikesoffice.com 06 May '24

06 May '24

I'm trying to add a new storage host into a Ceph cluster (quincy 17.2.6). The machine has boot drives, one free SSD and 10 HDDs. The plan is to have each HDD be an OSD with a DB on a equal size lvm of the SDD. This machine is newer but otherwise similar to other machines already in the cluster that are setup and running the same way. But I've been unable to add OSDs and unable to figure out why, or fix it. I have some experience, but I'm not an expert and could be missing something obvious. If anyone has any suggestions, I would appreciate it. I've tried to add OSDs a couple different ways. Via the dashboard, this has worked fine for previous machines. And it appears to succeed and gives no errors that I can find looking in /var/log/ceph and dashboard logs. The OSDs are never created. In fact, the drives still show up as available in Physical Disks and I can do the same creation procedure repeatedly. I've tried creating it in cephadm shell with the following, which has also worked in the past: ceph orch daemon add osd stor04.fqdn:data_devices=/dev/sdb,/dev/sdc,/dev/sdd,/dev/sde,/dev/sdf,/dev/sdg,/dev/sdh,/dev/sdi,/dev/sdj,/dev/sdk,db_devices=/dev/sda,osds_per_device=1 The command just hangs. Again I wasn't able to find any obvious errors. Although this one did seem to cause some slow op errors from the monitors that required restarting a monitor. And it could cause errors with the dashboard locking up and having to restart the manager as well. And I've tried setting 'ceph orch apply osd --all-available-devices --unmanaged=false' to let Ceph automatically add the drives. In the past, this would cause Ceph to automatically add the drives as OSDs but without having associated DBs on the SSD. The SSD would just be another OSD. This time it appears to have no affect and similar to the above, I wasn't able to find any obvious error feedback. -Mike

4 3

Have a problem with haproxy/keepalived/ganesha/docker
by Ruslan Nurabayev 06 May '24

06 May '24

Hello! I've installed my 5-node CEPH cluster next install NFS server by command: ceph nfs cluster create nfshacluster 5 --ingress --virtual_ip 192.168.171.48/26 --ingress-mode haproxy-protocol. I don't understand fully how this must be works but when i stop NFS daemon even on one of this nodes I've see that writing on NFS shares is disappear (testing via vdbench). As i understand it is wrong and IO from stopped daemon must switching to another NFS daemon without any impact on IO. Can someone help me with troubleshoot this issue? Or explain how done full-fledged Active-Active HA NFS Cluster for production use. Thanks! Руслан Нурабаев Старший инженер Сектор ИТ платформы Отдел развития опорной сети Департамент развития сети +77012119272 Ruslan.Nurabayev(a)kcell.kz -----Original Message----- From: ceph-users-request(a)ceph.io <ceph-users-request(a)ceph.io> Sent: Thursday, April 11, 2024 15:07 To: Ruslan Nurabayev <Ruslan.Nurabayev(a)kcell.kz> Subject: Welcome to the "ceph-users" mailing list [You don't often get email from ceph-users-request(a)ceph.io. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] Welcome to the "ceph-users" mailing list! To post to this list, send your email to: ceph-users(a)ceph.io You can unsubscribe or make adjustments to your options via email by sending a message to: ceph-users-request(a)ceph.io with the word 'help' in the subject or body (don't include the quotes), and you will get back a message with instructions. You will need your password to change your options, but for security purposes, this password is not included here. If you have forgotten your password you will need to reset it via the web UI. ________________________________ **************************************************************************************** Осы хабарлама және онымен берілетін кез келген файлдар құпия болып табылады және олар мекенжайда көрсетілген жеке немесе заңды тұлғалардың пайдалануына ғана арналған. Егер сіз болжамды алушы болып табылмайтын болсаңыз, осы арқылы осындай ақпаратты кез келген таратуға, жіберуге, көшіруге немесе пайдалануға қатаң тыйым салынатыны және осы электрондық хабарлама дереу жойылуға тиіс екендігін хабарлаймыз. KCELL осы хабарламадағы кез келген ақпараттың дәлдігіне немесе толықтығына қатысты ешқандай кепілдік бермейді және сол арқылы онда қамтылған ақпарат үшін немесе оны беру, қабылдау, сақтау немесе қандай да бір түрде пайдалану үшін кез келген жауапкершілікті болдырмайды. Осы хабарламада айтылған пікірлер тек жіберушіге ғана тиесілі және KCELL пікірін де білдіруі міндетті емес. Бұл электрондық хабарлама барлық танымал компьютерлік вирустарға тексерілді. **************************************************************************************** Данное сообщение и любые передаваемые с ним файлы являются конфиденциальными и предназначены исключительно для использования физическими или юридическими лицами, которым они адресованы. Если вы не являетесь предполагаемым получателем, настоящим уведомляем о том, что любое распространение, пересылка, копирование или использование такой информации строго запрещено, и данное электронное сообщение должно быть немедленно удалено. KCELL не дает никаких гарантий относительно точности или полноты любой информации, содержащейся в данном сообщении, и тем самым исключает любую ответственность за информацию, содержащуюся в нем, или за ее передачу, прием, хранение или использование каким-либо образом. Мнения, выраженные в данном сообщении, принадлежат только отправителю и не обязательно отражают мнение KCELL. Данное электронное сообщение было проверено на наличие всех известных компьютерных вирусов. **************************************************************************************** This e-mail and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you are not the intended recipient you are hereby notified that any dissemination, forwarding, copying or use of any of the information is strictly prohibited, and the e-mail should immediately be deleted. KCELL makes no warranty as to the accuracy or completeness of any information contained in this message and hereby excludes any liability of any kind for the information contained therein or for the information transmission, reception, storage or use of such in any way whatsoever. The opinions expressed in this message belong to sender alone and may not necessarily reflect the opinions of KCELL. This e-mail has been scanned for all known computer viruses. ****************************************************************************************

6 8

Remove failed OSD
by Zakhar Kirpichenko 04 May '24

04 May '24

Hi! An OSD failed in our 16.2.15 cluster. I prepared it for removal and ran `ceph orch daemon rm osd.19 --force`. Somehow that didn't work as expected, so now we still have osd.19 in the crush map: -10 122.66965 host ceph02 19 1.00000 osd.19 down 0 1.00000 But OSD has been cleaned up on the host, although incompletely, as both block and block.db LVs still exist. If I try to remove the OSD again, I get an error: # ceph orch daemon rm osd.19 --force Error EINVAL: Unable to find daemon(s) ['osd.19'] How can I clean up this OSD and get rid of it completely, including the crush map? I would appreciate any suggestions or pointers. Best regards, Zakhar

2 2