- ceph-users - lists.ceph.io

Monitors crash largely due to the structure of pg-upmap-primary
by Michal Strnad 25 Feb '25

25 Feb '25

Hi. We ran into an issue with "pg-upmap-primary", which resulted in our monitors crashing massively (around 1,000 crashes per day). According to [1], it should be possible to remove these pg-upmap-primaries. Unfortunately, we are unable to do so because these PGs, and therefore the pool, no longer exist. root(a)app001.clX ~ # ceph osd dump | grep 'pg_upmap_primary' | grep 24.ffc pg_upmap_primary 24.ffc 232 root(a)app001.clX ~ # ceph osd rm-pg-upmap-primary 24.ffc Error ENOENT: pgid '24.ffc' does not exist root(a)app001.clX ~ # ceph pg dump | grep "^24\." dumped all Is there any way to remove these structures? We also tried upgrading from the current version 18.2.1 to 18.2.4, but this led to a state on our three-node test cluster where one of the three monitors failed to start, along with a third of the OSDs, due to issues with the mentioned structure. Restarting the daemon didn’t help. Does anyone have a solution or an idea? This is becoming quite a problem for us. Below, I am attaching one of the many monitor crash logs. Thank you very much for any advice! Of course, we have also created a ticket in the tracker [https://tracker.ceph.com/issues/69760] where the same information I’m sending in this email is documented. Michal [1] https://tracker.ceph.com/issues/61948#note-32 { "assert_condition": "pg_upmap_primaries.empty()", "assert_file": "/builddir/build/BUILD/ceph-18.2.1/src/osd/OSDMap.cc",https://tracker.ceph.com/issues/69760 "assert_func": "void OSDMap::encode(ceph::buffer::v15_2_0::list&, uint64_t) const", "assert_line": 3239, "assert_msg": "/builddir/build/BUILD/ceph-18.2.1/src/osd/OSDMap.cc: In function 'void OSDMap::encode(ceph::buffer::v15_2_0::list&, uint64_t) const' thread 7f94216a3640 time 2025-02-02T19:16:03.629964+0100\n/builddir/build/BUILD/ceph-18.2.1/src/osd/OSDMap.cc: 3239: FAILED ceph_assert(pg_upmap_primaries.empty())\n", "assert_thread_name": "ms_dispatch", "backtrace": [ "/lib64/libc.so.6(+0x54db0) [0x7f9429054db0]", "/lib64/libc.so.6(+0xa365c) [0x7f94290a365c]", "raise()", "abort()", "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f9429d630df]", "/usr/lib64/ceph/libceph-common.so.2(+0x163243) [0x7f9429d63243]", "/usr/lib64/ceph/libceph-common.so.2(+0x1a0f38) [0x7f9429da0f38]", "(OSDMonitor::reencode_full_map(ceph::buffer::v15_2_0::list&, unsigned long)+0xe2) [0x55ca54957e22]", "(OSDMonitor::get_version_full(unsigned long, unsigned long, ceph::buffer::v15_2_0::list&)+0x1de) [0x55ca549596ae]", "(OSDMonitor::build_latest_full(unsigned long)+0x2a3) [0x55ca549599a3]", "(OSDMonitor::check_osdmap_sub(Subscription*)+0xc8) [0x55ca5495be98]", "(Monitor::handle_subscribe(boost::intrusive_ptr<MonOpRequest>)+0xf04) [0x55ca54834dd4]", "(Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x6a6) [0x55ca548359c6]", "(Monitor::_ms_dispatch(Message*)+0x779) [0x55ca54836d59]", "/usr/bin/ceph-mon(+0x2f3dfe) [0x55ca547f1dfe]", "(DispatchQueue::entry()+0x52a) [0x7f9429f5766a]", "/usr/lib64/ceph/libceph-common.so.2(+0x3e7321) [0x7f9429fe7321]", "/lib64/libc.so.6(+0xa1912) [0x7f94290a1912]", "/lib64/libc.so.6(+0x3f450) [0x7f942903f450]" ], "ceph_version": "18.2.1", "crash_id": "2025-02-02T18:16:03.632571Z_f5516ed0-6df5-4267-bada-71f5d8d764ba", "entity_name": "mon.mon001-clX", "os_id": "centos", "os_name": "CentOS Stream", "os_version": "9", "os_version_id": "9", "process_name": "ceph-mon", "stack_sig": "772ef523b041edc5147d1d9905926fb794d32b2635368a8199f6e2e4f2d688bf", "timestamp": "2025-02-02T18:16:03.632571Z", "utsname_hostname": "app001.clX", "utsname_machine": "x86_64", "utsname_release": "5.14.0-402.el9.x86_64", "utsname_sysname": "Linux", "utsname_version": "#1 SMP PREEMPT_DYNAMIC Thu Dec 21 19:46:35 UTC 2023" }

3 4

CSC Meeting Minutes | 2025-02-24
by Afreen Misbah 25 Feb '25

25 Feb '25

Hi all, Here are the meeting minutes from the 2025-02-24 call. Joseph shared LFX insights on ceph! - Link to the insights <https://insights.lfx.linuxfoundation.org/foundation/cephfoundation/overview…> QE updates (Yuri) - 6 outstanding PRs <https://github.com/ceph/ceph/pulls?q=is%3Aopen+is%3Apr+milestone%3Av18.2.5> - Eric is going to take a look at them and provide inputs - can start release process as soon as the PRs merge Release scheduling (Greg) Discussions from ongoing mail thread Ceph Tentacle release timeline — when? <https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/PNJB3CW43NF…> - People want fast EC - EC work in upstream making progress - Move the technical discussions to mailing list - distro sync is important for a lot of operators due to tooling available-by-default, even if using Ceph upstream packages - containers vs packages: Packages are stable enough - tentacle when fast EC ready, and freeze "U" in January for an April release next year? - Do we need/want to extend our major-version upgrade commitments for the shorter time frames? - Ceph does not give release for an entire year - Need to prepare a plan and see what resources and costs will be needed - Better to go for stability rather than quickness Tentacle Kickoff PR (Patrick) - https://github.com/ceph/ceph/pull/61321 - QA: https://tracker.ceph.com/issues/70000 - Merge soon to avoid rebases Ceph build PR (Patrick) - https://github.com/ceph/ceph-build/pull/2315 - Blocked on Dan Mick ,probably take a look soon Ceph Labs presentation (David) - During Ceph Infra call 2/27 12:30 Eastern - Feel free to join -- Afreen -- Afreen

2 1

[How to mount cephFS on a k8s pod?]
by Baijia Ye 25 Feb '25

25 Feb '25

Hello all, Is it possible to mount cephFS on a k8s pod? I tried but failed. [image: image.png] Thanks for your help! Best, Baijia

1 0

Cephalocon 2025 Sponsorships - Early interest survey
by Dan van der Ster 24 Feb '25

24 Feb '25

Hi all, After a fantastic Cephalocon at CERN last year, and based on the early results of the Ceph Events Survey (thanks to those who responded!) we are aiming to hold the next Cephalocon in North America in the fall of 2025 (Sep/Oct). We need to capture informal (not contractual) commitments from potential sponsors early so we can move ahead with planning the event. Levels will be similar to 2024's prospectus [1], with a possible increase in cost. However, for anyone who responds to this early commitment, we plan to hold the 3 major sponsorships (Platinum, Gold, and Silver) to 2024 pricing! We really appreciate your support in sending a quick response (by EOD Thursday, Feb 27th): https://docs.google.com/forms/d/e/1FAIpQLSfbC1nDP5qlBYMwzEuctsGdCa60mkeVIHc… [1] https://events.linuxfoundation.org/wp-content/uploads/2024/10/Sponsor-Ceph-… -- Dan van der Ster Ceph Executive Council

1 0

Why is the default include_parent value of `export-diff` True , and is it not allowed for users to set it?
by Zacharias Turing 24 Feb '25

24 Feb '25

Hi， Why not allow users to specify include_parent=False when doing export-diff? Wouldn't this save capacity during migration?

1 0

cephfs-mirror and acl
by ronny.lippold 24 Feb '25

24 Feb '25

hi, hope, somebody can help me,i did not found any ideas in the internet. we plan to migrate our fileserver storage to cephfs. our customer uses acl. for standby, it would be great to have a cold standby ... thats why cephfs-mirror. after a lot of testing, i got cephfs-mirror working. but now, is see ... there where no acl synced. i see the acl in the snapshot, but not on the other side. is somebody using cephfs-mirror, also with acl? do you have any other ideas for cold standby (rsync, lsyncd)? many thanks for help and regards, ronny

1 0

ASSISTANCE REQUEST: OSDs Stability Issues Post-Upgrade
by Nima AbolhassanBeigi 23 Feb '25

23 Feb '25

Dear Ceph Community, I hope this message finds you well. I am reaching out to seek assistance regarding a stability issue we have encountered after upgrading our Ceph cluster from version Pacific 16.2.3 to Quincy 17.2.8. Following the upgrade, we have observed that several of our Object Storage Daemons (OSDs) are experiencing erratic behavior. These OSDs frequently exhibit a "flapping" condition, where they unexpectedly go down and then come back up. This issue has predominantly affected the recently upgraded OSDs within the cluster. Upon reviewing the logs from the affected OSDs, we encountered the following messages: 2025-02-03T08:34:09.769+0000 7f0f11390780 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2025-02-03T08:38:22.920+0000 7feb9dd44780 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) In an attempt to resolve the issue, we executed the ceph-bluestore-tool fsck and repair commands. Although these commands executed successfully, they did not rectify the problem at hand. Additionally, we have captured the following crash information from the ceph logs: ceph crash info 2025-02-03T09:19:08.749233Z_9e2800fb-77f6-46cb-8087-203ea15a2039 { "assert_condition": "log.t.seq == log.seq_live", "assert_file": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/17.2.8/rpm/el9/BUILD/c eph-17.2.8/src/os/bluestore/BlueFS.cc", "assert_func": "uint64_t BlueFS::_log_advance_seq()", "assert_line": 3029, "assert_msg": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/17.2.8/rpm/el9/BUILD/ce ph-17.2.8/src/os/bluestore/BlueFS.cc: In function 'uint64_t BlueFS::_log_advance_seq()' thread 7ff983564640 time 2025-02-03T09:19:08.738781+0000\n/home/jenkins-build/build/workspace/ceph-bu ild/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/17.2.8/rpm/el9/BUILD/ceph-17.2.8/src/os/bluestore/BlueFS.cc: 3029: FAILED ceph_assert (log.t.seq == log.seq_live)\n", "assert_thread_name": "bstore_kv_sync", "backtrace": [ "/lib64/libc.so.6(+0x3e730) [0x7ff9930f5730]", "/lib64/libc.so.6(+0x8bbdc) [0x7ff993142bdc]", "raise()", "abort()", "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x179) [0x55882dfb7fdd]", "/usr/bin/ceph-osd(+0x36b13e) [0x55882dfb813e]", "/usr/bin/ceph-osd(+0x9cff3b) [0x55882e61cf3b]", "(BlueFS::_flush_and_sync_log_jump_D(unsigned long)+0x4e) [0x55882e6291ee]", "(BlueFS::_compact_log_async_LD_LNF_D()+0x59b) [0x55882e62e8fb]", "/usr/bin/ceph-osd(+0x9f2b15) [0x55882e63fb15]", "(BlueFS::fsync(BlueFS::FileWriter*)+0x1b9) [0x55882e631989]", "/usr/bin/ceph-osd(+0x9f4889) [0x55882e641889]", "/usr/bin/ceph-osd(+0xd74cd5) [0x55882e9c1cd5]", "(rocksdb::WritableFileWriter::SyncInternal(bool)+0x483) [0x55882eade393]", "(rocksdb::WritableFileWriter::Sync(bool)+0x120) [0x55882eae0b60]", "(rocksdb::DBImpl::WriteToWAL(rocksdb::WriteThread::WriteGroup const&, rocksdb::log::Writer*, unsigned long*, bool, bool, unsigned long)+0x337) [0x55882ea00ab7]", "(rocksdb::DBImpl::WriteImpl(rocksdb::WriteOptions const&, rocksdb::WriteBatch*, rocksdb::WriteCallback*, unsigned long*, unsigned long, bool, unsigned long*, unsigned long, rocksdb ::PreReleaseCallback*)+0x1935) [0x55882ea07675]", "(rocksdb::DBImpl::Write(rocksdb::WriteOptions const&, rocksdb::WriteBatch*)+0x35) [0x55882ea077c5]", "(RocksDBStore::submit_common(rocksdb::WriteOptions&, std::shared_ptr<KeyValueDB::TransactionImpl>)+0x83) [0x55882e992593]", "(RocksDBStore::submit_transaction_sync(std::shared_ptr<KeyValueDB::TransactionImpl>)+0x99) [0x55882e992ee9]", "(BlueStore::_kv_sync_thread()+0xf64) [0x55882e578e24]", "/usr/bin/ceph-osd(+0x8afb81) [0x55882e4fcb81]", "/lib64/libc.so.6(+0x89e92) [0x7ff993140e92]", "/lib64/libc.so.6(+0x10ef20) [0x7ff9931c5f20]" ], "ceph_version": "17.2.8", "crash_id": "2025-02-03T09:19:08.749233Z_9e2800fb-77f6-46cb-8087-203ea15a2039", "entity_name": "osd.211", "os_id": "centos", "os_name": "CentOS Stream", "os_version": "9", "os_version_id": "9", "process_name": "ceph-osd", "stack_sig": "ba90de24e2beba9c6a75249a4cce7c533987ca5127cfba5b835a3456174d6080", "timestamp": "2025-02-03T09:19:08.749233Z", "utsname_hostname": "afra-osd18", "utsname_machine": "x86_64", "utsname_release": "5.15.0-119-generic", "utsname_sysname": "Linux", "utsname_version": "#129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024" } The above crash logs highlight an assertion failure in the BlueFS component, specifically within the function BlueFS::_log_advance_seq(). Despite our efforts to analyze and resolve the issue, we have reached an impasse. For completeness, we have verified the health of our disks using smartctl, and they have all been deemed healthy. We kindly request guidance from the community on how to address this issue or any recommended steps for deeper diagnostics. We appreciate your support and expertise during this troubleshooting process. OSD logs: https://paste.mozilla.org/6STm6eum Thank you for your attention and assistance. Best regards Nima

2 2

User new to ceph
by Christian Hansen 23 Feb '25

23 Feb '25

So i have battled the last days to reclaim my disk. Basically long story short i installed proxmox cluster with a ceph cluster. I reinstalled the proxmox node and didnt remove the osd as you should trough the gui found out later, i tried a bunch of commands like zap the disk wipe the disk and so on. But whatever i do i cannot wipe the ceph disk and i cant seem to wipe the disk so i can reuse the disk. How do you go about wiping a ceph disk when you reinstalled the original OS that had the disk as the OSD. i tried DBAN, i tried ShredOS i tried Dells wipe disk feature but it says its looked. i tried hirens boot and wiping the disk i tried in diskpart in windows to wipe the disk all things fail and i dont get it. Is there no way to reclaim my disk? Thanks in advance. Best regards, Christian

4 3

ceph iscsi gateway
by Iban Cabrillo 22 Feb '25

22 Feb '25

Good morning, I wanted to inquire about the status of the Ceph iSCSI gateway service. We currently have several machines installed with this technology that are working correctly, although I have seen that it appears to be discontinued since 2022. My question is whether to continue down this path, adding machines, or whether it will be discontinued in new Ceph distributions. Best regards and thanks in advance. -- ================================================================ Ibán Cabrillo Bartolomé Instituto de Física de Cantabria (IFCA-CSIC) Santander, Spain Tel: +34942200969/+34669930421 Responsible for advanced computing service (RSC) ========================================================================================= ========================================================================================= All our suppliers must know and accept IFCA policy available at: https://confluence.ifca.es/display/IC/Information+Security+Policy+for+Exter… ==========================================================================================

10 11

understanding Ceph OSD Interaction with the Linux Kernel
by Lina SADI 21 Feb '25

21 Feb '25

Dear Ceph Community, I hope you are doing well. I am trying to understand how Ceph OSD interacts with the Linux kernel. The Linux I/O stack consists of layers such as VFS, filesystem, block layer, and device driver, but since Ceph does not use a traditional filesystem and instead relies on BlueStore as its storage backend, I am struggling to find details on how it interacts with the kernel. Specifically, I would like to understand: - What are the layers in the Linux I/O stack that a Ceph OSD request passes through? - Does BlueStore use submit_bio() to enter the block layer, or does it bypass it and interact directly with the device driver? - Where can I find resources or documentation that explain this in detail? Any insights, documentation links, or pointers to relevant parts of the Ceph source code would be greatly appreciated. Thank you in advance for your help! Best regards.

2 1