- ceph-users - lists.ceph.io

Bluefs spillover
by Ruben Bosch 26 Aug '24

26 Aug '24

Hi all, ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable) We are working on marking out OSDs on a host with EC4+2. The OSDs are HDDs. The OSDs have a separate DB on an NVMe disk. All the operations take ages. After some time we see BLUEFS_SPILLOVER. Telling the mentioned OSDs to compact sometimes helps, but not always. The OSDs have plenty space remaining in the db but the spillover does not disappear. [WRN] BLUEFS_SPILLOVER: 2 OSD(s) experiencing BlueFS spillover osd.91 spilled over 141 MiB metadata from 'db' device (15 GiB used of 50 GiB) to slow device osd.106 spilled over 70 MiB metadata from 'db' device (12 GiB used of 50 GiB) to slow device Has anyone seen similar behavior before and have they found a workaround or solution? Kind regards, Ruben Bosch

2 4

Is ceph-qa list still under administrated?
by Fred Liu 26 Aug '24

26 Aug '24

Hello, Just by chance, there are administrators here. Thank you. Fred

1 0

Paid support options?
by Paul Mezzanini 24 Aug '24

24 Aug '24

We've reached the point of our ceph cluster's life where it make sense to have an outside vendor in the passenger seat with us. I already know of a few but I see value in having a thread consolidating this information so I'm leaving it open ended. The two main questions I'm asking are: What vendors offer paid ceph support? Do they have specific requirements? (e.g. must run their version of ceph vs community, must be containerized vs bare metal) Thanks -paul -- Paul Mezzanini Platform Engineer III Research Computing Rochester Institute of Technology

10 9

Re: Snaptrim issue after nautilus to octopus upgrade
by Boris 23 Aug '24

23 Aug '24

Ah crap, I didn't reply to the list :D tldr for the rest of the list: move to next release because of bad bug https://tracker.ceph.com/issues/53729#note-65 I would go to pacific and then wait some time so the pglog_dups get cleaned up. Pacific and quincy are really nice releases that worked very well for us. Am Fr., 23. Aug. 2024 um 18:08 Uhr schrieb Özkan Göksu <ozkangksu(a)gmail.com >: > Hello Boris. Thank you so much for the recommendation. > > Actually my intention is jumping next releases until I reach Quincy. > What do you think about this? Am I gonna face with any problem while > jumping Octopus to Pacific and Quincy after ? > > Best regards. > > Boris <bb(a)kervyn.de>, 23 Ağu 2024 Cum, 18:56 tarihinde şunu yazdı: > >> Hi, I would suggest to move fast to the next release. >> There was a bug that still haunts my sleep that is known as the PGLOG_DUP >> bug. >> >> You can check if you are suffering from it here: >> https://tracker.ceph.com/issues/53729#note-65 >> >> Basically the pglog dups would not be cleaned anymore. And snaptrim seems >> to trigger that. The pacific release got the fix. >> >> Cheers >> Boris >> >> Am Fr., 23. Aug. 2024 um 16:48 Uhr schrieb Özkan Göksu < >> ozkangksu(a)gmail.com>: >> >>> Hello folks. >>> >>> We have a ceph cluster and we have 2000+ RBD drives on 20 nodes. >>> >>> We upgraded the cluster from 14.2.16 to 15.2.14 and after the upgrade we >>> started to see snap trim issues. >>> Without the "nosnaptrim" flag, the system is not usable right now. >>> >>> I think the problem is because of the omap conversion at Octopus upgrade. >>> >>> Note that the first time each OSD starts, it will do a format conversion >>> to >>> improve the accounting for “omap” data. This may take a few minutes to as >>> much as a few hours (for an HDD with lots of omap data). You can disable >>> this automatic conversion with: >>> >>> What should I do to solve this problem? >>> >>> Thanks. >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io >>> >> >> >> -- >> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im >> groÃƒ¼en Saal. >> > -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

2 1

Pull failed on cluster upgrade
by Nicola Mori 22 Aug '24

22 Aug '24

Dear Ceph users, during an upgrade from 18.2.2 to 18.2.4 the image pull from Dockerhub failed on one machine running a monitor daemon, while it succeeded on the previous ones. # ceph orch upgrade status { "target_image": "snack14/ceph-wizard@sha256:b1994328eb078778abdba0a17a7cf7b371e7d95ee1d543bd987892941ceb91c1", "in_progress": true, "which": "Upgrading all daemon types on all hosts", "services_complete": [ "mgr" ], "progress": "3/152 daemons upgraded", "message": "Error: UPGRADE_FAILED_PULL: Upgrade: failed to pull target image", "is_paused": true } # ceph health detail [WRN] UPGRADE_FAILED_PULL: Upgrade: failed to pull target image failed to pull snack14/ceph-wizard@sha256:b1994328eb078778abdba0a17a7cf7b371e7d95ee1d543bd987892941ceb91c1 on host aka On the affected host I can see the image has been correctly pulled, I can even delete and re-pull it manually: $ sudo docker pull snack14/ceph-wizard@sha256:b1994328eb078778abdba0a17a7cf7b371e7d95ee1d543bd987892941ceb91c1 docker.io/snack14/ceph-wizard@sha256:b1994328eb078778abdba0a17a7cf7b371e7d95ee1d543bd987892941ceb91c1: Pulling from snack14/ceph-wizard c4df4d1fcd03: Pull complete 676ca14fffd6: Pull complete cde0e2cfc7c9: Pull complete Digest: sha256:b1994328eb078778abdba0a17a7cf7b371e7d95ee1d543bd987892941ceb91c1 Status: Downloaded newer image for snack14/ceph-wizard@sha256:b1994328eb078778abdba0a17a7cf7b371e7d95ee1d543bd987892941ceb91c1 docker.io/snack14/ceph-wizard@sha256:b1994328eb078778abdba0a17a7cf7b371e7d95ee1d543bd987892941ceb91c1 so I believe I didn't hit Dockerhub pull limits. Yet the upgrade fails again in the same way when I resume it. I noticed that the docker logs show these warnings in coincidence with a failed pull triggered by the upgrade: Aug 05 18:24:28 aka dockerd[1564]: time="2024-08-05T18:24:28.087629058+02:00" level=warning msg="canonical references cannot be resolved: snack14/ceph-wizard@sha256:b1994328eb078778abdba0a17a7cf7b371e7d95ee1d543bd987892941ceb91c1" Aug 05 18:24:29 aka dockerd[1564]: time="2024-08-05T18:24:29.332143728+02:00" level=warning msg="reference for unknown type: " digest="sha256:b1994328eb078778abdba0a17a7cf7b371e7d95ee1d543bd987892941ceb91c1" remote="docker.io/snack14/ceph-wizard@sha256:b1994328eb078778> but only the second one appears when doing a successful manual pull, so I suspect only the first one is representative of the problem. The docker version on the affected machine is 26.1.3. Thanks in advance for any help, Nicola

8 12

Unable to recover cluster, error: unable to read magic from mon data
by RIT Computer Science House 22 Aug '24

22 Aug '24

Hello, Our cluster has become unresponsive after a teammate's work on the cluster. We are currently unable to get the full story on what he did to fully understand what is going on, and the only error we are able to see in any of the logs is the following: 2024-08-20T03:12:34.183+0000 7f3670246b80 -1 unable to read magic from mon data Any help would be greatly appreciated! I can provide any information necessary to help debugging. Thanks, Tyler System Administrator @ Computer Science House

2 3

Ceph XFS deadlock with Rook
by Raphaël Ducom 21 Aug '24

21 Aug '24

Hi I'm reaching out to check on the status of the XFS deadlock issue with RBD in hyperconverged environments, as detailed in Ceph tracker issue #43910 ( https://tracker.ceph.com/issues/43910?tab=history) It looks like there hasn’t been much activity on this for a while, and I'm wondering if there have been any updates or if it’s just been lost in the issue volume. The issue is a deadlock when using XFS, leading Rook to recommend using EXT4 instead. However, kernel 5.6 introduced PR_SET_IO_FLUSHER, which should allow Ceph to handle this scenario better. Versions lacking PR_SET_IO_FLUSHER are either already EOL or soon will be, with the exception of kernel 5.4. While EXT4 works according to the Ceph documentation, the limitations with XATTR on EXT4 are still a concern. It’s a bit unfortunate that Rook has to move away from XFS where Ceph recommends it. The issue on the Rook side : https://github.com/rook/rook/issues/3132#issuecomment-580508760 Could anyone provide an update on this Ceph issue or suggest how we might push it forward? Any insights would be really appreciated, as this impacts the broader use of XFS in hyperconverged Ceph setups deployed with Rook. Thanks in advance for your help.

2 1

weird outage of ceph
by Simon Oosthoek 21 Aug '24

21 Aug '24

Hi We had a really weird outage today of ceph and I wonder how it came about. The problem seems to have started around midnight, I still need to look if it was to the extend I found it in this morning or if it grew more gradually, but when I found it several osd servers had most or all osd processes down, to the point where our EC 8+3 buckets didn't work anymore. Restarting the servers or the services turned out to be the way to quickly recover from this. I see some of our OSDs are coming close to (but not quite) 80-85% full, There are many times when I've seen an overfull error lead to cascading and catastrophic failures. I suspect this may have been one of them. Which brings me to another question, why is our balancer doing so badly at balancing the OSDs? It's configured with upmap mode and it should work great with the amount of PGs per OSD we have, but it is letting some OSD's reach 80% full and others not yet 50% full (we're just over 61% full in total). The current health status is: HEALTH_WARN Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull pg 30.3fc is active+remapped+backfill_wait+backfill_toofull, acting [66,105,124,113,89,132,206,242,179] I've started reweighting again, because the balancer is not doing it's job in our cluster for some reason... Below is our dashboard overview, you can see the start and recovery in the 24h graph... Cheers /Simon [image: image.png] -- I'm using my gmail.com address, because the gmail.com dmarc policy is "none", some mail servers will reject this (microsoft?) others will instead allow this when I send mail to a mailling list which has not yet been configured to send mail "on behalf of" the sender, but rather do a kind of "forward". The latter situation causes dkim/dmarc failures and the dmarc policy will be applied. see https://wiki.list.org/DEV/DMARC for more details

4 5

Re: Cephfs mds node already exists crashes mds
by Tarrago, Eli (RIS-BCT) 21 Aug '24

21 Aug '24

Here is the backtrace from a ceph crash ceph crash info '2024-08-20T16:07:39.319197Z_8bcdf3df-f9b5-451a-b971-16f8190ab351' { "assert_condition": "!p", "assert_file": "/build/ceph-18.2.4/src/mds/MDCache.cc", "assert_func": "void MDCache::add_inode(CInode*)", "assert_line": 251, "assert_msg": "/build/ceph-18.2.4/src/mds/MDCache.cc: In function 'void MDCache::add_inode(CInode*)' thread 7f0551248700 time 2024-08-20T12:07:39.313490-0400\n/build/ceph-18.2.4/src/mds/MDCache.cc: 251: FAILED ceph_assert(!p)\n", "assert_thread_name": "ms_dispatch", "backtrace": [ "/lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f055736a420]", "gsignal()", "abort()", "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x182) [0x7f05576e12c9]", "/usr/lib/ceph/libceph-common.so.2(+0x32d42b) [0x7f05576e142b]", "(MDCache::add_inode(CInode*)+0x348) [0x55f4f007b7e8]", "(Server::prepare_new_inode(boost::intrusive_ptr<MDRequestImpl>&, CDir*, inodeno_t, unsigned int, file_layout_t const*)+0x7b3) [0x55f4f00023a3]", "(Server::handle_client_mkdir(boost::intrusive_ptr<MDRequestImpl>&)+0x1cd) [0x55f4f000546d]", "(MDSContext::complete(int)+0x5f) [0x55f4f02b72cf]", "(void finish_contexts<std::vector<MDSContext*, std::allocator<MDSContext*> > >(ceph::common::CephContext*, std::vector<MDSContext*, std::allocator<MDSContext*> >&, int)+0x90) [0x55f4eff5c730]", "(MDSCacheObject::finish_waiting(unsigned long, int)+0x5c) [0x55f4f02c7dbc]", "(Locker::eval_gather(SimpleLock*, bool, bool*, std::vector<MDSContext*, std::allocator<MDSContext*> >*)+0x12dc) [0x55f4f016fe2c]", "(Locker::dispatch(boost::intrusive_ptr<Message const> const&)+0x16c) [0x55f4f0183b1c]", "(MDSRank::_dispatch(boost::intrusive_ptr<Message const> const&, bool)+0x5a6) [0x55f4eff6b1a6]", "(MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr<Message const> const&)+0x5c) [0x55f4eff6b79c]", "(MDSDaemon::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x1bf) [0x55f4eff5530f]", "(Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0x468) [0x7f055795c448]", "(DispatchQueue::entry()+0x657) [0x7f05579599e7]", "(DispatchQueue::DispatchThread::entry()+0x11) [0x7f0557a20b11]", "/lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f055735e609]", "clone()" ], "ceph_version": "18.2.4", "crash_id": "2024-08-20T16:07:39.319197Z_8bcdf3df-f9b5-451a-b971-16f8190ab351", "entity_name": "mds.mds0202", "os_id": "ubuntu", "os_name": "Ubuntu", "os_version": "20.04.6 LTS (Focal Fossa)", "os_version_id": "20.04", "process_name": "ceph-mds", "stack_sig": "e51039f21ed17c3220d021c313d5af656ee59d9ccfb3e6d6f991bcf245dbf000", "timestamp": "2024-08-20T16:07:39.319197Z", "utsname_hostname": "mds02", "utsname_machine": "x86_64", "utsname_release": "5.4.0-190-generic", "utsname_sysname": "Linux", "utsname_version": "#210-Ubuntu SMP Fri Jul 5 17:03:38 UTC 2024" } ________________________________ The information contained in this e-mail message is intended only for the personal and confidential use of the recipient(s) named above. This message may be an attorney-client communication and/or work product and as such is privileged and confidential. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail, and delete the original message.

3 2

squid 19.1.1 RC QE validation status
by Yuri Weinstein 21 Aug '24

21 Aug '24

Details of this release are summarized here: https://tracker.ceph.com/issues/67340#note-1 Release Notes - N/A LRC upgrade - N/A Gibba upgrade -TBD Seeking approvals/reviews for: rados - Radek, Laura (https://github.com/ceph/ceph/pull/59020 is being tested and will be cherry-picked when ready) rgw - Eric, Adam E fs - Venky orch - Adam King rbd, krbd - Ilya quincy-x, reef-x - Laura, Neha powercycle - Brad crimson-rados - Matan, Samuel ceph-volume - Guillaume Pls let me know if any tests were missed from this list.

9 22