- ceph-users - lists.ceph.io

Squid Grafana Certificates
by Frank Frampton 31 Jan '25

31 Jan '25

Nothing I try will successfully deploy a Grafana daemon with the certificates I choose. The only way I can get it to use my certs is by manually over writing the certs placed on the host node in the binds section of the container and restarting the existing daemons. Replace file contents here '/var/lib/ceph/05aa38e6-8944-11ee-a398-f8f21e2ffe70/grafana.cephc01gw04/etc/grafana/certs/cert_file|cert_key' will result in Grafana using the desired certs. Using ceph config-key commands will add cert|key, however deployed containers do not use the cert|key defined. This process has worked from v16-v18, but now nothing, has something changed in v19? There are no errors stating that the certificates are invalid or that the json is invalid. Scanned By Microsoft EOP

2 1

Re: Grafana certificates storage and Squid
by Thorsten Fuchs 30 Jan '25

30 Jan '25

In the meantime we did update the `mgr/cephadm/cert_store.cert.grafana_cert` and `mgr/cephadm/cert_store.key.grafana_key` variables by hand and that worked out for us. I also tested the python script and it worked well. Makes the creation and management a much nicer experience. -- Thorsten Fuchs abaut GmbH Agnes-Pockels-Bogen 1, 80992 München

1 0

OSD latency
by Devender Singh 29 Jan '25

29 Jan '25

Hello Friends Seeking some help on OSD write latency in "osd performance graph" whereas "osd graph on dashboard" is not showing any high latency. So, is dashboard graph showing only read latency as I can see low latency in osd perf graph which matches the dashboard osd perf graph? OSD per Graph… Dashboard OSD perf Graph CEPH health… # ceph -s cluster: id: 384d7590-d018-11ee-b74c-5b2acfe0b35c health: HEALTH_OK services: mon: 3 daemons, quorum node1,node2,node3 (age 17h) mgr: node01.cvknae(active, since 18h), standbys: node2.foomun mds: 2/2 daemons up, 2 standby osd: 119 osds: 119 up (since 17h), 119 in (since 17h) rgw: 2 daemons active (2 hosts, 1 zones) data: volumes: 2/2 healthy pools: 18 pools, 817 pgs objects: 6.02M objects, 19 TiB usage: 33 TiB used, 383 TiB / 416 TiB avail pgs: 817 active+clean io: client: 129 KiB/s rd, 19 MiB/s wr, 42 op/s rd, 826 op/s wr Regards Dev

2 2

Cephfs mds not trimming after cluster outage
by Adam Prycki 29 Jan '25

29 Jan '25

Hello, we are having issues with cephfs cluster. Any help would be appreciated. We are running still on 18.2.0. During holidays we had outage caused by filling up rootfs. OSDs started randomly dying and we had time when not all PGs were active. This issue is already solved and all OSDs work fine but we're stuck with some MDS issues. warnings we are concerned about: [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs mds.arm-vol.k02r04nvm01.zaqebs(mds.0): 29 slow metadata IOs are blocked > 30 secs, oldest blocked for 1899 secs [WRN] MDS_TRIM: 1 MDSs behind on trimming mds.arm-vol.k02r04nvm01.zaqebs(mds.0): Behind on trimming (4851/128) max_segments: 128, num_segments: 4851 1. Out MDSs are not trimming. 2. our active MDS has metadata slow ops which we cannot understand Cephfs status look ok, main MDS is active. All metadata pool PGs are active and working, there are not laggy PGs. Trying to dump ops from mds also doesn't help ceph daemon ./ceph-mds.arm-vol.k02r04nvm01.zaqebs.asok dump_ops_in_flight { "ops": [], "num_ops": 0 } MDS failover or MDS restart also doesn't help. Metadata slow ops always return after MDS restart. (all MDSs have this issue) After failover main MDS is stuck in rejoin state for a long time. We've used mds_wipe_sessions config option to bring it quickly into active state. I'm guessing slow metadata ops are stopping MDS from trimming but we cannot figure out what is causing these slow ops. Best regards Adam Prycki

2 1

ceph-14.2.22 OSD crashing - PrimaryLogPG::hit_set_trim on unfound object
by Nikola Ciprich 29 Jan '25

29 Jan '25

Hello dear ceph users and developers, today, we've hit issue on one of our legacy clusters running 14.2.22. we've apparently lost two objects containing cache tier hit_set history. when hit_set_trim hits missing object, it causes OSD to crash: ceph version 14.2.22 (ca74598065096e6fcbd8433c8779a2be0c889351) nautilus (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x141) [0x55714332faad] 2: (()+0x4e8ca8) [0x55714332fca8] 3: (PrimaryLogPG::hit_set_trim(std::unique_ptr<PrimaryLogPG::OpContext, std::default_delete<PrimaryLogPG::OpContext> >&, unsigned int)+0xcd6) [0x557143626136] 4: (PrimaryLogPG::hit_set_remove_all()+0x2d5) [0x5571436264b5] 5: (PrimaryLogPG::on_activate()+0x55a) [0x5571436360aa] 6: (PG::RecoveryState::Active::react(PG::AllReplicasActivated const&)+0x130) [0x557143512b50] 7: (boost::statechart::simple_state<PG::RecoveryState::Active, PG::RecoveryState::Primary, PG::RecoveryState::Activating, (boost::statechart::history_mode)0>::react_impl(boost::statechart:: 8: (boost::statechart::simple_state<PG::RecoveryState::Activating, PG::RecoveryState::Active, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na 9: (PG::do_peering_event(std::shared_ptr<PGPeeringEvent>, PG::RecoveryCtx*)+0x15d) [0x55714353c7ed] 10: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0x211) [0x5571434555d1] 11: (PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x52) [0x557143723352] 12: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x582) [0x557143443122] 13: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x3eb) [0x557143ac2f4b] 14: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x557143ac5d30] 15: (()+0x7ea5) [0x7f22383e5ea5] 16: (clone()+0x6d) [0x7f22372a8b0d] I've found old report of similar problem here: https://www.spinics.net/lists/ceph-users/msg48660.html however using the same patch: diff -Naur ceph-14.2.22/src/osd/PrimaryLogPG.cc ceph-14.2.22-fix-hit_set_trim/src/osd/PrimaryLogPG.cc --- ceph-14.2.22/src/osd/PrimaryLogPG.cc 2024-12-29 12:34:17.000000000 +0100 +++ ceph-14.2.22-fix-hit_set_trim/src/osd/PrimaryLogPG.cc 2024-12-29 19:42:31.527632776 +0100 @@ -13932,11 +13932,13 @@ updated_hit_set_hist.history.pop_front(); ObjectContextRef obc = get_object_context(oid, false); - ceph_assert(obc); - --ctx->delta_stats.num_objects; - --ctx->delta_stats.num_objects_hit_set_archive; - ctx->delta_stats.num_bytes -= obc->obs.oi.size; - ctx->delta_stats.num_bytes_hit_set_archive -= obc->obs.oi.size; + //ceph_assert(obc); + if(obc){ + --ctx->delta_stats.num_objects; + --ctx->delta_stats.num_objects_hit_set_archive; + ctx->delta_stats.num_bytes -= obc->obs.oi.size; + ctx->delta_stats.num_bytes_hit_set_archive -= obc->obs.oi.size; + }; } } causes OSD to crash later: /usr/src/redhat/BUILD/ceph-14.2.22/src/osd/PrimaryLogPG.cc: In function 'virtual void PrimaryLogPG::op_applied(const eversion_t&)' thread 7f91e146d700 time 2024-12-29 19:58:13.880029 /usr/src/redhat/BUILD/ceph-14.2.22/src/osd/PrimaryLogPG.cc: 10457: FAILED ceph_assert(applied_version <= info.last_update) /usr/src/redhat/BUILD/ceph-14.2.22/src/osd/PrimaryLogPG.cc: In function 'virtual void PrimaryLogPG::op_applied(const eversion_t&)' thread 7f91df469700 time 2024-12-29 19:58:13.882629 /usr/src/redhat/BUILD/ceph-14.2.22/src/osd/PrimaryLogPG.cc: 10457: FAILED ceph_assert(applied_version <= info.last_update) ceph version 14.2.22 (ca74598065096e6fcbd8433c8779a2be0c889351) nautilus (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x122) [0x55e07e95b815] 2: (()+0x49899a) [0x55e07e95b99a] 3: (PrimaryLogPG::op_applied(eversion_t const&)+0x1f2) [0x55e07eb9fef2] 4: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr<PGTransaction, std::default_delete<PGTransaction> >&&, eversion_t const&, eversion_t const&, std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> > const&, boost::optional<pg_hit_set_history_t>&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr<OpRequest>)+0x720) [0x55e07ed411b0] 5: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0xdc5) [0x55e07eba0cd5] 6: (PrimaryLogPG::simple_opc_submit(std::unique_ptr<PrimaryLogPG::OpContext, std::default_delete<PrimaryLogPG::OpContext> >)+0x9b) [0x55e07eba2acb] 7: (PrimaryLogPG::hit_set_remove_all()+0x2e5) [0x55e07ebdcbf5] 8: (PrimaryLogPG::on_pool_change()+0xeb) [0x55e07ebddc7b] 9: (PG::handle_advance_map(std::shared_ptr<OSDMap const>, std::shared_ptr<OSDMap const>, std::vector<int, std::allocator<int> >&, int, std::vector<int, std::allocator<int> >&, int, PG::RecoveryCtx*)+0x34f) [0x55e07eaf8c4f] 10: (OSD::advance_pg(unsigned int, PG*, ThreadPool::TPHandle&, PG::RecoveryCtx*)+0x2f6) [0x55e07ea482e6] 11: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0x1b4) [0x55e07ea50a24] 12: (PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x4e) [0x55e07ecde60e] 13: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xfea) [0x55e07ea5527a] 14: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x415) [0x55e07f032345] 15: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55e07f034740] 16: (()+0x7ea5) [0x7f9203024ea5] 17: (clone()+0x6d) [0x7f9201ee7b0d] unfortunately here I don't have any clue how to proceed further. I've increased hit_set_count to 32 and hit_set_period to 36000 thus hopefully gaining some time (now OSD seem to be running). Any ideas on how to safely get from this mess? I can't easily get rid of cache tier now since it's used by running VMs and what is worse, I'm not sure I won't hit same problem when deleting cache pool anyways - OSDs are shared by cache pool and NVME data pool so it's pretty uncomfortable situation :-( I'm using bluestore for all OSDs, so can't even try copying other hit_set within filestore.. I'll be very grateful for any help with best regards nikola ciprich

2 1

link to grafana dashboard with osd / host % usage
by Marc 29 Jan '25

29 Jan '25

Is there an existing grafana dashboard/panel that sort of shows the % used on disks and hosts?

3 2

18.2.5 reediness for QE Validation
by Yuri Weinstein 28 Jan '25

28 Jan '25

Happy New Year! We are getting closer to start testing this point release. Almost all PRs marked with "milestone:v18.2.5" were merged. Dev Leads pls tag PRs that must be included appropriately, so we can test and merge them soon. TIA

3 3

AWS SDK change (CRC32 checksums on multiple objects)
by Dave Holland 28 Jan '25

28 Jan '25

Hi, a colleague brought this to my attention: | AWS recently updated their SDKs to enable CRC32 checksums on multiple | object operations by default. | R2 does not currently support CRC32 checksums, and the default | configurations will return header related errors such as Header | 'x-amz-checksum-algorithm' with value 'CRC32' not implemented. https://www.cloudflarestatus.com/incidents/t5nrjmpxc1cj Is this likely to affect Ceph/radosgw users, if they update their clients to the new AWS SDKs? Or does radosgw already support this? thanks, Dave -- ** Dave Holland ** Systems Support -- Informatics Systems Group ** ** dh3(a)sanger.ac.uk ** Wellcome Sanger Institute, Hinxton, UK ** ---------------------------------------------------------------------- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is Wellcome Sanger Institute, Wellcome Genome Campus, Hinxton, CB10 1SA.

5 4

Grafana certificates storage and Squid
by Thorsten Fuchs 28 Jan '25

28 Jan '25

We recently migrated our cluster from 18.2.4 to 19.2.0 and started having issues with Grafana. Ceph gives out the warning "CEPHADM_CERT_ERROR: Invalid grafana certificate on host cc-1: Invalid certificate key: [('PEM routines', '', 'no start line')]. Looking at the certificates they contain a line '# generated by cephadm' and are not the certificates we stored in the config, e.g. 'mgr/cephadm/cc-1/grafana_crt'. After some investigation we found that there was a change to how certificates are stored. Yet I could find no documentation on how to setup per hosts certificates with the new way. The commit that changed the certificate store is https://github.com/ceph/ceph/commit/bb7e715320e41f5d6b6291769e2b6d230eec74cc Maybe anyone can point us in the right direction on how to get our own certs back into Grafana. -- Thorsten Fuchs abaut GmbH Agnes-Pockels-Bogen 1, 80992 München

3 2

Cephfs bug in default Debian 12 (bookworm) kernel v6.1
by Frédéric Nass 28 Jan '25

28 Jan '25

Hello, I just wanted to share here that the default Debian 12 (bookworm) kernel v6.1 is affected by a bug that leads to cephfs client "failing to respond to capability release". This bug has been investigated here [1] and there [2]. To work around this, make sure you're using a Ceph version above v16.2.15 or v17.2.6 or v18.2.1 and an upstream kernel client above v6.5 (like kernel v6.12.9 from bookworm-backports repo) that includes this patch [3]. Cheers, Frédéric. [1] https://tracker.ceph.com/issues/57244 [2] https://tracker.ceph.com/issues/67221 [3] https://patchwork.kernel.org/project/ceph-devel/patch/20230618231011.9077-1…

1 0