Hi Folks,
The weekly performance meeting will be starting in approximately 1 hour
at 8AM PST! Today, Arun Raghunath from Intel will be presenting a talk
on Storage acceleration via Decoupled Cloud Storage Architecture. Hope
to see you there!
Etherpad:
https://pad.ceph.com/p/performance_weekly
Meeting URL:
https://meet.jit.si/ceph-performance
Mark
Hi folks,
I think "bucket sync markers" command is broken in the main branch. It always gives back this "error":
ERROR: sync.read_sync_status() returned error=0
which shouldn't be an error at all. And it doesn't display anything back. I traced it back to a last year's change into rgw_data_sync.cc:
rgw: Disentangle read_sync_status from RemoteBucketManager · ceph/ceph@c09155f
Since remote_info() call returns a negative value for an error and 0 for success, at line 5556 in this commit, it really should check if "ret < 0", not "!ret" for error cases. The consequence is that success is always interpreted as failure, which leads to the strange output from "bucket sync markers". Changing that condition check fixes the issue.
I don't think such a small change would warrant a issue ticket and hope someone could just piggyback it in his next PR.
Thanks,Yixin
# ceph windows tests
PR check will be made required once regressions are fixed
windows build currently depends on gcc11 which limits use of c++20
features. investigating newer gcc or clang toolchain
# 16.2.13 release
final testing in progress
# prometheus metric regressions
https://tracker.ceph.com/issues/59505
related to previous discussion on 4/12 about quincy backports
integration test coverage needed for ceph-exporter and the mgr module
# lab update
centos/rhel tests were failing due to problematic mirrorlists
fixed in https://github.com/ceph/ceph-cm-ansible/pull/731
more sanity checks in progress at
https://github.com/ceph/ceph-cm-ansible/pull/733
# cephalocon feedback
dev summit etherpads: https://pad.ceph.com/p/cephalocon-dev-summit-2023
collect more notes here: https://pad.ceph.com/p/cephalocon-2023-brainstorm
request for dev-focused longer term discussion
could have specific user-focused and dev-focused sessions
dense conference, hard to fit everything in 3 days
could have longer component updates during conf, with time for questions
perhaps 3 days of conf, dev-specific discussions a day before (no cfp,
one big room, then option for breakout), user-feedback sessions during
the normal con
Hi folks,
I noticed that rgw data logs are generated with a timestamp (cls_log_entry in cls_log_types.h), which seems to be used to ensure a certain order among all logs from peers in a zonegroup during sync. Since machine clocks are not perfectly in sync and using timestamps from different machines may lead to the different order than what truly has happen, how does ceph/rgw protect itself against this issue? Originally, I thought it gets a monotonic ID from the master zone or using master zone's timestamp. It doesn't seem to be the case or did I read the code wrong?
Thanks,Yixin
There was a lot of interest expressed at Cephalocon in using the read
balancer code and new commands to Quincy and Pacific. Until I evaluate the
possibility of backporting the feature, I would recommend using the read
balancer on Reef only, as this is where it the feature has been tested.
The main concern lies in the new commands we added, “pg-upmap-primary” and
“rm-pg-upmap-primary”, which are only available for Reef. Past Ceph
versions have a command “primary-temp”, as Stefan mentioned. However, this
command only changes the primary on the acting set, and also is not
maintained.
If you would like to test the read balancer code on older versions, please
do so with caution. Until the Reef commands are backported (if this proves
possible), we cannot guarantee intended behavior with primary-temp.
Thank you,
Laura
On Thu, Apr 20, 2023 at 9:31 AM Stefan Kooman <stefan(a)bit.nl> wrote:
> Hi,
>
> A word of caution for Ceph operators out there. Be careful with "ceph
> osd primary-temp" command. TL;DR: with primary_temp active, a CRUSH
> change might CRASH your OSDs ... and they won't come back online after a
> restart (in almost all cases).
>
> The bug is described in this tracker [1], and fixed with this PR [2]
> (thanks Igor!).
>
> The longer story is that we were inspired by the work done on the
> read-balancer [3] and wondered if we could leverage this on older
> clusters. It turned out this is indeed possible by using the
> "primary-temp" command, instead of "pg-primary-temp" that will be
> available from Reef onward. We compiled a main version of the
> "osdmaptool", fed it OSD maps from pacific clusters and have it
> calculate the optimal primary PG distributions. Then we replaced the
> pg-primary-temp command with "primary-temp" and applied the commands.
> That worked as expected. However, we hit a bug [1] in the Ceph code that
> could not handle a situation when there were changes in the CRUSH map
> when primary_temp where active. We added a new storage node to the
> cluster that triggered this condition as soon as we put it in the proper
> failure domain. It tried to make an OSD primary that was not in the
> active set anymore, and hence crashed (with a Segmentation Fault most
> often, or aborted). This resulted in multiple (many) OSD crashes across
> the failure domains and basically took down the whole cluster.
>
> If the Reef / main read-balancer code can suffer from the same bug is as
> of yet unknown (at least to us). We will try to build a Reef test
> cluster and find out.
>
> For those of you who want to know how we handled this incident can read
> the RFO ([4] in dutch, [5] in English).
>
> Gr. Stefan
>
> [1]: https://tracker.ceph.com/issues/59491?next_issue_id=59490
> [2]: https://github.com/ceph/ceph/pull/51160
> [3]: https://github.com/ljflores/ceph_read_balancer_2023
> [4]: https://www.bit.nl/uploads/images/PDF-Files/RFO-20230314-185335.pdf
> [5]: https://www.bit.nl/uploads/images/PDF-Files/2023.04.20%20RFO_Ceph
> Cluster_185335_EN.pdf
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
> --
Laura Flores
She/Her/Hers
Software Engineer, Ceph Storage <https://ceph.io>
Chicago, IL
lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com>
M: +17087388804
Hi Folks,
The weekly performance meeting will be starting in approximately 30
minutes at 8AM PST! Today we will catch up after Cephalocon! Please
fee free to add your own topic if you have something you want to talk about!
Etherpad:
https://pad.ceph.com/p/performance_weekly
Meeting URL:
https://meet.jit.si/ceph-performance
Mark
Details of this release are summarized here:
https://tracker.ceph.com/issues/59426#note-3
Release Notes - TBD
Seeking approvals/reviews for:
smoke - Josh approved?
orch - Adam King approved?
(there are infrastructure issues in the runs, but we want to release this ASAP)
Thx
YuriW
Hi folks,
Today we discussed:
- Just short of 1 exabyte of Ceph storage reported to Telemetry.
Telemetry's data is public and viewable at:
https://telemetry-public.ceph.com/d/ZFYuv1qWz/telemetry?orgId=1
If your cluster is not reporting to Telemetry, please consider it! :)
- A request from the Ceph Foundation Board to begin tracking component
(e.g. CephFS) roadmaps in docs (or somewhere else appropriate).
Concurrently, leads may also begin sending out status updates on a
~quarterly basis. To be discussed further.
- Cephalocon schedule is available: https://ceph2023.sched.com/
- A regression was reported for the exporter in 17.2.6:
https://github.com/ceph/ceph/pull/50718#issuecomment-1503376925
A follow-up hotfix/announcement is planned.
- Next week's meeting is canceled due to Cephalocon/travel.
Meeting minutes available here as always:
https://pad.ceph.com/p/clt-weekly-minutes
--
Patrick Donnelly, Ph.D.
He / Him / His
Red Hat Partner Engineer
IBM, Inc.
GPG: 19F28A586F808C2402351B93C3301A3E258DD79D
