- ceph-users - lists.ceph.io

Unable to move realm master between zonegroups -- radosgw-admin zonegroup ignoring the --rgw-zonegroup flag?
by Tim Hunter 26 Jun '24

26 Jun '24

Hi folks, We have a number of ceph clusters organized into one realm and three zonegroups. All the clusters are running ceph 17 built with cephadm. I am trying to move the metadata master from one zonegroup (us-east-1) to zone mn1 in zonegroup us-central-1, by following the steps in the documentation <https://docs.ceph.com/en/quincy/radosgw/multisite/#changing-the-metadata-ma…>, but it doesn't work as advised. I ran these commands on a system in the `mn1` zone in the us-central-1 zonegroup. sudo radosgw-admin zone modify --rgw-zone=mn1 --master sudo radosgw-admin zonegroup modify --rgw-zonegroup=us-central-1 --master sudo radosgw-admin period update --commit When I carry out those steps, the period update gives me this error: 2024-06-26T11:19:12.256+0000 7f1f5f610e40 0 Error updating periodmap, multiple master zonegroups configured 2024-06-26T11:19:12.256+0000 7f1f5f610e40 0 master zonegroup: 9ef7877b and a3abf9e1 Here we have problem (1) that the command doesn’t move the master from a3abf9e1 (us-east-1) to 9ef7877b (us-central-1). There is a fix advised by a 2016 post to ceph-users <https://www.spinics.net/lists/ceph-users/msg30873.html> -- explicitly mark the old zonegroup master as no longer the master. sudo radosgw-admin zonegroup modify --rgw-zonegroup=us-east-1 --master=false However, the json output of this command is a modified map of the us-central-1 zonegroup, showing that the 'is_master' flag has been set back to 'false' for that zonegroup. Thus we have problem (2) that I am unable to modify two different zonegroups from the same system, leaving me unable to change the realm master. `sudo radosgw-admin zonegroup get --rgw-zonegroup=us-east-1`, when run from the us-central-1 zonegroup, just shows me the us-central-1 zonegroup details. Changing the `=` to a space does not have any effect. My brief attempts at reading the code for rgw_admin.cc on github left me unenlightened. Can anyone offer any assistance here? -- Tim Hunter Senior Infrastructure Engineer telnyx.com [image: Telnyx logo]

1 0

Slow down RGW updates via orchestrator
by Boris 26 Jun '24

26 Jun '24

Hi, we've just updated our test cluster via ceph orch upgrade start --image quay.io/ceph/ceph:v18.2.0 During the update of the RGW service, all of the daemons went down at the same time. If I would do that on our production system it would cause a small but noticeable outage. Is there a way to only update 1 daemon at a time? Cheers Boris -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

2 2

Ceph Leadership Team Weekly Minutes 2024-06-17
by Laura Flores 26 Jun '24

26 Jun '24

Need to update the OS Recommendations doc to represent latest supported distros - https://docs.ceph.com/en/latest/start/os-recommendations/#platforms - PR from Zac to be reviewed CLT: https://github.com/ceph/ceph/pull/58092 arm64 CI check ready to be made required (request from Rongqi Sun, the arm check maintainer) - Agreed, although need to make sure all old issues are gone first Spam emails on various ceph lists (eg: ceph-users) - Need to improve configuration around spam check (SpamAssassin scoring seems unusually low for the types of emails coming through) - Currently version 3.4.6, which is relatively recent, but the scoring seems abnormally low for the content of the emails - Evaluate raw message headers to see this - Evaluate moderating subscription requests (require approval) [Zac] CQ#5 currently being compiled -- get your announcements in before July 10. https://pad.ceph.com/p/ceph_quarterly_2024_07 Release updates - 18.2.3: Moving along, Venky addressing issues in the fs suite - 19.1.0: Rados run scheduled; needs to be reviewed by rados - Etherpad created to track extra cherry-picks: https://pad.ceph.com/p/release-cherry-pick-coordination -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

2 1

Lot of spams on the list
by Alain Péan 25 Jun '24

25 Jun '24

Hi all, I already sent an email abbout this this morning (in France) but curiously, I never received it (blocked by my own email server From University Paris-saclay ?), although hundred of spams, seemingly coming from India, with sexual subject, have arrived in my email box (even if marked as spam by Thunderbird) . It lasts since about two weeks. I think I am not the only one on the list to get them. I am very upset to see that nothing seems to be done against these spams. Spammers should be unsubscribed for example priya.yt1 at gmail), and perhaps new subscription blocked for a time ? Best Regards, Alain -- Administrateur Système/Réseau C2N Centre de Nanosciences et Nanotechnologies (UMR 9001) Boulevard Thomas Gobert (ex Avenue de La Vauve), 91120 Palaiseau Tel : 01-70-27-06-88 Bureau A255

4 5

cephadm does not recreate OSD
by Luis Domingues 25 Jun '24

25 Jun '24

Hello all. After a disk changed, we see that cephadm does not recreate the OSD. Going all back to pvs command I ended up on this issue: https://tracker.ceph.com/issues/62862 and this PR: https://github.com/ceph/ceph/pull/53500. The PR is unfortunately closed. Is this a non-bug? I tries to replicate this pvs issue on various OS. And it looks like to be the good behavior at least on PVS side, if a lv was deleted in the middle of the disk. Example with a simple vg with 5 lvs were lv 3 was deleted. ``` root@debian:~# pvs --readonly -o pv_name,vg_name,lv_name PV VG LV /dev/vdb newvg lv1 /dev/vdb newvg lv2 /dev/vdb newvg /dev/vdb newvg lv4 /dev/vdb newvg lv5 /dev/vdb newvg ``` This output can seem weird, but if we expand the output lables ``` root@debian:~# pvs --segments -o+lv_name,seg_start_pe,segtype PV VG Fmt Attr PSize PFree Start SSize LV Start Type /dev/vdb newvg lvm2 a-- <60.00g <20.00g 0 2560 lv1 0 linear /dev/vdb newvg lvm2 a-- <60.00g <20.00g 2560 2560 lv2 0 linear /dev/vdb newvg lvm2 a-- <60.00g <20.00g 5120 2560 0 free /dev/vdb newvg lvm2 a-- <60.00g <20.00g 7680 2560 lv4 0 linear /dev/vdb newvg lvm2 a-- <60.00g <20.00g 10240 2560 lv5 0 linear /dev/vdb newvg lvm2 a-- <60.00g <20.00g 12800 2559 0 free ``` Anyway, this seems to cause the duplication. Will someone have a look into this issue? Or should we look into a workaround? Thanks, Luis

1 0

Incomplete PGs. Ceph Consultant Wanted
by cellosofia1＠gmail.com 25 Jun '24

25 Jun '24

Hi community! Recently we had a major outage in production and after running the automated ceph recovery, some PGs remain in "incomplete" state, and IO operations are blocked. Searching in documentation, forums, and this mailing list archive, I haven't found yet if this means this data is recoverable or not. We don't have any "unknown" objects or PGs, so I believe this is somehow an intermediate stage where we have to tell ceph which version of the objects to recover from. We are willing to work with a Ceph Consultant Specialist, because the data at stage is very critical, so if you're interested please let me know off-list, to discuss the details. Thanks in advance Best Regards Pablo

6 15

Incomplete PGs. Ceph Consultant Wanted
by cellosofia1＠gmail.com 25 Jun '24

25 Jun '24

Hi community! Recently we had a major outage in production and after running the automated ceph recovery, some PGs remain in "incomplete" state, and IO operations are blocked. Searching in documentation, forums, and this mailing list archive, I haven't found yet if this means this data is recoverable or not. We don't have any "unknown" objects or PGs, so I believe this is somehow an intermediate stage where we have to tell ceph which version of the objects to recover from. We are willing to work with a Ceph Consultant Specialist, because the data at stage is very critical, so if you're interested please let me know off-list, to discuss the details. Thanks in advance Best Regards Pablo

2 1

Test after list GC
by Anthony D'Atri 25 Jun '24

25 Jun '24

Here’s a test after de-crufting held messages. Grok the fullness. — aad

1 0

Re: Lot of spams on the list
by Anthony D'Atri 25 Jun '24

25 Jun '24

I’m not sure if I have access but I can try. > On Jun 24, 2024, at 4:37 PM, Kai Stian Olstad <kaistian(a)olstad.com> wrote: > > On 24.06.2024 19:15, Anthony D'Atri wrote: >> * Subscription is now moderated >> * The three worst spammers (you know who they are) have been removed >> * I’ve deleted tens of thousands of crufty mail messages from the queue >> The list should work normally now. Working on the backlog of held messages. 99% are bogus, but I want to be careful wrt baby and bathwater. > > Will the archive[1] also be clean up? > > [1] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/ > > -- > Kai Stian Olstad

1 0

June User + Dev Monthly Meeting [Recording Available]
by Noah Lehman 25 Jun '24

25 Jun '24

Hi Ceph users, A recording of June's user + developer monthly meeting is now available. Thank you to everyone who participated, asked questions, and shared insights. Your feedback is crucial to the growth and health of the ceph community! Watch it here: https://youtu.be/7D9otll-kjA?feature=shared Best, Noah

1 0