- ceph-users - lists.ceph.io

Upgrading RGW before cluster?
by Thomas Byrne - STFC UKRI 14 Aug '24

14 Aug '24

Hi all, The Ceph documentation has always recommended upgrading RGWs last when doing a upgrade. Is there a reason for this? As they're mostly just RADOS clients you could imagine the order doesn't matter as long as the cluster and RGW major versions are compatible. Our basic testing has shown no obvious issues (with Pacific RGWs and a Nautilus cluster FWIW). I'm asking because in our case it would be handy to upgrade our gateway infrastructure first, not for any new RGW features, but just for scheduling the operations. Is this a terrible idea? If it helps, this is the first step in a Nautilus -> Pacific -> Reef plan, with no cephadm. Thanks, Tom

4 3

Bluestore issue using 18.2.2
by Marianne Spiller 14 Aug '24

14 Aug '24

Hi, I am trying to gather experience on a Ceph STAGE cluster; it consists of virtual machines - which is not perfect, I know. The VMs are running Debian 12 and podman-4.3.1. There is practically no load on this Ceph - there is just one client using the storage, and it makes no noise. So this is what happened: * "During data consistency checks (scrub), at least one PG has been flagged as being damaged or inconsistent." * so I listed them (["2.3","2.58"]) * and tried to repair ("ceph pg repair 2.3", "ceph pg repair 2.58") * they both went well (resulting in "pgs: 129 active+clean"), but the cluster keeped its "HEALTH_WARN" state ("Too many repaired reads on 1 OSDs") * so I googled for this message; and the only thing I found was to restart the OSD to get rid of this message and - more important - the cluster WARN state ("ceph orch daemon restart osd.3") * after the restart, my cluster was still in WARN state - and complained about "2 PGs has been flagged as being damaged or inconsistent" - but other PGs on other OSDs * I "ceph pg repair"ed them, too, and the cluster's state was WARN afterwards, again ("Too many repaired reads on 1 OSDs") * when I restarted the OSD ("ceph orch daemon restart osd.2"), the crash occured; Ceph marked this OSD "down" and "out" and suspected a hardware issue, while the OSD HDDs in fact are QEMU "harddisks" * I can't judge whether it's a serious bug or just due to my non-optimal STAGE setup, so I'll attach the gzipped log of osd.2 I need help to understand what happened and how to prevent it in the future. What ist this "Too many repaired reads" and how to deal with it? Thanks a lot for reading, Marianne

4 5

Snapshot getting stuck
by Torkil Svensgaard 14 Aug '24

14 Aug '24

Ceph version 18.2.1. We have a nightly backup job snapshotting and exporting all RBDs used for libvirt VMs. Since a couple of weeks ago we've seen one or more getting stuck like this on 3 occasions, so intermittently: " zoperator@yggdrasil:~/backup$ cat /mnt/scratch/personal/zoperator/slurm-2888600.out Creating snap: 10% complete...2024-08-07T10:35:24.687+0200 7f5a2a44b640 0 --2- 172.21.14.135:0/3311921296 >> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] conn(0x7f59fc002060 0x7f59fc0094b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-08-07T10:36:02.064+0200 7f5a2a44b640 0 --2- 172.21.14.135:0/3311921296 >> [v2:172.21.15.150:3300/0,v1:172.21.15.150:6789/0] conn(0x7f59fc00a2b0 0x7f59fc015010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-08-07T10:37:38.191+0200 7f5a23fff640 0 --2- 172.21.14.135:0/3311921296 >> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] conn(0x7f5a0c063ea0 0x7f5a0c066320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-08-07T10:38:16.677+0200 7f5a28c48640 -1 librbd::ImageWatcher: 0x7f5a100076d0 image watch failed: 140024590898336, (107) Transport endpoint is not connected 2024-08-07T10:38:16.677+0200 7f5a28c48640 -1 librbd::Watcher: 0x7f5a100076d0 handle_error: handle=140024590898336: (107) Transport endpoint is not connected " The VM is also throwing stack traces from stuck I/O. Every VM seen affected by this map the RBD through multiple firewalls so that is likely a factor. Hypervisor <-> Palo Alto firewall <-> OpenBSD firewall <-> Ceph Any ideas? I haven't found anything in the ceph logs yet. Mvh. Torkil -- Torkil Svensgaard Sysadmin MR-Forskningssektionen, afs. 714 DRCMR, Danish Research Centre for Magnetic Resonance Hvidovre Hospital Kettegård Allé 30 DK-2650 Hvidovre Denmark Tel: +45 386 22828 E-mail: torkil(a)drcmr.dk

3 3

All MDS's Crashed, Failed Assert
by m＠silvenga.com 14 Aug '24

14 Aug '24

I'm looking for guidance around how to recover after all MDS continue to crash with a failed assert during journal replay (no MON damage). Context: So I've been working through failed MDS for the past day, likely caused by a large snaptrim operation that caused the cluster to grind to a halt. After evicting all clients and restarting the MDS's (it appears the clients were overwhelming the MDS's). The MDS are failing to start with: debug -1> 2024-07-24T18:44:52.674+0000 7f7878c22700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.2.2/rpm/el8/BUILD/ceph-18.2.2/src/osdc/Journaler.cc: In function 'bool Journaler::try_read_entry(ceph::bufferlist&)' thread 7f7878c22700 time 2024-07-24T18:44:52.676027+0000 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.2.2/rpm/el8/BUILD/ceph-18.2.2/src/osdc/Journaler.cc: 1256: FAILED ceph_assert(start_ptr == read_pos) ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f788aa32e15] 2: /usr/lib64/ceph/libceph-common.so.2(+0x2a9fdb) [0x7f788aa32fdb] 3: (Journaler::try_read_entry(ceph::buffer::v15_2_0::list&)+0x132) [0x55555847ef32] 4: (MDLog::_replay_thread()+0xda) [0x555558436bea] 5: (MDLog::ReplayThread::entry()+0x11) [0x5555580e52d1] 6: /lib64/libpthread.so.0(+0x81ca) [0x7f78897d81ca] 7: clone() debug 0> 2024-07-24T18:44:52.674+0000 7f7878c22700 -1 *** Caught signal (Aborted) ** in thread 7f7878c22700 thread_name:md_log_replay ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable) 1: /lib64/libpthread.so.0(+0x12d20) [0x7f78897e2d20] 2: gsignal() 3: abort() 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f788aa32e6f] 5: /usr/lib64/ceph/libceph-common.so.2(+0x2a9fdb) [0x7f788aa32fdb] 6: (Journaler::try_read_entry(ceph::buffer::v15_2_0::list&)+0x132) [0x55555847ef32] 7: (MDLog::_replay_thread()+0xda) [0x555558436bea] 8: (MDLog::ReplayThread::entry()+0x11) [0x5555580e52d1] 9: /lib64/libpthread.so.0(+0x81ca) [0x7f78897d81ca] 10: clone() Normally, there's three MDS are deployed, 1 active, one on hot standby. The cluster seems to believe that any restarted MDS is attempting to replay, but systemd reports an immediate crashed with a SIGABRT. ceph mds stat cephfs:1/1 {0=cephfs.sm1.esxjag=up:replay(laggy or crashed)} Redeploying the MDS's also continue to crash (suggesting a bad journal?)

3 2

Data recovery after resharding mishap
by Gauvain Pocentek 14 Aug '24

14 Aug '24

Hello list, We have made a mistake and dynamically resharded a bucket in a multi-site RGW setup running Quincy (support for this has been added in Reef). So we have now ~200 million objects still stored in the rados cluster, but completely removed from the bucket index (basically ceph has created a new index for the bucket). We would really like to recover these objects, but we are facing a few issues with our ideas. Any help would be appreciated. The main problem we face is that the rados objects contain binary data, where we expected json data. The RGW is configured to use zlib compression, so we think it can be the reason (although using zlib to decompress doesn't work). Has anyone already faced this, and managed to recover the data from rados objects? Another idea we have is to update the new bucket index to inject the old data. This looks possible as the object marker hasn't changed. We also have access to the old indexes/shards, so we could get all the omap key/value pairs and inject them in the new index. Has anyone been mad enough to try this by any chance? Any other idea to recover the data would help of course. Thank you! Gauvain

1 0

Fwd: [community] [OpenInfra Event Update] The CFP For OpenInfra Days NA is now open!
by Alvaro Soto 14 Aug '24

14 Aug '24

Hello Community!!!! I want to share this with you in case anybody wants to send anything to the CFP \o/ Feel free to reach out in case of questions; I hope to see you there. Cheers! ---------- Forwarded message --------- From: Helena Spease <helena(a)openinfra.dev> Date: Tue, Jul 16, 2024 at 1:46 PM Subject: [community] [OpenInfra Event Update] The CFP For OpenInfra Days NA is now open! To: <community(a)lists.openinfra.dev> Hi everyone, The CFP for the OpenInfra Days North America is now open! Submit your talk before the deadline on August 12. https://docs.google.com/forms/d/e/1FAIpQLSeFQBL1qGNegDWDHCVTLtu_wLf06w7JJkc… Pre-Event Timeline: - August 12: CFP ends - August 22: CFP notification of acceptance - August 26: Program announced ———OPENINFRA DAYS NORTH AMERICA——— October 15 & 16, 2024 IU Indianapolis Campus Center in Indianapolis, Indiana - Learn more: https://ittraining.iu.edu/explore-topics/titles/oid-iu/index.html - Sponsor: https://openinfra.dev/events/sponsorship#OIDNA ———OPENINFRA SUMMIT ASIA——— September 3 & 4, 2024 Suwon, South Korea - Learn more: https://2024.openinfraasia.org/ - Sponsor: https://2024.openinfraasia.org/sponsors/ - Schedule: https://2024.openinfraasia.org/a/schedule This schedule is not complete and only reflects OpenInfra Summit Asia sessions. OCP Summit Asia, Ceph Days Asia and OpenInfra Forum sessions are still being added. Session times are subject to change. Cheers, Helena helena(a)openinfra.dev Community Programs Coordinator The OpenInfra Foundation _______________________________________________ Community mailing list -- community(a)lists.openinfra.dev To unsubscribe send an email to community-leave(a)lists.openinfra.dev -- Alvaro Soto *Note: My work hours may not be your work hours. Please do not feel the need to respond during a time that is not convenient for you.* ---------------------------------------------------------- Great people talk about ideas, ordinary people talk about things, small people talk... about other people.

1 1

Announcing go-ceph v0.29.0
by Sven Anderson 14 Aug '24

14 Aug '24

We are happy to announce another release of the go-ceph API library. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.29.0 The library includes bindings that aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. The library also includes additional APIs that can be used to administer cephfs, rbd, and rgw subsystems. There are already a few consumers of this library in the wild, including the ceph-csi project. Sven

1 0

RBD Journaling seemingly getting stuck for some VMs after upgrade to Octopus
by Oliver Freyermuth 13 Aug '24

13 Aug '24

Dear Cephalopodians, we've successfully operated a "good old" Mimic cluster with primary RBD images, replicated via journaling to a "backup cluster" with Octopus, for the past years (i.e. one-way replication). We've now finally gotten around upgrading the cluster with the primary images to Octopus (and plan to upgrade further in the near future). After the upgrade, all MON+MGR-OSD+rbd_mirror daemons are running 15.2.17. We run three rbd-mirror daemons which all share the following client with auth in the "backup" cluster, to which they write: client.rbd_mirror_backup caps: [mon] profile rbd-mirror caps: [osd] profile rbd and the following shared client with auth in the "primary cluster" from which they are reading: client.rbd_mirror caps: [mon] profile rbd caps: [osd] profile rbd i.e. the same auth as described in the docs[0]. Checking on the primary cluster, we get: # rbd mirror pool status health: UNKNOWN daemon health: UNKNOWN image health: OK images: 288 total 288 replaying For some reason, some values are "unknown" here. But mirroring seems to work, as checking on the backup cluster reveals, see for example: # rbd mirror image status zabbix-test.example.com-disk2 zabbix-test.example.com-disk2: global_id: 1bdcb981-c1c5-4172-9583-be6a6cd996ec state: up+replaying description: replaying, {"bytes_per_second":8540.27,"entries_behind_primary":0,"entries_per_second":1.8,"non_primary_position":{"entry_tid":869176,"object_number":504,"tag_tid":1},"primary_position":{"entry_tid":11143,"object_number":7,"tag_tid":1}} service: rbd_mirror_backup on rbd-mirror002.example.com last_update: 2024-08-12 09:53:17 However, we do in some seemingly random cases see that journals are never advanced on the primary cluster — staying with the example above, on the primary cluster I find the following: # rbd journal status --image zabbix-test.physik.uni-bonn.de-disk2 minimum_set: 1 active_set: 126 registered clients: [id=, commit_position=[positions=[[object_number=7, tag_tid=1, entry_tid=11143], [object_number=6, tag_tid=1, entry_tid=11142], [object_number=5, tag_tid=1, entry_tid=11141], [object_number=4, tag_tid=1, entry_tid=11140]]], state=connected] [id=52b80bb0-a090-4f7d-9950-c8691ed8fee9, commit_position=[positions=[[object_number=505, tag_tid=1, entry_tid=869181], [object_number=504, tag_tid=1, entry_tid=869180], [object_number=507, tag_tid=1, entry_tid=869179], [object_number=506, tag_tid=1, entry_tid=869178]]], state=connected] As you can see, the minimum_set was not advanced. As can be seen in "mirror image status", it shows the strange output that non_primary_position seems much more advanced than primary_position. This seems to happen "at random" for only a few volumes... There are no other active clients apart from the actual VM (libvirt+qemu). As a quick fix, to purge journals piling up over and over, we've only found the "solution" to temporarily disable and then re-enable journaling for affected VM disks, which can be identified by: for A in $(rbd ls); do echo -n "$A: "; rbd --format=json journal status --image $A | jq '.active_set - .minimum_set'; done Any idea what is going wrong here? This did not happen with the primary cluster running Mimic and the backup cluster running Octopus before, and also did not happen when both were running Mimic. We plan to switch to snapshot-based mirroring in the future anyways, but it would be good to understand this strange issue in any case. Cheers, Oliver [0] https://docs.ceph.com/en/octopus/rbd/rbd-mirroring/ -- Oliver Freyermuth Universität Bonn Physikalisches Institut, Raum 1.047 Nußallee 12 53115 Bonn -- Tel.: +49 228 73 2367 Fax: +49 228 73 7869 --

2 6

Multi-Site sync error with multipart objects: Resource deadlock avoided
by Tino Lehnig 13 Aug '24

13 Aug '24

Hi, We've been trying to set up multi-site sync on two test VMs before rolling things out on actual production hardware. Both are running Ceph 18.2.4 deployed via cephadm. Host OS is Debian 12, container runtime is podman (switched from Debian 11 and docker.io, same error there). There is only one RGW daemon on each site. Ceph config is pretty much defaults. One thing I did change was setting rgw_relaxed_region_enforcement to true because the zonegroup got renamed from "default" during the switch to multi-site using the dashboard's assistant. There's nothing special like server-side encryption either. Our end goal is to replicate all RGW data from our current cluster to a new one. The Multi-Site configuration itself went pretty smoothly through the dashboard and pre-existing data started syncing right away. Unfortunately, not all objects made it. To be precise, none of the larger objects over the multipart threshold got synced. This is consistent for newly uploaded multipart objects as well. Curiously, it's working fine in the other direction, i.e. multipart uploads from the secondary zone do get synced to the master. Here are some relevant logs: From `radosgw-admin sync error list`: { "shard_id": 26, "entries": [ { "id": "1_1722598249.479766_23730.1", "section": "data", "name": "foobar/new:5160b406-4428-4fdc-9c5d-5ec9fe9404c0.12564119.3:7/logstash_1%3a8.12.2-1_amd64.deb", "timestamp": "2024-08-02T11:30:49.479766Z", "info": { "source_zone": "5160b406-4428-4fdc-9c5d-5ec9fe9404c0", "error_code": 35, "message": "failed to sync object(35) Resource deadlock avoided" } } ] }, From RGW on the receiving end: Aug 02 13:30:49 dev-ceph-single bash[754387]: debug 2024-08-02T11:30:49.474+0000 7f3a6243e640 0 rgw async rados processor: store->fetch_remote_obj() returned r=-35 Aug 02 13:30:49 dev-ceph-single bash[754387]: debug 2024-08-02T11:30:49.474+0000 7f3a36b7b640 2 req 7168648379339657593 0.000000000s :list_data_changes_log normalizing buckets and tenants Aug 02 13:30:49 dev-ceph-single bash[754387]: debug 2024-08-02T11:30:49.474+0000 7f3a36b7b640 2 req 7168648379339657593 0.003999872s :list_data_changes_log init permissions Aug 02 13:30:49 dev-ceph-single bash[754387]: debug 2024-08-02T11:30:49.478+0000 7f3a36b7b640 2 req 7168648379339657593 0.003999872s :list_data_changes_log recalculating target Aug 02 13:30:49 dev-ceph-single bash[754387]: debug 2024-08-02T11:30:49.478+0000 7f3a36b7b640 2 req 7168648379339657593 0.003999872s :list_data_changes_log reading permissions Aug 02 13:30:49 dev-ceph-single bash[754387]: debug 2024-08-02T11:30:49.478+0000 7f3a36b7b640 2 req 7168648379339657593 0.003999872s :list_data_changes_log init op Aug 02 13:30:49 dev-ceph-single bash[754387]: debug 2024-08-02T11:30:49.478+0000 7f3a36b7b640 2 req 7168648379339657593 0.003999872s :list_data_changes_log verifying op mask Aug 02 13:30:49 dev-ceph-single bash[754387]: debug 2024-08-02T11:30:49.478+0000 7f3a36b7b640 2 req 7168648379339657593 0.003999872s :list_data_changes_log verifying op permissions Aug 02 13:30:49 dev-ceph-single bash[754387]: debug 2024-08-02T11:30:49.478+0000 7f3a36b7b640 2 overriding permissions due to system operation Aug 02 13:30:49 dev-ceph-single bash[754387]: debug 2024-08-02T11:30:49.478+0000 7f3a36b7b640 2 req 7168648379339657593 0.003999872s :list_data_changes_log verifying op params Aug 02 13:30:49 dev-ceph-single bash[754387]: debug 2024-08-02T11:30:49.478+0000 7f3a5241e640 0 RGW-SYNC:data:sync:shard[28]:entry[foobar/new:5160b406-4428-4fdc-9c5d-5ec9fe9404c0.12564119.3:7[0]]:bucket_sync_sources[source=foobar:new[5160b406-4428-4fdc-9c5d-5ec9fe9404c0.12564119.3]):7:source_zone=5160b406-4428-4fdc-9c5d-5ec9fe9404c0]:bucket[foobar/new:5160b406-4428-4fdc-9c5d-5ec9fe9404c0.12564119.3<-foobar/new:5160b406-4428-4fdc-9c5d-5ec9fe9404c0.12564119.3:7]:inc_sync[foobar/new:5160b406-4428-4fdc-9c5d-5ec9fe9404c0.12564119.3:7]:entry[logstash_1%3a8.12.2-1_amd64.deb]: ERROR: failed to sync object: foobar/new:5160b406-4428-4fdc-9c5d-5ec9fe9404c0.12564119.3:7/logstash_1%3a8.12.2-1_amd64.deb And from the sender: Aug 02 13:30:49 test-ceph-single bash[885118]: debug 2024-08-02T11:30:49.476+0000 7f0acfdb2640 1 ====== req done req=0x7f0ab50e4710 op status=-104 http_status=200 latency=0.419986606s ====== Aug 02 13:30:49 test-ceph-single bash[885118]: debug 2024-08-02T11:30:49.476+0000 7f0ba9f66640 2 req 5943847843579143466 0.000000000s initializing for trans_id = tx00000527cca1f3381a52a-0066acc369-c052e6-eu2 Aug 02 13:30:49 test-ceph-single bash[885118]: debug 2024-08-02T11:30:49.476+0000 7f0acfdb2640 1 beast: 0x7f0ab50e4710: 10.139.0.151 - synchronization-user [02/Aug/2024:11:30:49.056 +0000] "GET /foobar%3Anew/logstash_1%253a8.12.2-1_amd64.deb?rgwx-zonegroup=9c1ee979-4362-45a1-ae70-2a83a30ea9fc&rgwx-prepend-metadata=true&rgwx-sync-manifest&rgwx-sync-cloudtiered&rgwx-skip-decrypt&rgwx-if-not-replicated-to=a0fab4b8-ec26-4a11-85dd-abab2e3205fa%3Afoobar%2Fnew%3A5160b406-4428-4fdc-9c5d-5ec9fe9404c0.12564119.3 HTTP/1.1" 200 138413732 - - - latency=0.419986606s Aug 02 13:30:49 test-ceph-single bash[885118]: debug 2024-08-02T11:30:49.476+0000 7f0ba9f66640 2 req 5943847843579143466 0.000000000s getting op 0 Aug 02 13:30:49 test-ceph-single bash[885118]: debug 2024-08-02T11:30:49.476+0000 7f0ba9f66640 2 req 5943847843579143466 0.000000000s :list_metadata_log verifying requester They all keep running into the same error: "failed to sync object(35) Resource deadlock avoided" Any ideas? Thanks! -- Mit freundlichem Gruß // Best regards, Tino Lehnig Cloud Architect Contabo GmbH Aschauer Straße 32a 81549 München https://contabo.com E-Mail: tino.lehnig(a)contabo.de Amtsgericht München HRB 180722 Vertretungsberechtigte Geschäftsführer: Dr. Christian Böing & Thomas Schimmel

1 1

Stable and fastest ceph version for RBD cluster.
by Özkan Göksu 13 Aug '24

13 Aug '24

Hello folks! I built a cluster in 2020 and it has been working great with Nautilus 14.2.16 for the past 4 years. I have 1000++ RBD drives for VM's running on Samsung MZ7LH3T8HMLT drives. Now I want to upgrade the ceph version with a fresh installation and I want to take your opinion on which version will be the best choice for me. I want to upgrade it once and I won't touch it again minimum of 2 years. Does anyone have RBD performance comparison from nautilus, octopus, pacific and quincy? I just want to learn the important changes and benefits of this upgrade. Best regards.

2 1