- ceph-users - lists.ceph.io

Quincy: osd_pool_default_crush_rule being ignored?
by Florian Haas 27 Sep '24

27 Sep '24

Hello everyone, my cluster has two CRUSH rules: the default replicated_rule (rule_id 0), and another rule named rack-aware (rule_id 1). Now, if I'm not misreading the config reference, I should be able to define that all future-created pools use the rack-aware rule, by setting osd_pool_default_crush_rule to 1. I've verified that this option is defined in src/common/options/global.yaml.in, so the "global" configuration section should be the applicable one (I did try with "mon" and "osd" also, for good measure). However, setting this option, in Quincy, apparently has no effect: # ceph config set global osd_pool_default_crush_rule 1 # ceph osd pool create foo pool 'foo' created # ceph osd pool ls detail | grep foo # pool 9 'foo' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 264 flags hashpspool stripe_width 0 I am seeing this behaviour in 17.2.7. After an upgrade to Reef (18.2.4) it is gone, the option behaves as documented, and new pools are created with a crush_rule of 1: # ceph osd pool create bar pool 'bar' created # ceph osd pool ls detail | grep bar pool 10 'bar' replicated size 3 min_size 2 crush_rule 1 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 302 flags hashpspool stripe_width 0 read_balance_score 4.00 However, the documentation at https://docs.ceph.com/en/quincy/rados/configuration/pool-pg-config-ref/#con… asserts that osd_pool_default_crush_rule should already work in Quincy, and the Reef release notes at https://docs.ceph.com/en/latest/releases/reef/ don't mention a fix covering this. Am I doing something wrong? Is this a documentation bug, and the option can't work in Quincy? Was this "accidentally" fixed at some point in the Reef cycle? Thanks in advance for any insight you might be able to share. Cheers, Florian

3 9

ceph can list volumes from a pool but can not remove the volume
by bryansoong21＠gmail.com 27 Sep '24

27 Sep '24

We have a volume in our cluster: [root(a)ceph-1.lab-a ~]# rbd ls volume-ssd volume-8a30615b-1c91-4e44-8482-3c7d15026c28 [root(a)ceph-1.lab-a ~]# rbd rm volume-ssd/volume-8a30615b-1c91-4e44-8482-3c7d15026c28 Removing image: 0% complete...failed. rbd: error opening image volume-8a30615b-1c91-4e44-8482-3c7d15026c28: (2) No such file or directory rbd: image has snapshots with linked clones - these must be deleted or flattened before the image can be removed. Any ideas on how can I remove the volume? Thanks

2 1

Cephalocon 2024 Agenda Announced – Sponsorship Opportunities Still Available!
by Dan van der Ster 27 Sep '24

27 Sep '24

Dear Ceph Community, We are excited to announce that the agenda for Cephalocon 2024 <https://events.linuxfoundation.org/cephalocon/program/schedule/> is now live! This year’s event, hosted at CERN, the birthplace of Ceph at scale, promises to be a unique opportunity to learn from experts, network with the community, and explore the latest innovations in software-defined storage. Whether you're new to Ceph or a long-time expert, there will be sessions and workshops tailored to all levels. We also want to highlight that sponsorship opportunities are still available. Sponsoring Cephalocon is an excellent way to increase your visibility in this influential community while supporting the ongoing growth and development of Ceph. Check out the full agenda on the Cephalocon website <https://events.linuxfoundation.org/cephalocon/> and get in touch with us at Sponsor | LF Events <https://events.linuxfoundation.org/cephalocon/sponsor/> to learn more about how you can become a sponsor. We look forward to seeing you in Geneva! Best regards, Neha, Josh, & Dan

2 1

cephfs +inotify = caps problem?
by Burkhard Linke 27 Sep '24

27 Sep '24

Hi, we are currently trying to debug and understand a problem with cephfs and inotify watchers. A user is running Visual Studio Code with a workspace on a cephfs mount. VSC uses inotify for monitoring files and directories in the workspace: root@cli:~# ./inotify-info ------------------------------------------------------------------------------ INotify Limits: max_queued_events 16,384 max_user_instances 128 max_user_watches 1,048,576 ------------------------------------------------------------------------------ Pid Uid App Watches Instances 3599940 1236 node 1,681 1 1 0 systemd 106 5 3600170 1236 node 54 1 874797 0 udevadm 17 1 3599118 0 systemd 7 3 3599707 1236 systemd 7 3 3599918 1236 node 6 1 2047 100 dbus-daemon 3 1 2054 0 sssd 2 1 2139 0 systemd-logind (deleted) 1 1 2446 0 agetty 1 1 3600001 1236 node 1 1 ------------------------------------------------------------------------------ Total inotify Watches: 1886 Total inotify Instances: 20 ------------------------------------------------------------------------------ root@cli:~# cat /sys/kernel/debug/ceph/XYZ.client354064780/caps | wc -l 1773083 root@cli:~# uname -a Linux cli 6.1.0-23-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.99-1 (2024-07-15) x86_64 GNU/Linux So roughly 1.700 watchers result in over 1.7 million caps. (some of the watchers might for files on different filesystems). I've also checked this on the MDS side, it also reports a very high number of caps for that client. Running tools like lsof on the host as root only reports very few open files (<50). So inotify seems to be responsible for the massive caps build up. Terminating VSC results in a sharp drop of the caps (just a few open files / directories left afterwards). Is this a known problem? Best regards, Burkhard Linke

2 1

[Ceph incident] PG stuck in peering.
by HARROUIN Loan (PRESTATAIRE CA-GIP) 26 Sep '24

26 Sep '24

Hello dear ceph community, We are facing a strange issue this weekend with a pg (13.6a) that is stuck in peering. Because of that we got lot of ops stuck of course. We are running a ceph in Pacific version 16.2.10, we have only SSD disk and are using erasure coding. cluster: id: f5c69b4a-89e0-4055-95f7-eddc6800d4fe health: HEALTH_WARN Reduced data availability: 1 pg inactive, 1 pg peering 256 slow ops, oldest one blocked for 5274 sec, osd.20 has slow ops services: mon: 3 daemons, quorum cos1-dal-ceph-mon-01,cos1-dal-ceph-mon-02,cos1-dal-ceph-mon-03 (age 17h) mgr: cos1-dal-ceph-mon-02(active, since 17h), standbys: cos1-dal-ceph-mon-03, cos1-dal-ceph-mon-01 osd: 647 osds: 646 up (since 27m), 643 in (since 2h) data: pools: 7 pools, 1921 pgs objects: 432.65M objects, 1.6 PiB usage: 2.4 PiB used, 2.0 PiB / 4.4 PiB avail pgs: 0.052% pgs not active 1916 active+clean 2 active+clean+scrubbing 2 active+clean+scrubbing+deep 1 peering The ‘ceph pg 13.6a query’ hung, so we must restart one of the osd that are part of this PG to temporary unhung the query (because during some seconds the pg isn’t peering yet). In that case, the query only retrieves the information about the shard that was hosted on the OSD that we restart. The result of the query is in attachment (shard 0). First when the issue occurs, we check the logs and restart all the osd linked to this PG. Sadly, it didn’t fix anything. We try to investigate the peering state to understand what was going on the primary OSD. We put the OSD in debug but at first glance anything seems strange (we are not use to deep dive that much into ceph). We find that CERN faced something similar a long time ago: https://indico.cern.ch/event/617118/contributions/2490930/attachments/14227… After reading it, we try to do the empty OSD method that they tried (diapo7). We identify that the shard0 seem in a weird state (and was primary) so it was our candidate. We wipe the OSD 11, 148 and 280 (one by one and waiting of course the peering to avoid data loss on other PGs). After that, the OSD.20 was now elected as the primary but still, the PG stay huge in peering and now all OPS are stuck on OSD.20. We are now in the dark. We plan to maybe deep dive deeper into the log of this new OSD.20, and see if we can plan to upgrade our ceph in order to have the most recent version. Any help or suggestion is welcome 😊

2 3

Backup strategies for rgw s3
by Adam Prycki 26 Sep '24

26 Sep '24

Hi, I'm currently working on a project which requires us to backup 2 separate s3 zones/realms and retain it for few months. Requirements were written by someone who doesn't know ceph rgw capabilities. We have to do incremental and full backups. Each type of backup has separate retention period. Is there a way to accomplish this with in a sensible way? My fist idea would be to create multisite replication to archive-zone. But I cannot really enforce data retention on archive zone. It would require us to overwrite lifecycle policies created by our users. As far as I know it's not possible to create zone level lifecycle policy. Users get their accounts are provisioned via openstack swift. Second idea would be to create custom backup script and copy all the buckets in the cluster to different s3 zone. Destination buckets could be all versioned to have desired retention. But this option feels very hackish and messy. Backing up 2 separate s3 zones to one could cause collision in bucket names. Prefixing bucket names with additional information is not safe because buckets have fixed name length. Prefixing object key name is also not ideal. Best regards Adam Prycki

6 6

Re: CephFS snaptrim bug?
by Linkriver Technology 26 Sep '24

26 Sep '24

Hello, We recently upgraded to Quincy (17.2.7) and I can see in the ceph logs many messages of the form: 1713256584.3135679 osd.28 (osd.28) 66398 : cluster 4 osd.28 found snap mapper error on pg 7.284 oid 7:214b503b:::100125de9b8.00000000:5c snaps in mapper: {}, oi: {5a} ...repaired 1713256584.3136106 osd.28 (osd.28) 66399 : cluster 4 osd.28 found snap mapper error on pg 7.284 oid 7:214b4f95:::1001654390d.00000000:5c snaps in mapper: {}, oi: {5a} ...repaired 1713256584.3136535 osd.28 (osd.28) 66400 : cluster 4 osd.28 found snap mapper error on pg 7.284 oid 7:214b4f3f:::1001549ed54.00000000:5c snaps in mapper: {}, oi: {5a} ...repaired 1713256584.9496887 osd.29 (osd.29) 70001 : cluster 4 osd.29 found snap mapper error on pg 7.b4 oid 7:2d089bdc:::10016105140.00000000:5c snaps in mapper: {}, oi: {5a} ...repaired 1713256590.9785151 osd.28 (osd.28) 66401 : cluster 4 osd.28 found snap mapper error on pg 7.284 oid 7:214b5179:::100128b85a0.00000cfe:5c snaps in mapper: {}, oi: {5a} ...repaired 1713256598.6286905 osd.29 (osd.29) 70002 : cluster 4 osd.29 found snap mapper error on pg 7.17c oid 7:3e877f95:::100151d8670.00000000:5c snaps in mapper: {}, oi: {5a} ...repaired ... A cursory reading of the code involved suggests that the scrubber in Quincy has acquired the capacity of detecting and removing the lost snapshots from Octopus, if I understand it correctly. Cheers, Linkriver Technology On Sat, 2022-06-25 at 19:36 +0000, Kári Bertilsson wrote: > Hello > > I am also having this issue after having > set osd_pg_max_concurrent_snap_trims = 0 previously to pause the > snaptrim. I upgraded to ceph 17.2.0. Have tried restarting, > repeering, deep-scrubbing all OSD's, so far nothing works. > > For one of the affected pools `cephfs_10k` I have tested removing > ALL data and it's still showing 26% usage. All snapshots have been > deleted and all pg's for the pool remain at SNAPTRIMQ_LEN = 0. All > pg's are active+clean. > > The pool still shows 589k object usage. When testing `rados get > object` on all the objects, it only works for 2.420 of them. The rest > seem to be in some kind of limbo and can not be read or deleted using > rados. > > # rados -p cephfs_10k listsnaps 10010539c22.00000000 > > > 10010539c22.00000000: > cloneid snaps size overlap > 288 288 30767656 [] > > # rados -p cephfs_10k get 10010539c22.00000000 10010539c22.00000000 > error getting cephfs_10k/10010539c22.00000000: (2) No such file or > directory > > # rados -p cephfs_10k rm 10010539c22.00000000 > error removing cephfs_10k>10010539c22.00000000: (2) No such file or > directory > > Is there some way to make the snap trimmer rediscover these objects > and remove them ? > > On Fri, Mar 18, 2022 at 2:21 PM Linkriver Technology > <technology(a)linkriver-capital.com> wrote: > > Hello, > > > > If I understand my issue correctly, it is in fact unrelated to > > CephFS itself, > > rather the problem happens at a lower level (in Ceph itself). IOW, > > it affects > > all kind of snapshots, not just CephFS ones. I believe my FS is > > healthy > > otherwise. In any case, here is the output of the command you > > asked: > > > > I ran it a few hours ago: > > > > "num_strays": 235, > > "num_strays_delayed": 38, > > "num_strays_enqueuing": 0, > > "strays_created": 5414436, > > "strays_enqueued": 5405983, > > "strays_reintegrated": 17892, > > "strays_migrated": 0, > > > > And just now: > > > > "num_strays": 186, > > "num_strays_delayed": 0, > > "num_strays_enqueuing": 0, > > "strays_created": 5540016, > > "strays_enqueued": 5531494, > > "strays_reintegrated": 18128, > > "strays_migrated": 0, > > > > > > Regards, > > > > LRT > > > > -----Original Message----- > > From: Arnaud M <arnaud.meauzoone(a)gmail.com> > > To: Linkriver Technology <technology(a)linkriver-capital.com> > > Cc: Dan van der Ster <dvanders(a)gmail.com>, Ceph Users > > <ceph-users(a)ceph.io> > > Subject: [ceph-users] Re: CephFS snaptrim bug? > > Date: Thu, 17 Mar 2022 21:48:18 +0100 > > > > Hello Linkriver > > > > I might have an issue close to your > > > > Can you tell us if your strays dirs are full ? > > > > What does this command output to you ? > > > > ceph tell mds.0 perf dump | grep strays > > > > Does the value change over time ? > > > > All the best > > > > Arnaud > > > > Le mer. 16 mars 2022 à 15:35, Linkriver Technology < > > technology(a)linkriver-capital.com> a écrit : > > > > > Hi, > > > > > > Has anyone figured whether those "lost" snaps are rediscoverable > > / > > > trimmable? > > > All pgs in the cluster have been deep scrubbed since my previous > > email and > > > I'm > > > not seeing any of that wasted space being recovered. > > > > > > Regards, > > > > > > LRT > > > > > > -----Original Message----- > > > From: Dan van der Ster <dvanders(a)gmail.com> > > > To: technology(a)linkriver-capital.com > > > Cc: Ceph Users <ceph-users(a)ceph.io>, Neha Ojha <nojha(a)redhat.com> > > > Subject: Re: [ceph-users] CephFS snaptrim bug? > > > Date: Thu, 24 Feb 2022 09:48:04 +0100 > > > > > > See https://tracker.ceph.com/issues/54396 > > > > > > I don't know how to tell the osds to rediscover those trimmed > > snaps. > > > Neha does that possible? > > > > > > Cheers, Dan > > > > > > On Thu, Feb 24, 2022 at 9:27 AM Dan van der Ster > > <dvanders(a)gmail.com> > > > wrote: > > > > > > > > Hi, > > > > > > > > I had a look at the code -- looks like there's a flaw in the > > logic: > > > > the snaptrim queue is cleared if > > osd_pg_max_concurrent_snap_trims = 0. > > > > > > > > I'll open a tracker and send a PR to restrict > > > > osd_pg_max_concurrent_snap_trims to >= 1. > > > > > > > > Cheers, Dan > > > > > > > > On Wed, Feb 23, 2022 at 9:44 PM Linkriver Technology > > > > <technology(a)linkriver-capital.com> wrote: > > > > > > > > > > Hello, > > > > > > > > > > I have upgraded our Ceph cluster from Nautilus to Octopus > > (15.2.15) > > > over the > > > > > weekend. The upgrade went well as far as I can tell. > > > > > > > > > > Earlier today, noticing that our CephFS data pool was > > approaching > > > capacity, I > > > > > removed some old CephFS snapshots (taken weekly at the root > > of the > > > filesystem), > > > > > keeping only the most recent one (created today, 2022-02-21). > > As > > > expected, a > > > > > good fraction of the PGs transitioned from active+clean to > > > active+clean+snaptrim > > > > > or active+clean+snaptrim_wait. In previous occasions when I > > removed a > > > snapshot > > > > > it took a few days for snaptrimming to complete. This would > > happen > > > without > > > > > noticeably impacting other workloads, and would also free up > > an > > > appreciable > > > > > amount of disk space. > > > > > > > > > > This time around, after a few hours of snaptrimming, users > > complained > > > of high IO > > > > > latency, and indeed Ceph reported "slow ops" on a number of > > OSDs and > > > on the > > > > > active MDS. I attributed this to the snaptrimming and decided > > to > > > reduce it by > > > > > initially setting osd_pg_max_concurrent_snap_trims to 1, > > which didn't > > > seem to > > > > > help much, so I then set it to 0, which had the surprising > > effect of > > > > > transitioning all PGs back to active+clean (is this > > intended?). I also > > > restarted > > > > > the MDS which seemed to be struggling. IO latency went back > > to normal > > > > > immediately. > > > > > > > > > > Outside of users' working hours, I decided to resume > > snaptrimming by > > > setting > > > > > osd_pg_max_concurrent_snap_trims back to 1. Much to my > > surprise, > > > nothing > > > > > happened. All PGs remained (and still remain at time of > > writing) in > > > the state > > > > > active+clean, even after restarting some of them. This > > definitely seems > > > > > abnormal, as I mentioned earlier, snaptrimming this FS > > previously > > > would take in > > > > > the order of multiple days. Moreover, if snaptrim were truly > > complete, > > > I would > > > > > expect pool usage to have dropped by appreciable amounts (at > > least a > > > dozen > > > > > terabytes), but that doesn't seem to be the case. > > > > > > > > > > A du on the CephFS root gives: > > > > > > > > > > # du -sh /mnt/pve/cephfs > > > > > 31T /mnt/pve/cephfs > > > > > > > > > > But: > > > > > > > > > > # ceph df > > > > > <snip> > > > > > --- POOLS --- > > > > > POOL ID PGS STORED OBJECTS USED > > %USED MAX > > > AVAIL > > > > > cephfs_data 7 512 43 TiB 190.83M 147 TiB > > 93.22 > > > 3.6 TiB > > > > > cephfs_metadata 8 32 89 GiB 694.60k 266 GiB > > 1.32 > > > 6.4 TiB > > > > > <snip> > > > > > > > > > > ceph pg dump reports a SNAPTRIMQ_LEN of 0 on all PGs. > > > > > > > > > > Did CephFS just leak a massive 12 TiB worth of objects...? It > > seems to > > > me that > > > > > the snaptrim operation did not complete at all. > > > > > > > > > > Perhaps relatedly: > > > > > > > > > > # ceph daemon mds.choi dump snaps > > > > > { > > > > > "last_created": 93, > > > > > "last_destroyed": 94, > > > > > "snaps": [ > > > > > { > > > > > "snapid": 93, > > > > > "ino": 1, > > > > > "stamp": "2022-02-21T00:00:01.245459+0800", > > > > > "name": "2022-02-21" > > > > > } > > > > > ] > > > > > } > > > > > > > > > > How can last_destroyed > last_created? The last snapshot to > > have been > > > taken on > > > > > this FS is indeed #93, and the removed snapshots were all > > created on > > > previous > > > > > weeks. > > > > > > > > > > Could someone shed some light please? Assuming that snaptrim > > didn't > > > run to > > > > > completion, how can I manually delete objects from now- > > removed > > > snapshots? I > > > > > believe this is what the Ceph documentation calls a > > "backwards scrub" > > > - but I > > > > > didn't find anything in the Ceph suite that can run such a > > scrub. This > > > pool is > > > > > filling up fast, I'll throw in some more OSDs for the moment > > to buy > > > some time, > > > > > but I certainly would appreciate your help! > > > > > > > > > > Happy to attach any logs or info you deem necessary. > > > > > > > > > > Regards, > > > > > > > > > > LRT > > > > > _______________________________________________ > > > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > _______________________________________________ > > > ceph-users mailing list -- ceph-users(a)ceph.io > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > > > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: [EXTERNAL] Re: Bucket Notifications v2 & Multisite Redundancy
by Alex Hussein-Kershaw (HE/HIM) 25 Sep '24

25 Sep '24

Thanks 🙂 I've raised: Bug #68102: rgw: "radosgw-admin topic list" may contain duplicated data and redundant nesting - rgw - Ceph<https://tracker.ceph.com/issues/68102?next_issue_id=984> Enhancement #68104: rgw: Add a "disable replication" flag to bucket notification configuration - rgw - Ceph<https://tracker.ceph.com/issues/68104> Also re-including the mailing list as it was dropped. ________________________________ From: Yuval Lifshitz <ylifshit(a)redhat.com> Sent: Tuesday, September 17, 2024 10:36 AM To: Alex Hussein-Kershaw (HE/HIM) <alexhus(a)microsoft.com> Subject: Re: [EXTERNAL] Re: [ceph-users] Bucket Notifications v2 & Multisite Redundancy ok, got it. not even sure we support attribute filtering for these types of events... i think you have to go with v1 for now. would be great if you also submit a tracker for the v1 topic list On Tue, Sep 17, 2024 at 12:24 PM Alex Hussein-Kershaw (HE/HIM) <alexhus(a)microsoft.com<mailto:alexhus@microsoft.com>> wrote: Indeed, I am using the s3:Replication:Create event (which I think is equivalent to s3:ObjectSynced:Create). But this does not solve the problem of the event being added to both topics on the site that received the replication. ________________________________ From: Yuval Lifshitz <ylifshit(a)redhat.com<mailto:ylifshit@redhat.com>> Sent: Tuesday, September 17, 2024 10:21 AM To: Alex Hussein-Kershaw (HE/HIM) <alexhus(a)microsoft.com<mailto:alexhus@microsoft.com>> Subject: Re: [EXTERNAL] Re: [ceph-users] Bucket Notifications v2 & Multisite Redundancy if you want to get a notification when an object is synced, you should use a different type of notification. the fact that an object is uploaded to siteB does not mean it is immediately synced to siteA. would recommend using s3:ObjectSynced:Create event type in this case. you will get this event only when an object is synced. On Tue, Sep 17, 2024 at 12:05 PM Alex Hussein-Kershaw (HE/HIM) <alexhus(a)microsoft.com<mailto:alexhus@microsoft.com>> wrote: Using the sites sounds sensible, and probably better than using my suggestion of application deployment name as they are not subject to changing (and actually we do genuinely use siteA and siteB exclusively as the zone names,). I want the siteA application to be notified of replication changes synced across from PUTs made on siteB, so I need the flip of what you are suggesting, i.e. create notification on siteA with filter "x-amz-metadata-site" == "siteB". That's fine I think (and doesn't really complicate the scenario), but requires the application to now be aware of the zone it is, which isn't information it has without some plumbing (and doesn't seem like something an application would typically be aware of?). That's on top of the plumbing to add the metadata header into every place where I do an S3 PUT. I still think I prefer using v1 and attempting to contribute an enhancement here to set a flag to disable multisite on a per notification basis, given the above; assuming you agree that this is a sensible enhancement. ________________________________ From: Yuval Lifshitz <ylifshit(a)redhat.com<mailto:ylifshit@redhat.com>> Sent: Monday, September 16, 2024 6:15 PM To: Alex Hussein-Kershaw (HE/HIM) <alexhus(a)microsoft.com<mailto:alexhus@microsoft.com>> Subject: Re: [EXTERNAL] Re: [ceph-users] Bucket Notifications v2 & Multisite Redundancy regarding the filter. i don't really follow. on siteA create a notification (with id "notifA") with filter "x-amz-metadata-site" == "siteA" that point to topicA (reachable in site A) and on siteB create a notification (with id "notifB") with filter "x-amz-metadata-site" == "siteB" that point to topicB (reachable in siteB) assuming we are in v2 and both notifications exist on the bucket. when app is uploading an object to a bucket in siteA, it should set "siteA" in the "x-amz-metadata-site" attribute. notifA will match the filter and send the notification to topicA, while it won't match the filter of notifB. when uploading an object to that bucket in siteB the opposite will happen. On Mon, Sep 16, 2024 at 7:14 PM Alex Hussein-Kershaw (HE/HIM) <alexhus(a)microsoft.com<mailto:alexhus@microsoft.com>> wrote: This is really an application level problem, but it's not trivial for me to determine the name of the remote site. So while I could add a metadata header and include the site name, then use "x-amz-metadata-site" == remote_site_name as my filter, it would be much more practical to say "x-amz-metadata-site" != local_site_name, if that were possible. I suspect that makes it firmly not your problem 🙂 but it does push me towards v1. That's what I assumed re the notifications. However, I did delete all the notifications and topics before disabling the feature and did still end up with the nesting shown below in the "ragosgw-admin topic list" command output. I also reproduced that installing a fresh Cluster at V19.1.1, immediately disabling notifications_v2 and then creating topics. although without the duplication this time. $ radosgw-admin topic list { "topics": [ { "topics": [ { "owner": "zone.user", "name": "ahk2", "dest": { "push_endpoint": "https://1.2.3.4:1234/", "push_endpoint_args": "verify-ssl=false", "push_endpoint_topic": "ahk2", "stored_secret": false, "persistent": true, "persistent_queue": ":ahk2", "time_to_live": "None", "max_retries": "None", "retry_sleep_duration": "10" }, "arn": "arn:aws:sns:geored_zg::ahk2", "opaqueData": "", "policy": "" } ] } ] } So maybe a bug in Squid? ________________________________ From: Yuval Lifshitz <ylifshit(a)redhat.com<mailto:ylifshit@redhat.com>> Sent: Monday, September 16, 2024 4:42 PM To: Alex Hussein-Kershaw (HE/HIM) <alexhus(a)microsoft.com<mailto:alexhus@microsoft.com>> Cc: ceph-users <ceph-users(a)ceph.io<mailto:ceph-users@ceph.io>> Subject: Re: [EXTERNAL] Re: [ceph-users] Bucket Notifications v2 & Multisite Redundancy inline On Mon, Sep 16, 2024 at 5:13 PM Alex Hussein-Kershaw (HE/HIM) <alexhus(a)microsoft.com<mailto:alexhus@microsoft.com>> wrote: Yes - that's correct. Thanks for the suggestions. I think the metadata suggestion probably does work, however it doesn't come easy for me as it seems not possible to do a negative match filter; what I would really like to do is setup a filter for objects where "x-amz-metadata-site" != "local-site-name". Seems that's an AWS S3 limitation. why do you need a negative match? the filter determines what is being notified not what to filter out. Likely leaning to disabling notifications_v2 for now and submitting a tracker for adding the flag as you suggested. I tried that but got some internal error from the tracker system, so will try again later. I'm keen to have a go at implementing as it will give us a path forward when v1 is removed. I'm curious if it's supported to go from notifications_v2 to v1? while we automatically migrate topics and notifications from v1 to v2, we do not do the opposite. if you had something on v2 and want to disable, best option is to delete all topics and notifications, disable v2 and define them again. Suspect it might not be and the bottom is an artifact of that. I tested disabling notifications_v2 and seem to have confused the "radosgw-admin topic list" command, which now returns some heavily nested JSON which looks a bit odd. Functionally might still be OK. I'm on the 19.1.1 RC, so not a stable release. I've replaced the actual topic data with "..." below, but it appears I am listing 4 copies of the same 2 topics. e.g.: $ radosgw-admin topic list { "topics": [ { "topics": [ ..., ... ], }, "topics": [ { "topics": [ ..., ... ], }, "topics": [ { "topics": [ ..., ... ], }, "topics": [ { "topics": [ ..., ... ], } ] } Thanks, Alex ________________________________ From: Yuval Lifshitz <ylifshit(a)redhat.com<mailto:ylifshit@redhat.com>> Sent: Monday, September 16, 2024 1:25 PM To: Alex Hussein-Kershaw (HE/HIM) <alexhus(a)microsoft.com<mailto:alexhus@microsoft.com>> Cc: ceph-users <ceph-users(a)ceph.io<mailto:ceph-users@ceph.io>> Subject: Re: [EXTERNAL] Re: [ceph-users] Bucket Notifications v2 & Multisite Redundancy just to see that i got it right. you are asking to disable the topic and notification replication. as you want to send to different topics based on the zone that got the update to the bucket? * one option is to disable "notifications_v2" on the zonegroup. but, this is probably not a good idea - there are other benefits to v2, and the v1 format will be deprecated eventually. * if you can control the client apps on each site, maybe you can add a metadata attribute (e.g. z-amz-...) to the requests (or maybe you already have one?) and filter the notifications based on that attribute? * please consider submitting a tracker for adding a "replication" flag to the notification configuration. so we can disable the replication (without disabling v2 notifications) on a granular level On Mon, Sep 16, 2024 at 1:18 PM Alex Hussein-Kershaw (HE/HIM) <alexhus(a)microsoft.com<mailto:alexhus@microsoft.com>> wrote: Following up on this, I've run into another issue during my prototyping. I have two Ceph Clusters, with a zone each, sharing a zonegroup and realm. I have a local application to each, that needs informed of replication changes. So I've created a topic and notification per site. Given the notification_v2 feature, these are replicated and both sites are aware of both topics and both notifications. I'm testing by writing to the bucket on the "siteA" Cluster, and I can see Replication:Create events pushed to both topics on the "siteB" Cluster. However, therein lies my problem; I don't want the Ceph Cluster on siteB to attempt to notify siteA of the replication (in-fact it can't route to it in the network at all, so it fails, and a queue builds up, which will eventually block subsequent writes if not dealt with). I think this is new in the notification_v2 feature, as a side-effect of the notifications being multisite. I had a look into filtering rules to see if there was a way to say filter by events in a specific zone, but didn't find anything. Is there a solution to this? Any suggestions very welcome 🙂 Kind regards, Alex ________________________________ From: Yuval Lifshitz <ylifshit(a)redhat.com<mailto:ylifshit@redhat.com>> Sent: Tuesday, September 3, 2024 2:56 PM To: Alex Hussein-Kershaw (HE/HIM) <alexhus(a)microsoft.com<mailto:alexhus@microsoft.com>> Cc: ceph-users <ceph-users(a)ceph.io<mailto:ceph-users@ceph.io>> Subject: Re: [EXTERNAL] Re: [ceph-users] Bucket Notifications v2 & Multisite Redundancy responded inline On Tue, Sep 3, 2024 at 3:05 PM Alex Hussein-Kershaw (HE/HIM) <alexhus(a)microsoft.com<mailto:alexhus@microsoft.com>> wrote: Hi Yuval, Thanks for the response. I did managed to disable the feature, however I hope you can understand my hesitancy to design our move away from pubsub onto a deprecated replacement (i.e. "notifications v1"). > agree. notification v2 provides better observability, e.g. list of buckets that use topic etc. The difference between this and the other operations that require forwarding to the master site, for me, is that I don't rely on any of those operations for day-to-day function. For example, S3 user creation I do at deployment time and never again; so if I lose the ability to do that in the event of a site failure that's OK. Perhaps I just need to implement a process to ensure the remaining zone is quickly promoted on a site failure. > this is probably the correct approach in multisite A further concern of the new feature... My intention is to use the bucket notification feature to send events generated by the multisite sync to my application server. If the topic is now multisite, will all kicks be duplicated? For example consider: SiteA receives an S3 write for object SiteA writes to the event queue and asynchronously delivers the event SiteB syncs object from siteA we do not send regular notifications upon sync. note that we have other type of notifications: s3:ObjectSynced:*/s3:Replication:* that are sent only when an object is synced SiteB writes to the event queue and asynchronously delivers the event Really I only want the second event in this example, which is what I get with the pubsub module, and I think is what I will get with "notifications_v1". Best Wishes, Alex ________________________________ From: Yuval Lifshitz <ylifshit(a)redhat.com<mailto:ylifshit@redhat.com>> Sent: Tuesday, September 3, 2024 12:15 PM To: Alex Hussein-Kershaw (HE/HIM) <alexhus(a)microsoft.com<mailto:alexhus@microsoft.com>> Cc: ceph-users <ceph-users(a)ceph.io<mailto:ceph-users@ceph.io>> Subject: [EXTERNAL] Re: [ceph-users] Bucket Notifications v2 & Multisite Redundancy Hi Alex, It should be possible to disable the v2 feature through an admin command. radosgw-admin onegroup modify --disable-feature=notification_v2 Also note that when creating a new squid cluster, v2 is enabled by default. But, when upgrading an existing cluster, you need to call: radosgw-admin onegroup modify --enable-feature=notification_v2 to enable v2. However, we do encourage moving to v2, as there are many benefits to the new data model (besides site syncing). Regarding the problem you encountered when the "master" site is down, how is this different from other operations that are forwarded to "master"? Yuval On Tue, Sep 3, 2024 at 11:31 AM Alex Hussein-Kershaw (HE/HIM) <alexhus(a)microsoft.com<mailto:alexhus@microsoft.com>> wrote: Hi Folks, I see in the pending release notes for Squid a description of the "notification_v2" feature. ceph/PendingReleaseNotes at main · ceph/ceph (github.com<http://github.com/>)<https://github.com/ceph/ceph/blob/main/PendingReleaseNotes#L43> I have some concerns about the multisite nature of this feature, I am using Ceph multisite to provide geographic redundancy for my application data. That seems to be a mainline use case. My application is designed to cope if an entire zone fails. My application currently relies on pubsub (and soon to be replaced by bucket notifications) to provide service. More specifically I rely on the ability to semi-regularly automatically reconfigure topics to adjust the HTTP endpoint. I believe with the "notification_v2" feature all adjustments of the topic are routed through the master for metadata site (much like creating an S3 user). Fundamentally I think this breaks the combination of Ceph multisite, notifications_v2 and reconfiguring a topic; you cannot reconfigure a topic unless both sites are up. I tested this, by setting up a deployment, shutting down a site and trying to reconfigure a topic. I get a 500 HTTP response to my createTopic, and can see this in the RGW logs: debug 2024-09-03T08:24:39.997+0000 7f53a3372640 20 ERROR: curl error: Couldn't connect to server req_data->error_buf=Failed to connect to 10.225.20.19 port 7480: No route to host debug 2024-09-03T08:24:39.997+0000 7f5329a7f640 4 req 2968544598970267460 0.686004043s sns:pubsub_topic_create CreateTopic forward_request_to_master returned ret = -2200 Have the implications of this been considered? For now I may be able to disable the notifications_v2 feature, however in the release note the predecessor is deprecated in Squid. Hopefully I'm missing something obvious. Kind regards, Alex _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io<mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave(a)ceph.io<mailto:ceph-users-leave@ceph.io>

1 2

Quincy: osd_pool_default_crush_rule being ignored?
by Florian Haas 24 Sep '24

24 Sep '24

Hello everyone, my cluster has two CRUSH rules: the default replicated_rule (rule_id 0), and another rule named rack-aware (rule_id 1). Now, if I'm not misreading the config reference, I should be able to define that all future-created pools use the rack-aware rule, by setting osd_pool_default_crush_rule to 1. I've verified that this option is defined in src/common/options/global.yaml.in, so the "global" configuration section should be the applicable one (I did try with "mon" and "osd" also, for good measure). However, setting this option, in Quincy, apparently has no effect: # ceph config set global osd_pool_default_crush_rule 1 # ceph osd pool create foo pool 'foo' created # ceph osd pool ls detail | grep foo # pool 9 'foo' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 264 flags hashpspool stripe_width 0 I am seeing this behaviour in 17.2.7. After an upgrade to Reef (18.2.4) it is gone, the option behaves as documented, and new pools are created with a crush_rule of 1: # ceph osd pool create bar pool 'bar' created # ceph osd pool ls detail | grep bar pool 10 'bar' replicated size 3 min_size 2 crush_rule 1 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 302 flags hashpspool stripe_width 0 read_balance_score 4.00 However, the documentation at https://docs.ceph.com/en/quincy/rados/configuration/pool-pg-config-ref/#con… asserts that osd_pool_default_crush_rule should already work in Quincy, and the Reef release notes at https://docs.ceph.com/en/latest/releases/reef/ don't mention a fix covering this. Am I doing something wrong? Is this a documentation bug, and the option can't work in Quincy? Was this "accidentally" fixed at some point in the Reef cycle? Thanks in advance for any insight you might be able to share. Cheers, Florian

1 0

Multisite sync: is metadata transferred in plain text?
by maryzhang0920＠gmail.com 24 Sep '24

24 Sep '24

Hi, We have a multisite Ceph configuration, with http (not https) sync endpoints. Are all sync traffic in plain text? We have concerns about metadata. For example, when syncing a newly created user and its access key and secret key from the Master zone to a secondary zone, is this traffic in plain text? If so, what are options to encrypt it? Thank you, Mary

4 4