- ceph-users - lists.ceph.io

Cannot remove bucket due to missing placement rule
by sinan＠turka.nl 28 Aug '24

28 Aug '24

I created a new placement target/pool. I don't have the exact commands anymore, but something similar to: --- $ radosgw-admin zonegroup placement add \ --rgw-zonegroup default \ --placement-id temporary $ radosgw-admin zone placement add \ --rgw-zone default \ --placement-id temporary \ --data-pool default.rgw.temporary.data \ --index-pool default.rgw.temporary.index \ --data-extra-pool default.rgw.temporary.non-ec --- I created a new bucket to use the "default.rgw.temporary.data" as its data pool. Without removing the bucket, I removed the placement target/rule. I realised that I still had a bucket, so tried to remove the bucket: --- $ radosgw-admin bucket rm --bucket=tempbucket 2024-08-26T14:12:08.722+0200 7f6c88a73fc0 0 could not find placement rule temporary within zonegroup 2024-08-26T14:12:08.722+0200 7f6c88a73fc0 0 ERROR: int RGWRados::Bucket::List::list_objects_unordered(const DoutPrefixProvider*, int64_t, std::vector<rgw_bucket_dir_entry>*, std::map<std::__cxx11::basic_string<char>, bool>*, bool*, optional_yield) cls_bucket_list_unordered returned -22 for :tempbucket[d9c26db8-925f-4c6c-838d-6e886ec345ca.694015388.44]) --- So I tried to remove the bucket in a different way: --- $ radosgw-admin metadata rm bucket:tempbucket $ radosgw-admin bucket rm --bucket=tempbucket --- This looked good. I thought the bucket was removed, but my rgw containers are crashing when trying to trim (so it seems the bucket is not really removed): --- debug -2> 2024-08-26T12:33:21.291+0000 7eff1bc70700 0 could not find placement rule temporary within zonegroup debug -1> 2024-08-26T12:33:21.291+0000 7eff1bc70700 0 ERROR: open_bucket_index_shard() returned ret=-22 debug 0> 2024-08-26T12:33:21.295+0000 7eff1bc70700 -1 *** Caught signal (Segmentation fault) ** in thread 7eff1bc70700 thread_name:sync-log-trim ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable) 1: (()+0x12ce0) [0x7eff4530cce0] 2: (()+0x97b01) [0x7eff4f8a6b01] 3: (librados::v14_2_0::IoCtx::aio_operate(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, librados::v14_2_0::AioCompletion*, librados::v14_2_0::ObjectWriteOperation*)+0x74) [0x7eff4f87f3b4] 4: (RGWRadosBILogTrimCR::send_request()+0x1c7) [0x7eff5029fb07] 5: (RGWSimpleCoroutine::state_send_request()+0x13) [0x7eff50297483] 6: (RGWSimpleCoroutine::operate()+0xac) [0x7eff5029ca5c] 7: (RGWCoroutinesStack::operate(RGWCoroutinesEnv*)+0x67) [0x7eff5029a287] 8: (RGWCoroutinesManager::run(std::__cxx11::list<RGWCoroutinesStack*, std::allocator<RGWCoroutinesStack*> >&)+0x271) [0x7eff5029b0b1] 9: (RGWSyncLogTrimThread::process()+0x200) [0x7eff503636d0] 10: (RGWRadosThread::Worker::entry()+0x176) [0x7eff5032bc26] 11: (()+0x81ca) [0x7eff453021ca] 12: (clone()+0x43) [0x7eff43948dd3] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- I can recreate (if required for the fix) the metadata info, since I have the output of "radosgw-admin metadata get bucket:tempbucket > tempbucket_metadata.json" by using: --- $ radosgw-admin metadata put bucket:tempbucket < tempbucket_metadata.json --- What is the best way to recover from this? Recreate the placement rule? Or are there other options? I just want to get rid of the bucket, so my rgw containers don't crash while trying to trim this bucket. Thanks!

2 3

Persistent Bucket Notification Reconfiguring with CreateTopic
by Alex Hussein-Kershaw (HE/HIM) 28 Aug '24

28 Aug '24

Hi folks, Currently running a test Ceph cluster on 19.1.0 and seeing some odd behaviour using the CreateTopic API. Initially I call the CreateTopic API, and all looks well: $ rga topic list { "topics": [ { "topics": [ { "owner": "ahk2", "name": "ahk2", "dest": { "push_endpoint": "http://1.2.3.6:4321", "push_endpoint_args": "", "push_endpoint_topic": "ahk2", "stored_secret": false, "persistent": true, "persistent_queue": ":ahk2", "time_to_live": "None", "max_retries": "None", "retry_sleep_duration": "None" }, "arn": "arn:aws:sns:geored_zg::ahk2", "opaqueData": "", "policy": "" } ] } ] } I call it again, in an identical manner: $ rga topic list { "topics": [ { "topics": [ { "owner": "ahk2", "name": "ahk2", "dest": { "push_endpoint": "http://1.2.3.6:4321", "push_endpoint_args": "", "push_endpoint_topic": "ahk2", "stored_secret": false, "persistent": true, "persistent_queue": "", "time_to_live": "None", "max_retries": "None", "retry_sleep_duration": "None" }, "arn": "arn:aws:sns:geored_zg::ahk2", "opaqueData": "", "policy": "" } ] } ] } Notice that the "persistent_queue" field is gone. I now hit errors trying to use my topic, for example: $ rga topic stats --topic ahk2 This topic does not have a persistent queue. Suspect this might be a bug in the RC. Happy to open a tracker if so. Unless I'm doing something stupid...? Best Wishes, Alex

1 1

Cephfs client capabilities
by wiz.chen＠gmail.com 27 Aug '24

27 Aug '24

Hi All, How to restrict a user that cannot create volume, subvolumegroup, subvolume of cephfs. This user just can access(mount) a subvolume only. Thanks in advance Yufan Chen

3 2

cephadm basic questions: image config, OS reimages
by Matthew Vernon 27 Aug '24

27 Aug '24

Hi, I've some experience with Ceph, but haven't used cephadm much before, and am trying to configure a pair of reef clusters with cephadm. A couple of newbie questions, if I may: * cephadm shell image I'm in an isolated environment, so pulling from a local repository. I bootstrapped OK with cephadm --image docker-registry.wikimedia.org/ceph bootstrap ... And that worked nicely, but if I want to run cephadm shell (to do any sort of admin), then I have to specify cephadm --image docker-registry.wikimedia.org/ceph shell (otherwise it just hangs failing to talk to quay.io). I found the docs, which refer to setting lots of other images, but not the one that cephadm uses: https://docs.ceph.com/en/reef/cephadm/install/#deployment-in-an-isolated-en… I found an old tracker in this area: https://tracker.ceph.com/issues/47274 ...but is there a good way to arrange for cephadm to use the already-downloaded image without having to remember to specify --image each time? * OS reimages We do OS upgrades by reimaging the server (which doesn't touch the storage disks); on an old-style deployment you could then use ceph-volume to re-start the OSDs and away you went; how does one do this in a cephadm cluster? [I presume involves telling cephadm to download a new image for podman to use and suchlike] Would the process be smoother if we arranged to leave /var/lib/ceph intact between reimages? Thanks, Matthew

3 4

cephfs client capabilities
by YuFan Chen 27 Aug '24

27 Aug '24

Hi all, How to restrict an user that cannot create volume, subvolume group, subvolume of cephfs? This user only can access(mount) a subvolume. Thanks in advance Yufan Chen

1 0

Connecting A Client To 2 Different Ceph Clusters
by duluxoz 27 Aug '24

27 Aug '24

Hi All, I haven't been able to find anything in the doco or on the web about this. I'm blessed with having available to me 2 different Ceph Clusters within our organisation (a "regular" Ceph Cluster and a Hyper-converged Proxmox Ceph Cluster). Both Ceph Clusters have a CephFS system on them (regular_cephfs and proxmox_cephfs). Obviously each Ceph Cluster has its own set of Manager, Monitor, and MDS Nodes, and each CephFS has its own user(s)/key(s). Also obviously, I've got the all the relevant key files install on my client in the /etc/ceph folder. How do I set up the ceph.conf file(s) on my clients so that I can use /etc/fstab to connect to each CephFS or each Ceph Cluster? My existing /etc/fstab file (extract of): ~~~ regular_cephfs_user@.regular_cephfs=/folder1 /mnt/folder1 ceph noatime,_netdev 0 0 proxmox_cephfs_user@.proxmox_cephfs=/folder2 /mnt/folder2 ceph noatime,_netdev 0 0 ~~~ Thanks in advance Dulux-Oz

2 4

Do you need to use a dedicated server for the MON service?
by Phong Tran Thanh 27 Aug '24

27 Aug '24

Hi Ceph users I am designing a CEPH system with 6 servers running full NVMe. Do I need to use 3 separate servers to run the MON services that communicate with OpenStack, or should I integrate the MON services into the OSD servers? What is the recommendation? Thank you. -- Email: tranphong079(a)gmail.com Skype: tranphong079

6 8

Snaptrim issue after nautilus to octopus upgrade
by Özkan Göksu 27 Aug '24

27 Aug '24

Hello folks. We have a ceph cluster and we have 2000+ RBD drives on 20 nodes. We upgraded the cluster from 14.2.16 to 15.2.14 and after the upgrade we started to see snap trim issues. Without the "nosnaptrim" flag, the system is not usable right now. I think the problem is because of the omap conversion at Octopus upgrade. Note that the first time each OSD starts, it will do a format conversion to improve the accounting for “omap” data. This may take a few minutes to as much as a few hours (for an HDD with lots of omap data). You can disable this automatic conversion with: What should I do to solve this problem? Thanks.

3 9

[PSA] New git version tag: v19.3.0
by Patrick Donnelly 26 Aug '24

26 Aug '24

TL;DR: we need an intermediate tag, v19.3.0, before v20.0.0 (tentacle) to get upgrade tests from squid -> main working. It is not a real release and should/will not be built/used. Right now upgrade tests are failing [1] from the squid RC to the main branch. This is because teuthology refuses to do an apparent downgrade (v19.1.X -> v19.0.0) during upgrade testing. I'm proposing an intermediate band-aid solution of creating a v19.3.0 **unsigned** tag on the current version of the main branch which would only exist in the git repository (possibly for only a short time). This version is chosen because it'll be greater than any squid release (v19.2.X) or release candidate (v19.1.X) using our current release schema while also not changing the major version (v19 -> v20). Changing the major version is an involved process touching many parts of Ceph and historically takes some time to merge. (I will be starting that process in the coming weeks.) [1] https://tracker.ceph.com/issues/67335 -- Patrick Donnelly, Ph.D. He / Him / His Red Hat Partner Engineer IBM, Inc. GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

tracing in ceph - tentacle release
by Yuval Lifshitz 26 Aug '24

26 Aug '24

Dear Community, jaeger based tracing was introduced to Ceph in the "reef" release, and a major improvement is done in the "squid" release, where we introduced end2end tracing between the RGW and the OSD. In "tentacle" we want to improve tracing further. Initial discussion was done as part of the RGW CDS (although this is not an RGW specific topic). Recording could be found here: https://www.youtube.com/watch?v=t3tT360Eub0&t=24m40s . Some ideas on where we want to take tracing in the upcoming releases are summarized here: https://gist.github.com/yuvalif/52032debe4065d8fb2ddc06308ed94d9 Would appreciate any feedback suggestions and requests on that topic! Yuval

1 0