- ceph-users - lists.ceph.io

lifecycle for versioned bucket
by Lukasz Borek 17 Sep '24

17 Sep '24

Hi, I'm having issue with lifecycle jobs for 18.2.4 cluster with versioning enabled bucket. /# radosgw-admin lc list [ { "bucket": ":mongobackup-prod:c3e0a369-71df-40f5-a5c0-51e859efe0e0.96754.1", "shard": "lc.0", "started": "Thu, 01 Jan 1970 00:00:00 GMT", "status": "COMPLETE" } ] During RGW startup : 2024-09-17T16:51:06.148+0000 7f944d02d640 2 garbage collection: garbage collection: start 2024-09-17T16:51:06.152+0000 7f944501d640 2 lifecycle: life cycle: start 2024-09-17T16:51:06.152+0000 7f944501d640 5 lifecycle: RGWLC::process(): ENTER: index: 26 worker ix: 0 2024-09-17T16:51:06.152+0000 7f943f812640 2 lifecycle: life cycle: start 2024-09-17T16:51:06.152+0000 7f943f812640 5 lifecycle: RGWLC::process(): ENTER: index: 18 worker ix: 1 2024-09-17T16:51:06.152+0000 7f944501d640 5 lifecycle: RGWLC::process() process shard rollover lc_shard=lc.26 head.marker= head.shard_rollover_date=0 2024-09-17T16:51:06.152+0000 7f944501d640 5 lifecycle: RGWLC::process() entry.bucket.empty() == true at START 1 (this is possible mainly before any lc policy has been stored or after removal of an lc_shard object) 2024-09-17T16:51:06.152+0000 7f943a007640 2 lifecycle: life cycle: start 2024-09-17T16:51:06.152+0000 7f943a007640 5 lifecycle: RGWLC::process(): ENTER: index: 17 worker ix: 2 2024-09-17T16:51:06.156+0000 7f943f812640 5 lifecycle: RGWLC::process() process shard rollover lc_shard=lc.18 head.marker= head.shard_rollover_date=0 2024-09-17T16:51:06.156+0000 7f943f812640 5 lifecycle: RGWLC::process() entry.bucket.empty() == true at START 1 (this is possible mainly before any lc policy has been stored or after removal of an lc_shard object) 2024-09-17T16:51:06.156+0000 7f944501d640 5 lifecycle: RGWLC::process(): ENTER: index: 9 worker ix: 0 [...] 2024-09-17T16:51:06.568+0000 7f94137ba640 2 lifecycle: life cycle: stop 2024-09-17T16:51:06.572+0000 7f94137ba640 5 lifecycle: schedule life cycle next start time: Wed Sep 18 01:00:00 2024 Scheduled job ends with: 2024-09-17T00:00:01.379+0000 7fd17f843700 0 lifecycle: RGWLC::process() head.marker !empty() at START for shard==lc.31 head last stored at Mon Sep 16 06:34:39 2024 2024-09-17T00:00:01.379+0000 7fd17f843700 0 lifecycle: RGWLC::process() sal_lc->get_entry(lc_shard, head.marker, entry) returned error ret==-2 Anyone can explain the logic behind that? Thanks. -- Łukasz Borek lukasz(a)borek.org.pl

1 0

Strange CephFS Permission error
by Carsten Feuls 17 Sep '24

17 Sep '24

Hello, a cluster from my friend has an strage issue with CephFS Permission. Ceph Version is 17.2.7 in cephadm. The Cluster was much older and was upgraded over years from 12 and moved to new hosts. When I create a cephfs user for my private cluster with "ceph fs authorize" I get permission like this. client.test2 key: <placeholder caps: [mds] allow r fsname=cephfs, allow rw fsname=cephfs path=/all caps: [mon] allow r fsname=cephfs caps: [osd] allow rw tag cephfs data=cephfs For my cluster this is working fine. But for the cluster from my friend I get only metadata access no file access. I figured it out that when I modifie the caps like this I get it working again. client.test2 key: <placeholder caps: [mds] allow r fsname=cephfs, allow rw fsname=cephfs path=/all caps: [mon] allow r fsname=cephfs caps: [osd] allow rw But this is a solution I do not wan't to keep and every auth createtd with "ceph fs authorize" is not working. ceph osd get-require-min-compat-client -> nautilus Any Ideas how to solve this? Cheers Carsten Feuls

1 0

Ceph RBD w/erasure coding
by andre＠drenet.net 17 Sep '24

17 Sep '24

I have a Ceph instance I'm wherein I'm trying to use erasure coding with RBD (and libvirt). I've followed https://docs.ceph.com/en/latest/rados/operations/erasure-code/#erasure-codi… and enabled overwrites so that RBD can be used. In doing so, I've set the data_pool to the erasure coded pool, with set to the replicated pool. I have the following in ceph.conf: rbd_default_data_pool = libvirt-pool-ec Here's an rbd info on the image I've created (notice the "data_pool" config): rbd image 'xxxxxxx': size 500 GiB in 128000 objects order 22 (4 MiB objects) snapshot_count: 0 id: 7e37e442592b data_pool: libvirt-pool-ec block_name_prefix: rbd_data.6.7e37e442592b format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, data-pool op_features: flags: create_timestamp: Fri Sep 13 16:34:06 2024 access_timestamp: Fri Sep 13 16:34:06 2024 modify_timestamp: Fri Sep 13 16:34:06 2024 The problem, when I attach this rbd image to a VM, I cannot write to it at all, I get an I/O error. But, when I create and rbd image without the "rbd_default_data_pool = libvirt-pool-ec" setting, I can write fine. Wanted to see if anyone has any ideas, maybe I'm missing something simple. Thanks in advance!

6 6

Re: Ceph RBD w/erasure coding
by andre＠drenet.net 17 Sep '24

17 Sep '24

My VM was running Reef, actually. But I determined the issue to be write perms on the erasure coded pool. The resolution in my case was editing my client.libvirt user adding to the mon caps, and allowing the erasure coded pool on the osd caps: ceph auth caps client.libvirt mon 'allow r, allow command "osd blacklist", allow command "osd blocklist", allow command "blacklistop", allow command "blocklistop"' osd 'allow class-read object_prefix rbd_children, allow rwx pool=libvirt-pool, allow rwx pool=images, allow rwx pool=libvirt-pool-ec' :) On 2024-09-13 14:58, Anthony D'Atri wrote: > Is your VM running something older than Luminous? Depending on how > your attachment is being done, you may need to specify both pools on > the client side too. > >> On Sep 13, 2024, at 1:45 PM, andre(a)drenet.net wrote: >> >> I have a Ceph instance I'm wherein I'm trying to use erasure coding >> with RBD (and libvirt). I've followed >> https://docs.ceph.com/en/latest/rados/operations/erasure-code/#erasure-codi… >> and enabled overwrites so that RBD can be used. In doing so, I've set >> the data_pool to the erasure coded pool, with set to the replicated >> pool. >> >> I have the following in ceph.conf: >> >> rbd_default_data_pool = libvirt-pool-ec >> >> Here's an rbd info on the image I've created (notice the "data_pool" >> config): >> >> rbd image 'xxxxxxx': >> size 500 GiB in 128000 objects >> order 22 (4 MiB objects) >> snapshot_count: 0 >> id: 7e37e442592b >> data_pool: libvirt-pool-ec >> block_name_prefix: rbd_data.6.7e37e442592b >> format: 2 >> features: layering, exclusive-lock, object-map, fast-diff, >> deep-flatten, data-pool >> op_features: >> flags: >> create_timestamp: Fri Sep 13 16:34:06 2024 >> access_timestamp: Fri Sep 13 16:34:06 2024 >> modify_timestamp: Fri Sep 13 16:34:06 2024 >> The problem, when I attach this rbd image to a VM, I cannot write to >> it at all, I get an I/O error. But, when I create and rbd image >> without the "rbd_default_data_pool = libvirt-pool-ec" setting, I can >> write fine. >> >> Wanted to see if anyone has any ideas, maybe I'm missing something >> simple. Thanks in advance! >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Blocking/Stuck file
by dominik.baack 17 Sep '24

17 Sep '24

Hi, we started upgrading our Ceph Cluster consisting of 7 Nodes from quincy to reef two days ago. This included the upgrade of the underlying OS and several other small changes. After hitting the osd_remove_queue bug, we could recover mostly, but are still in a non-healthy state because of changes on our network. (See attached image) But overall we can mount the filesystem on all nodes, read and write. The Problem now is that at least one file created by slurmctld exist which seems to be somehow compromised during this. It cannot be read or removed on any of the storage or working nodes. All operations (cat, rm, less, ...) are stuck until they are forcefully terminated. I checked all nodes manually and could not identify any process with an open handle to the file. If possible we would like to unblock the file, but removing would also be a possiblity. Cheers Dominik

2 2

Grand Piano 9/17
by Michelle Fangman 17 Sep '24

17 Sep '24

Hello, I'm offering my late husband's Yamaha Piano to any music enthusiast who may appreciate it. If you or someone you know might be interested in receiving this instrument for free, please feel free to reach out to me. Warm regards, Michelle

1 0

Ceph octopus version cluster not starting
by Amudhan P 17 Sep '24

17 Sep '24

Hi, Recently added one disk in Ceph cluster using "ceph-volume lvm create --data /dev/sdX" but the new OSD didn't start. After some rest of the other nodes OSD service also stopped. So, I restarted all nodes in the cluster now after restart. MON, MDS, MGR and OSD services are not starting. Could find any new logs also after restart it is totally silent in all nodes. Could find some logs in Ceph-volume service. Error in Ceph-volume logs :- [2024-09-15 23:38:15,080][ceph_volume.process][INFO ] stderr Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-5 --> Executable selinuxenabled not in PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-33cd42cd-8570-47de-8703-d7cab1acf2ae/osd-block-21968433-bb53-4415-b9e2-fdc36bc4a28e --path /var/lib/ceph/osd/ceph-5 --no-mon-config stderr: failed to read label for /dev/ceph-33cd42cd-8570-47de-8703-d7cab1acf2ae/osd-block-21968433-bb53-4415-b9e2-fdc36bc4a28e: (2) No such file or directory 2024-09-15T23:38:15.059+0530 7fe7767c8100 -1 bluestore(/dev/ceph-33cd42cd-8570-47de-8703-d7cab1acf2ae/osd-block-21968433-bb53-4415-b9e2-fdc36bc4a28e) _read_bdev_label failed to open /dev/ceph-33cd42cd-8570-47de-8703-d7cab1acf2ae/osd-block-21968433-bb53-4415-b9e2-fdc36bc4a28e: (2) No such file or directory --> RuntimeError: command returned non-zero exit status: 1 [2024-09-15 23:38:15,084][ceph_volume.process][INFO ] stderr Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-2 --> Executable selinuxenabled not in PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-9a9b8328-66ad-4997-8b9f-5216b56b73e8/osd-block-ac2ae41d-3b77-4bfd-ba5c-737e4266e988 --path /var/lib/ceph/osd/ceph-2 --no-mon-config stderr: failed to read label for /dev/ceph-9a9b8328-66ad-4997-8b9f-5216b56b73e8/osd-block-ac2ae41d-3b77-4bfd-ba5c-737e4266e988: (2) No such file or directory But I could find " /dev/ceph-9a9b8328-66ad-4997-8b9f-5216b56b73e8/osd-block-ac2ae41d-3b77-4bfd-ba5c-737e4266e988" the path valid and listing folder. Not sure how to proceed or where to start any idea or suggestion ?

3 11

Bucket Notifications v2 & Multisite Redundancy
by Alex Hussein-Kershaw (HE/HIM) 16 Sep '24

16 Sep '24

Hi Folks, I see in the pending release notes for Squid a description of the "notification_v2" feature. ceph/PendingReleaseNotes at main · ceph/ceph (github.com)<https://github.com/ceph/ceph/blob/main/PendingReleaseNotes#L43> I have some concerns about the multisite nature of this feature, I am using Ceph multisite to provide geographic redundancy for my application data. That seems to be a mainline use case. My application is designed to cope if an entire zone fails. My application currently relies on pubsub (and soon to be replaced by bucket notifications) to provide service. More specifically I rely on the ability to semi-regularly automatically reconfigure topics to adjust the HTTP endpoint. I believe with the "notification_v2" feature all adjustments of the topic are routed through the master for metadata site (much like creating an S3 user). Fundamentally I think this breaks the combination of Ceph multisite, notifications_v2 and reconfiguring a topic; you cannot reconfigure a topic unless both sites are up. I tested this, by setting up a deployment, shutting down a site and trying to reconfigure a topic. I get a 500 HTTP response to my createTopic, and can see this in the RGW logs: debug 2024-09-03T08:24:39.997+0000 7f53a3372640 20 ERROR: curl error: Couldn't connect to server req_data->error_buf=Failed to connect to 10.225.20.19 port 7480: No route to host debug 2024-09-03T08:24:39.997+0000 7f5329a7f640 4 req 2968544598970267460 0.686004043s sns:pubsub_topic_create CreateTopic forward_request_to_master returned ret = -2200 Have the implications of this been considered? For now I may be able to disable the notifications_v2 feature, however in the release note the predecessor is deprecated in Squid. Hopefully I'm missing something obvious. Kind regards, Alex

2 8

Metric or any information about disk (block) fragmentation
by Szabo, Istvan (Agoda) 16 Sep '24

16 Sep '24

Hi, Is there any way to pull out the fragmentation value from the disk somehow without shutdown the osd and run the ceph-objectstore-tool command? We had a total outage last week which is based on multiple bugs but most of them are down to disk fragmentation which is super high in our case. I'm trying to find in the logs and in ceph metrics some hint but I couldn't find anything how I can monitor or at least check this value: ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-218/ --command free-score block: { "fragmentation_rating": 0.90989583876508828 } bluefs-db: { "fragmentation_rating": 0.41012391001249843 } failure querying 'bluefs-wal' 2024-09-15T12:19:31.814+0700 7f90c66d00c0 -1 asok(0x55c637e7c000) AdminSocket: request '[{"prefix": "bluestore allocator score bluefs-wal"}]' not defined Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 1

telemetry
by Marc 16 Sep '24

16 Sep '24

maybe add filter for types as well? https://telemetry-public.ceph.com/d/x1_ISxiMz_01/models-per-vendor?orgId=1

1 0