- ceph-users - lists.ceph.io

lun allocation failure
by Opánszki Gábor 25 Aug '23

25 Aug '23

Hi folks, we deployed new reef cluster to our lab. all of the nodes are up and running, but we can't allocate lun to target. on the gui we got "disk create/update failed on ceph-iscsigw0. LUN allocation failure" message. We created images on gui do you have any idea? Thanks root@ceph-mgr0:~# ceph -s *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** cluster: id: ad0aede2-4100-11ee-bc14-1c40244f5c21 health: HEALTH_OK services: mon: 5 daemons, quorum ceph-mgr0,ceph-mgr1,ceph-osd5,ceph-osd7,ceph-osd6 (age 28h) mgr: ceph-mgr0.sapbav(active, since 45h), standbys: ceph-mgr1.zwzyuc osd: 44 osds: 44 up (since 4h), 44 in (since 4h) tcmu-runner: 1 portal active (1 hosts) data: pools: 5 pools, 3074 pgs objects: 27 objects, 453 KiB usage: 30 GiB used, 101 TiB / 101 TiB avail pgs: 3074 active+clean io: client: 2.7 KiB/s rd, 2 op/s rd, 0 op/s wr root@ceph-mgr0:~# root@ceph-mgr0:~# rados lspools .mgr ace1 1T-r3-01 ace0 x root@ceph-mgr0:~# rbd ls 1T-r3-01 111 aaaa bb pool2 teszt root@ceph-mgr0:~# rbd ls x x-a root@ceph-mgr0:~# root@ceph-mgr0:~# rbd info 1T-r3-01/111 rbd image '111': size 1 GiB in 256 objects order 22 (4 MiB objects) snapshot_count: 0 id: 5f927ce161de block_name_prefix: rbd_data.5f927ce161de format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten op_features: flags: create_timestamp: Thu Aug 24 17:33:37 2023 access_timestamp: Thu Aug 24 17:33:37 2023 modify_timestamp: Thu Aug 24 17:33:37 2023 root@ceph-mgr0:~# rbd info 1T-r3-01/aaaa rbd image 'aaaa': size 1 GiB in 256 objects order 22 (4 MiB objects) snapshot_count: 0 id: 5f926a0e299f block_name_prefix: rbd_data.5f926a0e299f format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten op_features: flags: create_timestamp: Thu Aug 24 17:18:06 2023 access_timestamp: Thu Aug 24 17:18:06 2023 modify_timestamp: Thu Aug 24 17:18:06 2023 root@ceph-mgr0:~# rbd info x/x-a rbd image 'x-a': size 1 GiB in 256 objects order 22 (4 MiB objects) snapshot_count: 0 id: 5f922dbdf6c6 block_name_prefix: rbd_data.5f922dbdf6c6 format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten op_features: flags: create_timestamp: Thu Aug 24 17:48:28 2023 access_timestamp: Thu Aug 24 17:48:28 2023 modify_timestamp: Thu Aug 24 17:48:28 2023 root@ceph-mgr0:~# root@ceph-mgr0:~# ceph orch ls --service_type iscsi *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH *** NAME PORTS RUNNING REFRESHED AGE PLACEMENT iscsi.gw-1 ?:5000 2/2 4m ago 6m ceph-iscsigw0;ceph-iscsigw1 root@ceph-mgr0:~# GW: root@ceph-iscsigw0:~# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES d677a8abd2d8 quay.io/ceph/ceph "/usr/bin/rbd-target…" 6 seconds ago Up 5 seconds ceph-ad0aede2-4100-11ee-bc14-1c40244f5c21-iscsi-gw-1-ceph-iscsigw0-fmuyhi ead503586cdd quay.io/ceph/ceph "/usr/bin/tcmu-runner" 6 seconds ago Up 5 seconds ceph-ad0aede2-4100-11ee-bc14-1c40244f5c21-iscsi-gw-1-ceph-iscsigw0-fmuyhi-tcmu 3ae0014bcc41 quay.io/ceph/ceph "/usr/bin/ceph-crash…" About an hour ago Up About an hour ceph-ad0aede2-4100-11ee-bc14-1c40244f5c21-crash-ceph-iscsigw0 1a7bc044ed8a quay.io/ceph/ceph "/usr/bin/ceph-expor…" About an hour ago Up About an hour ceph-ad0aede2-4100-11ee-bc14-1c40244f5c21-ceph-exporter-ceph-iscsigw0 c746a4da2bbb quay.io/prometheus/node-exporter:v1.5.0 "/bin/node_exporter …" About an hour ago Up About an hour ceph-ad0aede2-4100-11ee-bc14-1c40244f5c21-node-exporter-ceph-iscsigw0 root@ceph-iscsigw0:~# docker exec -it d677a8abd2d8 /bin/bash [root@ceph-iscsigw0 /]# gwcli ls o- / ......................................................................................................................... [...] o- cluster ......................................................................................................... [Clusters: 1] | o- ceph ............................................................................................................ [HEALTH_OK] | o- pools .......................................................................................................... [Pools: 5] | | o- .mgr .................................................................. [(x3), Commit: 0.00Y/33602764M (0%), Used: 3184K] | | o- 1T-r3-01 ................................................................ [(x3), Commit: 0.00Y/5793684M (0%), Used: 108K] | | o- ace0 ................................................................... [(2+1), Commit: 0.00Y/11587368M (0%), Used: 24K] | | o- ace1 ................................................................... [(2+1), Commit: 0.00Y/55665220M (0%), Used: 12K] | | o- x ....................................................................... [(x3), Commit: 0.00Y/33602764M (0%), Used: 36K] | o- topology ............................................................................................... [OSDs: 44,MONs: 5] o- disks ....................................................................................................... [0.00Y, Disks: 0] o- iscsi-targets ............................................................................... [DiscoveryAuth: None, Targets: 1] o- iqn.2001-07.com.ceph:1692892702115 ................................................................ [Auth: None, Gateways: 2] o- disks .......................................................................................................... [Disks: 0] o- gateways ............................................................................................ [Up: 2/2, Portals: 2] | o- ceph-iscsigw0 ............................................................................ [10.202.5.21,10.202.4.21 (UP)] | o- ceph-iscsigw1 ............................................................................ [10.202.3.21,10.202.2.21 (UP)] o- host-groups .................................................................................................. [Groups : 0] o- hosts ....................................................................................... [Auth: ACL_ENABLED, Hosts: 0] [root@ceph-iscsigw0 /]#

2 2

Problem when configuring S3 website domain go through Cloudflare DNS proxy
by Huy Nguyen 24 Aug '23

24 Aug '23

Hi, Currently, I trying to create a CNAME record point to a s3 website, for example: s3.example.com => s3.example.com.s3-website.myceph.com. So in this way, my subdomain s3. will have https. But then only http works. If I go to https://s3.example.com, it shows the metadata of index.html: This XML file does not appear to have any style information associated with it. The document tree is shown below. <ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> <Name>s3.example.com</Name> <Prefix/> <MaxKeys>1000</MaxKeys> <IsTruncated>false</IsTruncated> <Contents> <Key>index.html</Key> <LastModified>2023-08-24T10:03:14.046Z</LastModified> <ETag>"8e26caf000875221bf89d95f7f244927"</ETag> <Size>295</Size> <StorageClass>STANDARD</StorageClass> <Owner> <ID>d92ac19d934a4e9b90e7707372c64996</ID> <DisplayName>foo(a)example.com</DisplayName> </Owner> <Type>Normal</Type> </Contents> <Marker/> </ListBucketResult> Here is my rgw configuration: rgw_resolve_cname = true rgw_enable_static_website = true rgw_dns_s3website_name = ss-website.example.com rgw_trust_forwarded_https = true So how to make the https show the content of index.html (not its metadata)? Thanks in advance.

1 1

User + Dev Monthly Meeting Minutes 2023-08-24
by Laura Flores 24 Aug '23

24 Aug '23

Is there going to be another Pacific point release (16.2.14) in the pipeline? - Yes, 16.2.14 is going through QA right now. See https://www.spinics.net/lists/ceph-users/msg78528.html for updates. Need pacific backport for https://tracker.ceph.com/issues/59478 - Laura will check on this, although a Pacific backport is unlikely due to incompatibilities from the scrub backend refactoring. There are inconsistencies with the `ceph config dump` normal vs. json output. A fix has been proposed in https://tracker.ceph.com/issues/62379. Question for users: Will this change break any existing automation? See the tracker for more details, and reach out to @Sridhar Seshasayee <sseshasa(a)redhat.com> with any questions. -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

2 2

Rados object transformation
by Yixin Jin 24 Aug '23

24 Aug '23

Hi folks, Is it possible through object classes to transform object content? For example, I'd like this transformer to change the content of the object when it is read and when it is written. In this way, I can potentially encrypt the object content in storage without the need to make ceph/osd to do encryption/decryption. It could be taken care of by the object class itself. Thanks,Yixin

2 2

snaptrim number of objects
by Angelo Höngens 23 Aug '23

23 Aug '23

Hey guys, I'm trying to figure out what's happening to my backup cluster that often grinds to a halt when cephfs automatically removes snapshots. Almost all OSD's go to 100% CPU, ceph complains about slow ops, and CephFS stops doing client i/o. I'm graphing the cumulative value of the snaptrimq_len value, and that slowly decreases over time. One night it takes an hour, but other days, like today, my cluster has been down for almost 20 hours, and I think we're half way. Funny thing is that in both cases, the snaptrimq_len value initially goes to the same value, around 3000, and then slowly decreases, but my guess is that the number of objects that need to be trimmed varies hugely every day. Is there a way to show the size of cephfs snapshots, or get the number of objects or bytes that need snaptrimming? Perhaps I can graph that and see where the differences are. That won't explain why my cluster bogs down, but at least it gives some visibility. Running 17.2.6 everywhere by the way. Angelo.

7 9

User + Dev Monthly Meeting happening next week
by Laura Flores 23 Aug '23

23 Aug '23

Hi everyone, The User + Dev Monthly Meeting is happening next week on* Thursday, August 24th* *@ 2:00 PM UTC *at this link: https://meet.jit.si/ceph-user-dev-monthly (Note that the date has been rescheduled from the original date, August 17th.) Please add any topics you'd like to discuss to the agenda: https://pad.ceph.com/p/ceph-user-dev-monthly-minutes Thanks, Laura Flores -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage <https://ceph.io> Chicago, IL lflores(a)ibm.com | lflores(a)redhat.com <lflores(a)redhat.com> M: +17087388804

1 1

Patch change for CephFS subvolume
by Michal Strnad 23 Aug '23

23 Aug '23

Hi! I'm trying to figure out how to specify the path for a CephFS subvolume, as it's intended to represent a user's home directory. By default, it's located at /volumes/_nogroup/$NAME/$UUID. Is it possible to change this path somehow, or is using symbolic links the only option? Thank you Michal Strnad

3 5

[quincy] Migrating ceph cluster to new network, bind OSDs to multple public_nework
by Boris Behrens 23 Aug '23

23 Aug '23

Hi, I need to migrate a storage cluster to a new network. I added the new network to the ceph config via: ceph config set global public_network "old_network/64, new_network/64" I've added a set of new mon daemons with IP addresses in the new network and they are added to the quorum and seem to work as expected. But when I restart the OSD daemons, the do not bind to the new addresses. I would have expected that the OSDs try to bind to all networks but they are only bound to the old_network. The idea was to add the new set of network config to the current storage hosts, bind everything to ip addresses in both networks, shift over workload, and then remove the old network.

4 11

Re: Listing S3 buckets of a tenant using admin API
by Michal Strnad 23 Aug '23

23 Aug '23

Hi. Did you manage to access the specific bucket? Is it possible and if so, how to obtain information about a bucket storage_class and placement through the API, in addition to the regular information? Thanks Michal On 2/1/22 18:01, Daniel Iwan wrote: > It seems that REST API and CLI support listing either all buckets in the > system or buckets owned by a specific user by providing uid. > https://docs.ceph.com/en/latest/radosgw/adminops/#get-bucket-info > > I would like to list bucket of a specific tenant, possibly with stats, > therefore I would prefer to avoid fetching information for all buckets and > filtering in the application layer. > > Looks like --tenant in CLI would be ideal, but at the moment it expects > --uid also be provided. > > Is there any way to achieve that at the moment? > I'm on Ceph 16.2.7 > > Regards > Daniel Iwan > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Create OSDs MANUALLY
by Alfredo Rezinovsky 23 Aug '23

23 Aug '23

SSD driver works awful when full. Even if I set DB to ssd for 4 OSDs and theres 2 the dashboard daemon allocates all the ssd. I want to partition only 70% of the SSD for DB/WAL and leave the rest for SSD maneouvering. There's a way to create an OSD telling manually disk or partitions to user for data and DB (like the way I used to do it with ceph-deploy)? -- Alfrenovsky

2 1