- ceph-users - lists.ceph.io

User + Dev monthly meeting happening tomorrow, Feb. 16th!
by Laura Flores 16 Feb '23

16 Feb '23

Hi Ceph Users, The User + Dev monthly meeting is coming up tomorrow, Thursday, Feb. 16th at 3:00 PM UTC. Please add any topics you'd like to discuss to the agenda: https://pad.ceph.com/p/ceph-user-dev-monthly-minutes <https://pad.ceph.com/p/ceph-user-dev-monthly-minutes#L13> See you there, Laura Flores -- Laura Flores She/Her/Hers Software Engineer, Ceph Storage Red Hat Inc. <https://www.redhat.com> Chicago, IL lflores(a)redhat.com M: +17087388804 @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

1 1

RGW archive zone lifecycle
by ondrej＠kuuk.la 16 Feb '23

16 Feb '23

Hi, I have two Ceph clusters in a multi-zone setup. The first one (master zone) would be accessible to users for their interaction using RGW. The second one is set to sync from the master zone with the tier type of the zone set as an archive (to version all files). My question here is. Is there an option to set a lifecycle for the version files saved on the archive zone? For example, keep only 5 versions per file or delete version files older than one year? Thanks a lot.

2 1

how to sync data on two site CephFS
by zxcs 16 Feb '23

16 Feb '23

Hi, Experts, we already have a CephFS cluster, called A, and now we want to setup another CephFS cluster(called B) in other site. And we need to synchronize data with each other for some directory(if all directory can synchronize , then very very good), Means when we write a file in A cluster, this file can auto sync to B cluster, and when we create a file or directory on B Cluster, this file or directory can auto sync to A Cluster. our question is does there any best practices to do that on CephFS? Thanks in advance! Thanks, zx

3 2

Extremally need help. Openshift cluster is down :c
by kreept.sama＠gmail.com 16 Feb '23

16 Feb '23

Hello everyone and sorry. Maybe someone has already faced this problem. A day ago, we restored our Openshift cluster, however, at the moment, the PVCs cannot connect to the pod. We looked at the status of the ceph and found that our MDS were in standby mode, then found that the metadata was corrupted. After some manipulations, we were able to turn on our MDS daemons, but there is still no record on the cluster, the ceph status command shows the following. sh-4.4$ ceph -s cluster: id: 9213604e-b0b6-49d5-bcb3-f55ab3d79119 health: HEALTH_ERR 1 MDSs report damaged metadata 1 MDSs are read only 6 daemons have recently crashed services: mon: 5 daemons, quorum bd,bj,bm,bn,bo (age 26h) mgr: a(active, since 25h) mds: 1/1 daemons up, 1 hot standby osd: 9 osds: 9 up (since 41h), 9 in (since 42h) rgw: 1 daemon active (1 hosts, 1 zones) data: volumes: 1/1 healthy pools: 10 pools, 225 pgs objects: 1.60M objects, 234 GiB usage: 606 GiB used, 594 GiB / 1.2 TiB avail pgs: 225 active+clean io: client: 852 B/s rd, 1 op/s rd, 0 op/s wr Now we trying to follow this instructions: https://docs.ceph.com/en/latest/cephfs/disaster-recovery-experts/#recovery-… What else have we tried: cephfs-journal-tool --rank=1:0 event recover_dentries summary cephfs-journal-tool --rank=1:0 journal reset cephfs-table-tool all reset session ceph tell mds.gml--cephfs-a scrub start / recursive repair force ceph tell mds.gml--cephfs-b scrub start / recursive repair force ceph mds repaired 0 ceph tell mds.gml--cephfs-a damage ls [ { "damage_type": "dir_frag", "id": 26851730, "ino": 1100162409473, "frag": "*", "path": "/volumes/csi/csi-vol-5ad18c03-3205-11ed-9ba7-0a580a810206/e5664004-51e0-4bff-85c8-029944b431d8/store/096/096a1497-78ab-4802-a5a7-d09e011fd3a5/202301_1027796_1027796_0" }, ……… { "damage_type": "dir_frag", "id": 118336643, "ino": 1100162424469, "frag": "*", "path": "/volumes/csi/csi-vol-5ad18c03-3205-11ed-9ba7-0a580a810206/e5664004-51e0-4bff-85c8-029944b431d8/store/096/096a1497-78ab-4802-a5a7-d09e011fd3a5/202301_1027832_1027832_0" }, Now we trying: # Session table cephfs-table-tool 0 reset session # SnapServer cephfs-table-tool 0 reset snap # InoTable cephfs-table-tool 0 reset inode # Journal cephfs-journal-tool --rank=0 journal reset # Root inodes ("/" and MDS directory) cephfs-data-scan init cephfs-data-scan scan_extents <data pool> cephfs-data-scan scan_inodes <data pool> cephfs-data-scan scan_links Is it right way and cant it be our salvation? Thank you!

2 10

Renaming a ceph node
by Rice, Christian 16 Feb '23

16 Feb '23

Can anyone please point me at a doc that explains the most efficient procedure to rename a ceph node WITHOUT causing a massive misplaced objects churn? When my node came up with a new name, it properly joined the cluster and owned the OSDs, but the original node with no devices remained. I expect this affected the crush map such that a large qty of objects got reshuffled. I want no object movement, if possible. BTW this old cluster is on luminous. ☹

3 4

Ceph (cepadm) quincy: can't add osd from remote nodes.
by Anton Chivkunov 15 Feb '23

15 Feb '23

Hello! I stuck with a problem, while trying to create cluster of 3 nodes (AWS EC2 instancies): fa11 ~ # ceph orch host ls HOST ADDR LABELS STATUS fa11 172.16.24.67 _admin fa12 172.16.23.159 _admin fa13 172.16.25.119 _admin 3 hosts in cluster Each of them have 2 disks (all accepted by CEPH): fa11 ~ # ceph orch device ls HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS fa11 /dev/nvme1n1 ssd Amazon_Elastic_Block_Store_vol016651cf7f3b9c9dd 8589M Yes 7m ago fa11 /dev/nvme2n1 ssd Amazon_Elastic_Block_Store_vol034082d7d364dfbdb 5368M Yes 7m ago fa12 /dev/nvme1n1 ssd Amazon_Elastic_Block_Store_vol0ec193fa3f77fee66 8589M Yes 3m ago fa12 /dev/nvme2n1 ssd Amazon_Elastic_Block_Store_vol018736f7eeab725f5 5368M Yes 3m ago fa13 /dev/nvme1n1 ssd Amazon_Elastic_Block_Store_vol0443a031550be1024 8589M Yes 84s ago fa13 /dev/nvme2n1 ssd Amazon_Elastic_Block_Store_vol0870412d37717dc2c 5368M Yes 84s ago fa11 is first host, where from I manage cluster. Adding OSD from fa11 itself works fine: fa11 ~ # ceph orch daemon add osd fa11:/dev/nvme1n1 Created osd(s) 0 on host 'fa11' But it doesn't work for other 2 hosts (it hangs forever): fa11 ~ # ceph orch daemon add osd fa12:/dev/nvme1n1 ^CInterrupted Logs on fa12 shows that it hangs at following step: fa12 ~ # tail /var/log/ceph/a9ef6c26-ac38-11ed-9429-06e6bc29c1db/ceph-volume.log ... [2023-02-14 07:38:20,942][ceph_volume.process][INFO ] Running command: /usr/bin/ceph-authtool --gen-print-key [2023-02-14 07:38:20,964][ceph_volume.process][INFO ] Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new a51506c2-e910-4763-9a0c-f6c2194944e2 I'm not sure what might be the reason for this hanging? *Additional details: 1) cephadm installed, using curl (https://docs.ceph.com/en/quincy/cephadm/install/#curl-based-installation) 2) I use user "ceph", instead of "root" and port 2222 instead of 22. First node was bootstrapped, using below command: cephadm bootstrap --mon-ip 172.16.24.67 --allow-fqdn-hostname --ssh-user ceph --ssh-config /home/anton/ceph/ssh_config --cluster-network 172.16.16.0/20 --skip-monitoring-stack Content of /home/anton/ceph/ssh_config: fa11 ~ # cat /home/anton/ceph/ssh_config Host * User ceph Port 2222 IdentityFile /home/ceph/.ssh/id_rsa StrictHostKeyChecking no UserKnownHostsFile=/dev/null 3) Hosts fa12 and fa13 were added, using commnds: ceph orch host add fa12.testing.swiftserve.com 172.16.23.159 --labels _admin ceph orch host add fa13.testing.swiftserve.com 172.16.25.119 --labels _admin Thanks in advance! BR/Anton

3 3

Swift Public Access URL returns "NoSuchBucket" when rgw_swift_account_in_url is True
by Beaman, Joshua 15 Feb '23

15 Feb '23

Greetings, Regarding: https://tracker.ceph.com/issues/58019 And: https://github.com/ceph/ceph/pull/47341 We have completed upgrading our production pacific clusters to 16.2.11 and are still experiencing this bug. Can anyone confirm if this backport was included in 16.2.11. Could there be anything wrong with our configuration that is contributing? Thank you, Josh Beaman

1 0

Missing object in bucket list
by mahnoosh shahidi 15 Feb '23

15 Feb '23

Hi all, We have a cluster on 15.2.12. We are experiencing an unusual scenario in S3. User send PUT request to upload an object and RGW returns 200 as a response status code. The object has been uploaded and can be downloaded but it does not exist in the bucket list. We also tried to get the bucket index entry for that object but it does not exist. Below is the log of the RGW for the request. 1 ====== starting new request req=0x7f246c4426b0 ===== > 2 req 44161 0s initializing for trans_id = > tx00000000000000000ac81-0063e36653-17e18f0-default > 10 rgw api priority: s3=3 s3website=2 > 10 host=192.168.0.201 > 10 meta>> HTTP_X_AMZ_CONTENT_SHA256 > 10 meta>> HTTP_X_AMZ_DATE > 10 x>> > x-amz-content-sha256:30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58 > 10 x>> x-amz-date:20230208T090731Z > 10 handler=22RGWHandler_REST_Obj_S3 > 2 req 44161 0s getting op 1 > 10 req 44161 0s s3:put_obj scheduling with dmclock client=2 cost=1 > 10 op=21RGWPutObj_ObjStore_S3 > 2 req 44161 0s s3:put_obj verifying requester > 10 v4 signature format = > 7daee7e343e08d8121e843c6c77da3cc827bd4f4f179548e1c729c130a3e7745 > 10 v4 credential format = > 85ZYESW8HS34DC95MZBT/20230208/us-east-1/s3/aws4_request > 10 access key id = 85ZYESW8HS34DC95MZBT > 10 credential scope = 20230208/us-east-1/s3/aws4_request > 10 req 44161 0s canonical headers format = > content-md5:ttgbNgpWctgMJ0MPORU+LA== > host:192.168.0.201 > > x-amz-content-sha256:30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58 > x-amz-date:20230208T090731Z > > 10 payload request hash = > 30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58 > 10 canonical request = PUT > /test7/file508294 > > content-md5:ttgbNgpWctgMJ0MPORU+LA== > host:192.168.0.201 > > x-amz-content-sha256:30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58 > x-amz-date:20230208T090731Z > > content-md5;host;x-amz-content-sha256;x-amz-date > 30e14955ebf1352266dc2ff8067e68104607e750abb9d3b36582b8af909fcb58 > 10 canonical request hash = > 2ab4fe4f0fa402435c3237382bdc77e86203406e90a2768a70410f58754bb6ba > 10 string to sign = AWS4-HMAC-SHA256 > 20230208T090731Z > 20230208/us-east-1/s3/aws4_request > 2ab4fe4f0fa402435c3237382bdc77e86203406e90a2768a70410f58754bb6ba > 10 req 44161 0s delaying v4 auth > 10 date_k = > a9dc6afa32600995d313f1b6a4fa40be3a3cd574d25db8789ac966a8e7f43356 > 10 region_k = > b9193e8e261f702b88549da7e81e6a4a7672725996ea8a86269fed665b39670d > 10 service_k = > 34214c91aec1192bcc413e02044e346b31ed4f13df8c15830bdb1d7bd3565126 > 10 signing_k = > 7656d62334d92c982f8c21e0200e760054b214eebab6dbeab577fb655c00a6f4 > 10 generated signature = > 7daee7e343e08d8121e843c6c77da3cc827bd4f4f179548e1c729c130a3e7745 > 2 req 44161 0s s3:put_obj normalizing buckets and tenants > 10 s->object=file508294 s->bucket=test7 > 2 req 44161 0s s3:put_obj init permissions > 10 cache get: name=default.rgw.meta+root+test7 : expiry miss > 10 cache put: name=default.rgw.meta+root+test7 info.flags=0x16 > 10 adding default.rgw.meta+root+test7 to cache LRU end > 10 updating xattr: name=ceph.objclass.version bl.length()=42 > 10 cache get: name=default.rgw.meta+root+test7 : type miss > (requested=0x11, cached=0x16) > 10 cache put: name=default.rgw.meta+root+test7 info.flags=0x11 > 10 moving default.rgw.meta+root+test7 to cache LRU end > 10 cache get: name=default.rgw.meta+users.uid+storage : hit > (requested=0x6, cached=0x17) > 10 cache get: name=default.rgw.meta+users.uid+storage : hit > (requested=0x3, cached=0x17) > 2 req 44161 0.003999945s s3:put_obj recalculating target > 2 req 44161 0.003999945s s3:put_obj reading permissions > 2 req 44161 0.003999945s s3:put_obj init op > 2 req 44161 0.003999945s s3:put_obj verifying op mask > 2 req 44161 0.003999945s s3:put_obj verifying op permissions > 5 req 44161 0.003999945s s3:put_obj Searching permissions for > identity=rgw::auth::SysReqApplier -> > rgw::auth::LocalApplier(acct_user=storage, acct_name=storage, subuser=, > perm_mask=15, is_admin=0) mask=50 > 5 Searching permissions for uid=storage > 5 Found permission: 15 > 5 Searching permissions for group=1 mask=50 > 5 Permissions for group not found > 5 Searching permissions for group=2 mask=50 > 5 Permissions for group not found > 5 req 44161 0.003999945s s3:put_obj -- Getting permissions done for > identity=rgw::auth::SysReqApplier -> > rgw::auth::LocalApplier(acct_user=storage, acct_name=storage, subuser=, > perm_mask=15, is_admin=0), owner=storage, perm=2 > 37:31.066+0330 7f2479c5e700 10 req 44161 0.003999945s s3:put_obj > identity=rgw::auth::SysReqApplier -> > rgw::auth::LocalApplier(acct_user=storage, acct_name=storage, subuser=, > perm_mask=15, is_admin=0) requested perm (type)=2, policy perm=2, > user_perm_mask=2, acl perm=2 > 2 req 44161 0.003999945s s3:put_obj verifying op params > 2 req 44161 0.003999945s s3:put_obj pre-executing > 2 req 44161 0.003999945s s3:put_obj executing > 5 req 44161 0.023999668s s3:put_obj NOTICE: call to > do_aws4_auth_completion > 10 req 44161 0.023999668s s3:put_obj v4 auth ok -- do_aws4_auth_completion > 5 req 44161 0.023999668s s3:put_obj NOTICE: call to > do_aws4_auth_completion > -- > 0 RGWReshardLock::lock failed to acquire lock on > test7:c3b354de-c79f-444b-a647-5b272f8148d7.25388922.1 ret=-16 > 0 lifecycle: RGWLC::process() failed to acquire lock on lc.30, sleep 5, > try again > 0 lifecycle: RGWLC::process() failed to acquire lock on lc.30, sleep 5, > try again > 0 lifecycle: RGWLC::process() failed to acquire lock on lc.30, sleep 5, > try again > 0 RGWReshardLock::lock failed to acquire lock on > test7:c3b354de-c79f-444b-a647-5b272f8148d7.25388922.1 ret=-16 > 10 cache get: name=default.rgw.meta+root+test7 : hit (requested=0x6, > cached=0x17) > 10 cache get: name=default.rgw.meta+root+test7 : hit (requested=0x1, > cached=0x17) > -1 WARNING: The bucket info cache is inconsistent. This is a failure that > should be debugged. I am a nice machine, so I will try to recover. > 10 cache get: > name=default.rgw.meta+root+.bucket.meta.test7:c3b354de-c79f-444b-a647-5b272f8148d7.25388922.1 > : hit (requested=0x16, cached=0x17) > 10 cache get: > name=default.rgw.meta+root+.bucket.meta.test7:c3b354de-c79f-444b-a647-5b272f8148d7.25388922.1 > : hit (requested=0x13, cached=0x17) > 10 cache put: > name=default.rgw.meta+root+.bucket.meta.test7:c3b354de-c79f-444b-a647-5b272f8148d7.25388922.1 > info.flags=0x13 > 10 moving > default.rgw.meta+root+.bucket.meta.test7:c3b354de-c79f-444b-a647-5b272f8148d7.25388922.1 > to cache LRU end > 10 updating xattr: name=ceph.objclass.version bl.length()=42 > 10 updating xattr: name=user.rgw.acl bl.length()=147 > 10 chain_cache_entry: > cache_locator=default.rgw.meta+root+.bucket.meta.test7:c3b354de-c79f-444b-a647-5b272f8148d7.25388922.1 > -1 WARNING: The OSD has the same version I have. Something may have gone > squirrelly. An administrator may have forced a change; otherwise there is a > problem somewhere. > 0 lifecycle: RGWLC::process() failed to acquire lock on lc.30, sleep 5, > try again > 0 lifecycle: RGWLC::process() failed to acquire lock on lc.30, sleep 5, > try again > 10 manifest: total_size = 1048576 > 10 setting object > write_tag=c3b354de-c79f-444b-a647-5b272f8148d7.25041136.44161 > 10 cache get: name=default.rgw.log++bucket.sync-source-hints.test7 : hit > (negative entry) > 10 cache get: name=default.rgw.log++bucket.sync-target-hints.test7 : hit > (negative entry) > 10 chain_cache_entry: cache_locator= > 10 cache get: > name=default.rgw.log++pubsub.user.storage.bucket.test7/c3b354de-c79f-444b-a647-5b272f8148d7.25388922.1 > : hit (negative entry) > 2 req 44161 363.150981592s s3:put_obj completing > 4 write_data failed: Connection reset by peer > 0 ERROR: RESTFUL_IO(s)->complete_header() returned err=Connection reset > by peer > 2 req 44161 363.150981592s s3:put_obj op status=0 > 2 req 44161 363.150981592s s3:put_obj http status=200 > 1 ====== req done req=0x7f246c4426b0 op status=0 http_status=200 > latency=363.150981592s ====== Anybody has any idea about the reason for this behaviour? Best Regards, Mahnoosh

2 2

Re: iDRAC 9 version 6.10 shows 0% for write endurance on non-dell drives, work around?
by Konstantin Shalygin 14 Feb '23

14 Feb '23

Hi, You can use smartctl_exporter [1] for all your media, not only the SSD k [1] https://github.com/prometheus-community/smartctl_exporter Sent from my iPhone > On 14 Feb 2023, at 23:01, Drew Weaver <drew.weaver(a)thenap.com> wrote: > Hello, > > After upgrading a lot of iDRAC9 modules to version 6.10 in servers that are involved in a Ceph cluster we noticed that the iDRAC9 shows the write endurance as 0% on any non-certified disk. > > OMSA still shows the correct remaining write endurance but I am assuming that they are working feverishly to eliminate that too. > > I opened a support ticket with Dell once this was brought to my attention and they basically told me that I was lucky that it ever worked at all, which I thought was an odd response given that the iDRAC enterprise licenses cost several hundred dollars each. > > I know that the old Intel Datacenter Tool used to be able to reach through a MegaRAID controller and read the remaining write endurance but that tool is essentially defunct now. > > What are you folks using to monitor your write endurance on your SSDs that you couldn't buy from Dell because they had a 16 week lead time while the MFG could deliver the drives in 3 days? > > Thanks, > -Drew > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: Announcing go-ceph v0.17.0
by Sven Anderson 14 Feb '23

14 Feb '23

We are happy to announce another release of the go-ceph API library. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.20.0 Changes include additions to the rbd, rgw and cephfs packages. More details are available at the link above. The library includes bindings that aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. The library also includes additional APIs that can be used to administer cephfs, rbd, and rgw subsystems. There are already a few consumers of this library in the wild, including the ceph-csi project. Sven

1 1