- ceph-users - lists.ceph.io

The num of objects with cmd "rados -p xxx" not equal with s3 api?
by Louis Koo 12 Jun '23

12 Jun '23

List objects with rados cmd: "rados -p oss.rgw.buckets.index ls | grep "c2af65dc-b456-4f5a-be6a-2a142adeea75.335721.1" | awk '{print "rados listomapkeys -p oss.rgw.buckets.index "$1 }'|sh -x" some objects like this, I don't know what's it "?1000_DR-MON-1_20220307_224723/configs/lidars.cfgiUFWot6m-OJzb4cl6fC4czRzv9tGJMLw ?1000_DR-MON-1_20220307_224723/logs/sentry/DR_MON_1_20220307_224724_carstatus.jsoniYf0RYRvfYXWwwr6U8JDQ59ZSN2SLXg5 ?1000_MONDEO_A12/DR-MKZ-1_20220114_191640/coredumps/i.b2Je.JKFrSJAmlga7IIXLyKFSmXgAo ?1000_MONDEO_A12/DR-MKZ-1_20220114_191640/coredumps/iCuATmyPL9vXXfsSHENNkIpAqS.EZKFj ?1000_MONDEO_A12/DR-MKZ-1_20220211_190901/logs/gnss_adapter/i7g.vodrYiNsy4ljwy6UsQNptRHoY4Uz ?1000_MONDEO_A12/DR-MKZ-1_20220211_190901/logs/gnss_adapter/izHgzCRwJY2eGqV0ifuz9DaS50p9zmjA ?1000_MONDEO_A12/DR-MON-1_20211002_012629/config/iEYp.8-exWsChWoEGKTPBRzfFcm8EmK4 ?1000_MONDEO_A12/DR-MON-1_20211005_011823/log_other/iikTrMOJno2pKZFzxhBlJ04TqTrv37Be ?1000_MONDEO_A12/DR-MON-1_20211019_181039/task_information/iiyyJUW86YITRf023drwHV5TBNlxCIwJ ?1000_MONDEO_A12/DR-MON-1_20211019_181039/task_information/int4GOdWMZ7zXddTVtChac1TlbgwgJpJ ?1000_MONDEO_A12/DR-MON-1_20211110_235107_OT1495/config/irVFxhMgTrTI.cRhUPdt3YxMg-pX.GVy ?1000_MONDEO_A12/DR-MON-1_20211110_235107_OT1495/config/ixGn1hqiDGPR1IOmhW6aCGLJPELJAHNW ?1000_MONDEO_A12/DR-MON-1_20211130_230853/bag/important/ia7X4HvOe4VTk6i6jrgOLpAGzvUDlCpz ?1000_MONDEO_A12/DR-MON-1_20211130_230853/bag/important/ilyff1HsUSYEuWNr4smOpfS2lDyY.CoZ ?1000_MONDEO_A12/DR-MON-1_20211130_230853/task_information/task_description.jsoniDMuGFwTNPY2fTKuDIqSlVfy5AY6AydN ?1000_MONDEO_A12/DR-MON-1_20211203_203345/bag/bags_info.jsoniBEU6igyLo59VwAijFYQVl.iJMUP5s6F ?1000_MONDEO_A12/DR-MON-1_20211213_233958/config/lidars.cfgi.vISYc4d7gOj-qv2grSBhSZJ4LAPIti ?1000_MONDEO_A12/DR-MON-1_20211214_001357/task_information/sentry_config.jsonio8sOurJMZZReHUMu4jx116dz2buWwS0 ?1000_MONDEO_A12/DR-MON-1_20211215_195308/bag/important/iCOoK1O1ydDYHgDQsXAfiCdKc84gglEb ?1000_MONDEO_A12/DR-MON-1_20211215_195308/bag/important/iTQxsDHPef1UGyMmgQL9RNQBlxLHEHyT ?1000_MONDEO_A12/DR-MON-1_20211220_223504/log/DR_MON_1_20211220_223504_syslog.jsonig-E8J2jUe2IekrF8EIOebxaC3HZHWeF ?1000_MONDEO_A12/DR-MON-1_20211220_223504/log/DR_MON_1_20211220_223504_syslog.jsoniw2cN-1wvaz35.fkanuE.zEke95.UZ8V ?1000_MONDEO_A12/DR-MON-1_20211228_200413/log_other/gnss_imu_time.logiCL3l2NTALD2Kyi-KbxYAP4oQg.m6qjN ?1000_MONDEO_A12/DR-MON-1_20211228_200413/log_other/gnss_imu_time.logitAPux7n2MeOIGLejTDnCyirEpaQoQuQ ?1000_MONDEO_A12/DR-MON-1_20220104_195217/config/versioni0QqTxQh7EdOCgTIU3KHgaZfMXiAAQXF ?1000_MONDEO_A12/DR-MON-1_20220104_195217/config/versionida0hUxK8TREImCjsG" total objects: 395797 List obejects with s3 api: total objects: 67498

1 0

what are the options for config a CephFS client session
by Denis Polom 12 Jun '23

12 Jun '23

Hi, I didn't find any doc and any way how to get to know valid options to configure client session over mds socket: #> ceph tell mds.mds1 session config session config <client_id:int> <option> [<value>] : Config a CephFS client session Any hint on this? Thank you

2 1

Container image of Pacific latest version
by mahnoosh shahidi 12 Jun '23

12 Jun '23

Hi all, It seems the latest Pacific image in the registry is 16.2.11. Is there any plan to push the latest version of Pacific (16.2.13) in near future? Best Regards, Mahnoosh

3 4

Operations: cannot update immutable features
by Adam Boyhan 12 Jun '23

12 Jun '23

I have a small cluster on Pacific with roughly 600 RBD images. Out of those 600 images I have 2 which are in a somewhat odd state. root@cephmon:~# rbd info Cloud-Ceph1/vm-134-disk-0 rbd image 'vm-134-disk-0': size 1000 GiB in 256000 objects order 22 (4 MiB objects) snapshot_count: 11 id: 7c326b8b4567 block_name_prefix: rbd_data.7c326b8b4567 format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, operations op_features: snap-trash flags: create_timestamp: Fri Aug 14 07:11:44 2020 access_timestamp: Thu Jun 8 06:31:06 2023 modify_timestamp: Thu Jun 8 06:31:11 2023 Specifically, the feature "operations". I never set this feature; and I don't see it listed in documentation. This feature is causing my backup software to not be able to backup the rdb. Otherwise, it's working aok. I did attempt to remove the feature. root@cephmon:~# rbd feature disable Cloud-Ceph1/vm-464-disk-0 operations rbd: failed to update image features: 2023-06-08T07:50:21.899-0400 7fdea52ae340 -1 librbd::Operations: cannot update immutable features (22) Invalid argument Any help or input is great appreciated. This message and any attachments may contain information that is protected by law as privileged and confidential, and is transmitted for the sole use of the intended recipient(s). If you are not the intended recipient, you are hereby notified that any use, dissemination, copying or retention of this e-mail or the information contained herein is strictly prohibited. If you received this e-mail in error, please immediately notify the sender by e-mail, and permanently delete this e-mail.

2 1

Removed host still active, sort of?
by Jeremy Hansen 11 Jun '23

11 Jun '23

I’m going through the process of transitioning to new hardware. Pacific 16.2.11. I drained the host, all daemons were removed. Did the ceph orch host rm <hostname> [ceph: root@cn01 /]# ceph orch host rm cn03.ceph Error EINVAL: host cn03.ceph does not exist Yet I see it here: ceph osd crush tree |grep cn03 -10 0 host cn03 Web interface says: ceph health HEALTH_WARN 1 stray host(s) with 1 daemon(s) not managed by cephadm ceph orch host rm cn03.ceph --force Error EINVAL: host cn03.ceph does not exist No daemons are running. Something has a bad state. What can I do to clear this up? The previous host went without a problem and when all services were drained and I did the remove, it just completely disappeared as expected. Thanks -jeremy

1 3

Ceph drain not removing daemons
by Jeremy Hansen 09 Jun '23

09 Jun '23

I’m doing a drain on a host using cephadm, Pacific, 16.2.11. ceph orch host drain removed all the OSDs, but these daemons remain: grafana.cn06 cn06.ceph.la1 *:3000 stopped 5m ago 18M - - <unknown> <unknown> <unknown> mds.btc.cn06.euxhdu cn06.ceph.la1 running (2d) 5m ago 17M 29.4M - 16.2.11 de4b0b384ad4 017f7ef441ff mgr.cn06.rpkpwg cn06.ceph.la1 *:8443,9283 running (2d) 5m ago 10M 223M - 16.2.11 de4b0b384ad4 f1b89b453ef3 I manually stopped grafana. I expected these daemons to be removed as well. Is there an extra step I need to do here so I can remove the host cleanly? Thanks! -jeremy

1 1

Re: Disks are filling up
by Omar Siam 09 Jun '23

09 Jun '23

TL;DR: We could not fix this problem in the end and ended up with a Ceph fs in read only mode (so we could only backup, delete and restore) and one broken OSD (we deleted that and restored to a "new disk") I can now wrap up my whole experience with this problem. After the OSD usage growing to almost 2 TB x 3 OSDs (for a data that 'du' counted to be about 120GB), Ceph stopped filling up and in the week or two that followed, most of the space that was used showed up as free again. But there was one OSD that did not free up to meaningful values. To my surprise it was an OSD that is backed by SSDs not the one on HDD. It seems the biggest contributing factor was that I created the pools for the Ceph fs with autoscaling set to on (the default in cephadm dashboard). Now this pool never grew to more than 1 PG although it had a little over 100 GB on it. From what I read on this list this alone is prone to lock contention and other problematic behavior. Lessons learned: * If you use pool autoscaling, watch it if it actually does that. -> I opted for setting the new PG number for the replacement pools to 32 manually. * Ceph fs has a read only mode that at least lets you back up data in some bad states. -> That is good to know. It allows at least administrators to copy data to other storage devices. * If you use Ceph fs for persistent volumes in Kubernetes be aware that you probably loose all volumes at the same time when Ceph fs switches to read only. The CSI for Ceph fs does not work on read only Ceph fs, it always writes xattr on mount (the data pool that should be used and other internal data) and gives up if that fails. -> use a reasonable number of Ceph fs for Kubernetes persistent storage so you don't loose all PV at once. Other problems with my configuration I found: * We suffer from VMWare ESXi misreporting the type of disk physically attached. It reports some SAS SSDs as HDD. We also have real SAS HDD attached to some node. I suspected that to be a problem and we will exchange the HDDs for SSDs soon but the big problem was that the disktype was in the crushmap. -> Edited the crush map to ignore the type of storage as it is not very meaningful in our setup anyway * I had PGs stuck in undersized state for a long time and could not understand why ceph does not fix it. Then I checked the OSD weights (reweights) again and they were set to different values (1 and 0.85). After setting it to one on all OSDs Ceph actually started to bring all PG into the active+clean state. -> If all the OSDs actually are the same size i will either not reweight in the future or set it to the same value on all OSDs probably 1 So now after a noticable downtime of kubernetes and having to recreate most persistent volumes on the cluster Ceph health is HEALTH_OK again. I could upgrade to Ceph 16.2.13. I hope I can now upgrade to 17.2.6 without issues. Best regards -- Mag. Ing. Omar Siam Austrian Center for Digital Humanities and Cultural Heritage Österreichische Akademie der Wissenschaften | Austrian Academy of Sciences Stellvertretende Behindertenvertrauensperson | Deputy representative for disabled persons Bäckerstraße 13, 1010 Wien, Österreich | Vienna, Austria T: +43 1 51581-7295 omar.siam(a)oeaw.ac.at |www.oeaw.ac.at/acdh

1 0

ceph Pacific - MDS activity freezes when one the MDSs is restarted
by Emmanuel Jaep 09 Jun '23

09 Jun '23

Hi, we are currently running a ceph fs cluster at the following version: MDS version: ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) pacific (stable) The cluster is composed of 7 active MDSs and 1 standby MDS: RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active icadmin012 Reqs: 73 /s 1938k 1880k 85.3k 92.8k 1 active icadmin008 Reqs: 206 /s 2375k 2375k 7081 171k 2 active icadmin007 Reqs: 91 /s 5709k 5256k 149k 299k 3 active icadmin014 Reqs: 93 /s 679k 664k 40.1k 216k 4 active icadmin013 Reqs: 86 /s 3585k 3569k 12.7k 197k 5 active icadmin011 Reqs: 72 /s 225k 221k 8611 164k 6 active icadmin015 Reqs: 87 /s 1682k 1610k 27.9k 274k POOL TYPE USED AVAIL cephfs_metadata metadata 8552G 22.3T cephfs_data data 226T 22.3T STANDBY MDS icadmin006 When I restart one of the active MDSs, the standby MDS becomes active and its state becomes "replay". So far, so good! However, only one of the other "active" MDSs seems to remain active. All activities drop from the other ones: RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 active icadmin012 Reqs: 0 /s 1938k 1881k 85.3k 9720 1 active icadmin008 Reqs: 0 /s 2375k 2375k 7080 2505 2 active icadmin007 Reqs: 2 /s 5709k 5256k 149k 26.5k 3 active icadmin014 Reqs: 0 /s 679k 664k 40.1k 3259 4 replay icadmin006 801k 801k 1279 0 5 active icadmin011 Reqs: 0 /s 225k 221k 8611 9241 6 active icadmin015 Reqs: 0 /s 1682k 1610k 27.9k 34.8k POOL TYPE USED AVAIL cephfs_metadata metadata 8539G 22.8T cephfs_data data 225T 22.8T STANDBY MDS icadmin013 In effect, the cluster becomes almost unavailable until the newly promoted MDS finishes rejoining the cluster. Obviously, this defeats the purpose of having 7MDSs. Is this behavior? If not, what configuration items should I check to go back to "normal" operations? Best, Emmanuel

5 12

CreateMultipartUpload and Canned ACL - bucket-owner-full-control
by Rasool Almasi 09 Jun '23

09 Jun '23

A bucket with a policy that enforces "bucket-owner-full-control" results in Access Denied if multipart is used to upload the object. It is also discussed in an awscli issue: https://github.com/aws/aws-cli/issues/1674 aws client exits with "An error occurred (AccessDenied) when calling the CreateMultipartUpload operation", but it has been fixed on the server side rather than the client. Since without multi-part the object size in Rados Gateway has a limit of 5G and apparently increasing the limit is not recommended, how can I enforce "bucket-owner-full-control" and allow multipart upload? A sample bucket policy: { "Version": "2012-10-17", "Id": "read-write", "Statement": [ ... "Sid": "PolicyForAllowUploadWithCannedACL", "Effect": "Allow", "Principal": { "AWS": [ "arn:aws:iam::18f2c0870e3649b0f383504aec1d0bef:root" ] }, "Action": [ "s3:PutObject" ], "Resource": [ "arn:aws:s3:::mybucket", "arn:aws:s3:::mybucket/*" ], "Condition": { "StringEquals": { "s3:x-amz-acl": "bucket-owner-full-control" } } } ] } Regards, Rasool

1 0

S3 and Omap
by xadhoom76＠gmail.com 09 Jun '23

09 Jun '23

Hi, we have a ceph 17.2.6 with ragosgw and a couple of buckets in it. We use it for backup with lock directly from veeam. After few backups we got HEALTH_WARN 2 large omap objects │····················· [WRN] LARGE_OMAP_OBJECTS: 2 large omap objects │····················· 2 large objects found in pool 'backup.rgw.buckets.index' │····················· Search the cluster log for 'Large omap object found' for more details. What is causing this ? could we get a bigger threshold avout omap size and ignore safely this warning ? What's the issue? Best regards

2 1