- ceph-users - lists.ceph.io

cephadm file "/sbin/cephadm", line 10098 PK ^
by farhad kh 18 Dec '23

18 Dec '23

Hello, I downloaded cephadm from the link below. https://download.ceph.com/rpm-18.2.0/el8/noarch/ I change the address of the images to the address of my private registry, ``` DEFAULT_IMAGE = 'opkbhfpspsp0101.fns/ceph/ceph:v18' DEFAULT_IMAGE_IS_MAIN = False DEFAULT_IMAGE_RELEASE = 'reef' DEFAULT_PROMETHEUS_IMAGE = 'opkbhfpspsp0101.fns/ceph/prometheus:v2.43.0' DEFAULT_LOKI_IMAGE = 'opkbhfpspsp0101.fns/ceph/loki:2.4.0' DEFAULT_PROMTAIL_IMAGE = 'opkbhfpspsp0101.fns/ceph/promtail:2.4.0' DEFAULT_NODE_EXPORTER_IMAGE = 'opkbhfpspsp0101.fns/ceph/node-exporter:v1.5.0' DEFAULT_ALERT_MANAGER_IMAGE = 'opkbhfpspsp0101.fns/ceph/alertmanager:v0.25.0' DEFAULT_GRAFANA_IMAGE = 'opkbhfpspsp0101.fns/ceph/ceph-grafana:9.4.7' DEFAULT_HAPROXY_IMAGE = 'opkbhfpspsp0101.fns/ceph/haproxy:2.3' DEFAULT_KEEPALIVED_IMAGE = 'opkbhfpspsp0101.fns/ceph/keepalived:2.2.4' DEFAULT_SNMP_GATEWAY_IMAGE = 'opkbhfpspsp0101.fns/ceph/snmp-notifier:v1.2.1' DEFAULT_ELASTICSEARCH_IMAGE = 'opkbhfpspsp0101.fns/ceph/elasticsearch:6.8.23' DEFAULT_JAEGER_COLLECTOR_IMAGE = 'opkbhfpspsp0101.fns/ceph/jaeger-collector:1.29' DEFAULT_JAEGER_AGENT_IMAGE = 'opkbhfpspsp0101.fns/ceph/jaeger-agent:1.29' DEFAULT_JAEGER_QUERY_IMAGE = 'opkbhfpspsp0101.fns/ceph/jaeger-query:1.29' DEFAULT_REGISTRY = 'opkbhfpspsp0101.fns' # normalize unqualified digests to this ``` but I encounter this error ` File "/sbin/cephadm", line 10098 PK ^ `. Also, there is a wrong line at the beginning of cephadm file PK^C^D^T^@^@^@^@^@¥<9a>^CW<8e>^[º^×Ü^E^@×Ü^E^@^K^@^@^@__main__.py#!/usr/bin/python3

2 1

Ceph Cluster Deployment - Recommendation
by Amardeep Singh 18 Dec '23

18 Dec '23

Hi Everyone, We are in the process of planning a ceph cluster deployment for our data infrastructure. To provide you with a bit of context, we have deployed hardware across two data halls in our data center, and they are connected via a 10Gb interconnect. The hardware configuration for 4 x Ceph Cluster (2 x servers in both data halls) * 2 x AMD EPYC 7513 - 32 Cores / 64 Threads * 512GB RAM * 2 x 960 GB (OS DISKS) * 8x Micron 7450 PRO 7680GB NVMe - PCIe Gen4 * Intel X550-T2 - 10GbE Dual-Port RJ45 Server Adaptor Our primary usage is Object Gateway and will be running 4 x RGW Service. We are aiming to deploy using cephadm and utilize all nodes for MON/MGR/RGW and OSD's Given our limited experience with Ceph, we are reaching out to the knowledgeable members of this community for recommendations and best practices. We would greatly appreciate any insights or advice you can share regarding the following aspects: Cluster Topology: Considering our hardware setup with two data halls connected via a 10Gb interconnect, what would be the recommended cluster topology for optimal performance and fault tolerance? Best Practices for Deployment: Are there any recommended best practices for deploying Ceph in a similar environment? Any challenges we should be aware of? Thank you in advance for your time and assistance. Regards, Amar DISCLAIMER: This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error, please notify the sender. This message contains confidential information and is intended only for the individual named. If you are not the named addressee, you should not disseminate, distribute or copy this email. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. Thanks for your cooperation.

3 3

OSD has Rocksdb corruption that crashes ceph-bluestore-tool repair
by Malcolm Haak 18 Dec '23

18 Dec '23

Hello all, I had an OSD go offline due to UWE. When restarting the OSD service, to try and at least get it to drain cleanly of that data that wasn't damaged, the ceph-osd process would crash. I then attempted to repair it using ceph-bluestore-tool. I can run fsck and it will complete without issue, however when attempting to run repair it crashes in the exact same way that ceph-osd crashes. I'll attach the tail end of the output here: 2023-12-17T20:24:53.320+1000 7fdb7bf17740 -1 rocksdb: submit_common error: Corruption: block checksum mismatch: stored = 1106056583, computed = 657190205, type = 1 in db/020524.sst offset 21626321 size 4014 code = Rocksdb transaction: PutCF( prefix = S key = 'per_pool_omap' value size = 1) -442> 2023-12-17T20:24:53.386+1000 7fdb7bf17740 -1 /usr/src/debug/ceph/ceph-18.2.0/src/os/bluestore/BlueStore.cc: In function 'unsigned int BlueStoreRepairer::apply(KeyValueDB*)' thread 7fdb7bf17740 time 2023-12-17T20:24:53.341999+1000 /usr/src/debug/ceph/ceph-18.2.0/src/os/bluestore/BlueStore.cc: 17982: FAILED ceph_assert(ok) ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x136) [0x7fdb7b6502c9] 2: /usr/lib/ceph/libceph-common.so.2(+0x2504a4) [0x7fdb7b6504a4] 3: (BlueStoreRepairer::apply(KeyValueDB*)+0x5af) [0x559afb98cc7f] 4: (BlueStore::_fsck_on_open(BlueStore::FSCKDepth, bool)+0x45fc) [0x559afba2436c] 5: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x204) [0x559afba31014] 6: main() 7: /usr/lib/libc.so.6(+0x27cd0) [0x7fdb7ae45cd0] 8: __libc_start_main() 9: _start() -441> 2023-12-17T20:24:53.390+1000 7fdb7bf17740 -1 *** Caught signal (Aborted) ** in thread 7fdb7bf17740 thread_name:ceph-bluestore- ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable) 1: /usr/lib/libc.so.6(+0x3e710) [0x7fdb7ae5c710] 2: /usr/lib/libc.so.6(+0x8e83c) [0x7fdb7aeac83c] 3: raise() 4: abort() 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x191) [0x7fdb7b650324] 6: /usr/lib/ceph/libceph-common.so.2(+0x2504a4) [0x7fdb7b6504a4] 7: (BlueStoreRepairer::apply(KeyValueDB*)+0x5af) [0x559afb98cc7f] 8: (BlueStore::_fsck_on_open(BlueStore::FSCKDepth, bool)+0x45fc) [0x559afba2436c] 9: (BlueStore::_fsck(BlueStore::FSCKDepth, bool)+0x204) [0x559afba31014] 10: main() 11: /usr/lib/libc.so.6(+0x27cd0) [0x7fdb7ae45cd0] 12: __libc_start_main() 13: _start() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. The reason I need to get this OSD functioning is I had two other OSD's fail causing a single PG to be in down state. The weird thing is, I got one of those back up without issue (ceph-osd crashed due to root filling and alert not sending) but the PG is still down. So I need to get this other one back up (or the data extracted) to get that PG back from down. Thanks in advance

2 1

Cephfs MDS tunning for deep-learning data-flow
by mhnx 16 Dec '23

16 Dec '23

Hello everyone! How are you doing? I wasn't around for two years but I'm back and working on a new development. I deployed 2x ceph cluster: 1- user_data:5x node [8x4TB Sata SSD, 2x 25Gbit network], 2- data-gen: 3x node [8x4TB Sata SSD, 2x 25Gbit network], note: hardware is not my choice and I know I have TRIM issue and also I couldn't use any PCI-E nvme for wal+db because 1u servers and no empty slots --------------------- At test phase everything was good, I reached 1GB/s for 18 clients at the same time. But when I migrate to production (60 GPU server client + 40 CPU server client) the speed issue begin because of the default parameters as usual and now I'm working on adaptation by debugging current data work flow I have and I'm researching how can I improve my environment. So far, I couldn't find useful guide or informations in one place and I just wanted to share my findings, benchmarks and ideas with the community and if I'm lucky enough, maybe I will get awesome recommendations from some old friends and enjoy get in touch after a while. :) Starting from here, I will only share technical information about my environment: 1- Cluster user_data: 5x node [8x4TB Sata SSD, 2x 25Gbit network] = Replication 2 - A: I only have 1 pool in this cluster and information is below: - ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED ssd 146 TiB 106 TiB 40 TiB 40 TiB 27.50 TOTAL 146 TiB 106 TiB 40 TiB 40 TiB 27.50 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL .mgr 1 1 286 MiB 73 859 MiB 0 32 TiB cephfs.ud-data.meta 9 512 65 GiB 2.87M 131 GiB 0.13 48 TiB cephfs.ud-data.data 10 2048 23 TiB 95.34M 40 TiB 29.39 48 TiB - B: In this cluster, every user(50) has a subvolume and the quota is 1TB/for each users - C: In each subvolume, users has "home and data" directory. - D: home directory size 5-10GB and client uses it for docker home directory at each login - E: I'm also storing users personal or development data around 2TB/each user - F: I only have 1x active MDS server and 4x standby as below. - ceph fs status > ud-data - 84 clients > ======= > RANK STATE MDS ACTIVITY DNS INOS DIRS > CAPS > 0 active ud-data.ud-04.seggyv Reqs: 372 /s 4343k 4326k 69.7k > 2055k > POOL TYPE USED AVAIL > cephfs.ud-data.meta metadata 130G 47.5T > cephfs.ud-data.data data 39.5T 47.5T > STANDBY MDS > ud-data.ud-01.uatjle > ud-data.ud-02.xcoojt > ud-data.ud-05.rnhcfe > ud-data.ud-03.lhwkml > MDS version: ceph version 17.2.6 > (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) - What is my issue? 2023-12-15T21:07:47.175542+0000 mon.ud-01 [WRN] Health check failed: 1 clients failing to respond to cache pressure (MDS_CLIENT_RECALL) 2023-12-15T21:09:35.002112+0000 mon.ud-01 [INF] MDS health message cleared (mds.?): Client gpu-server-11 failing to respond to cache pressure 2023-12-15T21:09:35.391235+0000 mon.ud-01 [INF] Health check cleared: MDS_CLIENT_RECALL (was: 1 clients failing to respond to cache pressure) 2023-12-15T21:09:35.391304+0000 mon.ud-01 [INF] Cluster is now healthy 2023-12-15T21:10:00.000169+0000 mon.ud-01 [INF] overall HEALTH_OK For every read and write in client's trying to reach ceph MDS server and requests some data: issue 1: home data is around 5-10GB and users need all the time. I need to store it one time and prevent new requests. issue 2: users process generates new data by only reading some data one time and they write generated data one time. No need to cache this data at all. What I want to do ??? 1- I want to deploy 2x active MDS server for only "home" directory in each subvolume: - These 2x home MDS servers must send the data to client and cache in the client to reduce new requests even for simple "ls" command 2- I want to deploy 2x active MDS server for only "data" directory in each subvolume: - These 2x MDS servers must be configured to not hold any CACHE if it is not required constantly. The cache life time must be short and must be independent. - Constantly requested data from one client must be cached locally in that client to reduce requests and load on the MDS server. ------------------------------------------------------------ I believe you understand my data-flow and my needs. Let's talk what we can do about it. Note: I'm still researching and these are my finding and my plan so far. it is not completed, and this is the main reason why I'm writing this mail. ceph fs set $MYFS max_mds 4 mds_cache_memory_limit | default 4GiB --> 16GiB mds_cache_reservation | default 0.05 --> ?? mds_health_cache_threshold | default 1.5 --> ?? mds_cache_trim_threshold | default 256KiB --> ?? mds_cache_trim_decay_rate | default 1.0 --> ?? mds_cache_mid mds_decay_halflife mds_client_prealloc_inos mds_dirstat_min_interval mds_session_cache_liveness_magnitude mds_session_cache_liveness_decay_rate mds_max_caps_per_client mds_recall_max_caps mds_recall_max_decay_threshold mds_recall_max_decay_rate mds_recall_global_max_decay_threshold mds_session_cap_acquisition_throttle mds_session_cap_acquisition_decay_rate mds_session_max_caps_throttle_ratio mds_cap_acquisition_throttle_retry_request_timeout -Manually pinning directory trees to a particular rank As you can see, I'm at the beginning of this journey and I will be grateful if you can help me, share your knowledge, even I'm ready to help developers to use my system as a test bench to improve ceph as always! Best regards folks! - Özkan

1 1

rbd trash: snapshot id is protected from removal
by Eugen Block 15 Dec '23

15 Dec '23

Hi, I've been searching and trying things but to no avail yet. This is uncritical because it's a test cluster only, but I'd still like to have a solution in case this somehow will make it into our production clusters. It's an Openstack Victoria Cloud with Ceph backend. If one tries to remove a glance image (openstack image delete {UUID}' which usually has a protected snapshot it will fail to do so, but apparently the snapshot is actually moved to the trash namespace. And since it is protected, I can't remove it: storage01:~ # rbd -p images snap ls 278ffe2b-67a7-40d0-87b7-903f2fc9c3b4 --all SNAPID NAME SIZE PROTECTED TIMESTAMP NAMESPACE 159 1a97db13-307e-4820-8dc2-8549e9ba1ad7 39 MiB Thu Dec 14 08:29:56 2023 trash (snap) storage01:~ # rbd snap rm --snap-id 159 images/278ffe2b-67a7-40d0-87b7-903f2fc9c3b4 rbd: snapshot id 159 is protected from removal. storage01:~ # rbd snap ls images/278ffe2b-67a7-40d0-87b7-903f2fc9c3b4 storage01:~ # This is a small image and only a test environment, but these orphans could potentially fill up lots of space. In a newer openstack version (I tried with Antelope) this doesn't seem to work like that anymore, so that's good. But how would I get rid of that trash snapshot in this cluster? Thanks! Eugen

2 2

Corrupted and inconsistent reads from CephFS on EC pool
by aschmitz 15 Dec '23

15 Dec '23

Hi everyone, I'm seeing different results from reading files, depending on which OSDs are running, including some incorrect reads with all OSDs running, in CephFS from a pool with erasure coding. I'm running Ceph 17.2.6. # More detail In particular, I have a relatively large backup of some files, combined with SHA-256 hashes of the files (which were verified when the backup was created, approximately 7 months ago). Verifying these hashes currently gives several errors, both in large and small files, but somewhat tilted towards larger files. Investigating which PGs stored the relevant files (using cephfs-data-scan pg_files) didn't show the problem to be isolated to one PG, but did show several PGs that contained OSD 15 as an active member. Taking OSD 15 offline leads to *better* reads (more files with correct SHA-256 hashes), but not completely correct reads. Further investigation implicated OSD 34 as another potential issue, but taking it offline also results in more correct files but not completely. Bringing the stopped OSDs (15 and 34) back online results in the earlier (incorrect) hashes when reading files, as might be expected, but this seems to demonstrate that the correct information (or at least more correct information) is still on the drives. The hashes I receive for a given corrupted file are consistent from read to read (including on different hosts, to avoid caching as an issue), but obviously sometimes change if I take an affected OSD offline. # Recent history I have Ceph configured with a deep scrub interval of approximately 30 days, and they have completed regularly with no issues identified. However, within the past two weeks I added two additional drives to the cluster, and rebalancing took about two weeks to complete: the placement groups I took notice of having issues were not deep scrubbed since the replacement completed, so it is possible something got corrupted during the rebalance. Neither OSD 15 nor 34 is a new drive, and as far as I have experienced (and Ceph's health indications have shown), all of the existing OSDs have behaved correctly up to this point. # Configuration I created an erasure coding profile for the pool in question using the following command: ceph osd erasure-code-profile set erasure_k4_m2 \ plugin=jerasure \ k=4 m=2 \ technique=blaum_roth \ crush-device-class=hdd And the following CRUSH rule is used for the pool: rule erasure_k4_m2_hdd_rule { id 3 type erasure min_size 4 max_size 6 step take default class hdd step choose indep 3 type host step chooseleaf indep 2 type osd step emit } # Questions 1. Does this behavior ring a bell to anyone? Is there something obvious I'm missing or should do? 2. Is deep scrubbing likely to help the situation? Hurt it? (Hopefully not hurt: I've prioritized deep scrubbing of the PGs on OSD 15 and 34, and will likely follow up with the rest of the pool.) 3. Is there a way to force "full reads" or otherwise to use all of the EC chunks (potentially in tandem with on-disk checksums) to identify the correct data, rather than a combination of the data from the primary OSDs? Thanks for any insights you might have, aschmitz

1 0

Etag change of a parent object
by Rok Jaklič 15 Dec '23

15 Dec '23

Hi, shouldn't etag of a "parent" object change when "child" objects are added on s3? Example: 1. I add an object to test bucket: "example/" - size 0 "example/" has an etag XYZ1 2. I add an object to test bucket: "example/test1.txt" - size 12 "example/test1.txt" has an etag XYZ2 "example/" has an etag XYZ1 ... should this change? I understand that object storage is not hierarchical by design and objects are "not connected" by some other means than the bucket name. Kind regards, Rok

4 3

Howto: 'one line patch' in deployed cluster?
by Harry G Coin 14 Dec '23

14 Dec '23

Is there a 'Howto' or 'workflow' to implement a one-line patch in a running cluster? With full understanding it will be gone on the next upgrade? Hopefully without having to set up an entire packaging/development environment? Thanks! To implement: * /Subject/: Re: Permanent KeyError: 'TYPE' ->17.2.7: return self.blkid_api['TYPE'] == 'part' * /From/: Sascha Lucas <ceph-users@xxxxxxxxx> Problem found: in my case this is caused by DRBD secondary block devices, which can not be read until promoted to primary. ceph_volume/util/disk.py runs in blkid(): $ blkid -c /dev/null -p /dev/drbd4 blkid: error: /dev/drbd4: Wrong medium type but does not care about its return code. A quick fix is to use the get() method to automatically fall back to None for non existing keys: --- a/ceph_volume/util/device.py 2023-11-10 07:00:01.552497107 +0000 +++ b/ceph_volume/util/device.py 2023-11-10 08:54:40.320718690 +0000 @@ -476,13 +476,13 @@ @property def is_partition(self): self.load_blkid_api() if self.disk_api: return self.disk_api['TYPE'] == 'part' elif self.blkid_api: - return self.blkid_api['TYPE'] == 'part' + return self.blkid_api.get('TYPE') == 'part' return False Don't know why this is triggered in 17.2.7.

1 0

Ceph orch made block_db too small, not accounting for multiple nvmes, how-to fix it?
by Mikael Öhman 14 Dec '23

14 Dec '23

Hi Our hosts have 3 NVMEs and 48 spinning drives each. We found that ceph orch made the default lvm size for the block_db 1/3 the total size of the NVMEs. I suspect that ceph only considered one of the NVMEs when determining the size, based the closely related issue; https://tracker.ceph.com/issues/54541 We started having some bluefs spillover events now, so I'm looking for a way to fix this. Best idea I have so far is to manually specify the "block_db_size" in the osd_spec, then just recreating the entire block_db. Though I'm not sure if that means we'll hit the same issue https://tracker.ceph.com/issues/54541 instead. There would also be a lot of data to move in order to do this to a total of 588 OSD's. Maybe there is a way to just maybe remove and re-add (bigger) block_db? I would appreciate any suggestions or tips. Best regards, Mikael

1 1

Pool Migration / Import/Export
by duluxoz 14 Dec '23

14 Dec '23

Hi All, I find myself in the position of having to change the k/m values on an ec-pool. I've discovered that I simply can't change the ec-profile, but have to create a "new ec-profile" and a "new ec-pool" using the new values, then migrate the "old ec-pool" to the new (see: https://ceph.io/en/news/blog/2015/ceph-pool-migration/ => Using Rados Export/Import). (Yes, a PITA, but it has to be done, and better doing it now when the data-size isn't that big - yet!) My only concern is that the "old ec-pool" is a `--data-pool` and part of an rbd image (ie the image was created with `rbd create --size 2T ec_rbd_pool/disk01 --data-pool ec_pool --image-feature journaling`). So my Q is: What are the "gotchas" (if any) with this, or is it simpler to backup the data (already done), destroy and recreate the rbd image from scratch, and restore the data to the re-created pool(s)/image? FTR: space isn't an issue (I've got plenty to play with) but I'm looking for the quickest way to do this as its a live (but little used) system, so I can take it down for a few hours (or more), but don't particular *want* to have it off-line for longer than absolutely necessary. Thanks in advance for any advice/help/warnings/etc :-) Cheers Dulux-Oz

2 1