- ceph-users - lists.ceph.io

PSA: Bringing up an OSD really, really fast
by Tim Holloway 20 Jul '24

20 Jul '24

While running Ceph in a VM may not be considered the most efficient mode of operation, it's still fairly popular. I've put together a small project that can spin up an OSD VM in just a few minutes. Whereas Ceph given free rein will gladly acquire resources with all the enthusiasm as a sprig of mint in damp earth, what I've developed is more useful for a small farm with very irregular resources such as my own. And for research purposes! The project has 2 components: one of which gets deployed on the intended VM host, the other being an Ansible role and model playbook. Full instructions are provided. And incidentally, the Ansible role should be able to run against a base OS host, even though it was developed for VMs. Comments and suggestions welcome. https://gogs.mousetech.com/mtsinc7/instant_osd

1 0

Ceph filesystems deadlocking and freezing
by Domhnall McGuigan 19 Jul '24

19 Jul '24

Hi, we're trying to run our database package with CephFS as the backing store and we're seeing constant freezes on filesystem accesses, apparently due to deadlocks in Ceph (all versions tested on 5.x kernels, 16.2.6 up to 18.2.2). I've made an issue on the tracker for it here with the details I've found so far: https://tracker.ceph.com/issues/67008 But if that's the wrong place to raise it, or if there are any additional details I could provide, please let me know. *********************************************************************************************************************************************************************** This email, its contents and any files attached are a confidential communication and are intended only for the named addressees indicated in the message. If you are not the named addressee or if you have received this email in error, you may not, without the consent of First Derivatives, copy, use or rely on any information or attachments in any way. Please notify the sender by return email and delete it from your email system. Unless separately agreed, First Derivatives does not accept any responsibility for the accuracy or completeness of the contents of this email or its attachments. Please note that any views, opinion or advice contained in this communication are those of the sending individual and not those of First Derivatives and First Derivatives shall have no liability whatsoever in relation to this communication (or its content) unless separately agreed. ***********************************************************************************************************************************************************************

1 0

Cephadm has a small wart
by Tim Holloway 19 Jul '24

19 Jul '24

I've been setting up a cookbook OSD creation process and as I walked through the various stages, I noted that the /etc/redhat-release file said "CentOS Stream 8". I panicked, because IBM has pulled the Ceph archives for CentOS 8 and nuked the machine, then rebuilt it with more attention to detail. I consistently was seeing the correct redhat-release until I installed cephadm and ran cephadm shell. Cephadm is running a chroot jail, it seems, and it's plugging in the wrong value for that file. Presumably from its internal image instead of copying the actual file. The false release string does not leak outside of cephadm, though. So far this appears to be only important as a possible confusion point, but it is of concern since the repositories in /etc/yum.repos.d download from repositories whose URLs are derived from the OS release that YUM (dnf) thinks it's running under and I'd hate to find out the hard way that something got confused. Probably rpm and yum/dnf shouldn't be available from within cephadm, but since cephadm pulls stuff internally that may not be possible. This was seen on AlmaLinux 9.4, cephadm version 16.2.15.

3 4

v19.1.0 Squid RC0 released
by Yuri Weinstein 19 Jul '24

19 Jul '24

This is the first release candidate for Squid. Feature highlights: RGW The User Accounts feature unlocks several new AWS-compatible IAM APIs for the self-service management of users, keys, groups, roles, policy and more. RADOS BlueStore has been optimized for better performance in snapshot-intensive workloads. BlueStore RocksDB LZ4 compression is now enabled by default to improve average performance and "fast device" space usage. Other improvements include more flexible EC configurations, an OpTracker to help debug mgr module issues, and better scrub scheduling. Dashboard * Rearranged Navigation Layout: The navigation layout has been reorganized for improved usability and easier access to key features. CephFS Improvements * Support for managing CephFS snapshots and clones, as well as snapshot schedule management * Manage authorization capabilities for CephFS resources * Helpers on mounting a CephFS volume RGW Improvements * Support for managing bucket policies * Add/Remove bucket tags * ACL Management * Several UI/UX Improvements to the bucket form Monitoring: Grafana dashboards are now loaded into the container at runtime rather than building a grafana image with the grafana dashboards. Official Ceph grafana images can be found in quay.io/ceph/grafana * Monitoring: RGW S3 Analytics: A new Grafana dashboard is now available, enabling you to visualize per bucket and user analytics data, including total GETs, PUTs, Deletes, Copies, and list metrics. Crimson/Seastore Crimson's first tech preview release! Supporting RBD workloads on Replicated pools. For more information please visit: https://ceph.io/en/news/crimson If any of our community members would like to help us with performance investigations or regression testing of the Squid release candidate, please feel free to provide feedback via email or in https://pad.ceph.com/p/squid_scale_testing. For more active discussions, please use the #ceph-at-scale slack channel in ceph-storage.slack.com. * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-19.1.0.tar.gz * Containers at https://quay.io/repository/ceph/ceph * For packages, see https://docs.ceph.com/en/latest/install/get-packages/ * Release git sha1: 9025b9024baf597d63005552b5ee004013630404

3 2

How to detect condition for offline compaction of RocksDB?
by Rudenko Aleksandr 19 Jul '24

19 Jul '24

Hi, We have a big Ceph cluster (RGW case) with a lot of big buckets 10-500M objects with 31-1024 shards and a lot of io generated by many clients. Index pool placed on enterprise SSDs. We have about 120 SSDs (replication 3) and about 90Gb of OMAP data on each drive. About 75 PGs on each SSD for now. I think 75 is not enough for this amount of data, but I’m not sure that it is critical in our case. The problem: For the last few weeks, we can see big degradation of some SSD OSDs. We can see a lot of Ops (150-256 in perf dump) for a long time and we can see high avg ops time like 1-3 seconds (this metric is based on dump_historic_ops_by_duration and our avg calculation). These numbers are very unusual for our deployment. And it makes a big impact on our customers. For now, we make Offline compaction of ‘degraded’ OSDs and after compaction, we see that all PGs of this OSD are returned on this OSD (because recover is disabled during compaction) and we see that OSD works perfect for some time, all our metrics are dropped down for few weeks or only few days… And, I think the problem is in the rocksdb database which grows by levels and slows down requests. And I have few question about compaction: 1. Why can't automatic compaction manage this on its own? 2. How can I see RocksDB levels usage or some program metric which can be used as a condition for manual compacting? Because our metrics like request latency, OSD Ops count, OSD avg slow ops time are not 100% relative to rocksdb internal state. We try to use perf dump, bluestore.kv_xxx_lat metrics, but I think they are absolutely useless. Because we can see higher values after compaction and restart OSD then these metrics were before compaction, lol. We try to see OSD logs with debug_rocksdb=4, but we can’t understand this output compare it between good and bad OSDs: Uptime(secs): 3610809.5 total, 600.0 interval Flush(GB): cumulative 50.519, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 475.34 GB write, 0.13 MB/s write, 453.07 GB read, 0.13 MB/s read, 3346.3 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [p-0] ** ** Compaction Stats [p-1] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 0/0 0.00 KB 0.0 0.0 0.0 0.0 52.0 52.0 0.0 1.0 0.0 155.2 343.25 176.27 746 0.460 0 0 L1 4/0 204.00 MB 0.8 96.7 52.0 44.7 96.2 51.4 0.0 1.8 164.6 163.6 601.87 316.34 237 2.540 275M 1237K L2 34/0 2.09 GB 1.0 176.8 43.6 133.2 175.3 42.1 7.6 4.0 147.6 146.3 1226.72 590.20 535 2.293 519M 2672K L3 79/0 4.76 GB 0.3 163.6 38.9 124.7 140.7 16.1 8.8 3.6 152.6 131.3 1097.53 443.60 432 2.541 401M 79M L4 316/0 19.96 GB 0.1 0.9 0.4 0.5 0.8 0.3 19.7 2.0 151.1 131.1 5.94 2.53 4 1.485 2349K 385K Sum 433/0 27.01 GB 0.0 438.0 134.9 303.0 465.0 161.9 36.1 8.9 136.9 145.4 3275.31 1528.94 1954 1.676 1198M 83M Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 and this log is not usable as ‘metric’. Our Ceph version is: 16.2.13 and we use default bluestore_rocksdb_options.

3 12

Converting/Migrating EC pool to a replicated pool
by Burkhard Linke 19 Jul '24

19 Jul '24

Hi, we recently expanded a ceph cluster with a second site and want to distribute data accordingly. Two replicate should be present on one site, one replicate on the other site. This works well for replicated pools, but not for EC pool, e.g. the RGW data pool. I've added a second (replicated) storage class to RGW and moved most of the data to it via lifecycle operations. Now about 670 GB are left in the EC pool, mostly the head objects which cannot be moved. I'm looking for the best method to migrate the existing EC pool to a replicated pool, keeping the downtime a small as possible. So far two methods seem to be feasible: 1. Cache tiering Setup cache tiering with the EC pool as frontend pool and a replicated target pool as backend pool. Set cache mode to proxy and evict all objects from the EC pool the replicated pool. This will reduce the downtime to a minimum (stopping all RGW instances, changes the pool name in the placement definition, restart instance). But I'm not sure whether a EC pool can acts as frontend pool at all. And cache tiering is deprecated, so there might be other downfalls. 2. Export / Import Stop all RGW instances, export the EC pool's content with "rados export", and import it into the target pool. This should be a safe method, but will require a significant longer downtime (probably 24+ hours). Are there any other options? I also evaluated working bucket wise by creating a second placement set, disable access to a bucket, move all objects belonging to that bucket from one pool to the other (identifiable via the bucket's marker), change the placement setting of the bucket and reenable access. Unfortunately I've didn't found a good method to temporarily disable bucket access. Best regards, Burkhard Linke

3 2

Small issue with perms
by Albert Shih 18 Jul '24

18 Jul '24

Hi everyone. After my upgrade from 17.2.7 to 18.2.2 I notice after each time I restart I got a issue with perm on /var/lib/ceph/FSID/crash after the restart the owner/group are on nobody and I got Error scraping /var/lib/ceph/crash: [Errno 13] Permission denied: '/var/lib/ceph/crash' I need to do a chown ceph:ceph on the directory. But on the next reboot the dir return to nobody. Anyway to fix (beside some cron + chown) this minor bug ? Regards -- Albert SHIH 🦫 🐸 Observatoire de Paris France Heure locale/Local time: jeu. 18 juil. 2024 09:53:40 CEST

3 9

pg's stuck activating on osd create
by Richard Bade 18 Jul '24

18 Jul '24

Hi Everyone, I had an issue last night when I was bringing online some osds that I was rebuilding. When the osds created and came online 15pgs got stuck in activating. The first osd (osd.112) seemed to come online ok, but the second one (osd.113) triggered the issue. All the pgs in activating included osd.112 in the pg map and I resolved it by doing pg-upmap-items to map the pg back from osd.112 to where it currently was but it was painful having 10min of stuck i/o os an rbd pool with vm's running. Some details about the cluster: Pacific 16.2.15, upgraded from Nautilus fairly recently and Luminos back in the past. All osds were rebuilt on bluestore in Nautilus, as were the mons. The disks in question are Intel DC P4510 8TB nvme. I'm rebuilding them as I had previously had 4x2TB osd's per disk and now wanted to consolidate down to one osd per disk. There's around 300 osd's in the pool with 16384 pgs which means that the 2TB osds had 157pgs on them. However this means that the 8TB osds have 615pgs on them and I'm wondering if this is maybe the cause of the problem. There are no warnings about too many pgs per osd in the logs or ceph status. I have the default value of 250 for mon_max_pg_per_osd and default value of 3.0 for osd_max_pg_per_osd_hard_ratio. My plan is to reduce the number of pgs in the pool but I want to understand and prove what happened here. Is it likely I've hit pg overdose protection? If I have, how would I tell as I can't see anything in the cluster logs. Thanks, Rich

2 1

Heads up: New Ceph images require x86-64-v2 and possibly a qemu config change for virtual servers
by Dan van der Ster 18 Jul '24

18 Jul '24

Hey all, The upcoming community Ceph container images will be based on CentOS 9. In our Clyso CI testing lab we learned that el9-based images won't run on some (default) qemu VMs. Where our el8-based images run well, our new el9-based images get: error during bootstrap: Fatal glibc error: CPU does not support x86-64-v2 This is because RHEL 9 (and clones) added a hard requirement on x86-64-v2: * https://srobb.net/rhel9.html * https://forums.rockylinux.org/t/el9-will-require-x86-64-v2-support/5311 * https://developers.redhat.com/blog/2021/01/05/building-red-hat-enterprise-l… * https://access.redhat.com/solutions/6833751 The reason for the failure in our testing is that some of our virtual nodes use qemu's default CPU emulation (qemu64), which does not support x86-64-v2. The solution for these nodes is to switch to "host" or something else listed at https://www.qemu.org/docs/master/system/i386/cpu.html I would encourage cephadm/Rook users with daemons on VM hosts to verify their CPU modes on their VMs prior to any upgrades, and make the necessary changes to the hypervisor configuration ahead of time. Best, Dan -- Dan van der Ster CTO Clyso GmbH p: +49 89 215252722 | a: Vancouver, Canada w: https://clyso.com | e: dan.vanderster(a)clyso.com

3 2

RBD images can't be mapped anymore
by Daniele Rimoldi 18 Jul '24

18 Jul '24

Dear All, in the last few days we've been facing a strange problem with RBD mapping in our 5 host cluster. The cluster is running since 12 months and 2 weeks ago was updated from Quincy to Reef with no problems. Saturday, we decided to shut-down one of the 5 nodes di insert a test NVME drive, we have failure domain set at host level. Before this operation OSD flags were set (nodown, noout, norebalance... etc). With one node down, the cluster continued working correctly with one exception. Many RBD images mapped on various clients stopped working. This happened across various type of clients so both our external Proxmox cluster and our windows machines lost these RBD mapped devices. After bringing back node 5 to the cluster, the problem is still present. To make things even stranger, cluster is in "HEALTH OK" state and there are no apparent issues on OSDs. We then noticed that not all RBD images were lost, but only those created in pools placed on HDD class devices. There are a number rbd bench --io-type write test_hdd --pool=testhddof images in SSD class pools that are not affected by the problem, so we momentary moved (cloned) most important images to SSD pools to have them back and working. Now I'm seeking community help on how to investigate the problem. We did create a new HDD class pool with a new image for test purposes, but it can't be mapped. In a few tests, mapping in windows succeded but then, the device was immediately removed because of a stale connection and lack of communication with the device. Testing from a cluster node with: rbd bench --io-type write test_hdd --pool=testhdd works perfectly so OSDs and cluster seems to be fine... we are quite lost at this. Any suggestion on what to check? Thanks in advance for your help! Regards, Daniele

2 1