Hello everyone, We’re running a 30+ node cluster (+plan to add more) to provide S3 services to our consumers. Each node has: - 64 Cores, 300+ GB RAM (Monitors have less memory, we have 5 monitors separated from OSD nodes) - 14 NVMe (14TB) or 14 HDD (10TB) disks - Each OSD node also hosts RGW services(pods). - HP/Dell/Lenovo hardware - Around 6 PB Raw capacity. - roughly 3 CPU cores and 8GB RAM are allocated per OSD sizing. - We use Rook to deploy our cluster I know that some of the answers might be long, I also have my own experiences on some of them(which I included), but I would highly appreciate to see the state for Ceph S3 on 04/2025. and here we have several questions we’d love your input on: * Performance Counters - We want to enable performance counters to identify noisy buckets and diagnose issues when they occur. - Does enabling performance counters cause noticeable performance degradation? If so, by how much? - For now, we plan to keep them disabled.(Because we are not sure) * OS Tuning for Ceph Nodes - WriteCache is disabled for NVMe disks. - No multi-socket nodes (no NUMA configuration). - No other workloads running on Ceph nodes. - Plan is to keep the default installation unless strongly advised otherwise. - Is there anything else critical we should tune (kernel parameters, etc.)? * Tracking Over-utilization - How can we detect and identify users overloading the cluster (e.g., excessive object creation/deletion, heavy throughput)? * NVMe Configuration for Ceph - Beyond disabling cache, what are the best practices around discard/trimming operations for NVMe drives? * Buying New NVMe Disks - When we order disks, we request 2 batches per disk group (e.g., 100 disks from Batch1, 100 from Batch2). We are looking for more professional recommendations. Is it a good idea to have different batches or shall we combine two different vendor disk groups? - We always check the rebranding values for NVMe disks and try to group them with rebranding informations. * Tuning Ceph for Large Scale S3 - What default parameters should be adjusted for a large-scale S3 deployment? * Customer Limitations - In theory we don't need any limitations, but we don't want to see non-sharding objects, etc. , If we allow everything our customers will find ways to abuse this. What parameters will protect us from operational difficulties. - Are there standard limitations we should enforce, like maximum object size, maximum number of objects per bucket, etc. * Inventory Management and Netbox Integration - Is it a good practice to integrate Netbox into Ceph operations? - Like, should we register all disks in Netbox for easier daily maintenance, adding automation to the netbox and replacing the disks via inventory changes on netbox ? * Cluster Deployment Best Practices - Ansible, Rook, custom deployments, or manual host installations — what’s best for large clusters, might depend on the team and company structure but would be nice to see opinions? * Why Some Large Deployments Avoid Rook - I mostly see Rook deployments for Openstack HCI deployments only (maybe I am wrong on this) - Is it about operational complexity, or is Rook seen as unsuitable for very large clusters? * Monitoring Metrics - What are the most critical metrics for diagnosing S3 issues? * RocksDB Tuning and Archiving - How should we manage RocksDB in large-scale setups? - Should we periodically split/archive the DB? - How important is RocksDB latency monitoring? * OS-Level Monitoring - Besides standard exporter logs, what else should we monitor (e.g., network latency, OSD-to-OSD throughput)? - we are planning to create a separate exporter than the ceph_exporter to export some specific metrics (mostly disk and NW related) * Design Mistakes to Avoid - What are the common mistakes (e.g., settings that block easy upgrades) that we should be aware of when planning a large-scale S3 service? * Vendor Relations - Is there a resource to check disk types, endurance, wearout timelines, etc. When we order new HW the vendor doesn't provide us any data? - Maybe there is a group of Ceph users holding some internal information about different vendors and their HW, if not I would be happy to start this.. - What's the best approach when dealing with hardware vendors, getting the cheapest or getting the reliable is the right approach? * Scheduled Operations - Are there standard Ansible playbooks or procedures for recurring maintenance tasks? * Unsupported Ceph Versions - Some large deployments stick with Ceph 16.x (unsupported). - Why do they prefer being alone on the old version and backporting features themselves? this seems very dangerous (there has to be a valid reason why?) * Cluster Health Monitoring - Do large deployments rely only on HEALTH_OK? - Or do they combine metrics for a custom health status definition? * Security Best Practices - What's the best approach for securing the cluster? - How should we handle disk encryption, key rotation (frequency/method), and secure disposal? * Diversity of Pools - If our service is focused purely on S3, should we still consider using other pool types (RBD, CephFS)? * Disk Size Diversity - Is it problematic to have OSDs with different disk sizes? * Multisite Clusters - In a multisite setup, if replication is enabled for some pools, is the performance impact localized to those pools, or does it affect the entire cluster? * WAF or Smart Proxy for RGW nodes - Our idea is to have RGW on each of our OSD node, (Monitor and Mgr would be excluded on any operation), what would be the best idea besides having a LDB in front of these RGW services. How are the others designing RGW, is it safe enough to just use it directly? I would highly appreciate your answers/opinions on these topics. I also assume that most of these comments would come from real use cases and experiences. Thanks in advance, _ Senol Colak