Hi ceph-devel, I recently noticed a strange discrepancy between the docs/user intuition, and how the code actually works. The Problem Currently with a pool configured as size=3, min_size=2: - PG goes inactive if acting set drops below 2 - PG accepts writes when acting set is 2 or more - Writes wait for ALL acting replicas before ACK to client This means min_size only controls PG activation, not write acknowledgment. The documentation says min_size sets the minimum number of written replicas in order to acknowledge an I/O operation to the client but the code waits for all replicas. In src/osd/ReplicatedBackend.cc lines 627-629: op.waiting_for_commit.insert( parent->get_acting_recovery_backfill_shards().begin(), parent->get_acting_recovery_backfill_shards().end()); All acting shards are added to waiting_for_commit and the write only completes when this set is empty. Use Case: Hybrid SSD/HDD Pools Many deployments want to use hybrid storage with a CRUSH rule that places 2 replicas on SSDs and 1 on HDD using multiple EMIT blocks. The idea is SSDs provide low latency while HDDs provide capacity. With current behavior every write waits for the HDD (50-150ms commit latency) negating the SSD performance benefit. With min_size=2 we expected writes to ACK after 2 SSD replicas and let the HDD catch up asynchronously. Proposed Change Make min_size control write acknowledgment threshold: - Writes ACK after min_size replicas commit - Remaining replicas receive the write asynchronously - If async replica fails before receiving write standard recovery handles it This matches operator intuition that min_size=2 means I am OK with 2 copies for durability. Alternative: New Pool Parameter If changing min_size behavior is too disruptive we could add a new parameter like write_quorum that defaults to size (current behavior) but can be set lower. Questions 1. Is there philosophical opposition to quorum-based writes in Ceph? 2. Are there concerns about PG log divergence or recovery complexity? 3. Would this require a design document and CDM discussion? 4. Has this been proposed or discussed before? Thanks for any feedback, and sorry for the long email.