Hi ceph-devel,

I recently noticed a strange discrepancy between the docs/user intuition, and how the code actually works.

The Problem

Currently with a pool configured as size=3, min_size=2:
- PG goes inactive if acting set drops below 2
- PG accepts writes when acting set is 2 or more
- Writes wait for ALL acting replicas before ACK to client

This means min_size only controls PG activation, not write acknowledgment. The documentation says min_size sets the minimum number of written replicas in order to acknowledge an I/O operation to the client but the code waits for all replicas.

In src/osd/ReplicatedBackend.cc lines 627-629:

op.waiting_for_commit.insert(
    parent->get_acting_recovery_backfill_shards().begin(),
    parent->get_acting_recovery_backfill_shards().end());

All acting shards are added to waiting_for_commit and the write only completes when this set is empty.

Use Case: Hybrid SSD/HDD Pools

Many deployments want to use hybrid storage with a CRUSH rule that places 2 replicas on SSDs and 1 on HDD using multiple EMIT blocks. The idea is SSDs provide low latency while HDDs provide capacity.

With current behavior every write waits for the HDD (50-150ms commit latency) negating the SSD performance benefit. With min_size=2 we expected writes to ACK after 2 SSD replicas and let the HDD catch up asynchronously.

Proposed Change

Make min_size control write acknowledgment threshold:
- Writes ACK after min_size replicas commit
- Remaining replicas receive the write asynchronously
- If async replica fails before receiving write standard recovery handles it

This matches operator intuition that min_size=2 means I am OK with 2 copies for durability.

Alternative: New Pool Parameter

If changing min_size behavior is too disruptive we could add a new parameter like write_quorum that defaults to size (current behavior) but can be set lower.

Questions

1. Is there philosophical opposition to quorum-based writes in Ceph?
2. Are there concerns about PG log divergence or recovery complexity?
3. Would this require a design document and CDM discussion?
4. Has this been proposed or discussed before?

Thanks for any feedback, and sorry for the long email.