Problematic monitor command forwarding

The purpose of this email is to trigger a discussion on how we rectify the following situation so client commands are executed by the monitor only once and in the order they are submitted. https://tracker.ceph.com/issues/49428 describes a scenario where we can end up with commands executed more than once and out of order according to the client. In that tracker a client sends an 'erasure-code-profile rm' to mon.c and immediately receives an injected connection failure. The client then connects to mon.a and reissues the 'rm' command which returns the expected 'does not exist' result. The client code then issues an 'erasure-code-profile set' command. Shortly after this the original 'rm' command is forwarded to mon.a from mon.c and the 'set' command is cancelled because of this and the command, and the test, fails. From the client side the commands executed look like this. erasure-code-profile rm erasure-code-profile set From the mon side it looks like this. erasure-code-profile rm erasure-code-profile set erasure-code-profile rm I appreciate any feedback on the best way to tackle this one. -- Cheers, Brad