We are using the S3 ceph interface.  

Attached is a python file using boto3 which, when run against two different deployments of ceph (octopus ceph nano and our production nautilus 14.2.11 deployment), appears to repro a strange issue.

After running for a while, a recently uploaded file forever disappears from list_objects requests.  This file still appears to be visible to get_object if you know the specific name, but does not show up in list_objects.

There are more details about the experiment in the attached python file.

We produced a run of this experiment with debug logging, in which we see a trace message

In the same millisecond that the file was PUT.  

Reading the code, this comes from when a call to check_disk_state returns ENOENT, where we see

      /* object doesn't exist right now -- hopefully because it's
       * marked as !exists and got deleted */
    if (list_state.exists) {
      /* FIXME: what should happen now? Work out if there are any
       * non-bad ways this could happen (there probably are, but annoying
       * to handle!) */
    }
    // encode a suggested removal of that key
    list_state.ver.epoch = io_ctx.get_last_version();
    list_state.ver.pool = io_ctx.get_id();
    cls_rgw_encode_suggestion(CEPH_RGW_REMOVE, list_state, suggested_updates);
    return -ENOENT;
  }

It seems like this might be some kind of race between PUT and list_object in which some kind of object metadata is apparently deleted... the FIXME is at least a little suspicious :).  

I would love to know what's going on here, and if there is a fix or workaround we can do to prevent this behavior.  Let me know if there is any other information we can provide.

Thank you so much!

Best,

-Joseph Victor