27 Apr
2023
27 Apr
'23
4:50 a.m.
This came up again in the dev summit at cephalocon, so I figure it's
worth reviving this thread.
First, I'll try to recap the situation (Ilya, feel free to correct me
here). My understanding of the issue is that rbd has features (most
notably encryption) which depend on the librados SPARSE_READ operation
reflecting accurately which ranges have been written or trimmed at a
4k granularity. This appears to work correctly on replicated pools on
bluestore, but erasure coded pools always return the full object
contents up to the object size including regions the client has not
written to.
I don't think this was originally a guarantee of the interface. I
think the original guarantee was simply that SPARSE_READ would return
any non-zero regions, not that it was guaranteed not to return
unwritten or trimmed regions. The OSD does not track this state above
the ObjectStore layer -- SPARSE_READ and MAPEXT both rely directly on
ObjectStore::fiemap. MAPEXT actually returns -ENOTSUPP on erasure
coded pools.
Adam: the observed behavior is that fiemap on bluestore does
accurately reflect the client's written extents at a 4k granularity.
Is that reliable, or is it a property of only some bluestore
configurations?
As it appears desirable that we actually guarantee this, we probably
want to do two things:
1) codify this guarantee in the ObjectStore interface (4k in all
cases?), and ensure that all configurations satisfy it going forward
(including seastore)
2) update the ec implementation to track allocation at the granularity
of an EC stripe. HashInfo is the natural place to put the
information, probably? We'll need to also implement ZERO. Radek: I
know you're looking into EC for crimson, perhaps you can evaluate how
much work would be required here?
-Sam
On Mon, May 2, 2022 at 5:21 PM Sam Just <sjust(a)redhat.com> wrote:
I don't think fiemap was ever intended as anything more than an
optimization to permit a user to avoid transferring unnecessary
zeroes. SeaStore will probably not track sparseness at more than a 4k
granularity. I don't think the EC implementation is clever about
sparse reads/writes at all since that information would probably need
to be duplicated above the objectstore in the object_info.
-Sam
On Mon, May 2, 2022 at 7:47 AM Jeff Layton <jlayton(a)redhat.com> wrote:
>
> On Mon, 2022-05-02 at 16:41 +0200, Ilya Dryomov wrote:
> > On Mon, May 2, 2022 at 4:22 PM Jeff Layton <jlayton(a)redhat.com> wrote:
> > >
> > > (sorry for the resend, but the first message got rejected by the list
because it was from an unsubscribed address)
> > >
> > > On Mon, 2022-05-02 at 14:05 +0200, Ilya Dryomov wrote:
> > > > Hi Sam,
> > > >
> > > > I wanted to clarify ObjectStore::fiemap API and sparse-read OSD op
> > > > guarantees as this came up in Jeff's fscrypt work and just
recently in
> > > > RBD as well.
> > > >
> > > > In fscrypt for kcephfs, Jeff has opted to use sparse-read to ensure
> > > > that file holes (which must contain all zeroes logically) don't
get
> > > > "decrypted" into seemingly random junk. (Unlike ecryptfs,
fscrypt
> > > > framework doesn't attempt to protect the information about
existence
> > > > and location of holes in files, so logical holes generally
correspond
> > > > to physical holes.)
> > > >
> > >
> > > The fscrypt client infrastructure generally prevents you from reading a
> > > file when you don't have the key, but you could always analyze the
> > > backing device and determine where the holes are. The situation with
> > > cephfs is analogous.
> >
> > Yup.
> >
> > >
> > > I imagine this is the same with ecryptfs though. I don't believe it
> > > fills in the holes when you do a write past the EOF either. Were you
> > > thinking of LUKS? That operates at the device level, so finding holes
> > > there is a much different matter.
> >
> > I'm pretty sure ecryptfs always fills holes by encrypting logical zeroes
and
> > writing the resulting ciphertext out to the backing filesystem. Quoting the
> > FAQ:
> >
> > eCryptfs does not currently support sparse files. Sequences of encrypted
> > extents with all 0's could be interpreted as sparse regions in
eCryptfs
> > without too much implementation complexity. However, this would open up
> > a possible attack vector, since the fact that certain segments of data are
> > all 0's could betray strategic information that the user does not
> > necessarily want to reveal to an attacker. For instance, if the attacker
> > knows that a certain database file with patient medical data keeps
> > information about viral infections in one region of the file and
> > information about diabetes in another section of the file, then the very
> > fact that the segment for viral infection data is populated with data at
> > all would reveal that the patient has a viral infection.
> >
>
> I stand corrected then! That tends to be pretty horrible for performance
> though. Prepare to wait for a while if you do create a file and then
> start writing at the 2G offset.
>
> In principle, we could also have the client fill in holes instead. It
> may be worthwhile to have a mode where it does that. That might alsogive
> us a way to support this on non-bluestore pools if it's not feasible to
> allow for sparseness there).
> --
> Jeff Layton <jlayton(a)redhat.com>
>
28 Apr
28 Apr
12:46 a.m.
New subject: sparse-read OSD op guarantees
On Wed, 2023-04-26 at 14:50 -0700, Sam Just wrote:
This came up again in the dev summit at cephalocon, so
I figure it's
worth reviving this thread.
First, I'll try to recap the situation (Ilya, feel free to correct me
here). My understanding of the issue is that rbd has features (most
notably encryption) which depend on the librados SPARSE_READ operation
reflecting accurately which ranges have been written or trimmed at a
4k granularity. This appears to work correctly on replicated pools on
bluestore, but erasure coded pools always return the full object
contents up to the object size including regions the client has not
written to.
I don't think this was originally a guarantee of the interface. I
think the original guarantee was simply that SPARSE_READ would return
any non-zero regions, not that it was guaranteed not to return
unwritten or trimmed regions. The OSD does not track this state above
the ObjectStore layer -- SPARSE_READ and MAPEXT both rely directly on
ObjectStore::fiemap. MAPEXT actually returns -ENOTSUPP on erasure
coded pools.
Adam: the observed behavior is that fiemap on bluestore does
accurately reflect the client's written extents at a 4k granularity.
Is that reliable, or is it a property of only some bluestore
configurations?
As it appears desirable that we actually guarantee this, we probably
want to do two things:
1) codify this guarantee in the ObjectStore interface (4k in all
cases?), and ensure that all configurations satisfy it going forward
(including seastore)
2) update the ec implementation to track allocation at the granularity
of an EC stripe. HashInfo is the natural place to put the
information, probably? We'll need to also implement ZERO. Radek: I
know you're looking into EC for crimson, perhaps you can evaluate how
much work would be required here?
-Sam
The reason we need this is that with the advent of fscrypt, ceph becomes
(for all intents and purposes) a block-based filesystem. We have to
encrypt and decrypt the data in blocks. fscrypt does allow you to choose
the blocksize though.
We chose 4k blocks for the fscrypt prototype because...well, no real
reason really other than it lines up nicely with 4k page-based buffered
I/O that the VFS layer is handing down on commodity hw.
In theory we could handle other blocksizes, though ideally we'd want to
keep the blocksize a power of two.
On Mon, May 2, 2022 at 5:21 PM Sam Just
<sjust(a)redhat.com> wrote:
--
Jeff Layton <jlayton(a)redhat.com>
I don't think fiemap was ever intended as anything more than an
optimization to permit a user to avoid transferring unnecessary
zeroes. SeaStore will probably not track sparseness at more than a 4k
granularity. I don't think the EC implementation is clever about
sparse reads/writes at all since that information would probably need
to be duplicated above the objectstore in the object_info.
-Sam
On Mon, May 2, 2022 at 7:47 AM Jeff Layton <jlayton(a)redhat.com> wrote:
>
> On Mon, 2022-05-02 at 16:41 +0200, Ilya Dryomov wrote:
> > On Mon, May 2, 2022 at 4:22 PM Jeff Layton <jlayton(a)redhat.com> wrote:
> > >
> > > (sorry for the resend, but the first message got rejected by the list
because it was from an unsubscribed address)
> > >
> > > On Mon, 2022-05-02 at 14:05 +0200, Ilya Dryomov wrote:
> > > > Hi Sam,
> > > >
> > > > I wanted to clarify ObjectStore::fiemap API and sparse-read OSD op
> > > > guarantees as this came up in Jeff's fscrypt work and just
recently in
> > > > RBD as well.
> > > >
> > > > In fscrypt for kcephfs, Jeff has opted to use sparse-read to ensure
> > > > that file holes (which must contain all zeroes logically) don't
get
> > > > "decrypted" into seemingly random junk. (Unlike ecryptfs,
fscrypt
> > > > framework doesn't attempt to protect the information about
existence
> > > > and location of holes in files, so logical holes generally
correspond
> > > > to physical holes.)
> > > >
> > >
> > > The fscrypt client infrastructure generally prevents you from reading a
> > > file when you don't have the key, but you could always analyze the
> > > backing device and determine where the holes are. The situation with
> > > cephfs is analogous.
> >
> > Yup.
> >
> > >
> > > I imagine this is the same with ecryptfs though. I don't believe it
> > > fills in the holes when you do a write past the EOF either. Were you
> > > thinking of LUKS? That operates at the device level, so finding holes
> > > there is a much different matter.
> >
> > I'm pretty sure ecryptfs always fills holes by encrypting logical zeroes
and
> > writing the resulting ciphertext out to the backing filesystem. Quoting the
> > FAQ:
> >
> > eCryptfs does not currently support sparse files. Sequences of encrypted
> > extents with all 0's could be interpreted as sparse regions in
eCryptfs
> > without too much implementation complexity. However, this would open up
> > a possible attack vector, since the fact that certain segments of data are
> > all 0's could betray strategic information that the user does not
> > necessarily want to reveal to an attacker. For instance, if the attacker
> > knows that a certain database file with patient medical data keeps
> > information about viral infections in one region of the file and
> > information about diabetes in another section of the file, then the very
> > fact that the segment for viral infection data is populated with data at
> > all would reveal that the patient has a viral infection.
> >
>
> I stand corrected then! That tends to be pretty horrible for performance
> though. Prepare to wait for a while if you do create a file and then
> start writing at the 2G offset.
>
> In principle, we could also have the client fill in holes instead. It
> may be worthwhile to have a mode where it does that. That might alsogive
> us a way to support this on non-bluestore pools if it's not feasible to
> allow for sparseness there).
> --
> Jeff Layton <jlayton(a)redhat.com>
>
3:15 a.m.
New subject: sparse-read OSD op guarantees
On Wed, Apr 26, 2023 at 11:50 PM Sam Just <sjust(a)redhat.com> wrote:
This came up again in the dev summit at cephalocon, so I figure it's
worth reviving this thread.
First, I'll try to recap the situation (Ilya, feel free to correct me
here). My understanding of the issue is that rbd has features (most
notably encryption) which depend on the librados SPARSE_READ operation
reflecting accurately which ranges have been written or trimmed at a
4k granularity. This appears to work correctly on replicated pools on
bluestore, but erasure coded pools always return the full object
contents up to the object size including regions the client has not
written to.
Hi Sam,
As Jeff said in another email, fscrypt support in kcephfs has a hard
dependency on accurate allocation information. librbd wants to grow
a similar dependency to enhance its built-in LUKS encryption support
(currently reads from unallocated areas on encrypted images are handled
inconsistently: if the underlying object doesn't exist, zeroes are
returned; if it does exist, we are at the mercy of sparse-read behavior
and can return random garbage obtained by decrypting zeroes).
I don't think this was originally a guarantee of the interface. I
think the original guarantee was simply that SPARSE_READ would return
any non-zero regions, not that it was guaranteed not to return
unwritten or trimmed regions. The OSD does not track this state above
the ObjectStore layer -- SPARSE_READ and MAPEXT both rely directly on
ObjectStore::fiemap. MAPEXT actually returns -ENOTSUPP on erasure
coded pools.
Adam: the observed behavior is that fiemap on bluestore does
accurately reflect the client's written extents at a 4k granularity.
Is that reliable, or is it a property of only some bluestore
configurations?
As it appears desirable that we actually guarantee this, we probably
want to do two things:
1) codify this guarantee in the ObjectStore interface (4k in all
cases?), and ensure that all configurations satisfy it going forward
(including seastore)
2) update the ec implementation to track allocation at the granularity
of an EC stripe. HashInfo is the natural place to put the
information, probably? We'll need to also implement ZERO. Radek: I
know you're looking into EC for crimson, perhaps you can evaluate how
much work would be required here?
The EC stripe that is referred to here is configurable on a per-pool
basis with the default taken from osd_pool_erasure_code_stripe_unit,
right? If the user configures it to e.g. 16k for a particular pool
(EC profile), how would that interact with the 4k guarantee at the
ObjectStore layer?
Thanks,
Ilya
7:04 a.m.
New subject: sparse-read OSD op guarantees
EC wouldn't be able to directly use the ObjectStore's fiemap
implementation. I think we'd need to build that metadata into either
the object_info or the ECUtil::hash_info.
-Sam
On Thu, Apr 27, 2023 at 1:16 PM Ilya Dryomov <idryomov(a)gmail.com> wrote:
On Wed, Apr 26, 2023 at 11:50 PM Sam Just <sjust(a)redhat.com> wrote:
This came up again in the dev summit at cephalocon, so I figure it's
worth reviving this thread.
First, I'll try to recap the situation (Ilya, feel free to correct me
here). My understanding of the issue is that rbd has features (most
notably encryption) which depend on the librados SPARSE_READ operation
reflecting accurately which ranges have been written or trimmed at a
4k granularity. This appears to work correctly on replicated pools on
bluestore, but erasure coded pools always return the full object
contents up to the object size including regions the client has not
written to.
Hi Sam,
As Jeff said in another email, fscrypt support in kcephfs has a hard
dependency on accurate allocation information. librbd wants to grow
a similar dependency to enhance its built-in LUKS encryption support
(currently reads from unallocated areas on encrypted images are handled
inconsistently: if the underlying object doesn't exist, zeroes are
returned; if it does exist, we are at the mercy of sparse-read behavior
and can return random garbage obtained by decrypting zeroes).
I don't think this was originally a guarantee of the interface. I
think the original guarantee was simply that SPARSE_READ would return
any non-zero regions, not that it was guaranteed not to return
unwritten or trimmed regions. The OSD does not track this state above
the ObjectStore layer -- SPARSE_READ and MAPEXT both rely directly on
ObjectStore::fiemap. MAPEXT actually returns -ENOTSUPP on erasure
coded pools.
Adam: the observed behavior is that fiemap on bluestore does
accurately reflect the client's written extents at a 4k granularity.
Is that reliable, or is it a property of only some bluestore
configurations?
As it appears desirable that we actually guarantee this, we probably
want to do two things:
1) codify this guarantee in the ObjectStore interface (4k in all
cases?), and ensure that all configurations satisfy it going forward
(including seastore)
2) update the ec implementation to track allocation at the granularity
of an EC stripe. HashInfo is the natural place to put the
information, probably? We'll need to also implement ZERO. Radek: I
know you're looking into EC for crimson, perhaps you can evaluate how
much work would be required here?
The EC stripe that is referred to here is configurable on a per-pool
basis with the default taken from osd_pool_erasure_code_stripe_unit,
right? If the user configures it to e.g. 16k for a particular pool
(EC profile), how would that interact with the 4k guarantee at the
ObjectStore layer?
Thanks,
Ilya
372
days inactive
374
days old
3 comments
3 participants
participants (3)
-
Ilya Dryomov -
Jeff Layton -
Sam Just