30 Jan
2023
30 Jan
'23
11:39 a.m.
Dear all,
is there information available anywhere about the write amplification for
CephFS? I found quite some material on write amplification of VMs using
journaled file system on top of RBD but nothing as it relates to CephFS?
From my understanding I would expect the following:
- for X-rep, data needs to be written to X pgs (factor X)
- for k-m EC, data is written to k+m pgs (factor (k+m)/k)
Am I correct? How much is added by meta data, WAL? Do I miss anything?
Best wishes,
Manuel
30 Jan
30 Jan
1:25 p.m.
is there information available anywhere about the
write amplification for
CephFS? I found quite some material on write amplification of VMs using
journaled file system on top of RBD but nothing as it relates to CephFS?
From my understanding I would expect the following:
- for X-rep, data needs to be written to X pgs (factor X)
- for k-m EC, data is written to k+m pgs (factor (k+m)/k)
Am I correct?
Yes, but in the EC case, far smaller amounts of data will be sent to
each PG, so it depends on if you are calculating "amplification" in
terms of IOPS or in terms of MBytes (that needs to be sent over the
network and written to disk).
--
May the most significant bit of your life be positive.
1:33 p.m.
Dear Janne,
I'm concerned with the potential increased NVME wear. Assuming writes of
multiples of the block size, when I write 1GB of data to the CephFS, how
much data is written to the disks?
Best wishes,
Manuel
On Mon, Jan 30, 2023 at 2:25 PM Janne Johansson <icepic.dz(a)gmail.com> wrote:
is there
information available anywhere about the write amplification for
CephFS? I found quite some material on write amplification of VMs using
journaled file system on top of RBD but nothing as it relates to CephFS?
From my understanding I would expect the following:
- for X-rep, data needs to be written to X pgs (factor X)
- for k-m EC, data is written to k+m pgs (factor (k+m)/k)
Am I correct?
Yes, but in the EC case, far smaller amounts of data will be sent to
each PG, so it depends on if you are calculating "amplification" in
terms of IOPS or in terms of MBytes (that needs to be sent over the
network and written to disk).
--
May the most significant bit of your life be positive.
2:17 p.m.
I'm concerned with the potential increased NVME
wear. Assuming writes of multiples of the block size, when I write 1GB of data to the
CephFS, how much data is written to the disks?
In that case, repl=3 will write 1GB to three PGs. EC 8+3 would write
125M (ie, 1/8 of a GB) to 11 (8+3) drives.
--
May the most significant bit of your life be positive.
2:34 p.m.
OK. How much data will be written to the WAL and elsewhere?
On Mon, Jan 30, 2023 at 3:17 PM Janne Johansson <icepic.dz(a)gmail.com> wrote:
I'm
concerned with the potential increased NVME wear. Assuming writes of
multiples of
the block size, when I write 1GB of data to the CephFS, how
much data is written to the disks?
In that case, repl=3 will write 1GB to three PGs. EC 8+3 would write
125M (ie, 1/8 of a GB) to 11 (8+3) drives.
--
May the most significant bit of your life be positive.
3:09 p.m.
I think you will have to do the science there, it will be dependent on
thousands of factors like size of OSD, WAL, DB, placement of them,
size of incoming data if it is many small ops or few large ones and so
on. I don't think any one single answer there would work out for many
cases.
Most of us just buy a fast (nvme, or ssd) drive with a large DWPD and
put WAL/DB on those. My gut feeling says that WAL and DB will see a
lot of small writes, so "MB/s" might not be very important, but the
amount of writes on them is the important factor in terms of wear.
Then again, the point of just having WAL/DB separate might make the
OSD live lots longer since they bear the load of said smaller writes
for metadata.
Den mån 30 jan. 2023 kl 15:34 skrev Manuel Holtgrewe <zyklenfrei(a)gmail.com>om>:
OK. How much data will be written to the WAL and elsewhere?
On Mon, Jan 30, 2023 at 3:17 PM Janne Johansson <icepic.dz(a)gmail.com> wrote:
>
> > I'm concerned with the potential increased NVME wear. Assuming writes of
multiples of the block size, when I write 1GB of data to the CephFS, how much data is
written to the disks?
>
> In that case, repl=3 will write 1GB to three PGs. EC 8+3 would write
> 125M (ie, 1/8 of a GB) to 11 (8+3) drives.
>
> --
> May the most significant bit of your life be positive.
--
May the most significant bit of your life be positive.
454
days inactive
454
days old
5 comments
2 participants
participants (2)
-
Janne Johansson -
Manuel Holtgrewe