6 Apr
2023
6 Apr
'23
2:10 p.m.
We are testing an experimental Ceph cluster with server and controller at
subject.
The controller have not an HBA mode, but only a 'NonRAID' mode, come sort of
'auto RAID0' configuration.
We are using SSD SATA disks (MICRON MTFDDAK480TDT) that perform very well,
and SAS HDD disks (SEAGATE ST8000NM014A) that instead perform very bad
(particulary, very low IOPS).
There's some hint for disk/controller configuration/optimization?
Thanks.
--
Io credo nella chimica tanto quanto Giulio Cesare credeva nel caso...
mi va bene fino a quando non riguarda me :) (Emanuele Pucciarelli)
6 Apr
6 Apr
4:26 p.m.
How bizarre, I haven’t dealt with this specific SKU before. Some Dell / LSI HBAs call
this passthrough mode, some “personality”, some “jbod mode”, dunno why they can’t be
consistent.
We are testing an experimental Ceph cluster with
server and controller at
subject.
The controller have not an HBA mode, but only a 'NonRAID' mode, come sort of
'auto RAID0' configuration.
Dell’s CLI guide describes setting individual drives in Non-RAID, which *smells* like
passthrough, not the more-complex RAID0 workaround we had to do before passthrough.
https://www.dell.com/support/manuals/en-nz/perc-h750-sas/perc_cli_rg/set-dr…
We are using SSD SATA disks (MICRON MTFDDAK480TDT) that perform very well,
and SAS HDD disks (SEAGATE ST8000NM014A) that instead perform very bad
(particulary, very low IOPS).
Spinners are slow, this is news?
That said, how slow is slow? Testing commands and results or it didn’t happen.
Also, firmware matters. Run Dell’s DSU.
There's some hint for disk/controller
configuration/optimization?
Give us details, perccli /c0 show, test results etc.
Use a different HBA if you have to use an HBA, one that doesn’t suffer an RoC. Better
yet, take an expansive look at TCO and don’t write off NVMe as infeasible. If your
cluster is experimental hopefully you aren’t stuck with a lot of these. Add up the cost
of an RoC HBA, optionally with cache RAM and BBU/supercap, add in the cost delta for SAS
HDDs over SATA. Add in the operational hassle of managing WAL+DB on those boot SSDs. Add
in the extra HDDs you’ll need to provision because of IOPS.
Thanks.
--
Io credo nella chimica tanto quanto Giulio Cesare credeva nel caso...
mi va bene fino a quando non riguarda me :) (Emanuele Pucciarelli)
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
6:56 p.m.
spinners are slow anyway, but on top of that SAS disks often default to
writecache=off. In use as a single disk with no risk of raid
write-holes, you can turn on writecache. On SAS, I would assume the
firmware does not lie about writes reaching stable storage (flushes).
# turn on temporarily:
sdparm --set=WCE /dev/sdX
# turn on persistently:
sdparm --set=WCE --save /dev/sdX
To check current state:
sdparm --get=WCE /dev/sdf
/dev/sdf: SEAGATE ST2000NM0045 DS03
WCE 0 [cha: y, def: 0, sav: 0]
"WCE 0" means: off
"sav: 0" means: off next time the disk is powered on
Matthias
On Thu, Apr 06, 2023 at 09:26:27AM -0400, Anthony D'Atri wrote:
> How bizarre, I haven’t dealt with this specific SKU before. Some Dell / LSI HBAs
call this passthrough mode, some “personality”, some “jbod mode”, dunno why they can’t be
consistent.
>
>
> > We are testing an experimental Ceph cluster with server and controller at
> > subject.
> >
> > The controller have not an HBA mode, but only a 'NonRAID' mode, come
sort of
> > 'auto RAID0' configuration.
>
> Dell’s CLI guide describes setting individual drives in Non-RAID, which *smells* like
passthrough, not the more-complex RAID0 workaround we had to do before passthrough.
>
>
https://www.dell.com/support/manuals/en-nz/perc-h750-sas/perc_cli_rg/set-dr…
> >
> > We are using SSD SATA disks (MICRON MTFDDAK480TDT) that perform very well,
> > and SAS HDD disks (SEAGATE ST8000NM014A) that instead perform very bad
> > (particulary, very low IOPS).
>
> Spinners are slow, this is news?
>
> That said, how slow is slow? Testing commands and results or it didn’t happen.
>
> Also, firmware matters. Run Dell’s DSU.
>
> > There's some hint for disk/controller configuration/optimization?
>
> Give us details, perccli /c0 show, test results etc.
>
> Use a different HBA if you have to use an HBA, one that doesn’t suffer an RoC.
Better yet, take an expansive look at TCO and don’t write off NVMe as infeasible. If your
cluster is experimental hopefully you aren’t stuck with a lot of these. Add up the cost
of an RoC HBA, optionally with cache RAM and BBU/supercap, add in the cost delta for SAS
HDDs over SATA. Add in the operational hassle of managing WAL+DB on those boot SSDs. Add
in the extra HDDs you’ll need to provision because of IOPS.
>
> >
> >
> > Thanks.
> >
> > --
> > Io credo nella chimica tanto quanto Giulio Cesare credeva nel caso...
> > mi va bene fino a quando non riguarda me :) (Emanuele Pucciarelli)
> >
> > _______________________________________________
> > ceph-users mailing list -- ceph-users(a)ceph.io
> > To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
11 Apr
11 Apr
1:18 p.m.
Mandi! Matthias Ferdinand
In chel di` si favelave...
To check current state:
sdparm --get=WCE /dev/sdf
/dev/sdf: SEAGATE ST2000NM0045 DS03
WCE 0 [cha: y, def: 0, sav: 0]
"WCE 0" means: off
"sav: 0" means: off next time the disk is powered on
Checking current state lead to:
root@pppve1:~# sdparm --get=WCE /dev/sdd
/dev/sdd: ATA HGST HUS726T4TAL PV07
WCE 0 [cha: y]
So seems off, right?!
--
Voi non ci crederete
la mia ragazza sogna (R. Vecchioni)
1:14 p.m.
Mandi! Anthony D'Atri
In chel di` si favelave...
Dell???s CLI guide describes setting individual drives
in Non-RAID, which *smells* like passthrough, not the more-complex RAID0 workaround we had
to do before passthrough.
https://www.dell.com/support/manuals/en-nz/perc-h750-sas/perc_cli_rg/set-dr…
Exactly. NonRAID 'smells' also to us more like passthrough. The firs strange
things came from the fact that SATA/SSD disk are passed in a way that seems
'fully transparent' (eg, linux see even the disk S/N), while SAS disk are
passed more like an RAID0 disk, with a different S/N.
Spinners are slow, this is news?
;-)
That said, how slow is slow? Testing commands and
results or it didn???t happen.
A test done some month ago:
root@pppve1:~# fio --filename=/dev/sdc --direct=1 --rw=randrw --bs=128k --ioengine=libaio
--iodepth=256 --runtime=120 --numjobs=4 --time_based --group_reporting --name=hdd-rw-128
hdd-rw-128: (g=0): rw=randrw, bs=(R) 128KiB-128KiB, (W) 128KiB-128KiB, (T) 128KiB-128KiB,
ioengine=libaio, iodepth=256
...
fio-3.12
Starting 4 processes
Jobs: 4 (f=4): [m(4)][0.0%][eta 08d:17h:11m:29s]
hdd-rw-128: (groupid=0, jobs=4): err= 0: pid=26198: Wed May 18 19:11:04 2022
read: IOPS=84, BW=10.5MiB/s (11.0MB/s)(1279MiB/121557msec)
slat (usec): min=4, max=303887, avg=23029.19, stdev=61832.29
clat (msec): min=1329, max=6673, avg=4737.71, stdev=415.84
lat (msec): min=1543, max=6673, avg=4760.74, stdev=420.10
clat percentiles (msec):
| 1.00th=[ 2802], 5.00th=[ 4329], 10.00th=[ 4463], 20.00th=[ 4530],
| 30.00th=[ 4597], 40.00th=[ 4665], 50.00th=[ 4732], 60.00th=[ 4799],
| 70.00th=[ 4866], 80.00th=[ 4933], 90.00th=[ 5134], 95.00th=[ 5336],
| 99.00th=[ 5805], 99.50th=[ 6007], 99.90th=[ 6342], 99.95th=[ 6409],
| 99.99th=[ 6611]
bw ( KiB/s): min= 256, max= 5120, per=25.18%, avg=2713.08, stdev=780.45, samples=929
iops : min= 2, max= 40, avg=21.13, stdev= 6.10, samples=929
write: IOPS=87, BW=10.9MiB/s (11.5MB/s)(1328MiB/121557msec); 0 zone resets
slat (usec): min=9, max=309914, avg=23025.13, stdev=61676.77
clat (msec): min=1444, max=13086, avg=6943.12, stdev=2068.26
lat (msec): min=1543, max=13086, avg=6966.15, stdev=2069.28
clat percentiles (msec):
| 1.00th=[ 2769], 5.00th=[ 4597], 10.00th=[ 4799], 20.00th=[ 5067],
| 30.00th=[ 5403], 40.00th=[ 5873], 50.00th=[ 6409], 60.00th=[ 7148],
| 70.00th=[ 8020], 80.00th=[ 9060], 90.00th=[10134], 95.00th=[10671],
| 99.00th=[11610], 99.50th=[11879], 99.90th=[12550], 99.95th=[12550],
| 99.99th=[12684]
bw ( KiB/s): min= 256, max= 5376, per=24.68%, avg=2762.20, stdev=841.30, samples=926
iops : min= 2, max= 42, avg=21.52, stdev= 6.56, samples=926
cpu : usr=0.05%, sys=0.09%, ctx=2847, majf=0, minf=49
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.2%, 16=0.3%, 32=0.6%, >=64=98.8%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued rwts: total=10233,10627,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256
Run status group 0 (all jobs):
READ: bw=10.5MiB/s (11.0MB/s), 10.5MiB/s-10.5MiB/s (11.0MB/s-11.0MB/s), io=1279MiB
(1341MB), run=121557-121557msec
WRITE: bw=10.9MiB/s (11.5MB/s), 10.9MiB/s-10.9MiB/s (11.5MB/s-11.5MB/s), io=1328MiB
(1393MB), run=121557-121557msec
Disk stats (read/write):
sdc: ios=10282/10601, merge=0/0, ticks=3041312/27373721, in_queue=30373472, util=99.99%
Also, firmware matters. Run Dell???s DSU.
Controller does not have the latest-latest firmware, but a decent new one;
i've looked at chaneglogs and found nothing that seems relevant to
performance trouble.
Indeed, i'll do an upgrade ASAP.
Give us details, perccli /c0 show, test results etc.
root@pppve1:~# perccli /c0 show
Generating detailed summary of the adapter, it may take a while to complete.
CLI Version = 007.1910.0000.0000 Oct 08, 2021
Operating system = Linux 5.4.203-1-pve
Controller = 0
Status = Success
Description = None
Product Name = PERC H750 Adapter
Serial Number = 23L01Y6
SAS Address = 5f4ee0802ba3a400
PCI Address = 00:b3:00:00
System Time = 04/11/2023 12:11:50
Mfg. Date = 03/25/22
Controller Time = 04/11/2023 10:11:47
FW Package Build = 52.16.1-4405
BIOS Version = 7.16.00.0_0x07100501
FW Version = 5.160.02-3552
Driver Name = megaraid_sas
Driver Version = 07.713.01.00-rc1
Current Personality = RAID-Mode
Vendor Id = 0x1000
Device Id = 0x10E2
SubVendor Id = 0x1028
SubDevice Id = 0x2176
Host Interface = PCI-E
Device Interface = SAS-12G
Bus Number = 179
Device Number = 0
Function Number = 0
Domain ID = 0
Security Protocol = None
JBOD Drives = 6
JBOD LIST :
=========
---------------------------------------------------------------------------------
ID EID:Slt DID State Intf Med Size SeSz Model Vendor Port
---------------------------------------------------------------------------------
0 64:0 6 Onln SATA SSD 447.130 GB 512B MTFDDAK480TDT ATA x1
1 64:1 8 Onln SATA SSD 447.130 GB 512B MTFDDAK480TDT ATA x1
3 64:3 7 Onln SATA SSD 447.130 GB 512B MZ7KH480HAHQ0D3 ATA x1
5 64:5 9 Onln SATA HDD 3.638 TB 512B HGST HUS726T4TALA6L0 ATA x1
6 64:6 10 Onln SATA HDD 3.638 TB 512B HGST HUS726T4TALA6L0 ATA x1
7 64:7 11 Onln SATA HDD 3.638 TB 512B HGST HUS726T4TALA6L0 ATA x1
---------------------------------------------------------------------------------
ID=JBOD Target ID|EID=Enclosure Device ID|Slt=Slot No|DID=Device ID|Onln=Online
Offln=Offline|Intf=Interface|Med=Media Type|SeSz=Sector Size
SED=Self Encryptive Drive|PI=Protection Info|Sp=Spun|U=Up|D=Down
Physical Drives = 6
PD LIST :
=======
----------------------------------------------------------------------------------
EID:Slt DID State DG Size Intf Med SED PI SeSz Model Sp Type
----------------------------------------------------------------------------------
64:0 6 Onln - 447.130 GB SATA SSD N N 512B MTFDDAK480TDT U JBOD
64:1 8 Onln - 447.130 GB SATA SSD N N 512B MTFDDAK480TDT U JBOD
64:3 7 Onln - 447.130 GB SATA SSD N N 512B MZ7KH480HAHQ0D3 U JBOD
64:5 9 Onln - 3.638 TB SATA HDD N N 512B HGST HUS726T4TALA6L0 U JBOD
64:6 10 Onln - 3.638 TB SATA HDD N N 512B HGST HUS726T4TALA6L0 U JBOD
64:7 11 Onln - 3.638 TB SATA HDD N N 512B HGST HUS726T4TALA6L0 U JBOD
----------------------------------------------------------------------------------
EID=Enclosure Device ID|Slt=Slot No|DID=Device ID|DG=DriveGroup
DHS=Dedicated Hot Spare|UGood=Unconfigured Good|GHS=Global Hotspare
UBad=Unconfigured Bad|Sntze=Sanitize|Onln=Online|Offln=Offline|Intf=Interface
Med=Media Type|SED=Self Encryptive Drive|PI=Protection Info
SeSz=Sector Size|Sp=Spun|U=Up|D=Down|T=Transition|F=Foreign
UGUnsp=UGood Unsupported|UGShld=UGood shielded|HSPShld=Hotspare shielded
CFShld=Configured shielded|Cpybck=CopyBack|CBShld=Copyback Shielded
UBUnsp=UBad Unsupported|Rbld=Rebuild
Enclosures = 1
Enclosure LIST :
==============
--------------------------------------------------------------------
EID State Slots PD PS Fans TSs Alms SIM Port# ProdID VendorSpecific
--------------------------------------------------------------------
64 OK 8 6 0 0 0 0 0 - BP14G+ ?
--------------------------------------------------------------------
EID=Enclosure Device ID | PD=Physical drive count | PS=Power Supply count
TSs=Temperature sensor count | Alms=Alarm count | SIM=SIM Count | ProdID=Product ID
BBU_Info :
========
----------------------------------------------
Model State RetentionTime Temp Mode MfgDate
----------------------------------------------
BBU Optimal 0 hour(s) 47C - 0/00/00
----------------------------------------------
--
La tua tana sta in collina, Dr. Dobermann, seimilioni a metro quadro
e tua moglie sembra proprio una regina,
pero` e` la moglie di un ladro. (F. De Gregori)
5:46 p.m.
iops : min= 2, max= 40, avg=21.13,
stdev= 6.10, samples=929
iops : min= 2, max= 42, avg=21.52, stdev= 6.56, samples=926
That looks horrible. We also have a few SATA HDDs in Dell servers and they do about
100-150 IOP/s read or write. Originally, I was also a bit afraid that these disks would
drag performance down, but they are on par with the NL-SAS drives.
For ceph we use the cheapest Dell disk controller one can get (Dell HBA330 Mini
(Embedded)) and it works perfectly. All ceph-disks are configured non-raid, which is
equivalent to JBOD mode or pass-through. These controllers have no cache options, if your
do, disable all of them. Mode should be write-through.
For your disk type I saw "volatile write cache available = yes" on "the
internet". This looks a bit odd, but maybe these HDDs do have some volatile cache.
Try to disable it with smartctl and do the benchmark again.
Best regards,
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
14 Apr
14 Apr
7:06 p.m.
Mandi! Frank Schilder
In chel di` si favelave...
Exactly, horrible.
The strange thing is that we came from an homegrown Ceph cluster built using
old hardware (HP G6 servers) and spare disks, that perform 'better', or at
least perform 'more uniformly' that this 'new'.
With these embarassing IOPS, sooner or later we reach the point where
performances go to the ground, and sometimes it suffices to launch some
'find' on the filesystem involved...
iops
: min= 2, max= 40, avg=21.13, stdev= 6.10, samples=929
iops : min= 2, max= 42, avg=21.52, stdev= 6.56, samples=926
That
looks horrible. We also have a few SATA HDDs in Dell servers and they
do about 100-150 IOP/s read or write. Originally, I was also a bit afraid that these disks
would drag performance down, but they are on par with the NL-SAS drives.
For ceph we use the cheapest Dell disk controller one can get (Dell HBA330 Mini
(Embedded)) and it works perfectly. All ceph-disks are configured non-raid, which is
equivalent to JBOD mode or pass-through. These controllers have no cache options, if your
do, disable all of them. Mode should be write-through.
For your disk type I saw "volatile write cache available = yes" on "the
internet". This looks a bit odd, but maybe these HDDs do have some volatile cache.
Try to disable it with smartctl and do the benchmark again.
Sorry, i'm a bit puzzled here.
Matthias suggest to enable write cache, you suggest to disble it... or i'm
cache-confused?! ;-)
My actually controller configuration is:
root@pppve1:~# perccli /c0 show
Generating detailed summary of the adapter, it may take a while to complete.
CLI Version = 007.1910.0000.0000 Oct 08, 2021
Operating system = Linux 5.4.203-1-pve
Controller = 0
Status = Success
Description = None
Product Name = PERC H750 Adapter
Serial Number = 23L01Y6
SAS Address = 5f4ee0802ba3a400
PCI Address = 00:b3:00:00
System Time = 04/14/2023 18:03:24
Mfg. Date = 03/25/22
Controller Time = 04/14/2023 16:03:22
FW Package Build = 52.16.1-4405
BIOS Version = 7.16.00.0_0x07100501
FW Version = 5.160.02-3552
Driver Name = megaraid_sas
Driver Version = 07.713.01.00-rc1
Current Personality = RAID-Mode
Vendor Id = 0x1000
Device Id = 0x10E2
SubVendor Id = 0x1028
SubDevice Id = 0x2176
Host Interface = PCI-E
Device Interface = SAS-12G
Bus Number = 179
Device Number = 0
Function Number = 0
Domain ID = 0
Security Protocol = None
JBOD Drives = 6
JBOD LIST :
=========
---------------------------------------------------------------------------------
ID EID:Slt DID State Intf Med Size SeSz Model Vendor Port
---------------------------------------------------------------------------------
0 64:0 6 Onln SATA SSD 447.130 GB 512B MTFDDAK480TDT ATA x1
1 64:1 8 Onln SATA SSD 447.130 GB 512B MTFDDAK480TDT ATA x1
3 64:3 7 Onln SATA SSD 447.130 GB 512B MZ7KH480HAHQ0D3 ATA x1
5 64:5 9 Onln SATA HDD 3.638 TB 512B HGST HUS726T4TALA6L0 ATA x1
6 64:6 10 Onln SATA HDD 3.638 TB 512B HGST HUS726T4TALA6L0 ATA x1
7 64:7 11 Onln SATA HDD 3.638 TB 512B HGST HUS726T4TALA6L0 ATA x1
---------------------------------------------------------------------------------
ID=JBOD Target ID|EID=Enclosure Device ID|Slt=Slot No|DID=Device ID|Onln=Online
Offln=Offline|Intf=Interface|Med=Media Type|SeSz=Sector Size
SED=Self Encryptive Drive|PI=Protection Info|Sp=Spun|U=Up|D=Down
Physical Drives = 6
PD LIST :
=======
----------------------------------------------------------------------------------
EID:Slt DID State DG Size Intf Med SED PI SeSz Model Sp Type
----------------------------------------------------------------------------------
64:0 6 Onln - 447.130 GB SATA SSD N N 512B MTFDDAK480TDT U JBOD
64:1 8 Onln - 447.130 GB SATA SSD N N 512B MTFDDAK480TDT U JBOD
64:3 7 Onln - 447.130 GB SATA SSD N N 512B MZ7KH480HAHQ0D3 U JBOD
64:5 9 Onln - 3.638 TB SATA HDD N N 512B HGST HUS726T4TALA6L0 U JBOD
64:6 10 Onln - 3.638 TB SATA HDD N N 512B HGST HUS726T4TALA6L0 U JBOD
64:7 11 Onln - 3.638 TB SATA HDD N N 512B HGST HUS726T4TALA6L0 U JBOD
----------------------------------------------------------------------------------
EID=Enclosure Device ID|Slt=Slot No|DID=Device ID|DG=DriveGroup
DHS=Dedicated Hot Spare|UGood=Unconfigured Good|GHS=Global Hotspare
UBad=Unconfigured Bad|Sntze=Sanitize|Onln=Online|Offln=Offline|Intf=Interface
Med=Media Type|SED=Self Encryptive Drive|PI=Protection Info
SeSz=Sector Size|Sp=Spun|U=Up|D=Down|T=Transition|F=Foreign
UGUnsp=UGood Unsupported|UGShld=UGood shielded|HSPShld=Hotspare shielded
CFShld=Configured shielded|Cpybck=CopyBack|CBShld=Copyback Shielded
UBUnsp=UBad Unsupported|Rbld=Rebuild
Enclosures = 1
Enclosure LIST :
==============
--------------------------------------------------------------------
EID State Slots PD PS Fans TSs Alms SIM Port# ProdID VendorSpecific
--------------------------------------------------------------------
64 OK 8 6 0 0 0 0 0 - BP14G+ ?
--------------------------------------------------------------------
EID=Enclosure Device ID | PD=Physical drive count | PS=Power Supply count
TSs=Temperature sensor count | Alms=Alarm count | SIM=SIM Count | ProdID=Product ID
BBU_Info :
========
----------------------------------------------
Model State RetentionTime Temp Mode MfgDate
----------------------------------------------
BBU Optimal 0 hour(s) 48C - 0/00/00
----------------------------------------------
So BBU seems enabled.
--
Latex fetish implies the existence of a MS Word fetish.
(@nucleus)
15 Apr
15 Apr
12:49 p.m.
Hi,
Current controller mode is RAID. You can switch to HBA mode and disable cache in
controller settings at the BIOS
k
Sent from my iPhone
On 15 Apr 2023, at 12:11, Marco Gaiarin
<gaio(a)lilliput.linux.it> wrote:
Mandi! Frank Schilder
In chel di` si favelave...
Exactly, horrible.
The strange thing is that we came from an homegrown Ceph cluster built using
old hardware (HP G6 servers) and spare disks, that perform 'better', or at
least perform 'more uniformly' that this 'new'.
With these embarassing IOPS, sooner or later we reach the point where
performances go to the ground, and sometimes it suffices to launch some
'find' on the filesystem involved...
iops
: min= 2, max= 40, avg=21.13, stdev= 6.10, samples=929
iops : min= 2, max= 42, avg=21.52, stdev= 6.56, samples=926
That
looks horrible. We also have a few SATA HDDs in Dell servers and
they do about 100-150 IOP/s read or write. Originally, I was also a bit afraid that these
disks would drag performance down, but they are on par with the NL-SAS drives.
For ceph we use the cheapest Dell disk controller one can get (Dell HBA330 Mini
(Embedded)) and it works perfectly. All ceph-disks are configured non-raid, which is
equivalent to JBOD mode or pass-through. These controllers have no cache options, if your
do, disable all of them. Mode should be write-through.
For your disk type I saw "volatile write cache available = yes" on "the
internet". This looks a bit odd, but maybe these HDDs do have some volatile cache.
Try to disable it with smartctl and do the benchmark again.
Sorry, i'm a bit puzzled here.
Matthias suggest to enable write cache, you suggest to disble it... or i'm
cache-confused?! ;-)
My actually controller configuration is:
root@pppve1:~# perccli /c0 show
Generating detailed summary of the adapter, it may take a while to complete.
CLI Version = 007.1910.0000.0000 Oct 08, 2021
Operating system = Linux 5.4.203-1-pve
Controller = 0
Status = Success
Description = None
Product Name = PERC H750 Adapter
Serial Number = 23L01Y6
SAS Address = 5f4ee0802ba3a400
PCI Address = 00:b3:00:00
System Time = 04/14/2023 18:03:24
Mfg. Date = 03/25/22
Controller Time = 04/14/2023 16:03:22
FW Package Build = 52.16.1-4405
BIOS Version = 7.16.00.0_0x07100501
FW Version = 5.160.02-3552
Driver Name = megaraid_sas
Driver Version = 07.713.01.00-rc1
Current Personality = RAID-Mode
Vendor Id = 0x1000
Device Id = 0x10E2
SubVendor Id = 0x1028
SubDevice Id = 0x2176
Host Interface = PCI-E
Device Interface = SAS-12G
Bus Number = 179
Device Number = 0
Function Number = 0
Domain ID = 0
Security Protocol = None
JBOD Drives = 6
JBOD LIST :
=========
---------------------------------------------------------------------------------
ID EID:Slt DID State Intf Med Size SeSz Model Vendor Port
---------------------------------------------------------------------------------
0 64:0 6 Onln SATA SSD 447.130 GB 512B MTFDDAK480TDT ATA x1
1 64:1 8 Onln SATA SSD 447.130 GB 512B MTFDDAK480TDT ATA x1
3 64:3 7 Onln SATA SSD 447.130 GB 512B MZ7KH480HAHQ0D3 ATA x1
5 64:5 9 Onln SATA HDD 3.638 TB 512B HGST HUS726T4TALA6L0 ATA x1
6 64:6 10 Onln SATA HDD 3.638 TB 512B HGST HUS726T4TALA6L0 ATA x1
7 64:7 11 Onln SATA HDD 3.638 TB 512B HGST HUS726T4TALA6L0 ATA x1
---------------------------------------------------------------------------------
ID=JBOD Target ID|EID=Enclosure Device ID|Slt=Slot No|DID=Device ID|Onln=Online
Offln=Offline|Intf=Interface|Med=Media Type|SeSz=Sector Size
SED=Self Encryptive Drive|PI=Protection Info|Sp=Spun|U=Up|D=Down
Physical Drives = 6
PD LIST :
=======
----------------------------------------------------------------------------------
EID:Slt DID State DG Size Intf Med SED PI SeSz Model Sp Type
----------------------------------------------------------------------------------
64:0 6 Onln - 447.130 GB SATA SSD N N 512B MTFDDAK480TDT U JBOD
64:1 8 Onln - 447.130 GB SATA SSD N N 512B MTFDDAK480TDT U JBOD
64:3 7 Onln - 447.130 GB SATA SSD N N 512B MZ7KH480HAHQ0D3 U JBOD
64:5 9 Onln - 3.638 TB SATA HDD N N 512B HGST HUS726T4TALA6L0 U JBOD
64:6 10 Onln - 3.638 TB SATA HDD N N 512B HGST HUS726T4TALA6L0 U JBOD
64:7 11 Onln - 3.638 TB SATA HDD N N 512B HGST HUS726T4TALA6L0 U JBOD
----------------------------------------------------------------------------------
EID=Enclosure Device ID|Slt=Slot No|DID=Device ID|DG=DriveGroup
DHS=Dedicated Hot Spare|UGood=Unconfigured Good|GHS=Global Hotspare
UBad=Unconfigured Bad|Sntze=Sanitize|Onln=Online|Offln=Offline|Intf=Interface
Med=Media Type|SED=Self Encryptive Drive|PI=Protection Info
SeSz=Sector Size|Sp=Spun|U=Up|D=Down|T=Transition|F=Foreign
UGUnsp=UGood Unsupported|UGShld=UGood shielded|HSPShld=Hotspare shielded
CFShld=Configured shielded|Cpybck=CopyBack|CBShld=Copyback Shielded
UBUnsp=UBad Unsupported|Rbld=Rebuild
Enclosures = 1
Enclosure LIST :
==============
--------------------------------------------------------------------
EID State Slots PD PS Fans TSs Alms SIM Port# ProdID VendorSpecific
--------------------------------------------------------------------
64 OK 8 6 0 0 0 0 0 - BP14G+ ?
--------------------------------------------------------------------
EID=Enclosure Device ID | PD=Physical drive count | PS=Power Supply count
TSs=Temperature sensor count | Alms=Alarm count | SIM=SIM Count | ProdID=Product ID
BBU_Info :
========
----------------------------------------------
Model State RetentionTime Temp Mode MfgDate
----------------------------------------------
BBU Optimal 0 hour(s) 48C - 0/00/00
----------------------------------------------
So BBU seems enabled.
--
Latex fetish implies the existence of a MS Word fetish.
(@nucleus)
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io 
1:41 p.m.
With the LSI HBAs I’ve used, HBA cache seemed to only be used for VDs, not for passthrough
drives. And then with various nasty bugs. Be careful not to conflate HBA cache with cache
on the HDD itself.
> On Apr 15, 2023, at 11:51 AM, Konstantin Shalygin <k0ste(a)k0ste.ru> wrote:
>
> Hi,
>
> Current controller mode is RAID. You can switch to HBA mode and disable cache in
controller settings at the BIOS
>
>
> k
> Sent from my iPhone
>
>> On 15 Apr 2023, at 12:11, Marco Gaiarin <gaio(a)lilliput.linux.it> wrote:
>>
>> Mandi! Frank Schilder
>> In chel di` si favelave...
>>
>>>> iops : min= 2, max= 40, avg=21.13, stdev= 6.10, samples=929
>>>> iops : min= 2, max= 42, avg=21.52, stdev= 6.56, samples=926
>>> That looks horrible.
>>
>> Exactly, horrible.
>>
>> The strange thing is that we came from an homegrown Ceph cluster built using
>> old hardware (HP G6 servers) and spare disks, that perform 'better', or
at
>> least perform 'more uniformly' that this 'new'.
>>
>> With these embarassing IOPS, sooner or later we reach the point where
>> performances go to the ground, and sometimes it suffices to launch some
>> 'find' on the filesystem involved...
>>
>>
>>> We also have a few SATA HDDs in Dell servers and they do about 100-150 IOP/s
read or write. Originally, I was also a bit afraid that these disks would drag performance
down, but they are on par with the NL-SAS drives.
>>> For ceph we use the cheapest Dell disk controller one can get (Dell HBA330
Mini (Embedded)) and it works perfectly. All ceph-disks are configured non-raid, which is
equivalent to JBOD mode or pass-through. These controllers have no cache options, if your
do, disable all of them. Mode should be write-through.
>>> For your disk type I saw "volatile write cache available = yes" on
"the internet". This looks a bit odd, but maybe these HDDs do have some volatile
cache. Try to disable it with smartctl and do the benchmark again.
>>
>> Sorry, i'm a bit puzzled here.
>>
>> Matthias suggest to enable write cache, you suggest to disble it... or i'm
>> cache-confused?! ;-)
>>
>>
>> My actually controller configuration is:
>>
>> root@pppve1:~# perccli /c0 show
>> Generating detailed summary of the adapter, it may take a while to complete.
>>
>> CLI Version = 007.1910.0000.0000 Oct 08, 2021
>> Operating system = Linux 5.4.203-1-pve
>> Controller = 0
>> Status = Success
>> Description = None
>>
>> Product Name = PERC H750 Adapter
>> Serial Number = 23L01Y6
>> SAS Address = 5f4ee0802ba3a400
>> PCI Address = 00:b3:00:00
>> System Time = 04/14/2023 18:03:24
>> Mfg. Date = 03/25/22
>> Controller Time = 04/14/2023 16:03:22
>> FW Package Build = 52.16.1-4405
>> BIOS Version = 7.16.00.0_0x07100501
>> FW Version = 5.160.02-3552
>> Driver Name = megaraid_sas
>> Driver Version = 07.713.01.00-rc1
>> Current Personality = RAID-Mode
>> Vendor Id = 0x1000
>> Device Id = 0x10E2
>> SubVendor Id = 0x1028
>> SubDevice Id = 0x2176
>> Host Interface = PCI-E
>> Device Interface = SAS-12G
>> Bus Number = 179
>> Device Number = 0
>> Function Number = 0
>> Domain ID = 0
>> Security Protocol = None
>> JBOD Drives = 6
>>
>> JBOD LIST :
>> =========
>>
>>
---------------------------------------------------------------------------------
>> ID EID:Slt DID State Intf Med Size SeSz Model Vendor Port
>>
---------------------------------------------------------------------------------
>> 0 64:0 6 Onln SATA SSD 447.130 GB 512B MTFDDAK480TDT ATA x1
>> 1 64:1 8 Onln SATA SSD 447.130 GB 512B MTFDDAK480TDT ATA x1
>> 3 64:3 7 Onln SATA SSD 447.130 GB 512B MZ7KH480HAHQ0D3 ATA x1
>> 5 64:5 9 Onln SATA HDD 3.638 TB 512B HGST HUS726T4TALA6L0 ATA x1
>> 6 64:6 10 Onln SATA HDD 3.638 TB 512B HGST HUS726T4TALA6L0 ATA x1
>> 7 64:7 11 Onln SATA HDD 3.638 TB 512B HGST HUS726T4TALA6L0 ATA x1
>>
---------------------------------------------------------------------------------
>>
>> ID=JBOD Target ID|EID=Enclosure Device ID|Slt=Slot No|DID=Device ID|Onln=Online
>> Offln=Offline|Intf=Interface|Med=Media Type|SeSz=Sector Size
>> SED=Self Encryptive Drive|PI=Protection Info|Sp=Spun|U=Up|D=Down
>>
>> Physical Drives = 6
>>
>> PD LIST :
>> =======
>>
>>
----------------------------------------------------------------------------------
>> EID:Slt DID State DG Size Intf Med SED PI SeSz Model Sp Type
>>
----------------------------------------------------------------------------------
>> 64:0 6 Onln - 447.130 GB SATA SSD N N 512B MTFDDAK480TDT U JBOD
>> 64:1 8 Onln - 447.130 GB SATA SSD N N 512B MTFDDAK480TDT U JBOD
>> 64:3 7 Onln - 447.130 GB SATA SSD N N 512B MZ7KH480HAHQ0D3 U JBOD
>> 64:5 9 Onln - 3.638 TB SATA HDD N N 512B HGST HUS726T4TALA6L0 U JBOD
>> 64:6 10 Onln - 3.638 TB SATA HDD N N 512B HGST HUS726T4TALA6L0 U JBOD
>> 64:7 11 Onln - 3.638 TB SATA HDD N N 512B HGST HUS726T4TALA6L0 U JBOD
>>
----------------------------------------------------------------------------------
>>
>> EID=Enclosure Device ID|Slt=Slot No|DID=Device ID|DG=DriveGroup
>> DHS=Dedicated Hot Spare|UGood=Unconfigured Good|GHS=Global Hotspare
>> UBad=Unconfigured Bad|Sntze=Sanitize|Onln=Online|Offln=Offline|Intf=Interface
>> Med=Media Type|SED=Self Encryptive Drive|PI=Protection Info
>> SeSz=Sector Size|Sp=Spun|U=Up|D=Down|T=Transition|F=Foreign
>> UGUnsp=UGood Unsupported|UGShld=UGood shielded|HSPShld=Hotspare shielded
>> CFShld=Configured shielded|Cpybck=CopyBack|CBShld=Copyback Shielded
>> UBUnsp=UBad Unsupported|Rbld=Rebuild
>>
>> Enclosures = 1
>>
>> Enclosure LIST :
>> ==============
>>
>> --------------------------------------------------------------------
>> EID State Slots PD PS Fans TSs Alms SIM Port# ProdID VendorSpecific
>> --------------------------------------------------------------------
>> 64 OK 8 6 0 0 0 0 0 - BP14G+ ?
>> --------------------------------------------------------------------
>>
>> EID=Enclosure Device ID | PD=Physical drive count | PS=Power Supply count
>> TSs=Temperature sensor count | Alms=Alarm count | SIM=SIM Count | ProdID=Product
ID
>>
>>
>> BBU_Info :
>> ========
>>
>> ----------------------------------------------
>> Model State RetentionTime Temp Mode MfgDate
>> ----------------------------------------------
>> BBU Optimal 0 hour(s) 48C - 0/00/00
>> ----------------------------------------------
>>
>>
>> So BBU seems enabled.
>>
>> --
>> Latex fetish implies the existence of a MS Word fetish.
>> (@nucleus)
>>
>> _______________________________________________
>> ceph-users mailing list -- ceph-users(a)ceph.io
>> To unsubscribe send an email to ceph-users-leave(a)ceph.io
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
17 Apr
17 Apr
10:32 a.m.
Hi Marco.
You need to disable *volatile* write cache. In order to figure out what is volatile or
not, the best is to start disabling everything. This is usually the best way to run ceph
anyways. Some expensive controllers claim to have non-volatile write cache. This is
usually activated by disabling cache (yes, I know how this sounds), because this option
typically refers to volatile and not the non-volatile cache.
I would start by getting the disks into non-raid mode and disable all caches I can find.
Then benchmark again. Then, maybe, enable caches again one by one and see if anything
improves. If not or if it gets worse, disable it again.
Best regards,
Frank
=================
Frank Schilder
AIT Risø Campus
Bygning 109, rum S14
________________________________________
From: Konstantin Shalygin <k0ste(a)k0ste.ru>
Sent: Saturday, April 15, 2023 11:49 AM
To: Marco Gaiarin
Cc: Frank Schilder; ceph-users(a)ceph.io
Subject: Re: [ceph-users] Re: Some hint for a DELL PowerEdge T440/PERC H750 Controller...
Hi,
Current controller mode is RAID. You can switch to HBA mode and disable cache in
controller settings at the BIOS
k
Sent from my iPhone
For your disk
type I saw "volatile write cache available = yes" on "the internet".
This looks a bit odd, but maybe these HDDs do have some volatile cache. Try to disable it
with smartctl and do the benchmark again.
Sorry, i'm a bit puzzled here.
Matthias suggest to enable write cache, you suggest to disble it... or i'm
cache-confused?! ;-) On 15 Apr 2023, at 12:11, Marco Gaiarin
<gaio(a)lilliput.linux.it> wrote:
Mandi! Frank Schilder
In chel di` si favelave...
Exactly, horrible.
The strange thing is that we came from an homegrown Ceph cluster built using
old hardware (HP G6 servers) and spare disks, that perform 'better', or at
least perform 'more uniformly' that this 'new'.
With these embarassing IOPS, sooner or later we reach the point where
performances go to the ground, and sometimes it suffices to launch some
'find' on the filesystem involved...
iops
: min= 2, max= 40, avg=21.13, stdev= 6.10, samples=929
iops : min= 2, max= 42, avg=21.52, stdev= 6.56, samples=926
That
looks horrible. We also have a few SATA HDDs in Dell servers and
they do about 100-150 IOP/s read or write. Originally, I was also a bit afraid that these
disks would drag performance down, but they are on par with the NL-SAS drives.
For ceph we use the cheapest Dell disk controller one can get (Dell HBA330 Mini
(Embedded)) and it works perfectly. All ceph-disks are configured non-raid, which is
equivalent to JBOD mode or pass-through. These controllers have no cache options, if your
do, disable all of them. Mode should be write-through.
For your disk type I saw "volatile write cache available = yes" on "the
internet". This looks a bit odd, but maybe these HDDs do have some volatile cache.
Try to disable it with smartctl and do the benchmark again.
Sorry, i'm a bit puzzled here.
Matthias suggest to enable write cache, you suggest to disble it... or i'm
cache-confused?! ;-)
My actually controller configuration is:
root@pppve1:~# perccli /c0 show
Generating detailed summary of the adapter, it may take a while to complete.
CLI Version = 007.1910.0000.0000 Oct 08, 2021
Operating system = Linux 5.4.203-1-pve
Controller = 0
Status = Success
Description = None
Product Name = PERC H750 Adapter
Serial Number = 23L01Y6
SAS Address = 5f4ee0802ba3a400
PCI Address = 00:b3:00:00
System Time = 04/14/2023 18:03:24
Mfg. Date = 03/25/22
Controller Time = 04/14/2023 16:03:22
FW Package Build = 52.16.1-4405
BIOS Version = 7.16.00.0_0x07100501
FW Version = 5.160.02-3552
Driver Name = megaraid_sas
Driver Version = 07.713.01.00-rc1
Current Personality = RAID-Mode
Vendor Id = 0x1000
Device Id = 0x10E2
SubVendor Id = 0x1028
SubDevice Id = 0x2176
Host Interface = PCI-E
Device Interface = SAS-12G
Bus Number = 179
Device Number = 0
Function Number = 0
Domain ID = 0
Security Protocol = None
JBOD Drives = 6
JBOD LIST :
=========
---------------------------------------------------------------------------------
ID EID:Slt DID State Intf Med Size SeSz Model Vendor Port
---------------------------------------------------------------------------------
0 64:0 6 Onln SATA SSD 447.130 GB 512B MTFDDAK480TDT ATA x1
1 64:1 8 Onln SATA SSD 447.130 GB 512B MTFDDAK480TDT ATA x1
3 64:3 7 Onln SATA SSD 447.130 GB 512B MZ7KH480HAHQ0D3 ATA x1
5 64:5 9 Onln SATA HDD 3.638 TB 512B HGST HUS726T4TALA6L0 ATA x1
6 64:6 10 Onln SATA HDD 3.638 TB 512B HGST HUS726T4TALA6L0 ATA x1
7 64:7 11 Onln SATA HDD 3.638 TB 512B HGST HUS726T4TALA6L0 ATA x1
---------------------------------------------------------------------------------
ID=JBOD Target ID|EID=Enclosure Device ID|Slt=Slot No|DID=Device ID|Onln=Online
Offln=Offline|Intf=Interface|Med=Media Type|SeSz=Sector Size
SED=Self Encryptive Drive|PI=Protection Info|Sp=Spun|U=Up|D=Down
Physical Drives = 6
PD LIST :
=======
----------------------------------------------------------------------------------
EID:Slt DID State DG Size Intf Med SED PI SeSz Model Sp Type
----------------------------------------------------------------------------------
64:0 6 Onln - 447.130 GB SATA SSD N N 512B MTFDDAK480TDT U JBOD
64:1 8 Onln - 447.130 GB SATA SSD N N 512B MTFDDAK480TDT U JBOD
64:3 7 Onln - 447.130 GB SATA SSD N N 512B MZ7KH480HAHQ0D3 U JBOD
64:5 9 Onln - 3.638 TB SATA HDD N N 512B HGST HUS726T4TALA6L0 U JBOD
64:6 10 Onln - 3.638 TB SATA HDD N N 512B HGST HUS726T4TALA6L0 U JBOD
64:7 11 Onln - 3.638 TB SATA HDD N N 512B HGST HUS726T4TALA6L0 U JBOD
----------------------------------------------------------------------------------
EID=Enclosure Device ID|Slt=Slot No|DID=Device ID|DG=DriveGroup
DHS=Dedicated Hot Spare|UGood=Unconfigured Good|GHS=Global Hotspare
UBad=Unconfigured Bad|Sntze=Sanitize|Onln=Online|Offln=Offline|Intf=Interface
Med=Media Type|SED=Self Encryptive Drive|PI=Protection Info
SeSz=Sector Size|Sp=Spun|U=Up|D=Down|T=Transition|F=Foreign
UGUnsp=UGood Unsupported|UGShld=UGood shielded|HSPShld=Hotspare shielded
CFShld=Configured shielded|Cpybck=CopyBack|CBShld=Copyback Shielded
UBUnsp=UBad Unsupported|Rbld=Rebuild
Enclosures = 1
Enclosure LIST :
==============
--------------------------------------------------------------------
EID State Slots PD PS Fans TSs Alms SIM Port# ProdID VendorSpecific
--------------------------------------------------------------------
64 OK 8 6 0 0 0 0 0 - BP14G+ ?
--------------------------------------------------------------------
EID=Enclosure Device ID | PD=Physical drive count | PS=Power Supply count
TSs=Temperature sensor count | Alms=Alarm count | SIM=SIM Count | ProdID=Product ID
BBU_Info :
========
----------------------------------------------
Model State RetentionTime Temp Mode MfgDate
----------------------------------------------
BBU Optimal 0 hour(s) 48C - 0/00/00
----------------------------------------------
So BBU seems enabled.
--
Latex fetish implies the existence of a MS Word fetish.
(@nucleus)
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
11:28 p.m.
Hi,
> Matthias suggest to enable write cache, you
suggest to disble it... or i'm
> cache-confused?! ;-)
there were some discussions about write cache settings last year, e.g.
https://www.spinics.net/lists/ceph-users/msg73263.html
https://www.spinics.net/lists/ceph-users/msg69489.html
The entire threads are worth reading, but it is easy to get throroghly
confused.
In the first linked mail, Dan van der Ster points to this page:
https://docs.ceph.com/en/latest/start/hardware-recommendations/#write-caches
Essentially as Frank Schilder said: fio-test the drives with different
settings, as some perform better when write-back cache is disabled, and
some perform better with write-back cache enabled.
And in order to not lose all of your data at once in case of a power
outage: make sure your drives are not el-cheapo consumer ones which
might lie to you about write completion (on either flushes or
write-through writes). If a drive is not advertised for DC (data center)
use, most probably it is such a dangerous consumer-grade drive and you
better stay away from it.
In case of power loss, drives with "power loss prevention" (large
capacitor) do have a little more time to persist in-cache data to stable
storage. Such drives may safely signal back write completion _before_
data has actually reached flash chips (or platters), their capacitor
allows them to still persist the cache data on power loss. This gives
them more opportunity for internal optimizations without risking
durability, and could increase IOPS very much. Of course, individual
testing is still necessary to find optimal cache settings for each
drive.
Matthias
On Mon, Apr 17, 2023 at 07:32:42AM +0000, Frank Schilder wrote:
> Hi Marco.
>
> >> For your disk type I saw "volatile write cache available = yes" on
"the internet". This looks a bit odd, but maybe these HDDs do have some volatile
cache. Try to disable it with smartctl and do the benchmark again.
> >
> > Sorry, i'm a bit puzzled here.
> >
> Matthias suggest to enable write cache, you
suggest to disble it... or i'm
> cache-confused?! ;-)
>
> You need to disable *volatile* write cache. In order to figure out what is volatile
or not, the best is to start disabling everything. This is usually the best way to run
ceph anyways. Some expensive controllers claim to have non-volatile write cache. This is
usually activated by disabling cache (yes, I know how this sounds), because this option
typically refers to volatile and not the non-volatile cache.
>
> I would start by getting the disks into non-raid mode and disable all caches I can
find. Then benchmark again. Then, maybe, enable caches again one by one and see if
anything improves. If not or if it gets worse, disable it again.
>
> Best regards,
> Frank
>
> =================
> Frank Schilder
> AIT Risø Campus
> Bygning 109, rum S14
>
> ________________________________________
> From: Konstantin Shalygin <k0ste(a)k0ste.ru>
> Sent: Saturday, April 15, 2023 11:49 AM
> To: Marco Gaiarin
> Cc: Frank Schilder; ceph-users(a)ceph.io
> Subject: Re: [ceph-users] Re: Some hint for a DELL PowerEdge T440/PERC H750
Controller...
>
> Hi,
>
> Current controller mode is RAID. You can switch to HBA mode and disable cache in
controller settings at the BIOS
>
>
> k
> Sent from my iPhone
>
> > On 15 Apr 2023, at 12:11, Marco Gaiarin <gaio(a)lilliput.linux.it> wrote:
> >
> > Mandi! Frank Schilder
> > In chel di` si favelave...
> >
> >>> iops : min= 2, max= 40, avg=21.13, stdev= 6.10,
samples=929
> >>> iops : min= 2, max= 42, avg=21.52, stdev= 6.56,
samples=926
> >> That looks horrible.
> >
> > Exactly, horrible.
> >
> > The strange thing is that we came from an homegrown Ceph cluster built using
> > old hardware (HP G6 servers) and spare disks, that perform 'better', or
at
> > least perform 'more uniformly' that this 'new'.
> >
> > With these embarassing IOPS, sooner or later we reach the point where
> > performances go to the ground, and sometimes it suffices to launch some
> > 'find' on the filesystem involved...
> >
> >
> >> We also have a few SATA HDDs in Dell servers and they do about 100-150 IOP/s
read or write. Originally, I was also a bit afraid that these disks would drag performance
down, but they are on par with the NL-SAS drives.
> >> For ceph we use the cheapest Dell disk controller one can get (Dell HBA330
Mini (Embedded)) and it works perfectly. All ceph-disks are configured non-raid, which is
equivalent to JBOD mode or pass-through. These controllers have no cache options, if your
do, disable all of them. Mode should be write-through.
> >> For your disk type I saw "volatile write cache available = yes" on
"the internet". This looks a bit odd, but maybe these HDDs do have some volatile
cache. Try to disable it with smartctl and do the benchmark again.
> >
> > Sorry, i'm a bit puzzled here.
> >
> Matthias suggest to enable write cache, you
suggest to disble it... or i'm
> cache-confused?! ;-)
> >
> >
> > My actually controller configuration is:
> >
> > root@pppve1:~# perccli /c0 show
> > Generating detailed summary of the adapter, it may take a while to complete.
> >
> > CLI Version = 007.1910.0000.0000 Oct 08, 2021
> > Operating system = Linux 5.4.203-1-pve
> > Controller = 0
> > Status = Success
> > Description = None
> >
> > Product Name = PERC H750 Adapter
> > Serial Number = 23L01Y6
> > SAS Address = 5f4ee0802ba3a400
> > PCI Address = 00:b3:00:00
> > System Time = 04/14/2023 18:03:24
> > Mfg. Date = 03/25/22
> > Controller Time = 04/14/2023 16:03:22
> > FW Package Build = 52.16.1-4405
> > BIOS Version = 7.16.00.0_0x07100501
> > FW Version = 5.160.02-3552
> > Driver Name = megaraid_sas
> > Driver Version = 07.713.01.00-rc1
> > Current Personality = RAID-Mode
> > Vendor Id = 0x1000
> > Device Id = 0x10E2
> > SubVendor Id = 0x1028
> > SubDevice Id = 0x2176
> > Host Interface = PCI-E
> > Device Interface = SAS-12G
> > Bus Number = 179
> > Device Number = 0
> > Function Number = 0
> > Domain ID = 0
> > Security Protocol = None
> > JBOD Drives = 6
> >
> > JBOD LIST :
> > =========
> >
> >
---------------------------------------------------------------------------------
> > ID EID:Slt DID State Intf Med Size SeSz Model Vendor
Port
> >
---------------------------------------------------------------------------------
> > 0 64:0 6 Onln SATA SSD 447.130 GB 512B MTFDDAK480TDT ATA x1
> > 1 64:1 8 Onln SATA SSD 447.130 GB 512B MTFDDAK480TDT ATA x1
> > 3 64:3 7 Onln SATA SSD 447.130 GB 512B MZ7KH480HAHQ0D3 ATA x1
> > 5 64:5 9 Onln SATA HDD 3.638 TB 512B HGST HUS726T4TALA6L0 ATA x1
> > 6 64:6 10 Onln SATA HDD 3.638 TB 512B HGST HUS726T4TALA6L0 ATA x1
> > 7 64:7 11 Onln SATA HDD 3.638 TB 512B HGST HUS726T4TALA6L0 ATA x1
> >
---------------------------------------------------------------------------------
> >
> > ID=JBOD Target ID|EID=Enclosure Device ID|Slt=Slot No|DID=Device ID|Onln=Online
> > Offln=Offline|Intf=Interface|Med=Media Type|SeSz=Sector Size
> > SED=Self Encryptive Drive|PI=Protection Info|Sp=Spun|U=Up|D=Down
> >
> > Physical Drives = 6
> >
> > PD LIST :
> > =======
> >
> >
----------------------------------------------------------------------------------
> > EID:Slt DID State DG Size Intf Med SED PI SeSz Model Sp
Type
> >
----------------------------------------------------------------------------------
> > 64:0 6 Onln - 447.130 GB SATA SSD N N 512B MTFDDAK480TDT U
JBOD
> > 64:1 8 Onln - 447.130 GB SATA SSD N N 512B MTFDDAK480TDT U
JBOD
> > 64:3 7 Onln - 447.130 GB SATA SSD N N 512B MZ7KH480HAHQ0D3 U
JBOD
> > 64:5 9 Onln - 3.638 TB SATA HDD N N 512B HGST HUS726T4TALA6L0 U
JBOD
> > 64:6 10 Onln - 3.638 TB SATA HDD N N 512B HGST HUS726T4TALA6L0 U
JBOD
> > 64:7 11 Onln - 3.638 TB SATA HDD N N 512B HGST HUS726T4TALA6L0 U
JBOD
> >
----------------------------------------------------------------------------------
> >
> > EID=Enclosure Device ID|Slt=Slot No|DID=Device ID|DG=DriveGroup
> > DHS=Dedicated Hot Spare|UGood=Unconfigured Good|GHS=Global Hotspare
> > UBad=Unconfigured Bad|Sntze=Sanitize|Onln=Online|Offln=Offline|Intf=Interface
> > Med=Media Type|SED=Self Encryptive Drive|PI=Protection Info
> > SeSz=Sector Size|Sp=Spun|U=Up|D=Down|T=Transition|F=Foreign
> > UGUnsp=UGood Unsupported|UGShld=UGood shielded|HSPShld=Hotspare shielded
> > CFShld=Configured shielded|Cpybck=CopyBack|CBShld=Copyback Shielded
> > UBUnsp=UBad Unsupported|Rbld=Rebuild
> >
> > Enclosures = 1
> >
> > Enclosure LIST :
> > ==============
> >
> > --------------------------------------------------------------------
> > EID State Slots PD PS Fans TSs Alms SIM Port# ProdID VendorSpecific
> > --------------------------------------------------------------------
> > 64 OK 8 6 0 0 0 0 0 - BP14G+ ?
> > --------------------------------------------------------------------
> >
> > EID=Enclosure Device ID | PD=Physical drive count | PS=Power Supply count
> > TSs=Temperature sensor count | Alms=Alarm count | SIM=SIM Count | ProdID=Product
ID
> >
> >
> > BBU_Info :
> > ========
> >
> > ----------------------------------------------
> > Model State RetentionTime Temp Mode MfgDate
> > ----------------------------------------------
> > BBU Optimal 0 hour(s) 48C - 0/00/00
> > ----------------------------------------------
> >
> >
> > So BBU seems enabled.
> >
> > --
> > Latex fetish implies the existence of a MS Word fetish.
> > (@nucleus)
> >
> > _______________________________________________
> > ceph-users mailing list -- ceph-users(a)ceph.io
> > To unsubscribe send an email to ceph-users-leave(a)ceph.io
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
19 Apr
19 Apr
7:31 p.m.
Mandi! Matthias Ferdinand
In chel di` si favelave...
In the first linked mail, Dan van der Ster points to
this page:
https://docs.ceph.com/en/latest/start/hardware-recommendations/#write-caches
root@pppve1:~# for d in a b c d e f; do smartctl -g wcache /dev/sd$d | grep ^Write; done
Write cache is: Enabled
Write cache is: Enabled
Write cache is: Enabled
Write cache is: Disabled
Write cache is: Disabled
Write cache is: Disabled
configuration seems just correct: sda, sdb and sdc, SDD disks, have cache
enabled; sdd, sde and sdf that are HDD are disabled.
--
Se non trovi nessuno vuol dire che siamo scappati alle sei-shell (bash,
tcsh,csh...) (Possi)
7:25 p.m.
Mandi! Konstantin Shalygin
In chel di` si favelave...
Current controller mode is RAID. You can switch to HBA
mode and disable cache in controller settings at the BIOS
No, is a bit complex then that.
Controller does not have an 'HBA-mode', but a 'AutoRAID0' mode.
--
Documentation is like sex: when it is good, it is very, very good; and
when it is bad, it is better than nothing.
(from texinfo documentation) Dick Brandon
11:52 a.m.
Il giorno sab 15 apr 2023 alle ore 11:10 Marco Gaiarin <
gaio(a)lilliput.linux.it> ha scritto:
Sorry, i'm a bit puzzled here.
Matthias suggest to enable write cache, you suggest to disble it... or i'm
cache-confused?! ;-)
There is a cache in each disk, and a cache in the controller.
The disk cache is:
- volatile, if power goes away content is lost (some nvme disks have PLP
that is a capacitor so cache in this case is not volatile)
- ecc only in sas disks
- needed for ssd (most ssd wear leveling algorithms have problems without
cache)
The controller cache:
- is not volatile, if the controller has BBU
- usually has ecc ram
Usually when you have a BBU raid controller you disable single disks cache
and keep only controller cache which is protected by battery.
Ceph experts tell you to disable controller cache and raid but no one was
able to tell me what to do with disk cache:
- if you disable it disk performance becomes very slow and you risk
problems in ssd
- if you enable it disk will lose content if power goes away.
Now ceph experts will tell you to buy only disks with PLP to solve the
problem... but hdds do not have plp (if you find an hdd with plp tell me I
will buy it)
7:57 p.m.
Mandi! Mario Giammarco
In chel di` si favelave...
The disk cache is:
If the controller does not lie, disk cache is disabled, see my previous
messages.
The controller cache:
Manual say that for Non-RAID/Automatic RAID0 disks, «The only supported
cache policy for non???RAID disks is Write-Through»; so , AFAIK, disabled.
https://dl.dell.com/content/manual61357984-dell-poweredge-raid-controller-1…
reading through the controller docs, seems to me that there's very little to
tackle...
--
Il mondo del calcio sta andando a puttane!
Perché, hanno finito le letterine? (Marco Citi)
9:04 p.m.
Actually there was a firmware bug around that a while back. The HBA and storcli claimed
to not touch drive cache, but actually were enabling it and lying.
> On Apr 19, 2023, at 1:41 PM, Marco Gaiarin <gaio(a)lilliput.linux.it> wrote:
>
> Mandi! Mario Giammarco
> In chel di` si favelave...
>
>> The disk cache is:
>
> If the controller does not lie, disk cache is disabled, see my previous
> messages.
>
>
>> The controller cache:
>
> Manual say that for Non-RAID/Automatic RAID0 disks, «The only supported
> cache policy for non???RAID disks is Write-Through»; so , AFAIK, disabled.
>
>
https://dl.dell.com/content/manual61357984-dell-poweredge-raid-controller-1…
>
>
> reading through the controller docs, seems to me that there's very little to
> tackle...
>
> --
> Il mondo del calcio sta andando a puttane!
> Perché, hanno finito le letterine? (Marco Citi)
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
20 Apr
20 Apr
1:32 p.m.
Mandi! Anthony D'Atri
In chel di` si favelave...
Actually there was a firmware bug around that a while
back. The HBA and storcli claimed to not touch drive cache, but actually were enabling it
and lying.
Some pointer to the issue? I doubt hit me but...
Thanks.
--
ma l'impresa eccezionale, dammi retta
e` essere normale (L. Dalla)
11 Apr
11 Apr
7:41 p.m.
Hi,
do you want to hear the truth from real experience?
Or the myth?
The truth is that:
- hdd are too slow for ceph, the first time you need to do a rebalance or
similar you will discover...
- if you want to use hdds do a raid with your controller and use the
controller BBU cache (do not consider controllers with hdd cache), and
present the raid as one ceph disk.
- enabling single hdd write cache (that is not battery protected) is far
worse than enabling controller cache (which I assume is always protected by
BBU)
- anyway the best thing for ceph is to use nvme disks.
Mario
Il giorno gio 6 apr 2023 alle ore 13:40 Marco Gaiarin <
gaio(a)lilliput.linux.it> ha scritto:
We are testing an experimental Ceph cluster with server and controller at
subject.
The controller have not an HBA mode, but only a 'NonRAID' mode, come sort
of
'auto RAID0' configuration.
We are using SSD SATA disks (MICRON MTFDDAK480TDT) that perform very well,
and SAS HDD disks (SEAGATE ST8000NM014A) that instead perform very bad
(particulary, very low IOPS).
There's some hint for disk/controller configuration/optimization?
Thanks.
--
Io credo nella chimica tanto quanto Giulio Cesare credeva nel caso...
mi va bene fino a quando non riguarda me :) (Emanuele Pucciarelli)
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
8:23 p.m.
The truth is that:
- hdd are too slow for ceph, the first time you need to do a rebalance or
similar you will discover...
Depends on the needs. For cold storage, or sequential use-cases that aren't
performance-sensitive ... Can't say "too slow" without context. In
Marco's case, I wonder how the results might differ with numjobs=1 -- with a value of
4 as reported, seems to me like the drive will be seeking an awful lot. Mind you many
Ceph multi-client workloads exhibit the "IO Blender" effect where they present
to the drives as random, but this FIO job may not be entirely indicative.
If you have to expand just to get more IOPs, that's a different story.
- if you want to use hdds do a raid with your
controller and use the
controller BBU cache (do not consider controllers with hdd cache), and
present the raid as one ceph disk.
Take care regarding OSD and PG counts with that strategy. Plus, Ceph does replication, so
replication under the OSD layer can be ... gratuitous.
- enabling single hdd write cache (that is not battery
protected) is far
worse than enabling controller cache (which I assume is always protected by
BBU)
There are plenty of RoC HBAs out there without cache RAM or BBU/supercap, and also ones
with cache RAM but without BBU/supercap. These often default to writethrough caching and
arguably don't have much or any net benefit.
- anyway the best thing for ceph is to use nvme disks.
I wouldn't disagree, but it's not entirely cut and dried. Notably the cost and
hassle of an RoC HBA, cache, BBU/supercap, additional monitoring, replacement ... See my
post a few years back about reasons I don't like RoC HBAs. Go with a plain, non-RoC
HBA and the savings can help justify going with SATA SSDs at a minimum.
Mario
Il giorno gio 6 apr 2023 alle ore 13:40 Marco Gaiarin <
gaio(a)lilliput.linux.it> ha scritto:
We are testing an experimental Ceph cluster with server and controller at
subject.
The controller have not an HBA mode, but only a 'NonRAID' mode, come sort
of
'auto RAID0' configuration.
We are using SSD SATA disks (MICRON MTFDDAK480TDT) that perform very well,
and SAS HDD disks (SEAGATE ST8000NM014A) that instead perform very bad
(particulary, very low IOPS).
There's some hint for disk/controller configuration/optimization?
Thanks.
--
Io credo nella chimica tanto quanto Giulio Cesare credeva nel caso...
mi va bene fino a quando non riguarda me :) (Emanuele Pucciarelli)
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io
To unsubscribe send an email to ceph-users-leave(a)ceph.io
393
days inactive
407
days old
18 comments
7 participants
participants (7)
-
Anthony D'Atri -
Anthony D'Atri -
Frank Schilder -
Konstantin Shalygin -
Marco Gaiarin -
Mario Giammarco -
Matthias Ferdinand