To summarize this thread. 

I split NVME device to support both DB+WALL and OSD : 

nvme delete-ns /dev/nvme0 -n 1
nvme create-ns /dev/nvme0 --nsze=209715200 --ncap=209715200  --flbas=0 -dps=0
nvme create-ns /dev/nvme0 --nsze=2915912368 --ncap=2915912368  --flbas=0 -dps=0
nvme attach-ns /dev/nvme0 --namespace-id=1 -controllers=0x41
nvme attach-ns /dev/nvme0 --namespace-id=2 -controllers=0x41
echo 1 > /sys/class/nvme/nvme0/rescan_controller

All system and utils pools are redirected to replicated pools based on NVME OSDs. 

s3 write performance benchmark result increased from 1.7GB/s to 2.04 GB.s and there is traffic on index and non-ec : 

image.png

NVME device during benchmark: 

Device            r/s     rMB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wMB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dMB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
nvme0c0n1(db+wall)        0.00      0.00     0.00   0.00    0.00     0.00 1822.00      9.00   483.00  20.95    0.03     5.06    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.05  96.00
nvme0c0n2 (osd)           0.00      0.00     0.00   0.00    0.00     0.00  214.00      1.04    51.00  19.25    0.03     4.95    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.01  34.00

Thanks! 


On Tue, 4 Jun 2024 at 21:01, Lukasz Borek <lukasz@borek.org.pl> wrote:
You could check if your devices support NVMe namespaces and create more 
than one namespace on the device.
Wow, tricky. Will give it a try. 

Thanks!

____
Łukasz Borek
lukasz@borek.org.pl


On Tue, 4 Jun 2024 at 16:26, Robert Sander <r.sander@heinlein-support.de> wrote:
Hi,

On 6/4/24 16:15, Anthony D'Atri wrote:

> I've wondered for years what the practical differences are between using a namespace and a conventional partition.

Namespaces show up as separate block devices in the kernel.

The orchestrator will not touch any devices that contain a partition
table or logical volume signatures.

Regards
--
Robert Sander
Heinlein Consulting GmbH
Schwedter Str. 8/9b, 10119 Berlin

https://www.heinlein-support.de

Tel: 030 / 405051-43
Fax: 030 / 405051-19

Amtsgericht Berlin-Charlottenburg - HRB 220009 B
Geschäftsführer: Peer Heinlein - Sitz: Berlin
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-leave@ceph.io


--
Łukasz Borek
lukasz@borek.org.pl