24 Feb
2023
24 Feb
'23
7:48 p.m.
Hi Andrej,
That doesn't sound right -- I checked a couple of our clusters just
now and the mon filesystem is writing at just a few 100kBps.
debug_mon = 10 should clarify the root cause. Perhaps it's logm from
some persistent slow ops?
Cheers, Dan
On Fri, Feb 24, 2023 at 7:36 AM Andrej Filipcic <andrej.filipcic(a)ijs.si> wrote:
>
>
> Hi,
>
> on our large ceph cluster with 60 servers, 1600 OSDs, we have observed
> that small system nvmes are wearing out rapidly. Our monitoring shows
> mon writes on average about 10MB/s to store.db. For small system nvmes
> of 250GB and DWPD of ~1, this turns out to be too much, 0.8TB/day or
> 1.5PB in 5 years, too much even for 3DWPD of the same capacity.
>
> Apart from replacing the drives with larger ones, more durable,
> preferably both, do you have any suggestions if these writes can be
> reduced? Actually, the mon writes match 0.15Hz rate of .sst file
> creation of 64MB....
>
> Best regards,
> Andrej
>
> --
> _____________________________________________________________
> prof. dr. Andrej Filipcic, E-mail: Andrej.Filipcic(a)ijs.si
> Department of Experimental High Energy Physics - F9
> Jozef Stefan Institute, Jamova 39, P.o.Box 3000
> SI-1001 Ljubljana, Slovenia
> Tel.: +386-1-477-3674 Fax: +386-1-425-7074
> -------------------------------------------------------------
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io