On 2/27/23 03:22, Andrej Filipcic wrote: Compression is the first thing that comes to mind, though I'd caution you to be careful. Try it out on a test cluster first if possible. You could also try tuning rocksdb's other configurables, but it's been a long time since anyone looked at rocksdb tuning for the mons afaik. I'd suggest reading through our OSD rocksdb tuning article to get some ideas: https://ceph.io/en/news/blog/2022/rocksdb-tuning-deep-dive/ Hope that helps! Mark >> >> Cheers, Dan >> >> >> >> On Fri, Feb 24, 2023 at 7:36 AM Andrej Filipcic >> <andrej.filipcic(a)ijs.si> wrote: >>> >>> Hi, >>> >>> on our large ceph cluster with 60 servers, 1600 OSDs, we have observed >>> that small system nvmes are wearing out rapidly. Our monitoring shows >>> mon writes on average about 10MB/s to store.db. For small system nvmes >>> of 250GB and DWPD of ~1, this turns out to be too much, 0.8TB/day or >>> 1.5PB in 5 years, too much even for 3DWPD of the same capacity. >>> >>> Apart from replacing the drives with larger ones, more durable, >>> preferably  both, do you have any suggestions if these writes can be >>> reduced? Actually, the mon writes match 0.15Hz  rate of .sst file >>> creation of 64MB.... >>> >>> Best regards, >>> Andrej >>> >>> -- >>> _____________________________________________________________ >>>      prof. dr. Andrej Filipcic,   E-mail: Andrej.Filipcic(a)ijs.si >>>      Department of Experimental High Energy Physics - F9 >>>      Jozef Stefan Institute, Jamova 39, P.o.Box 3000 >>>      SI-1001 Ljubljana, Slovenia >>>      Tel.: +386-1-477-3674    Fax: +386-1-425-7074 >>> ------------------------------------------------------------- >>> _______________________________________________ >>> ceph-users mailing list -- ceph-users(a)ceph.io >>> To unsubscribe send an email to ceph-users-leave(a)ceph.io > >