sysctl: kernel.pid_max=4194303 kernel.threads-max=2097152 vm.max_map_count=524288 But really, why are you still running Hammer? Later releases handle a large number of OSDs *much* better. > On Jun 1, 2020, at 7:08 PM, 展荣臻（信泰） &lt;zhanrzh_xt(a)teamsun.com.cn&gt; wrote: > > Hi all, > We have a hammer ceph cluster with 3 monitor,324 osds. OSD daemon and kvm is collocated on node; > The ceph cluster are runing 2 years.Recently we added ~700 osds to the cluster,as process: > 1.ceph osd create > 2. mkdir -p /var/lib/ceph/osd/ceph-$osd > 3. mkfs.xfs -f /dev/$disk > 4. mount -o inode64,notime /dev/$disk /var/lib/ceph/osd/ceph-$osd > 5. ceph-osd -i 0 --mkfs --mkkey > 6.ceph auth add osd.$osd osd 'allow *' mon 'allow profile osd' -i /var/lib/ceph/osd/ceph-$osd/keyring > 7.ceph osd crush create-or-move $osd host=kvm101 root=default > Mabe we do that requently.After add 122 osds, osd.1-osd.8 failed > > 2020-05-14 16:48:29.881021 7f6727fb9700 -1 common/Thread.cc: In function 'void Thread::create(size_t)' thread 7f6727fb9700 time 2020-05-14 16:48:29.870051 > common/Thread.cc: 129: FAILED assert(ret == 0) > > ceph version 0.94.5 (9764da52395923e0b32908d83a9f7304401fee43) > 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0xbc8b55] > 2: (Thread::create(unsigned long)+0x8a) [0xbac50a] > 3: (Pipe::accept()+0x37fb) [0xca6c3b] > 4: (Pipe::reader()+0x1a0f) [0xcaa75f] > 5: (Pipe::Reader::entry()+0xd) [0xcb351d] > 6: (()+0x7dc5) [0x7f67a45ebdc5] > 7: (clone()+0x6d) [0x7f67a30cc1cd] > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. > > ulimit -u > 2061600 > open files 32768 > > > Does anyone know what's going on? Why create thread faild? > > > > > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io