[ceph-users] Thread::try_create(): pthread_create failed

Hi all, We have a hammer ceph cluster with 3 monitor,324 osds. OSD daemon and kvm is collocated on node; The ceph cluster are runing 2 years.Recently we added ~700 osds to the cluster,as process: 1.ceph osd create 2. mkdir -p /var/lib/ceph/osd/ceph-$osd 3. mkfs.xfs -f /dev/$disk 4. mount -o inode64,notime /dev/$disk /var/lib/ceph/osd/ceph-$osd 5. ceph-osd -i 0 --mkfs --mkkey 6.ceph auth add osd.$osd osd 'allow *' mon 'allow profile osd' -i /var/lib/ceph/osd/ceph-$osd/keyring 7.ceph osd crush create-or-move $osd host=kvm101 root=default Mabe we do that requently.After add 122 osds, osd.1-osd.8 failed 2020-05-14 16:48:29.881021 7f6727fb9700 -1 common/Thread.cc: In function 'void Thread::create(size_t)' thread 7f6727fb9700 time 2020-05-14 16:48:29.870051 common/Thread.cc: 129: FAILED assert(ret == 0) ceph version 0.94.5 (9764da52395923e0b32908d83a9f7304401fee43) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0xbc8b55] 2: (Thread::create(unsigned long)+0x8a) [0xbac50a] 3: (Pipe::accept()+0x37fb) [0xca6c3b] 4: (Pipe::reader()+0x1a0f) [0xcaa75f] 5: (Pipe::Reader::entry()+0xd) [0xcb351d] 6: (()+0x7dc5) [0x7f67a45ebdc5] 7: (clone()+0x6d) [0x7f67a30cc1cd] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. ulimit -u 2061600 open files 32768 Does anyone know what's going on? Why create thread faild?