Hi all,
We have a hammer ceph cluster with 3 monitor,324 osds. OSD daemon and kvm is collocated
on node;
The ceph cluster are runing 2 years.Recently we added ~700 osds to the cluster,as
process:
1.ceph osd create
2. mkdir -p /var/lib/ceph/osd/ceph-$osd
3. mkfs.xfs -f /dev/$disk
4. mount -o inode64,notime /dev/$disk /var/lib/ceph/osd/ceph-$osd
5. ceph-osd -i 0 --mkfs --mkkey
6.ceph auth add osd.$osd osd 'allow *' mon 'allow profile osd' -i
/var/lib/ceph/osd/ceph-$osd/keyring
7.ceph osd crush create-or-move $osd host=kvm101 root=default
Mabe we do that requently.After add 122 osds, osd.1-osd.8 failed
2020-05-14 16:48:29.881021 7f6727fb9700 -1 common/Thread.cc: In function 'void
Thread::create(size_t)' thread 7f6727fb9700 time 2020-05-14 16:48:29.870051
common/Thread.cc: 129: FAILED assert(ret == 0)
ceph version 0.94.5 (9764da52395923e0b32908d83a9f7304401fee43)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85)
[0xbc8b55]
2: (Thread::create(unsigned long)+0x8a) [0xbac50a]
3: (Pipe::accept()+0x37fb) [0xca6c3b]
4: (Pipe::reader()+0x1a0f) [0xcaa75f]
5: (Pipe::Reader::entry()+0xd) [0xcb351d]
6: (()+0x7dc5) [0x7f67a45ebdc5]
7: (clone()+0x6d) [0x7f67a30cc1cd]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to
interpret this.
ulimit -u
2061600
open files 32768
Does anyone know what's going on? Why create thread faild?
Show replies by date