[ceph-users] OSDs get full with bluestore logs

19 Apr 2020

Hi,

I have a 3 node cluster of mimic with 9 osds (3 osds on each node).
I use this cluster to test integration of an application with S3 api.

The problem is that after a few days all OSD starts filling up with
bluestore logs and goes down and out one by one!
I cannot stop the logs and I cannot find the setting to fix this leakage,
this should be a leakage in logs because it's not logical to fill up all
OSD with bluefs logs.

This is an example of logs which is being repeated in bluestore logs:

[root@server2 ~]# ceph-bluestore-tool --command bluefs-log-dump --path
/var/lib/ceph/osd/ceph-5
.
.
. 0x40d000:  op_file_update  file(ino 30 size 0x7a713 mtime 2020-04-18
15:54:29.056488 bdev 1 allocated 80000 extents
[1:0x130000+10000,1:0x100000+10000,1:0x140000+10000,1:0x150000+10000,1:0x160000+10000,1:0x170000+10000,1:0x180000+10000,1:0x190000+10000])
 0x40e000: txn(seq 1156100 len 0x78 crc 0x3e1c626f)
 0x40e000:  op_file_update  file(ino 30 size 0x7a72a mtime 2020-04-18
15:54:30.057828 bdev 1 allocated 80000 extents
[1:0x130000+10000,1:0x100000+10000,1:0x140000+10000,1:0x150000+10000,1:0x160000+10000,1:0x170000+10000,1:0x180000+10000,1:0x190000+10000])
 0x40f000: txn(seq 1156101 len 0x78 crc 0xc1f9ec5f)
 0x40f000:  op_file_update  file(ino 30 size 0x7a741 mtime 2020-04-18
15:54:31.059252 bdev 1 allocated 80000 extents
[1:0x130000+10000,1:0x100000+10000,1:0x140000+10000,1:0x150000+10000,1:0x160000+10000,1:0x170000+10000,1:0x180000+10000,1:0x190000+10000])
*** Caught signal (Segmentation fault) **
 in thread 7f4108cfe600 thread_name:ceph-bluestore-
 ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic
(stable)
 1: (()+0xf5f0) [0x7f40fd47f5f0]
 2: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned
long, unsigned long, ceph::buffer::list*, char*)+0x3bf) [0x55603a1ebb3f]
 3: (BlueFS::_replay(bool, bool)+0x214) [0x55603a1f6654]
 4: (BlueFS::log_dump(CephContext*, std::string const&,
std::vector<std::string, std::allocator<std::string> > const&)+0x3b)
[0x55603a1fad3b]
 5: (log_dump(CephContext*, std::string const&, std::vector<std::string,
std::allocator<std::string> > const&)+0x64) [0x55603a1da764]
 6: (main()+0x2f3e) [0x55603a105e3e]
 7: (__libc_start_main()+0xf5) [0x7f40fbe51505]
 8: (()+0x23115f) [0x55603a1d915f]
2020-04-18 21:08:25.410 7f4108cfe600 -1 *** Caught signal (Segmentation
fault) **
 in thread 7f4108cfe600 thread_name:ceph-bluestore-
.
.
.

And this is the output of daemonperf for one of remaining (filling up!)
OSDs:

[root@server2 ~]# ceph daemonperf osd.5
------bluefs------- ------------bluestore-------------
----------osd-----------
jlen j    wal  sst |fl_l k_l  io_l th_l s_l  c_l  r_l |ops  wr   rd   l
 rop |
 81M   0    0    0 |  0    0    0    0    0    0    0 |  0    0    0    0
 0
 81M 4.0k 1.8k   0 |  0    0    0    0    0    0    0 |  0    0    0    0
 0
 81M 4.0k 1.9k   0 |  0    0    0    0    0    0    0 |  0    0    0    0
 0
 81M 4.0k 1.9k   0 |  0    0    0    0    0    0    0 |  0    0    0    0
 0
 81M 4.0k 1.9k   0 |  0    0    0    0    0    0    0 |  0    0    0    0
 0
^C[root@server2 ~]#

And this is the enospc logs when trying to start OSD:

[root@server2 ~]# journalctl -u -f ceph-osd(a)4.service
Failed to add match &#39;ceph-osd(a)4.service&#39;ice': Invalid argument
Failed to add filters: Invalid argument
[root@server2 ~]# journalctl -f -u ceph-osd(a)4.service
-- Logs begin at Sat 2020-04-18 15:38:15 +0430. --
Apr 18 21:18:22 server2 ceph-osd[17485]: 22: (()+0x378c10) [0x557348ebac10]
Apr 18 21:18:22 server2 ceph-osd[17485]: NOTE: a copy of the executable, or
`objdump -rdS <executable>` is needed to interpret this.
Apr 18 21:18:22 server2 systemd[1]: ceph-osd(a)4.service: main process
exited, code=killed, status=6/ABRT
Apr 18 21:18:22 server2 systemd[1]: Unit ceph-osd(a)4.service entered failed
state.
Apr 18 21:18:22 server2 systemd[1]: ceph-osd(a)4.service failed.
Apr 18 21:18:42 server2 systemd[1]: ceph-osd(a)4.service holdoff time over,
scheduling restart.
Apr 18 21:18:42 server2 systemd[1]: Stopped Ceph object storage daemon
osd.4.
Apr 18 21:18:42 server2 systemd[1]: Starting Ceph object storage daemon
osd.4...
Apr 18 21:18:42 server2 systemd[1]: Started Ceph object storage daemon
osd.4.
Apr 18 21:18:43 server2 ceph-osd[17777]: starting osd.4 at - osd_data
/var/lib/ceph/osd/ceph-4 /var/lib/ceph/osd/ceph-4/journal
Apr 18 21:24:37 server2 ceph-osd[17777]:
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc:
In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t,
uint64_t)' thread 7f43c936bb80 time 2020-04-18 21:24:37.091289
Apr 18 21:24:37 server2 ceph-osd[17777]:
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc:
1704: FAILED assert(0 == "bluefs enospc")
Apr 18 21:24:37 server2 ceph-osd[17777]: 2020-04-18 21:24:37.090
7f43c936bb80 -1 bluefs _allocate failed to allocate 0x on bdev 2, dne
Apr 18 21:24:37 server2 ceph-osd[17777]: 2020-04-18 21:24:37.090
7f43c936bb80 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length:
0x794de9
Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8
(5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable)
Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (ceph::__ceph_assert_fail(char
const*, char const*, int, char const*)+0x14b) [0x7f43c074987b]
Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (()+0x26fa07) [0x7f43c0749a07]
Apr 18 21:24:37 server2 ceph-osd[17777]: 3:
(BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned
long)+0x1ac6) [0x55ad52029266]
Apr 18 21:24:37 server2 ceph-osd[17777]: 4:
(BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd]
Apr 18 21:24:37 server2 ceph-osd[17777]: 5:
(rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916]
Apr 18 21:24:37 server2 ceph-osd[17777]: 6:
(rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde]
Apr 18 21:24:37 server2 ceph-osd[17777]: 7:
(rocksdb::BuildTable(std::string const&, rocksdb::Env*,
rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&,
rocksdb::EnvOptions const&, rocksdb::TableCache*,
rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator,
std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*,
rocksdb::InternalKeyComparator const&,
std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
std::default_delete<rocksdb::IntTblPropCollectorFactory> >,
std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*,
unsigned int, std::string const&, std::vector<unsigned long,
std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*,
rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool,
rocksdb::InternalStats*, rocksdb::TableFileCreationReason,
rocksdb::EventLogger*, int, rocksdb::Env::IOPriority,
rocksdb::TableProperties*, int, unsigned long, unsigned long,
rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8]
Apr 18 21:24:37 server2 ceph-osd[17777]: 8:
(rocksdb::DBImpl::WriteLevel0TableForRecovery(int,
rocksdb::ColumnFamilyData*, rocksdb::MemTable*,
rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76]
Apr 18 21:24:37 server2 ceph-osd[17777]: 9:
(rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long,
std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b)
[0x55ad520b2dcb]
Apr 18 21:24:37 server2 ceph-osd[17777]: 10:
(rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor,
std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool,
bool)+0xa59) [0x55ad520b3d09]
Apr 18 21:24:37 server2 ceph-osd[17777]: 11:
(rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&,
std::vector<rocksdb::ColumnFamilyDescriptor,
std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
std::vector<rocksdb::ColumnFamilyHandle*,
std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**,
bool)+0x689) [0x55ad520b4ab9]
Apr 18 21:24:37 server2 ceph-osd[17777]: 12:
(rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&,
std::vector<rocksdb::ColumnFamilyDescriptor,
std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
std::vector<rocksdb::ColumnFamilyHandle*,
std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22)
[0x55ad520b62e2]
Apr 18 21:24:37 server2 ceph-osd[17777]: 13:
(RocksDBStore::do_open(std::ostream&, bool,
std::vector<KeyValueDB::ColumnFamily,
std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de]
Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (BlueStore::_open_db(bool,
bool)+0xcf4) [0x55ad51f527a4]
Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (BlueStore::_mount(bool,
bool)+0x4e9) [0x55ad51f828f9]
Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (OSD::init()+0x339)
[0x55ad51b1cf09]
Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (main()+0x23d2)
[0x55ad51a00d52]
Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (__libc_start_main()+0xf5)
[0x7f43bc6c7505]
Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (()+0x378c10) [0x55ad51ad8c10]
Apr 18 21:24:37 server2 ceph-osd[17777]: *** Caught signal (Aborted) **
Apr 18 21:24:37 server2 ceph-osd[17777]: in thread 7f43c936bb80
thread_name:ceph-osd
Apr 18 21:24:37 server2 ceph-osd[17777]: 2020-04-18 21:24:37.094
7f43c936bb80 -1
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc:
In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t,
uint64_t)' thread 7f43c936bb80 time 2020-04-18 21:24:37.091289
Apr 18 21:24:37 server2 ceph-osd[17777]:
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc:
1704: FAILED assert(0 == "bluefs enospc")
Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8
(5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable)
Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (ceph::__ceph_assert_fail(char
const*, char const*, int, char const*)+0x14b) [0x7f43c074987b]
Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (()+0x26fa07) [0x7f43c0749a07]
Apr 18 21:24:37 server2 ceph-osd[17777]: 3:
(BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned
long)+0x1ac6) [0x55ad52029266]
Apr 18 21:24:37 server2 ceph-osd[17777]: 4:
(BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd]
Apr 18 21:24:37 server2 ceph-osd[17777]: 5:
(rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916]
Apr 18 21:24:37 server2 ceph-osd[17777]: 6:
(rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde]
Apr 18 21:24:37 server2 ceph-osd[17777]: 7:
(rocksdb::BuildTable(std::string const&, rocksdb::Env*,
rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&,
rocksdb::EnvOptions const&, rocksdb::TableCache*,
rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator,
std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*,
rocksdb::InternalKeyComparator const&,
std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
std::default_delete<rocksdb::IntTblPropCollectorFactory> >,
std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*,
unsigned int, std::string const&, std::vector<unsigned long,
std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*,
rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool,
rocksdb::InternalStats*, rocksdb::TableFileCreationReason,
rocksdb::EventLogger*, int, rocksdb::Env::IOPriority,
rocksdb::TableProperties*, int, unsigned long, unsigned long,
rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8]
Apr 18 21:24:37 server2 ceph-osd[17777]: 8:
(rocksdb::DBImpl::WriteLevel0TableForRecovery(int,
rocksdb::ColumnFamilyData*, rocksdb::MemTable*,
rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76]
Apr 18 21:24:37 server2 ceph-osd[17777]: 9:
(rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long,
std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b)
[0x55ad520b2dcb]
Apr 18 21:24:37 server2 ceph-osd[17777]: 10:
(rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor,
std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool,
bool)+0xa59) [0x55ad520b3d09]
Apr 18 21:24:37 server2 ceph-osd[17777]: 11:
(rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&,
std::vector<rocksdb::ColumnFamilyDescriptor,
std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
std::vector<rocksdb::ColumnFamilyHandle*,
std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**,
bool)+0x689) [0x55ad520b4ab9]
Apr 18 21:24:37 server2 ceph-osd[17777]: 12:
(rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&,
std::vector<rocksdb::ColumnFamilyDescriptor,
std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
std::vector<rocksdb::ColumnFamilyHandle*,
std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22)
[0x55ad520b62e2]
Apr 18 21:24:37 server2 ceph-osd[17777]: 13:
(RocksDBStore::do_open(std::ostream&, bool,
std::vector<KeyValueDB::ColumnFamily,
std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de]
Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (BlueStore::_open_db(bool,
bool)+0xcf4) [0x55ad51f527a4]
Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (BlueStore::_mount(bool,
bool)+0x4e9) [0x55ad51f828f9]
Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (OSD::init()+0x339)
[0x55ad51b1cf09]
Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (main()+0x23d2)
[0x55ad51a00d52]
Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (__libc_start_main()+0xf5)
[0x7f43bc6c7505]
Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (()+0x378c10) [0x55ad51ad8c10]
Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8
(5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable)
Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (()+0xf5f0) [0x7f43bd6bb5f0]
Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (gsignal()+0x37)
[0x7f43bc6db337]
Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (abort()+0x148) [0x7f43bc6dca28]
Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (ceph::__ceph_assert_fail(char
const*, char const*, int, char const*)+0x248) [0x7f43c0749978]
Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (()+0x26fa07) [0x7f43c0749a07]
Apr 18 21:24:37 server2 ceph-osd[17777]: 6:
(BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned
long)+0x1ac6) [0x55ad52029266]
Apr 18 21:24:37 server2 ceph-osd[17777]: 7:
(BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd]
Apr 18 21:24:37 server2 ceph-osd[17777]: 8:
(rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916]
Apr 18 21:24:37 server2 ceph-osd[17777]: 9:
(rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde]
Apr 18 21:24:37 server2 ceph-osd[17777]: 10:
(rocksdb::BuildTable(std::string const&, rocksdb::Env*,
rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&,
rocksdb::EnvOptions const&, rocksdb::TableCache*,
rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator,
std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*,
rocksdb::InternalKeyComparator const&,
std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
std::default_delete<rocksdb::IntTblPropCollectorFactory> >,
std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*,
unsigned int, std::string const&, std::vector<unsigned long,
std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*,
rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool,
rocksdb::InternalStats*, rocksdb::TableFileCreationReason,
rocksdb::EventLogger*, int, rocksdb::Env::IOPriority,
rocksdb::TableProperties*, int, unsigned long, unsigned long,
rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8]
Apr 18 21:24:37 server2 ceph-osd[17777]: 11:
(rocksdb::DBImpl::WriteLevel0TableForRecovery(int,
rocksdb::ColumnFamilyData*, rocksdb::MemTable*,
rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76]
Apr 18 21:24:37 server2 ceph-osd[17777]: 12:
(rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long,
std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b)
[0x55ad520b2dcb]
Apr 18 21:24:37 server2 ceph-osd[17777]: 13:
(rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor,
std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool,
bool)+0xa59) [0x55ad520b3d09]
Apr 18 21:24:37 server2 ceph-osd[17777]: 14:
(rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&,
std::vector<rocksdb::ColumnFamilyDescriptor,
std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
std::vector<rocksdb::ColumnFamilyHandle*,
std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**,
bool)+0x689) [0x55ad520b4ab9]
Apr 18 21:24:37 server2 ceph-osd[17777]: 15:
(rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&,
std::vector<rocksdb::ColumnFamilyDescriptor,
std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
std::vector<rocksdb::ColumnFamilyHandle*,
std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22)
[0x55ad520b62e2]
Apr 18 21:24:37 server2 ceph-osd[17777]: 16:
(RocksDBStore::do_open(std::ostream&, bool,
std::vector<KeyValueDB::ColumnFamily,
std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de]
Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (BlueStore::_open_db(bool,
bool)+0xcf4) [0x55ad51f527a4]
Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (BlueStore::_mount(bool,
bool)+0x4e9) [0x55ad51f828f9]
Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (OSD::init()+0x339)
[0x55ad51b1cf09]
Apr 18 21:24:37 server2 ceph-osd[17777]: 20: (main()+0x23d2)
[0x55ad51a00d52]
Apr 18 21:24:37 server2 ceph-osd[17777]: 21: (__libc_start_main()+0xf5)
[0x7f43bc6c7505]
Apr 18 21:24:37 server2 ceph-osd[17777]: 22: (()+0x378c10) [0x55ad51ad8c10]
Apr 18 21:24:37 server2 ceph-osd[17777]: 2020-04-18 21:24:37.099
7f43c936bb80 -1 *** Caught signal (Aborted) **
Apr 18 21:24:37 server2 ceph-osd[17777]: in thread 7f43c936bb80
thread_name:ceph-osd
Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8
(5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable)
Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (()+0xf5f0) [0x7f43bd6bb5f0]
Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (gsignal()+0x37)
[0x7f43bc6db337]
Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (abort()+0x148) [0x7f43bc6dca28]
Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (ceph::__ceph_assert_fail(char
const*, char const*, int, char const*)+0x248) [0x7f43c0749978]
Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (()+0x26fa07) [0x7f43c0749a07]
Apr 18 21:24:37 server2 ceph-osd[17777]: 6:
(BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned
long)+0x1ac6) [0x55ad52029266]
Apr 18 21:24:37 server2 ceph-osd[17777]: 7:
(BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd]
Apr 18 21:24:37 server2 ceph-osd[17777]: 8:
(rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916]
Apr 18 21:24:37 server2 ceph-osd[17777]: 9:
(rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde]
Apr 18 21:24:37 server2 ceph-osd[17777]: 10:
(rocksdb::BuildTable(std::string const&, rocksdb::Env*,
rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&,
rocksdb::EnvOptions const&, rocksdb::TableCache*,
rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator,
std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*,
rocksdb::InternalKeyComparator const&,
std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
std::default_delete<rocksdb::IntTblPropCollectorFactory> >,
std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*,
unsigned int, std::string const&, std::vector<unsigned long,
std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*,
rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool,
rocksdb::InternalStats*, rocksdb::TableFileCreationReason,
rocksdb::EventLogger*, int, rocksdb::Env::IOPriority,
rocksdb::TableProperties*, int, unsigned long, unsigned long,
rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8]
Apr 18 21:24:37 server2 ceph-osd[17777]: 11:
(rocksdb::DBImpl::WriteLevel0TableForRecovery(int,
rocksdb::ColumnFamilyData*, rocksdb::MemTable*,
rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76]
Apr 18 21:24:37 server2 ceph-osd[17777]: 12:
(rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long,
std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b)
[0x55ad520b2dcb]
Apr 18 21:24:37 server2 ceph-osd[17777]: 13:
(rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor,
std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool,
bool)+0xa59) [0x55ad520b3d09]
Apr 18 21:24:37 server2 ceph-osd[17777]: 14:
(rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&,
std::vector<rocksdb::ColumnFamilyDescriptor,
std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
std::vector<rocksdb::ColumnFamilyHandle*,
std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**,
bool)+0x689) [0x55ad520b4ab9]
Apr 18 21:24:37 server2 ceph-osd[17777]: 15:
(rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&,
std::vector<rocksdb::ColumnFamilyDescriptor,
std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
std::vector<rocksdb::ColumnFamilyHandle*,
std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22)
[0x55ad520b62e2]
Apr 18 21:24:37 server2 ceph-osd[17777]: 16:
(RocksDBStore::do_open(std::ostream&, bool,
std::vector<KeyValueDB::ColumnFamily,
std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de]
Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (BlueStore::_open_db(bool,
bool)+0xcf4) [0x55ad51f527a4]
Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (BlueStore::_mount(bool,
bool)+0x4e9) [0x55ad51f828f9]
Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (OSD::init()+0x339)
[0x55ad51b1cf09]
Apr 18 21:24:37 server2 ceph-osd[17777]: 20: (main()+0x23d2)
[0x55ad51a00d52]
Apr 18 21:24:37 server2 ceph-osd[17777]: 21: (__libc_start_main()+0xf5)
[0x7f43bc6c7505]
Apr 18 21:24:37 server2 ceph-osd[17777]: 22: (()+0x378c10) [0x55ad51ad8c10]
Apr 18 21:24:37 server2 ceph-osd[17777]: NOTE: a copy of the executable, or
`objdump -rdS <executable>` is needed to interpret this.
Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.090
7f43c936bb80 -1 bluefs _allocate failed to allocate 0x on bdev 2, dne
Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.090
7f43c936bb80 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length:
0x794de9
Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.094
7f43c936bb80 -1
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc:
In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t,
uint64_t)' thread 7f43c936bb80 time 2020-04-18 21:24:37.091289
Apr 18 21:24:37 server2 ceph-osd[17777]:
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc:
1704: FAILED assert(0 == "bluefs enospc")
Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8
(5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable)
Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (ceph::__ceph_assert_fail(char
const*, char const*, int, char const*)+0x14b) [0x7f43c074987b]
Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (()+0x26fa07) [0x7f43c0749a07]
Apr 18 21:24:37 server2 ceph-osd[17777]: 3:
(BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned
long)+0x1ac6) [0x55ad52029266]
Apr 18 21:24:37 server2 ceph-osd[17777]: 4:
(BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd]
Apr 18 21:24:37 server2 ceph-osd[17777]: 5:
(rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916]
Apr 18 21:24:37 server2 ceph-osd[17777]: 6:
(rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde]
Apr 18 21:24:37 server2 ceph-osd[17777]: 7:
(rocksdb::BuildTable(std::string const&, rocksdb::Env*,
rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&,
rocksdb::EnvOptions const&, rocksdb::TableCache*,
rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator,
std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*,
rocksdb::InternalKeyComparator const&,
std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
std::default_delete<rocksdb::IntTblPropCollectorFactory> >,
std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*,
unsigned int, std::string const&, std::vector<unsigned long,
std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*,
rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool,
rocksdb::InternalStats*, rocksdb::TableFileCreationReason,
rocksdb::EventLogger*, int, rocksdb::Env::IOPriority,
rocksdb::TableProperties*, int, unsigned long, unsigned long,
rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8]
Apr 18 21:24:37 server2 ceph-osd[17777]: 8:
(rocksdb::DBImpl::WriteLevel0TableForRecovery(int,
rocksdb::ColumnFamilyData*, rocksdb::MemTable*,
rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76]
Apr 18 21:24:37 server2 ceph-osd[17777]: 9:
(rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long,
std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b)
[0x55ad520b2dcb]
Apr 18 21:24:37 server2 ceph-osd[17777]: 10:
(rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor,
std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool,
bool)+0xa59) [0x55ad520b3d09]
Apr 18 21:24:37 server2 ceph-osd[17777]: 11:
(rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&,
std::vector<rocksdb::ColumnFamilyDescriptor,
std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
std::vector<rocksdb::ColumnFamilyHandle*,
std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**,
bool)+0x689) [0x55ad520b4ab9]
Apr 18 21:24:37 server2 ceph-osd[17777]: 12:
(rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&,
std::vector<rocksdb::ColumnFamilyDescriptor,
std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
std::vector<rocksdb::ColumnFamilyHandle*,
std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22)
[0x55ad520b62e2]
Apr 18 21:24:37 server2 ceph-osd[17777]: 13:
(RocksDBStore::do_open(std::ostream&, bool,
std::vector<KeyValueDB::ColumnFamily,
std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de]
Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (BlueStore::_open_db(bool,
bool)+0xcf4) [0x55ad51f527a4]
Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (BlueStore::_mount(bool,
bool)+0x4e9) [0x55ad51f828f9]
Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (OSD::init()+0x339)
[0x55ad51b1cf09]
Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (main()+0x23d2)
[0x55ad51a00d52]
Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (__libc_start_main()+0xf5)
[0x7f43bc6c7505]
Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (()+0x378c10) [0x55ad51ad8c10]
Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.099
7f43c936bb80 -1 *** Caught signal (Aborted) **
Apr 18 21:24:37 server2 ceph-osd[17777]: in thread 7f43c936bb80
thread_name:ceph-osd
Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8
(5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable)
Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (()+0xf5f0) [0x7f43bd6bb5f0]
Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (gsignal()+0x37)
[0x7f43bc6db337]
Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (abort()+0x148) [0x7f43bc6dca28]
Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (ceph::__ceph_assert_fail(char
const*, char const*, int, char const*)+0x248) [0x7f43c0749978]
Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (()+0x26fa07) [0x7f43c0749a07]
Apr 18 21:24:37 server2 ceph-osd[17777]: 6:
(BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned
long)+0x1ac6) [0x55ad52029266]
Apr 18 21:24:37 server2 ceph-osd[17777]: 7:
(BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd]
Apr 18 21:24:37 server2 ceph-osd[17777]: 8:
(rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916]
Apr 18 21:24:37 server2 ceph-osd[17777]: 9:
(rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde]
Apr 18 21:24:37 server2 ceph-osd[17777]: 10:
(rocksdb::BuildTable(std::string const&, rocksdb::Env*,
rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&,
rocksdb::EnvOptions const&, rocksdb::TableCache*,
rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator,
std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*,
rocksdb::InternalKeyComparator const&,
std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
std::default_delete<rocksdb::IntTblPropCollectorFactory> >,
std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*,
unsigned int, std::string const&, std::vector<unsigned long,
std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*,
rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool,
rocksdb::InternalStats*, rocksdb::TableFileCreationReason,
rocksdb::EventLogger*, int, rocksdb::Env::IOPriority,
rocksdb::TableProperties*, int, unsigned long, unsigned long,
rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8]
Apr 18 21:24:37 server2 ceph-osd[17777]: 11:
(rocksdb::DBImpl::WriteLevel0TableForRecovery(int,
rocksdb::ColumnFamilyData*, rocksdb::MemTable*,
rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76]
Apr 18 21:24:37 server2 ceph-osd[17777]: 12:
(rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long,
std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b)
[0x55ad520b2dcb]
Apr 18 21:24:37 server2 ceph-osd[17777]: 13:
(rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor,
std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool,
bool)+0xa59) [0x55ad520b3d09]
Apr 18 21:24:37 server2 ceph-osd[17777]: 14:
(rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&,
std::vector<rocksdb::ColumnFamilyDescriptor,
std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
std::vector<rocksdb::ColumnFamilyHandle*,
std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**,
bool)+0x689) [0x55ad520b4ab9]
Apr 18 21:24:37 server2 ceph-osd[17777]: 15:
(rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&,
std::vector<rocksdb::ColumnFamilyDescriptor,
std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
std::vector<rocksdb::ColumnFamilyHandle*,
std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22)
[0x55ad520b62e2]
Apr 18 21:24:37 server2 ceph-osd[17777]: 16:
(RocksDBStore::do_open(std::ostream&, bool,
std::vector<KeyValueDB::ColumnFamily,
std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de]
Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (BlueStore::_open_db(bool,
bool)+0xcf4) [0x55ad51f527a4]
Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (BlueStore::_mount(bool,
bool)+0x4e9) [0x55ad51f828f9]
Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (OSD::init()+0x339)
[0x55ad51b1cf09]
Apr 18 21:24:37 server2 ceph-osd[17777]: 20: (main()+0x23d2)
[0x55ad51a00d52]
Apr 18 21:24:37 server2 ceph-osd[17777]: 21: (__libc_start_main()+0xf5)
[0x7f43bc6c7505]
Apr 18 21:24:37 server2 ceph-osd[17777]: 22: (()+0x378c10) [0x55ad51ad8c10]
Apr 18 21:24:37 server2 ceph-osd[17777]: NOTE: a copy of the executable, or
`objdump -rdS <executable>` is needed to interpret this.
Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.090
7f43c936bb80 -1 bluefs _allocate failed to allocate 0x on bdev 2, dne
Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.090
7f43c936bb80 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length:
0x794de9
Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.094
7f43c936bb80 -1
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc:
In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t,
uint64_t)' thread 7f43c936bb80 time 2020-04-18 21:24:37.091289
Apr 18 21:24:37 server2 ceph-osd[17777]:
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc:
1704: FAILED assert(0 == "bluefs enospc")
Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8
(5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable)
Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (ceph::__ceph_assert_fail(char
const*, char const*, int, char const*)+0x14b) [0x7f43c074987b]
Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (()+0x26fa07) [0x7f43c0749a07]
Apr 18 21:24:37 server2 ceph-osd[17777]: 3:
(BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned
long)+0x1ac6) [0x55ad52029266]
Apr 18 21:24:37 server2 ceph-osd[17777]: 4:
(BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd]
Apr 18 21:24:37 server2 ceph-osd[17777]: 5:
(rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916]
Apr 18 21:24:37 server2 ceph-osd[17777]: 6:
(rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde]
Apr 18 21:24:37 server2 ceph-osd[17777]: 7:
(rocksdb::BuildTable(std::string const&, rocksdb::Env*,
rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&,
rocksdb::EnvOptions const&, rocksdb::TableCache*,
rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator,
std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*,
rocksdb::InternalKeyComparator const&,
std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
std::default_delete<rocksdb::IntTblPropCollectorFactory> >,
std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*,
unsigned int, std::string const&, std::vector<unsigned long,
std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*,
rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool,
rocksdb::InternalStats*, rocksdb::TableFileCreationReason,
rocksdb::EventLogger*, int, rocksdb::Env::IOPriority,
rocksdb::TableProperties*, int, unsigned long, unsigned long,
rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8]
Apr 18 21:24:37 server2 ceph-osd[17777]: 8:
(rocksdb::DBImpl::WriteLevel0TableForRecovery(int,
rocksdb::ColumnFamilyData*, rocksdb::MemTable*,
rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76]
Apr 18 21:24:37 server2 ceph-osd[17777]: 9:
(rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long,
std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b)
[0x55ad520b2dcb]
Apr 18 21:24:37 server2 ceph-osd[17777]: 10:
(rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor,
std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool,
bool)+0xa59) [0x55ad520b3d09]
Apr 18 21:24:37 server2 ceph-osd[17777]: 11:
(rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&,
std::vector<rocksdb::ColumnFamilyDescriptor,
std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
std::vector<rocksdb::ColumnFamilyHandle*,
std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**,
bool)+0x689) [0x55ad520b4ab9]
Apr 18 21:24:37 server2 ceph-osd[17777]: 12:
(rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&,
std::vector<rocksdb::ColumnFamilyDescriptor,
std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
std::vector<rocksdb::ColumnFamilyHandle*,
std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22)
[0x55ad520b62e2]
Apr 18 21:24:37 server2 ceph-osd[17777]: 13:
(RocksDBStore::do_open(std::ostream&, bool,
std::vector<KeyValueDB::ColumnFamily,
std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de]
Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (BlueStore::_open_db(bool,
bool)+0xcf4) [0x55ad51f527a4]
Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (BlueStore::_mount(bool,
bool)+0x4e9) [0x55ad51f828f9]
Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (OSD::init()+0x339)
[0x55ad51b1cf09]
Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (main()+0x23d2)
[0x55ad51a00d52]
Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (__libc_start_main()+0xf5)
[0x7f43bc6c7505]
Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (()+0x378c10) [0x55ad51ad8c10]
Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.099
7f43c936bb80 -1 *** Caught signal (Aborted) **
Apr 18 21:24:37 server2 ceph-osd[17777]: in thread 7f43c936bb80
thread_name:ceph-osd
Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8
(5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable)
Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (()+0xf5f0) [0x7f43bd6bb5f0]
Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (gsignal()+0x37)
[0x7f43bc6db337]
Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (abort()+0x148) [0x7f43bc6dca28]
Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (ceph::__ceph_assert_fail(char
const*, char const*, int, char const*)+0x248) [0x7f43c0749978]
Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (()+0x26fa07) [0x7f43c0749a07]
Apr 18 21:24:37 server2 ceph-osd[17777]: 6:
(BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned
long)+0x1ac6) [0x55ad52029266]
Apr 18 21:24:37 server2 ceph-osd[17777]: 7:
(BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd]
Apr 18 21:24:37 server2 ceph-osd[17777]: 8:
(rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916]
Apr 18 21:24:37 server2 ceph-osd[17777]: 9:
(rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde]
Apr 18 21:24:37 server2 ceph-osd[17777]: 10:
(rocksdb::BuildTable(std::string const&, rocksdb::Env*,
rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&,
rocksdb::EnvOptions const&, rocksdb::TableCache*,
rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator,
std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*,
rocksdb::InternalKeyComparator const&,
std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
std::default_delete<rocksdb::IntTblPropCollectorFactory> >,
std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*,
unsigned int, std::string const&, std::vector<unsigned long,
std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*,
rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool,
rocksdb::InternalStats*, rocksdb::TableFileCreationReason,
rocksdb::EventLogger*, int, rocksdb::Env::IOPriority,
rocksdb::TableProperties*, int, unsigned long, unsigned long,
rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8]
Apr 18 21:24:37 server2 ceph-osd[17777]: 11:
(rocksdb::DBImpl::WriteLevel0TableForRecovery(int,
rocksdb::ColumnFamilyData*, rocksdb::MemTable*,
rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76]
Apr 18 21:24:37 server2 ceph-osd[17777]: 12:
(rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long,
std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b)
[0x55ad520b2dcb]
Apr 18 21:24:37 server2 ceph-osd[17777]: 13:
(rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor,
std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool,
bool)+0xa59) [0x55ad520b3d09]
Apr 18 21:24:37 server2 ceph-osd[17777]: 14:
(rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&,
std::vector<rocksdb::ColumnFamilyDescriptor,
std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
std::vector<rocksdb::ColumnFamilyHandle*,
std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**,
bool)+0x689) [0x55ad520b4ab9]
Apr 18 21:24:37 server2 ceph-osd[17777]: 15:
(rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&,
std::vector<rocksdb::ColumnFamilyDescriptor,
std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
std::vector<rocksdb::ColumnFamilyHandle*,
std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22)
[0x55ad520b62e2]
Apr 18 21:24:37 server2 ceph-osd[17777]: 16:
(RocksDBStore::do_open(std::ostream&, bool,
std::vector<KeyValueDB::ColumnFamily,
std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de]
Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (BlueStore::_open_db(bool,
bool)+0xcf4) [0x55ad51f527a4]
Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (BlueStore::_mount(bool,
bool)+0x4e9) [0x55ad51f828f9]
Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (OSD::init()+0x339)
[0x55ad51b1cf09]
Apr 18 21:24:37 server2 ceph-osd[17777]: 20: (main()+0x23d2)
[0x55ad51a00d52]
Apr 18 21:24:37 server2 ceph-osd[17777]: 21: (__libc_start_main()+0xf5)
[0x7f43bc6c7505]
Apr 18 21:24:37 server2 ceph-osd[17777]: 22: (()+0x378c10) [0x55ad51ad8c10]
Apr 18 21:24:37 server2 ceph-osd[17777]: NOTE: a copy of the executable, or
`objdump -rdS <executable>` is needed to interpret this.
Apr 18 21:24:37 server2 systemd[1]: ceph-osd(a)4.service: main process
exited, code=killed, status=6/ABRT
Apr 18 21:24:37 server2 systemd[1]: Unit ceph-osd(a)4.service entered failed
state.
Apr 18 21:24:37 server2 systemd[1]: ceph-osd(a)4.service failed.
Apr 18 21:24:57 server2 systemd[1]: ceph-osd(a)4.service holdoff time over,
scheduling restart.
Apr 18 21:24:57 server2 systemd[1]: Stopped Ceph object storage daemon
osd.4.
Apr 18 21:24:57 server2 systemd[1]: Starting Ceph object storage daemon
osd.4...
Apr 18 21:24:57 server2 systemd[1]: Started Ceph object storage daemon
osd.4.

and some more data:

[root@server1 ~]# ceph df detail
GLOBAL:
    SIZE       AVAIL      RAW USED     %RAW USED     OBJECTS
    38 GiB     35 GiB      2.6 GiB          6.83        303
POOLS:
    NAME                          ID     QUOTA OBJECTS     QUOTA BYTES
USED        %USED     MAX AVAIL     OBJECTS     DIRTY     READ        WRITE
      RAW USED
    .rgw.root                     18     N/A               N/A
1.1 KiB         0        50 GiB           4        4         63 B         4
B      1.1 KiB
    default.rgw.meta              19     N/A               N/A
4.1 KiB         0        50 GiB          22       22      1.5 KiB       182
B      9.0 KiB
    default.rgw.log               20     N/A               N/A
    0 B         0        50 GiB         107      107      307 KiB     205
KiB          0 B
    default.rgw.control           21     N/A               N/A
    0 B         0        50 GiB           5        5          0 B         0
B          0 B
    default.rgw.buckets.index     22     N/A               N/A
    0 B         0        50 GiB           5        5        501 B        60
B          0 B
    default.rgw.buckets.data      23     N/A               N/A
377 MiB      0.39        50 GiB         160      160        385 B     1.1
KiB      603 MiB
[root@server1 ~]#

[root@server1 ~]# ceph osd df tree
ID CLASS WEIGHT  REWEIGHT SIZE   USE     DATA    OMAP  META     AVAIL  %USE
VAR  PGS TYPE NAME
-1       0.16727        -    0 B     0 B     0 B   0 B      0 B    0 B    0
   0   - root default
-3       0.05576        -    0 B     0 B     0 B   0 B      0 B    0 B    0
   0   -     host server1
 0   hdd 0.01859  1.00000    0 B     0 B     0 B   0 B      0 B    0 B    0
   0   0         osd.0
 1   hdd 0.01859        0    0 B     0 B     0 B   0 B      0 B    0 B    0
   0   0         osd.1
 2   hdd 0.01859        0    0 B     0 B     0 B   0 B      0 B    0 B    0
   0   0         osd.2
-5       0.05576        - 19 GiB 1.4 GiB 360 MiB 3 KiB 1024 MiB 18 GiB    0
   0   -     host server2
 3   hdd 0.01859  1.00000    0 B     0 B     0 B   0 B      0 B    0 B    0
   0   0         osd.3
 4   hdd 0.01859        0    0 B     0 B     0 B   0 B      0 B    0 B    0
   0   0         osd.4
 5   hdd 0.01859  1.00000 19 GiB 1.4 GiB 360 MiB 3 KiB 1024 MiB 18 GiB 7.11
1.04  99         osd.5
-7       0.05576        -    0 B     0 B     0 B   0 B      0 B    0 B    0
   0   -     host server3
 6   hdd 0.01859  1.00000 19 GiB 1.2 GiB 249 MiB 3 KiB 1024 MiB 18 GiB 6.55
0.96  78         osd.6
 7   hdd 0.01859  1.00000    0 B     0 B     0 B   0 B      0 B    0 B    0
   0   0         osd.7
 8   hdd 0.01859  1.00000    0 B     0 B     0 B   0 B      0 B    0 B    0
   0   0         osd.8
                    TOTAL 38 GiB 2.6 GiB 610 MiB 6 KiB  2.0 GiB 35 GiB 6.83

MIN/MAX VAR: 0/1.04  STDDEV: 5.58
[root@server1 ~]#

I'm kind of newbie to ceph, so any help or hint would be appreciated.
Did I hit a bug or something is wrong with my configuration?

Thanks a lot,
Khodayar

2024

2023

2022

2021

2020

2019

[ceph-users] OSDs get full with bluestore logs