Indeed this assertion "FAILED assert(0 == "bluefs enospc")" indicates
lack of free space for RocksDB at both main and DB volumes.
OSD (RocksDB specifically) attempts to recover (and hence flush) some
data on OSD restart and is unable to allocate space for that. Hence it
crashes...
What volumes have you configured for BlueStore?
What's the output for "ceph-bluestore-tool --command bluefs-bdev-sizes
--path <actual-path-to-osd!!!>" command?
Thanks,
Igor
On 4/18/2020 7:56 PM, Khodayar Doustar wrote:
> Hi,
>
> I have a 3 node cluster of mimic with 9 osds (3 osds on each node).
> I use this cluster to test integration of an application with S3 api.
>
> The problem is that after a few days all OSD starts filling up with
> bluestore logs and goes down and out one by one!
> I cannot stop the logs and I cannot find the setting to fix this leakage,
> this should be a leakage in logs because it's not logical to fill up all
> OSD with bluefs logs.
>
> This is an example of logs which is being repeated in bluestore logs:
>
> [root@server2 ~]# ceph-bluestore-tool --command bluefs-log-dump --path
> /var/lib/ceph/osd/ceph-5
> .
> .
> . 0x40d000: op_file_update file(ino 30 size 0x7a713 mtime 2020-04-18
> 15:54:29.056488 bdev 1 allocated 80000 extents
>
[1:0x130000+10000,1:0x100000+10000,1:0x140000+10000,1:0x150000+10000,1:0x160000+10000,1:0x170000+10000,1:0x180000+10000,1:0x190000+10000])
> 0x40e000: txn(seq 1156100 len 0x78 crc 0x3e1c626f)
> 0x40e000: op_file_update file(ino 30 size 0x7a72a mtime 2020-04-18
> 15:54:30.057828 bdev 1 allocated 80000 extents
>
[1:0x130000+10000,1:0x100000+10000,1:0x140000+10000,1:0x150000+10000,1:0x160000+10000,1:0x170000+10000,1:0x180000+10000,1:0x190000+10000])
> 0x40f000: txn(seq 1156101 len 0x78 crc 0xc1f9ec5f)
> 0x40f000: op_file_update file(ino 30 size 0x7a741 mtime 2020-04-18
> 15:54:31.059252 bdev 1 allocated 80000 extents
>
[1:0x130000+10000,1:0x100000+10000,1:0x140000+10000,1:0x150000+10000,1:0x160000+10000,1:0x170000+10000,1:0x180000+10000,1:0x190000+10000])
> *** Caught signal (Segmentation fault) **
> in thread 7f4108cfe600 thread_name:ceph-bluestore-
> ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic
> (stable)
> 1: (()+0xf5f0) [0x7f40fd47f5f0]
> 2: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned
> long, unsigned long, ceph::buffer::list*, char*)+0x3bf) [0x55603a1ebb3f]
> 3: (BlueFS::_replay(bool, bool)+0x214) [0x55603a1f6654]
> 4: (BlueFS::log_dump(CephContext*, std::string const&,
> std::vector<std::string, std::allocator<std::string> > const&)+0x3b)
> [0x55603a1fad3b]
> 5: (log_dump(CephContext*, std::string const&, std::vector<std::string,
> std::allocator<std::string> > const&)+0x64) [0x55603a1da764]
> 6: (main()+0x2f3e) [0x55603a105e3e]
> 7: (__libc_start_main()+0xf5) [0x7f40fbe51505]
> 8: (()+0x23115f) [0x55603a1d915f]
> 2020-04-18 21:08:25.410 7f4108cfe600 -1 *** Caught signal (Segmentation
> fault) **
> in thread 7f4108cfe600 thread_name:ceph-bluestore-
> .
> .
> .
>
>
> And this is the output of daemonperf for one of remaining (filling up!)
> OSDs:
>
> [root@server2 ~]# ceph daemonperf osd.5
> ------bluefs------- ------------bluestore-------------
> ----------osd-----------
> jlen j wal sst |fl_l k_l io_l th_l s_l c_l r_l |ops wr rd l
> rop |
> 81M 0 0 0 | 0 0 0 0 0 0 0 | 0 0 0 0
> 0
> 81M 4.0k 1.8k 0 | 0 0 0 0 0 0 0 | 0 0 0 0
> 0
> 81M 4.0k 1.9k 0 | 0 0 0 0 0 0 0 | 0 0 0 0
> 0
> 81M 4.0k 1.9k 0 | 0 0 0 0 0 0 0 | 0 0 0 0
> 0
> 81M 4.0k 1.9k 0 | 0 0 0 0 0 0 0 | 0 0 0 0
> 0
> ^C[root@server2 ~]#
>
> And this is the enospc logs when trying to start OSD:
>
> [root@server2 ~]# journalctl -u -f ceph-osd(a)4.service
> Failed to add match 'ceph-osd(a)4.service'ice': Invalid argument
> Failed to add filters: Invalid argument
> [root@server2 ~]# journalctl -f -u ceph-osd(a)4.service
> -- Logs begin at Sat 2020-04-18 15:38:15 +0430. --
> Apr 18 21:18:22 server2 ceph-osd[17485]: 22: (()+0x378c10) [0x557348ebac10]
> Apr 18 21:18:22 server2 ceph-osd[17485]: NOTE: a copy of the executable, or
> `objdump -rdS <executable>` is needed to interpret this.
> Apr 18 21:18:22 server2 systemd[1]: ceph-osd(a)4.service: main process
> exited, code=killed, status=6/ABRT
> Apr 18 21:18:22 server2 systemd[1]: Unit ceph-osd(a)4.service entered failed
> state.
> Apr 18 21:18:22 server2 systemd[1]: ceph-osd(a)4.service failed.
> Apr 18 21:18:42 server2 systemd[1]: ceph-osd(a)4.service holdoff time over,
> scheduling restart.
> Apr 18 21:18:42 server2 systemd[1]: Stopped Ceph object storage daemon
> osd.4.
> Apr 18 21:18:42 server2 systemd[1]: Starting Ceph object storage daemon
> osd.4...
> Apr 18 21:18:42 server2 systemd[1]: Started Ceph object storage daemon
> osd.4.
> Apr 18 21:18:43 server2 ceph-osd[17777]: starting osd.4 at - osd_data
> /var/lib/ceph/osd/ceph-4 /var/lib/ceph/osd/ceph-4/journal
> Apr 18 21:24:37 server2 ceph-osd[17777]:
>
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc:
> In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t,
> uint64_t)' thread 7f43c936bb80 time 2020-04-18 21:24:37.091289
> Apr 18 21:24:37 server2 ceph-osd[17777]:
>
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc:
> 1704: FAILED assert(0 == "bluefs enospc")
> Apr 18 21:24:37 server2 ceph-osd[17777]: 2020-04-18 21:24:37.090
> 7f43c936bb80 -1 bluefs _allocate failed to allocate 0x on bdev 2, dne
> Apr 18 21:24:37 server2 ceph-osd[17777]: 2020-04-18 21:24:37.090
> 7f43c936bb80 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length:
> 0x794de9
> Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8
> (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable)
> Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (ceph::__ceph_assert_fail(char
> const*, char const*, int, char const*)+0x14b) [0x7f43c074987b]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (()+0x26fa07) [0x7f43c0749a07]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 3:
> (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned
> long)+0x1ac6) [0x55ad52029266]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 4:
> (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 5:
> (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 6:
> (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 7:
> (rocksdb::BuildTable(std::string const&, rocksdb::Env*,
> rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&,
> rocksdb::EnvOptions const&, rocksdb::TableCache*,
> rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator,
> std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*,
> rocksdb::InternalKeyComparator const&,
> std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
> std::default_delete<rocksdb::IntTblPropCollectorFactory> >,
> std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
> std::default_delete<rocksdb::IntTblPropCollectorFactory> > > >
const*,
> unsigned int, std::string const&, std::vector<unsigned long,
> std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*,
> rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool,
> rocksdb::InternalStats*, rocksdb::TableFileCreationReason,
> rocksdb::EventLogger*, int, rocksdb::Env::IOPriority,
> rocksdb::TableProperties*, int, unsigned long, unsigned long,
> rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 8:
> (rocksdb::DBImpl::WriteLevel0TableForRecovery(int,
> rocksdb::ColumnFamilyData*, rocksdb::MemTable*,
> rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 9:
> (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long,
> std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b)
> [0x55ad520b2dcb]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 10:
> (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor,
> std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool,
> bool)+0xa59) [0x55ad520b3d09]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 11:
> (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&,
> std::vector<rocksdb::ColumnFamilyDescriptor,
> std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
> std::vector<rocksdb::ColumnFamilyHandle*,
> std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**,
> bool)+0x689) [0x55ad520b4ab9]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 12:
> (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&,
> std::vector<rocksdb::ColumnFamilyDescriptor,
> std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
> std::vector<rocksdb::ColumnFamilyHandle*,
> std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22)
> [0x55ad520b62e2]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 13:
> (RocksDBStore::do_open(std::ostream&, bool,
> std::vector<KeyValueDB::ColumnFamily,
> std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (BlueStore::_open_db(bool,
> bool)+0xcf4) [0x55ad51f527a4]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (BlueStore::_mount(bool,
> bool)+0x4e9) [0x55ad51f828f9]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (OSD::init()+0x339)
> [0x55ad51b1cf09]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (main()+0x23d2)
> [0x55ad51a00d52]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (__libc_start_main()+0xf5)
> [0x7f43bc6c7505]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (()+0x378c10) [0x55ad51ad8c10]
> Apr 18 21:24:37 server2 ceph-osd[17777]: *** Caught signal (Aborted) **
> Apr 18 21:24:37 server2 ceph-osd[17777]: in thread 7f43c936bb80
> thread_name:ceph-osd
> Apr 18 21:24:37 server2 ceph-osd[17777]: 2020-04-18 21:24:37.094
> 7f43c936bb80 -1
>
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc:
> In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t,
> uint64_t)' thread 7f43c936bb80 time 2020-04-18 21:24:37.091289
> Apr 18 21:24:37 server2 ceph-osd[17777]:
>
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc:
> 1704: FAILED assert(0 == "bluefs enospc")
> Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8
> (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable)
> Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (ceph::__ceph_assert_fail(char
> const*, char const*, int, char const*)+0x14b) [0x7f43c074987b]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (()+0x26fa07) [0x7f43c0749a07]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 3:
> (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned
> long)+0x1ac6) [0x55ad52029266]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 4:
> (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 5:
> (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 6:
> (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 7:
> (rocksdb::BuildTable(std::string const&, rocksdb::Env*,
> rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&,
> rocksdb::EnvOptions const&, rocksdb::TableCache*,
> rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator,
> std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*,
> rocksdb::InternalKeyComparator const&,
> std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
> std::default_delete<rocksdb::IntTblPropCollectorFactory> >,
> std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
> std::default_delete<rocksdb::IntTblPropCollectorFactory> > > >
const*,
> unsigned int, std::string const&, std::vector<unsigned long,
> std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*,
> rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool,
> rocksdb::InternalStats*, rocksdb::TableFileCreationReason,
> rocksdb::EventLogger*, int, rocksdb::Env::IOPriority,
> rocksdb::TableProperties*, int, unsigned long, unsigned long,
> rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 8:
> (rocksdb::DBImpl::WriteLevel0TableForRecovery(int,
> rocksdb::ColumnFamilyData*, rocksdb::MemTable*,
> rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 9:
> (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long,
> std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b)
> [0x55ad520b2dcb]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 10:
> (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor,
> std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool,
> bool)+0xa59) [0x55ad520b3d09]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 11:
> (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&,
> std::vector<rocksdb::ColumnFamilyDescriptor,
> std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
> std::vector<rocksdb::ColumnFamilyHandle*,
> std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**,
> bool)+0x689) [0x55ad520b4ab9]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 12:
> (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&,
> std::vector<rocksdb::ColumnFamilyDescriptor,
> std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
> std::vector<rocksdb::ColumnFamilyHandle*,
> std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22)
> [0x55ad520b62e2]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 13:
> (RocksDBStore::do_open(std::ostream&, bool,
> std::vector<KeyValueDB::ColumnFamily,
> std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (BlueStore::_open_db(bool,
> bool)+0xcf4) [0x55ad51f527a4]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (BlueStore::_mount(bool,
> bool)+0x4e9) [0x55ad51f828f9]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (OSD::init()+0x339)
> [0x55ad51b1cf09]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (main()+0x23d2)
> [0x55ad51a00d52]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (__libc_start_main()+0xf5)
> [0x7f43bc6c7505]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (()+0x378c10) [0x55ad51ad8c10]
> Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8
> (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable)
> Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (()+0xf5f0) [0x7f43bd6bb5f0]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (gsignal()+0x37)
> [0x7f43bc6db337]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (abort()+0x148) [0x7f43bc6dca28]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (ceph::__ceph_assert_fail(char
> const*, char const*, int, char const*)+0x248) [0x7f43c0749978]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (()+0x26fa07) [0x7f43c0749a07]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 6:
> (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned
> long)+0x1ac6) [0x55ad52029266]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 7:
> (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 8:
> (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 9:
> (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 10:
> (rocksdb::BuildTable(std::string const&, rocksdb::Env*,
> rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&,
> rocksdb::EnvOptions const&, rocksdb::TableCache*,
> rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator,
> std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*,
> rocksdb::InternalKeyComparator const&,
> std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
> std::default_delete<rocksdb::IntTblPropCollectorFactory> >,
> std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
> std::default_delete<rocksdb::IntTblPropCollectorFactory> > > >
const*,
> unsigned int, std::string const&, std::vector<unsigned long,
> std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*,
> rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool,
> rocksdb::InternalStats*, rocksdb::TableFileCreationReason,
> rocksdb::EventLogger*, int, rocksdb::Env::IOPriority,
> rocksdb::TableProperties*, int, unsigned long, unsigned long,
> rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 11:
> (rocksdb::DBImpl::WriteLevel0TableForRecovery(int,
> rocksdb::ColumnFamilyData*, rocksdb::MemTable*,
> rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 12:
> (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long,
> std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b)
> [0x55ad520b2dcb]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 13:
> (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor,
> std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool,
> bool)+0xa59) [0x55ad520b3d09]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 14:
> (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&,
> std::vector<rocksdb::ColumnFamilyDescriptor,
> std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
> std::vector<rocksdb::ColumnFamilyHandle*,
> std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**,
> bool)+0x689) [0x55ad520b4ab9]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 15:
> (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&,
> std::vector<rocksdb::ColumnFamilyDescriptor,
> std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
> std::vector<rocksdb::ColumnFamilyHandle*,
> std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22)
> [0x55ad520b62e2]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 16:
> (RocksDBStore::do_open(std::ostream&, bool,
> std::vector<KeyValueDB::ColumnFamily,
> std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (BlueStore::_open_db(bool,
> bool)+0xcf4) [0x55ad51f527a4]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (BlueStore::_mount(bool,
> bool)+0x4e9) [0x55ad51f828f9]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (OSD::init()+0x339)
> [0x55ad51b1cf09]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 20: (main()+0x23d2)
> [0x55ad51a00d52]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 21: (__libc_start_main()+0xf5)
> [0x7f43bc6c7505]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 22: (()+0x378c10) [0x55ad51ad8c10]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 2020-04-18 21:24:37.099
> 7f43c936bb80 -1 *** Caught signal (Aborted) **
> Apr 18 21:24:37 server2 ceph-osd[17777]: in thread 7f43c936bb80
> thread_name:ceph-osd
> Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8
> (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable)
> Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (()+0xf5f0) [0x7f43bd6bb5f0]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (gsignal()+0x37)
> [0x7f43bc6db337]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (abort()+0x148) [0x7f43bc6dca28]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (ceph::__ceph_assert_fail(char
> const*, char const*, int, char const*)+0x248) [0x7f43c0749978]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (()+0x26fa07) [0x7f43c0749a07]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 6:
> (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned
> long)+0x1ac6) [0x55ad52029266]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 7:
> (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 8:
> (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 9:
> (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 10:
> (rocksdb::BuildTable(std::string const&, rocksdb::Env*,
> rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&,
> rocksdb::EnvOptions const&, rocksdb::TableCache*,
> rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator,
> std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*,
> rocksdb::InternalKeyComparator const&,
> std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
> std::default_delete<rocksdb::IntTblPropCollectorFactory> >,
> std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
> std::default_delete<rocksdb::IntTblPropCollectorFactory> > > >
const*,
> unsigned int, std::string const&, std::vector<unsigned long,
> std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*,
> rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool,
> rocksdb::InternalStats*, rocksdb::TableFileCreationReason,
> rocksdb::EventLogger*, int, rocksdb::Env::IOPriority,
> rocksdb::TableProperties*, int, unsigned long, unsigned long,
> rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 11:
> (rocksdb::DBImpl::WriteLevel0TableForRecovery(int,
> rocksdb::ColumnFamilyData*, rocksdb::MemTable*,
> rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 12:
> (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long,
> std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b)
> [0x55ad520b2dcb]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 13:
> (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor,
> std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool,
> bool)+0xa59) [0x55ad520b3d09]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 14:
> (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&,
> std::vector<rocksdb::ColumnFamilyDescriptor,
> std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
> std::vector<rocksdb::ColumnFamilyHandle*,
> std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**,
> bool)+0x689) [0x55ad520b4ab9]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 15:
> (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&,
> std::vector<rocksdb::ColumnFamilyDescriptor,
> std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
> std::vector<rocksdb::ColumnFamilyHandle*,
> std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22)
> [0x55ad520b62e2]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 16:
> (RocksDBStore::do_open(std::ostream&, bool,
> std::vector<KeyValueDB::ColumnFamily,
> std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (BlueStore::_open_db(bool,
> bool)+0xcf4) [0x55ad51f527a4]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (BlueStore::_mount(bool,
> bool)+0x4e9) [0x55ad51f828f9]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (OSD::init()+0x339)
> [0x55ad51b1cf09]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 20: (main()+0x23d2)
> [0x55ad51a00d52]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 21: (__libc_start_main()+0xf5)
> [0x7f43bc6c7505]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 22: (()+0x378c10) [0x55ad51ad8c10]
> Apr 18 21:24:37 server2 ceph-osd[17777]: NOTE: a copy of the executable, or
> `objdump -rdS <executable>` is needed to interpret this.
> Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.090
> 7f43c936bb80 -1 bluefs _allocate failed to allocate 0x on bdev 2, dne
> Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.090
> 7f43c936bb80 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length:
> 0x794de9
> Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.094
> 7f43c936bb80 -1
>
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc:
> In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t,
> uint64_t)' thread 7f43c936bb80 time 2020-04-18 21:24:37.091289
> Apr 18 21:24:37 server2 ceph-osd[17777]:
>
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc:
> 1704: FAILED assert(0 == "bluefs enospc")
> Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8
> (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable)
> Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (ceph::__ceph_assert_fail(char
> const*, char const*, int, char const*)+0x14b) [0x7f43c074987b]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (()+0x26fa07) [0x7f43c0749a07]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 3:
> (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned
> long)+0x1ac6) [0x55ad52029266]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 4:
> (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 5:
> (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 6:
> (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 7:
> (rocksdb::BuildTable(std::string const&, rocksdb::Env*,
> rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&,
> rocksdb::EnvOptions const&, rocksdb::TableCache*,
> rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator,
> std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*,
> rocksdb::InternalKeyComparator const&,
> std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
> std::default_delete<rocksdb::IntTblPropCollectorFactory> >,
> std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
> std::default_delete<rocksdb::IntTblPropCollectorFactory> > > >
const*,
> unsigned int, std::string const&, std::vector<unsigned long,
> std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*,
> rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool,
> rocksdb::InternalStats*, rocksdb::TableFileCreationReason,
> rocksdb::EventLogger*, int, rocksdb::Env::IOPriority,
> rocksdb::TableProperties*, int, unsigned long, unsigned long,
> rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 8:
> (rocksdb::DBImpl::WriteLevel0TableForRecovery(int,
> rocksdb::ColumnFamilyData*, rocksdb::MemTable*,
> rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 9:
> (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long,
> std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b)
> [0x55ad520b2dcb]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 10:
> (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor,
> std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool,
> bool)+0xa59) [0x55ad520b3d09]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 11:
> (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&,
> std::vector<rocksdb::ColumnFamilyDescriptor,
> std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
> std::vector<rocksdb::ColumnFamilyHandle*,
> std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**,
> bool)+0x689) [0x55ad520b4ab9]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 12:
> (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&,
> std::vector<rocksdb::ColumnFamilyDescriptor,
> std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
> std::vector<rocksdb::ColumnFamilyHandle*,
> std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22)
> [0x55ad520b62e2]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 13:
> (RocksDBStore::do_open(std::ostream&, bool,
> std::vector<KeyValueDB::ColumnFamily,
> std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (BlueStore::_open_db(bool,
> bool)+0xcf4) [0x55ad51f527a4]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (BlueStore::_mount(bool,
> bool)+0x4e9) [0x55ad51f828f9]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (OSD::init()+0x339)
> [0x55ad51b1cf09]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (main()+0x23d2)
> [0x55ad51a00d52]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (__libc_start_main()+0xf5)
> [0x7f43bc6c7505]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (()+0x378c10) [0x55ad51ad8c10]
> Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.099
> 7f43c936bb80 -1 *** Caught signal (Aborted) **
> Apr 18 21:24:37 server2 ceph-osd[17777]: in thread 7f43c936bb80
> thread_name:ceph-osd
> Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8
> (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable)
> Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (()+0xf5f0) [0x7f43bd6bb5f0]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (gsignal()+0x37)
> [0x7f43bc6db337]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (abort()+0x148) [0x7f43bc6dca28]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (ceph::__ceph_assert_fail(char
> const*, char const*, int, char const*)+0x248) [0x7f43c0749978]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (()+0x26fa07) [0x7f43c0749a07]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 6:
> (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned
> long)+0x1ac6) [0x55ad52029266]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 7:
> (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 8:
> (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 9:
> (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 10:
> (rocksdb::BuildTable(std::string const&, rocksdb::Env*,
> rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&,
> rocksdb::EnvOptions const&, rocksdb::TableCache*,
> rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator,
> std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*,
> rocksdb::InternalKeyComparator const&,
> std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
> std::default_delete<rocksdb::IntTblPropCollectorFactory> >,
> std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
> std::default_delete<rocksdb::IntTblPropCollectorFactory> > > >
const*,
> unsigned int, std::string const&, std::vector<unsigned long,
> std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*,
> rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool,
> rocksdb::InternalStats*, rocksdb::TableFileCreationReason,
> rocksdb::EventLogger*, int, rocksdb::Env::IOPriority,
> rocksdb::TableProperties*, int, unsigned long, unsigned long,
> rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 11:
> (rocksdb::DBImpl::WriteLevel0TableForRecovery(int,
> rocksdb::ColumnFamilyData*, rocksdb::MemTable*,
> rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 12:
> (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long,
> std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b)
> [0x55ad520b2dcb]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 13:
> (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor,
> std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool,
> bool)+0xa59) [0x55ad520b3d09]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 14:
> (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&,
> std::vector<rocksdb::ColumnFamilyDescriptor,
> std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
> std::vector<rocksdb::ColumnFamilyHandle*,
> std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**,
> bool)+0x689) [0x55ad520b4ab9]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 15:
> (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&,
> std::vector<rocksdb::ColumnFamilyDescriptor,
> std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
> std::vector<rocksdb::ColumnFamilyHandle*,
> std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22)
> [0x55ad520b62e2]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 16:
> (RocksDBStore::do_open(std::ostream&, bool,
> std::vector<KeyValueDB::ColumnFamily,
> std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (BlueStore::_open_db(bool,
> bool)+0xcf4) [0x55ad51f527a4]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (BlueStore::_mount(bool,
> bool)+0x4e9) [0x55ad51f828f9]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (OSD::init()+0x339)
> [0x55ad51b1cf09]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 20: (main()+0x23d2)
> [0x55ad51a00d52]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 21: (__libc_start_main()+0xf5)
> [0x7f43bc6c7505]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 22: (()+0x378c10) [0x55ad51ad8c10]
> Apr 18 21:24:37 server2 ceph-osd[17777]: NOTE: a copy of the executable, or
> `objdump -rdS <executable>` is needed to interpret this.
> Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.090
> 7f43c936bb80 -1 bluefs _allocate failed to allocate 0x on bdev 2, dne
> Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.090
> 7f43c936bb80 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length:
> 0x794de9
> Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.094
> 7f43c936bb80 -1
>
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc:
> In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t,
> uint64_t)' thread 7f43c936bb80 time 2020-04-18 21:24:37.091289
> Apr 18 21:24:37 server2 ceph-osd[17777]:
>
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.8/rpm/el7/BUILD/ceph-13.2.8/src/os/bluestore/BlueFS.cc:
> 1704: FAILED assert(0 == "bluefs enospc")
> Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8
> (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable)
> Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (ceph::__ceph_assert_fail(char
> const*, char const*, int, char const*)+0x14b) [0x7f43c074987b]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (()+0x26fa07) [0x7f43c0749a07]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 3:
> (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned
> long)+0x1ac6) [0x55ad52029266]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 4:
> (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 5:
> (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 6:
> (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 7:
> (rocksdb::BuildTable(std::string const&, rocksdb::Env*,
> rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&,
> rocksdb::EnvOptions const&, rocksdb::TableCache*,
> rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator,
> std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*,
> rocksdb::InternalKeyComparator const&,
> std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
> std::default_delete<rocksdb::IntTblPropCollectorFactory> >,
> std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
> std::default_delete<rocksdb::IntTblPropCollectorFactory> > > >
const*,
> unsigned int, std::string const&, std::vector<unsigned long,
> std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*,
> rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool,
> rocksdb::InternalStats*, rocksdb::TableFileCreationReason,
> rocksdb::EventLogger*, int, rocksdb::Env::IOPriority,
> rocksdb::TableProperties*, int, unsigned long, unsigned long,
> rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 8:
> (rocksdb::DBImpl::WriteLevel0TableForRecovery(int,
> rocksdb::ColumnFamilyData*, rocksdb::MemTable*,
> rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 9:
> (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long,
> std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b)
> [0x55ad520b2dcb]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 10:
> (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor,
> std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool,
> bool)+0xa59) [0x55ad520b3d09]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 11:
> (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&,
> std::vector<rocksdb::ColumnFamilyDescriptor,
> std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
> std::vector<rocksdb::ColumnFamilyHandle*,
> std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**,
> bool)+0x689) [0x55ad520b4ab9]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 12:
> (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&,
> std::vector<rocksdb::ColumnFamilyDescriptor,
> std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
> std::vector<rocksdb::ColumnFamilyHandle*,
> std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22)
> [0x55ad520b62e2]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 13:
> (RocksDBStore::do_open(std::ostream&, bool,
> std::vector<KeyValueDB::ColumnFamily,
> std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 14: (BlueStore::_open_db(bool,
> bool)+0xcf4) [0x55ad51f527a4]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 15: (BlueStore::_mount(bool,
> bool)+0x4e9) [0x55ad51f828f9]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 16: (OSD::init()+0x339)
> [0x55ad51b1cf09]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (main()+0x23d2)
> [0x55ad51a00d52]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (__libc_start_main()+0xf5)
> [0x7f43bc6c7505]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (()+0x378c10) [0x55ad51ad8c10]
> Apr 18 21:24:37 server2 ceph-osd[17777]: -324> 2020-04-18 21:24:37.099
> 7f43c936bb80 -1 *** Caught signal (Aborted) **
> Apr 18 21:24:37 server2 ceph-osd[17777]: in thread 7f43c936bb80
> thread_name:ceph-osd
> Apr 18 21:24:37 server2 ceph-osd[17777]: ceph version 13.2.8
> (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable)
> Apr 18 21:24:37 server2 ceph-osd[17777]: 1: (()+0xf5f0) [0x7f43bd6bb5f0]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 2: (gsignal()+0x37)
> [0x7f43bc6db337]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 3: (abort()+0x148) [0x7f43bc6dca28]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 4: (ceph::__ceph_assert_fail(char
> const*, char const*, int, char const*)+0x248) [0x7f43c0749978]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 5: (()+0x26fa07) [0x7f43c0749a07]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 6:
> (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned
> long)+0x1ac6) [0x55ad52029266]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 7:
> (BlueRocksWritableFile::Flush()+0x3d) [0x55ad520451bd]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 8:
> (rocksdb::WritableFileWriter::Flush()+0x196) [0x55ad521f7916]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 9:
> (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x55ad521f7bde]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 10:
> (rocksdb::BuildTable(std::string const&, rocksdb::Env*,
> rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&,
> rocksdb::EnvOptions const&, rocksdb::TableCache*,
> rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator,
> std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*,
> rocksdb::InternalKeyComparator const&,
> std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
> std::default_delete<rocksdb::IntTblPropCollectorFactory> >,
> std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory,
> std::default_delete<rocksdb::IntTblPropCollectorFactory> > > >
const*,
> unsigned int, std::string const&, std::vector<unsigned long,
> std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*,
> rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool,
> rocksdb::InternalStats*, rocksdb::TableFileCreationReason,
> rocksdb::EventLogger*, int, rocksdb::Env::IOPriority,
> rocksdb::TableProperties*, int, unsigned long, unsigned long,
> rocksdb::Env::WriteLifeTimeHint)+0x11d8) [0x55ad5221d5a8]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 11:
> (rocksdb::DBImpl::WriteLevel0TableForRecovery(int,
> rocksdb::ColumnFamilyData*, rocksdb::MemTable*,
> rocksdb::VersionEdit*)+0xbe6) [0x55ad520b0d76]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 12:
> (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long,
> std::allocator<unsigned long> > const&, unsigned long*, bool)+0x185b)
> [0x55ad520b2dcb]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 13:
> (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor,
> std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool,
> bool)+0xa59) [0x55ad520b3d09]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 14:
> (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&,
> std::vector<rocksdb::ColumnFamilyDescriptor,
> std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
> std::vector<rocksdb::ColumnFamilyHandle*,
> std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**,
> bool)+0x689) [0x55ad520b4ab9]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 15:
> (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&,
> std::vector<rocksdb::ColumnFamilyDescriptor,
> std::allocator<rocksdb::ColumnFamilyDescriptor> > const&,
> std::vector<rocksdb::ColumnFamilyHandle*,
> std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22)
> [0x55ad520b62e2]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 16:
> (RocksDBStore::do_open(std::ostream&, bool,
> std::vector<KeyValueDB::ColumnFamily,
> std::allocator<KeyValueDB::ColumnFamily> > const*)+0x164e) [0x55ad51fc65de]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 17: (BlueStore::_open_db(bool,
> bool)+0xcf4) [0x55ad51f527a4]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 18: (BlueStore::_mount(bool,
> bool)+0x4e9) [0x55ad51f828f9]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 19: (OSD::init()+0x339)
> [0x55ad51b1cf09]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 20: (main()+0x23d2)
> [0x55ad51a00d52]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 21: (__libc_start_main()+0xf5)
> [0x7f43bc6c7505]
> Apr 18 21:24:37 server2 ceph-osd[17777]: 22: (()+0x378c10) [0x55ad51ad8c10]
> Apr 18 21:24:37 server2 ceph-osd[17777]: NOTE: a copy of the executable, or
> `objdump -rdS <executable>` is needed to interpret this.
> Apr 18 21:24:37 server2 systemd[1]: ceph-osd(a)4.service: main process
> exited, code=killed, status=6/ABRT
> Apr 18 21:24:37 server2 systemd[1]: Unit ceph-osd(a)4.service entered failed
> state.
> Apr 18 21:24:37 server2 systemd[1]: ceph-osd(a)4.service failed.
> Apr 18 21:24:57 server2 systemd[1]: ceph-osd(a)4.service holdoff time over,
> scheduling restart.
> Apr 18 21:24:57 server2 systemd[1]: Stopped Ceph object storage daemon
> osd.4.
> Apr 18 21:24:57 server2 systemd[1]: Starting Ceph object storage daemon
> osd.4...
> Apr 18 21:24:57 server2 systemd[1]: Started Ceph object storage daemon
> osd.4.
>
>
>
>
>
> and some more data:
>
> [root@server1 ~]# ceph df detail
> GLOBAL:
> SIZE AVAIL RAW USED %RAW USED OBJECTS
> 38 GiB 35 GiB 2.6 GiB 6.83 303
> POOLS:
> NAME ID QUOTA OBJECTS QUOTA BYTES
> USED %USED MAX AVAIL OBJECTS DIRTY READ WRITE
> RAW USED
> .rgw.root 18 N/A N/A
> 1.1 KiB 0 50 GiB 4 4 63 B 4
> B 1.1 KiB
> default.rgw.meta 19 N/A N/A
> 4.1 KiB 0 50 GiB 22 22 1.5 KiB 182
> B 9.0 KiB
> default.rgw.log 20 N/A N/A
> 0 B 0 50 GiB 107 107 307 KiB 205
> KiB 0 B
> default.rgw.control 21 N/A N/A
> 0 B 0 50 GiB 5 5 0 B 0
> B 0 B
> default.rgw.buckets.index 22 N/A N/A
> 0 B 0 50 GiB 5 5 501 B 60
> B 0 B
> default.rgw.buckets.data 23 N/A N/A
> 377 MiB 0.39 50 GiB 160 160 385 B 1.1
> KiB 603 MiB
> [root@server1 ~]#
>
> [root@server1 ~]# ceph osd df tree
> ID CLASS WEIGHT REWEIGHT SIZE USE DATA OMAP META AVAIL %USE
> VAR PGS TYPE NAME
> -1 0.16727 - 0 B 0 B 0 B 0 B 0 B 0 B 0
> 0 - root default
> -3 0.05576 - 0 B 0 B 0 B 0 B 0 B 0 B 0
> 0 - host server1
> 0 hdd 0.01859 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0
> 0 0 osd.0
> 1 hdd 0.01859 0 0 B 0 B 0 B 0 B 0 B 0 B 0
> 0 0 osd.1
> 2 hdd 0.01859 0 0 B 0 B 0 B 0 B 0 B 0 B 0
> 0 0 osd.2
> -5 0.05576 - 19 GiB 1.4 GiB 360 MiB 3 KiB 1024 MiB 18 GiB 0
> 0 - host server2
> 3 hdd 0.01859 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0
> 0 0 osd.3
> 4 hdd 0.01859 0 0 B 0 B 0 B 0 B 0 B 0 B 0
> 0 0 osd.4
> 5 hdd 0.01859 1.00000 19 GiB 1.4 GiB 360 MiB 3 KiB 1024 MiB 18 GiB 7.11
> 1.04 99 osd.5
> -7 0.05576 - 0 B 0 B 0 B 0 B 0 B 0 B 0
> 0 - host server3
> 6 hdd 0.01859 1.00000 19 GiB 1.2 GiB 249 MiB 3 KiB 1024 MiB 18 GiB 6.55
> 0.96 78 osd.6
> 7 hdd 0.01859 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0
> 0 0 osd.7
> 8 hdd 0.01859 1.00000 0 B 0 B 0 B 0 B 0 B 0 B 0
> 0 0 osd.8
> TOTAL 38 GiB 2.6 GiB 610 MiB 6 KiB 2.0 GiB 35 GiB 6.83
>
> MIN/MAX VAR: 0/1.04 STDDEV: 5.58
> [root@server1 ~]#
>
>
> I'm kind of newbie to ceph, so any help or hint would be appreciated.
> Did I hit a bug or something is wrong with my configuration?
>
> Thanks a lot,
> Khodayar
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io
> To unsubscribe send an email to ceph-users-leave(a)ceph.io
>