[ceph-users] osd won't start

5 May 2020

I've been using CentOS 7 and 5.6.10-1.el7.elrepo.x86_64 Linux kernel.
After today's update and reboot osd's wont start.

# podman run --privileged --pid=host --cpuset-cpus 0,1 --memory 2g --name
ceph_osd0 --hostname ceph_osd0 --ip 172.30.0.10 -v /dev:/dev -v
/etc/localtime:/etc/localtime:ro -v /etc/ceph:/etc/ceph/ -v
/var/lib/ceph/osd/ceph-0:/var/lib/ceph/osd/ceph-0 -v /dev/:/dev/ -e
OSD_DEVICE=/dev/mapper/ceph-0block -e OSD_BLUESTORE=1 -e CLUSTER=ceph -e
OSD_ID=0 -e DEBUG=stayalive
ceph/daemon:v5.0.1-stable-5.0-octopus-centos-8-x86_64 osd_directory_single
2020-05-05 20:19:54  /opt/ceph-container/bin/entrypoint.sh: STAYALIVE:
container will not die if a command fails.
2020-05-05 20:19:54  /opt/ceph-container/bin/entrypoint.sh: static: does
not generate config
2020-05-05 20:19:54  /opt/ceph-container/bin/entrypoint.sh: Looks like OSD:
0 is not started, starting it...
2020-05-05 20:19:54  /opt/ceph-container/bin/entrypoint.sh: SUCCESS
exec: PID 45073: spawning ceph-osd --cluster ceph --setuser ceph --setgroup
ceph --log-to-stderr=true --err-to-stderr=true --default-log-to-file=false
--foreground -i 0 -k /var/lib/ceph/osd/ceph-0//keyring
exec: Waiting 45073 to quit
2020-05-05T20:19:54.932-0400 7fb999290ec0 -1 WARNING: the following
dangerous and experimental features are enabled: bluestore,rocksdb
2020-05-05T20:19:54.934-0400 7fb999290ec0 -1 WARNING: the following
dangerous and experimental features are enabled: bluestore,rocksdb
2020-05-05T20:19:54.934-0400 7fb999290ec0  0 set uid:gid to 167:167
(ceph:ceph)
2020-05-05T20:19:54.934-0400 7fb999290ec0  0 ceph version 15.2.1
(9fd2f65f91d9246fae2c841a6222d34d121680ee) octopus (stable), process
ceph-osd, pid 45073
2020-05-05T20:19:54.934-0400 7fb999290ec0  0 pidfile_write: ignore empty
--pid-file
2020-05-05T20:19:54.934-0400 7fb999290ec0 -1 WARNING: the following
dangerous and experimental features are enabled: bluestore,rocksdb
2020-05-05T20:19:54.934-0400 7fb999290ec0  5 object store type is bluestore
2020-05-05T20:19:54.935-0400 7fb999290ec0  1 bdev create path
/var/lib/ceph/osd/ceph-0/block type kernel
2020-05-05T20:19:54.935-0400 7fb999290ec0  1 bdev(0x561a009e0000
/var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
2020-05-05T20:19:54.935-0400 7fb999290ec0  1 bdev(0x561a009e0000
/var/lib/ceph/osd/ceph-0/block) open size 6001171365888 (0x57541b00000, 5.5
TiB) block_size 4096 (4 KiB) rotational discard not supported
2020-05-05T20:19:54.935-0400 7fb999290ec0  1
bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 1073741824
meta 0.4 kv 0.4 data 0.2
2020-05-05T20:19:54.935-0400 7fb999290ec0  1 bdev create path
/var/lib/ceph/osd/ceph-0/block.db type kernel
2020-05-05T20:19:54.935-0400 7fb999290ec0  1 bdev(0x561a009e0700
/var/lib/ceph/osd/ceph-0/block.db) open path
/var/lib/ceph/osd/ceph-0/block.db
2020-05-05T20:19:54.936-0400 7fb999290ec0  1 bdev(0x561a009e0700
/var/lib/ceph/osd/ceph-0/block.db) open size 49998200832 (0xba4200000, 47
GiB) block_size 4096 (4 KiB) non-rotational discard not supported
2020-05-05T20:19:54.936-0400 7fb999290ec0  1 bluefs add_block_device bdev 1
path /var/lib/ceph/osd/ceph-0/block.db size 47 GiB
2020-05-05T20:19:54.936-0400 7fb999290ec0  1 bdev create path
/var/lib/ceph/osd/ceph-0/block type kernel
2020-05-05T20:19:54.936-0400 7fb999290ec0  1 bdev(0x561a009e0a80
/var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
2020-05-05T20:19:54.936-0400 7fb999290ec0  1 bdev(0x561a009e0a80
/var/lib/ceph/osd/ceph-0/block) open size 6001171365888 (0x57541b00000, 5.5
TiB) block_size 4096 (4 KiB) rotational discard not supported
2020-05-05T20:19:54.936-0400 7fb999290ec0  1 bluefs add_block_device bdev 2
path /var/lib/ceph/osd/ceph-0/block size 5.5 TiB
2020-05-05T20:19:54.936-0400 7fb999290ec0  1 bdev create path
/var/lib/ceph/osd/ceph-0/block.wal type kernel
2020-05-05T20:19:54.936-0400 7fb999290ec0  1 bdev(0x561a009e0e00
/var/lib/ceph/osd/ceph-0/block.wal) open path
/var/lib/ceph/osd/ceph-0/block.wal
2020-05-05T20:19:54.937-0400 7fb999290ec0  1 bdev(0x561a009e0e00
/var/lib/ceph/osd/ceph-0/block.wal) open size 997195776 (0x3b700000, 951
MiB) block_size 4096 (4 KiB) non-rotational discard not supported
2020-05-05T20:19:54.937-0400 7fb999290ec0  1 bluefs add_block_device bdev 0
path /var/lib/ceph/osd/ceph-0/block.wal size 951 MiB
2020-05-05T20:19:54.937-0400 7fb999290ec0  1 bdev(0x561a009e0e00
/var/lib/ceph/osd/ceph-0/block.wal) close
2020-05-05T20:19:55.214-0400 7fb999290ec0  1 bdev(0x561a009e0700
/var/lib/ceph/osd/ceph-0/block.db) close
2020-05-05T20:19:55.453-0400 7fb999290ec0  1 bdev(0x561a009e0a80
/var/lib/ceph/osd/ceph-0/block) close
2020-05-05T20:19:55.714-0400 7fb999290ec0  1 bdev(0x561a009e0000
/var/lib/ceph/osd/ceph-0/block) close
2020-05-05T20:19:55.956-0400 7fb999290ec0  0 starting osd.0 osd_data
/var/lib/ceph/osd/ceph-0 /var/lib/ceph/osd/ceph-0/journal
2020-05-05T20:19:55.963-0400 7fb999290ec0  0 load: jerasure load: lrc load:
isa
2020-05-05T20:19:55.963-0400 7fb999290ec0  1 bdev create path
/var/lib/ceph/osd/ceph-0/block type kernel
2020-05-05T20:19:55.963-0400 7fb999290ec0  1 bdev(0x561a009e0000
/var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
2020-05-05T20:19:55.963-0400 7fb999290ec0  1 bdev(0x561a009e0000
/var/lib/ceph/osd/ceph-0/block) open size 6001171365888 (0x57541b00000, 5.5
TiB) block_size 4096 (4 KiB) rotational discard not supported
2020-05-05T20:19:55.963-0400 7fb999290ec0  1
bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 1073741824
meta 0.4 kv 0.4 data 0.2
2020-05-05T20:19:55.963-0400 7fb999290ec0  1 bdev(0x561a009e0000
/var/lib/ceph/osd/ceph-0/block) close
2020-05-05T20:19:56.235-0400 7fb999290ec0  0 osd.0:0.OSDShard using op
scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196)
2020-05-05T20:19:56.235-0400 7fb999290ec0  0 osd.0:1.OSDShard using op
scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196)
2020-05-05T20:19:56.235-0400 7fb999290ec0  0 osd.0:2.OSDShard using op
scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196)
2020-05-05T20:19:56.235-0400 7fb999290ec0  0 osd.0:3.OSDShard using op
scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196)
2020-05-05T20:19:56.235-0400 7fb999290ec0  0 osd.0:4.OSDShard using op
scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196)
2020-05-05T20:19:56.235-0400 7fb999290ec0  2 osd.0 0 init
/var/lib/ceph/osd/ceph-0 (looks like hdd)
2020-05-05T20:19:56.235-0400 7fb999290ec0  2 osd.0 0 journal
/var/lib/ceph/osd/ceph-0/journal
2020-05-05T20:19:56.235-0400 7fb999290ec0  1
bluestore(/var/lib/ceph/osd/ceph-0) _mount path /var/lib/ceph/osd/ceph-0
2020-05-05T20:19:56.235-0400 7fb999290ec0  1 bdev create path
/var/lib/ceph/osd/ceph-0/block type kernel
2020-05-05T20:19:56.235-0400 7fb999290ec0  1 bdev(0x561a009e0000
/var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
2020-05-05T20:19:56.236-0400 7fb999290ec0  1 bdev(0x561a009e0000
/var/lib/ceph/osd/ceph-0/block) open size 6001171365888 (0x57541b00000, 5.5
TiB) block_size 4096 (4 KiB) rotational discard not supported
2020-05-05T20:19:56.236-0400 7fb999290ec0  1
bluestore(/var/lib/ceph/osd/ceph-0) _set_cache_sizes cache_size 1073741824
meta 0.4 kv 0.4 data 0.2
2020-05-05T20:19:56.236-0400 7fb999290ec0  1 bdev create path
/var/lib/ceph/osd/ceph-0/block.db type kernel
2020-05-05T20:19:56.236-0400 7fb999290ec0  1 bdev(0x561a009e0e00
/var/lib/ceph/osd/ceph-0/block.db) open path
/var/lib/ceph/osd/ceph-0/block.db
2020-05-05T20:19:56.236-0400 7fb999290ec0  1 bdev(0x561a009e0e00
/var/lib/ceph/osd/ceph-0/block.db) open size 49998200832 (0xba4200000, 47
GiB) block_size 4096 (4 KiB) non-rotational discard not supported
2020-05-05T20:19:56.236-0400 7fb999290ec0  1 bluefs add_block_device bdev 1
path /var/lib/ceph/osd/ceph-0/block.db size 47 GiB
2020-05-05T20:19:56.242-0400 7fb999290ec0  1 bdev create path
/var/lib/ceph/osd/ceph-0/block type kernel
2020-05-05T20:19:56.242-0400 7fb999290ec0  1 bdev(0x561a009e0700
/var/lib/ceph/osd/ceph-0/block) open path /var/lib/ceph/osd/ceph-0/block
2020-05-05T20:19:56.243-0400 7fb999290ec0  1 bdev(0x561a009e0700
/var/lib/ceph/osd/ceph-0/block) open size 6001171365888 (0x57541b00000, 5.5
TiB) block_size 4096 (4 KiB) rotational discard not supported
2020-05-05T20:19:56.243-0400 7fb999290ec0  1 bluefs add_block_device bdev 2
path /var/lib/ceph/osd/ceph-0/block size 5.5 TiB
2020-05-05T20:19:56.243-0400 7fb999290ec0  1 bdev create path
/var/lib/ceph/osd/ceph-0/block.wal type kernel
2020-05-05T20:19:56.243-0400 7fb999290ec0  1 bdev(0x561a009e0a80
/var/lib/ceph/osd/ceph-0/block.wal) open path
/var/lib/ceph/osd/ceph-0/block.wal
2020-05-05T20:19:56.243-0400 7fb999290ec0  1 bdev(0x561a009e0a80
/var/lib/ceph/osd/ceph-0/block.wal) open size 997195776 (0x3b700000, 951
MiB) block_size 4096 (4 KiB) non-rotational discard not supported
2020-05-05T20:19:56.243-0400 7fb999290ec0  1 bluefs add_block_device bdev 0
path /var/lib/ceph/osd/ceph-0/block.wal size 951 MiB
2020-05-05T20:19:56.243-0400 7fb999290ec0  0  set rocksdb option
compaction_readahead_size = 2097152
2020-05-05T20:19:56.243-0400 7fb999290ec0  0  set rocksdb option
compression = kNoCompression
2020-05-05T20:19:56.243-0400 7fb999290ec0  0  set rocksdb option
max_background_compactions = 2
2020-05-05T20:19:56.243-0400 7fb999290ec0  0  set rocksdb option
max_write_buffer_number = 4
2020-05-05T20:19:56.243-0400 7fb999290ec0  0  set rocksdb option
min_write_buffer_number_to_merge = 1
2020-05-05T20:19:56.243-0400 7fb999290ec0  0  set rocksdb option
recycle_log_file_num = 4
2020-05-05T20:19:56.243-0400 7fb999290ec0  0  set rocksdb option
writable_file_max_buffer_size = 0
2020-05-05T20:19:56.243-0400 7fb999290ec0  0  set rocksdb option
write_buffer_size = 268435456
2020-05-05T20:19:56.243-0400 7fb999290ec0  1 bluefs mount
2020-05-05T20:19:56.243-0400 7fb999290ec0  1 bluefs _init_alloc id 0
alloc_size 0x100000 size 0x3b700000
2020-05-05T20:19:56.243-0400 7fb999290ec0  1 bluefs _init_alloc id 1
alloc_size 0x100000 size 0xba4200000
2020-05-05T20:19:56.244-0400 7fb999290ec0  1 bluefs _init_alloc id 2
alloc_size 0x10000 size 0x57541b00000
2020-05-05T20:19:56.343-0400 7fb999290ec0  1
bluestore(/var/lib/ceph/osd/ceph-0) _open_db set db_paths to db,47498290790
db.slow,5701112797593
2020-05-05T20:19:56.343-0400 7fb999290ec0  0  set rocksdb option
compaction_readahead_size = 2097152
2020-05-05T20:19:56.343-0400 7fb999290ec0  0  set rocksdb option
compression = kNoCompression
2020-05-05T20:19:56.343-0400 7fb999290ec0  0  set rocksdb option
max_background_compactions = 2
2020-05-05T20:19:56.343-0400 7fb999290ec0  0  set rocksdb option
max_write_buffer_number = 4
2020-05-05T20:19:56.343-0400 7fb999290ec0  0  set rocksdb option
min_write_buffer_number_to_merge = 1
2020-05-05T20:19:56.343-0400 7fb999290ec0  0  set rocksdb option
recycle_log_file_num = 4
2020-05-05T20:19:56.343-0400 7fb999290ec0  0  set rocksdb option
writable_file_max_buffer_size = 0
2020-05-05T20:19:56.343-0400 7fb999290ec0  0  set rocksdb option
write_buffer_size = 268435456
2020-05-05T20:19:56.343-0400 7fb999290ec0  0  set rocksdb option
compaction_readahead_size = 2097152
2020-05-05T20:19:56.343-0400 7fb999290ec0  0  set rocksdb option
compression = kNoCompression
2020-05-05T20:19:56.343-0400 7fb999290ec0  0  set rocksdb option
max_background_compactions = 2
2020-05-05T20:19:56.343-0400 7fb999290ec0  0  set rocksdb option
max_write_buffer_number = 4
2020-05-05T20:19:56.343-0400 7fb999290ec0  0  set rocksdb option
min_write_buffer_number_to_merge = 1
2020-05-05T20:19:56.343-0400 7fb999290ec0  0  set rocksdb option
recycle_log_file_num = 4
2020-05-05T20:19:56.343-0400 7fb999290ec0  0  set rocksdb option
writable_file_max_buffer_size = 0
2020-05-05T20:19:56.343-0400 7fb999290ec0  0  set rocksdb option
write_buffer_size = 268435456
2020-05-05T20:19:56.344-0400 7fb999290ec0  1 rocksdb: do_open column
families: [default]
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb: RocksDB version: 6.1.2

2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb: Git sha
rocksdb_build_git_sha:@0@
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb: Compile date Apr  8
2020
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb: DB SUMMARY

2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb: CURRENT file:  CURRENT

2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb: IDENTITY file:
 IDENTITY

2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb: MANIFEST file:
 MANIFEST-004159 size: 115013 Bytes

2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb: SST files in db dir,
Total Num: 174, files: 001179.sst 002875.sst 002961.sst 003070.sst
003102.sst 003217.sst 003290.sst 003324.sst 003399.sst

2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb: SST files in db.slow
dir, Total Num: 0, files:

2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb: Write Ahead Log file
in db.wal: 005071.log size: 254490391 ; 005073.log size: 254377461 ;

2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
  Options.error_if_exists: 0
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.create_if_missing: 0
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
  Options.paranoid_checks: 1
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
              Options.env: 0x5619ffd19b80
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
         Options.info_log: 0x5619ffd234e0
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
 Options.max_file_opening_threads: 16
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
       Options.statistics: (nil)
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
        Options.use_fsync: 0
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.max_log_file_size: 0
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
 Options.max_manifest_file_size: 1073741824
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.log_file_time_to_roll: 0
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.keep_log_file_num: 1000
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
 Options.recycle_log_file_num: 4
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
  Options.allow_fallocate: 1
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
 Options.allow_mmap_reads: 0
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.allow_mmap_writes: 0
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
 Options.use_direct_reads: 0
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
 Options.use_direct_io_for_flush_and_compaction: 0
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
 Options.create_missing_column_families: 0
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
       Options.db_log_dir:
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
          Options.wal_dir: db.wal
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
 Options.table_cache_numshardbits: 6
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
 Options.max_subcompactions: 1
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
 Options.max_background_flushes: -1
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
  Options.WAL_ttl_seconds: 0
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.WAL_size_limit_MB: 0
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.manifest_preallocation_size: 4194304
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.is_fd_close_on_exec: 1
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.advise_random_on_open: 1
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
 Options.db_write_buffer_size: 0
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
 Options.write_buffer_manager: 0x561a00a2b650
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.access_hint_on_compaction_start: 1
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
 Options.new_table_reader_for_compaction_inputs: 1
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.random_access_max_buffer_size: 1048576
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
 Options.use_adaptive_mutex: 0
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
     Options.rate_limiter: (nil)
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.sst_file_manager.rate_bytes_per_sec: 0
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.wal_recovery_mode: 0
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
 Options.enable_thread_tracking: 0
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
 Options.enable_pipelined_write: 0
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.allow_concurrent_memtable_write: 1
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
 Options.enable_write_thread_adaptive_yield: 1
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.write_thread_max_yield_usec: 100
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
 Options.write_thread_slow_yield_usec: 3
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
        Options.row_cache: None
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
       Options.wal_filter: None
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.avoid_flush_during_recovery: 0
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.allow_ingest_behind: 0
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.preserve_deletes: 0
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.two_write_queues: 0
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.manual_wal_flush: 0
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.atomic_flush: 0
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.avoid_unnecessary_blocking_io: 0
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.max_background_jobs: 2
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.max_background_compactions: 2
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.avoid_flush_during_shutdown: 0
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.writable_file_max_buffer_size: 0
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.delayed_write_rate : 16777216
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.max_total_wal_size: 0
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.delete_obsolete_files_period_micros: 21600000000
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.stats_dump_period_sec: 600
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.stats_persist_period_sec: 600
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.stats_history_buffer_size: 1048576
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
   Options.max_open_files: -1
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
   Options.bytes_per_sync: 0
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
 Options.wal_bytes_per_sync: 0
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.compaction_readahead_size: 2097152
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb: Compression
algorithms supported:
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
kZSTDNotFinalCompression supported: 0
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb: kZSTD supported: 0
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb: kXpressCompression
supported: 0
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb: kLZ4HCCompression
supported: 1
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb: kLZ4Compression
supported: 1
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb: kBZip2Compression
supported: 0
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb: kZlibCompression
supported: 1
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb: kSnappyCompression
supported: 1
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb: Fast CRC32 supported:
Supported on x86
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
[db/db_impl_readonly.cc:22] Opening the db in read only mode
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
[db/version_set.cc:3543] Recovering from manifest file: db/MANIFEST-004159

2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
[db/column_family.cc:477] --------------- Options for column family
[default]:

2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.comparator: leveldb.BytewiseComparator
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.merge_operator: .T:int64_array.b:bitwise_xor
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
 Options.compaction_filter: None
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
 Options.compaction_filter_factory: None
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.memtable_factory: SkipListFactory
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
 Options.table_factory: BlockBasedTable
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
 table_factory options:   flush_block_policy_factory:
FlushBlockBySizePolicyFactory (0x5619ffcb7360)
  cache_index_and_filter_blocks: 1
  cache_index_and_filter_blocks_with_high_priority: 1
  pin_l0_filter_and_index_blocks_in_cache: 0
  pin_top_level_index_and_filter: 1
  index_type: 0
  data_block_index_type: 0
  data_block_hash_table_util_ratio: 0.750000
  hash_index_allow_collision: 1
  checksum: 1
  no_block_cache: 0
  block_cache: 0x5619ffcfae10
  block_cache_name: BinnedLRUCache
  block_cache_options:
    capacity : 429496729
    num_shard_bits : 4
    strict_capacity_limit : 0
    high_pri_pool_ratio: 0.000
  block_cache_compressed: (nil)
  persistent_cache: (nil)
  block_size: 4096
  block_size_deviation: 10
  block_restart_interval: 16
  index_block_restart_interval: 1
  metadata_block_size: 4096
  partition_filters: 0
  use_delta_encoding: 1
  filter_policy: rocksdb.BuiltinBloomFilter
  whole_key_filtering: 1
  verify_compression: 0
  read_amp_bytes_per_bit: 0
  format_version: 2
  enable_index_compression: 1
  block_align: 0

2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
 Options.write_buffer_size: 268435456
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
 Options.max_write_buffer_number: 4
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
 Options.compression: NoCompression
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
 Options.bottommost_compression: Disabled
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.prefix_extractor: nullptr
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.memtable_insert_with_hint_prefix_extractor: nullptr
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.num_levels: 7
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
 Options.min_write_buffer_number_to_merge: 1
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.max_write_buffer_number_to_maintain: 0
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
 Options.bottommost_compression_opts.window_bits: -14
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
 Options.bottommost_compression_opts.level: 32767
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.bottommost_compression_opts.strategy: 0
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.bottommost_compression_opts.max_dict_bytes: 0
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.bottommost_compression_opts.zstd_max_train_bytes: 0
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
 Options.bottommost_compression_opts.enabled: false
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
 Options.compression_opts.window_bits: -14
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
 Options.compression_opts.level: 32767
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.compression_opts.strategy: 0
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.compression_opts.max_dict_bytes: 0
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.compression_opts.zstd_max_train_bytes: 0
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
 Options.compression_opts.enabled: false
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
 Options.level0_file_num_compaction_trigger: 4
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
 Options.level0_slowdown_writes_trigger: 20
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
 Options.level0_stop_writes_trigger: 36
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.target_file_size_base: 67108864
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.target_file_size_multiplier: 1
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
 Options.max_bytes_for_level_base: 268435456
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.level_compaction_dynamic_level_bytes: 0
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
 Options.max_bytes_for_level_multiplier: 10.000000
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[0]: 1
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[1]: 1
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[2]: 1
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[3]: 1
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[4]: 1
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[5]: 1
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.max_bytes_for_level_multiplier_addtl[6]: 1
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.max_sequential_skip_in_iterations: 8
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
 Options.max_compaction_bytes: 1677721600
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
 Options.arena_block_size: 33554432
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.soft_pending_compaction_bytes_limit: 68719476736
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.hard_pending_compaction_bytes_limit: 274877906944
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.rate_limit_delay_max_milliseconds: 100
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
 Options.disable_auto_compactions: 0
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
 Options.compaction_style: kCompactionStyleLevel
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
   Options.compaction_pri: kMinOverlappingRatio
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.compaction_options_universal.size_ratio: 1
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.compaction_options_universal.min_merge_width: 2
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.compaction_options_universal.max_merge_width: 4294967295
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.compaction_options_universal.max_size_amplification_percent: 200
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.compaction_options_universal.compression_size_percent: -1
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.compaction_options_universal.stop_style:
kCompactionStopStyleTotalSize
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.compaction_options_fifo.max_table_files_size: 1073741824
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.compaction_options_fifo.allow_compaction: 0
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.table_properties_collectors:
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.inplace_update_support: 0
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.inplace_update_num_locks: 10000
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.memtable_prefix_bloom_size_ratio: 0.000000
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.memtable_whole_key_filtering: 0
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
Options.memtable_huge_page_size: 0
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
    Options.bloom_locality: 0
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
 Options.max_successive_merges: 0
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
 Options.optimize_filters_for_hits: 0
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
 Options.paranoid_file_checks: 0
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
 Options.force_consistency_checks: 0
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
 Options.report_bg_io_stats: 0
2020-05-05T20:19:56.345-0400 7fb999290ec0  4 rocksdb:
        Options.ttl: 0
2020-05-05T20:19:56.352-0400 7fb999290ec0  4 rocksdb:
[db/version_set.cc:3757] Recovered from manifest file:db/MANIFEST-004159
succeeded,manifest_file_number is 4159, next_file_number is 5076,
last_sequence is 236178180, log_number is 5073,prev_log_number is
0,max_column_family is 0,min_log_number_to_keep is 0

2020-05-05T20:19:56.352-0400 7fb999290ec0  4 rocksdb:
[db/version_set.cc:3766] Column family [default] (ID 0), log number is 5073

2020-05-05T20:19:56.352-0400 7fb999290ec0  4 rocksdb: EVENT_LOG_v1
{"time_micros": 1588724396353552, "job": 1, "event":
"recovery_started",
"log_files": [5071, 5073]}
2020-05-05T20:19:56.352-0400 7fb999290ec0  4 rocksdb:
[db/db_impl_open.cc:583] Recovering log #5071 mode 0
2020-05-05T20:19:57.061-0400 7fb999290ec0  4 rocksdb:
[db/db_impl_open.cc:583] Recovering log #5073 mode 0
2020-05-05T20:19:57.451-0400 7fb999290ec0  4 rocksdb: EVENT_LOG_v1
{"time_micros": 1588724397452521, "job": 1, "event":
"recovery_finished"}
2020-05-05T20:19:57.451-0400 7fb999290ec0  1
bluestore(/var/lib/ceph/osd/ceph-0) _open_db opened rocksdb path db options
compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2
2020-05-05T20:19:57.477-0400 7fb999290ec0  1 freelist init
2020-05-05T20:19:57.563-0400 7fb999290ec0  1
bluestore(/var/lib/ceph/osd/ceph-0) _open_alloc opening allocation metadata
teardown: managing teardown after SIGTERM
teardown: Sending SIGTERM to PID 51610
teardown: Waiting PID 51610 to terminate
./opt/ceph-container/bin/docker_exec.sh: line 32: 51610 Terminated
     "$@"

teardown: Process 51610 is terminated
ERRO[0006] Cannot get exit code: error getting exit file for container
bbf03528bb8e9b7bd09e4b4215aaa3b24f9f46479f6181100994d9c7be080986: stat
/var/run/libpod/exits/bbf03528bb8e9b7bd09e4b4215aaa3b24f9f46479f6181100994d9c7be080986-old:
no such file or directory

octapus-latest image displays the same behavior.

Any help or guidance would be appreciated.
/Chris

2024

2023

2022

2021

2020

2019

[ceph-users] osd won't start