April 2021 - ceph-users - lists.ceph.io

Single OSD crash/restarting during scrub operation on specific PG
by Mark Johnson 21 Apr '21

21 Apr '21

We've recently recovered from a bit of a disaster where we had some power outages (combination of data centre power maintenance and us not having our redundant power supplies connected to the correct redundant power circuits - lesson learnt). We ended up with one OSD that wouldn't start - seems like perhaps filesystem corruption as an FSCK found and fixed a couple of errors but the OSD still wouldn't start so we ended up marking it lost and letting data backfill to other OSDs. Ended up with a handful of 'incomplete' or 'incomplete/down' pgs which was causing radosgw to stop accepting connections. Found a useful blog post from somebody that got us to a point of using the ceph-objectstore-tool to determine the correct remaining copy and mark pgs as complete. Backfill operations then wrote out pgs to different OSDs and the cluster returned to a health OK state, and radosgw started working normally. At least, that's what I thought. Now, I'm occasionally seeing one OSD crashing every now and then, sometimes after a few hours, sometimes after only 10 minutes. It always starts itself up again and the queued up backfills cancel and the cluster returns to OK until the next time. It's always the same OSD and going through the logs just now, it seems to always occur when performing a scrub operation on the same pg (although, I haven't checked every single instance to be completely sure). We're running Jewel (yes, I know it's old but we can't upgrade). Here's the last couple of lines from the OSD log when it crashes on two different occasions - I've used the hex code from the "Caught signal" line to reference events that match that same code in both instances. Looks roughly the same on both occasions in that it's always the same pg, however the last object shown in the log prior to the crash always seems to be different. -91> 2021-04-21 02:37:32.118290 7fed046e6700 5 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: false, divergent_priors: 0, writeout_from: 3110'3174946, trimmed: -90> 2021-04-21 02:37:32.118380 7fed046e6700 5 -- op tracker -- seq: 2219, time: 2021-04-21 02:37:32.118379, event: commit_queued_for_journal_write, op: osd_repop(client.3095191.0:19172420 30.65 30:a78d321d:::71b98f4a-2ef6-466a-bb1d-eb477b317c78.604098.18432929_1149756827.ogg:head v 3110'3174946) -1> 2021-04-21 02:37:32.748831 7fed046e6700 5 -- op tracker -- seq: 2221, time: 2021-04-21 02:37:32.748830, event: reached_pg, op: replica scrub(pg: 30.65,from:0'0,to:2923'3171906,epoch:3110,start:30:a63a08df:::71b98f4a-2ef6-466a-bb1d-eb477b317c78.577500.22382834__shadow_.cqXthfu1litKEyNZ53I_voGLwuhonVX_1:0,end:30:a63a13fd:::71b98f4a-2ef6-466a-bb1d-eb477b317c78.604098.26065941_1234799607.gsm:0,chunky:1,deep:1,seed:4294967295,version:6) 0> 2021-04-21 02:37:32.797826 7fed046e6700 -1 os/filestore/FileStore.cc: In function 'int FileStore::lfn_find(const ghobject_t&, const Index&, IndexedPath*)' thread 7fed046e6700 time 2021-04-21 02:37:32.790356 2021-04-21 02:37:32.859265 7fed046e6700 -1 *** Caught signal (Aborted) ** in thread 7fed046e6700 thread_name:tp_osd_tp 0> 2021-04-21 02:37:32.859265 7fed046e6700 -1 *** Caught signal (Aborted) ** in thread 7fed046e6700 thread_name:tp_osd_tp -17> 2021-04-21 03:55:09.090430 7f43382c7700 5 -- op tracker -- seq: 1596, time: 2021-04-21 03:55:09.090430, event: done, op: replica scrub(pg: 30.65,from:0'0,to:2979'3174652,epoch:3122,start:30:a639eb4a:::71b98f4a-2ef6-466a-bb1d-eb477b317c78.577500.17157485_1132337117.ogg:0,end:30:a639f7f1:::71b98f4a-2ef6-466a-bb1d-eb477b317c78.604098.21768594_1188151257.ogg:0,chunky:1,deep:1,seed:4294967295,version:6) -5> 2021-04-21 03:55:09.777503 7f43382c7700 5 -- op tracker -- seq: 1598, time: 2021-04-21 03:55:09.777476, event: reached_pg, op: replica scrub(pg: 30.65,from:0'0,to:2929'3172006,epoch:3122,start:30:a63a047c:::71b98f4a-2ef6-466a-bb1d-eb477b317c78.577500.18649542__shadow_.tKOWzKIibnLhX3Bu32FiiuG0FH1lIl4_1:0,end:30:a63a0ea6:::71b98f4a-2ef6-466a-bb1d-eb477b317c78.577500.14411965_1101425097.gsm:0,chunky:1,deep:1,seed:4294967295,version:6) 0> 2021-04-21 03:55:10.089217 7f43382c7700 -1 os/filestore/FileStore.cc: In function 'int FileStore::lfn_find(const ghobject_t&, const Index&, IndexedPath*)' thread 7f43382c7700 time 2021-04-21 03:55:10.081373 2021-04-21 03:55:10.157208 7f43382c7700 -1 *** Caught signal (Aborted) ** in thread 7f43382c7700 thread_name:tp_osd_tp 0> 2021-04-21 03:55:10.157208 7f43382c7700 -1 *** Caught signal (Aborted) ** in thread 7f43382c7700 thread_name:tp_osd_tp Any ideas what to do next? Regards, Mark Johnson

1 0

Issues upgrading to 16.2.1
by Radoslav Milanov 20 Apr '21

20 Apr '21

Hello Tried cephadm upgrade form 16.2.0 to 16.2.1 Managers were updated first then process halted on first monitor being upgraded. The monitor fails to start: root@host3:/var/lib/ceph/c8ee2878-9d54-11eb-bbca-1c34da4b9fb6/mon.host3# /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph-mon --privileged --group-add=disk --init --name ceph-c8ee2878-9d54-11eb-bbca-1c34da4b9fb6-mon.host3 -e CONTAINER_IMAGE=ceph/ceph@sha256:9b04c0f15704c49591640a37c7adfd40ffad0a4b42fecb950c3407687cb4f29a -e NODE_NAME=host3 -e CEPH_USE_RANDOM_NONCE=1 -v /var/run/ceph/c8ee2878-9d54-11eb-bbca-1c34da4b9fb6:/var/run/ceph:z -v /var/log/ceph/c8ee2878-9d54-11eb-bbca-1c34da4b9fb6:/var/log/ceph:z -v /var/lib/ceph/c8ee2878-9d54-11eb-bbca-1c34da4b9fb6/crash:/var/lib/ceph/crash:z -v /var/lib/ceph/c8ee2878-9d54-11eb-bbca-1c34da4b9fb6/mon.host3:/var/lib/ceph/mon/ceph-host3:z -v /var/lib/ceph/c8ee2878-9d54-11eb-bbca-1c34da4b9fb6/mon.host3/config:/etc/ceph/ceph.conf:z -v /dev:/dev -v /run/udev:/run/udev ceph/ceph@sha256:9b04c0f15704c49591640a37c7adfd40ffad0a4b42fecb950c3407687cb4f29a -n mon.host3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true '--default-log-stderr-prefix=debug ' --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true debug 2021-04-20T14:06:19.437+0000 7f11e080a700 0 set uid:gid to 167:167 (ceph:ceph) debug 2021-04-20T14:06:19.437+0000 7f11e080a700 0 ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable), process ceph-mon, pid 7 debug 2021-04-20T14:06:19.437+0000 7f11e080a700 0 pidfile_write: ignore empty --pid-file debug 2021-04-20T14:06:19.441+0000 7f11e080a700 0 load: jerasure load: lrc load: isa debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: RocksDB version: 6.8.1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Compile date Mar 30 2021 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: DB SUMMARY debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: CURRENT file: CURRENT debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: IDENTITY file: IDENTITY debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: MANIFEST file: MANIFEST-000152 size: 221 Bytes debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-host3/store.db dir, Total Num: 2, files: 000137.sst 000139.sst debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-host3/store.db: 000153.log size: 0 ; debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.error_if_exists: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.create_if_missing: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.paranoid_checks: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.env: 0x5641244cf1c0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.fs: Posix File System debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.info_log: 0x564126753220 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.max_file_opening_threads: 16 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.statistics: (nil) debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.use_fsync: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.max_log_file_size: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.max_manifest_file_size: 1073741824 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.log_file_time_to_roll: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.keep_log_file_num: 1000 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.recycle_log_file_num: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.allow_fallocate: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.allow_mmap_reads: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.allow_mmap_writes: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.use_direct_reads: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.create_missing_column_families: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.db_log_dir: debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-host3/store.db debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.table_cache_numshardbits: 6 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.max_subcompactions: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.max_background_flushes: -1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.WAL_ttl_seconds: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.WAL_size_limit_MB: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.manifest_preallocation_size: 4194304 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.is_fd_close_on_exec: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.advise_random_on_open: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.db_write_buffer_size: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.write_buffer_manager: 0x5641267e9e90 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.access_hint_on_compaction_start: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.use_adaptive_mutex: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.rate_limiter: (nil) debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.wal_recovery_mode: 2 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.enable_thread_tracking: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.enable_pipelined_write: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.unordered_write: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.write_thread_max_yield_usec: 100 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.row_cache: None debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.wal_filter: None debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.avoid_flush_during_recovery: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.allow_ingest_behind: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.preserve_deletes: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.two_write_queues: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.manual_wal_flush: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.atomic_flush: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.persist_stats_to_disk: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.write_dbid_to_manifest: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.log_readahead_size: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.sst_file_checksum_func: Unknown debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.max_background_jobs: 2 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.max_background_compactions: -1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.delayed_write_rate : 16777216 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.max_total_wal_size: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.stats_dump_period_sec: 600 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.stats_persist_period_sec: 600 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.stats_history_buffer_size: 1048576 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.max_open_files: -1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.bytes_per_sync: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.wal_bytes_per_sync: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.strict_bytes_per_sync: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.compaction_readahead_size: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Compression algorithms supported: debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: kZSTDNotFinalCompression supported: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: kZSTD supported: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: kXpressCompression supported: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: kLZ4HCCompression supported: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: kLZ4Compression supported: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: kBZip2Compression supported: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: kZlibCompression supported: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: kSnappyCompression supported: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Fast CRC32 supported: Supported on x86 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-host3/store.db/MANIFEST-000152 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.merge_operator: debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.compaction_filter: None debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.compaction_filter_factory: None debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.memtable_factory: SkipListFactory debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.table_factory: BlockBasedTable debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5641266f60d8) cache_index_and_filter_blocks: 1 cache_index_and_filter_blocks_with_high_priority: 0 pin_l0_filter_and_index_blocks_in_cache: 0 pin_top_level_index_and_filter: 1 index_type: 0 data_block_index_type: 0 index_shortening: 1 data_block_hash_table_util_ratio: 0.750000 hash_index_allow_collision: 1 checksum: 1 no_block_cache: 0 block_cache: 0x56412672ef10 block_cache_name: BinnedLRUCache block_cache_options: capacity : 536870912 num_shard_bits : 4 strict_capacity_limit : 0 high_pri_pool_ratio: 0.000 block_cache_compressed: (nil) persistent_cache: (nil) block_size: 4096 block_size_deviation: 10 block_restart_interval: 16 index_block_restart_interval: 1 metadata_block_size: 4096 partition_filters: 0 use_delta_encoding: 1 filter_policy: rocksdb.BuiltinBloomFilter whole_key_filtering: 1 verify_compression: 0 read_amp_bytes_per_bit: 0 format_version: 2 enable_index_compression: 1 block_align: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.write_buffer_size: 33554432 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.max_write_buffer_number: 2 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.compression: NoCompression debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.bottommost_compression: Disabled debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.prefix_extractor: nullptr debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.num_levels: 7 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.bottommost_compression_opts.enabled: false debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.compression_opts.window_bits: -14 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.compression_opts.level: 32767 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.compression_opts.strategy: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.compression_opts.enabled: false debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.level0_stop_writes_trigger: 36 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.target_file_size_base: 67108864 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.target_file_size_multiplier: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.max_compaction_bytes: 1677721600 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.arena_block_size: 4194304 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.disable_auto_compactions: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.table_properties_collectors: debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.inplace_update_support: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.inplace_update_num_locks: 10000 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.memtable_whole_key_filtering: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.memtable_huge_page_size: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.bloom_locality: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.max_successive_merges: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.optimize_filters_for_hits: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.paranoid_file_checks: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.force_consistency_checks: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.report_bg_io_stats: 0 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.ttl: 2592000 debug 2021-04-20T14:06:19.441+0000 7f11e080a700 4 rocksdb: Options.periodic_compaction_seconds: 0 debug 2021-04-20T14:06:19.445+0000 7f11e080a700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-host3/store.db/MANIFEST-000152 succeeded,manifest_file_number is 152, next_file_number is 154, last_sequence is 73904, log_number is 151,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 debug 2021-04-20T14:06:19.445+0000 7f11e080a700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 151 debug 2021-04-20T14:06:19.445+0000 7f11e080a700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1618927579447606, "job": 1, "event": "recovery_started", "log_files": [153]} debug 2021-04-20T14:06:19.445+0000 7f11e080a700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #153 mode 2 debug 2021-04-20T14:06:19.445+0000 7f11e080a700 4 rocksdb: [version_set.cc:3826] Creating manifest 155 debug 2021-04-20T14:06:19.445+0000 7f11e080a700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1618927579449426, "job": 1, "event": "recovery_finished"} debug 2021-04-20T14:06:19.445+0000 7f11e080a700 4 rocksdb: DB pointer 0x5641267f3800 debug 2021-04-20T14:06:19.449+0000 7f11c7624700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- debug 2021-04-20T14:06:19.449+0000 7f11c7624700 4 rocksdb: [db_impl/db_impl.cc:851] ** DB Stats ** Uptime(secs): 0.0 total, 0.0 interval Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s Interval stall: 00:00:0.000 H:M:S, 0.0 percent ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 1/0 3.10 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 L6 1/0 38.24 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Sum 2/0 41.34 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Uptime(secs): 0.0 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [default] ** ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 1/0 3.10 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 L6 1/0 38.24 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Sum 2/0 41.34 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Uptime(secs): 0.0 total, 0.0 interval Flush(GB): cumulative 0.000, interval 0.000 AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [default] ** debug 2021-04-20T14:06:19.449+0000 7f11e080a700 0 starting mon.host3 rank 2 at public addrs [v2:108.61.24.82:3300/0,v1:108.61.24.82:6789/0] at bind addrs [v2:108.61.24.82:3300/0,v1:108.61.24.82:6789/0] mon_data /var/lib/ceph/mon/ceph-host3 fsid c8ee2878-9d54-11eb-bbca-1c34da4b9fb6 debug 2021-04-20T14:06:19.449+0000 7f11e080a700 1 mon.host3@-1(???) e4 preinit fsid c8ee2878-9d54-11eb-bbca-1c34da4b9fb6 debug 2021-04-20T14:06:19.449+0000 7f11e080a700 0 mon.host3(a)-1(???).mds e1 new map debug 2021-04-20T14:06:19.449+0000 7f11e080a700 0 mon.host3(a)-1(???).mds e1 print_map e1 enable_multiple, ever_enabled_multiple: 0,0 compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} legacy client fscid: -1 No filesystems configured debug 2021-04-20T14:06:19.449+0000 7f11e080a700 0 mon.host3(a)-1(???).osd e293 crush map has features 3314933000852226048, adjusting msgr requires debug 2021-04-20T14:06:19.449+0000 7f11e080a700 0 mon.host3(a)-1(???).osd e293 crush map has features 288514051259236352, adjusting msgr requires debug 2021-04-20T14:06:19.449+0000 7f11e080a700 0 mon.host3(a)-1(???).osd e293 crush map has features 288514051259236352, adjusting msgr requires debug 2021-04-20T14:06:19.449+0000 7f11e080a700 0 mon.host3(a)-1(???).osd e293 crush map has features 288514051259236352, adjusting msgr requires audit 2021-04-14T20:10:31.225267+0000 mon.host1 (mon.0) 6360 : audit [DBG] from='client.? 108.61.24.80:0/2572567093' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch debug 2021-04-20T14:06:19.449+0000 7f11e080a700 1 mon.host3(a)-1(???).paxosservice(auth 1..86) refresh upgraded, format 0 -> 3 debug 2021-04-20T14:06:19.457+0000 7f11e080a700 0 mon.host3@-1(probing) e4 my rank is now 2 (was -1) debug 2021-04-20T14:06:19.661+0000 7f11c9e29700 0 mon.host3@2(probing) e5 removed from monmap, suicide.

1 0

v14.2.20 Nautilus released
by David Galloway 20 Apr '21

20 Apr '21

This is the 20th bugfix release in the Nautilus stable series. It addresses a security vulnerability in the Ceph authentication framework. We recommend users to update to this release. For a detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/releases/v14-2-20-nautilus-released Security Fixes -------------- * This release includes a security fix that ensures the global_id value (a numeric value that should be unique for every authenticated client or daemon in the cluster) is reclaimed after a network disconnect or ticket renewal in a secure fashion. Two new health alerts may appear during the upgrade indicating that there are clients or daemons that are not yet patched with the appropriate fix. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-14.2.20.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: 36274af6eb7f2a5055f2d53ad448f2694e9046a0

4 4

BlueFS spillover detected (Nautilus 14.2.16)
by by morphin 20 Apr '21

20 Apr '21

Hello. I'm trying to fix a wrong cluster deployment (Nautilus 14.2.16) Cluster usage is %40 EC pool with RGW Every node has: 20 x OSD = TOSHIBA MG08SCA16TEY 16.0TB 2 x DB = NVME PM1725b 1.6TB (linux mdadm raid1) NVME usage always goes around %90-99. With "iostat -xdh 1" r/s w/s rkB/s wkB/s rrqm/s wrqm/s %rrqm %wrqm r_await w_await aqu-sz rareq-sz wareq-sz svctm %util Device 168.00 3619.00 7.2M 367.7M 0.00 90510.00 0.0% 96.2% 1.10 9.21 22.86 43.8k 104.0k 0.25 96.0% nvme0c0n1 19.00 3670.00 1.7M 373.5M 0.00 90510.00 0.0% 96.1% 0.26 29.61 95.99 89.7k 104.2k 0.27 98.0% nvme1c1n1 The problem is: BLUEFS_SPILLOVER BlueFS spillover detected on 120 OSD(s) osd.194 spilled over 42 GiB metadata from 'db' device (39 GiB used of 50 GiB) to slow device osd.195 spilled over 34 GiB metadata from 'db' device (40 GiB used of 50 GiB) to slow device osd.196 spilled over 28 GiB metadata from 'db' device (40 GiB used of 50 GiB) to slow device osd.197 spilled over 25 GiB metadata from 'db' device (41 GiB used of 50 GiB) to slow device osd.198 spilled over 30 GiB metadata from 'db' device (41 GiB used of 50 GiB) to slow device Block and wal size: bluestore_block_db_size = 53687091200 bluestore_block_wal_size = 0 nvme0n1 259:2 0 1.5T 0 disk └─md0 9:0 0 1.5T 0 raid1 ├─md0p1 259:4 0 50G 0 md ├─md0p2 259:5 0 50G 0 md +n └─md0p20 259:22 0 50G 0 md How can I change the level up to 500MB --> 5GB --> 50GB ?

2 4

v16.2.1 Pacific released
by David Galloway 20 Apr '21

20 Apr '21

This is the first bugfix release in the Pacific stable series. It addresses a security vulnerability in the Ceph authentication framework. We recommend users to update to this release. For a detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/releases/v16-2-1-pacific-released Security Fixes -------------- * This release includes a security fix that ensures the global_id value (a numeric value that should be unique for every authenticated client or daemon in the cluster) is reclaimed after a network disconnect or ticket renewal in a secure fashion. Two new health alerts may appear during the upgrade indicating that there are clients or daemons that are not yet patched with the appropriate fix. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-16.2.1.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: afb9061ab4117f798c858c741efa6390e48ccf10

2 1

v15.2.11 Octopus released
by David Galloway 20 Apr '21

20 Apr '21

This is the 11th bugfix release in the Octopus stable series. It addresses a security vulnerability in the Ceph authentication framework. We recommend users to update to this release. For a detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/releases/v15-2-11-octopus-released Security Fixes -------------- * This release includes a security fix that ensures the global_id value (a numeric value that should be unique for every authenticated client or daemon in the cluster) is reclaimed after a network disconnect or ticket renewal in a secure fashion. Two new health alerts may appear during the upgrade indicating that there are clients or daemons that are not yet patched with the appropriate fix. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-15.2.11.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: e3523634d9c2227df9af89a4eac33d16738c49cb

2 1

any experience on using Bcache on top of HDD OSD
by huxiaoyu＠horebdata.cn 20 Apr '21

20 Apr '21

Dear Cephers, Just curious about any one who has some experience on using Bcache on top of HDD OSD to accelerate IOPS performance? If any, how about the stability and the performance improvement, and for how long the running time? I am evaluating whether its is worth to use Bcache on top of HDD OSD. I have done some experimental on using bcache with OSD, and would like to make sure before doing it on a production cluster. thanks in advance, samuel huxiaoyu(a)horebdata.cn

3 4

Documentation of the LVM metadata format
by Nico Schottelius 19 Apr '21

19 Apr '21

Good morning, is there any documentation available regarding the meta data stored within LVM that ceph-volume manages / creates? My background is that ceph-volume activate does not work on non-systemd Linux distributions, but if I know how to recreate the tmpfs, we can easily start the osd without systemd. Any pointers in the right direction are appreciated. Best regards, Nico -- Sustainable and modern Infrastructures by ungleich.ch

3 5

cleanup multipart in radosgw
by Boris Behrens 19 Apr '21

19 Apr '21

Hi, is there a way to remove multipart uploads that are older than X days? It doesn't need to be build into ceph or is automated to the end. Just something I don't need to build on my own. I currently try to debug a problem where ceph reports a lot more used space than it actually requires ( https://www.mail-archive.com/ceph-users@ceph.io/msg09810.html) I came across a lot of old _multipart_ files in some buckets and now I want to clean them up. I don't know if this will fix my problem but I would love to rule that out. radosgw-admin bucket check --bucket=bucket --check-objects --fix does not work because it is a shareded bucket. I have also some buckets that look like this, and contain 100% _multipart_ files which are >2 years old: "buckets": [ { "bucket": "ncprod", "tenant": "", "num_objects": -482, "num_shards": 0, "objects_per_shard": -482, "fill_status": "OVER 180143985094819%" } ] -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

2 2

Radosgw - WARNING: couldn't find acl header for object, generating default
by by morphin 19 Apr '21

19 Apr '21

Hello. I've a RGW bucket (versioning=on). And there was objects like this: radosgw-admin object stat --bucket=xdir --object=f5492238-50cb-4bc2-93fa-424869018946 { "name": "f5492238-50cb-4bc2-93fa-424869018946", "size": 0, "tag": "", "attrs": { "user.rgw.manifest": "", "user.rgw.olh.idtag": "5rs3x0qh152tn0j865k8ybo9xqy92qjn", "user.rgw.olh.info": "\u0001\u0001�", "user.rgw.olh.pending.00000000607c87b5pgo03tvm3sqt23i9": "\u0001\u0001\u0008", "user.rgw.olh.pending.00000000607c87b5pyv13ugk3fadvxw7": "\u0001\u0001\u0008", "user.rgw.olh.pending.00000000607c87b5qic02n0e54zsjkax": "\u0001\u0001\u0008", "user.rgw.olh.ver": "3" } } I'm not sure but I suspect that these objects maybe leftover from unfinished multisite sync. I've removed the zone and made it master (because I have to for something else) and I've created a newbucket with (versioning=off) and copied all objects from old rgw bucket to a new rgw bucket with "rclone" cmd: "rclone copy --files-from "object.list" old:bucket new:bucket --no-traverse -vv --progress --fast-list --no-check-dest --no-update-modtime" Config: [bucket] type = s3 provider = Ceph env_auth = false acl = private bucket = access_key_id = secret_access_key = endpoint = After the copy I've checked every object via "radosgw-admin object stat object=$i" and there was still these problematic objects. I've tried to overwrite these objects from the Backup-bucket at different cluster (these objects are fine) and I tried to be sure everything written as expected via object stat again and these pending object was gone and everything was ok. All of them was OK! After a while I started to see this warning in radosgw.log when the client GET or HEAD the objects. 2021-04-19 11:37:50.230 7f2d917eb700 1 ====== starting new request req=0x55a44414a710 ===== 2021-04-19 11:37:50.230 7f2d917eb700 0 WARNING: couldn't find acl header for object, generating default 2021-04-19 11:37:50.230 7f2d917eb700 1 ====== req done req=0x55a44414a710 op status=0 http_status=200 latency=0s ====== 2021-04-19 11:37:50.230 7f2d917eb700 1 beast: 0x55a44414a710: 10.10.10.1 - - [2021-04-19 11:37:50.0.230489s] "HEAD /xdir/f5492238-50cb-4bc2-93fa-424869018946 HTTP/1.1" 200 0 - "aws-sdk-java/1.11.638 Linux/3.10.0-1160.11.1.el7.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.281-b09 java/1.8.0_281 groovy/2.5.6 vendor/Oracle_Corporation" - These objects are problematic objects and somehow the overwritten objects is gone and these objects left after the WARNING. (versioning is still =off --> multisite = off --> the cluster alone in zonegroup and its masterzone) But I've checked every overwritten object via "radosgw-admin object stat object=$i" and there was all OK!!! How the hell they're gone and others came back? HOW? What should I do? Maybe "object rm" "log trim" and re-write again?

1 0

Octopus - unbalanced OSDs
by Ml Ml 19 Apr '21

19 Apr '21

Hello List, any ideas why my OSDs are that unbalanced ? root@ceph01:~# ceph -s cluster: id: 5436dd5d-83d4-4dc8-a93b-60ab5db145df health: HEALTH_WARN 1 nearfull osd(s) 4 pool(s) nearfull services: mon: 3 daemons, quorum ceph03,ceph01,ceph02 (age 2w) mgr: ceph03(active, since 4M), standbys: ceph02.jwvivm mds: backup:1 {0=backup.ceph06.hdjehi=up:active} 3 up:standby osd: 56 osds: 56 up (since 29h), 56 in (since 3d) task status: scrub status: mds.backup.ceph06.hdjehi: idle data: pools: 4 pools, 1185 pgs objects: 24.29M objects, 44 TiB usage: 151 TiB used, 55 TiB / 206 TiB avail pgs: 675 active+clean 476 active+clean+snaptrim_wait 30 active+clean+snaptrim 4 active+clean+scrubbing+deep root@ceph01:~# ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 206.79979 - 206 TiB 151 TiB 151 TiB 36 GiB 503 GiB 55 TiB 73.23 1.00 - root default -2 28.89995 - 29 TiB 20 TiB 20 TiB 5.5 GiB 74 GiB 8.9 TiB 69.19 0.94 - host ceph01 0 hdd 2.70000 1.00000 2.7 TiB 1.8 TiB 1.8 TiB 590 MiB 6.9 GiB 908 GiB 66.81 0.91 44 up osd.0 1 hdd 2.70000 1.00000 2.7 TiB 1.6 TiB 1.6 TiB 411 MiB 6.5 GiB 1.1 TiB 60.43 0.83 39 up osd.1 4 hdd 2.70000 1.00000 2.7 TiB 1.8 TiB 1.8 TiB 501 MiB 6.8 GiB 898 GiB 67.15 0.92 43 up osd.4 8 hdd 2.70000 1.00000 2.7 TiB 2.0 TiB 2.0 TiB 453 MiB 7.0 GiB 700 GiB 74.39 1.02 47 up osd.8 11 hdd 1.70000 1.00000 1.7 TiB 1.3 TiB 1.3 TiB 356 MiB 5.6 GiB 433 GiB 75.39 1.03 31 up osd.11 12 hdd 2.70000 1.00000 2.7 TiB 2.1 TiB 2.1 TiB 471 MiB 7.0 GiB 591 GiB 78.40 1.07 48 up osd.12 14 hdd 2.70000 1.00000 2.7 TiB 1.6 TiB 1.6 TiB 448 MiB 6.0 GiB 1.1 TiB 59.68 0.82 38 up osd.14 18 hdd 2.70000 1.00000 2.7 TiB 1.7 TiB 1.7 TiB 515 MiB 6.2 GiB 980 GiB 64.15 0.88 41 up osd.18 22 hdd 1.70000 1.00000 1.7 TiB 1.2 TiB 1.2 TiB 360 MiB 4.2 GiB 491 GiB 72.06 0.98 29 up osd.22 30 hdd 1.70000 1.00000 1.7 TiB 1.2 TiB 1.2 TiB 366 MiB 4.7 GiB 558 GiB 68.26 0.93 28 up osd.30 33 hdd 1.59999 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 406 MiB 4.9 GiB 427 GiB 74.28 1.01 29 up osd.33 64 hdd 3.29999 1.00000 3.3 TiB 2.4 TiB 2.4 TiB 736 MiB 8.6 GiB 915 GiB 73.22 1.00 60 up osd.64 -3 29.69995 - 30 TiB 22 TiB 22 TiB 5.4 GiB 81 GiB 7.9 TiB 73.20 1.00 - host ceph02 2 hdd 1.70000 1.00000 1.7 TiB 1.3 TiB 1.2 TiB 402 MiB 5.2 GiB 476 GiB 72.93 1.00 30 up osd.2 3 hdd 2.70000 1.00000 2.7 TiB 2.0 TiB 2.0 TiB 653 MiB 7.8 GiB 652 GiB 76.15 1.04 49 up osd.3 7 hdd 2.70000 1.00000 2.7 TiB 2.5 TiB 2.5 TiB 456 MiB 7.7 GiB 209 GiB 92.36 1.26 56 up osd.7 9 hdd 2.70000 1.00000 2.7 TiB 1.9 TiB 1.9 TiB 434 MiB 7.2 GiB 781 GiB 71.46 0.98 46 up osd.9 13 hdd 2.39999 1.00000 2.4 TiB 1.6 TiB 1.6 TiB 451 MiB 6.1 GiB 823 GiB 66.28 0.91 38 up osd.13 16 hdd 2.70000 1.00000 2.7 TiB 1.6 TiB 1.6 TiB 375 MiB 6.4 GiB 1.1 TiB 59.84 0.82 39 up osd.16 19 hdd 1.70000 1.00000 1.7 TiB 1.1 TiB 1.1 TiB 323 MiB 4.7 GiB 601 GiB 65.80 0.90 27 up osd.19 23 hdd 2.70000 1.00000 2.7 TiB 2.2 TiB 2.2 TiB 471 MiB 7.7 GiB 520 GiB 80.99 1.11 50 up osd.23 24 hdd 1.70000 1.00000 1.7 TiB 1.4 TiB 1.4 TiB 371 MiB 5.5 GiB 273 GiB 84.44 1.15 32 up osd.24 28 hdd 2.70000 1.00000 2.7 TiB 1.9 TiB 1.9 TiB 428 MiB 7.4 GiB 818 GiB 70.07 0.96 44 up osd.28 31 hdd 2.70000 1.00000 2.7 TiB 2.0 TiB 2.0 TiB 516 MiB 7.4 GiB 660 GiB 75.85 1.04 48 up osd.31 32 hdd 3.29999 1.00000 3.3 TiB 2.2 TiB 2.2 TiB 661 MiB 7.9 GiB 1.2 TiB 64.86 0.89 52 up osd.32 -4 26.29996 - 26 TiB 18 TiB 18 TiB 4.3 GiB 73 GiB 8.0 TiB 69.58 0.95 - host ceph03 5 hdd 1.70000 1.00000 1.7 TiB 1.2 TiB 1.2 TiB 298 MiB 5.2 GiB 541 GiB 69.21 0.95 29 up osd.5 6 hdd 1.70000 1.00000 1.7 TiB 1.0 TiB 1.0 TiB 321 MiB 4.4 GiB 697 GiB 60.34 0.82 25 up osd.6 10 hdd 2.70000 1.00000 2.7 TiB 1.9 TiB 1.9 TiB 431 MiB 7.5 GiB 796 GiB 70.89 0.97 46 up osd.10 15 hdd 2.70000 1.00000 2.7 TiB 1.9 TiB 1.9 TiB 500 MiB 6.6 GiB 805 GiB 70.55 0.96 44 up osd.15 17 hdd 1.59999 1.00000 1.6 TiB 1.1 TiB 1.1 TiB 377 MiB 4.9 GiB 530 GiB 68.05 0.93 27 up osd.17 20 hdd 1.70000 1.00000 1.7 TiB 1.0 TiB 1.0 TiB 223 MiB 4.7 GiB 685 GiB 61.03 0.83 25 up osd.20 21 hdd 2.70000 1.00000 2.7 TiB 1.7 TiB 1.7 TiB 392 MiB 6.7 GiB 951 GiB 65.23 0.89 42 up osd.21 25 hdd 1.70000 1.00000 1.7 TiB 1.1 TiB 1.1 TiB 157 MiB 5.1 GiB 601 GiB 65.83 0.90 27 up osd.25 26 hdd 2.70000 1.00000 2.7 TiB 2.1 TiB 2.1 TiB 512 MiB 7.6 GiB 573 GiB 79.06 1.08 50 up osd.26 27 hdd 2.70000 1.00000 2.7 TiB 1.9 TiB 1.9 TiB 473 MiB 7.6 GiB 805 GiB 70.55 0.96 46 up osd.27 29 hdd 2.70000 1.00000 2.7 TiB 2.1 TiB 2.1 TiB 478 MiB 7.3 GiB 539 GiB 80.29 1.10 50 up osd.29 63 hdd 1.70000 1.00000 1.7 TiB 1.1 TiB 1.1 TiB 195 MiB 5.1 GiB 646 GiB 63.23 0.86 26 up osd.63 -11 24.79999 - 25 TiB 18 TiB 18 TiB 4.1 GiB 59 GiB 6.3 TiB 74.51 1.02 - host ceph04 34 hdd 5.20000 1.00000 5.2 TiB 3.9 TiB 3.8 TiB 954 MiB 13 GiB 1.4 TiB 73.48 1.00 94 up osd.34 42 hdd 5.20000 1.00000 5.2 TiB 3.9 TiB 3.8 TiB 841 MiB 13 GiB 1.4 TiB 73.43 1.00 94 up osd.42 44 hdd 7.20000 1.00000 7.2 TiB 5.5 TiB 5.5 TiB 1.2 GiB 17 GiB 1.6 TiB 77.54 1.06 133 up osd.44 45 hdd 7.20000 1.00000 7.2 TiB 5.2 TiB 5.2 TiB 1.2 GiB 16 GiB 1.9 TiB 73.03 1.00 125 up osd.45 -13 30.09998 - 30 TiB 22 TiB 22 TiB 5.1 GiB 72 GiB 8.0 TiB 73.48 1.00 - host ceph05 39 hdd 7.20000 1.00000 7.2 TiB 5.6 TiB 5.6 TiB 1.4 GiB 17 GiB 1.6 TiB 77.89 1.06 126 up osd.39 40 hdd 7.20000 1.00000 7.2 TiB 5.3 TiB 5.3 TiB 1.2 GiB 17 GiB 1.9 TiB 73.87 1.01 124 up osd.40 41 hdd 7.20000 1.00000 7.2 TiB 5.4 TiB 5.3 TiB 1.1 GiB 17 GiB 1.8 TiB 74.92 1.02 128 up osd.41 43 hdd 5.20000 1.00000 5.2 TiB 3.7 TiB 3.7 TiB 853 MiB 13 GiB 1.5 TiB 71.28 0.97 91 up osd.43 60 hdd 3.29999 1.00000 3.3 TiB 2.1 TiB 2.1 TiB 573 MiB 7.9 GiB 1.2 TiB 63.53 0.87 52 up osd.60 -9 17.59999 - 18 TiB 12 TiB 12 TiB 3.0 GiB 40 GiB 5.3 TiB 70.10 0.96 - host ceph06 35 hdd 7.20000 1.00000 7.2 TiB 5.2 TiB 5.2 TiB 1.3 GiB 16 GiB 1.9 TiB 72.80 0.99 125 up osd.35 36 hdd 5.20000 1.00000 5.2 TiB 3.5 TiB 3.5 TiB 804 MiB 12 GiB 1.8 TiB 66.50 0.91 85 up osd.36 38 hdd 5.20000 1.00000 5.2 TiB 3.7 TiB 3.7 TiB 978 MiB 12 GiB 1.6 TiB 70.02 0.96 88 up osd.38 -15 24.89998 - 25 TiB 18 TiB 18 TiB 4.3 GiB 58 GiB 6.5 TiB 73.75 1.01 - host ceph07 66 hdd 7.20000 1.00000 7.2 TiB 5.3 TiB 5.3 TiB 1.1 GiB 17 GiB 1.8 TiB 74.74 1.02 126 up osd.66 67 hdd 7.20000 1.00000 7.2 TiB 5.3 TiB 5.3 TiB 1.2 GiB 17 GiB 1.8 TiB 74.51 1.02 121 up osd.67 68 hdd 3.29999 1.00000 3.3 TiB 2.3 TiB 2.3 TiB 720 MiB 7.9 GiB 1.0 TiB 68.63 0.94 55 up osd.68 69 hdd 7.20000 1.00000 7.2 TiB 5.3 TiB 5.3 TiB 1.2 GiB 17 GiB 1.8 TiB 74.40 1.02 129 up osd.69 -17 24.50000 - 24 TiB 20 TiB 20 TiB 4.1 GiB 47 GiB 4.4 TiB 82.08 1.12 - host ceph08 37 hdd 9.50000 1.00000 9.5 TiB 7.8 TiB 7.7 TiB 1.5 GiB 18 GiB 1.8 TiB 81.39 1.11 166 up osd.37 46 hdd 5.00000 1.00000 5.0 TiB 4.0 TiB 3.9 TiB 889 MiB 9.3 GiB 1.0 TiB 79.67 1.09 87 up osd.46 47 hdd 5.00000 1.00000 5.0 TiB 4.2 TiB 4.2 TiB 863 MiB 9.7 GiB 817 GiB 83.90 1.15 90 up osd.47 48 hdd 5.00000 1.00000 5.0 TiB 4.2 TiB 4.2 TiB 969 MiB 10 GiB 813 GiB 83.99 1.15 91 up osd.48 TOTAL 206 TiB 151 TiB 151 TiB 36 GiB 503 GiB 55 TiB 73.23 MIN/MAX VAR: 0.82/1.26 STDDEV: 7.00 root@ceph01:~# ceph balancer status { "active": true, "last_optimize_duration": "0:00:00.016174", "last_optimize_started": "Fri Apr 16 12:54:47 2021", "mode": "upmap", "optimize_result": "Unable to find further optimization, or pool(s) pg_num is decreasing, or distribution is already perfect", "plans": [] } root@ceph01:~# ceph versions { "mon": { "ceph version 15.2.5 (2c93eff00150f0cc5f106a559557a58d3d7b6f1f) octopus (stable)": 3 }, "mgr": { "ceph version 15.2.5 (2c93eff00150f0cc5f106a559557a58d3d7b6f1f) octopus (stable)": 2 }, "osd": { "ceph version 15.2.5 (2c93eff00150f0cc5f106a559557a58d3d7b6f1f) octopus (stable)": 56 }, "mds": { "ceph version 15.2.5 (2c93eff00150f0cc5f106a559557a58d3d7b6f1f) octopus (stable)": 4 }, "overall": { "ceph version 15.2.5 (2c93eff00150f0cc5f106a559557a58d3d7b6f1f) octopus (stable)": 65 } } root@ceph01:~# ceph osd crush rule ls replicated_ruleset root@ceph01:~# ceph osd crush rule dump replicated_ruleset { "rule_id": 0, "rule_name": "replicated_ruleset", "ruleset": 0, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] } Cheers, Michael

2 2

Abandon incomplete (damaged EC) pgs - How to manage the impact on cephfs?
by Joshua West 18 Apr '21

18 Apr '21

Hey everyone. Inside of cephfs, I have a directory which I setup a directory layout field to use an erasure coded (CLAY) pool, specific to the task. The rest of my cephfs is using normal replication. Fast forward some time, and the EC directory has been used pretty extensively, and through some bad luck and poor timing, ~200pgs are in an incomplete state, and the OSDs are completely gone and unrecoverable. (Specifically OSD 31 and 34, not that it matters at this point) # ceph pg ls incomplete --> is attached for reference. Fortunately, it's primarily (only) my on-site backups, and other replaceable data inside of I tried for a few days to recover the PGs: - Recreate blank OSDs with correct ID (was blocked by non-existant OSDs) - Deep Scrub - osd_find_best_info_ignore_history_les = true (`pg query` was showing related error) etc. I've finally just accepted this pool to be a lesson learned, and want to get the rest of my cephfs back to normal. My questions: -- `ceph osd force-create-pg` doesn't appear to fix pgs, even for pgs with 0 objects -- Deleting the pool seems like an appropriate step, but as I am using an xattr within cephfs, which is otherwise on another pool, I am not confident that this approach is safe? -- cephfs currently blocks when attemping to impact every third file in the EC directory. Once I delete the pool, how will I remove the files if even `rm` is blocking? Thank you for your time, Joshua West

3 8

Can't get one OSD (out of 14) to start
by Mark Johnson 17 Apr '21

17 Apr '21

Really not sure where to go with this one. Firstly, a description of my cluster. Yes, I know there are a lot of "not ideals" here but this is what I inherited. The cluster is running Jewel and has two storage/mon nodes and an additional mon only node, with a pool size of 2. Today, we had a some power issues in the data centre and we very ungracefully lost both storage servers at the same time. Node 1 came back online before node 2 but I could see there were a few OSDs that were down. When node 2 came back, I started trying to get OSDs up. Each node has 14 OSDs and I managed to get all OSDs up and in on node 2, but one of the OSDs on node 1 keeps starting and crashing and just won't stay up. I'm not finding the OSD log output to be much use. Current health status looks like this: # ceph health HEALTH_ERR 26 pgs are stuck inactive for more than 300 seconds; 26 pgs down; 26 pgs peering; 26 pgs stuck inactive; 26 pgs stuck unclean; 5 requests are blocked > 32 sec # ceph status cluster e2391bbf-15e0-405f-af12-943610cb4909 health HEALTH_ERR 26 pgs are stuck inactive for more than 300 seconds 26 pgs down 26 pgs peering 26 pgs stuck inactive 26 pgs stuck unclean 5 requests are blocked > 32 sec Any clues as to what I should be looking for or what sort of action I should be taking to troubleshoot this? Unfortunately, I'm a complete novice with Ceph. Here's a snippet from the OSD log that means little to me... --- begin dump of recent events --- 0> 2021-04-16 12:25:10.169340 7f2e23921ac0 -1 *** Caught signal (Aborted) ** in thread 7f2e23921ac0 thread_name:ceph-osd ceph version 10.2.11 (e4b061b47f07f583c92a050d9e84b1813a35671e) 1: (()+0x9f1c2a) [0x7f2e24330c2a] 2: (()+0xf5d0) [0x7f2e21ee95d0] 3: (gsignal()+0x37) [0x7f2e2049f207] 4: (abort()+0x148) [0x7f2e204a08f8] 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x267) [0x7f2e2442fd47] 6: (FileJournal::read_entry(ceph::buffer::list&, unsigned long&, bool*)+0x90c) [0x7f2e2417bc7c] 7: (JournalingObjectStore::journal_replay(unsigned long)+0x1ee) [0x7f2e240c8dce] 8: (FileStore::mount()+0x3cd6) [0x7f2e240a0546] 9: (OSD::init()+0x27d) [0x7f2e23d5828d] 10: (main()+0x2c18) [0x7f2e23c71088] 11: (__libc_start_main()+0xf5) [0x7f2e2048b3d5] 12: (()+0x3c8847) [0x7f2e23d07847] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Thanks in advance, Mark

2 5

what-does-nosuchkey-error-mean-while-subscribing-for-notification-in-ceph
by Szabo, Istvan (Agoda) 16 Apr '21

16 Apr '21

Hi, I am trying to follow this url https://docs.ceph.com/en/latest/radosgw/s3/bucketops/#create-notification to create a publisher for my bucket into a topic. My curl: curl -v -H 'Date: Fri, 16 Apr 2021 05:21:14 +0000' -H 'Authorization: AWS accessid:secretkey' -L -H 'content-type: text/xml' -H 'Content-MD5: pBRX39Oo7aAUYbilIYMoAw==' -T notif.xml http://ceph:8080/vig-test?notification and it returns me this error <?xml version="1.0" encoding="UTF-8"?> <Error> <Code>NoSuchKey</Code> <BucketName>vig-test</BucketName> <RequestId>tx0000000000000016ac570-0060791ecb-1c7e96b-hkg</RequestId> <HostId>1c7e96b-hkg-data</HostId> </Error> Does anybody know what does this error mean in Ceph? How can I proceed? Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

3 3

ceph/ceph-grafana docker image for arm64 missing
by mabi 16 Apr '21

16 Apr '21

Hello, I want to deploy a new ceph Octopus cluster using cephadm on arm64 architecture but unfortunately the ceph/ceph-grafana docker image for arm64 is missing. Is this mailing list the right place to report this? or where should I report that? Best regards, Mabi

1 0

ceph-iscsi issue after upgrading from nautilus to octopus
by icy chan 16 Apr '21

16 Apr '21

Hi, I had several clusters running as nautilus and pending upgrading to octopus. I am now testing the upgrade steps for ceph cluster from nautilus to octopus using cephadm adopt in lab referred to below link: - https://docs.ceph.com/en/octopus/cephadm/adoption/ Lab environment: 3 all-in-one nodes. OS: CentOS 7.9.2009 with podman 1.6.4. After the adoption, ceph health keep warns about tcme-runner not managed by cephadm. # ceph health detail HEALTH_WARN 12 stray daemon(s) not managed by cephadm; 1 pool(s) have no replicas configured [WRN] CEPHADM_STRAY_DAEMON: 12 stray daemon(s) not managed by cephadm stray daemon tcmu-runner.ceph-aio1:iSCSI/iscsi_image_01 on host ceph-aio1 not managed by cephadm stray daemon tcmu-runner.ceph-aio1:iSCSI/iscsi_image_02 on host ceph-aio1 not managed by cephadm stray daemon tcmu-runner.ceph-aio1:iSCSI/iscsi_image_03 on host ceph-aio1 not managed by cephadm stray daemon tcmu-runner.ceph-aio1:iSCSI/iscsi_image_test on host ceph-aio1 not managed by cephadm stray daemon tcmu-runner.ceph-aio2:iSCSI/iscsi_image_01 on host ceph-aio2 not managed by cephadm stray daemon tcmu-runner.ceph-aio2:iSCSI/iscsi_image_02 on host ceph-aio2 not managed by cephadm stray daemon tcmu-runner.ceph-aio2:iSCSI/iscsi_image_03 on host ceph-aio2 not managed by cephadm stray daemon tcmu-runner.ceph-aio2:iSCSI/iscsi_image_test on host ceph-aio2 not managed by cephadm stray daemon tcmu-runner.ceph-aio3:iSCSI/iscsi_image_01 on host ceph-aio3 not managed by cephadm stray daemon tcmu-runner.ceph-aio3:iSCSI/iscsi_image_02 on host ceph-aio3 not managed by cephadm stray daemon tcmu-runner.ceph-aio3:iSCSI/iscsi_image_03 on host ceph-aio3 not managed by cephadm stray daemon tcmu-runner.ceph-aio3:iSCSI/iscsi_image_test on host ceph-aio3 not managed by cephadm And tcmu-runner is still running with the old version. # ceph versions { "mon": { "ceph version 15.2.10 (27917a557cca91e4da407489bbaa64ad4352cc02) octopus (stable)": 3 }, "mgr": { "ceph version 15.2.10 (27917a557cca91e4da407489bbaa64ad4352cc02) octopus (stable)": 1 }, "osd": { "ceph version 15.2.10 (27917a557cca91e4da407489bbaa64ad4352cc02) octopus (stable)": 9 }, "mds": {}, "tcmu-runner": { "ceph version 14.2.18 (befbc92f3c11eedd8626487211d200c0b44786d9) nautilus (stable)": 12 }, "overall": { "ceph version 14.2.18 (befbc92f3c11eedd8626487211d200c0b44786d9) nautilus (stable)": 12, "ceph version 15.2.10 (27917a557cca91e4da407489bbaa64ad4352cc02) octopus (stable)": 13 } } I didn't find any ceph-iscsi related upgrade steps from the above reference link. Can anyone here point me to the right direction of ceph-iscsi version upgrade? Thanks. Regs, Icy

1 0

Fresh install of Ceph using Ansible
by Jared Jacob 15 Apr '21

15 Apr '21

I am looking to rebuild my ceph cluster using ansible.What is the best way to start this process?

2 1

How to handle bluestore fragmentation
by David Caro 15 Apr '21

15 Apr '21

Reading the thread "s3 requires twice the space it should use", Boris pointed out that the fragmentation for the osds is around 0.8-0.9: > On Thu, Apr 15, 2021 at 8:06 PM Boris Behrens <bb(a)kervyn.de> wrote: >> I also checked the fragmentation on the bluestore OSDs and it is around >> 0.80 - 0.89 on most OSDs. yikes. >> [root@s3db1 ~]# ceph daemon osd.23 bluestore allocator score block >> { >> "fragmentation_rating": 0.85906054329923576 >> } And that made me wonder what is the current recommended (and not recommended) way to handle and reduce the fragmentation of the existing OSDs. Reading around I would think of tweaking the min_alloc_size_{ssd,hdd} and redeploying those OSDs, but I was unable to find much else, I wonder what do people do? ps. There was another thread that got no replies asking something similar (and a bunch of other things): https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/3PITWZRNX7…

1 0

DocuBetter Meeting This Week -- 1630 UTC
by John Zachary Dover 15 Apr '21

15 Apr '21

This week's meeting will focus on the ongoing rewrite of the cephadm documentation and the upcoming Google Season of Docs project. Meeting: https://bluejeans.com/908675367 Etherpad: https://pad.ceph.com/p/Ceph_Documentation

2 1

Re: s3 requires twice the space it should use
by Boris Behrens 15 Apr '21

15 Apr '21

So, I need to live with it? A value of zero leads to use the default? [root@s3db1 ~]# ceph daemon osd.23 config get bluestore_min_alloc_size { "bluestore_min_alloc_size": "0" } I also checked the fragmentation on the bluestore OSDs and it is around 0.80 - 0.89 on most OSDs. yikes. [root@s3db1 ~]# ceph daemon osd.23 bluestore allocator score block { "fragmentation_rating": 0.85906054329923576 } The problem I currently have is, that I barely keep up with adding OSD disks. Am Do., 15. Apr. 2021 um 16:18 Uhr schrieb Amit Ghadge <amitg.b14(a)gmail.com >: > size_kb_actual are actually bucket object size but on OSD level the > bluestore_min_alloc_size default 64KB and SSD are 16KB > > > https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/3/html/a… > > -AmitG > > On Thu, Apr 15, 2021 at 7:29 PM Boris Behrens <bb(a)kervyn.de> wrote: > >> Hi, >> >> maybe it is just a problem in my understanding, but it looks like our s3 >> requires twice the space it should use. >> >> I ran "radosgw-admin bucket stats", and added all "size_kb_actual" values >> up and divided to TB (/1024/1024/1024). >> The resulting space is 135,1636733 TB. When I tripple it because of >> replication I end up with around 405TB which is nearly half the space of >> what ceph df tells me. >> >> Hope someone can help me. >> >> ceph df shows >> RAW STORAGE: >> CLASS SIZE AVAIL USED RAW USED %RAW USED >> hdd 1009 TiB 189 TiB 820 TiB 820 TiB 81.26 >> TOTAL 1009 TiB 189 TiB 820 TiB 820 TiB 81.26 >> >> POOLS: >> POOL ID PGS STORED >> OBJECTS >> USED %USED MAX AVAIL >> rbd 0 64 0 B >> 0 >> 0 B 0 18 TiB >> .rgw.root 1 64 99 KiB >> 119 >> 99 KiB 0 18 TiB >> eu-central-1.rgw.control 2 64 0 B >> 8 >> 0 B 0 18 TiB >> eu-central-1.rgw.data.root 3 64 1.0 MiB >> 3.15k >> 1.0 MiB 0 18 TiB >> eu-central-1.rgw.gc 4 64 71 MiB >> 32 >> 71 MiB 0 18 TiB >> eu-central-1.rgw.log 5 64 267 MiB >> 564 >> 267 MiB 0 18 TiB >> eu-central-1.rgw.users.uid 6 64 2.8 MiB >> 6.91k >> 2.8 MiB 0 18 TiB >> eu-central-1.rgw.users.keys 7 64 263 KiB >> 6.73k >> 263 KiB 0 18 TiB >> eu-central-1.rgw.meta 8 64 384 KiB >> 1k >> 384 KiB 0 18 TiB >> eu-central-1.rgw.users.email 9 64 40 B >> 1 >> 40 B 0 18 TiB >> eu-central-1.rgw.buckets.index 10 64 10 GiB >> 67.61k >> 10 GiB 0.02 18 TiB >> eu-central-1.rgw.buckets.data 11 2048 264 TiB >> 138.31M >> 264 TiB 83.37 18 TiB >> eu-central-1.rgw.buckets.non-ec 12 64 297 MiB >> 11.32k >> 297 MiB 0 18 TiB >> eu-central-1.rgw.usage 13 64 536 MiB >> 32 >> 536 MiB 0 18 TiB >> eu-msg-1.rgw.control 56 64 0 B >> 8 >> 0 B 0 18 TiB >> eu-msg-1.rgw.data.root 57 64 72 KiB >> 227 >> 72 KiB 0 18 TiB >> eu-msg-1.rgw.gc 58 64 300 KiB >> 32 >> 300 KiB 0 18 TiB >> eu-msg-1.rgw.log 59 64 835 KiB >> 242 >> 835 KiB 0 18 TiB >> eu-msg-1.rgw.users.uid 60 64 56 KiB >> 104 >> 56 KiB 0 18 TiB >> eu-msg-1.rgw.usage 61 64 37 MiB >> 25 >> 37 MiB 0 18 TiB >> eu-msg-1.rgw.users.keys 62 64 3.8 KiB >> 97 >> 3.8 KiB 0 18 TiB >> eu-msg-1.rgw.meta 63 64 607 KiB >> 1.60k >> 607 KiB 0 18 TiB >> eu-msg-1.rgw.buckets.index 64 64 71 MiB >> 119 >> 71 MiB 0 18 TiB >> eu-msg-1.rgw.users.email 65 64 0 B >> 0 >> 0 B 0 18 TiB >> eu-msg-1.rgw.buckets.data 66 64 2.9 TiB >> 1.16M >> 2.9 TiB 5.30 18 TiB >> eu-msg-1.rgw.buckets.non-ec 67 64 2.2 MiB >> 354 >> 2.2 MiB 0 18 TiB >> default.rgw.control 69 32 0 B >> 8 >> 0 B 0 18 TiB >> default.rgw.data.root 70 32 0 B >> 0 >> 0 B 0 18 TiB >> default.rgw.gc 71 32 0 B >> 0 >> 0 B 0 18 TiB >> default.rgw.log 72 32 0 B >> 0 >> 0 B 0 18 TiB >> default.rgw.users.uid 73 32 0 B >> 0 >> 0 B 0 18 TiB >> fra-1.rgw.control 74 32 0 B >> 8 >> 0 B 0 18 TiB >> fra-1.rgw.meta 75 32 0 B >> 0 >> 0 B 0 18 TiB >> fra-1.rgw.log 76 32 50 B >> 28 >> 50 B 0 18 TiB >> >> >> -- >> Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im >> groÃƒ¼en Saal. >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> > -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

1 0

s3 requires twice the space it should use
by Boris Behrens 15 Apr '21

15 Apr '21

Hi, maybe it is just a problem in my understanding, but it looks like our s3 requires twice the space it should use. I ran "radosgw-admin bucket stats", and added all "size_kb_actual" values up and divided to TB (/1024/1024/1024). The resulting space is 135,1636733 TB. When I tripple it because of replication I end up with around 405TB which is nearly half the space of what ceph df tells me. Hope someone can help me. ceph df shows RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 1009 TiB 189 TiB 820 TiB 820 TiB 81.26 TOTAL 1009 TiB 189 TiB 820 TiB 820 TiB 81.26 POOLS: POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL rbd 0 64 0 B 0 0 B 0 18 TiB .rgw.root 1 64 99 KiB 119 99 KiB 0 18 TiB eu-central-1.rgw.control 2 64 0 B 8 0 B 0 18 TiB eu-central-1.rgw.data.root 3 64 1.0 MiB 3.15k 1.0 MiB 0 18 TiB eu-central-1.rgw.gc 4 64 71 MiB 32 71 MiB 0 18 TiB eu-central-1.rgw.log 5 64 267 MiB 564 267 MiB 0 18 TiB eu-central-1.rgw.users.uid 6 64 2.8 MiB 6.91k 2.8 MiB 0 18 TiB eu-central-1.rgw.users.keys 7 64 263 KiB 6.73k 263 KiB 0 18 TiB eu-central-1.rgw.meta 8 64 384 KiB 1k 384 KiB 0 18 TiB eu-central-1.rgw.users.email 9 64 40 B 1 40 B 0 18 TiB eu-central-1.rgw.buckets.index 10 64 10 GiB 67.61k 10 GiB 0.02 18 TiB eu-central-1.rgw.buckets.data 11 2048 264 TiB 138.31M 264 TiB 83.37 18 TiB eu-central-1.rgw.buckets.non-ec 12 64 297 MiB 11.32k 297 MiB 0 18 TiB eu-central-1.rgw.usage 13 64 536 MiB 32 536 MiB 0 18 TiB eu-msg-1.rgw.control 56 64 0 B 8 0 B 0 18 TiB eu-msg-1.rgw.data.root 57 64 72 KiB 227 72 KiB 0 18 TiB eu-msg-1.rgw.gc 58 64 300 KiB 32 300 KiB 0 18 TiB eu-msg-1.rgw.log 59 64 835 KiB 242 835 KiB 0 18 TiB eu-msg-1.rgw.users.uid 60 64 56 KiB 104 56 KiB 0 18 TiB eu-msg-1.rgw.usage 61 64 37 MiB 25 37 MiB 0 18 TiB eu-msg-1.rgw.users.keys 62 64 3.8 KiB 97 3.8 KiB 0 18 TiB eu-msg-1.rgw.meta 63 64 607 KiB 1.60k 607 KiB 0 18 TiB eu-msg-1.rgw.buckets.index 64 64 71 MiB 119 71 MiB 0 18 TiB eu-msg-1.rgw.users.email 65 64 0 B 0 0 B 0 18 TiB eu-msg-1.rgw.buckets.data 66 64 2.9 TiB 1.16M 2.9 TiB 5.30 18 TiB eu-msg-1.rgw.buckets.non-ec 67 64 2.2 MiB 354 2.2 MiB 0 18 TiB default.rgw.control 69 32 0 B 8 0 B 0 18 TiB default.rgw.data.root 70 32 0 B 0 0 B 0 18 TiB default.rgw.gc 71 32 0 B 0 0 B 0 18 TiB default.rgw.log 72 32 0 B 0 0 B 0 18 TiB default.rgw.users.uid 73 32 0 B 0 0 B 0 18 TiB fra-1.rgw.control 74 32 0 B 8 0 B 0 18 TiB fra-1.rgw.meta 75 32 0 B 0 0 B 0 18 TiB fra-1.rgw.log 76 32 50 B 28 50 B 0 18 TiB -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

1 0

Cephadm upgrade to Pacific problem
by Radoslav Milanov 15 Apr '21

15 Apr '21

Hello, Cluster is 3 nodes Debian 10. Started cephadm upgrade on healthy 15.2.10 cluster. Managers were upgraded fine then first monitor went down for upgrade and never came back. Researching at the unit files container fails to run because of an error: root@host1:/var/lib/ceph/97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6/mon.host1# cat unit.run set -e /usr/bin/install -d -m0770 -o 167 -g 167 /var/run/ceph/97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6 # mon.host1 ! /usr/bin/docker rm -f ceph-97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6-mon.host1 2> /dev/null /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph-mon --privileged --group-add=disk --init --name ceph-97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6-mon.host1 -e CONTAINER_IMAGE=ceph/ceph@sha256:9b04c0f15704c49591640a37c7adfd40ffad0a4b42fecb950c3407687cb4f29a -e NODE_NAME=host1 -e CEPH_USE_RANDOM_NONCE=1 -v /var/run/ceph/97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6:/var/run/ceph:z -v /var/log/ceph/97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6:/var/log/ceph:z -v /var/lib/ceph/97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6/crash:/var/lib/ceph/crash:z -v /var/lib/ceph/97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6/mon.host1:/var/lib/ceph/mon/ceph-host1:z -v /var/lib/ceph/97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6/mon.host1/config:/etc/ceph/ceph.conf:z -v /dev:/dev -v /run/udev:/run/udev ceph/ceph@sha256:9b04c0f15704c49591640a37c7adfd40ffad0a4b42fecb950c3407687cb4f29a -n mon.host1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true '--default-log-stderr-prefix=debug ' --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true root@host1:/var/lib/ceph/97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6/mon.host1# /usr/bin/docker run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph-mon --privileged --group-add=disk --init --name ceph-97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6-mon.host1 -e CONTAINER_IMAGE=ceph/ceph@sha256:9b04c0f15704c49591640a37c7adfd40ffad0a4b42fecb950c3407687cb4f29a -e NODE_NAME=host1 -e CEPH_USE_RANDOM_NONCE=1 -v /var/run/ceph/97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6:/var/run/ceph:z -v /var/log/ceph/97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6:/var/log/ceph:z -v /var/lib/ceph/97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6/crash:/var/lib/ceph/crash:z -v /var/lib/ceph/97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6/mon.host1:/var/lib/ceph/mon/ceph-host1:z -v /var/lib/ceph/97d9f40e-9d33-11eb-8e3f-1c34da4b9fb6/mon.host1/config:/etc/ceph/ceph.conf:z -v /dev:/dev -v /run/udev:/run/udev ceph/ceph@sha256:9b04c0f15704c49591640a37c7adfd40ffad0a4b42fecb950c3407687cb4f29a -n mon.host1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true '--default-log-stderr-prefix=debug ' --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true /usr/bin/docker: Error response from daemon: OCI runtime create failed: container_linux.go:344: starting container process caused "exec: \"/dev/init\": stat /dev/init: no such file or directory": unknown. Any suggestions how to resolve that ? Thank you.

3 5

has anyone enabled bdev_enable_discard?
by Dan van der Ster 15 Apr '21

15 Apr '21

Hi all, bdev_enable_discard has been in ceph for several major releases now but it is still off by default. Did anyone try it recently -- is it safe to use? And do you have perf numbers before and after enabling? Cheers, Dan

3 5

Revisit Large OMAP Objects
by DHilsbos＠performair.com 15 Apr '21

15 Apr '21

All; We run 2 Nautilus clusters, with RADOSGW replication (14.2.11 --> 14.2.16). Initially our bucket grew very quickly, as I was loading old data into it and we quickly ran into Large OMAP Object warnings. I have since done a couple manual reshards, which has fixed the warning on the primary cluster. I have never been able to get rid of the issue on the cluster with the replica. I prior conversation on this list led me to this command: radosgw-admin reshard stale-instances list --yes-i-really-mean-it The results of which look like this: [ "nextcloud-ra:f91aeff8-a365-47b4-a1c8-928cd66134e8.185262.1", "nextcloud:f91aeff8-a365-47b4-a1c8-928cd66134e8.53761.6", "nextcloud:f91aeff8-a365-47b4-a1c8-928cd66134e8.53761.2", "nextcloud:f91aeff8-a365-47b4-a1c8-928cd66134e8.53761.5", "nextcloud:f91aeff8-a365-47b4-a1c8-928cd66134e8.53761.4", "nextcloud:f91aeff8-a365-47b4-a1c8-928cd66134e8.53761.3", "nextcloud:f91aeff8-a365-47b4-a1c8-928cd66134e8.53761.1", "3520ae821f974340afd018110c1065b8/OS Development:f91aeff8-a365-47b4-a1c8-928cd66134e8.4298264.1", "10dfdfadb7374ea1ba37bee1435d87ad/volumebackups:f91aeff8-a365-47b4-a1c8-928cd66134e8.4298264.2", "WorkOrder:f91aeff8-a365-47b4-a1c8-928cd66134e8.44130.1" ] I find this particularly interesting, as nextcloud-ra, <swift>/OS Development, <swift>/volumbackups, and WorkOrder buckets no longer exist. When I run: for obj in $(rados -p 300.rgw.buckets.index ls | grep f91aeff8-a365-47b4-a1c8-928cd66134e8.3512190.1); do printf "%-60s %7d\n" $obj $(rados -p 300.rgw.buckets.index listomapkeys $obj | wc -l); done I get the expected 64 entries, with counts around 20000 +/- 1000. Are the above listed stale instances ok to delete? If so, how do I go about doing so? Thank you, Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. DHilsbos(a)PerformAir.com www.PerformAir.com

4 5

ERROR: read_key_entry() idx= 1000_ ret=-2
by by morphin 14 Apr '21

14 Apr '21

Hello everyone! I'm running nautilus 14.2.16 and I'm using RGW with Beast frontend. I see this eror log in every SSD osd which is using for rgw index. Can you please tell me what is the problem? OSD LOG: cls_rgw.cc:1102: ERROR: read_key_entry() idx=�1000_matches/xdir/05/21/27260.jpg ret=-2 cls_rgw.cc:1102: ERROR: read_key_entry() idx=�1000_matches/xdir/05/21/27253.jpg ret=-2 RADOSGW LOG: 2021-04-15 01:53:54.385 7f2e0f8e7700 1 beast: 0x55a4439f8710: 10.151.101.15 - - [2021-04-15 01:53:54.0.385327s] "HEAD /xdir/04/13/704745.jpg HTTP/1.1" 200 0 - "aws-sdk-java/1.11.638 Linux/3.10.0-1062.12.1.el7.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.251-b08 java/1.8.0_251 groovy/2.4.3 vendor/Oracle_Corporation" - 2021-04-15 01:53:54.385 7f2d8b7df700 1 ====== starting new request req=0x55a4439f8710 ===== 2021-04-15 01:53:54.405 7f2e008c9700 1 ====== req done req=0x55a43dbc6710 op status=0 http_status=204 latency=0.300003s ====== 2021-04-15 01:53:54.405 7f2e008c9700 1 beast: 0x55a43dbc6710: 10.151.101.15 - - [2021-04-15 01:53:54.0.405327s] "DELETE /xdir/05/21/21586.gz HTTP/1.1" 204 0 - "aws-sdk-java/1.11.638 Linux/3.10.0-1062.12.1.el7.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.251-b08 java/1.8.0_251 groovy/2.4.3 vendor/Oracle_Corporation" - 2021-04-15 01:53:54.405 7f2d92fee700 1 ====== starting new request req=0x55a43dbc6710 ===== 2021-04-15 01:53:54.405 7f2d92fee700 0 WARNING: couldn't find acl header for object, generating default 2021-04-15 01:53:54.405 7f2d92fee700 1 ====== req done req=0x55a43dbc6710 op status=0 http_status=200 latency=0s ====== 2021-04-15 01:53:54.405 7f2d92fee700 1 beast: 0x55a43dbc6710: 10.151.101.15 - - [2021-04-15 01:53:54.0.405327s] "HEAD /xdir/2013/11/20/2a67508e-d7dd-4e0f-b959-d7575d5f65b1 HTTP/1.1" 200 0 - "aws-sdk-java/1.11.638 Linux/3.10.0-1160.11.1.el7.x86_64 Java_HotSpot(TM)_64-Bit_Server_VM/25.281-b09 java/1.8.0_281 groovy/2.5.6 vendor/Oracle_Corporation" - CEPH OSD DF ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS 19 ssd 0.87320 1.00000 894 GiB 436 GiB 101 GiB 332 GiB 2.5 GiB 458 GiB 48.75 1.84 115 up 208 ssd 0.87329 1.00000 894 GiB 161 GiB 87 GiB 73 GiB 978 MiB 733 GiB 18.00 0.68 113 up 199 ssd 0.87320 1.00000 894 GiB 272 GiB 106 GiB 163 GiB 2.4 GiB 623 GiB 30.37 1.14 123 up 202 ssd 0.87329 1.00000 894 GiB 239 GiB 73 GiB 165 GiB 1.4 GiB 655 GiB 26.77 1.01 106 up 39 ssd 0.87320 1.00000 894 GiB 450 GiB 87 GiB 361 GiB 2.3 GiB 444 GiB 50.36 1.90 113 up 207 ssd 0.87329 1.00000 894 GiB 204 GiB 100 GiB 98 GiB 6.0 GiB 691 GiB 22.76 0.86 118 up 59 ssd 0.87320 1.00000 894 GiB 372 GiB 107 GiB 263 GiB 3.0 GiB 522 GiB 41.64 1.57 122 up 203 ssd 0.87329 1.00000 894 GiB 206 GiB 79 GiB 124 GiB 2.4 GiB 689 GiB 23.00 0.87 117 up 79 ssd 0.87320 1.00000 894 GiB 447 GiB 103 GiB 342 GiB 1.8 GiB 447 GiB 49.97 1.88 120 up 206 ssd 0.87329 1.00000 894 GiB 200 GiB 81 GiB 119 GiB 1.0 GiB 694 GiB 22.38 0.84 94 up 99 ssd 0.87320 1.00000 894 GiB 333 GiB 87 GiB 244 GiB 2.0 GiB 562 GiB 37.19 1.40 106 up 205 ssd 0.87329 1.00000 894 GiB 316 GiB 83 GiB 232 GiB 1.1 GiB 579 GiB 35.29 1.33 117 up 114 ssd 0.87329 1.00000 894 GiB 256 GiB 100 GiB 154 GiB 1.7 GiB 638 GiB 28.61 1.08 113 up 200 ssd 0.87329 1.00000 894 GiB 266 GiB 100 GiB 165 GiB 1.1 GiB 628 GiB 29.76 1.12 128 up 139 ssd 0.87320 1.00000 894 GiB 234 GiB 79 GiB 153 GiB 1.7 GiB 660 GiB 26.14 0.98 104 up 204 ssd 0.87329 1.00000 894 GiB 173 GiB 113 GiB 59 GiB 1.2 GiB 721 GiB 19.37 0.73 124 up 119 ssd 0.87329 1.00000 894 GiB 248 GiB 108 GiB 139 GiB 1.9 GiB 646 GiB 27.76 1.05 130 up 159 ssd 0.87329 1.00000 894 GiB 196 GiB 94 GiB 99 GiB 2.6 GiB 699 GiB 21.87 0.82 109 up 179 ssd 0.87329 1.00000 894 GiB 427 GiB 81 GiB 341 GiB 4.7 GiB 467 GiB 47.73 1.80 114 up 201 ssd 0.87329 1.00000 894 GiB 346 GiB 102 GiB 242 GiB 1.8 GiB 548 GiB 38.71 1.46 128 up CEPH IOSTAT +---------------+---------------+---------------+---------------+---------------+---------------+ | Read | Write | Total | Read IOPS | Write IOPS | Total IOPS | +---------------+---------------+---------------+---------------+---------------+---------------+ | 329 MiB/s | 39 MiB/s | 368 MiB/s | 109027 | 1646 | 110673 | | 329 MiB/s | 39 MiB/s | 368 MiB/s | 109027 | 1646 | 110673 | | 331 MiB/s | 39 MiB/s | 371 MiB/s | 114915 | 1631 | 116547 | | 331 MiB/s | 39 MiB/s | 371 MiB/s | 114915 | 1631 | 116547 | | 308 MiB/s | 42 MiB/s | 350 MiB/s | 108469 | 1635 | 110104 | | 308 MiB/s | 42 MiB/s | 350 MiB/s | 108469 | 1635 | 110104 | | 291 MiB/s | 44 MiB/s | 335 MiB/s | 105828 | 1687 | 107516 |

1 1

How to disable ceph-grafana during cephadm bootstrap
by mabi 14 Apr '21

14 Apr '21

Hello, When bootstrapping a new ceph Octopus cluster with "cephadm bootstrap", how can I tell the cephadm bootstrap NOT to install the ceph-grafana container? Thank you very much in advance for your answer. Best regards, Mabi

2 2

Ceph Month June 2021 Event
by Mike Perez 14 Apr '21

14 Apr '21

Hi everyone, In June 2021, we're hosting a month of Ceph presentations, lightning talks, and unconference sessions such as BOFs. There is no registration or cost to attend this event. The CFP is now open until May 12th. https://ceph.io/events/ceph-month-june-2021/cfp Speakers will receive confirmation that their presentation is accepted and further instructions for scheduling by May 16th. The schedule will be available on May 19th. Join the Ceph community as we discuss how Ceph, the massively scalable, open-source, software-defined storage system, can radically improve the economics and management of data storage for your enterprise. -- Mike Perez

1 0

Monitor dissapears/stopped after testing monitor-host loss and recovery
by Kai Börnert 14 Apr '21

14 Apr '21

Hi, I'm currently testing some disaster scenarios. When removing one osd/monitor host, I see that a new quorum is build without the missing host. The missing host is listed in the dashboard under Not In Quorum, so probably everything as expected. After restarting the host, I see that the osd's come back online and everything appears to be working, however the quorum is still only with two monitors. Looking at the services, I can see that it is somehow stopped. Is this expected and I must start it manually somehow, or should it work? The whole cluster is deployed using cephadm (the node was the initial bootstrap one, if that is important) Greetings, Kai

1 0

Exporting CephFS using Samba preferred method
by Martin Palma 14 Apr '21

14 Apr '21

Hello, what is the currently preferred method, in terms of stability and performance, for exporting a CephFS directory with Samba? - locally mount the CephFS directory and export it via Samba? - using the "vfs_ceph" module of Samba? Best, Martin

4 3

As the cluster is filling up, write performance decreases
by zp_8483 14 Apr '21

14 Apr '21

Backend: XFS for the filestore back-end. In our testing, we found the performance decreases when cluster usage exceed default nearfull ratio(85%), is it by design?

3 2

ceph rgw why are reads faster for larger than 64kb object size
by Ronnie Puthukkeril 13 Apr '21

13 Apr '21

Environment: Ceph Nautilus 14.2.8 Object Storage Data nodes: 12 * HDD OSDs drives each with a 12TB capacity + 2 * SSD OSDs drives for rgw bucket index pool & rgw meta pool. Custom configs (since we dealing with a majority smaller sized objects) bluestore_min_alloc_size_ssd 4096 bluestore_min_alloc_size_hdd 4096 Observations from cosbench performance tests Stage Op-Type Op-Count Byte-Count Avg-ResTime s7-read1KB 48W read 2004202 2004202000 43.11 s13-read2KB 48W read 2013906 4027812000 42.9 s19-read4KB 48W read 2014701 8058804000 42.88 s25-read8KB 48W read 2002337 16018696000 43.15 s31-read16KB 48W read 1987785 31804560000 43.46 s37-read32KB 48W read 1976190 63238080000 43.7 s43-read64KB 48W read 1929183 123467712000 44.78 s49-read128KB 48W read 9965032 1275524096000 8.67 s55-read256KB 48W read 6505554 1665421824000 13.28 The response time improves drastically when the object size is greater than 64KB. What could be the reason? Thanks, Ronnie [https://d1dejaj6dcqv24.cloudfront.net/asset/image/email-banner-384-2x.png]<https://www.qualys.com/email-banner> This message may contain confidential and privileged information. If it has been sent to you in error, please reply to advise the sender of the error and then immediately delete it. If you are not the intended recipient, do not read, copy, disclose or otherwise use this message. The sender disclaims any liability for such unauthorized use. NOTE that all incoming emails sent to Qualys email accounts will be archived and may be scanned by us and/or by external service providers to detect and prevent threats to our systems, investigate illegal or inappropriate behavior, and/or eliminate unsolicited promotional emails ("spam"). If you have any concerns about this process, please contact us.

1 3

Announcing go-ceph v0.9.0
by John Mulligan 13 Apr '21

13 Apr '21

I'm happy to announce another release of the go-ceph API bindings. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.9.0 Changes in the release are detailed in the link above. The bindings aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. These API bindings require the use of cgo. There are already a few consumers of this library in the wild, including the ceph-csi project. Specific questions, comments, bugs etc are best directed at our github issues tracker or github discussions forum. -- John Mulligan phlogistonjohn(a)asynchrono.us jmulligan(a)redhat.com _______________________________________________ Dev mailing list -- dev(a)ceph.io To unsubscribe send an email to dev-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ Dev mailing list -- dev(a)ceph.io To unsubscribe send an email to dev-leave(a)ceph.io

1 0

BADAUTHORIZER in Nautilus, unknown PGs, slow peering, very slow client I/O
by Nico Schottelius 13 Apr '21

13 Apr '21

Good morning, I've look somewhat intensively through the list and it seems we are rather hard hit by this. Originally yesterday started on a mixed 14.2.9 and 14.2.16 cluster (osds, mons were all 14.2.16). We started phasing in 7 new osds, 6 of them throttled by reweighting to 0.1. Symptoms are many unknown PGs, very long stuck in peering (hours), slow activating and the infamous BADAUTHORIZER message. Client I/O is almost 0, not only of the pool with new OSDs, but also of other pools. We tried restarting all OSDs one by one, which seemed to clear the unknown PGs, however after around 1h they came back to unknown state. Tuning the --osd-max-backfills=.. and --osd-recovery-max-active=... to 1 does not improve the situation, so we are currently running at =7 for both of them. An excerpt from one of the osds which heavily logs this issue is below. Any pointer on how this problem was solved on nautilus before is much appreciated, as the issue started late last night. Best regards, Nico 2021-04-13 07:38:37.275 7f159a469700 4 rocksdb: (Original Log Time 2021/04/13-07:38:37.275255) EVENT_LOG_v1 {"time_micros": 1618292317275255, "job": 5, "event": "compaction_finished", "compaction_time_micros": 5288403, "compaction_time_cpu_micros": 2940370, "output_level": 2, "num_output_files": 6, "total_output_size": 370673432, "num_input_records": 983055, "num_output_records": 641964, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [0, 5, 35, 0, 0, 0, 0]} 2021-04-13 07:38:37.275 7f159a469700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1618292317275255, "job": 5, "event": "table_file_deletion", "file_number": 421257} 2021-04-13 07:38:37.275 7f159a469700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1618292317275255, "job": 5, "event": "table_file_deletion", "file_number": 401289} 2021-04-13 07:38:37.275 7f159a469700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1618292317275255, "job": 5, "event": "table_file_deletion", "file_number": 401288} 2021-04-13 07:38:37.275 7f159a469700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1618292317275255, "job": 5, "event": "table_file_deletion", "file_number": 401278} 2021-04-13 07:38:37.275 7f159a469700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1618292317275255, "job": 5, "event": "table_file_deletion", "file_number": 401277} 2021-04-13 07:38:37.275 7f159a469700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1618292317275255, "job": 5, "event": "table_file_deletion", "file_number": 401258} 2021-04-13 07:38:37.275 7f159a469700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1618292317275255, "job": 5, "event": "table_file_deletion", "file_number": 401257} 2021-04-13 07:38:37.275 7f159a469700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1618292317275255, "job": 5, "event": "table_file_deletion", "file_number": 401256} 2021-04-13 07:38:37.275 7f159a469700 4 rocksdb: [db/compaction_job.cc:1645] [default] [JOB 6] Compacting 1@1 + 5@2 files to L2, score 1.17 2021-04-13 07:38:37.275 7f159a469700 4 rocksdb: [db/compaction_job.cc:1649] [default] Compaction start summary: Base version 5 Base level 1, inputs: [421255(65MB)], [401318(65MB) 401319(65MB) 401320(65MB) 401321(65MB) 401322(65MB)] 2021-04-13 07:38:37.275 7f159a469700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1618292317275255, "job": 6, "event": "compaction_started", "compaction_reason": "LevelMaxLevelSize", "files_L1": [421255], "files_L2": [401318, 401319, 401320, 401321, 401322], "score": 1.17224, "input_data_size": 413630461} 2021-04-13 07:38:37.307 7f15a0704700 0 --1- [v2:[2a0a:e5c0:2:1:21b:21ff:febc:7cb6]:6816/5534,v1:[2a0a:e5c0:2:1:21b:21ff:febc:7cb6]:6817/5534] >> v1:[2a0a:e5c0:2:1:21b:21ff:febc:7cb6]:6862/8751 conn(0x55d520fbcd80 0x55d520ffa000 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_2 connect got BADAUTHORIZER 2021-04-13 07:38:37.315 7f15a372c700 0 --1- [v2:[2a0a:e5c0:2:1:21b:21ff:febc:7cb6]:6816/5534,v1:[2a0a:e5c0:2:1:21b:21ff:febc:7cb6]:6817/5534] >> v1:[2a0a:e5c0:2:1:21b:21ff:febc:5060]:6835/13651 conn(0x55d521b1bf80 0x55d521b19800 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_2 connect got BADAUTHORIZER 2021-04-13 07:38:37.707 7f15a0704700 0 --1- [v2:[2a0a:e5c0:2:1:21b:21ff:febc:7cb6]:6816/5534,v1:[2a0a:e5c0:2:1:21b:21ff:febc:7cb6]:6817/5534] >> v1:[2a0a:e5c0:2:1:21b:21ff:febc:7cb6]:6862/8751 conn(0x55d520fbcd80 0x55d520ffa000 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_2 connect got BADAUTHORIZER 2021-04-13 07:38:37.715 7f15a372c700 0 --1- [v2:[2a0a:e5c0:2:1:21b:21ff:febc:7cb6]:6816/5534,v1:[2a0a:e5c0:2:1:21b:21ff:febc:7cb6]:6817/5534] >> v1:[2a0a:e5c0:2:1:21b:21ff:febc:5060]:6835/13651 conn(0x55d521b1bf80 0x55d521b19800 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_2 connect got BADAUTHORIZER 2021-04-13 07:38:38.119 7f159a469700 4 rocksdb: [db/compaction_job.cc:1332] [default] [JOB 6] Generated table #421271: 130853 keys, 68923910 bytes 2021-04-13 07:38:38.119 7f159a469700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1618292318119320, "cf_name": "default", "job": 6, "event": "table_file_creation", "file_number": 421271, "file_size": 68923910, "table_properties": {"data_size": 67111841, "index_size": 1483972, "filter_size": 327237, "raw_key_size": 10909111, "raw_average_key_size": 83, "raw_value_size": 62457370, "raw_average_value_size": 477, "num_data_blocks": 16807, "num_entries": 130853, "filter_policy_name": "rocksdb.BuiltinBloomFilter"}} 2021-04-13 07:38:38.331 7f15a0704700 0 --1- [v2:[2a0a:e5c0:2:1:21b:21ff:febc:7cb6]:6816/5534,v1:[2a0a:e5c0:2:1:21b:21ff:febc:7cb6]:6817/5534] >> v1:[2a0a:e5c0:2:1:21b:21ff:febb:68f0]:6884/29245 conn(0x55d521b1a000 0x55d521b16800 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_2 connect got BADAUTHORIZER 2021-04-13 07:38:38.331 7f15a0704700 0 --1- [v2:[2a0a:e5c0:2:1:21b:21ff:febc:7cb6]:6816/5534,v1:[2a0a:e5c0:2:1:21b:21ff:febc:7cb6]:6817/5534] >> v1:[2a0a:e5c0:2:1:21b:21ff:febb:68f0]:6819/17174 conn(0x55d521b1c400 0x55d521b38000 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_2 connect got BADAUTHORIZER 2021-04-13 07:38:38.507 7f15a0704700 0 --1- [v2:[2a0a:e5c0:2:1:21b:21ff:febc:7cb6]:6816/5534,v1:[2a0a:e5c0:2:1:21b:21ff:febc:7cb6]:6817/5534] >> v1:[2a0a:e5c0:2:1:21b:21ff:febc:7cb6]:6862/8751 conn(0x55d520fbcd80 0x55d520ffa000 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_2 connect got BADAUTHORIZER 2021-04-13 07:38:38.515 7f15a372c700 0 --1- [v2:[2a0a:e5c0:2:1:21b:21ff:febc:7cb6]:6816/5534,v1:[2a0a:e5c0:2:1:21b:21ff:febc:7cb6]:6817/5534] >> v1:[2a0a:e5c0:2:1:21b:21ff:febc:5060]:6835/13651 conn(0x55d521b1bf80 0x55d521b19800 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_2 connect got BADAUTHORIZER 2021-04-13 07:38:38.731 7f15a0704700 0 --1- [v2:[2a0a:e5c0:2:1:21b:21ff:febc:7cb6]:6816/5534,v1:[2a0a:e5c0:2:1:21b:21ff:febc:7cb6]:6817/5534] >> v1:[2a0a:e5c0:2:1:21b:21ff:febb:68f0]:6884/29245 conn(0x55d521b1a000 0x55d521b16800 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_2 connect got BADAUTHORIZER 2021-04-13 07:38:38.731 7f15a0704700 0 --1- [v2:[2a0a:e5c0:2:1:21b:21ff:febc:7cb6]:6816/5534,v1:[2a0a:e5c0:2:1:21b:21ff:febc:7cb6]:6817/5534] >> v1:[2a0a:e5c0:2:1:21b:21ff:febb:68f0]:6819/17174 conn(0x55d521b1c400 0x55d521b38000 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_2 connect got BADAUTHORIZER -- Sustainable and modern Infrastructures by ungleich.ch

1 2

Enable Dashboard Active Alerts
by E Taka 13 Apr '21

13 Apr '21

Hi, this is documented with many links to other documents, which unfortunately only confused me. In our 6-Node-Ceph-Cluster (Pacific) the Dashboard tells me that I should "provide the URL to the API of Prometheus' Alertmanager". We only use Grafana and Prometheus which are deployed by cephadm. We did not configure anything unusual, with own containers or so. We use just the standard cephadm installation. What the documentation writes about how it "should look like" (https://docs.ceph.com/en/pacific/mgr/dashboard/#enabling-prometheus-alerting) seems to exist in the Docker-Container "prom/alertmanager:v0.20.0" in file /etc/alertmanager/alertmanager.yml […] - name: 'ceph-dashboard' webhook_configs: - url: 'https://ceph01:8443/api/prometheus_receiver' - url: 'https://10.149.12.22:8443/api/prometheus_receiver' […] (10.149.12.22 is the IP address for ceph01) Nevertheless I get the message above from the Dashboard. My questions: What do I have to write in which file or which commands, so that I can access the alerts via the dashboard? Of course this should survive reboots and updates. Thanks. Erich

1 0

HEALTH_WARN - Recovery Stuck?
by Ml Ml 12 Apr '21

12 Apr '21

Hello, i kind of ran out of disk space, so i added another host with osd.37. But it does not seem to move much data on it. (85MB in 2h) Any idea why the recovery process seems to be stuck? Should i fix the 4 backfillfull osds first? (by changing the weight)? root@ceph01:~# ceph -s cluster: id: 5436dd5d-83d4-4dc8-a93b-60ab5db145df health: HEALTH_WARN 4 backfillfull osd(s) 9 nearfull osd(s) Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull 4 pool(s) backfillfull services: mon: 3 daemons, quorum ceph03,ceph01,ceph02 (age 12d) mgr: ceph03(active, since 4M), standbys: ceph02.jwvivm mds: backup:1 {0=backup.ceph06.hdjehi=up:active} 3 up:standby osd: 53 osds: 53 up (since 2h), 53 in (since 2h); 235 remapped pgs task status: scrub status: mds.backup.ceph06.hdjehi: idle data: pools: 4 pools, 1185 pgs objects: 24.69M objects, 45 TiB usage: 149 TiB used, 42 TiB / 191 TiB avail pgs: 5388809/74059569 objects misplaced (7.276%) 950 active+clean 232 active+remapped+backfill_wait 2 active+remapped+backfilling 1 active+remapped+backfill_wait+backfill_toofull io: recovery: 0 B/s, 171 keys/s, 16 objects/s progress: Rebalancing after osd.37 marked in (2h) [............................] (remaining: 6d) root@ceph01:~# ceph health detail HEALTH_WARN 4 backfillfull osd(s); 9 nearfull osd(s); Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull; 4 pool(s) backfillfull [WRN] OSD_BACKFILLFULL: 4 backfillfull osd(s) osd.28 is backfill full osd.32 is backfill full osd.66 is backfill full osd.68 is backfill full [WRN] OSD_NEARFULL: 9 nearfull osd(s) osd.11 is near full osd.24 is near full osd.27 is near full osd.39 is near full osd.40 is near full osd.42 is near full osd.43 is near full osd.45 is near full osd.69 is near full [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull pg 23.295 is active+remapped+backfill_wait+backfill_toofull, acting [8,67,32] [WRN] POOL_BACKFILLFULL: 4 pool(s) backfillfull pool 'backurne-rbd' is backfillfull pool 'device_health_metrics' is backfillfull pool 'cephfs.backup.meta' is backfillfull pool 'cephfs.backup.data' is backfillfull root@ceph01:~# ceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 182.59897 - 191 TiB 149 TiB 149 TiB 35 GiB 503 GiB 42 TiB 77.96 1.00 - root default -2 24.62473 - 29 TiB 22 TiB 22 TiB 5.0 GiB 80 GiB 7.1 TiB 75.23 0.96 - host ceph01 0 hdd 2.39999 1.00000 2.7 TiB 2.2 TiB 2.2 TiB 665 MiB 8.0 GiB 480 GiB 82.43 1.06 53 up osd.0 1 hdd 2.29999 1.00000 2.7 TiB 2.1 TiB 2.1 TiB 446 MiB 7.5 GiB 590 GiB 78.44 1.01 49 up osd.1 4 hdd 2.67029 0.91066 2.7 TiB 2.2 TiB 2.2 TiB 484 MiB 7.9 GiB 440 GiB 83.90 1.08 53 up osd.4 8 hdd 2.39999 1.00000 2.7 TiB 2.1 TiB 2.1 TiB 490 MiB 7.9 GiB 533 GiB 80.49 1.03 51 up osd.8 11 hdd 1.71660 1.00000 1.7 TiB 1.5 TiB 1.5 TiB 406 MiB 5.5 GiB 200 GiB 88.60 1.14 36 up osd.11 12 hdd 1.29999 1.00000 2.7 TiB 1.2 TiB 1.2 TiB 366 MiB 4.9 GiB 1.5 TiB 43.89 0.56 28 up osd.12 14 hdd 2.20000 1.00000 2.7 TiB 2.0 TiB 2.0 TiB 418 MiB 7.1 GiB 693 GiB 74.66 0.96 47 up osd.14 18 hdd 2.20000 1.00000 2.7 TiB 2.0 TiB 1.9 TiB 434 MiB 7.3 GiB 737 GiB 73.05 0.94 47 up osd.18 22 hdd 1.00000 1.00000 1.7 TiB 890 GiB 886 GiB 110 MiB 3.6 GiB 868 GiB 50.62 0.65 20 up osd.22 30 hdd 1.50000 1.00000 1.7 TiB 1.4 TiB 1.3 TiB 361 MiB 4.9 GiB 370 GiB 78.93 1.01 32 up osd.30 33 hdd 1.59999 0.97437 1.6 TiB 1.4 TiB 1.4 TiB 397 MiB 5.4 GiB 213 GiB 87.20 1.12 34 up osd.33 64 hdd 3.33789 0.89752 3.3 TiB 2.7 TiB 2.7 TiB 573 MiB 9.9 GiB 647 GiB 81.07 1.04 64 up osd.64 -3 26.79504 - 30 TiB 24 TiB 24 TiB 6.2 GiB 89 GiB 5.4 TiB 81.80 1.05 - host ceph02 2 hdd 1.50000 1.00000 1.7 TiB 1.4 TiB 1.4 TiB 363 MiB 5.3 GiB 359 GiB 79.58 1.02 32 up osd.2 3 hdd 2.50000 1.00000 2.7 TiB 2.2 TiB 2.2 TiB 647 MiB 7.8 GiB 469 GiB 82.85 1.06 53 up osd.3 7 hdd 2.00000 1.00000 2.7 TiB 1.8 TiB 1.8 TiB 453 MiB 7.0 GiB 848 GiB 69.00 0.89 43 up osd.7 9 hdd 2.67029 0.98323 2.7 TiB 2.4 TiB 2.3 TiB 709 MiB 8.8 GiB 322 GiB 88.21 1.13 57 up osd.9 13 hdd 1.79999 1.00000 2.4 TiB 1.7 TiB 1.6 TiB 410 MiB 6.5 GiB 747 GiB 69.41 0.89 40 up osd.13 16 hdd 2.50000 1.00000 2.7 TiB 2.2 TiB 2.2 TiB 637 MiB 7.8 GiB 458 GiB 83.26 1.07 53 up osd.16 19 hdd 1.39999 1.00000 1.7 TiB 1.3 TiB 1.3 TiB 345 MiB 5.1 GiB 465 GiB 73.53 0.94 30 up osd.19 23 hdd 2.00000 1.00000 2.7 TiB 1.9 TiB 1.9 TiB 442 MiB 7.7 GiB 738 GiB 73.02 0.94 43 up osd.23 24 hdd 1.71660 0.95634 1.7 TiB 1.5 TiB 1.5 TiB 426 MiB 5.8 GiB 187 GiB 89.37 1.15 36 up osd.24 28 hdd 2.70000 1.00000 2.7 TiB 2.5 TiB 2.4 TiB 712 MiB 8.4 GiB 219 GiB 92.00 1.18 58 up osd.28 31 hdd 2.67029 0.92993 2.7 TiB 2.3 TiB 2.3 TiB 465 MiB 8.1 GiB 393 GiB 85.62 1.10 54 up osd.31 32 hdd 3.33789 1.00000 3.3 TiB 3.0 TiB 3.0 TiB 693 MiB 11 GiB 306 GiB 91.06 1.17 71 up osd.32 -4 24.52005 - 26 TiB 21 TiB 21 TiB 5.0 GiB 79 GiB 5.1 TiB 80.51 1.03 - host ceph03 5 hdd 1.71660 1.00000 1.7 TiB 1.5 TiB 1.5 TiB 392 MiB 5.6 GiB 223 GiB 87.34 1.12 35 up osd.5 6 hdd 1.71660 1.00000 1.7 TiB 1.5 TiB 1.5 TiB 397 MiB 5.6 GiB 221 GiB 87.41 1.12 35 up osd.6 10 hdd 2.50000 0.97487 2.7 TiB 2.2 TiB 2.2 TiB 497 MiB 7.7 GiB 480 GiB 82.46 1.06 52 up osd.10 15 hdd 2.29999 1.00000 2.7 TiB 2.1 TiB 2.1 TiB 474 MiB 7.6 GiB 586 GiB 78.57 1.01 49 up osd.15 17 hdd 1.39999 1.00000 1.6 TiB 1.2 TiB 1.2 TiB 352 MiB 5.6 GiB 384 GiB 76.88 0.99 30 up osd.17 20 hdd 1.59999 1.00000 1.7 TiB 1.4 TiB 1.4 TiB 234 MiB 5.4 GiB 331 GiB 81.15 1.04 33 up osd.20 21 hdd 2.00000 1.00000 2.7 TiB 1.8 TiB 1.8 TiB 611 MiB 7.0 GiB 868 GiB 68.27 0.88 44 up osd.21 25 hdd 1.70000 0.92348 1.7 TiB 1.4 TiB 1.4 TiB 407 MiB 5.6 GiB 274 GiB 84.41 1.08 35 up osd.25 26 hdd 2.50000 1.00000 2.7 TiB 2.2 TiB 2.2 TiB 464 MiB 7.8 GiB 441 GiB 83.88 1.08 52 up osd.26 27 hdd 2.70000 0.95955 2.7 TiB 2.4 TiB 2.4 TiB 674 MiB 8.3 GiB 318 GiB 88.35 1.13 57 up osd.27 29 hdd 2.67029 0.73337 2.7 TiB 1.8 TiB 1.8 TiB 436 MiB 6.7 GiB 885 GiB 67.63 0.87 43 up osd.29 63 hdd 1.71660 1.00000 1.7 TiB 1.5 TiB 1.5 TiB 226 MiB 5.7 GiB 224 GiB 87.26 1.12 35 up osd.63 -11 24.64297 - 25 TiB 21 TiB 21 TiB 4.9 GiB 66 GiB 3.4 TiB 86.48 1.11 - host ceph04 34 hdd 5.24519 0.85004 5.2 TiB 4.0 TiB 4.0 TiB 1002 MiB 13 GiB 1.2 TiB 76.37 0.98 97 up osd.34 42 hdd 5.24519 1.00000 5.2 TiB 4.7 TiB 4.7 TiB 1.1 GiB 15 GiB 545 GiB 89.86 1.15 113 up osd.42 44 hdd 7.00000 1.00000 7.2 TiB 6.3 TiB 6.3 TiB 1.4 GiB 19 GiB 901 GiB 87.70 1.12 150 up osd.44 45 hdd 7.15259 1.00000 7.2 TiB 6.5 TiB 6.4 TiB 1.5 GiB 19 GiB 718 GiB 90.20 1.16 154 up osd.45 -13 30.04085 - 30 TiB 26 TiB 26 TiB 5.8 GiB 81 GiB 4.2 TiB 86.11 1.10 - host ceph05 39 hdd 7.15259 1.00000 7.2 TiB 6.4 TiB 6.4 TiB 1.5 GiB 19 GiB 751 GiB 89.74 1.15 153 up osd.39 40 hdd 7.15259 1.00000 7.2 TiB 6.4 TiB 6.4 TiB 1.3 GiB 19 GiB 767 GiB 89.53 1.15 153 up osd.40 41 hdd 7.15259 0.90002 7.2 TiB 5.8 TiB 5.7 TiB 1.2 GiB 18 GiB 1.4 TiB 80.54 1.03 138 up osd.41 43 hdd 5.24519 1.00000 5.2 TiB 4.7 TiB 4.7 TiB 1.1 GiB 15 GiB 574 GiB 89.32 1.15 113 up osd.43 60 hdd 3.33789 0.85780 3.3 TiB 2.6 TiB 2.6 TiB 685 MiB 8.9 GiB 754 GiB 77.93 1.00 62 up osd.60 -9 17.64297 - 18 TiB 13 TiB 13 TiB 3.0 GiB 43 GiB 4.4 TiB 74.85 0.96 - host ceph06 35 hdd 7.15259 0.80005 7.2 TiB 5.2 TiB 5.2 TiB 1.0 GiB 16 GiB 2.0 TiB 72.31 0.93 124 up osd.35 36 hdd 5.24519 0.85004 5.2 TiB 4.0 TiB 4.0 TiB 985 MiB 13 GiB 1.2 TiB 76.65 0.98 97 up osd.36 38 hdd 5.24519 0.85004 5.2 TiB 4.0 TiB 4.0 TiB 1.0 GiB 13 GiB 1.2 TiB 76.50 0.98 97 up osd.38 -15 24.79565 - 25 TiB 22 TiB 22 TiB 4.7 GiB 66 GiB 3.1 TiB 87.64 1.12 - host ceph07 66 hdd 7.15259 1.00000 7.2 TiB 6.5 TiB 6.5 TiB 1.5 GiB 19 GiB 670 GiB 90.86 1.17 155 up osd.66 67 hdd 7.15259 0.91141 7.2 TiB 5.8 TiB 5.8 TiB 1.1 GiB 18 GiB 1.3 TiB 81.62 1.05 140 up osd.67 68 hdd 3.33789 1.00000 3.3 TiB 3.0 TiB 3.0 TiB 738 MiB 9.8 GiB 299 GiB 91.24 1.17 71 up osd.68 69 hdd 7.15259 1.00000 7.2 TiB 6.3 TiB 6.3 TiB 1.3 GiB 19 GiB 823 GiB 88.77 1.14 152 up osd.69 -17 9.53670 - 9.5 TiB 1.4 GiB 85 MiB 473 MiB 832 MiB 9.5 TiB 0.01 0 - host ceph08 37 hdd 9.53670 1.00000 9.5 TiB 1.4 GiB 85 MiB 473 MiB 832 MiB 9.5 TiB 0.01 0 2 up osd.37 TOTAL 191 TiB 149 TiB 149 TiB 35 GiB 503 GiB 42 TiB 77.96 MIN/MAX VAR: 0/1.18 STDDEV: 14.73

4 3

cephadm custom mgr modules
by Rob Haverkamp 12 Apr '21

12 Apr '21

Hi there, I'm developing a custom ceph-mgr module and have issues deploying this on a cluster deployed with cephadm. With a cluster deployed with ceph-deploy, I can just put my code under /usr/share/ceph/mgr/ and load the module. This works fine. I think I found 2 options to do this with cephadm: 1. build a custom container image: https://docs.ceph.com/en/octopus/cephadm/install/#deploying-custom-containe… 2. use the --shared_ceph_folder during cephadm bootstrap: 'Development mode. Several folders in containers are volumes mapped to different sub-folders in the ceph source folder' The shared folder method is only meant for development. So that is not an option in a production environment. Building a custom container image should be possible, but I don't think I want to go there. Are there more options? It would be nice if it was possible to deploy the managers with a custom service specification that for example mounts a folder from the host system to /usr/share/ceph/mgr/<module> in the container. Thanks! Rob Haverkamp

3 2

Ceph osd Reweight command in octopus
by Brent Kennedy 12 Apr '21

12 Apr '21

We have a ceph octopus cluster running 15.2.6, its indicating a near full osd which I can see is not weighted equally with the rest of the osds. I tried to do the usual "ceph osd reweight osd.0 0.95" to force it down a little bit, but unlike the nautilus clusters, I see no data movement when issuing the command. If I run a ceph osd tree, it shows the reweight setting, but no data movement appears to be occurring. Is there some new thing in ocotopus I am missing? I looked through the release notes for .7, .8 and .9 and didn't see any fixes that jumped out as resolving a bug related to this. The Octopus cluster was deployed using ceph-ansible and upgraded to 15.2.6. I plan to upgrade to 15.2.9 in the coming month. Any thoughts? Regards, -Brent Existing Clusters: Test: Ocotpus 15.2.5 ( all virtual on nvme ) US Production(HDD): Nautilus 14.2.11 with 11 osd servers, 3 mons, 4 gateways, 2 iscsi gateways UK Production(HDD): Nautilus 14.2.11 with 18 osd servers, 3 mons, 4 gateways, 2 iscsi gateways US Production(SSD): Nautilus 14.2.11 with 6 osd servers, 3 mons, 4 gateways, 2 iscsi gateways UK Production(SSD): Octopus 15.2.6 with 5 osd servers, 3 mons, 4 gateways

2 2

rbd info error opening image
by Marcel Kuiper 12 Apr '21

12 Apr '21

I hope someone can help out. I cannot run 'rbd info' on any image. # rbd ls openstack-volumes volume-628efc47-fc57-4630-8661-a13210a4e02c volume-e4fe1e24-fb26-4abc-a458-f936a4e75715 volume-1ce1439d-767b-4b1d-8217-51464a11c5cc volume-0a01d7e3-2c8f-4fab-9f9f-d84bbc7fa3c7 volume-a4aeb848-7283-4cd0-b5e6-ac2fc7f06dac # rbd info openstack-volumes/volume-a4aeb848-7283-4cd0-b5e6-ac2fc7f06dac rbd: error opening image volume-a4aeb848-7283-4cd0-b5e6-ac2fc7f06dac: (2) No such file or directory We're running nautilus 14.2.16 on ubuntu bionic Marcel

2 2

Nautilus, Ceph-Ansible, existing OSDs, and ceph.conf updates
by Dave Hall 12 Apr '21

12 Apr '21

Hello, A while back I asked about the troubles I was having with Ceph-Ansible when I kept existing OSDs in my inventory file when managing my Nautilus cluster. At the time it was suggested that once the OSDs have been configured they should be excluded from the inventory file. However, when processing certain configuration changes Ceph-Ansible updates ceph.conf on all cluster nodes and clients in the inventory file. Is there an alternative way to keep OSD nodes in the inventory file without listing them as OSD nodes, so they get other updates, but also so Ceph-Ansible doesn't try to do any of the ceph-volume stuff that seems to be failing after the OSDs are configured? Or maybe I just have something odd in my inventory file. I'd be glad to share - either in this list or off line. Thanks. -Dave -- Dave Hall Binghamton University kdhall(a)binghamton.edu

2 1

RGW failed to start after upgrade to pacific
by 胡玮文 12 Apr '21

12 Apr '21

Hi all, I upgraded to the pacific version with cephadm. However, all our RGW daemons cannot start anymore. any help is appreciated. Here are the logs when starting RGW. I set debug_rados and debug_rgw to 20/20 systemd[1]: Started Ceph rgw.smil.b7-1.gpu006.twfefs for e88d509a-f6fc-11ea-b25d-a0423f3ac864. bash[9823]: WARNING: Error loading config file: .dockercfg: $HOME is not defined bash[9823]: debug 2021-04-04T13:01:03.731+0000 7ff80f172440 0 deferred set uid:gid to 167:167 (ceph:ceph) bash[9823]: debug 2021-04-04T13:01:03.731+0000 7ff80f172440 0 ceph version 16.2.0 (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable), process radosgw, pid 7 bash[9823]: debug 2021-04-04T13:01:03.731+0000 7ff80f172440 0 framework: beast bash[9823]: debug 2021-04-04T13:01:03.731+0000 7ff80f172440 0 framework conf key: port, val: 7480 bash[9823]: debug 2021-04-04T13:01:03.731+0000 7ff80f172440 1 radosgw_Main not setting numa affinity bash[9823]: debug 2021-04-04T13:01:03.735+0000 7ff7f742b700 20 reqs_thread_entry: start bash[9823]: debug 2021-04-04T13:01:03.735+0000 7ff7f6c2a700 10 entry start bash[9823]: debug 2021-04-04T13:01:03.739+0000 7ff80f172440 1 librados: starting msgr at bash[9823]: debug 2021-04-04T13:01:03.739+0000 7ff80f172440 1 librados: starting objecter bash[9823]: debug 2021-04-04T13:01:03.739+0000 7ff80f172440 1 librados: setting wanted keys bash[9823]: debug 2021-04-04T13:01:03.739+0000 7ff80f172440 1 librados: calling monclient init bash[9823]: debug 2021-04-04T13:01:03.743+0000 7ff80f172440 1 librados: init done bash[9823]: debug 2021-04-04T13:01:03.743+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.743+0000 7ff80f172440 10 librados: wait_for_osdmap waiting bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 10 librados: wait_for_osdmap done waiting bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 10 librados: read oid=realms_names.smil nspace= bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 10 librados: Objecter returned from read r=0 bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 20 rados_obj.operate() r=0 bl.length=46 bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 10 librados: read oid=realms.6c92e1e6-c08b-4f4c-b211-7899893978fa nspace= bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 10 librados: Objecter returned from read r=0 bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 20 rados_obj.operate() r=0 bl.length=104 bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 20 realm smil 6c92e1e6-c08b-4f4c-b211-7899893978fa bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 10 librados: read oid=realms.6c92e1e6-c08b-4f4c-b211-7899893978fa nspace= bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 10 librados: Objecter returned from read r=0 bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 20 rados_obj.operate() r=0 bl.length=104 bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.747+0000 7ff80f172440 10 librados: read oid=periods.5528a87b-bca7-4936-b348-fcb8f213df20.latest_epoch nspace= bash[9823]: debug 2021-04-04T13:01:03.751+0000 7ff80f172440 10 librados: Objecter returned from read r=0 bash[9823]: debug 2021-04-04T13:01:03.751+0000 7ff80f172440 20 rados_obj.operate() r=0 bl.length=10 bash[9823]: debug 2021-04-04T13:01:03.751+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.751+0000 7ff80f172440 10 librados: read oid=periods.5528a87b-bca7-4936-b348-fcb8f213df20.1 nspace= bash[9823]: debug 2021-04-04T13:01:03.771+0000 7ff80f172440 10 librados: Objecter returned from read r=0 bash[9823]: debug 2021-04-04T13:01:03.771+0000 7ff80f172440 20 rados_obj.operate() r=0 bl.length=877 bash[9823]: debug 2021-04-04T13:01:03.771+0000 7ff80f172440 20 current period 5528a87b-bca7-4936-b348-fcb8f213df20 bash[9823]: debug 2021-04-04T13:01:03.771+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.771+0000 7ff80f172440 10 librados: read oid=converted nspace= bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 10 librados: Objecter returned from read r=-2 bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 20 rados_obj.operate() r=-2 bl.length=0 bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 10 librados: read oid=realms_names.smil nspace= bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 10 librados: Objecter returned from read r=0 bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 20 rados_obj.operate() r=0 bl.length=46 bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 10 librados: read oid=realms.6c92e1e6-c08b-4f4c-b211-7899893978fa nspace= bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 10 librados: Objecter returned from read r=0 bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 20 rados_obj.operate() r=0 bl.length=104 bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 10 librados: read oid=default.region nspace= bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 10 librados: Objecter returned from read r=-2 bash[9823]: debug 2021-04-04T13:01:03.775+0000 7ff80f172440 20 rados_obj.operate() r=-2 bl.length=0 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got default.zonegroup.6c92e1e6-c08b-4f4c-b211-7899893978fa bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got zonegroup_info.c9c9b732-ec1b-474a-afd6-fe95b8a6ac20 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got default.realm bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got periods.bd67eb14-7081-49ab-bfe5-718b29257410.1 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got zone_info.ceb42b3a-cf4b-493d-94f9-beb7f3a76f99 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got realms.6c92e1e6-c08b-4f4c-b211-7899893978fa.control bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got periods.5528a87b-bca7-4936-b348-fcb8f213df20.1 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got zone_names.b7-1 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got periods.6c92e1e6-c08b-4f4c-b211-7899893978fa:staging bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got realms.6c92e1e6-c08b-4f4c-b211-7899893978fa bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got default.zone.6c92e1e6-c08b-4f4c-b211-7899893978fa bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got periods.bd67eb14-7081-49ab-bfe5-718b29257410.latest_epoch bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got periods.5528a87b-bca7-4936-b348-fcb8f213df20.latest_epoch bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got realms_names.smil bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got period_config.6c92e1e6-c08b-4f4c-b211-7899893978fa bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 RGWRados::pool_iterate: got zonegroups_names.default bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 10 librados: read oid=zone_names.default nspace= bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 10 librados: Objecter returned from read r=-2 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 rados_obj.operate() r=-2 bl.length=0 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 10 librados: read oid=zonegroups_names.default nspace= bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 10 librados: Objecter returned from read r=0 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 rados_obj.operate() r=0 bl.length=46 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 10 librados: read oid=zonegroup_info.c9c9b732-ec1b-474a-afd6-fe95b8a6ac20 nspace= bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 10 librados: Objecter returned from read r=0 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 rados_obj.operate() r=0 bl.length=403 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 10 librados: read oid=region_map nspace= bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 10 librados: Objecter returned from read r=-2 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 rados_obj.operate() r=-2 bl.length=0 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 10 librados: read oid=realms_names.smil nspace= bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 10 librados: Objecter returned from read r=0 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 rados_obj.operate() r=0 bl.length=46 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.943+0000 7ff80f172440 10 librados: read oid=realms.6c92e1e6-c08b-4f4c-b211-7899893978fa nspace= bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: Objecter returned from read r=0 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados_obj.operate() r=0 bl.length=104 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: read oid=default.zonegroup.6c92e1e6-c08b-4f4c-b211-7899893978fa nspace= bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: Objecter returned from read r=0 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados_obj.operate() r=0 bl.length=46 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: read oid=zonegroup_info.c9c9b732-ec1b-474a-afd6-fe95b8a6ac20 nspace= bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: Objecter returned from read r=0 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados_obj.operate() r=0 bl.length=403 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 period zonegroup init ret 0 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 period zonegroup name default bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 using current period zonegroup default bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: read oid=zone_names.b7-1 nspace= bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: Objecter returned from read r=0 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados_obj.operate() r=0 bl.length=46 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: read oid=zone_info.ceb42b3a-cf4b-493d-94f9-beb7f3a76f99 nspace= bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: Objecter returned from read r=0 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados_obj.operate() r=0 bl.length=784 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 Cannot find current period zone using local zone bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: read oid=zone_info.ceb42b3a-cf4b-493d-94f9-beb7f3a76f99 nspace= bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: Objecter returned from read r=0 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados_obj.operate() r=0 bl.length=784 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 zone b7-1 found bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: call oid=bucket.sync-source-hints. nspace= bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: Objecter returned from call r=-2 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados_obj.operate() r=-2 bl.length=0 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 20 rados->read ofs=0 len=0 bash[9823]: debug 2021-04-04T13:01:03.947+0000 7ff80f172440 10 librados: call oid=bucket.sync-target-hints. nspace= bash[9823]: debug 2021-04-04T13:01:03.951+0000 7ff80f172440 10 librados: Objecter returned from call r=-2 bash[9823]: debug 2021-04-04T13:01:03.951+0000 7ff80f172440 20 rados_obj.operate() r=-2 bl.length=0 bash[9823]: debug 2021-04-04T13:01:03.951+0000 7ff80f172440 20 started sync module instance, tier type = bash[9823]: debug 2021-04-04T13:01:03.951+0000 7ff80f172440 20 started zone id=ceb42b3a-cf4b-493d-94f9-beb7f3a76f99 (name=b7-1) with tier type = bash[9823]: debug 2021-04-04T13:01:03.951+0000 7ff80f172440 10 librados: create oid=notify.0 nspace= bash[9823]: debug 2021-04-04T13:01:04.215+0000 7ff80f172440 10 librados: Objecter returned from create r=0 bash[9823]: debug 2021-04-04T13:01:04.219+0000 7ff80f172440 10 librados: create oid=notify.1 nspace= bash[9823]: debug 2021-04-04T13:01:04.255+0000 7ff80f172440 10 librados: Objecter returned from create r=0 bash[9823]: debug 2021-04-04T13:01:04.255+0000 7ff80f172440 10 librados: create oid=notify.2 nspace= bash[9823]: debug 2021-04-04T13:01:04.307+0000 7ff80f172440 10 librados: Objecter returned from create r=0 bash[9823]: debug 2021-04-04T13:01:04.307+0000 7ff80f172440 10 librados: create oid=notify.3 nspace= bash[9823]: debug 2021-04-04T13:01:04.367+0000 7ff80f172440 10 librados: Objecter returned from create r=0 bash[9823]: debug 2021-04-04T13:01:04.367+0000 7ff80f172440 10 librados: create oid=notify.4 nspace= bash[9823]: debug 2021-04-04T13:01:04.423+0000 7ff80f172440 10 librados: Objecter returned from create r=0 bash[9823]: debug 2021-04-04T13:01:04.423+0000 7ff80f172440 10 librados: create oid=notify.5 nspace= bash[9823]: debug 2021-04-04T13:01:04.431+0000 7ff80f172440 10 librados: Objecter returned from create r=0 bash[9823]: debug 2021-04-04T13:01:04.431+0000 7ff80f172440 10 librados: create oid=notify.6 nspace= bash[9823]: debug 2021-04-04T13:01:04.495+0000 7ff80f172440 10 librados: Objecter returned from create r=0 bash[9823]: debug 2021-04-04T13:01:04.495+0000 7ff80f172440 10 librados: create oid=notify.7 nspace= bash[9823]: debug 2021-04-04T13:01:04.515+0000 7ff80f172440 10 librados: Objecter returned from create r=0 bash[9823]: debug 2021-04-04T13:01:04.515+0000 7ff80f172440 20 add_watcher() i=0 bash[9823]: debug 2021-04-04T13:01:04.515+0000 7ff80f172440 20 add_watcher() i=1 bash[9823]: debug 2021-04-04T13:01:04.515+0000 7ff80f172440 20 add_watcher() i=2 bash[9823]: debug 2021-04-04T13:01:04.515+0000 7ff80f172440 20 add_watcher() i=3 bash[9823]: debug 2021-04-04T13:01:04.515+0000 7ff80f172440 20 add_watcher() i=4 bash[9823]: debug 2021-04-04T13:01:04.515+0000 7ff80f172440 20 add_watcher() i=5 bash[9823]: debug 2021-04-04T13:01:04.527+0000 7ff80f172440 20 add_watcher() i=6 bash[9823]: debug 2021-04-04T13:01:04.527+0000 7ff80f172440 20 add_watcher() i=7 bash[9823]: debug 2021-04-04T13:01:04.527+0000 7ff80f172440 2 all 8 watchers are set, enabling cache bash[9823]: debug 2021-04-04T13:01:04.527+0000 7ff80f172440 10 librados: call oid=data_log.0 nspace= bash[9823]: debug 2021-04-04T13:01:04.527+0000 7ff80f172440 10 librados: Objecter returned from call r=0 bash[9823]: debug 2021-04-04T13:01:04.527+0000 7ff80f172440 10 librados: call oid=data_log.1 nspace= bash[9823]: debug 2021-04-04T13:01:04.527+0000 7ff80f172440 10 librados: Objecter returned from call r=-2 bash[9823]: debug 2021-04-04T13:01:04.527+0000 7ff80f172440 10 librados: call oid=data_log.2 nspace= bash[9823]: debug 2021-04-04T13:01:04.531+0000 7ff80f172440 10 librados: Objecter returned from call r=-2 bash[9823]: debug 2021-04-04T13:01:04.531+0000 7ff80f172440 10 librados: call oid=data_log.3 nspace= bash[9823]: debug 2021-04-04T13:01:04.531+0000 7ff80f172440 10 librados: Objecter returned from call r=-2 ... bash[9823]: debug 2021-04-04T13:01:04.791+0000 7ff80f172440 10 librados: call oid=data_log.127 nspace= bash[9823]: debug 2021-04-04T13:01:04.791+0000 7ff80f172440 10 librados: Objecter returned from call r=-2 bash[9823]: debug 2021-04-04T13:01:04.791+0000 7ff80f172440 10 librados: call oid=data_log.0 nspace= bash[9823]: debug 2021-04-04T13:01:04.791+0000 7ff80f172440 10 librados: Objecter returned from call r=-61 bash[9823]: debug 2021-04-04T13:01:04.791+0000 7ff80f172440 10 librados: call oid=data_log.1 nspace= bash[9823]: debug 2021-04-04T13:01:04.795+0000 7ff80f172440 10 librados: Objecter returned from call r=-2 bash[9823]: debug 2021-04-04T13:01:04.795+0000 7ff80f172440 10 librados: call oid=data_log.2 nspace= bash[9823]: debug 2021-04-04T13:01:04.795+0000 7ff80f172440 10 librados: Objecter returned from call r=-2 bash[9823]: debug 2021-04-04T13:01:04.795+0000 7ff80f172440 10 librados: call oid=data_log.3 nspace= bash[9823]: debug 2021-04-04T13:01:04.795+0000 7ff80f172440 10 librados: Objecter returned from call r=-2 ... bash[9823]: debug 2021-04-04T13:01:04.959+0000 7ff80f172440 10 librados: call oid=data_log.127 nspace= bash[9823]: debug 2021-04-04T13:01:04.959+0000 7ff80f172440 10 librados: Objecter returned from call r=-2 bash[9823]: debug 2021-04-04T13:01:04.959+0000 7ff80f172440 10 librados: call oid=data_log.0 nspace= bash[9823]: debug 2021-04-04T13:01:04.995+0000 7ff80f172440 10 librados: Objecter returned from call r=-5 bash[9823]: debug 2021-04-04T13:01:04.995+0000 7ff80f172440 -1 static int rgw::cls::fifo::FIFO::create(librados::v14_2_0::IoCtx, std::__cxx11::string, std::unique_ptr<rgw::cls::fifo::FIFO>*, optional_yield, std::optional<rados::cls::fifo::objv>, std::optional<std::basic_string_view<char> >, bool, uint64_t, uint64_t):925 create_meta failed: r=-5 bash[9823]: debug 2021-04-04T13:01:04.995+0000 7ff80f172440 -1 int RGWDataChangesLog::start(const RGWZone*, const RGWZoneParams&, RGWSI_Cls*, librados::v14_2_0::Rados*): Error when starting backend: Input/output error bash[9823]: debug 2021-04-04T13:01:04.995+0000 7ff80f172440 0 ERROR: failed to start datalog_rados service ((5) Input/output error bash[9823]: debug 2021-04-04T13:01:04.995+0000 7ff80f172440 0 ERROR: failed to init services (ret=(5) Input/output error) bash[9823]: debug 2021-04-04T13:01:05.107+0000 7ff80f172440 10 librados: watch_flush enter bash[9823]: debug 2021-04-04T13:01:05.107+0000 7ff80f172440 10 librados: watch_flush exit bash[9823]: debug 2021-04-04T13:01:05.107+0000 7ff80f172440 20 remove_watcher() i=0 bash[9823]: debug 2021-04-04T13:01:05.107+0000 7ff80f172440 2 removed watcher, disabling cache bash[9823]: debug 2021-04-04T13:01:05.131+0000 7ff80f172440 10 librados: watch_flush enter bash[9823]: debug 2021-04-04T13:01:05.131+0000 7ff80f172440 10 librados: watch_flush exit bash[9823]: debug 2021-04-04T13:01:05.131+0000 7ff80f172440 20 remove_watcher() i=1 bash[9823]: debug 2021-04-04T13:01:05.159+0000 7ff80f172440 10 librados: watch_flush enter bash[9823]: debug 2021-04-04T13:01:05.159+0000 7ff80f172440 10 librados: watch_flush exit ... bash[9823]: debug 2021-04-04T13:01:05.275+0000 7ff80f172440 20 remove_watcher() i=7 bash[9823]: debug 2021-04-04T13:01:05.275+0000 7ff80f172440 10 librados: watch_flush enter bash[9823]: debug 2021-04-04T13:01:05.275+0000 7ff80f172440 10 librados: watch_flush exit bash[9823]: debug 2021-04-04T13:01:05.275+0000 7ff80f172440 1 librados: shutdown bash[9823]: debug 2021-04-04T13:01:05.279+0000 7ff80f172440 -1 Couldn't init storage provider (RADOS) systemd[1]: ceph-e88d509a-f6fc-11ea-b25d-a0423f3ac864(a)rgw.smil.b7-1.gpu006.twfefs.service: Main process exited, code=exited, status=5/NOTINSTALLED systemd[1]: ceph-e88d509a-f6fc-11ea-b25d-a0423f3ac864(a)rgw.smil.b7-1.gpu006.twfefs.service: Failed with result 'exit-code'. systemd[1]: ceph-e88d509a-f6fc-11ea-b25d-a0423f3ac864(a)rgw.smil.b7-1.gpu006.twfefs.service: Service hold-off time over, scheduling restart. systemd[1]: ceph-e88d509a-f6fc-11ea-b25d-a0423f3ac864(a)rgw.smil.b7-1.gpu006.twfefs.service: Scheduled restart job, restart counter is at 4. systemd[1]: Stopped Ceph rgw.smil.b7-1.gpu006.twfefs for e88d509a-f6fc-11ea-b25d-a0423f3ac864. Weiwen Hu

4 4

Ceph failover claster
by Várkonyi János 12 Apr '21

12 Apr '21

Hi All, Use anybody windows file server with ceph storage? Finally I can do the gateways. We've a ceph storage with 3 nodes and we can add this to windows via ceph-iscsi. I'd like to use it with 2 windows 2019 servers in failover cluster. I can connect to the storage each sides. But when I check the MPIO device Details all nods are connected and active, I've not "stand by" node. I'm not for sure it is right or it is a problem. I setup up the deatils from the ceph documment. TimeOutValue = 65; LinkDownTime = 25; SRBTimeoutDelta = 15. I try to validate failover cluster configuration and I get an error: "Failure issuing call to Persistent Reservation REGISTER AND IGNORE EXISTING on Test Disk 0 from node FS102.trafficom.hu when the disk has no existing registration. It is expected to succeed. The device is not ready." Did anybody see this error? jansz0

2 1

cephadm upgrade to pacific
by Peter Childs 11 Apr '21

11 Apr '21

I am attempting to upgrade a Ceph Upgrade cluster that was deployed with Octopus 15.2.8 and upgraded to 15.2.10 successfully. I'm not attempting to upgrade to 16.2.0 Pacific, and it is not going very well. I am using cephadm. It looks to have upgraded the managers and stopped, and not moved on to the monitors or anything else. I've attempted stopping the upgrade and restarting it, with debug on and I'm not seeing anything to say why it is not progressing any further. I've also tried rebooting machines and failing the managers over with no success. I'm currently thinking its stuck attempting to upgrade a manager that does not exist. Its a test cluster of 16 nodes, bit of a proof of concept, so if I've got something terribly wrong I'm happy to look at deploying, (running on top of CentOS 7 but I'm fast heading to using something else) (apart from anything its not really a production ready system yet) Just not sure where cephadm upgrade has crashed in 16.2.0 Thanks in advance Peter

2 2

working ansible based crush map?
by Philip Brown 09 Apr '21

09 Apr '21

I'm trying to follow the directions in ceph-ansible for having it automatically set up the crush map. I've also looked at https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/4/html/i… But.. my setup isnt working. ansible complains about: [WARNING]: While constructing a mapping from /etc/ansible/hosts.yml, line 5, column 9, found a duplicate dict key (hostA2). Using last defined value only. Could anyone explain what I'm doing wrong, please? Here's the relevant hosts.yml snippets all: children: osds: hosts: hostA1: osd_crush_location: host: 'hostA1' chassis: 'hostA' root: 'default' hostA2: osd_crush_location: host: 'hostA2' chassis: 'hostA' root: 'default' -- Philip Brown| Sr. Linux System Administrator | Medata, Inc. 5 Peters Canyon Rd Suite 250 Irvine CA 92606 Office 714.918.1310| Fax 714.918.1325 pbrown(a)medata.com| www.medata.com

1 2

Version of podman for Ceph 15.2.10
by mabi 09 Apr '21

09 Apr '21

Hello, I would like to install Ceph 15.2.10 using cephadm and just found the following table by checking the requirements on the host: https://docs.ceph.com/en/latest/cephadm/compatibility/#compatibility-with-p… Do I understand this table correctly that I should be using podman version 2.1? and what happens if I use the latest podman version 3.0 Best regards, Mabi

2 4

upgrade problem nautilus 14.2.15 -> 14.2.18? (Broken ceph!)
by Simon Oosthoek 09 Apr '21

09 Apr '21

Hi I'm in a bit of a panic :-( Recently we started attempting to configure a radosgw to our ceph cluster, which was until now only doing cephfs (and rbd wss working as well). We were messing about with ceph-ansible, as this was how we originally installed the cluster. Anyway, it installed nautilus 14.2.18 on the radosgw and I though it would be good to pull up the rest of the cluster to that level as well using our tried and tested ceph upgrade script (it basically does an update of all ceph nodes one by one and checks whether ceph is ok again before doing the next) After the 3rd mon/mgr was done, all pg's were unavailable :-( obviously, the script is not continuing, but ceph is also broken now... The message deceptively is: HEALTH_WARN Reduced data availability: 5568 pgs inactive That's all PGs! I tried as a desperate measure to upgrade one ceph OSD node, but that broke as well, the osd service on that node gets an interrupt from the kernel.... the versions are now like: 20:29 [root@cephmon1 ~]# ceph versions { "mon": { "ceph version 14.2.18 (befbc92f3c11eedd8626487211d200c0b44786d9) nautilus (stable)": 3 }, "mgr": { "ceph version 14.2.18 (befbc92f3c11eedd8626487211d200c0b44786d9) nautilus (stable)": 3 }, "osd": { "ceph version 14.2.15 (afdd217ae5fb1ed3f60e16bd62357ca58cc650e5) nautilus (stable)": 156 }, "mds": { "ceph version 14.2.15 (afdd217ae5fb1ed3f60e16bd62357ca58cc650e5) nautilus (stable)": 2 }, "overall": { "ceph version 14.2.15 (afdd217ae5fb1ed3f60e16bd62357ca58cc650e5) nautilus (stable)": 158, "ceph version 14.2.18 (befbc92f3c11eedd8626487211d200c0b44786d9) nautilus (stable)": 6 } } 12 OSDs are down # ceph -s cluster: id: b489547c-ba50-4745-a914-23eb78e0e5dc health: HEALTH_WARN Reduced data availability: 5568 pgs inactive services: mon: 3 daemons, quorum cephmon3,cephmon1,cephmon2 (age 50m) mgr: cephmon1(active, since 53m), standbys: cephmon3, cephmon2 mds: cephfs:1 {0=cephmds2=up:active} 1 up:standby osd: 168 osds: 156 up (since 28m), 156 in (since 18m); 1722 remapped pgs data: pools: 12 pools, 5568 pgs objects: 0 objects, 0 B usage: 0 B used, 0 B / 0 B avail pgs: 100.000% pgs unknown 5568 unknown progress: Rebalancing after osd.103 marked in [..............................]

5 11

short pages when listing RADOSGW buckets via Swift API
by Paul Collins 08 Apr '21

08 Apr '21

Hi, I noticed while using rclone to migrate some data from a Swift cluster into a RADOSGW cluster that sometimes when listing a bucket RADOSGW will not always return as many results as specified by the "limit" parameter, even when more objects remain to list. This results in rclone believing on subsequent runs that the objects do not exist, since it performs an initial comparison based on bucket listings, and so it needlessly recopies data. This seems contrary to how pagination is specified by Swift: https://docs.openstack.org/swift/latest/api/pagination.html Is this known behaviour, or should I go ahead and file a bug? I believe the cluster is running 15.2.8 or so, but will confirm. Thanks, Paul --- Further observations: * Here's a summary of the reply lengths I got when listing various buckets in our RADOSGW cluster. (This is not all of the buckets in the tenant; the other 100 or so are fine.) reply lengths: 1000 999 1000 1000 1000 1000 1000 1000 1000 1000 119 reply lengths: 1000 992 1000 1000 1000 1000 1000 935 1000 1000 257 reply lengths: 1000 1000 1000 1000 1000 975 1000 948 reply lengths: 953 1000 1000 1000 1000 1000 954 1000 1000 70 reply lengths: 1000 1000 1000 1000 998 15 reply lengths: 1000 1000 1000 1000 974 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 939 1000 1000 1000 1000 949 1000 1000 1000 644 reply lengths: 1000 1000 1000 1000 999 1000 1000 937 1000 1000 538 reply lengths: 1000 998 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 551 reply lengths: 1000 1000 1000 1000 1000 1000 1000 931 1000 986 1000 1000 1000 975 1000 989 1000 1000 1000 966 1000 998 921 994 1000 1000 973 58 reply lengths: 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 976 1000 366 reply lengths: 1000 1000 1000 1000 1000 983 1000 1000 1000 1000 1000 1000 1000 517 reply lengths: 1000 1000 1000 984 1000 1000 971 1000 1000 401 reply lengths: 949 1000 1000 1000 1000 1000 1000 403 reply lengths: 1000 998 532 reply lengths: 951 1000 1000 1000 1000 1000 976 1000 877 * rclone uses a default $limit of 1,000, in contrast to the Python swiftclient's default of 10,000. * The Swift API doc seems clear that $limit results should always be returned if at least $limit results are available, and that receiving less than $limit results indicates no more exist. (It doesn't *explicitly* say the last, but the document could be a lot shorter if it were not intended for that to follow.) * When swiftclient is asked to fetch a listing, and full_listing is set to True, instead of implementing pagingation as described in the document above, swiftclient simply keeps fetching pages until it receives an empty page. So Swift API implementations that don't strictly implement paging per the docs may not even be noticed by most users. * From a review of its code, swiftclient seems to have done this since the very beginning. Perhaps the code was written first and then pagination on the server side was nailed down later? -- Paul Collins Wellington, New Zealand

1 0

bluestore_min_alloc_size_hdd on Octopus (15.2.10) / XFS formatted RBDs
by David Orman 08 Apr '21

08 Apr '21

Now that the hybrid allocator appears to be enabled by default in Octopus, is it safe to change bluestore_min_alloc_size_hdd to 4k from 64k on Octopus 15.2.10 clusters, and then redeploy every OSD to switch to the smaller allocation size, without massive performance impact for RBD? We're seeing a lot of storage usage amplification on EC 8+3 clusters which are HDD backed that lines up with a lot of the mailing list posts we've seen here. Upgrading to Pacific before making this change is also a possibility once a more stable release arrives, if that's necessary. Second part of this question - we are using RBDs currently on the clusters impacted. These have XFS filesystems on top, which detect the sector size of the RBD as 512byte, and XFS has a block size of 4k. With the default of 64k for bluestore_min_alloc_size_hdd, let's say a 1G file is written out to the XFS filesystem backed by the RBD. On the ceph side, is this being seen as a lot of 4k objects thus a significant space waste is occurring, or is RBD able to coalesce these into 64k objects, even though XFS is using a 4k block size? XFS details below, you can see the allocation groups are quite large: meta-data=/dev/rbd0 isize=512 agcount=501, agsize=268435440 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=1, sparse=1, rmapbt=0 = reflink=1 data = bsize=4096 blocks=134217728000, imaxpct=1 = sunit=16 swidth=16 blks naming =version 2 bsize=4096 ascii-ci=0, ftype=1 log =internal log bsize=4096 blocks=521728, version=2 = sectsz=512 sunit=16 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 I'm curious if people have been tuning XFS on RBD for better performance, as well. Thank you!

2 1

Nautilus 14.2.19 radosgw ignoring ceph config
by Graham Allan 08 Apr '21

08 Apr '21

We just updated one of our ceph clusters from 14.2.15 to 14.2.19, and see some unexpected behavior by radosgw - it seems to ignore parameters set by the ceph config database. Specifically this is making it start up listening only on port 7480, and not the configured 80 and 443 (ssl) ports. Downgrading ceph on the rgw nodes back to 14.2.15 restores the expected behavior (I haven't yet tried any intermediate versions). The host OS is CentOS 7, if that matters... Here's a ceph config dump for one of the affected nodes, along with the radosgw startup log: > # ceph config dump|grep tier2-gw02 > client.rgw.tier2-gw02 basic log_file /var/log/ceph/radosgw.log * > client.rgw.tier2-gw02 advanced rgw_dns_name s3.msi.umn.edu * > client.rgw.tier2-gw02 advanced rgw_enable_usage_log true > client.rgw.tier2-gw02 basic rgw_frontends beast port=80 ssl_port=443 ssl_certificate=/etc/ceph/civetweb.pem * > client.rgw.tier2-gw02 basic rgw_thread_pool_size 512 > # tail /var/log/ceph/radosgw.log > 2021-04-08 11:51:07.956 7f420b78f700 -1 received signal: Terminated from /usr/lib/systemd/systemd --switched-root --system --deserialize 22 (PID: 1) UID: 0 > 2021-04-08 11:51:07.956 7f420b78f700 1 handle_sigterm > 2021-04-08 11:51:07.956 7f4220bc5900 -1 shutting down > 2021-04-08 11:51:07.956 7f420b78f700 1 handle_sigterm set alarm for 120 > 2021-04-08 11:51:08.010 7f4220bc5900 1 final shutdown > 2021-04-08 11:51:08.159 7f2ac6105900 0 deferred set uid:gid to 167:167 (ceph:ceph) > 2021-04-08 11:51:08.159 7f2ac6105900 0 ceph version 14.2.19 (bb796b9b5bab9463106022eef406373182465d11) nautilus (stable), process radosgw, pid 88256 > 2021-04-08 11:51:08.300 7f2ac6105900 0 starting handler: beast > 2021-04-08 11:51:08.302 7f2ac6105900 0 set uid:gid to 167:167 (ceph:ceph) > 2021-04-08 11:51:08.317 7f2ac6105900 1 mgrc service_daemon_register rgw.tier2-gw02 metadata {arch=x86_64,ceph_release=nautilus,ceph_version=ceph version 14.2.19 (bb796b9b5bab9463106022eef406373182465d11) nautilus (stable),ceph_version_short=14.2.19,cpu=AMD EPYC 7302P 16-Core Processor,distro=centos,distro_description=CentOS Linux 7 (Core),distro_version=7,frontend_config#0=beast port=7480,frontend_type#0=beast,hostname=tier2-gw02.msi.umn.edu,kernel_description=#1 SMP Tue Mar 16 18:28:22 UTC 2021,kernel_version=3.10.0-1160.21.1.el7.x86_64,mem_swap_kb=4194300,mem_total_kb=131754828,num_handles=1,os=Linux,pid=88256,zone_id=default,zone_name=default,zonegroup_id=default,zonegroup_name=default} BTW I can also change "rgw_frontends" to specify a civetweb frontend instead and it will still start the default beast... I haven't seen anyone else report such a problem so I wonder if this is something local to us - like perhaps I'm using "ceph config" incorrectly in a way which happened to be accepted before? Has anyone else seen this behavior? Graham -- Graham Allan - gta(a)umn.edu Associate Director of Operations - Minnesota Supercomputing Institute

2 1

Nautilus 14.2.19 mon 100% CPU
by Robert LeBlanc 08 Apr '21

08 Apr '21

I upgraded our Luminous cluster to Nautilus a couple of weeks ago and converted the last batch of FileStore OSDs to BlueStore about 36 hours ago. Yesterday our monitor cluster went nuts and started constantly calling elections because monitor nodes were at 100% and wouldn't respond to heartbeats. I reduced the monitor cluster to one to prevent the constant elections and that let the system limp along until the backfills finished. There are large amounts of time where ceph commands hang with the CPU is at 100%, when the CPU drops I see a lot of work getting done in the monitor logs which stops as soon as the CPU is at 100% again. I did a `perf top` on the node to see what's taking all the time and it appears to be in the rocksdb code path. I've set `mon_compact_on_start = true` in the ceph.conf but that does not appear to help. The `/var/lib/ceph/mon/` directory is 311MB which is down from 3.0 GB while the backfills were going on. I've tried adding a second monitor, but it goes back to the constant elections. I tried restarting all the services without luck. I also pulled the monitor from the network work and tried restarting the mon service isolated (this helped a couple of weeks ago when `ceph -s` would cause 100% CPU and lock up the service much worse than this) and didn't see the high CPU load. So I'm guessing it's triggered from some external source. I'm happy to provide more info, just let me know what would be helpful. Thank you, Robert LeBlanc [image: image.png] ---------------- Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1

2 4

Ceph CFP Coordination for 2021
by Mike Perez 08 Apr '21

08 Apr '21

Hi everyone, I cleaned up the CFP coordination etherpad with some events coming up. Please add other events you think the community should be considering proposing content on Ceph or adjacent projects like Rook. KubeCon NA CFP, for example, is ending April 11. Take a look: https://pad.ceph.com/p/cfp-coordination I have also added this to our wiki for discovery. https://tracker.ceph.com/projects/ceph/wiki/Community -- Mike Perez

1 1

KRBD failed to mount rbd image if mapping it to the host with read-only option
by Ha, Son Hai 08 Apr '21

08 Apr '21

Hi everyone, We encountered an issue with KRBD mounting after mapping it to the host with read-only option. We try to pinpoint where the problem is, but not able to do it. The image is mounted well if we map it without the "read-only" option. This leads to an issue that the pod in k8s cannot use the snapshotted persistent volume created by ceph-csi rbd provisioner. Thank you for reading. I have reported the bug here: Bug #50234: krbd failed to mount after map image with read-only option - Ceph - Ceph<https://tracker.ceph.com/issues/50234> Context - Using admin keyring - Linux Kernel: 3.10.0-1160.15.2.el7.x86_64 - Linux Distribution: Red Hat Enterprise Linux Server 7.8 (Maipo) - Ceph version: "ceph version 14.2.8 (2d095e947a02261ce61424021bb43bd3022d35cb) nautilus (stable)" rbd image 'csi-vol-85919409-9797-11eb-80ba-720b2b57c790': size 10 GiB in 2560 objects order 22 (4 MiB objects) snapshot_count: 0 id: 533a03bba388ea block_name_prefix: rbd_data.533a03bba388ea format: 2 features: layering op_features: flags: create_timestamp: Wed Apr 7 13:51:02 2021 access_timestamp: Wed Apr 7 13:51:02 2021 modify_timestamp: Wed Apr 7 13:51:02 2021 Bug Reproduction # Map RBD image WITH read-only option, CANNOT mount with both readonly or readwrite option sudo rbd device map -p k8s-sharedpool csi-vol-85919409-9797-11eb-80ba-720b2b57c790 -ro /dev/rbd0 sudo mount -v -r -t ext4 /dev/rbd0 /mnt/test1 mount: cannot mount /dev/rbd0 read-only sudo mount -v -r -t ext4 /dev/rbd0 /mnt/test1 mount: /dev/rbd0 is write-protected, mounting read-only mount: cannot mount /dev/rbd0 read-only # Map RBD image WITHOUT read-only option, CAN mount with both readonly or readwrite option sudo rbd device map -p k8s-sharedpool csi-vol-85919409-9797-11eb-80ba-720b2b57c790 /dev/rbd0 sudo mount -v -r -t ext4 /dev/rbd0 /mnt/test1 mount: /mnt/test1 does not contain SELinux labels. You just mounted an file system that supports labels which does not contain labels, onto an SELinux box. It is likely that confined applications will generate AVC messages and not be allowed access to this file system. For more details see restorecon(8) and mount(8). mount: /dev/rbd0 mounted on /mnt/test1. sudo mount -v -t ext4 /dev/rbd0 /mnt/test1 mount: /mnt/test1 does not contain SELinux labels. You just mounted an file system that supports labels which does not contain labels, onto an SELinux box. It is likely that confined applications will generate AVC messages and not be allowed access to this file system. For more details see restorecon(8) and mount(8). mount: /dev/rbd0 mounted on /mnt/test1. With my best regards, Son Hai Ha -- KPMG IT Service GmbH Sitz/Registergericht: Berlin/Amtsgericht Charlottenburg, HRB 87521 B Geschäftsführer: Hans-Christian Schwieger, Helmar Symmank Aufsichtsratsvorsitzender: WP StB Klaus Becker Allgemeine Informationen zur Datenverarbeitung im Rahmen unserer allgemeinen Geschäftstätigkeit sowie im Mandatsverhältnis gemäß EU Datenschutz-Grundverordnung sind hier <https://home.kpmg.com/content/dam/kpmg/de/pdf/Themen/2018/datenschutzinform…> abrufbar. Die Information in dieser E-Mail ist vertraulich und kann dem Berufsgeheimnis unterliegen. Sie ist ausschließlich für den Adressaten bestimmt. Jeglicher Zugriff auf diese E-Mail durch andere Personen als den Adressaten ist untersagt. Sollten Sie nicht der für diese E-Mail bestimmte Adressat sein, ist Ihnen jede Veröffentlichung, Vervielfältigung oder Weitergabe wie auch das Ergreifen oder Unterlassen von Maßnahmen im Vertrauen auf erlangte Information untersagt. In dieser E-Mail enthaltene Meinungen oder Empfehlungen unterliegen den Bedingungen des jeweiligen Mandatsverhältnisses mit dem Adressaten. The information in this e-mail is confidential and may be legally privileged. It is intended solely for the addressee. Access to this e-mail by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. Any opinions or advice contained in this e-mail are subject to the terms and conditions expressed in the governing KPMG client engagement letter.

2 2

cephadm/podman :: upgrade to pacific stuck
by Adrian Sevcenco 08 Apr '21

08 Apr '21

Hi! I have a single machine ceph installation and after trying to update to pacific the upgrade is stuck with: ceph -s cluster: id: d9f4c810-8270-11eb-97a7-faa3b09dcf67 health: HEALTH_WARN Upgrade: Need standby mgr daemon services: mon: 1 daemons, quorum sev.spacescience.ro (age 3w) mgr: sev.spacescience.ro.wpozds(active, since 2w) mds: sev-ceph:1 {0=sev-ceph.sev.vmvwrm=up:active} osd: 2 osds: 2 up (since 2w), 2 in (since 2w) data: pools: 4 pools, 194 pgs objects: 32 objects, 8.4 KiB usage: 2.0 GiB used, 930 GiB / 932 GiB avail pgs: 194 active+clean progress: Upgrade to docker.io/ceph/ceph:v16.2.0 (0s) [............................] How can i put the mgr on standby? so far i did not find anything relevant.. Thanks a lot! Adrian

3 4

cephadm and ha service for rgw
by Seba chanel 08 Apr '21

08 Apr '21

Hi everyone, I try to configure HA service for rgw with cephadm. I have 2 rgw on cnrgw1 et cnrgw2 for the same pool. i use a virtual IP address 192.168.0.15 cnrgwha and the config from https://docs.ceph.com/en/latest/cephadm/rgw/#high-availability-service-for-… # from root@cnrgw1 [root@cnrgw1 ~]# cat /etc/sysctl.conf net.ipv6.conf.all.disable_ipv6 = 1 net.ipv4.ip_forward = 1 net.ipv4.ip_nonlocal_bind = 1 [root@cnrgw1 ~]# sysctl -p net.ipv6.conf.all.disable_ipv6 = 1 net.ipv4.ip_forward = 1 net.ipv4.ip_nonlocal_bind = 1 #same from cnrgw2 #generate cert [vagrant@cn1 ~]# openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout ./rgwha.key -out ./rgwha.crt Generating a RSA private key .............+++++ ........................................................+++++ writing new private key to './rgwha.key' ----- You are about to be asked to enter information that will be incorporated into your certificate request. What you are about to enter is what is called a Distinguished Name or a DN. There are quite a few fields but you can leave some blank For some fields there will be a default value, If you enter '.', the field will be left blank. ----- Country Name (2 letter code) [XX]:fr State or Province Name (full name) []:est Locality Name (eg, city) [Default City]:sbg Organization Name (eg, company) [Default Company Ltd]:cephlab.org Organizational Unit Name (eg, section) []: Common Name (eg, your name or your server's hostname) []:cnrgwha Email Address []:root@localhost # write the YAML rgwha.yaml service_type: ha-rgw service_id: haproxy_for_rgw placement: hosts: - cnrgw1 - cnrgw2 spec: virtual_ip_interface: eth1 virtual_ip_address: 192.168.0.15/24 frontend_port: 8080 ha_proxy_port: 1967 ha_proxy_stats_enabled: true ha_proxy_stats_user: admin ha_proxy_stats_password: true ha_proxy_enable_prometheus_exporter: true ha_proxy_monitor_uri: /haproxy_health keepalived_user: admin keepalived_password: admin ha_proxy_frontend_ssl_certificate: [ "-----BEGIN CERTIFICATE-----", "MIICSzCCAfWgAwIBAgIUWKC9e+5tnIAjddECXOGc144p8E0wDQYJKoZIhvcNAQEL", "BQAwejELMAkGA1UEBhMCZnIxDDAKBgNVBAgMA2VzdDEMMAoGA1UEBwwDc2JnMRAw", "DgYDVQQKDAdjZXBobGFiMQwwCgYDVQQLDANvcmcxEDAOBgNVBAMMB2Nucmd3aGEx", "HTAbBgkqhkiG9w0BCQEWDnJvb3RAbG9jYWxob3N0MB4XDTIxMDMwOTE0MjI0N1oX", "DTIyMDMwOTE0MjI0N1owejELMAkGA1UEBhMCZnIxDDAKBgNVBAgMA2VzdDEMMAoG", "A1UEBwwDc2JnMRAwDgYDVQQKDAdjZXBobGFiMQwwCgYDVQQLDANvcmcxEDAOBgNV", "BAMMB2Nucmd3aGExHTAbBgkqhkiG9w0BCQEWDnJvb3RAbG9jYWxob3N0MFwwDQYJ", "KoZIhvcNAQEBBQADSwAwSAJBAMqji/AKBr6DbuHKOTWyIBWbeYkyZ7Jn7fqfZceE", "p7G321t1TvAjD7sa64FRT6n4x8CtzKPGXXpRr28o8oR1h70CAwEAAaNTMFEwHQYD", "VR0OBBYEFIQim5ZxojFny+srzQJIs1N8wLmYMB8GA1UdIwQYMBaAFIQim5ZxojFn", "y+srzQJIs1N8wLmYMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADQQCE", "eCwMQFNYtw+4I1QzTV13ewawuPkPdrhiNzcs0mgt93+quE0zBIeOY2jnFmlo6H/h", "syYGvwgcAh9VW9qo5fsk", "-----END CERTIFICATE-----", "-----BEGIN PRIVATE KEY-----", "MIIBVQIBADANBgkqhkiG9w0BAQEFAASCAT8wggE7AgEAAkEAyqOL8AoGvoNu4co5", "NbIgFZt5iTJnsmft+p9lx4SnsbfbW3VO8CMPuxrrgVFPqfjHwK3Mo8ZdelGvbyjy", "hHWHvQIDAQABAkB0kt2AO+RhWS9CyZlb4JtAku66FLs/ETcAxQ5CV3g5beq8/wRs", "x3xZhIsjdr7OZZ+BEoJYn+0upywoctXmwM8BAiEA+KG26RADqJfAdoRn640UrT9E", "pfF3drDrQg0WrKAf3N0CIQDQpOZa0pV2GL28u2NaU85uJCDeKDWhTnvFEqlLu/S4", "YQIhAPY+0/WIUtdLVOcMxA/bLrtXihoASR1Yo+hLJkXaYTRRAiB3Rh1txD6vEXu+", "Hb2xUIGNE1g6x+/ItA4rXfysD9nZYQIhAKYn3IdG55JwiwSKv8gVAEdX8xiUfEjY", "pnvk3p52VHHI", "-----END PRIVATE KEY-----" ] ha_proxy_frontend_ssl_port: 8090 ha_proxy_ssl_dh_param: 1024 ha_proxy_ssl_ciphers: ECDH+AESGCM:!MD5 ha_proxy_ssl_options: no-sslv3 haproxy_container_image: haproxy:2.4-dev3-alpine keepalived_container_image: arcts/keepalived:1.2.2 # apply the new config [ceph: root@cn1 ~]# ceph orch apply -i rgwha.yaml Error EINVAL: ServiceSpec: __init__() got an unexpected keyword argument 'virtual_ip_interface' Do you have any leads why it doesn't work? [ceph: root@cn1 /]# ceph versions { "mon": { "ceph version 15.2.9 (357616cbf726abb779ca75a551e8d02568e15b17) octopus (stable)": 5 }, "mgr": { "ceph version 15.2.9 (357616cbf726abb779ca75a551e8d02568e15b17) octopus (stable)": 2 }, "osd": { "ceph version 15.2.9 (357616cbf726abb779ca75a551e8d02568e15b17) octopus (stable)": 8 }, "mds": {}, "rgw": { "ceph version 15.2.9 (357616cbf726abb779ca75a551e8d02568e15b17) octopus (stable)": 2 }, "overall": { "ceph version 15.2.9 (357616cbf726abb779ca75a551e8d02568e15b17) octopus (stable)": 17 } }

2 2

Nautilus: rgw_max_chunk_size = 4M?
by Dave Hall 07 Apr '21

07 Apr '21

Hello, Pardon if this has been asked, but I'm just getting started with Rados Gateway. I looked around for some hints about performance tuning and found a reference to setting rgw_max_chunk_size = 4M. I suspect the material was written during Jewel or earlier, so I'm wondering about the best practices for Nautilus? 1) I couldn't find how to set this in Nautilus. 2) I found a mailing list post from August 2019 that talked about EC pools and using a multiple of k * 4M. Any insight, or a pointer to the right part of the docs would be greatly appreciated. Thanks. -Dave -- Dave Hall Binghamton University kdhall(a)binghamton.edu

1 0

which is definitive: /var/lib/ceph symlinks or ceph-volume?
by Philip Brown 07 Apr '21

07 Apr '21

I am in a situation where I see conflicting information. On the one hand, ls -l /var/lib/ceph/osd/ceph-7 shows a symlink for block device, but no block.db On the other hand, ceph-volume lvm list claims that there is a separate db device registered for osd 7 how can I know which one is correct? (This is currently ceph nautilus) -- Philip Brown| Sr. Linux System Administrator | Medata, Inc. 5 Peters Canyon Rd Suite 250 Irvine CA 92606 Office 714.918.1310| Fax 714.918.1325 pbrown(a)medata.com| www.medata.com

2 5

Increase of osd space usage on cephfs heavy load
by Olivier AUDRY 06 Apr '21

06 Apr '21

hello when I run my borgbackup over cephfs volume (10 subvolumes for 1.5To) I can see a big increase of osd space usage and 2 or 3 osd goes near full, or full, then out and finally the cluster goes in error. Any tips to prevent this ? My cluster is cephv15 with : 9 nodes : each node run : 2x6to hdd and 2x600to ssd the cephfs got data on hdd and metadata on ssd. the cephfs md cache is : 32Go 128pg for data and metadata (this is has been setup by auto balancer) Perhaps I can fix the pg num for each of cephfs pool and prevent autobalancer to run for them. what do you think ? thx you for your help and advices. UPDATE : I increase the pg number to 256 for data and 1024 for metadata Here the df during the backup started since 30min POOL ID STORED OBJECTS USED %USED MAX AVAIL cephfs-metadata 12 183 GiB 514.68k 550 GiB 7.16 2.3 TiB Before the backup the stored was 20GiB oau

2 3

Changing IP addresses
by Jean-Marc FONTANA 06 Apr '21

06 Apr '21

Hello everyone, We have installed a Nautilus Ceph cluster with 3 monitors, 5 osd and 1 RGW gateway. It works but now, we need to change the IP addresses of these machines to put them in DMZ. Are there any recommandations to go about doing this ? Best regards,

2 1

Problem using advanced OSD layout in octopus
by Gary Molenkamp 06 Apr '21

06 Apr '21

Before I upgrade our Nautilus ceph to Octopus, I would like to make sure I am able to replace existing OSDs when they fail. However, I have not been able to create an OSD in Octopus with the layout we are using in Nautilus. I am testing this on a VM cluster so as not to touch any production systems. Our existing servers partition one SSD as block_db for 4 HDD OSDs. ie: OST.0 block: /dev/sda, block.db: /dev/sdn1 OST.1 block: /dev/sdb, block.db: /dev/sdn2 OST.2 block: /dev/sdc, block.db: /dev/sdn3 OST.3 block: /dev/sdd, block.db: /dev/sdn4 etc. My understanding is that I will need to apply an advanced osd service to achieve this layout. As each server is is similar the above, I could create 4 services (osd.disk0 - osd.disk3) and apply it to each host. I tried something similar to this: service_type: osd service_id: disk0 placement: host_pattern: 'storage*' data_devices: paths: - /dev/sda db_devices: paths: - /dev/sdn1 But the yaml was rejected with "Exception: Failed to validate Drive Group: `paths` is only allowed for data_devices", although it appears to be valid in the data structures here: https://docs.ceph.com/en/latest/cephadm/osd/#deploy-osds https://people.redhat.com/bhubbard/nature/default/mgr/orchestrator_modules/ I tried to use a combination of size and db_slots for the db_device, but I could not get the OSD to put the block.db on the separate device. Is this possible using the advanced placement method to orchestration, or should I just focus on using "cephadm ceph-volume" to create the OSDs in the desired layout? NOTE: I am trying to avoid installing ceph directly on the host OS (to use ceph-volume) as I do like the containerized approach in octopus. Thank you, Gary -- Gary Molenkamp Computer Science/Science Technology Services Systems Administrator University of Western Ontario molenkam(a)uwo.ca http://www.csd.uwo.ca (519) 661-2111 x86882 (519) 661-3566

2 2

bug in ceph-volume create
by Philip Brown 06 Apr '21

06 Apr '21

I would file this as a potential bug.. but it takes too long to get approved, and tracker.ceph.com doesnt have straightfoward google signin enabled :-/ I believe that with the new lvm mandate, ceph-volume should not be complaining about "missing PARTUUID". This is stopping me from using my system. Details on how to recreate: 1. have a system with 1 SSD and multiple HDDS 2. create a buncha OSDs with your preferred frontend, which will eventualy come down to ceph-volume lvm batch --bluestore /dev/ssddevice /dev/sdA ... /dev/sdX THIS will work great. batch mode will appropriately carve up the SSD device into multiple LVMs, and allocate one of them to be a DB device for each of the HDDs. 3. try to repair/replace an HDD As soon as you have an HDD fail... you will need to recreate the OSD.. and you are then stuck. Because you cant use batch mode for it... and you cant do it more granularly, with ceph-volume --cluster ceph lvm create --bluestore --data /dev/sdg --block.db /dev/ceph-xx-xx-xx/ceph-osd-db-this-is-the-old-lvm-for-ssd here because ceph-volume will complain that, blkid could not detect a PARTUUID for device: /dev/ceph-xx-xx-xx/ceph-osd-db-this-is-the-old-lvm-for-ssd here but the lvm IS NOT SUPPOSED TO HAVE A PARTUUID. Which is provable first all by the fact that it isnt a partition. But secondly, that none of the other block-db LVMs it created on the SSD in batch mode, have an partuuid either!! So kindly quit checking for something that isnt supposed to be there in the first place?! (This is a bug all the way back in nautilus, through latest, I believe) -- Philip Brown| Sr. Linux System Administrator | Medata, Inc. 5 Peters Canyon Rd Suite 250 Irvine CA 92606 Office 714.918.1310| Fax 714.918.1325 pbrown(a)medata.com| www.medata.com

3 3

mkfs.xfs -f /dev/rbd0 hangs
by 展荣臻（信泰） 06 Apr '21

06 Apr '21

hi,all: I use 15.2.10 ceph cluster with ubuntu 18.04.I create a rbd device and map into a host（ubuntu 18.04 ceph 15.2.10）, When i type mkfs.fs -f /dev/rbd0 it is hangs,But it is ok that write data use "rados -p {poolname} put obj myfile". Has anyone encountered this kind of problem?

1 0

What is the upper limit of the numer of PGs in a ceph cluster
by huxiaoyu＠horebdata.cn 06 Apr '21

06 Apr '21

Dear Cepher, My client has 10 volume, each volume was assigned 8192 PGs, in total 81920 PGs. The ceph is with Luminous Bluestore. During a power outage, the cluster restarted, and we observed that OSD peering consumed a lot of CPU and memory resources, evne leading to some OSD flappings. My question is thus, 1) how to speed up OSD peering and avoid OSD flappings when there are a lot PGs in the Ceph cluster, and 2) Is there a practical limit on the number of PGs for a single cluster? best regards, Samuel huxiaoyu(a)horebdata.cn

1 0

RGW: Corrupted Bucket index with nautilus 14.2.16
by by morphin 06 Apr '21

06 Apr '21

Good morning. I have a bucket and it has 50M object in it. The bucket created with multisite sync and that is the masterzone and only zone now. After a health check, I saw weird objects in pending attr state. I've tried to remove them with "radosgw-admin object rm --bypas-gc" but I coldn't delete them. After sending delete command I see object attributes changing to pending state but it stays like that. Also I've tried rados rm and the object data is deleted and can't see with radosgw-admin stat. After that I tried to copy the object with rclone "source to dest" But I can not write the object in same path! Rclone gives error but the object is created anyway and I see 0 byte object with rados and radosgw-admin again! After that I've tried to copy the problematic object from /samebucket/objectpath/theObject --> /samebucket/theObject and the 0 byte object copied with success and its not 0 byte anymore! It looks like ok! Then after all these I wonder if its an index issue? I've run radosgw-admin bi list and I saw 29 entry for same object (I guess all these retries left most of them behind) Also I run bi list for a normal object and I see only 5 entry. (Versions). There is bi list for problematic object: https://paste.ubuntu.com/p/jRtH3cMC94/ I'm too scared of checking or re-creating with bucket index. Is there anyone have seen the same issue?

1 0

Pacific unable to configure NFS-Ganesha
by Robert Sander 05 Apr '21

05 Apr '21

Hi, I have a test cluster now running on Pacific with the cephadm orchestrator and upstream container images. In the Dashboard on the services tab I created a new service for NFS. The containers got deployed. But when I go to the NFS tab and try to create a new NFS share the Dashboard only returns a 500 error: Apr 05 19:38:49 ceph01 bash[35064]: debug 2021-04-05T17:38:49.146+0000 7f64468d1700 0 [dashboard ERROR exception] Internal Server Error Apr 05 19:38:49 ceph01 bash[35064]: Traceback (most recent call last): Apr 05 19:38:49 ceph01 bash[35064]: File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 46, in dashboard_exception_handler Apr 05 19:38:49 ceph01 bash[35064]: return handler(*args, **kwargs) Apr 05 19:38:49 ceph01 bash[35064]: File "/lib/python3.6/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__ Apr 05 19:38:49 ceph01 bash[35064]: return self.callable(*self.args, **self.kwargs) Apr 05 19:38:49 ceph01 bash[35064]: File "/usr/share/ceph/mgr/dashboard/controllers/__init__.py", line 694, in inner Apr 05 19:38:49 ceph01 bash[35064]: ret = func(*args, **kwargs) Apr 05 19:38:49 ceph01 bash[35064]: File "/usr/share/ceph/mgr/dashboard/controllers/nfsganesha.py", line 265, in fsals Apr 05 19:38:49 ceph01 bash[35064]: return Ganesha.fsals_available() Apr 05 19:38:49 ceph01 bash[35064]: File "/usr/share/ceph/mgr/dashboard/services/ganesha.py", line 154, in fsals_available Apr 05 19:38:49 ceph01 bash[35064]: if RgwClient.admin_instance().is_service_online() and \ Apr 05 19:38:49 ceph01 bash[35064]: File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 301, in admin_instance Apr 05 19:38:49 ceph01 bash[35064]: return RgwClient.instance(daemon_name=daemon_name) Apr 05 19:38:49 ceph01 bash[35064]: File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 241, in instance Apr 05 19:38:49 ceph01 bash[35064]: RgwClient._daemons = _get_daemons() Apr 05 19:38:49 ceph01 bash[35064]: File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 53, in _get_daemons Apr 05 19:38:49 ceph01 bash[35064]: raise NoRgwDaemonsException Apr 05 19:38:49 ceph01 bash[35064]: dashboard.services.rgw_client.NoRgwDaemonsException: No RGW service is running. Apr 05 19:38:49 ceph01 bash[35064]: debug 2021-04-05T17:38:49.150+0000 7f64468d1700 0 [dashboard ERROR request] [::ffff:10.0.44.42:39898] [GET] [500] [0.030s] [admin] [513.0B] /ui-api/nfs-ganesha/fsals Apr 05 19:38:49 ceph01 bash[35064]: debug 2021-04-05T17:38:49.150+0000 7f64468d1700 0 [dashboard ERROR request] [b'{"status": "500 Internal Server Error", "detail": "The server encountered an unexpected condition which prevented it from fulfilling the request.", "request_id": "e89b8519-352f-4e44-a364-6e6faf9dc533"} '] I have no radosgateways in that cluster (currently). There are the pools for radosgw (.rgw.root etc) but no running instance. Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

2 2

Cephfs: Migrating Data to a new Data Pool
by ceph＠fionera.de 05 Apr '21

05 Apr '21

Hello everyone, I currently have a CephFS running with about 60TB of Data. I created it with a replicated pool as default pool, an erasure coded one as additional data pool like it is described in the docs. Now I want to migrate the data from the replicated pool, to the new erasure coded one. I couldn't find any docs and was wondering if its even possible currently. Thank you very much, Fionera

3 4

understanding orchestration and cephadm
by Gary Molenkamp 05 Apr '21

05 Apr '21

Good morning all, I'm experimenting with ceph orchestration and cephadm after using ceph-deploy for several years, and I have a hopefully simple question. I've converted a basic nautilus cluster over to cephadm+orchestration and I tried adding, then removing a monitor. However, when I removed the host using 'ceph orch host rm', it removed two mons. I may have missed something in the adoption/upgrade that has left the cluster in a bad state. Any advice/pointers/clarification would be of assistance. Details: A nautilus cluster with two mons (I know this is not correct for quorum), a mgr, and a handful of osds. I went though the adoption process and enabled the ceph orch backend. [root@osdev-ctrl2 ~]# ceph orch ps NAME HOST STATUS REFRESHED AGE VERSION IMAGE NAME IMAGE ID CONTAINER ID mgr.osdev-ctrl2 osdev-ctrl2 running (18h) 50s ago 18h 15.2.10 docker.io/ceph/ceph:v15.2.10 5b724076c58f e73c19b51a09 mon.osdev-ctrl2 osdev-ctrl2 running (18h) 50s ago 18h 15.2.10 docker.io/ceph/ceph:v15.2.10 5b724076c58f a6bfc27221f0 mon.osdev-net1 osdev-net1 running (18h) 50s ago 18h 15.2.10 docker.io/ceph/ceph:v15.2.10 5b724076c58f f66e2bef3d44 osd.0 osdev-stor1 running (17h) 50s ago 17h 15.2.10 docker.io/ceph/ceph:v15.2.10 5b724076c58f ac59dbdc267c ... [root@osdev-ctrl2 ~]# ceph orch status Backend: cephadm Available: True [root@osdev-ctrl2 ~]# ceph orch host ls HOST ADDR LABELS STATUS osdev-ctrl2 osdev-ctrl2 mon mgr osdev-net1 osdev-net1 mon osdev-stor1 osdev-stor1 osd [root@osdev-ctrl2 ~]# ceph orch ls NAME RUNNING REFRESHED AGE PLACEMENT IMAGE NAME IMAGE ID mgr 1/1 9m ago 20h label:mgr docker.io/ceph/ceph:v15.2.10 5b724076c58f mon 2/2 9m ago 20h label:mon docker.io/ceph/ceph:v15.2.10 5b724076c58f I then added a new mon host: [root@osdev-ctrl2 ~]# ceph orch host add osdev-ctrl3 mon It did not spawn a mon container on osdev-ctrl3 until I defined the public network in the config: [root@osdev-ctrl2 ~]# ceph config set global public_network 10.10.10.0/24 At this point all is good with three running mon as expected. Now I wanted to delete the mon using [root@osdev-ctrl2 ~]# ceph orch host rm osdev-ctrl3 This had the effect of: 1. removing the osdev-ctrl3 mon from 'ceph orch ls' and 'ceph orch ps' 2. the mon on osdev-ctrl3 is still running, and is part of 'ceph -s' but reported as not managed by cephadm 3. (Big issue) the mon running on osdev-net1 was completely destroyed. Any ideas what is going on? Sorry for the long post, but I tried to be as clear as possible. -- Gary Molenkamp Computer Science/Science Technology Services Systems Administrator University of Western Ontario molenkam(a)uwo.ca http://www.csd.uwo.ca (519) 661-2111 x86882 (519) 661-3566

3 3

RGW S3 user.rgw.olh.pending - Can not overwrite on 0 byte objects rgw sync leftovers.
by by morphin 05 Apr '21

05 Apr '21

Hello. I was have one-way multisite S3 cluster and we've seen issues with rgw-sync due to sharding problems and I've stopped the multisite sync. This is not the topic just a knowledge about my story. I have some leftover 0 byte objects in destination and I'm trying to overwrite them with Rclone "path to path". But somehow I can not overwrite these objects. If I delete with rclone or rados rm and do rclone copy again, I got the result below. Rclone gives error but the object is created again "0 byte" with pending attrs. Why is this happening? I think somehow I need to clean these objects and copy from source again but how? What is "user.rgw.olh.pending" ? [root@SRV1]# radosgw-admin --id radosgw.prod1 object stat --bucket=mybucket --object=images/2019/05/29/ad4ba79c-bb66-4ff6-847a-09a1e0cff49f { "name": "images/2019/05/29/ad4ba79c-bb66-4ff6-847a-09a1e0cff49f", "size": 0, "tag": "713li30rvcrjfwhctx894mj7vf1wa1a8", "attrs": { "user.rgw.manifest": "", "user.rgw.olh.idtag": "v1m9jy4cjck38ptel09qebsbb10pe2af", "user.rgw.olh.info": "\u0001\u0001�", "user.rgw.olh.pending.00000000606b04728gs23ecq11b3i3l1": "\u0001\u0001\u0008", "user.rgw.olh.pending.00000000606b0472bfhdzxeb9wesd8t7": "\u0001\u0001\u0008", "user.rgw.olh.pending.00000000606b0472fv06t1dob3vmo4da": "\u0001\u0001\u0008", "user.rgw.olh.pending.00000000606b0472lql6c9o88rt211r9": "\u0001\u0001\u0008", "user.rgw.olh.ver": "" } } [root@SRV1]# rados listxattr -p prod.rgw.buckets.data c106b26b-xxx-xxxx-xxx-dee3ca5c0968.121384004.3_images/2019/05/29/ad4ba79c-bb66-4ff6-847a-09a1e0cff49f user.rgw.idtag user.rgw.olh.idtag user.rgw.olh.info user.rgw.olh.ver [root@SRV1]# rados -p prod.rgw.buckets.data stat c106b26b-xxx-xxxx-xxx-dee3ca5c0968.121384004.3_images/2019/05/29/ad4ba79c-bb66-4ff6-847a-09a1e0cff49f prod.rgw.buckets.data/c106b26b-xxx-xxxx-xxx-dee3ca5c0968.121384004.3_images/2019/05/29/ad4ba79c-bb66-4ff6-847a-09a1e0cff49f mtime 2021-04-05 17:10:55.000000, size 0

1 0

"unable to find any IP address in networks"
by Stephen Smith6 05 Apr '21

05 Apr '21

1 1

cephadm:: how to change the image for services
by Adrian Sevcenco 05 Apr '21

05 Apr '21

Hi! How/where can i change the image configured for a service? I tried to modify /var/lib/ceph/<fsid>/<service_name>/unit.{image,run} but after restarting ceph orch ps shows that the service use the same old image. What other configuration locations are there for the ceph components beside /etc/ceph (which is quite sparse) and /var/lib/ceph/<fsid> ? Thank you! Adrian

2 3

First 6 nodes cluster with Octopus
by mabi 05 Apr '21

05 Apr '21

Hello, I am planning to setup a small Ceph cluster for testing purpose with 6 Ubuntu nodes and have a few questions mostly regarding planning of the infra. 1) Based on the documentation the OS requirements mentions Ubuntu 18.04 LTS, is it ok to use Ubuntu 20.04 instead or should I stick with 18.04? 2) The documentation recommends using Cephadm for new deployments, so I will use that but I read that with Cephadm everything is running in containers, so is this the new way to go? Or is Ceph in containers kind of still experimental? 3) As I will be needing cephfs I will also need MDS servers so with a total of 6 nodes I am planning the following layout: Node 1: MGR+MON+MDS Node 2: MGR+MON+MDS Node 3: MGR+MON+MDS Node 4: OSD Node 5: OSD Node 6: OSD Does this make sense? I am mostly interested in stability and HA with this setup. 4) Is there any special kind of demand in terms of disks on the MGR+MON+MDS nodes? Or can I use have my OS disks on these nodes? As far as I understand the MDS will create a metadata pool on the OSDs. Thanks for the hints. Best, Mabi

4 7

Real world Timings of PG states
by Nico Schottelius 05 Apr '21

05 Apr '21

Good morning, I was wondering if there are any timing indications as to how long a PG should "usually" stay in a certain state? For instance, how long should a pg stay in - peering (seconds - minutes?) - activating (seconds?) - srubbing (+deep) The scrub process obviously depends on the number of objects in the PG, however is the same true for peering and activation? Since Nautilus we see longer (minutes long) peering states in the cluster, which we did not see before. Thanks for your input and hav a good start into the week! Best regards, Nico -- Sustainable and modern Infrastructures by ungleich.ch

1 0

Is metadata on SSD or bluestore cache better?
by Kai Börnert 05 Apr '21

05 Apr '21

Hi, I hope this mailgroup is ok for this kind of questions, if not please ignore. I'm currently in the process of planning a smaller ceph cluster mostly for cephfs use. The budget still allows for some SSD's in addition to the required harddisks. I see two options on how to use those, a) Make SSD only pools for the cephfs metadata b) Give every OSD a SSD for the bluestore cache I was not able to find any suggestions or benchmarks so far, does anyone has further resources or insight into those options? Greetings, Kai

2 1

Re: Upgrade and lost osds Operation not permitted
by Behzad Khoshbakhti 05 Apr '21

05 Apr '21

running as ceph user and not root. Following is the startup configuration which can be found via the https://paste.ubuntu.com/p/2kV8KhrRfV/. [Unit] Description=Ceph object storage daemon osd.%i PartOf=ceph-osd.target After=network-online.target local-fs.target time-sync.target Before=remote-fs-pre.target ceph-osd.target Wants=network-online.target local-fs.target time-sync.target remote-fs-pre.target ceph-osd.target [Service] Environment=CLUSTER=ceph EnvironmentFile=-/etc/default/ceph ExecReload=/bin/kill -HUP $MAINPID ExecStart=/usr/bin/ceph-osd -f --cluster ${CLUSTER} --id %i --setuser ceph --setgroup ceph ExecStartPre=/usr/libexec/ceph/ceph-osd-prestart.sh --cluster ${CLUSTER} --id %i LimitNOFILE=1048576 LimitNPROC=1048576 LockPersonality=true MemoryDenyWriteExecute=true # Need NewPrivileges via `sudo smartctl` NoNewPrivileges=false PrivateTmp=true ProtectClock=true ProtectControlGroups=true ProtectHome=true ProtectHostname=true ProtectKernelLogs=true ProtectKernelModules=true # flushing filestore requires access to /proc/sys/vm/drop_caches ProtectKernelTunables=false ProtectSystem=full Restart=on-failure RestartSec=10 RestrictSUIDSGID=true StartLimitBurst=3 StartLimitInterval=30min TasksMax=infinity [Install] WantedBy=ceph-osd.target When I issue the following command, the ceph osd starts successfully. However, when it is failed when launching from systemctl. root@osd03:~# /usr/bin/ceph-osd -f --cluster ceph --id 2 --setuser ceph --setgroup ceph 2021-04-05T11:24:08.823+0430 7f91772c5f00 -1 osd.2 496 log_to_monitors {default=true} 2021-04-05T11:24:09.943+0430 7f916f7b9700 -1 osd.2 496 set_numa_affinity unable to identify public interface 'ens160' numa node: (0) Success On Mon, Apr 5, 2021, 10:51 AM Behzad Khoshbakhti <khoshbakhtib(a)gmail.com> wrote: > running as ceph user > > On Mon, Apr 5, 2021, 10:49 AM Anthony D'Atri <anthony.datri(a)gmail.com> > wrote: > >> Running as root, or as ceph? >> >> > On Apr 4, 2021, at 3:51 AM, Behzad Khoshbakhti <khoshbakhtib(a)gmail.com> >> wrote: >> > >> > It worth mentioning as I issue the following command, the Ceph OSD >> starts >> > and joins the cluster: >> > /usr/bin/ceph-osd -f --cluster ceph --id 2 --setuser ceph --setgroup >> ceph >> > >> > >> > >> > On Sun, Apr 4, 2021 at 3:00 PM Behzad Khoshbakhti < >> khoshbakhtib(a)gmail.com> >> > wrote: >> > >> >> Hi all, >> >> >> >> As I have upgrade my Ceph cluster from 15.2.10 to 16.2.0, during the >> >> manual upgrade using the precompiled packages, the OSDs was down with >> the >> >> following messages: >> >> >> >> root@osd03:/var/lib/ceph/osd/ceph-2# ceph-volume lvm activate --all >> >> --> Activating OSD ID 2 FSID 2d3ffc61-e430-4b89-bcd4-105b2df26352 >> >> Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 >> >> Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph >> prime-osd-dir >> >> --dev >> >> >> /dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352 >> >> --path /var/lib/ceph/osd/ceph-2 --no-mon-config >> >> Running command: /usr/bin/ln -snf >> >> >> /dev/ceph-9d37674b-a269-4239-aa9e-66a3c74df76c/osd-block-2d3ffc61-e430-4b89-bcd4-105b2df26352 >> >> /var/lib/ceph/osd/ceph-2/block >> >> Running command: /usr/bin/chown -h ceph:ceph >> /var/lib/ceph/osd/ceph-2/block >> >> Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 >> >> Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 >> >> Running command: /usr/bin/systemctl enable >> >> ceph-volume@lvm-2-2d3ffc61-e430-4b89-bcd4-105b2df26352 >> >> Running command: /usr/bin/systemctl enable --runtime ceph-osd@2 >> >> Running command: /usr/bin/systemctl start ceph-osd@2 >> >> --> ceph-volume lvm activate successful for osd ID: 2 >> >> >> >> Content of /var/log/ceph/ceph-osd.2.log >> >> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 set uid:gid to 64045:64045 >> >> (ceph:ceph) >> >> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 ceph version 16.2.0 >> >> (0c2054e95bcd9b30fdd908a79ac1d8bbc3394442) pacific (stable), process >> >> ceph-osd, pid 5484 >> >> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 0 pidfile_write: ignore >> empty >> >> --pid-file >> >> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1* >> >> bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to >> open >> >> /var/lib/ceph/osd/ceph-2/block: (1) Operation not permitted* >> >> 2021-04-04T14:54:56.625+0430 7f4afbac0f00 -1 *** ERROR: unable to open >> >> OSD superblock on /var/lib/ceph/osd/ceph-2: (2) No such file or >> directory* >> >> >> >> >> >> root@osd03:/var/lib/ceph/osd/ceph-2# systemctl status ceph-osd@2 >> >> â— ceph-osd(a)2.service - Ceph object storage daemon osd.2 >> >> Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled; >> >> vendor preset: enabled) >> >> Active: failed (Result: exit-code) since Sun 2021-04-04 14:55:06 >> >> +0430; 50s ago >> >> Process: 5471 ExecStartPre=/usr/libexec/ceph/ceph-osd-prestart.sh >> >> --cluster ${CLUSTER} --id 2 (code=exited, status=0/SUCCESS) >> >> Process: 5484 ExecStart=/usr/bin/ceph-osd -f --cluster ${CLUSTER} >> --id >> >> 2 --setuser ceph --setgroup ceph (code=exited, status=1/FAILURE) >> >> Main PID: 5484 (code=exited, status=1/FAILURE) >> >> >> >> Apr 04 14:55:06 osd03 systemd[1]: ceph-osd(a)2.service: Scheduled >> restart >> >> job, restart counter is at 3. >> >> Apr 04 14:55:06 osd03 systemd[1]: Stopped Ceph object storage daemon >> osd.2. >> >> Apr 04 14:55:06 osd03 systemd[1]: ceph-osd(a)2.service: Start request >> >> repeated too quickly. >> >> Apr 04 14:55:06 osd03 systemd[1]: ceph-osd(a)2.service: Failed with >> result >> >> 'exit-code'. >> >> Apr 04 14:55:06 osd03 systemd[1]: Failed to start Ceph object storage >> >> daemon osd.2. >> >> root@osd03:/var/lib/ceph/osd/ceph-2# >> >> >> >> root@osd03:~# lsblk >> >> NAME MAJ:MIN RM SIZE RO TYPE >> MOUNTPOINT >> >> fd0 2:0 1 4K 0 disk >> >> loop0 7:0 0 55.5M 1 loop >> >> /snap/core18/1988 >> >> loop1 7:1 0 69.9M 1 loop >> >> /snap/lxd/19188 >> >> loop2 7:2 0 55.5M 1 loop >> >> /snap/core18/1997 >> >> loop3 7:3 0 70.4M 1 loop >> >> /snap/lxd/19647 >> >> loop4 7:4 0 32.3M 1 loop >> >> /snap/snapd/11402 >> >> loop5 7:5 0 32.3M 1 loop >> >> /snap/snapd/11107 >> >> sda 8:0 0 80G 0 disk >> >> ├─sda1 8:1 0 1M 0 part >> >> ├─sda2 8:2 0 1G 0 part /boot >> >> └─sda3 8:3 0 79G 0 part >> >> └─ubuntu--vg-ubuntu--lv 253:0 0 69.5G 0 lvm / >> >> sdb 8:16 0 16G 0 disk >> >> └─sdb1 8:17 0 16G 0 part >> >> >> >> >> └─ceph--9d37674b--a269--4239--aa9e--66a3c74df76c-osd--block--2d3ffc61--e430--4 >> >> b89--bcd4--105b2df26352 >> >> 253:1 0 16G 0 lvm >> >> root@osd03:~# >> >> >> >> root@osd03:/var/lib/ceph/osd/ceph-2# mount | grep -i ceph >> >> tmpfs on /var/lib/ceph/osd/ceph-2 type tmpfs (rw,relatime) >> >> root@osd03:/var/lib/ceph/osd/ceph-2# >> >> >> >> any help is much appreciated >> >> -- >> >> >> >> Regards >> >> Behzad Khoshbakhti >> >> Computer Network Engineer (CCIE #58887) >> >> >> >> >> > >> > -- >> > >> > Regards >> > Behzad Khoshbakhti >> > Computer Network Engineer (CCIE #58887) >> > +989128610474 >> > _______________________________________________ >> > ceph-users mailing list -- ceph-users(a)ceph.io >> > To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >>

1 0

Installation of Ceph on Ubuntu 18.04 TLS
by Michel Niyoyita 04 Apr '21

04 Apr '21

Dear Ceph users, Kindly help on how I can deploy ceph on ubuntu 18.04 TLS , I am learning Ceph from scratch , please your inputs are highly appreciated. Regards

3 3

Post
by Michel Niyoyita 03 Apr '21

03 Apr '21

micou12(a)gmail.com

1 0

Re: ceph orch update fails - got new digests
by Alexander Sporleder 03 Apr '21

03 Apr '21

1 0

Re: ceph orch update fails - got new digests
by Sage Weil 03 Apr '21

03 Apr '21

On Fri, Apr 2, 2021 at 12:08 PM Alexander Sporleder <asporleder(a)age.mpg.de> wrote: > > Hello Sage, thank you for your response! > > I had some problems updating 15.2.8 -> 15.2.9 but after updating Podman > to 3.0.1 and Ceph to 15.2.10 everything was fine again. > > Then I started the update 15.2.10 -> 16.2.0 and in the beging > everything worked well. But at some point the update got stucked and > something broke the dashboard (port is in use). I stopped the update > but it was not possible to start the update process again without the > loop. > > Now my mons, the mgr, a few OSDs and cephadm are V16.2.0. > > Unfortunately the mgr is not logging to a file after I converted the > cluster to cephadm. ceph config set global log_to_file true You might also need to chown -R 167.167 /var/log/ceph/*/. if you're on debian/ubuntu (the packages installed on the host may have fiddled with the ownership; the uid for fedora/rhel/centos is different than the one for debian/ubuntu). s

1 1

OSDs not starting after upgrade to pacific from 15.2.10
by Lomayani S. Laizer 03 Apr '21

03 Apr '21

Hello, I tried to upgrade one node in my cluster to pacific in Ubuntu 20.04.2 and kernel 5.8.0-49. After upgrade OSDs wont start giving below error message This non-cephadm cluster is running bluestore OSDs which many were created in Luminous. 2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 bluestore(/var/lib/ceph/osd/ceph-29/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-29/block: (1) Operation not permitted 2021-04-03T10:36:07.698+0300 7f9b8d075f00 -1 ESC[0;31m ** ERROR: unable to open OSD superblock on /var/lib/ceph/osd/ceph-29: (2) No such file or directoryESC[0m the directories are existing and have right permission. I have downgraded back to octopus and everything is running ok df -alh tmpfs 16G 28K 16G 1% /var/lib/ceph/osd/ceph-32 tmpfs 16G 28K 16G 1% /var/lib/ceph/osd/ceph-29 tmpfs 16G 28K 16G 1% /var/lib/ceph/osd/ceph-33 tmpfs 16G 28K 16G 1% /var/lib/ceph/osd/ceph-31 tmpfs 16G 28K 16G 1% /var/lib/ceph/osd/ceph-28 tmpfs 16G 52K 16G 1% /var/lib/ceph/osd/ceph-30

1 0

ceph orch update fails - got new digests
by Alexander Sporleder 02 Apr '21

02 Apr '21

Hello Ceph user list! I tried to update Ceph 15.2.10 to 16.2.0 via ceph orch. In the beginning everything seems to work fine and the new MGR and MONs where deployed. But now I enden up in a pulling loop and I am unable to fix the issue by my self. #ceph -W cephadm --watch-debu 2021-04-02T10:36:20.704960+0200 mgr.mon-a-02.tvcrfq [INF] Upgrade: Need to upgrade myself (mgr.mon-a-02.tvcrfq) 2021-04-02T10:36:21.837596+0200 mgr.mon-a-02.tvcrfq [INF] Upgrade: Pulling ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e0 0951949000 on mon-a-01 2021-04-02T10:36:24.591487+0200 mgr.mon-a-02.tvcrfq [INF] Upgrade: image ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e0 0951949000 pull on mon-a-01 got new digests ['docker.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6 c43120b5e5e00951949000', 'docker.io/ceph/ceph@sha256:9b04c0f15704c49591640a37c7adfd40ffad0a4b42f ecb950c3407687cb4f29a'] (not ['ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5 e00951949000']), restarting 5704c49591640a37c7adfd40ffad0a4b42fecb950c3407687cb4f29a']), restarting 2021-04-02T10:36:37.054786+0200 mgr.mon-a-02.tvcrfq [INF] Upgrade: Need to upgrade myself (mgr.mon-a-02.tvcrfq) 2021-04-02T10:36:38.419014+0200 mgr.mon-a-02.tvcrfq [INF] Upgrade: Pulling ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e0 0951949000 on mon-a-01 2021-04-02T10:36:41.172835+0200 mgr.mon-a-02.tvcrfq [INF] Upgrade: image ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5e0 0951949000 pull on mon-a-01 got new digests ['docker.io/ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6 c43120b5e5e00951949000', 'docker.io/ceph/ceph@sha256:9b04c0f15704c49591640a37c7adfd40ffad0a4b42f ecb950c3407687cb4f29a'] (not ['ceph/ceph@sha256:35b2786dc4cd535dd84f6a1a585503db4b43623ba6c43120b5e5 e00951949000']), restarting After I stoppend the update I got the following health error: Module 'cephadm' has failed: 'NoneType' object has no attribute 'target_digests' Thanks in advance! Best, Alex

2 2

Upmap balancer after node failure
by Andras Pataki 02 Apr '21

02 Apr '21

Dear ceph users, On one of our clusters I have some difficulties with the upmap balancer. We started with a reasonably well balanced cluster (using the balancer in upmap mode). After a node failure, we crush reweighted all the OSDs of the node to take it out of the cluster - and waited for the cluster to rebalance. Obviously, this significantly changes the crush map - hence the nice balance created by the balancer was gone. The recovery mostly completed - but some of the OSDs became too full - so we neded up with a few PGs that were backfill_toofull. The cluster has plenty of space (overall perhaps 65% full), only a few OSDs are >90% (we have backfillfull_ratio at 92%). The balancer refuses to change anything since the cluster is not clean. Yet - the cluster can't become clean without a few upmaps to help the top 3 or 4 most full OSDs. I would think this is a fairly common situation - trying to recover after some failure. Are there any recommendations on how to proceed? Obviously I can manually find and insert upmaps - but for a large cluster with tens of thousands of PGs, that isn't too practical. Is there a way to tell the balancer to still do something even though some PGs are undersized (with a quick look at the python module - I didn't see any)? The cluster is on Nautilus 14.2.15. Thanks, Andras

3 4

cephfs-top: "cluster ceph does not exist"
by Erwin Bogaard 02 Apr '21

02 Apr '21

Hi, just installed pacific on our test-cluster. This really is a minimal, but fully functional cluster. Everything works as expected, except for the new (and by me anticipated) cephfs-top. When I run that tool, it says: "cluster ceph does not exist" If I point it to the correct config file: # cephfs-top --conffile /etc/ceph/ceph.conf I still get the same error. Doesn't matter if I run this as the ceph user or as root. I added the "client.fstop"-user as required by the documentation. module "stats" is enabled and functioning (tested with "ceph fs perf stats"). Anyone any suggestions what might be wrong? Regards, Erwin

2 1

v16.2.0 Pacific released
by David Galloway 01 Apr '21

01 Apr '21

We're glad to announce the first release of the Pacific v16.2.0 stable series. There have been a lot of changes across components from the previous Ceph releases, and we advise everyone to go through the release and upgrade notes carefully. Major Changes from Octopus -------------------------- General ~~~~~~~ * Cephadm can automatically upgrade an Octopus cluster to Pacific with a single command to start the process. * Cephadm has improved significantly over the past year, with improved support for RGW (standalone and multisite), and new support for NFS and iSCSI. Most of these changes have already been backported to recent Octopus point releases, but with the Pacific release we will switch to backporting bug fixes only. * Packages are built for the following distributions: - CentOS 8 - Ubuntu 20.04 (Focal) - Ubuntu 18.04 (Bionic) - Debian Buster - Container image (based on CentOS 8) With the exception of Debian Buster, packages and containers are built for both x86_64 and aarch64 (arm64) architectures. Note that cephadm clusters may work on many other distributions, provided Python 3 and a recent version of Docker or Podman is available to manage containers. For more information, see `cephadm-host-requirements`. Dashboard ~~~~~~~~~ The `mgr-dashboard` brings improvements in the following management areas: * Orchestrator/Cephadm: - Host management: maintenance mode, labels. - Services: display placement specification. - OSD: disk replacement, display status of ongoing deletion, and improved health/SMART diagnostics reporting. * Official `mgr ceph api`: - OpenAPI v3 compliant. - Stability commitment starting from Pacific release. - Versioned via HTTP `Accept` header (starting with v1.0). - Thoroughly tested (>90% coverage and per Pull Request validation). - Fully documented. * RGW: - Multi-site synchronization monitoring. - Management of multiple RGW daemons and their resources (buckets and users). - Bucket and user quota usage visualization. - Improved configuration of S3 tenanted users. * Security (multiple enhancements and fixes resulting from a pen testing conducted by IBM): - Account lock-out after a configurable number of failed log-in attempts. - Improved cookie policies to mitigate XSS/CSRF attacks. - Reviewed and improved security in HTTP headers. - Sensitive information reviewed and removed from logs and error messages. - TLS 1.0 and 1.1 support disabled. - Debug mode when enabled triggers HEALTH_WARN. * Pools: - Improved visualization of replication and erasure coding modes. - CLAY erasure code plugin supported. * Alerts and notifications: - Alert triggered on MTU mismatches in the cluster network. - Favicon changes according cluster status. * Other: - Landing page: improved charts and visualization. - Telemetry configuration wizard. - OSDs: management of individual OSD flags. - RBD: per-RBD image Grafana dashboards. - CephFS: Dirs and Caps displayed. - NFS: v4 support only (v3 backward compatibility planned). - Front-end: Angular 10 update. RADOS ~~~~~ * Pacific introduces RocksDB sharding, which reduces disk space requirements. * Ceph now provides QoS between client I/O and background operations via the mclock scheduler. * The balancer is now on by default in upmap mode to improve distribution of PGs across OSDs. * The output of `ceph -s` has been improved to show recovery progress in one progress bar. More detailed progress bars are visible via the `ceph progress` command. RBD block storage ~~~~~~~~~~~~~~~~~ * Image live-migration feature has been extended to support external data sources. Images can now be instantly imported from local files, remote files served over HTTP(S) or remote S3 buckets in `raw` (`rbd export v1`) or basic `qcow` and `qcow2` formats. Support for `rbd export v2` format, advanced QCOW features and `rbd export-diff` snapshot differentials is expected in future releases. * Initial support for client-side encryption has been added. This is based on LUKS and in future releases will allow using per-image encryption keys while maintaining snapshot and clone functionality -- so that parent image and potentially multiple clone images can be encrypted with different keys. * A new persistent write-back cache is available. The cache operates in a log-structured manner, providing full point-in-time consistency for the backing image. It should be particularly suitable for PMEM devices. * A Windows client is now available in the form of `librbd.dll` and `rbd-wnbd` (Windows Network Block Device) daemon. It allows mapping, unmapping and manipulating images similar to `rbd-nbd`. * librbd API now offers quiesce/unquiesce hooks, allowing for coordinated snapshot creation. RGW object storage ~~~~~~~~~~~~~~~~~~ * Initial support for S3 Select. See `s3-select-feature-table` for supported queries. * Bucket notification topics can be configured as `persistent`, where events are recorded in rados for reliable delivery. * Bucket notifications can be delivered to SSL-enabled AMQP endpoints. * Lua scripts can be run during requests and access their metadata. * SSE-KMS now supports KMIP as a key management service. * Multisite data logs can now be deployed on `cls_fifo` to avoid large omap cluster warnings and make their trimming cheaper. See `rgw_data_log_backing`. CephFS distributed file system ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ * The CephFS MDS modifies on-RADOS metadata such that the new format is no longer backwards compatible. It is not possible to downgrade a file system from Pacific (or later) to an older release. * Multiple file systems in a single Ceph cluster is now stable. New Ceph clusters enable support for multiple file systems by default. Existing clusters must still set the "enable_multiple" flag on the FS. See also `cephfs-multifs`. * A new `mds_autoscaler` `ceph-mgr` plugin is available for automatically deploying MDS daemons in response to changes to the `max_mds` configuration. Expect further enhancements in the future to simplify and automate MDS scaling. * `cephfs-top` is a new utility for looking at performance metrics from CephFS clients. It is development preview quality and will have bugs. For more information, see `cephfs-top`. * A new `snap_schedule` `ceph-mgr` plugin provides a command toolset for scheduling snapshots on a CephFS file system. For more information, see `snap-schedule`. * First class NFS gateway support in Ceph is here! It's now possible to create scale-out ("active-active") NFS gateway clusters that export CephFS using a few commands. The gateways are deployed via cephadm (or Rook, in the future). For more information, see `cephfs-nfs`. * Multiple active MDS file system scrub is now stable. It is no longer necessary to set `max_mds` to 1 and wait for non-zero ranks to stop. Scrub commands can only be sent to rank 0: `ceph tell mds.<fs_name>:0 scrub start /path ...`. For more information, see `mds-scrub`. * Ephemeral pinning -- policy based subtree pinning -- is considered stable. `mds_export_ephemeral_random` and `mds_export_ephemeral_distributed` now default to true. For more information, see `cephfs-ephemeral-pinning`. * A new `cephfs-mirror` daemon is available to mirror CephFS file systems to a remote Ceph cluster. * A Windows client is now available for connecting to CephFS. This is offered through a new `ceph-dokan` utility which operates via the Dokan userspace API, similar to FUSE. For more information, see `ceph-dokan`. Upgrading from Octopus or Nautilus ---------------------------------- Before starting, make sure your cluster is stable and healthy (no down or recovering OSDs). (This is optional, but recommended.) Upgrading cephadm clusters ~~~~~~~~~~~~~~~~~~~~~~~~~~ If your cluster is deployed with cephadm (first introduced in Octopus), then the upgrade process is entirely automated. To initiate the upgrade, ceph orch upgrade start --ceph-version 16.2.0 The same process is used to upgrade to future minor releases. Upgrade progress can be monitored with `ceph -s` (which provides a simple progress bar) or more verbosely with ceph -W cephadm The upgrade can be paused or resumed with ceph orch upgrade pause # to pause ceph orch upgrade resume # to resume or canceled with ceph orch upgrade stop Note that canceling the upgrade simply stops the process; there is no ability to downgrade back to Octopus. Upgrading non-cephadm clusters ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If you cluster is running Octopus (15.2.x), you might choose to first convert it to use cephadm so that the upgrade to Pacific is automated (see above). For more information, see `cephadm-adoption`. #. Set the `noout` flag for the duration of the upgrade. (Optional, but recommended.): # ceph osd set noout #. Upgrade monitors by installing the new packages and restarting the monitor daemons. For example, on each monitor host,: # systemctl restart ceph-mon.target Once all monitors are up, verify that the monitor upgrade is complete by looking for the `octopus` string in the mon map. The command: # ceph mon dump | grep min_mon_release should report: min_mon_release 16 (pacific) If it doesn't, that implies that one or more monitors hasn't been upgraded and restarted and/or the quorum does not include all monitors. #. Upgrade `ceph-mgr` daemons by installing the new packages and restarting all manager daemons. For example, on each manager host,: # systemctl restart ceph-mgr.target Verify the `ceph-mgr` daemons are running by checking `ceph -s` # ceph -s ... services: mon: 3 daemons, quorum foo,bar,baz mgr: foo(active), standbys: bar, baz ... #. Upgrade all OSDs by installing the new packages and restarting the ceph-osd daemons on all OSD hosts: # systemctl restart ceph-osd.target Note that if you are upgrading from Nautilus, the first time each OSD starts, it will do a format conversion to improve the accounting for "omap" data. This may take a few minutes to as much as a few hours (for an HDD with lots of omap data). You can disable this automatic conversion with: # ceph config set osd bluestore_fsck_quick_fix_on_mount false You can monitor the progress of the OSD upgrades with the `ceph versions` or `ceph osd versions` commands: # ceph osd versions { "ceph version 14.2.5 (...) nautilus (stable)": 12, "ceph version 16.2.0 (...) pacific (stable)": 22, } #. Upgrade all CephFS MDS daemons. For each CephFS file system, #. Disable standby_replay: # ceph fs set <fs_name> allow_standby_replay false #. Reduce the number of ranks to 1. (Make note of the original number of MDS daemons first if you plan to restore it later.): # ceph status # ceph fs set <fs_name> max_mds 1 #. Wait for the cluster to deactivate any non-zero ranks by periodically checking the status: # ceph status #. Take all standby MDS daemons offline on the appropriate hosts with: # systemctl stop ceph-mds@<daemon_name> #. Confirm that only one MDS is online and is rank 0 for your FS: # ceph status #. Upgrade the last remaining MDS daemon by installing the new packages and restarting the daemon: # systemctl restart ceph-mds.target #. Restart all standby MDS daemons that were taken offline: # systemctl start ceph-mds.target #. Restore the original value of `max_mds` for the volume: # ceph fs set <fs_name> max_mds <original_max_mds> #. Upgrade all radosgw daemons by upgrading packages and restarting daemons on all hosts: # systemctl restart ceph-radosgw.target #. Complete the upgrade by disallowing pre-Pacific OSDs and enabling all new Pacific-only functionality: # ceph osd require-osd-release pacific #. If you set `noout` at the beginning, be sure to clear it with: # ceph osd unset noout #. Consider transitioning your cluster to use the cephadm deployment and orchestration framework to simplify cluster management and future upgrades. For more information on converting an existing cluster to cephadm, see `cephadm-adoption`. Post-upgrade ~~~~~~~~~~~~ #. Verify the cluster is healthy with `ceph health`. If your CRUSH tunables are older than Hammer, Ceph will now issue a health warning. If you see a health alert to that effect, you can revert this change with: ceph config set mon mon_crush_min_required_version firefly If Ceph does not complain, however, then we recommend you also switch any existing CRUSH buckets to straw2, which was added back in the Hammer release. If you have any 'straw' buckets, this will result in a modest amount of data movement, but generally nothing too severe.: ceph osd getcrushmap -o backup-crushmap ceph osd crush set-all-straw-buckets-to-straw2 If there are problems, you can easily revert with: ceph osd setcrushmap -i backup-crushmap Moving to 'straw2' buckets will unlock a few recent features, like the `crush-compat` `balancer <balancer>` mode added back in Luminous. #. If you did not already do so when upgrading from Mimic, we recommened you enable the new `v2 network protocol <msgr2>`, issue the following command: ceph mon enable-msgr2 This will instruct all monitors that bind to the old default port 6789 for the legacy v1 protocol to also bind to the new 3300 v2 protocol port. To see if all monitors have been updated,: ceph mon dump and verify that each monitor has both a `v2:` and `v1:` address listed. #. Consider enabling the `telemetry module <telemetry>` to send anonymized usage statistics and crash information to the Ceph upstream developers. To see what would be reported (without actually sending any information to anyone),: ceph mgr module enable telemetry ceph telemetry show If you are comfortable with the data that is reported, you can opt-in to automatically report the high-level cluster metadata with: ceph telemetry on The public dashboard that aggregates Ceph telemetry can be found at https://telemetry-public.ceph.com. For more information about the telemetry module, see `the telemetry documentation. Upgrade from pre-Nautilus releases (like Mimic or Luminous) ----------------------------------------------------------- You must first upgrade to Nautilus (14.2.z) or Octopus (15.2.z) before upgrading to Pacific. Notable Changes --------------- * A new library is available, libcephsqlite. It provides a SQLite Virtual File System (VFS) on top of RADOS. The database and journals are striped over RADOS across multiple objects for virtually unlimited scaling and throughput only limited by the SQLite client. Applications using SQLite may change to the Ceph VFS with minimal changes, usually just by specifying the alternate VFS. We expect the library to be most impactful and useful for applications that were storing state in RADOS omap, especially without striping which limits scalability. * New `bluestore_rocksdb_options_annex` config parameter. Complements `bluestore_rocksdb_options` and allows setting rocksdb options without repeating the existing defaults. * $pid expansion in config paths like `admin_socket` will now properly expand to the daemon pid for commands like `ceph-mds` or `ceph-osd`. Previously only `ceph-fuse`/`rbd-nbd` expanded `$pid` with the actual daemon pid. * The allowable options for some `radosgw-admin` commands have been changed. * `mdlog-list`, `datalog-list`, `sync-error-list` no longer accepts start and end dates, but does accept a single optional start marker. * `mdlog-trim`, `datalog-trim`, `sync-error-trim` only accept a single marker giving the end of the trimmed range. * Similarly the date ranges and marker ranges have been removed on the RESTful DATALog and MDLog list and trim operations. * ceph-volume: The `lvm batch` subcommand received a major rewrite. This closed a number of bugs and improves usability in terms of size specification and calculation, as well as idempotency behaviour and disk replacement process. Please refer to https://docs.ceph.com/en/latest/ceph-volume/lvm/batch/ for more detailed information. * Configuration variables for permitted scrub times have changed. The legal values for `osd_scrub_begin_hour` and `osd_scrub_end_hour` are 0 - 23. The use of 24 is now illegal. Specifying `0` for both values causes every hour to be allowed. The legal values for `osd_scrub_begin_week_day` and `osd_scrub_end_week_day` are 0 - 6. The use of 7 is now illegal. Specifying `0` for both values causes every day of the week to be allowed. * volume/nfs: Recently "ganesha-" prefix from cluster id and nfs-ganesha common config object was removed, to ensure consistent namespace across different orchestrator backends. Please delete any existing nfs-ganesha clusters prior to upgrading and redeploy new clusters after upgrading to Pacific. * A new health check, DAEMON_OLD_VERSION, will warn if different versions of Ceph are running on daemons. It will generate a health error if multiple versions are detected. This condition must exist for over mon_warn_older_version_delay (set to 1 week by default) in order for the health condition to be triggered. This allows most upgrades to proceed without falsely seeing the warning. If upgrade is paused for an extended time period, health mute can be used like this "ceph health mute DAEMON_OLD_VERSION --sticky". In this case after upgrade has finished use "ceph health unmute DAEMON_OLD_VERSION". * MGR: progress module can now be turned on/off, using the commands: `ceph progress on` and `ceph progress off`. * An AWS-compliant API: "GetTopicAttributes" was added to replace the existing "GetTopic" API. The new API should be used to fetch information about topics used for bucket notifications. * librbd: The shared, read-only parent cache's config option `immutable_object_cache_watermark` now has been updated to property reflect the upper cache utilization before space is reclaimed. The default `immutable_object_cache_watermark` now is `0.9`. If the capacity reaches 90% the daemon will delete cold cache. * OSD: the option `osd_fast_shutdown_notify_mon` has been introduced to allow the OSD to notify the monitor it is shutting down even if `osd_fast_shutdown` is enabled. This helps with the monitor logs on larger clusters, that may get many 'osd.X reported immediately failed by osd.Y' messages, and confuse tools. * The mclock scheduler has been refined. A set of built-in profiles are now available that provide QoS between the internal and external clients of Ceph. To enable the mclock scheduler, set the config option "osd_op_queue" to "mclock_scheduler". The "high_client_ops" profile is enabled by default, and allocates more OSD bandwidth to external client operations than to internal client operations (such as background recovery and scrubs). Other built-in profiles include "high_recovery_ops" and "balanced". These built-in profiles optimize the QoS provided to clients of mclock scheduler. * The balancer is now on by default in upmap mode. Since upmap mode requires `require_min_compat_client` luminous, new clusters will only support luminous and newer clients by default. Existing clusters can enable upmap support by running `ceph osd set-require-min-compat-client luminous`. It is still possible to turn the balancer off using the `ceph balancer off` command. In earlier versions, the balancer was included in the `always_on_modules` list, but needed to be turned on explicitly using the `ceph balancer on` command. * Version 2 of the cephx authentication protocol (`CEPHX_V2` feature bit) is now required by default. It was introduced in 2018, adding replay attack protection for authorizers and making msgr v1 message signatures stronger (CVE-2018-1128 and CVE-2018-1129). Support is present in Jewel 10.2.11, Luminous 12.2.6, Mimic 13.2.1, Nautilus 14.2.0 and later; upstream kernels 4.9.150, 4.14.86, 4.19 and later; various distribution kernels, in particular CentOS 7.6 and later. To enable older clients, set `cephx_require_version` and `cephx_service_require_version` config options to 1. * `blacklist` has been replaced with `blocklist` throughout. The following commands have changed: - `ceph osd blacklist ...` are now `ceph osd blocklist ...` - `ceph <tell|daemon> osd.<NNN> dump_blacklist` is now `ceph <tell|daemon> osd.<NNN> dump_blocklist` * The following config options have changed: - `mon osd blacklist default expire` is now `mon osd blocklist default expire` - `mon mds blacklist interval` is now `mon mds blocklist interval` - `mon mgr blacklist interval` is now ''mon mgr blocklist interval` - `rbd blacklist on break lock` is now `rbd blocklist on break lock` - `rbd blacklist expire seconds` is now `rbd blocklist expire seconds` - `mds session blacklist on timeout` is now `mds session blocklist on timeout` - `mds session blacklist on evict` is now `mds session blocklist on evict` * The following librados API calls have changed: - `rados_blacklist_add` is now `rados_blocklist_add`; the former will issue a deprecation warning and be removed in a future release. - `rados.blacklist_add` is now `rados.blocklist_add` in the C++ API. * The JSON output for the following commands now shows `blocklist` instead of `blacklist`: - `ceph osd dump` - `ceph <tell|daemon> osd.<N> dump_blocklist` * Monitors now have config option `mon_allow_pool_size_one`, which is disabled by default. However, if enabled, user now have to pass the `--yes-i-really-mean-it` flag to `osd pool set size 1`, if they are really sure of configuring pool size 1. * `ceph pg #.# list_unfound` output has been enhanced to provide might_have_unfound information which indicates which OSDs may contain the unfound objects. * OSD: A new configuration option `osd_compact_on_start` has been added which triggers an OSD compaction on start. Setting this option to `true` and restarting an OSD will result in an offline compaction of the OSD prior to booting. * OSD: the option named `bdev_nvme_retry_count` has been removed. Because in SPDK v20.07, there is no easy access to bdev_nvme options, and this option is hardly used, so it was removed. * Alpine build related script, documentation and test have been removed since the most updated APKBUILD script of Ceph is already included by Alpine Linux's aports repository. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-16.2.0.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: 0c2054e95bcd9b30fdd908a79ac1d8bbc3394442

3 2

Ceph User Survey Working Group - Next Steps
by Mike Perez 01 Apr '21

01 Apr '21

Hi everyone, We are approaching the April 2nd deadline in two weeks, so we should start proposing the next meeting to plan the survey results. Anybody in the community is welcome to join the Ceph Working Groups. Please add your name to: https://ceph.io/user-survey/ I have started a doodle: https://doodle.com/poll/y3t2ttdt8a3egz4v?utm_source=poll&utm_medium=link Please help promote the User Survey: https://twitter.com/Ceph/status/1369589099716349956 -- Mike Perez (thingee)

2 2

v14.2.19 Nautilus released
by David Galloway 01 Apr '21

01 Apr '21

This is the 19th update to the Ceph Nautilus release series. This is a hotfix release to prevent daemons from binding to loopback network interfaces. All nautilus users are advised to upgrade to this release. Notable Changes --------------- * This release fixes a regression introduced in v14.2.18 whereby in certain environments, OSDs will bind to 127.0.0.1. See https://tracker.ceph.com/issues/49938. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-14.2.19.tar.gz * For packages, see http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: bb796b9b5bab9463106022eef406373182465d11

3 5