- ceph-users - lists.ceph.io

Ceph Filesystem recovery with intact pools
by cyclic3.git＠gmail.com 02 Sep '20

02 Sep '20

Hi, I've had a complete monitor failure, which I have recovered from with the steps here: https://docs.ceph.com/docs/mimic/rados/troubleshooting/troubleshooting-mon/… The data and metadata pools are there and are completely intact, but ceph is reporting that there are no filesystems, where (before the failure) there was one. Is there any way of putting the filesystem back together again without having to resort to having to rebuild a new metadata pool with cephfs-data-scan? I'm on ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable) Thanks, Harlan

4 11

Octopus multisite centos 8 permission denied error
by Szabo, Istvan (Agoda) 02 Sep '20

02 Sep '20

Hi, Could someone help me what is the issue with our dployment steps please? Initial RGW Cluster_1 ===================================================================================ADD_RGW_TO_CLUSTERCreate Default Realm - sudo radosgw-admin realm create --rgw-realm=default --defaultCreate Default Zone Group - sudo radosgw-admin zonegroup create --rgw-zonegroup=default --master --defaultCreate Default Zone - sudo radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=default --master --defaultUpdate Period - sudo radosgw-admin period update --rgw-realm=default --commitCreate RGW on 3 OSD_Node - sudo ceph orch apply rgw default default --placement=“3 bk-otpsmon-1001 bk-otpsmon-1002 bk-otpsmon-1003” ===================================================================================Initial RGW Cluster_2 - sudo ceph orch apply rgw default default --placement=“3 hy-otpsmon-2001 hy-otpsmon-2002 bk-otpsmon-2003" HOW_TO_CONFIGURE_CEPH_OCTOPUS_MULTISITE =================================================================================================== configure multisite radosgw-admin realm create \ --rgw-realm=agoda \ --default radosgw-admin zonegroup create \ --rgw-zonegroup=data \ --endpoints=http://bk-otpsmon-1001:80,http://bk-otpsmon-1002:80,http://bk-otpsmon-1003:80 \ --rgw-realm=agoda \ --master \ --default ***** BUG Run Command-line => Can't send shot command-line ***** radosgw-admin zone create --rgw-zonegroup=data --rgw-zone=bk --master --default --endpoints=http://bk-otpsmon-1001:80,http://bk-otpsmon-1002:80,http://bk-otpsmon-1003:80 ***** BUG Can't Delete Pool ***** ***** ceph config set mon mon_allow_pool_delete true ***** ceph tell mon.\* injectargs '--mon-allow-pool-delete=true' ***** (optional) systemctl |grep -i ceph ***** (optional) systemctl restart ceph-85b0c358-e2a2-11ea-8864-000c29fa922a(a)mon.bk-otpsmon-1001.service # radosgw-admin zonegroup remove --rgw-zonegroup=default --rgw-zone=default # radosgw-admin period update --commit # radosgw-admin zone delete --rgw-zone=default # radosgw-admin period update --commit # radosgw-admin zonegroup delete --rgw-zonegroup=default # radosgw-admin period update --commit # ceph osd pool rm default.rgw.meta default.rgw.meta --yes-i-really-really-mean-it # ceph osd pool rm default.rgw.control default.rgw.control --yes-i-really-really-mean-it # ceph osd pool rm default.rgw.log default.rgw.log --yes-i-really-really-mean-it radosgw-admin user create --uid="ceph-sync" --display-name="ceph-sync" --system "user": "ceph-sync", "access_key": "VWV6566957QGVDV6ITJM", "secret_key": "qAEGXslUHBeWv7O6VMCmdo0z2AgMyBZlcKqg38H7" radosgw-admin zone modify \ --rgw-zone=bk \ --access-key=VWV6566957QGVDV6ITJM \ --secret=qAEGXslUHBeWv7O6VMCmdo0z2AgMyBZlcKqg38H7 radosgw-admin period update --rgw-realm=agoda --commit Configure Rados Gateway Client at RGW Nodes [client.rgw.bk-otpsmon-1001] host = bk-otpsmon-1001 rgw frontends = "civetweb port=80" rgw_zone=bk [client.rgw.bk-otpsmon-1002] host = bk-otpsmon-1002 rgw frontends = "civetweb port=80" rgw_zone=bk [client.rgw.bk-otpsmon-1003] host = bk-otpsmon-1003 rgw frontends = "civetweb port=80" rgw_zone=bk Restart All RGW Container Node rgw_1: systemctl restart ceph-89c72a6c-eb95-11ea-b88b-000c29147836(a)rgw.default.default.bk-otpsmon-1001.atdfmv.service rgw_2: systemctl restart ceph-89c72a6c-eb95-11ea-b88b-000c29147836(a)rgw.default.default.bk-otpsmon-1002.wkieqj.service rgw_3: systemctl restart ceph-89c72a6c-eb95-11ea-b88b-000c29147836(a)rgw.default.default.bk-otpsmon-1003.jpyzdq.service Enable All RGW Container Node rgw_1: systemctl enable ceph-89c72a6c-eb95-11ea-b88b-000c29147836(a)rgw.default.default.bk-otpsmon-1001.atdfmv.service rgw_2: systemctl enable ceph-89c72a6c-eb95-11ea-b88b-000c29147836(a)rgw.default.default.bk-otpsmon-1002.wkieqj.service rgw_3: systemctl enable ceph-89c72a6c-eb95-11ea-b88b-000c29147836(a)rgw.default.default.bk-otpsmon-1003.jpyzdq.service ===================================== Secondary Zones radosgw-admin realm pull --url={url-to-master-zone-gateway} --access-key={access-key} --secret={secret} radosgw-admin realm pull --url=http://bk-otpsmon-1001:80 --access-key=CL7NF0DLYL7D2YYVR9HA --secret=C49mBXDNgHl9fNwibdaQamvffB9QM2RNj5snUq03 radosgw-admin realm pull --url=http://bk-otpsmon-1001:80 --access-key=CL7NF0DLYL7D2YYVR9HA --secret=C49mBXDNgHl9fNwibdaQamvffB9QM2RNj5snUq03 --rgw-realm=agoda And here is the errors: Error From Cluster_2 After Realm Pull[root@hy-otpsmon-2001 ~]# radosgw-admin realm pull --url=http://bk-otpsmon-1001:80<http://bk-otpsmon-1001/> --access-key=CL7NF0DLYL7D2YYVR9HA --secret=C49mBXDNgHl9fNwibdaQamvffB9QM2RNj5snUq03 --rgw-realm=agoda request failed: (13) Permission denied If the realm has been changed on the master zone, the master zone's gateway may need to be restarted to recognize this user. [root@hy-otpsmon-2001 ~]# Error From Cluster_1 RGW Container LogSep 02 11:20:52 bk-otpsmon-1001 bash[1246]: debug 2020-09-02T04:20:52.963+0000 7f3b0747f700 1 ====== starting new request req=0x7f3b455118a0 ===== Sep 02 11:20:52 bk-otpsmon-1001 bash[1246]: debug 2020-09-02T04:20:52.967+0000 7f3b0747f700 1 op->ERRORHANDLER: err_no=-2028 new_err_no=-2028 Sep 02 11:20:52 bk-otpsmon-1001 bash[1246]: debug 2020-09-02T04:20:52.967+0000 7f3b0747f700 1 ====== req done req=0x7f3b455118a0 op status=0 http_status=403 latency=0.002999559s ====== All help appreciated. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 1

Essay Conclusion
by christwilliam26＠gmail.com 02 Sep '20

02 Sep '20

No student should have to feel stressed about their assignment deadlines! With total assignment help you can get professional Essay Conclusion every single time! https://www.totalassignmenthelp.com/blog/essay-conclusion/

1 0

how to rescue a cluster that is full filled
by chen kael 02 Sep '20

02 Sep '20

I have a self compiled ceph cluster that base on v14.2.9, and test writing to a pool until it`s full, after that osds start to panic, and no longer can be restarted. osd config: "mon_osd_nearfull_ratio": "0.850000", "mon_osd_full_ratio": "0.950000" "osd_failsafe_full_ratio": "0.970000", Although osd_failsafe_full_ratio is set to 0.97 but indeed %USE of osd can reach up to 99.7% I have tried to enlarge the LVM volume of osd and then use tools "ceph-bluestore-tool bluefs-bdev-expand" to notify bluestore. some osds were success started, but some failed with logs below log output: [root@oss-smaug-node4 ~]# ceph-bluestore-tool bluefs-bdev-expand --path /var/lib/ceph/osd/ceph-28/ inferring bluefs devices from bluestore path 2020-09-02 15:47:39.594 7fb33d836c00 -1 bluestore(/var/lib/ceph/osd/ceph-28) allocate_bluefs_freespace failed to allocate on 0x2ad0000 min_size 0x2ad0000 > allocated total 0x720000 bluefs_shared_alloc_size 0x10000 allocated 0x720000 available 0x b000 2020-09-02 15:47:39.594 7fb33d836c00 -1 bluefs _allocate failed to expand slow device to fit +0x2ac85c3 2020-09-02 15:47:39.594 7fb33d836c00 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: 0x2ac85c3 /root/rpmbuild/BUILD/ceph-1.0.0-5-g7a08a9e/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7fb33d836c00 time 2020-09-02 15:47:39.596122 /root/rpmbuild/BUILD/ceph-1.0.0-5-g7a08a9e/src/os/bluestore/BlueFS.cc: 2269: ceph_abort_msg("bluefs enospc") ceph version 1.0.0-5-g7a08a9e (7a08a9eaefdc4e8786cfbfcf4eb387a6e603c13c) nautilus (stable) 1: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0xdd) [0x7fb3339f3ef4] 2: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1cd7) [0x564bab3dcaf7] 3: (BlueFS::_flush(BlueFS::FileWriter*, bool)+0x10b) [0x564bab3dcceb] 4: (BlueRocksWritableFile::Flush()+0x3d) [0x564bab599afd] 5: (rocksdb::WritableFileWriter::Flush()+0x22c) [0x564bab7c0fdc] 6: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x564bab7c119e] 7: (rocksdb::BuildTable(std::string const&, rocksdb::Env*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::EnvOptions const&, rocksdb::TableCache*, rocksdb::InternalIteratorBase<rocksdb::Slice>*, std::vector<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> >, std::allocator<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> > > >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::string const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, unsigned long, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint)+0x2392) [0x564bab7ecb82] 8: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xb5f) [0x564bab686cdf] 9: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0x1cc7) [0x564bab6890d7] 10: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0xb23) [0x564bab689f93] 11: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool, bool)+0x9f7) [0x564bab684be7] 12: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x24) [0x564bab685eb4] 13: (RocksDBStore::do_open(std::ostream&, bool, bool, std::vector<KeyValueDB::ColumnFamily, std::allocator<KeyValueDB::ColumnFamily> > const*)+0xcf9) [0x564bab614689] 14: (BlueStore::_open_db(bool, bool, bool)+0xa41) [0x564bab429c61] 15: (BlueStore::_open_db_and_around(bool)+0x17e) [0x564bab43f6be] 16: (BlueStore::_mount(bool, bool)+0x5c2) [0x564bab482722] 17: (BlueStore::expand_devices(std::ostream&)+0x36) [0x564bab482bd6] 18: (main()+0x24a3) [0x564bab398903] 19: (__libc_start_main()+0xf5) [0x7fb3306923d5] 20: (()+0x1f59ef) [0x564bab3be9ef] obviously before expand take effect , _mount failed. Is there any other way to rescue this osds? I think this accident is likely happend in production Any suggestions would be greatly appreciated

1 0

Understanding op_r, op_w vs op_rw
by Mark Kirkwood 02 Sep '20

02 Sep '20

Hi, I'd like to gain a better understanding about what operations emit which of these performance counters, in particular when is 'op_rw' incremented instead of 'op_r' + 'op_w'? I've done a little bit of investigation (v12.2.13) , running various workoads and operations against an RBD volume (in a cluster with no other client activity): - Most RBD 'operations' (create, rm, features disable/enable, map, unmap) emit 'op_rw' and often 'op_w' too - Program reads and writes against a mounted RBD volume *only* emit 'op_r' and 'op_w' (never 'op_rw'), regardless of whether they are 'read + modify' of existing file data (or whether the writes are buffered, direct or sync) Is that correct? Or have I missed a program driven workload that will produce 'op_rw'? [1] In our production clusters I'm seeing similar numbers of 'op_w' and 'op_rw' (for a given OSD), which would suggest a lot of RBD operations if it is only them that cause 'op_rw' counters to be emitted. Cheers Mark [1] Tested using fio and pgbench (database benchmark). I mounted the volume using the kernel driver (I'll do some more experimentation using librbd)

1 2

Rbd image corrupt or locked somehow
by Salsa 01 Sep '20

01 Sep '20

Hi, I have set a 3 host cluster with 30 OSDs total. Cluster has health OK and no warning whatsoever. I set an RBD pool and 14 images which werer all rbd-mirrored to a second cluster (which was disconnected since problems began) and also an iSCSI interface. Then I connected a Windows 2019 Server through iSCSI, mounted all 14 drives and created a spanned volume with all the drives. Everything was working fine, but I had to disconnect the server, so I disconnected the iSCSI interface and when I tried to reconnect my volume was unusable and drives seemed stuck. I ended rebooting each cluster node and then later, since I still couldn't use my images, removed and recreated all images. in this second run all was good and I had a robocopy syncing files for almost a week to my ceph cluster and had copied more than 5TB of data already when my Windows Server got stuck. Still not sure why it got stuck, but some services like FTP were responding but others, including login, were not. So I reset Windows server and when it was back up my spanned volume was bad again. I've been trying to recover it for the last 2 days but without success. Right now all images are disconnected, I have no locks (found some at some point and removed, but not sure who was locking) and no watchers in any of the images, but the 3 images that had data in it are corrupt or locked somehow. Nothing I try works on them and the operation gets stuck. I can edit the images' config, but not these 3. I can create snapshots, but not these 3. I managed to mount images using iSCSI in a Linux box, but these 3 get Linux commands (fdisk, parted) hanging. Ceph dashboard shows some stats like read and write rate for all images, but these 3. It seems something inside the image is broken or stuck but as I said no locks on them. I tried a lot of options and somehow my cluster now has some RGW pools that I have no idea where they came from. Any idea what I should do? -- Salsa

1 0

cephadm daemons vs cephadm services -- what's the difference?
by John Zachary Dover 01 Sep '20

01 Sep '20

What is the difference between services and daemons? Specifically, what does it mean that "orch ps" lists cephadm daemons and "orch ls" lists cephadm services? This question will help me close this bug: https://tracker.ceph.com/issues/47142 Zac Dover Upstream Docs Ceph

2 1

osd regularly wrongly marked down
by Francois Legrand 01 Sep '20

01 Sep '20

Hi all, We have a ceph cluster in production with 6 osds servers (with 16x8TB disks), 3 mons/mgrs and 3 mdss. Both public and cluster networks are in 10GB and works well. After a major crash in april, we turned the option bluefs_buffered_io to false to workaround the large write bug when bluefs_buffered_io was true (we were in version 14.2.8 and the default value at this time was true). Since that time, we regularly have some osds wrongly marked down by the cluster after heartbeat timeout (heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f03f1384700' had timed out after 15). Generally the osd restart and the cluster is back healthy, but several time, after many of these kick-off the osd reach the osd_op_thread_suicide_timeout and goes down definitely. We increased the osd_op_thread_timeout and osd_op_thread_suicide_timeout... The problems still occurs (but less frequently). Few days ago, we upgraded to 14.2.11 and revert the timeout to their default value, hoping that it will solve the problem (we thought that it should be related to this bug https://tracker.ceph.com/issues/45943) but it didn't. We still have some osds wrongly marked down. Can somebody help us to fix this problem ? Thanks. Here is an extract of an osd log at failure time: --------------------------------- 2020-08-28 02:19:05.019 7f03f1384700 0 log_channel(cluster) log [DBG] : 44.7d scrub starts 2020-08-28 02:19:25.755 7f040e43d700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f03f1384700' had timed out after 15 2020-08-28 02:19:25.755 7f040dc3c700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f03f1384700' had timed out after 15 this last line is repeated more than 1000 times ... 2020-08-28 02:20:17.484 7f040d43b700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f03f1384700' had timed out after 15 2020-08-28 02:20:17.551 7f03f1384700 0 bluestore(/var/lib/ceph/osd/ceph-16) log_latency_fn slow operation observed for _collection_list, latency = 67.3532s, lat = 67s cid =44.7d_head start GHMAX end GHMAX max 25 ... 2020-08-28 02:20:22.600 7f040dc3c700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f03f1384700' had timed out after 15 2020-08-28 02:21:20.774 7f03f1384700 0 bluestore(/var/lib/ceph/osd/ceph-16) log_latency_fn slow operation observed for _collection_list, latency = 63.223s, lat = 63s cid =44.7d_head start #44:beffc78d:::rbd_data.1e48e8ab988992.00000000000011bd:0# end #MAX# max 2147483647 2020-08-28 02:21:20.774 7f03f1384700 1 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7f03f1384700' had timed out after 15 2020-08-28 02:21:20.805 7f03f1384700 0 log_channel(cluster) log [DBG] : 44.7d scrub ok 2020-08-28 02:21:21.099 7f03fd997700 0 log_channel(cluster) log [WRN] : Monitor daemon marked osd.16 down, but it is still running 2020-08-28 02:21:21.099 7f03fd997700 0 log_channel(cluster) log [DBG] : map e609411 wrongly marked me down at e609410 2020-08-28 02:21:21.099 7f03fd997700 1 osd.16 609411 start_waiting_for_healthy 2020-08-28 02:21:21.119 7f03fd997700 1 osd.16 609411 start_boot 2020-08-28 02:21:21.124 7f03f0b83700 1 osd.16 pg_epoch: 609410 pg[36.3d0( v 609409'481293 (449368'478292,609409'481293] local-lis/les=609403/609404 n=154651 ec=435353/435353 lis/c 609403/609403 les/c/f 609404/609404/0 609410/609410/608752) [25,72] r=-1 lpr=609410 pi=[609403,609410)/1 luod=0'0 lua=609392'481198 crt=609409'481293 lcod 609409'481292 active mbc={}] start_peering_interval up [25,72,16] -> [25,72], acting [25,72,16] -> [25,72], acting_primary 25 -> 25, up_primary 25 -> 25, role 2 -> -1, features acting 4611087854031667199 upacting 4611087854031667199 ... 2020-08-28 02:21:21.166 7f03f0b83700 1 osd.16 pg_epoch: 609411 pg[36.56( v 609409'480511 (449368'477424,609409'480511] local-lis/les=609403/609404 n=153854 ec=435353/435353 lis/c 609403/609403 les/c/f 609404/609404/0 609410/609410/609410) [103,102] r=-1 lpr=609410 pi=[609403,609410)/1 crt=609409'480511 lcod 609409'480510 unknown NOTIFY mbc={}] state<Start>: transitioning to Stray 2020-08-28 02:21:21.307 7f04073b0700 1 osd.16 609413 set_numa_affinity public network em1 numa node 0 2020-08-28 02:21:21.307 7f04073b0700 1 osd.16 609413 set_numa_affinity cluster network em2 numa node 0 2020-08-28 02:21:21.307 7f04073b0700 1 osd.16 609413 set_numa_affinity objectstore and network numa nodes do not match 2020-08-28 02:21:21.307 7f04073b0700 1 osd.16 609413 set_numa_affinity not setting numa affinity 2020-08-28 02:21:21.566 7f040a435700 1 osd.16 609413 tick checking mon for new map 2020-08-28 02:21:22.515 7f03fd997700 1 osd.16 609414 state: booting -> active 2020-08-28 02:21:22.515 7f03f0382700 1 osd.16 pg_epoch: 609414 pg[36.20( v 609409'483167 (449368'480117,609409'483167] local-lis/les=609403/609404 n=155171 ec=435353/435353 lis/c 609403/609403 les/c/f 609404/609404/0 609414/609414/609361) [97,16,72] r=1 lpr=609414 pi=[609403,609414)/1 crt=609409'483167 lcod 609409'483166 unknown NOTIFY mbc={}] start_peering_interval up [97,72] -> [97,16,72], acting [97,72] -> [97,16,72], acting_primary 97 -> 97, up_primary 97 -> 97, role -1 -> 1, features acting 4611087854031667199 upacting 4611087854031667199 ... 2020-08-28 02:21:22.522 7f03f1384700 1 osd.16 pg_epoch: 609414 pg[36.2f3( v 609409'479796 (449368'476712,609409'479796] local-lis/les=609403/609404 n=154451 ec=435353/435353 lis/c 609403/609403 les/c/f 609404/609404/0 609414/609414/609414) [16,34,21] r=0 lpr=609414 pi=[609403,609414)/1 crt=609409'479796 lcod 609409'479795 mlcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [34,21] -> [16,34,21], acting [34,21] -> [16,34,21], acting_primary 34 -> 16, up_primary 34 -> 16, role -1 -> 0, features acting 4611087854031667199 upacting 4611087854031667199 2020-08-28 02:21:22.522 7f03f1384700 1 osd.16 pg_epoch: 609414 pg[36.2f3( v 609409'479796 (449368'476712,609409'479796] local-lis/les=609403/609404 n=154451 ec=435353/435353 lis/c 609403/609403 les/c/f 609404/609404/0 609414/609414/609414) [16,34,21] r=0 lpr=609414 pi=[609403,609414)/1 crt=609409'479796 lcod 609409'479795 mlcod 0'0 unknown mbc={}] state<Start>: transitioning to Primary 2020-08-28 02:21:24.738 7f03f1384700 0 log_channel(cluster) log [DBG] : 36.2f3 scrub starts 2020-08-28 02:22:18.857 7f03f1384700 0 log_channel(cluster) log [DBG] : 36.2f3 scrub ok

3 4

Avail Online Home Help Services From Qualified Experts
by amarasmith488＠gmail.com 01 Sep '20

01 Sep '20

Are you unable to complete your homework assignment? Are you looking for a reliable online homework help service provider? LiveWebTutors is one of the best and most reliable companies when it comes to providing an assignment writing service. You can connect with the experts whether you are in need of a dissertation writing service or coursework help service. The professionals will ensure that your writing needs are covered within the given deadline and that too as per the instructions stated by the college professor! So, get your grades better by connecting with the professionals of LiveWebTutors now! Visit us: https://www.livewebtutors.com/usa/coursework-help

1 0

RandomCrashes on OSDs Attached to Mon Hosts with Octopus
by Denis Krienbühl 01 Sep '20

01 Sep '20

Hi! We've recently upgraded all our clusters from Mimic to Octopus (15.2.4). Since then, our largest cluster is experiencing random crashes on OSDs attached to the mon hosts. This is the crash we are seeing (cut for brevity, see links in post scriptum): { "ceph_version": "15.2.4", "utsname_release": "4.15.0-72-generic", "assert_condition": "r == 0", "assert_func": "void BlueStore::_txc_apply_kv(BlueStore::TransContext*, bool)", "assert_file": "/build/ceph-15.2.4/src/os/bluestore/BlueStore.cc <http://bluestore.cc/>", "assert_line": 11430, "assert_thread_name": "bstore_kv_sync", "assert_msg": "/build/ceph-15.2.4/src/os/bluestore/BlueStore.cc <http://bluestore.cc/>: In function 'void BlueStore::_txc_apply_kv(BlueStore::TransContext*, bool)' thread 7fc56311a700 time 2020-08-26T08:52:24.917083+0200\n/build/ceph-15.2.4/src/os/bluestore/BlueStore.cc <http://bluestore.cc/>: 11430: FAILED ceph_assert(r == 0)\n", "backtrace": [ "(()+0x12890) [0x7fc576875890]", "(gsignal()+0xc7) [0x7fc575527e97]", "(abort()+0x141) [0x7fc575529801]", "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a5) [0x559ef9ae97b5]", "(ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x559ef9ae993f]", "(BlueStore::_txc_apply_kv(BlueStore::TransContext*, bool)+0x3a0) [0x559efa0245b0]", "(BlueStore::_kv_sync_thread()+0xbdd) [0x559efa07745d]", "(BlueStore::KVSyncThread::entry()+0xd) [0x559efa09cd3d]", "(()+0x76db) [0x7fc57686a6db]", "(clone()+0x3f) [0x7fc57560a88f]" ] } Right before the crash occurs, we see the following message in the crash log: -3> 2020-08-26T08:52:24.787+0200 7fc569b2d700 2 rocksdb: [db/db_impl_compaction_flush.cc:2212 <http://db_impl_compaction_flush.cc:2212/>] Waiting after background compaction error: Corruption: block checksum mismatch: expected 2548200440, got 2324967102 in db/815839.sst offset 67107066 size 3808, Accumulated background error counts: 1 -2> 2020-08-26T08:52:24.852+0200 7fc56311a700 -1 rocksdb: submit_common error: Corruption: block checksum mismatch: expected 2548200440, got 2324967102 in db/815839.sst offset 67107066 size 3808 code = 2 Rocksdb transaction: In short, we see a Rocksdb corruption error after background compaction, when this happens. When an OSD crashes, which happens about 10-15 times a day, it restarts and resumes work without any further problems. We are pretty confident that this is not a hardware issue, due to the following facts: * The crashes occur on 5 different hosts over 3 different racks. * There is no smartctl/dmesg output that could explain it. * It usually happens to a different OSD that did not crash before. Still we checked the following on a few OSDs/hosts: * We can do a manual compaction, both offline and online. * We successfully ran "ceph-bluestore-tool fsck --deep yes" on one of the OSDs. * We manually compacted a number of OSDs, one of which crashed hours later. The only thing we have noticed so far: It only happens to OSDs that are attached to a mon host. *None* of the non-mon host OSDs have had a crash! Does anyone have a hint what could be causing this? We currently have no good theory that could explain this, much less have a fix or workaround. Any help would be greatly appreciated. Denis Crash: https://public-resources.objects.lpg.cloudscale.ch/osd-crash/meta.txt <https://public-resources.objects.lpg.cloudscale.ch/osd-crash/meta.txt> Log: https://public-resources.objects.lpg.cloudscale.ch/osd-crash/log.txt <https://public-resources.objects.lpg.cloudscale.ch/osd-crash/log.txt>

7 11