September 2020 - ceph-users

Cheapest essay writing service
by omeojones＠gmail.com 03 Sep '20

03 Sep '20

One of the unique features of our portal is our 24/7 live chat support platform. Students from any USA location can use this facility to directly receive essay assignment tips from our easy essay writer. https://myassignmenthelp.com/us/cheap-essay-writing-services.html

1 0

cephadm grafana url
by Robert Sander 03 Sep '20

03 Sep '20

Hi, How do I set the correct URL to Grafana in a new cephadm bootstrapped cluster? When I try to access the performance parts of the Ceph dashboard my browser tells me that it cannot resolve the short hostname that is presented in the URL to Grafana. cephadm seems to use only the hostname and not the FQDN which is needed when using the Dashboard from the browser (at least in this setup). This obviously did not work: root@ceph01:~# ceph dashboard get-grafana-api-url https://ceph01:3000 root@ceph01:~# ceph dashboard set-grafana-api-url https://ceph01.ceph.heinlein-akademie.de:3000 Option GRAFANA_API_URL updated root@ceph01:~# ceph config get mgr mgr/dashboard/GRAFANA_API_URL https://ceph01:3000 root@ceph01:~# ceph config set mgr mgr/dashboard/GRAFANA_API_URL https://ceph01.ceph.heinlein-akademie.de:3000 root@ceph01:~# ceph config get mgr mgr/dashboard/GRAFANA_API_URL https://ceph01:3000 root@ceph01:~# ceph dashboard get-grafana-api-url https://ceph01:3000 Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

3 4

java client cannot visit rgw behind nginx
by Zhenshi Zhou 03 Sep '20

03 Sep '20

Hi My fellows wanna use ceph rgw to store ES backup and Nexus blobs. But the services cannot connect to the rgw with s3 protocol when I provided them with the frontend nginx address(virtual ip). Only when they use the backend rgw's address(real ip) the ES and Nexus works well with rgw. Has anyone meet the same issue? Thanks

3 7

failed to authpin, subtree is being exported in 14.2.11
by Dan van der Ster 03 Sep '20

03 Sep '20

Hi all, We had a stuck mds ops this morning on a 14.2.11 cephfs cluster. I tried to ls the path from another client and that blocked. The ops were like this: # egrep 'desc|flag|age' ops.txt "description": "client_request(client.1212755100:37475 lookup #0x1003e229d38/analytics-logs 2020-09-03 04:30:45.607640 caller_uid=2004, caller_gid=2004{})", "age": 4975.7069706660004, "flag_point": "failed to authpin, subtree is being exported", "description": "client_request(client.1212755100:37477 lookup #0x1003e229d38/bundled-plugins 2020-09-03 04:31:01.064591 caller_uid=2004, caller_gid=2004{})", "age": 4960.2499044630003, "flag_point": "failed to authpin, subtree is being exported", ... The full list of stuck ops is at https://termbin.com/8itv We had only just yesterday enabled 2 active mds's on this cluster. I don't have much info about this client, other than that it has been running for several weeks successfully until we enabled the 2nd active mds. To clear out the slow ops we evicted client.1212755100, and then we reduced back to 1 active mds so it doesn't happen again. Any ideas what might have triggered this? Cheers, Dan

2 2

OSD memory (buffer_anon) grows once writing stops
by Wido den Hollander 03 Sep '20

03 Sep '20

Hi, The cluster I'm writing about has a long history (months) of instability mainly related to large RocksDB database and high memory consumption. The use-case is RGW with an EC8+3 pool for data. In the last months this cluster has been suffering from OSDs using much more memory then osd_memory_target and mainly allocated in buffer_anon. After removing a lot of data from the cluster and re-installing all OSDs there is one thing remaining: High memory usage when *NOT* writing data to the cluster. There is a script running which keeps writing data to RADOS in a slow pace. Once this stops we are observing the memory usage of the OSDs grow steadily and also see the RocksDB databases of the BlueStore OSDs grow. Once we start to write again memory usage (buffer_annon) reduces. I think this is related to the pglogs, but even trimming all the pglogs does not solve this issue. Has anybody seen this before or has any clues where to start looking? Ceph version 14.2.8 Wido

1 0

Ceph Filesystem recovery with intact pools
by cyclic3.git＠gmail.com 02 Sep '20

02 Sep '20

Hi, I've had a complete monitor failure, which I have recovered from with the steps here: https://docs.ceph.com/docs/mimic/rados/troubleshooting/troubleshooting-mon/… The data and metadata pools are there and are completely intact, but ceph is reporting that there are no filesystems, where (before the failure) there was one. Is there any way of putting the filesystem back together again without having to resort to having to rebuild a new metadata pool with cephfs-data-scan? I'm on ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable) Thanks, Harlan

4 11

Octopus multisite centos 8 permission denied error
by Szabo, Istvan (Agoda) 02 Sep '20

02 Sep '20

Hi, Could someone help me what is the issue with our dployment steps please? Initial RGW Cluster_1 ===================================================================================ADD_RGW_TO_CLUSTERCreate Default Realm - sudo radosgw-admin realm create --rgw-realm=default --defaultCreate Default Zone Group - sudo radosgw-admin zonegroup create --rgw-zonegroup=default --master --defaultCreate Default Zone - sudo radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=default --master --defaultUpdate Period - sudo radosgw-admin period update --rgw-realm=default --commitCreate RGW on 3 OSD_Node - sudo ceph orch apply rgw default default --placement=“3 bk-otpsmon-1001 bk-otpsmon-1002 bk-otpsmon-1003” ===================================================================================Initial RGW Cluster_2 - sudo ceph orch apply rgw default default --placement=“3 hy-otpsmon-2001 hy-otpsmon-2002 bk-otpsmon-2003" HOW_TO_CONFIGURE_CEPH_OCTOPUS_MULTISITE =================================================================================================== configure multisite radosgw-admin realm create \ --rgw-realm=agoda \ --default radosgw-admin zonegroup create \ --rgw-zonegroup=data \ --endpoints=http://bk-otpsmon-1001:80,http://bk-otpsmon-1002:80,http://bk-otpsmon-1003:80 \ --rgw-realm=agoda \ --master \ --default ***** BUG Run Command-line => Can't send shot command-line ***** radosgw-admin zone create --rgw-zonegroup=data --rgw-zone=bk --master --default --endpoints=http://bk-otpsmon-1001:80,http://bk-otpsmon-1002:80,http://bk-otpsmon-1003:80 ***** BUG Can't Delete Pool ***** ***** ceph config set mon mon_allow_pool_delete true ***** ceph tell mon.\* injectargs '--mon-allow-pool-delete=true' ***** (optional) systemctl |grep -i ceph ***** (optional) systemctl restart ceph-85b0c358-e2a2-11ea-8864-000c29fa922a(a)mon.bk-otpsmon-1001.service # radosgw-admin zonegroup remove --rgw-zonegroup=default --rgw-zone=default # radosgw-admin period update --commit # radosgw-admin zone delete --rgw-zone=default # radosgw-admin period update --commit # radosgw-admin zonegroup delete --rgw-zonegroup=default # radosgw-admin period update --commit # ceph osd pool rm default.rgw.meta default.rgw.meta --yes-i-really-really-mean-it # ceph osd pool rm default.rgw.control default.rgw.control --yes-i-really-really-mean-it # ceph osd pool rm default.rgw.log default.rgw.log --yes-i-really-really-mean-it radosgw-admin user create --uid="ceph-sync" --display-name="ceph-sync" --system "user": "ceph-sync", "access_key": "VWV6566957QGVDV6ITJM", "secret_key": "qAEGXslUHBeWv7O6VMCmdo0z2AgMyBZlcKqg38H7" radosgw-admin zone modify \ --rgw-zone=bk \ --access-key=VWV6566957QGVDV6ITJM \ --secret=qAEGXslUHBeWv7O6VMCmdo0z2AgMyBZlcKqg38H7 radosgw-admin period update --rgw-realm=agoda --commit Configure Rados Gateway Client at RGW Nodes [client.rgw.bk-otpsmon-1001] host = bk-otpsmon-1001 rgw frontends = "civetweb port=80" rgw_zone=bk [client.rgw.bk-otpsmon-1002] host = bk-otpsmon-1002 rgw frontends = "civetweb port=80" rgw_zone=bk [client.rgw.bk-otpsmon-1003] host = bk-otpsmon-1003 rgw frontends = "civetweb port=80" rgw_zone=bk Restart All RGW Container Node rgw_1: systemctl restart ceph-89c72a6c-eb95-11ea-b88b-000c29147836(a)rgw.default.default.bk-otpsmon-1001.atdfmv.service rgw_2: systemctl restart ceph-89c72a6c-eb95-11ea-b88b-000c29147836(a)rgw.default.default.bk-otpsmon-1002.wkieqj.service rgw_3: systemctl restart ceph-89c72a6c-eb95-11ea-b88b-000c29147836(a)rgw.default.default.bk-otpsmon-1003.jpyzdq.service Enable All RGW Container Node rgw_1: systemctl enable ceph-89c72a6c-eb95-11ea-b88b-000c29147836(a)rgw.default.default.bk-otpsmon-1001.atdfmv.service rgw_2: systemctl enable ceph-89c72a6c-eb95-11ea-b88b-000c29147836(a)rgw.default.default.bk-otpsmon-1002.wkieqj.service rgw_3: systemctl enable ceph-89c72a6c-eb95-11ea-b88b-000c29147836(a)rgw.default.default.bk-otpsmon-1003.jpyzdq.service ===================================== Secondary Zones radosgw-admin realm pull --url={url-to-master-zone-gateway} --access-key={access-key} --secret={secret} radosgw-admin realm pull --url=http://bk-otpsmon-1001:80 --access-key=CL7NF0DLYL7D2YYVR9HA --secret=C49mBXDNgHl9fNwibdaQamvffB9QM2RNj5snUq03 radosgw-admin realm pull --url=http://bk-otpsmon-1001:80 --access-key=CL7NF0DLYL7D2YYVR9HA --secret=C49mBXDNgHl9fNwibdaQamvffB9QM2RNj5snUq03 --rgw-realm=agoda And here is the errors: Error From Cluster_2 After Realm Pull[root@hy-otpsmon-2001 ~]# radosgw-admin realm pull --url=http://bk-otpsmon-1001:80<http://bk-otpsmon-1001/> --access-key=CL7NF0DLYL7D2YYVR9HA --secret=C49mBXDNgHl9fNwibdaQamvffB9QM2RNj5snUq03 --rgw-realm=agoda request failed: (13) Permission denied If the realm has been changed on the master zone, the master zone's gateway may need to be restarted to recognize this user. [root@hy-otpsmon-2001 ~]# Error From Cluster_1 RGW Container LogSep 02 11:20:52 bk-otpsmon-1001 bash[1246]: debug 2020-09-02T04:20:52.963+0000 7f3b0747f700 1 ====== starting new request req=0x7f3b455118a0 ===== Sep 02 11:20:52 bk-otpsmon-1001 bash[1246]: debug 2020-09-02T04:20:52.967+0000 7f3b0747f700 1 op->ERRORHANDLER: err_no=-2028 new_err_no=-2028 Sep 02 11:20:52 bk-otpsmon-1001 bash[1246]: debug 2020-09-02T04:20:52.967+0000 7f3b0747f700 1 ====== req done req=0x7f3b455118a0 op status=0 http_status=403 latency=0.002999559s ====== All help appreciated. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 1

Essay Conclusion
by christwilliam26＠gmail.com 02 Sep '20

02 Sep '20

No student should have to feel stressed about their assignment deadlines! With total assignment help you can get professional Essay Conclusion every single time! https://www.totalassignmenthelp.com/blog/essay-conclusion/

1 0

how to rescue a cluster that is full filled
by chen kael 02 Sep '20

02 Sep '20

I have a self compiled ceph cluster that base on v14.2.9, and test writing to a pool until it`s full, after that osds start to panic, and no longer can be restarted. osd config: "mon_osd_nearfull_ratio": "0.850000", "mon_osd_full_ratio": "0.950000" "osd_failsafe_full_ratio": "0.970000", Although osd_failsafe_full_ratio is set to 0.97 but indeed %USE of osd can reach up to 99.7% I have tried to enlarge the LVM volume of osd and then use tools "ceph-bluestore-tool bluefs-bdev-expand" to notify bluestore. some osds were success started, but some failed with logs below log output: [root@oss-smaug-node4 ~]# ceph-bluestore-tool bluefs-bdev-expand --path /var/lib/ceph/osd/ceph-28/ inferring bluefs devices from bluestore path 2020-09-02 15:47:39.594 7fb33d836c00 -1 bluestore(/var/lib/ceph/osd/ceph-28) allocate_bluefs_freespace failed to allocate on 0x2ad0000 min_size 0x2ad0000 > allocated total 0x720000 bluefs_shared_alloc_size 0x10000 allocated 0x720000 available 0x b000 2020-09-02 15:47:39.594 7fb33d836c00 -1 bluefs _allocate failed to expand slow device to fit +0x2ac85c3 2020-09-02 15:47:39.594 7fb33d836c00 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: 0x2ac85c3 /root/rpmbuild/BUILD/ceph-1.0.0-5-g7a08a9e/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7fb33d836c00 time 2020-09-02 15:47:39.596122 /root/rpmbuild/BUILD/ceph-1.0.0-5-g7a08a9e/src/os/bluestore/BlueFS.cc: 2269: ceph_abort_msg("bluefs enospc") ceph version 1.0.0-5-g7a08a9e (7a08a9eaefdc4e8786cfbfcf4eb387a6e603c13c) nautilus (stable) 1: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0xdd) [0x7fb3339f3ef4] 2: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1cd7) [0x564bab3dcaf7] 3: (BlueFS::_flush(BlueFS::FileWriter*, bool)+0x10b) [0x564bab3dcceb] 4: (BlueRocksWritableFile::Flush()+0x3d) [0x564bab599afd] 5: (rocksdb::WritableFileWriter::Flush()+0x22c) [0x564bab7c0fdc] 6: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x564bab7c119e] 7: (rocksdb::BuildTable(std::string const&, rocksdb::Env*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::EnvOptions const&, rocksdb::TableCache*, rocksdb::InternalIteratorBase<rocksdb::Slice>*, std::vector<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> >, std::allocator<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> > > >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::string const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, unsigned long, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint)+0x2392) [0x564bab7ecb82] 8: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xb5f) [0x564bab686cdf] 9: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0x1cc7) [0x564bab6890d7] 10: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0xb23) [0x564bab689f93] 11: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool, bool)+0x9f7) [0x564bab684be7] 12: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x24) [0x564bab685eb4] 13: (RocksDBStore::do_open(std::ostream&, bool, bool, std::vector<KeyValueDB::ColumnFamily, std::allocator<KeyValueDB::ColumnFamily> > const*)+0xcf9) [0x564bab614689] 14: (BlueStore::_open_db(bool, bool, bool)+0xa41) [0x564bab429c61] 15: (BlueStore::_open_db_and_around(bool)+0x17e) [0x564bab43f6be] 16: (BlueStore::_mount(bool, bool)+0x5c2) [0x564bab482722] 17: (BlueStore::expand_devices(std::ostream&)+0x36) [0x564bab482bd6] 18: (main()+0x24a3) [0x564bab398903] 19: (__libc_start_main()+0xf5) [0x7fb3306923d5] 20: (()+0x1f59ef) [0x564bab3be9ef] obviously before expand take effect , _mount failed. Is there any other way to rescue this osds? I think this accident is likely happend in production Any suggestions would be greatly appreciated

1 0

Understanding op_r, op_w vs op_rw
by Mark Kirkwood 02 Sep '20

02 Sep '20

Hi, I'd like to gain a better understanding about what operations emit which of these performance counters, in particular when is 'op_rw' incremented instead of 'op_r' + 'op_w'? I've done a little bit of investigation (v12.2.13) , running various workoads and operations against an RBD volume (in a cluster with no other client activity): - Most RBD 'operations' (create, rm, features disable/enable, map, unmap) emit 'op_rw' and often 'op_w' too - Program reads and writes against a mounted RBD volume *only* emit 'op_r' and 'op_w' (never 'op_rw'), regardless of whether they are 'read + modify' of existing file data (or whether the writes are buffered, direct or sync) Is that correct? Or have I missed a program driven workload that will produce 'op_rw'? [1] In our production clusters I'm seeing similar numbers of 'op_w' and 'op_rw' (for a given OSD), which would suggest a lot of RBD operations if it is only them that cause 'op_rw' counters to be emitted. Cheers Mark [1] Tested using fio and pgbench (database benchmark). I mounted the volume using the kernel driver (I'll do some more experimentation using librbd)

1 2

Rbd image corrupt or locked somehow
by Salsa 01 Sep '20

01 Sep '20

Hi, I have set a 3 host cluster with 30 OSDs total. Cluster has health OK and no warning whatsoever. I set an RBD pool and 14 images which werer all rbd-mirrored to a second cluster (which was disconnected since problems began) and also an iSCSI interface. Then I connected a Windows 2019 Server through iSCSI, mounted all 14 drives and created a spanned volume with all the drives. Everything was working fine, but I had to disconnect the server, so I disconnected the iSCSI interface and when I tried to reconnect my volume was unusable and drives seemed stuck. I ended rebooting each cluster node and then later, since I still couldn't use my images, removed and recreated all images. in this second run all was good and I had a robocopy syncing files for almost a week to my ceph cluster and had copied more than 5TB of data already when my Windows Server got stuck. Still not sure why it got stuck, but some services like FTP were responding but others, including login, were not. So I reset Windows server and when it was back up my spanned volume was bad again. I've been trying to recover it for the last 2 days but without success. Right now all images are disconnected, I have no locks (found some at some point and removed, but not sure who was locking) and no watchers in any of the images, but the 3 images that had data in it are corrupt or locked somehow. Nothing I try works on them and the operation gets stuck. I can edit the images' config, but not these 3. I can create snapshots, but not these 3. I managed to mount images using iSCSI in a Linux box, but these 3 get Linux commands (fdisk, parted) hanging. Ceph dashboard shows some stats like read and write rate for all images, but these 3. It seems something inside the image is broken or stuck but as I said no locks on them. I tried a lot of options and somehow my cluster now has some RGW pools that I have no idea where they came from. Any idea what I should do? -- Salsa

1 0

cephadm daemons vs cephadm services -- what's the difference?
by John Zachary Dover 01 Sep '20

01 Sep '20

What is the difference between services and daemons? Specifically, what does it mean that "orch ps" lists cephadm daemons and "orch ls" lists cephadm services? This question will help me close this bug: https://tracker.ceph.com/issues/47142 Zac Dover Upstream Docs Ceph

2 1

osd regularly wrongly marked down
by Francois Legrand 01 Sep '20

01 Sep '20

Hi all, We have a ceph cluster in production with 6 osds servers (with 16x8TB disks), 3 mons/mgrs and 3 mdss. Both public and cluster networks are in 10GB and works well. After a major crash in april, we turned the option bluefs_buffered_io to false to workaround the large write bug when bluefs_buffered_io was true (we were in version 14.2.8 and the default value at this time was true). Since that time, we regularly have some osds wrongly marked down by the cluster after heartbeat timeout (heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f03f1384700' had timed out after 15). Generally the osd restart and the cluster is back healthy, but several time, after many of these kick-off the osd reach the osd_op_thread_suicide_timeout and goes down definitely. We increased the osd_op_thread_timeout and osd_op_thread_suicide_timeout... The problems still occurs (but less frequently). Few days ago, we upgraded to 14.2.11 and revert the timeout to their default value, hoping that it will solve the problem (we thought that it should be related to this bug https://tracker.ceph.com/issues/45943) but it didn't. We still have some osds wrongly marked down. Can somebody help us to fix this problem ? Thanks. Here is an extract of an osd log at failure time: --------------------------------- 2020-08-28 02:19:05.019 7f03f1384700 0 log_channel(cluster) log [DBG] : 44.7d scrub starts 2020-08-28 02:19:25.755 7f040e43d700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f03f1384700' had timed out after 15 2020-08-28 02:19:25.755 7f040dc3c700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f03f1384700' had timed out after 15 this last line is repeated more than 1000 times ... 2020-08-28 02:20:17.484 7f040d43b700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f03f1384700' had timed out after 15 2020-08-28 02:20:17.551 7f03f1384700 0 bluestore(/var/lib/ceph/osd/ceph-16) log_latency_fn slow operation observed for _collection_list, latency = 67.3532s, lat = 67s cid =44.7d_head start GHMAX end GHMAX max 25 ... 2020-08-28 02:20:22.600 7f040dc3c700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f03f1384700' had timed out after 15 2020-08-28 02:21:20.774 7f03f1384700 0 bluestore(/var/lib/ceph/osd/ceph-16) log_latency_fn slow operation observed for _collection_list, latency = 63.223s, lat = 63s cid =44.7d_head start #44:beffc78d:::rbd_data.1e48e8ab988992.00000000000011bd:0# end #MAX# max 2147483647 2020-08-28 02:21:20.774 7f03f1384700 1 heartbeat_map reset_timeout 'OSD::osd_op_tp thread 0x7f03f1384700' had timed out after 15 2020-08-28 02:21:20.805 7f03f1384700 0 log_channel(cluster) log [DBG] : 44.7d scrub ok 2020-08-28 02:21:21.099 7f03fd997700 0 log_channel(cluster) log [WRN] : Monitor daemon marked osd.16 down, but it is still running 2020-08-28 02:21:21.099 7f03fd997700 0 log_channel(cluster) log [DBG] : map e609411 wrongly marked me down at e609410 2020-08-28 02:21:21.099 7f03fd997700 1 osd.16 609411 start_waiting_for_healthy 2020-08-28 02:21:21.119 7f03fd997700 1 osd.16 609411 start_boot 2020-08-28 02:21:21.124 7f03f0b83700 1 osd.16 pg_epoch: 609410 pg[36.3d0( v 609409'481293 (449368'478292,609409'481293] local-lis/les=609403/609404 n=154651 ec=435353/435353 lis/c 609403/609403 les/c/f 609404/609404/0 609410/609410/608752) [25,72] r=-1 lpr=609410 pi=[609403,609410)/1 luod=0'0 lua=609392'481198 crt=609409'481293 lcod 609409'481292 active mbc={}] start_peering_interval up [25,72,16] -> [25,72], acting [25,72,16] -> [25,72], acting_primary 25 -> 25, up_primary 25 -> 25, role 2 -> -1, features acting 4611087854031667199 upacting 4611087854031667199 ... 2020-08-28 02:21:21.166 7f03f0b83700 1 osd.16 pg_epoch: 609411 pg[36.56( v 609409'480511 (449368'477424,609409'480511] local-lis/les=609403/609404 n=153854 ec=435353/435353 lis/c 609403/609403 les/c/f 609404/609404/0 609410/609410/609410) [103,102] r=-1 lpr=609410 pi=[609403,609410)/1 crt=609409'480511 lcod 609409'480510 unknown NOTIFY mbc={}] state<Start>: transitioning to Stray 2020-08-28 02:21:21.307 7f04073b0700 1 osd.16 609413 set_numa_affinity public network em1 numa node 0 2020-08-28 02:21:21.307 7f04073b0700 1 osd.16 609413 set_numa_affinity cluster network em2 numa node 0 2020-08-28 02:21:21.307 7f04073b0700 1 osd.16 609413 set_numa_affinity objectstore and network numa nodes do not match 2020-08-28 02:21:21.307 7f04073b0700 1 osd.16 609413 set_numa_affinity not setting numa affinity 2020-08-28 02:21:21.566 7f040a435700 1 osd.16 609413 tick checking mon for new map 2020-08-28 02:21:22.515 7f03fd997700 1 osd.16 609414 state: booting -> active 2020-08-28 02:21:22.515 7f03f0382700 1 osd.16 pg_epoch: 609414 pg[36.20( v 609409'483167 (449368'480117,609409'483167] local-lis/les=609403/609404 n=155171 ec=435353/435353 lis/c 609403/609403 les/c/f 609404/609404/0 609414/609414/609361) [97,16,72] r=1 lpr=609414 pi=[609403,609414)/1 crt=609409'483167 lcod 609409'483166 unknown NOTIFY mbc={}] start_peering_interval up [97,72] -> [97,16,72], acting [97,72] -> [97,16,72], acting_primary 97 -> 97, up_primary 97 -> 97, role -1 -> 1, features acting 4611087854031667199 upacting 4611087854031667199 ... 2020-08-28 02:21:22.522 7f03f1384700 1 osd.16 pg_epoch: 609414 pg[36.2f3( v 609409'479796 (449368'476712,609409'479796] local-lis/les=609403/609404 n=154451 ec=435353/435353 lis/c 609403/609403 les/c/f 609404/609404/0 609414/609414/609414) [16,34,21] r=0 lpr=609414 pi=[609403,609414)/1 crt=609409'479796 lcod 609409'479795 mlcod 0'0 unknown NOTIFY mbc={}] start_peering_interval up [34,21] -> [16,34,21], acting [34,21] -> [16,34,21], acting_primary 34 -> 16, up_primary 34 -> 16, role -1 -> 0, features acting 4611087854031667199 upacting 4611087854031667199 2020-08-28 02:21:22.522 7f03f1384700 1 osd.16 pg_epoch: 609414 pg[36.2f3( v 609409'479796 (449368'476712,609409'479796] local-lis/les=609403/609404 n=154451 ec=435353/435353 lis/c 609403/609403 les/c/f 609404/609404/0 609414/609414/609414) [16,34,21] r=0 lpr=609414 pi=[609403,609414)/1 crt=609409'479796 lcod 609409'479795 mlcod 0'0 unknown mbc={}] state<Start>: transitioning to Primary 2020-08-28 02:21:24.738 7f03f1384700 0 log_channel(cluster) log [DBG] : 36.2f3 scrub starts 2020-08-28 02:22:18.857 7f03f1384700 0 log_channel(cluster) log [DBG] : 36.2f3 scrub ok

3 4

Avail Online Home Help Services From Qualified Experts
by amarasmith488＠gmail.com 01 Sep '20

01 Sep '20

Are you unable to complete your homework assignment? Are you looking for a reliable online homework help service provider? LiveWebTutors is one of the best and most reliable companies when it comes to providing an assignment writing service. You can connect with the experts whether you are in need of a dissertation writing service or coursework help service. The professionals will ensure that your writing needs are covered within the given deadline and that too as per the instructions stated by the college professor! So, get your grades better by connecting with the professionals of LiveWebTutors now! Visit us: https://www.livewebtutors.com/usa/coursework-help

1 0

RandomCrashes on OSDs Attached to Mon Hosts with Octopus
by Denis Krienbühl 01 Sep '20

01 Sep '20

Hi! We've recently upgraded all our clusters from Mimic to Octopus (15.2.4). Since then, our largest cluster is experiencing random crashes on OSDs attached to the mon hosts. This is the crash we are seeing (cut for brevity, see links in post scriptum): { "ceph_version": "15.2.4", "utsname_release": "4.15.0-72-generic", "assert_condition": "r == 0", "assert_func": "void BlueStore::_txc_apply_kv(BlueStore::TransContext*, bool)", "assert_file": "/build/ceph-15.2.4/src/os/bluestore/BlueStore.cc <http://bluestore.cc/>", "assert_line": 11430, "assert_thread_name": "bstore_kv_sync", "assert_msg": "/build/ceph-15.2.4/src/os/bluestore/BlueStore.cc <http://bluestore.cc/>: In function 'void BlueStore::_txc_apply_kv(BlueStore::TransContext*, bool)' thread 7fc56311a700 time 2020-08-26T08:52:24.917083+0200\n/build/ceph-15.2.4/src/os/bluestore/BlueStore.cc <http://bluestore.cc/>: 11430: FAILED ceph_assert(r == 0)\n", "backtrace": [ "(()+0x12890) [0x7fc576875890]", "(gsignal()+0xc7) [0x7fc575527e97]", "(abort()+0x141) [0x7fc575529801]", "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a5) [0x559ef9ae97b5]", "(ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x559ef9ae993f]", "(BlueStore::_txc_apply_kv(BlueStore::TransContext*, bool)+0x3a0) [0x559efa0245b0]", "(BlueStore::_kv_sync_thread()+0xbdd) [0x559efa07745d]", "(BlueStore::KVSyncThread::entry()+0xd) [0x559efa09cd3d]", "(()+0x76db) [0x7fc57686a6db]", "(clone()+0x3f) [0x7fc57560a88f]" ] } Right before the crash occurs, we see the following message in the crash log: -3> 2020-08-26T08:52:24.787+0200 7fc569b2d700 2 rocksdb: [db/db_impl_compaction_flush.cc:2212 <http://db_impl_compaction_flush.cc:2212/>] Waiting after background compaction error: Corruption: block checksum mismatch: expected 2548200440, got 2324967102 in db/815839.sst offset 67107066 size 3808, Accumulated background error counts: 1 -2> 2020-08-26T08:52:24.852+0200 7fc56311a700 -1 rocksdb: submit_common error: Corruption: block checksum mismatch: expected 2548200440, got 2324967102 in db/815839.sst offset 67107066 size 3808 code = 2 Rocksdb transaction: In short, we see a Rocksdb corruption error after background compaction, when this happens. When an OSD crashes, which happens about 10-15 times a day, it restarts and resumes work without any further problems. We are pretty confident that this is not a hardware issue, due to the following facts: * The crashes occur on 5 different hosts over 3 different racks. * There is no smartctl/dmesg output that could explain it. * It usually happens to a different OSD that did not crash before. Still we checked the following on a few OSDs/hosts: * We can do a manual compaction, both offline and online. * We successfully ran "ceph-bluestore-tool fsck --deep yes" on one of the OSDs. * We manually compacted a number of OSDs, one of which crashed hours later. The only thing we have noticed so far: It only happens to OSDs that are attached to a mon host. *None* of the non-mon host OSDs have had a crash! Does anyone have a hint what could be causing this? We currently have no good theory that could explain this, much less have a fix or workaround. Any help would be greatly appreciated. Denis Crash: https://public-resources.objects.lpg.cloudscale.ch/osd-crash/meta.txt <https://public-resources.objects.lpg.cloudscale.ch/osd-crash/meta.txt> Log: https://public-resources.objects.lpg.cloudscale.ch/osd-crash/log.txt <https://public-resources.objects.lpg.cloudscale.ch/osd-crash/log.txt>

7 11

rgw.none vs quota
by Jean-Sebastien Landry 01 Sep '20

01 Sep '20

Hi everyone, a bucket was overquota, (default quota of 300k objects per bucket), I enabled the object quota for this bucket and set a quota of 600k objects. We are on Luminous (12.2.12) and dynamic resharding is disabled, I manually do the resharding from 3 to 6 shards. Since then, radosgw-admin bucket stats report a `rgw.none` in the usage section for this bucket. I search the mailing-lists, bugzilla, github, it's look like I can ignore the rgw.none stats. (0 byte object, entry left in the index marked as cancelled...) but, the num_object in rgw.none is part of the quota usage. I bump the quota to 800k object to workaround the problem. (without resharding) Is there a way I can garbage collect the rgw.none? Is this problem fixed in Mimic/Nautilus/Octopus? "usage": { "rgw.none": { "size": 0, "size_actual": 0, "size_utilized": 0, "size_kb": 0, "size_kb_actual": 0, "size_kb_utilized": 0, "num_objects": 417827 }, "rgw.main": { "size": 1390778138502, "size_actual": 1391581007872, "size_utilized": 1390778138502, "size_kb": 1358181776, "size_kb_actual": 1358965828, "size_kb_utilized": 1358181776, "num_objects": 305637 } }, Thanks!

3 2

Re: cephfs needs access from two networks
by Simon Sutter 01 Sep '20

01 Sep '20

Hello again So I have changed the network configuration. Now my Ceph is reachable from outside, this also means all osd’s of all nodes are reachable. I still have the same behaviour which is a timeout. The client can resolve all nodes with their hostnames. The mon’s are still listening on the internal network so the nat rule is still there. I have set “public bind addr” to the external ip and restarted the mon but it’s still not working. [root@testnode1 ~]# ceph config get mon.public_bind_addr WHO MASK LEVEL OPTION VALUE RO mon advanced public_bind_addr v2:[ext-addr]:0/0 * Do I have to change them somewhere else too? Thanks in advance, Simon Von: Janne Johansson [mailto:icepic.dz@gmail.com] Gesendet: 27 August 2020 20:01 An: Simon Sutter <ssutter(a)hosttech.ch> Betreff: Re: [ceph-users] cephfs needs access from two networks Den tors 27 aug. 2020 kl 12:05 skrev Simon Sutter <ssutter(a)hosttech.ch<mailto:ssutter@hosttech.ch>>: Hello Janne Oh I missed that point. No, the client cannot talk directly to the osds. In this case it’s extremely difficult to set this up. This is an absolute requirement to be a ceph client. How is the mon telling the client, which host and port of the osd, it should connect to? The same port and ip that the ODS called into the mon with when it started up and joined the clusster. Can I have an influence on it? Well, you set the ip on the OSD hosts, and the port ranges in use for OSDs are changeable/settable, but it would not really help the above-mentioned client. Von: Janne Johansson [mailto:icepic.dz@gmail.com<mailto:icepic.dz@gmail.com>] Gesendet: 26 August 2020 15:09 An: Simon Sutter <ssutter(a)hosttech.ch<mailto:ssutter@hosttech.ch>> Cc: ceph-users(a)ceph.io<mailto:ceph-users@ceph.io> Betreff: Re: [ceph-users] cephfs needs access from two networks Den ons 26 aug. 2020 kl 14:16 skrev Simon Sutter <ssutter(a)hosttech.ch<mailto:ssutter@hosttech.ch>>: Hello, So I know, the mon services can only bind to just one ip. But I have to make it accessible to two networks because internal and external servers have to mount the cephfs. The internal ip is 10.99.10.1 and the external is some public-ip. I tried nat'ing it with this: "firewall-cmd --zone=public --add-forward-port=port=6789:proto=tcp:toport=6789:toaddr=10.99.10.1 -permanent" So the nat is working, because I get a "ceph v027" (alongside with some gibberish) when I do a telnet "telnet *public-ip* 6789" But when I try to mount it, I get just a timeout: mount -vvvv -t ceph *public-ip*:6789:/testing /mnt -o name=test,secretfile=/root/ceph.client. test.key mount error 110 = Connection timed out The tcpdump also recognizes a "Ceph Connect" packet, coming from the mon. How can I get around this problem? Is there something I have missed? Any ceph client will need direct access to all OSDs involved also. Your mail doesn't really say if the cephfs-mounting client can talk to OSDs? In ceph, traffic is not shuffled via mons, mons only tell the client which OSDs it needs to talk to, then all IO goes directly from client to any involved OSD servers. -- May the most significant bit of your life be positive. -- May the most significant bit of your life be positive.

2 1

Re: cephfs needs access from two networks]
by Marcel Kuiper 01 Sep '20

01 Sep '20

The mons get their bind address from the monmap I believe. So this means changing in the monmap the ip-addresses of the monitors with the monmaptool. Regards Marcel > Hello again > > So I have changed the network configuration. > Now my Ceph is reachable from outside, this also means all osdâs of all > nodes are reachable. > I still have the same behaviour which is a timeout. > > The client can resolve all nodes with their hostnames. > The monâs are still listening on the internal network so the nat rule is > still there. > I have set âpublic bind addrâ to the external ip and restarted the mon > but itâs still not working. > > [root@testnode1 ~]# ceph config get mon.public_bind_addr > WHO MASK LEVEL OPTION VALUE RO > mon advanced public_bind_addr v2:[ext-addr]:0/0 * > > Do I have to change them somewhere else too? > > Thanks in advance, > Simon >

1 0

setting bucket quota using admin API does not work
by Youzhong Yang 01 Sep '20

01 Sep '20

Hi all, I tried to set bucket quota using admin API as shown below: admin/user?quota&uid=bse&bucket=test&quota-type=bucket with payload in json format: { "enabled": true, "max_size": 1099511627776, "max_size_kb": 1073741824, "max_objects": -1 } it returned success but the quota change did not happen, as confirmed by 'radosgw-admin bucket stats --bucket=test' command. Am I missing something obvious? Please kindly advise/suggest. By the way, I am using ceph mimic (v13.2.4). Setting quota by radosgw-admin quota set --bucket=${BUCK} --max-size=1T --quota-scope=bucket works, but I want to do it programmatically. Thanks in advance, -Youzhong

1 1

Delete OSD spec (mgr)?
by Darrell Enns 01 Sep '20

01 Sep '20

Is there a way to remove an OSD spec from the mgr? I've got one in there that I don't want. It shows up when I do "ceph orch osd spec --preview", and I can't find any way to get rid of it.

1 0