- ceph-users - lists.ceph.io

Problem with centos7 repository
by Tadas 08 Jul '20

08 Jul '20

Hello. I'm having problem with Centos7 Nautilus repository. Since ~10 days ago, ( i guess after release of 14.2.10 packages), yum does not find earlier Nautilus releases anymore. They can be visible in repo if i browse it. But they are not in yum meta files i think so you cant install them via yum: N/S matched: librados2 ========================================================================================================== 1:librados2-10.2.5-4.el7.i686 : RADOS distributed object store client library 1:librados2-10.2.5-4.el7.x86_64 : RADOS distributed object store client library 1:librados2-10.2.5-4.el7.x86_64 : RADOS distributed object store client library 2:librados2-14.2.10-0.el7.x86_64 : RADOS distributed object store client library 1:librados2-devel-10.2.5-4.el7.i686 : RADOS headers 1:librados2-devel-10.2.5-4.el7.x86_64 : RADOS headers As you can see, yum finds only latest version of 14.2. release.

2 1

Ceph SSH orchestrator?
by Oliver Freyermuth 07 Jul '20

07 Jul '20

Dear Cephalopodians, as we all know, ceph-deploy is on its demise since a while and essentially in "maintenance mode". We've been eyeing the "ssh orchestrator" which was in Nautilus as the "successor in spirit" of ceph-deploy. While we have not tried it out just yet, I find this module seems to be gone without a trace in Octopus. There's still an Orchestrator module, but this seems to work "only" with containers. Is this true, or is there still an SSH orchestrator capable of bare-metal operation in Octopus (or are there plans to have something like this)? While I see many advantages of containers in many areas, and certainly also for smaller setups or test setups with Ceph, as any technology, they come with their own problems. Example issues (which all can be solved, but require extra work from the administrator) are: - Operation on machines without connectivity to the internet (you'd need to mirror the containers or run your own registry), - Ensuring automated security updates both outside the containers and inside the containers, or re-pull them regularly (and monitor that), - Integrate with existing logging and configuration management systems, - Potential hardware issues, such das InfiniBand RDMA. There's surely more (and there are also as many benefits), and as I said, all can be solved; the point I want to make is: Containers are not the best solution in all environments and also not for all admins. So my question is: Is there something like the SSH orchestrator still available? I guess essentially the cephadm orchestrator does something similar behind the screnes, with the added bells and whistles to manage the containers. Of course, a reduced feature-set would be expected (e.g. no "ceph orch upgrade"), but it would jump into the hole ceph-deploy has left. Maybe this is as easy as setting a configuration knob? Or is it also possible to switch to a "bare-metal edition" of cephadm (which might rely on users or existing configuration management to install the packages, e.g.)? Cheers, Oliver

6 7

BlueFS.cc: 1576 FAILED assert (Ceph mimic 13.2.8)
by Stefan Kooman 07 Jul '20

07 Jul '20

Hi List, On of our clusters there are a couple of OSDs that keep crashing: > /build/ceph-13.2.8/src/os/bluestore/BlueFS.cc: 1576: FAILED assert(r == 0) > > ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) > 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14e) [0x7fbae37bbf1e] > 2: (()+0x2fc0a7) [0x7fbae37bc0a7] > 3: (BlueFS::_flush_and_sync_log(std::unique_lock<std::mutex>&, unsigned long, unsigned long)+0x13ed) [0xc660dd] > 4: (BlueFS::_fsync(BlueFS::FileWriter*, std::unique_lock<std::mutex>&)+0x8a) [0xc667fa] > 5: (BlueRocksWritableFile::Sync()+0x63) [0xc81b13] > 6: (rocksdb::WritableFileWriter::SyncInternal(bool)+0x2a9) [0xe45c69] > 7: (rocksdb::WritableFileWriter::Sync(bool)+0x21e) [0xe47ece] > 8: (rocksdb::BuildTable(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::EnvOptions const&, rocksdb::TableCache*, rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint)+0x1e14) [0xe72af4] > 9: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xcb7) [0xcf5117] > 10: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0x1a0a) [0xcf742a] > 11: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0x5d4) [0xcf7fd4] > 12: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool)+0x6ab) [0xcf929b] > 13: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) [0xcfab92] > 14: (RocksDBStore::do_open(std::ostream&, bool, std::vector<KeyValueDB::ColumnFamily, std::allocator<KeyValueDB::ColumnFamily> > const*)+0x174a) [0xc0136a] > 15: (BlueStore::_open_db(bool, bool)+0xd8f) [0xb8cc4f] > 16: (BlueStore::_mount(bool, bool)+0x4cf) [0xbbcd4f] > 17: (OSD::init()+0x33d) [0x751f5d] > 18: (main()+0x24b0) [0x643560] > 19: (__libc_start_main()+0xf0) [0x7fbae1762830] > 20: (_start()+0x29) [0x70d9f9] Any hints on what can be the cause of this? Gr. Stefan -- | BIT BV https://www.bit.nl/ Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / info(a)bit.nl

2 4

Block sizes, small files and bluestore
by Dennis Benndorf 07 Jul '20

07 Jul '20

Hi Cephers, I have a lot of questions after reading a lot about bluestore, the min_alloc_size and the impact of writing small files to it through cepfs with erasure coding. In a setup using VMware ESX with its VMFS6 and its 1 MB block size on an ISCSI-LUN mapped from Ceph RBD (replicated) 1. Wouldnt it be better to reduce RBD block size to 1 MB also? 2. When a file with a size smaller than 4k is written to a filesystem within a virtual machine (and all the layers downwards) will the consumed space be 4k? So does the 4MB block size of RBD combine a lot of small files to one big object? When using Cephfs and erasure coding: 1. I assume using a 4k min_alloc_size_hdd would reduce wasted space, but increases fragmentation as Igor wrote. 2. How is the official way to deal with fragmentation in bluestore? Is there a defrag tool available or planned? From a performance perspective: My cluster runs on good old filestore using nvme journals. I am about to migrate to bluestore. 1. With a MaxIOSize of 512KB in VMware, wouldnt bluestore_prefer_deferred_size_hdd = 524288 give me a filestore like behavior? My aim is to have the write latency like in filestore because we have a lot of databases. 2. Are there any tradeoffs doing this? Regards, Dennis

1 0

OSDs taking too much memory, for buffer_anon
by Harald Staub 07 Jul '20

07 Jul '20

As a follow-up to our recent memory problems with OSDs (with high pglog values: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/LJPJZPBSQRJ… ), we also see high buffer_anon values. E.g. more than 4 GB, with "osd memory target" set to 3 GB. Is there a way to restrict it? As it is called "anon", I guess that it would first be necessary to find out what exactly is behind this? Well maybe it is just as Wido said, with lots of small objects, there will be several problems. Cheers Harry

4 8

Ceph Zabbix Monitoring : No such file or directory
by etiennemula＠gmail.com 07 Jul '20

07 Jul '20

Hello All, We are trying to make work the Zabbix module of our ceph cluster but im encountering an issue that got me stuck. Configuration of the module looks ok and we manage to send data using zabbix_sender to the host that is configured on Zabbix. We can also see this data/metric in the graphs that come with the template aswell. So far so good. However when testing from ceph module by doing ceph zabbix send , we get : Sending Data to Zabbix but when check ceph_mgr logs we see : 7fad8b61e700 0 mgr[zabbix] Exception when sending: [Errno 2] No such file or directory and no data is seen in zabbix itself. Any clue what we can check ? This ceph is running with an openstack cluster and ceph mgr instance is a docker container, not sure if we need to do anything in particular because of this architecture. Ceph Version : 12.2.4 Thank you, Etienne.

4 4

Ceph Zabbix Monitoring : No such file or directory
by etiennemula＠gmail.com 07 Jul '20

07 Jul '20

Hello All, We are trying to make work the Zabbix module of our ceph cluster but im encountering an issue that got me stuck. Configuration of the module looks ok and we manage to send data using zabbix_sender to the host that is configured on Zabbix. We can also see this data/metric in the graphs that come with the template aswell. So far so good. However when testing from ceph module by doing ceph zabbix send , we get : Sending Data to Zabbix but when check ceph_mgr logs we see : 7fad8b61e700 0 mgr[zabbix] Exception when sending: [Errno 2] No such file or directory and no data is seen in zabbix itself. Any clue what we can check ? This ceph is running with an openstack cluster and ceph mgr instance is a docker container, not sure if we need to do anything in particular because of this architecture. Ceph Version : 12.2.4 Thank you, Etienne.

3 2

Octopus: Recovery and backfilling causes OSDs to crash after upgrading from nautilus to octopus
by Wout van Heeswijk 06 Jul '20

06 Jul '20

Hi All, A customer of ours has upgraded the cluster from nautilus to octopus after experiencing issues with osds not being able to connect to each other, clients/mons/mgrs. The connectivity issues was related to the msgrV2 and require_osd_release setting not being set to nautilus. After fixing this the OSDs were restarted and all placement groups became active again. After unsetting the norecover and nobackfill flag some OSDs started crashing every few minutes. The OSD log, even with high debug settings, don't seem to reveal anything, it just stops logging mid log line. I've created a bug report: https://tracker.ceph.com/issues/46366 Has anyone experienced something similar? -- kind regards, Wout 42on

4 7

Upgrade from 14.2.6 to 15.2.4
by ST Wong (ITSC) 06 Jul '20

06 Jul '20

Hi all, We're planning to upgrade an existing testing 14.2.6 CEPH cluster to latest 15.2.4. Existing cluster was deployed using ceph-ansible and we're still searching steps to do the upgrade. Shall we do the upgrade with steps like following ? - Upgrade rpm on all nodes (mon/mgr first, then osd, then rgw) - Convert the cluster to use cephadm as mentioned here https://docs.ceph.com/docs/master/cephadm/adoption/ Apology for the newbie question. Thanks a lot. Regards, /ST Wong

1 0

Nautilus upgrade HEALTH_WARN legacy tunables
by jimf＠mninc.net 06 Jul '20

06 Jul '20

I just upgraded from Luminous to Nautilus. the cluster was originally hammer or kraken (can't recall), then jewel, on to luminous, and now Nautilus. The message i get from ceph -s is: "health: HEALTH_WARN crush map has legacy tunables (require firefly, min is hammer)" Not sure how to get rid of it. I tried "ceph osd set-require-min-compat-client firefly" per instructions but still have error. When I query osd: ceph osd dump | grep min_compat_client require_min_compat_client firefly min_compat_client firefly Instructions say I can ignore the error, but that seems like a bad idea. Windows VM's are running on qemu/kvm via rbd connection. Ceph and KVM are running CentOS7. Just upgraded all clients to "centos-release-7-8.2003.0.el7.centos.x86_64" I have rebooted all nodes since CentOS upgrade and again after Nautilus upgrade. tried setting "ceph osd set-require-min-compat-client jewel" but then heath_warn was still present and: ceph osd dump | grep min_compat_client require_min_compat_client jewel min_compat_client firefly Any help is appreciated.

4 3