July 2020 - ceph-users - lists.ceph.io

bucket index nvme
by Szabo, Istvan (Agoda) 09 Jul '20

09 Jul '20

Hello, Can someone explain me a bit about the objectstore indexing? It's not really clear, when redhat says one of the important tuning for objectstore is to put the indexes on a fast drive, when I check our current ceph cluster and I see petabytes of read operations but the size of the index pool is 0, so how I can size 1 nvme drive in each server with 4 osd to host the index pool? Also thinking to share the nvme with different partition for journaling and bucket index, but no problem to order 1 more for it. Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0

Activate Xfinity Channel Via xfinity.com authorize
by xfinityauthorize26＠gmail.com 09 Jul '20

09 Jul '20

Xfinity or Xfinity Stream is a streaming service which provides live broadcast channels, DVR recordings, linear cable channels and on-demand videos. You can watch live TV channels from nearly 200+ favourite networks. Additionally, it is possible to get on-demand videos for offline access. In the next guide, we will illustrate how to prepare and trigger the Xfinity Stream app on Roku apparatus. Currently, Xfinity is available in a beta version for Roku platforms, and plus It Doesn't incorporate the Entire set of features and functionality included in the very first variants . https://www.xfinitycomauthorize.com/

1 0

Ceph multisite secondary zone not sync new changes
by Amit Ghadge 09 Jul '20

09 Jul '20

Hello All, In our test environment we set up ceph multisite in Active/Passive. Cluster A migrated to the master zone without deleting any data and set up a fresh secondary zone. First we stop pushing data to master zone and secondary zone sync all buckets and objects but later 1 hour started uploads 1million objects with newly created buckets this new changes not sync by secondary. It still showing, # radosgw-admin sync status realm 2a7b2a08-3404-40ea-81ed-2d036ee6e54f (masifd) zonegroup 039c426b-4032-4621-9034-d3ee724967a0 (us) zone 213b25f9-e726-43c8-a82e-4f9f0d4ed55c (us-east) metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master data sync source: 0c6643f3-0289-4811-ab3e-dff2952e31c6 (us-west) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is caught up with source

1 0

ceph nautilus repository index is incomplet
by Francois Legrand 09 Jul '20

09 Jul '20

Hello, It seems that the index of https://download.ceph.com/rpm-nautilus/el7/x86_64/ repository is wrong. Only the 14.2.10-0.el7 version is available (all previous versions are missing despite the fact that the rpms are present in the repository). It thus seems that the index needs to be corrected. Who can I contact for that ? Thanks. F.

1 0

Install And Activate Nbc Channel On Roku
by nbcactivate26＠gmail.com 09 Jul '20

09 Jul '20

An American-English commercial real radio and tv system are a forerunner land of the NC Universal. Now, you can view the displays from round the NBC Universal Universal networks such as E!, Oxygen, SYFY, USA, Bravo. The NBC This station is currently available on particular streaming apparatus like Roku, Amazon Fire Stick. All of that should be done would be to bring the station on into the streaming device then need to trigger the station by seeing https://www.nbccomactivatetv.com/

1 0

Octopus upgrade breaks Ubuntu 18.04 libvirt
by Andrei Mikhailovsky 08 Jul '20

08 Jul '20

Hello, I've upgraded ceph to Octopus (15.2.3 from repo) on one of the Ubuntu 18.04 host servers. The update caused problem with libvirtd which hangs when it tries to access the storage pools. The problem doesn't exist on Nautilus. The libvirtd process simply hangs. Nothing seem to happen. The log file for the libvirtd shows: 2020-06-29 19:30:51.556+0000: 12040: debug : virNetlinkEventCallback:707 : dispatching to max 0 clients, called from event watch 11 2020-06-29 19:30:51.556+0000: 12040: debug : virNetlinkEventCallback:720 : event not handled. 2020-06-29 19:30:51.556+0000: 12040: debug : virNetlinkEventCallback:707 : dispatching to max 0 clients, called from event watch 11 2020-06-29 19:30:51.556+0000: 12040: debug : virNetlinkEventCallback:720 : event not handled. 2020-06-29 19:30:51.557+0000: 12040: debug : virNetlinkEventCallback:707 : dispatching to max 0 clients, called from event watch 11 2020-06-29 19:30:51.557+0000: 12040: debug : virNetlinkEventCallback:720 : event not handled. 2020-06-29 19:30:51.591+0000: 12040: debug : virNetlinkEventCallback:707 : dispatching to max 0 clients, called from event watch 11 2020-06-29 19:30:51.591+0000: 12040: debug : virNetlinkEventCallback:720 : event not handled. Running strace on the libvirtd process shows: root@ais-cloudhost1:/home/andrei# strace -p 12040 strace: Process 12040 attached restart_syscall(<... resuming interrupted poll ...> Nothing happens after that point. The same host server can get access to the ceph cluster and the pools by running ceph -s or rbd -p <pool> ls -l commands for example. Need some help to get the host servers working again with Octopus. Cheers

3 9

problem with ceph osd blacklist
by 380562518＠qq.com 08 Jul '20

08 Jul '20

Hi all. If cephfs client is in a slow or unreliable network environment, the client will be added to OSD blacklist and OSD map, and the default duration is 1 hour. During this time, the client will be forbidden to access CEPH. If I want to solve this problem and ensure client's normal I/O operation is not interrupted, are the following two options feasible? which one is better? 1. Set "mds_session_blacklist_on_timeout" to false, and forbid to add slow clients to blacklist; 2.Just reduce the duration of slow clients joining the blacklist, and change the default 1 hour to 5 minutes. (set the value of "mon_osd_blacklist_default_expire" to 5 minutes） Are the two schemes feasible? Will it have a great impact on the data security and integrity of CEPH? Can you give me some suggestions? Thanks.

2 1

RGW: rgw_qactive perf is constantly growing
by Jacek Suchenia 08 Jul '20

08 Jul '20

Hi On two of our clusters (all v14.2.8) we observe a very strange behavior: Over a time rgw_qactive perf is constantly growing, within 12h to 6k entries. [image: image.png] We observe this situation only on two of our clusters where the common thing is an app uploading a lot of files as multipart uploads via ssl. How can we debug this situation? How can we check what operations are in a queue or why a perf counter has not been decreased? Jacek -- Jacek Suchenia jacek.suchenia(a)gmail.com

2 2

Problem with centos7 repository
by Tadas 08 Jul '20

08 Jul '20

Hello. I'm having problem with Centos7 Nautilus repository. Since ~10 days ago, ( i guess after release of 14.2.10 packages), yum does not find earlier Nautilus releases anymore. They can be visible in repo if i browse it. But they are not in yum meta files i think so you cant install them via yum: N/S matched: librados2 ========================================================================================================== 1:librados2-10.2.5-4.el7.i686 : RADOS distributed object store client library 1:librados2-10.2.5-4.el7.x86_64 : RADOS distributed object store client library 1:librados2-10.2.5-4.el7.x86_64 : RADOS distributed object store client library 2:librados2-14.2.10-0.el7.x86_64 : RADOS distributed object store client library 1:librados2-devel-10.2.5-4.el7.i686 : RADOS headers 1:librados2-devel-10.2.5-4.el7.x86_64 : RADOS headers As you can see, yum finds only latest version of 14.2. release.

2 1

Ceph SSH orchestrator?
by Oliver Freyermuth 07 Jul '20

07 Jul '20

Dear Cephalopodians, as we all know, ceph-deploy is on its demise since a while and essentially in "maintenance mode". We've been eyeing the "ssh orchestrator" which was in Nautilus as the "successor in spirit" of ceph-deploy. While we have not tried it out just yet, I find this module seems to be gone without a trace in Octopus. There's still an Orchestrator module, but this seems to work "only" with containers. Is this true, or is there still an SSH orchestrator capable of bare-metal operation in Octopus (or are there plans to have something like this)? While I see many advantages of containers in many areas, and certainly also for smaller setups or test setups with Ceph, as any technology, they come with their own problems. Example issues (which all can be solved, but require extra work from the administrator) are: - Operation on machines without connectivity to the internet (you'd need to mirror the containers or run your own registry), - Ensuring automated security updates both outside the containers and inside the containers, or re-pull them regularly (and monitor that), - Integrate with existing logging and configuration management systems, - Potential hardware issues, such das InfiniBand RDMA. There's surely more (and there are also as many benefits), and as I said, all can be solved; the point I want to make is: Containers are not the best solution in all environments and also not for all admins. So my question is: Is there something like the SSH orchestrator still available? I guess essentially the cephadm orchestrator does something similar behind the screnes, with the added bells and whistles to manage the containers. Of course, a reduced feature-set would be expected (e.g. no "ceph orch upgrade"), but it would jump into the hole ceph-deploy has left. Maybe this is as easy as setting a configuration knob? Or is it also possible to switch to a "bare-metal edition" of cephadm (which might rely on users or existing configuration management to install the packages, e.g.)? Cheers, Oliver

6 7

BlueFS.cc: 1576 FAILED assert (Ceph mimic 13.2.8)
by Stefan Kooman 07 Jul '20

07 Jul '20

Hi List, On of our clusters there are a couple of OSDs that keep crashing: > /build/ceph-13.2.8/src/os/bluestore/BlueFS.cc: 1576: FAILED assert(r == 0) > > ceph version 13.2.8 (5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0) mimic (stable) > 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14e) [0x7fbae37bbf1e] > 2: (()+0x2fc0a7) [0x7fbae37bc0a7] > 3: (BlueFS::_flush_and_sync_log(std::unique_lock<std::mutex>&, unsigned long, unsigned long)+0x13ed) [0xc660dd] > 4: (BlueFS::_fsync(BlueFS::FileWriter*, std::unique_lock<std::mutex>&)+0x8a) [0xc667fa] > 5: (BlueRocksWritableFile::Sync()+0x63) [0xc81b13] > 6: (rocksdb::WritableFileWriter::SyncInternal(bool)+0x2a9) [0xe45c69] > 7: (rocksdb::WritableFileWriter::Sync(bool)+0x21e) [0xe47ece] > 8: (rocksdb::BuildTable(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Env*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::EnvOptions const&, rocksdb::TableCache*, rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint)+0x1e14) [0xe72af4] > 9: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xcb7) [0xcf5117] > 10: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0x1a0a) [0xcf742a] > 11: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0x5d4) [0xcf7fd4] > 12: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool)+0x6ab) [0xcf929b] > 13: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x22) [0xcfab92] > 14: (RocksDBStore::do_open(std::ostream&, bool, std::vector<KeyValueDB::ColumnFamily, std::allocator<KeyValueDB::ColumnFamily> > const*)+0x174a) [0xc0136a] > 15: (BlueStore::_open_db(bool, bool)+0xd8f) [0xb8cc4f] > 16: (BlueStore::_mount(bool, bool)+0x4cf) [0xbbcd4f] > 17: (OSD::init()+0x33d) [0x751f5d] > 18: (main()+0x24b0) [0x643560] > 19: (__libc_start_main()+0xf0) [0x7fbae1762830] > 20: (_start()+0x29) [0x70d9f9] Any hints on what can be the cause of this? Gr. Stefan -- | BIT BV https://www.bit.nl/ Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / info(a)bit.nl

2 4

Block sizes, small files and bluestore
by Dennis Benndorf 07 Jul '20

07 Jul '20

Hi Cephers, I have a lot of questions after reading a lot about bluestore, the min_alloc_size and the impact of writing small files to it through cepfs with erasure coding. In a setup using VMware ESX with its VMFS6 and its 1 MB block size on an ISCSI-LUN mapped from Ceph RBD (replicated) 1. Wouldnt it be better to reduce RBD block size to 1 MB also? 2. When a file with a size smaller than 4k is written to a filesystem within a virtual machine (and all the layers downwards) will the consumed space be 4k? So does the 4MB block size of RBD combine a lot of small files to one big object? When using Cephfs and erasure coding: 1. I assume using a 4k min_alloc_size_hdd would reduce wasted space, but increases fragmentation as Igor wrote. 2. How is the official way to deal with fragmentation in bluestore? Is there a defrag tool available or planned? From a performance perspective: My cluster runs on good old filestore using nvme journals. I am about to migrate to bluestore. 1. With a MaxIOSize of 512KB in VMware, wouldnt bluestore_prefer_deferred_size_hdd = 524288 give me a filestore like behavior? My aim is to have the write latency like in filestore because we have a lot of databases. 2. Are there any tradeoffs doing this? Regards, Dennis

1 0

OSDs taking too much memory, for buffer_anon
by Harald Staub 07 Jul '20

07 Jul '20

As a follow-up to our recent memory problems with OSDs (with high pglog values: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/LJPJZPBSQRJ… ), we also see high buffer_anon values. E.g. more than 4 GB, with "osd memory target" set to 3 GB. Is there a way to restrict it? As it is called "anon", I guess that it would first be necessary to find out what exactly is behind this? Well maybe it is just as Wido said, with lots of small objects, there will be several problems. Cheers Harry

4 8

Ceph Zabbix Monitoring : No such file or directory
by etiennemula＠gmail.com 07 Jul '20

07 Jul '20

Hello All, We are trying to make work the Zabbix module of our ceph cluster but im encountering an issue that got me stuck. Configuration of the module looks ok and we manage to send data using zabbix_sender to the host that is configured on Zabbix. We can also see this data/metric in the graphs that come with the template aswell. So far so good. However when testing from ceph module by doing ceph zabbix send , we get : Sending Data to Zabbix but when check ceph_mgr logs we see : 7fad8b61e700 0 mgr[zabbix] Exception when sending: [Errno 2] No such file or directory and no data is seen in zabbix itself. Any clue what we can check ? This ceph is running with an openstack cluster and ceph mgr instance is a docker container, not sure if we need to do anything in particular because of this architecture. Ceph Version : 12.2.4 Thank you, Etienne.

4 4

Ceph Zabbix Monitoring : No such file or directory
by etiennemula＠gmail.com 07 Jul '20

07 Jul '20

Hello All, We are trying to make work the Zabbix module of our ceph cluster but im encountering an issue that got me stuck. Configuration of the module looks ok and we manage to send data using zabbix_sender to the host that is configured on Zabbix. We can also see this data/metric in the graphs that come with the template aswell. So far so good. However when testing from ceph module by doing ceph zabbix send , we get : Sending Data to Zabbix but when check ceph_mgr logs we see : 7fad8b61e700 0 mgr[zabbix] Exception when sending: [Errno 2] No such file or directory and no data is seen in zabbix itself. Any clue what we can check ? This ceph is running with an openstack cluster and ceph mgr instance is a docker container, not sure if we need to do anything in particular because of this architecture. Ceph Version : 12.2.4 Thank you, Etienne.

3 2

Octopus: Recovery and backfilling causes OSDs to crash after upgrading from nautilus to octopus
by Wout van Heeswijk 06 Jul '20

06 Jul '20

Hi All, A customer of ours has upgraded the cluster from nautilus to octopus after experiencing issues with osds not being able to connect to each other, clients/mons/mgrs. The connectivity issues was related to the msgrV2 and require_osd_release setting not being set to nautilus. After fixing this the OSDs were restarted and all placement groups became active again. After unsetting the norecover and nobackfill flag some OSDs started crashing every few minutes. The OSD log, even with high debug settings, don't seem to reveal anything, it just stops logging mid log line. I've created a bug report: https://tracker.ceph.com/issues/46366 Has anyone experienced something similar? -- kind regards, Wout 42on

4 7

Upgrade from 14.2.6 to 15.2.4
by ST Wong (ITSC) 06 Jul '20

06 Jul '20

Hi all, We're planning to upgrade an existing testing 14.2.6 CEPH cluster to latest 15.2.4. Existing cluster was deployed using ceph-ansible and we're still searching steps to do the upgrade. Shall we do the upgrade with steps like following ? - Upgrade rpm on all nodes (mon/mgr first, then osd, then rgw) - Convert the cluster to use cephadm as mentioned here https://docs.ceph.com/docs/master/cephadm/adoption/ Apology for the newbie question. Thanks a lot. Regards, /ST Wong

1 0

Nautilus upgrade HEALTH_WARN legacy tunables
by jimf＠mninc.net 05 Jul '20

05 Jul '20

I just upgraded from Luminous to Nautilus. the cluster was originally hammer or kraken (can't recall), then jewel, on to luminous, and now Nautilus. The message i get from ceph -s is: "health: HEALTH_WARN crush map has legacy tunables (require firefly, min is hammer)" Not sure how to get rid of it. I tried "ceph osd set-require-min-compat-client firefly" per instructions but still have error. When I query osd: ceph osd dump | grep min_compat_client require_min_compat_client firefly min_compat_client firefly Instructions say I can ignore the error, but that seems like a bad idea. Windows VM's are running on qemu/kvm via rbd connection. Ceph and KVM are running CentOS7. Just upgraded all clients to "centos-release-7-8.2003.0.el7.centos.x86_64" I have rebooted all nodes since CentOS upgrade and again after Nautilus upgrade. tried setting "ceph osd set-require-min-compat-client jewel" but then heath_warn was still present and: ceph osd dump | grep min_compat_client require_min_compat_client jewel min_compat_client firefly Any help is appreciated.

4 3

rgw print continue
by Seena Fallah 05 Jul '20

05 Jul '20

Hi all. I see this sentence in many sites. Does anyone knows why? > Then turn off print continue. If you have it set to true, you may encounter problems with PUT operations I use nginx in front of my rgw and proxy pass expect header in it. Thanks.

1 0

Placement of block/db and WAL on SSD?
by Lindsay Mathieson 05 Jul '20

05 Jul '20

Nautilus install. Documentation seems a bit ambiguous to me - this is for a spinner + SSD, using ceph-volume If I put the block.db on the SSD with "ceph-volume lvm create --bluestore --data /dev/sdd --block.db /dev/sdc1" does the wal exists on the ssd (/dev/sdc1) as well, or does it remain on the hdd (/dev/sdd)? Conversely, what happens with the block.db if I place the wal with --block.wal Or do I have to setup separate partitions for the block.db and wal? -- Lindsay

2 3

High iops on bucket index
by Seena Fallah 05 Jul '20

05 Jul '20

Hi all. There are high iops on my bucket index pool when there is about 1K PUT request/s. Is there any way I can debug why there are so many iops on the bucket index pool? Thanks.

1 0

Showing OSD Disk config?
by Lindsay Mathieson 04 Jul '20

04 Jul '20

Is there a way to display an OSD's setup - data, data.db and WAL disks/partitions? -- Lindsay

3 3

Re: Ceph OSD not mounting after reboot
by Rodrigo Severo - Fábrica 04 Jul '20

04 Jul '20

Em sáb., 4 de jul. de 2020 às 11:27, Dave Hall <kdhall(a)binghamton.edu> escreveu: > > Rodrigo, > > I tried to send this to the list last night, but it looks like it didn't go through. I had a problem very much like this when I was first setting up my cluster. It turned out to be a missing systemd unit file. I would suggest that you check to see that you have an instance of ceph-volume@.service and an instance of ceph-osd@.service running for each OSD. I manually started my ceph-volume@ instances and then I managed to successfully restart my ceph-osd@ instances. My OSDs are back. Does anybody knows how is responsible for calling the ceph-volume@ instances on boot? Regards, Rodrigo

1 0

Re: Ceph OSD not mounting after reboot
by Rodrigo Severo - Fábrica 04 Jul '20

04 Jul '20

Em sáb., 4 de jul. de 2020 às 11:27, Dave Hall <kdhall(a)binghamton.edu> escreveu: > > Rodrigo, > > I tried to send this to the list last night, but it looks like it didn't go through. I had a problem very much like this when I was first setting up my cluster. It turned out to be a missing systemd unit file. I would suggest that you check to see that you have an instance of ceph-volume@.service and an instance of ceph-osd@.service running for each OSD. > > ceph-volume(a)lvm-10-a63a2465-9de2-497e-bf60-72ed4c3c4c33.service > ceph-volume(a)lvm-11-a2151f1f-84f5-407b-a730-9d2a9502a85f.service > ceph-volume(a)lvm-12-7a16123b-7a0f-40a2-9159-5555724d9978.service > ceph-volume(a)lvm-13-bda32c07-ecc7-40c4-8066-383977c5e795.service > ceph-volume(a)lvm-14-4cc89d3a-5390-415d-b12b-9a557b7b5950.service > ceph-volume(a)lvm-15-21bbdaf5-1994-43a9-aa80-982689f51438.service > ceph-volume(a)lvm-8-cda4394b-e132-4530-8044-3cbbfbcbea19.service > ceph-volume(a)lvm-9-72ff199d-a4e4-4d26-9a4f-00337f8fdc7c.service > > ceph-osd(a)10.service > ceph-osd(a)12.service > ceph-osd(a)14.service > ceph-osd(a)8.service > ceph-osd(a)11.service > ceph-osd(a)13.service > ceph-osd(a)15.service > ceph-osd(a)9.service > > In my case I found that the proto-unit for one of these (ceph-volume@.service, I think) was missing from /var/lib/systemd/system. In fact, it was missing from the Debian install package for some reason. I think I had to unpack a copy of the Debian SRC pacakge to retrieve the file. Once I added the missing file and made sure the unit was enabled, things started working better and making more sense. I don't recall if all of the instances created themselves or whether I had to do something addtional, but it worked. Hi Dave, I'm looking around but can't identify any missing file. ceph-osd@.service and ceph-volume@.service are present. I'm looking on the other servers for some other file that might be missing but can't find anything. Thanks for your help, Rodrigo

1 0

Re: Ceph OSD not mounting after reboot
by Rodrigo Severo - Fábrica 04 Jul '20

04 Jul '20

Em sex., 3 de jul. de 2020 às 17:41, Marc Roos <M.Roos(a)f1-outsourcing.eu> escreveu: > > So mount it, if it is empty Sure. That was my first impulse but as I said, in my other osd servers, these mounts are tmpfs filesystems. It's easy to manually mount them but how would I populate them? Regards, Rodrigo Severo > > > > -----Original Message----- > To: ceph-users > Subject: [ceph-users] Ceph OSD not mounting after reboot > > Hi, > > > Just rebooted one of my OSD servers after upgrading Ceph from 14.2.9 to > 14.2.10 and it's OSDs won't come up. > > I find the following messages on my log: > > 4991 Jul 3 17:24:03 osdserver1-df ceph-osd[1272]: 2020-07-03 > 17:24:03.036 7fcc497f1c00 -1 auth: unable to find a keyring on > /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory > 4992 Jul 3 17:24:03 osdserver1-df ceph-osd[1272]: 2020-07-03 > 17:24:03.036 7fcc497f1c00 -1 AuthRegistry(0x55e2ff810140) no keyring > found at /var/lib/ceph/osd/ceph-6/keyring, disabling cephx > > and my /var/lib/ceph/osd/ceph-6 directory is empty. > > I see that on my other servers these /var/lib/ceph/osd/ceph-? > directories are tmpfs mounts but I can't understand who is responsible > for mounting them as there are no entries for them in /etc/fstab. > > How can I fix this osd server? > > > Regards, > > Rodrigo Severo > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > >

2 1

Ceph OSD not mounting after reboot
by Rodrigo Severo - Fábrica 04 Jul '20

04 Jul '20

Hi, Just rebooted one of my OSD servers after upgrading Ceph from 14.2.9 to 14.2.10 and it's OSDs won't come up. I find the following messages on my log: 4991 Jul 3 17:24:03 osdserver1-df ceph-osd[1272]: 2020-07-03 17:24:03.036 7fcc497f1c00 -1 auth: unable to find a keyring on /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 4992 Jul 3 17:24:03 osdserver1-df ceph-osd[1272]: 2020-07-03 17:24:03.036 7fcc497f1c00 -1 AuthRegistry(0x55e2ff810140) no keyring found at /var/lib/ceph/osd/ceph-6/keyring, disabling cephx and my /var/lib/ceph/osd/ceph-6 directory is empty. I see that on my other servers these /var/lib/ceph/osd/ceph-? directories are tmpfs mounts but I can't understand who is responsible for mounting them as there are no entries for them in /etc/fstab. How can I fix this osd server? Regards, Rodrigo Severo

4 4

[Octopus] OSD won’t work with Docker
by Sean Johnson 03 Jul '20

03 Jul '20

I have a situation were OSDs won’t work as Docker containers with Octopus on an Ubuntu 2020.04 host. The cephadm adopt —style legacy —name osd.8 command works as expected, and sets up the /var/lib/ceph/ directory as expected: root@balin:~# ll /var/lib/ceph/c3d06c94-bb66-4f84-bf78-470a2364b667/osd.8/ total 64 drwx------ 2 167 167 4096 Jul 3 08:39 ./ drwx------ 8 167 167 4096 Jul 3 08:30 ../ lrwxrwxrwx 1 167 167 93 Jul 3 08:39 block -> /dev/ceph-590abe6f-abac-46b7-9455-80f69b63cf89/osd-block-5b229640-14e5-4e7f-9993-9368d172c30c -rw------- 1 167 167 37 Jul 3 08:39 ceph_fsid -rw------- 1 167 167 283 Jul 3 08:14 config -rw------- 1 167 167 37 Jul 3 08:39 fsid -rw------- 1 167 167 55 Jul 3 08:39 keyring -rw------- 1 167 167 6 Jul 3 08:39 ready -rw------- 1 167 167 3 May 14 21:41 require_osd_release -rw------- 1 167 167 10 Jul 3 08:39 type -rw------- 1 167 167 38 Jul 3 08:14 unit.configured -rw------- 1 167 167 48 May 14 21:41 unit.created -rw------- 1 167 167 28 Jul 3 08:14 unit.image -rw------- 1 167 167 825 Jul 3 08:14 unit.poststop -rw------- 1 167 167 2367 Jul 3 08:14 unit.run -rw------- 1 167 167 2 Jul 3 08:39 whoami However, the problem shows up when the container starts. Jul 03 08:39:40 balin systemd[1]: Started Ceph osd.8 for c3d06c94-bb66-4f84-bf78-470a2364b667. Jul 03 08:39:41 balin bash[5412]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-8 Jul 03 08:39:41 balin bash[5412]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-590abe6f-abac-46b7-9455-80f69b63cf89/osd-blo> Jul 03 08:39:41 balin bash[5412]: Running command: /usr/bin/ln -snf /dev/ceph-590abe6f-abac-46b7-9455-80f69b63cf89/osd-block-5b229640-14e5-4e7f-9993-9368d172c30c /var/li> Jul 03 08:39:41 balin bash[5412]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-8/block Jul 03 08:39:41 balin bash[5412]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 Jul 03 08:39:41 balin bash[5412]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-8 Jul 03 08:39:41 balin bash[5412]: --> ceph-volume lvm activate successful for osd ID: 8 Jul 03 08:39:41 balin bash[5482]: debug 2020-07-03T13:39:41.596+0000 7f52b1e70f40 0 set uid:gid to 167:167 (ceph:ceph) Jul 03 08:39:41 balin bash[5482]: debug 2020-07-03T13:39:41.596+0000 7f52b1e70f40 0 ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable), pro> Jul 03 08:39:41 balin bash[5482]: debug 2020-07-03T13:39:41.596+0000 7f52b1e70f40 0 pidfile_write: ignore empty --pid-file Jul 03 08:39:41 balin bash[5482]: debug 2020-07-03T13:39:41.600+0000 7f52b1e70f40 1 bdev create path /var/lib/ceph/osd/ceph-8/block type kernel Jul 03 08:39:41 balin bash[5482]: debug 2020-07-03T13:39:41.600+0000 7f52b1e70f40 1 bdev(0x55b992cd8000 /var/lib/ceph/osd/ceph-8/block) open path /var/lib/ceph/osd/ceph> Jul 03 08:39:41 balin bash[5482]: debug 2020-07-03T13:39:41.600+0000 7f52b1e70f40 1 bdev(0x55b992cd8000 /var/lib/ceph/osd/ceph-8/block) open size 12000134430720 (0xae9f> Jul 03 08:39:41 balin bash[5482]: debug 2020-07-03T13:39:41.600+0000 7f52b1e70f40 1 bluestore(/var/lib/ceph/osd/ceph-8) _set_cache_sizes cache_size 1073741824 meta 0.4 > Jul 03 08:39:41 balin bash[5482]: debug 2020-07-03T13:39:41.600+0000 7f52b1e70f40 1 bdev create path /var/lib/ceph/osd/ceph-8/block type kernel Jul 03 08:39:41 balin bash[5482]: debug 2020-07-03T13:39:41.600+0000 7f52b1e70f40 1 bdev(0x55b992cd8700 /var/lib/ceph/osd/ceph-8/block) open path /var/lib/ceph/osd/ceph> Jul 03 08:39:41 balin bash[5482]: debug 2020-07-03T13:39:41.600+0000 7f52b1e70f40 1 bdev(0x55b992cd8700 /var/lib/ceph/osd/ceph-8/block) open size 12000134430720 (0xae9f> Jul 03 08:39:41 balin bash[5482]: debug 2020-07-03T13:39:41.600+0000 7f52b1e70f40 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-8/block size 11 TiB Jul 03 08:39:41 balin bash[5482]: debug 2020-07-03T13:39:41.600+0000 7f52b1e70f40 1 bdev(0x55b992cd8700 /var/lib/ceph/osd/ceph-8/block) close Jul 03 08:39:41 balin bash[5482]: debug 2020-07-03T13:39:41.888+0000 7f52b1e70f40 1 bdev(0x55b992cd8000 /var/lib/ceph/osd/ceph-8/block) close Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.141+0000 7f52b1e70f40 0 starting osd.8 osd_data /var/lib/ceph/osd/ceph-8 /var/lib/ceph/osd/ceph-8/journal Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.153+0000 7f52b1e70f40 0 load: jerasure load: lrc load: isa Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.153+0000 7f52b1e70f40 1 bdev create path /var/lib/ceph/osd/ceph-8/block type kernel Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.153+0000 7f52b1e70f40 1 bdev(0x55b992cd8000 /var/lib/ceph/osd/ceph-8/block) open path /var/lib/ceph/osd/ceph> Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.153+0000 7f52b1e70f40 -1 bdev(0x55b992cd8000 /var/lib/ceph/osd/ceph-8/block) open open got: (13) Permission d> Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.157+0000 7f52b1e70f40 0 osd.8:0.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQu> Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.157+0000 7f52b1e70f40 0 osd.8:1.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQu> Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.157+0000 7f52b1e70f40 0 osd.8:2.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQu> Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.157+0000 7f52b1e70f40 0 osd.8:3.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQu> Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.157+0000 7f52b1e70f40 0 osd.8:4.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQu> Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.157+0000 7f52b1e70f40 -1 bluestore(/var/lib/ceph/osd/ceph-8/block) _read_bdev_label failed to open /var/lib/c> Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.157+0000 7f52b1e70f40 1 bluestore(/var/lib/ceph/osd/ceph-8) _mount path /var/lib/ceph/osd/ceph-8 Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.157+0000 7f52b1e70f40 -1 bluestore(/var/lib/ceph/osd/ceph-8/block) _read_bdev_label failed to open /var/lib/c> Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.157+0000 7f52b1e70f40 1 bdev create path /var/lib/ceph/osd/ceph-8/block type kernel Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.157+0000 7f52b1e70f40 1 bdev(0x55b992cd8000 /var/lib/ceph/osd/ceph-8/block) open path /var/lib/ceph/osd/ceph> Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.157+0000 7f52b1e70f40 -1 bdev(0x55b992cd8000 /var/lib/ceph/osd/ceph-8/block) open open got: (13) Permission d> Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.157+0000 7f52b1e70f40 -1 osd.8 0 OSD:init: unable to mount object store Jul 03 08:39:42 balin bash[5482]: debug 2020-07-03T13:39:42.157+0000 7f52b1e70f40 -1 ** ERROR: osd init failed: (13) Permission denied Jul 03 08:39:42 balin systemd[1]: ceph-c3d06c94-bb66-4f84-bf78-470a2364b667(a)osd.8.service: Main process exited, code=exited, status=1/FAILURE Jul 03 08:39:42 balin systemd[1]: ceph-c3d06c94-bb66-4f84-bf78-470a2364b667(a)osd.8.service: Failed with result 'exit-code'. Jul 03 08:39:53 balin systemd[1]: ceph-c3d06c94-bb66-4f84-bf78-470a2364b667(a)osd.8.service: Scheduled restart job, restart counter is at 6. Jul 03 08:39:53 balin systemd[1]: Stopped Ceph osd.8 for c3d06c94-bb66-4f84-bf78-470a2364b667. Jul 03 08:39:53 balin systemd[1]: ceph-c3d06c94-bb66-4f84-bf78-470a2364b667(a)osd.8.service: Start request repeated too quickly. Jul 03 08:39:53 balin systemd[1]: ceph-c3d06c94-bb66-4f84-bf78-470a2364b667(a)osd.8.service: Failed with result 'exit-code'. Jul 03 08:39:53 balin systemd[1]: Failed to start Ceph osd.8 for c3d06c94-bb66-4f84-bf78-470a2364b667. There’s a permission denied issue with the block, and I can’t find what might be causing that. If I set up an OSD fresh, it works fine with the docker container until I reboot, and then I get the same permission denied error. If I use ceph-volume lvm activate a standard ceph systemd service is created, and the OSD comes online without any problem, though it’s not visible to ceph orch.

1 1

Object Gateway not working within the dashboard anymore after network change
by Hendrik Peyerl 03 Jul '20

03 Jul '20

Hi all, we are currently experiencing a problem with the Obejct Gateway part of the dashboard not working anymore: We had a working setup were the RGW servers only had 1 network interface with an IP address that was reachable by the monitor servers and the dashboard was working as expected. After our initial tests everything was working great and we decided to add another physical link to the RGW Servers for the traffic to the clients. With that network change we also had to set the default gateway to the new interface while adding static routes for the rest of the ceph environment. To avoid issues with hostnames (the old hostname now resolves to the new interface) we added another hostname for the internal traffic, purged the gateways from ceph and added them again via ceph-deploy rgw create with the new hostname. The S3 communication is working perfectly fine as it did before, we can reach all buckets and the monitors can communicate with the Gateway. The Dashboard however throws the following error whenever we navigate to any of the object gateway menus: ————————————————————————————————— 2020-07-03 10:33:41.871 7fa0f9dbc700 0 mgr[dashboard] [03/Jul/2020:10:33:41] HTTP Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cherrypy/_cprequest.py", line 656, in respond response.body = self.handler() File "/usr/lib/python2.7/site-packages/cherrypy/lib/encoding.py", line 188, in __call__ self.body = self.oldhandler(*args, **kwargs) File "/usr/lib/python2.7/site-packages/cherrypy/_cptools.py", line 221, in wrap return self.newhandler(innerfunc, *args, **kwargs) File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 88, in dashboard_exception_handler return handler(*args, **kwargs) File "/usr/lib/python2.7/site-packages/cherrypy/_cpdispatch.py", line 34, in __call__ return self.callable(*self.args, **self.kwargs) File "/usr/share/ceph/mgr/dashboard/controllers/__init__.py", line 661, in inner ret = func(*args, **kwargs) File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 28, in status if not instance.is_service_online(): File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 507, in func_wrapper **kwargs) File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 321, in is_service_online _ = request({'format': 'json'}) File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 313, in __call__ data, raw_content) File "/usr/share/ceph/mgr/dashboard/rest_client.py", line 445, in do_request ex.args[0].reason.args[0]) File "/usr/lib64/python2.7/re.py", line 137, in match return _compile(pattern, flags).match(string) TypeError: expected string or buffer 2020-07-03 10:33:41.872 7fa0f9dbc700 0 mgr[dashboard] [2a02:2e0:13::a05:42784] [GET] [500] [45.044s] [plusline] [1.8K] /api/rgw/status 2020-07-03 10:33:41.872 7fa0f9dbc700 0 mgr[dashboard] ['{"status": "500 Internal Server Error", "version": "3.2.2", "traceback": "Traceback (most recent call last):\\n File \\"/usr/lib/python2.7/site- packages/cherrypy/_cprequest.py\\", line 656, in respond\\n response.body = self.handler()\\n File \\"/usr/lib/python2.7/site-packages/cherrypy/lib/encoding.py\\", line 188, in __call__\\n self.b ody = self.oldhandler(*args, **kwargs)\\n File \\"/usr/lib/python2.7/site-packages/cherrypy/_cptools.py\\", line 221, in wrap\\n return self.newhandler(innerfunc, *args, **kwargs)\\n File \\"/usr/s hare/ceph/mgr/dashboard/services/exception.py\\", line 88, in dashboard_exception_handler\\n return handler(*args, **kwargs)\\n File \\"/usr/lib/python2.7/site-packages/cherrypy/_cpdispatch.py\\", l ine 34, in __call__\\n return self.callable(*self.args, **self.kwargs)\\n File \\"/usr/share/ceph/mgr/dashboard/controllers/__init__.py\\", line 661, in inner\\n ret = func(*args, **kwargs)\\n F ile \\"/usr/share/ceph/mgr/dashboard/controllers/rgw.py\\", line 28, in status\\n if not instance.is_service_online():\\n File \\"/usr/share/ceph/mgr/dashboard/rest_client.py\\", line 507, in func_w rapper\\n **kwargs)\\n File \\"/usr/share/ceph/mgr/dashboard/services/rgw_client.py\\", line 321, in is_service_online\\n _ = request({\'format\': \'json\'})\\n File \\"/usr/share/ceph/mgr/dashb oard/rest_client.py\\", line 313, in __call__\\n data, raw_content)\\n File \\"/usr/share/ceph/mgr/dashboard/rest_client.py\\", line 445, in do_request\\n ex.args[0].reason.args[0])\\n File \\"/usr/lib64/python2.7/re.py\\", line 137, in match\\n return _compile(pattern, flags).match(string)\\nTypeError: expected string or buffer\\n", "detail": "The server encountered an unexpected condition which prevented it from fulfilling the request.", "request_id": "e0d6ff11-4dad-496a-9ee7-9db036c46ab7"}'] ————————————————————————————————— We are running ceph version 14.2.9 on CentOS 7.7. Any help on how to debug this would be greatly apreciated. Best Regards, Hendrik

2 3

Cannot remove cache tier
by Alexander E. Patrakov 03 Jul '20

03 Jul '20

Hello. I have tried to follow through the documented writeback cache tier removal procedure (https://docs.ceph.com/docs/master/rados/operations/cache-tiering/#removing-…) on a test cluster, and failed. I have successfully executed this command: ceph osd tier cache-mode alex-test-rbd-cache proxy Next, I am supposed to run this: rados -p alex-test-rbd-cache ls rados -p alex-test-rbd-cache cache-flush-evict-all The failure mode is that, while the client i/o still going on, I cannot get zero objects in the cache pool, even with the help of "rados -p alex-test-rbd-cache cache-flush-evict-all". And yes, I have waited more than 20 minutes (my cache tier has hit_set_count 10 and hit_set_period 120). I also tried to set both cache_target_dirty_ratio and cache_target_full_ratio to 0, it didn't help. Here is the relevant part of the pool setup: # ceph osd pool ls detail pool 25 'alex-test-rbd-metadata' replicated size 3 min_size 2 crush_rule 9 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode warn last_change 10973111 lfor 0/10971347/10971345 flags hashpspool,nodelete stripe_width 0 application rbd pool 26 'alex-test-rbd-data' erasure size 6 min_size 5 crush_rule 12 object_hash rjenkins pg_num 1024 pgp_num 1024 autoscale_mode warn last_change 10973112 lfor 10971705/10971705/10971705 flags hashpspool,ec_overwrites,nodelete,selfmanaged_snaps tiers 27 read_tier 27 write_tier 27 stripe_width 16384 application rbd removed_snaps [1~3] pool 27 'alex-test-rbd-cache' replicated size 3 min_size 2 crush_rule 9 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode warn last_change 10973113 lfor 10971705/10971705/10971705 flags hashpspool,incomplete_clones,nodelete,selfmanaged_snaps tier_of 26 cache_mode proxy target_bytes 10000000000 hit_set bloom{false_positive_probability: 0.05, target_size: 0, seed: 0} 120s x10 decay_rate 0 search_last_n 0 stripe_width 0 application rbd removed_snaps [1~3] The relevant crush rules are selecting ssds for the alex-test-rbd-cache and alex-test-rbd-metadata pools (plain old "replicated size 3" pools), and hdds for alex-test-rbd-data (which is EC 4+2). The client workload, which seemingly outpaces the eviction and flushing, is: for a in `seq 1000 2000` ; do time rbd import --data-pool alex-test-rbd-data ./Fedora-Cloud-Base-32-1.6.x86_64.raw alex-test-rbd-metadata/Fedora-copy-$a done The ceph version is "ceph version 14.2.9 (2afdc1f644870fb6315f25a777f9e4126dacc32d) nautilus (stable)" on all osds. The relevant part of "ceph df" is: RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 23 TiB 20 TiB 2.9 TiB 3.0 TiB 12.99 ssd 1.7 TiB 1.7 TiB 19 GiB 23 GiB 1.28 TOTAL 25 TiB 22 TiB 2.9 TiB 3.0 TiB 12.17 POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL <irrelevant pools omitted> alex-test-rbd-metadata 25 237 KiB 2.37k 59 MiB 0 564 GiB alex-test-rbd-data 26 691 GiB 198.57k 1.0 TiB 6.52 9.7 TiB alex-test-rbd-cache 27 5.1 GiB 2.99k 15 GiB 0.90 564 GiB The total size and the number of stored objects in the alex-test-rbd-cache pool oscillate around 5 GB and 3K, respectively, while "rados -p alex-test-rbd-cache cache-flush-evict-all" is running in a loop. Without it, the size grows to 6 GB and stays there. # ceph -s cluster: id: <omitted for privacy> health: HEALTH_WARN 1 cache pools at or near target size services: mon: 3 daemons, quorum xx-4a,xx-3a,xx-2a (age 10d) mgr: xx-3a(active, since 5w), standbys: xx-2b, xx-2a, xx-4a mds: cephfs:1 {0=xx-4b=up:active} 2 up:standby osd: 89 osds: 89 up (since 7d), 89 in (since 7d) rgw: 3 daemons active (xx-2b, xx-3b, xx-4b) tcmu-runner: 6 daemons active (<only irrelevant images here>) data: pools: 15 pools, 1976 pgs objects: 6.64M objects, 1.3 TiB usage: 3.1 TiB used, 22 TiB / 25 TiB avail pgs: 1976 active+clean io: client: 290 KiB/s rd, 251 MiB/s wr, 366 op/s rd, 278 op/s wr cache: 123 MiB/s flush, 72 MiB/s evict, 31 op/s promote, 3 PGs flushing, 1 PGs evicting Is there any workaround, short of somehow telling the client to stop creating new rbds? -- Alexander E. Patrakov CV: http://pc.cd/PLz7

1 0

Re: NFS Ganesha 2.7 in Xenial not available
by Victoria Martinez de la Cruz 03 Jul '20

03 Jul '20

Thanks Ramana and David. So we are using the Shaman search API to get the latest build for ceph_nautilus flavor of NFS Ganesha, and that's how we get to the mentioned build. We are doing this since it's part of our CI and it's better for automation. Should we use different repos? Thanks, V On Wed, Jun 24, 2020 at 3:33 PM Victoria Martinez de la Cruz < vkmc(a)redhat.com> wrote: > Thanks Ramana and David. > > So we are using the Shaman search API to get the latest build for > ceph_nautilus flavor of NFS Ganesha, and that's how we get to the mentioned > build. We are doing this since it's part of our CI and it's better for > automation. > > Should we use different repos? > > Thanks, > > V > > On Tue, Jun 23, 2020 at 2:42 PM David Galloway <dgallowa(a)redhat.com> > wrote: > >> >> >> On 6/23/20 1:21 PM, Ramana Venkatesh Raja wrote: >> > On Tue, Jun 23, 2020 at 6:59 PM Victoria Martinez de la Cruz >> > <victoria(a)redhat.com> wrote: >> >> >> >> Hi folks, >> >> >> >> I'm hitting issues with the nfs-ganesha-stable packages [0], the repo >> url >> >> [1] is broken. Is there a known issue for this? >> >> >> > >> > The missing packages in chacra could be due to the recent mishap in >> > the sepia long running cluster, >> > >> https://lists.ceph.io/hyperkitty/list/dev@ceph.io/thread/YQMAHTB7MUHL25QP7V… >> >> Hi Victoria, >> >> Ramana is correct. Do you need 2.7.4 specifically? If not, signed >> nfs-ganesha packages can also be found here: >> http://download.ceph.com/nfs-ganesha/ >> >> > >> >> Thanks, >> >> >> >> Victoria >> >> >> >> [0] >> >> >> https://shaman.ceph.com/repos/nfs-ganesha-stable/V2.7-stable/1a1fb71cdb811c… >> >> [1] >> >> >> https://chacra.ceph.com/r/nfs-ganesha-stable/V2.7-stable/1a1fb71cdb811c1bac… >> >> _______________________________________________ >> >> ceph-users mailing list -- ceph-users(a)ceph.io >> >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >> > >> >>

3 3

rbd audit
by Seena Fallah 02 Jul '20

02 Jul '20

Hi all. Is there any rbd audit like ceph.audit.log that could log which client runs which command from rbd client? Thanks.

1 0

RES: Debian install
by Rafael Quaglio 02 Jul '20

02 Jul '20

Thanks for your reply Anastasios, I was waiting for some answer. My /etc/apt/sources.list.d/ceph.list content is: deb https://download.ceph.com/debian-nautilus/ buster main Even if I do “apt-get update”, the packages still the same. The Ceph client (CephFS mount) is working well, but I can´t deploy new osds. The error that I posted occurs when I do : “ceph-deploy osd create --data /dev/sdb node1” I appreciate any help. Rafael. De: Anastasios Dados <tdados(a)hotmail.com> Enviada em: segunda-feira, 29 de junho de 2020 20:01 Para: Rafael Quaglio <quaglio(a)bol.com.br>; ceph-users(a)ceph.io Assunto: Re: [ceph-users] Debian install Hello Rafael, Can you check the apt sources list that exist from your ceph-deploy node? Maybe there you have put luminous debian packages version? Regards, Anastasios On Mon, 2020-06-29 at 06:59 -0300, Rafael Quaglio wrote: Hi, We have already installed a new Debian (10.4) server and I need put it in a Ceph cluster. When I execute the command to install ceph on this node: ceph-deploy install --release nautilus node1 It starts to install a version 12.x in my node... (...) [serifos][DEBUG ] After this operation, 183 MB of additional disk space will be used. [serifos][DEBUG ] Selecting previously unselected package python-cephfs. (Reading database ... 30440 files and directories currently installed.) [serifos][DEBUG ] Preparing to unpack .../python-cephfs_12.2.11+dfsg1-2.1+b1_amd64.deb ... [serifos][DEBUG ] Unpacking python-cephfs (12.2.11+dfsg1-2.1+b1) ... [serifos][DEBUG ] Selecting previously unselected package ceph-common. [serifos][DEBUG ] Preparing to unpack .../ceph-common_12.2.11+dfsg1-2.1+b1_amd64.deb ... [serifos][DEBUG ] Unpacking ceph-common (12.2.11+dfsg1-2.1+b1) ... (...) How do I upgrade this packages? Even installed packages in this version, the installation completes without erros. The question is due to an error message that I'm recieving when deploy a new osd. ceph-deploy osd create --data /dev/sdb node1 At this point: [ceph_deploy.osd][INFO ] Distro info: debian 10.4 buster [ceph_deploy.osd][DEBUG ] Deploying osd to node1 [node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf [node1][DEBUG ] find the location of an executable [node1][INFO ] Running command: sudo /usr/sbin/ceph-volume --cluster ceph lvm create --bluestore --data /dev/sdb [node1][WARNIN] --> RuntimeError: Unable to create a new OSD id [node1][DEBUG ] Running command: /bin/ceph-authtool --gen-print-key [node1][DEBUG ] Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 76da6c51-8385-4ffc-9a8e-0dfc11e31feb [node1][DEBUG ] stderr: /build/ceph-qtARip/ceph-12.2.11+dfsg1/src/mon/MonMap.cc: In function 'void MonMap::sanitize_mons(std::map<std::__cxx11::basic_string<char>, entity_addr_t>&)' thread 7f2bc7fff700 time 2020-06-29 06:56:17.331350 [node1][DEBUG ] stderr: /build/ceph-qtARip/ceph-12.2.11+dfsg1/src/mon/MonMap.cc: 77: FAILED assert(mon_info[p.first].public_addr == p.second) [node1][DEBUG ] stderr: ceph version 12.2.11 (26dc3775efc7bb286a1d6d66faee0ba30ea23eee) luminous (stable) [node1][DEBUG ] stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0xf5) [0x7f2bdaff5f75] [node1][DEBUG ] stderr: 2: (MonMap::sanitize_mons(std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, entity_addr_t, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, entity_addr_t> > &)+0x568) [0x7f2bdb050038] [node1][DEBUG ] stderr: 3: (MonMap::decode(ceph::buffer::list::iterator&)+0x4da) [0x7f2bdb05500a] [node1][DEBUG ] stderr: 4: (MonClient::handle_monmap(MMonMap*)+0x216) [0x7f2bdb042a06] [node1][DEBUG ] stderr: 5: (MonClient::ms_dispatch(Message*)+0x4ab) [0x7f2bdb04729b] [node1][DEBUG ] stderr: 6: (DispatchQueue::entry()+0xeba) [0x7f2bdb06bf5a] [node1][DEBUG ] stderr: 7: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f2bdb1576fd] [node1][DEBUG ] stderr: 8: (()+0x7fa3) [0x7f2be499dfa3] [node1][DEBUG ] stderr: 9: (clone()+0x3f) [0x7f2be45234cf] [node1][DEBUG ] stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. [node1][ERROR ] RuntimeError: command returned non-zero exit status: 1 [ceph_deploy.osd][ERROR ] Failed to execute command: /usr/sbin/ceph-volume --cluster ceph lvm create --bluestore --data /dev/sdb [ceph_deploy][ERROR ] GenericError: Failed to create 1 OSDs I think this error occurs because the wrong package that was installed. Thanks, Rafael _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave(a)ceph.io <mailto:ceph-users-leave@ceph.io>

1 1

[ceph-fs]questions about large omap objects
by norman 02 Jul '20

02 Jul '20

Hi guys, I'm using ceph fs , I met a problem of large object in my fs meta pool. Anyone can tell me how to avoid the problem? thanks. best regards, kern

1 0

Upgrade from Luminous to Nautilus 14.2.9 RBD issue?
by Daniel Stan - nav.ro 02 Jul '20

02 Jul '20

Hi, We are experiencing a weird issue after upgrading our clusters from ceph luminous to nautilus 14.2.9 - I am not even sure if this is ceph related but this started to happen exactly after we upgraded, so, I am trying my luck here. We have one ceph rbd pool size 3 min size 2 from all bluestore osds (KRBD) I will try to be clear enough.. though I cannot understand exactly whats happening or whats causing the issue. So, we have 1 virtual machine which uses a rbd image of 2TB - virtio-scsi device. Inside the VM we are trying to create ploop devices to be used for/by containers(inside the VM on the 2TB rbd image QEMU DISK). There is no way we can create ploop devices, it always crash, please check the crash below: https://pastebin.com/9khp9XS3 - sdb in the crash is the 2TB rbd image which the VM uses. There are no other read/write errors, we have health_ok, all OSDs are fine, no errors on any of the phisical disks - this happens only when we want to create ploop devices inside a VM and right after we upgraded our cluster to nautilus 14.2.9. I also did new images/other hosts.. same result. Did try a lot of different versions of ploop packages, same result. I would appreciate if someone else has encountered something similar and if there is a workaround. -- Best Regards, ------------------------------------------------------------------------ Daniel Stan Senior System Administrator | NAV Communications (RO) Office: +40 (21) 655-55-55 | E-Mail: daniel(a)nav.ro Site: www.nav.ro <https://www.nav.ro> | Client: https://client.ro

6 8

Lifecycle message on logs
by Marcelo Miziara 02 Jul '20

02 Jul '20

Hello...it's the first time I need to use the lifecycle, and I created a bucket and set it to expire in one day with s3cmd: s3cmd expire --expiry-days=1 s3://bucket The rgw_lifecycle_work_time is set to the default values(00:00-06:00). But I noticed in the rgw logs a lot of messages like: 2020-06-16 00:00:00.311369 7fe2cac87700 0 RGWLC::process() failed to get obj entry lc.8 2020-06-16 00:00:00.311623 7fe2c8c83700 0 RGWLC::process() failed to get obj entry lc.16 2020-06-16 00:00:00.311862 7fe2c6c7f700 0 RGWLC::process() failed to get obj entry lc.4 2020-06-16 00:00:00.319424 7fe2cac87700 0 RGWLC::process() failed to get obj entry lc.10 2020-06-16 00:00:00.319647 7fe2c8c83700 0 RGWLC::process() failed to get obj entry lc.18 2020-06-16 00:00:00.320682 7fe2c6c7f700 0 RGWLC::process() failed to get obj entry lc.16 2020-06-16 00:00:00.327770 7fe2cac87700 0 RGWLC::process() failed to get obj entry lc.6 2020-06-16 00:00:00.328941 7fe2c8c83700 0 RGWLC::process() failed to get obj entry lc.17 2020-06-16 00:00:00.332463 7fe2c6c7f700 0 RGWLC::process() failed to get obj entry lc.20 2020-06-16 00:00:00.336788 7fe2cac87700 0 RGWLC::process() failed to get obj entry lc.1 2020-06-16 00:00:00.336924 7fe2c8c83700 0 RGWLC::process() failed to get obj entry lc.24 2020-06-16 00:00:00.340915 7fe2c6c7f700 0 RGWLC::process() failed to get obj entry lc.2 The object was deleted, but these messages keep appearing. Is it safe to ignore them? For the records, i'm using redhat luminous 12.2.12 Thanks, Marcelo.

2 1

find rbd locks by client IP
by Void Star Nill 01 Jul '20

01 Jul '20

Hello, Is there a way to list all locks held by a client with the given IP address? Also, I read somewhere that removing the lock with "rbd lock rm..." automatically blacklists that client connection. Is that correct? How do I blacklist a client with the given IP address? Thanks, Shridhar

2 2

How to change 'ceph mon metadata' hostname value in octopus.
by Cem Zafer 01 Jul '20

01 Jul '20

Hi forum people, What is the best method to change monitor metadata in octopus? Thanks.

1 0