- ceph-users - lists.ceph.io

Cannot remove cache tier
by Alexander E. Patrakov 03 Jul '20

03 Jul '20

Hello. I have tried to follow through the documented writeback cache tier removal procedure (https://docs.ceph.com/docs/master/rados/operations/cache-tiering/#removing-…) on a test cluster, and failed. I have successfully executed this command: ceph osd tier cache-mode alex-test-rbd-cache proxy Next, I am supposed to run this: rados -p alex-test-rbd-cache ls rados -p alex-test-rbd-cache cache-flush-evict-all The failure mode is that, while the client i/o still going on, I cannot get zero objects in the cache pool, even with the help of "rados -p alex-test-rbd-cache cache-flush-evict-all". And yes, I have waited more than 20 minutes (my cache tier has hit_set_count 10 and hit_set_period 120). I also tried to set both cache_target_dirty_ratio and cache_target_full_ratio to 0, it didn't help. Here is the relevant part of the pool setup: # ceph osd pool ls detail pool 25 'alex-test-rbd-metadata' replicated size 3 min_size 2 crush_rule 9 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode warn last_change 10973111 lfor 0/10971347/10971345 flags hashpspool,nodelete stripe_width 0 application rbd pool 26 'alex-test-rbd-data' erasure size 6 min_size 5 crush_rule 12 object_hash rjenkins pg_num 1024 pgp_num 1024 autoscale_mode warn last_change 10973112 lfor 10971705/10971705/10971705 flags hashpspool,ec_overwrites,nodelete,selfmanaged_snaps tiers 27 read_tier 27 write_tier 27 stripe_width 16384 application rbd removed_snaps [1~3] pool 27 'alex-test-rbd-cache' replicated size 3 min_size 2 crush_rule 9 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode warn last_change 10973113 lfor 10971705/10971705/10971705 flags hashpspool,incomplete_clones,nodelete,selfmanaged_snaps tier_of 26 cache_mode proxy target_bytes 10000000000 hit_set bloom{false_positive_probability: 0.05, target_size: 0, seed: 0} 120s x10 decay_rate 0 search_last_n 0 stripe_width 0 application rbd removed_snaps [1~3] The relevant crush rules are selecting ssds for the alex-test-rbd-cache and alex-test-rbd-metadata pools (plain old "replicated size 3" pools), and hdds for alex-test-rbd-data (which is EC 4+2). The client workload, which seemingly outpaces the eviction and flushing, is: for a in `seq 1000 2000` ; do time rbd import --data-pool alex-test-rbd-data ./Fedora-Cloud-Base-32-1.6.x86_64.raw alex-test-rbd-metadata/Fedora-copy-$a done The ceph version is "ceph version 14.2.9 (2afdc1f644870fb6315f25a777f9e4126dacc32d) nautilus (stable)" on all osds. The relevant part of "ceph df" is: RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 23 TiB 20 TiB 2.9 TiB 3.0 TiB 12.99 ssd 1.7 TiB 1.7 TiB 19 GiB 23 GiB 1.28 TOTAL 25 TiB 22 TiB 2.9 TiB 3.0 TiB 12.17 POOLS: POOL ID STORED OBJECTS USED %USED MAX AVAIL <irrelevant pools omitted> alex-test-rbd-metadata 25 237 KiB 2.37k 59 MiB 0 564 GiB alex-test-rbd-data 26 691 GiB 198.57k 1.0 TiB 6.52 9.7 TiB alex-test-rbd-cache 27 5.1 GiB 2.99k 15 GiB 0.90 564 GiB The total size and the number of stored objects in the alex-test-rbd-cache pool oscillate around 5 GB and 3K, respectively, while "rados -p alex-test-rbd-cache cache-flush-evict-all" is running in a loop. Without it, the size grows to 6 GB and stays there. # ceph -s cluster: id: <omitted for privacy> health: HEALTH_WARN 1 cache pools at or near target size services: mon: 3 daemons, quorum xx-4a,xx-3a,xx-2a (age 10d) mgr: xx-3a(active, since 5w), standbys: xx-2b, xx-2a, xx-4a mds: cephfs:1 {0=xx-4b=up:active} 2 up:standby osd: 89 osds: 89 up (since 7d), 89 in (since 7d) rgw: 3 daemons active (xx-2b, xx-3b, xx-4b) tcmu-runner: 6 daemons active (<only irrelevant images here>) data: pools: 15 pools, 1976 pgs objects: 6.64M objects, 1.3 TiB usage: 3.1 TiB used, 22 TiB / 25 TiB avail pgs: 1976 active+clean io: client: 290 KiB/s rd, 251 MiB/s wr, 366 op/s rd, 278 op/s wr cache: 123 MiB/s flush, 72 MiB/s evict, 31 op/s promote, 3 PGs flushing, 1 PGs evicting Is there any workaround, short of somehow telling the client to stop creating new rbds? -- Alexander E. Patrakov CV: http://pc.cd/PLz7

1 0

Re: NFS Ganesha 2.7 in Xenial not available
by Victoria Martinez de la Cruz 03 Jul '20

03 Jul '20

Thanks Ramana and David. So we are using the Shaman search API to get the latest build for ceph_nautilus flavor of NFS Ganesha, and that's how we get to the mentioned build. We are doing this since it's part of our CI and it's better for automation. Should we use different repos? Thanks, V On Wed, Jun 24, 2020 at 3:33 PM Victoria Martinez de la Cruz < vkmc(a)redhat.com> wrote: > Thanks Ramana and David. > > So we are using the Shaman search API to get the latest build for > ceph_nautilus flavor of NFS Ganesha, and that's how we get to the mentioned > build. We are doing this since it's part of our CI and it's better for > automation. > > Should we use different repos? > > Thanks, > > V > > On Tue, Jun 23, 2020 at 2:42 PM David Galloway <dgallowa(a)redhat.com> > wrote: > >> >> >> On 6/23/20 1:21 PM, Ramana Venkatesh Raja wrote: >> > On Tue, Jun 23, 2020 at 6:59 PM Victoria Martinez de la Cruz >> > <victoria(a)redhat.com> wrote: >> >> >> >> Hi folks, >> >> >> >> I'm hitting issues with the nfs-ganesha-stable packages [0], the repo >> url >> >> [1] is broken. Is there a known issue for this? >> >> >> > >> > The missing packages in chacra could be due to the recent mishap in >> > the sepia long running cluster, >> > >> https://lists.ceph.io/hyperkitty/list/dev@ceph.io/thread/YQMAHTB7MUHL25QP7V… >> >> Hi Victoria, >> >> Ramana is correct. Do you need 2.7.4 specifically? If not, signed >> nfs-ganesha packages can also be found here: >> http://download.ceph.com/nfs-ganesha/ >> >> > >> >> Thanks, >> >> >> >> Victoria >> >> >> >> [0] >> >> >> https://shaman.ceph.com/repos/nfs-ganesha-stable/V2.7-stable/1a1fb71cdb811c… >> >> [1] >> >> >> https://chacra.ceph.com/r/nfs-ganesha-stable/V2.7-stable/1a1fb71cdb811c1bac… >> >> _______________________________________________ >> >> ceph-users mailing list -- ceph-users(a)ceph.io >> >> To unsubscribe send an email to ceph-users-leave(a)ceph.io >> >> >> > >> >>

3 3

rbd audit
by Seena Fallah 02 Jul '20

02 Jul '20

Hi all. Is there any rbd audit like ceph.audit.log that could log which client runs which command from rbd client? Thanks.

1 0

RES: Debian install
by Rafael Quaglio 02 Jul '20

02 Jul '20

Thanks for your reply Anastasios, I was waiting for some answer. My /etc/apt/sources.list.d/ceph.list content is: deb https://download.ceph.com/debian-nautilus/ buster main Even if I do “apt-get update”, the packages still the same. The Ceph client (CephFS mount) is working well, but I can´t deploy new osds. The error that I posted occurs when I do : “ceph-deploy osd create --data /dev/sdb node1” I appreciate any help. Rafael. De: Anastasios Dados <tdados(a)hotmail.com> Enviada em: segunda-feira, 29 de junho de 2020 20:01 Para: Rafael Quaglio <quaglio(a)bol.com.br>; ceph-users(a)ceph.io Assunto: Re: [ceph-users] Debian install Hello Rafael, Can you check the apt sources list that exist from your ceph-deploy node? Maybe there you have put luminous debian packages version? Regards, Anastasios On Mon, 2020-06-29 at 06:59 -0300, Rafael Quaglio wrote: Hi, We have already installed a new Debian (10.4) server and I need put it in a Ceph cluster. When I execute the command to install ceph on this node: ceph-deploy install --release nautilus node1 It starts to install a version 12.x in my node... (...) [serifos][DEBUG ] After this operation, 183 MB of additional disk space will be used. [serifos][DEBUG ] Selecting previously unselected package python-cephfs. (Reading database ... 30440 files and directories currently installed.) [serifos][DEBUG ] Preparing to unpack .../python-cephfs_12.2.11+dfsg1-2.1+b1_amd64.deb ... [serifos][DEBUG ] Unpacking python-cephfs (12.2.11+dfsg1-2.1+b1) ... [serifos][DEBUG ] Selecting previously unselected package ceph-common. [serifos][DEBUG ] Preparing to unpack .../ceph-common_12.2.11+dfsg1-2.1+b1_amd64.deb ... [serifos][DEBUG ] Unpacking ceph-common (12.2.11+dfsg1-2.1+b1) ... (...) How do I upgrade this packages? Even installed packages in this version, the installation completes without erros. The question is due to an error message that I'm recieving when deploy a new osd. ceph-deploy osd create --data /dev/sdb node1 At this point: [ceph_deploy.osd][INFO ] Distro info: debian 10.4 buster [ceph_deploy.osd][DEBUG ] Deploying osd to node1 [node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf [node1][DEBUG ] find the location of an executable [node1][INFO ] Running command: sudo /usr/sbin/ceph-volume --cluster ceph lvm create --bluestore --data /dev/sdb [node1][WARNIN] --> RuntimeError: Unable to create a new OSD id [node1][DEBUG ] Running command: /bin/ceph-authtool --gen-print-key [node1][DEBUG ] Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 76da6c51-8385-4ffc-9a8e-0dfc11e31feb [node1][DEBUG ] stderr: /build/ceph-qtARip/ceph-12.2.11+dfsg1/src/mon/MonMap.cc: In function 'void MonMap::sanitize_mons(std::map<std::__cxx11::basic_string<char>, entity_addr_t>&)' thread 7f2bc7fff700 time 2020-06-29 06:56:17.331350 [node1][DEBUG ] stderr: /build/ceph-qtARip/ceph-12.2.11+dfsg1/src/mon/MonMap.cc: 77: FAILED assert(mon_info[p.first].public_addr == p.second) [node1][DEBUG ] stderr: ceph version 12.2.11 (26dc3775efc7bb286a1d6d66faee0ba30ea23eee) luminous (stable) [node1][DEBUG ] stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0xf5) [0x7f2bdaff5f75] [node1][DEBUG ] stderr: 2: (MonMap::sanitize_mons(std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, entity_addr_t, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, entity_addr_t> > &)+0x568) [0x7f2bdb050038] [node1][DEBUG ] stderr: 3: (MonMap::decode(ceph::buffer::list::iterator&)+0x4da) [0x7f2bdb05500a] [node1][DEBUG ] stderr: 4: (MonClient::handle_monmap(MMonMap*)+0x216) [0x7f2bdb042a06] [node1][DEBUG ] stderr: 5: (MonClient::ms_dispatch(Message*)+0x4ab) [0x7f2bdb04729b] [node1][DEBUG ] stderr: 6: (DispatchQueue::entry()+0xeba) [0x7f2bdb06bf5a] [node1][DEBUG ] stderr: 7: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f2bdb1576fd] [node1][DEBUG ] stderr: 8: (()+0x7fa3) [0x7f2be499dfa3] [node1][DEBUG ] stderr: 9: (clone()+0x3f) [0x7f2be45234cf] [node1][DEBUG ] stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. [node1][ERROR ] RuntimeError: command returned non-zero exit status: 1 [ceph_deploy.osd][ERROR ] Failed to execute command: /usr/sbin/ceph-volume --cluster ceph lvm create --bluestore --data /dev/sdb [ceph_deploy][ERROR ] GenericError: Failed to create 1 OSDs I think this error occurs because the wrong package that was installed. Thanks, Rafael _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io <mailto:ceph-users@ceph.io> To unsubscribe send an email to ceph-users-leave(a)ceph.io <mailto:ceph-users-leave@ceph.io>

1 1

[ceph-fs]questions about large omap objects
by norman 02 Jul '20

02 Jul '20

Hi guys, I'm using ceph fs , I met a problem of large object in my fs meta pool. Anyone can tell me how to avoid the problem? thanks. best regards, kern

1 0

Upgrade from Luminous to Nautilus 14.2.9 RBD issue?
by Daniel Stan - nav.ro 02 Jul '20

02 Jul '20

Hi, We are experiencing a weird issue after upgrading our clusters from ceph luminous to nautilus 14.2.9 - I am not even sure if this is ceph related but this started to happen exactly after we upgraded, so, I am trying my luck here. We have one ceph rbd pool size 3 min size 2 from all bluestore osds (KRBD) I will try to be clear enough.. though I cannot understand exactly whats happening or whats causing the issue. So, we have 1 virtual machine which uses a rbd image of 2TB - virtio-scsi device. Inside the VM we are trying to create ploop devices to be used for/by containers(inside the VM on the 2TB rbd image QEMU DISK). There is no way we can create ploop devices, it always crash, please check the crash below: https://pastebin.com/9khp9XS3 - sdb in the crash is the 2TB rbd image which the VM uses. There are no other read/write errors, we have health_ok, all OSDs are fine, no errors on any of the phisical disks - this happens only when we want to create ploop devices inside a VM and right after we upgraded our cluster to nautilus 14.2.9. I also did new images/other hosts.. same result. Did try a lot of different versions of ploop packages, same result. I would appreciate if someone else has encountered something similar and if there is a workaround. -- Best Regards, ------------------------------------------------------------------------ Daniel Stan Senior System Administrator | NAV Communications (RO) Office: +40 (21) 655-55-55 | E-Mail: daniel(a)nav.ro Site: www.nav.ro <https://www.nav.ro> | Client: https://client.ro

6 8

Lifecycle message on logs
by Marcelo Miziara 02 Jul '20

02 Jul '20

Hello...it's the first time I need to use the lifecycle, and I created a bucket and set it to expire in one day with s3cmd: s3cmd expire --expiry-days=1 s3://bucket The rgw_lifecycle_work_time is set to the default values(00:00-06:00). But I noticed in the rgw logs a lot of messages like: 2020-06-16 00:00:00.311369 7fe2cac87700 0 RGWLC::process() failed to get obj entry lc.8 2020-06-16 00:00:00.311623 7fe2c8c83700 0 RGWLC::process() failed to get obj entry lc.16 2020-06-16 00:00:00.311862 7fe2c6c7f700 0 RGWLC::process() failed to get obj entry lc.4 2020-06-16 00:00:00.319424 7fe2cac87700 0 RGWLC::process() failed to get obj entry lc.10 2020-06-16 00:00:00.319647 7fe2c8c83700 0 RGWLC::process() failed to get obj entry lc.18 2020-06-16 00:00:00.320682 7fe2c6c7f700 0 RGWLC::process() failed to get obj entry lc.16 2020-06-16 00:00:00.327770 7fe2cac87700 0 RGWLC::process() failed to get obj entry lc.6 2020-06-16 00:00:00.328941 7fe2c8c83700 0 RGWLC::process() failed to get obj entry lc.17 2020-06-16 00:00:00.332463 7fe2c6c7f700 0 RGWLC::process() failed to get obj entry lc.20 2020-06-16 00:00:00.336788 7fe2cac87700 0 RGWLC::process() failed to get obj entry lc.1 2020-06-16 00:00:00.336924 7fe2c8c83700 0 RGWLC::process() failed to get obj entry lc.24 2020-06-16 00:00:00.340915 7fe2c6c7f700 0 RGWLC::process() failed to get obj entry lc.2 The object was deleted, but these messages keep appearing. Is it safe to ignore them? For the records, i'm using redhat luminous 12.2.12 Thanks, Marcelo.

2 1

find rbd locks by client IP
by Void Star Nill 02 Jul '20

02 Jul '20

Hello, Is there a way to list all locks held by a client with the given IP address? Also, I read somewhere that removing the lock with "rbd lock rm..." automatically blacklists that client connection. Is that correct? How do I blacklist a client with the given IP address? Thanks, Shridhar

2 2

How to change 'ceph mon metadata' hostname value in octopus.
by Cem Zafer 01 Jul '20

01 Jul '20

Hi forum people, What is the best method to change monitor metadata in octopus? Thanks.

1 0

Bluestore performance tuning for hdd with nvme db+wal
by Mark Kirkwood 01 Jul '20

01 Jul '20

Hi, We have recently added a new storage node to our Luminous (12.2.13) cluster. The prev nodes are all setup as Filestore: e.g 12 osds on hdd (Seagate Constellations) with one NVMe (Intel P4600) journal. With the new guy we decided to introduce Bluestore so it is configured as: (same HW) 12 osd with data on hdd and db + wal on one NVMe. We noticed there are periodic slow requests logged, and the implicated osds are the Bluestore ones 98% of the time! This suggests that we need to tweak our Bluestore settings in some way. Investigating I'm seeing: - A great deal of rocksdb debug info in the logs - perhaps we should tone that down? (debug_rocksdb 4/5 -> 1/5) - We look to have the default cache settings (bluestore_cache_size_hdd|ssd etc), we have memory to increase these - There are some buffered io settings (bluefs_buffered_io, bluestore_default_buffered_write), set to (default) false. Are these safe (or useful) to change? - We have default rocksdb options, should some of these be changed? (bluestore_rocksdb_options, in particular max_background_compactions=2 - should we have less, or more?) Also, anything else we should be looking at? regards Mark

4 6