July 2021 - ceph-users - lists.ceph.io

OpenSSL security update for Octopus container?
by Robert Sander 02 Oct '25

02 Oct '25

Hi, I am running a Ceph Octopus cluster in version 15.2.10 setup with the cephadm orchestrator and official Docker hub container images. Yesterday an important security fix was released for libssl and is already packaged for all major distributions. I tried to run "ceph orch upgrade check docker.io/ceph/ceph:v15" but it tells me that the containers do not need to be upgraded. How will this security fix of OpenSSL be deployed in a timely manner to users of the Ceph container images? Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 93818 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 1

Correct Migration Workflow Replicated -> Erasure Code
by Mac Wynkoop 19 Jun '24

19 Jun '24

Hi Everyone, So, I'm in the process of trying to migrate our rgw.buckets.data pool from a replicated rule pool to an erasure coded pool. I've gotten the EC pool set up, good EC profile and crush ruleset, pool created successfully, but when I go to "rados cppool xxx.rgw.buckets.data xxx.rgw.buckets.data.new", I get this error after it transfers 4GB of data: error copying object: (2) No such file or directory error copying pool xxx.rgw.buckets.data => xxx.rgw.buckets.data.new: (2) No such file or directory Is "rados cppool" still the blessed way to do the migration, or has something better/not deprecated been developed that I can use? Thanks, Mac

4 4

cephfs creation error
by Ramanathan S 03 Apr '24

03 Apr '24

Hi all, I just had created a ceph cluster to use cephfs. When i create the a ceph fs pool i get the filesystem below error. # ceph osd pool create cephfs_data 128 pool 'cephfs_data' created # ceph osd pool create cephfs_metadata 128 pool 'cephfs_metadata' created # ceph fs new cephfs cephfs_metadata cephfs_data new fs with metadata pool 6 and data pool 5 # ceph -s cluster: id: 1c27def45-f0f9-494d-sfke-eb4323432fd health: HEALTH_ERR 1 filesystem is offline 1 filesystem is online with fewer MDS than max_mds services: mon: 2 daemons, quorum ceph-mon01,ceph-mon02 mgr: ceph-adm01(active) mds: cephfs-0/0/1 up osd: 12 osds: 12 up, 12 in data: pools: 2 pools, 256 pgs objects: 0 objects, 0 B usage: 12 GiB used, 588 GiB / 600 GiB avail pgs: 256 active+clean but when i check the max_mds for the ceph fs it says 1 # ceph fs get cephfs | grep max_mds max_mds 1 Let anyone know what am i missing here? Any inputs is much appreciated. Regards, Ram Ceph-explorer..

3 3

Re: NoSuchKey on key that is visible in s3 list/radosgw bk
by Eric Ivancich 03 Apr '24

03 Apr '24

I have some questions for those who’ve experienced this issue. 1. It seems like those reporting this issue are seeing it strictly after upgrading to Octopus. From what version did each of these sites upgrade to Octopus? From Nautilus? Mimic? Luminous? 2. Does anyone have any lifecycle rules on a bucket experiencing this issue? If so, please describe. 3. Is anyone making copies of the affected objects (to same or to a different bucket) prior to seeing the issue? And if they are making copies, does the destination bucket have lifecycle rules? And if they are making copies, are those copies ever being removed? 4. Is anyone experiencing this issue willing to run their RGWs with 'debug_ms=1'? That would allow us to see a request from an RGW to either remove a tail object or decrement its reference counter (and when its counter reaches 0 it will be deleted). Thanks, Eric > On Nov 12, 2020, at 4:54 PM, huxiaoyu(a)horebdata.cn wrote: > > Looks like this is a very dangerous bug for data safety. Hope the bug would be quickly identified and fixed. > > best regards, > > Samuel > > > > huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn> > > From: Janek Bevendorff > Date: 2020-11-12 18:17 > To: huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn>; EDH - Manuel Rios; Rafael Lopez > CC: Robin H. Johnson; ceph-users > Subject: Re: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > I have never seen this on Luminous. I recently upgraded to Octopus and the issue started occurring only few weeks later. > > On 12/11/2020 16:37, huxiaoyu(a)horebdata.cn wrote: > which Ceph versions are affected by this RGW bug/issues? Luminous, Mimic, Octupos, or the latest? > > any idea? > > samuel > > > > huxiaoyu(a)horebdata.cn > > From: EDH - Manuel Rios > Date: 2020-11-12 14:27 > To: Janek Bevendorff; Rafael Lopez > CC: Robin H. Johnson; ceph-users > Subject: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > This same error caused us to wipe a full cluster of 300TB... will be related to some rados index/database bug not to s3. > > As Janek exposed is a mayor issue, because the error silent happend and you can only detect it with S3, when you're going to delete/purge a S3 bucket. Dropping NoSuchKey. Error is not related to S3 logic .. > > Hope this time dev's can take enought time to find and resolve the issue. Error happens with low ec profiles, even with replica x3 in some cases. > > Regards > > > > -----Mensaje original----- > De: Janek Bevendorff <janek.bevendorff(a)uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de>> > Enviado el: jueves, 12 de noviembre de 2020 14:06 > Para: Rafael Lopez <rafael.lopez(a)monash.edu <mailto:rafael.lopez@monash.edu>> > CC: Robin H. Johnson <robbat2(a)gentoo.org <mailto:robbat2@gentoo.org>>; ceph-users <ceph-users(a)ceph.io <mailto:ceph-users@ceph.io>> > Asunto: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > > Here is a bug report concerning (probably) this exact issue: > https://tracker.ceph.com/issues/47866 <https://tracker.ceph.com/issues/47866> > > I left a comment describing the situation and my (limited) experiences with it. > > > On 11/11/2020 10:04, Janek Bevendorff wrote: >> >> Yeah, that seems to be it. There are 239 objects prefixed >> .8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh in my dump. However, there are none >> of the multiparts from the other file to be found and the head object >> is 0 bytes. >> >> I checked another multipart object with an end pointer of 11. >> Surprisingly, it had way more than 11 parts (39 to be precise) named >> .1, .1_1 .1_2, .1_3, etc. Not sure how Ceph identifies those, but I >> could find them in the dump at least. >> >> I have no idea why the objects disappeared. I ran a Spark job over all >> buckets, read 1 byte of every object and recorded errors. Of the 78 >> buckets, two are missing objects. One bucket is missing one object, >> the other 15. So, luckily, the incidence is still quite low, but the >> problem seems to be expanding slowly. >> >> >> On 10/11/2020 23:46, Rafael Lopez wrote: >>> Hi Janek, >>> >>> What you said sounds right - an S3 single part obj won't have an S3 >>> multipart string as part of the prefix. S3 multipart string looks >>> like "2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme". >>> >>> From memory, single part S3 objects that don't fit in a single rados >>> object are assigned a random prefix that has nothing to do with >>> the object name, and the rados tail/data objects (not the head >>> object) have that prefix. >>> As per your working example, the prefix for that would be >>> '.8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh'. So there would be (239) "shadow" >>> objects with names containing that prefix, and if you add up the >>> sizes it should be the size of your S3 object. >>> >>> You should look at working and non working examples of both single >>> and multipart S3 objects, as they are probably all a bit different >>> when you look in rados. >>> >>> I agree it is a serious issue, because once objects are no longer in >>> rados, they cannot be recovered. If it was a case that there was a >>> link broken or rados objects renamed, then we could work to >>> recover...but as far as I can tell, it looks like stuff is just >>> vanishing from rados. The only explanation I can think of is some >>> (rgw or rados) background process is incorrectly doing something with >>> these objects (eg. renaming/deleting). I had thought perhaps it was a >>> bug with the rgw garbage collector..but that is pure speculation. >>> >>> Once you can articulate the problem, I'd recommend logging a bug >>> tracker upstream. >>> >>> >>> On Wed, 11 Nov 2020 at 06:33, Janek Bevendorff >>> <janek.bevendorff(a)uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de> >>> <mailto:janek.bevendorff@uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de>>> wrote: >>> >>> Here's something else I noticed: when I stat objects that work >>> via radosgw-admin, the stat info contains a "begin_iter" JSON >>> object with RADOS key info like this >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110924034843-crawl420/WIDE-20110924065228-02544.warc.gz", >>> "instance": "", >>> "ns": "" >>> } >>> >>> >>> and then "end_iter" with key info like this: >>> >>> >>> "key": { >>> "name": >>> ".8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh_239", >>> "instance": "", >>> "ns": "shadow" >>> } >>> >>> However, when I check the broken 0-byte object, the "begin_iter" >>> and "end_iter" keys look like this: >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.1", >>> "instance": "", >>> "ns": "multipart" >>> } >>> >>> [...] >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.19", >>> "instance": "", >>> "ns": "multipart" >>> } >>> >>> So, it's the full name plus a suffix and the namespace is >>> multipart, not shadow (or empty). This in itself may just be an >>> artefact of whether the object was uploaded in one go or as a >>> multipart object, but the second difference is that I cannot find >>> any of the multipart objects in my pool's object name dump. I >>> can, however, find the shadow RADOS object of the intact S3 object. >>> >>> >>> >>> >>> -- >>> *Rafael Lopez* >>> Devops Systems Engineer >>> Monash University eResearch Centre >>> >>> T: +61 3 9905 9118 <tel:%2B61%203%209905%209118> >>> E: rafael.lopez(a)monash.edu <mailto:rafael.lopez@monash.edu> >>> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

6 22

Why you might want packages not containers for Ceph deployments
by Matthew Vernon 26 Mar '24

26 Mar '24

Hi, In the discussion after the Ceph Month talks yesterday, there was a bit of chat about cephadm / containers / packages. IIRC, Sage observed that a common reason in the recent user survey for not using cephadm was that it only worked on containerised deployments. I think he then went on to say that he hadn't heard any compelling reasons why not to use containers, and suggested that resistance was essentially a user education question[0]. I'd like to suggest, briefly, that: * containerised deployments are more complex to manage, and this is not simply a matter of familiarity * reducing the complexity of systems makes admins' lives easier * the trade-off of the pros and cons of containers vs packages is not obvious, and will depend on deployment needs * Ceph users will benefit from both approaches being supported into the future We make extensive use of containers at Sanger, particularly for scientific workflows, and also for bundling some web apps (e.g. Grafana). We've also looked at a number of container runtimes (Docker, singularity, charliecloud). They do have advantages - it's easy to distribute a complex userland in a way that will run on (almost) any target distribution; rapid "cloud" deployment; some separation (via namespaces) of network/users/processes. For what I think of as a 'boring' Ceph deploy (i.e. install on a set of dedicated hardware and then run for a long time), I'm not sure any of these benefits are particularly relevant and/or compelling - Ceph upstream produce Ubuntu .debs and Canonical (via their Ubuntu Cloud Archive) provide .debs of a couple of different Ceph releases per Ubuntu LTS - meaning we can easily separate out OS upgrade from Ceph upgrade. And upgrading the Ceph packages _doesn't_ restart the daemons[1], meaning that we maintain control over restart order during an upgrade. And while we might briefly install packages from a PPA or similar to test a bugfix, we roll those (test-)cluster-wide, rather than trying to run a mixed set of versions on a single cluster - and I understand this single-version approach is best practice. Deployment via containers does bring complexity; some examples we've found at Sanger (not all Ceph-related, which we run from packages): * you now have 2 process supervision points - dockerd and systemd * docker updates (via distribution unattended-upgrades) have an unfortunate habit of rudely restarting everything * docker squats on a chunk of RFC 1918 space (and telling it not to can be a bore), which coincides with our internal network... * there is more friction if you need to look inside containers (particularly if you have a lot running on a host and are trying to find out what's going on) * you typically need to be root to build docker containers (unlike packages) * we already have package deployment infrastructure (which we'll need regardless of deployment choice) We also currently use systemd overrides to tweak some of the Ceph units (e.g. to do some network sanity checks before bringing up an OSD), and have some tools to pair OSD / journal / LVM / disk device up; I think these would be more fiddly in a containerised deployment. I'd accept that fixing these might just be a SMOP[2] on our part. Now none of this is show-stopping, and I am most definitely not saying "don't ship containers". But I think there is added complexity to your deployment from going the containers route, and that is not simply a "learn how to use containers" learning curve. I do think it is reasonable for an admin to want to reduce the complexity of what they're dealing with - after all, much of my job is trying to automate or simplify the management of complex systems! I can see from a software maintainer's point of view that just building one container and shipping it everywhere is easier than building packages for a number of different distributions (one of my other hats is a Debian developer, and I have a bunch of machinery for doing this sort of thing). But it would be a bit unfortunate if the general thrust of "let's make Ceph easier to set up and manage" was somewhat derailed with "you must use containers, even if they make your life harder". I'm not going to criticise anyone who decides to use a container-based deployment (and I'm sure there are plenty of setups where it's an obvious win), but if I were advising someone who wanted to set up and use a 'boring' Ceph cluster for the medium term, I'd still advise on using packages. I don't think this makes me a luddite :) Regards, and apologies for the wall of text, Matthew [0] I think that's a fair summary! [1] This hasn't always been true... [2] Simple (sic.) Matter of Programming -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.

26 65

rgw multisite sync not syncing data, error: RGW-SYNC:data:init_data_sync_status: ERROR: failed to read remote data log shards
by Christian Rohmann 20 Jul '23

20 Jul '23

Hey ceph-users, I setup a multisite sync between two freshly setup Octopus clusters. In the first cluster I created a bucket with some data just to test the replication of actual data later. I then followed the instructions on https://docs.ceph.com/en/octopus/radosgw/multisite/#migrating-a-single-site… to add a second zone. Things went well and both zones are now happily reaching each other and the API endpoints are talking. Also the metadata is in sync already - both sides are happy and I can see bucket listings and users are "in sync": > # radosgw-admin sync status > realm 13d1b8cb-dc76-4aed-8578-2ce5d3d010e8 (obst) > zonegroup 17a06c15-2665-484e-8c61-cbbb806e11d2 (obst-fra) > zone 6d2c1275-527e-432f-a57a-9614930deb61 (obst-rgn) > metadata sync no sync (zone is master) > data sync source: c07447eb-f93a-4d8f-bf7a-e52fade399f3 (obst-az1) > init > full sync: 128/128 shards > full sync: 0 buckets to sync > incremental sync: 0/128 shards > data is behind on 128 shards > behind shards: [0...127] > and on the other side ... > # radosgw-admin sync status > realm 13d1b8cb-dc76-4aed-8578-2ce5d3d010e8 (obst) > zonegroup 17a06c15-2665-484e-8c61-cbbb806e11d2 (obst-fra) > zone c07447eb-f93a-4d8f-bf7a-e52fade399f3 (obst-az1) > metadata sync syncing > full sync: 0/64 shards > incremental sync: 64/64 shards > metadata is caught up with master > data sync source: 6d2c1275-527e-432f-a57a-9614930deb61 (obst-rgn) > init > full sync: 128/128 shards > full sync: 0 buckets to sync > incremental sync: 0/128 shards > data is behind on 128 shards > behind shards: [0...127] > also the newly created buckets (read: their metadata) is synced. What is apparently not working in the sync of actual data. Upon startup the radosgw on the second site shows: > 2021-06-25T16:15:06.445+0000 7fe71eff5700 1 RGW-SYNC:meta: start > 2021-06-25T16:15:06.445+0000 7fe71eff5700 1 RGW-SYNC:meta: realm > epoch=2 period id=f4553d7c-5cc5-4759-9253-9a22b051e736 > 2021-06-25T16:15:11.525+0000 7fe71dff3700 0 > RGW-SYNC:data:sync:init_data_sync_status: ERROR: failed to read remote > data log shards > also when issuing # radosgw-admin data sync init --source-zone obst-rgn it throws > 2021-06-25T16:20:29.167+0000 7f87c2aec080 0 > RGW-SYNC:data:init_data_sync_status: ERROR: failed to read remote data > log shards Does anybody have any hints on where to look for what could be broken here? Thanks a bunch, Regards Christian

3 4

Small RGW objects and RADOS 64KB minimun size
by Loïc Dachary 01 Jun '23

01 Jun '23

Bonjour, Reading Karan's blog post about benchmarking the insertion of billions objects to Ceph via S3 / RGW[0] from last year, it reads: > we decided to lower bluestore_min_alloc_size_hdd to 18KB and re-test. As represented in chart-5, the object creation rate found to be notably reduced after lowering the bluestore_min_alloc_size_hdd parameter from 64KB (default) to 18KB. As such, for objects larger than the bluestore_min_alloc_size_hdd , the default values seems to be optimal, smaller objects further require more investigation if you intended to reduce bluestore_min_alloc_size_hdd parameter. There also is a mail thread dated 2018 on this topic as well, with the same conclusion although using RADOS directly and not RGW[3]. I read the RGW data layout page in the documentation[1] and concluded that by default every object inserted with S3 / RGW will indeed use at least 64kb. A pull request from last year[2] seems to confirm it and also suggests modifying bluestore_min_alloc_size_hdd has adverse side effects. That being said, I'm curious to know if people developed strategies to cope with this overhead. Someone mentioned packing objects together client side to make them larger. But maybe there are simpler ways to do the same? Cheers [0] https://www.redhat.com/en/blog/scaling-ceph-billion-objects-and-beyond [1] https://docs.ceph.com/en/latest/radosgw/layout/ [2] https://github.com/ceph/ceph/pull/32809 [3] https://www.spinics.net/lists/ceph-users/msg45755.html -- Loïc Dachary, Artisan Logiciel Libre

4 6

MDS crash on FAILED ceph_assert(cur->is_auth())
by Peter van Heusden 04 May '23

04 May '23

I am running Ceph 15.2.13 on CentOS 7.9.2009 and recently my MDS servers have started failing with the error message In function 'void Server::handle_client_open(MDRequestRef&)' thread 7f0ca9908700 time 2021-06-28T09:21:11.484768+0200 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/15.2.13/rpm/el7/BUILD/ceph-15.2.13/src/mds/Server.cc: 4149: FAILED ceph_assert(cur->is_auth()) Complete log is: https://gist.github.com/pvanheus/4da555a6de6b5fa5e46cbf74f5500fbd ceph status output is: # ceph status cluster: id: ed7b2c16-b053-45e2-a1fe-bf3474f90508 health: HEALTH_WARN 30 OSD(s) experiencing BlueFS spillover insufficient standby MDS daemons available 1 MDSs report slow requests 2 mgr modules have failed dependencies 4347046/326505282 objects misplaced (1.331%) 6 nearfull osd(s) 23 pgs not deep-scrubbed in time 23 pgs not scrubbed in time 8 pool(s) nearfull services: mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 22m) mgr: ceph-mon1(active, since 11w), standbys: ceph-mon2, ceph-mon3 mds: SANBI_FS:2 {0=ceph-mon1=up:active(laggy or crashed),1=ceph-mon2=up:stopping} osd: 54 osds: 54 up (since 2w), 54 in (since 11w); 50 remapped pgs data: pools: 8 pools, 833 pgs objects: 42.37M objects, 89 TiB usage: 159 TiB used, 105 TiB / 264 TiB avail pgs: 4347046/326505282 objects misplaced (1.331%) 782 active+clean 49 active+clean+remapped 1 active+clean+scrubbing+deep 1 active+clean+remapped+scrubbing io: client: 29 KiB/s rd, 427 KiB/s wr, 37 op/s rd, 48 op/s wr When restarting a MDS it goes through states replace, reconnect, resolve and finally sets itself to active before this crash happens. Any advice on what to do? Thanks, Peter P.S. apologies if you received this email more than once - I have had some trouble figuring out the correct mailing list to use.

3 5

cephadm cluster move /var/lib/docker to separate device fails
by Karsten Nielsen 26 Mar '23

26 Mar '23

Hi, I have setup a ceph cluster with cephadm with docker backend. I want to move /var/lib/docker to a separate device to get better performance and less load on the OS device. I tried that by stopping docker copy the content of /var/lib/docker to the new device and mount the new device to /var/lib/docker. The other containers started as expected and continues to work and run as expected. But the ceph containers seems to be broken. I am not able to get them back in working state. I have tried to remove the host with `ceph orch host rm itcnchn-bb4067` and readd it but no effect. The strange thing is that 2 of 4 containers comes up as expected. ceph orch ps itcnchn-bb4067 NAME HOST STATUS REFRESHED AGE VERSION IMAGE NAME IMAGE ID CONTAINER ID crash.itcnchn-bb4067 itcnchn-bb4067 running (18h) 10m ago 4w 15.2.7 docker.io/ceph/ceph:v15 2bc420ddb175 2af28c4571cf mds.cephfs.itcnchn-bb4067.qzoshl itcnchn-bb4067 error 10m ago 4w <unknown> docker.io/ceph/ceph:v15 <unknown> <unknown> mon.itcnchn-bb4067 itcnchn-bb4067 error 10m ago 18h <unknown> docker.io/ceph/ceph:v15 <unknown> <unknown> rgw.ikea.dc9-1.itcnchn-bb4067.gtqedc itcnchn-bb4067 running (18h) 10m ago 4w 15.2.7 docker.io/ceph/ceph:v15 2bc420ddb175 00d000aec32b Docker logs from the active manager does not say much about what is wrong debug 2021-01-05T09:57:52.537+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring mds.cephfs.itcnchn-bb4067.qzoshl (unknown last config time)... debug 2021-01-05T09:57:52.541+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring daemon mds.cephfs.itcnchn-bb4067.qzoshl on itcnchn-bb4067 debug 2021-01-05T09:57:52.973+0000 7fdb64e88700 0 log_channel(cluster) log [DBG] : pgmap v347: 241 pgs: 241 active+clean; 18 GiB data, 50 GiB used, 52 TiB / 52 TiB avail; 18 KiB/s rd, 78 KiB/s wr, 24 op/s debug 2021-01-05T09:57:53.085+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring mon.itcnchn-bb4067 (unknown last config time)... debug 2021-01-05T09:57:53.085+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring daemon mon.itcnchn-bb4067 on itcnchn-bb4067 debug 2021-01-05T09:57:53.625+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring rgw.ikea.dc9-1.itcnchn-bb4067.gtqedc (unknown last config time)... debug 2021-01-05T09:57:53.629+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring daemon rgw.ikea.dc9-1.itcnchn-bb4067.gtqedc on itcnchn-bb4067 debug 2021-01-05T09:57:54.141+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring crash.itcnchn-bb4067 (unknown last config time)... debug 2021-01-05T09:57:54.141+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring daemon crash.itcnchn-bb4067 on itcnchn-bb4067 - Karsten

2 1

Stuck OSD service specification - can't remove
by David Orman 15 Mar '23

15 Mar '23

Has anybody run into a 'stuck' OSD service specification? I've tried to delete it, but it's stuck in 'deleting' state, and has been for quite some time (even prior to upgrade, on 15.2.x). This is on 16.2.3: NAME PORTS RUNNING REFRESHED AGE PLACEMENT osd.osd_spec 504/525 <deleting> 12m label:osd root@ceph01:/# ceph orch rm osd.osd_spec Removed service osd.osd_spec From active monitor: debug 2021-05-06T23:14:48.909+0000 7f17d310b700 0 log_channel(cephadm) log [INF] : Remove service osd.osd_spec Yet in ls, it's still there, same as above. --export on it: root@ceph01:/# ceph orch ls osd.osd_spec --export service_type: osd service_id: osd_spec service_name: osd.osd_spec placement: {} unmanaged: true spec: filter_logic: AND objectstore: bluestore We've tried --force, as well, with no luck. To be clear, the --export even prior to delete looks nothing like the actual service specification we're using, even after I re-apply it, so something seems 'bugged'. Here's the OSD specification we're applying: service_type: osd service_id: osd_spec placement: label: "osd" data_devices: rotational: 1 db_devices: rotational: 0 db_slots: 12 I would appreciate any insight into how to clear this up (without removing the actual OSDs, we're just wanting to apply the updated service specification - we used to use host placement rules and are switching to label-based). Thanks, David

3 4

kernel client osdc ops stuck and mds slow reqs
by Dan van der Ster 23 Feb '23

23 Feb '23

Hi all, We are quite regularly (a couple times per week) seeing: HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests MDS_CLIENT_LATE_RELEASE 1 clients failing to respond to capability release mdshpc-be143(mds.0): Client hpc-be028.cern.ch: failing to respond to capability release client_id: 52919162 MDS_SLOW_REQUEST 1 MDSs report slow requests mdshpc-be143(mds.0): 1 slow requests are blocked > 30 secs Which is being caused by osdc ops stuck in a kernel client, e.g.: 10:57:18 root hpc-be028 /root → cat /sys/kernel/debug/ceph/4da6fd06-b069-49af-901f-c9513baabdbd.client52919162/osdc REQUESTS 9 homeless 0 46559317 osd243 3.ee6ffcdb 3.cdb [243,501,92]/243 [243,501,92]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a01.00000057 0x400014 1 read 46559322 osd243 3.ee6ffcdb 3.cdb [243,501,92]/243 [243,501,92]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a01.00000057 0x400014 1 read 46559323 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559341 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559342 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559345 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559621 osd243 3.6313e8ef 3.8ef [243,330,521]/243 [243,330,521]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a45.0000007a 0x400014 1 read 46559629 osd243 3.b280c852 3.852 [243,113,539]/243 [243,113,539]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a3a.0000007f 0x400014 1 read 46559928 osd243 3.1ee7bab4 3.ab4 [243,332,94]/243 [243,332,94]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f099ff.0000073f 0x400024 1 write LINGER REQUESTS BACKOFFS We can unblock those requests by doing `ceph osd down osd.243` (or restarting osd.243). This is ceph v14.2.6 and the client kernel is el7 3.10.0-957.27.2.el7.x86_64. Are there a better way to debug this? Best Regards, Dan

4 12

OSD slow ops warning not clearing after OSD down
by Frank Schilder 17 Jan '23

17 Jan '23

Dear cephers, I have a strange problem. An OSD went down and recovery finished. For some reason, I have a slow ops warning for the failed OSD stuck in the system: health: HEALTH_WARN 430 slow ops, oldest one blocked for 36 sec, osd.580 has slow ops The OSD is auto-out: | 580 | ceph-22 | 0 | 0 | 0 | 0 | 0 | 0 | autoout,exists | It is probably a warning dating back to just before the fail. How can I clear it? Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

4 7

[ERR] OSD_SCRUB_ERRORS: 2 scrub errors
by Szabo, Istvan (Agoda) 11 Jan '23

11 Jan '23

Hi, I’m continuously getting scrub errors in my index pool and log pool that I need to repair always. HEALTH_ERR 2 scrub errors; Possible data damage: 1 pg inconsistent [ERR] OSD_SCRUB_ERRORS: 2 scrub errors [ERR] PG_DAMAGED: Possible data damage: 1 pg inconsistent pg 20.19 is active+clean+inconsistent, acting [39,41,37] Why is this? I have no cue at all, no log entry no anything ☹ ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

6 9

v16.2.5 Pacific released
by David Galloway 09 Jul '21

09 Jul '21

We're happy to announce the 5th backport release in the Pacific series. We recommend users to update to this release. For a detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/en/news/blog/2021/v16-2-5-pacific-released Notable Changes --------------- * `ceph-mgr-modules-core` debian package does not recommend `ceph-mgr-rook` anymore. As the latter depends on `python3-numpy` which cannot be imported in different Python sub-interpreters multi-times if the version of `python3-numpy` is older than 1.19. Since `apt-get` installs the `Recommends` packages by default, `ceph-mgr-rook` was always installed along with `ceph-mgr` debian package as an indirect dependency. If your workflow depends on this behavior, you might want to install `ceph-mgr-rook` separately. * mgr/nfs: `nfs` module is moved out of volumes plugin. Prior using the `ceph nfs` commands, `nfs` mgr module must be enabled. * volumes/nfs: The `cephfs` cluster type has been removed from the `nfs cluster create` subcommand. Clusters deployed by cephadm can support an NFS export of both `rgw` and `cephfs` from a single NFS cluster instance. * The `nfs cluster update` command has been removed. You can modify the placement of an existing NFS service (and/or its associated ingress service) using `orch ls --export` and `orch apply -i ...`. * The `orch apply nfs` command no longer requires a pool or namespace argument. We strongly encourage users to use the defaults so that the `nfs cluster ls` and related commands will work properly. * The `nfs cluster delete` and `nfs export delete` commands are deprecated and will be removed in a future release. Please use `nfs cluster rm` and `nfs export rm` instead. * A long-standing bug that prevented 32-bit and 64-bit client/server interoperability under msgr v2 has been fixed. In particular, mixing armv7l (armhf) and x86_64 or aarch64 servers in the same cluster now works. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-16.2.5.tar.gz * For packages, see https://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: 0883bdea7337b95e4b611c768c0279868462204a

3 2

Cephfs slow, not busy, but doing high traffic in the metadata pool
by Flemming Frandsen 08 Jul '21

08 Jul '21

We have a nautilus cluster where any metadata write operation is very slow. We're seeing very light load from clients, as reported by dumping ops in flight, often it's zero. We're also seeing about 100 MB/s writes to the metadata pool, constantly, for weeks on end, which seems excessive, as only 22GB is utilized. Should the writes to the metadata pool not quiet down when there's nothing going on? Is there any way i can get information about why the MDSes are thrashing so badly?

2 3

RocksDB resharding does not work
by Robert Sander 08 Jul '21

08 Jul '21

Hi, I am trying to apply the resharding to a containerized OSD (16.2.4) as described here: https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/#r… # ceph osd set noout # ceph orch daemon stop osd.13 # cephadm shell --name osd.13 # ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-13 --sharding="m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L P" reshard 2021-07-08T15:12:03.392+0000 7f2e2173b3c0 -1 rocksdb: prepare_for_reshard failure parsing column options: block_cache={type=binned_lru} error resharding: (22) Invalid argument # exit # ceph orch daemon start osd.13 # ceph osd unset noout The OSD cannot start any more and has these errors in its log: Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.293+0000 7efc32db4080 0 osd.13:7.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196) Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.293+0000 7efc32db4080 1 bluestore(/var/lib/ceph/osd/ceph-13) _mount path /var/lib/ceph/osd/ceph-13 Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.293+0000 7efc32db4080 0 bluestore(/var/lib/ceph/osd/ceph-13) _open_db_and_around read-only:0 repair:0 Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.293+0000 7efc32db4080 1 bdev(0x564271210800 /var/lib/ceph/osd/ceph-13/block) open path /var/lib/ceph/osd/ceph-13/block Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.293+0000 7efc32db4080 1 bdev(0x564271210800 /var/lib/ceph/osd/ceph-13/block) open size 107369988096 (0x18ffc00000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.293+0000 7efc32db4080 1 bluestore(/var/lib/ceph/osd/ceph-13) _set_cache_sizes cache_size 3221225472 meta 0.45 kv 0.45 data 0.06 Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.293+0000 7efc32db4080 1 bdev(0x564271210c00 /var/lib/ceph/osd/ceph-13/block) open path /var/lib/ceph/osd/ceph-13/block Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.293+0000 7efc32db4080 1 bdev(0x564271210c00 /var/lib/ceph/osd/ceph-13/block) open size 107369988096 (0x18ffc00000, 100 GiB) block_size 4096 (4 KiB) non-rotational discard supported Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.293+0000 7efc32db4080 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-13/block size 100 GiB Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.293+0000 7efc32db4080 1 bluefs mount Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.293+0000 7efc32db4080 1 bluefs _init_alloc shared, id 1, capacity 0x18ffc00000, block size 0x10000 Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.325+0000 7efc32db4080 1 bluefs mount shared_bdev_used = 0 Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.325+0000 7efc32db4080 -1 rocksdb: verify_sharding extra columns in rocksdb. rocksdb columns = [default,m-0,m-1,m-2,p-0,p-1,p-2] target columns = [reshardingXcommencingXlocked] Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.325+0000 7efc32db4080 -1 bluestore(/var/lib/ceph/osd/ceph-13) _open_db erroring opening db: Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.325+0000 7efc32db4080 1 bluefs umount Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.325+0000 7efc32db4080 1 bdev(0x564271210c00 /var/lib/ceph/osd/ceph-13/block) close Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.577+0000 7efc32db4080 1 bdev(0x564271210800 /var/lib/ceph/osd/ceph-13/block) close Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.825+0000 7efc32db4080 -1 osd.13 0 OSD:init: unable to mount object store Jul 08 17:13:46 cephtest24 bash[4161252]: debug 2021-07-08T15:13:46.825+0000 7efc32db4080 -1 ** ERROR: osd init failed: (5) Input/output error How do I correct the issue? Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin http://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Zwangsangaben lt. §35a GmbHG: HRB 93818 B / Amtsgericht Berlin-Charlottenburg, Geschäftsführer: Peer Heinlein -- Sitz: Berlin

1 0

NVME hosts added to the clusters and it made old ssd hosts flapping osds
by Szabo, Istvan (Agoda) 08 Jul '21

08 Jul '21

Hi, I've added 4 nvme hosts with 2osd/nvme to my cluster and it made al the ssd osds flapping I don't understand why. It is under the same root but 2 different device classes, nvme and ssd. The pools are on the ssd on the nvme nothing at the moment. The only way to bring back the ssd osds alive to shutdown the nvmes. The new nvme servers have 25GB nics the old servers and the mons have 10GB but in aggregated mode. This is the crush rule dump: [ { "rule_id": 0, "rule_name": "replicated_ssd", "ruleset": 0, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -21, "item_name": "default~ssd" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 1, "rule_name": "replicated_nvme", "ruleset": 1, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -10, "item_name": "default~nvme" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] } ] This is the osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -19 561.15057 root default -1 38.03099 host server-2001 0 ssd 2.00000 osd.0 up 1.00000 1.00000 10 ssd 6.98499 osd.10 up 1.00000 1.00000 11 ssd 6.98599 osd.11 up 1.00000 1.00000 12 ssd 2.29799 osd.12 up 1.00000 1.00000 13 ssd 2.29799 osd.13 up 1.00000 1.00000 14 ssd 3.49300 osd.14 up 1.00000 1.00000 41 ssd 6.98499 osd.41 up 1.00000 1.00000 42 ssd 6.98599 osd.42 up 1.00000 1.00000 -3 38.03099 host server-2002 1 ssd 2.00000 osd.1 up 1.00000 1.00000 24 ssd 6.98499 osd.24 up 1.00000 1.00000 25 ssd 6.98599 osd.25 up 1.00000 1.00000 27 ssd 2.29799 osd.27 up 1.00000 1.00000 28 ssd 2.29799 osd.28 up 1.00000 1.00000 29 ssd 3.49300 osd.29 up 1.00000 1.00000 43 ssd 6.98499 osd.43 up 1.00000 1.00000 44 ssd 6.98599 osd.44 up 1.00000 1.00000 -6 38.03000 host server-2003 2 ssd 2.00000 osd.2 up 1.00000 1.00000 26 ssd 6.98499 osd.26 up 1.00000 1.00000 38 ssd 2.29999 osd.38 up 1.00000 1.00000 39 ssd 2.29500 osd.39 up 1.00000 1.00000 40 ssd 3.49300 osd.40 up 1.00000 1.00000 45 ssd 6.98499 osd.45 up 1.00000 1.00000 46 ssd 6.98599 osd.46 up 1.00000 1.00000 47 ssd 6.98599 osd.47 up 1.00000 1.00000 -17 111.76465 host server-2004 5 nvme 6.98529 osd.5 down 0 1.00000 9 nvme 6.98529 osd.9 down 0 1.00000 18 nvme 6.98529 osd.18 down 0 1.00000 22 nvme 6.98529 osd.22 down 0 1.00000 32 nvme 6.98529 osd.32 down 0 1.00000 36 nvme 6.98529 osd.36 down 0 1.00000 50 nvme 6.98529 osd.50 down 0 1.00000 54 nvme 6.98529 osd.54 down 0 1.00000 58 nvme 6.98529 osd.58 down 0 1.00000 62 nvme 6.98529 osd.62 down 0 1.00000 66 nvme 6.98529 osd.66 down 0 1.00000 70 nvme 6.98529 osd.70 down 0 1.00000 74 nvme 6.98529 osd.74 down 0 1.00000 78 nvme 6.98529 osd.78 down 0 1.00000 82 nvme 6.98529 osd.82 down 0 1.00000 86 nvme 6.98529 osd.86 down 0 1.00000 -14 111.76465 host server-2005 4 nvme 6.98529 osd.4 down 0 1.00000 8 nvme 6.98529 osd.8 down 0 1.00000 17 nvme 6.98529 osd.17 down 0 1.00000 21 nvme 6.98529 osd.21 down 0 1.00000 31 nvme 6.98529 osd.31 down 0 1.00000 35 nvme 6.98529 osd.35 down 0 1.00000 49 nvme 6.98529 osd.49 down 0 1.00000 53 nvme 6.98529 osd.53 down 0 1.00000 57 nvme 6.98529 osd.57 down 0 1.00000 61 nvme 6.98529 osd.61 down 0 1.00000 65 nvme 6.98529 osd.65 down 0 1.00000 69 nvme 6.98529 osd.69 down 0 1.00000 73 nvme 6.98529 osd.73 down 0 1.00000 77 nvme 6.98529 osd.77 down 0 1.00000 81 nvme 6.98529 osd.81 down 0 1.00000 85 nvme 6.98529 osd.85 down 0 1.00000 -22 111.76465 host server-2006 6 nvme 6.98529 osd.6 down 0 1.00000 15 nvme 6.98529 osd.15 down 0 1.00000 19 nvme 6.98529 osd.19 down 0 1.00000 23 nvme 6.98529 osd.23 down 0 1.00000 33 nvme 6.98529 osd.33 down 0 1.00000 37 nvme 6.98529 osd.37 down 0 1.00000 51 nvme 6.98529 osd.51 down 0 1.00000 55 nvme 6.98529 osd.55 down 0 1.00000 59 nvme 6.98529 osd.59 down 0 1.00000 63 nvme 6.98529 osd.63 up 0 1.00000 67 nvme 6.98529 osd.67 down 0 1.00000 71 nvme 6.98529 osd.71 up 0 1.00000 75 nvme 6.98529 osd.75 down 0 1.00000 79 nvme 6.98529 osd.79 down 0 1.00000 83 nvme 6.98529 osd.83 down 0 1.00000 87 nvme 6.98529 osd.87 down 0 1.00000 -11 111.76465 host server-2007 3 nvme 6.98529 osd.3 down 0 1.00000 7 nvme 6.98529 osd.7 down 0 1.00000 16 nvme 6.98529 osd.16 down 0 1.00000 20 nvme 6.98529 osd.20 down 0 1.00000 30 nvme 6.98529 osd.30 down 0 1.00000 34 nvme 6.98529 osd.34 down 0 1.00000 48 nvme 6.98529 osd.48 down 0 1.00000 52 nvme 6.98529 osd.52 down 0 1.00000 56 nvme 6.98529 osd.56 down 0 1.00000 60 nvme 6.98529 osd.60 down 0 1.00000 64 nvme 6.98529 osd.64 down 0 1.00000 68 nvme 6.98529 osd.68 down 0 1.00000 72 nvme 6.98529 osd.72 down 0 1.00000 76 nvme 6.98529 osd.76 down 0 1.00000 80 nvme 6.98529 osd.80 down 0 1.00000 84 nvme 6.98529 osd.84 down 0 1.00000 Pool info: pool 21 'dbs-realtime-staging-client' replicated size 3 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 27611 lfor 0/27512/27510 flags hashpspool,selfmanaged_snaps max_bytes 9999757606912 stripe_width 0 application rbd pool 24 'dbs-realtime-staging-w-financedb' replicated size 3 min_size 1 crush_rule 0 object_hash rjenkins pg_num 256 pgp_num 256 autoscale_mode on last_change 27613 flags hashpspool,selfmanaged_snaps max_bytes 19999515213824 stripe_width 0 application rbd pool 25 'dbs-realtime-staging-w-dstest' replicated size 3 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 27813 lfor 0/0/23856 flags hashpspool,selfmanaged_snaps max_bytes 99857989632 stripe_width 0 application rbd Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo(a)agoda.com<mailto:istvan.szabo@agoda.com> --------------------------------------------------- ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 1

Re: Fwd: ceph upgrade from luminous to nautils
by M Ranga Swami Reddy 08 Jul '21

08 Jul '21

Thanks Marc. Means, we can upgrade from Luminous to Nautils and later can upgrade the OSDs from ceph-disk to ceph-volmue.. On Thu, Jul 8, 2021 at 5:45 PM Marc <Marc(a)f1-outsourcing.eu> wrote: > I did the same upgrade from Luminous to Nautilus, and still have osd's > created with ceph-disk. I am slowly migrating to lvm and encryption. > > However I did have some issues with osd's not starting, you have to check > run levels and make sure the symlinks are still correct. I also had > something that I had to change ownership /dev/sdX before it would start. > > I still have this in my rc.local > > chown ceph.ceph /dev/sdb2 > chown ceph.ceph /dev/sdc2 > chown ceph.ceph /dev/sdd2 > chown ceph.ceph /dev/sde2 > chown ceph.ceph /dev/sdf2 > chown ceph.ceph /dev/sdg2 > chown ceph.ceph /dev/sdh2 > chown ceph.ceph /dev/sdi2 > chown ceph.ceph /dev/sdj2 > chown ceph.ceph /dev/sdk2 > > > > -----Original Message----- > > From: M Ranga Swami Reddy <swamireddy(a)gmail.com> > > Sent: Thursday, 8 July 2021 11:49 > > To: ceph-devel <ceph-devel(a)vger.kernel.org>; ceph-users <ceph- > > users(a)ceph.com> > > Subject: [ceph-users] Fwd: ceph upgrade from luminous to nautils > > > > ---------- Forwarded message --------- > > From: M Ranga Swami Reddy <swamireddy(a)gmail.com> > > Date: Thu, Jul 8, 2021 at 2:30 PM > > Subject: ceph upgrade from luminous to nautils > > To: ceph-devel <ceph-devel(a)vger.kernel.org> > > > > > > Dear All, > > I am using the Ceph with Luminous version with 2000+ OSDs. > > Planning to upgrade the ceph from Luminous to Nautils. > > Currently, all OSDs deployed via ceph-disk. > > Can I proceed with this upgrade? > > is the ceph-disk OSDs will work with ceph-volumes (as ceph-disk > > deprecated > > in memic release) > > > > Please advise. > > > > Thanks > > Swami > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io >

1 0

Wrong hostnames in "ceph mgr services" (Octopus)
by Sebastian Knust 08 Jul '21

08 Jul '21

Hi, After upgrading from 15.2.8. to 15.2.13 with cephadm on CentOS 8 (containerised installation done by cephadm), Grafana no longer shows new data. Additionally, when accessing the Dashboard-URL on a host currently not hosting the dashboard, I am redirected to a wrong hostname (as shown in ceph mgr services). I assume that this is caused by the same reason which leads to this output of `ceph mgr services`: { "dashboard": "https://ceph-<cluster-id>-mgr.iceph-11.tsmsqs:8443/", "prometheus": "http://ceph-<cluster-id>-mgr.iceph-11.tsmsqs:9283/" } The correct hostname is iceph-11 (without the tsmsqs part), FQDN is iceph-11.servernet. The hosts use DNS, the names (iceph-11 and iceph-11.servernet) are resolvable both from the hosts as well as from within the Podman containers. I have determined that podman by default sets the container name as a hostname alias (visible with `hostname -a` within the container), which somehow leads to Ceph mgr picking it up as the primary name? My workaround is to modify /var/lib/ceph/<cluster-id>/mgr.<hostname>.<random-6-char-string>/unit.run, adding --no-hosts as an additional argument to the "podman run" command. I could probably use a system-wide containers.conf as well. With this workaround and after restarting the Ceph mgr container (via systemctl) and then restarting Prometheus and Grafana (with ceph orch redeploy), I once again get data in Grafana and the correct redirect for the dashboard. `ceph mgr services` also shows expected and correct values. I am wondering if this kind of issue is known or whether there is something wrong with my setup. I expected Ceph mgr to use the primary hostname and not some seemingly random hostname alias. Maybe this issue can also be discussed in a troubleshooting section of the monitoring stack documentation. Cheers Sebastian

1 0

Stuck MDSs behind in trimming
by Zachary Ulissi 08 Jul '21

08 Jul '21

We're running a rook-ceph cluster that has gotten stuck in "1 MDSs behind on trimming". * 1 filesystem, three active MDS servers each with standby * Quite a few files (20M objects), daily snapshots. This might be a problem? * Ceph pacific 16.2.4 * `ceph health detail` doesn't provide much help (see below) * num_segments is very slowly increasing over time * Restarting all of the MDSs returns to the same point. * moderate CPU usage for each MDS server (~30% for the stuck one, ~80% of a core for the others) * logs for the stuck MDS looks clean, it hits rejoin_joint_start then standard 'updating MDS map to version XXX" messages * `ceph daemon mds.x ops` shows no active ops on each of the MDS servers * `mds_log_max_segments` is set to 128, setting to a higher number causes the warning to go away, but the filesystem remains degraded, and setting it back to 128 shows num_segments has not changed. * I've tried playing around with other MDS settings based on various posts on this list and elsewhere, to no avail * `cephfs-journal-tool journal inspect` for each rank says journal integrity is fine. Something similar happened last week and (probably by accident by removing/adding nodes?) I got the MDSs to start recovering and the filesystem went back to healthy. I'm at a bit of a loss for what else to try. Thanks! Zack `ceph health detail` HEALTH_WARN mons are allowing insecure global_id reclaim; 1 filesystem is degraded; 1 MDSs behind on trimming; mon x is low on available space [WRN] AUTH_INSECURE_GLOBAL_ID_RECLAIM_ALLOWED: mons are allowing insecure global_id reclaim mon.x has auth_allow_insecure_global_id_reclaim set to true mon.ad has auth_allow_insecure_global_id_reclaim set to true mon.af has auth_allow_insecure_global_id_reclaim set to true [WRN] FS_DEGRADED: 1 filesystem is degraded fs myfs is degraded [WRN] MDS_TRIM: 1 MDSs behind on trimming mds.myfs-d(mds.2): Behind on trimming (340/128) max_segments: 128, num_segments: 340 [WRN] MON_DISK_LOW: mon x is low on available space mon.x has 22% avail `ceph config get mds` WHO MASK LEVEL OPTION VALUE RO global basic log_file * global basic log_to_file false mds basic mds_cache_memory_limit 17179869184 mds advanced mds_cache_trim_decay_rate 1.000000 mds advanced mds_cache_trim_threshold 1048576 mds advanced mds_log_max_segments 128 mds advanced mds_recall_max_caps 5000 mds advanced mds_recall_max_decay_rate 2.500000 global advanced mon_allow_pool_delete true global advanced mon_allow_pool_size_one true global advanced mon_cluster_log_file global advanced mon_pg_warn_min_per_osd 0 global advanced osd_pool_default_pg_autoscale_mode on global advanced osd_scrub_auto_repair true global advanced rbd_default_features 3

1 1

RocksDB degradation / manual compaction vs. snaptrim operations choking Ceph to a halt
by Christian Rohmann 08 Jul '21

08 Jul '21

Hello ceph-users, after an upgrade from Ceph Nautilus to Octopus we ran into extreme performance issues leading to an unusable cluster when doing a larger snapshot delete and the cluster doing snaptrims, see i.e. https://tracker.ceph.com/issues/50511#note-13. Since this was not an issue prior to the upgrade, maybe the conversion of the OSD to OMAP caused this degradation of the RocksDB data structures, maybe not. (We were running bluefs_buffered_io=true, so that was NOT the issue here). But I've noticed there are a few reports of such issues which boil down to RocksDB being in a somewhat degraded state and running a simple compact fixed those issues, see: * https://tracker.ceph.com/issues/50511 * https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/XSEBOIT43TG… * https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/BTWAQIEXBBE… * https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/Z4ADQFTGC5H… * Maybe also: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/N74C4U2POOS… I know improvements in this regard are actively worked on for pg removal, i.e. * https://tracker.ceph.com/issues/47174 ** https://github.com/ceph/ceph/pull/37314 ** https://github.com/ceph/ceph/pull/37496 but am wondering if this will help with snaptrims as well? In any case I was just wondering if any of you also experienced this condition with RocksDB and am wondering what you do to monitor or to actively mitigate this prior to having flapping OSDs and queuing up (snaptrim) operations? With Ceph Pacific it's possible to enable offline compaction on every start of an OSD (osd_compact_on_start), but is this really sufficient then? Regards Christian

3 4

RGW Dedicated clusters vs Shared (RBD, RGW) clusters
by gustavo panizzo 08 Jul '21

08 Jul '21

Hello I have some experience with RBD clusters (for use with KVM/libvirt) but now I'm building my first cluster to use with RGW. The RGW cluster size will be around 70T RAW, current RBD cluster(s) are in similar (or smaller) size. I'll be deploying Octopus Since most of the tunning is pretty different (big number of PGs, bluestore_compression_*, bluestore_min_alloc_size_*) I wonder if makes sense to run both workloads in the same cluster or if it would be better to have dedicated clusters. However bigger clusters (AFAIK) are more stable, what is other people doing? Single cluster for all workloads or a cluster per workload? thanks! PS: Asking for the future, what about CepFS? should it share cluster? -- IRC: gfa GPG: 0x27263FA42553615F904A7EBE2A40A2ECB8DAD8D5 OLD GPG: 0x44BB1BA79F6C6333

1 0

Fwd: ceph upgrade from luminous to nautils
by M Ranga Swami Reddy 08 Jul '21

08 Jul '21

---------- Forwarded message --------- From: M Ranga Swami Reddy <swamireddy(a)gmail.com> Date: Thu, Jul 8, 2021 at 2:30 PM Subject: ceph upgrade from luminous to nautils To: ceph-devel <ceph-devel(a)vger.kernel.org> Dear All, I am using the Ceph with Luminous version with 2000+ OSDs. Planning to upgrade the ceph from Luminous to Nautils. Currently, all OSDs deployed via ceph-disk. Can I proceed with this upgrade? is the ceph-disk OSDs will work with ceph-volumes (as ceph-disk deprecated in memic release) Please advise. Thanks Swami

1 0

Upgrade tips from Luminous to Nautilus?
by Mark Schouten 08 Jul '21

08 Jul '21

Hi, We've done our fair share of Ceph cluster upgrades since Hammer, and have not seen much problems with them. I'm now at the point that I have to upgrade a rather large cluster running Luminous and I would like to hear from other users if they have experiences with issues I can expect so that I can anticipate on them beforehand. As said, the cluster is running Luminous (12.2.13) and has the following services active: services: mon: 3 daemons, quorum osdnode01,osdnode02,osdnode04 mgr: osdnode01(active), standbys: osdnode02, osdnode03 mds: pmrb-3/3/3 up {0=osdnode06=up:active,1=osdnode08=up:active,2=osdnode07=up:active}, 1 up:standby osd: 116 osds: 116 up, 116 in; rgw: 3 daemons active Of the OSD's, we have 11 SSD's and 105 HDD. The capacity of the cluster is 1.01PiB. We have 2 active crush-rules on 18 pools. All pools have a size of 3 there is a total of 5760 pgs. { "rule_id": 1, "rule_name": "hdd-data", "ruleset": 1, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -10, "item_name": "default~hdd" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 2, "rule_name": "ssd-data", "ruleset": 2, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -21, "item_name": "default~ssd" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] } rbd -> crush_rule: hdd-data .rgw.root -> crush_rule: hdd-data default.rgw.control -> crush_rule: hdd-data default.rgw.data.root -> crush_rule: ssd-data default.rgw.gc -> crush_rule: ssd-data default.rgw.log -> crush_rule: ssd-data default.rgw.users.uid -> crush_rule: hdd-data default.rgw.usage -> crush_rule: ssd-data default.rgw.users.email -> crush_rule: hdd-data default.rgw.users.keys -> crush_rule: hdd-data default.rgw.meta -> crush_rule: hdd-data default.rgw.buckets.index -> crush_rule: ssd-data default.rgw.buckets.data -> crush_rule: hdd-data default.rgw.users.swift -> crush_rule: hdd-data default.rgw.buckets.non-ec -> crush_rule: ssd-data DB0475 -> crush_rule: hdd-data cephfs_pmrb_data -> crush_rule: hdd-data cephfs_pmrb_metadata -> crush_rule: ssd-data All but four clients are running Luminous, the four are running Jewel (that needs upgrading before proceeding with this upgrade). So, normally, I would 'just' upgrade all Ceph packages on the monitor-nodes and restart mons and then mgrs. After that, I would upgrade all Ceph packages on the OSD nodes and restart all the OSD's. Then, after that, the MDSes and RGWs. Restarting the OSD's will probably take a while. If anyone has a hint on what I should expect to cause some extra load or waiting time, that would be great. Obviously, we have read https://ceph.com/releases/v14-2-0-nautilus-released/ , but I'm looking for real world experiences. Thanks! -- Mark Schouten | Tuxis B.V. KvK: 74698818 | http://www.tuxis.nl/ T: +31 318 200208 | info(a)tuxis.nl

4 11

list-type=2 requests
by Szabo, Istvan (Agoda) 08 Jul '21

08 Jul '21

Hi, Is there anybody know about list-type=2 request? GET /bucket?list-type=2&max-keys=2 We faced yesterday the 2nd big objectstore cluster outage due to this request. 1 user made the cluster down totally. The normal ceph iostat read operation is below 30k, when they deployed their release it jumped up to 350k which made rados-gateway died under the haproxy. Why ceph is so sensitive for this or what is this actually? I don't even found anything in google. Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo(a)agoda.com<mailto:istvan.szabo@agoda.com> --------------------------------------------------- ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 1

pgcalc tool removed (or moved?) from ceph.com ?
by Dominik Csapak 08 Jul '21

08 Jul '21

Hi, just wanted to ask if it is intentional that http://ceph.com/pgcalc/ results in a 404 error? is there any alternative url? it is still linked from the offical docs. with kind regards Dominik

2 4

Why does 'mute AUTH_INSECURE_GLOBAL_ID_RECLAIM_ALLOWED 2w' expire in less than a day?
by Harry G. Coin 08 Jul '21

08 Jul '21

Is this happening to anyone else? After this command: ceph health mute AUTH_INSECURE_GLOBAL_ID_RECLAIM_ALLOWED 2w The 'dashboard' shows 'Health OK', then after a few hours (perhaps a mon leadership change), it's back to 'degraded' and 'AUTH_INSECURE_GLOBAL_ID_RECLAIM_ALLOWED: mons are allowing insecure global_id reclaim' Pacific, 16.2.4 all in docker containers.

1 0

Create and listing topics with AWS4 fails
by Daniel Iwan 07 Jul '21

07 Jul '21

Hi I'm on Pacific 16.2.1 Documentation states that topic operations should be created using REST with application/x-www-form-urlencoded See https://docs.ceph.com/en/latest/radosgw/notifications/#topics However when attempting to create one using Postman (auth v4) operation fails. <?xml version="1.0" encoding="UTF-8"?> <Error> <Code>NotImplemented</Code> <RequestId>tx00000000000000000004e-0060d207d0-df5c88-abcf</RequestId> <HostId>df5c88-abc-default</HostId> </Error>. The same error is for listing topics. See log attached I think similar issues were reported on the threads - https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/23BZW2Q3TCU… - https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/WMKEYKTE5NH… - https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/2G4YWOUMVE2… I think the handling procedure is in https://github.com/ceph/ceph/blob/master/src/rgw/rgw_rest_s3.cc get_auth_data_v4()which does not have RGW_OP_PUBSUB_TOPIC_CREATE or similar topic-related operations. If that's the case this makes it more difficult to integrate those operations as majority of the sw uses AWSV4 authentication Can anyone confirm if the doc is wrong, auth V3 needs to be used etc? Is there a tracking issue for this already? Regards Daniel

2 4

CEPH logs to Graylog
by milosz＠dev.datacrunch.io 07 Jul '21

07 Jul '21

Hi, Want to have logs from cluster on Graylog but seems like CEPH send empty "host" field. Any one can help ? CEPH 16.2.3 # ceph config dump | grep graylog global advanced clog_to_graylog true global advanced clog_to_graylog_host xx.xx.xx.xx global basic err_to_graylog true global basic log_graylog_host xx.xx.xx.xx * global basic log_to_graylog true I see that my Graylog is hit by traffic from ceph on port 12201 udp and parsed by GELF udp Grylog logs: 2021-07-01 12:16:57,355 ERROR: org.graylog2.shared.buffers.processors.DecodingProcessor - Error processing message RawMessage{id=3ad5c6a1-da66-11eb-a55c-0242ac120005, messageQueueId=2810784, codec=gelf, payloadSize=340, timestamp=2021-07-01T12:16:57.354Z, remoteAddress=/xx.xx.xx.xx:34049} java.lang.IllegalArgumentException: GELF message <3ad5c6a1-da66-11eb-a55c-0242ac120005> (received from <xx.xx.xx.xx:34049>) has empty mandatory "host" field. at org.graylog2.inputs.codecs.GelfCodec.validateGELFMessage(GelfCodec.java:247) ~[graylog.jar:?] at org.graylog2.inputs.codecs.GelfCodec.decode(GelfCodec.java:140) ~[graylog.jar:?] at org.graylog2.shared.buffers.processors.DecodingProcessor.processMessage(DecodingProcessor.java:153) ~[graylog.jar:?] at org.graylog2.shared.buffers.processors.DecodingProcessor.onEvent(DecodingProcessor.java:94) [graylog.jar:?] at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:90) [graylog.jar:?] at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:47) [graylog.jar:?] at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?] at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292] Best regards Milosz

3 2

Continuing Ceph Issues with OSDs falling over
by Peter Childs 07 Jul '21

07 Jul '21

I'm still attempting to build a ceph cluster and I'm currently getting nowhere very very quickly. From what I can tell I have a slightly unstable setup and I'm yet to work out why. I currently have 24 servers and I'm planning to increase this to around 48 These servers are in three groups with different types of disks (and number) in each type. Currently I'm having an issue where every time I add a new server it adds the osd on the node and then a few random ods on the current hosts will all fall over and I'll only be able to get them up again by restart the daemons. I'm using cephadm. and the network is a QDR based IB network running IP over IB so its meant to be 40G but currently is behaving more like 10G (when I've tested it) Its still faster than the 1G management network I've also got. The machines are mostly running debian. There are a few machine running CentOS7 I'm meaning to redeploy when I get the time (so I can upgrade to Pacific) I'm running Octopus 15.2.13, I'm more than happy to change stuff I'm still trying to learn stuff so there is no data that I care about quite yet, I was looking for more stability before I go there. I really just want to know where to look for the problems rather than any exact answers, I'm yet to see any clues that might help Thanks in advance Peter Childs

2 1

Issue with cephadm not finding python3 after reboot
by mabi 07 Jul '21

07 Jul '21

Hello, After having done a rolling reboot of my Octopus 15.2.13 cluster of 8 nodes cephadm does not find python3 on the node and hence I get quite a few of the following warnings: [WRN] CEPHADM_HOST_CHECK_FAILED: 7 hosts fail cephadm check host ceph1f failed check: Can't communicate with remote host `ceph1f`, possibly because python3 is not installed there: [Errno 32] Broken pipe Here is the full stack trace from cephadm: 2021-07-06T06:03:20.798410+0000 mgr.ceph1a.xxqpph [ERR] Failed to apply osd.all-available-devices spec DriveGroupSpec(name=all-available-devices->placement=PlacementSpec(host_pattern='*'), service_id='all-available-devices', service_type='osd', data_devices=DeviceSelection(all=True), osd_id_claims={}, unmanaged=False, filter_logic='AND', preview_only=False): Can't communicate with remote host `ceph1d`, possibly because python3 is not installed there: [Errno 32] Broken pipe Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/module.py", line 1015, in _remote_connection conn, connr = self._get_connection(addr) File "/usr/share/ceph/mgr/cephadm/module.py", line 978, in _get_connection sudo=True if self.ssh_user != 'root' else False) File "/lib/python3.6/site-packages/remoto/backends/__init__.py", line 34, in __init__ self.gateway = self._make_gateway(hostname) File "/lib/python3.6/site-packages/remoto/backends/__init__.py", line 44, in _make_gateway self._make_connection_string(hostname) File "/lib/python3.6/site-packages/execnet/multi.py", line 134, in makegateway gw = gateway_bootstrap.bootstrap(io, spec) File "/lib/python3.6/site-packages/execnet/gateway_bootstrap.py", line 102, in bootstrap bootstrap_exec(io, spec) File "/lib/python3.6/site-packages/execnet/gateway_bootstrap.py", line 46, in bootstrap_exec "serve(io, id='%s-slave')" % spec.id, File "/lib/python3.6/site-packages/execnet/gateway_bootstrap.py", line 78, in sendexec io.write((repr(source) + "\n").encode("ascii")) File "/lib/python3.6/site-packages/execnet/gateway_base.py", line 409, in write self._write(data) BrokenPipeError: [Errno 32] Broken pipe During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/module.py", line 1019, in _remote_connection raise execnet.gateway_bootstrap.HostNotFound(msg) execnet.gateway_bootstrap.HostNotFound: Can't communicate with remote host `ceph1d`, possibly because python3 is not installed there: [Errno 32] Broken pipe The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/share/ceph/mgr/cephadm/serve.py", line 412, in _apply_all_services if self._apply_service(spec): File "/usr/share/ceph/mgr/cephadm/serve.py", line 450, in _apply_service self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec)) File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 51, in create_from_spec ret = create_from_spec_one(self.prepare_drivegroup(drive_group)) File "/usr/share/ceph/mgr/cephadm/utils.py", line 65, in forall_hosts_wrapper return CephadmOrchestrator.instance._worker_pool.map(do_work, vals) File "/lib64/python3.6/multiprocessing/pool.py", line 266, in map return self._map_async(func, iterable, mapstar, chunksize).get() File "/lib64/python3.6/multiprocessing/pool.py", line 644, in get raise self._value File "/lib64/python3.6/multiprocessing/pool.py", line 119, in worker result = (True, func(*args, **kwds)) File "/lib64/python3.6/multiprocessing/pool.py", line 44, in mapstar return list(map(*args)) File "/usr/share/ceph/mgr/cephadm/utils.py", line 59, in do_work return f(*arg) File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 47, in create_from_spec_one host, cmd, replace_osd_ids=osd_id_claims.get(host, []), env_vars=env_vars File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 56, in create_single_host out, err, code = self._run_ceph_volume_command(host, cmd, env_vars=env_vars) File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 271, in _run_ceph_volume_command error_ok=True) File "/usr/share/ceph/mgr/cephadm/module.py", line 1100, in _run_cephadm with self._remote_connection(host, addr) as tpl: File "/lib64/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/usr/share/ceph/mgr/cephadm/module.py", line 1046, in _remote_connection raise OrchestratorError(msg) from e orchestrator._interface.OrchestratorError: Can't communicate with remote host `ceph1d`, possibly because python3 is not installed there: [Errno 32] Broken pipe I checked directly on the nodes and I can execute "python3" command and I can also SSH into all nodes with the following test command: ssh -F ssh_config -i ~/cephadm_private_key root@nodeX So I don't really understand what could have broken the cephadm orchestrator... Any ideas? The cephfs itself is still working. Best regards, Mabi

1 1

ceph df (octopus) shows USED is 7 times higher than STORED in erasure coded pool
by Arkadiy Kulev 07 Jul '21

07 Jul '21

The pool *default.rgw.buckets.data* has *501 GiB* stored, but USED shows *3.5 TiB *(7 times higher!)*:* root@ceph-01:~# ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 196 TiB 193 TiB 3.5 TiB 3.6 TiB 1.85 TOTAL 196 TiB 193 TiB 3.5 TiB 3.6 TiB 1.85 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL device_health_metrics 1 1 19 KiB 12 56 KiB 0 61 TiB .rgw.root 2 32 2.6 KiB 6 1.1 MiB 0 61 TiB default.rgw.log 3 32 168 KiB 210 13 MiB 0 61 TiB default.rgw.control 4 32 0 B 8 0 B 0 61 TiB default.rgw.meta 5 8 4.8 KiB 11 1.9 MiB 0 61 TiB default.rgw.buckets.index 6 8 1.6 GiB 211 4.7 GiB 0 61 TiB default.rgw.buckets.data 10 128 501 GiB 5.36M 3.5 TiB 1.90 110 TiB The *default.rgw.buckets.data* pool is using erasure coding: root@ceph-01:~# ceph osd erasure-code-profile get EC_RGW_HOST crush-device-class=hdd crush-failure-domain=host crush-root=default jerasure-per-chunk-alignment=false k=6 m=4 plugin=jerasure technique=reed_sol_van w=8 If anyone could help explain why it's using up 7 times more space, it would help a lot. Versioning is disabled. ceph version 15.2.13 (octopus stable). Sincerely, Ark.

5 9

Pool size
by Rafael Quaglio 07 Jul '21

07 Jul '21

1 1

Ceph with BGP?
by German Anders 06 Jul '21

06 Jul '21

Hi All, I have an already created and functional ceph cluster (latest luminous release) with two networks one for the public (layer 2+3) and the other for the cluster, the public one uses VLAN and its 10GbE and the other one uses Infiniband with 56Gb/s, the cluster works ok. The public network uses Juniper QFX5100 switches with VLAN in layer2+3 configuration but the network team needs to move to a full layer3 and they want to use BGP, so the question is, how can we move to that schema? What are the considerations? Is it possible? Is there any step-by-step way to move to that schema? Also is anything better than BGP or other alternatives? In information will be really helpful Thanks in advance, Cheers, German

6 9

Spurious Read Errors: 0x6706be76
by Jay Sullivan 06 Jul '21

06 Jul '21

In the week since upgrading one of our clusters from Nautilus 14.2.21 to Pacific 16.2.4 I've seen four spurious read errors that always have the same bad checksum of 0x6706be76. I've never seen this in any of our clusters before. Here's an example of what I'm seeing in the logs: ceph-osd.132.log:2021-06-20T22:53:20.584-0400 7fde2e4fc700 -1 bluestore(/var/lib/ceph/osd/ceph-132) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x0, got 0x6706be76, expected 0xee74a56a, device location [0x18c81b40000~1000], logical extent 0x200000~1000, object #29:2d8210bf:::rbd_data.94f4232ae8944a.0000000000026c57:head# I'm not seeing any indication of inconsistent PGs, only the spurious read error. I don't see an explicit indication of a retry in the logs following the above message. Bluestore code to retry three times was introduced in 2018 following a similar issue with the same checksum: https://tracker.ceph.com/issues/22464 Here's an example of what my health detail looks like: HEALTH_WARN 1 OSD(s) have spurious read errors [WRN] BLUESTORE_SPURIOUS_READ_ERRORS: 1 OSD(s) have spurious read errors osd.117 reads with retries: 1 I followed this (unresolved) thread, too: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/DRBVFQLZ5ZY… I do have swap enabled, but I don't think memory pressure is an issue with 30GB available out of 96GB (and no sign I've been close to summoning the OOMkiller). The OSDs that have thrown the cluster into HEALTH_WARN with the spurious read errors are busy 12TB rotational HDDs and I _think_ it's only happening during a deep scrub. We're on Ubuntu 18.04; uname: 5.4.0-74-generic #83~18.04.1-Ubuntu SMP Tue May 11 16:01:00 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux. Does Pacific retry three times on a spurious read error? Would I see an indication of a retry in the logs? Thanks! ~Jay

3 4

At rest encryption and lockbox keys
by Stephen Smith6 06 Jul '21

06 Jul '21

1 0

Re: Ceph survey results
by Satoru Takeuchi 06 Jul '21

06 Jul '21

Hi, 2021年7月4日(日) 0:17 changcheng.liu <changcheng.liu(a)aliyun.com>: > > Hi all, > I'm reading the ceph survey results: https://ceph.io/community/2021-ceph-user-survey-results. > Do we have the data about which type of AsyncMessenger is used? TCP/RDMA/DPDK. > What's the reason that RDMA & DPDK isn't often used? > You can find both the report and raw data below. > > 2021 Ceph User Survey Results > 2021 Ceph User Survey Raw Data The above two links are 404. Could you fix them? Thanks, Satoru > > B.R. > Jerry > > > _______________________________________________ > Dev mailing list -- dev(a)ceph.io > To unsubscribe send an email to dev-leave(a)ceph.io

1 0

Re: XFS on RBD on EC painfully slow
by Sebastian Knust 06 Jul '21

06 Jul '21

Hi Reed, To add to this command by Weiwen: On 28.05.21 13:03, 胡玮文 wrote: > Have you tried just start multiple rsync process simultaneously to transfer different directories? Distributed system like ceph often benefits from more parallelism. When I migrated from XFS on iSCSI (legacy system, no Ceph) to CephFS a few months ago, I used msrsync [1] and was quite happy with the speed. For your use case, I would start with -p 12 but might experiment with up to -p 24 (as you only have 6C/12T in your CPU). With many small files, you also might want to increase -s from the default 1000. Note that msrsync does not work with the --delete rsync flag. As I was syncing a live system, I ended up with this workflow: - Initial sync with msrsync (something like ./msrsync -p 12 --progress --stats --rsync "-aS --numeric-ids" ...) - Second sync with msrsync (to sync changes during the first sync) - Take old storage off-line for users / read-only - Final rsync with --delete (i.e. rsync -aS --numeric-ids --delete ...) - Mount cephfs at location of old storage, adjust /etc/exports with fsid entries where necessary, turn system back on-line / read-write Cheers Sebastian [1] https://github.com/jbd/msrsync

3 2

Graphics in ceph dashboard
by Fyodor Ustinov 06 Jul '21

06 Jul '21

Hi! I installed fresh cluster 16.2.4 as described in https://docs.ceph.com/en/latest/cephadm/#cephadm Everything works except for one thing: there are only graphics in the hosts / overall performance, (only the CPU and the network). In all other places the inscription "no data". What could I have done wrong? WBR, Fyodor.

2 2

Haproxy config, multilple RGW on the same node with different ports haproxy ignore
by Szabo, Istvan (Agoda) 05 Jul '21

05 Jul '21

Hi, I have this config: https://jpst.it/2yBsD What I'm missing from the backend part to make it able to balance on the same server but different port? Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0

cephadm shell fails to start due to missing config files?
by Vladimir Brik 05 Jul '21

05 Jul '21

Hello I am getting an error on one node in my cluster (other nodes are fine) when trying to run "cephadm shell". Historically this machine has been used as the primary Ceph management host, so it would be nice if this could be fixed. ceph-1 ~ # cephadm -v shell container_init=False Inferring fsid 79656e6e-21e2-4092-ac04-d536f25a435d Inferring config /var/lib/ceph/79656e6e-21e2-4092-ac04-d536f25a435d/mon.ceph-1/config Running command: /usr/bin/podman images --filter label=ceph=True --filter dangling=false --format {{.Repository}}(a){{.Digest}} /usr/bin/podman: stdout docker.io/ceph/daemon-base@sha256:0810dc7db854150bc48cf8fc079875e28b3138d070990a630b8fb7cec7cd2ced /usr/bin/podman: stdout docker.io/ceph/ceph@sha256:54e95ae1e11404157d7b329d0bef866ebbb214b195a009e87aae4eba9d282949 /usr/bin/podman: stdout docker.io/ceph/ceph@sha256:16d37584df43bd6545d16e5aeba527de7d6ac3da3ca7b882384839d2d86acc7d Using recent ceph image docker.io/ceph/daemon-base@sha256:0810dc7db854150bc48cf8fc079875e28b3138d070990a630b8fb7cec7cd2ced Running command: /usr/bin/podman run --rm --ipc=host --net=host --entrypoint stat -e CONTAINER_IMAGE=docker.io/ceph/daemon-base@sha256:0810dc7db854150bc48cf8fc079875e28b3138d070990a630b8fb7cec7cd2ced -e NODE_NAME=ceph-1 docker.io/ceph/daemon-base@sha256:0810dc7db854150bc48cf8fc079875e28b3138d070990a630b8fb7cec7cd2ced -c %u %g /var/lib/ceph stat: stdout 167 167 Running command (timeout=None): /usr/bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk -it -e LANG=C -e PS1=[ceph: \u@\h \W]\$ -e CONTAINER_IMAGE=docker.io/ceph/daemon-base@sha256:0810dc7db854150bc48cf8fc079875e28b3138d070990a630b8fb7cec7cd2ced -e NODE_NAME=ceph-1 -v /var/run/ceph/79656e6e-21e2-4092-ac04-d536f25a435d:/var/run/ceph:z -v /var/log/ceph/79656e6e-21e2-4092-ac04-d536f25a435d:/var/log/ceph:z -v /var/lib/ceph/79656e6e-21e2-4092-ac04-d536f25a435d/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /var/lib/ceph/79656e6e-21e2-4092-ac04-d536f25a435d/mon.ceph-1/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z -v /var/lib/ceph/79656e6e-21e2-4092-ac04-d536f25a435d/home:/root --entrypoint bash docker.io/ceph/daemon-base@sha256:0810dc7db854150bc48cf8fc079875e28b3138d070990a630b8fb7cec7cd2ced Error: error checking path "/var/lib/ceph/79656e6e-21e2-4092-ac04-d536f25a435d/mon.ceph-1/config": stat /var/lib/ceph/79656e6e-21e2-4092-ac04-d536f25a435d/mon.ceph-1/config: no such file or directory The machine in question doesn't run a mon daemon (but it did a long time ago), so I am not sure why "cephadm shell" on this particular machine is looking for mon.ceph-1/config Can anybody help? Thanks, Vlad

2 1

Objectstore user IO and operations monitoring
by Szabo, Istvan (Agoda) 05 Jul '21

05 Jul '21

Hi, I'm looking for this long time ago, I have a lot of users and when 1 user can take down the cluster I want to know which one, but there isn't any bucket stats that could help. Anyone knows anything? Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo(a)agoda.com<mailto:istvan.szabo@agoda.com> --------------------------------------------------- ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0

Remove objectstore from a RBD RGW cluster
by Szabo, Istvan (Agoda) 05 Jul '21

05 Jul '21

Hi, I want to remove all the objectstore related things from my cluster and keep only for RBD. I've uninstalled the RGW services. Removed the haproxy config related to that. When I try to delete realm, zone, zonegroup it is finished but after coupe of minutes something recreate another zonegroup. I can't figure out what. What I miss? PS: The pools are still there, that will be the last step, hope I don't miss any necessary step. Istvan Szabo Senior Infrastructure Engineer --------------------------------------------------- Agoda Services Co., Ltd. e: istvan.szabo(a)agoda.com<mailto:istvan.szabo@agoda.com> --------------------------------------------------- ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 1

how to compare setting differences between two rbd images
by Philip Brown 02 Jul '21

02 Jul '21

I have two rbd images. running fio on them, one performs very well, and one performs poorly. I'd like to get more insight on why. I know that rbd info xx/yyy gives me a small amount of information. but I cant seem to find a detailed info dump, in the way you can do things like ceph daemon osd.9 config show I cant even get useful google hits on things like "ceph find rbd image stripe-unit" all the pages seem to detail how to set the value, but not how to query it. Can anyone point me in the right direction? -- Philip Brown| Sr. Linux System Administrator | Medata, Inc. 5 Peters Canyon Rd Suite 250 Irvine CA 92606 Office 714.918.1310| Fax 714.918.1325 pbrown(a)medata.com| www.medata.com

1 0

rbd: map failed: rbd: sysfs write failed -- (108) Cannot send after transport endpoint shutdown
by Oliver Dzombic 02 Jul '21

02 Jul '21

Hi, mapping of rbd volumes fails clusterwide. The volumes that are mapped, are ok, but new volumes wont map. Receiving errors liks: (108) Cannot send after transport endpoint shutdown or executing: #rbd -p CEPH map vm-120-disk-0 will show: rbd: sysfs write failed In some cases useful info is found in syslog - try "dmesg | tail". rbd: map failed: (108) Cannot send after transport endpoint shutdown I added some more debugging for the logs and got this: rbd: sysfs write failed 2021-07-01T10:28:44.258+0200 7f803c4a33c0 5 librbd::io::ImageRequestWQ: 0x7f802c0011f0 ImageRequestWQ: ictx=0x55846e936c40 2021-07-01T10:28:44.258+0200 7f803c4a33c0 20 librbd::ImageState: 0x7f802c001420 open 2021-07-01T10:28:44.258+0200 7f803c4a33c0 10 librbd::ImageState: 0x7f802c001420 0x7f802c001420 send_open_unlock 2021-07-01T10:28:44.258+0200 7f803c4a33c0 10 librbd::image::OpenRequest: 0x55846e979620 send_v2_detect_header 2021-07-01T10:28:44.262+0200 7f80297fa700 10 librbd::image::OpenRequest: handle_v2_detect_header: r=0 2021-07-01T10:28:44.262+0200 7f80297fa700 10 librbd::image::OpenRequest: 0x55846e979620 send_v2_get_id 2021-07-01T10:28:44.262+0200 7f80297fa700 10 librbd::image::OpenRequest: handle_v2_get_id: r=0 2021-07-01T10:28:44.262+0200 7f80297fa700 10 librbd::image::OpenRequest: 0x55846e979620 send_v2_get_initial_metadata 2021-07-01T10:28:44.262+0200 7f80297fa700 10 librbd::image::OpenRequest: handle_v2_get_initial_metadata: r=0 2021-07-01T10:28:44.262+0200 7f80297fa700 10 librbd::image::OpenRequest: 0x55846e979620 send_v2_get_create_timestamp 2021-07-01T10:28:44.262+0200 7f80297fa700 10 librbd::image::OpenRequest: 0x55846e979620 handle_v2_get_create_timestamp: r=0 2021-07-01T10:28:44.262+0200 7f80297fa700 10 librbd::image::OpenRequest: 0x55846e979620 send_v2_get_access_modify_timestamp 2021-07-01T10:28:44.266+0200 7f80297fa700 10 librbd::image::OpenRequest: 0x55846e979620 handle_v2_get_access_modify_timestamp: r=0 2021-07-01T10:28:44.266+0200 7f80297fa700 10 librbd::image::OpenRequest: 0x55846e979620 send_v2_get_data_pool 2021-07-01T10:28:44.266+0200 7f80297fa700 10 librbd::image::OpenRequest: 0x55846e979620 handle_v2_get_data_pool: r=0 2021-07-01T10:28:44.266+0200 7f80297fa700 10 librbd::ImageCtx: init_layout stripe_unit 4194304 stripe_count 1 object_size 4194304 prefix rbd_data.21f6f3a7ec4663 format rbd_data.21f6f3a7ec4663.%016llx 2021-07-01T10:28:44.266+0200 7f80297fa700 10 librbd::image::OpenRequest: 0x55846e979620 send_refresh 2021-07-01T10:28:44.266+0200 7f80297fa700 10 librbd::ConfigWatcher: init: 2021-07-01T10:28:44.266+0200 7f80297fa700 10 librbd::image::RefreshRequest: 0x7f8010011870 send_v2_get_mutable_metadata 2021-07-01T10:28:44.266+0200 7f80297fa700 10 librbd::image::RefreshRequest: 0x7f8010011870 handle_v2_get_mutable_metadata: r=0 2021-07-01T10:28:44.266+0200 7f80297fa700 10 librbd::image::RefreshRequest: 0x7f8010011870 send_v2_get_parent: legacy=0 2021-07-01T10:28:44.266+0200 7f80297fa700 10 librbd::image::RefreshRequest: 0x7f8010011870 handle_v2_get_parent: r=0 2021-07-01T10:28:44.266+0200 7f80297fa700 10 librbd::image::RefreshRequest: 0x7f8010011870 send_v2_get_metadata 2021-07-01T10:28:44.266+0200 7f80297fa700 15 librbd::image::GetMetadataRequest: 0x7f80100126a0 metadata_list: start_key=conf_ 2021-07-01T10:28:44.266+0200 7f80297fa700 15 librbd::image::GetMetadataRequest: 0x7f80100126a0 handle_metadata_list: r=0 2021-07-01T10:28:44.266+0200 7f80297fa700 15 librbd::image::GetMetadataRequest: 0x7f80100126a0 finish: r=0 2021-07-01T10:28:44.266+0200 7f80297fa700 10 librbd::image::RefreshRequest: 0x7f8010011870 handle_v2_get_metadata: r=0 2021-07-01T10:28:44.266+0200 7f80297fa700 10 librbd::image::RefreshRequest: 0x7f8010011870 send_v2_get_pool_metadata 2021-07-01T10:28:44.266+0200 7f80297fa700 15 librbd::image::GetMetadataRequest: 0x7f8010012c10 metadata_list: start_key=conf_ 2021-07-01T10:28:44.266+0200 7f80297fa700 15 librbd::image::GetMetadataRequest: 0x7f8010012c10 handle_metadata_list: r=0 2021-07-01T10:28:44.266+0200 7f80297fa700 15 librbd::image::GetMetadataRequest: 0x7f8010012c10 finish: r=0 2021-07-01T10:28:44.266+0200 7f80297fa700 10 librbd::image::RefreshRequest: 0x7f8010011870 handle_v2_get_pool_metadata: r=0 2021-07-01T10:28:44.266+0200 7f80297fa700 20 librbd::ImageCtx: apply_metadata 2021-07-01T10:28:44.266+0200 7f80297fa700 5 librbd::ImageCtx: 0x55846e936c40: disabling zero-copy writes 2021-07-01T10:28:44.266+0200 7f80297fa700 10 librbd::image::RefreshRequest: 0x7f8010011870 send_v2_get_group 2021-07-01T10:28:44.266+0200 7f80297fa700 10 librbd::image::RefreshRequest: 0x7f8010011870 handle_v2_get_group: r=0 2021-07-01T10:28:44.266+0200 7f80297fa700 10 librbd::image::RefreshRequest: 0x7f8010011870 send_v2_apply 2021-07-01T10:28:44.266+0200 7f8028ff9700 10 librbd::image::RefreshRequest: 0x7f8010011870 handle_v2_apply 2021-07-01T10:28:44.266+0200 7f8028ff9700 20 librbd::image::RefreshRequest: 0x7f8010011870 apply 2021-07-01T10:28:44.266+0200 7f8028ff9700 10 librbd::image::OpenRequest: handle_refresh: r=0 2021-07-01T10:28:44.266+0200 7f8028ff9700 10 librbd::image::OpenRequest: send_parent_cache: r=0 2021-07-01T10:28:44.266+0200 7f8028ff9700 10 librbd::image::OpenRequest: 0x55846e979620 send_init_cache 2021-07-01T10:28:44.266+0200 7f8028ff9700 5 librbd::cache::WriteAroundObjectDispatch: 0x7f80180020d0 init: 2021-07-01T10:28:44.266+0200 7f8028ff9700 5 librbd::io::ObjectDispatcher: 0x55846e979670 register_object_dispatch: object_dispatch_layer=1 2021-07-01T10:28:44.266+0200 7f8028ff9700 10 librbd::ImageState: 0x7f802c001420 0x7f802c001420 handle_open: r=0 In some cases useful info is found in syslog - try "dmesg | tail". 2021-07-01T10:28:44.266+0200 7f803c4a33c0 20 librbd::ImageState: 0x7f802c001420 close 2021-07-01T10:28:44.266+0200 7f803c4a33c0 10 librbd::ImageState: 0x7f802c001420 0x7f802c001420 send_close_unlock 2021-07-01T10:28:44.266+0200 7f803c4a33c0 10 librbd::ConfigWatcher: shut_down: 2021-07-01T10:28:44.266+0200 7f803c4a33c0 10 librbd::image::CloseRequest: 0x55846e97a510 send_block_image_watcher 2021-07-01T10:28:44.270+0200 7f803c4a33c0 10 librbd::ImageWatcher: 0x7f8010003e00 block_notifies 2021-07-01T10:28:44.270+0200 7f803c4a33c0 5 librbd::Watcher: 0x7f8010003e00 block_notifies: blocked_count=1 2021-07-01T10:28:44.270+0200 7f803c4a33c0 10 librbd::image::CloseRequest: 0x55846e97a510 handle_block_image_watcher: r=0 2021-07-01T10:28:44.270+0200 7f803c4a33c0 10 librbd::image::CloseRequest: 0x55846e97a510 send_shut_down_update_watchers 2021-07-01T10:28:44.270+0200 7f803c4a33c0 20 librbd::ImageState: 0x7f802c001420 shut_down_update_watchers 2021-07-01T10:28:44.270+0200 7f803c4a33c0 20 librbd::ImageState: 0x55846e80a230 ImageUpdateWatchers::shut_down 2021-07-01T10:28:44.270+0200 7f803c4a33c0 20 librbd::ImageState: 0x55846e80a230 ImageUpdateWatchers::shut_down: completing shut down 2021-07-01T10:28:44.270+0200 7f8028ff9700 10 librbd::image::CloseRequest: 0x55846e97a510 handle_shut_down_update_watchers: r=0 2021-07-01T10:28:44.270+0200 7f8028ff9700 10 librbd::image::CloseRequest: 0x55846e97a510 send_shut_down_io_queue 2021-07-01T10:28:44.270+0200 7f8028ff9700 5 librbd::io::ImageRequestWQ: 0x7f802c0011f0 shut_down: shut_down: in_flight=0 2021-07-01T10:28:44.270+0200 7f8028ff9700 20 librbd::io::AsyncOperation: 0x7f80180024c8 start_op 2021-07-01T10:28:44.270+0200 7f8028ff9700 20 librbd::io::ImageRequest: 0x7f8028ff3d30 send: aio_flush: ictx=0x55846e936c40, completion=0x7f80180023d0 2021-07-01T10:28:44.270+0200 7f8028ff9700 20 librbd::io::AioCompletion: 0x7f80180023d0 set_request_count: pending=1 2021-07-01T10:28:44.270+0200 7f8028ff9700 20 librbd::io::ObjectDispatcher: 0x55846e979670 send: object_dispatch_spec=0x7f8018002520 2021-07-01T10:28:44.270+0200 7f8028ff9700 20 librbd::cache::WriteAroundObjectDispatch: 0x7f80180020d0 flush: 2021-07-01T10:28:44.270+0200 7f8028ff9700 20 librbd::io::AioCompletion: 0x7f80180023d0 complete_request: cb=1, pending=0 2021-07-01T10:28:44.270+0200 7f8028ff9700 20 librbd::io::AioCompletion: 0x7f80180023d0 finalize: r=0 2021-07-01T10:28:44.270+0200 7f8028ff9700 10 librbd::image::CloseRequest: 0x55846e97a510 handle_shut_down_io_queue: r=0 2021-07-01T10:28:44.270+0200 7f8028ff9700 10 librbd::image::CloseRequest: 0x55846e97a510 send_flush 2021-07-01T10:28:44.270+0200 7f8028ff9700 20 librbd::io::AsyncOperation: 0x7f8018002718 start_op 2021-07-01T10:28:44.270+0200 7f8028ff9700 20 librbd::io::ImageRequest: 0x7f8028ff3c70 send: aio_flush: ictx=0x55846e936c40, completion=0x7f8018002620 2021-07-01T10:28:44.270+0200 7f8028ff9700 20 librbd::io::AioCompletion: 0x7f8018002620 set_request_count: pending=1 2021-07-01T10:28:44.270+0200 7f8028ff9700 20 librbd::io::AsyncOperation: 0x7f80180024c8 finish_op 2021-07-01T10:28:44.270+0200 7f8028ff9700 20 librbd::io::AsyncOperation: completed flush: 0x7f801000e6d0 2021-07-01T10:28:44.270+0200 7f8028ff9700 20 librbd::io::ObjectDispatcher: 0x55846e979670 send: object_dispatch_spec=0x7f8018002520 2021-07-01T10:28:44.270+0200 7f8028ff9700 20 librbd::cache::WriteAroundObjectDispatch: 0x7f80180020d0 flush: 2021-07-01T10:28:44.270+0200 7f8028ff9700 20 librbd::io::AioCompletion: 0x7f8018002620 complete_request: cb=1, pending=0 2021-07-01T10:28:44.270+0200 7f8028ff9700 20 librbd::io::AioCompletion: 0x7f8018002620 finalize: r=0 2021-07-01T10:28:44.270+0200 7f8028ff9700 10 librbd::image::CloseRequest: 0x55846e97a510 handle_flush: r=0 2021-07-01T10:28:44.270+0200 7f8028ff9700 10 librbd::image::CloseRequest: 0x55846e97a510 send_unregister_image_watcher 2021-07-01T10:28:44.270+0200 7f8028ff9700 10 librbd::ImageWatcher: 0x7f8010003e00 unregistering image watcher 2021-07-01T10:28:44.270+0200 7f8028ff9700 10 librbd::Watcher: 0x7f8010003e00 unregister_watch: 2021-07-01T10:28:44.270+0200 7f8028ff9700 20 librbd::io::AsyncOperation: 0x7f8018002718 finish_op 2021-07-01T10:28:44.270+0200 7f8022ffd700 10 librbd::image::CloseRequest: 0x55846e97a510 handle_unregister_image_watcher: r=0 2021-07-01T10:28:44.270+0200 7f8022ffd700 10 librbd::image::CloseRequest: 0x55846e97a510 send_flush_readahead 2021-07-01T10:28:44.270+0200 7f8028ff9700 10 librbd::image::CloseRequest: 0x55846e97a510 handle_flush_readahead: r=0 2021-07-01T10:28:44.270+0200 7f8028ff9700 10 librbd::image::CloseRequest: 0x55846e97a510 send_shut_down_object_dispatcher 2021-07-01T10:28:44.270+0200 7f8028ff9700 5 librbd::io::ObjectDispatcher: 0x55846e979670 shut_down: 2021-07-01T10:28:44.270+0200 7f8028ff9700 5 librbd::io::ObjectDispatch: 0x55846e97b430 shut_down: 2021-07-01T10:28:44.270+0200 7f8028ff9700 5 librbd::cache::WriteAroundObjectDispatch: 0x7f80180020d0 shut_down: 2021-07-01T10:28:44.270+0200 7f8028ff9700 10 librbd::image::CloseRequest: 0x55846e97a510 handle_shut_down_object_dispatcher: r=0 2021-07-01T10:28:44.270+0200 7f8028ff9700 10 librbd::image::CloseRequest: 0x55846e97a510 send_flush_op_work_queue 2021-07-01T10:28:44.270+0200 7f8028ff9700 10 librbd::image::CloseRequest: 0x55846e97a510 handle_flush_op_work_queue: r=0 2021-07-01T10:28:44.270+0200 7f8028ff9700 10 librbd::image::CloseRequest: 0x55846e97a510 handle_flush_image_watcher: r=0 2021-07-01T10:28:44.270+0200 7f8028ff9700 10 librbd::ImageState: 0x7f802c001420 0x7f802c001420 handle_close: r=0 rbd: map failed: (108) Cannot send after transport endpoint shutdown ... which does actually not tell me too much honestly. For now, i am out of idea's, can someone lead a way to analyse / fix this ? New volumes can be created and removed without issues. But they wont map. Thank you! -- Mit freundlichen Gruessen / Best regards Oliver Dzombic Layer7 Networks mailto:info@layer7.net Anschrift: Layer7 Networks GmbH Zum Sonnenberg 1-3 63571 Gelnhausen HRB 96293 beim Amtsgericht Hanau Geschäftsführung: Oliver Dzombic UST ID: DE259845632

2 1

configure fuse in fstab
by Simon Sutter 01 Jul '21

01 Jul '21

Hello Everyone! I'm trying to mount the ceph, with the fuse client under debian 9 (ceph-fuse 10.2.11-2). Ceph is on the latest Octopus release. The direct command is working, but writing it in fstab does not. Command I use: ceph-fuse --id dev.wsc -k /etc/ceph/ceph.clinet.dev.wsc.keyring -r /testing/dev.wsc/userdata /mnt/ceph /etc/ceph.conf: [global] fsid = [fsid] mon_host = 10.99.15.1 10.99.15.2 10.99.15.3 10.99.15.4 10.99.15.5 /etc/ceph/ceph.clinet.dev.wsc.keyring: [client.dev.wsc] key = [key] fstab: none /mnt/ceph fuse.ceph ceph.id=dev.wsc,ceph.conf=/etc/ceph/ceph.conf,ceph.client_mountpoint=/testing/dev.wsc/userdata,_netdev,defaults 0 0 With mount -a it tells me "ceph mount failed with (1) Operation not permitted". Where can I find authentication logs on the cluster? Thanks in advance and kind regards, Simon

2 2

Pacific: RadosGW crashing on multipart uploads.
by Chu, Vincent 01 Jul '21

01 Jul '21

Hi, I'm running into an issue with RadosGW where multipart uploads crash, but only on buckets with a hyphen, period or underscore in the bucket name and with a bucket policy applied. We've tested this in pacific 16.2.3 and pacific 16.2.4. Anyone run into this before? ubuntu@ubuntu:~/ubuntu$ aws --endpoint http://placeholder.com:7480 s3 cp ubuntu.iso s3://bucket.test upload failed: ./ubuntu.iso to s3://bucket.test/ubuntu.iso Connection was closed before we received a valid response from endpoint URL: "http://placeholder.com:7480/bucket.test/ubuntu.iso?uploads". Here is the crash log. -12> 2021-06-29T20:44:10.940+0000 7fae1f4ec700 1 ====== starting new request req=0x7fadf8998620 ===== -11> 2021-06-29T20:44:10.940+0000 7fae1f4ec700 2 req 2403 0.000000000s initializing for trans_id = tx000000000000000000963-0060db861a-17e77ee-default -10> 2021-06-29T20:44:10.940+0000 7fae1f4ec700 2 req 2403 0.000000000s getting op 4 -9> 2021-06-29T20:44:10.940+0000 7fae1f4ec700 2 req 2403 0.000000000s s3:init_multipart verifying requester -8> 2021-06-29T20:44:10.948+0000 7fae1f4ec700 2 req 2403 0.008000608s s3:init_multipart normalizing buckets and tenants -7> 2021-06-29T20:44:10.948+0000 7fae1f4ec700 2 req 2403 0.008000608s s3:init_multipart init permissions -6> 2021-06-29T20:44:10.954+0000 7faedf66c700 0 Supplied principal is discarded: arn:aws:iam::default:user -5> 2021-06-29T20:44:10.954+0000 7faedf66c700 2 req 2403 0.014001064s s3:init_multipart recalculating target -4> 2021-06-29T20:44:10.954+0000 7faedf66c700 2 req 2403 0.014001064s s3:init_multipart reading permissions -3> 2021-06-29T20:44:10.954+0000 7faedf66c700 2 req 2403 0.014001064s s3:init_multipart init op -2> 2021-06-29T20:44:10.954+0000 7faedf66c700 2 req 2403 0.014001064s s3:init_multipart verifying op mask -1> 2021-06-29T20:44:10.955+0000 7faedf66c700 2 req 2403 0.015001140s s3:init_multipart verifying op permissions 0> 2021-06-29T20:44:10.964+0000 7faedf66c700 -1 *** Caught signal (Segmentation fault) ** in thread 7faedf66c700 thread_name:radosgw ceph version 16.2.3 (381b476cb3900f9a92eb95d03b4850b953cfd79a) pacific (stable) 1: /lib64/libpthread.so.0(+0x12b20) [0x7faf2dd05b20] 2: (rgw_bucket::rgw_bucket(rgw_bucket const&)+0x23) [0x7faf38b4d083] 3: (rgw::sal::RGWObject::get_obj() const+0x20) [0x7faf38b7bcf0] 4: (RGWInitMultipart::verify_permission(optional_yield)+0x6c) [0x7faf38e6608c] 5: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, optional_yield, bool)+0x86a) [0x7faf38b2db1a] 6: (process_request(rgw::sal::RGWRadosStore*, RGWREST*, RGWRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*, int*)+0x26dd) [0x7faf38b3232d] 7: /lib64/libradosgw.so.2(+0x4a1c0b) [0x7faf38a83c0b] 8: /lib64/libradosgw.so.2(+0x4a36a4) [0x7faf38a856a4] 9: /lib64/libradosgw.so.2(+0x4a390e) [0x7faf38a8590e] 10: make_fcontext() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. -- Vincent Chu A-4: Advanced Research in Cyber Systems Los Alamos National Laboratory

2 1

cephadm dashboard errors
by Anthony Palermo 01 Jul '21

01 Jul '21

I am having issues accessing the dashboard [dashboard WARNING request] [::ffff:10.2.16.6:49696] [GET] [401] [0.045s] [463.0B] [d4a77c44-f9ee-45bc-8f90-870362e131bc] /api/mgr/module/telemetry Jun 30 20:51:16 cfnode01 bash[2655]: debug 2021-06-30T20:51:16.373+0000 7f536f15b700 0 [dashboard INFO request] [::ffff:10.2.16.6:49695] [GET] [200] [0.019s] [29.0B] [d3ecd267-948d-4dfa-887b-ccef517ad91b] /ui-api/langs Jun 30 20:51:16 cfnode01 bash[2655]: debug 2021-06-30T20:51:16.385+0000 7f536c956700 0 [dashboard ERROR exception] Internal Server Error Jun 30 20:51:16 cfnode01 bash[2655]: Traceback (most recent call last): Jun 30 20:51:16 cfnode01 bash[2655]: File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 46, in dashboard_exception_handler Jun 30 20:51:16 cfnode01 bash[2655]: return handler(*args, **kwargs) Jun 30 20:51:16 cfnode01 bash[2655]: File "/lib/python3.6/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__ Jun 30 20:51:16 cfnode01 bash[2655]: return self.callable(*self.args, **self.kwargs) Jun 30 20:51:16 cfnode01 bash[2655]: File "/usr/share/ceph/mgr/dashboard/controllers/home.py", line 135, in __call__ Jun 30 20:51:16 cfnode01 bash[2655]: return serve_file(full_path) Jun 30 20:51:16 cfnode01 bash[2655]: File "/lib/python3.6/site-packages/cherrypy/lib/static.py", line 77, in serve_file Jun 30 20:51:16 cfnode01 bash[2655]: cptools.validate_since() Jun 30 20:51:16 cfnode01 bash[2655]: File "/lib/python3.6/site-packages/cherrypy/lib/cptools.py", line 116, in validate_since Jun 30 20:51:16 cfnode01 bash[2655]: raise cherrypy.HTTPRedirect([], 304) Jun 30 20:51:16 cfnode01 bash[2655]: cherrypy._cperror.HTTPRedirect: ([], 304) Jun 30 20:51:16 cfnode01 bash[2655]: debug 2021-06-30T20:51:16.385+0000 7f536d157700 0 [dashboard WARNING request] [::ffff:10.2.16.6:49699] [GET] [401] [0.022s] [462.0B] [a59575fa-5f58-4db1-85aa-4f155e9ea639] /api/summary Jun 30 20:51:16 cfnode01 bash[2655]: debug 2021-06-30T20:51:16.385+0000 7f536a952700 0 [dashboard WARNING request] [::ffff:10.2.16.6:49697] [GET] [401] [0.007s] [463.0B] [f5eb4e44-060e-451e-a379-8990f8871ef8] /api/feature_toggles Jun 30 20:51:16 cfnode01 bash[2655]: debug 2021-06-30T20:51:16.385+0000 7f536c956700 0 [dashboard INFO request] [::ffff:10.2.16.6:49694] [GET] [304] [0.017s] [0.0B] [d90f8f90-2f65-401e-8bf3-ad6c1c62a368] /Ceph_Logo.487a0001b327fa7f5232.svg Jun 30 20:51:16 cfnode01 bash[2655]: debug 2021-06-30T20:51:16.457+0000 7f536e95a700 0 [dashboard INFO request] [::ffff:10.2.16.6:49695] [POST] [200] [0.005s] [40.0B] [9058a1ae-5f0a-47d1-b5fd-bcd646fdac48] /api/auth/check Jun 30 20:51:16 cfnode01 bash[2655]: debug 2021-06-30T20:51:16.461+0000 7f536e95a700 0 [dashboard ERROR frontend.error] (http://10.1.50.20:8080/#/login) Http failure response for http://10.1.50.20:8080/ui-api/standard_settings: 401 Unauthorized Jun 30 20:51:16 cfnode01 bash[2655]: None Jun 30 20:51:16 cfnode01 bash[2655]: debug 2021-06-30T20:51:16.465+0000 7f536e159700 0 [dashboard INFO request] [::ffff:10.2.16.6:49698] [GET] [200] [0.020s] [29.0B] [ae97598b-fc78-4510-88d1-d79d6251d6a2] /ui-api/langs Jun 30 20:51:16 cfnode01 bash[2655]: debug 2021-06-30T20:51:16.465+0000 7f536e95a700 0 [dashboard INFO request] [::ffff:10.2.16.6:49696] [POST] [200] [0.009s] [24.0B] [ad9954d6-7743-4215-9a4a-df3e908d84da] /ui-api/logging/js-error

1 0

Unprotect snapshot: device or resource busy
by Jan Kasprzak 01 Jul '21

01 Jul '21

Hello, Ceph users, How can I figure out why it is not possible to unprotect a snapshot in a RBD image? I use this RBD pool for OpenNebula, and somehow there is a snapshot in one image, which OpenNebula does not see. So I wanted to delete the snapshot: # rbd info one/one-1312 rbd image 'one-1312': size 8 GiB in 2048 objects order 22 (4 MiB objects) snapshot_count: 1 id: 6732dccd50fa75 block_name_prefix: rbd_data.6732dccd50fa75 format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten op_features: flags: create_timestamp: Wed Jun 30 10:41:59 2021 access_timestamp: Wed Jun 30 16:48:30 2021 modify_timestamp: Wed Jun 30 15:52:18 2021 # rbd snap ls one/one-1312 SNAPID NAME SIZE PROTECTED TIMESTAMP 1727 snap 8 GiB yes Wed Jun 30 16:11:39 2021 # rbd snap rm one/one-1312@snap Removing snap: 0% complete...failed. rbd: snapshot 'snap' is protected from removal. 2021-07-01 08:33:41.489 7f79c6ffd700 -1 librbd::Operations: snapshot is protected # rbd snap unprotect one/one-1312@snap 2021-07-01 08:28:40.747 7f3cb6ffd700 -1 librbd::SnapshotUnprotectRequest: cannot unprotect: at least 1 child(ren) [68ba8e7bace188] in pool 'one' 2021-07-01 08:28:40.749 7f3cb6ffd700 -1 librbd::SnapshotUnprotectRequest: encountered error: (16) Device or resource busy 2021-07-01 08:28:40.749 7f3cb6ffd700 -1 librbd::SnapshotUnprotectRequest: 0x56522f10e830 should_complete_error: ret_val=-16 rbd: unprotecting snap failed: 2021-07-01 08:28:40.751 7f3cb6ffd700 -1 librbd::SnapshotUnprotectRequest: 0x56522f10e830 should_complete_error: ret_val=-16 As far as I can see neither the snapshot nor the RBD image itself is used by a running qemu in my cluster. How can I delete the snapshot or debug the problem further? This is Ceph 14.2.21 on CentOS (mons are CentOS 8 now, and OSDs are CentOS 7). Thanks, -Yenya -- | Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> | | http://www.fi.muni.cz/~kas/ GPG: 4096R/A45477D5 | We all agree on the necessity of compromise. We just can't agree on when it's necessary to compromise. --Larry Wall

2 6

ceph tcp fastopen
by Seena Fallah 01 Jul '21

01 Jul '21

Hi, Is ceph using TCP_FASTOPEN for its sockets? If not why? Thanks.

1 0

Having issues to start more than 24 OSDs per host
by Jan.Jansen＠gdata.de 01 Jul '21

01 Jul '21

Hello We did try to use Cephadm with Podman to start 44 OSDs per host which consistently stop after adding 24 OSDs per host. We did look into the cephadm.log on the problematic host and saw that the command `cephadm ceph-volume lvm list --format json` did stuck. We were the output of the command wasn't complete. Therefore, we tried to use compacted JSON and we could increase the number to 36 OSDs per host. If you need more information just ask. Podman version: 3.2.1 Ceph version: 16.2.4 OS version: Suse Leap 15.3 Greetings, Jan

3 3

cephfs forward scrubbing docs
by Dan van der Ster 01 Jul '21

01 Jul '21

Hi, Today while debugging something we had a few questions that might lead to improving the cephfs forward scrub docs: https://docs.ceph.com/en/latest/cephfs/scrub/ tldr: 1. Should we document which sorts of issues that the forward scrub is able to fix? 2. Can we make it more visible (in docs) that scrubbing is not supported with multi-mds? 3. Isn't the new `ceph -s` scrub task status misleading with multi-mds? Details here: 1) We found a CephFS directory with a number of zero sized files: # ls -l ... -rw-r--r-- 1 1001890000 1001890000 0 Nov 3 11:58 upload_fc501199e3e7abe6b574101cf34aeefb.png -rw-r--r-- 1 1001890000 1001890000 0 Nov 3 12:23 upload_fce4f55348185fefa0abdd8d11095ba8.gif -rw-r--r-- 1 1001890000 1001890000 0 Nov 3 11:54 upload_fd95b8358851f0dac22fb775046a6163.png ... The user claims that those files were non-zero sized last week. The sequence of zero sized files includes *all* files written between Nov 2 and 9. The user claims that his client was running out of memory, but this is now fixed. So I suspect that his ceph client (kernel 3.10.0-1127.19.1.el7.x86_64) was not behaving well. Anyway, I noticed that even though the dentries list 0 bytes, the underlying rados objects have data, and the data looks good. E.g. # rados get -p cephfs_data 200212e68b5.00000000 --namespace=xxx 200212e68b5.00000000 # file 200212e68b5.00000000 200212e68b5.00000000: PNG image data, 960 x 815, 8-bit/color RGBA, non-interlaced So I managed to recover the files doing something like this (using an input file mapping inode to filename) [see PS 0]. But I'm wondering if a forward scrub is able to fix this sort of problem directly? Should we document which sorts of issues that the forward scrub is able to fix? I anyway tried to scrub it, which led to: # ceph tell mds.cephflax-mds-xxx scrub start /volumes/_nogroup/xxx recursive repair Scrub is not currently supported for multiple active MDS. Please reduce max_mds to 1 and then scrub. So ... 2) Shouldn't we update the doc to mention loud and clear that scrub is not currently supported for multiple active MDS? 3) I was somehow surprised by this, because I had thought that the new `ceph -s` multi-mds scrub status implied that multi-mds scrubbing was now working: task status: scrub status: mds.x: idle mds.y: idle mds.z: idle Is it worth reporting this task status for cephfs if we can't even scrub them? Thanks!! Dan [0] mkdir -p recovered while read -r a b; do for i in {0..9} do echo "rados stat --cluster=flax --pool=cephfs_data --namespace=xxx" $(printf "%x" $a).0000000$i "&&" "rados get --cluster=flax --pool=cephfs_data --namespace=xxx" $(printf "%x" $a).0000000$i $(printf "%x" $a).0000000$i done echo cat $(printf "%x" $a).* ">" $(printf "%x" $a) echo mv $(printf "%x" $a) recovered/$b done < inones_fnames.txt

2 2

Semantics of cephfs-mirror
by Manuel Holtgrewe 01 Jul '21

01 Jul '21

Dear all, I'm sorry if I'm asking for the obvious or missing a previous discussion of this but I could not find the answer to my question online. I'd be happy to be pointed to the right direction only. The cephfs-mirror tool in pacific looks extremely promising. How does it work exactly? Is it based on files and (recursive) ctime or rather based on object information? Does it handle incremental changes (only) between snapshots? There is an issue related to this that mentions recursive ctime. But that would mean that users could "rsync -a" data to the file system and this would not get synchronized. I have good experience with ZFS which is able to identify changes between two snapshots A and B and then only transfer these changes (using a sub-file level, on the ZFS equivalent of blocks to my understanding) to another server with the same file system that is in the exact state as snapshot A. Does cephfs-mirror work the same? Best wishes, Manuel

2 1