June 2021 - ceph-users - lists.ceph.io

OpenSSL security update for Octopus container?
by Robert Sander 02 Oct '25

02 Oct '25

Hi, I am running a Ceph Octopus cluster in version 15.2.10 setup with the cephadm orchestrator and official Docker hub container images. Yesterday an important security fix was released for libssl and is already packaged for all major distributions. I tried to run "ceph orch upgrade check docker.io/ceph/ceph:v15" but it tells me that the containers do not need to be upgraded. How will this security fix of OpenSSL be deployed in a timely manner to users of the Ceph container images? Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 93818 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 1

Correct Migration Workflow Replicated -> Erasure Code
by Mac Wynkoop 19 Jun '24

19 Jun '24

Hi Everyone, So, I'm in the process of trying to migrate our rgw.buckets.data pool from a replicated rule pool to an erasure coded pool. I've gotten the EC pool set up, good EC profile and crush ruleset, pool created successfully, but when I go to "rados cppool xxx.rgw.buckets.data xxx.rgw.buckets.data.new", I get this error after it transfers 4GB of data: error copying object: (2) No such file or directory error copying pool xxx.rgw.buckets.data => xxx.rgw.buckets.data.new: (2) No such file or directory Is "rados cppool" still the blessed way to do the migration, or has something better/not deprecated been developed that I can use? Thanks, Mac

4 4

cephfs creation error
by Ramanathan S 03 Apr '24

03 Apr '24

Hi all, I just had created a ceph cluster to use cephfs. When i create the a ceph fs pool i get the filesystem below error. # ceph osd pool create cephfs_data 128 pool 'cephfs_data' created # ceph osd pool create cephfs_metadata 128 pool 'cephfs_metadata' created # ceph fs new cephfs cephfs_metadata cephfs_data new fs with metadata pool 6 and data pool 5 # ceph -s cluster: id: 1c27def45-f0f9-494d-sfke-eb4323432fd health: HEALTH_ERR 1 filesystem is offline 1 filesystem is online with fewer MDS than max_mds services: mon: 2 daemons, quorum ceph-mon01,ceph-mon02 mgr: ceph-adm01(active) mds: cephfs-0/0/1 up osd: 12 osds: 12 up, 12 in data: pools: 2 pools, 256 pgs objects: 0 objects, 0 B usage: 12 GiB used, 588 GiB / 600 GiB avail pgs: 256 active+clean but when i check the max_mds for the ceph fs it says 1 # ceph fs get cephfs | grep max_mds max_mds 1 Let anyone know what am i missing here? Any inputs is much appreciated. Regards, Ram Ceph-explorer..

3 3

Re: NoSuchKey on key that is visible in s3 list/radosgw bk
by Eric Ivancich 03 Apr '24

03 Apr '24

I have some questions for those who’ve experienced this issue. 1. It seems like those reporting this issue are seeing it strictly after upgrading to Octopus. From what version did each of these sites upgrade to Octopus? From Nautilus? Mimic? Luminous? 2. Does anyone have any lifecycle rules on a bucket experiencing this issue? If so, please describe. 3. Is anyone making copies of the affected objects (to same or to a different bucket) prior to seeing the issue? And if they are making copies, does the destination bucket have lifecycle rules? And if they are making copies, are those copies ever being removed? 4. Is anyone experiencing this issue willing to run their RGWs with 'debug_ms=1'? That would allow us to see a request from an RGW to either remove a tail object or decrement its reference counter (and when its counter reaches 0 it will be deleted). Thanks, Eric > On Nov 12, 2020, at 4:54 PM, huxiaoyu(a)horebdata.cn wrote: > > Looks like this is a very dangerous bug for data safety. Hope the bug would be quickly identified and fixed. > > best regards, > > Samuel > > > > huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn> > > From: Janek Bevendorff > Date: 2020-11-12 18:17 > To: huxiaoyu(a)horebdata.cn <mailto:huxiaoyu@horebdata.cn>; EDH - Manuel Rios; Rafael Lopez > CC: Robin H. Johnson; ceph-users > Subject: Re: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > I have never seen this on Luminous. I recently upgraded to Octopus and the issue started occurring only few weeks later. > > On 12/11/2020 16:37, huxiaoyu(a)horebdata.cn wrote: > which Ceph versions are affected by this RGW bug/issues? Luminous, Mimic, Octupos, or the latest? > > any idea? > > samuel > > > > huxiaoyu(a)horebdata.cn > > From: EDH - Manuel Rios > Date: 2020-11-12 14:27 > To: Janek Bevendorff; Rafael Lopez > CC: Robin H. Johnson; ceph-users > Subject: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > This same error caused us to wipe a full cluster of 300TB... will be related to some rados index/database bug not to s3. > > As Janek exposed is a mayor issue, because the error silent happend and you can only detect it with S3, when you're going to delete/purge a S3 bucket. Dropping NoSuchKey. Error is not related to S3 logic .. > > Hope this time dev's can take enought time to find and resolve the issue. Error happens with low ec profiles, even with replica x3 in some cases. > > Regards > > > > -----Mensaje original----- > De: Janek Bevendorff <janek.bevendorff(a)uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de>> > Enviado el: jueves, 12 de noviembre de 2020 14:06 > Para: Rafael Lopez <rafael.lopez(a)monash.edu <mailto:rafael.lopez@monash.edu>> > CC: Robin H. Johnson <robbat2(a)gentoo.org <mailto:robbat2@gentoo.org>>; ceph-users <ceph-users(a)ceph.io <mailto:ceph-users@ceph.io>> > Asunto: [ceph-users] Re: NoSuchKey on key that is visible in s3 list/radosgw bk > > Here is a bug report concerning (probably) this exact issue: > https://tracker.ceph.com/issues/47866 <https://tracker.ceph.com/issues/47866> > > I left a comment describing the situation and my (limited) experiences with it. > > > On 11/11/2020 10:04, Janek Bevendorff wrote: >> >> Yeah, that seems to be it. There are 239 objects prefixed >> .8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh in my dump. However, there are none >> of the multiparts from the other file to be found and the head object >> is 0 bytes. >> >> I checked another multipart object with an end pointer of 11. >> Surprisingly, it had way more than 11 parts (39 to be precise) named >> .1, .1_1 .1_2, .1_3, etc. Not sure how Ceph identifies those, but I >> could find them in the dump at least. >> >> I have no idea why the objects disappeared. I ran a Spark job over all >> buckets, read 1 byte of every object and recorded errors. Of the 78 >> buckets, two are missing objects. One bucket is missing one object, >> the other 15. So, luckily, the incidence is still quite low, but the >> problem seems to be expanding slowly. >> >> >> On 10/11/2020 23:46, Rafael Lopez wrote: >>> Hi Janek, >>> >>> What you said sounds right - an S3 single part obj won't have an S3 >>> multipart string as part of the prefix. S3 multipart string looks >>> like "2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme". >>> >>> From memory, single part S3 objects that don't fit in a single rados >>> object are assigned a random prefix that has nothing to do with >>> the object name, and the rados tail/data objects (not the head >>> object) have that prefix. >>> As per your working example, the prefix for that would be >>> '.8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh'. So there would be (239) "shadow" >>> objects with names containing that prefix, and if you add up the >>> sizes it should be the size of your S3 object. >>> >>> You should look at working and non working examples of both single >>> and multipart S3 objects, as they are probably all a bit different >>> when you look in rados. >>> >>> I agree it is a serious issue, because once objects are no longer in >>> rados, they cannot be recovered. If it was a case that there was a >>> link broken or rados objects renamed, then we could work to >>> recover...but as far as I can tell, it looks like stuff is just >>> vanishing from rados. The only explanation I can think of is some >>> (rgw or rados) background process is incorrectly doing something with >>> these objects (eg. renaming/deleting). I had thought perhaps it was a >>> bug with the rgw garbage collector..but that is pure speculation. >>> >>> Once you can articulate the problem, I'd recommend logging a bug >>> tracker upstream. >>> >>> >>> On Wed, 11 Nov 2020 at 06:33, Janek Bevendorff >>> <janek.bevendorff(a)uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de> >>> <mailto:janek.bevendorff@uni-weimar.de <mailto:janek.bevendorff@uni-weimar.de>>> wrote: >>> >>> Here's something else I noticed: when I stat objects that work >>> via radosgw-admin, the stat info contains a "begin_iter" JSON >>> object with RADOS key info like this >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110924034843-crawl420/WIDE-20110924065228-02544.warc.gz", >>> "instance": "", >>> "ns": "" >>> } >>> >>> >>> and then "end_iter" with key info like this: >>> >>> >>> "key": { >>> "name": >>> ".8naRUHSG2zfgjqmwLnTPvvY1m6DZsgh_239", >>> "instance": "", >>> "ns": "shadow" >>> } >>> >>> However, when I check the broken 0-byte object, the "begin_iter" >>> and "end_iter" keys look like this: >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.1", >>> "instance": "", >>> "ns": "multipart" >>> } >>> >>> [...] >>> >>> >>> "key": { >>> "name": >>> "29/items/WIDE-20110903143858-crawl428/WIDE-20110903143858-01166.warc.gz.2~m5Y42lPMIeis5qgJAZJfuNnzOKd7lme.19", >>> "instance": "", >>> "ns": "multipart" >>> } >>> >>> So, it's the full name plus a suffix and the namespace is >>> multipart, not shadow (or empty). This in itself may just be an >>> artefact of whether the object was uploaded in one go or as a >>> multipart object, but the second difference is that I cannot find >>> any of the multipart objects in my pool's object name dump. I >>> can, however, find the shadow RADOS object of the intact S3 object. >>> >>> >>> >>> >>> -- >>> *Rafael Lopez* >>> Devops Systems Engineer >>> Monash University eResearch Centre >>> >>> T: +61 3 9905 9118 <tel:%2B61%203%209905%209118> >>> E: rafael.lopez(a)monash.edu <mailto:rafael.lopez@monash.edu> >>> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

6 22

Why you might want packages not containers for Ceph deployments
by Matthew Vernon 25 Mar '24

25 Mar '24

Hi, In the discussion after the Ceph Month talks yesterday, there was a bit of chat about cephadm / containers / packages. IIRC, Sage observed that a common reason in the recent user survey for not using cephadm was that it only worked on containerised deployments. I think he then went on to say that he hadn't heard any compelling reasons why not to use containers, and suggested that resistance was essentially a user education question[0]. I'd like to suggest, briefly, that: * containerised deployments are more complex to manage, and this is not simply a matter of familiarity * reducing the complexity of systems makes admins' lives easier * the trade-off of the pros and cons of containers vs packages is not obvious, and will depend on deployment needs * Ceph users will benefit from both approaches being supported into the future We make extensive use of containers at Sanger, particularly for scientific workflows, and also for bundling some web apps (e.g. Grafana). We've also looked at a number of container runtimes (Docker, singularity, charliecloud). They do have advantages - it's easy to distribute a complex userland in a way that will run on (almost) any target distribution; rapid "cloud" deployment; some separation (via namespaces) of network/users/processes. For what I think of as a 'boring' Ceph deploy (i.e. install on a set of dedicated hardware and then run for a long time), I'm not sure any of these benefits are particularly relevant and/or compelling - Ceph upstream produce Ubuntu .debs and Canonical (via their Ubuntu Cloud Archive) provide .debs of a couple of different Ceph releases per Ubuntu LTS - meaning we can easily separate out OS upgrade from Ceph upgrade. And upgrading the Ceph packages _doesn't_ restart the daemons[1], meaning that we maintain control over restart order during an upgrade. And while we might briefly install packages from a PPA or similar to test a bugfix, we roll those (test-)cluster-wide, rather than trying to run a mixed set of versions on a single cluster - and I understand this single-version approach is best practice. Deployment via containers does bring complexity; some examples we've found at Sanger (not all Ceph-related, which we run from packages): * you now have 2 process supervision points - dockerd and systemd * docker updates (via distribution unattended-upgrades) have an unfortunate habit of rudely restarting everything * docker squats on a chunk of RFC 1918 space (and telling it not to can be a bore), which coincides with our internal network... * there is more friction if you need to look inside containers (particularly if you have a lot running on a host and are trying to find out what's going on) * you typically need to be root to build docker containers (unlike packages) * we already have package deployment infrastructure (which we'll need regardless of deployment choice) We also currently use systemd overrides to tweak some of the Ceph units (e.g. to do some network sanity checks before bringing up an OSD), and have some tools to pair OSD / journal / LVM / disk device up; I think these would be more fiddly in a containerised deployment. I'd accept that fixing these might just be a SMOP[2] on our part. Now none of this is show-stopping, and I am most definitely not saying "don't ship containers". But I think there is added complexity to your deployment from going the containers route, and that is not simply a "learn how to use containers" learning curve. I do think it is reasonable for an admin to want to reduce the complexity of what they're dealing with - after all, much of my job is trying to automate or simplify the management of complex systems! I can see from a software maintainer's point of view that just building one container and shipping it everywhere is easier than building packages for a number of different distributions (one of my other hats is a Debian developer, and I have a bunch of machinery for doing this sort of thing). But it would be a bit unfortunate if the general thrust of "let's make Ceph easier to set up and manage" was somewhat derailed with "you must use containers, even if they make your life harder". I'm not going to criticise anyone who decides to use a container-based deployment (and I'm sure there are plenty of setups where it's an obvious win), but if I were advising someone who wanted to set up and use a 'boring' Ceph cluster for the medium term, I'd still advise on using packages. I don't think this makes me a luddite :) Regards, and apologies for the wall of text, Matthew [0] I think that's a fair summary! [1] This hasn't always been true... [2] Simple (sic.) Matter of Programming -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.

26 65

rgw multisite sync not syncing data, error: RGW-SYNC:data:init_data_sync_status: ERROR: failed to read remote data log shards
by Christian Rohmann 20 Jul '23

20 Jul '23

Hey ceph-users, I setup a multisite sync between two freshly setup Octopus clusters. In the first cluster I created a bucket with some data just to test the replication of actual data later. I then followed the instructions on https://docs.ceph.com/en/octopus/radosgw/multisite/#migrating-a-single-site… to add a second zone. Things went well and both zones are now happily reaching each other and the API endpoints are talking. Also the metadata is in sync already - both sides are happy and I can see bucket listings and users are "in sync": > # radosgw-admin sync status > realm 13d1b8cb-dc76-4aed-8578-2ce5d3d010e8 (obst) > zonegroup 17a06c15-2665-484e-8c61-cbbb806e11d2 (obst-fra) > zone 6d2c1275-527e-432f-a57a-9614930deb61 (obst-rgn) > metadata sync no sync (zone is master) > data sync source: c07447eb-f93a-4d8f-bf7a-e52fade399f3 (obst-az1) > init > full sync: 128/128 shards > full sync: 0 buckets to sync > incremental sync: 0/128 shards > data is behind on 128 shards > behind shards: [0...127] > and on the other side ... > # radosgw-admin sync status > realm 13d1b8cb-dc76-4aed-8578-2ce5d3d010e8 (obst) > zonegroup 17a06c15-2665-484e-8c61-cbbb806e11d2 (obst-fra) > zone c07447eb-f93a-4d8f-bf7a-e52fade399f3 (obst-az1) > metadata sync syncing > full sync: 0/64 shards > incremental sync: 64/64 shards > metadata is caught up with master > data sync source: 6d2c1275-527e-432f-a57a-9614930deb61 (obst-rgn) > init > full sync: 128/128 shards > full sync: 0 buckets to sync > incremental sync: 0/128 shards > data is behind on 128 shards > behind shards: [0...127] > also the newly created buckets (read: their metadata) is synced. What is apparently not working in the sync of actual data. Upon startup the radosgw on the second site shows: > 2021-06-25T16:15:06.445+0000 7fe71eff5700 1 RGW-SYNC:meta: start > 2021-06-25T16:15:06.445+0000 7fe71eff5700 1 RGW-SYNC:meta: realm > epoch=2 period id=f4553d7c-5cc5-4759-9253-9a22b051e736 > 2021-06-25T16:15:11.525+0000 7fe71dff3700 0 > RGW-SYNC:data:sync:init_data_sync_status: ERROR: failed to read remote > data log shards > also when issuing # radosgw-admin data sync init --source-zone obst-rgn it throws > 2021-06-25T16:20:29.167+0000 7f87c2aec080 0 > RGW-SYNC:data:init_data_sync_status: ERROR: failed to read remote data > log shards Does anybody have any hints on where to look for what could be broken here? Thanks a bunch, Regards Christian

3 4

Small RGW objects and RADOS 64KB minimun size
by Loïc Dachary 31 May '23

31 May '23

Bonjour, Reading Karan's blog post about benchmarking the insertion of billions objects to Ceph via S3 / RGW[0] from last year, it reads: > we decided to lower bluestore_min_alloc_size_hdd to 18KB and re-test. As represented in chart-5, the object creation rate found to be notably reduced after lowering the bluestore_min_alloc_size_hdd parameter from 64KB (default) to 18KB. As such, for objects larger than the bluestore_min_alloc_size_hdd , the default values seems to be optimal, smaller objects further require more investigation if you intended to reduce bluestore_min_alloc_size_hdd parameter. There also is a mail thread dated 2018 on this topic as well, with the same conclusion although using RADOS directly and not RGW[3]. I read the RGW data layout page in the documentation[1] and concluded that by default every object inserted with S3 / RGW will indeed use at least 64kb. A pull request from last year[2] seems to confirm it and also suggests modifying bluestore_min_alloc_size_hdd has adverse side effects. That being said, I'm curious to know if people developed strategies to cope with this overhead. Someone mentioned packing objects together client side to make them larger. But maybe there are simpler ways to do the same? Cheers [0] https://www.redhat.com/en/blog/scaling-ceph-billion-objects-and-beyond [1] https://docs.ceph.com/en/latest/radosgw/layout/ [2] https://github.com/ceph/ceph/pull/32809 [3] https://www.spinics.net/lists/ceph-users/msg45755.html -- Loïc Dachary, Artisan Logiciel Libre

4 6

MDS crash on FAILED ceph_assert(cur->is_auth())
by Peter van Heusden 04 May '23

04 May '23

I am running Ceph 15.2.13 on CentOS 7.9.2009 and recently my MDS servers have started failing with the error message In function 'void Server::handle_client_open(MDRequestRef&)' thread 7f0ca9908700 time 2021-06-28T09:21:11.484768+0200 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/15.2.13/rpm/el7/BUILD/ceph-15.2.13/src/mds/Server.cc: 4149: FAILED ceph_assert(cur->is_auth()) Complete log is: https://gist.github.com/pvanheus/4da555a6de6b5fa5e46cbf74f5500fbd ceph status output is: # ceph status cluster: id: ed7b2c16-b053-45e2-a1fe-bf3474f90508 health: HEALTH_WARN 30 OSD(s) experiencing BlueFS spillover insufficient standby MDS daemons available 1 MDSs report slow requests 2 mgr modules have failed dependencies 4347046/326505282 objects misplaced (1.331%) 6 nearfull osd(s) 23 pgs not deep-scrubbed in time 23 pgs not scrubbed in time 8 pool(s) nearfull services: mon: 3 daemons, quorum ceph-mon1,ceph-mon2,ceph-mon3 (age 22m) mgr: ceph-mon1(active, since 11w), standbys: ceph-mon2, ceph-mon3 mds: SANBI_FS:2 {0=ceph-mon1=up:active(laggy or crashed),1=ceph-mon2=up:stopping} osd: 54 osds: 54 up (since 2w), 54 in (since 11w); 50 remapped pgs data: pools: 8 pools, 833 pgs objects: 42.37M objects, 89 TiB usage: 159 TiB used, 105 TiB / 264 TiB avail pgs: 4347046/326505282 objects misplaced (1.331%) 782 active+clean 49 active+clean+remapped 1 active+clean+scrubbing+deep 1 active+clean+remapped+scrubbing io: client: 29 KiB/s rd, 427 KiB/s wr, 37 op/s rd, 48 op/s wr When restarting a MDS it goes through states replace, reconnect, resolve and finally sets itself to active before this crash happens. Any advice on what to do? Thanks, Peter P.S. apologies if you received this email more than once - I have had some trouble figuring out the correct mailing list to use.

3 5

cephadm cluster move /var/lib/docker to separate device fails
by Karsten Nielsen 25 Mar '23

25 Mar '23

Hi, I have setup a ceph cluster with cephadm with docker backend. I want to move /var/lib/docker to a separate device to get better performance and less load on the OS device. I tried that by stopping docker copy the content of /var/lib/docker to the new device and mount the new device to /var/lib/docker. The other containers started as expected and continues to work and run as expected. But the ceph containers seems to be broken. I am not able to get them back in working state. I have tried to remove the host with `ceph orch host rm itcnchn-bb4067` and readd it but no effect. The strange thing is that 2 of 4 containers comes up as expected. ceph orch ps itcnchn-bb4067 NAME HOST STATUS REFRESHED AGE VERSION IMAGE NAME IMAGE ID CONTAINER ID crash.itcnchn-bb4067 itcnchn-bb4067 running (18h) 10m ago 4w 15.2.7 docker.io/ceph/ceph:v15 2bc420ddb175 2af28c4571cf mds.cephfs.itcnchn-bb4067.qzoshl itcnchn-bb4067 error 10m ago 4w <unknown> docker.io/ceph/ceph:v15 <unknown> <unknown> mon.itcnchn-bb4067 itcnchn-bb4067 error 10m ago 18h <unknown> docker.io/ceph/ceph:v15 <unknown> <unknown> rgw.ikea.dc9-1.itcnchn-bb4067.gtqedc itcnchn-bb4067 running (18h) 10m ago 4w 15.2.7 docker.io/ceph/ceph:v15 2bc420ddb175 00d000aec32b Docker logs from the active manager does not say much about what is wrong debug 2021-01-05T09:57:52.537+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring mds.cephfs.itcnchn-bb4067.qzoshl (unknown last config time)... debug 2021-01-05T09:57:52.541+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring daemon mds.cephfs.itcnchn-bb4067.qzoshl on itcnchn-bb4067 debug 2021-01-05T09:57:52.973+0000 7fdb64e88700 0 log_channel(cluster) log [DBG] : pgmap v347: 241 pgs: 241 active+clean; 18 GiB data, 50 GiB used, 52 TiB / 52 TiB avail; 18 KiB/s rd, 78 KiB/s wr, 24 op/s debug 2021-01-05T09:57:53.085+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring mon.itcnchn-bb4067 (unknown last config time)... debug 2021-01-05T09:57:53.085+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring daemon mon.itcnchn-bb4067 on itcnchn-bb4067 debug 2021-01-05T09:57:53.625+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring rgw.ikea.dc9-1.itcnchn-bb4067.gtqedc (unknown last config time)... debug 2021-01-05T09:57:53.629+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring daemon rgw.ikea.dc9-1.itcnchn-bb4067.gtqedc on itcnchn-bb4067 debug 2021-01-05T09:57:54.141+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring crash.itcnchn-bb4067 (unknown last config time)... debug 2021-01-05T09:57:54.141+0000 7fdb69691700 0 log_channel(cephadm) log [INF] : Reconfiguring daemon crash.itcnchn-bb4067 on itcnchn-bb4067 - Karsten

2 1

Stuck OSD service specification - can't remove
by David Orman 15 Mar '23

15 Mar '23

Has anybody run into a 'stuck' OSD service specification? I've tried to delete it, but it's stuck in 'deleting' state, and has been for quite some time (even prior to upgrade, on 15.2.x). This is on 16.2.3: NAME PORTS RUNNING REFRESHED AGE PLACEMENT osd.osd_spec 504/525 <deleting> 12m label:osd root@ceph01:/# ceph orch rm osd.osd_spec Removed service osd.osd_spec From active monitor: debug 2021-05-06T23:14:48.909+0000 7f17d310b700 0 log_channel(cephadm) log [INF] : Remove service osd.osd_spec Yet in ls, it's still there, same as above. --export on it: root@ceph01:/# ceph orch ls osd.osd_spec --export service_type: osd service_id: osd_spec service_name: osd.osd_spec placement: {} unmanaged: true spec: filter_logic: AND objectstore: bluestore We've tried --force, as well, with no luck. To be clear, the --export even prior to delete looks nothing like the actual service specification we're using, even after I re-apply it, so something seems 'bugged'. Here's the OSD specification we're applying: service_type: osd service_id: osd_spec placement: label: "osd" data_devices: rotational: 1 db_devices: rotational: 0 db_slots: 12 I would appreciate any insight into how to clear this up (without removing the actual OSDs, we're just wanting to apply the updated service specification - we used to use host placement rules and are switching to label-based). Thanks, David

3 4

kernel client osdc ops stuck and mds slow reqs
by Dan van der Ster 23 Feb '23

23 Feb '23

Hi all, We are quite regularly (a couple times per week) seeing: HEALTH_WARN 1 clients failing to respond to capability release; 1 MDSs report slow requests MDS_CLIENT_LATE_RELEASE 1 clients failing to respond to capability release mdshpc-be143(mds.0): Client hpc-be028.cern.ch: failing to respond to capability release client_id: 52919162 MDS_SLOW_REQUEST 1 MDSs report slow requests mdshpc-be143(mds.0): 1 slow requests are blocked > 30 secs Which is being caused by osdc ops stuck in a kernel client, e.g.: 10:57:18 root hpc-be028 /root → cat /sys/kernel/debug/ceph/4da6fd06-b069-49af-901f-c9513baabdbd.client52919162/osdc REQUESTS 9 homeless 0 46559317 osd243 3.ee6ffcdb 3.cdb [243,501,92]/243 [243,501,92]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a01.00000057 0x400014 1 read 46559322 osd243 3.ee6ffcdb 3.cdb [243,501,92]/243 [243,501,92]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a01.00000057 0x400014 1 read 46559323 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559341 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559342 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559345 osd243 3.969cc573 3.573 [243,330,226]/243 [243,330,226]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a56.00000056 0x400014 1 read 46559621 osd243 3.6313e8ef 3.8ef [243,330,521]/243 [243,330,521]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a45.0000007a 0x400014 1 read 46559629 osd243 3.b280c852 3.852 [243,113,539]/243 [243,113,539]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f09a3a.0000007f 0x400014 1 read 46559928 osd243 3.1ee7bab4 3.ab4 [243,332,94]/243 [243,332,94]/243 e678697 fsvolumens_355f485c-6319-4ffe-acd6-94a07f2a14b4/10003f099ff.0000073f 0x400024 1 write LINGER REQUESTS BACKOFFS We can unblock those requests by doing `ceph osd down osd.243` (or restarting osd.243). This is ceph v14.2.6 and the client kernel is el7 3.10.0-957.27.2.el7.x86_64. Are there a better way to debug this? Best Regards, Dan

4 12

OSD slow ops warning not clearing after OSD down
by Frank Schilder 17 Jan '23

17 Jan '23

Dear cephers, I have a strange problem. An OSD went down and recovery finished. For some reason, I have a slow ops warning for the failed OSD stuck in the system: health: HEALTH_WARN 430 slow ops, oldest one blocked for 36 sec, osd.580 has slow ops The OSD is auto-out: | 580 | ceph-22 | 0 | 0 | 0 | 0 | 0 | 0 | autoout,exists | It is probably a warning dating back to just before the fail. How can I clear it? Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

4 7

[ERR] OSD_SCRUB_ERRORS: 2 scrub errors
by Szabo, Istvan (Agoda) 11 Jan '23

11 Jan '23

Hi, I’m continuously getting scrub errors in my index pool and log pool that I need to repair always. HEALTH_ERR 2 scrub errors; Possible data damage: 1 pg inconsistent [ERR] OSD_SCRUB_ERRORS: 2 scrub errors [ERR] PG_DAMAGED: Possible data damage: 1 pg inconsistent pg 20.19 is active+clean+inconsistent, acting [39,41,37] Why is this? I have no cue at all, no log entry no anything ☹ ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

6 9

Upgrade tips from Luminous to Nautilus?
by Mark Schouten 08 Jul '21

08 Jul '21

Hi, We've done our fair share of Ceph cluster upgrades since Hammer, and have not seen much problems with them. I'm now at the point that I have to upgrade a rather large cluster running Luminous and I would like to hear from other users if they have experiences with issues I can expect so that I can anticipate on them beforehand. As said, the cluster is running Luminous (12.2.13) and has the following services active: services: mon: 3 daemons, quorum osdnode01,osdnode02,osdnode04 mgr: osdnode01(active), standbys: osdnode02, osdnode03 mds: pmrb-3/3/3 up {0=osdnode06=up:active,1=osdnode08=up:active,2=osdnode07=up:active}, 1 up:standby osd: 116 osds: 116 up, 116 in; rgw: 3 daemons active Of the OSD's, we have 11 SSD's and 105 HDD. The capacity of the cluster is 1.01PiB. We have 2 active crush-rules on 18 pools. All pools have a size of 3 there is a total of 5760 pgs. { "rule_id": 1, "rule_name": "hdd-data", "ruleset": 1, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -10, "item_name": "default~hdd" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 2, "rule_name": "ssd-data", "ruleset": 2, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -21, "item_name": "default~ssd" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] } rbd -> crush_rule: hdd-data .rgw.root -> crush_rule: hdd-data default.rgw.control -> crush_rule: hdd-data default.rgw.data.root -> crush_rule: ssd-data default.rgw.gc -> crush_rule: ssd-data default.rgw.log -> crush_rule: ssd-data default.rgw.users.uid -> crush_rule: hdd-data default.rgw.usage -> crush_rule: ssd-data default.rgw.users.email -> crush_rule: hdd-data default.rgw.users.keys -> crush_rule: hdd-data default.rgw.meta -> crush_rule: hdd-data default.rgw.buckets.index -> crush_rule: ssd-data default.rgw.buckets.data -> crush_rule: hdd-data default.rgw.users.swift -> crush_rule: hdd-data default.rgw.buckets.non-ec -> crush_rule: ssd-data DB0475 -> crush_rule: hdd-data cephfs_pmrb_data -> crush_rule: hdd-data cephfs_pmrb_metadata -> crush_rule: ssd-data All but four clients are running Luminous, the four are running Jewel (that needs upgrading before proceeding with this upgrade). So, normally, I would 'just' upgrade all Ceph packages on the monitor-nodes and restart mons and then mgrs. After that, I would upgrade all Ceph packages on the OSD nodes and restart all the OSD's. Then, after that, the MDSes and RGWs. Restarting the OSD's will probably take a while. If anyone has a hint on what I should expect to cause some extra load or waiting time, that would be great. Obviously, we have read https://ceph.com/releases/v14-2-0-nautilus-released/ , but I'm looking for real world experiences. Thanks! -- Mark Schouten | Tuxis B.V. KvK: 74698818 | http://www.tuxis.nl/ T: +31 318 200208 | info(a)tuxis.nl

4 11

Create and listing topics with AWS4 fails
by Daniel Iwan 07 Jul '21

07 Jul '21

Hi I'm on Pacific 16.2.1 Documentation states that topic operations should be created using REST with application/x-www-form-urlencoded See https://docs.ceph.com/en/latest/radosgw/notifications/#topics However when attempting to create one using Postman (auth v4) operation fails. <?xml version="1.0" encoding="UTF-8"?> <Error> <Code>NotImplemented</Code> <RequestId>tx00000000000000000004e-0060d207d0-df5c88-abcf</RequestId> <HostId>df5c88-abc-default</HostId> </Error>. The same error is for listing topics. See log attached I think similar issues were reported on the threads - https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/23BZW2Q3TCU… - https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/WMKEYKTE5NH… - https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/2G4YWOUMVE2… I think the handling procedure is in https://github.com/ceph/ceph/blob/master/src/rgw/rgw_rest_s3.cc get_auth_data_v4()which does not have RGW_OP_PUBSUB_TOPIC_CREATE or similar topic-related operations. If that's the case this makes it more difficult to integrate those operations as majority of the sw uses AWSV4 authentication Can anyone confirm if the doc is wrong, auth V3 needs to be used etc? Is there a tracking issue for this already? Regards Daniel

2 4

ceph df (octopus) shows USED is 7 times higher than STORED in erasure coded pool
by Arkadiy Kulev 06 Jul '21

06 Jul '21

The pool *default.rgw.buckets.data* has *501 GiB* stored, but USED shows *3.5 TiB *(7 times higher!)*:* root@ceph-01:~# ceph df --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 196 TiB 193 TiB 3.5 TiB 3.6 TiB 1.85 TOTAL 196 TiB 193 TiB 3.5 TiB 3.6 TiB 1.85 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL device_health_metrics 1 1 19 KiB 12 56 KiB 0 61 TiB .rgw.root 2 32 2.6 KiB 6 1.1 MiB 0 61 TiB default.rgw.log 3 32 168 KiB 210 13 MiB 0 61 TiB default.rgw.control 4 32 0 B 8 0 B 0 61 TiB default.rgw.meta 5 8 4.8 KiB 11 1.9 MiB 0 61 TiB default.rgw.buckets.index 6 8 1.6 GiB 211 4.7 GiB 0 61 TiB default.rgw.buckets.data 10 128 501 GiB 5.36M 3.5 TiB 1.90 110 TiB The *default.rgw.buckets.data* pool is using erasure coding: root@ceph-01:~# ceph osd erasure-code-profile get EC_RGW_HOST crush-device-class=hdd crush-failure-domain=host crush-root=default jerasure-per-chunk-alignment=false k=6 m=4 plugin=jerasure technique=reed_sol_van w=8 If anyone could help explain why it's using up 7 times more space, it would help a lot. Versioning is disabled. ceph version 15.2.13 (octopus stable). Sincerely, Ark.

5 9

Spurious Read Errors: 0x6706be76
by Jay Sullivan 06 Jul '21

06 Jul '21

In the week since upgrading one of our clusters from Nautilus 14.2.21 to Pacific 16.2.4 I've seen four spurious read errors that always have the same bad checksum of 0x6706be76. I've never seen this in any of our clusters before. Here's an example of what I'm seeing in the logs: ceph-osd.132.log:2021-06-20T22:53:20.584-0400 7fde2e4fc700 -1 bluestore(/var/lib/ceph/osd/ceph-132) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x0, got 0x6706be76, expected 0xee74a56a, device location [0x18c81b40000~1000], logical extent 0x200000~1000, object #29:2d8210bf:::rbd_data.94f4232ae8944a.0000000000026c57:head# I'm not seeing any indication of inconsistent PGs, only the spurious read error. I don't see an explicit indication of a retry in the logs following the above message. Bluestore code to retry three times was introduced in 2018 following a similar issue with the same checksum: https://tracker.ceph.com/issues/22464 Here's an example of what my health detail looks like: HEALTH_WARN 1 OSD(s) have spurious read errors [WRN] BLUESTORE_SPURIOUS_READ_ERRORS: 1 OSD(s) have spurious read errors osd.117 reads with retries: 1 I followed this (unresolved) thread, too: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/DRBVFQLZ5ZY… I do have swap enabled, but I don't think memory pressure is an issue with 30GB available out of 96GB (and no sign I've been close to summoning the OOMkiller). The OSDs that have thrown the cluster into HEALTH_WARN with the spurious read errors are busy 12TB rotational HDDs and I _think_ it's only happening during a deep scrub. We're on Ubuntu 18.04; uname: 5.4.0-74-generic #83~18.04.1-Ubuntu SMP Tue May 11 16:01:00 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux. Does Pacific retry three times on a spurious read error? Would I see an indication of a retry in the logs? Thanks! ~Jay

3 4

Re: XFS on RBD on EC painfully slow
by Sebastian Knust 06 Jul '21

06 Jul '21

Hi Reed, To add to this command by Weiwen: On 28.05.21 13:03, 胡玮文 wrote: > Have you tried just start multiple rsync process simultaneously to transfer different directories? Distributed system like ceph often benefits from more parallelism. When I migrated from XFS on iSCSI (legacy system, no Ceph) to CephFS a few months ago, I used msrsync [1] and was quite happy with the speed. For your use case, I would start with -p 12 but might experiment with up to -p 24 (as you only have 6C/12T in your CPU). With many small files, you also might want to increase -s from the default 1000. Note that msrsync does not work with the --delete rsync flag. As I was syncing a live system, I ended up with this workflow: - Initial sync with msrsync (something like ./msrsync -p 12 --progress --stats --rsync "-aS --numeric-ids" ...) - Second sync with msrsync (to sync changes during the first sync) - Take old storage off-line for users / read-only - Final rsync with --delete (i.e. rsync -aS --numeric-ids --delete ...) - Mount cephfs at location of old storage, adjust /etc/exports with fsid entries where necessary, turn system back on-line / read-write Cheers Sebastian [1] https://github.com/jbd/msrsync

3 2

Pacific: RadosGW crashing on multipart uploads.
by Chu, Vincent 01 Jul '21

01 Jul '21

Hi, I'm running into an issue with RadosGW where multipart uploads crash, but only on buckets with a hyphen, period or underscore in the bucket name and with a bucket policy applied. We've tested this in pacific 16.2.3 and pacific 16.2.4. Anyone run into this before? ubuntu@ubuntu:~/ubuntu$ aws --endpoint http://placeholder.com:7480 s3 cp ubuntu.iso s3://bucket.test upload failed: ./ubuntu.iso to s3://bucket.test/ubuntu.iso Connection was closed before we received a valid response from endpoint URL: "http://placeholder.com:7480/bucket.test/ubuntu.iso?uploads". Here is the crash log. -12> 2021-06-29T20:44:10.940+0000 7fae1f4ec700 1 ====== starting new request req=0x7fadf8998620 ===== -11> 2021-06-29T20:44:10.940+0000 7fae1f4ec700 2 req 2403 0.000000000s initializing for trans_id = tx000000000000000000963-0060db861a-17e77ee-default -10> 2021-06-29T20:44:10.940+0000 7fae1f4ec700 2 req 2403 0.000000000s getting op 4 -9> 2021-06-29T20:44:10.940+0000 7fae1f4ec700 2 req 2403 0.000000000s s3:init_multipart verifying requester -8> 2021-06-29T20:44:10.948+0000 7fae1f4ec700 2 req 2403 0.008000608s s3:init_multipart normalizing buckets and tenants -7> 2021-06-29T20:44:10.948+0000 7fae1f4ec700 2 req 2403 0.008000608s s3:init_multipart init permissions -6> 2021-06-29T20:44:10.954+0000 7faedf66c700 0 Supplied principal is discarded: arn:aws:iam::default:user -5> 2021-06-29T20:44:10.954+0000 7faedf66c700 2 req 2403 0.014001064s s3:init_multipart recalculating target -4> 2021-06-29T20:44:10.954+0000 7faedf66c700 2 req 2403 0.014001064s s3:init_multipart reading permissions -3> 2021-06-29T20:44:10.954+0000 7faedf66c700 2 req 2403 0.014001064s s3:init_multipart init op -2> 2021-06-29T20:44:10.954+0000 7faedf66c700 2 req 2403 0.014001064s s3:init_multipart verifying op mask -1> 2021-06-29T20:44:10.955+0000 7faedf66c700 2 req 2403 0.015001140s s3:init_multipart verifying op permissions 0> 2021-06-29T20:44:10.964+0000 7faedf66c700 -1 *** Caught signal (Segmentation fault) ** in thread 7faedf66c700 thread_name:radosgw ceph version 16.2.3 (381b476cb3900f9a92eb95d03b4850b953cfd79a) pacific (stable) 1: /lib64/libpthread.so.0(+0x12b20) [0x7faf2dd05b20] 2: (rgw_bucket::rgw_bucket(rgw_bucket const&)+0x23) [0x7faf38b4d083] 3: (rgw::sal::RGWObject::get_obj() const+0x20) [0x7faf38b7bcf0] 4: (RGWInitMultipart::verify_permission(optional_yield)+0x6c) [0x7faf38e6608c] 5: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, optional_yield, bool)+0x86a) [0x7faf38b2db1a] 6: (process_request(rgw::sal::RGWRadosStore*, RGWREST*, RGWRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*, int*)+0x26dd) [0x7faf38b3232d] 7: /lib64/libradosgw.so.2(+0x4a1c0b) [0x7faf38a83c0b] 8: /lib64/libradosgw.so.2(+0x4a36a4) [0x7faf38a856a4] 9: /lib64/libradosgw.so.2(+0x4a390e) [0x7faf38a8590e] 10: make_fcontext() NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. -- Vincent Chu A-4: Advanced Research in Cyber Systems Los Alamos National Laboratory

2 1

Having issues to start more than 24 OSDs per host
by Jan.Jansen＠gdata.de 01 Jul '21

01 Jul '21

Hello We did try to use Cephadm with Podman to start 44 OSDs per host which consistently stop after adding 24 OSDs per host. We did look into the cephadm.log on the problematic host and saw that the command `cephadm ceph-volume lvm list --format json` did stuck. We were the output of the command wasn't complete. Therefore, we tried to use compacted JSON and we could increase the number to 36 OSDs per host. If you need more information just ask. Podman version: 3.2.1 Ceph version: 16.2.4 OS version: Suse Leap 15.3 Greetings, Jan

3 3

cephfs forward scrubbing docs
by Dan van der Ster 01 Jul '21

01 Jul '21

Hi, Today while debugging something we had a few questions that might lead to improving the cephfs forward scrub docs: https://docs.ceph.com/en/latest/cephfs/scrub/ tldr: 1. Should we document which sorts of issues that the forward scrub is able to fix? 2. Can we make it more visible (in docs) that scrubbing is not supported with multi-mds? 3. Isn't the new `ceph -s` scrub task status misleading with multi-mds? Details here: 1) We found a CephFS directory with a number of zero sized files: # ls -l ... -rw-r--r-- 1 1001890000 1001890000 0 Nov 3 11:58 upload_fc501199e3e7abe6b574101cf34aeefb.png -rw-r--r-- 1 1001890000 1001890000 0 Nov 3 12:23 upload_fce4f55348185fefa0abdd8d11095ba8.gif -rw-r--r-- 1 1001890000 1001890000 0 Nov 3 11:54 upload_fd95b8358851f0dac22fb775046a6163.png ... The user claims that those files were non-zero sized last week. The sequence of zero sized files includes *all* files written between Nov 2 and 9. The user claims that his client was running out of memory, but this is now fixed. So I suspect that his ceph client (kernel 3.10.0-1127.19.1.el7.x86_64) was not behaving well. Anyway, I noticed that even though the dentries list 0 bytes, the underlying rados objects have data, and the data looks good. E.g. # rados get -p cephfs_data 200212e68b5.00000000 --namespace=xxx 200212e68b5.00000000 # file 200212e68b5.00000000 200212e68b5.00000000: PNG image data, 960 x 815, 8-bit/color RGBA, non-interlaced So I managed to recover the files doing something like this (using an input file mapping inode to filename) [see PS 0]. But I'm wondering if a forward scrub is able to fix this sort of problem directly? Should we document which sorts of issues that the forward scrub is able to fix? I anyway tried to scrub it, which led to: # ceph tell mds.cephflax-mds-xxx scrub start /volumes/_nogroup/xxx recursive repair Scrub is not currently supported for multiple active MDS. Please reduce max_mds to 1 and then scrub. So ... 2) Shouldn't we update the doc to mention loud and clear that scrub is not currently supported for multiple active MDS? 3) I was somehow surprised by this, because I had thought that the new `ceph -s` multi-mds scrub status implied that multi-mds scrubbing was now working: task status: scrub status: mds.x: idle mds.y: idle mds.z: idle Is it worth reporting this task status for cephfs if we can't even scrub them? Thanks!! Dan [0] mkdir -p recovered while read -r a b; do for i in {0..9} do echo "rados stat --cluster=flax --pool=cephfs_data --namespace=xxx" $(printf "%x" $a).0000000$i "&&" "rados get --cluster=flax --pool=cephfs_data --namespace=xxx" $(printf "%x" $a).0000000$i $(printf "%x" $a).0000000$i done echo cat $(printf "%x" $a).* ">" $(printf "%x" $a) echo mv $(printf "%x" $a) recovered/$b done < inones_fnames.txt

2 2

Semantics of cephfs-mirror
by Manuel Holtgrewe 01 Jul '21

01 Jul '21

Dear all, I'm sorry if I'm asking for the obvious or missing a previous discussion of this but I could not find the answer to my question online. I'd be happy to be pointed to the right direction only. The cephfs-mirror tool in pacific looks extremely promising. How does it work exactly? Is it based on files and (recursive) ctime or rather based on object information? Does it handle incremental changes (only) between snapshots? There is an issue related to this that mentions recursive ctime. But that would mean that users could "rsync -a" data to the file system and this would not get synchronized. I have good experience with ZFS which is able to identify changes between two snapshots A and B and then only transfer these changes (using a sub-file level, on the ZFS equivalent of blocks to my understanding) to another server with the same file system that is in the exact state as snapshot A. Does cephfs-mirror work the same? Best wishes, Manuel

2 1

Re: upgrading from Nautilus on CentOS7 to Octopus on Ubuntu 20.04.2
by Radoslav Milanov 30 Jun '21

30 Jun '21

Sounds exactly like testing to me... On 30.6.2021 г. 17:46 ч., Teoman Onay wrote: > What do you mean by different? > > RHEL is a supported product while CentOS is not. You get bug/security > fixes sooner on RHEL than on CentOS as depending on their severity > level they are released during the z-stream releases. > > Stream is one minor release ahead than RHEL which means it already > contains part of the fixes which will be released a few months later > in RHEL. It could be considered even more stable as it already > contains part of the fixes. > > > > > On Wed, 30 Jun 2021, 15:15 Radoslav Milanov, > <radoslav.milanov(a)gmail.com <mailto:radoslav.milanov@gmail.com>> wrote: > > If stream is so great why is RHEL different ? > > On 30.6.2021 г. 03:49 ч., Teoman Onay wrote: > >> > >>> For similar reasons, CentOS 8 stream, as opposed to every > other CentOS > >> released before, is very experimental. I would never go in > production with > >> CentOS 8 stream. > >> > >> > > Experimental?? Looks like you still don't understand what CentOS > stream is. > > If you have some time just read this: > > > https://www.linkedin.com/pulse/why-you-should-have-already-been-centos-stre… > <https://www.linkedin.com/pulse/why-you-should-have-already-been-centos-stre…> > > > > He summarized quite well what CentOS stream is. > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > <mailto:ceph-users@ceph.io> > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > <mailto:ceph-users-leave@ceph.io> >

1 0

v14.2.22 Nautilus released
by David Galloway 30 Jun '21

30 Jun '21

We're happy to announce the 22nd and likely final backport release in the Nautilus series. Ultimately, we recommend all users upgrade to newer Ceph releases. For a detailed release notes with links & changelog please refer to the official blog entry at https://ceph.io/en/news/blog/2021/v14-2-22-nautilus-released Notable Changes --------------- * This release sets `bluefs_buffered_io` to true by default to improve performance for metadata heavy workloads. Enabling this option has been reported to occasionally cause excessive kernel swapping under certain workloads. Currently, the most consistent performing combination is to enable bluefs_buffered_io and disable system level swap. * The default value of `bluestore_cache_trim_max_skip_pinned` has been increased to 1000 to control memory growth due to onodes. * Several other bug fixes in BlueStore, including a fix for an unexpected ENOSPC bug in Avl/Hybrid allocators. * The trimming logic in the monitor has been made dynamic, with the introduction of `paxos_service_trim_max_multiplier`, a factor by which `paxos_service_trim_max` is multiplied to make trimming faster, when required. Setting it to 0 disables the upper bound check for trimming and makes the monitors trim at the maximum rate. * A `--max <n>` option is available with the `osd ok-to-stop` command to provide up to N OSDs that can be stopped together without making PGs unavailable. * OSD: the option `osd_fast_shutdown_notify_mon` has been introduced to allow the OSD to notify the monitor it is shutting down even if `osd_fast_shutdown` is enabled. This helps with the monitor logs on larger clusters, that may get many 'osd.X reported immediately failed by osd.Y' messages, and confuse tools. * A long-standing bug that prevented 32-bit and 64-bit client/server interoperability under msgr v2 has been fixed. In particular, mixing armv7l (armhf) and x86_64 or aarch64 servers in the same cluster now works. Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at https://download.ceph.com/tarballs/ceph-14.2.22.tar.gz * For packages, see https://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: ca74598065096e6fcbd8433c8779a2be0c889351

1 0

Ceph connect to openstack
by Szabo, Istvan (Agoda) 30 Jun '21

30 Jun '21

Hi, Is there any proper documentation how to connect ceph with openstack? ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

3 3

Unhandled exception from module 'devicehealth' while running on mgr.al111: 'NoneType' object has no attribute 'get'
by Robert Sander 30 Jun '21

30 Jun '21

Hi, ceph -s reports an error in the ceph-mgr process. Looking at the logfile I see: Jun 30 16:07:09 al111 bash[171790]: debug 2021-06-30T14:07:09.939+0000 7f2a31d64700 -1 log_channel(cluster) log [ERR] : Unhandled exception from module 'devicehealth' while running on mgr.al111: 'NoneType' object has no attribute 'get' Jun 30 16:07:09 al111 bash[171790]: debug 2021-06-30T14:07:09.939+0000 7f2a31d64700 -1 devicehealth.serve: Jun 30 16:07:09 al111 bash[171790]: debug 2021-06-30T14:07:09.939+0000 7f2a31d64700 -1 Traceback (most recent call last): Jun 30 16:07:09 al111 bash[171790]: File "/usr/share/ceph/mgr/devicehealth/module.py", line 330, in serve Jun 30 16:07:09 al111 bash[171790]: self.scrape_all() Jun 30 16:07:09 al111 bash[171790]: File "/usr/share/ceph/mgr/devicehealth/module.py", line 390, in scrape_all Jun 30 16:07:09 al111 bash[171790]: self.put_device_metrics(ioctx, device, data) Jun 30 16:07:09 al111 bash[171790]: File "/usr/share/ceph/mgr/devicehealth/module.py", line 477, in put_device_metrics Jun 30 16:07:09 al111 bash[171790]: wear_level = get_ata_wear_level(data) Jun 30 16:07:09 al111 bash[171790]: File "/usr/share/ceph/mgr/devicehealth/module.py", line 33, in get_ata_wear_level Jun 30 16:07:09 al111 bash[171790]: if page.get("number") != 7: Jun 30 16:07:09 al111 bash[171790]: AttributeError: 'NoneType' object has no attribute 'get' This is a containerized Ceph 16.4.2. What is happening here? Regards -- Robert Sander Heinlein Support GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 93818 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

1 0

Ceph DB
by Szabo, Istvan (Agoda) 30 Jun '21

30 Jun '21

Hi, If I set in a running cluster the rgw_bucket_default_quota_max_objects with the ceph config set (mgr/mon not sure which to make it global) rgw_bucket_default_quota_max_objects 1000000 will it overwrite the existing special buckets where I've already set higher values? Or keep in untouched? Ty ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

1 0

bluestore_min_alloc_size sizing
by Arkadiy Kulev 30 Jun '21

30 Jun '21

I know that this might be a dumb question, but asking just for future googlers' sake. If *bluestore_min_alloc_size* is set to *64KB*, will a *70KB* object be stored as *128KB* or *70KB*? Sincerely, Ark.

2 1

Re: upgrading from Nautilus on CentOS7 to Octopus on Ubuntu 20.04.2
by Marc 30 Jun '21

30 Jun '21

> I’ve been running a staging Ceph environment on CentOS 7/Nautilus for quite a while now. Because of many good reasons that you can probably guess, I am currently trying to move this staging environment to Octopus on Ubuntu 20.04.2. > > What made you decide to chose Ubuntu and not this rocky Linux or centos8 stream?

6 6

Re: Pacific: RadosGW crashing on multipart uploads.
by Chris Palmer 30 Jun '21

30 Jun '21

Hi Vincent It may be Bug 50556 <https://tracker.ceph.com/issues/50556>. I am having this problem, although I don't think that characters in the bucket name is relevant. Backport 51001 <https://tracker.ceph.com/issues/51001>has just been updated so looks as though it will be in 16.2.5. At a glance your symptoms sound similar but I'm not sure if the crash info is the same. Regards, Chris On 29/06/2021 22:35, Chu, Vincent wrote: > Hi, I'm running into an issue with RadosGW where multipart uploads crash, but only on buckets with a hyphen, period or underscore in the bucket name and with a bucket policy applied. We've tested this in pacific 16.2.3 and pacific 16.2.4. > > > Anyone run into this before? > > > ubuntu@ubuntu:~/ubuntu$ aws --endpointhttp://placeholder.com:7480 s3 cp ubuntu.iso s3://bucket.test > > upload failed: ./ubuntu.iso to s3://bucket.test/ubuntu.iso Connection was closed before we received a valid response from endpoint URL:"http://placeholder.com:7480/bucket.test/ubuntu.iso?uploads". > > > > Here is the crash log. > > -12> 2021-06-29T20:44:10.940+0000 7fae1f4ec700 1 ====== starting new request req=0x7fadf8998620 ===== > -11> 2021-06-29T20:44:10.940+0000 7fae1f4ec700 2 req 2403 0.000000000s initializing for trans_id = tx000000000000000000963-0060db861a-17e77ee-default > -10> 2021-06-29T20:44:10.940+0000 7fae1f4ec700 2 req 2403 0.000000000s getting op 4 > -9> 2021-06-29T20:44:10.940+0000 7fae1f4ec700 2 req 2403 0.000000000s s3:init_multipart verifying requester > -8> 2021-06-29T20:44:10.948+0000 7fae1f4ec700 2 req 2403 0.008000608s s3:init_multipart normalizing buckets and tenants > -7> 2021-06-29T20:44:10.948+0000 7fae1f4ec700 2 req 2403 0.008000608s s3:init_multipart init permissions > -6> 2021-06-29T20:44:10.954+0000 7faedf66c700 0 Supplied principal is discarded: arn:aws:iam::default:user > -5> 2021-06-29T20:44:10.954+0000 7faedf66c700 2 req 2403 0.014001064s s3:init_multipart recalculating target > -4> 2021-06-29T20:44:10.954+0000 7faedf66c700 2 req 2403 0.014001064s s3:init_multipart reading permissions > -3> 2021-06-29T20:44:10.954+0000 7faedf66c700 2 req 2403 0.014001064s s3:init_multipart init op > -2> 2021-06-29T20:44:10.954+0000 7faedf66c700 2 req 2403 0.014001064s s3:init_multipart verifying op mask > -1> 2021-06-29T20:44:10.955+0000 7faedf66c700 2 req 2403 0.015001140s s3:init_multipart verifying op permissions > 0> 2021-06-29T20:44:10.964+0000 7faedf66c700 -1 *** Caught signal (Segmentation fault) ** > in thread 7faedf66c700 thread_name:radosgw > > ceph version 16.2.3 (381b476cb3900f9a92eb95d03b4850b953cfd79a) pacific (stable) > 1: /lib64/libpthread.so.0(+0x12b20) [0x7faf2dd05b20] > 2: (rgw_bucket::rgw_bucket(rgw_bucket const&)+0x23) [0x7faf38b4d083] > 3: (rgw::sal::RGWObject::get_obj() const+0x20) [0x7faf38b7bcf0] > 4: (RGWInitMultipart::verify_permission(optional_yield)+0x6c) [0x7faf38e6608c] > 5: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, optional_yield, bool)+0x86a) [0x7faf38b2db1a] > 6: (process_request(rgw::sal::RGWRadosStore*, RGWREST*, RGWRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, optional_yield, rgw::dmclock::Scheduler*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*, int*)+0x26dd) [0x7faf38b3232d] > 7: /lib64/libradosgw.so.2(+0x4a1c0b) [0x7faf38a83c0b] > 8: /lib64/libradosgw.so.2(+0x4a36a4) [0x7faf38a856a4] > 9: /lib64/libradosgw.so.2(+0x4a390e) [0x7faf38a8590e] > 10: make_fcontext() > NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. > > > > > -- > > Vincent Chu > > A-4: Advanced Research in Cyber Systems > > Los Alamos National Laboratory > _______________________________________________ > ceph-users mailing list --ceph-users(a)ceph.io > To unsubscribe send an email toceph-users-leave(a)ceph.io

1 0

Multi-site failed to retrieve sync info: (13) Permission denied
by Владимир Клеусов 29 Jun '21

29 Jun '21

Hi, I have set up a multisite. Data (pools, buckets,users) from the master zone is synchronized to the secondary zone on the master radosgw-admin sync status realm 2194d6d2-0df4-400c-be8b-71dc74405ec2 (multisite-realm) zonegroup b41c6159-16e5-456f-a4e1-fb3dd280158f ((multisite-zg) zone 6843d5b5-5c6f-4ea3-a85b-12d8c3d58af8 (zone1) metadata sync no sync (zone is master) 2021-06-29T21:18:23.981+0300 7f2755878040 0 data sync zone:ca179040 ERROR: failed to fetch datalog info data sync source: ca179040-683f-4d9f-b8cc-00c7872dda35 (zone2) failed to retrieve sync info: (13) Permission denied on secondary radosgw-admin sync --rgw-realm=multisite-realm status realm 2194d6d2-0df4-400c-be8b-71dc74405ec2 (multisite-realm) zonegroup b41c6159-16e5-456f-a4e1-fb3dd280158f (multisite-zg) zone ca179040-683f-4d9f-b8cc-00c7872dda35 (zone2) metadata sync syncing full sync: 0/64 shards incremental sync: 64/64 shards metadata is caught up with master data sync source: 6843d5b5-5c6f-4ea3-a85b-12d8c3d58af8 (zone1) syncing full sync: 0/128 shards incremental sync: 128/128 shards data is caught up with source access key and secret key is the same in both zones ceph version octopus 15.2.13 Any ideas =)

1 0

upgrading from Nautilus on CentOS7 to Octopus on Ubuntu 20.04.2
by Jean-Philippe Méthot 29 Jun '21

29 Jun '21

Hi, I’ve been running a staging Ceph environment on CentOS 7/Nautilus for quite a while now. Because of many good reasons that you can probably guess, I am currently trying to move this staging environment to Octopus on Ubuntu 20.04.2. Since I’m trying to keep the data, but don’t mind downtime at all, my plan was to reinstall one server at a time, removing them from the cluster and adding them back to the cluster, since Octopus and Nautilus should still be compatible with each other. I started with one monitor and now I’m stuck in a weird state. Essentially : -Monitors see each other and are establishing connections to each other. -Monitor clocks are synchronized -Monmap was injected in the reinstalled monitor -New monitor is recognized in Ceph -s but stuck out of the quorum Is there something that could prevent a new monitor from establishing quorum if the monmap is the same, the clock is synchronized and it can contact other monitors on the network? Jean-Philippe Méthot Senior Openstack system administrator Administrateur système Openstack sénior PlanetHoster inc.

2 2

ceph-Dokan on windows 10 not working after upgrade to pacific
by Robert W. Eckert 29 Jun '21

29 Jun '21

Hi- I recently upgraded to pacific, and I am now getting an error connecting on my windows 10 machine: The error is the handle_auth_bad_method, I tried a few combinations of cephx,none on the monitors, but I keep getting the same error. The same config(With paths updated) and key ring works on my WSL instance running an old luminous client (I can't seem to get it to install a newer client ) Do you have any suggestions on where to look? Thanks, Rob. ----------------------------------------------------- PS C:\Program Files\Ceph\bin> .\ceph-dokan.exe --id rob -l Q 2021-05-14T12:19:58.172Eastern Daylight Time 5 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] failed to fetch mon config (--no-mon-config to skip) PS C:\Program Files\Ceph\bin> cat c:/ProgramData/ceph/ceph.client.rob.keyring [client.rob] key = <REDACTED> caps mon = "allow rwx" caps osd = "allow rwx" PS C:\Program Files\Ceph\bin> cat C:\ProgramData\Ceph\ceph.conf # minimal ceph.conf [global] log to stderr = true ; Uncomment the following in order to use the Windows Event Log log to syslog = true run dir = C:/ProgramData/ceph/out crash dir = C:/ProgramData/ceph/out ; Use the following to change the cephfs client log level debug client = 2 [global] fsid = <redacted> mon_host = [<redacted>] auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx [client] keyring = c:/ProgramData/ceph/ceph.client.rob.keyring log file = C:/ProgramData/ceph/out/$name.$pid.log admin socket = C:/ProgramData/ceph/out/$name.$pid.asok

2 1

docs dangers large raid
by Marc 29 Jun '21

29 Jun '21

Can someone point me to some good doc's describing the dangers of using a large amount of disks in a raid5/raid6? (Understandable for less techy people)

2 2

[cephadm] Unable to create multiple unmanaged OSDs per device
by Aggelos Avgerinos 29 Jun '21

29 Jun '21

Hi everyone, I'm currently trying to setup the OSDs in a fresh cluster using cephadm. The underlying devices are NVMe and I'm trying to provision 2 OSDs per device with the following spec: ``` service_type: osd service_id: all-nvmes unmanaged: true placement: label: osd data_devices: all: true osds_per_device: 2 ``` Since `ceph orch apply` just creates the service but not the daemons when `unmanaged: True`. Is there a way to enfore the `osds_per_device` setting when using `ceph orch daemon add osd`? Thanks in advance

1 0

Can we deprecate FileStore in Quincy?
by Neha Ojha 29 Jun '21

29 Jun '21

Hello everyone, Given that BlueStore has been the default and more widely used objectstore since quite some time, we would like to understand whether we can consider deprecating FileStore in our next release, Quincy and remove it in the R release. There is also a proposal [0] to add a health warning to report FileStore OSDs. We discussed this topic in the Ceph Month session today [1] and there were no objections from anybody on the call. I wanted to reach out to the list to check if there are any concerns about this or any users who will be impacted by this decision. Thanks, Neha [0] https://github.com/ceph/ceph/pull/39440 [1] https://pad.ceph.com/p/ceph-month-june-2021

10 15

Where did links to official MLs are moved?
by Satoru Takeuchi 28 Jun '21

28 Jun '21

Today I visited Ceph's official site and found that links to `resources` page seemed to be missing. https://ceph.io/en/ In addition, this page is no longer exists. https://ceph.io/resources/ Could you tell me where did they are moved? Thanks, Satoru

1 0

Nic bonding (lacp) settings for ceph
by mhnx 28 Jun '21

28 Jun '21

Hello. Im looking for proper way for the nic bonding for ceph. I was using the bonding driver with default settings ad_select=stable(default) and hash algorithm layer2. With this settings osds are using only one port and lacp only usefull for different nodes because layer2 hash algorithm using Mac. I've changed ad_select to bandwitdh and both nic is in use now but layer2 hash prevents dual nic usage for between two nodes (because layer2 using only Mac ). People advice using layer2+3 for best performance but it has no effect on osds because mac and ip is the same. I've tried layer3+4 to split by ports instead mac and it works. But i dont know what will the effect and also my switch is layer2. With "iperf -Parallel 2" now i can reach 19gbit on 2x 10gbit nics. I think there is no way to use both nic without parallel Usage with different ports. If I use same port and too many process I can only use one port at a time. What settings are you using? What is the best for ceph?

4 6

radosgw user "check_on_raw" setting
by Jared Jacob 28 Jun '21

28 Jun '21

Hello, I am setting up user quotas and I would like to enable the check on raw setting for my user's quota. I can't find any documentation on how to change this setting in any of the ceph documents. Do any of you know how to change this setting? Possibly using radosgw-admin? Thanks in advance! Jared Jacob

2 1

NFS Ganesha ingress parameter not valid?
by Oliver Weinmann 26 Jun '21

26 Jun '21

Dear All, I have deployed the latest CEPH Pacific release in my lab and started to check out the new ?stable? NFS Ganesha features. First of all I'm a bit confused which method to actually use to deploy the NFS cluster: cephadm or ceph nfs cluster create? I used "nfs cluster create" for now and noticed a minor problem in the docs. https://docs.ceph.com/en/latest/cephfs/fs-nfs-exports/#cephfs-nfs The command is stated as: $ ceph nfs cluster create <clusterid> [<placement>] [--ingress --virtual-ip <ip>] while as it needs a type (cephfs) to be specified nfs cluster create <type> <clusterid> [<placement>] : Create an NFS Cluster Also I can't manage to use the --ingress --virtual-ip parameter. Every time I try to use it I get this: [root@cephboot~]# ceph nfs cluster create cephfs ec9e031a-cd10-11eb-a3c3-005056b7db1f --ingress --virtual-ip 192.168.9.199 Invalid command: Unexpected argument '--ingress' nfs cluster create <type> <clusterid> [<placement>] : Create an NFS Cluster Error EINVAL: invalid command So i just deployed a NFS cluster without a VIP. Maybe I'm missing something? What about this note in the docs: >> From Pacific, the nfs mgr module must be enabled prior to use. << I can't find any info on how to enable it. Maybe this is already the case? ceph nfs cluster create cephfs ec9e031a-cd10-11eb-a3c3-005056b7db1f "cephnode01" This seems to be working fine. I managed to connect a CentOS 7 VM and I can access the NFS export just fine. Great stuff. For testing I tried to attach the same NFS export to a standalone ESXi 6.5 Server. This also works, but its diskspace is shown as 0 bytes: I'm not sure if this supported or I'm missing something. I could not find any clear info in the docs only some reddit posts where users mentioned that they were able to use it with VMware. Thanks and Best Regards, Oliver

1 1

Ceph Disk Prediction module issues
by Justas Balcas 25 Jun '21

25 Jun '21

Hello Folks, We are running Ceph Octopus 15.2.13 release and would like to use the disk prediction module. So far issues we faced are: 1. Ceph documentation does not mention to install `ceph-mgr-diskprediction-local.noarch` 2. Even if I install the needed package, after mgr restart, it does not appear on Ceph cluster. Detailed log is here: gist:b687798ea97ef13e36d466f2d7b1470a <https://gist.github.com/juztas/b687798ea97ef13e36d466f2d7b1470a> . Ceph -s shows [1]. Are you aware of this issue and are there any workarounds? Thanks! [1] # ceph -s cluster: id: 12d9d70a-e993-464c-a6f8-4f674db35136 health: HEALTH_WARN no active mgr services: mon: 3 daemons, quorum ceph-mon-cms-1,ceph-mon-cms-2,ceph-mon-cms-3 (age 2d) mgr: no daemons active (since 11m) mds: cephfs:1 {0=ceph-mds-cms-1=up:active} 1 up:standby ceph health detail HEALTH_WARN no active mgr [WRN] MGR_DOWN: no active mgr

1 0

Re: Why you might want packages not containers for Ceph deployments
by Marc 25 Jun '21

25 Jun '21

> but our experience so > far has been a big improvement over the complexity of managing package > dependencies across even just a handful of distros Do you have some charts or docs that show this complexity problem, because I have problems understanding it. This is very likely due to that my understanding of ceph internals is limited. For instance my view of the osd daemon. Now working with logical volumes for writing/reading data and then you have osd<->osd,mon,mgr communication. What dependency hell is there to be expected? > (Podman has been > the only real culprit here, tbh, but I give them a partial pass as the > tool is relatively new.) Is it not better for the sake of stability, security and future support to choose something with a proven record?

1 0

speeding up EC recovery
by Andrej Filipcic 25 Jun '21

25 Jun '21

Hi, on a large cluster with ~1600 OSDs, 60 servers and using 16+3 erasure coded pools, the recovery after OSD failure (HDD) is quite slow. Typical values are at 4GB/s with 125 ops/s and 32MB object sizes, which then takes 6-8 hours, during that time the pgs are degraded. I tried to speed it up with osd advanced osd_max_backfills 32 osd advanced osd_recovery_max_active 10 osd advanced osd_recovery_op_priority 63 osd advanced osd_recovery_sleep_hdd 0.000000 which at least kept the iops/s at a constant level. The recovery does not seem to be cpu or memory bound. Is there any way to speed it up? While testing the recovery on replicated pools, it reached 50GB/s. In contrast, replacing the failed drive with a new one and re-adding the OSD is quite fast, with 1GB/s recovery rate of misplaced pgs, or ~120MB/s average HDD write speed, which is not very far from HDD throughput. Regards, Andrej -- _____________________________________________________________ prof. dr. Andrej Filipcic, E-mail: Andrej.Filipcic(a)ijs.si Department of Experimental High Energy Physics - F9 Jozef Stefan Institute, Jamova 39, P.o.Box 3000 SI-1001 Ljubljana, Slovenia Tel.: +386-1-477-3674 Fax: +386-1-425-7074 -------------------------------------------------------------

2 1

Can not mount rbd device anymore
by Ml Ml 25 Jun '21

25 Jun '21

Hello List, oversudden i can not mount a specific rbd device anymore: root@proxmox-backup:~# rbd map backup-proxmox/cluster5 -k /etc/ceph/ceph.client.admin.keyring /dev/rbd0 root@proxmox-backup:~# mount /dev/rbd0 /mnt/backup-cluster5/ (just never times out) Any idea how to debug that mount? Tcpdump does show some active traffic. Cheers, Michael

4 9

Missing objects in pg
by Vadim Bulst 25 Jun '21

25 Jun '21

Dear List, since my update yesterday from 14.2.18 to 14.2.20 i got an unhealthy cluster. As I remember right, it appeared after rebooting the second server. They are 7 missing objects from pgs of a cache pool (pool 3). This pool is now changed writeback to proxy and i'm not able to flush all objects. root@scvirt06:/home/urzadmin/ceph_issue# ceph -s cluster: id: 5349724e-fa96-4fd6-8e44-8da2a39253f7 health: HEALTH_ERR 7/15893342 objects unfound (0.000%) Possible data damage: 7 pgs recovery_unfound Degraded data redundancy: 21/47680026 objects degraded (0.000%), 7 pgs degraded, 7 pgs undersized client is using insecure global_id reclaim mons are allowing insecure global_id reclaim services: mon: 3 daemons, quorum scvirt03,scvirt06,scvirt01 (age 19h) mgr: scvirt04(active, since 21m), standbys: scvirt03, scvirt02 mds: scfs:1 {0=scvirt04=up:active} 1 up:standby-replay 1 up:standby osd: 54 osds: 54 up (since 17m), 54 in (since 10w); 7 remapped pgs task status: scrub status: mds.scvirt03: idle data: pools: 5 pools, 704 pgs objects: 15.89M objects, 49 TiB usage: 139 TiB used, 145 TiB / 285 TiB avail pgs: 21/47680026 objects degraded (0.000%) 7/15893342 objects unfound (0.000%) 694 active+clean 7 active+recovery_unfound+undersized+degraded+remapped 3 active+clean+scrubbing+deep io: client: 3.7 MiB/s rd, 6.6 MiB/s wr, 40 op/s rd, 31 op/s wr my cluster: scvirt01 - mon,osds scvirt02 - mgr,osds scvirt03 - mon,mgr,mds,osds scvirt04 - mgr,mds,osds scvirt05 - osds scvirt06 - mon,mds,osds log of osd.49: root@scvirt03:/home/urzadmin# tail -f /var/log/ceph/ceph-osd.49.log AddFile(GB): cumulative 0.000, interval 0.000 AddFile(Total Files): cumulative 0, interval 0 AddFile(L0 Files): cumulative 0, interval 0 AddFile(Keys): cumulative 0, interval 0 Cumulative compaction: 0.64 GB write, 0.01 MB/s write, 0.54 GB read, 0.01 MB/s read, 6.5 seconds Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count ** File Read Latency Histogram By Level [default] ** 2021-06-24 08:53:08.865 7f88ab86c700 -1 log_channel(cluster) log [ERR] : 3.9 has 1 objects unfound and apparently lost 2021-06-24 08:53:08.865 7f88a505f700 -1 log_channel(cluster) log [ERR] : 3.1e has 1 objects unfound and apparently lost 2021-06-24 08:53:40.570 7f88ab86c700 -1 log_channel(cluster) log [ERR] : 3.9 has 1 objects unfound and apparently lost 2021-06-24 08:53:40.570 7f88a9067700 -1 log_channel(cluster) log [ERR] : 3.1e has 1 objects unfound and apparently lost 2021-06-24 08:54:45.042 7f88b487e700 4 rocksdb: [db/db_impl.cc:777] ------- DUMPING STATS ------- 2021-06-24 08:54:45.042 7f88b487e700 4 rocksdb: [db/db_impl.cc:778] ** DB Stats ** Uptime(secs): 85202.3 total, 600.0 interval Cumulative writes: 1148K writes, 8640K keys, 1148K commit groups, 1.0 writes per commit group, ingest: 1.24 GB, 0.01 MB/s Cumulative WAL: 1148K writes, 546K syncs, 2.10 writes per sync, written: 1.24 GB, 0.01 MB/s Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent Interval writes: 369 writes, 1758 keys, 369 commit groups, 1.0 writes per commit group, ingest: 0.41 MB, 0.00 MB/s Interval WAL: 369 writes, 155 syncs, 2.37 writes per sync, written: 0.00 MB, 0.00 MB/s Interval stall: 00:00:0.000 H:M:S, 0.0 percent ** Compaction Stats [default] ** Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- L0 3/0 104.40 MB 0.8 0.0 0.0 0.0 0.2 0.2 0.0 1.0 0.0 67.8 2.89 2.70 6 0.482 0 0 L1 2/0 131.98 MB 0.5 0.2 0.1 0.1 0.2 0.1 0.0 1.8 149.9 120.9 1.53 1.41 1 1.527 2293K 140K L2 16/0 871.57 MB 0.3 0.3 0.1 0.3 0.3 -0.0 0.0 5.2 158.1 132.3 2.05 1.93 1 2.052 3997K 1089K Sum 21/0 1.08 GB 0.0 0.5 0.2 0.4 0.6 0.2 0.0 3.3 85.5 100.8 6.47 6.03 8 0.809 6290K 1229K Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 If I run ceph pg repair 3.1e it doesn't change anything and i do not understand why these pgs are undersized. All OSDs are up. ceph.conf: [global] auth_client_required = cephx auth_cluster_required = cephx auth_service_required = cephx cluster_network = 10.10.144.0/24 filestore_xattr_use_omap = true fsid = 5349724e-fa96-4fd6-8e44-8da2a39253f7 mon_allow_pool_delete = true mon_cluster_log_file_level = info mon_host = 172.26.8.151,172.26.8.153,172.26.8.156 osd_journal_size = 5120 osd_pool_default_min_size = 1 public_network = 172.26.8.128/26 [client] keyring = /etc/pve/priv/$cluster.$name.keyring [mds] keyring = /var/lib/ceph/mds/ceph-$id/keyring [mds.scvirt03] host = scvirt03 mds_standby_for_rank = 0 mds_standby_replay = true [mds.scvirt04] host = scvirt04 mds standby for name = pve [mds.scvirt06] host = scvirt06 mds_standby_for_rank = 0 mds_standby_replay = true [mon.scvirt01] public_addr = 172.26.8.151 [mon.scvirt03] public_addr = 172.26.8.153 [mon.scvirt06] public_addr = 172.26.8.156 ceph health detail: HEALTH_ERR 7/15893333 objects unfound (0.000%); Possible data damage: 7 pgs recovery_unfound; Degraded data redundancy: 21/47679999 objects degraded (0.000%), 7 pgs degraded, 7 pgs undersized; client is using insecure global_id reclaim; mons are allowing insecure global_id reclaim OBJECT_UNFOUND 7/15893333 objects unfound (0.000%) pg 3.1e has 1 unfound objects pg 3.1f has 1 unfound objects pg 3.1b has 1 unfound objects pg 3.15 has 1 unfound objects pg 3.16 has 1 unfound objects pg 3.b has 1 unfound objects pg 3.9 has 1 unfound objects PG_DAMAGED Possible data damage: 7 pgs recovery_unfound pg 3.9 is active+recovery_unfound+undersized+degraded+remapped, acting [49,52], 1 unfound pg 3.b is active+recovery_unfound+undersized+degraded+remapped, acting [43,52], 1 unfound pg 3.15 is active+recovery_unfound+undersized+degraded+remapped, acting [44,52], 1 unfound pg 3.16 is active+recovery_unfound+undersized+degraded+remapped, acting [43,51], 1 unfound pg 3.1b is active+recovery_unfound+undersized+degraded+remapped, acting [43,52], 1 unfound pg 3.1e is active+recovery_unfound+undersized+degraded+remapped, acting [49,51], 1 unfound pg 3.1f is active+recovery_unfound+undersized+degraded+remapped, acting [48,51], 1 unfound PG_DEGRADED Degraded data redundancy: 21/47679999 objects degraded (0.000%), 7 pgs degraded, 7 pgs undersized pg 3.9 is stuck undersized for 64516.343966, current state active+recovery_unfound+undersized+degraded+remapped, last acting [49,52] pg 3.b is stuck undersized for 64516.351507, current state active+recovery_unfound+undersized+degraded+remapped, last acting [43,52] pg 3.15 is stuck undersized for 64521.368841, current state active+recovery_unfound+undersized+degraded+remapped, last acting [44,52] pg 3.16 is stuck undersized for 64516.351599, current state active+recovery_unfound+undersized+degraded+remapped, last acting [43,51] pg 3.1b is stuck undersized for 64517.427120, current state active+recovery_unfound+undersized+degraded+remapped, last acting [43,52] pg 3.1e is stuck undersized for 64521.369635, current state active+recovery_unfound+undersized+degraded+remapped, last acting [49,51] pg 3.1f is stuck undersized for 64517.426392, current state active+recovery_unfound+undersized+degraded+remapped, last acting [48,51] AUTH_INSECURE_GLOBAL_ID_RECLAIM client is using insecure global_id reclaim client.admin at 172.26.8.154:0/3925203408 is using insecure global_id reclaim mds.scvirt04 at [v2:172.26.8.154:6836/3778505565,v1:172.26.8.154:6837/3778505565] is using insecure global_id reclaim AUTH_INSECURE_GLOBAL_ID_RECLAIM_ALLOWED mons are allowing insecure global_id reclaim mon.scvirt03 has auth_allow_insecure_global_id_reclaim set to true mon.scvirt06 has auth_allow_insecure_global_id_reclaim set to true mon.scvirt01 has auth_allow_insecure_global_id_reclaim set to true ceph osd tree: ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 284.51312 root default -2 48.75215 host scvirt01 0 hdd 9.09560 osd.0 up 1.00000 1.00000 3 hdd 9.09560 osd.3 up 1.00000 1.00000 6 hdd 9.09560 osd.6 up 1.00000 1.00000 9 hdd 9.09560 osd.9 up 1.00000 1.00000 12 hdd 9.09560 osd.12 up 1.00000 1.00000 42 nvme 0.97029 osd.42 up 1.00000 1.00000 43 nvme 0.97029 osd.43 up 1.00000 1.00000 44 nvme 0.97029 osd.44 up 1.00000 1.00000 37 ssd 0.36330 osd.37 up 1.00000 1.00000 -3 48.75215 host scvirt02 1 hdd 9.09560 osd.1 up 1.00000 1.00000 4 hdd 9.09560 osd.4 up 1.00000 1.00000 7 hdd 9.09560 osd.7 up 1.00000 1.00000 10 hdd 9.09560 osd.10 up 1.00000 1.00000 13 hdd 9.09560 osd.13 up 1.00000 1.00000 45 nvme 0.97029 osd.45 up 1.00000 1.00000 46 nvme 0.97029 osd.46 up 1.00000 1.00000 47 nvme 0.97029 osd.47 up 1.00000 1.00000 38 ssd 0.36330 osd.38 up 1.00000 1.00000 -4 48.75224 host scvirt03 2 hdd 9.09569 osd.2 up 1.00000 1.00000 5 hdd 9.09560 osd.5 up 1.00000 1.00000 8 hdd 9.09560 osd.8 up 1.00000 1.00000 11 hdd 9.09560 osd.11 up 1.00000 1.00000 14 hdd 9.09560 osd.14 up 1.00000 1.00000 48 nvme 0.97029 osd.48 up 1.00000 1.00000 49 nvme 0.97029 osd.49 up 1.00000 1.00000 50 nvme 0.97029 osd.50 up 1.00000 1.00000 39 ssd 0.36330 osd.39 up 1.00000 1.00000 -9 56.75706 host scvirt04 15 hdd 9.09560 osd.15 up 1.00000 1.00000 17 hdd 9.09560 osd.17 up 1.00000 1.00000 20 hdd 9.09560 osd.20 up 1.00000 1.00000 22 hdd 9.09560 osd.22 up 1.00000 1.00000 23 hdd 9.09560 osd.23 up 1.00000 1.00000 25 hdd 3.63860 osd.25 up 1.00000 1.00000 26 hdd 3.63860 osd.26 up 1.00000 1.00000 27 hdd 3.63860 osd.27 up 1.00000 1.00000 40 ssd 0.36330 osd.40 up 1.00000 1.00000 -11 56.75706 host scvirt05 16 hdd 9.09560 osd.16 up 1.00000 1.00000 18 hdd 9.09560 osd.18 up 1.00000 1.00000 19 hdd 9.09560 osd.19 up 1.00000 1.00000 21 hdd 9.09560 osd.21 up 1.00000 1.00000 24 hdd 9.09560 osd.24 up 1.00000 1.00000 28 hdd 3.63860 osd.28 up 1.00000 1.00000 29 hdd 3.63860 osd.29 up 1.00000 1.00000 30 hdd 3.63860 osd.30 up 1.00000 1.00000 41 ssd 0.36330 osd.41 up 1.00000 1.00000 -13 24.74245 host scvirt06 31 hdd 3.63860 osd.31 up 1.00000 1.00000 32 hdd 3.63860 osd.32 up 1.00000 1.00000 33 hdd 3.63860 osd.33 up 1.00000 1.00000 34 hdd 3.63860 osd.34 up 1.00000 1.00000 35 hdd 3.63860 osd.35 up 1.00000 1.00000 36 hdd 3.63860 osd.36 up 1.00000 1.00000 51 nvme 0.97029 osd.51 up 1.00000 1.00000 52 nvme 0.97029 osd.52 up 1.00000 1.00000 53 nvme 0.97029 osd.53 up 1.00000 1.00000 Regards, Vadim -- Vadim Bulst Universität Leipzig / URZ 04109 Leipzig, Augustusplatz 10 phone: +49-341-97-33380 mail: vadim.bulst(a)uni-leipzig.de

1 1

ceph fs mv does copy, not move
by Frank Schilder 25 Jun '21

25 Jun '21

Dear all, some time ago I reported that the kernel client resorts to a copy instead of move when moving a file across quota domains. I was told that the fuse client does not have this problem. If enough space is available, a move should be a move, not a copy. Today, I tried to move a large file across quota domains testing botn, the kernel- and the fuse client. Both still resort to a copy even though this issue was addressed quite a while ago (https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/44AEIHNEGKV…) The versions I'm using are (CentOS 7) # yum list installed | grep ceph-fuse ceph-fuse.x86_64 2:13.2.10-0.el7 @ceph # uname -r 3.10.0-1160.31.1.el7.x86_64 Any suggestions how to get this to work? I have to move directories containing 100+ TB. Many thanks, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

4 10

native linux distribution host running ceph container ?
by marc boisis 25 Jun '21

25 Jun '21

Hi, We have a containerised ceph cluster in version 16.2.4 (15 hosts, 180 osds) deployed with ceph-ansible. Our host run on centos 7 (kernel 3.10) with ceph-deamon docker image based on centos 8. I cannot find in the documentation which native distribution is recommended, should it be the same as docker image (centos 8) ? About centos8 and the end of support announced for the end of the year, which distribution will ceph use in docker image ? Thanks

1 0

RGW topic created in wrong (default) tenant
by Daniel Iwan 25 Jun '21

25 Jun '21

Hi I'm using Ceph Pacific 16.2.1 I'm creating a topic as a user which belongs to a non-default tenant. I'm using AWS CLI 2 with v3 authentication enabled aws --profile=ceph-myprofile --endpoint=$HOST_S3_API --region="" sns create-topic --name=fishtopic --attributes='{"push-endpoint": " http://my-ceph-source-svc.default.svc.cluster.local"}' { "TopicArn": "arn:aws:sns:default::fishtopic" } topic is created in default tenant though. User can list topics but see topics from the default tenant. aws --profile=ceph-myprofile --endpoint=$HOST_S3_API --region="" sns list-topics { "Topics": [ { "TopicArn": "arn:aws:sns:default::fishtopic" } ] } Topic is in default tenant # radosgw-admin topic list --uid none { "topics": [ { "topic": { "user": "", "name": "fishtopic", "dest": { "bucket_name": "", "oid_prefix": "", "push_endpoint": " http://my-ceph-source-svc.default.svc.cluster.local", "push_endpoint_args": "Attributes.entry.1.key=push-endpoint&Attributes.entry.1.value= http://my-ceph-source-svc.default.svc.cluster.local &Version=2010-03-31&push-endpoint= http://my-ceph-source-svc.default.svc.cluster.local", "push_endpoint_topic": "fishtopic", "stored_secret": "false", "persistent": "false" }, "arn": "arn:aws:sns:default::fishtopic", "opaqueData": "" }, "subs": [] } ] } When I create a topic over HTTP with a federated user, the topic is created in the correct (user's) tenant. For some reason the "user" below is "marvel", which is actually the name of the tenant. Possibly the topic is not owned by the user but rather the tenant. radosgw-admin topic list --tenant marvel --uid none { "topics": [ { "topic": { "user": "marvel", "name": "MyTopic", "dest": { "bucket_name": "", "oid_prefix": "", "push_endpoint": "amqp://127.0.0.1", "push_endpoint_args": "amqp-exchange=rgw-exchange&push-endpoint=amqp://127.0.0.1 &use-ssl=false&verify-ssl=false", "push_endpoint_topic": "MyTopic", "stored_secret": "false", "persistent": "false" }, "arn": "arn:aws:sns:default:marvel:MyTopic", "opaqueData": "" }, "subs": [] } ] } Also, what permissions are checked when creating a topic? It seems so far I can create a topic without granting any special permissions? Regards Daniel

2 4

PG inconsistent+failed_repair
by Vladimir Prokofev 24 Jun '21

24 Jun '21

Hello. Today we've experienced a complete CEPH cluster outage - total loss of power in the whole infrastructure. 6 osd nodes and 3 monitors went down at the same time. CEPH 14.2.10 This resulted in unfound objects, which were "reverted" in a hurry with ceph pg <pg_id> mark_unfound_lost revert In retrospect that was probably a mistake as the "have" part stated 0'0. But then deep-scrubs started and they found inconsistent PGs. We tried repairing them, but they just switched to failed_repair. Here's a log example: 2021-06-25 00:08:07.693645 osd.0 [ERR] 3.c shard 6 3:3163e703:::rbd_data.be08c566ef438d.0000000000002445:head : missing 2021-06-25 00:08:07.693710 osd.0 [ERR] repair 3.c 3:3163e2ee:::rbd_data.efa86358d15f4a.000000000000004b:6ab1 : is an unexpected clone 2021-06-25 00:11:55.128951 osd.0 [ERR] 3.c repair 1 missing, 0 inconsistent objects 2021-06-25 00:11:55.128969 osd.0 [ERR] 3.c repair 2 errors, 1 fixed I tried manually deleting conflicting objects from secondary osds with ceph-objectstore-tool like this ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-22 --pgid 3.c rbd_data.efa86358d15f4a.000000000000004b:6ab1 remove it removes it but without any positive impact. Pretty sure I don't understand the concept. So currently I have the following thoughts: - is there any doc on the object placement specifics and what all of those numbers in their name mean? I've seen objects with similar prefix/mid but different suffix and I have no idea what does it mean; - I'm actually not sure what the production impact is at that point because everything seems to work so far. So I'm thinking if it's possible to kill replicas on secondary OSDd with ceph-objectstore-tool and just let CEPH create a replica from primary PG? I have 8 scrub errors and 4 inconsistent+failed_repair PGs, and I'm afraid that further deep scrubs will reveal more errors. Any thoughts appreciated.

1 1

iscsi, gwcli, and vmware version
by Philip Brown 24 Jun '21

24 Jun '21

I notice on https://docs.ceph.com/en/latest/rbd/iscsi-initiator-esx/ that it lists a requirement of "VMware ESX 6.5 or later using Virtual Machine compatibility 6.5 with VMFS 6." Could anyone enlighten me as to why this specific limit is in place? Officlaly knowing something like, "you have to use v6.5 or later, because X happens", would be very helpful to me when doing a writeup for potential deployment plans. -- Philip Brown| Sr. Linux System Administrator | Medata, Inc. 5 Peters Canyon Rd Suite 250 Irvine CA 92606 Office 714.918.1310| Fax 714.918.1325 pbrown(a)medata.com| www.medata.com

2 2

query about product use of rbd mirror for DR
by huxiaoyu＠horebdata.cn 24 Jun '21

24 Jun '21

Dear Ceph Folks， Does anyone has real experience of using rbd mirroring for disaster recovery over 1000 miles away? I am planning using Ceph rbd mirroring feature for DR, and has no real experience. Could anyone sharing good or bad experience here? I am thinking of using iSCSI over rbd-nbd map, with rbd mirror to a remote site using a dedicated link of 200Mb/s. Ceph version will be on Luminous 12.2.13 Any sharing, suggestions and comments are highly appreciated. best regards, samuel huxiaoyu(a)horebdata.cn

1 0

Ceph Month June Schedule Now Available
by Mike Perez 24 Jun '21

24 Jun '21

Hi everyone, The Ceph Month June schedule is now available: https://pad.ceph.com/p/ceph-month-june-2021 We have great sessions from component updates, performance best practices, Ceph on different architectures, BoF sessions to get more involved with working groups in the community, and more! You may also leave open discussion topics for the listed talks that we'll get to each Q/A portion. I will provide the video stream link on this thread and etherpad once it's available. You can also add the Ceph community calendar, which will have the Ceph Month sessions prefixed with "Ceph Month" to get local timezone conversions. https://calendar.google.com/calendar/embed?src=9ts9c7lt7u1vic2ijvvqqlfpo0%4… Thank you to our speakers for taking the time to share with us all the latest best practices and usage with Ceph! -- Mike Perez

2 13

Start a service on a specified node
by E Taka 24 Jun '21

24 Jun '21

If I understand the documentation for the placements in "ceph orch apply" correctly, I can place the daemons by number or on specific host. But what I want is: "Start 3 mgr services, and one of it should be started on node ceph01." How I can achieve this? Thanks!

1 0

pacific installation at ubuntu 20.04
by Jana Markwort 24 Jun '21

24 Jun '21

Hi all, I'm a new ceph user and try to install my first cluster. I try to install pacific but as result I get octopus. What's wrong here? I've done: # curl --silent --remote-name --location https://github.com/ceph/ceph/raw/pacific/src/cephadm/cephadm # chmod +x cephadm # ./cephadm add-repo --release pacific # ./cephadm install # cephadm install ceph-common # ceph -v ceph version 15.2.11 (e3523634d9c2227df9af89a4eac33d16738c49cb) octopus (stable) # cat /etc/apt/sources.list.d/ceph.list deb https://download.ceph.com/debian-pacific/ focal main ?? Kind regards, Jana

2 3

Re: In "ceph health detail", what's the diff between MDS_SLOW_METADATA_IO and MDS_SLOW_REQUEST?
by Patrick Donnelly 24 Jun '21

24 Jun '21

On Mon, Jun 21, 2021 at 8:13 PM opengers <zijian1012(a)gmail.com> wrote: > > Thanks for the answer, I still have some confusion when I see the explanation of "MDS_SLOW_REQUEST" from the document , as follows > ------ > MDS_SLOW_REQUEST > > Message > “N slow requests are blocked” > > Description > One or more client requests have not been completed promptly, indicating that the MDS is either running very slowly, or that the RADOS cluster is not acknowledging journal writes promptly, or that there is a bug. Use the ops admin socket command to list outstanding metadata operations. This message appears if any client requests have taken longer than mds_op_complaint_time (default 30s). > > FROM: https://docs.ceph.com/en/latest/cephfs/health-messages/ > ------ > > "or that the RADOS cluster is not acknowledging journal writes promptly", from this sentence, it seems that "MDS_SLOW_REQUEST" also contains OSD operations by the MDS? Yes. If you have slow metadata IO warnings you will likely also have slow request warnings. -- Patrick Donnelly, Ph.D. He / Him / His Principal Software Engineer Red Hat Sunnyvale, CA GPG: 19F28A586F808C2402351B93C3301A3E258DD79D

1 0

How to stop a rbd migration and recover
by Gilles Mocellin 23 Jun '21

23 Jun '21

Hello, As a follow-up of the thread "RBD migration between 2 EC pools : very slow". I'm running Octopus 15.2.13. RBD migration seems really fragile. I started a migration to change the data pool (from an EC 3+2 to an EC 8+2) : - rbd migration prepare - rbd migration execute => 4% after 6h, and no progress 12h later - rbd migration abort => does not return After that, the state of the migration on destination image is "unknown'. rbd info on source (in trash) and destination image shows migrating in the features. Debbugging another abord command shows that it tries to take/put a lock, and there's one already, due to either the cacncelled execute or the subsequent abort. rbd lock delete does not work, I have a strange message about a read only filesystem... So, I'm stuck. My production on the source image is stopped so far, because I use krbd and I must terminate the migration (either commit or abort) before being able to map and mount it. What I think I'm just understanding, is that abort is meant to be used only when the migration execution is finished, like commit ? Am I wrong ? If so, How can we stop an ongoinfg migration, and also, How do we recover the source image ? Do I need to : - delete the destination image - restore the source image from the trash What about the migrating feature that forbid krbd to map the image ?

1 0

Octopus 15.2.8 slow ops causing inactive PGs upon disk replacement
by Justin Goetz 23 Jun '21

23 Jun '21

Hello! We are in the process of expanding our CEPH cluster (by both adding OSD hosts and replacing smaller-sized HDDs on our existing hosts). So far we have gone host by host, removing the old OSDs, swapping the physical HDDs, and re-adding them. This process has gone smooth, aside from one issue: upon any action taken on the cluster (adding new OSDs, replacing old, etc), we have PGs get stuck "activating"which causes around 3.5% of PGs go inactive, causing IO to stop. Here is a current look at our ceph -s command: cluster: id: e8ffe2eb-f8fc-4110-a4bc-1715e878fb7b health: HEALTH_WARN Reduced data availability: 166 pgs inactive Degraded data redundancy: 137153907/3658405707 objects degraded (3.749%), 930 pgs degraded, 928 pgs undersized 10 pgs not deep-scrubbed in time 33709 slow ops, oldest one blocked for 35956 sec, daemons [osd.103,osd.104,osd.105,osd.106,osd.107,osd.109,osd.111,osd.112,osd.113,osd.114]... have slow ops. services: mon: 3 daemons, quorum lb3,lb2,lb1 (age 8w) mgr: lb1(active, since 6w), standbys: lb3, lb2 osd: 117 osds: 117 up (since 15m), 117 in (since 10h); 2033 remapped pgs rgw: 3 daemons active (lb1.rgw0, lb2.rgw0, lb3.rgw0) task status: data: pools: 8 pools, 5793 pgs objects: 609.74M objects, 169 TiB usage: 308 TiB used, 430 TiB / 738 TiB avail pgs: 2.866% pgs not active 137153907/3658405707 objects degraded (3.749%) 262215404/3658405707 objects misplaced (7.167%) 3754 active+clean 963 active+remapped+backfill_wait 892 active+undersized+degraded+remapped+backfill_wait 136 activating+remapped 27 activating+undersized+degraded+remapped 8 active+undersized+degraded+remapped+backfilling 6 active+clean+scrubbing+deep 3 activating+degraded+remapped 3 active+remapped+backfilling 1 active+undersized+remapped+backfill_wait io: client: 94 KiB/s rd, 94 op/s rd, 0 op/s wr recovery: 112 MiB/s, 372 objects/s progress: Rebalancing after osd.20 marked in (10h) [............................] (remaining: 11d) Rebalancing after osd.41 marked in (10h) [=...........................] (remaining: 8d) Rebalancing after osd.30 marked in (10h) [=...........................] (remaining: 9d) Rebalancing after osd.1 marked in (10h) [=======.....................] (remaining: 2h) Rebalancing after osd.10 marked in (10h) [............................] (remaining: 12d) Rebalancing after osd.50 marked in (10h) [............................] (remaining: 2w) Rebalancing after osd.71 marked out (10h) [==..........................] (remaining: 5d) What you may find interesting is the "slow ops" warnings. This is where our inactive PGs become stuck. Once the cluster gets into this state, I'm able to recover IO usually by restarting the OSDs with slow ops. However, what's extremely strange, is this workaround only works after about 12 hours since the last OSD addition. Restarting the slow ops OSDs before roughly 12 hours results in the slow ops returning immediately. Our first thought was hardware issues, however we ruled this out after the slow ops warnings appeared on brand new HDDs and OSD hosts. Monitoring the IO saturation of the OSDs reporting slow ops shows actual usage nowhere near saturation, and no hardware issues are present on the drives themselves. Looking at the journalctl logs of one of the affected OSDs above, we see the following repeated multiple times: osd.103 56934 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.467952.0:1520304537 8.6fbs0 8.1e6826fb (undecoded) ondisk+retry+write+known_if_redirected e56923 So far my procedure for the disk swaps have been as follows: 1. Set noout,norebalance, and norecover on the cluster. 2. Use ceph-ansible to remove the old disk OSD IDs 3. Swap physical HDDs, re-add with ceph-ansible 4. Unset noout,norebalance,norecover I should note this issue appears even with simple OSD additions (not removals), as we added 2 brand new hosts to the cluster and saw the same issue. I've been trying to think of any possible cause of this issue, I should mention our cluster is messy at the moment hardware-wise (we have a mix of 7T HDDs, 4T HDDs, and 10T HDDs - moving to all 10T HDDs but the process to swap has been taking a while). One warning I've noticed during the old disk removals is a warning about too many PGs per OSD, however this warning clears once the new OSDs are added, which is to be expected I assume. If anyone would be willing to provide any hints of where to look, it would be much appreciated! Thanks for your time. -- Justin Goetz Systems Engineer, TeraSwitch Inc. jgoetz(a)teraswitch.com 412-945-7045 (NOC) | 412-459-7945 (Direct)

2 2

How can I check my rgw quota ?
by Massimo Sgaravatto 23 Jun '21

23 Jun '21

Sorry for the very naive question: I know how to set/check the rgw quota for a user (using radosgw-admin) But how can a radosgw user check what is the quota assigned to his/her account , using the S3 and/or the swift interface ? I don't get this information using "swift stat", and I can't find a s3cmd quota related command ... Thanks, Massimo

3 2

when is krbd on osd nodes starting to get problematic?
by Marc 23 Jun '21

23 Jun '21

From what kernel / ceph version is krbd usage on a osd node problematic? Currently I am running Nautilus 14.2.11 and el7 3.10 kernel without any issues. I can remember using a cephfs mount without any issues as well, until some specific luminous update surprised me. So maybe nice to know when to expect this. > -----Original Message----- > Sent: Wednesday, 23 June 2021 11:25 > Subject: *****SPAM***** [ceph-users] Re: Can not mount rbd device > anymore > > On Wed, Jun 23, 2021 at 9:59 AM Matthias Ferdinand > wrote: > > > > On Tue, Jun 22, 2021 at 02:36:00PM +0200, Ml Ml wrote: > > > Hello List, > > > > > > oversudden i can not mount a specific rbd device anymore: > > > > > > root@proxmox-backup:~# rbd map backup-proxmox/cluster5 -k > > > /etc/ceph/ceph.client.admin.keyring > > > /dev/rbd0 > > > > > > root@proxmox-backup:~# mount /dev/rbd0 /mnt/backup-cluster5/ > > > (just never times out) > > > > > > Hi, > > > > there used to be some kernel lock issues when the kernel rbd client > > tried to access an OSD on the same machine. Not sure if these issues > > still exist (but I would guess so) and if you use your proxmox cluster > > in a hyperconverged manner (nodes providing VMs and storage service at > > the same time) you may just have been lucky that it had worked before. > > > > Instead of the kernel client mount you can try to export the volume as > > an NBD device (https://docs.ceph.com/en/latest/man/8/rbd-nbd/) and > > mounting that. rbd-nbd runs in userspace and should not have that > > locking problem. > > rbd-nbd is also susceptible to locking up in such setups, likely more > so than krbd. Don't forget that it also has a kernel component and > there are actually more opportunities for things to go sideways/lock up > because there is an extra daemon involved allocating some additional > memory for each I/O request. > > Thanks, > > Ilya > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

RBD migration between 2 EC pools : very slow
by Gilles Mocellin 23 Jun '21

23 Jun '21

Hello Cephers, On a capacitive Ceph cluster (13 nodes, 130 OSDs 8To HDD), I'm migrating a 40 To image from a 3+2 EC pool to a 8+2 one. The use case is Veeam backup on XFS filesystems, mounted via KRBD. Backups are running, and I can see 200MB/s Throughput. But my migration (rbd migrate prepare / execute) is staling at 4% for 6h now. When the backups are not running, I can see a little 20MB/s of throughput, certainly my migration. I need a month to migrate 40 to at that speed ! As I use a KRBD client, I cannot remap the rbd image straight after the rbd prepare. So the filesystem is not usable until the migration is completed. Not really usable for me... Is anyone has a clue either to speed up the rbd migration, or another method to move/copy an image between 2 pools, with the minimum downtime ? I thought of rbd export-diff | rbd import-diff, while mounted, and another unmapped before switching... But, it forces me to rename my image, because if I use another data pool, the metadata pool stays the same. Can you see another method ? -- Gilles

2 3

Ceph rbd-nbd performance benchmark
by Bobby 23 Jun '21

23 Jun '21

Hi, I am trying to benchmark the Ceph rbd-nbd performance. Are there any authentic existing benchmark results of rbd-nbd for comparison? BR Bobby

1 0

HDD <-> OSDs
by Thomas Roth 22 Jun '21

22 Jun '21

Hi all, newbie question: The documentation seems to suggest that with ceph-volume, one OSD is created for each HDD (cf. 4-HDD-example in https://docs.ceph.com/en/latest/rados/configuration/bluestore-config-ref/) This seems odd: what if a server has a finite number of disks? I was going to try cephfs on ~10 servers with 70 HDD each. That would make each system having to deal with 70 OSDs, on 70 LVs? Really no aggregation of the disks? Regards, Thomas -- -------------------------------------------------------------------- Thomas Roth Department: IT GSI Helmholtzzentrum für Schwerionenforschung GmbH www.gsi.de

7 8

OSD bootstrap time
by Jan-Philipp Litza 22 Jun '21

22 Jun '21

Hi everyone, recently I'm noticing that starting OSDs for the first time takes ages (like, more than an hour) before they are even picked up by the monitors as "up" and start backfilling. I'm not entirely sure if this is a new phenomenon or if it always was that way. Either way, I'd like to understand why. When I execute `ceph daemon osd.X status`, it says "state: preboot" and I can see the "newest_map" increase slowly. Apparently, a new OSD doesn't fetch the latest OSD map and gets to work, but instead fetches hundreds of thousands of OSD maps from the mon, burning CPU while parsing them. I wasn't able to find any good documentation on the OSDMap, in particular why its historical versions need to be kept and why the OSD seemingly needs so many of them. Can anybody point me in the right direction? Or is something wrong with my cluster? Best regards, Jan-Philipp Litza

3 6

Octopus support
by Shafiq Momin 22 Jun '21

22 Jun '21

Hi all, I see octopus is having limited Suport on Centos 7 I have prod cluster with 1.2 PTB data with nautilus 14.2 Can we upgrade on Centos 7 from nautilus to octopus or we foresee issue We have erasure coded pool Please guide on recommended approach and document if any Will yum upgrade will upgrade nautilus to octopus Regards -- Regards Shafiq 9029056566/9223316304

2 1

Fwd: In "ceph health detail", what's the diff between MDS_SLOW_METADATA_IO and MDS_SLOW_REQUEST?
by opengers 22 Jun '21

22 Jun '21

---------- Forwarded message --------- 发件人： opengers <zijian1012(a)gmail.com> Date: 2021年6月22日周二上午11:12 Subject: Re: [ceph-users] In "ceph health detail", what's the diff between MDS_SLOW_METADATA_IO and MDS_SLOW_REQUEST? To: Patrick Donnelly <pdonnell(a)redhat.com> Thanks for the answer, I still have some confusion when I see the explanation of "MDS_SLOW_REQUEST" from the document , as follows ------ MDS_SLOW_REQUEST Message “N slow requests are blocked” Description One or more client requests have not been completed promptly, indicating that the MDS is either running very slowly, or that the RADOS cluster is not acknowledging journal writes promptly, or that there is a bug. Use the ops admin socket command to list outstanding metadata operations. This message appears if any client requests have taken longer than mds_op_complaint_time (default 30s). FROM: https://docs.ceph.com/en/latest/cephfs/health-messages/ ------ "or that the RADOS cluster is not acknowledging journal writes promptly", from this sentence, it seems that "MDS_SLOW_REQUEST" also contains OSD operations by the MDS? Patrick Donnelly <pdonnell(a)redhat.com> 于2021年6月22日周二上午3:23写道： > Hello, > > On Mon, Jun 21, 2021 at 8:54 AM opengers <zijian1012(a)gmail.com> wrote: > > > > *$ *ceph health detail > > HEALTH_WARN 1 MDSs report slow metadata IOs; 1 MDSs report slow > > requests MDS_SLOW_METADATA_IO > > 1 MDSs report slow metadata IOs > > mds.fs-01(mds.0): 3 slow metadata IOs are blocked > 30 secs, > oldest > > blocked for 51123 secs MDS_SLOW_REQUEST 1 MDSs report slow requests > > MDS_SLOW_REQUEST: RPCs from the client to the MDS are "slow", i.e. not > complete in less than 30 seconds. > MDS_SLOW_METADATA_IO: OSD operations by the MDS are not yet complete > after 30 seconds. > > -- > Patrick Donnelly, Ph.D. > He / Him / His > Principal Software Engineer > Red Hat Sunnyvale, CA > GPG: 19F28A586F808C2402351B93C3301A3E258DD79D > >

1 0

mark_unfound_lost delete not deleting unfound objects
by Brian Andrus 21 Jun '21

21 Jun '21

I am wondering if anyone has experience with the mark_unfound_lost delete command seemingly not doing what it is supposed to, or if perhaps I have unreasonable expectations about its function. We have a EC pool making up a rgw data pool, and we have had a data loss scenario. I've attempted to manually recover shards from all listed peers in might_have_unfound, and had some success, but after extensive searching, I believe the time has come to let go of the data we are still missing in hopes of getting the cluster back to healthy and restoring service functionality. When I run "ceph pg 21.258e mark_unfound_lost delete", the command runs for some time, until a few minutes in the primary OSD drops out of the cluster but is still running. The logs would suggest this is because it is doing some intensive iterative operations and is unresponsive to other OSDs. Given we have tens of thousands of objects being marked lost, it would make sense this might take some time... but in the meantime, the OSD is marked out, another OSD takes its place, and the number of unfound objects for the PG increases over the next few hours back to the original amount. It seems so far, the primary OSD has not come back in every time I've tried this operation. My initial reaction was to restart the OSD when it dropped from the cluster (and its PG went DOWN state) in an attempt to keep the RGW functioning, but I realize that could have been counterproductive once I observed the logs of the primary iterating over objects. Yet even leaving the OSD to complete the iterative process, it doesn't seem to rejoin cluster without an intervention in the form of daemon restart. I'm wondering if anyone has experience deleting unfound objects at this scale, and if it is an asynchronous operation that eventually completes, or if we are encountering some unexpected behavior that warrants a bug report? I am also wondering if ceph-objectstore-tool might be employed to work on all shards of the PG at once and just start them back up together, minus the unfound objects? I haven't seen much useful documented use of the "fix-lost" operation, so I have hesitated to try it without a full understanding of what it does. Thank you to anyone who might be able to provide some information. -- Brian Andrus | Cloud Systems Engineer | DreamHost brian.andrus(a)DreamHost.com | www.dreamhost.com

1 0

In "ceph health detail", what's the diff between MDS_SLOW_METADATA_IO and MDS_SLOW_REQUEST?
by opengers 21 Jun '21

21 Jun '21

*$ *ceph health detail HEALTH_WARN 1 MDSs report slow metadata IOs; 1 MDSs report slow requests MDS_SLOW_METADATA_IO 1 MDSs report slow metadata IOs mds.fs-01(mds.0): 3 slow metadata IOs are blocked > 30 secs, oldest blocked for 51123 secs MDS_SLOW_REQUEST 1 MDSs report slow requests mds.fs-01(mds.0): 5 slow requests are blocked > 30 secs Do these two refer to cephfs metadata IO?

2 1

Re: Why you might want packages not containers for Ceph deployments
by Marc 21 Jun '21

21 Jun '21

> -----Original Message----- > Sent: Sunday, 20 June 2021 21:34 > To: ceph-users(a)ceph.io > Subject: *****SPAM***** [ceph-users] Re: Why you might want packages not > containers for Ceph deployments > > > > 3. Why is in this cephadm still being talked about systemd? Your > orchestrator should handle restarts,namespaces and failed tasks not? > There should be no need to have a systemd dependency, at least I have > not seen any container images relying on this. > > Podman uses systemd to manage containers so that it is daemonless, > contrast with Docker where one has to maintain a separate daemon and use > docker-specific tools to mange containers. If you assert that Podman > should not exist, please take that up with the Podman folks. If your OC uses systemd that means your OC is dependent on systemd and ceph not. Nobody here is discussing OC specifics. > > 4. Ok found the container images[2] (I think). Sorry but this has > ‘nothing’ to do with container thinking. I expected to find container > images for osd, msd, rgw separately and smaller. This looks more like an > OS deployment. > > Bundling all the daemons together into one container is *genius*. Much > simpler to build and maintain, one artifact vs a bunch. I wouldn’t be > surprised if there are memory usage efficiencies too. 😃 what a non-sense. If building container images is a problem, do not even get involved with containers. > > 7. If you are not setting cpu and memory limits on your cephadm > containers, then again there is an argument why even use containers. > > This seems like a non-sequitor. As many have written, CPU and memory > limits aren’t the reason for containerizing Ceph daemons. If there are > other container applications where doing so makes sense, that’s fine for > those applications. Indeed, so now we have concluded, cephadm does not use container functionality. > I suspect that artificial CPU limiting of Ceph daemons would have a > negative impact on latency, peering/flapping, and slow requests. Ceph > is a distributed system, not a massively parallel one. OSDs already > have a memory target that can be managed natively, vs an external > mechanism that arbitrarily cuts them off at the knees when they need it > the most. That approach would be addressing the symptoms, not the root > cause. Having been through a multi-day outage that was substantially > worsened by the OOMkiller (*), I personally want nothing to do with > blind external mechanisms deciding that they know better than Ceph > daemons whether or not they should be running. If your availability and > performance needs favor rigidly defined areas of doubt and uncertainty, > that’s your own lookout. Agreed, no real need for a using containers. > > * A release that didn’t have OSD memory target setting yet. Having that > would have helped dramatically. > > > 8. I still see lots of comments on the mailing list about accessing > logs. I have all my containers log to a remote syslog server, if you > still have your ceph daemons that can not do this (correctly). What > point is it even going to containers. > > With all possible respect, that’s another non-sequitor, or at the very > least, an assumption that your needs are everyone’s needs. Centralized > logging makes sense in some contexts. But not always, and not to > everyone. Back around the Hammer or Jewel releases there was a bug > where logging to syslog resulted in daemon crashes. I haven’t tried it > with newer releases, but assume that’s long been fixed. > > I’m not a an [r]syslog[ng] expert by far, but I suspect that central- > only logging might not deal well with situations like an OSD spewing > entries when the debug subsystem level is elevated. Moreover, many of > the issues one sees with Ceph clusters are network related. So when > there’s a network problem, I want to rely on the network to be able to > see logs? I’ve seen syslog drop entries under load, something I > personally wouldn’t want for Ceph daemons. There are of course many > strategies between the extremes. So your arguing is, if it does not work in <5% let's not use it? > > 9. I am updating my small cluster something like this: > > I’m guessing you’ve never updated between major releases. That process > tends to have additional steps and nuances, which is one of the > compelling arguments in favor of orchestration: when it’s done well, > most operators don’t need to rev their own homebrew orchestration to set > the right flags at the right time, etc. But one of the great things > about OSS is that you have the flexibility to roll you own if you so > choose. > > > I am never going to run a ‘ceph orch upgrade start –ceph-version > 16.2.0’. I want to see if everything is ok after each command I issue. I > want to see if scrubbing stopped, I want to see if osd have correctly > accepted the new config. > > So you want to do all the things that an orchestrated rolling upgrade > does for you. Check. > > > I have a small cluster so I do not see this procedure as a waste of > time. If I look at your telemetry data[3]. I see 600 clusters with 35k > osd’s, that is an average of 60 osd per cluster. So these are quite > small clusters, I would think these admins have a similar point of view > as I have. > > Careful with those inferences. We are not operating here on some website cpanel project. Due to the size of these storage solutions one should expect lots of third party data is stored there. So I would argue you should not want to have 'I only know Kubernetes commands' sysadmins operating ceph. > > * Operators who submit telemetry may not be a representative sample > * Sites may have many more than one cluster, If one has a 20 OSD lab > cluster and a 1000 OSD production cluster, perspectives and processes > are going to be different than someone with a single 60 OSD cluster. > * Average != median Whatever, prove me wrong in that not the vast majority of clusters are small. Logics just dictates this. > > I am rather getting the impression you need to have an easy deployment > tool for ceph than you want to really utilize containers. First there > was this ceph-deploy and ceph-ansible which I luckily skipped both > > That’s more than a little harsh. A lot of people get a lot of value out > of those tools. > > The ceph daemons seem to be not prepared for container use, ceph > containers can’t use cpu/memory limits > > They don’t make julienne fries either. What of it? So the argument for cephadm to move to containers is non-sense. If there would be a true container aspiration, you would have to want to apply at least a few of the suggestions. > > And last but not least you totally bypass that the (ceph) admin should > choose the OC platform and not you, because he probably has more than > just ceph nodes. > > Nobody’s stopping you from rolling your own containers, using > traditional packages, or heck even deploying with tarballs. That’s the > beauty of OSS. Let’s leave Orange County out of it though. I agree, there should be no strong relationship between ceph and any OC. I tend to think that Kubernetes or other OC's should be responsible for offering a ceph implementation. > > So my question to you: What problem is it actually that your cephadm > dev team is trying to solve? That is not clear to me. > > Asked and answered, sir Could you write it to me like I am six year old? I do not get from your text what problem the cephadm team is trying to solve.

1 0

how to set rgw parameters in Pacific
by Adrian Nicolae 20 Jun '21

20 Jun '21

Hi, I have some doubts regarding the best way to change some rgw parameters in Pacific. Let's say I want to change rgw_max_put_size and some rgw_gc default values like rgw_gc_max_concurrent_io. What is the recommended way to do it : - via 'ceph config set global' or - via 'ceph config set client.rgw' ? Thanks.

2 1

Ceph Managers dieing?
by Peter Childs 17 Jun '21

17 Jun '21

Lets try to stop this message turning into a mass moaning session about Ceph and try and get this newbie able to use it. I've got a Ceph Octopus cluster, its relatively new and deployed using cephadm. It was working fine, but now the managers start up run for about 30 seconds and then die, until systemctl gives up and I have to reset-fail them to get them to try again, when they fail. How do I work out why and get them working again? I've got 21 nodes and was looking to take it up to 32 over the next few weeks, but that is going to be difficult if the managers are not working. I did try Pacific and I'm happy to upgrade but that failed to deploy more than 6 osd's and I gave up and went back to Octopus. I'm about to give up on Ceph because it looks like its really really "fragile" and debugging what's going wrong is really difficult. I guess I could give up on cephadm and go with a different provisioning method but I'm not sure where to start on that. Thanks in advance. Peter.

4 4

Pulling Ceph Data Into Grafana
by Alcatraz 17 Jun '21

17 Jun '21

Hello All, I recently installed Ceph (v 16.2.4 pacific stable). I know Ceph creates and exposes two Prometheus instances (from what I've witnessed). To that end, I installed Grafana in a docker container, and am attempting to pull metrics from Ceph (Cluster Health, OSD information, etc), but I'm running into issues. If I setup a Prometheus data source to $ServerIP:9100 (Ceph internal metrics, CPU/RAM usage, etc), no issue arises. It automatically parses the URL to http://$ServerIP:9100/api/v1/query, which is expected. If I go to that URL and look at the /metrics endpoint, I can see the raw metrics. If I setup a Prometheus data source to $ServerIP:9283 (Ceph Cluster Health, OSD info, etc), Grafana produces an error in the Web UI stating "Unknown error during query transaction. Please check JS console logs.". Now, I decided to enable debug logging in Grafana, and checked the logs during setup of both the data source into Ceph internal metrics, and Ceph Cluster Health metrics. The only difference is that it shoots out a 301 status, which doesn't make sense, because the URLs are the same, as are the /metrics endpoints. Log Excerpts are below: *URL pointing to Ceph Node Metrics:* t=2021-06-17T16:09:08+0000 lvl=dbug msg="Received command to update data source" logger=datasources url=http://$ServerIP:9100/ t=2021-06-17T16:09:08+0000 lvl=dbug msg="Applying default URL parsing for this data source type" logger=datasource type=prometheus url=http://$ServerIP:9100/ t=2021-06-17T16:09:08+0000 lvl=dbug msg="Querying for data source via SQL store" logger=datasources id=1 orgId=1 t=2021-06-17T16:09:08+0000 lvl=dbug msg="Applying default URL parsing for this data source type" logger=datasource type=prometheus url=http://$ServerIP:9100/ t=2021-06-17T16:09:08+0000 lvl=info msg=Requesting logger=data-proxy-log url=http://172.16.168.3:9100/api/v1/query *URL Pointing to Ceph Cluster Metrics:* sg="Received command to update data source" logger=datasources url=http://$ServerIP:9283 t=2021-06-17T16:06:32+0000 lvl=dbug msg="Applying default URL parsing for this data source type" logger=datasource type=prometheus url=http://$ServerIP:9283 t=2021-06-17T16:06:33+0000 lvl=dbug msg="Querying for data source via SQL store" logger=datasources id=1 orgId=1 t=2021-06-17T16:06:33+0000 lvl=dbug msg="Applying default URL parsing for this data source type" logger=datasource type=prometheus url=http://$ServerIP:9283 t=2021-06-17T16:06:33+0000 lvl=info msg=Requesting logger=data-proxy-log url=http://$ServerIP:9283/api/v1/query t=2021-06-17T16:06:33+0000 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=POST path=/api/datasources/proxy/1/api/v1/query status=301 remote_addr=$LANIP time_ms=3 size=131 referer=http://$ServerIP:3001/datasources/edit/yh9QZAg7k I've ran out of ideas to sort this, because I'd really like to get Grafana to show Ceph cluster metrics. Thanks, Preston

1 0

Re: radosgw - Etags suffixed with #x0e
by André Cruz 17 Jun '21

17 Jun '21

Hello Ingo. Did the problem actually went away after you upgraded everything to Nautilus? I’m seeing the same issue in a Luminous cluster where a Nautilus node was introduced (with the intent of upgrading the whole cluster to Nautilus). When the problem happened we had: Mons, Mgr - Nautilus OSDs, rgw - Most on Luminous, 1 on Nautilus Afterwards the Nautilus RGW was disabled, but still we left the Nautilus OSDs, and the problem has never happened again. There’s also this issue which seems related, but which implies that it can also happened on an all Nautilus cluster: https://tracker.ceph.com/issues/47451 <https://tracker.ceph.com/issues/47451> Best regards, André

1 0

Podman pull error 'access denied'
by Samy Ascha 17 Jun '21

17 Jun '21

Hi! I have a problem after starting to upgrade to 16.2.4, from 15.2.13. I started the upgrade and it successfully redeployed 2 out of 3 mgr daemon containers. The third failed to upgrade and Cephadm started retrying to upgrade it forever. The only way I could stop this was to disable the cephadm module. I found out I had an old version of podman installed and proceeded to upgrade it to one of the fitting versions according to the requirements docs. I have 3.0.1 installed now. This solved some issue with being unable to start containers, due to a failing 'get podman version' command. (The Go template did not fit the output of the older version of podman.) Ok, so now it got a little further in the process, but enabling the cephadm module would still start to retry the above action indefinitely. It now fails with this log: https://pastebin.com/p3T1fbjs <https://pastebin.com/p3T1fbjs> At first I thought it had something to do with rate limits on docker.io <http://docker.io/>, but it seems I can pull other stuff without problems. I also setup an account and played around with cephadm registry-login, but did not get much further. When looking at the pull command in the logs, I see it is using some ID for the container image that needs to be resolved, I suppose. Could it maybe make an error here, resulting in a bad URL that hits a resource that it is not supposed to hit, resulting in access errors? Any other thoughts on how to fix this error, or somehow make cephadm stop retrying this action and fixing it? Thanks very much and with regards, Samy

1 0

Ceph monitor won't start after Ubuntu update
by Petr 17 Jun '21

17 Jun '21

Hello Ceph-users, I've upgraded my Ubuntu server from 18.04.5 LTS to Ubuntu 20.04.2 LTS via 'do-release-upgrade', during that process ceph packages were upgraded from Luminous to Octopus and now ceph-mon daemon(I have only one) won't start, log error is: "2021-06-15T20:23:41.843+0000 7fbb55e9b540 -1 mon.target@-1(probing) e2 current monmap has recorded min_mon_release 12 (luminous) is >2 releases older than installed 15 (octopus); you can only upgrade 2 releases at a time you should first upgrade to 13 (mimic) or 14 (nautilus) stopping." Is there any way to get cluster running or at least get data from OSDs? Will appreciate any help. Thank you -- Best regards, Petr

2 3

Likely date for Pacific backport for RGW fix?
by Chris Palmer 17 Jun '21

17 Jun '21

Hi Our first upgrade (non-cephadm) from Octopus to Pacific 16.0.4 went very smoothly. Thanks for all the effort. The only thing that has bitten us is https://tracker.ceph.com/issues/50556 <https://tracker.ceph.com/issues/50556> which prevents a multipart upload to an RGW bucket that has a bucket policy. While I've been able to rewrite the most urgent scripts to use s3api put-object (which doesn't do multipart), that only works for objects up to a certain size. Removing the bucket policies isn't an option. I can see that it has been fixed, and is now pending backport (https://tracker.ceph.com/issues/51001 <https://tracker.ceph.com/issues/51001>). Will this be included in 16.0.5? And do we have an estimated date for that? We can wait a little longer, but otherwise I have to do some more drastic changes to an application. Having an indication of date would help me choose which... Many thanks, Chris

2 1

Re: JSON output schema
by Vladimir Prokofev 16 Jun '21

16 Jun '21

This is a great start, thank you! Basically I can look through the code to get the keys I need. But maybe I'm approaching this task wrong? Maybe there's already some better solution to monitor cluster health details? ср, 16 июн. 2021 г. в 02:47, Anthony D'Atri <anthony.datri(a)gmail.com>: > Before Luminous, mon clock skew was part of the health status JSON. With > Luminous and later releases, one has to invoke a separate command to get > the info. > > This is a royal PITA for monitoring / metrics infrastructure and I’ve > never seen a reason why it was done. > > You might find the code here > https://github.com/digitalocean/ceph_exporter > useful. Note that there are multiple branches, which can be confusing. > > > On Jun 15, 2021, at 4:21 PM, Vladimir Prokofev <v(a)prokofev.me> wrote: > > > > Good day. > > > > I'm writing some code for parsing output data for monitoring purposes. > > The data is that of "ceph status -f json", "ceph df -f json", "ceph osd > > perf -f json" and "ceph osd pool stats -f json". > > I also need support for all major CEPH releases, starting with Jewel till > > Pacific. > > > > What I've stumbled upon is that: > > - keys in JSON output are not present if there's no appropriate data. > > For example the key ['pgmap', 'read_bytes_sec'] will not be present in > > "ceph status" output if there's no read activity in the cluster; > > - some keys changed between versions. For example ['health']['status'] > key > > is not present in Jewel, but is available in all the following versions; > > vice-versa, key ['osdmap', 'osdmap'] is not present in Pacific, but is in > > all the previous versions. > > > > So I need to get a list of all possible keys for all CEPH releases. Any > > ideas how this can be achieved? My only thought atm is to build a > "failing" > > cluster with all the possible states and get a reference data out of it. > > Not only this is tedious work since it requires each possible cluster > > version, but it is also prone for error. > > Is there any publicly available JSON schema for output? > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > >

1 0

osd_scrub_max_preemptions for large OSDs or large EC pgs
by Dave Hall 16 Jun '21

16 Jun '21

Hello, I would like to ask about osd_scrub_max_preemptions in 14.2.20 for large OSDs (mine are 12TB) and/or large k+m EC pools (mine are 8+2). I did search the archives for this list, but I did not see any reference. Symptoms: I have been seeing a behavior in my cluster over the past 2 or 3 weeks where, for no apparent reason, there are suddenly slow ops, followed by a brief OSD down, massive but brief degradation/activating/peering, and then back to normal. I had thought this might have to do with some backfill activity due to a recently failed (as in down and out and process wouldn't start), but now all of that is over and the cluster is mostly back to HEALTH_OK. Thinking this might be something that was introduced between 14.2.9 and 14.2.16, I upgraded to 14.2.20 this morning. However, I just saw the same kind of event happen twice again. At the time, the only non-client activity was a single deep-scrub. Question: The description for osd_scrub_max_preemptions indicates that a deep scrub process will allow itself to be preempted a fixed number of times by client I/O and will then block client I/O until it finishes. Although I don't fully understand the deep scrub process, it seems that either the size of the HDD or the k+m count of the EC Pool could affect the time needed to complete a deep scrub and thus increase the likelihood that more than the default 5 preemptions will occur. Please tell me if my understanding is correct. If so, is there any guideline for increasing osd_scrub_max_preemptions just enough balance between scrub progress and client responsiveness? Or perhaps there are other scrub attributes that should be tuned instead? Thanks. -Dave -- Dave Hall Binghamton University kdhall(a)binghamton.edu

1 0

Strange (incorrect?) upmap entries in OSD map
by Andras Pataki 16 Jun '21

16 Jun '21

I've been working on some improvements to our large cluster's space balancing, when I noticed that sometimes the OSD maps have strange upmap entries. Here is an example on a clean cluster (PGs are active+clean): { "pgid": "1.1cb7", ... "up": [ 891, 170, 1338 ], "acting": [ 891, 170, 1338 ], ... }, with an upmap entry: pg_upmap_items 1.1cb7 [170,891] this would make the "up" list [ 170, 170, 1338 ], which isn't allowed. So the cluster just seems to ignore this upmap. When I remove the upmap, nothing changes in the PG state, and I can even re-insert it (without any effect). Any ideas why this upmap doesn't simply get rejected/removed? However, if I were to insert an upmap [170, 892], it gets rejected correctly (since 891 and 892 are on the same host - violating crush rules). Any insights would be helpful, Andras

1 0

ceph osd df return null
by julien lenseigne 16 Jun '21

16 Jun '21

Hi, when i do ceph osd df, some osd returns null size. For example : 0 hdd 7.27699 1.00000 0B 0B 0B 0B 0B 0B 0 0 29 1 hdd 7.27698 1.00000 0B 0B 0B 0B 0B 0B 0 0 23 11 ssd 0.50000 1.00000 0B 0B 0B 0B 0B 0B 0 0 49 2 hdd 7.27699 1.00000 0B 0B 0B 0B 0B 0B 0 0 30 4 hdd 7.27699 1.00000 0B 0B 0B 0B 0B 0B 0 0 37 12 ssd 0.50000 1.00000 0B 0B 0B 0B 0B 0B 0 0 39 7 hdd 24.55698 1.00000 24.6TiB 663GiB 663GiB 0B 0B 23.9TiB 2.64 0.79 302 42 hdd 10.69179 1.00000 10.7TiB 346GiB 344GiB 30.0MiB 1.13GiB 10.4TiB 3.16 0.95 98 43 hdd 10.69179 1.00000 10.7TiB 357GiB 355GiB 49.1MiB 1.17GiB 10.3TiB 3.26 0.98 131 44 hdd 10.69179 1.00000 10.7TiB 298GiB 297GiB 30.4MiB 1.05GiB 10.4TiB 2.72 0.81 92 45 hdd 10.69179 1.00000 10.7TiB 342GiB 341GiB 26.6MiB 1.11GiB 10.4TiB 3.12 0.94 105 40 ssd 0.87270 1.00000 894GiB 180GiB 179GiB 33.5MiB 990MiB 714GiB 20.10 6.02 53 41 ssd 0.87270 1.00000 894GiB 254GiB 253GiB 39.0MiB 1.00GiB 640GiB 28.38 8.50 61 46 ssd 0.87270 1.00000 894GiB 255GiB 254GiB 26.8MiB 1.03GiB 639GiB 28.55 8.55 76 3 hdd 24.55699 1.00000 24.6TiB 537GiB 536GiB 67.9MiB 1.36GiB 24.0TiB 2.14 0.64 252 do you know where it might come from ? Thanks. -- Julien Lenseigne Responsable informatique LMD Tel: 0169335172 Ecole Polytechnique, route de saclay, 91128 Palaiseau Bat 83 - Bureau 83.30.13

2 1

Drop old SDD / HDD Host crushmap rules
by Denny Fuchs 16 Jun '21

16 Jun '21

Hello, i have from the beginning on one DC very old crush map rules, to split HDD and SSD disks. It is obsolete since Luminous and I want to drop them: # ceph osd crush rule ls replicated_rule fc-r02-ssdpool fc-r02-satapool fc-r02-ssd ================= [ { "rule_id": 0, "rule_name": "replicated_rule", "ruleset": 0, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -1, "item_name": "default" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 1, "rule_name": "fc-r02-ssdpool", "ruleset": 1, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -15, "item_name": "r02-ssds" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 2, "rule_name": "fc-r02-satapool", "ruleset": 2, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -16, "item_name": "r02-sata" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] }, { "rule_id": 3, "rule_name": "fc-r02-ssd", "ruleset": 3, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -4, "item_name": "default~ssd" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] } ] ========================== # ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -14 0 root sata -18 0 datacenter fc-sata -16 0 rack r02-sata -13 18.55234 root ssds -17 18.55234 datacenter fc-ssds -15 18.55234 rack r02-ssds -6 3.09206 host fc-r02-ceph-osd-01 41 nvme 0.36388 osd.41 up 1.00000 1.00000 0 ssd 0.45470 osd.0 up 1.00000 1.00000 1 ssd 0.45470 osd.1 up 1.00000 1.00000 2 ssd 0.45470 osd.2 up 1.00000 1.00000 3 ssd 0.45470 osd.3 up 1.00000 1.00000 4 ssd 0.45470 osd.4 up 1.00000 1.00000 5 ssd 0.45470 osd.5 up 1.00000 1.00000 -2 3.09206 host fc-r02-ceph-osd-02 36 nvme 0.36388 osd.36 up 1.00000 1.00000 6 ssd 0.45470 osd.6 up 1.00000 1.00000 7 ssd 0.45470 osd.7 up 1.00000 1.00000 8 ssd 0.45470 osd.8 up 1.00000 1.00000 9 ssd 0.45470 osd.9 up 1.00000 1.00000 10 ssd 0.45470 osd.10 up 1.00000 1.00000 29 ssd 0.45470 osd.29 up 1.00000 1.00000 -5 3.45593 host fc-r02-ceph-osd-03 38 nvme 0.36388 osd.38 up 1.00000 1.00000 40 nvme 0.36388 osd.40 up 1.00000 1.00000 11 ssd 0.45470 osd.11 up 1.00000 1.00000 12 ssd 0.45470 osd.12 up 1.00000 1.00000 13 ssd 0.45470 osd.13 up 1.00000 1.00000 14 ssd 0.45470 osd.14 up 1.00000 1.00000 15 ssd 0.45470 osd.15 up 1.00000 1.00000 16 ssd 0.45470 osd.16 up 1.00000 1.00000 -9 3.09206 host fc-r02-ceph-osd-04 37 nvme 0.36388 osd.37 up 1.00000 1.00000 30 ssd 0.45470 osd.30 up 1.00000 1.00000 31 ssd 0.45470 osd.31 up 1.00000 1.00000 32 ssd 0.45470 osd.32 up 1.00000 1.00000 33 ssd 0.45470 osd.33 up 1.00000 1.00000 34 ssd 0.45470 osd.34 up 1.00000 1.00000 35 ssd 0.45470 osd.35 up 1.00000 1.00000 -11 2.72818 host fc-r02-ceph-osd-05 17 ssd 0.45470 osd.17 up 1.00000 1.00000 18 ssd 0.45470 osd.18 up 1.00000 1.00000 19 ssd 0.45470 osd.19 up 1.00000 1.00000 20 ssd 0.45470 osd.20 up 1.00000 1.00000 21 ssd 0.45470 osd.21 up 1.00000 1.00000 22 ssd 0.45470 osd.22 up 1.00000 1.00000 -25 3.09206 host fc-r02-ceph-osd-06 39 nvme 0.36388 osd.39 up 1.00000 1.00000 23 ssd 0.45470 osd.23 up 1.00000 1.00000 24 ssd 0.45470 osd.24 up 1.00000 1.00000 25 ssd 0.45470 osd.25 up 1.00000 1.00000 26 ssd 0.45470 osd.26 up 1.00000 1.00000 27 ssd 0.45470 osd.27 up 1.00000 1.00000 28 ssd 0.45470 osd.28 up 1.00000 1.00000 -1 0 root default =================================================== the question is now: what is the best way, to drop them and use the defaults from Ceph. Any suggestions ? cu denny

1 0

RADOSGW Keystone integration - S3 bucket policies targeting not just other tenants / projects ?
by Christian Rohmann 16 Jun '21

16 Jun '21

Hallo Ceph-Users, I've been wondering about the state of OpenStack Keystone Auth in RADOSGW. 1) Even though the general documentation on RADOSGW S3 bucket policies is a little "misleading" https://docs.ceph.com/en/latest/radosgw/bucketpolicy/#creation-and-removal in showing users being referred as Principal, the documentation about Keystone integration at https://docs.ceph.com/en/latest/radosgw/keystone/#integrating-with-openstac… clearly states, that "A Ceph Object Gateway user is mapped into a Keystone <tenant>"||. In the keystone authentication code it strictly only takes the project from the authenticating user: * https://github.com/ceph/ceph/blob/6ce6874bae8fbac8921f0bdfc3931371fc61d4ff/… * https://github.com/ceph/ceph/blob/6ce6874bae8fbac8921f0bdfc3931371fc61d4ff/… This is rather unfortunate as this renders the usually powerful S3 bucket policies to be rather basic with granting access to all users (with a certain role) of a project or more importantly all users of another project / tenant, as in using arn:aws:iam::$OS_REMOTE_PROJECT_ID:root as principal. Or am I just misreading anything here or is this really all that can be done if using native keystone auth? 2) There is a PR open implementing generic external authentication https://github.com/ceph/ceph/pull/34093 Apparently this seems to also address the lack of support for subusers for Keystone - if I understand this correctly I could then grant access to users arn:aws:iam::$OS_REMOTE_PROJECT_ID:$user Are there any plans on the roadmap to extend the functionality in regards to keystone as authentication backend? I know a similar question as been asked before (https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/GY7VUKCQ5QU…) but unfortunately with no discussion / responses then. Regards Christian

1 0

libceph: monX session lost, hunting for new mon
by Magnus HAGDORN 16 Jun '21

16 Jun '21

Hi all, I know this came up before but I couldn't find a resolution. We get the error libceph: monX session lost, hunting for new mon a lot on our samba servers that reexport cephfs. A lot means more than once a minute. On other machines that are less busy we get it about every 10-30 minutes. We only use a single network for both client and backend traffic on bonded 10GE links. So, my questions are: is this expected and normal behaviour? how to track this problem down? Regards magnus The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. Is e buidheann carthannais a th’ ann an Oilthigh Dhùn Èideann, clàraichte an Alba, àireamh clàraidh SC005336.

1 0

Strategy for add new osds
by Jorge JP 16 Jun '21

16 Jun '21

Hello, I have a ceph cluster with 5 nodes (1 hdd each node). I want to add 5 more drives (hdd) to expand my cluster. What is the best strategy for this? I will add each drive in each node but is a good strategy add one drive and wait to rebalance the data to new osd for add new osd? or maybe.. I should be add the 5 drives without wait rebalancing and ceph rebalancing the data to all new osd? Thank you.

5 4

Fwd: Re: Issues with Ceph network redundancy using L2 MC-LAG
by Joe Comeau 16 Jun '21

16 Jun '21

We also run with Dell VLT switches (40 GB) everything is active/active, so multiple paths as Andrew describes in his config Our config allows us: bring down one of the switches for upgrades bring down an iscsi gatway for patching all the while at least one path is up and servicing Thanks Joe >>> Andrew Walker-Brown <andrew_jbrown(a)hotmail.com> 6/15/2021 10:26 AM >>> With an unstable link/port you could see the issues you describe. Ping doesn’t have the packet rate for you to necessarily have a packet in transit at exactly the same time as the port fails temporarily. Iperf on the other hand could certainly show the issue, higher packet rate and more likely to have packets in flight at the time of a link fail...combined with packet loss/retries gives poor throughput. Depending on what you want to happen, there are a number of tuning options both on the switches and Linux. If you want the LAG to be down if any link fails, the you should be able to config this on the switches and/or Linux (minimum number of links = 2 if you have 2 links in the lag). You can also tune the link monitoring, how frequently the links are checked (e.g. miimon) etc. Bringing this value down from the default of 100ms may allow you to detect a link failure more quickly. But you then run into the chance if detecting a transient failure that wouldn’t have caused any issues....and the LAG becoming more unstable. Flapping/unstable links are the worst kind of situation. Ideally you’d pick that up quickly from monitoring/alerts and either fix immediately or take the link down until you can fix it. I run 2x10G from my hosts into separate switches (Dell S series – VLT between switches). Pulling a single interface has no impact on Ceph, any packet loss is tiny and we’re not exceeding 10G bandwidth per host. If you’re running 1G links and the LAG is already busy, a link failure could be causing slow writes to the host, just down to congestion...which then starts to impact the wider cluster based on how Ceph works. Just caveating the above with - I’m relatively new to Ceph myself.... Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for Windows 10 From: huxiaoyu(a)horebdata.cn<mailto:huxiaoyu@horebdata.cn> Sent: 15 June 2021 17:52 To: Serkan Çoban<mailto:cobanserkan@gmail.com> Cc: ceph-users<mailto:ceph-users@ceph.io> Subject: [ceph-users] Re: Issues with Ceph network redundancy using L2 MC-LAG When i pull out the cable, then the bond is working properly. Does it mean that the port is somehow flapping? Ping can still work, but the iperf test yields very low results. huxiaoyu(a)horebdata.cn From: Serkan Çoban Date: 2021-06-15 18:47 To: huxiaoyu(a)horebdata.cn CC: ceph-users Subject: Re: [ceph-users] Issues with Ceph network redundancy using L2 MC-LAG Do you observe the same behaviour when you pull a cable? Maybe a flapping port might cause this kind of behaviour, other than that you should't see any network disconnects. Are you sure about LACP configuration, what is the output of 'cat /proc/net/bonding/bond0' On Tue, Jun 15, 2021 at 7:19 PM huxiaoyu(a)horebdata.cn <huxiaoyu(a)horebdata.cn> wrote: > > Dear Cephers, > > I encountered the following networking issue several times, and i wonder whether there is a solution for networking HA solution. > > We build ceph using L2 multi chassis link aggregation group (MC-LAG ) to provide switch redundancy. On each host, we use 802.3ad, LACP > mode for NIC redundancy. However, we observe several times, when a single network port, either the cable, or the SFP+ optical module fails, Ceph cluster is badly affected by networking, although in theory it should be able to tolerate. > > Did i miss something important here? and how to really achieve networking HA in Ceph cluster? > > best regards, > > Samuel > > > > > huxiaoyu(a)horebdata.cn > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

4 4

Re: Fwd: Re: Issues with Ceph network redundancy using L2 MC-LAG
by Serkan Çoban 16 Jun '21

16 Jun '21

You cannot do much if the link is flapping or the cable is bad. Maybe you can write some rules to shut the port down on the switch if the error packet ratio goes up. I also remember there are some config on the switch side for link flapping. On Wed, Jun 16, 2021 at 10:57 AM huxiaoyu(a)horebdata.cn <huxiaoyu(a)horebdata.cn> wrote: > > Is it true that MC-LAG and 803.2ad, by its default, are working on active-active. > > What else should i take care to ensure fault tolerance when one path is bad? > > best regards, > > samuel > > > > huxiaoyu(a)horebdata.cn > > From: Joe Comeau > Date: 2021-06-15 23:44 > To: ceph-users(a)ceph.io > Subject: [ceph-users] Fwd: Re: Issues with Ceph network redundancy using L2 MC-LAG > We also run with Dell VLT switches (40 GB) > everything is active/active, so multiple paths as Andrew describes in > his config > Our config allows us: > bring down one of the switches for upgrades > bring down an iscsi gatway for patching > all the while at least one path is up and servicing > Thanks Joe > > > >>> Andrew Walker-Brown <andrew_jbrown(a)hotmail.com> 6/15/2021 10:26 AM > >>> > With an unstable link/port you could see the issues you describe. Ping > doesn’t have the packet rate for you to necessarily have a packet in > transit at exactly the same time as the port fails temporarily. Iperf > on the other hand could certainly show the issue, higher packet rate and > more likely to have packets in flight at the time of a link > fail...combined with packet loss/retries gives poor throughput. > > Depending on what you want to happen, there are a number of tuning > options both on the switches and Linux. If you want the LAG to be down > if any link fails, the you should be able to config this on the switches > and/or Linux (minimum number of links = 2 if you have 2 links in the > lag). > > You can also tune the link monitoring, how frequently the links are > checked (e.g. miimon) etc. Bringing this value down from the default of > 100ms may allow you to detect a link failure more quickly. But you then > run into the chance if detecting a transient failure that wouldn’t have > caused any issues....and the LAG becoming more unstable. > > Flapping/unstable links are the worst kind of situation. Ideally you’d > pick that up quickly from monitoring/alerts and either fix immediately > or take the link down until you can fix it. > > I run 2x10G from my hosts into separate switches (Dell S series – VLT > between switches). Pulling a single interface has no impact on Ceph, > any packet loss is tiny and we’re not exceeding 10G bandwidth per host. > > If you’re running 1G links and the LAG is already busy, a link failure > could be causing slow writes to the host, just down to > congestion...which then starts to impact the wider cluster based on how > Ceph works. > > Just caveating the above with - I’m relatively new to Ceph myself.... > > Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for > Windows 10 > > From: huxiaoyu(a)horebdata.cn<mailto:huxiaoyu@horebdata.cn> > Sent: 15 June 2021 17:52 > To: Serkan Çoban<mailto:cobanserkan@gmail.com> > Cc: ceph-users<mailto:ceph-users@ceph.io> > Subject: [ceph-users] Re: Issues with Ceph network redundancy using L2 > MC-LAG > > When i pull out the cable, then the bond is working properly. > > Does it mean that the port is somehow flapping? Ping can still work, > but the iperf test yields very low results. > > > > > > huxiaoyu(a)horebdata.cn > > From: Serkan Çoban > Date: 2021-06-15 18:47 > To: huxiaoyu(a)horebdata.cn > CC: ceph-users > Subject: Re: [ceph-users] Issues with Ceph network redundancy using L2 > MC-LAG > Do you observe the same behaviour when you pull a cable? > Maybe a flapping port might cause this kind of behaviour, other than > that you should't see any network disconnects. > Are you sure about LACP configuration, what is the output of 'cat > /proc/net/bonding/bond0' > > On Tue, Jun 15, 2021 at 7:19 PM huxiaoyu(a)horebdata.cn > <huxiaoyu(a)horebdata.cn> wrote: > > > > Dear Cephers, > > > > I encountered the following networking issue several times, and i > wonder whether there is a solution for networking HA solution. > > > > We build ceph using L2 multi chassis link aggregation group (MC-LAG ) > to provide switch redundancy. On each host, we use 802.3ad, LACP > > mode for NIC redundancy. However, we observe several times, when a > single network port, either the cable, or the SFP+ optical module fails, > Ceph cluster is badly affected by networking, although in theory it > should be able to tolerate. > > > > Did i miss something important here? and how to really achieve > networking HA in Ceph cluster? > > > > best regards, > > > > Samuel > > > > > > > > > > huxiaoyu(a)horebdata.cn > > _______________________________________________ > > ceph-users mailing list -- ceph-users(a)ceph.io > > > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Re: cephfs mount problems with 5.11 kernel - not a ipv6 problem
by Ilya Dryomov 16 Jun '21

16 Jun '21

On Mon, May 3, 2021 at 12:27 PM Magnus Harlander <magnus(a)harlan.de> wrote: > > Am 03.05.21 um 12:25 schrieb Ilya Dryomov: > > ceph osd setmaxosd 10 > > Bingo! Mount works again. > > Veeeery strange things are going on here (-: > > Thanx a lot for now!! If I can help to track it down, please let me know. Good to know it helped! I'll think about this some more and probably plan to patch the kernel client to be less stringent and not choke on this sort of misconfiguration. Thanks, Ilya

3 6

Docs on Containerized Mon Maintenance
by Phil Merricks 16 Jun '21

16 Jun '21

Hey folks, I'm working through some basic ops drills, and noticed what I think is an inconsistency in the Cephadm Docs. Some Googling appears to show this is a known thing, but I didn't find a clear direction on cooking up a solution yet. On a cluster with 5 mons, 2 were abruptly removed when their host OS decided to do scheduled maintenance without asking first. Those hosts only had mons running on them (and mds/crash/node exporter), so I still have 3 mon quorum and the cluster is happy. It's not clear to me how I add these hosts back in as mons though. In the troubleshooting docs it describes bringing all mons down, then extracting a monmap. I tried this through various iterations of bringing all down, bringing one back up and entering the container; bringing all down and trying to use ceph-mon from cephadm shell and so on. I either got rocksdb lock issues presumably because a mon node was running, or an error that the path to the mon data didn't exist, presumably for the opposite reason. Is there guidance on the container-friendly way to perform the monmap maintenance? I did think that because I still have quorum, I could simply do ceph orch apply mon label:mon instead, but I am nervous this might upset my remaining mons. Looking at the ceph orch ls output I see: root@kida:/# ceph orch ls NAME PORTS RUNNING REFRESHED AGE PLACEMENT alertmanager 1/1 7m ago 2h count:1 crash 5/5 9m ago 2h * grafana 1/1 7m ago 2h count:1 mds.media 3/3 9m ago 2h thebends;okcomputer;amnesiac mgr 2/2 9m ago 2h count:2 mon 3/5 9m ago 2h label:mon node-exporter 5/5 9m ago 2h * osd.all-available-devices 5/10 9m ago 2h * prometheus 1/1 7m ago 2h count:1 root@kida:/# So is it expecting 2 more mons, or has it autoscaled down cleverly? Looking at ceph orch ps I see: root@kida:/# ceph orch ps NAME HOST PORTS STATUS REFRESHED AGE VERSION IMAGE ID CONTAINER ID alertmanager.kida kida *:9093,9094 running (2h) 8m ago 2h 0.20.0 0881eb8f169f 89c604455194 crash.amnesiac amnesiac running (11h) 8m ago 11h 16.2.4 8d91d370c2b8 bff086c930db crash.kida kida running (2h) 8m ago 2h 16.2.4 8d91d370c2b8 b0ac059be109 crash.kingoflimbs kingoflimbs running (13h) 8m ago 13h 16.2.4 8d91d370c2b8 b0955309a8b9 crash.okcomputer okcomputer running (2h) 10m ago 2h 16.2.4 8d91d370c2b8 a75cf65ef235 crash.thebends thebends running (2h) 8m ago 2h 16.2.4 8d91d370c2b8 befe9c1015f3 grafana.kida kida *:3000 running (2h) 8m ago 2h 6.7.4 ae5c36c3d3cd f85747138299 mds.media.amnesiac.uujwlk amnesiac running (11h) 8m ago 2h 16.2.4 8d91d370c2b8 512a2fcc0f97 mds.media.okcomputer.nednib okcomputer running (2h) 10m ago 2h 16.2.4 8d91d370c2b8 10c6244a9308 mds.media.thebends.pqsfeb thebends running (2h) 8m ago 2h 16.2.4 8d91d370c2b8 c1b75831a973 mgr.kida.kchysa kida *:9283 running (2h) 8m ago 2h 16.2.4 8d91d370c2b8 602acc0d8df3 mgr.okcomputer.rjtrqw okcomputer *:8443,9283 running (2h) 10m ago 2h 16.2.4 8d91d370c2b8 605a8a25a604 mon.amnesiac amnesiac stopped 8m ago 2h <unknown> <unknown> <unknown> mon.kida kida running (2h) 8m ago 2h 16.2.4 8d91d370c2b8 a441563a978d mon.kingoflimbs kingoflimbs stopped 8m ago 2h <unknown> <unknown> <unknown> mon.okcomputer okcomputer running (2h) 10m ago 2h 16.2.4 8d91d370c2b8 c4297efafe27 mon.thebends thebends running (2h) 8m ago 2h 16.2.4 8d91d370c2b8 e2394d5f152b node-exporter.amnesiac amnesiac *:9100 running (11h) 8m ago 2h 0.18.1 e5a616e4b9cf da3c69057c4f node-exporter.kida kida *:9100 running (2h) 8m ago 2h 0.18.1 e5a616e4b9cf 5c9219a29257 node-exporter.kingoflimbs kingoflimbs *:9100 running (13h) 8m ago 2h 0.18.1 e5a616e4b9cf c2236491fb6e node-exporter.okcomputer okcomputer *:9100 running (2h) 10m ago 2h 0.18.1 e5a616e4b9cf 2e53a82eed32 node-exporter.thebends thebends *:9100 running (2h) 8m ago 2h 0.18.1 e5a616e4b9cf def6bdd359d6 osd.0 kida running (2h) 8m ago 2h 16.2.4 8d91d370c2b8 c1419a29ddd8 osd.1 kida running (85m) 8m ago 2h 16.2.4 8d91d370c2b8 dcb172c628ec osd.2 thebends running (2h) 8m ago 2h 16.2.4 8d91d370c2b8 4826e3da8d14 osd.3 okcomputer running (2h) 10m ago 2h 16.2.4 8d91d370c2b8 5424d437c270 osd.4 thebends running (2h) 8m ago 2h 16.2.4 8d91d370c2b8 47e682c3727d prometheus.kida kida *:9095 running (2h) 8m ago 2h 2.18.1 de242295e225 4c8e7fdd89a8 root@kida:/# So those mon containers are still there, stopped. ceph orch daemon restart mon.amnesiac gives notice that a restart is scheduled on that mon. The container status updates in ceph orch ps to running, but version, image ID and container ID are <unknown> and I don't see that mon unit in any status output or log. cephadm unit --name mon.amnesiac restart --fsid yadda-yadda-yadda errors with daemon not found, it seems like the cephadm cli command is scoped to the daemons running on the same host it's being executed on, rather than cluster-wide like ceph orch. Any clues offered to further investigation are welcomed. Best regards Phil

1 0

Mon crash when client mounts CephFS
by Phil Merricks 16 Jun '21

16 Jun '21

Hey folks, I have deployed a 3 node dev cluster using cephadm. Deployment went smoothly and all seems well. If I try to mount a CephFS from a client node, 2/3 mons crash however. I've begun picking through the logs to see what I can see, but so far other than seeing the crash in the log itself, it's unclear what the cause of the crash is. Here's a log. <https://termbin.com/isaz>. You can see where the crash is occurring around the line that begins with "Jun 08 18:56:04 okcomputer podman[790987]:" I would welcome any advice on either what the cause may be, or how I can advance the analysis of what's wrong. Best regards Phil

3 3

JSON output schema
by Vladimir Prokofev 15 Jun '21

15 Jun '21

Good day. I'm writing some code for parsing output data for monitoring purposes. The data is that of "ceph status -f json", "ceph df -f json", "ceph osd perf -f json" and "ceph osd pool stats -f json". I also need support for all major CEPH releases, starting with Jewel till Pacific. What I've stumbled upon is that: - keys in JSON output are not present if there's no appropriate data. For example the key ['pgmap', 'read_bytes_sec'] will not be present in "ceph status" output if there's no read activity in the cluster; - some keys changed between versions. For example ['health']['status'] key is not present in Jewel, but is available in all the following versions; vice-versa, key ['osdmap', 'osdmap'] is not present in Pacific, but is in all the previous versions. So I need to get a list of all possible keys for all CEPH releases. Any ideas how this can be achieved? My only thought atm is to build a "failing" cluster with all the possible states and get a reference data out of it. Not only this is tedious work since it requires each possible cluster version, but it is also prone for error. Is there any publicly available JSON schema for output?

1 0

Re: Issues with Ceph network redundancy using L2 MC-LAG
by Serkan Çoban 15 Jun '21

15 Jun '21

Do you observe the same behaviour when you pull a cable? Maybe a flapping port might cause this kind of behaviour, other than that you should't see any network disconnects. Are you sure about LACP configuration, what is the output of 'cat /proc/net/bonding/bond0' On Tue, Jun 15, 2021 at 7:19 PM huxiaoyu(a)horebdata.cn <huxiaoyu(a)horebdata.cn> wrote: > > Dear Cephers, > > I encountered the following networking issue several times, and i wonder whether there is a solution for networking HA solution. > > We build ceph using L2 multi chassis link aggregation group (MC-LAG ) to provide switch redundancy. On each host, we use 802.3ad, LACP > mode for NIC redundancy. However, we observe several times, when a single network port, either the cable, or the SFP+ optical module fails, Ceph cluster is badly affected by networking, although in theory it should be able to tolerate. > > Did i miss something important here? and how to really achieve networking HA in Ceph cluster? > > best regards, > > Samuel > > > > > huxiaoyu(a)horebdata.cn > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

4 4

How to orch apply single site rgw with custom front-end
by Vladimir Brik 15 Jun '21

15 Jun '21

Hello How can I use ceph orch apply to deploy single site rgw daemons with custom frontend configuration? Basically, I have three servers in a DNS round-robin, each running a 15.2.12 rgw daemon with this configuration: rgw_frontends = civetweb num_threads=5000 port=443s ssl_certificate=/etc/ceph/rgw.crt I would like to deploy 16.2.4 rgw daemons, but I don't know how to configure them. When I used "ceph orch apply rgw <name> <placement>", it created a new entry in the monitor configuration database instead of using existing rgw_frontends entry. I am guessing that I need to name the config db entry correctly, but I don't know what name to use. Currently I have $ ceph config get client rgw_frontends civetweb num_threads=5000 port=443s ssl_certificate=/etc/ceph/rgw.crt Can anybody help? Thanks, Vlad

1 0

problem using gwcli; package dependancy lockout
by Philip Brown 15 Jun '21

15 Jun '21

I'm trying to update a ceph octopus install, to add an iscsi gateway, using ceph-ansible, and gwcli wont run for me. The ansible run went well.. but when I try to actually use gwcli, I get (blahblah) ImportError: No module named rados which isnt too surprising, since "python-rados" is not installed. HOWEVER. The ceph repos installed by ceph-ansible (5.0, the octopus release) are http://download.ceph.com/ceph-iscsi/3/rpm/el7/noarch which provides python3-rados This supposedly "obsoletes" python-rados. Except it doesnt, because python-rados is for python2. But even if it didnt.. there is no provided python v2 rados module in the ceph-iscsi repo, or the main ceph repo. and gwcli is still python 2. So, what am I supposed to do now? It seems like I need a python2 version of python rados, but the ceph repos dont provide one, so they basically provide a broken gwcli? -- Philip Brown| Sr. Linux System Administrator | Medata, Inc. 5 Peters Canyon Rd Suite 250 Irvine CA 92606 Office 714.918.1310| Fax 714.918.1325 pbrown(a)medata.com| www.medata.com

1 0

Failover with 2 nodes
by nORKy 15 Jun '21

15 Jun '21

Hi, I'm building a lab with virtual machines. I build a set up with only 2 nodes, 2 osd per nodes and I have a host that use mount.cephfs Each 2 ceph nodes runs services mon + mgr + mds and has cephadm command. If I stop a node, all commands hang. Can't use dashboard, can't use ceph -s or any ceph command, and my cephfs on the third host stop to respond too (ex: with ls command) All come back, when I power on the stopped node Why is there no failover ?? Thanks you 'Jof

6 6

Re: Issues with Ceph network redundancy using L2 MC-LAG
by huxiaoyu＠horebdata.cn 15 Jun '21

15 Jun '21

My big worry is about, when a single link under a bond breaks, it breaks hardly such that the whole bond does not work. How to make it "failover" in such cases? best regards, samuel huxiaoyu(a)horebdata.cn From: Anthony D'Atri Date: 2021-06-15 18:22 To: huxiaoyu(a)horebdata.cn Subject: Re: [ceph-users] Issues with Ceph network redundancy using L2 MC-LAG Which hash mode are you using on the hosts? layer 3+4 ? Are they set up active/active, or active/passive? I often see suboptimal bonding configurations that result in most or all traffic going over only one ilnk. > On Jun 15, 2021, at 9:19 AM, huxiaoyu(a)horebdata.cn wrote: > > Dear Cephers, > > I encountered the following networking issue several times, and i wonder whether there is a solution for networking HA solution. > > We build ceph using L2 multi chassis link aggregation group (MC-LAG ) to provide switch redundancy. On each host, we use 802.3ad, LACP > mode for NIC redundancy. However, we observe several times, when a single network port, either the cable, or the SFP+ optical module fails, Ceph cluster is badly affected by networking, although in theory it should be able to tolerate. > > Did i miss something important here? and how to really achieve networking HA in Ceph cluster? > > best regards, > > Samuel > > > > > huxiaoyu(a)horebdata.cn > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

ceph PGs issues
by Aly, Adel 15 Jun '21

15 Jun '21

Dears, We have a ceph cluster with 4096 PGs out of with +100 PGs are not active+clean. On top of the ceph cluster, we have a ceph FS, with 3 active MDS servers. It seems that we can’t get all the files out of it because of the affected PGs. The object store has more than 400 million objects. When we do “rados -p cephfs-data ls”, the listing stops (hangs) after listing +11 million objects. When we try to access an object which we can’t copy, the rados command hangs forever: ls -I <filename> 2199140525188 printf "%x\n" 2199140525188 20006fd6484 rados -p cephfs-data stat 20006fd6484.00000000 (hangs here) This is the current status of the ceph cluster: health: HEALTH_WARN 1 MDSs report slow metadata IOs 1 MDSs report slow requests 1 MDSs behind on trimming 1 osds exist in the crush map but not in the osdmap *Reduced data availability: 22 pgs inactive, 22 pgs incomplete* 240324 slow ops, oldest one blocked for 391503 sec, daemons [osd.144,osd.159,osd.180,osd.184,osd.242,osd.271,osd.275,osd.278,osd.280,osd.332]... h ave slow ops. mons xyz01,xyz02 are low on available space services: mon: 4 daemons, quorum abc001,abc002,xyz02,xyz01 mgr: abc002(active), standbys: xyz01, xyz02, abc001 mds: cephfs-3/3/3 up {0=xyz02=up:active,1=abc001=up:active,2=abc002=up:active}, 1 up:standby osd: 421 osds: 421 up, 421 in; 7 remapped pgs data: pools: 2 pools, 4096 pgs objects: 403.4 M objects, 846 TiB usage: 1.2 PiB used, 1.4 PiB / 2.6 PiB avail pgs: 0.537% pgs not active 3968 active+clean 96 active+clean+scrubbing+deep+repair 15 incomplete 10 active+clean+scrubbing 7 remapped+incomplete io: client: 89 KiB/s rd, 13 KiB/s wr, 34 op/s rd, 1 op/s wr The 100+ PGs have been in this state for a long time already. Sometimes when we try to copy some files the rsync process hangs and we can’t kill it and from the process stack, it seems to be hanging on ceph i/o operation. # cat /proc/51795/stack [<ffffffffc184206d>] ceph_mdsc_do_request+0xfd/0x280 [ceph] [<ffffffffc181e92e>] __ceph_do_getattr+0x9e/0x200 [ceph] [<ffffffffc181eb08>] ceph_getattr+0x28/0x100 [ceph] [<ffffffffab853689>] vfs_getattr+0x49/0x80 [<ffffffffab8537b5>] vfs_fstatat+0x75/0xc0 [<ffffffffab853bc1>] SYSC_newlstat+0x31/0x60 [<ffffffffab85402e>] SyS_newlstat+0xe/0x10 [<ffffffffabd93f92>] system_call_fastpath+0x25/0x2a [<ffffffffffffffff>] 0xffffffffffffffff # cat /proc/51795/mem cat: /proc/51795/mem: Input/output error Any idea on how to move forward with debugging and fixing this issue so we can get the data out of the ceph FS? Thank you in advance. Kind regards, adel This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, Atos’ liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted. On all offers and agreements under which Atos Nederland B.V. supplies goods and/or services of whatever nature, the Terms of Delivery from Atos Nederland B.V. exclusively apply. The Terms of Delivery shall be promptly submitted to you on your request.

2 3

Issues with Ceph network redundancy using L2 MC-LAG
by huxiaoyu＠horebdata.cn 15 Jun '21

15 Jun '21

Dear Cephers, I encountered the following networking issue several times, and i wonder whether there is a solution for networking HA solution. We build ceph using L2 multi chassis link aggregation group (MC-LAG ) to provide switch redundancy. On each host, we use 802.3ad, LACP mode for NIC redundancy. However, we observe several times, when a single network port, either the cable, or the SFP+ optical module fails, Ceph cluster is badly affected by networking, although in theory it should be able to tolerate. Did i miss something important here? and how to really achieve networking HA in Ceph cluster? best regards, Samuel huxiaoyu(a)horebdata.cn

1 0

CephFS mount fails after Centos 8.4 Upgrade
by Ackermann, Christoph 15 Jun '21

15 Jun '21

Hello all, after upgrading Centos clients to version 8.4 CephFS ( Kernel 4.18.0-305.3.1.el8 ) mount did fail. Message: *mount error 110 = Connection timed out* ..unfortunately the kernel log was flooded with zeros... :-( The monitor connection seems to be ok, but libceph said: kernel: libceph: corrupt full osdmap (-2) epoch off xxxxxxxxxxxxx of yyyyyyyyyyyyyyy After client VM restore with Kernel 4.18.0-240.22.1.el8_3.x86_64 everything runs well. Did someone recently upgrade clients to Centos 8.4? Best regards, Christoph

2 1

slow ops at restarting OSDs (octopus)
by Manuel Lausch 15 Jun '21

15 Jun '21

Hello, I implemented a new cluster with 48 Nodes á 24 OSDs. I have a replicated pool with 4 replica. The crushrule distributes the replicas to different racks. With this cluster I tested a upgrade from Nautilis (14.2.20) to Octopus (15.2.13). The update itself worked well until I began the restarts of the OSDs in the 4th rack. Since then I get slow ops while stopping OSDs. I think something happend here, after all replica partners are running on the new version. This issue remains after completing the upgrade. With Nautilus I had similar issues with slow ops when stopping OSDs. I could resolve this with the option „osd_fast_shutdown → false“. I let this option set to false while upgrading. For testing/debugging, I set this to true (default value) and got better results when stopping OSDs, but the problem is not completely vanished. Had someone else this problem and could fix it? What can I do to get rid of slow ops when resarting OSDs? All Servers are connected with 2x10G network links Manuel

3 11

Re: ceph df: pool stored vs bytes_used -- raw or not?
by Konstantin Shalygin 15 Jun '21

15 Jun '21

> On 20 May 2021, at 19:47, Igor Fedotov <ifedotov(a)suse.de> wrote: > > which PR/ticket are you referring to? This thread https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/QMHLZOAY7LW… <https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/QMHLZOAY7LW…> And this ticket https://tracker.ceph.com/issues/48385 <https://tracker.ceph.com/issues/48385> All pools show me stored == used, even if all osd in/up/weighted on 14.2.21 RAW STORAGE: CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 595 TiB 334 TiB 261 TiB 261 TiB 43.87 nvme 52 TiB 34 TiB 18 TiB 18 TiB 34.68 TOTAL 647 TiB 368 TiB 279 TiB 279 TiB 43.13 POOLS: POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL replicated_rbd 1 1024 29 TiB 7.74M 29 TiB 10.93 80 TiB erasure_rbd_meta 6 16 2.9 MiB 154 2.9 MiB 0 7.3 TiB erasure_rbd_data 7 512 44 TiB 11.64M 44 TiB 15.61 144 TiB .rgw.root 12 16 11 KiB 22 11 KiB 0 80 TiB default.rgw.control 13 16 0 B 8 0 B 0 80 TiB default.rgw.meta 14 16 45 KiB 211 45 KiB 0 7.3 TiB default.rgw.log 15 16 88 MiB 345 88 MiB 0 80 TiB default.rgw.buckets.index 16 16 6.3 GiB 717 6.3 GiB 0.03 7.3 TiB default.rgw.buckets.non-ec 17 16 810 KiB 5.74k 810 KiB 0 80 TiB default.rgw.buckets.data 18 512 15 TiB 33.99M 15 TiB 5.70 80 TiB replicated_rbd_nvme 19 256 3.3 TiB 880.95k 3.3 TiB 13.25 7.3 TiB unredundant_rbd_nvme 20 128 3.9 TiB 1.03M 3.9 TiB 15.27 11 TiB fs_data 21 256 11 TiB 12.32M 11 TiB 4.38 80 TiB fs_meta 22 16 4.8 GiB 636.79k 4.8 GiB 0.02 7.3 TiB fs_data_nvme 23 64 20 GiB 5.97k 20 GiB 0.09 7.3 TiB kubernetes_rbd 24 32 142 GiB 36.71k 142 GiB 0.06 80 TiB Thanks, k

3 4

Module 'devicehealth' has failed:
by Torkil Svensgaard 15 Jun '21

15 Jun '21

Hi Looking at this error in v15.2.13: " [ERR] MGR_MODULE_ERROR: Module 'devicehealth' has failed: Module 'devicehealth' has failed: " It used to work. Since the module is always on I can't seem to restart it and I've found no clue as to why it failed. I've tried rebooting all hosts to no avail. Suggestions? Thanks, Torkil

2 2