October 2020 - ceph-users

TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES
by Seena Fallah 14 Oct '20

14 Oct '20

Hi all, Is TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES configured just for filestore or can be used for bluestore, too? https://github.com/ceph/ceph/blob/master/etc/default/ceph#L7 Thanks.

2 2

Proxmox+Ceph Benchmark 2020
by Alwin Antreich 14 Oct '20

14 Oct '20

Hello fellow Ceph users, we have released our new Ceph benchmark paper [0]. The used platform and Hardware is Proxmox VE 6.2 with Ceph Octopus on a new AMD Epyc Zen2 CPU with U.2 SSDs (details in the paper). The paper should illustrate the performance that is possible with a 3x node cluster without significant tuning. I welcome everyone to share their experience and add to the discussion, perferred on our forum [1] thread with our fellow Proxmox VE users. -- Cheers, Alwin [0] https://proxmox.com/en/downloads/item/proxmox-ve-ceph-benchmark-2020-09 [1] https://forum.proxmox.com/threads/proxmox-ve-ceph-benchmark-2020-09-hyper-c…

6 8

ceph orch apply rgw - rgw fails to boot
by Karsten Nielsen 14 Oct '20

14 Oct '20

Hi, I have been setting up a new cluster with a combination of cephadm and ceph orch. I have run in to a problem with rgw daemons that does not start. I have been following the documentation: https://docs.ceph.com/en/latest/cephadm/install/ - the RGW section ceph orch apply rgw ikea cn-dc9-1 --port=8080 --placement="3 itcnchn-bb4065 itcnchn-bb4066 itcnchn-bb4067" Then wait for everything to settle and then rgw's was expected to start on nodes itcnchn-bb4065 itcnchn-bb4066 itcnchn-bb4067 but that was not the case. ceph orch ps itcnchn-bb4065 NAME HOST STATUS REFRESHED AGE VERSION IMAGE NAME IMAGE ID CONTAINER ID alertmanager.itcnchn-bb4065 itcnchn-bb4065 running (3w) 2m ago 4w 0.20.0 prom/alertmanager:v0.20.0 0881eb8f169f c595826b8867 crash.itcnchn-bb4065 itcnchn-bb4065 running (4w) 2m ago 4w 15.2.4 docker.io/ceph/ceph:v15 852b28cb10de 4598d37bf91c grafana.itcnchn-bb4065 itcnchn-bb4065 running (4w) 2m ago 4w 6.6.2 ceph/ceph-grafana:latest 87a51ecf0b1c ad8e16d12ef8 mgr.itcnchn-bb4065.jxtcer itcnchn-bb4065 running (4w) 2m ago 4w 15.2.4 docker.io/ceph/ceph:v15 852b28cb10de d9278a7e772c mon.itcnchn-bb4065 itcnchn-bb4065 running (4w) 2m ago 4w 15.2.4 docker.io/ceph/ceph:v15 852b28cb10de e0edd4a9038a prometheus.itcnchn-bb4065 itcnchn-bb4065 running (3w) 2m ago 4w 2.18.1 prom/prometheus:v2.18.1 de242295e225 1103b62e191b rgw.ikea.cn-dc9-1.itcnchn-bb4065.xpwict itcnchn-bb4065 error 2m ago 22h <unknown> docker.io/ceph/ceph:v15 <unknown> <unknown> I have logged into itcnchn-bb4065 and tried to start the rgw container my self, the command from (/var/lib/ceph/0aba4c3a-f735-11ea-af33-02423ee04865/rgw.ikea.cn-dc9-1.itcnchn-bb4065.xpwict/unit.run) /usr/bin/docker run --rm --net=host --ipc=host --name ceph-0aba4c3a-f735-11ea-af33-02423ee04865-rgw.ikea.cn-dc9-1.itcnchn-bb4065.xpwict -e CONTAINER_IMAGE=docker.io/ceph/ceph:v15 -e NODE_NAME=itcnchn-bb4065 -v /var/run/ceph/0aba4c3a-f735-11ea-af33-02423ee04865:/var/run/ceph:z -v /var/log/ceph/0aba4c3a-f735-11ea-af33-02423ee04865:/var/log/ceph:z -v /var/lib/ceph/0aba4c3a-f735-11ea-af33-02423ee04865/crash:/var/lib/ceph/crash:z -v /var/lib/ceph/0aba4c3a-f735-11ea-af33-02423ee04865/rgw.ikea.cn-dc9-1.itcnchn-bb4065.xpwict:/var/lib/ceph/radosgw/ceph-rgw.ikea.cn-dc9-1.itcnchn-bb4065.xpwict:z -v /var/lib/ceph/0aba4c3a-f735-11ea-af33-02423ee04865/rgw.ikea.cn-dc9-1.itcnchn-bb4065.xpwict/config:/etc/ceph/ceph.conf:z --entrypoint /usr/bin/radosgw docker.io/ceph/ceph:v15 -n client.rgw.ikea.cn-dc9-1.itcnchn-bb4065.xpwict -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix="debug " debug 2020-10-14T06:55:54.835+0000 7f87075b1280 0 deferred set uid:gid to 167:167 (ceph:ceph) debug 2020-10-14T06:55:54.835+0000 7f87075b1280 0 ceph version 15.2.4 (7447c15c6ff58d7fce91843b705a268a1917325c) octopus (stable), process radosgw, pid 1 debug 2020-10-14T06:55:54.835+0000 7f87075b1280 0 framework: beast debug 2020-10-14T06:55:54.835+0000 7f87075b1280 0 framework conf key: port, val: 8080 debug 2020-10-14T06:55:54.835+0000 7f87075b1280 1 radosgw_Main not setting numa affinity debug 2020-10-14T06:55:54.875+0000 7f87075b1280 1 Cannot find zone id= (name=cn-dc9-1), switching to local zonegroup configuration debug 2020-10-14T06:55:54.875+0000 7f87075b1280 -1 Cannot find zone id= (name=cn-dc9-1) debug 2020-10-14T06:55:54.875+0000 7f87075b1280 0 ERROR: failed to start notify service ((22) Invalid argument debug 2020-10-14T06:55:54.875+0000 7f87075b1280 0 ERROR: failed to init services (ret=(22) Invalid argument) debug 2020-10-14T06:55:54.879+0000 7f87075b1280 -1 Couldn't init storage provider (RADOS) I am not really sure how to get past this point. Any pointers / help would be very appreciated. Thanks, - Karsten

1 1

Bucket sharding
by Szabo, Istvan (Agoda) 14 Oct '20

14 Oct '20

Hello, I have a bucket which is close to 10 millions objects (9.1 millions), we have: rgw_dynamic_resharding = false rgw_override_bucket_index_max_shards = 100 rgw_max_objs_per_shard = 100000 Do I need to increase the numbers soon or it is not possible so they need to start to use new bucket? The version is luminous 12.2.8. I don't want outage so want to be prepared. Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 2

Ceph test cluster, how to estimate performance.
by Daniel Mezentsev 14 Oct '20

14 Oct '20

Hi Ceph users, Im working on common lisp client utilizing rados library. Got some results, but don't know how to estimate if i am getting correct performance. I'm running test cluster from laptop - 2 OSDs - VM, RAM 4Gb, 4 vCPU each, monitors and mgr are running from the same VM(s). As for storage, i have Samsung SSD 860 Pro, 512G. Disk is splitted into 2 logical volumes (LVMs), and that volumes are attached to VMs. I know that i can't expect too much from that layout, just want to know if im getting adequate numbers. Im doing read/write operations on very small objects - up to 1kb. In async write im getting ~7.5-8.0 KIOPS. Synchronouse read - pretty much the same 7.5-8.0 KIOPS. Async read is segfaulting don't know why. Disk itself is capable to deliver well above 50 KIOPS. Difference is magnitude. Any info is more welcome. Daniel Mezentsev, founder (+1) 604 313 8592. Soleks Data Group. Shaping the clouds.

3 3

DocuBetter Meeting 14 Oct 2020 -- 24 hours from the time of this email.
by John Zachary Dover 13 Oct '20

13 Oct '20

There is a general documentation meeting called the "DocuBetter Meeting", and it is held every two weeks. The next DocuBetter Meeting will be on 14 Oct 2020 at 1630 UTC, and will run for thirty minutes. Everyone with a documentation-related request or complaint is invited. The meeting will be held here: https://bluejeans.com/908675367 Send documentation-related requests and complaints to me by replying to this email and CCing me at zac.dover(a)gmail.com. The next DocuBetter meeting is scheduled for: 14 Oct 2020 1630 UTC Etherpad: https://pad.ceph.com/p/Ceph_Documentation Zac's docs whiteboard: https://pad.ceph.com/p/docs_whiteboard Report Documentation Bugs: https://pad.ceph.com/p/Report_Documentation_Bugs Meeting: https://bluejeans.com/908675367

1 0

Ubuntu 20 with octopus
by Seena Fallah 13 Oct '20

13 Oct '20

Hi all, Does anyone has any production cluster with ubuntu 20 (focal) or any suggestion or any bugs that prevents to deploy Ceph octopus on Ubuntu 20? Thanks.

5 6

Announcing go-ceph v0.6.0
by John Mulligan 13 Oct '20

13 Oct '20

I'm happy to announce the another release of the go-ceph API bindings. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.6.0 Changes in the release are detailed in the link above. The bindings aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. These API bindings require the use of cgo. There are already a few consumers of this library in the wild, including the ceph-csi project. Specific questions, comments, bugs etc are best directed at our github issues tracker. -- John Mulligan phlogistonjohn(a)asynchrono.us jmulligan(a)redhat.com

1 0

BlueFS spillover detected - correct response for 14.2.7?
by Dave Hall 13 Oct '20

13 Oct '20

Hello, We are running a 14.2.7 cluster - 3 nodes with 24 OSDs, I've recently started getting 'BlueFS spillover detected'. I'm up to 3 OSDs in this state. In scanning through the various online sources I haven't been able to determine how to respond to this condition. Please advise. Thanks. -Dave -- Dave Hall Binghamton University kdhall(a)binghamton.edu

2 1

cephadm numa aware config
by nokia ceph 13 Oct '20

13 Oct '20

Hi Team, I would like to validate cephadm on bare metal and use docker/podman as a container. Currently we use NUMA aware config on bare metal to improve performance . Is there any config I can apply in cephadm to run podman/docker use run with *–cpuset-cpus*=*num * and *–cpuset-mems*=*nodes *options in startup scripts ? Thanks, Muthu

1 0

librados documentation has gone
by Daniel Mezentsev 13 Oct '20

13 Oct '20

Hi Ceph users, librados (C interface) disappered ? All google provided links are broken: \ SORRY / \ / \ This page does / ] not exist yet. [ ,'| ] [ / | ]___ ___[ ,' | ] ]\ /[ [ |: | ] ] \ / [ [ |: | ] ] ] [ [ [ |: | ] ] ]__ __[ [ [ |: | ] ] ] ]\ _ /[ [ [ [ |: | ] ] ] ] (#) [ [ [ [ :====' ] ] ]_].nHn.[_[ [ [ ] ] ] HHHHH. [ [ [ ] ] / `HH("N \ [ [ ]__]/ HHH " \[__[ ] NNN [ ] N/" [ ] N H [ / N \ / q, \ / \ Can somebody gimme heads-up what is going on ? Daniel Mezentsev, founder (+1) 604 313 8592. Soleks Data Group. Shaping the clouds.

2 2

Re: MONs are down, the quorum is unable to resolve.
by Brian Topping 12 Oct '20

12 Oct '20

I see, maybe you want to look at these instructions. I don’t know if you are running Rook, but the point about getting the container alive by using `sleep` is important. Then you can get into the container with `exec` and do what you need to. https://rook.io/docs/rook/v1.4/ceph-disaster-recovery.html#restoring-mon-qu… > On Oct 12, 2020, at 4:16 PM, Gaël THEROND <gael.therond(a)bitswalk.com> wrote: > > Hi Brian! > > Thanks a lot for your quick answer, it was fast ! > > Yes, I’ve read this doc, yet I can’t perform appropriate commands as my OSDs are up and running. > > As my mon is a container if I try to use ceph-mon —extract it won’t work as the mon process is running and if I stop it the container will be restarted and I’ll be ousted off it. > > I can’t retrieve anything from ceph mon getmap as the quorum isn’t forming. > > Yep, I know that I would need three nodes and I have a third node available since recently for this lab. > > unfortunately it’s a lab cluster and so one of my colleagues just took the third node for testing purpose... I told you, a series of unfortunate events :-) > > I can’t get rid of the cluster as I can’t lost OSDs data. > > G. > > Le mar. 13 oct. 2020 à 00:01, Brian Topping <brian.topping(a)gmail.com <mailto:brian.topping@gmail.com>> a écrit : > Hi there! > > This isn’t a difficult problem to fix. For purposes of clarity, the monmap is just a part of the monitor database. You generally have all the details correct though. > > Have you looked at the process in https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-mon/#… <https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-mon/#…> > > Please do make sure you are working on the copy of the monitor database with the newest epoch. After removing the other monitors and getting your cluster back online, you can re-add monitors at will. > > Also note that a quorum is defined as "one-half the total number of nodes plus one”. In your case, quorum is defined by both nodes! Taking either down would cause this problem. So you need to have an odd number of nodes to provide the ability to take a node down, for instance in a rolling upgrade. > > Hope that helps! > > Brian > > >> On Oct 12, 2020, at 3:54 PM, Gaël THEROND <gael.therond(a)bitswalk.com <mailto:gael.therond@bitswalk.com>> wrote: >> > > >> Hi everyone, >> >> Because of unfortunate events, I’ve a containers based ceph cluster >> (nautilus) in a bad shape. >> >> One of the lab cluster which is only made of 2 nodes as control plane (I >> know it’s bad :-)) each of these nodes run a mon, a mgr and a rados-gw >> containerized ceph_daemon. >> >> They were installed using ceph-ansible if relevant for anyone. >> >> However, when I was performing an upgrade on one of the first nodes, the >> second went down too (electrical power outage). >> >> As soon as I saw that I stopped all current process within the upgrading >> node. >> >> For now, if I try to restart my second node, as the quorum is looking for >> two node the cluster isn’t available. >> >> The container start, the node elect itself as the master but all ceph >> commands are stuck forever, which is perfectly normal as the quorum still >> wait for one member to achieve the election process etc. >> >> So, my question is, as I can’t (to my knowledge) extract the monmap with >> this intermediary state, and as my first node will still be considered as a >> known mon and try to join back if started properly, can I just copy the >> /etc/ceph.conf and /var/lib/mon/<host>/keyring from the last living node >> (the second one) and copy everything at its own place within the first >> node? My mon keys were the same for both mon initially and if I’m not >> making any mistakes my first node being blank will try to create a default >> store, join the existing cluster and try to retrieve the appropriate monmap >> from the remaining node right? >> >> If not, is there a process to be able to save/extract the monmap when using >> a container based ceph ? I can perfectly exec on the remaining node if it >> make any difference. >> >> Thanks a lot! > >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io <mailto:ceph-users@ceph.io> >> To unsubscribe send an email to ceph-users-leave(a)ceph.io <mailto:ceph-users-leave@ceph.io> >

2 1

MONs are down, the quorum is unable to resolve.
by Gaël THEROND 12 Oct '20

12 Oct '20

Hi everyone, Because of unfortunate events, I’ve a containers based ceph cluster (nautilus) in a bad shape. One of the lab cluster which is only made of 2 nodes as control plane (I know it’s bad :-)) each of these nodes run a mon, a mgr and a rados-gw containerized ceph_daemon. They were installed using ceph-ansible if relevant for anyone. However, when I was performing an upgrade on one of the first nodes, the second went down too (electrical power outage). As soon as I saw that I stopped all current process within the upgrading node. For now, if I try to restart my second node, as the quorum is looking for two node the cluster isn’t available. The container start, the node elect itself as the master but all ceph commands are stuck forever, which is perfectly normal as the quorum still wait for one member to achieve the election process etc. So, my question is, as I can’t (to my knowledge) extract the monmap with this intermediary state, and as my first node will still be considered as a known mon and try to join back if started properly, can I just copy the /etc/ceph.conf and /var/lib/mon/<host>/keyring from the last living node (the second one) and copy everything at its own place within the first node? My mon keys were the same for both mon initially and if I’m not making any mistakes my first node being blank will try to create a default store, join the existing cluster and try to retrieve the appropriate monmap from the remaining node right? If not, is there a process to be able to save/extract the monmap when using a container based ceph ? I can perfectly exec on the remaining node if it make any difference. Thanks a lot!

2 1

Long heartbeat ping times
by Frank Schilder 12 Oct '20

12 Oct '20

Dear all, occasionally, I find messages like Health check update: Long heartbeat ping times on front interface seen, longest is 1043.153 msec (OSD_SLOW_PING_TIME_FRONT) in the cluster log. Unfortunately, I seem to be unable to find out which OSDs were affected (a-posteriori). I cannot find related messages in any OSD log and the messages I find in /var/log/messages do not contain IP addresses or OSD IDs. Is there a way to find out which OSDs/hosts were the problem after health status is back to healthy? Thanks! ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

1 0

Re: Cluster under stress - flapping OSDs?
by Kristof Coucke 12 Oct '20

12 Oct '20

Diving into the different logging and searching for answers, I came across the following: PG_DEGRADED Degraded data redundancy: 2101057/10339536570 objects degraded (0.020%), 3 pgs degraded, 3 pgs undersized pg 1.4b is stuck undersized for 63114.227655, current state active+undersized+degraded+remapped+backfilling, last acting [62,20,33,25,97,2,159,2147483647,88] pg 1.115 is stuck undersized for 67017.759147, current state active+undersized+degraded+remapped+backfilling, last acting [2147483647,6,28,48,171,160,51,7,84] pg 1.1ec is stuck undersized for 67017.772311, current state active+undersized+degraded+remapped+backfilling, last acting [65,82,2147483647,161,6,36,105,106,48] Note the PG# 2147483647... That doesn't seem correct. Any ideas?

2 1

Re: Cluster under stress - flapping OSDs?
by Kristof Coucke 12 Oct '20

12 Oct '20

I'll answer it myself: When CRUSH fails to find enough OSDs to map to a PG, it will show as a 2147483647 which is ITEM_NONE or no OSD found.

1 0

Re: Ubuntu 20 with octopus
by Seena Fallah 12 Oct '20

12 Oct '20

I've seen this PR that reverts the latest ubuntu version from 20.04 to 18.04 because of some failures! Are there any updates on this? https://github.com/ceph/ceph/pull/35110 On Mon, Oct 12, 2020 at 4:11 AM Robert Ruge <robert.ruge(a)deakin.edu.au> wrote: > I am using Ubuntu 20.04 LTS for a five node 1PB cephfs setup with no > problems that I can attribute to using Ubuntu 20. > > Regards > Robert Ruge > > > > -----Original Message----- > From: Seena Fallah <seenafallah(a)gmail.com> > Sent: Monday, 12 October 2020 11:35 AM > To: ceph-users <ceph-users(a)ceph.io> > Subject: [ceph-users] Re: Ubuntu 20 with octopus > > The main reason I asked is because I don’t see any Ubuntu 20 in this doc > https://docs.ceph.com/en/latest/start/os-recommendations/ > > On Mon, Oct 12, 2020 at 4:01 AM Seena Fallah <seenafallah(a)gmail.com> > wrote: > > > Hi all, > > > > Does anyone has any production cluster with ubuntu 20 (focal) or any > > suggestion or any bugs that prevents to deploy Ceph octopus on Ubuntu 20? > > > > Thanks. > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > > Important Notice: The contents of this email are intended solely for the > named addressee and are confidential; any unauthorised use, reproduction or > storage of the contents is expressly prohibited. If you have received this > email in error, please delete it and any attachments immediately and advise > the sender by return email or telephone. > > Deakin University does not warrant that this email and any attachments are > error or virus free. >

2 1

Cluster under stress - flapping OSDs?
by Kristof Coucke 12 Oct '20

12 Oct '20

Hi all, We're now having trouble over a week with our Ceph cluster. Short info regarding our situation: - Original cluster had 10 OSD nodes, each having 16 OSDs - Expansion was necessary, so another 6 nodes have been added - Version: 14.2.11 Last week we saw heavily loaded OSD servers, after help here we identified the disk load being too high due to compaction of the Rocks.db. Taking the disk offline and running ceph-kvstore-tool bluestore-kv /var/lib/ceph/osd/xxxx compact does take the load away temporarily as mentioned. Most of the new disks still have a weight of 0 as we want to get the system stable first, but there is something I simply don't understand. When setting the following flags: noout, norecover, nobackfill and norebalance prior taking the disk offline for compaction, we still get a raise in PGs degraded after the OSD is back marked as "up". This night, a flapping OSD was also temporarily marked as "offline", I assume because it was heavily loaded, causing again a rise in degraded PGs. I know that there is a flag "nodown", but I've never used it. Reading the docs, it states these flags are "temporary" and the blocking action will be performed anyhow afterwards... So I have a few questions: 1. Why is the cluster marking PGs as "degraded" and indicating degraded data redundancy, while this was not the case before? It rises (2196398/10339524249 objects degraded (0.021%)) and I simply cannot understand why it keeps going up... 2. The flag "nodown". Can I use this to prevent the flapping... I don't want to get a deeper mess. As far as I understood, it would help in our case as the OSDs are heavily used. 3. Is it a good idea to start adding the other disks as well (slowly increasing their weight)? Thansk, Kristof

1 0

Cephdeploy support
by Amudhan P 11 Oct '20

11 Oct '20

Hi, Future releases of Ceph support cephdeploy or only Cephadm will be the choice. Thanks, Amudhan

1 0

Is cephfs multi-volume support stable?
by Alexander E. Patrakov 10 Oct '20

10 Oct '20

Hello, I found that documentation on the Internet on the question whether I can safely have two instances of cephfs in my cluster is inconsistent. For the record, I don't use snapshots. FOSDEM 19 presentation by Sage Weil: https://archive.fosdem.org/2019/schedule/event/ceph_project_status_update/a… Slide 25 is specifically devoted to this topic, and declares multi-volume support as stable. But, https://docs.ceph.com/en/nautilus/cephfs/experimental-features/ declares that multiple filesystems in the same cluster are an experimental feature, and the "latest" version of the same doc makes the same claim. What should I believe - the presentation or the official docs? -- Alexander E. Patrakov CV: http://pc.cd/PLz7

2 1

Q on enabling application on the pool
by Void Star Nill 10 Oct '20

10 Oct '20

Hello, What is the necessity for enabling the application on the pool? As per the documentation, we need to enable application before using the pool. However, in my case, I have a single pool running on the cluster used for RBD. I am able to run all RBD operations on the pool even if I dont enable the application. So what is the downside of not enabling the application on a pool? Regards, Shridhar

1 0

Monitor recovery
by Brian Topping 10 Oct '20

10 Oct '20

Hello experts, I have accidentally created a situation where the only monitor in a cluster has been moved to a new node without it’s /var/lib/ceph contents. Not realizing what I had done, I decommissioned the original node, but still have the contents of it’s /var/lib/ceph. Can I shut down the monitor running on the new node, copy monitor data from the original node to the new node and restart the monitor? Or is there information in the monitor database that is tied to the original node? If that’s the case, I suspect I need to somehow recommission the original node. Thanks for any feedback on this situation! Brian

2 2

How to clear Health Warning status?
by Tecnología CHARNE.NET 10 Oct '20

10 Oct '20

Hello! Today, I started the morning with a WARNING STATUS on our Ceph cluster. # ceph health detail HEALTH_WARN Too many repaired reads on 1 OSDs [WRN] OSD_TOO_MANY_REPAIRS: Too many repaired reads on 1 OSDs osd.67 had 399911 reads repaired I made "ceph osd out 67" and PGs where migrated to another OSDs. I stopped the osd.67 daemon, inspected the logs, etc... Then I started the daemon and made "# ceph osd in 67". OSD started backfilling with some PGs and no other error appeared in the rest of the day, but Warning status still remains. Can I clear it? Shoud I remove the osd and start with a new one? Thanks in advance for your time! Javier.-

2 2

Re: another osd_pglog memory usage incident
by Dan van der Ster 09 Oct '20

09 Oct '20

On Fri, Oct 9, 2020 at 3:12 PM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > > > > > >1. The pg log contains 3000 entries by default (on nautilus). These > >3000 entries can legitimately consume gigabytes of ram for some > >use-cases. (I haven't determined exactly which ops triggered this > >today). > > How can I check how much ram my pg_logs are using? ceph daemon osd.x dump_mempools | jq .mempool.by_pool.osd_pglog > > > > -----Original Message----- > Cc: ceph-users > Subject: [ceph-users] Re: another osd_pglog memory usage incident > > On 09.10.20 13:55, Dan van der Ster wrote: > [...] > > I also noticed a possible relationship with scrubbing -- One week ago > > we increased to osd_max_scrubs=5 to clear out a scrubbing backlog; I > > wonder if the increased read/write ratio somehow led to an exploding > > buffer_anon. Do things stabilize on your side if you temporarily > > disable scrubbing? > > During the worst periods, we had disabled scrubbing. When we re-enabled, > we had our write-job to mitigate the problems. And currently, scrub load > is low. So I cannot tell, but it is very plausible. > > Cheers > Harry > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > >

1 0

Multisite replication speed
by Nicolas Moal 09 Oct '20

09 Oct '20

Hello everybody, We have two Ceph object clusters replicating over a very long-distance WAN link. Our version of Ceph is 14.2.10. Currently, replication speed seems to be capped around 70 MiB/s even if there's a 10Gb WAN link between the two clusters. The clusters themselves don't seem to suffer from any performance issue. The replication traffic leverages HAProxy VIPs, which means there's a single endpoint (the HAProxy VIP) in the multisite replication configuration. So, my questions are: - Is it possible to improve replication speed by adding more endpoints in the multisite replication configuration? The issue we are facing is that the secondary cluster is way behind the master cluster because of the relatively slow speed. - Is there anything else I can do to optimize replication speed ? Thanks for your comments ! Nicolas

3 4

another osd_pglog memory usage incident
by Dan van der Ster 09 Oct '20

09 Oct '20

Hi all, This morning some osds in our S3 cluster started going OOM, after restarting them I noticed that the osd_pglog is using >1.5GB per osd. (This is on an osd with osd_memory_target = 2GB, hosting 112PGs, all PGs are active+clean). After reading through this list and trying a few things, I'd like to share the following observations for your feedback: 1. The pg log contains 3000 entries by default (on nautilus). These 3000 entries can legitimately consume gigabytes of ram for some use-cases. (I haven't determined exactly which ops triggered this today). 2. The pg log length is decided by the primary osd -- setting osd_max_pg_log_entries/osd_min_pg_log_entries on one single OSD does not have a big effect (because most of the PGs are primaried somewhere else). You need to set it on all the osds for it to be applied to all PGs. 3. We eventually set osd_max_pg_log_entries = 500 everywhere. This decreased the osd_pglog mempool from more than 1.5GB on our largest osds to less that 500MB. 4. The osd_pglog mempool is not accounted for in the osd_memory_target (in nautilus). 5. I have opened a feature request to limit the pg_log length by memory size (https://tracker.ceph.com/issues/47775) This way we could allocate a fraction of memory to the pg log and it would shorten the pglog length (budget) accordingly. 6. Would it be feasible to add an osd option to 'trim pg log at boot' ? This way we could avoid the cumbersome ceph-objectstore-tool trim-pg-log in cases of disaster (osds going oom at boot). For those that had pglog memory usage incidents -- does this match your experience? Thanks! Dan

4 7

Nautilus RGW fails to open Jewel buckets (400 Bad Request)
by Wido den Hollander 09 Oct '20

09 Oct '20

Hi, Most of it is described here: https://tracker.ceph.com/issues/22928 Buckets created under Jewel don't always have the *placement_rule* set in their bucket metadata and this causes Nautilus RGWs to not serve requests for them. Snippet from the metadata: { "key": "bucket.instance:pbx:ams02.446941181.1", "ver": { "tag": "86lc3iVtQpPiJYkh95YCTnhu", "ver": 2 }, "mtime": "2020-10-09 09:12:04.744423Z", "data": { "bucket_info": { "bucket": { "name": "pbx", "marker": "ams02.241978.4", "bucket_id": "ams02.446941181.1", "tenant": "", "explicit_placement": { "data_pool": ".rgw.buckets", "data_extra_pool": "", "index_pool": ".rgw.buckets" } }, "creation_time": "2014-02-16 12:32:15.000000Z", "owner": "vdvm", "flags": 0, "zonegroup": "eu", "placement_rule": "", Notice that *placement_rule* is empty and that this bucket has *explicit_placement* set. There is no way to update the bucket.instance metadata as far as I know, otherwise I could have set a placement rule for the bucket. Earlier on the ML this has been discussed: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/ULKK5RU2VXL… People there compiled a manual version of RGW, something I'd rather stay away from. Has anybody seen this and if so: Have you found a solution? The commit that breaks these buckets is this one: https://github.com/ceph/ceph/commit/2a8e8a98d8c56cc374ec671846a20e2b0484bc75 14.2.0 was the first release with that code in there. So two things I'm thinking about and I don't know which one is best: - Update RGW and modify the if-statement added by commit 2a8e8a - Enhance 'bucket check --fix' to update the placement_rule if none is set for a bucket Any hints or suggestions? Wido

1 0

Weird performance issue with long heartbeat and slow ops warnings
by Void Star Nill 09 Oct '20

09 Oct '20

Hello, I have a ceph cluster running 14.2.11. I am running benchmark tests with FIO concurrently on ~2000 volumes of 10G each. During the time initial warm-up FIO creates a 10G file on each volume before it runs the actual read/write I/O operations. During this time, I start seeing the Ceph cluster reporting about 35GiB/s write throughput for a while, but after some time I start seeing "long heartbeat" and "slow ops" warnings and in a few mins the throughput drops to ~1GB/s and stays there until all FIO runs complete. The cluster has 5 monitor nodes and 10 data nodes - each with 10x3.2TB NVME drives. I have setup 3 OSD for each NVME, so there are a total of 300 OSDs. Each server has 200GB uplink and there's no apparent network bottleneck as the network is set up to support over 1Tbps bandwidth. I dont see any CPU or memory issues also on the servers. There is a single manager instance running on one of the mons. The pool is configured for 3 replication factor with min_size of 2. I tried to use pg_num of 8192 and 16384 and saw the issue with both settings. Could you please suggest if this is a known issue or if I can tune any parameters? Long heartbeat ping times on back interface seen, longest is 1202.120 msec Long heartbeat ping times on front interface seen, longest is 1535.191 msec 35 slow ops, oldest one blocked for 122 sec, daemons [osd.135,osd.14,osd.141,osd.143,osd.149,osd.15,osd.151,osd.153,osd.157,osd.162]... have slow ops. Regards, Shridhar

2 2

pg active+clean but can not handle io
by 古轶特 09 Oct '20

09 Oct '20

Hi all: I have a ceph cluster, the version is 12.2.12. this is my ceph osd tree: [root@node-1 ~]# ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -25 2.78760 root rack-test -26 0.92920 rack rack_1 -37 0.92920 host host_1 12 ssd 0.92920 osd.12 up 1.00000 1.00000 -27 0.92920 rack rack_2 -38 0.92920 host host_2 6 ssd 0.92920 osd.6 up 1.00000 1.00000 -28 0.92920 rack rack_3 -39 0.92920 host host_3 18 ssd 0.92920 osd.18 up 1.00000 1.00000 I have a pool in cluster: pool 14 'gyt-test' replicated size 2 min_size 1 crush_rule 2 object_hash rjenkins pg_num 128 pgp_num 128 last_change 5864 lfor 0/5828 flags hashpspool stripe_width 0 removed_snaps [1~3] crush_rule 2 dump: { "rule_id": 2, "rule_name": "replicated_rule_rack", "ruleset": 2, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -25, "item_name": "rack-test" }, { "op": "chooseleaf_firstn", "num": 0, "type": "rack" }, { "op": "emit" } ] } I have a rbd images in gyt-test pool: [root@node-1 ~]# rbd ls gyt-test gyt-test now, I use fio tool to test this rbd images: [root@node-1 ~]# fio --ioengine=rbd --pool=gyt-test --rbdname=gyt-test --rw=randwrite --bs=4k --numjobs=1 --runtime=120 --iodepth=128 --clientname=admin --direct=1 --name=test --time_based=1 --eta-newline 1 test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=rbd, iodepth=128 fio-3.1 Starting 1 process Jobs: 1 (f=1): [w(1)][2.5%][r=0KiB/s,w=42.2MiB/s][r=0,w=10.8k IOPS][eta 01m:57s] Jobs: 1 (f=1): [w(1)][4.2%][r=0KiB/s,w=57.7MiB/s][r=0,w=14.8k IOPS][eta 01m:55s] Jobs: 1 (f=1): [w(1)][5.8%][r=0KiB/s,w=52.4MiB/s][r=0,w=13.4k IOPS][eta 01m:53s] Jobs: 1 (f=1): [w(1)][7.5%][r=0KiB/s,w=61.1MiB/s][r=0,w=15.6k IOPS][eta 01m:51s] Jobs: 1 (f=1): [w(1)][9.2%][r=0KiB/s,w=30.0MiB/s][r=0,w=7927 IOPS][eta 01m:49s] Jobs: 1 (f=1): [w(1)][10.8%][r=0KiB/s,w=59.1MiB/s][r=0,w=15.1k IOPS][eta 01m:47s] Jobs: 1 (f=1): [w(1)][12.5%][r=0KiB/s,w=51.6MiB/s][r=0,w=13.2k IOPS][eta 01m:45s] Jobs: 1 (f=1): [w(1)][14.2%][r=0KiB/s,w=58.3MiB/s][r=0,w=14.9k IOPS][eta 01m:43s] Jobs: 1 (f=1): [w(1)][15.8%][r=0KiB/s,w=56.1MiB/s][r=0,w=14.4k IOPS][eta 01m:41s] Jobs: 1 (f=1): [w(1)][17.5%][r=0KiB/s,w=44.8MiB/s][r=0,w=11.5k IOPS][eta 01m:39s] This is normal And then, I move host_1 bucket to rack-test: [root@node-1 ~]# ceph osd crush move host_1 root=rack-test moved item id -37 name 'host_1' to location {root=gyt-test} in crush map [root@node-1 ~]# ceph osd tree -25 2.78760 root rack-test -37 0.92920 host host_1 12 ssd 0.92920 osd.12 up 1.00000 1.00000 -26 0 rack rack_1 -27 0.92920 rack rack_2 -38 0.92920 host host_2 6 ssd 0.92920 osd.6 up 1.00000 1.00000 -28 0.92920 rack rack_3 -39 0.92920 host host_3 18 ssd 0.92920 osd.18 up 1.00000 1.00000 use fio tool test gyt-test rbd again: [root@node-1 ~]# fio --ioengine=rbd --pool=gyt-test --rbdname=gyt-test --rw=randwrite --bs=4k --numjobs=1 --runtime=120 --iodepth=64 --clientname=admin --direct=1 --name=test --time_based=1 --eta-newline 1 test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=rbd, iodepth=64 fio-3.1 Starting 1 process Jobs: 1 (f=1): [w(1)][2.5%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 01m:57s] Jobs: 1 (f=1): [w(1)][4.2%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 01m:55s] Jobs: 1 (f=1): [w(1)][5.8%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 01m:53s] Jobs: 1 (f=1): [w(1)][7.5%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 01m:51s] Jobs: 1 (f=1): [w(1)][9.2%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 01m:49s] Jobs: 1 (f=1): [w(1)][10.8%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 01m:47s] Jobs: 1 (f=1): [w(1)][12.5%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 01m:45s] Jobs: 1 (f=1): [w(1)][14.2%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 01m:43s] Jobs: 1 (f=1): [w(1)][15.8%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 01m:41s] Jobs: 1 (f=1): [w(1)][17.5%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 01m:39s] Jobs: 1 (f=1): [w(1)][19.2%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 01m:37s] Jobs: 1 (f=1): [w(1)][20.8%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 01m:35s] Jobs: 1 (f=1): [w(1)][22.5%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 01m:33s] Jobs: 1 (f=1): [w(1)][24.2%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 01m:31s] Jobs: 1 (f=1): [w(1)][25.8%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 01m:29s] Jobs: 1 (f=1): [w(1)][27.5%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 01m:27s] Jobs: 1 (f=1): [w(1)][29.2%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 01m:25s] Jobs: 1 (f=1): [w(1)][30.8%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 01m:23s] Jobs: 1 (f=1): [w(1)][32.5%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 01m:21s] Jobs: 1 (f=1): [w(1)][34.2%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 01m:19s] Jobs: 1 (f=1): [w(1)][35.8%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 01m:17s] Jobs: 1 (f=1): [w(1)][37.5%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 01m:15s] Jobs: 1 (f=1): [w(1)][39.2%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 01m:13s] Jobs: 1 (f=1): [w(1)][40.8%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 01m:11s] rbd gyt-test can not handle io ceph status at this time: [root@node-1 ~]# ceph -s cluster: id: 896eeeed-6f14-4fb0-8d0d-00adda67a36c health: HEALTH_WARN 8782/299948 objects misplaced (2.928%) services: mon: 3 daemons, quorum node-1,node-2,node-3 mgr: node-1(active) osd: 18 osds: 18 up, 18 in; 75 remapped pgs flags nodeep-scrub rbd-mirror: 1 daemon active rgw: 1 daemon active data: pools: 14 pools, 1024 pgs objects: 104.25k objects, 371GiB usage: 641GiB used, 11.5TiB / 12.1TiB avail pgs: 8782/299948 objects misplaced (2.928%) 949 active+clean 75 active+clean+remapped io: client: 106KiB/s rd, 567KiB/s wr, 127op/s rd, 52op/s wr my all pgs is active+clean, remapped pg is already stop recovery, this is no normal. [root@node-1 ~]# ceph pg dump | grep ^14 | grep "\[12" dumped all 14.37 106 0 0 0 0 441110528 1733 1733 active+clean 2020-09-29 00:52:21.808536 6226'33265 6278:34441 [12,18] 12 [12,18] 12 0'0 2020-09-28 11:35:49.973497 0'0 2020-09-28 11:31:12.912584 0 14.1c 93 0 0 0 0 383131648 1512 1512 active+clean 2020-09-29 00:45:18.475265 6226'28692 6278:29883 [12,6] 12 [12,6] 12 0'0 2020-09-28 11:34:55.543085 0'0 2020-09-28 11:31:12.912584 0 14.10 82 0 0 0 0 340963328 1505 1505 active+clean 2020-09-29 00:45:18.474759 6226'26031 6278:27257 [12,6] 12 [12,6] 12 0'0 2020-09-28 11:34:37.400091 0'0 2020-09-28 11:31:12.912584 0 14.2c 85 0 0 0 0 353435648 1677 1677 active+clean 2020-09-29 00:51:24.528368 6226'25262 6278:26344 [12,18] 12 [12,18] 12 0'0 2020-09-28 11:35:28.806418 0'0 2020-09-28 11:31:12.912584 0 14.26 106 0 0 0 0 440807424 1534 1534 active+clean 2020-09-29 00:45:18.474631 6226'33003 6278:34306 [12,6] 12 [12,6] 12 0'0 2020-09-28 11:35:01.590900 0'0 2020-09-28 11:31:12.912584 0 14.6a 78 0 0 0 0 323850240 1671 1671 active+clean 2020-09-29 00:50:57.336812 6226'23954 6278:25009 [12,18] 12 [12,18] 12 0'0 2020-09-28 11:36:06.101612 0'0 2020-09-28 11:31:12.912584 0 these pg primary osd is 12, I feel it unable to understand that osd 12 already move out gyt-test pool crush rule. it seems a BUG?

1 0

Re: el6 / centos6 rpm's for luminous?
by Dan van der Ster 08 Oct '20

08 Oct '20

We had built some rpms locally for ceph-fuse, but AFAIR luminous needs systemd so the server rpms would be difficult. -- dan On Thu, Oct 8, 2020 at 11:12 AM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > > Nobody ever used luminous on el6? > > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 1

Wipe an Octopus install
by Samuel Taylor Liston 08 Oct '20

08 Oct '20

Wondering if anyone knows or has put together a way to wipe an Octopus install? I’ve looked for documentation on the process, but if it exists, I haven’t found it yet. I’m going through some test installs - working through the ins and outs of cephadm and containers and would love an easy way to tear things down and start over. In previous releases managed through ceph-deploy there were three very convenient commands that nuked the world. I am looking for something as complete for Octopus. Thanks, Sam Liston (sam.liston(a)utah.edu) ========================================== Center for High Performance Computing - Univ. of Utah 155 S. 1452 E. Rm 405 Salt Lake City, Utah 84112 (801)232-6932 ==========================================

3 6

Fwd: pool pgp_num not updated
by Mac Wynkoop 08 Oct '20

08 Oct '20

Just making sure this makes the list: Mac Wynkoop ---------- Forwarded message --------- From: 胡玮文 <huww98(a)outlook.com> Date: Wed, Oct 7, 2020 at 9:00 PM Subject: Re: pool pgp_num not updated To: Mac Wynkoop <mwynkoop(a)netdepot.com> Hi, You can read about this behavior at https://ceph.io/rados/new-in-nautilus-pg-merging-and-autotuning/ In short, ceph will not increase pgp_num if misplaced > 5% (by default), and once you got misplaced < 5%, it will increase pgp_num gradually, until reaching the value you set. This 5% can be configured by target_max_misplaced_ratio config option. 在 2020年10月8日，03:22，Mac Wynkoop <mwynkoop(a)netdepot.com> 写道： Well, backfilling sure, but will it allow me to actually change the pgp_num as more space frees up? Because the issue is that I cannot modify that value. Thanks, Mac Wynkoop, Senior Datacenter Engineer *NetDepot.com:* Cloud Servers; Delivered Houston | Atlanta | NYC | Colorado Springs 1-844-25-CLOUD Ext 806 On Wed, Oct 7, 2020 at 1:50 PM Eugen Block <eblock(a)nde.ag> wrote: Yes, I think that’s exactly the reason. As soon as the cluster has more space the backfill will continue. Zitat von Mac Wynkoop <mwynkoop(a)netdepot.com>: The cluster is currently in a warn state, here's the scrubbed output of ceph -s: *cluster: id: *redacted* health: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 22 nearfull osd(s) 2 pool(s) nearfull Low space hindering backfill (add storage if this doesn't resolve itself): 277 pgs backfill_toofull Degraded data redundancy: 32652738/3651947772 objects degraded (0.894%), 281 pgs degraded, 341 pgs undersized 1214 pgs not deep-scrubbed in time 2647 pgs not scrubbed in time 2 daemons have recently crashed services: mon: 5 daemons, *redacted* (age 44h) mgr: *redacted* osd: 162 osds: 162 up (since 44h), 162 in (since 4d); 971 remapped pgs flags noscrub,nodeep-scrub rgw: 3 daemons active *redacted* tcmu-runner: 18 daemons active *redacted* data: pools: 10 pools, 2648 pgs objects: 409.56M objects, 738 TiB usage: 1.3 PiB used, 580 TiB / 1.8 PiB avail pgs: 32652738/3651947772 objects degraded (0.894%) 517370913/3651947772 objects misplaced (14.167%) 1677 active+clean 477 active+remapped+backfill_wait 100 active+remapped+backfill_wait+backfill_toofull 80 active+undersized+degraded+remapped+backfill_wait 60 active+undersized+degraded+remapped+backfill_wait+backfill_toofull 42 active+undersized+degraded+remapped+backfill_toofull 33 active+undersized+degraded+remapped+backfilling 25 active+remapped+backfilling 25 active+remapped+backfill_toofull 24 active+undersized+remapped+backfilling 23 active+forced_recovery+undersized+degraded+remapped+backfill_wait 19 active+forced_recovery+undersized+degraded+remapped+backfill_wait+backfill_toofull 15 active+undersized+remapped+backfill_wait 14 active+undersized+remapped+backfill_wait+backfill_toofull 12 active+forced_recovery+undersized+degraded+remapped+backfill_toofull 12 active+forced_recovery+undersized+degraded+remapped+backfilling 5 active+undersized+remapped+backfill_toofull 3 active+remapped 1 active+undersized+remapped 1 active+forced_recovery+undersized+remapped+backfilling io: client: 287 MiB/s rd, 40 MiB/s wr, 1.94k op/s rd, 165 op/s wr recovery: 425 MiB/s, 225 objects/s* Now as you can see, we do have a lot of backfill operations going on at the moment. Does that actually prevent Ceph from modifying the pgp_num value of a pool? Thanks, Mac Wynkoop On Wed, Oct 7, 2020 at 8:57 AM Eugen Block <eblock(a)nde.ag> wrote: What is the current cluster status, is it healthy? Maybe increasing pg_num would hit the limit of mon_max_pg_per_osd? Can you share 'ceph -s' output? Zitat von Mac Wynkoop <mwynkoop(a)netdepot.com>: Right, both Norman and I set the pg_num before the pgp_num. For example, here is my current pool settings: *"pool 40 '*redacted*.rgw.buckets.data' erasure size 9 min_size 7 crush_rule 2 object_hash rjenkins pg_num 2048 pgp_num 1024 pgp_num_target 2048 last_change 8458830 lfor 0/0/8445757 flags hashpspool,ec_overwrites,nodelete,backfillfull stripe_width 24576 fast_read 1 application rgw"* So, when I set: "*ceph osd pool set hou-ec-1.rgw.buckets.data pgp_num 2048*" it returns: "*set pool 40 pgp_num to 2048*" But upon checking the pool details again: "*pool 40 '*redacted*.rgw.buckets.data' erasure size 9 min_size 7 crush_rule 2 object_hash rjenkins pg_num 2048 pgp_num 1024 pgp_num_target 2048 last_change 8458870 lfor 0/0/8445757 flags hashpspool,ec_overwrites,nodelete,backfillfull stripe_width 24576 fast_read 1 application rgw*" and the pgp_num value does not increase. Am I just doing something totally wrong? Thanks, Mac Wynkoop On Tue, Oct 6, 2020 at 2:32 PM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: pg_num and pgp_num need to be the same, not? 3.5.1. Set the Number of PGs To set the number of placement groups in a pool, you must specify the number of placement groups at the time you create the pool. See Create a Pool for details. Once you set placement groups for a pool, you can increase the number of placement groups (but you cannot decrease the number of placement groups). To increase the number of placement groups, execute the following: ceph osd pool set {pool-name} pg_num {pg_num} Once you increase the number of placement groups, you must also increase the number of placement groups for placement (pgp_num) before your cluster will rebalance. The pgp_num should be equal to the pg_num. To increase the number of placement groups for placement, execute the following: ceph osd pool set {pool-name} pgp_num {pgp_num} https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/4/html/s… -----Original Message----- To: norman Cc: ceph-users Subject: [ceph-users] Re: pool pgp_num not updated Hi everyone, I'm seeing a similar issue here. Any ideas on this? Mac Wynkoop, On Sun, Sep 6, 2020 at 11:09 PM norman <norman.kern(a)gmx.com> wrote: Hi guys, When I update the pg_num of a pool, I found it not worked(no rebalanced), anyone know the reason? Pool's info: pool 21 'openstack-volumes-rs' replicated size 3 min_size 2 crush_rule 21 object_hash rjenkins pg_num 1024 pgp_num 512 pgp_num_target 1024 autoscale_mode warn last_change 85103 lfor 82044/82044/82044 flags hashpspool,nodelete,selfmanaged_snaps stripe_width 0 application rbd removed_snaps [1~1e6,1e8~300,4e9~18,502~3f,542~11,554~1a,56f~1d7] pool 22 'openstack-vms-rs' replicated size 3 min_size 2 crush_rule 22 object_hash rjenkins pg_num 512 pgp_num 512 pg_num_target 256 pgp_num_target 256 autoscale_mode warn last_change 84769 lfor 0/0/55294 flags hashpspool,nodelete,selfmanaged_snaps stripe_width 0 application rbd The pgp_num_target is set, but pgp_num not set. I have scale out new OSDs and is backfilling before setting the value, is it the reason? _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

1 0

Error "Operation not permitted" using rbd pool init command
by floda 08 Oct '20

08 Oct '20

Hi guys, I have stumbled over an error that I cannot solve after more than 2 days trying. Now I turn to you in hope to get some help! :) First of all, I am totally new to Ceph and this is just a test setup for now. I have followed the instructions on the ceph.com homepage as far as I understand. I am about to set up a Ceph storage cluster on three different nodes (running Ubuntu server 20.04). I am using Ceph Octopus. On these 3 nodes I have installed 3 monitors, 3 managers and 12 OSDs (I do know that OSDs and Monitors on the same machines are not recommended but for this test setup I do it anyway). So far is everything working well for me. However, when I try to create an RBD pool it fails with some strange (at least strange for me) error message: 2020-10-08T07:48:36.601+0000 7f8b9b7fe700 -1 librbd::image::GetMetadataRequest: 0x564279b9d670 handle_metadata_list: failed to retrieve image metadata: (1) Operation not permitted 2020-10-08T07:48:36.601+0000 7f8bb6af8380 -1 librbd::PoolMetadata: list: failed listing metadata: (1) Operation not permitted 2020-10-08T07:48:36.601+0000 7f8bb6af8380 -1 librbd::Config: apply_pool_overrides: failed to read pool config overrides: (1) Operation not permitted 2020-10-08T07:48:36.601+0000 7f8b9b7fe700 -1 librbd::image::ValidatePoolRequest: handle_read_rbd_info: failed to read RBD info: (1) Operation not permitted rbd: pool already registered to a different application. I run the commands as the Linux root user and as the Ceph user client.admin (I have turned off apparmor and other hardening things as well). The chep user client.admin has the following setup in its keyring: [client.admin] key = ..... caps mds = "allow *" caps mgr = "allow *" caps mon = "allow *" For more information regarding permissions in the system see the attached file permissions.txt that contains the output of the ceph auth list command. I have also attached a file (report.txt) containing the output from the ceph report command. (NOTE: I have removed some information from the files, i.e. keys, fsid, fingerprints, uuid, and replaced them with "..." instead) Can anyone please help me understand why I get this error and what I need to do in order to solve it? Thanks in advanced! Best regards, Fredrik

2 2

el6 / centos6 rpm's for luminous?
by Marc Roos 08 Oct '20

08 Oct '20

Nobody ever used luminous on el6?

1 0

What are mon.<hostname>-safe containers?
by Sebastian Luna Valero 08 Oct '20

08 Oct '20

Hi, When I run `ceph orch ps` I see a couple of containers running on our MON nodes whose names end with the `-safe` suffix, and I was wondering what they are? I couldn't find information about it in https://docs.ceph.com This cluster is running Ceph 15.2.5, recently upgraded from 15.2.4 Many thanks, Sebastian

1 0

Quick/easy access to rbd on el6
by Marc Roos 07 Oct '20

07 Oct '20

Normally I would install ceph-common.rpm and access some rbd image via rbdmap. What would be the best way to do this on an old el6? There is not even a luminous el6 on download.ceph.com.

1 0

Kubernetes Luminous client acting on Nautilus pool: protocol feature mismatch: missing 200000 (CEPH_FEATURE_MON_GV ?)
by Fulvio Galeazzi 07 Oct '20

07 Oct '20

Hallo all! I am configuring a new storage class on my Kubernetes cluster, pointing to a pool on a Ceph cluster which was recently upgraded to Nautilus (was Luminous). The old storage class points to a Luminous pool in a separate cluster and works fine. On the new one, I think I did the configuration properly, yet when creating a volume I get this: 2020-10-07 10:00:53.849128 7f2f8c6f9700 0 -- 10.2.3.13:0/3520982056 >> 10.2.3.23:6789/0 pipe(0x7f2f780008c0 sd=3 :60192 s=1 pgs=0 cs=0 l=1 c=0x7f2f780068e0).connect protocol feature mismatch, my 27ffffffefdfbfff < peer 27fddff8efacbfff missing 200000 Looking at https://ceph.io/geen-categorie/feature-set-mismatch-error-on-ceph-kernel-cl… looks like the 200000 corresponds to CEPH_FEATURE_MON_GV, which by the way is listed here https://github.com/ceph/ceph/pull/8214 as a feature which could/should be removed. Things being as I described, I guess it would be safe to change the value of the tunable, correct? Unfortunately, I was unable to find any way to achieve this... the trivial "ceph osd crush set-tunable mon_gv 0" does not work. Any idea, please, how to fix my error? Upgrading Ceph packages on Kubernetes workers (now at Luminous) would help, may be? Thanks! Fulvio -- Fulvio Galeazzi GARR-CSD Department skype: fgaleazzi70 tel.: +39-334-6533-250

1 0

Pool quotas vs. rados df vs. RBD images (phantom objects in pool?)
by Miroslav Kalina 07 Oct '20

07 Oct '20

Hello everyone, I have single Ceph cluster with multiple pools for purposes of Kubernetes PVs and Virtualization (biggest pool, using snapshots here). Every client cluster has its own pool with max_bytes and max_objects quotas in place. When quotas are reached, Ceph halts all write I/O (which I suppose is OK). I am trying to setup best possible monitoring to prevent reaching pool quotas, but I am really struggling to find correct metrics Ceph is driven by, because almost every command I try gives me different results (rados df, rbd du, ceph health detail) and I feel like quotas are not usable for this purpose. After some fiddling with test pool I have somehow got it into state, when there are some phantom objects in stats (maybe also written to OSDs) which cannot be listed, accessed or deleted, but are counted into pool quotas. -> ceph version ceph version 14.2.10 (9f0d3f5a3ce352651da4c2437689144fcbec0131) nautilus (stable) I have my test pool here (quotas 4k objects, 15 GiB data): -> ceph osd pool ls detail | grep mirektest pool 2 'mirektest' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode warn last_change 21360 lfor 0/10929/10927 flags hashpspool,selfmanaged_snaps max_bytes 16106127360 max_objects 4000 stripe_width 0 target_size_bytes 10737418240 application rbd Cluster health says pool it getting full: -> ceph health detail HEALTH_WARN 1 pools nearfull POOL_NEAR_FULL 1 pools nearfull pool 'mirektest' has 3209 objects (max 4000) pool 'mirektest' has 13 GiB (max 15 GiB) But pool looks completely empty (no output, so I tried JSON output also): -> rbd -p mirektest --format json ls [] -> rbd -p mirektest --format json du {"images":[],"total_provisioned_size":0,"total_used_size":0} I am not using pool snapshots: -> rados -p mirektest lssnap 0 snaps But there are some RADOS object in it: -> rados -p mirektest df POOL_NAME *USED**OBJECTS*CLONES *COPIES*MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR mirektest *192 KiB* *3209* 0 *9627* 0 0 0 23998515 192 GiB 1221030 297 GiB 0 B 0 B Okay, lets try to list them: -> rados -p mirektest ls --all rbd_directory rbd_info rbd_trash Thats weird, expected more than 3 objetcs. Let's check it's sizes and content: -> rados -p mirektest stat rbd_directory mirektest/rbd_directory mtime 2020-10-07 11:55:16.000000, size 0 -> rados -p mirektest stat rbd_info mirektest/rbd_info mtime 2019-12-05 12:11:39.000000, size 19 -> rados -p mirektest get rbd_info - overwrite validated -> rados -p mirektest stat rbd_trash mirektest/rbd_trash mtime 2020-10-07 11:55:17.000000, size 0 Do any of you have some idea what's happening here? I am trying to find way how to cleanup pool without interrupting existing content in general. Also I have probably no idea how to replicate this and it's not the story of every pool in cluster. There is e.g. pool which was used for Kubernetes but it has no issues: -> rados df POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR pool12 192 KiB 3 0 9 0 0 0 8710688 807 GiB 5490880 576 GiB 0 B 0 B I don't know how to check other pools, because there is data in them . How can I rely on quotas when in empty cluster there is already 13GiB data and 3.2k objects (which btw corresponds because 3200 * 4MiB = 12.8 GiB). Also is there any way how to calculate same number of objects/bytes Ceph is using to enforce quotas? Thanks for you time if you read it until the end, really appreciated that! :-) Any idea or hint is welcome. -- Miroslav Kalina Systems development specialist miroslav.kalina(a)livesport.eu +420 773 071 848 Livesport s.r.o. Aspira Business Centre Bucharova 2928/14a, 158 00 Praha 5 www.livesport.eu

1 0

Slow ops on OSDs
by Kristof Coucke 07 Oct '20

07 Oct '20

Hi all, We have a Ceph cluster which has been expanded from 10 to 16 nodes. Each node has between 14 and 16 OSDs of which 2 are NVMe disks. Most disks (except NVMe's) are 16TB large. The expansion of 16 nodes went ok, but we've configured the system to prevent auto balance towards the new disks (weight was set to 0) so we could control the expansion. We started adding 6 disks last week (1 disk on each new node) which didn't give a lot of issues. When the Ceph status indicated the PG degraded was almost finished, we've added 2 disks on each node again. All seemed to go fine, till yesterday morning... IOs towards the system were slowing down. Diving onto the nodes we could see that the OSD daemons are consuming the CPU power, resulting in average CPU loads going near 10 (!). The RGWs nor monitors nor other involved servers are having CPU issues (except for the management server which is fighting with Prometheus), so it's latency seems to be related to the ODS hosts. All of the hosts are interconnected with 25Gbit connections, no bottlenecks are reached on the network either. Important piece of information: We are using erasure coding (6/3), and we do have a lot of small files... The current health detail indicates degraded health redundancy where 1192911/103387889228 objects are degraded. (1 pg degraded, 1 pg undersized). Diving into the historic ops of an OSD we can see that the main latency is found between the event "queued_for_pg" and "reached_pg". (Averaging +/- 3 secs) As the system load is quite high I assume the systems are busy recalculating the code chunks for using the new disks we've added (though not sure), but I was wondering how I can better fine tune the system or pinpoint the exact bottle neck. Latency towards the disks doesn't seem an issue at first sight... We are running Ceph 14.2.11 Who can give me some thoughts on how I can better pinpoint the bottle neck? Thanks Kristof

4 16

Ceph iSCSI Performance
by DHilsbos＠performair.com 06 Oct '20

06 Oct '20

All; I've finally gotten around to setting up iSCSI gateways on my primary production cluster, and performance is terrible. We're talking 1/4 to 1/3 of our current solution. I see no evidence of network congestion on any involved network link. I see no evidence CPU or memory being a problem on any involved server (MON / OSD / gateway /client). What can I look at to tune this, preferably on the iSCSI gateways? Thank you, Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International, Inc. DHilsbos(a)PerformAir.com www.PerformAir.com

5 8

pool pgp_num not updated
by norman 06 Oct '20

06 Oct '20

Hi guys, When I update the pg_num of a pool, I found it not worked(no rebalanced), anyone know the reason? Pool's info: pool 21 'openstack-volumes-rs' replicated size 3 min_size 2 crush_rule 21 object_hash rjenkins pg_num 1024 pgp_num 512 pgp_num_target 1024 autoscale_mode warn last_change 85103 lfor 82044/82044/82044 flags hashpspool,nodelete,selfmanaged_snaps stripe_width 0 application rbd removed_snaps [1~1e6,1e8~300,4e9~18,502~3f,542~11,554~1a,56f~1d7] pool 22 'openstack-vms-rs' replicated size 3 min_size 2 crush_rule 22 object_hash rjenkins pg_num 512 pgp_num 512 pg_num_target 256 pgp_num_target 256 autoscale_mode warn last_change 84769 lfor 0/0/55294 flags hashpspool,nodelete,selfmanaged_snaps stripe_width 0 application rbd The pgp_num_target is set, but pgp_num not set. I have scale out new OSDs and is backfilling before setting the value, is it the reason?

3 2

Re: Slow ops on OSDs
by Kristof Coucke 06 Oct '20

06 Oct '20

Thanks to @Anthony: Diving further I see that I probably was blinded by the CPU load... I see that some disks are very slow (so my first observations were incorrect), and the latency seen using iostat seems more or less the same as what we see in the dump_historic_ops. (+ 3s for r_await) So, it looks like a few OSDs are causing a bottleneck in the whole system. I'm now wondering what my options are to improve the performance... The main goal is to use the system again, and make sure write operations are not affected. - Putting weight on 0 for the slow OSDs (temporary)? This way they recovery can go on but new files are not written to that disk? - .... Still investigating...

2 1

CephFS user mapping
by René Bartsch 06 Oct '20

06 Oct '20

Hi, is there any documentation about mapping usernames, user-ids, groupnames and group-ids between hosts sharing the same CephFS storage? Thanx for any hint, Renne

4 3

Re: Massive Mon DB Size with noout on 14.2.11
by Dan van der Ster 06 Oct '20

06 Oct '20

The important metric is the difference between these two values: # ceph report | grep osdmap | grep committed report 3324953770 "osdmap_first_committed": 3441952, "osdmap_last_committed": 3442452, The mon stores osdmaps on disk, and trims the older versions whenever the PGs are clean. Trimming brings the osdmap_first_committed to be closer to osdmap_last_committed. In a cluster with no PGs backfilling or recovering, the mon should trim that difference to be within 500-750 epochs. If there are any PGs backfilling or recovering, then the mon will not trim beyond the osdmap epoch when the pools were clean. So if you are accumulating gigabytes of data in the mon dir, it suggests that you have unclean PGs/Pools. Cheers, dan On Fri, Oct 2, 2020 at 4:14 PM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > > Does this also count if your cluster is not healthy because of errors > like '2 pool(s) have no replicas configured' > I sometimes use these pools for testing, they are empty. > > > > > -----Original Message----- > Cc: ceph-users > Subject: [ceph-users] Re: Massive Mon DB Size with noout on 14.2.11 > > As long as the cluster is no healthy, the OSD will require much more > space, depending on the cluster size and other factors. Yes this is > somewhat normal. > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

4 4

Write access delay after OSD & Mon lost
by Mathieu Dupré 06 Oct '20

06 Oct '20

Hi everybody, Our need is to do VM failover using an image disk over RBD to avoid data loss.We want to limit the downtime as much as possible. We have: - Two hypervisors with a Ceph Monitor and a Ceph OSD. - A third machine with a Ceph Monitor and a Ceph Manager. VM are running over qemu.The VM disks are on a "replicated" rbd pool formed by the two OSDs.Ceph version: NautilusDistribution: Yocto Zeus The following test is performed: we electrically turn off one hypervisor (and therefore a Ceph Monitor and a Ceph OSD), which causes its VMs to switch to the second hypervisor. My main issue is that the mount time of a partition in rw is very slow in the case of a failover (after the loss of an OSD its monitor). With failover we can write on the device after ~25s:[ 25.609074] EXT4-fs (vda3): mounted filesystem with ordered data mode. Opts: (null) In normal boot we can write on the device after ~4s:[ 3.087412] EXT4-fs (vda3): mounted filesystem with ordered data mode. Opts: (null) I wasn't able to reduce this time by tweaking Ceph settings. I am wondering if someone could help me on that. Here is our configuration. ceph.conf[global] fsid = fa7a17d1-5351-459e-bf0e-07e7edc9a625 mon initial members = hypervisor1,hypervisor2,observer mon host = 192.168.217.131,192.168.217.132,192.168.217.133 public network = 192.168.217.0/24 auth cluster required = cephx auth service required = cephx auth client required = cephx osd journal size = 1024 osd pool default size = 2 osd pool default min size = 1 osd crush chooseleaf type = 1 mon osd adjust heartbeat grace = false mon osd min down reporters = 1[mon.hypervisor1] host = hypervisor1 mon addr = 192.168.217.131:6789[mon.hypervisor2] host = hypervisor2 mon addr = 192.168.217.132:6789[mon.observer] host = observer mon addr = 192.168.217.133:6789[osd.0] host = hypervisor1 public_addr = 192.168.217.131 cluster_addr = 192.168.217.131[osd.1] host = hypervisor2 public_addr = 192.168.217.132 cluster_addr = 192.168.217.13 # ceph config dump WHO MASK LEVEL OPTION VALUE RO global advanced mon_osd_adjust_down_out_interval false global advanced mon_osd_adjust_heartbeat_grace false global advanced mon_osd_down_out_interval 5 global advanced mon_osd_report_timeout 4 global advanced osd_beacon_report_interval 1 global advanced osd_heartbeat_grace 2 global advanced osd_heartbeat_interval 1 global advanced osd_mon_ack_timeout 1.000000 global advanced osd_mon_heartbeat_interval 2 global advanced osd_mon_report_interval 3 Thanks

3 2

Re: Slow ops on OSDs
by Kristof Coucke 06 Oct '20

06 Oct '20

Hi Anthony, Thnx for the reply Average values: User: 3.5 Idle: 78.4 Wait: 20 System: 1.2 /K. Op di 6 okt. 2020 om 10:18 schreef Anthony D'Atri <anthony.datri(a)gmail.com>: > > > > > > Diving onto the nodes we could see that the OSD daemons are consuming the > > CPU power, resulting in average CPU loads going near 10 (!). > > > FWIW, the load average doesn’t really tell you much on a multi-core > system. I’ve run 24x SSD OSD nodes with load averages routinely >30 that > hummed along just fine. > > What are the percentages for user/idle/wait/system ? > > > >

1 0

Consul as load balancer
by Szabo, Istvan (Agoda) 06 Oct '20

06 Oct '20

Hi, Is there anybody tried consul as a load balancer? Any experience? Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 2

[Ceph Octopus 15.2.3 ] MDS crashed suddently and failed to replay journal after restarting
by carlimeunier＠gmail.com 05 Oct '20

05 Oct '20

Hello, MDS process crashed suddently. After trying to restart it, it failed to replay journal and started to restart continually. Just to summarize, here is what happened : 1/ The cluster is up and running with 3 nodes (mon and mds in the same nodes) and 3 OSD. 2/ After a few days, 2 (standby-replay and standby) of the 3 MDS processes crashed. No pid. Ceph status indicates that the processes are down 3/ I try restart it : - Sometimes, the restarting fails with segmentation fault error. Here is the ceph-mds.log file : -20> 2020-07-17T13:50:27.888+0000 7fc8c6c51700 10 monclient: _renew_subs -19> 2020-07-17T13:50:27.888+0000 7fc8c6c51700 10 monclient: _send_mon_message to mon.2 at v1:172.31.36.98:6789/0 -18> 2020-07-17T13:50:27.888+0000 7fc8c6c51700 10 monclient: handle_get_version_reply finishing 0x559dcf9530c0 version 269 -17> 2020-07-17T13:50:27.888+0000 7fc8c6c51700 10 monclient: handle_get_version_reply finishing 0x559dcfa87520 version 269 -16> 2020-07-17T13:50:27.888+0000 7fc8c6c51700 10 monclient: handle_get_version_reply finishing 0x559dcfa875c0 version 269 -15> 2020-07-17T13:50:27.888+0000 7fc8c6c51700 10 monclient: handle_get_version_reply finishing 0x559dcfa871c0 version 269 -14> 2020-07-17T13:50:27.888+0000 7fc8c8c55700 10 monclient: get_auth_request con 0x559dcfada000 auth_method 0 -13> 2020-07-17T13:50:27.888+0000 7fc8c9456700 10 monclient: get_auth_request con 0x559dcfada800 auth_method 0 -12> 2020-07-17T13:50:27.892+0000 7fc8bfc43700 1 mds.282966.journaler.mdlog(ro) recover start -11> 2020-07-17T13:50:27.892+0000 7fc8bfc43700 1 mds.282966.journaler.mdlog(ro) read_head -10> 2020-07-17T13:50:27.892+0000 7fc8bfc43700 4 mds.0.log Waiting for journal 0x200 to recover... -9> 2020-07-17T13:50:27.893+0000 7fc8c0444700 1 mds.282966.journaler.mdlog(ro) _finish_read_head loghead(trim 4194304, expire 4231216, write 4329405, stream_format 1). probing for end of log (from 4329405)... -8> 2020-07-17T13:50:27.893+0000 7fc8c0444700 1 mds.282966.journaler.mdlog(ro) probing for end of the log -7> 2020-07-17T13:50:27.893+0000 7fc8c0444700 1 mds.282966.journaler.mdlog(ro) _finish_probe_end write_pos = 4329949 (header had 4329405). recovered. -6> 2020-07-17T13:50:27.893+0000 7fc8bfc43700 4 mds.0.log Journal 0x200 recovered. -5> 2020-07-17T13:50:27.893+0000 7fc8bfc43700 4 mds.0.log Recovered journal 0x200 in format 1 -4> 2020-07-17T13:50:27.893+0000 7fc8bfc43700 2 mds.0.0 Booting: 1: loading/discovering base inodes -3> 2020-07-17T13:50:27.893+0000 7fc8bfc43700 0 mds.0.cache creating system inode with ino:0x100 -2> 2020-07-17T13:50:27.894+0000 7fc8bfc43700 0 mds.0.cache creating system inode with ino:0x1 -1> 2020-07-17T13:50:27.894+0000 7fc8c0444700 2 mds.0.0 Booting: 2: replaying mds log 0> 2020-07-17T13:50:27.896+0000 7fc8bec41700 -1 *** Caught signal (Segmentation fault) ** in thread 7fc8bec41700 thread_name:md_log_replay - Sometimes, the restarting works but journal replay failed even after having reset it (# cephfs-journal-tool --rank=cephfs:0 journal reset) on the failed nodes. The cluster status look like this : # ceph status -w cluster: id: acd73aa2-8cdd-41a3-9941-fb397aa1d79e health: HEALTH_WARN 1 daemons have recently crashed services: mon: 3 daemons, quorum 2,0,1 (age 3w) mgr: mgr.0(active, since 11w), standbys: mgr.2, mgr.1 mds: cephfs:1 {0=node1=up:active} 1 up:standby-replay 1 up:standby osd: 3 osds: 3 up (since 33h), 3 in (since 11w) task status: scrub status: mds.node1: idle data: pools: 3 pools, 49 pgs objects: 165 objects, 157 MiB usage: 3.5 GiB used, 41 TiB / 41 TiB avail pgs: 49 active+clean io: client: 1.8 MiB/s rd, 4 op/s rd, 0 op/s wr 2020-10-05T13:32:03.798231+0000 mds.node0 [ERR] failure replaying journal (EMetaBlob) 2020-10-05T13:32:03.851986+0000 mon.2 [INF] daemon mds.node0 restarted 2020-10-05T13:32:04.605163+0000 mds.node0 [ERR] failure replaying journal (EMetaBlob) 2020-10-05T13:32:08.652989+0000 mon.2 [INF] daemon mds.node0 restarted 2020-10-05T13:32:08.916347+0000 mds.node0 [ERR] failure replaying journal (EMetaBlob) 2020-10-05T13:32:12.961902+0000 mon.2 [INF] daemon mds.node0 restarted 2020-10-05T13:32:13.974410+0000 mds.node0 [ERR] failure replaying journal (EMetaBlob) 2020-10-05T13:32:14.023126+0000 mon.2 [INF] daemon mds.node0 restarted 2020-10-05T13:32:14.610039+0000 mds.node0 [ERR] failure replaying journal (EMetaBlob) Question : - Why 2 of 3 MDS processes sometimes crash? I suspect the client ( kernel 4.20) on which there is a cephfs in-tree provisioner (not csi) for kubernetes. how can i highlight it? Thanks for your support

1 0

S3 multipart upload in Ceph 12.2.11 Luminous
by Eugeniy Khvastunov 05 Oct '20

05 Oct '20

Hi ceph-users, Looks like we faced a broken S3 multipart upload in Ceph 12.2.11 Luminous (MCP 2019.2.0) during thanos setup. The request to initialize the multipart fails. S3cmd log: DEBUG: Sending request method_string='POST', uri='/staging-prometheus-k8s-metrics/test4/dd-50M.rnd?uploads', headers={'x-amz-meta-s3cmd-attrs': 'uid:1011/gname: yevhen.kh/uname:yevhen.kh/gid:1011/mode:33188/mtime:1600198080/atime:1600198530/md5:730ee931c6058cad5bd4d87a6c8f3178/ctime:1600198363', 'content-type': 'application/octet-stream', 'Authorization': 'AWS ZLMVD7JZZ8V09VVK66C3:YX5Oe+Ig0FPDbRpIYO2m1WHITOE=', 'x-amz-date': 'Tue, 15 Sep 2020 19:38:33 +0000', 'x-amz-storage-class': 'STANDARD'}, body=(0 bytes) DEBUG: Response: {'status': 400, 'headers': {'content-length': '253', 'accept-ranges': 'bytes', 'server': 'nginx', 'connection': 'keep-alive', 'x-amz-request-id': 'tx0000000000000001e019a-005f611839-deb80f-default', 'date': 'Tue, 15 Sep 2020 19:38:37 GMT', 'content-type': 'application/xml'}, 'reason': 'Bad Request', 'data': '<?xml version="1.0" encoding="UTF-8"?><Error><Code>InvalidArgument</Code><BucketName>staging-prometheus-k8s-metrics</BucketName><RequestId>tx0000000000000001e019a-005f611839-deb80f-default</RequestId><HostId>deb80f-default-default</HostId></Error>'} DEBUG: ConnMan.put(): connection put back to pool ( https://s3.cloud.f.in.company.com:8080#1) DEBUG: S3Error: 400 (Bad Request) DEBUG: HttpHeader: content-length: 253 DEBUG: HttpHeader: accept-ranges: bytes DEBUG: HttpHeader: server: nginx DEBUG: HttpHeader: connection: keep-alive DEBUG: HttpHeader: x-amz-request-id: tx0000000000000001e019a-005f611839-deb80f-default DEBUG: HttpHeader: date: Tue, 15 Sep 2020 19:38:37 GMT DEBUG: HttpHeader: content-type: application/xml DEBUG: ErrorXML: Code: 'InvalidArgument' DEBUG: ErrorXML: BucketName: 'staging-prometheus-k8s-metrics' DEBUG: ErrorXML: RequestId: 'tx0000000000000001e019a-005f611839-deb80f-default' DEBUG: ErrorXML: HostId: 'deb80f-default-default' ERROR: S3 error: 400 (InvalidArgument) CEPH log: > 2020-09-10 08:26:26.238814 7f7482fdd700 1 -- 10.14.2.66:0/3902939317 <== > mon.0 10.14.2.66:6789/0 19 ==== pool_op_reply(tid 1290 (22) Invalid > argument v22601) v1 ==== 43+0+0 (3815426539 0 0) 0x55faeb4e4a80 con > 0x55faeabc8800 > 2020-09-10 08:26:26.241214 7f74607de700 2 req 484:1.595459:s3:POST > /staging-prometheus-k8s-metrics/01EH8QMFREFVE4Q32Y2QZRNTBN/chunks/000001:init_multipart:completing > 2020-09-10 08:26:26.241389 7f74607de700 2 req 484:1.595632:s3:POST > /staging-prometheus-k8s-metrics/01EH8QMFREFVE4Q32Y2QZRNTBN/chunks/000001:init_multipart:op > status=-22 > 2020-09-10 08:26:26.241414 7f74607de700 2 req 484:1.595658:s3:POST > /staging-prometheus-k8s-metrics/01EH8QMFREFVE4Q32Y2QZRNTBN/chunks/000001:init_multipart:http > status=400 > 2020-09-10 08:26:26.241420 7f74607de700 1 ====== req done > req=0x7f74607d8140 op status=-22 http_status=400 ====== > 2020-09-10 08:26:26.241443 7f74607de700 20 process_request() returned -22 > 2020-09-10 08:26:26.241492 7f74607de700 1 civetweb: 0x55faeaedf000: > 10.14.5.203 - - [10/Sep/2020:08:26:24 +0000] "POST > /staging-prometheus-k8s-metrics/01EH8QMFREFVE4Q32Y2QZRNTBN/chunks/000001?uploads= > HTTP/1.1" 400 0 - MinIO (linux; amd64) minio-go/v7.0.2 > thanos-sidecar/0.15.0 (go1.14.2) > The bucket exists. The issue is repeated on the newly created bucket and on the existing one. If s3cmd runs with `--multipart-chunk-size-mb=` option and the value is larger than the file size(for example, if the file is 50 mb, set --multipart-chunk-size-mb=200 mb) then the file uploads. Example: > s3cmd -c /root/f-s3-thanos.cfg put /root/dd-50M.rnd > s3://staging-prometheus-k8s-metrics/test4/dd-50M.rnd -d > --multipart-chunk-size-mb=200 > If s3cmd runs with `--disable-multipart` that works as well. Have you faced such problems? I would be grateful for your advice. -- wbr. Eugeniy Khvastunov, System administrator. [FMGH-UANIC] http://blog.unlimite.net

2 2

Feedback for proof of concept OSD Node
by Ignacio Ocampo 04 Oct '20

04 Oct '20

Hi All :), I would like to get your feedback about the components below to build a PoC OSD Node (I will build 3 of these). SSD for OS. NVMe for cache. HDD for storage. The Supermicro motherboard has 2 10Gb cards, and I will use ECC memories. [image: image.png] Thanks for your feedback! -- Ignacio Ocampo

6 12

ceph iscsi latency too high for esxi?
by Golasowski Martin 04 Oct '20

04 Oct '20

Hi, does anyone here use CEPH iSCSI with VMware ESXi? It seems that we are hitting the 5 second timeout limit on software HBA in ESXi. It appears whenever there is increased load on the cluster, like deep scrub or rebalance. Is it normal behaviour in production? Or is there something special we need to tune? We are on latest Nautilus, 12 x 10 TB OSDs (4 servers), 25 Gbit/s Ethernet, erasure coded rbd pool with 128 PGs, aroun 200 PGs per OSD total. ESXi Log: 2020-10-04T01:57:04.314Z cpu34:2098959)WARNING: iscsi_vmk: iscsivmk_ConnReceiveAtomic:517: vmhba64:CH:1 T:0 CN:0: Failed to receive data: Connection closed by peer 2020-10-04T01:57:04.314Z cpu34:2098959)iscsi_vmk: iscsivmk_ConnRxNotifyFailure:1235: vmhba64:CH:1 T:0 CN:0: Connection rx notifying failure: Failed to Receive. State=Bound 2020-10-04T01:57:04.566Z cpu19:2098979)WARNING: iscsi_vmk: iscsivmk_StopConnection:741: vmhba64:CH:1 T:0 CN:0: iSCSI connection is being marked "OFFLINE" (Event:4) 2020-10-04T01:57:04.654Z cpu7:2097866)WARNING: VMW_SATP_ALUA: satp_alua_issueCommandOnPath:788: Probe cmd 0xa3 failed for path "vmhba64:C2:T0:L0" (0x5/0x20/0x0). Check if failover mode is still ALUA. OSD Log: [303088.450088] Did not receive response to NOPIN on CID: 0, failing connection for I_T Nexus iqn.1994-05.com.redhat:esxi1,i,0x00023d000002,iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw,t,0x01 [324926.694077] Did not receive response to NOPIN on CID: 0, failing connection for I_T Nexus iqn.1994-05.com.redhat:esxi2,i,0x00023d000001,iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw,t,0x01 [407067.404538] ABORT_TASK: Found referenced iSCSI task_tag: 5891 [407076.077175] ABORT_TASK: Sending TMR_FUNCTION_COMPLETE for ref_tag: 5891 [411677.887690] ABORT_TASK: Found referenced iSCSI task_tag: 6722 [411683.297425] ABORT_TASK: Sending TMR_FUNCTION_COMPLETE for ref_tag: 6722 [481459.755876] ABORT_TASK: Found referenced iSCSI task_tag: 7930 [481460.787968] ABORT_TASK: Sending TMR_FUNCTION_COMPLETE for ref_tag: 7930 Cheers, Martin

5 8

Re: Ceph Tech Talk: Karan Singh - Scale Testing Ceph with 10Billion+ Objects
by Peter Sarossy 02 Oct '20

02 Oct '20

You can click "join without audio and video" at the bottom On Thu, Oct 1, 2020 at 1:10 PM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > Mike, > > Can you allow access without mic and cam? > > Thanks, > Marc > > > > -----Original Message----- > > To: ceph-users(a)ceph.io > Subject: *****SPAM***** [ceph-users] Ceph Tech Talk: Karan Singh - Scale > Testing Ceph with 10Billion+ Objects > > Hey all, > > We're live now with the latest Ceph tech talk! Join us: > > https://bluejeans.com/908675367/browser > > -- > > Mike Perez > > he/him > > Ceph Community Manager > > > M: +1-951-572-2633 > > 494C 5D25 2968 D361 65FB 3829 94BC D781 ADA8 8AEA @Thingee > <https://twitter.com/thingee> Thingee > <https://www.linkedin.com/thingee> <https://www.facebook.com/RedHatInc> > <https://www.redhat.com> > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an > email to ceph-users-leave(a)ceph.io > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io > -- Cheers, Peter Sarossy Technical Program Manager Data Center Data Security - Google LLC.

4 3

Massive Mon DB Size with noout on 14.2.11
by Andreas John 02 Oct '20

02 Oct '20

Hello, we observed massive and sudden growth of the mon db size on disk, from 50MB to 20GB+ (GB!) and thus reaching 100% disk usage on the mountpoint. As far as we can see, it happens if we set "noout" for a node reboot: After the node and the OSDs come back it looks like the mon db size increased drastically. We have 14.2.11, 10 OSD @ 2TB and cephfs in use. Is this a known issue? Should we avoid noout? TIA, derjohn -- Andreas John net-lab GmbH | Frankfurter Str. 99 | 63067 Offenbach Geschaeftsfuehrer: Andreas John | AG Offenbach, HRB40832 Tel: +49 69 8570033-1 | Fax: -2 | http://www.net-lab.net Facebook: https://www.facebook.com/netlabdotnet Twitter: https://twitter.com/netlabdotnet

4 3

rgw snapshots/backup
by Manuel Lausch 02 Oct '20

02 Oct '20

Hi, we evaluate rados gateway to provide S3 Storage. There is one question related to backup/snapshots. Is there any way to create snapshots of buckets and or backup a bucket? And how we can access data of a sapshot? I found only some very old information which indicates that this is not possible. Maybe there are new features which I didn't find. Regards Manuel

2 1

Upgrade failed, now ceph orch broken
by Ragan, Tj (Dr.) 02 Oct '20

02 Oct '20

Hi All, I have a cluster that that I tried upgrading from 15.2.4 to 15.2.5 using the command ‘ceph orch upgrade start —ceph-version 15.2.5’. After upgrading two of the three mgrs, the third mgr failed and the upgrade stopped. I was able to get the third mgr upgraded by changing the systemd file and manually removing the podman image. I have upgraded ceph-common outside the containers on all the mgr/mon nodes. The cluster repots HEALTH_OK. However, any ‘ceph orch’ command I issue now always returns: Error ENOENT: Module not found If I run ‘ceph orch ps —verbose’, I get: parsed_args: Namespace(admin_socket=None, block=False, cephconf=None, client_id=None, client_name=None, cluster=None, cluster_timeout=None, completion=False, help=False, input_file=None, output_file=None, output_format=None, period=1, setgroup=None, setuser=None, status=False, verbose=True, version=False, watch=False, watch_channel=None, watch_debug=False, watch_error=False, watch_info=False, watch_sec=False, watch_warn=False), childargs: ['orch', 'ps'] cmd000: pg stat ... better match: 2.5 > 1.5: orch ps [<hostname>] [<service_name>] [<daemon_type>] [<daemon_id>] [plain|json|json-pretty|yaml] [--refresh] bestcmds_sorted: [{'flags': 8, 'help': 'List daemons known to orchestrator', 'module': 'mgr', 'perm': 'r', 'sig': [argdesc(<class 'ceph_argparse.CephPrefix'>, req=True, name=prefix, n=1, numseen=0, prefix=orch), argdesc(<class 'ceph_argparse.CephPrefix'>, req=True, name=prefix, n=1, numseen=0, prefix=ps), argdesc(<class 'ceph_argparse.CephString'>, req=False, name=hostname, n=1, numseen=0), argdesc(<class 'ceph_argparse.CephString'>, req=False, name=service_name, n=1, numseen=0), argdesc(<class 'ceph_argparse.CephString'>, req=False, name=daemon_type, n=1, numseen=0), argdesc(<class 'ceph_argparse.CephString'>, req=False, name=daemon_id, n=1, numseen=0), argdesc(<class 'ceph_argparse.CephChoices'>, req=False, name=format, n=1, numseen=0, strings=plain|json|json-pretty|yaml), argdesc(<class 'ceph_argparse.CephBool'>, req=False, name=refresh, n=1, numseen=0)]}] Submitting command: {'prefix': 'orch ps', 'target': ('mon-mgr', '')} submit ['{"prefix": "orch ps", "target": ["mon-mgr", ""]}'] to mon-mgr Error ENOENT: Module not found Any help would be appreciated. Thanks, -TJ Ragan

1 0

Re: ceph-volume quite buggy compared to ceph-disk
by Matt Larson 02 Oct '20

02 Oct '20

Hi Marc, Did you have any success with `ceph-volume` for activating your OSD? I am having a similar problem where the command `ceph-bluestore-tool` fails to be able to read a label for a previously created OSD on an LVM partition. I had previously been using the OSD without issues, but after a reboot it fails to load. 1. I had initially created my OSD using Ceph Octopus 15.x with `ceph orch daemon add osd <my hostname>:boot/cephfs_meta` that was able to create an OSD on the LVM partition and bring up an OSD. 2. After a reboot, the OSD fails to come up, with error from `ceph-bluestore-tool` happening inside the container specifically being unable to read the label of the device. 3. When I query the symlinked device /dev/boot/cephfs_meta -> /dev/dm3, with `dmsetup info /dev/dm-3`, I can see the state is active and that it has a UUID, etc. 4. I installed `ceph-osd` CentOS package providing the ceph-bluestore-tool, and tried to manually test and `sudo ceph-bluestore-tool show-label --dev /dev/dm-3` fails to read the label. When I try with other OSD's that were created for entire disks this command is able to read the label and print out information. I am considering submitting a ticket to the ceph issue tracker, as I am unable to figure out why the ceph-bluestore-tool cannot read the labels and it seems either the OSD was initially created incorrectly or there is a bug in ceph-bluestore-tool. One possibility is that I did not have the LVM2 package installed on this host prior to the `ceph orch daemon add ..` command and this caused a particular issue with the LVM partition OSD. -Matt On Sat, Sep 19, 2020 at 9:11 AM Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > > > > [@]# ceph-volume lvm activate 36 82b94115-4dfb-4ed0-8801-def59a432b0a > Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-36 > Running command: /usr/bin/ceph-authtool > /var/lib/ceph/osd/ceph-36/lockbox.keyring --create-keyring --name > client.osd-lockbox.82b94115-4dfb-4ed0-8801-def59a432b0a --add-key > AQBxA2Zfj6avOBAAIIHqNNY2J22EnOZV+dNzFQ== > stdout: creating /var/lib/ceph/osd/ceph-36/lockbox.keyring > added entity client.osd-lockbox.82b94115-4dfb-4ed0-8801-def59a432b0a > auth(key=AQBxA2Zfj6avOBAAIIHqNNY2J22EnOZV+dNzFQ==) > Running command: /usr/bin/chown -R ceph:ceph > /var/lib/ceph/osd/ceph-36/lockbox.keyring > Running command: /usr/bin/ceph --cluster ceph --name > client.osd-lockbox.82b94115-4dfb-4ed0-8801-def59a432b0a --keyring > /var/lib/ceph/osd/ceph-36/lockbox.keyring config-key get > dm-crypt/osd/82b94115-4dfb-4ed0-8801-def59a432b0a/luks > Running command: /usr/sbin/cryptsetup --key-file - --allow-discards > luksOpen > /dev/ceph-9263e83b-7660-4f5b-843a-2111e882a17e/osd-block-82b94115-4dfb-4 > ed0-8801-def59a432b0a I8MyTZ-RQjx-gGmd-XSRw-kfa1-L60n-fgQpCb > stderr: Device I8MyTZ-RQjx-gGmd-XSRw-kfa1-L60n-fgQpCb already exists. > Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-36 > Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph > prime-osd-dir --dev /dev/mapper/I8MyTZ-RQjx-gGmd-XSRw-kfa1-L60n-fgQpCb > --path /var/lib/ceph/osd/ceph-36 --no-mon-config > stderr: failed to read label for > /dev/mapper/I8MyTZ-RQjx-gGmd-XSRw-kfa1-L60n-fgQpCb: (2) No such file or > directory > --> RuntimeError: command returned non-zero exit status: 1 > > dmsetup ls lists this???? > > Where is an option to set the weight? As far as I can see you can only > set this after peering started? > > How can I mount this tmpfs manually to inspect this? Maybe put in the > manual[1]? > > > [1] > https://docs.ceph.com/en/latest/ceph-volume/lvm/activate/ > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io -- Matt Larson, PhD Madison, WI 53705 U.S.A.

3 3

RFC: Possible replacement for ceph-disk
by Nico Schottelius 01 Oct '20

01 Oct '20

Good evening, since 2018 we have been using a custom script to create disks / partitions, because at the time both ceph-disk and ceph-volume exhibited bugs that made them unreliable for us. We recently re-tested ceph-volume and while it seems generally speaking [0] to work, using LVM seems to introduce an additional layer that is not needed from our side. The script we created is little less than 100 lines long and works by specifying the device and its device class: ./ceph-osd-create-start /dev/sdd ssd Everything else is determined by the script. As the script is very simple and work independently of any init system, we wanted to discuss whether it would make sense for anyone else to re-integrate something like it back into ceph upstream? We are aware that ceph-disk has been deprecated, however ceph-volume raw had some issues our setup [2]. The script itself can be found on [1]. At the end it has some ungleich specifics, but we'd be very open to remove that. Best regards, Nico [0] https://tracker.ceph.com/issues/47724 [1] https://code.ungleich.ch/ungleich-public/ungleich-tools/-/blob/master/ceph-… [2] [00:19:45] server8.place6:/var/lib/ceph/osd# ceph-volume raw prepare --bluestore --data /dev/sdn Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new d0fa7074-6cdf-4947-ac1e-e73dd0ec1fe8 Running command: /usr/bin/ceph-authtool --gen-print-key Running command: /bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-3 --> Executable selinuxenabled not in PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin Running command: /bin/chown -R ceph:ceph /dev/sdn Running command: /bin/ln -s /dev/sdn /var/lib/ceph/osd/ceph-3/block Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-3/activate.monmap stderr: 2020-09-28 00:20:48.276 7f8c90f27700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory 2020-09-28 00:20:48.276 7f8c90f27700 -1 AuthRegistry(0x7f8c8c081d08) no keyring found at /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx stderr: got monmap epoch 12 Running command: /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-3/keyring --create-keyring --name osd.3 --add-key AQA/EHFfINr/HhAAVFAP9NF2LLGFrJEGXbbMSw== stdout: creating /var/lib/ceph/osd/ceph-3/keyring added entity osd.3 auth(key=AQA/EHFfINr/HhAAVFAP9NF2LLGFrJEGXbbMSw==) Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3/keyring Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3/ Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 3 --monmap /var/lib/ceph/osd/ceph-3/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-3/ --osd-uuid d0fa7074-6cdf-4947-ac1e-e73dd0ec1fe8 --setuser ceph --setgroup ceph stderr: 2020-09-28 00:20:48.760 7f6e34ebfd80 -1 bluestore(/var/lib/ceph/osd/ceph-3/) _read_fsid unparsable uuid stderr: 2020-09-28 00:20:55.912 7f6e34ebfd80 -1 bdev(0x561a9a0f6700 /var/lib/ceph/osd/ceph-3//block) _lock flock failed on /var/lib/ceph/osd/ceph-3//block stderr: 2020-09-28 00:20:55.912 7f6e34ebfd80 -1 bdev(0x561a9a0f6700 /var/lib/ceph/osd/ceph-3//block) open failed to lock /var/lib/ceph/osd/ceph-3//block: (11) Resource temporarily unavailable stderr: 2020-09-28 00:20:55.912 7f6e34ebfd80 -1 OSD::mkfs: couldn't mount ObjectStore: error (11) Resource temporarily unavailable stderr: 2020-09-28 00:20:55.912 7f6e34ebfd80 -1 ** ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-3/: (11) Resource temporarily unavailable --> Was unable to complete a new OSD, will rollback changes Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.3 --yes-i-really-mean-it stderr: 2020-09-28 00:20:56.072 7f3e446a9700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory 2020-09-28 00:20:56.072 7f3e446a9700 -1 AuthRegistry(0x7f3e3c081d08) no keyring found at /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx stderr: purged osd.3 --> RuntimeError: Command failed with exit code 250: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 3 --monmap /var/lib/ceph/osd/ceph-3/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-3/ --osd-uuid d0fa7074-6cdf-4947-ac1e-e73dd0ec1fe8 --setuser ceph --setgroup ceph [00:20:56] server8.place6:/var/lib/ceph/osd# -- Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch

1 0

cephfs tag not working
by Andrej Filipcic 01 Oct '20

01 Oct '20

Hi, on octopus 15.2.4 I have an issue with cephfs tag auth. The following works fine: client.f9desktop key: .... caps: [mds] allow rw caps: [mon] allow r caps: [osd] allow rw pool=cephfs_data, allow rw pool=ssd_data, allow rw pool=fast_data, allow rw pool=arich_data, allow rw pool=ecfast_data but this one does not. client.f9desktopnew key: .... caps: [mds] allow rw caps: [mon] allow r caps: [osd] allow rw tag cephfs data=cephfs For the 2nd, mds works, files can be created or removed, but client read/write (native client, kernel version 5.7.4) fails with I/O error, so osd part does not seem to be working properly. Any clues what can be wrong? the cephfs was created in jewel... Another issue is: if osd caps are updated (adding data pool), then some clients refresh the caps, but most of them do not, and the only way to refresh it is to remount the filesystem. working tag would solve it. Best regards, Andrej -- _____________________________________________________________ prof. dr. Andrej Filipcic, E-mail: Andrej.Filipcic(a)ijs.si Department of Experimental High Energy Physics - F9 Jozef Stefan Institute, Jamova 39, P.o.Box 3000 SI-1001 Ljubljana, Slovenia Tel.: +386-1-477-3674 Fax: +386-1-477-3166 -------------------------------------------------------------

4 4

Ceph Tech Talk: Karan Singh - Scale Testing Ceph with 10Billion+ Objects
by Mike Perez 01 Oct '20

01 Oct '20

Hey all, We're live now with the latest Ceph tech talk! Join us: https://bluejeans.com/908675367/browser -- Mike Perez he/him Ceph Community Manager M: +1-951-572-2633 494C 5D25 2968 D361 65FB 3829 94BC D781 ADA8 8AEA @Thingee <https://twitter.com/thingee> Thingee <https://www.linkedin.com/thingee> <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

2 1

CEPH iSCSI issue - ESXi command timeout
by Golasowski Martin 01 Oct '20

01 Oct '20

Dear All, a week ago we had to reboot our ESXi nodes since our CEPH cluster sudennly stopped serving all I/O. We have identified a VM (vCenter appliance) which was swapping heavily and causing heavy load. However, since then we are experiencing strange issues, as if the cluster cannot handle any spike in I/O load like migration or VM reboot. The main problem is that the iSCSI commands issued by ESXi sometimes time out and ESXi reports inaccessible datastore. It disrupts the I/O heavily, we had to reboot the vmware cluster entirely several times. It started suddennly after approx 10 months of operation without problems. I can see a steadily increasing number of dropped Rx packets on the iSCSI network interfaces in the OSDs. Our CEPH setup is following: 4 OSDs, each having 3 10TB 7.2k rpm HDDs. The OSDs are connected by 25 Gbps Ethernet to the other nodes. For the RBD pools I have 64 PGs. The OSDs have 32 GB RAM, free is around 1G on each, I have seen even lower, though. OS is CentOS 7, CEPH release is Nautilus 14.2.11 deployed by ceph-ansible. MONs are virtualized in ESXi nodes on the local SSD drives. iSCSI NICs are on separate VLAN, other traffic is served via bond with balance-xor (LACP is unusable due to VMware limitation for using SW iSCSI HBA) in a different VLAN. Our network is Mellanox based - SN2100 switches and Connect-X 5 NICs. The iSCSI target serves 2 LUNs in RBD pool which is erasure coded. Yesterday I have increased the number of PGs for that pool from 64 to 128, without much effect after the cluster finished rebalancing. In OSD servers kernel log we see the following: [299560.618893] iSCSI Login negotiation failed. [303088.450088] Did not receive response to NOPIN on CID: 0, failing connection for I_T Nexus iqn.1994-05.com.redhat:esxi1,i,0x00023d000002,iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw,t,0x01 [324926.694077] Did not receive response to NOPIN on CID: 0, failing connection for I_T Nexus iqn.1994-05.com.redhat:esxi2,i,0x00023d000001,iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw,t,0x01 [407067.404538] ABORT_TASK: Found referenced iSCSI task_tag: 5891 [407076.077175] ABORT_TASK: Sending TMR_FUNCTION_COMPLETE for ref_tag: 5891 [411677.887690] ABORT_TASK: Found referenced iSCSI task_tag: 6722 [411683.297425] ABORT_TASK: Sending TMR_FUNCTION_COMPLETE for ref_tag: 6722 The error in ESXi looks like this: naa.60014053b46fc760ff0470dbd7980263" on path "vmhba64:C1:T0:L0" Failed: 2020-10-01T05:38:51.291Z cpu49:2144076)NMP: nmp_ThrottleLogForDevice:3856: Cmd 0x89 (0x459a5b1b9480, 2097241) to dev "naa.6001405a527d78935724451aa5f53513" on path "vmhba64:C2:T0:L1" Failed: 2020-10-01T05:38:57.098Z cpu44:2099346)NMP: nmp_ThrottleLogForDevice:3856: Cmd 0x8a (0x45ba96710ec0, 2107403) to dev "naa.60014053b46fc760ff0470dbd7980263" on path "vmhba64:C1:T0:L0" Failed: 2020-10-01T05:38:57.122Z cpu71:2098965)NMP: nmp_ThrottleLogForDevice:3856: Cmd 0x89 (0x45ba9676aec0, 2146212) to dev "naa.60014053b46fc760ff0470dbd7980263" on path "vmhba64:C1:T0:L0" Failed: 2020-10-01T05:38:57.256Z cpu65:2098959)NMP: nmp_ThrottleLogForDevice:3856: Cmd 0x89 (0x459a4179d8c0, 2146269) to dev "naa.6001405a527d78935724451aa5f53513" on path "vmhba64:C2:T0:L1" Failed: We would appreciate any help you can give us. Thank you very much. Regards, Martin Golasowski

1 0

bugs ceph-volume scripting
by Marc Roos 01 Oct '20

01 Oct '20

I have been creating lvm osd's with: ceph-volume lvm zap --destroy /dev/sdf && ceph-volume lvm create --data /dev/sdf --dmcrypt Because this procedure failed: ceph-volume lvm zap --destroy /dev/sdf (waiting on slow human typing) ceph-volume lvm create --data /dev/sdf --dmcrypt However when I was looking at the /var/lib/ceph/osd I expected these lvm mounts to list as world writable[1] So I decided to compare: [@osd]# cp -a ceph-33 ceph-33.bak2 [@osd]# service ceph-osd@33 stop [@osd]# service ceph-volume@lvm-33-9a7a9a7c-8fc8-441c-8380-acf7f8b1a670 start [BUG1] With the zap && create, I seem to be running the osd without the tmpfs mounted?! So that means that if I reboot this node and the content is different from the tmpfs. I have a serious problem. So I am trying to unmount ceph-33 [BUG2] Wtf ceph-osd@33 is running! Something started (ceph-volume?) the osd, without even having a chance to inspect the difference of these folders. If ceph-volume 'by design' starts the osd, then design is bad, nobody is expecting this behaviour and I have no idea what can go wrong if this startup data in tmpfs ceph-33 mount is different from the lvm create files on the os disk. Stopping again osd-33 [@osd]# service ceph-osd@33 stop Trying to unmount again ceph-33 [@osd]# service ceph-volume@lvm-33-9a7a9a7c-8fc8-441c-8380-acf7f8b1a670 stop [BUG3] service ceph-volume just does not unmount tmpfs. I have to unmount with umount /var/lib/ceph/osd/ceph-33 Inspecting differences of both, [@osd]# ls -l ceph-33.bak2 ceph-33.new ceph-33.new: total 28 lrwxrwxrwx 1 ceph ceph 50 Oct 1 10:06 block -> /dev/mapper/1K8AX3-D3Gv-VKdY-0wTW-qjgd-txAu-JbNJHo -rw------- 1 ceph ceph 37 Oct 1 10:06 ceph_fsid -rw------- 1 ceph ceph 37 Oct 1 10:06 fsid -rw------- 1 ceph ceph 56 Oct 1 10:06 keyring -rw------- 1 ceph ceph 106 Oct 1 10:06 lockbox.keyring -rw------- 1 ceph ceph 6 Oct 1 10:06 ready -rw------- 1 ceph ceph 10 Oct 1 10:06 type -rw------- 1 ceph ceph 3 Oct 1 10:06 whoami ceph-33.bak2: total 56 -rw-r----- 1 ceph ceph 373 Sep 30 21:23 activate.monmap lrwxrwxrwx 1 ceph ceph 50 Sep 30 21:23 block -> /dev/mapper/1K8AX3-D3Gv-VKdY-0wTW-qjgd-txAu-JbNJHo -rw------- 1 ceph ceph 2 Sep 30 21:23 bluefs -rw------- 1 ceph ceph 37 Sep 30 21:23 ceph_fsid -rw-r----- 1 ceph ceph 37 Sep 30 21:23 fsid -rw------- 1 ceph ceph 56 Sep 30 21:23 keyring -rw------- 1 ceph ceph 8 Sep 30 21:23 kv_backend -rw------- 1 ceph ceph 106 Sep 30 21:23 lockbox.keyring -rw------- 1 ceph ceph 21 Sep 30 21:23 magic -rw------- 1 ceph ceph 4 Sep 30 21:23 mkfs_done -rw------- 1 ceph ceph 41 Sep 30 21:23 osd_key -rw------- 1 ceph ceph 6 Sep 30 21:23 ready -rw------- 1 ceph ceph 3 Sep 30 21:23 require_osd_release -rw------- 1 ceph ceph 10 Sep 30 21:23 type -rw------- 1 ceph ceph 3 Sep 30 21:23 whoami The contents of the files in new (tmpfs) luckily are the same as in bak2 (ceph-volume create?). However as you can see I miss quite a few files in the tmpfs So I am giving it a try and start this osd.33 with the tmpfs mounted. [@osd]# service ceph-volume@lvm-33-9a7a9a7c-8fc8-441c-8380-acf7f8b1a670 start^C [@osd]# ps -ef | grep ceph-osd | grep 33 [@osd]# service ceph-volume@lvm-33-9a7a9a7c-8fc8-441c-8380-acf7f8b1a670 start Redirecting to /bin/systemctl start ceph-volume(a)lvm-33-9a7a9a7c-8fc8-441c-8380-acf7f8b1a670.service And indeed, again ceph-osd is started. [@osd]# ps -ef | grep ceph-osd | grep 33 ceph 1651105 1 48 11:29 ? 00:00:00 /usr/bin/ceph-osd -f --cluster ceph --id 33 --setuser ceph --setgroup ceph [QUESTION1] Should I just copy these files like kv_backend, mkfs_done to the tmpfs mount? I seem to have these files on other ceph-volume created osd's. [QUESTION2] Is there a reasonable explanation for running into such issues, before I start thinking this is shitty scripting? [1] https://tracker.ceph.com/issues/47549

1 0

S3 Buckets with "object-lock"
by Torsten Ennenbach 01 Oct '20

01 Oct '20

Hello, we are using Ceph 14.x for our s3 storages and some of our customers want to create a locked object bucket. BUT: While the creation of a locked bucket works, the objects are still deletable. Any ideas or hints? Best regards: Torsten

1 0

hdd pg's migrating when converting ssd class osd's
by Marc Roos 01 Oct '20

01 Oct '20

I have been converting ssd's osd's to dmcrypt, and I have noticed that pg's of pools are migrated that should be (and are?) on hdd class. On a healthy ok cluster I am getting, when I set the crush reweight to 0.0 of a ssd osd this: 17.35 10415 0 0 9907 0 36001743890 0 0 3045 3045 active+remapped+backfilling 2020-09-27 12:55:49.093054 83758'20725398 83758:100379720 [8,14,23] 8 [3,14,23] 3 83636'20718129 2020-09-27 00:58:07.098096 83300'20689151 2020-09-24 21:42:07.385360 0 However osds 3,14,23,8 are all hdd osd's Since this is a cluster from Kraken/Luminous, I am not sure if the device class of the replicated_ruleset[1] was set when the pool 17 was created. Weird thing is that all pg's of this pool seem to be on hdd osd[2] Q. How can I display the definition of 'crush_rule 0' at the time of the pool creation? (To be sure it had already this device class hdd configured) [1] [@~]# ceph osd pool ls detail | grep 'pool 17' pool 17 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode warn last_change 83712 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd [@~]# ceph osd crush rule dump replicated_ruleset { "rule_id": 0, "rule_name": "replicated_ruleset", "ruleset": 0, "type": 1, "min_size": 1, "max_size": 10, "steps": [ { "op": "take", "item": -10, "item_name": "default~hdd" }, { "op": "chooseleaf_firstn", "num": 0, "type": "host" }, { "op": "emit" } ] } [2] [@~]# for osd in `ceph pg dump pgs| grep '^17' | awk '{print $17" "$19}' | grep -oE '[0-9]{1,2}'| sort -u -n`; do ceph osd crush get-device-class osd.$osd ; done | sort -u dumped pgs hdd

5 19