- ceph-users - lists.ceph.io

Bucket sharding
by Szabo, Istvan (Agoda) 14 Oct '20

14 Oct '20

Hello, I have a bucket which is close to 10 millions objects (9.1 millions), we have: rgw_dynamic_resharding = false rgw_override_bucket_index_max_shards = 100 rgw_max_objs_per_shard = 100000 Do I need to increase the numbers soon or it is not possible so they need to start to use new bucket? The version is luminous 12.2.8. I don't want outage so want to be prepared. Thank you ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

2 2

Ceph test cluster, how to estimate performance.
by Daniel Mezentsev 14 Oct '20

14 Oct '20

Hi Ceph users, Im working on common lisp client utilizing rados library. Got some results, but don't know how to estimate if i am getting correct performance. I'm running test cluster from laptop - 2 OSDs - VM, RAM 4Gb, 4 vCPU each, monitors and mgr are running from the same VM(s). As for storage, i have Samsung SSD 860 Pro, 512G. Disk is splitted into 2 logical volumes (LVMs), and that volumes are attached to VMs. I know that i can't expect too much from that layout, just want to know if im getting adequate numbers. Im doing read/write operations on very small objects - up to 1kb. In async write im getting ~7.5-8.0 KIOPS. Synchronouse read - pretty much the same 7.5-8.0 KIOPS. Async read is segfaulting don't know why. Disk itself is capable to deliver well above 50 KIOPS. Difference is magnitude. Any info is more welcome. Daniel Mezentsev, founder (+1) 604 313 8592. Soleks Data Group. Shaping the clouds.

3 3

DocuBetter Meeting 14 Oct 2020 -- 24 hours from the time of this email.
by John Zachary Dover 13 Oct '20

13 Oct '20

There is a general documentation meeting called the "DocuBetter Meeting", and it is held every two weeks. The next DocuBetter Meeting will be on 14 Oct 2020 at 1630 UTC, and will run for thirty minutes. Everyone with a documentation-related request or complaint is invited. The meeting will be held here: https://bluejeans.com/908675367 Send documentation-related requests and complaints to me by replying to this email and CCing me at zac.dover(a)gmail.com. The next DocuBetter meeting is scheduled for: 14 Oct 2020 1630 UTC Etherpad: https://pad.ceph.com/p/Ceph_Documentation Zac's docs whiteboard: https://pad.ceph.com/p/docs_whiteboard Report Documentation Bugs: https://pad.ceph.com/p/Report_Documentation_Bugs Meeting: https://bluejeans.com/908675367

1 0

Ubuntu 20 with octopus
by Seena Fallah 13 Oct '20

13 Oct '20

Hi all, Does anyone has any production cluster with ubuntu 20 (focal) or any suggestion or any bugs that prevents to deploy Ceph octopus on Ubuntu 20? Thanks.

5 6

Announcing go-ceph v0.6.0
by John Mulligan 13 Oct '20

13 Oct '20

I'm happy to announce the another release of the go-ceph API bindings. This is a regular release following our every-two-months release cadence. https://github.com/ceph/go-ceph/releases/tag/v0.6.0 Changes in the release are detailed in the link above. The bindings aim to play a similar role to the "pybind" python bindings in the ceph tree but for the Go language. These API bindings require the use of cgo. There are already a few consumers of this library in the wild, including the ceph-csi project. Specific questions, comments, bugs etc are best directed at our github issues tracker. -- John Mulligan phlogistonjohn(a)asynchrono.us jmulligan(a)redhat.com

1 0

BlueFS spillover detected - correct response for 14.2.7?
by Dave Hall 13 Oct '20

13 Oct '20

Hello, We are running a 14.2.7 cluster - 3 nodes with 24 OSDs, I've recently started getting 'BlueFS spillover detected'. I'm up to 3 OSDs in this state. In scanning through the various online sources I haven't been able to determine how to respond to this condition. Please advise. Thanks. -Dave -- Dave Hall Binghamton University kdhall(a)binghamton.edu

2 1

cephadm numa aware config
by nokia ceph 13 Oct '20

13 Oct '20

Hi Team, I would like to validate cephadm on bare metal and use docker/podman as a container. Currently we use NUMA aware config on bare metal to improve performance . Is there any config I can apply in cephadm to run podman/docker use run with *–cpuset-cpus*=*num * and *–cpuset-mems*=*nodes *options in startup scripts ? Thanks, Muthu

1 0

librados documentation has gone
by Daniel Mezentsev 13 Oct '20

13 Oct '20

Hi Ceph users, librados (C interface) disappered ? All google provided links are broken: \ SORRY / \ / \ This page does / ] not exist yet. [ ,'| ] [ / | ]___ ___[ ,' | ] ]\ /[ [ |: | ] ] \ / [ [ |: | ] ] ] [ [ [ |: | ] ] ]__ __[ [ [ |: | ] ] ] ]\ _ /[ [ [ [ |: | ] ] ] ] (#) [ [ [ [ :====' ] ] ]_].nHn.[_[ [ [ ] ] ] HHHHH. [ [ [ ] ] / `HH("N \ [ [ ]__]/ HHH " \[__[ ] NNN [ ] N/" [ ] N H [ / N \ / q, \ / \ Can somebody gimme heads-up what is going on ? Daniel Mezentsev, founder (+1) 604 313 8592. Soleks Data Group. Shaping the clouds.

2 2

Re: MONs are down, the quorum is unable to resolve.
by Brian Topping 13 Oct '20

13 Oct '20

I see, maybe you want to look at these instructions. I don’t know if you are running Rook, but the point about getting the container alive by using `sleep` is important. Then you can get into the container with `exec` and do what you need to. https://rook.io/docs/rook/v1.4/ceph-disaster-recovery.html#restoring-mon-qu… > On Oct 12, 2020, at 4:16 PM, Gaël THEROND <gael.therond(a)bitswalk.com> wrote: > > Hi Brian! > > Thanks a lot for your quick answer, it was fast ! > > Yes, I’ve read this doc, yet I can’t perform appropriate commands as my OSDs are up and running. > > As my mon is a container if I try to use ceph-mon —extract it won’t work as the mon process is running and if I stop it the container will be restarted and I’ll be ousted off it. > > I can’t retrieve anything from ceph mon getmap as the quorum isn’t forming. > > Yep, I know that I would need three nodes and I have a third node available since recently for this lab. > > unfortunately it’s a lab cluster and so one of my colleagues just took the third node for testing purpose... I told you, a series of unfortunate events :-) > > I can’t get rid of the cluster as I can’t lost OSDs data. > > G. > > Le mar. 13 oct. 2020 à 00:01, Brian Topping <brian.topping(a)gmail.com <mailto:brian.topping@gmail.com>> a écrit : > Hi there! > > This isn’t a difficult problem to fix. For purposes of clarity, the monmap is just a part of the monitor database. You generally have all the details correct though. > > Have you looked at the process in https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-mon/#… <https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-mon/#…> > > Please do make sure you are working on the copy of the monitor database with the newest epoch. After removing the other monitors and getting your cluster back online, you can re-add monitors at will. > > Also note that a quorum is defined as "one-half the total number of nodes plus one”. In your case, quorum is defined by both nodes! Taking either down would cause this problem. So you need to have an odd number of nodes to provide the ability to take a node down, for instance in a rolling upgrade. > > Hope that helps! > > Brian > > >> On Oct 12, 2020, at 3:54 PM, Gaël THEROND <gael.therond(a)bitswalk.com <mailto:gael.therond@bitswalk.com>> wrote: >> > > >> Hi everyone, >> >> Because of unfortunate events, I’ve a containers based ceph cluster >> (nautilus) in a bad shape. >> >> One of the lab cluster which is only made of 2 nodes as control plane (I >> know it’s bad :-)) each of these nodes run a mon, a mgr and a rados-gw >> containerized ceph_daemon. >> >> They were installed using ceph-ansible if relevant for anyone. >> >> However, when I was performing an upgrade on one of the first nodes, the >> second went down too (electrical power outage). >> >> As soon as I saw that I stopped all current process within the upgrading >> node. >> >> For now, if I try to restart my second node, as the quorum is looking for >> two node the cluster isn’t available. >> >> The container start, the node elect itself as the master but all ceph >> commands are stuck forever, which is perfectly normal as the quorum still >> wait for one member to achieve the election process etc. >> >> So, my question is, as I can’t (to my knowledge) extract the monmap with >> this intermediary state, and as my first node will still be considered as a >> known mon and try to join back if started properly, can I just copy the >> /etc/ceph.conf and /var/lib/mon/<host>/keyring from the last living node >> (the second one) and copy everything at its own place within the first >> node? My mon keys were the same for both mon initially and if I’m not >> making any mistakes my first node being blank will try to create a default >> store, join the existing cluster and try to retrieve the appropriate monmap >> from the remaining node right? >> >> If not, is there a process to be able to save/extract the monmap when using >> a container based ceph ? I can perfectly exec on the remaining node if it >> make any difference. >> >> Thanks a lot! > >> _______________________________________________ >> ceph-users mailing list -- ceph-users(a)ceph.io <mailto:ceph-users@ceph.io> >> To unsubscribe send an email to ceph-users-leave(a)ceph.io <mailto:ceph-users-leave@ceph.io> >

2 1

MONs are down, the quorum is unable to resolve.
by Gaël THEROND 13 Oct '20

13 Oct '20

Hi everyone, Because of unfortunate events, I’ve a containers based ceph cluster (nautilus) in a bad shape. One of the lab cluster which is only made of 2 nodes as control plane (I know it’s bad :-)) each of these nodes run a mon, a mgr and a rados-gw containerized ceph_daemon. They were installed using ceph-ansible if relevant for anyone. However, when I was performing an upgrade on one of the first nodes, the second went down too (electrical power outage). As soon as I saw that I stopped all current process within the upgrading node. For now, if I try to restart my second node, as the quorum is looking for two node the cluster isn’t available. The container start, the node elect itself as the master but all ceph commands are stuck forever, which is perfectly normal as the quorum still wait for one member to achieve the election process etc. So, my question is, as I can’t (to my knowledge) extract the monmap with this intermediary state, and as my first node will still be considered as a known mon and try to join back if started properly, can I just copy the /etc/ceph.conf and /var/lib/mon/<host>/keyring from the last living node (the second one) and copy everything at its own place within the first node? My mon keys were the same for both mon initially and if I’m not making any mistakes my first node being blank will try to create a default store, join the existing cluster and try to retrieve the appropriate monmap from the remaining node right? If not, is there a process to be able to save/extract the monmap when using a container based ceph ? I can perfectly exec on the remaining node if it make any difference. Thanks a lot!

2 1