- ceph-users - lists.ceph.io

Newbie question - how do I add more monitors?
by c.stoddart＠sheffield.ac.uk 24 Apr '20

24 Apr '20

version 14.2.9 nautilus I have just set up my first cluster with one admin/monitor host and six OSDs. Everything appears to be working fine. I wanted to add a couple more monitors on the OSDs, and following the quick ceph deploy documentation I did: ceph-deploy mon add icephnode01 ceph-deploy mon add icephnode04 This installs and starts the monitor daemons on the two nodes, but they are stuck at probing. They have the right networking and firewall exemptions as far as I can tell, so I don't think it's that. If I get a copy of the monmap with 'ceph mon getmap' and use monmaptool to look at it I can only see that one host. Likewise ceph.conf only contains the one initial host. Should I have done something before the ceph-deploy mon add to modify the monmap? Or am I way off base? Thanks for your patience with a beginner question! Chris

3 2

Check if upmap is supported by client?
by Frank Schilder 24 Apr '20

24 Apr '20

Dear all, I would like to enable the balancer on a mimic 13.2.8 cluster in upmap mode. Unfortunately, I have a lot of ceph fs kernel clients that report their version as jewel, but might already support upmap. The ceph client kernel module received already a lot of back-ports and supports features of later ceph versions, for example, quotas. I guess they report back jewel, because not all luminous/mimic features are back-ported yet. Is there a way to check if a client supports upmap? Here some info: [root@gnosis ~]# ceph features { [...] "client": [ { "features": "0x27018fb86aa42ada", "release": "jewel", "num": 1676 }, { "features": "0x2f018fb86aa42ada", "release": "luminous", "num": 1 }, { "features": "0x3ffddff8ffacfffb", "release": "luminous", "num": 167 } ], The fs clients are the top two entries, the third entry is rbd clients. Note that the feature key for the fs clients is almost identical. Here a snippet from mds session ls for one such jewel client: { "id": 25641514, "num_leases": 0, "num_caps": 1, "state": "open", "request_load_avg": 0, "uptime": 588563.550276, "replay_requests": 0, "completed_requests": 0, "reconnecting": false, "inst": "client.25641514 192.168.57.124:0/3398308464", "client_metadata": { "features": "00000000000000ff", "entity_id": "con-fs2-hpc", "hostname": "sn253.hpc.ait.dtu.dk", "kernel_version": "3.10.0-957.12.2.el7.x86_64", "root": "/hpc/groups" } } Since I would like to use upmap right from the beginning, my alternative is to re-weight a few of the really bad outliers manually to simplify changing back. What would you suggest? Thanks and best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

3 4

Repeatedly OSD crashes in PrimaryLogPG::hit_set_trim()
by KOT MATPOCKuH 24 Apr '20

24 Apr '20

Hello all! I'm got repeatedlyOSD crashes for 3 of OSD which stracktrace: ceph version 13.2.4 (b10be4d44915a4d78a8e06aa31919e74927b142e) mimic (stable) 1: (()+0x911e70) [0x564d0067fe70] 2: (()+0xf5d0) [0x7f1272dad5d0] 3: (gsignal()+0x37) [0x7f1271dce2c7] 4: (abort()+0x148) [0x7f1271dcf9b8] 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x242) [0x7f12762252b2] 6: (()+0x25a337) [0x7f1276225337] 7: (PrimaryLogPG::hit_set_trim(std::unique_ptr<PrimaryLogPG::OpContext, std::default_delete<PrimaryLogPG::OpContext> >&, unsigned int)+0x930) [0x564d002ab480] 8: (PrimaryLogPG::hit_set_persist()+0xa0c) [0x564d002afafc] 9: (PrimaryLogPG::do_op(boost::intrusive_ptr<OpRequest>&)+0x2989) [0x564d002c5f09] 10: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0xc99) [0x564d002cac09] 11: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x1b7) [0x564d00124c87] 12: (PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x62) [0x564d0039d8c2] 13: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x592) [0x564d00144ae2] 14: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x3d3) [0x7f127622aec3] 15: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x7f127622bab0] 16: (()+0x7dd5) [0x7f1272da5dd5] 17: (clone()+0x6d) [0x7f1271e95f6d] I think this happens after some network troubles and one PG was marked recovery_unfound, to solve this problem this command was executed: ceph pg 2.f8 mark_unfound_lost revert Also last lines in debug output before crash are related to this PG: -10001> 2020-04-23 14:16:18.790 7f534915a700 10 osd.12 pg_epoch: 10476 pg[2.f8( v 8673'30651498 (6953'30648450,8673'30651498] local-lis/les=10475/10476 n=14 ec=66/66 lis/c 10475/10427 les/c/f 10476/10428/201 10475/10475/10473) [12,13,17] r=0 lpr=10475 pi=[10427,10475)/2 crt=8673'30651498 lcod 0'0 mlcod 0'0 active mbc={}] get_object_context: obc NOT found in cache: 2:1f000000:.ceph-internal::hit_set_2.f8_archive_2020-04-22 02%3a57%3a10.496532Z_2020-04-22 03%3a57%3a11.211949Z:head -10001> 2020-04-23 14:16:18.790 7f534915a700 10 osd.12 pg_epoch: 10476 pg[2.f8( v 8673'30651498 (6953'30648450,8673'30651498] local-lis/les=10475/10476 n=14 ec=66/66 lis/c 10475/10427 les/c/f 10476/10428/201 10475/10475/10473) [12,13,17] r=0 lpr=10475 pi=[10427,10475)/2 crt=8673'30651498 lcod 0'0 mlcod 0'0 active mbc={}] get_object_context: no obc for soid 2:1f000000:.ceph-internal::hit_set_2.f8_archive_2020-04-22 02%3a57%3a10.496532Z_2020-04-22 03%3a57%3a11.211949Z:head and !can_create I'm evicted must of PG's from cache pool, and currently we have only two rados objects in this PG: # rados --pgid 2.f8 ls rbd_data.10d4416b8b4567.0000000000001dfb rbd_header.1db946b8b4567 I tried to remove one of them: rados -p vms-cache rm rbd_header.1db946b8b4567 But command not completed for last ~8 hours, because repeatedly OSD crashes. Does we have any ways to solve this problem without data loss? -- MATPOCKuH

1 0

v13.2.10 Mimic released
by Abhishek Lekshmanan 24 Apr '20

24 Apr '20

We're happy to announce the availability of the tenth bugfix release of Ceph Mimic, this release fixes a RGW vulnerability affecting mimic, and we recommend that all mimic users upgrade. Notable Changes --------------- * CVE 2020 12059: Fix an issue with Post Object Requests with Tagging (#44967, Lei Cao, Abhishek Lekshmanan) Getting Ceph ------------ * Git at git://github.com/ceph/ceph.git * Tarball at http://download.ceph.com/tarballs/ceph-13.2.10.tar.gz * For packages: http://docs.ceph.com/docs/master/install/get-packages/ * Release git sha1: 564bdc4ae87418a232fc901524470e1a0f76d641 * Release blog: https://ceph.io/releases/v13-2-10-mimic-released -- Abhishek Lekshmanan SUSE Software Solutions Germany GmbH GF: Felix Imendörffer

1 0

Increase number of read and writes
by Bobby 23 Apr '20

23 Apr '20

Hi, I am using Ceph in developer mode. Currently I am implementing Librados examples which are also available in Introduction to Librados section https://docs.ceph.com/docs/master/rados/api/librados-intro/#step-3-creating…. It says once your app has a cluster handle and a connection to a Ceph Storage Cluster, you may create an I/O Context and begin reading and writing data. For example, *err = rados_write(io, "hw", "Hello World!", 12, 0); if (err < 0) { fprintf(stderr, "%s: Cannot write object \"neo-obj\" to pool %s: %s\n", argv[0], poolname, strerror(-err)); rados_ioctx_destroy(io); rados_shutdown(cluster); exit(1); } else { printf("\nWrote \"Hello World\" to object \"neo-obj\".\n"); }* My question, Is "12" is the number of writes? Because I want to test the with high number of read and writes. Looking for help !

2 2

Dear Abby: Why Is Architecting CEPH So Hard?
by Linus VanWeil 23 Apr '20

23 Apr '20

Hello, Looks like the original chain got deleted, but thank you to everyone who responded. Just to keep any new-comers in the loop, I have pasted the original positing below. To all the original contributors to this chain, I feel much more confident in my design theory for the storage nodes. However, I wanted to narrow the focus and see if I can get any elaborated comments on the two below topics. Does anyone have any real-world data on metrics I can use to size MONs? When are they active? When do they utilize CPU, RAM, Storage (ie. larger storage pools require more resources, resources are used during recovery, etc.)? For anyone that commented or has opinions on Storage node sizing: How does choosing EC vs 3X replication affect your sizing of CPU / RAM? IS the some kind of over-head generalization I can use if assuming EC (ie. add an extra core per OSD)? I understand that recoveries are where this is most important, so I am looking for sizing metrics based on living through worst case scenarios. --------------------------- ORIGINAL POSTING: Hey Folks, This is my first ever post here in the CEPH user group and I will preface with the fact that I know this is a lot of what many people ask frequently. Unlike what I assume to be a large majority of CEPH “users” in this forum, I am more of a CEPH “distributor.” My interests lie in how to build a CEPH environment to best fill an organization’s needs.I am here for the real-world experience and expertise so that I can learn to build CEPH “right.” I have spent the last couple years collecting data on general “best practices” through forum posts, CEPH documentation, CEPHLACON, etc. I wanted to post my findings to the forum to see where I can harden my stance. Below are two example designs that I might use when architecting a solution currently. I have specific questions around design elements in each that I would like you to approve for holding water or not. I want to focus on the hardware, so I am asking for generalizations where possible. Let’s assume in all scenarios that we are using Luminous and that the data type is mixed use. I am not expecting anyone to run through every question, so please feel free to comment on any piece you can. Tell me what is overkill and what is lacking! Example 1: 8x 60-Bay (8TB) Storage nodes (480x 8TB SAS Drives) Storage Node Spec: 2x 32C 2.9GHz AMD EPYC - Documentation mentions .5 cores per OSD for throughput optimized. Are they talking about .5 Physical cores or .5 Logical cores? - Is it better to pick my processors based on a total GHz measurement like 2GHz per OSD? - Would a theoretical 8C at 2GHz serve the same number of OSDs as a 16C at 1GHz? Would Threads be included in this calculation? 512GB Memory - I know this is the hot topic because of its role in recoveries. Basically, I am looking for the most generalized practice I can use as a safe number and a metric I can use as a nice to have. - Is it 1GB of RAM per TB of RAW OSD? 2x 3.2TB NVMe WAHLDB / Log Drive - Another hot topic that I am sure will bring many “it depends.” All I am looking for is experience on this. I know people have mentioned having at least 70GB of Flash for WAHLDB / Logs. - Can I use 70GB as a flat calculation per OSD or is it depend on the Size of the OSD? - I know more is better, but what is a number I can use to get started with minimal issues? 2x 56Gbit Links - I think this should be enough given the rule of thumb of 10Gbit for every 12 OSDs. 3x MON Node MON Node Spec: 1x 8C 3.2GHz AMD EPYC - I can’t really find good practices around when to increase your core count. Any suggestions? 128GB Memory - What do I need memory for in a MON? - When do I need to expand? 2x 480GB Boot SSDs - Any reason to look more closely into the sizing of these drives? 2x 25Gbit Uplinks - Should these match the output of the storage nodes for any reason? Example 2: 8x 12-Bay NVMe Storage nodes (96x 1.6TB NVMe Drives) Storage Node Spec: 2x 32C 2.9GHz AMD EPYC - I have read that each NMVe OSD should have 10 cores. I am not splitting Physical drives into multiple OSDs so let’s assume I have 12 OSD per Node. - Would threads count toward my 10 core quota or just physical cores? - Can I do a similar calculation as I mentioned before and just use 20GHz per OSD instead of focusing on cores specifically? 512GB Memory - I assume there is some reason I can’t use the same methodology of 1GB per TB of OSD since this is NVMe storage 2x 100Gbit Links - This is assuming about 1Gigabyte per second of real-world speed per disk 3x MON Node – What differences should MONs serving NVMe have compared to large NLSAS pools? MON Node Spec: 1x 8C 3.2GHz AMD Epyc 128GB Memory 2x 480GB Boot SSDs 2x 25Gbit Uplinks

1 0

Healthy objects trapped in incomplete pgs
by Jesper Lykkegaard Karlsen 23 Apr '20

23 Apr '20

Dear Cephers, A few days ago disaster struck the Ceph cluster (erasure-coded) I am administrating, as the UPS power was pull from the cluster causing a power outage. After rebooting the system, 6 osds were lost (spread over 5 osd nodes) as they could not mount anymore, several others had damages. This was more than the host-faliure domain was setup to handle and auto-recovery failed and osds started downing in a cascading maner. When the dust settled, there were 8 pgs (of 2048) inactive and a bunch of osds down. I managed to recover 5 pgs, mainly by ceph-objectstore-tool export/import/repair commands, but now I am left with 3 pgs that are inactive and incomplete. One of the pgs seems un-salvageable, as I cannot get to become active at all (repair/import/export/lowering min_size), but the two others I can get active if I export/import one of the pg shards and restart osd. Rebuilding then starts but after a while one of the osds holding the pgs goes down, with a "FAILED ceph_assert(clone_size.count(clone))" message in the log. If I set osds to noout nodown, then I can that it is only rather few objects e.g. 161 of a pg of >100000, that are failing to be remapped. Since most of the object in the two pgs seem intact, it would be sad to delete the whole pg (force-create-pg) and loose all that data. Is there a way to show and delete the failing objects? I have thought of a recovery plan and want to share that with you, so you can comment on this if it sounds doable or not? * Stop osds from recovering: ceph osd set norecover * bring back pgs active: ceph-objectstore-tool export/import and restart osd * find files in pgs: cephfs-data-scan pg_files <path> <pg id> * pull out as many as possible of those files to other location. * recreate pgs: ceph osd force-create-pg <pgid> * restart recovery: ceph osd unset norecover * copy back in the recovered files Would that work or do you have a better suggestion? Cheers, Jesper -------------------------- Jesper Lykkegaard Karlsen Scientific Computing Centre for Structural Biology Department of Molecular Biology and Genetics Aarhus University Gustav Wieds Vej 10 8000 Aarhus C E-mail: jelka(a)mbg.au.dk Tlf: +45 50906203

1 0

missing amqp-exchange on bucket-notification with AMQP endpoint
by Andreas Unterkircher 23 Apr '20

23 Apr '20

Hello List, I'm trying to create a (S3-)bucket-notification into RabbitMQ via AMQP - on Ceph v15.2.1 octopus, using the official .deb packages on Debian Buster. I've created the following topic (directly via S3, not via pubsub REST API): <ListTopicsResponse xmlns="https://sns.amazonaws.com/doc/2010-03-31/"> <ListTopicsResult> <Topics> <member> <User>testuser</User> <Name>testtopic</Name> <EndPoint> <EndpointAddress>amqp://rabbitmquser:rabbitmqpass@rabbitmq.example.com:5672</EndpointAddress> <EndpointArgs>Attributes.entry.1.key=amqp-exchange&Attributes.entry.1.value=amqp.direct&push-endpoint=amqp://rabbitmquser:rabbitmqpass@rabbitmq.example.com:5672</EndpointArgs> <EndpointTopic>testtopic</EndpointTopic> </EndPoint> <TopicArn>arn:aws:sns:de::testtopic</TopicArn> <OpaqueData></OpaqueData> </member> </Topics> </ListTopicsResult> ... </ListTopicsResponse> Then I've created the following bucket-notification <NotificationConfiguration> <TopicConfiguration> <Id>notify-psapp</Id> <Topic>arn:aws:sns:de::testtopic</Topic> <Event>s3:ObjectCreated:*</Event> <Event>s3:ObjectRemoved:*</Event> </TopicConfiguration> </NotificationConfiguration> When I upload a file into the bucket, the event itself seems to get fired, but radosgw keeps tell me that cmqp-exchange is not set 2020-04-20T12:24:29.935+0200 7ff01c5d3700 1 ====== starting new request req=0x7ff01c5cad50 ===== 2020-04-20T12:24:30.019+0200 7ff01c5d3700 1 ERROR: failed to create push endpoint: amqp://rabbitmquser:rabbitmqpass@rabbitmq.example.com:5672 due to: pubsub endpoint configuration error: AMQP: missing amqp-exchange But it's there in the EndpointArgs, right? Or do I miss it somewhere else? Best Regards, Andreas

2 4

How to remove a deamon from orch
by Ml Ml 23 Apr '20

23 Apr '20

Hello list, i somehow have this "mgr.cph02 ceph02 stopped " line here. root@ceph01:~# ceph orch ps NAME HOST STATUS REFRESHED AGE VERSION IMAGE NAME IMAGE ID CONTAINER ID mgr.ceph02 ceph02 running (2w) 2w ago - 15.2.0 docker.io/ceph/ceph:v15 204a01f9b0b6 4e349a382c6b mgr.ceph03 ceph03 running (2w) 2w ago - 15.2.0 docker.io/ceph/ceph:v15 204a01f9b0b6 2a9a037e5e2d mgr.cph02 ceph02 stopped 2w ago - <unknown> <unknown> <unknown> <unknown> mon.ceph02 ceph02 running (2w) 2w ago - 15.2.0 docker.io/ceph/ceph:v15 204a01f9b0b6 cf66ca51c0dd mon.ceph03 ceph03 running (2w) 2w ago - 15.2.0 docker.io/ceph/ceph:v15 204a01f9b0b6 fceaaa03b41f I actually cant remember how i did that. How can i remove that wrong "mgr.cph02" entry? Thanks, Michael

1 0

Re: docs.ceph.com certificate expired?
by Jos Collin 22 Apr '20

22 Apr '20

Fixed On 22/04/20 6:57 pm, Bobby wrote: > > Thanks ! When will it be back? > > On Wed, Apr 22, 2020 at 3:03 PM <ulrich.weigand(a)de.ibm.com > <mailto:ulrich.weigand@de.ibm.com>> wrote: > > Hello, > > trying to access the documentation on docs.ceph.com > <http://docs.ceph.com> now results in an error: The certificate > expired on April 22, 2020, 8:46 AM. > > Bye, > Ulrich > _______________________________________________ > Dev mailing list -- dev(a)ceph.io <mailto:dev@ceph.io> > To unsubscribe send an email to dev-leave(a)ceph.io > <mailto:dev-leave@ceph.io> > > > _______________________________________________ > Dev mailing list -- dev(a)ceph.io > To unsubscribe send an email to dev-leave(a)ceph.io

1 0