- ceph-users - lists.ceph.io

Object storage multisite
by Ignazio Cassano 18 Mar '20

18 Mar '20

Hello, I have two openstack installation on different sites. They do not share any service : each one have its keystone repository, and ceph cluster with object storage and block storage. I read about object storage multisite and I could modify my object storages to enable multisite active-active. I wonder if I can do it with two different keystone . How can I read containers on site B replicated by site A ? Ignazio

1 0

Upmap balancing - pools grouped together?
by Andras Pataki 18 Mar '20

18 Mar '20

I've been trying the upmap balancer on a new Nautilus cluster. We three main pools, a triple replicated pool (id:1) and two 6+3 erasure coded pools (id: 4 and 5). The balancer does a very nice job on the triple replicated pool, but does something strange on the EC pools. Here is a sample of the PG counts on OSDs (each line) per pool (columns): OSD 1 2 3 4 5 ALL 0 25 - - 80 34 139 1 26 - - 74 42 142 2 25 - - 74 42 141 3 26 - - 75 41 142 4 25 - - 83 31 139 5 26 - - 80 35 141 6 26 - - 79 36 141 7 26 - - 72 44 142 8 25 - - 74 42 141 9 26 - - 78 38 142 10 26 - - 70 46 142 11 25 - - 78 36 139 12 26 - - 74 41 141 13 26 - - 78 37 141 14 26 - - 76 40 142 15 25 - - 82 33 140 16 26 - - 77 37 140 17 26 - - 73 43 142 18 26 - - 77 37 140 19 26 - - 79 35 140 20 26 - - 74 42 142 21 26 - - 78 36 140 22 26 - - 79 35 140 23 26 - - 81 34 141 24 26 - - 77 38 141 25 26 - - 79 35 140 Pool #1 has 25 or 26 PGs on each OSD extremely consistently. But it looks like pool #4 and #5 have a lot of variance, but their *sum* seems to be well balanced (and hence the total PGs per OSD is also very consistent). As if the balancer was balancing the two pools together as one. Both pools are EC with 6+3 profile, and use the same crush rule. Could that be the reason? If not - any ideas? Also - a separate balancer related question: is there a way to have the balancer balace the size on each OSD per pool as opposed to the number of PGs? If not currently - is it something hard to implement? Andras

2 1

Inactive PGs
by Peter Eisch 18 Mar '20

18 Mar '20

Full cluster is 14.2.8. I had some OSD drop overnight which results now in 4 inactive PGs. The pools had three participant (2 ssd, 1 sas) OSDs. In each pool at least 1 ssd and 1 sas OSD is working without issue. I’ve ‘ceph pg repair <pg>’ but it doesn’t seem to make any changes. PG_AVAILABILITY Reduced data availability: 4 pgs inactive, 4 pgs incomplete pg 10.2e is incomplete, acting [59,67] pg 10.c3 is incomplete, acting [62,105] pg 10.f3 is incomplete, acting [62,59] pg 10.1d5 is incomplete, acting [87,106] Using `ceph pg <pg> query` I can see the OSD in each case of the ones which failed. Respectively they are: pg 10.2e participants: 59, 68, 77, 143 pg 10.c3 participants: 60, 62, 85, 102, 105, 106 pg 10.f3 participants: 59, 64, 75, 107 pg 10.1d5 participants: 64, 77, 87, 106 The OSDs which are now down/out and have been removed from the crush map and removed the auth are: 62, 64, 68 Of course I have lots of reports of slow OSDs now from OSDs worried about the inactive PGs. How do I properly kick these PGs to have them drop their usage of the OSDs which no longer exist? Thanks for you thoughts on this, peter Peter Eisch Senior Site Reliability Engineer T1.612.445.5135 virginpulse.com |virginpulse.com/global-challenge Australia | Bosnia and Herzegovina | Brazil | Canada | Singapore | Switzerland | United Kingdom | USA Confidentiality Notice: The information contained in this e-mail, including any attachment(s), is intended solely for use by the designated recipient(s). Unauthorized use, dissemination, distribution, or reproduction of this message by anyone other than the intended recipient(s), or a person designated as responsible for delivering such messages to the intended recipient, is strictly prohibited and may be unlawful. This e-mail may contain proprietary, confidential or privileged information. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Virgin Pulse, Inc. If you have received this message in error, or are not the named recipient(s), please immediately notify the sender and delete this e-mail message. v2.64

2 4

High memory ceph mgr 14.2.7
by hoannv46＠gmail.com 18 Mar '20

18 Mar '20

Hi all. My cluster in ceph version 14.2.6 Mgr process in top 3104786 ceph 20 0 20.2g 19.4g 18696 S 315.3 62.0 41:32.74 ceph-mgr All port ceph mgr open netstat -tulpn | grep ceph tcp 0 0 172.25.7.151:6800 0.0.0.0:* LISTEN 3104786/ceph-mgr tcp 0 0 172.25.7.151:6801 0.0.0.0:* LISTEN 3104786/ceph-mgr tcp 0 0 172.25.7.151:3300 0.0.0.0:* LISTEN 3407730/ceph-mon tcp 0 0 172.25.7.151:6789 0.0.0.0:* LISTEN 3407730/ceph-mon All connect to ceph mgr tcp 0 0 172.25.7.154:6800 172.25.5.23:42926 ESTABLISHED 250159/ceph-mgr tcp 0 0 172.25.7.154:6800 172.25.5.3:38732 ESTABLISHED 250159/ceph-mgr tcp 0 0 172.25.7.154:6801 172.25.5.2:42582 ESTABLISHED 250159/ceph-mgr tcp 0 0 172.25.7.154:6800 172.25.5.23:42930 ESTABLISHED 250159/ceph-mgr tcp 0 0 172.25.7.154:6800 172.25.6.2:39544 ESTABLISHED 250159/ceph-mgr tcp 0 0 172.25.7.154:6800 172.25.5.20:46618 ESTABLISHED 250159/ceph-mgr tcp 0 0 172.25.7.154:6801 172.25.5.4:35208 ESTABLISHED 250159/ceph-mgr netstat -plan | grep mgr | wc -l 2422 Connection connect to mgr is osd. How i can reduce mgr memory. Thanks.

6 8

Don't know how to use S3 notification
by jsobczak＠man.poznan.pl 17 Mar '20

17 Mar '20

Hello, I try to set up s3 http notifications but cant seem to figure it out. I tried with python boto3 like this: bucket_notification = s3.BucketNotification('wiaderko0') response = bucket_notification.put( NotificationConfiguration={ 'TopicConfigurations': [ { 'TopicArn': 'Kuba', 'Events': [ 's3:ObjectCreated:*' ] } ] } ) But get the following error: botocore.exceptions.ClientError: An error occurred (InvalidArgument) when calling the PutBucketNotificationConfiguration operation: Unknown I also tried using RGW REST api like with POST like this: http://172.16.97.1:7480/?Action=CreateTopic&Name=test&push-endpoint=http://… But got error like this: <?xml version="1.0" encoding="UTF-8"?> <Error> <Code>MethodNotAllowed</Code> <RequestId>tx0000000000000000d1e89-005e707912-11e4-default</RequestId> <HostId>11e4-default-default</HostId> </Error> Could any one offer some advice about it. I would appreciate some guide to setting it up as documentation is pretty unclear.

2 2

Point-in-Time Recovery
by Ml Ml 17 Mar '20

17 Mar '20

Hello List, when reading: https://docs.ceph.com/docs/master/rbd/rbd-mirroring/ it says: (...)Journal-based: This mode uses the RBD journaling image feature to ensure point-in-time, crash-consistent replication between clusters(...) Does this mean, that mean, that we have some kind of transaction logs where we will be able to "walk to" a specific time to restore a specific state? Like: https://blog.sleeplessbeastie.eu/2016/02/29/how-to-perform-postgresql-point… Thanks, Michael

2 1

For urgent help: OSD down under heavier workload
by huxiaoyu＠horebdata.cn 17 Mar '20

17 Mar '20

Dear Ceph folks, Can someoen with better ceph expertise help out? I am running a small ceph cluster of 3 nodes, Luminoous 12.2.12. On one node with weaker CPU, i have to reweight OSDs to keep them alive. If PGs on these OSDs exceeds some 40+, then these OSDs will continusouly restart (down) without success. I checked the CPU usage and it is actually not overloaded. What could be the root problem? best regards, Samuel *********************************************************************************************************************************************** lags = delete -42> 2020-03-17 17:13:53.374486 201a0afdb40 0 0x201d403edd0 6.d3 unexpected need for 6:cbf69ccd:::rbd_data.128dd6b8b4567.0000000000003e96:head have 217'1230 flags = delete tried to add 217'1230 flags = delete -41> 2020-03-17 17:13:53.374502 201a0afdb40 0 0x201d403edd0 6.d3 unexpected need for 6:cbf72b1e:::rbd_data.128f46b8b4567.00000000000005e4:head have 217'1103 flags = delete tried to add 217'1103 flags = delete -40> 2020-03-17 17:13:53.374518 201a0afdb40 0 0x201d403edd0 6.d3 unexpected need for 6:cbf91ecf:::rbd_data.8f3b66b8b4567.000000000000038e:head have 3093'11873 flags = delete tried to add 3093'11873 flags = delete -39> 2020-03-17 17:13:53.374533 201a0afdb40 0 0x201d403edd0 6.d3 unexpected need for 6:cbf9b36e:::rbd_data.2a0286b8b4567.0000000000000176:head have 1619'2253 flags = delete tried to add 1619'2253 flags = delete -38> 2020-03-17 17:13:53.374549 201a0afdb40 0 0x201d403edd0 6.d3 unexpected need for 6:cbf9dcdd:::rbd_data.128f46b8b4567.00000000000019b8:head have 217'1145 flags = delete tried to add 217'1145 flags = delete -37> 2020-03-17 17:13:53.374565 201a0afdb40 0 0x201d403edd0 6.d3 unexpected need for 6:cbfa7411:::rbd_data.73a8f6b8b4567.0000000000000231:head have 1934'8269 flags = delete tried to add 1934'8269 flags = delete -36> 2020-03-17 17:13:53.374581 201a0afdb40 0 0x201d403edd0 6.d3 unexpected need for 6:cbfc02e9:::rbd_data.128dd6b8b4567.00000000000002d0:head have 217'1111 flags = delete tried to add 217'1111 flags = delete -35> 2020-03-17 17:13:53.374597 201a0afdb40 0 0x201d403edd0 6.d3 unexpected need for 6:cbfccec4:::rbd_data.128f46b8b4567.0000000000000ea6:head have 217'1125 flags = delete tried to add 217'1125 flags = delete -34> 2020-03-17 17:13:53.374612 201a0afdb40 0 0x201d403edd0 6.d3 unexpected need for 6:cbfcf45a:::rbd_data.74a826b8b4567.000000000000004a:head have 1934'10159 flags = none tried to add 1934'10159 flags = none -33> 2020-03-17 17:13:53.374628 201a0afdb40 0 0x201d403edd0 6.d3 unexpected need for 6:cbfeb6f9:::rbd_data.8d7246b8b4567.00000000000042a5:head have 4114'12725 flags = none tried to add 4114'12725 flags = none -32> 2020-03-17 17:13:53.374643 201a0afdb40 0 0x201d403edd0 6.d3 unexpected need for 6:cbfeb9ce:::rbd_data.6e6fa6b8b4567.0000000000000093:head have 1934'10160 flags = none tried to add 1934'10160 flags = none -31> 2020-03-17 17:13:53.374659 201a0afdb40 0 0x201d403edd0 6.d3 unexpected need for 6:cbff1dd0:::rbd_data.128dd6b8b4567.0000000000004332:head have 217'1240 flags = delete tried to add 217'1240 flags = delete -30> 2020-03-17 17:13:53.374675 201a0afdb40 0 0x201d403edd0 6.d3 unexpected need for 6:cbffd7b5:::rbd_data.128dd6b8b4567.0000000000004330:head have 217'1239 flags = delete tried to add 217'1239 flags = delete -29> 2020-03-17 17:13:54.738477 201a12fdb40 0 0x201d4123390 4.318 unexpected need for 4:18c0b4c0:::rbd_data.9177446e87ccd.0000000000000b1d:head have 4711'802 flags = none tried to add 4711'802 flags = none -28> 2020-03-17 17:13:54.738516 201a12fdb40 0 0x201d4123390 4.318 unexpected need for 4:18c185d7:::rbd_data.589572ae8944a.0000000000000c3a:head have 1805'339 flags = delete tried to add 1805'339 flags = delete -27> 2020-03-17 17:13:54.738535 201a12fdb40 0 0x201d4123390 4.318 unexpected need for 4:18c1b3ab:::rbd_data.8922074b0dc51.0000000000003ea1:head have 3099'603 flags = delete tried to add 3099'603 flags = delete -26> 2020-03-17 17:13:54.738552 201a12fdb40 0 0x201d4123390 4.318 unexpected need for 4:18c28113:::rbd_data.9117a327b23c6.00000000000002bd:head have 3122'646 flags = none tried to add 3122'646 flags = none -25> 2020-03-17 17:13:54.738568 201a12fdb40 0 0x201d4123390 4.318 unexpected need for 4:18c2926a:::rbd_data.589572ae8944a.000000000000022d:head have 1805'337 flags = delete tried to add 1805'337 flags = delete -24> 2020-03-17 17:13:54.738584 201a12fdb40 0 0x201d4123390 4.318 unexpected need for 4:18c2bb8b:::rbd_data.5f15f625558ec.000000000000027f:head have 1913'383 flags = none tried to add 1913'383 flags = none -23> 2020-03-17 17:13:54.738600 201a12fdb40 0 0x201d4123390 4.318 unexpected need for 4:18c3ca0e:::rbd_data.8a738625558ec.0000000000005ca0:head have 3298'654 flags = none tried to add 3298'654 flags = none -22> 2020-03-17 17:13:54.738615 201a12fdb40 0 0x201d4123390 4.318 unexpected need for 4:18c4892e:::rbd_data.8ff1766334873.0000000000009ea0:head have 3092'523 flags = delete tried to add 3092'523 flags = delete -21> 2020-03-17 17:13:54.738631 201a12fdb40 0 0x201d4123390 4.318 unexpected need for 4:18c4ce18:::rbd_data.9177446e87ccd.0000000000000d58:head have 4750'803 flags = none tried to add 4750'803 flags = none -20> 2020-03-17 17:13:54.738647 201a12fdb40 0 0x201d4123390 4.318 unexpected need for 4:18c675ec:::rbd_data.8ff1766334873.00000000000018a2:head have 3092'468 flags = delete tried to add 3092'468 flags = delete -19> 2020-03-17 17:13:54.738662 201a12fdb40 0 0x201d4123390 4.318 unexpected need for 4:18cbb5d5:::rbd_data.9117a327b23c6.00000000000028a6:head have 3126'652 flags = none tried to add 3126'652 flags = none -18> 2020-03-17 17:13:54.738678 201a12fdb40 0 0x201d4123390 4.318 unexpected need for 4:18d2a04f:::rbd_data.110706b8b4567.0000000000000662:head have 1601'112 flags = none tried to add 1601'112 flags = none -17> 2020-03-17 17:13:54.738693 201a12fdb40 0 0x201d4123390 4.318 unexpected need for 4:18d424df:::rbd_data.9177446e87ccd.0000000000001ca7:head have 3122'649 flags = none tried to add 3122'649 flags = none -16> 2020-03-17 17:13:54.738708 201a12fdb40 0 0x201d4123390 4.318 unexpected need for 4:18d43cd1:::rbd_data.589572ae8944a.0000000000003a12:head have 1805'350 flags = delete tried to add 1805'350 flags = delete -15> 2020-03-17 17:13:54.738724 201a12fdb40 0 0x201d4123390 4.318 unexpected need for 4:18d91ef4:::rbd_data.589572ae8944a.0000000000004d34:head have 1805'354 flags = delete tried to add 1805'354 flags = delete -14> 2020-03-17 17:13:54.738739 201a12fdb40 0 0x201d4123390 4.318 unexpected need for 4:18e2953a:::rbd_data.5f15f625558ec.00000000000089d0:head have 1913'384 flags = none tried to add 1913'384 flags = none -13> 2020-03-17 17:13:54.738755 201a12fdb40 0 0x201d4123390 4.318 unexpected need for 4:18e514a7:::rbd_data.8f17c327b23c6.00000000000040a3:head have 2962'410 flags = delete tried to add 2962'410 flags = delete -12> 2020-03-17 17:13:54.738770 201a12fdb40 0 0x201d4123390 4.318 unexpected need for 4:18ea2e7b:::rbd_data.9177446e87ccd.000000000000095c:head have 4711'801 flags = none tried to add 4711'801 flags = none -11> 2020-03-17 17:13:54.738785 201a12fdb40 0 0x201d4123390 4.318 unexpected need for 4:18ed4244:::rbd_data.8ff1766334873.00000000000006a2:head have 3092'456 flags = delete tried to add 3092'456 flags = delete -10> 2020-03-17 17:13:54.738801 201a12fdb40 0 0x201d4123390 4.318 unexpected need for 4:18f0ccd9:::rbd_data.589572ae8944a.00000000000045e7:head have 1805'353 flags = delete tried to add 1805'353 flags = delete -9> 2020-03-17 17:13:54.738816 201a12fdb40 0 0x201d4123390 4.318 unexpected need for 4:18f69445:::rbd_data.589572ae8944a.0000000000000229:head have 1805'336 flags = delete tried to add 1805'336 flags = delete -8> 2020-03-17 17:13:54.738833 201a12fdb40 0 0x201d4123390 4.318 unexpected need for 4:18f88400:::rbd_data.589572ae8944a.0000000000004e31:head have 1805'355 flags = delete tried to add 1805'355 flags = delete -7> 2020-03-17 17:13:54.738848 201a12fdb40 0 0x201d4123390 4.318 unexpected need for 4:18f8e666:::rbd_data.9177446e87ccd.0000000000000387:head have 4416'800 flags = none tried to add 4416'800 flags = none -6> 2020-03-17 17:13:54.738864 201a12fdb40 0 0x201d4123390 4.318 unexpected need for 4:18fae8ee:::rbd_data.9177446e87ccd.0000000000005aa6:head have 3330'657 flags = none tried to add 3330'657 flags = none -5> 2020-03-17 17:13:54.738880 201a12fdb40 0 0x201d4123390 4.318 unexpected need for 4:18fdcb08:::rbd_data.110706b8b4567.0000000000001401:head have 1601'113 flags = none tried to add 1601'113 flags = none -4> 2020-03-17 17:14:19.091232 2000f81db40 0 -- 192.168.230.122:6812/171483 >> 192.168.230.11:0/1871093011 conn(0x20014cce840 :6812 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer -3> 2020-03-17 17:14:19.106328 2000ec51b40 0 -- 192.168.230.122:6812/171483 >> 192.168.230.12:0/4148513923 conn(0x20014109640 :6812 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer -2> 2020-03-17 17:14:19.155177 2000e3ddb40 0 -- 192.168.230.122:6812/171483 >> 192.168.230.13:0/768399090 conn(0x200141ac4d0 :6812 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer -1> 2020-03-17 17:14:36.138777 2000f81db40 0 -- 192.168.230.122:6812/171483 >> 192.168.230.202:0/3091162658 conn(0x200147d1f90 :6812 s=STATE_ACCEPTING_WAIT_CONNECT_MSG_AUTH pgs=0 cs=0 l=1).handle_connect_msg: challenging authorizer 0> 2020-03-17 17:15:06.927121 201a6afdb40 -1 *** Caught signal (Bus error) ** in thread 201a6afdb40 thread_name:tp_osd_tp ceph version 12.2.7 (3ec878d1e53e1aeb47a9f619c49d9e7c0aa384d5) luminous (stable) 1: (()+0x145882c) [0x2000245882c] 2: (()+0x19890) [0x2000d281890] 3: (BlueStore::ExtentMap::reshard(KeyValueDB*, std::shared_ptr<KeyValueDB::TransactionImpl>)+0x2df0) [0x2000229da60] 4: (BlueStore::_txc_write_nodes(BlueStore::TransContext*, std::shared_ptr<KeyValueDB::TransactionImpl>)+0x218) [0x2000229f888] 5: (BlueStore::queue_transactions(ObjectStore::Sequencer*, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x71c) [0x200022c7a6c] 6: (ObjectStore::queue_transaction(ObjectStore::Sequencer*, ObjectStore::Transaction&&, Context*, Context*, Context*, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x240) [0x20001c19ee0] 7: (PrimaryLogPG::queue_transaction(ObjectStore::Transaction&&, boost::intrusive_ptr<OpRequest>)+0x90) [0x20001e871b0] 8: (ReplicatedBackend::_do_pull_response(boost::intrusive_ptr<OpRequest>)+0x650) [0x2000203e5f0] 9: (ReplicatedBackend::_handle_message(boost::intrusive_ptr<OpRequest>)+0x3a4) [0x200020440c4] 10: (PGBackend::handle_message(boost::intrusive_ptr<OpRequest>)+0x94) [0x20001ecea74] 11: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x814) [0x20001de1384] 12: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x614) [0x20001b817d4] 13: (PGQueueable::RunVis::operator()(boost::intrusive_ptr<OpRequest> const&)+0xb8) [0x20001f98968] 14: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x1c24) [0x20001bb5fd4] 15: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0xab4) [0x200024d60a4] 16: (ShardedThreadPool::WorkThreadSharded::entry()+0x28) [0x200024da278] 17: (Thread::entry_wrapper()+0xec) [0x20002769b4c] 18: (Thread::_entry_func(void*)+0x20) [0x20002769ba0] 19: (()+0x80fc) [0x2000d2700fc] 20: (()+0x119854) [0x2000bed1854] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. huxiaoyu(a)horebdata.cn

1 0

OSD failing to restart with "no available blob id"
by Gilles Mocellin 17 Mar '20

17 Mar '20

Hi ! I'm stuck with "no available blob id" during the start of an OSD. It seems there's a workaround back-ported only in nautilus (Bug https://tracker.ceph.com/issues/38272) but I use mimic for now. Someone has an operational workaround ? Or should I recreate my OSD ? And what is the easiest way to do that ? Thanks, --- Gilles M

1 0

Forcibly move PGs from full to empty OSD
by Thomas Schneider 16 Mar '20

16 Mar '20

Hi, Ceph balancer is not working correctly; there's an open bug <https://tracker.ceph.com/issues/43752> report, too. Until this issue is not solved, I need a workaround because I get more and more warnings about "nearfull osd(s)". Therefore my question is: How can I forcibly move PGs from full OSD to empty OSD? THX

5 8

upmap balancer
by Frank R 16 Mar '20

16 Mar '20

Hi all, Does using the Upmap balancer require that all OSDs be the same size (per device class)? thx Frank

4 7