February 2021 - ceph-users

Re: RGW/Swift 404 error when listing/deleting a newly created empty bucket

by Mike Cave

I am bumping this email to hopefully get some more eyes on it. We are continuing to have this problem. Unfortunately the cluster is very lightly used currently until we go full production so we do not have the level of traffic that would generate a lot of statistics. We did update to 14.2.16 from 14.2.10 on Feb 1, 2021 and this seems to correlate with when the errors started popping up. Our current plan is to roll back the version to 14.2.10 again and rerun the test that causes the issue. I noted there was another email thread regarding latencies for a user who also updated to 14.2.16 recently and I'm not sure if this could be related or not to my issue. Any suggestions you may have are very welcomed. Cheers, -- Mike Cave On 2021-02-11, 8:37 AM, "Mike Cave" <mcave(a)uvic.ca> wrote: So, as the subject states I have an issue with buckets returning a 404 error when they are listed immediately after being created; as well the bucket fails to be deleted if you try to delete it immediately after creation. The behaviour is intermittent. If I leave the bucket in place for a few minutes, the bucket behaves normally. I’m thinking this is a metadata issue or something along those lines but I’m out of my depth now. To the best of our knowledge the cluster has not changed in any way since the same tests were run in December with no errors. We are running Ceph 14.2.16 on all parts of the cluster. I am using the python-swift client for the connection on a CentOS7 machine. Can replicate the results from the mons or an external client as well. I’m willing to share my test script as well if you would like to see how I’m generating the error. Here is a piece of the logs in case I missed something in the interpretation (log level at 20): 14:23:17.069 7faba00df700 1 ====== starting new request req=0x55fb7a138700 ===== 14:23:17.069 7faba00df700 2 req 148 0.000s initializing for trans_id = tx000000000000000000094-0060245cd5-2b8949-default 14:23:17.069 7faba00df700 10 rgw api priority: s3=8 s3website=7 14:23:17.069 7faba00df700 10 host=<NameRemoved> 14:23:17.069 7faba00df700 20 subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 14:23:17.069 7faba00df700 -1 res_query() failed 14:23:17.069 7faba00df700 20 final domain/bucket subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 s->info.domain= s->info.request_uri=/swift/v1/404test 14:23:17.069 7faba00df700 10 ver=v1 first=404test req= 14:23:17.069 7faba00df700 10 handler=28RGWHandler_REST_Bucket_SWIFT 14:23:17.069 7faba00df700 2 req 148 0.000s getting op 2 14:23:17.069 7faba00df700 10 req 148 0.000s swift:delete_bucket scheduling with dmclock client=3 cost=1 14:23:17.069 7faba00df700 10 op=30RGWDeleteBucket_ObjStore_SWIFT 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket verifying requester 14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket rgw::auth::swift::DefaultStrategy: trying rgw::auth::swift::TempURLEngine 14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket rgw::auth::swift::TempURLEngine denied with reason=-13 14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket rgw::auth::swift::DefaultStrategy: trying rgw::auth::swift::SignedTokenEngine 14:23:17.069 7faba00df700 10 req 148 0.000s swift:delete_bucket swift_user=xmcc:swift 14:23:17.069 7faba00df700 20 build_token token=0a000000786d63633a73776966748960ea4653df708a55ae2560e58acf01 14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket rgw::auth::swift::SignedTokenEngine granted access 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket normalizing buckets and tenants 14:23:17.069 7faba00df700 10 s->object=<NULL> s->bucket=404test 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket init permissions 14:23:17.069 7faba00df700 20 get_system_obj_state: rctx=0x55fb7a137770 obj=default.rgw.meta:root:404test state=0x55fb7a060ac0 s->prefetch_data=0 14:23:17.069 7faba00df700 10 cache get: name=default.rgw.meta+root+404test : hit (negative entry) 14:23:17.069 7faba00df700 20 get_system_obj_state: rctx=0x55fb7a137130 obj=default.rgw.meta:users.uid:xmcc state=0x55fb7a060f40 s->prefetch_data=0 14:23:17.069 7faba00df700 10 cache get: name=default.rgw.meta+users.uid+xmcc : hit (requested=0x6, cached=0x17) 14:23:17.069 7faba00df700 20 get_system_obj_state: s->obj_tag was set empty 14:23:17.069 7faba00df700 20 Read xattr: user.rgw.idtag 14:23:17.069 7faba00df700 20 get_system_obj_state: rctx=0x55fb7a137130 obj=default.rgw.meta:users.uid:xmcc state=0x55fb7a060f40 s->prefetch_data=0 14:23:17.069 7faba00df700 10 cache get: name=default.rgw.meta+users.uid+xmcc : hit (requested=0x6, cached=0x17) 14:23:17.069 7faba00df700 20 get_system_obj_state: s->obj_tag was set empty 14:23:17.069 7faba00df700 20 Read xattr: user.rgw.idtag 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket recalculating target 14:23:17.069 7faba00df700 10 Starting retarget 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket reading permissions 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket init op 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket verifying op mask 14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket required_mask= 4 user.op_mask=7 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket verifying op permissions 14:23:17.069 7faba00df700 20 req 148 0.000s swift:delete_bucket -- Getting permissions begin with perm_mask=50 14:23:17.069 7faba00df700 5 req 148 0.000s swift:delete_bucket Searching permissions for identity=rgw::auth::ThirdPartyAccountApplier() -> rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=xmcc, acct_name=xmcc, subuser=swift, perm_mask=15, is_admin=0) mask=50 14:23:17.069 7faba00df700 5 Searching permissions for uid=xmcc 14:23:17.069 7faba00df700 5 Found permission: 15 14:23:17.069 7faba00df700 5 Searching permissions for group=1 mask=50 14:23:17.069 7faba00df700 5 Permissions for group not found 14:23:17.069 7faba00df700 5 Searching permissions for group=2 mask=50 14:23:17.069 7faba00df700 5 Permissions for group not found 14:23:17.069 7faba00df700 5 req 148 0.000s swift:delete_bucket -- Getting permissions done for identity=rgw::auth::ThirdPartyAccountApplier() -> rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=xmcc, acct_name=xmcc, subuser=swift, perm_mask=15, is_admin=0), owner=xmcc, perm=2 14:23:17.069 7faba00df700 10 req 148 0.000s swift:delete_bucket identity=rgw::auth::ThirdPartyAccountApplier() -> rgw::auth::SysReqApplier -> rgw::auth::LocalApplier(acct_user=xmcc, acct_name=xmcc, subuser=swift, perm_mask=15, is_admin=0) requested perm (type)=2, policy perm=2, user_perm_mask=2, acl perm=2 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket verifying op params 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket pre-executing 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket executing 14:23:17.069 7faba00df700 0 req 148 0.000s swift:delete_bucket ERROR: bucket 404test not found 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket completing 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket op status=-2002 14:23:17.069 7faba00df700 2 req 148 0.000s swift:delete_bucket http status=404 14:23:17.069 7faba00df700 1 ====== req done req=0x55fb7a138700 op status=-2002 http_status=404 latency=0s ====== -- Mike Cave I acknowledge and respect the Lekwungen-speaking Peoples on whose traditional territories the university stands and the Songhees, Esquimalt and WSANEC peoples whose historical relationships with the land continue to this day. _______________________________________________ ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email to ceph-users-leave(a)ceph.io

3 years, 2 months

3
4
0 0

OSDs cannot join, MON leader at 100%

by Frank Schilder

Dear cephers, I was doing some maintenance yesterday involving shutdown-power up cycles of ceph servers. With the last server I run into a problem. The server runs an MDS and a couple of OSDs. After reboot, the MDS joined the MDS cluster without problems, but the OSDs didn't come up. This was 1 out of 12 servers and I had no such problems with the other 11. I also observed that "ceph status" was responding very slow. Upon further inspection, I found out that 2 of my 3 MONs (the leader and a peon) were running at 100% CPU. Client I/O was continuing, probably because the last cluster map remained valid. On our node performance monitoring I could see that the 2 busy MONs were showing extraordinary network activity. This state lasted for over one hour. After the MONs settled down, the OSDs finally joined as well and everything went back to normal. The other instance I have seen similar behaviour was, when I restarted a MON on an empty disk and the re-sync was extremely slow due to a too large value for mon_sync_max_payload_size. This time, I'm pretty sure it was MON-client communication; see below. Are there any settings similar to mon_sync_max_payload_size that could influence responsiveness of MONs in a similar way? Why do I suspect it is MON-client communication? In our monitoring, I do not see the huge amount of packages sent by the MONs arriving at any other ceph daemon. They seem to be distributed over client nodes, but since we have a large count of client nodes (>550) this is covered by the background network traffic. A second clue is that I have had such extended lock-ups before and, whenever I checked, I only observed these in case the leader had a large share of client sessions. For example, yesterday the client session count per MON was: ceph-01: 1339 (leader) ceph-02: 189 (peon) ceph-03: 839 (peon) I usually restart the leader when such a critical distribution occurs. As long as the leader has the fewest client sessions, I never observe this problem. Ceph version is 13.2.10 (564bdc4ae87418a232fc901524470e1a0f76d641) mimic (stable). Thanks for any clues! Best regards, ================= Frank Schilder AIT Risø Campus Bygning 109, rum S14

3 years, 2 months

2
5
0 0

after update to 14.2.16 osd daemons begin to crash

by Boris Behrens

Hi, currently we experience osd daemon crashes and I can't pin the issue. I hope someone can help me with it. * We operate multiple cluster (440 SSD - 1PB, 36 SSD - 126TB, 40SSD 100TB, 84HDD - 680TB) * All clusters were updated around the same time (2021-02-03) * We restarted ALL ceph daemons (systemctl restart ceph.target) on 2021-02-11 after we added OOMScoreAdjust=-900 the all service files. now in our main cluster (440SSD with 1PB) the OSD daemons begin to crash: # ceph crash ls ID ENTITY NEW 2020-03-06_17:37:54.031675Z_0bbbb807-ff2f-46df-9508-58d319b89bd6 osd.397 2020-05-28_12:23:27.677741Z_061f2449-9a36-4747-a2f8-624e72cd1ad0 osd.410 2021-02-05_07:03:35.943384Z_dffab245-4788-4de2-a677-76b735d5fc01 osd.403 2021-02-15_15:41:27.934194Z_97b57f8f-58f2-4390-9d3e-993874e0e000 osd.395 2021-02-15_18:01:19.774879Z_18160e65-4659-451f-8aae-def2984f1f29 osd.178 2021-02-17_04:51:05.101052Z_9f04c6e8-d0c7-442c-9a38-33d5164d2a83 osd.384 osd.384 and osd.395 are on the same node, which had some memory issues we fixed 2021-02-16_12:00:00 osd.384 was marked as out for >24h when the daemon crashed, and there no more misplaced objects in the cluster. Here is the latest crash dump --- begin dump of recent events --- -9999> 2021-02-17 03:31:31.305 7fcf7e136700 1 do_command 'perf dump' 'result is 30067 bytes -9998> 2021-02-17 03:31:31.626 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2882789376 unmapped: 956792832 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -9997> 2021-02-17 03:31:32.634 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2882789376 unmapped: 956792832 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -9996> 2021-02-17 03:31:33.639 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2882789376 unmapped: 956792832 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -9995> 2021-02-17 03:31:34.647 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2882789376 unmapped: 956792832 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -9994> 2021-02-17 03:31:35.651 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2882789376 unmapped: 956792832 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -9993> 2021-02-17 03:31:36.654 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2882789376 unmapped: 956792832 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -9992> 2021-02-17 03:31:37.657 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2882789376 unmapped: 956792832 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -9991> 2021-02-17 03:31:38.676 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2882789376 unmapped: 956792832 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -9990> 2021-02-17 03:31:39.680 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2882789376 unmapped: 956792832 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -9989> 2021-02-17 03:31:40.684 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2882789376 unmapped: 956792832 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -9988> 2021-02-17 03:31:41.193 7fcf7e136700 1 do_command 'perf dump' ' -9987> 2021-02-17 03:31:41.193 7fcf7e136700 1 do_command 'perf dump' 'result is 30067 bytes <snip> -31> 2021-02-17 05:50:41.158 7fcf7e136700 1 do_command 'perf dump' ' -30> 2021-02-17 05:50:41.159 7fcf7e136700 1 do_command 'perf dump' 'result is 30070 bytes -29> 2021-02-17 05:50:41.804 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851831808 unmapped: 987750400 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -28> 2021-02-17 05:50:42.813 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851831808 unmapped: 987750400 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -27> 2021-02-17 05:50:43.820 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -26> 2021-02-17 05:50:44.825 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -25> 2021-02-17 05:50:45.831 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -24> 2021-02-17 05:50:46.837 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -23> 2021-02-17 05:50:47.840 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -22> 2021-02-17 05:50:48.843 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -21> 2021-02-17 05:50:49.847 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -20> 2021-02-17 05:50:50.853 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -19> 2021-02-17 05:50:51.524 7fcf7e136700 1 do_command 'perf dump' ' -18> 2021-02-17 05:50:51.525 7fcf7e136700 1 do_command 'perf dump' 'result is 30070 bytes -17> 2021-02-17 05:50:51.859 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -16> 2021-02-17 05:50:52.862 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -15> 2021-02-17 05:50:53.871 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -14> 2021-02-17 05:50:54.875 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -13> 2021-02-17 05:50:55.886 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -12> 2021-02-17 05:50:56.891 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -11> 2021-02-17 05:50:57.905 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -10> 2021-02-17 05:50:58.911 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -9> 2021-02-17 05:50:59.917 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -8> 2021-02-17 05:51:00.929 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -7> 2021-02-17 05:51:01.566 7fcf7e136700 1 do_command 'perf dump' ' -6> 2021-02-17 05:51:01.567 7fcf7e136700 1 do_command 'perf dump' 'result is 30070 bytes -5> 2021-02-17 05:51:01.935 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -4> 2021-02-17 05:51:02.943 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851405824 unmapped: 988176384 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -3> 2021-02-17 05:51:03.949 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851102720 unmapped: 988479488 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -2> 2021-02-17 05:51:04.967 7fcf73be6700 5 prioritycache tune_memory target: 4294967296 mapped: 2851102720 unmapped: 988479488 heap: 3839582208 old mem: 2845415832 new mem: 2845415832 -1> 2021-02-17 05:51:05.091 7fcf743e7700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.16/rpm/el7/BUILD/ceph-14.2.16/src/os/bluestore/fastbmap_allocator_impl.h: In function 'uint64_t AllocatorLevel02<T>::claim_free_to_right(uint64_t) [with L1 = AllocatorLevel01Loose; uint64_t = long unsigned int]' thread 7fcf743e7700 time 2021-02-17 05:51:04.998475 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.16/rpm/el7/BUILD/ceph-14.2.16/src/os/bluestore/fastbmap_allocator_impl.h: 572: FAILED ceph_assert(available >= allocated) ceph version 14.2.16 (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus (stable) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x561c84cc2c7d] 2: (()+0x4d8e45) [0x561c84cc2e45] 3: (HybridAllocator::_add_to_tree(unsigned long, unsigned long)+0x49e) [0x561c853167de] 4: (AvlAllocator::_release(interval_set<unsigned long, std::map<unsigned long, unsigned long, std::less<unsigned long>, std::allocator<std::pair<unsigned long const, unsigned long> > > > const&)+0x60) [0x561c85310b20] 5: (HybridAllocator::release(interval_set<unsigned long, std::map<unsigned long, unsigned long, std::less<unsigned long>, std::allocator<std::pair<unsigned long const, unsigned long> > > > const&)+0x3a) [0x561c853143ca] 6: (BlueStore::_txc_release_alloc(BlueStore::TransContext*)+0x5f) [0x561c851ee83f] 7: (BlueStore::_txc_finish(BlueStore::TransContext*)+0x1be) [0x561c8522f4ae] 8: (BlueStore::_txc_state_proc(BlueStore::TransContext*)+0xaa) [0x561c8522fe9a] 9: (BlueStore::_kv_finalize_thread()+0x604) [0x561c85232ed4] 10: (BlueStore::KVFinalizeThread::entry()+0xd) [0x561c852625ed] 11: (()+0x7ea5) [0x7fcf840a2ea5] 12: (clone()+0x6d) [0x7fcf82f6596d] 0> 2021-02-17 05:51:05.145 7fcf743e7700 -1 *** Caught signal (Aborted) ** in thread 7fcf743e7700 thread_name:bstore_kv_final ceph version 14.2.16 (762032d6f509d5e7ee7dc008d80fe9c87086603c) nautilus (stable) 1: (()+0xf630) [0x7fcf840aa630] 2: (gsignal()+0x37) [0x7fcf82e9d387] 3: (abort()+0x148) [0x7fcf82e9ea78] 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x199) [0x561c84cc2ccc] 5: (()+0x4d8e45) [0x561c84cc2e45] 6: (HybridAllocator::_add_to_tree(unsigned long, unsigned long)+0x49e) [0x561c853167de] 7: (AvlAllocator::_release(interval_set<unsigned long, std::map<unsigned long, unsigned long, std::less<unsigned long>, std::allocator<std::pair<unsigned long const, unsigned long> > > > const&)+0x60) [0x561c85310b20] 8: (HybridAllocator::release(interval_set<unsigned long, std::map<unsigned long, unsigned long, std::less<unsigned long>, std::allocator<std::pair<unsigned long const, unsigned long> > > > const&)+0x3a) [0x561c853143ca] 9: (BlueStore::_txc_release_alloc(BlueStore::TransContext*)+0x5f) [0x561c851ee83f] 10: (BlueStore::_txc_finish(BlueStore::TransContext*)+0x1be) [0x561c8522f4ae] 11: (BlueStore::_txc_state_proc(BlueStore::TransContext*)+0xaa) [0x561c8522fe9a] 12: (BlueStore::_kv_finalize_thread()+0x604) [0x561c85232ed4] 13: (BlueStore::KVFinalizeThread::entry()+0xd) [0x561c852625ed] 14: (()+0x7ea5) [0x7fcf840a2ea5] 15: (clone()+0x6d) [0x7fcf82f6596d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. -- Die Selbsthilfegruppe "UTF-8-Probleme" trifft sich diesmal abweichend im groÃƒ¼en Saal.

3 years, 2 months

2
3
0 0

Ceph @ DevConf.CZ

by Mike Perez

Hi everyone, Ceph will be present at DevConf.CZ, February 18-20 in a joint booth with the Rook Community! https://www.devconf.cz If you're interested in more information about being present at the booth to provide expertise/content/presentations to our audience, please let me know privately. -- Mike Perez

3 years, 2 months

1
0
0 0

best use of NVMe drives

by Magnus HAGDORN

Hi there, we are in the process of growing our Nautilus ceph cluster. Currently, we have 6 nodes, 3 nodes with 2×5.5TB, 6x11TB disks and 8x186GB SSD and 3 nodes with 6×5.5TB and 6×7.5TB disks. All with dual link 10GE NICs. The SSDs are used for the CephFS metadata pool, the hard drives are used for the CephFS data pool. All OSD journals are kept on the drives themselves. Replication level is 3 for both data and metadata pools. The new servers have 12x12TB disks and 1 1.5TB NVMe drive. We expect to get another 3 similar nodes in the near future. My question is what is the most sensible thing to do with the NVMe drives. I would like to increase the replication level of the metadata pool. So my idea was to split the NVMes into say 4 partitions and add them to the metadata pool. Given the size of the drives and the metadata pool usage (~35GB) that seems overkill. Would it make sense to partition the drives further and stick the OSD journals on the NVMEs? Regards magnus The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

3 years, 2 months

2
1
0 0

SUSE POC - Dead in the water

by Schweiss, Chip

For the past several months I had been building a sizable Ceph cluster that will be up to 10PB with between 20 and 40 OSD servers this year. A few weeks ago I was informed that SUSE is shutting down SES and will no longer be selling it. We haven't licensed our proof of concept cluster that is currently at 14 OSD nodes, but it looks like SUSE is not going to be the answer here. I'm seeking recommendations for consulting help on this project since SUSE has let me down. I have Ceph installed and operating, however, I've been struggling with getting the pool configured properly for CephFS and getting very poor performance. The OSD servers have TLC NVMe for DB, and Optane NVMe for WAL, so I should be seeing decent performance with the current cluster. I'm not opposed to completely switching OS distributions. Ceph on SUSE was our first SUSE installation. Almost everything else we run is on CentOS, but that may change thanks to IBM cannibalizing CentOS. Please reach out to me if you can recommend someone to sell us consulting hours and/or a support contract. -Chip Schweiss chip.schweiss(a)wustl.edu Washington University School of Medicine

3 years, 2 months

4
6
0 0

share haproxy config for radosgw

by Marc

I was wondering if someone could post a config for haproxy. Is there something specific to configure? Like binding clients to a specific backend server, client timeouts, security specific to rgw etc.

3 years, 2 months

8
10
0 0

Consequences of setting bluestore_fsck_quick_fix_on_mount to false?

by Matthew Vernon

Hi, Looking at the Octopus upgrade instructions, I see "the first time each OSD starts, it will do a format conversion to improve the accounting for “omap” data. This may take a few minutes to as much as a few hours (for an HDD with lots of omap data)." and that I can disable this by setting bluestore_fsck_quick_fix_on_mount to false. A couple of questions about this: i) what are the consequences of turning off this "quick fix"? Is it possible to have it run in the background or similar? ii) is there any way to narrow down the time estimate? Our production cluster has 3060 OSDs on hdd (with block.db on NVME), and obviously 3000 lots of "a few hours" is an awful lot of time... I'll be doing some testing on our test cluster (by putting 10M objects into an S3 bucket before trying the upgrade), but it'd be useful to have some idea of how this is likely to work at scale... Thanks, Matthew -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.

3 years, 2 months

2
2
0 0

Upgrading Ceph luminous to mimic on debian-buster

by Jean-Marc FONTANA

Hello everyone, We just installed a Ceph cluster version luminous (12.2.11) on servers working with Debian buster (10.8) using ceph-deploy and we are trying to upgrade it to mimic but can't find a way to do it. We tried ceph-deploy install --release mimic mon1 mon2 mon3 (after having modified /etc/apt/sources.list.d/ceph.list) but this does nothing because the packets are said to be up to date. Could someone help us, please ? Best regards

3 years, 2 months

2
1
0 0

osds processes shutdown during outage

by Marcel Kuiper

Hi, (sorry if this gets posted twice. I forgot a subject in the first mail) We expereinced an outage this morning on a jewel cluster with 1559 osds. It appeared that a switch uplink in a rack misbehaved and once shutting that interface ceph health restored quickly. I have some questions though on osd behaviour that I hope someone can answer 1 - In a lot of osd logs I saw that neighbours reported the osd down (while the process was still running and obviously logging). Then after a while the logs shows * Got signal Interrupt * prepare_to_stop starting shutdown and the osd process stops Why does the osd proces stop? Is it instructed to do so by the monitor because neighbours reported it down and ceph wants to avoid flapping? 2 - The osds reported a lot of * heartbeat_check: no reply from #ip:#port When I telnet to the ip and port I get a connection just fine. Is there a way to run a heartbeat_check from the commandline so that we can try capture the traffic to determine why it fails Thanks Marcel

3 years, 2 months

1
0
0 0

2024

2023

2022

2021

2020

2019

ceph-users February 2021