13 Nov
2020
13 Nov
'20
12:30 a.m.
I didn't know there was a replacement for the radosgw role! I saw in the
ceph-ansible project mention of a radosgw load balancer but since I use
haproxy, I didn't dig into that. Is that what you are referring to?
Otherwise, I cant seem to find any mention of cive being replaced.
For the issue below, I guess the dev was using a single threaded process
that was out of control. They have done it a few times now and kills all
four gateways. I asked them to stop and so far no repeats. For deletes,
they should be using the bucket item aging anyways.
-Brent
-----Original Message-----
From: Eugen Block <eblock(a)nde.ag>
Sent: Friday, October 23, 2020 7:00 AM
To: ceph-users(a)ceph.io
Subject: [ceph-users] Re: Rados Crashing
Hi,
I read that civetweb and radosgw have a locking issue in combination with
ssl [1], just a thought based on
failed to acquire lock on
obj_delete_at_hint.0000000079
Since Nautilus the default rgw frontend is beast, have you thought about
switching?
Regards,
Eugen
[1] https://tracker.ceph.com/issues/22951
Zitat von Brent Kennedy <bkennedy(a)cfl.rr.com>om>:
> We are performing file maintenance( deletes essentially ) and when the
> process gets to a certain point, all four rados gateways crash with
> the
> following:
>
>
>
>
>
> Log output:
>
> -5> 2020-10-20 06:09:53.996 7f15f1543700 2 req 7 0.000s s3:delete_obj
> verifying op params
>
> -4> 2020-10-20 06:09:53.996 7f15f1543700 2 req 7 0.000s
> s3:delete_obj pre-executing
>
> -3> 2020-10-20 06:09:53.996 7f15f1543700 2 req 7 0.000s
> s3:delete_obj executing
>
> -2> 2020-10-20 06:09:53.997 7f161758f700 10 monclient:
> get_auth_request con 0x55d2c02ff800 auth_method 0
>
> -1> 2020-10-20 06:09:54.009 7f1609d74700 5 process_single_shard():
failed to acquire lock on
obj_delete_at_hint.0000000079
>
> 0> 2020-10-20 06:09:54.035 7f15f1543700 -1 *** Caught signal
> (Segmentation fault) **
>
> in thread 7f15f1543700 thread_name:civetweb-worker
>
>
>
> ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf)
> nautilus
> (stable)
>
> 1: (()+0xf5d0) [0x7f161d3405d0]
>
> 2: (()+0x2bec80) [0x55d2bcd1fc80]
>
> 3: (std::string::assign(std::string const&)+0x2e) [0x55d2bcd2870e]
>
> 4: (rgw_bucket::operator=(rgw_bucket const&)+0x11) [0x55d2bce3e551]
>
> 5: (RGWObjManifest::obj_iterator::update_location()+0x184)
> [0x55d2bced7114]
>
> 6: (RGWObjManifest::obj_iterator::operator++()+0x263) [0x55d2bd092793]
>
> 7: (RGWRados::update_gc_chain(rgw_obj&, RGWObjManifest&,
> cls_rgw_obj_chain*)+0x51a) [0x55d2bd0939ea]
>
> 8: (RGWRados::Object::complete_atomic_modification()+0x83)
> [0x55d2bd093c63]
>
> 9: (RGWRados::Object::Delete::delete_obj()+0x74d) [0x55d2bd0a87ad]
>
> 10: (RGWDeleteObj::execute()+0x915) [0x55d2bd04b6d5]
>
> 11: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*,
> req_state*, bool)+0x915) [0x55d2bcdfbb35]
>
> 12: (process_request(RGWRados*, RGWREST*, RGWRequest*, std::string
> const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*,
> OpsLogSocket*, optional_yield, rgw::dmclock::Scheduler*, int*)+0x1cd8)
> [0x55d2bcdfdea8]
>
> 13: (RGWCivetWebFrontend::process(mg_connection*)+0x38e)
> [0x55d2bcd41a1e]
>
> 14: (()+0x36bace) [0x55d2bcdccace]
>
> 15: (()+0x36d76f) [0x55d2bcdce76f]
>
> 16: (()+0x36dc18) [0x55d2bcdcec18]
>
> 17: (()+0x7dd5) [0x7f161d338dd5]
>
> 18: (clone()+0x6d) [0x7f161c84302d]
>
> NOTE: a copy of the executable, or `objdump -rdS <executable>` is
> needed to interpret this.
>
>
>
> My guess is that we need to add more resources to the gateways? They
> have 2 CPUs and 12GB of memory running as virtual machines on centOS
> 7.6 . Any thoughts?
>
>
>
> -Brent
>
> _______________________________________________
> ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an
> email to ceph-users-leave(a)ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users(a)ceph.io To unsubscribe send an email
to ceph-users-leave(a)ceph.io