- ceph-users - lists.ceph.io

17.2.6 Dashboard/RGW Signature Mismatch
by Chris Palmer 14 Apr '23

14 Apr '23

Hi I have 3 Ceph clusters, all configured similarly, which have been happy for some months on 17.2.5: 1. A test cluster 2. A small production cluster 3. A larger production cluster All are debian 11 built from packages - no cephadm. I upgraded (1) to 17.2.6 without any problems at all. In particular the Object Gateway sections of the dashboard work as usual. I then upgraded (2). Nothing seemed amiss, and everything seems to work except... when I try to access the Object Gateway sections of the dashboard I always get: *The Object Gateway Service is not configured* Error connecting to Object Gateway: RGW REST API failed request with status code 403 (b'{"Code":"SignatureDoesNotMatch","RequestId":"tx0000022ba920e82ac4a9c-0064381' b'934-10e73385-default","HostId":"10e73385-default-default"}') (Just the RequestId changes each time). Before the upgrade it worked just fine. Other info: * RGW requests using awscli and rclone all work with normal RGW accounts. It just seems to be the dashboard that's died. * Just the one zonegroup, no multisite/replication * "radosgw-admin user info --uid=rgwadmin" gives the correct output with the right access_key & secret_key. The other fields are as in (1). * "ceph dashboard get-rgw-api-access-key/get-rgw-api-secret-key" both give the right values. The rgw logs from (2) which fails show: 2023-04-13T16:36:28.720+0100 7fcc7966a700 1 ====== starting new request req=0x7fcd88c10720 ===== 2023-04-13T16:36:28.720+0100 7fcc80e79700 1 req 8090309398268968541 0.000000000s op->ERRORHANDLER: err_no=-2027 new_err_no=-2027 2023-04-13T16:36:28.724+0100 7fcc80e79700 1 ====== req done req=0x7fcd88c10720 op status=0 http_status=403 latency=0.003999980s ====== 2023-04-13T16:36:28.724+0100 7fcc80e79700 1 beast: 0x7fcd88c10720: 192.168.xx.xx - - [13/Apr/2023:16:36:28.720 +0100] "GET /admin/metadata/user?myself HTTP/1.1" 403 134 - "python-requests/2.25.1" - latency=0.003999980s (Note this does not have rgwadmin as the user, and is always the same URL) Whereas the rgw logs from (1) which works show things like: 2023-04-13T15:44:19.396+0000 7f8478da1700 1 ====== starting new request req=0x7f86284f5720 ===== 2023-04-13T15:44:19.412+0000 7f8478da1700 1 ====== req done req=0x7f86284f5720 op status=0 http_status=200 latency=0.016000060s ====== 2023-04-13T15:44:19.412+0000 7f8478da1700 1 beast: 0x7f86284f5720: 10.xx.xx.xx - rgwadmin [13/Apr/2023:15:44:19.396 +0000] "GET /admin/realm?list HTTP/1.1" 200 31 - "python-requests/2.25.1" - latency=0.016000060s (Note this has rgwadmin as the user, and various URLs) The only thing I can see in the release notes that looks even vaguely related is https://github.com/ceph/ceph/pull/47547, but it doesn't seem likely. I am really stumped on this, with no idea what has gone wrong on (2), and what the difference is between (1) and (2). I'm not going to touch (3) until I have resolved this. Grateful for any help... And thanks for all the good work. Regards, Chris

3 4

Restrict user to an RBD image in a pool
by Work Ceph 14 Apr '23

14 Apr '23

Hello guys! Is it possible to restrict user access to a single image in an RBD pool? I know that I can use namespaces, so users can only see images with a given namespaces. However, these users will still be able to create new RBD images. Is it possible to somehow block users from creating RBD images and only work with the already existing ones?

2 1

Cephadm only scheduling, not orchestrating daemons
by Thomas Widhalm 14 Apr '23

14 Apr '23

Hi, As you might know, I have a problem with MDS not starting. During the investigation with your help I found another issue that might be related. I can plan to restart, redeploy, reconfigure services via cephadm or dashboard just as I want but services won't react. I only see the action to be scheduled but that's all. 2023-04-13T17:11:15.690698+0000 mgr.ceph04.qaexpv (mgr.74906907) 37184 : cephadm [INF] Schedule redeploy daemon mds.mds01.ceph05.pqxmvt 2023-04-13T17:11:20.746743+0000 mgr.ceph04.qaexpv (mgr.74906907) 37190 : cephadm [INF] Schedule redeploy daemon mds.mds01.ceph06.rrxmks 2023-04-13T17:11:24.971226+0000 mgr.ceph04.qaexpv (mgr.74906907) 37195 : cephadm [INF] Schedule redeploy daemon mds.mds01.ceph07.omdisd It's the same for other daemons/services. I changed placement rules, scheduled changes, failed mgr, even rebooted hosts. I even was desperate enough to delete files for services from hosts before rebooting hoping I could trigger a manual redeploy. All I see are the same MDS stuck in "error" state. I removed them via "ceph orch rm" but they are still there. When I reissue the command it fails saying that the service isn't there. "ceph orch ps" still lists them. mds.mds01.ceph03.xqwdjy ceph03 error 2d ago 2M - - <unknown> <unknown> <unknown> mds.mds01.ceph04.hcmvae ceph04 error 2d ago 2d - - <unknown> <unknown> <unknown> mds.mds01.ceph05.pqxmvt ceph05 error 2d ago 10M - - <unknown> <unknown> <unknown> mds.mds01.ceph06.rrxmks ceph06 error 2d ago 10w - - <unknown> <unknown> <unknown> mds.mds01.ceph07.omdisd ceph07 error 2d ago 3M - - <unknown> <unknown> <unknown> Any idea how I can get rid of them? Or redeploy them? Additionally I'm just in the middle of an upgrade. { "target_image": "quay.io/ceph/ceph@sha256:1161e35e4e02cf377c93b913ce78773f8413f5a8d7c5eaee4b4773a4f9dd6635", "in_progress": true, "which": "Upgrading all daemon types on all hosts", "services_complete": [ "crash", "mgr", "mon", "osd" ], "progress": "18/40 daemons upgraded", "message": "Upgrade paused", "is_paused": true } I paused it on purpose to allow for manipulation of daemons. Cheers, Thomas

2 1

ceph 17.2.6 and iam roles (pr#48030)
by Christopher Durham 13 Apr '23

13 Apr '23

Hi, I see that this PR: https://github.com/ceph/ceph/pull/48030 made it into ceph 17.2.6, as per the change log at: https://docs.ceph.com/en/latest/releases/quincy/ That's great. But my scenario is as follows: I have two clusters set up as multisite. Because of the lack of replication for IAM roles, we have set things up so that roles on the primary 'manually' get replicated to the secondary site via a python script. Thus, if I create a role on the primary, add/delete users or buckets from said role, the role, including the AssumeRolePolicyDocument and policies, gets pushed to the replicated site. This has served us well for three years. With the advent of this fix, what should I do before I upgrade to 17.2.6 (currently on 17.2.5, rocky 8) I know that in my situation, roles of the same name have different RoleIDs on the two sites. What should I do before I upgrade? Possibilities that *could* happen if i dont rectify things as we upgrade: 1. The different RoleIDs lead to two roles of the same name on the replicated site, perhaps with the system unable to address/look at/modify either 2. Roles just don't get repiicated to the second site or other similar situations, all of which I want to avoid. Perhaps the safest thing to do is to remove all roles on the secondary site, upgrade, and then force a replication of roles (How would I *force* that for iAM roles if it is the correct answer?) Here is the original bug report: https://tracker.ceph.com/issues/57364 Thanks! -Chris

2 3

Live migrate RBD image with a client using it
by Work Ceph 13 Apr '23

13 Apr '23

Hello guys, We have been reading the docs, and trying to reproduce that process in our Ceph cluster. However, we always receive the following message: ``` librbd::Migration: prepare: image has watchers - not migrating rbd: preparing migration failed: (16) Device or resource busy ``` We tested with both RDB blocks mounted with kRBD, and with librbd via KVM/qEMU system. Both cases result in the same result. For KRBD, we understood that it is not supported right now, but for librbd it seems that it should be supported somehow. How do you guys handle those situations? We have the following use cases that might need an image to migrate between pools while the client is still consuming it. - RBD images that are consumed via the iSCSI gateways - RBD images mounted (rbd map) in hosts - RBD images used by KVM/Libvirt Does Ceph support a live migration of images between pools while the clients/consumers are still using those volumes? [1] https://docs.ceph.com/en/quincy/rbd/rbd-live-migration/

5 5

RGW is slowly after the ops increase
by Louis Koo 13 Apr '23

13 Apr '23

check the ops for rgw: [root@node06 ceph]# ceph daemon /var/run/ceph/ceph-client.rgw.os.dsglczutvqsgowpz.a.13.93908447458760.asok objecter_requests| jq ".ops" | jq 'length' 8 list subdir with s5cmd: [root@node01 deeproute]# time ./s5cmd --endpoint-url=http://10.x.x.x:80 ls s3://mlp-data-warehouse/ads_prediction/ DIR prediction_scenes/ DIR test_pai/ real 0m1.125s user 0m0.007s sys 0m0.016s after the ops increase: [root@node06 ceph]# ceph daemon /var/run/ceph/ceph-client.rgw.os.dsglczutvqsgowpz.a.13.93908447458760.asok objecter_requests| jq ".ops" | jq 'length' 264 list subdir with s5cmd: [root@node01 deeproute]# time ./s5cmd --endpoint-url=http://10.x.x.x:80 ls s3://mlp-data-warehouse/ads_prediction/ DIR prediction_scenes/ DIR test_pai/ real 0m8.822s user 0m0.004s sys 0m0.019s and if the ops increase to more 2000, it needs more than 100s to list the subdir, why?

1 0

Osd crash, looks like something related to PG recovery.
by Louis Koo 13 Apr '23

13 Apr '23

{ "archived": "2023-04-13 02:23:50.948191", "backtrace": [ "/lib64/libpthread.so.0(+0x12ce0) [0x7f2ee8198ce0]", "pthread_kill()", "(ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d const*, char const*, std::chrono::time_point<ceph::coarse_mono_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >)+0x48c) [0x563506e9934c]", "(ceph::HeartbeatMap::reset_timeout(ceph::heartbeat_handle_d*, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >)+0x23e) [0x563506e9973e]", "(PrimaryLogPG::scan_range(int, int, BackfillInterval*, ThreadPool::TPHandle&)+0x15a) [0x56350699a8da]", "(PrimaryLogPG::do_scan(boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x914) [0x56350699bd34]", "(PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x776) [0x56350699c826]", "(OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x309) [0x563506823fc9]", "(ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x68) [0x563506a82e78]", "(OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xc28) [0x5635068414c8]", "(ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5c4) [0x563506ebe2a4]", "(ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x563506ec1184]", "/lib64/libpthread.so.0(+0x81ca) [0x7f2ee818e1ca]", "clone()" ], "ceph_version": "16.2.10", "crash_id": "2023-04-12T16:53:45.988696Z_7e73aedd-3518-41f4-ae48-e5dbfe5750ec", "entity_name": "osd.74", "os_id": "centos", "os_name": "CentOS Stream", "os_version": "8", "os_version_id": "8", "process_name": "ceph-osd", "stack_sig": "1a2700ce6c68288739eb14ca1b2b5f49449c59a5baafbd1e71df3a4316e3bffe", "timestamp": "2023-04-12T16:53:45.988696Z", "utsname_hostname": "node03", "utsname_machine": "x86_64", "utsname_release": "3.10.0-1160.45.1.el7.x86_64", "utsname_sysname": "Linux", "utsname_version": "#1 SMP Wed Oct 13 17:20:51 UTC 2021" } releated issuse under rook: https://github.com/rook/rook/issues/11565

1 0

radosgw crash
by Louis Koo 13 Apr '23

13 Apr '23

{ "archived": "2023-04-09 01:22:40.755345", "backtrace": [ "/lib64/libpthread.so.0(+0x12ce0) [0x7f06dc1edce0]", "(boost::asio::detail::reactive_socket_service_base::start_op(boost::asio::detail::reactive_socket_service_base::base_implementation_type&, int, boost::asio::detail::reactor_op*, bool, bool, bool)+0x126) [0x7f06e6cf84d6]", "(void boost::asio::detail::reactive_socket_service_base::async_receive<boost::asio::mutable_buffer, boost::beast::detail::dynamic_read_ops::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::io_context::executor_type>, boost::beast::flat_static_buffer<65536ul>, boost::beast::http::detail::read_header_condition<true>, spawn::detail::coro_handler<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, unsigned long> >, boost::asio::detail::io_object_executor<boost::asio::io_context::executor_type> >(boost::asio::detail::reactive_socket_service_base::base_implementation_type&, boost::asio::mutable_buffer const&, int, boost::beast::detail::dynamic_read_ops::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::io_context::executor_type>, boost::beast::flat_static_buffer<65536ul>, boost::beast::http::detail::read_header_condition<true>, spawn::detail::coro_handler<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, unsigned long> >&, boost::asio::detail::io_object_executor<boost::asio::io_context::executor_type> const&)+0x1fa) [0x7f06e6d0bd1a]", "(boost::beast::detail::dynamic_read_ops::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::io_context::executor_type>, boost::beast::flat_static_buffer<65536ul>, boost::beast::http::detail::read_header_condition<true>, spawn::detail::coro_handler<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, unsigned long> >::operator()(boost::system::error_code, unsigned long, bool)+0x17f) [0x7f06e6d39f8f]", "(boost::asio::detail::executor_op<boost::asio::detail::binder2<boost::beast::detail::dynamic_read_ops::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::io_context::executor_type>, boost::beast::flat_static_buffer<65536ul>, boost::beast::http::detail::read_header_condition<true>, spawn::detail::coro_handler<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, unsigned long> >, boost::system::error_code, unsigned long>, std::allocator<void>, boost::asio::detail::scheduler_operation>::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long)+0x189) [0x7f06e6d3a7d9]", "(boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::executor_type const>::operator()()+0x8d) [0x7f06e6d0793d]", "(void boost::asio::io_context::executor_type::dispatch<boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::executor_type const>, std::allocator<void> >(boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::executor_type const>&&, std::allocator<void> const&) const+0x9c) [0x7f06e6d07b3c]", "(void boost::asio::detail::strand_executor_service::dispatch<boost::asio::io_context::executor_type const, boost::asio::detail::binder2<boost::beast::detail::dynamic_read_ops::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::io_context::executor_type>, boost::beast::flat_static_buffer<65536ul>, boost::beast::http::detail::read_header_condition<true>, spawn::detail::coro_handler<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, unsigned long> >, boost::system::error_code, unsigned long>, std::allocator<void> >(std::shared_ptr<boost::asio::detail::strand_executor_service::strand_impl> const&, boost::asio::io_context::executor_type const&, boost::asio::detail::binder2<boost::beast::detail::dynamic_read_ops::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::io_context::executor_type>, boost::beast::flat_static_buffer<65536ul>, boost::beast::http::detail::read_header_condition<true>, spawn::detail::coro_handler<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, unsigned long> >, boost::system::error_code, unsigned long>&&, std::allocator<void> const&)+0x2b6) [0x7f06e6d3a306]", "(boost::asio::detail::reactive_socket_recv_op<boost::asio::mutable_buffer, boost::beast::detail::dynamic_read_ops::read_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::io_context::executor_type>, boost::beast::flat_static_buffer<65536ul>, boost::beast::http::detail::read_header_condition<true>, spawn::detail::coro_handler<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, unsigned long> >, boost::asio::detail::io_object_executor<boost::asio::io_context::executor_type> >::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long)+0x1a1) [0x7f06e6d3a571]", "(boost::asio::detail::scheduler::run(boost::system::error_code&)+0x4f2) [0x7f06e6cfbad2]", "/lib64/libradosgw.so.2(+0x430376) [0x7f06e6cde376]", "/lib64/libstdc++.so.6(+0xc2ba3) [0x7f06db212ba3]", "/lib64/libpthread.so.0(+0x81ca) [0x7f06dc1e31ca]", "clone()" ], "ceph_version": "16.2.10", "crash_id": "2023-04-08T22:37:20.389262Z_88939939-522f-4b2c-a5fb-d4e49e9922a7", "entity_name": "client.rgw.os.dsglczutvqsgowpz.a", "os_id": "centos", "os_name": "CentOS Stream", "os_version": "8", "os_version_id": "8", "process_name": "radosgw", "stack_sig": "2535cd0a26a2ffcc7ca223d416ebf3d4ea172eeec60026bb8b36b2c97ea787da", "timestamp": "2023-04-08T22:37:20.389262Z", "utsname_hostname": "node01", "utsname_machine": "x86_64", "utsname_release": "3.10.0-1160.45.1.el7.x86_64", "utsname_sysname": "Linux", "utsname_version": "#1 SMP Wed Oct 13 17:20:51 UTC 2021" }

1 0

rookcmd: failed to configure devices: failed to generate osd keyring: failed to get or create auth key for client.bootstrap-osd:
by knawaz＠juniper.net 13 Apr '23

13 Apr '23

I am deploying Ceph via ROOK on K8s Cluster with following version metric ceph-version=17.2.5-0 Ubuntu 20.04 Kernel 5.4.0-135-generic But getting following error, does ceph-version=17.2.5-0 tested with Ubuntu 20.04 running Kernel 5.4.0-135-generic. Where I can find compatibility metric? 2023-04-12 21:34:41.958118 I | cephclient: getting or creating ceph auth key "client.bootstrap-osd" 2023-04-12 21:34:41.958184 D | exec: Running command: ceph auth get-or-create-key client.bootstrap-osd mon allow profile bootstrap-osd --connect-timeout=15 --cluster=rook-ceph --conf=/var/lib/rook/rook-ceph/rook-ceph.config --name=client.admin --keyring=/var/lib/rook/rook-ceph/client.admin.keyring --format json 2023-04-12 21:34:57.291895 C | rookcmd: failed to configure devices: failed to generate osd keyring: failed to get or create auth key for client.bootstrap-osd: failed get-or-create-key client.bootstrap-osd: exit status 1'

1 0

Upgrade from 17.2.5 to 17.2.6 stuck at MDS
by Thomas Widhalm 12 Apr '23

12 Apr '23

Hi, If you remember, I hit bug https://tracker.ceph.com/issues/58489 so I was very relieved when 17.2.6 was released and started to update immediately. But now I'm stuck again with my broken MDS. MDS won't get into up:active without the update but the update waits for them to get into up:active state. Seems like a deadlock / chicken-egg problem to me. Since I'm still relatively new to Ceph, could you help me? What I see when watching the update status: { "target_image": "quay.io/ceph/ceph@sha256:1161e35e4e02cf377c93b913ce78773f8413f5a8d7c5eaee4b4773a4f9dd6635", "in_progress": true, "which": "Upgrading all daemon types on all hosts", "services_complete": [ "crash", "mgr", "mon", "osd" ], "progress": "18/40 daemons upgraded", "message": "Error: UPGRADE_OFFLINE_HOST: Upgrade: Failed to connect to host ceph01 at addr (192.168.23.61)", "is_paused": false } (The offline host was one host that broke during the upgrade. I fixed that in the meantime and the update went on.) And in the log: 2023-04-10T19:23:48.750129+0000 mgr.ceph04.qaexpv [INF] Upgrade: Waiting for mds.mds01.ceph04.hcmvae to be up:active (currently up:replay) 2023-04-10T19:23:58.758141+0000 mgr.ceph04.qaexpv [WRN] Upgrade: No mds is up; continuing upgrade procedure to poke things in the right direction Please give me a hint what I can do. Cheers, Thomas -- http://www.widhalm.or.at GnuPG : 6265BAE6 , A84CB603 Threema: H7AV7D33 Telegram, Signal: widhalmt(a)widhalm.or.at

3 9