- ceph-users - lists.ceph.io

Degraded FS on 18.2.0 - two monitors per host????
by Robert W. Eckert 18 Aug '23

18 Aug '23

Hi - I have a 4 node cluster, and started to have some odd access issues to my file system "Home" When I started investigating, saw the message "1 MDSs behind on trimming", but I also noticed that I seem to have 2 MDSs running on each server - 3 Daemons up, with 5 standby. Is this expected behavior after the upgrade to 18.2? or did something go wrong? [root@cube ~]# ceph status cluster: id: fe3a7cb0-69ca-11eb-8d45-c86000d08867 health: HEALTH_WARN 1 filesystem is degraded 1 MDSs behind on trimming services: mon: 3 daemons, quorum rhel1,cube,hiho (age 23m) mgr: hiho.bphqff(active, since 23m), standbys: rhel1.owrvaz, cube.sdhftu mds: 3/3 daemons up, 5 standby osd: 16 osds: 16 up (since 23m), 16 in (since 26h) rgw: 4 daemons active (4 hosts, 1 zones) data: volumes: 0/1 healthy, 1 recovering pools: 12 pools, 769 pgs objects: 3.64M objects, 3.1 TiB usage: 17 TiB used, 49 TiB / 65 TiB avail pgs: 765 active+clean 4 active+clean+scrubbing+deep io: client: 154 MiB/s rd, 38 op/s rd, 0 op/s wr [root@cube ~]# ceph health detail HEALTH_WARN 1 filesystem is degraded; 1 MDSs behind on trimming [WRN] FS_DEGRADED: 1 filesystem is degraded fs home is degraded [WRN] MDS_TRIM: 1 MDSs behind on trimming mds.home.story.sodtjs(mds.0): Behind on trimming (5546/128) max_segments: 128, num_segments: 5546 [root@cube ~]# ceph fs status home home - 10 clients ==== RANK STATE MDS ACTIVITY DNS INOS DIRS CAPS 0 replay home.story.sodtjs<http://home.story.sodtjs> 802k 766k 36.7k 0 1 resolve home.cube.xljmfz<http://home.cube.xljmfz> 735k 680k 39.0k 0 2 resolve home.rhel1.nwpmbg<http://home.rhel1.nwpmbg> 322k 316k 17.5k 0 POOL TYPE USED AVAIL home.meta metadata 361G 14.9T home.data data 9206G 14.9T STANDBY MDS home.rhel1.ffrufi<http://home.rhel1.ffrufi> home.hiho.mssdyh<http://home.hiho.mssdyh> home.cube.kmpbku<http://home.cube.kmpbku> home.hiho.cfuswn<http://home.hiho.cfuswn> home.story.gmieio<http://home.story.gmieio> MDS version: ceph version 18.2.0 (5dd24139a1eada541a3bc16b6941c5dde975e26d) reef (stable)

2 2

Re: [ceph v16.2.10] radosgw crash
by 1187873955 18 Aug '23

18 Aug '23

Here is  the backtrace info in  core file (gdb) bt #0  0x00007f79da065b7f in raise () from /lib64/libpthread.so.0 #1  0x00007f79e5303563 in reraise_fatal (signum=6) at /usr/src/debug/ceph-16.2.10-0.el8.x86_64/src/global/signal_handler.cc:332 #2  handle_fatal_signal (signum=6) at /usr/src/debug/ceph-16.2.10-0.el8.x86_64/src/global/signal_handler.cc:332 #3  <signal handler called> #4  0x00007f79d86b6a9f in raise () from /lib64/libc.so.6 #5  0x00007f79d8689e05 in abort () from /lib64/libc.so.6 #6  0x00007f79d905809b in __gnu_cxx::__verbose_terminate_handler() [clone .cold.1] () from /lib64/libstdc++.so.6 #7  0x00007f79d905e53c in __cxxabiv1::__terminate(void (*)()) () from /lib64/libstdc++.so.6 #8  0x00007f79d905e597 in std::terminate() () from /lib64/libstdc++.so.6 #9  0x00007f79d905e52e in std::rethrow_exception(std::__exception_ptr::exception_ptr) () from /lib64/libstdc++.so.6 #10 0x00007f79e4b70a17 in spawn::detail::continuation_context::resume (this=<optimized out>) at /usr/include/c++/8/bits/exception_ptr.h:107 #11 0x00007f79e4b77b6a in spawn::detail::coro_handler<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, std::shared_lock<ceph::async::SharedMutex<boost::asio::io_context::executor_type> > >::operator() (value=..., ec=..., this=0x7f7916eeedd0) at /usr/include/c++/8/bits/shared_ptr_base.h:1018 #12 std::__invoke_impl<void, spawn::detail::coro_handler<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, std::shared_lock<ceph::async::SharedMutex<boost::asio::io_context::executor_type> > >, boost::system::error_code, std::shared_lock<ceph::async::SharedMutex<boost::asio::io_context::executor_type> > > (     __f=...) at /usr/include/c++/8/bits/invoke.h:60 #13 std::__invoke<spawn::detail::coro_handler<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, std::shared_lock<ceph::async::SharedMutex<boost::asio::io_context::executor_type> > >, boost::system::error_code, std::shared_lock<ceph::async::SharedMutex<boost::asio::io_context::executor_type> > > (__fn=...)     at /usr/include/c++/8/bits/invoke.h:95 #14 std::__apply_impl<spawn::detail::coro_handler<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, std::shared_lock<ceph::async::SharedMutex<boost::asio::io_context::executor_type> > >, std::tuple<boost::system::error_code, std::shared_lock<ceph::async::SharedMutex<boost::asio::io_context::executor_type> > >, 0ul, 1ul> (__t=..., __f=...) at /usr/include/c++/8/tuple:1678 #15 std::apply<spawn::detail::coro_handler<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, std::shared_lock<ceph::async::SharedMutex<boost::asio::io_context::executor_type> > >, std::tuple<boost::system::error_code, std::shared_lock<ceph::async::SharedMutex<boost::asio::io_context::executor_type> > > > (     __t=..., __f=...) at /usr/include/c++/8/tuple:1687 #16 ceph::async::CompletionHandler<spawn::detail::coro_handler<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, std::shared_lock<ceph::async::SharedMutex<boost::asio::io_context::executor_type> > >, std::tuple<boost::system::error_code, std::shared_lock<ceph::async::SharedMutex<boost::asio::io_context::executor_type> > > >::operator()() && (this=0x7f7916eeedd0) at /usr/src/debug/ceph-16.2.10-0.el8.x86_64/src/common/async/bind_handler.h:52 #17 ceph::async::ForwardingHandler<ceph::async::CompletionHandler<spawn::detail::coro_handler<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, std::shared_lock<ceph::async::SharedMutex<boost::asio::io_context::executor_type> > >, std::tuple<boost::system::error_code, std::shared_lock<ceph::async::SharedMutex<boost::asio::io_context::executor_type> > > > >::operator()<>() (this=0x7f7916eeedd0) at /usr/src/debug/ceph-16.2.10-0.el8.x86_64/src/common/async/forward_handler.h:47 #18 boost::asio::asio_handler_invoke<ceph::async::ForwardingHandler<ceph::async::CompletionHandler<spawn::detail::coro_handler<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, std::shared_lock<ceph::async::SharedMutex<boost::asio::io_context::executor_type> > >, std::tuple<boost::system::error_code, std::shared_lock<ceph::async::SharedMutex<boost::asio::io_context::executor_type> > > > > > (function=...)     at /usr/src/debug/ceph-16.2.10-0.el8.x86_64/build/boost/include/boost/asio/handler_invoke_hook.hpp:69 #19 boost_asio_handler_invoke_helpers::invoke<ceph::async::ForwardingHandler<ceph::async::CompletionHandler<spawn::detail::coro_handler<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, std::shared_lock<ceph::async::SharedMutex<boost::asio::io_context::executor_type> > >, std::tuple<boost::system::error_code, std::shared_lock<ceph::async::SharedMutex<boost::asio::io_context::executor_type> > > > >, ceph::async::ForwardingHandler<ceph::async::CompletionHandler<spawn::detail::coro_handler<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, std::shared_lock<ceph::async::SharedMutex<boost::asio::io_context::executor_type> > >, std::tuple<boost::system::error_code, std::shared_lock<ceph::async::SharedMutex<boost::asio::io_context::executor_type> > > > > > (context=..., function=...)     at /usr/src/debug/ceph-16.2.10-0.el8.x86_64/build/boost/include/boost/asio/detail/handler_invoke_helpers.hpp:37 #20 boost::asio::detail::executor_op<ceph::async::ForwardingHandler<ceph::async::CompletionHandler<spawn::detail::coro_handler<boost::asio::executor_binder<void (*)(), boost::asio::strand<boost::asio::io_context::executor_type> >, std::shared_lock<ceph::async::SharedMutex<boost::asio::io_context::executor_type> > >, std::tuple<boost::system::error_code, std::shared_lock<ceph::async::SharedMutex<boost::asio::io_context::executor_type> > > > >, std::allocator<void>, boost::asio::detail::scheduler_operation>::do_complete (owner=0x55fd64378910,     base=<optimized out>) at /usr/src/debug/ceph-16.2.10-0.el8.x86_64/build/boost/include/boost/asio/detail/executor_op.hpp:70 #21 0x00007f79e4b7f93d in boost::asio::detail::scheduler_operation::complete (bytes_transferred=0, ec=..., owner=<optimized out>, this=<optimized out>)     at /usr/src/debug/ceph-16.2.10-0.el8.x86_64/build/boost/include/boost/asio/detail/scheduler_operation.hpp:40 #22 boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::executor_type const>::operator() (this=this@entry=0x7f7916eeeed0)     at /usr/src/debug/ceph-16.2.10-0.el8.x86_64/build/boost/include/boost/asio/detail/impl/strand_executor_service.hpp:90 #23 0x00007f79e4b7fca6 in boost::asio::asio_handler_invoke<boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::executor_type const> > (function=...) --Type <RET> for more, q to quit, c to continue without paging--     at /usr/src/debug/ceph-16.2.10-0.el8.x86_64/build/boost/include/boost/asio/handler_invoke_hook.hpp:67 #24 boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::executor_type const>, boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::executor_type const> > (context=..., function=...)     at /usr/src/debug/ceph-16.2.10-0.el8.x86_64/build/boost/include/boost/asio/detail/handler_invoke_helpers.hpp:37 #25 boost::asio::detail::executor_op<boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::executor_type const>, boost::asio::detail::recycling_allocator<void, boost::asio::detail::thread_info_base::default_tag>, boost::asio::detail::scheduler_operation>::do_complete (owner=0x55fd63db8300, base=<optimized out>)     at /usr/src/debug/ceph-16.2.10-0.el8.x86_64/build/boost/include/boost/asio/detail/executor_op.hpp:70 #26 0x00007f79e4b73ad2 in boost::asio::detail::scheduler_operation::complete (bytes_transferred=0, ec=..., owner=0x55fd63db8300, this=<optimized out>)     at /usr/src/debug/ceph-16.2.10-0.el8.x86_64/build/boost/include/boost/asio/detail/scheduler_operation.hpp:40 #27 boost::asio::detail::scheduler::do_run_one (ec=..., this_thread=..., lock=..., this=0x55fd63db8300)     at /usr/src/debug/ceph-16.2.10-0.el8.x86_64/build/boost/include/boost/asio/detail/impl/scheduler.ipp:447 #28 boost::asio::detail::scheduler::run (this=0x55fd63db8300, ec=...) at /usr/src/debug/ceph-16.2.10-0.el8.x86_64/build/boost/include/boost/asio/detail/impl/scheduler.ipp:200 #29 0x00007f79e4b56376 in boost::asio::io_context::run (this=<optimized out>, ec=...)     at /usr/src/debug/ceph-16.2.10-0.el8.x86_64/build/boost/include/boost/asio/impl/io_context.ipp:71 #30 (anonymous namespace)::AsioFrontend::<lambda()>::operator() (__closure=0x55fd640cf148) at /usr/src/debug/ceph-16.2.10-0.el8.x86_64/src/rgw/rgw_asio_frontend.cc:1025 #31 std::__invoke_impl<void, (anonymous namespace)::AsioFrontend::run()::<lambda()> > (__f=...) at /usr/include/c++/8/bits/invoke.h:60 #32 std::__invoke<(anonymous namespace)::AsioFrontend::run()::<lambda()> > (__fn=...) at /usr/include/c++/8/bits/invoke.h:95 #33 std::thread::_Invoker<std::tuple<(anonymous namespace)::AsioFrontend::run()::<lambda()> > >::_M_invoke<0> (this=0x55fd640cf148) at /usr/include/c++/8/thread:244 #34 std::thread::_Invoker<std::tuple<(anonymous namespace)::AsioFrontend::run()::<lambda()> > >::operator() (this=0x55fd640cf148) at /usr/include/c++/8/thread:253 #35 std::thread::_State_impl<std::thread::_Invoker<std::tuple<(anonymous namespace)::AsioFrontend::run()::<lambda()> > > >::_M_run(void) (this=0x55fd640cf140)     at /usr/include/c++/8/thread:196 #36 0x00007f79d908aba3 in execute_native_thread_routine () from /lib64/libstdc++.so.6 #37 0x00007f79da05b1ca in start_thread () from /lib64/libpthread.so.0 #38 0x00007f79d86a1dd3 in clone () from /lib64/libc.so.6

1 0

Re: Check allocated RGW bucket/object size after enabling Bluestore compression
by Konstantin Shalygin 17 Aug '23

17 Aug '23

Hi, What you mean, Bluestore compression? The rgw compression is a rados compression, not the compress by rgw itself. You can setup different storage classes and upload to same pool uncompressed, or compressed objects The compression ratio you can determine with exporter [1] [1] https://github.com/blemmenes/radosgw_usage_exporter k Sent from my iPhone > On 17 Aug 2023, at 17:58, yosr.kchaou96(a)gmail.com wrote: > I see that if we enable RGW compression, we can see the actual (before compression) and utilized (after compression) size of the bucket. > Is it possible to find the same information but with Bluestore compression ?

3 4

Check allocated RGW bucket/object size after enabling Bluestore compression
by yosr.kchaou96＠gmail.com 17 Aug '23

17 Aug '23

Hello, I have enabled Bluestore compression and I see that my data is being compressed by checking the following metrics: bash-4.4$ ceph tell osd.0 perf dump | grep compre "compress_success_count": 24064, "compress_rejected_count": 1, "bluestore_compressed": 1617592, "bluestore_compressed_allocated": 98844672, "bluestore_compressed_original": 197689344, "bluestore_extent_compress": 107478, So far I was able to check the size the compressed data only on osd and pool level, but not on S3 bucket or object level. I see that if we enable RGW compression, we can see the actual (before compression) and utilized (after compression) size of the bucket. Is it possible to find the same information but with Bluestore compression ? Thanks Yosr

1 0

Lost buckets when moving OSD location
by Huy Nguyen 17 Aug '23

17 Aug '23

Hi, I'm using a v16.2.13 Ceph cluster. Yesterday, I just add some SSD node for replace HDD node. During the process, 1 SSD node have different MTU that cause the some PGs become unactive for a while. But after change the MTU, all the PGs is active+clean now. But after that, I can't access some buckets despite their metadata is still there: $ radosgw-admin bucket stats --bucket new-bucket failure: (2002) Unknown error 2002: $ radosgw-admin bucket list --uid owner_id [ ..., "new-bucket", "tesst" ] $ radosgw-admin metadata get bucket.instance:new-bucket:699abe5c-9603-4718-9073-938888ed8bbb.4961564.2 { "key": "bucket.instance:new-bucket:699abe5c-9603-4718-9073-938888ed8bbb.4961564.2", "ver": { "tag": "_ppS69qh0QYSDPuH1I-kp51a", "ver": 32 }, "mtime": "2023-04-03T01:54:24.445009Z", "data": { "bucket_info": { "bucket": { "name": "new-bucket", "marker": "699abe5c-9603-4718-9073-938888ed8bbb.4961564.2", "bucket_id": "699abe5c-9603-4718-9073-938888ed8bbb.4961564.2", "tenant": "", "explicit_placement": { "data_pool": "", "data_extra_pool": "", "index_pool": "" } }, "creation_time": "2022-03-15T08:14:13.199771Z", "owner": "0e0481b66c814e0c9d34b5bee8779205", "flags": 6, "zonegroup": "a453e527-0ded-4e8f-a51e-7eff2293b056", "placement_rule": "default-placement", "has_instance_obj": "true", "quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "num_shards": 0, "bi_shard_hash_type": 0, "requester_pays": "false", "has_website": "false", "swift_versioning": "false", "swift_ver_location": "", "index_type": 0, "mdsearch_config": [], "reshard_status": 0, "new_bucket_instance_id": "" }, "attrs": [ { "key": "user.rgw.acl", "val": "AgJNAQAAAwI+AAAAIAAAADBlMDQ4MWI2NmM4MTRlMGM5ZDM0YjViZWU4Nzc5MjA1FgAAAGFuaHZ0MTIxMjEyQHZjY2xvdWQudm4EAwMBAAABAQAAACAAAAAwZTA0ODFiNjZjODE0ZTBjOWQzNGI1YmVlODc3OTIwNQ8AAAACAAAAAAAAAAUDLAAAAAICBAAAAAIAAAAAAAAAAAAAAAAAAAACAgQAAAABAAAAAAAAAAEAAAAAAAAAIAAAADBlMDQ4MWI2NmM4MTRlMGM5ZDM0YjViZWU4Nzc5MjA1BQNiAAAAAgIEAAAAAAAAACAAAAAwZTA0ODFiNjZjODE0ZTBjOWQzNGI1YmVlODc3OTIwNQAAAAAAAAAAAgIEAAAADwAAABYAAABhbmh2dDEyMTIxMkB2Y2Nsb3VkLnZuAAAAAAAAAAABAAAAAQAAAAEAAAAAAAAA" }, { "key": "user.rgw.iam-policy", "val": "eyJWZXJzaW9uIjogIjIwMTItMTAtMTciLCAiU3RhdGVtZW50IjogW3siRWZmZWN0IjogIkFsbG93IiwgIlByaW5jaXBhbCI6IHsiQVdTIjogWyJhcm46YXdzOmlhbTo6OnVzZXIvdW5kZWZpbmVkOmh1eW5ucC1jcmVhdGUtc3VidXNlci1mcm9tLXJnd2FkbWluIl19LCAiQWN0aW9uIjogWyJzMzpDcmVhdGVCdWNrZXQiLCAiczM6RGVsZXRlQnVja2V0Il0sICJSZXNvdXJjZSI6IFsiYXJuOmF3czpzMzo6Om5ldy1idWNrZXQiLCAiYXJuOmF3czpzMzo6Om5ldy1idWNrZXQvKiJdfV19" } ] } } Do any of you have any ideas on how to get these buckets back? I really appreciate it. Thanks

1 0

Ceph Tech Talk for August 2023: Making Tehthology Friendly
by Mike Perez 16 Aug '23

16 Aug '23

Hi everyone, Join us tomorrow at 15:00 UTC to hear from our Google Summer of Code and Outreachy interns, Devansh Singh and Medhavi Singh, in this next Ceph Tech Talk on Making Teuthology Friendly. https://ceph.io/en/community/tech-talks/ If you want to give atechnicalpresentation for CephTechTalks, please contact me directly with a title and description. Thank you! -- Mike Perez Community Manager Ceph Foundation

1 1

ceph orch upgrade stuck between 16.2.7 and 16.2.13
by Robert Sander 16 Aug '23

16 Aug '23

Hi, A healthy 16.2.7 cluster should get an upgrade to 16.2.13. ceph orch upgrade start --ceph-version 16.2.13 did upgrade MONs, MGRs and 25% of the OSDs and is now stuck. We tried several "ceph orch upgrade stop" and starts again. We "failed" the active MGR but no progress. We set the debug logging with "ceph config set mgr mgr/cephadm/log_to_cluster_level debug" but it only tells that it starts: 2023-08-15T09:05:58.548896+0200 mgr.cephmon01 [INF] Upgrade: Started with target quay.io/ceph/ceph:v16.2.13 How can we check what is happening (or not happening) here? How do we get cephadm to complete the task? Current status is: # ceph orch upgrade status { "target_image": "quay.io/ceph/ceph:v16.2.13", "in_progress": true, "which": "Upgrading all daemon types on all hosts", "services_complete": [], "progress": "", "message": "", "is_paused": false } # ceph -s cluster: id: 3098199a-c7f5-4baf-901c-f178131be6f4 health: HEALTH_WARN There are daemons running an older version of ceph services: mon: 5 daemons, quorum cephmon02,cephmon01,cephmon03,cephmon04,cephmon05 (age 4d) mgr: cephmon03(active, since 8d), standbys: cephmon01, cephmon02 mds: 2/2 daemons up, 1 standby, 2 hot standby osd: 202 osds: 202 up (since 11d), 202 in (since 13d) rgw: 2 daemons active (2 hosts, 1 zones) data: volumes: 2/2 healthy pools: 11 pools, 4961 pgs objects: 98.84M objects, 347 TiB usage: 988 TiB used, 1.3 PiB / 2.3 PiB avail pgs: 4942 active+clean 19 active+clean+scrubbing+deep io: client: 89 MiB/s rd, 598 MiB/s wr, 25 op/s rd, 157 op/s wr progress: Upgrade to quay.io/ceph/ceph:v16.2.13 (0s) [............................] # ceph versions { "mon": { "ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)": 5 }, "mgr": { "ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)": 3 }, "osd": { "ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)": 48, "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)": 154 }, "mds": { "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)": 5 }, "rgw": { "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)": 2 }, "overall": { "ceph version 16.2.13 (5378749ba6be3a0868b51803968ee9cde4833a3e) pacific (stable)": 56, "ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)": 161 } } Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin https://www.heinlein-support.de Tel: 030 / 405051-43 Fax: 030 / 405051-19 Amtsgericht Berlin-Charlottenburg - HRB 220009 B Geschäftsführer: Peer Heinlein - Sitz: Berlin

3 8

CephFS metadata outgrow DISASTER during recovery
by Jakub Petrzilka 16 Aug '23

16 Aug '23

Hello everyone! Recently we had a very nasty incident with one of our CEPH storages. During basic backfill recovery operation due to faulty disk CephFS metadata started growing exponentially until they used all available space and whole cluster DIED. Usage graph screenshot in attachment. Everything was very fast and even when the OSDs were marked full they tripped failsafe and ate all the free blocks, still trying to allocate space and completely died without possibility to even start them again. Only solution was to copy whole bluestore to bigger SSD and resize underlying BS device. Just about 1/3 was able to start after moving but it was enough since we have very redundant settings for cephfs metadata. Basically metadata were moved from 12x 240g SSD to 12x 500GB SSD to have enough space to start again. Brief info about the cluster: - CephFS data are stored on ~500x 8TB SAS HDD using 10+2 ECC in 18 hosts. - CephFS metadata are stored on ~12x 500GB SAS/SATA SSD using 5x replication on 6 hosts. - Version was one of the latest 16.x.x Pacific at the time of the incident. - 3x Mon+mgr and 2 active and 2 hot standby MDS are on separate virtual servers. - typical file size to be stored is from hundreds of MBs to tens of GBs. - this cluster is not the biggest, not having the most HDDs, no special config, I simply see nothing special about this cluster. During investigation I found out the following: - Metadata are outgrowing any time recovery is running on any of maintained clusters (~15 clusters of different usages and sizes) but not this much, this was an extreme situation. - after recovery finish size went fine again. - i think there is slight correlation with recovery width (objects to be touched by recovery in order to recovery everything) and recovery (time) length. But i have no proof. - nothing much else I would like to find out why this happened because i think this can happen again sometime and someone might lose some data if they have less luck. Any ideas are appreciated, or even info if anyone have seen any similar behavior or if i am the only one struggling with issue like this :) Kind regards, Jakub Petrzilka

3 5

Re: Can't join new mon - lossy channel, failing
by Josef Johansson 16 Aug '23

16 Aug '23

Hi, Let's do some serious necromancy here. I just had this exact problem. Turns out that after rebooting all nodes (one at the time of course), the monitor could join perfectly. Why? You tell me. We did not see any traces of the ip address in any dumps that we could get a hold of. I restarted all ceph-mgr beforehand as well. Med vänliga hälsningar Josef Johansson On 10/5/21 15:37, Konstantin Shalygin wrote: > As last resort we've change ipaddr of this host, and mon successfully joined to quorum. When revert ipaddr back - mon can't join, we think there something on switch side or on old mon's side. From old mon's I was checked new mon process connectivity via telnet - all works > It's good to make a some reproducer of this network problem to know what exactly message of ceph protocol is broken > > > > k > > _______________________________________________ > ceph-users mailing list -- ceph-users(a)ceph.io > To unsubscribe send an email to ceph-users-leave(a)ceph.io

2 3

Cephadm adoption - service reconfiguration changes container image
by Iain Stott 16 Aug '23

16 Aug '23

Hi Everyone, We are looking at migrating all our production clusters from ceph-ansible to cephadm. We are currently experiencing an issue where when reconfiguring a service through ceph orch, it will change the running container image for that service which has led to the mgr services running an earlier version than the rest of the cluster, this has caused cephadm/ceph orch to not be able to manage services in the cluster. Does anyone have any help with this as I cannot find anything in the docs that would correct this. Cheers Iain [root@de1-ceph-mon-ceph-site-a-1 ~]# cephadm --image quay.io/ceph/ceph:v16.2.9 adopt --style legacy --name mon.$(hostname -s) [root@de1-ceph-mon-ceph-site-a-1 ~]# cat ./mon.yaml service_type: mon service_name: mon placement: host_pattern: '*mon*' extra_container_args: - -v - /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.client.admin.keyring [root@de1-ceph-mon-ceph-site-a-1 ~]# cephadm shell -- ceph config get mgr mgr/cephadm/container_image_base quay.io/ceph/ceph [root@de1-ceph-mon-ceph-site-a-1 ~]# podman ps -a | grep mon 55fe8bc10476 quay.io/ceph/ceph:v16.2.9 -n mgr.de1-ceph-m... 5 minutes ago Up 5 minutes ceph-da9e9837-a3cf-4482-9a13-790a721598cd-mgr-de1-ceph-mon-ceph-site-a-1 [root@de1-ceph-mon-ceph-site-a-1 ~]# cat ./mon.yaml | cephadm --image quay.io/ceph/ceph:v16.2.9 shell -- ceph orch apply -i - Inferring fsid da9e9837-a3cf-4482-9a13-790a721598cd Scheduled mon update... [root@de1-ceph-mon-ceph-site-a-1 ~]# podman ps -a | grep mon ecec1d62c719 docker.io/ceph/daemon-base:latest-pacific-devel -n mon.de1-ceph-m... 25 seconds ago Up 26 seconds ceph-da9e9837-a3cf-4482-9a13-790a721598cd-mon-de1-ceph-mon-ceph-site-a-1 [root@de1-ceph-mon-ceph-site-a-2 ~]# ceph versions { "mon": { "ceph version 16.2.5-387-g7282d81d (7282d81d2c500b5b0e929c07971b72444c6ac424) pacific (stable)": 3 }, "mgr": { "ceph version 16.2.9 (4c3647a322c0ff5a1dd2344e039859dcbd28c830) pacific (stable)": 3 }, "osd": { "ceph version 16.2.9 (4c3647a322c0ff5a1dd2344e039859dcbd28c830) pacific (stable)": 8 }, "mds": {}, "rgw": { "ceph version 16.2.9 (4c3647a322c0ff5a1dd2344e039859dcbd28c830) pacific (stable)": 3 }, "overall": { "ceph version 16.2.5-387-g7282d81d (7282d81d2c500b5b0e929c07971b72444c6ac424) pacific (stable)": 3, "ceph version 16.2.9 (4c3647a322c0ff5a1dd2344e039859dcbd28c830) pacific (stable)": 14 } } Iain Stott OpenStack Engineer Iain.Stott(a)thehutgroup.com [THG Ingenuity Logo]<https://www.thg.com> [https://i.imgur.com/wbpVRW6.png]<https://www.linkedin.com/company/thgplc/?originalSubdomain=uk> [https://i.imgur.com/c3040tr.png] <https://twitter.com/thgplc?lang=en>

4 4