- ceph-users - lists.ceph.io

Ceph Nautilous 14.2.22 slow OSD memory leak?
by huxiaoyu＠horebdata.cn 13 Jan '24

13 Jan '24

Dear Ceph folks, I am responsible for two Ceph clusters, running Nautilius 14.2.22 version, one with replication 3, and the other with EC 4+2. After around 400 days runing quietly and smoothly, recently the two clusters occured with similar problems: some of OSDs consume ca 18 GB while the memory target is setting at 2GB. What could wrong in the background? Does it mean any slow OSD memory leak issues with 14.2.22 which i do not know yet? I would be highly appreciated if some some provides any clues, ideas, comments ...... best regards, Samuel huxiaoyu(a)horebdata.cn

5 6

About ceph disk slowops effect to cluster
by Phong Tran Thanh 13 Jan '24

13 Jan '24

Hi community I'm currently facing a significant issue with my Ceph cluster. I have a cluster consisting of 10 nodes, and each node is equipped with 6 SSDs of 960GB used for block.db and 18 12TB drives used for data, network bonding 2x10Gbps for public and local networks. I am using a 4+2 erasure code for RBD in my Ceph cluster. When one node becomes unavailable, the cluster initiates the recovery process, and subsequently, slow operations (slowops) logs appear on the disk, impacting the entire cluster. Afterward, additional nodes are marked as failures. Is this phenomenon possibly due to the performance of SSDs and HDDs? When I check the I/O of the disk using the iostat command, the result shows that disk utilization has reached 80-90% Is using a combination of HDDs and SAS SSDs in Ceph a choice leading to poor performance? My Ceph cluster has a bandwidth of 1.9GB/s Thanks and hope someone can help me ---------------------------------------------------------------------------- Email: tranphong079(a)gmail.com Skype: tranphong079

4 7

RGW rate-limiting or anti-hammering for (external) auth requests // Anti-DoS measures
by Christian Rohmann 12 Jan '24

12 Jan '24

Hey Ceph-Users, RGW does have options [1] to rate limit ops or bandwidth per bucket or user. But those only come into play when the request is authenticated. I'd like to also protect the authentication subsystem from malicious or invalid requests. So in case e.g. some EC2 credentials are not valid (anymore) and clients start hammering the RGW with those requests, I'd like to make it cheap to deal with those requests. Especially in case some external authentication like OpenStack Keystone [2] is used, valid access tokens are cached within the RGW. But requests with invalid credentials end up being sent at full rate to the external API [3] as there is no negative caching. And even if there was, that would only limit the external auth requests for the same set of invalid credentials, but it would surely reduce the load in that case: Since the HTTP request is blocking .... > [...] > 2023-12-18T15:25:55.861+0000 7fec91dbb640 20 sending request to > https://keystone.example.com/v3/s3tokens > 2023-12-18T15:25:55.861+0000 7fec91dbb640 20 register_request > mgr=0x561a407ae0c0 req_data->id=778, curl_handle=0x7fedaccb36e0 > 2023-12-18T15:25:55.861+0000 7fec91dbb640 20 WARNING: blocking http > request > 2023-12-18T15:25:55.861+0000 7fede37fe640 20 link_request > req_data=0x561a40a418b0 req_data->id=778, curl_handle=0x7fedaccb36e0 > [...] this does not only stress the external authentication API (keystone in this case), but also blocks RGW threads for the duration of the external call. I am currently looking into using the capabilities of HAProxy to rate limit requests based on their resulting http-response [4]. So in essence to rate-limit or tarpit clients that "produce" a high number of 403 "InvalidAccessKeyId" responses. To have less collateral it might make sense to limit based on the presented credentials themselves. But this would require to extract and track HTTP headers or URL parameters (presigned URLs) [5] and to put them into tables. * What are your thoughts on the matter? * What kind of measures did you put in place? * Does it make sense to extend RGWs capabilities to deal with those cases itself? ** adding negative caching ** rate limits on concurrent external authentication requests (or is there a pool of connections for those requests?) Regards Christian [1] https://docs.ceph.com/en/latest/radosgw/admin/#rate-limit-management [2] https://docs.ceph.com/en/latest/radosgw/keystone/#integrating-with-openstac… [3] https://github.com/ceph/ceph/blob/86bb77eb9633bfd002e73b5e58b863bc2d0df594/… [4] https://www.haproxy.com/documentation/haproxy-configuration-manual/latest/#… [5] https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-reque…

2 3

Is there any way to merge an rbd image's full backup and a diff?
by Satoru Takeuchi 12 Jan '24

12 Jan '24

Hi, I'm developing RBD images' backup system. In my case, a backup data must be stored at least two weeks. To meet this requirement, I'd like to take backups as follows: 1. Take a full backup by rbd export first. 2. Take a differencial backups everyday. 3. Merge the full backup and the oldest (taken two weeks ago) diff. As a result of evaluation, I confirmed there is no problem in step 1 and 2. However, I found that step 3 couldn't be accomplished by `rbd merge-diff <full backup> <diff>` because `rbd merge-diff` only accepts a diff as a first parameter. Is there any way to merge a full backup and a diff? Thanks, Satoru

2 5

Pacific bluestore_volume_selection_policy
by Reed Dier 11 Jan '24

11 Jan '24

After ~3 uneventful weeks after upgrading from 15.2.17 to 16.2.14 I’ve started seeing OSD crashes with "cur >= fnode.size” and "cur >= p.length”, which seems to be resolved in the next point release for pacific later this month, but until then, I’d love to keep the OSDs from flapping. > $ for crash in $(ceph crash ls | grep osd | awk '{print $1}') ; do ceph crash info $crash | egrep "(assert_condition|crash_id)" ; done > "assert_condition": "cur >= fnode.size", > "crash_id": "2024-01-03T09:07:55.698213Z_348af2d3-d4a7-4c27-9f71-70e6dc7c1af7", > "assert_condition": "cur >= p.length", > "crash_id": "2024-01-03T14:21:55.794692Z_4557c416-ffca-4165-aa91-d63698d41454", > "assert_condition": "cur >= fnode.size", > "crash_id": "2024-01-03T22:53:43.010010Z_15dc2b2a-30fb-4355-84b9-2f9560f08ea7", > "assert_condition": "cur >= p.length", > "crash_id": "2024-01-04T02:34:34.408976Z_2954a2c2-25d2-478e-92ad-d79c42d3ba43", > "assert_condition": "cur2 >= p.length", > "crash_id": "2024-01-04T21:57:07.100877Z_12f89c2c-4209-4f5a-b243-f0445ba629d2", > "assert_condition": "cur >= p.length", > "crash_id": "2024-01-05T00:35:08.561753Z_a189d967-ab02-4c61-bf68-1229222fd259", > "assert_condition": "cur >= fnode.size", > "crash_id": "2024-01-05T04:11:48.625086Z_a598cbaf-2c4f-4824-9939-1271eeba13ea", > "assert_condition": "cur >= p.length", > "crash_id": "2024-01-05T13:49:34.911210Z_953e38b9-8ae4-4cfe-8f22-d4b7cdf65cea", > "assert_condition": "cur >= p.length", > "crash_id": "2024-01-05T13:54:25.732770Z_4924b1c0-309c-4471-8c5d-c3aaea49166c", > "assert_condition": "cur >= p.length", > "crash_id": "2024-01-05T16:35:16.485416Z_0bca3d2a-2451-4275-a049-a65c58c1aff1”, As noted in https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/YNJ35HXN4H… <https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/YNJ35HXN4H…> > You can apparently work around the issue by setting > 'bluestore_volume_selection_policy' config parameter to rocksdb_original. However, after trying to set that parameter with `ceph config set osd.$osd bluestore_volume_selection_policy rocksdb_original` it doesn’t appear to set? > $ ceph config show-with-defaults osd.0 | grep bluestore_volume_selection_policy > bluestore_volume_selection_policy use_some_extra > $ ceph config set osd.0 bluestore_volume_selection_policy rocksdb_original > $ ceph config show osd.0 | grep bluestore_volume_selection_policy > bluestore_volume_selection_policy use_some_extra default mom This, I assume, should reflect the new setting, however it still shows the default “use_some_extra” value. But then this seems to imply that the config is set? > $ ceph config dump | grep bluestore_volume_selection_policy > osd.0 dev bluestore_volume_selection_policy rocksdb_original * > [snip] > osd.9 dev bluestore_volume_selection_policy rocksdb_original * Does this need to be set in ceph.conf or is there another setting that also needs to be set? Even after bouncing the OSD daemon, `ceph config show` still reports “use_some_extra" Appreciate any help they can offer to point me towards to bridge the gap between now and the next point release. Thanks, Reed

3 6

Unable to execute radosgw command using cephx users on client side
by Alam Mohammad 11 Jan '24

11 Jan '24

Hello, In our Ceph cluster we encountered issues while attempting to execute "radosgw-admin" command on client side using cephx user having read only permission. Whenever we are executing "radosgw-admin user list" command it is throwing an error. "ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)" We have performed below steps in our environment Case-1 : First we created cephx user with below privileges client.rgw.username key: <-------key-------> caps: [mgr] allow r caps: [mon] allow r caps: [osd] allow r tag rgw *=* on client side we copied keyring and ceph.conf file What we noticed on client machine all general command like "ceph -s", "ceph health detail" "ceph df" running fine, even "radosgw-admin zonegroup list --id=rgw.username," command returned the expected output, but when attempting commands like "radosgw-admin user list," "radosgw-admin bucket list," or "radosgw-admin user info," errors were encountered. Below are the outputs that is throwing root@control:~# radosgw-admin user list --id=rgw.username 2024-01-03T17:34:06.498+0000 7f915ece1fc0 0 ERROR: failed reading data (obj=default.rgw.log:bucket.sync-source-hints.), r=-1 2024-01-03T17:34:06.498+0000 7f915ece1fc0 0 ERROR: failed to update sources index for bucket=:[]) r=-1 2024-01-03T17:34:06.498+0000 7f915ece1fc0 0 ERROR: failed to initialize bucket sync policy handler: get_bucket_sync_hints() on bucket=-- returned r=-1 2024-01-03T17:34:06.498+0000 7f915ece1fc0 -1 ERROR: could not initialize zone policy handler for zone=default 2024-01-03T17:34:06.498+0000 7f915ece1fc0 0 ERROR: failed to start notify service ((1) Operation not permitted 2024-01-03T17:34:06.498+0000 7f915ece1fc0 0 ERROR: failed to init services (ret=(1) Operation not permitted) couldn't init storage provider Case- 2 : In this case we granted read permissions to the rgw data pool and index pool for the user, client.rgw.username key: <----key----> caps: [mgr] allow r caps: [mon] allow r caps: [osd] allow r pool=default.rgw.log Despite this, while general commands worked perfectly fine on the client side, but "radosgw-admin" commands still failed to execute. Here is the output root@control:~# radosgw-admin user list --id=rgw.username 2024-01-03T17:43:38.071+0000 7f8b5a8bffc0 0 failed reading realm info: ret -1 (1) Operation not permitted 2024-01-03T17:43:38.071+0000 7f8b5a8bffc0 0 ERROR: failed to start notify service ((1) Operation not permitted 2024-01-03T17:43:38.071+0000 7f8b5a8bffc0 0 ERROR: failed to init services (ret=(1) Operation not permitted) couldn't init storage provider Have I overlooked anything in the process? Any guidance or insight would be greatly appreciated. Thanks, Mohammad Saif Ceph Enthusiast In the first step, we created a CephX user named client.rgw.saif with read permissions for the manager (mgr), monitor (mon), and object storage daemon (osd) components, along with specific RGW capabilities. On the client side, we successfully copied the keyring and ceph.conf, and certain commands, such as radosgw-admin zonegroup list --id=rgw.username,

3 5

Re: Stuck in upgrade process to reef
by Igor Fedotov 11 Jan '24

11 Jan '24

Hi Jan, unfortunately this wasn't very helpful. Moreover the log looks a bit messy - looks like a mixture of outputs from multiple running instances or something. I'm not an expert in using containerized setups though. Could you please simplify things by running ceph-osd process manually like you did for ceph-objectstore-tool. And enforce log output to a file. Command line should look somewhat the following: ceph-osd -i 0 --log-to-file --log-file <some-file> --debug-bluestore 5/20 --debug-prioritycache 10 Please don't forget to run repair prior to that. Also you haven't answered my questions about custom [memory] settings and RAM usage during OSD startup. It would be nice to hear some feedback. Thanks, Igor On 11/01/2024 16:47, Jan Marek wrote: > Hi Igor, > > I've tried to start osd.1 with debug_prioritycache and > debug_bluestore 5/20, see attached file... > > Sincerely > Jan > > Dne St, led 10, 2024 at 01:03:07 CET napsal(a) Igor Fedotov: >> Hi Jan, >> >> indeed this looks like some memory allocation problem - may be OSD's RAM >> usage threshold reached or something? >> >> Curious if you have any custom OSD settings or may be any memory caps for >> Ceph containers? >> >> Could you please set debug_bluestore to 5/20 and debug_prioritycache to 10 >> and try to start OSD once again. Please monitor process RAM usage along the >> process and share the resulting log. >> >> >> Thanks, >> >> Igor >> >> On 10/01/2024 11:20, Jan Marek wrote:

1 0

ceph-volume fails in all recent releases with IndexError
by Nico Schottelius 11 Jan '24

11 Jan '24

Hello dear fellow ceph users, it seems that for some months all current ceph releases (16.x, 17.x, 18.x) are having a bug in ceph-volume that causes disk activation to fail with the error "IndexError: list index out of range" (details below, [0]). It also seems there is already a fix for it available [1], but also that it hasn't been merged into any official release [2,3,4]. This has started to affect more and more nodes in our clusters and thus I was wondering if others are also seeing this issue and whether anyone knows whether it is planned to create a new release based on this soon? Best regards, Nico -------------------------------------------------------------------------------- [0] kubectl -n rook-ceph logs -c activate rook-ceph-osd-30-6558b7cf69-5cbbl + OSD_ID=30 + CEPH_FSID=bd3061a0-ecf3-4af6-9017-51b63c90b526 + OSD_UUID=319e5756-318c-46a0-b7e9-429e39069302 + OSD_STORE_FLAG=--bluestore + OSD_DATA_DIR=/var/lib/ceph/osd/ceph-30 + CV_MODE=raw + DEVICE=/dev/sdf + cp --no-preserve=mode /etc/temp-ceph/ceph.conf /etc/ceph/ceph.conf + python3 -c ' import configparser config = configparser.ConfigParser() config.read('\''/etc/ceph/ceph.conf'\'') if not config.has_section('\''global'\''): config['\''global'\''] = {} if not config.has_option('\''global'\'','\''fsid'\''): config['\''global'\'']['\''fsid'\''] = '\''....\'' with open('\''/etc/ceph/ceph.conf'\'', '\''w'\'') as configfile: config.write(configfile) ' + ceph -n client.admin auth get-or-create osd.30 mon 'allow profile osd' mgr 'allow profile osd' osd 'allow *' -k /etc/ceph/admin-keyring-store/keyring [osd.30] key = ... + [[ raw == \l\v\m ]] ++ mktemp + OSD_LIST=/tmp/tmp.OpZRJJOcrX + ceph-volume raw list /dev/sdf Traceback (most recent call last): File "/usr/sbin/ceph-volume", line 11, in <module> load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')() File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__ self.main(self.argv) File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc return f(*a, **kw) File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main terminal.dispatch(self.mapper, subcommand_args) File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch instance.main() File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/main.py", line 32, in main terminal.dispatch(self.mapper, self.argv) File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch instance.main() File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/list.py", line 166, in main self.list(args) File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root return func(*a, **kw) File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/list.py", line 122, in list report = self.generate(args.device) File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/list.py", line 91, in generate info_device = [info for info in info_devices if info['NAME'] == dev][0] IndexError: list index out of range [1] https://github.com/ceph/ceph/pull/49954 [2] https://github.com/ceph/ceph/pull/54705 [3] https://github.com/ceph/ceph/pull/54706 [4] https://github.com/ceph/ceph/pull/54707 -- Sustainable and modern Infrastructures by ungleich.ch

2 1

Rack outage test failing when nodes get integrated again
by Steve Baker 11 Jan '24

11 Jan '24

Hi, we're currently testing a ceph (v 16.2.14) cluster, 3 mon nodes, 6 osd nodes à 8 nvme ssd osds distributed over 3 racks. Daemons are deployed in containers with cephadm / podman. We got 2 pools on it, one with 3x replication and min_size=2, one with an EC (k3m3). With 1 mon node and 2 osd nodes in each rack, the crush rules are configured in a way (for the 3x pool chooseleaf_firstn rack, for the ec pool choose_indep 3 rack / chooseleaf_indep 2 host), so that a full rack can go down while the cluster stays accessible for client operations. Other options we have set are mon_osd_down_out_subtree_limit=host so that in case of a host/rack outage the cluster does not automatically start to backfill, but will continue to run in a degraded state until human interaction comes to fix it. Also we set mon_osd_reporter_subtree_level=rack. We tested - while under (synthetic test-)client load - what happens if we take a full rack (one mon node and 2 osd nodes) out of the cluster. We did that using iptables to block the nodes of the rack from other nodes of the cluster (global and cluster network), as well as from the clients. As expected, the remainder of the cluster continues to run in a degraded state without to start any backfilling or recovery processes. All client requests gets served while the rack is out. But then a strange thing happens when we take the rack (1mon node, 2 osd nodes) back into the cluster again by deleting all firewall rules with iptables -F at once. Some osds get integrated in the cluster again immediatelly but some others remain in state "down" for exactly 10 minutes. These osds that stay down for the 10 minutes are in a state where they still seem to not be able to reach other osd nodes (see heartbeat_check logs below). After these 10 minutes have passed, these osds come up as well but then at exactly that time, many PGs get stuck in peering state and other osds that were all the time in the cluster get slow requests and the cluster blocks client traffic (I think it's just the PGs stuck in peering soaking all the client threads). Then, exactly 45 minutes after the nodes of the rack were made reachable by iptables -F again, the situation recovers, peering succeeds and client load gets handled again. We have repeated this test several times and it's always exactly the same 10 min "down interval" and a 45 min affected client requests. When we integrate the nodes into the cluster again one after another with a delay of some minutes inbetween, this does not happen at all. I wonder what's happening there. It must be some kind of split-brain situation having to do with blocking the nodes using iptables but not rebooting them completelly. The 10 min and 45 min intervals I described occure every time. For the 10 minutes, some osds stay down after the hosts got integrated again. It's not all of the 16 osds from the 2 osd hosts that got integrated again but just some of them. Which ones varies randomly. Sometimes it's also only just one. We also observerd, the longer the hosts were out of the cluster, the more osds are affected. Then even after they get up again after 10 minutes, it takes another 45 minutes until the stuck peering situation resolves. Also during these 45 minutes, we see slow ops on osds thet remained into the cluster. #################################################### See here some OSD logs that get written after the reintegration: #################################################### 2024-01-04T08:25:03.856+0000 7f369132b700 -1 monclient: _check_auth_rotating possible clock skew, rotating keys expired way too early (before 2024-01-04T07:25:03.860426+0000) 2024-01-04T08:25:06.556+0000 7f3682882700 0 log_channel(cluster) log [WRN] : Monitor daemon marked osd.0 down, but it is still running 2024-01-04T08:25:06.556+0000 7f3682882700 0 log_channel(cluster) log [DBG] : map e62160 wrongly marked me down at e62136 2024-01-04T08:25:06.556+0000 7f3682882700 1 osd.0 62160 start_waiting_for_healthy 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.5:6810 osd.2 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.8:6814 osd.3 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.6:6822 osd.4 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.9:6830 osd.5 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.6:6830 osd.7 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.8:6830 osd.9 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.5:6802 osd.10 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.9:6802 osd.11 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.6:6802 osd.13 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.8:6802 osd.15 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.5:6806 osd.16 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.9:6806 osd.17 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.6:6806 osd.20 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.8:6806 osd.21 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.9:6810 osd.22 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.5:6814 osd.23 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.6:6810 osd.26 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.8:6810 osd.27 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.9:6814 osd.28 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.5:6818 osd.29 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.6:6814 osd.32 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.8:6818 osd.33 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.9:6818 osd.34 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.5:6822 osd.35 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.6:6818 osd.37 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.9:6822 osd.39 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.8:6822 osd.40 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.5:6826 osd.41 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.6:6826 osd.43 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.9:6826 osd.44 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.8:6826 osd.46 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) 2024-01-04T08:25:06.872+0000 7f368caa5700 -1 osd.0 62160 heartbeat_check: no reply from 1.2.3.5:6830 osd.47 ever on either front or back, first ping sent 2024-01-04T08:21:04.601131+0000 (oldest deadline 2024-01-04T08:21:24.601131+0000) [The block of loglines above gets repeated for 45 minutes until everything is fine again, the block below comes once at the beginning after the reintegration, then starts again after the 10 min interval and the repeats until the end of the 45 min interval until everything is fine again.] 2024-01-04T08:25:07.036+0000 7f3691b2c700 0 auth: could not find secret_id=1363 2024-01-04T08:25:07.036+0000 7f3691b2c700 0 cephx: verify_authorizer could not get service secret for service osd secret_id=1363 2024-01-04T08:25:07.036+0000 7f369132b700 0 auth: could not find secret_id=1363 2024-01-04T08:25:07.036+0000 7f369132b700 0 cephx: verify_authorizer could not get service secret for service osd secret_id=1363 2024-01-04T08:25:07.236+0000 7f369232d700 0 auth: could not find secret_id=1363 2024-01-04T08:25:07.236+0000 7f369232d700 0 cephx: verify_authorizer could not get service secret for service osd secret_id=1363 2024-01-04T08:25:07.236+0000 7f369232d700 0 auth: could not find secret_id=1363 2024-01-04T08:25:07.236+0000 7f369232d700 0 cephx: verify_authorizer could not get service secret for service osd secret_id=1363 2024-01-04T08:35:07.225+0000 7f369232d700 0 auth: could not find secret_id=1365 2024-01-04T08:35:07.225+0000 7f369232d700 0 cephx: verify_authorizer could not get service secret for service osd secret_id=1365 [The block below gets logged for 10 minutes until the osd is not down anymore] 2024-01-04T08:25:08.368+0000 7f368d2a6700 1 osd.0 62162 is_healthy false -- only 0/10 up peers (less than 33%) 2024-01-04T08:25:08.368+0000 7f368d2a6700 1 osd.0 62162 not healthy; waiting to boot 2024-01-04T08:25:09.340+0000 7f368d2a6700 1 osd.0 62162 is_healthy false -- only 0/10 up peers (less than 33%) 2024-01-04T08:25:09.340+0000 7f368d2a6700 1 osd.0 62162 not healthy; waiting to boot 2024-01-04T08:25:10.316+0000 7f368d2a6700 1 osd.0 62162 is_healthy false -- only 0/10 up peers (less than 33%) 2024-01-04T08:25:10.316+0000 7f368d2a6700 1 osd.0 62162 not healthy; waiting to boot After 10 minutes then, the osd seems to reboot: 2024-01-04T08:35:07.005+0000 7f368d2a6700 1 osd.0 62509 start_boot 2024-01-04T08:35:07.009+0000 7f368b2a2700 1 osd.0 62509 set_numa_affinity storage numa node 0 2024-01-04T08:35:07.009+0000 7f368b2a2700 -1 osd.0 62509 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-04T08:35:07.009+0000 7f368b2a2700 1 osd.0 62509 set_numa_affinity not setting numa affinity 2024-01-04T08:35:07.197+0000 7f367ea40700 2 osd.0 62509 ms_handle_reset con 0x561d78ec6000 session 0x561d8ae5f0e0 2024-01-04T08:35:07.213+0000 7f3682882700 1 osd.0 62521 state: booting -> active ############################################################## See here some logs of the active mon that get written after the reintegration: ############################################################## 2024-01-04T08:25:06.486+0000 7ff5fa87b700 5 mon.ceph-mon01(a)0(leader).osd e62160 send_latest to osd.0 v2:... start 62136 2024-01-04T08:25:06.486+0000 7ff5fa87b700 1 mon.ceph-mon01(a)0(leader).osd e62160 ignoring beacon from non-active osd.0 2024-01-04T08:25:06.490+0000 7ff5f9078700 0 log_channel(cluster) log [WRN] : osd.0 (root=default,rack=rack3,host=ceph-osd07) is down 2024-01-04T08:25:06.642+0000 7ff5f9078700 0 log_channel(cluster) log [INF] : osd.0 marked itself dead as of e62160 2024-01-04T08:25:07.434+0000 7ff5fa87b700 5 mon.ceph-mon01(a)0(leader).osd e62161 preprocess_failure dne(/dup?): osd.0 [v2:...,v1:...], from osd.45 2024-01-04T08:29:59.998+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : osd.0 (root=default,rack=rack3,host=ceph-osd07) is down 2024-01-04T08:29:59.998+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : Slow OSD heartbeats on back from osd.45 [rack3] to osd.0 [rack3] (down) 221701.273 msec 2024-01-04T08:29:59.998+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : Slow OSD heartbeats on front from osd.45 [rack3] to osd.0 [rack3] (down) 221700.153 msec 2024-01-04T08:31:43.019+0000 7ff5fa87b700 5 mon.ceph-mon01(a)0(leader).osd e62434 send_incremental [62416..62434] to osd.0 2024-01-04T08:32:13.915+0000 7ff5fa87b700 5 mon.ceph-mon01(a)0(leader).osd e62443 send_incremental [62435..62443] to osd.0 2024-01-04T08:32:44.891+0000 7ff5fa87b700 5 mon.ceph-mon01(a)0(leader).osd e62461 send_incremental [62444..62461] to osd.0 2024-01-04T08:33:15.752+0000 7ff5fa87b700 5 mon.ceph-mon01(a)0(leader).osd e62465 send_incremental [62462..62465] to osd.0 2024-01-04T08:33:39.148+0000 7ff5fa87b700 5 mon.ceph-mon01(a)0(leader).osd e62483 preprocess_failure from dead osd.0, ignoring 2024-01-04T08:33:39.148+0000 7ff5fa87b700 5 mon.ceph-mon01(a)0(leader).osd e62483 send_incremental [62466..62483] to osd.0 2024-01-04T08:33:39.148+0000 7ff5fa87b700 5 mon.ceph-mon01(a)0(leader).osd e62483 preprocess_failure from dead osd.0, ignoring 2024-01-04T08:33:39.148+0000 7ff5fa87b700 5 mon.ceph-mon01(a)0(leader).osd e62483 send_incremental [62466..62483] to osd.0 2024-01-04T08:33:39.148+0000 7ff5fa87b700 5 mon.ceph-mon01(a)0(leader).osd e62483 preprocess_failure from dead osd.0, ignoring 2024-01-04T08:33:39.148+0000 7ff5fa87b700 5 mon.ceph-mon01(a)0(leader).osd e62483 send_incremental [62466..62483] to osd.0 2024-01-04T08:33:39.148+0000 7ff5fa87b700 5 mon.ceph-mon01(a)0(leader).osd e62483 preprocess_failure from dead osd.0, ignoring 2024-01-04T08:33:39.148+0000 7ff5fa87b700 5 mon.ceph-mon01(a)0(leader).osd e62483 send_incremental [62466..62483] to osd.0 2024-01-04T08:33:39.148+0000 7ff5fa87b700 5 mon.ceph-mon01(a)0(leader).osd e62483 preprocess_failure from dead osd.0, ignoring 2024-01-04T08:33:39.148+0000 7ff5fa87b700 5 mon.ceph-mon01(a)0(leader).osd e62483 send_incremental [62466..62483] to osd.0 2024-01-04T08:33:39.148+0000 7ff5fa87b700 5 mon.ceph-mon01(a)0(leader).osd e62483 preprocess_failure from dead osd.0, ignoring 2024-01-04T08:33:39.148+0000 7ff5fa87b700 5 mon.ceph-mon01(a)0(leader).osd e62483 send_incremental [62466..62483] to osd.0 2024-01-04T08:33:39.148+0000 7ff5fa87b700 5 mon.ceph-mon01(a)0(leader).osd e62483 preprocess_failure from dead osd.0, ignoring 2024-01-04T08:33:39.148+0000 7ff5fa87b700 5 mon.ceph-mon01(a)0(leader).osd e62483 send_incremental [62466..62483] to osd.0 2024-01-04T08:33:39.148+0000 7ff5fa87b700 5 mon.ceph-mon01(a)0(leader).osd e62483 preprocess_failure from dead osd.0, ignoring 2024-01-04T08:33:39.148+0000 7ff5fa87b700 5 mon.ceph-mon01(a)0(leader).osd e62483 send_incremental [62466..62483] to osd.0 2024-01-04T08:33:39.148+0000 7ff5fa87b700 5 mon.ceph-mon01(a)0(leader).osd e62483 preprocess_failure from dead osd.0, ignoring 2024-01-04T08:33:39.148+0000 7ff5fa87b700 5 mon.ceph-mon01(a)0(leader).osd e62483 send_incremental [62466..62483] to osd.0 2024-01-04T08:33:39.148+0000 7ff5fa87b700 5 mon.ceph-mon01(a)0(leader).osd e62483 preprocess_failure from dead osd.0, ignoring 2024-01-04T08:33:39.148+0000 7ff5fa87b700 5 mon.ceph-mon01(a)0(leader).osd e62483 send_incremental [62466..62483] to osd.0 2024-01-04T08:33:51.240+0000 7ff5f9078700 5 mon.ceph-mon01(a)0(leader).osd e62484 send_incremental [62484..62484] to osd.0 2024-01-04T08:35:04.077+0000 7ff5fd080700 0 log_channel(cluster) log [INF] : Marking osd.0 out (has been down for 600 seconds) 2024-01-04T08:35:04.085+0000 7ff5fd080700 2 mon.ceph-mon01(a)0(leader).osd e62517 osd.0 OUT 2024-01-04T08:35:07.173+0000 7ff5fd080700 2 mon.ceph-mon01(a)0(leader).osd e62520 osd.0 UP [v2:...,v1:...] 2024-01-04T08:35:07.173+0000 7ff5fd080700 2 mon.ceph-mon01(a)0(leader).osd e62520 osd.0 IN This is what gets logged shortly before the situation recovers: 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : Health detail: HEALTH_WARN Reduced data availability: 292 pgs inactive, 292 pgs peering; Degraded data redundancy: 14614397/4091962683 objects degraded (0.357%), 322 pgs degraded, 360 pgs undersized; 2 pools have too many placement groups; 9472 slow ops, oldest one blocked for 2091 sec, daemons [osd.10,osd.11,osd.13,osd.14,osd.15,osd.16,osd.17> 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : [WRN] PG_AVAILABILITY: Reduced data availability: 292 pgs inactive, 292 pgs peering 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 14.212 is stuck peering for 34m, current state remapped+peering, last acting [4,21] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 14.220 is stuck peering for 34m, current state remapped+peering, last acting [3,35] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 14.227 is stuck peering for 34m, current state remapped+peering, last acting [44,26] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 14.228 is stuck peering for 34m, current state remapped+peering, last acting [23,5] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 14.22b is stuck peering for 34m, current state remapped+peering, last acting [32,9] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 14.234 is stuck peering for 34m, current state remapped+peering, last acting [32,44] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 14.24e is stuck peering for 34m, current state remapped+peering, last acting [17,32] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 14.255 is stuck peering for 34m, current state remapped+peering, last acting [4,22] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 14.260 is stuck peering for 34m, current state remapped+peering, last acting [17,47] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 14.267 is stuck peering for 34m, current state remapped+peering, last acting [44,23] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 14.27d is stuck peering for 34m, current state remapped+peering, last acting [4,21] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 14.289 is stuck peering for 34m, current state remapped+peering, last acting [13,9] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 14.292 is stuck peering for 34m, current state remapped+peering, last acting [15,10] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 14.297 is stuck peering for 34m, current state remapped+peering, last acting [2,21] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 14.29d is stuck peering for 34m, current state remapped+peering, last acting [40,23] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 14.2a8 is stuck peering for 34m, current state remapped+peering, last acting [33,4] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.20e is stuck inactive for 34m, current state remapped+peering, last acting [33,39,2147483647,2147483647,10,43] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.212 is stuck peering for 34m, current state remapped+peering, last acting [36,2147483647,22,40,10,43] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.219 is stuck peering for 34m, current state remapped+peering, last acting [13,10,21,34,2147483647,2147483647] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.21c is stuck peering for 34m, current state remapped+peering, last acting [41,4,2147483647,14,44,15] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.222 is stuck peering for 34m, current state remapped+peering, last acting [2147483647,2147483647,23,32,3,34] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.22d is stuck peering for 34m, current state remapped+peering, last acting [2147483647,45,41,20,17,33] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.233 is stuck peering for 34m, current state remapped+peering, last acting [4,2,27,34,14,2147483647] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.23a is stuck peering for 34m, current state remapped+peering, last acting [41,43,19,2147483647,34,33] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.23b is stuck peering for 34m, current state remapped+peering, last acting [2147483647,30,7,41,34,15] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.23e is stuck peering for 34m, current state remapped+peering, last acting [10,37,2147483647,2147483647,11,9] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.243 is stuck peering for 34m, current state remapped+peering, last acting [23,13,11,15,2147483647,45] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.244 is stuck peering for 34m, current state remapped+peering, last acting [13,35,14,2147483647,17,9] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.249 is stuck peering for 34m, current state remapped+peering, last acting [32,47,2147483647,2147483647,46,17] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.24a is stuck peering for 34m, current state remapped+peering, last acting [47,7,15,5,2147483647,2147483647] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.24b is stuck peering for 34m, current state remapped+peering, last acting [30,2147483647,46,28,4,29] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.24f is stuck peering for 34m, current state remapped+peering, last acting [2147483647,2147483647,13,35,33,5] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.254 is stuck peering for 34m, current state remapped+peering, last acting [15,39,2147483647,2147483647,4,16] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.257 is stuck peering for 34m, current state remapped+peering, last acting [13,10,2147483647,2147483647,22,40] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.25d is stuck peering for 34m, current state remapped+peering, last acting [2147483647,2147483647,20,16,34,46] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.26f is stuck peering for 34m, current state remapped+peering, last acting [33,17,2147483647,14,26,23] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.273 is stuck peering for 34m, current state remapped+peering, last acting [36,2147483647,29,4,17,21] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.27a is stuck peering for 34m, current state remapped+peering, last acting [40,34,2147483647,2147483647,26,23] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.27b is stuck peering for 34m, current state remapped+peering, last acting [41,37,2147483647,19,11,3] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.27e is stuck peering for 34m, current state remapped+peering, last acting [44,9,4,41,45,2147483647] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.281 is stuck peering for 34m, current state remapped+peering, last acting [2,43,21,17,31,2147483647] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.290 is stuck peering for 34m, current state remapped+peering, last acting [17,21,2147483647,2147483647,23,37] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.296 is stuck peering for 34m, current state remapped+peering, last acting [44,3,35,20,2147483647,2147483647] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.298 is stuck peering for 34m, current state remapped+peering, last acting [2147483647,36,43,23,44,9] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.2a5 is stuck peering for 34m, current state remapped+peering, last acting [33,11,19,2147483647,43,10] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.2a6 is stuck peering for 34m, current state remapped+peering, last acting [2147483647,2147483647,4,29,27,34] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.2a8 is stuck peering for 34m, current state remapped+peering, last acting [47,4,2147483647,2147483647,15,5] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.2ae is stuck peering for 34m, current state remapped+peering, last acting [23,32,2147483647,2147483647,3,22] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.2b4 is stuck peering for 34m, current state remapped+peering, last acting [2147483647,2147483647,23,20,15,34] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.2b7 is stuck peering for 34m, current state remapped+peering, last acting [17,3,2147483647,45,41,32] 2024-01-04T09:09:59.996+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : pg 15.2b8 is stuck peering for 34m, current state remapped+peering, last acting [40,44,2147483647,2147483647,43,10] 2024-01-04T09:10:02.572+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 14296421/4091962644 objects degraded (0.349%), 322 pgs degraded, 360 pgs undersized (PG_DEGRADED) 2024-01-04T09:10:02.572+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : Health check update: 9127 slow ops, oldest one blocked for 2096 sec, daemons [osd.10,osd.11,osd.13,osd.14,osd.15,osd.16,osd.17,osd.2,osd.20,osd.21]... have slow ops. (SLOW_OPS) 2024-01-04T09:10:27.608+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : Health check update: Slow OSD heartbeats on back (longest 429402.973ms) (OSD_SLOW_PING_TIME_BACK) 2024-01-04T09:10:27.608+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : Health check update: Slow OSD heartbeats on front (longest 429531.265ms) (OSD_SLOW_PING_TIME_FRONT) 2024-01-04T09:10:27.608+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : Health check update: Degraded data redundancy: 96918631/4091964189 objects degraded (2.369%), 404 pgs degraded, 241 pgs undersized (PG_DEGRADED) 2024-01-04T09:10:27.608+0000 7ff5fd080700 0 log_channel(cluster) log [WRN] : Health check update: 706 slow ops, oldest one blocked for 2121 sec, daemons [osd.10,osd.11,osd.12,osd.13,osd.15,osd.16,osd.17,osd.18,osd.2,osd.20]... have slow ops. (SLOW_OPS) Any ideas what's going on here ?

2 1

Re: Sending notification after multiple objects are created in a ceph bucket.
by Lokendra Rathour 11 Jan '24

11 Jan '24

Facing a similar situation, any support would be helpful. -Lokendra On Tue, Jan 9, 2024 at 10:47 PM Kushagr Gupta <kushagrguptasps.mun(a)gmail.com> wrote: > Hi Team, > > Features used: Rados gateway, ceph S3 buckets > > We are trying to create a data pipeline using the S3 buckets capability > and rado gateway in ceph. > Our endpoint is a kafka topic. > > Currently, as soon as an object is created, a notification is sent to > kafka topic. > Our goal is to create multiple object first and then send the notification. > Also if there is a way such that we can assign objects to a group and > collectively send notification on an event on that group. > > Could anyone please help me? > > Thanks and Regards, > Kushagra Gupta > -- ~ Lokendra skype: lokendrarathour

2 1