- ceph-users - lists.ceph.io

Re: monitoring apply_latency / commit_latency ?
by Konstantin Shalygin 03 Apr '23

03 Apr '23

Hi, > On 2 Apr 2023, at 23:14, Matthias Ferdinand <mf+ml.ceph(a)mfedv.net> wrote: > > I understand that grafana graphs are generated from prometheus metrics. > I just wanted to know which OSD daemon-perf values feed these prometheus > metrics (or if they are generated in some other way). Yep, this perf metrics is generated in some way 🙂 You can consult with ceph-mgr prometheus module source code [1] [1] https://github.com/ceph/ceph/blob/main/src/pybind/mgr/prometheus/module.py#… k

1 0

Set the Quality of Service configuration.
by farhad kh 02 Apr '23

02 Apr '23

how to i can set IO quota or limit for R/W for erasure coding pool in ceph ?

3 2

monitoring apply_latency / commit_latency ?
by Matthias Ferdinand 02 Apr '23

02 Apr '23

Hi, I would like to understand how the per-OSD data from "ceph osd perf" (i.e. apply_latency, commit_latency) is generated. So far I couldn't find documentation on this. "ceph osd perf" output is nice for a quick glimpse, but is not very well suited for graphing. Output values are from the most recent 5s-averages apparently. With "ceph daemon osd.X perf dump" OTOH, you get quite a lot of latency metrics, while it is just not obvious to me how they aggregate into apply_latency and commit_latency. Or some comparably easy read latency metric (something that is missing completely in "ceph osd perf"). Can somebody shed some light on this? Regards Matthias

2 4

avg apply latency went up after update from octopus to pacific
by Boris Behrens 02 Apr '23

02 Apr '23

Hi, today I did the first update from octopus to pacific, and it looks like the avg apply latency went up from 1ms to 2ms. All 36 OSDs are 4TB SSDs and nothing else changed. Someone knows if this is an issue, or am I just missing a config value? Cheers Boris

10 27

Failing to create monitor in a working cluster.
by pmestre＠gmail.com 01 Apr '23

01 Apr '23

Hello, i've been running a 3 node proxmox cluster with 4 ceph osd for 3 years as a production cluster. As a test for trying to move ceph cluster network, i destroyed one of the 3 working monitors and tried to recreate it. After destroying it, the new monitor refuses to join the cluster, even in the old network. I've tried all steps in documentation "Troubleshooting monitors" section. New monitor has this config extracted from ceph --admin-daemon file.asok { "name": "n3ceph", "rank": -1, "state": "probing", "election_epoch": 0, "quorum": [], "features": { "required_con": "2449958197560098820", "required_mon": [ "kraken", "luminous", "mimic", "osdmap-prune", "nautilus", "octopus", "pacific", "elector-pinging" ], "quorum_con": "0", "quorum_mon": [] }, "outside_quorum": [], "extra_probe_peers": [], "sync_provider": [], "monmap": { "epoch": 6, "fsid": "5e60d0bb-33b4-42db-bbe7-7032c35ee605", "modified": "2023-03-31T11:54:44.616569Z", "created": "2019-12-02T13:50:38.097448Z", "min_mon_release": 16, "min_mon_release_name": "pacific", "election_strategy": 1, "disallowed_leaders: ": "", "stretch_mode": false, "tiebreaker_mon": "", "removed_ranks: ": "1", "features": { "persistent": [ "kraken", "luminous", "mimic", "osdmap-prune", "nautilus", "octopus", "pacific", "elector-pinging" ], "optional": [] }, "mons": [ { "rank": 0, "name": "node1", "public_addrs": { "addrvec": [ { "type": "v2", "addr": "10.100.100.1:3300", "nonce": 0 }, { "type": "v1", "addr": "10.100.100.1:6789", "nonce": 0 } ] }, "addr": "10.100.100.1:6789/0", "public_addr": "10.100.100.1:6789/0", "priority": 0, "weight": 0, "crush_location": "{}" }, { "rank": 1, "name": "node2", "public_addrs": { "addrvec": [ { "type": "v2", "addr": "10.100.100.2:3300", "nonce": 0 }, { "type": "v1", "addr": "10.100.100.2:6789", "nonce": 0 } ] }, "addr": "10.100.100.2:6789/0", "public_addr": "10.100.100.2:6789/0", "priority": 0, "weight": 0, "crush_location": "{}" } ] }, "feature_map": { "mon": [ { "features": "0x3f01cfbdfffdffff", "release": "luminous", "num": 1 } ] }, "stretch_mode": false } The quorum mon stat is as follows: { "name": "node1", "rank": 0, "state": "leader", "election_epoch": 340, "quorum": [ 0, 1 ], "quorum_age": 13090, "features": { "required_con": "2449958747317026820", "required_mon": [ "kraken", "luminous", "mimic", "osdmap-prune", "nautilus", "octopus", "pacific", "elector-pinging" ], "quorum_con": "4540138314316775423", "quorum_mon": [ "kraken", "luminous", "mimic", "osdmap-prune", "nautilus", "octopus", "pacific", "elector-pinging" ] }, "outside_quorum": [], "extra_probe_peers": [], "sync_provider": [], "monmap": { "epoch": 6, "fsid": "5e60d0bb-33b4-42db-bbe7-7032c35ee605", "modified": "2023-03-31T11:54:44.616569Z", "created": "2019-12-02T13:50:38.097448Z", "min_mon_release": 16, "min_mon_release_name": "pacific", "election_strategy": 1, "disallowed_leaders: ": "", "stretch_mode": false, "tiebreaker_mon": "", "removed_ranks: ": "1", "features": { "persistent": [ "kraken", "luminous", "mimic", "osdmap-prune", "nautilus", "octopus", "pacific", "elector-pinging" ], "optional": [] }, "mons": [ { "rank": 0, "name": "node1", "public_addrs": { "addrvec": [ { "type": "v2", "addr": "10.100.100.1:3300", "nonce": 0 }, { "type": "v1", "addr": "10.100.100.1:6789", "nonce": 0 } ] }, "addr": "10.100.100.1:6789/0", "public_addr": "10.100.100.1:6789/0", "priority": 0, "weight": 0, "crush_location": "{}" }, { "rank": 1, "name": "node2", "public_addrs": { "addrvec": [ { "type": "v2", "addr": "10.100.100.2:3300", "nonce": 0 }, { "type": "v1", "addr": "10.100.100.2:6789", "nonce": 0 } ] }, "addr": "10.100.100.2:6789/0", "public_addr": "10.100.100.2:6789/0", "priority": 0, "weight": 0, "crush_location": "{}" } ] }, "feature_map": { "mon": [ { "features": "0x3f01cfbdfffdffff", "release": "luminous", "num": 1 } ], "osd": [ { "features": "0x3f01cfbdfffdffff", "release": "luminous", "num": 5 } ], "client": [ { "features": "0x2f018fb87aa4aafe", "release": "luminous", "num": 1 }, { "features": "0x3f01cfbdfffdffff", "release": "luminous", "num": 12 } ], "mgr": [ { "features": "0x3f01cfbdfffdffff", "release": "luminous", "num": 1 } ] }, "stretch_mode": false I tried to get a debug log with ceph daemon mon.n3ceph config set debug_mon 10/10 and restarting the service, but the ceph log file stoped working after i tried that setting. journalctl -u tells me: mar 31 17:35:22 node3 ceph-mon[240916]: 2023-03-31T17:35:22.926+0200 7f49e0699700 -1 mon.n3ceph@-1(probing) e6 get_health_metrics reporting 4 slow ops, oldest is log(1 entries from seq 1 at 2023-03-31T17:30:19.347379+0200) mar 31 17:35:27 node3 ceph-mon[240916]: 2023-03-31T17:35:27.926+0200 7f49e0699700 -1 mon.n3ceph@-1(probing) e6 get_health_metrics reporting 4 slow ops, oldest is log(1 entries from seq 1 at 2023-03-31T17:30:19.347379+0200) mar 31 17:35:32 node3 ceph-mon[240916]: 2023-03-31T17:35:32.926+0200 7f49e0699700 -1 mon.n3ceph@-1(probing) e6 get_health_metrics reporting 4 slow ops, oldest is log(1 entries from seq 1 at 2023-03-31T17:30:19.347379+0200). Any ideas? Cluster is running fine with two monitors, but a reboot in one of the nodes might be a big problem. Kind regards and many thanks.

2 1

Ceph Failure and OSD Node Stuck Incident
by petersun＠raksmart.com 31 Mar '23

31 Mar '23

We encountered a Ceph failure where the system became unresponsive with no IOPS or throughput after encountering a failed node. Upon investigation, it appears that the OSD process on one of the Ceph storage nodes is stuck, but ping is still responsive. However, during the failure, Ceph was unable to recognize the problematic node, which resulted in all other OSDs in the cluster experiencing slow operations and no IOPS in the cluster at all. Here's the timeline of the incident: - At 10:40, an alert is triggered, indicating a problem with the OSD. - After the alert, Ceph becomes unresponsive with no IOPS or throughput. - At 11:26, an engineer discovers that there is a gradual OSD failure, with 6 out of 12 OSDs on the node being down. - At 11:46, the Ceph engineer is unable to SSH into the faulty node and attempts a soft restart, but the "smartmontools" process is stuck while shutting down the server. Ping works during this time. - After waiting for about one or two minutes, a hard restart is attempted for the server. - At 11:57, after the Ceph node starts normally, service resumes as usual, indicating that the issue has been resolved. Here is some basic information about our services: - `Mon: 5 daemons, quorum host001, host002, host003, host004, host005 (age 4w)` - `Mgr: host005 (active, since 4w), standbys: host001, host002, host003, host004` - `Osd: 218 osds: 218 up (since 22h), 218 in (since 22h)` We have a cluster with 19 nodes, including 15 SSD nodes and 4 HDD nodes. In total, there are 218 OSDs. The SSD nodes have 11 OSDs with Samsung EVO 870 SSD and each drive DB/WAL by 1.6T NVME drive. We are using Ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable). Here is the health check detail: [root@node21 ~]# ceph health detail HEALTH_WARN 1 osds down; Reduced data availability: 12 pgs inactive, 12 pgs peering; Degraded data redundancy: 272273/43967625 objects degraded (0.619%), 88 pgs degraded, 5 pgs undersized; 18192 slow ops, oldest one blocked for 3730 sec, daemons [osd.0,osd.1,osd.101,osd.103,osd.107,osd.108,osd.109,osd.11,osd.111,osd.112]... have slow ops. [WRN] OSD_DOWN: 1 osds down osd.174 (root=default,host=hkhost031) is down [WRN] PG_AVAILABILITY: Reduced data availability: 12 pgs inactive, 12 pgs peering pg 2.dc is stuck peering for 49m, current state peering, last acting [87,95,172] pg 2.e2 is stuck peering for 15m, current state peering, last acting [51,177,97] ...... pg 2.f7e is active+undersized+degraded, acting [10,214] pg 2.f84 is active+undersized+degraded, acting [91,52] [WRN] SLOW_OPS: 18192 slow ops, oldest one blocked for 3730 sec, daemons [osd.0,osd.1,osd.101,osd.103,osd.107,osd.108,osd.109,osd.11,osd.111,osd.112]... have slow ops. I have the following questions: 1. Why couldn't Ceph detect the faulty node and automatically abandon its resources? Can anyone provide more troubleshooting guidance for this case? 2. What is Ceph's detection mechanism and where can I find related information? All of our production cloud machines were affected and suspended. If RBD is unstable, we cannot continue to use Ceph technology for our RBD source. 3. Did we miss any patches or bug fixes? 4. Is there anyone who can suggest improvements and how we can quickly detect and avoid similar issues in the future?

5 4

./install-deps.sh takes several hours
by Arvid Picciani 31 Mar '23

31 Mar '23

Hi again, something is very wrong with my hardware it seems and i'm slowly turning insane. I'm trying to debug why ceph has incredibly poor performance for us. we've got - 3 EPYC 7713 dual-cpu systems - datacenter nvme drives (3GB/s top) - 100G infiniband ceph does 800MB/s read max, CPU is idle, network is idle, I/O nowhere near saturation now while hunting for queues, i wanted to recompile ceph and poke around abit, but ./install-deps.sh is now running for 6 hours with the exact same problem. CPU is idle, network is idle, zero I/O what am i doing wrong? is there another computer resource we're bottlenecked on that i just dont know about? -- +4916093821054

1 0

ceph orch ps mon, mgr, osd shows <unknown> for version, image and container id
by anantha.adiga＠intel.com 31 Mar '23

31 Mar '23

Hi , Why is ceph orch ps showing ,unknown version, image and container id ? root@a002s002:~# cephadm shell ceph mon versions Inferring fsid 682863c2-812e-41c5-8d72-28fd3d228598 Using recent ceph image quay.io/ceph/daemon@sha256:9889075a79f425c2f5f5a59d03c8d5bf823856ab661113fa17a8a7572b16a997 { "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 } root@a002s002:~# cephadm shell ceph mgr versions Inferring fsid 682863c2-812e-41c5-8d72-28fd3d228598 Using recent ceph image quay.io/ceph/daemon@sha256:9889075a79f425c2f5f5a59d03c8d5bf823856ab661113fa17a8a7572b16a997 { "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)": 3 } root@a002s002:~# cephadm shell ceph orch ps --daemon-type mgr Inferring fsid 682863c2-812e-41c5-8d72-28fd3d228598 Using recent ceph image quay.io/ceph/daemon@sha256:9889075a79f425c2f5f5a59d03c8d5bf823856ab661113fa17a8a7572b16a997 NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID mgr.a002s002 a002s002 running 4m ago 11M - - <unknown> <unknown> mgr.a002s003 a002s003 running 87s ago 11M - - <unknown> <unknown> mgr.a002s004 a002s004 running 4m ago 11M - - <unknown> <unknown> root@a002s002:~# cephadm shell ceph orch ps --daemon-type mon Inferring fsid 682863c2-812e-41c5-8d72-28fd3d228598 Using recent ceph image quay.io/ceph/daemon@sha256:9889075a79f425c2f5f5a59d03c8d5bf823856ab661113fa17a8a7572b16a997 NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID mon.a002s002 a002s002 running 4m ago 11M - 2048M <unknown> <unknown> <unknown> mon.a002s003 a002s003 running 95s ago 11M - 2048M <unknown> <unknown> <unknown> mon.a002s004 a002s004 running (4w) 4m ago 5M 1172M 2048M 16.2.5 6e73176320aa d38b94e00d28 root@a002s002:~# cephadm shell ceph orch ps --daemon-type osd Inferring fsid 682863c2-812e-41c5-8d72-28fd3d228598 Using recent ceph image quay.io/ceph/daemon@sha256:9889075a79f425c2f5f5a59d03c8d5bf823856ab661113fa17a8a7572b16a997 NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID osd.0 a002s002 running 8m ago 11M - 10.9G <unknown> <unknown> osd.1 a002s003 running 5m ago 11M - 10.9G <unknown> <unknown> osd.10 a002s004 running 8m ago 11M - 10.9G <unknown> <unknown> osd.11 a002s003 running 5m ago 11M - 10.9G <unknown> <unknown> osd.12 a002s002 running 8m ago 11M - 10.9G <unknown> <unknown> osd.13 a002s004 running 8m ago 11M - 10.9G <unknown> <unknown> osd.14 a002s003 running 5m ago 11M - 10.9G <unknown> <unknown> osd.15 a002s002 running 8m ago 11M - 10.9G <unknown> <unknown> osd.16 a002s004 running 8m ago 11M - 10.9G <unknown> <unknown> osd.17 a002s003 running 5m ago 11M - 10.9G <unknown> <unknown> osd.18 a002s002 running 8m ago 11M - 10.9G <unknown> <unknown> osd.19 a002s004 running 8m ago 11M - 10.9G <unknown> <unknown> osd.2 a002s004 running 8m ago 11M - 10.9G <unknown> <unknown> osd.20 a002s003 running 5m ago 11M - 10.9G <unknown> <unknown> osd.21 a002s002 running 8m ago 11M - 10.9G <unknown> <unknown>

3 4

how to set block.db size
by li.xuehai＠99cloud.net 31 Mar '23

31 Mar '23

hi, how to understand ``Improvements in releases beginning with Nautilus 14.2.12 and Octopus 15.2.6 enable better utilization of arbitrary DB device sizes, and the Pacific release brings experimental dynamic level support. `` in the document ``https://docs.ceph.com/en/quincy/rados/configuration/bluestore-config-ref/#sizing`` is there a related patch??

1 0

Upgrade from 16.2.7. to 16.2.11 failing on OSDs
by Lo Re Giuseppe 31 Mar '23

31 Mar '23

Dear all, On one of our clusters I started the upgrade process from 16.2.7 to 16.2.11. Mon and mgr and crash processes were done easily/quickly, then at the first attempt of upgrading a OSD container the upgrade process stopped because of the OSD process is not able to start after the upgrade. Does anyone have any hint on how to unblock the upgrade? Some details below: Regards, Giuseppe I started the upgrade process with the cephadm command: “”” [root@naret-monitor01 ~]# ceph orch upgrade start --ceph-version 16.2.11 Initiating upgrade to quay.io/ceph/ceph:v16.2.11 “”” After a short time: “”” [root@naret-monitor01 ~]# ceph orch upgrade status { "target_image": quay.io/ceph/ceph@sha256:1b9803c8984bef8b82f05e233e8fe8ed8f0bba8e5cc2c57f6efaccbeea682add<mailto:quay.io/ceph/ceph@sha256:1b9803c8984bef8b82f05e233e8fe8ed8f0bba8e5cc2c57f6efaccbeea682add>, "in_progress": true, "which": "Upgrading all daemon types on all hosts", "services_complete": [ "crash", "mon", "mgr" ], "progress": "64/2039 daemons upgraded", "message": "Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon osd.4 on host naret-osd01 failed.", "is_paused": true } “”” The ceph health command reports: “”” [root@naret-monitor01 ~]# ceph health detail HEALTH_WARN 1 failed cephadm daemon(s); 1 osds down; Degraded data redundancy: 2654362/6721382840 objects degraded (0.039%), 14 pgs degraded, 14 pgs undersized; Upgrading daemon osd.4 on host naret-osd01 failed. [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) daemon osd.22 on naret-osd01 is in error state [WRN] OSD_DOWN: 1 osds down osd.4 (root=default,host=naret-osd01) is down [WRN] PG_DEGRADED: Degraded data redundancy: 2654362/6721382840 objects degraded (0.039%), 14 pgs degraded, 14 pgs undersized pg 28.88 is stuck undersized for 6m, current state active+undersized+degraded, last acting [1373,1337,1508,852,2147483647,483] pg 28.528 is stuck undersized for 6m, current state active+undersized+degraded, last acting [1063,793,2147483647,931,338,1777] pg 28.594 is stuck undersized for 6m, current state active+undersized+degraded, last acting [1208,891,1651,364,2147483647,53] pg 28.8b4 is stuck undersized for 6m, current state active+undersized+degraded, last acting [521,1273,1238,138,1539,2147483647] pg 28.a90 is stuck undersized for 6m, current state active+undersized+degraded, last acting [237,1665,1836,2147483647,192,1410] pg 28.ad6 is stuck undersized for 6m, current state active+undersized+degraded, last acting [870,466,350,885,1601,2147483647] pg 28.b34 is stuck undersized for 6m, current state active+undersized+degraded, last acting [920,1596,2147483647,115,201,941] pg 28.c14 is stuck undersized for 6m, current state active+undersized+degraded, last acting [1389,424,2147483647,268,1646,632] pg 28.dba is stuck undersized for 6m, current state active+undersized+degraded, last acting [1099,561,2147483647,1806,1874,1145] pg 28.ee2 is stuck undersized for 6m, current state active+undersized+degraded, last acting [1621,1904,1044,2147483647,1545,722] pg 29.163 is stuck undersized for 6m, current state active+undersized+degraded, last acting [1883,2147483647,1509,1697,1187,235] pg 29.1c1 is stuck undersized for 6m, current state active+undersized+degraded, last acting [122,1226,962,1254,1215,2147483647] pg 29.254 is stuck undersized for 6m, current state active+undersized+degraded, last acting [1782,1839,1545,412,196,2147483647] pg 29.2a1 is stuck undersized for 6m, current state active+undersized+degraded, last acting [370,2147483647,575,1423,1755,446] [WRN] UPGRADE_REDEPLOY_DAEMON: Upgrading daemon osd.4 on host naret-osd01 failed. Upgrade daemon: osd.4: cephadm exited with an error code: 1, stderr:Redeploy daemon osd.4 ... Non-zero exit code 1 from systemctl start ceph-63334166-d991-11eb-99de-40a6b72108d0(a)osd.4<mailto:ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4> systemctl: stderr Job for ceph-63334166-d991-11eb-99de-40a6b72108d0(a)osd.4.service<mailto:ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.service> failed because a timeout was exceeded. systemctl: stderr See "systemctl status ceph-63334166-d991-11eb-99de-40a6b72108d0(a)osd.4.service<mailto:ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.service>" and "journalctl -xe" for details. Traceback (most recent call last): File "/var/lib/ceph/63334166-d991-11eb-99de-40a6b72108d0/cephadm.8d0364fef6c92fc3580b0d022e32241348e6f11a7694d2b957cdafcb9d059ff2", line 9248, in <module> main() File "/var/lib/ceph/63334166-d991-11eb-99de-40a6b72108d0/cephadm.8d0364fef6c92fc3580b0d022e32241348e6f11a7694d2b957cdafcb9d059ff2", line 9236, in main r = ctx.func(ctx) File "/var/lib/ceph/63334166-d991-11eb-99de-40a6b72108d0/cephadm.8d0364fef6c92fc3580b0d022e32241348e6f11a7694d2b957cdafcb9d059ff2", line 1990, in _default_image return func(ctx) File "/var/lib/ceph/63334166-d991-11eb-99de-40a6b72108d0/cephadm.8d0364fef6c92fc3580b0d022e32241348e6f11a7694d2b957cdafcb9d059ff2", line 5041, in command_deploy ports=daemon_ports) File "/var/lib/ceph/63334166-d991-11eb-99de-40a6b72108d0/cephadm.8d0364fef6c92fc3580b0d022e32241348e6f11a7694d2b957cdafcb9d059ff2", line 2952, in deploy_daemon c, osd_fsid=osd_fsid, ports=ports) File "/var/lib/ceph/63334166-d991-11eb-99de-40a6b72108d0/cephadm.8d0364fef6c92fc3580b0d022e32241348e6f11a7694d2b957cdafcb9d059ff2", line 3197, in deploy_daemon_units call_throws(ctx, ['systemctl', 'start', unit_name]) File "/var/lib/ceph/63334166-d991-11eb-99de-40a6b72108d0/cephadm.8d0364fef6c92fc3580b0d022e32241348e6f11a7694d2b957cdafcb9d059ff2", line 1657, in call_throws raise RuntimeError(f'Failed command: {" ".join(command)}: {s}') RuntimeError: Failed command: systemctl start ceph-63334166-d991-11eb-99de-40a6b72108d0(a)osd.4<mailto:ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4>: Job for ceph-63334166-d991-11eb-99de-40a6b72108d0(a)osd.4.service<mailto:ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.service> failed because a timeout was exceeded. See "systemctl status ceph-63334166-d991-11eb-99de-40a6b72108d0(a)osd.4.service<mailto:ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.service>" and "journalctl -xe" for details. “”” On the OSD server we have: “”” [root@naret-osd01 ~]# uname -a Linux naret-osd01 4.18.0-425.10.1.el8_7.x86_64 #1 SMP Wed Dec 14 16:00:01 EST 2022 x86_64 x86_64 x86_64 GNU/Linux [root@naret-osd01 ~]# podman -v podman version 4.2.0 [root@naret-osd01 ~]# ceph -v ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable) [root@naret-osd01 ~]# cat /etc/os-release NAME="Red Hat Enterprise Linux" VERSION="8.7 (Ootpa)" ID="rhel" ID_LIKE="fedora" VERSION_ID="8.7" PLATFORM_ID="platform:el8" PRETTY_NAME="Red Hat Enterprise Linux 8.7 (Ootpa)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" HOME_URL=https://www.redhat.com/ DOCUMENTATION_URL=https://access.redhat.com/documentation/red_hat_enterpris… BUG_REPORT_URL=https://bugzilla.redhat.com/ REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" REDHAT_BUGZILLA_PRODUCT_VERSION=8.7 REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" REDHAT_SUPPORT_PRODUCT_VERSION="8.7" “”” Systemctl says: “”” systemctl status ceph-63334166-d991-11eb-99de-40a6b72108d0(a)osd.4.service<mailto:ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.service> … ● ceph-63334166-d991-11eb-99de-40a6b72108d0(a)osd.4.service<mailto:ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.service> - Ceph osd.4 for 63334166-d991-11eb-99de-40a6b72108d0 Loaded: loaded (/etc/systemd/system/ceph-63334166-d991-11eb-99de-40a6b72108d0@.service<mailto:/etc/systemd/system/ceph-63334166-d991-11eb-99de-40a6b72108d0@.service>; enabled; vendor preset: disabled) Active: failed (Result: timeout) since Mon 2023-03-27 15:34:29 CEST; 6min ago Process: 730621 ExecStopPost=/bin/rm -f /run/ceph-63334166-d991-11eb-99de-40a6b72108d0(a)osd.4.service-pid<mailto:/run/ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.service-pid> /run/ceph-63334166-d991-11eb-99de-40a6b72108d0(a)osd.4.service-cid<mailto:/run/ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.service-cid> (code=exited, status=0/SUCCESS) Process: 730209 ExecStopPost=/bin/bash /var/lib/ceph/63334166-d991-11eb-99de-40a6b72108d0/osd.4/unit.poststop (code=exited, status=0/SUCCESS) Process: 710355 ExecStartPre=/bin/rm -f /run/ceph-63334166-d991-11eb-99de-40a6b72108d0(a)osd.4.service-pid<mailto:/run/ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.service-pid> /run/ceph-63334166-d991-11eb-99de-40a6b72108d0(a)osd.4.service-cid<mailto:/run/ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.service-cid> (code=exited, status=0/SUCCESS) Main PID: 23025 (code=exited, status=0/SUCCESS) Tasks: 62 (limit: 1647878) Memory: 961.8M CGroup: /system.slice/system-ceph\x2d63334166\x2dd991\x2d11eb\x2d99de\x2d40a6b72108d0.slice/ceph-63334166-d991-11eb-99de-40a6b72108d0(a)osd.4.service ├─libpod-payload-b4f0ebebdfec38942b614756b6329b04d2939db29a0a9823e314b848680bc58e │ └─754976 /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug └─runtime └─754965 /usr/bin/conmon --api-version 1 -c b4f0ebebdfec38942b614756b6329b04d2939db29a0a9823e314b848680bc58e -u b4f0ebebdfec38942b614756b6329b04d2939db29a0a9823e314b848680bc58e -r /usr/bin/runc -b /var/lib/containers/storage/overlay-containers/b4f0ebebdfec38942b614756b6329b04d2939db29a0a9823e314b848680bc58e/userdata -p /run/containers/storage/overlay-containers/b4f0ebebdfec38942b614756b6329b04d2939db29a0a9823e314b848680bc58e/userdata/pidfile -n ceph-63334166-d991-11eb-99de-40a6b72108d0-osd-4 --exit-dir /run/libpod/exits --full-attach -l journald --log-level warning --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/run/containers/storage/overlay-containers/b4f0ebebdfec38942b614756b6329b04d2939db29a0a9823e314b848680bc58e/userdata/oci-log --conmon-pidfile /run/ceph-63334166-d991-11eb-99de-40a6b72108d0(a)osd.4.service-pid<mailto:/run/ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.service-pid> --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /run/containers/storage --exit-command-arg --log-level --exit-command-arg warning --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/libpod --exit-command-arg --network-config-dir --exit-command-arg --exit-command-arg --network-backend --exit-command-arg cni --exit-command-arg --volumepath --exit-command-arg /var/lib/containers/storage/volumes --exit-command-arg --runtime --exit-command-arg runc --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mountopt=nodev,metacopy=on --exit-command-arg --events-backend --exit-command-arg file --exit-command-arg container --exit-command-arg cleanup --exit-command-arg --rm --exit-command-arg b4f0ebebdfec38942b614756b6329b04d2939db29a0a9823e314b848680bc58e Mar 27 15:36:56 naret-osd01 ceph-63334166-d991-11eb-99de-40a6b72108d0-osd-4[754965]: debug 2023-03-27T13:36:56.886+0000 7f52e6ae1700 1 osd.4 pg_epoch: 821628 pg[28.dbas2( v 821618'4657799 (819107'4647770,821618'4657799] local-lis/les=749842/749843 n=239290 ec=130297/130290 lis/c=821623/749842 les/c/f=821624/749843/0 sis=821628 pruub=7.751406670s) [1099,561,4,1806,1874,1145]p1099(0) r=2 lpr=821628 pi=[749842,821628)/1 crt=821618'4657799 lcod 0'0 mlcod 0'0 unknown NOTIFY pruub 12.039081573s@ mbc={} ps=[4~6]] state<Start>: transitioning to Stray Mar 27 15:36:56 naret-osd01 ceph-63334166-d991-11eb-99de-40a6b72108d0-osd-4[754965]: debug 2023-03-27T13:36:56.886+0000 7f52e8ae5700 1 osd.4 pg_epoch: 821628 pg[29.163s1( v 821572'139334 (776804'129273,821572'139334] local-lis/les=749851/749852 n=65683 ec=130801/130801 lis/c=821623/749851 les/c/f=821624/749852/0 sis=821628 pruub=8.023463249s) [1883,4,1509,1697,1187,235]p1883(0) r=1 lpr=821628 pi=[749851,821628)/1 crt=821572'139334 lcod 0'0 mlcod 0'0 unknown NOTIFY pruub 12.311203003s@ mbc={}] start_peering_interval up [1883,4,1509,1697,1187,235] -> [1883,4,1509,1697,1187,235], acting [1883,2147483647,1509,1697,1187,235] -> [1883,4,1509,1697,1187,235], acting_primary 1883(0) -> 1883, up_primary 1883(0) -> 1883, role -1 -> 1, features acting 4540138297136906239 upacting 4540138297136906239 Mar 27 15:36:56 naret-osd01 ceph-63334166-d991-11eb-99de-40a6b72108d0-osd-4[754965]: debug 2023-03-27T13:36:56.886+0000 7f52e72e2700 1 osd.4 pg_epoch: 821628 pg[29.2a1s1( v 821500'140649 (776804'130601,821500'140649] local-lis/les=749849/749850 n=65848 ec=130801/130801 lis/c=821623/749849 les/c/f=821624/749850/0 sis=821628 pruub=7.845988274s) [370,4,575,1423,1755,446]p370(0) r=1 lpr=821628 pi=[749849,821628)/1 crt=821500'140649 lcod 0'0 mlcod 0'0 unknown NOTIFY pruub 12.133728981s@ mbc={}] start_peering_interval up [370,4,575,1423,1755,446] -> [370,4,575,1423,1755,446], acting [370,2147483647,575,1423,1755,446] -> [370,4,575,1423,1755,446], acting_primary 370(0) -> 370, up_primary 370(0) -> 370, role -1 -> 1, features acting 4540138297136906239 upacting 4540138297136906239 Mar 27 15:36:56 naret-osd01 ceph-63334166-d991-11eb-99de-40a6b72108d0-osd-4[754965]: debug 2023-03-27T13:36:56.887+0000 7f52e8ae5700 1 osd.4 pg_epoch: 821628 pg[29.163s1( v 821572'139334 (776804'129273,821572'139334] local-lis/les=749851/749852 n=65683 ec=130801/130801 lis/c=821623/749851 les/c/f=821624/749852/0 sis=821628 pruub=8.023443222s) [1883,4,1509,1697,1187,235]p1883(0) r=1 lpr=821628 pi=[749851,821628)/1 crt=821572'139334 lcod 0'0 mlcod 0'0 unknown NOTIFY pruub 12.311203003s@ mbc={}] state<Start>: transitioning to Stray Mar 27 15:36:56 naret-osd01 ceph-63334166-d991-11eb-99de-40a6b72108d0-osd-4[754965]: debug 2023-03-27T13:36:56.887+0000 7f52e72e2700 1 osd.4 pg_epoch: 821628 pg[29.2a1s1( v 821500'140649 (776804'130601,821500'140649] local-lis/les=749849/749850 n=65848 ec=130801/130801 lis/c=821623/749849 les/c/f=821624/749850/0 sis=821628 pruub=7.845966339s) [370,4,575,1423,1755,446]p370(0) r=1 lpr=821628 pi=[749849,821628)/1 crt=821500'140649 lcod 0'0 mlcod 0'0 unknown NOTIFY pruub 12.133728981s@ mbc={}] state<Start>: transitioning to Stray Mar 27 15:36:56 naret-osd01 ceph-63334166-d991-11eb-99de-40a6b72108d0-osd-4[754965]: debug 2023-03-27T13:36:56.887+0000 7f52e72e2700 1 osd.4 pg_epoch: 821628 pg[28.8b4s5( v 821618'2906095 (817032'2896088,821618'2906095] local-lis/les=749842/749843 n=239377 ec=130295/130290 lis/c=821623/749842 les/c/f=821624/749843/0 sis=821628 pruub=8.158309937s) [521,1273,1238,138,1539,4]p521(0) r=5 lpr=821628 pi=[749842,821628)/1 crt=821618'2906095 lcod 0'0 mlcod 0'0 unknown NOTIFY pruub 12.446221352s@ mbc={} ps=[4~6]] start_peering_interval up [521,1273,1238,138,1539,4] -> [521,1273,1238,138,1539,4], acting [521,1273,1238,138,1539,2147483647] -> [521,1273,1238,138,1539,4], acting_primary 521(0) -> 521, up_primary 521(0) -> 521, role -1 -> 5, features acting 4540138297136906239 upacting 4540138297136906239 Mar 27 15:36:56 naret-osd01 ceph-63334166-d991-11eb-99de-40a6b72108d0-osd-4[754965]: debug 2023-03-27T13:36:56.887+0000 7f52e72e2700 1 osd.4 pg_epoch: 821628 pg[28.8b4s5( v 821618'2906095 (817032'2896088,821618'2906095] local-lis/les=749842/749843 n=239377 ec=130295/130290 lis/c=821623/749842 les/c/f=821624/749843/0 sis=821628 pruub=8.158291817s) [521,1273,1238,138,1539,4]p521(0) r=5 lpr=821628 pi=[749842,821628)/1 crt=821618'2906095 lcod 0'0 mlcod 0'0 unknown NOTIFY pruub 12.446221352s@ mbc={} ps=[4~6]] state<Start>: transitioning to Stray Mar 27 15:39:36 naret-osd01 systemd[1]: ceph-63334166-d991-11eb-99de-40a6b72108d0(a)osd.4.service<mailto:ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.service>: Start request repeated too quickly. Mar 27 15:39:36 naret-osd01 systemd[1]: ceph-63334166-d991-11eb-99de-40a6b72108d0(a)osd.4.service<mailto:ceph-63334166-d991-11eb-99de-40a6b72108d0@osd.4.service>: Failed with result 'timeout'. Mar 27 15:39:36 naret-osd01 systemd[1]: Failed to start Ceph osd.4 for 63334166-d991-11eb-99de-40a6b72108d0. “””

2 1